Compare commits

..
Author SHA1 Message Date
Vito Sansevero f6e8977c79 fix(activity): indeterminate pill bar and the nothing-to-do line
The pill's mini bar sweeps while a job's total is unknown instead of
showing an empty fill. A finished thumbnail job with nothing to do shows
the server's final line. Coverage floors raised for gallery.js and
activity.mjs.
2026-10-23 16:31:58 -07:00
Vito Sansevero 7f75533a63 fix(gallery): cancel the stream's own thumbnail job by id
The settings panel and the startup dialog cancel the job named by the
stream's job frame, and only one their stream started. A Cancel before
the frame waits for it, a double Cancel posts once, and a job another
tab or System started keeps running. The startup dialog now settles on
Cancel instead of staying on "Cancelling...".
2026-10-23 16:31:57 -07:00
Vito Sansevero 6f78b4f46f fix(system): thumbnail job review fixes (#249 S5)
- The legacy stream's first frame is job {id, started}, sent after the
  subscription so no runner frame is missed; it reads the registry after
  prepare, so a job started in that gap is attached to, not refused.
- Every exit sets a truthful final status line; nothing to do never
  reads "Processing file 0 of 0".
- Cancel is checked per file during the planning walk; a job cancelled
  there publishes no start frame.
- The ffmpeg fallback keeps the chosen quality (-q:v) and the aspect
  ratio; the placeholder uses the chosen JPEG settings.
- A locked preview on Windows reads "Preview in use by another program;
  the old one was kept", not the raw WinError.
- Thumbnail work no longer defers the monitor's Needs review recording:
  it writes previews only, never records.
- The disconnect test now proves the job outlives its stream (one worker,
  9 files, three batches); the attach test waits on an event.
2026-10-23 16:31:56 -07:00
Vito Sansevero 2b4b415638 Merge remote-tracking branch 'origin/main' into feat/249-system-thumbnails
# Conflicts:
#	tools/vendor/frontend-coverage-baseline.json
#	web/js/workspace/activity-job-view.mjs
2026-10-23 15:14:30 -07:00
Vito b4c7a15045 feat(review): Needs review page, live queue and concept comparison (#288)
* feat(review): add review queue schema with restore validation

Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.

* feat(review): persist ambiguous sampler evidence from scans and new outputs

The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.

Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.

* feat(review): explicit association decisions with compatible linking

Needs review gains its queue reads and decisions:

- GET /prompt_manager/review/items lists pending items oldest first with
  the exact pending total and an HMAC keyset cursor bound to the storage
  epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
  label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
  for a capture of the item's bytes that the current file still hashes to
  and whose evidence is unchanged, through the ingester's compatible-only
  rules in the same transaction as the decision row; a conflict writes
  nothing. Leave unresolved records the decision and links nothing.

ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.

Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.

* fix(review): build the spelling validation test from native paths

On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.

* fix(review): Apply sees the file under every spelling and rechecks only its choice

Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.

Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.

The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.

* fix(review): validate stored review evidence once per startup

Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.

Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.

* fix(review): keep one capture worker for interactive requests

Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.

* fix(review): refresh the recorded spelling when a file is seen under a new one

A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.

* fix(review): final capture store close and recoverable stale cursors

Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.

A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.

* fix(review): make only the shutdown cleanup final

The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.

* refactor(admin): move Needs review recording into its own mixin

_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.

* fix(review): record the review fix round and the skip's residual risks in TESTING.md

The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.

* feat(review): bind queue cursors to the storage epoch only

A decision or a new item (the monitor records one per generation) no
longer makes an open cursor stale: the (created_at, id) keyset stays
ordered when items leave or are appended. A restore still answers
storage-changed and an unverifiable cursor review-conflict.

* feat(review): Needs review page, SAVED VIEWS row and Activity hook

The concept's Metadata review page at #/review: the notice with the
exact pending count, the queue's media cards (shared with Images through
image-card.mjs), and the Resolve the association panel with the sampler
context, the chosen path's prompt blocks, Leave for later and Apply
association. Apply takes a fresh capture; refusals keep the selection
and explain inline; an unanswered decision is resent once. SAVED VIEWS
lists Needs review after Favorites with its count (retires D50), and an
ended scan that recorded review items offers Review metadata in Activity
(D87). Queue items carry the card's media url.

* feat(review): re-read an open Needs review page when its queue changes

An open Needs review page now follows the queue without polling. A new
server-sent event stream, GET /prompt_manager/review/events, sends
"ready" when it opens and "change" after every committed queue change:
an item the scan or the monitor recorded, a spelling refresh, a decision
from any tab, and a restore. database/review_changes.py notifies after
the commit from whichever thread wrote it. The page re-reads its first
page on each event, keeps the open item, its choice and an unanswered
decision, and re-reads the SAVED VIEWS count with it. When the open item
left the queue, the next one opens with a note. A keepalive comment
every 25 s lets the server drop a closed tab; shutdown ends every stream.

Leaving the page mid-decision now releases that decision's capture.

D-NEW-F1 is renumbered D119 in markers and TESTING.md.

* test(review): Needs review page workflows

The review fixture now seeds a real monitor-fallback conflict (the
monitor pipeline links a new output to the most recent prompt, then
queues it), a limited item and a file under no current root, and can add
an output through the monitor or leave one for a metadata scan.

New workflows: Apply links the chosen pair to that file only; a real
association conflict leaves the links unchanged and explains; a limited
item and an unavailable file can only be left unresolved; a tag save
leaves the item pending; keyboard focus, labels and touch targets; an
open page shows items the monitor and a scan add and a decision made in
another tab; leaving the page releases captures, the change stream and
requests. The faked 409 conflict test is replaced by the real one.

* test(review): cover the card Tag button and raise the review floors

The card's Tag on Needs review now has a workflow, and the page's queue
change hook has no unused default. review.mjs and review-state.mjs
floors rise to their measured coverage; no other floor changes.

* feat(review): keep the loaded depth on live re-reads and read once per visit

A queue change re-read only the first 40 items, so the pages Load more had
shown vanished and the page jumped. Each re-read now reads page after page
until it holds as many items as were shown (a restore met after the first
page reads again from the start), so the cards and the scroll position stay.

The stream's ready made every visit read the queue twice. List reads now
answer the queue's version (library_epoch, review_revision) and ready names
the version read after subscribing; the page re-reads on ready only when the
queue it shows is older. The SAVED VIEWS count follows the page's total, so a
change costs no count read. A stream counts as open only once ready is sent.

* test(review): compare Needs review with the approved concept

review-concept.spec.mjs measures the whole Metadata review page as a tree
in the concept and in production, on the review fixture's new
concept-shaped queue (the concept's first three image names and sizes and
its first prompt). mapConceptReview applies only D29, D119, D3, production's
hooks and accessible names and sample data, and the page then matches at
1600x1100 and 390x844 in both themes. The gallery layout follows the
concept at every breakpoint, the SAVED VIEWS rows match with Needs review
after Favorites, the queue's Tag image dialog matches the needs-review
reference, and a dropped notice border or reordered actions fail.

The comparison found one gap: an unavailable file's card lost its media
area. Its empty media button now keeps the concept's 1.33 ratio (D119).
TESTING.md records the comparison, the extended D119 and a summary of the
image-tagging slice (routes, S1, capture limits, the record-changing job,
the no-download rule, the restart note). A ready whose version cannot be
read re-reads the queue (covered by a page test).

* test(review): give the multi-route shell comparison a fitting time budget

The dispatched CI run failed "shell geometry, tokens and controls match the
concept 390x844 light" on WebKit with a 30 s test timeout (no assertion
failed). Each of these tests makes twelve page loads with captures, hovers
and focus checks: 15-19 s on a CI WebKit runner on main, 22-31 s on this
run's loaded shard. With 2 CPUs and 3 workers it fails locally the same way
(6 of 12 timed out); with a 90 s budget, as the other multi-state concept
comparisons have, all 12 pass.

* fix(review): coalesce queue change events and answer queue thumbnails

- The change stream sends the first notice at once and collapses the
  notices of the next 0.4 s into one trailing change, so a busy scan costs
  an open page one re-read per 0.4 s at most (COALESCE_SECONDS).
- Queue items answer thumbnail_url, the same URL the Images listing builds
  (one shared helper, media_thumbnail_url), or null without a thumbnail.
- The "duplicate and refused decision send nothing" test spies on
  notify_review_change instead of reading the next event.

* fix(review): keep the live queue right through applies, Load more and lost streams

- A change heard while a decision is in flight is owed and read once the
  decision is refused or unanswered (I1).
- Load more has its own read: during a re-read it waits and continues from
  the re-read queue; a re-read started during Load more takes the click
  over, and a failed re-read leaves Load more clickable (I2, M7).
- A stream the browser gave up on (503, proxy error) reopens after 3 s,
  doubling to 30 s, with a quiet "Live updates are paused" line while it is
  down (I3, D-NEW-F2).
- A hidden tab closes its stream and a visible one reopens it; the ready
  version keeps that to at most one read (I4).
- Apply fences on the detail's epoch and a re-read under another epoch reads
  the open item again (M3); leaving during the observe releases its capture
  (M4); an unanswered decision that landed finishes as recorded (M5); cards
  follow a re-read's url, target and availability and show thumbnails
  (M6, M10).
- The unreadable-ready test asserts exactly one re-read (M1); the panel's
  long-file-name wrap carries its D119 marker and TESTING.md entry (M8).

* docs(review): number the live-updates-paused deviation D120

* fix(review): refuse monitor recording beside record-changing jobs

System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.

JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.

* test(backup): wait for the restore, not a 5 s timer, in the thread test

test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.

* fix(activity): drop Cancel job from a job that had ended before Activity opened

Since #289 the Activity list removed an item's actions only when they were
isConnected. On the first render the item is not in the page yet, so a job
that had already ended when Activity opened kept its "Cancel job" (and a
running job's progress was moved after its actions, then back). The list
now asks parentNode instead. The review spec "a scan job with review items
offers Review metadata in Activity" failed on this before the change
(Expected 0 buttons on a finished job, Received 1) and passes after it.

* fix(review): store non-finite evidence as null and count pending items

canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.

* fix(review): show the pending count on the System Overview review row

The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.

* feat(review): open Needs review from the System Overview review row

Review metadata on the Overview's review row now opens #/review, the
concept's go-review, instead of staying aria-disabled.

* fix(system): answer the Overview and Statistics only after every read ended

asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.

* fix(review): queue only ambiguous files that nothing links, and heal the queue

The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).

A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.

The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.

The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.

* fix(review): announce items that leave the queue, and seed only queueable files

#284 now queues a file only when its final outcome is ambiguous and unlinked,
and it removes the pending items of files that are now linked. That removal
is a queue change, so it calls notify_review_change after its commit, like
recording and decisions: an open Needs review page re-reads its queue, and
its count follows.

The review browser fixture wrote graphs that the scan's fallback links, and
it produced the conflict item through the monitor's fallback link. Neither is
queued any more. Its files now carry ComfyUI's Infinity stamp, which the
legacy parser cannot read, so they stay unlinked with the same two
candidates. The conflict item is queued first and its file is linked to
another prompt afterwards. "monitor" additions go through
record_review_file from a worker thread.

* fix(review): never queue an ambiguous file with nothing to choose

The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).

A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".

* test(review): seed the limited item from a graph with more candidates than stored

An A1111 block with two negative lines offers no candidate to choose, so
Needs review no longer queues it (#284). The fixture now records a graph of
17 samplers, which the recorder stores limited.
2026-10-23 15:02:18 -07:00
Vito Sansevero c0ef4b6090 Merge remote-tracking branch 'origin/main' into feat/249-system-thumbnails
# Conflicts:
#	py/api/__init__.py
2026-10-23 14:57:02 -07:00
Vito Sansevero 762fda1448 feat(system): review and start thumbnail jobs
Sources' Thumbnail generation -> Review options and Overview's Review
thumbnails open the concept's Thumbnail maintenance dialog: scope, quality
(preselected from Image loading quality) and preview size limit, a plan
line from the read-only thumbnail plan (D99) and Start, which starts the
thumbnails Activity job and opens it. A size outside 128-2048 requests
nothing; a refused start shows the busy link toast.
2026-10-23 14:34:30 -07:00
Vito 3f11ea6168 feat(annotations): Needs review queue backend (#284)
* feat(review): add review queue schema with restore validation

Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.

* feat(review): persist ambiguous sampler evidence from scans and new outputs

The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.

Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.

* feat(review): explicit association decisions with compatible linking

Needs review gains its queue reads and decisions:

- GET /prompt_manager/review/items lists pending items oldest first with
  the exact pending total and an HMAC keyset cursor bound to the storage
  epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
  label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
  for a capture of the item's bytes that the current file still hashes to
  and whose evidence is unchanged, through the ingester's compatible-only
  rules in the same transaction as the decision row; a conflict writes
  nothing. Leave unresolved records the decision and links nothing.

ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.

Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.

* fix(review): build the spelling validation test from native paths

On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.

* fix(review): Apply sees the file under every spelling and rechecks only its choice

Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.

Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.

The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.

* fix(review): validate stored review evidence once per startup

Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.

Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.

* fix(review): keep one capture worker for interactive requests

Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.

* fix(review): refresh the recorded spelling when a file is seen under a new one

A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.

* fix(review): final capture store close and recoverable stale cursors

Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.

A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.

* fix(review): make only the shutdown cleanup final

The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.

* refactor(admin): move Needs review recording into its own mixin

_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.

* fix(review): record the review fix round and the skip's residual risks in TESTING.md

The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.

* fix(review): refuse monitor recording beside record-changing jobs

System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.

JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.

* test(backup): wait for the restore, not a 5 s timer, in the thread test

test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.

* fix(review): store non-finite evidence as null and count pending items

canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.

* fix(review): show the pending count on the System Overview review row

The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.

* fix(system): answer the Overview and Statistics only after every read ended

asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.

* fix(review): queue only ambiguous files that nothing links, and heal the queue

The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).

A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.

The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.

The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.

* fix(review): never queue an ambiguous file with nothing to choose

The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).

A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".

* test(scan): pin only the per-source scope keys beside the review counters

* test(system): match the scan scope keys beside the review counters
2026-10-23 14:19:11 -07:00
Vito Sansevero cb1de9de88 feat(activity): per-kind job descriptors
One descriptor table (activity-kinds.mjs) decides each job kind's pill
verb, toasts, line, tiles, outcomes, openable rows and labels; the
store re-exports it. The scan and image tagging views are unchanged.
A thumbnail job counts Generated, Skipped and Errors, reads
"Generating thumbnails" in the pill and "Thumbnails finished · N
generated · M errors" when it ends.

Any other kind (the health check, a held legacy-page operation) gets
the generic descriptor with no tiles and no Results, instead of the
scan's zero tiles. A running job without a known total shows an
indeterminate progress bar and no percentage, and only a job the
server can cancel shows Cancel.
2026-10-23 13:53:14 -07:00
Vito Sansevero d5bcae301b feat(system): run thumbnail generation as an Activity job
GET /prompt_manager/images/thumbnails/plan counts what a run would do
(targets, fresh, remaining past the cap, source labels) and writes
nothing. POST /prompt_manager/jobs/thumbnails {mode, size, quality}
starts the "thumbnails" job: admission guarded, cancellable at a file
boundary, outcome groups generated / skipped / errors. size is the
longest edge (128-2048), quality the encoder, and rebuild writes fresh
previews again through the same atomic rename, so a failed file keeps
its old preview and originals are only read.

The legacy gallery stream now starts or attaches to the same job and
streams its unchanged frames. A client that goes away no longer stops
the work; the gallery's Cancel asks the server to cancel the job.
2026-10-23 13:31:19 -07:00
Vito 69de20a427 feat(system): Sources & scans (#309)
* feat(system): report configured, effective and watched sources

GET /prompt_manager/settings adds settings.monitoring_enabled, a sources
block (configured roots with valid/code/reason, effective roots with their
listing index and watch state, auto_detected) and the monitor state
(present, observer_alive, scheduled). Every path is a public path.

A source is "watching" only while a live observer scheduled it, "manual"
when monitoring is off or the process has no monitor, and otherwise
"restart_pending".

POST accepts monitoring_enabled as a strict bool, saved under
gallery.monitoring.enabled; restart_required_fields names it when the
running observer does not already match. A save that changes the sources
runs as a "Source change" records hold, so it is refused with 409 busy
while a scan runs. Case variants of one folder are saved once.

POST /prompt_manager/settings/validate-source {path} checks a path
lexically (no "..", no control characters, native path forms only) and
then against the gallery-root rules, answering a code and reason, and
writes nothing.

* fix(system): no existence oracle in source validation

A gallery root's containment in the allowed ComfyUI folders is now
decided before its existence and type, so a path outside them answers
"outside-allowed" whether or not it exists. validate-source also folds
the filesystem-root and home-directory codes into "outside-allowed", with
one reason, and logs the detail server-side only. Paths inside the
allowed folders still say "not-found" or "not-directory". The legacy
settings save shares the same check.

* feat(scan): scan one selected source

POST /prompt_manager/jobs/scan accepts roots: indexes into the effective
sources, strict non-negative ints (no bools or floats), deduplicated, at
most 64, non-empty when present. An index past the sources is a 400
"unknown-source" and starts nothing. The job walks only the selected
sources, resolved when it was requested, and keeps each file's listing
root_index. Its scope records files, roots (null for every source) and
output_folders, the number of folders walked, which the log's first line
counts. The legacy /scan route still scans every source, and attaching to
a scoped job keeps its payload keys.

* feat(system): manage sources and start scans natively

System > Sources & scans renders the concept's section on the server's
sources: one row per effective source with its watch state and a scan
scoped to that source, the 3.2.0 update notice with the legacy dismissal,
Add source checked by validate-source, Remove, monitoring and worker
threads as host drafts behind the one Save row, and Refresh files.
A save refused by a running scan names the job. Thumbnail generation
hands off to the legacy gallery until S5.

Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md.

* fix(security): decide source containment as text before any filesystem access

Every caller (validate-source, POST /settings, config.json at load) now
refuses a source through one check, sources.source_problem: the form
check first, then GalleryConfig.gallery_root_problem, which compares the
normalised spelling with the allowed folders before anything resolves,
stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a
device or any outside folder is refused with no SMB request, no
executor stall and no existence or timing oracle; only then is the path
resolved and contained again, which still catches a symlink out. The
filesystem root and the home directory answer outside-allowed with the
same reason as any outside path, so no caller learns the home path.

Also:
- POST /settings runs the form checks and parses the roots off the loop;
  an unusable source is a coded 400 invalid-source naming its public
  path (as text) and its index.
- ComfyUI folders outside the base directory are path anchors, so their
  sources round-trip and never collide by folder name.
- A per-source scan sends each index with its public path; a stale or
  shifted index is 409 sources-changed and starts nothing.
- A source still watched after monitoring was switched off is
  "stopping".
- Roots are compared as folders, not spellings, so an unchanged legacy
  resend takes no source-change hold.
- Tests: no realpath/stat/lstat of an outside path (native and Windows
  forms with ntpath fakes), the alike answers with /, ~ and a link to ~,
  an event instead of a timer wait, and a teardown that drains jobs
  before the temporary database goes.

* fix(system): say what Sources saved, refused and still watches

- A per-source scan sends the source's public path; when the sources
  changed since they were read the toast says the server's reason and
  the list and saved values are read again.
- "watching until restart" for a source the running monitor still
  watches after monitoring was saved off, and the status adds "Restart
  ComfyUI to apply: Watch for new outputs." from restart_required_fields.
- A save refused for one unusable source marks that row and the status
  is the server's message naming it.
- A valid configured source the effective list holds under another
  spelling reads "same folder as another source".
- The busy toast names the job the 409 named (ApiError keeps a
  validated running job and refused source), falling back to the list.
- The Add source dialog's listeners live on the page scope and Enter
  adds; its markup is compared with the concept (D127).
- TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the
  Sources contract describes the lexical containment, invalid-source and
  sources-changed.

* fix(security): guard the Windows form probes on a token of their own

The no-filesystem guard matched "C:", which on a Windows runner is also
the temporary ComfyUI tree the save may resolve for its trusted first
entry. The probe paths now carry a token no trusted directory holds.

* docs(system): number the Sources states deviation D128
2026-10-23 13:15:06 -07:00
Vito cbc53ee9f1 feat(resolver): trace multi-stage chains through pass-through nodes (#308)
* feat(resolver): trace multi-stage chains through pass-through nodes

The inspection resolver (server and local Inspect) now compares a sampler
with the earlier sampler its latent continues through image-space
pass-through nodes: decode, image filter, model upscale, scale and encode,
listed in a new validated `lineage` section of the semantics table. Equal
prompts resolve (hires-fix, refiner); different prompts are
multistage-differs. A later stage whose positive adds conditioning to the
shared base prompt through an additive composite (ConditioningCombine or
Concat) keeps the base pair, the text legacy linked (three-stage Flux).
Image-edit batches stay ambiguous and run-time text stays untraceable.
Ingestion keeps the default profile (D1).

A card no longer shows a seed when the inspector names a chain stage
whose seed differs from the parser's pick.

* feat(resolver): leave out run-time PromptManager parts, labelled

Maintainer option A: when a PromptManager prepend_text or append_text is
fed directly by a node whose output is decided at run time (LoraManager's
TriggerWord Toggle, now a runtime class, or PromptSearchList), the prompt
resolves to the node's own text, as legacy showed. The side's source is
runtime-parts-omitted and the resolved reason says that text added at run
time, such as trigger words, is not included; the prompt block shows that
line (D-NEW-R1). The toggle's output is never computed. Literal and
traced parts are combined as before; an indirect run-time part, a run-time
main text or an unknown feeder keep their outcome. The browser port
changes identically. Ingestion is unchanged.

* fix(resolver): keep run-time and added-conditioning labels honest

A text that left out a run-time PromptManager part is now labelled when
any node on its traced path omitted it, so the label survives encoders,
concatenations and chained PromptManagers. Comparisons include what a
text leaves out, so a partial prompt never equals a complete one: a
branch with trigger words and one without stay ambiguous, and such
stages are multistage-differs. Only classes listed as run time are left
out.

The added-conditioning floor labels its positive (D126, "Conditioning
added at a later stage is not included."); the run-time line is D125.

A card hides its seed only when the stage the inspector names reads a
different seed than the parser; an agreeing seed is kept, as on main.
The browser port moves its lineage, latent and part helpers into
local-resolver-stages.mjs, keeping parity.
2026-10-23 11:09:12 -07:00
Vito 7c25b351f7 test(comfyui): refuse a real-host port another run holds (#310)
The real-host config waited on the /system_stats URL, which any server on
the port answers. When two runs started on 8189 within a few seconds of
each other, the second run's ComfyUI failed to bind ("Port 8189 is already
in use"), but Playwright had already seen the first run's host answer and
ran the second run's tests against it. When the first run finished and
stopped its host, the second run failed mid-test with ECONNREFUSED,
ECONNRESET or ERR_CONNECTION_REFUSED, and passed when rerun alone.

The launcher now refuses a port that is in use, and reports ready only
when /system_stats names its own base directory. The config waits for
that line instead of the URL, and every test checks again that the host
answering the port is the one its run started.
2026-10-23 09:25:32 -07:00
Vito 785b744554 feat(system): Appearance section and browser display defaults (#307)
* feat(system): share browser display defaults

The workspace reads and writes the legacy gallery's gallerySettings object
through preferences.mjs: workspaceLayout and workspaceDensity join the
shared keys, a write re-reads the stored object and writes only the changed
keys, denied storage or a full quota keeps the change for this tab and says
so, and storage events report changes from other pages. The legacy gallery
Save keeps keys it does not edit, and the viewer reads its video defaults
through the shared module.

* feat(system): Appearance defaults used by Images and the viewer

System > Appearance is the concept's Browsing & display surface on real
values: the browser rows edit this browser's display defaults through the
shared gallerySettings object, and Open from ComfyUI and the node search
auto-hide save to the server settings with their revision. A new Images
visit starts from the default layout and size (the current view and saved
views keep theirs); metadata off waits for Load metadata; cache off re-reads
the shown file; paths off hides the Path row; the viewer reads the video
defaults. D97 adds card details, collapsed viewer details and Reset display
defaults; storage refused says Saved for this tab only.

* fix(system): flag changes made while System was closed and keep embedded galleries fixed

Final review fixes for System S3 (Appearance):

- I1: the on-demand metadata deviation is D124 (TESTING.md, inspector
  markers); TESTING.md names the spec that pins it, since no concept
  comparison renders that state.
- I2: System checks the display defaults for conflicts on mount, not only
  on storage events, and a browser write refuses a draft another page
  overtook, so a draft is never saved over another tab's value without
  the conflict status first. New two-tab spec.
- I3: the layout and size defaults start the Images page only; embedded
  galleries (a prompt history's Shared gallery and dialogs) keep
  Grid/Comfortable.
- M1: host rows whose /settings read failed say "Server settings could
  not be read."
- M2: a rejected timeout shows the "Some changes were not saved" status.
- M3: the browser loads on mount and on storage events never reject
  unhandled.
- M4: the next accepted write stores the tab-only changes and clears the
  overlay.
- M5: cache off never re-reads a source-changed selection, and refresh()
  never loads a deferred one.
- M6: the System fixture refuses a config.json outside its temp root and
  restores it with os.replace.
- M7: the inspection read counts wait on a request barrier, not frames.
- M8 recorded as an open saved-views question in TESTING.md.

* fix(system): answer a failed System read only after its other reads finish

CI (py3.10 on ubuntu) crashed with a segfault inside SQLite in
test_statistics_failure_is_an_error_envelope: the statistics and overview
routes used asyncio.gather without return_exceptions, so the 500 answered
on the first failure while get_statistics still ran in the executor, and
the test tore the database down under it. Both routes now await every
read before raising the first failure (failing before, passing after:
TestFailedReadsSettleFirst).

The frontend coverage merge failed on workspace/activity.mjs branches
(86.3 < 87): no test asserted Activity's empty and failed-read states, so
their branch was only reached in passing, depending on timing. A spec now
asserts both.
2026-10-23 07:49:39 -07:00
Vito 966a25d129 fix(scan): ingest metadata whose graph holds NaN or Infinity (#303)
* fix(scan): ingest metadata whose graph holds NaN or Infinity

ComfyUI writes its prompt with Python's default json.dumps, so a node whose
IS_CHANGED returns float("nan") saves "is_changed": [NaN] in every PNG. The
exact-pair ingestion stored the snapshots with allow_nan=False, so the scan
failed those files ("Out of range float values are not JSON compliant: nan")
and the monitor dropped their link. The legacy pages replaced NaN with null
before JSON.parse and ingested them.

- utils/json_strict.dumps_strict: strict JSON with non-finite numbers as
  null; identical bytes for every other value.
- Pair snapshots and provenance, and the plain prompt link, store strict
  JSON (the legacy link wrote bare NaN into the database).
- The browser-parser scan fallback parses records as the legacy pages do
  (parse_page_json), so a graph with NaN still traces its prompt.
- The prompt images route answers strict JSON instead of regex-replacing
  "NaN" (which also rewrote prompt text such as "ratio: NaN"); the NaN
  cleaners also null Infinity.

Prompt text and its hash are unchanged: only numbers become null.

* fix(scan): leave the browser-parser NaN parse to the resolver parity fix

The scan fallback's legacy-page parse (parse_page_json in _graph_prompt_text)
is owned by the resolver parity change, which adds the same helper with the
legacy pages' exact rule (NaN reads as null, Infinity still rejected). This
branch keeps only the strict serialization of stored and answered JSON.
2026-10-23 06:48:21 -07:00
139 changed files with 21957 additions and 962 deletions
+760 -27
View File
@@ -121,6 +121,13 @@ npm run --prefix tools/vendor test:comfyui
CI performs this setup automatically. `PM_COMFY_URL` can select another
`http://127.0.0.1:PORT/` address (default port 8189). This is an owned managed
server: a running host is never reused. Do not point it at your live checkout.
One port serves one run. The launcher refuses a port that is already in use and
reports ready only when `/system_stats` names its own base directory, so two runs
started together on 8189 fail at startup ("already in use" or "answered by a
ComfyUI host this run did not start") instead of one run testing the other's
host and losing it mid-test (`ECONNREFUSED`, `ERR_CONNECTION_REFUSED`, Network
Error). Every test checks the answering host again. Agents sharing a machine
give each run its own `PM_COMFY_URL` port.
The launcher verifies the upstream SHA and clean tracked files, snapshots only
tracked PromptManager files, and creates temporary user/input/output/database/
configuration/log directories. Personal untracked files are excluded. It runs
@@ -385,7 +392,8 @@ with source `zeroed-conditioning`. `easy ifElse` with a literal boolean follows
branch only; with a linked boolean both branches are traced, equal texts resolve and
different texts make the sampler `ambiguous` with one candidate per choice (at most
32, else `work-limit`). ConditioningCombine/Concat/Average resolve only when every
part has the same text, otherwise `unresolved` with `composite` naming the node.
part has the same text, otherwise `unresolved` with `composite` naming the node
(but see the added-conditioning floor below for a later stage).
Results that depend on a forced choice are never memoized. A `FaceDetailer`
(`detailer-stage`) is a sampler stage: the samplers reached through its `image`
ancestry are compared with it. Equal texts resolve with the detailer as the sampler;
@@ -396,9 +404,65 @@ A sampler's latent ancestry is walked through every link input of non-sampler
nodes within the same visit and depth budgets: with no earlier sampler it is a
single stage whatever the origin (VAEEncode, EmptyLatentBatch, EmptyFlux2LatentImage,
EmptyHunyuanLatentVideo, SetLatentNoiseMask, InpaintModelConditioning or
WanImageToVideo slot 2). An earlier sampler (img2img from a decode, hires fix) stays
`unresolved` "Multistage sampler ancestry is ambiguous" until stage texts are
compared; an unlisted node with sampler-shaped inputs (positive, negative, guider,
WanImageToVideo slot 2). Pass-through lineage (#304 follow-up, maintainer option B,
2026-10-23): the table's `lineage` section lists the outputs whose image or latent
continues one input's (`VAEDecode`/`VAEDecodeTiled` `samples`, `VAEEncode`/
`VAEEncodeTiled` `pixels`, `ImageUpscaleWithModel`, `ImageScale`, `ImageScaleBy`,
`ImageScaleToTotalPixels` `image`, `LatentUpscale`/`LatentUpscaleBy` `samples`,
cg-image-filter `Image Filter` slot 0 `images` and slot 1 `latents`; a selection of
the same batch), plus Reroutes. Lineage describes no prompt, so its classes are not
claimed by a prompt category. When the latent walk finds exactly one earlier sampler
and the latent descends from that sampler's output slot 0 only through lineage
nodes (a hires-fix or refiner chain, img2img from an in-graph decode, a direct
latent link), the two stages are compared like a detailer and its upstream stage:
equal role texts resolve with the later sampler named, different texts are
`ambiguous` `multistage-differs` with one candidate per stage, and an unreadable
earlier stage makes the chain `unresolved` with that stage's reason. Unanchored, the
earlier sampler is judged inside the chain when its output reaches no other file
output. A size or resolution node reading the same image beside the lineage (a
`ResolutionCalculator` feeding `ImageScale` width and height) is allowed, since the
walk then still finds only that sampler. Any other earlier sampler, or one reached
only through an unlisted image node, stays `unresolved` "Multistage sampler ancestry
is ambiguous". Added-conditioning floor: a later stage whose own positive fails only
as `composite` keeps its earlier stages' shared pair ("A later stage adds
conditioning to the base prompt", naming the earlier stage's sampler) when that
positive reaches, through Reroutes and conditioning pass-throughs, a composite the
table marks `additive` (ConditioningCombine, ConditioningConcat; never
ConditioningAverage, which can weight a part away), one of its parts is exactly the
earlier stages' positive text, and its negative text equals theirs. This is the
three-stage Flux run whose UltimateSDUpscale combines the PromptManager prompt with
a style-trigger encoder: legacy linked the PromptManager text and every stage is
conditioned on it. The positive is always labelled (maintainer ruling): its source is
`added-conditioning` (D126; with a D125 omission too,
`runtime-parts-omitted-and-added-conditioning`) and its provenance names the
composite (`added_by`), whatever the added text is; no length heuristic. A combine without the base text, a different negative, or
different earlier stages stay as they were (`composite` or `multistage-differs`).
An image-edit batch (several samplers with distinct prompts, one shared saver
prefix) stays `ambiguous`, and text generated at run time stays untraceable
(`tests/test_resolver_pass_through.py` on synthetic shapes from
`tests/passthrough_graphs.py`; the browser port is compared on the named
`select-*` chain cases and `passthrough_fuzz_cases`). Ingestion keeps the default
profile's historical latent rule (D1), so the admin scan still reaches these files
through its legacy-parser fallback. Run-time PromptManager parts (maintainer option
A, 2026-10-23): LoraManager's `TriggerWord Toggle` is a `runtime` class (its output is
the LoRA's trigger words as filtered at run time). A PromptManager (or
PromptManagerPair) `prepend_text` or `append_text` linked directly to a `runtime`
node is left out: the text is the node's own, combined with any other literal or
traced part exactly as before, the side's source is `runtime-parts-omitted`, and a
resolved pair's reason is "Text added at run time, such as trigger words, is not
included". The prompt block shows that line under the text (D125; the evidence
badge stays "Workflow inferred"). The label follows the text through encoders,
concatenations and chained PromptManagers: a side is labelled when any node on its
traced path left a part out, and its provenance lists those PromptManager nodes
(`omitted_parts`). Comparisons include what a text leaves out, so a prompt that left
out a part never equals a complete one (a branch with the toggle and one without stay
ambiguous; such stages are `multistage-differs`). Only classes in the table's
`runtime` list are left out, never another `runtime-value` source (a JoinStringMulti
with a linked list flag stays `runtime-value`). The toggle's output is never computed. A run-time node further upstream (inside a
concatenation), a run-time main `text`, or an unknown feeder keep their outcome
(`runtime-value` or `unsupported-node`). This is what legacy linked for the June
hires-fix runs (`TestRunTimePromptManagerParts`, the `select-runtime-*` parity
cases); an unlisted node with sampler-shaped inputs (positive, negative, guider,
latent_image, a pipe, video embeds, custom-sampler parts, or a `model` link together
with conditioning, neg_conditioning, samples or latent) is `unsupported-node`
naming it, here and in a detailer's image ancestry; a latent that is not a link or
@@ -563,8 +627,9 @@ Recorded shell deviations:
`a saved view row matches the concept's` saves one view in both documents and
compares the rows; `tests/browser/workspace-saved-views.spec.mjs` covers saving,
reopening, the defaults that replace rejected values, and failed reads and saves.
- **D50**: no "Needs review" row in SAVED VIEWS until the tag-review slice (maintainer
ruling, 2026-10-21). The comparison maps only that row out.
- **D50** (retired by #249 PR F, ruling R5): SAVED VIEWS now lists the concept's
"Needs review" row after Favorites, with its count; the comparison maps nothing out
for it (only the concept's sample count "3" reads as the fixture's "0").
- **D51** (maintainer ruling, 2026-10-21, beyond the concept): SAVED VIEWS starts with
the concept's Favorites row (star icon, name, count). Its count is the Favorites
listing's own `limit=1` total (`/images/output?mode=workspace&favorites=1`), follows
@@ -1288,8 +1353,8 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f
the selection bar's tagging shows is the concept's or the plan's (D85, D90, D91).
- **D89** (plan D66, Activity for the `image-tags` job): the job's Activity item and its
view are the scan's (D54, D56, D57, D59) with the job kind's own counters, from one
per-kind table (`JOB_VIEWS` in `activity-store.mjs`; a kind without one keeps the
scan's tiles under "Job log" and "Cancel job"). The titles are the server's: "Add
per-kind table (`activity-kinds.mjs` since S5, D111; a kind without one gets no
tiles, "Job log" and "Cancel job"). The titles are the server's: "Add
tags to images" and "Auto-tag images".
- The pill's verb is "Tagging" ("Tagging · 41% · 8,240 / 19,622").
- The item line and the view's heading line read "Started HH:MM · N images · closing
@@ -1322,7 +1387,7 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f
finished · N tagged · M errors", "<title> cancelled · N tagged" or "<title> failed ·
N tagged · M errors".
- Tests: `tests/js/workspace-activity-store.test.js` ("image-tags jobs get their own
tiles outcomes and pill verb", "unknown kinds keep the generic view");
tiles outcomes and pill verb", "unknown kinds get the generic view without the scan tiles");
`tests/browser/workspace-activity.spec.mjs` ("an image tagging job shows Tagged
Unchanged Skipped Errors and opens tagged rows", "cancel tagging stops at a file
boundary", "the report download names the kind", on fake `image-tags` jobs from
@@ -1816,6 +1881,187 @@ npm run --prefix tools/vendor test:workflows -- inspect-concept.spec.mjs workspa
npm run --prefix tools/vendor test:comfyui -- inspect-sender.spec.mjs
```
### Needs review page and recorded deviations (#249 PR F)
`#/review` is the concept's Metadata review page (app.js `renderReview` a749–760) over
the review routes (`web/js/workspace/review.mjs`, `review-state.mjs`): breadcrumb and
document title "Review" (the concept's), heading "Metadata review" with its line and
**Open activity** (`#/activity`), the `.notice` "N images need a decision" with the
concept's sentence, then `.gallery-layout`: the queue's concept media cards
(`image-card.mjs`, shared with Images; the image opens the shared viewer, Info selects
the item, Tag opens the shared Tag image dialog with the review scope note) and
`section.surface` "Resolve the association" (filename, Tag image, Sampler context,
the chosen path's prompt blocks, Leave for later and primary Apply association). The
first item is selected on load, as the concept's first image is. Sampler context lists
"Sampler <id> · <positive> / <negative>" per candidate (the server's label), disables
unusable ones, preselects the first workable one and ends with "Leave unresolved"
(rulings R2/R3); "Leave unresolved" shows the ambiguous status in both blocks. Apply
takes a fresh capture (`observe`) and posts the decision; Leave unresolved on a file
under no current root posts without a capture. Both actions return to Images (R7); a
recorded decision re-reads the row's count, as does every metadata scan that ends. A
refused decision keeps the selection and shows the server's sentence (the
association conflict's explanation included) as `p.warning-note` in the panel; a
restored storage or a changed queue reads the queue again; a request with no answer
keeps its capture and body, and the next Apply of the same choice resends it once.
The SAVED VIEWS row "Needs review" (review icon) follows Favorites with the exact
pending count (`GET /prompt_manager/review/items?limit=1`, its `total`).
- **D86** (plan D63, ruling R6): an empty queue shows only the notice, "No images need a
decision" and "Ambiguous images are added when a metadata scan or a new generation
finds more than one sampler path."
- **D87** (plan D64): an ended metadata scan whose `scope.review_items` is above 0
carries the concept's **Review metadata** (app.js a738 `go-review`): in its Activity
item after the item's line, and in its job view's heading after Download report. The
item's chip still names its state (D54), where the concept's says "Needs attention".
- **D119**: the queue cards have no selection checkbox and
no star (the page has no selection bar and the queue carries no favorite state); a
file under no current root has an empty, `aria-disabled` image button and Tag; more
than 40 items add the Images page's `.infinite-continuation` ("40 of 45 images",
Load more images) after the layout; loading and a failed read are the page's status
line (with Retry); a limited or unavailable file and the decision states are text in
the panel's `p.warning-note`; the toasts "Association accepted." and "Left
unresolved." replace the concept's prototype toast. The concept's "Sample review
queue" chip and "sample" wording are prototype copy (D29). When a re-read (below)
finds the open item gone, the next one opens with "<file> was decided elsewhere, so
the next image is open."; a failed re-read keeps the queue and says "The review queue
could not be read again. Reload the page to see the latest images." A queue card's
title is its file name and its line the extension only (an undecided item names no
prompt, and D11 omits the size and seed a queue item does not carry); its image's alt
is the file name; the panel's Copy and Send buttons carry no prompt title in their
names; the Tag image dialog opened from the queue heads with the file name. An
unavailable file's empty media button keeps the concept's 1.33 media area, so its
card lines up with the others. A long file name in the panel's `p.small` wraps inside
the panel (`overflow-wrap: anywhere`), where the concept's short sample name never
overflows. A queue card shows the Images listing's thumbnail when the server knows one
(`thumbnail_url`, the same URL the Images page uses), else the original; a re-read
rebuilds a card whose media, target or availability changed (a root came or went), so
Open and Tag follow it. When an unanswered decision's item leaves the queue in a
re-read, the decision was recorded: the page finishes as a recorded decision does (its
toast, back to Images), not with "decided elsewhere".
- **D120** (the concept has no live-update state): while the
change stream is down, the heading carries the quiet status line "Live updates are
paused: changes made elsewhere appear once the connection returns." (`p.small
.review-live`, `role="status"`); the stream's next `ready` hides it.
**Live queue (F2).** An open Needs review page holds one server-sent event stream,
`GET /prompt_manager/review/events` (`py/api/review_events.py`): `event: ready` when it
opens, then `event: change` after each committed queue change (an item the scan or the
monitor recorded, a spelling refresh, a decision from any tab; `database/review_changes.py`
notifies after the commit, from whichever thread wrote it) and after a restore. `change`
carries no data. `ready` carries the queue's version `{library_epoch, review_revision}`,
read after the stream subscribed, and every list read answers the same pair
(`review_revision` is the review index revision, which every queue change moves; a
restore moves the epoch). **One read per visit (F3):** the page re-reads on `ready` only
when the queue it shows is older than that version (an unknown version, `{}` when it
cannot be read, re-reads); a `ready` heard while the first read runs is compared once that
read answers. The test server counts a stream as open only once it has sent `ready`, so a
browser test that changes the queue after that count is a `change`, never part of the
version. The page re-reads on every `change`, keeping the open item, its choice and an
unanswered decision; re-reads asked for while one runs collapse into one more. A change
heard while a decision is in flight is owed: a refused or unanswered decision reads it
once the panel is idle again (a recorded one leaves the page). Load more asked for during
a re-read waits for it and continues after the queue it read; a re-read started during
Load more takes that click over and reads one page deeper, and a failed re-read leaves
Load more clickable. Apply fences on the epoch of the detail it decides: an observe under
another storage releases its capture and reads the queue and the item again, and any
re-read under another epoch reads the open item again. **Every re-read keeps the loaded depth (F3):** it
reads page after page (40 each, the route's limit) until it holds as many items as Load
more had shown, so the cards stay and the scroll position does not move; a restore met
after the first page reads once more from the first page. The row's count follows the
total the page shows (no count read per change). Nothing polls: a `: keepalive` comment
every 25 s only lets the server drop a closed tab's subscription. The server sends the
first change at once and collapses the notices of the next 0.4 s into one trailing
`change` (`COALESCE_SECONDS`), so a busy scan costs an open page one re-read per 0.4 s at
most. Shutdown (`on_shutdown`) ends every stream and answers a later one 503. A
reconnect the browser gives up on (a 503, or a proxy's error while ComfyUI restarts) is
opened again by the page after 3 s, doubling to 30 s, with D120's line shown while
the stream is down; its `ready` re-syncs the queue. The stream lives only while the tab is
visible: a hidden tab closes it and a visible one opens it again, and that `ready` reads
only when the queue changed meanwhile. A duplicate recording and a refused decision change
nothing and send nothing. Residual: each visible Needs review page holds one HTTP/1.1
connection (browsers allow six per origin, shared with ComfyUI's editor), so several
side-by-side visible windows on the page, during a metadata scan's job stream, can still
queue the origin's other requests; a re-read of a deep queue costs one request per 40
items shown.
Pinned by `tests/test_api_review_events.py`, `tests/js/workspace-review-state.test.js`
("refresh …", "a refresh keeps the loaded depth …", "ready naming …") and
`tests/browser/workspace-review.spec.mjs` ("an open page shows …", "a visit reads its
queue once …", "a live re-read keeps the loaded depth and the scroll position", "a stream
the browser gives up on …", "a hidden tab closes its stream …", "a re-read refreshes a
card …"); the controller's owed, Load more and epoch cases are "I1: …", "I2: …", "M3:
…", "M5: …" and "M7: …" in `workspace-review-state.test.js`, and the coalescing is "a
burst of changes …" in `test_api_review_events.py`.
Leaving the page mid-decision releases that decision's capture (the server may never see
the request), and leaving during the observe releases the capture it then answers
("leaving during the observe …").
**Concept comparison (F3).** `tests/browser/review-concept.spec.mjs` measures the whole
page (`reviewInventory` in `tests/browser/tagging-canonical-comparison.mjs`: every shown
box below `main` with its place, fonts, edges and layout, a leaf's text, a button's name,
a select's options) in the concept at `#review` and in production on the fixture's
concept-shaped queue (`POST /__test__/review/reset {items: 3, concept: true}`: the
concept's first three image names and sizes, and the concept's first prompt on the open
item), and `mapConceptReview` applies only: D29 (no "Sample review queue" chip, no
"sample" in the panel heading and Apply), D119 (above), D3 (the grid's
`position: relative`), production's hooks and accessible names (`review-queue`,
`review-panel`, the list roles, the region names) and sample data (the candidate options
except Leave unresolved; widths and heights that follow from the changed copy, moved by
exactly that much). It then matches at 1600×1100 and 390×844 in both themes on all four
projects; the gallery layout follows the concept at 1600, 1200, 900 and 390px (310px,
275px with a 15px gap, 280px, one column); the SAVED VIEWS rows match with Needs review
after Favorites; the queue's Tag image dialog matches the concept's needs-review dialog
(`needs-review-tagging.png`) under the C4 mapping; and the comparison is shown to notice
a dropped notice border and reordered actions. The paired captures (populated queue in
both themes and widths, conflict, limited item, unavailable file, empty queue) are test
artifacts; no screenshot baseline is added. The comparison found one gap, the collapsed
media area of an unavailable file's card, fixed above (`workspace-review.spec.mjs` "a
limited item and an unavailable file …" pins it).
- **D3** and **D20** apply to the queue grid (its cards stay in the grid's flow; it is
not virtualized) and to the panel's prompt blocks (their mount point is transparent).
Tests: `tests/browser/review-concept.spec.mjs` (concept comparison),
`tests/js/workspace-review-state.test.js` (controller),
`tests/js/workspace-state.test.js` (route and count),
`tests/browser/workspace-review.spec.mjs` (queue, count, apply, leave, Tag, empty
queue, Load more, Activity's Review metadata; items recorded by the real recorder in
`tests/browser/workspace_review_fixture.py`), and the SAVED VIEWS rows in
`shell-concept.spec.mjs` and `workspace-saved-views.spec.mjs`.
### Native image tagging and Needs review (#249 PRs A–F)
One place for the whole image-tagging slice; the details live in the sections named.
- **Routes.** Tagging: `POST /prompt_manager/image-annotations/observe`, `GET`/`DELETE
/prompt_manager/image-annotations/capture/<token>`, `POST
/prompt_manager/image-annotations/suggest`, `PUT /prompt_manager/image-annotations/tags`
("Tag image dialog and recorded deviations"). Needs review: `GET
/prompt_manager/review/items?limit=1..40&cursor=` (answers `library_epoch`,
`review_revision`, `total`), `GET /prompt_manager/review/items/<id>`, `POST
/prompt_manager/review/items/<id>/decision` and the change stream `GET
/prompt_manager/review/events` ("Needs review decisions and queue reads", "Needs review
page and recorded deviations"). Every legacy route keeps its payload; the only legacy
change is `409 model-busy` from the legacy Auto-tag routes while another model
operation holds the shared gate.
- **S1 (fingerprint-current reads, ruled 2026-10-22).** The Images Tags filter, search and
the inspector/viewer chips use a byte record only while the file's `(st_size,
st_mtime_ns)` equals its verified fingerprint; writes hash the bytes. Residual risk
under "Image annotations schema".
- **Capture limits.** 256 MiB per file, 8 completed captures, 512 MiB reserved, 2
capture/verify workers, 900 s idle, 1800 s hard; the browser fixtures release every
capture on reset, and leaving the Needs review page mid-decision releases its capture.
- **The record-changing job.** Bulk Add tags and Auto-tag selected run as the
`image-tags` Activity job (PR D); like the metadata scan it is a record-changing job, so
a start while another runs is refused (`busy`). Needs review items come from that scan
and from the live monitor, and a decision changes links only when compatible
(`association-conflict` otherwise, with zero writes).
- **No downloads.** No automated test downloads or loads a model: the browser server
injects `TestModel` (its `download_model` raises), native routes and jobs never
download, and every tagging spec asserts no `/autotag/download/` request.
- **Restart.** PRs A, B, D and E and PR F (the review change stream, the queue version)
change Python: the maintainer restarts ComfyUI once after merging. HTML, CSS and JS
changes need only a browser refresh.
### Native System (#249 System)
`#/system/<section>` is a native workspace route rendered by `web/js/workspace/system.mjs`
@@ -1834,8 +2080,155 @@ browser fields, then the host fields (`POST /prompt_manager/settings` with
highlights their controls; `settings-changed` is a conflict and keeps the draft. System
reads the host values on mount and again after a conflict; until a read returned a
revision, a host save fails without a request, so it never sends a blind
`expected_revision: null` (which S1 merges into the file on disk). No delivered section
edits a setting yet, so Save stays disabled until Appearance (S3).
`expected_revision: null` (which S1 merges into the file on disk). Save stays disabled
until a section has an unsaved change.
Appearance (`system-appearance.mjs`, S3) is the concept's "Browsing & display" surface
on real values. Its browser rows are this browser's display defaults in the legacy
gallery's own `gallerySettings` object (`web/js/workspace/preferences.mjs`, Q3): Default
layout and Thumbnail size (`workspaceLayout` grid/list/masonry, `workspaceDensity`
comfortable/large/compact, new keys), Load metadata on selection (`autoLoadMetadata`),
Cache metadata in this browser (`cacheMetadata`), Image loading quality (`imageQuality`:
Optimized→`medium`, Original→`high`, Fast→`low`; the thumbnail dialog's preselected quality
once S5 lands), the three video rows (`videoAutoplay`, `videoMute`, `videoLoop`) and Show
file paths (`showFilePaths`). A write re-reads the stored object and writes back only the
changed keys, every other key (legacy paging, thumbnail state, unknown keys) as stored;
the legacy gallery's Save also keeps the keys it does not edit, so the two pages stay in
sync. The workspace never reads the legacy paging keys (`defaultLimit`, `gridColumns`,
`infiniteScroll`; README continuous-browsing ruling). A browser that refuses storage
(denied, or a full quota that leaves the stored bytes as they were) keeps the change for
this tab only and the status says "Saved for this tab only; browser storage is
unavailable.", never "Saved in this browser.". Another tab's change (a `storage` event)
repaints the clean controls; a draft it overtook stays, highlighted, with the conflict
status. Open from ComfyUI (New tab→`newtab`, Popup window→`popup`) and Node search
auto-hide (seconds) (a whole number 0–300; anything else is highlighted and never sent)
are the server's `webui_display_mode` and `result_timeout`, saved through
`POST /prompt_manager/settings` with the revision. The consumers read the defaults on
their next visit, never the current view: a new Images visit starts from the layout and
size (the last visit keeps its own, a saved view opens at Grid and Comfortable as
before); with metadata off a selection reads nothing until "Load metadata"; with the
cache off selecting the shown file again re-reads it (on, the one current record is
reused); paths off leaves out only the Path row of File & provenance; the viewer reads
the video defaults.
Sources & scans (`system-sources.mjs`, S4) is the concept's "sources" body on the
server's sources. `GET /prompt_manager/settings` reports them (public paths only):
`sources.configured` (each `gallery_root_paths` entry with `valid`, `code` and the
server's `reason`), `sources.effective` (what is scanned and served, with its
`root_index` and `watch`: `watching` only when the image monitor's observer is alive,
scheduled that folder and monitoring is on, `stopping` when it still watches it but
monitoring was switched off, `manual` when monitoring is off or no monitor runs, else
`restart_pending`), `sources.auto_detected`, `monitor` and `settings.monitoring_enabled`.
`POST /prompt_manager/settings/validate-source {path}` checks one folder and writes
nothing. Every caller (validate-source, the settings save, config.json at load) refuses
a source through the same checks (`sources.source_problem`): its form first, then
containment decided as text (`normcase(normpath)` against the allowed folders' own
spellings), before anything resolves, stats or opens the path, so a UNC share, the
`\\?\` and `\\.\` namespaces, a device or any folder outside the allowed ones never
reaches the filesystem or the network (no SMB request, no existence or timing oracle).
Only then is the path resolved and contained again, which catches a symlink out. The
filesystem root and the home directory answer `outside-allowed` with the same reason as
any outside path. ComfyUI folders outside its base directory (`--output-directory`)
are path anchors too, so their sources keep a relative public path
(`renders/portraits`) that round-trips and never collides by folder name. A save whose
list holds a source the server cannot use is a 400 `invalid-source` naming its public
path (`source`, as text: a refused path is never resolved to name it), its `index` and
`reason`, and saves nothing. A save that changes the sources (compared as folders, not
spellings) holds the registry as `records` ("Source change"), so it is refused with
409 `busy` while a scan runs. `POST /prompt_manager/jobs/scan {scope, roots, paths}`
scans only the given effective `root_index`es (strict ints, at most 64), each with the
public path the page read for it; an index past the list or one that now names another
folder is 409 `sources-changed` and starts nothing (the page toasts the server's
reason and reads the sources again); the legacy `/scan` is unchanged. The section
lists one row per source ("<path> · watching | watching until restart | manual scan |
restart pending" with "Scan changes" for a watched source, else "Scan folder"), which
opens the shared "Review metadata scan" for that one source ("1 output folder") and
starts the session job over it; the job opens in Activity, and the toast only says it
started. Enrich
existing metadata and the notice's Review scan review every source. The notice shows
until this browser dismissed the 3.2.0 update notice (`pm_update_notice_dismissed`,
the legacy admin page's key); Later dismisses it without scanning, and Review scan
dismisses it as the legacy Scan does. "+ Add source" asks validate-source, then adds
the folder to the host draft of `gallery_root_paths`; Remove takes one out; "Watch for
new outputs" (`monitoring_enabled`) and Worker threads (`worker_threads`) are host
drafts too, and Save writes them with the revision. A save the server refuses because a
scan runs toasts "Another job is running: <title>." with a link to that job (D57) and
keeps the drafts; a save refused for one source says the server's message as the status
and marks that row. After a save the source list is read again; a monitoring change
waits for a restart (the running observer keeps watching until then, and the status
says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's
Review options opens the concept's "Thumbnail maintenance" dialog (D99, below).
A list that cannot be read shows "Sources could not be loaded. Your saved data has not
changed." and Retry, never an empty list.
Thumbnail generation is an Activity job (S5, `py/api/thumbnail_jobs.py`). `GET
/prompt_manager/images/thumbnails/plan?mode=missing|rebuild&size=128..2048&quality=low|medium|high`
counts the work (`targets`, `fresh`, `remaining` past the 10,000-file cap, the source
labels) and writes nothing; a value out of range is a 400 `invalid-request` that starts
nothing. `POST /prompt_manager/jobs/thumbnails {mode, size, quality}` starts the
`thumbnails` job ("Thumbnail generation"): admission `guarded` (never beside a scan or
another record-changing job, one at a time, a legacy clear refused while it runs),
cancellable at a file boundary (a file not yet started is left alone; written previews
stay), outcome groups Generated / Skipped (already fresh, listed without a log line
each) / Errors. `size` is the longest edge and `quality` the encoder (low: JPEG 70, no
optimize; medium: 85, optimized, what the legacy routes always wrote; high: 95,
optimized). Video previews take the same choice: the OpenCV frame and the placeholder
are saved with those JPEG settings, and the ffmpeg fallback (no OpenCV) maps them to
`-q:v` 8 / 4 / 2 and fits the frame inside the edge keeping its aspect ratio (`-vf
scale=E:E:force_original_aspect_ratio=decrease`, no longer a stretched square).
`rebuild` writes fresh previews again, each through the same temp-file
rename, so nothing is cleared first and a failed file keeps its old preview (a video
whose frame cannot be decoded keeps its preview instead of getting the placeholder);
originals are only read (`tests/test_thumbnail_jobs.py`). Cancel is also checked per
file during the planning walk: a job cancelled there stops the walk at the next file,
publishes no `start` frame and ends "Cancelled before any preview was written". Every
exit sets a final status line: "N previews written · M errors" (plus "· R more on the
next run" past the cap), "No previews to generate · N already up to date" (or "No
images or videos to preview"), "Cancelled · N previews written", or the failure; a job
with nothing to do never reads "Processing file 0 of 0" (`test_every_exit_sets_a_truthful_final_status_line`,
and "a finished thumbnail job with nothing to do says so" in `workspace-activity.spec.mjs`).
Windows: replacing a preview another process holds open (the gallery serving it, an
image viewer) fails with WinError 5 or 32; that file counts as an error reading
"Preview in use by another program; the old one was kept" (never the raw "[WinError
5]"), and its old preview stays. Any other PermissionError reads "Permission denied
writing the preview" (`test_a_locked_preview_is_a_readable_coded_error`).
Thumbnail work (the job and the gallery's generate and clear holds) is `guarded`, but it
writes only preview files and no records, so the image monitor's Needs review
recording runs beside it instead of being deferred for a whole rebuild
(`PREVIEW_ONLY_KINDS` in `py/api/review_recording.py`, `blocker(..., compatible=)`;
`test_a_running_thumbnail_job_does_not_defer_monitor_recording`). Other `guarded` and
all `records` jobs still defer it.
Deliberate change: the legacy gallery stream `POST /images/generate-thumbnails/progress`
now starts this job (missing previews, its quality preset as the size, medium encoding)
or attaches to the running one, and streams the job's unchanged `status`/`start`/
`progress`/`file_error`/`complete` frames after a first `job {id, started}` frame that
names the job and whether this stream started it. The stream reads the registry after
its headers are sent, so a job that starts in that gap is attached to, not refused. A
client that goes away no longer stops the work (`test_thumbnail_job_survives_client_disconnect`:
one worker, 9 files in three batches; the stream sees its client go after the first
batch and all 9 previews are written, where a stream that stops on disconnect leaves 1).
The gallery's Cancel (its settings panel and the startup dialog) cancels by that id,
and only a job its own stream started: a Cancel before the `job` frame keeps the stream
open until it arrives, a double Cancel posts once, and a job another tab or System
started keeps running while the gallery stops following it (`tests/js/gallery-thumbnails.test.js`).
A job cancelled from Activity ends the stream with the error frame "Thumbnail
generation was cancelled". The blocking
`POST /images/generate-thumbnails` and the clear stay legacy holds.
The workspace starts it from "Thumbnail maintenance" (`thumbnail-review.mjs`, the
concept's `thumbnail-review`), opened by Sources' Thumbnail generation → Review options
and Overview's Review thumbnails. Scope is Missing thumbnails only (`missing`, the
default) or Rebuild all (explicit opt-in) (`rebuild`); Quality is Medium · optimized /
High · original / Low · fastest (`medium` / `high` / `low`), preselected from
Appearance's Image loading quality (`gallerySettings.imageQuality`, the same values,
Q3); Preview size limit is the longest edge, 512 by default, a whole number from 128 to
2048. A size outside that range marks the field `aria-invalid`, the plan line says
"Enter a size from 128 to 2048 pixels." and nothing is requested; Start reports the
field's validity and keeps the dialog open. Start closes the dialog and starts the job
through the Activity store, which toasts "Thumbnail generation started."; the job opens
in Activity. A start refused because another job holds the slot toasts the D57 busy
link toast and stays. The legacy gallery's thumbnail tools are unchanged.
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
visit through `latestOnly()` on the section scope; a late answer after the section left
@@ -1864,7 +2257,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
reader can keep it byte-identical (`test_health_check_never_writes`).
Recorded deviations (plan D60–D64 renumbered D92–D96; D121, D122 and D123 assigned):
Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D128 assigned):
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
@@ -1877,13 +2270,15 @@ Recorded deviations (plan D60–D64 renumbered D92–D96; D121, D122 and D123 as
and active saved negatives and the indexed `generated_images` rows (Q2). "Database
integrity" is "Not checked" until a check ran this session, then "N issues" (checks
that found issues, even beside a check that could not run), "Passed" or "Incomplete".
"Ambiguous image links" reads "Not available yet." with its Review metadata action
`aria-disabled` (CSS under D93) until the Needs review queue reports a count (Q6).
The review row is "N ambiguous image links" (the Needs review queue's pending count,
`needs_review.pending`) with the concept's line; its Review metadata action opens
`#/review` (the concept's `go-review`). A count
that is not an integer reads "Ambiguous image links" / "Not available yet." (Q6).
"Backup status" reads "No retained backup record." until verified backups exist (S8).
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
missing thumbnails", "N missing thumbnails" (the pending count of
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid
opens the thumbnail dialog (D99). "One display configuration" says "Grid
and density settings are shared across views." (no paging). The sources row is
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
by "M of N configured sources available." or "None configured is available; ComfyUI's
@@ -1922,6 +2317,129 @@ Recorded deviations (plan D60–D64 renumbered D92–D96; D121, D122 and D123 as
the rest of Maintenance, above its D94 hand-off, as the native place that runs the
health check. A press shows "Checking…" (`aria-disabled`) and opens the report; a
failed check toasts "The health check could not finish. Nothing was changed."
- **D97** (Appearance): added rows for the legacy gallery settings the concept leaves
out, after its own rows: "Show image details on cards" (`showImageInfo`; off hides a
card's title and file line, Info and Tag stay at the end of the row), "Collapse viewer
details by default" (`sidebarCollapsedByDefault`; the viewer opens with
`#viewer-details` hidden and a "Show details"/"Hide details" toggle with
`aria-expanded`, the stage taking the whole width), and the operation "Reset display
defaults" ("Restore this browser’s display defaults, then save. Other settings are
kept.", button "Reset"), which drafts the gallery's defaults for the browser rows only.
The node search timeout shows the server's real value, not the sample 30. CSS under D97.
- **D98** (Sources): each configured source's row has a plain Remove after its scan
button; nothing changes until Save. A source added this session reads "<path> · not
saved yet" with Remove only until Save; a configured source the server cannot use
reads "<path> · <server reason>" with Remove only. A line under the list says when the
list is not the whole story: "No source is configured. After saving, ComfyUI’s output
folder is used." (the last one removed), "No source is configured; ComfyUI’s output
folder is used." and "No configured source is available; ComfyUI’s output folder is
used." (auto-detected rows, listed without Remove). "restart pending" is the third
watch state. The operation "Refresh files" ("Read every source again to find files
added or removed outside ComfyUI.", button "Refresh") closes the surface: it calls the
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
folders." or "Files could not be refreshed. Try again.". The concept's "sample path"
note is prototype copy and is dropped.
- **D99** (Sources/Overview, S5): the concept's `thumbnail-review` dialog ("Thumbnail
maintenance") with production copy. A `p.small[role=status]` line before the actions
says what the server's read-only plan (`GET /images/thumbnails/plan`) counts for the
chosen scope: "N previews to generate · M already up to date." or, for Rebuild all,
"Every preview is regenerated: N files.", with "· R more on the next run" past the
per-run cap; "Counting previews…" while it is read and "The previews could not be
counted." when the read fails (Start still starts). Each scope is read once per
dialog, since size and quality change no count. The primary is "Start" (the
concept's "Simulate thumbnail job"). Overview's Review thumbnails opens the same
dialog (the concept's row starts a sample "Thumbnail check" job directly). Maintenance's
Thumbnail cache row takes the dialog with S7. `thumbnail dialog matches the concept
from sources|overview` (system-concept.spec, both themes and sizes) maps the line and
"Start"; `thumbnail dialog comparison detects mutations` proves they are the only
difference and catches a renamed option, swapped options, swapped actions and a
changed style.
- **D111** (Activity, S5): each job kind's pill verb, toasts, line, tiles, outcomes,
openable rows and labels come from one descriptor table (`activity-kinds.mjs`;
`activity-store.mjs` re-exports it). The scan's (D56) and image tagging's (D89) are
unchanged. A thumbnail job reads "Generating thumbnails · N% · a / b" in the pill,
counts Generated (green), Skipped and Errors (red), lists Generated, Errors, Skipped
(Generated and Skipped rows open the viewer), says "Thumbnails finished · N generated ·
M errors" ("cancelled · N generated", "failed · …") and keeps "Job log" and "Cancel
job". Every other kind (the health check, a kind this page does not know) gets the
generic descriptor: title-based toasts and no tiles or Results, so it never shows the
scan's zero tiles (S1 review M6, S2 review M3). A running job without a known total
has an indeterminate `<progress>` (no `value`) and a pill without a percentage, whose
mini bar (`aria-hidden`; the pill's label carries the state) is marked `indeterminate`
and sweeps instead of showing an empty fill (a dimmed full bar under reduced motion). Only a
running job the server can cancel shows Cancel. Tests:
`tests/js/workspace-activity-kinds.test.js`, and in `workspace-activity.spec.mjs` "a
thumbnail job shows Generated Skipped Errors, its pill verb and its toasts" and "a held
legacy job and the health check show no tiles, no Results and no made-up percentage".
- **D112** (Activity, S5): a legacy-page operation held on the registry (not
cancellable, no outcome groups) reads as its server title ("Database restore (admin
page)", "Thumbnail clear (gallery page)"), with no tiles, no Results, no Cancel, an
indeterminate bar while it runs and "<title> finished." when it ends, whatever its
kind (the gallery's clear is a `thumbnails` hold).
- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the
concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's
sample 4).
- **D127** (Sources): the concept's `add-source` `simpleInput` with production
copy: the field is "Folder path" (the concept's "(sample only)" dropped), it starts
empty (not the demo "/images/new-collection"), and the action is "Add" (not "Apply in
prototype"); Enter in the field adds too. `add source dialog matches the concept`
(system-concept.spec) compares the open dialog with these three mapped, and `add
source comparison detects mutations` proves the mapping is the only difference. A
folder the server refuses keeps the dialog open with the field
`aria-invalid` and the server's reason in a `p.small[role=alert]` above the actions;
a folder the server names as one already listed says "This folder is already a
source.", and a check that cannot finish says "The folder could not be checked. Try
again."; neither adds anything.
The concept dialog does no check, so `invalid source shows the server reason`
(workspace-system-sources.spec) pins it.
- **D128** (Sources): states the concept's sample rows never show. A source the
running monitor still watches after "Watch for new outputs" was saved off reads
"<path> · watching until restart" (still "Scan changes"), and the save's status adds
"Restart ComfyUI to apply: Watch for new outputs." from `restart_required_fields`. A
configured source the effective list holds under another spelling reads "<path> ·
same folder as another source" with only Remove. A save refused for one source
(`invalid-source`) marks that row (`aria-invalid`, outlined like a D92 setting) and
the status is the server's "<path> can't be used: <reason>. Remove it or reconnect it,
then save." ("Remove it, then save." when reconnecting cannot help). Pinned by
`Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save
refused for one unusable source names it and marks its row`
(workspace-system-sources.spec) and the system-state and system-sources node tests.
- **D124** (inspector): with "Load metadata on selection" off, the Images panel and
the viewer's details show "Metadata loads when you ask for it." and a "Load metadata"
button instead of reading the file. The concept has no such state, so no concept
comparison renders it: `metadata off shows an explicit load and loads once on request`
(workspace-system.spec) pins it instead.
- **D125** (prompt blocks, maintainer option A): a resolved prompt whose source is
`runtime-parts-omitted` (a PromptManager `prepend_text` or `append_text` fed directly
by a listed run-time node, such as LoraManager's TriggerWord Toggle, was left out)
shows the muted line "Text added at run time, such as trigger words, is not
included." under its text. The evidence badge stays "Workflow inferred"; the line is
never sent or copied. The concept has no such state (`a prompt that left out run-time
trigger words says so under its text (D125)` in `tests/js/prompt-blocks.test.js`).
- **D126** (prompt blocks, maintainer ruling on the added-conditioning floor): a
resolved positive whose source is `added-conditioning` (a later stage combined more
conditioning with the base prompt, as UltimateSDUpscale's ConditioningCombine of the
PromptManager and a style trigger) shows "Conditioning added at a later stage is not
included." the same way. A prompt with both shows both lines, D125 first (source
`runtime-parts-omitted-and-added-conditioning`). Always labelled, with no length or
"short trigger" heuristic (`a prompt a later stage added conditioning to says so, and
both lines show together (D126)`).
Appearance scope and states. The Default layout and Thumbnail size defaults start the
Images page only (a new visit, not the current view or a saved view); the embedded
galleries (a prompt history's Shared gallery and its All images / Repeat run dialogs)
have no toolbar, so they keep Grid/Comfortable, while the metadata, cache and card
defaults apply there too. A host row whose `/settings` read failed stays disabled with
"Server settings could not be read."; a rejected timeout says "Some changes were not
saved. Review the highlighted settings." A display default another page saved under one
of this tab's drafts flags that draft with the conflict status, whether it arrived while
System was mounted (a `storage` event) or while it was not (checked again on mount, and
before a write); the draft is never written over the other page's value without that
status first. A change kept for this tab only is stored by the next write storage accepts.
Open question (saved-views follow-up, M8): a saved view stores no layout or size and
always opens at Grid/Comfortable, even when the Default layout is Masonry. Should saved
views store a layout, or follow the default?
`tests/browser/system-concept.spec.mjs` compares the shell and Overview with the concept
(dark and light, 1600×1100 and 390×844), mapping only the deviations above; unmapped,
@@ -1934,21 +2452,55 @@ source when there are several, replaces only the chips with the fixture's expect
results, and asserts the full unmapped path list; it maps D96 and compares the
Maintenance notice unchanged. `system comparison detects mutations` removes a row,
renames a metric label, swaps the `--line` token, recolours a chip, renames a check,
removes the totals table and renames a caption. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`) are test artifacts; there
removes the totals table and renames a caption. `appearance matches the concept` maps
only D97 and the real timeout (unmapped, exactly those paths differ), and `appearance
comparison detects mutations` removes a row, renames a label and an option and swaps
the `--line` token. `sources matches the concept with|without the notice` (both
themes and sizes; the fixture's monitor watches the first source, as the concept's
first sample is watched, and the concept's Later removes its notice) maps the real
sources for the two samples, D98's Remove and Refresh files, and D113; unmapped,
exactly those paths differ. `sources comparison detects mutations` removes the notice
and a source row, renames a label and the Add source button and swaps the `--line`
token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`,
`{concept,production}-appearance-{dark,light}-{1600,390}.png`,
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`,
`concept-thumbnail-review-{dark,light}-{1600,390}.png`,
`production-thumbnail-review-{sources,overview}-{dark,light}-{1600,390}.png`) are test artifacts; there
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
the health check and metric 4, a failed check, the thumbnail row against the status
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
route, a load failure with Retry and late answers after leaving. Its hooks are
endpoint (painted after the rest when its read is held, unavailable when it fails), its Review thumbnails
opening the thumbnail dialog and starting the job, the review row (Q6), the section links, the Statistics dialog against the
route, a load failure with Retry and late answers after leaving, and Appearance: new
visits versus the current view and a saved view, the embedded gallery keeping Grid,
metadata off and on request, the cache (counted up to a request barrier), hidden paths,
the video defaults, the host fields with their revision, a stale one and a failed read,
reload and a second tab, a second tab's save while System was closed, storage refused,
the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers the
notice's Later and Review scan, a scoped scan from one source into Activity, Add source
with its server check and Save, a refused folder, Remove down to auto-detect, an absent
configured source, the three watch states, the monitoring and worker rows, Refresh
files, the thumbnail dialog (its plan line per scope with the preselected quality, an
invalid size that requests nothing, Start into Activity, Rebuild all, a failed plan and
a refused start), a save refused by a running scan and a failed read. Its hooks are
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
failures, an absent source, a deleted linked file, unthumbnailed files) on the
concept-shaped fixture, with delays through the shared request holds.
failures, an absent source, a deleted linked file, unthumbnailed files, a fake image
monitor) on the
concept-shaped fixture, with delays through the shared request holds; reset also restores
the server's host settings and its `config.json`, which must be the one inside the server's
temp root (`tests/test_workspace_system_fixture.py`).
```sh
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library -v
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library \
tests.test_workspace_system_fixture -v
node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \
tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \
tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \
tests/js/workspace-system-sources.test.js tests/js/workspace-thumbnail-review.test.js
python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
workspace-system-sources.spec.mjs \
--project chromium --project touch --project firefox --project webkit
```
@@ -1993,8 +2545,24 @@ and skip the parse.
integers past 2**53 as their exact digits, so the card and the inspector read
the same digits. The parse runs after the file handle closes. A record over
2 MiB (UTF-8), a parser error, a recursion error, a placeholder, or a
non-integral, boolean or non-finite seed gives null. A card never shows a seed
that differs from the inspector's
non-integral, boolean or non-finite seed gives null. When the prompt graph's
sampler stages read different seed inputs and the inspector's resolution names
one sampler for a resolved pair (a detailer or pass-through chain, whose facts
are the named stage's, D19), the card compares that stage's seed (read by the
same parser, the other samplers hidden from it) with its own and shows no seed
only when they differ. A seed that agrees, or one the named stage does not make
known, is kept, as on main (a saved sampler beside a preview-only one keeps its
seed; an ambiguous chain, whose facts merge and fill from the parser, keeps it).
Only such graphs are resolved for the card, with `INSPECTION_PROFILE`
(`test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed`,
`test_a_seed_that_agrees_with_the_named_stage_is_kept`,
`test_a_named_ambiguous_chain_keeps_the_parser_seed`; a literal named seed needs no
second parse, and stages sharing one linked seed skip the check. Measured on a
200-card page of about 54 KB graphs, half multi-stage with per-stage seeds: 618 ms
cold against main's 387 ms, about 1.2 ms more per such card, paid once per file
through the header-facts cache below;
before this rule a KSampler then FaceDetailer chain with two seeds broke the
contract below). A card never shows a seed that differs from the inspector's
(`test_card_seed_equals_inspector_seed_when_both_present`, which `JSON.parse`s
the listing in Node and compares `String(seed)` with the inspector's seed,
including one past 2**53). The card shows no seed for A1111 `parameters` files
@@ -2110,6 +2678,167 @@ inspection's own stable stat (`tests/test_api_workspace_images.py`
browser fixtures clear `media_image_tags`, `media_annotation_bytes` and
`media_annotation_files` on reset and never the metadata row.
Review queue schema (#249, `database/review_schema.py`,
`tests/test_review_schema.py`): the group `media_review_metadata` (one row:
schema version, `review_index_revision`), `media_review_items` (one ambiguous
resolver result per `(byte_id, resolver_version, evidence_hash)`, citing an
annotation byte record `ON DELETE RESTRICT`, with the file's recorded
`spelled_path` (the locator is case-folded on Windows, a listing target is
not), 0..16 candidates, `limited`,
`state` pending/selected/unresolved and a revision) and `media_review_decisions`
(one row per item decision revision; `selected` exactly when a
`candidate_hash` is present; `pair_id`/`image_id` navigation is set to NULL,
never cascaded, when the pair or image is deleted, through
`idx_media_review_decisions_pair`/`_image`). Evidence is canonical JSON
(`canonical_evidence`: one object, sorted keys, no whitespace, ASCII only, no
NaN/infinity, at most 8 MiB) and `evidence_hash` is its SHA-256. The group is
valid only beside a valid annotation group (a review group without annotation
tables is refused, catalog only, before the annotation install), installs like
the annotation group (optional in old backups, required once current under
`require_library_state`, SAVEPOINT install validated before and after), and
validation re-canonicalizes every evidence document in one streamed pass. A
startup runs that pass once (`test_startup_validates_stored_evidence_once`; the
install trusts the annotation group it has just validated, and the SQL row
predicates check type and size only, because the streamed pass proves the
rest): 2.5 s to open a database of 40,000 items with 4 KiB of evidence each,
was 7.7 s with two passes. Restore and `verify_database_file` keep the full
validation, annotation group included. The browser fixtures clear
`media_review_decisions` and `media_review_items` before the annotation tables,
never the metadata row.
Needs review items (#249, `utils/review_capture.py`, `database/review.py`,
`tests/test_review.py`, `tests/test_review_outcome.py`,
`tests/test_review_choosable.py`): a file is queued only when its final outcome
is ambiguous and unlinked and there is something to choose. Only scans produce
items; the monitor links new outputs and clears pending items it resolves
(maintainer rulings, 2026-10-23). The output scan (Activity's
Scan for metadata and the legacy `/scan`) hands `record_ambiguous_file` a file
the ingestion resolver finds `ambiguous` only when the browser-parser fallback
then links nothing (status `ambiguous`) and no saved row links the file
already; an ambiguous file a saved row links is reported `unchanged` ("Already
linked to its prompt"), not Needs review. A file the fallback links, an exact
pair the ingester stores, and `conflict` and `unresolved` files are not
queued. Neither is an ambiguous file with nothing to choose: fewer than two of
its candidates are `resolved` with non-blank positive text (`reviewable`). With
none the file could not be traced; with exactly one, the untraced branches may
still have made the file, so that text is not certain either. Both report
`unresolved` ("could not be traced"), are never linked, and their pending,
undecided items whose own evidence offers no choice (a limited item counts as
a choice when it holds at least two candidates) leave the queue on the next
scan or monitor pass (`settle_untraced_review_items`). The maintainer's
second live check found 39 items whose 11 candidates were all "unresolved". The queue heals itself: after each batch, the scan removes the pending
items of every listed file (read or skipped as known) that a saved row now
links, and the monitor does the same for each ambiguous output it links
(`settle_linked_review_items`; the review index revision moves). Only an
undecided item goes (`pending`, revision 0, no decision row): a decision is
never deleted or changed, so no schema state was added. The monitor never
queues an output it linked; an unlinked file passed to `record_review_file`
follows the scan's rule (a file the fallback would link is `skipped`). The
maintainer's live check of #284/#288 found the queue at 10,232 against a review
group of 4,677: every fallback-linked ambiguous file had been queued. It captures
the file (scope `needs-review`), reads its metadata again from the capture,
resolves it with the default profile and stores only an ambiguous result,
bound to the capture's byte record (created or refreshed like an observe) and
fenced by the capture's storage epoch. Evidence is the resolver reason plus one
`{sampler_id, state, reason, positive, negative, hash}` per selectable
candidate (nested ambiguous stages flattened, duplicates dropped; each side
`{text, storage_text, provenance}`; `hash` = SHA-256 of the canonical candidate
without it). More than 16 candidates, or evidence over 8 MiB, is a `limited`
item without candidates. Recording is idempotent per `(byte_id,
resolver_version, evidence_hash)` and never touches an existing item or its
decisions; changed bytes add a new pending item and keep the old one. The
capture is released before the write; no model is loaded and no pixel is
decoded. Scan statuses, counters, outcome rows and the legacy SSE events are
unchanged apart from the already-linked and nothing-to-choose cases above; the job's `scope` gains
`review_items` (this scan's files that are in the pending queue after it, new
or already there: the review group's ambiguous rows, less files a capture could
not read or a decision already settled), `review_added` (new items),
`review_settled` (items removed because their file is linked now),
`review_untraced` (items removed because they offer nothing to choose) and
`review_errors`. The log ends with `? N images added to Needs review`,
`? N images already in Needs review`, `= N images left Needs review · now
linked`, `= N images left Needs review · nothing to choose` and `! N images could not be added to Needs review` (failures, e.g.
capture capacity), each only when N > 0. Recording captures in the background: it never takes the store's
last worker or capture slot, which stay free for an interactive Tag dialog or
Apply, and waits up to 2 s for room before counting the file as not added
(`test_background_recording_leaves_a_worker_for_the_tag_dialog`). The monitor's
pass runs after its own link, best effort: a failure is logged and the link
stays. Recording follows the registry's `records`
admission (#289): the scan's recording runs under its own `records` job, and
the monitor's (one file, not a job) is refused while a `records` or `guarded`
job or hold runs, except thumbnail work, which writes no records (S5 M7). It then records nothing, logs that the file waits for the
next scan, and answers `deferred`; the next "new" scan reads the ambiguous file
again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A
rescan does not copy or hash an ambiguous file whose current size and
`mtime_ns` equal the verified fingerprint of bytes an item already cites (a
stat compare); the file still stays in the default "new" scope, because it is
not a known image. The skip and a duplicate insert record the current spelling
of the locator on its items (`refresh_review_spelling`, the review index
revision moves, item revisions do not), so a case-only rename on Windows or
macOS keeps its item applicable. Residual risks of the skip, like S1's: an
in-place rewrite that keeps both size and `mtime_ns` is never re-queued (Apply
still re-hashes, so it cannot link those bytes), and a file that is deleted or
moved leaves its pending item counted in `total` (its target then formats as
`unavailable` or fails to open, and Leave unresolved clears it).
Needs review decisions and queue reads (#249, `py/api/review.py`,
`database/review.py`, `database/review_decisions.py`,
`tests/test_api_review.py`): `GET /prompt_manager/review/items?limit=1..40&cursor=`
answers `{success, library_epoch, total, items, next_cursor}` with pending items
oldest first (each item adds the card's `url`, the Images listing's
`/prompt_manager/images/serve/<path>?root=<i>&root_scope=<scope>` formatted without a
stat, null when the file is under no current root, and `is_video` from its extension); `total` is the exact pending count. The cursor is an HMAC keyset
on `(created_at, id)` bound to the storage epoch only (ruling M3, so "load more"
keeps working while the monitor adds an item per generation): a malformed one
is 400, a well-formed one this process cannot verify (tampered, or opened
before a ComfyUI restart: the secret is per process) is 409 `review-conflict`
with the current `total`, and one from before a restore is 409
`storage-changed`. A decision or a new item leaves an open cursor valid: a
decided item drops out of later pages and new items (stamped `utc_now_iso()`,
microseconds) append after it. Residual risk: an item recorded after a wall
clock step backwards sorts before an open cursor and shows only on reload.
Targets are formatted from `spelled_path` against the current roots (the
roots' real paths are the only file access); a file under no current root is
`unavailable` with a null target. Every read and decision lists the roots and
resolves real paths on a worker thread, never on the event loop. `GET .../items/{id}` adds `candidates`
(`hash`, `sampler_id`, `state` resolved/unresolved, `reason`, `label` "<positive
title> / <negative title>" from the saved prompt's Library title else the
first line, and the role texts), `dependency_fingerprint` (SHA-256 of the
file's legacy rows under every spelling of the same physical file, positive
hashes and pair links with their negative hashes, as the ingester reads them) and `sha256`; a detail over 8 MiB, or a
limited item, has no candidates. `POST .../items/{id}/decision
{expected_epoch, expected_revision, capture_token, sha256, dependency_fingerprint,
choice}` with a candidate hash (Apply) requires a capture of the item's file
whose bytes are the item's, re-hashes the current file (`file-changed`
otherwise, also for a fresh capture of new bytes), re-reads the capture and
requires that its bytes are still ambiguous and still offer the chosen
candidate (`review-conflict` otherwise; a resolver change that rewords the
reason or reorders candidates leaves the item applicable, and
`RESOLVER_VERSION` in `utils/prompt_pair_resolver.py` is the one version
ingestion provenance and review share), then in one `_pair_transaction`
checks the epoch, the item revision and state (`review-conflict` with the
current item for a repeat), the re-hashed digest, the dependency fingerprint
and the public ingester's compatible-only rules through
`_ingest_resolved_pair_in` (any refusal is 409 `association-conflict` with
zero writes, explained inline; Apply never reparents a link). Physical
identity: the roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, `subst` or a mapped
drive spells the same file otherwise. Every legacy row whose spelling ends with
the file's name (the link summaries' unindexed suffix lookup) is resolved by
`normcase(realpath)` against the locator, else `samefile`; two rows of the
file refuse the choice, one is enriched under the spelling it was linked
with, and a same-name row the request did not classify is a changed
association (`TestPhysicalIdentity`, symlinked root and a portable fake
junction), and writes
the link with `kind:"user-selection"` provenance (decision id, reviewed digest,
candidate hash, resolver version, role provenance), the decision row, the
item's new state and revision and `review_index_revision`. `"unresolved"`
records the decision only and accepts `capture_token: null`. A successful
decision releases its capture. No decision changes run or usage counts or
calls an execution writer; a Tag save never changes a review item. Errors carry
fixed sentences: `review-failed` / `decision-failed` 500s never echo exception
text or paths.
Capture store (#249, `utils/media_capture.py`, `tests/test_media_capture.py`):
one explicit action captures one rooted media file into an owner-only
`prompt-manager-captures-*` temp directory, streaming it in 1 MiB chunks while
@@ -2126,8 +2855,12 @@ directories are `CaptureDenied`; the locator is
`normcase(join(realpath(root), relative))`, so a symlinked root shares the
listing's identity. Fixed limits: 256 MiB per file (stat and stream), 8
completed captures, 512 MiB reserved, 2 capture/verify workers, 900 s idle,
1800 s hard; admission is nonblocking (`CaptureCapacity`) and lifetimes are
tested with an injected clock. An acquired capture outlives expiry and release
1800 s hard; interactive admission is nonblocking (`CaptureCapacity`), a
`background=True` capture (Needs review recording) never takes the last worker
or capture slot and waits up to `BACKGROUND_WAIT_SECONDS` (2 s) for room, and
lifetimes are tested with an injected clock. The host's on_cleanup close is
final: a recording still in flight gets a capture error instead of creating a
new temp directory after it. An acquired capture outlives expiry and release
until its last use ends. Header-only dimensions (4096 px per side,
16,777,216 px) and video mark Generate unsupported; `verify` re-hashes the
current file without a copy. Every handle closes before unlink (a patched
+18
View File
@@ -91,6 +91,9 @@ try:
if host_app is not None and hasattr(host_app, "on_cleanup"):
host_app.on_cleanup.append(api.cleanup_history_reads)
host_app.on_cleanup.append(api.cleanup_annotation_captures)
if host_app is not None and hasattr(host_app, "on_shutdown"):
# Open Needs review change streams end before the server waits on them.
host_app.on_shutdown.append(api.close_review_events)
except Exception as e:
# Log to internal logging system without console spam
@@ -122,6 +125,21 @@ try:
_global_prompt_tracker = get_prompt_tracker(_global_db)
_global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
# The Needs review pass for ambiguous new outputs runs through the API's
# capture store (the one the scan and the Tag dialog use): a linked output
# is never queued and its pending items leave the queue. It stops before
# that store is closed at shutdown.
_review_api = globals().get("api")
if _review_api is not None:
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
_review_host = getattr(config.server_instance, "app", None)
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
async def _stop_review_recording(app, monitor=_global_image_monitor):
monitor.set_review_recorder(None)
_review_host.on_cleanup.insert(0, _stop_review_recording)
# Start monitoring
_global_image_monitor.start_monitoring()
_init_logger.info("Global image monitoring system started successfully")
+5 -2
View File
@@ -637,7 +637,8 @@ class LibraryOperations:
pairs (every pair's positive is eligible). ``images`` counts indexed
``generated_images`` rows, not files on disk. An image has no rating
of its own (it shows its linked prompt's), so ``rated_images`` counts
the indexed images whose linked prompt is rated.
the indexed images whose linked prompt is rated. ``review_pending`` is
the Needs review queue's pending total.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
row = conn.execute(
@@ -645,13 +646,15 @@ class LibraryOperations:
"(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
"(SELECT COUNT(*) FROM generated_images),"
"(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)"
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL),"
"(SELECT COUNT(*) FROM media_review_items WHERE state='pending')"
).fetchone()
return {
"positive_prompts": row[0],
"negative_variants": row[1],
"images": row[2],
"rated_images": row[3],
"review_pending": row[4],
}
def usage_by_category(self) -> list[dict]:
+20 -3
View File
@@ -18,6 +18,11 @@ from typing import Optional, Set, Tuple
from .library_schema import library_schema_error, migrate_library_state
from .image_favorites import image_favorites_schema_error, migrate_image_favorites
from .annotation_schema import annotation_schema_error, migrate_annotation_state
from .review_schema import (
migrate_review_state,
review_schema_error,
review_without_annotations_error,
)
from .saved_views import migrate_saved_views, saved_views_schema_error
from .execution_history_schema import (
history_schema_error,
@@ -303,7 +308,14 @@ class PromptModel:
migrate_library_state(conn)
migrate_image_favorites(conn)
migrate_saved_views(conn)
# A review group without its annotation group is refused before the
# annotation install could make it look valid (catalog only; the rows
# are validated once, in migrate_review_state).
review_error = review_without_annotations_error(conn)
if review_error:
raise sqlite3.DatabaseError(review_error)
migrate_annotation_state(conn)
migrate_review_state(conn)
def _migrate_prompt_pairs(self, conn: sqlite3.Connection) -> None:
conn.execute("SAVEPOINT prompt_pair_schema")
@@ -1023,9 +1035,9 @@ class PromptModel:
Args:
path: File to inspect
require_library_state: Also require the tables every current
database has (Library state, image favorites, saved views
and image annotations), as a migrated restore stage must;
old backups may lack them.
database has (Library state, image favorites, saved views,
image annotations and the review queue), as a migrated
restore stage must; old backups may lack them.
Returns:
(ok, reason): reason is "ok" when the file is acceptable, otherwise
@@ -1095,6 +1107,11 @@ class PromptModel:
)
if annotation_error:
return False, annotation_error
review_error = review_schema_error(
conn, allow_missing=not require_library_state
)
if review_error:
return False, review_error
return True, "ok"
except sqlite3.Error as e:
return False, f"Invalid SQLite database: {e}"
+4
View File
@@ -14,6 +14,8 @@ from .pairs import PairDependencyError, PairOperations
from .library import LibraryOperations, titled
from .image_favorites import ImageFavoriteOperations
from .annotations import AnnotationOperations
from .review import ReviewOperations
from .review_decisions import ReviewDecisionOperations
from .saved_views import SavedViewOperations
from .library_commands import LibraryCommandOperations
from .vocabulary import VocabularyOperations
@@ -345,6 +347,8 @@ class PromptDatabase(
HistoryReadOperations,
ImageFavoriteOperations,
AnnotationOperations,
ReviewOperations,
ReviewDecisionOperations,
SavedViewOperations,
HealthOperations,
):
+133 -105
View File
@@ -9,9 +9,11 @@ from .operations import _coerce_file_info
try:
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
from ..utils.prompt_pair_resolver import RESOLVER_VERSION
except ImportError:
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
from utils.prompt_pair_resolver import RESOLVER_VERSION
def ingest_resolved_pair(
@@ -20,117 +22,143 @@ def ingest_resolved_pair(
state = resolution.get("state", "unresolved")
if state != "resolved":
return {"status": state, "pairs_added": 0, "links_added": 0}
with db._pair_transaction() as conn:
return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution)
def _conflict() -> Dict[str, Any]:
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
def _ingest_resolved_pair_in(
conn: Any,
db: Any,
image_path: str,
metadata: Dict[str, Any],
resolution: Dict[str, Any],
*,
provenance: Dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``.
The compatible-only rules are the public ingester's: an image with more
than one exact row, another positive, or a prior pair with another
negative is a ``conflict`` and nothing is written. The caller owns the
transaction, so its own later failure rolls the link back too.
``provenance`` replaces the link's default resolver provenance (Needs
review records the user's selection there).
"""
positive, negative = resolution["positive"], resolution["negative"]
positive_text = positive.get("storage_text", positive["text"])
negative_text = negative.get("storage_text", negative["text"])
normalized = normalize_image_path(image_path)
provenance = {"resolver_version": 1, "positive": positive, "negative": negative}
if provenance is None:
provenance = {
"resolver_version": RESOLVER_VERSION,
"positive": positive,
"negative": negative,
}
encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":"))
with db._pair_transaction() as conn:
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return _conflict()
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute(
"SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
snapshots = {
column: dumps_strict(metadata[key], sort_keys=True)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update(
{key: value for key, value in info.items() if value is not None}
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
if any(row["negative_hash"] != expected for row in prior):
return _conflict()
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
snapshots = {
column: dumps_strict(metadata[key], sort_keys=True)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update({key: value for key, value in info.items() if value is not None})
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
+561
View File
@@ -0,0 +1,561 @@
"""Needs review items (#249): persist and read capture-bound ambiguous evidence.
An item is one ambiguous resolver result for one annotation byte record (the
whole-file SHA-256 of one physical file at one locator), keyed by
``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the
same evidence for the same bytes is a duplicate and never touches an existing
item, its state, revision or decisions. Changed bytes are a new byte record
and therefore a new pending item; the old one stays.
Callers pass verified facts (the capture's digest, size and ``mtime_ns``);
nothing here reads the filesystem. Writes run in one ``_pair_transaction``
fenced by the storage epoch the capture was taken under.
"""
import hashlib
import json
import os
import re
import sqlite3
from typing import Iterator
try:
from ..utils.hashing import generate_prompt_hash
except ImportError:
from utils.hashing import generate_prompt_hash
from .annotation_schema import (
BYTES_TABLE,
FILES_TABLE,
METADATA_TABLE as ANNOTATION_METADATA_TABLE,
)
from .annotations import _new_id, _validate_locator, _validate_target
from .models import normalize_image_path, utc_now_iso
from .review_changes import notify_review_change
from .review_schema import (
DECISIONS_TABLE,
ITEMS_TABLE,
MAX_CANDIDATES,
MAX_SPELLED_PATH_BYTES,
METADATA_TABLE,
canonical_evidence,
evidence_digest,
evidence_offers_choice,
)
# Only this module's table constants are interpolated into the SQL below.
_INSERT_ITEM = (
f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path," # nosec B608
"resolver_version,evidence_hash,evidence,candidate_count,limited,state,"
"revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)"
)
_BUMP_REVIEW_INDEX = (
f"UPDATE {METADATA_TABLE} SET " # nosec B608
"review_index_revision=review_index_revision+1 WHERE singleton_id=1"
)
# A later spelling of the same locator (a case-only rename on a case-folding
# filesystem) replaces the recorded one, so targets keep formatting.
_REFRESH_SPELLING = (
f"UPDATE {ITEMS_TABLE} SET spelled_path=? " # nosec B608
f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)"
)
_REVIEW_INDEX = (
f"SELECT review_index_revision FROM {METADATA_TABLE} " # nosec B608
"WHERE singleton_id=1"
)
_BUMP_IMAGE_INDEX = (
f"UPDATE {ANNOTATION_METADATA_TABLE} SET " # nosec B608
"image_index_revision=image_index_revision+1 WHERE singleton_id=1"
)
_ITEMS_OF_LOCATOR = (
"SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash," # nosec B608
"i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at,"
f"i.updated_at,b.sha256 FROM {FILES_TABLE} f "
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? ORDER BY i.created_at,i.id"
)
_QUEUED = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"LIMIT 1"
)
_PENDING_AT = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"AND i.state='pending' LIMIT 1"
)
# Pending items of one file: the candidates a link settles.
_PENDING_OF_LOCATOR = (
f"SELECT i.id FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Pending items of one file with their evidence: an untraced file's settle.
_PENDING_EVIDENCE_OF_LOCATOR = (
f"SELECT i.id,i.evidence FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Only an undecided item goes: a decision row (RESTRICT) is never deleted.
_SETTLE_ITEM = (
f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' " # nosec B608
f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d "
f"WHERE d.item_id={ITEMS_TABLE}.id)"
)
_EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
_PENDING_COUNT = (
f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'" # nosec B608
)
_QUEUE_COLUMNS = (
"id",
"state",
"revision",
"created_at",
"candidate_count",
"limited",
"spelled_path",
"locator",
)
_QUEUE_SELECT = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i "
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id "
)
_QUEUE_FIRST = _QUEUE_SELECT + (
"WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?"
)
_QUEUE_AFTER = _QUEUE_SELECT + (
"WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) "
"ORDER BY i.created_at,i.id LIMIT ?"
)
_DETAIL_COLUMNS = _QUEUE_COLUMNS + (
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"updated_at",
"sha256",
)
_DETAIL = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
"i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version,"
"i.evidence_hash,i.evidence,i.updated_at,b.sha256 "
f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?"
)
# The legacy rows and pair links the ingester's compatibility rules read.
_DEPENDENT_IMAGES = (
"SELECT gi.id,p.hash,gi.image_path FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id"
)
_DEPENDENT_LINKS = (
"SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE ip.image_id=? ORDER BY ip.pair_id"
)
# Rows whose spelling ends with the file's name: the candidates for another
# spelling of the same physical file (the link summaries' filename fallback).
_SAME_NAME_ROWS = (
"SELECT id,image_path FROM generated_images "
"WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id"
)
_TITLES = (
"SELECT p.hash,s.title FROM prompts p "
"JOIN prompt_library_state s ON s.prompt_id=p.id "
"WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id"
)
_ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
_ITEM_COLUMNS = (
"id",
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"candidate_count",
"limited",
"state",
"revision",
"created_at",
"updated_at",
"sha256",
)
def _validate_spelling(spelled_path: object, locator: str) -> None:
if (
not isinstance(spelled_path, str)
or not spelled_path
or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES
or os.path.normcase(spelled_path) != locator
):
raise ValueError("spelled_path must be the file's spelling of its locator")
def dependent_images(conn: sqlite3.Connection, spellings) -> list:
"""``(id, image_path, positive_hash)`` of every legacy row of the file.
``spellings`` are the paths the file is linked under (the Apply path and
every alias of the same physical file); each is matched as the ingester
matches one path. Rows are distinct and ordered by id.
"""
found = {}
for spelled in dict.fromkeys(spellings):
for row in conn.execute(
_DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled)
):
found[row[0]] = (row[0], row[2], row[1])
return [found[key] for key in sorted(found)]
def same_name_rows(conn: sqlite3.Connection, name: str) -> list:
"""``(id, image_path)`` of rows whose spelling may end with ``name``."""
# Imported here: operations composes this module's mixin.
from .operations import escape_like
return [
(row[0], row[1])
for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",))
]
def dependency_fingerprint(
conn: sqlite3.Connection, image_path: str, aliases=()
) -> str:
"""SHA-256 of what Apply's compatibility depends on for ``image_path``.
The exact legacy rows of the file under ``image_path`` and every alias
spelling of it, with their positive hashes, and each row's pair links
with the negative hash, read as the ingester reads them. Unrelated prompt
changes (tags, titles) do not move it.
"""
rows = []
for image_id, _spelled, positive_hash in dependent_images(
conn, (image_path, *aliases)
):
links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall()
rows.append([image_id, positive_hash, [list(link) for link in links]])
text = canonical_evidence({"v": 1, "images": rows})
return hashlib.sha256(text.encode("utf-8")).hexdigest()
class ReviewOperations:
"""PromptDatabase mixin: insert and read Needs review items."""
def record_review_item(
self,
*,
locator: str,
spelled_path: str,
sha256: str,
size: int,
mtime_ns: int,
epoch: str,
resolver_version: int,
evidence: dict,
candidate_count: int,
limited: bool,
) -> bool:
"""Store one ambiguous result for these exact bytes; True when new.
The byte record is created (or its verified fingerprint refreshed)
like an observe. ``spelled_path`` is the file's own spelling (it must
fold to ``locator``); targets are formatted from it. Raises
``LibraryCommandError("storage-changed")`` when the storage epoch is
no longer ``epoch`` (a restore since the capture) and ``ValueError``
for invalid facts or evidence.
"""
_validate_target(locator, sha256, size, mtime_ns)
_validate_spelling(spelled_path, locator)
if type(resolver_version) is not int or resolver_version < 1:
raise ValueError("resolver_version must be a positive integer")
if (
type(candidate_count) is not int
or not 0 <= candidate_count <= MAX_CANDIDATES
):
raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}")
if not isinstance(limited, bool) or (limited and candidate_count):
raise ValueError("A limited item carries no candidates")
text = canonical_evidence(evidence)
evidence_hash = evidence_digest(text)
with self._pair_transaction() as conn:
self._library_epoch(conn, epoch)
now = utc_now_iso()
file_id = self._annotation_owner(conn, locator, now)
byte_id, _revision, created, moved = self._annotation_byte(
conn, file_id, sha256, size, mtime_ns, now
)
if created or moved:
conn.execute(_BUMP_IMAGE_INDEX)
cursor = conn.execute(
_INSERT_ITEM,
(
_new_id(),
byte_id,
spelled_path,
resolver_version,
evidence_hash,
text,
candidate_count,
int(limited),
now,
now,
),
)
inserted = cursor.rowcount > 0
refreshed = _refresh_spelling(conn, locator, spelled_path)
if inserted or refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
if inserted or refreshed:
notify_review_change(self.model.db_path)
return inserted
def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool:
"""Record ``spelled_path`` as the spelling of every item at ``locator``.
The rescan's skip calls it for a file it does not copy again: a
case-only rename keeps size and ``mtime_ns``, and a target formatted
from the old spelling would be refused. True when a spelling changed
(the review index revision moves; item revisions do not, so an open
decision stays valid).
"""
_validate_locator(locator)
_validate_spelling(spelled_path, locator)
with self._pair_transaction() as conn:
refreshed = _refresh_spelling(conn, locator, spelled_path)
if refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
if refreshed:
notify_review_change(self.model.db_path)
return refreshed
def review_index_revision(self) -> int:
"""The queue's revision (moved by every queue change)."""
row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
return row[0]
def review_queue_version(self) -> dict:
"""``{library_epoch, review_revision}`` of the queue, from one snapshot.
The pair a list read answers: equal pairs mean the same queue (a
restore moves the epoch, every queue change moves the revision).
"""
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
revision = conn.execute(_REVIEW_INDEX).fetchone()
if epoch is None or revision is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
return {"library_epoch": epoch[0], "review_revision": revision[0]}
def review_items_for(self, locator: str) -> list:
"""Every item of the file at ``locator``, oldest first, evidence parsed."""
_validate_locator(locator)
return list(self._iter_review_items(locator))
def _iter_review_items(self, locator: str) -> Iterator[dict]:
conn = self.model.get_connection()
for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)):
item = dict(zip(_ITEM_COLUMNS, tuple(row)))
item["evidence"] = json.loads(item["evidence"])
item["limited"] = bool(item["limited"])
yield item
# -- queue reads (#249 E3) -------------------------------------------------
def review_fingerprint_queued(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when an item cites a byte record verified at this exact stat.
A rescan uses it to skip the copy and hash of a file it already
queued: the same ``(size, mtime_ns)`` the bytes were verified with.
"""
row = (
self.model.get_connection()
.execute(_QUEUED, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_pending_at(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when a *pending* item cites bytes verified at this exact stat.
``review_fingerprint_queued`` also matches decided items; a file whose
only item is decided is not in the queue.
"""
row = (
self.model.get_connection()
.execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_file_linked(self, spellings) -> bool:
"""True when a saved row links the file under any of ``spellings``."""
with self._annotation_read() as conn:
return bool(dependent_images(conn, spellings))
def settle_linked_review_items(self, files) -> int:
"""Remove the pending, undecided items of files that are now linked.
``files`` holds ``(locator, spellings)`` per file. An item leaves the
queue only when it is still ``pending`` at revision 0 with no decision
row and a saved row links its file under one of ``spellings``; a
decided item is never touched. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
files = [(locator, tuple(spellings)) for locator, spellings in files]
for locator, _spellings in files:
_validate_locator(locator)
conn = self.model.get_connection()
candidates = [
(locator, spellings)
for locator, spellings in files
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not candidates:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator, spellings in candidates:
if not dependent_images(conn, spellings):
continue
for (item_id,) in conn.execute(
_PENDING_OF_LOCATOR, (locator,)
).fetchall():
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
if removed:
notify_review_change(self.model.db_path)
return removed
def settle_untraced_review_items(self, locators) -> int:
"""Remove the pending, undecided items that offer nothing to choose.
``locators`` are files the caller found ambiguous without a choice
(fewer than two candidates carry a prompt). Of their items, one leaves
the queue only when its own stored evidence offers no choice either
(``evidence_offers_choice``) and it is still ``pending`` at revision 0
with no decision row, the same safe delete as
``settle_linked_review_items``. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
locators = list(locators)
for locator in locators:
_validate_locator(locator)
conn = self.model.get_connection()
pending = [
locator
for locator in locators
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not pending:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator in pending:
rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall()
for item_id, evidence in rows:
if evidence_offers_choice(json.loads(evidence)):
continue
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
return removed
def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict:
"""One page of pending items, oldest first, from one snapshot.
``after`` is the ``(created_at, id)`` keyset of the last item already
shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is
the exact pending count and ``items`` carry their ``spelled_path`` and
``locator`` (server-side only).
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be a positive integer")
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
revision = conn.execute(_REVIEW_INDEX).fetchone()
if epoch is None or revision is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if after is None:
rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall()
else:
rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall()
items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]]
return {
"epoch": epoch[0],
"revision": revision[0],
"total": total,
"items": items,
"more": len(rows) > limit,
}
def review_item(self, item_id: str) -> dict | None:
"""One item with its parsed evidence and the pending total, or None."""
if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None:
return None
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
if epoch is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item["evidence"] = json.loads(item["evidence"])
item["epoch"] = epoch[0]
item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0]
return item
def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str:
with self._annotation_read() as conn:
return dependency_fingerprint(conn, image_path, aliases)
def review_same_name_rows(self, name: str) -> list:
"""``(id, image_path)`` of legacy rows that may spell a file ``name``."""
with self._annotation_read() as conn:
return same_name_rows(conn, name)
def stored_prompt_titles(self, texts) -> dict:
"""Library titles of saved prompts by text (dedup hash), when set."""
hashes = {generate_prompt_hash(text): text for text in texts if text.strip()}
if not hashes:
return {}
marks = ",".join("?" * len(hashes))
found: dict[str, tuple] = {}
conn = self.model.get_connection()
for prompt_hash, title in conn.execute(
_TITLES.format(marks=marks), list(hashes)
):
text = hashes[prompt_hash]
found[text] = found.get(text, ()) + (title,)
return found
def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool:
return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0
def _queue_row(row, columns) -> dict:
item = dict(zip(columns, tuple(row)))
item["limited"] = bool(item["limited"])
return item
+64
View File
@@ -0,0 +1,64 @@
"""Needs review queue change notices (#249 PR F2).
Every committed change to the pending queue (an item recorded by the scan or
the monitor, a spelling refresh, a decision) calls ``notify_review_change``
after its transaction, from whichever thread wrote it. The review event
stream (``py/api/review_events.py``) subscribes, so an open Needs review page
re-reads its queue when it changes instead of polling. Observers are keyed by
the database file, like ``subscribe_history_restore``; an observer that fails
is logged and never fails the write.
"""
import os
import threading
from typing import Callable
try:
from ..utils.logging_config import get_logger
except ImportError:
from utils.logging_config import get_logger
_logger = get_logger("prompt_manager.database.review_changes")
_lock = threading.Lock()
_observers: dict[str, dict[object, Callable[[], None]]] = {}
def _identity(db_path: str) -> str:
return os.path.normcase(os.path.realpath(db_path))
def subscribe_review_changes(
db_path: str, callback: Callable[[], None]
) -> Callable[[], None]:
"""Call ``callback()`` after each committed queue change; return the unsubscribe."""
identity = _identity(db_path)
token = object()
with _lock:
_observers.setdefault(identity, {})[token] = callback
def unsubscribe() -> None:
with _lock:
observers = _observers.get(identity)
if observers is not None:
observers.pop(token, None)
if not observers:
_observers.pop(identity, None)
return unsubscribe
def observer_count(db_path: str) -> int:
"""How many observers watch ``db_path`` (tests and diagnostics)."""
with _lock:
return len(_observers.get(_identity(db_path), {}))
def notify_review_change(db_path: str) -> None:
"""Tell every observer of ``db_path``; call it after the commit."""
with _lock:
callbacks = tuple(_observers.get(_identity(db_path), {}).values())
for callback in callbacks:
try:
callback()
except Exception:
_logger.warning("Needs review change observer failed", exc_info=True)
+291
View File
@@ -0,0 +1,291 @@
"""Needs review decisions (#249 E3): explicit, auditable and compatible-only.
A decision is one row of ``media_review_decisions`` for one item revision.
``selected`` links the chosen candidate's pair to the file through the public
ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same
``_pair_transaction`` as the decision row, the item's new state and revision
and the review index revision; any conflict or later failure rolls all of it
back. ``unresolved`` records the decision and links nothing. No decision
counts a run, touches usage or calls an execution writer.
Callers verify the file first (the capture must be the item's bytes and the
current file must still hash to them); nothing here reads the filesystem.
"""
import os
import sqlite3
from .annotation_schema import BYTES_TABLE
from .annotations import _new_id
from .library_commands import LibraryCommandError
from .models import utc_now_iso
from .review import (
_BUMP_REVIEW_INDEX,
_DETAIL,
_DETAIL_COLUMNS,
_PENDING_COUNT,
_queue_row,
dependency_fingerprint,
dependent_images,
same_name_rows,
)
from .review_changes import notify_review_change
from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence
DECISION_KIND = "user-selection"
_INSERT_DECISION = (
f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision," # nosec B608
"created_at,outcome,candidate_hash,evidence,pair_id,image_id) "
"VALUES(?,?,?,?,?,?,?,?,?)"
)
_ADVANCE_ITEM = (
f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1," # nosec B608
"updated_at=? WHERE id=? AND revision=?"
)
_ITEM_BYTES = (
f"SELECT b.sha256 FROM {ITEMS_TABLE} i " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?"
)
class ReviewCommandError(LibraryCommandError):
"""A decision lost a race or would rewrite an association.
``code`` is ``review-conflict`` (the item was decided or changed since it
was read), ``association-conflict`` (the choice is not compatible with
the file's saved links, or those links changed), ``file-changed`` (the
item is not about these bytes) or ``review-missing``. ``item`` is the
current item read inside the transaction, ``total`` the pending count and
``fingerprint`` the current dependency fingerprint, when known;
``changed`` marks an association conflict caused by links that changed
since the reviewer read them (rather than an incompatible choice).
"""
def __init__(
self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False
):
super().__init__(code, epoch)
self.item = item
self.total = total
self.fingerprint = fingerprint
self.changed = changed
# The public view of ``item``, set by the API when it formats one.
self.view = None
class _Rollback(Exception):
"""Raised inside the transaction so the context manager rolls back."""
def __init__(self, error: ReviewCommandError):
super().__init__(error.code)
self.error = error
class ReviewDecisionOperations:
"""PromptDatabase mixin: record one decision on one Needs review item."""
def decide_review_item(
self,
*,
item_id: str,
expected_epoch: str,
expected_revision: int,
sha256: str,
outcome: str,
candidate: dict | None = None,
image_path: str | None = None,
metadata: dict | None = None,
expected_fingerprint: str | None = None,
aliases: tuple = (),
known_rows: frozenset | None = None,
) -> dict:
"""Record the decision; return ``{item, total, pair}``.
``selected`` needs the evidence ``candidate`` (with its ``hash``), the
``image_path`` the scan and the monitor link under, the captured
``metadata`` and the ``expected_fingerprint`` the reviewer saw.
``aliases`` are the other spellings legacy rows of the same physical
file use (a symlinked, junctioned or mapped root) and ``known_rows``
the ids of every same-name row the caller classified; a same-name row
it did not see is a changed association.
Raises ``LibraryCommandError("storage-changed")`` after a restore and
:class:`ReviewCommandError` for every refused decision, with zero
writes either way.
"""
if outcome not in ("selected", "unresolved"):
raise ValueError("outcome must be selected or unresolved")
if outcome == "selected" and (
not isinstance(candidate, dict) or not image_path
):
raise ValueError("A selected decision needs a candidate and a file")
try:
with self._pair_transaction() as conn:
decided = self._decide_in(
conn,
item_id=item_id,
expected_epoch=expected_epoch,
expected_revision=expected_revision,
sha256=sha256,
outcome=outcome,
candidate=candidate,
image_path=image_path,
metadata=metadata or {},
expected_fingerprint=expected_fingerprint,
aliases=tuple(aliases),
known_rows=known_rows,
)
except _Rollback as rollback:
raise rollback.error from None
# After the commit: open Needs review pages re-read their queue.
notify_review_change(self.model.db_path)
return decided
def _decide_in(self, conn: sqlite3.Connection, **command) -> dict:
epoch = self._library_epoch(conn, command["expected_epoch"])
item_id = command["item_id"]
current = _current_item(conn, item_id)
if current is None:
raise _Rollback(ReviewCommandError("review-missing", epoch))
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if (
current["state"] != "pending"
or current["revision"] != command["expected_revision"]
):
raise _Rollback(
ReviewCommandError("review-conflict", epoch, item=current, total=total)
)
if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]:
raise _Rollback(ReviewCommandError("file-changed", epoch))
decision_id = _new_id()
pair = None
candidate = command["candidate"] if command["outcome"] == "selected" else None
if candidate is not None:
pair = self._apply_candidate(
conn, epoch, current, decision_id, total, **command
)
evidence = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"outcome": command["outcome"],
"reviewed_digest": command["sha256"],
"item_evidence_hash": current["evidence_hash"],
"resolver_version": current["resolver_version"],
"candidate": candidate,
"dependency_fingerprint": command["expected_fingerprint"],
"pair": pair,
}
now = utc_now_iso()
self._insert_review_decision(
conn,
(
decision_id,
item_id,
current["revision"] + 1,
now,
command["outcome"],
candidate["hash"] if candidate is not None else None,
canonical_evidence(evidence),
pair["id"] if pair else None,
pair["image_id"] if pair else None,
),
)
conn.execute(
_ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"])
)
conn.execute(_BUMP_REVIEW_INDEX)
return {
"item": _current_item(conn, item_id),
"total": conn.execute(_PENDING_COUNT).fetchone()[0],
"pair": pair,
"decision_id": decision_id,
}
def _apply_candidate(
self, conn, epoch, current, decision_id, total, **command
) -> dict:
"""Link the candidate's pair, compatible-only; raise on any conflict."""
# Imported here: pair_ingestion imports operations, which composes this.
from .pair_ingestion import _ingest_resolved_pair_in
image_path = command["image_path"]
aliases = command["aliases"]
fingerprint = dependency_fingerprint(conn, image_path, aliases)
known = command["known_rows"]
unseen = known is not None and any(
row_id not in known
for row_id, _path in same_name_rows(conn, os.path.basename(image_path))
)
if unseen or fingerprint != command["expected_fingerprint"]:
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
changed=True,
)
)
rows = dependent_images(conn, (image_path, *aliases))
if len(rows) > 1:
# One file linked under several spellings: no single row to enrich.
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
if rows:
# Enrich the one row of this file under the spelling it was
# linked with, so the ingester's compatibility rules see it.
image_path = rows[0][1]
candidate = command["candidate"]
positive, negative = candidate["positive"], candidate["negative"]
provenance = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"reviewed_digest": command["sha256"],
"candidate_hash": candidate["hash"],
"resolver_version": current["resolver_version"],
"positive": positive,
"negative": negative,
}
result = _ingest_resolved_pair_in(
conn,
self,
image_path,
command["metadata"],
{"state": "resolved", "positive": positive, "negative": negative},
provenance=provenance,
)
if result["status"] == "conflict":
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
return {
"id": result["pair_id"],
"image_id": result["image_id"],
"status": result["status"],
}
@staticmethod
def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None:
conn.execute(_INSERT_DECISION, values)
def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None:
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item.pop("evidence")
return item
+507
View File
@@ -0,0 +1,507 @@
"""Needs review schema group (#249): capture-bound ambiguous items and decisions.
The group installs like ``image_annotations``: optional in old backups,
required once a database is current, and valid only beside a valid image
annotation group, because every item cites one annotation byte record. It owns
- ``media_review_metadata``: one row, the group's schema version and the
review index revision (moved by every queue change);
- ``media_review_items``: one ambiguous resolver result for one byte record,
keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's
spelling when it was recorded (the locator is case-folded on Windows, a
target is not), its canonical evidence, candidate count, ``limited`` flag,
state and revision;
- ``media_review_decisions``: the auditable decision history of an item, one
row per decision revision, with optional navigation to the pair and image
the decision linked (nulled, never cascaded, when those are deleted).
Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted
keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound
agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is
the SHA-256 of the item's evidence text.
Installation never backfills anything and never touches the filesystem.
Validation reads only: catalog objects compared after whitespace compaction,
columns and foreign keys through the PRAGMAs, persisted values in single
bounded SQL passes, then one streamed pass that re-canonicalizes each evidence
document (one row in memory at a time).
"""
import hashlib
import json
import re
import sqlite3
from typing import Any, Optional
from .annotation_schema import BYTES_TABLE
from .annotation_schema import TABLES as ANNOTATION_TABLES
from .annotation_schema import _catalog as _annotation_catalog
from .annotation_schema import annotation_schema_error
try:
from ..utils.json_strict import dumps_strict
except ImportError:
from utils.json_strict import dumps_strict
SCHEMA_VERSION = 1
METADATA_TABLE = "media_review_metadata"
ITEMS_TABLE = "media_review_items"
DECISIONS_TABLE = "media_review_decisions"
TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
MAX_CANDIDATES = 16
# The file's own spelling when it was recorded (bytes of UTF-8 text). Locators
# are case-folded on Windows, so a target is formatted from this instead.
MAX_SPELLED_PATH_BYTES = 16384
ITEM_STATES = ("pending", "selected", "unresolved")
DECISION_OUTCOMES = ("selected", "unresolved")
_TIMESTAMP_BYTES = 64
_HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'"
_EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})"
_TABLE_DDL = {
METADATA_TABLE: (
f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
f"CHECK(schema_version={SCHEMA_VERSION}), "
"review_index_revision INTEGER NOT NULL DEFAULT 0 "
"CHECK(review_index_revision>=0))"
),
ITEMS_TABLE: (
f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, "
"spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), "
"resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL "
f"CHECK({_HEX64.format(c='evidence_hash')}), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"candidate_count INTEGER NOT NULL "
f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), "
"state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), "
"revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, "
"updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS_TABLE: (
f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY "
"CHECK(length(id)=36), item_id TEXT NOT NULL "
f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
f"({_HEX64.format(c='candidate_hash')})), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
_INDEX_DDL = {
# The queue reads pending items oldest first with a stable tiebreak.
"idx_media_review_items_queue": (
"CREATE INDEX idx_media_review_items_queue ON "
f"{ITEMS_TABLE}(state, created_at, id)"
),
# Deleting a pair or an image nulls decision navigation by index, not scan.
"idx_media_review_decisions_pair": (
"CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)"
),
"idx_media_review_decisions_image": (
"CREATE INDEX idx_media_review_decisions_image ON "
f"{DECISIONS_TABLE}(image_id)"
),
}
# ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key.
_AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2}
# (name, declared type, not null, default, primary key position)
_COLUMNS = {
METADATA_TABLE: (
("singleton_id", "INTEGER", 0, None, 1),
("schema_version", "INTEGER", 1, None, 0),
("review_index_revision", "INTEGER", 1, "0", 0),
),
ITEMS_TABLE: (
("id", "TEXT", 0, None, 1),
("byte_id", "TEXT", 1, None, 0),
("spelled_path", "TEXT", 1, None, 0),
("resolver_version", "INTEGER", 1, None, 0),
("evidence_hash", "TEXT", 1, None, 0),
("evidence", "TEXT", 1, None, 0),
("candidate_count", "INTEGER", 1, None, 0),
("limited", "INTEGER", 1, None, 0),
("state", "TEXT", 1, None, 0),
("revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("updated_at", "TEXT", 1, None, 0),
),
DECISIONS_TABLE: (
("id", "TEXT", 0, None, 1),
("item_id", "TEXT", 1, None, 0),
("decision_revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("outcome", "TEXT", 1, None, 0),
("candidate_hash", "TEXT", 0, None, 0),
("evidence", "TEXT", 1, None, 0),
("pair_id", "INTEGER", 0, None, 0),
("image_id", "INTEGER", 0, None, 0),
),
}
# (column, parent table, parent column, on delete)
_FOREIGN_KEYS = {
METADATA_TABLE: set(),
ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")},
DECISIONS_TABLE: {
("item_id", ITEMS_TABLE, "id", "RESTRICT"),
("pair_id", "prompt_pairs", "id", "SET NULL"),
("image_id", "generated_images", "id", "SET NULL"),
},
}
class _NonCanonical(ValueError):
pass
def _reject_constant(name: str) -> Any:
raise _NonCanonical(f"JSON constant {name} is not allowed")
# Built once: validation re-encodes every stored document at startup.
_ENCODER = json.JSONEncoder(
sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False
)
_DECODER = json.JSONDecoder(parse_constant=_reject_constant)
def canonical_evidence(value: Any) -> str:
"""Serialize one evidence object canonically; ``ValueError`` when invalid.
The text is a JSON object with sorted keys, no whitespace, ASCII only and
no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number
(ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's
policy everywhere (``utils.json_strict.dumps_strict``).
"""
if not isinstance(value, dict):
raise ValueError("Review evidence must be a JSON object")
try:
try:
text = _ENCODER.encode(value)
except ValueError as error:
if "Out of range float" not in str(error):
raise
text = dumps_strict(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
except (TypeError, RecursionError) as error:
raise ValueError(f"Review evidence is not serializable: {error}") from None
if len(text) > MAX_EVIDENCE_BYTES:
raise ValueError("Review evidence is larger than 8 MiB")
return text
def choosable(candidate: Any) -> bool:
"""True when a candidate offers a prompt to choose: it is ``resolved`` and
its positive side carries non-blank text. The page disables any other
candidate, so it is never something to choose.
"""
if not isinstance(candidate, dict) or candidate.get("state") != "resolved":
return False
positive = candidate.get("positive")
if not isinstance(positive, dict):
return False
text = positive.get("text")
return isinstance(text, str) and bool(text.strip())
# An item is worth a decision only when at least this many candidates offer
# a prompt; with fewer there is nothing to choose between (maintainer test of
# #284/#288: 39 items whose candidates were all "unresolved").
MIN_CHOOSABLE = 2
def evidence_offers_choice(evidence: Any) -> bool:
"""False when stored evidence provably offers nothing to choose.
Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A
limited item stores none, only ``candidate_total``; it may offer a choice
(more than ``MAX_CANDIDATES`` prompts) unless that total is too small.
"""
if not isinstance(evidence, dict):
return False
if evidence.get("limited"):
total = evidence.get("candidate_total")
return type(total) is int and total >= MIN_CHOOSABLE
candidates = evidence.get("candidates")
if not isinstance(candidates, list):
return False
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
def evidence_digest(text: str) -> str:
"""SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical)."""
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def _is_canonical(text: Any) -> bool:
if not isinstance(text, str):
return False
try:
value = _DECODER.decode(text)
return canonical_evidence(value) == text
except (ValueError, RecursionError):
return False
def _compact(sql: str) -> str:
return re.sub(r"\s+", "", sql).lower()
def _uuid(column: str) -> str:
"""SQL: *column* is a lowercase hyphenated 36-character UUID string."""
return (
f"typeof({column}) = 'text' AND length({column}) = 36 "
f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
f"AND length(replace({column}, '-', '')) = 32 "
f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
)
def _timestamp(column: str) -> str:
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
)
def _non_negative(column: str) -> str:
return f"typeof({column}) = 'integer' AND {column} >= 0"
def _hex64(column: str) -> str:
return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column)
def _evidence(column: str) -> str:
# Type and size only: the streamed pass below proves the text canonical
# (which implies printable ASCII), so a pattern scan here would read every
# document a second time.
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}"
)
def _nullable_integer(column: str) -> str:
return f"({column} IS NULL OR typeof({column}) = 'integer')"
def _in(column: str, values: tuple) -> str:
return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})"
# Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
_ROW_PREDICATES = {
ITEMS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("byte_id"),
"typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}",
"typeof(resolver_version) = 'integer'",
_hex64("evidence_hash"),
_evidence("evidence"),
_non_negative("candidate_count"),
f"candidate_count <= {MAX_CANDIDATES}",
"typeof(limited) = 'integer' AND limited IN (0, 1)",
_in("state", ITEM_STATES),
_non_negative("revision"),
_timestamp("created_at"),
_timestamp("updated_at"),
)
),
DECISIONS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("item_id"),
_non_negative("decision_revision"),
_timestamp("created_at"),
_in("outcome", DECISION_OUTCOMES),
"(CASE WHEN outcome = 'selected' THEN "
f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)",
_evidence("evidence"),
_nullable_integer("pair_id"),
_nullable_integer("image_id"),
)
),
}
def _expected_catalog() -> set:
objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in _AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def _catalog(conn: sqlite3.Connection) -> list:
names = TABLES + tuple(_INDEX_DDL)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return conn.execute( # only "?" placeholders are interpolated
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + TABLES,
).fetchall()
def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
for table, expected in _COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[4], row[5])
for row in conn.execute(f"PRAGMA table_info({table})")
)
if columns != expected:
return f"Review queue table {table} has invalid columns"
foreign_keys = {
(row[3], row[2], row[4], row[6])
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
}
if foreign_keys != _FOREIGN_KEYS[table]:
return f"Review queue table {table} has invalid foreign keys"
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
return f"Review queue table {table} contains foreign key violations"
return None
def _evidence_error(conn: sqlite3.Connection) -> Optional[str]:
"""Stream every evidence document once; one row is held at a time."""
for evidence_hash, text in conn.execute(
f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}" # nosec B608
):
if not _is_canonical(text) or evidence_digest(text) != evidence_hash:
return f"Review queue table {ITEMS_TABLE} contains invalid evidence"
for (text,) in conn.execute(
f"SELECT evidence FROM {DECISIONS_TABLE}" # nosec B608
):
if not _is_canonical(text):
return f"Review queue table {DECISIONS_TABLE} contains invalid evidence"
return None
def _value_error(conn: sqlite3.Connection) -> Optional[str]:
# Only this module's constants are interpolated into the SQL below.
rows = conn.execute(
"SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' " # nosec B608
"AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
f"AND schema_version = {SCHEMA_VERSION} "
f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}"
).fetchone()
if rows[0] != 1 or rows[1] != 1:
return "Review queue metadata is invalid"
for table, predicate in _ROW_PREDICATES.items():
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1" # nosec B608
).fetchone():
return f"Review queue table {table} contains invalid values"
return _evidence_error(conn)
def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
"""Catalog only: a review group present without its annotation tables.
Startup asks this before the annotation group installs, which would make
a review group copied without it look valid. It reads no rows; the full
validation runs once in :func:`migrate_review_state`.
"""
if not _catalog(conn):
return None
present = {
name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
}
if present != set(ANNOTATION_TABLES):
return (
"Review queue schema requires a valid image annotation schema: "
"Missing image annotation schema"
)
return None
def review_schema_error(
conn: sqlite3.Connection,
*,
allow_missing: bool = True,
annotations_valid: bool = False,
) -> Optional[str]:
"""Reject partial or unsupported review schemas and invalid rows.
Reads only: nothing is ever repaired. A database without any of the
group's objects is acceptable for backups that predate it unless
``allow_missing`` is false. A present group requires a present, valid
image annotation group; ``annotations_valid`` says the caller has just
validated it (startup's annotation install), so only its presence is
checked again.
"""
objects = _catalog(conn)
if not objects:
return None if allow_missing else "Missing review queue schema"
if annotations_valid:
annotation_error = review_without_annotations_error(conn)
else:
annotation_error = annotation_schema_error(conn, allow_missing=False)
if annotation_error:
annotation_error = (
"Review queue schema requires a valid image annotation schema: "
+ annotation_error
)
if annotation_error:
return annotation_error
present = {name for kind, name, _sql in objects if kind == "table"}
if present != set(TABLES):
return "Incomplete review queue schema"
actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
if actual != _expected_catalog():
return "Review queue has an invalid schema definition"
return _structure_error(conn) or _value_error(conn)
def migrate_review_state(conn: sqlite3.Connection) -> None:
"""Install the group atomically; validate before and after creating it.
Runs after ``migrate_annotation_state`` (which has just validated the
annotation group): the group is only valid beside the annotation group it
cites. A present group is validated once, evidence included.
"""
conn.execute("SAVEPOINT media_review_schema")
try:
installed = bool(_catalog(conn))
error = review_schema_error(conn, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
if installed:
# Already complete and valid: nothing to create, and the evidence
# pass is the expensive part of validation, so it runs once.
conn.execute("RELEASE media_review_schema")
return
for sql in _TABLE_DDL.values():
conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
for sql in _INDEX_DDL.values():
conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
conn.execute(
f"INSERT OR IGNORE INTO {METADATA_TABLE} "
"(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)",
(SCHEMA_VERSION,),
)
error = review_schema_error(conn, allow_missing=False, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE media_review_schema")
except BaseException:
conn.execute("ROLLBACK TO media_review_schema")
conn.execute("RELEASE media_review_schema")
raise
+16
View File
@@ -8,6 +8,10 @@ Split into domain-specific mixins for maintainability:
- AutotagRoutesMixin: auto-tagging model management and tagging
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
- ThumbnailJobsMixin: thumbnail plan, thumbnail job and the gallery stream on it
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
- ReviewEventsMixin: the Needs review change stream an open page re-reads on
- SystemRoutesMixin: System health check, Overview and usage statistics
"""
@@ -30,6 +34,9 @@ from .pairs import PairRoutesMixin
from .library import LibraryRoutesMixin
from .vocabulary import VocabularyRoutesMixin
from .annotations import AnnotationRoutesMixin
from .review import ReviewRoutesMixin
from .review_recording import ReviewRecordingMixin
from .review_events import ReviewEventsMixin
from .saved_views import SavedViewRoutesMixin
from .history import HistoryRoutesMixin
from .images import ImageRoutesMixin
@@ -37,6 +44,7 @@ from .admin import AdminRoutesMixin
from .settings_routes import SettingsRoutesMixin
from .job_routes import JobRoutesMixin
from .image_tag_jobs import ImageTagJobsMixin
from .thumbnail_jobs import ThumbnailJobsMixin
from .system import SystemRoutesMixin
from .jobs import JobRegistry
from .logging_routes import LoggingRoutesMixin
@@ -264,6 +272,9 @@ class PromptManagerAPI(
LibraryRoutesMixin,
VocabularyRoutesMixin,
AnnotationRoutesMixin,
ReviewRoutesMixin,
ReviewRecordingMixin,
ReviewEventsMixin,
SavedViewRoutesMixin,
HistoryRoutesMixin,
PromptRoutesMixin,
@@ -272,6 +283,7 @@ class PromptManagerAPI(
SettingsRoutesMixin,
JobRoutesMixin,
ImageTagJobsMixin,
ThumbnailJobsMixin,
SystemRoutesMixin,
LoggingRoutesMixin,
AutotagRoutesMixin,
@@ -304,6 +316,8 @@ class PromptManagerAPI(
# created on first use with their restore subscription.
self._captures = None
self._captures_unsubscribe = None
# Set by close_annotation_captures (shutdown): no store is recreated.
self._captures_closed = False
self._cached_output_dir = None # Lazy-cached by _find_comfyui_output_dir()
self._html_cache = {} # Cached HTML file contents keyed by path
self._gallery_cache = {} # dict: path_str -> (files, timestamp)
@@ -484,6 +498,8 @@ class PromptManagerAPI(
self._register_library_routes(routes)
self._register_vocabulary_routes(routes)
self._register_annotation_routes(routes)
self._register_review_routes(routes)
self._register_review_event_routes(routes)
self._register_saved_view_routes(routes)
self._register_history_routes(routes)
self._register_prompt_routes(routes)
+121 -28
View File
@@ -4,6 +4,7 @@ import asyncio
import contextlib
from contextlib import closing
import datetime
import functools
import hashlib
import json
import math
@@ -44,6 +45,7 @@ try:
parse_page_json,
)
from ...utils.prompt_pair_resolver import resolve_prompt_pair
from ...utils.review_capture import reviewable
except ImportError:
from database.operations import PromptDatabase
from database.pair_ingestion import ingest_resolved_pair
@@ -54,6 +56,7 @@ except ImportError:
parse_page_json,
)
from utils.prompt_pair_resolver import resolve_prompt_pair
from utils.review_capture import reviewable
try:
@@ -211,6 +214,30 @@ def _time_left(done, total, batches, elapsed):
return f" · about {minutes} min left"
def new_review_counts():
"""The scan's Needs review tally: ``added`` new items, ``queued`` files
in the queue after this scan (new or already there), ``settled`` items
removed because their file is linked now, ``untraced`` items removed
because they offer nothing to choose, and ``errors``."""
return {"added": 0, "queued": 0, "settled": 0, "untraced": 0, "errors": 0}
def _review_scope(review):
"""The scan job's ``scope`` keys for its Needs review tally.
``review_items`` is the number of this scan's files that are in the
pending queue after it: the review group's ambiguous files, less those a
capture could not read or a decision already settled.
"""
return {
"review_items": review["queued"],
"review_added": review["added"],
"review_settled": review["settled"],
"review_untraced": review["untraced"],
"review_errors": review["errors"],
}
def _scan_status_line(done, total, batches, elapsed):
""" "Processing file N of M", with an estimate once enough batches ran."""
line = f"Processing file {done:,} of {total:,}"
@@ -1344,28 +1371,34 @@ class AdminRoutesMixin:
}
)
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE):
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE, roots=None, output_dirs=None):
"""Register the scan job synchronously; its runner starts on the loop.
``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another
record-changing job runs.
``files`` is the scope (``SCAN_SCOPES``). ``roots`` selects sources by
their index in ``output_dirs`` (the effective sources the caller
resolved); None scans every source, resolved when the job starts.
Raises JobBusy while another record-changing job runs.
"""
if files not in SCAN_SCOPES:
raise ValueError(f"Unknown scan scope: {files!r}")
if roots is not None and (
output_dirs is None or not roots or max(roots) >= len(output_dirs)
):
raise ValueError("A scoped scan needs the sources its roots index")
job = self._jobs.start(
SCAN_JOB_KIND,
SCAN_JOB_TITLE,
record_changing=True,
scope={"files": files},
runner=self._run_scan_job,
scope={"files": files, "roots": None if roots is None else list(roots)},
runner=functools.partial(self._run_scan_job, output_dirs=output_dirs),
groups=SCAN_GROUP_NAMES,
)
self._scan_job = job
return job
async def _run_scan_job(self, job):
async def _run_scan_job(self, job, output_dirs=None):
final = None
async for payload in self._scan_images_events(job):
async for payload in self._scan_images_events(job, output_dirs):
job.publish(payload)
final = payload.get("type")
state = SCAN_FINAL_STATES.get(final, "finished")
@@ -1400,16 +1433,25 @@ class AdminRoutesMixin:
)
return response
async def _scan_images_events(self, job):
"""Yield progress payloads while scanning every configured output directory.
async def _scan_images_events(self, job, output_dirs=None):
"""Yield progress payloads while scanning the output directories.
The payloads are the legacy SSE events, unchanged; ``job`` also gets
its status line, totals and one outcome per processed file.
Every effective source is walked, or only those ``job.scope["roots"]``
selects (indexes into ``output_dirs``). The payloads are the legacy SSE
events, unchanged; ``job`` also gets its status line, totals and one
outcome per processed file. Rows keep each source's listing index.
"""
try:
self.logger.info("Starting image scan operation")
output_dirs = self._get_all_output_dirs()
if not output_dirs:
if output_dirs is None:
output_dirs = self._get_all_output_dirs()
selected = job.scope.get("roots")
walked = (
output_dirs
if selected is None
else [output_dirs[index] for index in selected]
)
if not walked:
self.logger.error("No output directories found")
message = (
"No output directories found. "
@@ -1419,18 +1461,18 @@ class AdminRoutesMixin:
yield {"type": "error", "message": message}
return
dir_names = [self._public_path(d) for d in output_dirs]
folders = _plural(len(output_dirs), "output folder")
job.scope["output_folders"] = len(output_dirs)
dir_names = [self._public_path(d) for d in walked]
folders = _plural(len(walked), "output folder")
job.scope["output_folders"] = len(walked)
scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")]
job.log_line(f"Scan started · {folders} · {scope}")
job.update(status_line=f"Looking for media files in {folders}")
yield self._scan_progress(
0, f"Scanning {len(output_dirs)} directory(ies) for media files..."
0, f"Scanning {len(walked)} directory(ies) for media files..."
)
media_files = await self._run_in_executor(
self._collect_output_media_sync, output_dirs
self._collect_output_media_sync, walked
)
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
job.update(
@@ -1445,10 +1487,13 @@ class AdminRoutesMixin:
known = set()
if media_files and job.scope.get("files") != "all":
known = await self._known_scan_files(job, media_files)
review = new_review_counts()
job.scope.update(_review_scope(review))
async for event in self._scan_batches(
media_files, counts, job, roots, known
media_files, counts, job, roots, known, review
):
yield event
self._log_review_items(job, review)
self.logger.info(
f"Scan completed: processed={counts['processed']}, "
@@ -1485,12 +1530,16 @@ class AdminRoutesMixin:
sizes = {str(media_file): _file_size(media_file) for media_file in media_files}
return self.db.get_known_resolved_paths(sizes)
async def _scan_batches(self, media_files, counts, job, roots=(), known=()):
async def _scan_batches(
self, media_files, counts, job, roots=(), known=(), review=None
):
"""Process ``media_files`` in executor batches, updating ``counts`` in place.
Records each file's outcome in ``job`` and yields one progress event
per batch. Files in ``known`` (path strings) are not read: they count
as processed and unchanged and are recorded as skipped.
as processed and unchanged and are recorded as skipped. ``review``
counts the Needs review items the batches record (``job.scope``
``review_items`` / ``review_errors`` follow it after every batch).
"""
total = len(media_files)
started = time.monotonic()
@@ -1513,9 +1562,16 @@ class AdminRoutesMixin:
batch_results,
roots=roots,
outcomes=outcomes,
review=review,
)
for key in counts:
counts[key] += batch_counts[key]
if review is not None:
# After the batch's links: a file linked now leaves the queue.
await self._run_in_executor(
self._settle_scanned_reviews, batch, roots, review
)
job.scope.update(_review_scope(review))
for group, row, log_line in outcomes:
job.record(group, row, log_line)
@@ -1565,12 +1621,20 @@ class AdminRoutesMixin:
return map_parallel(extract, file_batch, PromptManagerConfig.WORKER_THREADS)
def _ingest_scan_batch_sync(self, batch_results, roots=(), outcomes=None):
def _ingest_scan_batch_sync(
self, batch_results, roots=(), outcomes=None, review=None
):
"""Store the prompts found in a metadata batch (blocking DB work).
Returns the legacy counts. When ``outcomes`` is a list, every processed
file appends its ``(group, row, log_line)`` for the scan job, with the
path relative to its root in ``roots`` (see ``_scan_roots``).
path relative to its root in ``roots`` (see ``_scan_roots``). A file
whose final outcome is ambiguous and unlinked (the resolver found it
ambiguous and the fallback did not link it) is recorded for Needs
review (best effort, counted in ``review`` when given); one a saved
row already links is reported ``unchanged`` and never queued. An
ambiguous file with nothing to choose is reported ``unresolved`` and
its pending items without a choice leave the queue.
"""
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
@@ -1582,13 +1646,27 @@ class AdminRoutesMixin:
if self._scan_job and self._scan_job.cancelled:
break
counts["processed"] += 1
ambiguous, untraced = [], []
try:
outcome = self._ingest_scanned_file(media_file, metadata, detailed=True)
outcome = self._ingest_scanned_file(
media_file,
metadata,
detailed=True,
ambiguous=ambiguous,
untraced=untraced,
)
except Exception as e:
self.logger.error(f"Error processing {media_file.name}: {e}")
counts["failed"] += 1
note(media_file, "error")
continue
if ambiguous and outcome == "ambiguous":
if self._review_file_linked(media_file, roots):
outcome = "unchanged" # already linked: not a review case
else:
self._record_scanned_review(media_file, roots, review)
elif untraced and outcome == "unresolved":
self._settle_untraced_reviews(untraced, roots, review)
if outcome is None:
note(media_file, None)
continue
@@ -1622,13 +1700,21 @@ class AdminRoutesMixin:
return None
return positive.strip() or None
def _ingest_scanned_file(self, media_file, metadata, detailed=False):
def _ingest_scanned_file(
self, media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
"""Enrich exact pairs; retain legacy positive-only import for absent pairs.
Returns the file's scan status, or None without prompt metadata. With
``detailed``, a stored or linked file returns a dict instead: its
``status``, the prompt's display ``title`` and, for a resolved pair,
the pair counts.
the pair counts. When the resolver finds the file ambiguous (whether
or not the browser parser then links it) and ``ambiguous`` is a list,
``media_file`` is appended to it; only a file that then returns
``"ambiguous"`` (nothing linked it) is a Needs review case. An
ambiguous result with nothing to choose (``reviewable`` is False: fewer
than two candidates carry a prompt) that nothing links returns
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
"""
if not metadata:
return None
@@ -1644,6 +1730,13 @@ class AdminRoutesMixin:
return None
_with_file_size(canonical, media_file)
resolution = resolve_prompt_pair(canonical)
# Ambiguous without a choice: unresolved unless something links it.
unchoosable = resolution["state"] == "ambiguous" and not reviewable(resolution)
if unchoosable:
if untraced is not None:
untraced.append(media_file)
elif resolution["state"] == "ambiguous" and ambiguous is not None:
ambiguous.append(media_file)
if resolution["state"] == "resolved":
result = ingest_resolved_pair(
self.db, str(media_file), canonical, resolution
@@ -1663,9 +1756,9 @@ class AdminRoutesMixin:
# Graphs without a recognized sampler keep the legacy extractor below.
prompt_text = self._graph_prompt_text(metadata)
if not prompt_text:
return resolution["state"]
return "unresolved" if unchoosable else resolution["state"]
elif resolution["state"] == "ambiguous":
return "ambiguous"
return "unresolved" if unchoosable else "ambiguous"
elif resolution["state"] == "unresolved" and not isinstance(
canonical.get("parameters"), str
):
+35 -13
View File
@@ -13,6 +13,7 @@ import datetime
import functools
import os
import re
import threading
from pathlib import Path
from aiohttp import web
@@ -73,6 +74,9 @@ _HEX64 = re.compile(r"[0-9a-f]{64}")
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
# secrets.token_urlsafe(32) is 43 characters; the bound only rejects junk.
_TOKEN = re.compile(r"[A-Za-z0-9_-]{1,128}")
# The store is created on first use by the event loop, the scan's executor or
# the monitor's worker (Needs review), so its creation is serialized.
_CAPTURE_STORE_LOCK = threading.Lock()
_PREVIEW_TYPES = {
".png": "image/png",
@@ -259,31 +263,46 @@ class AnnotationRoutesMixin:
Created lazily so an API object that never observes a file (most tests,
a server that never opens the Tag dialog) owns no temp directory. A
restore of the database dooms every capture (``invalidate_all``).
Thread-safe: Needs review recording calls it off the event loop.
Raises ``CaptureError`` after ``close_annotation_captures``.
"""
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
with _CAPTURE_STORE_LOCK:
if getattr(self, "_captures_closed", False):
# A recording still in flight at shutdown must not create a
# new temp directory after the cleanup removed the old one.
raise CaptureError("The capture store is closed")
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
def _invalidate_captures(self) -> None:
store = self._captures
if store is not None:
store.invalidate_all()
def close_annotation_captures(self) -> None:
"""Remove every capture file and the owned directory. Idempotent."""
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
def close_annotation_captures(self, *, final: bool = False) -> None:
"""Remove every capture file and the owned directory. Idempotent.
Taken under the store lock. ``final`` (the shutdown cleanup) also makes
``_capture_store`` refuse to create another store, so a recording
still in flight cannot leak a new temp directory after it.
"""
with _CAPTURE_STORE_LOCK:
if final:
self._captures_closed = True
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
store, self._captures = self._captures, None
if unsubscribe is not None:
unsubscribe()
store, self._captures = self._captures, None
if store is not None:
store.close()
async def cleanup_annotation_captures(self, app: web.Application) -> None:
self.close_annotation_captures()
self.close_annotation_captures(final=True)
# -- shared request steps --------------------------------------------------
@@ -329,7 +348,10 @@ class AnnotationRoutesMixin:
root, error = self._scoped_root(index, scope)
if error is not None:
return error
store = self._capture_store()
try:
store = self._capture_store()
except CaptureError as exc: # the server is shutting down
return _capture_error(exc)
try:
epoch = await self._run_in_executor(self.db.get_execution_epoch)
capture = await _settled(
+192 -183
View File
@@ -20,7 +20,7 @@ from typing import Callable, NamedTuple
from aiohttp import web
from PIL import Image, UnidentifiedImageError
from .jobs import THUMBNAILS_JOB_KIND
from .jobs import MAX_OUTCOME_ROWS, THUMBNAILS_JOB_KIND
from .prompts import bad_request, parse_page_params, publish_image_paths
from .prompts import thumbnail_url_for
from .prompts import safe_error_message as _safe_error
@@ -235,6 +235,8 @@ MEDIA_EXTENSIONS = GALLERY_IMAGE_EXTENSIONS | VIDEO_EXTENSIONS
# Caps on request-triggered filesystem work.
MAX_SCAN_DEPTH = 12
MAX_THUMBNAIL_FILES = 10_000
# Already-fresh files a thumbnail job lists under Skipped (its counter stays exact).
MAX_FRESH_LISTED = MAX_OUTCOME_ROWS
def _worker_threads():
@@ -271,15 +273,52 @@ def _atomic_save(img, dst, fmt=None, **save_kwargs):
tmp.unlink()
def _write_image_thumbnail(src, dst, thumbnail_size):
"""Resize *src* into *dst* (blocking PIL work), atomically."""
# Encoder settings per thumbnail quality (the System thumbnail dialog's
# "Medium · optimized", "High · original", "Low · fastest"). The legacy
# routes always encode as "medium", which is what they always wrote.
THUMBNAIL_ENCODINGS = {
"low": {"quality": 70, "optimize": False},
"medium": {"quality": 85, "optimize": True},
"high": {"quality": 95, "optimize": True},
}
DEFAULT_THUMBNAIL_ENCODING = THUMBNAIL_ENCODINGS["medium"]
# ffmpeg's JPEG scale (-q:v, 2 best to 31 worst) for each encoding's quality.
_FFMPEG_QSCALE = {95: 2, 85: 4, 70: 8}
# A preview that could not be replaced (#249 S5 M6). Windows refuses to rename
# over a file another process has open (WinError 5 or 32), such as a preview
# the gallery is serving; the old preview stays and the file counts as an error.
PREVIEW_IN_USE = "Preview in use by another program; the old one was kept"
PREVIEW_NOT_WRITABLE = "Permission denied writing the preview"
_WINDOWS_LOCKED = (5, 32) # ERROR_ACCESS_DENIED, ERROR_SHARING_VIOLATION
def _preview_write_error(exc):
"""The readable error for a PermissionError while writing a preview."""
if getattr(exc, "winerror", None) in _WINDOWS_LOCKED:
return PREVIEW_IN_USE
return PREVIEW_NOT_WRITABLE
def _ffmpeg_qscale(encoding):
"""ffmpeg's ``-q:v`` for a THUMBNAIL_ENCODINGS entry."""
return str(_FFMPEG_QSCALE.get(encoding["quality"], 4))
def _write_image_thumbnail(src, dst, thumbnail_size, encoding=None):
"""Resize *src* into *dst* (blocking PIL work), atomically.
``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); a PNG
preview keeps only its ``optimize`` flag.
"""
encoding = encoding or DEFAULT_THUMBNAIL_ENCODING
with Image.open(src) as img:
if img.mode in ("RGBA", "LA", "P"):
img = img.convert("RGB")
img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS)
save_kwargs = {"quality": 85, "optimize": True}
save_kwargs = dict(encoding)
if dst.suffix.lower() == ".png":
save_kwargs = {"optimize": True}
save_kwargs = {"optimize": encoding["optimize"]}
_atomic_save(img, dst, **save_kwargs)
@@ -671,6 +710,25 @@ def _iter_media_files(output_path, max_files):
yield root_path / name
def media_thumbnail_url(output_path, rel_path, root_index, *, root_scope=None):
"""The Images listing's thumbnail URL for *rel_path* under *output_path*,
or None when no thumbnail exists (one blocking stat). The Needs review
queue cards use the same URL (#249 F, M10)."""
rel_path = Path(rel_path)
extension = rel_path.suffix.lower()
thumb_rel = (
f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb"
f"{'.jpg' if extension in VIDEO_EXTENSIONS else extension}"
)
thumbnail_url = thumbnail_url_for(thumb_rel, Path(output_path) / thumb_rel)
if thumbnail_url is None:
return None
thumbnail_url += f"&root={root_index}"
if root_scope is not None:
thumbnail_url += f"&root_scope={root_scope}"
return thumbnail_url
def _output_image_entry(
media_path, output_path, root_index, public_root=None, *, root_scope=None
):
@@ -686,19 +744,13 @@ def _output_image_entry(
extension = media_path.suffix.lower()
is_video = extension in VIDEO_EXTENSIONS
thumb_rel = (
f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb"
f"{'.jpg' if is_video else extension}"
thumbnail_url = media_thumbnail_url(
output_path, rel_path, root_index, root_scope=root_scope
)
thumbnail_url = thumbnail_url_for(thumb_rel, output_path / thumb_rel)
if thumbnail_url is not None:
thumbnail_url += f"&root={root_index}"
encoded_path = urllib.parse.quote(rel_path.as_posix(), safe="/")
url = f"/prompt_manager/images/serve/{encoded_path}?root={root_index}"
if root_scope is not None:
url += f"&root_scope={root_scope}"
if thumbnail_url is not None:
thumbnail_url += f"&root_scope={root_scope}"
entry = {
"id": hashlib.sha1(
@@ -932,6 +984,10 @@ class ImageRoutesMixin:
async def thumbnail_status_route(request):
return await self.get_thumbnail_status(request)
@routes.get("/prompt_manager/images/thumbnails/plan")
async def thumbnail_plan_route(request):
return await self.get_thumbnail_plan(request)
@routes.post("/prompt_manager/images/generate-thumbnails")
async def generate_thumbnails_route(request):
return await self.generate_thumbnails(request)
@@ -1804,15 +1860,35 @@ class ImageRoutesMixin:
self.logger.info(payload["message"])
return payload
def _plan_gallery_thumbnails(self, output_dirs: list[Path]) -> dict:
"""Plan every gallery root with one shared per-run generation cap."""
combined = {"targets": [], "skipped": 0, "remaining": 0}
def _plan_gallery_thumbnails(
self, output_dirs: list[Path], rebuild: bool = False, stop=None
) -> dict:
"""Plan every gallery root with one shared per-run generation cap.
See ``_plan_thumbnails``; ``fresh_files`` keeps at most
MAX_FRESH_LISTED files across the roots. ``stop()`` true ends the walk
at the next file (a cancelled job); the partial plan is returned.
"""
combined = {
"targets": [],
"skipped": 0,
"remaining": 0,
"fresh": 0,
"fresh_files": [],
}
for root in output_dirs:
if stop is not None and stop():
break
root = Path(root)
plan = self._plan_thumbnails(root, root / "thumbnails")
plan = self._plan_thumbnails(
root, root / "thumbnails", rebuild=rebuild, stop=stop
)
available = max(0, MAX_THUMBNAIL_FILES - len(combined["targets"]))
combined["targets"].extend(plan["targets"][:available])
combined["skipped"] += plan["skipped"]
combined["fresh"] += plan["fresh"]
room = max(0, MAX_FRESH_LISTED - len(combined["fresh_files"]))
combined["fresh_files"].extend(plan["fresh_files"][:room])
combined["remaining"] += plan["remaining"] + max(
0, len(plan["targets"]) - available
)
@@ -1845,22 +1921,30 @@ class ImageRoutesMixin:
{"success": False, "error": _safe_error(exc)}, status=500
)
def _plan_thumbnails(self, output_path, thumbnails_dir):
def _plan_thumbnails(self, output_path, thumbnails_dir, rebuild=False, stop=None):
"""Decide which media files need a thumbnail (blocking walk).
Returns ``{"targets": [(src, dst, is_video)], "skipped": n, "remaining": n}``.
Files whose thumbnail is already newer than the source are counted as
skipped and never queued, so a large library that is mostly done only
pays for the files that changed. The walk is bounded like the gallery
listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES, no symlinks) and at most
MAX_THUMBNAIL_FILES pending files are queued per run; ``remaining``
counts the pending files beyond that cap.
Returns ``{"targets": [(src, dst, is_video)], "skipped": n,
"remaining": n, "fresh": n, "fresh_files": [src]}``. Files whose
thumbnail is already newer than the source are ``fresh``. Without
``rebuild`` they are counted as skipped and never queued, so a large
library that is mostly done only pays for the files that changed;
``fresh_files`` lists the first MAX_FRESH_LISTED of them. With
``rebuild`` every file is a target (nothing is skipped). The walk is
bounded like the gallery listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES,
no symlinks) and at most MAX_THUMBNAIL_FILES targets are queued per
run; ``remaining`` counts the targets beyond that cap. ``stop()`` true
ends the walk before the next file.
"""
output_path = Path(output_path)
targets = []
skipped = 0
remaining = 0
fresh = 0
fresh_files = []
for src in _iter_media_files(output_path, MAX_GALLERY_FILES):
if stop is not None and stop():
break
suffix = src.suffix.lower()
is_video = suffix in VIDEO_EXTENSIONS
rel_no_ext = src.relative_to(output_path).with_suffix("")
@@ -1870,8 +1954,12 @@ class ImageRoutesMixin:
self.logger.warning(f"Skipping thumbnail outside safe dir: {src.name}")
continue
if _thumbnail_is_fresh(src, dst):
skipped += 1
continue
fresh += 1
if not rebuild:
skipped += 1
if len(fresh_files) < MAX_FRESH_LISTED:
fresh_files.append(src)
continue
if len(targets) >= MAX_THUMBNAIL_FILES:
remaining += 1
continue
@@ -1881,190 +1969,81 @@ class ImageRoutesMixin:
f"Thumbnail run capped at {MAX_THUMBNAIL_FILES} new files; "
f"{remaining} more will be generated on the next run"
)
return {"targets": targets, "skipped": skipped, "remaining": remaining}
return {
"targets": targets,
"skipped": skipped,
"remaining": remaining,
"fresh": fresh,
"fresh_files": fresh_files,
}
def _thumbnail_targets(self, output_path, thumbnails_dir):
"""Pending (src, dst, is_video) triples; see _plan_thumbnails."""
return self._plan_thumbnails(output_path, thumbnails_dir)["targets"]
def _generate_batch(self, targets, thumbnail_size):
def _generate_batch(self, targets, thumbnail_size, **options):
"""Create thumbnails for ``targets`` on the configured worker threads.
Blocking; returns one result dict per target, in order.
Blocking; returns one result dict per target, in order. ``options``
go to ``_generate_one`` (``encoding``, ``rebuild``, ``stop``).
"""
def one(target):
src, dst, is_video = target
return self._generate_one(src, dst, thumbnail_size, is_video)
return self._generate_one(src, dst, thumbnail_size, is_video, **options)
return map_parallel(one, targets, _worker_threads())
def _generate_one(self, src, dst, thumbnail_size, is_video):
"""Create one thumbnail (blocking). Never raises; returns a result dict."""
def _generate_one(
self,
src,
dst,
thumbnail_size,
is_video,
encoding=None,
rebuild=False,
stop=None,
):
"""Create one thumbnail (blocking). Never raises; returns a result dict.
``stop()`` true before the file starts gives ``action: "cancelled"``
and touches nothing (a job's file boundary). ``rebuild`` writes a
fresh preview again and never replaces an existing video preview with
the placeholder. Every write is atomic, so a failure keeps the old
preview.
"""
result = {
"file": src.name,
"dir": src.parent.name,
"type": "video" if is_video else "image",
}
try:
if stop is not None and stop():
return {**result, "action": "cancelled"}
if not src.is_file():
return {**result, "action": "error", "error": "File no longer exists"}
if _thumbnail_is_fresh(src, dst):
if not rebuild and _thumbnail_is_fresh(src, dst):
return {**result, "action": "skipped"}
dst.parent.mkdir(parents=True, exist_ok=True)
if is_video:
if not self._generate_video_thumbnail(src, dst, thumbnail_size):
made = self._generate_video_thumbnail(
src, dst, thumbnail_size, encoding, keep_existing=rebuild
)
if not made:
return {**result, "action": "error", "error": "Video decode failed"}
else:
_write_image_thumbnail(src, dst, thumbnail_size)
_write_image_thumbnail(src, dst, thumbnail_size, encoding)
return {**result, "action": "generated"}
except UnidentifiedImageError:
return {**result, "action": "error", "error": "Not a valid image file"}
except PermissionError as e:
message = _preview_write_error(e)
self.logger.warning(f"Thumbnail failed for {src.name}: {message}")
return {**result, "action": "error", "error": message}
except Exception as e:
self.logger.warning(f"Thumbnail failed for {src.name}", exc_info=True)
return {**result, "action": "error", "error": _safe_error(e)}
async def generate_thumbnails_with_progress(self, request):
"""Generate thumbnails with Server-Sent Events progress updates (POST).
``quality`` is read from the query string so a POST-capable SSE
client can keep using the same URL shape. Every blocking step (scan,
PIL/ffmpeg) runs in the executor; the event loop only writes SSE
frames between files.
"""
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
response = web.StreamResponse(
status=200,
headers={
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
},
)
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
if busy is not None:
# The gallery reads this stream: refuse with its error event.
await response.prepare(request)
message = f"{busy.title} already running"
await self._emit_sse(
response, "error", {"error": message, "message": message}
)
return response
async with self._jobs.hold(
THUMBNAILS_JOB_KIND, THUMBNAIL_GENERATION_TITLE, admission="guarded"
) as job:
await response.prepare(request)
if not await self._thumbnail_progress_stream(response, thumbnail_size):
job.finish("failed") # an error event ended the stream
return response
async def _thumbnail_progress_stream(self, response, thumbnail_size) -> bool:
"""Stream one generation run; False when it ended with an error event."""
try:
output_dirs = self._get_all_output_dirs()
if not output_dirs:
await self._emit_sse(
response, "error", {"error": "ComfyUI output directory not found"}
)
return False
await self._emit_sse(
response,
"status",
{"phase": "scanning", "message": "Scanning for images and videos..."},
)
plan = await self._run_in_executor(
self._plan_gallery_thumbnails, output_dirs
)
await self._stream_thumbnails(
response,
plan["targets"],
thumbnail_size,
skipped=plan["skipped"],
remaining=plan["remaining"],
)
self.invalidate_gallery_cache()
return True
except Exception:
self.logger.exception("Thumbnail generation failed")
await self._emit_sse(
response,
"error",
{
"error": "An internal error occurred",
"message": "Thumbnail generation failed. Check server logs.",
},
)
return False
async def _stream_thumbnails(
self, response, targets, thumbnail_size, skipped=0, remaining=0
):
"""Process pending *targets* in worker-thread batches, emitting SSE per file.
``skipped`` files already had fresh thumbnails and ``remaining`` were
left for the next run; both only feed the start/complete payloads.
"""
total = len(targets)
video_count = sum(1 for _, _, is_video in targets if is_video)
message = (
f"Found {total - video_count} images and {video_count} videos to process"
)
if skipped:
message += f" ({skipped} already have thumbnails)"
connected = await self._emit_sse(
response,
"start",
{
"total_images": total,
"skipped": skipped,
"remaining": remaining,
"phase": "processing",
"image_count": total - video_count,
"video_count": video_count,
"message": message,
},
)
if not connected:
return
stats = _new_thumbnail_stats()
stats["skipped"] = skipped
start = _time.monotonic()
batch_size = _thumbnail_batch_size(_worker_threads())
index = 0
for batch_start in range(0, total, batch_size):
batch = targets[batch_start : batch_start + batch_size]
results = await self._run_in_executor(
self._generate_batch, batch, thumbnail_size
)
for result in results:
index += 1
_record_thumbnail_result(stats, result)
if result["action"] == "error":
await self._emit_sse(
response,
"file_error",
{"file": result["file"], "error": result["error"]},
)
connected = await self._emit_sse(
response,
"progress",
_thumbnail_progress_payload(index, total, stats, result, start),
)
if not connected:
self.logger.info("Thumbnail client disconnected; stopping early")
return
await asyncio.sleep(0) # let other requests run between batches
payload = _thumbnail_complete_payload(
total + skipped + remaining,
stats,
start,
remaining=remaining,
processed=total,
)
await self._emit_sse(response, "complete", payload)
self.logger.info(payload["message"])
async def _emit_sse(self, response, event, data) -> bool:
"""Write one SSE frame. Returns False once the client has gone away."""
try:
@@ -2075,8 +2054,25 @@ class ImageRoutesMixin:
self.logger.warning(f"Failed to send SSE message: {e}")
return False
def _generate_video_thumbnail(self, video_path, thumbnail_path, thumbnail_size):
"""Generate thumbnail from video file. Returns True if successful."""
def _generate_video_thumbnail(
self,
video_path,
thumbnail_path,
thumbnail_size,
encoding=None,
keep_existing=False,
):
"""Generate thumbnail from video file. Returns True if successful.
``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); it
encodes the OpenCV frame, ffmpeg's frame (as ``-q:v``) and the
placeholder alike. Both decoders fit the frame inside
``thumbnail_size`` keeping its aspect ratio. With ``keep_existing`` an
existing preview is kept, and the run fails, when no frame could be
decoded, rather than replaced by the placeholder. A PermissionError
while replacing the preview propagates (``_generate_one`` names it).
"""
encoding = encoding or DEFAULT_THUMBNAIL_ENCODING
try:
# Try using OpenCV first (most reliable)
try:
@@ -2100,7 +2096,7 @@ class ImageRoutesMixin:
img = Image.fromarray(frame_rgb)
img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS)
_atomic_save(img, thumbnail_path, "JPEG", quality=85, optimize=True)
_atomic_save(img, thumbnail_path, "JPEG", **encoding)
self.logger.debug(
f"Generated video thumbnail using OpenCV: {thumbnail_path}"
)
@@ -2122,8 +2118,11 @@ class ImageRoutesMixin:
"00:00:01",
"-vframes",
"1",
"-s",
f"{thumbnail_size[0]}x{thumbnail_size[1]}",
"-vf",
f"scale={thumbnail_size[0]}:{thumbnail_size[1]}"
":force_original_aspect_ratio=decrease",
"-q:v",
_ffmpeg_qscale(encoding),
"-y",
str(tmp_path),
]
@@ -2148,6 +2147,12 @@ class ImageRoutesMixin:
except (NameError, OSError):
pass
if keep_existing and thumbnail_path.exists():
self.logger.warning(
f"No frame decoded for {video_path.name}; keeping its preview"
)
return False
# Last resort: create a placeholder thumbnail
try:
from PIL import ImageDraw, ImageFont
@@ -2182,18 +2187,22 @@ class ImageRoutesMixin:
except (OSError, AttributeError):
pass
_atomic_save(img, thumbnail_path, "JPEG", quality=85)
_atomic_save(img, thumbnail_path, "JPEG", **encoding)
self.logger.debug(
f"Generated placeholder video thumbnail: {thumbnail_path}"
)
return True
except PermissionError:
raise
except Exception as e:
self.logger.warning(
f"Failed to create placeholder thumbnail for {video_path}: {e}"
)
return False
except PermissionError:
raise
except Exception as e:
self.logger.error(
f"Video thumbnail generation failed for {video_path}: {e}"
+99 -18
View File
@@ -1,7 +1,8 @@
"""Session job routes (Activity): list, inspect, follow and cancel jobs.
The registry itself is ``py/api/jobs.py``. The scan job's runner is in
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``.
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``
and the thumbnail job's in ``py/api/thumbnail_jobs.py``.
Every response leaves out absolute server paths: job rows already carry
root-relative paths and public root labels.
"""
@@ -18,29 +19,74 @@ def _not_found():
return web.json_response({"success": False, "error": "Job not found"}, status=404)
def _bad_request(message):
return web.json_response({"success": False, "error": message}, status=400)
MAX_SCAN_ROOTS = 64
_SCOPE_ERROR = 'The scan scope must be "new" or "all"'
_ROOTS_ERROR = (
f"roots must be a non-empty list of up to {MAX_SCAN_ROOTS} source indexes, "
"with paths naming the public path of each"
)
SOURCES_CHANGED_ERROR = "The sources changed since they were loaded. Reload them."
async def _scan_scope(request):
"""The requested scan scope ("new" or "all"), or None when the body is invalid.
def _bad_request(message, code=None):
body = {"success": False, "error": message}
if code is not None:
body["code"] = code
return web.json_response(body, status=400)
An empty body (also a chunked one with no bytes), or one without
``scope``, asks for the default.
def _scan_roots(body):
"""``(roots, error)``: ``{index: public path}`` of the selected sources, or None.
Absent ``roots`` means every source. When present it is a non-empty list
of at most MAX_SCAN_ROOTS non-negative ints (never bools or floats), and
``paths`` the same-length list of the public path the caller read for
each (GET /settings), so a stale index never scans another folder.
"""
if "roots" not in body:
return None, None
raw = body["roots"]
paths = body.get("paths")
if not isinstance(raw, list) or not raw or len(raw) > MAX_SCAN_ROOTS:
return None, _ROOTS_ERROR
if not isinstance(paths, list) or len(paths) != len(raw):
return None, _ROOTS_ERROR
selected = {}
for value, path in zip(raw, paths):
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
return None, _ROOTS_ERROR
if not isinstance(path, str) or selected.get(value, path) != path:
return None, _ROOTS_ERROR
selected[value] = path
return dict(sorted(selected.items())), None
async def _scan_request(request):
"""``({files, roots}, None)`` for a scan start, or ``(None, message)``.
``files`` is the scope ("new" or "all"); ``roots`` the selected source
indexes or None for every source. An empty body (also a chunked one with
no bytes), or one without ``scope``, asks for the default scope.
"""
default = {"files": DEFAULT_SCAN_SCOPE, "roots": None}
if not request.can_read_body:
return DEFAULT_SCAN_SCOPE
return default, None
text = await request.text()
if not text.strip(): # e.g. a chunked POST that sent no bytes
return DEFAULT_SCAN_SCOPE
return default, None
try:
body = json.loads(text)
except ValueError:
return None
return None, _SCOPE_ERROR
if not isinstance(body, dict):
return None
scope = body.get("scope", DEFAULT_SCAN_SCOPE)
return scope if scope in SCAN_SCOPES else None
return None, _SCOPE_ERROR
files = body.get("scope", DEFAULT_SCAN_SCOPE)
if files not in SCAN_SCOPES:
return None, _SCOPE_ERROR
roots, error = _scan_roots(body)
if error:
return None, error
return {"files": files, "roots": roots}, None
def busy_response(busy: JobBusy):
@@ -64,6 +110,13 @@ def _summary_event(job):
class JobRoutesMixin:
"""Mixin providing the /prompt_manager/jobs endpoints."""
def _roots_still_listed(self, selected, output_dirs):
"""Whether each ``{index: public path}`` still names that source."""
return all(
index < len(output_dirs) and self._public_path(output_dirs[index]) == path
for index, path in selected.items()
)
def _register_job_routes(self, routes):
@routes.get("/prompt_manager/jobs")
async def list_jobs_route(request):
@@ -77,6 +130,10 @@ class JobRoutesMixin:
async def start_image_tag_job_route(request):
return await self.start_image_tag_job(request)
@routes.post("/prompt_manager/jobs/thumbnails")
async def start_thumbnail_job_route(request):
return await self.start_thumbnail_job(request)
@routes.get("/prompt_manager/jobs/{job_id}")
async def job_detail_route(request):
return await self.job_detail(request)
@@ -132,13 +189,37 @@ class JobRoutesMixin:
"""Start the metadata scan as a job (202), or 409 naming the busy job.
The body's ``scope`` is "new" (the default: new and unresolved files)
or "all"; anything else is a 400.
or "all". ``roots``, when given, selects sources by their index in
the effective sources (``_get_all_output_dirs``), and ``paths`` names
the public path read for each. An index past them, or one whose
source now has another public path (the list changed since it was
read), is a 409 ``sources-changed`` and starts nothing. The job walks
the sources resolved here, so a later source change cannot move it.
"""
files = await _scan_scope(request)
if files is None:
return _bad_request('The scan scope must be "new" or "all"')
scan, error = await _scan_request(request)
if error:
return _bad_request(error)
output_dirs = None
roots = None
if scan["roots"] is not None:
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
listed = await self._run_in_executor(
self._roots_still_listed, scan["roots"], output_dirs
)
if not listed:
return web.json_response(
{
"success": False,
"code": "sources-changed",
"error": SOURCES_CHANGED_ERROR,
},
status=409,
)
roots = list(scan["roots"])
try:
job = self._start_scan_job(files)
job = self._start_scan_job(
scan["files"], roots=roots, output_dirs=output_dirs
)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
+49 -10
View File
@@ -72,6 +72,9 @@ SCAN_GROUPS = {
IMAGE_TAG_JOB_KIND = "image-tags"
IMAGE_TAG_GROUP_NAMES = ("tagged", "unchanged", "skipped", "errors")
# Thumbnail generation (#249 System S5): previews written, already fresh, failed.
THUMBNAIL_GROUP_NAMES = ("generated", "skipped", "errors")
ADMISSIONS = ("read", "guarded", "records")
# What each admission class is refused beside (besides its own kind).
_REFUSED_BESIDE = {
@@ -247,6 +250,32 @@ class Job:
)
self.log_line(log_line)
def record_counted(self, group, count, rows=()):
"""Count ``count`` outcomes in ``group`` at once, listing ``rows``.
For outcomes that need no log line each (a thumbnail job's already
fresh previews): the counter is exact, ``rows`` (``record``'s row
shape) are listed up to MAX_OUTCOME_ROWS, and nothing is logged.
"""
if group not in self.counters:
raise ValueError(f"Unknown outcome group {group!r} for {self.kind}")
self.counters[group] += max(0, int(count))
listed = self.outcomes[group]
for row in rows:
if len(listed) >= MAX_OUTCOME_ROWS:
break
index = row.get("root_index")
if isinstance(index, bool) or not isinstance(index, int) or index < 0:
index = None
listed.append(
{
"path": _relative_path(row.get("path")),
"root": str(row.get("root") or ""),
"root_index": index,
"detail": str(row.get("detail") or ""),
}
)
def finish(self, state="finished"):
"""End the job in ``state`` and release every subscriber.
@@ -263,18 +292,24 @@ class Job:
for queue in list(self._subscribers):
queue.put_nowait(None)
async def events(self):
async def events(self, lead=None):
"""Yield the latest event, then each later one, until the job finishes.
``lead``, when given, is yielded first, after the subscription is in
place, so nothing published while the consumer sends it is missed.
Consume it under ``contextlib.aclosing`` so a consumer that stops early
(a closed socket) drops its queue at once instead of at garbage collection.
"""
queue = asyncio.Queue()
self._subscribers.add(queue)
# As of the subscription: a later event or the end reaches the queue.
snapshot, ended = self.last_event, self.finished
try:
if self.last_event is not None:
yield self.last_event
if self.finished:
if lead is not None:
yield lead
if snapshot is not None:
yield snapshot
if ended:
return
while True:
payload = await queue.get()
@@ -425,20 +460,24 @@ class JobRegistry:
self._jobs[job.id] = job
return job
def blocker(self, kind, *, admission=None, record_changing=None):
def blocker(self, kind, *, admission=None, record_changing=None, compatible=()):
"""The running job a new ``kind`` job would be refused for, or None.
The same kind refuses first; then the oldest running job whose class
the new job's admission class may not run beside.
the new job's admission class may not run beside. Running jobs of a
kind in ``compatible`` never refuse it: the caller knows that work
cannot conflict with its own (the monitor's Needs review recording
beside thumbnail work, which writes previews and no records).
"""
same_kind = self.active(kind)
if same_kind is not None:
return same_kind
refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)]
running = [job for job in self._jobs.values() if job.running]
# A snapshot: the monitor's worker thread asks too (Needs review).
running = [job for job in tuple(self._jobs.values()) if job.running]
running.sort(key=lambda job: job._started_seq)
for job in running:
if job.admission in refused:
if job.admission in refused and job.kind not in compatible:
return job
return None
@@ -466,8 +505,8 @@ class JobRegistry:
return self._jobs.get(job_id)
def active(self, kind):
"""The running job of ``kind``, or None."""
for job in self._jobs.values():
"""The running job of ``kind``, or None (safe off the event loop)."""
for job in tuple(self._jobs.values()):
if job.kind == kind and job.running:
return job
return None
+706
View File
@@ -0,0 +1,706 @@
"""Needs review routes (#249 E3): the pending queue, item detail, decisions.
``GET /prompt_manager/review/items`` lists pending items oldest first behind
an HMAC keyset cursor bound to the storage epoch only (a restore makes an
old cursor stale; a decision or a new item does not, because the
``(created_at, id)`` keyset stays ordered when items leave or are appended);
``total`` is the exact pending count. ``GET .../items/{id}`` adds the
candidates and the dependency fingerprint. ``POST .../items/{id}/decision``
records one explicit decision: Apply links the chosen sampler's pair only
when the capture is the item's bytes, the current file still hashes to them,
the evidence they yield is unchanged and the ingester's compatible-only rules
allow it; Leave unresolved links nothing.
Targets are formatted from the recorded spelling against the current roots
(no file access beyond the roots' real paths); a file under no current root
is ``unavailable``. Responses carry no server path, no locator and no
exception text.
"""
import base64
import binascii
import hashlib
import hmac
import json
import os
import re
import urllib.parse
from aiohttp import web
from .annotations import _HEX64, _TOKEN, _UUID, _capture_error, _error
from .images import media_thumbnail_url
from .inspection_service import VIDEO_EXTENSIONS
from .prompts import bad_request
try:
from ...database.library import display_title
from ...database.library_commands import LibraryCommandError
from ...database.review_decisions import ReviewCommandError
from ...utils.media_capture import CaptureError
from ...utils.review_capture import captured_review, review_evidence
except ImportError:
from database.library import display_title
from database.library_commands import LibraryCommandError
from database.review_decisions import ReviewCommandError
from utils.media_capture import CaptureError
from utils.review_capture import captured_review, review_evidence
ITEMS_URL = "/prompt_manager/review/items"
DEFAULT_LIMIT = 40
MAX_LIMIT = 40
MAX_CURSOR = 4096
MAX_BODY_BYTES = 64 * 1024
# The serialized detail stays under this; past it the detail is ``limited``.
MAX_DETAIL_BYTES = 8 * 1024 * 1024
UNRESOLVED = "unresolved"
_DECISION_FIELDS = frozenset(
{
"expected_epoch",
"expected_revision",
"capture_token",
"sha256",
"dependency_fingerprint",
"choice",
}
)
_CURSOR_SECRET = os.urandom(32)
# The real path of a spelling; module-level so a test can fake a junction.
_realpath = os.path.realpath
_LIMIT = re.compile(r"[0-9]{1,3}")
_NO_STORE = {"Cache-Control": "no-store"}
_STATUS = {
"review-missing": 404,
"review-conflict": 409,
"association-conflict": 409,
"file-changed": 409,
"storage-changed": 409,
"source-changed": 409,
"capture-expired": 409,
}
_MESSAGES = {
"review-missing": "This review item no longer exists. Reload the queue.",
"review-conflict": (
"This item was decided or changed since it was opened. Reload the queue."
),
"association-conflict": (
"This image is already linked to another prompt or negative, so this "
"sampler path cannot be attached safely. The existing links were "
"preserved."
),
"dependencies-changed": (
"The image's saved links changed since this item was opened. Reopen it "
"to see the current association."
),
"file-changed": (
"The file changed since it was reviewed. A metadata scan adds its new "
"bytes as a new item."
),
"storage-changed": "Prompt storage was restored; reload the review queue.",
"source-changed": (
"The source changed. Refresh the image collection before reviewing this "
"file."
),
"capture-expired": "The preview expired. Open the item again.",
"other-file": "The preview belongs to another file. Open the item again.",
"evidence-changed": (
"This file's metadata no longer gives the evidence that was reviewed. "
"Reload the queue."
),
}
_DEFAULT_MESSAGE = "The decision could not be saved. Reload the queue."
def _coded(code: str, message_key: str | None = None, **extra) -> web.Response:
message = _MESSAGES.get(message_key or code, _DEFAULT_MESSAGE)
return _error(_STATUS.get(code, 409), code, message, **extra)
def _failed() -> web.Response:
return _error(500, "review-failed", "The review queue could not be read.")
def format_review_target(spelled_path, locator, real_roots, *, pathmod=os.path):
"""``(root_index, relative)`` of the first root that lists the file, or None.
``spelled_path`` is the file's recorded spelling, ``locator`` its folded
identity and ``real_roots`` ``(index, realpath)`` pairs. The relative
path keeps the recorded spelling (a Windows locator is case-folded, a
listing target is not) and must fold back to the locator.
"""
sep = pathmod.sep
for index, real_root in real_roots:
head = spelled_path[: len(real_root)]
if pathmod.normcase(head) != pathmod.normcase(real_root):
continue
rest = spelled_path[len(real_root) :]
if not real_root.endswith(sep):
if not rest.startswith(sep):
continue
rest = rest[1:]
parts = rest.split(sep)
if any(
part in ("", ".", "..") or "/" in part or "\\" in part or ":" in part
for part in parts
):
continue
if pathmod.normcase(pathmod.join(real_root, *parts)) != locator:
continue
return index, "/".join(parts)
return None
class _StaleCursor(ValueError):
"""A well-formed cursor this process cannot verify (e.g. after a restart)."""
def _encode_cursor(epoch: str, key: tuple) -> str:
payload = json.dumps(
{"v": 2, "e": epoch, "k": list(key)},
separators=(",", ":"),
ensure_ascii=True,
).encode()
signature = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=")
def _decode_cursor(token: str) -> tuple:
"""``(epoch, (created_at, id))`` of a cursor.
``ValueError`` when malformed; :class:`_StaleCursor` when well-formed but
signed by another secret (the per-process secret changes on every start,
so an open page's cursor is stale after a restart, not forged).
"""
try:
if len(token) > MAX_CURSOR:
raise ValueError()
raw = base64.b64decode(
token + "=" * (-len(token) % 4), altchars=b"-_", validate=True
)
signature, payload = raw[:32], raw[32:]
if len(signature) != 32 or not payload:
raise ValueError()
expected = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
if not hmac.compare_digest(signature, expected):
raise _StaleCursor("This page of the queue is out of date")
data = json.loads(payload)
key = data["k"]
if (
data["v"] != 2
or not isinstance(data["e"], str)
or not isinstance(key, list)
or len(key) != 2
or not all(isinstance(part, str) for part in key)
):
raise ValueError()
return data["e"], tuple(key)
except _StaleCursor:
raise
except (ValueError, KeyError, TypeError, UnicodeError, binascii.Error):
raise ValueError("Invalid review cursor; reload the queue") from None
def _limit(value):
if value is None:
return DEFAULT_LIMIT
if _LIMIT.fullmatch(value) is None or not 1 <= int(value) <= MAX_LIMIT:
return None
return int(value)
def _side_text(side) -> str | None:
if isinstance(side, dict) and isinstance(side.get("text"), str):
return side["text"]
return None
def _workable(candidate: dict) -> bool:
return (
candidate.get("state") == "resolved"
and _side_text(candidate.get("positive")) is not None
and _side_text(candidate.get("negative")) is not None
)
def _side_label(titles: dict, side, empty: str) -> str:
"""The Library title of the side's saved prompt, else its first line."""
text = _side_text(side)
if text is None:
return empty
stored = side.get("storage_text")
stored = stored if isinstance(stored, str) else text
return display_title(titles.get(stored, ()), text) or empty
class ReviewRoutesMixin:
"""The Needs review queue over E1's schema and E2's recorded items."""
def _register_review_routes(self, routes: web.RouteTableDef) -> None:
routes.get(ITEMS_URL)(self.list_review_items)
routes.get(ITEMS_URL + "/{item_id}")(self.get_review_item)
routes.post(ITEMS_URL + "/{item_id}/decision")(self.decide_review_item)
# -- shared ----------------------------------------------------------------
def _review_roots(self):
"""``(roots, real_roots)``: the current gallery roots and their realpaths.
Blocking (it lists the roots and resolves them): call it on a worker.
"""
roots = list(self._get_all_output_dirs())
return roots, [(i, _realpath(str(root))) for i, root in enumerate(roots)]
def _review_view(self, row: dict, roots, real_roots) -> dict:
"""The public ReviewItem of one stored row (no path, no locator)."""
found = format_review_target(row["spelled_path"], row["locator"], real_roots)
target = url = thumbnail_url = None
if found is not None:
index, relative = found
target = {
"root_index": index,
"root_scope": self._gallery_root_scope(roots[index], index),
"path": relative,
}
# The Images listing's media URL for the same file (formatting only).
url = (
"/prompt_manager/images/serve/"
f"{urllib.parse.quote(relative, safe='/')}?root={index}"
f"&root_scope={target['root_scope']}"
)
# The Images listing's thumbnail, when one exists (one stat).
thumbnail_url = media_thumbnail_url(
roots[index], relative, index, root_scope=target["root_scope"]
)
filename = os.path.basename(row["spelled_path"])
return {
"id": row["id"],
"state": row["state"],
"revision": row["revision"],
"created_at": row["created_at"],
"filename": filename,
"target": target,
"unavailable": target is None,
"candidate_count": row["candidate_count"],
"limited": row["limited"],
"url": url,
"thumbnail_url": thumbnail_url,
"is_video": os.path.splitext(filename)[1].lower() in VIDEO_EXTENSIONS,
}
@staticmethod
def _review_image_path(row: dict, roots, target) -> str:
"""The path the scan and the monitor link this file under."""
if target is None:
return row["spelled_path"]
root = roots[target["root_index"]]
return os.path.join(str(root), *target["path"].split("/"))
# -- queue -----------------------------------------------------------------
async def list_review_items(self, request: web.Request) -> web.Response:
"""GET ``?limit=1..40&cursor=``: one page of pending items."""
limit = _limit(request.query.get("limit"))
if limit is None:
return bad_request(f"limit must be an integer from 1 to {MAX_LIMIT}")
bound = None
cursor = request.query.get("cursor")
if cursor:
try:
bound = _decode_cursor(cursor)
except _StaleCursor:
return await self._stale_cursor()
except ValueError as exc:
return bad_request(str(exc))
try:
page, views = await self._run_in_executor(
self._review_page, limit, bound[1] if bound else None
)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
if bound is not None and bound[0] != page["epoch"]:
return _coded("storage-changed", library_epoch=page["epoch"])
items = page["items"]
next_cursor = None
if page["more"] and items:
last = items[-1]
next_cursor = _encode_cursor(
page["epoch"], (last["created_at"], last["id"])
)
return web.json_response(
{
"success": True,
"library_epoch": page["epoch"],
# F3: the queue version the change stream's ``ready`` names.
"review_revision": page["revision"],
"total": page["total"],
"items": views,
"next_cursor": next_cursor,
},
headers=_NO_STORE,
)
async def _stale_cursor(self) -> web.Response:
"""409 ``review-conflict`` with the current total: reload the queue."""
try:
page = await self._run_in_executor(self.db.review_queue_page, limit=1)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
return _coded(
"review-conflict", total=page["total"], library_epoch=page["epoch"]
)
def _review_page(self, limit: int, after):
"""Worker thread: one queue page and the public views of its items."""
page = self.db.review_queue_page(limit=limit, after=after)
roots, real_roots = self._review_roots()
return page, [
self._review_view(row, roots, real_roots) for row in page["items"]
]
# -- detail ----------------------------------------------------------------
async def get_review_item(self, request: web.Request) -> web.Response:
"""GET one item with its candidates and dependency fingerprint."""
try:
read = await self._run_in_executor(
self._review_detail, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if read is None:
return _coded("review-missing")
item, view, fingerprint, candidates = read
detail = {
"success": True,
"library_epoch": item["epoch"],
**view,
"candidates": candidates,
"dependency_fingerprint": fingerprint,
"sha256": item["sha256"],
"total": item["total"],
}
if len(json.dumps(detail).encode("utf-8")) > MAX_DETAIL_BYTES:
detail = {**detail, "candidates": [], "limited": True}
return web.json_response(detail, headers=_NO_STORE)
def _review_detail(self, item_id: str):
"""Worker thread: ``(item, view, fingerprint, candidates)`` or None."""
item = self.db.review_item(item_id)
if item is None:
return None
roots, real_roots = self._review_roots()
view = self._review_view(item, roots, real_roots)
image_path = self._review_image_path(item, roots, view["target"])
aliases, _known = physical_spellings(self.db, image_path, item["locator"])
fingerprint = self.db.review_dependency_fingerprint(image_path, aliases)
return item, view, fingerprint, self._review_candidates(item)
def _review_candidates(self, item: dict) -> list:
"""The detail's candidates: role texts, a label and selectability."""
if item["limited"]:
return []
candidates = [
c for c in item["evidence"].get("candidates", []) if isinstance(c, dict)
]
texts = set()
for candidate in candidates:
for role in ("positive", "negative"):
side = candidate.get(role)
if isinstance(side, dict):
for key in ("text", "storage_text"):
if isinstance(side.get(key), str):
texts.add(side[key])
titles = self.db.stored_prompt_titles(texts)
rows = []
for candidate in candidates:
positive = _side_text(candidate.get("positive"))
negative = _side_text(candidate.get("negative"))
label = (
_side_label(titles, candidate.get("positive"), "No positive")
+ " / "
+ _side_label(titles, candidate.get("negative"), "No negative")
)
rows.append(
{
"hash": candidate.get("hash"),
"sampler_id": candidate.get("sampler_id"),
"state": "resolved" if _workable(candidate) else "unresolved",
"reason": candidate.get("reason") or "",
"label": label,
"positive": {"text": positive} if positive is not None else None,
"negative": {"text": negative} if negative is not None else None,
}
)
return rows
# -- decision --------------------------------------------------------------
async def decide_review_item(self, request: web.Request) -> web.Response:
"""POST one explicit decision on one pending item."""
data, error = await self._read_json_body(request, max_bytes=MAX_BODY_BYTES)
if error is not None:
return error
guards, error = _decision_guards(data)
if error is not None:
return error
try:
item = await self._run_in_executor(
self.db.review_item, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if item is None:
return _coded("review-missing")
choice = guards["choice"]
candidate = None
if choice != UNRESOLVED:
candidate = next(
(
c
for c in item["evidence"].get("candidates", [])
if isinstance(c, dict) and c.get("hash") == choice
),
None,
)
if candidate is None or item["limited"]:
return bad_request("choice is not one of this item's candidates")
if not _workable(candidate):
return bad_request("This sampler path cannot be applied")
if guards["capture_token"] is None:
return bad_request("Apply needs a capture of the file")
verified = None
if guards["capture_token"] is not None:
verified, error = await self._verified_review_capture(
item, guards, need_metadata=candidate is not None
)
if error is not None:
return error
elif guards["sha256"] != item["sha256"]:
return bad_request("sha256 does not match the item")
return await self._record_decision(item, guards, candidate, verified)
async def _verified_review_capture(self, item, guards, *, need_metadata):
"""``((capture, facts, image_path, metadata), None)`` or ``(None, error)``.
The capture must be the item's file and bytes, under the expected
epoch and a current root; the current file must re-hash to the same
bytes; for Apply, the capture's metadata must still yield exactly the
reviewed evidence.
"""
store = self._captures
capture = store.get(guards["capture_token"]) if store is not None else None
if capture is None:
return None, _coded("capture-expired")
if capture.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if guards["sha256"] != capture.sha256:
return None, bad_request("sha256 does not match the preview")
if capture.sha256 != item["sha256"]:
return None, _coded("file-changed")
if guards["expected_epoch"] != capture.epoch:
current = await self._run_in_executor(self.db.get_execution_epoch)
return None, _coded("storage-changed", library_epoch=current)
try:
root, error = await self._run_in_executor(
self._scoped_root, capture.root_index, capture.scope
)
if error is not None:
return None, error
read = await self._run_in_executor(
_verify_and_read, store, capture, item, root, need_metadata
)
except CaptureError as exc:
return None, _capture_error(exc)
if read is None:
return None, _coded("source-changed")
relative, facts, captured = read
if facts.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if facts.sha256 != item["sha256"]:
return None, _coded("file-changed")
metadata = None
if need_metadata:
if captured is None or not _candidate_still_offered(
captured[1], guards["choice"]
):
return None, _coded("review-conflict", "evidence-changed")
metadata = captured[0]
image_path = os.path.join(str(root), *relative.split("/"))
return (capture, facts, image_path, metadata), None
def _decide_sync(self, item, guards, candidate, verified):
"""Worker thread: resolve the file's spellings, decide, build the view.
Returns ``(result, view)``; a refusal raises with ``exc.view`` set.
"""
_capture, facts, image_path, metadata = verified or (None, None, None, None)
aliases, known = (), None
if candidate is not None:
aliases, known = physical_spellings(self.db, image_path, item["locator"])
try:
result = self.db.decide_review_item(
item_id=item["id"],
expected_epoch=guards["expected_epoch"],
expected_revision=guards["expected_revision"],
# The bytes the re-hash proved, or the body's for Leave unresolved.
sha256=facts.sha256 if facts is not None else guards["sha256"],
outcome="selected" if candidate is not None else UNRESOLVED,
candidate=candidate,
image_path=image_path,
metadata=metadata,
expected_fingerprint=guards["dependency_fingerprint"],
aliases=aliases,
known_rows=known,
)
except ReviewCommandError as exc:
if exc.item is not None:
roots, real_roots = self._review_roots()
exc.view = self._review_view(exc.item, roots, real_roots)
raise
roots, real_roots = self._review_roots()
return result, self._review_view(result["item"], roots, real_roots)
async def _record_decision(self, item, guards, candidate, verified):
capture = verified[0] if verified else None
try:
result, view = await self._run_in_executor(
self._decide_sync, item, guards, candidate, verified
)
except ReviewCommandError as exc:
return self._decision_refused(exc)
except LibraryCommandError as exc:
return _coded(exc.code, library_epoch=exc.epoch)
except Exception:
self.logger.exception("Review decision failed")
return _error(500, "decision-failed", "The decision could not be saved.")
if capture is not None:
# The decision used the capture; the page returns to Images.
self._captures.release(capture.token)
body = {
"success": True,
"library_epoch": guards["expected_epoch"],
"item": view,
"total": result["total"],
}
if result["pair"] is not None:
body["pair"] = result["pair"]
return web.json_response(body, headers=_NO_STORE)
def _decision_refused(self, exc: ReviewCommandError) -> web.Response:
extra = {"library_epoch": exc.epoch}
if exc.total is not None:
extra["total"] = exc.total
if exc.view is not None:
extra["item"] = exc.view
message = None
if exc.fingerprint is not None:
extra["dependency_fingerprint"] = exc.fingerprint
if exc.code == "association-conflict" and exc.changed:
message = "dependencies-changed"
return _coded(exc.code, message, **extra)
def _decision_guards(data: dict):
"""The validated decision fields, or ``(None, error)``."""
if set(data) != _DECISION_FIELDS:
return None, bad_request(
"Request must contain exactly " + ", ".join(sorted(_DECISION_FIELDS))
)
epoch, revision = data["expected_epoch"], data["expected_revision"]
if not isinstance(epoch, str) or _UUID.fullmatch(epoch) is None:
return None, bad_request("expected_epoch must be a canonical UUID")
if type(revision) is not int or revision < 0:
return None, bad_request("expected_revision must be a non-negative integer")
token = data["capture_token"]
if token is not None and (
not isinstance(token, str) or _TOKEN.fullmatch(token) is None
):
return None, bad_request("Invalid capture token")
for name in ("sha256", "dependency_fingerprint"):
value = data[name]
if not isinstance(value, str) or _HEX64.fullmatch(value) is None:
return None, bad_request(
f"{name} must be 64 lowercase hexadecimal characters"
)
choice = data["choice"]
if not isinstance(choice, str) or (
choice != UNRESOLVED and _HEX64.fullmatch(choice) is None
):
return None, bad_request("choice must be a candidate hash or unresolved")
return {**data}, None
def _verify_and_read(store, capture, item, root, need_metadata: bool):
"""Worker thread: ``(relative, facts, captured)``, or None off the root.
Formats the item's target under the capture's root (resolving the root's
real path), re-hashes the current file, then reads the capture.
"""
found = format_review_target(
item["spelled_path"],
item["locator"],
[(capture.root_index, _realpath(str(root)))],
)
if found is None:
return None
relative = found[1]
facts = store.verify(root, relative)
captured = captured_review(store, capture) if need_metadata else None
return relative, facts, captured
def _same_file(spelled: str, image_path: str, locator: str) -> bool:
"""True when ``spelled`` reaches the physical file at ``locator``."""
try:
if os.path.normcase(_realpath(spelled)) == locator:
return True
return os.path.samefile(spelled, image_path)
except (OSError, ValueError):
return False
def physical_spellings(db, image_path: str, locator: str):
"""``(aliases, known_rows)`` for the legacy rows of one physical file.
The roots are real paths, while the monitor links a file under the
directory it watches, so a symlinked or junctioned root, a ``subst`` or a
mapped drive spells the same file otherwise. Every row whose spelling
ends with the file's name (the link summaries' fallback lookup) is a
candidate; ``aliases`` are the candidates that resolve to ``locator``
(``normcase(realpath)``, as the capture forms it) or are the same file,
and ``known_rows`` every candidate id, so a decision can refuse a
same-name row it did not classify. Blocking: call it on a worker.
"""
rows = db.review_same_name_rows(os.path.basename(image_path))
aliases = tuple(
spelled
for _row_id, spelled in rows
if isinstance(spelled, str)
and spelled != image_path
and _same_file(spelled, image_path, locator)
)
return tuple(dict.fromkeys(aliases)), frozenset(row_id for row_id, _ in rows)
def _candidate_still_offered(resolution: dict, choice: str) -> bool:
"""True when the captured bytes still offer the chosen candidate.
Only what the decision depends on is re-checked: the bytes are still
``ambiguous`` and the chosen candidate (its hash covers its sampler,
sides and provenance) is still among their candidates. A resolver change
that rewords the top-level reason or reorders candidates leaves a
pending item applicable.
"""
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
return False
evidence, _count, limited = review_evidence(resolution)
if limited:
return False
return any(c.get("hash") == choice for c in evidence.get("candidates", ()))
+171
View File
@@ -0,0 +1,171 @@
"""Needs review change stream (#249 PR F2).
``GET /prompt_manager/review/events`` is a server-sent event stream for an
open Needs review page: ``event: ready`` when it opens, then
``event: change`` after each committed queue change (an item the scan or the
monitor recorded, a decision from any tab, a spelling refresh) and after a
restore. The first notice is sent at once; notices in the next
``COALESCE_SECONDS`` collapse into one trailing ``change`` (M9: a busy scan
records file after file, and each event costs every open page a re-read).
The page re-reads its queue on each ``change``; nothing polls. ``change``
carries no data (``{}``). ``ready`` carries the queue's version
``{library_epoch, review_revision}``, read after the stream subscribed, the
same pair a list read answers: a page whose first read already answered that
version skips the re-read (F3, one read per visit), and any later change is a
``change`` event. Nothing else about the queue is pushed.
A ``: keepalive`` comment every ``HEARTBEAT_SECONDS`` lets the server notice
a closed tab (its write fails) and drop the subscription; clients ignore
comments. Shutdown (``close_review_events``, an ``on_shutdown`` hook) ends
every stream, and a stream asked for afterwards is a 503.
"""
import asyncio
import json
from aiohttp import web
try:
from ...database.execution_history import subscribe_history_restore
from ...database.review_changes import subscribe_review_changes
except ImportError:
from database.execution_history import subscribe_history_restore
from database.review_changes import subscribe_review_changes
EVENTS_URL = "/prompt_manager/review/events"
HEARTBEAT_SECONDS = 25
# After a ``change``, the next one waits this long, collapsing the notices in between.
COALESCE_SECONDS = 0.4
_CHANGE = b"event: change\ndata: {}\n\n"
_KEEPALIVE = b": keepalive\n\n"
def _ready(version: dict) -> bytes:
data = json.dumps(version, separators=(",", ":"), sort_keys=True)
return f"event: ready\ndata: {data}\n\n".encode("utf-8")
class _Stream:
"""One open stream: a change flag set from any thread, and a close flag."""
def __init__(self, loop: asyncio.AbstractEventLoop) -> None:
self._loop = loop
self.changed = asyncio.Event()
self.closed = asyncio.Event()
self.ready = False
def wake(self) -> None:
"""Thread-safe: mark a change (the loop may already be gone)."""
try:
self._loop.call_soon_threadsafe(self.changed.set)
except RuntimeError: # the loop closed
pass
async def next(self, timeout: float) -> str:
"""``change``, ``closed`` or ``idle`` (nothing within ``timeout``)."""
waits = {
asyncio.ensure_future(self.changed.wait()): "change",
asyncio.ensure_future(self.closed.wait()): "closed",
}
try:
done, _ = await asyncio.wait(
waits, timeout=timeout, return_when=asyncio.FIRST_COMPLETED
)
finally:
for waiter in waits:
waiter.cancel()
if self.closed.is_set():
return "closed"
if done:
self.changed.clear()
return "change"
return "idle"
async def closed_within(self, timeout: float) -> bool:
"""Wait up to ``timeout``; true when the stream closed meanwhile."""
try:
await asyncio.wait_for(self.closed.wait(), timeout)
except asyncio.TimeoutError:
return False
return True
class ReviewEventsMixin:
"""The Needs review change stream."""
def _register_review_event_routes(self, routes: web.RouteTableDef) -> None:
routes.get(EVENTS_URL)(self.review_events)
def _review_streams(self) -> set:
streams = getattr(self, "_open_review_streams", None)
if streams is None:
streams = self._open_review_streams = set()
return streams
def review_stream_count(self) -> int:
"""How many change streams have sent ``ready`` and are open (tests and
diagnostics): a queue change after this count moved is a ``change``
event on them, never part of the version ``ready`` named."""
return sum(1 for stream in self._review_streams() if stream.ready)
async def close_review_events(self, app) -> None:
"""End every open stream; later requests get 503 (shutdown hook)."""
self._review_events_closed = True
for stream in list(self._review_streams()):
stream.closed.set()
async def _queue_version(self) -> dict:
"""The queue version for ``ready``, read after subscribing (so a change
after this read is an event); ``{}`` when it cannot be read, and the
page then re-reads."""
try:
return await self._run_in_executor(self.db.review_queue_version)
except Exception:
self.logger.warning("Needs review version read failed", exc_info=True)
return {}
async def review_events(self, request: web.Request) -> web.StreamResponse:
if getattr(self, "_review_events_closed", False):
return web.json_response(
{"success": False, "error": "The server is shutting down."},
status=503,
)
stream = _Stream(asyncio.get_running_loop())
db_path = self.db.model.db_path
unsubscribers = (
subscribe_review_changes(db_path, stream.wake),
subscribe_history_restore(db_path, stream.wake),
)
streams = self._review_streams()
streams.add(stream)
response = web.StreamResponse(
status=200,
headers={
"Content-Type": "text/event-stream",
"Cache-Control": "no-store",
"X-Accel-Buffering": "no",
},
)
try:
version = await self._queue_version()
await response.prepare(request)
await response.write(_ready(version))
stream.ready = True
while True:
outcome = await stream.next(HEARTBEAT_SECONDS)
if outcome == "closed":
break
if outcome == "idle":
await response.write(_KEEPALIVE)
continue
await response.write(_CHANGE)
if await stream.closed_within(COALESCE_SECONDS):
break
await response.write_eof()
except (ConnectionError, RuntimeError) as exc: # the tab went away
self.logger.debug(f"Needs review stream closed ({exc})")
finally:
streams.discard(stream)
for unsubscribe in unsubscribers:
unsubscribe()
return response
+251
View File
@@ -0,0 +1,251 @@
"""Needs review recording for the output scan and the live monitor (#249).
The rule (maintainer test of #284/#288): a file is queued only when its
final outcome is ambiguous and unlinked. The strict resolver must find it
``ambiguous``, the scan's legacy fallback (the browser parser,
``_graph_prompt_text``) must not link it, and no saved row may link it
already. A file the fallback links, an exact pair the ingester stored and an
already linked file are never queued. Neither is an ambiguous file with
nothing to choose: fewer than two of its candidates carry a prompt
(``utils.review_capture.reviewable``); the scan reports it unresolved ("could
not be traced").
The queue heals itself: a pending item whose file is now linked (by the
fallback, a later resolution or the monitor) is removed by the next scan
that lists the file or by the monitor's next pass over it, and so is a
pending item without a choice whose file is still ambiguous without one
(``settle_untraced_review_items``). Only an undecided item goes
(``pending``, revision 0, no decision row); a decision is never deleted or
changed (``settle_linked_review_items``).
The scan records through ``_record_scanned_review`` (best effort: a failure
is logged and counted); the monitor calls ``record_review_file`` from its
worker thread after its own link. Both go through
``utils.review_capture.record_ambiguous_file`` on the API's capture store.
Nothing here links a file.
Recording changes records, so it follows the registry's ``records``
admission (#289). The scan records under its own ``records`` job. The
monitor's pass lasts one file and is not a job: it is refused while a
``records`` or ``guarded`` job runs (``deferred``), and the next scan reads
the file again. Thumbnail work (the job and the gallery's generate and clear
holds, kind ``thumbnails``) is ``guarded`` but only writes and removes preview
files under ``thumbnails/`` and never touches the database, so it does not
defer the recording (#249 S5 M7): a long rebuild no longer holds Needs review
back for its whole run.
"""
import os
from pathlib import Path
from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots
from .jobs import THUMBNAILS_JOB_KIND
# The kind the monitor's recording checks admission under (never registered).
REVIEW_RECORDING_KIND = "review_recording"
# Guarded kinds that change no records, so the recording may run beside them.
PREVIEW_ONLY_KINDS = (THUMBNAILS_JOB_KIND,)
# record_ambiguous_file outcomes that leave the file in the pending queue.
QUEUED_OUTCOMES = ("recorded", "limited", "duplicate")
NEW_OUTCOMES = ("recorded", "limited")
try:
from ...utils.review_capture import record_ambiguous_file
except ImportError:
from utils.review_capture import record_ambiguous_file
# The scan log's reason for items that leave the queue with nothing to choose.
NOTHING_TO_CHOOSE = "nothing to choose"
class ReviewRecordingMixin:
"""Record the scan's and the monitor's ambiguous files for Needs review."""
def _record_scanned_review(self, media_file, roots, review=None):
"""Record one ambiguous, unlinked scanned file (logged and counted)."""
try:
outcome = self._record_review_sync(media_file, roots, review)
except Exception:
self.logger.warning(
f"{media_file.name} could not be added to Needs review",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return
if review is None:
return
if outcome in NEW_OUTCOMES:
review["added"] += 1
if outcome in QUEUED_OUTCOMES:
review["queued"] += 1
def _record_review_sync(self, media_file, roots, review=None):
"""``record_ambiguous_file`` for a file under one of ``roots`` (blocking).
An ``untraced`` file (its captured bytes offer nothing to choose) also
loses its pending items that offer nothing either.
"""
located = self._review_location(media_file, roots)
if located is None:
return "skipped"
root, index, relative, _spelled = located
outcome = record_ambiguous_file(
self.db, self._capture_store(), Path(root), index, relative
)
if outcome == "untraced":
self._settle_untraced_reviews([media_file], roots, review)
return outcome
@staticmethod
def _review_location(media_file, roots, real_roots=None):
"""``(root, root_index, relative, spelled)`` of a listed file, or None.
``spelled`` is the capture's spelling of the file (the real root joined
with the relative path); ``os.path.normcase`` of it is the locator
review items are stored under. ``real_roots`` caches each root's
``realpath`` across a batch.
"""
relative, _label, index = _scan_location(media_file, roots)
root = next((r for r, _l, i in roots if i == index), None)
if index is None or root is None:
return None
if real_roots is None:
real_roots = {}
real = real_roots.get(index)
if real is None:
real = real_roots[index] = os.path.realpath(os.fspath(root))
spelled = os.path.join(real, *relative.split("/"))
return root, index, relative, spelled
def _review_file_linked(self, media_file, roots):
"""True when a saved row links the scanned file (any spelling)."""
located = self._review_location(media_file, roots)
spellings = (str(media_file),) + ((located[3],) if located else ())
return self.db.review_file_linked(spellings)
def _settle_scanned_reviews(self, media_files, roots, review=None):
"""Remove the pending items of ``media_files`` that are now linked.
Blocking; best effort like recording (a failure is logged and
counted in ``review["errors"]``). Returns how many items left.
"""
real_roots = {}
files = []
for media_file in media_files:
located = self._review_location(media_file, roots, real_roots)
if located is not None:
spelled = located[3]
files.append((os.path.normcase(spelled), (str(media_file), spelled)))
try:
settled = self.db.settle_linked_review_items(files) if files else 0
except Exception:
self.logger.warning(
"Needs review items of linked files could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["settled"] += settled
return settled
def _settle_untraced_reviews(self, media_files, roots, review=None):
"""Remove the pending items without a choice of untraced ``media_files``.
``media_files`` are files found ambiguous with fewer than two
candidates that carry a prompt. Blocking; best effort like the linked
settle (a failure is logged and counted in ``review["errors"]``).
Returns how many items left.
"""
locators = []
for media_file in media_files:
located = self._review_location(media_file, roots)
if located is not None:
locators.append(os.path.normcase(located[3]))
try:
removed = self.db.settle_untraced_review_items(locators) if locators else 0
except Exception:
self.logger.warning(
"Needs review items without a choice could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["untraced"] += removed
return removed
def _fallback_links(self, metadata):
"""True when the scan's legacy fallback would link these records."""
return bool(metadata.get("prompt")) and bool(self._graph_prompt_text(metadata))
def record_review_file(self, image_path):
"""Apply the Needs review rule to one monitor output (its worker thread).
Maps ``image_path`` to the gallery root that lists it; a file under no
current root is ``skipped``. A linked file is never queued: its
pending items are removed and the answer is ``linked``. An unlinked
file the scan's fallback would link is ``skipped`` (the next scan
links it); any other file is recorded when it is ambiguous with a
choice, and is ``untraced`` (its pending items without a choice
removed) when it is ambiguous without one. While a
job runs that a ``records`` job may not run beside, other than
thumbnail work (``PREVIEW_ONLY_KINDS``), nothing is written and the
answer is ``deferred``. Errors propagate: the monitor logs
them and keeps its link.
"""
busy = self._jobs.blocker(
REVIEW_RECORDING_KIND, admission="records", compatible=PREVIEW_ONLY_KINDS
)
if busy is not None:
self.logger.info(
f"Needs review was not updated for "
f"{os.path.basename(str(image_path))} while {busy.title} runs; "
"the next scan reads it again"
)
return "deferred"
full = os.path.abspath(str(image_path))
located = Path(os.path.realpath(os.path.dirname(full))) / os.path.basename(full)
roots = _scan_roots(self._get_all_output_dirs())
if self._review_location(located, roots) is None:
return "skipped"
if self._review_file_linked(located, roots):
self._settle_scanned_reviews([located], roots)
return "linked"
if self._fallback_links(self._extract_comfyui_metadata(str(located))):
return "skipped"
return self._record_review_sync(located, roots)
@staticmethod
def _log_review_items(job, review):
"""The scan log's Needs review summary lines (none when nothing moved)."""
prefix = SCAN_LOG_PREFIXES["review"]
if review["added"]:
job.log_line(
f"{prefix} {_plural(review['added'], 'image')} added to Needs review"
)
already = review["queued"] - review["added"]
if already:
job.log_line(
f"{prefix} {_plural(already, 'image')} already in Needs review"
)
if review["settled"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['settled'], 'image')} left Needs review · now linked"
)
if review["untraced"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['untraced'], 'image')} left Needs review · "
f"{NOTHING_TO_CHOOSE}"
)
if review["errors"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['errors']} "
f"{_plural(review['errors'], 'image')} could not be added to "
"Needs review"
)
+246 -31
View File
@@ -4,8 +4,15 @@ A save validates the payload, commits it to config.json through the config
store and publishes it at runtime inside that commit; a save that cannot be
written changes nothing at runtime and says so. ``_config_commit_failure``
is shared with the LoRA integration's save.
GET also reports the image sources and the monitor (``sources.py``). A save
that changes the sources runs as a ``source_change`` registry hold, so it is
refused while a record-changing job (a scan) runs. ``monitoring_enabled``
takes effect at the next restart. ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing.
"""
import contextlib
import os
from aiohttp import web
@@ -15,7 +22,10 @@ from ..config_store import (
RevisionConflict,
get_config_store,
)
from . import sources
from .admin import _current_image_monitor
from .job_routes import busy_response
from .jobs import JobBusy
try:
from ...utils.validators import validate_result_timeout, validate_worker_threads
@@ -23,13 +33,75 @@ except ImportError:
from utils.validators import validate_result_timeout, validate_worker_threads
# Settings owned by POST /settings: config attribute -> (section, key).
# Settings owned by POST /settings: config attribute -> its keys in config.json.
_SETTINGS_FILE_KEYS = {
"RESULT_TIMEOUT": ("web_ui", "result_timeout"),
"WEBUI_DISPLAY_MODE": ("web_ui", "webui_display_mode"),
"INFINITE_SCROLL": ("web_ui", "infinite_scroll"),
"WORKER_THREADS": ("performance", "worker_threads"),
"MONITORING_ENABLED": ("gallery", "monitoring", "enabled"),
}
# Attributes that live on GalleryConfig; the others on PromptManagerConfig.
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"})
SOURCE_CHANGE_JOB_KIND = "source_change"
SOURCE_CHANGE_JOB_TITLE = "Source change"
def _source_identity(path):
"""The case-folded spelling two roots are the same directory by (Windows)."""
return os.path.normcase(path)
def _stat_identity(path):
"""``(st_dev, st_ino)`` of an existing directory, or None.
Catches case variants on a case-insensitive filesystem whose spelling
normcase keeps (macOS).
"""
try:
stat = os.stat(path)
except (OSError, ValueError):
return None
return (stat.st_dev, stat.st_ino) if stat.st_ino else None
# Refusals that a reconnected drive or folder can lift.
_RECONNECTABLE = frozenset({"not-found", "not-directory"})
def _root_key(root):
"""The directory a configured root names (resolved), else its spelling."""
from ..config import GalleryConfig
try:
return GalleryConfig.resolve_gallery_root(root)
except (OSError, TypeError, ValueError, AttributeError):
return root
def _request_error(message):
return {"code": "invalid-request", "error": message}
def _invalid_source(index, entry, code, reason):
"""The 400 body for the source at ``index`` that cannot be used.
Names it by its public spelling (computed as text: a refused path is
never resolved), so the page can mark that row and say why.
"""
source = sources.public_spelling(entry)
advice = (
"Remove it or reconnect it, then save."
if code in _RECONNECTABLE
else "Remove it, then save."
)
return {
"code": "invalid-source",
"source": source,
"index": index,
"reason": reason,
"error": f"{source} can't be used: {reason}. {advice}",
}
def _section(config, name):
@@ -47,10 +119,8 @@ def _settings_mutation(updates, new_roots):
def mutate(config):
for name, value in updates.items():
section, key = _SETTINGS_FILE_KEYS[name]
merged = _section(config, section)
merged[key] = value
config[section] = merged
*sections, key = _SETTINGS_FILE_KEYS[name]
_set_nested(config, sections, key, value)
if new_roots is not None:
gallery = _section(config, "gallery")
monitoring = _section(gallery, "monitoring")
@@ -62,6 +132,16 @@ def _settings_mutation(updates, new_roots):
return mutate
def _set_nested(config, sections, key, value):
"""``config[s1][s2]...[key] = value``, each section a fresh dict."""
if not sections:
config[key] = value
return
merged = _section(config, sections[0])
_set_nested(merged, sections[1:], key, value)
config[sections[0]] = merged
async def _read_json(request):
"""Parse a JSON object body through the package's capped reader."""
from . import _read_json_body
@@ -81,6 +161,10 @@ class SettingsRoutesMixin:
async def save_settings_route(request):
return await self.save_settings(request)
@routes.post("/prompt_manager/settings/validate-source")
async def validate_source_route(request):
return await self.validate_source(request)
async def get_settings(self, request):
"""Get current settings."""
try:
@@ -90,11 +174,15 @@ class SettingsRoutesMixin:
self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES
]
revision = await self._run_in_executor(self._config_revision)
source_status, monitor = await self._run_in_executor(self._source_status)
return web.json_response(
{
"success": True,
**revision,
"sources": source_status,
"monitor": monitor,
"settings": {
"monitoring_enabled": GalleryConfig.MONITORING_ENABLED,
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
@@ -115,6 +203,61 @@ class SettingsRoutesMixin:
status=500,
)
def _source_status(self):
"""``(sources, monitor)`` of GET /settings (blocking)."""
from ..config import GalleryConfig
output_dirs, auto_detected = self._output_dirs_with_origin()
return sources.describe_sources(
configured=list(GalleryConfig.MONITORING_DIRECTORIES),
output_dirs=output_dirs,
auto_detected=auto_detected,
monitor=_current_image_monitor(),
enabled=GalleryConfig.MONITORING_ENABLED,
)
async def validate_source(self, request):
"""Check one source path ``{path}``; writes nothing.
200 ``{success, valid, label, path, code, reason}``: a valid source
answers its label and public path, an invalid one its code and reason
only. A body without a string ``path`` is a 400 ``invalid-request``.
"""
data, error_response = await _read_json(request)
if error_response is not None:
too_large = error_response.status == 413
return web.json_response(
{
"success": False,
"code": "too-large" if too_large else "invalid-request",
"error": (
"Request body too large"
if too_large
else "Request body must be a JSON object"
),
},
status=error_response.status,
)
path = data.get("path")
if not isinstance(path, str):
return web.json_response(
{
"success": False,
"code": "invalid-request",
"error": "path must be a string",
},
status=400,
)
try:
result = await self._run_in_executor(sources.validate_source, path)
except Exception:
self.logger.exception("Source validation failed")
return web.json_response(
{"success": False, "code": None, "error": "Failed to check the source"},
status=500,
)
return web.json_response({"success": True, **result})
def _monitored_directories(self):
"""Directories the running image monitor watches, else the configured roots."""
from ..config import GalleryConfig
@@ -151,9 +294,15 @@ class SettingsRoutesMixin:
{"success": False, "error": str(ve)}, status=400
)
new_roots, error = self._parse_gallery_roots(data)
# Off the loop: an accepted root is resolved (filesystem access).
new_roots, error = await self._run_in_executor(
self._parse_gallery_roots, data
)
if error:
return web.json_response({"success": False, "error": error}, status=400)
return web.json_response({"success": False, **error}, status=400)
roots_change = new_roots is not None and await self._run_in_executor(
self._roots_change, new_roots
)
published = {}
@@ -162,15 +311,30 @@ class SettingsRoutesMixin:
# they were written, so the runtime ends on the last one.
published.update(self._publish_settings(updates, new_roots))
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
return self._config_commit_failure(exc)
async with contextlib.AsyncExitStack() as stack:
hold = None
if roots_change:
try:
hold = await stack.enter_async_context(
self._jobs.hold(
SOURCE_CHANGE_JOB_KIND,
SOURCE_CHANGE_JOB_TITLE,
admission="records",
)
)
except JobBusy as busy:
return busy_response(busy)
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
if hold is not None:
hold.finish("failed")
return self._config_commit_failure(exc)
return web.json_response(
{
@@ -196,10 +360,11 @@ class SettingsRoutesMixin:
as ``runtime_applied: false`` with the fields that need a restart;
nothing is retried or rolled back. Never raises.
"""
from ..config import PromptManagerConfig
from ..config import GalleryConfig, PromptManagerConfig
for name, value in updates.items():
setattr(PromptManagerConfig, name, value)
owner = GalleryConfig if name in _GALLERY_SETTINGS else PromptManagerConfig
setattr(owner, name, value)
restart_fields = []
runtime_applied = True
if new_roots is not None:
@@ -212,11 +377,40 @@ class SettingsRoutesMixin:
)
runtime_applied = False
restart_fields.append("gallery_root_paths")
if "MONITORING_ENABLED" in updates:
# The monitor starts with the server: a restart applies the setting
# unless the observer already matches it.
if updates["MONITORING_ENABLED"] != self._observer_alive():
restart_fields.append("monitoring_enabled")
return {
"runtime_applied": runtime_applied,
"restart_required_fields": restart_fields,
}
@staticmethod
def _roots_change(new_roots):
"""Whether a save changes the sources (blocking: it resolves them).
Compared as directories, not spellings: the legacy page resends the
roots resolved, while config.json may hold them as hand-written
(relative, another case), and an unchanged resend is no change.
"""
from ..config import GalleryConfig
if new_roots is None:
return False
return [_root_key(root) for root in new_roots] != [
_root_key(root) for root in GalleryConfig.MONITORING_DIRECTORIES
]
@staticmethod
def _observer_alive():
"""Whether the image monitor's observer is running now. Never raises."""
try:
return sources.monitor_state(_current_image_monitor())[1]
except Exception:
return False
def _config_commit_failure(self, exc):
"""The response for a config commit that wrote nothing."""
not_saved = {"success": False, "persisted": False, "runtime_applied": False}
@@ -283,6 +477,10 @@ class SettingsRoutesMixin:
if not isinstance(data["infinite_scroll"], bool):
raise ValueError("infinite_scroll must be true or false")
updates["INFINITE_SCROLL"] = data["infinite_scroll"]
if "monitoring_enabled" in data:
if not isinstance(data["monitoring_enabled"], bool):
raise ValueError("monitoring_enabled must be true or false")
updates["MONITORING_ENABLED"] = data["monitoring_enabled"]
if "worker_threads" in data:
updates["WORKER_THREADS"] = validate_worker_threads(
data["worker_threads"], PromptManagerConfig.max_worker_threads()
@@ -295,15 +493,21 @@ class SettingsRoutesMixin:
Accepts ``gallery_root_paths`` (list, preferred) or the legacy single
``gallery_root_path`` string. Blank entries are dropped.
Each entry passes the same checks as validate-source
(``sources.source_problem``): the form, then containment as text,
before anything is resolved. Blocking (an accepted root is resolved).
Returns:
(roots, error): ``roots`` is the validated list, or ``None`` when
the payload carries no gallery root key; ``error`` is a message
when validation failed (and ``roots`` is then ``None``).
the payload carries no gallery root key; ``error`` is the coded
body of the 400 when validation failed (``roots`` is then None).
An unusable source is ``invalid-source`` naming its public path
and its position in the list.
"""
if "gallery_root_paths" in data:
raw = data["gallery_root_paths"]
if not isinstance(raw, list):
return None, "gallery_root_paths must be a list"
return None, _request_error("gallery_root_paths must be a list")
elif "gallery_root_path" in data:
raw = [data["gallery_root_path"]]
else:
@@ -312,27 +516,38 @@ class SettingsRoutesMixin:
from ..config import GalleryConfig
roots = []
for entry in raw:
seen = set()
for index, entry in enumerate(raw):
if not isinstance(entry, str):
return None, "Gallery root paths must be strings"
return None, _request_error("Gallery root paths must be strings")
entry = entry.strip()
if not entry:
continue
ok, reason = GalleryConfig.validate_gallery_root(entry)
if not ok:
return (
None,
f"Invalid gallery root '{os.path.basename(entry)}': {reason}",
)
roots.append(GalleryConfig.resolve_gallery_root(entry))
code, reason = sources.source_problem(entry)
if code is not None:
return None, _invalid_source(index, entry, code, reason)
root = GalleryConfig.resolve_gallery_root(entry)
keys = {("path", _source_identity(root)), ("file", _stat_identity(root))}
keys.discard(("file", None))
if keys & seen:
continue # another spelling of a root already listed
seen |= keys
roots.append(root)
return roots, None
def _apply_gallery_roots(self, roots):
"""Install validated gallery roots; returns True when they changed."""
"""Install validated gallery roots; returns True when they changed.
Another spelling of the same directories installs the resolved
spelling and is no change (nothing to restart for).
"""
from ..config import GalleryConfig
if roots == list(GalleryConfig.MONITORING_DIRECTORIES):
return False
if not self._roots_change(roots):
GalleryConfig.MONITORING_DIRECTORIES = roots
return False
GalleryConfig.MONITORING_DIRECTORIES = roots
self._cached_output_dir = None
self.reset_gallery_roots_cache()
+280
View File
@@ -0,0 +1,280 @@
"""Image sources as System › Sources & scans shows them (#249 S4).
- ``source_form_error`` is the lexical check a submitted source path passes
before any filesystem access: no control characters, no ``..`` part, and
only the native path forms (on Windows a drive or a UNC share, never the
``\\\\?\\`` and ``\\\\.\\`` namespaces, a drive-relative path, an alternate
data stream, a device name or a part Windows would silently trim; on POSIX
no Windows drive or backslash form).
- ``source_problem`` is the one refusal every caller answers (validate-source
and the settings save): the form check, then
``GalleryConfig.gallery_root_problem``, which decides containment as text
before it touches the filesystem.
- ``describe_sources`` (blocking: it resolves paths) reports the configured
sources with their validity, the effective sources with their listing
index and watch state, and the image monitor's state. Every path in it is
a public path (``_public_path``), never an absolute one.
A source's watch state is ``watching`` when the monitor's observer is alive,
scheduled that directory and monitoring is on; ``stopping`` when it still
watches it but monitoring was switched off (it stops at the next restart);
``manual`` when monitoring is off or this process has no monitor; else
``restart_pending`` (a source added since the monitor started, monitoring
switched on, or an observer that stopped).
"""
import logging
import ntpath
import os
import re
logger = logging.getLogger("prompt_manager.api")
MAX_SOURCE_PATH = 4096 # characters
FORM_REASONS = {
"empty": "Gallery root must be a non-empty path",
"invalid-form": "Gallery root is not a folder path this server can use",
"traversal": "Gallery root must not contain '..'",
}
_DRIVE = re.compile(r"[A-Za-z]:")
_POSIX_FOREIGN = re.compile(r"^[A-Za-z]:")
# Reserved device names: Windows opens the device for any folder of that name.
_DEVICE_NAME = re.compile(r"^(con|prn|aux|nul|com[0-9¹²³]|lpt[0-9¹²³])(\..*)?$", re.I)
# folder_paths getters and the label of the directory each one reports.
_COMFYUI_LABELS = (
("get_output_directory", "ComfyUI output"),
("get_input_directory", "ComfyUI input"),
("get_temp_directory", "ComfyUI temp"),
("get_user_directory", "ComfyUI user"),
)
WATCH_STATES = ("watching", "stopping", "manual", "restart_pending")
def source_form_error(text, *, windows=None):
"""The form code that refuses ``text`` as a source path, or None.
``windows`` selects the path rules (default: this OS). No filesystem
access happens here.
"""
if windows is None:
windows = os.name == "nt"
if not isinstance(text, str) or len(text) > MAX_SOURCE_PATH:
return "invalid-form"
if any(ord(char) < 32 or ord(char) == 127 for char in text):
return "invalid-form"
text = text.strip()
if not text:
return "empty"
return _windows_form_error(text) if windows else _posix_form_error(text)
def _posix_form_error(text):
if "\\" in text or _POSIX_FOREIGN.match(text):
return "invalid-form" # a Windows path names nothing on this server
return "traversal" if ".." in text.split("/") else None
def _windows_form_error(text):
portable = text.replace("/", "\\")
if portable.startswith(("\\\\?\\", "\\\\.\\")):
return "invalid-form" # verbatim and device namespaces
if portable.startswith("\\\\"):
drive, rest = ntpath.splitdrive(portable)
share = drive[2:].split("\\")
if len(share) != 2 or any(part in ("", ".", "..") for part in share):
return "invalid-form"
else:
drive, rest = ntpath.splitdrive(portable)
if drive and (not _DRIVE.fullmatch(drive) or not rest.startswith("\\")):
return "invalid-form" # "C:folder" is relative to that drive's cwd
if ":" in rest:
return "invalid-form" # an alternate data stream
parts = rest.split("\\")
if ".." in parts:
return "traversal"
for part in parts:
if part in ("", "."):
continue
if part[-1] in ". " or _DEVICE_NAME.match(part):
return "invalid-form"
return None
def source_problem(text):
"""``(code, reason)`` that refuses ``text`` as a source, else ``(None, "")``.
The form check runs first and touches nothing; then containment is
decided as text before any filesystem access (see
``GalleryConfig.gallery_root_problem``). Every caller answers alike.
"""
from ..config import GalleryConfig
code = source_form_error(text)
if code is not None:
return code, FORM_REASONS[code]
return GalleryConfig.gallery_root_problem(text)
def public_spelling(text):
"""How a refused source is named in a response, computed as text only.
Relative to the path anchor it lies under (its public path), else its
last part: the caller sent the path, so naming it reveals nothing, and a
refused path is never resolved to name it.
"""
from ..config import GalleryConfig, _lexical_path, _lexically_inside
text = str(text).strip()
try:
lexical = os.path.normpath(text)
key = _lexical_path(text) # the same length as ``lexical``
if not os.path.isabs(text):
return lexical.replace("\\", "/")
for anchor in GalleryConfig.path_anchors():
if _lexically_inside(key, [anchor]):
rest = lexical[len(anchor.rstrip(os.sep)) :].lstrip(os.sep)
return rest.replace(os.sep, "/") or "."
return os.path.basename(lexical) or lexical
except (TypeError, ValueError):
return ""
def identity(path):
"""The comparison key of a directory: resolved and case-folded (Windows)."""
return os.path.normcase(os.path.realpath(str(path)))
def _comfyui_labels():
"""``{identity: label}`` of the directories ComfyUI reports."""
try:
import folder_paths
except ImportError:
return {}
labels = {}
for getter_name, label in _COMFYUI_LABELS:
getter = getattr(folder_paths, getter_name, None)
if getter is None:
continue
try:
directory = getter()
except Exception:
continue
if isinstance(directory, str) and directory:
labels.setdefault(identity(directory), label)
return labels
def source_label(path, labels=None):
""" "ComfyUI output" for ComfyUI's own directories, else the folder name."""
if labels is None:
labels = _comfyui_labels()
label = labels.get(identity(path))
if label:
return label
return os.path.basename(os.path.normpath(str(path))) or str(path)
def _configured(entry, labels, public_path):
from ..config import GalleryConfig
code, reason = GalleryConfig.gallery_root_problem(entry)
try:
resolved = GalleryConfig.resolve_gallery_root(entry)
except (OSError, TypeError, ValueError):
resolved = str(entry)
return {
"label": source_label(resolved, labels),
"path": public_path(resolved),
"valid": code is None,
"code": code,
"reason": reason or None,
}
def monitor_state(monitor):
"""``(present, observer_alive, scheduled directories)`` of the image monitor."""
if monitor is None:
return False, False, []
status = monitor.get_status()
scheduled = [str(d) for d in status.get("monitored_directories") or []]
return True, bool(status.get("observer_alive")), scheduled
def watch_state(path, *, present, alive, scheduled_keys, enabled):
"""One of ``WATCH_STATES`` (see the module docstring)."""
if alive and identity(path) in scheduled_keys:
return "watching" if enabled else "stopping"
if not present or not enabled:
return "manual"
return "restart_pending"
def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled):
"""``(sources, monitor)`` for GET /settings (blocking; public paths only).
``configured`` is the configured roots, ``output_dirs`` and
``auto_detected`` what ``_output_dirs_with_origin`` returned, ``monitor``
the running ImageMonitor or None, ``enabled`` the monitoring setting.
"""
from . import _public_path
labels = _comfyui_labels()
present, alive, scheduled = monitor_state(monitor)
scheduled_keys = {identity(path) for path in scheduled}
effective = [
{
"label": source_label(path, labels),
"path": _public_path(path),
"root_index": index,
"watch": watch_state(
path,
present=present,
alive=alive,
scheduled_keys=scheduled_keys,
enabled=enabled,
),
}
for index, path in enumerate(output_dirs)
]
return (
{
"configured": [
_configured(entry, labels, _public_path) for entry in configured
],
"effective": effective,
"auto_detected": bool(auto_detected),
},
{
"present": present,
"observer_alive": alive,
"scheduled": [_public_path(path) for path in scheduled],
},
)
def validate_source(text):
"""``{valid, label, path, code, reason}`` for one submitted path (blocking).
Writes nothing. An invalid path answers its code and reason with no
label or path, so a response never echoes what was typed.
"""
from ..config import GalleryConfig
from . import _public_path
code, reason = source_problem(text)
if code is not None:
return {
"valid": False,
"label": None,
"path": None,
"code": code,
"reason": reason,
}
resolved = GalleryConfig.resolve_gallery_root(text)
return {
"valid": True,
"label": source_label(resolved),
"path": _public_path(resolved),
"code": None,
"reason": None,
}
+20 -6
View File
@@ -9,9 +9,10 @@
kept.
- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate.
Every value is real or null: there is no backup status until backups report
one, no health result until a check ran, and no Needs-review count until
the review queue exists. The missing-thumbnail count is not part of it:
the page reads ``/images/thumbnails/status``, which walks every source.
one and no health result until a check ran; ``needs_review.pending`` is the
Needs review queue's pending total. The missing-thumbnail count is not part
of it: the page reads ``/images/thumbnails/status``, which walks every
source.
- ``GET /prompt_manager/system/statistics`` lists runs and ratings per
category, with the legacy ``/stats`` totals and the Overview's count of
rated images (images whose linked prompt is rated).
@@ -40,6 +41,19 @@ _HEALTH_STATUS_LINES = {
_StoredReport = namedtuple("_StoredReport", "generation report")
async def _gather_settled(*reads):
"""Await every read, then raise the first failure.
Unlike a plain gather, a failure never answers while the other executor
reads still run against the database.
"""
results = await asyncio.gather(*reads, return_exceptions=True)
for result in results:
if isinstance(result, BaseException):
raise result
return results
def _failure(message, status=500, code=None):
return web.json_response(
{"success": False, "code": code, "error": message}, status=status
@@ -154,7 +168,7 @@ class SystemRoutesMixin:
from ..config import GalleryConfig
try:
(output_dirs, auto_detected), counts = await asyncio.gather(
(output_dirs, auto_detected), counts = await _gather_settled(
self._run_in_executor(self._output_dirs_with_origin),
self._run_in_executor(self.db.library_counts),
)
@@ -174,7 +188,7 @@ class SystemRoutesMixin:
},
"health": health_summary(self._current_health_report()),
"backup": None, # no verified backup status exists yet (S8)
"needs_review": {"pending": None}, # no review queue yet (Q6)
"needs_review": {"pending": counts["review_pending"]},
"sources": {
"configured": len(GalleryConfig.MONITORING_DIRECTORIES),
"effective": len(output_dirs),
@@ -188,7 +202,7 @@ class SystemRoutesMixin:
async def get_system_statistics(self, request):
try:
by_category, totals, counts = await asyncio.gather(
by_category, totals, counts = await _gather_settled(
self._run_in_executor(self.db.usage_by_category),
self._run_in_executor(self.db.get_statistics),
self._run_in_executor(self.db.library_counts),
+494
View File
@@ -0,0 +1,494 @@
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
- ``GET /prompt_manager/images/thumbnails/plan?mode=&size=&quality=`` counts
what a run would do over every gallery root and writes nothing:
``{success, targets, fresh, remaining, roots, mode, size, quality}``.
- ``POST /prompt_manager/jobs/thumbnails {mode, size, quality}`` starts the
``thumbnails`` job (202 ``{job}``). Its admission class is ``guarded``: it
never runs beside a record-changing job, and one thumbnail job runs at a
time. It stops at a file boundary when cancelled, keeping the previews it
wrote. Its outcome groups are ``generated``, ``skipped`` (already fresh)
and ``errors``.
``mode`` is ``missing`` (previews that are absent or older than their file)
or ``rebuild`` (every preview again, fresh ones too). ``size`` is the
preview's longest edge, 128 to 2048 pixels. ``quality`` picks the encoder
settings (``THUMBNAIL_ENCODINGS``). Every preview is written next to the old
one and renamed over it, so nothing is cleared first and a failure keeps the
old preview; originals are only read.
The legacy gallery stream ``POST /images/generate-thumbnails/progress``
starts this job (its quality preset still sizes the previews) or attaches to
the running one. Its first frame, ``job {id, started}``, names that job and
whether this stream started it (the gallery cancels by that id, and only a
job it started); then come the job's legacy ``status``/``start``/
``progress``/``file_error``/``complete`` frames. A client that goes away no
longer stops the work: the job outlives the request, as a scan does.
"""
import asyncio
import functools
import json
import time
from dataclasses import dataclass
from aiohttp import web
from .admin import _plural, _scan_location, _scan_roots, _scan_status_line
from .images import (
THUMBNAIL_ENCODINGS,
_BodyTooLarge,
_new_thumbnail_stats,
_read_json_body,
_record_thumbnail_result,
_thumbnail_batch_size,
_thumbnail_complete_payload,
_thumbnail_progress_payload,
_worker_threads,
)
from .job_routes import busy_response
from .jobs import THUMBNAIL_GROUP_NAMES, THUMBNAILS_JOB_KIND, JobBusy
THUMBNAIL_JOB_TITLE = "Thumbnail generation"
THUMBNAIL_MODES = ("missing", "rebuild")
THUMBNAIL_QUALITIES = tuple(THUMBNAIL_ENCODINGS)
MIN_THUMBNAIL_EDGE = 128
MAX_THUMBNAIL_EDGE = 2048
# The concept's "Preview size limit" default.
DEFAULT_THUMBNAIL_EDGE = 512
DEFAULT_THUMBNAIL_OPTIONS = {
"mode": "missing",
"size": DEFAULT_THUMBNAIL_EDGE,
"quality": "medium",
}
_OPTIONS_ERROR = (
'mode must be "missing" or "rebuild", quality "low", "medium" or "high", '
f"and size a whole number of pixels from {MIN_THUMBNAIL_EDGE} "
f"to {MAX_THUMBNAIL_EDGE}"
)
CANCELLED_ERROR = "Thumbnail generation was cancelled"
CANCELLED_BEFORE_WRITING = "Cancelled before any preview was written"
NOTHING_TO_PREVIEW = "No images or videos to preview"
_MODE_LOG = {"missing": "missing previews", "rebuild": "every preview"}
# Outcome rows: what happened to each file's preview.
_DETAILS = {
"generated": "Preview written",
"skipped": "Preview already up to date",
}
_LOG_PREFIXES = {"generated": "+", "errors": "!"}
@dataclass(frozen=True)
class ThumbnailOptions:
"""One run's choices: the mode, the preview's longest edge and quality."""
mode: str
edge: int
quality: str
@property
def size(self):
return (self.edge, self.edge)
@property
def encoding(self):
return THUMBNAIL_ENCODINGS[self.quality]
@property
def rebuild(self):
return self.mode == "rebuild"
def scope(self):
return {"mode": self.mode, "size": self.edge, "quality": self.quality}
def _edge(value, *, from_query):
"""The preview edge as an int, or None when it is not one in range."""
if from_query and isinstance(value, str):
if not value.isascii() or not value.isdigit() or len(value) > 5:
return None
value = int(value)
if isinstance(value, bool) or not isinstance(value, int):
return None
return value if MIN_THUMBNAIL_EDGE <= value <= MAX_THUMBNAIL_EDGE else None
def thumbnail_options(raw, *, from_query=False):
"""``(ThumbnailOptions, None)`` from a body or query, or ``(None, message)``.
Absent fields take ``DEFAULT_THUMBNAIL_OPTIONS``. Query values are
strings, so a query ``size`` may be decimal digits; a JSON ``size`` must
be an int.
"""
if not isinstance(raw, dict) and not hasattr(raw, "get"):
return None, _OPTIONS_ERROR
mode = raw.get("mode", DEFAULT_THUMBNAIL_OPTIONS["mode"])
quality = raw.get("quality", DEFAULT_THUMBNAIL_OPTIONS["quality"])
edge = _edge(raw.get("size", DEFAULT_THUMBNAIL_EDGE), from_query=from_query)
if mode not in THUMBNAIL_MODES or quality not in THUMBNAIL_QUALITIES:
return None, _OPTIONS_ERROR
if edge is None:
return None, _OPTIONS_ERROR
return ThumbnailOptions(mode, edge, quality), None
def _invalid(message):
return web.json_response(
{"success": False, "code": "invalid-request", "error": message}, status=400
)
def _frame(event, data):
"""A legacy stream frame as the job publishes it."""
return {"event": event, "data": data}
def _finished_line(plan, stats):
"""A finished job's status line: what it wrote, or that it had nothing to do."""
if not plan["targets"]:
if plan["skipped"]:
return f"No previews to generate · {plan['skipped']:,} already up to date"
return NOTHING_TO_PREVIEW
parts = [f"{_plural(stats['generated'], 'preview')} written"]
if stats["errors"]:
parts.append(_plural(len(stats["errors"]), "error"))
if plan["remaining"]:
parts.append(f"{plan['remaining']:,} more on the next run")
return " · ".join(parts)
class ThumbnailJobsMixin:
"""The thumbnail plan, the thumbnail job and the legacy stream on it."""
async def get_thumbnail_plan(self, request):
"""What a thumbnail run would do (read-only; capped like a run)."""
options, error = thumbnail_options(request.query, from_query=True)
if error:
return _invalid(error)
try:
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
plan = await self._run_in_executor(
self._plan_gallery_thumbnails, output_dirs, rebuild=options.rebuild
)
except Exception:
self.logger.exception("Thumbnail plan failed")
return web.json_response(
{
"success": False,
"code": None,
"error": "The thumbnail plan could not be read",
},
status=500,
)
return web.json_response(
{
"success": True,
"targets": len(plan["targets"]),
"fresh": plan["fresh"],
"remaining": plan["remaining"],
"roots": [self._public_path(root) for root in output_dirs],
**options.scope(),
},
headers={"Cache-Control": "no-store"},
)
async def start_thumbnail_job(self, request):
"""Start the thumbnail job (202), or 409 naming the job in the way."""
body = {}
if request.can_read_body:
try:
body = await _read_json_body(request)
except _BodyTooLarge:
return web.json_response(
{"success": False, "code": "too-large", "error": "Body too large"},
status=413,
)
except (ValueError, UnicodeDecodeError):
return _invalid(_OPTIONS_ERROR)
options, error = thumbnail_options(body)
if error:
return _invalid(error)
try:
job = self._start_thumbnail_job(options)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
def _start_thumbnail_job(self, options):
"""Register the thumbnail job synchronously; raises JobBusy."""
return self._jobs.start(
THUMBNAILS_JOB_KIND,
THUMBNAIL_JOB_TITLE,
admission="guarded",
cancellable=True,
scope=options.scope(),
runner=functools.partial(self._run_thumbnail_job, options=options),
groups=THUMBNAIL_GROUP_NAMES,
)
# ── the job ───────────────────────────────────────────────────────
async def _run_thumbnail_job(self, job, options):
try:
state = await self._thumbnail_job_body(job, options)
except Exception:
self.logger.exception("Thumbnail generation failed")
job.update(status_line="Thumbnail generation stopped on an internal error.")
job.log_line("Thumbnails failed · internal error")
job.publish(
_frame(
"error",
{
"error": "An internal error occurred",
"message": "Thumbnail generation failed. Check server logs.",
},
)
)
state = "failed"
job.finish(state)
async def _thumbnail_job_body(self, job, options):
"""Plan, then generate in worker batches; returns the final state.
Every exit sets the job's final status line (I3): what was written,
that there was nothing to do, or what a cancel kept.
"""
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
if not output_dirs:
message = "ComfyUI output directory not found"
job.update(status_line=message)
job.log_line(f"Thumbnails failed · {message}")
job.publish(_frame("error", {"error": message}))
return "failed"
folders = _plural(len(output_dirs), "output folder")
job.scope["output_folders"] = len(output_dirs)
job.update(status_line=f"Looking for media files in {folders}")
job.log_line(
f"Thumbnails started · {folders} · {_MODE_LOG[options.mode]} · "
f"{options.edge} px · {options.quality}"
)
job.publish(
_frame(
"status",
{"phase": "scanning", "message": "Scanning for images and videos..."},
)
)
plan = await self._run_in_executor(
self._plan_gallery_thumbnails,
output_dirs,
rebuild=options.rebuild,
stop=lambda: job.cancelled,
)
if job.cancelled: # M2: stopped during the walk, before any start frame
return self._end_cancelled(job, 0, CANCELLED_BEFORE_WRITING)
roots = _scan_roots(output_dirs)
self._record_fresh(job, plan, roots)
stats, start = await self._generate_thumbnail_targets(job, options, plan, roots)
self.invalidate_gallery_cache()
generated = _plural(stats["generated"], "preview")
if job.cancelled:
return self._end_cancelled(
job, stats["generated"], f"Cancelled · {generated} written"
)
payload = _thumbnail_complete_payload(
len(plan["targets"]) + plan["skipped"] + plan["remaining"],
stats,
start,
remaining=plan["remaining"],
processed=len(plan["targets"]),
)
job.update(progress=100, status_line=_finished_line(plan, stats))
if plan["remaining"]:
job.log_line(
f"{plan['remaining']:,} more previews are left for the next run"
)
errors = _plural(len(stats["errors"]), "error")
job.log_line(f"Thumbnails finished · {generated} written · {errors}")
self.logger.info(payload["message"])
job.publish(_frame("complete", payload))
return "finished"
@staticmethod
def _end_cancelled(job, generated, status_line):
"""Close a cancelled job: its status line, log line and error frame."""
job.update(status_line=status_line)
job.log_line(f"Thumbnails cancelled · {_plural(generated, 'preview')} written")
job.publish(
_frame("error", {"error": CANCELLED_ERROR, "message": CANCELLED_ERROR})
)
return "cancelled"
def _record_fresh(self, job, plan, roots):
"""List the already fresh previews under Skipped (no log line each)."""
rows = []
for src in plan["fresh_files"]:
path, label, index = _scan_location(src, roots)
rows.append(
{
"path": path,
"root": label,
"root_index": index,
"detail": _DETAILS["skipped"],
}
)
job.record_counted("skipped", plan["skipped"], rows)
if plan["skipped"]:
job.log_line(f"= {plan['skipped']:,} previews already up to date")
async def _generate_thumbnail_targets(self, job, options, plan, roots):
"""Generate ``plan``'s targets in worker batches, publishing each file."""
targets = plan["targets"]
total = len(targets)
video_count = sum(1 for _, _, is_video in targets if is_video)
message = (
f"Found {total - video_count} images and {video_count} videos to process"
)
if plan["skipped"]:
message += f" ({plan['skipped']} already have thumbnails)"
job.update(total=total)
if total: # never "Processing file 0 of 0" (I3)
job.update(status_line=_scan_status_line(0, total, 0, 0))
job.publish(
_frame(
"start",
{
"total_images": total,
"skipped": plan["skipped"],
"remaining": plan["remaining"],
"phase": "processing",
"image_count": total - video_count,
"video_count": video_count,
"message": message,
},
)
)
stats = _new_thumbnail_stats()
stats["skipped"] = plan["skipped"]
start = time.monotonic()
batch_size = _thumbnail_batch_size(_worker_threads())
generate = {
"encoding": options.encoding,
"rebuild": options.rebuild,
"stop": lambda: job.cancelled,
}
index = 0
batches = 0
for batch_start in range(0, total, batch_size):
if job.cancelled:
break
batch = targets[batch_start : batch_start + batch_size]
results = await self._run_in_executor(
self._generate_batch, batch, options.size, **generate
)
for (src, _dst, _video), result in zip(batch, results):
if result["action"] == "cancelled":
continue
index += 1
_record_thumbnail_result(stats, result)
self._record_thumbnail_outcome(job, src, result, roots)
job.update(processed=index, progress=index * 100 // total)
if result["action"] == "error":
job.publish(
_frame(
"file_error",
{"file": result["file"], "error": result["error"]},
)
)
job.publish(
_frame(
"progress",
_thumbnail_progress_payload(index, total, stats, result, start),
)
)
batches += 1
job.update(
status_line=_scan_status_line(
index, total, batches, time.monotonic() - start
)
)
await asyncio.sleep(0) # let other requests run between batches
return stats, start
@staticmethod
def _record_thumbnail_outcome(job, src, result, roots):
path, label, index = _scan_location(src, roots)
where = f"{label}/{path}" if label else path
if result["action"] == "error":
group, detail = "errors", result.get("error") or "failed"
elif result["action"] == "skipped":
group, detail = "skipped", _DETAILS["skipped"]
else:
group, detail = "generated", _DETAILS["generated"]
row = {"path": path, "root": label, "root_index": index, "detail": detail}
if group == "skipped":
job.record_counted(group, 1, [row])
return
job.record(group, row, f"{_LOG_PREFIXES[group]} {where} · {detail}")
# ── the legacy gallery stream ─────────────────────────────────────
async def generate_thumbnails_with_progress(self, request):
"""The gallery's thumbnail stream (POST, server-sent events).
Starts the thumbnail job with the gallery's quality preset as the
preview size (missing previews, medium encoding, as before), or
attaches to the running thumbnail job, and streams its frames. A
legacy hold (the blocking generate or clear) or a record-changing
job refuses it with the stream's error frame. The job goes on when
the client goes away.
"""
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
response = web.StreamResponse(
status=200,
headers={
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
},
)
await response.prepare(request)
# M3: read the registry after the last await, so a job that started
# while the headers went out is attached to, not refused.
job, started, busy = self._legacy_stream_job(thumbnail_size)
if busy is not None:
message = f"{busy.title} already running"
await self._emit_sse(
response, "error", {"error": message, "message": message}
)
return response
# No await between the start and the subscription: the lead frame is
# sent after it, so the stream misses none of the runner's frames.
events = job.events(lead=_frame("job", {"id": job.id, "started": started}))
try:
async for payload in events:
frame = (
f"event: {payload['event']}\ndata: {json.dumps(payload['data'])}"
)
await response.write(f"{frame}\n\n".encode("utf-8"))
except Exception as exc: # a write failed: the client went away
self.logger.info(
f"Thumbnail progress client disconnected ({exc}); "
"the thumbnails continue in the background"
)
finally:
# Also when aiohttp cancels the handler on disconnect: the job is
# its own task and goes on.
await events.aclose()
return response
def _legacy_stream_job(self, thumbnail_size):
"""``(job, started, busy)`` for the legacy stream, synchronously.
Attaches to the running cancellable thumbnail job, or starts one with
the gallery's preset as the size (missing previews, medium encoding,
as the gallery always wrote), or names the job in the way.
"""
running = self._jobs.active(THUMBNAILS_JOB_KIND)
if running is not None and running.cancellable:
return running, False, None
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
if busy is not None:
return None, False, busy
# blocker() said free and nothing awaited since, so the start is admitted.
options = ThumbnailOptions("missing", thumbnail_size[0], "medium")
return self._start_thumbnail_job(options), True, None
+121 -25
View File
@@ -31,7 +31,6 @@ routes = server_instance.routes
extension_uri = None # Will be set in __init__.py
import os # noqa: E402
from pathlib import Path # noqa: E402
from typing import Any, Dict, List, Optional, Tuple # noqa: E402
# Import logging system
@@ -51,6 +50,14 @@ config_logger = get_logger("prompt_manager.config")
# may be used as gallery roots in addition to ComfyUI's own directories.
EXTRA_GALLERY_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
# The one answer for every path outside the allowed folders. The filesystem
# root and the home directory answer it too, so no caller learns the server's
# layout (home path, user name) from a refusal (#249 S4).
OUTSIDE_ALLOWED_REASON = (
"Gallery root must be inside a ComfyUI directory (output, input, temp, user) "
f"or one listed in {EXTRA_GALLERY_ROOTS_ENV}"
)
# Environment variable overriding where config.json is read from and saved to.
CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH"
@@ -89,6 +96,29 @@ def write_private_json(path: str, data: Any) -> None:
os.chmod(path, 0o600)
def _lexical_path(path: str) -> str:
"""``path`` normalised for comparison without touching the filesystem.
normpath collapses ``.`` and ``..`` as text and normcase folds case on
Windows; neither opens, stats nor resolves anything (no SMB request for a
UNC path). A symlink is only seen by :func:`_canonical_path` later.
"""
return os.path.normcase(os.path.normpath(path))
def _lexically_inside(path: str, parents: List[str]) -> bool:
"""Whether the normalised ``path`` is one of ``parents`` or below one.
Pure text on normalised spellings (:func:`_lexical_path` or
:func:`_canonical_path`): no filesystem access.
"""
sep = os.path.sep
return any(
path == parent or path.startswith(parent.rstrip(sep) + sep)
for parent in parents
)
def _is_filesystem_root(path: str) -> bool:
"""True for '/' on POSIX and drive roots such as 'C:\\' on Windows."""
return os.path.dirname(path) == path
@@ -268,7 +298,12 @@ class GalleryConfig:
"""
monitoring = new_config.get("monitoring", {})
if "enabled" in monitoring:
cls.MONITORING_ENABLED = monitoring["enabled"]
if isinstance(monitoring["enabled"], bool):
cls.MONITORING_ENABLED = monitoring["enabled"]
else: # a hand-edited "false" would otherwise switch it on
config_logger.warning(
"Ignoring gallery monitoring enabled: expected true or false"
)
if "directories" in monitoring:
directories = cls._validated_directories(monitoring["directories"])
if directories is not None:
@@ -337,31 +372,56 @@ class GalleryConfig:
Filesystem roots are never allowed as parents, even when listed in
the environment, because that would re-open the whole disk.
"""
parents = []
return [canonical for canonical, _spelling in cls._allowed_parent_pairs()]
@classmethod
def _allowed_parent_pairs(cls) -> List[Tuple[str, str]]:
"""``(canonical, lexical spelling)`` of each allowed parent.
Only trusted directories (ComfyUI's own and the environment's) are
resolved here; the spelling is how ComfyUI or the environment wrote
the directory, so a submitted path can be compared without resolving it.
"""
pairs = []
seen = set()
for candidate in _comfyui_directories() + _extra_gallery_roots():
try:
canonical = _canonical_path(candidate)
spelling = _lexical_path(os.path.abspath(candidate))
except (OSError, ValueError):
continue
if _is_filesystem_root(canonical) or not os.path.isdir(canonical):
continue
if canonical not in parents:
parents.append(canonical)
return parents
if canonical not in seen:
seen.add(canonical)
pairs.append((canonical, spelling))
return pairs
@classmethod
def path_anchors(cls) -> List[str]:
"""Canonical directories that relative gallery paths are resolved against.
The ComfyUI base directory comes first, followed by the parent of each
directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so an extra
root is addressed by its own name (``gallery/sub``) rather than by an
absolute path. The same anchors drive the public (relative) form of
paths in API responses.
ComfyUI directory outside it (``--output-directory D:\\Renders``) and of
each directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so such a
directory is addressed by its own name (``Renders/sub``) rather than by
an absolute path, and two folders of one name stay apart. The same
anchors drive the public (relative) form of paths in API responses.
"""
anchors = []
base = _comfyui_base_directory()
candidates = [base] if base else []
try:
base_key = _canonical_path(base) if base else None
except (OSError, ValueError):
base_key = None
for directory in _comfyui_directories():
try:
canonical = _canonical_path(directory)
except (OSError, ValueError):
continue
if base_key is None or not _lexically_inside(canonical, [base_key]):
candidates.append(os.path.dirname(canonical))
candidates.extend(os.path.dirname(root) for root in _extra_gallery_roots())
for candidate in candidates:
try:
@@ -399,33 +459,69 @@ class GalleryConfig:
(True, "") when the directory lies inside one of
:meth:`allowed_gallery_parents`; otherwise (False, reason).
"""
code, reason = cls.gallery_root_problem(path)
return code is None, reason
@classmethod
def gallery_root_problem(cls, path: Any) -> Tuple[Optional[str], str]:
"""``(code, reason)`` why ``path`` may not be a gallery root, else (None, "").
Codes: ``empty``, ``unresolvable``, ``outside-allowed``, ``not-found``
and ``not-directory``. No reason names the path.
Containment is decided twice. First lexically, with no filesystem
access at all: a path (UNC, ``\\\\?\\``, a device, anything) whose
normalised spelling lies outside every allowed parent answers
``outside-allowed`` before it is resolved, so the server never opens
or stats it (no SMB connection, no existence or timing oracle). Only
then is it resolved and checked again, which catches a symlink out.
The filesystem root and the home directory answer ``outside-allowed``
too: one answer for every caller (validate-source, the settings save,
config.json), and the detail goes to the log only.
"""
if not isinstance(path, str) or not path.strip():
return False, "Gallery root must be a non-empty path"
return "empty", "Gallery root must be a non-empty path"
pairs = cls._allowed_parent_pairs()
spellings = [spelling for pair in pairs for spelling in pair]
try:
lexical = cls._lexical_candidates(path)
except (TypeError, ValueError):
return "unresolvable", "Gallery root could not be resolved"
if not any(_lexically_inside(candidate, spellings) for candidate in lexical):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
try:
canonical = cls.resolve_gallery_root(path)
except (OSError, ValueError):
return False, "Gallery root could not be resolved"
return "unresolvable", "Gallery root could not be resolved"
if _is_filesystem_root(canonical):
return False, "Gallery root cannot be a filesystem root"
config_logger.info("Gallery root refused: a filesystem root")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if canonical == _canonical_path(os.path.expanduser("~")):
return False, "Gallery root cannot be the home directory"
config_logger.info("Gallery root refused: the home directory")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if not _lexically_inside(canonical, [canonical for canonical, _ in pairs]):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if not os.path.exists(canonical):
return False, "Gallery root does not exist"
return "not-found", "Gallery root does not exist"
if not os.path.isdir(canonical):
return False, "Gallery root is not a directory"
return "not-directory", "Gallery root is not a directory"
return None, ""
candidate = Path(canonical)
for parent in cls.allowed_gallery_parents():
if candidate.is_relative_to(Path(parent)):
return True, ""
@classmethod
def _lexical_candidates(cls, path: str) -> List[str]:
"""Every spelling ``path`` may resolve to, normalised as text (no IO).
return (
False,
"Gallery root must be inside a ComfyUI directory "
f"(output, input, temp, user) or one listed in {EXTRA_GALLERY_ROOTS_ENV}",
)
An absolute path is itself; a relative one is joined to each
:meth:`path_anchors` entry, as :meth:`resolve_gallery_root` tries them.
"""
path = path.strip()
if os.path.isabs(path):
return [_lexical_path(path)]
anchors = cls.path_anchors()
return [_lexical_path(os.path.join(anchor, path)) for anchor in anchors]
class IntegrationConfig:
+2
View File
@@ -127,6 +127,8 @@ export async function openProduction(page,route,theme,motion='no-preference') {
await expect(page.locator('#primary-nav .nav-count')).toHaveCount(3);
// SAVED VIEWS' Favorites count (D51), the concept's starred images in the concept fixture.
await expect(page.locator('#saved-nav a[data-route="favorites"] .nav-count')).toHaveCount(1);
// SAVED VIEWS' Needs review count (ruling R5): the exact pending count.
await expect(page.locator('#saved-nav a[data-route="review"] .nav-count')).toHaveCount(1);
}
/** Computed outline of the focused element, after keyboard focus so `:focus-visible` applies. */
+212
View File
@@ -0,0 +1,212 @@
// Needs review against the approved concept (#249 PR F, plan Task F3, Mock inventory M4: app.js renderReview
// a749–760, style.css s407–415, s796–814 and their breakpoints, screenshot needs-review-tagging.png). Production runs
// on the review fixture's concept-shaped queue (workspace_review_fixture.py `concept: true`: the concept's first three
// image names and sizes, and the concept's first prompt on the open item), so every remaining difference is a recorded
// deviation applied by mapConceptReview (D3, D29, D119) or a test failure. Paired captures of every state are attached
// as artifacts; no screenshot baseline is added.
import { test, expect } from './fixtures.mjs';
import { openConcept, openProduction, inventoryOf, mutate, saveImageTags, diffInventory, SIZES, THEMES } from './concept-inventory.mjs';
import { reviewInventory, mapConceptReview, taggingInventory, mapConceptTagging, diffTaggingInventory } from './tagging-canonical-comparison.mjs';
const CONCEPT_FILE = 'alpine_00001.png';
const CONCEPT_TAGS = ['cinematic', 'blue_hour', 'architecture'];
const SURFACE = ['h2', 'p.small', 'div.row-actions.spaced-15', 'div.field', 'section.positive.prompt-section',
'section.negative.prompt-section', 'div.dialog-actions'];
// The fake model's server default instructions (tests/browser/server.py TestModel.default_prompt), D84.
const INSTRUCTIONS = 'Describe the image';
const panel = page => page.locator('#workspace-main section.review-panel');
const tagDialog = page => page.getByRole('dialog', { name: 'Tag image', exact: true });
async function seed(request, data = {}) {
await expect(await request.post('/__test__/review/reset', { data: { items: 3, concept: true, ...data } })).toBeOK();
}
/** Production's Needs review page with its first item's sampler paths loaded. */
async function openReview(page, theme) {
await openProduction(page, 'review', theme);
await expect(panel(page).getByRole('combobox', { name: 'Sampler context', exact: true })).toBeEnabled();
await settle(page);
}
async function settle(page) {
await page.mouse.move(0, 0);
await page.evaluate(async () => {
await document.fonts.ready;
await Promise.all([...document.images].map(image => image.decode().catch(() => {})));
});
}
const conceptInventory = page => page.evaluate(reviewInventory, '#main');
const productionInventory = page => page.evaluate(reviewInventory, '#workspace-main');
const node = (tree, prefix) => tree.tag?.startsWith(prefix) ? tree : tree.children?.map(child => node(child, prefix)).find(Boolean) ?? null;
/** Computed values of CSS tokens, read through a probe element in the open page. */
const tokens = (page, declarations) => page.evaluate(text => {
const probe = document.createElement('i');
probe.style.cssText = text;
document.body.append(probe);
const style = getComputedStyle(probe), values = [style.borderTopColor, style.backgroundColor];
probe.remove();
return values;
}, declarations);
async function compareAt(page, request, width, height, theme) {
await seed(request);
await page.setViewportSize({ width, height });
await openConcept(page, 'review', theme);
await settle(page);
const concept = await conceptInventory(page);
await openReview(page, theme);
const actual = await productionInventory(page);
return { concept, actual, reference: mapConceptReview(concept, actual) };
}
test.beforeEach(async ({ page }) => {
page.errors = [];
page.on('pageerror', error => page.errors.push(error.message));
});
test.afterEach(async ({ page, request }) => {
expect(page.errors).toEqual([]);
await request.post('/__test__/jobs/restart');
});
for (const [width, height] of SIZES) for (const theme of THEMES) {
test(`concept and production Metadata review match at ${width}x${height} ${theme}`, async ({ page, request }) => {
const { actual, reference } = await compareAt(page, request, width, height, theme);
expect(diffInventory(reference, actual), `${width}x${height} ${theme}`).toEqual([]);
// Pinned concept values (M4), so a mapping mistake on both sides cannot pass unseen.
const main = actual.main;
expect(main.children.map(child => child.tag)).toEqual(['div.heading', 'div.notice', 'div.gallery-layout']);
const notice = node(main, 'div.notice');
const [accent, soft] = await tokens(page, 'border-top:1px solid var(--ca-accent);background:var(--soft-blue)');
expect([notice.borderTopColor, notice.borderLeftColor, notice.backgroundColor]).toEqual([accent, accent, soft]);
expect([notice.borderTopWidth, notice.borderTopLeftRadius, notice.paddingTop, notice.paddingLeft, notice.marginBottom])
.toEqual(['1px', '6px', '15px', '18px', '22px']);
expect(notice.flexDirection).toBe(width <= 700 ? 'column' : 'row');
expect(notice.children.map(child => child.children.map(item => [item.tag, item.text])))
.toEqual([[['h2', '3 images need a decision'], ['p', 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.']]]);
const layout = node(main, 'div.gallery-layout');
expect(layout.gridTemplateColumns).toBe(width <= 700 ? `${layout.width}px` : `${layout.width - 22 - 310}px 310px`);
const surface = node(main, 'section.review-panel');
expect(surface.children.map(child => child.tag)).toEqual(SURFACE);
expect(surface.children[0].text).toBe('Resolve the association');
expect(surface.children[1].text).toBe(CONCEPT_FILE);
const actions = surface.children.at(-1);
expect(actions.children.map(button => [button.text, button.primary])).toEqual([['Leave for later', false], ['Apply association', true]]);
expect(node(surface, 'select').options.at(-1)).toEqual(['Leave unresolved', false]);
expect(node(main, 'div.image-grid').children.map(card => node(card, 'h3').text))
.toEqual(['alpine_00001.png', 'dunes_00001.png', 'forest_00001.png']);
expect(actual.noPageScroll).toBe(true);
});
}
test('the gallery layout follows the concept at every breakpoint', async ({ page, request }) => {
await seed(request);
// 310px beside the queue, 275px and a 15px gap at ≤1250px, 280px at ≤950px, one column at ≤700px.
for (const [width, columns, gap] of [[1600, '310px', '22px'], [1200, '275px', '15px'], [900, '280px', '15px'], [390, null, '15px']]) {
await page.setViewportSize({ width, height: 1000 });
await openConcept(page, 'review', 'dark');
const concept = node((await conceptInventory(page)).main, 'div.gallery-layout');
await openReview(page, 'dark');
const actual = node((await productionInventory(page)).main, 'div.gallery-layout');
expect([actual.gridTemplateColumns, actual.columnGap], `${width}px`).toEqual([concept.gridTemplateColumns, concept.columnGap]);
expect([actual.gridTemplateColumns.split(' ').at(-1), actual.columnGap], `${width}px`).toEqual([columns ?? `${actual.width}px`, gap]);
expect(actual.gridTemplateColumns.split(' ').length, `${width}px`).toBe(width <= 700 ? 1 : 2);
}
});
test('SAVED VIEWS rows match the concept with Needs review after Favorites', async ({ page, request }) => {
await seed(request);
await page.setViewportSize({ width: 1600, height: 1100 });
await openConcept(page, 'review', 'dark');
const concept = (await inventoryOf(page, '#main')).chrome.navs.find(nav => nav.id === 'saved-nav');
await openReview(page, 'dark');
const actual = (await inventoryOf(page, '#workspace-main')).chrome.navs.find(nav => nav.id === 'saved-nav');
// Sample data: the concept stars three images; this queue's fixture stars none. Needs review counts the queue (3).
const reference = { ...concept, rows: concept.rows.map(row => row.text === 'Favorites' ? { ...row, count: { ...row.count, text: '0' } } : row) };
expect(diffInventory(reference, actual)).toEqual([]);
expect(actual.rows.map(row => [row.text, row.count?.text, row.current])).toEqual([['Favorites', '0', null], ['Needs review', '3', 'page']]);
// The count's auto start margin follows the row's own text; every other count token is shared.
const { text: favoriteCount, marginInlineStart: favoriteMargin, ...favoriteStyle } = actual.rows[0].count;
const { text: reviewCount, marginInlineStart: reviewMargin, ...reviewStyle } = actual.rows[1].count;
expect(reviewStyle).toEqual(favoriteStyle);
expect(actual.rows[1].icon.d).toBe('M4 3h16v18H4zM8 7h8M8 12h8M8 17h5');
});
test('review Tag dialog matches the needs-review-tagging reference', async ({ page, request }) => {
await seed(request);
await saveImageTags(request, `review/${CONCEPT_FILE}`, CONCEPT_TAGS);
await page.setViewportSize({ width: 1600, height: 1100 });
await openConcept(page, 'review', 'dark');
await page.locator('#main section.surface').getByRole('button', { name: 'Tag image', exact: true }).click();
await expect(tagDialog(page)).toBeVisible();
await settle(page);
const opened = await page.locator('dialog#modal').evaluate(taggingInventory);
await page.keyboard.press('Escape');
await openReview(page, 'dark');
await panel(page).getByRole('button', { name: 'Tag image', exact: true }).click();
await expect(tagDialog(page).locator('img.tag-image-preview')).toHaveAttribute('src', /\/image-annotations\/capture\//);
await expect(tagDialog(page).locator('#manual-image-tags')).toHaveValue(CONCEPT_TAGS.join(', '));
await settle(page);
const actual = await page.locator('dialog#modal').evaluate(taggingInventory);
const numberField = opened.modelPanel.fields.find(field => field.control.id === 'image-general-threshold');
const mapped = mapConceptTagging(opened, { instructions: INSTRUCTIONS, chip: opened.heading.size, numberField });
// D119: an undecided queue item names no prompt, so the heading and the preview's alt are the file name.
const reference = { ...mapped, heading: { ...mapped.heading, h3: { ...mapped.heading.h3, text: CONCEPT_FILE },
preview: { ...mapped.heading.preview, alt: CONCEPT_FILE } } };
expect(actual.scopeNote.text).toBe('Tags belong to this image. Resolve its prompt association separately in Needs review.');
expect(diffTaggingInventory(reference, actual)).toEqual([]);
await page.keyboard.press('Escape');
await expect(tagDialog(page)).toHaveCount(0);
});
test('the comparison notices a dropped notice border and reordered actions', async ({ page, request }) => {
const { actual, reference } = await compareAt(page, request, 1600, 1100, 'dark');
expect(diffInventory(reference, actual)).toEqual([]);
const undoBorder = await mutate(page, { css: '#workspace-main .notice { border: 0 !important; }' });
const borderless = await productionInventory(page);
expect(diffInventory(reference, borderless).map(item => item.path)).toEqual(expect.arrayContaining([
'main.children.1.borderTopWidth', 'main.children.1.borderTopStyle']));
await undoBorder();
await panel(page).locator('.dialog-actions').evaluate(row => row.prepend(row.lastElementChild));
const reordered = await productionInventory(page);
expect(diffInventory(reference, reordered).map(item => item.path)).toEqual(expect.arrayContaining([
'main.children.2.children.1.children.6.children.0.text', 'main.children.2.children.1.children.6.children.0.primary']));
await panel(page).locator('.dialog-actions').evaluate(row => row.prepend(row.lastElementChild));
expect(diffInventory(reference, await productionInventory(page))).toEqual([]);
});
// Paired captures for the PR (no baselines): the populated queue beside the concept in both themes and widths, then
// the states the concept lacks (conflict, limited item, unavailable file, empty queue) as production captures.
test('paired Needs review captures', async ({ page, request }, testInfo) => {
test.setTimeout(120_000);
const shot = async name => {
await settle(page);
const file = testInfo.outputPath(`${name}.png`);
await page.screenshot({ path: file, fullPage: true });
await testInfo.attach(name, { path: file, contentType: 'image/png' });
};
await seed(request);
for (const [width, height] of SIZES) for (const theme of THEMES) {
await page.setViewportSize({ width, height });
await openConcept(page, 'review', theme);
await shot(`concept-review-${width}-${theme}`);
await openReview(page, theme);
await shot(`production-review-${width}-${theme}`);
}
await page.setViewportSize({ width: 1600, height: 1100 });
await seed(request, { items: 0, conflict: true, limited: true, unavailable: true });
await openReview(page, 'dark');
await panel(page).getByRole('combobox', { name: 'Sampler context', exact: true }).selectOption({ index: 0 });
const decided = page.waitForResponse(item => /\/decision$/.test(item.url()));
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
expect((await decided).status()).toBe(409);
await expect(panel(page).locator('.warning-note')).toBeVisible();
await shot('production-review-conflict');
for (const [index, name] of [[1, 'limited'], [2, 'unavailable']]) {
await page.locator('#workspace-main .review-queue article').nth(index).getByRole('button', { name: /^Inspect / }).click();
await expect(panel(page).locator('.warning-note')).toBeVisible();
await shot(`production-review-${name}`);
}
await seed(request, { items: 0 });
await openProduction(page, 'review', 'dark');
await expect(page.locator('#workspace-main .notice h2')).toHaveText('No images need a decision');
await shot('production-review-empty');
});
+8
View File
@@ -40,6 +40,7 @@ from tests.browser.workspace_library_editor_fixture import (
)
from tests.browser.workspace_vocabulary_fixture import register_vocabulary_commands
from tests.browser.workspace_tagging_fixture import register_tagging_commands
from tests.browser.workspace_review_fixture import register_review_commands
CANONICAL_ROOT = Path(__file__).resolve().parents[2]
CANONICAL_ASSETS = {
@@ -197,6 +198,7 @@ def create_app(root: Path) -> web.Application:
library_editor_commands = register_library_editor_commands(routes, api, root)
register_vocabulary_commands(routes, api, root)
register_tagging_commands(routes, api, root, model)
register_review_commands(routes, api, root)
reveal_stub = register_reveal_stub(routes, api, root)
register_activity_commands(app, routes, api)
register_system_commands(app, routes, api, root)
@@ -226,6 +228,8 @@ def create_app(root: Path) -> web.Application:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -591,9 +595,13 @@ def create_app(root: Path) -> web.Application:
return web.json_response("dark")
async def cleanup(application: web.Application) -> None:
# A scan of an ambiguous fixture file captures it for Needs review.
api.close_annotation_captures()
api.db.close_all()
seed()
# Open Needs review change streams end first, as on the real host.
app.on_shutdown.append(api.close_review_events)
app.on_cleanup.append(cleanup)
app.add_routes(routes)
return app
+6 -5
View File
@@ -68,12 +68,12 @@ function remainingFooter(footer,spans) {
const OVERVIEW_GLYPH='M3 11 12 4l9 7M5 10v10h14V10M10 20v-6h4v6';
/**
* The concept's SAVED VIEWS rows (app.js a331–335) as production lists them: Favorites with its count (D51), then the
* saved views. D50: no "Needs review" row until the tag-review slice (maintainer ruling 2026-10-21).
* The concept's SAVED VIEWS rows (app.js a331–335) as production lists them: Favorites with its count (D51), Needs
* review with the exact pending count (ruling R5), then the saved views. The concept's "3" is sample data: the concept
* fixture records no review items, so production's count reads "0".
*/
function savedNav(nav) {
const rows=nav.rows.filter(row=>row.text!=='Needs review');
return {...nav,rows:rows.map((row,index)=>index===rows.length-1?{...row,gapAfter:null}:row)};
return {...nav,rows:nav.rows.map(row=>row.text==='Needs review'?{...row,count:{...row.count,text:'0'}}:row)};
}
/** Recorded shell deviations only (spec §8). Everything else in the concept inventory must equal production. */
@@ -243,8 +243,9 @@ for(const [width,height] of SIZES)test(`a saved view row matches the concept's $
await openProduction(page,'images','dark');
const actual=await save('Save','#workspace-main');
expect(diffInventory(mapShellConcept(reference).chrome.navs,actual.chrome.navs)).toEqual([]);
const referenceRows=reference.chrome.navs.find(nav=>nav.id==='saved-nav').rows;
expect(actual.chrome.navs.find(nav=>nav.id==='saved-nav').rows.map(row=>[row.text,row.count]))
.toEqual([['Favorites',reference.chrome.navs.find(nav=>nav.id==='saved-nav').rows[0].count],['Golden hour',null]]);
.toEqual([['Favorites',referenceRows[0].count],['Needs review',{...referenceRows[1].count,text:'0'}],['Golden hour',null]]);
});
test('the inventory reports a missing shell element by its path instead of failing',async({page})=>{
+501 -6
View File
@@ -169,9 +169,9 @@ function mapOverviewConcept(reference, overview) {
const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts),
'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) };
const rows = {
// D93 (Q6): no review queue yet, so no count and the action is unavailable (dimmed, as the Save row's D92).
'3 ambiguous image links': row => ({ ...row, title: 'Ambiguous image links', description: 'Not available yet.',
action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }),
// D93: the real pending count; Review metadata opens Needs review (the concept's go-review).
'3 ambiguous image links': row => ({ ...row,
title: `${number(overview.needs_review.pending)} ambiguous image link${overview.needs_review.pending === 1 ? '' : 's'}` }),
// D93: no backup has been recorded (verified backups arrive with S8).
'Backup status': row => ({ ...row, description: 'No retained backup record.' }),
// D93: there is no paging, so the shared settings are grid and density.
@@ -230,12 +230,11 @@ for (const theme of THEMES) for (const [width, height] of SIZES) {
});
}
// The recorded differences, by path: D93 the review row (title, Not available yet, dimmed action), the backup row, the
// The recorded differences, by path: D93 the review row (its real count), the backup row, the
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
const UNMAPPED_PATHS = [
'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity',
'columns.surfaces.0.rows.0.description', 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
'save.button.cursor', 'save.button.opacity', 'save.status.text',
@@ -408,3 +407,499 @@ for (const theme of THEMES) test(`statistics and health dialogs match the concep
await page.evaluate(() => { document.querySelector('#modal-body caption').textContent = 'Runs per category'; });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)).toEqual(['tables.0.caption']);
});
/** Runs in the page: Appearance's surface, every row's text and control, and the concept's row geometry and tokens. */
function measureAppearance(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const surface = layout.querySelector(':scope > section.surface');
const control = row => {
const select = row.querySelector(':scope > select');
if (select) return { kind: 'select', id: select.id, options: [...select.options].map(option => option.textContent),
selected: select.selectedOptions[0]?.textContent ?? null };
const input = row.querySelector(':scope > input');
if (input) return { kind: `input.${input.type}`, id: input.id, value: input.value };
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const chip = row.querySelector(':scope > .chip');
return chip ? { kind: name(chip), text: text(chip), ...css(chip, 'color', 'backgroundColor', 'fontSize') } : null;
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
const operations = [...surface.querySelectorAll(':scope > .operation')];
const firstRow = rows[0];
return {
layout: [...layout.children].map(name),
surface: { tree: [...surface.children].map(name), heading: text(surface.querySelector(':scope > h2')),
intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius') },
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: operations.map(row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), action: text(row.querySelector('.row-actions > button')) })),
row: css(firstRow, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopWidth',
'borderTopColor', 'marginTop'),
label: css(firstRow.querySelector('label'), 'fontSize', 'fontWeight', 'color'),
description: css(firstRow.querySelector('p'), 'fontSize', 'color', 'marginTop', 'maxWidth'),
select: css(firstRow.querySelector('select'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor', 'color', 'borderTopColor'),
heading3: css(surface.querySelector('.setting-row h3'), 'fontSize', 'fontWeight', 'color'),
number: css(surface.querySelector('.setting-row input[type="number"]'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D97: the rows Appearance adds for the legacy gallery settings, after the concept's own, as production renders them.
const ADDED_APPEARANCE_ROWS = [
['card-details', 'Show image details on cards', 'Show each card’s title and file details under the preview.', true],
['viewer-details', 'Collapse viewer details by default', 'Open the image viewer with its details panel hidden.', false],
];
/** The concept's Appearance with the recorded deviations applied, given the server's real settings. */
function mapAppearanceConcept(reference, settings) {
const toggleRow = reference.rows.find(row => row.control?.kind === 'switch');
const added = ADDED_APPEARANCE_ROWS.map(([id, label, description, checked]) => ({
tree: toggleRow.tree, text: [['label', label, id], ['p', description, null]],
control: { ...toggleRow.control, id, checked } }));
return {
...reference,
// D97: two added toggles and the "Reset display defaults" operation close the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.setting-row', 'div.setting-row', 'div.operation'] },
rows: [...reference.rows.map(row => row.control?.id === 'timeout'
// The node search timeout is the server's real `result_timeout`, not the concept's sample 30.
? { ...row, control: { ...row.control, value: String(settings.result_timeout) } } : row), ...added],
operations: [{ tree: ['div', 'div.row-actions'], title: 'Reset display defaults',
description: 'Restore this browser’s display defaults, then save. Other settings are kept.', action: 'Reset' }],
};
}
async function conceptAppearance(page, theme) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Appearance', exact: true }).click();
await expect(page.locator('#main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#main');
}
async function productionAppearance(page, theme) {
await openProduction(page, 'system/appearance', theme);
await expect(page.locator('#workspace-main #timeout')).toBeEnabled();
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`appearance matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-appearance-${theme}-${width}.png`), fullPage: true });
const production = await productionAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-appearance-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapAppearanceConcept(concept, settings), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('appearance comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, 'dark');
const expected = mapAppearanceConcept(concept, settings);
const production = await productionAppearance(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D97 and the real timeout are recorded.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.0', 'operations.length', 'rows.11', 'rows.12', 'rows.13', 'rows.length', 'surface.tree'].sort());
const measure = () => page.evaluate(measureAppearance, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .setting-row:nth-of-type(3)'] });
expect(await paths()).toEqual(expect.arrayContaining(['rows.length', 'surface.tree.length']));
await undoRow();
// A changed label and a changed option.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail sizes'; });
expect(await paths()).toEqual(['rows.1.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail size'; });
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Medium'; });
expect(await paths()).toEqual(expect.arrayContaining(['rows.5.control.options.0']));
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Optimized'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['row.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: Sources & scans (app.js 679–681): the optional notice, then the "Image sources" surface with its
* panel heading, the source list, the two setting rows and the operations, with the concept's geometry and tokens.
*/
function measureSources(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') });
const operation = row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), actions: [...row.querySelectorAll('.row-actions > button')].map(button) });
const notice = layout.querySelector(':scope > .notice');
const surface = layout.querySelector(':scope > section.surface');
const heading = surface.querySelector(':scope > .panel-heading');
const list = surface.querySelector(':scope > #source-list');
const control = row => {
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const input = row.querySelector(':scope > input');
return { kind: `input.${input.type}`, id: input.id, min: input.min, max: input.max, value: input.value,
...css(input, 'width', 'accentColor') };
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
return {
layout: [...layout.children].map(name),
notice: notice && { tree: [...notice.children].map(name),
text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, text(node)]),
actions: [...notice.querySelectorAll(':scope > .row-actions > button')].map(button),
...css(notice, 'display', 'alignItems', 'columnGap', 'borderTopColor', 'backgroundColor', 'paddingTop', 'marginBottom') },
surface: { tree: [...surface.children].map(name), intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
heading: { tree: [...heading.children].map(name), title: text(heading.querySelector('h2')),
action: button(heading.querySelector('button')), ...css(heading, 'display', 'alignItems', 'justifyContent', 'marginBottom'),
h2: css(heading.querySelector('h2'), 'fontSize', 'fontWeight', 'color') },
sources: [...list.children].map(operation),
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: [...surface.querySelectorAll(':scope > .operation')].map(operation),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop',
'paddingBottom', 'borderTopWidth', 'borderTopColor'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
p: css(list.querySelector('.operation p'), 'fontSize', 'color', 'marginTop'),
row: css(rows[0], 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D128: "watching until restart" after monitoring was switched off while the observer still runs.
const WATCH_TEXT = { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', restart_pending: 'restart pending' };
const WATCHED = new Set(['watching', 'stopping']);
// D98: the operation Sources adds after the concept's own, as production renders it.
const REFRESH_FILES = { title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
label: 'Refresh' };
/** The concept's Sources with the recorded deviations applied, given the server's real settings and sources. */
function mapSourcesConcept(reference, { settings, sources }) {
const [sample] = reference.sources;
const scan = sample.actions[0];
return {
...reference,
// The real sources replace the two samples (without the "sample path" note). D98: each configured one has Remove,
// a plain button after its scan.
sources: sources.effective.map(source => ({ ...sample, title: source.label,
description: `${source.path} · ${WATCH_TEXT[source.watch]}`,
actions: [{ ...scan, text: WATCHED.has(source.watch) ? 'Scan changes' : 'Scan folder' }, { ...scan, text: 'Remove' }] })),
rows: reference.rows.map(row => {
if (row.control.id === 'monitoring') return { ...row, control: { ...row.control, checked: settings.monitoring_enabled } };
// D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count.
return { ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } };
}),
// D98: "Refresh files" closes the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.operation'] },
operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title,
description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }],
};
}
async function conceptSources(page, theme, { notice }) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Sources & scans', exact: true }).click();
await expect(page.locator('#main .panel-heading > h2')).toHaveText('Image sources');
if (!notice) await page.locator('#main .notice').getByRole('button', { name: 'Later', exact: true }).click();
await expect(page.locator('#main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#main');
}
async function productionSources(page, theme, { notice }) {
// about:blank has no storage; the workspace page reads the dismissal.
if (!notice) await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
await openProduction(page, 'system/sources', theme);
await expect(page.locator('#workspace-main #source-list .operation')).toHaveCount(2);
await expect(page.locator('#workspace-main #workers')).toBeEnabled();
await expect(page.locator('#workspace-main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#workspace-main');
}
/** The fixture's sources as the concept shows its samples: the first one watched, the second not. */
async function watchedSources(request) {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
return serverJson(request, '/prompt_manager/settings');
}
for (const theme of THEMES) for (const [width, height] of SIZES) for (const notice of [true, false]) {
const variant = notice ? 'with' : 'without';
test(`sources matches the concept ${variant} the notice ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const server = await watchedSources(request);
const concept = await conceptSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`concept-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
const production = await productionSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`production-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapSourcesConcept(concept, server), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('sources comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const server = await watchedSources(request);
const concept = await conceptSources(page, 'dark', { notice: true });
const expected = mapSourcesConcept(concept, server);
const production = await productionSources(page, 'dark', { notice: true });
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only in the real sources, the D98 operation and the D113 range.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.2', 'operations.length', 'rows.1', 'sources.0', 'sources.1', 'surface.tree'].sort());
const measure = () => page.evaluate(measureSources, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed notice and a removed source row.
const undoNotice = await mutate(page, { remove: ['#workspace-main .settings-layout > .notice'] });
expect(await paths()).toEqual(expect.arrayContaining(['layout.length', 'notice.tree.length']));
await undoNotice();
const undoRow = await mutate(page, { remove: ['#workspace-main #source-list > .operation:nth-child(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['sources.length']));
await undoRow();
// A changed label and a changed button.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for outputs'; });
expect(await paths()).toEqual(['rows.0.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for new outputs'; });
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = 'Add source'; });
expect(await paths()).toEqual(['heading.action.text']);
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = '+ Add source'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['operation.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: the open "Add image source" dialog (the concept's `simpleInput`, app.js 1138 and 1497): its title,
* the body's tree, the field's label and input, and the actions, with the concept's geometry and tokens.
*/
function measureAddSource() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const body = dialog.querySelector('#modal-body');
const field = body.querySelector(':scope > .field');
const label = field?.querySelector(':scope > label');
const input = field?.querySelector(':scope > input');
const actions = body.querySelector(':scope > .dialog-actions');
const rect = dialog.getBoundingClientRect();
return {
title: text(dialog.querySelector('#modal-title')),
tree: [...body.children].map(name),
field: field && { tree: [...field.children].map(name), ...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom') },
label: label && { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
input: input && { id: input.id, type: input.type, value: input.value,
...css(input, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
width: Math.round(rect.width),
};
}
/** D127: the field is "Folder path", starts empty (no sample folder) and its action is "Add". */
function mapAddSourceConcept(reference) {
const [cancel, apply] = reference.actions.buttons;
return { ...reference, label: { ...reference.label, text: 'Folder path' }, input: { ...reference.input, value: '' },
actions: { ...reference.actions, buttons: [cancel, { ...apply, text: 'Add' }] } };
}
async function conceptAddSource(page, theme) {
await conceptSources(page, theme, { notice: false });
await page.locator('#main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
async function productionAddSource(page, theme) {
await productionSources(page, theme, { notice: false });
await page.locator('#workspace-main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`add source dialog matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
await watchedSources(request);
const concept = await conceptAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-add-source-${theme}-${width}.png`) });
const production = await productionAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-add-source-${theme}-${width}.png`) });
expect(diffInventory(mapAddSourceConcept(concept), production)).toEqual([]);
});
}
test('add source comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
await watchedSources(request);
const concept = await conceptAddSource(page, 'dark');
const expected = mapAddSourceConcept(concept);
const production = await productionAddSource(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D127 is recorded.
expect(diffInventory(concept, production).map(item => item.path).sort())
.toEqual(['actions.buttons.1.text', 'input.value', 'label.text']);
const paths = async () => diffInventory(expected, await page.evaluate(measureAddSource)).map(item => item.path);
// A changed label, a swapped action order and a changed style.
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder'; });
expect(await paths()).toEqual(['label.text']);
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder path'; });
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
const undoStyle = await mutate(page, { css: 'dialog#modal #action-input { border-color: rgb(255, 0, 0); } dialog#modal .dialog-actions { gap: 31px; }' });
expect(await paths()).toEqual(expect.arrayContaining(['input.borderTopColor', 'actions.columnGap']));
await undoStyle();
expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]);
});
/**
* Runs in the page: the open "Thumbnail maintenance" dialog (the concept's `thumbnail-review`, app.js 1433–1437): its
* title, the body's tree, each `.field` (label, control, options, selection, limits), the `p.small` line and the
* actions, with the concept's geometry and tokens. The concept has no `p.small`, so a probe one stands in for its style.
*/
function measureThumbnailReview() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const body = dialog.querySelector('#modal-body');
const fields = [...body.querySelectorAll(':scope > .field')].map(field => {
const label = field.querySelector(':scope > label');
const control = field.querySelector(':scope > select, :scope > input');
return {
tree: [...field.children].map(name),
...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom'),
label: { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
control: { id: control.id, tag: control.localName, type: control.type,
options: control.options ? [...control.options].map(option => option.textContent) : null,
selected: control.options ? control.selectedIndex : null,
value: control.localName === 'input' ? control.value : null,
min: control.getAttribute('min'), max: control.getAttribute('max'),
...css(control, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
};
});
let line = body.querySelector(':scope > p.small');
const probe = !line;
if (probe) { line = document.createElement('p'); line.className = 'small'; body.append(line); }
const plan = { text: probe ? null : text(line), role: probe ? null : line.getAttribute('role'),
...css(line, 'fontSize', 'color', 'marginTop', 'marginBottom') };
if (probe) line.remove();
const actions = body.querySelector(':scope > .dialog-actions');
const rect = dialog.getBoundingClientRect();
return {
title: text(dialog.querySelector('#modal-title')),
tree: [...body.children].map(name),
fields,
plan,
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
width: Math.round(rect.width),
};
}
/**
* D99: the planned counts line (`p.small[role=status]`) before the actions, from the server's plan for the chosen
* options, and the primary "Start" (the concept's "Simulate thumbnail job").
*/
function mapThumbnailConcept(reference, plan) {
const [cancel, start] = reference.actions.buttons;
const tree = [...reference.tree];
tree.splice(tree.indexOf('div.dialog-actions'), 0, 'p.small');
return { ...reference, tree,
plan: { ...reference.plan, role: 'status',
text: `${number(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ${number(plan.fresh)} already up to date.` },
actions: { ...reference.actions, buttons: [cancel, { ...start, text: 'Start' }] } };
}
async function conceptThumbnailReview(page, theme) {
await conceptSources(page, theme, { notice: false });
const row = page.locator('#main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
await row.getByRole('button', { name: 'Review options', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
await page.mouse.move(0, 0);
return page.evaluate(measureThumbnailReview);
}
/** Production's dialog from Sources' Review options or Overview's Review thumbnails, once its plan line answered. */
async function productionThumbnailReview(page, theme, entry) {
if (entry === 'sources') {
await productionSources(page, theme, { notice: false });
const row = page.locator('#workspace-main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
await row.getByRole('button', { name: 'Review options', exact: true }).click();
} else {
await openProduction(page, 'system/overview', theme);
await page.locator('#workspace-main').getByRole('button', { name: 'Review thumbnails', exact: true }).click();
}
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
await expect(page.locator('dialog#modal[open] #modal-body > p.small')).toHaveText(/ already up to date\.$/);
await page.mouse.move(0, 0);
return page.evaluate(measureThumbnailReview);
}
const thumbnailPlan = request => serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
for (const entry of ['sources', 'overview']) for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`thumbnail dialog matches the concept from ${entry} ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
await watchedSources(request);
const plan = await thumbnailPlan(request);
const concept = await conceptThumbnailReview(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-thumbnail-review-${theme}-${width}.png`) });
const production = await productionThumbnailReview(page, theme, entry);
await page.screenshot({ path: testInfo.outputPath(`production-thumbnail-review-${entry}-${theme}-${width}.png`) });
expect(diffInventory(mapThumbnailConcept(concept, plan), production)).toEqual([]);
});
}
test('thumbnail dialog comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
await watchedSources(request);
const plan = await thumbnailPlan(request);
const concept = await conceptThumbnailReview(page, 'dark');
const expected = mapThumbnailConcept(concept, plan);
const production = await productionThumbnailReview(page, 'dark', 'sources');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D99 is recorded.
expect(diffInventory(concept, production).map(item => item.path).sort())
.toEqual(['actions.buttons.1.text', 'plan.role', 'plan.text', 'tree.3', 'tree.4', 'tree.length']);
const paths = async () => diffInventory(expected, await page.evaluate(measureThumbnailReview)).map(item => item.path);
// A changed option label, a swapped option order, a swapped action order and a changed style.
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium'; });
expect(await paths()).toEqual(['fields.1.control.options.0']);
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium · optimized'; });
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); });
expect(await paths()).toEqual(expect.arrayContaining(['fields.0.control.options.0', 'fields.0.control.options.1']));
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); select.selectedIndex = 0; });
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
const undoStyle = await mutate(page, { css: 'dialog#modal #thumb-limit { border-color: rgb(255, 0, 0); } dialog#modal #modal-body > p.small { color: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['fields.2.control.borderTopColor', 'plan.color']));
await undoStyle();
expect(diffInventory(expected, await page.evaluate(measureThumbnailReview))).toEqual([]);
});
@@ -140,3 +140,152 @@ export function mapConceptTagging(reference,{instructions,limit=10,chip,numberFi
export function withoutChipGeometry(inventory) {
return {...inventory,suggestions:{...inventory.suggestions,rows:inventory.suggestions.rows.map(({paddingTop,paddingLeft,alignItems,...row})=>row)}};
}
/**
* Needs review (plan Task F3, Mock inventory M4: app.js renderReview a749–760, style.css s407–415, s796–814 and their
* breakpoints): the page's main element as a tree, runs in the page. Every shown box below `main` to `depth` levels is
* recorded with its tag and classes, its box relative to its parent, its fonts, edges and layout, a leaf's text, a
* button's label and icon, a select's options; a `display: contents` node is replaced by its children (D20). A flex
* row of actions also records the gaps between its children and its right inset, so a row whose button copy differs
* (D29) still compares by order, spacing and alignment.
*/
export function reviewInventory(selector) {
const main=document.querySelector(selector);
// Half pixels: the concept's 13.5px line heights add up exactly, so a box that loses one line moves by 13.5px.
const px=value=>Math.round(value*2)/2;
const css=(node,...keys)=>{const style=getComputedStyle(node);return Object.fromEntries(keys.map(key=>[key,style[key]]));};
const rect=node=>node.getBoundingClientRect();
const shown=node=>node.getClientRects().length>0&&getComputedStyle(node).visibility!=='hidden';
const relative=(node,origin)=>{const a=rect(node),b=rect(origin);return {left:px(a.left-b.left),top:px(a.top-b.top),width:px(a.width),height:px(a.height)};};
const font=node=>css(node,'fontFamily','fontSize','fontWeight','lineHeight','letterSpacing','color','textTransform','textAlign');
const edges=node=>css(node,'paddingTop','paddingRight','paddingBottom','paddingLeft','marginTop','marginRight','marginBottom','marginLeft',
'borderTopWidth','borderTopStyle','borderTopColor','borderRightWidth','borderRightStyle','borderRightColor','borderBottomWidth',
'borderBottomStyle','borderBottomColor','borderLeftWidth','borderLeftStyle','borderLeftColor','borderTopLeftRadius','backgroundColor',
'boxShadow','outlineStyle');
const layout=node=>css(node,'display','position','flexDirection','flexWrap','justifyContent','alignItems','columnGap','rowGap',
'gridTemplateColumns','overflowX','overflowY','objectFit','minHeight','minWidth');
const tagOf=node=>`${node.localName}${[...node.classList].sort().map(name=>`.${name}`).join('')}`;
const textOf=node=>node.textContent.replace(/\s+/g,' ').trim();
const boxed=node=>[...node.children].flatMap(child=>getComputedStyle(child).display==='contents'?boxed(child):[child]);
const LEAVES=new Set(['button','select','input','img','svg','textarea','progress']);
function nodeOf(node,origin,depth) {
const base={tag:tagOf(node),...relative(node,origin),...font(node),...edges(node),...layout(node)};
const name=node.localName;
if(name==='svg')return {tag:base.tag,width:base.width,height:base.height,d:node.querySelector('path')?.getAttribute('d')??null,
...css(node,'strokeWidth','color','verticalAlign')};
if(name==='button')return {...base,text:textOf(node),label:node.getAttribute('aria-label'),pressed:node.getAttribute('aria-pressed'),
disabled:node.getAttribute('aria-disabled')==='true'||node.disabled,primary:node.classList.contains('ca-primary'),
children:boxed(node).filter(shown).map(child=>nodeOf(child,node,0))};
if(name==='select')return {...base,id:node.id,options:[...node.options].map(option=>[option.textContent,option.disabled])};
if(name==='input')return {...base,id:node.id,type:node.type,label:node.getAttribute('aria-label')};
if(name==='img')return {...base,alt:node.getAttribute('alt')};
const children=boxed(node).filter(shown);
const item={...base,id:node.id||null,role:node.getAttribute('role'),label:node.getAttribute('aria-label'),
text:children.length?null:textOf(node),htmlFor:node.htmlFor||null};
if(/(^|\s)(dialog-actions|row-actions|image-card-actions)(\s|$)/.test(node.className)&&children.length) {
const boxes=children.map(rect);
item.gaps=boxes.slice(1).map((box,index)=>px(box.left-boxes[index].right));
item.rightInset=px(rect(node).right-boxes.at(-1).right);
}
if(depth>0&&!LEAVES.has(name))item.children=children.map(child=>nodeOf(child,node,depth-1));
return item;
}
return {main:nodeOf(main,main,8),noPageScroll:document.documentElement.scrollWidth<=innerWidth+1};
}
/**
* `tree` with `edit(node)` applied to every node `match` picks (a tag prefix or a predicate), depth first, new objects
* only.
*/
function editNodes(tree,match,edit) {
const children=tree.children?.map(child=>editNodes(child,match,edit));
const node=children?{...tree,children}:tree;
const picked=typeof match==='function'?match(node):node.tag?.startsWith(match);
return picked?edit(node):node;
}
const findNode=(tree,prefix)=>tree.tag?.startsWith(prefix)?tree:tree.children?.map(child=>findNode(child,prefix)).find(Boolean)??null;
/**
* The concept's Metadata review inventory (reviewInventory) with the recorded deviations applied, the remaining sample
* data taken from production (`actual`) only where the concept's own copy cannot say it:
* - D29: no "Sample review queue" chip in the notice; "Resolve the association" and "Apply association" without the
* prototype's "sample";
* - D119: queue cards carry no selection checkbox and no star (the overlay row stays, empty); a card's title is its
* file name (an undecided item names no prompt) and its line is the extension only (D11 omits the unknown size and
* seed, which a queue item does not carry); the image's alt is the file name; the prompt blocks' Copy and Send names
* carry no prompt title;
* - D3: the grid keeps the Images grid's `position: relative` (its cards stay in its flow);
* - production's hooks and names: the `review-queue` / `review-panel` classes, the queue's list roles and name, the
* panel heading's id (the region's name) and the prompt sections' region names;
* - sample data: the Sampler context's candidate options (the file's own evidence; the last option, Leave unresolved,
* stays compared) and the widths that follow from copy the deviations above change (a card's text column, the
* panel's action buttons; their gaps and right inset stay compared).
* Where that copy changes the height of a box, the boxes after it move by exactly that much: the stacked notice (390)
* loses the chip and its row gap; a card whose concept line wraps (390: "1536 × 1152 · PNG · Seed 42001") takes
* production's one-line height, its row of the grid shrinks by the most any of its cards lost, and in one column the
* panel below the grid moves up by the grid's loss. The actions keep production's centred place in the card's info.
*/
export function mapConceptReview(reference,actual) {
const named=(node,id)=>({...node,id});
const actualCards=findNode(actual.main,'div.image-grid')?.children??[];
const actualActions=findNode(actual.main,'div.dialog-actions');
let cardIndex=0;
let main={...reference.main,id:actual.main.id};
let noticeLoss=0;
main=editNodes(main,'div.notice',notice=>{
const chip=notice.children.find(child=>child.tag==='span.chip');
if(chip&&notice.flexDirection==='column')noticeLoss=chip.height+parseFloat(notice.rowGap);
return {...notice,height:notice.height-noticeLoss,children:notice.children.filter(child=>child!==chip)};
});
const noticeIndex=main.children.findIndex(child=>child.tag==='div.notice');
main={...main,height:main.height-noticeLoss,children:main.children.map((child,index)=>index>noticeIndex?{...child,top:child.top-noticeLoss}:child)};
const losses=[];
main=editNodes(main,'div.image-grid',grid=>({...grid,tag:'div.image-grid.review-queue',position:'relative',role:'list',label:'Review queue'}));
main=editNodes(main,'article.media-card',card=>{
const twin=actualCards[cardIndex++];
const filename=findNode(card,'button.image-open').label.replace(/^Open /,'');
const art=editNodes(card.children[0],'div.image-overlays',overlays=>({...overlays,height:0,text:'',children:[]}));
const withAlt=editNodes(art,'img',img=>({...img,alt:filename}));
const info=card.children[1];
const [copy,...rest]=info.children;
const twinCopy=twin?.children[1]?.children[0];
const [h3,p]=copy.children;
const twinLine=twinCopy?.children[1];
const loss=twinLine?p.height-twinLine.height:0;
losses.push({top:card.top,loss});
const mappedCopy={...copy,width:twinCopy?.width,height:copy.height-loss,children:[{...h3,text:filename,width:twinCopy?.children[0]?.width},
{...p,text:'PNG',width:twinLine?.width,height:p.height-loss}]};
const [actions,...after]=rest;
const twinActions=twin?.children[1]?.children[1];
const mappedActions=actions&&loss?{...actions,top:twinActions?.top}:actions;
return {...card,role:'listitem',height:card.height-loss,
children:[withAlt,{...info,height:info.height-loss,children:[mappedCopy,...(actions?[mappedActions]:[]),...after]},...card.children.slice(2)]};
});
// Each grid row loses what its most shortened card lost; later rows move up by the rows before them.
const rowLoss=new Map();
for(const {top,loss} of losses)rowLoss.set(top,Math.max(rowLoss.get(top)??0,loss));
const lossBefore=top=>[...rowLoss].filter(([rowTop])=>rowTop<top).reduce((sum,[,loss])=>sum+loss,0);
const gridLoss=[...rowLoss.values()].reduce((sum,loss)=>sum+loss,0);
main=editNodes(main,'div.image-grid',grid=>({...grid,height:grid.height-gridLoss,
children:grid.children.map(card=>({...card,top:card.top-lossBefore(card.top)}))}));
main=editNodes(main,'div.gallery-layout',layout=>{
if(!gridLoss||layout.gridTemplateColumns.trim().split(/\s+/).length>1)return layout;
return {...layout,height:layout.height-gridLoss,children:layout.children.map((child,index)=>index?{...child,top:child.top-gridLoss}:child)};
});
if(gridLoss&&findNode(main,'div.gallery-layout').gridTemplateColumns.trim().split(/\s+/).length===1)main={...main,height:main.height-gridLoss};
main=editNodes(main,'section.surface',surface=>{
const [h2,...rest]=surface.children;
return {...surface,tag:'section.review-panel.surface',children:[named({...h2,text:h2.text.replace('Resolve the sample association','Resolve the association')},'review-panel-title'),...rest]};
});
main=editNodes(main,'select',select=>{
const twin=findNode(actual.main,'select');
return select.id==='review-context'?{...select,options:[...twin.options.slice(0,-1),select.options.at(-1)]}:select;
});
main=editNodes(main,node=>/^section\..*\bprompt-section\b/.test(node.tag),section=>({...section,label:section.tag.includes('positive')?'Positive prompt':'Negative prompt'}));
main=editNodes(main,node=>node.tag?.startsWith('button')&&/^(Copy|Send) (positive|negative) prompt/.test(node.label??''),
button=>({...button,label:button.label.replace(/: .+$/,'')}));
main=editNodes(main,'div.dialog-actions',actions=>({...actions,children:actions.children.map((button,index)=>({...button,
text:button.text.replace('Apply sample association','Apply association'),
left:actualActions?.children[index]?.left,width:actualActions?.children[index]?.width}))}));
return {...reference,main};
}
+108 -2
View File
@@ -54,8 +54,8 @@ async function step(request, id, data) {
await expect(await request.post(`/__test__/jobs/${id}/step`, { data })).toBeOK();
}
async function finish(request, id, state = 'finished') {
const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state } });
async function finish(request, id, state = 'finished', extra = {}) {
const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state, ...extra } });
await expect(response).toBeOK();
return (await response.json()).job;
}
@@ -284,6 +284,19 @@ test('reload during a scan reattaches to the running job', async ({ page, reques
await expect(pill(page)).toBeHidden();
});
test('an empty Activity says no job has run, and a list that cannot be read says so', async ({ page }) => {
// The empty and failed states were only reached in passing by other tests, so their coverage depended on timing.
const empty = main(page).locator('section.surface > p.empty-state');
await page.goto(`${workspace}#/activity`);
await expect(empty).toHaveText('No jobs have run since ComfyUI started. Scans and other long operations appear here.');
await expect(empty).not.toHaveAttribute('role');
await expect(items(page)).toHaveCount(0);
await page.route(url => url.pathname === '/prompt_manager/jobs', route => route.abort());
await page.reload();
await expect(empty).toHaveText('Activity could not be read. It is tried again shortly.');
await expect(empty).toHaveAttribute('role', 'alert');
});
test('lost server clears the pill and marks jobs ended', async ({ page, request }) => {
// A poll interval after the restart, then failed re-reads backing off for 1 + 2 + 4 s.
test.setTimeout(90_000);
@@ -995,3 +1008,96 @@ test('the report download names the kind', async ({ page, request }) => {
expect(report.outcomes.tagged.rows.map(row => [row.path, row.detail])).toEqual([['sub/a.png', 'Added 1 tag']]);
await expect(toast(page)).toHaveText('Job report downloaded.');
});
// D111 (System S5): each kind's own descriptor. A thumbnail job counts Generated (green), Skipped and Errors (red),
// its pill says "Generating thumbnails" and its toasts "Thumbnails finished · N generated · M errors".
const THUMB_JOB = { kind: 'thumbnails', total: 10, output_folders: 2 };
test('a thumbnail job shows Generated Skipped Errors, its pill verb and its toasts', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const job = await startJob(request, THUMB_JOB);
await step(request, job.id, { processed: 4, records: [
{ group: 'generated', path: 'sub/a.png', root: 'output', detail: 'Preview written' },
{ group: 'skipped', path: 'b.png', root: 'input', detail: 'Preview already up to date' },
{ group: 'errors', path: 'broken.png', root: 'output', detail: 'Not a valid image file' },
] });
await page.goto(`${workspace}#/images`);
await expect(pill(page)).toHaveText('Generating thumbnails · 40% · 4 / 10');
await page.goto(`${workspace}#/activity`);
await expect(items(page).first().locator('p')).toHaveText(/^Started \d\d:\d\d · 2 output folders · closing this view does not cancel it$/);
await expect(items(page).first().locator('progress')).toHaveAttribute('value', '40');
await openJob(page, job.id);
await expect(heading(page).locator('h1')).toHaveText('Thumbnail generation');
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel job']);
expect(await tileValues(page)).toEqual([['stat added', 'Generated', '1'], ['stat', 'Skipped', '1'], ['stat bad', 'Errors', '1']]);
await expect(outcomeButtons(page)).toHaveText(['Generated1', 'Errors1', 'Skipped1']);
await expect(fileRows(page).first().locator('.path')).toHaveText('sub/a.png');
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('button');
await outcomeButtons(page).filter({ hasText: 'Errors' }).click();
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('div');
await expect(scanLog(page).locator('summary')).toHaveText('● Job log — raw detail, click to show');
await finish(request, job.id);
await expect(toast(page)).toHaveText('Thumbnails finished · 1 generated · 1 error');
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d · 1 generated · 1 error$/);
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
});
test('a held legacy job and the health check show no tiles, no Results and no made-up percentage', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const response = await request.post('/__test__/jobs/hold',
{ data: { kind: 'thumbnails', title: 'Thumbnail clear (gallery page)', admission: 'guarded' } });
await expect(response).toBeOK();
const held = (await response.json()).job;
await page.goto(`${workspace}#/images`);
// D112: the pill names the held operation and shows no percentage; its bar is indeterminate, not an empty fill
// (S5 review M5), and stays hidden from assistive tech, which reads the label.
await expect(pill(page)).toHaveText('Thumbnail clear (gallery page)');
await expect(pill(page)).toHaveAttribute('aria-label', 'Thumbnail clear (gallery page)');
await expect(pill(page).locator('.activity-pill-bar')).toHaveClass('activity-pill-bar indeterminate');
await expect(pill(page).locator('.activity-pill-bar')).toHaveAttribute('aria-hidden', 'true');
expect(await pill(page).locator('.activity-pill-bar i').evaluate(node => node.style.width)).toBe('');
await page.goto(`${workspace}#/activity`);
const item = items(page).filter({ has: page.getByRole('link', { name: /Thumbnail clear \(gallery page\)$/ }) });
await expect(item.locator('p')).toHaveText(/^Started \d\d:\d\d · closing this view does not cancel it$/);
await expect(item.locator('progress')).toHaveCount(1);
await expect(item.locator('progress')).not.toHaveAttribute('value');
await expect(item.getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
await item.getByRole('link').click();
await expect(heading(page).locator('h1')).toHaveText('Thumbnail clear (gallery page)');
await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value');
await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0);
await expect(resultsSection(page)).toHaveCount(0);
await expect(scanLog(page)).toHaveCount(1);
await finish(request, held.id);
await expect(toast(page)).toHaveText('Thumbnail clear (gallery page) finished.');
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d$/);
await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100');
// The health check (a read job of its own kind): its status line, no tiles, no Cancel.
const health = await startJob(request, { kind: 'health', total: 0 });
await openJob(page, health.id);
await expect(heading(page).locator('h1')).toHaveText('Health check');
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Checking the database and image links');
await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value');
await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0);
await expect(resultsSection(page)).toHaveCount(0);
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
await expect(pill(page)).toHaveText('Health check');
await finish(request, health.id);
await expect(toast(page)).toHaveText('Health check finished.');
});
// S5 review I3: a thumbnail job with nothing to generate ends on the server's final line ("No previews to generate · N
// already up to date"), never on a stale "Processing file 0 of 0" under the Completed chip.
test('a finished thumbnail job with nothing to do says so', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const job = await startJob(request, { ...THUMB_JOB, total: 0 });
await step(request, job.id, { total: 0, processed: 0 }); // the stale line an older server left
await openJob(page, job.id);
const line = progressSection(page).locator('.panel-heading > span.small');
await finish(request, job.id, 'finished', { status_line: 'No previews to generate · 24 already up to date' });
await expect(line).toHaveText('No previews to generate · 24 already up to date');
await expect(page.getByText('Processing file 0 of 0')).toHaveCount(0);
await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100');
});
+556
View File
@@ -0,0 +1,556 @@
// Needs review (#249 PR F, plan Tasks F1 and F2, Mock inventory M4): the SAVED VIEWS row and its count, the Metadata
// review page over the real review routes, and Activity's Review metadata. Items are recorded by the real recorder
// (tests/browser/workspace_review_fixture.py) through the monitor's entry point; the conflict item's file was linked
// to another prompt after it was queued. An open page re-reads its queue from the review change stream (F2).
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
const review = '/prompt_manager/workspace#/review';
const main = page => page.locator('#workspace-main');
const savedNav = page => page.getByRole('navigation', { name: 'Saved views', exact: true });
const reviewRow = page => savedNav(page).locator('a[data-route="review"]');
const panel = page => main(page).getByRole('region', { name: 'Resolve the association', exact: true });
const context = page => panel(page).getByRole('combobox', { name: 'Sampler context', exact: true });
const cards = page => main(page).locator('.review-queue article.media-card');
const toast = page => page.locator('#workspace-toast');
const NOTICE = 'Multiple sampler paths were found. The existing image links were preserved. Choose the correct context or leave them unresolved.';
const EMPTY = 'Ambiguous images are added when a metadata scan or a new generation finds more than one sampler path.';
const OPTIONS = ['Sampler 11 · Golden dunes at dusk / Soft negative, blur', 'Sampler 13 · Harbor lights at night / Soft negative, blur',
'Leave unresolved'];
async function seed(request, items = 3, special = {}) {
const response = await request.post('/__test__/review/reset', { data: { items, ...special } });
await expect(response).toBeOK();
return response.json();
}
async function pending(request) {
return (await (await request.get('/prompt_manager/review/items?limit=1')).json()).total;
}
const links = async request => (await (await request.get('/__test__/review/links')).json()).links;
const streams = async request => (await (await request.get('/__test__/review/streams')).json());
const apply = page => panel(page).getByRole('button', { name: 'Apply association', exact: true });
const warning = page => panel(page).locator('.warning-note');
/** The page with its first item's sampler paths loaded. */
async function open(page) {
await page.goto(review);
await expect(context(page)).toBeEnabled();
}
test.beforeEach(async ({ page, browserName }) => {
await startCoverage(page, browserName);
page.reviewErrors = [];
page.on('pageerror', error => page.reviewErrors.push(error.message));
});
test.afterEach(async ({ page, browserName, request }) => {
await saveCoverage(page, browserName);
expect(page.reviewErrors).toEqual([]);
await request.post('/__test__/jobs/restart');
});
test('queue shows persisted items and the nav count', async ({ page, request }) => {
await seed(request, 3);
await open(page);
// SAVED VIEWS: Favorites, then Needs review with the exact pending count (ruling R5).
await expect(savedNav(page).locator('a').nth(1)).toHaveAttribute('data-route', 'review');
await expect(reviewRow(page)).toHaveText('Needs review3');
await expect(reviewRow(page)).toHaveAttribute('aria-current', 'page');
await expect(page).toHaveTitle('Review · Prompt Manager');
await expect(page.locator('#workspace-breadcrumb')).toHaveText('Review');
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Metadata review');
await expect(main(page).locator('.heading p.muted')).toHaveText('Uncertain evidence stays visible until you decide.');
await expect(main(page).locator('.notice h2')).toHaveText('3 images need a decision');
await expect(main(page).locator('.notice p')).toHaveText(NOTICE);
await expect(cards(page)).toHaveCount(3);
await expect(cards(page).first()).toHaveClass(/selected/);
await expect(panel(page).locator('p.small')).toHaveText('review_0000.png');
await expect(context(page).locator('option')).toHaveText(OPTIONS);
await expect(context(page)).toHaveValue(/^[0-9a-f]{64}$/);
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Golden dunes at dusk');
await expect(panel(page).locator('.prompt-section.negative .prompt-text')).toHaveText('Soft negative, blur');
await context(page).selectOption({ label: OPTIONS[1] });
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Harbor lights at night');
await context(page).selectOption({ label: 'Leave unresolved' });
await expect(panel(page).locator('.prompt-section.positive .prompt-text')).toHaveText('Multiple possible prompts. Source could not be determined.');
// Info selects another item.
await cards(page).nth(1).getByRole('button', { name: 'Inspect review_0001.png', exact: true }).click();
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
await expect(cards(page).nth(1)).toHaveClass(/selected/);
await expect(context(page)).toBeEnabled();
// The card's image opens the shared viewer.
await cards(page).nth(1).getByRole('button', { name: 'Open review_0001.png', exact: true }).click();
await expect(page.locator('#viewer-dialog')).toBeVisible();
await page.keyboard.press('Escape');
await expect(page.locator('#viewer-dialog')).toHaveCount(0);
// Open activity.
await main(page).getByRole('button', { name: 'Open activity', exact: true }).click();
await expect(page).toHaveURL(/#\/activity$/);
});
test('apply candidate links the pair and the item leaves the queue and the count drops', async ({ page, request }) => {
await seed(request, 2);
await open(page);
await expect(reviewRow(page)).toHaveText('Needs review2');
const decided = page.waitForResponse(item => /\/review\/items\/[^/]+\/decision$/.test(item.url()));
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
expect((await decided).status()).toBe(200);
await expect(toast(page)).toHaveText('Association accepted.');
await expect(page).toHaveURL(/#\/images$/);
await expect(reviewRow(page)).toHaveText('Needs review1');
expect(await pending(request)).toBe(1);
// The first sampler path's pair is linked to that file, and only to it.
expect(await links(request)).toEqual([{ filename: 'review_0000.png', prompt: 'Golden dunes at dusk',
pair_id: expect.any(Number), positive: 'Golden dunes at dusk', negative: 'Soft negative, blur' }]);
});
test('conflict leaves links unchanged and explains', async ({ page, request }) => {
// Queued first, then linked to another prompt (before a scan or monitor pass could settle it).
await seed(request, 0, { conflict: true });
const before = await links(request);
expect(before).toEqual([{ filename: 'conflict.png', prompt: 'Misty forest path', pair_id: null, positive: null,
negative: null }]);
await open(page);
await expect(panel(page).locator('p.small')).toHaveText('conflict.png');
await expect(context(page).locator('option')).toHaveText(OPTIONS);
const decided = page.waitForResponse(item => /\/decision$/.test(item.url()));
await apply(page).click();
const answer = await decided;
expect(answer.status()).toBe(409);
expect((await answer.json()).code).toBe('association-conflict');
await expect(warning(page)).toHaveText('This image is already linked to another prompt or negative, so this sampler path cannot be attached safely. The existing links were preserved.');
await expect(page).toHaveURL(/#\/review$/);
await expect(context(page)).toBeEnabled();
await expect(context(page)).toHaveValue(/^[0-9a-f]{64}$/);
expect(await links(request)).toEqual(before);
expect(await pending(request)).toBe(1);
await expect(reviewRow(page)).toHaveText('Needs review1');
// The file can still be left unresolved, which links nothing either.
await context(page).selectOption({ label: 'Leave unresolved' });
await apply(page).click();
await expect(toast(page)).toHaveText('Left unresolved.');
expect(await links(request)).toEqual(before);
expect(await pending(request)).toBe(0);
});
test('a limited item and an unavailable file can only be left unresolved', async ({ page, request }) => {
await seed(request, 0, { limited: true, unavailable: true });
await open(page);
await expect(cards(page)).toHaveCount(2);
await expect(panel(page).locator('p.small')).toHaveText('limited.png');
await expect(context(page).locator('option')).toHaveText(['Leave unresolved']);
await expect(warning(page)).toHaveText('This file has too many sampler paths to list. You can leave it unresolved.');
// The unavailable file: no current root lists it, so its media and Tag are inert.
const unavailable = cards(page).nth(1);
await expect(unavailable.locator('.image-open')).toHaveAttribute('aria-disabled', 'true');
await expect(unavailable.locator('.image-card-actions button').last()).toHaveAttribute('aria-disabled', 'true');
// F3: its empty media button keeps the concept's 1.33 media area, so the card lines up with its neighbours.
await expect(cards(page).first().locator('.image-open img')).toBeVisible();
const sizes = await cards(page).evaluateAll(nodes => nodes.map(node => {
const art = node.querySelector('.image-open').getBoundingClientRect();
return [Math.round(art.width), Math.round(art.height), Math.round(node.getBoundingClientRect().height)];
}));
expect(sizes[1]).toEqual(sizes[0]);
await unavailable.getByRole('button', { name: 'Inspect unavailable.png', exact: true }).click();
await expect(panel(page).locator('p.small')).toHaveText('unavailable.png');
await expect(warning(page)).toHaveText('This file is not under a current image source, so it cannot be opened. You can leave it unresolved.');
await expect(panel(page).getByRole('button', { name: 'Tag image', exact: true })).toHaveAttribute('aria-disabled', 'true');
const observes = [];
page.on('request', item => { if (item.url().endsWith('/image-annotations/observe')) observes.push(item.url()); });
await apply(page).click();
await expect(toast(page)).toHaveText('Left unresolved.');
expect(observes).toEqual([]);
expect(await pending(request)).toBe(1);
});
test('leave unresolved removes the item from pending', async ({ page, request }) => {
await seed(request, 2);
await open(page);
await context(page).selectOption({ label: 'Leave unresolved' });
await panel(page).getByRole('button', { name: 'Apply association', exact: true }).click();
await expect(toast(page)).toHaveText('Left unresolved.');
await expect(page).toHaveURL(/#\/images$/);
expect(await pending(request)).toBe(1);
});
test('leave for later writes nothing and goes to Images', async ({ page, request }) => {
await seed(request, 2);
await open(page);
const writes = [];
page.on('request', item => { if (item.method() !== 'GET') writes.push(item.url()); });
await panel(page).getByRole('button', { name: 'Leave for later', exact: true }).click();
await expect(page).toHaveURL(/#\/images$/);
expect(writes).toEqual([]);
expect(await pending(request)).toBe(2);
});
test('tag image from review saves tags and the item remains pending', async ({ page, request }) => {
await seed(request, 1);
await open(page);
const before = await links(request);
await panel(page).getByRole('button', { name: 'Tag image', exact: true }).click();
const dialog = page.getByRole('dialog', { name: 'Tag image', exact: true });
await expect(dialog).toBeVisible();
await expect(dialog.locator('p.tag-scope-note')).toHaveText('Tags belong to this image. Resolve its prompt association separately in Needs review.');
await dialog.getByRole('textbox', { name: 'Image tags · edit manually', exact: true }).fill('dunes');
await dialog.getByRole('button', { name: 'Save manual tags', exact: true }).click();
await expect(toast(page)).toHaveText('1 tags saved for review_0000.png. Prompt association still awaits review.');
await expect(dialog).toHaveCount(0);
expect(await pending(request)).toBe(1);
await expect(page).toHaveURL(/#\/review$/);
// The item is untouched: still listed, still open, nothing linked.
await expect(cards(page)).toHaveCount(1);
await expect(context(page)).toBeEnabled();
expect(await links(request)).toEqual(before);
const [item] = (await (await request.get('/prompt_manager/review/items')).json()).items;
expect([item.state, item.revision]).toEqual(['pending', 0]);
// The card's Tag opens the same dialog.
await cards(page).first().getByRole('button', { name: 'Tag or auto-tag review_0000.png', exact: true }).click();
await expect(dialog).toBeVisible();
await expect(dialog.locator('p.tag-scope-note')).toBeVisible();
await page.keyboard.press('Escape');
await expect(dialog).toHaveCount(0);
});
test('an empty queue shows the empty notice only', async ({ page, request }) => {
await seed(request, 0);
await page.goto(review);
await expect(main(page).locator('.notice h2')).toHaveText('No images need a decision');
await expect(main(page).locator('.notice p')).toHaveText(EMPTY);
await expect(main(page).locator('.gallery-layout')).toHaveCount(0);
await expect(reviewRow(page)).toHaveText('Needs review0');
});
test('load more appends the next page', async ({ page, request }) => {
await seed(request, 45);
await open(page);
await expect(cards(page)).toHaveCount(40);
const continuation = main(page).locator('.infinite-continuation');
await expect(continuation.locator('p')).toHaveText('40 of 45 images');
await continuation.getByRole('button', { name: 'Load more images', exact: true }).click();
await expect(cards(page)).toHaveCount(45);
await expect(continuation).toHaveCount(0);
});
test('a scan job with review items offers Review metadata in Activity', async ({ page, request }) => {
await seed(request, 1);
const started = await (await request.post('/__test__/jobs/start', { data: { total: 10, review_items: 2 } })).json();
await expect(await request.post(`/__test__/jobs/${started.job.id}/finish`, { data: {} })).toBeOK();
const plain = await (await request.post('/__test__/jobs/start', { data: { total: 10 } })).json();
await expect(await request.post(`/__test__/jobs/${plain.job.id}/finish`, { data: {} })).toBeOK();
await page.goto('/prompt_manager/workspace#/activity');
const item = main(page).locator(`.activity-item[data-job-id="${started.job.id}"]`);
await expect(item.getByRole('button', { name: 'Review metadata', exact: true })).toBeVisible();
await expect(main(page).locator(`.activity-item[data-job-id="${plain.job.id}"] button`)).toHaveCount(0);
await page.goto(`/prompt_manager/workspace#/activity/${started.job.id}`);
const actions = main(page).locator('.heading-actions button');
await expect(actions).toHaveText(['Download report', 'Review metadata']);
await actions.nth(1).click();
await expect(page).toHaveURL(/#\/review$/);
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Metadata review');
});
test('a failed item read says so and Retry reads it again', async ({ page, request }) => {
await seed(request, 1);
await page.route(url => /\/prompt_manager\/review\/items\/[0-9a-f-]{36}$/.test(url.pathname),
route => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, code: 'review-failed', error: 'x' }) }),
{ times: 1 });
await page.goto(review);
await expect(panel(page).locator('.warning-note')).toHaveText('This item could not be read. Select it again to retry.');
await expect(panel(page).getByText('Could not read prompts from this file.', { exact: true })).toBeVisible();
await expect(panel(page).getByRole('button', { name: 'Apply association', exact: true })).toHaveAttribute('aria-disabled', 'true');
await panel(page).getByRole('button', { name: 'Retry prompt details', exact: true }).click();
await expect(context(page)).toBeEnabled();
await expect(context(page).locator('option')).toHaveText(OPTIONS);
await expect(panel(page).locator('.warning-note')).toBeHidden();
});
test('a failed queue read offers Retry', async ({ page, request }) => {
await seed(request, 1);
await page.route(url => url.pathname.endsWith('/prompt_manager/review/items') && url.searchParams.get('limit') === '40',
route => route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, code: 'review-failed', error: 'x' }) }),
{ times: 1 });
await page.goto(review);
await expect(main(page).getByRole('alert')).toHaveText('The review queue could not be read.');
await main(page).getByRole('button', { name: 'Retry', exact: true }).click();
await expect(context(page)).toBeEnabled();
await expect(main(page).locator('.notice h2')).toHaveText('1 image needs a decision');
});
test('keyboard focus labels and touch targets', async ({ page, request }) => {
await seed(request, 2);
await open(page);
const first = cards(page).first();
for (const [name, role] of [['Open review_0000.png', 'button'], ['Inspect review_0000.png', 'button'],
['Tag or auto-tag review_0000.png', 'button']]) await expect(first.getByRole(role, { name, exact: true })).toBeVisible();
await expect(main(page).getByRole('list', { name: 'Review queue', exact: true })).toBeVisible();
await expect(panel(page)).toBeVisible();
// Keyboard: Info on the second card opens it, and the panel's controls take focus in their order.
const info = cards(page).nth(1).getByRole('button', { name: 'Inspect review_0001.png', exact: true });
await info.focus();
await page.keyboard.press('Enter');
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
await expect(context(page)).toBeEnabled();
await context(page).focus();
await expect(context(page)).toBeFocused();
const order = await panel(page).evaluate(node => [...node.querySelectorAll('button, select')]
.filter(control => control.offsetParent !== null).map(control => control.getAttribute('aria-label') || control.textContent.trim()));
expect(order[0]).toBe('Tag image');
expect(order.slice(-2)).toEqual(['Leave for later', 'Apply association']);
// Targets: the panel's controls are 40px on a coarse pointer (the concept's coarse buttons) and 24px otherwise (the
// WCAG 2.2 minimum); the card's buttons keep the concept's compact size, never under 24px (its ≤700px rule).
const coarse = await page.evaluate(() => matchMedia('(pointer: coarse)').matches);
const sized = [
[panel(page).getByRole('button', { name: 'Tag image', exact: true }), coarse ? 40 : 24],
[context(page), coarse ? 40 : 24],
[panel(page).getByRole('button', { name: 'Leave for later', exact: true }), coarse ? 40 : 24],
[apply(page), coarse ? 40 : 24],
[first.getByRole('button', { name: 'Inspect review_0000.png', exact: true }), 24],
[first.getByRole('button', { name: 'Tag or auto-tag review_0000.png', exact: true }), 24],
];
for (const [control, minimum] of sized) {
const box = await control.boundingBox();
const label = await control.evaluate(node => node.getAttribute('aria-label') || node.textContent.trim().slice(0, 40));
expect(box.height, label).toBeGreaterThanOrEqual(minimum);
expect(box.width, label).toBeGreaterThanOrEqual(24);
}
// Enter on Leave for later returns to Images.
await panel(page).getByRole('button', { name: 'Leave for later', exact: true }).focus();
await page.keyboard.press('Enter');
await expect(page).toHaveURL(/#\/images$/);
});
test('an open page shows items the monitor adds and keeps the open choice', async ({ page, request }) => {
await seed(request, 1);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await context(page).selectOption({ label: OPTIONS[1] });
await expect(reviewRow(page)).toHaveText('Needs review1');
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(2);
await expect(main(page).locator('.notice h2')).toHaveText('2 images need a decision');
await expect(reviewRow(page)).toHaveText('Needs review2');
await expect(cards(page).first()).toHaveClass(/selected/);
await expect(panel(page).locator('p.small')).toHaveText('review_0000.png');
await expect(context(page).locator('option:checked')).toHaveText(OPTIONS[1]);
await expect(warning(page)).toBeHidden();
});
test('an open page shows items a metadata scan adds', async ({ page, request }) => {
await seed(request, 1);
await expect(await request.post('/__test__/review/add', { data: { kind: 'file' } })).toBeOK();
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await expect(cards(page)).toHaveCount(1);
const started = await request.post('/prompt_manager/jobs/scan', { data: { scope: 'new' } });
expect(started.status()).toBe(202);
await expect(cards(page)).toHaveCount(2);
await expect(cards(page).nth(1).getByRole('button', { name: 'Inspect added_0000.png', exact: true })).toBeVisible();
await expect(reviewRow(page)).toHaveText('Needs review2');
});
test('a decision in another tab leaves the open page and opens the next image', async ({ page, request, context: browserContext }) => {
await seed(request, 2);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
const other = await browserContext.newPage();
await open(other);
await context(other).selectOption({ label: 'Leave unresolved' });
await apply(other).click();
await expect(other).toHaveURL(/#\/images$/);
await other.close();
await expect(cards(page)).toHaveCount(1);
await expect(panel(page).locator('p.small')).toHaveText('review_0001.png');
await expect(warning(page)).toHaveText('review_0000.png was decided elsewhere, so the next image is open.');
await expect(main(page).locator('.notice h2')).toHaveText('1 image needs a decision');
await expect(reviewRow(page)).toHaveText('Needs review1');
await expect(context(page)).toBeEnabled();
});
/** The queue reads (`?limit=40`, cursors) and count reads (`?limit=1`) the page sends from now on. */
function reviewReads(page) {
const reads = [];
page.on('request', item => {
const url = new URL(item.url());
if (url.pathname === '/prompt_manager/review/items') reads.push(url.searchParams.get('cursor') ? 'more' : url.search);
});
return reads;
}
test('a visit reads its queue once, and a change reads it once more', async ({ page, request }) => {
// F3 ruling: the stream's ready must not read a queue the first read already holds.
await seed(request, 1);
const reads = reviewReads(page);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(2);
await expect(reviewRow(page)).toHaveText('Needs review2');
// The page's first read and the change's re-read; the shell's count read at start, and the row follows the page.
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
expect(reads.filter(search => search === '?limit=1')).toHaveLength(1);
});
test('a ready whose version cannot be read re-reads the queue once', async ({ page, request }) => {
// The stream's ready names no readable version (an older server, a proxy that rewrote it): the page reads again.
await seed(request, 1);
const reads = reviewReads(page);
// M1: `retry` keeps the browser from reconnecting the ended stream during the test, so exactly one re-read follows.
await page.route(url => url.pathname === '/prompt_manager/review/events', route => route.fulfill({
status: 200, headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-store' },
body: 'retry: 3600000\nevent: ready\ndata: not json\n\n' }));
let reread = 0;
const second = page.waitForResponse(item => item.url().endsWith('/prompt_manager/review/items?limit=40') && ++reread === 2);
await open(page);
await second;
await expect(cards(page)).toHaveCount(1);
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
});
const live = page => main(page).locator('.heading .review-live');
const NOT_LIVE = 'Live updates are paused: changes made elsewhere appear once the connection returns.';
test('a stream the browser gives up on is opened again, and the page says it is not live meanwhile', async ({ page, request }) => {
// I3: a 503 (shutdown, or a proxy while the backend restarts) fails an EventSource for good.
await seed(request, 1);
let attempts = 0;
await page.route(url => url.pathname === '/prompt_manager/review/events', route => {
attempts += 1;
if (attempts === 1) return route.fulfill({ status: 503, contentType: 'application/json', body: '{"success":false}' });
return route.fallback();
});
await open(page);
await expect(live(page)).toHaveText(NOT_LIVE);
await expect(live(page)).toBeVisible();
// The page opens the stream again (after 3 s) and is live once it sends ready.
await expect.poll(async () => (await streams(request)).streams, { timeout: 15000 }).toBe(1);
await expect(live(page)).toBeHidden();
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(2);
expect(attempts).toBe(2);
});
/** Make the page hidden or visible as a tab switch does. */
async function setVisible(page, visible) {
await page.evaluate(shown => {
Object.defineProperty(document, 'hidden', { configurable: true, get: () => !shown });
Object.defineProperty(document, 'visibilityState', { configurable: true, get: () => (shown ? 'visible' : 'hidden') });
document.dispatchEvent(new Event('visibilitychange'));
}, visible);
}
test('a hidden tab closes its stream and a visible one reads only what changed meanwhile', async ({ page, request }) => {
// I4: background tabs must not hold one of the origin's six connections.
await seed(request, 1);
const reads = reviewReads(page);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await setVisible(page, false);
// The server drops a closed stream at its next write: this monitor addition is one.
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect.poll(async () => (await streams(request)).streams).toBe(0);
await expect(cards(page)).toHaveCount(1);
expect(reads.filter(search => search === '?limit=40')).toHaveLength(1);
// Visible again: the reopened stream's ready names a newer version, so the page reads once.
const reread = page.waitForResponse(item => item.url().endsWith('/prompt_manager/review/items?limit=40'));
await setVisible(page, true);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await reread;
await expect(cards(page)).toHaveCount(2);
await expect(live(page)).toBeHidden();
expect(reads.filter(search => search === '?limit=40')).toHaveLength(2);
});
test('a re-read refreshes a card whose media or target changed', async ({ page, request }) => {
// M6 and M10: a root came or went between reads; the card follows, and a known thumbnail is its media.
await seed(request, 2);
let rewrite = false;
await page.route(url => url.pathname === '/prompt_manager/review/items' && url.search === '?limit=40', async route => {
if (!rewrite) return route.fallback();
const response = await route.fetch();
const body = await response.json();
body.items[0] = { ...body.items[0], thumbnail_url: `${body.items[0].url}&thumb=1` };
body.items[1] = { ...body.items[1], url: null, thumbnail_url: null, target: null, unavailable: true };
return route.fulfill({ response, json: body });
});
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
const second = cards(page).nth(1);
await expect(second.getByRole('button', { name: 'Open review_0001.png', exact: true })).not.toHaveAttribute('aria-disabled', 'true');
rewrite = true;
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(cards(page)).toHaveCount(3);
await expect(cards(page).first().locator('img')).toHaveAttribute('src', /&thumb=1$/);
await expect(second.getByRole('button', { name: 'Open review_0001.png', exact: true })).toHaveAttribute('aria-disabled', 'true');
await expect(second.locator('img')).toHaveCount(0);
await expect(second.locator('.image-card-actions button').last()).toHaveAttribute('aria-disabled', 'true');
await expect(cards(page).first()).toHaveClass(/selected/);
});
test('leaving during the observe releases the capture it answers', async ({ page, request }) => {
// M4: the server made the capture; the page left before the answer came.
await seed(request, 1);
await open(page);
let held;
const sent = new Promise(resolve => { held = resolve; });
await page.route(url => url.pathname === '/prompt_manager/image-annotations/observe', async route => {
const response = await route.fetch();
held(() => route.fulfill({ response }));
});
await apply(page).click();
const answer = await sent;
expect((await streams(request)).captures).toBe(1);
await page.evaluate(() => { window.location.hash = '#/tags'; });
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Tags & categories');
await answer();
await expect.poll(async () => (await streams(request)).captures).toBe(0);
});
test('a live re-read keeps the loaded depth and the scroll position', async ({ page, request }) => {
// F3 ruling: a queue change re-reads as many items as Load more had shown, and the page does not jump.
await seed(request, 45);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
await main(page).getByRole('button', { name: 'Load more images', exact: true }).click();
await expect(cards(page)).toHaveCount(45);
await cards(page).last().scrollIntoViewIfNeeded();
const scrollTop = () => cards(page).first().evaluate(node => {
for (let el = node.parentElement; el; el = el.parentElement) {
const style = getComputedStyle(el);
if (/(auto|scroll)/.test(style.overflowY) && el.scrollHeight > el.clientHeight) return el.scrollTop;
}
return document.scrollingElement.scrollTop;
});
const before = await scrollTop();
expect(before).toBeGreaterThan(0);
const reads = reviewReads(page);
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect(main(page).locator('.notice h2')).toHaveText('46 images need a decision');
await expect(cards(page)).toHaveCount(46);
expect(await scrollTop()).toBe(before);
expect(reads).toEqual(['?limit=40', 'more']);
await expect(main(page).locator('.infinite-continuation')).toHaveCount(0);
});
test('teardown releases captures listeners and requests', async ({ page, request }) => {
await seed(request, 1);
await open(page);
await expect.poll(async () => (await streams(request)).streams).toBe(1);
const inFlight = new Set();
page.on('request', item => { if (item.url().includes('/prompt_manager/review/')) inFlight.add(item); });
page.on('requestfinished', item => inFlight.delete(item));
page.on('requestfailed', item => inFlight.delete(item));
// A decision whose answer never comes: leaving the page must still release its capture.
let held;
const sent = new Promise(resolve => { held = resolve; });
await page.route(url => /\/decision$/.test(url.pathname), route => held(route));
await apply(page).click();
const route = await sent;
expect((await streams(request)).captures).toBe(1);
await page.evaluate(() => { window.location.hash = '#/tags'; });
await expect(main(page).getByRole('heading', { level: 1 })).toHaveText('Tags & categories');
await expect.poll(async () => (await streams(request)).captures).toBe(0);
await route.abort().catch(() => {});
// The server drops a closed stream at its next write: a monitor addition is one.
await expect(await request.post('/__test__/review/add', { data: { kind: 'monitor' } })).toBeOK();
await expect.poll(async () => (await streams(request)).streams).toBe(0);
await expect.poll(() => inFlight.size).toBe(0);
});
+1 -1
View File
@@ -372,7 +372,7 @@ test('Favorites is the first SAVED VIEWS row, with a real count that follows eve
expect(await favoritesRow(page).evaluate(link => [...link.childNodes].map(node => node.nodeType === Node.TEXT_NODE
? `text:${node.textContent}` : `${node.localName}.${node.getAttribute('class')}`)))
.toEqual(['svg.icon', 'text:Favorites', 'span.nav-count']);
expect(await savedNav(page).locator('a').evaluateAll(links => links.map(link => link.dataset.route))).toEqual(['favorites']);
expect(await savedNav(page).locator('a').evaluateAll(links => links.map(link => link.dataset.route))).toEqual(['favorites', 'review']);
await star(page, '1:shared/same.png');
await expect(toast(page)).toHaveText('Added to favorites.');
@@ -0,0 +1,536 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Sources & scans (#249 System S4 Task 11) on the concept-shaped real fixture, whose two sources are this
// server's `output` and `input` folders: the source list, monitoring and worker rows read GET /settings (S4 Task 9),
// Add/Remove edit the host draft behind the one Save row, and every scan starts the session job through the shared
// "Review metadata scan" (D55), scoped to one source from its row (S4 Task 10). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const SETTINGS_URL = '/prompt_manager/settings';
const NOTICE_KEY = 'pm_update_notice_dismissed';
const OUTSIDE_REASON = 'Gallery root must be inside a ComfyUI directory (output, input, temp, user) or one listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS';
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/jobs/restart');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
const toast = page => page.locator('#workspace-toast');
const layoutOf = page => page.locator('#workspace-main .settings-layout');
const sourceList = page => layoutOf(page).locator('#source-list');
const statusOf = page => layoutOf(page).locator('#settings-status');
const saveButton = page => layoutOf(page).getByRole('button', { name: 'Save preferences', exact: true });
const dialog = page => page.locator('dialog#modal[open]');
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
/** Open Sources with the update notice dismissed (or shown, `notice: true`) once the source list has loaded. */
async function openSources(page, { notice = false } = {}) {
if (!notice) await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layoutOf(page).locator('#workers')).toBeEnabled();
return layoutOf(page);
}
/** The source list's rows: `[h3, p, [buttons]]`, then its note line when there is one. */
function rowsOf(page) {
return sourceList(page).evaluate(list => [...list.children].map(row => row.matches('.operation')
? [row.querySelector('h3').textContent, row.querySelector('p').textContent,
[...row.querySelectorAll('.row-actions > button')].map(button => button.textContent)]
: [row.localName, row.className, row.textContent]));
}
const sourceRow = (page, title) => sourceList(page).locator('.operation').filter({ has: page.getByRole('heading', { name: title, exact: true }) });
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed) without scanning', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const layout = await openSources(page, { notice: true });
const notice = layout.locator(':scope > .notice');
await expect(notice.locator('h2')).toHaveText('Bring older images into your prompt history');
await expect(notice.locator('p')).toHaveText('The schema upgrades automatically. An optional metadata scan links existing positives and negatives without regenerating all thumbnails.');
await expect(notice.locator('.row-actions > button')).toHaveText(['Later', 'Review scan']);
await notice.getByRole('button', { name: 'Later', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await expect(dialog(page)).toHaveCount(0);
// The dismissal is the legacy admin page's own, so neither page shows the notice again.
await page.reload();
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(writes).toEqual([]);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Review scan opens the shared Review metadata scan with all sources', async ({ page, request }) => {
const layout = await openSources(page, { notice: true });
// The notice's Review scan dismisses it, as the legacy page's does, and reviews every source.
await layout.locator(':scope > .notice').getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
// Enrich existing metadata is the same review.
const enrich = layout.locator('.operation').filter({ hasText: 'Enrich existing metadata' });
await expect(enrich.locator('p')).toHaveText('Resolve both prompts, deduplicate recipes, preserve uncertain links.');
await enrich.getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Scan changes opens it with that one source ("1 output folder") and starts a scoped job that opens in Activity', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['input'] } })).toBeOK();
const bodies = [];
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan')) bodies.push(item.postDataJSON()); });
await openSources(page);
const input = sourceRow(page, 'ComfyUI input');
await expect(input.locator('p')).toHaveText('input · watching');
await input.getByRole('button', { name: 'Scan changes', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['1 output folder', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
// The toast says it started (or, for a scan this quick, that it already finished); Activity shows its progress.
await expect(toast(page)).toHaveText(/^Scan (started\.|finished · )/);
expect(bodies).toEqual([{ scope: 'new', roots: [1], paths: ['input'] }]);
const id = page.url().split('/').pop();
// The job walks only that source (the count is recorded once the walk has listed it).
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope)
.toMatchObject({ files: 'new', roots: [1], output_folders: 1 });
await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata');
});
test('Add source validates on the server, then marks the host draft dirty; Save persists', async ({ page, request }) => {
// A folder inside ComfyUI's output folder that is not a source yet.
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK();
const layout = await openSources(page);
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
]);
await expect(layout.locator('.panel-heading > h2')).toHaveText('Image sources');
const add = layout.locator('.panel-heading').getByRole('button', { name: '+ Add source', exact: true });
await add.click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Add image source');
await expect(dialog(page).locator('label[for="action-input"]')).toHaveText('Folder path');
await expect(dialog(page).locator('#action-input')).toHaveValue('');
await expect(dialog(page).locator('.dialog-actions > button')).toHaveText(['Cancel', 'Add']);
await dialog(page).locator('#action-input').fill('output/unthumbnailed');
const validated = page.waitForResponse(response => response.url().endsWith('/prompt_manager/settings/validate-source'));
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
expect(await (await validated).json()).toEqual({ success: true, valid: true, label: 'unthumbnailed', path: 'output/unthumbnailed', code: null, reason: null });
await expect(dialog(page)).toHaveCount(0);
await expect(add).toBeFocused();
// Validation wrote nothing: the draft is dirty and the server still has two sources.
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
['unthumbnailed', 'output/unthumbnailed · not saved yet', ['Remove']],
]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.effective).toHaveLength(2);
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const body = (await saved).postDataJSON();
expect(body.gallery_root_paths).toEqual(['output', 'input', 'output/unthumbnailed']);
expect(typeof body.expected_revision).toBe('string');
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// The list is the server's again: three effective sources, each scannable.
await expect(sourceRow(page, 'unthumbnailed').locator('p')).toHaveText('output/unthumbnailed · manual scan');
await expect(sourceRow(page, 'unthumbnailed').getByRole('button', { name: 'Scan folder', exact: true })).toBeVisible();
expect((await serverJson(request, SETTINGS_URL)).sources.effective.map(source => source.path)).toEqual(['output', 'input', 'output/unthumbnailed']);
});
test('invalid source shows the server reason', async ({ page, request }) => {
const layout = await openSources(page);
await layout.getByRole('button', { name: '+ Add source', exact: true }).click();
const field = dialog(page).locator('#action-input');
for (const [path, reason, submit] of [
['output/not-there', 'Gallery root does not exist', 'click'],
// Enter in the field adds too (M6).
['/etc', OUTSIDE_REASON, 'Enter'],
]) {
await field.fill(path);
if (submit === 'Enter') await field.press('Enter');
else await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText(reason);
await expect(field).toHaveAttribute('aria-invalid', 'true');
await expect(field).toHaveValue(path);
}
// A folder already listed (the server's answer names it) is not added twice.
await field.fill('input');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('This folder is already a source.');
// A check that cannot finish says so and adds nothing.
await page.route('**/prompt_manager/settings/validate-source', route => route.fulfill({ status: 500,
json: { success: false, code: null, error: 'Failed to check the source' } }));
await field.fill('output/not-checked');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('The folder could not be checked. Try again.');
await page.unroute('**/prompt_manager/settings/validate-source');
// Typing again clears the reason.
await field.fill('output/unthumbnailed');
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveCount(0);
await expect(field).not.toHaveAttribute('aria-invalid', 'true');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
});
test('Remove marks the draft; removing the last entry states auto-detect', async ({ page, request }) => {
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await sourceRow(page, 'ComfyUI output').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['p', 'small', 'No source is configured. After saving, ComfyUI’s output folder is used.']]);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// Saved: ComfyUI's output folder is detected and listed without Remove.
await expect.poll(rowsOf.bind(null, page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder']],
['p', 'small', 'No source is configured; ComfyUI’s output folder is used.'],
]);
const { sources } = await serverJson(request, SETTINGS_URL);
expect([sources.configured, sources.auto_detected]).toEqual([[], true]);
});
test('a configured source that does not exist says why and can be removed', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await openSources(page);
// The fixture's unmounted drive is not inside a ComfyUI folder, so the server says that (it never probes outside).
expect((await rowsOf(page)).at(-1)).toEqual(['unmounted-drive', `unmounted-drive · ${OUTSIDE_REASON}`, ['Remove']]);
await sourceRow(page, 'unmounted-drive').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output', 'input']);
});
test('watching only with a live observer; restart pending otherwise', async ({ page, request }) => {
// No image monitor in this process: every source is scanned by hand.
await openSources(page);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · manual scan', 'input · manual scan']);
// A live observer that scheduled output: output is watched; input was added since it started.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · restart pending']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan folder']);
// An observer that died watches nothing.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'], alive: false } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · restart pending', 'input · restart pending']);
});
test('Watch for new outputs saves monitoring_enabled; a restart applies it', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'] } })).toBeOK();
const layout = await openSources(page);
const monitoring = layout.getByRole('switch', { name: 'Watch for new outputs', exact: true });
await expect(monitoring).toBeChecked();
await expect(layout.locator('.setting-row').filter({ has: page.locator('#monitoring') }).locator('p'))
.toHaveText('New files enter the same library. Existing scans do not count as new runs.');
await monitoring.uncheck();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await (await saved).json();
expect(answer.restart_required_fields).toEqual(['monitoring_enabled']);
// D128: saved for the next start, and the status says a restart applies it.
await expect(statusOf(page)).toHaveText('Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The running observer keeps watching until ComfyUI restarts, and the list says so.
const { settings, monitor } = await serverJson(request, SETTINGS_URL);
expect([settings.monitoring_enabled, monitor.observer_alive]).toEqual([false, true]);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching until restart', 'input · watching until restart']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan changes']);
// Switched back on before the restart: the observer still runs, so nothing waits for one.
await monitoring.check();
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · watching']);
});
test('worker range max is cpu_count', async ({ page, request }) => {
const { settings } = await serverJson(request, SETTINGS_URL);
const layout = await openSources(page);
const workers = layout.locator('#workers');
await expect(layout.locator('label[for="workers"]')).toHaveText('Worker threads');
await expect(workers).toHaveAttribute('type', 'range');
await expect(workers).toHaveAttribute('min', '1');
await expect(workers).toHaveAttribute('max', String(settings.cpu_count));
await expect(workers).toHaveValue(String(settings.worker_threads));
const next = settings.worker_threads === 1 ? settings.cpu_count : 1;
await workers.fill(String(next));
await expect(statusOf(page)).toHaveText(next === settings.worker_threads ? 'No unsaved changes.' : 'Unsaved changes.');
test.skip(next === settings.worker_threads, 'One core: the range has a single value');
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
expect((await saved).postDataJSON().worker_threads).toBe(next);
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.worker_threads).toBe(next);
});
test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page }) => {
const layout = await openSources(page);
const refresh = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Refresh files', exact: true }) });
await expect(refresh.locator('p')).toHaveText('Read every source again to find files added or removed outside ComfyUI.');
const answered = page.waitForResponse(response => response.url().endsWith('/prompt_manager/gallery/rescan'));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
const { total, roots } = await (await answered).json();
expect(roots).toBe(2);
await expect(toast(page)).toHaveText(`Files refreshed. ${total.toLocaleString('en-US')} files in 2 output folders.`);
// A failed refresh says so.
await page.route('**/prompt_manager/gallery/rescan', route => route.fulfill({ status: 500, json: { success: false, error: 'busy disk' } }));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.');
});
const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan';
const THUMBNAIL_JOB_URL = '/prompt_manager/jobs/thumbnails';
const planCount = value => value.toLocaleString('en-US');
/** Open "Thumbnail maintenance" from Sources' Thumbnail generation → Review options. */
async function openThumbnailReview(page) {
const layout = await openSources(page);
const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.');
await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
return dialog(page);
}
/** The thumbnail requests this page sends: plan reads (their query) and job starts (their body). */
function thumbnailRequests(page) {
const seen = { plans: [], starts: [] };
page.on('request', item => {
const url = new URL(item.url());
if (url.pathname === THUMBNAIL_PLAN_URL) seen.plans.push(Object.fromEntries(url.searchParams));
if (item.method() === 'POST' && url.pathname === THUMBNAIL_JOB_URL) seen.starts.push(item.postDataJSON());
});
return seen;
}
const missingLine = plan => `${planCount(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ` +
`${planCount(plan.fresh)} already up to date.`;
test('Review options shows the plan counts for the chosen scope/quality/size before Start', async ({ page, request }) => {
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
// Image loading quality (Appearance, Q3) preselects the dialog's quality.
await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('gallerySettings', JSON.stringify({ imageQuality: 'high' })); });
const seen = thumbnailRequests(page);
const plan = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=missing&size=512&quality=high`);
expect(plan.targets).toBe(3);
const review = await openThumbnailReview(page);
await expect(review.locator('#modal-body > *')).toHaveCount(5);
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
await expect(review.locator('#thumb-mode option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)']);
await expect(review.locator('#thumb-quality option')).toHaveText(['Medium · optimized', 'High · original', 'Low · fastest']);
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
await expect(review.locator('#thumb-quality')).toHaveValue('high');
await expect(review.locator('#thumb-limit')).toHaveValue('512');
await expect(review.locator('#thumb-limit')).toHaveAttribute('min', '128');
await expect(review.locator('#thumb-limit')).toHaveAttribute('max', '2048');
const line = review.locator('#modal-body > p.small');
await expect(line).toHaveAttribute('role', 'status');
await expect(line).toHaveText(missingLine(plan));
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'high' }]);
await expect(review.locator('.dialog-actions > button')).toHaveText(['Cancel', 'Start']);
// Reviewing starts nothing; Cancel closes.
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(dialog(page)).toHaveCount(0);
expect(seen.starts).toEqual([]);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('size outside 128–2048 reports validity and requests nothing', async ({ page, request }) => {
const seen = thumbnailRequests(page);
const review = await openThumbnailReview(page);
const line = review.locator('#modal-body > p.small');
await expect(line).toHaveText(/ already up to date\.$/);
const size = review.locator('#thumb-limit');
for (const value of ['4096', '64', '']) {
await size.fill(value);
await expect(size).toHaveAttribute('aria-invalid', 'true');
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
expect(await size.evaluate(input => input.validity.valid)).toBe(false);
// Start keeps the dialog open and sends nothing; neither does a scope change.
await review.getByRole('button', { name: 'Start', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
await review.locator('#thumb-mode').selectOption('rebuild');
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
await review.locator('#thumb-mode').selectOption('missing');
}
expect(seen.starts).toEqual([]);
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }]);
await size.fill('1024');
await expect(size).not.toHaveAttribute('aria-invalid');
await expect(line).toHaveText(/ already up to date\.$/);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Start closes the dialog, toasts, and opens the job in Activity', async ({ page, request }) => {
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
const seen = thumbnailRequests(page);
const review = await openThumbnailReview(page);
await review.locator('#thumb-quality').selectOption('low');
await review.locator('#thumb-limit').fill('256');
await review.getByRole('button', { name: 'Start', exact: true }).click();
await expect(dialog(page)).toHaveCount(0);
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
// The toast says it started (or, for a run this quick, that it already finished); Activity shows its progress.
await expect(toast(page)).toHaveText(/^(Thumbnail generation started\.|Thumbnails finished · 2 generated · 0 errors)$/);
expect(seen.starts).toEqual([{ mode: 'missing', size: 256, quality: 'low' }]);
const id = page.url().split('/').pop();
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
expect(job).toMatchObject({ kind: 'thumbnails', title: 'Thumbnail generation', scope: { mode: 'missing', size: 256, quality: 'low' } });
await expect(page.locator('#workspace-main h1')).toHaveText('Thumbnail generation');
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
});
test('Rebuild all is explicit and the plan line says every preview is regenerated', async ({ page, request }) => {
const seen = thumbnailRequests(page);
const rebuild = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=rebuild&size=512&quality=medium`);
expect(rebuild.targets).toBeGreaterThan(0);
const review = await openThumbnailReview(page);
// Missing thumbnails only is the default; rebuilding is the explicit second option.
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
const line = review.locator('#modal-body > p.small');
await expect(line).toHaveText(/ already up to date\.$/);
await review.locator('#thumb-mode').selectOption('rebuild');
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }, { mode: 'rebuild', size: '512', quality: 'medium' }]);
// Back and forth reads each scope once.
await review.locator('#thumb-mode').selectOption('missing');
await expect(line).toHaveText(/ already up to date\.$/);
await review.locator('#thumb-mode').selectOption('rebuild');
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
expect(seen.plans).toHaveLength(2);
await review.getByRole('button', { name: 'Start', exact: true }).click();
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
expect(seen.starts).toEqual([{ mode: 'rebuild', size: 512, quality: 'medium' }]);
const id = page.url().split('/').pop();
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
});
test('a plan that cannot be read says so; Start still starts', async ({ page }) => {
await page.route(`**${THUMBNAIL_PLAN_URL}?*`, route => route.fulfill({ status: 500, json: { success: false, error: 'disk' } }));
const review = await openThumbnailReview(page);
await expect(review.locator('#modal-body > p.small')).toHaveText('The previews could not be counted.');
await expect(review.getByRole('button', { name: 'Start', exact: true })).not.toHaveAttribute('aria-disabled');
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
});
test('a refused start shows the busy link toast', async ({ page, request }) => {
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
const review = await openThumbnailReview(page);
await review.getByRole('button', { name: 'Start', exact: true }).click();
await expect(dialog(page)).toHaveCount(0);
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await expect(page).toHaveURL(`${workspace}#/system/sources`);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['scan']);
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
});
test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => {
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
// The draft is kept for the next Save, and the server still has both sources.
await expect(sourceList(page).locator('.operation')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
// An answer that does not name its job (an older server): the toast names the running one from the job list.
await page.route(`**${SETTINGS_URL}`, route => (route.request().method() === 'POST'
? route.fulfill({ status: 409, json: { success: false, code: 'busy', error: 'Scan for metadata already running' } })
: route.continue()));
await page.locator('#workspace-toast').evaluate(node => { node.textContent = ''; });
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await page.unroute(`**${SETTINGS_URL}`);
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
});
test('a source list that cannot be read says so and retries', async ({ page }) => {
await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
let failing = true;
await page.route(`**${SETTINGS_URL}`, route => (failing && route.request().method() === 'GET'
? route.fulfill({ status: 500, json: { success: false, error: 'Injected failure' } }) : route.continue()));
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).getByRole('alert')).toHaveText('Sources could not be loaded. Your saved data has not changed.');
await expect(sourceList(page).locator('.operation')).toHaveCount(0);
// The server settings rows say so too, and stay disabled.
await expect(layoutOf(page).locator('#workers')).toBeDisabled();
failing = false;
await sourceList(page).getByRole('button', { name: 'Retry', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
});
test('a scan from a list that changed since it was read starts nothing, says why and reads the list again', async ({ page, request }) => {
await openSources(page);
// The legacy page (or another tab) removes ComfyUI output: the page's "input" is index 1, now past the end.
await expect(await request.post(SETTINGS_URL, { data: { gallery_root_paths: ['input'] } })).toBeOK();
const refused = page.waitForResponse(response => response.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
const answer = await refused;
expect([answer.status(), (await answer.json()).code]).toEqual([409, 'sources-changed']);
await expect(toast(page)).toHaveText('The sources changed since they were loaded. Reload them.');
// No scan started (the other save ran as its own source-change hold).
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['source_change']);
// Read again: the one source is index 0 now, and its scan starts.
await expect.poll(rowsOf.bind(null, page)).toEqual([['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('No unsaved changes.');
const started = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
expect((await started).postDataJSON()).toEqual({ scope: 'new', roots: [0], paths: ['input'] });
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
});
test('a save refused for one unusable source names it and marks its row', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source', { data: { inside: true } })).toBeOK();
await openSources(page);
await expect(sourceRow(page, 'unplugged').locator('p')).toHaveText('output/unplugged · manual scan');
// The drive is unplugged after the page read the list; removing another source resends the whole list.
await expect(await request.post('/__test__/system/unplug')).toBeOK();
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await saved;
expect(answer.status()).toBe(400);
expect(await answer.json()).toMatchObject({ code: 'invalid-source', source: 'output/unplugged', index: 1 });
await expect(statusOf(page)).toHaveText("output/unplugged can't be used: Gallery root does not exist. Remove it or reconnect it, then save.");
await expect(sourceRow(page, 'unplugged')).toHaveAttribute('aria-invalid', 'true');
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(3);
// Removed, the list saves.
await sourceRow(page, 'unplugged').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(0);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output']);
});
+406 -12
View File
@@ -11,7 +11,7 @@ const number = value => value.toLocaleString('en-US');
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
page.manifest = await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
@@ -55,6 +55,8 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]);
// I2: the Overview never walks the sources for thumbnails; the row reads the status route itself.
expect(overview).not.toHaveProperty('thumbnails');
// The fixture records no ambiguous file: the Needs review queue is empty.
expect(overview.needs_review).toEqual({ pending: 0 });
const layout = await openOverview(page);
expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item =>
[item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([
@@ -64,7 +66,7 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
['Not checked', 'Database integrity'],
]);
expect(await rowsOf(layout, 'Library health')).toEqual([
['Ambiguous image links', 'Not available yet.', 'Review metadata'],
['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'],
['Backup status', 'No retained backup record.', 'View backups'],
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
]);
@@ -138,9 +140,32 @@ test('thumbnail row reflects the status endpoint', async ({ page, request }) =>
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('3 missing thumbnails');
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources.
});
test('Overview Review thumbnails opens the same dialog', async ({ page, request }) => {
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
const plan = await serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
const starts = [];
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/thumbnails')) starts.push(item.postDataJSON()); });
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('2 missing thumbnails');
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/sources`);
const review = page.locator('dialog#modal[open]');
await expect(review.locator('#modal-title')).toHaveText('Thumbnail maintenance');
await expect(page).toHaveURL(`${workspace}#/system/overview`);
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
await expect(review.locator('#modal-body > p.small')).toHaveText(`2 previews to generate · ${number(plan.fresh)} already up to date.`);
// Escape closes it and focus returns to the row's button.
await page.keyboard.press('Escape');
await expect(review).toHaveCount(0);
await expect(row.getByRole('button', { name: 'Review thumbnails', exact: true })).toBeFocused();
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
await review.getByRole('button', { name: 'Start', exact: true }).click();
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
expect(starts).toEqual([{ mode: 'missing', size: 512, quality: 'medium' }]);
const id = page.url().split('/').pop();
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
});
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
@@ -166,14 +191,17 @@ test('a failed thumbnail read is unavailable, never zero', async ({ page }) => {
await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24');
});
test('ambiguous row reads Not available yet (Q6)', async ({ page }) => {
test('ambiguous row reads the pending count and Review metadata opens Needs review', async ({ page }) => {
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ hasText: 'Ambiguous image links' });
await expect(row.locator('p')).toHaveText('Not available yet.');
const row = layout.locator('.operation').filter({ hasText: 'ambiguous image links' });
await expect(row.locator('h3')).toHaveText('0 ambiguous image links');
await expect(row.locator('p')).toHaveText('Review metadata before assigning a positive / negative pair.');
const review = row.getByRole('button', { name: 'Review metadata', exact: true });
await expect(review).toHaveAttribute('aria-disabled', 'true');
await review.click({ force: true });
await expect(page).toHaveURL(`${workspace}#/system/overview`);
await expect(review).not.toHaveAttribute('aria-disabled');
await review.click();
// The concept's go-review: the Needs review page.
await expect(page).toHaveURL(`${workspace}#/review`);
await expect(page.getByRole('heading', { name: 'Metadata review', exact: true })).toBeVisible();
});
test('View backups / Appearance / Manage sources switch sections', async ({ page }) => {
@@ -237,10 +265,11 @@ test('out-of-order overview responses never repaint a newer section', async ({ p
await nav.getByRole('button', { name: 'Appearance', exact: true }).click();
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect.poll(held).toBe(2);
await nav.getByRole('button', { name: 'Sources & scans', exact: true }).click();
// Integrations still hands off (S10), so its body is one known panel.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await expect(await request.post('/__test__/release')).toBeOK();
await expect.poll(held).toBe(0);
// Both late answers reached a section that had left: Sources keeps its own body.
// Both late answers reached a section that had left: Integrations keeps its own body.
await expect(layout.locator(':scope > section.handoff')).toBeVisible();
await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve))));
await expect(layout.locator('.summary-metrics')).toHaveCount(0);
@@ -250,3 +279,368 @@ test('out-of-order overview responses never repaint a newer section', async ({ p
await expect(layout.locator('.summary-metrics')).toHaveCount(1);
await expect(layout.locator('.two-col')).toHaveCount(1);
});
// System › Appearance (#249 System S3 Task 8): the browser display defaults (`gallerySettings`, preferences.mjs) and the
// two host fields behind the one Save row, and the consumers that read them on their next visit.
const SETTINGS_URL = '/prompt_manager/settings';
const INSPECT_PATH = '/prompt_manager/images/inspect/';
const statusOf = page => page.locator('#workspace-main #settings-status');
const saveOf = page => page.locator('#workspace-main .save-row > button');
const storedSettings = page => page.evaluate(() => JSON.parse(localStorage.getItem('gallerySettings') || 'null'));
const imageCards = page => page.locator('#workspace-main article.media-card');
const panel = page => page.getByRole('complementary', { name: 'Selected image details', exact: true });
async function openAppearance(page) {
await page.goto(`${workspace}#/system/appearance`);
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
// The host rows are enabled once the server settings answered.
await expect(page.locator('#timeout')).toBeEnabled();
}
async function saveAppearance(page, text) {
await expect(saveOf(page)).not.toHaveAttribute('aria-disabled', 'true');
await saveOf(page).click();
await expect(statusOf(page)).toHaveText(text);
}
/** Set toggles (by row id) through their Toolkit switches. */
async function toggle(page, values) {
for (const [id, checked] of Object.entries(values)) await page.locator(`#${id}`).setChecked(checked, { force: true });
}
/**
* A request barrier: one request issued and answered after everything the page has asked for so far, so a read the
* last action started has been seen (Playwright reports a page's requests in order).
*/
async function requestBarrier(page) {
expect(await page.evaluate(url => fetch(url, { cache: 'no-store' }).then(response => response.ok), SETTINGS_URL)).toBe(true);
}
async function openImages(page) {
await page.goto(`${workspace}#/images`);
await expect(imageCards(page).first()).toBeVisible();
}
test('defaults apply to a new visit but not to the current view or a saved view', async ({ page, request }) => {
const created = await request.post('/prompt_manager/saved-views', { data: { page: 'images', name: 'Everything',
state: { q: '', category: '', sort: 'newest', source: 'all', folder: '', media: 'all', tags: [], favorites: false } } });
expect(created.status()).toBe(201);
const view = (await created.json()).view;
try {
await openImages(page);
// The current view: the toolbar sets List.
await page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'List', exact: true }).click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await openAppearance(page);
await expect(page.locator('#default-view')).toHaveValue('grid');
await page.locator('#default-view').selectOption('masonry');
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect([stored.workspaceLayout, stored.workspaceDensity]).toEqual(['masonry', 'compact']);
// Back in Images (the page session's last visit): the current view keeps its own layout and size.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// The saved view opens as before.
await page.goto(`${workspace}#/saved/${view.id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// A new visit starts from the defaults.
await openImages(page);
await page.reload();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'masonry');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
await expect(page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'Masonry', exact: true }))
.toHaveAttribute('aria-pressed', 'true');
} finally {
await request.delete(`/prompt_manager/saved-views/${view.id}`);
}
});
test('the defaults reach the Images page only; embedded galleries keep their own presentation', async ({ page }) => {
await openAppearance(page);
await page.locator('#default-view').selectOption('list');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
// The prompt history's shared gallery has no toolbar to change them, so it opens as it always did.
await page.goto(`${workspace}#/library/${page.manifest.prompts[0].positive_id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-density', 'comfortable');
// The Images page starts from them.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
});
test('metadata off shows an explicit load and loads once on request', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'metadata-auto': false });
await saveAppearance(page, 'Saved in this browser.');
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
await openImages(page);
const first = imageCards(page).first();
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).getByRole('status')).toHaveText('Metadata loads when you ask for it.');
const load = panel(page).getByRole('button', { name: 'Load metadata', exact: true });
await expect(load).toBeVisible();
expect(reads).toEqual([]);
await load.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(1);
// The same file again reads nothing more (the cache is on).
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await requestBarrier(page);
expect(reads).toHaveLength(1);
// The viewer's details use the same inspector: another file waits there too, and loads on request.
await imageCards(page).nth(1).locator('.image-open').click();
const viewerDetails = page.locator('#viewer-details');
await expect(viewerDetails.getByRole('status')).toHaveText('Metadata loads when you ask for it.');
expect(reads).toHaveLength(1);
await viewerDetails.getByRole('button', { name: 'Load metadata', exact: true }).click();
await expect(viewerDetails.locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(2);
});
test('cache off rereads the current selection; on reuses the one current record', async ({ page }) => {
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
for (const cache of [true, false]) {
if (!cache) {
await openAppearance(page);
await toggle(page, { 'metadata-cache': false });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
}
await openImages(page);
const info = imageCards(page).first().locator('.image-card-actions button').first();
await info.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await expect.poll(() => reads.length).toBeGreaterThan(0);
await requestBarrier(page);
reads.length = 0;
// Inspect the selected file twice more.
for (const _ of [1, 2]) {
const answered = cache ? null : page.waitForResponse(response => new URL(response.url()).pathname.startsWith(INSPECT_PATH));
await info.click();
if (answered) await answered;
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
}
await requestBarrier(page);
expect(reads, `cache ${cache}`).toHaveLength(cache ? 0 : 2);
}
});
test('paths hidden only changes display', async ({ page }) => {
const rows = async () => page.locator('#viewer-details details').filter({ hasText: 'File & provenance' }).locator('dt')
.evaluateAll(terms => terms.map(term => term.textContent));
const openViewer = async () => {
await openImages(page);
await imageCards(page).first().locator('.image-open').click();
const provenance = page.locator('#viewer-details details').filter({ hasText: 'File & provenance' });
await provenance.locator('summary').click();
await expect(provenance.locator('dl')).toBeVisible();
return provenance;
};
let provenance = await openViewer();
expect(await rows()).toContain('Path');
const before = { show: await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count(),
download: await provenance.getByRole('link', { name: 'Download original', exact: true }).getAttribute('href') };
await page.keyboard.press('Escape');
await openAppearance(page);
await toggle(page, { paths: false });
await saveAppearance(page, 'Saved in this browser.');
provenance = await openViewer();
const shown = await rows();
expect(shown).not.toContain('Path');
expect(shown).toEqual(expect.arrayContaining(['Source', 'Size']));
// Everything else is as it was: the folder action and the original download stay.
expect(await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count()).toBe(before.show);
await expect(provenance.getByRole('link', { name: 'Download original', exact: true })).toHaveAttribute('href', before.download);
});
test('video defaults reach the viewer', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'video-auto': true, 'video-muted': false, 'video-loop': false });
await saveAppearance(page, 'Saved in this browser.');
expect(await page.evaluate(async () => (await import('/prompt_manager/js/workspace/media-viewer.mjs')).readViewerPreferences()))
.toEqual({ videoAutoplay: true, videoMute: false, videoLoop: false });
const stored = await storedSettings(page);
expect([stored.videoAutoplay, stored.videoMute, stored.videoLoop]).toEqual([true, false, false]);
});
test('host fields round-trip through /settings with revision', async ({ page, request }) => {
const posts = [];
page.on('request', item => { if (item.method() === 'POST' && new URL(item.url()).pathname === SETTINGS_URL) posts.push(item.postDataJSON()); });
const before = await serverJson(request, SETTINGS_URL);
await openAppearance(page);
await expect(page.locator('#open-mode')).toHaveValue(before.settings.webui_display_mode);
await expect(page.locator('#timeout')).toHaveValue(String(before.settings.result_timeout));
// Only whole seconds from 0 to 300: anything else is highlighted and never sent.
for (const bad of ['301', '-1', '2.5']) {
await page.locator('#timeout').fill(bad);
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).toHaveAttribute('aria-invalid', 'true');
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
await expect(saveOf(page)).toHaveAttribute('aria-disabled', 'true');
}
await page.locator('#timeout').fill('0');
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).not.toHaveAttribute('aria-invalid');
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved to the server configuration.');
expect(posts).toEqual([{ webui_display_mode: 'popup', result_timeout: 0, expected_revision: before.revision }]);
const after = await serverJson(request, SETTINGS_URL);
expect([after.settings.webui_display_mode, after.settings.result_timeout]).toEqual(['popup', 0]);
expect(after.revision).not.toBe(before.revision);
// No browser key was written for a host field.
expect(await storedSettings(page)).toBeNull();
});
test('server settings that cannot be read say so in their rows', async ({ page }) => {
await page.route(url => url.pathname === SETTINGS_URL, route => (route.request().method() === 'GET'
? route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, error: 'Injected failure' }) })
: route.continue()));
await page.goto(`${workspace}#/system/appearance`);
for (const id of ['open-mode', 'timeout']) {
const row = page.locator('#workspace-main .setting-row').filter({ has: page.locator(`#${id}`) });
await expect(row.locator('p.small')).toHaveText('Server settings could not be read.');
await expect(page.locator(`#${id}`)).toBeDisabled();
}
// The browser rows are this browser's, and work as before.
await expect(page.locator('#default-view')).toBeEnabled();
await expect(page.locator('#workspace-main .setting-row p.small')).toHaveCount(2);
});
test('stale revision shows the conflict status', async ({ page, request }) => {
await openAppearance(page);
// Another page saves after this one read the settings.
const current = await serverJson(request, SETTINGS_URL);
await expect(await request.post(SETTINGS_URL, { data: { result_timeout: 42, expected_revision: current.revision } })).toBeOK();
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#open-mode')).toHaveAttribute('aria-invalid', 'true');
// The draft is kept and the section now shows the other page's value.
await expect(page.locator('#open-mode')).toHaveValue('popup');
await expect(page.locator('#timeout')).toHaveValue('42');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('newtab');
// With the reloaded revision, Save succeeds.
await saveAppearance(page, 'Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('popup');
});
test('reload keeps saved values; a new tab reads them', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#image-quality').selectOption('high');
await toggle(page, { 'card-details': false });
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved in this browser. Saved to the server configuration.');
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#image-quality')).toHaveValue('high');
await expect(page.locator('#card-details')).not.toBeChecked();
await expect(page.locator('#open-mode')).toHaveValue('popup');
const other = await context.newPage();
await openAppearance(other);
await expect(other.locator('#image-quality')).toHaveValue('high');
await expect(other.locator('#card-details')).not.toBeChecked();
// A change saved in one tab reaches the other's clean controls; a draft it overtakes is flagged and kept.
await other.locator('#density').selectOption('large');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#density')).toHaveValue('large');
await expect(other.locator('#settings-status')).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(other.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await page.locator('#image-quality').selectOption('low');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#image-quality')).toHaveValue('low');
await other.close();
});
test('a change another tab saved while System was closed is a conflict on return, never saved over silently', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
// Leave System for Images: the draft stays for the page session, but System no longer listens.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
const other = await context.newPage();
await openAppearance(other);
await other.locator('#density').selectOption('large');
await saveAppearance(other, 'Saved in this browser.');
await other.close();
// Back in System: the draft is kept and flagged, and nothing was written over the other tab's value.
await page.evaluate(() => { location.hash = '#/system/appearance'; });
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await expect(statusOf(page)).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await expect(page.locator('#density')).toHaveValue('compact');
expect((await storedSettings(page)).workspaceDensity).toBe('large');
// Saving after the conflict was shown is an explicit choice: it wins.
await saveAppearance(page, 'Saved in this browser.');
expect((await storedSettings(page)).workspaceDensity).toBe('compact');
await expect(page.locator('#density')).not.toHaveAttribute('aria-invalid');
});
test('a browser that refuses storage is saved for this tab only, never "Saved"', async ({ page }) => {
await openAppearance(page);
await page.evaluate(() => {
Storage.prototype.setItem = function setItem() { throw new DOMException('Quota exceeded', 'QuotaExceededError'); };
});
await page.locator('#default-view').selectOption('list');
await saveAppearance(page, 'Saved for this tab only; browser storage is unavailable.');
expect(await storedSettings(page)).toBeNull();
// This tab's next Images visit still uses it.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
});
test('added rows reach their consumers', async ({ page }) => {
await openImages(page);
await expect(imageCards(page).first().locator('.media-info h3')).toBeVisible();
await openAppearance(page);
await toggle(page, { 'card-details': false, 'viewer-details': true });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
await openImages(page);
// Show image details on cards off: no title or file line; Info and Tag stay.
const card = imageCards(page).first();
await expect(card.locator('.media-info h3')).toBeHidden();
await expect(card.locator('.image-card-actions button')).toHaveCount(2);
// Collapse viewer details by default: the viewer opens with its details hidden behind one toggle.
await card.locator('.image-open').click();
const details = page.locator('#viewer-details');
await expect(page.locator('#viewer-dialog')).toBeVisible();
await expect(details).toBeHidden();
const show = page.locator('#viewer-toolbar').getByRole('button', { name: 'Show details', exact: true });
await expect(show).toHaveAttribute('aria-expanded', 'false');
await show.click();
await expect(details).toBeVisible();
await expect(page.locator('#viewer-toolbar').getByRole('button', { name: 'Hide details', exact: true })).toHaveAttribute('aria-expanded', 'true');
});
test('reset restores defaults without touching other keys', async ({ page }) => {
await page.goto(`${workspace}#/system/appearance`);
await page.evaluate(() => localStorage.setItem('gallerySettings', JSON.stringify({ gridColumns: 5, defaultLimit: 200,
thumbnailPromptDismissed: 'kept', workspaceLayout: 'list', videoMute: false, showImageInfo: false })));
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#default-view')).toHaveValue('list');
const reset = page.locator('#workspace-main .operation').filter({ hasText: 'Reset display defaults' }).getByRole('button', { name: 'Reset', exact: true });
await reset.click();
await expect(page.locator('#default-view')).toHaveValue('grid');
await expect(page.locator('#video-muted')).toBeChecked();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect(stored).toEqual({ gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'kept', workspaceLayout: 'grid',
videoMute: true, showImageInfo: true });
});
+15 -11
View File
@@ -22,6 +22,8 @@ const routes = [
['system', 'System', null],
// #249 D51: SAVED VIEWS' Favorites, Images filtered to favorites.
['favorites', 'Favorites', null],
// #249 PR F (ruling R5): SAVED VIEWS' Needs review, the concept's Metadata review page (its own heading).
['review', 'Needs review', null, 'Metadata review'],
// #249 Activity-1: the concept's Activity page in the tools-and-system group.
['activity', 'Activity', null],
];
@@ -77,10 +79,10 @@ test('workspace starts with real persisted statistics without mutating data', as
});
test('all shell destinations deep-link, reload and expose working existing applications', async ({ page, request }) => {
for (const [route, label, legacy] of routes) {
for (const [route, label, legacy, title = label] of routes) {
await page.goto(`${workspace}#/${route}`);
const main = page.locator('#workspace-main');
await expect(main.getByRole('heading', { level: 1, name: label, exact: true })).toBeVisible();
await expect(main.getByRole('heading', { level: 1, name: title, exact: true })).toBeVisible();
await expect(sidebarLink(page, label)).toHaveAttribute('aria-current', 'page');
if (route === 'tags' || route === 'inspect') await expect(main.locator('.handoff')).toHaveCount(0);
if (route === 'inspect') await expect(main.locator('#local-file')).toBeAttached();
@@ -89,7 +91,7 @@ test('all shell destinations deep-link, reload and expose working existing appli
await expect(await request.get(legacy.split('#')[0])).toBeOK();
}
await page.reload();
await expect(main.getByRole('heading', { level: 1, name: label, exact: true })).toBeVisible();
await expect(main.getByRole('heading', { level: 1, name: title, exact: true })).toBeVisible();
}
// D29 and #249 E3: the legacy gallery and Library are reached from Overview's tool cards, not from their pages.
await page.goto(`${workspace}#/overview`);
@@ -309,9 +311,9 @@ test('overview reports loading, offers retry after failure and renders real empt
test('small screens keep navigation and the theme accessible without horizontal page overflow', async ({ page }) => {
await page.setViewportSize({ width: 375, height: 812 });
await page.goto(`${workspace}#/overview`);
for (const [, label] of routes) {
for (const [, label, , title = label] of routes) {
await sidebarLink(page, label).click();
await expect(page.locator('#workspace-main h1')).toHaveText(label);
await expect(page.locator('#workspace-main h1')).toHaveText(title);
}
await expect(page.getByRole('combobox', { name: 'Theme', exact: true })).toBeVisible();
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
@@ -425,8 +427,7 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance',
'Backup & export', 'Diagnostics & logs'];
const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics'];
const SYSTEM_HANDOFFS = { appearance: ['/prompt_manager/gallery', '/prompt_manager/admin'], sources: ['/prompt_manager/admin', '/prompt_manager/gallery'],
maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
const SYSTEM_HANDOFFS = { maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => {
const mutations = [];
@@ -462,14 +463,17 @@ test('System is native: the concept shell, section navigation, sections, hand-of
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
// The section changes in place: focus stays on the pressed section, not the page title.
await expect(button).toBeFocused();
// S2 Task 6: Overview is native (workspace-system.spec.mjs); Maintenance shows its health-check notice above
// the hand-off (D123); every other section hands off.
// S2 Task 6: Overview is native (workspace-system.spec.mjs), and so are Appearance (S3 Task 8) and Sources & scans
// (S4 Task 11, workspace-system-sources.spec.mjs, its update notice shown in a fresh browser); Maintenance shows
// its health-check notice above the hand-off (D123); every other section hands off.
if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible();
if (hash === 'sources') await expect(layout.locator('#source-list .operation')).toHaveCount(2);
expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'],
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'div.save-row'],
sources: ['div.notice', 'section.surface', 'div.save-row'],
maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash]
?? ['section.surface handoff', 'div.save-row']);
if (hash === 'overview') continue;
if (['overview', 'appearance', 'sources'].includes(hash)) continue;
const handoff = layout.locator('section.handoff');
await expect(handoff.locator('h2')).toHaveText('Your current tools are ready');
expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href'))))
+47 -10
View File
@@ -5,18 +5,23 @@ reading files, so a test sets its progress and counters exactly:
- ``POST /__test__/jobs/start`` starts one (``kind``, ``title``, ``total``,
``output_folders``; for ``kind: "image-tags"`` the image tagging job's
groups and its ``{mode, images}`` scope instead) and returns its summary,
or 409 while one runs;
groups and its ``{mode, images}`` scope instead; for ``kind:
"thumbnails"`` the thumbnail job's groups, ``guarded`` admission and its
``{mode, size, quality, output_folders}`` scope; for ``kind: "health"`` the
health check's: no groups, ``read``, not cancellable, with its status
line) and returns its summary, or 409 while one runs;
- ``POST /__test__/jobs/{id}/step`` records ``records`` (``{group, path,
root, root_index, detail}`` file outcomes, counted, listed and logged as the scan
records them) and ``bulk`` (``{group, count, root}``: ``count`` numbered
rows), sets ``processed`` (and ``total``) and ``counters``, then publishes
the progress to every subscriber;
- ``POST /__test__/jobs/{id}/finish`` ends it as ``finished``, ``cancelled``
or ``failed``; a cancel through the real route ends it ``cancelled``;
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``):
a running, non-cancellable ``records`` job, as a legacy restore or
maintenance request takes, until ``finish`` ends it;
or ``failed``, with the runner's final ``status_line`` when given; a cancel
through the real route ends it ``cancelled``;
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``,
``admission``, default ``records``): a running, non-cancellable job with no
outcome groups, as a legacy restore, maintenance or thumbnail clear request
takes, until ``finish`` ends it;
- ``POST /__test__/jobs/restart`` acts out a ComfyUI restart: the open job
streams drop without a final event and the registry starts empty.
"""
@@ -35,6 +40,9 @@ PREFIXES = {"added": "+", "known": "=", "review": "?", "errors": "!"}
PREFIXES["no_metadata"] = "-"
# The image tagging job's groups (#249 PR D), logged with the runner's prefixes.
PREFIXES.update({"tagged": "+", "unchanged": "=", "skipped": "-"})
# The thumbnail job's (#249 System S5).
PREFIXES["generated"] = "+"
HEALTH_STATUS_LINE = "Checking the database and image links"
ROOT_INDEXES = {"output": 0, "input": 1}
@@ -80,9 +88,12 @@ def outcome_records(data):
def register_activity_commands(app, routes, api):
from py.api.jobs import (
HEALTH_JOB_KIND,
IMAGE_TAG_GROUP_NAMES,
IMAGE_TAG_JOB_KIND,
SCAN_GROUP_NAMES,
THUMBNAIL_GROUP_NAMES,
THUMBNAILS_JOB_KIND,
JobBusy,
JobRegistry,
)
@@ -138,7 +149,21 @@ def register_activity_commands(app, routes, api):
data = await body(request)
ending = asyncio.get_running_loop().create_future()
kind = str(data.get("kind", "scan"))
if kind == IMAGE_TAG_JOB_KIND:
admission, cancellable = "records", True
if kind == THUMBNAILS_JOB_KIND:
title = "Thumbnail generation"
scope = {
"mode": str(data.get("mode", "missing")),
"size": count(data.get("size"), 512),
"quality": str(data.get("quality", "medium")),
"output_folders": count(data.get("output_folders"), 2),
}
groups = THUMBNAIL_GROUP_NAMES
admission = "guarded"
elif kind == HEALTH_JOB_KIND:
title, scope, groups = "Health check", {}, ()
admission, cancellable = "read", False
elif kind == IMAGE_TAG_JOB_KIND:
title = "Add tags to images"
scope = {
"mode": str(data.get("mode", "add")),
@@ -150,13 +175,20 @@ def register_activity_commands(app, routes, api):
scope = {
"files": str(data.get("files", "new")),
"output_folders": count(data.get("output_folders"), 2),
# A real scan's count of its files in the Needs review queue.
**(
{"review_items": count(data["review_items"])}
if "review_items" in data
else {}
),
}
groups = SCAN_GROUP_NAMES
try:
job = api._jobs.start(
kind,
str(data.get("title", title)),
record_changing=True,
admission=admission,
cancellable=cancellable,
scope=scope,
runner=fake_runner(ending),
groups=groups,
@@ -164,6 +196,8 @@ def register_activity_commands(app, routes, api):
except JobBusy:
raise web.HTTPConflict()
job.update(total=count(data.get("total")))
if kind == HEALTH_JOB_KIND:
job.update(status_line=HEALTH_STATUS_LINE)
endings[job.id] = ending
return web.json_response({"success": True, "job": job.summary()})
@@ -178,7 +212,7 @@ def register_activity_commands(app, routes, api):
async with api._jobs.hold(
str(data.get("kind", "restore")),
str(data.get("title", "Database restore (admin page)")),
admission="records",
admission=str(data.get("admission", "records")),
) as job:
entered.set_result(job)
state = await ending
@@ -226,9 +260,12 @@ def register_activity_commands(app, routes, api):
@routes.post("/__test__/jobs/{job_id}/finish")
async def finish_job(request):
job = job_or_404(request)
state = (await body(request)).get("state", "finished")
data = await body(request)
state = data.get("state", "finished")
if state not in FINAL_STATES:
raise web.HTTPBadRequest()
if isinstance(data.get("status_line"), str): # the runner's final line
job.update(status_line=data["status_line"])
ending = endings[job.id]
if not ending.done():
ending.set_result(state)
@@ -176,6 +176,8 @@ _TABLES = (
"execution_context_texts",
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -28,6 +28,8 @@ EDITOR_METADATA = {
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+368
View File
@@ -0,0 +1,368 @@
"""Owned, deterministic Needs review items for the workspace page (#249 PR F).
Each file is a PNG whose workflow has two samplers with their own positive
encoders and one shared negative, so the resolver finds it ``ambiguous``. Its
loader node carries ComfyUI's ``Infinity`` stamp, which the legacy pages'
parser (the scan's fallback) cannot read, so nothing links the file and it is
a Needs review case (#284: only ambiguous, unlinked files are queued). It is
recorded through the producers' entry point (``record_review_file``). No path
or SQL comes from a request.
``POST /__test__/review/reset`` also takes the special items F2 needs, queued
after the plain ones in this order:
- ``conflict``: an item whose file was linked to another prompt ("Misty
forest path") after it was queued (the next scan or monitor pass would
remove the item); applying a sampler path then conflicts with that link.
- ``limited``: a graph of 17 samplers, each with its own prompt, which the
recorder stores ``limited`` (more than 16 candidates, so none are stored).
An A1111 block with two negative lines is not queued any more: it offers
no candidate to choose (#284).
- ``unavailable``: a file recorded under the ``temp`` directory while it was a
gallery root; the root is then removed again, so no current root lists it.
``concept: true`` (F3, the concept comparison) names and sizes the plain
files as the concept's first three images (app.js ``images``) and gives
their first sampler the concept's first prompt (``alpine``: its positive and
its first negative, the one the concept's panel shows), so the panel's text
wraps as the concept's does.
``POST /__test__/review/add {kind}`` adds one file while a page is open:
``monitor`` records a new ambiguous, unlinked output from a worker thread
through ``record_review_file`` (the monitor's own entry point; the monitor
never queues an output it linked itself), ``file`` only writes one (a
metadata scan records it). ``GET
/__test__/review/links`` answers each review file's prompt and pair links, and
``GET /__test__/review/streams`` the open change streams and captures.
"""
from __future__ import annotations
import json
import os
import threading
from pathlib import Path
import shutil
import time
from typing import TYPE_CHECKING
from aiohttp import web
from PIL import Image
from PIL.PngImagePlugin import PngInfo
if TYPE_CHECKING:
from py.api import PromptManagerAPI
POSITIVES = ("Golden dunes at dusk", "Harbor lights at night")
NEGATIVE = "Soft negative, blur"
# The prompt the conflict item's file is linked to after it was queued.
FALLBACK = "Misty forest path"
SPECIAL = ("conflict", "limited", "unavailable")
# The concept's first three images (concept app.js `images`) and first prompt.
CONCEPT_IMAGES = (
("alpine_00001.png", (1536, 1152)),
("dunes_00001.png", (1024, 1536)),
("forest_00001.png", (1280, 1280)),
)
CONCEPT_POSITIVE = (
"A remote alpine observatory at blue hour, warm light through glass, dramatic "
"mountain peaks, cinematic composition, atmospheric mist, fine detail"
)
CONCEPT_NEGATIVE = "blurry, low quality, low resolution"
# More candidates than an item stores (16): the ``limited`` item.
LIMITED_SAMPLERS = 17
# More than one queue page (40), so Load more has a second page.
MAX_ITEMS = 45
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
"image_prompt_pairs",
"prompt_pair_tags",
"prompt_pairs",
"generated_images",
"prompt_tags",
"tags",
"prompt_library_state",
"prompts",
)
def _graph(
encoder_class: str = "CLIPTextEncode",
positives: tuple = POSITIVES,
negative: str = NEGATIVE,
) -> dict:
graph = {
# ``Infinity`` (as ComfyUI writes it): the legacy parser cannot read the
# graph, so the scan's fallback links nothing and the file is queued.
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "m"},
"is_changed": [float("inf")],
},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
}
for index, text in enumerate(positives):
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
graph[encoder] = {"class_type": encoder_class, "inputs": {"text": text}}
graph[sampler] = {
"class_type": "KSampler",
"inputs": {
"model": ["1", 0],
"positive": [encoder, 0],
"negative": ["2", 0],
},
}
return graph
def _png(
path: Path,
colour: tuple[int, int, int],
*,
graph: dict | None = None,
parameters: str | None = None,
size: tuple[int, int] = (96, 64),
) -> None:
info = PngInfo()
if parameters is not None:
info.add_text("parameters", parameters)
else:
info.add_text("prompt", json.dumps(graph or _graph()))
path.parent.mkdir(parents=True, exist_ok=True)
with Image.new("RGB", size, colour) as image:
image.save(path, pnginfo=info)
def _record(api: PromptManagerAPI, path: Path, expected: str) -> None:
outcome = api.record_review_file(str(path))
if outcome != expected:
raise RuntimeError(f"{path.name} was {outcome}, not {expected}")
def _stamp(path: Path, age: float) -> None:
stamp = time.time() - age
os.utime(path, (stamp, stamp))
class _Monitor:
"""The monitor's recorder over this server's database (one per server)."""
def __init__(self, api: PromptManagerAPI) -> None:
self.api = api
self.lock = threading.Lock()
def new_output(self, path: Path) -> None:
"""The monitor's Needs review pass for a new output, from a worker thread."""
outcome = []
with self.lock:
worker = threading.Thread(
target=lambda: outcome.append(self.api.record_review_file(str(path)))
)
worker.start()
worker.join()
if outcome != ["recorded"]:
raise RuntimeError(f"{path.name} was {outcome}, not recorded")
def _monitored_output(api: PromptManagerAPI, monitor: _Monitor, path: Path) -> None:
_png(path, (150, 60, 40))
monitor.new_output(path)
def _linked_after_queueing(api: PromptManagerAPI, path: Path) -> None:
from utils.hashing import generate_prompt_hash
_png(path, (150, 60, 40))
_record(api, path, "recorded")
prompt = api.db.get_prompt_by_hash(generate_prompt_hash(FALLBACK))
prompt_id = (
prompt["id"]
if prompt is not None
else api.db.save_prompt(FALLBACK, prompt_hash=generate_prompt_hash(FALLBACK))
)
api.db.link_image_to_prompt(prompt_id, str(path))
def _unavailable(api: PromptManagerAPI, root: Path, path: Path) -> None:
from py.config import GalleryConfig
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
GalleryConfig.MONITORING_DIRECTORIES = [*roots, str(root / "temp")]
try:
_record(api, path, "recorded")
finally:
GalleryConfig.MONITORING_DIRECTORIES = roots
def seed_review(
api: PromptManagerAPI,
root: Path,
items: int,
special: tuple = (),
monitor: _Monitor | None = None,
concept: bool = False,
) -> dict:
"""Replace this server's disposable rows and files with ``items`` queued files."""
with api.db.model.get_connection() as conn:
for table in _TABLES:
conn.execute(f"DELETE FROM {table}")
api.close_annotation_captures()
for name in ("output", "input"):
shutil.rmtree(root / name)
(root / name).mkdir()
shutil.rmtree(root / "temp" / "review", ignore_errors=True)
output = root / "output"
files = []
for number in range(items):
relative = f"review/review_{number:04}.png"
options = {}
if concept:
name, size = CONCEPT_IMAGES[number % len(CONCEPT_IMAGES)]
relative = f"review/{name}" if number < len(CONCEPT_IMAGES) else relative
graph = _graph(
positives=(CONCEPT_POSITIVE, POSITIVES[1]), negative=CONCEPT_NEGATIVE
)
options = {"graph": graph, "size": size}
_png(output / relative, (40 + number * 4, 90, 140), **options)
_stamp(output / relative, 60 + number)
_record(api, output / relative, "recorded")
files.append({"path": relative, "filename": Path(relative).name})
for kind in SPECIAL:
if kind not in special:
continue
relative = f"review/{kind}.png"
path = (root / "temp" if kind == "unavailable" else output) / relative
if kind == "conflict":
_linked_after_queueing(api, path)
elif kind == "limited":
many = tuple(f"{POSITIVES[0]} {n}" for n in range(LIMITED_SAMPLERS))
_png(path, (90, 90, 90), graph=_graph(positives=many))
_record(api, path, "limited")
else:
_png(path, (20, 140, 90))
_unavailable(api, root, path)
files.append({"path": relative, "filename": path.name, "kind": kind})
api.invalidate_gallery_cache()
return {
"files": files,
"positives": list(POSITIVES),
"negative": NEGATIVE,
"fallback": FALLBACK,
}
def add_review_file(
api: PromptManagerAPI, root: Path, kind: str, monitor: _Monitor
) -> dict:
"""One more ambiguous output: through the monitor, or only written."""
output = root / "output"
number = (
len(list((output / "added").glob("*.png")))
if (output / "added").exists()
else 0
)
relative = f"added/added_{number:04}.png"
if kind == "monitor":
_monitored_output(api, monitor, output / relative)
else:
_png(output / relative, (200, 200, 40))
api.invalidate_gallery_cache()
return {"path": relative, "filename": Path(relative).name}
_LINKS = (
"SELECT gi.filename, p.text AS prompt, pp.id AS pair_id, pos.text AS positive, "
"neg.text AS negative FROM generated_images gi "
"JOIN prompts p ON p.id = gi.prompt_id "
"LEFT JOIN image_prompt_pairs ipp ON ipp.image_id = gi.id "
"LEFT JOIN prompt_pairs pp ON pp.id = ipp.pair_id "
"LEFT JOIN prompts pos ON pos.id = pp.positive_prompt_id "
"LEFT JOIN prompts neg ON neg.id = pp.negative_prompt_id "
"ORDER BY gi.filename, gi.id, pp.id"
)
def review_links(api: PromptManagerAPI) -> list:
"""Every file's prompt link and pair links, as plain rows."""
conn = api.db.model.get_connection()
return [dict(row) for row in conn.execute(_LINKS)]
def register_review_commands(
routes: web.RouteTableDef, api: PromptManagerAPI, root: Path
) -> None:
"""Register the review fixture's commands; no path or SQL comes from a request."""
monitors = []
def monitor() -> _Monitor:
if not monitors:
monitors.append(_Monitor(api))
return monitors[0]
@routes.post("/__test__/review/reset")
async def reset(request: web.Request) -> web.Response:
data = await request.json() if request.can_read_body else {}
if (
request.query
or not isinstance(data, dict)
or set(data) - {"items", "concept", *SPECIAL}
):
raise web.HTTPBadRequest()
items = data.get("items", 3)
if isinstance(items, bool) or not isinstance(items, int):
raise web.HTTPBadRequest()
if not 0 <= items <= MAX_ITEMS:
raise web.HTTPBadRequest()
if any(
not isinstance(data.get(kind, False), bool)
for kind in (*SPECIAL, "concept")
):
raise web.HTTPBadRequest()
special = tuple(kind for kind in SPECIAL if data.get(kind))
manifest = await api._run_in_executor(
seed_review,
api,
root,
items,
special,
monitor(),
data.get("concept", False),
)
return web.json_response({"success": True, **manifest})
@routes.post("/__test__/review/add")
async def add(request: web.Request) -> web.Response:
data = await request.json() if request.can_read_body else None
if request.query or not isinstance(data, dict) or set(data) != {"kind"}:
raise web.HTTPBadRequest()
if data["kind"] not in ("monitor", "file"):
raise web.HTTPBadRequest()
added = await api._run_in_executor(
add_review_file, api, root, data["kind"], monitor()
)
return web.json_response({"success": True, **added})
@routes.get("/__test__/review/links")
async def links(request: web.Request) -> web.Response:
if request.query:
raise web.HTTPBadRequest()
return web.json_response(
{"links": await api._run_in_executor(review_links, api)}
)
@routes.get("/__test__/review/streams")
async def streams(request: web.Request) -> web.Response:
"""Open review change streams and held captures (teardown checks)."""
if request.query:
raise web.HTTPBadRequest()
store = api._captures
captures = store.stats()["captures"] if store is not None else 0
return web.json_response(
{"streams": api.review_stream_count(), "captures": captures}
)
+96 -6
View File
@@ -11,11 +11,18 @@ read. These per-server hooks only change what a System test needs on top:
requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes'
500 error envelope;
- ``POST /__test__/system/missing-source`` configures one more source that
does not exist (an unmounted drive);
does not exist (an unmounted drive); with ``{"inside": true}`` it is an
existing folder inside the output folder (``output/unplugged``), which
``POST /__test__/system/unplug`` then removes (a drive unplugged while the
page shows it);
- ``POST /__test__/system/missing-file`` deletes the newest linked image's
file from disk, leaving its database row (a missing-file link);
- ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain
PNGs with no thumbnail into the output folder.
PNGs with no thumbnail into the output folder;
- ``POST /__test__/system/monitor`` ``{watching, alive}`` stands in a fake
image monitor whose observer is ``alive`` (default true) and schedules the
named folders of this server's root (``["output"]``), so Sources reads
watching, manual scan and restart pending states (S4).
Delays use the shared request holds (``request_hold.py``).
"""
@@ -31,9 +38,31 @@ MAX_UNTHUMBNAILED = 50
def register_system_commands(app, routes, api, root: Path):
from py.config import GalleryConfig
# System › Appearance saves the host fields to this server's config.json (S3);
# reset restores both. The file is this server's own, inside its temp root:
# reset rewrites or deletes it, so a server whose configuration points
# anywhere else (a real config.json) refuses to register these hooks.
config_path = root / "config.json"
configured_path = os.environ.get("PROMPT_MANAGER_CONFIG_PATH")
if (
configured_path is None
or Path(configured_path).resolve() != config_path.resolve()
):
raise RuntimeError(
f"PROMPT_MANAGER_CONFIG_PATH must be {config_path}, not {configured_path!r}"
)
from py.config import GalleryConfig, PromptManagerConfig
from utils import image_monitor
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
config_bytes = config_path.read_bytes() if config_path.exists() else None
host_values = (
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
)
monitor = getattr(image_monitor, "_monitor_instance", None)
failures: dict[str, int] = {}
@web.middleware
@@ -48,10 +77,28 @@ def register_system_commands(app, routes, api, root: Path):
app.middlewares.append(inject_failures)
unplugged = root / "output" / "unplugged"
def reset():
failures.clear()
if unplugged.is_dir():
unplugged.rmdir()
GalleryConfig.MONITORING_DIRECTORIES = list(configured)
api._health_report = None
(
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
) = host_values
image_monitor._monitor_instance = monitor
if config_bytes is None:
config_path.unlink(missing_ok=True)
else:
# Replace in one step, as the server writes it (Windows preflight).
staged = config_path.with_name(config_path.name + ".reset")
staged.write_bytes(config_bytes)
os.replace(staged, config_path)
@routes.post("/__test__/system/reset")
async def system_reset(request):
@@ -75,14 +122,26 @@ def register_system_commands(app, routes, api, root: Path):
@routes.post("/__test__/system/missing-source")
async def missing_source(request):
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
data = await request.json() if request.can_read_body else {}
if data.get("inside") is True:
unplugged.mkdir(exist_ok=True)
absent = str(unplugged)
else:
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent]
return web.json_response(
{"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)}
)
@routes.post("/__test__/system/unplug")
async def unplug(request):
if not unplugged.is_dir():
raise web.HTTPConflict()
unplugged.rmdir()
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-file")
async def missing_file(request):
with api.db.model.get_connection() as conn:
@@ -113,4 +172,35 @@ def register_system_commands(app, routes, api, root: Path):
api.invalidate_gallery_cache()
return web.json_response({"success": True})
@routes.post("/__test__/system/monitor")
async def fake_monitor(request):
data = await request.json()
watching, alive = data.get("watching", []), data.get("alive", True)
if (
not isinstance(watching, list)
or not all(name in ("output", "input") for name in watching)
or not isinstance(alive, bool)
):
raise web.HTTPBadRequest()
image_monitor._monitor_instance = _FakeMonitor(
[str(root / name) for name in watching], alive
)
return web.json_response({"success": True})
return reset
class _FakeMonitor:
"""The image monitor's status surface (``get_status``) with no observer thread."""
def __init__(self, directories, alive):
self.monitored_directories = directories
self._alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": list(self.monitored_directories),
"handler_active": True,
"observer_alive": self._alive,
}
@@ -44,6 +44,8 @@ MAX_EXTRA_IMAGES = 160
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -114,6 +114,8 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -81,6 +81,8 @@ MAX_EXTRA_CATEGORIES = 200
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+10 -1
View File
@@ -1,11 +1,20 @@
import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
import runtime from '../../tools/vendor/comfyui-runtime.json' with { type: 'json' };
import { hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
export function installHostHooks(test, expectedTypes = ['PromptManager', 'PromptManagerText', 'PromptSearchList']) {
test.beforeEach(async ({ page, context, request }, testInfo) => {
const response = await request.get('/system_stats');
expect(response.ok()).toBe(true);
const info = (await response.json()).system;
const stats = await response.json();
// comfyui.config.mjs's launcher reports the host it owns. The execution suite has no
// web server: it starts its own host per worker behind a port refusal.
if (testInfo.config.webServer) {
const ownedBase = process.env.PM_COMFY_HOST_BASE;
expect(ownedBase, 'the launcher reported the host it started').toBeTruthy();
expect(hostBaseDirectory(stats), 'the host answering the port is the one this run started').toBe(ownedBase);
}
const info = stats.system;
expect(info.comfyui_version).toBe(runtime.serverVersion);
expect(info.pytorch_version).toBe(`${runtime.torchVersion}+cpu`);
expect(info.required_frontend_version).toBe(runtime.frontendVersion);
+4 -4
View File
@@ -594,12 +594,12 @@
"anchor": null,
"expect": {
"metadata_status": "available",
"state": "ambiguous",
"reason": "Multiple sampler branches",
"state": "resolved",
"reason": "",
"reason_code": null,
"selection": null,
"positive": null,
"negative": null,
"positive": "a red fox in snow, detailed fur",
"negative": "blurry, lowres",
"candidate_count": 0,
"facts": {
"seed": {
File diff suppressed because one or more lines are too long
+17
View File
@@ -355,3 +355,20 @@ test("keepalive is forwarded only when asked, so a request can outlive its page"
for (const call of fetch.calls) call.resolve(jsonResponse(200, { success: true }));
await Promise.all(writes);
});
test("ApiError keeps a validated running job on busy and a validated source on invalid-source", () => {
const body = { running: { id: "a".repeat(32), title: "Scan for metadata" }, source: "output/gone" };
const busy = new ApiError(409, "busy", "busy", null, body);
assert.deepEqual([busy.running, busy.source], [{ id: "a".repeat(32), title: "Scan for metadata" }, null]);
assert.notEqual(busy.running, body.running, "server objects must be copied");
const invalid = new ApiError(400, "refused", "invalid-source", null, body);
assert.deepEqual([invalid.running, invalid.source], [null, "output/gone"]);
for (const running of [null, "x", { id: 1, title: "t" }, { id: "a b", title: "t" }, { id: "a", title: "" },
{ id: "a", title: "x".repeat(201) }, { id: "a".repeat(65), title: "t" }]) {
assert.equal(new ApiError(409, "busy", "busy", null, { running }).running, null);
}
for (const source of [null, 7, "", "x".repeat(4097)]) {
assert.equal(new ApiError(400, "refused", "invalid-source", null, { source }).source, null);
}
assert.deepEqual([new ApiError(409, "old caller").running, new ApiError(409, "old caller").source], [null, null]);
});
+47
View File
@@ -61,3 +61,50 @@ test("infinite scrolling is off by default and must be a boolean", () => {
assert.equal(normalizeGallerySettings({ infiniteScroll: true }).infiniteScroll, true);
assert.equal(normalizeGallerySettings({ infiniteScroll: "yes" }).infiniteScroll, false);
});
test("the workspace keys have their own defaults and validation", () => {
const { WORKSPACE_LAYOUTS, WORKSPACE_DENSITIES } = require("../../web/js/gallery-settings.js");
assert.equal(DEFAULT_SETTINGS.workspaceLayout, "grid");
assert.equal(DEFAULT_SETTINGS.workspaceDensity, "comfortable");
assert.deepEqual(WORKSPACE_LAYOUTS, ["grid", "list", "masonry"]);
assert.deepEqual(WORKSPACE_DENSITIES, ["comfortable", "large", "compact"]);
assert.equal(normalizeGallerySettings({ workspaceLayout: "masonry" }).workspaceLayout, "masonry");
assert.equal(normalizeGallerySettings({ workspaceLayout: "mosaic" }).workspaceLayout, "grid");
assert.equal(normalizeGallerySettings({ workspaceDensity: "large" }).workspaceDensity, "large");
assert.equal(normalizeGallerySettings({ workspaceDensity: 3 }).workspaceDensity, "comfortable");
// The legacy gallery's own view mode is separate: it has no masonry.
assert.equal(normalizeGallerySettings({ defaultViewMode: "masonry" }).defaultViewMode, "grid");
});
test("a legacy gallery Save keeps the workspace keys and every other stored key", () => {
const fs = require("node:fs");
const path = require("node:path");
const vm = require("node:vm");
const settings = require("../../web/js/gallery-settings.js");
const storage = new Map([["gallerySettings", JSON.stringify({ workspaceLayout: "masonry", workspaceDensity: "compact", someOtherTool: 1 })]]);
const controls = {
lazyLoadingToggle: { checked: true }, imageQualitySelect: { value: "high" }, defaultViewModeSelect: { value: "list" },
defaultLimitSelect: { value: "200" }, gridColumnsSelect: { value: "6" }, showImageInfoToggle: { checked: false },
autoLoadMetadataToggle: { checked: true }, cacheMetadataToggle: { checked: true }, showFilePathsToggle: { checked: false },
sidebarCollapsedByDefaultToggle: { checked: true }, infiniteScrollToggle: { checked: false }, debugModeToggle: { checked: false },
apiTimeoutInput: { value: "30" }, checkThumbnailsAtStartup: { checked: true }, videoAutoplayToggle: { checked: true },
videoMuteToggle: { checked: false }, videoLoopToggle: { checked: true },
};
const context = vm.createContext({
document: { addEventListener() {}, getElementById: id => controls[id] },
window: {}, console: { debug() {}, warn() {}, error() {} }, GallerySettings: settings,
localStorage: { getItem: key => storage.get(key) ?? null, setItem: (key, value) => storage.set(key, value) },
});
const file = path.join(__dirname, "../../web/js/gallery.js");
const Manager = vm.runInContext(`${fs.readFileSync(file, "utf8")}\nGalleryManager;`, context, { filename: file });
const manager = Object.create(Manager.prototype);
Object.assign(manager, { showNotification() {}, hideSettings() {}, applySettings() {} });
manager.saveSettings();
const saved = JSON.parse(storage.get("gallerySettings"));
assert.equal(saved.workspaceLayout, "masonry");
assert.equal(saved.workspaceDensity, "compact");
assert.equal(saved.someOtherTool, 1);
assert.equal(saved.defaultViewMode, "list");
assert.equal(saved.showImageInfo, false);
assert.equal(saved.videoMute, false);
});
+120
View File
@@ -110,3 +110,123 @@ test('saving gallery settings retains the remembered thumbnail dismissal', () =>
f.manager.saveSettings();
assert.equal(JSON.parse(f.storage.get('gallerySettings')).thumbnailPromptDismissed, 'remembered-state');
});
// #249 System S5: the stream's work is now an Activity job that outlives the page's connection, so the gallery's
// Cancel asks the server to stop the job. The stream's first frame, `job {id, started}`, names the job; the gallery
// cancels by that id, only a job this stream started, and a Cancel before the id waits for it (review I1).
const JOB = 'b'.repeat(32);
function streamFixture({ modal = false, postFails = false } = {}) {
const posts = [];
const warnings = [];
const gets = [];
const listeners = {};
const source = { closed: 0, readyState: 1, CLOSED: 2, url: null,
addEventListener(type, handler) { (listeners[type] ||= []).push(handler); },
close() { this.closed += 1; } };
const element = () => ({ classList: { add() {}, remove() {} }, style: {}, textContent: '', handlers: {},
addEventListener(type, handler) { this.handlers[type] = handler; },
removeEventListener(type, handler) { if (this.handlers[type] === handler) delete this.handlers[type]; },
click() { this.handlers.click?.(); } });
const elements = {};
const document = { addEventListener() {}, getElementById: id => elements[id] ||= element(),
createElement: () => element(), body: { appendChild() {}, removeChild() {} } };
const context = vm.createContext({ document, window: {},
console: { debug() {}, warn: (...args) => warnings.push(args), error() {} },
setTimeout() {}, localStorage: { setItem() {} },
SseStream: { connect: (url, options) => { source.url = url; source.options = options; return source; } } });
const file = path.join(__dirname, '../../web/js/gallery.js');
const Manager = vm.runInContext(fs.readFileSync(file, 'utf8') + '\nGalleryManager;', context, { filename: file });
const manager = Object.create(Manager.prototype);
Object.assign(manager, { getSettings: () => ({}),
api: { get: async url => { gets.push(url); return { success: true, jobs: [] }; },
post: async (url, body) => {
posts.push([url, body]);
if (postFails) throw new Error('offline');
return { success: true };
} } });
const emit = (type, data) => (listeners[type] || []).forEach(handler => handler({ data: JSON.stringify(data) }));
const ui = { addStatusMessage() {}, progressText: element(), progressPercent: element(), progressBar: element(),
progressDetails: element(), progressETA: element(), isCancelled: () => false,
onCancel: handler => { ui.cancel = handler; } };
const result = modal ? manager.generateThumbnailsWithProgressInModal(ui)
: manager.generateThumbnailsWithProgress({ quality: 'medium' });
const outcome = result.then(value => ({ value }), error => ({ error: error.message }));
const cancel = () => (modal ? ui.cancel() : elements.cancelThumbnailsBtn.click());
const settle = () => new Promise(resolve => setImmediate(resolve));
return { emit, source, posts, gets, warnings, outcome, cancel, settle };
}
const cancelUrl = id => `/prompt_manager/jobs/${id}/cancel`;
test('Cancel before the job id arrives cancels that job once it does', async () => {
const f = streamFixture();
f.cancel();
await f.settle();
assert.deepEqual(f.posts, []);
assert.equal(f.source.closed, 0); // kept open until the stream names its job
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
f.emit('status', { phase: 'scanning', message: 'Scanning…' });
f.emit('job', { id: JOB, started: true });
await f.settle();
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
assert.equal(f.source.closed, 1);
f.emit('job', { id: 'c'.repeat(32), started: true }); // a stray second frame changes nothing
await f.settle();
assert.equal(f.posts.length, 1);
assert.deepEqual(f.gets, []); // never looks a job up by kind
});
test('Cancel after the id cancels it at once, and a double Cancel posts once', async () => {
const f = streamFixture();
f.emit('job', { id: JOB, started: true });
f.cancel();
f.cancel();
await f.settle();
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
assert.equal(f.source.closed, 1);
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
assert.deepEqual(f.gets, []);
});
test("Cancel leaves another tab's or System's job running and only stops following it", async () => {
const f = streamFixture();
f.emit('job', { id: JOB, started: false });
f.cancel();
await f.settle();
assert.deepEqual(f.posts, []);
assert.deepEqual(f.gets, []);
assert.equal(f.source.closed, 1);
assert.match((await f.outcome).error, /keeps running/);
});
test('a Cancel the stream never names a job for posts nothing and closes when it ends', async () => {
const f = streamFixture();
f.cancel();
f.emit('error', { error: 'Scan for metadata already running' });
await f.settle();
assert.deepEqual(f.posts, []);
assert.equal(f.source.closed, 1);
});
test('a cancel post that fails is reported, not thrown', async () => {
const f = streamFixture({ postFails: true });
f.emit('job', { id: JOB, started: true });
f.cancel();
await f.settle();
assert.equal(f.posts.length, 1);
assert.equal(f.warnings.length, 1);
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
});
test('the startup dialog cancels its own job by id too', async () => {
const f = streamFixture({ modal: true });
f.cancel();
await f.settle();
assert.deepEqual(f.posts, []);
f.emit('job', { id: JOB, started: true });
await f.settle();
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
assert.equal(f.source.closed, 1);
assert.match((await f.outcome).error, /cancelled/);
});
+26
View File
@@ -82,6 +82,32 @@ test('a zeroed conditioning negative shows as an explicit empty negative without
}finally{scope.dispose();f.restore();}
});
test('a prompt that left out run-time trigger words says so under its text (D125)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div'),sent=[];
const note='Text added at run time, such as trigger words, is not included.';
try{
module.mountPromptBlocks(container,{parentScope:scope,sender:{open:roles=>sent.push(roles)},prompts:{positive:{status:'resolved',text:'own text',source:'runtime-parts-omitted'},negative:{status:'resolved',text:'blur',source:'graph-inferred'}}});
const notes=f.walk(container,node=>node.textContent===note);
assert.equal(notes.length,1,'one note, under the positive only');
assert.equal(notes[0].parent.attributes['aria-label'],'Positive prompt');
container.click(f.walk(container,node=>node.dataset.sendRole==='pair')[0]);
assert.deepEqual(sent,[{positive:'own text',negative:'blur'}],'the note is never sent');
}finally{scope.dispose();f.restore();}
});
test('a prompt a later stage added conditioning to says so, and both lines show together (D126)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div');
const added='Conditioning added at a later stage is not included.', runtime='Text added at run time, such as trigger words, is not included.';
try{
const view=module.mountPromptBlocks(container,{parentScope:scope,prompts:{positive:{status:'resolved',text:'base',source:'added-conditioning'}}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[added]);
view.update({positive:{status:'resolved',text:'base',source:'runtime-parts-omitted-and-added-conditioning'}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[runtime,added]);
view.update({positive:{status:'resolved',text:'base',source:'graph-inferred'}});
assert.equal(f.walk(container,node=>node.className==='small muted').length,0);
}finally{scope.dispose();f.restore();}
});
const unresolved=message=>({status:'unresolved',text:null,source:null,...(message===undefined?{}:{message})});
const statusLines=(f,container)=>f.walk(container,node=>node.tagName==='p'&&node.className==='prompt-text muted').map(node=>node.textContent);
test('unsupported node status names the class',()=>{
+148
View File
@@ -0,0 +1,148 @@
// Per-kind Activity descriptors (#249 System S5 Task 13, D111): each job kind's pill verb, toasts, line, tiles,
// outcomes, openable rows and labels, from one table (web/js/workspace/activity-kinds.mjs).
const test = require('node:test');
const assert = require('node:assert/strict');
const source = import('../../web/js/workspace/activity-kinds.mjs');
const store = import('../../web/js/workspace/activity-store.mjs');
const ID = 'a'.repeat(32);
function job(overrides = {}) {
return { id: ID, kind: 'scan', title: 'Scan for metadata', state: 'running', started_at: 1, finished_at: null,
progress: 42, status_line: '', processed: 8240, total: 19622,
counters: { added: 1, known: 0, review: 0, no_metadata: 0, errors: 1 }, scope: {}, listed: {}, log_length: 0,
cancellable: true, ...overrides };
}
test('scan descriptors are unchanged', async () => {
const { kindOf } = await source;
const scan = kindOf('scan');
assert.equal(scan.pillVerb, 'Scanning');
assert.equal(scan.startToast(job()), 'Scan started.');
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]);
assert.deepEqual(scan.outcomes, [['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
['no_metadata', 'No usable metadata'], ['known', 'Already known']]);
assert.deepEqual([...scan.openable].sort(), ['added', 'known']);
assert.deepEqual([scan.log, scan.cancelLabel], ['Scan log', 'Cancel scan']);
assert.equal(scan.finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error');
assert.equal(scan.finishToast(job({ state: 'cancelled' })), 'Scan cancelled · 1 added');
assert.equal(scan.finishToast(job({ state: 'failed' })), 'Scan failed · 1 added · 1 error');
assert.equal(scan.line(job()), '1 added · 1 error');
assert.equal(scan.kept(job()), '1 added');
assert.equal(scan.scope(job({ scope: { output_folders: 2 } })), '2 output folders');
// The store keeps answering with the same table.
const { jobView, finishToast, startToast, pillParts, outcomeLine } = await store;
assert.equal(jobView('scan'), scan);
assert.equal(finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error');
assert.equal(startToast(job()), 'Scan started.');
assert.equal(pillParts(job()).text, 'Scanning · 42% · 8,240 / 19,622');
assert.equal(outcomeLine(job()), '1 added · 1 error');
});
test('thumbnail job shows its own tiles and toasts', async () => {
const { kindOf, viewOf } = await source;
const { finishToast, startToast, pillParts, outcomeLine } = await store;
const thumbs = job({ kind: 'thumbnails', title: 'Thumbnail generation', processed: 4, total: 10, progress: 40,
counters: { generated: 1420, skipped: 3, errors: 1 }, scope: { mode: 'missing', size: 512, quality: 'medium',
output_folders: 2 } });
const view = kindOf('thumbnails');
assert.equal(viewOf(thumbs), view);
assert.deepEqual(view.tiles, [['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]);
assert.deepEqual(view.outcomes, [['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]);
assert.deepEqual([...view.openable].sort(), ['generated', 'skipped']);
assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']);
assert.equal(pillParts(thumbs).text, 'Generating thumbnails · 40% · 4 / 10');
assert.equal(startToast(thumbs), 'Thumbnail generation started.');
assert.equal(finishToast({ ...thumbs, state: 'finished' }), 'Thumbnails finished · 1,420 generated · 1 error');
assert.equal(finishToast({ ...thumbs, state: 'failed' }), 'Thumbnails failed · 1,420 generated · 1 error');
assert.equal(finishToast({ ...thumbs, state: 'cancelled' }), 'Thumbnails cancelled · 1,420 generated');
assert.equal(finishToast({ ...thumbs, counters: { generated: 0, skipped: 0, errors: 2 }, state: 'finished' }),
'Thumbnails finished · 0 generated · 2 errors');
assert.equal(outcomeLine(thumbs), '1,420 generated · 1 error');
assert.equal(view.kept(thumbs), '1,420 generated');
assert.equal(view.scope(thumbs), '2 output folders');
});
test('non-cancellable jobs show no Cancel job', async () => {
const { canCancel } = await source;
assert.equal(canCancel(job()), true);
assert.equal(canCancel(job({ cancellable: false })), false);
for (const state of ['finished', 'cancelled', 'failed', 'lost']) assert.equal(canCancel(job({ state })), false);
// The health check is a read job the server cannot stop.
assert.equal(canCancel(job({ kind: 'health', title: 'Health check', cancellable: false, counters: {} })), false);
});
test('indeterminate progress has no value attribute', async () => {
const { indeterminate, showProgress } = await source;
const { pillParts } = await store;
function node() {
const attributes = new Map();
return { attributes, textContent: '', set value(v) { attributes.set('value', String(v)); },
get value() { return Number(attributes.get('value') ?? 0); },
setAttribute(key, v) { attributes.set(key, String(v)); }, removeAttribute(key) { attributes.delete(key); } };
}
// Running with no total yet: the server has not said how much there is, so there is no percentage.
const health = job({ kind: 'health', title: 'Health check', progress: 0, processed: 0, total: 0, counters: {},
cancellable: false, status_line: 'Checking the database and image links' });
assert.equal(indeterminate(health), true);
const bar = node();
bar.value = 37;
showProgress(bar, health);
assert.equal(bar.attributes.has('value'), false);
assert.equal(bar.textContent, '');
assert.equal(pillParts(health).text, 'Health check');
assert.deepEqual(pillParts(health), { label: 'Health check', files: '', text: 'Health check' });
// A known total is determinate, and so is any ended job.
const scan = job();
assert.equal(indeterminate(scan), false);
showProgress(bar, scan);
assert.equal(bar.attributes.get('value'), '42');
assert.equal(bar.textContent, '42%');
assert.equal(indeterminate({ ...health, state: 'finished', progress: 100 }), false);
// A scan before its file count is known is indeterminate too, never "0%".
assert.equal(pillParts(job({ total: 0, processed: 0, progress: 0 })).text, 'Scanning');
});
test('held legacy jobs read "(admin page)" with no outcomes', async () => {
const { viewOf } = await source;
const { jobDetail } = await import('../../web/js/workspace/activity.mjs');
const { finishToast, outcomeLine } = await store;
// A hold has no outcome groups (counters {}), whatever its kind: thumbnails held by the gallery's clear included.
for (const [kind, title] of [['restore', 'Database restore (admin page)'],
['thumbnails', 'Thumbnail clear (gallery page)'], ['maintenance', 'Database maintenance (admin page)']]) {
const held = job({ kind, title, counters: {}, cancellable: false, progress: 0, processed: 0, total: 0 });
const view = viewOf(held);
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind);
assert.equal(outcomeLine(held), '');
assert.match(jobDetail(held), /^Started \d\d:\d\d · closing this view does not cancel it$/);
const ended = { ...held, state: 'finished', finished_at: 2 };
assert.match(jobDetail(ended), /^Finished \d\d:\d\d$/);
assert.equal(finishToast(ended), `${title} finished.`);
assert.ok(title.endsWith('page)'));
}
});
test('unknown and generic kinds show no tiles of another kind', async () => {
const { kindOf, viewOf } = await source;
for (const kind of ['maintenance', '', 'constructor', '__proto__', 'health']) {
const view = kindOf(kind);
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind);
assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']);
assert.equal(view.pillVerb, null);
assert.equal(view.line(job({ kind })), '');
assert.equal(view.kept(job({ kind })), '');
}
const backup = job({ kind: 'backup', title: 'Backup' });
assert.equal(kindOf('backup').startToast(backup), 'Backup started.');
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'finished' }), 'Backup finished.');
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'cancelled' }), 'Backup was cancelled.');
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'failed' }), 'Backup failed.');
// A kind's tiles are only the groups its job counts: no zero tile for a group the server never reported.
const partial = viewOf(job({ counters: { added: 3, errors: 0 } }));
assert.deepEqual(partial.tiles, [['added', 'Added to Library', 'added'], ['errors', 'Errors', 'bad']]);
assert.deepEqual(partial.outcomes, [['added', 'Added to Library'], ['errors', 'Errors']]);
assert.deepEqual([...partial.openable], ['added']);
assert.equal(partial.log, 'Scan log');
});
+7 -4
View File
@@ -289,8 +289,10 @@ test('summaries are validated and the copy follows the spec', async () => {
const scan = parseJob(summary({ progress: 42, processed: 8240, total: 19622, counters: { added: 1, errors: 1 } }));
assert.deepEqual(pillParts(scan), { label: 'Scanning · 42%', files: ' · 8,240 / 19,622', text: 'Scanning · 42% · 8,240 / 19,622' });
assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning · 42%');
assert.equal(pillParts({ ...scan, kind: 'thumbnails', title: 'Thumbnails' }).text, 'Thumbnails · 42% · 8,240 / 19,622');
// D111: without a total the progress is indeterminate, so the pill shows no percentage.
assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning');
// A kind without a pill verb shows its title (D111: thumbnails have their own, see workspace-activity-kinds.test.js).
assert.equal(pillParts({ ...scan, kind: 'backup', title: 'Backup' }).text, 'Backup · 42% · 8,240 / 19,622');
assert.equal(finishToast({ ...scan, state: 'finished' }), 'Scan finished · 1 added · 1 error');
assert.equal(finishToast({ ...scan, state: 'cancelled' }), 'Scan cancelled · 1 added');
assert.equal(finishToast({ ...scan, state: 'failed' }), 'Scan failed · 1 added · 1 error');
@@ -332,7 +334,8 @@ test('image-tags jobs get their own tiles outcomes and pill verb', async () => {
'Add tags to images finished · 0 tagged · 0 errors');
});
test('unknown kinds keep the generic view', async () => {
// D111 (S5): an unknown kind no longer borrows the scan's tiles (S1 review M6, S2 review M3); it has none.
test('unknown kinds get the generic view without the scan tiles', async () => {
const { jobView } = await source;
const scan = jobView('scan');
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
@@ -343,7 +346,7 @@ test('unknown kinds keep the generic view', async () => {
assert.deepEqual([scan.log, scan.cancel], ['Scan log', 'Cancel scan']);
for (const kind of ['maintenance', '', 'constructor', '__proto__']) {
const view = jobView(kind);
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [scan.tiles, scan.outcomes, [...scan.openable]]);
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []]);
assert.deepEqual([view.log, view.cancel], ['Job log', 'Cancel job']);
}
});
+16
View File
@@ -478,3 +478,19 @@ test('a refresh deferred behind a page load is dropped when the stream is dispos
await new Promise(resolve=>setImmediate(resolve));
assert.equal(requests.length,2,'no refresh runs after dispose');
});
test('defaults apply to a new visit but not to the current view or a saved view', async () => {
const { initialImagesView } = await source;
assert.equal(typeof initialImagesView, 'function', 'initialImagesView is exported');
const preferences = { workspaceLayout: 'masonry', workspaceDensity: 'compact' };
// A new visit starts from System › Appearance.
assert.deepEqual(initialImagesView({ preferences }), { layout: 'masonry', density: 'compact' });
// The current view (the last visit) keeps its own toolbar choices.
assert.deepEqual(initialImagesView({ prior: { layout: 'list', density: 'large' }, preferences }), { layout: 'list', density: 'large' });
assert.deepEqual(initialImagesView({ prior: {}, preferences }), { layout: 'grid', density: 'comfortable' });
// A saved view opens as it always did.
assert.deepEqual(initialImagesView({ saved: { q: '' }, preferences }), { layout: 'grid', density: 'comfortable' });
// Without readable preferences nothing changes.
assert.deepEqual(initialImagesView({}), { layout: 'grid', density: 'comfortable' });
assert.deepEqual(initialImagesView({ preferences: {} }), { layout: 'grid', density: 'comfortable' });
});
@@ -825,3 +825,73 @@ test('refresh re-reads the selected file and keeps the shown record until the ne
assert.equal(controller.state.record.identity.path, 'folder/b.png');
controller.dispose(); assert.equal(await controller.refresh(), undefined);
});
test('metadata off shows an explicit load and loads once on request', async () => {
const { api, requests } = pendingApi(), updates = [];
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
controller.subscribe(state => updates.push(state));
// Selecting reads nothing: the state says the metadata waits for a request.
assert.equal(controller.select(image()), undefined);
assert.equal(requests.length, 0);
assert.equal(controller.state.deferred, true);
assert.equal(controller.state.loading, false);
assert.equal(controller.state.record, null);
// Selecting the same file again still reads nothing.
controller.select({ ...image() });
assert.equal(requests.length, 0);
const load = controller.load();
assert.equal(requests.length, 1);
assert.equal(controller.state.loading, true);
assert.equal(controller.state.deferred, false);
// A second press while it reads, and the same file selected again once it loaded, read nothing more.
controller.load();
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await load;
assert.equal(controller.state.record.generation.seed.value, '9007199254740993');
controller.load(); controller.select(image());
assert.equal(requests.length, 1);
// Another file waits for its own request again.
controller.select({ ...image(), path: 'folder/b.png', filename: 'b.png', url: image().url.replace('a%20%23%25', 'b') });
assert.equal(controller.state.deferred, true);
assert.equal(requests.length, 1);
// The default controller loads on selection, as before.
const eager = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts });
eager.select(image());
assert.equal(requests.length, 2);
assert.equal(eager.state.deferred, false);
eager.dispose(); controller.dispose();
assert.equal(controller.load(), undefined);
});
test('cache off rereads the current selection; on reuses the one current record', async () => {
for (const cache of [true, false]) {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache });
const first = controller.select(image());
// A read in flight is shared either way.
controller.select(image());
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await first;
const again = controller.select(image());
assert.equal(requests.length, cache ? 1 : 2, `cache ${cache}`);
if (!cache) { requests[1].resolve(envelope()); await again; assert.equal(controller.state.record.identity.path, 'folder/a #%.png'); }
controller.dispose();
}
});
test('cache off never rereads a selection whose source changed, and refresh never loads a deferred one', async () => {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache: false });
const first = controller.select(image());
requests[0].reject(new ApiError(409, refreshMessage, 'source-changed')); await first;
assert.equal(controller.state.refreshRequired, true);
controller.select(image());
assert.equal(requests.length, 1);
controller.dispose();
const deferred = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
deferred.select(image());
assert.equal(deferred.refresh(), undefined);
assert.equal(requests.length, 1);
assert.equal(deferred.state.deferred, true);
deferred.dispose();
});
+11
View File
@@ -136,6 +136,17 @@ test('an unusable semantics table is refused', () => {
for (const doc of [null, {schema: 2}, {schema: 1, samplers: []}]) assert.throws(() => resolver.compileSemantics(doc), resolver.SemanticsError);
});
test('the lineage section is compiled, and a bad one refuses the table', () => {
const doc = JSON.parse(fs.readFileSync(new URL('../../web/js/inspection-semantics.json', import.meta.url), 'utf8'));
assert.deepEqual(semantics.lineage.get('Image Filter\u00001'), { strategy: 'continue', input: 'latents' });
// Lineage describes no prompt: its classes are not table classes for the stage walk.
assert.equal(semantics.allClasses.has('VAEEncode'), false);
const row = { class: 'VAEEncode', slot: 0, strategy: 'continue', input: 'pixels' };
for (const lineage of [undefined, {}, [null], [{ ...row, slot: -1 }], [{ ...row, strategy: 'passthrough' }], [{ ...row, input: '' }], [row, row]]) {
assert.throws(() => resolver.compileSemantics({ ...doc, lineage }), resolver.SemanticsError, JSON.stringify(lineage));
}
});
// The committed server answers (tests/fixtures/local-inspect/resolver-vectors.json, written by
// `python3 -m tests.local_resolver_cases --write` and checked current by the Python suite).
test('committed server answers equal the local evidence', () => {
+193
View File
@@ -0,0 +1,193 @@
// Browser display defaults (#249 System S3 Task 7): the workspace reads and writes the legacy gallery's own
// `gallerySettings` object (Q3, "One display configuration"), writes only the keys it changed, never consumes the
// legacy paging keys, and says when this browser refused to keep a change.
import test from 'node:test';
import assert from 'node:assert/strict';
import gallerySettings from '../../web/js/gallery-settings.js';
const module = await import('../../web/js/workspace/preferences.mjs').catch(error => {
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
return {};
});
globalThis.GallerySettings = gallerySettings;
const KEY = 'gallerySettings';
/** A localStorage stand-in holding `text` under the settings key; `setItem` can be made to throw. */
function storage(text = null, { quota = false } = {}) {
const writes = [];
const store = {
text,
writes,
getItem(key) { assert.equal(key, KEY); return store.text; },
setItem(key, value) {
assert.equal(key, KEY);
writes.push(value);
if (quota) { const error = new Error('Quota exceeded'); error.name = 'QuotaExceededError'; throw error; }
store.text = value;
},
removeItem() { assert.fail('Preferences never remove the shared settings'); },
};
return store;
}
const denied = () => { const error = new Error('Denied'); error.name = 'SecurityError'; throw error; };
test('normalizes new workspace keys and keeps legacy keys', () => {
assert.equal(typeof module.readPreferences, 'function', 'preferences.mjs exports readPreferences');
module.resetTabPreferences();
const legacy = { defaultViewMode: 'list', gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'sig', extra: 1 };
const store = storage(JSON.stringify({ ...legacy, workspaceLayout: 'masonry', workspaceDensity: 'compact', showFilePaths: false }));
const { values, persisted } = module.readPreferences(() => store);
assert.equal(persisted, true);
assert.equal(values.workspaceLayout, 'masonry');
assert.equal(values.workspaceDensity, 'compact');
assert.equal(values.showFilePaths, false);
// Invalid workspace values fall back to the defaults, as every legacy key does.
store.text = JSON.stringify({ workspaceLayout: 'mosaic', workspaceDensity: 7, videoMute: 'no' });
const fallback = module.readPreferences(() => store).values;
assert.equal(fallback.workspaceLayout, 'grid');
assert.equal(fallback.workspaceDensity, 'comfortable');
assert.equal(fallback.videoMute, true);
// The legacy normalizer knows the new keys and still keeps unknown ones.
const normalized = gallerySettings.normalizeGallerySettings({ ...legacy, workspaceLayout: 'list' });
assert.equal(normalized.workspaceLayout, 'list');
assert.equal(normalized.workspaceDensity, 'comfortable');
assert.equal(normalized.extra, 1);
assert.equal(normalized.thumbnailPromptDismissed, 'sig');
assert.deepEqual(gallerySettings.WORKSPACE_LAYOUTS, ['grid', 'list', 'masonry']);
assert.deepEqual(gallerySettings.WORKSPACE_DENSITIES, ['comfortable', 'large', 'compact']);
});
test('write touches only dirty keys', () => {
module.resetTabPreferences();
const stored = { defaultViewMode: 'list', gridColumns: 5, thumbnailsGenerated: true, extra: { kept: true }, videoMute: false };
const store = storage(JSON.stringify(stored));
const result = module.writePreferences({ workspaceLayout: 'masonry', showImageInfo: false }, () => store);
assert.deepEqual(result, { persisted: true, conflicts: [] });
assert.equal(store.writes.length, 1);
// Every other key, legacy or unknown, is written back exactly as it was stored: no defaults are filled in.
assert.deepEqual(JSON.parse(store.text), { ...stored, workspaceLayout: 'masonry', showImageInfo: false });
// It re-reads the stored object, so a key another page wrote after this one read is kept.
store.text = JSON.stringify({ ...JSON.parse(store.text), gridColumns: 9 });
module.writePreferences({ videoLoop: false }, () => store);
assert.equal(JSON.parse(store.text).gridColumns, 9);
assert.equal(JSON.parse(store.text).workspaceLayout, 'masonry');
// An empty or malformed stored value starts from an empty object, never the defaults.
for (const text of [null, '{broken', '[]', '42']) {
const fresh = storage(text);
assert.equal(module.writePreferences({ cacheMetadata: false }, () => fresh).persisted, true);
assert.deepEqual(JSON.parse(fresh.text), { cacheMetadata: false });
}
// Only workspace keys with valid values are accepted; nothing is written for a bad change.
for (const bad of [{ gridColumns: 4 }, { defaultLimit: 50 }, { workspaceLayout: 'mosaic' }, { videoMute: 'yes' }, { imageQuality: 'ultra' }]) {
const untouched = storage(JSON.stringify(stored));
assert.throws(() => module.writePreferences(bad, () => untouched), TypeError);
assert.equal(untouched.writes.length, 0);
}
// A key changed elsewhere since this tab read it (its baseline) is reported, and the explicit save still wins.
const elsewhere = storage(JSON.stringify({ videoLoop: false }));
const outcome = module.writePreferences({ videoLoop: true, videoMute: false }, () => elsewhere, { baseline: { videoLoop: true, videoMute: true } });
assert.deepEqual(outcome, { persisted: true, conflicts: ['videoLoop'] });
assert.equal(JSON.parse(elsewhere.text).videoLoop, true);
});
test('denied storage reports tab-only', () => {
module.resetTabPreferences();
const read = module.readPreferences(denied);
assert.equal(read.persisted, false);
assert.equal(read.values.workspaceLayout, 'grid');
const result = module.writePreferences({ workspaceLayout: 'list', videoAutoplay: true }, denied);
assert.deepEqual(result, { persisted: false, conflicts: [] });
// The change is kept for this tab: every reader in it (a new Images visit, the viewer) sees it.
const tab = module.readPreferences(denied);
assert.equal(tab.persisted, false);
assert.equal(tab.values.workspaceLayout, 'list');
assert.equal(tab.values.videoAutoplay, true);
// A getItem that throws is denied storage too.
const throwing = { getItem: denied, setItem: denied };
assert.equal(module.readPreferences(() => throwing).values.workspaceLayout, 'list');
assert.equal(module.writePreferences({ workspaceDensity: 'large' }, () => throwing).persisted, false);
// Once storage accepts a write, the tab copy of that key gives way to the stored value.
const working = storage(JSON.stringify({ workspaceLayout: 'masonry' }));
assert.equal(module.writePreferences({ workspaceLayout: 'masonry' }, () => working).persisted, true);
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
assert.equal(module.readPreferences(denied).values.videoAutoplay, false);
});
test('quota error keeps previous bytes', () => {
module.resetTabPreferences();
const before = JSON.stringify({ gridColumns: 6, workspaceDensity: 'large' });
const store = storage(before, { quota: true });
const result = module.writePreferences({ workspaceDensity: 'compact' }, () => store);
assert.deepEqual(result, { persisted: false, conflicts: [] });
assert.equal(store.writes.length, 1);
assert.equal(store.text, before);
// This tab keeps the change it could not store.
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'compact');
module.resetTabPreferences();
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'large');
});
/** A window stand-in that records its storage listeners. */
function fakeWindow() {
const listeners = new Set();
return {
listeners,
addEventListener(type, listener) { assert.equal(type, 'storage'); listeners.add(listener); },
removeEventListener(type, listener) { assert.equal(type, 'storage'); listeners.delete(listener); },
fire(event) { for (const listener of [...listeners]) listener(event); },
};
}
test('storage event updates clean values and flags dirty conflicts', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ workspaceLayout: 'list', videoMute: false }));
const win = fakeWindow();
const seen = [];
const unsubscribe = module.subscribePreferences(win, next => seen.push(next), () => store);
assert.equal(win.listeners.size, 1);
// Another key, or another storage area's event, is not ours.
win.fire({ key: 'pm_update_notice_dismissed', storageArea: store });
win.fire({ key: KEY, storageArea: { other: true } });
assert.equal(seen.length, 0);
win.fire({ key: KEY, storageArea: store });
win.fire({ key: null, storageArea: store }); // localStorage.clear() elsewhere
assert.equal(seen.length, 2);
assert.equal(seen[0].values.workspaceLayout, 'list');
unsubscribe();
assert.equal(win.listeners.size, 0);
// The reconciliation of the System drafts: clean values follow the other page; a key this tab has a draft for and
// the other page changed is a conflict, and the draft stays.
const baseline = { workspaceLayout: 'grid', videoMute: true, videoLoop: true };
const next = { workspaceLayout: 'list', videoMute: true, videoLoop: false };
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { videoMute: false, videoLoop: false }, next }), { conflicts: [] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'masonry' }, next }), { conflicts: ['workspaceLayout'] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'list' }, next }), { conflicts: [] });
});
test('legacy paging keys are never consumed by the workspace', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ defaultLimit: 500, gridColumns: 12, infiniteScroll: true, lazyLoading: false,
defaultViewMode: 'list', debugMode: true, apiTimeout: 9, thumbnailsGenerated: true, checkThumbnailsAtStartup: false }));
const { values } = module.readPreferences(() => store);
assert.deepEqual(Object.keys(values).sort(), [...module.WORKSPACE_PREFERENCE_KEYS].sort());
for (const key of ['defaultLimit', 'gridColumns', 'infiniteScroll', 'lazyLoading', 'defaultViewMode', 'debugMode', 'apiTimeout',
'thumbnailsGenerated', 'checkThumbnailsAtStartup', 'thumbnailPromptDismissed']) assert.equal(Object.hasOwn(values, key), false, key);
// The display defaults are the gallery's own defaults for these keys.
const defaults = module.preferenceDefaults();
for (const key of module.WORKSPACE_PREFERENCE_KEYS) assert.equal(defaults[key], gallerySettings.DEFAULT_SETTINGS[key], key);
});
test('a later save flushes the changes kept for this tab only', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ gridColumns: 4 }), { quota: true });
assert.equal(module.writePreferences({ workspaceLayout: 'list' }, () => store).persisted, false);
// Storage accepts writes again: saving another key stores the tab-only change with it, so "Saved" is true for both.
const working = storage(store.text);
assert.deepEqual(module.writePreferences({ videoLoop: false }, () => working), { persisted: true, conflicts: [] });
assert.deepEqual(JSON.parse(working.text), { gridColumns: 4, workspaceLayout: 'list', videoLoop: false });
// Nothing is overlaid any more: another page's later change reads through.
working.text = JSON.stringify({ ...JSON.parse(working.text), workspaceLayout: 'masonry' });
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
});
+788
View File
@@ -0,0 +1,788 @@
// Run with: node --test tests/js/workspace-review-state.test.js
const test = require('node:test');
const assert = require('node:assert/strict');
const { createApiClient } = require('../../web/js/api-client.js');
// Only an absent module is tolerated, so RED is an assertion failure, not a crash.
const source = import('../../web/js/workspace/review-state.mjs').catch(error => {
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
return {};
});
async function load() {
const module = await source;
assert.equal(typeof module.createReviewController, 'function', 'review-state.mjs is not implemented');
return module;
}
const EPOCH = '11111111-1111-4111-8111-111111111111';
const SCOPE = 'c'.repeat(64);
const SHA = 'd'.repeat(64);
const PRINT = 'e'.repeat(64);
const HASH_A = 'a'.repeat(64);
const HASH_B = 'b'.repeat(64);
const HASH_C = 'f'.repeat(64);
const ITEMS = '/prompt_manager/review/items';
const OBSERVE = '/prompt_manager/image-annotations/observe';
const CAPTURE = '/prompt_manager/image-annotations/capture/';
const flush = () => new Promise(resolve => setImmediate(resolve));
async function settle(times = 8) { for (let i = 0; i < times; i++) await flush(); }
function response(status, body) {
return { ok: status >= 200 && status < 300, status, statusText: `Status ${status}`, json: async () => body };
}
const fail = (status, code, error = `failure ${code}`, extra = {}) => response(status, { success: false, code, error, ...extra });
function item(id, overrides = {}) {
const path = `dunes/${id}.png`;
return {
id, state: 'pending', revision: 0, created_at: `2026-10-22T10:00:0${id.length}+00:00`, filename: `${id}.png`,
target: { root_index: 0, root_scope: SCOPE, path }, unavailable: false, candidate_count: 3, limited: false,
url: `/prompt_manager/images/serve/${path}?root=0&root_scope=${SCOPE}`, is_video: false, ...overrides,
};
}
const candidate = (hash, sampler, label, positive, negative, state = 'resolved') => ({
hash, sampler_id: sampler, state, reason: '', label,
positive: positive === null ? null : { text: positive }, negative: negative === null ? null : { text: negative },
});
function detailOf(entry, overrides = {}) {
return {
success: true, library_epoch: EPOCH, ...entry, total: 2, sha256: SHA, dependency_fingerprint: PRINT,
candidates: [
candidate(HASH_C, '9', 'No positive / Soft negative', null, 'blur', 'unresolved'),
candidate(HASH_A, '12', 'Dunes / Soft negative', 'golden dunes', 'blur'),
candidate(HASH_B, '28', 'Dunes / Harsh negative', 'golden dunes', 'noise'),
],
...overrides,
};
}
const page = (items, overrides = {}) => ({ success: true, library_epoch: EPOCH, total: items.length, items, next_cursor: null, ...overrides });
const observed = (token = 'capture-1') => ({
success: true, library_epoch: EPOCH, target: item('one').target, filename: 'one.png', title: 'one.png', tags: [], tag_revision: 0,
capture: { token, sha256: SHA, preview_url: CAPTURE + token, size: 10, width: 8, height: 8, media_type: 'image',
expires_at: '2026-10-22T12:00:00+00:00', generate: { supported: true, reason: null } },
});
/** Real createApiClient over a fetch whose answers come from routes['METHOD path'](call). */
function harness(routes = {}) {
const calls = [];
const fetch = (url, init = {}) => {
const call = { url, method: init.method, body: init.body === undefined ? undefined : JSON.parse(init.body) };
calls.push(call);
const path = url.startsWith(CAPTURE) ? CAPTURE : url.split('?')[0];
const route = routes[`${call.method} ${path}`];
return new Promise((resolve, reject) => {
init.signal?.addEventListener('abort', () => reject(new DOMException('Aborted', 'AbortError')));
Promise.resolve().then(() => (route ? route(call) : response(200, { success: true }))).then(resolve, reject);
});
};
return {
api: createApiClient(fetch), calls,
of: (method, prefix) => calls.filter(call => call.method === method && call.url.startsWith(prefix)),
};
}
const QUEUE = [item('one'), item('two')];
function routes(extra = {}) {
return {
[`GET ${ITEMS}`]: () => response(200, page(QUEUE)),
[`GET ${ITEMS}/one`]: () => response(200, detailOf(QUEUE[0])),
[`GET ${ITEMS}/two`]: () => response(200, detailOf(QUEUE[1], { candidates: [], limited: true, candidate_count: 0 })),
[`POST ${OBSERVE}`]: () => response(200, observed()),
[`POST ${ITEMS}/one/decision`]: call => response(200, { success: true, library_epoch: EPOCH, total: 1,
item: { ...QUEUE[0], state: call.body.choice === 'unresolved' ? 'unresolved' : 'selected', revision: 1 } }),
...extra,
};
}
async function loaded(extra) {
const { createReviewController } = await load();
const h = harness(routes(extra));
const c = createReviewController({ api: h.api });
await c.load();
await settle();
return { c, h };
}
test('notice titles for 0, 1 and 3', async () => {
const { noticeTitle } = await load();
assert.equal(noticeTitle(0), 'No images need a decision');
assert.equal(noticeTitle(1), '1 image needs a decision');
assert.equal(noticeTitle(3), '3 images need a decision');
assert.equal(noticeTitle(1234), '1,234 images need a decision');
});
test('loads pending items and the exact count', async () => {
const { c, h } = await loaded();
assert.equal(c.state.status, 'ready');
assert.equal(c.state.total, 2);
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
const [list] = h.of('GET', ITEMS);
assert.equal(list.url, `${ITEMS}?limit=40`);
// The first item is selected, as the concept selects its first queue image.
assert.equal(c.state.selectedId, 'one');
assert.ok(Object.isFrozen(c.state));
});
test('selecting a card loads detail and option labels, first workable preselected, unusable disabled', async () => {
const { c } = await loaded();
assert.equal(c.state.detailStatus, 'ready');
assert.deepEqual(c.state.options.map(option => [option.label, option.disabled]), [
['Sampler 9 · No positive / Soft negative', true],
['Sampler 12 · Dunes / Soft negative', false],
['Sampler 28 · Dunes / Harsh negative', false],
['Leave unresolved', false],
]);
assert.equal(c.state.options.at(-1).value, 'unresolved');
assert.equal(c.state.choice, HASH_A);
assert.deepEqual(c.state.prompts, { positive: { status: 'resolved', text: 'golden dunes' }, negative: { status: 'resolved', text: 'blur' } });
c.choose(HASH_C);
assert.equal(c.state.choice, HASH_A, 'a disabled option cannot be chosen');
c.choose(HASH_B);
assert.equal(c.state.prompts.negative.text, 'noise');
c.choose('unresolved');
assert.equal(c.state.prompts.positive.status, 'ambiguous');
await c.select('two');
await settle();
assert.equal(c.state.selectedId, 'two');
assert.deepEqual(c.state.options.map(option => option.label), ['Leave unresolved']);
assert.equal(c.state.choice, 'unresolved');
assert.match(c.state.warning, /too many sampler paths/i);
});
test('apply sends the selected candidate hash or unresolved with a fresh capture', async () => {
const { c, h } = await loaded();
c.choose(HASH_B);
const done = await c.apply();
assert.equal(done, true);
const [observe] = h.of('POST', OBSERVE);
assert.deepEqual(observe.body, { target: QUEUE[0].target });
const [decision] = h.of('POST', `${ITEMS}/one/decision`);
assert.deepEqual(decision.body, { expected_epoch: EPOCH, expected_revision: 0, capture_token: 'capture-1', sha256: SHA,
dependency_fingerprint: PRINT, choice: HASH_B });
assert.equal(c.state.phase, 'done');
assert.equal(c.state.total, 1);
assert.equal(c.state.toast, 'Association accepted.');
// The server releases a used capture itself.
assert.equal(h.of('DELETE', CAPTURE).length, 0);
const again = await loaded();
again.c.choose('unresolved');
assert.equal(await again.c.apply(), true);
const [unresolved] = again.h.of('POST', `${ITEMS}/one/decision`);
assert.equal(unresolved.body.choice, 'unresolved');
assert.equal(unresolved.body.capture_token, 'capture-1');
assert.equal(again.c.state.toast, 'Left unresolved.');
});
test('an unavailable file can only be left unresolved, without a capture', async () => {
const gone = item('one', { target: null, unavailable: true, url: null });
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page([gone])),
[`GET ${ITEMS}/one`]: () => response(200, detailOf(gone)),
});
assert.ok(c.state.options.slice(0, -1).every(option => option.disabled));
assert.equal(c.state.choice, 'unresolved');
assert.match(c.state.warning, /not under a current image source/i);
assert.equal(await c.apply(), true);
assert.equal(h.of('POST', OBSERVE).length, 0);
const [decision] = h.of('POST', `${ITEMS}/one/decision`);
assert.equal(decision.body.capture_token, null);
assert.equal(decision.body.sha256, SHA);
});
test('association conflict keeps the selection and shows the inline warning', async () => {
const message = 'This image is already linked to another prompt or negative, so this sampler path cannot be attached safely.';
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => fail(409, 'association-conflict', message) });
assert.equal(await c.apply(), false);
assert.equal(c.state.selectedId, 'one');
assert.equal(c.state.choice, HASH_A);
assert.equal(c.state.phase, 'idle');
assert.equal(c.state.warning, message);
// The refused decision's capture is released.
assert.equal(h.of('DELETE', CAPTURE).length, 1);
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'no reload');
});
test('storage-changed requires reload', async () => {
let lists = 0;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => { lists += 1; return response(200, page(QUEUE)); },
[`POST ${ITEMS}/one/decision`]: () => fail(409, 'storage-changed', 'Prompt storage was restored; reload the review queue.'),
});
assert.equal(await c.apply(), false);
await settle();
assert.equal(lists, 2, 'the queue is read again under the new epoch');
assert.equal(c.state.warning, 'Prompt storage was restored; reload the review queue.');
assert.equal(c.state.canRetry, false);
assert.equal(h.of('DELETE', CAPTURE).length, 1);
});
test('lost response keeps the selection and retry resends once', async () => {
let attempts = 0;
const { c, h } = await loaded({
[`POST ${ITEMS}/one/decision`]: call => {
attempts += 1;
if (attempts === 1) return Promise.reject(new TypeError('Failed to fetch'));
return response(200, { success: true, library_epoch: EPOCH, total: 1, item: { ...QUEUE[0], state: 'selected', revision: 1 } });
},
});
assert.equal(await c.apply(), false);
assert.equal(c.state.selectedId, 'one');
assert.equal(c.state.canRetry, true);
assert.match(c.state.warning, /may not have been saved/i);
assert.equal(h.of('DELETE', CAPTURE).length, 0, 'the capture is kept for the retry');
const [first] = h.of('POST', `${ITEMS}/one/decision`);
assert.equal(await c.retry(), true);
const sent = h.of('POST', `${ITEMS}/one/decision`);
assert.equal(sent.length, 2);
assert.deepEqual(sent[1].body, first.body, 'the identical decision is resent');
assert.equal(h.of('POST', OBSERVE).length, 1, 'no second capture');
assert.equal(await c.retry(), false, 'nothing left to resend');
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 2);
});
test('a retry answered by review-conflict reloads the queue', async () => {
let attempts = 0, lists = 0;
const { c } = await loaded({
[`GET ${ITEMS}`]: () => { lists += 1; return response(200, page(lists === 1 ? QUEUE : [QUEUE[1]])); },
[`POST ${ITEMS}/one/decision`]: () => {
attempts += 1;
if (attempts === 1) return Promise.reject(new TypeError('Failed to fetch'));
return fail(409, 'review-conflict', 'This item was decided or changed since it was opened. Reload the queue.');
},
});
await c.apply();
assert.equal(await c.retry(), false);
await settle();
assert.equal(lists, 2);
assert.equal(c.state.selectedId, 'two');
});
test('a review-conflict on a still pending item reads its detail again', async () => {
let details = 0, decisions = 0;
const { c, h } = await loaded({
[`GET ${ITEMS}/one`]: () => { details += 1; return response(200, detailOf({ ...QUEUE[0], revision: details - 1 })); },
[`POST ${ITEMS}/one/decision`]: () => {
decisions += 1;
return decisions === 1 ? fail(409, 'review-conflict', 'This item was decided or changed since it was opened. Reload the queue.')
: response(200, { success: true, library_epoch: EPOCH, total: 1, item: { ...QUEUE[0], state: 'selected', revision: 2 } });
},
});
assert.equal(await c.apply(), false);
await settle();
assert.equal(details, 2, 'the item is read again');
assert.equal(c.state.detail.revision, 1);
assert.equal(c.state.warning, 'This item was decided or changed since it was opened. Reload the queue.');
assert.equal(await c.apply(), true);
assert.equal(h.of('POST', `${ITEMS}/one/decision`).at(-1).body.expected_revision, 1);
});
test('leave for later writes nothing', async () => {
const { c, h } = await loaded();
c.leave();
assert.equal(c.state.phase, 'done');
assert.equal(c.state.toast, null);
assert.equal(h.of('POST', '/').length, 0);
assert.equal(h.of('DELETE', '/').length, 0);
});
test('load more pages with the cursor and keeps the selection', async () => {
const more = [item('three')];
const { c, h } = await loaded({
[`GET ${ITEMS}`]: call => response(200, call.url.includes('cursor=')
? page(more, { total: 3 }) : page(QUEUE, { total: 3, next_cursor: 'abc_-1' })),
});
assert.equal(c.state.nextCursor, 'abc_-1');
await c.loadMore();
assert.equal(h.of('GET', ITEMS).at(-1).url, `${ITEMS}?limit=40&cursor=abc_-1`);
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two', 'three']);
assert.equal(c.state.nextCursor, null);
assert.equal(c.state.selectedId, 'one');
});
test('a failed queue read is an error state and load retries', async () => {
let lists = 0;
const { c } = await loaded({ [`GET ${ITEMS}`]: () => { lists += 1; return lists === 1 ? fail(500, 'review-failed') : response(200, page(QUEUE)); } });
assert.equal(c.state.status, 'error');
await c.load();
await settle();
assert.equal(c.state.status, 'ready');
});
test('tag save toast carries the review suffix and the item stays pending', async () => {
const { c, h } = await loaded({
['GET /prompt_manager/autotag/models']: () => response(200, { success: true, models: {}, default_prompt: 'x',
wd14_general_threshold: 0.35, wd14_character_threshold: 0.85, suggestion_limit: 10 }),
['PUT /prompt_manager/image-annotations/tags']: call => response(200, { success: true, library_epoch: EPOCH, sha256: SHA,
tags: call.body.tags, tag_revision: 1, replayed: false }),
});
const { createImageTaggingController } = await import('../../web/js/workspace/image-tagging-state.mjs');
const before = h.calls.filter(call => call.url.startsWith(ITEMS)).length;
// The Tag dialog opened from Needs review is the shared controller with the review source.
const tagging = createImageTaggingController({ api: h.api, target: QUEUE[0].target, source: 'review' });
assert.equal(tagging.state.showScopeNote, true);
await tagging.open();
await settle();
tagging.setManual('dunes');
await tagging.save();
await settle();
assert.equal(tagging.state.toast, '1 tags saved for one.png. Prompt association still awaits review.');
tagging.dispose();
assert.equal(h.calls.filter(call => call.url.startsWith(ITEMS)).length, before, 'a tag save neither decides nor re-reads the queue');
assert.deepEqual(c.state.items.map(entry => [entry.id, entry.state]), [['one', 'pending'], ['two', 'pending']]);
assert.equal(c.state.phase, 'idle');
});
test('dispose releases a kept capture and stops reads', async () => {
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => Promise.reject(new TypeError('Failed to fetch')) });
await c.apply();
c.dispose();
await settle();
const released = h.of('DELETE', CAPTURE);
assert.equal(released.length, 1);
assert.equal(released[0].url, CAPTURE + 'capture-1');
assert.equal(await c.retry(), false);
});
// F2: an open page re-reads its queue when the server says it changed (the review change stream).
function refreshOf(c) {
assert.equal(typeof c.refresh, 'function', 'refresh is not implemented');
return c.refresh();
}
test('refresh re-reads the queue quietly and keeps the selection and choice', async () => {
let queue = QUEUE;
const { c, h } = await loaded({ [`GET ${ITEMS}`]: () => response(200, page(queue)) });
c.choose(HASH_B);
const statuses = [];
c.subscribe(state => statuses.push(state.status));
queue = [...QUEUE, item('three')];
await refreshOf(c);
await settle();
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two', 'three']);
assert.equal(c.state.total, 3);
assert.equal(c.state.selectedId, 'one');
assert.equal(c.state.choice, HASH_B);
assert.equal(c.state.warning, '');
assert.ok(statuses.every(status => status === 'ready'), `no loading state: ${statuses}`);
assert.equal(h.of('GET', `${ITEMS}/one`).length, 1, 'the open item is not read again');
});
test('refreshes asked for during a refresh collapse into one more read', async () => {
let release;
const held = new Promise(resolve => { release = resolve; });
let lists = 0;
const { c, h } = await loaded({ [`GET ${ITEMS}`]: async () => {
lists += 1;
if (lists === 2) await held;
return response(200, page(QUEUE));
} });
const first = refreshOf(c);
await settle();
refreshOf(c);
refreshOf(c);
release();
await first;
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 3, 'the load, the refresh and one collapsed re-read');
assert.equal(c.state.status, 'ready');
});
test('a refresh after the open item was decided elsewhere opens the next one and says so', async () => {
let queue = QUEUE;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue)),
[`GET ${ITEMS}/three`]: () => response(200, detailOf(item('three'))),
});
queue = [item('three')];
await refreshOf(c);
await settle();
assert.equal(c.state.selectedId, 'three');
assert.equal(c.state.detailStatus, 'ready');
assert.equal(c.state.warning, 'one.png was decided elsewhere, so the next image is open.');
assert.equal(h.of('GET', `${ITEMS}/three`).length, 1);
});
test('a refresh to an empty queue clears the selection', async () => {
let queue = QUEUE;
const { c } = await loaded({ [`GET ${ITEMS}`]: () => response(200, page(queue)) });
queue = [];
await refreshOf(c);
await settle();
assert.equal(c.state.total, 0);
assert.deepEqual(c.state.items, []);
assert.equal(c.state.selectedId, null);
assert.equal(c.state.status, 'ready');
});
test('a failed refresh keeps the queue and says so', async () => {
let lists = 0;
const { c } = await loaded({ [`GET ${ITEMS}`]: () => { lists += 1; return lists === 1 ? response(200, page(QUEUE)) : fail(500, 'review-failed'); } });
await refreshOf(c);
await settle();
assert.equal(c.state.status, 'ready');
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
assert.equal(c.state.warning, 'The review queue could not be read again. Reload the page to see the latest images.');
});
test('dispose during a decision releases its capture', async () => {
const { c, h } = await loaded({ [`POST ${ITEMS}/one/decision`]: () => new Promise(() => {}) });
const applying = c.apply();
await settle();
assert.equal(c.state.phase, 'applying');
c.dispose();
assert.equal(await applying, false);
await settle();
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'capture-1']);
await refreshOf(c);
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'a disposed controller reads nothing');
});
test('a change heard during the first read re-reads once after it, and a failed first read stays an error', async () => {
const { createReviewController } = await load();
let release;
let lists = 0;
const h = harness(routes({ [`GET ${ITEMS}`]: async () => {
lists += 1;
if (lists === 1) await new Promise(resolve => { release = resolve; });
return response(200, page(QUEUE));
} }));
const c = createReviewController({ api: h.api });
const first = c.load();
await settle();
refreshOf(c);
assert.equal(lists, 1, 'the first read is not interrupted');
release();
await first;
await settle();
assert.equal(lists, 2, 'one re-read after the first read');
assert.equal(c.state.status, 'ready');
const failing = harness(routes({ [`GET ${ITEMS}`]: () => fail(500, 'review-failed') }));
const broken = createReviewController({ api: failing.api });
const reading = broken.load();
refreshOf(broken);
await reading;
await settle();
assert.equal(broken.state.status, 'error');
assert.equal(failing.of('GET', `${ITEMS}?`).length, 1, 'Retry is the way back, not a re-read');
});
// F3 rulings: a live re-read keeps the loaded depth, and the stream's `ready` reads nothing when the first read is current.
/** A keyset pager over `source()`: pages of 40 with an index cursor, each answering `version()`. */
function pager(source, version = () => 0, { onPage } = {}) {
return call => {
const query = new URL(call.url, 'http://x').searchParams;
const cursor = query.get('cursor');
const all = source();
const from = cursor ? Number(cursor.slice(1)) : 0;
const answer = onPage?.(cursor);
if (answer) return answer;
const items = all.slice(from, from + 40);
const next = from + 40 < all.length ? `c${from + 40}` : null;
return response(200, page(items, { total: all.length, next_cursor: next, review_revision: version() }));
};
}
const many = (count, prefix = 'i') => Array.from({ length: count }, (_, index) => item(`${prefix}${String(index).padStart(3, '0')}`));
function syncOf(c, version) {
assert.equal(typeof c.sync, 'function', 'sync is not implemented');
return c.sync(version);
}
const version = revision => ({ library_epoch: EPOCH, review_revision: revision });
test('a refresh keeps the loaded depth: it re-reads as many items as were loaded', async () => {
let queue = many(100);
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue) });
await c.loadMore();
assert.equal(c.state.items.length, 80);
const before = h.of('GET', `${ITEMS}?`).length;
queue = [...queue, item('late')];
await refreshOf(c);
await settle();
assert.equal(c.state.items.length, 80, 'the Load more page is kept');
assert.deepEqual(c.state.items.map(entry => entry.id), queue.slice(0, 80).map(entry => entry.id));
assert.equal(c.state.total, 101);
assert.equal(c.state.nextCursor, 'c80', 'Load more continues after the kept depth');
assert.deepEqual(h.of('GET', `${ITEMS}?`).slice(before).map(call => call.url),
[`${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`]);
assert.equal(c.state.selectedId, 'i000');
});
test('a refresh after decisions elsewhere fills the loaded depth from what is left', async () => {
let queue = many(85);
const { c } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue) });
await c.loadMore();
await c.loadMore();
assert.equal(c.state.items.length, 85);
queue = queue.filter((_, index) => index % 10 !== 5);
await refreshOf(c);
await settle();
assert.equal(c.state.items.length, queue.length);
assert.equal(c.state.nextCursor, null);
});
test('a deep refresh whose next page meets a restore reads again from the first page', async () => {
let restored = false;
let queue = many(60);
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => queue, () => 0, {
onPage: cursor => {
if (cursor && restored) { restored = false; return fail(409, 'storage-changed', 'The library was restored.'); }
return null;
},
}) });
await c.loadMore();
const before = h.of('GET', `${ITEMS}?`).length;
restored = true;
queue = many(60, 'r');
await refreshOf(c);
await settle();
assert.deepEqual(h.of('GET', `${ITEMS}?`).slice(before).map(call => call.url),
[`${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`, `${ITEMS}?limit=40`, `${ITEMS}?limit=40&cursor=c40`]);
assert.equal(c.state.items.length, 60);
assert.equal(c.state.items[0].id, 'r000');
assert.notEqual(c.state.warning, 'The review queue could not be read again. Reload the page to see the latest images.',
'a restore met mid-read is not a failed re-read');
assert.equal(c.state.status, 'ready');
});
test('ready naming the version the first read answered reads nothing more', async () => {
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => QUEUE, () => 4) });
await syncOf(c, version(4));
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'one list read per visit');
assert.equal(h.of('GET', `${ITEMS}/one`).length, 1);
});
test('ready naming a newer version, another epoch or no version re-reads once', async () => {
const { c, h } = await loaded({ [`GET ${ITEMS}`]: pager(() => QUEUE, () => 4) });
await syncOf(c, version(5));
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 2);
await syncOf(c, { library_epoch: '22222222-2222-4222-8222-222222222222', review_revision: 4 });
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 3);
await syncOf(c, {});
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 4, 'an unknown version is read, as before versions existed');
});
test('ready heard during the first read re-reads only when that read is older', async () => {
const { createReviewController } = await load();
for (const [readRevision, readyRevision, reads] of [[4, 4, 1], [5, 4, 1], [4, 5, 2]]) {
let release;
let lists = 0;
const answer = pager(() => QUEUE, () => (lists === 1 ? readRevision : readyRevision));
const h = harness(routes({ [`GET ${ITEMS}`]: async call => {
lists += 1;
if (lists === 1) await new Promise(resolve => { release = resolve; });
return answer(call);
} }));
const c = createReviewController({ api: h.api });
const first = c.load();
await settle();
syncOf(c, version(readyRevision));
release();
await first;
await settle();
assert.equal(lists, reads, `first read r${readRevision}, ready r${readyRevision}`);
c.dispose();
}
});
test('a change heard during the first read still re-reads after a current ready', async () => {
const { createReviewController } = await load();
let release;
let lists = 0;
const h = harness(routes({ [`GET ${ITEMS}`]: async call => {
lists += 1;
if (lists === 1) await new Promise(resolve => { release = resolve; });
return pager(() => QUEUE, () => 4)(call);
} }));
const c = createReviewController({ api: h.api });
const first = c.load();
await settle();
syncOf(c, version(4));
refreshOf(c);
release();
await first;
await settle();
assert.equal(lists, 2);
});
// Final review round (I1, I2, M3–M5, M7).
test('I1: a change heard during a refused Apply is re-read once the phase is idle again', async () => {
let queue = [item('one')], releaseDecision;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue, { review_revision: queue.length })),
[`POST ${ITEMS}/one/decision`]: () => new Promise(resolve => {
releaseDecision = () => resolve(fail(409, 'association-conflict', 'conflict'));
}),
});
const applying = c.apply();
await settle();
assert.equal(c.state.phase, 'applying');
queue = [item('one'), item('two')]; // the monitor records an item while the decision is in flight
c.refresh();
await settle();
assert.equal(h.of('GET', `${ITEMS}?`).length, 1, 'no read while the decision is in flight');
releaseDecision();
await applying;
await settle();
assert.equal(c.state.phase, 'idle');
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
assert.equal(c.state.total, 2);
assert.equal(c.state.warning, 'conflict', 'the refusal stays the warning');
assert.equal(h.of('GET', `${ITEMS}?`).length, 2, 'exactly one owed re-read');
});
test('I1: a change heard during an Apply whose observe is refused is re-read too', async () => {
let queue = [item('one')], releaseObserve;
const { c } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue)),
[`POST ${OBSERVE}`]: () => new Promise(resolve => {
releaseObserve = () => resolve(fail(409, 'capture-capacity', 'busy'));
}),
});
const applying = c.apply();
await settle();
queue = [item('one'), item('two')];
c.refresh();
releaseObserve();
await applying;
await settle();
assert.deepEqual(c.state.items.map(entry => entry.id), ['one', 'two']);
});
const forty = Array.from({ length: 41 }, (_, i) => item(`i${String(i).padStart(2, '0')}`));
function deepRoutes(state) {
return {
[`GET ${ITEMS}`]: call => {
const cursor = new URL(call.url, 'http://x').searchParams.get('cursor');
if (cursor) {
if (state.moreGate) return state.moreGate.then(() => state.more ?? response(200, page([forty[40]], { total: 41 })));
return state.more ?? response(200, page([forty[40]], { total: 41 }));
}
const body = response(200, page(state.first, { total: state.first.length + 1, review_revision: state.revision ?? 1, next_cursor: 'cur' }));
return state.gate ? state.gate.then(() => state.fail ?? body) : body;
},
[`GET ${ITEMS}/i00`]: () => response(200, detailOf(forty[0])),
};
}
test('I2: Load more during a re-read waits for it and continues from the re-read queue', async () => {
const state = { first: forty.slice(0, 40) };
const { c } = await loaded(deepRoutes(state));
// Another tab decides i05; the re-read is slow and the user clicks Load more meanwhile.
let open;
state.first = forty.slice(0, 40).filter(entry => entry.id !== 'i05');
state.gate = new Promise(resolve => { open = resolve; });
const refreshing = c.refresh();
await settle();
const more = c.loadMore();
await settle();
assert.equal(c.state.more, 'loading', 'the click shows as loading while the re-read runs');
open();
await Promise.all([refreshing, more]);
await settle();
const ids = c.state.items.map(entry => entry.id);
assert.ok(!ids.includes('i05'), 'the re-read was applied');
assert.equal(ids.at(-1), 'i40', 'Load more appended the next page after the re-read');
assert.equal(c.state.more, 'idle');
});
test('I2: a re-read started during Load more takes over the click and reads one page deeper', async () => {
const state = { first: forty.slice(0, 40) };
const { c, h } = await loaded(deepRoutes(state));
let openMore;
state.moreGate = new Promise(resolve => { openMore = resolve; });
const more = c.loadMore();
await settle();
state.moreGate = null;
state.first = forty.slice(0, 40).filter(entry => entry.id !== 'i05');
await c.refresh();
openMore();
await more;
await settle();
const ids = c.state.items.map(entry => entry.id);
assert.ok(!ids.includes('i05'));
assert.equal(ids.at(-1), 'i40', 'the aborted Load more page is part of the re-read');
assert.equal(c.state.more, 'idle');
assert.equal(h.of('GET', `${ITEMS}?`).length, 4, 'load, the aborted more, and the re-read of two pages');
});
test('M7: a failed re-read that took over Load more clears its loading state', async () => {
const state = { first: forty.slice(0, 40) };
const { c } = await loaded(deepRoutes(state));
state.moreGate = new Promise(() => {});
c.loadMore();
await settle();
assert.equal(c.state.more, 'loading');
state.moreGate = null;
state.gate = Promise.resolve();
state.fail = fail(500, 'review-failed');
await c.refresh();
await settle();
assert.equal(c.state.more, 'idle', 'Load more can be clicked again');
assert.equal(c.state.warning, COPY_REFRESH_FAILED);
});
const COPY_REFRESH_FAILED = 'The review queue could not be read again. Reload the page to see the latest images.';
test('M3: Apply fences on the epoch of the detail it decides', async () => {
const OTHER = '22222222-2222-4222-8222-222222222222';
const { c, h } = await loaded({ [`POST ${OBSERVE}`]: () => response(200, { ...observed(), library_epoch: OTHER }) });
assert.equal(await c.apply(), false);
await settle();
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 0, 'a restored storage sends no decision');
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'capture-1']);
assert.equal(h.of('GET', `${ITEMS}?`).length, 2, 'the queue is read again');
assert.equal(h.of('GET', `${ITEMS}/one`).length, 2, 'the open item is read again under the new storage');
assert.equal(c.state.phase, 'idle');
});
test('M3: the decision carries the detail epoch', async () => {
const { c, h } = await loaded();
assert.equal(await c.apply(), true);
assert.equal(h.of('POST', `${ITEMS}/one/decision`)[0].body.expected_epoch, EPOCH);
});
test('M3: a quiet re-read under another epoch reads the open item again', async () => {
const OTHER = '22222222-2222-4222-8222-222222222222';
let epoch = EPOCH;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(QUEUE, { library_epoch: epoch })),
[`GET ${ITEMS}/one`]: () => response(200, detailOf(QUEUE[0], { library_epoch: epoch })),
});
epoch = OTHER;
await c.refresh();
await settle();
assert.equal(h.of('GET', `${ITEMS}/one`).length, 2);
assert.equal(c.state.detail.library_epoch, OTHER);
});
test('M4: leaving during the observe releases the capture it then answers', async () => {
let answer;
const { c, h } = await loaded({ [`POST ${OBSERVE}`]: () => new Promise(resolve => { answer = resolve; }) });
const applying = c.apply();
await settle();
c.dispose();
answer(response(200, observed('late')));
assert.equal(await applying, false);
await settle();
assert.deepEqual(h.of('DELETE', CAPTURE).map(call => call.url), [CAPTURE + 'late']);
assert.equal(h.of('POST', `${ITEMS}/one/decision`).length, 0);
});
test('M5: an unanswered decision that landed finishes as recorded, not as decided elsewhere', async () => {
let queue = QUEUE;
const { c, h } = await loaded({
[`GET ${ITEMS}`]: () => response(200, page(queue)),
[`POST ${ITEMS}/one/decision`]: () => Promise.reject(new TypeError('Failed to fetch')),
});
await c.apply();
assert.equal(c.state.canRetry, true);
queue = [QUEUE[1]]; // the server did record it: its change arrives
await c.refresh();
await settle();
assert.equal(c.state.phase, 'done');
assert.equal(c.state.toast, 'Association accepted.');
assert.doesNotMatch(c.state.warning, /decided elsewhere/);
assert.equal(h.of('DELETE', CAPTURE).length, 0, 'the server used the capture');
});
+33 -10
View File
@@ -4,7 +4,7 @@ const source = import('../../web/js/workspace/state.mjs');
test('workspace routes resolve only supported deep links', async () => {
const { resolveRoute } = await source;
for (const route of ['overview', 'images', 'library', 'tags', 'inspect', 'system', 'activity']) {
for (const route of ['overview', 'images', 'library', 'tags', 'inspect', 'system', 'activity', 'review']) {
assert.equal(resolveRoute(`#/${route}`).id, route);
}
for (const hash of ['', '#', '#/unknown', '#/images/other', '#/<script>', '#/activity/abc', `#/activity/${'A'.repeat(32)}`]) {
@@ -24,7 +24,7 @@ test('tags is a native route titled Tags; only unbuilt routes hand off to legacy
}
assert.equal(routeTitle(resolveRoute('#/library/7')), 'Prompt library');
// System is native from S2 (#249 System): its undelivered sections hand off per section (D94), not the route.
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'review', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
});
test('system resolves its sections; #/overview stays the production hub', async () => {
@@ -157,9 +157,10 @@ test('positive library deep links preserve the selected identity without duplica
test('nav counts reject negative, non-integer and missing totals', async () => {
const { summarizeNavCounts } = await source;
const counts = summarizeNavCounts({images: {total: 24}, favorites: {total: 3}, library: {total: 0}, tags: {total: 7}});
const counts = summarizeNavCounts({images: {total: 24}, favorites: {total: 3}, library: {total: 0}, tags: {total: 7},
review: {total: 3}});
assert.ok(counts instanceof Map);
assert.deepEqual([...counts], [['images', 24], ['favorites', 3], ['library', 0], ['tags', 7]]);
assert.deepEqual([...counts], [['images', 24], ['favorites', 3], ['library', 0], ['tags', 7], ['review', 3]]);
assert.equal(counts.has('inspect'), false, 'Inspect a file has no count');
for (const invalid of [undefined, null, {}, {total: -1}, {total: 1.5}, {total: '3'}, {total: NaN},
{total: Infinity}, {total: Number.MAX_SAFE_INTEGER + 1}, {total: null}, [], 'x']) {
@@ -169,7 +170,8 @@ test('nav counts reject negative, non-integer and missing totals', async () => {
assert.equal(result.get('library'), 2);
assert.equal(result.get('tags'), null);
}
assert.deepEqual([...summarizeNavCounts()], [['images', null], ['favorites', null], ['library', null], ['tags', null]]);
assert.deepEqual([...summarizeNavCounts()], [['images', null], ['favorites', null], ['library', null], ['tags', null],
['review', null]]);
});
test('nav counts settle independently so a slow images read never delays the others', async () => {
@@ -185,14 +187,14 @@ test('nav counts settle independently so a slow images read never delays the oth
const settled = [];
const done = loadNavCounts(api, new AbortController().signal, (id, count) => settled.push([id, count]));
await new Promise(resolve => setTimeout(resolve, 0));
assert.deepEqual(settled.sort(), [['favorites', 2], ['library', 3], ['tags', null]]);
assert.deepEqual(settled.sort(), [['favorites', 2], ['library', 3], ['review', null], ['tags', null]]);
releaseImages({success: true, total: 40});
const counts = await done;
assert.deepEqual(settled.at(-1), ['images', 40]);
assert.deepEqual([...counts], [['images', 40], ['favorites', 2], ['library', 3], ['tags', null]]);
assert.deepEqual([...counts], [['images', 40], ['favorites', 2], ['library', 3], ['tags', null], ['review', null]]);
});
test('nav counts issue four bounded requests and drop only the failed count', async () => {
test('nav counts issue five bounded requests and drop only the failed count', async () => {
const { loadNavCounts } = await source;
const calls = [];
const signal = new AbortController().signal;
@@ -200,13 +202,15 @@ test('nav counts issue four bounded requests and drop only the failed count', as
calls.push([url, options.signal]);
if (url.startsWith('/prompt_manager/library')) throw new Error('Unavailable');
if (url.startsWith('/prompt_manager/vocabulary/tags')) return {success: true, total: 5};
if (url.startsWith('/prompt_manager/review/items')) return {success: true, total: 2};
return {success: true, total: 12};
}};
const counts = await loadNavCounts(api, signal);
assert.deepEqual([...counts], [['images', 12], ['favorites', 12], ['library', null], ['tags', 5]]);
assert.deepEqual([...counts], [['images', 12], ['favorites', 12], ['library', null], ['tags', 5], ['review', 2]]);
const urls = calls.map(([url]) => new URL(url, 'http://host'));
assert.deepEqual(urls.map(url => url.pathname),
['/prompt_manager/images/output', '/prompt_manager/images/output', '/prompt_manager/library', '/prompt_manager/vocabulary/tags']);
['/prompt_manager/images/output', '/prompt_manager/images/output', '/prompt_manager/library', '/prompt_manager/vocabulary/tags',
'/prompt_manager/review/items']);
for (const url of urls) assert.equal(url.searchParams.get('limit'), '1');
assert.equal(urls[0].searchParams.get('mode'), 'workspace');
assert.equal(urls[0].searchParams.has('favorites'), false);
@@ -288,3 +292,22 @@ test('saved-view links resolve to their id and re-render only when the id change
assert.deepEqual(seen,[['saved',3],['saved',4],['images',undefined]]);
dispose();
});
test('Needs review is a native SAVED VIEWS route titled Review, with its own count', async () => {
const { resolveRoute, isHandoff, routeTitle, routes, loadNavCounts, REVIEW_WRITE_COUNTS, NAV_COUNT_ROUTES } = await source;
const route = resolveRoute('#/review');
assert.equal(route.id, 'review');
assert.equal(route.label, 'Needs review');
assert.equal(routeTitle(route), 'Review');
assert.equal(isHandoff(route), false);
// No href: Overview lists no legacy card for it.
assert.equal(routes.find(item => item.id === 'review').href, undefined);
assert.ok(NAV_COUNT_ROUTES.includes('review'));
assert.deepEqual([...REVIEW_WRITE_COUNTS], ['review']);
assert.ok(Object.isFrozen(REVIEW_WRITE_COUNTS));
const calls = [];
const api = {get: async url => { calls.push(url); return {success: true, total: 4}; }};
const counts = await loadNavCounts(api, new AbortController().signal, () => {}, REVIEW_WRITE_COUNTS);
assert.deepEqual(calls, ['/prompt_manager/review/items?limit=1']);
assert.deepEqual([...counts], [['review', 4]]);
});
@@ -0,0 +1,188 @@
// System › Appearance (#249 System S3 Task 8): the concept's "Browsing & display" rows (app.js 676–678) on the drafts
// of the one Save row, their stored values (Q3 mappings), the host timeout's 0–300 whole seconds and the D97 rows.
const test = require('node:test');
const assert = require('node:assert/strict');
const gallerySettings = require('../../web/js/gallery-settings.js');
const appearanceModule = import('../../web/js/workspace/system-appearance.mjs');
const stateModule = import('../../web/js/workspace/system-state.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
globalThis.GallerySettings = gallerySettings;
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
remove() { if (this.parentNode) this.parentNode.children = this.parentNode.children.filter(child => child !== this); this.parentNode = null; }
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
const BROWSER = { workspaceLayout: 'grid', workspaceDensity: 'comfortable', autoLoadMetadata: true, cacheMetadata: true,
imageQuality: 'medium', videoAutoplay: false, videoMute: true, videoLoop: true, showFilePaths: true, showImageInfo: true,
sidebarCollapsedByDefault: false };
async function mounted({ host = { webui_display_mode: 'newtab', result_timeout: 5 }, browser = BROWSER, hostFails = false } = {}) {
const { mountAppearance } = await appearanceModule;
const { createSystemDrafts } = await stateModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
let hostValues = null;
const drafts = createSystemDrafts({
readBrowser: () => ({ ...browser }), readRuntime: () => ({}),
readHost: async () => { if (hostFails) throw new Error('offline'); return { values: hostValues, revision: 'sha256:one' }; },
writeBrowser: async () => ({ persisted: true }), writeHost: async () => ({ success: true, persisted: true }), writeRuntime: async () => ({}),
});
await drafts.load('browser');
const scope = createScope();
const [surface] = mountAppearance(null, { scope, drafts });
const byId = id => dom.find(surface, node => node.dataset?.setting === id);
return { dom, drafts, scope, surface, byId, loadHost: async () => { hostValues = host; await drafts.load('host'); },
fixHost: () => { hostFails = false; } };
}
test('the surface is the concept section with the concept rows in order, then the D97 rows', async () => {
const { APPEARANCE_ROWS } = await appearanceModule;
const { dom, surface, scope } = await mounted();
try {
assert.equal(surface.className, 'surface');
assert.equal(surface.children[0].textContent, 'Browsing & display');
assert.equal(surface.children[1].textContent, 'One set of defaults; toolbar changes apply to the current view.');
const rows = surface.children.slice(2);
const titles = rows.map(row => row.children[0].children[0].textContent);
assert.deepEqual(titles, ['Default layout', 'Thumbnail size', 'Continuous browsing', 'Load metadata on selection',
'Cache metadata in this browser', 'Image loading quality', 'Autoplay videos', 'Mute videos by default', 'Loop videos',
'Show file paths', 'Open from ComfyUI', 'Node search auto-hide (seconds)', 'Show image details on cards',
'Collapse viewer details by default', 'Reset display defaults']);
const continuous = rows[2];
assert.deepEqual(dom.tags(continuous.children[0]), ['h3', 'p']);
assert.equal(continuous.children[1].className, 'chip blue');
assert.equal(continuous.children[1].textContent, 'Infinite scroll');
assert.equal(rows.at(-1).className, 'operation');
assert.equal(APPEARANCE_ROWS.filter(Array.isArray).length, 13);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
test('added rows reach their consumers: each control edits its stored key with the Q3 values', async () => {
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
// Host rows stay disabled until the server settings answered.
assert.equal(byId('open-mode').disabled, true);
assert.equal(byId('timeout').disabled, true);
await loadHost();
assert.equal(byId('open-mode').disabled, false);
assert.equal(byId('timeout').value, '5');
assert.deepEqual(byId('image-quality').children.map(option => [option.value, option.textContent]),
[['medium', 'Optimized'], ['high', 'Original'], ['low', 'Fast']]);
assert.deepEqual(byId('open-mode').children.map(option => [option.value, option.textContent]),
[['newtab', 'New tab'], ['popup', 'Popup window']]);
const edits = [['default-view', 'masonry'], ['density', 'large'], ['image-quality', 'low'], ['open-mode', 'popup']];
for (const [id, value] of edits) { byId(id).value = value; byId(id).dispatch('change'); }
for (const id of ['metadata-auto', 'metadata-cache', 'video-muted', 'video-loop', 'paths', 'card-details']) {
byId(id).checked = false; byId(id).dispatch('change');
}
for (const id of ['video-auto', 'viewer-details']) { byId(id).checked = true; byId(id).dispatch('change'); }
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'masonry', workspaceDensity: 'large', imageQuality: 'low',
autoLoadMetadata: false, cacheMetadata: false, videoMute: false, videoLoop: false, showFilePaths: false, showImageInfo: false,
videoAutoplay: true, sidebarCollapsedByDefault: true });
assert.deepEqual(drafts.changes('host'), { webui_display_mode: 'popup' });
} finally { scope.dispose(); dom.restore(); }
});
test('the auto-hide timeout takes whole seconds from 0 to 300 only', async () => {
const { resultTimeout } = await appearanceModule;
assert.deepEqual(['0', '30', ' 300 ', '301', '-1', '2.5', '', 'abc', '1e2'].map(resultTimeout), [0, 30, 300, null, null, null, null, null, null]);
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
await loadHost();
const timeout = byId('timeout');
timeout.value = '45'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 45 });
timeout.value = '301'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), {});
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
timeout.value = '0'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 0 });
assert.deepEqual([...drafts.failed('host')], []);
} finally { scope.dispose(); dom.restore(); }
});
test('reset restores defaults without touching other keys', async () => {
const { DISPLAY_KEYS } = await appearanceModule;
const changed = { ...BROWSER, workspaceLayout: 'list', videoMute: false, showImageInfo: false, sidebarCollapsedByDefault: true };
const { dom, drafts, scope, surface, loadHost } = await mounted({ browser: changed });
try {
await loadHost();
drafts.set('host', 'result_timeout', 9);
const reset = dom.find(surface.children.at(-1), node => node.tagName === 'BUTTON');
assert.equal(reset.textContent, 'Reset');
reset.click();
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'grid', videoMute: true, showImageInfo: true, sidebarCollapsedByDefault: false });
// The host fields and every key outside the display defaults are left as they are.
assert.deepEqual(drafts.changes('host'), { result_timeout: 9 });
assert.deepEqual([...DISPLAY_KEYS].sort(), Object.keys(BROWSER).sort());
} finally { scope.dispose(); dom.restore(); }
});
test('host rows that could not be read stay disabled and say so; a later read clears it', async () => {
const { HOST_UNAVAILABLE } = await appearanceModule;
const { dom, drafts, scope, byId, surface, fixHost } = await mounted({ hostFails: true });
const notes = () => surface.children.slice(2).flatMap(row => row.children[0].children.slice(2).map(note => [row.children[0].children[0].textContent, note.className, note.textContent]));
try {
assert.deepEqual(notes(), []);
assert.equal(await drafts.reloadHost(), false);
assert.deepEqual(notes(), [['Open from ComfyUI', 'small', HOST_UNAVAILABLE], ['Node search auto-hide (seconds)', 'small', HOST_UNAVAILABLE]]);
assert.equal(byId('timeout').disabled, true);
// Repaints keep one line per row.
drafts.set('browser', 'videoMute', false);
assert.equal(notes().length, 2);
fixHost();
assert.equal(await drafts.reloadHost(), true);
assert.deepEqual(notes(), []);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
+145
View File
@@ -0,0 +1,145 @@
// System › Sources & scans (#249 System S4 Task 11): the concept's "Image sources" list (app.js 679–681) built from the
// server's configured, effective and watched sources (GET /settings), the host draft of `gallery_root_paths`, the
// 3.2.0 update notice's dismissal and the Refresh files toast (D98).
const test = require('node:test');
const assert = require('node:assert/strict');
const sourcesModule = import('../../web/js/workspace/system-sources.mjs');
const STATUS = Object.freeze({
configured: [
{ label: 'ComfyUI output', path: 'output', valid: true, code: null, reason: null },
{ label: 'ComfyUI input', path: 'input', valid: true, code: null, reason: null },
],
effective: [
{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'watching' },
{ label: 'ComfyUI input', path: 'input', root_index: 1, watch: 'manual' },
],
auto_detected: false,
});
test('each effective source is the concept row: label, "<path> · watching|manual scan", Scan changes or Scan folder', async () => {
const { sourceRows } = await sourcesModule;
const saved = ['output', 'input'];
const { rows, note } = sourceRows({ sources: STATUS, saved, current: saved });
assert.equal(note, null);
assert.deepEqual(rows, [
{ key: 'output', title: 'ComfyUI output', description: 'output · watching', scan: { label: 'Scan changes', root: 0, path: 'output' },
remove: 'output' },
{ key: 'input', title: 'ComfyUI input', description: 'input · manual scan', scan: { label: 'Scan folder', root: 1, path: 'input' },
remove: 'input' },
]);
});
test('watching only with a live observer; restart pending otherwise (D98)', async () => {
const { sourceRows, WATCH_COPY } = await sourcesModule;
assert.deepEqual(WATCH_COPY, { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan',
restart_pending: 'restart pending' });
const sources = { ...STATUS, effective: STATUS.effective.map(source => ({ ...source, watch: 'restart_pending' })) };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · restart pending', 'Scan folder'], ['input · restart pending', 'Scan folder']]);
});
test('a configured source that cannot be used shows the server reason and only Remove', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'unmounted-drive', path: 'unmounted-drive', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const saved = ['output', 'input', 'unmounted-drive'];
const { rows } = sourceRows({ sources, saved, current: saved });
assert.deepEqual(rows.at(-1), { key: 'unmounted-drive', title: 'unmounted-drive',
description: 'unmounted-drive · Gallery root does not exist', scan: null, remove: 'unmounted-drive' });
});
test('an added source is listed as not saved yet with its validated label; a removed one leaves the list', async () => {
const { sourceRows } = await sourcesModule;
const labels = new Map([['output/archive', 'archive']]);
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['input', 'output/archive'], labels });
assert.equal(note, null);
assert.deepEqual(rows.map(row => [row.title, row.description, row.scan?.label ?? null, row.remove]), [
['ComfyUI input', 'input · manual scan', 'Scan folder', 'input'],
['archive', 'output/archive · not saved yet', null, 'output/archive'],
]);
// Without a recorded label (a reload of System), the folder name stands in.
assert.equal(sourceRows({ sources: STATUS, saved: [], current: ['output/new-collection'] }).rows[0].title, 'new-collection');
});
test('removing the last entry states auto-detect (D98)', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: [] });
assert.deepEqual(rows, []);
assert.equal(note, NOTES.removedAll);
assert.equal(NOTES.removedAll, 'No source is configured. After saving, ComfyUI’s output folder is used.');
});
test('auto-detected sources are listed without Remove, with the line that says how they were chosen', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const auto = { configured: [], effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'manual' }], auto_detected: true };
assert.deepEqual(sourceRows({ sources: auto, saved: [], current: [] }), { rows: [
{ key: 'auto:0', title: 'ComfyUI output', description: 'output · manual scan', scan: { label: 'Scan folder', root: 0, path: 'output' },
remove: null },
], note: NOTES.autoDetected });
// Every configured source unusable: ComfyUI's output folder is used instead.
const fallback = { ...auto, configured: [{ label: 'gone', path: 'gone', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const result = sourceRows({ sources: fallback, saved: ['gone'], current: ['gone'] });
assert.deepEqual(result.rows.map(row => [row.title, row.remove]), [['gone', 'gone'], ['ComfyUI output', null]]);
assert.equal(result.note, NOTES.fallback);
assert.equal(sourceRows({ sources: { configured: [], effective: [], auto_detected: false }, saved: [], current: [] }).note, NOTES.none);
});
test('a path listed twice in the draft is one row', async () => {
const { sourceRows } = await sourcesModule;
assert.equal(sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['output', 'output'] }).rows.length, 1);
});
test('roots equal to the saved list are the saved list itself, so the draft is clean again', async () => {
const { draftRoots } = await sourcesModule;
const saved = ['output', 'input'];
assert.equal(draftRoots(['output', 'input'], saved), saved);
assert.deepEqual(draftRoots(['input'], saved), ['input']);
assert.notEqual(draftRoots(['input', 'output'], saved), saved);
assert.deepEqual(draftRoots([], undefined), []);
});
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed); storage that throws keeps it for this view', async () => {
const { noticeDismissed, dismissNotice, NOTICE_KEY, NOTICE_VERSION } = await sourcesModule;
assert.deepEqual([NOTICE_KEY, NOTICE_VERSION], ['pm_update_notice_dismissed', '3.2.0']);
const values = new Map();
const storage = { getItem: key => values.get(key) ?? null, setItem: (key, value) => values.set(key, String(value)) };
assert.equal(noticeDismissed(() => storage), false);
values.set(NOTICE_KEY, '3.1.0');
assert.equal(noticeDismissed(() => storage), false);
assert.equal(dismissNotice(() => storage), true);
assert.equal(values.get(NOTICE_KEY), '3.2.0');
assert.equal(noticeDismissed(() => storage), true);
const denied = () => { throw new Error('SecurityError'); };
assert.equal(noticeDismissed(denied), false);
assert.equal(dismissNotice(denied), false);
assert.equal(dismissNotice(() => ({ setItem() { throw new Error('QuotaExceededError'); } })), false);
});
test('Refresh files toasts the rescan counts, or says it finished when they are missing', async () => {
const { refreshedText } = await sourcesModule;
assert.equal(refreshedText({ total: 1234, roots: 2 }), 'Files refreshed. 1,234 files in 2 output folders.');
assert.equal(refreshedText({ total: 1, roots: 1 }), 'Files refreshed. 1 file in 1 output folder.');
assert.equal(refreshedText({ total: -1, roots: 1 }), 'Files refreshed.');
assert.equal(refreshedText(null), 'Files refreshed.');
});
test('monitoring switched off: a source still watched says so until the restart (D128)', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, effective: [{ ...STATUS.effective[0], watch: 'stopping' }, STATUS.effective[1]] };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · watching until restart', 'Scan changes'], ['input · manual scan', 'Scan folder']]);
});
test('a valid configured source missing from the effective list is another spelling of a listed folder (D128)', async () => {
const { sourceRows, DUPLICATE_SOURCE } = await sourcesModule;
assert.equal(DUPLICATE_SOURCE, 'same folder as another source');
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'output', path: 'output/.', valid: true, code: null, reason: null }] };
const saved = ['output', 'input', 'output/.'];
assert.deepEqual(sourceRows({ sources, saved, current: saved }).rows.at(-1), { key: 'output/.', title: 'output',
description: 'output/. · same folder as another source', scan: null, remove: 'output/.' });
});
+105
View File
@@ -192,6 +192,7 @@ test('status text per outcome', async () => {
partial: 'Some changes were not saved. Review the highlighted settings.',
tabOnly: 'Saved for this tab only; browser storage is unavailable.',
conflict: 'Settings changed elsewhere. Reload this section to see them.',
restart: 'Restart ComfyUI to apply:',
});
const clean = { browser: 'clean', host: 'clean', runtime: 'clean' };
assert.equal(statusText(clean), 'No unsaved changes.');
@@ -259,3 +260,107 @@ test('a failed read keeps the store usable and drafts survive a reload (Q10)', a
assert.equal(failing.dirty(), false);
assert.throws(() => createSystemDrafts({ ...domains, writeHost: null }), TypeError);
});
test('a rejected value drops its draft and is highlighted; a conflict elsewhere keeps the drafts', async () => {
const domains = adapters({ browser: { videoMute: true, videoLoop: true }, host: { result_timeout: 5 } });
const drafts = await loaded(domains);
assert.deepEqual(drafts.saved('browser'), { videoMute: true, videoLoop: true });
drafts.set('host', 'result_timeout', 12);
assert.equal(drafts.dirty('host'), true);
drafts.reject('host', 'result_timeout');
assert.equal(drafts.dirty('host'), false);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.equal(drafts.get('host', 'result_timeout'), 5);
// A valid value clears the highlight.
drafts.set('host', 'result_timeout', 7);
assert.deepEqual([...drafts.failed('host')], []);
drafts.discard();
drafts.set('browser', 'videoMute', false);
drafts.conflict('browser', []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
drafts.conflict('browser', ['videoMute']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
assert.deepEqual([...drafts.failed('browser')], ['videoMute']);
assert.equal(drafts.get('browser', 'videoMute'), false);
assert.throws(() => drafts.reject('nowhere', 'x'), TypeError);
});
test('a rejected value says why in the status until the next edit', async () => {
const drafts = await loaded(adapters({ host: { result_timeout: 5 } }));
drafts.reject('host', 'result_timeout');
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
drafts.set('host', 'result_timeout', 9);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a browser write refused as changed elsewhere keeps the draft and reads the browser values again', async () => {
const domains = adapters({ browser: { workspaceDensity: 'comfortable' } });
const drafts = await loaded(domains);
drafts.set('browser', 'workspaceDensity', 'compact');
domains.outcome.browser = () => { throw Object.assign(new Error('changed'), { code: 'settings-changed' }); };
assert.deepEqual(await drafts.save(), { browser: 'conflict', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls.map(([name]) => name), ['writeBrowser', 'readBrowser']);
assert.equal(drafts.get('browser', 'workspaceDensity'), 'compact');
assert.deepEqual([...drafts.failed('browser')], ['workspaceDensity']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
});
test('host read state says whether the server settings could be read', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 5 } });
let fail = true;
const drafts = createSystemDrafts({ ...domains, readHost: async () => { if (fail) throw new Error('offline'); return { values: {}, revision: 'r' }; } });
assert.equal(drafts.hostState(), 'pending');
assert.equal(await drafts.reloadHost(), false);
assert.equal(drafts.hostState(), 'failed');
fail = false;
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.hostState(), 'loaded');
});
test('a host save names the settings that apply after a restart (D128)', async () => {
const { statusText, RESTART_LABELS } = await source;
assert.deepEqual(RESTART_LABELS, { monitoring_enabled: 'Watch for new outputs' });
const saved = { browser: 'clean', host: 'saved', runtime: 'clean' };
assert.equal(statusText({ ...saved, restart: ['monitoring_enabled'] }),
'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// A field without a row label (the sources say "restart pending" per row) adds nothing.
assert.equal(statusText({ ...saved, restart: ['gallery_root_paths'] }), 'Saved to the server configuration.');
assert.equal(statusText({ ...saved, restart: [] }), 'Saved to the server configuration.');
const domains = adapters({ host: { monitoring_enabled: true } });
const drafts = await loaded(domains);
domains.outcome.host = () => ({ success: true, persisted: true, revision: 'sha256:two',
restart_required_fields: ['monitoring_enabled', 7] });
drafts.set('host', 'monitoring_enabled', false);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'saved', runtime: 'clean', restart: ['monitoring_enabled'] });
assert.equal(drafts.statusText(), 'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The next edit is unsaved again.
drafts.set('host', 'monitoring_enabled', true);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a host save refused for one source says the server reason and names it until the next edit', async () => {
const domains = adapters({ host: { gallery_root_paths: ['output', 'output/gone'] } });
const drafts = await loaded(domains);
const message = "output/gone can't be used: Gallery root does not exist. Remove it or reconnect it, then save.";
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source', source: 'output/gone' }); };
drafts.set('host', 'gallery_root_paths', ['output/gone']);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'clean', host: 'failed', runtime: 'clean',
invalid: { source: 'output/gone', message } });
assert.equal(drafts.statusText(), message);
assert.equal(drafts.invalidSource(), 'output/gone');
assert.deepEqual([...drafts.failed('host')], ['gallery_root_paths']);
// Any other failure keeps the generic copy and names no source.
domains.outcome.host = () => { throw Object.assign(new Error('busy'), { code: 'busy' }); };
await drafts.save();
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
assert.equal(drafts.invalidSource(), null);
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source' }); };
await drafts.save();
assert.equal(drafts.invalidSource(), null);
drafts.set('host', 'gallery_root_paths', ['output']);
assert.equal(drafts.invalidSource(), null);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
@@ -0,0 +1,93 @@
// System › thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (app.js 1433–1437,
// `thumbnail-review`) on the thumbnail job (S5 Task 12). Pure parts: the option values the job takes, the preview size
// check, the Image loading quality preselection (Q3), the D99 plan line and the start outcome.
const test = require('node:test');
const assert = require('node:assert/strict');
const reviewModule = import('../../web/js/workspace/thumbnail-review.mjs');
test('the concept options map to the job values in the concept order', async () => {
const { SCOPE_OPTIONS, QUALITY_OPTIONS, SIZE_LIMIT } = await reviewModule;
assert.deepEqual(SCOPE_OPTIONS, [['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]);
assert.deepEqual(QUALITY_OPTIONS, [['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]);
assert.deepEqual(SIZE_LIMIT, { min: 128, max: 2048, value: 512 });
});
test('size outside 128–2048 reports validity: only whole pixels in range are a size', async () => {
const { previewSize } = await reviewModule;
assert.equal(previewSize('512'), 512);
assert.equal(previewSize('128'), 128);
assert.equal(previewSize('2048'), 2048);
for (const text of ['', ' ', '127', '2049', '4096', '512.5', '-512', 'abc', '1e3', '0x200']) {
assert.equal(previewSize(text), null, text);
}
});
test('Image loading quality preselects the dialog quality (Q3); anything else is Medium', async () => {
const { preselectedQuality } = await reviewModule;
assert.equal(preselectedQuality('low'), 'low');
assert.equal(preselectedQuality('high'), 'high');
assert.equal(preselectedQuality('medium'), 'medium');
assert.equal(preselectedQuality(undefined), 'medium');
assert.equal(preselectedQuality('original'), 'medium');
});
test('the plan line counts what a missing-only run generates and what is already up to date (D99)', async () => {
const { planText } = await reviewModule;
assert.equal(planText({ mode: 'missing', targets: 12, fresh: 30, remaining: 0 }), '12 previews to generate · 30 already up to date.');
assert.equal(planText({ mode: 'missing', targets: 1, fresh: 1, remaining: 0 }), '1 preview to generate · 1 already up to date.');
assert.equal(planText({ mode: 'missing', targets: 0, fresh: 0, remaining: 0 }), '0 previews to generate · 0 already up to date.');
assert.equal(planText({ mode: 'missing', targets: 10000, fresh: 5, remaining: 2500 }),
'10,000 previews to generate · 5 already up to date · 2,500 more on the next run.');
});
test('Rebuild all says every preview is regenerated (D99)', async () => {
const { planText } = await reviewModule;
assert.equal(planText({ mode: 'rebuild', targets: 42, fresh: 30, remaining: 0 }), 'Every preview is regenerated: 42 files.');
assert.equal(planText({ mode: 'rebuild', targets: 1, fresh: 1, remaining: 0 }), 'Every preview is regenerated: 1 file.');
assert.equal(planText({ mode: 'rebuild', targets: 10000, fresh: 9000, remaining: 3 }),
'Every preview is regenerated: 10,000 files · 3 more on the next run.');
});
test('an answer without counts is a failed plan, never zeros', async () => {
const { planText, PLAN_FAILED } = await reviewModule;
assert.equal(planText(null), PLAN_FAILED);
assert.equal(planText({ mode: 'missing', targets: 2 }), PLAN_FAILED);
assert.equal(planText({ mode: 'missing', targets: -1, fresh: 0, remaining: 0 }), PLAN_FAILED);
assert.equal(planText({ mode: 'other', targets: 1, fresh: 0, remaining: 0 }), PLAN_FAILED);
});
const scopeOf = () => ({ signal: new AbortController().signal });
test('Start starts the thumbnail job with the chosen options and opens it in Activity', async () => {
const { startThumbnails } = await reviewModule;
const calls = [];
const navigated = [];
const store = { start: async (kind, body) => { calls.push([kind, body]); return { job: { id: 'a'.repeat(32) } }; } };
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'rebuild', size: 1024, quality: 'high' },
navigate: hash => navigated.push(hash), notify: () => assert.fail('no toast of its own') });
assert.deepEqual(calls, [['thumbnails', { mode: 'rebuild', size: 1024, quality: 'high' }]]);
assert.deepEqual(navigated, [`#/activity/${'a'.repeat(32)}`]);
});
test('a refused start shows the busy link toast and stays', async () => {
const { startThumbnails } = await reviewModule;
const toasts = [];
const store = { start: async () => ({ busy: { id: 'b'.repeat(32), title: 'Scan for metadata' } }) };
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
assert.deepEqual(toasts, [['Another job is running: ', { text: 'Scan for metadata', href: `#/activity/${'b'.repeat(32)}` }, '.']]);
});
test('a failed start says so; a start after the section left says nothing', async () => {
const { startThumbnails, START_FAILED } = await reviewModule;
const toasts = [];
const store = { start: async () => { throw new Error('offline'); } };
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
assert.deepEqual(toasts, [START_FAILED]);
const gone = new AbortController();
gone.abort();
await startThumbnails({ store, scope: { signal: gone.signal }, options: { mode: 'missing', size: 512, quality: 'medium' },
navigate: () => assert.fail('stays'), notify: () => assert.fail('silent') });
});
+2
View File
@@ -29,6 +29,7 @@ VECTORS_PATH = (
# Committed record sets stay small; the large bound cases are built at test time.
MAX_VECTOR_CHARS = 20_000
COMMITTED_FUZZ = (0, 60)
COMMITTED_PASSTHROUGH = (0, 40)
SPACES = ["", " ", "\t", "\n", "\x1c", "\x85", " ", "", "
 "]
WORDS = ["fox", "snow", "café", "猫", "\U0001f600", "a,", "detailed fur", ""]
@@ -1380,6 +1381,7 @@ def committed_vectors():
}
cases.update(fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.selection_fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.passthrough_fuzz_cases(*COMMITTED_PASSTHROUGH))
vectors = [
{
"name": name,
+304 -1
View File
@@ -12,8 +12,10 @@ import json
import random
try:
from tests import passthrough_graphs as shapes
from tests.local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import passthrough_graphs as shapes
from local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
@@ -437,10 +439,213 @@ def _parameters():
return cases
def _passthrough():
"""Chains joined through lineage nodes, and the floor for added conditioning."""
cases = {
"hires-fix": shapes.hires_fix(),
"hires-fix-differs": shapes.hires_fix(second_positive=shapes.OTHER),
"hires-fix-unlisted": shapes.hires_fix(filter_class="MysteryImageEffect"),
"hires-fix-generated": shapes.generated_text_chain(),
"flux-three-stage": shapes.flux_three_stage(),
"flux-concat": shapes.flux_three_stage(composite="ConditioningConcat"),
"flux-average": shapes.flux_three_stage(composite="ConditioningAverage"),
"edit-batch": shapes.edit_batch(3),
}
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
cases["flux-without-base"] = graph
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-other-negative"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
graph["10"]["inputs"]["latent_image"] = ["20", 0]
cases["hires-fix-cycle"] = graph
graph = shapes.hires_fix()
graph["17"]["inputs"]["pixels"] = ["18", 0]
graph["18"] = {"class_type": "Reroute", "inputs": {"value": ["16", 0]}}
cases["hires-fix-reroute"] = graph
graph = shapes.hires_fix()
graph["24"] = {"class_type": "SaveImage", "inputs": {"images": ["11", 0]}}
cases["hires-fix-both-saved"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["99", 0]
cases["hires-fix-missing-latent"] = graph
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-negative-unsupported"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-reroute-input"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-encoder-parts"] = graph
cases.update(_passthrough_edges())
cases.update(_runtime_parts())
return cases
def _runtime_parts():
"""Option A: a PromptManager part fed by a run-time node is left out."""
cases = {"trigger-prepend": shapes.hires_fix_with_trigger_words()}
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="masterpiece", append_text=["8", 0])
graph["20"]["inputs"]["latent_image"] = ["4", 0]
cases["trigger-append-single"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
cases["traced-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
cases["unknown-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["indirect-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="", text=["8", 0])
cases["runtime-main-text"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["class_type"] = "PromptManagerPair"
graph["2"]["inputs"]["negative_text"] = shapes.NEGATIVE
graph["10"]["inputs"]["negative"] = ["2", 1]
graph["20"]["inputs"]["negative"] = ["2", 1]
cases["trigger-pair"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["30"] = {"class_type": "PromptSearchList", "inputs": {"search": "x"}}
graph["2"]["inputs"]["append_text"] = ["30", 0]
cases["search-list-append"] = graph
for shape in ("text-encode", "chained", "concat"):
cases[f"through-{shape}"] = shapes.trigger_words_through(shape)
cases["branches-with-and-without"] = (
shapes.branches_with_and_without_trigger_words()
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
cases["stage-with-and-without"] = graph
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["join-multi-flag"] = graph
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
cases["flux-other-full-prompt"] = graph
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
cases["flux-trigger"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-chained"] = graph
return {f"runtime-{name}": graph for name, graph in cases.items()}
def _passthrough_edges():
"""Lineage and floor edges: branching refiners, other slots, cycles, guiders."""
cases = {}
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.4)
graph["31"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-two-refiners"] = graph
graph = shapes.hires_fix()
graph["10"]["class_type"] = "SamplerCustom"
graph["11"]["inputs"]["samples"] = ["10", 1]
cases["hires-fix-denoised-slot"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["30"]["inputs"]["latent_image"] = ["32", 0]
cases["hires-fix-self-latent"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["32", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["32", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["33"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-lineage-cycle"] = graph
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
cases["flux-base-stages-differ"] = graph
graph = shapes.flux_three_stage()
del graph["60"]["inputs"]["negative"]
cases["flux-no-last-negative"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
graph["40"]["inputs"]["positive"] = ["39", 0]
graph["50"]["inputs"]["positive"] = ["39", 0]
cases["flux-unreadable-base"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "MysteryUpscaler",
"inputs": {"image": ["51", 0], "model": ["1", 0], "positive": ["34", 0]},
}
graph["52"]["inputs"]["images"] = ["39", 0]
cases["flux-unsupported-ancestry"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["62", 0],
"guider": ["63", 0],
"sampler": ["64", 0],
"sigmas": ["65", 0],
"latent_image": ["66", 0],
},
}
graph["62"] = {"class_type": "RandomNoise", "inputs": {"noise_seed": 1}}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"model": ["1", 0], "positive": ["35", 0], "negative": ["33", 0]},
}
graph["64"] = {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}}
graph["65"] = {"class_type": "BasicScheduler", "inputs": {"steps": 4}}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["52", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-last"] = graph
return {f"edge-{name}": graph for name, graph in cases.items()}
def selection_cases():
"""Named record sets for every Task 9 path, each a ``read_inspection_file`` dict."""
cases = {}
for group in (_samplers(), _previews(), _detailers()):
for group in (_samplers(), _previews(), _detailers(), _passthrough()):
for name, graph in group.items():
cases[f"select-{name}"] = {"prompt": json.dumps(graph, ensure_ascii=False)}
cases.update(_parameters())
@@ -572,3 +777,101 @@ def selection_fuzz_cases(seed, count):
records.update(builder.records)
cases[f"select-fuzz-{seed}-{index}"] = records
return cases
_LINEAGE_HOPS = (
("VAEDecode", "samples"),
("VAEDecodeTiled", "samples"),
("Image Filter", "images"),
("ImageUpscaleWithModel", "image"),
("ImageScale", "image"),
("ImageScaleBy", "image"),
("VAEEncode", "pixels"),
("LatentUpscaleBy", "samples"),
("Reroute", "value"),
("MysteryImageEffect", "image"),
)
def passthrough_fuzz_cases(seed, count):
"""Seeded sampler chains joined by random lineage (and unlisted) hops.
Each later stage reads the earlier one's output through zero to four hops,
with a shared, a different, an added-to (combine, concat, average) or a
run-time positive, and a shared or different negative, under a saver or a
preview. The server is the oracle; nothing states the expected answer.
"""
rng = random.Random(seed)
cases = {}
for index in range(count):
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": shapes.prompt_manager(rng.choice([shapes.POSITIVE, "fox"])),
"3": shapes.encode(shapes.NEGATIVE),
"4": {"class_type": "EmptyLatentBatch", "inputs": {"batch_size": 2}},
"5": shapes.encode(shapes.STYLE),
"6": shapes.encode(shapes.OTHER),
}
previous, next_id = ["4", 0], 10
for stage in range(rng.randint(2, 3)):
positive = ["2", 0]
choice = rng.randrange(8)
if stage and choice == 1:
positive = ["6", 0]
elif stage and choice in (2, 3, 4):
name = (
"ConditioningCombine",
"ConditioningConcat",
"ConditioningAverage",
)[choice - 2]
names = shapes._COMPOSITE_INPUTS[name]
parts = [["2", 0], rng.choice([["5", 0], ["6", 0]])]
rng.shuffle(parts)
graph[str(next_id)] = {
"class_type": name,
"inputs": dict(zip(names, parts)),
}
positive, next_id = [str(next_id), 0], next_id + 1
elif stage and choice == 5:
graph[str(next_id)] = {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["2", 0]},
}
positive, next_id = [str(next_id), 0], next_id + 1
negative = ["6", 0] if stage and rng.random() < 0.15 else ["3", 0]
latent = previous
if stage:
image = previous
for _ in range(rng.randrange(5)):
name, key = rng.choice(_LINEAGE_HOPS)
graph[str(next_id)] = {"class_type": name, "inputs": {key: image}}
image, next_id = [str(next_id), 0], next_id + 1
latent = image
sampler = shapes.ksampler(positive, negative, latent)
if stage and rng.random() < 0.2:
sampler = {
"class_type": "UltimateSDUpscale",
"inputs": {
"image": latent,
"model": ["1", 0],
"positive": positive,
"negative": negative,
},
}
graph[str(next_id)] = sampler
previous, next_id = [str(next_id), 0], next_id + 1
if rng.random() < 0.3:
saver = rng.choice(["SaveImage", "PreviewImage"])
graph[str(next_id)] = {
"class_type": saver,
"inputs": {"images": previous},
}
next_id += 1
graph[str(next_id)] = {
"class_type": "SaveImage",
"inputs": {"images": previous},
}
cases[f"select-passthrough-fuzz-{seed}-{index}"] = {
"prompt": json.dumps(graph, ensure_ascii=False)
}
return cases
+332
View File
@@ -0,0 +1,332 @@
"""Synthetic multi-stage graphs for the pass-through lineage rule.
Each builder reproduces one graph shape with neutral texts and generic names:
a hires-fix or refiner chain (sampler, decode, image filter, model upscale,
scale, encode, sampler), a three-stage Flux run whose last stage combines the
base prompt with a style trigger, an image-edit batch of samplers with distinct
prompts, and a chain whose prompt is generated at run time. They are shared by
the resolver tests and the browser-port parity cases; no user file is involved.
"""
import copy
POSITIVE = "a lighthouse on a cliff, golden hour"
NEGATIVE = "blurry, lowres"
OTHER = "a red bicycle in the rain"
STYLE = "film grain, analog photo"
def encode(text):
return {"class_type": "CLIPTextEncode", "inputs": {"text": text, "clip": ["1", 1]}}
def prompt_manager(text, prepend=""):
return {
"class_type": "PromptManager",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
"clip": ["1", 1],
},
}
def ksampler(positive, negative, latent, denoise=1.0):
return {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"steps": 20,
"cfg": 7.0,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": denoise,
"model": ["1", 0],
"positive": positive,
"negative": negative,
"latent_image": latent,
},
}
def hires_fix(second_positive=None, filter_class="Image Filter"):
"""Two KSamplers joined in image space, like a hires-fix or refiner run.
Sampler 10 runs on a custom batch latent; its output goes through a tiled
decode, an interactive image filter, a model upscale and a scale (sized by a
resolution node reading the filtered image), then is encoded again for
sampler 20 (denoise 0.25), which is decoded and saved.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": prompt_manager(POSITIVE),
"3": encode(NEGATIVE),
"4": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1024, "height": 512, "batch_size": 4},
},
"10": ksampler(["2", 0], ["3", 0], ["4", 0]),
"11": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["10", 0], "vae": ["1", 2]},
},
"12": {
"class_type": filter_class,
"inputs": {"timeout": 600, "ontimeout": "send none", "images": ["11", 0]},
},
"13": {"class_type": "UpscaleModelLoader", "inputs": {"model_name": "x4"}},
"14": {
"class_type": "ImageUpscaleWithModel",
"inputs": {"upscale_model": ["13", 0], "image": ["12", 0]},
},
"15": {
"class_type": "ResolutionCalculator",
"inputs": {"scale_factor": 1.5, "image": ["12", 0]},
},
"16": {
"class_type": "ImageScale",
"inputs": {
"upscale_method": "nearest-exact",
"width": ["15", 0],
"height": ["15", 1],
"crop": "disabled",
"image": ["14", 0],
},
},
"17": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["16", 0], "vae": ["1", 2]},
},
"20": ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.25),
"21": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["20", 0], "vae": ["1", 2]},
},
"22": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "hires", "images": ["21", 0]},
},
"23": {"class_type": "PlaySound|pysssss", "inputs": {"any": ["21", 0]}},
}
if second_positive is not None:
graph["5"] = encode(second_positive)
graph["20"]["inputs"]["positive"] = ["5", 0]
return graph
_COMPOSITE_INPUTS = {
"ConditioningCombine": ("conditioning_1", "conditioning_2"),
"ConditioningConcat": ("conditioning_to", "conditioning_from"),
"ConditioningAverage": ("conditioning_to", "conditioning_from"),
}
def flux_three_stage(combined=(("30", 0), ("31", 0)), composite="ConditioningCombine"):
"""FluxGuidance base, a refiner through an image filter, then a tiled upscale.
Samplers 40 and 50 share FluxGuidance(PromptManager 30) and a zeroed empty
negative. Sampler 50 refines 40's filtered image (denoise 0.25). The final
UltimateSDUpscale 60 reads 50's image, scaled down, and a positive that is
``composite`` of ``combined`` (by default the base PromptManager and a
style-trigger encoder 31).
"""
names = _COMPOSITE_INPUTS[composite]
graph = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux"}},
"2": {"class_type": "DualCLIPLoader", "inputs": {"type": "flux"}},
"3": {"class_type": "VAELoader", "inputs": {"vae_name": "ae"}},
"30": prompt_manager(POSITIVE),
"31": encode(STYLE),
"32": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "", "clip": ["2", 0]},
},
"33": {
"class_type": "ConditioningZeroOut",
"inputs": {"conditioning": ["32", 0]},
},
"34": {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["30", 0]},
},
"35": {
"class_type": composite,
"inputs": dict(zip(names, map(list, combined))),
},
"36": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1080, "height": 1920, "batch_size": 4},
},
"40": ksampler(["34", 0], ["33", 0], ["36", 0]),
"41": {
"class_type": "VAEDecode",
"inputs": {"samples": ["40", 0], "vae": ["3", 0]},
},
"42": {
"class_type": "Image Filter",
"inputs": {"timeout": 600, "images": ["41", 0]},
},
"43": {"class_type": "PreviewImage", "inputs": {"images": ["42", 0]}},
"44": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["42", 0], "vae": ["3", 0]},
},
"50": ksampler(["34", 0], ["33", 0], ["44", 0], denoise=0.25),
"51": {
"class_type": "VAEDecode",
"inputs": {"samples": ["50", 0], "vae": ["3", 0]},
},
"52": {
"class_type": "ImageScaleDownBy",
"inputs": {"scale_by": 0.75, "images": ["51", 0]},
},
"53": {"class_type": "PreviewImage", "inputs": {"images": ["51", 0]}},
"54": {
"class_type": "Image Comparer (rgthree)",
"inputs": {"image_a": ["51", 0], "image_b": ["42", 0]},
},
"60": {
"class_type": "UltimateSDUpscale",
"inputs": {
"upscale_by": 2.0,
"seed": 7,
"steps": 4,
"denoise": 0.35,
"image": ["52", 0],
"model": ["1", 0],
"positive": ["35", 0],
"negative": ["33", 0],
"vae": ["3", 0],
},
},
"61": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "flux", "images": ["60", 0]},
},
}
return graph
def edit_batch(count=3):
"""Image-edit samplers with distinct prompts, all saved with one prefix."""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "LoadImage", "inputs": {"image": "ref.png"}},
"3": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["2", 0], "vae": ["1", 2]},
},
"4": {"class_type": "PrimitiveString", "inputs": {"value": "edit"}},
}
for index in range(count):
base = 100 + index * 10
graph[str(base)] = {
"class_type": "TextEncodeQwenImageEdit",
"inputs": {
"prompt": f"edit number {index}: change the pose",
"clip": ["1", 1],
},
}
graph[str(base + 1)] = ksampler([str(base), 0], [str(base), 0], ["3", 0])
graph[str(base + 2)] = {
"class_type": "VAEDecode",
"inputs": {"samples": [str(base + 1), 0], "vae": ["1", 2]},
}
graph[str(base + 3)] = {
"class_type": "SaveImage",
"inputs": {"filename_prefix": ["4", 0], "images": [str(base + 2), 0]},
}
return graph
def trigger_toggle(words="trig9word"):
"""LoraManager's TriggerWord Toggle: its output is decided at run time."""
return {
"class_type": "TriggerWord Toggle (LoraManager)",
"inputs": {
"group_mode": True,
"default_active": True,
"toggle_trigger_words": {"__value__": [{"text": words, "active": True}]},
"orinalMessage": words,
"trigger_words": ["7", 2],
},
}
def hires_fix_with_trigger_words():
"""``hires_fix`` whose PromptManager prepends LoRA trigger words (node 8)."""
graph = hires_fix()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["1", 1], "text": "<lora:x:1>"},
}
graph["8"] = trigger_toggle()
graph["2"]["inputs"]["prepend_text"] = ["8", 0]
return graph
def prompt_manager_text(text, prepend=""):
"""PromptManagerText: a STRING PromptManager (prepend/main/append)."""
return {
"class_type": "PromptManagerText",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
},
}
def trigger_words_through(shape):
"""``hires_fix_with_trigger_words`` with nodes between the toggle and sampler.
``text-encode``: PromptManagerText (prepend <- toggle) -> CLIPTextEncode.
``chained``: a PromptManager whose ``text`` reads that PromptManagerText.
``concat``: PromptManagerText -> StringConcatenate(..., "extra") -> encoder.
"""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager_text(POSITIVE, ["8", 0])
if shape == "chained":
graph["2"]["inputs"].update(prepend_text="", text=["9", 0])
elif shape == "text-encode":
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["9", 0]}}
else:
graph["19"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": ["9", 0], "string_b": "extra", "delimiter": ", "},
}
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["19", 0]}}
return graph
def branches_with_and_without_trigger_words():
"""Two saved branches: one PromptManager prepends trigger words, one does not."""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager(POSITIVE)
graph["20"]["inputs"].update(positive=["9", 0], latent_image=["4", 0])
return graph
def generated_text_chain():
"""``hires_fix`` whose positive text comes from a run-time text generator."""
graph = hires_fix()
graph["2"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["6", 0], "clip": ["1", 1]},
}
graph["6"] = {
"class_type": "TextGenerate",
"inputs": {"prompt": "write a prompt", "model": ["1", 0]},
}
return graph
def copied(graph):
return copy.deepcopy(graph)
+11 -2
View File
@@ -30,6 +30,11 @@ FILES = "media_annotation_files"
BYTES = "media_annotation_bytes"
IMAGE_TAGS = "media_image_tags"
TABLES = (METADATA, FILES, BYTES, IMAGE_TAGS)
REVIEW_TABLES = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
@@ -202,10 +207,14 @@ class AnnotationSchemaTests(unittest.TestCase):
)
def drop_group(self, path):
"""Remove the group from a closed database file (an old backup)."""
"""Remove the group from a closed database file (an old backup).
The review queue group cites annotation bytes and predates nothing, so
an old backup lacks it as well.
"""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in reversed(TABLES):
for table in REVIEW_TABLES + tuple(reversed(TABLES)):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
+28 -3
View File
@@ -52,6 +52,7 @@ class AdminAPITestCase(AioHTTPTestCase):
self.outside_dir = Path(self.tmpdir) / "outside"
self.outside_dir.mkdir()
self.addAsyncCleanup(self._drain_and_remove)
self.config_path = os.path.join(self.tmpdir, "config.json")
self._orig_env = {
key: os.environ.pop(key, None) for key in (CONFIG_PATH_ENV, EXTRA_ROOTS_ENV)
@@ -89,6 +90,26 @@ class AdminAPITestCase(AioHTTPTestCase):
sys.modules.pop("folder_paths", None)
else:
sys.modules["folder_paths"] = self._orig_folder_paths
async def _drain_and_remove(self):
"""The last cleanup: no job outlives the temporary database.
Registered first, so it runs after every cleanup a test added (which
may release its own gated jobs). A job still running then, such as a
scan a failed assertion never awaited, is cancelled and awaited
before the database and the tree are removed.
"""
api = getattr(self, "api", None)
jobs = api._jobs.list() if api is not None else []
tasks = [
job.task for job in jobs if job.task is not None and not job.task.done()
]
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
if api is not None:
# A scan of an ambiguous file captures it for Needs review.
api.close_annotation_captures()
PromptModel.close_all_instances() # restore may have swapped api.db
assert_closed(self, self.tmpdir)
shutil.rmtree(self.tmpdir, ignore_errors=True)
@@ -2221,7 +2242,9 @@ class TestActivityJobs(AdminAPITestCase):
for name in names:
(self.output_dir / name).write_bytes(b"not an image")
def fake_ingest(media_file, metadata, detailed=False):
def fake_ingest(
media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
key = Path(media_file).relative_to(self.output_dir).as_posix()
outcome = names[key]
if isinstance(outcome, Exception):
@@ -2350,8 +2373,10 @@ class TestActivityJobs(AdminAPITestCase):
first_done, release = threading.Event(), threading.Event()
original = self.api._ingest_scan_batch_sync
def gated(batch_results, roots=(), outcomes=None):
counts = original(batch_results, roots=roots, outcomes=outcomes)
def gated(batch_results, roots=(), outcomes=None, review=None):
counts = original(
batch_results, roots=roots, outcomes=outcomes, review=review
)
first_done.set()
release.wait(5)
return counts
+9 -1
View File
@@ -1045,7 +1045,15 @@ class TestFailures(AnnotationAPITestCase):
await self.api.cleanup_annotation_captures(None)
self.assertFalse(os.path.exists(directory))
self.assertNotIn(identity, observers._restore_observers)
# A later observation starts a fresh store.
# The shutdown cleanup is final: a later observation (or a monitor
# recording still in flight) creates no new store or directory.
status, body = await self.observe({"target": self.target("sub/a.png")})
self.assertEqual((status, body["code"]), (500, "capture-error"), body)
self.assertIsNone(self.api._captures)
self.assertNotIn(identity, observers._restore_observers)
# An explicit, non-final close still lets a later observation start
# a fresh store (the tests' own store swaps rely on it).
self.api._captures_closed = False
await self.observe_ok()
self.assertNotEqual(self.api._captures.directory, directory)
+13
View File
@@ -471,6 +471,19 @@ class TestUiRoutes(CoreRoutesTestCase):
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
async def test_review_routes_are_registered(self):
table = {
(route.method, route.resource.canonical)
for route in self.app.router.routes()
if route.resource is not None
}
for entry in (
("GET", "/prompt_manager/review/items"),
("GET", "/prompt_manager/review/items/{item_id}"),
("POST", "/prompt_manager/review/items/{item_id}/decision"),
):
self.assertIn(entry, table)
class TestStaticRoutes(CoreRoutesTestCase):
+7 -25
View File
@@ -413,11 +413,11 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
lock = threading.Lock()
original = images_module._write_image_thumbnail
def recording(src, dst, size):
def recording(src, dst, size, *args):
with lock:
seen.add(threading.current_thread().name)
time.sleep(0.05)
return original(src, dst, size)
return original(src, dst, size, *args)
images_module._write_image_thumbnail = recording
self.addCleanup(setattr, images_module, "_write_image_thumbnail", original)
@@ -1939,7 +1939,11 @@ class TestGalleryRescan(ImageAPITestCase):
class TestOutputScanHelpers(ImageAPITestCase):
"""Stable ids for filesystem entries, no symlink reads, stop on disconnect."""
"""Stable ids for filesystem entries and no symlink reads.
The thumbnail stream no longer stops when its client goes away: see
``tests/test_thumbnail_jobs.py`` (the work is a job that outlives it).
"""
def test_output_entry_id_is_a_stable_digest_of_the_path(self):
import hashlib
@@ -1964,28 +1968,6 @@ class TestOutputScanHelpers(ImageAPITestCase):
names = sorted(p.name for p in images_module._iter_media_files(output, 100))
self.assertEqual(names, ["real.png"])
async def test_thumbnail_stream_stops_when_the_client_disconnects(self):
generated = []
def fake_generate(src, dst, size, is_video):
generated.append(src)
return {
"action": "generated",
"file": src.name,
"dir": src.parent.name,
"type": "image",
}
self.api._generate_one = fake_generate
class GoneResponse:
async def write(self, data):
raise ConnectionResetError("client went away")
targets = [(Path(f"/x/{i}.png"), Path(f"/x/t{i}.jpg"), False) for i in range(3)]
await self.api._stream_thumbnails(GoneResponse(), targets, (256, 256))
self.assertEqual(generated, [])
class TestThumbnailStartupStatus(ThumbnailTestCase):
"""Startup counts and generation cover the same configured collection."""
File diff suppressed because it is too large Load Diff
+364
View File
@@ -0,0 +1,364 @@
"""Needs review change notices (#249 PR F2).
``GET /prompt_manager/review/events`` is a server-sent event stream: one
``ready`` event when it opens, then one ``change`` event after every
committed queue change (an item recorded by the scan or the monitor, a
spelling refresh, a decision from any tab) and after a restore. An open
Needs review page re-reads its queue on ``change``; nothing polls. A refused
decision and a duplicate recording change nothing and send nothing.
"""
import asyncio
import json
import sqlite3
import threading
try:
from tests.test_api_review import ReviewRoutesTestCase
from tests.test_review import unlinkable_graph, write_png
except ImportError: # discovered with tests/ as the top-level directory
from test_api_review import ReviewRoutesTestCase
from test_review import unlinkable_graph, write_png
try: # guarded so a missing module fails assertions, not test loading
from database import review_changes
except ImportError:
review_changes = None
try:
from py.api import review_events
except ImportError:
review_events = None
EVENTS_URL = "/prompt_manager/review/events"
# A safety bound for one read; every wait below ends on an event first.
READ_SECONDS = 10
class ReviewEventsTestCase(ReviewRoutesTestCase):
def _require_events(self):
self.assertIsNotNone(review_changes, "database.review_changes is missing")
self.assertIsNotNone(review_events, "py.api.review_events is missing")
async def _stream(self):
resp = await self.client.get(EVENTS_URL)
self.assertEqual(resp.status, 200)
self.assertEqual(resp.headers["Content-Type"], "text/event-stream")
self.assertEqual(resp.headers["Cache-Control"], "no-store")
self.addAsyncCleanup(self._close, resp)
self.assertEqual(await self._event(resp), "ready")
return resp
@staticmethod
async def _close(resp):
resp.close()
async def _event(self, resp):
"""The next event's name (keepalive comments are skipped)."""
return (await self._event_with_data(resp))[0]
async def _event_with_data(self, resp):
"""The next event's name and its parsed ``data`` line."""
name, data = None, None
while True:
line = await asyncio.wait_for(resp.content.readline(), READ_SECONDS)
self.assertTrue(line, "the stream ended")
text = line.decode("utf-8").rstrip("\n")
if text.startswith("event: "):
name = text[len("event: ") :]
elif text.startswith("data: "):
data = json.loads(text[len("data: ") :])
elif text == "" and name is not None:
return name, data
async def _ready_version(self):
resp = await self.client.get(EVENTS_URL)
self.assertEqual(resp.status, 200)
self.addAsyncCleanup(self._close, resp)
name, data = await self._event_with_data(resp)
self.assertEqual(name, "ready")
return data
def _queueable(self, name="two.png", color=(255, 0, 0)):
"""A file the producers queue: ambiguous, and nothing links it."""
path = self.output_dir / name
path.parent.mkdir(parents=True, exist_ok=True)
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=color)
return path
async def _record_in_thread(self, path):
"""Record as the monitor does: from a worker thread, not the loop."""
self._roots()
outcomes = []
thread = threading.Thread(
target=lambda: outcomes.append(self.api.record_review_file(str(path)))
)
thread.start()
await asyncio.get_running_loop().run_in_executor(None, thread.join)
return outcomes[0]
class TestReviewEvents(ReviewEventsTestCase):
async def test_monitor_recording_sends_one_change(self):
self._require_events()
stream = await self._stream()
self.assertEqual(await self._record_in_thread(self._queueable()), "recorded")
self.assertEqual(await self._event(stream), "change")
async def test_duplicate_recording_and_refused_decision_send_nothing(self):
# A spy on the notice itself: a wrong notice would otherwise pass as
# the next real change's event.
self._require_events()
from unittest.mock import patch
from database import review as review_db
from database import review_decisions
calls = []
original = review_changes.notify_review_change
def spy(db_path):
calls.append(db_path)
original(db_path)
path = self._queueable("two.png")
self._record(path)
with (
patch.object(review_db, "notify_review_change", spy),
patch.object(review_decisions, "notify_review_change", spy),
):
# The same bytes again: nothing changes.
self.assertEqual(await self._record_in_thread(path), "duplicate")
self.assertEqual(calls, [], "a duplicate recording sent a notice")
item, detail, observed = await self._open()
# A stale revision: refused with zero writes.
body = {
**self._body(detail, observed, "unresolved"),
"expected_revision": 9,
}
status, data = await self._decide(item["id"], body)
self.assertEqual(status, 409, data)
self.assertEqual(calls, [], "a refused decision sent a notice")
self._record(self._ambiguous("other.png", color=(0, 0, 9)))
self.assertEqual(len(calls), 1, "a real recording sends one notice")
self.assertEqual(len(self._items()), 2)
async def test_a_burst_of_changes_coalesces_into_one_more_event(self):
# M9: a busy scan records file after file; an open page hears the
# first at once and the rest as one trailing ``change``.
self._require_events()
from unittest.mock import patch
path = self.api.db.model.db_path
with (
patch.object(review_events, "COALESCE_SECONDS", 0.3),
patch.object(review_events, "HEARTBEAT_SECONDS", 0.6),
):
stream = await self._stream()
review_changes.notify_review_change(path)
self.assertEqual(await self._event(stream), "change")
for _ in range(5):
review_changes.notify_review_change(path)
await asyncio.sleep(0.02)
# Every change event up to the next keepalive: exactly one.
changes = 0
while True:
line = await asyncio.wait_for(stream.content.readline(), READ_SECONDS)
self.assertTrue(line, "the stream ended")
if line == b": keepalive\n":
break
changes += line == b"event: change\n"
self.assertEqual(changes, 1)
async def test_decision_from_any_tab_sends_a_change(self):
self._require_events()
self._record(self._ambiguous())
stream = await self._stream()
item, detail, observed = await self._open()
status, data = await self._decide(
item["id"], self._body(detail, observed, "unresolved")
)
self.assertEqual(status, 200, data)
self.assertEqual(await self._event(stream), "change")
async def test_restore_sends_a_change(self):
self._require_events()
stream = await self._stream()
from database.execution_history import notify_history_restore
notify_history_restore(self.api.db.model.db_path)
self.assertEqual(await self._event(stream), "change")
async def test_scan_recording_sends_a_change(self):
self._require_events()
self._roots()
self._queueable("sub/two.png")
stream = await self._stream()
job = await self._scan("all")
self.assertEqual(job["scope"]["review_items"], 1)
self.assertEqual(await self._event(stream), "change")
async def test_a_linked_file_leaving_the_queue_sends_a_change(self):
"""The queue heals (#284 maintainer test): a pending item whose file is
linked now leaves the queue, and the open page hears about it."""
self._require_events()
self._roots()
path = self._ambiguous("two.png")
self._record(path) # as the pre-fix scan queued fallback-linked files
stream = await self._stream()
job = await self._scan("new")
self.assertEqual(job["scope"]["review_settled"], 1)
self.assertEqual(self._items(), [])
self.assertEqual(await self._event(stream), "change")
async def test_ready_carries_the_version_a_list_read_answers(self):
# F3 (no double read per visit): the page skips the re-read on
# ``ready`` when its first read already answered this version.
self._require_events()
self._record(self._ambiguous())
page = await self._list()
self.assertIsInstance(page.get("review_revision"), int, page)
version = await self._ready_version()
self.assertEqual(
version,
{
"library_epoch": page["library_epoch"],
"review_revision": page["review_revision"],
},
)
async def test_a_queue_change_moves_the_version_ready_carries(self):
self._require_events()
before = await self._list()
self._record(self._ambiguous())
after = await self._list()
self.assertIn("review_revision", after)
self.assertGreater(after["review_revision"], before["review_revision"])
version = await self._ready_version()
self.assertEqual(version["review_revision"], after["review_revision"])
self.assertNotEqual(version["review_revision"], before["review_revision"])
async def test_ready_names_no_version_when_it_cannot_be_read(self):
# The page then re-reads, as it did before versions existed.
self._require_events()
def broken():
raise sqlite3.DatabaseError("locked")
self.api.db.review_queue_version = broken
self.assertEqual(await self._ready_version(), {})
async def test_a_stream_counts_as_open_once_it_has_sent_ready(self):
# The browser fixture waits on this count before it changes the
# queue, so a change after it is a ``change`` event and never part of
# the version ``ready`` named.
self._require_events()
reading, release = asyncio.Event(), asyncio.Event()
original = self.api._queue_version
async def held():
reading.set()
await release.wait()
return await original()
self.api._queue_version = held
opening = asyncio.ensure_future(self._ready_version())
await asyncio.wait_for(reading.wait(), READ_SECONDS)
self.assertEqual(self.api.review_stream_count(), 0)
release.set()
await asyncio.wait_for(opening, READ_SECONDS)
self.assertEqual(self.api.review_stream_count(), 1)
async def test_change_events_carry_no_data(self):
self._require_events()
stream = await self._stream()
self._record(self._ambiguous())
self.assertEqual(await self._event_with_data(stream), ("change", {}))
async def test_closing_the_stream_unsubscribes(self):
self._require_events()
path = self.api.db.model.db_path
stream = await self._stream()
self.assertEqual(review_changes.observer_count(path), 1)
self.assertEqual(self.api.review_stream_count(), 1)
stream.close()
# The server notices the closed socket at its next write: a change.
self._record(self._ambiguous())
for _ in range(100):
if not self.api.review_stream_count():
break
await asyncio.sleep(0.02)
self.assertEqual(self.api.review_stream_count(), 0)
self.assertEqual(review_changes.observer_count(path), 0)
async def test_shutdown_ends_every_stream(self):
self._require_events()
stream = await self._stream()
await self.api.close_review_events(None)
line = await asyncio.wait_for(stream.content.read(), READ_SECONDS)
self.assertEqual(line, b"")
self.assertEqual(self.api.review_stream_count(), 0)
# A stream opened after shutdown ends at once.
resp = await self.client.get(EVENTS_URL)
self.assertEqual(resp.status, 503)
resp.close()
async def test_keepalive_comment_is_skipped_by_clients(self):
self._require_events()
with self.patch_heartbeat(0.05):
stream = await self._stream()
line = await asyncio.wait_for(stream.content.readline(), READ_SECONDS)
self.assertEqual(line, b": keepalive\n")
def patch_heartbeat(self, seconds):
from unittest.mock import patch
return patch.object(review_events, "HEARTBEAT_SECONDS", seconds)
class TestReviewChangeRegistry(ReviewEventsTestCase):
def test_a_failing_observer_never_fails_a_write(self):
self._require_events()
path = self.api.db.model.db_path
seen = []
def broken():
raise RuntimeError("observer failed")
unsubscribe_broken = review_changes.subscribe_review_changes(path, broken)
unsubscribe = review_changes.subscribe_review_changes(
path, lambda: seen.append(1)
)
try:
with self.assertLogs(
"prompt_manager.database.review_changes", level="WARNING"
):
self._record(self._ambiguous())
self.assertEqual(seen, [1])
finally:
unsubscribe_broken()
unsubscribe()
self.assertEqual(review_changes.observer_count(path), 0)
def test_notice_comes_after_the_commit(self):
self._require_events()
path = self.api.db.model.db_path
seen = []
def committed():
# A separate connection sees only committed rows.
other = sqlite3.connect(path)
try:
seen.append(
other.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[
0
]
)
finally:
other.close()
unsubscribe = review_changes.subscribe_review_changes(path, committed)
try:
self._record(self._ambiguous())
finally:
unsubscribe()
self.assertEqual(seen, [1])
+57
View File
@@ -68,6 +68,55 @@ class CardSeed(unittest.TestCase):
self.assertIn(expected, (111, 222))
self.assertEqual(info_seed(info), str(expected))
def test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed(self):
# The inspector shows the named final stage's seed (D19, pass-through
# chains); the parser shows another stage's. Never two seeds: none.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {
"class_type": "FaceDetailer",
"inputs": {
"seed": 333,
"image": ["20", 0],
"model": ["4", 0],
"positive": ["6", 0],
"negative": ["7", 0],
},
}
info = graph(seeded(111), extra=extra)
self.assertEqual(extract_from_records(info["prompt"], None)["seed"], 111)
self.assertIsNone(info_seed(info))
# The same chain on one seed keeps it; so does a refiner over a decode.
extra["21"]["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), extra=extra)), "111")
refiner = seeded(222, latent_image=["22", 0])
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
del extra["21"]
self.assertIsNone(info_seed(graph(seeded(111), refiner, extra=extra)))
refiner["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_a_seed_that_agrees_with_the_named_stage_is_kept(self):
# Review I3: a saved sampler (seed 7) beside a preview-only sampler
# (seed 5): the parser and the inspector both show 7, as on main.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {"class_type": "SaveImage", "inputs": {"images": ["20", 0]}}
extra["30"] = {"class_type": "VAEDecode", "inputs": {"samples": ["11", 0]}}
extra["31"] = {"class_type": "PreviewImage", "inputs": {"images": ["30", 0]}}
info = graph(seeded(7), seeded(5), extra=extra)
self.assertEqual(info_seed(info), "7")
def test_a_named_ambiguous_chain_keeps_the_parser_seed(self):
# Review M2: multistage-differs names a sampler, but its facts merge,
# and the inspector fills the parser's seed: the card keeps it.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["8"] = encoder("another scene")
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
refiner = ksampler(("8", 0), ("7", 0), seed=222, latent_image=["22", 0])
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_card_seed_linked_input_follows_shared_parser(self):
primitive = {"class_type": "PrimitiveNode", "inputs": {"value": 555}}
info = graph(seeded(["20", 0]), extra={"20": primitive})
@@ -207,6 +256,7 @@ class CardSeed(unittest.TestCase):
self.assertIsNone(parsed_positive(unparsable))
def test_every_shared_case_reads_the_projected_graphs(self):
withheld = []
for name, case in CASES.items():
with self.subTest(case=name):
info = {k: v for k, v in case.items() if v is not None}
@@ -225,11 +275,18 @@ class CardSeed(unittest.TestCase):
except GenerationParamsError:
self.assertIsNone(params)
continue
if name == "shared-prompt-pipeline":
# The inspector names the detailer (seed 333); the parser
# reads the base sampler (111): the card shows none.
self.assertEqual(expected["seed"], 111)
expected = {**expected, "seed": None}
withheld.append(name)
self.assertEqual(params, expected)
# Prompt text is the same as from a plain parse of the records.
if len(values) == len(info):
raw = extract_from_records(case["prompt"], case["workflow"])
self.assertEqual(params["positivePrompt"], raw["positivePrompt"])
self.assertEqual(withheld, ["shared-prompt-pipeline"])
if __name__ == "__main__":
+67
View File
@@ -466,6 +466,9 @@ class TestRestoreFromFile(BackupTestCase):
def test_restore_old_backup_without_annotations(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -502,6 +505,70 @@ class TestRestoreFromFile(BackupTestCase):
0,
)
def test_restore_annotations_era_backup_without_review_queue(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
src = self._make_source_db(["predates the review queue"])
with closing(sqlite3.connect(src)) as conn:
for table in tables:
conn.execute(f"DROP TABLE {table}")
conn.commit()
self.assertTrue(PromptModel.verify_database_file(src)[0])
ok, reason = PromptModel.verify_database_file(src, require_library_state=True)
self.assertFalse(ok)
self.assertEqual(reason, "Missing review queue schema")
self.db.model.restore_from_file(src)
self.assertEqual(_texts(self.path), ["predates the review queue"])
with closing(sqlite3.connect(self.path)) as conn:
self.assertEqual(
conn.execute("SELECT * FROM media_review_metadata").fetchall(),
[(1, 1, 0)],
)
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[0],
0,
)
ok, reason = PromptModel.verify_database_file(
self.path, require_library_state=True
)
self.assertTrue(ok, reason)
with closing(sqlite3.connect(src)) as conn: # the source is untouched
self.assertEqual(
conn.execute(
"SELECT COUNT(*) FROM sqlite_master WHERE name LIKE 'media_review%'"
).fetchone()[0],
0,
)
def test_restore_rejects_review_queue_without_annotations(self):
src = self._make_source_db(["review without annotations"])
with closing(sqlite3.connect(src)) as conn:
conn.execute("PRAGMA foreign_keys=OFF")
for table in (
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
"media_annotation_metadata",
):
conn.execute(f"DROP TABLE {table}")
conn.commit()
before = Path(src).read_bytes()
live_before = _texts(self.path)
ok, reason = PromptModel.verify_database_file(src)
self.assertFalse(ok)
self.assertIn("image annotation", reason)
with self.assertRaises(ValueError):
self.db.model.restore_from_file(src)
self.assertEqual(Path(src).read_bytes(), before)
self.assertEqual(_texts(self.path), live_before)
def test_restore_preserves_favorites_and_closes_handles(self):
self.db.set_image_favorite("output", "live.png")
with tempfile.TemporaryDirectory() as directory:
+18 -3
View File
@@ -23,7 +23,11 @@ sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
from py.config import ( # noqa: E402
OUTSIDE_ALLOWED_REASON,
GalleryConfig,
PromptManagerConfig,
)
EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
@@ -158,9 +162,10 @@ class GalleryRootTestCase(unittest.TestCase):
class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_filesystem_root_rejected(self):
"""Refused as outside the allowed folders: no reason names the layout."""
ok, reason = GalleryConfig.validate_gallery_root(os.path.abspath(os.sep))
self.assertFalse(ok)
self.assertIn("root", reason.lower())
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
@unittest.skipUnless(os.name == "nt", "drive roots only exist on Windows")
def test_windows_drive_root_rejected(self):
@@ -170,7 +175,17 @@ class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_home_directory_rejected(self):
ok, reason = GalleryConfig.validate_gallery_root(os.path.expanduser("~"))
self.assertFalse(ok)
self.assertIn("home", reason.lower())
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
def test_home_directory_inside_an_allowed_parent_is_still_refused(self):
"""Even when the environment opens the home's parent, home answers
like any outside path."""
home = os.path.expanduser("~")
os.environ[EXTRA_ROOTS_ENV] = os.path.dirname(home)
self.assertEqual(
GalleryConfig.gallery_root_problem(home),
("outside-allowed", OUTSIDE_ALLOWED_REASON),
)
def test_directory_outside_comfyui_rejected(self):
outside = Path(self.tmpdir) / "elsewhere"
+8 -3
View File
@@ -800,10 +800,15 @@ class TestUpstreamVerifiedNodeSemantics(unittest.TestCase):
prompt["31"] = dict(prompt["3"], inputs=dict(prompt["3"]["inputs"]))
prompt["31"]["inputs"]["latent_image"] = ["3", 0]
prompt["8"]["inputs"]["samples"] = ["31", 0]
# Unanchored, two listed samplers are two branches, as for KSampler.
self.assertEqual(_inspect({"prompt": prompt})["state"], "ambiguous")
# 31 continues 3's latent: the stages are compared, as for KSampler.
for anchor in (None, "9"):
result = _inspect({"prompt": prompt}, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "31")
)
prompt["31"]["inputs"]["positive"] = ["7", 0]
result = _inspect({"prompt": prompt}, output_node_id="9")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
self.assertEqual(result["reason_code"], "multistage-differs")
def test_seed_variance_enhancer_passes_its_conditioning(self):
prompt = _f02(
+105 -3
View File
@@ -55,6 +55,9 @@ def _minimal():
"runtime": [],
"savers": {},
"non_file_outputs": [],
"lineage": [
{"class": "VAEEncode", "slot": 0, "strategy": "continue", "input": "pixels"}
],
"stage_inputs": {"shaped": ["positive"], "with_model": ["samples"]},
"facts": {
"configuration_outputs": {},
@@ -199,16 +202,26 @@ class SpecEntriesTest(unittest.TestCase):
},
)
# Combine and concat keep every part in effect (additive); an average
# blends its parts, so a part may be weighted away.
self.assertEqual(
table.composites["ConditioningCombine"],
{"strategy": "combine", "inputs": ("conditioning_1", "conditioning_2")},
{
"strategy": "combine",
"inputs": ("conditioning_1", "conditioning_2"),
"additive": True,
},
)
for name in ("ConditioningConcat", "ConditioningAverage"):
for name, additive in (
("ConditioningConcat", {"additive": True}),
("ConditioningAverage", {}),
):
self.assertEqual(
table.composites[name],
{
"strategy": "combine",
"inputs": ("conditioning_to", "conditioning_from"),
**additive,
},
)
@@ -257,7 +270,12 @@ class SpecEntriesTest(unittest.TestCase):
)
# D18: decided at run time.
self.assertEqual(table.runtime, frozenset({"PromptSearchList"}))
# LoraManager's TriggerWord Toggle outputs the LoRA's trigger words as
# filtered at run time (option A leaves them out of a PromptManager).
self.assertEqual(
table.runtime,
frozenset({"PromptSearchList", "TriggerWord Toggle (LoraManager)"}),
)
comfy = "{name}_{counter:05}_.{ext}"
for name, image in (
@@ -464,6 +482,89 @@ class SpecEntriesTest(unittest.TestCase):
self.assertIsInstance(table.composites["ConditioningCombine"]["inputs"], tuple)
class LineageTableTest(_TableCase):
"""``lineage``: outputs whose image or latent continues one input's."""
def test_repository_lineage_lists_the_pass_through_nodes(self):
table = load_semantics()
self.assertEqual(
{key: dict(entry) for key, entry in table.lineage.items()},
{
("VAEDecode", 0): {"strategy": "continue", "input": "samples"},
("VAEDecodeTiled", 0): {"strategy": "continue", "input": "samples"},
("VAEEncode", 0): {"strategy": "continue", "input": "pixels"},
("VAEEncodeTiled", 0): {"strategy": "continue", "input": "pixels"},
("ImageUpscaleWithModel", 0): {
"strategy": "continue",
"input": "image",
},
("ImageScale", 0): {"strategy": "continue", "input": "image"},
("ImageScaleBy", 0): {"strategy": "continue", "input": "image"},
("ImageScaleToTotalPixels", 0): {
"strategy": "continue",
"input": "image",
},
("LatentUpscale", 0): {"strategy": "continue", "input": "samples"},
("LatentUpscaleBy", 0): {"strategy": "continue", "input": "samples"},
("Image Filter", 0): {"strategy": "continue", "input": "images"},
("Image Filter", 1): {"strategy": "continue", "input": "latents"},
},
)
# Lineage describes no prompt: its classes are not prompt categories.
self.assertFalse({name for name, _ in table.lineage} & table.all_classes)
def test_lineage_rows_are_validated(self):
def row(**changes):
entry = {
"class": "VAEEncode",
"slot": 0,
"strategy": "continue",
"input": "pixels",
}
entry.update(changes)
return entry
cases = (
({"lineage": {}}, "must be a list"),
({"lineage": ["VAEEncode"]}, "must be an object"),
({"lineage": [row(strategy="passthrough")]}, "unknown strategy"),
({"lineage": [row(slot=-1)]}, "slot"),
({"lineage": [row(slot=True)]}, "slot"),
({"lineage": [row(input="")]}, "input name"),
({"lineage": [row(note="x")]}, "field"),
({"lineage": [row(), row()]}, "duplicate"),
({"lineage": [row(**{"class": ""})]}, "class"),
)
for index, (change, fragment) in enumerate(cases):
with self.subTest(case=index):
doc = _minimal()
doc.update(change)
self.assertRejected(doc, fragment)
doc = _minimal()
doc["lineage"].append(row(slot=1, input="latents"))
self.assertEqual(len(self.load_doc(doc).lineage), 2)
def test_a_lineage_class_may_also_carry_conditioning(self):
# InpaintModelConditioning-like: conditioning slots, a latent slot.
doc = _minimal()
doc["lineage"].append(
{"class": "FluxGuidance", "slot": 1, "strategy": "continue", "input": "x"}
)
self.assertIn(("FluxGuidance", 1), self.load_doc(doc).lineage)
def test_composite_additive_must_be_a_boolean(self):
doc = _minimal()
doc["composites"]["ConditioningCombine"] = {
"strategy": "combine",
"inputs": ["conditioning_1", "conditioning_2"],
"additive": "yes",
}
self.assertRejected(doc, "additive must be a boolean")
doc["composites"]["ConditioningCombine"]["additive"] = True
table = self.load_doc(doc)
self.assertIs(table.composites["ConditioningCombine"]["additive"], True)
class ValidationTest(_TableCase):
def test_minimal_table_loads(self):
table = self.load_doc(_minimal())
@@ -501,6 +602,7 @@ class ValidationTest(_TableCase):
{"literal", "strip-newlines", "join-multi", "concat"}
),
"savers": frozenset({"comfy-counter", "vhs-counter"}),
"lineage": frozenset({"continue"}),
},
)
with self.assertRaises(TypeError):
+75
View File
@@ -101,6 +101,35 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase):
self.assertEqual(await _collect(job), [{"type": "complete"}])
async def test_lead_comes_first_and_nothing_published_meanwhile_is_lost(self):
"""S5 I1: the ``job`` frame leads, with the subscription already in place."""
job = Job("thumbnails", "Thumbnail generation")
job.publish({"type": "status"})
lead = {"type": "job"}
seen = []
async with contextlib.aclosing(job.events(lead=lead)) as events:
async for payload in events:
seen.append(payload)
if payload is lead:
# Published while the consumer is still sending the lead.
job.publish({"type": "progress"})
job.publish({"type": "complete"})
job.finish()
self.assertEqual(
seen,
[lead, {"type": "status"}, {"type": "progress"}, {"type": "complete"}],
)
async def test_lead_on_a_finished_job_then_its_last_event(self):
job = Job("thumbnails", "Thumbnail generation")
job.publish({"type": "complete"})
job.finish()
seen = []
async with contextlib.aclosing(job.events(lead={"type": "job"})) as events:
async for payload in events:
seen.append(payload)
self.assertEqual(seen, [{"type": "job"}, {"type": "complete"}])
async def test_departed_subscriber_is_forgotten(self):
job = Job("scan", "Scan")
job.publish({"type": "progress", "progress": 1})
@@ -151,6 +180,30 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase):
class TestJobRegistry(unittest.IsolatedAsyncioTestCase):
async def test_a_compatible_kind_never_blocks(self):
"""S5 M7: a caller may name running kinds that cannot conflict with it."""
registry = JobRegistry()
gate = asyncio.Event()
thumbs = registry.start(
"thumbnails",
"Thumbnail generation",
admission="guarded",
runner=_gated_runner(gate),
)
self.assertIs(registry.blocker("probe", admission="records"), thumbs)
self.assertIsNone(
registry.blocker("probe", admission="records", compatible=("thumbnails",))
)
backup = registry.start(
"backup", "Backup", admission="guarded", runner=_gated_runner(gate)
)
self.assertIs(
registry.blocker("probe", admission="records", compatible=("thumbnails",)),
backup,
)
gate.set()
await asyncio.wait_for(asyncio.gather(thumbs.task, backup.task), 2)
async def test_start_lists_and_finishes_a_job(self):
registry = JobRegistry()
gate = asyncio.Event()
@@ -421,6 +474,28 @@ class TestJobOutcomesAndLog(unittest.TestCase):
SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors")
)
def test_record_counted_counts_exactly_lists_capped_and_logs_nothing(self):
job = Job("thumbnails", "Thumbnail generation", groups=("skipped",))
rows = [
{"path": os.path.abspath(f"out{os.sep}f{i}.png"), "root": "output"}
for i in range(MAX_OUTCOME_ROWS + 5)
]
job.record_counted("skipped", 10_000, rows)
job.record_counted("skipped", 1, [{"path": "late.png", "root_index": True}])
self.assertEqual(job.counters["skipped"], 10_001)
self.assertEqual(len(job.outcomes["skipped"]), MAX_OUTCOME_ROWS)
self.assertEqual(job.outcomes["skipped"][0]["path"], "f0.png")
self.assertIsNone(job.outcomes["skipped"][0]["root_index"])
self.assertEqual(job.log, [])
with self.assertRaises(ValueError):
job.record_counted("bogus", 1)
def test_thumbnail_job_groups(self):
self.assertEqual(
jobs_module.THUMBNAIL_GROUP_NAMES, ("generated", "skipped", "errors")
)
self.assertEqual(jobs_module.THUMBNAILS_JOB_KIND, "thumbnails")
def test_image_tag_job_kind_and_groups(self):
# #249 D1: bulk Add tags / Auto-tag, counter-tile order.
self.assertEqual(getattr(jobs_module, "IMAGE_TAG_JOB_KIND", None), "image-tags")
+1
View File
@@ -1744,6 +1744,7 @@ class LibraryCounts(unittest.TestCase):
"negative_variants": 0,
"images": 0,
"rated_images": 0,
"review_pending": 0,
},
)
+28 -1
View File
@@ -52,7 +52,7 @@ from utils.prompt_inspector import (
)
from utils.prompt_inspector import inspect_prompt_evidence, unreadable_evidence
from utils.prompt_pair_resolver import INSPECTION_PROFILE
from utils.prompt_pair_resolver import ADDED_CONDITIONING_REASON, INSPECTION_PROFILE
try:
from tests import inspection_parity as parity
@@ -1293,6 +1293,33 @@ class LocalResolverParityTests(unittest.TestCase):
self.assertEqual(
answers["select-preview-dropped"]["resolution"]["sampler_id"], "3"
)
# Pass-through lineage: the chain shapes reach every new outcome.
self.assertEqual(
{
name: answers[f"select-{name}"]["resolution"]["reason"]
for name in ("hires-fix", "flux-three-stage", "hires-fix-differs")
},
{
"hires-fix": "",
"flux-three-stage": ADDED_CONDITIONING_REASON,
"hires-fix-differs": "Sampler stages have different prompts",
},
)
def test_local_roles_match_fuzzed_pass_through_chains(self):
cases = {}
for seed in range(4):
cases.update(selection_cases.passthrough_fuzz_cases(seed, 120))
answers = self._compare(cases)
reasons = [answer["resolution"]["reason"] for answer in answers.values()]
for reason in (
"",
"Sampler stages have different prompts",
"A later stage adds conditioning to the base prompt",
"Combined conditioning has different texts",
"Multiple sampler branches",
):
self.assertGreater(reasons.count(reason), 5, reason)
def test_local_roles_match_fuzzed_selection_sets(self):
cases = {}
+88
View File
@@ -471,6 +471,94 @@ class CaptureStoreTestCase(unittest.TestCase):
self.assertEqual(results["capture"].size, 60)
self.capture(workers, "b.png")
def _hold_worker(self, store, relative, **options):
"""Start a capture that holds a worker until ``proceed`` is set."""
entered, proceed = threading.Event(), threading.Event()
original = CaptureStore._read
results = {}
def blocking(store_, stream, size):
if threading.current_thread() is thread and not entered.is_set():
entered.set()
self.assertTrue(proceed.wait(WAIT))
return original(store_, stream, size)
def worker():
try:
results["capture"] = self.capture(store, relative, **options)
except Exception as error: # pragma: no cover - reported below
results["error"] = error
patcher = patch.object(CaptureStore, "_read", blocking)
patcher.start()
self.addCleanup(patcher.stop)
thread = threading.Thread(target=worker)
thread.start()
self.assertTrue(entered.wait(WAIT))
def finish():
proceed.set()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
return results["capture"]
self.addCleanup(proceed.set)
return finish
def test_background_capture_never_takes_the_last_worker(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
# The second background capture waits, then fails: one worker is free
# but it is the interactive request's.
started = time.monotonic()
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.05):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertGreaterEqual(time.monotonic() - started, 0.04)
# An interactive capture (Tag dialog, Apply) still gets the last one.
interactive = self.capture(store, "c.png")
self.assertEqual(interactive.size, 60)
finish()
def test_background_capture_waits_for_a_worker_to_free(self):
for name in ("a", "b"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
waited = {}
def background():
waited["capture"] = self.capture(store, "b.png", background=True)
thread = threading.Thread(target=background)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", WAIT):
thread.start()
finish()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertEqual(waited["capture"].size, 60)
def test_single_worker_store_admits_no_background_capture(self):
self.write("a.png", bytes(60))
store = self.make_store(max_workers=1)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "a.png", background=True)
self.assertEqual(self.capture(store, "a.png").size, 60)
def test_background_capture_leaves_the_last_capture_slot(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_captures=2)
self.capture(store, "a.png")
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertEqual(self.capture(store, "c.png").size, 60)
# -- lifetime ------------------------------------------------------------
def test_idle_and_hard_expiry_and_in_use_retention(self):
+66
View File
@@ -160,3 +160,69 @@ class TestPairIngestion(unittest.TestCase):
info = self.db.get_image_prompt_info(self.path)
self.assertEqual(info["pair_context"]["positive_text"], "cat, lora_trigger")
self.assertEqual(info["pair_context"]["text_source"], "execution")
def test_public_ingest_resolved_pair_contract_unchanged(self):
"""The public ingester keeps its result shape and its own transaction;
the extracted ``_ingest_resolved_pair_in`` runs in the caller's."""
from database import pair_ingestion
self.assertTrue(
hasattr(pair_ingestion, "_ingest_resolved_pair_in"),
"_ingest_resolved_pair_in is not extracted",
)
for state in ("ambiguous", "unresolved", "conflict"):
self.assertEqual(
ingest_resolved_pair(self.db, self.path, {}, {"state": state}),
{"status": state, "pairs_added": 0, "links_added": 0},
)
first = ingest_resolved_pair(self.db, self.path, {}, resolved())
self.assertEqual(
set(first),
{"status", "pairs_added", "links_added", "pair_id", "image_id"},
)
self.assertEqual(first["status"], "added")
(link,) = self.db.get_image_prompt_pairs(first["image_id"])
self.assertEqual(
link["provenance"],
{
"resolver_version": 1,
"positive": resolved()["positive"],
"negative": resolved()["negative"],
},
)
conn = self.db.model.get_connection()
self.assertFalse(conn.in_transaction)
# Inside the caller's transaction, with the caller's provenance; an
# exception afterwards rolls the link back with the caller's writes.
provenance = {"kind": "user-selection", "resolver_version": 1}
with self.assertRaises(RuntimeError):
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
self.assertEqual(inner["status"], "added")
raise RuntimeError("caller failed")
self.assertIsNone(self.db.get_image_prompt_info(self.path + "2"))
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
(link,) = self.db.get_image_prompt_pairs(inner["image_id"])
self.assertEqual(link["provenance"], provenance)
with self.db._pair_transaction() as held:
conflict = pair_ingestion._ingest_resolved_pair_in(
held, self.db, self.path, {}, resolved("bird")
)
self.assertEqual(
conflict, {"status": "conflict", "pairs_added": 0, "links_added": 0}
)
+4 -2
View File
@@ -82,7 +82,8 @@ class TestPairScan(AdminAPITestCase):
"class_type": "BasicGuider",
"inputs": {"width": 2048, "height": 2048},
}
self.assertEqual(self._scanned("untraceable.png", g), "ambiguous")
# Ambiguous with nothing to choose: unresolved, never a review item.
self.assertEqual(self._scanned("untraceable.png", g), "unresolved")
self.assertEqual(self.api.db.list_prompt_pairs()["total"], 0)
self.assertEqual(self._texts(), [])
@@ -159,9 +160,10 @@ class TestPairScan(AdminAPITestCase):
"\nSteps: 20, Sampler: Euler"
)
}
# No candidate to choose: reported unresolved, still not guessed.
self.assertEqual(
self.api._ingest_scanned_file(self.output_dir / "ambiguous.png", meta),
"ambiguous",
"unresolved",
)
self.assertEqual(self.api.db.get_recent_prompts()["prompts"], [])
+4
View File
@@ -258,6 +258,10 @@ class PairSchemaTests(unittest.TestCase):
],
[(1, 1, 0)],
)
self.assertEqual(
[tuple(r) for r in conn.execute("SELECT * FROM media_review_metadata")],
[(1, 1, 0)],
)
self.assertTrue(PromptModel.verify_database_file(live.db_path)[0])
self.assertTrue(
PromptModel.verify_database_file(live.db_path, require_library_state=True)[
+31 -20
View File
@@ -1245,7 +1245,7 @@ class TestGuiderSamplers(unittest.TestCase):
source["3"]["inputs"]["guider"] = [previous, 0]
self.assertEqual(_inspect(source)["reason_code"], "depth-limit")
def test_custom_advanced_upstream_of_a_sampler_is_multistage(self):
def test_custom_advanced_upstream_of_a_sampler_is_a_compared_stage(self):
source = _guided()
source["41"] = {
"class_type": "KSampler",
@@ -1256,15 +1256,18 @@ class TestGuiderSamplers(unittest.TestCase):
},
}
source["8"]["inputs"]["samples"] = ["41", 0]
# 41 continues 3's latent and reads the same texts as 3's guider.
for anchor in ("9", None):
result = _inspect(source, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "41")
)
source["41"]["inputs"]["negative"] = ["6", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["sampler_id"], "41")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], "Multistage sampler ancestry is ambiguous")
unanchored = _inspect(source)
self.assertEqual(unanchored["state"], "ambiguous")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
sorted(item["sampler_id"] for item in unanchored["candidates"]),
["3", "41"],
sorted(item["sampler_id"] for item in result["candidates"]), ["3", "41"]
)
@@ -2632,17 +2635,19 @@ class TestLatentAncestry(unittest.TestCase):
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
self.assertEqual(_inspect(source, output_node_id="9"), result)
def test_upstream_sampler_through_vaedecode_vaeencode_is_multistage(self):
for positive in (OTHER, parity.POS):
with self.subTest(positive=positive):
result = _inspect(_image_to_image(positive), output_node_id="63")
# Until PR3 compares stage texts, never one stage's prompt.
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason"], MULTISTAGE)
self.assertIsNone(result["reason_code"])
self.assertIsNone(result["positive"])
self.assertIsNone(result["negative"])
def test_upstream_sampler_through_vaedecode_vaeencode_is_compared(self):
# The latent continues the first stage's through lineage nodes: the
# stages are compared (spec 9.3), equal prompts resolve, others differ.
result = _inspect(_image_to_image(parity.POS), output_node_id="63")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "61"))
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
result = _inspect(_image_to_image(OTHER), output_node_id="63")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
[item["sampler_id"] for item in result["candidates"]], ["61", "3"]
)
# An upstream sampler reached through a non-latent input is a stage too.
source = _latent_from(
{
@@ -2959,9 +2964,15 @@ class TestLatentAncestry(unittest.TestCase):
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"], {"id": "60", "class_type": "MysteryUpscaler"})
# The listed stage is now an earlier sampler stage in the ancestry.
# The listed stage is now an earlier sampler stage, continued through
# VAEEncode: its prompt equals this stage's, so the chain resolves.
source["60"]["class_type"] = "UltimateSDUpscale"
result = _inspect(source, output_node_id="9")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "3"))
# Reached only through an unlisted image node, it stays unattributed.
source["61"] = {"class_type": "MysteryFilter", "inputs": {"image": ["60", 0]}}
source["5"]["inputs"]["pixels"] = ["61", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_latent_walk_budget_exhaustion_is_work_limit(self):
+419
View File
@@ -0,0 +1,419 @@
"""Multi-stage chains joined through image-space pass-through nodes.
A hires-fix or refiner run decodes sampler A's latent, filters, upscales and
scales the image, then encodes it again for sampler B. The semantics table's
``lineage`` section lists the nodes through which the latent or image carries
on unchanged in kind. When B's latent descends from A only through those nodes
and the stages share one pair, the chain resolves to it. A last stage that adds
conditioning to the base prompt (ConditioningCombine of the base and a style
trigger) keeps the base pair: the floor legacy linked. Everything else stays
ambiguous or unresolved. Ingestion (the default profile) is unchanged (D1).
"""
import json
import unittest
from utils import prompt_pair_resolver as resolver
from utils.prompt_inspector import inspect_prompt_evidence
from utils.prompt_pair_resolver import resolve_prompt_pair
try:
from tests import passthrough_graphs as shapes
except ImportError: # discovered with ``-s tests``
import passthrough_graphs as shapes
def _inspect(graph, **options):
return resolve_prompt_pair(
{"prompt": graph}, profile=resolver.INSPECTION_PROFILE, **options
)
def _texts(result):
return tuple(
result[role]["text"] if isinstance(result[role], dict) else None
for role in ("positive", "negative")
)
class TestHiresFixChain(unittest.TestCase):
def test_shared_pair_resolves_through_image_space_pass_through(self):
graph = shapes.hires_fix()
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(result["sampler_id"], "20")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertIsNone(result["reason_code"])
def test_the_earlier_stage_is_judged_in_its_chain(self):
# Sampler 10 only feeds sampler 20, so the file is 20's: one candidate.
result = _inspect(shapes.hires_fix())
self.assertEqual(result["candidates"], [])
def test_inspector_shows_the_shared_prompt(self):
evidence = inspect_prompt_evidence({"prompt": json.dumps(shapes.hires_fix())})
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
[evidence["prompts"][role]["text"] for role in ("positive", "negative")],
[shapes.POSITIVE, shapes.NEGATIVE],
)
def test_different_stage_prompts_are_multistage_ambiguous(self):
result = _inspect(shapes.hires_fix(second_positive=shapes.OTHER))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
sorted(_texts(item)[0] for item in result["candidates"]),
sorted([shapes.POSITIVE, shapes.OTHER]),
)
def test_an_unlisted_image_node_breaks_the_lineage(self):
# The walk still finds the earlier sampler, but not through listed
# pass-through nodes: the stages are never compared.
result = _inspect(shapes.hires_fix(filter_class="MysteryImageEffect"))
self.assertEqual(result["state"], "ambiguous")
later = next(
item for item in result["candidates"] if item["sampler_id"] == "20"
)
self.assertEqual(later["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_second_earlier_sampler_in_the_ancestry_stays_unattributed(self):
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {
"class_type": "ImageBlend",
"inputs": {"image1": ["14", 0], "image2": ["31", 0], "blend_factor": 0.5},
}
graph["16"]["inputs"]["image"] = ["32", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_direct_latent_link_is_a_chain_too(self):
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "20"))
def test_an_unreadable_earlier_stage_blocks_the_chain(self):
graph = shapes.hires_fix(second_positive=shapes.POSITIVE)
graph["10"]["inputs"]["positive"] = ["6", 0]
graph["6"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryPrompt")
def test_generated_text_stays_untraceable(self):
result = _inspect(shapes.generated_text_chain())
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "TextGenerate")
def test_ingestion_is_unchanged(self):
# D1: the default profile keeps its historical latent rule.
result = resolve_prompt_pair({"prompt": shapes.hires_fix()})
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(
{item["reason"] for item in result["candidates"]},
{"Unsupported latent ancestry"},
)
class TestFluxThreeStage(unittest.TestCase):
def test_added_style_conditioning_keeps_the_base_pair(self):
graph = shapes.flux_three_stage()
for anchor in (None, "61"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
self.assertEqual(result["sampler_id"], "50")
# D126: the positive says a later stage added conditioning.
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
self.assertEqual(
result["negative"]["provenance"]["text_source"],
"zeroed-conditioning",
)
def test_any_added_conditioning_is_labelled_never_measured(self):
# Ruling I2: a whole other prompt combined in is labelled the same way;
# no length or "short trigger" heuristic.
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], shapes.POSITIVE)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
def test_trigger_words_and_added_conditioning_carry_both_labels(self):
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_AND_ADDED_SOURCE,
)
def test_a_custom_advanced_last_stage_keeps_its_failure(self):
# Review M2: a guider's combined positive is not floored.
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
result = _inspect(graph, output_node_id="61")
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_concat_also_adds_to_the_base_prompt(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningConcat"))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
def test_an_average_blends_the_base_away(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningAverage"))
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_combine_without_the_base_prompt_is_not_the_base_pair(self):
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_different_last_negative_is_not_the_base_pair(self):
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_different_base_stages_are_never_floored(self):
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertNotEqual(result["state"], "resolved")
def test_a_rerouted_combine_is_followed(self):
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
def test_an_unreadable_last_negative_or_encoder_parts_keep_the_failure(self):
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
self.assertEqual(_inspect(graph)["reason_code"], "composite")
# Differing encoder parts are a composite too, but not an added stage.
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
self.assertEqual(result["node"]["class_type"], "CLIPTextEncodeSDXL")
def test_inspector_shows_the_base_prompt(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.flux_three_stage())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(evidence["prompts"]["positive"]["text"], shapes.POSITIVE)
class TestRunTimePromptManagerParts(unittest.TestCase):
"""Maintainer option A: a run-time prepend or append is left out, labelled."""
def test_trigger_words_prepend_resolves_to_the_prompt_manager_text(self):
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(
shapes.hires_fix_with_trigger_words(), output_node_id=anchor
)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
provenance = result["positive"]["provenance"]
self.assertEqual(
provenance["text_source"], resolver.RUNTIME_PARTS_SOURCE
)
self.assertEqual(result["positive"]["storage_text"], shapes.POSITIVE)
# The negative is untouched by the positive's run-time part.
self.assertEqual(
result["negative"]["provenance"]["text_source"], "graph-inferred"
)
def test_append_and_a_single_sampler_are_labelled_too(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = "masterpiece"
graph["2"]["inputs"]["append_text"] = ["8", 0]
graph["20"]["inputs"]["latent_image"] = ["4", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "resolved")
# The literal prepend is still combined exactly as before.
self.assertEqual(_texts(result)[0], "masterpiece " + shapes.POSITIVE)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
def test_the_inspector_shows_the_prompt_and_its_source(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.hires_fix_with_trigger_words())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
evidence["prompts"]["positive"],
{
"status": "resolved",
"text": shapes.POSITIVE,
"source": resolver.RUNTIME_PARTS_SOURCE,
},
)
def test_literal_and_traceable_parts_are_combined_as_before(self):
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
result = _inspect(graph)
self.assertEqual(_texts(result)[0], "trig9word " + shapes.POSITIVE)
self.assertEqual(result["reason"], "")
self.assertEqual(
result["positive"]["provenance"]["text_source"], "graph-inferred"
)
def test_only_a_run_time_feeder_is_left_out(self):
# An unknown feeder stays unsupported, naming it.
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
result = _inspect(graph)
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryWords")
# A run-time node deeper inside the part makes the whole part run time,
# but it is not a direct feeder: never leave out a traced literal.
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_main_text_from_a_run_time_node_is_never_left_out(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = ""
graph["2"]["inputs"]["text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_label_survives_nodes_between_the_prompt_manager_and_sampler(self):
# Review C1: encoders, chained PromptManagers and concatenations.
expected = {
"text-encode": shapes.POSITIVE,
"chained": shapes.POSITIVE,
"concat": shapes.POSITIVE + ", extra",
}
for shape, text in expected.items():
with self.subTest(shape=shape):
result = _inspect(shapes.trigger_words_through(shape))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], text)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_PARTS_SOURCE,
)
def test_a_partial_prompt_never_equals_a_complete_one(self):
# Review C2: never collapse a branch that left out trigger words with
# one that did not, nor compare such stages as equal.
result = _inspect(shapes.branches_with_and_without_trigger_words())
self.assertEqual(
(result["state"], result["reason"]),
("ambiguous", "Multiple sampler branches"),
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(
(result["state"], result["reason_code"]),
("ambiguous", "multistage-differs"),
)
# Two PromptManagers leaving out two different toggles differ too.
graph["8b"] = shapes.trigger_toggle("otherword")
graph["9"]["inputs"]["prepend_text"] = ["8b", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["reason_code"], "multistage-differs")
def test_only_listed_run_time_classes_are_left_out(self):
# Review M1: a JoinStringMulti with a linked list flag is run time
# too, but it is not a listed run-time node: never left out.
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "runtime-value")
)
self.assertEqual(result["node"]["class_type"], "JoinStringMulti")
def test_ingestion_is_unchanged(self):
result = resolve_prompt_pair({"prompt": shapes.hires_fix_with_trigger_words()})
self.assertEqual(result["state"], "ambiguous")
class TestStillAmbiguous(unittest.TestCase):
def test_edit_batch_with_distinct_prompts_stays_ambiguous(self):
result = _inspect(shapes.edit_batch(4))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason"], "Multiple sampler branches")
self.assertEqual(len({_texts(item)[0] for item in result["candidates"]}), 4)
if __name__ == "__main__":
unittest.main()
+1091
View File
File diff suppressed because it is too large Load Diff
+335
View File
@@ -0,0 +1,335 @@
"""Needs review queues a file only when there is something to choose.
The maintainer's live queue (#284 / #288) held 39 items whose 11 candidates
all read "unresolved", "No positive / No negative": the ingestion resolver
found several sampler branches but could trace no prompt on any of them.
An ambiguous result is reviewable only when at least two candidates carry a
resolved prompt text. With none, the file could not be traced; with exactly
one, the other branches may still have made the file, so the one text is
not certain either (the resolver never flattens unknown semantics). Both
stay unresolved and are never queued, and a rescan removes the pending,
undecided items such files already have.
"""
import os
import unittest
from unittest.mock import patch
try:
from tests.test_review import DELIMITERS, write_png
from tests.test_review_outcome import OutcomeTestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_review import DELIMITERS, write_png
from test_review_outcome import OutcomeTestCase
from database.review_schema import evidence_offers_choice
from utils import review_capture
def opaque_graph(*opaque, traced=()):
"""Several KSamplers: ``opaque`` positives come from an encoder the
ingestion resolver does not know, ``traced`` ones from CLIPTextEncode.
The first sampler is opaque, so the scan's browser-parser fallback links
nothing either.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
}
sides = [("OpaqueTextEncode", "prompt", t) for t in opaque]
sides += [("CLIPTextEncode", "text", t) for t in traced]
for index, (encoder_class, key, text) in enumerate(sides):
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
graph[encoder] = {"class_type": encoder_class, "inputs": {key: text}}
graph[sampler] = {
"class_type": "KSampler",
"inputs": {
"model": ["1", 0],
"positive": [encoder, 0],
"negative": ["2", 0],
},
}
return graph
def _always_reviewable(_resolution):
return True
class ChoosableTestCase(OutcomeTestCase):
def _record_pre_fix(self, name):
"""An item as the pre-fix recorder stored it (no choosable test)."""
with patch.object(review_capture, "reviewable", _always_reviewable):
self._record(name)
def _write(self, name, graph):
write_png(self.output_dir / name, prompt=graph)
return self.output_dir / name
class TestChoosableCandidates(unittest.TestCase):
def _resolution(self, *states):
candidates = []
for index, state in enumerate(states):
side = {"text": f"t{index}"} if state == "resolved" else None
candidates.append(
{"sampler_id": str(index), "state": state, "positive": side}
)
return {"state": "ambiguous", "candidates": candidates}
def test_two_resolved_candidates_are_reviewable(self):
resolution = self._resolution("resolved", "resolved", "unresolved")
self.assertTrue(review_capture.reviewable(resolution))
def test_none_or_one_resolved_candidate_is_not_reviewable(self):
for states in ((), ("unresolved",) * 11, ("resolved", "unresolved")):
with self.subTest(states=states):
resolution = self._resolution(*states)
self.assertFalse(review_capture.reviewable(resolution))
def test_blank_or_missing_text_does_not_count(self):
resolution = self._resolution("resolved", "resolved")
resolution["candidates"][0]["positive"] = {"text": " "}
self.assertFalse(review_capture.reviewable(resolution))
resolution["candidates"][0]["positive"] = None
self.assertFalse(review_capture.reviewable(resolution))
def test_nested_ambiguous_stage_counts_its_own_candidates(self):
inner = self._resolution("resolved", "resolved")["candidates"]
resolution = {
"state": "ambiguous",
"candidates": [{"state": "ambiguous", "candidates": inner}],
}
self.assertTrue(review_capture.reviewable(resolution))
def test_a_result_that_is_not_ambiguous_is_not_reviewable(self):
resolution = self._resolution("resolved", "resolved")
resolution["state"] = "resolved"
self.assertFalse(review_capture.reviewable(resolution))
class TestStoredEvidenceOffersChoice(unittest.TestCase):
def test_unlimited_evidence_needs_two_choosable_candidates(self):
side = {"text": "cat"}
two = [{"state": "resolved", "positive": side}] * 2
self.assertTrue(evidence_offers_choice({"candidates": two}))
one = two[:1] + [{"state": "unresolved", "positive": None}]
self.assertFalse(evidence_offers_choice({"candidates": one}))
def test_limited_evidence_offers_a_choice_only_with_two_candidates(self):
self.assertTrue(
evidence_offers_choice({"limited": True, "candidate_total": 18})
)
for total in (0, 1, None, "18", True):
with self.subTest(total=total):
evidence = {"limited": True, "candidate_total": total}
self.assertFalse(evidence_offers_choice(evidence))
def test_malformed_evidence_offers_nothing(self):
for evidence in (None, [], {"candidates": "x"}, {}):
with self.subTest(evidence=evidence):
self.assertFalse(evidence_offers_choice(evidence))
class TestNothingToChooseIsNotQueued(ChoosableTestCase):
async def test_no_candidate_with_text_could_not_be_traced(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_items"], 0)
(row,) = job["outcomes"]["review"]
self.assertEqual(row["path"], "none.png")
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"]["review"], 1, job["counters"])
self.assertEqual(self._capture_files(), [])
async def test_exactly_one_candidate_with_text_stays_unresolved(self):
"""One traced branch beside untraced ones is not certain: no link."""
self._roots()
self._write("one.png", opaque_graph("cat", traced=("dog",)))
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"].get("added", 0), 0, job["counters"])
self.assertEqual(job["counters"].get("known", 0), 0, job["counters"])
async def test_two_candidates_with_text_are_queued(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
(row,) = job["outcomes"]["review"]
self.assertIn("more than one prompt could match", row["detail"])
async def test_parameters_delimiter_ambiguity_is_not_queued(self):
"""Two A1111 negative delimiters offer no candidate to choose."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
def test_recorder_reports_untraced_and_stores_nothing(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
for name in ("none.png", "one.png"):
with self.subTest(name=name):
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertEqual(outcome, "untraced")
self.assertEqual(self._items(), [])
self.assertEqual(self._capture_files(), [])
def test_monitor_does_not_queue_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
class TestUntracedItemsLeaveTheQueue(ChoosableTestCase):
async def test_rescan_removes_pending_items_without_a_choice(self):
"""The maintainer's 39 items: queued before the rule, gone on rescan."""
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
for name in ("none.png", "one.png", "two.png"):
self._record_pre_fix(name)
revision = self._review_revision()
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(self._pending(), 1)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_untraced"], 2)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertIn(
"= 2 images left Needs review · nothing to choose", "\n".join(job["log"])
)
async def test_decided_items_of_untraced_files_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
self._write(name, opaque_graph("cat", traced=(name, name + "2")))
self._record_pre_fix(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# The files now carry graphs without a choice, queued pre-fix.
for name in ("sel.png", "unres.png", "pending.png"):
self._write(name, opaque_graph("a cow", "dog"))
self._record_pre_fix(name)
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_untraced"], 3)
async def test_an_item_with_a_choice_survives_beside_an_untraced_one(self):
"""Removal judges each item by its own evidence: an item that offers
a choice is not removed although the file's new bytes offer none."""
self._roots()
self._write("f.png", opaque_graph("x", traced=("cat", "dog")))
self._record_pre_fix("f.png")
(choosable,) = self._items()
self._write("f.png", opaque_graph("a cow", "dog"))
self._record_pre_fix("f.png")
self.assertEqual(self._pending(), 2)
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [choosable["id"]])
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_limited_items_go_only_without_enough_candidates(self):
"""A pre-fix delimiter item (no candidate) goes; a limited item of 18
candidates may hold a choice and stays, though the bytes now offer
none."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
self._record_pre_fix("delims.png")
many = [f"prompt {n}" for n in range(17)]
self._write("many.png", opaque_graph("x", traced=many))
self._record_pre_fix("many.png")
kept = [i["id"] for i in self._items() if "many" in i["locator"]]
self._write("many.png", opaque_graph("a cow", "dog"))
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], kept)
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_choosable_pending_item_stays(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
await self._scan()
(item,) = self._items()
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"]["review_untraced"], 0)
async def test_a_failed_removal_is_logged_and_counted_not_fatal(self):
import logging
import sqlite3
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
with (
patch.object(
self.api.db,
"settle_untraced_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan("all")
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
await self._scan("all")
self.assertEqual(self._pending(), 0)
def test_monitor_removes_the_pending_item_of_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
# A pre-fix item of other bytes keeps the stat shortcut out of the way.
self._write("none.png", opaque_graph("a cow", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
if __name__ == "__main__":
unittest.main()
+374
View File
@@ -0,0 +1,374 @@
"""Needs review takes only files whose final outcome is ambiguous and unlinked.
The maintainer's live scan (#284 / #288) queued every file the strict
resolver found ambiguous, also those the legacy browser-parser fallback then
linked and those already linked, so the queue (10,232) was far larger than
the scan's Needs review group. The rule now: a file is queued only when the
resolver finds it ambiguous, the fallback does not link it and no saved row
links it. A pending, undecided item whose file is linked (by the fallback, a
later resolution or the monitor) is removed on the next scan or monitor
pass; an item that carries a decision is never touched.
"""
import json
import logging
import os
import sqlite3
import unittest
from unittest.mock import patch
try:
from tests.test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
except ImportError: # discovered with tests/ as the top-level directory
from test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
from utils.prompt_tracker import PromptTracker
from utils import review_capture
def nan_stamped(graph):
"""ComfyUI's ``is_changed: [NaN]`` stamp (written as a bare NaN literal)."""
for node in graph.values():
node["is_changed"] = [float("nan")]
return graph
class OutcomeTestCase(ReviewTestCase):
def _pending(self):
return self.api.db.library_counts()["review_pending"]
def _record(self, name):
"""An item as the pre-fix producers recorded it (any ambiguous file)."""
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertIn(outcome, ("recorded", "limited"))
def _decide(self, item, outcome):
conn = self.api.db.model.get_connection()
candidate = None
if outcome == "selected":
candidate = json.loads(item["evidence"])["candidates"][0]["hash"]
with conn:
conn.execute(
"UPDATE media_review_items SET state=?, revision=1 WHERE id=?",
(outcome, item["id"]),
)
conn.execute(
"INSERT INTO media_review_decisions(id,item_id,decision_revision,"
"created_at,outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)",
(
f"00000000-0000-4000-8000-{len(self._decisions()) + 1:012d}",
item["id"],
1,
NOW,
outcome,
candidate,
'{"kind":"user-selection"}',
),
)
def _decisions(self):
conn = self.api.db.model.get_connection()
return [
tuple(row)
for row in conn.execute(
"SELECT id,item_id,outcome FROM media_review_decisions ORDER BY id"
)
]
def _link(self, path, text="already saved"):
prompt_id = self.api.db.save_prompt(
text=text, prompt_hash=generate_prompt_hash(text)
)
self.api.db.link_image_to_prompt(prompt_id, str(path))
return prompt_id
@staticmethod
def _rows(job, group):
return sorted(row["path"] for row in job["outcomes"].get(group, []))
class TestOnlyUnlinkedAmbiguousFilesAreQueued(OutcomeTestCase):
async def test_nan_stamped_graph_the_fallback_links_is_not_queued(self):
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self.assertIn(b"NaN", path.read_bytes())
job = await self._scan()
self.assertEqual(job["counters"]["added"], 1, job["counters"])
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._pending(), 0)
async def test_fallback_link_to_a_known_prompt_is_not_queued(self):
self._roots()
write_png(self.output_dir / "a.png", prompt=ambiguous_graph("cat", "dog"))
await self._scan()
write_png(self.output_dir / "b.png", prompt=ambiguous_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
async def test_already_linked_file_is_not_queued(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._link(path)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._rows(job, "review"), [])
self.assertEqual(self._rows(job, "known"), ["linked.png"])
async def test_genuinely_ambiguous_file_is_queued(self):
self._roots()
path = self.output_dir / "two.png"
write_png(path, prompt=nan_stamped(unlinkable_graph("cat", "dog")))
# Beside it, a file the fallback links: only the first is queued.
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["state"], "pending")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertEqual(self._rows(job, "review"), ["two.png"])
self.assertEqual(self._pending(), 1)
self.assertIn("? 1 image added to Needs review", "\n".join(job["log"]))
async def test_counter_and_queue_agree_with_the_review_group(self):
"""``review_items`` and the queue total equal the review group less
its unresolved files, on the first scan and on a rescan."""
self._roots()
for name in ("u1.png", "u2.png"):
write_png(self.output_dir / name, prompt=unlinkable_graph("cat", name))
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
unresolved = {"1": {"class_type": "KSampler", "inputs": {}}}
write_png(self.output_dir / "none.png", prompt=unresolved)
write_png(self.output_dir / "one.png", prompt=resolved_graph())
for _ in range(2):
job = await self._scan()
review = self._rows(job, "review")
unresolved_rows = sorted(
row["path"]
for row in job["outcomes"]["review"]
if "could not be traced" in row["detail"]
)
queued = [name for name in review if name not in unresolved_rows]
# delims.png has nothing to choose: "could not be traced".
self.assertEqual(queued, ["u1.png", "u2.png"], job)
self.assertIn("delims.png", unresolved_rows)
self.assertEqual(job["scope"]["review_items"], 2)
self.assertEqual(self._pending(), 2)
self.assertIn("? 2 images already in Needs review", "\n".join(job["log"]))
class TestQueueSelfHeals(OutcomeTestCase):
async def test_pending_item_of_a_fallback_linked_file_is_removed(self):
"""The maintainer's case: queued by the old rule, linked on rescan."""
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self._record("nan.png")
revision = self._review_revision()
self.assertEqual(self._pending(), 1)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_settled"], 1)
self.assertIn("= 1 image left Needs review · now linked", "\n".join(job["log"]))
async def test_pending_item_of_an_already_linked_file_is_removed(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("linked.png")
self._link(path)
await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_pending_item_of_a_skipped_known_file_is_removed(self):
"""A file the "new" scope skips as known still settles its old item."""
self._roots()
path = self.output_dir / "one.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("one.png")
write_png(path, prompt=resolved_graph())
first = await self._scan()
self.assertEqual(self._rows(first, "added"), ["one.png"])
# An item a pre-fix scan left for the file's old bytes.
self.assertEqual(self._items(), [])
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 9, 0))
self._record("one.png")
write_png(path, prompt=resolved_graph())
second = await self._scan()
self.assertEqual(self._rows(second, "known"), ["one.png"])
self.assertEqual(self._items(), [])
async def test_a_failed_settle_is_logged_and_counted_not_fatal(self):
self._roots()
write_png(self.output_dir / "nan.png", prompt=ambiguous_graph("cat", "dog"))
self._record("nan.png")
with (
patch.object(
self.api.db,
"settle_linked_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan()
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertEqual(job["scope"]["review_settled"], 0)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
# The next scan settles it.
await self._scan()
self.assertEqual(self._pending(), 0)
async def test_still_unlinked_pending_item_stays(self):
self._roots()
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
await self._scan()
(item,) = self._items()
job = await self._scan()
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"].get("review_settled", 0), 0)
async def test_decided_items_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
write_png(self.output_dir / name, prompt=ambiguous_graph("cat", name))
self._record(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# A pending neighbour the fallback links: it alone leaves the queue.
write_png(self.output_dir / "pending.png", prompt=ambiguous_graph("x", "y"))
self._record("pending.png")
self.assertEqual(self._pending(), 1)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_settled"], 1)
class TestMonitorRule(OutcomeTestCase):
def _handler(self):
handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db))
handler.review_recorder = self.api.record_review_file
return handler
def _context(self):
prompt_id = self.api.db.save_prompt(
text="latest", prompt_hash=generate_prompt_hash("latest")
)
return {"id": prompt_id, "text": "latest", "fallback": True}
async def test_monitor_linked_ambiguous_output_is_not_queued(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
async def test_monitor_link_removes_the_files_pending_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
self.assertEqual(self._pending(), 1)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_monitor_never_touches_a_decided_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
(item,) = self._items()
self._decide(item, "unresolved")
before = (self._items(), self._decisions())
other = self.output_dir / "other.png"
write_png(other, prompt=graph, color=(0, 0, 9))
self._record("other.png")
handler, context = self._handler(), self._context()
handler.link_image_to_prompt(str(path), context, {"prompt": graph})
handler.link_image_to_prompt(str(other), context, {"prompt": graph})
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
async def test_record_review_file_applies_the_scan_rule(self):
self._roots()
write_png(self.output_dir / "fallback.png", prompt=ambiguous_graph("a", "b"))
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
linked = self.output_dir / "linked.png"
write_png(linked, prompt=unlinkable_graph("cat", "dog"))
self._link(linked)
record = self.api.record_review_file
self.assertEqual(record(str(self.output_dir / "fallback.png")), "skipped")
self.assertEqual(record(str(linked)), "linked")
self.assertEqual(record(str(self.output_dir / "two.png")), "recorded")
self.assertEqual(len(self._items()), 1)
if __name__ == "__main__":
unittest.main()
+918
View File
@@ -0,0 +1,918 @@
"""Needs review schema group (#249): install, validation and restore safety."""
import hashlib
import json
import re
import sqlite3
import tempfile
import unittest
from contextlib import closing
from pathlib import Path
from unittest.mock import patch
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
from database.models import PromptModel
from database.operations import PromptDatabase
try: # guarded so a missing module fails assertions, not test loading
from database.review_schema import (
canonical_evidence,
evidence_digest,
migrate_review_state,
review_schema_error,
)
except ImportError:
canonical_evidence = None
evidence_digest = None
migrate_review_state = None
review_schema_error = None
METADATA = "media_review_metadata"
ITEMS = "media_review_items"
DECISIONS = "media_review_decisions"
TABLES = (METADATA, ITEMS, DECISIONS)
ANNOTATION_TABLES = (
"media_annotation_metadata",
"media_annotation_files",
"media_annotation_bytes",
"media_image_tags",
)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
f"CREATE TABLE {METADATA}(singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
"CHECK(schema_version=1), review_index_revision INTEGER NOT NULL "
"DEFAULT 0 CHECK(review_index_revision>=0))"
),
ITEMS: (
f"CREATE TABLE {ITEMS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
"byte_id TEXT NOT NULL REFERENCES media_annotation_bytes(id) "
"ON DELETE RESTRICT, spelled_path TEXT NOT NULL CHECK(length(CAST("
"spelled_path AS BLOB)) BETWEEN 1 AND 16384), "
"resolver_version INTEGER NOT NULL, "
"evidence_hash TEXT NOT NULL CHECK(length(evidence_hash)=64 AND "
"evidence_hash NOT GLOB '*[^0-9a-f]*'), evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"candidate_count INTEGER NOT NULL CHECK(candidate_count BETWEEN 0 AND 16), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), state TEXT NOT NULL "
"CHECK(state IN ('pending','selected','unresolved')), revision INTEGER "
"NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, updated_at TEXT "
"NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS: (
f"CREATE TABLE {DECISIONS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"item_id TEXT NOT NULL REFERENCES {ITEMS}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
"(length(candidate_hash)=64 AND candidate_hash NOT GLOB '*[^0-9a-f]*')), "
"evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
INDEXES = {
"idx_media_review_items_queue": (
f"CREATE INDEX idx_media_review_items_queue ON {ITEMS}"
"(state, created_at, id)"
),
"idx_media_review_decisions_pair": (
f"CREATE INDEX idx_media_review_decisions_pair ON {DECISIONS}(pair_id)"
),
"idx_media_review_decisions_image": (
f"CREATE INDEX idx_media_review_decisions_image ON {DECISIONS}(image_id)"
),
}
AUTOINDEXES = {ITEMS: 2, DECISIONS: 2}
LEGACY_TABLES = (
"prompts",
"generated_images",
"tags",
"prompt_tags",
"prompt_pairs",
"prompt_pair_tags",
"image_favorites",
) + ANNOTATION_TABLES
FILE_ID = "0f8e4b2a-1c3d-4e5f-8a9b-0c1d2e3f4a5b"
BYTE_ID = "6a1b2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d"
ITEM_ID = "1d2e3f4a-5b6c-4d7e-8f9a-0b1c2d3e4f5a"
DECISION_ID = "2e3f4a5b-6c7d-4e8f-9a0b-1c2d3e4f5a6b"
OTHER_ID = "7b2c3d4e-5f6a-4b7c-9d0e-1f2a3b4c5d6e"
DIGEST = hashlib.sha256(b"img").hexdigest()
CANDIDATE = hashlib.sha256(b"candidate").hexdigest()
NOW = "2026-10-22T00:00:00.000000+00:00"
EVIDENCE = '{"candidates":[],"sha256":"' + DIGEST + '"}'
EVIDENCE_HASH = hashlib.sha256(EVIDENCE.encode("ascii")).hexdigest()
def _compact(sql):
return re.sub(r"\s+", "", sql or "").lower()
def expected_catalog():
objects = {("table", name, _compact(sql)) for name, sql in DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in INDEXES.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def catalog(conn):
names = tuple(TABLES) + tuple(INDEXES)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return {
(kind, name, _compact(sql))
for kind, name, sql in conn.execute(
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + tuple(TABLES),
)
}
def rows(conn, sql, *params):
"""Plain tuples whatever row factory *conn* uses."""
return [tuple(row) for row in conn.execute(sql, params)]
def metadata(conn):
return rows(conn, f"SELECT schema_version,review_index_revision FROM {METADATA}")
def table_rows(conn, tables):
found = {}
for table in tables:
present = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
).fetchone()
if present:
found[table] = rows(conn, f"SELECT * FROM {table}") # nosec B608
return found
def insert_byte(conn, file_id=FILE_ID, byte_id=BYTE_ID, locator="/out/img.png"):
conn.execute(
"INSERT INTO media_annotation_files VALUES(?,?,?)", (file_id, locator, NOW)
)
conn.execute(
"INSERT INTO media_annotation_bytes VALUES(?,?,?,?,?,?,?,0)",
(byte_id, file_id, DIGEST, 3, 1_700_000_000_000_000_000, NOW, NOW),
)
def insert_item(conn, *, item_id=ITEM_ID, evidence=EVIDENCE, candidates=2, **extra):
values = {
"id": item_id,
"byte_id": BYTE_ID,
"spelled_path": "/out/img.png",
"resolver_version": 1,
"evidence_hash": hashlib.sha256(evidence.encode("utf-8")).hexdigest(),
"evidence": evidence,
"candidate_count": candidates,
"limited": 0,
"state": "pending",
"revision": 0,
"created_at": NOW,
"updated_at": NOW,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {ITEMS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_decision(conn, *, decision_id=DECISION_ID, **extra):
values = {
"id": decision_id,
"item_id": ITEM_ID,
"decision_revision": 1,
"created_at": NOW,
"outcome": "selected",
"candidate_hash": CANDIDATE,
"evidence": EVIDENCE,
"pair_id": None,
"image_id": None,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {DECISIONS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_group_rows(conn, **item):
insert_byte(conn)
insert_item(conn, **item)
insert_decision(conn)
def populate_legacy(conn):
conn.execute(
"INSERT INTO prompts(id,text,hash,category,notes,run_count) VALUES"
"(7,'positive','p','keep','notes',12)"
)
conn.execute("INSERT INTO prompts(id,text,hash) VALUES(9,'negative','n')")
conn.execute(
"INSERT INTO generated_images(id,prompt_id,image_path,filename,file_path) "
"VALUES(4,7,'/tmp/image.png','image.png','/tmp/image.png')"
)
conn.execute("INSERT INTO tags(id,name) VALUES(3,'keep-tag')")
conn.execute("INSERT INTO prompt_tags VALUES(7,3)")
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(6,7,9)"
)
conn.execute("INSERT INTO prompt_pair_tags VALUES(6,3)")
conn.execute("INSERT INTO image_favorites VALUES('root','a.png',?)", (NOW,))
def full_capacity_evidence():
"""Sixteen candidates of control, lone-surrogate, astral and combining
text, exactly 8 MiB."""
text = '\x00\x07\x1f
\ud800 é é "q" \\ \U0001f9ea'
candidates = [
{
"hash": hashlib.sha256(str(n).encode()).hexdigest(),
"negative": {"text": text * 3},
"positive": {"text": text * 5},
"sampler_id": str(n),
"state": "resolved",
}
for n in range(16)
]
value = {"candidates": candidates, "pad": ""}
base = len(canonical_evidence(value).encode("utf-8"))
value["pad"] = "x" * (MAX_EVIDENCE_BYTES - base)
return value
class ReviewSchemaTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.addCleanup(assert_closed, self, self.temp.name)
self.addCleanup(PromptModel.reset_schema_cache)
self.models = []
self.addCleanup(self.close_models)
self.path = str(Path(self.temp.name) / "live.db")
def close_models(self):
for model in self.models:
model.close_all()
def model(self, path=None):
model = PromptModel(path or self.path)
self.models.append(model)
return model
def require_module(self):
self.assertIsNotNone(
review_schema_error, "database.review_schema is not implemented"
)
def drop_tables(self, path, tables=TABLES):
"""Remove tables from a closed database file (an old backup)."""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in reversed(tables):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
insert_group_rows(conn)
snapshot = table_rows(conn, TABLES)
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(table_rows(conn, TABLES), snapshot)
# Running the migration again on a current database changes nothing.
migrate_review_state(conn)
self.assertEqual(table_rows(conn, TABLES), snapshot)
model.close_all()
# Items and decisions keep the byte record and the item they cite.
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
with self.assertRaises(sqlite3.IntegrityError):
conn.execute("DELETE FROM media_annotation_files")
with self.assertRaises(sqlite3.IntegrityError):
conn.execute(f"DELETE FROM {ITEMS}")
self.assertEqual(table_rows(conn, TABLES), snapshot)
def test_startup_validates_stored_evidence_once(self):
"""A reopen streams the evidence once and the annotation group twice
(its own install), never again for the review group (I3)."""
self.require_module()
import database.annotation_schema as annotation_module
import database.review_schema as review_module
model = self.model()
with model.get_connection() as conn:
insert_group_rows(conn)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
with (
patch.object(
review_module,
"_evidence_error",
wraps=review_module._evidence_error,
) as evidence,
patch.object(
review_module,
"annotation_schema_error",
wraps=review_module.annotation_schema_error,
) as nested,
patch.object(
annotation_module,
"_value_error",
wraps=annotation_module._value_error,
) as annotation_rows,
):
model = self.model()
model.get_connection()
self.assertEqual(evidence.call_count, 1)
self.assertEqual(nested.call_count, 0)
self.assertEqual(annotation_rows.call_count, 2)
model.close_all()
def test_old_database_without_group_migrates_without_touching_legacy_rows(self):
self.require_module()
for label, dropped in (
("annotations only", TABLES),
("predates annotations", ANNOTATION_TABLES + TABLES),
):
with self.subTest(label):
path = str(Path(self.temp.name) / f"{label.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if dropped == TABLES:
insert_byte(conn)
model.close_all()
self.drop_tables(path, dropped)
with closing(sqlite3.connect(path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(
review_schema_error(raw, allow_missing=False),
"Missing review queue schema",
)
legacy_rows_before = table_rows(raw, LEGACY_TABLES)
self.assertTrue(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[
0
]
)
PromptModel.reset_schema_cache()
model = self.model(path)
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
counts = [
conn.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0]
for t in (ITEMS, DECISIONS)
]
legacy_rows_after = {
table: value
for table, value in table_rows(conn, LEGACY_TABLES).items()
if table in legacy_rows_before
}
self.assertEqual(counts, [0, 0])
self.assertEqual(legacy_rows_after, legacy_rows_before)
self.assertEqual(
PromptModel.verify_database_file(path, require_library_state=True),
(True, "ok"),
)
model.close_all()
def test_annotations_only_installation_is_valid(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
model.close_all()
self.drop_tables(self.path)
with closing(sqlite3.connect(self.path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
def test_review_group_without_annotation_group_is_invalid(self):
self.require_module()
model = self.model()
model.close_all()
self.drop_tables(self.path, ANNOTATION_TABLES)
with closing(sqlite3.connect(self.path)) as raw:
for allow_missing in (True, False):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("image annotation", error.lower())
self.assertFalse(PromptModel.verify_database_file(self.path)[0])
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model()
# An invalid annotation group invalidates the review group beside it.
path = self.tampered(
"bad annotations",
("UPDATE media_annotation_bytes SET sha256=upper(sha256)",),
checks_off=True,
)
with closing(sqlite3.connect(path)) as raw:
self.assertIn("image annotation", review_schema_error(raw).lower())
def tampered(self, name, statements, *, checks_off=False, rows_in=True):
"""A closed current database whose group was altered after the fact."""
path = str(Path(self.temp.name) / f"{name.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if rows_in:
insert_group_rows(conn)
model.close_all()
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
if checks_off:
raw.execute("PRAGMA ignore_check_constraints=ON")
for statement in statements:
if isinstance(statement, tuple):
raw.execute(*statement)
else:
raw.execute(statement)
return path
def assert_rejected_unrepaired(self, path):
with closing(sqlite3.connect(path)) as raw:
schema_before = raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall()
rows_before = table_rows(raw, TABLES)
for allow_missing in (False, True):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("review", error.lower())
self.assertFalse(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[0]
)
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model(path)
PromptModel.reset_schema_cache()
with closing(sqlite3.connect(path)) as raw:
self.assertEqual(
raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall(),
schema_before,
)
self.assertEqual(table_rows(raw, TABLES), rows_before)
def test_partial_or_malformed_group_is_rejected_without_repair(self):
self.require_module()
cases = {
"missing metadata": (f"DROP TABLE {METADATA}",),
"missing items": (f"DROP TABLE {DECISIONS}", f"DROP TABLE {ITEMS}"),
"missing decisions": (f"DROP TABLE {DECISIONS}",),
"reserved view": (
f"DROP TABLE {DECISIONS}",
f"CREATE VIEW {DECISIONS} AS SELECT 'x' AS id",
),
"extra trigger": (
f"CREATE TRIGGER media_review_trigger AFTER INSERT ON {ITEMS} "
"BEGIN SELECT 1; END",
),
"extra index": (
f"CREATE INDEX idx_media_review_extra ON {ITEMS}(updated_at)",
),
"extra column": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("evidence TEXT", "note TEXT, evidence TEXT"),
),
"wrong primary key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("id TEXT PRIMARY KEY", "id TEXT", 1),
),
"wrong foreign key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(
"generated_images(id) ON DELETE SET NULL",
"generated_images(id) ON DELETE CASCADE",
),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"wrong check": (
f"DROP TABLE {METADATA}",
DDL[METADATA].replace("schema_version=1", "schema_version=2"),
f"INSERT INTO {METADATA} VALUES(1,2,0)",
),
"missing unique": (
"DROP INDEX idx_media_review_decisions_pair",
"DROP INDEX idx_media_review_decisions_image",
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(" UNIQUE(item_id, decision_revision),", ""),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"missing queue index": ("DROP INDEX idx_media_review_items_queue",),
"wrong index columns": (
"DROP INDEX idx_media_review_items_queue",
INDEXES["idx_media_review_items_queue"].replace(", id", ""),
),
"orphan item": ((f"UPDATE {ITEMS} SET byte_id=?", (OTHER_ID,)),),
"orphan decision": ((f"UPDATE {DECISIONS} SET item_id=?", (OTHER_ID,)),),
"dangling pair": (f"UPDATE {DECISIONS} SET pair_id=404",),
"bad item uuid": (
(f"UPDATE {ITEMS} SET id=?", ("x" * 36,)),
(f"UPDATE {DECISIONS} SET item_id=?", ("x" * 36,)),
),
"uppercase decision uuid": (f"UPDATE {DECISIONS} SET id=upper(id)",),
"null decision id": (f"UPDATE {DECISIONS} SET id=NULL",),
"blob ids": (f"UPDATE {DECISIONS} SET id=CAST(id AS BLOB)",),
"no metadata row": (f"DELETE FROM {METADATA}",),
"long created_at": ((f"UPDATE {ITEMS} SET created_at=?", ("x" * 65,)),),
"blob updated_at": (f"UPDATE {ITEMS} SET updated_at=X'00'",),
"text resolver version": (f"UPDATE {ITEMS} SET resolver_version='one'",),
"blob spelled path": (
f"UPDATE {ITEMS} SET spelled_path=CAST(spelled_path AS BLOB)",
),
"evidence hash mismatch": (
(f"UPDATE {ITEMS} SET evidence_hash=?", (CANDIDATE,)),
),
}
for label, statements in cases.items():
with self.subTest(label):
self.assert_rejected_unrepaired(self.tampered(label, statements))
unchecked = {
"two metadata rows": (f"INSERT INTO {METADATA} VALUES(2,1,0)",),
"wrong schema version": (f"UPDATE {METADATA} SET schema_version=2",),
"negative index revision": (
f"UPDATE {METADATA} SET review_index_revision=-1",
),
"uppercase evidence hash": (
f"UPDATE {ITEMS} SET evidence_hash=upper(evidence_hash)",
),
"bad state": (f"UPDATE {ITEMS} SET state='done'",),
"limited two": (f"UPDATE {ITEMS} SET limited=2",),
"empty spelled path": (f"UPDATE {ITEMS} SET spelled_path=''",),
"negative revision": (f"UPDATE {ITEMS} SET revision=-1",),
"negative candidates": (f"UPDATE {ITEMS} SET candidate_count=-1",),
"bad outcome": (f"UPDATE {DECISIONS} SET outcome='maybe'",),
"short candidate hash": (
f"UPDATE {DECISIONS} SET candidate_hash=substr(candidate_hash,1,63)",
),
"negative decision revision": (
f"UPDATE {DECISIONS} SET decision_revision=-1",
),
}
for label, statements in unchecked.items():
with self.subTest(label):
path = self.tampered(label, statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_selected_decision_requires_candidate_hash(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
insert_item(conn)
for outcome, candidate in (("selected", None), ("unresolved", CANDIDATE)):
with self.subTest(outcome):
with self.assertRaises(sqlite3.IntegrityError):
insert_decision(conn, outcome=outcome, candidate_hash=candidate)
insert_decision(conn, outcome="unresolved", candidate_hash=None)
insert_decision(conn, decision_id=OTHER_ID, decision_revision=2)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(
rows(conn, f"SELECT outcome FROM {DECISIONS} ORDER BY 1"),
[("selected",), ("unresolved",)],
)
with self.assertRaises(sqlite3.IntegrityError): # one per revision
insert_decision(conn, decision_id=ITEM_ID, decision_revision=2)
model.close_all()
path = self.tampered(
"selected without hash",
(f"UPDATE {DECISIONS} SET candidate_hash=NULL",),
checks_off=True,
)
self.assert_rejected_unrepaired(path)
def test_oversize_or_noncanonical_evidence_rejected(self):
self.require_module()
self.assertEqual(
canonical_evidence({"b": [1, "é"], "a": None}),
'{"a":null,"b":[1,"\\u00e9"]}',
)
self.assertEqual(
evidence_digest(EVIDENCE), hashlib.sha256(EVIDENCE.encode()).hexdigest()
)
# The store's non-finite policy (utils.json_strict): written as null.
self.assertEqual(
canonical_evidence({"x": float("nan"), "y": [float("-inf")]}),
'{"x":null,"y":[null]}',
)
for bad in (
[],
"text",
{"x": object()},
{"x": object(), "y": float("nan")},
{1: 1, "a": 2},
):
with self.subTest(value=repr(bad)), self.assertRaises(ValueError):
canonical_evidence(bad)
with self.assertRaises(ValueError):
canonical_evidence({"pad": "x" * MAX_EVIDENCE_BYTES})
oversize = '{"pad":"' + "x" * (MAX_EVIDENCE_BYTES - 9) + '"}'
self.assertEqual(len(oversize), MAX_EVIDENCE_BYTES + 1)
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, evidence=oversize)
model.close_all()
noncanonical = {
"spaces": '{"candidates": [], "sha256": "x"}',
"unsorted": '{"sha256":"x","candidates":[]}',
"raw unicode": '{"text":"é"}',
"duplicate key": '{"a":1,"a":2}',
"not an object": "[]",
"not json": "{",
"nan": '{"a":NaN}',
"empty": "",
}
for label, evidence in noncanonical.items():
for table in (ITEMS, DECISIONS):
with self.subTest(label, table=table):
digest = hashlib.sha256(evidence.encode("utf-8")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (evidence,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
if table == DECISIONS:
statements.pop()
self.assert_rejected_unrepaired(
self.tampered(f"{label} {table}", statements)
)
for table in (ITEMS, DECISIONS):
with self.subTest("oversize", table=table):
digest = hashlib.sha256(oversize.encode("ascii")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (oversize,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
path = self.tampered(f"oversize {table}", statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_seventeen_candidates_rejected(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, candidates=17)
insert_item(conn, candidates=16)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
model.close_all()
path = self.tampered(
"seventeen", (f"UPDATE {ITEMS} SET candidate_count=17",), checks_off=True
)
self.assert_rejected_unrepaired(path)
def test_prompt_or_image_deletion_nulls_navigation_and_keeps_decision(self):
self.require_module()
db = PromptDatabase(self.path)
self.models.append(db.model)
with db.model.get_connection() as conn:
populate_legacy(conn)
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(8,9,NULL)"
)
insert_byte(conn)
insert_item(conn)
insert_decision(conn, pair_id=8, image_id=4)
conn.commit()
self.assertTrue(db.delete_prompt_pair(8))
with db.model.get_connection() as conn:
self.assertEqual(
rows(conn, f"SELECT pair_id, image_id FROM {DECISIONS}"), [(None, 4)]
)
with db.model.get_connection() as conn:
conn.execute("DELETE FROM prompt_pairs WHERE id=6")
conn.commit()
self.assertTrue(db.delete_prompt(7))
with db.model.get_connection() as conn:
self.assertEqual(
rows(
conn,
f"SELECT id, item_id, outcome, candidate_hash, pair_id, image_id "
f"FROM {DECISIONS}",
),
[(DECISION_ID, ITEM_ID, "selected", CANDIDATE, None, None)],
)
self.assertEqual(rows(conn, f"SELECT id FROM {ITEMS}"), [(ITEM_ID,)])
self.assertIsNone(review_schema_error(conn, allow_missing=False))
# The SET NULL actions look decisions up by index, not by a scan.
for column in ("pair_id", "image_id"):
plan = " ".join(
str(row[3])
for row in conn.execute(
f"EXPLAIN QUERY PLAN UPDATE {DECISIONS} " # nosec B608
f"SET {column}=NULL WHERE {column}=?",
(1,),
)
)
self.assertIn(f"idx_media_review_decisions_{column[:-3]}", plan)
def test_full_capacity_unicode_control_pair_round_trips_within_8_mib(self):
self.require_module()
value = full_capacity_evidence()
evidence = canonical_evidence(value)
self.assertEqual(len(evidence.encode("utf-8")), MAX_EVIDENCE_BYTES)
self.assertTrue(evidence.isascii())
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_byte(conn)
insert_item(conn, evidence=evidence, candidates=16)
insert_decision(conn, evidence=evidence)
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
for restore_from in (source_path, backup_file):
with self.subTest(Path(restore_from).name):
live.restore_from_file(restore_from)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
stored = conn.execute(
f"SELECT i.evidence, d.evidence FROM {ITEMS} i "
f"JOIN {DECISIONS} d ON d.item_id = i.id"
).fetchone()
self.assertEqual(json.loads(stored[0]), value)
self.assertEqual(json.loads(stored[1]), value)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_install_failure_rolls_back_and_retries(self):
self.require_module()
with patch(
"database.review_schema.review_schema_error",
side_effect=[None, "injected final validation failure"],
):
with self.assertRaisesRegex(sqlite3.DatabaseError, "injected"):
self.model()
with closing(sqlite3.connect(self.path)) as raw:
self.assertEqual(catalog(raw), set())
self.assertIsNotNone(
raw.execute(
"SELECT 1 FROM sqlite_master WHERE name='media_annotation_bytes'"
).fetchone()
)
PromptModel.reset_schema_cache()
model = self.model()
self.assertEqual(
PromptModel.verify_database_file(model.db_path, require_library_state=True),
(True, "ok"),
)
with model.get_connection() as conn:
self.assertEqual(metadata(conn), [(1, 0)])
def test_wal_backup_restore_round_trips_group(self):
self.require_module()
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_group_rows(conn)
conn.execute(f"UPDATE {METADATA} SET review_index_revision=3")
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
source_before = Path(source_path).read_bytes()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
with closing(sqlite3.connect(backup_file)) as raw:
self.assertEqual(table_rows(raw, TABLES), source_rows)
self.assertIsNone(review_schema_error(raw, allow_missing=False))
safety = live.restore_from_file(source_path)
self.assertEqual(Path(source_path).read_bytes(), source_before)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertEqual(metadata(conn), [(1, 3)])
with closing(sqlite3.connect(safety)) as raw:
self.assertEqual(rows(raw, f"SELECT id FROM {ITEMS}"), [(OTHER_ID,)])
PromptModel.reset_schema_cache()
live.restore_from_file(backup_file)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_staged_failure_preserves_source_live_safety_and_handles(self):
self.require_module()
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
live_rows = table_rows(conn, TABLES)
bad = self.tampered(
"restore bad", (f"UPDATE {ITEMS} SET state='done'",), checks_off=True
)
old = str(Path(self.temp.name) / "old.db")
self.model(old).close_all()
self.drop_tables(old)
for source, patched in ((bad, False), (old, True)):
with self.subTest(source=Path(source).name):
before = Path(source).read_bytes()
safety = sorted(Path(self.temp.name).glob("live.db.backup_*"))
if patched:
context = patch(
"database.models.migrate_review_state",
side_effect=sqlite3.OperationalError("staging failed"),
)
expected = sqlite3.OperationalError
else:
context = patch.object(PromptModel, "backup_database")
expected = ValueError
with context as hook, self.assertRaises(expected):
live.restore_from_file(source)
if not patched:
hook.assert_not_called()
self.assertEqual(Path(source).read_bytes(), before)
self.assertEqual(
sorted(Path(self.temp.name).glob("live.db.backup_*")), safety
)
assert_closed(self, source)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), live_rows)
self.assertFalse(
PromptModel.verify_database_file(old, require_library_state=True)[0]
)
self.assertTrue(PromptModel.verify_database_file(old)[0])
def test_verify_requires_group_only_with_require_library_state(self):
self.require_module()
self.model().close_all()
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
self.drop_tables(self.path)
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
if __name__ == "__main__":
unittest.main()
+237 -5
View File
@@ -38,6 +38,11 @@ def _graph(text):
}
def _scope_part(scope):
"""The per-source scope keys this module pins (other features add counters)."""
return {key: scope[key] for key in ("files", "roots", "output_folders")}
class ScanScopeTestCase(AdminAPITestCase):
"""Helpers: link files the way the scan and the monitor do, spy on reads."""
@@ -260,7 +265,7 @@ class TestScanScope(ScanScopeTestCase):
async def test_empty_read_body_asks_for_the_default_scope(self):
"""A chunked POST with no bytes can read a body: still the default."""
from py.api.job_routes import _scan_scope
from py.api.job_routes import _scan_request
class Request:
can_read_body = True
@@ -271,10 +276,21 @@ class TestScanScope(ScanScopeTestCase):
async def text(self):
return self._text
self.assertEqual(await _scan_scope(Request("")), "new")
self.assertEqual(await _scan_scope(Request(" \n")), "new")
self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all")
self.assertIsNone(await _scan_scope(Request("{")))
default = ({"files": "new", "roots": None}, None)
self.assertEqual(await _scan_request(Request("")), default)
self.assertEqual(await _scan_request(Request(" \n")), default)
self.assertEqual(
await _scan_request(
Request(
'{"scope": "all", "roots": [2, 0, 2],'
' "paths": ["output/c", "output/a", "output/c"]}'
)
),
({"files": "all", "roots": {0: "output/a", 2: "output/c"}}, None),
)
request, error = await _scan_request(Request("{"))
self.assertIsNone(request)
self.assertIsInstance(error, str)
async def test_invalid_scope_is_refused(self):
for body in ({"scope": "everything"}, {"scope": 1}, ["new"]):
@@ -292,6 +308,222 @@ class TestScanScope(ScanScopeTestCase):
self.assertIsNone(self.api._scan_job)
class TestPerSourceScope(ScanScopeTestCase):
"""``roots``: scan only the selected effective sources (#249 S4 Task 10)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
_write_png(self.a / "in_a.png", prompt_text="a prompt in a")
_write_png(self.b / "in_b.png", prompt_text="a prompt in b")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
async def _refused(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 400, data)
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_scoped_scan_only_walks_selected_root(self):
walked = []
original = self.api._collect_output_media_sync
def spy(output_dirs):
walked.append([os.path.basename(str(d)) for d in output_dirs])
return original(output_dirs)
self.api._collect_output_media_sync = spy
reads = self._spy_reads()
job = await self._scan({"scope": "all", "roots": [1], "paths": ["output/b"]})
self.assertEqual(walked, [["b"]])
self.assertEqual(reads, ["in_b.png"])
self.assertEqual(
_scope_part(job["scope"]),
{"files": "all", "roots": [1], "output_folders": 1},
)
self.assertEqual(
job["outcomes"]["added"],
[
{
"path": "in_b.png",
"root": "output/b",
"root_index": 1,
"detail": job["outcomes"]["added"][0]["detail"],
}
],
)
self.assertEqual(job["counters"]["added"], 1)
self.assertEqual(self.api._scan_job.last_event["directories"], ["output/b"])
async def test_duplicate_roots_are_one_selection(self):
job = await self._scan(
{"roots": [1, 0, 1], "paths": ["output/b", "output/a", "output/b"]}
)
self.assertEqual(job["scope"]["roots"], [0, 1])
self.assertEqual(job["scope"]["output_folders"], 2)
self.assertEqual(job["counters"]["added"], 2)
async def test_scope_line_counts_selected_folders(self):
job = await self._scan({"roots": [0], "paths": ["output/a"]})
self.assertTrue(
job["log"][0].endswith(
"Scan started · 1 output folder · new and unresolved files"
),
job["log"][0],
)
self.assertEqual(job["scope"]["output_folders"], 1)
unscoped = await self._scan({"scope": "all"})
self.assertEqual(
_scope_part(unscoped["scope"]),
{"files": "all", "roots": None, "output_folders": 2},
)
self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0])
async def _stale(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 409, data)
self.assertEqual(data["code"], "sources-changed")
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_index_past_the_sources_is_409_without_job(self):
await self._stale({"roots": [2], "paths": ["output/c"]})
await self._stale({"roots": [0, 64], "paths": ["output/a", "output/x"]})
async def test_stale_index_never_scans_another_source(self):
"""The list changed after the page read it: source 0 was removed, so
old index 1 (b) is past the end and old index 0 now names b."""
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.b)]
await self._stale({"roots": [1], "paths": ["output/b"]})
await self._stale({"roots": [0], "paths": ["output/a"]})
job = await self._scan({"roots": [0], "paths": ["output/b"]})
self.assertEqual(job["outcomes"]["added"][0]["root"], "output/b")
async def test_roots_need_their_paths(self):
for body in (
{"roots": [0]},
{"roots": [0], "paths": None},
{"roots": [0], "paths": "output/a"},
{"roots": [0], "paths": []},
{"roots": [0, 1], "paths": ["output/a"]},
{"roots": [0], "paths": [0]},
{"roots": [0, 0], "paths": ["output/a", "output/b"]},
):
with self.subTest(body=body):
await self._refused(body)
async def test_bool_and_float_roots_rejected(self):
for roots in (
[True],
[False],
[0.0],
[1.5],
["0"],
[None],
[-1],
[0, None],
[],
None,
0,
"0",
{"0": 0},
list(range(65)),
):
with self.subTest(roots=roots):
paths = ["output/a"] * len(roots) if isinstance(roots, list) else []
await self._refused({"roots": roots, "paths": paths})
async def test_legacy_scan_unscoped_unchanged(self):
from py.api.admin import SCAN_COUNT_KEYS
resp = await self.client.request("POST", "/prompt_manager/scan", data="{}")
payloads = _sse_payloads(await resp.text())
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a", "output/b"])
self.assertEqual(final["added"], 2)
self.assertEqual(
_scope_part(self.api._scan_job.scope),
{"files": "all", "roots": None, "output_folders": 2},
)
async def test_legacy_scan_attaches_to_a_scoped_job_as_is(self):
import threading
from py.api.admin import SCAN_COUNT_KEYS
gate = threading.Event()
original = self.api._collect_output_media_sync
def gated(output_dirs):
gate.wait(5)
return original(output_dirs)
self.api._collect_output_media_sync = gated
# The legacy route has decided to attach once it streams the job.
attached = asyncio.Event()
stream = self.api._stream_scan_job
async def attaching(request, job):
attached.set()
return await stream(request, job)
self.api._stream_scan_job = attaching
resp = await self.client.request(
"POST",
"/prompt_manager/jobs/scan",
json={"roots": [0], "paths": ["output/a"]},
)
self.assertEqual(resp.status, 202)
started = (await resp.json())["job"]
legacy = asyncio.ensure_future(
self.client.request("POST", "/prompt_manager/scan", data="{}")
)
try:
await asyncio.wait_for(attached.wait(), 10)
finally:
gate.set()
payloads = _sse_payloads(await (await legacy).text())
await asyncio.wait_for(self.api._scan_job.task, 10)
self.assertEqual(self.api._scan_job.id, started["id"])
self.assertEqual(len(self.api._jobs.list()), 1)
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a"])
self.assertEqual(final["added"], 1)
async def test_scoped_scan_uses_the_sources_it_was_started_with(self):
"""Sources saved after the start never change what the job walks."""
resolved = []
original = self.api._get_all_output_dirs
def counting():
resolved.append(1)
return original()
self.api._get_all_output_dirs = counting
job = await self._scan({"roots": [1], "paths": ["output/b"]})
self.assertEqual(len(resolved), 1) # the request's own resolution only
self.assertEqual(job["outcomes"]["added"][0]["root_index"], 1)
class TestKnownResolvedPaths(ScanScopeTestCase):
"""``PromptDatabase.get_known_resolved_paths``: one streamed query."""
+959
View File
@@ -0,0 +1,959 @@
"""System › Sources & scans backend (#249 S4 Task 9).
``GET /prompt_manager/settings`` reports the configured, effective and
watched sources and the monitoring setting; ``POST`` saves
``monitoring_enabled`` (a restart applies it) and refuses a source change
while a record-changing job runs; ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing. No response carries an absolute path.
"""
import asyncio
import functools
import json
import ntpath
import os
import sys
import types
import unittest
from contextlib import contextmanager
from unittest.mock import patch
try:
from tests.test_api_admin import AdminAPITestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase
from py import config as config_module # noqa: E402
from py.api import settings_routes, sources # noqa: E402
from py.config import GalleryConfig # noqa: E402
VALIDATE = "/prompt_manager/settings/validate-source"
SETTINGS = "/prompt_manager/settings"
@contextmanager
def no_filesystem_access_for(marker, test):
"""Fail ``test`` if anything resolves, stats or opens a path holding ``marker``.
Trusted directories (ComfyUI's, the anchors) may still be resolved.
"""
real = {
"realpath": os.path.realpath,
"stat": os.stat,
"lstat": os.lstat,
}
def guard(name):
def call(path, *args, **kwargs):
if isinstance(path, (str, bytes)) and marker in os.fsdecode(path):
test.fail(f"{name}({path!r}) touched an outside path")
return real[name](path, *args, **kwargs)
return call
with (
patch("os.path.realpath", guard("realpath")),
patch("os.stat", guard("stat")),
patch("os.lstat", guard("lstat")),
):
yield
class FakeMonitor:
"""The image monitor as get_settings sees it: status and scheduled roots."""
def __init__(self, scheduled, alive=True):
self.monitored_directories = [str(path) for path in scheduled]
self.alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": self.monitored_directories,
"handler_active": True,
"observer_alive": self.alive,
}
class SourcesTestCase(AdminAPITestCase):
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
enabled = GalleryConfig.MONITORING_ENABLED
self.addCleanup(setattr, GalleryConfig, "MONITORING_ENABLED", enabled)
GalleryConfig.MONITORING_ENABLED = True
self._install_monitor(None)
def _install_monitor(self, monitor):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = monitor
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
async def _get(self):
resp = await self.client.request("GET", "/prompt_manager/settings")
body = await resp.text()
self.assertEqual(resp.status, 200, body)
self.assertNotIn(self.tmpdir, body)
return json.loads(body)
async def _post(self, url, payload):
resp = await self.client.request("POST", url, json=payload)
body = await resp.text()
self.assertNotIn(self.tmpdir, body)
return resp.status, json.loads(body)
def _config_bytes(self):
if not os.path.exists(self.config_path):
return None
with open(self.config_path, "rb") as f:
return f.read()
def _states(self, data):
return {row["path"]: row["watch"] for row in data["sources"]["effective"]}
class TestSourceStatus(SourcesTestCase):
async def test_sources_list_configured_effective_and_auto_detected(self):
missing = self.output_dir / "gone"
missing.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.a),
self._root(missing),
]
missing.rmdir() # an unmounted drive: configured, no longer a directory
data = await self._get()
self.assertEqual(
data["sources"]["configured"],
[
{
"label": "a",
"path": "output/a",
"valid": True,
"code": None,
"reason": None,
},
{
"label": "gone",
"path": "output/gone",
"valid": False,
"code": "not-found",
"reason": "Gallery root does not exist",
},
],
)
self.assertEqual(
data["sources"]["effective"],
[{"label": "a", "path": "output/a", "root_index": 0, "watch": "manual"}],
)
self.assertFalse(data["sources"]["auto_detected"])
# The legacy keys are unchanged.
self.assertEqual(
data["settings"]["gallery_root_paths"], ["output/a", "output/gone"]
)
async def test_empty_list_means_auto_detect(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["x"]}}})
# Detection (the upward search for a ComfyUI tree) finds the output.
self.api._find_comfyui_output_dir = lambda: str(self.output_dir)
status, saved = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": []}
)
self.assertEqual(status, 200, saved)
self.assertEqual(
self._read_config()["gallery"]["monitoring"]["directories"], []
)
data = await self._get()
self.assertEqual(data["sources"]["configured"], [])
self.assertTrue(data["sources"]["auto_detected"])
self.assertEqual(
data["sources"]["effective"],
[
{
"label": "ComfyUI output",
"path": "output",
"root_index": 0,
"watch": "manual",
}
],
)
async def test_watching_requires_live_observer_with_root(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
# No monitor in this process: nothing is watched, a restart cannot help.
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": False, "observer_alive": False, "scheduled": []},
)
self.assertEqual(
self._states(data), {"output/a": "manual", "output/b": "manual"}
)
# A dead observer watches nothing, even the roots it scheduled.
self._install_monitor(FakeMonitor([self.a], alive=False))
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": True, "observer_alive": False, "scheduled": ["output/a"]},
)
self.assertEqual(
self._states(data),
{"output/a": "restart_pending", "output/b": "restart_pending"},
)
# A live observer watches its scheduled root; a root configured since
# it started waits for a restart.
self._install_monitor(FakeMonitor([self.a]))
data = await self._get()
self.assertTrue(data["monitor"]["observer_alive"])
self.assertEqual(
self._states(data), {"output/a": "watching", "output/b": "restart_pending"}
)
# Monitoring switched off: the live observer still watches until the
# restart ("stopping"), and nothing else is pending.
GalleryConfig.MONITORING_ENABLED = False
data = await self._get()
self.assertFalse(data["settings"]["monitoring_enabled"])
self.assertEqual(
self._states(data), {"output/a": "stopping", "output/b": "manual"}
)
async def test_scheduled_root_matches_through_its_spelling(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
spelled = os.path.join(str(self.output_dir), "b", "..", "a")
self._install_monitor(FakeMonitor([spelled]))
data = await self._get()
self.assertEqual(self._states(data), {"output/a": "watching"})
self.assertEqual(data["monitor"]["scheduled"], ["output/a"])
async def test_status_failure_is_500_without_paths(self):
def boom(*args, **kwargs):
raise OSError(2, "No such file", f"{self.tmpdir}/secret")
with patch.object(sources, "describe_sources", boom):
resp = await self.client.request("GET", "/prompt_manager/settings")
self.assertEqual(resp.status, 500)
self.assertNotIn(self.tmpdir, await resp.text())
class TestMonitoringSetting(SourcesTestCase):
async def test_monitoring_enabled_strict_bool_and_restart_flag(self):
self._write_config({"custom": {"keep": True}})
before = self._config_bytes()
for value in ("false", 0, 1, None, [], {"on": True}):
with self.subTest(value=value):
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": value}
)
self.assertEqual(status, 400, data)
self.assertFalse(data["success"])
self.assertEqual(self._config_bytes(), before)
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
self._install_monitor(FakeMonitor([self.output_dir]))
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": False}
)
self.assertEqual(status, 200, data)
self.assertTrue(data["persisted"])
self.assertTrue(data["restart_required"])
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
saved = self._read_config()
self.assertIs(saved["gallery"]["monitoring"]["enabled"], False)
self.assertEqual(saved["custom"], {"keep": True})
self.assertFalse(GalleryConfig.MONITORING_ENABLED)
self.assertFalse((await self._get())["settings"]["monitoring_enabled"])
# Back on while the observer still runs: nothing waits for a restart.
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertIs(self._read_config()["gallery"]["monitoring"]["enabled"], True)
async def test_enabling_without_a_live_observer_needs_a_restart(self):
GalleryConfig.MONITORING_ENABLED = False
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
def test_config_file_enabled_must_be_a_bool(self):
GalleryConfig.MONITORING_ENABLED = True
with self.assertLogs("prompt_manager.config", level="WARNING"):
GalleryConfig.update_config({"monitoring": {"enabled": "false"}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, True)
GalleryConfig.update_config({"monitoring": {"enabled": False}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, False)
class TestSourceChangeHold(SourcesTestCase):
async def test_root_save_refused_while_scan_job_runs(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["kept"]}}})
before = self._config_bytes()
gate = asyncio.Event()
async def runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan", "Scan for metadata", record_changing=True, runner=runner
)
try:
status, data = await self._post(
"/prompt_manager/settings",
{"gallery_root_paths": [str(self.b)]},
)
self.assertEqual(status, 409, data)
self.assertEqual(data["code"], "busy")
self.assertEqual(
data["running"], {"id": scan.id, "title": "Scan for metadata"}
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.a)])
# A save without sources, or one resending the current sources
# (the legacy page always does), is not a source change.
status, data = await self._post(
"/prompt_manager/settings", {"result_timeout": 9}
)
self.assertEqual(status, 200, data)
status, data = await self._post(
"/prompt_manager/settings",
{"result_timeout": 8, "gallery_root_paths": [str(self.a)]},
)
self.assertEqual(status, 200, data)
finally:
gate.set()
await asyncio.wait_for(scan.task, 5)
async def test_root_save_runs_as_a_source_change_hold(self):
seen = []
original = settings_routes.get_config_store
def store():
seen.extend(
(job.kind, job.title, job.admission, job.cancellable)
for job in self.api._jobs.list()
if job.running
)
return original()
with patch.object(settings_routes, "get_config_store", store):
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual(status, 200, data)
self.assertIn(("source_change", "Source change", "records", False), seen)
self.assertFalse(any(job.running for job in self.api._jobs.list()))
async def test_failed_root_save_ends_the_hold_failed(self):
os.makedirs(self.config_path) # config.json is a directory: unreadable
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertGreaterEqual(status, 400, data)
held = [j for j in self.api._jobs.list() if j.kind == "source_change"]
self.assertEqual([job.state for job in held], ["failed"])
async def test_mixed_case_windows_paths_dedupe(self):
"""Two spellings of one Windows folder are saved once (normcase)."""
with (
patch.object(sources, "source_problem", return_value=(None, "")),
patch.object(
GalleryConfig, "resolve_gallery_root", side_effect=lambda path: path
),
patch.object(settings_routes, "_source_identity", ntpath.normcase),
patch.object(self.api, "_apply_gallery_roots", return_value=True),
):
roots, error = self.api._parse_gallery_roots(
{
"gallery_root_paths": [
"C:\\ComfyUI\\Output\\Renders",
"c:/comfyui/output/renders",
"C:\\ComfyUI\\output\\other",
]
}
)
self.assertIsNone(error)
self.assertEqual(
roots, ["C:\\ComfyUI\\Output\\Renders", "C:\\ComfyUI\\output\\other"]
)
async def test_case_variant_of_one_folder_is_saved_once_natively(self):
variant = str(self.a)[:-1] + "A"
if not os.path.isdir(variant):
self.skipTest("case-sensitive filesystem")
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.a), variant]}
)
self.assertEqual(status, 200, data)
self.assertEqual(len(GalleryConfig.MONITORING_DIRECTORIES), 1)
class TestValidateSource(SourcesTestCase):
async def test_validate_source_writes_nothing(self):
self._write_config({"gallery": {"monitoring": {"directories": []}}})
before = self._config_bytes()
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 200, data)
self.assertEqual(
data,
{
"success": True,
"valid": True,
"label": "a",
"path": "output/a",
"code": None,
"reason": None,
},
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, roots)
self.assertEqual(self.api._jobs.list(), [])
async def test_relative_path_resolves_against_comfyui(self):
status, data = await self._post(VALIDATE, {"path": "output/b"})
self.assertEqual((status, data["valid"], data["path"]), (200, True, "output/b"))
status, data = await self._post(VALIDATE, {"path": "output"})
self.assertEqual(data["label"], "ComfyUI output")
async def test_invalid_sources_answer_a_code(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
cases = {
"": "empty",
" ": "empty",
str(self.output_dir / "missing"): "not-found",
str(a_file): "not-directory",
str(self.outside_dir): "outside-allowed",
os.path.abspath(os.sep): "outside-allowed",
os.path.expanduser("~"): "outside-allowed",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
"output/../output/a": "traversal",
str(self.a) + "\x00": "invalid-form",
"\\\\?\\C:\\ComfyUI\\output": "invalid-form",
"x" * 5000: "invalid-form",
}
if os.name != "nt":
cases["C:\\ComfyUI\\output"] = "invalid-form"
for path, code in cases.items():
with self.subTest(code=code, path=path[:40]):
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(status, 200, data)
self.assertFalse(data["valid"])
self.assertEqual(data["code"], code)
self.assertIsInstance(data["reason"], str)
self.assertIsNone(data["label"])
self.assertIsNone(data["path"])
self.assertNotIn(os.path.expanduser("~"), json.dumps(data))
async def test_outside_paths_answer_alike_whether_they_exist(self):
"""No existence oracle: outside the allowed folders every path looks
the same, existing or not, file or folder."""
a_file = self.outside_dir / "file.txt"
a_file.write_bytes(b"x")
answers = []
for path in (
self.outside_dir,
self.outside_dir / "missing",
a_file,
os.path.join(self.tmpdir, "no", "such", "tree"),
os.path.abspath(os.sep),
os.path.expanduser("~"),
):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append((status, data))
self.assertEqual(answers[0][1]["code"], "outside-allowed")
self.assertEqual([answer for answer in answers], [answers[0]] * len(answers))
async def test_settings_save_refuses_outside_paths_alike(self):
"""The legacy save answers every outside path alike too: the
filesystem root, the home directory and a link to it included."""
(self.outside_dir / "x").mkdir()
link = os.path.join(self.tmpdir, "link-home")
try:
os.symlink(os.path.expanduser("~"), link, target_is_directory=True)
except (OSError, NotImplementedError):
link = None
paths = [
str(self.outside_dir / "x"),
self.tmpdir + "/nothing/x",
os.path.abspath(os.sep),
os.path.expanduser("~"),
]
if link:
paths.append(link)
answers = []
for path in paths:
resp = await self.client.request(
"POST", SETTINGS, json={"gallery_root_paths": [path]}
)
body = await resp.text()
self.assertNotIn(os.path.expanduser("~"), body)
data = json.loads(body)
# Only the name the caller sent may differ.
answers.append(
(
resp.status,
{k: v for k, v in data.items() if k not in ("source", "error")},
)
)
self.assertEqual(
data["error"],
f"{data['source']} can't be used: "
f"{config_module.OUTSIDE_ALLOWED_REASON}. Remove it, then save.",
)
self.assertEqual(
answers[0],
(
400,
{
"success": False,
"code": "invalid-source",
"index": 0,
"reason": config_module.OUTSIDE_ALLOWED_REASON,
},
),
)
self.assertEqual(answers, [answers[0]] * len(answers))
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
def test_every_caller_folds_root_and_home_alike(self):
for path in (os.path.abspath(os.sep), os.path.expanduser("~")):
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
GalleryConfig.validate_gallery_root(path),
(False, config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
async def test_outside_path_is_never_resolved_or_stat(self):
"""Containment is decided as text first: no realpath, stat or lstat of
an outside path, existing or not, by any caller (I2)."""
probe = self.outside_dir / "probe-x"
probe.mkdir()
(probe / "deep").mkdir()
cases = [str(probe), str(probe / "deep"), str(probe / "missing")]
with no_filesystem_access_for("probe-x", self):
for path in cases:
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path)[0], "outside-allowed"
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": [path]}
)
self.assertEqual(error["code"], "invalid-source")
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(data["code"], "outside-allowed")
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
# A traversal that normalises outside is refused as text too.
sneaky = os.path.join(
str(self.output_dir), "..", "..", "outside", "probe-x"
)
self.assertEqual(
GalleryConfig.gallery_root_problem(sneaky)[0], "outside-allowed"
)
async def test_settings_save_applies_the_form_checks(self):
"""POST /settings refuses what validate-source refuses, by its form."""
for path, code in {
str(self.a) + "\x00": "invalid-form",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
}.items():
with self.subTest(code=code):
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
self.assertEqual(data["reason"], sources.FORM_REASONS[code])
windows = functools.partial(sources.source_form_error, windows=True)
with (
patch.object(sources, "source_form_error", windows),
# The probes hold a token no trusted directory does (on Windows the
# temporary ComfyUI tree itself sits on C:).
no_filesystem_access_for("PMProbe", self),
):
for path in (
"\\\\?\\C:\\PMProbe\\output",
"\\\\.\\pipe\\PMProbe",
"C:\\PMProbe\\output\\CON",
"C:\\PMProbe\\output\\a:stream",
"C:PMProbe",
):
with self.subTest(path=path):
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": ["output/a", path]}
)
self.assertEqual(error["code"], "invalid-source")
self.assertEqual(error["index"], 1)
self.assertEqual(
error["reason"], sources.FORM_REASONS["invalid-form"]
)
async def test_inside_paths_say_why_they_are_not_usable(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
status, missing = await self._post(
VALIDATE, {"path": str(self.output_dir / "missing")}
)
status, not_dir = await self._post(VALIDATE, {"path": str(a_file)})
self.assertEqual(
(missing["code"], not_dir["code"]), ("not-found", "not-directory")
)
async def test_malformed_request_is_400(self):
for body in ({}, {"path": 42}, {"path": None}, ["x"]):
with self.subTest(body=body):
status, data = await self._post(VALIDATE, body)
self.assertEqual(status, 400, data)
self.assertEqual(data["code"], "invalid-request")
self.assertFalse(data["success"])
class TestSourceEdges(SourcesTestCase):
async def test_validation_failure_is_500_without_paths(self):
def boom(path):
raise OSError(13, "Permission denied", f"{self.tmpdir}/secret")
with patch.object(sources, "validate_source", boom):
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 500)
self.assertEqual(data["error"], "Failed to check the source")
async def test_oversized_body_is_413_too_large(self):
resp = await self.client.request(
"POST", VALIDATE, data=json.dumps({"path": "x" * (2 << 20)})
)
self.assertEqual(resp.status, 413)
self.assertEqual((await resp.json())["code"], "too-large")
async def test_unreadable_monitor_counts_as_no_live_observer(self):
class Broken:
def get_status(self):
raise RuntimeError("observer gone")
self._install_monitor(Broken())
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
def test_labels_skip_getters_that_fail_or_are_missing(self):
import sys
import types
def fails():
raise RuntimeError("no input dir")
fake = types.SimpleNamespace(
get_output_directory=lambda: str(self.output_dir),
get_input_directory=fails,
get_temp_directory=lambda: None,
)
with patch.dict(sys.modules, {"folder_paths": fake}):
self.assertEqual(sources.source_label(self.output_dir), "ComfyUI output")
self.assertEqual(sources.source_label(self.a), "a")
with patch.dict(sys.modules, {"folder_paths": None}):
self.assertEqual(sources.source_label(self.output_dir), "output")
class TestOutsideBaseDirectories(SourcesTestCase):
"""ComfyUI folders outside the base directory (``--output-directory``):
their sources round-trip and two folders of one name stay apart (I3)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.renders = self.outside_dir / "renders"
self.input_dir = self.comfy_dir / "input"
(self.renders / "portraits").mkdir(parents=True)
(self.input_dir / "portraits").mkdir(parents=True)
sys.modules["folder_paths"] = types.SimpleNamespace(
base_path=str(self.comfy_dir),
get_output_directory=lambda: str(self.renders),
get_input_directory=lambda: str(self.input_dir),
)
async def test_source_outside_base_round_trips(self):
status, rendered = await self._post(
VALIDATE, {"path": str(self.renders / "portraits")}
)
self.assertEqual(
(rendered["valid"], rendered["path"]), (True, "renders/portraits")
)
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [rendered["path"]]}
)
self.assertEqual(status, 200, data)
self.assertEqual(
GalleryConfig.MONITORING_DIRECTORIES,
[self._root(self.renders / "portraits")],
)
configured = (await self._get())["sources"]["configured"]
self.assertEqual(
[(row["path"], row["valid"]) for row in configured],
[("renders/portraits", True)],
)
# The page resends what it read: still the same folder, no change.
status, data = await self._post(
SETTINGS, {"gallery_root_paths": ["renders/portraits"]}
)
self.assertEqual((status, data["restart_required_fields"]), (200, []))
async def test_same_name_folders_stay_apart(self):
answers = []
for path in (self.renders / "portraits", self.input_dir / "portraits"):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append(data["path"])
self.assertEqual(answers, ["renders/portraits", "input/portraits"])
status, data = await self._post(SETTINGS, {"gallery_root_paths": answers})
self.assertEqual(status, 200, data)
sources_ = (await self._get())["sources"]
self.assertEqual([row["path"] for row in sources_["configured"]], answers)
self.assertEqual([row["path"] for row in sources_["effective"]], answers)
class TestUnusableSourceSave(SourcesTestCase):
async def test_unusable_source_is_named_and_saves_nothing(self):
"""A configured source that went missing blocks the save and is named."""
gone = self.output_dir / "gone"
gone.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(gone)]
self._write_config({"custom": 1})
before = self._config_bytes()
gone.rmdir() # an unplugged drive
status, data = await self._post(
SETTINGS,
{
"gallery_root_paths": ["output/a", "output/gone", "output/b"],
"worker_threads": 2,
},
)
self.assertEqual(status, 400, data)
self.assertEqual(
data,
{
"success": False,
"code": "invalid-source",
"source": "output/gone",
"index": 1,
"reason": "Gallery root does not exist",
"error": "output/gone can't be used: Gallery root does not exist. "
"Remove it or reconnect it, then save.",
},
)
self.assertEqual(self._config_bytes(), before)
async def test_malformed_roots_are_coded_invalid_request(self):
for roots in ("output/a", [1]):
with self.subTest(roots=roots):
status, data = await self._post(SETTINGS, {"gallery_root_paths": roots})
self.assertEqual((status, data["code"]), (400, "invalid-request"))
class TestUnchangedRootsTakeNoHold(SourcesTestCase):
async def test_resent_spelling_of_configured_roots_is_no_change(self):
"""config.json holds a hand-written spelling; the legacy page resends
it resolved: no source change, so no hold and no 409 during a scan (M1)."""
GalleryConfig.MONITORING_DIRECTORIES = ["output/a"]
gate = asyncio.Event()
async def runner(job):
await gate.wait()
job.finish("finished")
self.api._jobs.start(
"scan", "Metadata scan", record_changing=True, runner=runner
)
try:
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.a)], "result_timeout": 30}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertEqual([j.kind for j in self.api._jobs.list()], ["scan"])
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual((status, data["code"]), (409, "busy"))
finally:
gate.set()
class TestWindowsContainment(unittest.TestCase):
"""Windows path forms, with ntpath fakes: anything outside the allowed
folders (a UNC share, the verbatim and device namespaces, another drive)
is refused as text, before any path is resolved or opened (I2)."""
ALLOWED = "c:\\comfyui\\output"
NAS = "\\\\nas\\share\\renders"
def setUp(self):
self.touched = []
def realpath(path, *args, **kwargs):
self.touched.append(path)
return ntpath.normcase(ntpath.normpath(path))
def probe(path):
self.touched.append(path)
return True
names = ("normcase", "normpath", "isabs", "join", "dirname", "basename", "sep")
fake_path = types.SimpleNamespace(
**{name: getattr(ntpath, name) for name in names},
realpath=realpath,
exists=probe,
isdir=probe,
expanduser=lambda path: "c:\\users\\someone",
)
fake_os = types.SimpleNamespace(
path=fake_path, environ={}, pathsep=";", name="nt"
)
for target, value in (
(patch.object(config_module, "os", fake_os), None),
(
patch.object(
GalleryConfig,
"_allowed_parent_pairs",
classmethod(
lambda cls: [(self.ALLOWED, self.ALLOWED), (self.NAS, self.NAS)]
),
),
None,
),
(
patch.object(
GalleryConfig,
"path_anchors",
classmethod(lambda cls: ["c:\\comfyui", "\\\\nas\\share"]),
),
None,
),
):
target.start()
self.addCleanup(target.stop)
def test_outside_forms_are_refused_untouched(self):
for path in (
"\\\\attacker\\share\\x",
"//attacker/share/x",
"\\\\?\\C:\\ComfyUI\\output\\a",
"\\\\?\\UNC\\attacker\\share",
"\\\\.\\pipe\\x",
"\\\\.\\C:\\ComfyUI\\output",
"D:\\Renders",
"C:\\Windows",
"C:\\ComfyUI\\output\\..\\..\\Windows",
"C:\\ComfyUI\\output2",
"\\\\nas\\share\\other",
"..\\..\\Windows",
):
with self.subTest(path=path):
self.touched.clear()
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(self.touched, [])
def test_allowed_spellings_are_accepted(self):
for path in (
"C:\\ComfyUI\\output\\a",
"c:/comfyui/OUTPUT/a",
"output\\a",
"\\\\NAS\\share\\renders\\a", # a UNC spelling of an allowed extra root
):
with self.subTest(path=path):
self.assertEqual(GalleryConfig.gallery_root_problem(path), (None, ""))
class TestSourceFormError(unittest.TestCase):
"""The lexical check that runs before any filesystem access."""
def test_windows_forms(self):
err = sources.source_form_error
for path in (
"C:\\ComfyUI\\output",
"C:/ComfyUI/output",
"\\\\server\\share\\output",
"D:\\images",
):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=True))
for path, code in {
"C:output": "invalid-form", # drive-relative
"C:\\out\\a:stream": "invalid-form", # alternate data stream
"C:\\out\\CON": "invalid-form",
"C:\\out\\nul.txt": "invalid-form",
"C:\\out\\folder.": "invalid-form",
"C:\\out\\folder \\sub": "invalid-form",
"\\\\?\\C:\\out": "invalid-form",
"//?/C:/out": "invalid-form",
"\\\\.\\pipe\\x": "invalid-form",
"\\\\server": "invalid-form",
"C:\\out\\..\\in": "traversal",
"output\\..\\input": "traversal",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=True), code)
def test_posix_forms(self):
err = sources.source_form_error
for path in ("/srv/comfy/output", "output/renders", "out/CON", "a b/c."):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=False))
for path, code in {
"C:\\ComfyUI\\output": "invalid-form",
"C:/ComfyUI/output": "invalid-form",
"\\\\server\\share": "invalid-form",
"out\\renders": "invalid-form",
"/srv/../etc": "traversal",
"../output": "traversal",
"/srv/out\x00": "invalid-form",
"/srv/out\n": "invalid-form",
"": "empty",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=False), code)
if __name__ == "__main__":
unittest.main()
+58 -1
View File
@@ -292,7 +292,7 @@ class TestOverviewRoute(SystemAPITestCase):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertIsNone(body["health"])
self.assertIsNone(body["backup"])
self.assertEqual(body["needs_review"], {"pending": None})
self.assertEqual(body["needs_review"], {"pending": 0})
self.save("Any")
await self.run_health()
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
@@ -416,6 +416,63 @@ class TestStatisticsRoute(SystemAPITestCase):
)
class TestFailedReadsSettleFirst(SystemAPITestCase):
"""A failed read answers only after the route's other reads finished.
Answering on the first failure (``asyncio.gather`` without
``return_exceptions``) left the other executor reads running against a
database the caller may already be closing; on Python 3.10 CI that
crashed the test process inside SQLite.
"""
async def assert_answers_after_reads(self, url, failing, slow, error):
raised, release, finished = (threading.Event() for _ in range(3))
real = getattr(*slow)
def fail(*args, **kwargs):
raised.set()
raise RuntimeError("secret")
def held(*args, **kwargs):
release.wait()
try:
return real(*args, **kwargs)
finally:
finished.set()
with (
patch.object(*failing, side_effect=fail),
patch.object(*slow, side_effect=held),
self.assertLogs("prompt_manager.api", "ERROR"),
):
answer = asyncio.ensure_future(self.json("GET", url, 500))
try:
await asyncio.get_running_loop().run_in_executor(None, raised.wait)
done, _ = await asyncio.wait({answer}, timeout=0.5)
self.assertEqual(done, set(), "answered while a read still ran")
finally:
release.set()
body = await answer
self.assertTrue(finished.is_set())
self.assertEqual(body, {"success": False, "code": None, "error": error})
async def test_statistics(self):
await self.assert_answers_after_reads(
"/prompt_manager/system/statistics",
(self.api.db, "usage_by_category"),
(self.api.db, "get_statistics"),
"Failed to read statistics",
)
async def test_overview(self):
await self.assert_answers_after_reads(
"/prompt_manager/system/overview",
(self.api.db, "library_counts"),
(self.api, "_output_dirs_with_origin"),
"Failed to read the overview",
)
if __name__ == "__main__":
import unittest
+846
View File
@@ -0,0 +1,846 @@
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
``GET /prompt_manager/images/thumbnails/plan`` counts what a run would do and
writes nothing. ``POST /prompt_manager/jobs/thumbnails`` starts the
``thumbnails`` job (admission ``guarded``, cancellable at a file boundary).
The legacy gallery stream ``POST /images/generate-thumbnails/progress`` starts
or attaches to that job and streams its unchanged events from it, so a client
that goes away no longer stops the work.
Ordering is driven by threading/asyncio events, never by timers.
"""
import asyncio
import hashlib
import os
import sys
import threading
import unittest
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
try:
from tests.test_api_images import ThumbnailTestCase, make_png, parse_sse
except ImportError: # discovered with tests/ as the top-level directory
from test_api_images import ThumbnailTestCase, make_png, parse_sse
from PIL import Image # noqa: E402
import py.api.images as images_module # noqa: E402
from py.api.thumbnail_jobs import thumbnail_options # noqa: E402
from py.config import PromptManagerConfig # noqa: E402
PLAN_URL = "/prompt_manager/images/thumbnails/plan"
JOB_URL = "/prompt_manager/jobs/thumbnails"
def _tree_state(root):
"""{relative path: (size, mtime_ns, sha256)} of every file under ``root``."""
state = {}
for folder, _dirs, files in os.walk(root):
for name in files:
path = Path(folder) / name
data = path.read_bytes()
stat = path.stat()
state[path.relative_to(root).as_posix()] = (
stat.st_size,
stat.st_mtime_ns,
hashlib.sha256(data).hexdigest(),
)
return state
class ThumbnailJobCase(ThumbnailTestCase):
"""Helpers: start, gate and wait for thumbnail jobs."""
async def _start(self, **body):
resp = await self.client.post(JOB_URL, json=body)
return resp.status, await resp.json()
async def _finished(self, job_id):
job = self.api._jobs.get(job_id)
await asyncio.wait_for(job.task, 10)
return job
def _gate_generation(self):
"""Block every file in the worker until released; returns (entered, release)."""
entered = threading.Event()
release = threading.Event()
original = self.api._generate_one
def gated(*args, **kwargs):
entered.set()
release.wait(10)
return original(*args, **kwargs)
self.api._generate_one = gated
self.addCleanup(release.set)
return entered, release
async def _wait(self, event):
loop = asyncio.get_running_loop()
self.assertTrue(await loop.run_in_executor(None, event.wait, 10))
@staticmethod
def _watch_subscribers(job):
"""(subscribed, left): threading events for a stream joining or leaving."""
subscribed, left = threading.Event(), threading.Event()
class Watched(set):
def add(self, item):
super().add(item)
subscribed.set()
def discard(self, item):
super().discard(item)
left.set()
job._subscribers = Watched(job._subscribers)
return subscribed, left
def _status_lines(self):
"""Every status line any job is given, in order."""
from py.api.jobs import Job
lines = []
original = Job.update
def update(job, **kwargs):
if kwargs.get("status_line") is not None:
lines.append(kwargs["status_line"])
return original(job, **kwargs)
patcher = patch.object(Job, "update", update)
patcher.start()
self.addCleanup(patcher.stop)
return lines
def _one_worker(self):
original = PromptManagerConfig.WORKER_THREADS
PromptManagerConfig.WORKER_THREADS = 1 # batches of 4
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
def _run_scan(self):
gate = asyncio.Event()
async def runner(job):
await gate.wait()
job = self.api._jobs.start(
"scan", "Scan for metadata", admission="records", scope={}, runner=runner
)
async def release():
gate.set()
await asyncio.wait_for(job.task, 5)
self.addAsyncCleanup(release)
return job
class TestThumbnailPlan(ThumbnailJobCase):
async def test_plan_is_read_only_and_counts_cap(self):
for name in ("a", "b", "c"):
make_png(self.output_dir / f"{name}.png", size=(64, 64))
fresh = make_png(self._thumb_path("a"))
source_time = (self.output_dir / "a.png").stat().st_mtime
os.utime(fresh, (source_time + 10, source_time + 10))
before = _tree_state(self.comfy_dir)
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1):
missing = await self.client.get(
PLAN_URL, params={"mode": "missing", "size": "512", "quality": "high"}
)
rebuild = await self.client.get(
PLAN_URL, params={"mode": "rebuild", "size": "256", "quality": "low"}
)
self.assertEqual(missing.status, 200)
self.assertEqual(
await missing.json(),
{
"success": True,
"targets": 1,
"fresh": 1,
"remaining": 1,
"roots": ["output"],
"mode": "missing",
"size": 512,
"quality": "high",
},
)
data = await rebuild.json()
# Rebuild regenerates fresh previews too: all three are work, one fits the cap.
self.assertEqual((data["targets"], data["fresh"], data["remaining"]), (1, 1, 2))
self.assertEqual(
(data["mode"], data["size"], data["quality"]), ("rebuild", 256, "low")
)
self.assertEqual(_tree_state(self.comfy_dir), before)
self.assertEqual(self.api._jobs.list(), [])
async def test_plan_defaults_to_missing_previews_at_the_concept_size(self):
make_png(self.output_dir / "a.png")
data = await (await self.client.get(PLAN_URL)).json()
self.assertEqual(
(data["mode"], data["size"], data["quality"], data["targets"]),
("missing", 512, "medium", 1),
)
async def test_size_bounds_validated(self):
make_png(self.output_dir / "a.png")
for params in (
{"size": "127"},
{"size": "2049"},
{"size": "512.5"},
{"size": "big"},
{"mode": "all"},
{"quality": "ultra"},
):
with self.subTest(params=params):
resp = await self.client.get(PLAN_URL, params=params)
body = await resp.json()
self.assertEqual(resp.status, 400)
self.assertEqual(
(body["success"], body["code"]), (False, "invalid-request")
)
for body in (
{"size": 127},
{"size": 2049},
{"size": "512"},
{"size": 512.0},
{"size": True},
{"mode": "rebuild-all"},
{"quality": None},
[],
):
with self.subTest(body=body):
resp = await self.client.post(JOB_URL, json=body)
self.assertEqual(resp.status, 400)
self.assertEqual((await resp.json())["code"], "invalid-request")
for edge in (128, 2048):
with self.subTest(edge=edge):
resp = await self.client.get(PLAN_URL, params={"size": str(edge)})
self.assertEqual(resp.status, 200)
self.assertEqual(self.api._jobs.list(), [])
self.assertFalse(self._thumb_path("a").exists())
class TestThumbnailJob(ThumbnailJobCase):
async def test_job_generates_missing_previews_at_the_chosen_edge(self):
make_png(self.output_dir / "a.png", size=(800, 400))
make_png(self.output_dir / "sub" / "b.png", size=(64, 64))
(self.output_dir / "bad.png").write_bytes(b"\x89PNG not really")
status, body = await self._start(mode="missing", size=200, quality="medium")
self.assertEqual(status, 202)
summary = body["job"]
self.assertEqual(
(summary["kind"], summary["title"], summary["admission"]),
("thumbnails", "Thumbnail generation", "guarded"),
)
self.assertTrue(summary["cancellable"])
self.assertEqual(set(summary["counters"]), {"generated", "skipped", "errors"})
self.assertEqual(
summary["scope"], {"mode": "missing", "size": 200, "quality": "medium"}
)
job = await self._finished(summary["id"])
self.assertEqual(job.state, "finished")
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 1})
self.assertEqual((job.processed, job.total, job.progress), (3, 3, 100))
self.assertEqual(job.scope["output_folders"], 1)
with Image.open(self._thumb_path("a")) as thumb:
self.assertEqual(thumb.size, (200, 100))
rows = {row["path"]: row for row in job.outcomes["generated"]}
self.assertEqual(set(rows), {"a.png", "sub/b.png"})
self.assertEqual(
(rows["a.png"]["root"], rows["a.png"]["root_index"]), ("output", 0)
)
self.assertEqual(job.outcomes["errors"][0]["path"], "bad.png")
self.assertTrue(any("Thumbnails finished" in line for line in job.log))
# Rows and log lines never carry the server's absolute layout.
self.assertNotIn(str(self.comfy_dir), repr(job.detail()))
async def test_fresh_previews_are_counted_as_skipped(self):
make_png(self.output_dir / "a.png")
make_png(self.output_dir / "b.png")
fresh = make_png(self._thumb_path("a"))
source_time = (self.output_dir / "a.png").stat().st_mtime
os.utime(fresh, (source_time + 10, source_time + 10))
_, body = await self._start()
job = await self._finished(body["job"]["id"])
self.assertEqual(job.counters, {"generated": 1, "skipped": 1, "errors": 0})
self.assertEqual([r["path"] for r in job.outcomes["skipped"]], ["a.png"])
async def test_rebuild_never_touches_originals(self):
for name in ("a", "b"):
make_png(self.output_dir / f"{name}.png", size=(400, 400))
_, body = await self._start(size=128)
await self._finished(body["job"]["id"])
originals = {
name: hashlib.sha256(
(self.output_dir / f"{name}.png").read_bytes()
).hexdigest()
for name in ("a", "b")
}
_, body = await self._start(mode="rebuild", size=256, quality="high")
job = await self._finished(body["job"]["id"])
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 0})
for name in ("a", "b"):
digest = hashlib.sha256(
(self.output_dir / f"{name}.png").read_bytes()
).hexdigest()
self.assertEqual(digest, originals[name], name)
with Image.open(self._thumb_path(name)) as thumb:
self.assertEqual(thumb.size, (256, 256))
async def test_rebuild_keeps_old_preview_when_generation_fails(self):
source = make_png(self.output_dir / "a.png", size=(64, 64))
_, body = await self._start()
await self._finished(body["job"]["id"])
preview = self._thumb_path("a").read_bytes()
source.write_bytes(b"\x89PNG broken now")
later = self._thumb_path("a").stat().st_mtime + 10
os.utime(self._thumb_path("a"), (later, later)) # still fresh
_, body = await self._start(mode="rebuild")
job = await self._finished(body["job"]["id"])
self.assertEqual(job.counters["errors"], 1)
self.assertEqual(self._thumb_path("a").read_bytes(), preview)
leftovers = [p.name for p in self._thumb_path("a").parent.iterdir()]
self.assertEqual(leftovers, ["a_thumb.png"]) # no temp file left
async def test_rebuild_keeps_an_old_video_preview_instead_of_a_placeholder(self):
clip = self.output_dir / "clip.mp4"
clip.write_bytes(b"\x00")
preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.jpg")
old = preview.read_bytes()
with (
patch.dict(sys.modules, {"cv2": None}),
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
):
_, body = await self._start(mode="rebuild")
job = await self._finished(body["job"]["id"])
self.assertEqual(job.counters["errors"], 1)
self.assertEqual(preview.read_bytes(), old)
async def test_quality_sets_the_encoder_not_the_edge(self):
noise = bytes(
(i * 7919 + (i >> 3) * 104729) % 251 for i in range(600 * 600 * 3)
)
with Image.frombytes("RGB", (600, 600), noise) as img:
img.save(self.output_dir / "photo.jpg", "JPEG", quality=95)
sizes = {}
for quality in ("low", "high"):
_, body = await self._start(mode="rebuild", size=300, quality=quality)
await self._finished(body["job"]["id"])
thumb = self._thumb_path("photo", ".jpg")
with Image.open(thumb) as image:
self.assertEqual(image.size, (300, 300))
sizes[quality] = thumb.stat().st_size
self.assertLess(sizes["low"], sizes["high"])
async def test_no_output_folder_fails_the_job(self):
self.api._get_all_output_dirs = lambda: []
_, body = await self._start()
job = await self._finished(body["job"]["id"])
self.assertEqual(job.state, "failed")
self.assertEqual(job.status_line, "ComfyUI output directory not found")
async def test_cancel_stops_at_file_boundary_keeping_written_previews(self):
for i in range(6):
make_png(self.output_dir / f"p{i}.png")
original = PromptManagerConfig.WORKER_THREADS
PromptManagerConfig.WORKER_THREADS = 1
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
written = []
real_write = images_module._write_image_thumbnail
job_ids = []
def counting(src, dst, *args, **kwargs):
real_write(src, dst, *args, **kwargs)
written.append(dst)
if len(written) == 2:
self.assertTrue(self.api._jobs.cancel(job_ids[0]))
with patch.object(images_module, "_write_image_thumbnail", counting):
_, body = await self._start()
job_ids.append(body["job"]["id"])
job = await self._finished(body["job"]["id"])
self.assertEqual(job.state, "cancelled")
self.assertEqual(job.counters["generated"], 2)
self.assertEqual(len(written), 2)
for dst in written:
self.assertTrue(dst.is_file())
made = sorted((self.output_dir / "thumbnails").glob("*_thumb.png"))
self.assertEqual(made, sorted(written))
self.assertTrue(any("Thumbnails cancelled" in line for line in job.log))
async def test_a_second_thumbnail_job_is_refused_naming_the_first(self):
make_png(self.output_dir / "a.png")
entered, release = self._gate_generation()
_, first = await self._start()
await self._wait(entered)
status, body = await self._start(mode="rebuild")
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["id"], first["job"]["id"])
release.set()
await self._finished(first["job"]["id"])
async def test_thumbnail_job_refused_while_scan_runs(self):
make_png(self.output_dir / "a.png")
self._run_scan()
status, body = await self._start()
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["title"], "Scan for metadata")
self.assertFalse(self._thumb_path("a").exists())
async def test_scan_refused_while_thumbnail_job_runs(self):
make_png(self.output_dir / "a.png")
entered, release = self._gate_generation()
_, started = await self._start()
await self._wait(entered)
resp = await self.client.post("/prompt_manager/jobs/scan")
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["title"], "Thumbnail generation")
release.set()
await self._finished(started["job"]["id"])
async def test_clear_refused_while_thumbnail_job_runs(self):
make_png(self.output_dir / "a.png")
kept = make_png(self.output_dir / "thumbnails" / "old_thumb.png")
entered, release = self._gate_generation()
_, started = await self._start()
await self._wait(entered)
resp = await self.client.post("/prompt_manager/images/clear-thumbnails")
self.assertEqual(resp.status, 409)
self.assertEqual(
await resp.json(),
{"success": False, "error": "Thumbnail generation already running"},
)
self.assertTrue(kept.exists())
release.set()
await self._finished(started["job"]["id"])
async def test_every_exit_sets_a_truthful_final_status_line(self):
"""I3: finished, nothing to do, cancelled and failed each say what happened."""
lines = self._status_lines()
make_png(self.output_dir / "a.png")
(self.output_dir / "bad.png").write_bytes(b"\x89PNG nope")
_, body = await self._start()
job = await self._finished(body["job"]["id"])
self.assertEqual(job.status_line, "1 preview written · 1 error")
# Every preview fresh: the job finishes with nothing to do.
source_time = (self.output_dir / "a.png").stat().st_mtime
bad_thumb = make_png(self._thumb_path("bad"))
for thumb in (self._thumb_path("a"), bad_thumb):
os.utime(thumb, (source_time + 10, source_time + 10))
os.utime(self.output_dir / "bad.png", (source_time, source_time))
_, body = await self._start()
job = await self._finished(body["job"]["id"])
self.assertEqual((job.state, job.total), ("finished", 0))
self.assertEqual(
job.status_line, "No previews to generate · 2 already up to date"
)
self.assertNotIn("Processing file 0 of 0", lines)
async def test_an_empty_library_says_there_is_nothing_to_preview(self):
lines = self._status_lines()
_, body = await self._start()
job = await self._finished(body["job"]["id"])
self.assertEqual(
(job.state, job.status_line), ("finished", "No images or videos to preview")
)
self.assertNotIn("Processing file 0 of 0", lines)
async def test_a_cancelled_job_says_what_it_kept(self):
for i in range(6):
make_png(self.output_dir / f"p{i}.png")
self._one_worker()
real_write = images_module._write_image_thumbnail
job_ids = []
written = []
def counting(src, dst, *args, **kwargs):
real_write(src, dst, *args, **kwargs)
written.append(dst)
if len(written) == 1:
self.api._jobs.cancel(job_ids[0])
with patch.object(images_module, "_write_image_thumbnail", counting):
_, body = await self._start()
job_ids.append(body["job"]["id"])
job = await self._finished(body["job"]["id"])
self.assertEqual(job.state, "cancelled")
self.assertEqual(job.status_line, "Cancelled · 1 preview written")
async def test_an_internal_error_says_the_job_failed(self):
def broken(*_args, **_kwargs):
raise RuntimeError("boom")
self.api._plan_gallery_thumbnails = broken
_, body = await self._start()
job = await self._finished(body["job"]["id"])
self.assertEqual(
(job.state, job.status_line),
("failed", "Thumbnail generation stopped on an internal error."),
)
async def test_cancel_during_planning_stops_the_walk_and_starts_nothing(self):
"""M2: the walk checks Cancel per file; no ``start`` frame is published."""
for i in range(10):
make_png(self.output_dir / f"p{i}.png")
entered, release = threading.Event(), threading.Event()
self.addCleanup(release.set)
checked = []
real_fresh = images_module._thumbnail_is_fresh
def gated_fresh(src, dst):
checked.append(src.name)
if len(checked) == 1:
entered.set()
release.wait(10)
return real_fresh(src, dst)
with patch.object(images_module, "_thumbnail_is_fresh", gated_fresh):
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
await self._wait(entered)
job = self.api._jobs.active("thumbnails")
self.assertTrue(self.api._jobs.cancel(job.id))
release.set()
events = parse_sse(await (await stream).text())
await asyncio.wait_for(job.task, 10)
self.assertEqual(checked, ["p0.png"]) # the walk stopped at the next file
self.assertEqual([name for name, _ in events], ["job", "status", "error"])
self.assertEqual(job.state, "cancelled")
self.assertEqual(job.status_line, "Cancelled before any preview was written")
self.assertEqual(job.total, 0)
self.assertFalse((self.output_dir / "thumbnails").exists())
async def test_a_locked_preview_is_a_readable_coded_error(self):
"""M6: Windows refuses to replace a preview another process has open."""
make_png(self.output_dir / "a.png")
locked = PermissionError(13, "Access is denied")
locked.winerror = 5
def refuse(*_args, **_kwargs):
raise locked
with patch.object(images_module, "_write_image_thumbnail", refuse):
_, body = await self._start(mode="rebuild")
job = await self._finished(body["job"]["id"])
self.assertEqual(job.counters["errors"], 1)
[row] = job.outcomes["errors"]
self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE)
self.assertNotIn("WinError", repr(job.detail()))
self.assertTrue(any(images_module.PREVIEW_IN_USE in line for line in job.log))
async def test_a_locked_video_preview_is_the_same_coded_error(self):
clip = self.output_dir / "clip.mp4"
clip.write_bytes(b"\x00")
locked = PermissionError(13, "Access is denied")
locked.winerror = 32
def refuse(*_args, **_kwargs):
raise locked
with (
patch.dict(sys.modules, {"cv2": None}),
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
patch.object(images_module, "_atomic_save", refuse),
):
_, body = await self._start()
job = await self._finished(body["job"]["id"])
[row] = job.outcomes["errors"]
self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE)
async def test_a_permission_error_off_windows_names_the_permission(self):
make_png(self.output_dir / "a.png")
def refuse(*_args, **_kwargs):
raise PermissionError(13, "Permission denied")
with patch.object(images_module, "_write_image_thumbnail", refuse):
_, body = await self._start()
job = await self._finished(body["job"]["id"])
[row] = job.outcomes["errors"]
self.assertEqual(row["detail"], images_module.PREVIEW_NOT_WRITABLE)
async def test_ffmpeg_fallback_keeps_quality_and_aspect_ratio(self):
"""M1: without OpenCV, ffmpeg keeps the chosen quality and the aspect ratio."""
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
commands = []
def fake_ffmpeg(cmd, **_kwargs):
commands.append(cmd)
Path(cmd[-1]).write_bytes(b"jpeg")
return type("Done", (), {"returncode": 0, "stderr": ""})()
for quality, qscale in (("high", "2"), ("medium", "4"), ("low", "8")):
with (
patch.dict(sys.modules, {"cv2": None}),
patch("subprocess.run", side_effect=fake_ffmpeg),
):
_, body = await self._start(mode="rebuild", size=640, quality=quality)
job = await self._finished(body["job"]["id"])
self.assertEqual(job.counters["generated"], 1, quality)
cmd = commands[-1]
self.assertNotIn("-s", cmd)
self.assertEqual(
cmd[cmd.index("-vf") + 1],
"scale=640:640:force_original_aspect_ratio=decrease",
)
self.assertEqual(cmd[cmd.index("-q:v") + 1], qscale, quality)
async def test_the_placeholder_uses_the_chosen_quality(self):
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
saved = []
real_save = images_module._atomic_save
def recording(img, dst, fmt=None, **kwargs):
saved.append(kwargs)
return real_save(img, dst, fmt, **kwargs)
with (
patch.dict(sys.modules, {"cv2": None}),
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
patch.object(images_module, "_atomic_save", recording),
):
_, body = await self._start(quality="low")
await self._finished(body["job"]["id"])
self.assertEqual(saved, [images_module.THUMBNAIL_ENCODINGS["low"]])
# The legacy stream's frames and their keys, as the gallery page reads them.
LEGACY_KEYS = {
"job": {"id", "started"},
"status": {"phase", "message"},
"start": {
"total_images",
"skipped",
"remaining",
"phase",
"image_count",
"video_count",
"message",
},
"progress": {
"processed",
"total_images",
"generated",
"skipped",
"error_count",
"percentage",
"rate",
"eta",
"elapsed",
"current_file",
"file_type",
"action",
},
"complete": {
"count",
"skipped",
"total_images",
"remaining",
"errors",
"error_count",
"elapsed_time",
"processing_rate",
"message",
},
}
class TestLegacyStreamOnTheJob(ThumbnailJobCase):
async def test_legacy_progress_events_unchanged_on_success(self):
make_png(self.output_dir / "a.png", size=(400, 400))
make_png(self.output_dir / "b.png")
resp = await self.client.post(f"{self.PROGRESS_URL}?quality=low")
events = parse_sse(await resp.text())
self.assertEqual(
[name for name, _ in events],
["job", "status", "start", "progress", "progress", "complete"],
)
for name, data in events:
self.assertEqual(set(data), LEGACY_KEYS[name], name)
complete = events[-1][1]
self.assertEqual((complete["count"], complete["total_images"]), (2, 2))
self.assertEqual(
events[1][1],
{"phase": "scanning", "message": "Scanning for images and videos..."},
)
with Image.open(self._thumb_path("a")) as thumb:
self.assertEqual(max(thumb.size), 150) # the legacy preset still sizes
[job] = self.api._jobs.list()
self.assertEqual(
(job.kind, job.title, job.state, job.cancellable),
("thumbnails", "Thumbnail generation", "finished", True),
)
self.assertEqual(job.counters["generated"], 2)
# I1: the first frame names the job this stream started, for its Cancel.
self.assertEqual(events[0][1], {"id": job.id, "started": True})
async def test_legacy_file_errors_still_stream(self):
make_png(self.output_dir / "good.png")
(self.output_dir / "bad.png").write_bytes(b"\x89PNG nope")
resp = await self.client.post(self.PROGRESS_URL)
events = parse_sse(await resp.text())
self.assertEqual(
[d for e, d in events if e == "file_error"],
[{"file": "bad.png", "error": "Not a valid image file"}],
)
async def test_thumbnail_job_survives_client_disconnect(self):
"""I2: the stream sees its client go after batch 1; every batch still runs.
One worker thread makes batches of 4, so 9 files take three batches. The
first file waits until the client is gone; the second batch waits until
the stream has seen it go (its subscription dropped) and records whether
the job was still running then. A stream that stopped the work on
disconnect would leave 4 previews; the job writes all 9.
"""
for i in range(9):
make_png(self.output_dir / f"p{i}.png")
self._one_worker()
first, closed = threading.Event(), threading.Event()
state = {"calls": 0, "left": None, "running_when_left": None}
original = self.api._generate_one
def gated(*args, **kwargs):
state["calls"] += 1
if state["calls"] == 1:
first.set()
closed.wait(10)
elif state["calls"] == 5: # the second batch
state["running_when_left"] = state["left"].wait(10)
return original(*args, **kwargs)
self.api._generate_one = gated
self.addCleanup(closed.set)
resp = await self.client.post(self.PROGRESS_URL)
self.assertEqual(resp.status, 200)
await self._wait(first)
job = self.api._jobs.active("thumbnails")
_subscribed, state["left"] = self._watch_subscribers(job)
resp.close() # the gallery tab went away
closed.set()
await self._wait(state["left"]) # the server saw the client go
self.assertEqual(job.subscriber_count, 0)
await asyncio.wait_for(job.task, 10)
self.assertEqual(job.counters["generated"], 9)
self.assertEqual(job.state, "finished")
self.assertTrue(state["running_when_left"])
for i in range(9):
self.assertTrue(self._thumb_path(f"p{i}").is_file())
async def test_legacy_stream_attaches_to_a_running_job(self):
make_png(self.output_dir / "a.png")
entered, release = self._gate_generation()
_, started = await self._start(mode="missing", size=256)
await self._wait(entered)
job = self.api._jobs.get(started["job"]["id"])
subscribed, _left = self._watch_subscribers(job)
attached = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
await self._wait(subscribed) # M4: an event with a ceiling, never a poll
release.set()
events = parse_sse(await (await attached).text())
self.assertEqual(events[-1][0], "complete")
# I1: the stream names the job it did not start: the gallery never cancels it.
self.assertEqual(events[0], ("job", {"id": job.id, "started": False}))
# One job: the stream followed the running one with its own options.
[listed] = self.api._jobs.list()
self.assertIs(listed, job)
self.assertEqual(job.scope["size"], 256)
async def test_cancel_from_activity_ends_the_legacy_stream_with_an_error(self):
for i in range(2):
make_png(self.output_dir / f"p{i}.png")
entered, release = self._gate_generation()
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
await self._wait(entered)
job = self.api._jobs.active("thumbnails")
self.assertTrue(self.api._jobs.cancel(job.id))
release.set()
events = parse_sse(await (await stream).text())
self.assertEqual(events[-1][0], "error")
self.assertEqual(events[-1][1]["error"], "Thumbnail generation was cancelled")
await asyncio.wait_for(job.task, 10)
self.assertEqual(job.state, "cancelled")
async def test_a_job_started_while_the_stream_prepares_is_attached(self):
"""M3: a job that starts during ``prepare`` is followed, not refused."""
from aiohttp import web
make_png(self.output_dir / "a.png")
entered, release = self._gate_generation()
real_prepare = web.StreamResponse.prepare
raced = []
async def prepare(response, request):
if not raced and type(response) is web.StreamResponse:
options, _ = thumbnail_options({"size": 256})
raced.append(self.api._start_thumbnail_job(options))
return await real_prepare(response, request)
with patch.object(web.StreamResponse, "prepare", prepare):
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
await self._wait(entered)
release.set()
events = parse_sse(await (await stream).text())
[job] = raced
self.assertEqual(events[0], ("job", {"id": job.id, "started": False}))
self.assertEqual(events[-1][0], "complete")
self.assertEqual(self.api._jobs.list(), [job])
async def test_a_refused_stream_names_no_job(self):
make_png(self.output_dir / "a.png")
self._run_scan()
events = parse_sse(await (await self.client.post(self.PROGRESS_URL)).text())
self.assertEqual(
events,
[
(
"error",
{
"error": "Scan for metadata already running",
"message": "Scan for metadata already running",
},
)
],
)
if __name__ == "__main__":
unittest.main()
+38
View File
@@ -0,0 +1,38 @@
"""The System browser hooks only ever reset their own server's config.json (#249 S3)."""
import os
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from tests.browser.workspace_system_fixture import register_system_commands
class TestWorkspaceSystemFixture(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
self.root = Path(self.tmp.name) / "server"
self.root.mkdir()
def test_refuses_a_config_path_outside_its_root(self):
elsewhere = Path(self.tmp.name) / "config.json"
elsewhere.write_bytes(b'{"real": true}')
for value in (str(elsewhere), None):
env = {
k: v for k, v in os.environ.items() if k != "PROMPT_MANAGER_CONFIG_PATH"
}
if value is not None:
env["PROMPT_MANAGER_CONFIG_PATH"] = value
with (
self.subTest(value=value),
mock.patch.dict(os.environ, env, clear=True),
):
with self.assertRaisesRegex(RuntimeError, "PROMPT_MANAGER_CONFIG_PATH"):
register_system_commands(None, None, None, self.root)
self.assertEqual(elsewhere.read_bytes(), b'{"real": true}')
if __name__ == "__main__":
unittest.main()
+14
View File
@@ -51,3 +51,17 @@ test('CI retains stock, execution and pack suites in order with uploaded executi
assert.deepEqual(commands, ['test:comfyui', 'test:executions', 'test:compat']);
assert.match(workflow, /path: tools\/vendor\/test-results\//);
});
test('real-host config waits for its own launcher, never for whichever server holds the port', async t => {
const previous = process.env.PM_COMFY_ROOT;
process.env.PM_COMFY_ROOT = '/unused-pinned-checkout';
t.after(() => { if (previous === undefined) delete process.env.PM_COMFY_ROOT; else process.env.PM_COMFY_ROOT = previous; });
const { default: config } = await import('../../tools/vendor/comfyui.config.mjs');
const [server] = config.webServer;
assert.equal(server.url, undefined);
assert.equal(server.port, undefined);
assert.equal(server.reuseExistingServer, false);
const ready = server.wait.stdout.exec('Starting server\nPM_COMFY_HOST_READY base-directory=/tmp/comfyassets-host-a b\r\nlater');
assert.equal(ready.groups.pm_comfy_host_base, '/tmp/comfyassets-host-a b');
assert.equal(server.wait.stdout.exec('To see the GUI go to: http://127.0.0.1:8189'), null);
});
+50 -1
View File
@@ -6,7 +6,8 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { once } from 'node:events';
import { createServer } from 'node:net';
import { launchComfyHost } from '../../tools/vendor/comfyui-host.mjs';
import { createServer as createHttpServer } from 'node:http';
import { launchComfyHost, assertPortFree, waitForOwnHost, hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
async function fixture(t, program) {
const directory = await mkdtemp(join(tmpdir(), 'pm-host-test-'));
@@ -232,3 +233,51 @@ test('execution launcher refuses an occupied port without touching an existing s
await new Promise(resolve => server.close(resolve));
}
});
// A stand-in ComfyUI that answers /system_stats with its own argv, as the real host does.
const answersOwnArgv = `const args = process.argv.slice(1);
require('node:http').createServer((request, response) => {
response.setHeader('content-type', 'application/json');
response.end(JSON.stringify({ system: { argv: args } }));
}).listen(Number(args[args.indexOf('--port') + 1]), '127.0.0.1');`;
async function listen(t, port, handler = (request, response) => response.end()) {
const server = createHttpServer(handler);
await new Promise((resolve, reject) => { server.once('error', reject); server.listen(port, '127.0.0.1', resolve); });
t.after(() => new Promise(resolve => server.close(resolve)));
return server;
}
test('real-host launcher refuses a port another run already holds and leaves that server running', async t => {
const server = await listen(t, 8199);
await assert.rejects(assertPortFree(8199), /8199 is already in use.*never shared/s);
assert.equal(server.listening, true);
});
test('real-host launcher is ready only when its own ComfyUI answers the port', async t => {
const options = await fixture(t, answersOwnArgv);
await assertPortFree(options.port);
const host = await launchComfyHost(options);
t.after(() => host.stop());
await waitForOwnHost(host, options.port);
const stats = await (await fetch(`http://127.0.0.1:${options.port}/system_stats`)).json();
assert.equal(hostBaseDirectory(stats), host.stateDirectory);
});
test('real-host launcher refuses a host it did not start that answers before its own binds', async t => {
await listen(t, 8199, (request, response) => {
response.setHeader('content-type', 'application/json');
response.end(JSON.stringify({ system: { argv: ['main.py', '--base-directory', '/other-run-state'] } }));
});
const options = await fixture(t, idle);
const host = await launchComfyHost(options);
t.after(() => host.stop());
await assert.rejects(waitForOwnHost(host, options.port), /did not start \(base directory \/other-run-state\)/);
});
test('real-host launcher reports a host that exits before it answers', async t => {
const options = await fixture(t, 'process.exit(3);');
const host = await launchComfyHost(options);
t.after(() => host.stop());
await assert.rejects(waitForOwnHost(host, options.port), /exited before it answered on port 8199 \(exit=3/);
});
+58 -3
View File
@@ -1,4 +1,5 @@
import { writeFile } from 'node:fs/promises';
import { createServer } from 'node:net';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { launchComfyHost as launchSharedHost } from '@comfyassets/toolkit/testing/host';
@@ -7,6 +8,9 @@ import runtime from './comfyui-runtime.json' with { type: 'json' };
export { launchSharedHost };
const repositoryRoot = fileURLToPath(new URL('../../', import.meta.url));
// comfyui.config.mjs waits for this line; its named group becomes PM_COMFY_HOST_BASE in the test workers.
export const HOST_READY_MARKER = 'PM_COMFY_HOST_READY';
const READY_POLL_MS = 250;
export async function preparePromptManagerPlugin({ installed, stateDirectory }) {
const configPath = join(installed, 'config.json');
@@ -27,17 +31,68 @@ export function launchComfyHost({ checkout, repository = repositoryRoot,
});
}
export async function assertPortFree(port) {
await new Promise((resolve, reject) => {
const probe = createServer();
probe.once('error', error => reject(new Error(`ComfyUI host port ${port} is already in use, often by another `
+ 'real-host run. A managed host is never shared: stop that run or set PM_COMFY_URL to a free '
+ 'http://127.0.0.1:PORT/ address.', { cause: error })));
probe.listen({ host: '127.0.0.1', port, exclusive: true }, () => probe.close(error => error ? reject(error) : resolve()));
});
}
export function hostBaseDirectory(stats) {
const argv = stats?.system?.argv;
if (!Array.isArray(argv)) return null;
const index = argv.indexOf('--base-directory');
return index >= 0 ? argv[index + 1] ?? null : null;
}
// Ready means this launcher's own ComfyUI answers: /system_stats reports its base directory.
// Any other answer on the port comes from a host this run did not start, so the run must not use it.
export async function waitForOwnHost(host, port, { fetchImpl = fetch } = {}) {
let exited = null;
host.finished.then(result => { exited = result; });
const url = `http://127.0.0.1:${port}/system_stats`;
for (;;) {
let stats = null;
try { stats = await (await fetchImpl(url)).json(); }
catch { stats = null; }
if (stats) {
const base = hostBaseDirectory(stats);
if (base === host.stateDirectory) return;
throw new Error(`Port ${port} is answered by a ComfyUI host this run did not start `
+ `(base directory ${base ?? 'unknown'}); another run is using the port.`);
}
if (exited) {
throw new Error(`ComfyUI host exited before it answered on port ${port} `
+ `(exit=${exited.code}, signal=${exited.signal || 'none'}).`);
}
await new Promise(resolve => setTimeout(resolve, READY_POLL_MS));
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
try {
const host = await launchComfyHost({ checkout: process.env.PM_COMFY_ROOT,
port: Number(process.env.PM_COMFY_PORT || 8189) });
const port = Number(process.env.PM_COMFY_PORT || 8189);
await assertPortFree(port);
const host = await launchComfyHost({ checkout: process.env.PM_COMFY_ROOT, port });
const stop = () => { host.stop().catch(error => { console.error(error.message); process.exitCode = 1; }); };
process.once('SIGINT', stop);
process.once('SIGTERM', stop);
let refused = false;
try {
await waitForOwnHost(host, port);
console.log(`${HOST_READY_MARKER} base-directory=${host.stateDirectory}`);
} catch (error) {
refused = true;
console.error(error.message);
await host.stop();
}
const result = await host.finished;
process.removeListener('SIGINT', stop);
process.removeListener('SIGTERM', stop);
if (result.code) console.error(`ComfyUI host exited with status ${result.code}; see tools/vendor/test-results/comfyui-host.log`);
process.exitCode = result.code;
process.exitCode = refused ? 1 : result.code;
} catch (error) { console.error(error.message); process.exitCode = 1; }
}
+6 -1
View File
@@ -1,5 +1,6 @@
import { defineConfig } from './node_modules/@playwright/test/index.mjs';
import { fileURLToPath } from 'node:url';
import { HOST_READY_MARKER } from './comfyui-host.mjs';
if (!process.env.PM_COMFY_ROOT) throw new Error('Set PM_COMFY_ROOT to a pinned disposable ComfyUI checkout');
const host = new URL(process.env.PM_COMFY_URL || 'http://127.0.0.1:8189/');
@@ -7,7 +8,11 @@ if (host.protocol !== 'http:' || host.hostname !== '127.0.0.1' || !host.port) th
const servers = [{
command: 'node tools/vendor/comfyui-host.mjs',
cwd: fileURLToPath(new URL('../../', import.meta.url)),
url: new URL('system_stats', host).href, timeout: 120000, reuseExistingServer: false,
// No URL wait: any server on the port would satisfy it, including another run's host that
// then stops mid-test. The launcher refuses a used port and prints this line only once its
// own ComfyUI answers; the group reaches the tests as PM_COMFY_HOST_BASE.
wait: { stdout: new RegExp(`^${HOST_READY_MARKER} base-directory=(?<pm_comfy_host_base>.+?)\r?$`, 'm') },
timeout: 120000, reuseExistingServer: false,
env: { PM_COMFY_PORT: host.port },
gracefulShutdown: { signal: 'SIGTERM', timeout: 10000 },
}];

Some files were not shown because too many files have changed in this diff Show More