Compare commits

...
Author SHA1 Message Date
Vito Sansevero d2ff73d299 test(system): match the scan scope keys beside the review counters 2026-10-23 13:59:23 -07:00
Vito Sansevero 40cecbced2 test(scan): pin only the per-source scope keys beside the review counters 2026-10-23 13:41:08 -07:00
Vito Sansevero f9a781c30a Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts:
#	tests/test_api_admin.py
2026-10-23 13:34:28 -07:00
Vito 69de20a427 feat(system): Sources & scans (#309)
* feat(system): report configured, effective and watched sources

GET /prompt_manager/settings adds settings.monitoring_enabled, a sources
block (configured roots with valid/code/reason, effective roots with their
listing index and watch state, auto_detected) and the monitor state
(present, observer_alive, scheduled). Every path is a public path.

A source is "watching" only while a live observer scheduled it, "manual"
when monitoring is off or the process has no monitor, and otherwise
"restart_pending".

POST accepts monitoring_enabled as a strict bool, saved under
gallery.monitoring.enabled; restart_required_fields names it when the
running observer does not already match. A save that changes the sources
runs as a "Source change" records hold, so it is refused with 409 busy
while a scan runs. Case variants of one folder are saved once.

POST /prompt_manager/settings/validate-source {path} checks a path
lexically (no "..", no control characters, native path forms only) and
then against the gallery-root rules, answering a code and reason, and
writes nothing.

* fix(system): no existence oracle in source validation

A gallery root's containment in the allowed ComfyUI folders is now
decided before its existence and type, so a path outside them answers
"outside-allowed" whether or not it exists. validate-source also folds
the filesystem-root and home-directory codes into "outside-allowed", with
one reason, and logs the detail server-side only. Paths inside the
allowed folders still say "not-found" or "not-directory". The legacy
settings save shares the same check.

* feat(scan): scan one selected source

POST /prompt_manager/jobs/scan accepts roots: indexes into the effective
sources, strict non-negative ints (no bools or floats), deduplicated, at
most 64, non-empty when present. An index past the sources is a 400
"unknown-source" and starts nothing. The job walks only the selected
sources, resolved when it was requested, and keeps each file's listing
root_index. Its scope records files, roots (null for every source) and
output_folders, the number of folders walked, which the log's first line
counts. The legacy /scan route still scans every source, and attaching to
a scoped job keeps its payload keys.

* feat(system): manage sources and start scans natively

System > Sources & scans renders the concept's section on the server's
sources: one row per effective source with its watch state and a scan
scoped to that source, the 3.2.0 update notice with the legacy dismissal,
Add source checked by validate-source, Remove, monitoring and worker
threads as host drafts behind the one Save row, and Refresh files.
A save refused by a running scan names the job. Thumbnail generation
hands off to the legacy gallery until S5.

Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md.

* fix(security): decide source containment as text before any filesystem access

Every caller (validate-source, POST /settings, config.json at load) now
refuses a source through one check, sources.source_problem: the form
check first, then GalleryConfig.gallery_root_problem, which compares the
normalised spelling with the allowed folders before anything resolves,
stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a
device or any outside folder is refused with no SMB request, no
executor stall and no existence or timing oracle; only then is the path
resolved and contained again, which still catches a symlink out. The
filesystem root and the home directory answer outside-allowed with the
same reason as any outside path, so no caller learns the home path.

Also:
- POST /settings runs the form checks and parses the roots off the loop;
  an unusable source is a coded 400 invalid-source naming its public
  path (as text) and its index.
- ComfyUI folders outside the base directory are path anchors, so their
  sources round-trip and never collide by folder name.
- A per-source scan sends each index with its public path; a stale or
  shifted index is 409 sources-changed and starts nothing.
- A source still watched after monitoring was switched off is
  "stopping".
- Roots are compared as folders, not spellings, so an unchanged legacy
  resend takes no source-change hold.
- Tests: no realpath/stat/lstat of an outside path (native and Windows
  forms with ntpath fakes), the alike answers with /, ~ and a link to ~,
  an event instead of a timer wait, and a teardown that drains jobs
  before the temporary database goes.

* fix(system): say what Sources saved, refused and still watches

- A per-source scan sends the source's public path; when the sources
  changed since they were read the toast says the server's reason and
  the list and saved values are read again.
- "watching until restart" for a source the running monitor still
  watches after monitoring was saved off, and the status adds "Restart
  ComfyUI to apply: Watch for new outputs." from restart_required_fields.
- A save refused for one unusable source marks that row and the status
  is the server's message naming it.
- A valid configured source the effective list holds under another
  spelling reads "same folder as another source".
- The busy toast names the job the 409 named (ApiError keeps a
  validated running job and refused source), falling back to the list.
- The Add source dialog's listeners live on the page scope and Enter
  adds; its markup is compared with the concept (D127).
- TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the
  Sources contract describes the lexical containment, invalid-source and
  sources-changed.

* fix(security): guard the Windows form probes on a token of their own

The no-filesystem guard matched "C:", which on a Windows runner is also
the temporary ComfyUI tree the save may resolve for its trusted first
entry. The probe paths now carry a token no trusted directory holds.

* docs(system): number the Sources states deviation D128
2026-10-23 13:15:06 -07:00
Vito Sansevero 18d81c9d96 fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).

A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
2026-10-23 13:03:10 -07:00
Vito Sansevero 9e125ee4a3 Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend 2026-10-23 11:09:28 -07:00
Vito cbc53ee9f1 feat(resolver): trace multi-stage chains through pass-through nodes (#308)
* feat(resolver): trace multi-stage chains through pass-through nodes

The inspection resolver (server and local Inspect) now compares a sampler
with the earlier sampler its latent continues through image-space
pass-through nodes: decode, image filter, model upscale, scale and encode,
listed in a new validated `lineage` section of the semantics table. Equal
prompts resolve (hires-fix, refiner); different prompts are
multistage-differs. A later stage whose positive adds conditioning to the
shared base prompt through an additive composite (ConditioningCombine or
Concat) keeps the base pair, the text legacy linked (three-stage Flux).
Image-edit batches stay ambiguous and run-time text stays untraceable.
Ingestion keeps the default profile (D1).

A card no longer shows a seed when the inspector names a chain stage
whose seed differs from the parser's pick.

* feat(resolver): leave out run-time PromptManager parts, labelled

Maintainer option A: when a PromptManager prepend_text or append_text is
fed directly by a node whose output is decided at run time (LoraManager's
TriggerWord Toggle, now a runtime class, or PromptSearchList), the prompt
resolves to the node's own text, as legacy showed. The side's source is
runtime-parts-omitted and the resolved reason says that text added at run
time, such as trigger words, is not included; the prompt block shows that
line (D-NEW-R1). The toggle's output is never computed. Literal and
traced parts are combined as before; an indirect run-time part, a run-time
main text or an unknown feeder keep their outcome. The browser port
changes identically. Ingestion is unchanged.

* fix(resolver): keep run-time and added-conditioning labels honest

A text that left out a run-time PromptManager part is now labelled when
any node on its traced path omitted it, so the label survives encoders,
concatenations and chained PromptManagers. Comparisons include what a
text leaves out, so a partial prompt never equals a complete one: a
branch with trigger words and one without stay ambiguous, and such
stages are multistage-differs. Only classes listed as run time are left
out.

The added-conditioning floor labels its positive (D126, "Conditioning
added at a later stage is not included."); the run-time line is D125.

A card hides its seed only when the stage the inspector names reads a
different seed than the parser; an agreeing seed is kept, as on main.
The browser port moves its lineage, latent and part helpers into
local-resolver-stages.mjs, keeping parity.
2026-10-23 11:09:12 -07:00
Vito Sansevero 95f88d3966 fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).

A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.

The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.

The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
2026-10-23 09:41:27 -07:00
Vito 7c25b351f7 test(comfyui): refuse a real-host port another run holds (#310)
The real-host config waited on the /system_stats URL, which any server on
the port answers. When two runs started on 8189 within a few seconds of
each other, the second run's ComfyUI failed to bind ("Port 8189 is already
in use"), but Playwright had already seen the first run's host answer and
ran the second run's tests against it. When the first run finished and
stopped its host, the second run failed mid-test with ECONNREFUSED,
ECONNRESET or ERR_CONNECTION_REFUSED, and passed when rerun alone.

The launcher now refuses a port that is in use, and reports ready only
when /system_stats names its own base directory. The config waits for
that line instead of the URL, and every test checks again that the host
answering the port is the one its run started.
2026-10-23 09:25:32 -07:00
Vito Sansevero 9bfb5992b3 Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts:
#	py/api/system.py
#	tests/test_system_api.py
2026-10-23 09:24:07 -07:00
Vito 785b744554 feat(system): Appearance section and browser display defaults (#307)
* feat(system): share browser display defaults

The workspace reads and writes the legacy gallery's gallerySettings object
through preferences.mjs: workspaceLayout and workspaceDensity join the
shared keys, a write re-reads the stored object and writes only the changed
keys, denied storage or a full quota keeps the change for this tab and says
so, and storage events report changes from other pages. The legacy gallery
Save keeps keys it does not edit, and the viewer reads its video defaults
through the shared module.

* feat(system): Appearance defaults used by Images and the viewer

System > Appearance is the concept's Browsing & display surface on real
values: the browser rows edit this browser's display defaults through the
shared gallerySettings object, and Open from ComfyUI and the node search
auto-hide save to the server settings with their revision. A new Images
visit starts from the default layout and size (the current view and saved
views keep theirs); metadata off waits for Load metadata; cache off re-reads
the shown file; paths off hides the Path row; the viewer reads the video
defaults. D97 adds card details, collapsed viewer details and Reset display
defaults; storage refused says Saved for this tab only.

* fix(system): flag changes made while System was closed and keep embedded galleries fixed

Final review fixes for System S3 (Appearance):

- I1: the on-demand metadata deviation is D124 (TESTING.md, inspector
  markers); TESTING.md names the spec that pins it, since no concept
  comparison renders that state.
- I2: System checks the display defaults for conflicts on mount, not only
  on storage events, and a browser write refuses a draft another page
  overtook, so a draft is never saved over another tab's value without
  the conflict status first. New two-tab spec.
- I3: the layout and size defaults start the Images page only; embedded
  galleries (a prompt history's Shared gallery and dialogs) keep
  Grid/Comfortable.
- M1: host rows whose /settings read failed say "Server settings could
  not be read."
- M2: a rejected timeout shows the "Some changes were not saved" status.
- M3: the browser loads on mount and on storage events never reject
  unhandled.
- M4: the next accepted write stores the tab-only changes and clears the
  overlay.
- M5: cache off never re-reads a source-changed selection, and refresh()
  never loads a deferred one.
- M6: the System fixture refuses a config.json outside its temp root and
  restores it with os.replace.
- M7: the inspection read counts wait on a request barrier, not frames.
- M8 recorded as an open saved-views question in TESTING.md.

* fix(system): answer a failed System read only after its other reads finish

CI (py3.10 on ubuntu) crashed with a segfault inside SQLite in
test_statistics_failure_is_an_error_envelope: the statistics and overview
routes used asyncio.gather without return_exceptions, so the 500 answered
on the first failure while get_statistics still ran in the executor, and
the test tore the database down under it. Both routes now await every
read before raising the first failure (failing before, passing after:
TestFailedReadsSettleFirst).

The frontend coverage merge failed on workspace/activity.mjs branches
(86.3 < 87): no test asserted Activity's empty and failed-read states, so
their branch was only reached in passing, depending on timing. A spec now
asserts both.
2026-10-23 07:49:39 -07:00
Vito Sansevero 519ab59d71 fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
2026-10-23 07:45:57 -07:00
Vito Sansevero 369f38be9a fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
2026-10-23 07:11:56 -07:00
Vito Sansevero e6ed2914e1 fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
2026-10-23 06:55:31 -07:00
Vito Sansevero 878612c22d Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts:
#	database/pair_ingestion.py
#	py/api/__init__.py
2026-10-23 06:55:29 -07:00
Vito 966a25d129 fix(scan): ingest metadata whose graph holds NaN or Infinity (#303)
* fix(scan): ingest metadata whose graph holds NaN or Infinity

ComfyUI writes its prompt with Python's default json.dumps, so a node whose
IS_CHANGED returns float("nan") saves "is_changed": [NaN] in every PNG. The
exact-pair ingestion stored the snapshots with allow_nan=False, so the scan
failed those files ("Out of range float values are not JSON compliant: nan")
and the monitor dropped their link. The legacy pages replaced NaN with null
before JSON.parse and ingested them.

- utils/json_strict.dumps_strict: strict JSON with non-finite numbers as
  null; identical bytes for every other value.
- Pair snapshots and provenance, and the plain prompt link, store strict
  JSON (the legacy link wrote bare NaN into the database).
- The browser-parser scan fallback parses records as the legacy pages do
  (parse_page_json), so a graph with NaN still traces its prompt.
- The prompt images route answers strict JSON instead of regex-replacing
  "NaN" (which also rewrote prompt text such as "ratio: NaN"); the NaN
  cleaners also null Infinity.

Prompt text and its hash are unchanged: only numbers become null.

* fix(scan): leave the browser-parser NaN parse to the resolver parity fix

The scan fallback's legacy-page parse (parse_page_json in _graph_prompt_text)
is owned by the resolver parity change, which adds the same helper with the
legacy pages' exact rule (NaN reads as null, Infinity still rejected). This
branch keeps only the strict serialization of stored and answered JSON.
2026-10-23 06:48:21 -07:00
Vito 7511a0fd38 fix(activity): no Cancel on jobs that ended before Activity opened (#306)
* fix(activity): drop Cancel job from a job that had ended before Activity opened

Since #289 the Activity list removed an item's actions only when they were
isConnected. On the first render the item is not in the page yet, so a job
that had already ended when Activity opened kept its "Cancel job" (and a
running job's progress was moved after its actions, then back). The list
now asks parentNode instead. The review spec "a scan job with review items
offers Review metadata in Activity" failed on this before the change
(Expected 0 buttons on a finished job, Received 1) and passes after it.

* test(backup): wait for the restore, not a 5 s timer, in the thread test

test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.

* test(activity): no Cancel job on jobs that ended before Activity opened

Since #289 a list item dropped its actions only when they were isConnected,
so the first render of jobs the store already knew (Activity opened from
another page) kept "Cancel job" on a finished or cancelled job and moved a
running job's progress after its actions. Activity's own list read renders
again a moment later and hides it, so the new spec holds that read and checks
the first render: no progress or Cancel job on the ended jobs, and a running
job's children in order heading, detail, progress, actions.

With the parentNode fix reverted it fails on chromium and touch (Expected 0,
Received 1 on the finished job); with it, it passes.
2026-10-23 06:29:45 -07:00
Vito add035d940 fix(images): release a collection viewer image Viewer.js has not built yet (#293)
* fix(images): release a collection viewer image Viewer.js has not built yet

Leaving a viewer image while it was still loading (Previous, Next, the
arrow keys, Retry) deferred the inline Viewer.js destroy() until the
image's view settled, failed or a 10 s fallback ran. Before its image
loads Viewer.js has not built, so nothing could settle: the stage
dropped the image, the late load ran Viewer.js's build() on an image
without a parent, and getComputedStyle(null) threw a page error
("Argument 1 is not an object" in Firefox, "must be an instance of
Element" in WebKit). Under CI load this failed "collection viewer
follows filtered loaded order beyond mounted cards".

A viewer that is not ready yet is now destroyed at once, which cancels
its pending build, and its image src is released. A built viewer with
a still-loading view keeps the existing deferral.

* fix(images): build Viewer.js only on a loaded viewer image

Destroying an unbuilt inline Viewer.js at once (429b3a8) is not safe in
one state. Viewer.js's own cancel removes its image listeners only while
`image.complete` is false, so an image that had finished downloading
with its load event still queued kept them. Closing the viewer then
released the src, the queued load built Viewer.js anyway, and its first
view found no listed image: "can't access property querySelector, c is
undefined" (workspace-history.spec.mjs:222 on Firefox in CI).

The viewer now constructs Viewer.js from its own load listener, which the
media scope removes when the image is left. Before the load no Viewer.js
exists, so leaving the image only releases its src; after it Viewer.js
sees a complete image and only its zero-delay build is pending, which
destroy() cancels. Transforms report Viewer.js as unavailable only when
it did not load at all.

* fix(images): describe transforms by the Viewer.js an image was built with

1086436 creates Viewer.js only once the image has loaded, so the
"Viewer.js could not load" reason moved from the instance to the live
window.Viewer. An image shown natively while Viewer.js was missing then
offered transforms as soon as Viewer.js reappeared, which
full-inspector.spec.mjs:854 ("collection viewer preserves original
download and rooted retries") caught on every project.

Each rebuild now captures the Viewer.js it builds with. The reason, the
native fallback and the later construction all use that capture.
2026-10-23 06:08:44 -07:00
Vito 88e5b78207 fix(gallery): check duplicate deletes in the scanned root and refresh the listing (#305)
The duplicate cleanup workflow failed in CI with the deleted file still
served (200). The delete was real: the scan reads only the output root,
and the browser fixture's input root holds a different shared/same.png.
The test checked a bare /images/serve/<path> URL, which tries every
gallery root, so after the output copy went it answered with the input
copy. That only happened when the kernel stamped a bulk copy a fraction
of a millisecond before the wall-clock stamp on shared/same.png, which
made shared/same.png a Delete candidate (CI trace: 0000 .8605533,
same .8606777, 0001 .8618972).

The workflow now checks each file in its own root (?root=N), checks the
input copy survives, and a new workflow forces the CI order through the
scan response so the collision runs every time.

The server log in that failure also showed the gallery listing a file
the cleanup had deleted: delete_duplicate_images left the 30 s listing
cache, so the gallery kept counting the file. A successful delete now
invalidates the gallery cache.
2026-10-23 05:42:59 -07:00
Vito b0a025a326 fix(resolver): read NaN as null in the scan's legacy-parser fallback (#304)
ComfyUI stamps "is_changed": [NaN] on nodes that always re-run, the
PromptManager node among them. The legacy gallery and admin pages replaced
NaN with null before JSON.parse, so they traced these files' prompts. The
scan's fallback parsed records strictly, so every NaN-stamped file the
resolver could not decide (hires-fix and refiner chains, custom latent
nodes) fell to "Needs review" although legacy linked its prompt.

parse_page_json parses as the legacy pages did, but only the NaN number
reads as null: the word inside prompt text is kept. Infinity still makes
the record unreadable, as it did in the pages.
2026-10-23 05:17:10 -07:00
Vito c86739178d feat(system): System page shell, review dialog and Overview (#298)
* feat(system): read-only health check and truthful overview counts

POST /prompt_manager/system/health runs the health check off-loop as a
read registry job of kind "health" (one at a time, beside a scan, not
cancellable, no file outcomes) and keeps the last report; GET returns it
or null. The check reads one snapshot on its own mode=ro connection,
closed in finally: integrity_check(100), foreign_key_check, stored hashes
against utils.hashing, duplicate pair rows, the legacy consistency
queries (now shared with check_consistency), and image links whose file
is gone, counted only under readable sources. An unavailable source, or
the absent Needs-review queue, makes its check unavailable and the
report incomplete instead of producing a count.

GET /prompt_manager/system/overview returns Library-eligible positive
prompts and active negative variants, indexed images, the last health
summary or null, backup null, the Needs-review count null, the effective
sources with whether they were auto-detected, and the thumbnail plan's
missing count. GET /prompt_manager/system/statistics lists runs and
average rating per category of Library parents, with the /stats totals.

* feat(system): native System route, drafts and shared review dialog

- #/system/<section> renders the concept's System shell: section nav,
  settings layout and Save row with truthful status (D92); each section
  not delivered yet hands off to its legacy tools (D94). Overview's System
  card opens #/system/overview (Q12).
- A module-level draft store saves browser, then host (expected_revision),
  then runtime; failed domains stay dirty, a stale revision is a conflict.
- System row builders (setting, operation, section nav) on the scope.
- openReviewDialog and backupNotice extracted from the scan review, which
  is now a thin caller with identical markup.
- Overview images gain a "rated" count (images whose linked prompt is
  rated); the missing-files health check reports per source, so partial
  availability is visible.

* feat(system): native Overview with health report and statistics

System > Overview renders the concept's metrics and its Library health and
Workspace preferences surfaces from GET /system/overview: real counts,
"Not checked" until a health check ran, no invented review count, backup
or thumbnail status (D93). Explore opens "Usage statistics" with the real
runs and ratings per category and the Library totals (D96); the
statistics route now also returns rated_images.

Maintenance shows the concept's notice with Run health checks above its
hand-off: the read-only check runs as a registry read job and opens the
"Health report" with a chip per real result, missing files per source
(D95). A load failure shows Retry and no zeros; late answers never
repaint a newer section.

Adds the System concept comparison (shell, Overview, both dialogs, the
notice, mutation checks), the System workflow spec and its fixture
hooks, and the TESTING.md "Native System" section.

* test(system): coverage floors for the native Overview

Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes.

* fix(system): keep health reports per database and off the event loop

- A restore that replaces the database forgets the stored health report,
  and a check that started before the swap is not kept (database
  generation stamped at start).
- The Overview no longer walks every source for the missing-thumbnail
  count; the page reads /images/thumbnails/status on its own.
- Image sources for the health check and the Overview resolve in the
  executor; a failure to start the check answers the coded envelope.
- Any exception fails only its own check; missing files reads its rows in
  the snapshot and stats the files after it ended.
- The read-only URI keeps Windows UNC paths (empty authority).
- An integrity count at SQLite's limit is flagged as capped.

* fix(system): thumbnail row read, tracked health job and guarded host save

- The Overview's thumbnail row is its own read of the thumbnail status:
  "Checking thumbnails…" until it answers, unavailable when it fails.
- The health job is tracked in the Activity store (new store.track) and
  followed through its event stream instead of polling every 150 ms.
- A host save before a read returned a revision fails without a request;
  System reads the host values on mount and after a conflict.
- "100+ problems found" when the integrity count is capped.
- D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone.
- The concept spec builds the health report from the concept's own rows
  and asserts the full unmapped path list; surface tops pin the 18px
  offset.
- TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI
  and the new behavior. Coverage floors raised for the touched files.

* fix(system): make two health tests portable to Windows

- Windows forbids "?" in file names, so the URI-character open test
  uses "#", "%" and a space there.
- ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of
  raising, so the bad-path test raises as POSIX does to reach the
  missing-files failure path on every platform.
2026-10-23 04:59:18 -07:00
Vito Sansevero 4e5c810f96 test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
2026-10-23 04:05:44 -07:00
Vito b50bf862f0 fix(inspect): hold a route change until a starting video settles (#294)
* fix(inspect): keep a replaced video in place until its pipeline has started

WebKit (GStreamer) deadlocks the page when a video leaves the page or the
viewport before its pipeline has started: the main thread changes the
pipeline state and waits for the source's streaming thread, which waits on
the main thread in a synchronous bus handler (native stacks of the hung WPE
web process in the pinned image). Removing, hiding or merely pushing the
video below the fold all hung; none of pause(), removing src, load() or the
revoke was the cause on its own.

A video replaced while still starting now stays where it is, inert and
aria-hidden, with the new preview inserted after it, and the session's
revoke of its URL waits. Once it reports metadata or an error (or after
15 s) it is paused, loses its src, is reset with load() and removed, and
only then is its URL revoked. Every released video is reset with load(), so
no decoder is kept per replaced video.

The loadstart replacement spec now runs on WebKit too (15 fresh pages,
replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause.

* fix(inspect): hold a route change until a starting video settles

A route change within the workspace detaches the Inspect page, and WebKit
(GStreamer) deadlocks when a video still starting is detached. Route
changes at loadstart + 0-7 ms hung 4-5 of 120 fresh pages (14 of 120 with
no page code releasing anything, so the detach itself is the trigger).

Inspect now exposes leaveGuard(): a promise while a video on the page is
still starting (current or replaced). The shell renders the next route
once it settles (metadata, error, or the 15 s bound); the latest route
wins. Route changes at loadstart + 0-7 ms now hang 0 of 120.

Not changed, with numbers in TESTING.md D62:
- A replaced video still shows above the new preview until it settles.
  Same-slot stacking changes the video's own box and hung again
  (absolute 5/120, absolute + opacity 0 8/120, one-cell grid 4/120).
- A real page leave cannot wait. Navigating away hung 8/120 and reload
  22/120 even with no page-side release, so it stays a known limitation.
2026-10-23 03:51:38 -07:00
Vito Sansevero fcf6445de0 fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.

JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
2026-10-23 03:43:21 -07:00
Vito Sansevero 0f3559bedf Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts:
#	py/api/__init__.py
#	tests/test_api_core.py
2026-10-23 03:34:00 -07:00
Vito 4f1c6e236b test(browser): report a slow test reset with every thread's stack (#299)
* test(browser): report a slow test reset with every thread's stack

On #291 (run 39904288262, WebKit shard 2/2) one /__test__/reset
{chronological_history} waited 16 s on its worker's server, with nothing
on the server's stderr. The cause did not reproduce: the same worker's
file sequence (library, library editor, tagging lifecycle, visual) at
2 CPUs three times, and WebKit shard 2/2 at 2 CPUs, kept every reset
under 2 s.

A reset still running after 2 s now reports, while it waits, every
thread's stack (faulthandler), the default executor's threads and queue,
the open connections and WAL size, and on Linux each thread's scheduler
state and kernel wait channel. When it ends it reports the total and how
long each executor call queued and ran. The lines carry a
"pm-slow-reset| " prefix and the worker fixture copies them to the job
log, so the next one shows up even in a test that passes.

The report never reads another thread's connection. Reading even
in_transaction on a connection busy in SQLite blocks on its mutex with
the GIL held: a first version of this report froze the whole server
until the 30 s busy timeout. A test holds a real write lock behind a
reset and checks the server keeps answering while it waits.

* test(browser): name each thread in the slow-reset stacks

Before Python 3.14 faulthandler prints thread ids only, so the report
now lists which name each id belongs to. CI on 3.10 to 3.13 failed the
unit test that looks for the blocked thread by name.

* test(browser): keep the slow-reset summary on a coarse clock

On Windows the report timer can fire a few milliseconds short of the
threshold, and the summary was then dropped. A reset that was reported
now always gets its summary. The server test uses the plain reset, which
the Windows jobs can seed, and puts the server stderr in its failure
messages.
2026-10-23 03:24:20 -07:00
Vito 5b2c21bdaf test(annotations): capture the details panel after it has loaded (#292)
"Tag opens the dialog without selecting the card" read the panel while it
still said "Loading file details…", then compared it with the same card's
finished details. On a slow WebKit runner the read raced the load.
2026-10-23 03:03:38 -07:00
Vito Sansevero 7f1b5b64bd fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
2026-10-22 17:42:01 -07:00
Vito Sansevero f00933ee16 refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
2026-10-22 17:41:15 -07:00
Vito Sansevero beaba9ae84 fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
2026-10-22 17:41:14 -07:00
Vito Sansevero ab47d7529b fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.

A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
2026-10-22 17:38:09 -07:00
Vito Sansevero b84e6ea111 fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
2026-10-22 17:37:00 -07:00
Vito Sansevero fe5a0017a9 fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
2026-10-22 17:35:06 -07:00
Vito Sansevero d24f3f989f fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.

Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
2026-10-22 17:33:46 -07:00
Vito Sansevero 32cc071d07 fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.

Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.

The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
2026-10-22 17:29:15 -07:00
Vito Sansevero c4ac90e9ec Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend 2026-10-22 17:22:15 -07:00
Vito Sansevero ed392e77e2 fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
2026-10-22 16:50:46 -07:00
Vito Sansevero 3e20c533fa feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:

- GET /prompt_manager/review/items lists pending items oldest first with
  the exact pending total and an HMAC keyset cursor bound to the storage
  epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
  label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
  for a capture of the item's bytes that the current file still hashes to
  and whose evidence is unchanged, through the ingester's compatible-only
  rules in the same transaction as the decision row; a conflict writes
  nothing. Leave unresolved records the decision and links nothing.

ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.

Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
2026-10-22 16:27:49 -07:00
Vito Sansevero d25c2cb4f2 Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend 2026-10-22 16:09:06 -07:00
Vito Sansevero f3366a13f8 feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.

Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
2026-10-22 15:56:58 -07:00
Vito Sansevero de6af76064 feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
2026-10-22 15:33:26 -07:00
135 changed files with 21072 additions and 596 deletions
+603 -12
View File
@@ -50,6 +50,13 @@ stdin open and the server (`--exit-on-stdin-eof`) shuts down cleanly, removing i
temp root, when that pipe closes, so a worker killed without teardown leaves no
server behind. A test that fails gets the stderr its server wrote during the test
attached to the report as `server-stderr` (an aiohttp traceback behind a 500).
A `/__test__/.../reset` still running after 2 s writes a report to the job log,
passing test or not: every thread's stack taken while it waits, the default
executor's threads and queue, the open connections and WAL size, then the total
and how long each executor call queued and ran. Its lines start with
`pm-slow-reset| ` (`tests/browser/slow_reset_watch.py`). The report never reads a
connection another thread is using: one that is busy inside SQLite would block
that read with the GIL held and freeze the whole server.
The coverage run can be split the way CI splits it, and each shard keeps the raw
V8 files it writes under `coverage/js/raw`:
@@ -114,6 +121,13 @@ npm run --prefix tools/vendor test:comfyui
CI performs this setup automatically. `PM_COMFY_URL` can select another
`http://127.0.0.1:PORT/` address (default port 8189). This is an owned managed
server: a running host is never reused. Do not point it at your live checkout.
One port serves one run. The launcher refuses a port that is already in use and
reports ready only when `/system_stats` names its own base directory, so two runs
started together on 8189 fail at startup ("already in use" or "answered by a
ComfyUI host this run did not start") instead of one run testing the other's
host and losing it mid-test (`ECONNREFUSED`, `ERR_CONNECTION_REFUSED`, Network
Error). Every test checks the answering host again. Agents sharing a machine
give each run its own `PM_COMFY_URL` port.
The launcher verifies the upstream SHA and clean tracked files, snapshots only
tracked PromptManager files, and creates temporary user/input/output/database/
configuration/log directories. Personal untracked files are excluded. It runs
@@ -378,7 +392,8 @@ with source `zeroed-conditioning`. `easy ifElse` with a literal boolean follows
branch only; with a linked boolean both branches are traced, equal texts resolve and
different texts make the sampler `ambiguous` with one candidate per choice (at most
32, else `work-limit`). ConditioningCombine/Concat/Average resolve only when every
part has the same text, otherwise `unresolved` with `composite` naming the node.
part has the same text, otherwise `unresolved` with `composite` naming the node
(but see the added-conditioning floor below for a later stage).
Results that depend on a forced choice are never memoized. A `FaceDetailer`
(`detailer-stage`) is a sampler stage: the samplers reached through its `image`
ancestry are compared with it. Equal texts resolve with the detailer as the sampler;
@@ -389,9 +404,65 @@ A sampler's latent ancestry is walked through every link input of non-sampler
nodes within the same visit and depth budgets: with no earlier sampler it is a
single stage whatever the origin (VAEEncode, EmptyLatentBatch, EmptyFlux2LatentImage,
EmptyHunyuanLatentVideo, SetLatentNoiseMask, InpaintModelConditioning or
WanImageToVideo slot 2). An earlier sampler (img2img from a decode, hires fix) stays
`unresolved` "Multistage sampler ancestry is ambiguous" until stage texts are
compared; an unlisted node with sampler-shaped inputs (positive, negative, guider,
WanImageToVideo slot 2). Pass-through lineage (#304 follow-up, maintainer option B,
2026-10-23): the table's `lineage` section lists the outputs whose image or latent
continues one input's (`VAEDecode`/`VAEDecodeTiled` `samples`, `VAEEncode`/
`VAEEncodeTiled` `pixels`, `ImageUpscaleWithModel`, `ImageScale`, `ImageScaleBy`,
`ImageScaleToTotalPixels` `image`, `LatentUpscale`/`LatentUpscaleBy` `samples`,
cg-image-filter `Image Filter` slot 0 `images` and slot 1 `latents`; a selection of
the same batch), plus Reroutes. Lineage describes no prompt, so its classes are not
claimed by a prompt category. When the latent walk finds exactly one earlier sampler
and the latent descends from that sampler's output slot 0 only through lineage
nodes (a hires-fix or refiner chain, img2img from an in-graph decode, a direct
latent link), the two stages are compared like a detailer and its upstream stage:
equal role texts resolve with the later sampler named, different texts are
`ambiguous` `multistage-differs` with one candidate per stage, and an unreadable
earlier stage makes the chain `unresolved` with that stage's reason. Unanchored, the
earlier sampler is judged inside the chain when its output reaches no other file
output. A size or resolution node reading the same image beside the lineage (a
`ResolutionCalculator` feeding `ImageScale` width and height) is allowed, since the
walk then still finds only that sampler. Any other earlier sampler, or one reached
only through an unlisted image node, stays `unresolved` "Multistage sampler ancestry
is ambiguous". Added-conditioning floor: a later stage whose own positive fails only
as `composite` keeps its earlier stages' shared pair ("A later stage adds
conditioning to the base prompt", naming the earlier stage's sampler) when that
positive reaches, through Reroutes and conditioning pass-throughs, a composite the
table marks `additive` (ConditioningCombine, ConditioningConcat; never
ConditioningAverage, which can weight a part away), one of its parts is exactly the
earlier stages' positive text, and its negative text equals theirs. This is the
three-stage Flux run whose UltimateSDUpscale combines the PromptManager prompt with
a style-trigger encoder: legacy linked the PromptManager text and every stage is
conditioned on it. The positive is always labelled (maintainer ruling): its source is
`added-conditioning` (D126; with a D125 omission too,
`runtime-parts-omitted-and-added-conditioning`) and its provenance names the
composite (`added_by`), whatever the added text is; no length heuristic. A combine without the base text, a different negative, or
different earlier stages stay as they were (`composite` or `multistage-differs`).
An image-edit batch (several samplers with distinct prompts, one shared saver
prefix) stays `ambiguous`, and text generated at run time stays untraceable
(`tests/test_resolver_pass_through.py` on synthetic shapes from
`tests/passthrough_graphs.py`; the browser port is compared on the named
`select-*` chain cases and `passthrough_fuzz_cases`). Ingestion keeps the default
profile's historical latent rule (D1), so the admin scan still reaches these files
through its legacy-parser fallback. Run-time PromptManager parts (maintainer option
A, 2026-10-23): LoraManager's `TriggerWord Toggle` is a `runtime` class (its output is
the LoRA's trigger words as filtered at run time). A PromptManager (or
PromptManagerPair) `prepend_text` or `append_text` linked directly to a `runtime`
node is left out: the text is the node's own, combined with any other literal or
traced part exactly as before, the side's source is `runtime-parts-omitted`, and a
resolved pair's reason is "Text added at run time, such as trigger words, is not
included". The prompt block shows that line under the text (D125; the evidence
badge stays "Workflow inferred"). The label follows the text through encoders,
concatenations and chained PromptManagers: a side is labelled when any node on its
traced path left a part out, and its provenance lists those PromptManager nodes
(`omitted_parts`). Comparisons include what a text leaves out, so a prompt that left
out a part never equals a complete one (a branch with the toggle and one without stay
ambiguous; such stages are `multistage-differs`). Only classes in the table's
`runtime` list are left out, never another `runtime-value` source (a JoinStringMulti
with a linked list flag stays `runtime-value`). The toggle's output is never computed. A run-time node further upstream (inside a
concatenation), a run-time main `text`, or an unknown feeder keep their outcome
(`runtime-value` or `unsupported-node`). This is what legacy linked for the June
hires-fix runs (`TestRunTimePromptManagerParts`, the `select-runtime-*` parity
cases); an unlisted node with sampler-shaped inputs (positive, negative, guider,
latent_image, a pipe, video embeds, custom-sampler parts, or a `model` link together
with conditioning, neg_conditioning, samples or latent) is `unsupported-node`
naming it, here and in a detailer's image ancestry; a latent that is not a link or
@@ -1677,8 +1748,20 @@ mapped differences"):
A video replaced while still starting (a source, but neither metadata nor an error yet)
stays where it is, inert and `aria-hidden`, with the new preview inserted after it, and
the session's revoke of its URL waits; once it reports metadata or an error (or after
15 s) it is released as above and only then is its URL revoked. Page leave and dispose
release it at once. Cause, found with native stacks of the hung WPE web process in the
15 s) it is released as above and only then is its URL revoked. For that moment (the
time to metadata, or at most 15 s for a video that never settles) the old video and the
new preview both show, the new one below it: a known, brief overlap. Stacking them in one
slot instead hung again, because it changes the video's own box: the old video
absolutely positioned in its own box hung 5 of 120 pages (8 of 120 with `opacity: 0`),
and a one-cell grid 4 of 120. Only `opacity: 0` on the video, left in the flow, did not
hang (0 of 120), and alone it would leave an empty hole above the new preview. A route
change within the workspace waits the same way: the shell renders the next route once a
video still starting on Inspect settles (`leaveGuard`; the latest route wins). Route
changes at `loadstart` + 0–7 ms hung 4–5 of 120 pages before and 0 of 120 after.
A real page leave cannot wait. With no page code releasing anything, navigating
away hung 8 of 120 pages and reload 22 of 120 (3 and 8 of 120 with the page's own
release), so that WebKit teardown stays a known limitation; dispose still releases at
once. Cause, found with native stacks of the hung WPE web process in the
pinned image: WebKit (GStreamer) deadlocks when a video leaves the page or the viewport
before its pipeline has started. The main thread changes the pipeline state and waits
for the source's streaming thread, which is itself waiting on the main thread in a
@@ -1689,8 +1772,9 @@ mapped differences"):
`loadedmetadata` hung 0 of 180. Specs: `replacing a video that just started loading
never blocks the page`, and `a video replaced while still starting is reset once it
settles`, which now runs on WebKit too (15 fresh pages, replaced at `loadstart` + 0–4
ms). The viewer's own video teardown was checked the same way (`closing the viewer on a
video that just started never blocks the page`, 15 fresh pages on WebKit) and needed no
ms), and `leaving for another route while a video starts waits for it to settle`. The
viewer's own video teardown was checked the same way (`closing the viewer on a video
that just started never blocks the page`, 15 fresh pages on WebKit) and needed no
change.
- **D63** (status line): one `p.small.spaced-16` in `#local-preview`, for states the
concept never shows: "Drop one file at a time.", "Folders cannot be inspected. Choose
@@ -1796,6 +1880,337 @@ npm run --prefix tools/vendor test:workflows -- inspect-concept.spec.mjs workspa
npm run --prefix tools/vendor test:comfyui -- inspect-sender.spec.mjs
```
### Native System (#249 System)
`#/system/<section>` is a native workspace route rendered by `web/js/workspace/system.mjs`
on the concept's `renderSystem` markup (app.js a698–706; `concept.css` 709–895). `#/system`
is Overview under the sidebar's own hash; `#/system/overview|appearance|sources|
integrations|tagging|maintenance|backups|diagnostics` select a section, and an unknown
child (the concept's `#/system/general`) is replaced by `#/system/overview`. The
production hub `#/overview` is unchanged (D28); its System card opens
`#/system/overview` (Q12). Switching sections keeps the shell, the drafts and focus on
the pressed section button and disposes the previous section's scope, so its reads,
listeners and dialogs never outlive it.
Drafts (`system-state.mjs`) live for the page session (Q10). Save writes the dirty
browser fields, then the host fields (`POST /prompt_manager/settings` with
`expected_revision`, #220), then the runtime fields. A failed domain keeps its drafts and
highlights their controls; `settings-changed` is a conflict and keeps the draft. System
reads the host values on mount and again after a conflict; until a read returned a
revision, a host save fails without a request, so it never sends a blind
`expected_revision: null` (which S1 merges into the file on disk). Save stays disabled
until a section has an unsaved change.
Appearance (`system-appearance.mjs`, S3) is the concept's "Browsing & display" surface
on real values. Its browser rows are this browser's display defaults in the legacy
gallery's own `gallerySettings` object (`web/js/workspace/preferences.mjs`, Q3): Default
layout and Thumbnail size (`workspaceLayout` grid/list/masonry, `workspaceDensity`
comfortable/large/compact, new keys), Load metadata on selection (`autoLoadMetadata`),
Cache metadata in this browser (`cacheMetadata`), Image loading quality (`imageQuality`:
Optimized→`medium`, Original→`high`, Fast→`low`; the thumbnail dialog's preselected quality
once S5 lands), the three video rows (`videoAutoplay`, `videoMute`, `videoLoop`) and Show
file paths (`showFilePaths`). A write re-reads the stored object and writes back only the
changed keys, every other key (legacy paging, thumbnail state, unknown keys) as stored;
the legacy gallery's Save also keeps the keys it does not edit, so the two pages stay in
sync. The workspace never reads the legacy paging keys (`defaultLimit`, `gridColumns`,
`infiniteScroll`; README continuous-browsing ruling). A browser that refuses storage
(denied, or a full quota that leaves the stored bytes as they were) keeps the change for
this tab only and the status says "Saved for this tab only; browser storage is
unavailable.", never "Saved in this browser.". Another tab's change (a `storage` event)
repaints the clean controls; a draft it overtook stays, highlighted, with the conflict
status. Open from ComfyUI (New tab→`newtab`, Popup window→`popup`) and Node search
auto-hide (seconds) (a whole number 0–300; anything else is highlighted and never sent)
are the server's `webui_display_mode` and `result_timeout`, saved through
`POST /prompt_manager/settings` with the revision. The consumers read the defaults on
their next visit, never the current view: a new Images visit starts from the layout and
size (the last visit keeps its own, a saved view opens at Grid and Comfortable as
before); with metadata off a selection reads nothing until "Load metadata"; with the
cache off selecting the shown file again re-reads it (on, the one current record is
reused); paths off leaves out only the Path row of File & provenance; the viewer reads
the video defaults.
Sources & scans (`system-sources.mjs`, S4) is the concept's "sources" body on the
server's sources. `GET /prompt_manager/settings` reports them (public paths only):
`sources.configured` (each `gallery_root_paths` entry with `valid`, `code` and the
server's `reason`), `sources.effective` (what is scanned and served, with its
`root_index` and `watch`: `watching` only when the image monitor's observer is alive,
scheduled that folder and monitoring is on, `stopping` when it still watches it but
monitoring was switched off, `manual` when monitoring is off or no monitor runs, else
`restart_pending`), `sources.auto_detected`, `monitor` and `settings.monitoring_enabled`.
`POST /prompt_manager/settings/validate-source {path}` checks one folder and writes
nothing. Every caller (validate-source, the settings save, config.json at load) refuses
a source through the same checks (`sources.source_problem`): its form first, then
containment decided as text (`normcase(normpath)` against the allowed folders' own
spellings), before anything resolves, stats or opens the path, so a UNC share, the
`\\?\` and `\\.\` namespaces, a device or any folder outside the allowed ones never
reaches the filesystem or the network (no SMB request, no existence or timing oracle).
Only then is the path resolved and contained again, which catches a symlink out. The
filesystem root and the home directory answer `outside-allowed` with the same reason as
any outside path. ComfyUI folders outside its base directory (`--output-directory`)
are path anchors too, so their sources keep a relative public path
(`renders/portraits`) that round-trips and never collides by folder name. A save whose
list holds a source the server cannot use is a 400 `invalid-source` naming its public
path (`source`, as text: a refused path is never resolved to name it), its `index` and
`reason`, and saves nothing. A save that changes the sources (compared as folders, not
spellings) holds the registry as `records` ("Source change"), so it is refused with
409 `busy` while a scan runs. `POST /prompt_manager/jobs/scan {scope, roots, paths}`
scans only the given effective `root_index`es (strict ints, at most 64), each with the
public path the page read for it; an index past the list or one that now names another
folder is 409 `sources-changed` and starts nothing (the page toasts the server's
reason and reads the sources again); the legacy `/scan` is unchanged. The section
lists one row per source ("<path> · watching | watching until restart | manual scan |
restart pending" with "Scan changes" for a watched source, else "Scan folder"), which
opens the shared "Review metadata scan" for that one source ("1 output folder") and
starts the session job over it; the job opens in Activity, and the toast only says it
started. Enrich
existing metadata and the notice's Review scan review every source. The notice shows
until this browser dismissed the 3.2.0 update notice (`pm_update_notice_dismissed`,
the legacy admin page's key); Later dismisses it without scanning, and Review scan
dismisses it as the legacy Scan does. "+ Add source" asks validate-source, then adds
the folder to the host draft of `gallery_root_paths`; Remove takes one out; "Watch for
new outputs" (`monitoring_enabled`) and Worker threads (`worker_threads`) are host
drafts too, and Save writes them with the revision. A save the server refuses because a
scan runs toasts "Another job is running: <title>." with a link to that job (D57) and
keeps the drafts; a save refused for one source says the server's message as the status
and marks that row. After a save the source list is read again; a monitoring change
waits for a restart (the running observer keeps watching until then, and the status
says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's
Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5).
A list that cannot be read shows "Sources could not be loaded. Your saved data has not
changed." and Retry, never an empty list.
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
visit through `latestOnly()` on the section scope; a late answer after the section left
paints nothing. Until it answers, the section shows "Loading the overview…"; a failure
shows "The overview could not be loaded. Your saved data has not changed." and Retry,
never zeros. The thumbnail row is a separate read of `/images/thumbnails/status` (a
walk of every source), so the other values never wait for it: "Checking thumbnails…"
until it answers. The health check (`POST /prompt_manager/system/health`) is a `read`
registry job of kind `health`: one at a time, beside a scan, not cancellable and with
no file outcomes. The page tracks the started job in the Activity store (its event
stream; the store marks it lost if the server stops listing it), never by polling. A press while one runs (another tab) waits
for that one. The last report lives for this server session (`GET
/prompt_manager/system/health`) and describes the database it was taken from: a
restore that replaces the database forgets it, and a check that started before the
swap is not kept. The check's own connection opens a `mode=ro` URI that keeps a Windows
UNC path (`file:////server/share/prompts.db`; pathlib's `file://server/share/...` is a
remote authority SQLite refuses). A check that raises fails only itself, and the
missing-files check reads its rows inside the snapshot but stats the files after it
ended, so a slow source never pins the WAL. Statistics
read `GET /prompt_manager/system/statistics` (`by_category`, the legacy `/stats` totals
and `rated_images`).
Read-only standard for a check or preview (S2's health check; S6 and S8 previews use
the same): the database and `-wal` bytes are unchanged, the `-shm` size is unchanged,
the connection's `total_changes` is 0 at close and the data reads back unchanged. The
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
reader can keep it byte-identical (`test_health_check_never_writes`).
Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned):
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
configuration." / "Applied until the server restarts.", "Some changes were not saved.
Review the highlighted settings.", "Saved for this tab only; browser storage is
unavailable." and "Settings changed elsewhere. Reload this section to see them."
"Save preferences" is `aria-disabled` (opacity .55, `not-allowed`) while clean or
saving; a setting the last save could not save gets a red outline. CSS under D92.
- **D93** (Overview): real values. The metrics are the Library's eligible positives
and active saved negatives and the indexed `generated_images` rows (Q2). "Database
integrity" is "Not checked" until a check ran this session, then "N issues" (checks
that found issues, even beside a check that could not run), "Passed" or "Incomplete".
The review row is "N ambiguous image links" (the Needs review queue's pending count,
`needs_review.pending`) with the concept's line; its Review metadata action is
`aria-disabled` (CSS under D93) until the Needs review page ships (#288). A count
that is not an integer reads "Ambiguous image links" / "Not available yet." (Q6).
"Backup status" reads "No retained backup record." until verified backups exist (S8).
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
missing thumbnails", "N missing thumbnails" (the pending count of
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid
and density settings are shared across views." (no paging). The sources row is
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
by "M of N configured sources available." or "None configured is available; ComfyUI's
output folder is used." when that is the case.
- **D94** (all sections): a section not delivered yet shows the hand-off panel "Your
current tools are ready" with a link to each legacy tool that still owns it (the
plan's legacy coverage table); retired section by section, gone after S12.
- **D95** (health report): the concept's `health-check` dialog titled "Health report",
with no sample note. It lists every check in the server's order: Integrity check,
Foreign keys & pair references, Canonical hashes, Pair uniqueness (added), Missing
files, Ambiguous metadata, Consistency (added). Missing files has one row per image
source when there are several ("Missing files · output"). Each chip is the real
result: "Passed", a count with what was found ("2 broken references", "1 link needs
review", the concept's "N links need review" and "N images need review"; "100+
problems found" when SQLite's integrity check stopped at its limit), "Not
available" (an unavailable source), "Not available yet" (Ambiguous metadata, Q6) or
"Could not check". A `p.small` after the list says when it was checked and whether
every check passed, some could not run, or issues need review. Close / Review missing
links (to Maintenance, its hand-off until S7). Nothing is changed by a check.
- **D96** (Statistics): "Usage statistics" without the sample note: the concept's
"Runs by category" table (Category / Runs / Avg. rating) with the real runs and
average rating of eligible parents per category ("Uncategorized", "Not rated"),
then a second table "Library totals" (Total / Count: Prompts, Tags, Images from the
legacy statistics, and Rated images: indexed images whose linked prompt is rated).
The Overview row says "Runs and ratings by category, with library totals." because
that is what the dialog shows (the concept's own dialog shows only "Runs by
category"); per-prompt run counts, prompt tags and dated runs are recorded, but no
table of them exists yet. A failed read toasts "Statistics could not
be loaded. Your saved data has not changed."
- **D121** (production hub): the Overview hub's System card reads the concept's System
heading and "Open System"; the concept has no hub card.
- **D122** (shell): the section nav's `aria-label` is "System sections"; the concept's
literal label is its prototype action name "system-tab sections".
- **D123** (Maintenance): the concept's Maintenance notice
("Check first. Review changes. Then apply." + Run health checks) is delivered ahead of
the rest of Maintenance, above its D94 hand-off, as the native place that runs the
health check. A press shows "Checking…" (`aria-disabled`) and opens the report; a
failed check toasts "The health check could not finish. Nothing was changed."
- **D97** (Appearance): added rows for the legacy gallery settings the concept leaves
out, after its own rows: "Show image details on cards" (`showImageInfo`; off hides a
card's title and file line, Info and Tag stay at the end of the row), "Collapse viewer
details by default" (`sidebarCollapsedByDefault`; the viewer opens with
`#viewer-details` hidden and a "Show details"/"Hide details" toggle with
`aria-expanded`, the stage taking the whole width), and the operation "Reset display
defaults" ("Restore this browser’s display defaults, then save. Other settings are
kept.", button "Reset"), which drafts the gallery's defaults for the browser rows only.
The node search timeout shows the server's real value, not the sample 30. CSS under D97.
- **D98** (Sources): each configured source's row has a plain Remove after its scan
button; nothing changes until Save. A source added this session reads "<path> · not
saved yet" with Remove only until Save; a configured source the server cannot use
reads "<path> · <server reason>" with Remove only. A line under the list says when the
list is not the whole story: "No source is configured. After saving, ComfyUI’s output
folder is used." (the last one removed), "No source is configured; ComfyUI’s output
folder is used." and "No configured source is available; ComfyUI’s output folder is
used." (auto-detected rows, listed without Remove). "restart pending" is the third
watch state. The operation "Refresh files" ("Read every source again to find files
added or removed outside ComfyUI.", button "Refresh") closes the surface: it calls the
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
folders." or "Files could not be refreshed. Try again.". The concept's "sample path"
note is prototype copy and is dropped.
- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the
concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's
sample 4).
- **D127** (Sources): the concept's `add-source` `simpleInput` with production
copy: the field is "Folder path" (the concept's "(sample only)" dropped), it starts
empty (not the demo "/images/new-collection"), and the action is "Add" (not "Apply in
prototype"); Enter in the field adds too. `add source dialog matches the concept`
(system-concept.spec) compares the open dialog with these three mapped, and `add
source comparison detects mutations` proves the mapping is the only difference. A
folder the server refuses keeps the dialog open with the field
`aria-invalid` and the server's reason in a `p.small[role=alert]` above the actions;
a folder the server names as one already listed says "This folder is already a
source.", and a check that cannot finish says "The folder could not be checked. Try
again."; neither adds anything.
The concept dialog does no check, so `invalid source shows the server reason`
(workspace-system-sources.spec) pins it.
- **D128** (Sources): states the concept's sample rows never show. A source the
running monitor still watches after "Watch for new outputs" was saved off reads
"<path> · watching until restart" (still "Scan changes"), and the save's status adds
"Restart ComfyUI to apply: Watch for new outputs." from `restart_required_fields`. A
configured source the effective list holds under another spelling reads "<path> ·
same folder as another source" with only Remove. A save refused for one source
(`invalid-source`) marks that row (`aria-invalid`, outlined like a D92 setting) and
the status is the server's "<path> can't be used: <reason>. Remove it or reconnect it,
then save." ("Remove it, then save." when reconnecting cannot help). Pinned by
`Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save
refused for one unusable source names it and marks its row`
(workspace-system-sources.spec) and the system-state and system-sources node tests.
- **D124** (inspector): with "Load metadata on selection" off, the Images panel and
the viewer's details show "Metadata loads when you ask for it." and a "Load metadata"
button instead of reading the file. The concept has no such state, so no concept
comparison renders it: `metadata off shows an explicit load and loads once on request`
(workspace-system.spec) pins it instead.
- **D125** (prompt blocks, maintainer option A): a resolved prompt whose source is
`runtime-parts-omitted` (a PromptManager `prepend_text` or `append_text` fed directly
by a listed run-time node, such as LoraManager's TriggerWord Toggle, was left out)
shows the muted line "Text added at run time, such as trigger words, is not
included." under its text. The evidence badge stays "Workflow inferred"; the line is
never sent or copied. The concept has no such state (`a prompt that left out run-time
trigger words says so under its text (D125)` in `tests/js/prompt-blocks.test.js`).
- **D126** (prompt blocks, maintainer ruling on the added-conditioning floor): a
resolved positive whose source is `added-conditioning` (a later stage combined more
conditioning with the base prompt, as UltimateSDUpscale's ConditioningCombine of the
PromptManager and a style trigger) shows "Conditioning added at a later stage is not
included." the same way. A prompt with both shows both lines, D125 first (source
`runtime-parts-omitted-and-added-conditioning`). Always labelled, with no length or
"short trigger" heuristic (`a prompt a later stage added conditioning to says so, and
both lines show together (D126)`).
Appearance scope and states. The Default layout and Thumbnail size defaults start the
Images page only (a new visit, not the current view or a saved view); the embedded
galleries (a prompt history's Shared gallery and its All images / Repeat run dialogs)
have no toolbar, so they keep Grid/Comfortable, while the metadata, cache and card
defaults apply there too. A host row whose `/settings` read failed stays disabled with
"Server settings could not be read."; a rejected timeout says "Some changes were not
saved. Review the highlighted settings." A display default another page saved under one
of this tab's drafts flags that draft with the conflict status, whether it arrived while
System was mounted (a `storage` event) or while it was not (checked again on mount, and
before a write); the draft is never written over the other page's value without that
status first. A change kept for this tab only is stored by the next write storage accepts.
Open question (saved-views follow-up, M8): a saved view stores no layout or size and
always opens at Grid/Comfortable, even when the Default layout is Masonry. Should saved
views store a layout, or follow the default?
`tests/browser/system-concept.spec.mjs` compares the shell and Overview with the concept
(dark and light, 1600×1100 and 390×844), mapping only the deviations above; unmapped,
the difference is exactly the D92, D93, D96 and D122 paths. Each surface's `top` is
measured, which pins the concept's `.surface + .surface` 18px offset that concept.css
also applies inside `.two-col` (copied as is). `statistics and health dialogs match
the concept` maps D95 from the concept's own five rows (labels verbatim): it inserts
Pair uniqueness after Canonical hashes and Consistency last, splits Missing files per
source when there are several, replaces only the chips with the fixture's expected
results, and asserts the full unmapped path list; it maps D96 and compares the
Maintenance notice unchanged. `system comparison detects mutations` removes a row,
renames a metric label, swaps the `--line` token, recolours a chip, renames a check,
removes the totals table and renames a caption. `appearance matches the concept` maps
only D97 and the real timeout (unmapped, exactly those paths differ), and `appearance
comparison detects mutations` removes a row, renames a label and an option and swaps
the `--line` token. `sources matches the concept with|without the notice` (both
themes and sizes; the fixture's monitor watches the first source, as the concept's
first sample is watched, and the concept's Later removes its notice) maps the real
sources for the two samples, D98's Remove and Refresh files, and D113; unmapped,
exactly those paths differ. `sources comparison detects mutations` removes the notice
and a source row, renames a label and the Add source button and swaps the `--line`
token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`,
`{concept,production}-appearance-{dark,light}-{1600,390}.png`,
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
the health check and metric 4, a failed check, the thumbnail row against the status
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
route, a load failure with Retry and late answers after leaving, and Appearance: new
visits versus the current view and a saved view, the embedded gallery keeping Grid,
metadata off and on request, the cache (counted up to a request barrier), hidden paths,
the video defaults, the host fields with their revision, a stale one and a failed read,
reload and a second tab, a second tab's save while System was closed, storage refused,
the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers the
notice's Later and Review scan, a scoped scan from one source into Activity, Add source
with its server check and Save, a refused folder, Remove down to auto-detect, an absent
configured source, the three watch states, the monitoring and worker rows, Refresh
files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
failures, an absent source, a deleted linked file, unthumbnailed files, a fake image
monitor) on the
concept-shaped fixture, with delays through the shared request holds; reset also restores
the server's host settings and its `config.json`, which must be the one inside the server's
temp root (`tests/test_workspace_system_fixture.py`).
```sh
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library \
tests.test_workspace_system_fixture -v
node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \
tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \
tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \
tests/js/workspace-system-sources.test.js
python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
workspace-system-sources.spec.mjs \
--project chromium --project touch --project firefox --project webkit
```
## Workspace data fields for the concept pages (#249)
These fields are additive. The legacy `/prompt_manager/images/output` listing (no
@@ -1837,8 +2252,24 @@ and skip the parse.
integers past 2**53 as their exact digits, so the card and the inspector read
the same digits. The parse runs after the file handle closes. A record over
2 MiB (UTF-8), a parser error, a recursion error, a placeholder, or a
non-integral, boolean or non-finite seed gives null. A card never shows a seed
that differs from the inspector's
non-integral, boolean or non-finite seed gives null. When the prompt graph's
sampler stages read different seed inputs and the inspector's resolution names
one sampler for a resolved pair (a detailer or pass-through chain, whose facts
are the named stage's, D19), the card compares that stage's seed (read by the
same parser, the other samplers hidden from it) with its own and shows no seed
only when they differ. A seed that agrees, or one the named stage does not make
known, is kept, as on main (a saved sampler beside a preview-only one keeps its
seed; an ambiguous chain, whose facts merge and fill from the parser, keeps it).
Only such graphs are resolved for the card, with `INSPECTION_PROFILE`
(`test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed`,
`test_a_seed_that_agrees_with_the_named_stage_is_kept`,
`test_a_named_ambiguous_chain_keeps_the_parser_seed`; a literal named seed needs no
second parse, and stages sharing one linked seed skip the check. Measured on a
200-card page of about 54 KB graphs, half multi-stage with per-stage seeds: 618 ms
cold against main's 387 ms, about 1.2 ms more per such card, paid once per file
through the header-facts cache below;
before this rule a KSampler then FaceDetailer chain with two seeds broke the
contract below). A card never shows a seed that differs from the inspector's
(`test_card_seed_equals_inspector_seed_when_both_present`, which `JSON.parse`s
the listing in Node and compares `String(seed)` with the inspector's seed,
including one past 2**53). The card shows no seed for A1111 `parameters` files
@@ -1954,6 +2385,162 @@ inspection's own stable stat (`tests/test_api_workspace_images.py`
browser fixtures clear `media_image_tags`, `media_annotation_bytes` and
`media_annotation_files` on reset and never the metadata row.
Review queue schema (#249, `database/review_schema.py`,
`tests/test_review_schema.py`): the group `media_review_metadata` (one row:
schema version, `review_index_revision`), `media_review_items` (one ambiguous
resolver result per `(byte_id, resolver_version, evidence_hash)`, citing an
annotation byte record `ON DELETE RESTRICT`, with the file's recorded
`spelled_path` (the locator is case-folded on Windows, a listing target is
not), 0..16 candidates, `limited`,
`state` pending/selected/unresolved and a revision) and `media_review_decisions`
(one row per item decision revision; `selected` exactly when a
`candidate_hash` is present; `pair_id`/`image_id` navigation is set to NULL,
never cascaded, when the pair or image is deleted, through
`idx_media_review_decisions_pair`/`_image`). Evidence is canonical JSON
(`canonical_evidence`: one object, sorted keys, no whitespace, ASCII only, no
NaN/infinity, at most 8 MiB) and `evidence_hash` is its SHA-256. The group is
valid only beside a valid annotation group (a review group without annotation
tables is refused, catalog only, before the annotation install), installs like
the annotation group (optional in old backups, required once current under
`require_library_state`, SAVEPOINT install validated before and after), and
validation re-canonicalizes every evidence document in one streamed pass. A
startup runs that pass once (`test_startup_validates_stored_evidence_once`; the
install trusts the annotation group it has just validated, and the SQL row
predicates check type and size only, because the streamed pass proves the
rest): 2.5 s to open a database of 40,000 items with 4 KiB of evidence each,
was 7.7 s with two passes. Restore and `verify_database_file` keep the full
validation, annotation group included. The browser fixtures clear
`media_review_decisions` and `media_review_items` before the annotation tables,
never the metadata row.
Needs review items (#249, `utils/review_capture.py`, `database/review.py`,
`tests/test_review.py`, `tests/test_review_outcome.py`,
`tests/test_review_choosable.py`): a file is queued only when its final outcome
is ambiguous and unlinked and there is something to choose. Only scans produce
items; the monitor links new outputs and clears pending items it resolves
(maintainer rulings, 2026-10-23). The output scan (Activity's
Scan for metadata and the legacy `/scan`) hands `record_ambiguous_file` a file
the ingestion resolver finds `ambiguous` only when the browser-parser fallback
then links nothing (status `ambiguous`) and no saved row links the file
already; an ambiguous file a saved row links is reported `unchanged` ("Already
linked to its prompt"), not Needs review. A file the fallback links, an exact
pair the ingester stores, and `conflict` and `unresolved` files are not
queued. Neither is an ambiguous file with nothing to choose: fewer than two of
its candidates are `resolved` with non-blank positive text (`reviewable`). With
none the file could not be traced; with exactly one, the untraced branches may
still have made the file, so that text is not certain either. Both report
`unresolved` ("could not be traced"), are never linked, and their pending,
undecided items whose own evidence offers no choice (a limited item counts as
a choice when it holds at least two candidates) leave the queue on the next
scan or monitor pass (`settle_untraced_review_items`). The maintainer's
second live check found 39 items whose 11 candidates were all "unresolved". The queue heals itself: after each batch, the scan removes the pending
items of every listed file (read or skipped as known) that a saved row now
links, and the monitor does the same for each ambiguous output it links
(`settle_linked_review_items`; the review index revision moves). Only an
undecided item goes (`pending`, revision 0, no decision row): a decision is
never deleted or changed, so no schema state was added. The monitor never
queues an output it linked; an unlinked file passed to `record_review_file`
follows the scan's rule (a file the fallback would link is `skipped`). The
maintainer's live check of #284/#288 found the queue at 10,232 against a review
group of 4,677: every fallback-linked ambiguous file had been queued. It captures
the file (scope `needs-review`), reads its metadata again from the capture,
resolves it with the default profile and stores only an ambiguous result,
bound to the capture's byte record (created or refreshed like an observe) and
fenced by the capture's storage epoch. Evidence is the resolver reason plus one
`{sampler_id, state, reason, positive, negative, hash}` per selectable
candidate (nested ambiguous stages flattened, duplicates dropped; each side
`{text, storage_text, provenance}`; `hash` = SHA-256 of the canonical candidate
without it). More than 16 candidates, or evidence over 8 MiB, is a `limited`
item without candidates. Recording is idempotent per `(byte_id,
resolver_version, evidence_hash)` and never touches an existing item or its
decisions; changed bytes add a new pending item and keep the old one. The
capture is released before the write; no model is loaded and no pixel is
decoded. Scan statuses, counters, outcome rows and the legacy SSE events are
unchanged apart from the already-linked and nothing-to-choose cases above; the job's `scope` gains
`review_items` (this scan's files that are in the pending queue after it, new
or already there: the review group's ambiguous rows, less files a capture could
not read or a decision already settled), `review_added` (new items),
`review_settled` (items removed because their file is linked now),
`review_untraced` (items removed because they offer nothing to choose) and
`review_errors`. The log ends with `? N images added to Needs review`,
`? N images already in Needs review`, `= N images left Needs review · now
linked`, `= N images left Needs review · nothing to choose` and `! N images could not be added to Needs review` (failures, e.g.
capture capacity), each only when N > 0. Recording captures in the background: it never takes the store's
last worker or capture slot, which stay free for an interactive Tag dialog or
Apply, and waits up to 2 s for room before counting the file as not added
(`test_background_recording_leaves_a_worker_for_the_tag_dialog`). The monitor's
pass runs after its own link, best effort: a failure is logged and the link
stays. Recording follows the registry's `records`
admission (#289): the scan's recording runs under its own `records` job, and
the monitor's (one file, not a job) is refused while a `records` or `guarded`
job or hold runs. It then records nothing, logs that the file waits for the
next scan, and answers `deferred`; the next "new" scan reads the ambiguous file
again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A
rescan does not copy or hash an ambiguous file whose current size and
`mtime_ns` equal the verified fingerprint of bytes an item already cites (a
stat compare); the file still stays in the default "new" scope, because it is
not a known image. The skip and a duplicate insert record the current spelling
of the locator on its items (`refresh_review_spelling`, the review index
revision moves, item revisions do not), so a case-only rename on Windows or
macOS keeps its item applicable. Residual risks of the skip, like S1's: an
in-place rewrite that keeps both size and `mtime_ns` is never re-queued (Apply
still re-hashes, so it cannot link those bytes), and a file that is deleted or
moved leaves its pending item counted in `total` (its target then formats as
`unavailable` or fails to open, and Leave unresolved clears it).
Needs review decisions and queue reads (#249, `py/api/review.py`,
`database/review.py`, `database/review_decisions.py`,
`tests/test_api_review.py`): `GET /prompt_manager/review/items?limit=1..40&cursor=`
answers `{success, library_epoch, total, items, next_cursor}` with pending items
oldest first; `total` is the exact pending count. The cursor is an HMAC keyset
on `(created_at, id)` bound to the storage epoch and `review_index_revision`: a
malformed one is 400, a well-formed one this process cannot verify (tampered,
or opened before a ComfyUI restart: the secret is per process) is 409
`review-conflict` with the current `total`, one from before a restore is 409
`storage-changed`, one from before any queue change (a decision, a new item, a
refreshed spelling) is 409 `review-conflict`.
Targets are formatted from `spelled_path` against the current roots (the
roots' real paths are the only file access); a file under no current root is
`unavailable` with a null target. Every read and decision lists the roots and
resolves real paths on a worker thread, never on the event loop. `GET .../items/{id}` adds `candidates`
(`hash`, `sampler_id`, `state` resolved/unresolved, `reason`, `label` "<positive
title> / <negative title>" from the saved prompt's Library title else the
first line, and the role texts), `dependency_fingerprint` (SHA-256 of the
file's legacy rows under every spelling of the same physical file, positive
hashes and pair links with their negative hashes, as the ingester reads them) and `sha256`; a detail over 8 MiB, or a
limited item, has no candidates. `POST .../items/{id}/decision
{expected_epoch, expected_revision, capture_token, sha256, dependency_fingerprint,
choice}` with a candidate hash (Apply) requires a capture of the item's file
whose bytes are the item's, re-hashes the current file (`file-changed`
otherwise, also for a fresh capture of new bytes), re-reads the capture and
requires that its bytes are still ambiguous and still offer the chosen
candidate (`review-conflict` otherwise; a resolver change that rewords the
reason or reorders candidates leaves the item applicable, and
`RESOLVER_VERSION` in `utils/prompt_pair_resolver.py` is the one version
ingestion provenance and review share), then in one `_pair_transaction`
checks the epoch, the item revision and state (`review-conflict` with the
current item for a repeat), the re-hashed digest, the dependency fingerprint
and the public ingester's compatible-only rules through
`_ingest_resolved_pair_in` (any refusal is 409 `association-conflict` with
zero writes, explained inline; Apply never reparents a link). Physical
identity: the roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, `subst` or a mapped
drive spells the same file otherwise. Every legacy row whose spelling ends with
the file's name (the link summaries' unindexed suffix lookup) is resolved by
`normcase(realpath)` against the locator, else `samefile`; two rows of the
file refuse the choice, one is enriched under the spelling it was linked
with, and a same-name row the request did not classify is a changed
association (`TestPhysicalIdentity`, symlinked root and a portable fake
junction), and writes
the link with `kind:"user-selection"` provenance (decision id, reviewed digest,
candidate hash, resolver version, role provenance), the decision row, the
item's new state and revision and `review_index_revision`. `"unresolved"`
records the decision only and accepts `capture_token: null`. A successful
decision releases its capture. No decision changes run or usage counts or
calls an execution writer; a Tag save never changes a review item. Errors carry
fixed sentences: `review-failed` / `decision-failed` 500s never echo exception
text or paths.
Capture store (#249, `utils/media_capture.py`, `tests/test_media_capture.py`):
one explicit action captures one rooted media file into an owner-only
`prompt-manager-captures-*` temp directory, streaming it in 1 MiB chunks while
@@ -1970,8 +2557,12 @@ directories are `CaptureDenied`; the locator is
`normcase(join(realpath(root), relative))`, so a symlinked root shares the
listing's identity. Fixed limits: 256 MiB per file (stat and stream), 8
completed captures, 512 MiB reserved, 2 capture/verify workers, 900 s idle,
1800 s hard; admission is nonblocking (`CaptureCapacity`) and lifetimes are
tested with an injected clock. An acquired capture outlives expiry and release
1800 s hard; interactive admission is nonblocking (`CaptureCapacity`), a
`background=True` capture (Needs review recording) never takes the last worker
or capture slot and waits up to `BACKGROUND_WAIT_SECONDS` (2 s) for room, and
lifetimes are tested with an injected clock. The host's on_cleanup close is
final: a recording still in flight gets a capture error instead of creating a
new temp directory after it. An acquired capture outlives expiry and release
until its last use ends. Header-only dimensions (4096 px per side,
16,777,216 px) and video mark Generate unsupported; `verify` re-hashes the
current file without a copy. Every handle closes before unlink (a patched
+15
View File
@@ -122,6 +122,21 @@ try:
_global_prompt_tracker = get_prompt_tracker(_global_db)
_global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
# The Needs review pass for ambiguous new outputs runs through the API's
# capture store (the one the scan and the Tag dialog use): a linked output
# is never queued and its pending items leave the queue. It stops before
# that store is closed at shutdown.
_review_api = globals().get("api")
if _review_api is not None:
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
_review_host = getattr(config.server_instance, "app", None)
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
async def _stop_review_recording(app, monitor=_global_image_monitor):
monitor.set_review_recorder(None)
_review_host.on_cleanup.insert(0, _stop_review_recording)
# Start monitoring
_global_image_monitor.start_monitoring()
_init_logger.info("Global image monitoring system started successfully")
+354
View File
@@ -0,0 +1,354 @@
"""The read-only health check behind System › Overview and Maintenance.
``run_health_checks`` opens its own ``mode=ro`` connection, reads one
snapshot and closes the connection in ``finally``; it never writes. Each
check reports ``passed``, ``issues`` (with an exact ``count``; the
integrity check stops at ``INTEGRITY_LIMIT`` problems and then says
``capped``),
``unavailable`` (it could not be checked here, so ``count`` is None, or
``missing_files`` checked only some sources and found nothing) or ``failed``
(it raised; the other checks still run). ``missing_files`` also reports each
source and whether the check was ``partial``; it reads the image rows inside
the snapshot and checks their files after the snapshot ended, so a slow source
never holds the WAL. At most ``MAX_DETAIL`` detail strings are returned per
check, and none of them carries a server path.
"""
import datetime
import logging
import ntpath
import os
import sqlite3
from pathlib import PurePosixPath, PureWindowsPath
from urllib.parse import quote
try:
from ..utils.hashing import generate_prompt_hash
from .models import normalize_image_path
except ImportError:
from utils.hashing import generate_prompt_hash
from database.models import normalize_image_path
logger = logging.getLogger("prompt_manager.database.health")
MAX_DETAIL = 10 # detail strings returned per check
INTEGRITY_LIMIT = 100 # PRAGMA integrity_check(N): problems SQLite reports
BATCH_ROWS = 100 # image rows read per fetch
REVIEW_UNAVAILABLE = "Needs review is not available yet."
CHECK_FAILED = "The check could not finish."
# Check ids and the labels of the concept's health report, in its order;
# pair uniqueness and consistency are the added rows (D63, now D95).
CHECKS = (
("integrity", "Integrity check"),
("foreign_keys", "Foreign keys & pair references"),
("canonical_hashes", "Canonical hashes"),
("pair_uniqueness", "Pair uniqueness"),
("missing_files", "Missing files"),
("ambiguous_metadata", "Ambiguous metadata"),
("consistency", "Consistency"),
)
_DATABASE_CHECKS = (
"integrity",
"foreign_keys",
"canonical_hashes",
"pair_uniqueness",
"missing_files",
"consistency",
)
def iter_consistency_issues(conn):
"""The legacy consistency issues: tag and image rows of a missing prompt."""
for ref in conn.execute(
"SELECT pt.prompt_id FROM prompt_tags pt "
"LEFT JOIN prompts p ON pt.prompt_id = p.id WHERE p.id IS NULL"
):
yield f"prompt_tags entry references non-existent prompt {ref[0]}"
for ref in conn.execute(
"SELECT gi.id, gi.prompt_id FROM generated_images gi "
"LEFT JOIN prompts p ON gi.prompt_id = p.id WHERE p.id IS NULL"
):
yield f"Image {ref[0]} references non-existent prompt {ref[1]}"
def _result(count, detail, *, status=None):
return {
"status": status or ("issues" if count else "passed"),
"count": count,
"detail": detail[:MAX_DETAIL],
}
def _counted(lines):
"""(count, first MAX_DETAIL lines) of an iterable, consumed once."""
count, detail = 0, []
for line in lines:
count += 1
if len(detail) < MAX_DETAIL:
detail.append(line)
return _result(count, detail)
def readonly_uri(db_path, *, windows=None):
"""The ``mode=ro`` SQLite URI of *db_path*, on this platform or the given one.
pathlib spells a UNC path ``\\\\server\\share\\x.db`` as
``file://server/share/x.db``, a remote authority SQLite refuses to open.
With an empty authority (``file:////server/share/x.db``) SQLite hands
Windows the ``//server/share`` path, which it opens as the UNC path.
"""
if windows is None:
windows = os.name == "nt"
db_path = os.path.abspath(db_path)
if not windows:
return PurePosixPath(db_path).as_uri() + "?mode=ro"
path = PureWindowsPath(db_path)
if ntpath.splitdrive(str(path))[0].startswith("\\\\"):
return "file:" + quote("//" + path.as_posix()) + "?mode=ro"
return path.as_uri() + "?mode=ro"
def _integrity(conn, _roots):
rows = [
row[0] for row in conn.execute(f"PRAGMA integrity_check({INTEGRITY_LIMIT})")
]
if rows == ["ok"]:
return {**_result(0, []), "capped": False}
# SQLite stops listing at the limit: there may be more problems than that.
return {**_result(len(rows), rows), "capped": len(rows) >= INTEGRITY_LIMIT}
def _foreign_keys(conn, _roots):
return _counted(
f"{table} row {rowid} references a missing {parent} row"
for table, rowid, parent, _fkid in conn.execute("PRAGMA foreign_key_check")
)
def _prompt_hash(text):
return generate_prompt_hash(text) if isinstance(text, str) else None
def _canonical_hashes(conn, _roots):
conn.create_function("pm_prompt_hash", 1, _prompt_hash, deterministic=True)
return _counted(
f"Prompt {row[0]}: the stored hash does not match its text"
for row in conn.execute(
"SELECT id FROM prompts WHERE hash IS NOT pm_prompt_hash(text) ORDER BY id"
)
)
def _pair_uniqueness(conn, _roots):
groups = conn.execute(
"SELECT positive_prompt_id,negative_prompt_id,COUNT(*) FROM prompt_pairs "
"GROUP BY positive_prompt_id,negative_prompt_id HAVING COUNT(*)>1 "
"ORDER BY positive_prompt_id,negative_prompt_id"
).fetchall()
detail = [
f"Positive prompt {positive} with "
+ ("no negative" if negative is None else f"negative {negative}")
+ f": {rows} pair rows"
for positive, negative, rows in groups
]
# The surplus rows: what keeping one row per pair would leave out.
return _result(sum(rows - 1 for _p, _n, rows in groups), detail)
def _consistency(conn, _roots):
return _counted(iter_consistency_issues(conn))
def _readable(root):
return os.path.isdir(root) and os.access(root, os.R_OK | os.X_OK)
def _root_prefix(root):
return normalize_image_path(root).rstrip("\\/") + os.sep
def _source_name(root, index):
"""A source's folder name for the report (never its server path)."""
return os.path.basename(str(root).rstrip("\\/")) or f"Source {index + 1}"
def _owner(path, prefixes):
"""Index of the longest source prefix *path* lies under, or None."""
owner, longest = None, -1
for index, prefix in enumerate(prefixes):
if path.startswith(prefix) and len(prefix) > longest:
owner, longest = index, len(prefix)
return owner
def _image_owner(image_path, file_path, prefixes):
spelled = file_path or normalize_image_path(image_path)
owner = _owner(spelled, prefixes)
if owner is None:
owner = _owner(normalize_image_path(os.path.realpath(image_path)), prefixes)
return owner
def _unavailable_line(unavailable, *, partial):
noun = "source is" if unavailable == 1 else "sources are"
effect = "its files were" if unavailable == 1 else "their files were"
if not partial:
effect = "missing files were"
return f"{unavailable} image {noun} unavailable, so {effect} not counted."
def _missing_files(conn, roots):
"""Links whose file is gone, counted per source (only readable ones).
Each source reports ``checked`` with its own count, or ``unavailable``
(an unmounted drive) with no count: its links are never turned into a
count, while the readable sources still are. The check's ``count`` is the
readable sources' total (None when none is readable) and ``partial`` says
some sources were not checked. Links outside every source are named in the
detail, not counted.
Inside the snapshot this only reads the image rows; it returns the step
that checks their files, which runs after the snapshot ended.
"""
if not roots:
result = _result(None, ["No image source is available."], status="unavailable")
return {**result, "partial": False, "sources": []}
rows = []
cursor = conn.execute(
"SELECT id,image_path,filename,file_path FROM generated_images ORDER BY id"
)
while batch := cursor.fetchmany(BATCH_ROWS):
rows.extend(batch)
return lambda: _check_files(rows, roots)
def _check_files(rows, roots):
"""The ``missing_files`` result of the image *rows*, from the file system."""
readable = [_readable(root) for root in roots]
sources = [
{
"name": _source_name(root, index),
"status": "checked" if ok else "unavailable",
"count": 0 if ok else None,
}
for index, (root, ok) in enumerate(zip(roots, readable))
]
unavailable = readable.count(False)
if not any(readable):
result = _result(
None, [_unavailable_line(unavailable, partial=False)], status="unavailable"
)
return {**result, "partial": False, "sources": sources}
prefixes = [_root_prefix(root) for root in roots]
outside, missing = 0, []
for image_id, image_path, filename, file_path in rows:
owner = _image_owner(image_path, file_path, prefixes)
if owner is None:
outside += 1
elif readable[owner] and not os.path.exists(image_path):
sources[owner]["count"] += 1
if len(missing) < MAX_DETAIL:
name = filename or os.path.basename(image_path)
missing.append(f"Image {image_id}: {name} is missing")
count = sum(source["count"] or 0 for source in sources)
detail = [_unavailable_line(unavailable, partial=True)] if unavailable else []
detail += missing
if outside:
links = "link" if outside == 1 else "links"
verb = "was" if outside == 1 else "were"
detail.append(
f"{outside} {links} outside the image sources {verb} not checked."
)
status = None if count or not unavailable else "unavailable"
result = _result(count, detail, status=status)
return {**result, "partial": bool(unavailable), "sources": sources}
def _ambiguous(review_count):
if review_count is None:
return _result(None, [REVIEW_UNAVAILABLE], status="unavailable")
return _result(review_count, [])
def overall_status(checks) -> str:
"""``passed`` only when every check passed; any check not run is incomplete."""
statuses = {check["status"] for check in checks}
if statuses & {"unavailable", "failed"} or any(
check.get("partial") for check in checks
):
return "incomplete"
return "issues" if "issues" in statuses else "passed"
def _failed():
return _result(None, [CHECK_FAILED], status="failed")
def _run_check(check_id, step, *args):
"""One check's result, or None (reported as failed) when it raised."""
try:
return step(*args)
except Exception: # any error fails this check only; the others still run
logger.exception(f"Health check {check_id} failed")
return None
def _run_database_checks(db_path, roots):
"""Each database check's result (None when it raised), from one snapshot.
A check may return a step instead of its result: the step runs after the
snapshot ended and the connection closed (``missing_files``' file checks).
"""
results = dict.fromkeys(_DATABASE_CHECKS)
try:
conn = sqlite3.connect(readonly_uri(db_path), uri=True)
except sqlite3.Error:
logger.exception("Health check could not open the database")
return results
try:
conn.execute("BEGIN") # one read snapshot for every check
for check_id in _DATABASE_CHECKS:
runner = globals()["_" + check_id] # looked up per run (tests patch it)
results[check_id] = _run_check(check_id, runner, conn, roots)
except sqlite3.Error:
logger.exception("Health check could not read the database")
finally:
conn.close()
for check_id, result in results.items():
if callable(result):
results[check_id] = _run_check(check_id, result)
return results
class HealthOperations:
"""Mixin for PromptDatabase: the read-only System health check."""
def run_health_checks(self, roots, *, review_count=None) -> dict:
"""Run every check and return a ``HealthReport``.
Args:
roots: The image source directories to judge missing files under:
the effective sources plus any configured source that is not
available (reported as an unavailable source of ``missing_files``).
review_count: The Needs-review pending count, or None while no
review queue exists (``ambiguous_metadata`` is then unavailable).
"""
if review_count is not None and (
type(review_count) is not int or review_count < 0
):
raise ValueError("review_count must be a non-negative integer or None")
roots = [str(root) for root in roots]
database = _run_database_checks(self.model.db_path, roots)
checks = []
for check_id, label in CHECKS:
if check_id == "ambiguous_metadata":
result = _ambiguous(review_count)
else:
result = database[check_id] or _failed()
checks.append({"id": check_id, "label": label, **result})
return {
"success": True,
"checked_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"status": overall_status(checks),
"checks": checks,
}
+54
View File
@@ -628,3 +628,57 @@ class LibraryOperations:
"kind": "prompt-manager-library",
"positives": [owners[key] for key in order],
}
def library_counts(self) -> dict:
"""System › Overview counts from one snapshot, writing nothing.
``positive_prompts`` and ``negative_variants`` are the unfiltered
Library's ``total`` and ``variant_total``: eligible parents and active
pairs (every pair's positive is eligible). ``images`` counts indexed
``generated_images`` rows, not files on disk. An image has no rating
of its own (it shows its linked prompt's), so ``rated_images`` counts
the indexed images whose linked prompt is rated. ``review_pending`` is
the Needs review queue's pending total.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
row = conn.execute(
"SELECT (SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + "),"
"(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
"(SELECT COUNT(*) FROM generated_images),"
"(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL),"
"(SELECT COUNT(*) FROM media_review_items WHERE state='pending')"
).fetchone()
return {
"positive_prompts": row[0],
"negative_variants": row[1],
"images": row[2],
"rated_images": row[3],
"review_pending": row[4],
}
def usage_by_category(self) -> list[dict]:
"""Runs and average rating of the Library's parents, per category.
Only eligible parents count, so a negative-only prompt's runs (which
repeat its pairs' runs) are never added twice. A blank category groups
with no category as ``None``. Unrated parents are left out of the
average, which is ``None`` when no parent is rated. Most runs first.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
rows = conn.execute(
"SELECT * FROM (SELECT NULLIF(TRIM(p.category),'') AS category,"
"COALESCE(SUM(p.run_count),0) AS runs,AVG(p.rating) AS rating "
"FROM prompts p WHERE " + _ELIGIBLE + " GROUP BY 1) "
"ORDER BY runs DESC,category IS NULL,category"
).fetchall()
return [
{
"category": row["category"],
"runs": row["runs"],
"average_rating": (
None if row["rating"] is None else round(row["rating"], 2)
),
}
for row in rows
]
+20 -3
View File
@@ -18,6 +18,11 @@ from typing import Optional, Set, Tuple
from .library_schema import library_schema_error, migrate_library_state
from .image_favorites import image_favorites_schema_error, migrate_image_favorites
from .annotation_schema import annotation_schema_error, migrate_annotation_state
from .review_schema import (
migrate_review_state,
review_schema_error,
review_without_annotations_error,
)
from .saved_views import migrate_saved_views, saved_views_schema_error
from .execution_history_schema import (
history_schema_error,
@@ -303,7 +308,14 @@ class PromptModel:
migrate_library_state(conn)
migrate_image_favorites(conn)
migrate_saved_views(conn)
# A review group without its annotation group is refused before the
# annotation install could make it look valid (catalog only; the rows
# are validated once, in migrate_review_state).
review_error = review_without_annotations_error(conn)
if review_error:
raise sqlite3.DatabaseError(review_error)
migrate_annotation_state(conn)
migrate_review_state(conn)
def _migrate_prompt_pairs(self, conn: sqlite3.Connection) -> None:
conn.execute("SAVEPOINT prompt_pair_schema")
@@ -1023,9 +1035,9 @@ class PromptModel:
Args:
path: File to inspect
require_library_state: Also require the tables every current
database has (Library state, image favorites, saved views
and image annotations), as a migrated restore stage must;
old backups may lack them.
database has (Library state, image favorites, saved views,
image annotations and the review queue), as a migrated
restore stage must; old backups may lack them.
Returns:
(ok, reason): reason is "ok" when the file is acceptable, otherwise
@@ -1095,6 +1107,11 @@ class PromptModel:
)
if annotation_error:
return False, annotation_error
review_error = review_schema_error(
conn, allow_missing=not require_library_state
)
if review_error:
return False, review_error
return True, "ok"
except sqlite3.Error as e:
return False, f"Invalid SQLite database: {e}"
+20 -31
View File
@@ -4,6 +4,7 @@ Database operations for KikoTextEncode prompt storage and retrieval.
import sqlite3
import json
import math
import datetime
import os
from typing import Optional, List, Dict, Any, Set, Tuple
@@ -13,6 +14,8 @@ from .pairs import PairDependencyError, PairOperations
from .library import LibraryOperations, titled
from .image_favorites import ImageFavoriteOperations
from .annotations import AnnotationOperations
from .review import ReviewOperations
from .review_decisions import ReviewDecisionOperations
from .saved_views import SavedViewOperations
from .library_commands import LibraryCommandOperations
from .vocabulary import VocabularyOperations
@@ -20,11 +23,13 @@ from .execution_history import ExecutionHistoryOperations
from .execution_contexts import ExecutionContextOperations, LIVE_HINT_CTES
from .execution_outputs import ExecutionOutputOperations
from .history_reads import HistoryReadOperations
from .health import HealthOperations, iter_consistency_issues
# Import logging system
try:
from ..utils.logging_config import get_logger
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
except ImportError:
import sys
@@ -32,6 +37,7 @@ except ImportError:
sys.path.insert(0, current_dir)
from utils.logging_config import get_logger
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
# Subquery to fetch tags from junction table, embedded in SELECT statements
@@ -341,7 +347,10 @@ class PromptDatabase(
HistoryReadOperations,
ImageFavoriteOperations,
AnnotationOperations,
ReviewOperations,
ReviewDecisionOperations,
SavedViewOperations,
HealthOperations,
):
"""Database operations class for managing prompts."""
@@ -1695,9 +1704,9 @@ class PromptDatabase(
width,
height,
image_format,
json.dumps(metadata.get("workflow", {})),
json.dumps(metadata.get("prompt", {})),
json.dumps(metadata.get("parameters", {})),
dumps_strict(metadata.get("workflow", {})),
dumps_strict(metadata.get("prompt", {})),
dumps_strict(metadata.get("parameters", {})),
),
)
conn.commit()
@@ -1902,13 +1911,13 @@ class PromptDatabase(
obj: The object to clean (dict, list, or scalar)
Returns:
Cleaned object with NaN values replaced by None
Cleaned object with NaN and Infinity values replaced by None
"""
if isinstance(obj, dict):
return {key: self._clean_nan_values(value) for key, value in obj.items()}
elif isinstance(obj, list):
return [self._clean_nan_values(item) for item in obj]
elif isinstance(obj, float) and str(obj) == "nan":
elif isinstance(obj, float) and not math.isfinite(obj):
return None
else:
return obj
@@ -2570,33 +2579,13 @@ class PromptDatabase(
return cursor.rowcount
def check_consistency(self) -> List[str]:
"""Run consistency checks and return a list of issue descriptions."""
issues: List[str] = []
with self.model.get_connection() as conn:
# Check for orphaned prompt_tags entries
cursor = conn.execute("""
SELECT pt.prompt_id, pt.tag_id FROM prompt_tags pt
LEFT JOIN prompts p ON pt.prompt_id = p.id
WHERE p.id IS NULL
""")
for ref in cursor.fetchall():
issues.append(
"prompt_tags entry references non-existent prompt "
f"{ref['prompt_id']}"
)
"""Run consistency checks and return a list of issue descriptions.
# Check for orphaned image entries
cursor = conn.execute("""
SELECT gi.id, gi.prompt_id FROM generated_images gi
LEFT JOIN prompts p ON gi.prompt_id = p.id
WHERE p.id IS NULL
""")
for ref in cursor.fetchall():
issues.append(
f"Image {ref['id']} references non-existent prompt "
f"{ref['prompt_id']}"
)
return issues
The same queries back the System health check's Consistency row
(database/health.py).
"""
with self.model.get_connection() as conn:
return list(iter_consistency_issues(conn))
def _image_row_to_dict(self, row: sqlite3.Row) -> Dict[str, Any]:
"""
+136 -108
View File
@@ -1,6 +1,5 @@
"""Atomic metadata enrichment for a single known generation context."""
import json
import os
from typing import Any, Dict
@@ -9,8 +8,12 @@ from .operations import _coerce_file_info
try:
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
from ..utils.prompt_pair_resolver import RESOLVER_VERSION
except ImportError:
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
from utils.prompt_pair_resolver import RESOLVER_VERSION
def ingest_resolved_pair(
@@ -19,118 +22,143 @@ def ingest_resolved_pair(
state = resolution.get("state", "unresolved")
if state != "resolved":
return {"status": state, "pairs_added": 0, "links_added": 0}
with db._pair_transaction() as conn:
return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution)
def _conflict() -> Dict[str, Any]:
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
def _ingest_resolved_pair_in(
conn: Any,
db: Any,
image_path: str,
metadata: Dict[str, Any],
resolution: Dict[str, Any],
*,
provenance: Dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``.
The compatible-only rules are the public ingester's: an image with more
than one exact row, another positive, or a prior pair with another
negative is a ``conflict`` and nothing is written. The caller owns the
transaction, so its own later failure rolls the link back too.
``provenance`` replaces the link's default resolver provenance (Needs
review records the user's selection there).
"""
positive, negative = resolution["positive"], resolution["negative"]
positive_text = positive.get("storage_text", positive["text"])
negative_text = negative.get("storage_text", negative["text"])
normalized = normalize_image_path(image_path)
provenance = {"resolver_version": 1, "positive": positive, "negative": negative}
encoded = json.dumps(
provenance, sort_keys=True, separators=(",", ":"), allow_nan=False
)
with db._pair_transaction() as conn:
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
if provenance is None:
provenance = {
"resolver_version": RESOLVER_VERSION,
"positive": positive,
"negative": negative,
}
encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":"))
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return _conflict()
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute(
"SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
snapshots = {
column: json.dumps(metadata[key], sort_keys=True, allow_nan=False)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update(
{key: value for key, value in info.items() if value is not None}
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
if any(row["negative_hash"] != expected for row in prior):
return _conflict()
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
snapshots = {
column: dumps_strict(metadata[key], sort_keys=True)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update({key: value for key, value in info.items() if value is not None})
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
+541
View File
@@ -0,0 +1,541 @@
"""Needs review items (#249): persist and read capture-bound ambiguous evidence.
An item is one ambiguous resolver result for one annotation byte record (the
whole-file SHA-256 of one physical file at one locator), keyed by
``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the
same evidence for the same bytes is a duplicate and never touches an existing
item, its state, revision or decisions. Changed bytes are a new byte record
and therefore a new pending item; the old one stays.
Callers pass verified facts (the capture's digest, size and ``mtime_ns``);
nothing here reads the filesystem. Writes run in one ``_pair_transaction``
fenced by the storage epoch the capture was taken under.
"""
import hashlib
import json
import os
import re
import sqlite3
from typing import Iterator
try:
from ..utils.hashing import generate_prompt_hash
except ImportError:
from utils.hashing import generate_prompt_hash
from .annotation_schema import (
BYTES_TABLE,
FILES_TABLE,
METADATA_TABLE as ANNOTATION_METADATA_TABLE,
)
from .annotations import _new_id, _validate_locator, _validate_target
from .models import normalize_image_path, utc_now_iso
from .review_schema import (
DECISIONS_TABLE,
ITEMS_TABLE,
MAX_CANDIDATES,
MAX_SPELLED_PATH_BYTES,
METADATA_TABLE,
canonical_evidence,
evidence_digest,
evidence_offers_choice,
)
# Only this module's table constants are interpolated into the SQL below.
_INSERT_ITEM = (
f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path," # nosec B608
"resolver_version,evidence_hash,evidence,candidate_count,limited,state,"
"revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)"
)
_BUMP_REVIEW_INDEX = (
f"UPDATE {METADATA_TABLE} SET " # nosec B608
"review_index_revision=review_index_revision+1 WHERE singleton_id=1"
)
# A later spelling of the same locator (a case-only rename on a case-folding
# filesystem) replaces the recorded one, so targets keep formatting.
_REFRESH_SPELLING = (
f"UPDATE {ITEMS_TABLE} SET spelled_path=? " # nosec B608
f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)"
)
_REVIEW_INDEX = (
f"SELECT review_index_revision FROM {METADATA_TABLE} " # nosec B608
"WHERE singleton_id=1"
)
_BUMP_IMAGE_INDEX = (
f"UPDATE {ANNOTATION_METADATA_TABLE} SET " # nosec B608
"image_index_revision=image_index_revision+1 WHERE singleton_id=1"
)
_ITEMS_OF_LOCATOR = (
"SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash," # nosec B608
"i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at,"
f"i.updated_at,b.sha256 FROM {FILES_TABLE} f "
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? ORDER BY i.created_at,i.id"
)
_QUEUED = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"LIMIT 1"
)
_PENDING_AT = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"AND i.state='pending' LIMIT 1"
)
# Pending items of one file: the candidates a link settles.
_PENDING_OF_LOCATOR = (
f"SELECT i.id FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Pending items of one file with their evidence: an untraced file's settle.
_PENDING_EVIDENCE_OF_LOCATOR = (
f"SELECT i.id,i.evidence FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Only an undecided item goes: a decision row (RESTRICT) is never deleted.
_SETTLE_ITEM = (
f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' " # nosec B608
f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d "
f"WHERE d.item_id={ITEMS_TABLE}.id)"
)
_EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
_PENDING_COUNT = (
f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'" # nosec B608
)
_QUEUE_COLUMNS = (
"id",
"state",
"revision",
"created_at",
"candidate_count",
"limited",
"spelled_path",
"locator",
)
_QUEUE_SELECT = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i "
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id "
)
_QUEUE_FIRST = _QUEUE_SELECT + (
"WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?"
)
_QUEUE_AFTER = _QUEUE_SELECT + (
"WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) "
"ORDER BY i.created_at,i.id LIMIT ?"
)
_DETAIL_COLUMNS = _QUEUE_COLUMNS + (
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"updated_at",
"sha256",
)
_DETAIL = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
"i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version,"
"i.evidence_hash,i.evidence,i.updated_at,b.sha256 "
f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?"
)
# The legacy rows and pair links the ingester's compatibility rules read.
_DEPENDENT_IMAGES = (
"SELECT gi.id,p.hash,gi.image_path FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id"
)
_DEPENDENT_LINKS = (
"SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE ip.image_id=? ORDER BY ip.pair_id"
)
# Rows whose spelling ends with the file's name: the candidates for another
# spelling of the same physical file (the link summaries' filename fallback).
_SAME_NAME_ROWS = (
"SELECT id,image_path FROM generated_images "
"WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id"
)
_TITLES = (
"SELECT p.hash,s.title FROM prompts p "
"JOIN prompt_library_state s ON s.prompt_id=p.id "
"WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id"
)
_ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
_ITEM_COLUMNS = (
"id",
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"candidate_count",
"limited",
"state",
"revision",
"created_at",
"updated_at",
"sha256",
)
def _validate_spelling(spelled_path: object, locator: str) -> None:
if (
not isinstance(spelled_path, str)
or not spelled_path
or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES
or os.path.normcase(spelled_path) != locator
):
raise ValueError("spelled_path must be the file's spelling of its locator")
def dependent_images(conn: sqlite3.Connection, spellings) -> list:
"""``(id, image_path, positive_hash)`` of every legacy row of the file.
``spellings`` are the paths the file is linked under (the Apply path and
every alias of the same physical file); each is matched as the ingester
matches one path. Rows are distinct and ordered by id.
"""
found = {}
for spelled in dict.fromkeys(spellings):
for row in conn.execute(
_DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled)
):
found[row[0]] = (row[0], row[2], row[1])
return [found[key] for key in sorted(found)]
def same_name_rows(conn: sqlite3.Connection, name: str) -> list:
"""``(id, image_path)`` of rows whose spelling may end with ``name``."""
# Imported here: operations composes this module's mixin.
from .operations import escape_like
return [
(row[0], row[1])
for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",))
]
def dependency_fingerprint(
conn: sqlite3.Connection, image_path: str, aliases=()
) -> str:
"""SHA-256 of what Apply's compatibility depends on for ``image_path``.
The exact legacy rows of the file under ``image_path`` and every alias
spelling of it, with their positive hashes, and each row's pair links
with the negative hash, read as the ingester reads them. Unrelated prompt
changes (tags, titles) do not move it.
"""
rows = []
for image_id, _spelled, positive_hash in dependent_images(
conn, (image_path, *aliases)
):
links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall()
rows.append([image_id, positive_hash, [list(link) for link in links]])
text = canonical_evidence({"v": 1, "images": rows})
return hashlib.sha256(text.encode("utf-8")).hexdigest()
class ReviewOperations:
"""PromptDatabase mixin: insert and read Needs review items."""
def record_review_item(
self,
*,
locator: str,
spelled_path: str,
sha256: str,
size: int,
mtime_ns: int,
epoch: str,
resolver_version: int,
evidence: dict,
candidate_count: int,
limited: bool,
) -> bool:
"""Store one ambiguous result for these exact bytes; True when new.
The byte record is created (or its verified fingerprint refreshed)
like an observe. ``spelled_path`` is the file's own spelling (it must
fold to ``locator``); targets are formatted from it. Raises
``LibraryCommandError("storage-changed")`` when the storage epoch is
no longer ``epoch`` (a restore since the capture) and ``ValueError``
for invalid facts or evidence.
"""
_validate_target(locator, sha256, size, mtime_ns)
_validate_spelling(spelled_path, locator)
if type(resolver_version) is not int or resolver_version < 1:
raise ValueError("resolver_version must be a positive integer")
if (
type(candidate_count) is not int
or not 0 <= candidate_count <= MAX_CANDIDATES
):
raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}")
if not isinstance(limited, bool) or (limited and candidate_count):
raise ValueError("A limited item carries no candidates")
text = canonical_evidence(evidence)
evidence_hash = evidence_digest(text)
with self._pair_transaction() as conn:
self._library_epoch(conn, epoch)
now = utc_now_iso()
file_id = self._annotation_owner(conn, locator, now)
byte_id, _revision, created, moved = self._annotation_byte(
conn, file_id, sha256, size, mtime_ns, now
)
if created or moved:
conn.execute(_BUMP_IMAGE_INDEX)
cursor = conn.execute(
_INSERT_ITEM,
(
_new_id(),
byte_id,
spelled_path,
resolver_version,
evidence_hash,
text,
candidate_count,
int(limited),
now,
now,
),
)
inserted = cursor.rowcount > 0
refreshed = _refresh_spelling(conn, locator, spelled_path)
if inserted or refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
return inserted
def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool:
"""Record ``spelled_path`` as the spelling of every item at ``locator``.
The rescan's skip calls it for a file it does not copy again: a
case-only rename keeps size and ``mtime_ns``, and a target formatted
from the old spelling would be refused. True when a spelling changed
(the review index revision moves; item revisions do not, so an open
decision stays valid).
"""
_validate_locator(locator)
_validate_spelling(spelled_path, locator)
with self._pair_transaction() as conn:
refreshed = _refresh_spelling(conn, locator, spelled_path)
if refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
return refreshed
def review_index_revision(self) -> int:
"""The queue's revision (moved by every queue change)."""
row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
return row[0]
def review_items_for(self, locator: str) -> list:
"""Every item of the file at ``locator``, oldest first, evidence parsed."""
_validate_locator(locator)
return list(self._iter_review_items(locator))
def _iter_review_items(self, locator: str) -> Iterator[dict]:
conn = self.model.get_connection()
for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)):
item = dict(zip(_ITEM_COLUMNS, tuple(row)))
item["evidence"] = json.loads(item["evidence"])
item["limited"] = bool(item["limited"])
yield item
# -- queue reads (#249 E3) -------------------------------------------------
def review_fingerprint_queued(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when an item cites a byte record verified at this exact stat.
A rescan uses it to skip the copy and hash of a file it already
queued: the same ``(size, mtime_ns)`` the bytes were verified with.
"""
row = (
self.model.get_connection()
.execute(_QUEUED, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_pending_at(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when a *pending* item cites bytes verified at this exact stat.
``review_fingerprint_queued`` also matches decided items; a file whose
only item is decided is not in the queue.
"""
row = (
self.model.get_connection()
.execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_file_linked(self, spellings) -> bool:
"""True when a saved row links the file under any of ``spellings``."""
with self._annotation_read() as conn:
return bool(dependent_images(conn, spellings))
def settle_linked_review_items(self, files) -> int:
"""Remove the pending, undecided items of files that are now linked.
``files`` holds ``(locator, spellings)`` per file. An item leaves the
queue only when it is still ``pending`` at revision 0 with no decision
row and a saved row links its file under one of ``spellings``; a
decided item is never touched. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
files = [(locator, tuple(spellings)) for locator, spellings in files]
for locator, _spellings in files:
_validate_locator(locator)
conn = self.model.get_connection()
candidates = [
(locator, spellings)
for locator, spellings in files
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not candidates:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator, spellings in candidates:
if not dependent_images(conn, spellings):
continue
for (item_id,) in conn.execute(
_PENDING_OF_LOCATOR, (locator,)
).fetchall():
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
return removed
def settle_untraced_review_items(self, locators) -> int:
"""Remove the pending, undecided items that offer nothing to choose.
``locators`` are files the caller found ambiguous without a choice
(fewer than two candidates carry a prompt). Of their items, one leaves
the queue only when its own stored evidence offers no choice either
(``evidence_offers_choice``) and it is still ``pending`` at revision 0
with no decision row, the same safe delete as
``settle_linked_review_items``. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
locators = list(locators)
for locator in locators:
_validate_locator(locator)
conn = self.model.get_connection()
pending = [
locator
for locator in locators
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not pending:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator in pending:
rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall()
for item_id, evidence in rows:
if evidence_offers_choice(json.loads(evidence)):
continue
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
return removed
def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict:
"""One page of pending items, oldest first, from one snapshot.
``after`` is the ``(created_at, id)`` keyset of the last item already
shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is
the exact pending count and ``items`` carry their ``spelled_path`` and
``locator`` (server-side only).
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be a positive integer")
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
revision = conn.execute(_REVIEW_INDEX).fetchone()
if epoch is None or revision is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if after is None:
rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall()
else:
rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall()
items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]]
return {
"epoch": epoch[0],
"revision": revision[0],
"total": total,
"items": items,
"more": len(rows) > limit,
}
def review_item(self, item_id: str) -> dict | None:
"""One item with its parsed evidence and the pending total, or None."""
if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None:
return None
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
if epoch is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item["evidence"] = json.loads(item["evidence"])
item["epoch"] = epoch[0]
item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0]
return item
def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str:
with self._annotation_read() as conn:
return dependency_fingerprint(conn, image_path, aliases)
def review_same_name_rows(self, name: str) -> list:
"""``(id, image_path)`` of legacy rows that may spell a file ``name``."""
with self._annotation_read() as conn:
return same_name_rows(conn, name)
def stored_prompt_titles(self, texts) -> dict:
"""Library titles of saved prompts by text (dedup hash), when set."""
hashes = {generate_prompt_hash(text): text for text in texts if text.strip()}
if not hashes:
return {}
marks = ",".join("?" * len(hashes))
found: dict[str, tuple] = {}
conn = self.model.get_connection()
for prompt_hash, title in conn.execute(
_TITLES.format(marks=marks), list(hashes)
):
text = hashes[prompt_hash]
found[text] = found.get(text, ()) + (title,)
return found
def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool:
return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0
def _queue_row(row, columns) -> dict:
item = dict(zip(columns, tuple(row)))
item["limited"] = bool(item["limited"])
return item
+287
View File
@@ -0,0 +1,287 @@
"""Needs review decisions (#249 E3): explicit, auditable and compatible-only.
A decision is one row of ``media_review_decisions`` for one item revision.
``selected`` links the chosen candidate's pair to the file through the public
ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same
``_pair_transaction`` as the decision row, the item's new state and revision
and the review index revision; any conflict or later failure rolls all of it
back. ``unresolved`` records the decision and links nothing. No decision
counts a run, touches usage or calls an execution writer.
Callers verify the file first (the capture must be the item's bytes and the
current file must still hash to them); nothing here reads the filesystem.
"""
import os
import sqlite3
from .annotation_schema import BYTES_TABLE
from .annotations import _new_id
from .library_commands import LibraryCommandError
from .models import utc_now_iso
from .review import (
_BUMP_REVIEW_INDEX,
_DETAIL,
_DETAIL_COLUMNS,
_PENDING_COUNT,
_queue_row,
dependency_fingerprint,
dependent_images,
same_name_rows,
)
from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence
DECISION_KIND = "user-selection"
_INSERT_DECISION = (
f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision," # nosec B608
"created_at,outcome,candidate_hash,evidence,pair_id,image_id) "
"VALUES(?,?,?,?,?,?,?,?,?)"
)
_ADVANCE_ITEM = (
f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1," # nosec B608
"updated_at=? WHERE id=? AND revision=?"
)
_ITEM_BYTES = (
f"SELECT b.sha256 FROM {ITEMS_TABLE} i " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?"
)
class ReviewCommandError(LibraryCommandError):
"""A decision lost a race or would rewrite an association.
``code`` is ``review-conflict`` (the item was decided or changed since it
was read), ``association-conflict`` (the choice is not compatible with
the file's saved links, or those links changed), ``file-changed`` (the
item is not about these bytes) or ``review-missing``. ``item`` is the
current item read inside the transaction, ``total`` the pending count and
``fingerprint`` the current dependency fingerprint, when known;
``changed`` marks an association conflict caused by links that changed
since the reviewer read them (rather than an incompatible choice).
"""
def __init__(
self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False
):
super().__init__(code, epoch)
self.item = item
self.total = total
self.fingerprint = fingerprint
self.changed = changed
# The public view of ``item``, set by the API when it formats one.
self.view = None
class _Rollback(Exception):
"""Raised inside the transaction so the context manager rolls back."""
def __init__(self, error: ReviewCommandError):
super().__init__(error.code)
self.error = error
class ReviewDecisionOperations:
"""PromptDatabase mixin: record one decision on one Needs review item."""
def decide_review_item(
self,
*,
item_id: str,
expected_epoch: str,
expected_revision: int,
sha256: str,
outcome: str,
candidate: dict | None = None,
image_path: str | None = None,
metadata: dict | None = None,
expected_fingerprint: str | None = None,
aliases: tuple = (),
known_rows: frozenset | None = None,
) -> dict:
"""Record the decision; return ``{item, total, pair}``.
``selected`` needs the evidence ``candidate`` (with its ``hash``), the
``image_path`` the scan and the monitor link under, the captured
``metadata`` and the ``expected_fingerprint`` the reviewer saw.
``aliases`` are the other spellings legacy rows of the same physical
file use (a symlinked, junctioned or mapped root) and ``known_rows``
the ids of every same-name row the caller classified; a same-name row
it did not see is a changed association.
Raises ``LibraryCommandError("storage-changed")`` after a restore and
:class:`ReviewCommandError` for every refused decision, with zero
writes either way.
"""
if outcome not in ("selected", "unresolved"):
raise ValueError("outcome must be selected or unresolved")
if outcome == "selected" and (
not isinstance(candidate, dict) or not image_path
):
raise ValueError("A selected decision needs a candidate and a file")
try:
with self._pair_transaction() as conn:
return self._decide_in(
conn,
item_id=item_id,
expected_epoch=expected_epoch,
expected_revision=expected_revision,
sha256=sha256,
outcome=outcome,
candidate=candidate,
image_path=image_path,
metadata=metadata or {},
expected_fingerprint=expected_fingerprint,
aliases=tuple(aliases),
known_rows=known_rows,
)
except _Rollback as rollback:
raise rollback.error from None
def _decide_in(self, conn: sqlite3.Connection, **command) -> dict:
epoch = self._library_epoch(conn, command["expected_epoch"])
item_id = command["item_id"]
current = _current_item(conn, item_id)
if current is None:
raise _Rollback(ReviewCommandError("review-missing", epoch))
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if (
current["state"] != "pending"
or current["revision"] != command["expected_revision"]
):
raise _Rollback(
ReviewCommandError("review-conflict", epoch, item=current, total=total)
)
if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]:
raise _Rollback(ReviewCommandError("file-changed", epoch))
decision_id = _new_id()
pair = None
candidate = command["candidate"] if command["outcome"] == "selected" else None
if candidate is not None:
pair = self._apply_candidate(
conn, epoch, current, decision_id, total, **command
)
evidence = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"outcome": command["outcome"],
"reviewed_digest": command["sha256"],
"item_evidence_hash": current["evidence_hash"],
"resolver_version": current["resolver_version"],
"candidate": candidate,
"dependency_fingerprint": command["expected_fingerprint"],
"pair": pair,
}
now = utc_now_iso()
self._insert_review_decision(
conn,
(
decision_id,
item_id,
current["revision"] + 1,
now,
command["outcome"],
candidate["hash"] if candidate is not None else None,
canonical_evidence(evidence),
pair["id"] if pair else None,
pair["image_id"] if pair else None,
),
)
conn.execute(
_ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"])
)
conn.execute(_BUMP_REVIEW_INDEX)
return {
"item": _current_item(conn, item_id),
"total": conn.execute(_PENDING_COUNT).fetchone()[0],
"pair": pair,
"decision_id": decision_id,
}
def _apply_candidate(
self, conn, epoch, current, decision_id, total, **command
) -> dict:
"""Link the candidate's pair, compatible-only; raise on any conflict."""
# Imported here: pair_ingestion imports operations, which composes this.
from .pair_ingestion import _ingest_resolved_pair_in
image_path = command["image_path"]
aliases = command["aliases"]
fingerprint = dependency_fingerprint(conn, image_path, aliases)
known = command["known_rows"]
unseen = known is not None and any(
row_id not in known
for row_id, _path in same_name_rows(conn, os.path.basename(image_path))
)
if unseen or fingerprint != command["expected_fingerprint"]:
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
changed=True,
)
)
rows = dependent_images(conn, (image_path, *aliases))
if len(rows) > 1:
# One file linked under several spellings: no single row to enrich.
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
if rows:
# Enrich the one row of this file under the spelling it was
# linked with, so the ingester's compatibility rules see it.
image_path = rows[0][1]
candidate = command["candidate"]
positive, negative = candidate["positive"], candidate["negative"]
provenance = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"reviewed_digest": command["sha256"],
"candidate_hash": candidate["hash"],
"resolver_version": current["resolver_version"],
"positive": positive,
"negative": negative,
}
result = _ingest_resolved_pair_in(
conn,
self,
image_path,
command["metadata"],
{"state": "resolved", "positive": positive, "negative": negative},
provenance=provenance,
)
if result["status"] == "conflict":
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
return {
"id": result["pair_id"],
"image_id": result["image_id"],
"status": result["status"],
}
@staticmethod
def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None:
conn.execute(_INSERT_DECISION, values)
def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None:
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item.pop("evidence")
return item
+507
View File
@@ -0,0 +1,507 @@
"""Needs review schema group (#249): capture-bound ambiguous items and decisions.
The group installs like ``image_annotations``: optional in old backups,
required once a database is current, and valid only beside a valid image
annotation group, because every item cites one annotation byte record. It owns
- ``media_review_metadata``: one row, the group's schema version and the
review index revision (moved by every queue change);
- ``media_review_items``: one ambiguous resolver result for one byte record,
keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's
spelling when it was recorded (the locator is case-folded on Windows, a
target is not), its canonical evidence, candidate count, ``limited`` flag,
state and revision;
- ``media_review_decisions``: the auditable decision history of an item, one
row per decision revision, with optional navigation to the pair and image
the decision linked (nulled, never cascaded, when those are deleted).
Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted
keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound
agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is
the SHA-256 of the item's evidence text.
Installation never backfills anything and never touches the filesystem.
Validation reads only: catalog objects compared after whitespace compaction,
columns and foreign keys through the PRAGMAs, persisted values in single
bounded SQL passes, then one streamed pass that re-canonicalizes each evidence
document (one row in memory at a time).
"""
import hashlib
import json
import re
import sqlite3
from typing import Any, Optional
from .annotation_schema import BYTES_TABLE
from .annotation_schema import TABLES as ANNOTATION_TABLES
from .annotation_schema import _catalog as _annotation_catalog
from .annotation_schema import annotation_schema_error
try:
from ..utils.json_strict import dumps_strict
except ImportError:
from utils.json_strict import dumps_strict
SCHEMA_VERSION = 1
METADATA_TABLE = "media_review_metadata"
ITEMS_TABLE = "media_review_items"
DECISIONS_TABLE = "media_review_decisions"
TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
MAX_CANDIDATES = 16
# The file's own spelling when it was recorded (bytes of UTF-8 text). Locators
# are case-folded on Windows, so a target is formatted from this instead.
MAX_SPELLED_PATH_BYTES = 16384
ITEM_STATES = ("pending", "selected", "unresolved")
DECISION_OUTCOMES = ("selected", "unresolved")
_TIMESTAMP_BYTES = 64
_HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'"
_EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})"
_TABLE_DDL = {
METADATA_TABLE: (
f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
f"CHECK(schema_version={SCHEMA_VERSION}), "
"review_index_revision INTEGER NOT NULL DEFAULT 0 "
"CHECK(review_index_revision>=0))"
),
ITEMS_TABLE: (
f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, "
"spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), "
"resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL "
f"CHECK({_HEX64.format(c='evidence_hash')}), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"candidate_count INTEGER NOT NULL "
f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), "
"state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), "
"revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, "
"updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS_TABLE: (
f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY "
"CHECK(length(id)=36), item_id TEXT NOT NULL "
f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
f"({_HEX64.format(c='candidate_hash')})), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
_INDEX_DDL = {
# The queue reads pending items oldest first with a stable tiebreak.
"idx_media_review_items_queue": (
"CREATE INDEX idx_media_review_items_queue ON "
f"{ITEMS_TABLE}(state, created_at, id)"
),
# Deleting a pair or an image nulls decision navigation by index, not scan.
"idx_media_review_decisions_pair": (
"CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)"
),
"idx_media_review_decisions_image": (
"CREATE INDEX idx_media_review_decisions_image ON "
f"{DECISIONS_TABLE}(image_id)"
),
}
# ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key.
_AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2}
# (name, declared type, not null, default, primary key position)
_COLUMNS = {
METADATA_TABLE: (
("singleton_id", "INTEGER", 0, None, 1),
("schema_version", "INTEGER", 1, None, 0),
("review_index_revision", "INTEGER", 1, "0", 0),
),
ITEMS_TABLE: (
("id", "TEXT", 0, None, 1),
("byte_id", "TEXT", 1, None, 0),
("spelled_path", "TEXT", 1, None, 0),
("resolver_version", "INTEGER", 1, None, 0),
("evidence_hash", "TEXT", 1, None, 0),
("evidence", "TEXT", 1, None, 0),
("candidate_count", "INTEGER", 1, None, 0),
("limited", "INTEGER", 1, None, 0),
("state", "TEXT", 1, None, 0),
("revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("updated_at", "TEXT", 1, None, 0),
),
DECISIONS_TABLE: (
("id", "TEXT", 0, None, 1),
("item_id", "TEXT", 1, None, 0),
("decision_revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("outcome", "TEXT", 1, None, 0),
("candidate_hash", "TEXT", 0, None, 0),
("evidence", "TEXT", 1, None, 0),
("pair_id", "INTEGER", 0, None, 0),
("image_id", "INTEGER", 0, None, 0),
),
}
# (column, parent table, parent column, on delete)
_FOREIGN_KEYS = {
METADATA_TABLE: set(),
ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")},
DECISIONS_TABLE: {
("item_id", ITEMS_TABLE, "id", "RESTRICT"),
("pair_id", "prompt_pairs", "id", "SET NULL"),
("image_id", "generated_images", "id", "SET NULL"),
},
}
class _NonCanonical(ValueError):
pass
def _reject_constant(name: str) -> Any:
raise _NonCanonical(f"JSON constant {name} is not allowed")
# Built once: validation re-encodes every stored document at startup.
_ENCODER = json.JSONEncoder(
sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False
)
_DECODER = json.JSONDecoder(parse_constant=_reject_constant)
def canonical_evidence(value: Any) -> str:
"""Serialize one evidence object canonically; ``ValueError`` when invalid.
The text is a JSON object with sorted keys, no whitespace, ASCII only and
no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number
(ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's
policy everywhere (``utils.json_strict.dumps_strict``).
"""
if not isinstance(value, dict):
raise ValueError("Review evidence must be a JSON object")
try:
try:
text = _ENCODER.encode(value)
except ValueError as error:
if "Out of range float" not in str(error):
raise
text = dumps_strict(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
except (TypeError, RecursionError) as error:
raise ValueError(f"Review evidence is not serializable: {error}") from None
if len(text) > MAX_EVIDENCE_BYTES:
raise ValueError("Review evidence is larger than 8 MiB")
return text
def choosable(candidate: Any) -> bool:
"""True when a candidate offers a prompt to choose: it is ``resolved`` and
its positive side carries non-blank text. The page disables any other
candidate, so it is never something to choose.
"""
if not isinstance(candidate, dict) or candidate.get("state") != "resolved":
return False
positive = candidate.get("positive")
if not isinstance(positive, dict):
return False
text = positive.get("text")
return isinstance(text, str) and bool(text.strip())
# An item is worth a decision only when at least this many candidates offer
# a prompt; with fewer there is nothing to choose between (maintainer test of
# #284/#288: 39 items whose candidates were all "unresolved").
MIN_CHOOSABLE = 2
def evidence_offers_choice(evidence: Any) -> bool:
"""False when stored evidence provably offers nothing to choose.
Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A
limited item stores none, only ``candidate_total``; it may offer a choice
(more than ``MAX_CANDIDATES`` prompts) unless that total is too small.
"""
if not isinstance(evidence, dict):
return False
if evidence.get("limited"):
total = evidence.get("candidate_total")
return type(total) is int and total >= MIN_CHOOSABLE
candidates = evidence.get("candidates")
if not isinstance(candidates, list):
return False
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
def evidence_digest(text: str) -> str:
"""SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical)."""
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def _is_canonical(text: Any) -> bool:
if not isinstance(text, str):
return False
try:
value = _DECODER.decode(text)
return canonical_evidence(value) == text
except (ValueError, RecursionError):
return False
def _compact(sql: str) -> str:
return re.sub(r"\s+", "", sql).lower()
def _uuid(column: str) -> str:
"""SQL: *column* is a lowercase hyphenated 36-character UUID string."""
return (
f"typeof({column}) = 'text' AND length({column}) = 36 "
f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
f"AND length(replace({column}, '-', '')) = 32 "
f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
)
def _timestamp(column: str) -> str:
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
)
def _non_negative(column: str) -> str:
return f"typeof({column}) = 'integer' AND {column} >= 0"
def _hex64(column: str) -> str:
return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column)
def _evidence(column: str) -> str:
# Type and size only: the streamed pass below proves the text canonical
# (which implies printable ASCII), so a pattern scan here would read every
# document a second time.
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}"
)
def _nullable_integer(column: str) -> str:
return f"({column} IS NULL OR typeof({column}) = 'integer')"
def _in(column: str, values: tuple) -> str:
return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})"
# Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
_ROW_PREDICATES = {
ITEMS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("byte_id"),
"typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}",
"typeof(resolver_version) = 'integer'",
_hex64("evidence_hash"),
_evidence("evidence"),
_non_negative("candidate_count"),
f"candidate_count <= {MAX_CANDIDATES}",
"typeof(limited) = 'integer' AND limited IN (0, 1)",
_in("state", ITEM_STATES),
_non_negative("revision"),
_timestamp("created_at"),
_timestamp("updated_at"),
)
),
DECISIONS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("item_id"),
_non_negative("decision_revision"),
_timestamp("created_at"),
_in("outcome", DECISION_OUTCOMES),
"(CASE WHEN outcome = 'selected' THEN "
f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)",
_evidence("evidence"),
_nullable_integer("pair_id"),
_nullable_integer("image_id"),
)
),
}
def _expected_catalog() -> set:
objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in _AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def _catalog(conn: sqlite3.Connection) -> list:
names = TABLES + tuple(_INDEX_DDL)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return conn.execute( # only "?" placeholders are interpolated
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + TABLES,
).fetchall()
def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
for table, expected in _COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[4], row[5])
for row in conn.execute(f"PRAGMA table_info({table})")
)
if columns != expected:
return f"Review queue table {table} has invalid columns"
foreign_keys = {
(row[3], row[2], row[4], row[6])
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
}
if foreign_keys != _FOREIGN_KEYS[table]:
return f"Review queue table {table} has invalid foreign keys"
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
return f"Review queue table {table} contains foreign key violations"
return None
def _evidence_error(conn: sqlite3.Connection) -> Optional[str]:
"""Stream every evidence document once; one row is held at a time."""
for evidence_hash, text in conn.execute(
f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}" # nosec B608
):
if not _is_canonical(text) or evidence_digest(text) != evidence_hash:
return f"Review queue table {ITEMS_TABLE} contains invalid evidence"
for (text,) in conn.execute(
f"SELECT evidence FROM {DECISIONS_TABLE}" # nosec B608
):
if not _is_canonical(text):
return f"Review queue table {DECISIONS_TABLE} contains invalid evidence"
return None
def _value_error(conn: sqlite3.Connection) -> Optional[str]:
# Only this module's constants are interpolated into the SQL below.
rows = conn.execute(
"SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' " # nosec B608
"AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
f"AND schema_version = {SCHEMA_VERSION} "
f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}"
).fetchone()
if rows[0] != 1 or rows[1] != 1:
return "Review queue metadata is invalid"
for table, predicate in _ROW_PREDICATES.items():
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1" # nosec B608
).fetchone():
return f"Review queue table {table} contains invalid values"
return _evidence_error(conn)
def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
"""Catalog only: a review group present without its annotation tables.
Startup asks this before the annotation group installs, which would make
a review group copied without it look valid. It reads no rows; the full
validation runs once in :func:`migrate_review_state`.
"""
if not _catalog(conn):
return None
present = {
name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
}
if present != set(ANNOTATION_TABLES):
return (
"Review queue schema requires a valid image annotation schema: "
"Missing image annotation schema"
)
return None
def review_schema_error(
conn: sqlite3.Connection,
*,
allow_missing: bool = True,
annotations_valid: bool = False,
) -> Optional[str]:
"""Reject partial or unsupported review schemas and invalid rows.
Reads only: nothing is ever repaired. A database without any of the
group's objects is acceptable for backups that predate it unless
``allow_missing`` is false. A present group requires a present, valid
image annotation group; ``annotations_valid`` says the caller has just
validated it (startup's annotation install), so only its presence is
checked again.
"""
objects = _catalog(conn)
if not objects:
return None if allow_missing else "Missing review queue schema"
if annotations_valid:
annotation_error = review_without_annotations_error(conn)
else:
annotation_error = annotation_schema_error(conn, allow_missing=False)
if annotation_error:
annotation_error = (
"Review queue schema requires a valid image annotation schema: "
+ annotation_error
)
if annotation_error:
return annotation_error
present = {name for kind, name, _sql in objects if kind == "table"}
if present != set(TABLES):
return "Incomplete review queue schema"
actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
if actual != _expected_catalog():
return "Review queue has an invalid schema definition"
return _structure_error(conn) or _value_error(conn)
def migrate_review_state(conn: sqlite3.Connection) -> None:
"""Install the group atomically; validate before and after creating it.
Runs after ``migrate_annotation_state`` (which has just validated the
annotation group): the group is only valid beside the annotation group it
cites. A present group is validated once, evidence included.
"""
conn.execute("SAVEPOINT media_review_schema")
try:
installed = bool(_catalog(conn))
error = review_schema_error(conn, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
if installed:
# Already complete and valid: nothing to create, and the evidence
# pass is the expensive part of validation, so it runs once.
conn.execute("RELEASE media_review_schema")
return
for sql in _TABLE_DDL.values():
conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
for sql in _INDEX_DDL.values():
conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
conn.execute(
f"INSERT OR IGNORE INTO {METADATA_TABLE} "
"(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)",
(SCHEMA_VERSION,),
)
error = review_schema_error(conn, allow_missing=False, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE media_review_schema")
except BaseException:
conn.execute("ROLLBACK TO media_review_schema")
conn.execute("RELEASE media_review_schema")
raise
+34 -7
View File
@@ -8,6 +8,9 @@ Split into domain-specific mixins for maintainability:
- AutotagRoutesMixin: auto-tagging model management and tagging
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
- SystemRoutesMixin: System health check, Overview and usage statistics
"""
import asyncio
@@ -15,6 +18,7 @@ import datetime
import functools
import gzip as gzip_module
import json
import math
import ntpath
import os
import re
@@ -28,6 +32,8 @@ from .pairs import PairRoutesMixin
from .library import LibraryRoutesMixin
from .vocabulary import VocabularyRoutesMixin
from .annotations import AnnotationRoutesMixin
from .review import ReviewRoutesMixin
from .review_recording import ReviewRecordingMixin
from .saved_views import SavedViewRoutesMixin
from .history import HistoryRoutesMixin
from .images import ImageRoutesMixin
@@ -35,6 +41,7 @@ from .admin import AdminRoutesMixin
from .settings_routes import SettingsRoutesMixin
from .job_routes import JobRoutesMixin
from .image_tag_jobs import ImageTagJobsMixin
from .system import SystemRoutesMixin
from .jobs import JobRegistry
from .logging_routes import LoggingRoutesMixin
from .autotag_routes import AutotagRoutesMixin
@@ -261,6 +268,8 @@ class PromptManagerAPI(
LibraryRoutesMixin,
VocabularyRoutesMixin,
AnnotationRoutesMixin,
ReviewRoutesMixin,
ReviewRecordingMixin,
SavedViewRoutesMixin,
HistoryRoutesMixin,
PromptRoutesMixin,
@@ -269,6 +278,7 @@ class PromptManagerAPI(
SettingsRoutesMixin,
JobRoutesMixin,
ImageTagJobsMixin,
SystemRoutesMixin,
LoggingRoutesMixin,
AutotagRoutesMixin,
LoraIntegrationMixin,
@@ -300,6 +310,8 @@ class PromptManagerAPI(
# created on first use with their restore subscription.
self._captures = None
self._captures_unsubscribe = None
# Set by close_annotation_captures (shutdown): no store is recreated.
self._captures_closed = False
self._cached_output_dir = None # Lazy-cached by _find_comfyui_output_dir()
self._html_cache = {} # Cached HTML file contents keyed by path
self._gallery_cache = {} # dict: path_str -> (files, timestamp)
@@ -325,6 +337,11 @@ class PromptManagerAPI(
self._scan_job = None
# The current duplicate scan (a py/api/jobs.Job outside the registry).
self._duplicate_job = None
# The last System health report this session and the database
# generation it describes, bumped when a restore replaces self.db
# (py/api/system.py).
self._health_report = None
self._db_generation = 0
# Run cleanup on initialization to remove any existing duplicates
try:
@@ -475,6 +492,7 @@ class PromptManagerAPI(
self._register_library_routes(routes)
self._register_vocabulary_routes(routes)
self._register_annotation_routes(routes)
self._register_review_routes(routes)
self._register_saved_view_routes(routes)
self._register_history_routes(routes)
self._register_prompt_routes(routes)
@@ -482,6 +500,7 @@ class PromptManagerAPI(
self._register_admin_routes(routes)
self._register_settings_routes(routes)
self._register_job_routes(routes)
self._register_system_routes(routes)
self._register_logging_routes(routes)
self._register_autotag_routes(routes)
self._register_lora_routes(routes)
@@ -600,12 +619,12 @@ class PromptManagerAPI(
return prompts
def _clean_nan_recursive(self, obj):
"""Recursively clean NaN values from nested data structures."""
"""Recursively replace NaN and Infinity in nested data with None."""
if isinstance(obj, dict):
return {key: self._clean_nan_recursive(value) for key, value in obj.items()}
elif isinstance(obj, list):
return [self._clean_nan_recursive(item) for item in obj]
elif isinstance(obj, float) and str(obj) == "nan":
elif isinstance(obj, float) and not math.isfinite(obj):
return None
else:
return obj
@@ -728,6 +747,13 @@ class PromptManagerAPI(
Returns:
List[Path]: Validated directory paths, possibly empty.
"""
return self._output_dirs_with_origin()[0]
def _output_dirs_with_origin(self):
"""``(dirs, auto_detected)``: the ``_get_all_output_dirs`` list, and
whether it comes from detection or ComfyUI's output rather than from
a valid configured root (False when there is no directory at all).
"""
from ..config import GalleryConfig
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
@@ -746,15 +772,16 @@ class PromptManagerAPI(
output_dirs.append(Path(real))
if output_dirs:
return output_dirs
return output_dirs, False
if configured:
# Every configured root was rejected: never let auto-detection
# resurrect it, serve ComfyUI's own output directory instead.
return self._comfyui_output_dir_list()
fallback = self._find_comfyui_output_dir()
return [Path(fallback)] if fallback else []
fallback_dirs = self._comfyui_output_dir_list()
else:
fallback = self._find_comfyui_output_dir()
fallback_dirs = [Path(fallback)] if fallback else []
return fallback_dirs, bool(fallback_dirs)
def _comfyui_output_dir_list(self):
"""ComfyUI's output directory as a one-element list, or [] if unknown."""
+132 -32
View File
@@ -4,6 +4,7 @@ import asyncio
import contextlib
from contextlib import closing
import datetime
import functools
import hashlib
import json
import math
@@ -41,9 +42,10 @@ try:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_js_json,
parse_page_json,
)
from ...utils.prompt_pair_resolver import resolve_prompt_pair
from ...utils.review_capture import reviewable
except ImportError:
from database.operations import PromptDatabase
from database.pair_ingestion import ingest_resolved_pair
@@ -51,9 +53,10 @@ except ImportError:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_js_json,
parse_page_json,
)
from utils.prompt_pair_resolver import resolve_prompt_pair
from utils.review_capture import reviewable
try:
@@ -211,6 +214,30 @@ def _time_left(done, total, batches, elapsed):
return f" · about {minutes} min left"
def new_review_counts():
"""The scan's Needs review tally: ``added`` new items, ``queued`` files
in the queue after this scan (new or already there), ``settled`` items
removed because their file is linked now, ``untraced`` items removed
because they offer nothing to choose, and ``errors``."""
return {"added": 0, "queued": 0, "settled": 0, "untraced": 0, "errors": 0}
def _review_scope(review):
"""The scan job's ``scope`` keys for its Needs review tally.
``review_items`` is the number of this scan's files that are in the
pending queue after it: the review group's ambiguous files, less those a
capture could not read or a decision already settled.
"""
return {
"review_items": review["queued"],
"review_added": review["added"],
"review_settled": review["settled"],
"review_untraced": review["untraced"],
"review_errors": review["errors"],
}
def _scan_status_line(done, total, batches, elapsed):
""" "Processing file N of M", with an estimate once enough batches ran."""
line = f"Processing file {done:,} of {total:,}"
@@ -804,6 +831,9 @@ class AdminRoutesMixin:
result = await self._run_in_executor(
self._delete_duplicate_images_sync, image_paths, output_dir
)
if result["deleted_count"]:
# The cached listing would keep (and count) the deleted files.
self.invalidate_gallery_cache()
response_data = {
"success": True,
@@ -1271,6 +1301,7 @@ class AdminRoutesMixin:
self.logger.info(f"Removed {len(removed)} old safety backups")
previous = self.db
self.db = PromptDatabase(model.db_path)
self._database_replaced() # a health report described the old one
# Its connections would otherwise linger (a file lock on Windows)
previous.close_all()
prompt_count = self.db.model.get_database_info().get("total_prompts", 0)
@@ -1340,28 +1371,34 @@ class AdminRoutesMixin:
}
)
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE):
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE, roots=None, output_dirs=None):
"""Register the scan job synchronously; its runner starts on the loop.
``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another
record-changing job runs.
``files`` is the scope (``SCAN_SCOPES``). ``roots`` selects sources by
their index in ``output_dirs`` (the effective sources the caller
resolved); None scans every source, resolved when the job starts.
Raises JobBusy while another record-changing job runs.
"""
if files not in SCAN_SCOPES:
raise ValueError(f"Unknown scan scope: {files!r}")
if roots is not None and (
output_dirs is None or not roots or max(roots) >= len(output_dirs)
):
raise ValueError("A scoped scan needs the sources its roots index")
job = self._jobs.start(
SCAN_JOB_KIND,
SCAN_JOB_TITLE,
record_changing=True,
scope={"files": files},
runner=self._run_scan_job,
scope={"files": files, "roots": None if roots is None else list(roots)},
runner=functools.partial(self._run_scan_job, output_dirs=output_dirs),
groups=SCAN_GROUP_NAMES,
)
self._scan_job = job
return job
async def _run_scan_job(self, job):
async def _run_scan_job(self, job, output_dirs=None):
final = None
async for payload in self._scan_images_events(job):
async for payload in self._scan_images_events(job, output_dirs):
job.publish(payload)
final = payload.get("type")
state = SCAN_FINAL_STATES.get(final, "finished")
@@ -1396,16 +1433,25 @@ class AdminRoutesMixin:
)
return response
async def _scan_images_events(self, job):
"""Yield progress payloads while scanning every configured output directory.
async def _scan_images_events(self, job, output_dirs=None):
"""Yield progress payloads while scanning the output directories.
The payloads are the legacy SSE events, unchanged; ``job`` also gets
its status line, totals and one outcome per processed file.
Every effective source is walked, or only those ``job.scope["roots"]``
selects (indexes into ``output_dirs``). The payloads are the legacy SSE
events, unchanged; ``job`` also gets its status line, totals and one
outcome per processed file. Rows keep each source's listing index.
"""
try:
self.logger.info("Starting image scan operation")
output_dirs = self._get_all_output_dirs()
if not output_dirs:
if output_dirs is None:
output_dirs = self._get_all_output_dirs()
selected = job.scope.get("roots")
walked = (
output_dirs
if selected is None
else [output_dirs[index] for index in selected]
)
if not walked:
self.logger.error("No output directories found")
message = (
"No output directories found. "
@@ -1415,18 +1461,18 @@ class AdminRoutesMixin:
yield {"type": "error", "message": message}
return
dir_names = [self._public_path(d) for d in output_dirs]
folders = _plural(len(output_dirs), "output folder")
job.scope["output_folders"] = len(output_dirs)
dir_names = [self._public_path(d) for d in walked]
folders = _plural(len(walked), "output folder")
job.scope["output_folders"] = len(walked)
scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")]
job.log_line(f"Scan started · {folders} · {scope}")
job.update(status_line=f"Looking for media files in {folders}")
yield self._scan_progress(
0, f"Scanning {len(output_dirs)} directory(ies) for media files..."
0, f"Scanning {len(walked)} directory(ies) for media files..."
)
media_files = await self._run_in_executor(
self._collect_output_media_sync, output_dirs
self._collect_output_media_sync, walked
)
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
job.update(
@@ -1441,10 +1487,13 @@ class AdminRoutesMixin:
known = set()
if media_files and job.scope.get("files") != "all":
known = await self._known_scan_files(job, media_files)
review = new_review_counts()
job.scope.update(_review_scope(review))
async for event in self._scan_batches(
media_files, counts, job, roots, known
media_files, counts, job, roots, known, review
):
yield event
self._log_review_items(job, review)
self.logger.info(
f"Scan completed: processed={counts['processed']}, "
@@ -1481,12 +1530,16 @@ class AdminRoutesMixin:
sizes = {str(media_file): _file_size(media_file) for media_file in media_files}
return self.db.get_known_resolved_paths(sizes)
async def _scan_batches(self, media_files, counts, job, roots=(), known=()):
async def _scan_batches(
self, media_files, counts, job, roots=(), known=(), review=None
):
"""Process ``media_files`` in executor batches, updating ``counts`` in place.
Records each file's outcome in ``job`` and yields one progress event
per batch. Files in ``known`` (path strings) are not read: they count
as processed and unchanged and are recorded as skipped.
as processed and unchanged and are recorded as skipped. ``review``
counts the Needs review items the batches record (``job.scope``
``review_items`` / ``review_errors`` follow it after every batch).
"""
total = len(media_files)
started = time.monotonic()
@@ -1509,9 +1562,16 @@ class AdminRoutesMixin:
batch_results,
roots=roots,
outcomes=outcomes,
review=review,
)
for key in counts:
counts[key] += batch_counts[key]
if review is not None:
# After the batch's links: a file linked now leaves the queue.
await self._run_in_executor(
self._settle_scanned_reviews, batch, roots, review
)
job.scope.update(_review_scope(review))
for group, row, log_line in outcomes:
job.record(group, row, log_line)
@@ -1561,12 +1621,20 @@ class AdminRoutesMixin:
return map_parallel(extract, file_batch, PromptManagerConfig.WORKER_THREADS)
def _ingest_scan_batch_sync(self, batch_results, roots=(), outcomes=None):
def _ingest_scan_batch_sync(
self, batch_results, roots=(), outcomes=None, review=None
):
"""Store the prompts found in a metadata batch (blocking DB work).
Returns the legacy counts. When ``outcomes`` is a list, every processed
file appends its ``(group, row, log_line)`` for the scan job, with the
path relative to its root in ``roots`` (see ``_scan_roots``).
path relative to its root in ``roots`` (see ``_scan_roots``). A file
whose final outcome is ambiguous and unlinked (the resolver found it
ambiguous and the fallback did not link it) is recorded for Needs
review (best effort, counted in ``review`` when given); one a saved
row already links is reported ``unchanged`` and never queued. An
ambiguous file with nothing to choose is reported ``unresolved`` and
its pending items without a choice leave the queue.
"""
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
@@ -1578,13 +1646,27 @@ class AdminRoutesMixin:
if self._scan_job and self._scan_job.cancelled:
break
counts["processed"] += 1
ambiguous, untraced = [], []
try:
outcome = self._ingest_scanned_file(media_file, metadata, detailed=True)
outcome = self._ingest_scanned_file(
media_file,
metadata,
detailed=True,
ambiguous=ambiguous,
untraced=untraced,
)
except Exception as e:
self.logger.error(f"Error processing {media_file.name}: {e}")
counts["failed"] += 1
note(media_file, "error")
continue
if ambiguous and outcome == "ambiguous":
if self._review_file_linked(media_file, roots):
outcome = "unchanged" # already linked: not a review case
else:
self._record_scanned_review(media_file, roots, review)
elif untraced and outcome == "unresolved":
self._settle_untraced_reviews(untraced, roots, review)
if outcome is None:
note(media_file, None)
continue
@@ -1602,11 +1684,14 @@ class AdminRoutesMixin:
@staticmethod
def _graph_prompt_text(metadata):
"""The positive prompt the browser parser traces from the graphs, or None."""
"""The positive prompt the browser parser traces from the graphs, or None.
Records parse as the legacy pages parsed them (NaN reads as null).
"""
records = {}
for key in ("prompt", "workflow"):
value = metadata.get(key)
records[key] = parse_js_json(value) if isinstance(value, str) else value
records[key] = parse_page_json(value) if isinstance(value, str) else value
try:
positive = extract_generation_params(records)["positivePrompt"]
except GenerationParamsError:
@@ -1615,13 +1700,21 @@ class AdminRoutesMixin:
return None
return positive.strip() or None
def _ingest_scanned_file(self, media_file, metadata, detailed=False):
def _ingest_scanned_file(
self, media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
"""Enrich exact pairs; retain legacy positive-only import for absent pairs.
Returns the file's scan status, or None without prompt metadata. With
``detailed``, a stored or linked file returns a dict instead: its
``status``, the prompt's display ``title`` and, for a resolved pair,
the pair counts.
the pair counts. When the resolver finds the file ambiguous (whether
or not the browser parser then links it) and ``ambiguous`` is a list,
``media_file`` is appended to it; only a file that then returns
``"ambiguous"`` (nothing linked it) is a Needs review case. An
ambiguous result with nothing to choose (``reviewable`` is False: fewer
than two candidates carry a prompt) that nothing links returns
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
"""
if not metadata:
return None
@@ -1637,6 +1730,13 @@ class AdminRoutesMixin:
return None
_with_file_size(canonical, media_file)
resolution = resolve_prompt_pair(canonical)
# Ambiguous without a choice: unresolved unless something links it.
unchoosable = resolution["state"] == "ambiguous" and not reviewable(resolution)
if unchoosable:
if untraced is not None:
untraced.append(media_file)
elif resolution["state"] == "ambiguous" and ambiguous is not None:
ambiguous.append(media_file)
if resolution["state"] == "resolved":
result = ingest_resolved_pair(
self.db, str(media_file), canonical, resolution
@@ -1656,9 +1756,9 @@ class AdminRoutesMixin:
# Graphs without a recognized sampler keep the legacy extractor below.
prompt_text = self._graph_prompt_text(metadata)
if not prompt_text:
return resolution["state"]
return "unresolved" if unchoosable else resolution["state"]
elif resolution["state"] == "ambiguous":
return "ambiguous"
return "unresolved" if unchoosable else "ambiguous"
elif resolution["state"] == "unresolved" and not isinstance(
canonical.get("parameters"), str
):
+35 -13
View File
@@ -13,6 +13,7 @@ import datetime
import functools
import os
import re
import threading
from pathlib import Path
from aiohttp import web
@@ -73,6 +74,9 @@ _HEX64 = re.compile(r"[0-9a-f]{64}")
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
# secrets.token_urlsafe(32) is 43 characters; the bound only rejects junk.
_TOKEN = re.compile(r"[A-Za-z0-9_-]{1,128}")
# The store is created on first use by the event loop, the scan's executor or
# the monitor's worker (Needs review), so its creation is serialized.
_CAPTURE_STORE_LOCK = threading.Lock()
_PREVIEW_TYPES = {
".png": "image/png",
@@ -259,31 +263,46 @@ class AnnotationRoutesMixin:
Created lazily so an API object that never observes a file (most tests,
a server that never opens the Tag dialog) owns no temp directory. A
restore of the database dooms every capture (``invalidate_all``).
Thread-safe: Needs review recording calls it off the event loop.
Raises ``CaptureError`` after ``close_annotation_captures``.
"""
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
with _CAPTURE_STORE_LOCK:
if getattr(self, "_captures_closed", False):
# A recording still in flight at shutdown must not create a
# new temp directory after the cleanup removed the old one.
raise CaptureError("The capture store is closed")
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
def _invalidate_captures(self) -> None:
store = self._captures
if store is not None:
store.invalidate_all()
def close_annotation_captures(self) -> None:
"""Remove every capture file and the owned directory. Idempotent."""
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
def close_annotation_captures(self, *, final: bool = False) -> None:
"""Remove every capture file and the owned directory. Idempotent.
Taken under the store lock. ``final`` (the shutdown cleanup) also makes
``_capture_store`` refuse to create another store, so a recording
still in flight cannot leak a new temp directory after it.
"""
with _CAPTURE_STORE_LOCK:
if final:
self._captures_closed = True
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
store, self._captures = self._captures, None
if unsubscribe is not None:
unsubscribe()
store, self._captures = self._captures, None
if store is not None:
store.close()
async def cleanup_annotation_captures(self, app: web.Application) -> None:
self.close_annotation_captures()
self.close_annotation_captures(final=True)
# -- shared request steps --------------------------------------------------
@@ -329,7 +348,10 @@ class AnnotationRoutesMixin:
root, error = self._scoped_root(index, scope)
if error is not None:
return error
store = self._capture_store()
try:
store = self._capture_store()
except CaptureError as exc: # the server is shutting down
return _capture_error(exc)
try:
epoch = await self._run_in_executor(self.db.get_execution_epoch)
capture = await _settled(
+9 -21
View File
@@ -3,6 +3,7 @@
import asyncio
import base64
import binascii
import functools
import hashlib
import hmac
import json
@@ -37,6 +38,7 @@ try:
from ...database.models import normalize_image_path
from ...utils.card_seed import card_seed, header_params, parsed_positive
from ...utils.parallel import map_parallel
from ...utils.json_strict import dumps_strict
from ...utils.inspection_response import (
encode_inspection_response,
library_metadata,
@@ -56,6 +58,7 @@ except ImportError:
from database.models import normalize_image_path
from utils.card_seed import card_seed, header_params, parsed_positive
from utils.parallel import map_parallel
from utils.json_strict import dumps_strict
from utils.inspection_response import (
encode_inspection_response,
library_metadata,
@@ -982,27 +985,12 @@ class ImageRoutesMixin:
# Add url and thumbnail_url so the frontend can serve them
cleaned_images = self._present_images(cleaned_images)
# Additional fallback: convert to JSON string and clean NaN values manually
try:
response_data = {"success": True, "images": cleaned_images}
# Convert to JSON string
json_str = json.dumps(response_data, default=str)
# Clean any remaining NaN values with regex
json_str = re.sub(r":\s*NaN", ": null", json_str)
json_str = re.sub(r"\[\s*NaN\s*\]", "[null]", json_str)
json_str = re.sub(r",\s*NaN\s*,", ", null,", json_str)
json_str = re.sub(r",\s*NaN\s*\]", ", null]", json_str)
json_str = re.sub(r"\[\s*NaN\s*,", "[null,", json_str)
# Parse back to verify it's valid JSON
cleaned_data = json.loads(json_str)
return web.json_response(cleaned_data)
except Exception as json_error:
self.logger.error(f"JSON cleaning error: {json_error}")
# Fallback to original response
return web.json_response({"success": True, "images": cleaned_images})
# Rows an older version stored may still hold NaN/Infinity: answer
# strict JSON, and leave prompt text that spells "NaN" untouched.
return web.json_response(
{"success": True, "images": cleaned_images},
dumps=functools.partial(dumps_strict, default=str),
)
except Exception as e:
self.logger.error(f"Get prompt images error: {e}")
+93 -17
View File
@@ -18,29 +18,74 @@ def _not_found():
return web.json_response({"success": False, "error": "Job not found"}, status=404)
def _bad_request(message):
return web.json_response({"success": False, "error": message}, status=400)
MAX_SCAN_ROOTS = 64
_SCOPE_ERROR = 'The scan scope must be "new" or "all"'
_ROOTS_ERROR = (
f"roots must be a non-empty list of up to {MAX_SCAN_ROOTS} source indexes, "
"with paths naming the public path of each"
)
SOURCES_CHANGED_ERROR = "The sources changed since they were loaded. Reload them."
async def _scan_scope(request):
"""The requested scan scope ("new" or "all"), or None when the body is invalid.
def _bad_request(message, code=None):
body = {"success": False, "error": message}
if code is not None:
body["code"] = code
return web.json_response(body, status=400)
An empty body (also a chunked one with no bytes), or one without
``scope``, asks for the default.
def _scan_roots(body):
"""``(roots, error)``: ``{index: public path}`` of the selected sources, or None.
Absent ``roots`` means every source. When present it is a non-empty list
of at most MAX_SCAN_ROOTS non-negative ints (never bools or floats), and
``paths`` the same-length list of the public path the caller read for
each (GET /settings), so a stale index never scans another folder.
"""
if "roots" not in body:
return None, None
raw = body["roots"]
paths = body.get("paths")
if not isinstance(raw, list) or not raw or len(raw) > MAX_SCAN_ROOTS:
return None, _ROOTS_ERROR
if not isinstance(paths, list) or len(paths) != len(raw):
return None, _ROOTS_ERROR
selected = {}
for value, path in zip(raw, paths):
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
return None, _ROOTS_ERROR
if not isinstance(path, str) or selected.get(value, path) != path:
return None, _ROOTS_ERROR
selected[value] = path
return dict(sorted(selected.items())), None
async def _scan_request(request):
"""``({files, roots}, None)`` for a scan start, or ``(None, message)``.
``files`` is the scope ("new" or "all"); ``roots`` the selected source
indexes or None for every source. An empty body (also a chunked one with
no bytes), or one without ``scope``, asks for the default scope.
"""
default = {"files": DEFAULT_SCAN_SCOPE, "roots": None}
if not request.can_read_body:
return DEFAULT_SCAN_SCOPE
return default, None
text = await request.text()
if not text.strip(): # e.g. a chunked POST that sent no bytes
return DEFAULT_SCAN_SCOPE
return default, None
try:
body = json.loads(text)
except ValueError:
return None
return None, _SCOPE_ERROR
if not isinstance(body, dict):
return None
scope = body.get("scope", DEFAULT_SCAN_SCOPE)
return scope if scope in SCAN_SCOPES else None
return None, _SCOPE_ERROR
files = body.get("scope", DEFAULT_SCAN_SCOPE)
if files not in SCAN_SCOPES:
return None, _SCOPE_ERROR
roots, error = _scan_roots(body)
if error:
return None, error
return {"files": files, "roots": roots}, None
def busy_response(busy: JobBusy):
@@ -64,6 +109,13 @@ def _summary_event(job):
class JobRoutesMixin:
"""Mixin providing the /prompt_manager/jobs endpoints."""
def _roots_still_listed(self, selected, output_dirs):
"""Whether each ``{index: public path}`` still names that source."""
return all(
index < len(output_dirs) and self._public_path(output_dirs[index]) == path
for index, path in selected.items()
)
def _register_job_routes(self, routes):
@routes.get("/prompt_manager/jobs")
async def list_jobs_route(request):
@@ -132,13 +184,37 @@ class JobRoutesMixin:
"""Start the metadata scan as a job (202), or 409 naming the busy job.
The body's ``scope`` is "new" (the default: new and unresolved files)
or "all"; anything else is a 400.
or "all". ``roots``, when given, selects sources by their index in
the effective sources (``_get_all_output_dirs``), and ``paths`` names
the public path read for each. An index past them, or one whose
source now has another public path (the list changed since it was
read), is a 409 ``sources-changed`` and starts nothing. The job walks
the sources resolved here, so a later source change cannot move it.
"""
files = await _scan_scope(request)
if files is None:
return _bad_request('The scan scope must be "new" or "all"')
scan, error = await _scan_request(request)
if error:
return _bad_request(error)
output_dirs = None
roots = None
if scan["roots"] is not None:
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
listed = await self._run_in_executor(
self._roots_still_listed, scan["roots"], output_dirs
)
if not listed:
return web.json_response(
{
"success": False,
"code": "sources-changed",
"error": SOURCES_CHANGED_ERROR,
},
status=409,
)
roots = list(scan["roots"])
try:
job = self._start_scan_job(files)
job = self._start_scan_job(
scan["files"], roots=roots, output_dirs=output_dirs
)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
+7 -3
View File
@@ -88,6 +88,9 @@ DUPLICATE_CLEANUP_JOB_KIND = "duplicate_cleanup"
THUMBNAILS_JOB_KIND = "thumbnails"
LORA_IMPORT_JOB_KIND = "lora_import"
AUTOTAG_BATCH_JOB_KIND = "autotag"
# The System health check (py/api/system.py): a read-only ``read`` job with
# no file outcomes, so it may run beside a scan.
HEALTH_JOB_KIND = "health"
def resolve_admission(admission=None, record_changing=None) -> str:
@@ -432,7 +435,8 @@ class JobRegistry:
if same_kind is not None:
return same_kind
refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)]
running = [job for job in self._jobs.values() if job.running]
# A snapshot: the monitor's worker thread asks too (Needs review).
running = [job for job in tuple(self._jobs.values()) if job.running]
running.sort(key=lambda job: job._started_seq)
for job in running:
if job.admission in refused:
@@ -463,8 +467,8 @@ class JobRegistry:
return self._jobs.get(job_id)
def active(self, kind):
"""The running job of ``kind``, or None."""
for job in self._jobs.values():
"""The running job of ``kind``, or None (safe off the event loop)."""
for job in tuple(self._jobs.values()):
if job.kind == kind and job.running:
return job
return None
+689
View File
@@ -0,0 +1,689 @@
"""Needs review routes (#249 E3): the pending queue, item detail, decisions.
``GET /prompt_manager/review/items`` lists pending items oldest first behind
an HMAC keyset cursor bound to the storage epoch and the review index
revision (any decision, new item or restore makes an old cursor stale);
``total`` is the exact pending count. ``GET .../items/{id}`` adds the
candidates and the dependency fingerprint. ``POST .../items/{id}/decision``
records one explicit decision: Apply links the chosen sampler's pair only
when the capture is the item's bytes, the current file still hashes to them,
the evidence they yield is unchanged and the ingester's compatible-only rules
allow it; Leave unresolved links nothing.
Targets are formatted from the recorded spelling against the current roots
(no file access beyond the roots' real paths); a file under no current root
is ``unavailable``. Responses carry no server path, no locator and no
exception text.
"""
import base64
import binascii
import hashlib
import hmac
import json
import os
import re
from aiohttp import web
from .annotations import _HEX64, _TOKEN, _UUID, _capture_error, _error
from .prompts import bad_request
try:
from ...database.library import display_title
from ...database.library_commands import LibraryCommandError
from ...database.review_decisions import ReviewCommandError
from ...utils.media_capture import CaptureError
from ...utils.review_capture import captured_review, review_evidence
except ImportError:
from database.library import display_title
from database.library_commands import LibraryCommandError
from database.review_decisions import ReviewCommandError
from utils.media_capture import CaptureError
from utils.review_capture import captured_review, review_evidence
ITEMS_URL = "/prompt_manager/review/items"
DEFAULT_LIMIT = 40
MAX_LIMIT = 40
MAX_CURSOR = 4096
MAX_BODY_BYTES = 64 * 1024
# The serialized detail stays under this; past it the detail is ``limited``.
MAX_DETAIL_BYTES = 8 * 1024 * 1024
UNRESOLVED = "unresolved"
_DECISION_FIELDS = frozenset(
{
"expected_epoch",
"expected_revision",
"capture_token",
"sha256",
"dependency_fingerprint",
"choice",
}
)
_CURSOR_SECRET = os.urandom(32)
# The real path of a spelling; module-level so a test can fake a junction.
_realpath = os.path.realpath
_LIMIT = re.compile(r"[0-9]{1,3}")
_NO_STORE = {"Cache-Control": "no-store"}
_STATUS = {
"review-missing": 404,
"review-conflict": 409,
"association-conflict": 409,
"file-changed": 409,
"storage-changed": 409,
"source-changed": 409,
"capture-expired": 409,
}
_MESSAGES = {
"review-missing": "This review item no longer exists. Reload the queue.",
"review-conflict": (
"This item was decided or changed since it was opened. Reload the queue."
),
"association-conflict": (
"This image is already linked to another prompt or negative, so this "
"sampler path cannot be attached safely. The existing links were "
"preserved."
),
"dependencies-changed": (
"The image's saved links changed since this item was opened. Reopen it "
"to see the current association."
),
"file-changed": (
"The file changed since it was reviewed. A metadata scan adds its new "
"bytes as a new item."
),
"storage-changed": "Prompt storage was restored; reload the review queue.",
"source-changed": (
"The source changed. Refresh the image collection before reviewing this "
"file."
),
"capture-expired": "The preview expired. Open the item again.",
"other-file": "The preview belongs to another file. Open the item again.",
"evidence-changed": (
"This file's metadata no longer gives the evidence that was reviewed. "
"Reload the queue."
),
}
_DEFAULT_MESSAGE = "The decision could not be saved. Reload the queue."
def _coded(code: str, message_key: str | None = None, **extra) -> web.Response:
message = _MESSAGES.get(message_key or code, _DEFAULT_MESSAGE)
return _error(_STATUS.get(code, 409), code, message, **extra)
def _failed() -> web.Response:
return _error(500, "review-failed", "The review queue could not be read.")
def format_review_target(spelled_path, locator, real_roots, *, pathmod=os.path):
"""``(root_index, relative)`` of the first root that lists the file, or None.
``spelled_path`` is the file's recorded spelling, ``locator`` its folded
identity and ``real_roots`` ``(index, realpath)`` pairs. The relative
path keeps the recorded spelling (a Windows locator is case-folded, a
listing target is not) and must fold back to the locator.
"""
sep = pathmod.sep
for index, real_root in real_roots:
head = spelled_path[: len(real_root)]
if pathmod.normcase(head) != pathmod.normcase(real_root):
continue
rest = spelled_path[len(real_root) :]
if not real_root.endswith(sep):
if not rest.startswith(sep):
continue
rest = rest[1:]
parts = rest.split(sep)
if any(
part in ("", ".", "..") or "/" in part or "\\" in part or ":" in part
for part in parts
):
continue
if pathmod.normcase(pathmod.join(real_root, *parts)) != locator:
continue
return index, "/".join(parts)
return None
class _StaleCursor(ValueError):
"""A well-formed cursor this process cannot verify (e.g. after a restart)."""
def _encode_cursor(epoch: str, revision: int, key: tuple) -> str:
payload = json.dumps(
{"v": 1, "e": epoch, "r": revision, "k": list(key)},
separators=(",", ":"),
ensure_ascii=True,
).encode()
signature = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=")
def _decode_cursor(token: str) -> tuple:
"""``(epoch, revision, (created_at, id))`` of a cursor.
``ValueError`` when malformed; :class:`_StaleCursor` when well-formed but
signed by another secret (the per-process secret changes on every start,
so an open page's cursor is stale after a restart, not forged).
"""
try:
if len(token) > MAX_CURSOR:
raise ValueError()
raw = base64.b64decode(
token + "=" * (-len(token) % 4), altchars=b"-_", validate=True
)
signature, payload = raw[:32], raw[32:]
if len(signature) != 32 or not payload:
raise ValueError()
expected = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
if not hmac.compare_digest(signature, expected):
raise _StaleCursor("This page of the queue is out of date")
data = json.loads(payload)
key = data["k"]
if (
data["v"] != 1
or not isinstance(data["e"], str)
or type(data["r"]) is not int
or not isinstance(key, list)
or len(key) != 2
or not all(isinstance(part, str) for part in key)
):
raise ValueError()
return data["e"], data["r"], tuple(key)
except _StaleCursor:
raise
except (ValueError, KeyError, TypeError, UnicodeError, binascii.Error):
raise ValueError("Invalid review cursor; reload the queue") from None
def _limit(value):
if value is None:
return DEFAULT_LIMIT
if _LIMIT.fullmatch(value) is None or not 1 <= int(value) <= MAX_LIMIT:
return None
return int(value)
def _side_text(side) -> str | None:
if isinstance(side, dict) and isinstance(side.get("text"), str):
return side["text"]
return None
def _workable(candidate: dict) -> bool:
return (
candidate.get("state") == "resolved"
and _side_text(candidate.get("positive")) is not None
and _side_text(candidate.get("negative")) is not None
)
def _side_label(titles: dict, side, empty: str) -> str:
"""The Library title of the side's saved prompt, else its first line."""
text = _side_text(side)
if text is None:
return empty
stored = side.get("storage_text")
stored = stored if isinstance(stored, str) else text
return display_title(titles.get(stored, ()), text) or empty
class ReviewRoutesMixin:
"""The Needs review queue over E1's schema and E2's recorded items."""
def _register_review_routes(self, routes: web.RouteTableDef) -> None:
routes.get(ITEMS_URL)(self.list_review_items)
routes.get(ITEMS_URL + "/{item_id}")(self.get_review_item)
routes.post(ITEMS_URL + "/{item_id}/decision")(self.decide_review_item)
# -- shared ----------------------------------------------------------------
def _review_roots(self):
"""``(roots, real_roots)``: the current gallery roots and their realpaths.
Blocking (it lists the roots and resolves them): call it on a worker.
"""
roots = list(self._get_all_output_dirs())
return roots, [(i, _realpath(str(root))) for i, root in enumerate(roots)]
def _review_view(self, row: dict, roots, real_roots) -> dict:
"""The public ReviewItem of one stored row (no path, no locator)."""
found = format_review_target(row["spelled_path"], row["locator"], real_roots)
target = None
if found is not None:
index, relative = found
target = {
"root_index": index,
"root_scope": self._gallery_root_scope(roots[index], index),
"path": relative,
}
return {
"id": row["id"],
"state": row["state"],
"revision": row["revision"],
"created_at": row["created_at"],
"filename": os.path.basename(row["spelled_path"]),
"target": target,
"unavailable": target is None,
"candidate_count": row["candidate_count"],
"limited": row["limited"],
}
@staticmethod
def _review_image_path(row: dict, roots, target) -> str:
"""The path the scan and the monitor link this file under."""
if target is None:
return row["spelled_path"]
root = roots[target["root_index"]]
return os.path.join(str(root), *target["path"].split("/"))
# -- queue -----------------------------------------------------------------
async def list_review_items(self, request: web.Request) -> web.Response:
"""GET ``?limit=1..40&cursor=``: one page of pending items."""
limit = _limit(request.query.get("limit"))
if limit is None:
return bad_request(f"limit must be an integer from 1 to {MAX_LIMIT}")
bound = None
cursor = request.query.get("cursor")
if cursor:
try:
bound = _decode_cursor(cursor)
except _StaleCursor:
return await self._stale_cursor()
except ValueError as exc:
return bad_request(str(exc))
try:
page, views = await self._run_in_executor(
self._review_page, limit, bound[2] if bound else None
)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
if bound is not None and bound[0] != page["epoch"]:
return _coded("storage-changed", library_epoch=page["epoch"])
if bound is not None and bound[1] != page["revision"]:
return _coded("review-conflict", total=page["total"])
items = page["items"]
next_cursor = None
if page["more"] and items:
last = items[-1]
next_cursor = _encode_cursor(
page["epoch"], page["revision"], (last["created_at"], last["id"])
)
return web.json_response(
{
"success": True,
"library_epoch": page["epoch"],
"total": page["total"],
"items": views,
"next_cursor": next_cursor,
},
headers=_NO_STORE,
)
async def _stale_cursor(self) -> web.Response:
"""409 ``review-conflict`` with the current total: reload the queue."""
try:
page = await self._run_in_executor(self.db.review_queue_page, limit=1)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
return _coded(
"review-conflict", total=page["total"], library_epoch=page["epoch"]
)
def _review_page(self, limit: int, after):
"""Worker thread: one queue page and the public views of its items."""
page = self.db.review_queue_page(limit=limit, after=after)
roots, real_roots = self._review_roots()
return page, [
self._review_view(row, roots, real_roots) for row in page["items"]
]
# -- detail ----------------------------------------------------------------
async def get_review_item(self, request: web.Request) -> web.Response:
"""GET one item with its candidates and dependency fingerprint."""
try:
read = await self._run_in_executor(
self._review_detail, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if read is None:
return _coded("review-missing")
item, view, fingerprint, candidates = read
detail = {
"success": True,
"library_epoch": item["epoch"],
**view,
"candidates": candidates,
"dependency_fingerprint": fingerprint,
"sha256": item["sha256"],
"total": item["total"],
}
if len(json.dumps(detail).encode("utf-8")) > MAX_DETAIL_BYTES:
detail = {**detail, "candidates": [], "limited": True}
return web.json_response(detail, headers=_NO_STORE)
def _review_detail(self, item_id: str):
"""Worker thread: ``(item, view, fingerprint, candidates)`` or None."""
item = self.db.review_item(item_id)
if item is None:
return None
roots, real_roots = self._review_roots()
view = self._review_view(item, roots, real_roots)
image_path = self._review_image_path(item, roots, view["target"])
aliases, _known = physical_spellings(self.db, image_path, item["locator"])
fingerprint = self.db.review_dependency_fingerprint(image_path, aliases)
return item, view, fingerprint, self._review_candidates(item)
def _review_candidates(self, item: dict) -> list:
"""The detail's candidates: role texts, a label and selectability."""
if item["limited"]:
return []
candidates = [
c for c in item["evidence"].get("candidates", []) if isinstance(c, dict)
]
texts = set()
for candidate in candidates:
for role in ("positive", "negative"):
side = candidate.get(role)
if isinstance(side, dict):
for key in ("text", "storage_text"):
if isinstance(side.get(key), str):
texts.add(side[key])
titles = self.db.stored_prompt_titles(texts)
rows = []
for candidate in candidates:
positive = _side_text(candidate.get("positive"))
negative = _side_text(candidate.get("negative"))
label = (
_side_label(titles, candidate.get("positive"), "No positive")
+ " / "
+ _side_label(titles, candidate.get("negative"), "No negative")
)
rows.append(
{
"hash": candidate.get("hash"),
"sampler_id": candidate.get("sampler_id"),
"state": "resolved" if _workable(candidate) else "unresolved",
"reason": candidate.get("reason") or "",
"label": label,
"positive": {"text": positive} if positive is not None else None,
"negative": {"text": negative} if negative is not None else None,
}
)
return rows
# -- decision --------------------------------------------------------------
async def decide_review_item(self, request: web.Request) -> web.Response:
"""POST one explicit decision on one pending item."""
data, error = await self._read_json_body(request, max_bytes=MAX_BODY_BYTES)
if error is not None:
return error
guards, error = _decision_guards(data)
if error is not None:
return error
try:
item = await self._run_in_executor(
self.db.review_item, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if item is None:
return _coded("review-missing")
choice = guards["choice"]
candidate = None
if choice != UNRESOLVED:
candidate = next(
(
c
for c in item["evidence"].get("candidates", [])
if isinstance(c, dict) and c.get("hash") == choice
),
None,
)
if candidate is None or item["limited"]:
return bad_request("choice is not one of this item's candidates")
if not _workable(candidate):
return bad_request("This sampler path cannot be applied")
if guards["capture_token"] is None:
return bad_request("Apply needs a capture of the file")
verified = None
if guards["capture_token"] is not None:
verified, error = await self._verified_review_capture(
item, guards, need_metadata=candidate is not None
)
if error is not None:
return error
elif guards["sha256"] != item["sha256"]:
return bad_request("sha256 does not match the item")
return await self._record_decision(item, guards, candidate, verified)
async def _verified_review_capture(self, item, guards, *, need_metadata):
"""``((capture, facts, image_path, metadata), None)`` or ``(None, error)``.
The capture must be the item's file and bytes, under the expected
epoch and a current root; the current file must re-hash to the same
bytes; for Apply, the capture's metadata must still yield exactly the
reviewed evidence.
"""
store = self._captures
capture = store.get(guards["capture_token"]) if store is not None else None
if capture is None:
return None, _coded("capture-expired")
if capture.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if guards["sha256"] != capture.sha256:
return None, bad_request("sha256 does not match the preview")
if capture.sha256 != item["sha256"]:
return None, _coded("file-changed")
if guards["expected_epoch"] != capture.epoch:
current = await self._run_in_executor(self.db.get_execution_epoch)
return None, _coded("storage-changed", library_epoch=current)
try:
root, error = await self._run_in_executor(
self._scoped_root, capture.root_index, capture.scope
)
if error is not None:
return None, error
read = await self._run_in_executor(
_verify_and_read, store, capture, item, root, need_metadata
)
except CaptureError as exc:
return None, _capture_error(exc)
if read is None:
return None, _coded("source-changed")
relative, facts, captured = read
if facts.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if facts.sha256 != item["sha256"]:
return None, _coded("file-changed")
metadata = None
if need_metadata:
if captured is None or not _candidate_still_offered(
captured[1], guards["choice"]
):
return None, _coded("review-conflict", "evidence-changed")
metadata = captured[0]
image_path = os.path.join(str(root), *relative.split("/"))
return (capture, facts, image_path, metadata), None
def _decide_sync(self, item, guards, candidate, verified):
"""Worker thread: resolve the file's spellings, decide, build the view.
Returns ``(result, view)``; a refusal raises with ``exc.view`` set.
"""
_capture, facts, image_path, metadata = verified or (None, None, None, None)
aliases, known = (), None
if candidate is not None:
aliases, known = physical_spellings(self.db, image_path, item["locator"])
try:
result = self.db.decide_review_item(
item_id=item["id"],
expected_epoch=guards["expected_epoch"],
expected_revision=guards["expected_revision"],
# The bytes the re-hash proved, or the body's for Leave unresolved.
sha256=facts.sha256 if facts is not None else guards["sha256"],
outcome="selected" if candidate is not None else UNRESOLVED,
candidate=candidate,
image_path=image_path,
metadata=metadata,
expected_fingerprint=guards["dependency_fingerprint"],
aliases=aliases,
known_rows=known,
)
except ReviewCommandError as exc:
if exc.item is not None:
roots, real_roots = self._review_roots()
exc.view = self._review_view(exc.item, roots, real_roots)
raise
roots, real_roots = self._review_roots()
return result, self._review_view(result["item"], roots, real_roots)
async def _record_decision(self, item, guards, candidate, verified):
capture = verified[0] if verified else None
try:
result, view = await self._run_in_executor(
self._decide_sync, item, guards, candidate, verified
)
except ReviewCommandError as exc:
return self._decision_refused(exc)
except LibraryCommandError as exc:
return _coded(exc.code, library_epoch=exc.epoch)
except Exception:
self.logger.exception("Review decision failed")
return _error(500, "decision-failed", "The decision could not be saved.")
if capture is not None:
# The decision used the capture; the page returns to Images.
self._captures.release(capture.token)
body = {
"success": True,
"library_epoch": guards["expected_epoch"],
"item": view,
"total": result["total"],
}
if result["pair"] is not None:
body["pair"] = result["pair"]
return web.json_response(body, headers=_NO_STORE)
def _decision_refused(self, exc: ReviewCommandError) -> web.Response:
extra = {"library_epoch": exc.epoch}
if exc.total is not None:
extra["total"] = exc.total
if exc.view is not None:
extra["item"] = exc.view
message = None
if exc.fingerprint is not None:
extra["dependency_fingerprint"] = exc.fingerprint
if exc.code == "association-conflict" and exc.changed:
message = "dependencies-changed"
return _coded(exc.code, message, **extra)
def _decision_guards(data: dict):
"""The validated decision fields, or ``(None, error)``."""
if set(data) != _DECISION_FIELDS:
return None, bad_request(
"Request must contain exactly " + ", ".join(sorted(_DECISION_FIELDS))
)
epoch, revision = data["expected_epoch"], data["expected_revision"]
if not isinstance(epoch, str) or _UUID.fullmatch(epoch) is None:
return None, bad_request("expected_epoch must be a canonical UUID")
if type(revision) is not int or revision < 0:
return None, bad_request("expected_revision must be a non-negative integer")
token = data["capture_token"]
if token is not None and (
not isinstance(token, str) or _TOKEN.fullmatch(token) is None
):
return None, bad_request("Invalid capture token")
for name in ("sha256", "dependency_fingerprint"):
value = data[name]
if not isinstance(value, str) or _HEX64.fullmatch(value) is None:
return None, bad_request(
f"{name} must be 64 lowercase hexadecimal characters"
)
choice = data["choice"]
if not isinstance(choice, str) or (
choice != UNRESOLVED and _HEX64.fullmatch(choice) is None
):
return None, bad_request("choice must be a candidate hash or unresolved")
return {**data}, None
def _verify_and_read(store, capture, item, root, need_metadata: bool):
"""Worker thread: ``(relative, facts, captured)``, or None off the root.
Formats the item's target under the capture's root (resolving the root's
real path), re-hashes the current file, then reads the capture.
"""
found = format_review_target(
item["spelled_path"],
item["locator"],
[(capture.root_index, _realpath(str(root)))],
)
if found is None:
return None
relative = found[1]
facts = store.verify(root, relative)
captured = captured_review(store, capture) if need_metadata else None
return relative, facts, captured
def _same_file(spelled: str, image_path: str, locator: str) -> bool:
"""True when ``spelled`` reaches the physical file at ``locator``."""
try:
if os.path.normcase(_realpath(spelled)) == locator:
return True
return os.path.samefile(spelled, image_path)
except (OSError, ValueError):
return False
def physical_spellings(db, image_path: str, locator: str):
"""``(aliases, known_rows)`` for the legacy rows of one physical file.
The roots are real paths, while the monitor links a file under the
directory it watches, so a symlinked or junctioned root, a ``subst`` or a
mapped drive spells the same file otherwise. Every row whose spelling
ends with the file's name (the link summaries' fallback lookup) is a
candidate; ``aliases`` are the candidates that resolve to ``locator``
(``normcase(realpath)``, as the capture forms it) or are the same file,
and ``known_rows`` every candidate id, so a decision can refuse a
same-name row it did not classify. Blocking: call it on a worker.
"""
rows = db.review_same_name_rows(os.path.basename(image_path))
aliases = tuple(
spelled
for _row_id, spelled in rows
if isinstance(spelled, str)
and spelled != image_path
and _same_file(spelled, image_path, locator)
)
return tuple(dict.fromkeys(aliases)), frozenset(row_id for row_id, _ in rows)
def _candidate_still_offered(resolution: dict, choice: str) -> bool:
"""True when the captured bytes still offer the chosen candidate.
Only what the decision depends on is re-checked: the bytes are still
``ambiguous`` and the chosen candidate (its hash covers its sampler,
sides and provenance) is still among their candidates. A resolver change
that rewords the top-level reason or reorders candidates leaves a
pending item applicable.
"""
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
return False
evidence, _count, limited = review_evidence(resolution)
if limited:
return False
return any(c.get("hash") == choice for c in evidence.get("candidates", ()))
+241
View File
@@ -0,0 +1,241 @@
"""Needs review recording for the output scan and the live monitor (#249).
The rule (maintainer test of #284/#288): a file is queued only when its
final outcome is ambiguous and unlinked. The strict resolver must find it
``ambiguous``, the scan's legacy fallback (the browser parser,
``_graph_prompt_text``) must not link it, and no saved row may link it
already. A file the fallback links, an exact pair the ingester stored and an
already linked file are never queued. Neither is an ambiguous file with
nothing to choose: fewer than two of its candidates carry a prompt
(``utils.review_capture.reviewable``); the scan reports it unresolved ("could
not be traced").
The queue heals itself: a pending item whose file is now linked (by the
fallback, a later resolution or the monitor) is removed by the next scan
that lists the file or by the monitor's next pass over it, and so is a
pending item without a choice whose file is still ambiguous without one
(``settle_untraced_review_items``). Only an undecided item goes
(``pending``, revision 0, no decision row); a decision is never deleted or
changed (``settle_linked_review_items``).
The scan records through ``_record_scanned_review`` (best effort: a failure
is logged and counted); the monitor calls ``record_review_file`` from its
worker thread after its own link. Both go through
``utils.review_capture.record_ambiguous_file`` on the API's capture store.
Nothing here links a file.
Recording changes records, so it follows the registry's ``records``
admission (#289). The scan records under its own ``records`` job. The
monitor's pass lasts one file and is not a job: it is refused while a
``records`` or ``guarded`` job runs (``deferred``), and the next scan reads
the file again.
"""
import os
from pathlib import Path
from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots
# The kind the monitor's recording checks admission under (never registered).
REVIEW_RECORDING_KIND = "review_recording"
# record_ambiguous_file outcomes that leave the file in the pending queue.
QUEUED_OUTCOMES = ("recorded", "limited", "duplicate")
NEW_OUTCOMES = ("recorded", "limited")
try:
from ...utils.review_capture import record_ambiguous_file
except ImportError:
from utils.review_capture import record_ambiguous_file
# The scan log's reason for items that leave the queue with nothing to choose.
NOTHING_TO_CHOOSE = "nothing to choose"
class ReviewRecordingMixin:
"""Record the scan's and the monitor's ambiguous files for Needs review."""
def _record_scanned_review(self, media_file, roots, review=None):
"""Record one ambiguous, unlinked scanned file (logged and counted)."""
try:
outcome = self._record_review_sync(media_file, roots, review)
except Exception:
self.logger.warning(
f"{media_file.name} could not be added to Needs review",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return
if review is None:
return
if outcome in NEW_OUTCOMES:
review["added"] += 1
if outcome in QUEUED_OUTCOMES:
review["queued"] += 1
def _record_review_sync(self, media_file, roots, review=None):
"""``record_ambiguous_file`` for a file under one of ``roots`` (blocking).
An ``untraced`` file (its captured bytes offer nothing to choose) also
loses its pending items that offer nothing either.
"""
located = self._review_location(media_file, roots)
if located is None:
return "skipped"
root, index, relative, _spelled = located
outcome = record_ambiguous_file(
self.db, self._capture_store(), Path(root), index, relative
)
if outcome == "untraced":
self._settle_untraced_reviews([media_file], roots, review)
return outcome
@staticmethod
def _review_location(media_file, roots, real_roots=None):
"""``(root, root_index, relative, spelled)`` of a listed file, or None.
``spelled`` is the capture's spelling of the file (the real root joined
with the relative path); ``os.path.normcase`` of it is the locator
review items are stored under. ``real_roots`` caches each root's
``realpath`` across a batch.
"""
relative, _label, index = _scan_location(media_file, roots)
root = next((r for r, _l, i in roots if i == index), None)
if index is None or root is None:
return None
if real_roots is None:
real_roots = {}
real = real_roots.get(index)
if real is None:
real = real_roots[index] = os.path.realpath(os.fspath(root))
spelled = os.path.join(real, *relative.split("/"))
return root, index, relative, spelled
def _review_file_linked(self, media_file, roots):
"""True when a saved row links the scanned file (any spelling)."""
located = self._review_location(media_file, roots)
spellings = (str(media_file),) + ((located[3],) if located else ())
return self.db.review_file_linked(spellings)
def _settle_scanned_reviews(self, media_files, roots, review=None):
"""Remove the pending items of ``media_files`` that are now linked.
Blocking; best effort like recording (a failure is logged and
counted in ``review["errors"]``). Returns how many items left.
"""
real_roots = {}
files = []
for media_file in media_files:
located = self._review_location(media_file, roots, real_roots)
if located is not None:
spelled = located[3]
files.append((os.path.normcase(spelled), (str(media_file), spelled)))
try:
settled = self.db.settle_linked_review_items(files) if files else 0
except Exception:
self.logger.warning(
"Needs review items of linked files could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["settled"] += settled
return settled
def _settle_untraced_reviews(self, media_files, roots, review=None):
"""Remove the pending items without a choice of untraced ``media_files``.
``media_files`` are files found ambiguous with fewer than two
candidates that carry a prompt. Blocking; best effort like the linked
settle (a failure is logged and counted in ``review["errors"]``).
Returns how many items left.
"""
locators = []
for media_file in media_files:
located = self._review_location(media_file, roots)
if located is not None:
locators.append(os.path.normcase(located[3]))
try:
removed = self.db.settle_untraced_review_items(locators) if locators else 0
except Exception:
self.logger.warning(
"Needs review items without a choice could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["untraced"] += removed
return removed
def _fallback_links(self, metadata):
"""True when the scan's legacy fallback would link these records."""
return bool(metadata.get("prompt")) and bool(self._graph_prompt_text(metadata))
def record_review_file(self, image_path):
"""Apply the Needs review rule to one monitor output (its worker thread).
Maps ``image_path`` to the gallery root that lists it; a file under no
current root is ``skipped``. A linked file is never queued: its
pending items are removed and the answer is ``linked``. An unlinked
file the scan's fallback would link is ``skipped`` (the next scan
links it); any other file is recorded when it is ambiguous with a
choice, and is ``untraced`` (its pending items without a choice
removed) when it is ambiguous without one. While a
job runs that a ``records`` job may not run beside, nothing is written
and the answer is ``deferred``. Errors propagate: the monitor logs
them and keeps its link.
"""
busy = self._jobs.blocker(REVIEW_RECORDING_KIND, admission="records")
if busy is not None:
self.logger.info(
f"Needs review was not updated for "
f"{os.path.basename(str(image_path))} while {busy.title} runs; "
"the next scan reads it again"
)
return "deferred"
full = os.path.abspath(str(image_path))
located = Path(os.path.realpath(os.path.dirname(full))) / os.path.basename(full)
roots = _scan_roots(self._get_all_output_dirs())
if self._review_location(located, roots) is None:
return "skipped"
if self._review_file_linked(located, roots):
self._settle_scanned_reviews([located], roots)
return "linked"
if self._fallback_links(self._extract_comfyui_metadata(str(located))):
return "skipped"
return self._record_review_sync(located, roots)
@staticmethod
def _log_review_items(job, review):
"""The scan log's Needs review summary lines (none when nothing moved)."""
prefix = SCAN_LOG_PREFIXES["review"]
if review["added"]:
job.log_line(
f"{prefix} {_plural(review['added'], 'image')} added to Needs review"
)
already = review["queued"] - review["added"]
if already:
job.log_line(
f"{prefix} {_plural(already, 'image')} already in Needs review"
)
if review["settled"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['settled'], 'image')} left Needs review · now linked"
)
if review["untraced"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['untraced'], 'image')} left Needs review · "
f"{NOTHING_TO_CHOOSE}"
)
if review["errors"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['errors']} "
f"{_plural(review['errors'], 'image')} could not be added to "
"Needs review"
)
+246 -31
View File
@@ -4,8 +4,15 @@ A save validates the payload, commits it to config.json through the config
store and publishes it at runtime inside that commit; a save that cannot be
written changes nothing at runtime and says so. ``_config_commit_failure``
is shared with the LoRA integration's save.
GET also reports the image sources and the monitor (``sources.py``). A save
that changes the sources runs as a ``source_change`` registry hold, so it is
refused while a record-changing job (a scan) runs. ``monitoring_enabled``
takes effect at the next restart. ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing.
"""
import contextlib
import os
from aiohttp import web
@@ -15,7 +22,10 @@ from ..config_store import (
RevisionConflict,
get_config_store,
)
from . import sources
from .admin import _current_image_monitor
from .job_routes import busy_response
from .jobs import JobBusy
try:
from ...utils.validators import validate_result_timeout, validate_worker_threads
@@ -23,13 +33,75 @@ except ImportError:
from utils.validators import validate_result_timeout, validate_worker_threads
# Settings owned by POST /settings: config attribute -> (section, key).
# Settings owned by POST /settings: config attribute -> its keys in config.json.
_SETTINGS_FILE_KEYS = {
"RESULT_TIMEOUT": ("web_ui", "result_timeout"),
"WEBUI_DISPLAY_MODE": ("web_ui", "webui_display_mode"),
"INFINITE_SCROLL": ("web_ui", "infinite_scroll"),
"WORKER_THREADS": ("performance", "worker_threads"),
"MONITORING_ENABLED": ("gallery", "monitoring", "enabled"),
}
# Attributes that live on GalleryConfig; the others on PromptManagerConfig.
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"})
SOURCE_CHANGE_JOB_KIND = "source_change"
SOURCE_CHANGE_JOB_TITLE = "Source change"
def _source_identity(path):
"""The case-folded spelling two roots are the same directory by (Windows)."""
return os.path.normcase(path)
def _stat_identity(path):
"""``(st_dev, st_ino)`` of an existing directory, or None.
Catches case variants on a case-insensitive filesystem whose spelling
normcase keeps (macOS).
"""
try:
stat = os.stat(path)
except (OSError, ValueError):
return None
return (stat.st_dev, stat.st_ino) if stat.st_ino else None
# Refusals that a reconnected drive or folder can lift.
_RECONNECTABLE = frozenset({"not-found", "not-directory"})
def _root_key(root):
"""The directory a configured root names (resolved), else its spelling."""
from ..config import GalleryConfig
try:
return GalleryConfig.resolve_gallery_root(root)
except (OSError, TypeError, ValueError, AttributeError):
return root
def _request_error(message):
return {"code": "invalid-request", "error": message}
def _invalid_source(index, entry, code, reason):
"""The 400 body for the source at ``index`` that cannot be used.
Names it by its public spelling (computed as text: a refused path is
never resolved), so the page can mark that row and say why.
"""
source = sources.public_spelling(entry)
advice = (
"Remove it or reconnect it, then save."
if code in _RECONNECTABLE
else "Remove it, then save."
)
return {
"code": "invalid-source",
"source": source,
"index": index,
"reason": reason,
"error": f"{source} can't be used: {reason}. {advice}",
}
def _section(config, name):
@@ -47,10 +119,8 @@ def _settings_mutation(updates, new_roots):
def mutate(config):
for name, value in updates.items():
section, key = _SETTINGS_FILE_KEYS[name]
merged = _section(config, section)
merged[key] = value
config[section] = merged
*sections, key = _SETTINGS_FILE_KEYS[name]
_set_nested(config, sections, key, value)
if new_roots is not None:
gallery = _section(config, "gallery")
monitoring = _section(gallery, "monitoring")
@@ -62,6 +132,16 @@ def _settings_mutation(updates, new_roots):
return mutate
def _set_nested(config, sections, key, value):
"""``config[s1][s2]...[key] = value``, each section a fresh dict."""
if not sections:
config[key] = value
return
merged = _section(config, sections[0])
_set_nested(merged, sections[1:], key, value)
config[sections[0]] = merged
async def _read_json(request):
"""Parse a JSON object body through the package's capped reader."""
from . import _read_json_body
@@ -81,6 +161,10 @@ class SettingsRoutesMixin:
async def save_settings_route(request):
return await self.save_settings(request)
@routes.post("/prompt_manager/settings/validate-source")
async def validate_source_route(request):
return await self.validate_source(request)
async def get_settings(self, request):
"""Get current settings."""
try:
@@ -90,11 +174,15 @@ class SettingsRoutesMixin:
self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES
]
revision = await self._run_in_executor(self._config_revision)
source_status, monitor = await self._run_in_executor(self._source_status)
return web.json_response(
{
"success": True,
**revision,
"sources": source_status,
"monitor": monitor,
"settings": {
"monitoring_enabled": GalleryConfig.MONITORING_ENABLED,
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
@@ -115,6 +203,61 @@ class SettingsRoutesMixin:
status=500,
)
def _source_status(self):
"""``(sources, monitor)`` of GET /settings (blocking)."""
from ..config import GalleryConfig
output_dirs, auto_detected = self._output_dirs_with_origin()
return sources.describe_sources(
configured=list(GalleryConfig.MONITORING_DIRECTORIES),
output_dirs=output_dirs,
auto_detected=auto_detected,
monitor=_current_image_monitor(),
enabled=GalleryConfig.MONITORING_ENABLED,
)
async def validate_source(self, request):
"""Check one source path ``{path}``; writes nothing.
200 ``{success, valid, label, path, code, reason}``: a valid source
answers its label and public path, an invalid one its code and reason
only. A body without a string ``path`` is a 400 ``invalid-request``.
"""
data, error_response = await _read_json(request)
if error_response is not None:
too_large = error_response.status == 413
return web.json_response(
{
"success": False,
"code": "too-large" if too_large else "invalid-request",
"error": (
"Request body too large"
if too_large
else "Request body must be a JSON object"
),
},
status=error_response.status,
)
path = data.get("path")
if not isinstance(path, str):
return web.json_response(
{
"success": False,
"code": "invalid-request",
"error": "path must be a string",
},
status=400,
)
try:
result = await self._run_in_executor(sources.validate_source, path)
except Exception:
self.logger.exception("Source validation failed")
return web.json_response(
{"success": False, "code": None, "error": "Failed to check the source"},
status=500,
)
return web.json_response({"success": True, **result})
def _monitored_directories(self):
"""Directories the running image monitor watches, else the configured roots."""
from ..config import GalleryConfig
@@ -151,9 +294,15 @@ class SettingsRoutesMixin:
{"success": False, "error": str(ve)}, status=400
)
new_roots, error = self._parse_gallery_roots(data)
# Off the loop: an accepted root is resolved (filesystem access).
new_roots, error = await self._run_in_executor(
self._parse_gallery_roots, data
)
if error:
return web.json_response({"success": False, "error": error}, status=400)
return web.json_response({"success": False, **error}, status=400)
roots_change = new_roots is not None and await self._run_in_executor(
self._roots_change, new_roots
)
published = {}
@@ -162,15 +311,30 @@ class SettingsRoutesMixin:
# they were written, so the runtime ends on the last one.
published.update(self._publish_settings(updates, new_roots))
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
return self._config_commit_failure(exc)
async with contextlib.AsyncExitStack() as stack:
hold = None
if roots_change:
try:
hold = await stack.enter_async_context(
self._jobs.hold(
SOURCE_CHANGE_JOB_KIND,
SOURCE_CHANGE_JOB_TITLE,
admission="records",
)
)
except JobBusy as busy:
return busy_response(busy)
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
if hold is not None:
hold.finish("failed")
return self._config_commit_failure(exc)
return web.json_response(
{
@@ -196,10 +360,11 @@ class SettingsRoutesMixin:
as ``runtime_applied: false`` with the fields that need a restart;
nothing is retried or rolled back. Never raises.
"""
from ..config import PromptManagerConfig
from ..config import GalleryConfig, PromptManagerConfig
for name, value in updates.items():
setattr(PromptManagerConfig, name, value)
owner = GalleryConfig if name in _GALLERY_SETTINGS else PromptManagerConfig
setattr(owner, name, value)
restart_fields = []
runtime_applied = True
if new_roots is not None:
@@ -212,11 +377,40 @@ class SettingsRoutesMixin:
)
runtime_applied = False
restart_fields.append("gallery_root_paths")
if "MONITORING_ENABLED" in updates:
# The monitor starts with the server: a restart applies the setting
# unless the observer already matches it.
if updates["MONITORING_ENABLED"] != self._observer_alive():
restart_fields.append("monitoring_enabled")
return {
"runtime_applied": runtime_applied,
"restart_required_fields": restart_fields,
}
@staticmethod
def _roots_change(new_roots):
"""Whether a save changes the sources (blocking: it resolves them).
Compared as directories, not spellings: the legacy page resends the
roots resolved, while config.json may hold them as hand-written
(relative, another case), and an unchanged resend is no change.
"""
from ..config import GalleryConfig
if new_roots is None:
return False
return [_root_key(root) for root in new_roots] != [
_root_key(root) for root in GalleryConfig.MONITORING_DIRECTORIES
]
@staticmethod
def _observer_alive():
"""Whether the image monitor's observer is running now. Never raises."""
try:
return sources.monitor_state(_current_image_monitor())[1]
except Exception:
return False
def _config_commit_failure(self, exc):
"""The response for a config commit that wrote nothing."""
not_saved = {"success": False, "persisted": False, "runtime_applied": False}
@@ -283,6 +477,10 @@ class SettingsRoutesMixin:
if not isinstance(data["infinite_scroll"], bool):
raise ValueError("infinite_scroll must be true or false")
updates["INFINITE_SCROLL"] = data["infinite_scroll"]
if "monitoring_enabled" in data:
if not isinstance(data["monitoring_enabled"], bool):
raise ValueError("monitoring_enabled must be true or false")
updates["MONITORING_ENABLED"] = data["monitoring_enabled"]
if "worker_threads" in data:
updates["WORKER_THREADS"] = validate_worker_threads(
data["worker_threads"], PromptManagerConfig.max_worker_threads()
@@ -295,15 +493,21 @@ class SettingsRoutesMixin:
Accepts ``gallery_root_paths`` (list, preferred) or the legacy single
``gallery_root_path`` string. Blank entries are dropped.
Each entry passes the same checks as validate-source
(``sources.source_problem``): the form, then containment as text,
before anything is resolved. Blocking (an accepted root is resolved).
Returns:
(roots, error): ``roots`` is the validated list, or ``None`` when
the payload carries no gallery root key; ``error`` is a message
when validation failed (and ``roots`` is then ``None``).
the payload carries no gallery root key; ``error`` is the coded
body of the 400 when validation failed (``roots`` is then None).
An unusable source is ``invalid-source`` naming its public path
and its position in the list.
"""
if "gallery_root_paths" in data:
raw = data["gallery_root_paths"]
if not isinstance(raw, list):
return None, "gallery_root_paths must be a list"
return None, _request_error("gallery_root_paths must be a list")
elif "gallery_root_path" in data:
raw = [data["gallery_root_path"]]
else:
@@ -312,27 +516,38 @@ class SettingsRoutesMixin:
from ..config import GalleryConfig
roots = []
for entry in raw:
seen = set()
for index, entry in enumerate(raw):
if not isinstance(entry, str):
return None, "Gallery root paths must be strings"
return None, _request_error("Gallery root paths must be strings")
entry = entry.strip()
if not entry:
continue
ok, reason = GalleryConfig.validate_gallery_root(entry)
if not ok:
return (
None,
f"Invalid gallery root '{os.path.basename(entry)}': {reason}",
)
roots.append(GalleryConfig.resolve_gallery_root(entry))
code, reason = sources.source_problem(entry)
if code is not None:
return None, _invalid_source(index, entry, code, reason)
root = GalleryConfig.resolve_gallery_root(entry)
keys = {("path", _source_identity(root)), ("file", _stat_identity(root))}
keys.discard(("file", None))
if keys & seen:
continue # another spelling of a root already listed
seen |= keys
roots.append(root)
return roots, None
def _apply_gallery_roots(self, roots):
"""Install validated gallery roots; returns True when they changed."""
"""Install validated gallery roots; returns True when they changed.
Another spelling of the same directories installs the resolved
spelling and is no change (nothing to restart for).
"""
from ..config import GalleryConfig
if roots == list(GalleryConfig.MONITORING_DIRECTORIES):
return False
if not self._roots_change(roots):
GalleryConfig.MONITORING_DIRECTORIES = roots
return False
GalleryConfig.MONITORING_DIRECTORIES = roots
self._cached_output_dir = None
self.reset_gallery_roots_cache()
+280
View File
@@ -0,0 +1,280 @@
"""Image sources as System › Sources & scans shows them (#249 S4).
- ``source_form_error`` is the lexical check a submitted source path passes
before any filesystem access: no control characters, no ``..`` part, and
only the native path forms (on Windows a drive or a UNC share, never the
``\\\\?\\`` and ``\\\\.\\`` namespaces, a drive-relative path, an alternate
data stream, a device name or a part Windows would silently trim; on POSIX
no Windows drive or backslash form).
- ``source_problem`` is the one refusal every caller answers (validate-source
and the settings save): the form check, then
``GalleryConfig.gallery_root_problem``, which decides containment as text
before it touches the filesystem.
- ``describe_sources`` (blocking: it resolves paths) reports the configured
sources with their validity, the effective sources with their listing
index and watch state, and the image monitor's state. Every path in it is
a public path (``_public_path``), never an absolute one.
A source's watch state is ``watching`` when the monitor's observer is alive,
scheduled that directory and monitoring is on; ``stopping`` when it still
watches it but monitoring was switched off (it stops at the next restart);
``manual`` when monitoring is off or this process has no monitor; else
``restart_pending`` (a source added since the monitor started, monitoring
switched on, or an observer that stopped).
"""
import logging
import ntpath
import os
import re
logger = logging.getLogger("prompt_manager.api")
MAX_SOURCE_PATH = 4096 # characters
FORM_REASONS = {
"empty": "Gallery root must be a non-empty path",
"invalid-form": "Gallery root is not a folder path this server can use",
"traversal": "Gallery root must not contain '..'",
}
_DRIVE = re.compile(r"[A-Za-z]:")
_POSIX_FOREIGN = re.compile(r"^[A-Za-z]:")
# Reserved device names: Windows opens the device for any folder of that name.
_DEVICE_NAME = re.compile(r"^(con|prn|aux|nul|com[0-9¹²³]|lpt[0-9¹²³])(\..*)?$", re.I)
# folder_paths getters and the label of the directory each one reports.
_COMFYUI_LABELS = (
("get_output_directory", "ComfyUI output"),
("get_input_directory", "ComfyUI input"),
("get_temp_directory", "ComfyUI temp"),
("get_user_directory", "ComfyUI user"),
)
WATCH_STATES = ("watching", "stopping", "manual", "restart_pending")
def source_form_error(text, *, windows=None):
"""The form code that refuses ``text`` as a source path, or None.
``windows`` selects the path rules (default: this OS). No filesystem
access happens here.
"""
if windows is None:
windows = os.name == "nt"
if not isinstance(text, str) or len(text) > MAX_SOURCE_PATH:
return "invalid-form"
if any(ord(char) < 32 or ord(char) == 127 for char in text):
return "invalid-form"
text = text.strip()
if not text:
return "empty"
return _windows_form_error(text) if windows else _posix_form_error(text)
def _posix_form_error(text):
if "\\" in text or _POSIX_FOREIGN.match(text):
return "invalid-form" # a Windows path names nothing on this server
return "traversal" if ".." in text.split("/") else None
def _windows_form_error(text):
portable = text.replace("/", "\\")
if portable.startswith(("\\\\?\\", "\\\\.\\")):
return "invalid-form" # verbatim and device namespaces
if portable.startswith("\\\\"):
drive, rest = ntpath.splitdrive(portable)
share = drive[2:].split("\\")
if len(share) != 2 or any(part in ("", ".", "..") for part in share):
return "invalid-form"
else:
drive, rest = ntpath.splitdrive(portable)
if drive and (not _DRIVE.fullmatch(drive) or not rest.startswith("\\")):
return "invalid-form" # "C:folder" is relative to that drive's cwd
if ":" in rest:
return "invalid-form" # an alternate data stream
parts = rest.split("\\")
if ".." in parts:
return "traversal"
for part in parts:
if part in ("", "."):
continue
if part[-1] in ". " or _DEVICE_NAME.match(part):
return "invalid-form"
return None
def source_problem(text):
"""``(code, reason)`` that refuses ``text`` as a source, else ``(None, "")``.
The form check runs first and touches nothing; then containment is
decided as text before any filesystem access (see
``GalleryConfig.gallery_root_problem``). Every caller answers alike.
"""
from ..config import GalleryConfig
code = source_form_error(text)
if code is not None:
return code, FORM_REASONS[code]
return GalleryConfig.gallery_root_problem(text)
def public_spelling(text):
"""How a refused source is named in a response, computed as text only.
Relative to the path anchor it lies under (its public path), else its
last part: the caller sent the path, so naming it reveals nothing, and a
refused path is never resolved to name it.
"""
from ..config import GalleryConfig, _lexical_path, _lexically_inside
text = str(text).strip()
try:
lexical = os.path.normpath(text)
key = _lexical_path(text) # the same length as ``lexical``
if not os.path.isabs(text):
return lexical.replace("\\", "/")
for anchor in GalleryConfig.path_anchors():
if _lexically_inside(key, [anchor]):
rest = lexical[len(anchor.rstrip(os.sep)) :].lstrip(os.sep)
return rest.replace(os.sep, "/") or "."
return os.path.basename(lexical) or lexical
except (TypeError, ValueError):
return ""
def identity(path):
"""The comparison key of a directory: resolved and case-folded (Windows)."""
return os.path.normcase(os.path.realpath(str(path)))
def _comfyui_labels():
"""``{identity: label}`` of the directories ComfyUI reports."""
try:
import folder_paths
except ImportError:
return {}
labels = {}
for getter_name, label in _COMFYUI_LABELS:
getter = getattr(folder_paths, getter_name, None)
if getter is None:
continue
try:
directory = getter()
except Exception:
continue
if isinstance(directory, str) and directory:
labels.setdefault(identity(directory), label)
return labels
def source_label(path, labels=None):
""" "ComfyUI output" for ComfyUI's own directories, else the folder name."""
if labels is None:
labels = _comfyui_labels()
label = labels.get(identity(path))
if label:
return label
return os.path.basename(os.path.normpath(str(path))) or str(path)
def _configured(entry, labels, public_path):
from ..config import GalleryConfig
code, reason = GalleryConfig.gallery_root_problem(entry)
try:
resolved = GalleryConfig.resolve_gallery_root(entry)
except (OSError, TypeError, ValueError):
resolved = str(entry)
return {
"label": source_label(resolved, labels),
"path": public_path(resolved),
"valid": code is None,
"code": code,
"reason": reason or None,
}
def monitor_state(monitor):
"""``(present, observer_alive, scheduled directories)`` of the image monitor."""
if monitor is None:
return False, False, []
status = monitor.get_status()
scheduled = [str(d) for d in status.get("monitored_directories") or []]
return True, bool(status.get("observer_alive")), scheduled
def watch_state(path, *, present, alive, scheduled_keys, enabled):
"""One of ``WATCH_STATES`` (see the module docstring)."""
if alive and identity(path) in scheduled_keys:
return "watching" if enabled else "stopping"
if not present or not enabled:
return "manual"
return "restart_pending"
def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled):
"""``(sources, monitor)`` for GET /settings (blocking; public paths only).
``configured`` is the configured roots, ``output_dirs`` and
``auto_detected`` what ``_output_dirs_with_origin`` returned, ``monitor``
the running ImageMonitor or None, ``enabled`` the monitoring setting.
"""
from . import _public_path
labels = _comfyui_labels()
present, alive, scheduled = monitor_state(monitor)
scheduled_keys = {identity(path) for path in scheduled}
effective = [
{
"label": source_label(path, labels),
"path": _public_path(path),
"root_index": index,
"watch": watch_state(
path,
present=present,
alive=alive,
scheduled_keys=scheduled_keys,
enabled=enabled,
),
}
for index, path in enumerate(output_dirs)
]
return (
{
"configured": [
_configured(entry, labels, _public_path) for entry in configured
],
"effective": effective,
"auto_detected": bool(auto_detected),
},
{
"present": present,
"observer_alive": alive,
"scheduled": [_public_path(path) for path in scheduled],
},
)
def validate_source(text):
"""``{valid, label, path, code, reason}`` for one submitted path (blocking).
Writes nothing. An invalid path answers its code and reason with no
label or path, so a response never echoes what was typed.
"""
from ..config import GalleryConfig
from . import _public_path
code, reason = source_problem(text)
if code is not None:
return {
"valid": False,
"label": None,
"path": None,
"code": code,
"reason": reason,
}
resolved = GalleryConfig.resolve_gallery_root(text)
return {
"valid": True,
"label": source_label(resolved),
"path": _public_path(resolved),
"code": None,
"reason": None,
}
+221
View File
@@ -0,0 +1,221 @@
"""System routes: the read-only health check, the Overview and usage statistics.
- ``POST /prompt_manager/system/health`` starts the health check
(``database/health.py``) off-loop as a ``read`` job of kind ``health``: one
at a time, beside a scan, not cancellable and with no file outcomes. The
API keeps the last report for this session; ``GET`` returns it or null.
A report describes the database it was taken from: replacing the database
(a restore) forgets it, and a check that started before the swap is not
kept.
- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate.
Every value is real or null: there is no backup status until backups report
one and no health result until a check ran; ``needs_review.pending`` is the
Needs review queue's pending total. The missing-thumbnail count is not part
of it: the page reads ``/images/thumbnails/status``, which walks every
source.
- ``GET /prompt_manager/system/statistics`` lists runs and ratings per
category, with the legacy ``/stats`` totals and the Overview's count of
rated images (images whose linked prompt is rated).
New routes answer errors with ``{success:false, code, error}``.
"""
import asyncio
import os
from collections import namedtuple
from aiohttp import web
from .job_routes import busy_response
from .jobs import HEALTH_JOB_KIND, JobBusy
HEALTH_JOB_TITLE = "Health check"
_HEALTH_STATUS_LINES = {
"passed": "Health check passed",
"issues": "Health check found issues",
"incomplete": "Health check incomplete",
}
# The last health report and the database generation it was taken from.
_StoredReport = namedtuple("_StoredReport", "generation report")
async def _gather_settled(*reads):
"""Await every read, then raise the first failure.
Unlike a plain gather, a failure never answers while the other executor
reads still run against the database.
"""
results = await asyncio.gather(*reads, return_exceptions=True)
for result in results:
if isinstance(result, BaseException):
raise result
return results
def _failure(message, status=500, code=None):
return web.json_response(
{"success": False, "code": code, "error": message}, status=status
)
def health_summary(report):
"""What the Overview shows of a report: when, the outcome, checks with issues."""
if report is None:
return None
return {
"checked_at": report["checked_at"],
"status": report["status"],
"issues": sum(1 for check in report["checks"] if check["status"] == "issues"),
}
class SystemRoutesMixin:
"""Mixin providing the /prompt_manager/system endpoints."""
def _register_system_routes(self, routes):
@routes.post("/prompt_manager/system/health")
async def start_health_check_route(request):
return await self.start_health_check(request)
@routes.get("/prompt_manager/system/health")
async def health_report_route(request):
return await self.get_health_report(request)
@routes.get("/prompt_manager/system/overview")
async def system_overview_route(request):
return await self.get_system_overview(request)
@routes.get("/prompt_manager/system/statistics")
async def system_statistics_route(request):
return await self.get_system_statistics(request)
# ── health ────────────────────────────────────────────────────────
def _health_roots(self):
"""The sources missing files are judged under.
The effective sources, plus every configured source that is not an
existing directory (an unmounted drive): the check then reports
missing files as unavailable instead of counting that source's links.
"""
from ..config import GalleryConfig
roots = [str(path) for path in self._get_all_output_dirs()]
for configured in GalleryConfig.MONITORING_DIRECTORIES:
try:
resolved = GalleryConfig.resolve_gallery_root(configured)
except (OSError, TypeError, ValueError):
continue
if not os.path.isdir(resolved):
roots.append(resolved)
return roots
def _database_replaced(self):
"""Forget what described the previous database (call after swapping ``db``)."""
self._db_generation += 1
self._health_report = None
def _current_health_report(self):
"""The last report, or None when none was kept for the current database."""
stored = self._health_report
generation = self._db_generation
if isinstance(stored, _StoredReport) and stored.generation == generation:
return stored.report
return None
async def start_health_check(self, request):
"""Start the health check job (202), or 409 naming the running check."""
generation = self._db_generation
db = self.db
try:
roots = await self._run_in_executor(self._health_roots)
except Exception:
self.logger.exception("Health check could not resolve the image sources")
return _failure("Failed to start the health check")
async def run(job):
job.update(status_line="Checking the database and image links")
report = await self._run_in_executor(db.run_health_checks, roots)
if generation == self._db_generation:
self._health_report = _StoredReport(generation, report)
else:
self.logger.info("Health report dropped: the database was replaced")
job.update(progress=100, status_line=_HEALTH_STATUS_LINES[report["status"]])
try:
job = self._jobs.start(
HEALTH_JOB_KIND,
HEALTH_JOB_TITLE,
admission="read",
cancellable=False,
scope={},
runner=run,
)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
async def get_health_report(self, request):
return web.json_response(
{"success": True, "report": self._current_health_report()}
)
# ── overview ──────────────────────────────────────────────────────
async def get_system_overview(self, request):
from ..config import GalleryConfig
try:
(output_dirs, auto_detected), counts = await _gather_settled(
self._run_in_executor(self._output_dirs_with_origin),
self._run_in_executor(self.db.library_counts),
)
except Exception:
self.logger.exception("System overview failed")
return _failure("Failed to read the overview")
return web.json_response(
{
"success": True,
"library": {
"positive_prompts": counts["positive_prompts"],
"negative_variants": counts["negative_variants"],
},
"images": {
"indexed": counts["images"],
"rated": counts["rated_images"],
},
"health": health_summary(self._current_health_report()),
"backup": None, # no verified backup status exists yet (S8)
"needs_review": {"pending": counts["review_pending"]},
"sources": {
"configured": len(GalleryConfig.MONITORING_DIRECTORIES),
"effective": len(output_dirs),
"auto_detected": auto_detected,
},
},
headers={"Cache-Control": "no-store"},
)
# ── statistics ────────────────────────────────────────────────────
async def get_system_statistics(self, request):
try:
by_category, totals, counts = await _gather_settled(
self._run_in_executor(self.db.usage_by_category),
self._run_in_executor(self.db.get_statistics),
self._run_in_executor(self.db.library_counts),
)
except Exception:
self.logger.exception("System statistics failed")
return _failure("Failed to read statistics")
return web.json_response(
{
"success": True,
"by_category": by_category,
"totals": totals,
"rated_images": counts["rated_images"],
},
headers={"Cache-Control": "no-store"},
)
+121 -25
View File
@@ -31,7 +31,6 @@ routes = server_instance.routes
extension_uri = None # Will be set in __init__.py
import os # noqa: E402
from pathlib import Path # noqa: E402
from typing import Any, Dict, List, Optional, Tuple # noqa: E402
# Import logging system
@@ -51,6 +50,14 @@ config_logger = get_logger("prompt_manager.config")
# may be used as gallery roots in addition to ComfyUI's own directories.
EXTRA_GALLERY_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
# The one answer for every path outside the allowed folders. The filesystem
# root and the home directory answer it too, so no caller learns the server's
# layout (home path, user name) from a refusal (#249 S4).
OUTSIDE_ALLOWED_REASON = (
"Gallery root must be inside a ComfyUI directory (output, input, temp, user) "
f"or one listed in {EXTRA_GALLERY_ROOTS_ENV}"
)
# Environment variable overriding where config.json is read from and saved to.
CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH"
@@ -89,6 +96,29 @@ def write_private_json(path: str, data: Any) -> None:
os.chmod(path, 0o600)
def _lexical_path(path: str) -> str:
"""``path`` normalised for comparison without touching the filesystem.
normpath collapses ``.`` and ``..`` as text and normcase folds case on
Windows; neither opens, stats nor resolves anything (no SMB request for a
UNC path). A symlink is only seen by :func:`_canonical_path` later.
"""
return os.path.normcase(os.path.normpath(path))
def _lexically_inside(path: str, parents: List[str]) -> bool:
"""Whether the normalised ``path`` is one of ``parents`` or below one.
Pure text on normalised spellings (:func:`_lexical_path` or
:func:`_canonical_path`): no filesystem access.
"""
sep = os.path.sep
return any(
path == parent or path.startswith(parent.rstrip(sep) + sep)
for parent in parents
)
def _is_filesystem_root(path: str) -> bool:
"""True for '/' on POSIX and drive roots such as 'C:\\' on Windows."""
return os.path.dirname(path) == path
@@ -268,7 +298,12 @@ class GalleryConfig:
"""
monitoring = new_config.get("monitoring", {})
if "enabled" in monitoring:
cls.MONITORING_ENABLED = monitoring["enabled"]
if isinstance(monitoring["enabled"], bool):
cls.MONITORING_ENABLED = monitoring["enabled"]
else: # a hand-edited "false" would otherwise switch it on
config_logger.warning(
"Ignoring gallery monitoring enabled: expected true or false"
)
if "directories" in monitoring:
directories = cls._validated_directories(monitoring["directories"])
if directories is not None:
@@ -337,31 +372,56 @@ class GalleryConfig:
Filesystem roots are never allowed as parents, even when listed in
the environment, because that would re-open the whole disk.
"""
parents = []
return [canonical for canonical, _spelling in cls._allowed_parent_pairs()]
@classmethod
def _allowed_parent_pairs(cls) -> List[Tuple[str, str]]:
"""``(canonical, lexical spelling)`` of each allowed parent.
Only trusted directories (ComfyUI's own and the environment's) are
resolved here; the spelling is how ComfyUI or the environment wrote
the directory, so a submitted path can be compared without resolving it.
"""
pairs = []
seen = set()
for candidate in _comfyui_directories() + _extra_gallery_roots():
try:
canonical = _canonical_path(candidate)
spelling = _lexical_path(os.path.abspath(candidate))
except (OSError, ValueError):
continue
if _is_filesystem_root(canonical) or not os.path.isdir(canonical):
continue
if canonical not in parents:
parents.append(canonical)
return parents
if canonical not in seen:
seen.add(canonical)
pairs.append((canonical, spelling))
return pairs
@classmethod
def path_anchors(cls) -> List[str]:
"""Canonical directories that relative gallery paths are resolved against.
The ComfyUI base directory comes first, followed by the parent of each
directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so an extra
root is addressed by its own name (``gallery/sub``) rather than by an
absolute path. The same anchors drive the public (relative) form of
paths in API responses.
ComfyUI directory outside it (``--output-directory D:\\Renders``) and of
each directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so such a
directory is addressed by its own name (``Renders/sub``) rather than by
an absolute path, and two folders of one name stay apart. The same
anchors drive the public (relative) form of paths in API responses.
"""
anchors = []
base = _comfyui_base_directory()
candidates = [base] if base else []
try:
base_key = _canonical_path(base) if base else None
except (OSError, ValueError):
base_key = None
for directory in _comfyui_directories():
try:
canonical = _canonical_path(directory)
except (OSError, ValueError):
continue
if base_key is None or not _lexically_inside(canonical, [base_key]):
candidates.append(os.path.dirname(canonical))
candidates.extend(os.path.dirname(root) for root in _extra_gallery_roots())
for candidate in candidates:
try:
@@ -399,33 +459,69 @@ class GalleryConfig:
(True, "") when the directory lies inside one of
:meth:`allowed_gallery_parents`; otherwise (False, reason).
"""
code, reason = cls.gallery_root_problem(path)
return code is None, reason
@classmethod
def gallery_root_problem(cls, path: Any) -> Tuple[Optional[str], str]:
"""``(code, reason)`` why ``path`` may not be a gallery root, else (None, "").
Codes: ``empty``, ``unresolvable``, ``outside-allowed``, ``not-found``
and ``not-directory``. No reason names the path.
Containment is decided twice. First lexically, with no filesystem
access at all: a path (UNC, ``\\\\?\\``, a device, anything) whose
normalised spelling lies outside every allowed parent answers
``outside-allowed`` before it is resolved, so the server never opens
or stats it (no SMB connection, no existence or timing oracle). Only
then is it resolved and checked again, which catches a symlink out.
The filesystem root and the home directory answer ``outside-allowed``
too: one answer for every caller (validate-source, the settings save,
config.json), and the detail goes to the log only.
"""
if not isinstance(path, str) or not path.strip():
return False, "Gallery root must be a non-empty path"
return "empty", "Gallery root must be a non-empty path"
pairs = cls._allowed_parent_pairs()
spellings = [spelling for pair in pairs for spelling in pair]
try:
lexical = cls._lexical_candidates(path)
except (TypeError, ValueError):
return "unresolvable", "Gallery root could not be resolved"
if not any(_lexically_inside(candidate, spellings) for candidate in lexical):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
try:
canonical = cls.resolve_gallery_root(path)
except (OSError, ValueError):
return False, "Gallery root could not be resolved"
return "unresolvable", "Gallery root could not be resolved"
if _is_filesystem_root(canonical):
return False, "Gallery root cannot be a filesystem root"
config_logger.info("Gallery root refused: a filesystem root")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if canonical == _canonical_path(os.path.expanduser("~")):
return False, "Gallery root cannot be the home directory"
config_logger.info("Gallery root refused: the home directory")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if not _lexically_inside(canonical, [canonical for canonical, _ in pairs]):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if not os.path.exists(canonical):
return False, "Gallery root does not exist"
return "not-found", "Gallery root does not exist"
if not os.path.isdir(canonical):
return False, "Gallery root is not a directory"
return "not-directory", "Gallery root is not a directory"
return None, ""
candidate = Path(canonical)
for parent in cls.allowed_gallery_parents():
if candidate.is_relative_to(Path(parent)):
return True, ""
@classmethod
def _lexical_candidates(cls, path: str) -> List[str]:
"""Every spelling ``path`` may resolve to, normalised as text (no IO).
return (
False,
"Gallery root must be inside a ComfyUI directory "
f"(output, input, temp, user) or one listed in {EXTRA_GALLERY_ROOTS_ENV}",
)
An absolute path is itself; a relative one is joined to each
:meth:`path_anchors` entry, as :meth:`resolve_gallery_root` tries them.
"""
path = path.strip()
if os.path.isabs(path):
return [_lexical_path(path)]
anchors = cls.path_anchors()
return [_lexical_path(os.path.join(anchor, path)) for anchor in anchors]
class IntegrationConfig:
+8
View File
@@ -19,7 +19,9 @@ from PIL import Image
from PIL.PngImagePlugin import PngInfo
from tests.browser.request_hold import register_request_holds
from tests.browser import slow_reset_watch
from tests.browser.workspace_activity_fixture import register_activity_commands
from tests.browser.workspace_system_fixture import register_system_commands
from tests.browser.workspace_concept_fixture import (
MAX_EXTRA_IMAGES as CONCEPT_MAX_EXTRA_IMAGES,
MAX_SEEDLESS_IMAGES as CONCEPT_MAX_SEEDLESS_IMAGES,
@@ -197,7 +199,9 @@ def create_app(root: Path) -> web.Application:
register_tagging_commands(routes, api, root, model)
reveal_stub = register_reveal_stub(routes, api, root)
register_activity_commands(app, routes, api)
register_system_commands(app, routes, api, root)
request_holds = register_request_holds(app, routes)
slow_reset_watch.install(app, api)
async def canonical_asset(request: web.Request) -> web.Response:
filename, content_type = CANONICAL_ASSETS[request.path]
@@ -222,6 +226,8 @@ def create_app(root: Path) -> web.Application:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -587,6 +593,8 @@ def create_app(root: Path) -> web.Application:
return web.json_response("dark")
async def cleanup(application: web.Application) -> None:
# A scan of an ambiguous fixture file captures it for Needs review.
api.close_annotation_captures()
api.db.close_all()
seed()
+237
View File
@@ -0,0 +1,237 @@
"""Report a slow test reset with what the server was doing while it waited.
A reset normally answers within about a second. Once one runs past
SLOW_RESET_SECONDS, this writes to stderr, while the reset is still waiting:
- every thread's stack, so a reset waiting on a lock, a full executor or a
slow disk shows where it waits and who holds it;
- the default executor's threads and queued work;
- how many database connections are open, and the WAL size;
- on Linux, each thread's scheduler state and the kernel function it sleeps
in (``D`` with a journal or writeback wait is a slow disk flush, a nanosleep
is SQLite's busy handler, a futex is a Python lock or an idle worker).
When the reset ends it writes how long it took and, for each executor call it
made, how long the call queued and how long it ran.
Every report line starts with REPORT_PREFIX. The worker fixture
(tools/vendor/start-browser-server.mjs) copies those lines to the job log,
so a slow reset is visible even in a test that still passes.
"""
import asyncio
import contextvars
import faulthandler
import os
import sys
import tempfile
import threading
import time
from typing import Callable, TextIO
from aiohttp import web
SLOW_RESET_SECONDS = 2.0
REPORT_PREFIX = "pm-slow-reset| "
_current: contextvars.ContextVar["ResetWatch | None"] = contextvars.ContextVar(
"pm_reset_watch", default=None
)
def is_reset_path(path: str) -> bool:
return path.startswith("/__test__/") and path.endswith("/reset")
class ResetWatch:
"""Time one reset; dump the server's state if it passes the threshold."""
def __init__(
self,
label: str,
describe: Callable[[], list[str]],
*,
threshold: float = SLOW_RESET_SECONDS,
stream: TextIO | None = None,
) -> None:
self.label = label
self.threshold = threshold
self._describe = describe
self._stream = stream
self._start = time.monotonic()
self._calls: list[str] = []
self._calls_lock = threading.Lock()
self._stalled = threading.Event()
self._timer = threading.Timer(threshold, self._report_stall)
self._timer.daemon = True
self._timer.name = "pm-slow-reset-watch"
def start(self) -> "ResetWatch":
self._timer.start()
return self
def elapsed(self) -> float:
return time.monotonic() - self._start
def record_call(self, name: str, queued: float, ran: float) -> None:
with self._calls_lock:
self._calls.append(
f"executor call {name}: queued {queued:.3f}s, ran {ran:.3f}s"
)
def finish(self, outcome: str) -> None:
self._timer.cancel()
elapsed = self.elapsed()
# A coarse clock (Windows) can fire the timer just short of the
# threshold; a reset that was reported always gets its summary.
if elapsed < self.threshold and not self._stalled.is_set():
return
with self._calls_lock:
calls = list(self._calls)
self._write([f"{self.label} {outcome} after {elapsed:.3f}s", *calls])
def _report_stall(self) -> None:
self._stalled.set()
lines = [f"{self.label} still running after {self.elapsed():.3f}s"]
try:
lines.extend(self._describe())
except Exception as error: # a diagnostic never breaks the reset
lines.append(f"state unavailable: {error!r}")
lines.extend(thread_stacks())
self._write(lines)
def _write(self, lines: list[str]) -> None:
stream = self._stream or sys.stderr
text = "".join(
f"{REPORT_PREFIX}{part}\n" for line in lines for part in line.splitlines()
)
try:
stream.write(text)
stream.flush()
except (OSError, ValueError):
pass
def thread_stacks() -> list[str]:
"""Every thread's stack, from faulthandler.
Formatting sys._current_frames() showed a thread blocked in SQLite as idle
(Python 3.14); faulthandler reports where each thread really is.
"""
# Before Python 3.14 faulthandler prints ids only; name each id here.
names = [
f"thread 0x{thread.ident:016x} is {thread.name}"
for thread in threading.enumerate()
if thread.ident is not None
]
with tempfile.TemporaryFile() as dump:
faulthandler.dump_traceback(file=dump, all_threads=True)
dump.seek(0)
return [*names, *dump.read().decode("utf-8", "replace").splitlines()]
def executor_state(loop: asyncio.AbstractEventLoop) -> list[str]:
executor = getattr(loop, "_default_executor", None)
if executor is None:
return ["default executor: not started"]
threads = getattr(executor, "_threads", ())
queue = getattr(executor, "_work_queue", None)
queued = queue.qsize() if queue is not None else "?"
return [
f"default executor: {len(threads)} of "
f"{getattr(executor, '_max_workers', '?')} threads, {queued} queued"
]
def database_state(model) -> list[str]:
"""How many connections are open, and the WAL size.
Never touches another thread's connection: a connection busy in SQLite
holds its mutex, and reading even ``in_transaction`` from here would block
on it while holding the GIL, freezing every thread until that query ends.
"""
with model._conn_lock:
count = len(model._connections)
wal = model.db_path + "-wal"
size = os.path.getsize(wal) if os.path.exists(wal) else 0
return [f"database: {count} connections, WAL {size} bytes"]
def kernel_wait_state() -> list[str]:
"""Each thread's state and kernel wait channel, where /proc has them."""
lines = []
for thread in threading.enumerate():
task = f"/proc/self/task/{thread.native_id}"
try:
with open(f"{task}/stat", encoding="ascii") as stat:
state = stat.read().rsplit(")", 1)[1].split()[0]
with open(f"{task}/wchan", encoding="ascii") as wchan:
channel = wchan.read().strip() or "0"
except (OSError, IndexError):
continue
# The id matches the "Thread 0x..." headers of the stacks below.
lines.append(
f"kernel: {thread.name} (0x{thread.ident:016x}) state {state}, "
f"waiting in {channel}"
)
return lines
def install(app: web.Application, api) -> None:
"""Watch every reset route, and time the executor calls each one makes."""
run_in_executor = api._run_in_executor
async def timed_run_in_executor(func, *args, **kwargs):
watch = _current.get()
if watch is None:
return await run_in_executor(func, *args, **kwargs)
queued_at = time.monotonic()
started: list[float] = []
def timed(*call_args, **call_kwargs):
started.append(time.monotonic())
return func(*call_args, **call_kwargs)
try:
return await run_in_executor(timed, *args, **kwargs)
finally:
end = time.monotonic()
begin = started[0] if started else end
watch.record_call(
getattr(func, "__qualname__", repr(func)),
begin - queued_at,
end - begin,
)
api._run_in_executor = timed_run_in_executor
@web.middleware
async def watch_resets(request: web.Request, handler):
if not is_reset_path(request.path):
return await handler(request)
loop = asyncio.get_running_loop()
def describe() -> list[str]:
return [
*executor_state(loop),
*database_state(api.db.model),
*kernel_wait_state(),
]
watch = ResetWatch(
f"{request.method} {request.path}", describe, threshold=SLOW_RESET_SECONDS
).start()
token = _current.set(watch)
outcome = "failed"
try:
response = await handler(request)
outcome = f"answered {response.status}"
return response
except web.HTTPException as error:
outcome = f"answered {error.status}"
raise
finally:
_current.reset(token)
watch.finish(outcome)
app.middlewares.append(watch_resets)
+779
View File
@@ -0,0 +1,779 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { SIZES, THEMES, seedConcept, openConcept, openProduction, diffInventory, mutate } from './concept-inventory.mjs';
// #249 System S2 Task 6: System's shell and its Overview section (app.js renderSystem 658–706), the Statistics dialog
// ("Usage statistics", a1600–1604) and the health report (a1439–1456) against the immutable concept, on the
// concept-shaped real fixture. Each concept measurement is mapped only by the recorded deviations (D92–D96, D121,
// D122, D123), each commented where it applies; after that the difference must be empty, and a mutation of a
// row, a label or a token must be noticed.
test.use({ locale: 'en-US', timezoneId: 'UTC' });
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.conceptErrors = [];
page.on('pageerror', error => page.conceptErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/system/reset');
expect(page.conceptErrors).toEqual([]);
});
/** Runs in the page; self-contained. System's heading, section nav, Overview body and Save row under `mainSelector`. */
function measureSystem(mainSelector) {
const main = document.querySelector(mainSelector);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const box = (node, origin = main) => {
const rect = node.getBoundingClientRect(), base = origin.getBoundingClientRect();
return { left: Math.round(rect.left - base.left), width: Math.round(rect.width) };
};
// A surface's top: from `.two-col` beside its neighbour, else from the bottom of the surface above it. This pins the
// concept's `.surface + .surface` 18px offset, which concept.css applies inside `.two-col` too (copied, not ruled).
const top = node => {
const rect = node.getBoundingClientRect(), previous = node.previousElementSibling?.getBoundingClientRect();
if (previous && rect.top >= previous.bottom) return Math.round(rect.top - previous.bottom);
return Math.round(rect.top - node.parentElement.getBoundingClientRect().top);
};
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor',
'borderTopLeftRadius', 'opacity', 'cursor') });
const heading = main.querySelector(':scope > .heading');
const nav = main.querySelector(':scope > .section-nav');
const layout = main.querySelector(':scope > .settings-layout');
const metrics = layout.querySelector(':scope > .summary-metrics');
const columns = layout.querySelector(':scope > .two-col');
const save = layout.querySelector(':scope > .save-row');
const firstMetric = metrics.querySelector('.metric');
const firstSurface = columns.querySelector('.surface');
const firstOperation = firstSurface.querySelector('.operation');
return {
tree: [...main.children].map(name),
heading: {
title: text(heading.querySelector('h1')), description: text(heading.querySelector('p.muted')),
actions: [...(heading.querySelector('.heading-actions')?.children ?? [])].map(name),
},
nav: {
role: nav.getAttribute('role'), label: nav.getAttribute('aria-label'),
...css(nav, 'display', 'flexWrap', 'columnGap', 'rowGap', 'paddingBottom', 'marginBottom', 'borderBottomWidth', 'borderBottomColor'),
box: box(nav),
buttons: [...nav.children].map(item => ({ pressed: item.getAttribute('aria-pressed'), ...button(item) })),
},
layout: { tree: [...layout.children].map(name), ...css(layout, 'maxWidth'), box: box(layout) },
metrics: {
...css(metrics, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap', 'marginBottom'), box: box(metrics),
items: [...metrics.children].map(item => ({ tree: [...item.children].map(name), value: text(item.querySelector('strong')),
label: text(item.querySelector('span')), box: box(item) })),
metric: css(firstMetric, 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius', 'backgroundColor'),
strong: css(firstMetric.querySelector('strong'), 'display', 'fontSize', 'fontWeight', 'color'),
span: css(firstMetric.querySelector('span'), 'fontSize', 'color', 'marginTop'),
},
columns: {
...css(columns, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap'), box: box(columns),
surfaces: [...columns.children].map(surface => ({ tree: [...surface.children].map(name), box: box(surface), top: top(surface),
heading: text(surface.querySelector(':scope > h2')),
rows: [...surface.querySelectorAll(':scope > .operation')].map(row => ({ tree: [...row.children].map(name),
title: text(row.querySelector('h3')), description: text(row.querySelector('p')),
action: button(row.querySelector('.row-actions > button')) })) })),
surface: css(firstSurface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius'),
h2: css(firstSurface.querySelector('h2'), 'fontSize', 'fontWeight', 'color', 'marginBottom'),
operation: css(firstOperation, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom',
'borderTopWidth', 'borderTopColor'),
h3: css(firstOperation.querySelector('h3'), 'fontSize', 'fontWeight', 'color'),
p: css(firstOperation.querySelector('p'), 'fontSize', 'color', 'marginTop'),
actions: css(firstOperation.querySelector('.row-actions'), 'display', 'columnGap', 'flexShrink'),
},
save: {
tree: [...save.children].map(name), ...css(save, 'display', 'alignItems', 'justifyContent', 'columnGap', 'marginTop', 'paddingTop'),
status: { text: text(save.querySelector('#settings-status')), ...css(save.querySelector('#settings-status'), 'fontSize', 'color') },
button: button(save.querySelector('button')),
},
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
/** Runs in the page: the open dialog's title, body sequence, tables, operation rows and actions. */
function measureDialog() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const body = dialog.querySelector('#modal-body');
const firstCell = selector => body.querySelector(selector);
const list = body.querySelector(':scope > .operation-list');
const actions = body.querySelector(':scope > .dialog-actions');
return {
title: dialog.querySelector('#modal-title').textContent,
body: [...body.children].map(name),
tables: [...body.querySelectorAll(':scope > table')].map(table => ({
caption: table.caption?.textContent ?? null,
head: [...table.tHead.rows[0].cells].map(cell => [cell.localName, cell.scope, cell.textContent]),
rows: [...table.tBodies[0].rows].map(row => [...row.cells].map(cell => cell.textContent)),
})),
tableStyle: firstCell(':scope > table') && {
table: css(firstCell(':scope > table'), 'width', 'borderCollapse', 'fontSize', 'marginTop'),
caption: css(firstCell(':scope > table caption'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingBottom'),
th: css(firstCell(':scope > table thead th'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingTop', 'paddingLeft',
'borderBottomWidth', 'borderBottomColor'),
td: css(firstCell(':scope > table tbody td'), 'textAlign', 'fontSize', 'color', 'paddingTop', 'paddingLeft',
'borderBottomWidth', 'borderBottomColor'),
},
operations: list && {
...css(list, 'display', 'rowGap', 'marginTop'),
rows: [...list.children].map(row => ({ tree: [...row.children].map(name), title: row.querySelector('h3').textContent,
chip: row.querySelector('.chip').textContent })),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'paddingTop', 'borderTopWidth'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
chip: css(list.querySelector('.operation .chip'), 'fontSize', 'color', 'backgroundColor', 'borderTopLeftRadius', 'paddingLeft'),
},
actions: actions && {
...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(item => ({ text: item.textContent, className: item.className,
...css(item, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') })),
},
};
}
/** Runs in the page: a `.notice` (Maintenance's "Check first…") with its copy. */
function measureNotice(mainSelector) {
const notice = document.querySelector(`${mainSelector} .settings-layout > .notice`);
const css = (node, ...keys) => Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]]));
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const run = notice.querySelector(':scope > button');
return {
tree: [...notice.children].map(name), text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, node.textContent]),
frame: css(notice, 'display', 'alignItems', 'justifyContent', 'columnGap', 'borderTopWidth', 'borderTopColor', 'backgroundColor',
'borderTopLeftRadius', 'paddingTop', 'paddingLeft', 'marginBottom'),
h2: css(notice.querySelector('h2'), 'fontSize', 'fontWeight', 'color'),
p: css(notice.querySelector('p'), 'fontSize', 'color', 'marginTop'),
run: { text: run.textContent, className: run.className, ...css(run, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') },
};
}
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
const number = value => value.toLocaleString('en-US');
const ratingText = value => value === null ? 'Not rated' : `${number(value)} / 5`;
/** The concept's shell and Overview with the recorded deviations applied, given the server's real overview. */
function mapOverviewConcept(reference, overview) {
const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts),
'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) };
const rows = {
// D93: the real pending count; the action is unavailable (dimmed, as the Save row's D92) until the review page.
'3 ambiguous image links': row => ({ ...row,
title: `${number(overview.needs_review.pending)} ambiguous image link${overview.needs_review.pending === 1 ? '' : 's'}`,
action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }),
// D93: no backup has been recorded (verified backups arrive with S8).
'Backup status': row => ({ ...row, description: 'No retained backup record.' }),
// D93: there is no paging, so the shared settings are grid and density.
'One display configuration': row => ({ ...row, description: 'Grid and density settings are shared across views.' }),
// D93: the real sources, not samples.
'2 sample output sources': row => ({ ...row, title: `${number(overview.sources.effective)} output sources` }),
// D96: the dialog shows runs and ratings by category and library totals, not prompt, tag or date usage.
Statistics: row => ({ ...row, description: 'Runs and ratings by category, with library totals.' }),
};
return {
...reference,
// D92: no "Sample configuration" chip.
heading: { ...reference.heading, actions: [] },
// D122: the concept's literal label is its prototype action name ("system-tab sections").
nav: { ...reference.nav, label: 'System sections' },
metrics: { ...reference.metrics, items: reference.metrics.items.map(item => ({ ...item, value: relabel[item.label] })) },
columns: { ...reference.columns, surfaces: reference.columns.surfaces.map(surface => ({ ...surface,
rows: surface.rows.map(row => (rows[row.title] ?? (value => value))(row)) })) },
save: { ...reference.save,
// D92: truthful status copy, and "Save preferences" is aria-disabled (dimmed) while nothing is unsaved.
status: { ...reference.save.status, text: 'No unsaved changes.' },
button: { ...reference.save.button, opacity: '0.55', cursor: 'not-allowed' } },
};
}
async function conceptOverview(page, theme) {
await openConcept(page, 'system', theme);
await expect(page.locator('#main .summary-metrics')).toBeVisible();
await page.mouse.move(0, 0);
return page.evaluate(measureSystem, '#main');
}
async function productionOverview(page, theme) {
await openProduction(page, 'system/overview', theme);
await expect(page.locator('#workspace-main .summary-metrics')).toBeVisible();
// The thumbnail row is its own read (I2): compare once it answered.
await expect(page.locator('#workspace-main .operation h3').filter({ hasText: 'thumbnails' })).toHaveText('No missing thumbnails');
await page.mouse.move(0, 0);
return page.evaluate(measureSystem, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`system shell and overview match the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const overview = await serverJson(request, '/prompt_manager/system/overview');
// The fixture's thumbnails all exist, so the concept's "No missing thumbnails" is also the real title.
expect((await serverJson(request, '/prompt_manager/images/thumbnails/status')).pending).toBe(0);
const concept = await conceptOverview(page, theme);
expect(concept.heading.actions).toEqual(['span.chip']);
// Paired captures are artifacts only; there are no PNG baselines.
await page.screenshot({ path: testInfo.outputPath(`concept-system-${theme}-${width}.png`), fullPage: true });
const production = await productionOverview(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-system-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapOverviewConcept(concept, overview), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
// The recorded differences, by path: D93 the review row (its real count, dimmed action), the backup row, the
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
const UNMAPPED_PATHS = [
'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity',
'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
'save.button.cursor', 'save.button.opacity', 'save.status.text',
];
test('system comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const overview = await serverJson(request, '/prompt_manager/system/overview');
const concept = await conceptOverview(page, 'dark');
const expected = mapOverviewConcept(concept, overview);
const production = await productionOverview(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs exactly where a deviation is recorded, and nowhere else.
expect(diffInventory(concept, production).map(item => item.path).sort()).toEqual(UNMAPPED_PATHS);
const measure = () => page.evaluate(measureSystem, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .two-col > .surface:first-child > .operation:nth-of-type(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['columns.surfaces.0.rows.length', 'columns.surfaces.0.tree.length']));
await undoRow();
// A changed label.
await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompt'; });
expect(await paths()).toEqual(['metrics.items.0.label']);
await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompts'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['metrics.metric.borderTopColor', 'columns.surface.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/** D96: the concept's table, its rows the real ones; then "Library totals" in the same table markup. */
function mapStatisticsConcept(reference, statistics) {
const [runs] = reference.tables;
return {
...reference,
// No sample note (prototype copy); a second table follows the first.
body: ['table', 'table'],
tables: [
{ ...runs, rows: statistics.by_category.map(row => [row.category ?? 'Uncategorized', number(row.runs), ratingText(row.average_rating)]) },
{ caption: 'Library totals', head: [['th', 'col', 'Total'], ['th', 'col', 'Count']],
rows: [['Prompts', number(statistics.totals.total_prompts)], ['Tags', number(statistics.totals.total_tags)],
['Images', number(statistics.totals.total_images)], ['Rated images', number(statistics.rated_images)]] },
],
};
}
// The fixture's image sources and each check's real chip on it: nothing is damaged, Needs review does not exist yet (Q6).
const FIXTURE_SOURCES = ['output', 'input'];
const FIXTURE_CHIPS = Object.freeze({ 'Integrity check': 'Passed', 'Foreign keys & pair references': 'Passed',
'Canonical hashes': 'Passed', 'Pair uniqueness': 'Passed', 'Missing files': 'Passed', 'Ambiguous metadata': 'Not available yet',
Consistency: 'Passed' });
const chipOf = title => {
if (!(title in FIXTURE_CHIPS)) throw new Error(`No expected chip for ${title}`);
return FIXTURE_CHIPS[title];
};
/**
* D95: the concept's health report from its own five rows, labels verbatim, titled "Health report". Only the D95 rows
* are inserted where they are recorded (Pair uniqueness after Canonical hashes, Consistency last); Missing files has a
* row per image source when there are several; each chip is the fixture's expected result.
*/
function mapHealthConcept(reference, sources = FIXTURE_SOURCES) {
const [first] = reference.operations.rows;
const added = title => ({ ...first, title, chip: chipOf(title) });
const rows = reference.operations.rows.flatMap(row => {
const mapped = row.title === 'Missing files' && sources.length > 1
? sources.map(source => ({ ...row, title: `Missing files · ${source}`, chip: chipOf(row.title) }))
: [{ ...row, chip: chipOf(row.title) }];
return row.title === 'Canonical hashes' ? [...mapped, added('Pair uniqueness')] : mapped;
});
return {
...reference,
title: 'Health report',
// No sample note (prototype copy); D95 adds the checked line after the list.
body: ['div.operation-list', 'p.small', 'div.dialog-actions'],
operations: { ...reference.operations, rows: [...rows, added('Consistency')] },
};
}
// Unmapped, the health report differs exactly where D95 is recorded: the title, the body sequence (no sample note, the
// checked line), the rows after Canonical hashes (Pair uniqueness inserted, Missing files per source) and the chips of
// the concept's sample findings.
const UNMAPPED_HEALTH_PATHS = [
'body.0', 'body.1',
...[3, 4].flatMap(row => [`operations.rows.${row}.chip`, `operations.rows.${row}.title`]),
// The concept's five rows end at row 4; rows 5–7 exist only with the D95 rows and the second source.
...[5, 6, 7].flatMap(row => ['chip', 'title', 'tree.0', 'tree.1', 'tree.length'].map(key => `operations.rows.${row}.${key}`)),
'operations.rows.length', 'title',
];
async function conceptDialogs(page, theme, testInfo) {
await openConcept(page, 'system', theme);
await page.locator('#main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
await page.mouse.move(0, 0);
const statistics = await page.evaluate(measureDialog);
await page.screenshot({ path: testInfo.outputPath(`concept-statistics-${theme}.png`) });
await page.keyboard.press('Escape');
await page.locator('#main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
const notice = await page.evaluate(measureNotice, '#main');
await page.locator('#main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Sample health report');
await page.mouse.move(0, 0);
await page.screenshot({ path: testInfo.outputPath(`concept-health-${theme}.png`) });
return { statistics, notice, health: await page.evaluate(measureDialog) };
}
async function productionDialogs(page, theme, testInfo) {
await openProduction(page, 'system/overview', theme);
await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
await page.mouse.move(0, 0);
const statistics = await page.evaluate(measureDialog);
await page.screenshot({ path: testInfo.outputPath(`production-statistics-${theme}.png`) });
await page.keyboard.press('Escape');
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
await expect(page.locator('#workspace-main .notice')).toBeVisible();
const notice = await page.evaluate(measureNotice, '#workspace-main');
await page.locator('#workspace-main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Health report');
await page.mouse.move(0, 0);
await page.screenshot({ path: testInfo.outputPath(`production-health-${theme}.png`) });
return { statistics, notice, health: await page.evaluate(measureDialog) };
}
for (const theme of THEMES) test(`statistics and health dialogs match the concept ${theme}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const concept = await conceptDialogs(page, theme, testInfo);
expect(concept.statistics.body).toEqual(['p.demo-note', 'table']);
expect(concept.health.body).toEqual(['p.demo-note', 'div.operation-list', 'div.dialog-actions']);
const production = await productionDialogs(page, theme, testInfo);
const statistics = await serverJson(request, '/prompt_manager/system/statistics');
const { report } = await serverJson(request, '/prompt_manager/system/health');
// The fixture the expected chips describe: its two sources, and the concept's checks plus the D95 ones.
const missing = report.checks.find(check => check.id === 'missing_files');
expect(missing.sources.map(source => source.name)).toEqual(FIXTURE_SOURCES);
expect(diffInventory(mapStatisticsConcept(concept.statistics, statistics), production.statistics)).toEqual([]);
expect(diffInventory(mapHealthConcept(concept.health), production.health)).toEqual([]);
// D123: Maintenance's notice is the concept's own, delivered ahead of the rest of the section.
expect(diffInventory(concept.notice, production.notice)).toEqual([]);
// Unmapped, the dialogs differ only where D95 and D96 are recorded.
expect(diffInventory(concept.health, production.health).map(item => item.path).sort()).toEqual(UNMAPPED_HEALTH_PATHS);
expect(diffInventory(concept.statistics, production.statistics).map(item => item.path.split('.')[0])
.filter((part, index, all) => all.indexOf(part) === index).sort()).toEqual(['body', 'tables']);
// Mutation check: a changed chip token in the health report is noticed, then undone to an exact match again.
const undo = await mutate(page, { css: '#modal .operation .chip { background: rgb(255, 0, 0); }' });
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(['operations.chip.backgroundColor']);
await undo();
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog))).toEqual([]);
// A renamed check (database/health.py's label) is noticed against the concept's own label.
await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity checks'; });
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(['operations.rows.0.title']);
await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity check'; });
// And a removed "Library totals" table and a changed caption in the statistics dialog.
await page.keyboard.press('Escape');
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
const expectedStatistics = mapStatisticsConcept(concept.statistics, statistics);
const undoTable = await mutate(page, { remove: ['#modal-body > table:nth-of-type(2)'] });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(expect.arrayContaining(['body.length', 'tables.length']));
await undoTable();
await page.evaluate(() => { document.querySelector('#modal-body caption').textContent = 'Runs per category'; });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)).toEqual(['tables.0.caption']);
});
/** Runs in the page: Appearance's surface, every row's text and control, and the concept's row geometry and tokens. */
function measureAppearance(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const surface = layout.querySelector(':scope > section.surface');
const control = row => {
const select = row.querySelector(':scope > select');
if (select) return { kind: 'select', id: select.id, options: [...select.options].map(option => option.textContent),
selected: select.selectedOptions[0]?.textContent ?? null };
const input = row.querySelector(':scope > input');
if (input) return { kind: `input.${input.type}`, id: input.id, value: input.value };
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const chip = row.querySelector(':scope > .chip');
return chip ? { kind: name(chip), text: text(chip), ...css(chip, 'color', 'backgroundColor', 'fontSize') } : null;
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
const operations = [...surface.querySelectorAll(':scope > .operation')];
const firstRow = rows[0];
return {
layout: [...layout.children].map(name),
surface: { tree: [...surface.children].map(name), heading: text(surface.querySelector(':scope > h2')),
intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius') },
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: operations.map(row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), action: text(row.querySelector('.row-actions > button')) })),
row: css(firstRow, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopWidth',
'borderTopColor', 'marginTop'),
label: css(firstRow.querySelector('label'), 'fontSize', 'fontWeight', 'color'),
description: css(firstRow.querySelector('p'), 'fontSize', 'color', 'marginTop', 'maxWidth'),
select: css(firstRow.querySelector('select'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor', 'color', 'borderTopColor'),
heading3: css(surface.querySelector('.setting-row h3'), 'fontSize', 'fontWeight', 'color'),
number: css(surface.querySelector('.setting-row input[type="number"]'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D97: the rows Appearance adds for the legacy gallery settings, after the concept's own, as production renders them.
const ADDED_APPEARANCE_ROWS = [
['card-details', 'Show image details on cards', 'Show each card’s title and file details under the preview.', true],
['viewer-details', 'Collapse viewer details by default', 'Open the image viewer with its details panel hidden.', false],
];
/** The concept's Appearance with the recorded deviations applied, given the server's real settings. */
function mapAppearanceConcept(reference, settings) {
const toggleRow = reference.rows.find(row => row.control?.kind === 'switch');
const added = ADDED_APPEARANCE_ROWS.map(([id, label, description, checked]) => ({
tree: toggleRow.tree, text: [['label', label, id], ['p', description, null]],
control: { ...toggleRow.control, id, checked } }));
return {
...reference,
// D97: two added toggles and the "Reset display defaults" operation close the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.setting-row', 'div.setting-row', 'div.operation'] },
rows: [...reference.rows.map(row => row.control?.id === 'timeout'
// The node search timeout is the server's real `result_timeout`, not the concept's sample 30.
? { ...row, control: { ...row.control, value: String(settings.result_timeout) } } : row), ...added],
operations: [{ tree: ['div', 'div.row-actions'], title: 'Reset display defaults',
description: 'Restore this browser’s display defaults, then save. Other settings are kept.', action: 'Reset' }],
};
}
async function conceptAppearance(page, theme) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Appearance', exact: true }).click();
await expect(page.locator('#main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#main');
}
async function productionAppearance(page, theme) {
await openProduction(page, 'system/appearance', theme);
await expect(page.locator('#workspace-main #timeout')).toBeEnabled();
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`appearance matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-appearance-${theme}-${width}.png`), fullPage: true });
const production = await productionAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-appearance-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapAppearanceConcept(concept, settings), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('appearance comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, 'dark');
const expected = mapAppearanceConcept(concept, settings);
const production = await productionAppearance(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D97 and the real timeout are recorded.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.0', 'operations.length', 'rows.11', 'rows.12', 'rows.13', 'rows.length', 'surface.tree'].sort());
const measure = () => page.evaluate(measureAppearance, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .setting-row:nth-of-type(3)'] });
expect(await paths()).toEqual(expect.arrayContaining(['rows.length', 'surface.tree.length']));
await undoRow();
// A changed label and a changed option.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail sizes'; });
expect(await paths()).toEqual(['rows.1.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail size'; });
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Medium'; });
expect(await paths()).toEqual(expect.arrayContaining(['rows.5.control.options.0']));
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Optimized'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['row.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: Sources & scans (app.js 679–681): the optional notice, then the "Image sources" surface with its
* panel heading, the source list, the two setting rows and the operations, with the concept's geometry and tokens.
*/
function measureSources(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') });
const operation = row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), actions: [...row.querySelectorAll('.row-actions > button')].map(button) });
const notice = layout.querySelector(':scope > .notice');
const surface = layout.querySelector(':scope > section.surface');
const heading = surface.querySelector(':scope > .panel-heading');
const list = surface.querySelector(':scope > #source-list');
const control = row => {
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const input = row.querySelector(':scope > input');
return { kind: `input.${input.type}`, id: input.id, min: input.min, max: input.max, value: input.value,
...css(input, 'width', 'accentColor') };
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
return {
layout: [...layout.children].map(name),
notice: notice && { tree: [...notice.children].map(name),
text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, text(node)]),
actions: [...notice.querySelectorAll(':scope > .row-actions > button')].map(button),
...css(notice, 'display', 'alignItems', 'columnGap', 'borderTopColor', 'backgroundColor', 'paddingTop', 'marginBottom') },
surface: { tree: [...surface.children].map(name), intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
heading: { tree: [...heading.children].map(name), title: text(heading.querySelector('h2')),
action: button(heading.querySelector('button')), ...css(heading, 'display', 'alignItems', 'justifyContent', 'marginBottom'),
h2: css(heading.querySelector('h2'), 'fontSize', 'fontWeight', 'color') },
sources: [...list.children].map(operation),
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: [...surface.querySelectorAll(':scope > .operation')].map(operation),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop',
'paddingBottom', 'borderTopWidth', 'borderTopColor'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
p: css(list.querySelector('.operation p'), 'fontSize', 'color', 'marginTop'),
row: css(rows[0], 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D128: "watching until restart" after monitoring was switched off while the observer still runs.
const WATCH_TEXT = { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', restart_pending: 'restart pending' };
const WATCHED = new Set(['watching', 'stopping']);
// D98: the operation Sources adds after the concept's own, as production renders it.
const REFRESH_FILES = { title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
label: 'Refresh' };
/** The concept's Sources with the recorded deviations applied, given the server's real settings and sources. */
function mapSourcesConcept(reference, { settings, sources }) {
const [sample] = reference.sources;
const scan = sample.actions[0];
return {
...reference,
// The real sources replace the two samples (without the "sample path" note). D98: each configured one has Remove,
// a plain button after its scan.
sources: sources.effective.map(source => ({ ...sample, title: source.label,
description: `${source.path} · ${WATCH_TEXT[source.watch]}`,
actions: [{ ...scan, text: WATCHED.has(source.watch) ? 'Scan changes' : 'Scan folder' }, { ...scan, text: 'Remove' }] })),
rows: reference.rows.map(row => {
if (row.control.id === 'monitoring') return { ...row, control: { ...row.control, checked: settings.monitoring_enabled } };
// D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count.
return { ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } };
}),
// D98: "Refresh files" closes the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.operation'] },
operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title,
description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }],
};
}
async function conceptSources(page, theme, { notice }) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Sources & scans', exact: true }).click();
await expect(page.locator('#main .panel-heading > h2')).toHaveText('Image sources');
if (!notice) await page.locator('#main .notice').getByRole('button', { name: 'Later', exact: true }).click();
await expect(page.locator('#main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#main');
}
async function productionSources(page, theme, { notice }) {
// about:blank has no storage; the workspace page reads the dismissal.
if (!notice) await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
await openProduction(page, 'system/sources', theme);
await expect(page.locator('#workspace-main #source-list .operation')).toHaveCount(2);
await expect(page.locator('#workspace-main #workers')).toBeEnabled();
await expect(page.locator('#workspace-main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#workspace-main');
}
/** The fixture's sources as the concept shows its samples: the first one watched, the second not. */
async function watchedSources(request) {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
return serverJson(request, '/prompt_manager/settings');
}
for (const theme of THEMES) for (const [width, height] of SIZES) for (const notice of [true, false]) {
const variant = notice ? 'with' : 'without';
test(`sources matches the concept ${variant} the notice ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const server = await watchedSources(request);
const concept = await conceptSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`concept-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
const production = await productionSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`production-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapSourcesConcept(concept, server), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('sources comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const server = await watchedSources(request);
const concept = await conceptSources(page, 'dark', { notice: true });
const expected = mapSourcesConcept(concept, server);
const production = await productionSources(page, 'dark', { notice: true });
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only in the real sources, the D98 operation and the D113 range.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.2', 'operations.length', 'rows.1', 'sources.0', 'sources.1', 'surface.tree'].sort());
const measure = () => page.evaluate(measureSources, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed notice and a removed source row.
const undoNotice = await mutate(page, { remove: ['#workspace-main .settings-layout > .notice'] });
expect(await paths()).toEqual(expect.arrayContaining(['layout.length', 'notice.tree.length']));
await undoNotice();
const undoRow = await mutate(page, { remove: ['#workspace-main #source-list > .operation:nth-child(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['sources.length']));
await undoRow();
// A changed label and a changed button.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for outputs'; });
expect(await paths()).toEqual(['rows.0.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for new outputs'; });
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = 'Add source'; });
expect(await paths()).toEqual(['heading.action.text']);
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = '+ Add source'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['operation.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: the open "Add image source" dialog (the concept's `simpleInput`, app.js 1138 and 1497): its title,
* the body's tree, the field's label and input, and the actions, with the concept's geometry and tokens.
*/
function measureAddSource() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const body = dialog.querySelector('#modal-body');
const field = body.querySelector(':scope > .field');
const label = field?.querySelector(':scope > label');
const input = field?.querySelector(':scope > input');
const actions = body.querySelector(':scope > .dialog-actions');
const rect = dialog.getBoundingClientRect();
return {
title: text(dialog.querySelector('#modal-title')),
tree: [...body.children].map(name),
field: field && { tree: [...field.children].map(name), ...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom') },
label: label && { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
input: input && { id: input.id, type: input.type, value: input.value,
...css(input, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
width: Math.round(rect.width),
};
}
/** D127: the field is "Folder path", starts empty (no sample folder) and its action is "Add". */
function mapAddSourceConcept(reference) {
const [cancel, apply] = reference.actions.buttons;
return { ...reference, label: { ...reference.label, text: 'Folder path' }, input: { ...reference.input, value: '' },
actions: { ...reference.actions, buttons: [cancel, { ...apply, text: 'Add' }] } };
}
async function conceptAddSource(page, theme) {
await conceptSources(page, theme, { notice: false });
await page.locator('#main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
async function productionAddSource(page, theme) {
await productionSources(page, theme, { notice: false });
await page.locator('#workspace-main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`add source dialog matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
await watchedSources(request);
const concept = await conceptAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-add-source-${theme}-${width}.png`) });
const production = await productionAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-add-source-${theme}-${width}.png`) });
expect(diffInventory(mapAddSourceConcept(concept), production)).toEqual([]);
});
}
test('add source comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
await watchedSources(request);
const concept = await conceptAddSource(page, 'dark');
const expected = mapAddSourceConcept(concept);
const production = await productionAddSource(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D127 is recorded.
expect(diffInventory(concept, production).map(item => item.path).sort())
.toEqual(['actions.buttons.1.text', 'input.value', 'label.text']);
const paths = async () => diffInventory(expected, await page.evaluate(measureAddSource)).map(item => item.path);
// A changed label, a swapped action order and a changed style.
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder'; });
expect(await paths()).toEqual(['label.text']);
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder path'; });
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
const undoStyle = await mutate(page, { css: 'dialog#modal #action-input { border-color: rgb(255, 0, 0); } dialog#modal .dialog-actions { gap: 31px; }' });
expect(await paths()).toEqual(expect.arrayContaining(['input.borderTopColor', 'actions.columnGap']));
await undoStyle();
expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]);
});
+76 -24
View File
@@ -354,39 +354,48 @@ test('prompt deletion requires confirmation and can retry after a network failur
}
});
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
// The duplicate scan reads the output root only, but the fixture's input root holds a different
// shared/same.png. A bare /images/serve/<path> URL tries every gallery root in turn, so once the
// output copy is deleted it answers 200 with the input copy: check each file in its own root.
const serveIn = (root, path) => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}?root=${root}`;
const rootOf = url => new URL(url, 'http://gallery.test').searchParams.get('root');
// Which copy the scan keeps depends on file times: the fixture stamps shared/same.png with the
// wall clock, and the kernel can stamp its bulk copies a fraction of a millisecond earlier.
// listOrder rewrites the finished scan's groups so a test can pick the order it needs.
async function openDuplicateScan(page, request, listOrder = null) {
await expect(await request.post('/__test__/reset', { data: { extra_images: 2 } })).toBeOK();
if (listOrder) {
await page.route('**/prompt_manager/scan_duplicates/stream', async route => {
const response = await route.fetch();
const body = (await response.text()).replace(/^data: (.*)$/gm, (line, json) => {
const event = JSON.parse(json);
if (event.type !== 'complete') return line;
for (const group of event.duplicates) group.images = listOrder(group.images);
return `data: ${JSON.stringify(event)}`;
});
await route.fulfill({ response, body });
});
}
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(4);
const originals = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
const output = rootOf(originals.find(url => url.includes('/bulk/')));
const input = rootOf(originals.find(url => url.includes('/shared/same.png') && rootOf(url) !== output));
expect(output).not.toBeNull();
expect(input).not.toBeNull();
await page.locator('#settingsBtn').click();
await page.locator('#scanDuplicatesBtn').click();
await expect(page.locator('#duplicatesCount')).toHaveText('1');
const candidates = page.locator('.duplicate-checkbox');
await expect(candidates).toHaveCount(2);
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
const serve = path => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}`;
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serve(selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serve(selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
expect((await request.get(serve(selected))).status()).toBe(404);
await expect(await request.get(serve(retained))).toBeOK();
return { originals, output, input, candidates };
}
async function expectOnlySelectedGone(page, request, { originals, output, input }, selected, retained) {
expect((await request.get(serveIn(output, selected))).status()).toBe(404);
await expect(await request.get(serveIn(output, retained))).toBeOK();
await expect(await request.get(serveIn(input, 'shared/same.png'))).toBeOK();
await page.reload();
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(3);
const remaining = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
@@ -395,4 +404,47 @@ test('duplicate cleanup cancels safely and deletes only the selected file after
expect(originals).toContain(url);
await expect(await request.get(url)).toBeOK();
}
}
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
const scan = await openDuplicateScan(page, request);
const { output, candidates } = scan;
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serveIn(output, selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serveIn(output, selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
test('deleting a duplicate whose path another gallery root also holds removes only the scanned copy', async ({ page, request }) => {
// The order CI saw: a bulk copy stamped first, so shared/same.png is the first Delete candidate.
const ciOrder = ['bulk/0000.png', 'shared/same.png', 'bulk/0001.png'];
const rank = image => ciOrder.indexOf(image.path.replace(/\\/g, '/'));
const scan = await openDuplicateScan(page, request, images => [...images].sort((a, b) => rank(a) - rank(b)));
const selected = await scan.candidates.first().getAttribute('data-image-path');
const retained = await scan.candidates.last().getAttribute('data-image-path');
expect(selected.replace(/\\/g, '/')).toBe('shared/same.png');
await scan.candidates.first().check();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
+49
View File
@@ -284,6 +284,19 @@ test('reload during a scan reattaches to the running job', async ({ page, reques
await expect(pill(page)).toBeHidden();
});
test('an empty Activity says no job has run, and a list that cannot be read says so', async ({ page }) => {
// The empty and failed states were only reached in passing by other tests, so their coverage depended on timing.
const empty = main(page).locator('section.surface > p.empty-state');
await page.goto(`${workspace}#/activity`);
await expect(empty).toHaveText('No jobs have run since ComfyUI started. Scans and other long operations appear here.');
await expect(empty).not.toHaveAttribute('role');
await expect(items(page)).toHaveCount(0);
await page.route(url => url.pathname === '/prompt_manager/jobs', route => route.abort());
await page.reload();
await expect(empty).toHaveText('Activity could not be read. It is tried again shortly.');
await expect(empty).toHaveAttribute('role', 'alert');
});
test('lost server clears the pill and marks jobs ended', async ({ page, request }) => {
// A poll interval after the restart, then failed re-reads backing off for 1 + 2 + 4 s.
test.setTimeout(90_000);
@@ -724,6 +737,42 @@ test('a held legacy job shows no Cancel in the list or its view', async ({ page,
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
});
// Since #289 an item dropped its Cancel job only when it was already in the page, so the first render of jobs the
// store already knew (Activity opened from another page) kept a Cancel job on a job that had ended, and moved a
// running job's progress after its actions. The read Activity starts on opening is held, so that first render is
// what the page shows (the read's own render would otherwise repair it a moment later).
test('jobs that ended before Activity opened show no Cancel job, a running one keeps its progress first', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const finished = await startJob(request, { title: 'Finished scan' });
await finish(request, finished.id);
const cancelled = await startJob(request, { title: 'Cancelled scan' });
await finish(request, cancelled.id, 'cancelled');
await startJob(request, { title: 'Running scan' });
// The page's first list read has all three jobs once the running one shows in the pill.
await page.goto(`${workspace}#/images`);
await expect(pill(page)).toBeVisible();
let hold;
const held = new Promise(resolve => { hold = resolve; });
// Only the first read is held; any later one goes through.
await page.route(url => url.pathname === '/prompt_manager/jobs', route => {
if (!hold) return route.continue();
hold(route);
hold = null;
});
await activityButton(page).click();
const route = await held;
await expect(items(page)).toHaveCount(3);
const item = title => items(page).filter({ has: page.getByRole('link', { name: new RegExp(`${title}$`) }) });
for (const ended of ['Finished scan', 'Cancelled scan']) {
await expect(item(ended).locator('progress, .row-actions')).toHaveCount(0);
await expect(item(ended).getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
}
await expect(item('Running scan').getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(1);
expect(await item('Running scan').evaluate(node => [...node.children].map(child => child.localName + (child.className ? `.${child.className}` : ''))))
.toEqual(['div.panel-heading', 'p', 'progress', 'div.row-actions']);
await route.continue();
});
test('busy start names the running job', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
// The page's timers are Playwright's, so the toast's lifetime below is run out rather than waited for.
+44
View File
@@ -463,6 +463,50 @@ test('collection viewer ignores delayed media callbacks and keeps inspection ret
await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0);
});
test('collection viewer leaving an image Viewer.js has not built yet releases it at once',async({page})=>{
await page.goto(workspace);await expect(cards(page)).toHaveCount(2);
let release,held;const waiting=new Promise(resolve=>{held=resolve;});const pending=new Promise(resolve=>{release=resolve;});
await page.route('**/images/serve/shared/same.png?*',async route=>{
const url=new URL(route.request().url());
if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();await pending;}
await route.continue().catch(()=>{});
});
await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting;
const preview=page.locator('dialog#viewer-dialog');
await page.evaluate(()=>{window.unbuiltImage=document.querySelector('#viewer-stage > img');});
await preview.getByRole('button',{name:'Next media',exact:true}).click();
await expect(preview.getByRole('button',{name:'Zoom in',exact:true})).toHaveAttribute('aria-disabled','false');
// The left image is off the stage. Viewer.js has not built it, so the viewer is destroyed and the src released at
// once: its late load must never build Viewer.js into an image without a parent (getComputedStyle(null)).
release();
await expect.poll(()=>page.evaluate(()=>[window.unbuiltImage.isConnected,window.unbuiltImage.hasAttribute('src'),!!window.unbuiltImage.viewer])).toEqual([false,false,false]);
await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0);
});
test('collection viewer closed while its image finishes loading never builds Viewer.js afterwards',async({page})=>{
await page.goto(workspace);await expect(cards(page)).toHaveCount(2);
let held;const waiting=new Promise(resolve=>{held=resolve;});
// The image is held for the whole test: its late load is delivered below, at the moment Viewer.js cannot cancel.
await page.route('**/images/serve/shared/same.png?*',async route=>{
const url=new URL(route.request().url());
if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();return;}
await route.continue().catch(()=>{});
});
await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting;
const preview=page.locator('dialog#viewer-dialog');
// Viewer.js's own cancel removes its image listeners only while `image.complete` is false. An image that has
// finished downloading but whose load event is still queued is already complete: close in that state, then deliver it.
await page.evaluate(()=>{window.finishingImage=document.querySelector('#viewer-stage > img');Object.defineProperty(window.finishingImage,'complete',{get:()=>true});});
await page.keyboard.press('Escape');await expect(preview).toHaveCount(0);
const after=await page.evaluate(async()=>{
const image=window.finishingImage;image.dispatchEvent(new Event('load'));
// Viewer.js builds an inline viewer in a zero-delay timeout after its image loads; this runs after it.
await new Promise(resolve=>setTimeout(resolve));
return {src:image.hasAttribute('src'),built:!!image.parentNode?.querySelector('.viewer-container')};
});
expect(after).toEqual({src:false,built:false});
});
test('collection viewer describes video transforms and releases native playback on navigation',async({page,request})=>{
await expect(await request.post('/__test__/reset',{data:{include_video:true}})).toBeOK();await page.goto(workspace);
await page.locator('article[data-image-key="0:clip.webm"] .image-open').click();
+48
View File
@@ -577,6 +577,54 @@ test('a video replaced while still starting is reset once it settles', async ({
}
});
test('leaving for another route while a video starts waits for it to settle', async ({ context, browserName }) => {
test.setTimeout(180000);
// A WebKit (GStreamer) video detached before its pipeline had started deadlocked the page (TESTING.md D62); a
// route change detaches the whole page, so the shell renders the next route once the video reports metadata or an
// error. The route changes from the video's own `loadstart`, or a few milliseconds after it.
for (let attempt = 0; attempt < 15; attempt++) {
const page = await context.newPage();
try {
if (attempt === 0) await startCoverage(page, browserName);
await page.addInitScript(() => {
const load = HTMLMediaElement.prototype.load;
HTMLMediaElement.prototype.load = function () {
if (!this.hasAttribute('src') && window.__starting?.node === this) window.__starting.reset = { readyState: this.readyState, failed: this.error !== null, connected: this.isConnected };
return load.call(this);
};
});
await openInspect(page);
await page.evaluate(delay => {
const slot = document.querySelector('#local-preview');
new MutationObserver((records, observer) => {
const video = slot.querySelector('video');
if (!video) return;
observer.disconnect();
const leave = () => {
window.__starting = { node: video, readyState: video.readyState };
location.hash = '#/overview';
};
video.addEventListener('loadstart', () => { if (delay) setTimeout(leave, delay); else leave(); }, { once: true });
}).observe(slot, { childList: true });
}, attempt % 5);
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(main(page).getByRole('heading', { level: 1, name: 'Overview', exact: true })).toBeVisible({ timeout: 8000 });
const { readyState, reset } = await page.evaluate(() => ({ readyState: window.__starting.readyState, reset: window.__starting.reset }));
if (attempt % 5 === 0) expect(readyState).toBe(0);
// The video was reset and left the page only once it had settled.
expect(reset.readyState >= 1 || reset.failed).toBe(true);
expect(reset.connected).toBe(true);
// The page still answers.
await page.locator('aside.sidebar nav').getByRole('link', { name: 'Inspect a file', exact: true }).click();
await chooser(page).setInputFiles(parityPng('f02.png'), { timeout: 8000 });
await expect(preview(page).locator('img.local-file-preview')).toBeVisible({ timeout: 8000 });
if (attempt === 0) await saveCoverage(page, browserName);
} finally {
await page.close();
}
}
});
test('closing the viewer on a video that just started never blocks the page', async ({ context, browserName }) => {
test.setTimeout(180000);
// The viewer's video teardown resets its element; on WebKit a reset of a still-starting pipeline blocked the page.
@@ -0,0 +1,401 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Sources & scans (#249 System S4 Task 11) on the concept-shaped real fixture, whose two sources are this
// server's `output` and `input` folders: the source list, monitoring and worker rows read GET /settings (S4 Task 9),
// Add/Remove edit the host draft behind the one Save row, and every scan starts the session job through the shared
// "Review metadata scan" (D55), scoped to one source from its row (S4 Task 10). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const SETTINGS_URL = '/prompt_manager/settings';
const NOTICE_KEY = 'pm_update_notice_dismissed';
const OUTSIDE_REASON = 'Gallery root must be inside a ComfyUI directory (output, input, temp, user) or one listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS';
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/jobs/restart');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
const toast = page => page.locator('#workspace-toast');
const layoutOf = page => page.locator('#workspace-main .settings-layout');
const sourceList = page => layoutOf(page).locator('#source-list');
const statusOf = page => layoutOf(page).locator('#settings-status');
const saveButton = page => layoutOf(page).getByRole('button', { name: 'Save preferences', exact: true });
const dialog = page => page.locator('dialog#modal[open]');
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
/** Open Sources with the update notice dismissed (or shown, `notice: true`) once the source list has loaded. */
async function openSources(page, { notice = false } = {}) {
if (!notice) await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layoutOf(page).locator('#workers')).toBeEnabled();
return layoutOf(page);
}
/** The source list's rows: `[h3, p, [buttons]]`, then its note line when there is one. */
function rowsOf(page) {
return sourceList(page).evaluate(list => [...list.children].map(row => row.matches('.operation')
? [row.querySelector('h3').textContent, row.querySelector('p').textContent,
[...row.querySelectorAll('.row-actions > button')].map(button => button.textContent)]
: [row.localName, row.className, row.textContent]));
}
const sourceRow = (page, title) => sourceList(page).locator('.operation').filter({ has: page.getByRole('heading', { name: title, exact: true }) });
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed) without scanning', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const layout = await openSources(page, { notice: true });
const notice = layout.locator(':scope > .notice');
await expect(notice.locator('h2')).toHaveText('Bring older images into your prompt history');
await expect(notice.locator('p')).toHaveText('The schema upgrades automatically. An optional metadata scan links existing positives and negatives without regenerating all thumbnails.');
await expect(notice.locator('.row-actions > button')).toHaveText(['Later', 'Review scan']);
await notice.getByRole('button', { name: 'Later', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await expect(dialog(page)).toHaveCount(0);
// The dismissal is the legacy admin page's own, so neither page shows the notice again.
await page.reload();
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(writes).toEqual([]);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Review scan opens the shared Review metadata scan with all sources', async ({ page, request }) => {
const layout = await openSources(page, { notice: true });
// The notice's Review scan dismisses it, as the legacy page's does, and reviews every source.
await layout.locator(':scope > .notice').getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
// Enrich existing metadata is the same review.
const enrich = layout.locator('.operation').filter({ hasText: 'Enrich existing metadata' });
await expect(enrich.locator('p')).toHaveText('Resolve both prompts, deduplicate recipes, preserve uncertain links.');
await enrich.getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Scan changes opens it with that one source ("1 output folder") and starts a scoped job that opens in Activity', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['input'] } })).toBeOK();
const bodies = [];
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan')) bodies.push(item.postDataJSON()); });
await openSources(page);
const input = sourceRow(page, 'ComfyUI input');
await expect(input.locator('p')).toHaveText('input · watching');
await input.getByRole('button', { name: 'Scan changes', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['1 output folder', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
// The toast says it started (or, for a scan this quick, that it already finished); Activity shows its progress.
await expect(toast(page)).toHaveText(/^Scan (started\.|finished · )/);
expect(bodies).toEqual([{ scope: 'new', roots: [1], paths: ['input'] }]);
const id = page.url().split('/').pop();
// The job walks only that source (the count is recorded once the walk has listed it).
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope)
.toMatchObject({ files: 'new', roots: [1], output_folders: 1 });
await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata');
});
test('Add source validates on the server, then marks the host draft dirty; Save persists', async ({ page, request }) => {
// A folder inside ComfyUI's output folder that is not a source yet.
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK();
const layout = await openSources(page);
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
]);
await expect(layout.locator('.panel-heading > h2')).toHaveText('Image sources');
const add = layout.locator('.panel-heading').getByRole('button', { name: '+ Add source', exact: true });
await add.click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Add image source');
await expect(dialog(page).locator('label[for="action-input"]')).toHaveText('Folder path');
await expect(dialog(page).locator('#action-input')).toHaveValue('');
await expect(dialog(page).locator('.dialog-actions > button')).toHaveText(['Cancel', 'Add']);
await dialog(page).locator('#action-input').fill('output/unthumbnailed');
const validated = page.waitForResponse(response => response.url().endsWith('/prompt_manager/settings/validate-source'));
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
expect(await (await validated).json()).toEqual({ success: true, valid: true, label: 'unthumbnailed', path: 'output/unthumbnailed', code: null, reason: null });
await expect(dialog(page)).toHaveCount(0);
await expect(add).toBeFocused();
// Validation wrote nothing: the draft is dirty and the server still has two sources.
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
['unthumbnailed', 'output/unthumbnailed · not saved yet', ['Remove']],
]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.effective).toHaveLength(2);
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const body = (await saved).postDataJSON();
expect(body.gallery_root_paths).toEqual(['output', 'input', 'output/unthumbnailed']);
expect(typeof body.expected_revision).toBe('string');
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// The list is the server's again: three effective sources, each scannable.
await expect(sourceRow(page, 'unthumbnailed').locator('p')).toHaveText('output/unthumbnailed · manual scan');
await expect(sourceRow(page, 'unthumbnailed').getByRole('button', { name: 'Scan folder', exact: true })).toBeVisible();
expect((await serverJson(request, SETTINGS_URL)).sources.effective.map(source => source.path)).toEqual(['output', 'input', 'output/unthumbnailed']);
});
test('invalid source shows the server reason', async ({ page, request }) => {
const layout = await openSources(page);
await layout.getByRole('button', { name: '+ Add source', exact: true }).click();
const field = dialog(page).locator('#action-input');
for (const [path, reason, submit] of [
['output/not-there', 'Gallery root does not exist', 'click'],
// Enter in the field adds too (M6).
['/etc', OUTSIDE_REASON, 'Enter'],
]) {
await field.fill(path);
if (submit === 'Enter') await field.press('Enter');
else await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText(reason);
await expect(field).toHaveAttribute('aria-invalid', 'true');
await expect(field).toHaveValue(path);
}
// A folder already listed (the server's answer names it) is not added twice.
await field.fill('input');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('This folder is already a source.');
// A check that cannot finish says so and adds nothing.
await page.route('**/prompt_manager/settings/validate-source', route => route.fulfill({ status: 500,
json: { success: false, code: null, error: 'Failed to check the source' } }));
await field.fill('output/not-checked');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('The folder could not be checked. Try again.');
await page.unroute('**/prompt_manager/settings/validate-source');
// Typing again clears the reason.
await field.fill('output/unthumbnailed');
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveCount(0);
await expect(field).not.toHaveAttribute('aria-invalid', 'true');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
});
test('Remove marks the draft; removing the last entry states auto-detect', async ({ page, request }) => {
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await sourceRow(page, 'ComfyUI output').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['p', 'small', 'No source is configured. After saving, ComfyUI’s output folder is used.']]);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// Saved: ComfyUI's output folder is detected and listed without Remove.
await expect.poll(rowsOf.bind(null, page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder']],
['p', 'small', 'No source is configured; ComfyUI’s output folder is used.'],
]);
const { sources } = await serverJson(request, SETTINGS_URL);
expect([sources.configured, sources.auto_detected]).toEqual([[], true]);
});
test('a configured source that does not exist says why and can be removed', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await openSources(page);
// The fixture's unmounted drive is not inside a ComfyUI folder, so the server says that (it never probes outside).
expect((await rowsOf(page)).at(-1)).toEqual(['unmounted-drive', `unmounted-drive · ${OUTSIDE_REASON}`, ['Remove']]);
await sourceRow(page, 'unmounted-drive').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output', 'input']);
});
test('watching only with a live observer; restart pending otherwise', async ({ page, request }) => {
// No image monitor in this process: every source is scanned by hand.
await openSources(page);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · manual scan', 'input · manual scan']);
// A live observer that scheduled output: output is watched; input was added since it started.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · restart pending']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan folder']);
// An observer that died watches nothing.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'], alive: false } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · restart pending', 'input · restart pending']);
});
test('Watch for new outputs saves monitoring_enabled; a restart applies it', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'] } })).toBeOK();
const layout = await openSources(page);
const monitoring = layout.getByRole('switch', { name: 'Watch for new outputs', exact: true });
await expect(monitoring).toBeChecked();
await expect(layout.locator('.setting-row').filter({ has: page.locator('#monitoring') }).locator('p'))
.toHaveText('New files enter the same library. Existing scans do not count as new runs.');
await monitoring.uncheck();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await (await saved).json();
expect(answer.restart_required_fields).toEqual(['monitoring_enabled']);
// D128: saved for the next start, and the status says a restart applies it.
await expect(statusOf(page)).toHaveText('Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The running observer keeps watching until ComfyUI restarts, and the list says so.
const { settings, monitor } = await serverJson(request, SETTINGS_URL);
expect([settings.monitoring_enabled, monitor.observer_alive]).toEqual([false, true]);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching until restart', 'input · watching until restart']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan changes']);
// Switched back on before the restart: the observer still runs, so nothing waits for one.
await monitoring.check();
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · watching']);
});
test('worker range max is cpu_count', async ({ page, request }) => {
const { settings } = await serverJson(request, SETTINGS_URL);
const layout = await openSources(page);
const workers = layout.locator('#workers');
await expect(layout.locator('label[for="workers"]')).toHaveText('Worker threads');
await expect(workers).toHaveAttribute('type', 'range');
await expect(workers).toHaveAttribute('min', '1');
await expect(workers).toHaveAttribute('max', String(settings.cpu_count));
await expect(workers).toHaveValue(String(settings.worker_threads));
const next = settings.worker_threads === 1 ? settings.cpu_count : 1;
await workers.fill(String(next));
await expect(statusOf(page)).toHaveText(next === settings.worker_threads ? 'No unsaved changes.' : 'Unsaved changes.');
test.skip(next === settings.worker_threads, 'One core: the range has a single value');
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
expect((await saved).postDataJSON().worker_threads).toBe(next);
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.worker_threads).toBe(next);
});
test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page }) => {
const layout = await openSources(page);
const refresh = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Refresh files', exact: true }) });
await expect(refresh.locator('p')).toHaveText('Read every source again to find files added or removed outside ComfyUI.');
const answered = page.waitForResponse(response => response.url().endsWith('/prompt_manager/gallery/rescan'));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
const { total, roots } = await (await answered).json();
expect(roots).toBe(2);
await expect(toast(page)).toHaveText(`Files refreshed. ${total.toLocaleString('en-US')} files in 2 output folders.`);
// A failed refresh says so.
await page.route('**/prompt_manager/gallery/rescan', route => route.fulfill({ status: 500, json: { success: false, error: 'busy disk' } }));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.');
});
test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => {
const layout = await openSources(page);
const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.');
const popup = page.waitForEvent('popup');
await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click();
const gallery = await popup;
expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery');
await gallery.close();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
});
test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => {
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
// The draft is kept for the next Save, and the server still has both sources.
await expect(sourceList(page).locator('.operation')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
// An answer that does not name its job (an older server): the toast names the running one from the job list.
await page.route(`**${SETTINGS_URL}`, route => (route.request().method() === 'POST'
? route.fulfill({ status: 409, json: { success: false, code: 'busy', error: 'Scan for metadata already running' } })
: route.continue()));
await page.locator('#workspace-toast').evaluate(node => { node.textContent = ''; });
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await page.unroute(`**${SETTINGS_URL}`);
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
});
test('a source list that cannot be read says so and retries', async ({ page }) => {
await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
let failing = true;
await page.route(`**${SETTINGS_URL}`, route => (failing && route.request().method() === 'GET'
? route.fulfill({ status: 500, json: { success: false, error: 'Injected failure' } }) : route.continue()));
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).getByRole('alert')).toHaveText('Sources could not be loaded. Your saved data has not changed.');
await expect(sourceList(page).locator('.operation')).toHaveCount(0);
// The server settings rows say so too, and stay disabled.
await expect(layoutOf(page).locator('#workers')).toBeDisabled();
failing = false;
await sourceList(page).getByRole('button', { name: 'Retry', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
});
test('a scan from a list that changed since it was read starts nothing, says why and reads the list again', async ({ page, request }) => {
await openSources(page);
// The legacy page (or another tab) removes ComfyUI output: the page's "input" is index 1, now past the end.
await expect(await request.post(SETTINGS_URL, { data: { gallery_root_paths: ['input'] } })).toBeOK();
const refused = page.waitForResponse(response => response.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
const answer = await refused;
expect([answer.status(), (await answer.json()).code]).toEqual([409, 'sources-changed']);
await expect(toast(page)).toHaveText('The sources changed since they were loaded. Reload them.');
// No scan started (the other save ran as its own source-change hold).
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['source_change']);
// Read again: the one source is index 0 now, and its scan starts.
await expect.poll(rowsOf.bind(null, page)).toEqual([['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('No unsaved changes.');
const started = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
expect((await started).postDataJSON()).toEqual({ scope: 'new', roots: [0], paths: ['input'] });
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
});
test('a save refused for one unusable source names it and marks its row', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source', { data: { inside: true } })).toBeOK();
await openSources(page);
await expect(sourceRow(page, 'unplugged').locator('p')).toHaveText('output/unplugged · manual scan');
// The drive is unplugged after the page read the list; removing another source resends the whole list.
await expect(await request.post('/__test__/system/unplug')).toBeOK();
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await saved;
expect(answer.status()).toBe(400);
expect(await answer.json()).toMatchObject({ code: 'invalid-source', source: 'output/unplugged', index: 1 });
await expect(statusOf(page)).toHaveText("output/unplugged can't be used: Gallery root does not exist. Remove it or reconnect it, then save.");
await expect(sourceRow(page, 'unplugged')).toHaveAttribute('aria-invalid', 'true');
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(3);
// Removed, the list saves.
await sourceRow(page, 'unplugged').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(0);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output']);
});
+621
View File
@@ -0,0 +1,621 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Overview (#249 System S2 Task 6) on the concept-shaped real fixture: the metrics, rows, Statistics dialog and
// Health report read the server, never sample values (D93, D95, D96). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const OVERVIEW_URL = '/prompt_manager/system/overview';
const THUMBNAILS_URL = '/prompt_manager/images/thumbnails/status';
const number = value => value.toLocaleString('en-US');
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
page.manifest = await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
const layoutOf = page => page.locator('#workspace-main .settings-layout');
async function openOverview(page) {
await page.goto(`${workspace}#/system/overview`);
await expect(layoutOf(page).locator(':scope > .summary-metrics')).toBeVisible();
return layoutOf(page);
}
/** The rows of one Overview surface: `[h3, p, button]`. */
function rowsOf(layout, heading) {
return layout.locator('.two-col > section.surface').filter({ has: layout.page().getByRole('heading', { name: heading, exact: true }) })
.locator('.operation').evaluateAll(rows => rows.map(row => [row.querySelector('h3').textContent,
row.querySelector('p').textContent, row.querySelector('.row-actions button').textContent]));
}
test('overview shows real counts and Not checked', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const overview = await serverJson(request, OVERVIEW_URL);
const thumbnails = await serverJson(request, THUMBNAILS_URL);
// The concept fixture: six positives, their saved negatives and 24 linked images, all with thumbnails.
expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]);
// I2: the Overview never walks the sources for thumbnails; the row reads the status route itself.
expect(overview).not.toHaveProperty('thumbnails');
// The fixture records no ambiguous file: the Needs review queue is empty.
expect(overview.needs_review).toEqual({ pending: 0 });
const layout = await openOverview(page);
expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item =>
[item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([
[number(overview.library.positive_prompts), 'Positive prompts'],
[number(overview.library.negative_variants), 'Negative variants'],
[number(overview.images.indexed), 'Images'],
['Not checked', 'Database integrity'],
]);
expect(await rowsOf(layout, 'Library health')).toEqual([
['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'],
['Backup status', 'No retained backup record.', 'View backups'],
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
]);
expect(await rowsOf(layout, 'Workspace preferences')).toEqual([
['One display configuration', 'Grid and density settings are shared across views.', 'Appearance'],
['2 output sources', 'Monitoring and enrichment share one source list.', 'Manage sources'],
['Statistics', 'Runs and ratings by category, with library totals.', 'Explore'],
]);
expect(writes).toEqual([]);
});
test('Run health checks opens Health report and updates metric 4', async ({ page, request }) => {
await page.goto(`${workspace}#/system/maintenance`);
const layout = layoutOf(page);
await expect(layout.locator(':scope > .notice h2')).toHaveText('Check first. Review changes. Then apply.');
// The rest of Maintenance is still the existing tools' (D94).
await expect(layout.locator(':scope > section.handoff h2')).toHaveText('Your current tools are ready');
const run = layout.getByRole('button', { name: 'Run health checks', exact: true });
await run.click();
const dialog = page.locator('dialog#modal[open]');
await expect(dialog.locator('#modal-title')).toHaveText('Health report');
const { report } = await serverJson(request, '/prompt_manager/system/health');
expect(report.status).toBe('incomplete');
expect(await dialog.locator('.operation-list > .operation').evaluateAll(rows => rows.map(row =>
[row.querySelector('h3').textContent, row.querySelector('span.chip').textContent]))).toEqual([
['Integrity check', 'Passed'], ['Foreign keys & pair references', 'Passed'], ['Canonical hashes', 'Passed'],
['Pair uniqueness', 'Passed'], ['Missing files · output', 'Passed'], ['Missing files · input', 'Passed'],
['Ambiguous metadata', 'Not available yet'], ['Consistency', 'Passed'],
]);
await expect(dialog.locator('#modal-body > p.small')).toHaveText(/^Checked .+\. Some checks could not run, so this report is incomplete\.$/);
await expect(dialog.locator('.dialog-actions button')).toHaveText(['Close', 'Review missing links']);
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(run).toBeFocused();
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('Incomplete');
// A missing file and an unmounted source: the report counts the readable sources and names the other one.
await expect(await request.post('/__test__/system/missing-file')).toBeOK();
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
await layout.getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(dialog.locator('.operation').filter({ hasText: 'Missing files' }).locator('.chip'))
.toHaveText(['1 link needs review', 'Passed', 'Not available']);
await expect(dialog.locator('.operation h3').filter({ hasText: 'Missing files' }))
.toHaveText(['Missing files · output', 'Missing files · input', 'Missing files · unmounted-drive']);
// Review missing links goes to Maintenance, where the cleanup's existing tool is linked.
await dialog.getByRole('button', { name: 'Review missing links', exact: true }).click();
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(page).toHaveURL(`${workspace}#/system/maintenance`);
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('1 issue');
});
test('a health check that fails says so and keeps the last result', async ({ page, request }) => {
await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/health' } })).toBeOK();
await page.goto(`${workspace}#/system/maintenance`);
const run = layoutOf(page).getByRole('button', { name: 'Run health checks', exact: true });
await run.click();
await expect(page.locator('#workspace-toast')).toHaveText('The health check could not finish. Nothing was changed.');
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(run).not.toHaveAttribute('aria-disabled', 'true');
await openOverview(page);
await expect(layoutOf(page).locator('.metric').nth(3).locator('strong')).toHaveText('Not checked');
});
test('thumbnail row reflects the status endpoint', async ({ page, request }) => {
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
const status = await serverJson(request, '/prompt_manager/images/thumbnails/status');
expect(status.pending).toBe(3);
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('3 missing thumbnails');
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources.
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/sources`);
});
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
const held = async () => (await serverJson(request, '/__test__/hold')).held[THUMBNAILS_URL] ?? 0;
await expect(await request.post('/__test__/hold', { data: { path: THUMBNAILS_URL } })).toBeOK();
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
// The other values do not wait for the walk of every source.
await expect(row.locator('h3')).toHaveText('Checking thumbnails…');
await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4);
await expect.poll(held).toBe(1);
await expect(await request.post('/__test__/release')).toBeOK();
await expect(row.locator('h3')).toHaveText('No missing thumbnails');
});
test('a failed thumbnail read is unavailable, never zero', async ({ page }) => {
await page.route(`**${THUMBNAILS_URL}`, route => route.fulfill({ status: 500, contentType: 'application/json',
body: JSON.stringify({ success: false, error: 'Injected failure' }) }));
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('Thumbnail status unavailable');
await expect(layout.getByRole('alert')).toHaveCount(0);
await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24');
});
test('ambiguous row reads the pending count; its action waits for the review page', async ({ page }) => {
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ hasText: 'ambiguous image links' });
await expect(row.locator('h3')).toHaveText('0 ambiguous image links');
await expect(row.locator('p')).toHaveText('Review metadata before assigning a positive / negative pair.');
const review = row.getByRole('button', { name: 'Review metadata', exact: true });
await expect(review).toHaveAttribute('aria-disabled', 'true');
await review.click({ force: true });
await expect(page).toHaveURL(`${workspace}#/system/overview`);
});
test('View backups / Appearance / Manage sources switch sections', async ({ page }) => {
for (const [label, section, pressed] of [['View backups', 'backups', 'Backup & export'], ['Appearance', 'appearance', 'Appearance'],
['Manage sources', 'sources', 'Sources & scans']]) {
const layout = await openOverview(page);
await layout.getByRole('button', { name: label, exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/${section}`);
await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText(pressed);
await expect(layout.locator(':scope > .summary-metrics')).toHaveCount(0);
}
});
test('statistics dialog lists runs by category and library totals', async ({ page, request }) => {
const statistics = await serverJson(request, '/prompt_manager/system/statistics');
expect(statistics.by_category.length).toBeGreaterThan(1);
const layout = await openOverview(page);
const explore = layout.getByRole('button', { name: 'Explore', exact: true });
await explore.click();
const dialog = page.locator('dialog#modal[open]');
await expect(dialog.locator('#modal-title')).toHaveText('Usage statistics');
const tables = dialog.locator('#modal-body > table');
await expect(tables.locator('caption')).toHaveText(['Runs by category', 'Library totals']);
await expect(tables.first().locator('thead th')).toHaveText(['Category', 'Runs', 'Avg. rating']);
const cells = table => table.locator('tbody tr').evaluateAll(rows => rows.map(row => [...row.cells].map(cell => cell.textContent)));
expect(await cells(tables.first())).toEqual(statistics.by_category.map(row => [row.category ?? 'Uncategorized',
number(row.runs), row.average_rating === null ? 'Not rated' : `${number(row.average_rating)} / 5`]));
expect(await cells(tables.nth(1))).toEqual([['Prompts', number(statistics.totals.total_prompts)],
['Tags', number(statistics.totals.total_tags)], ['Images', number(statistics.totals.total_images)],
['Rated images', number(statistics.rated_images)]]);
await page.keyboard.press('Escape');
await expect(explore).toBeFocused();
// A statistics read that fails says so; the Overview stays.
await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/statistics' } })).toBeOK();
await explore.click();
await expect(page.locator('#workspace-toast')).toHaveText('Statistics could not be loaded. Your saved data has not changed.');
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
});
test('load failure shows Retry and no zeros', async ({ page, request }) => {
await expect(await request.post('/__test__/system/fail', { data: { path: OVERVIEW_URL } })).toBeOK();
await page.goto(`${workspace}#/system/overview`);
const layout = layoutOf(page);
await expect(layout.getByRole('alert')).toHaveText('The overview could not be loaded. Your saved data has not changed.');
await expect(layout.locator('.metric')).toHaveCount(0);
await expect(layout).not.toContainText('0');
await layout.getByRole('button', { name: 'Retry', exact: true }).click();
await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4);
await expect(layout.getByRole('alert')).toHaveCount(0);
});
test('out-of-order overview responses never repaint a newer section', async ({ page, request }) => {
const held = async () => (await serverJson(request, '/__test__/hold')).held[OVERVIEW_URL] ?? 0;
await expect(await request.post('/__test__/hold', { data: { path: OVERVIEW_URL } })).toBeOK();
await page.goto(`${workspace}#/system/overview`);
const layout = layoutOf(page);
await expect(layout.getByRole('status')).toHaveText('Loading the overview…');
await expect.poll(held).toBe(1);
// Leave for Appearance, come back (a second read), then leave again before either answers.
const nav = page.locator('#workspace-main .section-nav');
await nav.getByRole('button', { name: 'Appearance', exact: true }).click();
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect.poll(held).toBe(2);
// Integrations still hands off (S10), so its body is one known panel.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await expect(await request.post('/__test__/release')).toBeOK();
await expect.poll(held).toBe(0);
// Both late answers reached a section that had left: Integrations keeps its own body.
await expect(layout.locator(':scope > section.handoff')).toBeVisible();
await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve))));
await expect(layout.locator('.summary-metrics')).toHaveCount(0);
expect(await layout.evaluate(node => [...node.children].map(child => child.className))).toEqual(['surface handoff', 'save-row']);
// Back on Overview, one fresh read paints one set of metrics.
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.summary-metrics')).toHaveCount(1);
await expect(layout.locator('.two-col')).toHaveCount(1);
});
// System › Appearance (#249 System S3 Task 8): the browser display defaults (`gallerySettings`, preferences.mjs) and the
// two host fields behind the one Save row, and the consumers that read them on their next visit.
const SETTINGS_URL = '/prompt_manager/settings';
const INSPECT_PATH = '/prompt_manager/images/inspect/';
const statusOf = page => page.locator('#workspace-main #settings-status');
const saveOf = page => page.locator('#workspace-main .save-row > button');
const storedSettings = page => page.evaluate(() => JSON.parse(localStorage.getItem('gallerySettings') || 'null'));
const imageCards = page => page.locator('#workspace-main article.media-card');
const panel = page => page.getByRole('complementary', { name: 'Selected image details', exact: true });
async function openAppearance(page) {
await page.goto(`${workspace}#/system/appearance`);
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
// The host rows are enabled once the server settings answered.
await expect(page.locator('#timeout')).toBeEnabled();
}
async function saveAppearance(page, text) {
await expect(saveOf(page)).not.toHaveAttribute('aria-disabled', 'true');
await saveOf(page).click();
await expect(statusOf(page)).toHaveText(text);
}
/** Set toggles (by row id) through their Toolkit switches. */
async function toggle(page, values) {
for (const [id, checked] of Object.entries(values)) await page.locator(`#${id}`).setChecked(checked, { force: true });
}
/**
* A request barrier: one request issued and answered after everything the page has asked for so far, so a read the
* last action started has been seen (Playwright reports a page's requests in order).
*/
async function requestBarrier(page) {
expect(await page.evaluate(url => fetch(url, { cache: 'no-store' }).then(response => response.ok), SETTINGS_URL)).toBe(true);
}
async function openImages(page) {
await page.goto(`${workspace}#/images`);
await expect(imageCards(page).first()).toBeVisible();
}
test('defaults apply to a new visit but not to the current view or a saved view', async ({ page, request }) => {
const created = await request.post('/prompt_manager/saved-views', { data: { page: 'images', name: 'Everything',
state: { q: '', category: '', sort: 'newest', source: 'all', folder: '', media: 'all', tags: [], favorites: false } } });
expect(created.status()).toBe(201);
const view = (await created.json()).view;
try {
await openImages(page);
// The current view: the toolbar sets List.
await page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'List', exact: true }).click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await openAppearance(page);
await expect(page.locator('#default-view')).toHaveValue('grid');
await page.locator('#default-view').selectOption('masonry');
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect([stored.workspaceLayout, stored.workspaceDensity]).toEqual(['masonry', 'compact']);
// Back in Images (the page session's last visit): the current view keeps its own layout and size.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// The saved view opens as before.
await page.goto(`${workspace}#/saved/${view.id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// A new visit starts from the defaults.
await openImages(page);
await page.reload();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'masonry');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
await expect(page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'Masonry', exact: true }))
.toHaveAttribute('aria-pressed', 'true');
} finally {
await request.delete(`/prompt_manager/saved-views/${view.id}`);
}
});
test('the defaults reach the Images page only; embedded galleries keep their own presentation', async ({ page }) => {
await openAppearance(page);
await page.locator('#default-view').selectOption('list');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
// The prompt history's shared gallery has no toolbar to change them, so it opens as it always did.
await page.goto(`${workspace}#/library/${page.manifest.prompts[0].positive_id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-density', 'comfortable');
// The Images page starts from them.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
});
test('metadata off shows an explicit load and loads once on request', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'metadata-auto': false });
await saveAppearance(page, 'Saved in this browser.');
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
await openImages(page);
const first = imageCards(page).first();
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).getByRole('status')).toHaveText('Metadata loads when you ask for it.');
const load = panel(page).getByRole('button', { name: 'Load metadata', exact: true });
await expect(load).toBeVisible();
expect(reads).toEqual([]);
await load.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(1);
// The same file again reads nothing more (the cache is on).
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await requestBarrier(page);
expect(reads).toHaveLength(1);
// The viewer's details use the same inspector: another file waits there too, and loads on request.
await imageCards(page).nth(1).locator('.image-open').click();
const viewerDetails = page.locator('#viewer-details');
await expect(viewerDetails.getByRole('status')).toHaveText('Metadata loads when you ask for it.');
expect(reads).toHaveLength(1);
await viewerDetails.getByRole('button', { name: 'Load metadata', exact: true }).click();
await expect(viewerDetails.locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(2);
});
test('cache off rereads the current selection; on reuses the one current record', async ({ page }) => {
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
for (const cache of [true, false]) {
if (!cache) {
await openAppearance(page);
await toggle(page, { 'metadata-cache': false });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
}
await openImages(page);
const info = imageCards(page).first().locator('.image-card-actions button').first();
await info.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await expect.poll(() => reads.length).toBeGreaterThan(0);
await requestBarrier(page);
reads.length = 0;
// Inspect the selected file twice more.
for (const _ of [1, 2]) {
const answered = cache ? null : page.waitForResponse(response => new URL(response.url()).pathname.startsWith(INSPECT_PATH));
await info.click();
if (answered) await answered;
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
}
await requestBarrier(page);
expect(reads, `cache ${cache}`).toHaveLength(cache ? 0 : 2);
}
});
test('paths hidden only changes display', async ({ page }) => {
const rows = async () => page.locator('#viewer-details details').filter({ hasText: 'File & provenance' }).locator('dt')
.evaluateAll(terms => terms.map(term => term.textContent));
const openViewer = async () => {
await openImages(page);
await imageCards(page).first().locator('.image-open').click();
const provenance = page.locator('#viewer-details details').filter({ hasText: 'File & provenance' });
await provenance.locator('summary').click();
await expect(provenance.locator('dl')).toBeVisible();
return provenance;
};
let provenance = await openViewer();
expect(await rows()).toContain('Path');
const before = { show: await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count(),
download: await provenance.getByRole('link', { name: 'Download original', exact: true }).getAttribute('href') };
await page.keyboard.press('Escape');
await openAppearance(page);
await toggle(page, { paths: false });
await saveAppearance(page, 'Saved in this browser.');
provenance = await openViewer();
const shown = await rows();
expect(shown).not.toContain('Path');
expect(shown).toEqual(expect.arrayContaining(['Source', 'Size']));
// Everything else is as it was: the folder action and the original download stay.
expect(await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count()).toBe(before.show);
await expect(provenance.getByRole('link', { name: 'Download original', exact: true })).toHaveAttribute('href', before.download);
});
test('video defaults reach the viewer', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'video-auto': true, 'video-muted': false, 'video-loop': false });
await saveAppearance(page, 'Saved in this browser.');
expect(await page.evaluate(async () => (await import('/prompt_manager/js/workspace/media-viewer.mjs')).readViewerPreferences()))
.toEqual({ videoAutoplay: true, videoMute: false, videoLoop: false });
const stored = await storedSettings(page);
expect([stored.videoAutoplay, stored.videoMute, stored.videoLoop]).toEqual([true, false, false]);
});
test('host fields round-trip through /settings with revision', async ({ page, request }) => {
const posts = [];
page.on('request', item => { if (item.method() === 'POST' && new URL(item.url()).pathname === SETTINGS_URL) posts.push(item.postDataJSON()); });
const before = await serverJson(request, SETTINGS_URL);
await openAppearance(page);
await expect(page.locator('#open-mode')).toHaveValue(before.settings.webui_display_mode);
await expect(page.locator('#timeout')).toHaveValue(String(before.settings.result_timeout));
// Only whole seconds from 0 to 300: anything else is highlighted and never sent.
for (const bad of ['301', '-1', '2.5']) {
await page.locator('#timeout').fill(bad);
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).toHaveAttribute('aria-invalid', 'true');
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
await expect(saveOf(page)).toHaveAttribute('aria-disabled', 'true');
}
await page.locator('#timeout').fill('0');
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).not.toHaveAttribute('aria-invalid');
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved to the server configuration.');
expect(posts).toEqual([{ webui_display_mode: 'popup', result_timeout: 0, expected_revision: before.revision }]);
const after = await serverJson(request, SETTINGS_URL);
expect([after.settings.webui_display_mode, after.settings.result_timeout]).toEqual(['popup', 0]);
expect(after.revision).not.toBe(before.revision);
// No browser key was written for a host field.
expect(await storedSettings(page)).toBeNull();
});
test('server settings that cannot be read say so in their rows', async ({ page }) => {
await page.route(url => url.pathname === SETTINGS_URL, route => (route.request().method() === 'GET'
? route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, error: 'Injected failure' }) })
: route.continue()));
await page.goto(`${workspace}#/system/appearance`);
for (const id of ['open-mode', 'timeout']) {
const row = page.locator('#workspace-main .setting-row').filter({ has: page.locator(`#${id}`) });
await expect(row.locator('p.small')).toHaveText('Server settings could not be read.');
await expect(page.locator(`#${id}`)).toBeDisabled();
}
// The browser rows are this browser's, and work as before.
await expect(page.locator('#default-view')).toBeEnabled();
await expect(page.locator('#workspace-main .setting-row p.small')).toHaveCount(2);
});
test('stale revision shows the conflict status', async ({ page, request }) => {
await openAppearance(page);
// Another page saves after this one read the settings.
const current = await serverJson(request, SETTINGS_URL);
await expect(await request.post(SETTINGS_URL, { data: { result_timeout: 42, expected_revision: current.revision } })).toBeOK();
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#open-mode')).toHaveAttribute('aria-invalid', 'true');
// The draft is kept and the section now shows the other page's value.
await expect(page.locator('#open-mode')).toHaveValue('popup');
await expect(page.locator('#timeout')).toHaveValue('42');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('newtab');
// With the reloaded revision, Save succeeds.
await saveAppearance(page, 'Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('popup');
});
test('reload keeps saved values; a new tab reads them', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#image-quality').selectOption('high');
await toggle(page, { 'card-details': false });
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved in this browser. Saved to the server configuration.');
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#image-quality')).toHaveValue('high');
await expect(page.locator('#card-details')).not.toBeChecked();
await expect(page.locator('#open-mode')).toHaveValue('popup');
const other = await context.newPage();
await openAppearance(other);
await expect(other.locator('#image-quality')).toHaveValue('high');
await expect(other.locator('#card-details')).not.toBeChecked();
// A change saved in one tab reaches the other's clean controls; a draft it overtakes is flagged and kept.
await other.locator('#density').selectOption('large');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#density')).toHaveValue('large');
await expect(other.locator('#settings-status')).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(other.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await page.locator('#image-quality').selectOption('low');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#image-quality')).toHaveValue('low');
await other.close();
});
test('a change another tab saved while System was closed is a conflict on return, never saved over silently', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
// Leave System for Images: the draft stays for the page session, but System no longer listens.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
const other = await context.newPage();
await openAppearance(other);
await other.locator('#density').selectOption('large');
await saveAppearance(other, 'Saved in this browser.');
await other.close();
// Back in System: the draft is kept and flagged, and nothing was written over the other tab's value.
await page.evaluate(() => { location.hash = '#/system/appearance'; });
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await expect(statusOf(page)).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await expect(page.locator('#density')).toHaveValue('compact');
expect((await storedSettings(page)).workspaceDensity).toBe('large');
// Saving after the conflict was shown is an explicit choice: it wins.
await saveAppearance(page, 'Saved in this browser.');
expect((await storedSettings(page)).workspaceDensity).toBe('compact');
await expect(page.locator('#density')).not.toHaveAttribute('aria-invalid');
});
test('a browser that refuses storage is saved for this tab only, never "Saved"', async ({ page }) => {
await openAppearance(page);
await page.evaluate(() => {
Storage.prototype.setItem = function setItem() { throw new DOMException('Quota exceeded', 'QuotaExceededError'); };
});
await page.locator('#default-view').selectOption('list');
await saveAppearance(page, 'Saved for this tab only; browser storage is unavailable.');
expect(await storedSettings(page)).toBeNull();
// This tab's next Images visit still uses it.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
});
test('added rows reach their consumers', async ({ page }) => {
await openImages(page);
await expect(imageCards(page).first().locator('.media-info h3')).toBeVisible();
await openAppearance(page);
await toggle(page, { 'card-details': false, 'viewer-details': true });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
await openImages(page);
// Show image details on cards off: no title or file line; Info and Tag stay.
const card = imageCards(page).first();
await expect(card.locator('.media-info h3')).toBeHidden();
await expect(card.locator('.image-card-actions button')).toHaveCount(2);
// Collapse viewer details by default: the viewer opens with its details hidden behind one toggle.
await card.locator('.image-open').click();
const details = page.locator('#viewer-details');
await expect(page.locator('#viewer-dialog')).toBeVisible();
await expect(details).toBeHidden();
const show = page.locator('#viewer-toolbar').getByRole('button', { name: 'Show details', exact: true });
await expect(show).toHaveAttribute('aria-expanded', 'false');
await show.click();
await expect(details).toBeVisible();
await expect(page.locator('#viewer-toolbar').getByRole('button', { name: 'Hide details', exact: true })).toHaveAttribute('aria-expanded', 'true');
});
test('reset restores defaults without touching other keys', async ({ page }) => {
await page.goto(`${workspace}#/system/appearance`);
await page.evaluate(() => localStorage.setItem('gallerySettings', JSON.stringify({ gridColumns: 5, defaultLimit: 200,
thumbnailPromptDismissed: 'kept', workspaceLayout: 'list', videoMute: false, showImageInfo: false })));
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#default-view')).toHaveValue('list');
const reset = page.locator('#workspace-main .operation').filter({ hasText: 'Reset display defaults' }).getByRole('button', { name: 'Reset', exact: true });
await reset.click();
await expect(page.locator('#default-view')).toHaveValue('grid');
await expect(page.locator('#video-muted')).toBeChecked();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect(stored).toEqual({ gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'kept', workspaceLayout: 'grid',
videoMute: true, showImageInfo: true });
});
+3
View File
@@ -340,6 +340,9 @@ test('Tag opens the dialog without selecting the card', async ({ page }) => {
const panel = page.getByRole('complementary', { name: 'Selected image details', exact: true });
await page.goto(images);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
// Capture the panel once it shows the first card's details, not while it is still reading them.
await expect(panel.getByRole('heading', { level: 2 })).toBeVisible();
await expect(panel).not.toContainText('Loading file details');
const before = await panel.innerText();
await openTag(page, SECOND, { goto: false });
await page.keyboard.press('Escape');
+98 -1
View File
@@ -18,7 +18,8 @@ const routes = [
['tags', 'Tags & categories', null],
// #249 local Inspect: the native Inspect a file page; Overview's tool card still links the legacy metadata page.
['inspect', 'Inspect a file', null],
['system', 'System', '/prompt_manager/admin'],
// #249 System S2: native System; its undelivered sections hand off per section (D94, asserted below).
['system', 'System', null],
// #249 D51: SAVED VIEWS' Favorites, Images filtered to favorites.
['favorites', 'Favorites', null],
// #249 Activity-1: the concept's Activity page in the tools-and-system group.
@@ -409,6 +410,8 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
await sidebarLink(page, 'System').click();
// System's Overview is native (S2 Task 6); a section not delivered yet still hands off (D94).
await main.locator('.section-nav').getByRole('button', { name: 'Integrations', exact: true }).click();
const handoff = main.locator('section.handoff');
await expect(handoff).toHaveClass(/\bsurface\b/);
expect(await handoff.evaluate(surfaceStyle)).toMatchObject(SURFACE);
@@ -417,6 +420,100 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
});
// #249 System S2 Task 5: the concept's renderSystem shell (app.js 698–706) at `#/system/<section>`, every section a D94
// hand-off until its PR, and the Save row's truthful status (D92).
const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance',
'Backup & export', 'Diagnostics & logs'];
const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics'];
const SYSTEM_HANDOFFS = { maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => {
const mutations = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) mutations.push(item.url()); });
await page.goto(`${workspace}#/system`);
const main = page.locator('#workspace-main');
await expect(main.locator('h1')).toHaveText('System');
await expect(main.locator('.heading p.muted')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.');
// D92: no "Sample configuration" chip.
await expect(main.locator('.heading .chip')).toHaveCount(0);
expect(await main.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual(['div.heading', 'div.section-nav', 'div.settings-layout']);
const nav = main.getByRole('group', { name: 'System sections', exact: true });
await expect(nav.getByRole('button')).toHaveText(SYSTEM_SECTION_LABELS);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page');
const layout = main.locator('.settings-layout');
await expect(layout.locator(':scope > .save-row > span.small#settings-status')).toHaveText('No unsaved changes.');
const save = layout.locator(':scope > .save-row > button.ca-primary');
await expect(save).toHaveText('Save preferences');
await expect(save).toHaveAttribute('aria-disabled', 'true');
// Playwright waits on an aria-disabled control, so the ignored press is forced.
await save.click({ force: true });
await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.');
for (const [index, label] of SYSTEM_SECTION_LABELS.entries()) {
const hash = SYSTEM_HASHES[index];
const button = nav.getByRole('button', { name: label, exact: true });
await button.click();
await expect(page).toHaveURL(`${workspace}#/system/${hash}`);
await expect(button).toHaveAttribute('aria-pressed', 'true');
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
// The section changes in place: focus stays on the pressed section, not the page title.
await expect(button).toBeFocused();
// S2 Task 6: Overview is native (workspace-system.spec.mjs), and so are Appearance (S3 Task 8) and Sources & scans
// (S4 Task 11, workspace-system-sources.spec.mjs, its update notice shown in a fresh browser); Maintenance shows
// its health-check notice above the hand-off (D123); every other section hands off.
if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible();
if (hash === 'sources') await expect(layout.locator('#source-list .operation')).toHaveCount(2);
expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'div.save-row'],
sources: ['div.notice', 'section.surface', 'div.save-row'],
maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash]
?? ['section.surface handoff', 'div.save-row']);
if (['overview', 'appearance', 'sources'].includes(hash)) continue;
const handoff = layout.locator('section.handoff');
await expect(handoff.locator('h2')).toHaveText('Your current tools are ready');
expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href'))))
.toEqual(SYSTEM_HANDOFFS[hash] ?? ['/prompt_manager/admin']);
await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.');
await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page');
}
// Deep links, reload and history keep the section; an unknown section is Overview under its own hash.
await page.reload();
await expect(nav.getByRole('button', { name: 'Diagnostics & logs', exact: true })).toHaveAttribute('aria-pressed', 'true');
await page.goBack();
await expect(page).toHaveURL(`${workspace}#/system/backups`);
await expect(nav.getByRole('button', { name: 'Backup & export', exact: true })).toHaveAttribute('aria-pressed', 'true');
await page.goto(`${workspace}#/system/general`);
await expect(page).toHaveURL(`${workspace}#/system/overview`);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
// The sidebar's System returns to the Overview section.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await sidebarLink(page, 'System').click();
await expect(page).toHaveURL(`${workspace}#/system`);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
// Leaving System and coming back renders it once, with a fresh section.
await sidebarLink(page, 'Images').click();
await expect(main.locator('h1')).toHaveText('Images');
await expect(main.locator('.section-nav')).toHaveCount(0);
await page.goBack();
await expect(main.locator('.section-nav')).toHaveCount(1);
await expect(main.locator('.settings-layout')).toHaveCount(1);
expect(mutations).toEqual([]);
});
test('Overview opens native System from its System card (Q12)', async ({ page }) => {
await page.goto(`${workspace}#/overview`);
const card = page.locator('#workspace-main .tool-card').filter({ has: page.getByRole('heading', { name: 'System', exact: true }) });
await expect(card.locator('p.small')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.');
const link = card.getByRole('link', { name: 'Open System', exact: true });
await expect(link).toHaveAttribute('href', '#/system/overview');
await link.click();
await expect(page.locator('#workspace-main h1')).toHaveText('System');
await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText('Overview');
});
// Task A3: the concept sidebar (index.html h16-41, app.js nav()) rendered from real data (D27, D28).
const NAV_TOTALS = [
['images', 'Images', '/prompt_manager/images/output?mode=workspace&limit=1'],
@@ -176,6 +176,8 @@ _TABLES = (
"execution_context_texts",
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -28,6 +28,8 @@ EDITOR_METADATA = {
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+206
View File
@@ -0,0 +1,206 @@
"""System test hooks for the browser tests (#249 System S2 Task 6).
The data is the concept-shaped real fixture (``POST /__test__/concept/reset``,
``workspace_concept_fixture.py``): its parents, saved pairs, images, ratings
and runs are what the Overview, the Statistics dialog and the health check
read. These per-server hooks only change what a System test needs on top:
- ``POST /__test__/system/reset`` restores the configured sources, forgets the
session's health report and ends every injected failure;
- ``POST /__test__/system/fail`` ``{path, times}`` answers the next ``times``
requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes'
500 error envelope;
- ``POST /__test__/system/missing-source`` configures one more source that
does not exist (an unmounted drive); with ``{"inside": true}`` it is an
existing folder inside the output folder (``output/unplugged``), which
``POST /__test__/system/unplug`` then removes (a drive unplugged while the
page shows it);
- ``POST /__test__/system/missing-file`` deletes the newest linked image's
file from disk, leaving its database row (a missing-file link);
- ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain
PNGs with no thumbnail into the output folder;
- ``POST /__test__/system/monitor`` ``{watching, alive}`` stands in a fake
image monitor whose observer is ``alive`` (default true) and schedules the
named folders of this server's root (``["output"]``), so Sources reads
watching, manual scan and restart pending states (S4).
Delays use the shared request holds (``request_hold.py``).
"""
import os
from pathlib import Path
from aiohttp import web
from PIL import Image
FAILABLE_PREFIX = "/prompt_manager/system/"
MAX_UNTHUMBNAILED = 50
def register_system_commands(app, routes, api, root: Path):
# System › Appearance saves the host fields to this server's config.json (S3);
# reset restores both. The file is this server's own, inside its temp root:
# reset rewrites or deletes it, so a server whose configuration points
# anywhere else (a real config.json) refuses to register these hooks.
config_path = root / "config.json"
configured_path = os.environ.get("PROMPT_MANAGER_CONFIG_PATH")
if (
configured_path is None
or Path(configured_path).resolve() != config_path.resolve()
):
raise RuntimeError(
f"PROMPT_MANAGER_CONFIG_PATH must be {config_path}, not {configured_path!r}"
)
from py.config import GalleryConfig, PromptManagerConfig
from utils import image_monitor
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
config_bytes = config_path.read_bytes() if config_path.exists() else None
host_values = (
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
)
monitor = getattr(image_monitor, "_monitor_instance", None)
failures: dict[str, int] = {}
@web.middleware
async def inject_failures(request, handler):
left = failures.get(request.path, 0)
if left <= 0:
return await handler(request)
failures[request.path] = left - 1
return web.json_response(
{"success": False, "code": None, "error": "Injected failure"}, status=500
)
app.middlewares.append(inject_failures)
unplugged = root / "output" / "unplugged"
def reset():
failures.clear()
if unplugged.is_dir():
unplugged.rmdir()
GalleryConfig.MONITORING_DIRECTORIES = list(configured)
api._health_report = None
(
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
) = host_values
image_monitor._monitor_instance = monitor
if config_bytes is None:
config_path.unlink(missing_ok=True)
else:
# Replace in one step, as the server writes it (Windows preflight).
staged = config_path.with_name(config_path.name + ".reset")
staged.write_bytes(config_bytes)
os.replace(staged, config_path)
@routes.post("/__test__/system/reset")
async def system_reset(request):
reset()
return web.json_response({"success": True})
@routes.post("/__test__/system/fail")
async def system_fail(request):
data = await request.json()
path, times = data.get("path"), data.get("times", 1)
if (
not isinstance(path, str)
or not path.startswith(FAILABLE_PREFIX)
or isinstance(times, bool)
or not isinstance(times, int)
or not 1 <= times <= 10
):
raise web.HTTPBadRequest()
failures[path] = times
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-source")
async def missing_source(request):
data = await request.json() if request.can_read_body else {}
if data.get("inside") is True:
unplugged.mkdir(exist_ok=True)
absent = str(unplugged)
else:
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent]
return web.json_response(
{"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)}
)
@routes.post("/__test__/system/unplug")
async def unplug(request):
if not unplugged.is_dir():
raise web.HTTPConflict()
unplugged.rmdir()
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-file")
async def missing_file(request):
with api.db.model.get_connection() as conn:
row = conn.execute(
"SELECT file_path FROM generated_images ORDER BY id DESC LIMIT 1"
).fetchone()
if row is None:
raise web.HTTPNotFound()
Path(row[0]).unlink()
api.invalidate_gallery_cache()
return web.json_response({"success": True, "filename": Path(row[0]).name})
@routes.post("/__test__/system/unthumbnailed")
async def unthumbnailed(request):
data = await request.json()
count = data.get("count")
if (
isinstance(count, bool)
or not isinstance(count, int)
or not 1 <= count <= MAX_UNTHUMBNAILED
):
raise web.HTTPBadRequest()
folder = root / "output" / "unthumbnailed"
folder.mkdir(exist_ok=True)
for number in range(count):
with Image.new("RGB", (64, 64), "#2b3442") as image:
image.save(folder / f"plain_{number + 1:03}.png")
api.invalidate_gallery_cache()
return web.json_response({"success": True})
@routes.post("/__test__/system/monitor")
async def fake_monitor(request):
data = await request.json()
watching, alive = data.get("watching", []), data.get("alive", True)
if (
not isinstance(watching, list)
or not all(name in ("output", "input") for name in watching)
or not isinstance(alive, bool)
):
raise web.HTTPBadRequest()
image_monitor._monitor_instance = _FakeMonitor(
[str(root / name) for name in watching], alive
)
return web.json_response({"success": True})
return reset
class _FakeMonitor:
"""The image monitor's status surface (``get_status``) with no observer thread."""
def __init__(self, directories, alive):
self.monitored_directories = directories
self._alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": list(self.monitored_directories),
"handler_active": True,
"observer_alive": self._alive,
}
@@ -44,6 +44,8 @@ MAX_EXTRA_IMAGES = 160
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -114,6 +114,8 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -81,6 +81,8 @@ MAX_EXTRA_CATEGORIES = 200
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+10 -1
View File
@@ -1,11 +1,20 @@
import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
import runtime from '../../tools/vendor/comfyui-runtime.json' with { type: 'json' };
import { hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
export function installHostHooks(test, expectedTypes = ['PromptManager', 'PromptManagerText', 'PromptSearchList']) {
test.beforeEach(async ({ page, context, request }, testInfo) => {
const response = await request.get('/system_stats');
expect(response.ok()).toBe(true);
const info = (await response.json()).system;
const stats = await response.json();
// comfyui.config.mjs's launcher reports the host it owns. The execution suite has no
// web server: it starts its own host per worker behind a port refusal.
if (testInfo.config.webServer) {
const ownedBase = process.env.PM_COMFY_HOST_BASE;
expect(ownedBase, 'the launcher reported the host it started').toBeTruthy();
expect(hostBaseDirectory(stats), 'the host answering the port is the one this run started').toBe(ownedBase);
}
const info = stats.system;
expect(info.comfyui_version).toBe(runtime.serverVersion);
expect(info.pytorch_version).toBe(`${runtime.torchVersion}+cpu`);
expect(info.required_frontend_version).toBe(runtime.frontendVersion);
+4 -4
View File
@@ -594,12 +594,12 @@
"anchor": null,
"expect": {
"metadata_status": "available",
"state": "ambiguous",
"reason": "Multiple sampler branches",
"state": "resolved",
"reason": "",
"reason_code": null,
"selection": null,
"positive": null,
"negative": null,
"positive": "a red fox in snow, detailed fur",
"negative": "blurry, lowres",
"candidate_count": 0,
"facts": {
"seed": {
File diff suppressed because one or more lines are too long
+17
View File
@@ -355,3 +355,20 @@ test("keepalive is forwarded only when asked, so a request can outlive its page"
for (const call of fetch.calls) call.resolve(jsonResponse(200, { success: true }));
await Promise.all(writes);
});
test("ApiError keeps a validated running job on busy and a validated source on invalid-source", () => {
const body = { running: { id: "a".repeat(32), title: "Scan for metadata" }, source: "output/gone" };
const busy = new ApiError(409, "busy", "busy", null, body);
assert.deepEqual([busy.running, busy.source], [{ id: "a".repeat(32), title: "Scan for metadata" }, null]);
assert.notEqual(busy.running, body.running, "server objects must be copied");
const invalid = new ApiError(400, "refused", "invalid-source", null, body);
assert.deepEqual([invalid.running, invalid.source], [null, "output/gone"]);
for (const running of [null, "x", { id: 1, title: "t" }, { id: "a b", title: "t" }, { id: "a", title: "" },
{ id: "a", title: "x".repeat(201) }, { id: "a".repeat(65), title: "t" }]) {
assert.equal(new ApiError(409, "busy", "busy", null, { running }).running, null);
}
for (const source of [null, 7, "", "x".repeat(4097)]) {
assert.equal(new ApiError(400, "refused", "invalid-source", null, { source }).source, null);
}
assert.deepEqual([new ApiError(409, "old caller").running, new ApiError(409, "old caller").source], [null, null]);
});
+47
View File
@@ -61,3 +61,50 @@ test("infinite scrolling is off by default and must be a boolean", () => {
assert.equal(normalizeGallerySettings({ infiniteScroll: true }).infiniteScroll, true);
assert.equal(normalizeGallerySettings({ infiniteScroll: "yes" }).infiniteScroll, false);
});
test("the workspace keys have their own defaults and validation", () => {
const { WORKSPACE_LAYOUTS, WORKSPACE_DENSITIES } = require("../../web/js/gallery-settings.js");
assert.equal(DEFAULT_SETTINGS.workspaceLayout, "grid");
assert.equal(DEFAULT_SETTINGS.workspaceDensity, "comfortable");
assert.deepEqual(WORKSPACE_LAYOUTS, ["grid", "list", "masonry"]);
assert.deepEqual(WORKSPACE_DENSITIES, ["comfortable", "large", "compact"]);
assert.equal(normalizeGallerySettings({ workspaceLayout: "masonry" }).workspaceLayout, "masonry");
assert.equal(normalizeGallerySettings({ workspaceLayout: "mosaic" }).workspaceLayout, "grid");
assert.equal(normalizeGallerySettings({ workspaceDensity: "large" }).workspaceDensity, "large");
assert.equal(normalizeGallerySettings({ workspaceDensity: 3 }).workspaceDensity, "comfortable");
// The legacy gallery's own view mode is separate: it has no masonry.
assert.equal(normalizeGallerySettings({ defaultViewMode: "masonry" }).defaultViewMode, "grid");
});
test("a legacy gallery Save keeps the workspace keys and every other stored key", () => {
const fs = require("node:fs");
const path = require("node:path");
const vm = require("node:vm");
const settings = require("../../web/js/gallery-settings.js");
const storage = new Map([["gallerySettings", JSON.stringify({ workspaceLayout: "masonry", workspaceDensity: "compact", someOtherTool: 1 })]]);
const controls = {
lazyLoadingToggle: { checked: true }, imageQualitySelect: { value: "high" }, defaultViewModeSelect: { value: "list" },
defaultLimitSelect: { value: "200" }, gridColumnsSelect: { value: "6" }, showImageInfoToggle: { checked: false },
autoLoadMetadataToggle: { checked: true }, cacheMetadataToggle: { checked: true }, showFilePathsToggle: { checked: false },
sidebarCollapsedByDefaultToggle: { checked: true }, infiniteScrollToggle: { checked: false }, debugModeToggle: { checked: false },
apiTimeoutInput: { value: "30" }, checkThumbnailsAtStartup: { checked: true }, videoAutoplayToggle: { checked: true },
videoMuteToggle: { checked: false }, videoLoopToggle: { checked: true },
};
const context = vm.createContext({
document: { addEventListener() {}, getElementById: id => controls[id] },
window: {}, console: { debug() {}, warn() {}, error() {} }, GallerySettings: settings,
localStorage: { getItem: key => storage.get(key) ?? null, setItem: (key, value) => storage.set(key, value) },
});
const file = path.join(__dirname, "../../web/js/gallery.js");
const Manager = vm.runInContext(`${fs.readFileSync(file, "utf8")}\nGalleryManager;`, context, { filename: file });
const manager = Object.create(Manager.prototype);
Object.assign(manager, { showNotification() {}, hideSettings() {}, applySettings() {} });
manager.saveSettings();
const saved = JSON.parse(storage.get("gallerySettings"));
assert.equal(saved.workspaceLayout, "masonry");
assert.equal(saved.workspaceDensity, "compact");
assert.equal(saved.someOtherTool, 1);
assert.equal(saved.defaultViewMode, "list");
assert.equal(saved.showImageInfo, false);
assert.equal(saved.videoMute, false);
});
+26
View File
@@ -82,6 +82,32 @@ test('a zeroed conditioning negative shows as an explicit empty negative without
}finally{scope.dispose();f.restore();}
});
test('a prompt that left out run-time trigger words says so under its text (D125)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div'),sent=[];
const note='Text added at run time, such as trigger words, is not included.';
try{
module.mountPromptBlocks(container,{parentScope:scope,sender:{open:roles=>sent.push(roles)},prompts:{positive:{status:'resolved',text:'own text',source:'runtime-parts-omitted'},negative:{status:'resolved',text:'blur',source:'graph-inferred'}}});
const notes=f.walk(container,node=>node.textContent===note);
assert.equal(notes.length,1,'one note, under the positive only');
assert.equal(notes[0].parent.attributes['aria-label'],'Positive prompt');
container.click(f.walk(container,node=>node.dataset.sendRole==='pair')[0]);
assert.deepEqual(sent,[{positive:'own text',negative:'blur'}],'the note is never sent');
}finally{scope.dispose();f.restore();}
});
test('a prompt a later stage added conditioning to says so, and both lines show together (D126)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div');
const added='Conditioning added at a later stage is not included.', runtime='Text added at run time, such as trigger words, is not included.';
try{
const view=module.mountPromptBlocks(container,{parentScope:scope,prompts:{positive:{status:'resolved',text:'base',source:'added-conditioning'}}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[added]);
view.update({positive:{status:'resolved',text:'base',source:'runtime-parts-omitted-and-added-conditioning'}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[runtime,added]);
view.update({positive:{status:'resolved',text:'base',source:'graph-inferred'}});
assert.equal(f.walk(container,node=>node.className==='small muted').length,0);
}finally{scope.dispose();f.restore();}
});
const unresolved=message=>({status:'unresolved',text:null,source:null,...(message===undefined?{}:{message})});
const statusLines=(f,container)=>f.walk(container,node=>node.tagName==='p'&&node.className==='prompt-text muted').map(node=>node.textContent);
test('unsupported node status names the class',()=>{
+18
View File
@@ -182,6 +182,24 @@ test('start returns busy with the running job', async () => {
world.dispose();
});
test('track follows a job another route started, with no read of its own', async () => {
const { createActivityStore } = await source;
const { world, ...deps } = harness();
const store = createActivityStore(deps);
const seen = [];
store.subscribe(() => seen.push(store.find(ID_A)?.state));
const job = store.track(summary({ kind: 'health', title: 'Health check' }));
assert.equal(job.id, ID_A);
assert.equal(world.gets, 0, 'the started job is known without a list read');
assert.deepEqual(world.open().map(stream => stream.url), [`/prompt_manager/jobs/${ID_A}/events`]);
assert.deepEqual(world.toasts, [], 'the route that started it says so itself');
send(world.open()[0], summary({ kind: 'health', title: 'Health check', state: 'finished', progress: 100 }));
assert.deepEqual(seen, ['running', 'finished']);
assert.equal(world.open().length, 0, 'an ended job is no longer followed');
assert.throws(() => store.track({ id: 'nope' }), /Invalid job/);
world.dispose();
});
test('a list that is not a job list counts as a failed read', async () => {
const { createActivityStore } = await source;
const { world, api, ...deps } = harness();
+16
View File
@@ -478,3 +478,19 @@ test('a refresh deferred behind a page load is dropped when the stream is dispos
await new Promise(resolve=>setImmediate(resolve));
assert.equal(requests.length,2,'no refresh runs after dispose');
});
test('defaults apply to a new visit but not to the current view or a saved view', async () => {
const { initialImagesView } = await source;
assert.equal(typeof initialImagesView, 'function', 'initialImagesView is exported');
const preferences = { workspaceLayout: 'masonry', workspaceDensity: 'compact' };
// A new visit starts from System › Appearance.
assert.deepEqual(initialImagesView({ preferences }), { layout: 'masonry', density: 'compact' });
// The current view (the last visit) keeps its own toolbar choices.
assert.deepEqual(initialImagesView({ prior: { layout: 'list', density: 'large' }, preferences }), { layout: 'list', density: 'large' });
assert.deepEqual(initialImagesView({ prior: {}, preferences }), { layout: 'grid', density: 'comfortable' });
// A saved view opens as it always did.
assert.deepEqual(initialImagesView({ saved: { q: '' }, preferences }), { layout: 'grid', density: 'comfortable' });
// Without readable preferences nothing changes.
assert.deepEqual(initialImagesView({}), { layout: 'grid', density: 'comfortable' });
assert.deepEqual(initialImagesView({ preferences: {} }), { layout: 'grid', density: 'comfortable' });
});
@@ -825,3 +825,73 @@ test('refresh re-reads the selected file and keeps the shown record until the ne
assert.equal(controller.state.record.identity.path, 'folder/b.png');
controller.dispose(); assert.equal(await controller.refresh(), undefined);
});
test('metadata off shows an explicit load and loads once on request', async () => {
const { api, requests } = pendingApi(), updates = [];
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
controller.subscribe(state => updates.push(state));
// Selecting reads nothing: the state says the metadata waits for a request.
assert.equal(controller.select(image()), undefined);
assert.equal(requests.length, 0);
assert.equal(controller.state.deferred, true);
assert.equal(controller.state.loading, false);
assert.equal(controller.state.record, null);
// Selecting the same file again still reads nothing.
controller.select({ ...image() });
assert.equal(requests.length, 0);
const load = controller.load();
assert.equal(requests.length, 1);
assert.equal(controller.state.loading, true);
assert.equal(controller.state.deferred, false);
// A second press while it reads, and the same file selected again once it loaded, read nothing more.
controller.load();
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await load;
assert.equal(controller.state.record.generation.seed.value, '9007199254740993');
controller.load(); controller.select(image());
assert.equal(requests.length, 1);
// Another file waits for its own request again.
controller.select({ ...image(), path: 'folder/b.png', filename: 'b.png', url: image().url.replace('a%20%23%25', 'b') });
assert.equal(controller.state.deferred, true);
assert.equal(requests.length, 1);
// The default controller loads on selection, as before.
const eager = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts });
eager.select(image());
assert.equal(requests.length, 2);
assert.equal(eager.state.deferred, false);
eager.dispose(); controller.dispose();
assert.equal(controller.load(), undefined);
});
test('cache off rereads the current selection; on reuses the one current record', async () => {
for (const cache of [true, false]) {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache });
const first = controller.select(image());
// A read in flight is shared either way.
controller.select(image());
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await first;
const again = controller.select(image());
assert.equal(requests.length, cache ? 1 : 2, `cache ${cache}`);
if (!cache) { requests[1].resolve(envelope()); await again; assert.equal(controller.state.record.identity.path, 'folder/a #%.png'); }
controller.dispose();
}
});
test('cache off never rereads a selection whose source changed, and refresh never loads a deferred one', async () => {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache: false });
const first = controller.select(image());
requests[0].reject(new ApiError(409, refreshMessage, 'source-changed')); await first;
assert.equal(controller.state.refreshRequired, true);
controller.select(image());
assert.equal(requests.length, 1);
controller.dispose();
const deferred = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
deferred.select(image());
assert.equal(deferred.refresh(), undefined);
assert.equal(requests.length, 1);
assert.equal(deferred.state.deferred, true);
deferred.dispose();
});
+11
View File
@@ -136,6 +136,17 @@ test('an unusable semantics table is refused', () => {
for (const doc of [null, {schema: 2}, {schema: 1, samplers: []}]) assert.throws(() => resolver.compileSemantics(doc), resolver.SemanticsError);
});
test('the lineage section is compiled, and a bad one refuses the table', () => {
const doc = JSON.parse(fs.readFileSync(new URL('../../web/js/inspection-semantics.json', import.meta.url), 'utf8'));
assert.deepEqual(semantics.lineage.get('Image Filter\u00001'), { strategy: 'continue', input: 'latents' });
// Lineage describes no prompt: its classes are not table classes for the stage walk.
assert.equal(semantics.allClasses.has('VAEEncode'), false);
const row = { class: 'VAEEncode', slot: 0, strategy: 'continue', input: 'pixels' };
for (const lineage of [undefined, {}, [null], [{ ...row, slot: -1 }], [{ ...row, strategy: 'passthrough' }], [{ ...row, input: '' }], [row, row]]) {
assert.throws(() => resolver.compileSemantics({ ...doc, lineage }), resolver.SemanticsError, JSON.stringify(lineage));
}
});
// The committed server answers (tests/fixtures/local-inspect/resolver-vectors.json, written by
// `python3 -m tests.local_resolver_cases --write` and checked current by the Python suite).
test('committed server answers equal the local evidence', () => {
+40 -7
View File
@@ -124,18 +124,51 @@ test('real inspection navigation preserves slideshow across two advances and def
} finally {view.dispose();controller.dispose();parentScope.dispose();globalThis.setTimeout=oldSetTimeout;globalThis.clearTimeout=oldClearTimeout;f.restore();}
});
test('an inline Viewer still loading its image is destroyed only after it has viewed it', async () => {
test('Viewer.js is built only on a loaded image and an unbuilt viewer is released at once', async () => {
const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs');
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c')];
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c'),occurrenceMedia('2',0,'d')];
const instances = [];
// Viewer.js's inline destroy() leaves an in-flight view running; its callback then needs a live instance.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.viewed = false; instances.push(this); }
// On a loaded image Viewer.js schedules its inline build; until it runs `ready` is false and destroy() cancels it.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = false; this.viewed = false; instances.push(this); }
destroy() { this.destroyed++; } resize() {} };
const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame};
Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}});
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}});
try {
view.open({items,selectedKey:'c'.repeat(64)});
const stageImage = () => f.walk(f.container,n => n.tagName === 'img')[0];
view.open({items,selectedKey:'d'.repeat(64)});
// Viewer.js's own cancel misses an image whose load event is still queued, so none exists before the load.
const unloaded = stageImage(), dropped = [];unloaded.removeAttribute = key => dropped.push(key);
assert.equal(instances.length, 0, 'no Viewer.js before the image loads');
f.walk(f.container,n => n.attributes['aria-label'] === 'Previous media')[0].fire('click');
assert.deepEqual(dropped, ['src'], 'an image left before it loads is released at once');
unloaded.fire('load');
assert.equal(instances.length, 0, 'its late load builds no Viewer.js');
stageImage().fire('load');
const [loading] = instances, image = loading.node, removed = [];
image.removeAttribute = key => removed.push(key);
// Next drops the image whose build is pending; a deferred destroy would let that build run there.
f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click');
stageImage().fire('load');assert.equal(instances.length, 2, 'the next image builds its own Viewer.js once loaded');
assert.equal(loading.destroyed, 1, 'an unbuilt viewer is destroyed at once');
assert.deepEqual(removed, ['src'], 'and its image is released at once');
loading.options.viewed.call(loading);
assert.equal(loading.destroyed, 1, 'never destroyed twice');
} finally { view.dispose(); parentScope.dispose(); f.restore(); Object.assign(globalThis, globals); }
});
test('an inline Viewer still loading its image is destroyed only after it has viewed it', async () => {
const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs');
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c')];
const instances = [];
// A built (`ready`) inline Viewer.js's destroy() leaves an in-flight view running; its callback then needs a live instance.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = true; this.viewed = false; instances.push(this); }
destroy() { this.destroyed++; } resize() {} };
const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame};
Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}});
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}});
try {
view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load');
const [loading] = instances, image = loading.node, removed = [];
image.removeAttribute = key => removed.push(key);
f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click');
@@ -148,7 +181,7 @@ test('an inline Viewer still loading its image is destroyed only after it has vi
loading.options.viewed.call(loading);
assert.equal(loading.destroyed, 1, 'never destroyed twice');
view.open({items,selectedKey:'c'.repeat(64)});
view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load');
const shown = instances[1]; shown.viewed = true; shown.options.viewed.call(shown);
f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click');
assert.equal(shown.destroyed, 1, 'a viewed image is destroyed at once');
@@ -236,7 +269,7 @@ test('viewer uses adapter.mediaUrl for image and video', async () => {
const img = f.walk(f.container,n => n.tagName === 'img')[0];
assert.equal(img.src, image.url, 'the blob URL is used exactly');
assert.doesNotMatch(img.src, /\?/);
assert.equal(instances[0].node, img, 'Viewer.js shows the blob image');
img.fire('load');assert.equal(instances[0].node, img, 'Viewer.js shows the blob image');
const download = f.walk(f.container,n => n.attributes['aria-label'] === 'Download original')[0];
assert.equal(download.href, image.url);assert.equal(download.download, 'a.png');
f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click');
+193
View File
@@ -0,0 +1,193 @@
// Browser display defaults (#249 System S3 Task 7): the workspace reads and writes the legacy gallery's own
// `gallerySettings` object (Q3, "One display configuration"), writes only the keys it changed, never consumes the
// legacy paging keys, and says when this browser refused to keep a change.
import test from 'node:test';
import assert from 'node:assert/strict';
import gallerySettings from '../../web/js/gallery-settings.js';
const module = await import('../../web/js/workspace/preferences.mjs').catch(error => {
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
return {};
});
globalThis.GallerySettings = gallerySettings;
const KEY = 'gallerySettings';
/** A localStorage stand-in holding `text` under the settings key; `setItem` can be made to throw. */
function storage(text = null, { quota = false } = {}) {
const writes = [];
const store = {
text,
writes,
getItem(key) { assert.equal(key, KEY); return store.text; },
setItem(key, value) {
assert.equal(key, KEY);
writes.push(value);
if (quota) { const error = new Error('Quota exceeded'); error.name = 'QuotaExceededError'; throw error; }
store.text = value;
},
removeItem() { assert.fail('Preferences never remove the shared settings'); },
};
return store;
}
const denied = () => { const error = new Error('Denied'); error.name = 'SecurityError'; throw error; };
test('normalizes new workspace keys and keeps legacy keys', () => {
assert.equal(typeof module.readPreferences, 'function', 'preferences.mjs exports readPreferences');
module.resetTabPreferences();
const legacy = { defaultViewMode: 'list', gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'sig', extra: 1 };
const store = storage(JSON.stringify({ ...legacy, workspaceLayout: 'masonry', workspaceDensity: 'compact', showFilePaths: false }));
const { values, persisted } = module.readPreferences(() => store);
assert.equal(persisted, true);
assert.equal(values.workspaceLayout, 'masonry');
assert.equal(values.workspaceDensity, 'compact');
assert.equal(values.showFilePaths, false);
// Invalid workspace values fall back to the defaults, as every legacy key does.
store.text = JSON.stringify({ workspaceLayout: 'mosaic', workspaceDensity: 7, videoMute: 'no' });
const fallback = module.readPreferences(() => store).values;
assert.equal(fallback.workspaceLayout, 'grid');
assert.equal(fallback.workspaceDensity, 'comfortable');
assert.equal(fallback.videoMute, true);
// The legacy normalizer knows the new keys and still keeps unknown ones.
const normalized = gallerySettings.normalizeGallerySettings({ ...legacy, workspaceLayout: 'list' });
assert.equal(normalized.workspaceLayout, 'list');
assert.equal(normalized.workspaceDensity, 'comfortable');
assert.equal(normalized.extra, 1);
assert.equal(normalized.thumbnailPromptDismissed, 'sig');
assert.deepEqual(gallerySettings.WORKSPACE_LAYOUTS, ['grid', 'list', 'masonry']);
assert.deepEqual(gallerySettings.WORKSPACE_DENSITIES, ['comfortable', 'large', 'compact']);
});
test('write touches only dirty keys', () => {
module.resetTabPreferences();
const stored = { defaultViewMode: 'list', gridColumns: 5, thumbnailsGenerated: true, extra: { kept: true }, videoMute: false };
const store = storage(JSON.stringify(stored));
const result = module.writePreferences({ workspaceLayout: 'masonry', showImageInfo: false }, () => store);
assert.deepEqual(result, { persisted: true, conflicts: [] });
assert.equal(store.writes.length, 1);
// Every other key, legacy or unknown, is written back exactly as it was stored: no defaults are filled in.
assert.deepEqual(JSON.parse(store.text), { ...stored, workspaceLayout: 'masonry', showImageInfo: false });
// It re-reads the stored object, so a key another page wrote after this one read is kept.
store.text = JSON.stringify({ ...JSON.parse(store.text), gridColumns: 9 });
module.writePreferences({ videoLoop: false }, () => store);
assert.equal(JSON.parse(store.text).gridColumns, 9);
assert.equal(JSON.parse(store.text).workspaceLayout, 'masonry');
// An empty or malformed stored value starts from an empty object, never the defaults.
for (const text of [null, '{broken', '[]', '42']) {
const fresh = storage(text);
assert.equal(module.writePreferences({ cacheMetadata: false }, () => fresh).persisted, true);
assert.deepEqual(JSON.parse(fresh.text), { cacheMetadata: false });
}
// Only workspace keys with valid values are accepted; nothing is written for a bad change.
for (const bad of [{ gridColumns: 4 }, { defaultLimit: 50 }, { workspaceLayout: 'mosaic' }, { videoMute: 'yes' }, { imageQuality: 'ultra' }]) {
const untouched = storage(JSON.stringify(stored));
assert.throws(() => module.writePreferences(bad, () => untouched), TypeError);
assert.equal(untouched.writes.length, 0);
}
// A key changed elsewhere since this tab read it (its baseline) is reported, and the explicit save still wins.
const elsewhere = storage(JSON.stringify({ videoLoop: false }));
const outcome = module.writePreferences({ videoLoop: true, videoMute: false }, () => elsewhere, { baseline: { videoLoop: true, videoMute: true } });
assert.deepEqual(outcome, { persisted: true, conflicts: ['videoLoop'] });
assert.equal(JSON.parse(elsewhere.text).videoLoop, true);
});
test('denied storage reports tab-only', () => {
module.resetTabPreferences();
const read = module.readPreferences(denied);
assert.equal(read.persisted, false);
assert.equal(read.values.workspaceLayout, 'grid');
const result = module.writePreferences({ workspaceLayout: 'list', videoAutoplay: true }, denied);
assert.deepEqual(result, { persisted: false, conflicts: [] });
// The change is kept for this tab: every reader in it (a new Images visit, the viewer) sees it.
const tab = module.readPreferences(denied);
assert.equal(tab.persisted, false);
assert.equal(tab.values.workspaceLayout, 'list');
assert.equal(tab.values.videoAutoplay, true);
// A getItem that throws is denied storage too.
const throwing = { getItem: denied, setItem: denied };
assert.equal(module.readPreferences(() => throwing).values.workspaceLayout, 'list');
assert.equal(module.writePreferences({ workspaceDensity: 'large' }, () => throwing).persisted, false);
// Once storage accepts a write, the tab copy of that key gives way to the stored value.
const working = storage(JSON.stringify({ workspaceLayout: 'masonry' }));
assert.equal(module.writePreferences({ workspaceLayout: 'masonry' }, () => working).persisted, true);
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
assert.equal(module.readPreferences(denied).values.videoAutoplay, false);
});
test('quota error keeps previous bytes', () => {
module.resetTabPreferences();
const before = JSON.stringify({ gridColumns: 6, workspaceDensity: 'large' });
const store = storage(before, { quota: true });
const result = module.writePreferences({ workspaceDensity: 'compact' }, () => store);
assert.deepEqual(result, { persisted: false, conflicts: [] });
assert.equal(store.writes.length, 1);
assert.equal(store.text, before);
// This tab keeps the change it could not store.
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'compact');
module.resetTabPreferences();
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'large');
});
/** A window stand-in that records its storage listeners. */
function fakeWindow() {
const listeners = new Set();
return {
listeners,
addEventListener(type, listener) { assert.equal(type, 'storage'); listeners.add(listener); },
removeEventListener(type, listener) { assert.equal(type, 'storage'); listeners.delete(listener); },
fire(event) { for (const listener of [...listeners]) listener(event); },
};
}
test('storage event updates clean values and flags dirty conflicts', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ workspaceLayout: 'list', videoMute: false }));
const win = fakeWindow();
const seen = [];
const unsubscribe = module.subscribePreferences(win, next => seen.push(next), () => store);
assert.equal(win.listeners.size, 1);
// Another key, or another storage area's event, is not ours.
win.fire({ key: 'pm_update_notice_dismissed', storageArea: store });
win.fire({ key: KEY, storageArea: { other: true } });
assert.equal(seen.length, 0);
win.fire({ key: KEY, storageArea: store });
win.fire({ key: null, storageArea: store }); // localStorage.clear() elsewhere
assert.equal(seen.length, 2);
assert.equal(seen[0].values.workspaceLayout, 'list');
unsubscribe();
assert.equal(win.listeners.size, 0);
// The reconciliation of the System drafts: clean values follow the other page; a key this tab has a draft for and
// the other page changed is a conflict, and the draft stays.
const baseline = { workspaceLayout: 'grid', videoMute: true, videoLoop: true };
const next = { workspaceLayout: 'list', videoMute: true, videoLoop: false };
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { videoMute: false, videoLoop: false }, next }), { conflicts: [] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'masonry' }, next }), { conflicts: ['workspaceLayout'] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'list' }, next }), { conflicts: [] });
});
test('legacy paging keys are never consumed by the workspace', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ defaultLimit: 500, gridColumns: 12, infiniteScroll: true, lazyLoading: false,
defaultViewMode: 'list', debugMode: true, apiTimeout: 9, thumbnailsGenerated: true, checkThumbnailsAtStartup: false }));
const { values } = module.readPreferences(() => store);
assert.deepEqual(Object.keys(values).sort(), [...module.WORKSPACE_PREFERENCE_KEYS].sort());
for (const key of ['defaultLimit', 'gridColumns', 'infiniteScroll', 'lazyLoading', 'defaultViewMode', 'debugMode', 'apiTimeout',
'thumbnailsGenerated', 'checkThumbnailsAtStartup', 'thumbnailPromptDismissed']) assert.equal(Object.hasOwn(values, key), false, key);
// The display defaults are the gallery's own defaults for these keys.
const defaults = module.preferenceDefaults();
for (const key of module.WORKSPACE_PREFERENCE_KEYS) assert.equal(defaults[key], gallerySettings.DEFAULT_SETTINGS[key], key);
});
test('a later save flushes the changes kept for this tab only', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ gridColumns: 4 }), { quota: true });
assert.equal(module.writePreferences({ workspaceLayout: 'list' }, () => store).persisted, false);
// Storage accepts writes again: saving another key stores the tab-only change with it, so "Saved" is true for both.
const working = storage(store.text);
assert.deepEqual(module.writePreferences({ videoLoop: false }, () => working), { persisted: true, conflicts: [] });
assert.deepEqual(JSON.parse(working.text), { gridColumns: 4, workspaceLayout: 'list', videoLoop: false });
// Nothing is overlaid any more: another page's later change reads through.
working.text = JSON.stringify({ ...JSON.parse(working.text), workspaceLayout: 'masonry' });
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
});
+44 -2
View File
@@ -23,8 +23,50 @@ test('tags is a native route titled Tags; only unbuilt routes hand off to legacy
assert.equal(routeTitle(route), route.label);
}
assert.equal(routeTitle(resolveRoute('#/library/7')), 'Prompt library');
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
for (const id of ['system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), true, id);
// System is native from S2 (#249 System): its undelivered sections hand off per section (D94), not the route.
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
});
test('system resolves its sections; #/overview stays the production hub', async () => {
const { resolveRoute, routeTitle, isHandoff, routes } = await source;
// `#/system` is the Overview section and keeps the sidebar's hash.
const system = resolveRoute('#/system');
assert.deepEqual([system.id, system.section, system.hash, routeTitle(system), isHandoff(system)],
['system', 'overview', '#/system', 'System', false]);
for (const section of ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics']) {
const route = resolveRoute(`#/system/${section}`);
assert.deepEqual([route.id, route.section, route.hash], ['system', section, `#/system/${section}`]);
}
// An unknown child is the Overview section with a replaced hash.
for (const hash of ['#/system/general', '#/system/', '#/system/__proto__', '#/system/backups/1', '#/system/<b>']) {
const route = resolveRoute(hash);
assert.deepEqual([route.id, route.section, route.hash], ['system', 'overview', '#/system/overview'], hash);
}
// D28: the production hub is untouched.
const hub = resolveRoute('#/overview');
assert.deepEqual([hub.id, hub.section], ['overview', undefined]);
// Q12: Overview's System card opens the native System Overview, with no legacy href or action.
const card = routes.find(route => route.id === 'system');
assert.equal(card.href, '#/system/overview');
assert.equal(card.native, true);
assert.equal(card.action, 'Open System');
});
test('the router renders again when only the System section changes', async () => {
const { mountRouter } = await source;
const win = new EventTarget();
win.location = { hash: '#/system' };
const replacements = [];
win.history = { state: null, replaceState(state, unused, hash) { replacements.push(hash); win.location.hash = hash; } };
const seen = [];
const dispose = mountRouter(win, route => seen.push([route.id, route.section]));
for (const hash of ['#/system/appearance', '#/system/appearance', '#/system/bogus', '#/system/overview', '#/images']) {
win.location.hash = hash;
win.dispatchEvent(new Event('hashchange'));
}
assert.deepEqual(seen, [['system', 'overview'], ['system', 'appearance'], ['system', 'overview'], ['images', undefined]]);
assert.deepEqual(replacements, ['#/system/overview']);
dispose();
});
test('inspect is the native Inspect a file page with the concept description', async () => {
@@ -0,0 +1,188 @@
// System › Appearance (#249 System S3 Task 8): the concept's "Browsing & display" rows (app.js 676–678) on the drafts
// of the one Save row, their stored values (Q3 mappings), the host timeout's 0–300 whole seconds and the D97 rows.
const test = require('node:test');
const assert = require('node:assert/strict');
const gallerySettings = require('../../web/js/gallery-settings.js');
const appearanceModule = import('../../web/js/workspace/system-appearance.mjs');
const stateModule = import('../../web/js/workspace/system-state.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
globalThis.GallerySettings = gallerySettings;
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
remove() { if (this.parentNode) this.parentNode.children = this.parentNode.children.filter(child => child !== this); this.parentNode = null; }
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
const BROWSER = { workspaceLayout: 'grid', workspaceDensity: 'comfortable', autoLoadMetadata: true, cacheMetadata: true,
imageQuality: 'medium', videoAutoplay: false, videoMute: true, videoLoop: true, showFilePaths: true, showImageInfo: true,
sidebarCollapsedByDefault: false };
async function mounted({ host = { webui_display_mode: 'newtab', result_timeout: 5 }, browser = BROWSER, hostFails = false } = {}) {
const { mountAppearance } = await appearanceModule;
const { createSystemDrafts } = await stateModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
let hostValues = null;
const drafts = createSystemDrafts({
readBrowser: () => ({ ...browser }), readRuntime: () => ({}),
readHost: async () => { if (hostFails) throw new Error('offline'); return { values: hostValues, revision: 'sha256:one' }; },
writeBrowser: async () => ({ persisted: true }), writeHost: async () => ({ success: true, persisted: true }), writeRuntime: async () => ({}),
});
await drafts.load('browser');
const scope = createScope();
const [surface] = mountAppearance(null, { scope, drafts });
const byId = id => dom.find(surface, node => node.dataset?.setting === id);
return { dom, drafts, scope, surface, byId, loadHost: async () => { hostValues = host; await drafts.load('host'); },
fixHost: () => { hostFails = false; } };
}
test('the surface is the concept section with the concept rows in order, then the D97 rows', async () => {
const { APPEARANCE_ROWS } = await appearanceModule;
const { dom, surface, scope } = await mounted();
try {
assert.equal(surface.className, 'surface');
assert.equal(surface.children[0].textContent, 'Browsing & display');
assert.equal(surface.children[1].textContent, 'One set of defaults; toolbar changes apply to the current view.');
const rows = surface.children.slice(2);
const titles = rows.map(row => row.children[0].children[0].textContent);
assert.deepEqual(titles, ['Default layout', 'Thumbnail size', 'Continuous browsing', 'Load metadata on selection',
'Cache metadata in this browser', 'Image loading quality', 'Autoplay videos', 'Mute videos by default', 'Loop videos',
'Show file paths', 'Open from ComfyUI', 'Node search auto-hide (seconds)', 'Show image details on cards',
'Collapse viewer details by default', 'Reset display defaults']);
const continuous = rows[2];
assert.deepEqual(dom.tags(continuous.children[0]), ['h3', 'p']);
assert.equal(continuous.children[1].className, 'chip blue');
assert.equal(continuous.children[1].textContent, 'Infinite scroll');
assert.equal(rows.at(-1).className, 'operation');
assert.equal(APPEARANCE_ROWS.filter(Array.isArray).length, 13);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
test('added rows reach their consumers: each control edits its stored key with the Q3 values', async () => {
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
// Host rows stay disabled until the server settings answered.
assert.equal(byId('open-mode').disabled, true);
assert.equal(byId('timeout').disabled, true);
await loadHost();
assert.equal(byId('open-mode').disabled, false);
assert.equal(byId('timeout').value, '5');
assert.deepEqual(byId('image-quality').children.map(option => [option.value, option.textContent]),
[['medium', 'Optimized'], ['high', 'Original'], ['low', 'Fast']]);
assert.deepEqual(byId('open-mode').children.map(option => [option.value, option.textContent]),
[['newtab', 'New tab'], ['popup', 'Popup window']]);
const edits = [['default-view', 'masonry'], ['density', 'large'], ['image-quality', 'low'], ['open-mode', 'popup']];
for (const [id, value] of edits) { byId(id).value = value; byId(id).dispatch('change'); }
for (const id of ['metadata-auto', 'metadata-cache', 'video-muted', 'video-loop', 'paths', 'card-details']) {
byId(id).checked = false; byId(id).dispatch('change');
}
for (const id of ['video-auto', 'viewer-details']) { byId(id).checked = true; byId(id).dispatch('change'); }
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'masonry', workspaceDensity: 'large', imageQuality: 'low',
autoLoadMetadata: false, cacheMetadata: false, videoMute: false, videoLoop: false, showFilePaths: false, showImageInfo: false,
videoAutoplay: true, sidebarCollapsedByDefault: true });
assert.deepEqual(drafts.changes('host'), { webui_display_mode: 'popup' });
} finally { scope.dispose(); dom.restore(); }
});
test('the auto-hide timeout takes whole seconds from 0 to 300 only', async () => {
const { resultTimeout } = await appearanceModule;
assert.deepEqual(['0', '30', ' 300 ', '301', '-1', '2.5', '', 'abc', '1e2'].map(resultTimeout), [0, 30, 300, null, null, null, null, null, null]);
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
await loadHost();
const timeout = byId('timeout');
timeout.value = '45'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 45 });
timeout.value = '301'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), {});
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
timeout.value = '0'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 0 });
assert.deepEqual([...drafts.failed('host')], []);
} finally { scope.dispose(); dom.restore(); }
});
test('reset restores defaults without touching other keys', async () => {
const { DISPLAY_KEYS } = await appearanceModule;
const changed = { ...BROWSER, workspaceLayout: 'list', videoMute: false, showImageInfo: false, sidebarCollapsedByDefault: true };
const { dom, drafts, scope, surface, loadHost } = await mounted({ browser: changed });
try {
await loadHost();
drafts.set('host', 'result_timeout', 9);
const reset = dom.find(surface.children.at(-1), node => node.tagName === 'BUTTON');
assert.equal(reset.textContent, 'Reset');
reset.click();
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'grid', videoMute: true, showImageInfo: true, sidebarCollapsedByDefault: false });
// The host fields and every key outside the display defaults are left as they are.
assert.deepEqual(drafts.changes('host'), { result_timeout: 9 });
assert.deepEqual([...DISPLAY_KEYS].sort(), Object.keys(BROWSER).sort());
} finally { scope.dispose(); dom.restore(); }
});
test('host rows that could not be read stay disabled and say so; a later read clears it', async () => {
const { HOST_UNAVAILABLE } = await appearanceModule;
const { dom, drafts, scope, byId, surface, fixHost } = await mounted({ hostFails: true });
const notes = () => surface.children.slice(2).flatMap(row => row.children[0].children.slice(2).map(note => [row.children[0].children[0].textContent, note.className, note.textContent]));
try {
assert.deepEqual(notes(), []);
assert.equal(await drafts.reloadHost(), false);
assert.deepEqual(notes(), [['Open from ComfyUI', 'small', HOST_UNAVAILABLE], ['Node search auto-hide (seconds)', 'small', HOST_UNAVAILABLE]]);
assert.equal(byId('timeout').disabled, true);
// Repaints keep one line per row.
drafts.set('browser', 'videoMute', false);
assert.equal(notes().length, 2);
fixHost();
assert.equal(await drafts.reloadHost(), true);
assert.deepEqual(notes(), []);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
+214
View File
@@ -0,0 +1,214 @@
// System controls (#249 System S2 Task 5): the concept's `setting()` (app.js 624–634, its toggles hydrated into Toolkit
// switches by `hydrateSwitches` 708–717), `operation()` (645–654) and `sectionNav()` (306–308) as DOM builders whose
// listeners live on the caller's scope.
const test = require('node:test');
const assert = require('node:assert/strict');
const controlsModule = import('../../web/js/workspace/system-controls.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
test('settingRow is the concept row: div > label[for] + p, then the control', async () => {
const { settingRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const changes = [];
const row = settingRow({ id: 'timeout', label: 'Node search auto-hide (seconds)',
description: 'Node search behavior only. Use 0 to keep results visible.',
control: { type: 'number', value: 30, min: 0, max: 300 }, scope, onChange: value => changes.push(value) });
assert.equal(row.className, 'setting-row');
assert.deepEqual(dom.tags(row), ['div', 'input']);
const [text, input] = row.children;
assert.deepEqual(dom.tags(text), ['label', 'p']);
assert.equal(text.children[0].textContent, 'Node search auto-hide (seconds)');
assert.equal(text.children[0].htmlFor, 'timeout');
assert.equal(text.children[1].textContent, 'Node search behavior only. Use 0 to keep results visible.');
assert.deepEqual([input.id, input.type, input.value, input.min, input.max], ['timeout', 'number', '30', '0', '300']);
assert.equal(input.dataset.setting, 'timeout');
input.value = '45';
input.dispatch('change');
assert.deepEqual(changes, [45]);
const select = settingRow({ id: 'open-mode', label: 'Open from ComfyUI', description: 'Keep the node pack’s popup / new-tab preference.',
control: { type: 'select', options: [['newtab', 'New tab'], ['popup', 'Popup window']], value: 'popup' }, scope,
onChange: value => changes.push(value) }).children[1];
assert.equal(select.tagName, 'SELECT');
assert.deepEqual(select.children.map(option => [option.value, option.textContent]), [['newtab', 'New tab'], ['popup', 'Popup window']]);
assert.equal(select.value, 'popup');
select.value = 'newtab';
select.dispatch('change');
assert.deepEqual(changes, [45, 'newtab']);
// The concept's range is 1–16 unless the row says otherwise (D113: the CPU count).
const range = settingRow({ id: 'workers', label: 'Worker threads', description: 'Balance background scans.',
control: { type: 'range', value: 4 } }).children[1];
assert.deepEqual([range.type, range.min, range.max, range.value], ['range', '1', '16', '4']);
const password = settingRow({ id: 'civitai-key', label: 'CivitAI API key', description: 'Credentials stay on the server.',
control: { type: 'password', value: '' } }).children[1];
assert.equal(password.type, 'password');
// A ready-made control is placed as given and takes the row id.
const custom = document.createElement('textarea');
const customRow = settingRow({ id: 'custom', label: 'Custom', description: 'Any control.', control: custom });
assert.equal(customRow.children[1], custom);
assert.equal(custom.id, 'custom');
assert.throws(() => settingRow({ id: '', label: 'x', description: 'y', control: { type: 'number' } }), TypeError);
assert.throws(() => settingRow({ id: 'x', label: 'x', description: 'y', control: { type: 'colour' } }), TypeError);
} finally {
scope.dispose();
dom.restore();
}
});
test('a toggle row holds the Toolkit switch, labelled by the row label', async () => {
const { settingRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const changes = [];
const row = settingRow({ id: 'video-auto', label: 'Autoplay videos', description: 'Videos stay within the same gallery.',
control: { type: 'toggle', checked: false }, scope, onChange: value => changes.push(value) });
const field = row.children[1];
// createSwitch('', '', id): `label.ca-switch` > input[type=checkbox][role=switch] + an empty label text.
assert.equal(field.tagName, 'LABEL');
assert.equal(field.className, 'ca-switch');
const input = field.children[0];
assert.deepEqual([input.type, input.role, input.id, input.checked], ['checkbox', 'switch', 'video-auto', false]);
assert.equal(row.children[0].children[0].htmlFor, input.id);
assert.equal(input.dataset.setting, 'video-auto');
input.checked = true;
input.dispatch('change');
assert.deepEqual(changes, [true]);
// The concept's toggles are on unless configured off (`val !== false`).
const on = settingRow({ id: 'video-loop', label: 'Loop videos', description: 'Repeat a clip.', control: { type: 'toggle' } });
assert.equal(on.children[1].children[0].checked, true);
} finally {
scope.dispose();
dom.restore();
}
});
test('operationRow is the concept operation with a danger class only when flagged', async () => {
const { operationRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const clicks = [];
const row = operationRow({ title: 'Missing-file links',
description: 'Remove stale database links only. Files on disk are never deleted by this operation.',
label: 'Review plan', danger: true, scope, onClick: button => clicks.push(button) });
assert.equal(row.className, 'operation');
assert.deepEqual(dom.tags(row), ['div', 'div']);
assert.deepEqual(dom.tags(row.children[0]), ['h3', 'p']);
assert.equal(row.children[0].children[0].textContent, 'Missing-file links');
assert.equal(row.children[1].className, 'row-actions');
const button = row.children[1].children[0];
assert.deepEqual([button.tagName, button.type, button.textContent, button.className], ['BUTTON', 'button', 'Review plan', 'danger']);
button.click();
assert.deepEqual(clicks, [button]);
const plain = operationRow({ title: 'Statistics', description: 'Usage by prompt, category, tag and date.', label: 'Explore',
scope, onClick() {} });
assert.equal(plain.children[1].children[0].className, '');
assert.throws(() => operationRow({ title: 'x', description: 'y', label: 'z', onClick() {} }), TypeError);
} finally {
scope.dispose();
dom.restore();
}
});
test('sectionNav marks the active section and reports the chosen one', async () => {
const { sectionNav } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const chosen = [];
const nav = sectionNav({ label: 'System sections', sections: [['overview', 'Overview'], ['appearance', 'Appearance']],
active: 'appearance', scope, onSelect: id => chosen.push(id) });
assert.equal(nav.className, 'section-nav');
assert.equal(nav.getAttribute('role'), 'group');
assert.equal(nav.getAttribute('aria-label'), 'System sections');
assert.deepEqual(nav.children.map(button => [button.textContent, button.getAttribute('aria-pressed'), button.dataset.value]),
[['Overview', 'false', 'overview'], ['Appearance', 'true', 'appearance']]);
nav.children[0].click();
assert.deepEqual(chosen, ['overview']);
} finally {
scope.dispose();
dom.restore();
}
});
test('disposing the scope leaves no listener behind', async () => {
const { settingRow, operationRow, sectionNav } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
settingRow({ id: 'a', label: 'A', description: 'a', control: { type: 'toggle' }, scope, onChange() {} });
settingRow({ id: 'b', label: 'B', description: 'b', control: { type: 'select', options: ['Grid', 'List'] }, scope, onChange() {} });
operationRow({ title: 'C', description: 'c', label: 'Go', scope, onClick() {} });
sectionNav({ label: 'System sections', sections: [['overview', 'Overview']], active: 'overview', scope, onSelect() {} });
assert.equal(dom.listenerCount() > 0, true);
scope.dispose();
assert.equal(dom.listenerCount(), 0);
} finally {
scope.dispose();
dom.restore();
}
});
+255
View File
@@ -0,0 +1,255 @@
// System › Overview (#249 System S2 Task 6): the concept's Overview copy on real values (D93), the Health report rows
// (D95), the Statistics tables (D96) and the health check run that waits for its read job.
const test = require('node:test');
const assert = require('node:assert/strict');
const overviewModule = import('../../web/js/workspace/system-overview.mjs');
const OVERVIEW = Object.freeze({
library: { positive_prompts: 1234, negative_variants: 7 },
images: { indexed: 24, rated: 20 },
health: null,
backup: null,
needs_review: { pending: null },
sources: { configured: 2, effective: 2, auto_detected: false },
});
test('metrics are the server counts and Not checked before any health check', async () => {
const { metricValues } = await overviewModule;
assert.deepEqual(metricValues(OVERVIEW), [
[(1234).toLocaleString(), 'Positive prompts'], ['7', 'Negative variants'], ['24', 'Images'],
['Not checked', 'Database integrity'],
]);
});
test('metric 4 is the last health result', async () => {
const { integrityText } = await overviewModule;
assert.equal(integrityText(null), 'Not checked');
assert.equal(integrityText({ status: 'passed', issues: 0 }), 'Passed');
assert.equal(integrityText({ status: 'issues', issues: 1 }), '1 issue');
assert.equal(integrityText({ status: 'issues', issues: 3 }), '3 issues');
assert.equal(integrityText({ status: 'incomplete', issues: 0 }), 'Incomplete');
// Issues found beside a check that could not run (Needs review, Q6) are still named.
assert.equal(integrityText({ status: 'incomplete', issues: 2 }), '2 issues');
});
test('library health rows never invent a review count, a backup or thumbnails', async () => {
const { ambiguousTitle, backupText } = await overviewModule;
assert.equal(ambiguousTitle({ pending: null }), 'Ambiguous image links');
assert.equal(ambiguousTitle(undefined), 'Ambiguous image links');
assert.equal(ambiguousTitle({ pending: 1 }), '1 ambiguous image link');
assert.equal(ambiguousTitle({ pending: 3 }), '3 ambiguous image links');
assert.equal(backupText(null), 'No retained backup record.');
assert.equal(backupText({ verified_at: 'not a time' }), 'No retained backup record.');
assert.match(backupText({ verified_at: '2026-10-22T09:30:00Z' }), /^Last verified backup .+\.$/);
});
test('the thumbnail row is its own read of the thumbnail status', async () => {
const { thumbnailTitle, THUMBNAILS_LOADING } = await overviewModule;
// Until `/images/thumbnails/status` answers the row says it is checking; a failed read is unavailable, never 0.
assert.equal(THUMBNAILS_LOADING, 'Checking thumbnails…');
assert.equal(thumbnailTitle(undefined), THUMBNAILS_LOADING);
assert.equal(thumbnailTitle({ pending: 0 }), 'No missing thumbnails');
assert.equal(thumbnailTitle({ pending: 1 }), '1 missing thumbnail');
assert.equal(thumbnailTitle({ pending: 12 }), '12 missing thumbnails');
assert.equal(thumbnailTitle(null), 'Thumbnail status unavailable');
assert.equal(thumbnailTitle({ pending: 'x' }), 'Thumbnail status unavailable');
// With no image source there is nothing to count.
assert.equal(thumbnailTitle({ pending: 0 }, { effective: 0 }), 'Thumbnail status unavailable');
assert.equal(thumbnailTitle({ pending: 2 }, { effective: 1 }), '2 missing thumbnails');
});
test('an integrity count SQLite stopped at reads as a lower bound', async () => {
const { checkChip } = await overviewModule;
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 100, capped: true }), '100+ problems found');
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 37, capped: false }), '37 problems found');
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 1 }), '1 problem found');
});
test('the sources row tells configured, available and detected sources apart', async () => {
const { sourcesCopy } = await overviewModule;
const shared = 'Monitoring and enrichment share one source list.';
assert.deepEqual(sourcesCopy({ configured: 2, effective: 2, auto_detected: false }), ['2 output sources', shared]);
assert.deepEqual(sourcesCopy({ configured: 2, effective: 1, auto_detected: false }),
['1 output source', `1 of 2 configured sources available. ${shared}`]);
assert.deepEqual(sourcesCopy({ configured: 0, effective: 1, auto_detected: true }),
['1 output source', `None configured is available; ComfyUI's output folder is used. ${shared}`]);
});
const REPORT = Object.freeze({
checked_at: '2026-10-23T08:00:00Z',
status: 'issues',
checks: [
{ id: 'integrity', label: 'Integrity check', status: 'passed', count: 0 },
{ id: 'foreign_keys', label: 'Foreign keys & pair references', status: 'issues', count: 2 },
{ id: 'canonical_hashes', label: 'Canonical hashes', status: 'issues', count: 1 },
{ id: 'pair_uniqueness', label: 'Pair uniqueness', status: 'failed', count: null },
{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 2,
sources: [{ name: 'output', status: 'checked', count: 2 }] },
{ id: 'ambiguous_metadata', label: 'Ambiguous metadata', status: 'unavailable', count: null },
{ id: 'consistency', label: 'Consistency', status: 'issues', count: null },
],
});
test('health rows give each check a chip from its real status, in the server order', async () => {
const { healthRows } = await overviewModule;
assert.deepEqual(healthRows(REPORT), [
['Integrity check', 'Passed'],
['Foreign keys & pair references', '2 broken references'],
['Canonical hashes', '1 hash does not match its text'],
['Pair uniqueness', 'Could not check'],
['Missing files', '2 links need review'],
['Ambiguous metadata', 'Not available yet'],
['Consistency', 'Issues found'],
]);
});
test('missing files is reported per image source when there are several', async () => {
const { healthRows } = await overviewModule;
const report = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 1, partial: true,
sources: [{ name: 'output', status: 'checked', count: 1 }, { name: 'input', status: 'checked', count: 0 },
{ name: 'archive', status: 'unavailable', count: null }] }] };
assert.deepEqual(healthRows(report), [
['Missing files · output', '1 link needs review'],
['Missing files · input', 'Passed'],
['Missing files · archive', 'Not available'],
]);
const unavailable = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'unavailable', count: null, sources: [] }] };
assert.deepEqual(healthRows(unavailable), [['Missing files', 'Not available']]);
});
test('the report summary says when it was checked and whether it is complete', async () => {
const { healthSummary } = await overviewModule;
const when = new Date(REPORT.checked_at).toLocaleString();
assert.equal(healthSummary({ ...REPORT, status: 'passed' }), `Checked ${when}. Every check passed.`);
assert.equal(healthSummary({ ...REPORT, status: 'incomplete' }), `Checked ${when}. Some checks could not run, so this report is incomplete.`);
assert.equal(healthSummary({ ...REPORT, checked_at: 'x' }), 'Nothing was changed; review the checks with issues.');
});
test('statistics rows are the real runs and ratings, then the library totals', async () => {
const { categoryRows, totalsRows } = await overviewModule;
const statistics = {
by_category: [{ category: 'Landscapes', runs: 1200, average_rating: 4.5 },
{ category: null, runs: 3, average_rating: null }],
totals: { total_prompts: 12, total_tags: 15, total_images: 24 },
rated_images: 20,
};
assert.deepEqual(categoryRows(statistics), [['Landscapes', (1200).toLocaleString(), '4.5 / 5'], ['Uncategorized', '3', 'Not rated']]);
assert.deepEqual(totalsRows(statistics), [['Prompts', '12'], ['Tags', '15'], ['Images', '24'], ['Rated images', '20']]);
});
/** A fake api answering each `get`/`post` from a queue per URL, recording the calls. */
function fakeApi(answers) {
const calls = [];
const answer = (method, url) => {
calls.push(`${method} ${url}`);
const queue = answers[`${method} ${url}`];
if (!queue?.length) throw new Error(`unexpected ${method} ${url}`);
const next = queue.shift();
return next instanceof Error ? Promise.reject(next) : Promise.resolve(next);
};
return { calls, get: url => answer('GET', url), post: url => answer('POST', url) };
}
const JOB = '0123456789abcdef0123456789abcdef';
const job = (state, id = JOB, kind = 'health') => ({ id, kind, state });
/**
* A fake Activity store holding `jobs`: `track(job)` lists a started job, `load()` applies the next of `reads` (the
* server's job lists) and `push(job)` is a stream update; each notifies as the real store does. It never polls.
*/
function fakeActivity({ jobs = [], reads = [] } = {}) {
const listeners = new Set();
const emit = () => { for (const listener of [...listeners]) listener(); };
const store = {
loads: 0,
find: id => jobs.find(item => item.id === id),
running: () => jobs.filter(item => item.state === 'running'),
subscribe(listener) { listeners.add(listener); return () => listeners.delete(listener); },
listeners: () => listeners.size,
async load() {
store.loads += 1;
if (reads.length) jobs = reads.shift();
emit();
},
track(item) {
if (!item?.id) throw new TypeError('Invalid job');
store.push(item);
return item;
},
push(item) {
jobs = [...jobs.filter(other => other.id !== item.id), item];
emit();
},
};
return store;
}
test('a health check tracks its job in the Activity store, then reads the report', async () => {
const { runHealthCheck } = await overviewModule;
const api = fakeApi({
'POST /prompt_manager/system/health': [{ job: job('running') }],
'GET /prompt_manager/system/health': [{ report: REPORT }],
});
const activity = fakeActivity();
const checking = runHealthCheck(api, { activity });
await new Promise(resolve => setImmediate(resolve));
// The store follows the started job's event stream; nothing reads or polls the job here.
assert.equal(activity.loads, 0);
assert.equal(activity.find(JOB).state, 'running');
assert.deepEqual(api.calls, ['POST /prompt_manager/system/health']);
activity.push(job('finished'));
assert.equal(await checking, REPORT);
assert.deepEqual(api.calls, ['POST /prompt_manager/system/health', 'GET /prompt_manager/system/health']);
assert.equal(activity.listeners(), 0);
});
test('a check already running elsewhere is waited for, not refused', async () => {
const { runHealthCheck } = await overviewModule;
const busy = Object.assign(new Error('busy'), { status: 409, code: 'busy' });
const api = fakeApi({
'POST /prompt_manager/system/health': [busy],
'GET /prompt_manager/system/health': [{ report: REPORT }],
});
const activity = fakeActivity({ reads: [[job('running', 'f'.repeat(32), 'scan'), job('running')]] });
const checking = runHealthCheck(api, { activity });
await new Promise(resolve => setImmediate(resolve));
assert.equal(activity.loads, 1);
activity.push(job('finished'));
assert.equal(await checking, REPORT);
// The other check ended between the refusal and the read: its report is the latest.
const ended = fakeApi({ 'POST /prompt_manager/system/health': [busy], 'GET /prompt_manager/system/health': [{ report: REPORT }] });
assert.equal(await runHealthCheck(ended, { activity: fakeActivity({ reads: [[]] }) }), REPORT);
});
test('a failed, lost or reportless check rejects; other errors pass through', async () => {
const { runHealthCheck, waitForJob, HEALTH_FAILED_TEXT } = await overviewModule;
const failed = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('failed') }] });
await assert.rejects(runHealthCheck(failed, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT });
// The store marks a job lost when the server no longer lists it (ComfyUI restarted).
const restarted = fakeActivity({ jobs: [job('running')] });
const waiting = waitForJob(restarted, JOB);
restarted.push(job('lost'));
assert.equal(await waiting, 'lost');
const empty = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('finished') }],
'GET /prompt_manager/system/health': [{ report: null }] });
await assert.rejects(runHealthCheck(empty, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT });
const broken = fakeApi({ 'POST /prompt_manager/system/health': [Object.assign(new Error('server'), { status: 500 })] });
await assert.rejects(runHealthCheck(broken, { activity: fakeActivity() }), { message: 'server' });
const invalid = fakeApi({ 'POST /prompt_manager/system/health': [{ job: null }] });
await assert.rejects(runHealthCheck(invalid, { activity: fakeActivity() }), { message: 'Invalid job' });
});
test('waiting stops when its signal aborts', async () => {
const { waitForJob } = await overviewModule;
const controller = new AbortController();
const activity = fakeActivity({ jobs: [job('running')] });
const waiting = waitForJob(activity, JOB, { signal: controller.signal });
await new Promise(resolve => setImmediate(resolve));
controller.abort(new Error('left'));
await assert.rejects(waiting, { message: 'left' });
assert.equal(activity.listeners(), 0);
const gone = new AbortController();
gone.abort(new Error('already'));
await assert.rejects(waitForJob(fakeActivity(), JOB, { signal: gone.signal }), { message: 'already' });
});
+145
View File
@@ -0,0 +1,145 @@
// System › Sources & scans (#249 System S4 Task 11): the concept's "Image sources" list (app.js 679–681) built from the
// server's configured, effective and watched sources (GET /settings), the host draft of `gallery_root_paths`, the
// 3.2.0 update notice's dismissal and the Refresh files toast (D98).
const test = require('node:test');
const assert = require('node:assert/strict');
const sourcesModule = import('../../web/js/workspace/system-sources.mjs');
const STATUS = Object.freeze({
configured: [
{ label: 'ComfyUI output', path: 'output', valid: true, code: null, reason: null },
{ label: 'ComfyUI input', path: 'input', valid: true, code: null, reason: null },
],
effective: [
{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'watching' },
{ label: 'ComfyUI input', path: 'input', root_index: 1, watch: 'manual' },
],
auto_detected: false,
});
test('each effective source is the concept row: label, "<path> · watching|manual scan", Scan changes or Scan folder', async () => {
const { sourceRows } = await sourcesModule;
const saved = ['output', 'input'];
const { rows, note } = sourceRows({ sources: STATUS, saved, current: saved });
assert.equal(note, null);
assert.deepEqual(rows, [
{ key: 'output', title: 'ComfyUI output', description: 'output · watching', scan: { label: 'Scan changes', root: 0, path: 'output' },
remove: 'output' },
{ key: 'input', title: 'ComfyUI input', description: 'input · manual scan', scan: { label: 'Scan folder', root: 1, path: 'input' },
remove: 'input' },
]);
});
test('watching only with a live observer; restart pending otherwise (D98)', async () => {
const { sourceRows, WATCH_COPY } = await sourcesModule;
assert.deepEqual(WATCH_COPY, { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan',
restart_pending: 'restart pending' });
const sources = { ...STATUS, effective: STATUS.effective.map(source => ({ ...source, watch: 'restart_pending' })) };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · restart pending', 'Scan folder'], ['input · restart pending', 'Scan folder']]);
});
test('a configured source that cannot be used shows the server reason and only Remove', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'unmounted-drive', path: 'unmounted-drive', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const saved = ['output', 'input', 'unmounted-drive'];
const { rows } = sourceRows({ sources, saved, current: saved });
assert.deepEqual(rows.at(-1), { key: 'unmounted-drive', title: 'unmounted-drive',
description: 'unmounted-drive · Gallery root does not exist', scan: null, remove: 'unmounted-drive' });
});
test('an added source is listed as not saved yet with its validated label; a removed one leaves the list', async () => {
const { sourceRows } = await sourcesModule;
const labels = new Map([['output/archive', 'archive']]);
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['input', 'output/archive'], labels });
assert.equal(note, null);
assert.deepEqual(rows.map(row => [row.title, row.description, row.scan?.label ?? null, row.remove]), [
['ComfyUI input', 'input · manual scan', 'Scan folder', 'input'],
['archive', 'output/archive · not saved yet', null, 'output/archive'],
]);
// Without a recorded label (a reload of System), the folder name stands in.
assert.equal(sourceRows({ sources: STATUS, saved: [], current: ['output/new-collection'] }).rows[0].title, 'new-collection');
});
test('removing the last entry states auto-detect (D98)', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: [] });
assert.deepEqual(rows, []);
assert.equal(note, NOTES.removedAll);
assert.equal(NOTES.removedAll, 'No source is configured. After saving, ComfyUI’s output folder is used.');
});
test('auto-detected sources are listed without Remove, with the line that says how they were chosen', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const auto = { configured: [], effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'manual' }], auto_detected: true };
assert.deepEqual(sourceRows({ sources: auto, saved: [], current: [] }), { rows: [
{ key: 'auto:0', title: 'ComfyUI output', description: 'output · manual scan', scan: { label: 'Scan folder', root: 0, path: 'output' },
remove: null },
], note: NOTES.autoDetected });
// Every configured source unusable: ComfyUI's output folder is used instead.
const fallback = { ...auto, configured: [{ label: 'gone', path: 'gone', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const result = sourceRows({ sources: fallback, saved: ['gone'], current: ['gone'] });
assert.deepEqual(result.rows.map(row => [row.title, row.remove]), [['gone', 'gone'], ['ComfyUI output', null]]);
assert.equal(result.note, NOTES.fallback);
assert.equal(sourceRows({ sources: { configured: [], effective: [], auto_detected: false }, saved: [], current: [] }).note, NOTES.none);
});
test('a path listed twice in the draft is one row', async () => {
const { sourceRows } = await sourcesModule;
assert.equal(sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['output', 'output'] }).rows.length, 1);
});
test('roots equal to the saved list are the saved list itself, so the draft is clean again', async () => {
const { draftRoots } = await sourcesModule;
const saved = ['output', 'input'];
assert.equal(draftRoots(['output', 'input'], saved), saved);
assert.deepEqual(draftRoots(['input'], saved), ['input']);
assert.notEqual(draftRoots(['input', 'output'], saved), saved);
assert.deepEqual(draftRoots([], undefined), []);
});
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed); storage that throws keeps it for this view', async () => {
const { noticeDismissed, dismissNotice, NOTICE_KEY, NOTICE_VERSION } = await sourcesModule;
assert.deepEqual([NOTICE_KEY, NOTICE_VERSION], ['pm_update_notice_dismissed', '3.2.0']);
const values = new Map();
const storage = { getItem: key => values.get(key) ?? null, setItem: (key, value) => values.set(key, String(value)) };
assert.equal(noticeDismissed(() => storage), false);
values.set(NOTICE_KEY, '3.1.0');
assert.equal(noticeDismissed(() => storage), false);
assert.equal(dismissNotice(() => storage), true);
assert.equal(values.get(NOTICE_KEY), '3.2.0');
assert.equal(noticeDismissed(() => storage), true);
const denied = () => { throw new Error('SecurityError'); };
assert.equal(noticeDismissed(denied), false);
assert.equal(dismissNotice(denied), false);
assert.equal(dismissNotice(() => ({ setItem() { throw new Error('QuotaExceededError'); } })), false);
});
test('Refresh files toasts the rescan counts, or says it finished when they are missing', async () => {
const { refreshedText } = await sourcesModule;
assert.equal(refreshedText({ total: 1234, roots: 2 }), 'Files refreshed. 1,234 files in 2 output folders.');
assert.equal(refreshedText({ total: 1, roots: 1 }), 'Files refreshed. 1 file in 1 output folder.');
assert.equal(refreshedText({ total: -1, roots: 1 }), 'Files refreshed.');
assert.equal(refreshedText(null), 'Files refreshed.');
});
test('monitoring switched off: a source still watched says so until the restart (D128)', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, effective: [{ ...STATUS.effective[0], watch: 'stopping' }, STATUS.effective[1]] };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · watching until restart', 'Scan changes'], ['input · manual scan', 'Scan folder']]);
});
test('a valid configured source missing from the effective list is another spelling of a listed folder (D128)', async () => {
const { sourceRows, DUPLICATE_SOURCE } = await sourcesModule;
assert.equal(DUPLICATE_SOURCE, 'same folder as another source');
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'output', path: 'output/.', valid: true, code: null, reason: null }] };
const saved = ['output', 'input', 'output/.'];
assert.deepEqual(sourceRows({ sources, saved, current: saved }).rows.at(-1), { key: 'output/.', title: 'output',
description: 'output/. · same folder as another source', scan: null, remove: 'output/.' });
});
+366
View File
@@ -0,0 +1,366 @@
// System draft store (#249 System S2 Task 5): one Save row for browser, host and runtime fields (concept app.js
// renderSystem `.save-row` 705, `save-settings` 1320). It saves browser, then host with the GET revision, then runtime;
// a domain that fails keeps its fields dirty, and the status copy (D92) says what really happened.
const test = require('node:test');
const assert = require('node:assert/strict');
const source = import('../../web/js/workspace/system-state.mjs');
/** Adapters that record every call; `host` answers like POST /prompt_manager/settings. */
function adapters({ browser = {}, host = {}, runtime = {}, revision = 'sha256:one' } = {}) {
const calls = [];
const outcome = { browser: null, host: null, runtime: null };
return {
calls,
outcome,
readBrowser: () => { calls.push(['readBrowser']); return { ...browser }; },
readHost: async () => { calls.push(['readHost']); return { values: { ...host }, revision }; },
readRuntime: async () => { calls.push(['readRuntime']); return { ...runtime }; },
writeBrowser: async changes => {
calls.push(['writeBrowser', changes]);
if (outcome.browser) return outcome.browser(changes);
return { persisted: true };
},
writeHost: async (changes, options) => {
calls.push(['writeHost', changes, options]);
if (outcome.host) return outcome.host(changes, options);
return { success: true, persisted: true, revision: 'sha256:two', runtime_applied: true };
},
writeRuntime: async changes => {
calls.push(['writeRuntime', changes]);
if (outcome.runtime) return outcome.runtime(changes);
return { success: true };
},
};
}
async function loaded(domains) {
const { createSystemDrafts } = await source;
const drafts = createSystemDrafts(domains);
await Promise.all(['browser', 'host', 'runtime'].map(domain => drafts.load(domain)));
domains.calls.length = 0;
return drafts;
}
test('sections follow the concept order and the production hashes', async () => {
const { SYSTEM_SECTIONS, isSystemSection } = await source;
assert.deepEqual(SYSTEM_SECTIONS.map(([id, label]) => [id, label]), [
['overview', 'Overview'], ['appearance', 'Appearance'], ['sources', 'Sources & scans'],
['integrations', 'Integrations'], ['tagging', 'Auto-tagging'], ['maintenance', 'Maintenance'],
['backups', 'Backup & export'], ['diagnostics', 'Diagnostics & logs'],
]);
assert.equal(isSystemSection('backups'), true);
for (const value of ['general', '', 'constructor', '__proto__', undefined]) assert.equal(isSystemSection(value), false, value);
});
test('drafts read baselines and report dirty only for changed values', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } });
const drafts = await loaded(domains);
assert.equal(drafts.get('browser', 'layout'), 'grid');
assert.equal(drafts.get('host', 'result_timeout'), 30);
assert.equal(drafts.dirty(), false);
drafts.set('host', 'result_timeout', 45);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.equal(drafts.dirty(), true);
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.dirty('browser'), false);
// Setting the saved value back is clean again.
drafts.set('host', 'result_timeout', 30);
assert.equal(drafts.dirty(), false);
assert.throws(() => drafts.set('elsewhere', 'x', 1), TypeError);
assert.throws(() => drafts.get('elsewhere', 'x'), TypeError);
});
test('save commits browser then host then runtime', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 }, runtime: { level: 'INFO' } });
const drafts = await loaded(domains);
drafts.set('runtime', 'level', 'DEBUG');
drafts.set('host', 'result_timeout', 45);
drafts.set('browser', 'layout', 'list');
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'saved', host: 'saved', runtime: 'applied' });
assert.deepEqual(domains.calls, [
['writeBrowser', { layout: 'list' }],
['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }],
['writeRuntime', { level: 'DEBUG' }],
]);
assert.equal(drafts.dirty(), false);
// The next host save sends the revision the last one returned.
drafts.set('host', 'result_timeout', 60);
await drafts.save();
assert.deepEqual(domains.calls.at(-1), ['writeHost', { result_timeout: 60 }, { expected_revision: 'sha256:two' }]);
});
test('host failure keeps host fields dirty and browser saved', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } });
const drafts = await loaded(domains);
domains.outcome.host = () => { throw Object.assign(new Error('Settings could not be written to config.json.'), { code: 'config-write-failed' }); };
drafts.set('browser', 'layout', 'list');
drafts.set('host', 'result_timeout', 45);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'saved', host: 'failed', runtime: 'clean' });
assert.equal(drafts.dirty('browser'), false);
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.deepEqual([...drafts.failed('browser')], []);
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
// A save answered `persisted: false` without an error is not saved either.
domains.outcome.host = () => ({ success: false, persisted: false });
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' });
// Editing the failed field clears its highlight and reads as unsaved again.
drafts.set('host', 'result_timeout', 50);
assert.deepEqual([...drafts.failed('host')], []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('stale revision reports conflict and keeps the draft', async () => {
const domains = adapters({ host: { result_timeout: 30 } });
const drafts = await loaded(domains);
domains.outcome.host = () => { throw Object.assign(new Error('Settings changed since they were loaded.'), { code: 'settings-changed' }); };
drafts.set('host', 'result_timeout', 45);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'conflict', runtime: 'clean' });
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
// The conflict read the host values again (their new revision), keeping the draft.
assert.deepEqual(domains.calls.slice(-2).map(([name]) => name), ['writeHost', 'readHost']);
// Reloading the section reads the new revision and keeps the draft (Q10).
domains.outcome.host = null;
await drafts.load('host');
assert.equal(drafts.get('host', 'result_timeout'), 45);
await drafts.save();
assert.deepEqual(domains.calls.at(-1)[2], { expected_revision: 'sha256:one' });
});
test('host save without a read revision never posts', async () => {
const { createSystemDrafts } = await source;
// Never read: an edit made before the host values (and their revision) arrived must not merge blindly.
const unread = adapters({ host: { result_timeout: 30 } });
const drafts = createSystemDrafts(unread);
drafts.set('host', 'result_timeout', 45);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' });
assert.deepEqual(unread.calls, []);
assert.equal(drafts.dirty('host'), true);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
// A read of an unreadable configuration has no revision either.
const unreadable = adapters({ host: { result_timeout: 30 }, revision: null });
const blind = createSystemDrafts(unreadable);
await blind.load('host');
blind.set('host', 'result_timeout', 45);
assert.equal((await blind.save()).host, 'failed');
assert.equal(unreadable.calls.filter(([name]) => name === 'writeHost').length, 0);
// Once read, the same draft saves with the revision.
await drafts.load('host');
assert.equal((await drafts.save()).host, 'saved');
assert.deepEqual(unread.calls.at(-1), ['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }]);
});
test('reloadHost reads the host values and says whether it could', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 30 } });
const drafts = createSystemDrafts(domains);
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.get('host', 'result_timeout'), 30);
const broken = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } });
assert.equal(await broken.reloadHost(), false);
broken.set('host', 'result_timeout', 45);
assert.equal((await broken.save()).host, 'failed');
});
test('no domain called when clean', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls, []);
drafts.set('browser', 'layout', 'list');
drafts.discard();
assert.equal(drafts.dirty(), false);
assert.equal(drafts.get('browser', 'layout'), 'grid');
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls, []);
});
test('status text per outcome', async () => {
const { statusText, SETTINGS_STATUS } = await source;
assert.deepEqual(SETTINGS_STATUS, {
clean: 'No unsaved changes.',
dirty: 'Unsaved changes.',
browser: 'Saved in this browser.',
host: 'Saved to the server configuration.',
runtime: 'Applied until the server restarts.',
partial: 'Some changes were not saved. Review the highlighted settings.',
tabOnly: 'Saved for this tab only; browser storage is unavailable.',
conflict: 'Settings changed elsewhere. Reload this section to see them.',
restart: 'Restart ComfyUI to apply:',
});
const clean = { browser: 'clean', host: 'clean', runtime: 'clean' };
assert.equal(statusText(clean), 'No unsaved changes.');
assert.equal(statusText({ ...clean, browser: 'saved' }), 'Saved in this browser.');
assert.equal(statusText({ ...clean, host: 'saved' }), 'Saved to the server configuration.');
assert.equal(statusText({ ...clean, runtime: 'applied' }), 'Applied until the server restarts.');
assert.equal(statusText({ ...clean, browser: 'tab-only' }), 'Saved for this tab only; browser storage is unavailable.');
assert.equal(statusText({ browser: 'saved', host: 'saved', runtime: 'applied' }),
'Saved in this browser. Saved to the server configuration. Applied until the server restarts.');
assert.equal(statusText({ ...clean, browser: 'saved', runtime: 'failed' }),
'Some changes were not saved. Review the highlighted settings.');
assert.equal(statusText({ browser: 'failed', host: 'conflict', runtime: 'clean' }),
'Settings changed elsewhere. Reload this section to see them.');
});
test('the store says unsaved, then the save outcome, and notifies subscribers', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
const seen = [];
const unsubscribe = drafts.subscribe(() => seen.push(drafts.statusText()));
assert.equal(drafts.statusText(), 'No unsaved changes.');
drafts.set('browser', 'layout', 'list');
domains.outcome.browser = () => ({ persisted: false });
await drafts.save();
assert.deepEqual(seen, ['Unsaved changes.', 'Saved for this tab only; browser storage is unavailable.']);
unsubscribe();
drafts.set('browser', 'layout', 'masonry');
assert.equal(seen.length, 2);
// The tab-only value is this tab's baseline now.
drafts.set('browser', 'layout', 'list');
assert.equal(drafts.dirty(), false);
});
test('an edit made while a save runs stays dirty', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
let release;
domains.outcome.browser = () => new Promise(resolve => { release = () => resolve({ persisted: true }); });
drafts.set('browser', 'layout', 'list');
const saving = drafts.save();
// A second Save while one runs joins it instead of writing twice.
assert.equal(drafts.save(), saving);
assert.equal(drafts.saving(), true);
await new Promise(resolve => setImmediate(resolve));
drafts.set('browser', 'layout', 'masonry');
release();
assert.deepEqual(await saving, { browser: 'saved', host: 'clean', runtime: 'clean' });
assert.equal(drafts.saving(), false);
assert.equal(drafts.dirty('browser'), true);
assert.equal(drafts.get('browser', 'layout'), 'masonry');
assert.equal(domains.calls.filter(([name]) => name === 'writeBrowser').length, 1);
});
test('a failed read keeps the store usable and drafts survive a reload (Q10)', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = createSystemDrafts(domains);
await drafts.load('browser');
drafts.set('browser', 'layout', 'list');
await drafts.load('browser');
assert.equal(drafts.get('browser', 'layout'), 'list');
assert.equal(drafts.dirty(), true);
const failing = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } });
await assert.rejects(failing.load('host'), /offline/);
assert.equal(failing.dirty(), false);
assert.throws(() => createSystemDrafts({ ...domains, writeHost: null }), TypeError);
});
test('a rejected value drops its draft and is highlighted; a conflict elsewhere keeps the drafts', async () => {
const domains = adapters({ browser: { videoMute: true, videoLoop: true }, host: { result_timeout: 5 } });
const drafts = await loaded(domains);
assert.deepEqual(drafts.saved('browser'), { videoMute: true, videoLoop: true });
drafts.set('host', 'result_timeout', 12);
assert.equal(drafts.dirty('host'), true);
drafts.reject('host', 'result_timeout');
assert.equal(drafts.dirty('host'), false);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.equal(drafts.get('host', 'result_timeout'), 5);
// A valid value clears the highlight.
drafts.set('host', 'result_timeout', 7);
assert.deepEqual([...drafts.failed('host')], []);
drafts.discard();
drafts.set('browser', 'videoMute', false);
drafts.conflict('browser', []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
drafts.conflict('browser', ['videoMute']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
assert.deepEqual([...drafts.failed('browser')], ['videoMute']);
assert.equal(drafts.get('browser', 'videoMute'), false);
assert.throws(() => drafts.reject('nowhere', 'x'), TypeError);
});
test('a rejected value says why in the status until the next edit', async () => {
const drafts = await loaded(adapters({ host: { result_timeout: 5 } }));
drafts.reject('host', 'result_timeout');
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
drafts.set('host', 'result_timeout', 9);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a browser write refused as changed elsewhere keeps the draft and reads the browser values again', async () => {
const domains = adapters({ browser: { workspaceDensity: 'comfortable' } });
const drafts = await loaded(domains);
drafts.set('browser', 'workspaceDensity', 'compact');
domains.outcome.browser = () => { throw Object.assign(new Error('changed'), { code: 'settings-changed' }); };
assert.deepEqual(await drafts.save(), { browser: 'conflict', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls.map(([name]) => name), ['writeBrowser', 'readBrowser']);
assert.equal(drafts.get('browser', 'workspaceDensity'), 'compact');
assert.deepEqual([...drafts.failed('browser')], ['workspaceDensity']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
});
test('host read state says whether the server settings could be read', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 5 } });
let fail = true;
const drafts = createSystemDrafts({ ...domains, readHost: async () => { if (fail) throw new Error('offline'); return { values: {}, revision: 'r' }; } });
assert.equal(drafts.hostState(), 'pending');
assert.equal(await drafts.reloadHost(), false);
assert.equal(drafts.hostState(), 'failed');
fail = false;
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.hostState(), 'loaded');
});
test('a host save names the settings that apply after a restart (D128)', async () => {
const { statusText, RESTART_LABELS } = await source;
assert.deepEqual(RESTART_LABELS, { monitoring_enabled: 'Watch for new outputs' });
const saved = { browser: 'clean', host: 'saved', runtime: 'clean' };
assert.equal(statusText({ ...saved, restart: ['monitoring_enabled'] }),
'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// A field without a row label (the sources say "restart pending" per row) adds nothing.
assert.equal(statusText({ ...saved, restart: ['gallery_root_paths'] }), 'Saved to the server configuration.');
assert.equal(statusText({ ...saved, restart: [] }), 'Saved to the server configuration.');
const domains = adapters({ host: { monitoring_enabled: true } });
const drafts = await loaded(domains);
domains.outcome.host = () => ({ success: true, persisted: true, revision: 'sha256:two',
restart_required_fields: ['monitoring_enabled', 7] });
drafts.set('host', 'monitoring_enabled', false);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'saved', runtime: 'clean', restart: ['monitoring_enabled'] });
assert.equal(drafts.statusText(), 'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The next edit is unsaved again.
drafts.set('host', 'monitoring_enabled', true);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a host save refused for one source says the server reason and names it until the next edit', async () => {
const domains = adapters({ host: { gallery_root_paths: ['output', 'output/gone'] } });
const drafts = await loaded(domains);
const message = "output/gone can't be used: Gallery root does not exist. Remove it or reconnect it, then save.";
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source', source: 'output/gone' }); };
drafts.set('host', 'gallery_root_paths', ['output/gone']);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'clean', host: 'failed', runtime: 'clean',
invalid: { source: 'output/gone', message } });
assert.equal(drafts.statusText(), message);
assert.equal(drafts.invalidSource(), 'output/gone');
assert.deepEqual([...drafts.failed('host')], ['gallery_root_paths']);
// Any other failure keeps the generic copy and names no source.
domains.outcome.host = () => { throw Object.assign(new Error('busy'), { code: 'busy' }); };
await drafts.save();
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
assert.equal(drafts.invalidSource(), null);
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source' }); };
await drafts.save();
assert.equal(drafts.invalidSource(), null);
drafts.set('host', 'gallery_root_paths', ['output']);
assert.equal(drafts.invalidSource(), null);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
+143
View File
@@ -76,6 +76,7 @@ function fakeDOM() {
get activeElement() { return active ?? body; },
createElement: tag => new Element(tag),
createElementNS: (namespace, tag) => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const all = (root, predicate, found = []) => {
if (predicate(root)) found.push(root);
@@ -790,3 +791,145 @@ test('the Filters button shows the active refinement count', async () => {
dom.restore();
}
});
// #249 System S2 Task 5: the scan review's dialog and backup box (D55) generalised for every record-changing review.
const scanModule = import('../../web/js/workspace/activity-scan.mjs');
const bodyShape = dom => byId(dom, 'modal-body').children.map(node => [node.tagName.toLowerCase(), node.className].filter(Boolean).join('.'));
test('backupNotice is the concept .notice with its copy and one Back up now action', async () => {
const { backupNotice, SCAN_BACKUP_COPY } = await dialogsModule;
const dom = fakeDOM();
try {
let calls = 0;
const notice = backupNotice({ copy: SCAN_BACKUP_COPY, onBackup: () => { calls++; } });
assert.equal(notice.className, 'notice');
assert.deepEqual(tags(notice), ['div', 'div']);
assert.deepEqual(tags(notice.children[0]), ['h3', 'p']);
assert.equal(notice.children[0].children[0].textContent, 'Back up the database first (recommended)');
assert.equal(notice.children[0].children[1].textContent,
'Downloads a copy of the prompts database before the scan changes any records.');
assert.equal(notice.children[1].className, 'row-actions');
const action = button(notice, 'Back up now');
action.click();
assert.equal(calls, 1);
assert.throws(() => backupNotice({ copy: { heading: '', text: 'x' }, onBackup() {} }), TypeError);
assert.throws(() => backupNotice({ copy: SCAN_BACKUP_COPY }), TypeError);
} finally {
dom.restore();
}
});
test('openReviewDialog lays out heading, grid, options, sentence, backup box and actions', async () => {
const { openReviewDialog, closeWorkspaceDialog } = await dialogsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const trigger = new dom.Element('button');
dom.body.append(trigger);
const option = new dom.Element('div');
option.className = 'setting-row';
const started = [];
const review = openReviewDialog({ scope, title: 'Review maintenance plan', heading: 'Missing-file links',
grid: [['Scope', '2 stale image links'], ['Media on disk', 'No source files removed']], options: [option],
sentence: 'Cancelling leaves all data untouched.', backup: { copy: { heading: 'Back up first', text: 'A copy.' }, onBackup() {} },
cancelLabel: 'Cancel', primary: { label: 'Apply reviewed operation' }, onStart: () => started.push('start'), trigger });
assert.equal(byId(dom, 'modal-title').textContent, 'Review maintenance plan');
assert.deepEqual(bodyShape(dom), ['h3', 'dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']);
const grid = byId(dom, 'modal-body').children[1];
assert.deepEqual(grid.children.map(entry => [tags(entry), entry.children[0].textContent, entry.children[1].textContent]), [
[['dt', 'dd'], 'Scope', '2 stale image links'], [['dt', 'dd'], 'Media on disk', 'No source files removed']]);
review.setValue('Scope', '3 stale image links');
assert.equal(grid.children[0].children[1].textContent, '3 stale image links');
const actions = byId(dom, 'modal-body').children.at(-1);
assert.deepEqual(actions.children.map(node => [node.textContent, node.className]),
[['Cancel', ''], ['Apply reviewed operation', 'ca-primary']]);
actions.children[1].click();
assert.deepEqual(started, ['start']);
assert.equal(modal(dom).open, false);
assert.equal(document.activeElement, trigger);
// A danger primary (the concept's Restore) has no sentence, heading, options or backup box when none is given.
openReviewDialog({ scope, title: 'Restore preview', grid: [['Integrity', 'Verified']], backup: null,
cancelLabel: 'Cancel', primary: { label: 'Restore', danger: true }, onStart() {} });
assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.dialog-actions']);
assert.equal(button(byId(dom, 'modal-body'), 'Restore').className, 'danger');
button(byId(dom, 'modal-body'), 'Cancel').click();
assert.equal(modal(dom).open, false);
assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [['a']], cancelLabel: 'Cancel', primary: { label: 'Go' },
onStart() {} }), TypeError);
assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [], cancelLabel: 'Cancel', primary: {}, onStart() {} }), TypeError);
} finally {
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
test('review dialog backup box keeps the dialog open and toasts the outcome', async () => {
const { openReviewDialog, closeWorkspaceDialog, SCAN_BACKUP_COPY } = await dialogsModule;
const { downloadDatabaseBackup, BACKUP_DONE, BACKUP_FAILED } = await scanModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
const created = URL.createObjectURL;
const revoked = URL.revokeObjectURL;
URL.createObjectURL = () => 'blob:backup';
URL.revokeObjectURL = () => {};
try {
const toasts = [];
let answer;
const fetchUrl = () => new Promise(resolve => { answer = resolve; });
openReviewDialog({ scope, title: 'Review metadata scan', grid: [['Sources', '2 output folders']],
backup: { copy: SCAN_BACKUP_COPY, onBackup: () => downloadDatabaseBackup({ fetchUrl, notify: text => toasts.push(text) }) },
cancelLabel: 'Cancel', primary: { label: 'Start scan' }, onStart() { throw new Error('not started'); } });
const action = button(byId(dom, 'modal-body'), 'Back up now');
action.click();
assert.equal(action.textContent, 'Backing up…');
assert.equal(action.getAttribute('aria-disabled'), 'true');
// A second press while the copy downloads is ignored.
action.click();
answer({ ok: true, blob: async () => new Blob(['db']), headers: { get: () => 'attachment; filename="prompts_backup_1.db"' } });
await tick();
assert.deepEqual(toasts, [BACKUP_DONE]);
assert.equal(modal(dom).open, true);
assert.equal(action.textContent, 'Back up now');
assert.equal(action.getAttribute('aria-disabled'), null);
action.click();
answer({ ok: false, status: 500 });
await tick();
assert.deepEqual(toasts, [BACKUP_DONE, BACKUP_FAILED]);
assert.equal(modal(dom).open, true);
} finally {
URL.createObjectURL = created;
URL.revokeObjectURL = revoked;
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
test('the scan review is a thin openReviewDialog caller with its scope switch', async () => {
const { openScanReviewDialog, closeWorkspaceDialog } = await dialogsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const starts = [];
openScanReviewDialog({ scope, sources: 2, onBackup() {}, onStart: value => starts.push(value) });
assert.equal(byId(dom, 'modal-title').textContent, 'Review metadata scan');
assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']);
const grid = byId(dom, 'modal-body').children[0];
assert.deepEqual(grid.children.map(entry => entry.textContent), ['Sources2 output folders', 'ScopeNew and unresolved files']);
const allFiles = byId(dom, 'scan-all-files');
allFiles.checked = true;
allFiles.dispatchEvent(new Event('change'));
assert.equal(grid.children[1].children[1].textContent, 'All files');
button(byId(dom, 'modal-body'), 'Start scan').click();
assert.deepEqual(starts, [{ scope: 'all' }]);
} finally {
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
+2
View File
@@ -29,6 +29,7 @@ VECTORS_PATH = (
# Committed record sets stay small; the large bound cases are built at test time.
MAX_VECTOR_CHARS = 20_000
COMMITTED_FUZZ = (0, 60)
COMMITTED_PASSTHROUGH = (0, 40)
SPACES = ["", " ", "\t", "\n", "\x1c", "\x85", " ", "", "
 "]
WORDS = ["fox", "snow", "café", "猫", "\U0001f600", "a,", "detailed fur", ""]
@@ -1380,6 +1381,7 @@ def committed_vectors():
}
cases.update(fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.selection_fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.passthrough_fuzz_cases(*COMMITTED_PASSTHROUGH))
vectors = [
{
"name": name,
+304 -1
View File
@@ -12,8 +12,10 @@ import json
import random
try:
from tests import passthrough_graphs as shapes
from tests.local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import passthrough_graphs as shapes
from local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
@@ -437,10 +439,213 @@ def _parameters():
return cases
def _passthrough():
"""Chains joined through lineage nodes, and the floor for added conditioning."""
cases = {
"hires-fix": shapes.hires_fix(),
"hires-fix-differs": shapes.hires_fix(second_positive=shapes.OTHER),
"hires-fix-unlisted": shapes.hires_fix(filter_class="MysteryImageEffect"),
"hires-fix-generated": shapes.generated_text_chain(),
"flux-three-stage": shapes.flux_three_stage(),
"flux-concat": shapes.flux_three_stage(composite="ConditioningConcat"),
"flux-average": shapes.flux_three_stage(composite="ConditioningAverage"),
"edit-batch": shapes.edit_batch(3),
}
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
cases["flux-without-base"] = graph
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-other-negative"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
graph["10"]["inputs"]["latent_image"] = ["20", 0]
cases["hires-fix-cycle"] = graph
graph = shapes.hires_fix()
graph["17"]["inputs"]["pixels"] = ["18", 0]
graph["18"] = {"class_type": "Reroute", "inputs": {"value": ["16", 0]}}
cases["hires-fix-reroute"] = graph
graph = shapes.hires_fix()
graph["24"] = {"class_type": "SaveImage", "inputs": {"images": ["11", 0]}}
cases["hires-fix-both-saved"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["99", 0]
cases["hires-fix-missing-latent"] = graph
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-negative-unsupported"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-reroute-input"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-encoder-parts"] = graph
cases.update(_passthrough_edges())
cases.update(_runtime_parts())
return cases
def _runtime_parts():
"""Option A: a PromptManager part fed by a run-time node is left out."""
cases = {"trigger-prepend": shapes.hires_fix_with_trigger_words()}
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="masterpiece", append_text=["8", 0])
graph["20"]["inputs"]["latent_image"] = ["4", 0]
cases["trigger-append-single"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
cases["traced-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
cases["unknown-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["indirect-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="", text=["8", 0])
cases["runtime-main-text"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["class_type"] = "PromptManagerPair"
graph["2"]["inputs"]["negative_text"] = shapes.NEGATIVE
graph["10"]["inputs"]["negative"] = ["2", 1]
graph["20"]["inputs"]["negative"] = ["2", 1]
cases["trigger-pair"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["30"] = {"class_type": "PromptSearchList", "inputs": {"search": "x"}}
graph["2"]["inputs"]["append_text"] = ["30", 0]
cases["search-list-append"] = graph
for shape in ("text-encode", "chained", "concat"):
cases[f"through-{shape}"] = shapes.trigger_words_through(shape)
cases["branches-with-and-without"] = (
shapes.branches_with_and_without_trigger_words()
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
cases["stage-with-and-without"] = graph
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["join-multi-flag"] = graph
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
cases["flux-other-full-prompt"] = graph
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
cases["flux-trigger"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-chained"] = graph
return {f"runtime-{name}": graph for name, graph in cases.items()}
def _passthrough_edges():
"""Lineage and floor edges: branching refiners, other slots, cycles, guiders."""
cases = {}
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.4)
graph["31"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-two-refiners"] = graph
graph = shapes.hires_fix()
graph["10"]["class_type"] = "SamplerCustom"
graph["11"]["inputs"]["samples"] = ["10", 1]
cases["hires-fix-denoised-slot"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["30"]["inputs"]["latent_image"] = ["32", 0]
cases["hires-fix-self-latent"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["32", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["32", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["33"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-lineage-cycle"] = graph
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
cases["flux-base-stages-differ"] = graph
graph = shapes.flux_three_stage()
del graph["60"]["inputs"]["negative"]
cases["flux-no-last-negative"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
graph["40"]["inputs"]["positive"] = ["39", 0]
graph["50"]["inputs"]["positive"] = ["39", 0]
cases["flux-unreadable-base"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "MysteryUpscaler",
"inputs": {"image": ["51", 0], "model": ["1", 0], "positive": ["34", 0]},
}
graph["52"]["inputs"]["images"] = ["39", 0]
cases["flux-unsupported-ancestry"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["62", 0],
"guider": ["63", 0],
"sampler": ["64", 0],
"sigmas": ["65", 0],
"latent_image": ["66", 0],
},
}
graph["62"] = {"class_type": "RandomNoise", "inputs": {"noise_seed": 1}}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"model": ["1", 0], "positive": ["35", 0], "negative": ["33", 0]},
}
graph["64"] = {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}}
graph["65"] = {"class_type": "BasicScheduler", "inputs": {"steps": 4}}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["52", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-last"] = graph
return {f"edge-{name}": graph for name, graph in cases.items()}
def selection_cases():
"""Named record sets for every Task 9 path, each a ``read_inspection_file`` dict."""
cases = {}
for group in (_samplers(), _previews(), _detailers()):
for group in (_samplers(), _previews(), _detailers(), _passthrough()):
for name, graph in group.items():
cases[f"select-{name}"] = {"prompt": json.dumps(graph, ensure_ascii=False)}
cases.update(_parameters())
@@ -572,3 +777,101 @@ def selection_fuzz_cases(seed, count):
records.update(builder.records)
cases[f"select-fuzz-{seed}-{index}"] = records
return cases
_LINEAGE_HOPS = (
("VAEDecode", "samples"),
("VAEDecodeTiled", "samples"),
("Image Filter", "images"),
("ImageUpscaleWithModel", "image"),
("ImageScale", "image"),
("ImageScaleBy", "image"),
("VAEEncode", "pixels"),
("LatentUpscaleBy", "samples"),
("Reroute", "value"),
("MysteryImageEffect", "image"),
)
def passthrough_fuzz_cases(seed, count):
"""Seeded sampler chains joined by random lineage (and unlisted) hops.
Each later stage reads the earlier one's output through zero to four hops,
with a shared, a different, an added-to (combine, concat, average) or a
run-time positive, and a shared or different negative, under a saver or a
preview. The server is the oracle; nothing states the expected answer.
"""
rng = random.Random(seed)
cases = {}
for index in range(count):
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": shapes.prompt_manager(rng.choice([shapes.POSITIVE, "fox"])),
"3": shapes.encode(shapes.NEGATIVE),
"4": {"class_type": "EmptyLatentBatch", "inputs": {"batch_size": 2}},
"5": shapes.encode(shapes.STYLE),
"6": shapes.encode(shapes.OTHER),
}
previous, next_id = ["4", 0], 10
for stage in range(rng.randint(2, 3)):
positive = ["2", 0]
choice = rng.randrange(8)
if stage and choice == 1:
positive = ["6", 0]
elif stage and choice in (2, 3, 4):
name = (
"ConditioningCombine",
"ConditioningConcat",
"ConditioningAverage",
)[choice - 2]
names = shapes._COMPOSITE_INPUTS[name]
parts = [["2", 0], rng.choice([["5", 0], ["6", 0]])]
rng.shuffle(parts)
graph[str(next_id)] = {
"class_type": name,
"inputs": dict(zip(names, parts)),
}
positive, next_id = [str(next_id), 0], next_id + 1
elif stage and choice == 5:
graph[str(next_id)] = {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["2", 0]},
}
positive, next_id = [str(next_id), 0], next_id + 1
negative = ["6", 0] if stage and rng.random() < 0.15 else ["3", 0]
latent = previous
if stage:
image = previous
for _ in range(rng.randrange(5)):
name, key = rng.choice(_LINEAGE_HOPS)
graph[str(next_id)] = {"class_type": name, "inputs": {key: image}}
image, next_id = [str(next_id), 0], next_id + 1
latent = image
sampler = shapes.ksampler(positive, negative, latent)
if stage and rng.random() < 0.2:
sampler = {
"class_type": "UltimateSDUpscale",
"inputs": {
"image": latent,
"model": ["1", 0],
"positive": positive,
"negative": negative,
},
}
graph[str(next_id)] = sampler
previous, next_id = [str(next_id), 0], next_id + 1
if rng.random() < 0.3:
saver = rng.choice(["SaveImage", "PreviewImage"])
graph[str(next_id)] = {
"class_type": saver,
"inputs": {"images": previous},
}
next_id += 1
graph[str(next_id)] = {
"class_type": "SaveImage",
"inputs": {"images": previous},
}
cases[f"select-passthrough-fuzz-{seed}-{index}"] = {
"prompt": json.dumps(graph, ensure_ascii=False)
}
return cases
+332
View File
@@ -0,0 +1,332 @@
"""Synthetic multi-stage graphs for the pass-through lineage rule.
Each builder reproduces one graph shape with neutral texts and generic names:
a hires-fix or refiner chain (sampler, decode, image filter, model upscale,
scale, encode, sampler), a three-stage Flux run whose last stage combines the
base prompt with a style trigger, an image-edit batch of samplers with distinct
prompts, and a chain whose prompt is generated at run time. They are shared by
the resolver tests and the browser-port parity cases; no user file is involved.
"""
import copy
POSITIVE = "a lighthouse on a cliff, golden hour"
NEGATIVE = "blurry, lowres"
OTHER = "a red bicycle in the rain"
STYLE = "film grain, analog photo"
def encode(text):
return {"class_type": "CLIPTextEncode", "inputs": {"text": text, "clip": ["1", 1]}}
def prompt_manager(text, prepend=""):
return {
"class_type": "PromptManager",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
"clip": ["1", 1],
},
}
def ksampler(positive, negative, latent, denoise=1.0):
return {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"steps": 20,
"cfg": 7.0,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": denoise,
"model": ["1", 0],
"positive": positive,
"negative": negative,
"latent_image": latent,
},
}
def hires_fix(second_positive=None, filter_class="Image Filter"):
"""Two KSamplers joined in image space, like a hires-fix or refiner run.
Sampler 10 runs on a custom batch latent; its output goes through a tiled
decode, an interactive image filter, a model upscale and a scale (sized by a
resolution node reading the filtered image), then is encoded again for
sampler 20 (denoise 0.25), which is decoded and saved.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": prompt_manager(POSITIVE),
"3": encode(NEGATIVE),
"4": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1024, "height": 512, "batch_size": 4},
},
"10": ksampler(["2", 0], ["3", 0], ["4", 0]),
"11": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["10", 0], "vae": ["1", 2]},
},
"12": {
"class_type": filter_class,
"inputs": {"timeout": 600, "ontimeout": "send none", "images": ["11", 0]},
},
"13": {"class_type": "UpscaleModelLoader", "inputs": {"model_name": "x4"}},
"14": {
"class_type": "ImageUpscaleWithModel",
"inputs": {"upscale_model": ["13", 0], "image": ["12", 0]},
},
"15": {
"class_type": "ResolutionCalculator",
"inputs": {"scale_factor": 1.5, "image": ["12", 0]},
},
"16": {
"class_type": "ImageScale",
"inputs": {
"upscale_method": "nearest-exact",
"width": ["15", 0],
"height": ["15", 1],
"crop": "disabled",
"image": ["14", 0],
},
},
"17": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["16", 0], "vae": ["1", 2]},
},
"20": ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.25),
"21": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["20", 0], "vae": ["1", 2]},
},
"22": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "hires", "images": ["21", 0]},
},
"23": {"class_type": "PlaySound|pysssss", "inputs": {"any": ["21", 0]}},
}
if second_positive is not None:
graph["5"] = encode(second_positive)
graph["20"]["inputs"]["positive"] = ["5", 0]
return graph
_COMPOSITE_INPUTS = {
"ConditioningCombine": ("conditioning_1", "conditioning_2"),
"ConditioningConcat": ("conditioning_to", "conditioning_from"),
"ConditioningAverage": ("conditioning_to", "conditioning_from"),
}
def flux_three_stage(combined=(("30", 0), ("31", 0)), composite="ConditioningCombine"):
"""FluxGuidance base, a refiner through an image filter, then a tiled upscale.
Samplers 40 and 50 share FluxGuidance(PromptManager 30) and a zeroed empty
negative. Sampler 50 refines 40's filtered image (denoise 0.25). The final
UltimateSDUpscale 60 reads 50's image, scaled down, and a positive that is
``composite`` of ``combined`` (by default the base PromptManager and a
style-trigger encoder 31).
"""
names = _COMPOSITE_INPUTS[composite]
graph = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux"}},
"2": {"class_type": "DualCLIPLoader", "inputs": {"type": "flux"}},
"3": {"class_type": "VAELoader", "inputs": {"vae_name": "ae"}},
"30": prompt_manager(POSITIVE),
"31": encode(STYLE),
"32": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "", "clip": ["2", 0]},
},
"33": {
"class_type": "ConditioningZeroOut",
"inputs": {"conditioning": ["32", 0]},
},
"34": {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["30", 0]},
},
"35": {
"class_type": composite,
"inputs": dict(zip(names, map(list, combined))),
},
"36": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1080, "height": 1920, "batch_size": 4},
},
"40": ksampler(["34", 0], ["33", 0], ["36", 0]),
"41": {
"class_type": "VAEDecode",
"inputs": {"samples": ["40", 0], "vae": ["3", 0]},
},
"42": {
"class_type": "Image Filter",
"inputs": {"timeout": 600, "images": ["41", 0]},
},
"43": {"class_type": "PreviewImage", "inputs": {"images": ["42", 0]}},
"44": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["42", 0], "vae": ["3", 0]},
},
"50": ksampler(["34", 0], ["33", 0], ["44", 0], denoise=0.25),
"51": {
"class_type": "VAEDecode",
"inputs": {"samples": ["50", 0], "vae": ["3", 0]},
},
"52": {
"class_type": "ImageScaleDownBy",
"inputs": {"scale_by": 0.75, "images": ["51", 0]},
},
"53": {"class_type": "PreviewImage", "inputs": {"images": ["51", 0]}},
"54": {
"class_type": "Image Comparer (rgthree)",
"inputs": {"image_a": ["51", 0], "image_b": ["42", 0]},
},
"60": {
"class_type": "UltimateSDUpscale",
"inputs": {
"upscale_by": 2.0,
"seed": 7,
"steps": 4,
"denoise": 0.35,
"image": ["52", 0],
"model": ["1", 0],
"positive": ["35", 0],
"negative": ["33", 0],
"vae": ["3", 0],
},
},
"61": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "flux", "images": ["60", 0]},
},
}
return graph
def edit_batch(count=3):
"""Image-edit samplers with distinct prompts, all saved with one prefix."""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "LoadImage", "inputs": {"image": "ref.png"}},
"3": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["2", 0], "vae": ["1", 2]},
},
"4": {"class_type": "PrimitiveString", "inputs": {"value": "edit"}},
}
for index in range(count):
base = 100 + index * 10
graph[str(base)] = {
"class_type": "TextEncodeQwenImageEdit",
"inputs": {
"prompt": f"edit number {index}: change the pose",
"clip": ["1", 1],
},
}
graph[str(base + 1)] = ksampler([str(base), 0], [str(base), 0], ["3", 0])
graph[str(base + 2)] = {
"class_type": "VAEDecode",
"inputs": {"samples": [str(base + 1), 0], "vae": ["1", 2]},
}
graph[str(base + 3)] = {
"class_type": "SaveImage",
"inputs": {"filename_prefix": ["4", 0], "images": [str(base + 2), 0]},
}
return graph
def trigger_toggle(words="trig9word"):
"""LoraManager's TriggerWord Toggle: its output is decided at run time."""
return {
"class_type": "TriggerWord Toggle (LoraManager)",
"inputs": {
"group_mode": True,
"default_active": True,
"toggle_trigger_words": {"__value__": [{"text": words, "active": True}]},
"orinalMessage": words,
"trigger_words": ["7", 2],
},
}
def hires_fix_with_trigger_words():
"""``hires_fix`` whose PromptManager prepends LoRA trigger words (node 8)."""
graph = hires_fix()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["1", 1], "text": "<lora:x:1>"},
}
graph["8"] = trigger_toggle()
graph["2"]["inputs"]["prepend_text"] = ["8", 0]
return graph
def prompt_manager_text(text, prepend=""):
"""PromptManagerText: a STRING PromptManager (prepend/main/append)."""
return {
"class_type": "PromptManagerText",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
},
}
def trigger_words_through(shape):
"""``hires_fix_with_trigger_words`` with nodes between the toggle and sampler.
``text-encode``: PromptManagerText (prepend <- toggle) -> CLIPTextEncode.
``chained``: a PromptManager whose ``text`` reads that PromptManagerText.
``concat``: PromptManagerText -> StringConcatenate(..., "extra") -> encoder.
"""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager_text(POSITIVE, ["8", 0])
if shape == "chained":
graph["2"]["inputs"].update(prepend_text="", text=["9", 0])
elif shape == "text-encode":
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["9", 0]}}
else:
graph["19"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": ["9", 0], "string_b": "extra", "delimiter": ", "},
}
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["19", 0]}}
return graph
def branches_with_and_without_trigger_words():
"""Two saved branches: one PromptManager prepends trigger words, one does not."""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager(POSITIVE)
graph["20"]["inputs"].update(positive=["9", 0], latent_image=["4", 0])
return graph
def generated_text_chain():
"""``hires_fix`` whose positive text comes from a run-time text generator."""
graph = hires_fix()
graph["2"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["6", 0], "clip": ["1", 1]},
}
graph["6"] = {
"class_type": "TextGenerate",
"inputs": {"prompt": "write a prompt", "model": ["1", 0]},
}
return graph
def copied(graph):
return copy.deepcopy(graph)
+11 -2
View File
@@ -30,6 +30,11 @@ FILES = "media_annotation_files"
BYTES = "media_annotation_bytes"
IMAGE_TAGS = "media_image_tags"
TABLES = (METADATA, FILES, BYTES, IMAGE_TAGS)
REVIEW_TABLES = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
@@ -202,10 +207,14 @@ class AnnotationSchemaTests(unittest.TestCase):
)
def drop_group(self, path):
"""Remove the group from a closed database file (an old backup)."""
"""Remove the group from a closed database file (an old backup).
The review queue group cites annotation bytes and predates nothing, so
an old backup lacks it as well.
"""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in reversed(TABLES):
for table in REVIEW_TABLES + tuple(reversed(TABLES)):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
+45 -3
View File
@@ -52,6 +52,7 @@ class AdminAPITestCase(AioHTTPTestCase):
self.outside_dir = Path(self.tmpdir) / "outside"
self.outside_dir.mkdir()
self.addAsyncCleanup(self._drain_and_remove)
self.config_path = os.path.join(self.tmpdir, "config.json")
self._orig_env = {
key: os.environ.pop(key, None) for key in (CONFIG_PATH_ENV, EXTRA_ROOTS_ENV)
@@ -89,6 +90,26 @@ class AdminAPITestCase(AioHTTPTestCase):
sys.modules.pop("folder_paths", None)
else:
sys.modules["folder_paths"] = self._orig_folder_paths
async def _drain_and_remove(self):
"""The last cleanup: no job outlives the temporary database.
Registered first, so it runs after every cleanup a test added (which
may release its own gated jobs). A job still running then, such as a
scan a failed assertion never awaited, is cancelled and awaited
before the database and the tree are removed.
"""
api = getattr(self, "api", None)
jobs = api._jobs.list() if api is not None else []
tasks = [
job.task for job in jobs if job.task is not None and not job.task.done()
]
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
if api is not None:
# A scan of an ambiguous file captures it for Needs review.
api.close_annotation_captures()
PromptModel.close_all_instances() # restore may have swapped api.db
assert_closed(self, self.tmpdir)
shutil.rmtree(self.tmpdir, ignore_errors=True)
@@ -868,6 +889,23 @@ class TestDeleteDuplicatesEndpoint(AdminAPITestCase):
self.assertFalse(target.exists())
self.assertTrue(thumb_dir.exists())
async def test_deleted_files_leave_the_cached_gallery_listing(self):
# The listing is cached for 30 s; a delete must not leave the gallery
# listing (and counting) a file that is gone.
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
for name in ("keep.png", "gone.png"):
(self.output_dir / name).write_bytes(b"same bytes")
async def listed():
resp = await self.client.get("/prompt_manager/images/output")
data = await resp.json()
return data["total"], sorted(i["filename"] for i in data["images"])
self.assertEqual(await listed(), (2, ["gone.png", "keep.png"]))
resp = await self._post({"image_paths": ["gone.png"]})
self.assertEqual((await resp.json())["deleted_count"], 1)
self.assertEqual(await listed(), (1, ["keep.png"]))
def test_unexpected_error_per_file_is_reported_without_path(self):
self.api._delete_one_duplicate = lambda *a: (_ for _ in ()).throw(
PermissionError(13, "Permission denied", os.path.join(self.tmpdir, "x.png"))
@@ -2204,7 +2242,9 @@ class TestActivityJobs(AdminAPITestCase):
for name in names:
(self.output_dir / name).write_bytes(b"not an image")
def fake_ingest(media_file, metadata, detailed=False):
def fake_ingest(
media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
key = Path(media_file).relative_to(self.output_dir).as_posix()
outcome = names[key]
if isinstance(outcome, Exception):
@@ -2333,8 +2373,10 @@ class TestActivityJobs(AdminAPITestCase):
first_done, release = threading.Event(), threading.Event()
original = self.api._ingest_scan_batch_sync
def gated(batch_results, roots=(), outcomes=None):
counts = original(batch_results, roots=roots, outcomes=outcomes)
def gated(batch_results, roots=(), outcomes=None, review=None):
counts = original(
batch_results, roots=roots, outcomes=outcomes, review=review
)
first_done.set()
release.wait(5)
return counts
+9 -1
View File
@@ -1045,7 +1045,15 @@ class TestFailures(AnnotationAPITestCase):
await self.api.cleanup_annotation_captures(None)
self.assertFalse(os.path.exists(directory))
self.assertNotIn(identity, observers._restore_observers)
# A later observation starts a fresh store.
# The shutdown cleanup is final: a later observation (or a monitor
# recording still in flight) creates no new store or directory.
status, body = await self.observe({"target": self.target("sub/a.png")})
self.assertEqual((status, body["code"]), (500, "capture-error"), body)
self.assertIsNone(self.api._captures)
self.assertNotIn(identity, observers._restore_observers)
# An explicit, non-final close still lets a later observation start
# a fresh store (the tests' own store swaps rely on it).
self.api._captures_closed = False
await self.observe_ok()
self.assertNotEqual(self.api._captures.directory, directory)
+13
View File
@@ -471,6 +471,19 @@ class TestUiRoutes(CoreRoutesTestCase):
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
async def test_review_routes_are_registered(self):
table = {
(route.method, route.resource.canonical)
for route in self.app.router.routes()
if route.resource is not None
}
for entry in (
("GET", "/prompt_manager/review/items"),
("GET", "/prompt_manager/review/items/{item_id}"),
("POST", "/prompt_manager/review/items/{item_id}/decision"),
):
self.assertIn(entry, table)
class TestStaticRoutes(CoreRoutesTestCase):
File diff suppressed because it is too large Load Diff
+234
View File
@@ -0,0 +1,234 @@
"""A slow page-workflow reset reports where the server waited.
CI run 39904288262 had one ``/__test__/reset`` wait 16 s with nothing in the
server's stderr. tests/browser/slow_reset_watch.py reports any reset past
SLOW_RESET_SECONDS with every thread's stack, taken while it still waits.
These tests hold a real SQLite write lock behind a real test server's reset.
"""
import glob
import io
import os
import sqlite3
import subprocess
import sys
import tempfile
import threading
import time
import unittest
import urllib.request
from pathlib import Path
from tests.browser import slow_reset_watch
from tests.browser.slow_reset_watch import REPORT_PREFIX, ResetWatch, is_reset_path
REPO_ROOT = Path(__file__).resolve().parent.parent
SERVER_TIMEOUT_S = 60
HOLD_SECONDS = slow_reset_watch.SLOW_RESET_SECONDS + 4
# The reset may take a little longer than the hold, never the 30 s busy timeout.
RESET_MARGIN_SECONDS = 10
# While the reset waits, the server still answers other requests this fast.
# Reading the waiting connection from the report's thread froze it (the GIL is
# held while SQLite's connection mutex is awaited) until the lock was released.
RESPONSIVE_SECONDS = 2.0
PROBE_INTERVAL_SECONDS = 0.1
def blocked_here(gate: threading.Event) -> None:
gate.wait()
class TestResetWatch(unittest.TestCase):
def test_reset_paths_are_the_test_reset_routes(self):
self.assertTrue(is_reset_path("/__test__/reset"))
self.assertTrue(is_reset_path("/__test__/tagging/reset"))
self.assertFalse(is_reset_path("/prompt_manager/reset"))
self.assertFalse(is_reset_path("/__test__/reset/extra"))
def test_a_stalled_reset_reports_state_and_stacks_while_it_waits(self):
stream = io.StringIO()
reported = threading.Event()
class Stream(io.StringIO):
def flush(self):
reported.set()
stream = Stream()
gate = threading.Event()
worker = threading.Thread(target=blocked_here, args=(gate,), name="held-up")
worker.start()
self.addCleanup(worker.join)
self.addCleanup(gate.set)
watch = ResetWatch(
"POST /__test__/reset",
lambda: ["database: 1 connections"],
threshold=0.05,
stream=stream,
).start()
self.assertTrue(reported.wait(SERVER_TIMEOUT_S))
report = stream.getvalue().splitlines()
self.assertTrue(all(line.startswith(REPORT_PREFIX) for line in report))
text = "\n".join(report)
self.assertIn("POST /__test__/reset still running after", text)
self.assertIn("database: 1 connections", text)
self.assertIn(f"thread 0x{worker.ident:016x} is held-up", text)
self.assertIn("in blocked_here", text)
watch.record_call("seed", 0.25, 1.5)
watch.finish("answered 200")
self.assertIn("POST /__test__/reset answered 200 after", stream.getvalue())
self.assertIn(
"executor call seed: queued 0.250s, ran 1.500s", stream.getvalue()
)
def test_a_fast_reset_writes_nothing_and_stops_its_timer(self):
stream = io.StringIO()
watch = ResetWatch("reset", lambda: [], threshold=60, stream=stream).start()
watch.finish("answered 200")
self.assertTrue(watch._timer.finished.is_set())
self.assertEqual(stream.getvalue(), "")
def test_a_failing_state_probe_still_reports_the_stacks(self):
def broken():
raise RuntimeError("no state")
reported = threading.Event()
class Stream(io.StringIO):
def flush(self):
reported.set()
stream = Stream()
ResetWatch("reset", broken, threshold=0.01, stream=stream).start()
self.assertTrue(reported.wait(SERVER_TIMEOUT_S))
self.assertIn("state unavailable: RuntimeError('no state')", stream.getvalue())
self.assertIn("most recent call first", stream.getvalue())
class TestSlowResetOnTheServer(unittest.TestCase):
"""A reset behind another process's write lock, on tests/browser/server.py."""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
env = {**os.environ, "TMPDIR": self.tmp.name}
env.update(TEMP=self.tmp.name, TMP=self.tmp.name)
self.stderr = tempfile.TemporaryFile()
self.addCleanup(self.stderr.close)
process = subprocess.Popen(
[sys.executable, "-m", "tests.browser.server", "--port", "0"],
cwd=REPO_ROOT,
env=env,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=self.stderr,
)
def stop():
if process.poll() is None:
process.kill()
process.wait(timeout=SERVER_TIMEOUT_S)
for stream in (process.stdin, process.stdout):
stream.close()
self.addCleanup(stop)
line = process.stdout.readline().decode()
self.assertTrue(line.startswith("PM_TEST_SERVER_PORT="), line)
self.base = f"http://127.0.0.1:{int(line.split('=', 1)[1])}"
deadline = time.monotonic() + SERVER_TIMEOUT_S
while True:
try:
with urllib.request.urlopen( # nosec B310
self.base + "/prompt_manager/test", timeout=1
):
break
except OSError:
self.assertIsNone(process.poll(), "the server exited early")
self.assertLess(time.monotonic(), deadline, "never ready")
time.sleep(0.1)
[self.db_path] = glob.glob(
os.path.join(self.tmp.name, "pm-browser-*", "browser.db")
)
def reset(self) -> tuple[int, float]:
request = urllib.request.Request(
self.base + "/__test__/reset",
data=b"{}",
headers={"Content-Type": "application/json"},
method="POST",
)
start = time.monotonic()
with urllib.request.urlopen( # nosec B310
request, timeout=SERVER_TIMEOUT_S
) as response:
return response.status, time.monotonic() - start
def server_stderr(self) -> str:
self.stderr.seek(0)
return self.stderr.read().decode("utf-8", "replace")
def test_a_reset_held_behind_a_write_lock_names_the_wait_and_ends_with_it(self):
self.assertEqual(self.reset()[0], 200, self.server_stderr())
self.assertNotIn(REPORT_PREFIX, self.server_stderr())
held = threading.Event()
def hold_write_lock():
holder = sqlite3.connect(self.db_path, timeout=SERVER_TIMEOUT_S)
try:
holder.execute("BEGIN IMMEDIATE")
held.set()
time.sleep(HOLD_SECONDS)
holder.rollback()
finally:
holder.close()
lock = threading.Thread(target=hold_write_lock)
lock.start()
self.addCleanup(lock.join)
self.assertTrue(held.wait(SERVER_TIMEOUT_S))
outcome = []
resetting = threading.Thread(target=lambda: outcome.append(self.reset()))
resetting.start()
self.addCleanup(resetting.join)
slowest = 0.0
while resetting.is_alive():
start = time.monotonic()
with urllib.request.urlopen( # nosec B310
self.base + "/prompt_manager/test", timeout=SERVER_TIMEOUT_S
):
slowest = max(slowest, time.monotonic() - start)
time.sleep(PROBE_INTERVAL_SECONDS)
resetting.join()
lock.join()
self.assertEqual(len(outcome), 1, self.server_stderr())
[(status, elapsed)] = outcome
self.assertLess(slowest, RESPONSIVE_SECONDS)
self.assertEqual(status, 200)
self.assertGreaterEqual(elapsed, HOLD_SECONDS - 0.5)
self.assertLess(elapsed, HOLD_SECONDS + RESET_MARGIN_SECONDS)
report = [
line[len(REPORT_PREFIX) :]
for line in self.server_stderr().splitlines()
if line.startswith(REPORT_PREFIX)
]
text = "\n".join(report)
self.assertIn("POST /__test__/reset still running after", text)
self.assertIn("default executor:", text)
if sys.platform.startswith("linux"):
# SQLite's busy handler sleeps between its retries.
self.assertRegex(
text, r"kernel: asyncio_\d+ \(0x[0-9a-f]{16}\) state \w, waiting in "
)
# The plain reset's seed() waits on its first DELETE.
self.assertRegex(text, r"line \d+ in seed\n")
self.assertIn("POST /__test__/reset answered 200 after", text)
self.assertRegex(
text,
r"executor call create_app.<locals>.seed: queued \d+\.\d+s, ran \d+\.\d+s",
)
if __name__ == "__main__":
unittest.main()
+57
View File
@@ -68,6 +68,55 @@ class CardSeed(unittest.TestCase):
self.assertIn(expected, (111, 222))
self.assertEqual(info_seed(info), str(expected))
def test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed(self):
# The inspector shows the named final stage's seed (D19, pass-through
# chains); the parser shows another stage's. Never two seeds: none.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {
"class_type": "FaceDetailer",
"inputs": {
"seed": 333,
"image": ["20", 0],
"model": ["4", 0],
"positive": ["6", 0],
"negative": ["7", 0],
},
}
info = graph(seeded(111), extra=extra)
self.assertEqual(extract_from_records(info["prompt"], None)["seed"], 111)
self.assertIsNone(info_seed(info))
# The same chain on one seed keeps it; so does a refiner over a decode.
extra["21"]["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), extra=extra)), "111")
refiner = seeded(222, latent_image=["22", 0])
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
del extra["21"]
self.assertIsNone(info_seed(graph(seeded(111), refiner, extra=extra)))
refiner["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_a_seed_that_agrees_with_the_named_stage_is_kept(self):
# Review I3: a saved sampler (seed 7) beside a preview-only sampler
# (seed 5): the parser and the inspector both show 7, as on main.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {"class_type": "SaveImage", "inputs": {"images": ["20", 0]}}
extra["30"] = {"class_type": "VAEDecode", "inputs": {"samples": ["11", 0]}}
extra["31"] = {"class_type": "PreviewImage", "inputs": {"images": ["30", 0]}}
info = graph(seeded(7), seeded(5), extra=extra)
self.assertEqual(info_seed(info), "7")
def test_a_named_ambiguous_chain_keeps_the_parser_seed(self):
# Review M2: multistage-differs names a sampler, but its facts merge,
# and the inspector fills the parser's seed: the card keeps it.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["8"] = encoder("another scene")
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
refiner = ksampler(("8", 0), ("7", 0), seed=222, latent_image=["22", 0])
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_card_seed_linked_input_follows_shared_parser(self):
primitive = {"class_type": "PrimitiveNode", "inputs": {"value": 555}}
info = graph(seeded(["20", 0]), extra={"20": primitive})
@@ -207,6 +256,7 @@ class CardSeed(unittest.TestCase):
self.assertIsNone(parsed_positive(unparsable))
def test_every_shared_case_reads_the_projected_graphs(self):
withheld = []
for name, case in CASES.items():
with self.subTest(case=name):
info = {k: v for k, v in case.items() if v is not None}
@@ -225,11 +275,18 @@ class CardSeed(unittest.TestCase):
except GenerationParamsError:
self.assertIsNone(params)
continue
if name == "shared-prompt-pipeline":
# The inspector names the detailer (seed 333); the parser
# reads the base sampler (111): the card shows none.
self.assertEqual(expected["seed"], 111)
expected = {**expected, "seed": None}
withheld.append(name)
self.assertEqual(params, expected)
# Prompt text is the same as from a plain parse of the records.
if len(values) == len(info):
raw = extract_from_records(case["prompt"], case["workflow"])
self.assertEqual(params["positivePrompt"], raw["positivePrompt"])
self.assertEqual(withheld, ["shared-prompt-pipeline"])
if __name__ == "__main__":
+84 -5
View File
@@ -24,6 +24,9 @@ from database.operations import PromptDatabase
from tests.open_handles import assert_closed
from utils.hashing import generate_prompt_hash
# A bound on thread hand-offs that only a hung test reaches (never a pacing timer).
THREAD_WAIT_SECONDS = 120
def _count_prompts(path):
"""Row count read through a brand-new connection (closed before return)."""
@@ -350,7 +353,13 @@ class TestRestoreFromFile(BackupTestCase):
def worker():
self.db.search_prompts(text="old") # opens this thread's connection
ready.set()
proceed.wait(5)
# Wait for the restore itself, not for a timer: a stalled Windows
# runner once took longer than 5 s to restore, and the worker then
# read the old database before the restore had run.
if not proceed.wait(THREAD_WAIT_SECONDS):
results["error"] = "the restore did not finish"
self.db.close()
return
try:
results["texts"] = [p["text"] for p in self.db.search_prompts()]
except Exception as exc: # noqa: BLE001 - reported through results
@@ -360,12 +369,15 @@ class TestRestoreFromFile(BackupTestCase):
thread = threading.Thread(target=worker)
thread.start()
ready.wait(5)
self.assertTrue(ready.wait(THREAD_WAIT_SECONDS))
self.db.model.restore_from_file(src)
proceed.set()
thread.join(5)
try:
self.db.model.restore_from_file(src)
finally:
proceed.set()
thread.join(THREAD_WAIT_SECONDS)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
self.assertEqual(results["texts"], ["restored"])
self.assertEqual([p["text"] for p in self.db.search_prompts()], ["restored"])
@@ -454,6 +466,9 @@ class TestRestoreFromFile(BackupTestCase):
def test_restore_old_backup_without_annotations(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -490,6 +505,70 @@ class TestRestoreFromFile(BackupTestCase):
0,
)
def test_restore_annotations_era_backup_without_review_queue(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
src = self._make_source_db(["predates the review queue"])
with closing(sqlite3.connect(src)) as conn:
for table in tables:
conn.execute(f"DROP TABLE {table}")
conn.commit()
self.assertTrue(PromptModel.verify_database_file(src)[0])
ok, reason = PromptModel.verify_database_file(src, require_library_state=True)
self.assertFalse(ok)
self.assertEqual(reason, "Missing review queue schema")
self.db.model.restore_from_file(src)
self.assertEqual(_texts(self.path), ["predates the review queue"])
with closing(sqlite3.connect(self.path)) as conn:
self.assertEqual(
conn.execute("SELECT * FROM media_review_metadata").fetchall(),
[(1, 1, 0)],
)
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[0],
0,
)
ok, reason = PromptModel.verify_database_file(
self.path, require_library_state=True
)
self.assertTrue(ok, reason)
with closing(sqlite3.connect(src)) as conn: # the source is untouched
self.assertEqual(
conn.execute(
"SELECT COUNT(*) FROM sqlite_master WHERE name LIKE 'media_review%'"
).fetchone()[0],
0,
)
def test_restore_rejects_review_queue_without_annotations(self):
src = self._make_source_db(["review without annotations"])
with closing(sqlite3.connect(src)) as conn:
conn.execute("PRAGMA foreign_keys=OFF")
for table in (
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
"media_annotation_metadata",
):
conn.execute(f"DROP TABLE {table}")
conn.commit()
before = Path(src).read_bytes()
live_before = _texts(self.path)
ok, reason = PromptModel.verify_database_file(src)
self.assertFalse(ok)
self.assertIn("image annotation", reason)
with self.assertRaises(ValueError):
self.db.model.restore_from_file(src)
self.assertEqual(Path(src).read_bytes(), before)
self.assertEqual(_texts(self.path), live_before)
def test_restore_preserves_favorites_and_closes_handles(self):
self.db.set_image_favorite("output", "live.png")
with tempfile.TemporaryDirectory() as directory:
+614
View File
@@ -0,0 +1,614 @@
"""The read-only health check (System › Overview and Maintenance).
``run_health_checks`` reads one snapshot on its own ``mode=ro`` connection:
SQLite's integrity and foreign-key checks, stored prompt hashes against
``utils.hashing``, duplicate pair rows, the legacy consistency queries, image
links whose file is gone (only under readable sources) and the Needs-review
count when one is supplied. It never writes, and it never turns a check it
could not run into a count.
"""
import hashlib
import ntpath
import os
import sqlite3
import sys
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from database import health # noqa: E402
from database.operations import PromptDatabase # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
try:
from tests.open_handles import assert_closed # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed # noqa: E402
CHECK_IDS = [
"integrity",
"foreign_keys",
"canonical_hashes",
"pair_uniqueness",
"missing_files",
"ambiguous_metadata",
"consistency",
]
def _digest(path):
if not os.path.exists(path):
return None
with open(path, "rb") as handle:
return hashlib.sha256(handle.read()).hexdigest()
class HealthTestCase(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.root = Path(os.path.realpath(self.temp.name))
self.output = self.root / "output"
self.output.mkdir()
self.db_path = str(self.root / "prompts.db")
self.db = PromptDatabase(self.db_path)
def tearDown(self):
self.db.close_all()
assert_closed(self, self.root)
self.temp.cleanup()
def save(self, text, **metadata):
return self.db.save_prompt(
text, prompt_hash=generate_prompt_hash(text), **metadata
)
def image(self, prompt_id, name, *, exists=True, folder=None):
folder = Path(folder or self.output)
path = folder / name
if exists:
path.write_bytes(b"png")
return self.db.link_image_to_prompt(prompt_id, str(path))
def run_checks(self, roots=None, **kwargs):
roots = [str(self.output)] if roots is None else roots
return self.db.run_health_checks(roots, **kwargs)
@staticmethod
def check(report, check_id):
return next(c for c in report["checks"] if c["id"] == check_id)
def raw(self):
"""A plain connection for planting damage the app itself never writes."""
conn = sqlite3.connect(self.db_path)
conn.execute("PRAGMA foreign_keys=OFF")
return conn
class TestHealthReport(HealthTestCase):
def test_clean_library_passes_except_unavailable_review(self):
prompt = self.save("A clean prompt")
self.db.save_prompt_pair("Cat", "Blur")
self.image(prompt, "a.png")
report = self.run_checks(review_count=0)
self.assertTrue(report["success"])
self.assertEqual([c["id"] for c in report["checks"]], CHECK_IDS)
self.assertEqual(
[c["label"] for c in report["checks"]],
[
"Integrity check",
"Foreign keys & pair references",
"Canonical hashes",
"Pair uniqueness",
"Missing files",
"Ambiguous metadata",
"Consistency",
],
)
for check in report["checks"]:
self.assertEqual(check["status"], "passed", check)
self.assertEqual(check["count"], 0, check)
self.assertEqual(report["status"], "passed")
self.assertRegex(report["checked_at"], r"^\d{4}-\d\d-\d\dT.*\+00:00$")
def test_health_check_never_writes(self):
prompt = self.save("Untouched")
self.image(prompt, "kept.png")
self.image(prompt, "gone.png", exists=False)
with self.raw() as conn:
conn.execute("UPDATE prompts SET hash='stale' WHERE id=?", (prompt,))
conn.close()
# The app's own connection stays open, as in the running server.
self.db.get_statistics()
files = [self.db_path, self.db_path + "-wal"]
before = [_digest(path) for path in files]
shm_size = os.path.getsize(self.db_path + "-shm")
media = sorted(os.listdir(self.output))
opened = []
real_connect = sqlite3.connect
class Recording(sqlite3.Connection):
changes_at_close = None
def close(self):
Recording.changes_at_close = self.total_changes
super().close()
def connect(*args, **kwargs):
conn = real_connect(*args, factory=Recording, **kwargs)
opened.append((args, kwargs, conn))
return conn
with patch.object(health.sqlite3, "connect", side_effect=connect):
report = self.run_checks()
self.assertEqual(report["status"], "incomplete")
self.assertEqual(len(opened), 1)
args, kwargs, conn = opened[0]
self.assertTrue(str(args[0]).endswith("?mode=ro"), args)
self.assertTrue(kwargs.get("uri"))
self.assertEqual(Recording.changes_at_close, 0)
with self.assertRaises(sqlite3.ProgrammingError): # closed in finally
conn.execute("SELECT 1")
self.assertEqual([_digest(path) for path in files], before)
# The -shm holds WAL read marks every reader updates; it never grows.
self.assertEqual(os.path.getsize(self.db_path + "-shm"), shm_size)
self.assertEqual(sorted(os.listdir(self.output)), media)
with self.db.model.get_connection() as live:
stored = live.execute(
"SELECT hash FROM prompts WHERE id=?", (prompt,)
).fetchone()[0]
self.assertEqual(stored, "stale")
def test_connection_closed_when_a_check_raises(self):
self.save("Any")
with (
patch.object(
health,
"_canonical_hashes",
side_effect=sqlite3.OperationalError("boom"),
),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
canonical = self.check(report, "canonical_hashes")
self.assertEqual(canonical["status"], "failed")
self.assertIsNone(canonical["count"])
self.assertEqual(canonical["detail"], ["The check could not finish."])
self.assertEqual(self.check(report, "integrity")["status"], "passed")
self.assertEqual(report["status"], "incomplete")
self.db.close_all()
assert_closed(self, self.root)
def test_unopenable_database_fails_every_database_check(self):
missing = PromptDatabase.__new__(PromptDatabase)
missing.model = type("M", (), {"db_path": str(self.root / "nope.db")})()
with self.assertLogs("prompt_manager.database.health", "ERROR"):
report = missing.run_health_checks([str(self.output)], review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses["integrity"], "failed")
self.assertEqual(statuses["missing_files"], "failed")
self.assertEqual(statuses["ambiguous_metadata"], "passed")
self.assertEqual(report["status"], "incomplete")
self.assertFalse(os.path.exists(self.root / "nope.db"))
def test_integrity_and_foreign_key_issues_reported(self):
prompt = self.save("Linked")
with self.raw() as conn:
conn.execute("CREATE TABLE probe(a)")
conn.execute("CREATE INDEX probe_a ON probe(a)")
conn.executemany("INSERT INTO probe VALUES(?)", [(i,) for i in range(16)])
conn.execute(
"INSERT INTO generated_images(prompt_id,image_path,filename,file_path)"
" VALUES(?,?,?,?)",
(9999, str(self.output / "orphan.png"), "orphan.png", "orphan"),
)
conn.execute(
"INSERT INTO prompt_tags(prompt_id,tag_id) VALUES(?,?)", (prompt, 777)
)
conn.close()
with self.raw() as conn:
conn.execute("PRAGMA writable_schema=ON")
conn.execute(
"UPDATE sqlite_master SET sql='CREATE INDEX probe_a ON probe(a DESC)'"
" WHERE name='probe_a'"
)
conn.close()
self.db.close_all() # the next read sees the edited schema
(self.output / "orphan.png").write_bytes(b"png")
with self.raw() as conn:
expected = [r[0] for r in conn.execute("PRAGMA integrity_check(100)")]
conn.close()
self.assertGreater(len(expected), 10)
report = self.run_checks(review_count=0)
integrity = self.check(report, "integrity")
self.assertEqual(integrity["status"], "issues")
self.assertEqual(integrity["count"], len(expected))
self.assertEqual(integrity["detail"], expected[:10])
self.assertIn("missing from index probe_a", integrity["detail"][0])
foreign = self.check(report, "foreign_keys")
self.assertEqual(foreign["status"], "issues")
self.assertEqual(foreign["count"], 2)
self.assertEqual(
sorted(foreign["detail"]),
[
"generated_images row 1 references a missing prompts row",
"prompt_tags row 1 references a missing tags row",
],
)
consistency = self.check(report, "consistency")
self.assertEqual(consistency["status"], "issues")
self.assertEqual(consistency["count"], 1)
self.assertEqual(
consistency["detail"], ["Image 1 references non-existent prompt 9999"]
)
self.assertEqual(report["status"], "issues")
def test_canonical_hash_mismatch_counts_rows(self):
ids = [self.save(f"Prompt {n}") for n in range(14)]
with self.raw() as conn:
for n, prompt_id in enumerate(ids[:12]):
conn.execute(
"UPDATE prompts SET hash=? WHERE id=?", (f"bad{n}", prompt_id)
)
conn.execute("UPDATE prompts SET hash=NULL WHERE id=?", (ids[12],))
conn.close()
# Normalised spellings hash alike: this one is not a mismatch.
self.save(" MIXED Case ")
report = self.run_checks(review_count=0)
canonical = self.check(report, "canonical_hashes")
self.assertEqual(canonical["status"], "issues")
self.assertEqual(canonical["count"], 13)
self.assertEqual(len(canonical["detail"]), 10)
self.assertEqual(
canonical["detail"][0],
f"Prompt {ids[0]}: the stored hash does not match its text",
)
def test_duplicate_pair_rows_reported(self):
pair = self.db.save_prompt_pair("Cat", "")
positive = self.db.get_prompt_pair(pair)["positive_prompt_id"]
self.db.save_prompt_pair("Dog", "Blur")
with self.raw() as conn:
# Only the partial index keeps positive-only pairs unique; a restored
# or hand-edited database may have lost it.
conn.execute("DROP INDEX idx_prompt_pairs_positive_only")
conn.executemany(
"INSERT INTO prompt_pairs(positive_prompt_id,negative_prompt_id)"
" VALUES(?,NULL)",
[(positive,), (positive,)],
)
conn.close()
report = self.run_checks(review_count=0)
pairs = self.check(report, "pair_uniqueness")
self.assertEqual(pairs["status"], "issues")
self.assertEqual(pairs["count"], 2)
self.assertEqual(
pairs["detail"],
[f"Positive prompt {positive} with no negative: 3 pair rows"],
)
def test_integrity_count_at_the_limit_is_flagged_capped(self):
class Rows:
def __init__(self, rows):
self.rows = rows
def execute(self, _sql):
return iter([(row,) for row in self.rows])
capped = health._integrity(Rows(["bad"] * health.INTEGRITY_LIMIT), [])
self.assertEqual(capped["status"], "issues")
self.assertEqual(capped["count"], health.INTEGRITY_LIMIT)
self.assertTrue(capped["capped"])
exact = health._integrity(Rows(["bad"] * 3), [])
self.assertEqual(exact["count"], 3)
self.assertFalse(exact["capped"])
self.assertFalse(health._integrity(Rows(["ok"]), [])["capped"])
def test_an_unexpected_error_fails_only_its_own_check(self):
self.save("Any")
with (
patch.object(health, "_pair_uniqueness", side_effect=KeyError("boom")),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses.pop("pair_uniqueness"), "failed")
self.assertEqual(set(statuses.values()), {"passed"})
class TestDatabaseUri(unittest.TestCase):
"""The ``mode=ro`` URI of the database file, for every path form SQLite opens."""
def test_posix_path_is_a_local_file_uri(self):
self.assertEqual(
health.readonly_uri("/data/my db#1?.db", windows=False),
"file:///data/my%20db%231%3F.db?mode=ro",
)
def test_windows_drive_path(self):
self.assertEqual(
health.readonly_uri(r"C:\Users\Me\prompts.db", windows=True),
"file:///C:/Users/Me/prompts.db?mode=ro",
)
def test_windows_unc_path_has_an_empty_authority(self):
# pathlib spells \\server\share as file://server/share, which SQLite
# rejects (a non-local authority); an empty one keeps the UNC path.
uri = health.readonly_uri(r"\\server\share\Comfy UI\prompts.db", windows=True)
self.assertEqual(uri, "file:////server/share/Comfy%20UI/prompts.db?mode=ro")
self.assertTrue(ntpath.isabs(r"\\server\share\Comfy UI\prompts.db"))
def test_uri_opens_a_path_with_uri_characters(self):
with tempfile.TemporaryDirectory() as temp:
# "?" is not a valid Windows file name character; "#", "%" and spaces are.
name = "a #b%20 c.db" if os.name == "nt" else "a #b?c%20.db"
path = os.path.join(os.path.realpath(temp), name)
conn = sqlite3.connect(path)
try:
conn.execute("CREATE TABLE t(a)")
conn.commit()
finally:
conn.close()
ro = sqlite3.connect(health.readonly_uri(path), uri=True)
try:
self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,))
with self.assertRaises(sqlite3.OperationalError):
ro.execute("INSERT INTO t VALUES(1)")
finally:
ro.close()
@unittest.skipUnless(os.name == "nt", "a native UNC path exists on Windows only")
def test_native_unc_path_opens(self):
with tempfile.TemporaryDirectory() as temp:
path = os.path.join(os.path.realpath(temp), "prompts.db")
conn = sqlite3.connect(path)
try:
conn.execute("CREATE TABLE t(a)")
conn.commit()
finally:
conn.close()
drive, rest = ntpath.splitdrive(path)
unc = "\\\\localhost\\" + drive[0] + "$" + rest
if not os.path.exists(unc):
self.skipTest("the administrative share is not reachable here")
ro = sqlite3.connect(health.readonly_uri(unc), uri=True)
try:
self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,))
finally:
ro.close()
class TestMissingFiles(HealthTestCase):
def test_missing_files_counted_under_readable_sources_only(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
for n in range(12):
self.image(prompt, f"gone-{n}.png", exists=False)
elsewhere = self.root / "elsewhere"
elsewhere.mkdir()
self.image(prompt, "outside.png", exists=False, folder=elsewhere)
report = self.run_checks(review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 12)
self.assertEqual(len(missing["detail"]), 10)
self.assertRegex(missing["detail"][0], r"^Image \d+: gone-0\.png is missing$")
self.assertNotIn(str(self.root), " ".join(missing["detail"]))
with self.db.model.get_connection() as conn:
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM generated_images").fetchone()[0],
14,
)
def test_links_outside_every_source_are_named_not_counted(self):
prompt = self.save("Images")
elsewhere = self.root / "elsewhere"
elsewhere.mkdir()
self.image(prompt, "outside.png", exists=False, folder=elsewhere)
missing = self.check(self.run_checks(review_count=0), "missing_files")
self.assertEqual(missing["status"], "passed")
self.assertEqual(missing["count"], 0)
self.assertEqual(
missing["detail"],
["1 link outside the image sources was not checked."],
)
def test_missing_file_vs_unavailable_root(self):
# Per source (S2 ruling): the readable source is counted, the absent
# one reports "unavailable" with no count, and the report is incomplete.
prompt = self.save("Images")
absent = self.root / "unmounted"
absent.mkdir()
self.image(prompt, "on-nas.png", folder=absent)
self.image(prompt, "gone.png", exists=False)
os.unlink(absent / "on-nas.png")
absent.rmdir()
report = self.run_checks([str(self.output), str(absent)], review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 1)
self.assertTrue(missing["partial"])
self.assertEqual(
missing["sources"],
[
{"name": "output", "status": "checked", "count": 1},
{"name": "unmounted", "status": "unavailable", "count": None},
],
)
self.assertEqual(
missing["detail"][0],
"1 image source is unavailable, so its files were not counted.",
)
self.assertRegex(missing["detail"][1], r"^Image \d+: gone\.png is missing$")
# The link under the unavailable source is neither counted nor "outside".
self.assertEqual(len(missing["detail"]), 2)
self.assertNotIn(str(self.root), " ".join(missing["detail"]))
self.assertEqual(report["status"], "incomplete")
def test_partial_sources_with_no_missing_file_are_not_passed(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
absent = self.root / "unmounted"
missing = self.check(
self.run_checks([str(self.output), str(absent)], review_count=0),
"missing_files",
)
self.assertEqual(missing["status"], "unavailable")
self.assertEqual(missing["count"], 0)
self.assertTrue(missing["partial"])
self.assertEqual(
[s["status"] for s in missing["sources"]], ["checked", "unavailable"]
)
def test_every_source_unavailable_counts_nothing(self):
prompt = self.save("Images")
absent = self.root / "unmounted"
self.image(prompt, "gone.png", exists=False, folder=self.output)
report = self.run_checks([str(absent)], review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "unavailable")
self.assertIsNone(missing["count"])
self.assertFalse(missing["partial"])
self.assertEqual(
missing["sources"],
[{"name": "unmounted", "status": "unavailable", "count": None}],
)
self.assertEqual(
missing["detail"],
["1 image source is unavailable, so missing files were not counted."],
)
def test_each_source_counts_its_own_missing_files(self):
prompt = self.save("Images")
archive = self.root / "archive"
archive.mkdir()
self.image(prompt, "a.png", exists=False)
self.image(prompt, "b.png", exists=False, folder=archive)
self.image(prompt, "c.png", exists=False, folder=archive)
missing = self.check(
self.run_checks([str(self.output), str(archive)], review_count=0),
"missing_files",
)
self.assertEqual(missing["count"], 3)
self.assertFalse(missing["partial"])
self.assertEqual(
[(s["name"], s["count"]) for s in missing["sources"]],
[("output", 1), ("archive", 2)],
)
def test_no_sources_is_unavailable(self):
self.save("Images")
missing = self.check(self.run_checks([], review_count=0), "missing_files")
self.assertEqual(missing["status"], "unavailable")
self.assertIsNone(missing["count"])
self.assertEqual(missing["detail"], ["No image source is available."])
self.assertEqual(missing["sources"], [])
self.assertFalse(missing["partial"])
def test_a_bad_stored_path_fails_only_missing_files(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
with self.raw() as conn:
conn.execute(
"INSERT INTO generated_images(prompt_id,image_path,filename)"
" VALUES(?,?,?)",
(prompt, str(self.root / "else\x00where.png"), "bad.png"),
)
conn.close()
real_realpath = os.path.realpath
def realpath(path, *args, **kwargs):
# POSIX (and Windows before Python 3.11) raise for the NUL byte;
# newer ntpath returns it unchanged. Raise everywhere, as POSIX does.
if "\x00" in str(path):
raise ValueError("embedded null byte")
return real_realpath(path, *args, **kwargs)
with (
patch.object(health.os.path, "realpath", side_effect=realpath),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses.pop("missing_files"), "failed")
self.assertEqual(set(statuses.values()), {"passed"})
def test_files_are_checked_after_the_read_snapshot_ends(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
self.image(prompt, "gone.png", exists=False)
opened = []
real_connect = sqlite3.connect
def connect(*args, **kwargs):
conn = real_connect(*args, **kwargs)
opened.append(conn)
return conn
real_exists = os.path.exists
in_snapshot = []
def exists(path):
for conn in opened:
try:
in_snapshot.append(conn.in_transaction)
except sqlite3.ProgrammingError: # closed: no snapshot
in_snapshot.append(False)
return real_exists(path)
with (
patch.object(health.sqlite3, "connect", side_effect=connect),
patch.object(health.os.path, "exists", side_effect=exists),
):
missing = self.check(self.run_checks(review_count=0), "missing_files")
self.assertEqual((missing["status"], missing["count"]), ("issues", 1))
self.assertEqual(len(opened), 1)
self.assertTrue(in_snapshot)
self.assertNotIn(True, in_snapshot)
class TestAmbiguousMetadata(HealthTestCase):
def test_ambiguous_metadata_unavailable_without_review_queue(self):
self.save("Any")
report = self.run_checks()
ambiguous = self.check(report, "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "unavailable")
self.assertIsNone(ambiguous["count"])
self.assertEqual(ambiguous["detail"], ["Needs review is not available yet."])
self.assertEqual(report["status"], "incomplete")
def test_review_count_reports_images_needing_review(self):
report = self.run_checks(review_count=3)
ambiguous = self.check(report, "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "issues")
self.assertEqual(ambiguous["count"], 3)
self.assertEqual(ambiguous["detail"], [])
self.assertEqual(report["status"], "issues")
def test_invalid_review_count_is_rejected(self):
for value in (-1, True, "3", 1.5):
with self.assertRaises(ValueError):
self.run_checks(review_count=value)
class TestOverallStatus(unittest.TestCase):
def test_overall_status_precedence(self):
def checks(*statuses):
return [{"status": status} for status in statuses]
self.assertEqual(health.overall_status(checks("passed", "passed")), "passed")
self.assertEqual(health.overall_status(checks("passed", "issues")), "issues")
self.assertEqual(
health.overall_status(checks("issues", "unavailable")), "incomplete"
)
self.assertEqual(health.overall_status(checks("failed")), "incomplete")
# A check that found issues under only some sources is incomplete too.
partial = [{"status": "issues", "partial": True}, {"status": "passed"}]
self.assertEqual(health.overall_status(partial), "incomplete")
if __name__ == "__main__":
unittest.main()
+18 -3
View File
@@ -23,7 +23,11 @@ sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
from py.config import ( # noqa: E402
OUTSIDE_ALLOWED_REASON,
GalleryConfig,
PromptManagerConfig,
)
EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
@@ -158,9 +162,10 @@ class GalleryRootTestCase(unittest.TestCase):
class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_filesystem_root_rejected(self):
"""Refused as outside the allowed folders: no reason names the layout."""
ok, reason = GalleryConfig.validate_gallery_root(os.path.abspath(os.sep))
self.assertFalse(ok)
self.assertIn("root", reason.lower())
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
@unittest.skipUnless(os.name == "nt", "drive roots only exist on Windows")
def test_windows_drive_root_rejected(self):
@@ -170,7 +175,17 @@ class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_home_directory_rejected(self):
ok, reason = GalleryConfig.validate_gallery_root(os.path.expanduser("~"))
self.assertFalse(ok)
self.assertIn("home", reason.lower())
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
def test_home_directory_inside_an_allowed_parent_is_still_refused(self):
"""Even when the environment opens the home's parent, home answers
like any outside path."""
home = os.path.expanduser("~")
os.environ[EXTRA_ROOTS_ENV] = os.path.dirname(home)
self.assertEqual(
GalleryConfig.gallery_root_problem(home),
("outside-allowed", OUTSIDE_ALLOWED_REASON),
)
def test_directory_outside_comfyui_rejected(self):
outside = Path(self.tmpdir) / "elsewhere"
+45
View File
@@ -5,7 +5,10 @@ returns for the same input (tests/test_generation_params_js_parity.py checks
that directly when Node is available).
"""
import json
import os
import shutil
import subprocess # nosec B404 - runs a fixed Node one-liner
import sys
import unittest
@@ -17,6 +20,7 @@ from utils.generation_params import ( # noqa: E402
extract_generation_params,
extract_from_records,
parse_js_json,
parse_page_json,
)
try:
@@ -219,6 +223,47 @@ class ParsesLikeJavaScript(unittest.TestCase):
run("seed-beyond-double-precision")["seed"], 4974052037430000001.0
)
def test_page_parse_reads_nan_numbers_as_null(self):
# gallery.js and admin.js replace NaN with null before JSON.parse.
self.assertEqual(
parse_page_json('{"a": NaN, "b": [NaN], "t": "NaN"}'),
{"a": None, "b": [None], "t": "NaN"},
)
self.assertEqual(
parse_page_json('{"seed": 9007199254740993}'), {"seed": 9007199254740992.0}
)
for text in ('{"a": Infinity}', "[-Infinity]", "not json", "", None):
with self.subTest(text=text):
self.assertIsNone(parse_page_json(text))
@unittest.skipUnless(shutil.which("node"), "Node is not installed")
def test_page_parse_equals_the_legacy_pages_for_nan_numbers(self):
# gallery.js/admin.js: two NaN regexes, then JSON.parse (Infinity throws).
texts = [
'{"1": {"is_changed": [NaN]}, "2": {"is_changed": NaN, "v": 1.5}}',
'{"a":NaN,"b":[NaN, 2]}',
'{"a": Infinity}',
'{"seed": 9007199254740993}',
]
script = (
"const texts = JSON.parse(require('fs').readFileSync(0, 'utf8'));"
"console.log(JSON.stringify(texts.map((t) => { try { return {v: JSON.parse("
"t.replace(/:\\s*NaN\\b/g, ': null').replace(/\\bNaN\\b/g, 'null'))}; }"
" catch (e) { return null; } })));"
)
completed = subprocess.run( # nosec B603 B607 - fixed argv, no shell
["node", "-e", script],
input=json.dumps(texts),
capture_output=True,
text=True,
check=True,
timeout=60,
)
expected = json.loads(completed.stdout)
for text, page in zip(texts, expected):
with self.subTest(text=text):
self.assertEqual(parse_page_json(text), page and page["v"])
def test_duplicate_keys_keep_the_last_value(self):
self.assertEqual(parse_js_json('{"a": 1, "a": 2}'), {"a": 2})
+8 -3
View File
@@ -800,10 +800,15 @@ class TestUpstreamVerifiedNodeSemantics(unittest.TestCase):
prompt["31"] = dict(prompt["3"], inputs=dict(prompt["3"]["inputs"]))
prompt["31"]["inputs"]["latent_image"] = ["3", 0]
prompt["8"]["inputs"]["samples"] = ["31", 0]
# Unanchored, two listed samplers are two branches, as for KSampler.
self.assertEqual(_inspect({"prompt": prompt})["state"], "ambiguous")
# 31 continues 3's latent: the stages are compared, as for KSampler.
for anchor in (None, "9"):
result = _inspect({"prompt": prompt}, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "31")
)
prompt["31"]["inputs"]["positive"] = ["7", 0]
result = _inspect({"prompt": prompt}, output_node_id="9")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
self.assertEqual(result["reason_code"], "multistage-differs")
def test_seed_variance_enhancer_passes_its_conditioning(self):
prompt = _f02(
+105 -3
View File
@@ -55,6 +55,9 @@ def _minimal():
"runtime": [],
"savers": {},
"non_file_outputs": [],
"lineage": [
{"class": "VAEEncode", "slot": 0, "strategy": "continue", "input": "pixels"}
],
"stage_inputs": {"shaped": ["positive"], "with_model": ["samples"]},
"facts": {
"configuration_outputs": {},
@@ -199,16 +202,26 @@ class SpecEntriesTest(unittest.TestCase):
},
)
# Combine and concat keep every part in effect (additive); an average
# blends its parts, so a part may be weighted away.
self.assertEqual(
table.composites["ConditioningCombine"],
{"strategy": "combine", "inputs": ("conditioning_1", "conditioning_2")},
{
"strategy": "combine",
"inputs": ("conditioning_1", "conditioning_2"),
"additive": True,
},
)
for name in ("ConditioningConcat", "ConditioningAverage"):
for name, additive in (
("ConditioningConcat", {"additive": True}),
("ConditioningAverage", {}),
):
self.assertEqual(
table.composites[name],
{
"strategy": "combine",
"inputs": ("conditioning_to", "conditioning_from"),
**additive,
},
)
@@ -257,7 +270,12 @@ class SpecEntriesTest(unittest.TestCase):
)
# D18: decided at run time.
self.assertEqual(table.runtime, frozenset({"PromptSearchList"}))
# LoraManager's TriggerWord Toggle outputs the LoRA's trigger words as
# filtered at run time (option A leaves them out of a PromptManager).
self.assertEqual(
table.runtime,
frozenset({"PromptSearchList", "TriggerWord Toggle (LoraManager)"}),
)
comfy = "{name}_{counter:05}_.{ext}"
for name, image in (
@@ -464,6 +482,89 @@ class SpecEntriesTest(unittest.TestCase):
self.assertIsInstance(table.composites["ConditioningCombine"]["inputs"], tuple)
class LineageTableTest(_TableCase):
"""``lineage``: outputs whose image or latent continues one input's."""
def test_repository_lineage_lists_the_pass_through_nodes(self):
table = load_semantics()
self.assertEqual(
{key: dict(entry) for key, entry in table.lineage.items()},
{
("VAEDecode", 0): {"strategy": "continue", "input": "samples"},
("VAEDecodeTiled", 0): {"strategy": "continue", "input": "samples"},
("VAEEncode", 0): {"strategy": "continue", "input": "pixels"},
("VAEEncodeTiled", 0): {"strategy": "continue", "input": "pixels"},
("ImageUpscaleWithModel", 0): {
"strategy": "continue",
"input": "image",
},
("ImageScale", 0): {"strategy": "continue", "input": "image"},
("ImageScaleBy", 0): {"strategy": "continue", "input": "image"},
("ImageScaleToTotalPixels", 0): {
"strategy": "continue",
"input": "image",
},
("LatentUpscale", 0): {"strategy": "continue", "input": "samples"},
("LatentUpscaleBy", 0): {"strategy": "continue", "input": "samples"},
("Image Filter", 0): {"strategy": "continue", "input": "images"},
("Image Filter", 1): {"strategy": "continue", "input": "latents"},
},
)
# Lineage describes no prompt: its classes are not prompt categories.
self.assertFalse({name for name, _ in table.lineage} & table.all_classes)
def test_lineage_rows_are_validated(self):
def row(**changes):
entry = {
"class": "VAEEncode",
"slot": 0,
"strategy": "continue",
"input": "pixels",
}
entry.update(changes)
return entry
cases = (
({"lineage": {}}, "must be a list"),
({"lineage": ["VAEEncode"]}, "must be an object"),
({"lineage": [row(strategy="passthrough")]}, "unknown strategy"),
({"lineage": [row(slot=-1)]}, "slot"),
({"lineage": [row(slot=True)]}, "slot"),
({"lineage": [row(input="")]}, "input name"),
({"lineage": [row(note="x")]}, "field"),
({"lineage": [row(), row()]}, "duplicate"),
({"lineage": [row(**{"class": ""})]}, "class"),
)
for index, (change, fragment) in enumerate(cases):
with self.subTest(case=index):
doc = _minimal()
doc.update(change)
self.assertRejected(doc, fragment)
doc = _minimal()
doc["lineage"].append(row(slot=1, input="latents"))
self.assertEqual(len(self.load_doc(doc).lineage), 2)
def test_a_lineage_class_may_also_carry_conditioning(self):
# InpaintModelConditioning-like: conditioning slots, a latent slot.
doc = _minimal()
doc["lineage"].append(
{"class": "FluxGuidance", "slot": 1, "strategy": "continue", "input": "x"}
)
self.assertIn(("FluxGuidance", 1), self.load_doc(doc).lineage)
def test_composite_additive_must_be_a_boolean(self):
doc = _minimal()
doc["composites"]["ConditioningCombine"] = {
"strategy": "combine",
"inputs": ["conditioning_1", "conditioning_2"],
"additive": "yes",
}
self.assertRejected(doc, "additive must be a boolean")
doc["composites"]["ConditioningCombine"]["additive"] = True
table = self.load_doc(doc)
self.assertIs(table.composites["ConditioningCombine"]["additive"], True)
class ValidationTest(_TableCase):
def test_minimal_table_loads(self):
table = self.load_doc(_minimal())
@@ -501,6 +602,7 @@ class ValidationTest(_TableCase):
{"literal", "strip-newlines", "join-multi", "concat"}
),
"savers": frozenset({"comfy-counter", "vhs-counter"}),
"lineage": frozenset({"continue"}),
},
)
with self.assertRaises(TypeError):
+98
View File
@@ -1706,6 +1706,104 @@ class TestLibrary(unittest.TestCase):
self.db.list_library_images(positive, pair_id=foreign, archive="all")
class LibraryCounts(unittest.TestCase):
"""System › Overview counts and usage statistics follow Library eligibility."""
setUp = TestLibrary.setUp
tearDown = TestLibrary.tearDown
save = TestLibrary.save
state = TestLibrary.state
def test_counts_use_library_eligibility(self):
self.db.save_prompt_pair("Cat", "Blur")
archived = self.db.save_prompt_pair("Cat", "Noise")
self.db.archive_prompt_pair(archived)
titled_pair = self.db.save_prompt_pair("Dog", "Titled negative")
titled = self.db.get_prompt_pair(titled_pair)["negative_prompt_id"]
self.state(titled, "A titled parent") # zero pairs of its own: counted
legacy = self.save("Legacy")
self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "a.png"))
self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "b.png"))
counts = self.db.library_counts()
# Cat, Dog, the titled negative and Legacy; "Blur" and "Noise" are
# negative-only.
self.assertEqual(counts["positive_prompts"], 4)
# Cat+Blur and Dog+Titled negative; the archived pair is excluded.
self.assertEqual(counts["negative_variants"], 2)
self.assertEqual(counts["images"], 2)
self.assertEqual(counts["rated_images"], 0)
listed = self.db.list_library()
self.assertEqual(counts["positive_prompts"], listed["total"])
self.assertEqual(counts["negative_variants"], listed["variant_total"])
def test_counts_of_an_empty_library_are_zero(self):
self.assertEqual(
self.db.library_counts(),
{
"positive_prompts": 0,
"negative_variants": 0,
"images": 0,
"rated_images": 0,
"review_pending": 0,
},
)
def test_rated_images_follow_their_linked_prompt_rating(self):
# An image carries no rating of its own: it shows its linked prompt's
# (the Images "Highest rated" sort), so "rated" counts those images.
folder = Path(self.temp.name)
rated = self.save("Rated", rating=4)
unrated = self.save("Unrated")
for name in ("a.png", "b.png", "c.png"):
self.db.link_image_to_prompt(rated, str(folder / name))
self.db.link_image_to_prompt(unrated, str(folder / "d.png"))
counts = self.db.library_counts()
self.assertEqual(counts["images"], 4)
self.assertEqual(counts["rated_images"], 3)
self.db.update_prompt_rating(unrated, 2)
self.assertEqual(self.db.library_counts()["rated_images"], 4)
def test_statistics_sum_real_run_counts(self):
a = self.save("Alpha", category="Landscapes", rating=5)
b = self.save("Beta", category="Landscapes", rating=4)
self.save("Gamma", category="Landscapes") # unrated: not in the average
c = self.save("Delta", category="Portraits", rating=3)
blank = self.save("Epsilon", category=" ")
uncategorised = self.save("Zeta")
self.db.record_prompt_use(a, times=3)
self.db.record_prompt_use(b, times=2)
self.db.record_prompt_use(c)
self.db.record_prompt_use(blank)
self.db.record_prompt_use(uncategorised, times=4)
# A negative-only prompt's runs are the pair's runs: never counted twice.
pair = self.db.save_prompt_pair("Alpha", "Only negative")
negative = self.db.get_prompt_pair(pair)["negative_prompt_id"]
with self.db.model.get_connection() as connection:
connection.execute(
"UPDATE prompts SET category='Landscapes',run_count=50 WHERE id=?",
(negative,),
)
rows = self.db.usage_by_category()
self.assertEqual(
rows,
[
{"category": "Landscapes", "runs": 5, "average_rating": 4.5},
{"category": None, "runs": 5, "average_rating": None},
{"category": "Portraits", "runs": 1, "average_rating": 3.0},
],
)
def test_counts_and_usage_write_nothing(self):
self.save("Alpha", category="A")
with self.db.model.get_connection() as connection:
before = connection.total_changes
self.db.library_counts()
self.db.usage_by_category()
with self.db.model.get_connection() as connection:
self.assertEqual(connection.total_changes, before)
self.assertFalse(connection.in_transaction)
class DisplayTitleUnits(unittest.TestCase):
"""The first-line title is cut like the Library's JavaScript title()."""
+28 -1
View File
@@ -52,7 +52,7 @@ from utils.prompt_inspector import (
)
from utils.prompt_inspector import inspect_prompt_evidence, unreadable_evidence
from utils.prompt_pair_resolver import INSPECTION_PROFILE
from utils.prompt_pair_resolver import ADDED_CONDITIONING_REASON, INSPECTION_PROFILE
try:
from tests import inspection_parity as parity
@@ -1293,6 +1293,33 @@ class LocalResolverParityTests(unittest.TestCase):
self.assertEqual(
answers["select-preview-dropped"]["resolution"]["sampler_id"], "3"
)
# Pass-through lineage: the chain shapes reach every new outcome.
self.assertEqual(
{
name: answers[f"select-{name}"]["resolution"]["reason"]
for name in ("hires-fix", "flux-three-stage", "hires-fix-differs")
},
{
"hires-fix": "",
"flux-three-stage": ADDED_CONDITIONING_REASON,
"hires-fix-differs": "Sampler stages have different prompts",
},
)
def test_local_roles_match_fuzzed_pass_through_chains(self):
cases = {}
for seed in range(4):
cases.update(selection_cases.passthrough_fuzz_cases(seed, 120))
answers = self._compare(cases)
reasons = [answer["resolution"]["reason"] for answer in answers.values()]
for reason in (
"",
"Sampler stages have different prompts",
"A later stage adds conditioning to the base prompt",
"Combined conditioning has different texts",
"Multiple sampler branches",
):
self.assertGreater(reasons.count(reason), 5, reason)
def test_local_roles_match_fuzzed_selection_sets(self):
cases = {}
+88
View File
@@ -471,6 +471,94 @@ class CaptureStoreTestCase(unittest.TestCase):
self.assertEqual(results["capture"].size, 60)
self.capture(workers, "b.png")
def _hold_worker(self, store, relative, **options):
"""Start a capture that holds a worker until ``proceed`` is set."""
entered, proceed = threading.Event(), threading.Event()
original = CaptureStore._read
results = {}
def blocking(store_, stream, size):
if threading.current_thread() is thread and not entered.is_set():
entered.set()
self.assertTrue(proceed.wait(WAIT))
return original(store_, stream, size)
def worker():
try:
results["capture"] = self.capture(store, relative, **options)
except Exception as error: # pragma: no cover - reported below
results["error"] = error
patcher = patch.object(CaptureStore, "_read", blocking)
patcher.start()
self.addCleanup(patcher.stop)
thread = threading.Thread(target=worker)
thread.start()
self.assertTrue(entered.wait(WAIT))
def finish():
proceed.set()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
return results["capture"]
self.addCleanup(proceed.set)
return finish
def test_background_capture_never_takes_the_last_worker(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
# The second background capture waits, then fails: one worker is free
# but it is the interactive request's.
started = time.monotonic()
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.05):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertGreaterEqual(time.monotonic() - started, 0.04)
# An interactive capture (Tag dialog, Apply) still gets the last one.
interactive = self.capture(store, "c.png")
self.assertEqual(interactive.size, 60)
finish()
def test_background_capture_waits_for_a_worker_to_free(self):
for name in ("a", "b"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
waited = {}
def background():
waited["capture"] = self.capture(store, "b.png", background=True)
thread = threading.Thread(target=background)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", WAIT):
thread.start()
finish()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertEqual(waited["capture"].size, 60)
def test_single_worker_store_admits_no_background_capture(self):
self.write("a.png", bytes(60))
store = self.make_store(max_workers=1)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "a.png", background=True)
self.assertEqual(self.capture(store, "a.png").size, 60)
def test_background_capture_leaves_the_last_capture_slot(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_captures=2)
self.capture(store, "a.png")
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertEqual(self.capture(store, "c.png").size, 60)
# -- lifetime ------------------------------------------------------------
def test_idle_and_hard_expiry_and_in_use_retention(self):
+360
View File
@@ -0,0 +1,360 @@
"""ComfyUI's ``NaN``/``Infinity`` metadata ingests, and is stored as strict JSON.
ComfyUI serializes the prompt with Python's ``json.dumps``, so a node whose
``IS_CHANGED`` returns ``float("nan")`` (the "always re-run" idiom) writes
``"is_changed": [NaN]`` into every saved PNG; widgets can hold ``Infinity``.
The legacy pages replaced ``NaN`` with ``null`` before ``JSON.parse``, so such
files must ingest. Whatever is stored or answered as JSON stays strict (the
non-finite numbers become ``null``) and the prompt text and hash never change.
"""
import json
import logging
import os
import sqlite3
import sys
import tempfile
import time
import types
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from PIL import Image, PngImagePlugin
from database.operations import PromptDatabase
from tests.history_waits import drain_writer
from tests.test_api_admin import AdminAPITestCase
from tests.test_api_images import ImageAPITestCase
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
# The words "NaN" and "Infinity" inside the prompt are text, never numbers.
POSITIVE = " A cat named NaN beside an Infinity pool, ratio: NaN, [NaN] "
NEGATIVE = "blurry"
NAN, INF = float("nan"), float("inf")
def graph(positive=POSITIVE, negative=NEGATIVE):
"""An API prompt graph with non-finite numbers where ComfyUI writes them."""
g = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": positive}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
"3": {
"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["2", 0], "model": ["4", 0]},
},
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"5": {
"class_type": "Image Filter",
"inputs": {"strength": INF, "floor": -INF},
"_meta": {"title": "Image Filter"},
"is_changed": [NAN],
},
}
return g
WORKFLOW = {
"nodes": [
{"id": 5, "type": "Image Filter", "widgets_values": [NAN, INF, -INF]},
],
"links": [],
}
def comfy_text(value):
"""The record text ComfyUI writes: Python's default, non-strict JSON."""
text = json.dumps(value)
assert "NaN" in text and "Infinity" in text
return text
def write_png(path, prompt_graph=None, workflow=WORKFLOW):
info = PngImagePlugin.PngInfo()
info.add_text("prompt", comfy_text(prompt_graph or graph()))
if workflow is not None:
info.add_text("workflow", comfy_text(workflow))
path.parent.mkdir(parents=True, exist_ok=True)
with Image.new("RGB", (4, 4)) as image:
image.save(path, pnginfo=info)
return path
def _reject(name):
raise ValueError(f"non-strict JSON constant {name}")
def strict_loads(text):
return json.loads(text, parse_constant=_reject)
class StrictRows:
"""Assertions over the rows a write left behind."""
def image_rows(self, db):
with db.model.get_connection() as conn:
return [
dict(row)
for row in conn.execute(
"SELECT prompt_id, workflow_data, prompt_metadata, parameters "
"FROM generated_images ORDER BY id"
)
]
def assert_strict_snapshot(self, row, *, workflow=True):
prompt = strict_loads(row["prompt_metadata"])
self.assertEqual(prompt["5"]["is_changed"], [None])
self.assertEqual(prompt["5"]["inputs"], {"strength": None, "floor": None})
self.assertEqual(prompt["1"]["inputs"]["text"], POSITIVE)
stored_workflow = strict_loads(row["workflow_data"])
if workflow:
self.assertEqual(
stored_workflow["nodes"][0]["widgets_values"], [None, None, None]
)
if row["parameters"] is not None:
strict_loads(row["parameters"])
def assert_prompt_text(self, db, prompt_id):
with db.model.get_connection() as conn:
row = conn.execute(
"SELECT text, hash FROM prompts WHERE id=?", (prompt_id,)
).fetchone()
self.assertEqual(row["text"], POSITIVE.strip())
self.assertEqual(row["hash"], generate_prompt_hash(POSITIVE))
def dumps_strict(value, **kwargs):
from utils.json_strict import dumps_strict as dumps
return dumps(value, **kwargs)
class TestDumpsStrict(unittest.TestCase):
def test_finite_values_serialize_byte_identically(self):
value = {"b": [1, 2.5, "NaN", None, True], "a": {"x": "Infinity"}}
for kwargs in ({}, {"sort_keys": True, "separators": (",", ":")}):
with self.subTest(kwargs=kwargs):
self.assertEqual(
dumps_strict(value, **kwargs),
json.dumps(value, allow_nan=False, **kwargs),
)
def test_non_finite_numbers_become_null_and_strings_stay(self):
value = {"n": NAN, "i": [INF, -INF], "s": "NaN Infinity", "t": (NAN, 1)}
encoded = dumps_strict(value, sort_keys=True)
self.assertEqual(
strict_loads(encoded),
{"n": None, "i": [None, None], "s": "NaN Infinity", "t": [None, 1]},
)
def test_other_errors_still_raise(self):
cycle = []
cycle.append(cycle)
for value in (cycle, [NAN, cycle]):
with self.assertRaisesRegex(ValueError, "Circular"):
dumps_strict(value)
with self.assertRaises(TypeError):
dumps_strict({"x": object()})
self.assertEqual(
dumps_strict({"x": object()}, default=lambda _: NAN), '{"x": null}'
)
class TestScanIngestsNonFiniteMetadata(StrictRows, AdminAPITestCase):
def ingest(self, name, prompt_graph=None):
path = write_png(self.output_dir / name, prompt_graph)
extracted = self.api._extract_comfyui_metadata(str(path))
return self.api._ingest_scan_batch_sync([(path, extracted)])
def test_resolved_pair_with_nan_is_added_with_strict_snapshots(self):
counts = self.ingest("nan.png")
self.assertEqual(counts["failed"], 0)
self.assertEqual(counts["added"], 1)
rows = self.image_rows(self.api.db)
self.assertEqual(len(rows), 1)
self.assert_strict_snapshot(rows[0])
self.assert_prompt_text(self.api.db, rows[0]["prompt_id"])
pairs = self.api.db.list_prompt_pairs()["pairs"]
self.assertEqual(
[(p["positive_text"], p["negative_text"]) for p in pairs],
[(POSITIVE.strip(), NEGATIVE)],
)
# A repeated scan compares the same strict snapshot: unchanged.
self.assertEqual(self.ingest("nan.png")["unchanged"], 1)
async def test_scan_route_reports_no_failures(self):
from py.config import GalleryConfig
write_png(self.output_dir / "nan.png")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp = await self.client.request("POST", "/prompt_manager/scan")
body = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn('"type": "complete"', body)
self.assertIn('"added": 1', body)
self.assertIn('"failed": 0', body)
class TestMonitorIngestsNonFiniteMetadata(StrictRows, unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.db = PromptDatabase(str(Path(self.temp.name) / "db.sqlite"))
self.pid = self.db.save_prompt(
POSITIVE.strip(), prompt_hash=generate_prompt_hash(POSITIVE)
)
self.tracker = SimpleNamespace(
pop_next_prompt=lambda: None,
get_current_prompt=lambda: None,
extend_prompt_timeout=lambda *args: None,
)
self.handler = ImageGenerationHandler(self.db, self.tracker)
self.handler.logger = logging.getLogger("test.nonfinite.monitor")
self.handler.logger.disabled = True
def tearDown(self):
self.db.close_all()
assert_closed(self, self.temp.name)
self.temp.cleanup()
def test_new_image_with_nan_links_its_pair(self):
# The monitor links only images a PromptManager node produced.
g = graph()
g["1"]["class_type"] = "PromptManager"
path = write_png(Path(self.temp.name) / "out" / "nan.png", g)
self.handler.process_new_image(str(path))
rows = self.image_rows(self.db)
self.assertEqual([row["prompt_id"] for row in rows], [self.pid])
self.assert_strict_snapshot(rows[0])
self.assert_prompt_text(self.db, self.pid)
pairs = self.db.list_prompt_pairs()["pairs"]
self.assertEqual(len(pairs), 1)
self.assertEqual(pairs[0]["image_count"], 1)
def test_legacy_link_stores_strict_json(self):
# The plain prompt link (no exact pair) must not write NaN either.
path = str(Path(self.temp.name) / "plain.png")
metadata = {"prompt": graph(), "workflow": WORKFLOW, "parameters": {"x": NAN}}
self.assertTrue(self.db.link_image_to_prompt(self.pid, path, metadata))
row = self.image_rows(self.db)[0]
self.assert_strict_snapshot(row)
self.assertEqual(strict_loads(row["parameters"]), {"x": None})
def test_stored_images_read_back_without_non_finite_numbers(self):
path = str(Path(self.temp.name) / "legacy.png")
self.db.link_image_to_prompt(self.pid, path, {"prompt": graph()})
with self.db.model.get_connection() as conn:
# A row an older version wrote with Python's default json.dumps.
conn.execute(
"UPDATE generated_images SET prompt_metadata=?", (comfy_text(graph()),)
)
conn.commit()
images = self.db.get_prompt_images(str(self.pid))
strict_loads(json.dumps(images, default=str))
class TestInspectorAnswersStrictJSON(ImageAPITestCase):
async def test_inspect_route_resolves_and_answers_strict_json(self):
write_png(self.output_dir / "nan.png")
response = await self.client.get(
"/prompt_manager/images/inspect/nan.png", params={"root": "0"}
)
self.assertEqual(response.status, 200)
result = strict_loads(await response.text())
self.assertEqual(result["resolution"]["state"], "resolved")
self.assertEqual(
result["prompts"]["positive"],
{"status": "resolved", "text": POSITIVE, "source": "graph-inferred"},
)
async def test_prompt_images_route_answers_strict_json(self):
pid = self.api.db.save_prompt(
POSITIVE.strip(), prompt_hash=generate_prompt_hash(POSITIVE)
)
path = str(write_png(self.output_dir / "linked.png"))
self.api.db.link_image_to_prompt(pid, path, {"prompt": graph()})
with self.api.db.model.get_connection() as conn:
conn.execute(
"UPDATE generated_images SET prompt_metadata=?", (comfy_text(graph()),)
)
conn.commit()
response = await self.client.get(f"/prompt_manager/prompts/{pid}/images")
self.assertEqual(response.status, 200)
result = strict_loads(await response.text())
self.assertEqual(len(result["images"]), 1)
stored = result["images"][0]["prompt_metadata"]
self.assertEqual(stored["1"]["inputs"]["text"], POSITIVE)
self.assertEqual(stored["5"]["is_changed"], [None])
class TestHistoryCapturesNonFiniteGraphs(unittest.TestCase):
"""The queued graph ComfyUI hands the hook may carry float('nan')."""
JOB = "a0c26da3-eb26-405c-b143-c662a84c3112"
def setUp(self):
self.directory = tempfile.TemporaryDirectory()
self.addCleanup(self.directory.cleanup)
self.db = PromptDatabase(os.path.join(self.directory.name, "history.db"))
self.addCleanup(self.db.model.reset_schema_cache)
self.addCleanup(self.db.model.close_all)
def test_queued_graph_with_nan_records_its_run_and_contexts(self):
from tests.test_execution_history_hooks import HostQueue, status
from utils.execution_context_capture import record_runtime_context
from utils.execution_history import register_history_hook
queue = HostQueue()
registration = register_history_hook(
SimpleNamespace(prompt_queue=queue), lambda: self.db
)
self.addCleanup(lambda: registration.dispose(3))
self.addCleanup(registration.writer.invalidate)
deadline = time.monotonic() + 3
while not registration.writer.status().accepting:
self.assertLess(time.monotonic(), deadline, "writer never accepted")
time.sleep(0.005)
g = {
"p": {
"class_type": "PromptManagerPair",
"inputs": {"text": POSITIVE, "negative_text": NEGATIVE},
},
"s": {
"class_type": "KSampler",
"inputs": {"positive": ["p", 0], "negative": ["p", 1], "cfg": INF},
"is_changed": [NAN],
},
"5": graph()["5"],
}
extra = {"extra_pnginfo": {"workflow": WORKFLOW}}
queue.results.append(((0, self.JOB, g, extra, ["s"]), 1))
queue.get()
host = types.ModuleType("comfy_execution.utils")
host.get_executing_context = lambda: SimpleNamespace(
prompt_id=self.JOB, node_id="p", list_index=0
)
with patch.dict(sys.modules, {"comfy_execution.utils": host}):
for role, text in (("positive", POSITIVE), ("negative", NEGATIVE)):
record_runtime_context(
"PromptManagerPair", text, "", "", text, role=role
)
queue.task_done(1, {}, status("execution_success", job=self.JOB))
drain_writer(registration.writer, clock=time.monotonic)
with self.db.model.get_connection() as conn:
conn.row_factory = sqlite3.Row
runs = [dict(r) for r in conn.execute("SELECT * FROM execution_runs")]
self.assertEqual([run["outcome"] for run in runs], ["succeeded"])
contexts = self.db.get_execution_contexts(runs[0]["run_id"])
self.assertEqual(contexts["state"], "complete")
self.assertEqual(contexts["contexts"][0]["positive"]["raw"], POSITIVE)
strict_loads(json.dumps(contexts))
if __name__ == "__main__":
unittest.main()
+66
View File
@@ -160,3 +160,69 @@ class TestPairIngestion(unittest.TestCase):
info = self.db.get_image_prompt_info(self.path)
self.assertEqual(info["pair_context"]["positive_text"], "cat, lora_trigger")
self.assertEqual(info["pair_context"]["text_source"], "execution")
def test_public_ingest_resolved_pair_contract_unchanged(self):
"""The public ingester keeps its result shape and its own transaction;
the extracted ``_ingest_resolved_pair_in`` runs in the caller's."""
from database import pair_ingestion
self.assertTrue(
hasattr(pair_ingestion, "_ingest_resolved_pair_in"),
"_ingest_resolved_pair_in is not extracted",
)
for state in ("ambiguous", "unresolved", "conflict"):
self.assertEqual(
ingest_resolved_pair(self.db, self.path, {}, {"state": state}),
{"status": state, "pairs_added": 0, "links_added": 0},
)
first = ingest_resolved_pair(self.db, self.path, {}, resolved())
self.assertEqual(
set(first),
{"status", "pairs_added", "links_added", "pair_id", "image_id"},
)
self.assertEqual(first["status"], "added")
(link,) = self.db.get_image_prompt_pairs(first["image_id"])
self.assertEqual(
link["provenance"],
{
"resolver_version": 1,
"positive": resolved()["positive"],
"negative": resolved()["negative"],
},
)
conn = self.db.model.get_connection()
self.assertFalse(conn.in_transaction)
# Inside the caller's transaction, with the caller's provenance; an
# exception afterwards rolls the link back with the caller's writes.
provenance = {"kind": "user-selection", "resolver_version": 1}
with self.assertRaises(RuntimeError):
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
self.assertEqual(inner["status"], "added")
raise RuntimeError("caller failed")
self.assertIsNone(self.db.get_image_prompt_info(self.path + "2"))
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
(link,) = self.db.get_image_prompt_pairs(inner["image_id"])
self.assertEqual(link["provenance"], provenance)
with self.db._pair_transaction() as held:
conflict = pair_ingestion._ingest_resolved_pair_in(
held, self.db, self.path, {}, resolved("bird")
)
self.assertEqual(
conflict, {"status": "conflict", "pairs_added": 0, "links_added": 0}
)
+120 -2
View File
@@ -82,7 +82,8 @@ class TestPairScan(AdminAPITestCase):
"class_type": "BasicGuider",
"inputs": {"width": 2048, "height": 2048},
}
self.assertEqual(self._scanned("untraceable.png", g), "ambiguous")
# Ambiguous with nothing to choose: unresolved, never a review item.
self.assertEqual(self._scanned("untraceable.png", g), "unresolved")
self.assertEqual(self.api.db.list_prompt_pairs()["total"], 0)
self.assertEqual(self._texts(), [])
@@ -159,12 +160,129 @@ class TestPairScan(AdminAPITestCase):
"\nSteps: 20, Sampler: Euler"
)
}
# No candidate to choose: reported unresolved, still not guessed.
self.assertEqual(
self.api._ingest_scanned_file(self.output_dir / "ambiguous.png", meta),
"ambiguous",
"unresolved",
)
self.assertEqual(self.api.db.get_recent_prompts()["prompts"], [])
def _nan_scanned(self, name, prompt_graph):
"""Scan ``prompt_graph`` as ComfyUI writes it: ``is_changed`` is NaN."""
stamped = {
node_id: {**node, "is_changed": [float("nan")]}
for node_id, node in prompt_graph.items()
}
text = json.dumps(stamped)
self.assertIn("NaN", text)
return self._scanned_text(name, text)
def _scanned_text(self, name, prompt_text):
meta = {"prompt": prompt_text}
return self.api._ingest_scanned_file(self.output_dir / name, meta)
@staticmethod
def _hires_fix_graph(positive):
"""A PromptManager prompt on a base sampler and a hires-fix sampler.
The base latent comes from a custom batch node and the hires pass from
a custom image filter: no sampler branch traces, so the scan links the
prompt the legacy pages show.
"""
return {
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "m.safetensors"},
},
"2": {
"class_type": "MyTriggerWords",
"inputs": {"trigger_words": ["1", 2]},
},
"3": {
"class_type": "PromptManager",
"inputs": {
"text": positive,
"prepend_text": ["2", 0],
"append_text": "",
"clip": ["1", 1],
},
},
"4": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "blur", "clip": ["1", 1]},
},
"5": {"class_type": "MyLatentBatch", "inputs": {"batch_size": 6}},
"6": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["5", 0],
},
},
"7": {
"class_type": "VAEDecode",
"inputs": {"samples": ["6", 0], "vae": ["1", 2]},
},
"8": {"class_type": "MyImageFilter", "inputs": {"images": ["7", 0]}},
"9": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["8", 0], "vae": ["1", 2]},
},
"14": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"denoise": 0.25,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["9", 0],
},
},
"15": {
"class_type": "VAEDecode",
"inputs": {"samples": ["14", 0], "vae": ["1", 2]},
},
"16": {"class_type": "SaveImage", "inputs": {"images": ["15", 0]}},
}
def test_hires_fix_prompt_links_with_or_without_nan_is_changed(self):
# Legacy parity: the legacy pages read NaN as null before parsing, so they
# showed this prompt; the scan left NaN-stamped files in review.
for name, scan in (("plain.png", None), ("stamped.png", "nan")):
with self.subTest(name=name):
g = self._hires_fix_graph("cat on a sofa")
if scan:
status = self._nan_scanned(name, g)
else:
status = self._scanned(name, g)
self.assertIn(status, ("added", "linked"))
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_untraced_single_sampler_links_the_legacy_prompt_with_nan(self):
g = self._hires_fix_graph("cat on a sofa")
for node_id in ("7", "8", "9", "14", "15"):
del g[node_id]
g["16"]["inputs"]["images"] = ["6", 0]
self.assertEqual(self._nan_scanned("single.png", g), "added")
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_nan_inside_prompt_text_is_kept(self):
# Only the NaN number reads as null; legacy's regex also rewrote text.
g = self._hires_fix_graph("NaN cat, value: NaN")
self.assertEqual(self._nan_scanned("text.png", g), "added")
self.assertEqual(self._texts(), ["NaN cat, value: NaN"])
def test_infinity_is_still_unreadable_like_the_legacy_pages(self):
text = json.dumps(self._hires_fix_graph("cat")).replace(
'"seed": 7', '"seed": Infinity'
)
self.assertIn(self._scanned_text("inf.png", text), ("ambiguous", "unresolved"))
self.assertEqual(self._texts(), [])
def test_raw_png_workflow_is_stored_as_object_and_repeat_is_unchanged(self):
path = self.output_dir / "workflow.png"
workflow = {"nodes": [{"id": 1, "type": "PromptManagerPair"}]}
+4
View File
@@ -258,6 +258,10 @@ class PairSchemaTests(unittest.TestCase):
],
[(1, 1, 0)],
)
self.assertEqual(
[tuple(r) for r in conn.execute("SELECT * FROM media_review_metadata")],
[(1, 1, 0)],
)
self.assertTrue(PromptModel.verify_database_file(live.db_path)[0])
self.assertTrue(
PromptModel.verify_database_file(live.db_path, require_library_state=True)[
+31 -20
View File
@@ -1245,7 +1245,7 @@ class TestGuiderSamplers(unittest.TestCase):
source["3"]["inputs"]["guider"] = [previous, 0]
self.assertEqual(_inspect(source)["reason_code"], "depth-limit")
def test_custom_advanced_upstream_of_a_sampler_is_multistage(self):
def test_custom_advanced_upstream_of_a_sampler_is_a_compared_stage(self):
source = _guided()
source["41"] = {
"class_type": "KSampler",
@@ -1256,15 +1256,18 @@ class TestGuiderSamplers(unittest.TestCase):
},
}
source["8"]["inputs"]["samples"] = ["41", 0]
# 41 continues 3's latent and reads the same texts as 3's guider.
for anchor in ("9", None):
result = _inspect(source, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "41")
)
source["41"]["inputs"]["negative"] = ["6", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["sampler_id"], "41")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], "Multistage sampler ancestry is ambiguous")
unanchored = _inspect(source)
self.assertEqual(unanchored["state"], "ambiguous")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
sorted(item["sampler_id"] for item in unanchored["candidates"]),
["3", "41"],
sorted(item["sampler_id"] for item in result["candidates"]), ["3", "41"]
)
@@ -2632,17 +2635,19 @@ class TestLatentAncestry(unittest.TestCase):
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
self.assertEqual(_inspect(source, output_node_id="9"), result)
def test_upstream_sampler_through_vaedecode_vaeencode_is_multistage(self):
for positive in (OTHER, parity.POS):
with self.subTest(positive=positive):
result = _inspect(_image_to_image(positive), output_node_id="63")
# Until PR3 compares stage texts, never one stage's prompt.
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason"], MULTISTAGE)
self.assertIsNone(result["reason_code"])
self.assertIsNone(result["positive"])
self.assertIsNone(result["negative"])
def test_upstream_sampler_through_vaedecode_vaeencode_is_compared(self):
# The latent continues the first stage's through lineage nodes: the
# stages are compared (spec 9.3), equal prompts resolve, others differ.
result = _inspect(_image_to_image(parity.POS), output_node_id="63")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "61"))
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
result = _inspect(_image_to_image(OTHER), output_node_id="63")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
[item["sampler_id"] for item in result["candidates"]], ["61", "3"]
)
# An upstream sampler reached through a non-latent input is a stage too.
source = _latent_from(
{
@@ -2959,9 +2964,15 @@ class TestLatentAncestry(unittest.TestCase):
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"], {"id": "60", "class_type": "MysteryUpscaler"})
# The listed stage is now an earlier sampler stage in the ancestry.
# The listed stage is now an earlier sampler stage, continued through
# VAEEncode: its prompt equals this stage's, so the chain resolves.
source["60"]["class_type"] = "UltimateSDUpscale"
result = _inspect(source, output_node_id="9")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "3"))
# Reached only through an unlisted image node, it stays unattributed.
source["61"] = {"class_type": "MysteryFilter", "inputs": {"image": ["60", 0]}}
source["5"]["inputs"]["pixels"] = ["61", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_latent_walk_budget_exhaustion_is_work_limit(self):
+419
View File
@@ -0,0 +1,419 @@
"""Multi-stage chains joined through image-space pass-through nodes.
A hires-fix or refiner run decodes sampler A's latent, filters, upscales and
scales the image, then encodes it again for sampler B. The semantics table's
``lineage`` section lists the nodes through which the latent or image carries
on unchanged in kind. When B's latent descends from A only through those nodes
and the stages share one pair, the chain resolves to it. A last stage that adds
conditioning to the base prompt (ConditioningCombine of the base and a style
trigger) keeps the base pair: the floor legacy linked. Everything else stays
ambiguous or unresolved. Ingestion (the default profile) is unchanged (D1).
"""
import json
import unittest
from utils import prompt_pair_resolver as resolver
from utils.prompt_inspector import inspect_prompt_evidence
from utils.prompt_pair_resolver import resolve_prompt_pair
try:
from tests import passthrough_graphs as shapes
except ImportError: # discovered with ``-s tests``
import passthrough_graphs as shapes
def _inspect(graph, **options):
return resolve_prompt_pair(
{"prompt": graph}, profile=resolver.INSPECTION_PROFILE, **options
)
def _texts(result):
return tuple(
result[role]["text"] if isinstance(result[role], dict) else None
for role in ("positive", "negative")
)
class TestHiresFixChain(unittest.TestCase):
def test_shared_pair_resolves_through_image_space_pass_through(self):
graph = shapes.hires_fix()
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(result["sampler_id"], "20")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertIsNone(result["reason_code"])
def test_the_earlier_stage_is_judged_in_its_chain(self):
# Sampler 10 only feeds sampler 20, so the file is 20's: one candidate.
result = _inspect(shapes.hires_fix())
self.assertEqual(result["candidates"], [])
def test_inspector_shows_the_shared_prompt(self):
evidence = inspect_prompt_evidence({"prompt": json.dumps(shapes.hires_fix())})
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
[evidence["prompts"][role]["text"] for role in ("positive", "negative")],
[shapes.POSITIVE, shapes.NEGATIVE],
)
def test_different_stage_prompts_are_multistage_ambiguous(self):
result = _inspect(shapes.hires_fix(second_positive=shapes.OTHER))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
sorted(_texts(item)[0] for item in result["candidates"]),
sorted([shapes.POSITIVE, shapes.OTHER]),
)
def test_an_unlisted_image_node_breaks_the_lineage(self):
# The walk still finds the earlier sampler, but not through listed
# pass-through nodes: the stages are never compared.
result = _inspect(shapes.hires_fix(filter_class="MysteryImageEffect"))
self.assertEqual(result["state"], "ambiguous")
later = next(
item for item in result["candidates"] if item["sampler_id"] == "20"
)
self.assertEqual(later["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_second_earlier_sampler_in_the_ancestry_stays_unattributed(self):
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {
"class_type": "ImageBlend",
"inputs": {"image1": ["14", 0], "image2": ["31", 0], "blend_factor": 0.5},
}
graph["16"]["inputs"]["image"] = ["32", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_direct_latent_link_is_a_chain_too(self):
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "20"))
def test_an_unreadable_earlier_stage_blocks_the_chain(self):
graph = shapes.hires_fix(second_positive=shapes.POSITIVE)
graph["10"]["inputs"]["positive"] = ["6", 0]
graph["6"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryPrompt")
def test_generated_text_stays_untraceable(self):
result = _inspect(shapes.generated_text_chain())
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "TextGenerate")
def test_ingestion_is_unchanged(self):
# D1: the default profile keeps its historical latent rule.
result = resolve_prompt_pair({"prompt": shapes.hires_fix()})
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(
{item["reason"] for item in result["candidates"]},
{"Unsupported latent ancestry"},
)
class TestFluxThreeStage(unittest.TestCase):
def test_added_style_conditioning_keeps_the_base_pair(self):
graph = shapes.flux_three_stage()
for anchor in (None, "61"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
self.assertEqual(result["sampler_id"], "50")
# D126: the positive says a later stage added conditioning.
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
self.assertEqual(
result["negative"]["provenance"]["text_source"],
"zeroed-conditioning",
)
def test_any_added_conditioning_is_labelled_never_measured(self):
# Ruling I2: a whole other prompt combined in is labelled the same way;
# no length or "short trigger" heuristic.
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], shapes.POSITIVE)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
def test_trigger_words_and_added_conditioning_carry_both_labels(self):
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_AND_ADDED_SOURCE,
)
def test_a_custom_advanced_last_stage_keeps_its_failure(self):
# Review M2: a guider's combined positive is not floored.
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
result = _inspect(graph, output_node_id="61")
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_concat_also_adds_to_the_base_prompt(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningConcat"))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
def test_an_average_blends_the_base_away(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningAverage"))
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_combine_without_the_base_prompt_is_not_the_base_pair(self):
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_different_last_negative_is_not_the_base_pair(self):
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_different_base_stages_are_never_floored(self):
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertNotEqual(result["state"], "resolved")
def test_a_rerouted_combine_is_followed(self):
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
def test_an_unreadable_last_negative_or_encoder_parts_keep_the_failure(self):
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
self.assertEqual(_inspect(graph)["reason_code"], "composite")
# Differing encoder parts are a composite too, but not an added stage.
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
self.assertEqual(result["node"]["class_type"], "CLIPTextEncodeSDXL")
def test_inspector_shows_the_base_prompt(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.flux_three_stage())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(evidence["prompts"]["positive"]["text"], shapes.POSITIVE)
class TestRunTimePromptManagerParts(unittest.TestCase):
"""Maintainer option A: a run-time prepend or append is left out, labelled."""
def test_trigger_words_prepend_resolves_to_the_prompt_manager_text(self):
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(
shapes.hires_fix_with_trigger_words(), output_node_id=anchor
)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
provenance = result["positive"]["provenance"]
self.assertEqual(
provenance["text_source"], resolver.RUNTIME_PARTS_SOURCE
)
self.assertEqual(result["positive"]["storage_text"], shapes.POSITIVE)
# The negative is untouched by the positive's run-time part.
self.assertEqual(
result["negative"]["provenance"]["text_source"], "graph-inferred"
)
def test_append_and_a_single_sampler_are_labelled_too(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = "masterpiece"
graph["2"]["inputs"]["append_text"] = ["8", 0]
graph["20"]["inputs"]["latent_image"] = ["4", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "resolved")
# The literal prepend is still combined exactly as before.
self.assertEqual(_texts(result)[0], "masterpiece " + shapes.POSITIVE)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
def test_the_inspector_shows_the_prompt_and_its_source(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.hires_fix_with_trigger_words())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
evidence["prompts"]["positive"],
{
"status": "resolved",
"text": shapes.POSITIVE,
"source": resolver.RUNTIME_PARTS_SOURCE,
},
)
def test_literal_and_traceable_parts_are_combined_as_before(self):
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
result = _inspect(graph)
self.assertEqual(_texts(result)[0], "trig9word " + shapes.POSITIVE)
self.assertEqual(result["reason"], "")
self.assertEqual(
result["positive"]["provenance"]["text_source"], "graph-inferred"
)
def test_only_a_run_time_feeder_is_left_out(self):
# An unknown feeder stays unsupported, naming it.
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
result = _inspect(graph)
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryWords")
# A run-time node deeper inside the part makes the whole part run time,
# but it is not a direct feeder: never leave out a traced literal.
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_main_text_from_a_run_time_node_is_never_left_out(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = ""
graph["2"]["inputs"]["text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_label_survives_nodes_between_the_prompt_manager_and_sampler(self):
# Review C1: encoders, chained PromptManagers and concatenations.
expected = {
"text-encode": shapes.POSITIVE,
"chained": shapes.POSITIVE,
"concat": shapes.POSITIVE + ", extra",
}
for shape, text in expected.items():
with self.subTest(shape=shape):
result = _inspect(shapes.trigger_words_through(shape))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], text)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_PARTS_SOURCE,
)
def test_a_partial_prompt_never_equals_a_complete_one(self):
# Review C2: never collapse a branch that left out trigger words with
# one that did not, nor compare such stages as equal.
result = _inspect(shapes.branches_with_and_without_trigger_words())
self.assertEqual(
(result["state"], result["reason"]),
("ambiguous", "Multiple sampler branches"),
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(
(result["state"], result["reason_code"]),
("ambiguous", "multistage-differs"),
)
# Two PromptManagers leaving out two different toggles differ too.
graph["8b"] = shapes.trigger_toggle("otherword")
graph["9"]["inputs"]["prepend_text"] = ["8b", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["reason_code"], "multistage-differs")
def test_only_listed_run_time_classes_are_left_out(self):
# Review M1: a JoinStringMulti with a linked list flag is run time
# too, but it is not a listed run-time node: never left out.
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "runtime-value")
)
self.assertEqual(result["node"]["class_type"], "JoinStringMulti")
def test_ingestion_is_unchanged(self):
result = resolve_prompt_pair({"prompt": shapes.hires_fix_with_trigger_words()})
self.assertEqual(result["state"], "ambiguous")
class TestStillAmbiguous(unittest.TestCase):
def test_edit_batch_with_distinct_prompts_stays_ambiguous(self):
result = _inspect(shapes.edit_batch(4))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason"], "Multiple sampler branches")
self.assertEqual(len({_texts(item)[0] for item in result["candidates"]}), 4)
if __name__ == "__main__":
unittest.main()
+1053
View File
File diff suppressed because it is too large Load Diff
+335
View File
@@ -0,0 +1,335 @@
"""Needs review queues a file only when there is something to choose.
The maintainer's live queue (#284 / #288) held 39 items whose 11 candidates
all read "unresolved", "No positive / No negative": the ingestion resolver
found several sampler branches but could trace no prompt on any of them.
An ambiguous result is reviewable only when at least two candidates carry a
resolved prompt text. With none, the file could not be traced; with exactly
one, the other branches may still have made the file, so the one text is
not certain either (the resolver never flattens unknown semantics). Both
stay unresolved and are never queued, and a rescan removes the pending,
undecided items such files already have.
"""
import os
import unittest
from unittest.mock import patch
try:
from tests.test_review import DELIMITERS, write_png
from tests.test_review_outcome import OutcomeTestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_review import DELIMITERS, write_png
from test_review_outcome import OutcomeTestCase
from database.review_schema import evidence_offers_choice
from utils import review_capture
def opaque_graph(*opaque, traced=()):
"""Several KSamplers: ``opaque`` positives come from an encoder the
ingestion resolver does not know, ``traced`` ones from CLIPTextEncode.
The first sampler is opaque, so the scan's browser-parser fallback links
nothing either.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
}
sides = [("OpaqueTextEncode", "prompt", t) for t in opaque]
sides += [("CLIPTextEncode", "text", t) for t in traced]
for index, (encoder_class, key, text) in enumerate(sides):
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
graph[encoder] = {"class_type": encoder_class, "inputs": {key: text}}
graph[sampler] = {
"class_type": "KSampler",
"inputs": {
"model": ["1", 0],
"positive": [encoder, 0],
"negative": ["2", 0],
},
}
return graph
def _always_reviewable(_resolution):
return True
class ChoosableTestCase(OutcomeTestCase):
def _record_pre_fix(self, name):
"""An item as the pre-fix recorder stored it (no choosable test)."""
with patch.object(review_capture, "reviewable", _always_reviewable):
self._record(name)
def _write(self, name, graph):
write_png(self.output_dir / name, prompt=graph)
return self.output_dir / name
class TestChoosableCandidates(unittest.TestCase):
def _resolution(self, *states):
candidates = []
for index, state in enumerate(states):
side = {"text": f"t{index}"} if state == "resolved" else None
candidates.append(
{"sampler_id": str(index), "state": state, "positive": side}
)
return {"state": "ambiguous", "candidates": candidates}
def test_two_resolved_candidates_are_reviewable(self):
resolution = self._resolution("resolved", "resolved", "unresolved")
self.assertTrue(review_capture.reviewable(resolution))
def test_none_or_one_resolved_candidate_is_not_reviewable(self):
for states in ((), ("unresolved",) * 11, ("resolved", "unresolved")):
with self.subTest(states=states):
resolution = self._resolution(*states)
self.assertFalse(review_capture.reviewable(resolution))
def test_blank_or_missing_text_does_not_count(self):
resolution = self._resolution("resolved", "resolved")
resolution["candidates"][0]["positive"] = {"text": " "}
self.assertFalse(review_capture.reviewable(resolution))
resolution["candidates"][0]["positive"] = None
self.assertFalse(review_capture.reviewable(resolution))
def test_nested_ambiguous_stage_counts_its_own_candidates(self):
inner = self._resolution("resolved", "resolved")["candidates"]
resolution = {
"state": "ambiguous",
"candidates": [{"state": "ambiguous", "candidates": inner}],
}
self.assertTrue(review_capture.reviewable(resolution))
def test_a_result_that_is_not_ambiguous_is_not_reviewable(self):
resolution = self._resolution("resolved", "resolved")
resolution["state"] = "resolved"
self.assertFalse(review_capture.reviewable(resolution))
class TestStoredEvidenceOffersChoice(unittest.TestCase):
def test_unlimited_evidence_needs_two_choosable_candidates(self):
side = {"text": "cat"}
two = [{"state": "resolved", "positive": side}] * 2
self.assertTrue(evidence_offers_choice({"candidates": two}))
one = two[:1] + [{"state": "unresolved", "positive": None}]
self.assertFalse(evidence_offers_choice({"candidates": one}))
def test_limited_evidence_offers_a_choice_only_with_two_candidates(self):
self.assertTrue(
evidence_offers_choice({"limited": True, "candidate_total": 18})
)
for total in (0, 1, None, "18", True):
with self.subTest(total=total):
evidence = {"limited": True, "candidate_total": total}
self.assertFalse(evidence_offers_choice(evidence))
def test_malformed_evidence_offers_nothing(self):
for evidence in (None, [], {"candidates": "x"}, {}):
with self.subTest(evidence=evidence):
self.assertFalse(evidence_offers_choice(evidence))
class TestNothingToChooseIsNotQueued(ChoosableTestCase):
async def test_no_candidate_with_text_could_not_be_traced(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_items"], 0)
(row,) = job["outcomes"]["review"]
self.assertEqual(row["path"], "none.png")
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"]["review"], 1, job["counters"])
self.assertEqual(self._capture_files(), [])
async def test_exactly_one_candidate_with_text_stays_unresolved(self):
"""One traced branch beside untraced ones is not certain: no link."""
self._roots()
self._write("one.png", opaque_graph("cat", traced=("dog",)))
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"].get("added", 0), 0, job["counters"])
self.assertEqual(job["counters"].get("known", 0), 0, job["counters"])
async def test_two_candidates_with_text_are_queued(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
(row,) = job["outcomes"]["review"]
self.assertIn("more than one prompt could match", row["detail"])
async def test_parameters_delimiter_ambiguity_is_not_queued(self):
"""Two A1111 negative delimiters offer no candidate to choose."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
def test_recorder_reports_untraced_and_stores_nothing(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
for name in ("none.png", "one.png"):
with self.subTest(name=name):
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertEqual(outcome, "untraced")
self.assertEqual(self._items(), [])
self.assertEqual(self._capture_files(), [])
def test_monitor_does_not_queue_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
class TestUntracedItemsLeaveTheQueue(ChoosableTestCase):
async def test_rescan_removes_pending_items_without_a_choice(self):
"""The maintainer's 39 items: queued before the rule, gone on rescan."""
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
for name in ("none.png", "one.png", "two.png"):
self._record_pre_fix(name)
revision = self._review_revision()
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(self._pending(), 1)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_untraced"], 2)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertIn(
"= 2 images left Needs review · nothing to choose", "\n".join(job["log"])
)
async def test_decided_items_of_untraced_files_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
self._write(name, opaque_graph("cat", traced=(name, name + "2")))
self._record_pre_fix(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# The files now carry graphs without a choice, queued pre-fix.
for name in ("sel.png", "unres.png", "pending.png"):
self._write(name, opaque_graph("a cow", "dog"))
self._record_pre_fix(name)
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_untraced"], 3)
async def test_an_item_with_a_choice_survives_beside_an_untraced_one(self):
"""Removal judges each item by its own evidence: an item that offers
a choice is not removed although the file's new bytes offer none."""
self._roots()
self._write("f.png", opaque_graph("x", traced=("cat", "dog")))
self._record_pre_fix("f.png")
(choosable,) = self._items()
self._write("f.png", opaque_graph("a cow", "dog"))
self._record_pre_fix("f.png")
self.assertEqual(self._pending(), 2)
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [choosable["id"]])
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_limited_items_go_only_without_enough_candidates(self):
"""A pre-fix delimiter item (no candidate) goes; a limited item of 18
candidates may hold a choice and stays, though the bytes now offer
none."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
self._record_pre_fix("delims.png")
many = [f"prompt {n}" for n in range(17)]
self._write("many.png", opaque_graph("x", traced=many))
self._record_pre_fix("many.png")
kept = [i["id"] for i in self._items() if "many" in i["locator"]]
self._write("many.png", opaque_graph("a cow", "dog"))
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], kept)
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_choosable_pending_item_stays(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
await self._scan()
(item,) = self._items()
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"]["review_untraced"], 0)
async def test_a_failed_removal_is_logged_and_counted_not_fatal(self):
import logging
import sqlite3
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
with (
patch.object(
self.api.db,
"settle_untraced_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan("all")
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
await self._scan("all")
self.assertEqual(self._pending(), 0)
def test_monitor_removes_the_pending_item_of_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
# A pre-fix item of other bytes keeps the stat shortcut out of the way.
self._write("none.png", opaque_graph("a cow", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
if __name__ == "__main__":
unittest.main()
+374
View File
@@ -0,0 +1,374 @@
"""Needs review takes only files whose final outcome is ambiguous and unlinked.
The maintainer's live scan (#284 / #288) queued every file the strict
resolver found ambiguous, also those the legacy browser-parser fallback then
linked and those already linked, so the queue (10,232) was far larger than
the scan's Needs review group. The rule now: a file is queued only when the
resolver finds it ambiguous, the fallback does not link it and no saved row
links it. A pending, undecided item whose file is linked (by the fallback, a
later resolution or the monitor) is removed on the next scan or monitor
pass; an item that carries a decision is never touched.
"""
import json
import logging
import os
import sqlite3
import unittest
from unittest.mock import patch
try:
from tests.test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
except ImportError: # discovered with tests/ as the top-level directory
from test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
from utils.prompt_tracker import PromptTracker
from utils import review_capture
def nan_stamped(graph):
"""ComfyUI's ``is_changed: [NaN]`` stamp (written as a bare NaN literal)."""
for node in graph.values():
node["is_changed"] = [float("nan")]
return graph
class OutcomeTestCase(ReviewTestCase):
def _pending(self):
return self.api.db.library_counts()["review_pending"]
def _record(self, name):
"""An item as the pre-fix producers recorded it (any ambiguous file)."""
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertIn(outcome, ("recorded", "limited"))
def _decide(self, item, outcome):
conn = self.api.db.model.get_connection()
candidate = None
if outcome == "selected":
candidate = json.loads(item["evidence"])["candidates"][0]["hash"]
with conn:
conn.execute(
"UPDATE media_review_items SET state=?, revision=1 WHERE id=?",
(outcome, item["id"]),
)
conn.execute(
"INSERT INTO media_review_decisions(id,item_id,decision_revision,"
"created_at,outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)",
(
f"00000000-0000-4000-8000-{len(self._decisions()) + 1:012d}",
item["id"],
1,
NOW,
outcome,
candidate,
'{"kind":"user-selection"}',
),
)
def _decisions(self):
conn = self.api.db.model.get_connection()
return [
tuple(row)
for row in conn.execute(
"SELECT id,item_id,outcome FROM media_review_decisions ORDER BY id"
)
]
def _link(self, path, text="already saved"):
prompt_id = self.api.db.save_prompt(
text=text, prompt_hash=generate_prompt_hash(text)
)
self.api.db.link_image_to_prompt(prompt_id, str(path))
return prompt_id
@staticmethod
def _rows(job, group):
return sorted(row["path"] for row in job["outcomes"].get(group, []))
class TestOnlyUnlinkedAmbiguousFilesAreQueued(OutcomeTestCase):
async def test_nan_stamped_graph_the_fallback_links_is_not_queued(self):
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self.assertIn(b"NaN", path.read_bytes())
job = await self._scan()
self.assertEqual(job["counters"]["added"], 1, job["counters"])
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._pending(), 0)
async def test_fallback_link_to_a_known_prompt_is_not_queued(self):
self._roots()
write_png(self.output_dir / "a.png", prompt=ambiguous_graph("cat", "dog"))
await self._scan()
write_png(self.output_dir / "b.png", prompt=ambiguous_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
async def test_already_linked_file_is_not_queued(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._link(path)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._rows(job, "review"), [])
self.assertEqual(self._rows(job, "known"), ["linked.png"])
async def test_genuinely_ambiguous_file_is_queued(self):
self._roots()
path = self.output_dir / "two.png"
write_png(path, prompt=nan_stamped(unlinkable_graph("cat", "dog")))
# Beside it, a file the fallback links: only the first is queued.
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["state"], "pending")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertEqual(self._rows(job, "review"), ["two.png"])
self.assertEqual(self._pending(), 1)
self.assertIn("? 1 image added to Needs review", "\n".join(job["log"]))
async def test_counter_and_queue_agree_with_the_review_group(self):
"""``review_items`` and the queue total equal the review group less
its unresolved files, on the first scan and on a rescan."""
self._roots()
for name in ("u1.png", "u2.png"):
write_png(self.output_dir / name, prompt=unlinkable_graph("cat", name))
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
unresolved = {"1": {"class_type": "KSampler", "inputs": {}}}
write_png(self.output_dir / "none.png", prompt=unresolved)
write_png(self.output_dir / "one.png", prompt=resolved_graph())
for _ in range(2):
job = await self._scan()
review = self._rows(job, "review")
unresolved_rows = sorted(
row["path"]
for row in job["outcomes"]["review"]
if "could not be traced" in row["detail"]
)
queued = [name for name in review if name not in unresolved_rows]
# delims.png has nothing to choose: "could not be traced".
self.assertEqual(queued, ["u1.png", "u2.png"], job)
self.assertIn("delims.png", unresolved_rows)
self.assertEqual(job["scope"]["review_items"], 2)
self.assertEqual(self._pending(), 2)
self.assertIn("? 2 images already in Needs review", "\n".join(job["log"]))
class TestQueueSelfHeals(OutcomeTestCase):
async def test_pending_item_of_a_fallback_linked_file_is_removed(self):
"""The maintainer's case: queued by the old rule, linked on rescan."""
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self._record("nan.png")
revision = self._review_revision()
self.assertEqual(self._pending(), 1)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_settled"], 1)
self.assertIn("= 1 image left Needs review · now linked", "\n".join(job["log"]))
async def test_pending_item_of_an_already_linked_file_is_removed(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("linked.png")
self._link(path)
await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_pending_item_of_a_skipped_known_file_is_removed(self):
"""A file the "new" scope skips as known still settles its old item."""
self._roots()
path = self.output_dir / "one.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("one.png")
write_png(path, prompt=resolved_graph())
first = await self._scan()
self.assertEqual(self._rows(first, "added"), ["one.png"])
# An item a pre-fix scan left for the file's old bytes.
self.assertEqual(self._items(), [])
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 9, 0))
self._record("one.png")
write_png(path, prompt=resolved_graph())
second = await self._scan()
self.assertEqual(self._rows(second, "known"), ["one.png"])
self.assertEqual(self._items(), [])
async def test_a_failed_settle_is_logged_and_counted_not_fatal(self):
self._roots()
write_png(self.output_dir / "nan.png", prompt=ambiguous_graph("cat", "dog"))
self._record("nan.png")
with (
patch.object(
self.api.db,
"settle_linked_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan()
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertEqual(job["scope"]["review_settled"], 0)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
# The next scan settles it.
await self._scan()
self.assertEqual(self._pending(), 0)
async def test_still_unlinked_pending_item_stays(self):
self._roots()
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
await self._scan()
(item,) = self._items()
job = await self._scan()
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"].get("review_settled", 0), 0)
async def test_decided_items_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
write_png(self.output_dir / name, prompt=ambiguous_graph("cat", name))
self._record(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# A pending neighbour the fallback links: it alone leaves the queue.
write_png(self.output_dir / "pending.png", prompt=ambiguous_graph("x", "y"))
self._record("pending.png")
self.assertEqual(self._pending(), 1)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_settled"], 1)
class TestMonitorRule(OutcomeTestCase):
def _handler(self):
handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db))
handler.review_recorder = self.api.record_review_file
return handler
def _context(self):
prompt_id = self.api.db.save_prompt(
text="latest", prompt_hash=generate_prompt_hash("latest")
)
return {"id": prompt_id, "text": "latest", "fallback": True}
async def test_monitor_linked_ambiguous_output_is_not_queued(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
async def test_monitor_link_removes_the_files_pending_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
self.assertEqual(self._pending(), 1)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_monitor_never_touches_a_decided_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
(item,) = self._items()
self._decide(item, "unresolved")
before = (self._items(), self._decisions())
other = self.output_dir / "other.png"
write_png(other, prompt=graph, color=(0, 0, 9))
self._record("other.png")
handler, context = self._handler(), self._context()
handler.link_image_to_prompt(str(path), context, {"prompt": graph})
handler.link_image_to_prompt(str(other), context, {"prompt": graph})
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
async def test_record_review_file_applies_the_scan_rule(self):
self._roots()
write_png(self.output_dir / "fallback.png", prompt=ambiguous_graph("a", "b"))
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
linked = self.output_dir / "linked.png"
write_png(linked, prompt=unlinkable_graph("cat", "dog"))
self._link(linked)
record = self.api.record_review_file
self.assertEqual(record(str(self.output_dir / "fallback.png")), "skipped")
self.assertEqual(record(str(linked)), "linked")
self.assertEqual(record(str(self.output_dir / "two.png")), "recorded")
self.assertEqual(len(self._items()), 1)
if __name__ == "__main__":
unittest.main()
+918
View File
@@ -0,0 +1,918 @@
"""Needs review schema group (#249): install, validation and restore safety."""
import hashlib
import json
import re
import sqlite3
import tempfile
import unittest
from contextlib import closing
from pathlib import Path
from unittest.mock import patch
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
from database.models import PromptModel
from database.operations import PromptDatabase
try: # guarded so a missing module fails assertions, not test loading
from database.review_schema import (
canonical_evidence,
evidence_digest,
migrate_review_state,
review_schema_error,
)
except ImportError:
canonical_evidence = None
evidence_digest = None
migrate_review_state = None
review_schema_error = None
METADATA = "media_review_metadata"
ITEMS = "media_review_items"
DECISIONS = "media_review_decisions"
TABLES = (METADATA, ITEMS, DECISIONS)
ANNOTATION_TABLES = (
"media_annotation_metadata",
"media_annotation_files",
"media_annotation_bytes",
"media_image_tags",
)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
f"CREATE TABLE {METADATA}(singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
"CHECK(schema_version=1), review_index_revision INTEGER NOT NULL "
"DEFAULT 0 CHECK(review_index_revision>=0))"
),
ITEMS: (
f"CREATE TABLE {ITEMS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
"byte_id TEXT NOT NULL REFERENCES media_annotation_bytes(id) "
"ON DELETE RESTRICT, spelled_path TEXT NOT NULL CHECK(length(CAST("
"spelled_path AS BLOB)) BETWEEN 1 AND 16384), "
"resolver_version INTEGER NOT NULL, "
"evidence_hash TEXT NOT NULL CHECK(length(evidence_hash)=64 AND "
"evidence_hash NOT GLOB '*[^0-9a-f]*'), evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"candidate_count INTEGER NOT NULL CHECK(candidate_count BETWEEN 0 AND 16), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), state TEXT NOT NULL "
"CHECK(state IN ('pending','selected','unresolved')), revision INTEGER "
"NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, updated_at TEXT "
"NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS: (
f"CREATE TABLE {DECISIONS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"item_id TEXT NOT NULL REFERENCES {ITEMS}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
"(length(candidate_hash)=64 AND candidate_hash NOT GLOB '*[^0-9a-f]*')), "
"evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
INDEXES = {
"idx_media_review_items_queue": (
f"CREATE INDEX idx_media_review_items_queue ON {ITEMS}"
"(state, created_at, id)"
),
"idx_media_review_decisions_pair": (
f"CREATE INDEX idx_media_review_decisions_pair ON {DECISIONS}(pair_id)"
),
"idx_media_review_decisions_image": (
f"CREATE INDEX idx_media_review_decisions_image ON {DECISIONS}(image_id)"
),
}
AUTOINDEXES = {ITEMS: 2, DECISIONS: 2}
LEGACY_TABLES = (
"prompts",
"generated_images",
"tags",
"prompt_tags",
"prompt_pairs",
"prompt_pair_tags",
"image_favorites",
) + ANNOTATION_TABLES
FILE_ID = "0f8e4b2a-1c3d-4e5f-8a9b-0c1d2e3f4a5b"
BYTE_ID = "6a1b2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d"
ITEM_ID = "1d2e3f4a-5b6c-4d7e-8f9a-0b1c2d3e4f5a"
DECISION_ID = "2e3f4a5b-6c7d-4e8f-9a0b-1c2d3e4f5a6b"
OTHER_ID = "7b2c3d4e-5f6a-4b7c-9d0e-1f2a3b4c5d6e"
DIGEST = hashlib.sha256(b"img").hexdigest()
CANDIDATE = hashlib.sha256(b"candidate").hexdigest()
NOW = "2026-10-22T00:00:00.000000+00:00"
EVIDENCE = '{"candidates":[],"sha256":"' + DIGEST + '"}'
EVIDENCE_HASH = hashlib.sha256(EVIDENCE.encode("ascii")).hexdigest()
def _compact(sql):
return re.sub(r"\s+", "", sql or "").lower()
def expected_catalog():
objects = {("table", name, _compact(sql)) for name, sql in DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in INDEXES.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def catalog(conn):
names = tuple(TABLES) + tuple(INDEXES)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return {
(kind, name, _compact(sql))
for kind, name, sql in conn.execute(
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + tuple(TABLES),
)
}
def rows(conn, sql, *params):
"""Plain tuples whatever row factory *conn* uses."""
return [tuple(row) for row in conn.execute(sql, params)]
def metadata(conn):
return rows(conn, f"SELECT schema_version,review_index_revision FROM {METADATA}")
def table_rows(conn, tables):
found = {}
for table in tables:
present = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
).fetchone()
if present:
found[table] = rows(conn, f"SELECT * FROM {table}") # nosec B608
return found
def insert_byte(conn, file_id=FILE_ID, byte_id=BYTE_ID, locator="/out/img.png"):
conn.execute(
"INSERT INTO media_annotation_files VALUES(?,?,?)", (file_id, locator, NOW)
)
conn.execute(
"INSERT INTO media_annotation_bytes VALUES(?,?,?,?,?,?,?,0)",
(byte_id, file_id, DIGEST, 3, 1_700_000_000_000_000_000, NOW, NOW),
)
def insert_item(conn, *, item_id=ITEM_ID, evidence=EVIDENCE, candidates=2, **extra):
values = {
"id": item_id,
"byte_id": BYTE_ID,
"spelled_path": "/out/img.png",
"resolver_version": 1,
"evidence_hash": hashlib.sha256(evidence.encode("utf-8")).hexdigest(),
"evidence": evidence,
"candidate_count": candidates,
"limited": 0,
"state": "pending",
"revision": 0,
"created_at": NOW,
"updated_at": NOW,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {ITEMS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_decision(conn, *, decision_id=DECISION_ID, **extra):
values = {
"id": decision_id,
"item_id": ITEM_ID,
"decision_revision": 1,
"created_at": NOW,
"outcome": "selected",
"candidate_hash": CANDIDATE,
"evidence": EVIDENCE,
"pair_id": None,
"image_id": None,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {DECISIONS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_group_rows(conn, **item):
insert_byte(conn)
insert_item(conn, **item)
insert_decision(conn)
def populate_legacy(conn):
conn.execute(
"INSERT INTO prompts(id,text,hash,category,notes,run_count) VALUES"
"(7,'positive','p','keep','notes',12)"
)
conn.execute("INSERT INTO prompts(id,text,hash) VALUES(9,'negative','n')")
conn.execute(
"INSERT INTO generated_images(id,prompt_id,image_path,filename,file_path) "
"VALUES(4,7,'/tmp/image.png','image.png','/tmp/image.png')"
)
conn.execute("INSERT INTO tags(id,name) VALUES(3,'keep-tag')")
conn.execute("INSERT INTO prompt_tags VALUES(7,3)")
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(6,7,9)"
)
conn.execute("INSERT INTO prompt_pair_tags VALUES(6,3)")
conn.execute("INSERT INTO image_favorites VALUES('root','a.png',?)", (NOW,))
def full_capacity_evidence():
"""Sixteen candidates of control, lone-surrogate, astral and combining
text, exactly 8 MiB."""
text = '\x00\x07\x1f
\ud800 é é "q" \\ \U0001f9ea'
candidates = [
{
"hash": hashlib.sha256(str(n).encode()).hexdigest(),
"negative": {"text": text * 3},
"positive": {"text": text * 5},
"sampler_id": str(n),
"state": "resolved",
}
for n in range(16)
]
value = {"candidates": candidates, "pad": ""}
base = len(canonical_evidence(value).encode("utf-8"))
value["pad"] = "x" * (MAX_EVIDENCE_BYTES - base)
return value
class ReviewSchemaTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.addCleanup(assert_closed, self, self.temp.name)
self.addCleanup(PromptModel.reset_schema_cache)
self.models = []
self.addCleanup(self.close_models)
self.path = str(Path(self.temp.name) / "live.db")
def close_models(self):
for model in self.models:
model.close_all()
def model(self, path=None):
model = PromptModel(path or self.path)
self.models.append(model)
return model
def require_module(self):
self.assertIsNotNone(
review_schema_error, "database.review_schema is not implemented"
)
def drop_tables(self, path, tables=TABLES):
"""Remove tables from a closed database file (an old backup)."""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in reversed(tables):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
insert_group_rows(conn)
snapshot = table_rows(conn, TABLES)
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(table_rows(conn, TABLES), snapshot)
# Running the migration again on a current database changes nothing.
migrate_review_state(conn)
self.assertEqual(table_rows(conn, TABLES), snapshot)
model.close_all()
# Items and decisions keep the byte record and the item they cite.
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
with self.assertRaises(sqlite3.IntegrityError):
conn.execute("DELETE FROM media_annotation_files")
with self.assertRaises(sqlite3.IntegrityError):
conn.execute(f"DELETE FROM {ITEMS}")
self.assertEqual(table_rows(conn, TABLES), snapshot)
def test_startup_validates_stored_evidence_once(self):
"""A reopen streams the evidence once and the annotation group twice
(its own install), never again for the review group (I3)."""
self.require_module()
import database.annotation_schema as annotation_module
import database.review_schema as review_module
model = self.model()
with model.get_connection() as conn:
insert_group_rows(conn)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
with (
patch.object(
review_module,
"_evidence_error",
wraps=review_module._evidence_error,
) as evidence,
patch.object(
review_module,
"annotation_schema_error",
wraps=review_module.annotation_schema_error,
) as nested,
patch.object(
annotation_module,
"_value_error",
wraps=annotation_module._value_error,
) as annotation_rows,
):
model = self.model()
model.get_connection()
self.assertEqual(evidence.call_count, 1)
self.assertEqual(nested.call_count, 0)
self.assertEqual(annotation_rows.call_count, 2)
model.close_all()
def test_old_database_without_group_migrates_without_touching_legacy_rows(self):
self.require_module()
for label, dropped in (
("annotations only", TABLES),
("predates annotations", ANNOTATION_TABLES + TABLES),
):
with self.subTest(label):
path = str(Path(self.temp.name) / f"{label.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if dropped == TABLES:
insert_byte(conn)
model.close_all()
self.drop_tables(path, dropped)
with closing(sqlite3.connect(path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(
review_schema_error(raw, allow_missing=False),
"Missing review queue schema",
)
legacy_rows_before = table_rows(raw, LEGACY_TABLES)
self.assertTrue(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[
0
]
)
PromptModel.reset_schema_cache()
model = self.model(path)
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
counts = [
conn.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0]
for t in (ITEMS, DECISIONS)
]
legacy_rows_after = {
table: value
for table, value in table_rows(conn, LEGACY_TABLES).items()
if table in legacy_rows_before
}
self.assertEqual(counts, [0, 0])
self.assertEqual(legacy_rows_after, legacy_rows_before)
self.assertEqual(
PromptModel.verify_database_file(path, require_library_state=True),
(True, "ok"),
)
model.close_all()
def test_annotations_only_installation_is_valid(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
model.close_all()
self.drop_tables(self.path)
with closing(sqlite3.connect(self.path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
def test_review_group_without_annotation_group_is_invalid(self):
self.require_module()
model = self.model()
model.close_all()
self.drop_tables(self.path, ANNOTATION_TABLES)
with closing(sqlite3.connect(self.path)) as raw:
for allow_missing in (True, False):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("image annotation", error.lower())
self.assertFalse(PromptModel.verify_database_file(self.path)[0])
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model()
# An invalid annotation group invalidates the review group beside it.
path = self.tampered(
"bad annotations",
("UPDATE media_annotation_bytes SET sha256=upper(sha256)",),
checks_off=True,
)
with closing(sqlite3.connect(path)) as raw:
self.assertIn("image annotation", review_schema_error(raw).lower())
def tampered(self, name, statements, *, checks_off=False, rows_in=True):
"""A closed current database whose group was altered after the fact."""
path = str(Path(self.temp.name) / f"{name.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if rows_in:
insert_group_rows(conn)
model.close_all()
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
if checks_off:
raw.execute("PRAGMA ignore_check_constraints=ON")
for statement in statements:
if isinstance(statement, tuple):
raw.execute(*statement)
else:
raw.execute(statement)
return path
def assert_rejected_unrepaired(self, path):
with closing(sqlite3.connect(path)) as raw:
schema_before = raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall()
rows_before = table_rows(raw, TABLES)
for allow_missing in (False, True):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("review", error.lower())
self.assertFalse(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[0]
)
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model(path)
PromptModel.reset_schema_cache()
with closing(sqlite3.connect(path)) as raw:
self.assertEqual(
raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall(),
schema_before,
)
self.assertEqual(table_rows(raw, TABLES), rows_before)
def test_partial_or_malformed_group_is_rejected_without_repair(self):
self.require_module()
cases = {
"missing metadata": (f"DROP TABLE {METADATA}",),
"missing items": (f"DROP TABLE {DECISIONS}", f"DROP TABLE {ITEMS}"),
"missing decisions": (f"DROP TABLE {DECISIONS}",),
"reserved view": (
f"DROP TABLE {DECISIONS}",
f"CREATE VIEW {DECISIONS} AS SELECT 'x' AS id",
),
"extra trigger": (
f"CREATE TRIGGER media_review_trigger AFTER INSERT ON {ITEMS} "
"BEGIN SELECT 1; END",
),
"extra index": (
f"CREATE INDEX idx_media_review_extra ON {ITEMS}(updated_at)",
),
"extra column": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("evidence TEXT", "note TEXT, evidence TEXT"),
),
"wrong primary key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("id TEXT PRIMARY KEY", "id TEXT", 1),
),
"wrong foreign key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(
"generated_images(id) ON DELETE SET NULL",
"generated_images(id) ON DELETE CASCADE",
),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"wrong check": (
f"DROP TABLE {METADATA}",
DDL[METADATA].replace("schema_version=1", "schema_version=2"),
f"INSERT INTO {METADATA} VALUES(1,2,0)",
),
"missing unique": (
"DROP INDEX idx_media_review_decisions_pair",
"DROP INDEX idx_media_review_decisions_image",
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(" UNIQUE(item_id, decision_revision),", ""),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"missing queue index": ("DROP INDEX idx_media_review_items_queue",),
"wrong index columns": (
"DROP INDEX idx_media_review_items_queue",
INDEXES["idx_media_review_items_queue"].replace(", id", ""),
),
"orphan item": ((f"UPDATE {ITEMS} SET byte_id=?", (OTHER_ID,)),),
"orphan decision": ((f"UPDATE {DECISIONS} SET item_id=?", (OTHER_ID,)),),
"dangling pair": (f"UPDATE {DECISIONS} SET pair_id=404",),
"bad item uuid": (
(f"UPDATE {ITEMS} SET id=?", ("x" * 36,)),
(f"UPDATE {DECISIONS} SET item_id=?", ("x" * 36,)),
),
"uppercase decision uuid": (f"UPDATE {DECISIONS} SET id=upper(id)",),
"null decision id": (f"UPDATE {DECISIONS} SET id=NULL",),
"blob ids": (f"UPDATE {DECISIONS} SET id=CAST(id AS BLOB)",),
"no metadata row": (f"DELETE FROM {METADATA}",),
"long created_at": ((f"UPDATE {ITEMS} SET created_at=?", ("x" * 65,)),),
"blob updated_at": (f"UPDATE {ITEMS} SET updated_at=X'00'",),
"text resolver version": (f"UPDATE {ITEMS} SET resolver_version='one'",),
"blob spelled path": (
f"UPDATE {ITEMS} SET spelled_path=CAST(spelled_path AS BLOB)",
),
"evidence hash mismatch": (
(f"UPDATE {ITEMS} SET evidence_hash=?", (CANDIDATE,)),
),
}
for label, statements in cases.items():
with self.subTest(label):
self.assert_rejected_unrepaired(self.tampered(label, statements))
unchecked = {
"two metadata rows": (f"INSERT INTO {METADATA} VALUES(2,1,0)",),
"wrong schema version": (f"UPDATE {METADATA} SET schema_version=2",),
"negative index revision": (
f"UPDATE {METADATA} SET review_index_revision=-1",
),
"uppercase evidence hash": (
f"UPDATE {ITEMS} SET evidence_hash=upper(evidence_hash)",
),
"bad state": (f"UPDATE {ITEMS} SET state='done'",),
"limited two": (f"UPDATE {ITEMS} SET limited=2",),
"empty spelled path": (f"UPDATE {ITEMS} SET spelled_path=''",),
"negative revision": (f"UPDATE {ITEMS} SET revision=-1",),
"negative candidates": (f"UPDATE {ITEMS} SET candidate_count=-1",),
"bad outcome": (f"UPDATE {DECISIONS} SET outcome='maybe'",),
"short candidate hash": (
f"UPDATE {DECISIONS} SET candidate_hash=substr(candidate_hash,1,63)",
),
"negative decision revision": (
f"UPDATE {DECISIONS} SET decision_revision=-1",
),
}
for label, statements in unchecked.items():
with self.subTest(label):
path = self.tampered(label, statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_selected_decision_requires_candidate_hash(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
insert_item(conn)
for outcome, candidate in (("selected", None), ("unresolved", CANDIDATE)):
with self.subTest(outcome):
with self.assertRaises(sqlite3.IntegrityError):
insert_decision(conn, outcome=outcome, candidate_hash=candidate)
insert_decision(conn, outcome="unresolved", candidate_hash=None)
insert_decision(conn, decision_id=OTHER_ID, decision_revision=2)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(
rows(conn, f"SELECT outcome FROM {DECISIONS} ORDER BY 1"),
[("selected",), ("unresolved",)],
)
with self.assertRaises(sqlite3.IntegrityError): # one per revision
insert_decision(conn, decision_id=ITEM_ID, decision_revision=2)
model.close_all()
path = self.tampered(
"selected without hash",
(f"UPDATE {DECISIONS} SET candidate_hash=NULL",),
checks_off=True,
)
self.assert_rejected_unrepaired(path)
def test_oversize_or_noncanonical_evidence_rejected(self):
self.require_module()
self.assertEqual(
canonical_evidence({"b": [1, "é"], "a": None}),
'{"a":null,"b":[1,"\\u00e9"]}',
)
self.assertEqual(
evidence_digest(EVIDENCE), hashlib.sha256(EVIDENCE.encode()).hexdigest()
)
# The store's non-finite policy (utils.json_strict): written as null.
self.assertEqual(
canonical_evidence({"x": float("nan"), "y": [float("-inf")]}),
'{"x":null,"y":[null]}',
)
for bad in (
[],
"text",
{"x": object()},
{"x": object(), "y": float("nan")},
{1: 1, "a": 2},
):
with self.subTest(value=repr(bad)), self.assertRaises(ValueError):
canonical_evidence(bad)
with self.assertRaises(ValueError):
canonical_evidence({"pad": "x" * MAX_EVIDENCE_BYTES})
oversize = '{"pad":"' + "x" * (MAX_EVIDENCE_BYTES - 9) + '"}'
self.assertEqual(len(oversize), MAX_EVIDENCE_BYTES + 1)
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, evidence=oversize)
model.close_all()
noncanonical = {
"spaces": '{"candidates": [], "sha256": "x"}',
"unsorted": '{"sha256":"x","candidates":[]}',
"raw unicode": '{"text":"é"}',
"duplicate key": '{"a":1,"a":2}',
"not an object": "[]",
"not json": "{",
"nan": '{"a":NaN}',
"empty": "",
}
for label, evidence in noncanonical.items():
for table in (ITEMS, DECISIONS):
with self.subTest(label, table=table):
digest = hashlib.sha256(evidence.encode("utf-8")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (evidence,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
if table == DECISIONS:
statements.pop()
self.assert_rejected_unrepaired(
self.tampered(f"{label} {table}", statements)
)
for table in (ITEMS, DECISIONS):
with self.subTest("oversize", table=table):
digest = hashlib.sha256(oversize.encode("ascii")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (oversize,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
path = self.tampered(f"oversize {table}", statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_seventeen_candidates_rejected(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, candidates=17)
insert_item(conn, candidates=16)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
model.close_all()
path = self.tampered(
"seventeen", (f"UPDATE {ITEMS} SET candidate_count=17",), checks_off=True
)
self.assert_rejected_unrepaired(path)
def test_prompt_or_image_deletion_nulls_navigation_and_keeps_decision(self):
self.require_module()
db = PromptDatabase(self.path)
self.models.append(db.model)
with db.model.get_connection() as conn:
populate_legacy(conn)
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(8,9,NULL)"
)
insert_byte(conn)
insert_item(conn)
insert_decision(conn, pair_id=8, image_id=4)
conn.commit()
self.assertTrue(db.delete_prompt_pair(8))
with db.model.get_connection() as conn:
self.assertEqual(
rows(conn, f"SELECT pair_id, image_id FROM {DECISIONS}"), [(None, 4)]
)
with db.model.get_connection() as conn:
conn.execute("DELETE FROM prompt_pairs WHERE id=6")
conn.commit()
self.assertTrue(db.delete_prompt(7))
with db.model.get_connection() as conn:
self.assertEqual(
rows(
conn,
f"SELECT id, item_id, outcome, candidate_hash, pair_id, image_id "
f"FROM {DECISIONS}",
),
[(DECISION_ID, ITEM_ID, "selected", CANDIDATE, None, None)],
)
self.assertEqual(rows(conn, f"SELECT id FROM {ITEMS}"), [(ITEM_ID,)])
self.assertIsNone(review_schema_error(conn, allow_missing=False))
# The SET NULL actions look decisions up by index, not by a scan.
for column in ("pair_id", "image_id"):
plan = " ".join(
str(row[3])
for row in conn.execute(
f"EXPLAIN QUERY PLAN UPDATE {DECISIONS} " # nosec B608
f"SET {column}=NULL WHERE {column}=?",
(1,),
)
)
self.assertIn(f"idx_media_review_decisions_{column[:-3]}", plan)
def test_full_capacity_unicode_control_pair_round_trips_within_8_mib(self):
self.require_module()
value = full_capacity_evidence()
evidence = canonical_evidence(value)
self.assertEqual(len(evidence.encode("utf-8")), MAX_EVIDENCE_BYTES)
self.assertTrue(evidence.isascii())
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_byte(conn)
insert_item(conn, evidence=evidence, candidates=16)
insert_decision(conn, evidence=evidence)
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
for restore_from in (source_path, backup_file):
with self.subTest(Path(restore_from).name):
live.restore_from_file(restore_from)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
stored = conn.execute(
f"SELECT i.evidence, d.evidence FROM {ITEMS} i "
f"JOIN {DECISIONS} d ON d.item_id = i.id"
).fetchone()
self.assertEqual(json.loads(stored[0]), value)
self.assertEqual(json.loads(stored[1]), value)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_install_failure_rolls_back_and_retries(self):
self.require_module()
with patch(
"database.review_schema.review_schema_error",
side_effect=[None, "injected final validation failure"],
):
with self.assertRaisesRegex(sqlite3.DatabaseError, "injected"):
self.model()
with closing(sqlite3.connect(self.path)) as raw:
self.assertEqual(catalog(raw), set())
self.assertIsNotNone(
raw.execute(
"SELECT 1 FROM sqlite_master WHERE name='media_annotation_bytes'"
).fetchone()
)
PromptModel.reset_schema_cache()
model = self.model()
self.assertEqual(
PromptModel.verify_database_file(model.db_path, require_library_state=True),
(True, "ok"),
)
with model.get_connection() as conn:
self.assertEqual(metadata(conn), [(1, 0)])
def test_wal_backup_restore_round_trips_group(self):
self.require_module()
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_group_rows(conn)
conn.execute(f"UPDATE {METADATA} SET review_index_revision=3")
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
source_before = Path(source_path).read_bytes()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
with closing(sqlite3.connect(backup_file)) as raw:
self.assertEqual(table_rows(raw, TABLES), source_rows)
self.assertIsNone(review_schema_error(raw, allow_missing=False))
safety = live.restore_from_file(source_path)
self.assertEqual(Path(source_path).read_bytes(), source_before)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertEqual(metadata(conn), [(1, 3)])
with closing(sqlite3.connect(safety)) as raw:
self.assertEqual(rows(raw, f"SELECT id FROM {ITEMS}"), [(OTHER_ID,)])
PromptModel.reset_schema_cache()
live.restore_from_file(backup_file)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_staged_failure_preserves_source_live_safety_and_handles(self):
self.require_module()
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
live_rows = table_rows(conn, TABLES)
bad = self.tampered(
"restore bad", (f"UPDATE {ITEMS} SET state='done'",), checks_off=True
)
old = str(Path(self.temp.name) / "old.db")
self.model(old).close_all()
self.drop_tables(old)
for source, patched in ((bad, False), (old, True)):
with self.subTest(source=Path(source).name):
before = Path(source).read_bytes()
safety = sorted(Path(self.temp.name).glob("live.db.backup_*"))
if patched:
context = patch(
"database.models.migrate_review_state",
side_effect=sqlite3.OperationalError("staging failed"),
)
expected = sqlite3.OperationalError
else:
context = patch.object(PromptModel, "backup_database")
expected = ValueError
with context as hook, self.assertRaises(expected):
live.restore_from_file(source)
if not patched:
hook.assert_not_called()
self.assertEqual(Path(source).read_bytes(), before)
self.assertEqual(
sorted(Path(self.temp.name).glob("live.db.backup_*")), safety
)
assert_closed(self, source)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), live_rows)
self.assertFalse(
PromptModel.verify_database_file(old, require_library_state=True)[0]
)
self.assertTrue(PromptModel.verify_database_file(old)[0])
def test_verify_requires_group_only_with_require_library_state(self):
self.require_module()
self.model().close_all()
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
self.drop_tables(self.path)
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
if __name__ == "__main__":
unittest.main()
+237 -5
View File
@@ -38,6 +38,11 @@ def _graph(text):
}
def _scope_part(scope):
"""The per-source scope keys this module pins (other features add counters)."""
return {key: scope[key] for key in ("files", "roots", "output_folders")}
class ScanScopeTestCase(AdminAPITestCase):
"""Helpers: link files the way the scan and the monitor do, spy on reads."""
@@ -260,7 +265,7 @@ class TestScanScope(ScanScopeTestCase):
async def test_empty_read_body_asks_for_the_default_scope(self):
"""A chunked POST with no bytes can read a body: still the default."""
from py.api.job_routes import _scan_scope
from py.api.job_routes import _scan_request
class Request:
can_read_body = True
@@ -271,10 +276,21 @@ class TestScanScope(ScanScopeTestCase):
async def text(self):
return self._text
self.assertEqual(await _scan_scope(Request("")), "new")
self.assertEqual(await _scan_scope(Request(" \n")), "new")
self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all")
self.assertIsNone(await _scan_scope(Request("{")))
default = ({"files": "new", "roots": None}, None)
self.assertEqual(await _scan_request(Request("")), default)
self.assertEqual(await _scan_request(Request(" \n")), default)
self.assertEqual(
await _scan_request(
Request(
'{"scope": "all", "roots": [2, 0, 2],'
' "paths": ["output/c", "output/a", "output/c"]}'
)
),
({"files": "all", "roots": {0: "output/a", 2: "output/c"}}, None),
)
request, error = await _scan_request(Request("{"))
self.assertIsNone(request)
self.assertIsInstance(error, str)
async def test_invalid_scope_is_refused(self):
for body in ({"scope": "everything"}, {"scope": 1}, ["new"]):
@@ -292,6 +308,222 @@ class TestScanScope(ScanScopeTestCase):
self.assertIsNone(self.api._scan_job)
class TestPerSourceScope(ScanScopeTestCase):
"""``roots``: scan only the selected effective sources (#249 S4 Task 10)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
_write_png(self.a / "in_a.png", prompt_text="a prompt in a")
_write_png(self.b / "in_b.png", prompt_text="a prompt in b")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
async def _refused(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 400, data)
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_scoped_scan_only_walks_selected_root(self):
walked = []
original = self.api._collect_output_media_sync
def spy(output_dirs):
walked.append([os.path.basename(str(d)) for d in output_dirs])
return original(output_dirs)
self.api._collect_output_media_sync = spy
reads = self._spy_reads()
job = await self._scan({"scope": "all", "roots": [1], "paths": ["output/b"]})
self.assertEqual(walked, [["b"]])
self.assertEqual(reads, ["in_b.png"])
self.assertEqual(
_scope_part(job["scope"]),
{"files": "all", "roots": [1], "output_folders": 1},
)
self.assertEqual(
job["outcomes"]["added"],
[
{
"path": "in_b.png",
"root": "output/b",
"root_index": 1,
"detail": job["outcomes"]["added"][0]["detail"],
}
],
)
self.assertEqual(job["counters"]["added"], 1)
self.assertEqual(self.api._scan_job.last_event["directories"], ["output/b"])
async def test_duplicate_roots_are_one_selection(self):
job = await self._scan(
{"roots": [1, 0, 1], "paths": ["output/b", "output/a", "output/b"]}
)
self.assertEqual(job["scope"]["roots"], [0, 1])
self.assertEqual(job["scope"]["output_folders"], 2)
self.assertEqual(job["counters"]["added"], 2)
async def test_scope_line_counts_selected_folders(self):
job = await self._scan({"roots": [0], "paths": ["output/a"]})
self.assertTrue(
job["log"][0].endswith(
"Scan started · 1 output folder · new and unresolved files"
),
job["log"][0],
)
self.assertEqual(job["scope"]["output_folders"], 1)
unscoped = await self._scan({"scope": "all"})
self.assertEqual(
_scope_part(unscoped["scope"]),
{"files": "all", "roots": None, "output_folders": 2},
)
self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0])
async def _stale(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 409, data)
self.assertEqual(data["code"], "sources-changed")
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_index_past_the_sources_is_409_without_job(self):
await self._stale({"roots": [2], "paths": ["output/c"]})
await self._stale({"roots": [0, 64], "paths": ["output/a", "output/x"]})
async def test_stale_index_never_scans_another_source(self):
"""The list changed after the page read it: source 0 was removed, so
old index 1 (b) is past the end and old index 0 now names b."""
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.b)]
await self._stale({"roots": [1], "paths": ["output/b"]})
await self._stale({"roots": [0], "paths": ["output/a"]})
job = await self._scan({"roots": [0], "paths": ["output/b"]})
self.assertEqual(job["outcomes"]["added"][0]["root"], "output/b")
async def test_roots_need_their_paths(self):
for body in (
{"roots": [0]},
{"roots": [0], "paths": None},
{"roots": [0], "paths": "output/a"},
{"roots": [0], "paths": []},
{"roots": [0, 1], "paths": ["output/a"]},
{"roots": [0], "paths": [0]},
{"roots": [0, 0], "paths": ["output/a", "output/b"]},
):
with self.subTest(body=body):
await self._refused(body)
async def test_bool_and_float_roots_rejected(self):
for roots in (
[True],
[False],
[0.0],
[1.5],
["0"],
[None],
[-1],
[0, None],
[],
None,
0,
"0",
{"0": 0},
list(range(65)),
):
with self.subTest(roots=roots):
paths = ["output/a"] * len(roots) if isinstance(roots, list) else []
await self._refused({"roots": roots, "paths": paths})
async def test_legacy_scan_unscoped_unchanged(self):
from py.api.admin import SCAN_COUNT_KEYS
resp = await self.client.request("POST", "/prompt_manager/scan", data="{}")
payloads = _sse_payloads(await resp.text())
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a", "output/b"])
self.assertEqual(final["added"], 2)
self.assertEqual(
_scope_part(self.api._scan_job.scope),
{"files": "all", "roots": None, "output_folders": 2},
)
async def test_legacy_scan_attaches_to_a_scoped_job_as_is(self):
import threading
from py.api.admin import SCAN_COUNT_KEYS
gate = threading.Event()
original = self.api._collect_output_media_sync
def gated(output_dirs):
gate.wait(5)
return original(output_dirs)
self.api._collect_output_media_sync = gated
# The legacy route has decided to attach once it streams the job.
attached = asyncio.Event()
stream = self.api._stream_scan_job
async def attaching(request, job):
attached.set()
return await stream(request, job)
self.api._stream_scan_job = attaching
resp = await self.client.request(
"POST",
"/prompt_manager/jobs/scan",
json={"roots": [0], "paths": ["output/a"]},
)
self.assertEqual(resp.status, 202)
started = (await resp.json())["job"]
legacy = asyncio.ensure_future(
self.client.request("POST", "/prompt_manager/scan", data="{}")
)
try:
await asyncio.wait_for(attached.wait(), 10)
finally:
gate.set()
payloads = _sse_payloads(await (await legacy).text())
await asyncio.wait_for(self.api._scan_job.task, 10)
self.assertEqual(self.api._scan_job.id, started["id"])
self.assertEqual(len(self.api._jobs.list()), 1)
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a"])
self.assertEqual(final["added"], 1)
async def test_scoped_scan_uses_the_sources_it_was_started_with(self):
"""Sources saved after the start never change what the job walks."""
resolved = []
original = self.api._get_all_output_dirs
def counting():
resolved.append(1)
return original()
self.api._get_all_output_dirs = counting
job = await self._scan({"roots": [1], "paths": ["output/b"]})
self.assertEqual(len(resolved), 1) # the request's own resolution only
self.assertEqual(job["outcomes"]["added"][0]["root_index"], 1)
class TestKnownResolvedPaths(ScanScopeTestCase):
"""``PromptDatabase.get_known_resolved_paths``: one streamed query."""
+959
View File
@@ -0,0 +1,959 @@
"""System › Sources & scans backend (#249 S4 Task 9).
``GET /prompt_manager/settings`` reports the configured, effective and
watched sources and the monitoring setting; ``POST`` saves
``monitoring_enabled`` (a restart applies it) and refuses a source change
while a record-changing job runs; ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing. No response carries an absolute path.
"""
import asyncio
import functools
import json
import ntpath
import os
import sys
import types
import unittest
from contextlib import contextmanager
from unittest.mock import patch
try:
from tests.test_api_admin import AdminAPITestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase
from py import config as config_module # noqa: E402
from py.api import settings_routes, sources # noqa: E402
from py.config import GalleryConfig # noqa: E402
VALIDATE = "/prompt_manager/settings/validate-source"
SETTINGS = "/prompt_manager/settings"
@contextmanager
def no_filesystem_access_for(marker, test):
"""Fail ``test`` if anything resolves, stats or opens a path holding ``marker``.
Trusted directories (ComfyUI's, the anchors) may still be resolved.
"""
real = {
"realpath": os.path.realpath,
"stat": os.stat,
"lstat": os.lstat,
}
def guard(name):
def call(path, *args, **kwargs):
if isinstance(path, (str, bytes)) and marker in os.fsdecode(path):
test.fail(f"{name}({path!r}) touched an outside path")
return real[name](path, *args, **kwargs)
return call
with (
patch("os.path.realpath", guard("realpath")),
patch("os.stat", guard("stat")),
patch("os.lstat", guard("lstat")),
):
yield
class FakeMonitor:
"""The image monitor as get_settings sees it: status and scheduled roots."""
def __init__(self, scheduled, alive=True):
self.monitored_directories = [str(path) for path in scheduled]
self.alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": self.monitored_directories,
"handler_active": True,
"observer_alive": self.alive,
}
class SourcesTestCase(AdminAPITestCase):
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
enabled = GalleryConfig.MONITORING_ENABLED
self.addCleanup(setattr, GalleryConfig, "MONITORING_ENABLED", enabled)
GalleryConfig.MONITORING_ENABLED = True
self._install_monitor(None)
def _install_monitor(self, monitor):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = monitor
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
async def _get(self):
resp = await self.client.request("GET", "/prompt_manager/settings")
body = await resp.text()
self.assertEqual(resp.status, 200, body)
self.assertNotIn(self.tmpdir, body)
return json.loads(body)
async def _post(self, url, payload):
resp = await self.client.request("POST", url, json=payload)
body = await resp.text()
self.assertNotIn(self.tmpdir, body)
return resp.status, json.loads(body)
def _config_bytes(self):
if not os.path.exists(self.config_path):
return None
with open(self.config_path, "rb") as f:
return f.read()
def _states(self, data):
return {row["path"]: row["watch"] for row in data["sources"]["effective"]}
class TestSourceStatus(SourcesTestCase):
async def test_sources_list_configured_effective_and_auto_detected(self):
missing = self.output_dir / "gone"
missing.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.a),
self._root(missing),
]
missing.rmdir() # an unmounted drive: configured, no longer a directory
data = await self._get()
self.assertEqual(
data["sources"]["configured"],
[
{
"label": "a",
"path": "output/a",
"valid": True,
"code": None,
"reason": None,
},
{
"label": "gone",
"path": "output/gone",
"valid": False,
"code": "not-found",
"reason": "Gallery root does not exist",
},
],
)
self.assertEqual(
data["sources"]["effective"],
[{"label": "a", "path": "output/a", "root_index": 0, "watch": "manual"}],
)
self.assertFalse(data["sources"]["auto_detected"])
# The legacy keys are unchanged.
self.assertEqual(
data["settings"]["gallery_root_paths"], ["output/a", "output/gone"]
)
async def test_empty_list_means_auto_detect(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["x"]}}})
# Detection (the upward search for a ComfyUI tree) finds the output.
self.api._find_comfyui_output_dir = lambda: str(self.output_dir)
status, saved = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": []}
)
self.assertEqual(status, 200, saved)
self.assertEqual(
self._read_config()["gallery"]["monitoring"]["directories"], []
)
data = await self._get()
self.assertEqual(data["sources"]["configured"], [])
self.assertTrue(data["sources"]["auto_detected"])
self.assertEqual(
data["sources"]["effective"],
[
{
"label": "ComfyUI output",
"path": "output",
"root_index": 0,
"watch": "manual",
}
],
)
async def test_watching_requires_live_observer_with_root(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
# No monitor in this process: nothing is watched, a restart cannot help.
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": False, "observer_alive": False, "scheduled": []},
)
self.assertEqual(
self._states(data), {"output/a": "manual", "output/b": "manual"}
)
# A dead observer watches nothing, even the roots it scheduled.
self._install_monitor(FakeMonitor([self.a], alive=False))
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": True, "observer_alive": False, "scheduled": ["output/a"]},
)
self.assertEqual(
self._states(data),
{"output/a": "restart_pending", "output/b": "restart_pending"},
)
# A live observer watches its scheduled root; a root configured since
# it started waits for a restart.
self._install_monitor(FakeMonitor([self.a]))
data = await self._get()
self.assertTrue(data["monitor"]["observer_alive"])
self.assertEqual(
self._states(data), {"output/a": "watching", "output/b": "restart_pending"}
)
# Monitoring switched off: the live observer still watches until the
# restart ("stopping"), and nothing else is pending.
GalleryConfig.MONITORING_ENABLED = False
data = await self._get()
self.assertFalse(data["settings"]["monitoring_enabled"])
self.assertEqual(
self._states(data), {"output/a": "stopping", "output/b": "manual"}
)
async def test_scheduled_root_matches_through_its_spelling(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
spelled = os.path.join(str(self.output_dir), "b", "..", "a")
self._install_monitor(FakeMonitor([spelled]))
data = await self._get()
self.assertEqual(self._states(data), {"output/a": "watching"})
self.assertEqual(data["monitor"]["scheduled"], ["output/a"])
async def test_status_failure_is_500_without_paths(self):
def boom(*args, **kwargs):
raise OSError(2, "No such file", f"{self.tmpdir}/secret")
with patch.object(sources, "describe_sources", boom):
resp = await self.client.request("GET", "/prompt_manager/settings")
self.assertEqual(resp.status, 500)
self.assertNotIn(self.tmpdir, await resp.text())
class TestMonitoringSetting(SourcesTestCase):
async def test_monitoring_enabled_strict_bool_and_restart_flag(self):
self._write_config({"custom": {"keep": True}})
before = self._config_bytes()
for value in ("false", 0, 1, None, [], {"on": True}):
with self.subTest(value=value):
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": value}
)
self.assertEqual(status, 400, data)
self.assertFalse(data["success"])
self.assertEqual(self._config_bytes(), before)
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
self._install_monitor(FakeMonitor([self.output_dir]))
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": False}
)
self.assertEqual(status, 200, data)
self.assertTrue(data["persisted"])
self.assertTrue(data["restart_required"])
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
saved = self._read_config()
self.assertIs(saved["gallery"]["monitoring"]["enabled"], False)
self.assertEqual(saved["custom"], {"keep": True})
self.assertFalse(GalleryConfig.MONITORING_ENABLED)
self.assertFalse((await self._get())["settings"]["monitoring_enabled"])
# Back on while the observer still runs: nothing waits for a restart.
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertIs(self._read_config()["gallery"]["monitoring"]["enabled"], True)
async def test_enabling_without_a_live_observer_needs_a_restart(self):
GalleryConfig.MONITORING_ENABLED = False
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
def test_config_file_enabled_must_be_a_bool(self):
GalleryConfig.MONITORING_ENABLED = True
with self.assertLogs("prompt_manager.config", level="WARNING"):
GalleryConfig.update_config({"monitoring": {"enabled": "false"}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, True)
GalleryConfig.update_config({"monitoring": {"enabled": False}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, False)
class TestSourceChangeHold(SourcesTestCase):
async def test_root_save_refused_while_scan_job_runs(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["kept"]}}})
before = self._config_bytes()
gate = asyncio.Event()
async def runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan", "Scan for metadata", record_changing=True, runner=runner
)
try:
status, data = await self._post(
"/prompt_manager/settings",
{"gallery_root_paths": [str(self.b)]},
)
self.assertEqual(status, 409, data)
self.assertEqual(data["code"], "busy")
self.assertEqual(
data["running"], {"id": scan.id, "title": "Scan for metadata"}
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.a)])
# A save without sources, or one resending the current sources
# (the legacy page always does), is not a source change.
status, data = await self._post(
"/prompt_manager/settings", {"result_timeout": 9}
)
self.assertEqual(status, 200, data)
status, data = await self._post(
"/prompt_manager/settings",
{"result_timeout": 8, "gallery_root_paths": [str(self.a)]},
)
self.assertEqual(status, 200, data)
finally:
gate.set()
await asyncio.wait_for(scan.task, 5)
async def test_root_save_runs_as_a_source_change_hold(self):
seen = []
original = settings_routes.get_config_store
def store():
seen.extend(
(job.kind, job.title, job.admission, job.cancellable)
for job in self.api._jobs.list()
if job.running
)
return original()
with patch.object(settings_routes, "get_config_store", store):
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual(status, 200, data)
self.assertIn(("source_change", "Source change", "records", False), seen)
self.assertFalse(any(job.running for job in self.api._jobs.list()))
async def test_failed_root_save_ends_the_hold_failed(self):
os.makedirs(self.config_path) # config.json is a directory: unreadable
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertGreaterEqual(status, 400, data)
held = [j for j in self.api._jobs.list() if j.kind == "source_change"]
self.assertEqual([job.state for job in held], ["failed"])
async def test_mixed_case_windows_paths_dedupe(self):
"""Two spellings of one Windows folder are saved once (normcase)."""
with (
patch.object(sources, "source_problem", return_value=(None, "")),
patch.object(
GalleryConfig, "resolve_gallery_root", side_effect=lambda path: path
),
patch.object(settings_routes, "_source_identity", ntpath.normcase),
patch.object(self.api, "_apply_gallery_roots", return_value=True),
):
roots, error = self.api._parse_gallery_roots(
{
"gallery_root_paths": [
"C:\\ComfyUI\\Output\\Renders",
"c:/comfyui/output/renders",
"C:\\ComfyUI\\output\\other",
]
}
)
self.assertIsNone(error)
self.assertEqual(
roots, ["C:\\ComfyUI\\Output\\Renders", "C:\\ComfyUI\\output\\other"]
)
async def test_case_variant_of_one_folder_is_saved_once_natively(self):
variant = str(self.a)[:-1] + "A"
if not os.path.isdir(variant):
self.skipTest("case-sensitive filesystem")
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.a), variant]}
)
self.assertEqual(status, 200, data)
self.assertEqual(len(GalleryConfig.MONITORING_DIRECTORIES), 1)
class TestValidateSource(SourcesTestCase):
async def test_validate_source_writes_nothing(self):
self._write_config({"gallery": {"monitoring": {"directories": []}}})
before = self._config_bytes()
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 200, data)
self.assertEqual(
data,
{
"success": True,
"valid": True,
"label": "a",
"path": "output/a",
"code": None,
"reason": None,
},
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, roots)
self.assertEqual(self.api._jobs.list(), [])
async def test_relative_path_resolves_against_comfyui(self):
status, data = await self._post(VALIDATE, {"path": "output/b"})
self.assertEqual((status, data["valid"], data["path"]), (200, True, "output/b"))
status, data = await self._post(VALIDATE, {"path": "output"})
self.assertEqual(data["label"], "ComfyUI output")
async def test_invalid_sources_answer_a_code(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
cases = {
"": "empty",
" ": "empty",
str(self.output_dir / "missing"): "not-found",
str(a_file): "not-directory",
str(self.outside_dir): "outside-allowed",
os.path.abspath(os.sep): "outside-allowed",
os.path.expanduser("~"): "outside-allowed",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
"output/../output/a": "traversal",
str(self.a) + "\x00": "invalid-form",
"\\\\?\\C:\\ComfyUI\\output": "invalid-form",
"x" * 5000: "invalid-form",
}
if os.name != "nt":
cases["C:\\ComfyUI\\output"] = "invalid-form"
for path, code in cases.items():
with self.subTest(code=code, path=path[:40]):
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(status, 200, data)
self.assertFalse(data["valid"])
self.assertEqual(data["code"], code)
self.assertIsInstance(data["reason"], str)
self.assertIsNone(data["label"])
self.assertIsNone(data["path"])
self.assertNotIn(os.path.expanduser("~"), json.dumps(data))
async def test_outside_paths_answer_alike_whether_they_exist(self):
"""No existence oracle: outside the allowed folders every path looks
the same, existing or not, file or folder."""
a_file = self.outside_dir / "file.txt"
a_file.write_bytes(b"x")
answers = []
for path in (
self.outside_dir,
self.outside_dir / "missing",
a_file,
os.path.join(self.tmpdir, "no", "such", "tree"),
os.path.abspath(os.sep),
os.path.expanduser("~"),
):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append((status, data))
self.assertEqual(answers[0][1]["code"], "outside-allowed")
self.assertEqual([answer for answer in answers], [answers[0]] * len(answers))
async def test_settings_save_refuses_outside_paths_alike(self):
"""The legacy save answers every outside path alike too: the
filesystem root, the home directory and a link to it included."""
(self.outside_dir / "x").mkdir()
link = os.path.join(self.tmpdir, "link-home")
try:
os.symlink(os.path.expanduser("~"), link, target_is_directory=True)
except (OSError, NotImplementedError):
link = None
paths = [
str(self.outside_dir / "x"),
self.tmpdir + "/nothing/x",
os.path.abspath(os.sep),
os.path.expanduser("~"),
]
if link:
paths.append(link)
answers = []
for path in paths:
resp = await self.client.request(
"POST", SETTINGS, json={"gallery_root_paths": [path]}
)
body = await resp.text()
self.assertNotIn(os.path.expanduser("~"), body)
data = json.loads(body)
# Only the name the caller sent may differ.
answers.append(
(
resp.status,
{k: v for k, v in data.items() if k not in ("source", "error")},
)
)
self.assertEqual(
data["error"],
f"{data['source']} can't be used: "
f"{config_module.OUTSIDE_ALLOWED_REASON}. Remove it, then save.",
)
self.assertEqual(
answers[0],
(
400,
{
"success": False,
"code": "invalid-source",
"index": 0,
"reason": config_module.OUTSIDE_ALLOWED_REASON,
},
),
)
self.assertEqual(answers, [answers[0]] * len(answers))
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
def test_every_caller_folds_root_and_home_alike(self):
for path in (os.path.abspath(os.sep), os.path.expanduser("~")):
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
GalleryConfig.validate_gallery_root(path),
(False, config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
async def test_outside_path_is_never_resolved_or_stat(self):
"""Containment is decided as text first: no realpath, stat or lstat of
an outside path, existing or not, by any caller (I2)."""
probe = self.outside_dir / "probe-x"
probe.mkdir()
(probe / "deep").mkdir()
cases = [str(probe), str(probe / "deep"), str(probe / "missing")]
with no_filesystem_access_for("probe-x", self):
for path in cases:
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path)[0], "outside-allowed"
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": [path]}
)
self.assertEqual(error["code"], "invalid-source")
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(data["code"], "outside-allowed")
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
# A traversal that normalises outside is refused as text too.
sneaky = os.path.join(
str(self.output_dir), "..", "..", "outside", "probe-x"
)
self.assertEqual(
GalleryConfig.gallery_root_problem(sneaky)[0], "outside-allowed"
)
async def test_settings_save_applies_the_form_checks(self):
"""POST /settings refuses what validate-source refuses, by its form."""
for path, code in {
str(self.a) + "\x00": "invalid-form",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
}.items():
with self.subTest(code=code):
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
self.assertEqual(data["reason"], sources.FORM_REASONS[code])
windows = functools.partial(sources.source_form_error, windows=True)
with (
patch.object(sources, "source_form_error", windows),
# The probes hold a token no trusted directory does (on Windows the
# temporary ComfyUI tree itself sits on C:).
no_filesystem_access_for("PMProbe", self),
):
for path in (
"\\\\?\\C:\\PMProbe\\output",
"\\\\.\\pipe\\PMProbe",
"C:\\PMProbe\\output\\CON",
"C:\\PMProbe\\output\\a:stream",
"C:PMProbe",
):
with self.subTest(path=path):
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": ["output/a", path]}
)
self.assertEqual(error["code"], "invalid-source")
self.assertEqual(error["index"], 1)
self.assertEqual(
error["reason"], sources.FORM_REASONS["invalid-form"]
)
async def test_inside_paths_say_why_they_are_not_usable(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
status, missing = await self._post(
VALIDATE, {"path": str(self.output_dir / "missing")}
)
status, not_dir = await self._post(VALIDATE, {"path": str(a_file)})
self.assertEqual(
(missing["code"], not_dir["code"]), ("not-found", "not-directory")
)
async def test_malformed_request_is_400(self):
for body in ({}, {"path": 42}, {"path": None}, ["x"]):
with self.subTest(body=body):
status, data = await self._post(VALIDATE, body)
self.assertEqual(status, 400, data)
self.assertEqual(data["code"], "invalid-request")
self.assertFalse(data["success"])
class TestSourceEdges(SourcesTestCase):
async def test_validation_failure_is_500_without_paths(self):
def boom(path):
raise OSError(13, "Permission denied", f"{self.tmpdir}/secret")
with patch.object(sources, "validate_source", boom):
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 500)
self.assertEqual(data["error"], "Failed to check the source")
async def test_oversized_body_is_413_too_large(self):
resp = await self.client.request(
"POST", VALIDATE, data=json.dumps({"path": "x" * (2 << 20)})
)
self.assertEqual(resp.status, 413)
self.assertEqual((await resp.json())["code"], "too-large")
async def test_unreadable_monitor_counts_as_no_live_observer(self):
class Broken:
def get_status(self):
raise RuntimeError("observer gone")
self._install_monitor(Broken())
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
def test_labels_skip_getters_that_fail_or_are_missing(self):
import sys
import types
def fails():
raise RuntimeError("no input dir")
fake = types.SimpleNamespace(
get_output_directory=lambda: str(self.output_dir),
get_input_directory=fails,
get_temp_directory=lambda: None,
)
with patch.dict(sys.modules, {"folder_paths": fake}):
self.assertEqual(sources.source_label(self.output_dir), "ComfyUI output")
self.assertEqual(sources.source_label(self.a), "a")
with patch.dict(sys.modules, {"folder_paths": None}):
self.assertEqual(sources.source_label(self.output_dir), "output")
class TestOutsideBaseDirectories(SourcesTestCase):
"""ComfyUI folders outside the base directory (``--output-directory``):
their sources round-trip and two folders of one name stay apart (I3)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.renders = self.outside_dir / "renders"
self.input_dir = self.comfy_dir / "input"
(self.renders / "portraits").mkdir(parents=True)
(self.input_dir / "portraits").mkdir(parents=True)
sys.modules["folder_paths"] = types.SimpleNamespace(
base_path=str(self.comfy_dir),
get_output_directory=lambda: str(self.renders),
get_input_directory=lambda: str(self.input_dir),
)
async def test_source_outside_base_round_trips(self):
status, rendered = await self._post(
VALIDATE, {"path": str(self.renders / "portraits")}
)
self.assertEqual(
(rendered["valid"], rendered["path"]), (True, "renders/portraits")
)
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [rendered["path"]]}
)
self.assertEqual(status, 200, data)
self.assertEqual(
GalleryConfig.MONITORING_DIRECTORIES,
[self._root(self.renders / "portraits")],
)
configured = (await self._get())["sources"]["configured"]
self.assertEqual(
[(row["path"], row["valid"]) for row in configured],
[("renders/portraits", True)],
)
# The page resends what it read: still the same folder, no change.
status, data = await self._post(
SETTINGS, {"gallery_root_paths": ["renders/portraits"]}
)
self.assertEqual((status, data["restart_required_fields"]), (200, []))
async def test_same_name_folders_stay_apart(self):
answers = []
for path in (self.renders / "portraits", self.input_dir / "portraits"):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append(data["path"])
self.assertEqual(answers, ["renders/portraits", "input/portraits"])
status, data = await self._post(SETTINGS, {"gallery_root_paths": answers})
self.assertEqual(status, 200, data)
sources_ = (await self._get())["sources"]
self.assertEqual([row["path"] for row in sources_["configured"]], answers)
self.assertEqual([row["path"] for row in sources_["effective"]], answers)
class TestUnusableSourceSave(SourcesTestCase):
async def test_unusable_source_is_named_and_saves_nothing(self):
"""A configured source that went missing blocks the save and is named."""
gone = self.output_dir / "gone"
gone.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(gone)]
self._write_config({"custom": 1})
before = self._config_bytes()
gone.rmdir() # an unplugged drive
status, data = await self._post(
SETTINGS,
{
"gallery_root_paths": ["output/a", "output/gone", "output/b"],
"worker_threads": 2,
},
)
self.assertEqual(status, 400, data)
self.assertEqual(
data,
{
"success": False,
"code": "invalid-source",
"source": "output/gone",
"index": 1,
"reason": "Gallery root does not exist",
"error": "output/gone can't be used: Gallery root does not exist. "
"Remove it or reconnect it, then save.",
},
)
self.assertEqual(self._config_bytes(), before)
async def test_malformed_roots_are_coded_invalid_request(self):
for roots in ("output/a", [1]):
with self.subTest(roots=roots):
status, data = await self._post(SETTINGS, {"gallery_root_paths": roots})
self.assertEqual((status, data["code"]), (400, "invalid-request"))
class TestUnchangedRootsTakeNoHold(SourcesTestCase):
async def test_resent_spelling_of_configured_roots_is_no_change(self):
"""config.json holds a hand-written spelling; the legacy page resends
it resolved: no source change, so no hold and no 409 during a scan (M1)."""
GalleryConfig.MONITORING_DIRECTORIES = ["output/a"]
gate = asyncio.Event()
async def runner(job):
await gate.wait()
job.finish("finished")
self.api._jobs.start(
"scan", "Metadata scan", record_changing=True, runner=runner
)
try:
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.a)], "result_timeout": 30}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertEqual([j.kind for j in self.api._jobs.list()], ["scan"])
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual((status, data["code"]), (409, "busy"))
finally:
gate.set()
class TestWindowsContainment(unittest.TestCase):
"""Windows path forms, with ntpath fakes: anything outside the allowed
folders (a UNC share, the verbatim and device namespaces, another drive)
is refused as text, before any path is resolved or opened (I2)."""
ALLOWED = "c:\\comfyui\\output"
NAS = "\\\\nas\\share\\renders"
def setUp(self):
self.touched = []
def realpath(path, *args, **kwargs):
self.touched.append(path)
return ntpath.normcase(ntpath.normpath(path))
def probe(path):
self.touched.append(path)
return True
names = ("normcase", "normpath", "isabs", "join", "dirname", "basename", "sep")
fake_path = types.SimpleNamespace(
**{name: getattr(ntpath, name) for name in names},
realpath=realpath,
exists=probe,
isdir=probe,
expanduser=lambda path: "c:\\users\\someone",
)
fake_os = types.SimpleNamespace(
path=fake_path, environ={}, pathsep=";", name="nt"
)
for target, value in (
(patch.object(config_module, "os", fake_os), None),
(
patch.object(
GalleryConfig,
"_allowed_parent_pairs",
classmethod(
lambda cls: [(self.ALLOWED, self.ALLOWED), (self.NAS, self.NAS)]
),
),
None,
),
(
patch.object(
GalleryConfig,
"path_anchors",
classmethod(lambda cls: ["c:\\comfyui", "\\\\nas\\share"]),
),
None,
),
):
target.start()
self.addCleanup(target.stop)
def test_outside_forms_are_refused_untouched(self):
for path in (
"\\\\attacker\\share\\x",
"//attacker/share/x",
"\\\\?\\C:\\ComfyUI\\output\\a",
"\\\\?\\UNC\\attacker\\share",
"\\\\.\\pipe\\x",
"\\\\.\\C:\\ComfyUI\\output",
"D:\\Renders",
"C:\\Windows",
"C:\\ComfyUI\\output\\..\\..\\Windows",
"C:\\ComfyUI\\output2",
"\\\\nas\\share\\other",
"..\\..\\Windows",
):
with self.subTest(path=path):
self.touched.clear()
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(self.touched, [])
def test_allowed_spellings_are_accepted(self):
for path in (
"C:\\ComfyUI\\output\\a",
"c:/comfyui/OUTPUT/a",
"output\\a",
"\\\\NAS\\share\\renders\\a", # a UNC spelling of an allowed extra root
):
with self.subTest(path=path):
self.assertEqual(GalleryConfig.gallery_root_problem(path), (None, ""))
class TestSourceFormError(unittest.TestCase):
"""The lexical check that runs before any filesystem access."""
def test_windows_forms(self):
err = sources.source_form_error
for path in (
"C:\\ComfyUI\\output",
"C:/ComfyUI/output",
"\\\\server\\share\\output",
"D:\\images",
):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=True))
for path, code in {
"C:output": "invalid-form", # drive-relative
"C:\\out\\a:stream": "invalid-form", # alternate data stream
"C:\\out\\CON": "invalid-form",
"C:\\out\\nul.txt": "invalid-form",
"C:\\out\\folder.": "invalid-form",
"C:\\out\\folder \\sub": "invalid-form",
"\\\\?\\C:\\out": "invalid-form",
"//?/C:/out": "invalid-form",
"\\\\.\\pipe\\x": "invalid-form",
"\\\\server": "invalid-form",
"C:\\out\\..\\in": "traversal",
"output\\..\\input": "traversal",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=True), code)
def test_posix_forms(self):
err = sources.source_form_error
for path in ("/srv/comfy/output", "output/renders", "out/CON", "a b/c."):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=False))
for path, code in {
"C:\\ComfyUI\\output": "invalid-form",
"C:/ComfyUI/output": "invalid-form",
"\\\\server\\share": "invalid-form",
"out\\renders": "invalid-form",
"/srv/../etc": "traversal",
"../output": "traversal",
"/srv/out\x00": "invalid-form",
"/srv/out\n": "invalid-form",
"": "empty",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=False), code)
if __name__ == "__main__":
unittest.main()
+479
View File
@@ -0,0 +1,479 @@
"""System routes (py/api/system.py): health check job, Overview and statistics.
The health check runs off-loop as a ``read`` registry job of kind ``health``
(one at a time, beside a scan), and the API keeps its last report. The
Overview reads only real values: Library-eligible counts, indexed images, the
last health result or null, no backup until one exists, and the effective
sources and whether they were auto-detected. The missing-thumbnail count is not
part of it: the Overview reads ``/images/thumbnails/status`` on its own (its
walk of every source must not hold the other values back).
"""
import asyncio
import os
import sqlite3
import sys
import threading
from pathlib import Path
from unittest.mock import MagicMock, patch
_mock_server = MagicMock()
_mock_server.PromptServer.instance.routes = MagicMock()
sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.api.jobs import HEALTH_JOB_KIND # noqa: E402
from py.config import GalleryConfig # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
try:
from tests.test_api_admin import AdminAPITestCase # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase # noqa: E402
WAIT = 5
class SystemAPITestCase(AdminAPITestCase):
async def get_application(self):
app = await super().get_application()
# Auto-detection searches the disk; here it finds the fake ComfyUI only.
self.api._find_comfyui_output_dir = lambda: (
str(self.output_dir) if self.output_dir.is_dir() else None
)
return app
def save(self, text, **metadata):
return self.api.db.save_prompt(
text, prompt_hash=generate_prompt_hash(text), **metadata
)
def image(self, prompt_id, name, *, exists=True, folder=None):
path = Path(folder or self.output_dir) / name
if exists:
path.write_bytes(b"png")
return self.api.db.link_image_to_prompt(prompt_id, str(path))
async def json(self, method, url, status=200):
response = await self.client.request(method, url)
self.assertEqual(response.status, status, await response.text())
return await response.json()
async def restore_snapshot(self):
"""Restore the live database from a copy of itself through the legacy route."""
from contextlib import closing
from aiohttp import FormData
snapshot = os.path.join(self.tmpdir, "snapshot.db")
with closing(sqlite3.connect(self.api.db.model.db_path)) as src:
with closing(sqlite3.connect(snapshot)) as dst:
src.backup(dst)
with open(snapshot, "rb") as handle:
upload = handle.read()
form = FormData()
form.add_field("database_file", upload, filename="upload.db")
response = await self.client.request(
"POST", "/prompt_manager/restore", data=form
)
self.assertEqual(response.status, 200, await response.text())
async def run_health(self):
body = await self.json("POST", "/prompt_manager/system/health", 202)
job = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
return body["job"], job
class TestHealthRoute(SystemAPITestCase):
async def test_no_report_before_a_check(self):
body = await self.json("GET", "/prompt_manager/system/health")
self.assertEqual(body, {"success": True, "report": None})
async def test_health_check_runs_as_a_read_job_and_keeps_its_report(self):
prompt = self.save("Images")
self.image(prompt, "kept.png")
self.image(prompt, "gone.png", exists=False)
summary, job = await self.run_health()
self.assertEqual(summary["kind"], HEALTH_JOB_KIND)
self.assertEqual(summary["title"], "Health check")
self.assertEqual(summary["admission"], "read")
self.assertFalse(summary["cancellable"])
self.assertEqual(summary["counters"], {})
self.assertEqual(job.state, "finished")
self.assertEqual(job.progress, 100)
self.assertEqual(job.status_line, "Health check incomplete")
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
self.assertEqual(report["status"], "incomplete")
missing = next(c for c in report["checks"] if c["id"] == "missing_files")
self.assertEqual((missing["status"], missing["count"]), ("issues", 1))
ambiguous = next(c for c in report["checks"] if c["id"] == "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "unavailable")
self.assertNotIn(self.tmpdir, str(report))
async def test_health_check_is_single_flight_and_runs_beside_a_scan(self):
release = threading.Event()
entered = threading.Event()
real = self.api.db.run_health_checks
def gated(*args, **kwargs):
entered.set()
release.wait(WAIT)
return real(*args, **kwargs)
scan_gate = asyncio.Event()
async def scanning(job):
await scan_gate.wait()
scan = self.api._jobs.start(
"scan", "Metadata scan", admission="records", runner=scanning
)
with patch.object(self.api.db, "run_health_checks", side_effect=gated):
first = await self.json("POST", "/prompt_manager/system/health", 202)
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, entered.wait, WAIT)
busy = await self.json("POST", "/prompt_manager/system/health", 409)
self.assertEqual(busy["code"], "busy")
self.assertEqual(
busy["running"], {"id": first["job"]["id"], "title": "Health check"}
)
release.set()
await asyncio.wait_for(self.api._jobs.get(first["job"]["id"]).task, WAIT)
scan_gate.set()
await asyncio.wait_for(scan.task, WAIT)
async def test_failed_check_keeps_the_previous_report(self):
await self.run_health()
before = (await self.json("GET", "/prompt_manager/system/health"))["report"]
with (
patch.object(
self.api.db, "run_health_checks", side_effect=RuntimeError("boom")
),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("POST", "/prompt_manager/system/health", 202)
job = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.state, "failed")
after = (await self.json("GET", "/prompt_manager/system/health"))["report"]
self.assertEqual(after, before)
async def test_absent_configured_source_is_reported_per_source(self):
prompt = self.save("Images")
self.image(prompt, "gone.png", exists=False)
absent = self.comfy_dir / "output" / "unmounted"
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.output_dir),
str(absent),
]
await self.run_health()
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
missing = next(c for c in report["checks"] if c["id"] == "missing_files")
# Per source: the readable output is counted, the absent one is not.
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 1)
self.assertTrue(missing["partial"])
self.assertEqual(
[(s["status"], s["count"]) for s in missing["sources"]],
[("checked", 1), ("unavailable", None)],
)
self.assertEqual(report["status"], "incomplete")
async def test_restore_clears_health_report(self):
self.save("Any")
await self.run_health()
self.assertIsNotNone(
(await self.json("GET", "/prompt_manager/system/health"))["report"]
)
await self.restore_snapshot()
body = await self.json("GET", "/prompt_manager/system/health")
self.assertEqual(body, {"success": True, "report": None})
overview = await self.json("GET", "/prompt_manager/system/overview")
self.assertIsNone(overview["health"])
async def test_check_finishing_after_a_restore_is_not_kept(self):
# A check that started on the old database describes it, not the new one.
self.save("Any")
release = threading.Event()
entered = threading.Event()
real = self.api.db.run_health_checks
def gated(*args, **kwargs):
entered.set()
release.wait(WAIT)
return real(*args, **kwargs)
with patch.object(self.api.db, "run_health_checks", side_effect=gated):
body = await self.json("POST", "/prompt_manager/system/health", 202)
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, entered.wait, WAIT)
await self.restore_snapshot()
release.set()
job = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.state, "finished")
body = await self.json("GET", "/prompt_manager/system/health")
self.assertEqual(body, {"success": True, "report": None})
# A check started on the restored database is kept again.
await self.run_health()
self.assertIsNotNone(
(await self.json("GET", "/prompt_manager/system/health"))["report"]
)
async def test_health_sources_resolve_off_the_event_loop(self):
loop_thread = threading.get_ident()
threads = []
real = self.api._health_roots
def recording():
threads.append(threading.get_ident())
return real()
with patch.object(self.api, "_health_roots", side_effect=recording):
await self.run_health()
self.assertEqual(len(threads), 1)
self.assertNotEqual(threads[0], loop_thread)
async def test_health_start_failure_is_an_error_envelope(self):
with (
patch.object(self.api, "_health_roots", side_effect=RuntimeError("secret")),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("POST", "/prompt_manager/system/health", 500)
self.assertEqual(
body,
{
"success": False,
"code": None,
"error": "Failed to start the health check",
},
)
self.assertEqual(self.api._jobs.list(), [])
class TestOverviewRoute(SystemAPITestCase):
async def test_overview_reports_real_counts(self):
self.api.db.save_prompt_pair("Cat", "Blur")
archived = self.api.db.save_prompt_pair("Cat", "Noise")
self.api.db.archive_prompt_pair(archived)
legacy = self.save("Legacy")
self.image(legacy, "a.png")
self.image(legacy, "b.png")
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertTrue(body["success"])
self.assertEqual(
body["library"], {"positive_prompts": 2, "negative_variants": 1}
)
self.assertEqual(body["images"], {"indexed": 2, "rated": 0})
self.assertEqual(
set(body),
{
"success",
"library",
"images",
"health",
"backup",
"needs_review",
"sources",
},
)
async def test_overview_rated_counts_images_of_rated_prompts(self):
rated = self.save("Rated", rating=5)
self.image(rated, "a.png")
self.image(rated, "b.png")
self.image(self.save("Unrated"), "c.png")
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(body["images"], {"indexed": 3, "rated": 2})
async def test_overview_never_fabricates_backup_or_health(self):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertIsNone(body["health"])
self.assertIsNone(body["backup"])
self.assertEqual(body["needs_review"], {"pending": 0})
self.save("Any")
await self.run_health()
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["health"],
{"checked_at": report["checked_at"], "status": "incomplete", "issues": 0},
)
self.assertIsNone(body["backup"])
async def test_overview_health_counts_checks_with_issues(self):
prompt = self.save("Images")
self.image(prompt, "gone.png", exists=False)
self.image(prompt, "gone-too.png", exists=False)
await self.run_health()
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(body["health"]["issues"], 1)
async def test_overview_sources_distinguish_auto_detect(self):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["sources"], {"configured": 0, "effective": 1, "auto_detected": True}
)
second = self.output_dir / "archive"
second.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.output_dir),
self._root(second),
]
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["sources"], {"configured": 2, "effective": 2, "auto_detected": False}
)
# Every configured source unavailable: ComfyUI's output is used instead.
GalleryConfig.MONITORING_DIRECTORIES = [str(self.comfy_dir / "absent")]
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["sources"], {"configured": 1, "effective": 1, "auto_detected": True}
)
async def test_overview_never_walks_the_sources_for_thumbnails(self):
(self.output_dir / "one.png").write_bytes(b"png")
with patch.object(
self.api,
"_plan_gallery_thumbnails",
side_effect=AssertionError("walked"),
):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertTrue(body["success"])
self.assertNotIn("thumbnails", body)
async def test_overview_sources_resolve_off_the_event_loop(self):
loop_thread = threading.get_ident()
threads = []
real = self.api._output_dirs_with_origin
def recording():
threads.append(threading.get_ident())
return real()
with patch.object(self.api, "_output_dirs_with_origin", side_effect=recording):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(body["sources"]["effective"], 1)
self.assertEqual(len(threads), 1)
self.assertNotEqual(threads[0], loop_thread)
async def test_overview_failure_is_an_error_envelope(self):
with (
patch.object(
self.api.db, "library_counts", side_effect=RuntimeError("secret")
),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("GET", "/prompt_manager/system/overview", 500)
self.assertEqual(
body,
{"success": False, "code": None, "error": "Failed to read the overview"},
)
class TestStatisticsRoute(SystemAPITestCase):
async def test_statistics_sum_real_run_counts(self):
a = self.save("Alpha", category="Landscapes", rating=5)
b = self.save("Beta", category="Portraits", rating=2)
self.api.db.record_prompt_use(a, times=3)
self.api.db.record_prompt_use(b)
body = await self.json("GET", "/prompt_manager/system/statistics")
self.assertEqual(
body["by_category"],
[
{"category": "Landscapes", "runs": 3, "average_rating": 5.0},
{"category": "Portraits", "runs": 1, "average_rating": 2.0},
],
)
self.assertEqual(body["totals"], self.api.db.get_statistics())
self.assertTrue(body["success"])
async def test_statistics_lists_rated_images_for_library_totals(self):
# D96: the Statistics dialog's "Library totals" reads the rated count
# with the runs (one read), from the same images the Overview counts.
rated = self.save("Rated", rating=4)
self.image(rated, "a.png")
self.image(self.save("Unrated"), "b.png")
body = await self.json("GET", "/prompt_manager/system/statistics")
self.assertEqual(body["rated_images"], 1)
self.assertEqual(
set(body), {"success", "by_category", "totals", "rated_images"}
)
async def test_statistics_failure_is_an_error_envelope(self):
with (
patch.object(
self.api.db, "usage_by_category", side_effect=RuntimeError("secret")
),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("GET", "/prompt_manager/system/statistics", 500)
self.assertEqual(
body,
{"success": False, "code": None, "error": "Failed to read statistics"},
)
class TestFailedReadsSettleFirst(SystemAPITestCase):
"""A failed read answers only after the route's other reads finished.
Answering on the first failure (``asyncio.gather`` without
``return_exceptions``) left the other executor reads running against a
database the caller may already be closing; on Python 3.10 CI that
crashed the test process inside SQLite.
"""
async def assert_answers_after_reads(self, url, failing, slow, error):
raised, release, finished = (threading.Event() for _ in range(3))
real = getattr(*slow)
def fail(*args, **kwargs):
raised.set()
raise RuntimeError("secret")
def held(*args, **kwargs):
release.wait()
try:
return real(*args, **kwargs)
finally:
finished.set()
with (
patch.object(*failing, side_effect=fail),
patch.object(*slow, side_effect=held),
self.assertLogs("prompt_manager.api", "ERROR"),
):
answer = asyncio.ensure_future(self.json("GET", url, 500))
try:
await asyncio.get_running_loop().run_in_executor(None, raised.wait)
done, _ = await asyncio.wait({answer}, timeout=0.5)
self.assertEqual(done, set(), "answered while a read still ran")
finally:
release.set()
body = await answer
self.assertTrue(finished.is_set())
self.assertEqual(body, {"success": False, "code": None, "error": error})
async def test_statistics(self):
await self.assert_answers_after_reads(
"/prompt_manager/system/statistics",
(self.api.db, "usage_by_category"),
(self.api.db, "get_statistics"),
"Failed to read statistics",
)
async def test_overview(self):
await self.assert_answers_after_reads(
"/prompt_manager/system/overview",
(self.api.db, "library_counts"),
(self.api, "_output_dirs_with_origin"),
"Failed to read the overview",
)
if __name__ == "__main__":
import unittest
unittest.main()
+38
View File
@@ -0,0 +1,38 @@
"""The System browser hooks only ever reset their own server's config.json (#249 S3)."""
import os
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from tests.browser.workspace_system_fixture import register_system_commands
class TestWorkspaceSystemFixture(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
self.root = Path(self.tmp.name) / "server"
self.root.mkdir()
def test_refuses_a_config_path_outside_its_root(self):
elsewhere = Path(self.tmp.name) / "config.json"
elsewhere.write_bytes(b'{"real": true}')
for value in (str(elsewhere), None):
env = {
k: v for k, v in os.environ.items() if k != "PROMPT_MANAGER_CONFIG_PATH"
}
if value is not None:
env["PROMPT_MANAGER_CONFIG_PATH"] = value
with (
self.subTest(value=value),
mock.patch.dict(os.environ, env, clear=True),
):
with self.assertRaisesRegex(RuntimeError, "PROMPT_MANAGER_CONFIG_PATH"):
register_system_commands(None, None, None, self.root)
self.assertEqual(elsewhere.read_bytes(), b'{"real": true}')
if __name__ == "__main__":
unittest.main()
+24 -1
View File
@@ -6,7 +6,7 @@ import { once } from 'node:events';
import { readFileSync } from 'node:fs';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { setTimeout as delay } from 'node:timers/promises';
import { startBrowserServer, createServerSupervisor, attachServerStderr } from '../../tools/vendor/start-browser-server.mjs';
import { startBrowserServer, createServerSupervisor, attachServerStderr, forwardPrefixedLines, SLOW_RESET_PREFIX } from '../../tools/vendor/start-browser-server.mjs';
import { workflowWorkers } from '../../tools/vendor/test-workers.mjs';
const fakeServer = fileURLToPath(new URL('./fake-browser-server.mjs', import.meta.url));
@@ -193,6 +193,29 @@ test('a failed test gets the server stderr written during it attached; a passing
assert.equal(String(attached[0].body), 'Traceback (most recent call last): boom\n');
});
test('only complete slow-reset report lines are forwarded, however the chunks split them', () => {
const written = [];
const forward = forwardPrefixedLines(SLOW_RESET_PREFIX, text => written.push(text));
forward(`aiohttp access line\n${SLOW_RESET_PREFIX}POST /__test__/reset still`);
assert.deepEqual(written, [], 'a report line waits for its newline');
forward(` running after 2.000s\nplain stderr\n${SLOW_RESET_PREFIX}Thread 0x1 [asyncio_0]\n${SLOW_RESET_PREFIX.slice(0, 5)}`);
forward(`${SLOW_RESET_PREFIX.slice(5)} File "server.py", line 1\nnot a report ${SLOW_RESET_PREFIX}\n`);
assert.deepEqual(written, [
`${SLOW_RESET_PREFIX}POST /__test__/reset still running after 2.000s\n${SLOW_RESET_PREFIX}Thread 0x1 [asyncio_0]\n`,
`${SLOW_RESET_PREFIX} File "server.py", line 1\n`,
]);
});
test('a slow-reset report on server stderr reaches the worker stderr; other stderr does not', async t => {
const reported = [];
const server = await startBrowserServer({ ...fake(), report: text => reported.push(text) });
t.after(() => server.stop());
await writeStderr(server, 'Traceback: ordinary stderr\n');
await writeStderr(server, `${SLOW_RESET_PREFIX}POST /__test__/reset answered 200 after 16.064s\n`);
assert.deepEqual(reported, [`${SLOW_RESET_PREFIX}POST /__test__/reset answered 200 after 16.064s\n`]);
assert.ok(server.stderrTail().includes('Traceback: ordinary stderr'), 'the stderr tail still keeps everything');
});
test('a server exits with its worker even when the worker is SIGKILLed', async () => {
const pid = await serverOfKilledWorker(`{ command: process.execPath, args: port => [${JSON.stringify(fakeServer)}, '--port', String(port), '--exit-on-stdin-eof'] }`);
await waitFor(() => !alive(pid), `server ${pid} to exit after its worker died`);
+14
View File
@@ -51,3 +51,17 @@ test('CI retains stock, execution and pack suites in order with uploaded executi
assert.deepEqual(commands, ['test:comfyui', 'test:executions', 'test:compat']);
assert.match(workflow, /path: tools\/vendor\/test-results\//);
});
test('real-host config waits for its own launcher, never for whichever server holds the port', async t => {
const previous = process.env.PM_COMFY_ROOT;
process.env.PM_COMFY_ROOT = '/unused-pinned-checkout';
t.after(() => { if (previous === undefined) delete process.env.PM_COMFY_ROOT; else process.env.PM_COMFY_ROOT = previous; });
const { default: config } = await import('../../tools/vendor/comfyui.config.mjs');
const [server] = config.webServer;
assert.equal(server.url, undefined);
assert.equal(server.port, undefined);
assert.equal(server.reuseExistingServer, false);
const ready = server.wait.stdout.exec('Starting server\nPM_COMFY_HOST_READY base-directory=/tmp/comfyassets-host-a b\r\nlater');
assert.equal(ready.groups.pm_comfy_host_base, '/tmp/comfyassets-host-a b');
assert.equal(server.wait.stdout.exec('To see the GUI go to: http://127.0.0.1:8189'), null);
});
+50 -1
View File
@@ -6,7 +6,8 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { once } from 'node:events';
import { createServer } from 'node:net';
import { launchComfyHost } from '../../tools/vendor/comfyui-host.mjs';
import { createServer as createHttpServer } from 'node:http';
import { launchComfyHost, assertPortFree, waitForOwnHost, hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
async function fixture(t, program) {
const directory = await mkdtemp(join(tmpdir(), 'pm-host-test-'));
@@ -232,3 +233,51 @@ test('execution launcher refuses an occupied port without touching an existing s
await new Promise(resolve => server.close(resolve));
}
});
// A stand-in ComfyUI that answers /system_stats with its own argv, as the real host does.
const answersOwnArgv = `const args = process.argv.slice(1);
require('node:http').createServer((request, response) => {
response.setHeader('content-type', 'application/json');
response.end(JSON.stringify({ system: { argv: args } }));
}).listen(Number(args[args.indexOf('--port') + 1]), '127.0.0.1');`;
async function listen(t, port, handler = (request, response) => response.end()) {
const server = createHttpServer(handler);
await new Promise((resolve, reject) => { server.once('error', reject); server.listen(port, '127.0.0.1', resolve); });
t.after(() => new Promise(resolve => server.close(resolve)));
return server;
}
test('real-host launcher refuses a port another run already holds and leaves that server running', async t => {
const server = await listen(t, 8199);
await assert.rejects(assertPortFree(8199), /8199 is already in use.*never shared/s);
assert.equal(server.listening, true);
});
test('real-host launcher is ready only when its own ComfyUI answers the port', async t => {
const options = await fixture(t, answersOwnArgv);
await assertPortFree(options.port);
const host = await launchComfyHost(options);
t.after(() => host.stop());
await waitForOwnHost(host, options.port);
const stats = await (await fetch(`http://127.0.0.1:${options.port}/system_stats`)).json();
assert.equal(hostBaseDirectory(stats), host.stateDirectory);
});
test('real-host launcher refuses a host it did not start that answers before its own binds', async t => {
await listen(t, 8199, (request, response) => {
response.setHeader('content-type', 'application/json');
response.end(JSON.stringify({ system: { argv: ['main.py', '--base-directory', '/other-run-state'] } }));
});
const options = await fixture(t, idle);
const host = await launchComfyHost(options);
t.after(() => host.stop());
await assert.rejects(waitForOwnHost(host, options.port), /did not start \(base directory \/other-run-state\)/);
});
test('real-host launcher reports a host that exits before it answers', async t => {
const options = await fixture(t, 'process.exit(3);');
const host = await launchComfyHost(options);
t.after(() => host.stop());
await assert.rejects(waitForOwnHost(host, options.port), /exited before it answered on port 8199 \(exit=3/);
});
+58 -3
View File
@@ -1,4 +1,5 @@
import { writeFile } from 'node:fs/promises';
import { createServer } from 'node:net';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { launchComfyHost as launchSharedHost } from '@comfyassets/toolkit/testing/host';
@@ -7,6 +8,9 @@ import runtime from './comfyui-runtime.json' with { type: 'json' };
export { launchSharedHost };
const repositoryRoot = fileURLToPath(new URL('../../', import.meta.url));
// comfyui.config.mjs waits for this line; its named group becomes PM_COMFY_HOST_BASE in the test workers.
export const HOST_READY_MARKER = 'PM_COMFY_HOST_READY';
const READY_POLL_MS = 250;
export async function preparePromptManagerPlugin({ installed, stateDirectory }) {
const configPath = join(installed, 'config.json');
@@ -27,17 +31,68 @@ export function launchComfyHost({ checkout, repository = repositoryRoot,
});
}
export async function assertPortFree(port) {
await new Promise((resolve, reject) => {
const probe = createServer();
probe.once('error', error => reject(new Error(`ComfyUI host port ${port} is already in use, often by another `
+ 'real-host run. A managed host is never shared: stop that run or set PM_COMFY_URL to a free '
+ 'http://127.0.0.1:PORT/ address.', { cause: error })));
probe.listen({ host: '127.0.0.1', port, exclusive: true }, () => probe.close(error => error ? reject(error) : resolve()));
});
}
export function hostBaseDirectory(stats) {
const argv = stats?.system?.argv;
if (!Array.isArray(argv)) return null;
const index = argv.indexOf('--base-directory');
return index >= 0 ? argv[index + 1] ?? null : null;
}
// Ready means this launcher's own ComfyUI answers: /system_stats reports its base directory.
// Any other answer on the port comes from a host this run did not start, so the run must not use it.
export async function waitForOwnHost(host, port, { fetchImpl = fetch } = {}) {
let exited = null;
host.finished.then(result => { exited = result; });
const url = `http://127.0.0.1:${port}/system_stats`;
for (;;) {
let stats = null;
try { stats = await (await fetchImpl(url)).json(); }
catch { stats = null; }
if (stats) {
const base = hostBaseDirectory(stats);
if (base === host.stateDirectory) return;
throw new Error(`Port ${port} is answered by a ComfyUI host this run did not start `
+ `(base directory ${base ?? 'unknown'}); another run is using the port.`);
}
if (exited) {
throw new Error(`ComfyUI host exited before it answered on port ${port} `
+ `(exit=${exited.code}, signal=${exited.signal || 'none'}).`);
}
await new Promise(resolve => setTimeout(resolve, READY_POLL_MS));
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
try {
const host = await launchComfyHost({ checkout: process.env.PM_COMFY_ROOT,
port: Number(process.env.PM_COMFY_PORT || 8189) });
const port = Number(process.env.PM_COMFY_PORT || 8189);
await assertPortFree(port);
const host = await launchComfyHost({ checkout: process.env.PM_COMFY_ROOT, port });
const stop = () => { host.stop().catch(error => { console.error(error.message); process.exitCode = 1; }); };
process.once('SIGINT', stop);
process.once('SIGTERM', stop);
let refused = false;
try {
await waitForOwnHost(host, port);
console.log(`${HOST_READY_MARKER} base-directory=${host.stateDirectory}`);
} catch (error) {
refused = true;
console.error(error.message);
await host.stop();
}
const result = await host.finished;
process.removeListener('SIGINT', stop);
process.removeListener('SIGTERM', stop);
if (result.code) console.error(`ComfyUI host exited with status ${result.code}; see tools/vendor/test-results/comfyui-host.log`);
process.exitCode = result.code;
process.exitCode = refused ? 1 : result.code;
} catch (error) { console.error(error.message); process.exitCode = 1; }
}
+6 -1
View File
@@ -1,5 +1,6 @@
import { defineConfig } from './node_modules/@playwright/test/index.mjs';
import { fileURLToPath } from 'node:url';
import { HOST_READY_MARKER } from './comfyui-host.mjs';
if (!process.env.PM_COMFY_ROOT) throw new Error('Set PM_COMFY_ROOT to a pinned disposable ComfyUI checkout');
const host = new URL(process.env.PM_COMFY_URL || 'http://127.0.0.1:8189/');
@@ -7,7 +8,11 @@ if (host.protocol !== 'http:' || host.hostname !== '127.0.0.1' || !host.port) th
const servers = [{
command: 'node tools/vendor/comfyui-host.mjs',
cwd: fileURLToPath(new URL('../../', import.meta.url)),
url: new URL('system_stats', host).href, timeout: 120000, reuseExistingServer: false,
// No URL wait: any server on the port would satisfy it, including another run's host that
// then stops mid-test. The launcher refuses a used port and prints this line only once its
// own ComfyUI answers; the group reaches the tests as PM_COMFY_HOST_BASE.
wait: { stdout: new RegExp(`^${HOST_READY_MARKER} base-directory=(?<pm_comfy_host_base>.+?)\r?$`, 'm') },
timeout: 120000, reuseExistingServer: false,
env: { PM_COMFY_PORT: host.port },
gracefulShutdown: { signal: 'SIGTERM', timeout: 10000 },
}];
+47 -5
View File
@@ -7,7 +7,7 @@
},
"api-client.js": {
"lines": 98,
"branches": 95,
"branches": 96,
"functions": 98,
"statements": 98
},
@@ -37,15 +37,15 @@
},
"gallery-settings.js": {
"lines": 98,
"branches": 90,
"branches": 94,
"functions": 98,
"statements": 98
},
"gallery.js": {
"lines": 40,
"lines": 41,
"branches": 63,
"functions": 44,
"statements": 40
"statements": 41
},
"html-escape.js": {
"lines": 98,
@@ -163,7 +163,7 @@
},
"workspace/activity-scan.mjs": {
"lines": 96,
"branches": 73,
"branches": 78,
"functions": 98,
"statements": 96
},
@@ -401,6 +401,12 @@
"functions": 98,
"statements": 98
},
"workspace/preferences.mjs": {
"lines": 98,
"branches": 96,
"functions": 98,
"statements": 98
},
"workspace/prompt-blocks.mjs": {
"lines": 90,
"branches": 92,
@@ -443,6 +449,42 @@
"functions": 98,
"statements": 98
},
"workspace/system-appearance.mjs": {
"lines": 98,
"branches": 93,
"functions": 98,
"statements": 98
},
"workspace/system-controls.mjs": {
"lines": 98,
"branches": 87,
"functions": 89,
"statements": 98
},
"workspace/system-overview.mjs": {
"lines": 96,
"branches": 92,
"functions": 98,
"statements": 96
},
"workspace/system-sources.mjs": {
"lines": 98,
"branches": 88,
"functions": 95,
"statements": 98
},
"workspace/system-state.mjs": {
"lines": 98,
"branches": 94,
"functions": 98,
"statements": 98
},
"workspace/system.mjs": {
"lines": 98,
"branches": 92,
"functions": 86,
"statements": 98
},
"workspace/theme.mjs": {
"lines": 98,
"branches": 93,
+19
View File
@@ -17,6 +17,22 @@ const HOST = '127.0.0.1';
// tests/browser/server.py prints this once it holds its listening socket. Only a
// complete line counts: a chunk may end mid-number.
const PORT_ANNOUNCEMENT = /^PM_TEST_SERVER_PORT=(\d+)\r?\n/m;
// tests/browser/slow_reset_watch.py starts every line of a slow-reset report with
// this. Those lines go to the worker's stderr, so the job log shows a slow reset
// even when its test passes.
export const SLOW_RESET_PREFIX = 'pm-slow-reset| ';
// A stderr chunk handler that passes every complete line starting with `prefix`
// to `write`. A line split across chunks is held until its newline arrives.
export function forwardPrefixedLines(prefix, write) {
let pending = '';
return chunk => {
const lines = (pending + chunk).split('\n');
pending = lines.pop().slice(-STDERR_TAIL_BYTES);
const matched = lines.filter(line => line.startsWith(prefix));
if (matched.length) write(matched.map(line => `${line}\n`).join(''));
};
}
// The server exits when its stdin closes, so a worker that dies without teardown
// (SIGKILL) leaves no server behind; the launcher holds the pipe's other end.
@@ -40,6 +56,7 @@ export async function startBrowserServer({
cwd = repositoryRoot,
timeoutMs = STARTUP_TIMEOUT_MS,
stderrLogChars = STDERR_LOG_CHARS,
report = chunk => process.stderr.write(chunk),
} = {}) {
if (!Number.isInteger(port) || port < 0 || port > 65535) {
throw new Error(`startBrowserServer needs a port from 0 to 65535, got ${port}`);
@@ -50,7 +67,9 @@ export async function startBrowserServer({
let stderr = '';
let stderrTotal = 0;
child.stderr.setEncoding('utf8');
const forwardReport = forwardPrefixedLines(SLOW_RESET_PREFIX, report);
child.stderr.on('data', chunk => {
forwardReport(chunk);
stderr = (stderr + chunk).slice(-Math.max(stderrLogChars, STDERR_TAIL_BYTES));
stderrTotal += chunk.length;
});
+95 -3
View File
@@ -6,19 +6,28 @@ the inspection projection (``project_graph``: strict JSON, the depth and node
bounds, big integers and decimals as text), and only graphs it parsed, through
the shared parser (``extract_generation_params``, the Python twin of the
browser parser the inspector fills from since #251). No separate seed
heuristic, no graph resolution and no extra file reads. One parse feeds both
the seed and the positive that confirms a first-line card title.
heuristic and no extra file reads. One parse feeds both the seed and the
positive that confirms a first-line card title. One exception keeps the card
from ever showing a seed the inspector does not: when sampler stages read
different seed inputs and the inspector's resolution names one of them for a
resolved pair (whose facts are that stage's), the card shows no seed when the
named stage's seed differs from the parser's. Only such graphs are resolved,
with the inspector's own profile.
"""
import json
import re
from collections.abc import Mapping
from .generation_params import (
PLACEHOLDERS,
GenerationParamsError,
_is_sampler,
extract_generation_params,
)
from .inspection_response import project_graph
from .inspection_semantics import SemanticsError, get_semantics
from .prompt_pair_resolver import INSPECTION_PROFILE, resolve_prompt_pair
__all__ = [
"MAX_RECORD_BYTES",
@@ -73,10 +82,93 @@ def header_params(info) -> dict | None:
if not graphs:
return None
try:
return extract_generation_params(graphs)
params = extract_generation_params(graphs)
except (GenerationParamsError, RecursionError):
# Defence in depth: a parser failure gives no card facts, never a 500.
return None
if isinstance(params, dict) and _inspector_names_another_seed(
graphs, card_seed(params)
):
return {**params, "seed": None}
return params
_SEED_INPUTS = ("seed", "noise_seed", "noise")
def _stage_seed(graph: Mapping, node_id: str) -> str | None:
"""The seed the parser reads for one sampler node, or None when unknown.
A literal seed input is that seed. A linked one is read by the parser
itself when it treats the node as a sampler, with every other such node
hidden from it (no ``positive`` input), so linked seeds resolve as they do
on the card; any other linked seed is unknown.
"""
node = graph.get(node_id)
if not isinstance(node, Mapping) or not isinstance(node.get("inputs"), Mapping):
return None
inputs = node["inputs"]
literal = next((inputs[k] for k in _SEED_INPUTS if k in inputs), None)
if not isinstance(literal, (list, tuple)):
# A literal needs no second parse (the cost is measured in TESTING.md).
return card_seed({"seed": literal})
if _is_sampler(node):
only = {
key: (
{**other, "inputs": {**other["inputs"], "positive": None}}
if key != node_id and _is_sampler(other)
else other
)
for key, other in graph.items()
}
try:
return card_seed(extract_generation_params({"prompt": only}))
except (GenerationParamsError, RecursionError):
return None
return None
def _inspector_names_another_seed(graphs: dict, parsed: str | None) -> bool:
"""Whether the inspector shows another stage's seed than the parser's.
True when the prompt graph's sampler stages read different seed inputs, the
inspection resolution names one sampler for a resolved pair (the
inspector's facts are then that stage's, D19), and that stage's seed is
known and differs from ``parsed``. A seed that agrees is kept.
"""
if parsed is None:
return False
graph = graphs.get("prompt")
if not isinstance(graph, Mapping):
return False
try:
samplers = get_semantics().samplers
except SemanticsError:
return False
seeds = set()
for node in graph.values():
if not isinstance(node, Mapping) or not isinstance(node.get("inputs"), Mapping):
continue
if not isinstance(node.get("class_type"), str):
continue
if node["class_type"] not in samplers:
continue
inputs = node["inputs"]
value = next((inputs[key] for key in _SEED_INPUTS if key in inputs), None)
seeds.add(json.dumps(value, sort_keys=True, default=str))
if len(seeds) < 2:
return False
try:
resolution = resolve_prompt_pair(
{"prompt": dict(graph)}, profile=INSPECTION_PROFILE
)
except RecursionError:
return True
named = resolution["sampler_id"]
if resolution["state"] != "resolved" or named is None:
return False
seed = _stage_seed(graph, named)
return seed is not None and seed != parsed
def card_seed(params: dict | None) -> str | None:
+25
View File
@@ -28,6 +28,7 @@ __all__ = [
"extract_generation_params",
"extract_from_records",
"parse_js_json",
"parse_page_json",
]
PLACEHOLDERS = MappingProxyType(
@@ -953,6 +954,30 @@ def parse_js_json(text):
return None
def _nan_as_null(name):
"""The legacy pages' NaN cleaning; Infinity still throws in ``JSON.parse``."""
if name == "NaN":
return None
raise ValueError(f"{name} is not JSON")
def parse_page_json(text):
"""A metadata record as the legacy gallery and admin pages parsed it.
They replace ``NaN`` with ``null`` and then ``JSON.parse`` (ComfyUI writes
``"is_changed": [NaN]`` for nodes that always re-run, PromptManager among
them). Only the NaN number reads as null here: the pages' regex also
rewrote the word inside strings, which changed prompt text. None where the
parse throws.
"""
if not isinstance(text, str):
return None
try:
return json.loads(text, parse_constant=_nan_as_null, parse_int=_double_integer)
except (ValueError, RecursionError):
return None
def extract_from_records(prompt_text, workflow_text):
"""``extract_generation_params`` for raw ``prompt``/``workflow`` record text.

Some files were not shown because too many files have changed in this diff Show More