Compare commits

..
Author SHA1 Message Date
Vito Sansevero 23af132532 Merge remote-tracking branch 'origin/main' into fix/duplicate-delete-serve 2026-10-23 05:17:13 -07:00
Vito b0a025a326 fix(resolver): read NaN as null in the scan's legacy-parser fallback (#304)
ComfyUI stamps "is_changed": [NaN] on nodes that always re-run, the
PromptManager node among them. The legacy gallery and admin pages replaced
NaN with null before JSON.parse, so they traced these files' prompts. The
scan's fallback parsed records strictly, so every NaN-stamped file the
resolver could not decide (hires-fix and refiner chains, custom latent
nodes) fell to "Needs review" although legacy linked its prompt.

parse_page_json parses as the legacy pages did, but only the NaN number
reads as null: the word inside prompt text is kept. Infinity still makes
the record unreadable, as it did in the pages.
2026-10-23 05:17:10 -07:00
Vito c86739178d feat(system): System page shell, review dialog and Overview (#298)
* feat(system): read-only health check and truthful overview counts

POST /prompt_manager/system/health runs the health check off-loop as a
read registry job of kind "health" (one at a time, beside a scan, not
cancellable, no file outcomes) and keeps the last report; GET returns it
or null. The check reads one snapshot on its own mode=ro connection,
closed in finally: integrity_check(100), foreign_key_check, stored hashes
against utils.hashing, duplicate pair rows, the legacy consistency
queries (now shared with check_consistency), and image links whose file
is gone, counted only under readable sources. An unavailable source, or
the absent Needs-review queue, makes its check unavailable and the
report incomplete instead of producing a count.

GET /prompt_manager/system/overview returns Library-eligible positive
prompts and active negative variants, indexed images, the last health
summary or null, backup null, the Needs-review count null, the effective
sources with whether they were auto-detected, and the thumbnail plan's
missing count. GET /prompt_manager/system/statistics lists runs and
average rating per category of Library parents, with the /stats totals.

* feat(system): native System route, drafts and shared review dialog

- #/system/<section> renders the concept's System shell: section nav,
  settings layout and Save row with truthful status (D92); each section
  not delivered yet hands off to its legacy tools (D94). Overview's System
  card opens #/system/overview (Q12).
- A module-level draft store saves browser, then host (expected_revision),
  then runtime; failed domains stay dirty, a stale revision is a conflict.
- System row builders (setting, operation, section nav) on the scope.
- openReviewDialog and backupNotice extracted from the scan review, which
  is now a thin caller with identical markup.
- Overview images gain a "rated" count (images whose linked prompt is
  rated); the missing-files health check reports per source, so partial
  availability is visible.

* feat(system): native Overview with health report and statistics

System > Overview renders the concept's metrics and its Library health and
Workspace preferences surfaces from GET /system/overview: real counts,
"Not checked" until a health check ran, no invented review count, backup
or thumbnail status (D93). Explore opens "Usage statistics" with the real
runs and ratings per category and the Library totals (D96); the
statistics route now also returns rated_images.

Maintenance shows the concept's notice with Run health checks above its
hand-off: the read-only check runs as a registry read job and opens the
"Health report" with a chip per real result, missing files per source
(D95). A load failure shows Retry and no zeros; late answers never
repaint a newer section.

Adds the System concept comparison (shell, Overview, both dialogs, the
notice, mutation checks), the System workflow spec and its fixture
hooks, and the TESTING.md "Native System" section.

* test(system): coverage floors for the native Overview

Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes.

* fix(system): keep health reports per database and off the event loop

- A restore that replaces the database forgets the stored health report,
  and a check that started before the swap is not kept (database
  generation stamped at start).
- The Overview no longer walks every source for the missing-thumbnail
  count; the page reads /images/thumbnails/status on its own.
- Image sources for the health check and the Overview resolve in the
  executor; a failure to start the check answers the coded envelope.
- Any exception fails only its own check; missing files reads its rows in
  the snapshot and stats the files after it ended.
- The read-only URI keeps Windows UNC paths (empty authority).
- An integrity count at SQLite's limit is flagged as capped.

* fix(system): thumbnail row read, tracked health job and guarded host save

- The Overview's thumbnail row is its own read of the thumbnail status:
  "Checking thumbnails…" until it answers, unavailable when it fails.
- The health job is tracked in the Activity store (new store.track) and
  followed through its event stream instead of polling every 150 ms.
- A host save before a read returned a revision fails without a request;
  System reads the host values on mount and after a conflict.
- "100+ problems found" when the integrity count is capped.
- D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone.
- The concept spec builds the health report from the concept's own rows
  and asserts the full unmapped path list; surface tops pin the 18px
  offset.
- TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI
  and the new behavior. Coverage floors raised for the touched files.

* fix(system): make two health tests portable to Windows

- Windows forbids "?" in file names, so the URI-character open test
  uses "#", "%" and a space there.
- ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of
  raising, so the bad-path test raises as POSIX does to reach the
  missing-files failure path on every platform.
2026-10-23 04:59:18 -07:00
Vito Sansevero a4b75c3eae fix(gallery): check duplicate deletes in the scanned root and refresh the listing
The duplicate cleanup workflow failed in CI with the deleted file still
served (200). The delete was real: the scan reads only the output root,
and the browser fixture's input root holds a different shared/same.png.
The test checked a bare /images/serve/<path> URL, which tries every
gallery root, so after the output copy went it answered with the input
copy. That only happened when the kernel stamped a bulk copy a fraction
of a millisecond before the wall-clock stamp on shared/same.png, which
made shared/same.png a Delete candidate (CI trace: 0000 .8605533,
same .8606777, 0001 .8618972).

The workflow now checks each file in its own root (?root=N), checks the
input copy survives, and a new workflow forces the CI order through the
scan response so the collision runs every time.

The server log in that failure also showed the gallery listing a file
the cleanup had deleted: delete_duplicate_images left the 30 s listing
cache, so the gallery kept counting the file. A successful delete now
invalidates the gallery cache.
2026-10-23 04:11:16 -07:00
Vito b50bf862f0 fix(inspect): hold a route change until a starting video settles (#294)
* fix(inspect): keep a replaced video in place until its pipeline has started

WebKit (GStreamer) deadlocks the page when a video leaves the page or the
viewport before its pipeline has started: the main thread changes the
pipeline state and waits for the source's streaming thread, which waits on
the main thread in a synchronous bus handler (native stacks of the hung WPE
web process in the pinned image). Removing, hiding or merely pushing the
video below the fold all hung; none of pause(), removing src, load() or the
revoke was the cause on its own.

A video replaced while still starting now stays where it is, inert and
aria-hidden, with the new preview inserted after it, and the session's
revoke of its URL waits. Once it reports metadata or an error (or after
15 s) it is paused, loses its src, is reset with load() and removed, and
only then is its URL revoked. Every released video is reset with load(), so
no decoder is kept per replaced video.

The loadstart replacement spec now runs on WebKit too (15 fresh pages,
replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause.

* fix(inspect): hold a route change until a starting video settles

A route change within the workspace detaches the Inspect page, and WebKit
(GStreamer) deadlocks when a video still starting is detached. Route
changes at loadstart + 0-7 ms hung 4-5 of 120 fresh pages (14 of 120 with
no page code releasing anything, so the detach itself is the trigger).

Inspect now exposes leaveGuard(): a promise while a video on the page is
still starting (current or replaced). The shell renders the next route
once it settles (metadata, error, or the 15 s bound); the latest route
wins. Route changes at loadstart + 0-7 ms now hang 0 of 120.

Not changed, with numbers in TESTING.md D62:
- A replaced video still shows above the new preview until it settles.
  Same-slot stacking changes the video's own box and hung again
  (absolute 5/120, absolute + opacity 0 8/120, one-cell grid 4/120).
- A real page leave cannot wait. Navigating away hung 8/120 and reload
  22/120 even with no page-side release, so it stays a known limitation.
2026-10-23 03:51:38 -07:00
Vito 4f1c6e236b test(browser): report a slow test reset with every thread's stack (#299)
* test(browser): report a slow test reset with every thread's stack

On #291 (run 39904288262, WebKit shard 2/2) one /__test__/reset
{chronological_history} waited 16 s on its worker's server, with nothing
on the server's stderr. The cause did not reproduce: the same worker's
file sequence (library, library editor, tagging lifecycle, visual) at
2 CPUs three times, and WebKit shard 2/2 at 2 CPUs, kept every reset
under 2 s.

A reset still running after 2 s now reports, while it waits, every
thread's stack (faulthandler), the default executor's threads and queue,
the open connections and WAL size, and on Linux each thread's scheduler
state and kernel wait channel. When it ends it reports the total and how
long each executor call queued and ran. The lines carry a
"pm-slow-reset| " prefix and the worker fixture copies them to the job
log, so the next one shows up even in a test that passes.

The report never reads another thread's connection. Reading even
in_transaction on a connection busy in SQLite blocks on its mutex with
the GIL held: a first version of this report froze the whole server
until the 30 s busy timeout. A test holds a real write lock behind a
reset and checks the server keeps answering while it waits.

* test(browser): name each thread in the slow-reset stacks

Before Python 3.14 faulthandler prints thread ids only, so the report
now lists which name each id belongs to. CI on 3.10 to 3.13 failed the
unit test that looks for the blocked thread by name.

* test(browser): keep the slow-reset summary on a coarse clock

On Windows the report timer can fire a few milliseconds short of the
threshold, and the summary was then dropped. A reset that was reported
now always gets its summary. The server test uses the plain reset, which
the Windows jobs can seed, and puts the server stderr in its failure
messages.
2026-10-23 03:24:20 -07:00
Vito 5b2c21bdaf test(annotations): capture the details panel after it has loaded (#292)
"Tag opens the dialog without selecting the card" read the panel while it
still said "Loading file details…", then compared it with the same card's
finished details. On a slow WebKit runner the read raced the load.
2026-10-23 03:03:38 -07:00
Vito 25e7ae158b test(history): isolate restore-rotation subtests and wait on the rotation event (#300)
On a slow Windows runner the restore that the rotation test injects between
the writer's transactions outlived wait_for's 3 s worker deadline. The
subtest failed, and because every subtest shared the process-wide context
budget and left its writer running, the still-held restore cascaded: the
next subtest saw the in-flight 621-byte payload as charged, and the late
restore rotated the epoch under the last subtest's writer.

Each subtest now owns its writer and budget and closes them before the next
starts, and the handoff waits on an event set once the rotation returns,
with the existing BLOCK_SECONDS ceiling, before the short pending wait.
A new test holds a restore between transactions with events and proves the
payload is charged only while in flight, released afterwards, and that the
writer re-probes and accepts new evidence.
2026-10-23 02:36:07 -07:00
Vito 08151ca93d test(library): read one "now" in the fixture and the page across midnight (#287)
* test(library): read one "now" in the fixture and the page across midnight

The concept fixture recorded each prompt's last use with the database's wall
clock at seeding, while the page computed "today" from its own clock at
render. A run seeded at 23:59:5x UTC and rendered after midnight read "Used
yesterday" against the concept's "Used today" (PR #284, frontend coverage
shard 2/3).

The concept fixture now takes one instant (optional "now" through the
concept reset hook, the wall clock otherwise) for every last use, mtime and
run time it writes and returns it as manifest.now; seedConceptAt fixes the
browser's Date at that instant. The Library and Images concept specs seed
through it, and the two workspace specs that assert "Used today" and the
"Today" caption fix the page clock at the time they patch or seed. A new
test starts the browser at 23:59:58 UTC, seeds at that instant and passes
midnight before either document opens.

* test(activity): judge the lost-server backoff by its gaps, not a timed window

"lost server clears the pill and marks jobs ended" counted failed job-list
re-reads in a 3.5 s waitForTimeout window that started once the test saw
the ended job. Re-reads back off 1 s, 2 s, then 4 s, so the third lands
4 s after the second. A runner that noticed the ended job more than 0.5 s
late moved the window over the third read (firefox shard 1/2 on #287:
3 reads, expected at most 2).

The test now waits for the third failed read and asserts the gaps between
reads: at least 2 s, then at least 4 s, with 100 ms allowed for the
request being reported late.

* test(activity): replace the spec's remaining timer waits with gaps and a run-out clock

Three checks in workspace-activity.spec.mjs waited on wall-clock timers:

- "lost server": waitForTimeout(2500), then at most 2 list reads. The read
  after the one that found the restart must be the periodic poll, so the
  test now waits for it and asserts the gap is at least POLL_MS (less
  100 ms for late request reporting). Any extra read lands in that gap.
- "200-row cap note" (review M12): waitForTimeout(1500), then no detail
  read. The page's timers are now Playwright's clock. The test runs one
  read-throttle interval out, so any read the counter update scheduled is
  sent before the log opens. Once the log's read has answered, exactly one
  read must be listed.
- "busy start" (review M14): waitForTimeout(7000), then the link toast is
  still visible. The test now runs the page's clock 7 s on, which fires a
  6.5 s clear however slowly the runner got there.

RED, each against a temporary product change: POLL_MS 3000 (gap 3004 ms,
expected at least 14900), detail re-read on new log lines while the log is
closed (an unexpected read), LINK_TOAST_MS 6500 (toast link gone), and
BACKOFF_FIRST_MS 200 for the earlier backoff check (gap 407 ms, expected at
least 1900).
2026-10-23 01:58:02 -07:00
Vito b2cc489fd4 feat(inspect): copy settings summary and local workflow roles (#286)
* feat(inspect): copy settings summary in the full inspector

The legacy metadata page's COPY ALL (LI-4) becomes "Copy settings summary"
after the six facts of the shared full inspector, so the Images viewer,
the recorded history's current file and Inspect all offer it; the Images
side panel keeps the concept's footer. The text is the legacy copyAllData
template with its placeholders, filled from what the inspector shows. A
unit test runs the template read from web/metadata.html on the legacy
parse of every parity PNG: 22 of 25 are byte-equal, and the other three
differ only by the ZeroOut guard (#249) or facts the legacy parse missed.

Copying follows the concept's prompt copy: the toast "Settings summary
copied.", or the select-to-copy dialog (now titled per copy) without a
clipboard. The concept comparisons map the button as D-NEW-1.

* feat(inspect): local execution evidence and prompt tracing

A browser port of the server's inspection resolver for one sampler
(PR L3 Task 8, LI-1). local-resolver.mjs answers inspect_prompt_evidence's
{prompts, resolution} for a local read's records: execution evidence
(read_pair_execution and its per-node match), every traced strategy
(encoders, strings, concatenations, PromptManager and its historical
classes, passthrough, zeroed, selector branches, composites, guiders),
the latent stage walk, the R1 overwritten-text guard and the inspection
profile's visit, depth, text, path, built and scanned bounds, with the
same reason codes and outcome lines. py-json.mjs reads the records as
CPython's json.loads does (dict order and last-wins keys, int versus
float, BigInt past 2^53, NaN) and measures and strips text as str does.
Several sampler candidates, detailer stages and A1111 parameters raise
LocalResolverGap until Task 9. Nothing is wired into Inspect yet.

Python is the oracle: test_local_inspect_parity compares the port with
the server on the parity corpus, 107 named record sets, every work bound
at its edge and 600 seeded fuzzed graphs, and requires gaps to be exactly
the Task 9 paths. Sixteen porting mutations each fail the suite. The
Node unit test replays committed server answers, which the Python suite
keeps current.

* fix(inspect): a list or object workflow node type answers malformed

The R1 overwritten-text check looked every UI workflow node's `type` up
among the subgraph IDs. A `type` that is a JSON list or object raised
TypeError there, so the inspect route answered 500 "Prompt inspection
failed". Such a workflow is now malformed evidence: the resolution is
unresolved with reason_code `malformed` and the inspector's own
"Prompt evidence is malformed" line, never exception text.

The browser port answers the same, and the parity corpus gains the
list, object, nested-subgraph and skipped-ID shapes.

* fix(inspect): a list or object graph class_type is never a preview output

With several sampler candidates the inspector first drops stages whose
outputs reach only previews. That pass looked each API graph node's
`class_type` up in the preview class set, so a list or object
`class_type` raised TypeError and the inspect route answered 500. Only
a string names a class now; such a node is an ordinary terminal, as the
rest of the resolver already treats it.

* feat(inspect): local candidate selection matching the inspector

The browser port of the inspection resolver now answers every record set
the server does, so no path is left to the server any more:

- several sampler candidates, with the identical-branch rule (#270): a
  compare grid whose branches all resolve to one pair (text and recipe)
  is resolved, any other mix is ambiguous;
- stages whose outputs reach only previews are dropped first;
- detailer stages are judged with the samplers in their image ancestry
  (multistage-differs, cycles, depth and the per-stage visit budgets),
  and a sampler that only feeds detailers is not a candidate of its own;
- A1111 `parameters`, with Python's multiline anchors, `\s` and Unicode
  `\d`, the bare footer and Template trailers, counted in code points.

The parity corpus gains the selection cases and a seeded fuzzer, and
every inspection vector now gives the server's prompts, resolution and
facts locally. 16 of 17 mutations are caught; the uncaught one (not
flattening an ambiguous upstream stage) cannot change any answer.

* feat(inspect): local prompts resolved like the server's

A file chosen on the Inspect page now shows the prompts and resolution
the server's inspector gives the same records: buildLocalPayload takes
its roles from the local resolver instead of the interim open roles, so
the legacy fill applies exactly where it does for Images.

The resolver reads the server's own node table. The page fetches
web/js/inspection-semantics.json once through the API client as it
mounts, before any selection; the static route serves that one JSON
file by exact path (every other JSON stays forbidden). A table that
cannot be fetched answers "The inspector's node table is unavailable."
as the server does, and a later selection asks again.

The Copy settings summary deviation is numbered D117.

* fix(inspect): skip a workflow node whose type is a list or object

The overwritten-PromptManager walk now skips such a node, on the server
and in the browser port, instead of marking the whole file malformed.
A prompt that resolves still resolves; the other nodes still feed the
check.

* fix(inspect): one JSON depth cap for both browser parsers

py-json.mjs and local-video.mjs now refuse past the same 1,000 nested
arrays or objects, counting every container as CPython's scanner does.
Boundary vectors at 999, 1,000 and 1,001 for a deep prompt and a deep
workflow run through the Python oracle, and TESTING.md gives the
measured per-version server limits and both divergence directions.

* fix(inspect): realpath containment for the exact-path node table route

inspection-semantics.json now goes through the same realpath check as
every other static file, so a symlink out of web/ is refused. The
forbidden-path test asserts exactly 403.

* fix(inspect): abortable, stepped local reads with a busy state

The wait for the node table now ends with the selection's signal. A
local read resolves and projects in bounded steps with a turn of the
event loop between them, and the inspector carries aria-busy while it
loads. A budget test covers maximal 2 MiB graphs and checks the stepped
read is byte-identical to one synchronous build. The D117 wrap rule now
targets only the summary button, and TESTING.md scopes the restart line
to PRs L1 and L2.

* fix(inspect): record the two CPython-JSON emulations as a follow-up
2026-10-23 01:38:36 -07:00
Vito 347de6a023 test(logging): keep prior.log closed against in-flight records (#296)
TestPackageLoggerIsolation installed a plain logging.FileHandler on
prior.log and closed it while it was still attached to the live
"prompt_manager" logger. FileHandler.emit reopens the file whenever
stream is None, so a record another thread had already routed to the
handler (other tests leave image workers and watchdog threads logging
under prompt_manager.*) reopened prior.log after close and kept it open
through TemporaryDirectory.cleanup(), which Windows refuses (WinError 32).

Use a prior handler that refuses to emit once closed (the same guard
as the product's _OwnedRotatingFileHandler), detach handlers before
closing them, add a deterministic in-flight-record reproduction, and
assert no open handles under the temp directory before its cleanup.
2026-10-23 00:52:00 -07:00
Vito fd747a0ffc fix(inspect): keep a replaced video in place until its pipeline has started (#291)
WebKit (GStreamer) deadlocks the page when a video leaves the page or the
viewport before its pipeline has started: the main thread changes the
pipeline state and waits for the source's streaming thread, which waits on
the main thread in a synchronous bus handler (native stacks of the hung WPE
web process in the pinned image). Removing, hiding or merely pushing the
video below the fold all hung; none of pause(), removing src, load() or the
revoke was the cause on its own.

A video replaced while still starting now stays where it is, inert and
aria-hidden, with the new preview inserted after it, and the session's
revoke of its URL waits. Once it reports metadata or an error (or after
15 s) it is paused, loses its src, is reset with load() and removed, and
only then is its URL revoked. Every released video is reset with load(), so
no decoder is kept per replaced video.

The loadstart replacement spec now runs on WebKit too (15 fresh pages,
replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause.
2026-10-23 00:31:37 -07:00
Vito 7a4b6eebb2 test(shell): split the shell concept comparison per route and motion (#297)
"shell geometry, tokens and controls match the concept" compared all three
routes (Images, Library, Tags) under both motions in one test per size and
theme: 12 page loads, 12 captures and 36 hovers under the 30 s default
budget. Over the last 40 Viewer.js runs it took 22 s median and up to
33 s on CI WebKit, and timed out 6 times (main run 39886719087 among them).
The CI traces show steady progress with no stalled step (the longest one
is a 0.8 s navigation), so the cause is the amount of work in one test,
not a hang or a race.

Each route and motion is now its own test, still covering both sizes and
both themes, with the same assertions and captures.

Under run-pw-ci.sh on WebKit at 2 CPUs, --repeat-each=10:
before 18/40 timed out (median 26.9 s, max 32.5 s);
after 0/240 (median 6.5 s, p90 9.8 s, max 17.2 s).
2026-10-23 00:08:43 -07:00
Vito a64e99402b test(images): wait for the resized grid row before asserting no overflow (#295)
At 700px and 390px the Images grid has two columns, so the poll for a card at
the row start was already true of the 700px layout. Under CI load WebKit can run
no rendering frame between setViewportSize and the overflow check, so the cards
still carried the 700px widths and scrollWidth exceeded the viewport by 296px.
Wait until the first row fits the resized grid instead.
2026-10-22 22:46:50 -07:00
Vito 3b1d00501f fix(config): serialized config store and registry admission (#289)
* fix(config): report settings persistence failures (#220)

Settings saves now go through one serialized config store
(py/config_store.py). A commit re-reads config.json under a process
lock, applies the change to a copy and replaces the file atomically
(temp file beside it, fsync, close, os.replace), so a failed write
leaves the old file in place.

POST /prompt_manager/settings commits first and only then publishes
the values at runtime. A write failure is 500 config-write-failed, an
unparseable config.json is 409 config-unreadable (the file is kept
instead of replaced) and a stale expected_revision is 409
settings-changed. A save reports persisted, revision, runtime_applied
and restart_required_fields; GET adds the revision. Only submitted keys
are written; other sections and the CivitAI key are kept.

* fix(lora): report integration save failures (#220)

POST /prompt_manager/lora/enable commits integrations.lora_manager
through the config store and publishes the runtime values only after
config.json holds them. Before, it set the runtime values, rewrote the
whole runtime config (dropping sections it did not know) and reported
success even when the write had failed.

A write failure is 500 config-write-failed and changes nothing at
runtime; an unparseable config.json is 409 config-unreadable. The
body goes through the capped JSON reader and the flags must be
booleans. A trigger word cache that fails to load is reported as
persisted:true, cache_loaded:false. The CivitAI key is never echoed.
The LoRA tests now write to a temporary config path.

* fix(jobs): refuse legacy record-changing operations while a job runs

The job registry gains admission classes. A records job is refused
while any records or guarded job runs, a guarded job while a records
job runs, and read jobs share only the one-job-per-kind rule.
record_changing=True stays as the spelling of records. Job summaries
add cancellable and admission, and a non-cancellable job ignores
cancel.

JobRegistry.hold() registers a running, non-cancellable job for a
legacy-page operation and ends it when the request does. The legacy
restore, maintenance, prompt cleanup, duplicate image delete, LoRA
import and batch auto-tag routes hold as records; thumbnail generation
(both routes) and thumbnail clear hold as guarded. A maintenance run
of only read checks holds as read. So a restore during a scan, or a
scan during a legacy restore, is refused with the running job named
instead of interleaving; this closes the gap noted on #269.

Refusals keep each route's shape: the JSON routes return the legacy
409 "<title> already running", and the thumbnail and LoRA streams
send their existing error event.

* test(lora): keep the preserved-section fixture free of secret values

CodeQL flagged the fixture that writes the secret constant into a test
config file as clear-text storage. The test only needs a marker value
to prove the stored key survives a save.

* fix(config): read under the commit lock and publish saves in write order

- ConfigStore.read takes the commit lock, so no read handle is open while
  config.json is replaced (Windows refuses that replace).
- os.replace is retried briefly on a Windows PermissionError (antivirus or
  indexer handles) before the save is reported as not written.
- The temp file is written beside realpath(config.json), so a symlinked
  config keeps its link and a bind mount fails truthfully.
- commit() takes a publish callback that runs under the lock after the
  write; settings and LoRA saves publish there, so two concurrent saves
  reach the runtime in write order and the runtime ends on the last one.
- The concurrency tests now wait until the second caller is blocked on the
  lock (tests/lock_probe.py) before releasing the first.

* fix(jobs): require an admission class, admit holds on entry, true Activity states

- JobRegistry.start raises TypeError when neither admission nor
  record_changing is given, so a new job never runs as "read" beside a
  scan by omission.
- hold() admits when its block is entered, so a hold that is never entered
  leaves no running records job behind; callers that answer a refusal in
  their own way check blocker() first, with no await before the block.
- A held SSE operation that ends in an error event (thumbnail generation,
  legacy batch auto-tag) is listed as failed, and refused input (duplicate
  delete without paths, an unreadable thumbnail body) is answered before
  the hold, so it is not listed as a job.
- Activity keeps a job's cancellable flag (true when an older server omits
  it) and shows Cancel only for a running, cancellable job, in Recent work
  and in the job view. A held legacy job no longer offers a Cancel that
  does nothing.

* fix(admin): legacy pages show why a request was refused

A job running in the workspace refuses the legacy record-changing requests,
and a corrupt config.json refuses a settings save. The legacy pages now
name the reason instead of hiding it:

- LoRA import: the busy and failure frames carry status and progress, and
  the admin modal handles type "error": it shows the message, notifies and
  closes, instead of staying on "Initializing..." at 0%.
- Legacy batch auto-tag: a refusal by another job is a 200 stream with one
  error frame ({type, status, message}), which both pages already render;
  a second batch keeps its exact JSON 409.
- Gallery cache clear: a refused server clear shows the reason and stops
  before the success toast and the "Not generated" reset.
- Admin settings save and maintenance show the reply's error ("config.json
  cannot be read...", "Scan for metadata already running") instead of
  "Failed to save settings" or "HTTP 409: Conflict".
- Tests cover the read-only /maintenance branch: a read-only selection runs
  during a scan as a read hold; a mixed or default selection is refused.

* refactor(api): move the settings routes out of admin.py

GET and POST /prompt_manager/settings, their validation, the config commit
and runtime publication, and the shared commit-failure reply now live in
py/api/settings_routes.py as SettingsRoutesMixin, before S3 adds host
fields there. No behaviour change: the same routes, payloads and status
codes; admin.py drops about 300 lines.

* chore(coverage): raise the floors of the frontend files this round touched

admin.js, gallery.js, activity.mjs and activity-job-view.mjs, to their
measured coverage. No floor is lowered.

* fix(admin): the gallery starts no thumbnail status read while it leaves

The gallery's startup thumbnail check runs one second after load. When that
second ended while the page was leaving (CI, WebKit: the workflow test
navigates from the gallery to the admin page), WebKit failed the new read
with "Fetch API cannot load ... due to access control checks." and
Playwright reported a page error.

Like the admin page (#290), the gallery now has a page lifetime that ends
on beforeunload and pagehide (and starts again on a back/forward restore).
The check starts no read once it has ended and passes it as the read's
signal. A page-clock workflow test that leaves before the second ends
failed on both events, in Chromium and WebKit, before this change.
2026-10-22 21:39:01 -07:00
Vito 5123654b0d fix(admin): start no list or stats read while the admin page leaves (#290)
On WebKit, a reload cancels the page's reads between beforeunload and
pagehide. After an in-place edit, finishEdit awaits loadStatistics, which
swallows its cancelled read, and then calls refreshList. That /recent read
starts while the page is leaving, and WebKit fails it with "Fetch API cannot
load ... due to access control checks." Playwright reports that console
error as a page error. This failed workflows.spec on WebKit in CI.

It is not an unhandled rejection: the page catches the failure. The read
should never start.

The admin page now has a lifetime (AbortController) that ends on
beforeunload, unless a running scan asks to stay, and on pagehide.
- ApiClient.latestOnly({lifetime}) starts no request once the lifetime has
  ended, aborts the one in flight, and resolves both to STALE.
- loadStatistics skips and silences a read that ended with the page.
- Real errors still propagate and show as before.
- A page restored from the back/forward cache, or one whose navigation never
  left, gets a new lifetime on pageshow or its next pointer or key input.

Tests:
- The test server can hold a real request at the server (request_hold.py),
  because a page.route hold is cancelled differently by WebKit.
- A WebKit regression test holds /prompt_manager/stats after an in-place
  edit and reloads. It failed 5/5 before the fix with the CI message and
  passes 5/5 after it.
- A test checks that a called-off leave reads the list again.
- api-client unit tests cover the lifetime.
2026-10-22 19:49:14 -07:00
Vito a8aff3a39d feat(annotations): bulk Add tags and Auto-tag selected as Activity jobs (#283)
* feat(activity): image tagging jobs for bulk Add tags and Auto-tag

POST /prompt_manager/jobs/image-tags starts one record-changing
"image-tags" job over 1..500 listed images (202, or the same 409 busy
envelope as /jobs/scan). Add tags hashes each current file and unions a
fixed set into that byte record; Auto-tag captures each still image, asks
one legacy engine for its top N suggestions on the capture (count from
the options or autotag.defaults.suggestion_limit), re-hashes the file and
unions only onto the same bytes. Outcomes are grouped as tagged,
unchanged, skipped (video, too large, file changed, file not found, not a
still image) and errors; cancel stops at a file boundary.

Auto-tag holds the shared model gate for the whole job (409 model-busy at
start when it is held), loads the model once and never downloads one: an
unavailable model fails the job before the first file. The registry gains
blocker() so a start answers busy before taking the gate.

* feat(annotations): Add tags and Auto-tag selected in the selection bar

The Images selection bar gains the concept's Add tags and, beside it,
Auto-tag selected (D85). Add tags starts the image-tags Activity job with
the dialog's comma-separated names; Auto-tag selected reuses the Tag image
model panel (server defaults, Suggestions count) without Generate, refuses
a model that is not downloaded, and either starts the job (Apply
automatically) or walks the selection one Tag image dialog at a time
(Review each image, D90). Busy, model-busy and refused starts toast why;
the selection stays; a finished job's files are read again.

* feat(activity): show image tagging jobs with their own outcomes

Image tagging jobs get their own Activity view (D89): the pill reads
"Tagging", the item and heading line name the images and end with
"N tagged · M errors", and the job view shows Tagged, Unchanged, Skipped
and Errors tiles and outcomes, opens Tagged and Unchanged rows in the
viewer, and reads "Tagging log" and "Cancel tagging". The per-kind table
lives in activity-store.mjs; the scan's view is unchanged.

The selection bar's copy without a concept line is recorded as D117.

* fix(annotations): final review round for bulk image tagging

- Rename the D117 copy deviation to D118 (D117 belongs to another PR).
- The tagging job's progress line gives the scan's "about X min left"
  estimate, with each finished file as one batch.
- The start body cap fits 500 targets of the longest legal path, even
  escaped; one byte over answers 413.
- A capture store at capacity is retried with a short backoff, then the
  file is skipped with a rerun sentence instead of a Tag-dialog error.
- Output roots and their labels are resolved once per job in the
  executor, not twice per file on the event loop.
- Review each image holds Stop reviewing, Escape and x while a save is in
  flight, and its primary reads "Save and next" when only the manual
  field would be saved.
- An Add tags field that parses to no names keeps the dialog and its
  text and reports the message on the field.
- Image tagging finish toasts carry the counts ("... finished · N tagged
  · M errors").
- Specs: the bulk spec waits for the answered Start before checking that
  the invalid press sent nothing; the Activity spec uses a real error row.
2026-10-22 18:55:24 -07:00
Vito 9dcad31441 chore(coverage): raise the activity-scan floor to its measured coverage (#285)
The floor sat at 64% branches while every run measures 75%, so each
branch's coverage:update raised it and had to revert the change by hand.
2026-10-22 16:26:02 -07:00
119 changed files with 17350 additions and 791 deletions
+420 -24
View File
@@ -50,6 +50,13 @@ stdin open and the server (`--exit-on-stdin-eof`) shuts down cleanly, removing i
temp root, when that pipe closes, so a worker killed without teardown leaves no
server behind. A test that fails gets the stderr its server wrote during the test
attached to the report as `server-stderr` (an aiohttp traceback behind a 500).
A `/__test__/.../reset` still running after 2 s writes a report to the job log,
passing test or not: every thread's stack taken while it waits, the default
executor's threads and queue, the open connections and WAL size, then the total
and how long each executor call queued and ran. Its lines start with
`pm-slow-reset| ` (`tests/browser/slow_reset_watch.py`). The report never reads a
connection another thread is using: one that is busy inside SQLite would block
that read with the GIL held and freeze the whole server.
The coverage run can be split the way CI splits it, and each shard keeps the raw
V8 files it writes under `coverage/js/raw`:
@@ -398,6 +405,13 @@ naming it, here and in a detailer's image ancestry; a latent that is not a link
reaches its own sampler is `malformed`; running out of budget is `work-limit` or
`depth-limit`, never single stage. Each sampler's walk runs once per resolution;
selector branch runs share it and other candidates are charged its visit cost.
A UI workflow node whose `type` is a list or an object (read only for the overwritten
PromptManager check) is skipped like a non-object node, since it can be neither a
subgraph instance nor a PromptManager node, so a prompt that resolves still resolves
and the other nodes still feed the check; a list or object `class_type` is never a
preview output. Before #286 both raised and the inspect route answered 500
(`test_unhashable_*` in `tests/test_api_prompt_inspector.py`, the `workflow-type-*`
golden cases on both sides).
Text encoders: `single` encoders (the Qwen image encoders, `TextEncodeQwenImage21`
slot 0 `prompt` and slot 1 `negative_prompt`, `KikoTextEncode`) read one string
input unchanged. `parts-equal` encoders (CLIPTextEncodeSDXL, Flux, SD3) resolve when
@@ -861,9 +875,10 @@ Recorded deviations on this page (spec §8):
- **D7**: the star is a real persisted favorite with "Added to favorites." /
"Removed from favorites." and a rollback toast on failure.
- **D8**: every Images card has the concept's select checkbox before the star; the
`#bulkbar` shows "N selected" and only the real actions in the concept's order (Set
category, Export metadata, Clear). Add tags and Add to collection wait for image tags
and collections; "Remove links…" is dropped. The selection survives Load more and
`#bulkbar` shows "N selected" and only the real actions in the concept's order (Add
tags, then D85's Auto-tag selected, Set category, Export metadata, Clear). Add to
collection waits for collections; "Remove links…" is dropped. (Amended by #249 PR D:
Add tags ships, see "Bulk tagging from the selection bar".) The selection survives Load more and
bounded re-mounting and is cleared whenever the collection is replaced (search,
category, sort, Filters, source change, rescan, reload). A return to Images restores
the saved collection unreplaced and keeps its selection; a full page reload starts
@@ -1201,6 +1216,142 @@ production-versus-concept difference of PR C and its record:
| Retry leads the actions when the controller allows it | D116 (ruling T5) |
| States the concept lacks (busy, conflict, changed storage or file, unconfirmed save, unavailable model, video) | Global Constraints: `#tag-generation-status` text only |
### Bulk tagging from the selection bar (#249 PR D)
The Images selection bar shows the concept's **Add tags** after the count, then
**Auto-tag selected** (D85), before Set category. Both take the checked files when
pressed, in listing order, as their reveal identity (`{root_index, root_scope, path}`);
more than 500 toasts "Select at most 500 images." and opens nothing. The selection
stays after every start, refusal and review. The job's start body is bounded so 500
targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) fit:
`500 × (1024 × 6 + 256) + 64 KiB`; a larger body answers 413
(`tests/test_image_tag_jobs.py` "test_start_body_bounds").
- **Add tags** opens the concept's `simpleInput` (app.js a1378–1383) as Set category
does: "Tag selected images", "Tags, separated by commas" (empty: the concept's
"cinematic, concept" is sample text), Cancel / Apply. The names are parsed as the Tag
image dialog's manual field parses them (trimmed, case-insensitive repeats dropped,
first spelling wins); a field that parses to no names keeps the dialog open with the
typed text and reports "Enter at least one tag." on the field (its native validity,
cleared on the next edit), and nothing is sent. Apply starts the
`image-tags` job (`POST /prompt_manager/jobs/image-tags`, `{mode: "add", targets,
tags}`) and toasts "Tagging started."; Activity shows its progress.
- A refused start toasts why and changes nothing: another record-changing job (a scan)
gives "Another job is running: <title>." with a link to it (D57's toast), a held model
gives the server's `model-busy` sentence, any other coded refusal its own sentence,
and anything else "Tagging could not start. Try again." / "Auto-tagging could not
start. Try again.".
- When a job this page started ends having tagged files, those files are read again
once: the selected file's inspection (its chips, D88) and, under a Tags filter or a
search, the loaded range, as after a Tag save.
- **D85** (plan D62, ruling L1, beyond the concept): **Auto-tag selected** opens
"Auto-tag selected images" in the Tag image dialog's shell (`tagging-modal`) with
its model panel, headed "Auto-tag <N> image(s)" (the panel's "Auto-tag this image"
for N files), without the preview, the manual field, the suggestions or Generate.
Its fields show the server's defaults (`GET /prompt_manager/autotag/models`, the
D91 Suggestions count included) once that read lands; Start waits for it. A model
the status lists as not downloaded keeps Start `aria-disabled` and
`#tag-generation-status` says "<model> is not downloaded. Download it from the
legacy Auto-tag page." (the server's own sentence), so no job starts only to fail.
Start validates the fields as Generate does (native validity; nothing is sent).
The concept comparison (`images-concept.spec.mjs`) uses the concept page itself as
the oracle, with Add to collection and Remove links… removed and a copy of its Add
tags button reading "Auto-tag selected" inserted after it.
- **D90** (plan D67, ruling T3-bulk): the dialog's **How to apply** radio group
(`.field[role=radiogroup]`, radios beside their labels): **Apply automatically**
(checked; primary "Start auto-tagging") starts the `image-tags` job
(`{mode: "auto", targets, model_type, options}`) and toasts "Auto-tagging started.";
**Review each image** (primary "Start review") walks the selection in the browser
(`images-tagging-review.mjs`). Each image opens the Tag image dialog titled "Tag
image · Image <i> of <M>" on the chosen model and options with Generate already
running; its actions are Stop reviewing, Skip and Approve and next (Approve and
finish on the last). Approve saves the manual field plus the remaining chips as Save
does, without its per-image toast, and opens the next image; Skip writes nothing and
advances; Stop reviewing, Escape and × end the session. While Approve's save is in
flight, Skip and Stop reviewing are `aria-disabled` and Escape and × do nothing, so the
closing count includes that save. When no suggestions are on screen (a model change
cleared them, or Generate was refused) the primary saves the manual field only and
reads "Save and next" ("Save and finish" on the last image). Each image is one gated
`suggest` call, so the session holds no model between images; a `model-busy` answer
stays on that image with its sentence in `#tag-generation-status` and D116's Retry
until the user retries or skips. Ending toasts "<n> of <M> images tagged." and
returns focus to Auto-tag selected.
- **D91** extends to the bulk dialog: its Suggestions field is the job's and the
review's `suggestion_limit`.
- **D118** (copy the concept has no line for; the concept's `bulk-apply` only toasts a
demo sentence): an Add tags field that parses to no names reports "Enter at least one
tag." on the field; a start refused for no coded reason toasts "Tagging could not start. Try
again." or "Auto-tagging could not start. Try again." (as D55's "The scan could not
start. Try again."); and the bulk dialog's model panel is headed "Auto-tag <N>
image(s)" where the Tag image dialog's reads "Auto-tag this image". Everything else
the selection bar's tagging shows is the concept's or the plan's (D85, D90, D91).
- **D89** (plan D66, Activity for the `image-tags` job): the job's Activity item and its
view are the scan's (D54, D56, D57, D59) with the job kind's own counters, from one
per-kind table (`JOB_VIEWS` in `activity-store.mjs`; a kind without one keeps the
scan's tiles under "Job log" and "Cancel job"). The titles are the server's: "Add
tags to images" and "Auto-tag images".
- The pill's verb is "Tagging" ("Tagging · 41% · 8,240 / 19,622").
- The item line and the view's heading line read "Started HH:MM · N images · closing
this view does not cancel it" (the job's image count, grouped "en-US"), "Finished
HH:MM · N tagged · M errors", "Cancelled HH:MM at a file boundary · N tagged ·
completed work is kept" or "Stopped on an error at HH:MM · N tagged · M errors".
- The view's actions are Download report and, while running, "Cancel tagging"
(`.danger`); the progress bar is "<title> progress".
- Four `dl.stat-row` tiles in the mockup's tones: Tagged (`.added`, green),
Unchanged, Skipped (`.warn`, warning), Errors (`.bad`, red). They keep the
mockup's five-column row (two columns on a phone), so a tile is the scan's width
and the fifth cell stays empty.
- The outcome list reads Tagged, Errors, Skipped, Unchanged (what got tagged and what
went wrong first, as the scan's), and the view opens on the first with files.
- Rows under Tagged and Unchanged are D57's `button.file-row`s that open the file in
the viewer by its listing identity; Skipped and Errors rows are plain text.
- The log is "● Tagging log — raw detail, click to show".
- The row and log wording is the job's (D1), since neither the concept nor the
mockup has a line for it: "Added N tag(s)", "No new tags", and the skip reasons
"Video: manual tags only", "Too large to verify", "File changed", "File not found",
"Not a supported still image" and "The file could not be read now. Run the job again
for it." (the capture store stayed at capacity, for example with Tag dialogs open in
other tabs, after a short backoff of retries). Log prefixes are `+` tagged, `=`
unchanged, `-` skipped and `!` error.
- The progress line is the scan's: "Processing file N of M · about X min left", the
estimate (D56's `_time_left`) shown once two files are done, each file one batch.
- Download report saves `prompt-manager-image-tags-<YYYYMMDD-HHMM>.json`.
- The start toasts are "Tagging started." / "Auto-tagging started." (D85); the end
toasts follow the design's pattern with the job's counts, as the scan's: "<title>
finished · N tagged · M errors", "<title> cancelled · N tagged" or "<title> failed ·
N tagged · M errors".
- Tests: `tests/js/workspace-activity-store.test.js` ("image-tags jobs get their own
tiles outcomes and pill verb", "unknown kinds keep the generic view");
`tests/browser/workspace-activity.spec.mjs` ("an image tagging job shows Tagged
Unchanged Skipped Errors and opens tagged rows", "cancel tagging stops at a file
boundary", "the report download names the kind", on fake `image-tags` jobs from
`workspace_activity_fixture.py`); `tests/browser/activity-concept.spec.mjs` ("image
tagging job view follows the approved mockup", dark and light, with a mutation
check). The scan's view is unchanged.
Browser tests (`tests/browser/workspace-tagging-bulk.spec.mjs`, on the tagging
fixture): "selection bar shows Add tags then Auto-tag selected before Set category",
"Add tags runs a job and the cards show the tags after it finishes", "Auto-tag
selected with Apply automatically runs a job with the chosen model, thresholds and
suggestion count", "a model that is not downloaded keeps Start disabled and says
why", "Suggestions field prefills from the setting and caps chips at the entered
count", "Review each image walks the selection one dialog at a time and only approved
images gain tags", "Review holds Stop reviewing, Escape and × while a save is in
flight, and names a manual-only save" and "a running scan makes the start report
busy". Units:
`tests/js/workspace-images-tagging-bulk.test.js`,
`tests/js/workspace-images-tagging-review.test.js`.
| Difference | Record |
| --- | --- |
| Add tags ships (Add to collection and Remove links… still omitted) | D8 (amended) |
| Auto-tag selected and its dialog | D85 (ruling L1) |
| How to apply and the Review each image session | D90 (ruling T3-bulk) |
| Suggestions count in the bulk dialog | D91 |
| Copy without a concept line: no tags, start fallbacks, the panel heading | D118 |
| Activity item and job view for image tagging jobs | D89 (plan D66) |
Native image tagging (#249 PR B routes, PR C UI): `POST
/prompt_manager/image-annotations/observe` (capture one listed file, its saved set and
`tag_revision`), `GET`/`DELETE /prompt_manager/image-annotations/capture/<token>`
@@ -1444,7 +1595,43 @@ AVI/WMV preview only), chosen or dropped, previews inline. The heading's one act
the shared viewer with the full inspector on a record read entirely in this browser
(`web/js/workspace/inspect.mjs`, `local-session.mjs`, `local-inspection.mjs` and the L1
readers `png-metadata.js`, `local-video.mjs`, `local-projection.mjs`). Nothing is
uploaded, imported or stored. Only HTML and JS change, so no ComfyUI restart is needed.
uploaded, imported or stored. PRs L1 and L2 change only HTML and JS, so they need no
ComfyUI restart; PR L3 does (below).
Local roles (#249 PR L3, decision LI-1): the prompts and resolution of a local record are
the server inspector's for the same records. `web/js/workspace/local-resolver.mjs` (with
`local-resolver-support.mjs`, `local-resolver-stages.mjs`, `local-parameters.mjs` and
`py-json.mjs`) ports `inspect_prompt_evidence` with `INSPECTION_PROFILE`: execution
evidence, graph tracing, every bound, several sampler candidates with the
identical-branch rule (#270), preview-only stages, detailer stages (multistage) and A1111
`parameters`. It reads the server's own node table, `web/js/inspection-semantics.json`,
which the page fetches once through the API client as it mounts, before any selection
(`local-semantics.mjs`; `/prompt_manager/js/inspection-semantics.json` is the one JSON
file the static route serves, by exact path). A table that cannot be fetched answers
"The inspector's node table is unavailable." as the server does. The legacy fill in
`projectInspectionPayload` then applies to a local record exactly where it applies to the
Images answer for the same bytes. Python is the oracle:
`tests/test_local_inspect_parity.py` `LocalResolverParityTests` compares the local
evidence byte for byte with `inspect_prompt_evidence` for every parity PNG and inspection
vector (prompts, resolution and facts), the golden and selection record sets
(`tests/local_resolver_cases.py`, `tests/local_resolver_selection_cases.py`), every work
bound and 1,200 seeded fuzz cases; `tests/fixtures/local-inspect/resolver-vectors.json`
holds the committed answers the Node unit test reads. The browser spec `chooser shows the
Images roles and resolution for every parity PNG` compares the rail with Images for all
25 committed parity PNGs. The static route change and the resolver fixes (Python) need a
ComfyUI restart.
Main thread (M4): the resolver runs on the page's thread, not a worker. On a desktop
(Node 26) `resolveLocalEvidence` on maximal 2 MiB graphs (a 19,990-sampler chain, 6,600
independent samplers, a consumer-dense scan, a 100,000-node PromptManager workflow) took
130–455 ms; the server took 80–905 ms on the same inputs. `localFileAdapter.read` runs in
bounded steps (read, table, parse and resolve, project, fit) with a turn of the event
loop between them, every wait ends with the selection's signal (a stalled table request
included), and the inspector carries `aria-busy="true"` with "Loading file details…"
until the answer lands. `tests/js/workspace-local-inspection.test.js` pins a 2,500 ms
budget per step on maximal independent-sampler and stage-chain graphs, checks that the
stepped read answers byte for byte as one synchronous build, and that an abort at any
step ends the read.
Bounds (the server's own budgets, ported by PR L1; see "Follow-up: local Inspect parity"
under "Chronological Library history" for the known divergences):
@@ -1463,7 +1650,8 @@ under "Chronological Library history" for the known divergences):
Privacy, asserted by `local inspection makes no content request or storage write` and
`local roles send exactly` in `tests/browser/workspace-inspect.spec.mjs`, and by the
managed-host spec below: no request carries the file's bytes, name or text (fetch, XHR,
managed-host spec below: the only static file Inspect reads is its node table, once per
page and before any selection; no request carries the file's bytes, name or text (fetch, XHR,
beacon, WebSocket, EventSource); Inspect writes nothing to `localStorage`,
`sessionStorage` or IndexedDB; only an explicit Send crosses the host bridge, and its
payload carries neither the `blob:` URL nor the filename; no console message names the
@@ -1490,19 +1678,40 @@ mapped differences"):
`image/png,image/jpeg,image/webp` plus `video/*,.mp4,.mov,.m4v,.webm,.mkv,.avi,.wmv`. A
video previews in the same `#local-preview` slot as `video.local-file-preview` with
native `controls`, `playsInline`, `preload="metadata"` and `muted`, never autoplays
inline, and is paused, has its `src` removed and is removed from the page on replacement
or dispose, before its URL is revoked. Each preview owns its listeners, so the page never
pins a released node. A settled video (metadata read, or failed) is then reset with
`load()` at once; one still starting is reset only once it reports metadata or an error,
because WebKit (GStreamer) blocked the page when such a pipeline was reset (`replacing a
video that just started loading never blocks the page`, `a video replaced while still
starting is reset once it settles`). A WebKit load cut off by the revoked URL usually
stalls and never settles; a fallback then only drops the waiting listeners. The viewer's
own video teardown was checked the same way (`closing the viewer on a video that just
started never blocks the page`, 15 fresh pages on WebKit) and needed no change.
Follow-up: on WebKit, a video released exactly at its `loadstart` sometimes blocks the
page, with or without any reset and also before the deferred reset (2 of 24 runs in the
pinned image), so the `loadstart` spec is skipped on WebKit until that is investigated.
inline. On replacement or dispose it is paused, has its `src` removed, is reset with
`load()` (which frees its decoder) and is removed from the page, before its URL is
revoked. Each preview owns its listeners, so the page never pins a released node.
A video replaced while still starting (a source, but neither metadata nor an error yet)
stays where it is, inert and `aria-hidden`, with the new preview inserted after it, and
the session's revoke of its URL waits; once it reports metadata or an error (or after
15 s) it is released as above and only then is its URL revoked. For that moment (the
time to metadata, or at most 15 s for a video that never settles) the old video and the
new preview both show, the new one below it: a known, brief overlap. Stacking them in one
slot instead hung again, because it changes the video's own box: the old video
absolutely positioned in its own box hung 5 of 120 pages (8 of 120 with `opacity: 0`),
and a one-cell grid 4 of 120. Only `opacity: 0` on the video, left in the flow, did not
hang (0 of 120), and alone it would leave an empty hole above the new preview. A route
change within the workspace waits the same way: the shell renders the next route once a
video still starting on Inspect settles (`leaveGuard`; the latest route wins). Route
changes at `loadstart` + 0–7 ms hung 4–5 of 120 pages before and 0 of 120 after.
A real page leave cannot wait. With no page code releasing anything, navigating
away hung 8 of 120 pages and reload 22 of 120 (3 and 8 of 120 with the page's own
release), so that WebKit teardown stays a known limitation; dispose still releases at
once. Cause, found with native stacks of the hung WPE web process in the
pinned image: WebKit (GStreamer) deadlocks when a video leaves the page or the viewport
before its pipeline has started. The main thread changes the pipeline state and waits
for the source's streaming thread, which is itself waiting on the main thread in a
synchronous bus handler. Removing the video, hiding it, or only pushing it below the
fold with a taller preview inserted before it all hung (7 to 21 of 120 fresh pages each,
replaced at `loadstart` + 0–7 ms), and none of the earlier steps (`pause()`, removing
`src`, `load()`, revoking the URL) was the cause on its own; pushing it out after
`loadedmetadata` hung 0 of 180. Specs: `replacing a video that just started loading
never blocks the page`, and `a video replaced while still starting is reset once it
settles`, which now runs on WebKit too (15 fresh pages, replaced at `loadstart` + 0–4
ms), and `leaving for another route while a video starts waits for it to settle`. The
viewer's own video teardown was checked the same way (`closing the viewer on a video
that just started never blocks the page`, 15 fresh pages on WebKit) and needed no
change.
- **D63** (status line): one `p.small.spaced-16` in `#local-preview`, for states the
concept never shows: "Drop one file at a time.", "Folders cannot be inspected. Choose
one image or video file.", "This file type cannot be previewed or inspected here.
@@ -1527,6 +1736,29 @@ mapped differences"):
Modified with no Path row and no Show in folder, and ends with "Local file; no saved
library association was checked." and Download original. There is no Prompt history
action and no category or tag chips.
- **D117** (Copy settings summary, LI-4): the legacy metadata
page's COPY ALL, consolidated into the shared full inspector (the viewer's
`#viewer-details`, so Images, the recorded history's current file and Inspect all have
it). The concept has no such action. One plain button "Copy settings summary" follows
the six facts (before "File & provenance"); the Images side panel keeps the concept's
footer and has none. Its text is the legacy `copyAllData` template, line for line:
`Checkpoint`, `Positive Prompt`, `Negative Prompt`, `Steps`, `CFG Scale`, `Sampler`,
`Seed`, with the legacy placeholders ("Unknown", "No prompt found", "No negative prompt
found"). The values are the ones the inspector shows: a confirmed prompt and a known
fact verbatim, anything else (an explicitly empty prompt included, as the legacy parse
treats '') the placeholder. Against the legacy page, for the same file, only these
differ: numbers keep the server's spelling ("8.0" where the legacy page wrote 8); a fact
the legacy parse could not read can be known; and a negative the #249 ZeroOut guard
zeroed reads "No negative prompt found" where the legacy parse read the positive's text
through the zero-out. `workspace-file-inspector.test.js` runs the template taken from
`web/metadata.html` itself on the legacy parse of every parity PNG: 22 of 25 are
byte-equal, f01b and f23 differ by the ZeroOut guard and f17 by facts the legacy parse
missed. Copying works as the concept's prompt copy: the toast "Settings summary
copied.", or, without a clipboard or when it is refused, the select-to-copy dialog
titled "Copy settings summary". `.ca-tools #viewer-details > div > button` wraps
(`white-space: normal`) in the narrow rail at 200% zoom, as D20's buttons do. The concept
comparisons add the button to the concept's viewer at runtime (a copy of its own plain
button) and to the Inspect rail's section order.
The chooser keeps naming the chosen file, as the concept's does: it is cleared as the
picker opens (so choosing the same file again is a new selection), not after `change`,
@@ -1540,9 +1772,8 @@ Legacy floor (`web/metadata.html`), dispositions:
- Full text (the old "Show more"): the prompt blocks show the whole text.
- Raw workflow view and Download workflow JSON: kept, in "Workflow & raw metadata".
- Copy all summary and filename copy (LI-4): the filename is selectable text in the rail
and Download original saves the file under its name. The ruled "Copy settings summary"
button after the six facts of the shared full inspector is not part of PR L2; until it
lands, the legacy page keeps COPY ALL and stays reachable from Overview.
and Download original saves the file under its name. COPY ALL is the shared full
inspector's "Copy settings summary" (D117), with the legacy text.
- Fully browser-local reading: kept and asserted (privacy above).
- Case-variant keys `Workflow|comfy|ComfyUI|Prompt|Parameters` (LI-5): a separate
backend issue; Inspect follows the server's readers, so the two never disagree.
@@ -1585,6 +1816,142 @@ npm run --prefix tools/vendor test:workflows -- inspect-concept.spec.mjs workspa
npm run --prefix tools/vendor test:comfyui -- inspect-sender.spec.mjs
```
### Native System (#249 System)
`#/system/<section>` is a native workspace route rendered by `web/js/workspace/system.mjs`
on the concept's `renderSystem` markup (app.js a698–706; `concept.css` 709–895). `#/system`
is Overview under the sidebar's own hash; `#/system/overview|appearance|sources|
integrations|tagging|maintenance|backups|diagnostics` select a section, and an unknown
child (the concept's `#/system/general`) is replaced by `#/system/overview`. The
production hub `#/overview` is unchanged (D28); its System card opens
`#/system/overview` (Q12). Switching sections keeps the shell, the drafts and focus on
the pressed section button and disposes the previous section's scope, so its reads,
listeners and dialogs never outlive it.
Drafts (`system-state.mjs`) live for the page session (Q10). Save writes the dirty
browser fields, then the host fields (`POST /prompt_manager/settings` with
`expected_revision`, #220), then the runtime fields. A failed domain keeps its drafts and
highlights their controls; `settings-changed` is a conflict and keeps the draft. System
reads the host values on mount and again after a conflict; until a read returned a
revision, a host save fails without a request, so it never sends a blind
`expected_revision: null` (which S1 merges into the file on disk). No delivered section
edits a setting yet, so Save stays disabled until Appearance (S3).
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
visit through `latestOnly()` on the section scope; a late answer after the section left
paints nothing. Until it answers, the section shows "Loading the overview…"; a failure
shows "The overview could not be loaded. Your saved data has not changed." and Retry,
never zeros. The thumbnail row is a separate read of `/images/thumbnails/status` (a
walk of every source), so the other values never wait for it: "Checking thumbnails…"
until it answers. The health check (`POST /prompt_manager/system/health`) is a `read`
registry job of kind `health`: one at a time, beside a scan, not cancellable and with
no file outcomes. The page tracks the started job in the Activity store (its event
stream; the store marks it lost if the server stops listing it), never by polling. A press while one runs (another tab) waits
for that one. The last report lives for this server session (`GET
/prompt_manager/system/health`) and describes the database it was taken from: a
restore that replaces the database forgets it, and a check that started before the
swap is not kept. The check's own connection opens a `mode=ro` URI that keeps a Windows
UNC path (`file:////server/share/prompts.db`; pathlib's `file://server/share/...` is a
remote authority SQLite refuses). A check that raises fails only itself, and the
missing-files check reads its rows inside the snapshot but stats the files after it
ended, so a slow source never pins the WAL. Statistics
read `GET /prompt_manager/system/statistics` (`by_category`, the legacy `/stats` totals
and `rated_images`).
Read-only standard for a check or preview (S2's health check; S6 and S8 previews use
the same): the database and `-wal` bytes are unchanged, the `-shm` size is unchanged,
the connection's `total_changes` is 0 at close and the data reads back unchanged. The
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
reader can keep it byte-identical (`test_health_check_never_writes`).
Recorded deviations (plan D60–D64 renumbered D92–D96; D121, D122 and D123 assigned):
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
configuration." / "Applied until the server restarts.", "Some changes were not saved.
Review the highlighted settings.", "Saved for this tab only; browser storage is
unavailable." and "Settings changed elsewhere. Reload this section to see them."
"Save preferences" is `aria-disabled` (opacity .55, `not-allowed`) while clean or
saving; a setting the last save could not save gets a red outline. CSS under D92.
- **D93** (Overview): real values. The metrics are the Library's eligible positives
and active saved negatives and the indexed `generated_images` rows (Q2). "Database
integrity" is "Not checked" until a check ran this session, then "N issues" (checks
that found issues, even beside a check that could not run), "Passed" or "Incomplete".
"Ambiguous image links" reads "Not available yet." with its Review metadata action
`aria-disabled` (CSS under D93) until the Needs review queue reports a count (Q6).
"Backup status" reads "No retained backup record." until verified backups exist (S8).
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
missing thumbnails", "N missing thumbnails" (the pending count of
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid
and density settings are shared across views." (no paging). The sources row is
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
by "M of N configured sources available." or "None configured is available; ComfyUI's
output folder is used." when that is the case.
- **D94** (all sections): a section not delivered yet shows the hand-off panel "Your
current tools are ready" with a link to each legacy tool that still owns it (the
plan's legacy coverage table); retired section by section, gone after S12.
- **D95** (health report): the concept's `health-check` dialog titled "Health report",
with no sample note. It lists every check in the server's order: Integrity check,
Foreign keys & pair references, Canonical hashes, Pair uniqueness (added), Missing
files, Ambiguous metadata, Consistency (added). Missing files has one row per image
source when there are several ("Missing files · output"). Each chip is the real
result: "Passed", a count with what was found ("2 broken references", "1 link needs
review", the concept's "N links need review" and "N images need review"; "100+
problems found" when SQLite's integrity check stopped at its limit), "Not
available" (an unavailable source), "Not available yet" (Ambiguous metadata, Q6) or
"Could not check". A `p.small` after the list says when it was checked and whether
every check passed, some could not run, or issues need review. Close / Review missing
links (to Maintenance, its hand-off until S7). Nothing is changed by a check.
- **D96** (Statistics): "Usage statistics" without the sample note: the concept's
"Runs by category" table (Category / Runs / Avg. rating) with the real runs and
average rating of eligible parents per category ("Uncategorized", "Not rated"),
then a second table "Library totals" (Total / Count: Prompts, Tags, Images from the
legacy statistics, and Rated images: indexed images whose linked prompt is rated).
The Overview row says "Runs and ratings by category, with library totals." because
that is what the dialog shows (the concept's own dialog shows only "Runs by
category"); per-prompt run counts, prompt tags and dated runs are recorded, but no
table of them exists yet. A failed read toasts "Statistics could not
be loaded. Your saved data has not changed."
- **D121** (production hub): the Overview hub's System card reads the concept's System
heading and "Open System"; the concept has no hub card.
- **D122** (shell): the section nav's `aria-label` is "System sections"; the concept's
literal label is its prototype action name "system-tab sections".
- **D123** (Maintenance): the concept's Maintenance notice
("Check first. Review changes. Then apply." + Run health checks) is delivered ahead of
the rest of Maintenance, above its D94 hand-off, as the native place that runs the
health check. A press shows "Checking…" (`aria-disabled`) and opens the report; a
failed check toasts "The health check could not finish. Nothing was changed."
`tests/browser/system-concept.spec.mjs` compares the shell and Overview with the concept
(dark and light, 1600×1100 and 390×844), mapping only the deviations above; unmapped,
the difference is exactly the D92, D93, D96 and D122 paths. Each surface's `top` is
measured, which pins the concept's `.surface + .surface` 18px offset that concept.css
also applies inside `.two-col` (copied as is). `statistics and health dialogs match
the concept` maps D95 from the concept's own five rows (labels verbatim): it inserts
Pair uniqueness after Canonical hashes and Consistency last, splits Missing files per
source when there are several, replaces only the chips with the fixture's expected
results, and asserts the full unmapped path list; it maps D96 and compares the
Maintenance notice unchanged. `system comparison detects mutations` removes a row,
renames a metric label, swaps the `--line` token, recolours a chip, renames a check,
removes the totals table and renames a caption. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`) are test artifacts; there
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
the health check and metric 4, a failed check, the thumbnail row against the status
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
route, a load failure with Retry and late answers after leaving. Its hooks are
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
failures, an absent source, a deleted linked file, unthumbnailed files) on the
concept-shaped fixture, with delays through the shared request holds.
```sh
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library -v
node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
--project chromium --project touch --project firefox --project webkit
```
## Workspace data fields for the concept pages (#249)
These fields are additive. The legacy `/prompt_manager/images/output` listing (no
@@ -2135,12 +2502,41 @@ skip without ffmpeg). The remaining known divergences, checked against ffprobe 9
`unreadable`); ffprobe reads such files.
- An EBML DocType other than `matroska`/`webm` is an invalid container locally; ffmpeg
only warns and reads it.
- Envelope JSON deeper than 1,000 levels is refused locally as Python's default
recursion limit refuses it; the server's real limit depends on the interpreter and
call stack. A duplicate key in an object enclosing a nest deeper than the running
- Deep JSON (local roles, PR L3, and video envelopes): both browser CPython-JSON
parsers (`py-json.mjs`, `local-video.mjs`) share one cap, `JSON_MAX_DEPTH` = 1,000
nested arrays or objects (each counts, an empty one too); past it a record is invalid
JSON locally. The server's `json.loads` raises RecursionError at a limit that depends
on the Python, measured on Linux x86-64: 993 levels from a bare script on 3.10 and
3.11 (the 1,000 recursion limit minus the caller's frames, so about 990 inside the
server), 9,997 on 3.12 and 9,998 on 3.13 (the C recursion counter; lower on Windows
builds), and about 52,150 on 3.14 (stack-based, varies with the thread's stack).
The record kinds answer: a deep `prompt` graph (or `comfyassets_prompt_pairs`
envelope) is malformed; a deep `workflow` is ignored, so the overwritten PromptManager
check (R1) does not run and a prompt can resolve; a deep video `comment` or
`description` envelope makes the file unreadable on both sides (the server's
RecursionError reaches `read_inspection_file`). That gives two directions:
- 3.10/3.11, between the server's limit and 1,000: the server refuses and the local
page reads, so a deep prompt resolves locally while the server says malformed, and
a deep workflow can make R1 refuse locally where the server resolves.
- 3.12 and newer, from 1,001 up to the server's limit: the local page refuses and
the server reads, so a deep prompt is malformed locally while the server resolves,
and a deep workflow is ignored locally while the server can refuse it under R1.
Past both limits they agree. `test_local_json_depth_cap_at_its_boundary` checks the
local answer at 999, 1,000 and 1,001 for both kinds and that any server difference
is only one of the two directions above.
- Local roles (PR L3): `\d` in the A1111 settings footer is any Unicode decimal
digit on both sides, but the browser's Unicode version can be newer than the server
Python's, so a footer whose step count uses a digit added after Unicode 15 can differ.
`str.lower()` and `toLowerCase()` differ only for characters no node class name uses.
- A duplicate key in an object enclosing a nest deeper than the running
Python's json limit is `malformed` locally and `limited` on that Python.
- Decimal exponent limits assume CPython's C `_decimal`; on a `_pydecimal` interpreter
the server would read the over-limit exponents the local projection refuses.
- Follow-up (two CPython-JSON emulations, review M6): `local-video.mjs`
(`pythonJsonLoads`/`pythonFloatRepr`, numbers kept as lexemes) and `py-json.mjs`
(`pyJsonLoads`/`pyFloatRepr`, Python value kinds) both emulate `json.loads`; only the
depth cap is shared so far. A fix to one parser can miss the other, so the video
reader should move onto `py-json.mjs` in a later PR.
Preserve raw logs, failed traces, paired PNGs, geometry, runtime versions and
source hashes before later managed commands clear their output directories.
+354
View File
@@ -0,0 +1,354 @@
"""The read-only health check behind System › Overview and Maintenance.
``run_health_checks`` opens its own ``mode=ro`` connection, reads one
snapshot and closes the connection in ``finally``; it never writes. Each
check reports ``passed``, ``issues`` (with an exact ``count``; the
integrity check stops at ``INTEGRITY_LIMIT`` problems and then says
``capped``),
``unavailable`` (it could not be checked here, so ``count`` is None, or
``missing_files`` checked only some sources and found nothing) or ``failed``
(it raised; the other checks still run). ``missing_files`` also reports each
source and whether the check was ``partial``; it reads the image rows inside
the snapshot and checks their files after the snapshot ended, so a slow source
never holds the WAL. At most ``MAX_DETAIL`` detail strings are returned per
check, and none of them carries a server path.
"""
import datetime
import logging
import ntpath
import os
import sqlite3
from pathlib import PurePosixPath, PureWindowsPath
from urllib.parse import quote
try:
from ..utils.hashing import generate_prompt_hash
from .models import normalize_image_path
except ImportError:
from utils.hashing import generate_prompt_hash
from database.models import normalize_image_path
logger = logging.getLogger("prompt_manager.database.health")
MAX_DETAIL = 10 # detail strings returned per check
INTEGRITY_LIMIT = 100 # PRAGMA integrity_check(N): problems SQLite reports
BATCH_ROWS = 100 # image rows read per fetch
REVIEW_UNAVAILABLE = "Needs review is not available yet."
CHECK_FAILED = "The check could not finish."
# Check ids and the labels of the concept's health report, in its order;
# pair uniqueness and consistency are the added rows (D63, now D95).
CHECKS = (
("integrity", "Integrity check"),
("foreign_keys", "Foreign keys & pair references"),
("canonical_hashes", "Canonical hashes"),
("pair_uniqueness", "Pair uniqueness"),
("missing_files", "Missing files"),
("ambiguous_metadata", "Ambiguous metadata"),
("consistency", "Consistency"),
)
_DATABASE_CHECKS = (
"integrity",
"foreign_keys",
"canonical_hashes",
"pair_uniqueness",
"missing_files",
"consistency",
)
def iter_consistency_issues(conn):
"""The legacy consistency issues: tag and image rows of a missing prompt."""
for ref in conn.execute(
"SELECT pt.prompt_id FROM prompt_tags pt "
"LEFT JOIN prompts p ON pt.prompt_id = p.id WHERE p.id IS NULL"
):
yield f"prompt_tags entry references non-existent prompt {ref[0]}"
for ref in conn.execute(
"SELECT gi.id, gi.prompt_id FROM generated_images gi "
"LEFT JOIN prompts p ON gi.prompt_id = p.id WHERE p.id IS NULL"
):
yield f"Image {ref[0]} references non-existent prompt {ref[1]}"
def _result(count, detail, *, status=None):
return {
"status": status or ("issues" if count else "passed"),
"count": count,
"detail": detail[:MAX_DETAIL],
}
def _counted(lines):
"""(count, first MAX_DETAIL lines) of an iterable, consumed once."""
count, detail = 0, []
for line in lines:
count += 1
if len(detail) < MAX_DETAIL:
detail.append(line)
return _result(count, detail)
def readonly_uri(db_path, *, windows=None):
"""The ``mode=ro`` SQLite URI of *db_path*, on this platform or the given one.
pathlib spells a UNC path ``\\\\server\\share\\x.db`` as
``file://server/share/x.db``, a remote authority SQLite refuses to open.
With an empty authority (``file:////server/share/x.db``) SQLite hands
Windows the ``//server/share`` path, which it opens as the UNC path.
"""
if windows is None:
windows = os.name == "nt"
db_path = os.path.abspath(db_path)
if not windows:
return PurePosixPath(db_path).as_uri() + "?mode=ro"
path = PureWindowsPath(db_path)
if ntpath.splitdrive(str(path))[0].startswith("\\\\"):
return "file:" + quote("//" + path.as_posix()) + "?mode=ro"
return path.as_uri() + "?mode=ro"
def _integrity(conn, _roots):
rows = [
row[0] for row in conn.execute(f"PRAGMA integrity_check({INTEGRITY_LIMIT})")
]
if rows == ["ok"]:
return {**_result(0, []), "capped": False}
# SQLite stops listing at the limit: there may be more problems than that.
return {**_result(len(rows), rows), "capped": len(rows) >= INTEGRITY_LIMIT}
def _foreign_keys(conn, _roots):
return _counted(
f"{table} row {rowid} references a missing {parent} row"
for table, rowid, parent, _fkid in conn.execute("PRAGMA foreign_key_check")
)
def _prompt_hash(text):
return generate_prompt_hash(text) if isinstance(text, str) else None
def _canonical_hashes(conn, _roots):
conn.create_function("pm_prompt_hash", 1, _prompt_hash, deterministic=True)
return _counted(
f"Prompt {row[0]}: the stored hash does not match its text"
for row in conn.execute(
"SELECT id FROM prompts WHERE hash IS NOT pm_prompt_hash(text) ORDER BY id"
)
)
def _pair_uniqueness(conn, _roots):
groups = conn.execute(
"SELECT positive_prompt_id,negative_prompt_id,COUNT(*) FROM prompt_pairs "
"GROUP BY positive_prompt_id,negative_prompt_id HAVING COUNT(*)>1 "
"ORDER BY positive_prompt_id,negative_prompt_id"
).fetchall()
detail = [
f"Positive prompt {positive} with "
+ ("no negative" if negative is None else f"negative {negative}")
+ f": {rows} pair rows"
for positive, negative, rows in groups
]
# The surplus rows: what keeping one row per pair would leave out.
return _result(sum(rows - 1 for _p, _n, rows in groups), detail)
def _consistency(conn, _roots):
return _counted(iter_consistency_issues(conn))
def _readable(root):
return os.path.isdir(root) and os.access(root, os.R_OK | os.X_OK)
def _root_prefix(root):
return normalize_image_path(root).rstrip("\\/") + os.sep
def _source_name(root, index):
"""A source's folder name for the report (never its server path)."""
return os.path.basename(str(root).rstrip("\\/")) or f"Source {index + 1}"
def _owner(path, prefixes):
"""Index of the longest source prefix *path* lies under, or None."""
owner, longest = None, -1
for index, prefix in enumerate(prefixes):
if path.startswith(prefix) and len(prefix) > longest:
owner, longest = index, len(prefix)
return owner
def _image_owner(image_path, file_path, prefixes):
spelled = file_path or normalize_image_path(image_path)
owner = _owner(spelled, prefixes)
if owner is None:
owner = _owner(normalize_image_path(os.path.realpath(image_path)), prefixes)
return owner
def _unavailable_line(unavailable, *, partial):
noun = "source is" if unavailable == 1 else "sources are"
effect = "its files were" if unavailable == 1 else "their files were"
if not partial:
effect = "missing files were"
return f"{unavailable} image {noun} unavailable, so {effect} not counted."
def _missing_files(conn, roots):
"""Links whose file is gone, counted per source (only readable ones).
Each source reports ``checked`` with its own count, or ``unavailable``
(an unmounted drive) with no count: its links are never turned into a
count, while the readable sources still are. The check's ``count`` is the
readable sources' total (None when none is readable) and ``partial`` says
some sources were not checked. Links outside every source are named in the
detail, not counted.
Inside the snapshot this only reads the image rows; it returns the step
that checks their files, which runs after the snapshot ended.
"""
if not roots:
result = _result(None, ["No image source is available."], status="unavailable")
return {**result, "partial": False, "sources": []}
rows = []
cursor = conn.execute(
"SELECT id,image_path,filename,file_path FROM generated_images ORDER BY id"
)
while batch := cursor.fetchmany(BATCH_ROWS):
rows.extend(batch)
return lambda: _check_files(rows, roots)
def _check_files(rows, roots):
"""The ``missing_files`` result of the image *rows*, from the file system."""
readable = [_readable(root) for root in roots]
sources = [
{
"name": _source_name(root, index),
"status": "checked" if ok else "unavailable",
"count": 0 if ok else None,
}
for index, (root, ok) in enumerate(zip(roots, readable))
]
unavailable = readable.count(False)
if not any(readable):
result = _result(
None, [_unavailable_line(unavailable, partial=False)], status="unavailable"
)
return {**result, "partial": False, "sources": sources}
prefixes = [_root_prefix(root) for root in roots]
outside, missing = 0, []
for image_id, image_path, filename, file_path in rows:
owner = _image_owner(image_path, file_path, prefixes)
if owner is None:
outside += 1
elif readable[owner] and not os.path.exists(image_path):
sources[owner]["count"] += 1
if len(missing) < MAX_DETAIL:
name = filename or os.path.basename(image_path)
missing.append(f"Image {image_id}: {name} is missing")
count = sum(source["count"] or 0 for source in sources)
detail = [_unavailable_line(unavailable, partial=True)] if unavailable else []
detail += missing
if outside:
links = "link" if outside == 1 else "links"
verb = "was" if outside == 1 else "were"
detail.append(
f"{outside} {links} outside the image sources {verb} not checked."
)
status = None if count or not unavailable else "unavailable"
result = _result(count, detail, status=status)
return {**result, "partial": bool(unavailable), "sources": sources}
def _ambiguous(review_count):
if review_count is None:
return _result(None, [REVIEW_UNAVAILABLE], status="unavailable")
return _result(review_count, [])
def overall_status(checks) -> str:
"""``passed`` only when every check passed; any check not run is incomplete."""
statuses = {check["status"] for check in checks}
if statuses & {"unavailable", "failed"} or any(
check.get("partial") for check in checks
):
return "incomplete"
return "issues" if "issues" in statuses else "passed"
def _failed():
return _result(None, [CHECK_FAILED], status="failed")
def _run_check(check_id, step, *args):
"""One check's result, or None (reported as failed) when it raised."""
try:
return step(*args)
except Exception: # any error fails this check only; the others still run
logger.exception(f"Health check {check_id} failed")
return None
def _run_database_checks(db_path, roots):
"""Each database check's result (None when it raised), from one snapshot.
A check may return a step instead of its result: the step runs after the
snapshot ended and the connection closed (``missing_files``' file checks).
"""
results = dict.fromkeys(_DATABASE_CHECKS)
try:
conn = sqlite3.connect(readonly_uri(db_path), uri=True)
except sqlite3.Error:
logger.exception("Health check could not open the database")
return results
try:
conn.execute("BEGIN") # one read snapshot for every check
for check_id in _DATABASE_CHECKS:
runner = globals()["_" + check_id] # looked up per run (tests patch it)
results[check_id] = _run_check(check_id, runner, conn, roots)
except sqlite3.Error:
logger.exception("Health check could not read the database")
finally:
conn.close()
for check_id, result in results.items():
if callable(result):
results[check_id] = _run_check(check_id, result)
return results
class HealthOperations:
"""Mixin for PromptDatabase: the read-only System health check."""
def run_health_checks(self, roots, *, review_count=None) -> dict:
"""Run every check and return a ``HealthReport``.
Args:
roots: The image source directories to judge missing files under:
the effective sources plus any configured source that is not
available (reported as an unavailable source of ``missing_files``).
review_count: The Needs-review pending count, or None while no
review queue exists (``ambiguous_metadata`` is then unavailable).
"""
if review_count is not None and (
type(review_count) is not int or review_count < 0
):
raise ValueError("review_count must be a non-negative integer or None")
roots = [str(root) for root in roots]
database = _run_database_checks(self.model.db_path, roots)
checks = []
for check_id, label in CHECKS:
if check_id == "ambiguous_metadata":
result = _ambiguous(review_count)
else:
result = database[check_id] or _failed()
checks.append({"id": check_id, "label": label, **result})
return {
"success": True,
"checked_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"status": overall_status(checks),
"checks": checks,
}
+51
View File
@@ -628,3 +628,54 @@ class LibraryOperations:
"kind": "prompt-manager-library",
"positives": [owners[key] for key in order],
}
def library_counts(self) -> dict:
"""System › Overview counts from one snapshot, writing nothing.
``positive_prompts`` and ``negative_variants`` are the unfiltered
Library's ``total`` and ``variant_total``: eligible parents and active
pairs (every pair's positive is eligible). ``images`` counts indexed
``generated_images`` rows, not files on disk. An image has no rating
of its own (it shows its linked prompt's), so ``rated_images`` counts
the indexed images whose linked prompt is rated.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
row = conn.execute(
"SELECT (SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + "),"
"(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
"(SELECT COUNT(*) FROM generated_images),"
"(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)"
).fetchone()
return {
"positive_prompts": row[0],
"negative_variants": row[1],
"images": row[2],
"rated_images": row[3],
}
def usage_by_category(self) -> list[dict]:
"""Runs and average rating of the Library's parents, per category.
Only eligible parents count, so a negative-only prompt's runs (which
repeat its pairs' runs) are never added twice. A blank category groups
with no category as ``None``. Unrated parents are left out of the
average, which is ``None`` when no parent is rated. Most runs first.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
rows = conn.execute(
"SELECT * FROM (SELECT NULLIF(TRIM(p.category),'') AS category,"
"COALESCE(SUM(p.run_count),0) AS runs,AVG(p.rating) AS rating "
"FROM prompts p WHERE " + _ELIGIBLE + " GROUP BY 1) "
"ORDER BY runs DESC,category IS NULL,category"
).fetchall()
return [
{
"category": row["category"],
"runs": row["runs"],
"average_rating": (
None if row["rating"] is None else round(row["rating"], 2)
),
}
for row in rows
]
+8 -26
View File
@@ -20,6 +20,7 @@ from .execution_history import ExecutionHistoryOperations
from .execution_contexts import ExecutionContextOperations, LIVE_HINT_CTES
from .execution_outputs import ExecutionOutputOperations
from .history_reads import HistoryReadOperations
from .health import HealthOperations, iter_consistency_issues
# Import logging system
try:
@@ -342,6 +343,7 @@ class PromptDatabase(
ImageFavoriteOperations,
AnnotationOperations,
SavedViewOperations,
HealthOperations,
):
"""Database operations class for managing prompts."""
@@ -2570,33 +2572,13 @@ class PromptDatabase(
return cursor.rowcount
def check_consistency(self) -> List[str]:
"""Run consistency checks and return a list of issue descriptions."""
issues: List[str] = []
with self.model.get_connection() as conn:
# Check for orphaned prompt_tags entries
cursor = conn.execute("""
SELECT pt.prompt_id, pt.tag_id FROM prompt_tags pt
LEFT JOIN prompts p ON pt.prompt_id = p.id
WHERE p.id IS NULL
""")
for ref in cursor.fetchall():
issues.append(
"prompt_tags entry references non-existent prompt "
f"{ref['prompt_id']}"
)
"""Run consistency checks and return a list of issue descriptions.
# Check for orphaned image entries
cursor = conn.execute("""
SELECT gi.id, gi.prompt_id FROM generated_images gi
LEFT JOIN prompts p ON gi.prompt_id = p.id
WHERE p.id IS NULL
""")
for ref in cursor.fetchall():
issues.append(
f"Image {ref['id']} references non-existent prompt "
f"{ref['prompt_id']}"
)
return issues
The same queries back the System health check's Consistency row
(database/health.py).
"""
with self.model.get_connection() as conn:
return list(iter_consistency_issues(conn))
def _image_row_to_dict(self, row: sqlite3.Row) -> Dict[str, Any]:
"""
+61 -11
View File
@@ -7,6 +7,8 @@ Split into domain-specific mixins for maintainability:
- LoggingRoutesMixin: log management endpoints
- AutotagRoutesMixin: auto-tagging model management and tagging
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
- SystemRoutesMixin: System health check, Overview and usage statistics
"""
import asyncio
@@ -31,7 +33,10 @@ from .saved_views import SavedViewRoutesMixin
from .history import HistoryRoutesMixin
from .images import ImageRoutesMixin
from .admin import AdminRoutesMixin
from .settings_routes import SettingsRoutesMixin
from .job_routes import JobRoutesMixin
from .image_tag_jobs import ImageTagJobsMixin
from .system import SystemRoutesMixin
from .jobs import JobRegistry
from .logging_routes import LoggingRoutesMixin
from .autotag_routes import AutotagRoutesMixin
@@ -105,16 +110,20 @@ STATIC_MIME_TYPES = {
".html": "text/html",
}
_STATIC_SEPARATORS = re.compile(r"[\\/]+")
# Served by exact request path beside the extension allow-list: the browser's
# copy of the inspection node table (local Inspect resolves with it, #249).
STATIC_EXACT_FILES = {("js", "inspection-semantics.json"): "application/json"}
def _resolve_static_file(root, filepath):
def _resolve_static_file(root, filepath, allowed=STATIC_MIME_TYPES):
"""Real path of ``filepath`` under ``root``, or None when it must not be served.
The request path is untrusted. It is rejected when it is empty, carries a
``..`` segment, is absolute on any platform (a leading slash or backslash,
a drive letter such as ``C:`` or a UNC ``\\\\server\\share`` prefix all
re-anchor ``os.path.join`` on Windows), has an extension outside
STATIC_MIME_TYPES, or resolves (symlinks included) outside ``root``.
``allowed`` (STATIC_MIME_TYPES), or resolves (symlinks included) outside
``root``.
"""
if not isinstance(filepath, str) or not filepath:
return None
@@ -127,7 +136,7 @@ def _resolve_static_file(root, filepath):
segments = _STATIC_SEPARATORS.split(filepath)
if any(segment in ("", ".", "..") for segment in segments):
return None
if os.path.splitext(filepath)[1].lower() not in STATIC_MIME_TYPES:
if os.path.splitext(filepath)[1].lower() not in allowed:
return None
try:
root_real = Path(os.path.normcase(os.path.realpath(root)))
@@ -259,7 +268,10 @@ class PromptManagerAPI(
PromptRoutesMixin,
ImageRoutesMixin,
AdminRoutesMixin,
SettingsRoutesMixin,
JobRoutesMixin,
ImageTagJobsMixin,
SystemRoutesMixin,
LoggingRoutesMixin,
AutotagRoutesMixin,
LoraIntegrationMixin,
@@ -316,6 +328,11 @@ class PromptManagerAPI(
self._scan_job = None
# The current duplicate scan (a py/api/jobs.Job outside the registry).
self._duplicate_job = None
# The last System health report this session and the database
# generation it describes, bumped when a restore replaces self.db
# (py/api/system.py).
self._health_report = None
self._db_generation = 0
# Run cleanup on initialization to remove any existing duplicates
try:
@@ -358,6 +375,24 @@ class PromptManagerAPI(
"""409 envelope for a long job that is already running."""
return _json_error(f"{label} already running", 409)
async def _run_held(self, kind, title, admission, operation):
"""Run a legacy-page operation under a registry hold (D80).
While ``operation()`` runs, the registry lists it as a running,
non-cancellable job, so workspace jobs its admission class conflicts
with are refused, naming it. When the registry refuses the hold, the
legacy 409 envelope names the running job and nothing is done. A
response of 400 or above ends the hold ``failed``.
"""
busy = self._jobs.blocker(kind, admission=admission)
if busy is not None:
return self._long_job_busy_response(busy.title)
async with self._jobs.hold(kind, title, admission=admission) as job:
response = await operation()
if getattr(response, "status", 200) >= 400:
job.finish("failed")
return response
async def _run_in_executor(self, func, *args, **kwargs):
"""Run a blocking function in the default thread pool executor.
@@ -453,7 +488,9 @@ class PromptManagerAPI(
self._register_prompt_routes(routes)
self._register_image_routes(routes)
self._register_admin_routes(routes)
self._register_settings_routes(routes)
self._register_job_routes(routes)
self._register_system_routes(routes)
self._register_logging_routes(routes)
self._register_autotag_routes(routes)
self._register_lora_routes(routes)
@@ -475,15 +512,20 @@ class PromptManagerAPI(
def _serve_static_file(self, subdir, filepath):
"""Serve ``web/<subdir>/<filepath>``; 403 outside it, 404 when missing."""
root = os.path.join(_get_project_root(), "web", subdir)
file_path = _resolve_static_file(root, filepath)
exact = STATIC_EXACT_FILES.get((subdir, filepath))
# An exact file passes the same containment check, its extension only.
allowed = STATIC_MIME_TYPES if exact is None else {".json"}
file_path = _resolve_static_file(root, filepath, allowed)
if file_path is None:
return web.Response(text="Forbidden", status=403)
content_type = (
exact or STATIC_MIME_TYPES[os.path.splitext(file_path)[1].lower()]
)
if not os.path.isfile(file_path):
return web.Response(text="Not Found", status=404)
ext = os.path.splitext(file_path)[1].lower()
with open(file_path, "rb") as f:
content = f.read()
return web.Response(body=content, content_type=STATIC_MIME_TYPES[ext])
return web.Response(body=content, content_type=content_type)
def _serve_html_page(self, filename, title, cache=True):
"""Serve ``web/<filename>``; 404 when missing, 500 when unreadable."""
@@ -695,6 +737,13 @@ class PromptManagerAPI(
Returns:
List[Path]: Validated directory paths, possibly empty.
"""
return self._output_dirs_with_origin()[0]
def _output_dirs_with_origin(self):
"""``(dirs, auto_detected)``: the ``_get_all_output_dirs`` list, and
whether it comes from detection or ComfyUI's output rather than from
a valid configured root (False when there is no directory at all).
"""
from ..config import GalleryConfig
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
@@ -713,15 +762,16 @@ class PromptManagerAPI(
output_dirs.append(Path(real))
if output_dirs:
return output_dirs
return output_dirs, False
if configured:
# Every configured root was rejected: never let auto-detection
# resurrect it, serve ComfyUI's own output directory instead.
return self._comfyui_output_dir_list()
fallback = self._find_comfyui_output_dir()
return [Path(fallback)] if fallback else []
fallback_dirs = self._comfyui_output_dir_list()
else:
fallback = self._find_comfyui_output_dir()
fallback_dirs = [Path(fallback)] if fallback else []
return fallback_dirs, bool(fallback_dirs)
def _comfyui_output_dir_list(self):
"""ComfyUI's output directory as a one-element list, or [] if unknown."""
+102 -239
View File
@@ -22,16 +22,17 @@ from .jobs import (
SCAN_GROUP_NAMES,
SCAN_GROUPS,
SCAN_SCOPES,
DUPLICATE_CLEANUP_JOB_KIND,
MAINTENANCE_JOB_KIND,
RESTORE_JOB_KIND,
Job,
JobBusy,
)
try:
from ...utils.parallel import map_parallel
from ...utils.validators import validate_result_timeout, validate_worker_threads
except ImportError:
from utils.parallel import map_parallel
from utils.validators import validate_result_timeout, validate_worker_threads
try:
from ...database.operations import PromptDatabase
@@ -40,7 +41,7 @@ try:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_js_json,
parse_page_json,
)
from ...utils.prompt_pair_resolver import resolve_prompt_pair
except ImportError:
@@ -50,7 +51,7 @@ except ImportError:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_js_json,
parse_page_json,
)
from utils.prompt_pair_resolver import resolve_prompt_pair
@@ -198,13 +199,22 @@ def _plural(count, word):
return f"{count} {word}" if count == 1 else f"{count} {word}s"
def _time_left(done, total, batches, elapsed):
""" " · about X min left" once enough batches ran, else "" (D56).
Shared by every Activity job that reports "Processing file N of M": the
scan counts its batches, the image tagging job each file.
"""
if batches < MIN_BATCHES_FOR_ESTIMATE or done <= 0 or done >= total:
return ""
minutes = max(1, math.ceil(elapsed / done * (total - done) / 60))
return f" · about {minutes} min left"
def _scan_status_line(done, total, batches, elapsed):
""" "Processing file N of M", with an estimate once enough batches ran."""
line = f"Processing file {done:,} of {total:,}"
if batches < MIN_BATCHES_FOR_ESTIMATE or done <= 0 or done >= total:
return line
minutes = max(1, math.ceil(elapsed / done * (total - done) / 60))
return f"{line} · about {minutes} min left"
return line + _time_left(done, total, batches, elapsed)
def _file_size(path):
@@ -493,14 +503,6 @@ class AdminRoutesMixin:
async def get_stats_route(request):
return await self.get_statistics(request)
@routes.get("/prompt_manager/settings")
async def get_settings_route(request):
return await self.get_settings(request)
@routes.post("/prompt_manager/settings")
async def save_settings_route(request):
return await self.save_settings(request)
@routes.get("/prompt_manager/backup")
async def backup_database_route(request):
return await self.backup_database(request)
@@ -642,7 +644,15 @@ class AdminRoutesMixin:
job.finish()
async def cleanup_duplicates_endpoint(self, request):
"""Cleanup duplicate prompts endpoint."""
"""Cleanup duplicate prompts endpoint (holds the registry: records)."""
return await self._run_held(
MAINTENANCE_JOB_KIND,
"Prompt cleanup (admin page)",
"records",
self._cleanup_duplicates,
)
async def _cleanup_duplicates(self):
try:
removed_count = await self._run_in_executor(self.db.cleanup_duplicates)
@@ -757,30 +767,46 @@ class AdminRoutesMixin:
return hash_sha256.hexdigest()
async def delete_duplicate_images_endpoint(self, request):
"""Delete duplicate image files from disk (single flight with the scan)."""
job = self._acquire_long_job(self.LONG_JOB_DUPLICATES)
if job is None:
"""Delete duplicate image files from disk.
Single flight with the duplicate scan, and a records hold on the
registry, so no scan, restore or tagging job runs meanwhile. A body
without paths is refused before the hold (it is no job).
"""
data, error_response = await _read_json(request)
if error_response is not None:
return error_response
image_paths = data.get("image_paths", [])
if not image_paths or not isinstance(image_paths, list):
return web.json_response(
{"success": False, "error": "No image paths provided"},
status=400,
)
guard = self._acquire_long_job(self.LONG_JOB_DUPLICATES)
if guard is None:
return self._long_job_busy_response("Duplicate scan or delete")
async with job:
return await self._delete_duplicate_images(request)
async def _delete_duplicate_images(self, request):
async def delete():
# No await since the check above: the lock is still free.
async with guard:
return await self._delete_duplicate_images(image_paths)
return await self._run_held(
DUPLICATE_CLEANUP_JOB_KIND,
"Duplicate image delete (gallery page)",
"records",
delete,
)
async def _delete_duplicate_images(self, image_paths):
try:
data, error_response = await _read_json(request)
if error_response is not None:
return error_response
image_paths = data.get("image_paths", [])
if not image_paths or not isinstance(image_paths, list):
return web.json_response(
{"success": False, "error": "No image paths provided"},
status=400,
)
output_dir = await self._run_in_executor(self._find_comfyui_output_dir)
result = await self._run_in_executor(
self._delete_duplicate_images_sync, image_paths, output_dir
)
if result["deleted_count"]:
# The cached listing would keep (and count) the deleted files.
self.invalidate_gallery_cache()
response_data = {
"success": True,
@@ -873,208 +899,6 @@ class AdminRoutesMixin:
status=500,
)
async def get_settings(self, request):
"""Get current settings."""
try:
from ..config import PromptManagerConfig, GalleryConfig
root_paths = [
self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES
]
return web.json_response(
{
"success": True,
"settings": {
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
"worker_threads": PromptManagerConfig.WORKER_THREADS,
"cpu_count": PromptManagerConfig.max_worker_threads(),
"gallery_root_paths": root_paths,
"gallery_root_path": root_paths[0] if root_paths else "",
"monitored_directories": [
self._public_path(d) for d in self._monitored_directories()
],
},
}
)
except Exception as e:
self.logger.error(f"Get settings error: {e}", exc_info=True)
return web.json_response(
{"success": False, "error": "Failed to get settings"},
status=500,
)
def _monitored_directories(self):
"""Directories the running image monitor watches, else the configured roots."""
from ..config import GalleryConfig
monitor = _current_image_monitor()
if monitor is not None:
return list(getattr(monitor, "monitored_directories", []))
return list(GalleryConfig.MONITORING_DIRECTORIES)
async def save_settings(self, request):
"""Save settings: the whole payload is validated before anything changes."""
try:
from ..config import PromptManagerConfig
data, error_response = await _read_json(request)
if error_response is not None:
return error_response
try:
updates = self._validate_settings(data)
except ValueError as ve:
return web.json_response(
{"success": False, "error": str(ve)}, status=400
)
new_roots, error = self._parse_gallery_roots(data)
if error:
return web.json_response({"success": False, "error": error}, status=400)
for name, value in updates.items():
setattr(PromptManagerConfig, name, value)
restart_required = False
if new_roots is not None:
restart_required = self._apply_gallery_roots(new_roots)
self._persist_settings()
return web.json_response(
{
"success": True,
"message": "Settings saved successfully",
"restart_required": restart_required,
}
)
except Exception as e:
self.logger.error(f"Save settings error: {e}", exc_info=True)
return web.json_response(
{"success": False, "error": "Failed to save settings"},
status=500,
)
@staticmethod
def _validate_settings(data):
"""Scalar settings of a payload as ``{CONFIG_ATTR: value}``.
Raises ValueError with a user-facing message on the first bad value;
nothing is applied until every value passed.
"""
from ..config import PromptManagerConfig
updates = {}
if "result_timeout" in data:
updates["RESULT_TIMEOUT"] = validate_result_timeout(data["result_timeout"])
if "webui_display_mode" in data:
mode = data["webui_display_mode"]
if mode not in ("popup", "newtab"):
raise ValueError("webui_display_mode must be 'popup' or 'newtab'")
updates["WEBUI_DISPLAY_MODE"] = mode
if "infinite_scroll" in data:
if not isinstance(data["infinite_scroll"], bool):
raise ValueError("infinite_scroll must be true or false")
updates["INFINITE_SCROLL"] = data["infinite_scroll"]
if "worker_threads" in data:
updates["WORKER_THREADS"] = validate_worker_threads(
data["worker_threads"], PromptManagerConfig.max_worker_threads()
)
return updates
def _parse_gallery_roots(self, data):
"""Extract and validate gallery roots from a settings payload.
Accepts ``gallery_root_paths`` (list, preferred) or the legacy single
``gallery_root_path`` string. Blank entries are dropped.
Returns:
(roots, error): ``roots`` is the validated list, or ``None`` when
the payload carries no gallery root key; ``error`` is a message
when validation failed (and ``roots`` is then ``None``).
"""
if "gallery_root_paths" in data:
raw = data["gallery_root_paths"]
if not isinstance(raw, list):
return None, "gallery_root_paths must be a list"
elif "gallery_root_path" in data:
raw = [data["gallery_root_path"]]
else:
return None, None
from ..config import GalleryConfig
roots = []
for entry in raw:
if not isinstance(entry, str):
return None, "Gallery root paths must be strings"
entry = entry.strip()
if not entry:
continue
ok, reason = GalleryConfig.validate_gallery_root(entry)
if not ok:
return (
None,
f"Invalid gallery root '{os.path.basename(entry)}': {reason}",
)
roots.append(GalleryConfig.resolve_gallery_root(entry))
return roots, None
def _apply_gallery_roots(self, roots):
"""Install validated gallery roots; returns True when they changed."""
from ..config import GalleryConfig
if roots == list(GalleryConfig.MONITORING_DIRECTORIES):
return False
GalleryConfig.MONITORING_DIRECTORIES = roots
self._cached_output_dir = None
self.reset_gallery_roots_cache()
return True
def _persist_settings(self):
"""Merge the user-editable settings into the configured config.json.
Only the keys this endpoint owns are rewritten; every other section
(integrations, database, performance, ...) and every other key in
``web_ui``/``gallery`` is kept as it was. A config file that cannot
be parsed is replaced.
"""
from ..config import GalleryConfig, PromptManagerConfig, write_private_json
config_file = PromptManagerConfig.get_config_path()
existing = self._read_config_file(config_file)
web_ui = dict(existing.get("web_ui") or {})
web_ui["result_timeout"] = PromptManagerConfig.RESULT_TIMEOUT
web_ui["webui_display_mode"] = PromptManagerConfig.WEBUI_DISPLAY_MODE
web_ui["infinite_scroll"] = PromptManagerConfig.INFINITE_SCROLL
gallery = dict(existing.get("gallery") or {})
monitoring = dict(gallery.get("monitoring") or {})
monitoring["directories"] = list(GalleryConfig.MONITORING_DIRECTORIES)
gallery["monitoring"] = monitoring
performance = dict(existing.get("performance") or {})
performance["worker_threads"] = PromptManagerConfig.WORKER_THREADS
config_data = {
**existing,
"web_ui": web_ui,
"gallery": gallery,
"performance": performance,
}
try:
write_private_json(config_file, config_data)
self.logger.info(f"Settings saved to {config_file}")
except OSError as save_err:
self.logger.warning(f"Could not save config file: {save_err}")
def _read_config_file(self, config_file):
"""Current contents of config.json as a dict, or {} when unusable."""
try:
with open(config_file, "r", encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError):
return {}
return data if isinstance(data, dict) else {}
async def run_diagnostics(self, request):
"""Run comprehensive system diagnostics and health checks."""
try:
@@ -1167,6 +991,11 @@ class AdminRoutesMixin:
"vacuum",
"cleanup_orphaned_images",
)
# Operations that only read; a run of only these holds the registry as
# "read", every other run as "records".
READ_ONLY_MAINTENANCE_OPERATIONS = frozenset(
{"check_hash_duplicates", "statistics", "check_consistency"}
)
async def run_maintenance(self, request):
"""Perform database maintenance operations and report each outcome."""
@@ -1179,7 +1008,27 @@ class AdminRoutesMixin:
operations = data.get(
"operations", list(self.DEFAULT_MAINTENANCE_OPERATIONS)
)
except Exception as e:
self.logger.error(f"Maintenance error: {e}", exc_info=True)
return web.json_response(
{"success": False, "error": "Maintenance failed"}, status=500
)
writes = not (
isinstance(operations, list)
and all(
isinstance(name, str) and name in self.READ_ONLY_MAINTENANCE_OPERATIONS
for name in operations
)
)
return await self._run_held(
MAINTENANCE_JOB_KIND,
"Maintenance (admin page)",
"records" if writes else "read",
lambda: self._run_maintenance(operations),
)
async def _run_maintenance(self, operations):
try:
results = await self._run_in_executor(
self._run_maintenance_operations, operations
)
@@ -1333,8 +1182,18 @@ class AdminRoutesMixin:
The upload is streamed to a temp file (capped at restore_max_bytes),
verified with PromptModel.verify_database_file and only then copied
over the live database, which is backed up first.
over the live database, which is backed up first. The whole request
holds the registry (records): it is refused while a scan or another
record-changing job runs, and those are refused while it runs.
"""
return await self._run_held(
RESTORE_JOB_KIND,
"Database restore (admin page)",
"records",
lambda: self._restore_database(request),
)
async def _restore_database(self, request):
max_bytes = getattr(self, "restore_max_bytes", 100 * 1024 * 1024)
temp_path = None
try:
@@ -1415,6 +1274,7 @@ class AdminRoutesMixin:
self.logger.info(f"Removed {len(removed)} old safety backups")
previous = self.db
self.db = PromptDatabase(model.db_path)
self._database_replaced() # a health report described the old one
# Its connections would otherwise linger (a file lock on Windows)
previous.close_all()
prompt_count = self.db.model.get_database_info().get("total_prompts", 0)
@@ -1746,11 +1606,14 @@ class AdminRoutesMixin:
@staticmethod
def _graph_prompt_text(metadata):
"""The positive prompt the browser parser traces from the graphs, or None."""
"""The positive prompt the browser parser traces from the graphs, or None.
Records parse as the legacy pages parsed them (NaN reads as null).
"""
records = {}
for key in ("prompt", "workflow"):
value = metadata.get(key)
records[key] = parse_js_json(value) if isinstance(value, str) else value
records[key] = parse_page_json(value) if isinstance(value, str) else value
try:
positive = extract_generation_params(records)["positivePrompt"]
except GenerationParamsError:
+35
View File
@@ -8,6 +8,7 @@ from pathlib import Path
from typing import Optional
from urllib.parse import quote
from .jobs import AUTOTAG_BATCH_JOB_KIND
from .prompts import thumbnail_url_for
from .. import model_gate
@@ -63,6 +64,12 @@ def path_is_within(candidate, root) -> bool:
return child != parent and child.is_relative_to(parent)
async def _refusal(title):
"""The single error event of an auto-tag batch refused by another job."""
message = f"{title} already running"
yield {"type": "error", "status": "error", "message": message}
class AutotagRoutesMixin:
"""Mixin providing auto-tagging API endpoints."""
@@ -425,6 +432,8 @@ class AutotagRoutesMixin:
"general_threshold and character_threshold must be numeric"
)
errors = [] # an error event ends the run; Activity lists it failed
async def events():
try:
from ..autotag import get_autotag_service
@@ -432,14 +441,40 @@ class AutotagRoutesMixin:
async for event in self._autotag_batch_events(
get_autotag_service(), opts
):
if event.get("type") == "error":
errors.append(event)
yield event
except Exception:
self.logger.exception("AutoTag error")
errors.append(_INTERNAL_ERROR)
yield {
"type": "error",
"message": _INTERNAL_ERROR,
}
# The batch writes prompt tags: a records hold on the registry, so it
# never runs beside a scan, restore or the workspace tagging job.
busy = self._jobs.blocker(AUTOTAG_BATCH_JOB_KIND, admission="records")
if busy is not None:
if busy.kind == AUTOTAG_BATCH_JOB_KIND: # unchanged payload
return web.json_response(
{"success": False, "error": _BUSY_ERROR}, status=409
)
# Both legacy pages read this route as a stream, where a 409 reads
# as a lost connection: refuse with one error frame instead (the
# admin page keys on ``type``, the gallery on ``status``).
return await self._stream_sse(request, _refusal(busy.title))
async with self._jobs.hold(
AUTOTAG_BATCH_JOB_KIND,
"Auto-tag images (legacy page)",
admission="records",
) as job:
response = await self._start_autotag_batch(request, opts, events)
if response.status >= 400 or errors:
job.finish("failed")
return response
async def _start_autotag_batch(self, request, opts, events):
if not _autotag_batch_lock.acquire(blocking=False):
return web.json_response(
{"success": False, "error": _BUSY_ERROR}, status=409
+509
View File
@@ -0,0 +1,509 @@
"""Bulk image tagging as an Activity job: Add tags and Auto-tag (#249 D1).
``POST /prompt_manager/jobs/image-tags`` starts one ``image-tags`` job over
1..500 listed images, named by the listing identity ``{root_index,
root_scope, path}``. The job is record-changing (it writes image tag sets and
new vocabulary names), so it never runs beside the metadata scan.
- **Add tags** (``mode: "add"``) hashes each current file and unions a fixed
tag set into the record of exactly those bytes.
- **Auto-tag** (``mode: "auto"``) copies each still image into a private
capture, asks one legacy engine for its top N tags on the capture, re-hashes
the current file and unions the suggestions only when the bytes are still
the captured ones. It holds the shared model gate for the whole job, loads
the model once and never downloads one.
Every step reuses the Tag dialog's pieces: the capture store, the model gate,
the suggestion worker and the storage call ``add_image_tags``. Files are
handled one at a time and the job stops at a file boundary when cancelled,
keeping what it already wrote. Rows and log lines carry root-relative paths
and public root labels only.
"""
import asyncio
import functools
import os
import time
from dataclasses import dataclass, field
from aiohttp import web
from .. import model_gate
from .admin import _plural, _time_left
from .annotation_suggest import (
OptionError,
unavailable_message,
validate_options,
)
from .annotations import (
_MESSAGES,
MAX_TARGET_PATH,
_coded,
_error,
_pinned_suggestions,
_settled,
_submit,
_target,
)
from .autotag_routes import model_busy_response
from .images import _public_path
from .job_routes import busy_response
from .jobs import IMAGE_TAG_GROUP_NAMES, IMAGE_TAG_JOB_KIND, JobBusy
from .prompts import bad_request
try:
from ...database.annotations import ImageTagLimitError, normalize_image_tags
from ...database.library_commands import LibraryCommandError
from ...utils.media_capture import (
CaptureCapacity,
CaptureChanged,
CaptureDenied,
CaptureError,
CaptureTooLarge,
)
from ...utils.video_metadata import is_video_path
except ImportError:
from database.annotations import ImageTagLimitError, normalize_image_tags
from database.library_commands import LibraryCommandError
from utils.media_capture import (
CaptureCapacity,
CaptureChanged,
CaptureDenied,
CaptureError,
CaptureTooLarge,
)
from utils.video_metadata import is_video_path
MAX_JOB_TARGETS = 500
# 500 maximal targets plus the options: a path of MAX_TARGET_PATH code points
# takes at most 6 bytes each on the wire (a \uXXXX escape; raw UTF-8 is at most
# 4), and the root index and scope fit in 256. The annotation routes' 64 KiB
# bound is for one target, too small for a full selection.
MAX_JOB_BODY_BYTES = MAX_JOB_TARGETS * (MAX_TARGET_PATH * 6 + 256) + 64 * 1024
# A busy capture worker (a Tag dialog in another tab) is waited for this long,
# try by try, before the file is left for a rerun.
CAPACITY_BACKOFF = (0.1, 0.2, 0.4, 0.8)
ADD_TITLE = "Add tags to images"
AUTO_TITLE = "Auto-tag images"
GATE_OWNER_KIND = "image-tag-job"
_FIELDS = {
"add": frozenset({"mode", "targets", "tags"}),
"auto": frozenset({"mode", "targets", "model_type", "options"}),
}
_VERBS = {"add": "Add tags", "auto": "Auto-tag"}
_LOG_PREFIXES = {"tagged": "+", "unchanged": "=", "skipped": "-", "errors": "!"}
NO_NEW_TAGS = "No new tags"
VIDEO_SKIPPED = "Video: manual tags only"
TOO_LARGE = "Too large to verify"
FILE_CHANGED = "File changed"
FILE_NOT_FOUND = "File not found"
NOT_A_STILL = "Not a supported still image"
INFERENCE_FAILED = "Auto-tagging failed. Check server logs for details."
SAVE_FAILED = "The tags could not be saved. Check server logs for details."
READ_FAILED = "The file could not be read."
SOURCE_CHANGED = "The source changed. Refresh the image collection."
STORAGE_RESTORED = "Prompt storage was restored; start the job again."
BUSY_RETRY = "The file could not be read now. Run the job again for it."
_monotonic = time.monotonic # the job's clock (a test can fake it)
class _JobStopped(Exception):
"""No later file can be written (prompt storage was restored)."""
@dataclass
class _Run:
"""One running job: its request, the gate owner and the resolved roots.
``roots`` maps each target's ``(root_index, root_scope)`` to ``(root,
label)``, read once per job off the event loop; ``root`` is None when the
scope no longer names a current root.
"""
job: object
spec: object
epoch: str
owner: object
service: object = None
roots: dict = field(default_factory=dict)
@dataclass(frozen=True)
class _Spec:
"""A validated start request."""
mode: str
targets: tuple
tags: tuple = ()
model_type: str | None = None
options: dict | None = None
@property
def title(self) -> str:
return ADD_TITLE if self.mode == "add" else AUTO_TITLE
def scope(self) -> dict:
scope = {"mode": self.mode, "images": len(self.targets)}
if self.mode == "auto":
scope["model_type"] = self.model_type
return scope
def _parse_targets(raw):
"""Unique validated targets in request order, or ``(None, error)``."""
if not isinstance(raw, list) or not 1 <= len(raw) <= MAX_JOB_TARGETS:
return None, bad_request(f"targets must list 1 to {MAX_JOB_TARGETS} images")
seen, targets = set(), []
for item in raw:
target, error = _target(item)
if error is not None:
return None, error
if target not in seen:
seen.add(target)
targets.append(target)
return tuple(targets), None
def _parse_add(targets, data):
try:
tags = normalize_image_tags(data["tags"])
except ImageTagLimitError as exc:
return None, _error(400, "tag-limit", str(exc))
except ValueError as exc:
return None, bad_request(str(exc))
if not tags:
return None, bad_request("tags must name at least one tag")
return _Spec("add", targets, tags=tuple(tags)), None
def _parse_auto(targets, data):
from ..config import MAX_SUGGESTION_LIMIT, AutotagConfig
try:
model_type, options = validate_options(
data["model_type"],
data["options"],
default_limit=AutotagConfig.suggestion_limit(),
max_limit=MAX_SUGGESTION_LIMIT,
)
except OptionError as exc:
return None, bad_request(str(exc))
return _Spec("auto", targets, model_type=model_type, options=options), None
def parse_image_tag_request(data: dict):
"""A :class:`_Spec` for a start body, or ``(None, error response)``.
``{mode:"add", targets, tags}`` or ``{mode:"auto", targets, model_type,
options}`` exactly; options follow the Tag dialog's Generate rules.
"""
fields = _FIELDS.get(data.get("mode"))
if fields is None:
return None, bad_request('mode must be "add" or "auto"')
if set(data) != fields:
return None, bad_request(
"Request must contain exactly " + ", ".join(sorted(fields))
)
targets, error = _parse_targets(data["targets"])
if error is not None:
return None, error
if data["mode"] == "add":
return _parse_add(targets, data)
return _parse_auto(targets, data)
def _capture_outcome(exc: CaptureError, root, relative: str):
"""The (group, detail) of a file the capture store refused to read."""
if isinstance(exc, CaptureTooLarge):
return "skipped", TOO_LARGE
if isinstance(exc, CaptureDenied):
return "skipped", FILE_NOT_FOUND
if isinstance(exc, CaptureChanged):
path = os.path.join(root, *relative.split("/"))
return "skipped", FILE_CHANGED if os.path.lexists(path) else FILE_NOT_FOUND
if isinstance(exc, CaptureCapacity):
return "skipped", BUSY_RETRY
return "errors", READ_FAILED
async def _capacity_retried(job, attempt):
"""``await attempt()``, tried again while the capture store is at capacity.
Raises the last ``CaptureCapacity`` once ``CAPACITY_BACKOFF`` is spent or the
job was cancelled meanwhile.
"""
for delay in CAPACITY_BACKOFF:
try:
return await attempt()
except CaptureCapacity:
if job.cancelled:
raise
await asyncio.sleep(delay)
return await attempt()
def _load_model(service, model_type: str, use_gpu: bool) -> None:
"""Worker thread: load (or swap to) *model_type*; never downloads."""
if service.is_model_loaded() and service.get_loaded_model_type() == model_type:
return
if service.load_model(model_type, use_gpu) is False:
raise RuntimeError(f"{model_type} could not be loaded")
class ImageTagJobsMixin:
"""The ``image-tags`` job kind: start route and runner."""
async def start_image_tag_job(self, request: web.Request) -> web.Response:
"""Start bulk tagging (202), or 409 ``busy`` / ``model-busy``.
Every target's root scope is checked before the job starts
(``source-changed``). Auto-tag takes the model gate here, so a start
while anything else holds it is refused at once.
"""
data, error = await self._read_json_body(request, max_bytes=MAX_JOB_BODY_BYTES)
if error is not None:
return error
spec, error = parse_image_tag_request(data)
if error is not None:
return error
roots = await self._run_in_executor(self._job_roots, spec.targets)
if any(root is None for root, _label in roots.values()):
return _coded("source-changed")
epoch = await self._run_in_executor(self.db.get_execution_epoch)
# A running job in the way answers first: it says which job to wait for.
blocker = self._jobs.blocker(IMAGE_TAG_JOB_KIND, record_changing=True)
if blocker is not None:
return busy_response(JobBusy(blocker))
owner = None
if spec.mode == "auto":
owner = model_gate.new_owner(GATE_OWNER_KIND)
if not model_gate.try_acquire(owner):
return model_busy_response()
try:
job = self._jobs.start(
IMAGE_TAG_JOB_KIND,
spec.title,
record_changing=True,
scope=spec.scope(),
runner=functools.partial(self._run_image_tag_job, spec, epoch, owner),
groups=IMAGE_TAG_GROUP_NAMES,
)
except JobBusy as busy:
if owner is not None:
model_gate.release(owner)
return busy_response(busy)
if owner is not None:
# Also when the task is cancelled before its runner ever starts.
job.task.add_done_callback(lambda _task: model_gate.release(owner))
return web.json_response({"success": True, "job": job.summary()}, status=202)
# -- runner ----------------------------------------------------------------
def _job_roots(self, targets) -> dict:
"""Worker thread: ``{(index, scope): (root or None, label)}`` (see _Run)."""
roots = list(self._get_all_output_dirs())
resolved = {}
for index, scope in dict.fromkeys((t[0], t[1]) for t in targets):
label = _public_path(roots[index]) if index < len(roots) else ""
error = self._gallery_scope_error(
{"root_scope": scope, "root": str(index)}, roots
)
resolved[(index, scope)] = (roots[index] if error is None else None, label)
return resolved
async def _run_image_tag_job(self, spec: _Spec, epoch: str, owner, job) -> None:
verb = _VERBS[spec.mode]
total = len(spec.targets)
job.update(total=total, status_line=f"Processing file 0 of {total:,}")
started = f"{verb} started · {_plural(total, 'image')}"
if spec.mode == "auto":
started += f" · {spec.model_type}"
job.log_line(started)
run = _Run(job, spec, epoch, owner)
try:
if spec.mode == "auto":
run.service = await self._prepare_tagging_model(job, spec, owner)
if run.service is None:
job.finish("failed")
return
if not await self._tag_each_file(run):
job.log_line(f"{verb} stopped · {STORAGE_RESTORED}")
job.finish("failed")
return
state = "cancelled" if job.cancelled else "finished"
job.log_line(f"{verb} {state} · {_plural(job.processed, 'file')} processed")
job.publish({"type": state, "progress": job.progress})
job.finish(state)
finally:
if owner is not None:
model_gate.release(owner)
async def _tag_each_file(self, run: _Run) -> bool:
"""Tag the targets in order; False when the job had to stop."""
job, total = run.job, len(run.spec.targets)
run.roots = await self._run_in_executor(self._job_roots, run.spec.targets)
started = _monotonic()
for number, target in enumerate(run.spec.targets, 1):
if job.cancelled:
break
# The scan's estimate (D56), with each finished file as one batch.
left = _time_left(number - 1, total, number - 1, _monotonic() - started)
job.update(status_line=f"Processing file {number:,} of {total:,}{left}")
job.publish({"type": "progress", "progress": job.progress})
stopped = False
try:
group, detail = await self._tag_one_file(run, target)
except _JobStopped:
stopped, group, detail = True, "errors", STORAGE_RESTORED
except Exception:
self.logger.exception("Image tagging failed for one file")
group, detail = "errors", SAVE_FAILED
self._record_tag_outcome(run, target, group, detail)
job.update(processed=number, progress=number * 100 // total)
if stopped:
return False
return True
async def _prepare_tagging_model(self, job, spec: _Spec, owner):
"""The loaded service, or None (logged) when the job cannot run."""
from ..autotag import MODELS, get_autotag_service
service = get_autotag_service()
status = await self._run_in_executor(service.get_models_status)
if not status.get(spec.model_type, {}).get("downloaded"):
name = MODELS.get(spec.model_type, {}).get("name", spec.model_type)
message = unavailable_message(name)
job.update(status_line=message)
job.log_line(f"! {message}")
return None
try:
await model_gate.run(
owner,
_load_model,
service,
spec.model_type,
spec.options["use_gpu"],
)
except Exception:
self.logger.exception("The auto-tag job could not load its model")
job.update(status_line=INFERENCE_FAILED)
job.log_line(f"! {INFERENCE_FAILED}")
return None
return service
async def _verified(self, run: _Run, store, root, relative):
"""The current file's facts, waiting out a busy capture store."""
return await _capacity_retried(
run.job, lambda: self._run_in_executor(store.verify, root, relative)
)
async def _tag_one_file(self, run: _Run, target):
"""The (group, detail) of one target; raises _JobStopped on a restore."""
index, scope, relative = target
root, _label = run.roots[(index, scope)]
if root is None:
return "errors", SOURCE_CHANGED
store = self._capture_store()
if run.spec.mode == "add":
try:
facts = await self._verified(run, store, root, relative)
except CaptureError as exc:
return _capture_outcome(exc, root, relative)
return await self._union_image_tags(facts, list(run.spec.tags), run.epoch)
if is_video_path(relative):
return "skipped", VIDEO_SKIPPED
try:
capture = await _capacity_retried(
run.job,
lambda: _settled(
_submit(
store.capture,
root,
relative,
epoch=run.epoch,
scope=scope,
root_index=index,
),
lambda made: made is not None and store.release(made.token),
),
)
except CaptureError as exc:
return _capture_outcome(exc, root, relative)
try:
return await self._auto_tag_capture(run, store, capture, root, relative)
finally:
store.release(capture.token)
async def _auto_tag_capture(self, run: _Run, store, capture, root, relative):
"""Suggest on the capture, then union only onto the same current bytes."""
if not capture.inferable:
return "skipped", NOT_A_STILL
if store.acquire(capture.token) is None:
raise _JobStopped # every capture was invalidated (restore)
spec = run.spec
try:
pairs = await model_gate.run(
run.owner,
_pinned_suggestions,
store,
capture.token,
run.service,
capture,
spec.model_type,
spec.options,
on_skipped=functools.partial(store.release_use, capture.token),
)
except Exception:
self.logger.exception("Auto-tagging failed for one file")
return "errors", INFERENCE_FAILED
try:
facts = await self._verified(run, store, root, relative)
except CaptureError as exc:
return _capture_outcome(exc, root, relative)
if facts.locator != capture.locator or facts.sha256 != capture.sha256:
return "skipped", FILE_CHANGED
names = [str(name) for name, _score in pairs]
if not names:
return "unchanged", NO_NEW_TAGS
return await self._union_image_tags(facts, names, run.epoch)
async def _union_image_tags(self, facts, tags: list, epoch: str):
"""``add_image_tags`` on the verified bytes, as a (group, detail)."""
try:
receipt = await self._run_in_executor(
self.db.add_image_tags,
locator=facts.locator,
sha256=facts.sha256,
size=facts.size,
mtime_ns=facts.mtime_ns,
expected_epoch=epoch,
tags=tags,
)
except LibraryCommandError as exc:
if exc.code == "storage-changed":
raise _JobStopped from None
if exc.code == "tag-limit":
return "errors", _MESSAGES["tag-limit"]
self.logger.warning(f"Image tags not saved ({exc.code})")
return "errors", SAVE_FAILED
except ValueError:
self.logger.exception("Image tags were refused")
return "errors", SAVE_FAILED
added = receipt.get("added", 0)
if added:
return "tagged", f"Added {_plural(added, 'tag')}"
return "unchanged", NO_NEW_TAGS
def _record_tag_outcome(self, run: _Run, target, group: str, detail: str) -> None:
index, scope, relative = target
_root, label = run.roots[(index, scope)]
where = f"{label}/{relative}" if label else relative
run.job.record(
group,
{"path": relative, "root": label, "root_index": index, "detail": detail},
f"{_LOG_PREFIXES[group]} {where} · {detail}",
)
+54 -8
View File
@@ -19,6 +19,7 @@ from typing import Callable, NamedTuple
from aiohttp import web
from PIL import Image, UnidentifiedImageError
from .jobs import THUMBNAILS_JOB_KIND
from .prompts import bad_request, parse_page_params, publish_image_paths
from .prompts import thumbnail_url_for
from .prompts import safe_error_message as _safe_error
@@ -104,6 +105,10 @@ def _card_header(path: Path) -> tuple:
return inspection.width, inspection.height, records
# The legacy gallery page's thumbnail generation, as Activity lists it (D80).
THUMBNAIL_GENERATION_TITLE = "Thumbnail generation (gallery page)"
class _CardFacts(NamedTuple):
"""One image file's card facts, derived from its header alone.
@@ -1736,11 +1741,24 @@ class ImageRoutesMixin:
return cls.THUMBNAIL_SIZES.get(quality, cls.THUMBNAIL_SIZES["medium"])
async def generate_thumbnails(self, request):
"""Generate every thumbnail in one executor job (POST, blocking)."""
"""Generate every thumbnail in one executor job (POST, blocking).
Holds the registry as guarded work: refused while a record-changing
job runs, and such jobs are refused while it runs. A body that cannot
be read is refused before the hold, so it is never listed as a job.
"""
data, error = await _json_body_or_error(request)
if error is not None:
return error
return await self._run_held(
THUMBNAILS_JOB_KIND,
THUMBNAIL_GENERATION_TITLE,
"guarded",
lambda: self._generate_thumbnails(data),
)
async def _generate_thumbnails(self, data):
try:
data, error = await _json_body_or_error(request)
if error is not None:
return error
thumbnail_size = self._thumbnail_size(data.get("quality", "medium"))
output_dirs = self._get_all_output_dirs()
@@ -1935,14 +1953,32 @@ class ImageRoutesMixin:
"Connection": "keep-alive",
},
)
await response.prepare(request)
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
if busy is not None:
# The gallery reads this stream: refuse with its error event.
await response.prepare(request)
message = f"{busy.title} already running"
await self._emit_sse(
response, "error", {"error": message, "message": message}
)
return response
async with self._jobs.hold(
THUMBNAILS_JOB_KIND, THUMBNAIL_GENERATION_TITLE, admission="guarded"
) as job:
await response.prepare(request)
if not await self._thumbnail_progress_stream(response, thumbnail_size):
job.finish("failed") # an error event ended the stream
return response
async def _thumbnail_progress_stream(self, response, thumbnail_size) -> bool:
"""Stream one generation run; False when it ended with an error event."""
try:
output_dirs = self._get_all_output_dirs()
if not output_dirs:
await self._emit_sse(
response, "error", {"error": "ComfyUI output directory not found"}
)
return response
return False
await self._emit_sse(
response,
@@ -1960,6 +1996,7 @@ class ImageRoutesMixin:
remaining=plan["remaining"],
)
self.invalidate_gallery_cache()
return True
except Exception:
self.logger.exception("Thumbnail generation failed")
await self._emit_sse(
@@ -1970,7 +2007,7 @@ class ImageRoutesMixin:
"message": "Thumbnail generation failed. Check server logs.",
},
)
return response
return False
async def _stream_thumbnails(
self, response, targets, thumbnail_size, skipped=0, remaining=0
@@ -2179,8 +2216,17 @@ class ImageRoutesMixin:
"""Clear generated thumbnails across gallery roots, preserving originals.
Invalid thumbnail directories return 400; unavailable gallery roots
return 404, and unexpected filesystem failures return 500.
return 404, and unexpected filesystem failures return 500. Holds the
registry as guarded work, like generation.
"""
return await self._run_held(
THUMBNAILS_JOB_KIND,
"Thumbnail clear (gallery page)",
"guarded",
self._clear_thumbnails,
)
async def _clear_thumbnails(self) -> web.Response:
try:
output_dirs = self._get_all_output_dirs()
if not output_dirs:
+23 -13
View File
@@ -1,8 +1,9 @@
"""Session job routes (Activity): list, inspect, follow and cancel jobs.
The registry itself is ``py/api/jobs.py``; the scan job's runner is in
``py/api/admin.py``. Every response leaves out absolute server paths: job
rows already carry root-relative paths and public root labels.
The registry itself is ``py/api/jobs.py``. The scan job's runner is in
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``.
Every response leaves out absolute server paths: job rows already carry
root-relative paths and public root labels.
"""
import contextlib
@@ -42,6 +43,20 @@ async def _scan_scope(request):
return scope if scope in SCAN_SCOPES else None
def busy_response(busy: JobBusy):
"""``409 busy``, naming the running job that holds the slot."""
running = busy.running
return web.json_response(
{
"success": False,
"code": "busy",
"error": str(busy),
"running": {"id": running.id, "title": running.title},
},
status=409,
)
def _summary_event(job):
return f"data: {json.dumps(job.summary())}\n\n".encode("utf-8")
@@ -58,6 +73,10 @@ class JobRoutesMixin:
async def start_scan_job_route(request):
return await self.start_scan_job(request)
@routes.post("/prompt_manager/jobs/image-tags")
async def start_image_tag_job_route(request):
return await self.start_image_tag_job(request)
@routes.get("/prompt_manager/jobs/{job_id}")
async def job_detail_route(request):
return await self.job_detail(request)
@@ -121,16 +140,7 @@ class JobRoutesMixin:
try:
job = self._start_scan_job(files)
except JobBusy as busy:
running = busy.running
return web.json_response(
{
"success": False,
"code": "busy",
"error": str(busy),
"running": {"id": running.id, "title": running.title},
},
status=409,
)
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
async def job_events(self, request):
+175 -23
View File
@@ -9,8 +9,21 @@ as a snapshot and then every event published after it, until the job ends.
Jobs live only in server memory and are gone after a restart. The registry
keeps the running jobs and this session's last ``MAX_FINISHED_JOBS`` finished
ones. Record-changing jobs (the metadata scan, and later maintenance, cleanup
and restore) never run together, and one job of a kind runs at a time.
ones. One job of a kind runs at a time, and each job has an admission class
(``ADMISSIONS``) that decides what may run beside it:
- ``records`` changes library records or files (the metadata scan, bulk image
tagging, and the legacy restore, maintenance, cleanup, duplicate delete,
LoRA import and batch auto-tag routes). It is refused while any
``records`` or ``guarded`` job runs.
- ``guarded`` must not overlap record changes but may overlap other guarded
work (thumbnail generation and clearing, later backup). It is refused while
a ``records`` job runs.
- ``read`` only reads; it is refused only by a running job of its own kind.
A short legacy-page operation takes a ``hold``: a running, non-cancellable
job that lasts as long as the request, so the Activity page lists it and a
workspace job is refused while it runs (and the other way round).
"""
import asyncio
@@ -54,6 +67,49 @@ SCAN_GROUPS = {
"failed": "errors",
}
# Bulk image tagging (#249 D1): Add tags and Auto-tag over selected images.
# Record-changing: it writes image tag sets and new vocabulary names.
IMAGE_TAG_JOB_KIND = "image-tags"
IMAGE_TAG_GROUP_NAMES = ("tagged", "unchanged", "skipped", "errors")
ADMISSIONS = ("read", "guarded", "records")
# What each admission class is refused beside (besides its own kind).
_REFUSED_BESIDE = {
"read": frozenset(),
"guarded": frozenset({"records"}),
"records": frozenset({"records", "guarded"}),
}
# Kinds of the legacy-page operations that hold the registry (D80; titles in
# their handlers end in "(admin page)", "(gallery page)" or "(legacy page)").
RESTORE_JOB_KIND = "restore"
MAINTENANCE_JOB_KIND = "maintenance"
DUPLICATE_CLEANUP_JOB_KIND = "duplicate_cleanup"
THUMBNAILS_JOB_KIND = "thumbnails"
LORA_IMPORT_JOB_KIND = "lora_import"
AUTOTAG_BATCH_JOB_KIND = "autotag"
# The System health check (py/api/system.py): a read-only ``read`` job with
# no file outcomes, so it may run beside a scan.
HEALTH_JOB_KIND = "health"
def resolve_admission(admission=None, record_changing=None) -> str:
"""The admission class for a start call; ``record_changing`` is an alias.
``record_changing=True`` means ``records`` and ``False`` means ``read``;
with neither given the class is ``read``. Raises ValueError for an
unknown class or one that contradicts ``record_changing``.
"""
if admission is None:
return "records" if record_changing else "read"
if admission not in ADMISSIONS:
raise ValueError(f"Unknown job admission {admission!r}")
if record_changing is not None and record_changing != (admission == "records"):
raise ValueError(
f"record_changing={record_changing!r} contradicts admission {admission!r}"
)
return admission
def _relative_path(path) -> str:
"""``path`` as a root-relative POSIX path; an absolute one keeps its name."""
@@ -79,13 +135,23 @@ class Job:
"""One tracked operation: state, counters, outcomes, log and subscribers."""
def __init__(
self, kind, title, *, record_changing=False, scope=None, groups=(), clock=None
self,
kind,
title,
*,
admission=None,
record_changing=None,
cancellable=True,
scope=None,
groups=(),
clock=None,
):
self._clock = clock or time.time
self.id = uuid.uuid4().hex
self.kind = kind
self.title = title
self.record_changing = record_changing
self.admission = resolve_admission(admission, record_changing)
self.cancellable = bool(cancellable)
self.scope = dict(scope or {})
self.state = "running"
self.started_at = self._clock()
@@ -104,6 +170,10 @@ class Job:
self.task = None
self._subscribers = set()
@property
def record_changing(self):
return self.admission == "records"
@property
def running(self):
return self.state == "running"
@@ -234,6 +304,9 @@ class Job:
# What detail() lists, so a view re-reads it only when it changed.
"listed": {group: len(rows) for group, rows in self.outcomes.items()},
"log_length": len(self.log),
# A view offers Cancel only when the server can stop the job.
"cancellable": self.cancellable,
"admission": self.admission,
}
def detail(self):
@@ -285,27 +358,89 @@ class JobRegistry:
self._logger = logger_ or logger
self._jobs = {}
def start(self, kind, title, *, record_changing, scope, runner, groups=()) -> Job:
def start(
self,
kind,
title,
*,
admission=None,
record_changing=None,
cancellable=True,
scope=None,
runner,
groups=(),
) -> Job:
"""Create a job and run ``runner(job)`` as a task.
Raises JobBusy, naming the job in the way, when a job of ``kind`` is
running or when ``record_changing`` and another record-changing job
runs. Must be called from the event loop.
``admission`` is one of ``ADMISSIONS``; ``record_changing=True`` is
the older spelling of ``records`` and ``False`` of ``read``. One of
the two is required (TypeError otherwise), so no new job runs as
``read`` beside a scan by omission. Raises JobBusy, naming the job in
the way, when the admission class refuses it (see the module
docstring). Must be called from the event loop.
"""
if admission is None and record_changing is None:
raise TypeError(
f"JobRegistry.start({kind!r}) needs an admission class "
"(admission= or record_changing=)"
)
job = self._admit(
kind,
title,
admission=admission,
record_changing=record_changing,
cancellable=cancellable,
scope=scope,
groups=groups,
)
job.task = asyncio.ensure_future(self._run(job, runner))
return job
def hold(self, kind, title, *, admission, scope=None):
"""A running, non-cancellable job for a legacy-page operation.
Returns an async context manager. Entering it registers the job and
yields it, or raises JobBusy at once, without waiting, when the
admission class refuses it; nothing is registered until then, so a
hold that is never entered leaves no job behind. The job ends when
the block exits: ``finished``, or ``failed`` when it raised (unless
the body already ended it). A caller that answers a refusal in its
own way checks ``blocker(kind, admission=...)`` first, with no await
before the ``async with``.
"""
resolve_admission(admission) # an unknown class fails at the call
return _Hold(
lambda: self._admit(
kind, title, admission=admission, cancellable=False, scope=scope
)
)
def _admit(self, kind, title, *, admission, record_changing=None, **job_args):
resolved = resolve_admission(admission, record_changing)
blocker = self.blocker(kind, admission=resolved)
if blocker is not None:
raise JobBusy(blocker)
self._prune()
job = Job(kind, title, admission=resolved, **job_args)
self._jobs[job.id] = job
return job
def blocker(self, kind, *, admission=None, record_changing=None):
"""The running job a new ``kind`` job would be refused for, or None.
The same kind refuses first; then the oldest running job whose class
the new job's admission class may not run beside.
"""
same_kind = self.active(kind)
if same_kind is not None:
raise JobBusy(same_kind)
if record_changing:
for job in self._jobs.values():
if job.running and job.record_changing:
raise JobBusy(job)
self._prune()
job = Job(
kind, title, record_changing=record_changing, scope=scope, groups=groups
)
self._jobs[job.id] = job
job.task = asyncio.ensure_future(self._run(job, runner))
return job
return same_kind
refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)]
running = [job for job in self._jobs.values() if job.running]
running.sort(key=lambda job: job._started_seq)
for job in running:
if job.admission in refused:
return job
return None
async def _run(self, job, runner):
try:
@@ -349,11 +484,28 @@ class JobRegistry:
def cancel(self, job_id) -> bool:
"""Ask a running job to stop at its next safe boundary.
Returns whether a cancellation was requested: False for a finished or
unknown job, so a cancel after the end is a no-op.
Returns whether a cancellation was requested: False for a finished,
unknown or non-cancellable job, so a cancel after the end is a no-op.
"""
job = self._jobs.get(job_id)
if job is None or job.finished:
if job is None or job.finished or not job.cancellable:
return False
job.cancelled = True
return True
class _Hold:
"""The async context manager ``JobRegistry.hold`` returns."""
def __init__(self, admit):
self._admit = admit
self._job = None
async def __aenter__(self):
self._job = self._admit()
return self._job
async def __aexit__(self, exc_type, exc, tb):
if self._job.running:
self._job.finish("finished" if exc_type is None else "failed")
return False
+104 -20
View File
@@ -5,12 +5,47 @@ from pathlib import Path
from aiohttp import web
from ..config_store import ConfigUnreadable, RevisionConflict, get_config_store
from .jobs import LORA_IMPORT_JOB_KIND
# Flags of POST /lora/enable; each must be a JSON boolean when present.
_LORA_FLAGS = ("enabled", "trigger_words_enabled", "clear_civitai_api_key")
_LORA_PATH_ERROR = (
"LoraManager path must be a directory under ComfyUI's custom_nodes "
"whose name contains 'lora'"
)
def _lora_settings_mutation(values):
"""The config commit for a LoRA save: only ``integrations.lora_manager``.
The submitted keys replace their values; other LoRA keys, other
integrations and every other section stay as they are on disk.
"""
def mutate(config):
integrations = config.get("integrations")
integrations = dict(integrations) if isinstance(integrations, dict) else {}
lora = integrations.get("lora_manager")
lora = dict(lora) if isinstance(lora, dict) else {}
lora.update(values)
integrations["lora_manager"] = lora
config["integrations"] = integrations
return config
return mutate
def _lora_error_frame(message):
"""An import stream's error event, as the admin page's modal reads it.
The modal shows ``status`` and ``progress`` from every frame and closes
on ``type: error``, naming ``message``.
"""
return {"type": "error", "status": message, "progress": 0, "message": message}
class LoraIntegrationMixin:
"""Mixin providing LoraManager detection, scanning, and trigger word endpoints."""
@@ -131,14 +166,25 @@ class LoraIntegrationMixin:
return allowed, None
async def lora_enable(self, request):
"""Enable or disable LoraManager integration and save to config.json."""
"""Enable or disable LoraManager integration and save it to config.json.
The values are committed through the config store first (#220) and
published at runtime only once config.json holds them; a save that
cannot be written reports ``persisted: false`` and changes nothing.
The CivitAI key is never echoed back.
"""
try:
data = await request.json()
data, error = await self._read_json_body(request)
if error is not None:
return error
for flag in _LORA_FLAGS:
if flag in data and not isinstance(data[flag], bool):
return self._bad_lora_request(f"{flag} must be true or false")
enabled = data.get("enabled", False)
path = data.get("path", "")
trigger_words = data.get("trigger_words_enabled", False)
from ..config import IntegrationConfig, PromptManagerConfig
from ..config import IntegrationConfig
from ..lora_utils import get_trigger_cache
# The key is never echoed back to the UI, so a blank or absent
@@ -161,20 +207,40 @@ class LoraIntegrationMixin:
if error is not None:
return error
# Update in-memory config
IntegrationConfig.LORA_MANAGER_ENABLED = enabled
IntegrationConfig.LORA_MANAGER_PATH = path
IntegrationConfig.LORA_TRIGGER_WORDS_ENABLED = trigger_words
IntegrationConfig.CIVITAI_API_KEY = civitai_key
values = {
"enabled": enabled,
"path": path,
"trigger_words_enabled": trigger_words,
"civitai_api_key": civitai_key,
}
# Persist to the configured config.json
PromptManagerConfig.save_to_file()
def publish(_config):
# Under the commit lock, so the last save written is the one
# the runtime keeps.
IntegrationConfig.LORA_MANAGER_ENABLED = enabled
IntegrationConfig.LORA_MANAGER_PATH = path
IntegrationConfig.LORA_TRIGGER_WORDS_ENABLED = trigger_words
IntegrationConfig.CIVITAI_API_KEY = civitai_key
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_lora_settings_mutation(values),
expected_revision=None,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
return self._config_commit_failure(exc)
# Load trigger word cache if enabling
cache = get_trigger_cache()
if enabled and trigger_words and path:
count = await self._run_in_executor(cache.load, path)
self.logger.info(f"Trigger word cache loaded: {count} LoRAs")
try:
count = await self._run_in_executor(cache.load, path)
self.logger.info(f"Trigger word cache loaded: {count} LoRAs")
except Exception as e:
# Saved and enabled; only the trigger words are missing.
self.logger.warning(f"Trigger word cache could not be loaded: {e}")
cache.clear()
elif not enabled:
cache.clear()
@@ -184,6 +250,10 @@ class LoraIntegrationMixin:
"enabled": enabled,
"path": self._public_path(path),
"trigger_words_enabled": trigger_words,
"persisted": True,
"revision": revision,
"runtime_applied": True,
"cache_loaded": cache.is_loaded,
}
)
except Exception as e:
@@ -218,19 +288,33 @@ class LoraIntegrationMixin:
reason="OK",
headers={"Content-Type": "text/event-stream"},
)
await response.prepare(request)
async def send_progress(data):
line = f"data: {json.dumps(data)}\n\n"
await response.write(line.encode("utf-8"))
try:
await self._lora_scan_stream(allowed, send_progress)
except Exception as e:
self.logger.exception("LoRA scan failed")
await send_progress({"type": "error", "message": self._public_error(e)})
# The import replaces prompts and images: a records hold, so it
# never runs beside a scan, restore or tagging job.
busy = self._jobs.blocker(LORA_IMPORT_JOB_KIND, admission="records")
if busy is not None:
await response.prepare(request)
message = f"{busy.title} already running"
await send_progress(_lora_error_frame(message))
await response.write_eof()
return response
await response.write_eof()
async with self._jobs.hold(
LORA_IMPORT_JOB_KIND, "LoRA import (admin page)", admission="records"
) as job:
await response.prepare(request)
try:
await self._lora_scan_stream(allowed, send_progress)
except Exception as e:
self.logger.exception("LoRA scan failed")
job.finish("failed")
await send_progress(_lora_error_frame(self._public_error(e)))
await response.write_eof()
return response
except Exception as e:
+339
View File
@@ -0,0 +1,339 @@
"""Settings API routes: GET and POST /prompt_manager/settings (#220).
A save validates the payload, commits it to config.json through the config
store and publishes it at runtime inside that commit; a save that cannot be
written changes nothing at runtime and says so. ``_config_commit_failure``
is shared with the LoRA integration's save.
"""
import os
from aiohttp import web
from ..config_store import (
ConfigUnreadable,
RevisionConflict,
get_config_store,
)
from .admin import _current_image_monitor
try:
from ...utils.validators import validate_result_timeout, validate_worker_threads
except ImportError:
from utils.validators import validate_result_timeout, validate_worker_threads
# Settings owned by POST /settings: config attribute -> (section, key).
_SETTINGS_FILE_KEYS = {
"RESULT_TIMEOUT": ("web_ui", "result_timeout"),
"WEBUI_DISPLAY_MODE": ("web_ui", "webui_display_mode"),
"INFINITE_SCROLL": ("web_ui", "infinite_scroll"),
"WORKER_THREADS": ("performance", "worker_threads"),
}
def _section(config, name):
"""``config[name]`` as a fresh dict (a non-object section starts empty)."""
value = config.get(name)
return dict(value) if isinstance(value, dict) else {}
def _settings_mutation(updates, new_roots):
"""The config commit for a settings save: only the submitted keys change.
Every other section and key (integrations and the CivitAI key, database,
unknown sections) is kept as it is on disk.
"""
def mutate(config):
for name, value in updates.items():
section, key = _SETTINGS_FILE_KEYS[name]
merged = _section(config, section)
merged[key] = value
config[section] = merged
if new_roots is not None:
gallery = _section(config, "gallery")
monitoring = _section(gallery, "monitoring")
monitoring["directories"] = list(new_roots)
gallery["monitoring"] = monitoring
config["gallery"] = gallery
return config
return mutate
async def _read_json(request):
"""Parse a JSON object body through the package's capped reader."""
from . import _read_json_body
return await _read_json_body(request)
class SettingsRoutesMixin:
"""Mixin providing the settings read and save endpoints."""
def _register_settings_routes(self, routes):
@routes.get("/prompt_manager/settings")
async def get_settings_route(request):
return await self.get_settings(request)
@routes.post("/prompt_manager/settings")
async def save_settings_route(request):
return await self.save_settings(request)
async def get_settings(self, request):
"""Get current settings."""
try:
from ..config import PromptManagerConfig, GalleryConfig
root_paths = [
self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES
]
revision = await self._run_in_executor(self._config_revision)
return web.json_response(
{
"success": True,
**revision,
"settings": {
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
"worker_threads": PromptManagerConfig.WORKER_THREADS,
"cpu_count": PromptManagerConfig.max_worker_threads(),
"gallery_root_paths": root_paths,
"gallery_root_path": root_paths[0] if root_paths else "",
"monitored_directories": [
self._public_path(d) for d in self._monitored_directories()
],
},
}
)
except Exception as e:
self.logger.error(f"Get settings error: {e}", exc_info=True)
return web.json_response(
{"success": False, "error": "Failed to get settings"},
status=500,
)
def _monitored_directories(self):
"""Directories the running image monitor watches, else the configured roots."""
from ..config import GalleryConfig
monitor = _current_image_monitor()
if monitor is not None:
return list(getattr(monitor, "monitored_directories", []))
return list(GalleryConfig.MONITORING_DIRECTORIES)
async def save_settings(self, request):
"""Save settings: validate, commit to config.json, then publish (#220).
Nothing changes at runtime until config.json holds the new values, so
a save that cannot be written reports ``persisted: false`` and leaves
the running settings as they were. ``expected_revision`` (the
``revision`` from GET) refuses to overwrite a file changed meanwhile;
without it the payload merges into what is on disk (legacy page).
"""
try:
data, error_response = await _read_json(request)
if error_response is not None:
return error_response
expected_revision = data.get("expected_revision")
if expected_revision is not None and not isinstance(expected_revision, str):
return web.json_response(
{"success": False, "error": "expected_revision must be a string"},
status=400,
)
try:
updates = self._validate_settings(data)
except ValueError as ve:
return web.json_response(
{"success": False, "error": str(ve)}, status=400
)
new_roots, error = self._parse_gallery_roots(data)
if error:
return web.json_response({"success": False, "error": error}, status=400)
published = {}
def publish(_config):
# Under the commit lock: saves reach the runtime in the order
# they were written, so the runtime ends on the last one.
published.update(self._publish_settings(updates, new_roots))
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
return self._config_commit_failure(exc)
return web.json_response(
{
"success": True,
"message": "Settings saved successfully",
"restart_required": bool(published["restart_required_fields"]),
"persisted": True,
"revision": revision,
**published,
}
)
except Exception as e:
self.logger.error(f"Save settings error: {e}", exc_info=True)
return web.json_response(
{"success": False, "error": "Failed to save settings"},
status=500,
)
def _publish_settings(self, updates, new_roots):
"""Install committed settings at runtime (inside the config commit).
The values are already durable. A publication that fails is reported
as ``runtime_applied: false`` with the fields that need a restart;
nothing is retried or rolled back. Never raises.
"""
from ..config import PromptManagerConfig
for name, value in updates.items():
setattr(PromptManagerConfig, name, value)
restart_fields = []
runtime_applied = True
if new_roots is not None:
try:
if self._apply_gallery_roots(new_roots):
restart_fields.append("gallery_root_paths")
except Exception as e:
self.logger.error(
f"Saved gallery roots could not be applied: {e}", exc_info=True
)
runtime_applied = False
restart_fields.append("gallery_root_paths")
return {
"runtime_applied": runtime_applied,
"restart_required_fields": restart_fields,
}
def _config_commit_failure(self, exc):
"""The response for a config commit that wrote nothing."""
not_saved = {"success": False, "persisted": False, "runtime_applied": False}
if isinstance(exc, ConfigUnreadable):
self.logger.warning(f"Settings not saved: {exc}")
return web.json_response(
{
**not_saved,
"code": "config-unreadable",
"error": (
"config.json cannot be read, so nothing was saved. "
"Fix or remove the file, then save again."
),
},
status=409,
)
if isinstance(exc, RevisionConflict):
return web.json_response(
{
**not_saved,
"code": "settings-changed",
"error": "Settings changed since they were loaded. Reload them.",
"revision": exc.current,
},
status=409,
)
self.logger.error(f"Could not write config.json: {exc}")
return web.json_response(
{
**not_saved,
"code": "config-write-failed",
"error": "Settings could not be written to config.json.",
},
status=500,
)
@staticmethod
def _config_revision():
"""``{"revision": ...}``, or a null revision with ``config_error``."""
try:
_data, revision = get_config_store().read()
except ConfigUnreadable:
return {"revision": None, "config_error": "config-unreadable"}
return {"revision": revision}
@staticmethod
def _validate_settings(data):
"""Scalar settings of a payload as ``{CONFIG_ATTR: value}``.
Raises ValueError with a user-facing message on the first bad value;
nothing is applied until every value passed.
"""
from ..config import PromptManagerConfig
updates = {}
if "result_timeout" in data:
updates["RESULT_TIMEOUT"] = validate_result_timeout(data["result_timeout"])
if "webui_display_mode" in data:
mode = data["webui_display_mode"]
if mode not in ("popup", "newtab"):
raise ValueError("webui_display_mode must be 'popup' or 'newtab'")
updates["WEBUI_DISPLAY_MODE"] = mode
if "infinite_scroll" in data:
if not isinstance(data["infinite_scroll"], bool):
raise ValueError("infinite_scroll must be true or false")
updates["INFINITE_SCROLL"] = data["infinite_scroll"]
if "worker_threads" in data:
updates["WORKER_THREADS"] = validate_worker_threads(
data["worker_threads"], PromptManagerConfig.max_worker_threads()
)
return updates
def _parse_gallery_roots(self, data):
"""Extract and validate gallery roots from a settings payload.
Accepts ``gallery_root_paths`` (list, preferred) or the legacy single
``gallery_root_path`` string. Blank entries are dropped.
Returns:
(roots, error): ``roots`` is the validated list, or ``None`` when
the payload carries no gallery root key; ``error`` is a message
when validation failed (and ``roots`` is then ``None``).
"""
if "gallery_root_paths" in data:
raw = data["gallery_root_paths"]
if not isinstance(raw, list):
return None, "gallery_root_paths must be a list"
elif "gallery_root_path" in data:
raw = [data["gallery_root_path"]]
else:
return None, None
from ..config import GalleryConfig
roots = []
for entry in raw:
if not isinstance(entry, str):
return None, "Gallery root paths must be strings"
entry = entry.strip()
if not entry:
continue
ok, reason = GalleryConfig.validate_gallery_root(entry)
if not ok:
return (
None,
f"Invalid gallery root '{os.path.basename(entry)}': {reason}",
)
roots.append(GalleryConfig.resolve_gallery_root(entry))
return roots, None
def _apply_gallery_roots(self, roots):
"""Install validated gallery roots; returns True when they changed."""
from ..config import GalleryConfig
if roots == list(GalleryConfig.MONITORING_DIRECTORIES):
return False
GalleryConfig.MONITORING_DIRECTORIES = roots
self._cached_output_dir = None
self.reset_gallery_roots_cache()
return True
+207
View File
@@ -0,0 +1,207 @@
"""System routes: the read-only health check, the Overview and usage statistics.
- ``POST /prompt_manager/system/health`` starts the health check
(``database/health.py``) off-loop as a ``read`` job of kind ``health``: one
at a time, beside a scan, not cancellable and with no file outcomes. The
API keeps the last report for this session; ``GET`` returns it or null.
A report describes the database it was taken from: replacing the database
(a restore) forgets it, and a check that started before the swap is not
kept.
- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate.
Every value is real or null: there is no backup status until backups report
one, no health result until a check ran, and no Needs-review count until
the review queue exists. The missing-thumbnail count is not part of it:
the page reads ``/images/thumbnails/status``, which walks every source.
- ``GET /prompt_manager/system/statistics`` lists runs and ratings per
category, with the legacy ``/stats`` totals and the Overview's count of
rated images (images whose linked prompt is rated).
New routes answer errors with ``{success:false, code, error}``.
"""
import asyncio
import os
from collections import namedtuple
from aiohttp import web
from .job_routes import busy_response
from .jobs import HEALTH_JOB_KIND, JobBusy
HEALTH_JOB_TITLE = "Health check"
_HEALTH_STATUS_LINES = {
"passed": "Health check passed",
"issues": "Health check found issues",
"incomplete": "Health check incomplete",
}
# The last health report and the database generation it was taken from.
_StoredReport = namedtuple("_StoredReport", "generation report")
def _failure(message, status=500, code=None):
return web.json_response(
{"success": False, "code": code, "error": message}, status=status
)
def health_summary(report):
"""What the Overview shows of a report: when, the outcome, checks with issues."""
if report is None:
return None
return {
"checked_at": report["checked_at"],
"status": report["status"],
"issues": sum(1 for check in report["checks"] if check["status"] == "issues"),
}
class SystemRoutesMixin:
"""Mixin providing the /prompt_manager/system endpoints."""
def _register_system_routes(self, routes):
@routes.post("/prompt_manager/system/health")
async def start_health_check_route(request):
return await self.start_health_check(request)
@routes.get("/prompt_manager/system/health")
async def health_report_route(request):
return await self.get_health_report(request)
@routes.get("/prompt_manager/system/overview")
async def system_overview_route(request):
return await self.get_system_overview(request)
@routes.get("/prompt_manager/system/statistics")
async def system_statistics_route(request):
return await self.get_system_statistics(request)
# ── health ────────────────────────────────────────────────────────
def _health_roots(self):
"""The sources missing files are judged under.
The effective sources, plus every configured source that is not an
existing directory (an unmounted drive): the check then reports
missing files as unavailable instead of counting that source's links.
"""
from ..config import GalleryConfig
roots = [str(path) for path in self._get_all_output_dirs()]
for configured in GalleryConfig.MONITORING_DIRECTORIES:
try:
resolved = GalleryConfig.resolve_gallery_root(configured)
except (OSError, TypeError, ValueError):
continue
if not os.path.isdir(resolved):
roots.append(resolved)
return roots
def _database_replaced(self):
"""Forget what described the previous database (call after swapping ``db``)."""
self._db_generation += 1
self._health_report = None
def _current_health_report(self):
"""The last report, or None when none was kept for the current database."""
stored = self._health_report
generation = self._db_generation
if isinstance(stored, _StoredReport) and stored.generation == generation:
return stored.report
return None
async def start_health_check(self, request):
"""Start the health check job (202), or 409 naming the running check."""
generation = self._db_generation
db = self.db
try:
roots = await self._run_in_executor(self._health_roots)
except Exception:
self.logger.exception("Health check could not resolve the image sources")
return _failure("Failed to start the health check")
async def run(job):
job.update(status_line="Checking the database and image links")
report = await self._run_in_executor(db.run_health_checks, roots)
if generation == self._db_generation:
self._health_report = _StoredReport(generation, report)
else:
self.logger.info("Health report dropped: the database was replaced")
job.update(progress=100, status_line=_HEALTH_STATUS_LINES[report["status"]])
try:
job = self._jobs.start(
HEALTH_JOB_KIND,
HEALTH_JOB_TITLE,
admission="read",
cancellable=False,
scope={},
runner=run,
)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
async def get_health_report(self, request):
return web.json_response(
{"success": True, "report": self._current_health_report()}
)
# ── overview ──────────────────────────────────────────────────────
async def get_system_overview(self, request):
from ..config import GalleryConfig
try:
(output_dirs, auto_detected), counts = await asyncio.gather(
self._run_in_executor(self._output_dirs_with_origin),
self._run_in_executor(self.db.library_counts),
)
except Exception:
self.logger.exception("System overview failed")
return _failure("Failed to read the overview")
return web.json_response(
{
"success": True,
"library": {
"positive_prompts": counts["positive_prompts"],
"negative_variants": counts["negative_variants"],
},
"images": {
"indexed": counts["images"],
"rated": counts["rated_images"],
},
"health": health_summary(self._current_health_report()),
"backup": None, # no verified backup status exists yet (S8)
"needs_review": {"pending": None}, # no review queue yet (Q6)
"sources": {
"configured": len(GalleryConfig.MONITORING_DIRECTORIES),
"effective": len(output_dirs),
"auto_detected": auto_detected,
},
},
headers={"Cache-Control": "no-store"},
)
# ── statistics ────────────────────────────────────────────────────
async def get_system_statistics(self, request):
try:
by_category, totals, counts = await asyncio.gather(
self._run_in_executor(self.db.usage_by_category),
self._run_in_executor(self.db.get_statistics),
self._run_in_executor(self.db.library_counts),
)
except Exception:
self.logger.exception("System statistics failed")
return _failure("Failed to read statistics")
return web.json_response(
{
"success": True,
"by_category": by_category,
"totals": totals,
"rated_images": counts["rated_images"],
},
headers={"Cache-Control": "no-store"},
)
+197
View File
@@ -0,0 +1,197 @@
"""The one serialized writer of config.json (#220).
Every save that changes config.json goes through ``ConfigStore.commit``: the
settings page, the LoRA integration and, later, every System section. A
commit runs under one process-wide lock, re-reads the file inside it, applies
the caller's change to a copy and replaces the file atomically. So two saves
never overwrite each other's keys, sections this process does not know are
kept byte-for-byte in meaning, and a write that fails leaves the old file in
place and raises instead of being logged and reported as saved.
A revision names the file content a client last saw: ``"sha256:<hex>"`` of
the file bytes, or ``"absent"`` when there is no file. A commit that names a
revision refuses to write over different content (``RevisionConflict``).
Reads take the same lock, so a read handle is never open while the file is
replaced (Windows refuses that), and a caller's ``publish`` runs inside the
lock right after its write, so runtimes publish saves in the order they were
written. A symlinked config.json is written beside its real target.
config.json can hold the CivitAI key, so the file is written owner-only.
"""
import copy
import hashlib
import json
import os
import tempfile
import threading
import time
from typing import Callable, Optional, Tuple
ABSENT_REVISION = "absent"
# One lock for the process: every ConfigStore writes the same file. It is
# re-entrant so a commit's ``publish`` callback may read the config again.
_COMMIT_LOCK = threading.RLock()
# Windows: an antivirus, indexer or backup handle on config.json makes
# os.replace fail with PermissionError for a moment, so it is retried briefly.
_REPLACE_ATTEMPTS = 5 if os.name == "nt" else 1
_REPLACE_BACKOFF_SECONDS = 0.05
class ConfigUnreadable(Exception):
"""config.json exists but is not a readable JSON object; it is left alone."""
class RevisionConflict(Exception):
"""config.json changed since the revision the caller saw."""
def __init__(self, current: str):
super().__init__("config.json changed since it was read")
self.current = current
def revision_of(data: Optional[bytes]) -> str:
"""The revision of a file's bytes; ``None`` means the file is absent."""
if data is None:
return ABSENT_REVISION
return "sha256:" + hashlib.sha256(data).hexdigest()
def _default_path() -> str:
from .config import PromptManagerConfig
return PromptManagerConfig.get_config_path()
def _replace(source: str, target: str) -> None:
"""``os.replace`` with a short retry on a transient Windows PermissionError."""
for attempt in range(1, _REPLACE_ATTEMPTS + 1):
try:
os.replace(source, target)
return
except PermissionError:
if attempt == _REPLACE_ATTEMPTS:
raise
time.sleep(_REPLACE_BACKOFF_SECONDS)
def _write_atomically(path: str, payload: bytes) -> None:
"""Replace ``path`` with ``payload``: temp file beside it, fsync, replace.
The descriptor is closed before ``os.replace`` (Windows refuses to replace
over an open handle), and on any failure the temp file is closed and
removed, so a failed write leaves the old file and no stray temp file.
``path`` is resolved first: a symlink keeps pointing at the updated
target. A single-file bind mount cannot be replaced (EBUSY) and fails as
an OSError, which callers report as not saved.
"""
path = os.path.realpath(path)
parent = os.path.dirname(path) or "."
os.makedirs(parent, exist_ok=True)
fd, temp_path = tempfile.mkstemp(dir=parent, prefix=".config-", suffix=".tmp")
try:
try:
handle = os.fdopen(fd, "wb")
except BaseException:
os.close(fd)
raise
with handle: # closed before the replace, also when a write fails
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
if os.name == "posix":
os.chmod(temp_path, 0o600)
_replace(temp_path, path)
except BaseException:
try:
os.unlink(temp_path)
except OSError:
pass
raise
class ConfigStore:
"""Serialized, atomic reads and commits of config.json."""
def __init__(self, path_getter: Optional[Callable[[], str]] = None):
self._path_getter = path_getter or _default_path
@property
def path(self) -> str:
"""The config file this store reads and writes (resolved per call)."""
return self._path_getter()
def read(self) -> Tuple[dict, str]:
"""``(config, revision)``; a missing file is ``({}, "absent")``.
Raises ConfigUnreadable when the file cannot be read, is not JSON, or
is not a JSON object. Waits for a running commit (see the module
docstring).
"""
with _COMMIT_LOCK:
return self._read(self.path)
@staticmethod
def _read(path: str) -> Tuple[dict, str]:
try:
with open(path, "rb") as handle:
raw = handle.read()
except (FileNotFoundError, NotADirectoryError):
# No file there (a parent that is a file cannot hold one either);
# a commit then fails on the write, as config-write-failed.
return {}, ABSENT_REVISION
except OSError as exc:
raise ConfigUnreadable(f"config.json cannot be read: {exc}") from exc
try:
data = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, ValueError) as exc:
raise ConfigUnreadable("config.json is not valid JSON") from exc
if not isinstance(data, dict):
raise ConfigUnreadable("config.json is not a JSON object")
return data, revision_of(raw)
def commit(
self,
mutate: Callable[[dict], dict],
*,
expected_revision: Optional[str],
publish: Optional[Callable[[dict], None]] = None,
) -> Tuple[dict, str]:
"""Apply ``mutate`` to the current config and write the result.
``mutate`` receives a deep copy of the file's content and returns the
whole new config (an object). ``expected_revision`` None merges into
whatever is on disk (legacy partial updates); a revision refuses to
write over different content. Returns ``(config, revision)`` as
written. Raises ConfigUnreadable, RevisionConflict, TypeError (a
non-object result) or OSError (the write failed; the old file stays).
``publish(config)``, when given, runs under the lock once the file
holds the new config, so concurrent saves reach the runtime in the
order they were written. It must handle its own errors: the values
are already durable when it runs.
"""
with _COMMIT_LOCK:
path = self.path
current, revision = self._read(path)
if expected_revision is not None and expected_revision != revision:
raise RevisionConflict(revision)
candidate = mutate(copy.deepcopy(current))
if not isinstance(candidate, dict):
raise TypeError("A config commit must produce a JSON object")
payload = json.dumps(candidate, indent=2).encode("utf-8")
_write_atomically(path, payload)
if publish is not None:
publish(candidate)
return candidate, revision_of(payload)
_store = ConfigStore()
def get_config_store() -> ConfigStore:
"""The process's config store (on ``PromptManagerConfig.get_config_path``)."""
return _store
+31 -5
View File
@@ -249,14 +249,16 @@ const EXPECTED_JOB_VIEW = {
pressed: { border: true, background: true, added: true, bad: true, warn: true },
};
async function openRunningJob(page, request, theme, viewport = { width: 1600, height: 1100 }) {
const SCAN_START = { start: { total: 19622 }, step: { processed: 8240,
counters: { known: 6511, review: 37, no_metadata: 254, errors: 18 },
records: [{ group: 'added', path: 'CCC/mabes9sans-dataset-v1/IMG_GEN_00369_.png', detail: 'Added to the Library as “Mabes portrait”' }] } };
async function openRunningJob(page, request, theme, viewport = { width: 1600, height: 1100 }, data = SCAN_START) {
await page.setViewportSize(viewport);
const started = await request.post('/__test__/jobs/start', { data: { total: 19622 } });
const started = await request.post('/__test__/jobs/start', { data: data.start });
await expect(started).toBeOK();
const { job } = await started.json();
await expect(await request.post(`/__test__/jobs/${job.id}/step`, { data: { processed: 8240,
counters: { known: 6511, review: 37, no_metadata: 254, errors: 18 },
records: [{ group: 'added', path: 'CCC/mabes9sans-dataset-v1/IMG_GEN_00369_.png', detail: 'Added to the Library as “Mabes portrait”' }] } })).toBeOK();
await expect(await request.post(`/__test__/jobs/${job.id}/step`, { data: data.step })).toBeOK();
await openProduction(page, `activity/${job.id}`, theme);
await expect(page.locator('#workspace-main .file-row')).toHaveCount(1);
return job;
@@ -278,6 +280,30 @@ for (const theme of THEMES) test(`job view follows the approved mockup ${theme}`
expect(await page.evaluate(measureJobView)).toEqual(EXPECTED_JOB_VIEW);
});
// D89 (plan D66): an image tagging job's view is the same mockup with its own four tiles (Tagged green, Unchanged,
// Skipped warning, Errors red) in the mockup's five-column row, its own outcome list and "Tagging log".
const TAG_START = { start: { kind: 'image-tags', title: 'Auto-tag images', mode: 'auto', images: 4, total: 4 }, step: { processed: 2,
counters: { unchanged: 1, errors: 1 }, records: [{ group: 'tagged', path: 'CCC/mabes9sans-dataset-v1/IMG_GEN_00369_.png', detail: 'Added 3 tags' }] } };
const EXPECTED_TAG_VIEW = {
...EXPECTED_JOB_VIEW,
progress: [MOCKUP_PROGRESS[0], MOCKUP_PROGRESS[1], 'dl.stat-row(div.added.stat,div.stat,div.stat.warn,div.bad.stat)'],
summary: '● Tagging log — raw detail, click to show',
};
for (const theme of THEMES) test(`image tagging job view follows the approved mockup ${theme}`, async ({ page, request }) => {
await seedConcept(request);
await openRunningJob(page, request, theme, undefined, TAG_START);
expect(await page.evaluate(measureJobView)).toEqual(EXPECTED_TAG_VIEW);
await expect(page.locator('#workspace-main .outcome-list > button')).toHaveText(['Tagged1', 'Errors1', 'Skipped0', 'Unchanged1']);
// Mutation check: the scan's five tiles and the scan's log label are both noticed.
expect(diffInventory(EXPECTED_JOB_VIEW, await page.evaluate(measureJobView)).map(item => item.path).sort())
.toEqual(['progress.2', 'summary']);
const undoCss = await mutate(page, { css: '#workspace-main .stat-row .stat.warn dd { color: red; }' });
expect(diffInventory(EXPECTED_TAG_VIEW, await page.evaluate(measureJobView)).map(item => item.path)).toEqual(['pressed.warn']);
await undoCss();
expect(await page.evaluate(measureJobView)).toEqual(EXPECTED_TAG_VIEW);
});
test('job view fits a phone without page scroll', async ({ page, request }) => {
await seedConcept(request);
await openRunningJob(page, request, 'dark', { width: 390, height: 844 });
+23 -4
View File
@@ -78,6 +78,17 @@ export async function seedConcept(request,data={}) {
return response.json();
}
/**
* Seed the concept fixture and fix the browser's clock at the fixture's own instant (`manifest.now`), so the page and
* the fixture read one "now": a prompt used now reads "Used today" and the caption names the fixture's day even when
* the run crosses midnight between seeding and rendering. Timers keep running; only `Date` stands still.
*/
export async function seedConceptAt(page,request,data={}) {
const manifest=await seedConcept(request,data);
await page.clock.setFixedTime(new Date(manifest.now*1000));
return manifest;
}
/**
* Save `tags` as the current image tag set of the listed file at `path` (root 0) through the real observe and tag
* routes (#249 PR C), releasing the capture afterwards; returns the receipt's saved names.
@@ -138,10 +149,12 @@ export async function hoverStateOf(page,locator) {
/**
* Mutate the open page so a comparison can be shown to notice a change (the A7 mutation check): `css` is appended as
* the last stylesheet, so it wins over equally specific concept rules, and every element matching a `remove` selector
* is detached. Returns an async undo that restores the stylesheet and the removed elements in place.
* is detached; each `clone` ({selector, text}) inserts a copy of the first match after it, holding `text` and no
* `data-action` (a control production adds beside a concept one, D85). Returns an async undo that restores the
* stylesheet and the removed elements in place and drops the copies.
*/
export async function mutate(page,{css=null,remove=[]}={}) {
const token=await page.evaluate(({css,remove})=>{
export async function mutate(page,{css=null,remove=[],clone=[]}={}) {
const token=await page.evaluate(({css,remove,clone})=>{
const token=`mutation-${Math.random().toString(36).slice(2)}`;
const undo=[];
if(css) {
@@ -156,9 +169,15 @@ export async function mutate(page,{css=null,remove=[]}={}) {
node.remove();undo.push(()=>parent.insertBefore(node,next));
}
}
for(const {selector,text} of clone) {
const node=document.querySelector(selector);
if(!node)throw new Error(`mutate: nothing matches ${selector}`);
const copy=node.cloneNode(true);copy.textContent=text;copy.removeAttribute('data-action');
node.after(copy);undo.push(()=>copy.remove());
}
(window.__conceptMutations??=new Map()).set(token,undo);
return token;
},{css,remove});
},{css,remove,clone});
return ()=>page.evaluate(token=>{
const undo=window.__conceptMutations?.get(token);
if(!undo)throw new Error('mutate: this mutation was already undone or the page reloaded');
+5 -2
View File
@@ -237,9 +237,12 @@ const viewerStructure=details=>details.evaluate(node=>{
test('viewer details hold provenance, raw records and downloads',async({page,request})=>{
await page.setViewportSize({width:1600,height:1100});const reads=await patchInspection(page);
await fullFixture(page,request);let details=await viewerDetails(page,'complete.png');
// The panel content without the cover, then the two disclosures, then the pair send and history buttons (D20).
// The panel content without the cover, D117's Copy settings summary after the facts, then the two disclosures,
// then the pair send and history buttons (D20).
expect(await viewerStructure(details)).toEqual({outer:['div','button.send','button'],
inner:['div.panel-heading','p.small','div.tags','section.prompt-section.positive','section.prompt-section.negative','div.tags','dl.meta-grid','details','details']});
inner:['div.panel-heading','p.small','div.tags','section.prompt-section.positive','section.prompt-section.negative','div.tags','dl.meta-grid','button.settings-summary','details','details']});
await expect(details.locator(':scope > div > button')).toHaveText('Copy settings summary');
await expect(details.locator(':scope > div > button.settings-summary')).toHaveCSS('white-space','normal');
await expect(details.locator('.detail-cover,.detail-inner,.detail-footer')).toHaveCount(0);
await expect(details.locator('.panel-heading h2')).toHaveText(POSITIVE_TITLE);await expect(details.locator('.panel-heading .chip')).toHaveText('PNG');
await expect(details.locator(':scope > div > p.small')).toHaveText('complete.png');
+20 -13
View File
@@ -1,7 +1,7 @@
import {writeFile} from 'node:fs/promises';
import {test,expect} from './fixtures.mjs';
import {captureCanonicalState} from './history-canonical-comparison.mjs';
import {SIZES,THEMES,MOTIONS,seedConcept,openConcept,openProduction,inventoryOf,diffInventory,mutate,stripExact,countCopy,reflow,withoutItems} from './concept-inventory.mjs';
import {SIZES,THEMES,MOTIONS,seedConceptAt,openConcept,openProduction,inventoryOf,diffInventory,mutate,stripExact,countCopy,reflow,withoutItems} from './concept-inventory.mjs';
// Task C7 of #249: the production Images page against the immutable concept, on the concept-shaped real fixture (24
// images of the concept's six prompts, its seeds, sizes and favorites). Heading, toolbar, filter row, caption, grid,
@@ -30,9 +30,11 @@ function captionDay(seconds,now=new Date()) {
return `${name} · ${date.toLocaleDateString('en-US',options)}`;
}
/** D8: the selection bar's real actions only; Add tags, Add to collection and Remove links… are left out. */
const OMITTED_BULK=Object.freeze(['Add tags','Add to collection','Remove links…']);
const OMITTED_BULK_SELECTORS=Object.freeze(['bulk-tags','bulk-collection','bulk-remove'].map(action=>`#bulkbar [data-action="${action}"]`));
/** D8 (amended by #249 PR D): the selection bar's real actions only; Add to collection and Remove links… are left out. */
const OMITTED_BULK=Object.freeze(['Add to collection','Remove links…']);
const OMITTED_BULK_SELECTORS=Object.freeze(['bulk-collection','bulk-remove'].map(action=>`#bulkbar [data-action="${action}"]`));
/** D85: Auto-tag selected follows the concept's Add tags, a concept button of its own text. */
const AUTO_TAG_CLONE=Object.freeze([{selector:'#bulkbar [data-action="bulk-tags"]',text:'Auto-tag selected'}]);
/**
* `withoutItems` for a row with padding and a border (the `.bulkbar`): its items flow again inside the content box,
@@ -83,7 +85,7 @@ function listInfo(card,actionsWidth) {
/** Recorded Images deviations only (spec §8); everything else in the concept inventory must equal production. */
const nocase=(a,b)=>{const x=a.toLowerCase(),y=b.toLowerCase();return x<y?-1:x>y?1:a<b?-1:a>b?1:0;};
export function mapImagesConcept(reference,{manifest,now=new Date()}) {
export function mapImagesConcept(reference,{manifest,now=new Date(manifest.now*1000)}) {
const paths=new Map(manifest.images.map(image=>[image.filename,image.path]));
// D14: the sort appends production's two sorts. Save view is the concept's own (saved views, #249 D16 retired).
const toolbar={...reference.toolbar};
@@ -316,7 +318,7 @@ async function behaviorsOf(page,scope) {
return {hover,opened,viewerTitle,focusAfterViewer,selected,layouts,search,single};
}
test.beforeEach(async({request},testInfo)=>{testInfo.manifest=await seedConcept(request);});
test.beforeEach(async({page,request},testInfo)=>{testInfo.manifest=await seedConceptAt(page,request);});
for(const [width,height] of SIZES)for(const theme of THEMES)test(`images page geometry, tokens, controls and behaviors match the concept ${width}x${height} ${theme}`,async({page,request},testInfo)=>{
test.setTimeout(240_000);
@@ -346,7 +348,7 @@ for(const [width,height] of SIZES)for(const theme of THEMES)test(`images page ge
importDiff:diffInventory(mapImportConcept(referenceImport),actualImport),referenceBehaviors,actualBehaviors};
}
// Load more in production: a fixture past one listing page appends one batch and shows the concept's toast.
testInfo.manifest=await seedConcept(request,{extra_images:LOAD_MORE_EXTRA});
testInfo.manifest=await seedConceptAt(page,request,{extra_images:LOAD_MORE_EXTRA});
for(const motion of MOTIONS) {
const total=FIXTURE_IMAGES+LOAD_MORE_EXTRA;
await openProductionLoaded(page,theme,motion,total);
@@ -443,6 +445,10 @@ for(const [width,height] of SIZES)for(const theme of THEMES)test(`images selecti
const restore=await mutate(page,{remove:[...OMITTED_BULK_SELECTORS]});
const removed=(await inventoryOf(page,'#main')).bulkbar;
await restore();
// The oracle for production's bar: the concept page itself without the omitted actions and with D85's button.
const restoreProduction=await mutate(page,{remove:[...OMITTED_BULK_SELECTORS],clone:AUTO_TAG_CLONE});
const oracle=await inventoryOf(page,'#main');
await restoreProduction();
await captureCanonicalState(page,testInfo,`concept-images-selected-${width}-${theme}`);
await openProductionLoaded(page,theme,'no-preference');
const emptyActual=await inventoryOf(page,'#workspace-main');
@@ -454,12 +460,13 @@ for(const [width,height] of SIZES)for(const theme of THEMES)test(`images selecti
expect.soft(reference.bulkbar.bar.items.map(item=>item.text),'concept bar').toEqual(['1 selected','Add tags','Set category',
'Add to collection','Export metadata','Remove links…','Clear']);
expect.soft(diffInventory(stripExact(mapBulkbar(reference.bulkbar,0).bulkbar.bar),stripExact(removed.bar)),'bar without the omitted actions').toEqual([]);
expect.soft(actual.bulkbar.bar.items.map(item=>item.text),'production bar').toEqual(['1 selected','Set category','Export metadata','Clear']);
expect.soft(actual.bulkbar.bar.items.map(item=>item.text),'production bar').toEqual(['1 selected','Add tags','Auto-tag selected',
'Set category','Export metadata','Clear']);
expect.soft(compare(empty,emptyActual,manifest,['bulkbar','cards','panel']),'nothing checked').toEqual([]);
expect.soft(compare(reference,actual,manifest,['bulkbar','cards','panel']),'one card checked').toEqual([]);
expect.soft(compare(oracle,actual,manifest,['bulkbar','cards','panel']),'one card checked').toEqual([]);
// The bar comparison notices a change to the production bar.
await mutate(page,{css:'.bulkbar { padding: 4px }'});
expect(compare(reference,await inventoryOf(page,'#workspace-main'),manifest,['bulkbar']).map(item=>item.path)).toContain('bulkbar.bar.paddingTop');
expect(compare(oracle,await inventoryOf(page,'#workspace-main'),manifest,['bulkbar']).map(item=>item.path)).toContain('bulkbar.bar.paddingTop');
});
// The comparison is only worth something if it notices a change (spec §7 mutation check): one CSS mutation (the
@@ -507,7 +514,7 @@ test('one appended image reads in the singular',async({page,request})=>{
test.setTimeout(120_000);
await page.setViewportSize({width:1600,height:1100});
const total=PRODUCTION_BATCH+1;
await seedConcept(request,{extra_images:total-FIXTURE_IMAGES});
await seedConceptAt(page,request,{extra_images:total-FIXTURE_IMAGES});
await openProductionLoaded(page,'dark','no-preference',total);
await expect(page.locator('.infinite-continuation > p').first()).toHaveText(countCopy(`${PRODUCTION_BATCH} of ${total} images · more load as you scroll`));
// A dispatched press: scrolling the button into view lets the page's own observer load the one image first.
@@ -523,7 +530,7 @@ test('one image appended by scrolling reads in the singular',async({page,request
test.setTimeout(120_000);
await page.setViewportSize({width:1600,height:1100});
const total=PRODUCTION_BATCH+1;
await seedConcept(request,{extra_images:total-FIXTURE_IMAGES});
await seedConceptAt(page,request,{extra_images:total-FIXTURE_IMAGES});
await openProductionLoaded(page,'dark','no-preference',total);
await expect(page.locator('#result-count')).toHaveText(new RegExp(`^${PRODUCTION_BATCH} of ${total} images`));
await page.locator('.infinite-continuation').scrollIntoViewIfNeeded();
@@ -544,7 +551,7 @@ test('one footer height for the collection (D3)',async({page,request})=>{
const reference=await inventoryOf(page,'#main');
expect(reference.grid.columns).toBe(2);
expect(reference.cards.every(card=>card.info.p.height>parseFloat(card.info.p.lineHeight))).toBe(true);
const manifest=await seedConcept(request,{seedless_images:2,short_seedless:true});
const manifest=await seedConceptAt(page,request,{seedless_images:2,short_seedless:true});
await openProductionLoaded(page,'dark','no-preference',FIXTURE_IMAGES+2);
const actual=await inventoryOf(page,'#workspace-main');
const box=card=>({left:card.left,top:card.top,width:card.width,height:card.height});
+10 -5
View File
@@ -117,7 +117,7 @@ function measureViewer() {
['dl.meta-grid', node => node.closest('details') ? null : 'facts'],
['details > summary', node => text(node)],
['button.send', node => node.closest('.prompt-section') ? `send ${node.closest('.prompt-section').matches('.positive') ? 'positive' : 'negative'}` : 'send pair'],
['button', node => /Prompt history/.test(node.textContent) ? 'Prompt history' : null],
['button', node => /Prompt history/.test(node.textContent) ? 'Prompt history' : text(node) === 'Copy settings summary' ? 'settings summary' : null],
];
const sections = [...rail.querySelectorAll('*')].map(node => {
for (const [selector, label] of SECTIONS) if (node.matches(selector)) return label(node);
@@ -158,13 +158,18 @@ async function withProductionCopy(page) {
}, { COPY, ACCEPT });
}
/** D66: the viewer of a local file is titled by its filename and has no tag chips and no Prompt history. */
/**
* D66: the viewer of a local file is titled by its filename and has no tag chips and no Prompt history. D117 (LI-4):
* "Copy settings summary" follows the six facts.
*/
function mapViewerConcept(reference) {
const sections = reference.sections.filter(section => section !== 'tag chips' && section !== 'Prompt history');
const facts = sections.indexOf('facts');
return {
...reference,
title: FILE,
railTitle: FILE,
sections: reference.sections.filter(section => section !== 'tag chips' && section !== 'Prompt history'),
sections: [...sections.slice(0, facts + 1), 'settings summary', ...sections.slice(facts + 1)],
};
}
@@ -269,9 +274,9 @@ for (const [width, height] of SIZES) for (const theme of THEMES) {
// The chooser names the chosen file after a real setInputFiles, as the concept's does.
expect(production.initial.chosen).toEqual({ name: '', files: [] });
expect(production.selected.chosen).toEqual({ name: FILE, files: [FILE] });
// D66 in the rail: no tag chips and no Prompt history; the title is the filename.
// D66 in the rail: no tag chips and no Prompt history; the title is the filename. D117 follows the facts.
expect(production.viewer.sections).toEqual(['heading', 'filename', 'badges', 'positive', 'send positive', 'negative',
'send negative', 'facts', 'File & provenance', 'Workflow & raw metadata', 'send pair']);
'send negative', 'facts', 'settings summary', 'File & provenance', 'Workflow & raw metadata', 'send pair']);
});
}
+19 -7
View File
@@ -45,10 +45,10 @@ function withCategoryScope(inventory) {
}
/**
* Recorded inspector deviations only (spec §8, TESTING.md D3, D20–D22, D34 and D38); everything else must equal
* production. `reference.viewerPlain` is the concept's own full details
* with its tag chips made plain (D20) and, under a coarse pointer, D34's targets, both at runtime: the browser lays out
* what follows them, as the Images spec's removed toolbar items do.
* Recorded inspector deviations only (spec §8, TESTING.md D3, D20–D22, D34, D38 and D117); everything else must
* equal production. `reference.viewerPlain` is the concept's own full details
* with its tag chips made plain (D20), D117's summary button added and, under a coarse pointer, D34's targets, all at
* runtime: the browser lays out what follows them, as the Images spec's removed toolbar items do.
*/
export function mapInspectorConcept(reference,{viewport}) {
const {panel,viewerPlain,fileControls}=reference;
@@ -105,7 +105,11 @@ async function openProductionPanel(page,theme) {
await settle(page);
}
/** The concept's tag chips in the open viewer as plain `span.chip`s (D20) and D34's coarse targets; returns an undo. */
/**
* The concept's tag chips in the open viewer as plain `span.chip`s (D20), D34's coarse targets and D117's "Copy
* settings summary" after the six facts (LI-4), drawn as the concept's own plain button (its "Download sample JSON",
* without its action); returns an undo.
*/
async function plainViewerChips(page) {
const restoreTargets=await mutate(page,{css:D34_COARSE_TARGETS});
await page.evaluate(()=>{
@@ -113,10 +117,15 @@ async function plainViewerChips(page) {
const span=document.createElement('span');span.className='chip';span.textContent=button.textContent;
button.replaceWith(span);return [span,button];
});
const summary=document.querySelector('#viewer-details [data-action="download-workflow"]').cloneNode(false);
for(const key of Object.keys(summary.dataset))delete summary.dataset[key];
summary.textContent='Copy settings summary';summary.classList.add('settings-summary');
document.querySelector('#viewer-details > div > dl.meta-grid').after(summary);window.__summaryButton=summary;
});
await settle(page);
return async()=>{
await page.evaluate(()=>{for(const [span,button] of window.__plainChips)span.replaceWith(button);delete window.__plainChips;});
await page.evaluate(()=>{for(const [span,button] of window.__plainChips)span.replaceWith(button);delete window.__plainChips;
window.__summaryButton.remove();delete window.__summaryButton;});
await restoreTargets();
};
}
@@ -342,7 +351,7 @@ for(const [width,height] of SIZES)test(`image tags saved on the concept image ke
// The comparison is only worth something if it notices a change (spec §7 mutation check): on the panel one CSS
// mutation (the role label's letter spacing) and one DOM mutation (the footer removed); in the viewer (final review m5)
// its pair actions removed and D38's Show in folder removed; each applied after an exact match.
// its pair actions removed, D38's Show in folder removed and D117's summary removed; each applied after an exact match.
test('inspector comparison detects mutations',async({page})=>{
test.setTimeout(120_000);
const viewport={width:1600,height:1100};
@@ -366,6 +375,9 @@ test('inspector comparison detects mutations',async({page})=>{
expect(()=>assertInspectorMatch(reference,withoutActions,viewport)).toThrow(/viewerDetails\.actions/);
const withoutReveal=await viewerMutated({remove:['#viewer-details button[aria-label$=" in folder"]']});
expect(()=>assertInspectorMatch(reference,withoutReveal,viewport)).toThrow(/fileControls/);
// D117: the summary action after the facts is part of the mapped concept, so losing it is a difference.
const withoutSummary=await viewerMutated({remove:['#viewer-details > div > button']});
expect(()=>assertInspectorMatch(reference,withoutSummary,viewport)).toThrow(/viewerDetails\.structure\.inner/);
assertInspectorMatch(reference,await viewerMutated(),viewport);
await mutate(page,{remove:['#image-inspector > .detail-footer']});
+26 -3
View File
@@ -1,7 +1,7 @@
import {writeFile} from 'node:fs/promises';
import {test,expect} from './fixtures.mjs';
import {captureCanonicalState} from './history-canonical-comparison.mjs';
import {SIZES,THEMES,MOTIONS,seedConcept,openConcept,openProduction,inventoryOf,diffInventory,mutate,stripExact,countCopy,reflow} from './concept-inventory.mjs';
import {SIZES,THEMES,MOTIONS,seedConceptAt,openConcept,openProduction,inventoryOf,diffInventory,mutate,stripExact,countCopy,reflow} from './concept-inventory.mjs';
// Task E3 of #249: the production Prompt library shelf against the immutable concept (app.js renderLibrary a572–586,
// style.css s648–708), on the concept-shaped real fixture (the concept's six prompts with their negatives, categories,
@@ -37,7 +37,8 @@ export function mapLibraryConcept(reference) {
// D43: tag chips come in the store's alphabetical order; the row flows again from the concept's own chips.
tags:sortedTags(card.tags),
// D23: the stars are aria-hidden, with a visually hidden "Rated N out of 5" spoken in their place. The fixture's
// run_count (4) and last_used_at (now) give the concept's own "4 runs" and "Used today". The two-line clamp is
// run_count (4) and last_used_at (its own "now", where the page's clock stands) give the concept's own "4 runs"
// and "Used today". The two-line clamp is
// applied below through `clampedCards`, measured on the concept with `CLAMP_CSS` (its samples wrap at 390px).
footer:{...card.footer,facts:{...card.footer.facts,spoken:card.footer.facts.spoken.replace(/^(★+)/,stars=>`Rated ${stars.length} out of 5`)}}}));
return {...reference,library:{...library,toolbar,
@@ -238,7 +239,8 @@ async function openHost(context) {
}
test.beforeEach(async({page,request},testInfo)=>{
testInfo.manifest=await seedConcept(request);
// The page's clock stands at the fixture's own "now", so its last use reads "Used today" on either side of midnight.
testInfo.manifest=await seedConceptAt(page,request);
// Both documents copy through the clipboard; the stub records what they wrote.
await page.addInitScript(()=>{window.copied=[];Object.defineProperty(navigator,'clipboard',{configurable:true,value:{writeText:async text=>{window.copied.push(text);}}});});
});
@@ -322,6 +324,27 @@ for(const theme of THEMES)test(`library grid and cards match the concept 1800x11
expect(compare(reference,actual,{sections:['library'],library:['grid','cards','trailing']})).toEqual([]);
});
// PR #284's CI run: the fixture recorded the last use at 23:59:5x UTC and the shelf rendered at 00:00:05, so production
// read "Used yesterday" against the concept's "Used today". Here the browser's wall clock starts at 23:59:58 UTC, the
// fixture records its use at that instant, and midnight passes before either document opens.
const BEFORE_MIDNIGHT=Date.UTC(2026,9,22,23,59,58);
const PAST_MIDNIGHT_MS=5_000;
test('a run that crosses midnight still reads the concept fixture\'s day',async({page,request})=>{
await page.setViewportSize({width:1800,height:1100});
await page.clock.install({time:BEFORE_MIDNIGHT});
const manifest=await seedConceptAt(page,request,{now:BEFORE_MIDNIGHT/1000});
expect(manifest.now*1000,'the fixture\'s now').toBe(BEFORE_MIDNIGHT);
const used=(await (await request.get('/prompt_manager/library')).json()).items.map(item=>Date.parse(item.last_used_at));
expect(used,'every last use at the fixture\'s now').toEqual(Array(CONCEPT_PROMPTS).fill(BEFORE_MIDNIGHT));
await page.clock.fastForward(PAST_MIDNIGHT_MS);
await openConceptLibrary(page,'dark','no-preference');
const reference=await conceptInventory(page);
await openProductionLibrary(page,'dark','no-preference');
const actual=await inventoryOf(page,'#workspace-main');
expect(actual.library.cards.map(card=>card.footer.facts.text.replace(/^★+ · /,''))).toEqual(Array(CONCEPT_PROMPTS).fill('Used today'));
expect(compare(reference,actual,{sections:['library'],library:['grid','cards','trailing']})).toEqual([]);
});
// The comparison is only worth something if it notices a change (spec §7 mutation check): one CSS mutation (the footer
// padding) and one DOM mutation (the negative-variants chip removed), each applied after an exact match.
test('library comparison detects mutations',async({page})=>{
+106
View File
@@ -0,0 +1,106 @@
"""Hold real requests at this test server, so a page can leave while one is in flight.
A Playwright ``page.route`` hold answers inside the browser's interception layer,
which WebKit cancels differently from a request the server has not answered yet.
These commands hold the request at the server itself:
- ``POST /__test__/hold`` with ``{"path": "/prompt_manager/..."}`` holds every
later request for that path until it is released. With ``"stage": "body"``
the server answers first and holds after sending the headers and the first
half of the body, so the browser has a response whose body is still loading.
- ``GET /__test__/hold`` answers ``{"held": {path: count}}``: how many requests
for each held path have arrived and are waiting.
- ``POST /__test__/release`` lets every held request go on, and stops holding.
"""
import asyncio
from aiohttp import web
HOLDABLE_PREFIX = "/prompt_manager/"
class RequestHolds:
def __init__(self) -> None:
self._gates: dict[str, asyncio.Event] = {}
self._waiting: dict[str, int] = {}
self._stages: dict[str, str] = {}
def hold(self, path: str, stage: str = "request") -> None:
self._gates.setdefault(path, asyncio.Event())
self._waiting.setdefault(path, 0)
self._stages[path] = stage
def stage(self, path: str) -> str | None:
return self._stages.get(path) if path in self._gates else None
def held(self) -> dict[str, int]:
return dict(self._waiting)
def release(self) -> None:
for gate in self._gates.values():
gate.set()
self._gates.clear()
self._waiting.clear()
self._stages.clear()
async def wait(self, path: str) -> None:
gate = self._gates.get(path)
if gate is None:
return
self._waiting[path] = self._waiting.get(path, 0) + 1
try:
await gate.wait()
finally:
if path in self._waiting:
self._waiting[path] -= 1
def register_request_holds(app: web.Application, routes) -> RequestHolds:
holds = RequestHolds()
@web.middleware
async def hold_middleware(request: web.Request, handler):
stage = holds.stage(request.path)
if stage == "request":
await holds.wait(request.path)
response = await handler(request)
if stage != "body" or not isinstance(response, web.Response):
return response
body = response.body if isinstance(response.body, bytes) else b""
streamed = web.StreamResponse(status=response.status, headers=response.headers)
streamed.content_length = len(body)
await streamed.prepare(request)
await streamed.write(body[: len(body) // 2])
await holds.wait(request.path)
await streamed.write(body[len(body) // 2 :])
await streamed.write_eof()
return streamed
app.middlewares.append(hold_middleware)
@routes.post("/__test__/hold")
async def hold(request: web.Request) -> web.Response:
data = await request.json() if request.can_read_body else {}
path = data.get("path") if isinstance(data, dict) else None
stage = data.get("stage", "request") if isinstance(data, dict) else None
if (
not isinstance(path, str)
or not path.startswith(HOLDABLE_PREFIX)
or set(data) - {"path", "stage"}
or stage not in ("request", "body")
):
raise web.HTTPBadRequest()
holds.hold(path, stage)
return web.json_response({"success": True})
@routes.get("/__test__/hold")
async def held(request: web.Request) -> web.Response:
return web.json_response({"success": True, "held": holds.held()})
@routes.post("/__test__/release")
async def release(request: web.Request) -> web.Response:
holds.release()
return web.json_response({"success": True})
return holds
+19 -1
View File
@@ -18,10 +18,14 @@ from aiohttp.web_runner import GracefulExit
from PIL import Image
from PIL.PngImagePlugin import PngInfo
from tests.browser.request_hold import register_request_holds
from tests.browser import slow_reset_watch
from tests.browser.workspace_activity_fixture import register_activity_commands
from tests.browser.workspace_system_fixture import register_system_commands
from tests.browser.workspace_concept_fixture import (
MAX_EXTRA_IMAGES as CONCEPT_MAX_EXTRA_IMAGES,
MAX_SEEDLESS_IMAGES as CONCEPT_MAX_SEEDLESS_IMAGES,
MAX_EPOCH_SECONDS as CONCEPT_MAX_EPOCH_SECONDS,
seed_workspace_concept,
)
from tests.browser.workspace_visual_fixture import seed_workspace_visual
@@ -195,6 +199,9 @@ def create_app(root: Path) -> web.Application:
register_tagging_commands(routes, api, root, model)
reveal_stub = register_reveal_stub(routes, api, root)
register_activity_commands(app, routes, api)
register_system_commands(app, routes, api, root)
request_holds = register_request_holds(app, routes)
slow_reset_watch.install(app, api)
async def canonical_asset(request: web.Request) -> web.Response:
filename, content_type = CANONICAL_ASSETS[request.path]
@@ -374,6 +381,7 @@ def create_app(root: Path) -> web.Application:
@routes.post("/__test__/reset")
async def reset(request: web.Request) -> web.Response:
nonlocal history_fixture
request_holds.release()
library_editor_commands.reset()
data = await request.json() if request.can_read_body else {}
if not isinstance(data, dict):
@@ -486,8 +494,17 @@ def create_app(root: Path) -> web.Application:
if (
request.query
or not isinstance(data, dict)
or set(data) - set(limits) - {"short_seedless"}
or set(data) - set(limits) - {"short_seedless", "now"}
or not isinstance(data.get("short_seedless", False), bool)
# A chosen "now" (epoch seconds) pins every time the fixture writes.
or (
"now" in data
and (
isinstance(data["now"], bool)
or not isinstance(data["now"], (int, float))
or not 0 < data["now"] < CONCEPT_MAX_EPOCH_SECONDS
)
)
or any(
isinstance(data.get(key, 0), bool)
or not isinstance(data.get(key, 0), int)
@@ -507,6 +524,7 @@ def create_app(root: Path) -> web.Application:
extra_images=data.get("extra_images", 0),
seedless_images=data.get("seedless_images", 0),
short_seedless=data.get("short_seedless", False),
now=data.get("now"),
)
)
return web.json_response({"success": True, **manifest})
+21 -20
View File
@@ -178,27 +178,28 @@ async function captureFooter(page,testInfo,name) {
test.beforeEach(async({request})=>{await seedConcept(request);});
for(const [width,height] of SIZES)for(const theme of THEMES)test(`shell geometry, tokens and controls match the concept ${width}x${height} ${theme}`,async({page},testInfo)=>{
// One test per route, size, theme and motion. All of them in one test per size and theme (12 page loads, 12 captures,
// 36 hovers) steadily took 22 s median and up to 33 s on CI WebKit, past the 30 s budget with no step stalled; one
// route in one motion is a sixth of that work under the same budget.
for(const [route,breadcrumb] of ROUTES)for(const [width,height] of SIZES)for(const theme of THEMES)for(const motion of MOTIONS)test(`shell geometry, tokens and controls match the concept on ${route} ${width}x${height} ${theme} ${motion}`,async({page},testInfo)=>{
await page.setViewportSize({width,height});const result={};
for(const motion of MOTIONS)for(const [route,breadcrumb] of ROUTES) {
await openConcept(page,route,theme,motion);await page.mouse.move(0,0);
const reference=await inventoryOf(page,'#main');
if(motion==='no-preference')await captureCanonicalState(page,testInfo,`concept-shell-${route}-${width}-${theme}`);
const referenceRings=FULL_HEADING.has(route)?await ringsOf(page,'#main'):null;
const referenceHovers=await hoversOf(page);
if(motion==='no-preference')await captureFooter(page,testInfo,`concept-footer-${route}-${width}-${theme}`);
await openProduction(page,route,theme,motion);await page.mouse.move(0,0);
const actual=await inventoryOf(page,'#workspace-main');
if(motion==='no-preference')await captureCanonicalState(page,testInfo,`production-shell-${route}-${width}-${theme}`);
const actualRings=FULL_HEADING.has(route)?await ringsOf(page,'#workspace-main'):null;
const actualHovers=await hoversOf(page);
if(motion==='no-preference')await captureFooter(page,testInfo,`production-footer-${route}-${width}-${theme}`);
result[`${motion} ${route}`]={route,breadcrumb,reference,actual,diff:compare(route,reference,actual),referenceRings,actualRings,
referenceHovers,actualHovers};
}
const path=testInfo.outputPath('shell-geometry.json');await writeFile(path,JSON.stringify(result,null,2));
await testInfo.attach('shell-geometry',{path,contentType:'application/json'});
for(const [key,{route,breadcrumb,reference,actual,diff,referenceRings,actualRings,referenceHovers,actualHovers}] of Object.entries(result)) {
await openConcept(page,route,theme,motion);await page.mouse.move(0,0);
const reference=await inventoryOf(page,'#main');
if(motion==='no-preference')await captureCanonicalState(page,testInfo,`concept-shell-${route}-${width}-${theme}`);
const referenceRings=FULL_HEADING.has(route)?await ringsOf(page,'#main'):null;
const referenceHovers=await hoversOf(page);
if(motion==='no-preference')await captureFooter(page,testInfo,`concept-footer-${route}-${width}-${theme}`);
await openProduction(page,route,theme,motion);await page.mouse.move(0,0);
const actual=await inventoryOf(page,'#workspace-main');
if(motion==='no-preference')await captureCanonicalState(page,testInfo,`production-shell-${route}-${width}-${theme}`);
const actualRings=FULL_HEADING.has(route)?await ringsOf(page,'#workspace-main'):null;
const actualHovers=await hoversOf(page);
if(motion==='no-preference')await captureFooter(page,testInfo,`production-footer-${route}-${width}-${theme}`);
result[`${motion} ${route}`]={reference,actual,diff:compare(route,reference,actual),referenceRings,actualRings,
referenceHovers,actualHovers};
const path=testInfo.outputPath(`shell-geometry-${route}-${motion}.json`);await writeFile(path,JSON.stringify(result,null,2));
await testInfo.attach(`shell-geometry-${route}-${motion}`,{path,contentType:'application/json'});
for(const [key,{reference,actual,diff,referenceRings,actualRings,referenceHovers,actualHovers}] of Object.entries(result)) {
// The concept itself is what this comparison assumes: its sample counts and its shell controls.
expect.soft(reference.chrome.navs[0].rows.map(row=>[row.text,row.count?.text??null]),`${key} concept WORKSPACE rows`)
.toEqual([['Images','24'],['Prompt library','6'],['Tags & categories','15'],['Inspect a file',null]]);
+237
View File
@@ -0,0 +1,237 @@
"""Report a slow test reset with what the server was doing while it waited.
A reset normally answers within about a second. Once one runs past
SLOW_RESET_SECONDS, this writes to stderr, while the reset is still waiting:
- every thread's stack, so a reset waiting on a lock, a full executor or a
slow disk shows where it waits and who holds it;
- the default executor's threads and queued work;
- how many database connections are open, and the WAL size;
- on Linux, each thread's scheduler state and the kernel function it sleeps
in (``D`` with a journal or writeback wait is a slow disk flush, a nanosleep
is SQLite's busy handler, a futex is a Python lock or an idle worker).
When the reset ends it writes how long it took and, for each executor call it
made, how long the call queued and how long it ran.
Every report line starts with REPORT_PREFIX. The worker fixture
(tools/vendor/start-browser-server.mjs) copies those lines to the job log,
so a slow reset is visible even in a test that still passes.
"""
import asyncio
import contextvars
import faulthandler
import os
import sys
import tempfile
import threading
import time
from typing import Callable, TextIO
from aiohttp import web
SLOW_RESET_SECONDS = 2.0
REPORT_PREFIX = "pm-slow-reset| "
_current: contextvars.ContextVar["ResetWatch | None"] = contextvars.ContextVar(
"pm_reset_watch", default=None
)
def is_reset_path(path: str) -> bool:
return path.startswith("/__test__/") and path.endswith("/reset")
class ResetWatch:
"""Time one reset; dump the server's state if it passes the threshold."""
def __init__(
self,
label: str,
describe: Callable[[], list[str]],
*,
threshold: float = SLOW_RESET_SECONDS,
stream: TextIO | None = None,
) -> None:
self.label = label
self.threshold = threshold
self._describe = describe
self._stream = stream
self._start = time.monotonic()
self._calls: list[str] = []
self._calls_lock = threading.Lock()
self._stalled = threading.Event()
self._timer = threading.Timer(threshold, self._report_stall)
self._timer.daemon = True
self._timer.name = "pm-slow-reset-watch"
def start(self) -> "ResetWatch":
self._timer.start()
return self
def elapsed(self) -> float:
return time.monotonic() - self._start
def record_call(self, name: str, queued: float, ran: float) -> None:
with self._calls_lock:
self._calls.append(
f"executor call {name}: queued {queued:.3f}s, ran {ran:.3f}s"
)
def finish(self, outcome: str) -> None:
self._timer.cancel()
elapsed = self.elapsed()
# A coarse clock (Windows) can fire the timer just short of the
# threshold; a reset that was reported always gets its summary.
if elapsed < self.threshold and not self._stalled.is_set():
return
with self._calls_lock:
calls = list(self._calls)
self._write([f"{self.label} {outcome} after {elapsed:.3f}s", *calls])
def _report_stall(self) -> None:
self._stalled.set()
lines = [f"{self.label} still running after {self.elapsed():.3f}s"]
try:
lines.extend(self._describe())
except Exception as error: # a diagnostic never breaks the reset
lines.append(f"state unavailable: {error!r}")
lines.extend(thread_stacks())
self._write(lines)
def _write(self, lines: list[str]) -> None:
stream = self._stream or sys.stderr
text = "".join(
f"{REPORT_PREFIX}{part}\n" for line in lines for part in line.splitlines()
)
try:
stream.write(text)
stream.flush()
except (OSError, ValueError):
pass
def thread_stacks() -> list[str]:
"""Every thread's stack, from faulthandler.
Formatting sys._current_frames() showed a thread blocked in SQLite as idle
(Python 3.14); faulthandler reports where each thread really is.
"""
# Before Python 3.14 faulthandler prints ids only; name each id here.
names = [
f"thread 0x{thread.ident:016x} is {thread.name}"
for thread in threading.enumerate()
if thread.ident is not None
]
with tempfile.TemporaryFile() as dump:
faulthandler.dump_traceback(file=dump, all_threads=True)
dump.seek(0)
return [*names, *dump.read().decode("utf-8", "replace").splitlines()]
def executor_state(loop: asyncio.AbstractEventLoop) -> list[str]:
executor = getattr(loop, "_default_executor", None)
if executor is None:
return ["default executor: not started"]
threads = getattr(executor, "_threads", ())
queue = getattr(executor, "_work_queue", None)
queued = queue.qsize() if queue is not None else "?"
return [
f"default executor: {len(threads)} of "
f"{getattr(executor, '_max_workers', '?')} threads, {queued} queued"
]
def database_state(model) -> list[str]:
"""How many connections are open, and the WAL size.
Never touches another thread's connection: a connection busy in SQLite
holds its mutex, and reading even ``in_transaction`` from here would block
on it while holding the GIL, freezing every thread until that query ends.
"""
with model._conn_lock:
count = len(model._connections)
wal = model.db_path + "-wal"
size = os.path.getsize(wal) if os.path.exists(wal) else 0
return [f"database: {count} connections, WAL {size} bytes"]
def kernel_wait_state() -> list[str]:
"""Each thread's state and kernel wait channel, where /proc has them."""
lines = []
for thread in threading.enumerate():
task = f"/proc/self/task/{thread.native_id}"
try:
with open(f"{task}/stat", encoding="ascii") as stat:
state = stat.read().rsplit(")", 1)[1].split()[0]
with open(f"{task}/wchan", encoding="ascii") as wchan:
channel = wchan.read().strip() or "0"
except (OSError, IndexError):
continue
# The id matches the "Thread 0x..." headers of the stacks below.
lines.append(
f"kernel: {thread.name} (0x{thread.ident:016x}) state {state}, "
f"waiting in {channel}"
)
return lines
def install(app: web.Application, api) -> None:
"""Watch every reset route, and time the executor calls each one makes."""
run_in_executor = api._run_in_executor
async def timed_run_in_executor(func, *args, **kwargs):
watch = _current.get()
if watch is None:
return await run_in_executor(func, *args, **kwargs)
queued_at = time.monotonic()
started: list[float] = []
def timed(*call_args, **call_kwargs):
started.append(time.monotonic())
return func(*call_args, **call_kwargs)
try:
return await run_in_executor(timed, *args, **kwargs)
finally:
end = time.monotonic()
begin = started[0] if started else end
watch.record_call(
getattr(func, "__qualname__", repr(func)),
begin - queued_at,
end - begin,
)
api._run_in_executor = timed_run_in_executor
@web.middleware
async def watch_resets(request: web.Request, handler):
if not is_reset_path(request.path):
return await handler(request)
loop = asyncio.get_running_loop()
def describe() -> list[str]:
return [
*executor_state(loop),
*database_state(api.db.model),
*kernel_wait_state(),
]
watch = ResetWatch(
f"{request.method} {request.path}", describe, threshold=SLOW_RESET_SECONDS
).start()
token = _current.set(watch)
outcome = "failed"
try:
response = await handler(request)
outcome = f"answered {response.status}"
return response
except web.HTTPException as error:
outcome = f"answered {error.status}"
raise
finally:
_current.reset(token)
watch.finish(outcome)
app.middlewares.append(watch_resets)
+410
View File
@@ -0,0 +1,410 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { SIZES, THEMES, seedConcept, openConcept, openProduction, diffInventory, mutate } from './concept-inventory.mjs';
// #249 System S2 Task 6: System's shell and its Overview section (app.js renderSystem 658–706), the Statistics dialog
// ("Usage statistics", a1600–1604) and the health report (a1439–1456) against the immutable concept, on the
// concept-shaped real fixture. Each concept measurement is mapped only by the recorded deviations (D92–D96, D121,
// D122, D123), each commented where it applies; after that the difference must be empty, and a mutation of a
// row, a label or a token must be noticed.
test.use({ locale: 'en-US', timezoneId: 'UTC' });
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.conceptErrors = [];
page.on('pageerror', error => page.conceptErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/system/reset');
expect(page.conceptErrors).toEqual([]);
});
/** Runs in the page; self-contained. System's heading, section nav, Overview body and Save row under `mainSelector`. */
function measureSystem(mainSelector) {
const main = document.querySelector(mainSelector);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const box = (node, origin = main) => {
const rect = node.getBoundingClientRect(), base = origin.getBoundingClientRect();
return { left: Math.round(rect.left - base.left), width: Math.round(rect.width) };
};
// A surface's top: from `.two-col` beside its neighbour, else from the bottom of the surface above it. This pins the
// concept's `.surface + .surface` 18px offset, which concept.css applies inside `.two-col` too (copied, not ruled).
const top = node => {
const rect = node.getBoundingClientRect(), previous = node.previousElementSibling?.getBoundingClientRect();
if (previous && rect.top >= previous.bottom) return Math.round(rect.top - previous.bottom);
return Math.round(rect.top - node.parentElement.getBoundingClientRect().top);
};
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor',
'borderTopLeftRadius', 'opacity', 'cursor') });
const heading = main.querySelector(':scope > .heading');
const nav = main.querySelector(':scope > .section-nav');
const layout = main.querySelector(':scope > .settings-layout');
const metrics = layout.querySelector(':scope > .summary-metrics');
const columns = layout.querySelector(':scope > .two-col');
const save = layout.querySelector(':scope > .save-row');
const firstMetric = metrics.querySelector('.metric');
const firstSurface = columns.querySelector('.surface');
const firstOperation = firstSurface.querySelector('.operation');
return {
tree: [...main.children].map(name),
heading: {
title: text(heading.querySelector('h1')), description: text(heading.querySelector('p.muted')),
actions: [...(heading.querySelector('.heading-actions')?.children ?? [])].map(name),
},
nav: {
role: nav.getAttribute('role'), label: nav.getAttribute('aria-label'),
...css(nav, 'display', 'flexWrap', 'columnGap', 'rowGap', 'paddingBottom', 'marginBottom', 'borderBottomWidth', 'borderBottomColor'),
box: box(nav),
buttons: [...nav.children].map(item => ({ pressed: item.getAttribute('aria-pressed'), ...button(item) })),
},
layout: { tree: [...layout.children].map(name), ...css(layout, 'maxWidth'), box: box(layout) },
metrics: {
...css(metrics, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap', 'marginBottom'), box: box(metrics),
items: [...metrics.children].map(item => ({ tree: [...item.children].map(name), value: text(item.querySelector('strong')),
label: text(item.querySelector('span')), box: box(item) })),
metric: css(firstMetric, 'paddingTop', 'paddingLeft', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius', 'backgroundColor'),
strong: css(firstMetric.querySelector('strong'), 'display', 'fontSize', 'fontWeight', 'color'),
span: css(firstMetric.querySelector('span'), 'fontSize', 'color', 'marginTop'),
},
columns: {
...css(columns, 'display', 'gridTemplateColumns', 'columnGap', 'rowGap'), box: box(columns),
surfaces: [...columns.children].map(surface => ({ tree: [...surface.children].map(name), box: box(surface), top: top(surface),
heading: text(surface.querySelector(':scope > h2')),
rows: [...surface.querySelectorAll(':scope > .operation')].map(row => ({ tree: [...row.children].map(name),
title: text(row.querySelector('h3')), description: text(row.querySelector('p')),
action: button(row.querySelector('.row-actions > button')) })) })),
surface: css(firstSurface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius'),
h2: css(firstSurface.querySelector('h2'), 'fontSize', 'fontWeight', 'color', 'marginBottom'),
operation: css(firstOperation, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom',
'borderTopWidth', 'borderTopColor'),
h3: css(firstOperation.querySelector('h3'), 'fontSize', 'fontWeight', 'color'),
p: css(firstOperation.querySelector('p'), 'fontSize', 'color', 'marginTop'),
actions: css(firstOperation.querySelector('.row-actions'), 'display', 'columnGap', 'flexShrink'),
},
save: {
tree: [...save.children].map(name), ...css(save, 'display', 'alignItems', 'justifyContent', 'columnGap', 'marginTop', 'paddingTop'),
status: { text: text(save.querySelector('#settings-status')), ...css(save.querySelector('#settings-status'), 'fontSize', 'color') },
button: button(save.querySelector('button')),
},
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
/** Runs in the page: the open dialog's title, body sequence, tables, operation rows and actions. */
function measureDialog() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const body = dialog.querySelector('#modal-body');
const firstCell = selector => body.querySelector(selector);
const list = body.querySelector(':scope > .operation-list');
const actions = body.querySelector(':scope > .dialog-actions');
return {
title: dialog.querySelector('#modal-title').textContent,
body: [...body.children].map(name),
tables: [...body.querySelectorAll(':scope > table')].map(table => ({
caption: table.caption?.textContent ?? null,
head: [...table.tHead.rows[0].cells].map(cell => [cell.localName, cell.scope, cell.textContent]),
rows: [...table.tBodies[0].rows].map(row => [...row.cells].map(cell => cell.textContent)),
})),
tableStyle: firstCell(':scope > table') && {
table: css(firstCell(':scope > table'), 'width', 'borderCollapse', 'fontSize', 'marginTop'),
caption: css(firstCell(':scope > table caption'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingBottom'),
th: css(firstCell(':scope > table thead th'), 'textAlign', 'fontSize', 'fontWeight', 'color', 'paddingTop', 'paddingLeft',
'borderBottomWidth', 'borderBottomColor'),
td: css(firstCell(':scope > table tbody td'), 'textAlign', 'fontSize', 'color', 'paddingTop', 'paddingLeft',
'borderBottomWidth', 'borderBottomColor'),
},
operations: list && {
...css(list, 'display', 'rowGap', 'marginTop'),
rows: [...list.children].map(row => ({ tree: [...row.children].map(name), title: row.querySelector('h3').textContent,
chip: row.querySelector('.chip').textContent })),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'paddingTop', 'borderTopWidth'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
chip: css(list.querySelector('.operation .chip'), 'fontSize', 'color', 'backgroundColor', 'borderTopLeftRadius', 'paddingLeft'),
},
actions: actions && {
...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(item => ({ text: item.textContent, className: item.className,
...css(item, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') })),
},
};
}
/** Runs in the page: a `.notice` (Maintenance's "Check first…") with its copy. */
function measureNotice(mainSelector) {
const notice = document.querySelector(`${mainSelector} .settings-layout > .notice`);
const css = (node, ...keys) => Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]]));
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const run = notice.querySelector(':scope > button');
return {
tree: [...notice.children].map(name), text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, node.textContent]),
frame: css(notice, 'display', 'alignItems', 'justifyContent', 'columnGap', 'borderTopWidth', 'borderTopColor', 'backgroundColor',
'borderTopLeftRadius', 'paddingTop', 'paddingLeft', 'marginBottom'),
h2: css(notice.querySelector('h2'), 'fontSize', 'fontWeight', 'color'),
p: css(notice.querySelector('p'), 'fontSize', 'color', 'marginTop'),
run: { text: run.textContent, className: run.className, ...css(run, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft') },
};
}
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
const number = value => value.toLocaleString('en-US');
const ratingText = value => value === null ? 'Not rated' : `${number(value)} / 5`;
/** The concept's shell and Overview with the recorded deviations applied, given the server's real overview. */
function mapOverviewConcept(reference, overview) {
const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts),
'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) };
const rows = {
// D93 (Q6): no review queue yet, so no count and the action is unavailable (dimmed, as the Save row's D92).
'3 ambiguous image links': row => ({ ...row, title: 'Ambiguous image links', description: 'Not available yet.',
action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }),
// D93: no backup has been recorded (verified backups arrive with S8).
'Backup status': row => ({ ...row, description: 'No retained backup record.' }),
// D93: there is no paging, so the shared settings are grid and density.
'One display configuration': row => ({ ...row, description: 'Grid and density settings are shared across views.' }),
// D93: the real sources, not samples.
'2 sample output sources': row => ({ ...row, title: `${number(overview.sources.effective)} output sources` }),
// D96: the dialog shows runs and ratings by category and library totals, not prompt, tag or date usage.
Statistics: row => ({ ...row, description: 'Runs and ratings by category, with library totals.' }),
};
return {
...reference,
// D92: no "Sample configuration" chip.
heading: { ...reference.heading, actions: [] },
// D122: the concept's literal label is its prototype action name ("system-tab sections").
nav: { ...reference.nav, label: 'System sections' },
metrics: { ...reference.metrics, items: reference.metrics.items.map(item => ({ ...item, value: relabel[item.label] })) },
columns: { ...reference.columns, surfaces: reference.columns.surfaces.map(surface => ({ ...surface,
rows: surface.rows.map(row => (rows[row.title] ?? (value => value))(row)) })) },
save: { ...reference.save,
// D92: truthful status copy, and "Save preferences" is aria-disabled (dimmed) while nothing is unsaved.
status: { ...reference.save.status, text: 'No unsaved changes.' },
button: { ...reference.save.button, opacity: '0.55', cursor: 'not-allowed' } },
};
}
async function conceptOverview(page, theme) {
await openConcept(page, 'system', theme);
await expect(page.locator('#main .summary-metrics')).toBeVisible();
await page.mouse.move(0, 0);
return page.evaluate(measureSystem, '#main');
}
async function productionOverview(page, theme) {
await openProduction(page, 'system/overview', theme);
await expect(page.locator('#workspace-main .summary-metrics')).toBeVisible();
// The thumbnail row is its own read (I2): compare once it answered.
await expect(page.locator('#workspace-main .operation h3').filter({ hasText: 'thumbnails' })).toHaveText('No missing thumbnails');
await page.mouse.move(0, 0);
return page.evaluate(measureSystem, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`system shell and overview match the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const overview = await serverJson(request, '/prompt_manager/system/overview');
// The fixture's thumbnails all exist, so the concept's "No missing thumbnails" is also the real title.
expect((await serverJson(request, '/prompt_manager/images/thumbnails/status')).pending).toBe(0);
const concept = await conceptOverview(page, theme);
expect(concept.heading.actions).toEqual(['span.chip']);
// Paired captures are artifacts only; there are no PNG baselines.
await page.screenshot({ path: testInfo.outputPath(`concept-system-${theme}-${width}.png`), fullPage: true });
const production = await productionOverview(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-system-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapOverviewConcept(concept, overview), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
// The recorded differences, by path: D93 the review row (title, Not available yet, dimmed action), the backup row, the
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
const UNMAPPED_PATHS = [
'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity',
'columns.surfaces.0.rows.0.description', 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
'save.button.cursor', 'save.button.opacity', 'save.status.text',
];
test('system comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const overview = await serverJson(request, '/prompt_manager/system/overview');
const concept = await conceptOverview(page, 'dark');
const expected = mapOverviewConcept(concept, overview);
const production = await productionOverview(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs exactly where a deviation is recorded, and nowhere else.
expect(diffInventory(concept, production).map(item => item.path).sort()).toEqual(UNMAPPED_PATHS);
const measure = () => page.evaluate(measureSystem, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .two-col > .surface:first-child > .operation:nth-of-type(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['columns.surfaces.0.rows.length', 'columns.surfaces.0.tree.length']));
await undoRow();
// A changed label.
await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompt'; });
expect(await paths()).toEqual(['metrics.items.0.label']);
await page.evaluate(() => { document.querySelector('#workspace-main .metric span').textContent = 'Positive prompts'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['metrics.metric.borderTopColor', 'columns.surface.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/** D96: the concept's table, its rows the real ones; then "Library totals" in the same table markup. */
function mapStatisticsConcept(reference, statistics) {
const [runs] = reference.tables;
return {
...reference,
// No sample note (prototype copy); a second table follows the first.
body: ['table', 'table'],
tables: [
{ ...runs, rows: statistics.by_category.map(row => [row.category ?? 'Uncategorized', number(row.runs), ratingText(row.average_rating)]) },
{ caption: 'Library totals', head: [['th', 'col', 'Total'], ['th', 'col', 'Count']],
rows: [['Prompts', number(statistics.totals.total_prompts)], ['Tags', number(statistics.totals.total_tags)],
['Images', number(statistics.totals.total_images)], ['Rated images', number(statistics.rated_images)]] },
],
};
}
// The fixture's image sources and each check's real chip on it: nothing is damaged, Needs review does not exist yet (Q6).
const FIXTURE_SOURCES = ['output', 'input'];
const FIXTURE_CHIPS = Object.freeze({ 'Integrity check': 'Passed', 'Foreign keys & pair references': 'Passed',
'Canonical hashes': 'Passed', 'Pair uniqueness': 'Passed', 'Missing files': 'Passed', 'Ambiguous metadata': 'Not available yet',
Consistency: 'Passed' });
const chipOf = title => {
if (!(title in FIXTURE_CHIPS)) throw new Error(`No expected chip for ${title}`);
return FIXTURE_CHIPS[title];
};
/**
* D95: the concept's health report from its own five rows, labels verbatim, titled "Health report". Only the D95 rows
* are inserted where they are recorded (Pair uniqueness after Canonical hashes, Consistency last); Missing files has a
* row per image source when there are several; each chip is the fixture's expected result.
*/
function mapHealthConcept(reference, sources = FIXTURE_SOURCES) {
const [first] = reference.operations.rows;
const added = title => ({ ...first, title, chip: chipOf(title) });
const rows = reference.operations.rows.flatMap(row => {
const mapped = row.title === 'Missing files' && sources.length > 1
? sources.map(source => ({ ...row, title: `Missing files · ${source}`, chip: chipOf(row.title) }))
: [{ ...row, chip: chipOf(row.title) }];
return row.title === 'Canonical hashes' ? [...mapped, added('Pair uniqueness')] : mapped;
});
return {
...reference,
title: 'Health report',
// No sample note (prototype copy); D95 adds the checked line after the list.
body: ['div.operation-list', 'p.small', 'div.dialog-actions'],
operations: { ...reference.operations, rows: [...rows, added('Consistency')] },
};
}
// Unmapped, the health report differs exactly where D95 is recorded: the title, the body sequence (no sample note, the
// checked line), the rows after Canonical hashes (Pair uniqueness inserted, Missing files per source) and the chips of
// the concept's sample findings.
const UNMAPPED_HEALTH_PATHS = [
'body.0', 'body.1',
...[3, 4].flatMap(row => [`operations.rows.${row}.chip`, `operations.rows.${row}.title`]),
// The concept's five rows end at row 4; rows 5–7 exist only with the D95 rows and the second source.
...[5, 6, 7].flatMap(row => ['chip', 'title', 'tree.0', 'tree.1', 'tree.length'].map(key => `operations.rows.${row}.${key}`)),
'operations.rows.length', 'title',
];
async function conceptDialogs(page, theme, testInfo) {
await openConcept(page, 'system', theme);
await page.locator('#main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
await page.mouse.move(0, 0);
const statistics = await page.evaluate(measureDialog);
await page.screenshot({ path: testInfo.outputPath(`concept-statistics-${theme}.png`) });
await page.keyboard.press('Escape');
await page.locator('#main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
const notice = await page.evaluate(measureNotice, '#main');
await page.locator('#main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Sample health report');
await page.mouse.move(0, 0);
await page.screenshot({ path: testInfo.outputPath(`concept-health-${theme}.png`) });
return { statistics, notice, health: await page.evaluate(measureDialog) };
}
async function productionDialogs(page, theme, testInfo) {
await openProduction(page, 'system/overview', theme);
await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
await page.mouse.move(0, 0);
const statistics = await page.evaluate(measureDialog);
await page.screenshot({ path: testInfo.outputPath(`production-statistics-${theme}.png`) });
await page.keyboard.press('Escape');
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
await expect(page.locator('#workspace-main .notice')).toBeVisible();
const notice = await page.evaluate(measureNotice, '#workspace-main');
await page.locator('#workspace-main .notice').getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Health report');
await page.mouse.move(0, 0);
await page.screenshot({ path: testInfo.outputPath(`production-health-${theme}.png`) });
return { statistics, notice, health: await page.evaluate(measureDialog) };
}
for (const theme of THEMES) test(`statistics and health dialogs match the concept ${theme}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const concept = await conceptDialogs(page, theme, testInfo);
expect(concept.statistics.body).toEqual(['p.demo-note', 'table']);
expect(concept.health.body).toEqual(['p.demo-note', 'div.operation-list', 'div.dialog-actions']);
const production = await productionDialogs(page, theme, testInfo);
const statistics = await serverJson(request, '/prompt_manager/system/statistics');
const { report } = await serverJson(request, '/prompt_manager/system/health');
// The fixture the expected chips describe: its two sources, and the concept's checks plus the D95 ones.
const missing = report.checks.find(check => check.id === 'missing_files');
expect(missing.sources.map(source => source.name)).toEqual(FIXTURE_SOURCES);
expect(diffInventory(mapStatisticsConcept(concept.statistics, statistics), production.statistics)).toEqual([]);
expect(diffInventory(mapHealthConcept(concept.health), production.health)).toEqual([]);
// D123: Maintenance's notice is the concept's own, delivered ahead of the rest of the section.
expect(diffInventory(concept.notice, production.notice)).toEqual([]);
// Unmapped, the dialogs differ only where D95 and D96 are recorded.
expect(diffInventory(concept.health, production.health).map(item => item.path).sort()).toEqual(UNMAPPED_HEALTH_PATHS);
expect(diffInventory(concept.statistics, production.statistics).map(item => item.path.split('.')[0])
.filter((part, index, all) => all.indexOf(part) === index).sort()).toEqual(['body', 'tables']);
// Mutation check: a changed chip token in the health report is noticed, then undone to an exact match again.
const undo = await mutate(page, { css: '#modal .operation .chip { background: rgb(255, 0, 0); }' });
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(['operations.chip.backgroundColor']);
await undo();
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog))).toEqual([]);
// A renamed check (database/health.py's label) is noticed against the concept's own label.
await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity checks'; });
expect(diffInventory(mapHealthConcept(concept.health), await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(['operations.rows.0.title']);
await page.evaluate(() => { document.querySelector('#modal .operation h3').textContent = 'Integrity check'; });
// And a removed "Library totals" table and a changed caption in the statistics dialog.
await page.keyboard.press('Escape');
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await page.locator('#workspace-main').getByRole('button', { name: 'Explore', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Usage statistics');
const expectedStatistics = mapStatisticsConcept(concept.statistics, statistics);
const undoTable = await mutate(page, { remove: ['#modal-body > table:nth-of-type(2)'] });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path))
.toEqual(expect.arrayContaining(['body.length', 'tables.length']));
await undoTable();
await page.evaluate(() => { document.querySelector('#modal-body caption').textContent = 'Runs per category'; });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)).toEqual(['tables.0.caption']);
});
+125 -24
View File
@@ -43,6 +43,55 @@ test('actual gallery form saves a prompt that the admin page can edit and reload
expect((await saved.json()).prompt_id).not.toBe(Number(id));
});
// WebKit fails a fetch that a leaving page starts ("Fetch API cannot load … due to access control checks."), and
// Playwright reports that console error as a page error. A save's follow-up reads must not start once the page hides:
// here the statistics read after an in-place edit is held at the server while the page reloads.
test('an in-place edit whose follow-up reads outlive the page starts no read while it leaves', async ({ page, request }) => {
await page.goto('/prompt_manager/admin');
const text = page.locator('.prompt-text').filter({ hasText: 'Cat portrait' });
const id = await text.getAttribute('data-id');
await expect(await request.post('/__test__/hold', { data: { path: '/prompt_manager/stats' } })).toBeOK();
await page.locator(`[data-action="edit"][data-prompt-id="${id}"]`).click();
const editor = page.locator(`.prompt-text[data-id="${id}"]`);
await editor.fill('Cat portrait at dusk');
await editor.press('Enter');
const held = async () => (await (await request.get('/__test__/hold')).json()).held['/prompt_manager/stats'];
await expect.poll(held).toBe(1);
await page.reload({ waitUntil: 'commit' });
await expect(await request.post('/__test__/release')).toBeOK();
await expect(page.locator('.prompt-text').filter({ hasText: 'Cat portrait at dusk' })).toBeVisible();
});
// A leave that never happens (a stopped navigation, or one answered with a download) must not leave the list dead.
test('a page whose leave was called off reads its list again on the next input', async ({ page }) => {
await page.goto('/prompt_manager/admin');
await expect(page.locator('.prompt-text').first()).toBeVisible();
await page.evaluate(() => window.dispatchEvent(new Event('beforeunload')));
await page.mouse.click(1, 1);
const reread = page.waitForResponse(response => new URL(response.url()).pathname === '/prompt_manager/recent');
await page.locator('#sortBy').selectOption('created_asc');
expect(new URL((await reread).url()).searchParams.get('sort')).toBe('created_asc');
await expect(page.locator('#resultsTitle')).toHaveText('Recent Prompts');
});
// The gallery's startup thumbnail check runs a second after load. When that second ends while the page is leaving,
// the check must start no read: WebKit fails it with "Fetch API cannot load … due to access control checks." and
// Playwright reports a page error (seen in CI on PR #289). The page clock makes the second end after the leave began.
for (const leave of ['beforeunload', 'pagehide']) {
test(`the gallery startup check starts no read after ${leave}`, async ({ page }) => {
await page.clock.install();
const statusReads = [];
page.on('request', req => { if (req.url().includes('/images/thumbnails/status')) statusReads.push(req.url()); });
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(2);
await page.evaluate(type => window.dispatchEvent(new Event(type)), leave);
await page.clock.runFor(1500);
// A read made after the clock ran is listed, so the listener saw every status read.
await page.evaluate(() => fetch('/prompt_manager/images/thumbnails/status?probe=1'));
expect(statusReads.map(url => new URL(url).search)).toEqual(['?probe=1']);
});
}
test('actual gallery review generates and persists tags for the selected root', async ({ page, request }) => {
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(2);
@@ -305,39 +354,48 @@ test('prompt deletion requires confirmation and can retry after a network failur
}
});
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
// The duplicate scan reads the output root only, but the fixture's input root holds a different
// shared/same.png. A bare /images/serve/<path> URL tries every gallery root in turn, so once the
// output copy is deleted it answers 200 with the input copy: check each file in its own root.
const serveIn = (root, path) => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}?root=${root}`;
const rootOf = url => new URL(url, 'http://gallery.test').searchParams.get('root');
// Which copy the scan keeps depends on file times: the fixture stamps shared/same.png with the
// wall clock, and the kernel can stamp its bulk copies a fraction of a millisecond earlier.
// listOrder rewrites the finished scan's groups so a test can pick the order it needs.
async function openDuplicateScan(page, request, listOrder = null) {
await expect(await request.post('/__test__/reset', { data: { extra_images: 2 } })).toBeOK();
if (listOrder) {
await page.route('**/prompt_manager/scan_duplicates/stream', async route => {
const response = await route.fetch();
const body = (await response.text()).replace(/^data: (.*)$/gm, (line, json) => {
const event = JSON.parse(json);
if (event.type !== 'complete') return line;
for (const group of event.duplicates) group.images = listOrder(group.images);
return `data: ${JSON.stringify(event)}`;
});
await route.fulfill({ response, body });
});
}
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(4);
const originals = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
const output = rootOf(originals.find(url => url.includes('/bulk/')));
const input = rootOf(originals.find(url => url.includes('/shared/same.png') && rootOf(url) !== output));
expect(output).not.toBeNull();
expect(input).not.toBeNull();
await page.locator('#settingsBtn').click();
await page.locator('#scanDuplicatesBtn').click();
await expect(page.locator('#duplicatesCount')).toHaveText('1');
const candidates = page.locator('.duplicate-checkbox');
await expect(candidates).toHaveCount(2);
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
const serve = path => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}`;
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serve(selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serve(selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
expect((await request.get(serve(selected))).status()).toBe(404);
await expect(await request.get(serve(retained))).toBeOK();
return { originals, output, input, candidates };
}
async function expectOnlySelectedGone(page, request, { originals, output, input }, selected, retained) {
expect((await request.get(serveIn(output, selected))).status()).toBe(404);
await expect(await request.get(serveIn(output, retained))).toBeOK();
await expect(await request.get(serveIn(input, 'shared/same.png'))).toBeOK();
await page.reload();
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(3);
const remaining = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
@@ -346,4 +404,47 @@ test('duplicate cleanup cancels safely and deletes only the selected file after
expect(originals).toContain(url);
await expect(await request.get(url)).toBeOK();
}
}
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
const scan = await openDuplicateScan(page, request);
const { output, candidates } = scan;
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serveIn(output, selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serveIn(output, selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
test('deleting a duplicate whose path another gallery root also holds removes only the scanned copy', async ({ page, request }) => {
// The order CI saw: a bulk copy stamped first, so shared/same.png is the first Delete candidate.
const ciOrder = ['bulk/0000.png', 'shared/same.png', 'bulk/0001.png'];
const rank = image => ciOrder.indexOf(image.path.replace(/\\/g, '/'));
const scan = await openDuplicateScan(page, request, images => [...images].sort((a, b) => rank(a) - rank(b)));
const selected = await scan.candidates.first().getAttribute('data-image-path');
const retained = await scan.candidates.last().getAttribute('data-image-path');
expect(selected.replace(/\\/g, '/')).toBe('shared/same.png');
await scan.candidates.first().check();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
+169 -9
View File
@@ -2,6 +2,13 @@ import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage
import { test, expect } from './fixtures.mjs';
import { seedConcept, openConcept, openProduction } from './concept-inventory.mjs';
// The Activity store's periodic list read (activity-store.mjs POLL_MS) and the detail view's read throttle
// (activity-job-view.mjs DETAIL_INTERVAL_MS).
const POLL_MS = 15_000;
const DETAIL_INTERVAL_MS = 1_000;
// A request is reported to the test a little after the browser sends it; that delay is not a shorter wait.
const REPORT_SLACK_MS = 100;
// Activity (#249 Activity-1): the concept's Activity page (app.js renderActivity 732–738), its top-bar button
// (index.html 51–52) and sidebar row (app.js ~341), with the recorded D52 (counts of running jobs), D53 (the progress
// pill) and D54 (real job items). Fake jobs come from tests/browser/workspace_activity_fixture.py.
@@ -278,6 +285,8 @@ test('reload during a scan reattaches to the running job', async ({ page, reques
});
test('lost server clears the pill and marks jobs ended', async ({ page, request }) => {
// A poll interval after the restart, then failed re-reads backing off for 1 + 2 + 4 s.
test.setTimeout(90_000);
await page.setViewportSize({ width: 1600, height: 1100 });
const job = await startJob(request);
await step(request, job.id, { processed: 4000 });
@@ -294,9 +303,12 @@ test('lost server clears the pill and marks jobs ended', async ({ page, request
await expect(items(page).first().locator('p')).toHaveText(LOST);
await expect(items(page).first().locator('progress, .row-actions')).toHaveCount(0);
await expect(navRow(page).locator('.nav-count')).toHaveCount(0);
// One re-read found the restart; there is no retry storm and no finish toast for a job that only vanished.
await page.waitForTimeout(2500);
expect(reads.length).toBeLessThanOrEqual(2);
// One re-read found the restart; there is no retry storm and no finish toast for a job that only vanished. The
// read after it is the periodic one, a whole poll interval later: any extra read lands in that gap and fails it.
const found = reads.length;
expect(found, 'reads that found the restart').toBeGreaterThanOrEqual(1);
await expect.poll(() => reads.length, { timeout: POLL_MS + 10_000 }).toBeGreaterThan(found);
expect(reads[found] - reads[found - 1], 'the next read waits for the poll').toBeGreaterThanOrEqual(POLL_MS - REPORT_SLACK_MS);
await expect(toast(page)).toHaveText('');
// A server that cannot be read at all also ends the running job, then is read again with a growing backoff.
@@ -308,9 +320,12 @@ test('lost server clears the pill and marks jobs ended', async ({ page, request
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
await expect(pill(page)).toBeHidden();
await expect(items(page).filter({ hasText: LOST })).toHaveCount(1);
await page.waitForTimeout(3500);
// 1 s, then 2 s: at most two re-reads in 3.5 s.
expect(failed.length).toBeLessThanOrEqual(2);
// 1 s, then 2 s, then 4 s: each failed re-read waits twice as long as the one before. The gaps between the
// reads themselves are the evidence, so a slow runner noticing the ended job late cannot add a read to a window.
await expect.poll(() => failed.length, { timeout: 20_000 }).toBeGreaterThanOrEqual(3);
const gaps = failed.slice(1, 3).map((at, index) => at - failed[index]);
expect(gaps[0]).toBeGreaterThanOrEqual(2000 - REPORT_SLACK_MS);
expect(gaps[1]).toBeGreaterThanOrEqual(4000 - REPORT_SLACK_MS);
await page.unroute('**/prompt_manager/jobs**');
expect(second.id).not.toBe(job.id);
});
@@ -607,6 +622,8 @@ test('filter narrows the selected outcome', async ({ page, request }) => {
test('job view renders 200-row cap note', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
// The page's timers are Playwright's, running at real speed, so the M12 check below can run a read throttle out.
await page.clock.install();
const job = await startJob(request);
await step(request, job.id, { processed: 260, bulk: { group: 'added', count: 250 }, records: [
...Array.from({ length: 10 }, (_, index) => ({ group: 'no_metadata', path: `misc/plain_${index}.png`, detail: 'No prompt metadata in the file' })),
@@ -646,11 +663,15 @@ test('job view renders 200-row cap note', async ({ page, request }) => {
page.on('request', item => { if (new URL(item.url()).pathname === `/prompt_manager/jobs/${job.id}`) reads.push(Date.now()); });
await step(request, job.id, { processed: 300, bulk: { group: 'added', count: 40 } });
await expect(tiles(page).first().locator('dd')).toHaveText('290');
await page.waitForTimeout(1500);
// Every read the counter update could have scheduled is due within one throttle interval: run it out, so any such
// read is sent before the log opens. Requests reach the test in the order they were sent, so once the log's read
// has answered, a read sent before it would already be listed.
await page.clock.runFor(DETAIL_INTERVAL_MS);
expect(reads).toEqual([]);
await scanLog(page).locator('summary').click();
await expect.poll(() => reads.length).toBe(1);
await expect(scanLog(page).locator('pre')).toContainText('bulk/file_00039.png');
expect(reads, 'only the log\'s own read').toHaveLength(1);
});
test('cancel scan ends cancelled with partial counts', async ({ page, request }) => {
@@ -678,16 +699,44 @@ test('cancel scan ends cancelled with partial counts', async ({ page, request })
await expect(heading(page).getByRole('button', { name: 'Cancel scan', exact: true })).not.toHaveAttribute('aria-disabled', 'true');
});
// Truth rule: a held legacy-page job (a restore from the admin page) cannot be cancelled, so neither the Activity
// list nor the job's view offers Cancel; the cancellable scan beside it keeps its own (review I3).
test('a held legacy job shows no Cancel in the list or its view', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const response = await request.post('/__test__/jobs/hold', { data: { kind: 'restore', title: 'Database restore (admin page)' } });
await expect(response).toBeOK();
const held = (await response.json()).job;
expect(held.cancellable).toBe(false);
await page.goto(`${workspace}#/activity`);
const item = items(page).filter({ has: page.getByRole('link', { name: /Database restore \(admin page\)$/ }) });
await expect(item.locator('.chip')).toHaveText('Running');
await expect(item.locator('progress')).toHaveCount(1);
await expect(main(page).getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
await item.getByRole('link').click();
await expect(heading(page).locator('h1')).toHaveText('Database restore (admin page)');
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
await expect(heading(page).getByRole('button', { name: /^Cancel/ })).toHaveCount(0);
// When the hold ends the view shows the end, still without a Cancel.
await finish(request, held.id);
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished/);
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
});
test('busy start names the running job', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
// The page's timers are Playwright's, so the toast's lifetime below is run out rather than waited for.
await page.clock.install();
const running = await startJob(request, { title: 'Database maintenance', kind: 'maintenance' });
await startFromImport(page);
await expect(toast(page)).toHaveText('Another job is running: Database maintenance.');
await expect(toast(page).getByRole('link', { name: 'Database maintenance', exact: true })).toHaveAttribute('href', `#/activity/${running.id}`);
await expect(page).toHaveURL(/#\/images$/);
expect((await listed(request)).map(item => item.id)).toEqual([running.id]);
// Review M14: a toast with a link outlasts the plain 6.5 s, so the link stays reachable.
await page.waitForTimeout(7000);
// Review M14: a toast with a link outlasts the plain 6.5 s, so the link stays reachable. Running the page's timers
// 7 s on fires a 6.5 s clear for certain, however slowly this runner reached this line.
await page.clock.runFor(7000);
await expect(toast(page).getByRole('link', { name: 'Database maintenance', exact: true })).toBeVisible();
await toast(page).getByRole('link', { name: 'Database maintenance', exact: true }).click();
await expect(heading(page).locator('h1')).toHaveText('Database maintenance');
@@ -799,3 +848,114 @@ test('a failed report download says so and leaves the button usable', async ({ p
await expect(toast(page)).toHaveText('The report could not be downloaded. Try again.');
await expect(button).not.toHaveAttribute('aria-disabled', 'true');
});
// ── #249 PR D Task D3 (D89, plan D66): image tagging jobs in Activity ─────────────────────────
const TAG_JOB = { kind: 'image-tags', title: 'Auto-tag images', mode: 'auto', images: 4, total: 4 };
test('an image tagging job shows Tagged Unchanged Skipped Errors and opens tagged rows', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const job = await startJob(request, TAG_JOB);
await step(request, job.id, { processed: 3, records: [
{ group: 'tagged', path: 'shared/same.png', root: 'output', detail: 'Added 2 tags' },
{ group: 'unchanged', path: 'shared/same.png', root: 'input', detail: 'No new tags' },
{ group: 'skipped', path: 'video/clip_0004.webm', root: 'output', detail: 'Video: manual tags only' },
] });
await page.goto(`${workspace}#/images`);
// The pill names the kind's verb.
await expect(pill(page)).toHaveText('Tagging · 75% · 3 / 4');
await openJob(page, job.id);
await expect(heading(page).locator('h1')).toHaveText('Auto-tag images');
await expect(page).toHaveTitle('Auto-tag images · Prompt Manager');
// The header's scope is the number of images, and the job cancels as "Cancel tagging" (danger).
await expect(heading(page).locator('p.muted')).toHaveText(/^Started \d\d:\d\d · 4 images · closing this view does not cancel it$/);
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel tagging']);
await expect(heading(page).getByRole('button', { name: 'Cancel tagging', exact: true })).toHaveClass(/danger/);
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Processing file 3 of 4');
await expect(progressSection(page).locator('progress')).toHaveAttribute('aria-label', 'Auto-tag images progress');
// Four tiles in the scan's tones: Tagged green, Unchanged plain, Skipped warning, Errors red.
expect(await tileValues(page)).toEqual([
['stat added', 'Tagged', '1'], ['stat', 'Unchanged', '1'], ['stat warn', 'Skipped', '1'], ['stat bad', 'Errors', '0'],
]);
// What got tagged and what went wrong first, as the scan's list does.
await expect(outcomeButtons(page)).toHaveText(['Tagged1', 'Errors0', 'Skipped1', 'Unchanged1']);
await expect(outcomeButtons(page).first()).toHaveAttribute('aria-pressed', 'true');
// Tagged rows open the viewer on that file (D57's identity).
const tagged = fileRows(page).first();
expect(await tagged.evaluate(node => node.localName)).toBe('button');
await expect(tagged.locator('.path')).toHaveText('shared/same.png');
await expect(tagged.locator('.why')).toHaveText('Added 2 tags');
await expect(resultsNote(page)).toHaveText('Each row opens the image in the viewer.');
await tagged.click();
const viewer = page.locator('dialog#viewer-dialog[open]');
await expect(viewer.locator('#viewer-stage img').first()).toHaveAttribute('src', /[?&]root=0(&|$)/);
await page.keyboard.press('Escape');
await expect(viewer).toHaveCount(0);
await expect(tagged).toBeFocused();
// Unchanged rows open too, here the input root's file.
await outcomeButtons(page).filter({ hasText: 'Unchanged' }).click();
await fileRows(page).first().click();
await expect(page.locator('dialog#viewer-dialog[open] #viewer-stage img').first()).toHaveAttribute('src', /[?&]root=1(&|$)/);
await page.keyboard.press('Escape');
// Skipped rows are plain text with their reason.
await outcomeButtons(page).filter({ hasText: 'Skipped' }).click();
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('div');
await expect(fileRows(page).first().locator('.why')).toHaveText('Video: manual tags only');
await expect(resultsNote(page)).toHaveCount(0);
// The log is the job's own: "Tagging log", with the runner's prefixes.
await expect(scanLog(page).locator('summary')).toHaveText('● Tagging log — raw detail, click to show');
await scanLog(page).locator('summary').click();
await expect(scanLog(page).locator('pre')).toContainText('+ shared/same.png · Added 2 tags');
// An error, then the end: the header and the Activity item read "N tagged · M errors".
await step(request, job.id, { processed: 4, records: [
{ group: 'errors', path: 'broken/truncated.png', root: 'output', detail: 'Auto-tagging failed. Check server logs for details.' },
] });
await expect(tiles(page).last().locator('dd')).toHaveText('1');
await finish(request, job.id);
await expect(toast(page)).toHaveText('Auto-tag images finished · 1 tagged · 1 error');
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d · 1 tagged · 1 error$/);
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('4 of 4 files processed');
await page.goto(`${workspace}#/activity`);
await expect(items(page).first().locator('h3')).toHaveText('✓ Auto-tag images');
await expect(items(page).first().locator('p')).toHaveText(/^Finished \d\d:\d\d · 1 tagged · 1 error$/);
});
test('cancel tagging stops at a file boundary', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const job = await startJob(request, { ...TAG_JOB, title: 'Add tags to images', mode: 'add', images: 1250, total: 1250 });
await step(request, job.id, { processed: 2, counters: { tagged: 2 } });
await page.goto(`${workspace}#/activity`);
await expect(items(page).first().locator('p')).toHaveText(/^Started \d\d:\d\d · 1,250 images · closing this view does not cancel it$/);
await openJob(page, job.id);
await heading(page).getByRole('button', { name: 'Cancel tagging', exact: true }).focus();
await page.keyboard.press('Enter');
await expect(toast(page)).toHaveText('Add tags to images cancelled · 2 tagged');
await expect(heading(page).locator('h1')).toBeFocused();
await expect(heading(page).locator('p.muted')).toHaveText(/^Cancelled \d\d:\d\d at a file boundary · 2 tagged · completed work is kept$/);
await expect(heading(page).getByRole('button', { name: 'Cancel tagging', exact: true })).toHaveCount(0);
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Cancelled after 2 of 1,250 files');
expect((await tileValues(page)).map(([, , value]) => value)).toEqual(['2', '0', '0', '0']);
expect((await listed(request))[0].state).toBe('cancelled');
});
test('the report download names the kind', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const job = await startJob(request, TAG_JOB);
await step(request, job.id, { processed: 1, records: [
{ group: 'tagged', path: 'sub/a.png', root: 'output', detail: 'Added 1 tag' },
] });
await finish(request, job.id);
await openJob(page, job.id);
const button = heading(page).getByRole('button', { name: 'Download report', exact: true });
const [download] = await Promise.all([page.waitForEvent('download'), button.click()]);
expect(download.suggestedFilename()).toMatch(/^prompt-manager-image-tags-\d{8}-\d{4}\.json$/);
const report = JSON.parse(await (await import('node:fs/promises')).readFile(await download.path(), 'utf8'));
expect(report.job.kind).toBe('image-tags');
expect(report.job.scope).toEqual({ mode: 'auto', images: 4 });
expect(Object.keys(report.outcomes)).toEqual(['tagged', 'unchanged', 'skipped', 'errors']);
expect(report.outcomes.tagged.rows.map(row => [row.path, row.detail])).toEqual([['sub/a.png', 'Added 1 tag']]);
await expect(toast(page)).toHaveText('Job report downloaded.');
});
+17
View File
@@ -494,6 +494,23 @@ test('the inline current file names its prompts apart from the frozen run fields
expect(await exact(dialog(page).getByRole('region',{name:'Positive prompt',exact:true}).locator('p.prompt-text'))).toBe(before);
});
// LI-4 (D117): the recorded history's current-file viewer is the shared full inspector, so it copies the legacy summary.
test('the current file viewer copies its settings summary',async({page})=>{
await page.addInitScript(()=>{window.__copied=[];Object.defineProperty(navigator,'clipboard',{configurable:true,value:{writeText:async text=>{window.__copied.push(text);}}});});
await page.route(url=>url.pathname.endsWith('/inspection'),async route=>{const response=await route.fetch();const body=await response.json();
body.prompts={positive:{status:'resolved',text:'current file positive',source:'graph-inferred'},negative:{status:'resolved',text:'current file negative',source:'graph-inferred'}};
await route.fulfill({response,json:body});});
await open(page,page.fixture.detail_id);await marker(page,page.fixture.runs.shared).click();await expect(dialog(page).locator('[data-receipt]')).toHaveCount(5);
await dialog(page).locator('[data-receipt]').first().getByRole('button',{name:'View current file',exact:true}).click();
const details=page.locator('dialog#viewer-dialog #viewer-details');await expect(details.locator('.panel-heading')).toBeVisible();
const facts=await details.locator(':scope > div > dl.meta-grid dd').allTextContents();
await details.getByRole('button',{name:'Copy settings summary',exact:true}).click();
await expect(page.locator('#workspace-toast')).toHaveText('Settings summary copied.');
const [seed,steps,cfg,sampler,,model]=facts;
expect(await page.evaluate(()=>window.__copied)).toEqual([[`Checkpoint: ${model}`,'Positive Prompt: current file positive','Negative Prompt: current file negative',
`Steps: ${steps}`,`CFG Scale: ${cfg}`,`Sampler: ${sampler}`,`Seed: ${seed}`].join('\n').replace(/: (Not recorded|Unresolved|Ambiguous|Unavailable within inspection limits)(?=\n|$)/g,': Unknown')]);
});
test('the recorded run dialog with a resolved current file has no duplicate button or region names',async({page})=>{
await page.route(url=>url.pathname.endsWith('/inspection'),async route=>{const response=await route.fetch();const body=await response.json();
body.prompts={positive:{status:'resolved',text:'current file positive',source:'graph-inferred'},negative:{status:'resolved',text:'current file negative',source:'graph-inferred'}};
+12 -2
View File
@@ -787,7 +787,12 @@ test('density changes and resize preserve identity focus and selection', async (
await page.getByRole('button',{name:'Grid',exact:true}).click();
for (const [width,columns] of [[951,3],[950,3],[700,2],[390,2]]) {
await page.setViewportSize({width,height:844});
await expect.poll(()=>cards(page).nth(columns).evaluate(node=>node.offsetLeft)).toBe(0);
// 700px and 390px both lay out two columns, so a card at the row start alone is also true of the previous
// width's cards before WebKit runs the first frame after the resize; wait for the row to fit the new grid.
await expect.poll(()=>page.locator('.image-grid').evaluate((grid,columns)=>{
const row=[...grid.querySelectorAll(':scope > article[data-image-key]')];
return row[columns].offsetLeft===0&&row[columns-1].getBoundingClientRect().right<=grid.getBoundingClientRect().right+1;
},columns)).toBe(true);
await first.getByRole('button',{name:/^Open /}).focus();await page.keyboard.press('ArrowDown');
await expect(cards(page).nth(columns).getByRole('button',{name:/^Open /})).toBeFocused();
expect(await page.evaluate(()=>document.documentElement.scrollWidth-document.documentElement.clientWidth)).toBeLessThanOrEqual(1);
@@ -902,6 +907,10 @@ test('browser Back closes the open Images dialog and leaves the next page usable
test('count line shows images and positive prompts', async ({ page, request }) => {
await page.addInitScript(() => { window.IntersectionObserver = undefined; });
// D17's "Today" is the newest file's day: the page's clock stands at that file's mtime, so the fixture's day and
// the page's agree on either side of midnight.
const { images } = await (await request.get('/prompt_manager/images/output?mode=workspace&limit=40')).json();
await page.clock.setFixedTime(new Date(Math.max(...images.map(image => image.modified_time)) * 1000));
await page.goto(workspace);
await expect(cards(page)).toHaveCount(2);
await expect(caption(page).locator('#result-count')).toHaveText('2 of 2 images · 2 positive prompts');
@@ -1702,7 +1711,8 @@ test('card checkboxes select images into the concept bar and Clear empties it',
const selectedBefore = await cards(page).evaluateAll(nodes => nodes.map(node => node.classList.contains('selected')));
await checkbox(first, 'same.png').check();
await expect(bulkbar(page).locator('.bulkbar > strong')).toHaveText('1 selected');
await expect(barButtons(page)).toHaveText(['Set category', 'Export metadata', 'Clear']);
// D85: Add tags (the concept's) and Auto-tag selected lead the real actions.
await expect(barButtons(page)).toHaveText(['Add tags', 'Auto-tag selected', 'Set category', 'Export metadata', 'Clear']);
// Toggling neither opens the viewer, nor inspects, nor moves focus.
await expect(page.locator('dialog[open]')).toHaveCount(0);
await expect(checkbox(first, 'same.png')).toBeFocused();
+190 -20
View File
@@ -207,6 +207,40 @@ test('chooser opens the shared full inspector without a second read', async ({ p
expect(await page.evaluate(() => window.__slices)).toBe(0);
});
// PR L3 Task 10: local roles are the server inspector's (local-resolver.mjs), so for every parity PNG the rail shows the
// prompts, their status lines and the facts Images shows for the same bytes in the output folder.
const PARITY_PNGS = ['f01b', 'f02', 'f03', 'f06a', 'f06b', 'f06c', 'f06d', 'f06e', 'f07a', 'f07b', 'f08', 'f09', 'f10', 'f11', 'f12',
'f14', 'f15', 'f16', 'f17', 'f19', 'f20', 'f21', 'f22', 'f23', 'f24'];
const roleSections = locator => locator.evaluate(node => [...node.querySelectorAll('.prompt-section')].map(item => item.textContent));
test('chooser shows the Images roles and resolution for every parity PNG', async ({ page, request }) => {
test.slow();
for (const name of PARITY_PNGS) {
await expect(await request.post('/__test__/add-local-inspect-file', { data: { name: `${name}.png` } })).toBeOK();
}
const served = {};
await page.goto(`${workspace}#/images`);
for (const name of PARITY_PNGS) {
await page.locator(`article[data-image-key="0:local-inspect/${name}.png"] .image-open`).click();
await expect(viewer(page).locator('#viewer-title')).toHaveText(`${name}.png`);
await expect(rail(page).locator('.panel-heading')).toBeVisible();
await expect(rail(page).locator('.prompt-section')).not.toHaveCount(0);
served[name] = { content: await railContent(rail(page)), roles: await roleSections(rail(page)) };
await page.keyboard.press('Escape');
await expect(viewer(page)).toHaveCount(0);
}
await openInspect(page);
for (const name of PARITY_PNGS) {
await choose(page, parityPng(`${name}.png`));
await openViewer(page);
await expect(viewer(page).locator('#viewer-title')).toHaveText(`${name}.png`);
await expect(rail(page).locator('.prompt-section')).not.toHaveCount(0);
expect({ content: await railContent(rail(page)), roles: await roleSections(rail(page)) }, name).toEqual(served[name]);
await page.keyboard.press('Escape');
await expect(viewer(page)).toHaveCount(0);
}
});
test('Import → Inspect a local image lands here', async ({ page }) => {
await page.goto(`${workspace}#/images`);
await page.getByRole('button', { name: 'Import', exact: true }).click();
@@ -481,15 +515,13 @@ test('replacing a video that just started loading never blocks the page', async
});
test('a video replaced while still starting is reset once it settles', async ({ context, browserName }) => {
// A WebKit (GStreamer) video released exactly at its `loadstart` sometimes blocks the page, with or without any
// reset and also with the code before the deferred reset (2 of 24 runs); tracked in TESTING.md (D62). WebKit's
// no-hang guard is `replacing a video that just started loading never blocks the page`.
test.skip(browserName === 'webkit', 'WebKit release at loadstart is a tracked follow-up (D62)');
test.setTimeout(120000);
// The video is replaced from its own `loadstart` (readyState HAVE_NOTHING). It leaves the page with no src before
// its URL is revoked, and is reset (load() with no src, which frees its decoder) only once it reports metadata or
// an error, so the page never blocks. Fresh pages share one browser process.
for (let attempt = 0; attempt < 5; attempt++) {
test.setTimeout(180000);
// The video is replaced from its own `loadstart` (readyState HAVE_NOTHING), or a few milliseconds after it. On
// WebKit (GStreamer) a video that left the page or the viewport before its pipeline had started deadlocked the
// page (TESTING.md D62), so it stays in place, inert and still playing its URL, with the new preview after it.
// Once it reports metadata or an error it is paused, loses its src, is reset (load() with no src, which frees its
// decoder) and leaves the page, and only then is its URL revoked. Fresh pages share one browser process.
for (let attempt = 0; attempt < 15; attempt++) {
const page = await context.newPage();
try {
if (attempt === 0) await startCoverage(page, browserName);
@@ -510,10 +542,12 @@ test('a video replaced while still starting is reset once it settles', async ({
const video = slot.querySelector('video');
if (!video) return;
observer.disconnect();
video.addEventListener('loadstart', () => {
const replace = () => {
window.__starting = { node: video, url: video.getAttribute('src'), readyState: video.readyState };
input.files = window.__pngFiles; input.dispatchEvent(new Event('change', { bubbles: true }));
}, { once: true });
window.__starting.replaced = { connected: video.isConnected, src: video.hasAttribute('src'), inert: video.inert, resets: video.__resets ?? 0 };
};
video.addEventListener('loadstart', () => { if (window.__delay) setTimeout(replace, window.__delay); else replace(); }, { once: true });
}).observe(slot, { childList: true });
});
const png = parityPng('f02.png');
@@ -522,13 +556,17 @@ test('a video replaced while still starting is reset once it settles', async ({
transfer.items.add(new File([new Uint8Array(bytes)], 'f02.png', { type: 'image/png' }));
window.__pngFiles = transfer.files;
}, [...png.buffer]);
await page.evaluate(delay => { window.__delay = delay; }, attempt % 5);
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(preview(page).locator('img.local-file-preview')).toBeVisible({ timeout: 8000 });
await expect(preview(page).locator('video')).toHaveCount(0);
// Released while starting, and never reset synchronously: the URL went once the node had no src.
expect(await page.evaluate(() => ({ readyState: window.__starting.readyState, revoked: window.__starting.revoked })))
.toEqual({ readyState: 0, revoked: { src: false, connected: false, resets: 0 } });
await expect.poll(() => page.evaluate(() => window.__starting.node.__resets ?? 0), { timeout: 15000 }).toBe(1);
// Replaced while starting (always so at `loadstart` itself): kept in place and untouched. A video that had
// settled is released at once. Either way it is reset once, and its URL goes last.
await expect.poll(() => page.evaluate(() => window.__starting.revoked ?? null)).toEqual({ src: false, connected: false, resets: 1 });
const { readyState, replaced } = await page.evaluate(() => ({ readyState: window.__starting.readyState, replaced: window.__starting.replaced }));
if (attempt % 5 === 0) expect(readyState).toBe(0);
expect(replaced).toEqual(readyState === 0 ? { connected: true, src: true, inert: true, resets: 0 } : { connected: false, src: false, inert: false, resets: 1 });
expect(await page.evaluate(() => window.__starting.node.__resets)).toBe(1);
// The page still answers.
await chooser(page).setInputFiles(parityPng('f02.png', 'after.png'), { timeout: 8000 });
await expect(preview(page).locator('img.local-file-preview')).toBeVisible({ timeout: 8000 });
@@ -539,6 +577,54 @@ test('a video replaced while still starting is reset once it settles', async ({
}
});
test('leaving for another route while a video starts waits for it to settle', async ({ context, browserName }) => {
test.setTimeout(180000);
// A WebKit (GStreamer) video detached before its pipeline had started deadlocked the page (TESTING.md D62); a
// route change detaches the whole page, so the shell renders the next route once the video reports metadata or an
// error. The route changes from the video's own `loadstart`, or a few milliseconds after it.
for (let attempt = 0; attempt < 15; attempt++) {
const page = await context.newPage();
try {
if (attempt === 0) await startCoverage(page, browserName);
await page.addInitScript(() => {
const load = HTMLMediaElement.prototype.load;
HTMLMediaElement.prototype.load = function () {
if (!this.hasAttribute('src') && window.__starting?.node === this) window.__starting.reset = { readyState: this.readyState, failed: this.error !== null, connected: this.isConnected };
return load.call(this);
};
});
await openInspect(page);
await page.evaluate(delay => {
const slot = document.querySelector('#local-preview');
new MutationObserver((records, observer) => {
const video = slot.querySelector('video');
if (!video) return;
observer.disconnect();
const leave = () => {
window.__starting = { node: video, readyState: video.readyState };
location.hash = '#/overview';
};
video.addEventListener('loadstart', () => { if (delay) setTimeout(leave, delay); else leave(); }, { once: true });
}).observe(slot, { childList: true });
}, attempt % 5);
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(main(page).getByRole('heading', { level: 1, name: 'Overview', exact: true })).toBeVisible({ timeout: 8000 });
const { readyState, reset } = await page.evaluate(() => ({ readyState: window.__starting.readyState, reset: window.__starting.reset }));
if (attempt % 5 === 0) expect(readyState).toBe(0);
// The video was reset and left the page only once it had settled.
expect(reset.readyState >= 1 || reset.failed).toBe(true);
expect(reset.connected).toBe(true);
// The page still answers.
await page.locator('aside.sidebar nav').getByRole('link', { name: 'Inspect a file', exact: true }).click();
await chooser(page).setInputFiles(parityPng('f02.png'), { timeout: 8000 });
await expect(preview(page).locator('img.local-file-preview')).toBeVisible({ timeout: 8000 });
if (attempt === 0) await saveCoverage(page, browserName);
} finally {
await page.close();
}
}
});
test('closing the viewer on a video that just started never blocks the page', async ({ context, browserName }) => {
test.setTimeout(180000);
// The viewer's video teardown resets its element; on WebKit a reset of a still-starting pipeline blocked the page.
@@ -652,13 +738,19 @@ test('local roles send exactly', async ({ page, context }) => {
test('local inspection makes no content request or storage write', async ({ page }) => {
await instrument(page);
await openInspect(page);
// The shell's own reads (counts, Activity) settle before the baseline.
await expect(page.locator('aside.sidebar nav [data-route="images"] .nav-count')).toBeAttached();
const baseline = (await urls(page)).writes.length;
const requests = [];
page.on('request', request => requests.push({ url: request.url(), method: request.method(), body: request.postData() }));
page.on('websocket', socket => requests.push({ url: socket.url(), method: 'WS', body: null }));
const table = /\/prompt_manager\/js\/inspection-semantics\.json$/;
const tableRead = page.waitForResponse(response => table.test(response.url()));
await openInspect(page);
// The one static file Inspect reads, its node table, is fetched as the page mounts: before any selection.
expect((await tableRead).ok()).toBe(true);
// The shell's own reads (counts, Activity) settle before the baseline.
await expect(page.locator('aside.sidebar nav [data-route="images"] .nav-count')).toBeAttached();
const baseline = (await urls(page)).writes.length;
const before = requests.length;
expect(requests.filter(entry => table.test(entry.url))).toHaveLength(1);
for (const file of [sendablePng('private-name.png'), localFixture('core-tags.webm', 'video/webm', 'private-clip.webm')]) {
await choose(page, file);
await openViewer(page);
@@ -666,7 +758,8 @@ test('local inspection makes no content request or storage write', async ({ page
await page.keyboard.press('Escape');
await expect(viewer(page)).toHaveCount(0);
}
const network = requests.filter(entry => !entry.url.startsWith('blob:') && !entry.url.startsWith('data:'));
const network = requests.slice(before).filter(entry => !entry.url.startsWith('blob:') && !entry.url.startsWith('data:'));
expect(requests.filter(entry => table.test(entry.url)), 'the table is read once per page').toHaveLength(1);
expect(network.filter(entry => /private|lighthouse|fox/.test(`${entry.url} ${entry.body ?? ''}`))).toEqual([]);
expect(network.filter(entry => entry.method !== 'GET')).toEqual([]);
expect(network.filter(entry => !/\/prompt_manager\/(activity|images\/output|library|vocabulary)/.test(entry.url))).toEqual([]);
@@ -685,3 +778,80 @@ test('touch layout at 390 has no horizontal overflow at 200% zoom', async ({ pag
expect(area).toBe(true);
}
});
// LI-4 (D117): "Copy settings summary" after the six facts is the legacy metadata page's COPY ALL, in the shared full
// inspector, so a file served in Images and the same bytes chosen here copy the same summary of what the rail shows.
/** Records clipboard writes in the page, or (`clipboard: false`) leaves the page without a clipboard. */
async function stubClipboard(page, { clipboard = true } = {}) {
await page.addInitScript(available => {
window.__copied = [];
Object.defineProperty(navigator, 'clipboard', { configurable: true,
value: available ? { writeText: async text => { window.__copied.push(text); } } : undefined });
}, clipboard);
}
/** The summary the rail's own prompts and facts make, in the legacy COPY ALL lines and placeholders. */
const expectedSummary = locator => locator.evaluate(node => {
const facts = Object.fromEntries([...node.querySelectorAll(':scope .meta-grid:not(details .meta-grid) > div')]
.map(item => [item.querySelector('dt').textContent, item.querySelector('dd').textContent]));
// A confirmed prompt has its Copy action; an explicitly empty one takes the legacy placeholder too.
const prompt = role => {
const text = node.querySelector(`.prompt-section.${role} [data-copy-role]`) ? node.querySelector(`.prompt-section.${role} p.prompt-text`).textContent : null;
return text === `No ${role} prompt (explicitly empty)` ? null : text;
};
return [`Checkpoint: ${facts.Model}`, `Positive Prompt: ${prompt('positive') ?? 'No prompt found'}`,
`Negative Prompt: ${prompt('negative') ?? 'No negative prompt found'}`, `Steps: ${facts.Steps}`, `CFG Scale: ${facts.CFG}`,
`Sampler: ${facts.Sampler}`, `Seed: ${facts.Seed}`].join('\n');
});
async function copySummary(page) {
const button = rail(page).getByRole('button', { name: 'Copy settings summary', exact: true });
// It follows the six facts, before "File & provenance".
expect(await button.evaluate(node => [node.previousElementSibling?.localName, node.nextElementSibling?.localName])).toEqual(['dl', 'details']);
await button.click();
}
test('Copy settings summary copies the same legacy summary in Images and Inspect', async ({ page, request }) => {
await stubClipboard(page);
await expect(await request.post('/__test__/add-local-inspect-file', { data: { name: 'f02.png' } })).toBeOK();
await page.goto(`${workspace}#/images`);
await page.locator('article[data-image-key="0:local-inspect/f02.png"] .image-open').click();
await expect(rail(page).locator('.prompt-section.positive p.prompt-text')).toBeVisible();
const served = await expectedSummary(rail(page));
expect(served).toMatch(/^Checkpoint: .+\nPositive Prompt: [\s\S]+\nNegative Prompt: [\s\S]+\nSteps: .+\nCFG Scale: .+\nSampler: .+\nSeed: .+$/);
expect(served).not.toContain('Unknown');
await copySummary(page);
await expect(page.locator('#workspace-toast')).toHaveText('Settings summary copied.');
expect(await page.evaluate(() => window.__copied)).toEqual([served]);
await page.keyboard.press('Escape');
await expect(viewer(page)).toHaveCount(0);
await openInspect(page);
await choose(page, parityPng('f02.png'));
await openViewer(page);
expect(await expectedSummary(rail(page))).toBe(served);
await copySummary(page);
await expect.poll(() => page.evaluate(() => window.__copied)).toEqual([served, served]);
// The panel variant (the Images side panel) keeps the concept's footer: no summary action there.
await page.keyboard.press('Escape');
await page.goto(`${workspace}#/images`);
await expect(page.locator('#image-inspector .panel-heading')).toBeVisible();
await expect(page.locator('#image-inspector').getByRole('button', { name: 'Copy settings summary', exact: true })).toHaveCount(0);
});
test('Copy settings summary without a clipboard opens the select-to-copy dialog', async ({ page }) => {
await stubClipboard(page, { clipboard: false });
await openInspect(page);
await choose(page, sendablePng());
await openViewer(page);
const summary = await expectedSummary(rail(page));
expect(summary).toContain(`Positive Prompt: ${POSITIVE_LINES}`);
await copySummary(page);
const fallback = page.getByRole('dialog', { name: 'Copy settings summary', exact: true });
await expect(fallback).toBeVisible();
await expect(fallback.getByRole('textbox', { name: 'Select and copy this text', exact: true })).toHaveValue(summary);
await expect(fallback.getByRole('textbox', { name: 'Select and copy this text', exact: true })).toBeFocused();
await page.keyboard.press('Escape');
await expect(fallback).toHaveCount(0);
// Escape closes only the copy dialog; focus returns to the action in the still-open viewer.
await expect(viewer(page)).toBeVisible();
await expect(rail(page).getByRole('button', { name: 'Copy settings summary', exact: true })).toBeFocused();
});
+3 -1
View File
@@ -610,7 +610,9 @@ test('card matches the concept and never wraps the category mid-word',async({pag
await expect(await request.post('/prompt_manager/bulk/category',{data:{prompt_ids:[dual.id],category:''}})).toBeOK();
// A long saved title fills the title row, so only the chip's own min-content keeps "Uncategorized" and "asian" whole.
const wide='A remarkably long saved positive prompt title that fills the whole card title row';
await patchShelf(page,item=>item.id===cat.id?{run_count:4,last_used_at:new Date().toISOString(),rating:5}:item.id===dual.id||item.id===dog.id?{title:wide,run_count:1,last_used_at:null,rating:null}:{});
// One "now" for the patched last use and the page's clock, so it reads "Used today" on either side of midnight.
const usedAt=new Date();await page.clock.setFixedTime(usedAt);
await patchShelf(page,item=>item.id===cat.id?{run_count:4,last_used_at:usedAt.toISOString(),rating:5}:item.id===dual.id||item.id===dog.id?{title:wide,run_count:1,last_used_at:null,rating:null}:{});
for(const [width,height] of [[1600,1100],[390,844]]) {
await page.setViewportSize({width,height});await page.goto(route);await expect(cards(page)).toHaveCount(3);
const card=page.locator(`article[data-prompt-id="${cat.id}"]`);
+252
View File
@@ -0,0 +1,252 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Overview (#249 System S2 Task 6) on the concept-shaped real fixture: the metrics, rows, Statistics dialog and
// Health report read the server, never sample values (D93, D95, D96). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const OVERVIEW_URL = '/prompt_manager/system/overview';
const THUMBNAILS_URL = '/prompt_manager/images/thumbnails/status';
const number = value => value.toLocaleString('en-US');
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
const layoutOf = page => page.locator('#workspace-main .settings-layout');
async function openOverview(page) {
await page.goto(`${workspace}#/system/overview`);
await expect(layoutOf(page).locator(':scope > .summary-metrics')).toBeVisible();
return layoutOf(page);
}
/** The rows of one Overview surface: `[h3, p, button]`. */
function rowsOf(layout, heading) {
return layout.locator('.two-col > section.surface').filter({ has: layout.page().getByRole('heading', { name: heading, exact: true }) })
.locator('.operation').evaluateAll(rows => rows.map(row => [row.querySelector('h3').textContent,
row.querySelector('p').textContent, row.querySelector('.row-actions button').textContent]));
}
test('overview shows real counts and Not checked', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const overview = await serverJson(request, OVERVIEW_URL);
const thumbnails = await serverJson(request, THUMBNAILS_URL);
// The concept fixture: six positives, their saved negatives and 24 linked images, all with thumbnails.
expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]);
// I2: the Overview never walks the sources for thumbnails; the row reads the status route itself.
expect(overview).not.toHaveProperty('thumbnails');
const layout = await openOverview(page);
expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item =>
[item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([
[number(overview.library.positive_prompts), 'Positive prompts'],
[number(overview.library.negative_variants), 'Negative variants'],
[number(overview.images.indexed), 'Images'],
['Not checked', 'Database integrity'],
]);
expect(await rowsOf(layout, 'Library health')).toEqual([
['Ambiguous image links', 'Not available yet.', 'Review metadata'],
['Backup status', 'No retained backup record.', 'View backups'],
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
]);
expect(await rowsOf(layout, 'Workspace preferences')).toEqual([
['One display configuration', 'Grid and density settings are shared across views.', 'Appearance'],
['2 output sources', 'Monitoring and enrichment share one source list.', 'Manage sources'],
['Statistics', 'Runs and ratings by category, with library totals.', 'Explore'],
]);
expect(writes).toEqual([]);
});
test('Run health checks opens Health report and updates metric 4', async ({ page, request }) => {
await page.goto(`${workspace}#/system/maintenance`);
const layout = layoutOf(page);
await expect(layout.locator(':scope > .notice h2')).toHaveText('Check first. Review changes. Then apply.');
// The rest of Maintenance is still the existing tools' (D94).
await expect(layout.locator(':scope > section.handoff h2')).toHaveText('Your current tools are ready');
const run = layout.getByRole('button', { name: 'Run health checks', exact: true });
await run.click();
const dialog = page.locator('dialog#modal[open]');
await expect(dialog.locator('#modal-title')).toHaveText('Health report');
const { report } = await serverJson(request, '/prompt_manager/system/health');
expect(report.status).toBe('incomplete');
expect(await dialog.locator('.operation-list > .operation').evaluateAll(rows => rows.map(row =>
[row.querySelector('h3').textContent, row.querySelector('span.chip').textContent]))).toEqual([
['Integrity check', 'Passed'], ['Foreign keys & pair references', 'Passed'], ['Canonical hashes', 'Passed'],
['Pair uniqueness', 'Passed'], ['Missing files · output', 'Passed'], ['Missing files · input', 'Passed'],
['Ambiguous metadata', 'Not available yet'], ['Consistency', 'Passed'],
]);
await expect(dialog.locator('#modal-body > p.small')).toHaveText(/^Checked .+\. Some checks could not run, so this report is incomplete\.$/);
await expect(dialog.locator('.dialog-actions button')).toHaveText(['Close', 'Review missing links']);
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(run).toBeFocused();
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('Incomplete');
// A missing file and an unmounted source: the report counts the readable sources and names the other one.
await expect(await request.post('/__test__/system/missing-file')).toBeOK();
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Maintenance', exact: true }).click();
await layout.getByRole('button', { name: 'Run health checks', exact: true }).click();
await expect(dialog.locator('.operation').filter({ hasText: 'Missing files' }).locator('.chip'))
.toHaveText(['1 link needs review', 'Passed', 'Not available']);
await expect(dialog.locator('.operation h3').filter({ hasText: 'Missing files' }))
.toHaveText(['Missing files · output', 'Missing files · input', 'Missing files · unmounted-drive']);
// Review missing links goes to Maintenance, where the cleanup's existing tool is linked.
await dialog.getByRole('button', { name: 'Review missing links', exact: true }).click();
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(page).toHaveURL(`${workspace}#/system/maintenance`);
await page.locator('#workspace-main .section-nav').getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.metric').nth(3).locator('strong')).toHaveText('1 issue');
});
test('a health check that fails says so and keeps the last result', async ({ page, request }) => {
await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/health' } })).toBeOK();
await page.goto(`${workspace}#/system/maintenance`);
const run = layoutOf(page).getByRole('button', { name: 'Run health checks', exact: true });
await run.click();
await expect(page.locator('#workspace-toast')).toHaveText('The health check could not finish. Nothing was changed.');
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
await expect(run).not.toHaveAttribute('aria-disabled', 'true');
await openOverview(page);
await expect(layoutOf(page).locator('.metric').nth(3).locator('strong')).toHaveText('Not checked');
});
test('thumbnail row reflects the status endpoint', async ({ page, request }) => {
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
const status = await serverJson(request, '/prompt_manager/images/thumbnails/status');
expect(status.pending).toBe(3);
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('3 missing thumbnails');
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources.
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/sources`);
});
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
const held = async () => (await serverJson(request, '/__test__/hold')).held[THUMBNAILS_URL] ?? 0;
await expect(await request.post('/__test__/hold', { data: { path: THUMBNAILS_URL } })).toBeOK();
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
// The other values do not wait for the walk of every source.
await expect(row.locator('h3')).toHaveText('Checking thumbnails…');
await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4);
await expect.poll(held).toBe(1);
await expect(await request.post('/__test__/release')).toBeOK();
await expect(row.locator('h3')).toHaveText('No missing thumbnails');
});
test('a failed thumbnail read is unavailable, never zero', async ({ page }) => {
await page.route(`**${THUMBNAILS_URL}`, route => route.fulfill({ status: 500, contentType: 'application/json',
body: JSON.stringify({ success: false, error: 'Injected failure' }) }));
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
await expect(row.locator('h3')).toHaveText('Thumbnail status unavailable');
await expect(layout.getByRole('alert')).toHaveCount(0);
await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24');
});
test('ambiguous row reads Not available yet (Q6)', async ({ page }) => {
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ hasText: 'Ambiguous image links' });
await expect(row.locator('p')).toHaveText('Not available yet.');
const review = row.getByRole('button', { name: 'Review metadata', exact: true });
await expect(review).toHaveAttribute('aria-disabled', 'true');
await review.click({ force: true });
await expect(page).toHaveURL(`${workspace}#/system/overview`);
});
test('View backups / Appearance / Manage sources switch sections', async ({ page }) => {
for (const [label, section, pressed] of [['View backups', 'backups', 'Backup & export'], ['Appearance', 'appearance', 'Appearance'],
['Manage sources', 'sources', 'Sources & scans']]) {
const layout = await openOverview(page);
await layout.getByRole('button', { name: label, exact: true }).click();
await expect(page).toHaveURL(`${workspace}#/system/${section}`);
await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText(pressed);
await expect(layout.locator(':scope > .summary-metrics')).toHaveCount(0);
}
});
test('statistics dialog lists runs by category and library totals', async ({ page, request }) => {
const statistics = await serverJson(request, '/prompt_manager/system/statistics');
expect(statistics.by_category.length).toBeGreaterThan(1);
const layout = await openOverview(page);
const explore = layout.getByRole('button', { name: 'Explore', exact: true });
await explore.click();
const dialog = page.locator('dialog#modal[open]');
await expect(dialog.locator('#modal-title')).toHaveText('Usage statistics');
const tables = dialog.locator('#modal-body > table');
await expect(tables.locator('caption')).toHaveText(['Runs by category', 'Library totals']);
await expect(tables.first().locator('thead th')).toHaveText(['Category', 'Runs', 'Avg. rating']);
const cells = table => table.locator('tbody tr').evaluateAll(rows => rows.map(row => [...row.cells].map(cell => cell.textContent)));
expect(await cells(tables.first())).toEqual(statistics.by_category.map(row => [row.category ?? 'Uncategorized',
number(row.runs), row.average_rating === null ? 'Not rated' : `${number(row.average_rating)} / 5`]));
expect(await cells(tables.nth(1))).toEqual([['Prompts', number(statistics.totals.total_prompts)],
['Tags', number(statistics.totals.total_tags)], ['Images', number(statistics.totals.total_images)],
['Rated images', number(statistics.rated_images)]]);
await page.keyboard.press('Escape');
await expect(explore).toBeFocused();
// A statistics read that fails says so; the Overview stays.
await expect(await request.post('/__test__/system/fail', { data: { path: '/prompt_manager/system/statistics' } })).toBeOK();
await explore.click();
await expect(page.locator('#workspace-toast')).toHaveText('Statistics could not be loaded. Your saved data has not changed.');
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
});
test('load failure shows Retry and no zeros', async ({ page, request }) => {
await expect(await request.post('/__test__/system/fail', { data: { path: OVERVIEW_URL } })).toBeOK();
await page.goto(`${workspace}#/system/overview`);
const layout = layoutOf(page);
await expect(layout.getByRole('alert')).toHaveText('The overview could not be loaded. Your saved data has not changed.');
await expect(layout.locator('.metric')).toHaveCount(0);
await expect(layout).not.toContainText('0');
await layout.getByRole('button', { name: 'Retry', exact: true }).click();
await expect(layout.locator('.summary-metrics > .metric')).toHaveCount(4);
await expect(layout.getByRole('alert')).toHaveCount(0);
});
test('out-of-order overview responses never repaint a newer section', async ({ page, request }) => {
const held = async () => (await serverJson(request, '/__test__/hold')).held[OVERVIEW_URL] ?? 0;
await expect(await request.post('/__test__/hold', { data: { path: OVERVIEW_URL } })).toBeOK();
await page.goto(`${workspace}#/system/overview`);
const layout = layoutOf(page);
await expect(layout.getByRole('status')).toHaveText('Loading the overview…');
await expect.poll(held).toBe(1);
// Leave for Appearance, come back (a second read), then leave again before either answers.
const nav = page.locator('#workspace-main .section-nav');
await nav.getByRole('button', { name: 'Appearance', exact: true }).click();
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect.poll(held).toBe(2);
await nav.getByRole('button', { name: 'Sources & scans', exact: true }).click();
await expect(await request.post('/__test__/release')).toBeOK();
await expect.poll(held).toBe(0);
// Both late answers reached a section that had left: Sources keeps its own body.
await expect(layout.locator(':scope > section.handoff')).toBeVisible();
await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve))));
await expect(layout.locator('.summary-metrics')).toHaveCount(0);
expect(await layout.evaluate(node => [...node.children].map(child => child.className))).toEqual(['surface handoff', 'save-row']);
// Back on Overview, one fresh read paints one set of metrics.
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect(layout.locator('.summary-metrics')).toHaveCount(1);
await expect(layout.locator('.two-col')).toHaveCount(1);
});
@@ -0,0 +1,323 @@
// The Images selection bar's Add tags and Auto-tag selected (#249 PR D, plan Task D2, Mock inventory M3; D85, D90,
// D91) against the real `image-tags` job route and the injected fake model (tests/browser/workspace_tagging_fixture.py).
// No model is ever downloaded or loaded for real: the fake answers, and every test checks no download request was sent.
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
const images = '/prompt_manager/workspace#/images';
const FIRST = 'tagging/dunes_00001.png', SECOND = 'tagging/dunes_00002.png', UNLINKED = 'loose/unlinked.png';
const card = (page, path) => page.locator(`article[data-image-key="0:${path}"]`);
const bar = page => page.locator('#workspace-main > #bulkbar > .bulkbar');
const barButton = (page, name) => bar(page).getByRole('button', { name, exact: true });
const modal = page => page.locator('dialog#modal[open]');
const title = page => modal(page).locator('#modal-title');
const toast = page => page.locator('#workspace-toast');
const toasts = page => page.evaluate(() => window.__toasts);
const chips = page => modal(page).locator('#tag-suggestions .chip.tag-suggestion');
const actionButtons = page => modal(page).locator('.dialog-actions button');
const panel = page => page.getByRole('complementary', { name: 'Selected image details', exact: true });
const tagRow = page => panel(page).locator('.detail-inner > .tags').last();
const NOT_DOWNLOADED = 'WD14 SwinV2 is not downloaded. Download it from the legacy Auto-tag page.';
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
// Every toast text in order: a fast job's finish toast may replace its start toast before an assertion reads it.
await page.addInitScript(() => {
window.__toasts = [];
new MutationObserver(() => {
const text = document.getElementById('workspace-toast')?.textContent ?? '';
if (text && window.__toasts.at(-1) !== text) window.__toasts.push(text);
}).observe(document, { subtree: true, childList: true, characterData: true });
});
await expect(await request.post('/__test__/tagging/reset', { data: {} })).toBeOK();
page.taggingErrors = [];
page.on('pageerror', error => page.taggingErrors.push(error.message));
page.downloads = [];
page.on('request', item => { if (/\/autotag\/download\//.test(item.url())) page.downloads.push(item.url()); });
});
test.afterEach(async ({ page, browserName }) => {
await saveCoverage(page, browserName);
expect(page.taggingErrors).toEqual([]);
expect(page.downloads).toEqual([]);
});
/** Images with `paths` checked, in that order. */
async function select(page, ...paths) {
await page.goto(images);
for (const path of paths) await card(page, path).getByRole('checkbox').check();
await expect(bar(page).locator('strong')).toHaveText(`${paths.length} selected`);
}
async function scopeOf(request) {
const listing = await (await request.get('/prompt_manager/images/output?mode=workspace&limit=40')).json();
return listing.images[0].root_scope;
}
/** The saved tags of `path` through the real observe route, then that capture's release. */
async function savedTags(request, path) {
const target = { root_index: 0, root_scope: await scopeOf(request), path };
const observed = await (await request.post('/prompt_manager/image-annotations/observe', { data: { target } })).json();
expect(observed.success).toBe(true);
await request.delete(`/prompt_manager/image-annotations/capture/${observed.capture.token}`);
return observed.tags;
}
/** Wait until the session's job `id` has ended; returns its summary. */
async function ended(request, id) {
let job;
await expect.poll(async () => {
const { jobs } = await (await request.get('/prompt_manager/jobs')).json();
job = jobs.find(item => item.id === id);
return job?.state;
}).toBe('finished');
return job;
}
const jobStart = page => page.waitForRequest(item => item.method() === 'POST' && new URL(item.url()).pathname === '/prompt_manager/jobs/image-tags');
test('selection bar shows Add tags then Auto-tag selected before Set category', async ({ page }) => {
await select(page, FIRST);
await expect(bar(page).locator(':scope > *')).toHaveText(['1 selected', 'Add tags', 'Auto-tag selected', 'Set category', 'Export metadata', 'Clear']);
});
test('Add tags runs a job and the cards show the tags after it finishes', async ({ page, request }) => {
await select(page, SECOND, FIRST);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
await barButton(page, 'Add tags').click();
await expect(title(page)).toHaveText('Tag selected images');
const field = modal(page).getByRole('textbox', { name: 'Tags, separated by commas', exact: true });
await expect(field).toHaveValue('');
await expect(actionButtons(page)).toHaveText(['Cancel', 'Apply']);
let posted = 0;
page.on('request', item => { if (new URL(item.url()).pathname === '/prompt_manager/jobs/image-tags') posted += 1; });
// A field that parses to no names keeps the dialog and its text, and reports why on the field (D118).
await field.fill(' , ,');
await actionButtons(page).last().click();
await expect.poll(() => field.evaluate(node => node.validationMessage)).toBe('Enter at least one tag.');
await expect(title(page)).toHaveText('Tag selected images');
await expect(field).toHaveValue(' , ,');
await field.fill('sunset, Sunset , blue_hour,,');
expect(await field.evaluate(node => node.validity.valid)).toBe(true);
const started = jobStart(page);
const answer = page.waitForResponse(item => new URL(item.url()).pathname === '/prompt_manager/jobs/image-tags');
await actionButtons(page).last().click();
const scope = await scopeOf(request);
// Listing order (the newer FIRST before SECOND), the reveal identity, the dialog's dedupe.
expect((await started).postDataJSON()).toEqual({ mode: 'add', tags: ['sunset', 'blue_hour'], targets: [
{ root_index: 0, root_scope: scope, path: FIRST }, { root_index: 0, root_scope: scope, path: SECOND }] });
const { job } = await (await answer).json();
// The refused Apply sent nothing: only the valid one was posted.
expect(posted).toBe(1);
await expect.poll(() => toasts(page)).toContain('Tagging started.');
expect(await toasts(page)).not.toContain('Enter at least one tag.');
await expect(modal(page)).toHaveCount(0);
await expect(bar(page).locator('strong')).toHaveText('2 selected');
expect((await ended(request, job.id)).counters).toEqual({ tagged: 2, unchanged: 0, skipped: 0, errors: 0 });
// The finished job's files are read again: the selected card's panel shows its new image tags.
await expect(tagRow(page).locator('button.chip')).toHaveText(['sunset', 'blue_hour']);
expect(await savedTags(request, SECOND)).toEqual(['sunset', 'blue_hour']);
expect(await savedTags(request, UNLINKED)).toEqual([]);
});
test('Auto-tag selected with Apply automatically runs a job with the chosen model, thresholds and suggestion count', async ({ page, request }) => {
await select(page, FIRST, UNLINKED);
await barButton(page, 'Auto-tag selected').click();
await expect(title(page)).toHaveText('Auto-tag selected images');
await expect(modal(page)).toHaveClass(/(^| )tagging-modal( |$)/);
const body = modal(page).locator('#modal-body');
expect(await body.evaluate(node => [...node.children].map(child => child.className))).toEqual(['tag-model-panel', 'field', 'small', 'dialog-actions']);
await expect(body.locator('.tag-model-panel h3')).toHaveText('Auto-tag 2 images');
// No preview, manual field, suggestions or Generate: only the model panel's fields.
await expect(body.getByRole('button', { name: 'Generate suggestions', exact: true })).toHaveCount(0);
await expect(body.locator('#manual-image-tags, .tag-suggestion-panel, .tag-image-preview')).toHaveCount(0);
const group = body.getByRole('radiogroup', { name: 'How to apply', exact: true });
await expect(group.getByRole('radio', { name: 'Apply automatically', exact: true })).toBeChecked();
await expect(group.getByRole('radio', { name: 'Review each image', exact: true })).not.toBeChecked();
await expect(actionButtons(page)).toHaveText(['Cancel', 'Start auto-tagging']);
await body.getByRole('combobox', { name: 'Tagging model', exact: true }).selectOption('wd14-vit');
await body.getByRole('spinbutton', { name: 'General confidence', exact: true }).fill('0.5');
await body.getByRole('spinbutton', { name: 'Character confidence', exact: true }).fill('0.9');
const limit = body.getByRole('spinbutton', { name: 'Suggestions', exact: true });
// An invalid count reports its validity and sends nothing.
await limit.fill('0');
let posted = 0;
page.on('request', item => { if (new URL(item.url()).pathname === '/prompt_manager/jobs/image-tags') posted += 1; });
await actionButtons(page).last().click();
expect(await limit.evaluate(node => node.validity.valid)).toBe(false);
await expect(title(page)).toHaveText('Auto-tag selected images');
await limit.fill('2');
const started = jobStart(page);
const answer = page.waitForResponse(item => new URL(item.url()).pathname === '/prompt_manager/jobs/image-tags');
await actionButtons(page).last().click();
const scope = await scopeOf(request);
expect((await started).postDataJSON()).toEqual({ mode: 'auto', model_type: 'wd14-vit',
options: { general_threshold: 0.5, character_threshold: 0.9, suggestion_limit: 2 },
targets: [{ root_index: 0, root_scope: scope, path: FIRST }, { root_index: 0, root_scope: scope, path: UNLINKED }] });
const { job } = await (await answer).json();
// The invalid press sent nothing: only this Start was posted.
expect(posted).toBe(1);
await expect.poll(() => toasts(page)).toContain('Auto-tagging started.');
await expect(modal(page)).toHaveCount(0);
await expect(bar(page).locator('strong')).toHaveText('2 selected');
expect((await ended(request, job.id)).counters).toEqual({ tagged: 2, unchanged: 0, skipped: 0, errors: 0 });
expect(await savedTags(request, FIRST)).toEqual(['sandstone', 'desert']);
expect(await savedTags(request, UNLINKED)).toEqual(['sandstone', 'desert']);
expect(await savedTags(request, SECOND)).toEqual([]);
});
test('a model that is not downloaded keeps Start disabled and says why', async ({ page, request }) => {
await expect(await request.post('/__test__/tagging/models', { data: { unavailable: ['wd14-swinv2'] } })).toBeOK();
await select(page, FIRST);
await barButton(page, 'Auto-tag selected').click();
const status = modal(page).locator('#tag-generation-status');
await expect(status).toHaveAttribute('role', 'status');
await expect(status).toHaveText(NOT_DOWNLOADED);
const start = actionButtons(page).last();
await expect(start).toHaveAttribute('aria-disabled', 'true');
let posted = 0;
page.on('request', item => { if (new URL(item.url()).pathname === '/prompt_manager/jobs/image-tags') posted += 1; });
// aria-disabled: a press does nothing.
await start.dispatchEvent('click');
await expect(title(page)).toHaveText('Auto-tag selected images');
await modal(page).getByRole('combobox', { name: 'Tagging model', exact: true }).selectOption('wd14-vit');
await expect(status).toHaveText('');
await expect(start).not.toHaveAttribute('aria-disabled', 'true');
await modal(page).getByRole('combobox', { name: 'Tagging model', exact: true }).selectOption('wd14-swinv2');
await expect(start).toHaveAttribute('aria-disabled', 'true');
await actionButtons(page).first().click();
await expect(modal(page)).toHaveCount(0);
await expect(barButton(page, 'Auto-tag selected')).toBeFocused();
expect(posted).toBe(0);
});
test('Suggestions field prefills from the setting and caps chips at the entered count', async ({ page, request }) => {
await expect(await request.post('/__test__/tagging/defaults', { data: { general: 0.5, character: 0.9, limit: 7 } })).toBeOK();
await select(page, FIRST);
await barButton(page, 'Auto-tag selected').click();
const limit = modal(page).getByRole('spinbutton', { name: 'Suggestions', exact: true });
await expect(limit).toHaveValue('7');
await expect(modal(page).getByRole('spinbutton', { name: 'General confidence', exact: true })).toHaveValue('0.5');
await expect(modal(page).getByRole('spinbutton', { name: 'Character confidence', exact: true })).toHaveValue('0.9');
await modal(page).getByRole('radio', { name: 'Review each image', exact: true }).check();
await expect(actionButtons(page)).toHaveText(['Cancel', 'Start review']);
await limit.fill('3');
const suggest = page.waitForRequest(item => item.url().endsWith('/image-annotations/suggest'));
await actionButtons(page).last().click();
await expect(title(page)).toHaveText('Tag image · Image 1 of 1');
expect((await suggest).postDataJSON().options).toEqual({ general_threshold: 0.5, character_threshold: 0.9, suggestion_limit: 3 });
await expect(chips(page)).toHaveCount(3);
await expect(modal(page).getByRole('spinbutton', { name: 'Suggestions', exact: true })).toHaveValue('3');
await expect(actionButtons(page)).toHaveText(['Stop reviewing', 'Skip', 'Approve and finish']);
await page.keyboard.press('Escape');
await expect(modal(page)).toHaveCount(0);
await expect(toast(page)).toHaveText('0 of 1 images tagged.');
expect(await savedTags(request, FIRST)).toEqual([]);
});
test('Review each image walks the selection one dialog at a time and only approved images gain tags', async ({ page, request }) => {
await select(page, FIRST, SECOND, UNLINKED);
// The second image's first Generate finds the model busy (a legacy batch): the dialog waits for Retry or Skip.
let suggests = 0;
await page.route('**/prompt_manager/image-annotations/suggest', async route => {
suggests += 1;
if (suggests === 2) {
await route.fulfill({ status: 409, json: { success: false, code: 'model-busy', error: 'The tagging model is busy. Try again when it finishes.' } });
return;
}
await route.continue();
});
const puts = [];
page.on('request', item => { if (item.method() === 'PUT' && item.url().endsWith('/image-annotations/tags')) puts.push(item.postDataJSON()); });
await barButton(page, 'Auto-tag selected').click();
await modal(page).getByRole('radio', { name: 'Review each image', exact: true }).check();
await modal(page).getByRole('spinbutton', { name: 'Suggestions', exact: true }).fill('2');
await actionButtons(page).last().click();
await expect(title(page)).toHaveText('Tag image · Image 1 of 3');
await expect(modal(page).getByRole('combobox', { name: 'Tagging model', exact: true })).toHaveValue('wd14-swinv2');
await expect(chips(page)).toHaveCount(2);
await expect(actionButtons(page)).toHaveText(['Stop reviewing', 'Skip', 'Approve and next']);
await expect(actionButtons(page).last()).toHaveClass(/ca-primary/);
// Approve saves the dialog's manual field plus its remaining chips.
await modal(page).getByRole('textbox', { name: 'Image tags · edit manually', exact: true }).fill('mine');
await modal(page).getByRole('button', { name: 'Remove desert', exact: true }).click();
await actionButtons(page).last().click();
await expect(title(page)).toHaveText('Tag image · Image 2 of 3');
await expect(modal(page).locator('#tag-generation-status')).toHaveText('The tagging model is busy. Try again when it finishes.');
// No suggestions on screen: the primary would save the manual field only (D90).
await expect(actionButtons(page)).toHaveText(['Retry', 'Stop reviewing', 'Skip', 'Save and next']);
await actionButtons(page).nth(2).click();
await expect(title(page)).toHaveText('Tag image · Image 3 of 3');
await expect(chips(page)).toHaveCount(2);
await expect(actionButtons(page)).toHaveText(['Stop reviewing', 'Skip', 'Approve and finish']);
await actionButtons(page).last().click();
await expect(modal(page)).toHaveCount(0);
await expect(toast(page)).toHaveText('2 of 3 images tagged.');
await expect(barButton(page, 'Auto-tag selected')).toBeFocused();
await expect(bar(page).locator('strong')).toHaveText('3 selected');
expect(puts.map(body => [body.target.path, body.tags])).toEqual([[FIRST, ['mine', 'sandstone']], [UNLINKED, ['sandstone', 'desert']]]);
expect(await savedTags(request, FIRST)).toEqual(['mine', 'sandstone']);
expect(await savedTags(request, SECOND)).toEqual([]);
expect(await savedTags(request, UNLINKED)).toEqual(['sandstone', 'desert']);
// The reviewed file's panel shows its saved tags without reselecting.
await expect(tagRow(page).locator('button.chip')).toHaveText(['mine', 'sandstone']);
await expect.poll(async () => (await (await request.get('/__test__/tagging/captures')).json()).captures).toBe(0);
});
test('Review holds Stop reviewing, Escape and × while a save is in flight, and names a manual-only save', async ({ page, request }) => {
await select(page, FIRST, SECOND);
let release;
const held = new Promise(resolve => { release = resolve; });
await page.route('**/prompt_manager/image-annotations/tags', async route => {
if (route.request().method() === 'PUT') await held;
await route.continue();
});
await barButton(page, 'Auto-tag selected').click();
await modal(page).getByRole('radio', { name: 'Review each image', exact: true }).check();
await actionButtons(page).last().click();
await expect(title(page)).toHaveText('Tag image · Image 1 of 2');
await expect(chips(page)).not.toHaveCount(0);
await expect(actionButtons(page)).toHaveText(['Stop reviewing', 'Skip', 'Approve and next']);
// D90: a model change clears the suggestions, so the primary saves the manual field only and says so.
await modal(page).getByRole('combobox', { name: 'Tagging model', exact: true }).selectOption('wd14-vit');
await expect(chips(page)).toHaveCount(0);
await expect(actionButtons(page)).toHaveText(['Stop reviewing', 'Skip', 'Save and next']);
await modal(page).getByRole('textbox', { name: 'Image tags · edit manually', exact: true }).fill('mine');
const put = page.waitForRequest(item => item.method() === 'PUT' && item.url().endsWith('/image-annotations/tags'));
await actionButtons(page).last().click();
await put;
const stop = actionButtons(page).first();
await expect(stop).toHaveAttribute('aria-disabled', 'true');
await expect(actionButtons(page).nth(1)).toHaveAttribute('aria-disabled', 'true');
// None of these ends the session while the save is in flight; the count then includes it.
await page.keyboard.press('Escape');
await modal(page).locator('#modal-close').click();
await stop.dispatchEvent('click'); // aria-disabled: Playwright's click would wait for it
release();
await expect(title(page)).toHaveText('Tag image · Image 2 of 2');
await expect(chips(page)).not.toHaveCount(0);
await expect(actionButtons(page)).toHaveText(['Stop reviewing', 'Skip', 'Approve and finish']);
await expect(actionButtons(page).first()).not.toHaveAttribute('aria-disabled', 'true');
await actionButtons(page).first().click();
await expect(modal(page)).toHaveCount(0);
await expect(toast(page)).toHaveText('1 of 2 images tagged.');
expect(await savedTags(request, FIRST)).toEqual(['mine']);
expect(await savedTags(request, SECOND)).toEqual([]);
});
test('a running scan makes the start report busy', async ({ page, request }) => {
const scan = await (await request.post('/__test__/jobs/start', { data: { kind: 'scan', title: 'Scan for metadata' } })).json();
try {
await select(page, FIRST, SECOND);
await barButton(page, 'Add tags').click();
await modal(page).getByRole('textbox', { name: 'Tags, separated by commas', exact: true }).fill('sunset');
await actionButtons(page).last().click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${scan.job.id}`);
await expect(bar(page).locator('strong')).toHaveText('2 selected');
// Auto-tag is refused the same way.
await barButton(page, 'Auto-tag selected').click();
await expect(actionButtons(page).last()).not.toHaveAttribute('aria-disabled', 'true');
await actionButtons(page).last().click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(bar(page).locator('strong')).toHaveText('2 selected');
expect(await savedTags(request, FIRST)).toEqual([]);
} finally {
await request.post(`/__test__/jobs/${scan.job.id}/finish`, { data: { state: 'finished' } });
}
});
+3
View File
@@ -340,6 +340,9 @@ test('Tag opens the dialog without selecting the card', async ({ page }) => {
const panel = page.getByRole('complementary', { name: 'Selected image details', exact: true });
await page.goto(images);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
// Capture the panel once it shows the first card's details, not while it is still reading them.
await expect(panel.getByRole('heading', { level: 2 })).toBeVisible();
await expect(panel).not.toContainText('Loading file details');
const before = await panel.innerText();
await openTag(page, SECOND, { goto: false });
await page.keyboard.press('Escape');
+96 -1
View File
@@ -18,7 +18,8 @@ const routes = [
['tags', 'Tags & categories', null],
// #249 local Inspect: the native Inspect a file page; Overview's tool card still links the legacy metadata page.
['inspect', 'Inspect a file', null],
['system', 'System', '/prompt_manager/admin'],
// #249 System S2: native System; its undelivered sections hand off per section (D94, asserted below).
['system', 'System', null],
// #249 D51: SAVED VIEWS' Favorites, Images filtered to favorites.
['favorites', 'Favorites', null],
// #249 Activity-1: the concept's Activity page in the tools-and-system group.
@@ -409,6 +410,8 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
await sidebarLink(page, 'System').click();
// System's Overview is native (S2 Task 6); a section not delivered yet still hands off (D94).
await main.locator('.section-nav').getByRole('button', { name: 'Integrations', exact: true }).click();
const handoff = main.locator('section.handoff');
await expect(handoff).toHaveClass(/\bsurface\b/);
expect(await handoff.evaluate(surfaceStyle)).toMatchObject(SURFACE);
@@ -417,6 +420,98 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true);
});
// #249 System S2 Task 5: the concept's renderSystem shell (app.js 698–706) at `#/system/<section>`, every section a D94
// hand-off until its PR, and the Save row's truthful status (D92).
const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance',
'Backup & export', 'Diagnostics & logs'];
const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics'];
const SYSTEM_HANDOFFS = { appearance: ['/prompt_manager/gallery', '/prompt_manager/admin'], sources: ['/prompt_manager/admin', '/prompt_manager/gallery'],
maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => {
const mutations = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) mutations.push(item.url()); });
await page.goto(`${workspace}#/system`);
const main = page.locator('#workspace-main');
await expect(main.locator('h1')).toHaveText('System');
await expect(main.locator('.heading p.muted')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.');
// D92: no "Sample configuration" chip.
await expect(main.locator('.heading .chip')).toHaveCount(0);
expect(await main.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual(['div.heading', 'div.section-nav', 'div.settings-layout']);
const nav = main.getByRole('group', { name: 'System sections', exact: true });
await expect(nav.getByRole('button')).toHaveText(SYSTEM_SECTION_LABELS);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page');
const layout = main.locator('.settings-layout');
await expect(layout.locator(':scope > .save-row > span.small#settings-status')).toHaveText('No unsaved changes.');
const save = layout.locator(':scope > .save-row > button.ca-primary');
await expect(save).toHaveText('Save preferences');
await expect(save).toHaveAttribute('aria-disabled', 'true');
// Playwright waits on an aria-disabled control, so the ignored press is forced.
await save.click({ force: true });
await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.');
for (const [index, label] of SYSTEM_SECTION_LABELS.entries()) {
const hash = SYSTEM_HASHES[index];
const button = nav.getByRole('button', { name: label, exact: true });
await button.click();
await expect(page).toHaveURL(`${workspace}#/system/${hash}`);
await expect(button).toHaveAttribute('aria-pressed', 'true');
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
// The section changes in place: focus stays on the pressed section, not the page title.
await expect(button).toBeFocused();
// S2 Task 6: Overview is native (workspace-system.spec.mjs); Maintenance shows its health-check notice above
// the hand-off (D123); every other section hands off.
if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible();
expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'],
maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash]
?? ['section.surface handoff', 'div.save-row']);
if (hash === 'overview') continue;
const handoff = layout.locator('section.handoff');
await expect(handoff.locator('h2')).toHaveText('Your current tools are ready');
expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href'))))
.toEqual(SYSTEM_HANDOFFS[hash] ?? ['/prompt_manager/admin']);
await expect(layout.locator('#settings-status')).toHaveText('No unsaved changes.');
await expect(page.locator('aside.sidebar a[href="#/system"]')).toHaveAttribute('aria-current', 'page');
}
// Deep links, reload and history keep the section; an unknown section is Overview under its own hash.
await page.reload();
await expect(nav.getByRole('button', { name: 'Diagnostics & logs', exact: true })).toHaveAttribute('aria-pressed', 'true');
await page.goBack();
await expect(page).toHaveURL(`${workspace}#/system/backups`);
await expect(nav.getByRole('button', { name: 'Backup & export', exact: true })).toHaveAttribute('aria-pressed', 'true');
await page.goto(`${workspace}#/system/general`);
await expect(page).toHaveURL(`${workspace}#/system/overview`);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
// The sidebar's System returns to the Overview section.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await sidebarLink(page, 'System').click();
await expect(page).toHaveURL(`${workspace}#/system`);
await expect(nav.getByRole('button', { name: 'Overview', exact: true })).toHaveAttribute('aria-pressed', 'true');
// Leaving System and coming back renders it once, with a fresh section.
await sidebarLink(page, 'Images').click();
await expect(main.locator('h1')).toHaveText('Images');
await expect(main.locator('.section-nav')).toHaveCount(0);
await page.goBack();
await expect(main.locator('.section-nav')).toHaveCount(1);
await expect(main.locator('.settings-layout')).toHaveCount(1);
expect(mutations).toEqual([]);
});
test('Overview opens native System from its System card (Q12)', async ({ page }) => {
await page.goto(`${workspace}#/overview`);
const card = page.locator('#workspace-main .tool-card').filter({ has: page.getByRole('heading', { name: 'System', exact: true }) });
await expect(card.locator('p.small')).toHaveText('Configuration, maintenance and recovery—with browsing kept in the workspace.');
const link = card.getByRole('link', { name: 'Open System', exact: true });
await expect(link).toHaveAttribute('href', '#/system/overview');
await link.click();
await expect(page.locator('#workspace-main h1')).toHaveText('System');
await expect(page.locator('#workspace-main .section-nav [aria-pressed="true"]')).toHaveText('Overview');
});
// Task A3: the concept sidebar (index.html h16-41, app.js nav()) rendered from real data (D27, D28).
const NAV_TOTALS = [
['images', 'Images', '/prompt_manager/images/output?mode=workspace&limit=1'],
+66 -9
View File
@@ -4,7 +4,9 @@ A fake job is a real registry job whose runner waits for commands instead of
reading files, so a test sets its progress and counters exactly:
- ``POST /__test__/jobs/start`` starts one (``kind``, ``title``, ``total``,
``output_folders``) and returns its summary, or 409 while one runs;
``output_folders``; for ``kind: "image-tags"`` the image tagging job's
groups and its ``{mode, images}`` scope instead) and returns its summary,
or 409 while one runs;
- ``POST /__test__/jobs/{id}/step`` records ``records`` (``{group, path,
root, root_index, detail}`` file outcomes, counted, listed and logged as the scan
records them) and ``bulk`` (``{group, count, root}``: ``count`` numbered
@@ -12,6 +14,9 @@ reading files, so a test sets its progress and counters exactly:
the progress to every subscriber;
- ``POST /__test__/jobs/{id}/finish`` ends it as ``finished``, ``cancelled``
or ``failed``; a cancel through the real route ends it ``cancelled``;
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``):
a running, non-cancellable ``records`` job, as a legacy restore or
maintenance request takes, until ``finish`` ends it;
- ``POST /__test__/jobs/restart`` acts out a ComfyUI restart: the open job
streams drop without a final event and the registry starts empty.
"""
@@ -28,6 +33,8 @@ POLL_SECONDS = 0.02
PREFIXES = {"added": "+", "known": "=", "review": "?", "errors": "!"}
PREFIXES["no_metadata"] = "-"
# The image tagging job's groups (#249 PR D), logged with the runner's prefixes.
PREFIXES.update({"tagged": "+", "unchanged": "=", "skipped": "-"})
ROOT_INDEXES = {"output": 0, "input": 1}
@@ -72,10 +79,17 @@ def outcome_records(data):
def register_activity_commands(app, routes, api):
from py.api.jobs import SCAN_GROUP_NAMES, JobBusy, JobRegistry
from py.api.jobs import (
IMAGE_TAG_GROUP_NAMES,
IMAGE_TAG_JOB_KIND,
SCAN_GROUP_NAMES,
JobBusy,
JobRegistry,
)
streams = set()
endings = {}
holders = set()
@web.middleware
async def track_streams(request, handler):
@@ -123,17 +137,29 @@ def register_activity_commands(app, routes, api):
async def start_job(request):
data = await body(request)
ending = asyncio.get_running_loop().create_future()
kind = str(data.get("kind", "scan"))
if kind == IMAGE_TAG_JOB_KIND:
title = "Add tags to images"
scope = {
"mode": str(data.get("mode", "add")),
"images": count(data.get("images"), 0),
}
groups = IMAGE_TAG_GROUP_NAMES
else:
title = "Scan for metadata"
scope = {
"files": str(data.get("files", "new")),
"output_folders": count(data.get("output_folders"), 2),
}
groups = SCAN_GROUP_NAMES
try:
job = api._jobs.start(
str(data.get("kind", "scan")),
str(data.get("title", "Scan for metadata")),
kind,
str(data.get("title", title)),
record_changing=True,
scope={
"files": str(data.get("files", "new")),
"output_folders": count(data.get("output_folders"), 2),
},
scope=scope,
runner=fake_runner(ending),
groups=SCAN_GROUP_NAMES,
groups=groups,
)
except JobBusy:
raise web.HTTPConflict()
@@ -141,6 +167,36 @@ def register_activity_commands(app, routes, api):
endings[job.id] = ending
return web.json_response({"success": True, "job": job.summary()})
@routes.post("/__test__/jobs/hold")
async def hold_job(request):
data = await body(request)
loop = asyncio.get_running_loop()
ending = loop.create_future()
entered = loop.create_future()
async def hold():
async with api._jobs.hold(
str(data.get("kind", "restore")),
str(data.get("title", "Database restore (admin page)")),
admission="records",
) as job:
entered.set_result(job)
state = await ending
if state != "finished":
job.finish(state)
holder = asyncio.ensure_future(hold())
holders.add(holder)
holder.add_done_callback(holders.discard)
await asyncio.wait({entered, holder}, return_when=asyncio.FIRST_COMPLETED)
if not entered.done():
if isinstance(holder.exception(), JobBusy):
raise web.HTTPConflict()
raise holder.exception()
job = entered.result()
endings[job.id] = ending
return web.json_response({"success": True, "job": job.summary()})
@routes.post("/__test__/jobs/{job_id}/step")
async def step_job(request):
job = job_or_404(request)
@@ -188,6 +244,7 @@ def register_activity_commands(app, routes, api):
for transport in list(streams):
transport.close()
tasks = [job.task for job in old.list() if job.task and not job.task.done()]
tasks += [holder for holder in holders if not holder.done()]
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
+35 -2
View File
@@ -11,6 +11,7 @@ the same sidebar counts (24 images, 6 prompts, 15 tags). Nothing is mocked.
from __future__ import annotations
from dataclasses import dataclass
import datetime
import json
import math
import os
@@ -157,6 +158,8 @@ CONCEPT_MODEL = "illustrative-model.safetensors"
MAX_EXTRA_IMAGES = 120
# Unlinked, seedless images (D11 "W × H · PNG") newer than the concept's, for D3 checks.
MAX_SEEDLESS_IMAGES = 2
# A test's chosen "now" (epoch seconds) stays before the year 3000.
MAX_EPOCH_SECONDS = 32_503_680_000
SEEDLESS_SIZE = (1536, 1152)
# The same 4:3 shape with a short "96 × 72 · PNG" subtitle that stays on one line at
# 390px beside the card's Info and Tag (the D3 footer check).
@@ -342,6 +345,17 @@ def _bounded(name: str, value: int, maximum: int) -> None:
raise ValueError(f"{name} is out of range")
def _instant(now: float | None) -> float:
"""The fixture's one "now": ``now`` itself, or the wall clock when omitted."""
if now is None:
return time.time()
if isinstance(now, bool) or not isinstance(now, (int, float)):
raise TypeError("now must be a number of epoch seconds")
if not 0 < now < MAX_EPOCH_SECONDS:
raise ValueError("now is out of range")
return float(now)
def seed_workspace_concept(
api: PromptManagerAPI,
root: Path,
@@ -349,9 +363,16 @@ def seed_workspace_concept(
extra_images: int = 0,
seedless_images: int = 0,
short_seedless: bool = False,
now: float | None = None,
) -> dict:
"""Replace this server's disposable rows and files with the concept's data.
Every relative time the fixture writes (the prompts' last use, the files'
mtimes, the runs' observation times) is taken from one instant, ``now``
(epoch seconds, the wall clock when omitted), returned as ``manifest["now"]``.
A browser whose clock is fixed at that instant reads the concept's "Used
today" and caption day however close to midnight the run starts.
``extra_images`` continues the concept's rounds (same prompts, dimensions and
seed rule) after its 24 images, for bounded-rendering and Load more checks.
``seedless_images`` adds unlinked PNGs without metadata, newest of all;
@@ -361,6 +382,10 @@ def seed_workspace_concept(
raise TypeError("short_seedless must be a boolean")
_bounded("extra_images", extra_images, MAX_EXTRA_IMAGES)
_bounded("seedless_images", seedless_images, MAX_SEEDLESS_IMAGES)
now = _instant(now)
used_at = datetime.datetime.fromtimestamp(now, datetime.timezone.utc).isoformat(
timespec="microseconds"
)
_clear(api, root)
db = api.db
epoch = db.get_execution_epoch()
@@ -385,8 +410,15 @@ def seed_workspace_concept(
db.update_prompt_pair_metadata(pair_ids[0], title=first_name)
for name, text in prompt.negatives[1:]:
pair_ids.append(db.save_prompt_pair(prompt.positive, text, title=name))
# Real queue-hook counting (D23): four runs, the last one now.
# Real queue-hook counting (D23): four runs, the last one now, at the
# fixture's own instant rather than the database's later wall clock.
db.record_prompt_use(positive_id, times=CONCEPT_RUN_COUNT)
with db.model.get_connection() as conn:
conn.execute(
"UPDATE prompts SET last_used_at = ? WHERE id = ?",
(used_at, positive_id),
)
conn.commit()
prompts.insert(
0, {"id": prompt.id, "positive_id": positive_id, "pair_ids": pair_ids}
)
@@ -400,7 +432,6 @@ def seed_workspace_concept(
favorite_root = _favorite_root_key(list(api._get_all_output_dirs())[0])
total = len(CONCEPT_PROMPTS) * CONCEPT_IMAGE_ROUNDS + extra_images
rounds = math.ceil(total / len(CONCEPT_PROMPTS))
now = time.time()
images, runs = [], []
slots = [
(round_index, prompt_index, prompt)
@@ -484,6 +515,8 @@ def seed_workspace_concept(
api.invalidate_gallery_cache()
return {
"epoch": epoch,
# The fixture's "now" (epoch seconds), for the browser's fixed clock.
"now": now,
# The newest file's mtime (epoch seconds), for the D17 caption date.
"newest_modified": now - 10 if seedless_images else now - 60,
"seedless": seedless,
+116
View File
@@ -0,0 +1,116 @@
"""System test hooks for the browser tests (#249 System S2 Task 6).
The data is the concept-shaped real fixture (``POST /__test__/concept/reset``,
``workspace_concept_fixture.py``): its parents, saved pairs, images, ratings
and runs are what the Overview, the Statistics dialog and the health check
read. These per-server hooks only change what a System test needs on top:
- ``POST /__test__/system/reset`` restores the configured sources, forgets the
session's health report and ends every injected failure;
- ``POST /__test__/system/fail`` ``{path, times}`` answers the next ``times``
requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes'
500 error envelope;
- ``POST /__test__/system/missing-source`` configures one more source that
does not exist (an unmounted drive);
- ``POST /__test__/system/missing-file`` deletes the newest linked image's
file from disk, leaving its database row (a missing-file link);
- ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain
PNGs with no thumbnail into the output folder.
Delays use the shared request holds (``request_hold.py``).
"""
import os
from pathlib import Path
from aiohttp import web
from PIL import Image
FAILABLE_PREFIX = "/prompt_manager/system/"
MAX_UNTHUMBNAILED = 50
def register_system_commands(app, routes, api, root: Path):
from py.config import GalleryConfig
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
failures: dict[str, int] = {}
@web.middleware
async def inject_failures(request, handler):
left = failures.get(request.path, 0)
if left <= 0:
return await handler(request)
failures[request.path] = left - 1
return web.json_response(
{"success": False, "code": None, "error": "Injected failure"}, status=500
)
app.middlewares.append(inject_failures)
def reset():
failures.clear()
GalleryConfig.MONITORING_DIRECTORIES = list(configured)
api._health_report = None
@routes.post("/__test__/system/reset")
async def system_reset(request):
reset()
return web.json_response({"success": True})
@routes.post("/__test__/system/fail")
async def system_fail(request):
data = await request.json()
path, times = data.get("path"), data.get("times", 1)
if (
not isinstance(path, str)
or not path.startswith(FAILABLE_PREFIX)
or isinstance(times, bool)
or not isinstance(times, int)
or not 1 <= times <= 10
):
raise web.HTTPBadRequest()
failures[path] = times
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-source")
async def missing_source(request):
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent]
return web.json_response(
{"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)}
)
@routes.post("/__test__/system/missing-file")
async def missing_file(request):
with api.db.model.get_connection() as conn:
row = conn.execute(
"SELECT file_path FROM generated_images ORDER BY id DESC LIMIT 1"
).fetchone()
if row is None:
raise web.HTTPNotFound()
Path(row[0]).unlink()
api.invalidate_gallery_cache()
return web.json_response({"success": True, "filename": Path(row[0]).name})
@routes.post("/__test__/system/unthumbnailed")
async def unthumbnailed(request):
data = await request.json()
count = data.get("count")
if (
isinstance(count, bool)
or not isinstance(count, int)
or not 1 <= count <= MAX_UNTHUMBNAILED
):
raise web.HTTPBadRequest()
folder = root / "output" / "unthumbnailed"
folder.mkdir(exist_ok=True)
for number in range(count):
with Image.new("RGB", (64, 64), "#2b3442") as image:
image.save(folder / f"plain_{number + 1:03}.png")
api.invalidate_gallery_cache()
return web.json_response({"success": True})
return reset
File diff suppressed because one or more lines are too long
+29
View File
@@ -236,6 +236,35 @@ test("latestOnly: errors from the current call propagate, errors from superseded
assert.equal(await first, STALE);
});
test("latestOnly with a lifetime: once it ends, no request starts and the call is STALE", async () => {
const fetch = fakeFetch();
const api = createApiClient(fetch);
const page = new AbortController();
const latest = latestOnly({ lifetime: page.signal });
page.abort();
assert.equal(await latest((signal) => api.get("/recent", { signal })), STALE);
assert.equal(fetch.calls.length, 0, "a leaving page starts no read");
});
test("latestOnly with a lifetime: its end aborts the request in flight, which is STALE, not an error", async () => {
const fetch = fakeFetch();
const api = createApiClient(fetch);
const page = new AbortController();
const latest = latestOnly({ lifetime: page.signal });
const call = latest((signal) => api.get("/recent", { signal }));
page.abort();
assert.equal(fetch.calls[0].aborted, true, "the read in flight is aborted");
fetch.calls[0].reject(new DOMException("The operation was aborted.", "AbortError"));
assert.equal(await call, STALE);
});
test("latestOnly with a lifetime: real errors still propagate while the page lives", async () => {
const page = new AbortController();
const latest = latestOnly({ lifetime: page.signal });
await assert.rejects(latest(async () => { throw new Error("server down"); }), /server down/);
assert.equal(await latest(async () => "fine"), "fine");
});
test("ApiError exposes only validated history recovery actions on all error paths", async () => {
const permitted = [null, { action: "refresh-history" }, { action: "reload-outputs" },
{ action: "retry-read" }, { action: "smaller-limit", limit: 1 },
+22
View File
@@ -73,6 +73,28 @@ test('Not Now persists the collection state and suppresses the unchanged prompt'
assert.equal(f.prompts.length, 1);
});
// WebKit fails a fetch a leaving page starts ("Fetch API cannot load … due to access control checks.") and logs it
// as a page error: the delayed startup check starts no read once the page has begun to leave (as admin.js, #290).
test('the startup check starts no read once the page is leaving', async () => {
const f = fixture({ success: true, total_images: 100, pending: 30, collection_id: 'a' });
f.manager.pageLifetime = new AbortController();
f.manager.pageLifetime.abort();
await f.check();
assert.equal(f.requests.length, 0);
assert.equal(f.prompts.length, 0);
});
test('the startup check passes the page lifetime to its read', async () => {
const f = fixture({ success: true, total_images: 100, pending: 30, collection_id: 'a' });
const signals = [];
f.manager.pageLifetime = new AbortController();
const get = f.manager.api.get;
f.manager.api.get = async (url, options) => { signals.push(options?.signal); return get(url); };
await f.check();
assert.deepEqual(signals, [f.manager.pageLifetime.signal]);
assert.equal(f.prompts.length, 1);
});
test('disabled checks do not request collection status', async () => {
const f = fixture({}, { checkThumbnailsAtStartup: false });
await f.check();
+150
View File
@@ -0,0 +1,150 @@
// The legacy admin and gallery pages show a server refusal's reason (#220, PR #289 review I1 and M8): a job that
// runs in the workspace refuses their record-changing requests, and a corrupt config.json refuses a settings save.
// Each page must name the reason, not report a lost connection, a bare HTTP status or a success.
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const vm = require('node:vm');
function elements() {
const nodes = {};
return {
nodes,
getElementById: id => nodes[id] ||= { id, textContent: '', innerHTML: '', disabled: false, style: {},
value: '', checked: false, addEventListener() {}, querySelectorAll: () => [] },
};
}
function jsonResponse(status, body) {
return { ok: status >= 200 && status < 300, status, statusText: status === 409 ? 'Conflict' : 'OK',
json: async () => body };
}
function sseResponse(frames) {
const bytes = new TextEncoder().encode(frames.map(frame => `data: ${JSON.stringify(frame)}\n\n`).join(''));
let sent = false;
return { ok: true, status: 200, body: { getReader: () => ({
read: async () => (sent ? { done: true } : (sent = true, { done: false, value: bytes })),
}) } };
}
function load(file, className, fetchUrl, extra = {}, dom = elements()) {
const timers = [];
const notifications = [];
const storage = new Map([['pm_cached', '1'], ['userData', 'keep']]);
const localStorage = {
getItem: key => storage.get(key) ?? null,
setItem: (key, value) => storage.set(key, String(value)),
removeItem: key => storage.delete(key),
};
const context = vm.createContext({
document: { readyState: 'loading', addEventListener() {}, getElementById: dom.getElementById,
querySelectorAll: () => [] },
window: {}, console: { error() {}, warn() {}, debug() {} },
setTimeout: callback => timers.push(callback),
confirm: () => true,
localStorage,
Object: { ...Object, keys: target => (target === localStorage ? [...storage.keys()] : Object.keys(target)) },
TextDecoder, TextEncoder, JSON,
ApiClient: { fetchUrl },
...extra,
});
const source = fs.readFileSync(path.join(__dirname, `../../web/js/${file}`), 'utf8');
const Page = vm.runInContext(`${source}\n${className};`, context, { filename: path.join(__dirname, `../../web/js/${file}`) });
const page = Object.create(Page.prototype);
page.showNotification = (message, type) => notifications.push([type, message]);
page.showModal = id => notifications.push(['show', id]);
page.hideModal = id => notifications.push(['hide', id]);
return { page, dom, timers, notifications, storage };
}
const BUSY = 'Scan for metadata already running';
test('gallery cache clear reports a refused server clear and never claims success', async () => {
const calls = [];
const { page, dom, notifications, storage } = load('gallery.js', 'GalleryManager', async url => {
calls.push(url);
return jsonResponse(409, { success: false, error: BUSY });
});
page.getSettings = () => ({ thumbnailsGenerated: true });
dom.getElementById('thumbnailStatus').textContent = 'Generated';
await page.clearCache();
assert.deepEqual(calls, ['/prompt_manager/images/clear-thumbnails']);
assert.deepEqual(notifications, [['error', `Thumbnails were not cleared: ${BUSY}`]]);
assert.equal(dom.nodes.thumbnailStatus.textContent, 'Generated');
assert.equal(storage.get('gallerySettings'), undefined); // thumbnailsGenerated is not reset
});
test('gallery cache clear still says so when the server clears', async () => {
const { page, dom, notifications, storage } = load('gallery.js', 'GalleryManager',
async () => jsonResponse(200, { success: true, cleared_count: 3 }));
page.getSettings = () => ({ thumbnailsGenerated: true });
await page.clearCache();
assert.deepEqual(notifications, [['success', 'PromptManager cache cleared successfully! Original images untouched.']]);
assert.equal(dom.nodes.thumbnailStatus.textContent, 'Not generated');
assert.equal(JSON.parse(storage.get('gallerySettings')).thumbnailsGenerated, false);
assert.equal(storage.get('userData'), 'keep');
assert.equal(storage.has('pm_cached'), false);
});
test('admin LoRA import shows a refusal frame and closes its modal', async () => {
const { page, dom, timers, notifications } = load('admin.js', 'PromptAdmin',
async () => sseResponse([{ type: 'error', status: BUSY, progress: 0, message: BUSY }]));
page.saveLoraSettings = async () => true;
page.loadStatistics = () => assert.fail('a refused import must not reload statistics');
await page.runLoraImport();
assert.equal(dom.nodes.loraImportStatus.textContent, `Error: ${BUSY}`);
assert.deepEqual(notifications.filter(([type]) => type === 'error'), [['error', `LoRA import failed: ${BUSY}`]]);
assert.equal(timers.length, 1);
timers[0]();
assert.deepEqual(notifications.at(-1), ['hide', 'loraImportModal']);
});
test('admin settings save shows why config.json refused it', async () => {
const message = 'config.json cannot be read, so nothing was saved. Fix or remove the file, then save again.';
const { page, notifications } = load('admin.js', 'PromptAdmin',
async () => jsonResponse(409, { success: false, persisted: false, code: 'config-unreadable', error: message }));
page.settings = {};
page._collectScanPaths = () => [];
page.saveLoraSettings = async () => assert.fail('nothing else is saved after a refusal');
await page.saveSettings();
assert.deepEqual(notifications, [['error', `Failed to save settings: ${message}`]]);
});
test('admin settings save keeps its plain message when the reply has no reason', async () => {
const { page, notifications } = load('admin.js', 'PromptAdmin', async () => ({ ok: false, status: 500,
json: async () => { throw new SyntaxError('not json'); } }));
page.settings = {};
page._collectScanPaths = () => [];
await page.saveSettings();
assert.deepEqual(notifications, [['error', 'Failed to save settings']]);
});
test('admin maintenance names the job that refused it', async () => {
const { page, dom, notifications } = load('admin.js', 'PromptAdmin',
async () => jsonResponse(409, { success: false, error: BUSY }), {
escapeHtml: text => String(text),
document: { readyState: 'loading', addEventListener() {}, getElementById: id => dom.getElementById(id),
querySelectorAll: selector => (selector === '.maintenance-option:checked' ? [{ id: 'vacuum' }] : []) },
});
await page.runMaintenance();
const results = dom.nodes.maintenanceResults.innerHTML;
assert.match(results, /Maintenance Failed/);
assert.match(results, new RegExp(BUSY));
assert.doesNotMatch(results, /HTTP 409/);
assert.deepEqual(notifications, [['error', '❌ Maintenance failed']]);
assert.equal(dom.nodes.runMaintenanceBtn.disabled, false);
});
+34
View File
@@ -14,6 +14,11 @@
// the carrier rule alone, as the server selects from ffprobe's tag map.
// A {records} entry (records null = an unreadable file) answers
// {classified: null, result: null, metadata} for the projection alone.
// A {mode: 'resolve'} entry (PR L3) answers {evidence}: resolveLocalEvidence of its {records} texts
// with the repository semantics table. With a {path} it reads the file locally and also answers
// {facts} (and {unreadable: true} with the server's unreadable evidence when nothing could be read):
// extractGenerationFacts on the local payload through projectInspectionPayload, exactly as
// tests/js/support/inspection-parity-facts.mjs reads the server's response for vectors.json.
import fs from 'node:fs';
import path from 'node:path';
import { createRequire } from 'node:module';
@@ -28,6 +33,22 @@ const web = path.join(here, '..', '..', '..', 'web', 'js', 'workspace');
const { selectVideoCarrier } = await import(moduleUrl(path.join(web, 'local-video.mjs')));
const { projectMetadata } = await import(moduleUrl(path.join(web, 'local-projection.mjs')));
const { readLocalFile, toInspectionRead } = await import(moduleUrl(path.join(web, 'local-inspection.mjs')));
const { resolveLocalEvidence, compileSemantics, unreadableEvidence } = await import(moduleUrl(path.join(web, 'local-resolver.mjs')));
const { buildLocalPayload } = await import(moduleUrl(path.join(web, 'local-inspection.mjs')));
const { projectInspectionPayload } = await import(moduleUrl(path.join(web, 'inspection-state.mjs')));
const semantics = compileSemantics(JSON.parse(fs.readFileSync(path.join(here, '..', '..', '..', 'web', 'js', 'inspection-semantics.json'), 'utf8')));
const { extractGenerationFacts } = require(path.join(here, '..', '..', '..', 'web', 'js', 'comfy-metadata.js'));
/** The local evidence for one set of record texts. */
const resolved = records => ({ evidence: resolveLocalEvidence(records, semantics) });
/** The local wire's evidence and six facts for one read (no legacy fill, as in the vectors' own harness). */
function localPayload(read, file, classification) {
const payload = buildLocalPayload({ read, file, classification, semantics });
const record = projectInspectionPayload(payload, { kind: 'local-file', key: 'local:1', path: file.name }, null, extractGenerationFacts);
const facts = Object.fromEntries(Object.entries(record.generation).map(([field, fact]) => [field, { status: fact.status, value: fact.value }]));
return { evidence: { prompts: payload.prompts, resolution: payload.resolution }, facts };
}
const selectionRead = (selection) => selection.status === 'complete' ?
{ records: selection.records, record_formats: selection.record_formats, carrierText: selection.carrierText, limited: selection.limited } : null;
@@ -37,6 +58,19 @@ for await (const chunk of process.stdin) chunks.push(chunk);
const output = {};
for (const entry of JSON.parse(Buffer.concat(chunks).toString('utf8'))) {
const { name } = entry;
if (entry.mode === 'resolve') {
if (entry.records !== undefined) { output[name] = resolved(entry.records); continue; }
const blob = await fs.openAsBlob(entry.path);
const { classification, result } = await readLocalFile(blob, { pngMetadata });
const read = classification === null ? null : toInspectionRead(classification, result).read;
if (classification === null) { output[name] = { unreadable: true }; continue; }
// The evidence is the local wire's own (buildLocalPayload), and it equals resolveLocalEvidence of the records.
const local = localPayload(result, { name: path.basename(entry.path), size: blob.size }, classification);
const direct = read === null ? unreadableEvidence() : resolved(read.records).evidence;
if (JSON.stringify(direct) !== JSON.stringify(local.evidence)) throw new Error(`${name}: the wire's roles are not the resolver's`);
output[name] = read === null ? { unreadable: true, ...local } : local;
continue;
}
if (entry.records !== undefined) {
const read = entry.records === null ? null : {
records: entry.records,
+66 -1
View File
@@ -182,6 +182,24 @@ test('start returns busy with the running job', async () => {
world.dispose();
});
test('track follows a job another route started, with no read of its own', async () => {
const { createActivityStore } = await source;
const { world, ...deps } = harness();
const store = createActivityStore(deps);
const seen = [];
store.subscribe(() => seen.push(store.find(ID_A)?.state));
const job = store.track(summary({ kind: 'health', title: 'Health check' }));
assert.equal(job.id, ID_A);
assert.equal(world.gets, 0, 'the started job is known without a list read');
assert.deepEqual(world.open().map(stream => stream.url), [`/prompt_manager/jobs/${ID_A}/events`]);
assert.deepEqual(world.toasts, [], 'the route that started it says so itself');
send(world.open()[0], summary({ kind: 'health', title: 'Health check', state: 'finished', progress: 100 }));
assert.deepEqual(seen, ['running', 'finished']);
assert.equal(world.open().length, 0, 'an ended job is no longer followed');
assert.throws(() => store.track({ id: 'nope' }), /Invalid job/);
world.dispose();
});
test('a list that is not a job list counts as a failed read', async () => {
const { createActivityStore } = await source;
const { world, api, ...deps } = harness();
@@ -261,8 +279,13 @@ test('summaries are validated and the copy follows the spec', async () => {
const loose = parseJob({ id: ID_A, state: 'running', progress: 140.4, processed: -1, total: 1.5, counters: { added: 'x', errors: 2 },
title: ' ', started_at: 'x', scope: [] });
assert.deepEqual(loose, { id: ID_A, kind: '', title: 'Background job', state: 'running', started_at: null, finished_at: null,
progress: 100, status_line: '', processed: 0, total: 0, counters: { added: 0, errors: 2 }, scope: {}, listed: {}, log_length: 0 });
progress: 100, status_line: '', processed: 0, total: 0, counters: { added: 0, errors: 2 }, scope: {}, listed: {}, log_length: 0,
cancellable: true });
assert.equal(parseJob(summary({ progress: -3 })).progress, 0);
// A held legacy job cannot be cancelled; an older server sends no key, which keeps Cancel.
assert.equal(parseJob(summary({ cancellable: false })).cancellable, false);
assert.equal(parseJob(summary({ cancellable: true })).cancellable, true);
assert.equal(parseJob(summary()).cancellable, true);
const scan = parseJob(summary({ progress: 42, processed: 8240, total: 19622, counters: { added: 1, errors: 1 } }));
assert.deepEqual(pillParts(scan), { label: 'Scanning · 42%', files: ' · 8,240 / 19,622', text: 'Scanning · 42% · 8,240 / 19,622' });
@@ -283,6 +306,48 @@ test('summaries are validated and the copy follows the spec', async () => {
assert.deepEqual([-1, 0, 1, 4, 5, 6, 1e9, 0.5].map(backoffDelay), [1000, 1000, 2000, 16000, 30000, 30000, 30000, 1000]);
});
// D89 (plan D66): an image tagging job's view has its own tiles, outcomes, openable rows, log and cancel labels.
test('image-tags jobs get their own tiles outcomes and pill verb', async () => {
const { parseJob, jobView, pillParts, outcomeLine, startToast } = await source;
assert.equal(typeof jobView, 'function');
const job = parseJob(summary({ kind: 'image-tags', title: 'Auto-tag images', progress: 40, processed: 2, total: 5,
counters: { tagged: 1420, unchanged: 3, skipped: 2, errors: 1 }, scope: { mode: 'auto', images: 5, model_type: 'wd14-vit' } }));
const view = jobView(job.kind);
assert.deepEqual(view.tiles, [['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'],
['errors', 'Errors', 'bad']]);
assert.deepEqual(view.outcomes, [['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'], ['unchanged', 'Unchanged']]);
assert.deepEqual([...view.openable].sort(), ['tagged', 'unchanged']);
assert.equal(view.log, 'Tagging log');
assert.equal(view.cancel, 'Cancel tagging');
assert.deepEqual(pillParts(job), { label: 'Tagging · 40%', files: ' · 2 / 5', text: 'Tagging · 40% · 2 / 5' });
assert.equal(outcomeLine(job), '1,420 tagged · 1 error');
assert.equal(outcomeLine({ ...job, counters: {} }), '0 tagged · 0 errors');
assert.equal(startToast(job), 'Auto-tagging started.');
// The design's toast pattern, as the scan's: "<Job> finished · N tagged · M errors".
const { finishToast } = await source;
assert.equal(finishToast({ ...job, state: 'finished' }), 'Auto-tag images finished · 1,420 tagged · 1 error');
assert.equal(finishToast({ ...job, state: 'cancelled' }), 'Auto-tag images cancelled · 1,420 tagged');
assert.equal(finishToast({ ...job, state: 'failed' }), 'Auto-tag images failed · 1,420 tagged · 1 error');
assert.equal(finishToast({ ...job, title: 'Add tags to images', counters: {}, state: 'finished' }),
'Add tags to images finished · 0 tagged · 0 errors');
});
test('unknown kinds keep the generic view', async () => {
const { jobView } = await source;
const scan = jobView('scan');
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]);
assert.deepEqual(scan.outcomes, [['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
['no_metadata', 'No usable metadata'], ['known', 'Already known']]);
assert.deepEqual([...scan.openable].sort(), ['added', 'known']);
assert.deepEqual([scan.log, scan.cancel], ['Scan log', 'Cancel scan']);
for (const kind of ['maintenance', '', 'constructor', '__proto__']) {
const view = jobView(kind);
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [scan.tiles, scan.outcomes, [...scan.openable]]);
assert.deepEqual([view.log, view.cancel], ['Job log', 'Cancel job']);
}
});
test('pagehide stops polling, closes streams and aborts the in-flight read; pageshow resumes', async () => {
const { createActivityStore } = await source;
const { world, api, ...deps } = harness();
+148
View File
@@ -215,3 +215,151 @@ test('none and unavailable render no image chips', async () => {
assert.deepEqual(inspectorTagChips(tagged({state, tags:[]}, {status:'none', tags:[], tags_scope:null}), {filterable:true}), []);
}
});
// LI-4 (D117): "Copy settings summary" after the six facts of the full inspector consolidates the legacy metadata
// page's COPY ALL (web/metadata.html copyAllData). The legacy template is read from that page itself and run on the
// legacy parse, so the summary's text is the legacy page's, line for line, wherever the two agree on the values.
const METADATA_PAGE = fs.readFileSync(new URL('../../web/metadata.html', import.meta.url), 'utf8');
function legacyCopyAll(params) {
const template = METADATA_PAGE.match(/const allData = `([\s\S]*?)`;/)[1];
return new Function('currentWorkflowData', `return \`${template}\`;`)(params);
}
const legacyParams = record => ComfyMetadata.extractGenerationParams({prompt:record.metadata.prompt_graph.status === 'parsed' ? record.metadata.prompt_graph.value : null,
workflow:record.metadata.workflow_graph.status === 'parsed' ? record.metadata.workflow_graph.value : null});
/**
* Each summary line against the legacy page's, allowing only the recorded differences (D117): a number keeps the
* inspector's (the server's) spelling ("8.0" where the legacy page wrote 8); a fact the legacy parse left "Unknown" may
* be known; and a negative the #249 ZeroOut guard zeroed reads "No negative prompt found" where the legacy parse read
* the positive's text through the zero-out.
*/
function assertLegacySummary(summary, legacy, record) {
const lines = summary.split('\n'), expected = legacy.split('\n');
assert.deepEqual(lines.map(line => line.split(':')[0]), expected.map(line => line.split(':')[0]), 'the legacy labels, in order');
lines.forEach((line, index) => {
if (line === expected[index]) return;
const [label, value] = [line.slice(0, line.indexOf(': ')), line.slice(line.indexOf(': ') + 2)];
const old = expected[index].slice(label.length + 2);
if (label === 'Negative Prompt' && record.prompts.negative?.source === 'zeroed-conditioning' && value === 'No negative prompt found') return;
if (old === 'Unknown' && !/Prompt$/.test(label)) return;
assert.ok(value.trim() !== '' && Number(value) === Number(old), `${label}: "${value}" differs from the legacy "${old}"`);
});
}
test('settings summary is the legacy COPY ALL text for a local file', async () => {
const {settingsSummary} = await load();
assert.equal(typeof settingsSummary, 'function', 'settingsSummary is exported');
const record = await localRecord(localPng('fox.png'));
assertLegacySummary(settingsSummary(record), legacyCopyAll(legacyParams(record)), record);
assert.equal(settingsSummary(record), ['Checkpoint: synthetic-v1.safetensors', 'Positive Prompt: a red fox in snow, detailed fur',
'Negative Prompt: blurry, lowres', 'Steps: 20', 'CFG Scale: 8.0', 'Sampler: euler', 'Seed: 7'].join('\n'));
// Every parity PNG read in this browser: the legacy page's text but for the recorded differences, and exactly the
// legacy text for most of them (f09 among them: the server is ambiguous, so the record shows the legacy fill).
const exact = [];
for (const name of fs.readdirSync(PARITY).filter(item => item.endsWith('.png')).sort()) {
const parity = await localRecord(new File([fs.readFileSync(new URL(name, PARITY))], name));
const summary = settingsSummary(parity), legacy = legacyCopyAll(legacyParams(parity));
assertLegacySummary(summary, legacy, parity);
if (summary === legacy) exact.push(name);
}
assert.ok(exact.includes('f09.png'));
assert.deepEqual(fs.readdirSync(PARITY).filter(item => item.endsWith('.png') && !exact.includes(item)).sort(), ['f01b.png', 'f17.png', 'f23.png']);
});
test('settings summary uses the legacy placeholders for what the record does not know', async () => {
const {settingsSummary} = await load();
const unknown = {...record(), prompts:{positive:{status:'unresolved'}, negative:{status:'ambiguous'}}, generation:{}};
assert.equal(settingsSummary(unknown), legacyCopyAll({}).replace('Positive Prompt: undefined', 'Positive Prompt: No prompt found')
.replace('Negative Prompt: undefined', 'Negative Prompt: No negative prompt found'));
assert.equal(settingsSummary(unknown), ['Checkpoint: Unknown', 'Positive Prompt: No prompt found', 'Negative Prompt: No negative prompt found',
'Steps: Unknown', 'CFG Scale: Unknown', 'Sampler: Unknown', 'Seed: Unknown'].join('\n'));
// A server answer is shown as the inspector shows it: known facts verbatim, every other state the legacy "Unknown".
const server = {...record(), prompts:{positive:{status:'resolved', text:'a\n猫'}, negative:{status:'resolved', text:''}}, generation:{
model:{status:'known', value:'m.safetensors'}, steps:{status:'not-recorded', value:null}, cfg:{status:'known', value:'6.50'},
sampler:{status:'ambiguous', value:null}, seed:{status:'known', value:'9007199254740993'}, scheduler:{status:'known', value:'karras'}}};
assert.equal(settingsSummary(server), ['Checkpoint: m.safetensors', 'Positive Prompt: a\n猫', 'Negative Prompt: No negative prompt found',
'Steps: Unknown', 'CFG Scale: 6.50', 'Sampler: Unknown', 'Seed: 9007199254740993'].join('\n'));
});
/** The full inspector with a toast node and `clipboard` as `navigator.clipboard`; returns the summary button. */
function renderSummary(record, {variant = 'full', clipboard} = {}) {
const f = mountedDom(), scope = createScope(), fallbacks = [], toast = new f.Node('div');
f.doc.getElementById = id => id === 'workspace-toast' ? toast : null;
const previous = Object.getOwnPropertyDescriptor(globalThis, 'navigator');
Object.defineProperty(globalThis, 'navigator', {configurable:true, value:clipboard === undefined ? {} : {clipboard}});
const view = mountFileInspector(f.container, {sender:{open() {}}, parentScope:scope, variant, copyFallback:(text, options) => fallbacks.push({text, ...options})});
view.setState({loading:false, error:false, record});
const button = f.walk(f.container, n => n.tagName === 'button' && n.textContent === 'Copy settings summary')[0] ?? null;
return {f, button, toast, fallbacks, scope, done() {
view.dispose(); scope.dispose(); f.restore();
if (previous) Object.defineProperty(globalThis, 'navigator', previous); else delete globalThis.navigator;
}};
}
test('Copy settings summary follows the facts in the full inspector only', async () => {
const full = renderSummary(record());
try {
assert.ok(full.button, 'the full inspector offers Copy settings summary');
const inner = full.button.parent, index = inner.children.indexOf(full.button);
assert.equal(inner.children[index - 1].tagName, 'dl', 'it follows the six facts');
assert.equal(inner.children[index + 1].tagName, 'details', 'File & provenance follows it');
// M7: the D117 wrap rule targets this button by its class, never any button of a details block.
assert.equal(full.button.className, 'settings-summary');
} finally { full.done(); }
const panel = renderSummary(record(), {variant:'panel'});
try { assert.equal(panel.button, null, 'the Images side panel keeps the concept footer'); } finally { panel.done(); }
});
test('Copy settings summary copies the summary and shows the shared toast', async () => {
const {settingsSummary} = await load(), copied = [];
const view = renderSummary(record(), {clipboard:{writeText:async text => {copied.push(text);}}});
try {
await view.button.listeners.get('click')[0].callback({target:view.button});
assert.deepEqual(copied, [settingsSummary(record())]);
assert.equal(view.toast.textContent, 'Settings summary copied.');
assert.equal(view.fallbacks.length, 0);
} finally { view.done(); }
});
test('Copy settings summary without a clipboard opens the select-to-copy fallback', async () => {
const {settingsSummary} = await load();
for (const clipboard of [undefined, {writeText:async () => {throw new Error('Denied');}}]) {
const view = renderSummary(record(), {clipboard});
try {
await view.button.listeners.get('click')[0].callback({target:view.button});
assert.equal(view.fallbacks.length, 1);
assert.equal(view.fallbacks[0].text, settingsSummary(record()));
assert.equal(view.fallbacks[0].title, 'Copy settings summary');
assert.equal(view.fallbacks[0].trigger, view.button);
assert.notEqual(view.toast.textContent, 'Settings summary copied.');
} finally { view.done(); }
}
});
test('the inspector is busy while a read is loading, then not', async () => {
const f = mountedDom(), scope = createScope();
const view = mountFileInspector(f.container, {sender:{open() {}}, parentScope:scope, variant:'full'});
try {
view.setState({loading:true, error:false, record:null});
assert.equal(f.container.getAttribute('aria-busy'), 'true');
assert.ok(f.walk(f.container, n => n.textContent === 'Loading file details…').length);
view.setState({loading:false, error:false, record:record()});
assert.equal(f.container.getAttribute('aria-busy'), null);
view.setState({loading:true, error:false, record:null});
view.setState({loading:false, error:true, record:null});
assert.equal(f.container.getAttribute('aria-busy'), null);
view.setState({loading:true, error:false, record:null});
} finally {
view.dispose();
assert.equal(f.container.getAttribute('aria-busy'), null, 'dispose clears the busy state');
scope.dispose(); f.restore();
}
});
test('the D117 wrap rule targets only the settings summary button', async () => {
const fs = await import('node:fs');
const css = fs.readFileSync(new URL('../../web/js/workspace/workspace.css', import.meta.url), 'utf8');
const rules = css.split('\n').filter(line => line.includes('D117'));
assert.deepEqual(rules.map(line => line.trim()), ['.ca-tools #viewer-details .settings-summary { white-space: normal; /* D117 */ }']);
assert.doesNotMatch(css, /#viewer-details > div > button/);
});
@@ -0,0 +1,198 @@
import test from 'node:test';
import assert from 'node:assert/strict';
// Guarded: before the module exists every test fails on an assertion, not on the import.
const loaded = await import('../../web/js/workspace/images-tagging-bulk.mjs').catch(() => null);
const module = () => { assert.ok(loaded, 'images-tagging-bulk.mjs is not implemented'); return loaded; };
const state = await import('../../web/js/workspace/image-tagging-state.mjs');
const { startToast, parseJob } = await import('../../web/js/workspace/activity-store.mjs');
const SCOPE = 'b'.repeat(64);
const JOB_ID = 'c'.repeat(32);
const image = (path, extra = {}) => ({ root_index: 0, root_scope: SCOPE, path, filename: path.split('/').pop(), ...extra });
const pageScope = () => {
const controller = new AbortController();
const cleanups = [];
return { signal: controller.signal, cleanup: fn => cleanups.push(fn), abort: () => controller.abort(), cleanups };
};
test('targets keep listing order and identity', () => {
const { bulkTargets } = module();
const listing = [image('b/2.png', { width: 9 }), image('a/1.png'), { ...image('x.png'), root_scope: 'nope' }, image('c/3.webm')];
assert.deepEqual(bulkTargets(listing), [
{ root_index: 0, root_scope: SCOPE, path: 'b/2.png' },
{ root_index: 0, root_scope: SCOPE, path: 'a/1.png' },
{ root_index: 0, root_scope: SCOPE, path: 'c/3.webm' },
]);
assert.deepEqual(bulkTargets([]), []);
});
test('over 500 selected refuses with the toast', () => {
const { bulkTargets, openAddTags, openAutoTag, MAX_BULK_IMAGES, TOO_MANY } = module();
assert.equal(MAX_BULK_IMAGES, 500);
assert.equal(TOO_MANY, 'Select at most 500 images.');
const many = Array.from({ length: 501 }, (_, index) => image(`bulk/${index}.png`));
assert.equal(bulkTargets(many), null);
assert.equal(bulkTargets(many.slice(0, 500)).length, 500);
const toasts = [];
const store = { start: () => assert.fail('no job starts') };
// Refused before any dialog opens (no DOM here).
openAddTags({ scope: pageScope(), store, images: many, notify: text => toasts.push(text) });
openAutoTag({ api: {}, scope: pageScope(), store, images: many, notify: text => toasts.push(text) });
assert.deepEqual(toasts, [TOO_MANY, TOO_MANY]);
});
test('add parses comma tags with the same dedupe as the dialog', () => {
const { parseBulkTags, ADD_DIALOG } = module();
assert.deepEqual(ADD_DIALOG, { title: 'Tag selected images', label: 'Tags, separated by commas', action: 'Apply' });
assert.deepEqual(parseBulkTags(' Sunset, sunset ,, blue_hour,SUNSET, Blue_Hour '), ['Sunset', 'blue_hour']);
assert.deepEqual(parseBulkTags(' , ,'), []);
assert.equal(parseBulkTags, state.parseManualTags);
});
test('Add tags keeps its dialog and the typed text when the field parses to no names', async () => {
const { openAddTags, NO_TAGS } = module();
const toasts = [];
const posted = [];
const store = { start: async (kind, body) => { posted.push(body); return { job: { id: JOB_ID, state: 'running' } }; },
find: () => null, subscribe: () => () => {} };
let dialog = null;
openAddTags({ scope: pageScope(), store, images: [image('a.png')], trigger: null, notify: text => toasts.push(text),
openInput: args => { dialog = args; } });
assert.equal(typeof dialog?.validate, 'function');
// The dialog reports the message on its field and stays open: nothing toasts, nothing is sent.
assert.equal(dialog.validate(' , ,'), NO_TAGS);
assert.equal(dialog.validate('sunset'), null);
assert.deepEqual(toasts, []);
await dialog.onApply('sunset, Sunset');
assert.deepEqual(posted.map(body => body.tags), [['sunset']]);
});
test('busy start keeps the selection and names the running job', async () => {
const { startTagJob } = module();
const toasts = [];
const posted = [];
const running = { id: JOB_ID, title: 'Scan for metadata' };
const store = { start: async (kind, body) => { posted.push([kind, body]); return { busy: running, message: 'busy' }; } };
const body = { mode: 'add', targets: [{ root_index: 0, root_scope: SCOPE, path: 'a.png' }], tags: ['x'] };
assert.equal(await startTagJob({ store, scope: pageScope(), body, notify: text => toasts.push(text) }), null);
assert.deepEqual(posted, [['image-tags', body]]);
assert.deepEqual(toasts, [['Another job is running: ', { text: 'Scan for metadata', href: `#/activity/${JOB_ID}` }, '.']]);
});
test('a started job is returned without a toast of its own; refusals toast their sentence', async () => {
const { startTagJob } = module();
const toasts = [];
const notify = text => toasts.push(text);
const job = { id: JOB_ID, kind: 'image-tags' };
assert.equal(await startTagJob({ store: { start: async () => ({ job }) }, scope: pageScope(), body: { mode: 'add' }, notify }), job);
const refused = Object.assign(new Error('The tagging model is busy. Try again when it finishes.'), { name: 'ApiError', status: 409, code: 'model-busy' });
assert.equal(await startTagJob({ store: { start: async () => { throw refused; } }, scope: pageScope(), body: { mode: 'auto' }, notify }), null);
assert.equal(await startTagJob({ store: { start: async () => { throw new TypeError('down'); } }, scope: pageScope(), body: { mode: 'auto' }, notify }), null);
assert.equal(await startTagJob({ store: { start: async () => { throw new TypeError('down'); } }, scope: pageScope(), body: { mode: 'add' }, notify }), null);
assert.deepEqual(toasts, ['The tagging model is busy. Try again when it finishes.', 'Auto-tagging could not start. Try again.',
'Tagging could not start. Try again.']);
// A page left meanwhile says nothing.
const left = pageScope();
left.abort();
assert.equal(await startTagJob({ store: { start: async () => { throw refused; } }, scope: left, body: { mode: 'auto' }, notify }), null);
assert.equal(toasts.length, 3);
});
test('the start toast names the tagging mode', () => {
const job = mode => parseJob({ id: JOB_ID, state: 'running', kind: 'image-tags', title: 'Add tags to images', scope: { mode, images: 2 } });
assert.equal(startToast(job('add')), 'Tagging started.');
assert.equal(startToast(job('auto')), 'Auto-tagging started.');
assert.equal(startToast(job('other')), 'Add tags to images started.');
assert.equal(startToast(parseJob({ id: JOB_ID, state: 'running', kind: 'scan', title: 'Scan for metadata' })), 'Scan started.');
});
test('a followed job reports its end once', () => {
const { followJobEnd } = module();
let jobs = [{ id: JOB_ID, state: 'running', counters: {} }];
const listeners = new Set();
const store = { find: id => jobs.find(job => job.id === id), subscribe: fn => { listeners.add(fn); return () => listeners.delete(fn); } };
const ended = [];
const scope = pageScope();
followJobEnd({ store, scope, id: JOB_ID, onEnd: job => ended.push(job.state) });
assert.equal(listeners.size, 1);
for (const fn of [...listeners]) fn(store);
assert.deepEqual(ended, []);
jobs = [{ id: JOB_ID, state: 'finished', counters: { tagged: 1 } }];
for (const fn of [...listeners]) fn(store);
for (const fn of [...listeners]) fn(store);
assert.deepEqual(ended, ['finished']);
assert.equal(listeners.size, 0);
assert.equal(scope.cleanups.length, 1);
});
test('a model that is not downloaded is named with the server sentence', () => {
const { modelUnavailable } = state;
assert.equal(modelUnavailable(null, 'wd14-vit'), '');
assert.equal(modelUnavailable({ 'wd14-vit': { downloaded: true } }, 'wd14-vit'), '');
assert.equal(modelUnavailable({ 'wd14-vit': { downloaded: false } }, 'wd14-vit'), 'WD14 ViT is not downloaded. Download it from the legacy Auto-tag page.');
assert.equal(modelUnavailable({ gguf: { downloaded: false, name: 'GGUF (Recommended)' } }, 'gguf'),
'GGUF (Recommended) is not downloaded. Download it from the legacy Auto-tag page.');
assert.throws(() => modelUnavailable({}, 'nope'), TypeError);
});
test('the bulk options start from the server defaults and wait for them before Start', async () => {
const { createTaggingOptionsController } = state;
let answer;
const api = { get: () => new Promise(resolve => { answer = resolve; }) };
const options = createTaggingOptionsController({ api });
const seen = [];
options.subscribe(value => seen.push(value.optionsRevision));
assert.equal(options.state.canStart, false);
assert.equal(options.state.canGenerate, false);
const opening = options.open();
answer({ wd14_general_threshold: 0.5, wd14_character_threshold: 0.9, default_prompt: 'Say it', suggestion_limit: 7,
models: { 'wd14-swinv2': { downloaded: true }, 'wd14-vit': { downloaded: false }, gguf: { downloaded: true } } });
await opening;
assert.equal(options.state.canStart, true);
assert.deepEqual(options.request(), { model_type: 'wd14-swinv2', options: { general_threshold: 0.5, character_threshold: 0.9, suggestion_limit: 7 } });
options.setOption('suggestion_limit', '3');
options.setModel('wd14-vit');
assert.equal(options.state.canStart, false);
assert.match(options.state.unavailable, /^WD14 ViT is not downloaded\./);
options.setModel('gguf');
assert.deepEqual(options.request(), { model_type: 'gguf', options: { instructions: 'Say it', suggestion_limit: 3 } });
options.setOption('instructions', ' ');
assert.equal(options.request(), null);
assert.deepEqual(seen, [1, 1, 2, 3, 3]);
assert.throws(() => options.setModel('nope'), TypeError);
assert.throws(() => options.setOption('nope', 1), TypeError);
options.dispose();
options.setOption('suggestion_limit', '4');
assert.equal(options.state.options.suggestion_limit, '3');
});
test('the bulk options keep the documented defaults when the read fails', async () => {
const { createTaggingOptionsController } = state;
const options = createTaggingOptionsController({ api: { get: async () => { throw new Error('offline'); } } });
await options.open();
await options.open();
assert.equal(options.state.canStart, true);
assert.equal(options.state.unavailable, '');
assert.deepEqual(options.request(), { model_type: 'wd14-swinv2', options: { general_threshold: 0.35, character_threshold: 0.85, suggestion_limit: 10 } });
assert.throws(() => createTaggingOptionsController({}), TypeError);
});
test('a preset starts the Tag image dialog on the chosen model and options', async () => {
const { createImageTaggingController } = state;
const target = { root_index: 0, root_scope: SCOPE, path: 'a.png' };
const api = {
get: async () => ({ wd14_general_threshold: 0.35, wd14_character_threshold: 0.85, default_prompt: 'Describe', suggestion_limit: 10 }),
post: async () => ({ success: true, library_epoch: '12345678-1234-1234-1234-123456789abc', filename: 'a.png', title: null, tags: [], tag_revision: 0,
capture: { token: 'tok', sha256: 'd'.repeat(64), preview_url: '/prompt_manager/image-annotations/capture/tok', width: 4, height: 4,
media_type: 'image', generate: { supported: true, reason: null } } }),
put: async () => ({}), del: async () => ({}),
};
const controller = createImageTaggingController({ api, target, source: 'bulk', preset: { model: 'gguf', options: { instructions: 'Name it', suggestion_limit: 3 } } });
assert.equal(controller.state.model, 'gguf');
await controller.open();
assert.deepEqual([controller.state.model, controller.state.options.instructions, controller.state.options.suggestion_limit], ['gguf', 'Name it', 3]);
controller.dispose();
assert.throws(() => createImageTaggingController({ api, target, preset: { model: 'nope', options: {} } }), TypeError);
assert.throws(() => createImageTaggingController({ api, target, preset: { model: 'gguf', options: { instructions: '' } } }), TypeError);
assert.throws(() => createImageTaggingController({ api, target, preset: { model: 'gguf', options: { nope: 1 } } }), TypeError);
});
@@ -0,0 +1,110 @@
import test from 'node:test';
import assert from 'node:assert/strict';
// Guarded: before the module exists every test fails on an assertion, not on the import.
const loaded = await import('../../web/js/workspace/images-tagging-review.mjs').catch(() => null);
const module = () => { assert.ok(loaded, 'images-tagging-review.mjs is not implemented'); return loaded; };
const SCOPE = 'e'.repeat(64);
const images = ['a.png', 'b.png', 'c.png'].map(path => ({ root_index: 0, root_scope: SCOPE, path, filename: path }));
const OPTIONS = { general_threshold: 0.4, character_threshold: 0.8, suggestion_limit: 3 };
/** A session over fake dialogs: `dialogs` holds every open call's arguments and whether it was closed. */
function session(extra = {}) {
const dialogs = [];
const toasts = [];
let closes = 0;
const saved = [];
const handle = module().startReviewSession({
api: {}, scope: {}, images, model_type: 'wd14-vit', options: OPTIONS, trigger: 'auto-tag',
onSaved: event => saved.push(event.filename),
open: args => {
const dialog = { args, closed: 0 };
dialogs.push(dialog);
return { close: () => { dialog.closed += 1; args.onStop(); } };
},
close: () => { closes += 1; },
notify: text => toasts.push(text),
...extra,
});
return { handle, dialogs, toasts, saved, closes: () => closes };
}
const approve = (dialog, tags = ['x']) => dialog.args.onSaved({ filename: dialog.args.image.filename, tags, target: {} });
test('the first image opens at once with the session model, options and its position', () => {
const { dialogs } = session();
assert.equal(dialogs.length, 1);
const { args } = dialogs[0];
assert.equal(args.image, images[0]);
assert.deepEqual(args.position, { index: 1, total: 3 });
assert.deepEqual(args.preset, { model: 'wd14-vit', options: OPTIONS });
assert.equal(args.autoGenerate, true);
assert.equal(args.actions, 'review');
assert.equal(args.source, 'bulk');
assert.equal(args.trigger, 'auto-tag');
assert.throws(() => module().startReviewSession({ images: [] }), TypeError);
});
test('advances only on approve or skip and writes nothing on skip or stop', () => {
const { dialogs, toasts, saved } = session();
// Nothing from the dialog (a Generate in progress, a model-busy answer): the session waits on that image.
assert.equal(dialogs.length, 1);
dialogs[0].args.onSkip();
assert.equal(dialogs.length, 2);
assert.deepEqual(dialogs[1].args.position, { index: 2, total: 3 });
// A second report from a dialog already left changes nothing.
dialogs[0].args.onSkip();
dialogs[0].args.onStop();
assert.equal(dialogs.length, 2);
approve(dialogs[1]);
assert.equal(dialogs.length, 3);
dialogs[2].args.onStop();
assert.deepEqual(saved, ['b.png']);
assert.deepEqual(toasts, ['1 of 3 images tagged.']);
assert.equal(dialogs.length, 3);
});
test('approve saves exactly the dialog\'s manual-plus-chips set', () => {
// The session never writes itself: each approve is the dialog's own Save (its event carries the saved set).
const forwarded = [];
const { dialogs } = session({ onSaved: event => forwarded.push(event.tags) });
approve(dialogs[0], ['manual', 'chip']);
assert.deepEqual(forwarded, [['manual', 'chip']]);
assert.deepEqual(dialogs[1].args.preset, { model: 'wd14-vit', options: OPTIONS });
});
test('last image offers Approve and finish', () => {
const { dialogs, toasts, closes } = session();
approve(dialogs[0]);
dialogs[1].args.onSkip();
assert.deepEqual(dialogs[2].args.position, { index: 3, total: 3 });
approve(dialogs[2]);
// The last approve ends the session: the dialog closes and the count is toasted once.
assert.equal(closes(), 1);
assert.deepEqual(toasts, ['2 of 3 images tagged.']);
dialogs[2].args.onStop();
assert.deepEqual(toasts, ['2 of 3 images tagged.']);
});
test('model-busy on one image waits for retry or skip', () => {
const { dialogs, toasts } = session();
// The dialog shows the refusal with Retry; a Retry that succeeds then approves, all on the same image.
assert.equal(dialogs.length, 1);
approve(dialogs[0]);
assert.equal(dialogs.length, 2);
assert.deepEqual(toasts, []);
});
test('stop releases listeners and in-flight requests', () => {
const { handle, dialogs, toasts, saved } = session();
handle.stop();
assert.equal(dialogs[0].closed, 1);
assert.deepEqual(toasts, ['0 of 3 images tagged.']);
// A save that settles after the stop still counts as written, but nothing opens or toasts again.
approve(dialogs[0]);
dialogs[0].args.onSkip();
handle.stop();
assert.equal(dialogs.length, 1);
assert.deepEqual(saved, ['a.png']);
assert.deepEqual(toasts, ['0 of 3 images tagged.']);
assert.equal(module().reviewOutcome(1, 1), '1 of 1 images tagged.');
});
+160 -2
View File
@@ -20,6 +20,16 @@ const { extractGenerationFacts } = ComfyMetadata;
// The page loads both as scripts; the legacy fill and the PNG reader find them on the global.
globalThis.PngMetadata = PngMetadata;
globalThis.ComfyMetadata = ComfyMetadata;
// The page's API client and the node table it serves (local-semantics.mjs fetches it once).
const SEMANTICS = fs.readFileSync(path.join(here, "..", "..", "web", "js", "inspection-semantics.json"), "utf8");
globalThis.window = { ApiClient: { resolveUrl: (value) => value } };
// A test can hold the table request (`tableGate`) to stand for a slow or stalled proxy.
let tableGate = null;
globalThis.fetch = async (url) => {
if (url !== "/prompt_manager/js/inspection-semantics.json") throw new Error(`unexpected request ${url}`);
if (tableGate) await tableGate;
return new Response(SEMANTICS, { status: 200 });
};
const m = await import("../../web/js/workspace/local-inspection.mjs").catch((error) => {
if (error.code !== "ERR_MODULE_NOT_FOUND") throw error;
return {};
@@ -59,6 +69,37 @@ test("contract", () => {
assert.ok(Object.isFrozen(m.localFileAdapter));
});
// M3: the first read waits for the page's one table request; an aborted, replaced or cleared selection must not.
test("a read waiting for the node table ends with its selection", async () => {
let release;
tableGate = new Promise((resolve) => { release = resolve; });
try {
const file = pngFile("slow.png", [tEXt("prompt", PROMPT)]);
const aborted = new AbortController();
const reading = m.localFileAdapter.read(await handleFor(file), { signal: aborted.signal });
const settled = (promise) => Promise.race([promise.then(() => "resolved", (error) => error.name), new Promise((resolve) => setTimeout(() => resolve("pending"), 200))]);
await new Promise((resolve) => setTimeout(resolve, 20));
aborted.abort();
assert.equal(await settled(reading), "AbortError");
// Through the controller: clearing the selection ends its read; nothing is published for it.
const controller = createInspectionController({ api: null, adapter: m.localFileAdapter, extractFacts: extractGenerationFacts });
const states = [];
controller.subscribe((state) => states.push(state));
const selected = controller.select(await handleFor(file));
await new Promise((resolve) => setTimeout(resolve, 20));
controller.clear();
assert.equal(await settled(selected), "resolved");
assert.ok(states.every((state) => state.record === null && state.error === false), "no record or error for a cleared selection");
controller.dispose();
} finally {
tableGate = null;
release();
}
// The shared request still lands for the next selection.
const { wire } = await recordFor(pngFile("next.png", [tEXt("prompt", PROMPT)]));
assert.equal(wire.resolution.state, "resolved");
});
test("a PNG File becomes the server-shaped wire with local provenance", async () => {
const file = pngFile("Fox Render.PNG", [tEXt("prompt", PROMPT), tEXt("workflow", WORKFLOW)]);
const { wire, record, handle } = await recordFor(file);
@@ -66,8 +107,12 @@ test("a PNG File becomes the server-shaped wire with local provenance", async ()
size: file.size, width: 640, height: 480, modified_time: 1_700_000_000.123 });
assert.equal(wire.success, true);
assert.equal(wire.inspection_version, 1);
assert.deepEqual(wire.prompts, { positive: { status: "unresolved" }, negative: { status: "unresolved" } });
assert.deepEqual(wire.resolution, { state: "unresolved", sampler_id: null, reason: "Resolved in this browser from the workflow.", reason_code: null, node: null });
// The roles the server's inspector resolves for the same records (local-resolver.mjs).
assert.deepEqual(wire.prompts, {
positive: { status: "resolved", text: "a red fox in snow, detailed fur", source: "graph-inferred" },
negative: { status: "resolved", text: "blurry, lowres", source: "graph-inferred" },
});
assert.deepEqual(wire.resolution, { state: "resolved", sampler_id: "3", reason: "", reason_code: null, node: null });
assert.deepEqual(wire.library, { status: "unavailable", scope: "saved-library", tags: [], category_scope: null, tags_scope: null });
assert.equal(wire.metadata.status, "available");
assert.deepEqual(wire.metadata.raw, { prompt: PROMPT, workflow: WORKFLOW });
@@ -96,6 +141,33 @@ test("a PNG File becomes the server-shaped wire with local provenance", async ()
assert.equal(exported.file.source_label, "Selected local file");
});
test("roles answer as the server's inspector: unreadable, unavailable table, ambiguous", async () => {
const png = { media: "image", kind: "png" };
const unreadable = m.buildLocalPayload({ read: null, file: { name: "a.png" }, classification: png });
assert.deepEqual(unreadable.resolution, { state: "unresolved", sampler_id: null, reason: "Media metadata could not be read", reason_code: null, node: null });
assert.equal(unreadable.prompts.positive.message, "Metadata in this file could not be read.");
const records = { prompt: PROMPT };
const read = { status: "complete", records, limited: [], width: 2, height: 3 };
// Without a table (it could not be fetched) the roles stay open, with the server's own line.
const missing = m.buildLocalPayload({ read, file: { name: "a.png" }, classification: png });
assert.equal(missing.resolution.reason, "Inspection node semantics are unavailable");
assert.equal(missing.prompts.positive.message, "The inspector's node table is unavailable.");
const semantics = await (await import("../../web/js/workspace/local-semantics.mjs")).loadInspectionSemantics();
const graph = JSON.parse(PROMPT);
graph["9"] = { ...graph["3"], inputs: { ...graph["3"].inputs, positive: ["8", 0] } };
graph["8"] = { class_type: "CLIPTextEncode", inputs: { text: "a cat", clip: ["4", 1] } };
const several = m.buildLocalPayload({ read: { ...read, records: { prompt: JSON.stringify(graph) } }, file: { name: "a.png" }, classification: png, semantics });
assert.equal(several.resolution.reason, "Multiple sampler branches");
assert.deepEqual(several.prompts.positive, { status: "ambiguous", text: null, source: null });
// The legacy fill then applies exactly as it does to the server's answer for the same bytes in Images.
const handle = await handleFor(pngFile("grid.png", [tEXt("prompt", JSON.stringify(graph))]));
const local = m.localFileAdapter.toRecord(several, handle, extractGenerationFacts);
const asServed = { ...several, file: { ...several.file, observation: "current-file", source_label: "output" } };
const served = projectInspectionPayload(asServed, { kind: "rooted-file", key: "0:grid.png", root_index: 0, path: "grid.png", root_scope: null }, null, extractGenerationFacts);
assert.deepEqual(local.prompts, served.prompts);
assert.deepEqual(local.generation, served.generation);
});
test("file facts follow the server: extension like Path.suffix, null when unknown", async () => {
const cases = [["a.b.PnG", ".png"], [".hidden", ""], ["noext", ""], ["dot.", ""], ["x.tar.gz", ".gz"]];
for (const [name, extension] of cases) {
@@ -307,3 +379,89 @@ test("the parity corpus PNGs read into renderable records", async () => {
assert.ok(["parsed", "limited", "malformed"].includes(wire.metadata.prompt_graph.status));
}
});
// M4: resolution runs on the page's main thread, so a read yields between its bounded steps (the busy state paints
// and input is handled) and no step may exceed a budget on the largest record the readers keep (2 MiB of roles).
const ROLE_BYTES = 2 * 1024 * 1024;
// Generous for a shared CI runner: a desktop resolves these in 130-455 ms.
const STEP_BUDGET_MS = 2500;
const ENCODE = (text) => ({ class_type: "CLIPTextEncode", inputs: { text, clip: ["4", 1] } });
const KSAMPLER = (positive, negative, latent) => ({ class_type: "KSampler", inputs: { seed: 1, steps: 20, cfg: 8.0, sampler_name: "euler", scheduler: "normal",
denoise: 1.0, model: ["4", 0], positive, negative, latent_image: latent } });
const LOADERS = { "4": { class_type: "CheckpointLoaderSimple", inputs: { ckpt_name: "m.safetensors" } }, "5": { class_type: "EmptyLatentImage", inputs: { width: 64, height: 64, batch_size: 1 } } };
/** The largest graph of this shape whose JSON fits the role budget. */
function maximal(build, high) {
let low = 1;
while (low < high) {
const middle = (low + high + 1) >> 1;
if (Buffer.byteLength(JSON.stringify(build(middle)), "utf8") <= ROLE_BYTES) low = middle;
else high = middle - 1;
}
return JSON.stringify(build(low));
}
const MAXIMAL_GRAPHS = {
// Many candidates: every sampler is a final stage with its own pair.
independent: maximal((count) => {
const graph = { ...LOADERS };
for (let index = 0; index < count; index++) {
graph[`s${index}`] = KSAMPLER([`p${index}`, 0], [`n${index}`, 0], ["5", 0]);
graph[`p${index}`] = ENCODE(`fox ${index}`);
graph[`n${index}`] = ENCODE(`blur ${index}`);
}
return graph;
}, 6666),
// One long stage chain through the latent: the deepest stage walk.
chain: maximal((count) => {
const graph = { ...LOADERS, "6": ENCODE("fox"), "7": ENCODE("blur") };
for (let index = 0; index < count; index++) graph[`s${index}`] = KSAMPLER(["6", 0], ["7", 0], index ? [`s${index - 1}`, 0] : ["5", 0]);
return graph;
}, 19996),
};
test("resolution of a maximal 2 MiB graph stays within the step budget", async () => {
const { resolveLocalEvidence } = await import("../../web/js/workspace/local-resolver.mjs");
const semantics = await (await import("../../web/js/workspace/local-semantics.mjs")).loadInspectionSemantics();
for (const [name, prompt] of Object.entries(MAXIMAL_GRAPHS)) {
assert.ok(Buffer.byteLength(prompt, "utf8") > ROLE_BYTES - 1024, `${name} fills the role budget`);
const started = performance.now();
const evidence = resolveLocalEvidence({ prompt }, semantics);
const elapsed = performance.now() - started;
assert.ok(elapsed < STEP_BUDGET_MS, `${name}: ${Math.round(elapsed)} ms`);
assert.notEqual(evidence.resolution.reason_code, "malformed", name);
}
});
test("a read yields between bounded steps and answers byte for byte as one synchronous build", async () => {
const semantics = await (await import("../../web/js/workspace/local-semantics.mjs")).loadInspectionSemantics();
const file = pngFile("maximal.png", [tEXt("prompt", MAXIMAL_GRAPHS.independent)]);
const handle = await handleFor(file);
const marks = [];
const yieldControl = async (signal) => {
signal?.throwIfAborted();
marks.push(performance.now());
await new Promise((resolve) => setTimeout(resolve, 0));
};
const started = performance.now();
const wire = await m.localFileAdapter.read(handle, { signal: new AbortController().signal, yieldControl });
marks.push(performance.now());
// Resolution and the projection each run in a turn of their own after the table is in.
assert.ok(marks.length >= 3, `yields: ${marks.length - 1}`);
const gaps = marks.map((mark, index) => mark - (index ? marks[index - 1] : started));
assert.ok(Math.max(...gaps) < STEP_BUDGET_MS, `longest step ${Math.round(Math.max(...gaps))} ms`);
const { result } = await m.readLocalFile(file, { classification: handle.classification });
const direct = await m.fitLocalPayload(m.buildLocalPayload({ read: result, file: { name: handle.filename, size: file.size, lastModified: file.lastModified },
classification: handle.classification, semantics }));
assert.equal(JSON.stringify(wire), JSON.stringify(direct));
assert.equal(wire.resolution.state, "ambiguous");
// An abort at any yield ends the read there.
for (let stop = 0; stop < marks.length - 1; stop++) {
const controller = new AbortController();
let seen = 0;
const stopping = async (signal) => {
if (seen++ === stop) controller.abort();
signal.throwIfAborted();
await new Promise((resolve) => setTimeout(resolve, 0));
};
await assert.rejects(m.localFileAdapter.read(handle, { signal: controller.signal, yieldControl: stopping }), { name: "AbortError" }, `yield ${stop}`);
}
});
+161
View File
@@ -0,0 +1,161 @@
// PR L3 Tasks 8-9: the server's inspection resolver in this browser. The values below are what CPython answers
// (json.loads, len, str.strip, repr, utils/prompt_inspector.inspect_prompt_evidence); the full comparison with the
// server runs in tests/test_local_inspect_parity.py through tests/js/support/local-inspect-oracle.mjs.
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
const py = await import('../../web/js/workspace/py-json.mjs');
const resolver = await import('../../web/js/workspace/local-resolver.mjs');
const semantics = resolver.compileSemantics(JSON.parse(fs.readFileSync(new URL('../../web/js/inspection-semantics.json', import.meta.url), 'utf8')));
test('json.loads keeps Python value kinds and dict order', () => {
const value = py.pyJsonLoads('{"9": 1, "10": 1.0, "a": NaN, "9": [true, -0, 9007199254740993, -Infinity]}');
assert.ok(value instanceof Map);
// A repeated key keeps its first position and takes its last value; integer-like keys keep text order.
assert.deepEqual([...value.keys()], ['9', '10', 'a']);
const [flag, zero, big, negative] = value.get('9');
assert.equal(flag, true); assert.equal(zero, 0); assert.equal(big, 9007199254740993n);
assert.ok(negative instanceof py.PyFloat); assert.equal(negative.value, -Infinity);
assert.ok(value.get('10') instanceof py.PyFloat, '1.0 is a float, not an int');
assert.ok(Number.isNaN(value.get('a').value));
assert.equal(py.isInt(value.get('10')), false);
assert.equal(py.pyJsonLoads('"\\ud83d\\ude00\\ud800"'), '\u{1f600}\ud800');
for (const bad of ['{}', '{"a":1,}', '[01]', '{"a" 1}', '"\u0001"', '"\\x"', '1 2', '-', `${'9'.repeat(4301)}`, '"\\ud800\\uzzzz"']) {
assert.throws(() => py.pyJsonLoads(bad), py.PyValueError, bad);
}
assert.equal(py.pyJsonLoads(` ${'9'.repeat(4300)}\n`), BigInt('9'.repeat(4300)));
});
// One depth cap for both browser CPython-JSON parsers (py-json.mjs here, local-video.mjs for video envelopes): every
// array or object counts, an empty one too, as CPython's scanner enters recursion for each; past the cap it refuses.
test('json.loads refuses nesting past the shared depth cap, counting every container', () => {
const video = import('../../web/js/workspace/local-video.mjs');
assert.equal(py.JSON_MAX_DEPTH, 1000);
const nest = (depth, inner = '') => '['.repeat(depth) + inner + ']'.repeat(depth);
const objects = (depth, inner) => '{"a":'.repeat(depth) + inner + '}'.repeat(depth);
for (const text of [nest(1000), nest(1000, '1'), nest(999, '{}'), objects(1000, '1'), objects(999, '[]')]) assert.doesNotThrow(() => py.pyJsonLoads(text));
for (const text of [nest(1001), nest(1001, '1'), nest(1000, '{}'), objects(1001, '1'), objects(1000, '[]')]) {
assert.throws(() => py.pyJsonLoads(text), py.PyValueError);
}
return video.then(({ pythonJsonLoads, PythonJsonError }) => {
for (const text of [nest(1000), nest(1000, '1'), objects(999, '[]')]) assert.doesNotThrow(() => pythonJsonLoads(text));
for (const text of [nest(1001), nest(1001, '1'), objects(1000, '[]')]) {
assert.throws(() => pythonJsonLoads(text), error => error instanceof PythonJsonError && error.recursion === true);
}
});
});
test('str helpers measure, slice and strip as CPython', () => {
assert.equal(py.pyLen('a\u{1f600}\ud800'), 3);
assert.equal(py.pySlice('\u{1f600}\u{1f600}x', 2), '\u{1f600}\u{1f600}');
// str.isspace() includes \x1c-\x1f and \x85 and excludes , unlike String.prototype.trim.
assert.equal(py.pyStrip('\x1c\x85 a b \x1f'), 'a b');
assert.equal(py.pyStrip('a'), 'a');
assert.deepEqual([1.0, 0.0001, 1e-5, 1e16, 123456789012345.6, -2.5, 1e22].map(py.pyFloatRepr),
['1.0', '0.0001', '1e-05', '1e+16', '123456789012345.6', '-2.5', '1e+22']);
assert.deepEqual([null, true, 0, 1n, new py.PyFloat(2), '', [], new Map()].map(py.pyTruthy), [false, true, false, true, true, false, false, false]);
assert.deepEqual([1, 1n, true, new py.PyFloat(1), '1', 2].map(py.pyEqualsOne), [true, true, true, true, false, false]);
});
const encode = text => ({class_type: 'CLIPTextEncode', inputs: {text, clip: ['4', 1]}});
const sampler = (extra = {}) => ({class_type: 'KSampler', inputs: {seed: 1, positive: ['6', 0], negative: ['7', 0], latent_image: ['5', 0], model: ['4', 0], ...extra}});
const base = (nodes = {}) => ({'3': sampler(), '4': {class_type: 'CheckpointLoaderSimple', inputs: {ckpt_name: 'm.safetensors'}},
'5': {class_type: 'EmptyLatentImage', inputs: {width: 64, height: 64, batch_size: 1}}, '6': encode('a fox'), '7': encode('blurry'), ...nodes});
test('a single chain resolves both roles with their sources', () => {
const evidence = resolver.resolveLocalEvidence({prompt: JSON.stringify(base())}, semantics);
assert.deepEqual(evidence, {
prompts: {positive: {status: 'resolved', text: 'a fox', source: 'graph-inferred'}, negative: {status: 'resolved', text: 'blurry', source: 'graph-inferred'}},
resolution: {state: 'resolved', sampler_id: '3', reason: '', reason_code: null, node: null},
});
});
test('records without roles, malformed text and a missing sampler answer as the server', () => {
assert.deepEqual(resolver.resolveLocalEvidence({workflow: '{}'}, semantics).resolution,
{state: 'absent', sampler_id: null, reason: 'No parsed prompt metadata', reason_code: null, node: null});
const malformed = resolver.resolveLocalEvidence({prompt: '{broken'}, semantics);
assert.equal(malformed.resolution.reason, 'Prompt evidence is malformed');
assert.equal(malformed.prompts.positive.message, 'Prompt metadata in this file is malformed.');
const none = resolver.resolveLocalEvidence({prompt: JSON.stringify({'1': encode('x'), '2': {class_type: 'MySampler', inputs: {positive: ['1', 0]}}})}, semantics);
assert.deepEqual(none.resolution, {state: 'unresolved', sampler_id: null, reason: 'No supported sampler pair found (saw MySampler)', reason_code: 'no-sampler', node: null});
assert.equal(none.prompts.negative.message, 'No supported sampler was found.');
assert.equal(resolver.resolveLocalEvidence({prompt: JSON.stringify(base()), parameters: 'x'}, null).resolution.reason, 'Inspection node semantics are unavailable');
});
test('a float output slot is not a link, and a runtime node is named', () => {
const floatSlot = resolver.resolveLocalEvidence({prompt: JSON.stringify(base()).replace('"positive":["6",0]', '"positive":["6",0.0]')}, semantics);
assert.deepEqual(floatSlot.resolution, {state: 'unresolved', sampler_id: '3', reason: 'Expected a typed graph link or text input', reason_code: 'malformed', node: null});
const runtime = resolver.resolveLocalEvidence({prompt: JSON.stringify(base({'6': {class_type: 'PromptSearchList', inputs: {}}}))}, semantics);
assert.deepEqual(runtime.resolution.node, {id: '6', class_type: 'PromptSearchList'});
assert.equal(runtime.prompts.positive.message, 'Prompt is chosen at run time (PromptSearchList).');
});
test('execution evidence replaces a PromptManagerPair text when it matches the node inputs', () => {
const pair = {class_type: 'PromptManagerPair', inputs: {text: 'fox', negative_text: 'blur', prepend_text: '', append_text: ''}};
const record = {text: 'fox', negative_text: 'blur', positive_text: 'fox, executed', negative_effective_text: 'blur', prepend_text: '', append_text: ''};
const graph = base({'3': sampler({positive: ['8', 0], negative: ['8', 1]}), '8': pair});
const evidence = resolver.resolveLocalEvidence({prompt: JSON.stringify(graph), comfyassets_prompt_pairs: JSON.stringify({version: 1, nodes: {'8': {records: [record]}}})}, semantics);
assert.deepEqual(evidence.prompts.positive, {status: 'resolved', text: 'fox, executed', source: 'execution'});
const stale = resolver.resolveLocalEvidence({prompt: JSON.stringify(graph), comfyassets_prompt_pairs: JSON.stringify({version: 1, nodes: {'8': {records: [{...record, text: 'other'}]}}})}, semantics);
assert.equal(stale.resolution.reason, 'Execution evidence does not match recorded node inputs');
});
test('identical sampler branches resolve to their shared pair, differing ones stay ambiguous (#270)', () => {
const shared = resolver.resolveLocalEvidence({prompt: JSON.stringify(base({'9': sampler()}))}, semantics);
assert.deepEqual(shared.resolution, {state: 'resolved', sampler_id: null, reason: 'Every sampler branch uses the same prompt pair', reason_code: null, node: null});
assert.equal(shared.prompts.positive.text, 'a fox');
const differ = resolver.resolveLocalEvidence({prompt: JSON.stringify(base({'9': sampler({positive: ['8', 0]}), '8': encode('a cat')}))}, semantics);
assert.deepEqual(differ.resolution, {state: 'ambiguous', sampler_id: null, reason: 'Multiple sampler branches', reason_code: null, node: null});
assert.equal(differ.prompts.positive.status, 'ambiguous');
});
test('a detailer stage is compared with the sampler in its image ancestry', () => {
const graph = base({'10': {class_type: 'VAEDecode', inputs: {samples: ['3', 0]}}, '8': encode('a cat'), '12': {class_type: 'SaveImage', inputs: {images: ['11', 0]}},
'11': {class_type: 'FaceDetailer', inputs: {image: ['10', 0], model: ['4', 0], positive: ['8', 0], negative: ['7', 0]}}});
const evidence = resolver.resolveLocalEvidence({prompt: JSON.stringify(graph)}, semantics);
assert.deepEqual(evidence.resolution, {state: 'ambiguous', sampler_id: '11', reason: 'Sampler stages have different prompts', reason_code: 'multistage-differs', node: null});
assert.equal(evidence.prompts.positive.message, undefined);
});
test('A1111 parameters split on the unique settings footer as Python does', () => {
const read = text => resolver.resolveLocalEvidence({parameters: text}, semantics);
assert.deepEqual(read('a fox\nNegative prompt: blur\nSteps: 20, Sampler: euler').prompts, {
positive: {status: 'resolved', text: 'a fox', source: 'embedded-parameters'}, negative: {status: 'resolved', text: 'blur', source: 'embedded-parameters'},
});
// Python's multiline ^ and $ break only at \n: a U+2028 or a lone \r never starts the footer line.
for (const text of ['a fox\u2028Steps: 20, x', 'a fox\rSteps: 20, x']) assert.equal(read(text).resolution.reason, 'A1111 settings footer is not unique');
// \d is any Unicode decimal digit and \s spans line breaks, as in Python's re.
assert.equal(read('a fox\nSteps: \u0662\u0660, x').resolution.reason, 'Negative prompt delimiter is absent');
assert.equal(read('a fox\nSteps:\n 20 \n, x').resolution.state, 'absent');
assert.equal(read('a fox\nSteps: 1, x\nNegative Template: n\nTemplate: t').resolution.reason_code, 'malformed');
assert.equal(read('\u{1f600}'.repeat(60000) + '\nSteps: 1, x').resolution.state, 'absent', 'the length limit counts code points');
});
test('an unusable semantics table is refused', () => {
for (const doc of [null, {schema: 2}, {schema: 1, samplers: []}]) assert.throws(() => resolver.compileSemantics(doc), resolver.SemanticsError);
});
// The committed server answers (tests/fixtures/local-inspect/resolver-vectors.json, written by
// `python3 -m tests.local_resolver_cases --write` and checked current by the Python suite).
test('committed server answers equal the local evidence', () => {
const {vectors} = JSON.parse(fs.readFileSync(new URL('../fixtures/local-inspect/resolver-vectors.json', import.meta.url), 'utf8'));
const canonical = value => Array.isArray(value) ? value.map(canonical) :
value !== null && typeof value === 'object' ? Object.fromEntries(Object.keys(value).sort().map(key => [key, canonical(value[key])])) : value;
let compared = 0;
for (const {name, records, evidence} of vectors) {
assert.deepEqual(canonical(resolver.resolveLocalEvidence(records, semantics)), canonical(evidence), name);
compared++;
}
assert.ok(compared > 280, `compared ${compared}`);
});
test('a list or object workflow node type is skipped, as on the server (#286)', () => {
const graph = base({'6': {class_type: 'PromptManager', inputs: {text: 'a fox', clip: ['4', 1]}}});
for (const type of [['PromptManager'], {name: 'PromptManager'}]) {
const evidence = resolver.resolveLocalEvidence({prompt: JSON.stringify(graph), workflow: JSON.stringify({nodes: [{id: 6, type, widgets_values: ['x']}]})}, semantics);
assert.equal(evidence.resolution.state, 'resolved');
assert.equal(evidence.prompts.positive.status, 'resolved');
assert.equal(evidence.prompts.positive.text, 'a fox');
}
});
@@ -0,0 +1,43 @@
// web/js/workspace/local-semantics.mjs (#249 local Inspect, PR L3 Task 10): the node table local Inspect resolves
// with is the server's own web/js/inspection-semantics.json, fetched once through the API client before a file is
// chosen. A table that cannot be fetched or used answers as the server does when its table is unusable.
import test from "node:test";
import assert from "node:assert/strict";
import fs from "node:fs";
const TABLE = fs.readFileSync(new URL("../../web/js/inspection-semantics.json", import.meta.url), "utf8");
let instance = 0;
/** A fresh module instance (its cache is per page) with `fetch` and the API client stubbed. */
async function load(respond) {
const calls = [];
globalThis.window = { ApiClient: { resolveUrl: (path) => `/proxy${path}` } };
globalThis.fetch = async (url, init) => { calls.push([url, init]); return respond(); };
const module = await import(`../../web/js/workspace/local-semantics.mjs?instance=${++instance}`);
return { module, calls };
}
test.after(() => { delete globalThis.window; delete globalThis.fetch; });
test("the table is fetched once through the API client and compiled", async () => {
const { module, calls } = await load(() => new Response(TABLE, { status: 200 }));
const [first, second] = await Promise.all([module.loadInspectionSemantics(), module.loadInspectionSemantics()]);
assert.equal(first, second);
assert.ok(first.samplers.has("KSampler"));
assert.equal(await module.loadInspectionSemantics(), first);
assert.deepEqual(calls, [["/proxy/prompt_manager/js/inspection-semantics.json", { credentials: "same-origin" }]]);
});
test("a failed or unusable table is null, and a later selection asks again", async () => {
for (const respond of [() => new Response("nope", { status: 404 }), () => new Response("{\"schema\": 2}", { status: 200 }), () => { throw new TypeError("offline"); }]) {
const { module, calls } = await load(respond);
assert.equal(await module.loadInspectionSemantics(), null);
assert.equal(await module.loadInspectionSemantics(), null);
assert.equal(calls.length, 2);
}
});
test("without the API client nothing is fetched", async () => {
const { module, calls } = await load(() => new Response(TABLE));
delete globalThis.window;
assert.equal(await module.loadInspectionSemantics(), null);
assert.deepEqual(calls, []);
});
+2 -2
View File
@@ -193,8 +193,8 @@ test("unclassified input creates no object URL", async () => {
// declares a supported type it opens the inspector's unreadable state, with no preview and no URL (ledger ruling).
const gone = pngFile("gone.png", "fox", { gate: () => Promise.reject(new DOMException("The file was removed", "NotReadableError")) });
assert.equal(value.select([gone]), "accepted");
await settle();
const unreadable = value.controller.state;
// The read yields between its steps (local-inspection.mjs): wait for the answer, not a fixed number of turns.
const unreadable = await recorded(value.controller);
assert.ok(unreadable.record, "the unreadable record is published");
assert.deepEqual([value.state.phase, value.state.message, value.state.preview], ["ready", m.LOCAL_SESSION_MESSAGES.unreadable, null]);
assert.equal(m.LOCAL_SESSION_MESSAGES.unreadable, "Embedded metadata could not be read.");
+44 -2
View File
@@ -23,8 +23,50 @@ test('tags is a native route titled Tags; only unbuilt routes hand off to legacy
assert.equal(routeTitle(route), route.label);
}
assert.equal(routeTitle(resolveRoute('#/library/7')), 'Prompt library');
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
for (const id of ['system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), true, id);
// System is native from S2 (#249 System): its undelivered sections hand off per section (D94), not the route.
for (const id of ['images', 'library', 'tags', 'inspect', 'overview', 'activity', 'system']) assert.equal(isHandoff(resolveRoute(`#/${id}`)), false, id);
});
test('system resolves its sections; #/overview stays the production hub', async () => {
const { resolveRoute, routeTitle, isHandoff, routes } = await source;
// `#/system` is the Overview section and keeps the sidebar's hash.
const system = resolveRoute('#/system');
assert.deepEqual([system.id, system.section, system.hash, routeTitle(system), isHandoff(system)],
['system', 'overview', '#/system', 'System', false]);
for (const section of ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics']) {
const route = resolveRoute(`#/system/${section}`);
assert.deepEqual([route.id, route.section, route.hash], ['system', section, `#/system/${section}`]);
}
// An unknown child is the Overview section with a replaced hash.
for (const hash of ['#/system/general', '#/system/', '#/system/__proto__', '#/system/backups/1', '#/system/<b>']) {
const route = resolveRoute(hash);
assert.deepEqual([route.id, route.section, route.hash], ['system', 'overview', '#/system/overview'], hash);
}
// D28: the production hub is untouched.
const hub = resolveRoute('#/overview');
assert.deepEqual([hub.id, hub.section], ['overview', undefined]);
// Q12: Overview's System card opens the native System Overview, with no legacy href or action.
const card = routes.find(route => route.id === 'system');
assert.equal(card.href, '#/system/overview');
assert.equal(card.native, true);
assert.equal(card.action, 'Open System');
});
test('the router renders again when only the System section changes', async () => {
const { mountRouter } = await source;
const win = new EventTarget();
win.location = { hash: '#/system' };
const replacements = [];
win.history = { state: null, replaceState(state, unused, hash) { replacements.push(hash); win.location.hash = hash; } };
const seen = [];
const dispose = mountRouter(win, route => seen.push([route.id, route.section]));
for (const hash of ['#/system/appearance', '#/system/appearance', '#/system/bogus', '#/system/overview', '#/images']) {
win.location.hash = hash;
win.dispatchEvent(new Event('hashchange'));
}
assert.deepEqual(seen, [['system', 'overview'], ['system', 'appearance'], ['system', 'overview'], ['images', undefined]]);
assert.deepEqual(replacements, ['#/system/overview']);
dispose();
});
test('inspect is the native Inspect a file page with the concept description', async () => {
+214
View File
@@ -0,0 +1,214 @@
// System controls (#249 System S2 Task 5): the concept's `setting()` (app.js 624–634, its toggles hydrated into Toolkit
// switches by `hydrateSwitches` 708–717), `operation()` (645–654) and `sectionNav()` (306–308) as DOM builders whose
// listeners live on the caller's scope.
const test = require('node:test');
const assert = require('node:assert/strict');
const controlsModule = import('../../web/js/workspace/system-controls.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
test('settingRow is the concept row: div > label[for] + p, then the control', async () => {
const { settingRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const changes = [];
const row = settingRow({ id: 'timeout', label: 'Node search auto-hide (seconds)',
description: 'Node search behavior only. Use 0 to keep results visible.',
control: { type: 'number', value: 30, min: 0, max: 300 }, scope, onChange: value => changes.push(value) });
assert.equal(row.className, 'setting-row');
assert.deepEqual(dom.tags(row), ['div', 'input']);
const [text, input] = row.children;
assert.deepEqual(dom.tags(text), ['label', 'p']);
assert.equal(text.children[0].textContent, 'Node search auto-hide (seconds)');
assert.equal(text.children[0].htmlFor, 'timeout');
assert.equal(text.children[1].textContent, 'Node search behavior only. Use 0 to keep results visible.');
assert.deepEqual([input.id, input.type, input.value, input.min, input.max], ['timeout', 'number', '30', '0', '300']);
assert.equal(input.dataset.setting, 'timeout');
input.value = '45';
input.dispatch('change');
assert.deepEqual(changes, [45]);
const select = settingRow({ id: 'open-mode', label: 'Open from ComfyUI', description: 'Keep the node pack’s popup / new-tab preference.',
control: { type: 'select', options: [['newtab', 'New tab'], ['popup', 'Popup window']], value: 'popup' }, scope,
onChange: value => changes.push(value) }).children[1];
assert.equal(select.tagName, 'SELECT');
assert.deepEqual(select.children.map(option => [option.value, option.textContent]), [['newtab', 'New tab'], ['popup', 'Popup window']]);
assert.equal(select.value, 'popup');
select.value = 'newtab';
select.dispatch('change');
assert.deepEqual(changes, [45, 'newtab']);
// The concept's range is 1–16 unless the row says otherwise (D113: the CPU count).
const range = settingRow({ id: 'workers', label: 'Worker threads', description: 'Balance background scans.',
control: { type: 'range', value: 4 } }).children[1];
assert.deepEqual([range.type, range.min, range.max, range.value], ['range', '1', '16', '4']);
const password = settingRow({ id: 'civitai-key', label: 'CivitAI API key', description: 'Credentials stay on the server.',
control: { type: 'password', value: '' } }).children[1];
assert.equal(password.type, 'password');
// A ready-made control is placed as given and takes the row id.
const custom = document.createElement('textarea');
const customRow = settingRow({ id: 'custom', label: 'Custom', description: 'Any control.', control: custom });
assert.equal(customRow.children[1], custom);
assert.equal(custom.id, 'custom');
assert.throws(() => settingRow({ id: '', label: 'x', description: 'y', control: { type: 'number' } }), TypeError);
assert.throws(() => settingRow({ id: 'x', label: 'x', description: 'y', control: { type: 'colour' } }), TypeError);
} finally {
scope.dispose();
dom.restore();
}
});
test('a toggle row holds the Toolkit switch, labelled by the row label', async () => {
const { settingRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const changes = [];
const row = settingRow({ id: 'video-auto', label: 'Autoplay videos', description: 'Videos stay within the same gallery.',
control: { type: 'toggle', checked: false }, scope, onChange: value => changes.push(value) });
const field = row.children[1];
// createSwitch('', '', id): `label.ca-switch` > input[type=checkbox][role=switch] + an empty label text.
assert.equal(field.tagName, 'LABEL');
assert.equal(field.className, 'ca-switch');
const input = field.children[0];
assert.deepEqual([input.type, input.role, input.id, input.checked], ['checkbox', 'switch', 'video-auto', false]);
assert.equal(row.children[0].children[0].htmlFor, input.id);
assert.equal(input.dataset.setting, 'video-auto');
input.checked = true;
input.dispatch('change');
assert.deepEqual(changes, [true]);
// The concept's toggles are on unless configured off (`val !== false`).
const on = settingRow({ id: 'video-loop', label: 'Loop videos', description: 'Repeat a clip.', control: { type: 'toggle' } });
assert.equal(on.children[1].children[0].checked, true);
} finally {
scope.dispose();
dom.restore();
}
});
test('operationRow is the concept operation with a danger class only when flagged', async () => {
const { operationRow } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const clicks = [];
const row = operationRow({ title: 'Missing-file links',
description: 'Remove stale database links only. Files on disk are never deleted by this operation.',
label: 'Review plan', danger: true, scope, onClick: button => clicks.push(button) });
assert.equal(row.className, 'operation');
assert.deepEqual(dom.tags(row), ['div', 'div']);
assert.deepEqual(dom.tags(row.children[0]), ['h3', 'p']);
assert.equal(row.children[0].children[0].textContent, 'Missing-file links');
assert.equal(row.children[1].className, 'row-actions');
const button = row.children[1].children[0];
assert.deepEqual([button.tagName, button.type, button.textContent, button.className], ['BUTTON', 'button', 'Review plan', 'danger']);
button.click();
assert.deepEqual(clicks, [button]);
const plain = operationRow({ title: 'Statistics', description: 'Usage by prompt, category, tag and date.', label: 'Explore',
scope, onClick() {} });
assert.equal(plain.children[1].children[0].className, '');
assert.throws(() => operationRow({ title: 'x', description: 'y', label: 'z', onClick() {} }), TypeError);
} finally {
scope.dispose();
dom.restore();
}
});
test('sectionNav marks the active section and reports the chosen one', async () => {
const { sectionNav } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const chosen = [];
const nav = sectionNav({ label: 'System sections', sections: [['overview', 'Overview'], ['appearance', 'Appearance']],
active: 'appearance', scope, onSelect: id => chosen.push(id) });
assert.equal(nav.className, 'section-nav');
assert.equal(nav.getAttribute('role'), 'group');
assert.equal(nav.getAttribute('aria-label'), 'System sections');
assert.deepEqual(nav.children.map(button => [button.textContent, button.getAttribute('aria-pressed'), button.dataset.value]),
[['Overview', 'false', 'overview'], ['Appearance', 'true', 'appearance']]);
nav.children[0].click();
assert.deepEqual(chosen, ['overview']);
} finally {
scope.dispose();
dom.restore();
}
});
test('disposing the scope leaves no listener behind', async () => {
const { settingRow, operationRow, sectionNav } = await controlsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
settingRow({ id: 'a', label: 'A', description: 'a', control: { type: 'toggle' }, scope, onChange() {} });
settingRow({ id: 'b', label: 'B', description: 'b', control: { type: 'select', options: ['Grid', 'List'] }, scope, onChange() {} });
operationRow({ title: 'C', description: 'c', label: 'Go', scope, onClick() {} });
sectionNav({ label: 'System sections', sections: [['overview', 'Overview']], active: 'overview', scope, onSelect() {} });
assert.equal(dom.listenerCount() > 0, true);
scope.dispose();
assert.equal(dom.listenerCount(), 0);
} finally {
scope.dispose();
dom.restore();
}
});
+255
View File
@@ -0,0 +1,255 @@
// System › Overview (#249 System S2 Task 6): the concept's Overview copy on real values (D93), the Health report rows
// (D95), the Statistics tables (D96) and the health check run that waits for its read job.
const test = require('node:test');
const assert = require('node:assert/strict');
const overviewModule = import('../../web/js/workspace/system-overview.mjs');
const OVERVIEW = Object.freeze({
library: { positive_prompts: 1234, negative_variants: 7 },
images: { indexed: 24, rated: 20 },
health: null,
backup: null,
needs_review: { pending: null },
sources: { configured: 2, effective: 2, auto_detected: false },
});
test('metrics are the server counts and Not checked before any health check', async () => {
const { metricValues } = await overviewModule;
assert.deepEqual(metricValues(OVERVIEW), [
[(1234).toLocaleString(), 'Positive prompts'], ['7', 'Negative variants'], ['24', 'Images'],
['Not checked', 'Database integrity'],
]);
});
test('metric 4 is the last health result', async () => {
const { integrityText } = await overviewModule;
assert.equal(integrityText(null), 'Not checked');
assert.equal(integrityText({ status: 'passed', issues: 0 }), 'Passed');
assert.equal(integrityText({ status: 'issues', issues: 1 }), '1 issue');
assert.equal(integrityText({ status: 'issues', issues: 3 }), '3 issues');
assert.equal(integrityText({ status: 'incomplete', issues: 0 }), 'Incomplete');
// Issues found beside a check that could not run (Needs review, Q6) are still named.
assert.equal(integrityText({ status: 'incomplete', issues: 2 }), '2 issues');
});
test('library health rows never invent a review count, a backup or thumbnails', async () => {
const { ambiguousTitle, backupText } = await overviewModule;
assert.equal(ambiguousTitle({ pending: null }), 'Ambiguous image links');
assert.equal(ambiguousTitle(undefined), 'Ambiguous image links');
assert.equal(ambiguousTitle({ pending: 1 }), '1 ambiguous image link');
assert.equal(ambiguousTitle({ pending: 3 }), '3 ambiguous image links');
assert.equal(backupText(null), 'No retained backup record.');
assert.equal(backupText({ verified_at: 'not a time' }), 'No retained backup record.');
assert.match(backupText({ verified_at: '2026-10-22T09:30:00Z' }), /^Last verified backup .+\.$/);
});
test('the thumbnail row is its own read of the thumbnail status', async () => {
const { thumbnailTitle, THUMBNAILS_LOADING } = await overviewModule;
// Until `/images/thumbnails/status` answers the row says it is checking; a failed read is unavailable, never 0.
assert.equal(THUMBNAILS_LOADING, 'Checking thumbnails…');
assert.equal(thumbnailTitle(undefined), THUMBNAILS_LOADING);
assert.equal(thumbnailTitle({ pending: 0 }), 'No missing thumbnails');
assert.equal(thumbnailTitle({ pending: 1 }), '1 missing thumbnail');
assert.equal(thumbnailTitle({ pending: 12 }), '12 missing thumbnails');
assert.equal(thumbnailTitle(null), 'Thumbnail status unavailable');
assert.equal(thumbnailTitle({ pending: 'x' }), 'Thumbnail status unavailable');
// With no image source there is nothing to count.
assert.equal(thumbnailTitle({ pending: 0 }, { effective: 0 }), 'Thumbnail status unavailable');
assert.equal(thumbnailTitle({ pending: 2 }, { effective: 1 }), '2 missing thumbnails');
});
test('an integrity count SQLite stopped at reads as a lower bound', async () => {
const { checkChip } = await overviewModule;
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 100, capped: true }), '100+ problems found');
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 37, capped: false }), '37 problems found');
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 1 }), '1 problem found');
});
test('the sources row tells configured, available and detected sources apart', async () => {
const { sourcesCopy } = await overviewModule;
const shared = 'Monitoring and enrichment share one source list.';
assert.deepEqual(sourcesCopy({ configured: 2, effective: 2, auto_detected: false }), ['2 output sources', shared]);
assert.deepEqual(sourcesCopy({ configured: 2, effective: 1, auto_detected: false }),
['1 output source', `1 of 2 configured sources available. ${shared}`]);
assert.deepEqual(sourcesCopy({ configured: 0, effective: 1, auto_detected: true }),
['1 output source', `None configured is available; ComfyUI's output folder is used. ${shared}`]);
});
const REPORT = Object.freeze({
checked_at: '2026-10-23T08:00:00Z',
status: 'issues',
checks: [
{ id: 'integrity', label: 'Integrity check', status: 'passed', count: 0 },
{ id: 'foreign_keys', label: 'Foreign keys & pair references', status: 'issues', count: 2 },
{ id: 'canonical_hashes', label: 'Canonical hashes', status: 'issues', count: 1 },
{ id: 'pair_uniqueness', label: 'Pair uniqueness', status: 'failed', count: null },
{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 2,
sources: [{ name: 'output', status: 'checked', count: 2 }] },
{ id: 'ambiguous_metadata', label: 'Ambiguous metadata', status: 'unavailable', count: null },
{ id: 'consistency', label: 'Consistency', status: 'issues', count: null },
],
});
test('health rows give each check a chip from its real status, in the server order', async () => {
const { healthRows } = await overviewModule;
assert.deepEqual(healthRows(REPORT), [
['Integrity check', 'Passed'],
['Foreign keys & pair references', '2 broken references'],
['Canonical hashes', '1 hash does not match its text'],
['Pair uniqueness', 'Could not check'],
['Missing files', '2 links need review'],
['Ambiguous metadata', 'Not available yet'],
['Consistency', 'Issues found'],
]);
});
test('missing files is reported per image source when there are several', async () => {
const { healthRows } = await overviewModule;
const report = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'issues', count: 1, partial: true,
sources: [{ name: 'output', status: 'checked', count: 1 }, { name: 'input', status: 'checked', count: 0 },
{ name: 'archive', status: 'unavailable', count: null }] }] };
assert.deepEqual(healthRows(report), [
['Missing files · output', '1 link needs review'],
['Missing files · input', 'Passed'],
['Missing files · archive', 'Not available'],
]);
const unavailable = { ...REPORT, checks: [{ id: 'missing_files', label: 'Missing files', status: 'unavailable', count: null, sources: [] }] };
assert.deepEqual(healthRows(unavailable), [['Missing files', 'Not available']]);
});
test('the report summary says when it was checked and whether it is complete', async () => {
const { healthSummary } = await overviewModule;
const when = new Date(REPORT.checked_at).toLocaleString();
assert.equal(healthSummary({ ...REPORT, status: 'passed' }), `Checked ${when}. Every check passed.`);
assert.equal(healthSummary({ ...REPORT, status: 'incomplete' }), `Checked ${when}. Some checks could not run, so this report is incomplete.`);
assert.equal(healthSummary({ ...REPORT, checked_at: 'x' }), 'Nothing was changed; review the checks with issues.');
});
test('statistics rows are the real runs and ratings, then the library totals', async () => {
const { categoryRows, totalsRows } = await overviewModule;
const statistics = {
by_category: [{ category: 'Landscapes', runs: 1200, average_rating: 4.5 },
{ category: null, runs: 3, average_rating: null }],
totals: { total_prompts: 12, total_tags: 15, total_images: 24 },
rated_images: 20,
};
assert.deepEqual(categoryRows(statistics), [['Landscapes', (1200).toLocaleString(), '4.5 / 5'], ['Uncategorized', '3', 'Not rated']]);
assert.deepEqual(totalsRows(statistics), [['Prompts', '12'], ['Tags', '15'], ['Images', '24'], ['Rated images', '20']]);
});
/** A fake api answering each `get`/`post` from a queue per URL, recording the calls. */
function fakeApi(answers) {
const calls = [];
const answer = (method, url) => {
calls.push(`${method} ${url}`);
const queue = answers[`${method} ${url}`];
if (!queue?.length) throw new Error(`unexpected ${method} ${url}`);
const next = queue.shift();
return next instanceof Error ? Promise.reject(next) : Promise.resolve(next);
};
return { calls, get: url => answer('GET', url), post: url => answer('POST', url) };
}
const JOB = '0123456789abcdef0123456789abcdef';
const job = (state, id = JOB, kind = 'health') => ({ id, kind, state });
/**
* A fake Activity store holding `jobs`: `track(job)` lists a started job, `load()` applies the next of `reads` (the
* server's job lists) and `push(job)` is a stream update; each notifies as the real store does. It never polls.
*/
function fakeActivity({ jobs = [], reads = [] } = {}) {
const listeners = new Set();
const emit = () => { for (const listener of [...listeners]) listener(); };
const store = {
loads: 0,
find: id => jobs.find(item => item.id === id),
running: () => jobs.filter(item => item.state === 'running'),
subscribe(listener) { listeners.add(listener); return () => listeners.delete(listener); },
listeners: () => listeners.size,
async load() {
store.loads += 1;
if (reads.length) jobs = reads.shift();
emit();
},
track(item) {
if (!item?.id) throw new TypeError('Invalid job');
store.push(item);
return item;
},
push(item) {
jobs = [...jobs.filter(other => other.id !== item.id), item];
emit();
},
};
return store;
}
test('a health check tracks its job in the Activity store, then reads the report', async () => {
const { runHealthCheck } = await overviewModule;
const api = fakeApi({
'POST /prompt_manager/system/health': [{ job: job('running') }],
'GET /prompt_manager/system/health': [{ report: REPORT }],
});
const activity = fakeActivity();
const checking = runHealthCheck(api, { activity });
await new Promise(resolve => setImmediate(resolve));
// The store follows the started job's event stream; nothing reads or polls the job here.
assert.equal(activity.loads, 0);
assert.equal(activity.find(JOB).state, 'running');
assert.deepEqual(api.calls, ['POST /prompt_manager/system/health']);
activity.push(job('finished'));
assert.equal(await checking, REPORT);
assert.deepEqual(api.calls, ['POST /prompt_manager/system/health', 'GET /prompt_manager/system/health']);
assert.equal(activity.listeners(), 0);
});
test('a check already running elsewhere is waited for, not refused', async () => {
const { runHealthCheck } = await overviewModule;
const busy = Object.assign(new Error('busy'), { status: 409, code: 'busy' });
const api = fakeApi({
'POST /prompt_manager/system/health': [busy],
'GET /prompt_manager/system/health': [{ report: REPORT }],
});
const activity = fakeActivity({ reads: [[job('running', 'f'.repeat(32), 'scan'), job('running')]] });
const checking = runHealthCheck(api, { activity });
await new Promise(resolve => setImmediate(resolve));
assert.equal(activity.loads, 1);
activity.push(job('finished'));
assert.equal(await checking, REPORT);
// The other check ended between the refusal and the read: its report is the latest.
const ended = fakeApi({ 'POST /prompt_manager/system/health': [busy], 'GET /prompt_manager/system/health': [{ report: REPORT }] });
assert.equal(await runHealthCheck(ended, { activity: fakeActivity({ reads: [[]] }) }), REPORT);
});
test('a failed, lost or reportless check rejects; other errors pass through', async () => {
const { runHealthCheck, waitForJob, HEALTH_FAILED_TEXT } = await overviewModule;
const failed = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('failed') }] });
await assert.rejects(runHealthCheck(failed, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT });
// The store marks a job lost when the server no longer lists it (ComfyUI restarted).
const restarted = fakeActivity({ jobs: [job('running')] });
const waiting = waitForJob(restarted, JOB);
restarted.push(job('lost'));
assert.equal(await waiting, 'lost');
const empty = fakeApi({ 'POST /prompt_manager/system/health': [{ job: job('finished') }],
'GET /prompt_manager/system/health': [{ report: null }] });
await assert.rejects(runHealthCheck(empty, { activity: fakeActivity() }), { message: HEALTH_FAILED_TEXT });
const broken = fakeApi({ 'POST /prompt_manager/system/health': [Object.assign(new Error('server'), { status: 500 })] });
await assert.rejects(runHealthCheck(broken, { activity: fakeActivity() }), { message: 'server' });
const invalid = fakeApi({ 'POST /prompt_manager/system/health': [{ job: null }] });
await assert.rejects(runHealthCheck(invalid, { activity: fakeActivity() }), { message: 'Invalid job' });
});
test('waiting stops when its signal aborts', async () => {
const { waitForJob } = await overviewModule;
const controller = new AbortController();
const activity = fakeActivity({ jobs: [job('running')] });
const waiting = waitForJob(activity, JOB, { signal: controller.signal });
await new Promise(resolve => setImmediate(resolve));
controller.abort(new Error('left'));
await assert.rejects(waiting, { message: 'left' });
assert.equal(activity.listeners(), 0);
const gone = new AbortController();
gone.abort(new Error('already'));
await assert.rejects(waitForJob(fakeActivity(), JOB, { signal: gone.signal }), { message: 'already' });
});
+261
View File
@@ -0,0 +1,261 @@
// System draft store (#249 System S2 Task 5): one Save row for browser, host and runtime fields (concept app.js
// renderSystem `.save-row` 705, `save-settings` 1320). It saves browser, then host with the GET revision, then runtime;
// a domain that fails keeps its fields dirty, and the status copy (D92) says what really happened.
const test = require('node:test');
const assert = require('node:assert/strict');
const source = import('../../web/js/workspace/system-state.mjs');
/** Adapters that record every call; `host` answers like POST /prompt_manager/settings. */
function adapters({ browser = {}, host = {}, runtime = {}, revision = 'sha256:one' } = {}) {
const calls = [];
const outcome = { browser: null, host: null, runtime: null };
return {
calls,
outcome,
readBrowser: () => { calls.push(['readBrowser']); return { ...browser }; },
readHost: async () => { calls.push(['readHost']); return { values: { ...host }, revision }; },
readRuntime: async () => { calls.push(['readRuntime']); return { ...runtime }; },
writeBrowser: async changes => {
calls.push(['writeBrowser', changes]);
if (outcome.browser) return outcome.browser(changes);
return { persisted: true };
},
writeHost: async (changes, options) => {
calls.push(['writeHost', changes, options]);
if (outcome.host) return outcome.host(changes, options);
return { success: true, persisted: true, revision: 'sha256:two', runtime_applied: true };
},
writeRuntime: async changes => {
calls.push(['writeRuntime', changes]);
if (outcome.runtime) return outcome.runtime(changes);
return { success: true };
},
};
}
async function loaded(domains) {
const { createSystemDrafts } = await source;
const drafts = createSystemDrafts(domains);
await Promise.all(['browser', 'host', 'runtime'].map(domain => drafts.load(domain)));
domains.calls.length = 0;
return drafts;
}
test('sections follow the concept order and the production hashes', async () => {
const { SYSTEM_SECTIONS, isSystemSection } = await source;
assert.deepEqual(SYSTEM_SECTIONS.map(([id, label]) => [id, label]), [
['overview', 'Overview'], ['appearance', 'Appearance'], ['sources', 'Sources & scans'],
['integrations', 'Integrations'], ['tagging', 'Auto-tagging'], ['maintenance', 'Maintenance'],
['backups', 'Backup & export'], ['diagnostics', 'Diagnostics & logs'],
]);
assert.equal(isSystemSection('backups'), true);
for (const value of ['general', '', 'constructor', '__proto__', undefined]) assert.equal(isSystemSection(value), false, value);
});
test('drafts read baselines and report dirty only for changed values', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } });
const drafts = await loaded(domains);
assert.equal(drafts.get('browser', 'layout'), 'grid');
assert.equal(drafts.get('host', 'result_timeout'), 30);
assert.equal(drafts.dirty(), false);
drafts.set('host', 'result_timeout', 45);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.equal(drafts.dirty(), true);
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.dirty('browser'), false);
// Setting the saved value back is clean again.
drafts.set('host', 'result_timeout', 30);
assert.equal(drafts.dirty(), false);
assert.throws(() => drafts.set('elsewhere', 'x', 1), TypeError);
assert.throws(() => drafts.get('elsewhere', 'x'), TypeError);
});
test('save commits browser then host then runtime', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 }, runtime: { level: 'INFO' } });
const drafts = await loaded(domains);
drafts.set('runtime', 'level', 'DEBUG');
drafts.set('host', 'result_timeout', 45);
drafts.set('browser', 'layout', 'list');
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'saved', host: 'saved', runtime: 'applied' });
assert.deepEqual(domains.calls, [
['writeBrowser', { layout: 'list' }],
['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }],
['writeRuntime', { level: 'DEBUG' }],
]);
assert.equal(drafts.dirty(), false);
// The next host save sends the revision the last one returned.
drafts.set('host', 'result_timeout', 60);
await drafts.save();
assert.deepEqual(domains.calls.at(-1), ['writeHost', { result_timeout: 60 }, { expected_revision: 'sha256:two' }]);
});
test('host failure keeps host fields dirty and browser saved', async () => {
const domains = adapters({ browser: { layout: 'grid' }, host: { result_timeout: 30 } });
const drafts = await loaded(domains);
domains.outcome.host = () => { throw Object.assign(new Error('Settings could not be written to config.json.'), { code: 'config-write-failed' }); };
drafts.set('browser', 'layout', 'list');
drafts.set('host', 'result_timeout', 45);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'saved', host: 'failed', runtime: 'clean' });
assert.equal(drafts.dirty('browser'), false);
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.deepEqual([...drafts.failed('browser')], []);
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
// A save answered `persisted: false` without an error is not saved either.
domains.outcome.host = () => ({ success: false, persisted: false });
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' });
// Editing the failed field clears its highlight and reads as unsaved again.
drafts.set('host', 'result_timeout', 50);
assert.deepEqual([...drafts.failed('host')], []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('stale revision reports conflict and keeps the draft', async () => {
const domains = adapters({ host: { result_timeout: 30 } });
const drafts = await loaded(domains);
domains.outcome.host = () => { throw Object.assign(new Error('Settings changed since they were loaded.'), { code: 'settings-changed' }); };
drafts.set('host', 'result_timeout', 45);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'conflict', runtime: 'clean' });
assert.equal(drafts.dirty('host'), true);
assert.equal(drafts.get('host', 'result_timeout'), 45);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
// The conflict read the host values again (their new revision), keeping the draft.
assert.deepEqual(domains.calls.slice(-2).map(([name]) => name), ['writeHost', 'readHost']);
// Reloading the section reads the new revision and keeps the draft (Q10).
domains.outcome.host = null;
await drafts.load('host');
assert.equal(drafts.get('host', 'result_timeout'), 45);
await drafts.save();
assert.deepEqual(domains.calls.at(-1)[2], { expected_revision: 'sha256:one' });
});
test('host save without a read revision never posts', async () => {
const { createSystemDrafts } = await source;
// Never read: an edit made before the host values (and their revision) arrived must not merge blindly.
const unread = adapters({ host: { result_timeout: 30 } });
const drafts = createSystemDrafts(unread);
drafts.set('host', 'result_timeout', 45);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'failed', runtime: 'clean' });
assert.deepEqual(unread.calls, []);
assert.equal(drafts.dirty('host'), true);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
// A read of an unreadable configuration has no revision either.
const unreadable = adapters({ host: { result_timeout: 30 }, revision: null });
const blind = createSystemDrafts(unreadable);
await blind.load('host');
blind.set('host', 'result_timeout', 45);
assert.equal((await blind.save()).host, 'failed');
assert.equal(unreadable.calls.filter(([name]) => name === 'writeHost').length, 0);
// Once read, the same draft saves with the revision.
await drafts.load('host');
assert.equal((await drafts.save()).host, 'saved');
assert.deepEqual(unread.calls.at(-1), ['writeHost', { result_timeout: 45 }, { expected_revision: 'sha256:one' }]);
});
test('reloadHost reads the host values and says whether it could', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 30 } });
const drafts = createSystemDrafts(domains);
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.get('host', 'result_timeout'), 30);
const broken = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } });
assert.equal(await broken.reloadHost(), false);
broken.set('host', 'result_timeout', 45);
assert.equal((await broken.save()).host, 'failed');
});
test('no domain called when clean', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls, []);
drafts.set('browser', 'layout', 'list');
drafts.discard();
assert.equal(drafts.dirty(), false);
assert.equal(drafts.get('browser', 'layout'), 'grid');
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls, []);
});
test('status text per outcome', async () => {
const { statusText, SETTINGS_STATUS } = await source;
assert.deepEqual(SETTINGS_STATUS, {
clean: 'No unsaved changes.',
dirty: 'Unsaved changes.',
browser: 'Saved in this browser.',
host: 'Saved to the server configuration.',
runtime: 'Applied until the server restarts.',
partial: 'Some changes were not saved. Review the highlighted settings.',
tabOnly: 'Saved for this tab only; browser storage is unavailable.',
conflict: 'Settings changed elsewhere. Reload this section to see them.',
});
const clean = { browser: 'clean', host: 'clean', runtime: 'clean' };
assert.equal(statusText(clean), 'No unsaved changes.');
assert.equal(statusText({ ...clean, browser: 'saved' }), 'Saved in this browser.');
assert.equal(statusText({ ...clean, host: 'saved' }), 'Saved to the server configuration.');
assert.equal(statusText({ ...clean, runtime: 'applied' }), 'Applied until the server restarts.');
assert.equal(statusText({ ...clean, browser: 'tab-only' }), 'Saved for this tab only; browser storage is unavailable.');
assert.equal(statusText({ browser: 'saved', host: 'saved', runtime: 'applied' }),
'Saved in this browser. Saved to the server configuration. Applied until the server restarts.');
assert.equal(statusText({ ...clean, browser: 'saved', runtime: 'failed' }),
'Some changes were not saved. Review the highlighted settings.');
assert.equal(statusText({ browser: 'failed', host: 'conflict', runtime: 'clean' }),
'Settings changed elsewhere. Reload this section to see them.');
});
test('the store says unsaved, then the save outcome, and notifies subscribers', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
const seen = [];
const unsubscribe = drafts.subscribe(() => seen.push(drafts.statusText()));
assert.equal(drafts.statusText(), 'No unsaved changes.');
drafts.set('browser', 'layout', 'list');
domains.outcome.browser = () => ({ persisted: false });
await drafts.save();
assert.deepEqual(seen, ['Unsaved changes.', 'Saved for this tab only; browser storage is unavailable.']);
unsubscribe();
drafts.set('browser', 'layout', 'masonry');
assert.equal(seen.length, 2);
// The tab-only value is this tab's baseline now.
drafts.set('browser', 'layout', 'list');
assert.equal(drafts.dirty(), false);
});
test('an edit made while a save runs stays dirty', async () => {
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = await loaded(domains);
let release;
domains.outcome.browser = () => new Promise(resolve => { release = () => resolve({ persisted: true }); });
drafts.set('browser', 'layout', 'list');
const saving = drafts.save();
// A second Save while one runs joins it instead of writing twice.
assert.equal(drafts.save(), saving);
assert.equal(drafts.saving(), true);
await new Promise(resolve => setImmediate(resolve));
drafts.set('browser', 'layout', 'masonry');
release();
assert.deepEqual(await saving, { browser: 'saved', host: 'clean', runtime: 'clean' });
assert.equal(drafts.saving(), false);
assert.equal(drafts.dirty('browser'), true);
assert.equal(drafts.get('browser', 'layout'), 'masonry');
assert.equal(domains.calls.filter(([name]) => name === 'writeBrowser').length, 1);
});
test('a failed read keeps the store usable and drafts survive a reload (Q10)', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ browser: { layout: 'grid' } });
const drafts = createSystemDrafts(domains);
await drafts.load('browser');
drafts.set('browser', 'layout', 'list');
await drafts.load('browser');
assert.equal(drafts.get('browser', 'layout'), 'list');
assert.equal(drafts.dirty(), true);
const failing = createSystemDrafts({ ...domains, readHost: async () => { throw new Error('offline'); } });
await assert.rejects(failing.load('host'), /offline/);
assert.equal(failing.dirty(), false);
assert.throws(() => createSystemDrafts({ ...domains, writeHost: null }), TypeError);
});
+147
View File
@@ -76,6 +76,7 @@ function fakeDOM() {
get activeElement() { return active ?? body; },
createElement: tag => new Element(tag),
createElementNS: (namespace, tag) => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const all = (root, predicate, found = []) => {
if (predicate(root)) found.push(root);
@@ -521,6 +522,10 @@ test('copy fallback selects the text', async () => {
assert.equal(note.className, 'small');
assert.equal(note.textContent, 'Automatic clipboard access requires a secure browser context.');
assert.throws(() => openCopyFallback(42, { scope }), TypeError);
// LI-4 (D117): the settings summary names its own dialog; the select-to-copy body is the same.
openCopyFallback('Seed: 7', { scope, title: 'Copy settings summary' });
assert.equal(byId(dom, 'modal-title').textContent, 'Copy settings summary');
assert.equal(byId(dom, 'copy-text').value, 'Seed: 7');
} finally {
closeWorkspaceDialog();
scope.dispose();
@@ -786,3 +791,145 @@ test('the Filters button shows the active refinement count', async () => {
dom.restore();
}
});
// #249 System S2 Task 5: the scan review's dialog and backup box (D55) generalised for every record-changing review.
const scanModule = import('../../web/js/workspace/activity-scan.mjs');
const bodyShape = dom => byId(dom, 'modal-body').children.map(node => [node.tagName.toLowerCase(), node.className].filter(Boolean).join('.'));
test('backupNotice is the concept .notice with its copy and one Back up now action', async () => {
const { backupNotice, SCAN_BACKUP_COPY } = await dialogsModule;
const dom = fakeDOM();
try {
let calls = 0;
const notice = backupNotice({ copy: SCAN_BACKUP_COPY, onBackup: () => { calls++; } });
assert.equal(notice.className, 'notice');
assert.deepEqual(tags(notice), ['div', 'div']);
assert.deepEqual(tags(notice.children[0]), ['h3', 'p']);
assert.equal(notice.children[0].children[0].textContent, 'Back up the database first (recommended)');
assert.equal(notice.children[0].children[1].textContent,
'Downloads a copy of the prompts database before the scan changes any records.');
assert.equal(notice.children[1].className, 'row-actions');
const action = button(notice, 'Back up now');
action.click();
assert.equal(calls, 1);
assert.throws(() => backupNotice({ copy: { heading: '', text: 'x' }, onBackup() {} }), TypeError);
assert.throws(() => backupNotice({ copy: SCAN_BACKUP_COPY }), TypeError);
} finally {
dom.restore();
}
});
test('openReviewDialog lays out heading, grid, options, sentence, backup box and actions', async () => {
const { openReviewDialog, closeWorkspaceDialog } = await dialogsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const trigger = new dom.Element('button');
dom.body.append(trigger);
const option = new dom.Element('div');
option.className = 'setting-row';
const started = [];
const review = openReviewDialog({ scope, title: 'Review maintenance plan', heading: 'Missing-file links',
grid: [['Scope', '2 stale image links'], ['Media on disk', 'No source files removed']], options: [option],
sentence: 'Cancelling leaves all data untouched.', backup: { copy: { heading: 'Back up first', text: 'A copy.' }, onBackup() {} },
cancelLabel: 'Cancel', primary: { label: 'Apply reviewed operation' }, onStart: () => started.push('start'), trigger });
assert.equal(byId(dom, 'modal-title').textContent, 'Review maintenance plan');
assert.deepEqual(bodyShape(dom), ['h3', 'dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']);
const grid = byId(dom, 'modal-body').children[1];
assert.deepEqual(grid.children.map(entry => [tags(entry), entry.children[0].textContent, entry.children[1].textContent]), [
[['dt', 'dd'], 'Scope', '2 stale image links'], [['dt', 'dd'], 'Media on disk', 'No source files removed']]);
review.setValue('Scope', '3 stale image links');
assert.equal(grid.children[0].children[1].textContent, '3 stale image links');
const actions = byId(dom, 'modal-body').children.at(-1);
assert.deepEqual(actions.children.map(node => [node.textContent, node.className]),
[['Cancel', ''], ['Apply reviewed operation', 'ca-primary']]);
actions.children[1].click();
assert.deepEqual(started, ['start']);
assert.equal(modal(dom).open, false);
assert.equal(document.activeElement, trigger);
// A danger primary (the concept's Restore) has no sentence, heading, options or backup box when none is given.
openReviewDialog({ scope, title: 'Restore preview', grid: [['Integrity', 'Verified']], backup: null,
cancelLabel: 'Cancel', primary: { label: 'Restore', danger: true }, onStart() {} });
assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.dialog-actions']);
assert.equal(button(byId(dom, 'modal-body'), 'Restore').className, 'danger');
button(byId(dom, 'modal-body'), 'Cancel').click();
assert.equal(modal(dom).open, false);
assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [['a']], cancelLabel: 'Cancel', primary: { label: 'Go' },
onStart() {} }), TypeError);
assert.throws(() => openReviewDialog({ scope, title: 'x', grid: [], cancelLabel: 'Cancel', primary: {}, onStart() {} }), TypeError);
} finally {
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
test('review dialog backup box keeps the dialog open and toasts the outcome', async () => {
const { openReviewDialog, closeWorkspaceDialog, SCAN_BACKUP_COPY } = await dialogsModule;
const { downloadDatabaseBackup, BACKUP_DONE, BACKUP_FAILED } = await scanModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
const created = URL.createObjectURL;
const revoked = URL.revokeObjectURL;
URL.createObjectURL = () => 'blob:backup';
URL.revokeObjectURL = () => {};
try {
const toasts = [];
let answer;
const fetchUrl = () => new Promise(resolve => { answer = resolve; });
openReviewDialog({ scope, title: 'Review metadata scan', grid: [['Sources', '2 output folders']],
backup: { copy: SCAN_BACKUP_COPY, onBackup: () => downloadDatabaseBackup({ fetchUrl, notify: text => toasts.push(text) }) },
cancelLabel: 'Cancel', primary: { label: 'Start scan' }, onStart() { throw new Error('not started'); } });
const action = button(byId(dom, 'modal-body'), 'Back up now');
action.click();
assert.equal(action.textContent, 'Backing up…');
assert.equal(action.getAttribute('aria-disabled'), 'true');
// A second press while the copy downloads is ignored.
action.click();
answer({ ok: true, blob: async () => new Blob(['db']), headers: { get: () => 'attachment; filename="prompts_backup_1.db"' } });
await tick();
assert.deepEqual(toasts, [BACKUP_DONE]);
assert.equal(modal(dom).open, true);
assert.equal(action.textContent, 'Back up now');
assert.equal(action.getAttribute('aria-disabled'), null);
action.click();
answer({ ok: false, status: 500 });
await tick();
assert.deepEqual(toasts, [BACKUP_DONE, BACKUP_FAILED]);
assert.equal(modal(dom).open, true);
} finally {
URL.createObjectURL = created;
URL.revokeObjectURL = revoked;
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
test('the scan review is a thin openReviewDialog caller with its scope switch', async () => {
const { openScanReviewDialog, closeWorkspaceDialog } = await dialogsModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
const scope = createScope();
try {
const starts = [];
openScanReviewDialog({ scope, sources: 2, onBackup() {}, onStart: value => starts.push(value) });
assert.equal(byId(dom, 'modal-title').textContent, 'Review metadata scan');
assert.deepEqual(bodyShape(dom), ['dl.meta-grid', 'div.setting-row', 'p.small', 'div.notice', 'div.dialog-actions']);
const grid = byId(dom, 'modal-body').children[0];
assert.deepEqual(grid.children.map(entry => entry.textContent), ['Sources2 output folders', 'ScopeNew and unresolved files']);
const allFiles = byId(dom, 'scan-all-files');
allFiles.checked = true;
allFiles.dispatchEvent(new Event('change'));
assert.equal(grid.children[1].children[1].textContent, 'All files');
button(byId(dom, 'modal-body'), 'Start scan').click();
assert.deepEqual(starts, [{ scope: 'all' }]);
} finally {
scope.dispose();
closeWorkspaceDialog();
dom.restore();
}
});
File diff suppressed because it is too large Load Diff
+574
View File
@@ -0,0 +1,574 @@
"""Candidate-selection record sets for the local resolver parity test (PR L3 Task 9).
``local_resolver_cases`` covers one sampler's tracing. These cases reach the
rest of ``_resolve``: several sampler candidates and the identical-branch rule
(#270), stages that only reach previews, detailer stages and their upstream
samplers (multistage), and A1111 ``parameters`` with every footer shape. The
server's ``inspect_prompt_evidence`` is the oracle; nothing here states an
expected answer. All content is synthetic.
"""
import json
import random
try:
from tests.local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
from local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
def _encode(text):
return {"class_type": "CLIPTextEncode", "inputs": {"text": text, "clip": ["4", 1]}}
def _sampler(positive, negative, latent=None, name="KSampler"):
inputs = {"seed": 1, "model": ["4", 0], "positive": positive}
if negative is not None:
inputs["negative"] = negative
inputs["latent_image"] = latent or ["5", 0]
return {"class_type": name, "inputs": inputs}
def _decode(sampler_id):
return {"class_type": "VAEDecode", "inputs": {"samples": [sampler_id, 0]}}
def _save(image, name="SaveImage"):
return {"class_type": name, "inputs": {"images": image}}
def _detailer(image, positive, negative, name="FaceDetailer", **extra):
inputs = {"image": image, "model": ["4", 0], "positive": positive}
inputs["negative"] = negative
inputs.update(extra)
return {"class_type": name, "inputs": inputs}
def _base(**nodes):
graph = {
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"5": {"class_type": "EmptyLatentImage", "inputs": {"width": 64}},
"6": _encode("a fox"),
"7": _encode("blurry"),
"8": _encode("a cat"),
}
graph.update(nodes)
return graph
def _samplers():
"""Several unanchored samplers: identical, differing and mixed branches."""
same = _sampler(["6", 0], ["7", 0])
cases = {
"two-identical": _base(**{"3": same, "9": same}),
"two-differ": _base(**{"3": same, "9": _sampler(["8", 0], ["7", 0])}),
"two-negative-differs": _base(**{"3": same, "9": _sampler(["6", 0], ["8", 0])}),
"identical-without-negative": _base(
**{"3": _sampler(["6", 0], None), "9": _sampler(["6", 0], None)}
),
"one-unresolved": _base(**{"3": same, "9": _sampler(["missing", 0], ["7", 0])}),
"both-unresolved": _base(
**{"3": _sampler(["x", 0], ["7", 0]), "9": _sampler(["y", 0], ["7", 0])}
),
"three-identical-one-advanced": _base(
**{
"3": same,
"9": same,
"10": _sampler(["6", 0], ["7", 0], name="KSamplerAdvanced"),
}
),
# Equal texts, different recipes: a PromptManager pair against an encoder.
"same-text-other-recipe": _base(
**{
"3": same,
"9": _sampler(["11", 0], ["7", 0]),
"11": {
"class_type": "PromptManager",
"inputs": {
"text": "fox",
"prepend_text": "a",
"append_text": "",
"clip": ["4", 1],
},
},
}
),
"multistage-latent": _base(
**{"3": same, "9": _sampler(["6", 0], ["7", 0], latent=["3", 0])}
),
}
selector = {
"class_type": "easy ifElse",
"inputs": {"boolean": ["4", 0], "on_true": ["6", 0], "on_false": ["8", 0]},
}
cases["selector-and-plain"] = _base(
**{"3": same, "9": _sampler(["12", 0], ["7", 0]), "12": selector}
)
return cases
def _previews():
"""Stages whose outputs reach only previews are not the saved file's."""
same, other = _sampler(["6", 0], ["7", 0]), _sampler(["8", 0], ["7", 0])
return {
"preview-dropped": _base(
**{
"3": same,
"9": other,
"13": _decode("3"),
"14": _decode("9"),
"15": _save(["13", 0]),
"16": _save(["14", 0], "PreviewImage"),
}
),
"all-previews-kept": _base(
**{
"3": same,
"9": other,
"13": _decode("3"),
"14": _decode("9"),
"15": _save(["13", 0], "PreviewImage"),
"16": _save(["14", 0], "Image Comparer (rgthree)"),
}
),
"preview-and-file": _base(
**{
"3": same,
"9": other,
"13": _decode("3"),
"14": _decode("9"),
"15": _save(["13", 0]),
"16": _save(["14", 0], "PreviewImage"),
"17": _save(["14", 0]),
}
),
"unconsumed-stage-kept": _base(
**{
"3": same,
"9": other,
"14": _decode("9"),
"16": _save(["14", 0], "PreviewImage"),
}
),
# A list or object class_type is never a preview class (#286).
"preview-unhashable-class": _base(
**{
"3": same,
"9": other,
"14": _decode("9"),
"16": {"class_type": ["PreviewImage"], "inputs": {"images": ["14", 0]}},
"17": {"class_type": {"a": 1}, "inputs": {"images": ["3", 0]}},
}
),
"preview-through-bad-node": _base(
**{
"3": same,
"9": other,
"13": _decode("3"),
"15": _save(["13", 0]),
"16": _save(["9", 0], "PreviewImage"),
"18": "not a node",
"19": {"class_type": "Note", "inputs": {"x": ["18", 0]}},
}
),
}
def _detailers():
"""Detailer stages judged with the samplers in their image ancestry."""
same = _sampler(["6", 0], ["7", 0])
cases = {
"detailer-same-prompt": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-differs": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(["10", 0], ["8", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-and-own-output": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(["10", 0], ["8", 0], ["7", 0]),
"12": _save(["11", 0]),
"13": _save(["10", 0]),
}
),
"detailer-image-literal": _base(
**{
"11": _detailer("not a link", ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-cycle": _base(
**{
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"10": {"class_type": "ImageScale", "inputs": {"image": ["11", 0]}},
"12": _save(["11", 0]),
}
),
"detailer-chain": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"21": _detailer(
["11", 0], ["8", 0], ["7", 0], name="UltimateSDUpscale"
),
"12": _save(["21", 0]),
}
),
"detailer-chain-same": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"21": _detailer(["11", 0], ["6", 0], ["7", 0], name="DetailerForEach"),
"12": _save(["21", 0]),
}
),
"detailer-two-upstream": _base(
**{
"3": same,
"9": _sampler(["8", 0], ["7", 0]),
"10": {
"class_type": "ImageBlend",
"inputs": {"image1": ["13", 0], "image2": ["14", 0]},
},
"13": _decode("3"),
"14": _decode("9"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-unsupported-stage": _base(
**{
"10": {
"class_type": "MysterySampler",
"inputs": {"positive": ["6", 0], "model": ["4", 0]},
},
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-upstream-unresolved": _base(
**{
"3": _sampler(["gone", 0], ["7", 0]),
"10": _decode("3"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-upstream-selector": _base(
**{
"3": _sampler(["15", 0], ["7", 0]),
"15": {
"class_type": "easy ifElse",
"inputs": {
"boolean": ["4", 0],
"on_true": ["6", 0],
"on_false": ["8", 0],
},
},
"10": _decode("3"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
"detailer-wildcard": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(["10", 0], ["6", 0], ["7", 0], wildcard="[x|y]"),
"12": _save(["11", 0]),
}
),
"detailer-no-upscaled-image": _base(
**{
"3": same,
"10": _decode("3"),
"11": _detailer(
["10", 0], ["6", 0], ["7", 0], name="UltimateSDUpscaleNoUpscale"
),
"12": _save(["11", 0]),
}
),
"detailer-no-sampler-upstream": _base(
**{
"10": {"class_type": "LoadImage", "inputs": {"image": "a.png"}},
"11": _detailer(["10", 0], ["6", 0], ["7", 0]),
"12": _save(["11", 0]),
}
),
}
deep = {"3": _sampler(["6", 0], ["7", 0]), "d0": _decode("3")}
for index in range(1, 70):
deep[f"d{index}"] = {
"class_type": "ImageScale",
"inputs": {"image": [f"d{index - 1}", 0]},
}
deep["11"] = _detailer(["d69", 0], ["6", 0], ["7", 0])
deep["12"] = _save(["11", 0])
cases["detailer-deep-ancestry"] = _base(**deep)
# More nested detailer stages than the depth bound.
stacked = {"3": _sampler(["6", 0], ["7", 0]), "s0": _decode("3")}
previous = "s0"
for index in range(1, 67):
stacked[f"s{index}"] = _detailer([previous, 0], ["6", 0], ["7", 0])
previous = f"s{index}"
stacked["12"] = _save([previous, 0])
cases["detailer-stacked-past-depth"] = _base(**stacked)
# Each detailer walks a wide shared image ancestry: the total visit budget
# runs out across stages before any candidate is read.
wide = {
f"l{index}": {"class_type": "LoadImage", "inputs": {}} for index in range(1500)
}
wide["hub"] = {
"class_type": "ImageBatchMany",
"inputs": {f"i{index}": [f"l{index}", 0] for index in range(1500)},
}
for index in range(15):
wide[f"w{index}"] = _detailer(["hub", 0], ["6", 0], ["7", 0])
wide[f"o{index}"] = _save([f"w{index}", 0])
cases["detailers-exhaust-total-visits"] = _base(**wide)
cases["detailer-mutual-cycle"] = _base(
**{
"11": _detailer(["21", 0], ["6", 0], ["7", 0]),
"21": _detailer(["11", 0], ["6", 0], ["7", 0]),
"12": _save(["21", 0]),
}
)
# The sampler's output reaches a preview through more nodes than one
# detailer's allowance: the walk runs out, so the sampler is not absorbed.
long = {"3": _sampler(["6", 0], ["7", 0]), "10": _decode("3")}
previous = "10"
for index in range(2100):
long[f"c{index}"] = {
"class_type": "ImageScale",
"inputs": {"image": [previous, 0]},
}
previous = f"c{index}"
long["p"] = _save([previous, 0], "PreviewImage")
long["11"] = _detailer(["10", 0], ["6", 0], ["7", 0])
long["12"] = _save(["11", 0])
cases["detailer-output-walk-exhausted"] = _base(**long)
# Every detailer's ancestry walk has its own allowance: the first one is
# wide, and the second still absorbs the first.
chained = {"3": _sampler(["6", 0], ["7", 0]), "10": _decode("3")}
chained["h1"] = {
"class_type": "ImageBatchMany",
"inputs": {
"i": ["10", 0],
**{f"i{index}": [f"a{index}", 0] for index in range(1500)},
},
}
chained["h2"] = {
"class_type": "ImageBatchMany",
"inputs": {
"i": ["11", 0],
**{f"i{index}": [f"b{index}", 0] for index in range(600)},
},
}
for index in range(1500):
chained[f"a{index}"] = {"class_type": "LoadImage", "inputs": {}}
for index in range(600):
chained[f"b{index}"] = {"class_type": "LoadImage", "inputs": {}}
chained["11"] = _detailer(["h1", 0], ["6", 0], ["7", 0])
chained["21"] = _detailer(["h2", 0], ["6", 0], ["7", 0])
chained["12"] = _save(["21", 0])
cases["detailers-each-reset"] = _base(**chained)
return cases
_FOOTER = "Steps: 20, Sampler: euler, CFG scale: 7"
def _parameters():
"""A1111 ``parameters`` texts: every footer shape and delimiter rule."""
texts = {
"basic": f"a fox\nNegative prompt: blur\n{_FOOTER}",
"no-negative": f"a fox\n{_FOOTER}",
"two-delimiters": f"a\nNegative prompt: b\nNegative prompt: c\n{_FOOTER}",
"leading-delimiter": f"Negative prompt: blur\n{_FOOTER}",
"blank-positive": f" \x85\nNegative prompt: blur\n{_FOOTER}",
"footer-not-last": f"a fox\n{_FOOTER}\nmore text",
"two-footers": f"a\n{_FOOTER}\n{_FOOTER}",
"bare-footer": "a fox\nNegative prompt: blur\nSteps: 20",
"bare-and-full": f"a fox\nSteps: 5\n{_FOOTER}",
"two-bare": "a\nSteps: 1\nSteps: 2",
"template": f"a fox\n{_FOOTER}\nTemplate: {{a|b}}",
"both-templates": f"a\n{_FOOTER}\nTemplate: x\nNegative Template: y",
"templates-reversed": f"a\n{_FOOTER}\nNegative Template: y\nTemplate: x",
"two-templates": f"a\n{_FOOTER}\nTemplate: x\nTemplate: y",
"template-multiline": f"a\n{_FOOTER}\nTemplate: x\nmore",
"crlf": f"a fox\r\nNegative prompt: blur\r\n{_FOOTER}\r\n",
"cr-only": f"a fox\rNegative prompt: blur\r{_FOOTER}",
"cr-template": f"a\n{_FOOTER}\rTemplate: x",
"line-separator": f"a fox
{_FOOTER}",
"next-line": f"a fox\x85{_FOOTER}",
"spaced-steps": "a fox\nSteps:\n 20 \n, x",
"steps-newline-bare": "a fox\nSteps:\n20\n",
"arabic-digits": "a fox\nSteps: ٢٠, Sampler: euler",
"fullwidth-digits": "a fox\nSteps: 2, x",
"no-digits": "a fox\nSteps: , x",
"trailing-spaces": f"a fox \nNegative prompt: blur \n{_FOOTER} \n \n",
"empty-negative": f"a fox\nNegative prompt:\n{_FOOTER}",
"astral-within-limit": "\U0001f600" * 60_000 + f"\n{_FOOTER}",
"over-limit": "x" * 100_001,
"at-limit": "x" * (100_000 - len(_FOOTER) - 1) + "\n" + _FOOTER,
"empty": "",
"only-footer": _FOOTER,
"lone-surrogate": f"\ud800 fox\nNegative prompt: \udfff\n{_FOOTER}",
}
cases = {f"parameters-{name}": {"parameters": text} for name, text in texts.items()}
cases["parameters-with-execution"] = {
"parameters": texts["basic"],
EXECUTION_KEY: json.dumps({"version": 1, "nodes": {}}),
}
return cases
def selection_cases():
"""Named record sets for every Task 9 path, each a ``read_inspection_file`` dict."""
cases = {}
for group in (_samplers(), _previews(), _detailers()):
for name, graph in group.items():
cases[f"select-{name}"] = {"prompt": json.dumps(graph, ensure_ascii=False)}
cases.update(_parameters())
return cases
class _Selection(_Builder):
"""Random graphs with several samplers, outputs, previews and detailers."""
def sampler(self):
rng = self.rng
positive = self.conditioning(0)
negative = self.conditioning(0) if rng.random() < 0.85 else None
name = rng.choice(["KSampler", "KSampler", "KSamplerAdvanced", "SamplerCustom"])
node_id = self.new_id()
self.nodes[node_id] = _sampler(positive, negative, name=name)
return node_id
def shared(self, ids):
"""A copy of an earlier sampler: an identical branch."""
node_id = self.new_id()
self.nodes[node_id] = json.loads(json.dumps(self.nodes[self.rng.choice(ids)]))
return node_id
def output(self, image):
rng = self.rng
choice = rng.randrange(5)
if choice == 4:
return
name = ["SaveImage", "SaveImage", "PreviewImage", "PreviewAny"][choice]
self.nodes[self.new_id()] = _save(image, name)
def detailer(self, image):
rng = self.rng
node_id = self.new_id()
positive = self.conditioning(1) if rng.random() < 0.5 else None
name = rng.choice(list(_DETAILER_NAMES))
node = _detailer(image, positive, self.conditioning(1), name=name)
if positive is None:
node["inputs"]["positive"] = self.reuse_positive()
if name == "UltimateSDUpscaleNoUpscale":
node["inputs"]["upscaled_image"] = node["inputs"].pop("image")
if name in ("FaceDetailer", "DetailerForEach") and rng.random() < 0.15:
node["inputs"]["wildcard"] = rng.choice(["", "x", ["4", 0]])
self.nodes[node_id] = node
return node_id
def reuse_positive(self):
samplers = [
node
for node in self.nodes.values()
if node.get("class_type")
in ("KSampler", "KSamplerAdvanced", "SamplerCustom")
]
if samplers:
return self.rng.choice(samplers)["inputs"]["positive"]
return self.conditioning(1)
def build(self):
rng = self.rng
ids = [self.sampler()]
for _ in range(rng.randrange(3)):
ids.append(self.shared(ids) if rng.random() < 0.4 else self.sampler())
for sampler_id in list(ids):
decoded = self.new_id()
self.nodes[decoded] = _decode(sampler_id)
image = [decoded, 0]
for _ in range(rng.randrange(3)):
if rng.random() < 0.5:
image = [self.detailer(image), 0]
else:
self.output(image)
self.output(image)
if rng.random() < 0.1:
self.nodes[sampler_id]["inputs"]["latent_image"] = [rng.choice(ids), 0]
graph = {
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"5": {"class_type": "EmptyLatentImage", "inputs": {"width": 64}},
}
graph.update(self.nodes)
return graph
_DETAILER_NAMES = (
"FaceDetailer",
"DetailerForEach",
"UltimateSDUpscale",
"UltimateSDUpscaleNoUpscale",
)
_PARAMETER_LINES = [
"Negative prompt:",
"Negative prompt: blur",
"Steps: 20, Sampler: euler",
"Steps: 7",
"Steps: x",
"Template: t",
"Negative Template: n",
"Steps:٣, a",
]
def _parameter_text(rng):
parts = []
for _ in range(rng.randint(1, 5)):
if rng.random() < 0.5:
parts.append(rng.choice(_PARAMETER_LINES))
else:
parts.append(rng.choice(SPACES) + rng.choice(WORDS) + rng.choice(SPACES))
joints = ["\n", "\n", "\r\n", "\r", "
", " "]
return "".join(part + rng.choice(joints) for part in parts).rstrip(
rng.choice(["", "\n", " "])
)
def selection_fuzz_cases(seed, count):
"""Seeded several-sampler graphs (sometimes corrupted) and parameters texts."""
rng = random.Random(seed)
cases = {}
for index in range(count):
if rng.random() < 0.2:
cases[f"select-fuzz-{seed}-{index}"] = {"parameters": _parameter_text(rng)}
continue
builder = _Selection(rng)
graph = builder.build()
if rng.random() < 0.2:
builder.nodes = graph
builder.corrupt()
records = {"prompt": json.dumps(graph, ensure_ascii=rng.random() < 0.5)}
records.update(builder.records)
cases[f"select-fuzz-{seed}-{index}"] = records
return cases
+35
View File
@@ -0,0 +1,35 @@
"""A lock stand-in that reports when a second caller has to wait for it.
Concurrency tests patch it over a module's lock so they can wait until the
second caller is really blocked on the lock, instead of hoping a thread
start or one event-loop tick got it there.
"""
import threading
class ProbeLock:
"""A re-entrant lock whose ``contended`` event is set when a caller blocks."""
def __init__(self):
self._lock = threading.RLock()
self.contended = threading.Event()
def acquire(self, blocking=True, timeout=-1):
if self._lock.acquire(blocking=False):
return True
self.contended.set()
if not blocking:
return False
return self._lock.acquire(timeout=timeout)
def release(self):
self._lock.release()
def __enter__(self):
self.acquire()
return self
def __exit__(self, *exc):
self.release()
return False
+186 -9
View File
@@ -5,6 +5,7 @@ database and a stub ``folder_paths`` module, so the tests run without ComfyUI.
"""
import asyncio
import hashlib
import json
import os
import shutil
@@ -734,9 +735,9 @@ class TestSettingsReadMonitorThroughModule(AdminAPITestCase):
def test_no_sys_modules_scan(self):
import inspect
from py.api.admin import AdminRoutesMixin
from py.api.settings_routes import SettingsRoutesMixin
source = inspect.getsource(AdminRoutesMixin.get_settings)
source = inspect.getsource(SettingsRoutesMixin.get_settings)
self.assertNotIn("sys.modules", source)
async def test_reports_directories_of_running_monitor(self):
@@ -867,6 +868,23 @@ class TestDeleteDuplicatesEndpoint(AdminAPITestCase):
self.assertFalse(target.exists())
self.assertTrue(thumb_dir.exists())
async def test_deleted_files_leave_the_cached_gallery_listing(self):
# The listing is cached for 30 s; a delete must not leave the gallery
# listing (and counting) a file that is gone.
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
for name in ("keep.png", "gone.png"):
(self.output_dir / name).write_bytes(b"same bytes")
async def listed():
resp = await self.client.get("/prompt_manager/images/output")
data = await resp.json()
return data["total"], sorted(i["filename"] for i in data["images"])
self.assertEqual(await listed(), (2, ["gone.png", "keep.png"]))
resp = await self._post({"image_paths": ["gone.png"]})
self.assertEqual((await resp.json())["deleted_count"], 1)
self.assertEqual(await listed(), (1, ["keep.png"]))
def test_unexpected_error_per_file_is_reported_without_path(self):
self.api._delete_one_duplicate = lambda *a: (_ for _ in ()).throw(
PermissionError(13, "Permission denied", os.path.join(self.tmpdir, "x.png"))
@@ -1059,14 +1077,29 @@ class TestSettingsMisc(AdminAPITestCase):
saved["gallery"]["monitoring"]["directories"], [self._root(self.output_dir)]
)
async def test_save_replaces_a_corrupt_config_file(self):
with open(self.config_path, "w") as f:
f.write("{not json")
async def test_save_refuses_a_corrupt_config_file(self):
# #220: a config.json that cannot be parsed is the user's file, never
# silently replaced; the save is refused and nothing is published.
Path(self.config_path).write_bytes(b"{not json")
before_timeout = PromptManagerConfig.RESULT_TIMEOUT
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": 4}
)
self.assertEqual(resp.status, 200)
self.assertEqual(self._read_config()["web_ui"]["result_timeout"], 4)
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "config-unreadable"))
self.assertIs(body["success"], False)
self.assertIs(body["persisted"], False)
self.assertEqual(PromptManagerConfig.RESULT_TIMEOUT, before_timeout)
self.assertEqual(Path(self.config_path).read_bytes(), b"{not json")
async def test_get_settings_reports_an_unreadable_config(self):
Path(self.config_path).write_bytes(b"{not json")
resp = await self.client.request("GET", "/prompt_manager/settings")
body = await resp.json()
self.assertEqual(resp.status, 200) # the legacy page still loads
self.assertIsNone(body["revision"])
self.assertEqual(body["config_error"], "config-unreadable")
self.assertIn("result_timeout", body["settings"])
async def test_saved_config_file_is_private_to_the_user(self):
import stat
@@ -1107,12 +1140,156 @@ class TestSettingsMisc(AdminAPITestCase):
self.assertEqual(resp.status, 200)
self.assertFalse(data["restart_required"])
async def test_unwritable_config_path_still_succeeds(self):
os.environ[CONFIG_PATH_ENV] = os.path.join(self.tmpdir, "missing", "c.json")
async def test_unwritable_config_path_reports_failure_without_runtime_change(self):
# #220: the parent is a file, so config.json cannot be written. The
# save reports the failure and publishes nothing.
blocker = Path(self.tmpdir) / "not-a-dir"
blocker.write_text("x")
os.environ[CONFIG_PATH_ENV] = str(blocker / "c.json")
GalleryConfig.MONITORING_DIRECTORIES = []
before_timeout = PromptManagerConfig.RESULT_TIMEOUT
with self.assertLogs("prompt_manager.api", level="ERROR"):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={
"result_timeout": before_timeout + 1,
"gallery_root_paths": [self._root(self.output_dir)],
},
)
body = await resp.json()
self.assertEqual(
(resp.status, body["code"], body["persisted"]),
(500, "config-write-failed", False),
)
self.assertIs(body["success"], False)
self.assertIs(body["runtime_applied"], False)
self.assertEqual(PromptManagerConfig.RESULT_TIMEOUT, before_timeout)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
self.assertEqual(blocker.read_text(), "x")
async def test_save_reports_the_commit(self):
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": 3}
)
body = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(body["message"], "Settings saved successfully")
self.assertIs(body["persisted"], True)
self.assertIs(body["runtime_applied"], True)
self.assertIs(body["restart_required"], False)
self.assertEqual(body["restart_required_fields"], [])
digest = hashlib.sha256(Path(self.config_path).read_bytes()).hexdigest()
self.assertEqual(body["revision"], "sha256:" + digest)
async def test_changed_roots_name_the_restart_field(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [self._root(self.output_dir)]},
)
body = await resp.json()
self.assertEqual(resp.status, 200)
self.assertIs(body["restart_required"], True)
self.assertEqual(body["restart_required_fields"], ["gallery_root_paths"])
async def test_runtime_publication_failure_after_commit_is_reported(self):
# The values are durable; the response must not claim they are live.
self.api.reset_gallery_roots_cache = _raise
with self.assertLogs("prompt_manager.api", level="ERROR"):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [self._root(self.output_dir)]},
)
body = await resp.json()
self.assertEqual(resp.status, 200)
self.assertIs(body["persisted"], True)
self.assertIs(body["runtime_applied"], False)
self.assertIs(body["restart_required"], True)
self.assertEqual(body["restart_required_fields"], ["gallery_root_paths"])
self.assertEqual(
self._read_config()["gallery"]["monitoring"]["directories"],
[self._root(self.output_dir)],
)
async def test_get_settings_reports_revision(self):
resp = await self.client.request("GET", "/prompt_manager/settings")
body = await resp.json()
self.assertEqual(body["revision"], "absent")
self.assertNotIn("config_error", body)
self._write_config({"web_ui": {"result_timeout": 2}})
body = await (
await self.client.request("GET", "/prompt_manager/settings")
).json()
digest = hashlib.sha256(Path(self.config_path).read_bytes()).hexdigest()
self.assertEqual(body["revision"], "sha256:" + digest)
async def test_native_save_with_stale_revision_is_409(self):
self._write_config({"web_ui": {"result_timeout": 2}})
body = await (
await self.client.request("GET", "/prompt_manager/settings")
).json()
seen = body["revision"]
self._write_config({"web_ui": {"result_timeout": 6}}) # another writer
before = Path(self.config_path).read_bytes()
before_timeout = PromptManagerConfig.RESULT_TIMEOUT
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"result_timeout": 9, "expected_revision": seen},
)
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "settings-changed"))
self.assertIs(body["persisted"], False)
digest = hashlib.sha256(before).hexdigest()
self.assertEqual(body["revision"], "sha256:" + digest)
self.assertEqual(Path(self.config_path).read_bytes(), before)
self.assertEqual(PromptManagerConfig.RESULT_TIMEOUT, before_timeout)
async def test_native_save_with_current_revision_commits(self):
body = await (
await self.client.request("GET", "/prompt_manager/settings")
).json()
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"result_timeout": 9, "expected_revision": body["revision"]},
)
self.assertEqual(resp.status, 200)
self.assertEqual(self._read_config()["web_ui"]["result_timeout"], 9)
async def test_expected_revision_must_be_a_string(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"result_timeout": 9, "expected_revision": 5},
)
self.assertEqual(resp.status, 400)
self.assertFalse(os.path.exists(self.config_path))
async def test_legacy_partial_update_without_revision_still_merges(self):
self._write_config(
{
"integrations": {"lora_manager": {"civitai_api_key": "secret"}},
"web_ui": {"result_timeout": 1, "webui_display_mode": "popup"},
"custom_section": {"keep": True},
}
)
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": 8}
)
self.assertEqual(resp.status, 200)
self.assertNotIn("secret", await resp.text())
saved = self._read_config()
self.assertEqual(
saved["integrations"]["lora_manager"]["civitai_api_key"], "secret"
)
self.assertEqual(saved["custom_section"], {"keep": True})
self.assertEqual(saved["web_ui"]["result_timeout"], 8)
self.assertEqual(saved["web_ui"]["webui_display_mode"], "popup") # not sent
async def test_save_unexpected_error_is_500(self):
self.api._parse_gallery_roots = _raise
+83
View File
@@ -5,12 +5,16 @@ stream the upload to disk, reject anything that is not a healthy prompts
database and leave the live database untouched when it does.
"""
import asyncio
import glob
import hashlib
import os
import sqlite3
import sys
import tempfile
import threading
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
_mock_server = MagicMock()
@@ -142,6 +146,85 @@ class TestBackupRoute(BackupRestoreTestCase):
self.assertIn("backup", data["error"].lower())
class TestRestoreHoldsTheRegistry(BackupRestoreTestCase):
"""A legacy restore and a workspace job refuse each other (#269 gap)."""
def _digest(self):
return {
suffix: hashlib.sha256(Path(self.db_path + suffix).read_bytes()).hexdigest()
for suffix in ("", "-wal")
if os.path.exists(self.db_path + suffix)
}
async def test_legacy_restore_refused_while_scan_job_runs(self):
self._save("keep me")
upload = self._make_db_file(["restored"])
gate = asyncio.Event()
async def runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan", "Scan for metadata", record_changing=True, scope={}, runner=runner
)
before = self._digest()
try:
resp = await self._restore(upload)
self.assertEqual(resp.status, 409)
self.assertEqual(
await resp.json(),
{"success": False, "error": "Scan for metadata already running"},
)
self.assertEqual(self._digest(), before)
self.assertEqual(
[p["text"] for p in self.api.db.search_prompts()], ["keep me"]
)
self.assertEqual(glob.glob(self.db_path + ".backup_*"), [])
finally:
gate.set()
await asyncio.wait_for(scan.task, 5)
async def test_scan_refused_while_legacy_restore_holds(self):
upload = self._make_db_file(["restored"])
entered = threading.Event()
release = threading.Event()
model = self.api.db.model
real_verify = model.verify_database_file
def gated(path, **kwargs):
entered.set()
release.wait(10)
return real_verify(path, **kwargs)
loop = asyncio.get_running_loop()
with patch.object(model, "verify_database_file", gated):
restore = asyncio.ensure_future(self._restore(upload))
self.assertTrue(await loop.run_in_executor(None, entered.wait, 10))
resp = await self.client.post("/prompt_manager/jobs/scan")
body = await resp.json()
listed = (await (await self.client.get("/prompt_manager/jobs")).json())[
"jobs"
]
release.set()
restored = await restore
self.assertEqual((resp.status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["title"], "Database restore (admin page)")
self.assertEqual(
[(j["kind"], j["state"], j["cancellable"]) for j in listed],
[("restore", "running", False)],
)
self.assertEqual(restored.status, 200)
self.assertEqual(self.api._jobs.list()[0].state, "finished")
self.assertIsNone(self.api._jobs.active("restore"))
async def test_rejected_upload_ends_its_hold_failed(self):
resp = await self._restore(self._write("notes.txt", b"not a database"))
self.assertEqual(resp.status, 400)
job = self.api._jobs.list()[0]
self.assertEqual((job.kind, job.state), ("restore", "failed"))
class TestRestoreRoute(BackupRestoreTestCase):
async def test_restore_replaces_database_and_reports_backup(self):
self._save("old prompt")
+55
View File
@@ -462,6 +462,15 @@ class TestUiRoutes(CoreRoutesTestCase):
):
self.assertIn(entry, table)
async def test_image_tag_job_route_is_registered_beside_the_scan_job(self):
table = {
(route.method, route.resource.canonical)
for route in self.app.router.routes()
if route.resource is not None
}
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
class TestStaticRoutes(CoreRoutesTestCase):
@@ -477,6 +486,30 @@ class TestStaticRoutes(CoreRoutesTestCase):
self.assertEqual(resp.status, 200)
self.assertIn("application/javascript", resp.headers["Content-Type"])
async def test_inspection_semantics_table_is_the_one_json_served(self):
"""Local Inspect reads the node table the server resolves with (#249 PR L3).
Only that exact file is served; every other JSON stays forbidden.
"""
resp = await self.client.request(
"GET", "/prompt_manager/js/inspection-semantics.json"
)
self.assertEqual(resp.status, 200)
self.assertIn("application/json", resp.headers["Content-Type"])
table = Path(__file__).resolve().parents[1] / "web" / "js"
self.assertEqual(
await resp.read(), (table / "inspection-semantics.json").read_bytes()
)
for path in (
"js/shared/toolkit/provenance.json",
"lib/viewerjs/provenance.json",
"lib/inspection-semantics.json",
"js/workspace/..%2Finspection-semantics.json",
"js/INSPECTION-SEMANTICS.json",
):
resp = await self.client.request("GET", f"/prompt_manager/{path}")
self.assertEqual(resp.status, 403, path)
async def test_lib_served(self):
web_dir = self._fake_root()
(web_dir / "lib" / "x.css").write_text("body{}")
@@ -569,6 +602,28 @@ class TestStaticRoutes(CoreRoutesTestCase):
resp = await self.client.request("GET", "/prompt_manager/lib/link.js")
self.assertEqual(resp.status, 403)
async def test_inspection_semantics_symlink_escaping_web_dir_is_403(self):
"""The exact-path JSON gets the same realpath containment as any file."""
web_dir = self._fake_root()
secret = Path(self.tmpdir) / "secret.json"
secret.write_text('{"secret": true}')
try:
os.symlink(secret, web_dir / "js" / "inspection-semantics.json")
except (OSError, NotImplementedError):
self.skipTest("symlinks not available")
resp = await self.client.request(
"GET", "/prompt_manager/js/inspection-semantics.json"
)
self.assertEqual(resp.status, 403)
self.assertNotIn("secret", await resp.text())
async def test_inspection_semantics_missing_is_404(self):
self._fake_root()
resp = await self.client.request(
"GET", "/prompt_manager/js/inspection-semantics.json"
)
self.assertEqual(resp.status, 404)
async def test_empty_path_is_not_served(self):
for prefix in ("lib", "js"):
resp = await self.client.request("GET", f"/prompt_manager/{prefix}/")
+89
View File
@@ -5,11 +5,13 @@ Uses aiohttp's test client against a real PromptManagerAPI wired to a
temporary SQLite database and a temporary output directory.
"""
import asyncio
import json
import os
import pathlib
import sys
import tempfile
import threading
import unittest
from contextlib import ExitStack
from pathlib import Path
@@ -638,6 +640,93 @@ class TestGenerateThumbnailsPost(ThumbnailTestCase):
self.assertEqual(data["total_images"], 0)
class TestThumbnailRoutesHoldTheRegistry(ThumbnailTestCase):
"""Thumbnail work is ``guarded``: never beside a record-changing job."""
def _run_scan(self):
gate = asyncio.Event()
async def runner(job):
await gate.wait()
job = self.api._jobs.start(
"scan", "Scan for metadata", admission="records", scope={}, runner=runner
)
async def release():
gate.set()
await asyncio.wait_for(job.task, 5)
self.addAsyncCleanup(release)
return job
async def test_thumbnail_generate_refused_while_scan_runs(self):
make_png(self.output_dir / "a.png")
self._run_scan()
resp = await self.client.request(
"POST", "/prompt_manager/images/generate-thumbnails", json={}
)
self.assertEqual(resp.status, 409)
self.assertEqual(
await resp.json(),
{"success": False, "error": "Scan for metadata already running"},
)
resp = await self.client.request("POST", self.PROGRESS_URL)
text = await resp.text()
self.assertEqual(resp.status, 200) # the gallery reads an SSE stream
self.assertEqual(
text,
"event: error\ndata: "
+ json.dumps(
{
"error": "Scan for metadata already running",
"message": "Scan for metadata already running",
}
)
+ "\n\n",
)
thumbs = self.output_dir / "thumbnails"
make_png(thumbs / "b_thumb.png")
resp = await self.client.request(
"POST", "/prompt_manager/images/clear-thumbnails"
)
self.assertEqual(resp.status, 409)
self.assertFalse(self._thumb_path("a").exists())
self.assertTrue((thumbs / "b_thumb.png").exists())
async def test_scan_refused_while_thumbnails_generate(self):
make_png(self.output_dir / "a.png")
entered = threading.Event()
release = threading.Event()
real = self.api._generate_gallery_thumbnails_sync
def gated(*args):
entered.set()
release.wait(10)
return real(*args)
self.api._generate_gallery_thumbnails_sync = gated
loop = asyncio.get_running_loop()
generate = asyncio.ensure_future(
self.client.request(
"POST", "/prompt_manager/images/generate-thumbnails", json={}
)
)
self.assertTrue(await loop.run_in_executor(None, entered.wait, 10))
resp = await self.client.request("POST", "/prompt_manager/jobs/scan")
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "busy"))
self.assertEqual(
body["running"]["title"], "Thumbnail generation (gallery page)"
)
release.set()
self.assertEqual((await generate).status, 200)
self.assertTrue(self._thumb_path("a").exists())
class TestRecentImagesCaps(ImageAPITestCase):
"""Every path through /images/recent hands the DB a bounded window."""
+340 -12
View File
@@ -4,11 +4,13 @@ Filesystem detection and network downloads are patched out so the tests run
anywhere without a LoraManager install or a CivitAI connection.
"""
import asyncio
import json
import os
import shutil
import sys
import tempfile
import threading
import types
import unittest
from pathlib import Path
@@ -34,11 +36,20 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from py import lora_utils
from py import config_store, lora_utils
from py.config import IntegrationConfig, PromptManagerConfig
from py.api import settings_routes as settings_module
from py.api import lora_integration as lora_module
from py.lora_utils import get_trigger_cache
from utils.hashing import generate_prompt_hash
try:
from tests.lock_probe import ProbeLock
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from lock_probe import ProbeLock
from open_handles import assert_closed
SECRET_KEY = "sk-civitai-super-secret-0123456789"
@@ -56,7 +67,13 @@ class LoraAPITestCase(AioHTTPTestCase):
return_value=[self.custom_nodes],
)
self._custom_nodes_patch.start()
self._real_save = PromptManagerConfig.save_to_file
# Never the repository's config.json (live user data): every save in
# these tests goes to a file in this test's temp directory.
self.config_file = self.tmp_root / "config.json"
self._env_patch = patch.dict(
os.environ, {"PROMPT_MANAGER_CONFIG_PATH": str(self.config_file)}
)
self._env_patch.start()
self._config_patch = patch.multiple(
IntegrationConfig,
@@ -70,8 +87,6 @@ class LoraAPITestCase(AioHTTPTestCase):
"py.lora_utils.detect_lora_manager", return_value=None
)
self.detect_mock = self._detect_patch.start()
self._save_patch = patch.object(PromptManagerConfig, "save_to_file")
self.save_mock = self._save_patch.start()
get_trigger_cache().clear()
app = web.Application()
@@ -83,7 +98,7 @@ class LoraAPITestCase(AioHTTPTestCase):
async def tearDownAsync(self):
await super().tearDownAsync() # closes the aiohttp test client
self._save_patch.stop()
self._env_patch.stop()
self._detect_patch.stop()
self._config_patch.stop()
self._custom_nodes_patch.stop()
@@ -96,8 +111,19 @@ class LoraAPITestCase(AioHTTPTestCase):
):
if os.path.exists(path):
os.unlink(path)
assert_closed(self, self.tmp_root)
shutil.rmtree(self.tmp_root, ignore_errors=True)
def _saved(self):
return json.loads(self.config_file.read_text())
def _make_unwritable(self):
"""Point the config path under a regular file, so no write can succeed."""
blocker = self.tmp_root / "not-a-dir"
blocker.write_text("x")
self.config_file = blocker / "config.json"
os.environ["PROMPT_MANAGER_CONFIG_PATH"] = str(self.config_file)
async def _post_json(self, url, body):
return await self.client.request(
"POST",
@@ -250,7 +276,7 @@ class TestLoraEnableRoute(LoraAPITestCase):
)
self.assertEqual(resp.status, 400)
self.assertFalse(IntegrationConfig.LORA_MANAGER_ENABLED)
self.save_mock.assert_not_called()
self.assertFalse(self.config_file.exists())
async def test_enable_persists_and_loads_trigger_cache(self):
lm = _make_lora_manager(self.custom_nodes)
@@ -285,7 +311,15 @@ class TestLoraEnableRoute(LoraAPITestCase):
os.path.normcase(os.path.realpath(str(lm))),
)
self.assertEqual(IntegrationConfig.CIVITAI_API_KEY, SECRET_KEY)
self.save_mock.assert_called_once_with()
saved = self._saved()["integrations"]["lora_manager"]
self.assertEqual(saved["civitai_api_key"], SECRET_KEY)
self.assertEqual(saved["path"], IntegrationConfig.LORA_MANAGER_PATH)
self.assertIs(saved["enabled"], True)
self.assertIs(saved["trigger_words_enabled"], True)
self.assertEqual(
(data["persisted"], data["runtime_applied"], data["cache_loaded"]),
(True, True, True),
)
self.assertTrue(get_trigger_cache().is_loaded)
self.assertEqual(get_trigger_cache().get_trigger_words("neon"), ["glow"])
@@ -319,7 +353,7 @@ class TestLoraEnableRoute(LoraAPITestCase):
self.assertFalse(json.loads(body)["success"])
self.assertNotIn(str(self.tmp_root), body)
self.assertFalse(IntegrationConfig.LORA_MANAGER_ENABLED)
self.save_mock.assert_not_called()
self.assertFalse(self.config_file.exists())
async def test_enable_with_custom_node_lacking_lora_in_its_name_is_400(self):
lm = _make_lora_manager(self.custom_nodes).parent / "ComfyUI-Other"
@@ -344,7 +378,6 @@ class TestLoraEnableRoute(LoraAPITestCase):
async def test_enable_writes_the_configured_config_file(self):
config_file = self.tmp_root / "cfg" / "config.json"
self.save_mock.side_effect = self._real_save
with patch.dict(os.environ, {"PROMPT_MANAGER_CONFIG_PATH": str(config_file)}):
resp = await self._post_json(
@@ -353,7 +386,6 @@ class TestLoraEnableRoute(LoraAPITestCase):
)
self.assertEqual(resp.status, 200)
self.save_mock.assert_called_once_with()
self.assertTrue(config_file.is_file())
saved = json.loads(config_file.read_text())["integrations"]["lora_manager"]
self.assertFalse(saved["enabled"])
@@ -414,9 +446,299 @@ class TestLoraEnableRoute(LoraAPITestCase):
self.assertEqual(resp.status, 400)
async def test_persist_failure_is_500(self):
self.save_mock.side_effect = OSError("read-only")
resp = await self._post_json("/prompt_manager/lora/enable", {"enabled": False})
self._make_unwritable()
with self.assertLogs("prompt_manager.api", level="ERROR"):
resp = await self._post_json(
"/prompt_manager/lora/enable", {"enabled": False}
)
self.assertEqual(resp.status, 500)
self.assertEqual((await resp.json())["code"], "config-write-failed")
async def test_enable_write_failure_reports_and_keeps_runtime(self):
# #220: a save that cannot be written publishes nothing and says so.
lm = _make_lora_manager(self.custom_nodes)
self.detect_mock.return_value = str(lm)
IntegrationConfig.CIVITAI_API_KEY = SECRET_KEY
self._make_unwritable()
with self.assertLogs("prompt_manager.api", level="ERROR"):
resp = await self._post_json(
"/prompt_manager/lora/enable",
{
"enabled": True,
"path": str(lm),
"trigger_words_enabled": True,
"clear_civitai_api_key": True,
},
)
text = await resp.text()
body = json.loads(text)
self.assertEqual(resp.status, 500)
self.assertEqual(body["code"], "config-write-failed")
self.assertEqual((body["persisted"], body["runtime_applied"]), (False, False))
self.assertFalse(IntegrationConfig.LORA_MANAGER_ENABLED)
self.assertEqual(IntegrationConfig.LORA_MANAGER_PATH, "")
self.assertFalse(IntegrationConfig.LORA_TRIGGER_WORDS_ENABLED)
self.assertEqual(IntegrationConfig.CIVITAI_API_KEY, SECRET_KEY)
self.assertFalse(get_trigger_cache().is_loaded)
self.assertNotIn(SECRET_KEY, text)
self.assertNotIn(str(self.tmp_root), text)
async def test_enable_preserves_unknown_sections(self):
stored = "stored-value" # a plain marker: what is on disk must stay
self.config_file.write_text(
json.dumps(
{
"custom_section": {"keep": True},
"web_ui": {"result_timeout": 11},
"integrations": {
"lora_manager": {"civitai_api_key": stored, "extra": 1},
"other_tool": {"on": True},
},
}
)
)
IntegrationConfig.CIVITAI_API_KEY = stored
resp = await self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "trigger_words_enabled": True},
)
self.assertEqual(resp.status, 200)
saved = self._saved()
self.assertEqual(saved["custom_section"], {"keep": True})
self.assertEqual(saved["web_ui"], {"result_timeout": 11})
self.assertEqual(saved["integrations"]["other_tool"], {"on": True})
lora = saved["integrations"]["lora_manager"]
self.assertEqual(lora["extra"], 1)
self.assertEqual(lora["civitai_api_key"], stored)
self.assertIs(lora["trigger_words_enabled"], True)
self.assertNotIn("database", saved) # no runtime defaults written
async def test_enable_refuses_a_corrupt_config_file(self):
self.config_file.write_bytes(b"{not json")
resp = await self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "trigger_words_enabled": True},
)
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "config-unreadable"))
self.assertEqual(self.config_file.read_bytes(), b"{not json")
self.assertFalse(IntegrationConfig.LORA_TRIGGER_WORDS_ENABLED)
async def test_enable_requires_boolean_flags_and_body_cap(self):
for body in (
{"enabled": "yes"},
{"enabled": False, "trigger_words_enabled": 1},
{"enabled": False, "clear_civitai_api_key": "true"},
):
with self.subTest(body=body):
resp = await self._post_json("/prompt_manager/lora/enable", body)
self.assertEqual(resp.status, 400)
resp = await self._post_json("/prompt_manager/lora/enable", [True])
self.assertEqual(resp.status, 400)
resp = await self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "civitai_api_key": "k" * (2 * 1024 * 1024)},
)
self.assertEqual(resp.status, 413)
self.assertFalse(self.config_file.exists())
self.assertFalse(IntegrationConfig.LORA_MANAGER_ENABLED)
async def test_blank_key_unchanged_and_explicit_clear(self):
self.config_file.write_text(
json.dumps({"integrations": {"lora_manager": {"civitai_api_key": "old"}}})
)
IntegrationConfig.CIVITAI_API_KEY = "old"
resp = await self._post_json(
"/prompt_manager/lora/enable", {"enabled": False, "civitai_api_key": " "}
)
self.assertEqual(resp.status, 200)
self.assertEqual(
self._saved()["integrations"]["lora_manager"]["civitai_api_key"], "old"
)
self.assertEqual(IntegrationConfig.CIVITAI_API_KEY, "old")
resp = await self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "clear_civitai_api_key": True},
)
self.assertEqual(resp.status, 200)
self.assertEqual(
self._saved()["integrations"]["lora_manager"]["civitai_api_key"], ""
)
self.assertEqual(IntegrationConfig.CIVITAI_API_KEY, "")
async def test_cache_load_failure_reports_persisted_but_unloaded(self):
lm = _make_lora_manager(self.custom_nodes)
self.detect_mock.return_value = str(lm)
with (
patch.object(
type(get_trigger_cache()), "load", side_effect=OSError("unreadable")
),
self.assertLogs("prompt_manager.api", level="WARNING"),
):
resp = await self._post_json(
"/prompt_manager/lora/enable",
{"enabled": True, "path": str(lm), "trigger_words_enabled": True},
)
text = await resp.text()
body = json.loads(text)
self.assertEqual(resp.status, 200)
self.assertIs(body["success"], True)
self.assertEqual((body["persisted"], body["cache_loaded"]), (True, False))
self.assertTrue(IntegrationConfig.LORA_MANAGER_ENABLED)
self.assertIs(self._saved()["integrations"]["lora_manager"]["enabled"], True)
self.assertNotIn("unreadable", text)
class TestConfigWritersShareOneStore(LoraAPITestCase):
async def asyncSetUp(self):
await super().asyncSetUp()
self._orig_timeout = PromptManagerConfig.RESULT_TIMEOUT
async def asyncTearDown(self):
PromptManagerConfig.RESULT_TIMEOUT = self._orig_timeout
await super().asyncTearDown()
async def test_settings_and_lora_saves_do_not_overwrite_each_other(self):
# #220: the LoRA save is sent while the settings save is inside its
# commit (lock held, file already read). The LoRA commit re-reads the
# file under the same lock, so both edits survive and nothing else
# is lost (before, the LoRA save rewrote the whole runtime config).
self.config_file.write_text(json.dumps({"custom_section": {"keep": True}}))
inside = threading.Event()
release = threading.Event()
real_mutation = settings_module._settings_mutation
def gated_mutation(updates, new_roots):
mutate = real_mutation(updates, new_roots)
def gated(config):
inside.set()
release.wait(10)
return mutate(config)
return gated
lora_calls = []
real_lora_mutation = lora_module._lora_settings_mutation
def recorded_lora_mutation(values):
mutate = real_lora_mutation(values)
def recorded(config):
lora_calls.append(dict(config))
return mutate(config)
return recorded
probe = ProbeLock()
loop = asyncio.get_running_loop()
with (
patch.object(settings_module, "_settings_mutation", gated_mutation),
patch.object(
lora_module, "_lora_settings_mutation", recorded_lora_mutation
),
patch.object(config_store, "_COMMIT_LOCK", probe),
):
settings = asyncio.ensure_future(
self._post_json("/prompt_manager/settings", {"result_timeout": 13})
)
self.assertTrue(await loop.run_in_executor(None, inside.wait, 10))
lora = asyncio.ensure_future(
self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "trigger_words_enabled": True},
)
)
# Wait until the LoRA commit is blocked on the lock the settings
# commit holds; its mutation must not have run yet.
self.assertTrue(await loop.run_in_executor(None, probe.contended.wait, 10))
self.assertEqual(lora_calls, [])
release.set()
settings_resp, lora_resp = await asyncio.gather(settings, lora)
self.assertEqual(len(lora_calls), 1)
self.assertEqual(lora_calls[0]["web_ui"]["result_timeout"], 13)
self.assertEqual((settings_resp.status, lora_resp.status), (200, 200))
saved = self._saved()
self.assertEqual(saved["custom_section"], {"keep": True})
self.assertEqual(saved["web_ui"]["result_timeout"], 13)
self.assertIs(
saved["integrations"]["lora_manager"]["trigger_words_enabled"], True
)
settings_body = await settings_resp.json()
lora_body = await lora_resp.json()
self.assertNotEqual(settings_body["revision"], lora_body["revision"])
def _delay_first_commit_result(self):
"""Hold back the first config commit's result from its handler.
The commit itself has run (the file is written); only the handler's
continuation waits until the returned event is set. Before #220's fix
round the handler published after this point, so a later save could
publish first and the earlier one overwrite it at runtime.
"""
original = self.api._run_in_executor
written = asyncio.Event()
resume = asyncio.Event()
delayed = []
async def run(func, *args, **kwargs):
result = await original(func, *args, **kwargs)
if getattr(func, "__func__", None) is config_store.ConfigStore.commit:
if not delayed:
delayed.append(func)
written.set()
await resume.wait()
return result
patcher = patch.object(self.api, "_run_in_executor", run)
patcher.start()
self.addCleanup(patcher.stop)
return written, resume
async def test_concurrent_settings_saves_end_on_the_last_written(self):
written, resume = self._delay_first_commit_result()
first = asyncio.ensure_future(
self._post_json("/prompt_manager/settings", {"result_timeout": 11})
)
await asyncio.wait_for(written.wait(), 10)
second = await self._post_json(
"/prompt_manager/settings", {"result_timeout": 22}
)
resume.set()
first = await first
self.assertEqual((first.status, second.status), (200, 200))
self.assertEqual(self._saved()["web_ui"]["result_timeout"], 22)
self.assertEqual(PromptManagerConfig.RESULT_TIMEOUT, 22)
async def test_concurrent_lora_saves_end_on_the_last_written(self):
written, resume = self._delay_first_commit_result()
first = asyncio.ensure_future(
self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "trigger_words_enabled": True},
)
)
await asyncio.wait_for(written.wait(), 10)
second = await self._post_json(
"/prompt_manager/lora/enable",
{"enabled": False, "trigger_words_enabled": False},
)
resume.set()
first = await first
self.assertEqual((first.status, second.status), (200, 200))
lora = self._saved()["integrations"]["lora_manager"]
self.assertIs(lora["trigger_words_enabled"], False)
self.assertIs(IntegrationConfig.LORA_TRIGGER_WORDS_ENABLED, False)
class TestLoraScanRoute(LoraAPITestCase):
@@ -445,6 +767,12 @@ class TestLoraScanRoute(LoraAPITestCase):
body = await resp.text()
self.assertNotIn(str(self.tmp_root), body)
self.assertIn("x.db", body)
# The admin page's modal shows the failure as its status and closes.
error = self._sse_events(body)[-1]
self.assertEqual(
(error["type"], error["status"], error["progress"]),
("error", error["message"], 0),
)
@staticmethod
def _sse_events(text):
+56
View File
@@ -1643,6 +1643,62 @@ class TestPromptInspector(ImageAPITestCase):
str(path), False, report_ambiguity=True
)
async def test_unhashable_workflow_node_type_is_skipped(self):
"""A UI workflow node whose ``type`` is a list or dict is skipped (#286).
The R1 check looks each node's ``type`` up among subgraph IDs; an
unhashable ``type`` used to raise there and the route answered 500.
Such a node can be neither a subgraph instance nor a PromptManager
node, so the walk skips it and a prompt that resolves still resolves.
"""
source = graph()
source["1"]["class_type"] = "PromptManager"
for kind in (["PromptManager"], {"name": "PromptManager"}):
with self.subTest(kind=type(kind).__name__):
workflow = json.dumps(
{"nodes": [{"id": 1, "type": kind, "widgets_values": ["x"]}]}
)
png(
self.output_dir / "image.png",
prompt=json.dumps(source),
workflow=workflow,
)
response, result = await self.inspect()
self.assertEqual(response.status, 200)
self.assertEqual(result["resolution"]["state"], "resolved")
self.assertEqual(result["resolution"]["sampler_id"], "3")
self.assertEqual(result["prompts"]["positive"]["status"], "resolved")
self.assertEqual(result["prompts"]["positive"]["text"], "portrait")
self.assertEqual(result["prompts"]["negative"]["text"], "blurry")
self.assertNotIn("unhashable", json.dumps(result))
async def test_unhashable_graph_class_type_is_not_a_preview_output(self):
"""A list or object ``class_type`` never names a preview output (#286).
With several samplers the inspector first drops stages that reach only
previews; looking such a ``class_type`` up among the preview classes
used to raise, and the route answered 500.
"""
for kind in (["PreviewImage"], {"name": "PreviewImage"}):
with self.subTest(kind=type(kind).__name__):
source = graph()
source["9"] = {
"class_type": "KSampler",
"inputs": {"positive": ["8", 0], "negative": ["2", 0]},
}
source["8"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": "cat"},
}
source["10"] = {"class_type": kind, "inputs": {"images": ["9", 0]}}
png(self.output_dir / "image.png", prompt=json.dumps(source))
response, result = await self.inspect()
self.assertEqual(response.status, 200)
self.assertEqual(
result["resolution"]["reason"], "Multiple sampler branches"
)
self.assertEqual(result["resolution"]["state"], "ambiguous")
class TestInspectorImageTags(ImageAPITestCase):
"""B3: the inspector envelope's ``image_tags`` facet (spec amendment S1)."""
+234
View File
@@ -0,0 +1,234 @@
"""A slow page-workflow reset reports where the server waited.
CI run 39904288262 had one ``/__test__/reset`` wait 16 s with nothing in the
server's stderr. tests/browser/slow_reset_watch.py reports any reset past
SLOW_RESET_SECONDS with every thread's stack, taken while it still waits.
These tests hold a real SQLite write lock behind a real test server's reset.
"""
import glob
import io
import os
import sqlite3
import subprocess
import sys
import tempfile
import threading
import time
import unittest
import urllib.request
from pathlib import Path
from tests.browser import slow_reset_watch
from tests.browser.slow_reset_watch import REPORT_PREFIX, ResetWatch, is_reset_path
REPO_ROOT = Path(__file__).resolve().parent.parent
SERVER_TIMEOUT_S = 60
HOLD_SECONDS = slow_reset_watch.SLOW_RESET_SECONDS + 4
# The reset may take a little longer than the hold, never the 30 s busy timeout.
RESET_MARGIN_SECONDS = 10
# While the reset waits, the server still answers other requests this fast.
# Reading the waiting connection from the report's thread froze it (the GIL is
# held while SQLite's connection mutex is awaited) until the lock was released.
RESPONSIVE_SECONDS = 2.0
PROBE_INTERVAL_SECONDS = 0.1
def blocked_here(gate: threading.Event) -> None:
gate.wait()
class TestResetWatch(unittest.TestCase):
def test_reset_paths_are_the_test_reset_routes(self):
self.assertTrue(is_reset_path("/__test__/reset"))
self.assertTrue(is_reset_path("/__test__/tagging/reset"))
self.assertFalse(is_reset_path("/prompt_manager/reset"))
self.assertFalse(is_reset_path("/__test__/reset/extra"))
def test_a_stalled_reset_reports_state_and_stacks_while_it_waits(self):
stream = io.StringIO()
reported = threading.Event()
class Stream(io.StringIO):
def flush(self):
reported.set()
stream = Stream()
gate = threading.Event()
worker = threading.Thread(target=blocked_here, args=(gate,), name="held-up")
worker.start()
self.addCleanup(worker.join)
self.addCleanup(gate.set)
watch = ResetWatch(
"POST /__test__/reset",
lambda: ["database: 1 connections"],
threshold=0.05,
stream=stream,
).start()
self.assertTrue(reported.wait(SERVER_TIMEOUT_S))
report = stream.getvalue().splitlines()
self.assertTrue(all(line.startswith(REPORT_PREFIX) for line in report))
text = "\n".join(report)
self.assertIn("POST /__test__/reset still running after", text)
self.assertIn("database: 1 connections", text)
self.assertIn(f"thread 0x{worker.ident:016x} is held-up", text)
self.assertIn("in blocked_here", text)
watch.record_call("seed", 0.25, 1.5)
watch.finish("answered 200")
self.assertIn("POST /__test__/reset answered 200 after", stream.getvalue())
self.assertIn(
"executor call seed: queued 0.250s, ran 1.500s", stream.getvalue()
)
def test_a_fast_reset_writes_nothing_and_stops_its_timer(self):
stream = io.StringIO()
watch = ResetWatch("reset", lambda: [], threshold=60, stream=stream).start()
watch.finish("answered 200")
self.assertTrue(watch._timer.finished.is_set())
self.assertEqual(stream.getvalue(), "")
def test_a_failing_state_probe_still_reports_the_stacks(self):
def broken():
raise RuntimeError("no state")
reported = threading.Event()
class Stream(io.StringIO):
def flush(self):
reported.set()
stream = Stream()
ResetWatch("reset", broken, threshold=0.01, stream=stream).start()
self.assertTrue(reported.wait(SERVER_TIMEOUT_S))
self.assertIn("state unavailable: RuntimeError('no state')", stream.getvalue())
self.assertIn("most recent call first", stream.getvalue())
class TestSlowResetOnTheServer(unittest.TestCase):
"""A reset behind another process's write lock, on tests/browser/server.py."""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
env = {**os.environ, "TMPDIR": self.tmp.name}
env.update(TEMP=self.tmp.name, TMP=self.tmp.name)
self.stderr = tempfile.TemporaryFile()
self.addCleanup(self.stderr.close)
process = subprocess.Popen(
[sys.executable, "-m", "tests.browser.server", "--port", "0"],
cwd=REPO_ROOT,
env=env,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=self.stderr,
)
def stop():
if process.poll() is None:
process.kill()
process.wait(timeout=SERVER_TIMEOUT_S)
for stream in (process.stdin, process.stdout):
stream.close()
self.addCleanup(stop)
line = process.stdout.readline().decode()
self.assertTrue(line.startswith("PM_TEST_SERVER_PORT="), line)
self.base = f"http://127.0.0.1:{int(line.split('=', 1)[1])}"
deadline = time.monotonic() + SERVER_TIMEOUT_S
while True:
try:
with urllib.request.urlopen( # nosec B310
self.base + "/prompt_manager/test", timeout=1
):
break
except OSError:
self.assertIsNone(process.poll(), "the server exited early")
self.assertLess(time.monotonic(), deadline, "never ready")
time.sleep(0.1)
[self.db_path] = glob.glob(
os.path.join(self.tmp.name, "pm-browser-*", "browser.db")
)
def reset(self) -> tuple[int, float]:
request = urllib.request.Request(
self.base + "/__test__/reset",
data=b"{}",
headers={"Content-Type": "application/json"},
method="POST",
)
start = time.monotonic()
with urllib.request.urlopen( # nosec B310
request, timeout=SERVER_TIMEOUT_S
) as response:
return response.status, time.monotonic() - start
def server_stderr(self) -> str:
self.stderr.seek(0)
return self.stderr.read().decode("utf-8", "replace")
def test_a_reset_held_behind_a_write_lock_names_the_wait_and_ends_with_it(self):
self.assertEqual(self.reset()[0], 200, self.server_stderr())
self.assertNotIn(REPORT_PREFIX, self.server_stderr())
held = threading.Event()
def hold_write_lock():
holder = sqlite3.connect(self.db_path, timeout=SERVER_TIMEOUT_S)
try:
holder.execute("BEGIN IMMEDIATE")
held.set()
time.sleep(HOLD_SECONDS)
holder.rollback()
finally:
holder.close()
lock = threading.Thread(target=hold_write_lock)
lock.start()
self.addCleanup(lock.join)
self.assertTrue(held.wait(SERVER_TIMEOUT_S))
outcome = []
resetting = threading.Thread(target=lambda: outcome.append(self.reset()))
resetting.start()
self.addCleanup(resetting.join)
slowest = 0.0
while resetting.is_alive():
start = time.monotonic()
with urllib.request.urlopen( # nosec B310
self.base + "/prompt_manager/test", timeout=SERVER_TIMEOUT_S
):
slowest = max(slowest, time.monotonic() - start)
time.sleep(PROBE_INTERVAL_SECONDS)
resetting.join()
lock.join()
self.assertEqual(len(outcome), 1, self.server_stderr())
[(status, elapsed)] = outcome
self.assertLess(slowest, RESPONSIVE_SECONDS)
self.assertEqual(status, 200)
self.assertGreaterEqual(elapsed, HOLD_SECONDS - 0.5)
self.assertLess(elapsed, HOLD_SECONDS + RESET_MARGIN_SECONDS)
report = [
line[len(REPORT_PREFIX) :]
for line in self.server_stderr().splitlines()
if line.startswith(REPORT_PREFIX)
]
text = "\n".join(report)
self.assertIn("POST /__test__/reset still running after", text)
self.assertIn("default executor:", text)
if sys.platform.startswith("linux"):
# SQLite's busy handler sleeps between its retries.
self.assertRegex(
text, r"kernel: asyncio_\d+ \(0x[0-9a-f]{16}\) state \w, waiting in "
)
# The plain reset's seed() waits on its first DELETE.
self.assertRegex(text, r"line \d+ in seed\n")
self.assertIn("POST /__test__/reset answered 200 after", text)
self.assertRegex(
text,
r"executor call create_app.<locals>.seed: queued \d+\.\d+s, ran \d+\.\d+s",
)
if __name__ == "__main__":
unittest.main()
+419
View File
@@ -0,0 +1,419 @@
"""Tests for the serialized config store (py/config_store.py, #220).
Every test works on a config.json inside its own temporary directory; the
repository's config.json (live user data) is never read or written.
"""
import hashlib
import json
import os
import shutil
import sys
import tempfile
import threading
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
_mock_server = MagicMock()
_mock_server.PromptServer.instance.routes = MagicMock()
sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py import config_store # noqa: E402
from py.config_store import ( # noqa: E402
ABSENT_REVISION,
ConfigStore,
ConfigUnreadable,
RevisionConflict,
get_config_store,
revision_of,
)
try:
from tests.lock_probe import ProbeLock # noqa: E402
from tests.open_handles import assert_closed # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from lock_probe import ProbeLock # noqa: E402
from open_handles import assert_closed # noqa: E402
class ConfigStoreTests(unittest.TestCase):
def setUp(self):
self.tmpdir = os.path.realpath(tempfile.mkdtemp())
self.path = os.path.join(self.tmpdir, "config.json")
self.store = ConfigStore(lambda: self.path)
def tearDown(self):
try:
assert_closed(self, self.tmpdir)
finally:
shutil.rmtree(self.tmpdir, ignore_errors=True)
def _write(self, data):
Path(self.path).write_text(json.dumps(data), encoding="utf-8")
def _bytes(self):
return Path(self.path).read_bytes()
def _leftovers(self):
return sorted(name for name in os.listdir(self.tmpdir) if name != "config.json")
# ── read ──────────────────────────────────────────────────────────
def test_read_of_a_missing_file_is_empty_and_absent(self):
self.assertEqual(self.store.read(), ({}, ABSENT_REVISION))
def test_read_reports_the_sha256_of_the_file_bytes(self):
self._write({"web_ui": {"result_timeout": 3}})
data, revision = self.store.read()
self.assertEqual(data, {"web_ui": {"result_timeout": 3}})
expected = "sha256:" + hashlib.sha256(self._bytes()).hexdigest()
self.assertEqual(revision, expected)
self.assertEqual(revision_of(self._bytes()), expected)
def test_non_object_config_is_unreadable(self):
self._write([1, 2])
with self.assertRaises(ConfigUnreadable):
self.store.read()
def test_unreadable_path_is_unreadable(self):
os.mkdir(self.path) # a directory where the file should be
with self.assertRaises(ConfigUnreadable):
self.store.read()
# ── commit ────────────────────────────────────────────────────────
def test_commit_preserves_unknown_sections_and_secret(self):
self._write(
{
"custom_section": {"keep": True},
"integrations": {"lora_manager": {"civitai_api_key": "secret"}},
"web_ui": {"result_timeout": 1, "show_test_button": True},
}
)
def mutate(config):
config.setdefault("web_ui", {})["result_timeout"] = 7
return config
saved, revision = self.store.commit(mutate, expected_revision=None)
on_disk = json.loads(self._bytes())
self.assertEqual(saved, on_disk)
self.assertEqual(on_disk["custom_section"], {"keep": True})
self.assertEqual(
on_disk["integrations"]["lora_manager"]["civitai_api_key"], "secret"
)
self.assertEqual(
on_disk["web_ui"], {"result_timeout": 7, "show_test_button": True}
)
self.assertEqual(revision, revision_of(self._bytes()))
self.assertEqual(self.store.read(), (on_disk, revision))
def test_commit_creates_a_missing_file_and_its_parent(self):
self.path = os.path.join(self.tmpdir, "nested", "config.json")
saved, revision = self.store.commit(
lambda config: {**config, "a": 1}, expected_revision=ABSENT_REVISION
)
self.assertEqual(saved, {"a": 1})
self.assertEqual(json.loads(Path(self.path).read_text()), {"a": 1})
self.assertNotEqual(revision, ABSENT_REVISION)
def test_mutate_works_on_a_copy(self):
self._write({"web_ui": {"result_timeout": 1}})
seen = {}
def mutate(config):
seen["config"] = config
config["web_ui"]["result_timeout"] = 2
raise RuntimeError("validation failed late")
before = self._bytes()
with self.assertRaises(RuntimeError):
self.store.commit(mutate, expected_revision=None)
self.assertEqual(self._bytes(), before)
self.assertEqual(self.store.read()[0], {"web_ui": {"result_timeout": 1}})
def test_mutate_must_return_an_object(self):
self._write({"a": 1})
before = self._bytes()
with self.assertRaises(TypeError):
self.store.commit(lambda config: None, expected_revision=None)
self.assertEqual(self._bytes(), before)
def test_malformed_existing_config_is_refused_and_untouched(self):
Path(self.path).write_bytes(b"{not json")
calls = []
with self.assertRaises(ConfigUnreadable):
self.store.commit(
lambda config: calls.append(config) or config, expected_revision=None
)
self.assertEqual(calls, [])
self.assertEqual(self._bytes(), b"{not json")
self.assertEqual(self._leftovers(), [])
def test_write_failure_leaves_bytes_and_raises(self):
self._write({"a": 1})
before = self._bytes()
with patch.object(config_store.os, "fsync", side_effect=OSError("disk full")):
with self.assertRaises(OSError):
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
self.assertEqual(self._bytes(), before)
self.assertEqual(self._leftovers(), [])
def test_unwritable_directory_raises_oserror(self):
blocker = os.path.join(self.tmpdir, "afile")
Path(blocker).write_text("x")
self.path = os.path.join(blocker, "config.json") # parent is a file
with self.assertRaises(OSError):
self.store.commit(lambda c: {"a": 1}, expected_revision=None)
def test_stale_revision_conflicts_without_write(self):
self._write({"a": 1})
_data, revision = self.store.read()
self._write({"a": 2}) # another writer got there first
before = self._bytes()
calls = []
with self.assertRaises(RevisionConflict) as caught:
self.store.commit(
lambda c: calls.append(c) or c, expected_revision=revision
)
self.assertEqual(caught.exception.current, revision_of(before))
self.assertEqual(calls, [])
self.assertEqual(self._bytes(), before)
def test_matching_revision_commits(self):
self._write({"a": 1})
_data, revision = self.store.read()
saved, _ = self.store.commit(
lambda c: {**c, "b": 2}, expected_revision=revision
)
self.assertEqual(saved, {"a": 1, "b": 2})
def test_temp_file_removed_after_replace_failure(self):
self._write({"a": 1})
before = self._bytes()
with patch.object(
config_store.os, "replace", side_effect=PermissionError("locked")
):
with self.assertRaises(OSError):
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
self.assertEqual(self._bytes(), before)
self.assertEqual(self._leftovers(), [])
@unittest.skipUnless(os.name == "posix", "POSIX file modes")
def test_committed_file_is_private_to_the_user(self):
import stat
self._write({"a": 1})
os.chmod(self.path, 0o644)
self.store.commit(lambda c: c, expected_revision=None)
self.assertEqual(stat.S_IMODE(os.stat(self.path).st_mode), 0o600)
def _probe_lock(self):
probe = ProbeLock()
patcher = patch.object(config_store, "_COMMIT_LOCK", probe)
patcher.start()
self.addCleanup(patcher.stop)
return probe
def _hold_lock_in_a_commit(self, value="first"):
"""Start a commit that holds the lock until the returned event is set."""
inside = threading.Event()
release = threading.Event()
errors = []
def slow(config):
inside.set()
release.wait(10)
config[value] = 1
return config
def run():
try:
self.store.commit(slow, expected_revision=None)
except Exception as exc: # pragma: no cover - reported by the test
errors.append(exc)
thread = threading.Thread(target=run)
thread.start()
self.assertTrue(inside.wait(10))
return thread, release, errors
def test_concurrent_commits_serialize(self):
# The second commit is observed blocked on the lock before the first
# one may finish, so this fails whenever commit stops taking the lock.
self._write({"keep": True})
probe = self._probe_lock()
first, release, errors = self._hold_lock_in_a_commit()
calls = []
def fast(config):
calls.append(dict(config))
config["second"] = 2
return config
second = threading.Thread(
target=lambda: self.store.commit(fast, expected_revision=None)
)
second.start()
self.assertTrue(probe.contended.wait(10), "second commit never waited")
self.assertEqual(calls, []) # not called while the first holds the lock
release.set()
first.join(10)
second.join(10)
self.assertEqual(errors, [])
self.assertEqual(calls, [{"keep": True, "first": 1}]) # re-read inside
self.assertEqual(
json.loads(self._bytes()), {"keep": True, "first": 1, "second": 2}
)
def test_read_waits_for_a_running_commit(self):
# Windows cannot replace a file another handle has open, so a read
# runs under the commit lock and never overlaps a replace.
self._write({"keep": True})
probe = self._probe_lock()
first, release, errors = self._hold_lock_in_a_commit()
seen = []
reader = threading.Thread(target=lambda: seen.append(self.store.read()))
reader.start()
self.assertTrue(probe.contended.wait(10), "read did not take the lock")
self.assertEqual(seen, [])
release.set()
first.join(10)
reader.join(10)
self.assertEqual(errors, [])
self.assertEqual(seen[0][0], {"keep": True, "first": 1})
self.assertEqual(seen[0][1], revision_of(self._bytes()))
def test_replace_is_retried_after_a_transient_permission_error(self):
# Windows: an antivirus or indexer handle makes os.replace fail with
# PermissionError for a moment; the commit retries before failing.
self._write({"a": 1})
real_replace = os.replace
attempts = []
def flaky(src, dst):
attempts.append(dst)
if len(attempts) == 1:
raise PermissionError("in use by another process")
return real_replace(src, dst)
with (
patch.object(config_store, "_REPLACE_ATTEMPTS", 5),
patch.object(config_store.os, "replace", side_effect=flaky),
patch.object(config_store.time, "sleep") as sleep,
):
saved, _ = self.store.commit(
lambda c: {**c, "b": 2}, expected_revision=None
)
self.assertEqual(saved, {"a": 1, "b": 2})
self.assertEqual(json.loads(self._bytes()), {"a": 1, "b": 2})
self.assertEqual(len(attempts), 2)
sleep.assert_called_once_with(config_store._REPLACE_BACKOFF_SECONDS)
self.assertEqual(self._leftovers(), [])
def test_replace_gives_up_after_the_last_attempt(self):
self._write({"a": 1})
before = self._bytes()
with (
patch.object(config_store, "_REPLACE_ATTEMPTS", 3),
patch.object(
config_store.os, "replace", side_effect=PermissionError("locked")
) as replace,
patch.object(config_store.time, "sleep") as sleep,
):
with self.assertRaises(PermissionError):
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
self.assertEqual((replace.call_count, sleep.call_count), (3, 2))
self.assertEqual(self._bytes(), before)
self.assertEqual(self._leftovers(), [])
def test_replace_is_tried_once_outside_windows(self):
expected = 5 if os.name == "nt" else 1
self.assertEqual(config_store._REPLACE_ATTEMPTS, expected)
@unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks")
def test_symlinked_config_is_written_through_the_link(self):
# A config.json that is a symlink (or a single-file bind mount) is
# written beside its real target, so the link stays a link.
real_dir = os.path.join(self.tmpdir, "real")
os.mkdir(real_dir)
target = os.path.join(real_dir, "settings.json")
Path(target).write_text(json.dumps({"a": 1}), encoding="utf-8")
try:
os.symlink(target, self.path)
except (OSError, NotImplementedError): # pragma: no cover - Windows
self.skipTest("symlinks are not permitted here")
saved, revision = self.store.commit(
lambda c: {**c, "b": 2}, expected_revision=None
)
self.assertTrue(os.path.islink(self.path))
self.assertEqual(json.loads(Path(target).read_text()), {"a": 1, "b": 2})
self.assertEqual(self.store.read(), (saved, revision))
self.assertEqual(os.listdir(real_dir), ["settings.json"])
def test_publish_runs_under_the_lock_after_the_write(self):
# Runtimes publish inside the commit, so two saves publish in the
# order they were written and the runtime ends on the last one.
self._write({"a": 1})
seen = []
# The lock is re-entrant, so another thread probes whether it is held.
def probe_publish(config):
result = []
thread = threading.Thread(
target=lambda: result.append(
config_store._COMMIT_LOCK.acquire(blocking=False)
)
)
thread.start()
thread.join(10)
seen.append((config, json.loads(self._bytes()), result[0]))
saved, _ = self.store.commit(
lambda c: {**c, "b": 2}, expected_revision=None, publish=probe_publish
)
self.assertEqual(seen, [(saved, {"a": 1, "b": 2}, False)])
def test_publish_is_skipped_when_the_write_fails(self):
self._write({"a": 1})
published = []
with patch.object(config_store.os, "fsync", side_effect=OSError("disk full")):
with self.assertRaises(OSError):
self.store.commit(
lambda c: {**c, "b": 2},
expected_revision=None,
publish=published.append,
)
self.assertEqual(published, [])
def test_handles_closed_after_commit(self):
self._write({"a": 1})
self.store.read()
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
assert_closed(self, self.tmpdir)
os.unlink(self.path) # Windows refuses this while a handle is open
self.assertEqual(self._leftovers(), [])
class SingletonTests(unittest.TestCase):
def test_get_config_store_is_one_process_store_on_the_configured_path(self):
store = get_config_store()
self.assertIs(store, get_config_store())
tmpdir = tempfile.mkdtemp()
try:
custom = os.path.join(tmpdir, "c.json")
with patch.dict(os.environ, {"PROMPT_MANAGER_CONFIG_PATH": custom}):
self.assertEqual(store.path, custom)
finally:
shutil.rmtree(tmpdir, ignore_errors=True)
if __name__ == "__main__":
unittest.main()
+614
View File
@@ -0,0 +1,614 @@
"""The read-only health check (System › Overview and Maintenance).
``run_health_checks`` reads one snapshot on its own ``mode=ro`` connection:
SQLite's integrity and foreign-key checks, stored prompt hashes against
``utils.hashing``, duplicate pair rows, the legacy consistency queries, image
links whose file is gone (only under readable sources) and the Needs-review
count when one is supplied. It never writes, and it never turns a check it
could not run into a count.
"""
import hashlib
import ntpath
import os
import sqlite3
import sys
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from database import health # noqa: E402
from database.operations import PromptDatabase # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
try:
from tests.open_handles import assert_closed # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed # noqa: E402
CHECK_IDS = [
"integrity",
"foreign_keys",
"canonical_hashes",
"pair_uniqueness",
"missing_files",
"ambiguous_metadata",
"consistency",
]
def _digest(path):
if not os.path.exists(path):
return None
with open(path, "rb") as handle:
return hashlib.sha256(handle.read()).hexdigest()
class HealthTestCase(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.root = Path(os.path.realpath(self.temp.name))
self.output = self.root / "output"
self.output.mkdir()
self.db_path = str(self.root / "prompts.db")
self.db = PromptDatabase(self.db_path)
def tearDown(self):
self.db.close_all()
assert_closed(self, self.root)
self.temp.cleanup()
def save(self, text, **metadata):
return self.db.save_prompt(
text, prompt_hash=generate_prompt_hash(text), **metadata
)
def image(self, prompt_id, name, *, exists=True, folder=None):
folder = Path(folder or self.output)
path = folder / name
if exists:
path.write_bytes(b"png")
return self.db.link_image_to_prompt(prompt_id, str(path))
def run_checks(self, roots=None, **kwargs):
roots = [str(self.output)] if roots is None else roots
return self.db.run_health_checks(roots, **kwargs)
@staticmethod
def check(report, check_id):
return next(c for c in report["checks"] if c["id"] == check_id)
def raw(self):
"""A plain connection for planting damage the app itself never writes."""
conn = sqlite3.connect(self.db_path)
conn.execute("PRAGMA foreign_keys=OFF")
return conn
class TestHealthReport(HealthTestCase):
def test_clean_library_passes_except_unavailable_review(self):
prompt = self.save("A clean prompt")
self.db.save_prompt_pair("Cat", "Blur")
self.image(prompt, "a.png")
report = self.run_checks(review_count=0)
self.assertTrue(report["success"])
self.assertEqual([c["id"] for c in report["checks"]], CHECK_IDS)
self.assertEqual(
[c["label"] for c in report["checks"]],
[
"Integrity check",
"Foreign keys & pair references",
"Canonical hashes",
"Pair uniqueness",
"Missing files",
"Ambiguous metadata",
"Consistency",
],
)
for check in report["checks"]:
self.assertEqual(check["status"], "passed", check)
self.assertEqual(check["count"], 0, check)
self.assertEqual(report["status"], "passed")
self.assertRegex(report["checked_at"], r"^\d{4}-\d\d-\d\dT.*\+00:00$")
def test_health_check_never_writes(self):
prompt = self.save("Untouched")
self.image(prompt, "kept.png")
self.image(prompt, "gone.png", exists=False)
with self.raw() as conn:
conn.execute("UPDATE prompts SET hash='stale' WHERE id=?", (prompt,))
conn.close()
# The app's own connection stays open, as in the running server.
self.db.get_statistics()
files = [self.db_path, self.db_path + "-wal"]
before = [_digest(path) for path in files]
shm_size = os.path.getsize(self.db_path + "-shm")
media = sorted(os.listdir(self.output))
opened = []
real_connect = sqlite3.connect
class Recording(sqlite3.Connection):
changes_at_close = None
def close(self):
Recording.changes_at_close = self.total_changes
super().close()
def connect(*args, **kwargs):
conn = real_connect(*args, factory=Recording, **kwargs)
opened.append((args, kwargs, conn))
return conn
with patch.object(health.sqlite3, "connect", side_effect=connect):
report = self.run_checks()
self.assertEqual(report["status"], "incomplete")
self.assertEqual(len(opened), 1)
args, kwargs, conn = opened[0]
self.assertTrue(str(args[0]).endswith("?mode=ro"), args)
self.assertTrue(kwargs.get("uri"))
self.assertEqual(Recording.changes_at_close, 0)
with self.assertRaises(sqlite3.ProgrammingError): # closed in finally
conn.execute("SELECT 1")
self.assertEqual([_digest(path) for path in files], before)
# The -shm holds WAL read marks every reader updates; it never grows.
self.assertEqual(os.path.getsize(self.db_path + "-shm"), shm_size)
self.assertEqual(sorted(os.listdir(self.output)), media)
with self.db.model.get_connection() as live:
stored = live.execute(
"SELECT hash FROM prompts WHERE id=?", (prompt,)
).fetchone()[0]
self.assertEqual(stored, "stale")
def test_connection_closed_when_a_check_raises(self):
self.save("Any")
with (
patch.object(
health,
"_canonical_hashes",
side_effect=sqlite3.OperationalError("boom"),
),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
canonical = self.check(report, "canonical_hashes")
self.assertEqual(canonical["status"], "failed")
self.assertIsNone(canonical["count"])
self.assertEqual(canonical["detail"], ["The check could not finish."])
self.assertEqual(self.check(report, "integrity")["status"], "passed")
self.assertEqual(report["status"], "incomplete")
self.db.close_all()
assert_closed(self, self.root)
def test_unopenable_database_fails_every_database_check(self):
missing = PromptDatabase.__new__(PromptDatabase)
missing.model = type("M", (), {"db_path": str(self.root / "nope.db")})()
with self.assertLogs("prompt_manager.database.health", "ERROR"):
report = missing.run_health_checks([str(self.output)], review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses["integrity"], "failed")
self.assertEqual(statuses["missing_files"], "failed")
self.assertEqual(statuses["ambiguous_metadata"], "passed")
self.assertEqual(report["status"], "incomplete")
self.assertFalse(os.path.exists(self.root / "nope.db"))
def test_integrity_and_foreign_key_issues_reported(self):
prompt = self.save("Linked")
with self.raw() as conn:
conn.execute("CREATE TABLE probe(a)")
conn.execute("CREATE INDEX probe_a ON probe(a)")
conn.executemany("INSERT INTO probe VALUES(?)", [(i,) for i in range(16)])
conn.execute(
"INSERT INTO generated_images(prompt_id,image_path,filename,file_path)"
" VALUES(?,?,?,?)",
(9999, str(self.output / "orphan.png"), "orphan.png", "orphan"),
)
conn.execute(
"INSERT INTO prompt_tags(prompt_id,tag_id) VALUES(?,?)", (prompt, 777)
)
conn.close()
with self.raw() as conn:
conn.execute("PRAGMA writable_schema=ON")
conn.execute(
"UPDATE sqlite_master SET sql='CREATE INDEX probe_a ON probe(a DESC)'"
" WHERE name='probe_a'"
)
conn.close()
self.db.close_all() # the next read sees the edited schema
(self.output / "orphan.png").write_bytes(b"png")
with self.raw() as conn:
expected = [r[0] for r in conn.execute("PRAGMA integrity_check(100)")]
conn.close()
self.assertGreater(len(expected), 10)
report = self.run_checks(review_count=0)
integrity = self.check(report, "integrity")
self.assertEqual(integrity["status"], "issues")
self.assertEqual(integrity["count"], len(expected))
self.assertEqual(integrity["detail"], expected[:10])
self.assertIn("missing from index probe_a", integrity["detail"][0])
foreign = self.check(report, "foreign_keys")
self.assertEqual(foreign["status"], "issues")
self.assertEqual(foreign["count"], 2)
self.assertEqual(
sorted(foreign["detail"]),
[
"generated_images row 1 references a missing prompts row",
"prompt_tags row 1 references a missing tags row",
],
)
consistency = self.check(report, "consistency")
self.assertEqual(consistency["status"], "issues")
self.assertEqual(consistency["count"], 1)
self.assertEqual(
consistency["detail"], ["Image 1 references non-existent prompt 9999"]
)
self.assertEqual(report["status"], "issues")
def test_canonical_hash_mismatch_counts_rows(self):
ids = [self.save(f"Prompt {n}") for n in range(14)]
with self.raw() as conn:
for n, prompt_id in enumerate(ids[:12]):
conn.execute(
"UPDATE prompts SET hash=? WHERE id=?", (f"bad{n}", prompt_id)
)
conn.execute("UPDATE prompts SET hash=NULL WHERE id=?", (ids[12],))
conn.close()
# Normalised spellings hash alike: this one is not a mismatch.
self.save(" MIXED Case ")
report = self.run_checks(review_count=0)
canonical = self.check(report, "canonical_hashes")
self.assertEqual(canonical["status"], "issues")
self.assertEqual(canonical["count"], 13)
self.assertEqual(len(canonical["detail"]), 10)
self.assertEqual(
canonical["detail"][0],
f"Prompt {ids[0]}: the stored hash does not match its text",
)
def test_duplicate_pair_rows_reported(self):
pair = self.db.save_prompt_pair("Cat", "")
positive = self.db.get_prompt_pair(pair)["positive_prompt_id"]
self.db.save_prompt_pair("Dog", "Blur")
with self.raw() as conn:
# Only the partial index keeps positive-only pairs unique; a restored
# or hand-edited database may have lost it.
conn.execute("DROP INDEX idx_prompt_pairs_positive_only")
conn.executemany(
"INSERT INTO prompt_pairs(positive_prompt_id,negative_prompt_id)"
" VALUES(?,NULL)",
[(positive,), (positive,)],
)
conn.close()
report = self.run_checks(review_count=0)
pairs = self.check(report, "pair_uniqueness")
self.assertEqual(pairs["status"], "issues")
self.assertEqual(pairs["count"], 2)
self.assertEqual(
pairs["detail"],
[f"Positive prompt {positive} with no negative: 3 pair rows"],
)
def test_integrity_count_at_the_limit_is_flagged_capped(self):
class Rows:
def __init__(self, rows):
self.rows = rows
def execute(self, _sql):
return iter([(row,) for row in self.rows])
capped = health._integrity(Rows(["bad"] * health.INTEGRITY_LIMIT), [])
self.assertEqual(capped["status"], "issues")
self.assertEqual(capped["count"], health.INTEGRITY_LIMIT)
self.assertTrue(capped["capped"])
exact = health._integrity(Rows(["bad"] * 3), [])
self.assertEqual(exact["count"], 3)
self.assertFalse(exact["capped"])
self.assertFalse(health._integrity(Rows(["ok"]), [])["capped"])
def test_an_unexpected_error_fails_only_its_own_check(self):
self.save("Any")
with (
patch.object(health, "_pair_uniqueness", side_effect=KeyError("boom")),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses.pop("pair_uniqueness"), "failed")
self.assertEqual(set(statuses.values()), {"passed"})
class TestDatabaseUri(unittest.TestCase):
"""The ``mode=ro`` URI of the database file, for every path form SQLite opens."""
def test_posix_path_is_a_local_file_uri(self):
self.assertEqual(
health.readonly_uri("/data/my db#1?.db", windows=False),
"file:///data/my%20db%231%3F.db?mode=ro",
)
def test_windows_drive_path(self):
self.assertEqual(
health.readonly_uri(r"C:\Users\Me\prompts.db", windows=True),
"file:///C:/Users/Me/prompts.db?mode=ro",
)
def test_windows_unc_path_has_an_empty_authority(self):
# pathlib spells \\server\share as file://server/share, which SQLite
# rejects (a non-local authority); an empty one keeps the UNC path.
uri = health.readonly_uri(r"\\server\share\Comfy UI\prompts.db", windows=True)
self.assertEqual(uri, "file:////server/share/Comfy%20UI/prompts.db?mode=ro")
self.assertTrue(ntpath.isabs(r"\\server\share\Comfy UI\prompts.db"))
def test_uri_opens_a_path_with_uri_characters(self):
with tempfile.TemporaryDirectory() as temp:
# "?" is not a valid Windows file name character; "#", "%" and spaces are.
name = "a #b%20 c.db" if os.name == "nt" else "a #b?c%20.db"
path = os.path.join(os.path.realpath(temp), name)
conn = sqlite3.connect(path)
try:
conn.execute("CREATE TABLE t(a)")
conn.commit()
finally:
conn.close()
ro = sqlite3.connect(health.readonly_uri(path), uri=True)
try:
self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,))
with self.assertRaises(sqlite3.OperationalError):
ro.execute("INSERT INTO t VALUES(1)")
finally:
ro.close()
@unittest.skipUnless(os.name == "nt", "a native UNC path exists on Windows only")
def test_native_unc_path_opens(self):
with tempfile.TemporaryDirectory() as temp:
path = os.path.join(os.path.realpath(temp), "prompts.db")
conn = sqlite3.connect(path)
try:
conn.execute("CREATE TABLE t(a)")
conn.commit()
finally:
conn.close()
drive, rest = ntpath.splitdrive(path)
unc = "\\\\localhost\\" + drive[0] + "$" + rest
if not os.path.exists(unc):
self.skipTest("the administrative share is not reachable here")
ro = sqlite3.connect(health.readonly_uri(unc), uri=True)
try:
self.assertEqual(ro.execute("SELECT COUNT(*) FROM t").fetchone(), (0,))
finally:
ro.close()
class TestMissingFiles(HealthTestCase):
def test_missing_files_counted_under_readable_sources_only(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
for n in range(12):
self.image(prompt, f"gone-{n}.png", exists=False)
elsewhere = self.root / "elsewhere"
elsewhere.mkdir()
self.image(prompt, "outside.png", exists=False, folder=elsewhere)
report = self.run_checks(review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 12)
self.assertEqual(len(missing["detail"]), 10)
self.assertRegex(missing["detail"][0], r"^Image \d+: gone-0\.png is missing$")
self.assertNotIn(str(self.root), " ".join(missing["detail"]))
with self.db.model.get_connection() as conn:
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM generated_images").fetchone()[0],
14,
)
def test_links_outside_every_source_are_named_not_counted(self):
prompt = self.save("Images")
elsewhere = self.root / "elsewhere"
elsewhere.mkdir()
self.image(prompt, "outside.png", exists=False, folder=elsewhere)
missing = self.check(self.run_checks(review_count=0), "missing_files")
self.assertEqual(missing["status"], "passed")
self.assertEqual(missing["count"], 0)
self.assertEqual(
missing["detail"],
["1 link outside the image sources was not checked."],
)
def test_missing_file_vs_unavailable_root(self):
# Per source (S2 ruling): the readable source is counted, the absent
# one reports "unavailable" with no count, and the report is incomplete.
prompt = self.save("Images")
absent = self.root / "unmounted"
absent.mkdir()
self.image(prompt, "on-nas.png", folder=absent)
self.image(prompt, "gone.png", exists=False)
os.unlink(absent / "on-nas.png")
absent.rmdir()
report = self.run_checks([str(self.output), str(absent)], review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 1)
self.assertTrue(missing["partial"])
self.assertEqual(
missing["sources"],
[
{"name": "output", "status": "checked", "count": 1},
{"name": "unmounted", "status": "unavailable", "count": None},
],
)
self.assertEqual(
missing["detail"][0],
"1 image source is unavailable, so its files were not counted.",
)
self.assertRegex(missing["detail"][1], r"^Image \d+: gone\.png is missing$")
# The link under the unavailable source is neither counted nor "outside".
self.assertEqual(len(missing["detail"]), 2)
self.assertNotIn(str(self.root), " ".join(missing["detail"]))
self.assertEqual(report["status"], "incomplete")
def test_partial_sources_with_no_missing_file_are_not_passed(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
absent = self.root / "unmounted"
missing = self.check(
self.run_checks([str(self.output), str(absent)], review_count=0),
"missing_files",
)
self.assertEqual(missing["status"], "unavailable")
self.assertEqual(missing["count"], 0)
self.assertTrue(missing["partial"])
self.assertEqual(
[s["status"] for s in missing["sources"]], ["checked", "unavailable"]
)
def test_every_source_unavailable_counts_nothing(self):
prompt = self.save("Images")
absent = self.root / "unmounted"
self.image(prompt, "gone.png", exists=False, folder=self.output)
report = self.run_checks([str(absent)], review_count=0)
missing = self.check(report, "missing_files")
self.assertEqual(missing["status"], "unavailable")
self.assertIsNone(missing["count"])
self.assertFalse(missing["partial"])
self.assertEqual(
missing["sources"],
[{"name": "unmounted", "status": "unavailable", "count": None}],
)
self.assertEqual(
missing["detail"],
["1 image source is unavailable, so missing files were not counted."],
)
def test_each_source_counts_its_own_missing_files(self):
prompt = self.save("Images")
archive = self.root / "archive"
archive.mkdir()
self.image(prompt, "a.png", exists=False)
self.image(prompt, "b.png", exists=False, folder=archive)
self.image(prompt, "c.png", exists=False, folder=archive)
missing = self.check(
self.run_checks([str(self.output), str(archive)], review_count=0),
"missing_files",
)
self.assertEqual(missing["count"], 3)
self.assertFalse(missing["partial"])
self.assertEqual(
[(s["name"], s["count"]) for s in missing["sources"]],
[("output", 1), ("archive", 2)],
)
def test_no_sources_is_unavailable(self):
self.save("Images")
missing = self.check(self.run_checks([], review_count=0), "missing_files")
self.assertEqual(missing["status"], "unavailable")
self.assertIsNone(missing["count"])
self.assertEqual(missing["detail"], ["No image source is available."])
self.assertEqual(missing["sources"], [])
self.assertFalse(missing["partial"])
def test_a_bad_stored_path_fails_only_missing_files(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
with self.raw() as conn:
conn.execute(
"INSERT INTO generated_images(prompt_id,image_path,filename)"
" VALUES(?,?,?)",
(prompt, str(self.root / "else\x00where.png"), "bad.png"),
)
conn.close()
real_realpath = os.path.realpath
def realpath(path, *args, **kwargs):
# POSIX (and Windows before Python 3.11) raise for the NUL byte;
# newer ntpath returns it unchanged. Raise everywhere, as POSIX does.
if "\x00" in str(path):
raise ValueError("embedded null byte")
return real_realpath(path, *args, **kwargs)
with (
patch.object(health.os.path, "realpath", side_effect=realpath),
self.assertLogs("prompt_manager.database.health", "ERROR"),
):
report = self.run_checks(review_count=0)
statuses = {c["id"]: c["status"] for c in report["checks"]}
self.assertEqual(statuses.pop("missing_files"), "failed")
self.assertEqual(set(statuses.values()), {"passed"})
def test_files_are_checked_after_the_read_snapshot_ends(self):
prompt = self.save("Images")
self.image(prompt, "here.png")
self.image(prompt, "gone.png", exists=False)
opened = []
real_connect = sqlite3.connect
def connect(*args, **kwargs):
conn = real_connect(*args, **kwargs)
opened.append(conn)
return conn
real_exists = os.path.exists
in_snapshot = []
def exists(path):
for conn in opened:
try:
in_snapshot.append(conn.in_transaction)
except sqlite3.ProgrammingError: # closed: no snapshot
in_snapshot.append(False)
return real_exists(path)
with (
patch.object(health.sqlite3, "connect", side_effect=connect),
patch.object(health.os.path, "exists", side_effect=exists),
):
missing = self.check(self.run_checks(review_count=0), "missing_files")
self.assertEqual((missing["status"], missing["count"]), ("issues", 1))
self.assertEqual(len(opened), 1)
self.assertTrue(in_snapshot)
self.assertNotIn(True, in_snapshot)
class TestAmbiguousMetadata(HealthTestCase):
def test_ambiguous_metadata_unavailable_without_review_queue(self):
self.save("Any")
report = self.run_checks()
ambiguous = self.check(report, "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "unavailable")
self.assertIsNone(ambiguous["count"])
self.assertEqual(ambiguous["detail"], ["Needs review is not available yet."])
self.assertEqual(report["status"], "incomplete")
def test_review_count_reports_images_needing_review(self):
report = self.run_checks(review_count=3)
ambiguous = self.check(report, "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "issues")
self.assertEqual(ambiguous["count"], 3)
self.assertEqual(ambiguous["detail"], [])
self.assertEqual(report["status"], "issues")
def test_invalid_review_count_is_rejected(self):
for value in (-1, True, "3", 1.5):
with self.assertRaises(ValueError):
self.run_checks(review_count=value)
class TestOverallStatus(unittest.TestCase):
def test_overall_status_precedence(self):
def checks(*statuses):
return [{"status": status} for status in statuses]
self.assertEqual(health.overall_status(checks("passed", "passed")), "passed")
self.assertEqual(health.overall_status(checks("passed", "issues")), "issues")
self.assertEqual(
health.overall_status(checks("issues", "unavailable")), "incomplete"
)
self.assertEqual(health.overall_status(checks("failed")), "incomplete")
# A check that found issues under only some sources is incomplete too.
partial = [{"status": "issues", "partial": True}, {"status": "passed"}]
self.assertEqual(health.overall_status(partial), "incomplete")
if __name__ == "__main__":
unittest.main()
+113 -32
View File
@@ -299,45 +299,126 @@ class OutputWriterTests(fixtures.unittest.TestCase):
)
self.assertEqual(writer.context_budget.charged_bytes, 0)
def isolated_writer(self) -> HistoryWriter:
"""A ready writer whose budget no other writer or test can charge."""
writer = HistoryWriter(
lambda: self.db, clock=self.clock, context_budget=ContextBudget()
)
self.writers.append(writer)
writer.start()
self.wait_for(lambda: writer.status().accepting)
return writer
def rotate_between_transactions(self, writer, start, after, restore, hold=None):
"""Wrap ``after`` so a restore or invalidation lands right after it.
Returns the patch and an event set once that rotation has returned on
the worker. A restore is a full verify/migrate/backup/replace cycle, so
the handoff waits on this event with the generous ``BLOCK_SECONDS``
ceiling instead of the short worker-behavior deadline (TESTING.md,
Windows portability preflight).
"""
backup = os.path.join(self.directory.name, f"restore-{start.run_id}.db")
self.assertTrue(self.db.model.backup_database(backup))
original = getattr(self.db, after)
rotated = threading.Event()
def write(*args, **kwargs):
result = original(*args, **kwargs)
try:
if hold is not None:
entered, release = hold
entered.set()
release.wait(BLOCK_SECONDS)
if restore:
self.db.model.restore_from_file(backup)
else:
writer.invalidate()
finally:
rotated.set()
return result
return patch.object(self.db, after, write), rotated
def test_restore_or_generation_rotation_between_transactions_discards_evidence(
self,
):
for after in ("write_execution_finish", "write_execution_contexts_for_finish"):
for restore in (False, True):
with self.subTest(after=after, restore=restore):
writer = self.writer()
start = self.start_command(len(self.writers))
self.db.write_execution_start(start)
backup = os.path.join(self.directory.name, "restore.db")
self.assertTrue(self.db.model.backup_database(backup))
original = getattr(self.db, after)
# Each subtest owns its writer and budget and retires them
# before the next starts, so one slow restore cannot leave
# in-flight charges or an epoch rotation for the next.
writer = self.isolated_writer()
try:
start = self.start_command(len(self.writers))
self.db.write_execution_start(start)
rotation, rotated = self.rotate_between_transactions(
writer, start, after, restore
)
with rotation:
self.finalization(writer, start)
self.assertTrue(rotated.wait(BLOCK_SECONDS))
self.wait_for(lambda: writer.status().pending == 0)
self.assertEqual(
self.db.get_execution_outputs(start.run_id)["state"],
"not_captured",
)
self.assertEqual(
self.db.get_execution_contexts(start.run_id)["state"],
(
"complete"
if after == "write_execution_contexts_for_finish"
and not restore
else "not_captured"
),
)
self.assertEqual(writer.context_budget.charged_bytes, 0)
finally:
self.assertTrue(writer.close(BLOCK_SECONDS))
def write(*args, **kwargs):
result = original(*args, **kwargs)
if restore:
self.db.model.restore_from_file(backup)
else:
writer.invalidate()
return result
with patch.object(self.db, after, write):
self.finalization(writer, start)
self.wait_for(lambda: writer.status().pending == 0)
self.assertEqual(
self.db.get_execution_outputs(start.run_id)["state"],
"not_captured",
)
self.assertEqual(
self.db.get_execution_contexts(start.run_id)["state"],
(
"complete"
if after == "write_execution_contexts_for_finish"
and not restore
else "not_captured"
),
)
self.assertEqual(writer.context_budget.charged_bytes, 0)
self.assertTrue(writer.close())
def test_slow_restore_between_transactions_holds_charge_only_while_in_flight(
self,
):
writer = self.isolated_writer()
start = self.start_command()
self.db.write_execution_start(start)
entered, release = self.block()
rotation, rotated = self.rotate_between_transactions(
writer, start, "write_execution_finish", True, (entered, release)
)
with rotation:
context = self.sealed(writer)
outputs = self.outputs(writer, context.lease.byte_count)
charge = writer.context_budget.charged_bytes
self.assertGreater(charge, 0)
self.assertTrue(
writer.submit_finalization(self.finish_command(start), context, outputs)
)
self.assertTrue(entered.wait(BLOCK_SECONDS))
# The worker is between transactions: the payload is owned, not leaked.
self.assertEqual(writer.status().pending, 1)
self.assertEqual(writer.context_budget.charged_bytes, charge)
release.set()
self.assertTrue(rotated.wait(BLOCK_SECONDS))
self.wait_for(lambda: writer.status().pending == 0)
self.assertEqual(writer.context_budget.charged_bytes, 0)
self.assertFalse(writer.status().accepting)
self.assertEqual(
self.db.get_execution_outputs(start.run_id)["state"], "not_captured"
)
# The restored epoch is probed again and new evidence is accepted.
self.clock.advance(5)
self.wait_for(lambda: writer.status().accepting)
restored = self.start_command(2)
self.db.write_execution_start(restored)
self.finalization(writer, restored)
self.wait_for(lambda: writer.status().pending == 0)
self.assertEqual(
self.db.get_execution_outputs(restored.run_id)["state"], "complete"
)
self.assertEqual(writer.context_budget.charged_bytes, 0)
self.assertTrue(writer.close(BLOCK_SECONDS))
def test_invalidation_during_output_preparation_rechecks_actual_commit(self):
import database.execution_outputs as outputs_db
+45
View File
@@ -5,7 +5,10 @@ returns for the same input (tests/test_generation_params_js_parity.py checks
that directly when Node is available).
"""
import json
import os
import shutil
import subprocess # nosec B404 - runs a fixed Node one-liner
import sys
import unittest
@@ -17,6 +20,7 @@ from utils.generation_params import ( # noqa: E402
extract_generation_params,
extract_from_records,
parse_js_json,
parse_page_json,
)
try:
@@ -219,6 +223,47 @@ class ParsesLikeJavaScript(unittest.TestCase):
run("seed-beyond-double-precision")["seed"], 4974052037430000001.0
)
def test_page_parse_reads_nan_numbers_as_null(self):
# gallery.js and admin.js replace NaN with null before JSON.parse.
self.assertEqual(
parse_page_json('{"a": NaN, "b": [NaN], "t": "NaN"}'),
{"a": None, "b": [None], "t": "NaN"},
)
self.assertEqual(
parse_page_json('{"seed": 9007199254740993}'), {"seed": 9007199254740992.0}
)
for text in ('{"a": Infinity}', "[-Infinity]", "not json", "", None):
with self.subTest(text=text):
self.assertIsNone(parse_page_json(text))
@unittest.skipUnless(shutil.which("node"), "Node is not installed")
def test_page_parse_equals_the_legacy_pages_for_nan_numbers(self):
# gallery.js/admin.js: two NaN regexes, then JSON.parse (Infinity throws).
texts = [
'{"1": {"is_changed": [NaN]}, "2": {"is_changed": NaN, "v": 1.5}}',
'{"a":NaN,"b":[NaN, 2]}',
'{"a": Infinity}',
'{"seed": 9007199254740993}',
]
script = (
"const texts = JSON.parse(require('fs').readFileSync(0, 'utf8'));"
"console.log(JSON.stringify(texts.map((t) => { try { return {v: JSON.parse("
"t.replace(/:\\s*NaN\\b/g, ': null').replace(/\\bNaN\\b/g, 'null'))}; }"
" catch (e) { return null; } })));"
)
completed = subprocess.run( # nosec B603 B607 - fixed argv, no shell
["node", "-e", script],
input=json.dumps(texts),
capture_output=True,
text=True,
check=True,
timeout=60,
)
expected = json.loads(completed.stdout)
for text, page in zip(texts, expected):
with self.subTest(text=text):
self.assertEqual(parse_page_json(text), page and page["v"])
def test_duplicate_keys_keep_the_last_value(self):
self.assertEqual(parse_js_json('{"a": 1, "a": 2}'), {"a": 2})
+804
View File
@@ -0,0 +1,804 @@
"""Bulk image tagging as an Activity job: Add tags and Auto-tag (#249 D1).
``POST /prompt_manager/jobs/image-tags`` starts one record-changing
``image-tags`` job over 1..500 listed images. Add tags unions a fixed set
into each file's byte record; Auto-tag runs one legacy engine on a private
capture of each still image (under the shared model gate, never
downloading) and unions its top N suggestions. No real model is loaded: the
engine is B2's fake service.
"""
import asyncio
import hashlib
import json
import sqlite3
import threading
import unittest
from unittest.mock import patch
try:
from test_api_annotation_suggest import SCORED, WD14, FakeTagger, SuggestTestCase
from test_api_images import make_png
except ImportError:
from tests.test_api_annotation_suggest import (
SCORED,
WD14,
FakeTagger,
SuggestTestCase,
)
from tests.test_api_images import make_png
from database.execution_history_schema import rotate_history_epoch
from py import model_gate
from py.api import jobs as jobs_module
from py.config import AutotagConfig
from utils.media_capture import CaptureCapacity
START = "/prompt_manager/jobs/image-tags"
WAIT = 30.0
KIND = getattr(jobs_module, "IMAGE_TAG_JOB_KIND", "image-tags")
GROUPS = ("tagged", "unchanged", "skipped", "errors")
class HookedTagger(FakeTagger):
"""The B2 fake plus a hook that runs inside each inference."""
def __init__(self):
super().__init__()
self.on_generate = None
def generate_tag_suggestions(self, image_path, **kwargs):
result = super().generate_tag_suggestions(image_path, **kwargs)
if self.on_generate is not None:
self.on_generate(image_path)
return result
class ImageTagJobTestCase(SuggestTestCase):
async def get_application(self):
app = await super().get_application()
self.tagger = HookedTagger()
patcher = patch("py.autotag.get_autotag_service", return_value=self.tagger)
patcher.start()
self.addCleanup(patcher.stop)
return app
async def tearDownAsync(self):
# Every job of the test has ended before the app goes away.
for job in self.api._jobs.list():
if job.task is not None and not job.task.done():
job.task.cancel()
await asyncio.gather(job.task, return_exceptions=True)
await super().tearDownAsync()
# -- helpers ---------------------------------------------------------
def png(self, path, size=(8, 6), color=(200, 120, 40)):
make_png(self.output_dir / path, size=size, color=color)
return self.output_dir / path
def digest(self, path):
return hashlib.sha256((self.output_dir / path).read_bytes()).hexdigest()
def stored(self, path):
return self.api.db.read_image_tags(self.locator(path), self.digest(path))
def add_body(self, paths, tags, **changes):
return {
"mode": "add",
"targets": [self.target(p) for p in paths],
"tags": list(tags),
**changes,
}
def auto_body(self, paths, model_type="wd14-vit", options=None, **changes):
return {
"mode": "auto",
"targets": [self.target(p) for p in paths],
"model_type": model_type,
"options": dict(WD14) if options is None else options,
**changes,
}
async def start(self, body):
response = await self.client.post(START, json=body)
return response.status, await self.json_of(response)
async def start_ok(self, body):
status, data = await self.start(body)
self.assertEqual(status, 202, data)
return self.api._jobs.get(data["job"]["id"])
async def run_job(self, body):
job = await self.start_ok(body)
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
return job
def rows(self, job, group):
return [(row["path"], row["detail"]) for row in job.outcomes[group]]
async def wait_for(self, event):
ok = await asyncio.get_running_loop().run_in_executor(None, event.wait, WAIT)
self.assertTrue(ok, "the fake engine was never entered")
class TestAddTags(ImageTagJobTestCase):
async def test_add_job_unions_tags_on_each_target_and_records_outcomes(self):
for name in ("a.png", "b.png", "c.png"):
self.png(name, color=(len(name), ord(name[0]), 9))
self.saved_set("a.png", (self.output_dir / "a.png").read_bytes(), ["keep_me"])
self.saved_set(
"c.png", (self.output_dir / "c.png").read_bytes(), ["Sandstone", "desert"]
)
job = await self.run_job(
self.add_body(
["a.png", "b.png", "c.png", "gone.png"], ["sandstone", "desert"]
)
)
self.assertEqual(job.kind, KIND)
self.assertEqual(job.title, "Add tags to images")
self.assertEqual(job.state, "finished")
self.assertTrue(job.record_changing)
self.assertEqual(job.scope, {"mode": "add", "images": 4})
self.assertEqual(
job.counters, {"tagged": 2, "unchanged": 1, "skipped": 1, "errors": 0}
)
self.assertEqual(
self.stored("a.png")["tags"], ["keep_me", "sandstone", "desert"]
)
self.assertEqual(self.stored("b.png")["tags"], ["sandstone", "desert"])
# The stored spelling wins; nothing new means no write.
self.assertEqual(self.stored("c.png")["tags"], ["Sandstone", "desert"])
self.assertEqual(self.stored("c.png")["tag_revision"], 1)
self.assertEqual(
self.rows(job, "tagged"),
[("a.png", "Added 2 tags"), ("b.png", "Added 2 tags")],
)
self.assertEqual(self.rows(job, "unchanged"), [("c.png", "No new tags")])
self.assertEqual(self.rows(job, "skipped"), [("gone.png", "File not found")])
self.assertEqual(job.outcomes["tagged"][0]["root_index"], 0)
self.assertEqual((job.processed, job.total, job.progress), (4, 4, 100))
self.assertTrue(any("+ " in line and "a.png" in line for line in job.log))
self.assertTrue(job.log[-1].endswith("Add tags finished · 4 files processed"))
# Add tags never needs the model.
self.assertEqual(self.tagger.load_calls, [])
self.assertIsNone(model_gate.holder())
async def test_add_tags_vocabulary_names_become_global_tags(self):
self.png("a.png")
await self.run_job(self.add_body(["a.png"], ["brand_new_tag"]))
names = set(self.api.db.get_all_tags())
self.assertIn("brand_new_tag", names)
async def test_add_over_the_tag_cap_is_an_error_row_with_zero_writes(self):
self.png("a.png")
full = [f"t{i:03d}" for i in range(100)]
self.saved_set("a.png", (self.output_dir / "a.png").read_bytes(), full)
job = await self.run_job(self.add_body(["a.png"], ["one_more"]))
self.assertEqual(job.counters["errors"], 1)
self.assertEqual(
self.rows(job, "errors"),
[("a.png", "An image can carry at most 100 tags.")],
)
self.assertEqual(self.stored("a.png")["tags"], full)
async def test_video_skipped_in_auto_and_tagged_in_add(self):
(self.output_dir / "clip.mp4").write_bytes(b"\x00\x00\x00\x18ftypmp42" * 8)
added = await self.run_job(self.add_body(["clip.mp4"], ["motion"]))
self.assertEqual(self.rows(added, "tagged"), [("clip.mp4", "Added 1 tag")])
self.assertEqual(self.stored("clip.mp4")["tags"], ["motion"])
auto = await self.run_job(self.auto_body(["clip.mp4"]))
self.assertEqual(
self.rows(auto, "skipped"), [("clip.mp4", "Video: manual tags only")]
)
self.assertEqual(self.tagger.calls, [])
self.assertEqual(self.stored("clip.mp4")["tags"], ["motion"])
async def test_not_a_still_image_and_too_large_are_skipped(self):
(self.output_dir / "broken.png").write_bytes(b"not an image")
self.png("big.png")
self.api._capture_store().max_file = 16
auto = await self.run_job(self.auto_body(["big.png"]))
self.assertEqual(
self.rows(auto, "skipped"), [("big.png", "Too large to verify")]
)
self.api._capture_store().max_file = 2**20
auto = await self.run_job(self.auto_body(["broken.png"]))
self.assertEqual(
self.rows(auto, "skipped"), [("broken.png", "Not a supported still image")]
)
self.api._capture_store().max_file = 16
added = await self.run_job(self.add_body(["big.png"], ["x"]))
self.assertEqual(
self.rows(added, "skipped"), [("big.png", "Too large to verify")]
)
self.assertEqual(self.tagger.calls, [])
class TestAutoTag(ImageTagJobTestCase):
async def test_auto_job_applies_the_requested_count_per_file_without_removing_existing( # noqa: E501
self,
):
self.png("a.png")
self.png("b.png", color=(1, 2, 3))
self.saved_set("a.png", (self.output_dir / "a.png").read_bytes(), ["keep_me"])
options = {**WD14, "suggestion_limit": 2}
job = await self.run_job(self.auto_body(["a.png", "b.png"], options=options))
self.assertEqual(job.title, "Auto-tag images")
self.assertEqual(
job.scope, {"mode": "auto", "images": 2, "model_type": "wd14-vit"}
)
self.assertEqual(job.state, "finished")
self.assertEqual(
self.api.db.read_image_tags(self.locator("a.png"), self.digest("a.png"))[
"tags"
],
["keep_me", "sandstone", "desert"],
)
self.assertEqual(self.stored("b.png")["tags"], ["sandstone", "desert"])
self.assertEqual(
job.counters, {"tagged": 2, "unchanged": 0, "skipped": 0, "errors": 0}
)
# Inference read the private captures only, never the current files.
for path in self.tagger.read_paths:
self.assertNotIn(str(self.output_dir), path)
self.assertEqual([call["limit"] for call in self.tagger.calls], [2, 2])
# limit 3: at most three names are added to a file.
self.png("c.png", color=(9, 9, 9))
options = {**WD14, "suggestion_limit": 3}
job = await self.run_job(self.auto_body(["c.png"], options=options))
self.assertEqual(self.stored("c.png")["tags"], [name for name, _ in SCORED[:3]])
self.assertEqual(self.rows(job, "tagged"), [("c.png", "Added 3 tags")])
# Omitted: the configured default.
AutotagConfig.SUGGESTION_LIMIT = 4
self.png("d.png", color=(7, 7, 7))
await self.run_job(self.auto_body(["d.png"]))
self.assertEqual(self.tagger.calls[-1]["limit"], 4)
self.assertEqual(len(self.stored("d.png")["tags"]), 4)
# Suggestions the file already carries add nothing.
job = await self.run_job(self.auto_body(["d.png"]))
self.assertEqual(self.rows(job, "unchanged"), [("d.png", "No new tags")])
async def test_auto_job_holds_gate_and_never_downloads(self):
self.png("a.png")
self.png("b.png", color=(1, 2, 3))
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
job = await self.start_ok(self.auto_body(["a.png", "b.png"]))
await self.wait_for(entered)
holder = model_gate.holder()
self.assertIsNotNone(holder)
self.assertTrue(holder.startswith("image-tag-job:"), holder)
# A legacy route cannot swap the model under the running job.
response = await self.client.post("/prompt_manager/autotag/unload")
self.assertEqual(response.status, 409)
self.assertEqual((await response.json())["code"], "model-busy")
proceed.set()
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.counters["tagged"], 2)
self.assertEqual(self.tagger.load_calls, [("wd14-vit", True)])
self.tagger.download_model.assert_not_called()
self.assertIsNone(model_gate.holder())
async def test_model_unavailable_fails_the_job_before_the_first_file(self):
self.png("a.png")
self.tagger.downloaded["wd14-vit"] = False
job = await self.run_job(self.auto_body(["a.png"]))
self.assertEqual(job.state, "failed")
self.assertEqual(job.processed, 0)
self.assertEqual(job.counters, dict.fromkeys(GROUPS, 0))
self.assertTrue(
any(
line.endswith(
"is not downloaded. Download it from the legacy Auto-tag page."
)
for line in job.log
),
job.log,
)
self.assertEqual(self.tagger.load_calls, [])
self.tagger.download_model.assert_not_called()
self.assertIsNone(model_gate.holder())
self.assertEqual(self.api._capture_store().stats()["captures"], 0)
async def test_model_load_failure_fails_the_job_and_frees_the_gate(self):
self.png("a.png")
self.tagger.load_error = RuntimeError("/secret/models/wd14 exploded")
job = await self.run_job(self.auto_body(["a.png"]))
self.assertEqual(job.state, "failed")
self.assertEqual(job.processed, 0)
self.assertNotIn("/secret", json.dumps(job.detail()))
self.assertIsNone(model_gate.holder())
async def test_inference_failure_is_an_error_row_and_the_job_goes_on(self):
self.png("a.png")
self.png("b.png", color=(1, 2, 3))
failing = {"first": True}
def explode(_path):
if failing.pop("first", False):
raise RuntimeError("/secret/path boom")
self.tagger.on_generate = explode
job = await self.run_job(self.auto_body(["a.png", "b.png"]))
self.assertEqual(job.counters["errors"], 1)
self.assertEqual(job.counters["tagged"], 1)
self.assertEqual(
self.rows(job, "errors"),
[("a.png", "Auto-tagging failed. Check server logs for details.")],
)
self.assertNotIn("/secret", json.dumps(job.report()))
self.assertIsNone(model_gate.holder())
async def test_changed_file_is_skipped_not_tagged(self):
self.png("a.png")
before = self.digest("a.png")
def rewrite(_capture_path):
make_png(self.output_dir / "a.png", size=(8, 6), color=(1, 1, 1))
self.tagger.on_generate = rewrite
job = await self.run_job(self.auto_body(["a.png"]))
self.assertEqual(self.rows(job, "skipped"), [("a.png", "File changed")])
self.assertEqual(job.counters["tagged"], 0)
locator = self.locator("a.png")
self.assertEqual(self.api.db.read_image_tags(locator, before)["tags"], [])
self.assertEqual(self.stored("a.png")["tags"], [])
async def test_captures_released_and_handles_closed(self):
try:
from open_handles import assert_closed
except ImportError:
from tests.open_handles import assert_closed
for index in range(3):
self.png(f"f{index}.png", color=(index, 0, 0))
await self.run_job(self.auto_body([f"f{index}.png" for index in range(3)]))
store = self.api._capture_store()
self.assertEqual(store.stats()["captures"], 0)
assert_closed(self, store.directory)
assert_closed(self, str(self.output_dir))
class TestStartAndExclusion(ImageTagJobTestCase):
async def test_start_returns_model_busy_or_job_busy(self):
self.png("a.png")
other = model_gate.new_owner("legacy-batch")
self.assertTrue(model_gate.try_acquire(other))
try:
status, body = await self.start(self.auto_body(["a.png"]))
self.assertEqual((status, body["code"]), (409, "model-busy"))
self.assertEqual(self.api._jobs.list(), [])
# Add tags needs no model: it starts while the gate is held.
job = await self.run_job(self.add_body(["a.png"], ["x"]))
self.assertEqual(job.state, "finished")
finally:
model_gate.release(other)
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
job = await self.start_ok(self.auto_body(["a.png"]))
await self.wait_for(entered)
try:
for body in (self.add_body(["a.png"], ["y"]), self.auto_body(["a.png"])):
resp = await self.client.post(START, json=body)
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "busy"))
self.assertEqual(
body["running"], {"id": job.id, "title": "Auto-tag images"}
)
finally:
proceed.set()
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
# A refused start never leaves the gate taken.
self.assertIsNone(model_gate.holder())
async def test_image_tag_job_refused_while_legacy_restore_holds(self):
self.png("a.png")
async with self.api._jobs.hold(
"restore", "Database restore (admin page)", admission="records"
):
status, body = await self.start(self.add_body(["a.png"], ["x"]))
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["title"], "Database restore (admin page)")
status, body = await self.start(self.auto_body(["a.png"]))
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertIsNone(model_gate.holder())
async with self.api._jobs.hold(
"thumbnails", "Thumbnail generation (gallery page)", admission="guarded"
):
status, body = await self.start(self.add_body(["a.png"], ["x"]))
self.assertEqual((status, body["code"]), (409, "busy"))
async def test_record_changing_excludes_running_scan(self):
self.png("a.png")
gate = asyncio.Event()
async def scan_runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan",
"Scan for metadata",
record_changing=True,
scope={"files": "new"},
runner=scan_runner,
)
try:
status, body = await self.start(self.add_body(["a.png"], ["x"]))
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["title"], "Scan for metadata")
status, body = await self.start(self.auto_body(["a.png"]))
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertIsNone(model_gate.holder())
finally:
gate.set()
await scan.task
# The other way round: a running tagging job keeps the scan out.
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
job = await self.start_ok(self.auto_body(["a.png"]))
await self.wait_for(entered)
try:
resp = await self.client.post("/prompt_manager/jobs/scan", json={})
self.assertEqual(resp.status, 409)
self.assertEqual((await resp.json())["running"]["id"], job.id)
finally:
proceed.set()
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
async def test_a_start_that_loses_the_slot_race_frees_the_gate(self):
self.png("a.png")
gate = asyncio.Event()
async def scan_runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan",
"Scan for metadata",
record_changing=True,
scope={},
runner=scan_runner,
)
try:
# The pre-check saw a free slot; registry.start still refuses.
with patch.object(self.api._jobs, "blocker", side_effect=[None, scan]):
status, body = await self.start(self.auto_body(["a.png"]))
self.assertEqual((status, body["code"]), (409, "busy"))
self.assertIsNone(model_gate.holder())
finally:
gate.set()
await scan.task
async def test_cancel_stops_at_file_boundary_and_keeps_written_files(self):
for index in range(3):
self.png(f"f{index}.png", color=(index, 5, 5))
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
job = await self.start_ok(self.auto_body([f"f{i}.png" for i in range(3)]))
await self.wait_for(entered)
response = await self.client.post(f"/prompt_manager/jobs/{job.id}/cancel")
self.assertTrue((await response.json())["cancellation_requested"])
proceed.set()
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.state, "cancelled")
self.assertEqual(job.counters["tagged"], 1)
self.assertEqual(job.processed, 1)
self.assertEqual(len(self.stored("f0.png")["tags"]), 10)
self.assertEqual(self.stored("f1.png")["tags"], [])
self.assertEqual(len(self.tagger.calls), 1)
self.assertTrue(job.log[-1].endswith("Auto-tag cancelled · 1 file processed"))
self.assertIsNone(model_gate.holder())
self.assertEqual(self.api._capture_store().stats()["captures"], 0)
async def test_stale_root_scope_refuses_the_start(self):
self.png("a.png")
body = self.add_body(["a.png"], ["x"])
body["targets"][0]["root_scope"] = "0" * 64
status, data = await self.start(body)
self.assertEqual((status, data["code"]), (409, "source-changed"))
self.assertEqual(self.api._jobs.list(), [])
async def test_restore_during_the_job_stops_it_without_writes(self):
self.png("a.png")
self.png("b.png", color=(1, 2, 3))
def rotate(_path):
conn = sqlite3.connect(self._temp_db.name)
try:
rotate_history_epoch(conn)
finally:
conn.close()
self.tagger.on_generate = rotate
job = await self.run_job(self.auto_body(["a.png", "b.png"]))
self.assertEqual(job.state, "failed")
self.assertEqual(job.counters["tagged"], 0)
self.assertEqual(
self.rows(job, "errors"),
[("a.png", "Prompt storage was restored; start the job again.")],
)
self.assertEqual(self.stored("a.png")["tags"], [])
self.assertEqual(len(self.tagger.calls), 1)
self.assertIsNone(model_gate.holder())
async def test_invalid_bodies_rejected(self):
self.png("a.png")
target = self.target("a.png")
bad_bodies = [
{"mode": "add", "targets": [], "tags": ["x"]},
{"mode": "add", "targets": [target] * 501, "tags": ["x"]},
{"mode": "add", "targets": "a.png", "tags": ["x"]},
{"mode": "add", "targets": [{**target, "path": "../a.png"}], "tags": ["x"]},
{"mode": "add", "targets": [{**target, "root_index": True}], "tags": ["x"]},
{"mode": "add", "targets": [{**target, "extra": 1}], "tags": ["x"]},
{"mode": "add", "targets": [{**target, "path": "a.txt"}], "tags": ["x"]},
{"mode": "add", "targets": [target], "tags": []},
{"mode": "add", "targets": [target], "tags": "x"},
{"mode": "add", "targets": [target], "tags": ["bad\nname"]},
{"mode": "add", "targets": [target]},
{"mode": "add", "targets": [target], "tags": ["x"], "extra": 1},
{"mode": "add", "targets": [target], "tags": ["x"], "model_type": "hf"},
{"mode": "remove", "targets": [target], "tags": ["x"]},
{"targets": [target], "tags": ["x"]},
{"mode": "auto", "targets": [target], "model_type": "wd14-vit"},
{"mode": "auto", "targets": [target], "model_type": "x", "options": {}},
{
"mode": "auto",
"targets": [target],
"model_type": "wd14-vit",
"options": {},
},
{
"mode": "auto",
"targets": [target],
"model_type": "wd14-vit",
"options": {**WD14, "general_threshold": 2},
},
{
"mode": "auto",
"targets": [target],
"model_type": "wd14-vit",
"options": {**WD14, "suggestion_limit": 0},
},
{
"mode": "auto",
"targets": [target],
"model_type": "gguf",
"options": dict(WD14),
},
{
"mode": "auto",
"targets": [target],
"model_type": "wd14-vit",
"options": dict(WD14),
"tags": ["x"],
},
]
for body in bad_bodies:
with self.subTest(body=json.dumps(body)[:120]):
status, data = await self.start(body)
self.assertIn(status, (400, 403), data)
self.assertFalse(data["success"])
too_many = {
"mode": "add",
"targets": [target],
"tags": [f"t{i}" for i in range(101)],
}
status, data = await self.start(too_many)
self.assertEqual((status, data["code"]), (400, "tag-limit"))
response = await self.client.post(START, data="not json")
self.assertEqual(response.status, 400)
self.assertEqual(self.api._jobs.list(), [])
self.assertIsNone(model_gate.holder())
async def test_duplicate_targets_are_tagged_once(self):
self.png("a.png")
job = await self.run_job(self.add_body(["a.png", "a.png"], ["x"]))
self.assertEqual(job.scope["images"], 1)
self.assertEqual(job.counters["tagged"], 1)
async def test_rows_and_report_carry_no_absolute_paths(self):
self.png("sub/a.png")
self.png("sub/b.png", color=(3, 3, 3))
job = await self.run_job(
self.auto_body(["sub/a.png", "sub/b.png", "sub/missing.png"])
)
for url in (
START.rsplit("/", 1)[0],
f"/prompt_manager/jobs/{job.id}",
f"/prompt_manager/jobs/{job.id}/report",
):
response = await self.client.get(url)
self.assertEqual(response.status, 200)
body = json.loads(await response.text())
self.assert_no_paths(body)
detail = job.detail()
self.assertEqual(
[row["path"] for row in detail["outcomes"]["tagged"]],
["sub/a.png", "sub/b.png"],
)
self.assertTrue(
all(row["root"] == "output" for row in detail["outcomes"]["tagged"])
)
self.assertTrue(any("output/sub/a.png" in line for line in detail["log"]))
class TestFixRound(ImageTagJobTestCase):
"""The final review's fixes: estimate, body bound, busy captures, roots."""
async def test_status_line_estimates_minutes_left_after_the_minimum(self):
for index in range(3):
self.png(f"f{index}.png", color=(index, 7, 7))
clock = iter([0.0, 0.0, 120.0, 240.0])
lines = []
self.tagger.on_generate = lambda _path: lines.append(
self.api._jobs.list()[0].status_line
)
with patch(
"py.api.image_tag_jobs._monotonic", side_effect=lambda: next(clock, 240.0)
):
await self.run_job(self.auto_body([f"f{i}.png" for i in range(3)]))
# Two files make the minimum (the scan's MIN_BATCHES_FOR_ESTIMATE); then
# 120 s per file leaves about 2 min for the third.
self.assertEqual(
lines,
[
"Processing file 1 of 3",
"Processing file 2 of 3",
"Processing file 3 of 3 · about 2 min left",
],
)
def maximal_target(self, number):
"""A distinct target whose path is 1024 non-ASCII code points long."""
head = f"{number:03d}/" + ("界" * 200 + "/") * 5
path = head + "界" * (1024 - len(head) - 4) + ".png"
self.assertEqual(len(path), 1024)
return self.target(path)
async def test_start_body_bounds(self):
from py.api import image_tag_jobs
body = {
"mode": "add",
"targets": [self.maximal_target(n) for n in range(500)],
"tags": ["x"],
}
# Escaped as \uXXXX (6 bytes a code point), far over the old 1 MiB.
payload = json.dumps(body).encode("utf-8")
self.assertGreater(len(payload), 3 * 1000 * 1000)
response = await self.client.post(
START, data=payload, headers={"Content-Type": "application/json"}
)
data = await self.json_of(response)
self.assertEqual(response.status, 202, data)
self.assertEqual(data["job"]["scope"]["images"], 500)
job = self.api._jobs.get(data["job"]["id"])
await self.client.post(f"/prompt_manager/jobs/{job.id}/cancel")
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
cap = image_tag_jobs.MAX_JOB_BODY_BYTES
self.assertGreaterEqual(cap, len(payload))
over = b'{"mode":"add","targets":[],"tags":["x"]'
over += b" " * (cap + 1 - len(over) - 1) + b"}"
self.assertEqual(len(over), cap + 1)
response = await self.client.post(
START, data=over, headers={"Content-Type": "application/json"}
)
self.assertEqual(response.status, 413)
def saturate(self, store):
with store._lock:
store._workers += store.max_workers
self.addCleanup(self.unsaturate, store)
def unsaturate(self, store):
with store._lock:
if store._workers >= store.max_workers:
store._workers -= store.max_workers
async def test_busy_capture_workers_are_retried_then_left_for_a_rerun(self):
self.png("a.png")
self.png("b.png", color=(1, 2, 3))
store = self.api._capture_store()
self.saturate(store)
with patch("py.api.image_tag_jobs.CAPACITY_BACKOFF", (0.001, 0.001)):
added = await self.run_job(self.add_body(["a.png"], ["x"]))
auto = await self.run_job(self.auto_body(["b.png"]))
retry = "The file could not be read now. Run the job again for it."
for job, path in ((added, "a.png"), (auto, "b.png")):
self.assertEqual(job.state, "finished")
self.assertEqual(self.rows(job, "skipped"), [(path, retry)])
self.assertEqual(job.counters["errors"], 0)
self.assertNotIn("Tag dialog", json.dumps(job.detail()))
self.assertEqual(self.tagger.calls, [])
self.assertEqual(self.stored("a.png")["tags"], [])
async def test_a_busy_capture_worker_that_frees_up_is_tagged(self):
self.png("a.png")
store = self.api._capture_store()
real = store.verify
tries = []
def verify(root, relative):
tries.append(relative)
if len(tries) < 3:
raise CaptureCapacity("Too many captures are in progress")
return real(root, relative)
with (
patch.object(store, "verify", side_effect=verify),
patch("py.api.image_tag_jobs.CAPACITY_BACKOFF", (0.001, 0.001, 0.001)),
):
job = await self.run_job(self.add_body(["a.png"], ["x"]))
self.assertEqual(len(tries), 3)
self.assertEqual(self.rows(job, "tagged"), [("a.png", "Added 1 tag")])
self.assertEqual(self.stored("a.png")["tags"], ["x"])
async def test_output_roots_are_read_once_per_job_off_the_event_loop(self):
for index in range(3):
self.png(f"f{index}.png", color=(index, 1, 1))
body = self.add_body([f"f{i}.png" for i in range(3)], ["x"])
original = self.api._get_all_output_dirs
threads = []
def lookup():
threads.append(threading.current_thread())
return original()
with patch.object(self.api, "_get_all_output_dirs", side_effect=lookup):
job = await self.run_job(body)
self.assertEqual(job.counters["tagged"], 3)
# One read when the start checks the scopes, one for the whole run.
self.assertEqual(len(threads), 2)
self.assertNotIn(threading.main_thread(), threads)
self.assertEqual(job.outcomes["tagged"][0]["root"], "output")
if __name__ == "__main__":
unittest.main()
+72 -5
View File
@@ -12,6 +12,7 @@ import json
import os
import sys
import tempfile
import threading
import types
import unittest
import unittest.mock as mock
@@ -368,10 +369,29 @@ class TestConsoleBanner(unittest.TestCase):
self.assertIn("Loaded:", banner)
class PriorFileHandler(logging.FileHandler):
"""A caller-owned sink that stays closed once its owner closes it.
``FileHandler.emit`` reopens the file when ``stream`` is None, so a record
that selected this handler before ``close()`` would reopen prior.log after
it, and Windows then refuses the temp directory cleanup (WinError 32).
``Handler.handle`` and ``FileHandler.close`` hold the same handler lock, so
checking ``_closed`` in ``emit`` is race free (as in the product's
``_OwnedRotatingFileHandler``).
"""
def emit(self, record):
if not self._closed:
super().emit(record)
class TestPackageLoggerIsolation(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
# Runs after restore_logging, before the cleanup Windows refuses while
# prior.log is open; /proc makes Linux fail the same way.
self.addCleanup(self.assert_tmp_has_no_open_handles)
self.logger = logging.getLogger("prompt_manager")
self.previous = (
self.logger.handlers[:],
@@ -381,7 +401,7 @@ class TestPackageLoggerIsolation(unittest.TestCase):
)
self.addCleanup(self.restore_logging)
self.path = os.path.join(self.tmp.name, "prior.log")
self.handler = logging.FileHandler(self.path)
self.handler = PriorFileHandler(self.path)
self.logger.handlers = [self.handler]
self.logger.setLevel(logging.DEBUG)
self.logger.propagate = True
@@ -398,16 +418,20 @@ class TestPackageLoggerIsolation(unittest.TestCase):
self.server.PromptServer.instance.prompt_queue.task_done = lambda *a, **k: None
def restore_logging(self):
for handler in self.logger.handlers:
if handler not in self.previous[0]:
handler.close()
self.handler.close()
# Detach before closing, so new records stop selecting these handlers.
retired = [h for h in self.logger.handlers if h not in self.previous[0]]
(
self.logger.handlers,
self.logger.level,
self.logger.propagate,
self.logger.disabled,
) = self.previous
for handler in retired:
handler.close()
self.handler.close()
def assert_tmp_has_no_open_handles(self):
self.assertEqual(open_handles_under(self.tmp.name), [])
def assert_logging_restored(self, handlers, registration):
for key, module in self.saved_modules.items():
@@ -456,6 +480,49 @@ class TestPackageLoggerIsolation(unittest.TestCase):
self.assertEqual(open_at_disposal, [True])
self.assert_logging_restored(handlers, registration)
def test_record_in_flight_at_restore_cannot_reopen_prior_log(self):
"""A record that selected prior.log before restore never reopens it.
Leaked daemon threads of other tests (image workers, watchdog) log
under ``prompt_manager.*``. One that reaches the prior handler while
restore_logging closes it would, with a plain FileHandler, reopen the
file on emit and keep it open through TemporaryDirectory.cleanup().
"""
message = "record selected before restore_logging"
selected = threading.Event()
resume = threading.Event()
errors = []
def pause_before_handler_lock(record):
if record.getMessage() == message:
selected.set()
if not resume.wait(5):
raise RuntimeError("logging thread was not released")
return True
def log_from_background_thread():
try:
logging.getLogger("prompt_manager.late_emitter").warning(message)
except BaseException as error:
errors.append(error)
self.handler.addFilter(pause_before_handler_lock)
worker = threading.Thread(target=log_from_background_thread, daemon=True)
worker.start()
try:
self.assertTrue(selected.wait(5), "record did not select prior.log")
self.restore_logging()
self.assertIsNone(self.handler.stream)
finally:
resume.set()
worker.join(5)
self.assertFalse(worker.is_alive(), "logging thread did not finish")
self.assertEqual(errors, [])
self.assertIsNone(self.handler.stream)
self.assert_tmp_has_no_open_handles()
with open(self.path, encoding="utf-8") as log:
self.assertNotIn(message, log.read())
def test_failed_package_import_closes_handlers_and_restores_logging(self):
name = "prompt_manager_pkg_logger_failure"
captured = {}
+552
View File
@@ -0,0 +1,552 @@
"""Registry admission classes and the legacy routes that hold the registry.
Record-changing work never runs together (Activity design): a ``records``
job is refused while any ``records`` or ``guarded`` job runs, a ``guarded``
job (thumbnails) while a ``records`` job runs, and ``read`` jobs only share
the one-per-kind rule. The legacy admin and gallery routes that change
records or files take a registry hold for their duration, so a workspace job
and a legacy-page operation refuse each other, naming the one that runs.
Fake runners and gated executor calls only; ordering is driven by events.
"""
import asyncio
import hashlib
import json
import os
import sys
import threading
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
_mock_server = MagicMock()
_mock_server.PromptServer.instance.routes = MagicMock()
sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.api.jobs import ( # noqa: E402
ADMISSIONS,
Job,
JobBusy,
JobRegistry,
)
try:
from tests.test_api_admin import AdminAPITestCase # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase # noqa: E402
def _gated(gate):
async def runner(job):
await gate.wait()
return runner
async def _noop(job):
return None
class TestAdmissionClasses(unittest.IsolatedAsyncioTestCase):
def _start(self, registry, kind, admission, gate=None, **kwargs):
return registry.start(
kind,
kind.title(),
admission=admission,
scope={},
runner=_gated(gate) if gate is not None else _noop,
**kwargs,
)
async def test_admission_names_and_explicit_read(self):
self.assertEqual(ADMISSIONS, ("read", "guarded", "records"))
registry = JobRegistry()
job = registry.start(
"report", "Report", admission="read", scope={}, runner=_noop
)
await asyncio.wait_for(job.task, 2)
self.assertEqual(job.admission, "read")
self.assertFalse(job.record_changing)
with self.assertRaises(ValueError):
registry.start("x", "X", admission="exclusive", scope={}, runner=_noop)
with self.assertRaises(ValueError):
registry.start(
"x",
"X",
admission="read",
record_changing=True,
scope={},
runner=_noop,
)
async def test_start_requires_an_admission_class(self):
# A job that forgets its class must not silently run as "read" beside
# a scan or restore: every start call decides.
registry = JobRegistry()
with self.assertRaises(TypeError) as raised:
registry.start("review", "Review", scope={}, runner=_noop)
self.assertIn("admission", str(raised.exception))
self.assertEqual(registry.list(), [])
job = registry.start(
"review", "Review", record_changing=False, scope={}, runner=_noop
)
await asyncio.wait_for(job.task, 2)
self.assertEqual(job.admission, "read")
async def test_record_changing_is_an_alias_for_records(self):
registry = JobRegistry()
gate = asyncio.Event()
job = registry.start(
"scan", "Scan", record_changing=True, scope={}, runner=_gated(gate)
)
self.assertEqual(job.admission, "records")
self.assertTrue(job.record_changing)
self.assertEqual(job.summary()["admission"], "records")
self.assertIs(
registry.blocker("image-tags", record_changing=True), job
) # the existing call shape keeps working
gate.set()
await asyncio.wait_for(job.task, 2)
async def test_records_job_refused_while_guarded_runs(self):
registry = JobRegistry()
gate = asyncio.Event()
thumbs = self._start(registry, "thumbnails", "guarded", gate)
with self.assertRaises(JobBusy) as raised:
self._start(registry, "scan", "records")
self.assertIs(raised.exception.running, thumbs)
self.assertIs(registry.blocker("restore", admission="records"), thumbs)
gate.set()
await asyncio.wait_for(thumbs.task, 2)
scan = self._start(registry, "scan", "records")
await asyncio.wait_for(scan.task, 2)
self.assertEqual(scan.state, "finished")
async def test_guarded_refused_while_records_runs(self):
registry = JobRegistry()
gate = asyncio.Event()
scan = self._start(registry, "scan", "records", gate)
with self.assertRaises(JobBusy) as raised:
self._start(registry, "thumbnails", "guarded")
self.assertIs(raised.exception.running, scan)
# A second guarded kind may run beside a guarded one, not beside records.
gate.set()
await asyncio.wait_for(scan.task, 2)
hold = asyncio.Event()
thumbs = self._start(registry, "thumbnails", "guarded", hold)
backup = self._start(registry, "backup", "guarded")
await asyncio.wait_for(backup.task, 2)
self.assertEqual(backup.state, "finished")
hold.set()
await asyncio.wait_for(thumbs.task, 2)
async def test_read_jobs_run_in_parallel(self):
registry = JobRegistry()
gate = asyncio.Event()
scan = self._start(registry, "scan", "records", gate)
health = self._start(registry, "health", "read", gate)
duplicates = self._start(registry, "duplicates", "read", gate)
self.assertTrue(all(j.running for j in (scan, health, duplicates)))
with self.assertRaises(JobBusy) as raised: # one job of a kind
self._start(registry, "health", "read")
self.assertIs(raised.exception.running, health)
gate.set()
await asyncio.wait_for(
asyncio.gather(scan.task, health.task, duplicates.task), 2
)
async def test_non_cancellable_job_ignores_cancel(self):
registry = JobRegistry()
gate = asyncio.Event()
job = self._start(registry, "restore", "records", gate, cancellable=False)
self.assertIs(job.summary()["cancellable"], False)
self.assertFalse(registry.cancel(job.id))
self.assertFalse(job.cancelled)
gate.set()
await asyncio.wait_for(job.task, 2)
self.assertEqual(job.state, "finished")
async def test_hold_registers_a_running_non_cancellable_job(self):
registry = JobRegistry()
held = registry.hold(
"restore", "Database restore (admin page)", admission="records"
)
async with held as job:
self.assertIs(registry.active("restore"), job)
self.assertEqual(job.state, "running")
self.assertEqual(
(job.summary()["cancellable"], job.summary()["admission"]),
(False, "records"),
)
self.assertFalse(registry.cancel(job.id))
with self.assertRaises(JobBusy) as raised:
self._start(registry, "scan", "records")
self.assertIn("Database restore (admin page)", str(raised.exception))
self.assertEqual(job.state, "finished")
self.assertIsNone(registry.active("restore"))
self.assertIs(registry.list()[0], job)
async def test_hold_refused_without_waiting(self):
registry = JobRegistry()
gate = asyncio.Event()
scan = self._start(registry, "scan", "records", gate)
with self.assertRaises(JobBusy) as raised:
async with registry.hold("maintenance", "Maintenance", admission="records"):
self.fail("a refused hold must not run its body")
self.assertIs(raised.exception.running, scan)
self.assertEqual([j.kind for j in registry.list()], ["scan"])
gate.set()
await asyncio.wait_for(scan.task, 2)
async def test_hold_released_on_exception(self):
registry = JobRegistry()
with self.assertRaises(RuntimeError):
async with registry.hold(
"maintenance", "Maintenance", admission="records"
) as job:
raise RuntimeError("boom")
self.assertEqual(job.state, "failed")
again = registry.hold("maintenance", "Maintenance", admission="records")
async with again as second:
second.finish("failed") # the body may end it itself
self.assertEqual(second.state, "failed")
async def test_hold_admits_on_entry_not_at_the_call(self):
# A hold created but never entered (an await or an exception between
# the call and ``async with``) must not leave a running records job
# behind that blocks all record work until a restart.
registry = JobRegistry()
held = registry.hold("restore", "Restore", admission="records")
self.assertEqual(registry.list(), [])
self.assertIsNone(registry.blocker("scan", admission="records"))
del held
async with registry.hold("restore", "Restore", admission="records") as job:
self.assertIs(registry.active("restore"), job)
self.assertEqual(job.state, "finished")
def test_job_defaults_stay_read_and_cancellable(self):
job = Job("duplicates", "Duplicate scan")
self.assertEqual((job.admission, job.cancellable), ("read", True))
class _LegacyHoldCase(AdminAPITestCase):
"""The admin test app plus a running workspace job on demand."""
def _run_job(self, kind="scan", title="Scan for metadata", admission="records"):
gate = asyncio.Event()
job = self.api._jobs.start(
kind, title, admission=admission, scope={}, runner=_gated(gate)
)
self.addAsyncCleanup(self._release, gate, job)
return gate, job
@staticmethod
async def _release(gate, job):
gate.set()
await asyncio.wait_for(job.task, 5)
def _db_digest(self):
path = Path(self.api.db.model.db_path)
return {
suffix: hashlib.sha256(Path(f"{path}{suffix}").read_bytes()).hexdigest()
for suffix in ("", "-wal")
if Path(f"{path}{suffix}").exists()
}
async def _busy(self, resp, title="Scan for metadata"):
body = await resp.json()
self.assertEqual(resp.status, 409)
self.assertEqual(body, {"success": False, "error": f"{title} already running"})
class TestLegacyRoutesHoldTheRegistry(_LegacyHoldCase):
async def test_hold_is_listed_as_running_job(self):
held = self.api._jobs.hold(
"restore", "Database restore (admin page)", admission="records"
)
async with held:
resp = await self.client.request("GET", "/prompt_manager/jobs")
jobs = (await resp.json())["jobs"]
self.assertEqual(len(jobs), 1)
job = jobs[0]
self.assertEqual(
(job["kind"], job["title"], job["state"]),
("restore", "Database restore (admin page)", "running"),
)
self.assertIs(job["cancellable"], False)
self.assertEqual(job["admission"], "records")
async def test_legacy_maintenance_and_cleanup_refused_while_scan_runs(self):
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
before = self._db_digest()
self._run_job()
resp = await self.client.request(
"POST",
"/prompt_manager/maintenance",
json={"operations": ["cleanup_duplicates", "vacuum"]},
)
await self._busy(resp)
resp = await self.client.request("POST", "/prompt_manager/cleanup")
await self._busy(resp)
self.assertEqual(self._db_digest(), before)
async def test_read_only_maintenance_runs_during_scan(self):
# The legacy statistics and consistency checks only read: they hold
# the registry as "read", so a long scan does not block them.
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
_gate, scan = self._run_job()
listed = []
real = self.api._run_maintenance_operations
def observed(operations):
listed.extend(
(job.kind, job.admission, job.state) for job in self.api._jobs.list()
)
return real(operations)
self.api._run_maintenance_operations = observed
resp = await self.client.request(
"POST",
"/prompt_manager/maintenance",
json={
"operations": [
"statistics",
"check_consistency",
"check_hash_duplicates",
]
},
)
body = await resp.json()
self.assertEqual(resp.status, 200)
self.assertIs(body["success"], True)
self.assertEqual(
set(body["results"]),
{"statistics", "check_consistency", "check_hash_duplicates"},
)
self.assertIn(("maintenance", "read", "running"), listed)
self.assertEqual(scan.state, "running")
maintenance = self.api._jobs.list()[-1]
self.assertEqual(
(maintenance.kind, maintenance.admission, maintenance.state),
("maintenance", "read", "finished"),
)
async def test_mixed_maintenance_selection_is_refused_during_scan(self):
# One write operation among read-only ones makes the run "records".
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
before = self._db_digest()
self._run_job()
resp = await self.client.request(
"POST",
"/prompt_manager/maintenance",
json={"operations": ["statistics", "vacuum"]},
)
await self._busy(resp)
self.assertEqual(self._db_digest(), before)
async def test_default_maintenance_selection_is_refused_during_scan(self):
# No "operations" key means the default list, which vacuums.
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
before = self._db_digest()
self._run_job()
resp = await self.client.request("POST", "/prompt_manager/maintenance", json={})
await self._busy(resp)
self.assertEqual(self._db_digest(), before)
self.assertEqual([j.kind for j in self.api._jobs.list()], ["scan"])
async def test_legacy_maintenance_refused_while_image_tags_runs(self):
self._run_job("image-tags", "Add tags to 3 images")
resp = await self.client.request("POST", "/prompt_manager/maintenance")
await self._busy(resp, "Add tags to 3 images")
async def test_scan_refused_while_legacy_maintenance_holds(self):
entered = threading.Event()
release = threading.Event()
real = self.api._run_maintenance_operations
def gated(operations):
entered.set()
release.wait(10)
return real(operations)
self.api._run_maintenance_operations = gated
loop = asyncio.get_running_loop()
maintenance = asyncio.ensure_future(
self.client.request(
"POST", "/prompt_manager/maintenance", json={"operations": ["vacuum"]}
)
)
self.assertTrue(await loop.run_in_executor(None, entered.wait, 10))
resp = await self.client.request("POST", "/prompt_manager/jobs/scan")
body = await resp.json()
self.assertEqual((resp.status, body["code"]), (409, "busy"))
self.assertEqual(body["running"]["title"], "Maintenance (admin page)")
release.set()
self.assertEqual((await maintenance).status, 200)
listed = (
await (await self.client.request("GET", "/prompt_manager/jobs")).json()
)["jobs"]
self.assertEqual(
[(j["kind"], j["state"]) for j in listed], [("maintenance", "finished")]
)
async def test_legacy_duplicate_delete_refused_while_scan_runs(self):
victim = self.output_dir / "dup.png"
victim.write_bytes(b"x")
self._run_job()
resp = await self.client.request(
"POST",
"/prompt_manager/delete_duplicate_images",
json={"image_paths": [str(victim)]},
)
await self._busy(resp)
self.assertTrue(victim.exists())
# The duplicate family lock was not taken by the refused request.
self.assertIsNotNone(self.api._acquire_long_job(self.api.LONG_JOB_DUPLICATES))
async def test_lora_scan_refused_while_scan_runs(self):
from py.config import IntegrationConfig
lm = Path(self.tmpdir) / "custom_nodes" / "ComfyUI-Lora-Manager"
(lm / "py").mkdir(parents=True)
self._run_job()
imported = []
with (
patch.multiple(
IntegrationConfig, LORA_MANAGER_ENABLED=True, LORA_MANAGER_PATH=str(lm)
),
patch("py.lora_utils.resolve_lora_manager_path", return_value=str(lm)),
patch.object(
self.api, "_lora_scan_stream", side_effect=lambda *a: imported.append(a)
),
):
resp = await self.client.request("POST", "/prompt_manager/lora/scan")
text = await resp.text()
self.assertEqual(resp.status, 200) # the legacy page reads a stream
events = [json.loads(line[6:]) for line in text.split("\n\n") if line]
# The admin page's modal shows ``status`` and closes on ``type: error``.
message = "Scan for metadata already running"
self.assertEqual(
events,
[{"type": "error", "status": message, "progress": 0, "message": message}],
)
self.assertEqual(imported, [])
async def test_legacy_autotag_refused_while_scan_runs(self):
# Both pages read this route as a stream (a 409 reads as a lost
# connection there), so a refusal by another job is one error frame:
# the admin page keys on ``type``, the gallery on ``status``.
self._run_job()
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start", json={}
)
text = await resp.text()
self.assertEqual(resp.status, 200)
self.assertEqual(resp.headers["Content-Type"], "text/event-stream")
events = [json.loads(line[6:]) for line in text.split("\n\n") if line]
message = "Scan for metadata already running"
self.assertEqual(
events, [{"type": "error", "status": "error", "message": message}]
)
self.assertEqual(
[j.kind for j in self.api._jobs.list() if j.kind == "autotag"], []
)
async def test_second_legacy_autotag_batch_keeps_its_json_409(self):
from py.api.autotag_routes import _BUSY_ERROR
self._run_job("autotag", "Auto-tag images (legacy page)")
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start", json={}
)
self.assertEqual(resp.status, 409)
self.assertEqual(await resp.json(), {"success": False, "error": _BUSY_ERROR})
async def test_legacy_payloads_unchanged_when_idle(self):
resp = await self.client.request(
"POST", "/prompt_manager/maintenance", json={"operations": ["vacuum"]}
)
body = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(
set(body),
{"success", "operations_completed", "all_successful", "results", "message"},
)
resp = await self.client.request("POST", "/prompt_manager/cleanup")
body = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(set(body), {"success", "message", "duplicates_removed"})
resp = await self.client.request(
"POST", "/prompt_manager/delete_duplicate_images", json={"image_paths": []}
)
self.assertEqual(
(resp.status, await resp.json()),
(400, {"success": False, "error": "No image paths provided"}),
)
# Each hold ended with the request; refused input never took one.
listed = await self._listed()
self.assertEqual(
[(j["kind"], j["state"]) for j in listed],
[("maintenance", "finished"), ("maintenance", "finished")],
)
self.assertEqual(
[j["title"] for j in listed],
["Prompt cleanup (admin page)", "Maintenance (admin page)"],
)
async def _listed(self):
resp = await self.client.request("GET", "/prompt_manager/jobs")
return (await resp.json())["jobs"]
class TestHeldJobStates(_LegacyHoldCase):
"""Activity lists what a held legacy operation really did (Minor 7)."""
async def _listed(self):
resp = await self.client.request("GET", "/prompt_manager/jobs")
return [(j["kind"], j["state"]) for j in (await resp.json())["jobs"]]
async def test_refused_input_takes_no_hold(self):
resp = await self.client.request(
"POST", "/prompt_manager/delete_duplicate_images", json={}
)
self.assertEqual(resp.status, 400)
resp = await self.client.request(
"POST",
"/prompt_manager/images/generate-thumbnails",
data=b"{not json",
headers={"Content-Type": "application/json"},
)
self.assertEqual(resp.status, 400)
self.assertEqual(await self._listed(), [])
async def test_thumbnail_stream_that_ends_in_error_is_failed(self):
with patch.object(self.api, "_get_all_output_dirs", return_value=[]):
resp = await self.client.request(
"POST", "/prompt_manager/images/generate-thumbnails/progress"
)
text = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn("event: error", text)
self.assertEqual(await self._listed(), [("thumbnails", "failed")])
async def test_legacy_autotag_that_ends_in_error_is_failed(self):
service = MagicMock()
service.get_models_status.return_value = {} # model not downloaded
with patch("py.autotag.get_autotag_service", return_value=service):
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start", json={}
)
text = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn('"type": "error"', text)
self.assertEqual(await self._listed(), [("autotag", "failed")])
if __name__ == "__main__":
unittest.main()
+14
View File
@@ -12,6 +12,7 @@ import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.api import jobs as jobs_module # noqa: E402
from py.api.jobs import ( # noqa: E402
DEFAULT_SCAN_SCOPE,
LOG_CAP_LINE,
@@ -181,8 +182,13 @@ class TestJobRegistry(unittest.IsolatedAsyncioTestCase):
"scope",
"listed",
"log_length",
"cancellable",
"admission",
},
)
self.assertEqual(
(summary["cancellable"], summary["admission"]), (True, "records")
)
self.assertEqual(summary["scope"], {"files": "new", "output_folders": 2})
# What the detail would list, so a view reads it only when that changed.
self.assertEqual((summary["listed"], summary["log_length"]), ({}, 0))
@@ -415,6 +421,14 @@ class TestJobOutcomesAndLog(unittest.TestCase):
SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors")
)
def test_image_tag_job_kind_and_groups(self):
# #249 D1: bulk Add tags / Auto-tag, counter-tile order.
self.assertEqual(getattr(jobs_module, "IMAGE_TAG_JOB_KIND", None), "image-tags")
self.assertEqual(
getattr(jobs_module, "IMAGE_TAG_GROUP_NAMES", None),
("tagged", "unchanged", "skipped", "errors"),
)
if __name__ == "__main__":
unittest.main()
+97
View File
@@ -1706,6 +1706,103 @@ class TestLibrary(unittest.TestCase):
self.db.list_library_images(positive, pair_id=foreign, archive="all")
class LibraryCounts(unittest.TestCase):
"""System › Overview counts and usage statistics follow Library eligibility."""
setUp = TestLibrary.setUp
tearDown = TestLibrary.tearDown
save = TestLibrary.save
state = TestLibrary.state
def test_counts_use_library_eligibility(self):
self.db.save_prompt_pair("Cat", "Blur")
archived = self.db.save_prompt_pair("Cat", "Noise")
self.db.archive_prompt_pair(archived)
titled_pair = self.db.save_prompt_pair("Dog", "Titled negative")
titled = self.db.get_prompt_pair(titled_pair)["negative_prompt_id"]
self.state(titled, "A titled parent") # zero pairs of its own: counted
legacy = self.save("Legacy")
self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "a.png"))
self.db.link_image_to_prompt(legacy, str(Path(self.temp.name) / "b.png"))
counts = self.db.library_counts()
# Cat, Dog, the titled negative and Legacy; "Blur" and "Noise" are
# negative-only.
self.assertEqual(counts["positive_prompts"], 4)
# Cat+Blur and Dog+Titled negative; the archived pair is excluded.
self.assertEqual(counts["negative_variants"], 2)
self.assertEqual(counts["images"], 2)
self.assertEqual(counts["rated_images"], 0)
listed = self.db.list_library()
self.assertEqual(counts["positive_prompts"], listed["total"])
self.assertEqual(counts["negative_variants"], listed["variant_total"])
def test_counts_of_an_empty_library_are_zero(self):
self.assertEqual(
self.db.library_counts(),
{
"positive_prompts": 0,
"negative_variants": 0,
"images": 0,
"rated_images": 0,
},
)
def test_rated_images_follow_their_linked_prompt_rating(self):
# An image carries no rating of its own: it shows its linked prompt's
# (the Images "Highest rated" sort), so "rated" counts those images.
folder = Path(self.temp.name)
rated = self.save("Rated", rating=4)
unrated = self.save("Unrated")
for name in ("a.png", "b.png", "c.png"):
self.db.link_image_to_prompt(rated, str(folder / name))
self.db.link_image_to_prompt(unrated, str(folder / "d.png"))
counts = self.db.library_counts()
self.assertEqual(counts["images"], 4)
self.assertEqual(counts["rated_images"], 3)
self.db.update_prompt_rating(unrated, 2)
self.assertEqual(self.db.library_counts()["rated_images"], 4)
def test_statistics_sum_real_run_counts(self):
a = self.save("Alpha", category="Landscapes", rating=5)
b = self.save("Beta", category="Landscapes", rating=4)
self.save("Gamma", category="Landscapes") # unrated: not in the average
c = self.save("Delta", category="Portraits", rating=3)
blank = self.save("Epsilon", category=" ")
uncategorised = self.save("Zeta")
self.db.record_prompt_use(a, times=3)
self.db.record_prompt_use(b, times=2)
self.db.record_prompt_use(c)
self.db.record_prompt_use(blank)
self.db.record_prompt_use(uncategorised, times=4)
# A negative-only prompt's runs are the pair's runs: never counted twice.
pair = self.db.save_prompt_pair("Alpha", "Only negative")
negative = self.db.get_prompt_pair(pair)["negative_prompt_id"]
with self.db.model.get_connection() as connection:
connection.execute(
"UPDATE prompts SET category='Landscapes',run_count=50 WHERE id=?",
(negative,),
)
rows = self.db.usage_by_category()
self.assertEqual(
rows,
[
{"category": "Landscapes", "runs": 5, "average_rating": 4.5},
{"category": None, "runs": 5, "average_rating": None},
{"category": "Portraits", "runs": 1, "average_rating": 3.0},
],
)
def test_counts_and_usage_write_nothing(self):
self.save("Alpha", category="A")
with self.db.model.get_connection() as connection:
before = connection.total_changes
self.db.library_counts()
self.db.usage_by_category()
with self.db.model.get_connection() as connection:
self.assertEqual(connection.total_changes, before)
self.assertFalse(connection.in_transaction)
class DisplayTitleUnits(unittest.TestCase):
"""The first-line title is cut like the Library's JavaScript title()."""
+308
View File
@@ -51,12 +51,19 @@ from utils.prompt_inspector import (
read_inspection_file,
)
from utils.prompt_inspector import inspect_prompt_evidence, unreadable_evidence
from utils.prompt_pair_resolver import INSPECTION_PROFILE
try:
from tests import inspection_parity as parity
from tests import local_resolver_cases as resolver_cases
from tests import local_resolver_selection_cases as selection_cases
from tests.open_handles import assert_closed
from tests.test_utils_video_metadata import HAVE_FFMPEG, write_video
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import inspection_parity as parity
import local_resolver_cases as resolver_cases
import local_resolver_selection_cases as selection_cases
from open_handles import assert_closed
from test_utils_video_metadata import HAVE_FFMPEG, write_video
@@ -1111,5 +1118,306 @@ class LocalProjectionParityTests(unittest.TestCase):
self.assertTrue({"parsed", "malformed", "limited"} <= seen)
def _server_evidence(records):
return inspect_prompt_evidence(records, None, profile=INSPECTION_PROFILE)
class LocalResolverParityTests(unittest.TestCase):
"""``resolveLocalEvidence`` answers as ``inspect_prompt_evidence`` (L3 Tasks 8-9).
Python is the oracle: every expected value is the server's own answer on the
same record texts. Task 8 ported one sampler's tracing; Task 9 ports
candidate selection (several samplers, the identical-branch rule of #270,
preview-only stages, detailer stages) and A1111 ``parameters``, so no
record set is left to the server any more.
"""
def setUp(self):
temporary = tempfile.TemporaryDirectory()
self.addCleanup(temporary.cleanup)
self.addCleanup(assert_closed, self, temporary.name)
self.directory = Path(temporary.name)
def _compare(self, cases):
"""Assert each record set's local evidence equal; return the server answers."""
observed = _run_oracle(
self,
[
{"name": name, "mode": "resolve", "records": records}
for name, records in cases.items()
],
)
answers = {}
for name, records in cases.items():
with self.subTest(case=name):
local = observed[name]
server = _server_evidence(records)
answers[name] = server
self.assertEqual(
_canonical_bytes(local["evidence"]), _canonical_bytes(server)
)
return answers
def test_local_roles_match_execution_and_single_chain_vectors(self):
files = {}
for name in parity.COMMITTED:
files[name] = parity.fixture_path(name)
for name in parity.TEST_TIME:
path = parity.fixture_path(name, self.directory)
parity.write_png(path, parity.build(name))
files[name] = path
observed = _run_oracle(
self,
[
{"name": name, "mode": "resolve", "path": str(path)}
for name, path in files.items()
],
)
vectors = {
vector["fixture"]: vector
for vector in json.loads(parity.VECTORS_PATH.read_text(encoding="utf-8"))
if vector["profile"] == "inspection"
}
compared = 0
for name, path in files.items():
with self.subTest(fixture=name):
local = observed[name]
try:
records = read_inspection_file(path).metadata
except MetadataReadError:
self.assertTrue(local["unreadable"])
continue
# The single-chain subset (Task 8); the rest is Task 9's test below.
if not resolver_cases.single_chain(records):
continue
server = _server_evidence(records)
self.assertEqual(
_canonical_bytes(local["evidence"]), _canonical_bytes(server)
)
# The committed vector agrees with the server read here.
expect = vectors[name]["expect"]
self.assertEqual(server["resolution"]["state"], expect["state"])
self.assertEqual(
server["resolution"]["reason_code"], expect["reason_code"]
)
compared += 1
self.assertGreaterEqual(compared, 20)
def test_local_roles_match_inspection_vectors(self):
"""Every inspection vector: local prompts, resolution and facts equal Images'.
The facts are ``extractGenerationFacts`` (with the legacy fill) on the
local payload, against the facts the committed vectors pin for the
server's response on the same file.
"""
vectors = [
vector
for vector in json.loads(parity.VECTORS_PATH.read_text(encoding="utf-8"))
if vector["profile"] == "inspection"
]
files = {}
for vector in vectors:
name = vector["fixture"]
if name in parity.TEST_TIME:
path = parity.fixture_path(name, self.directory)
parity.write_png(path, parity.build(name))
else:
path = parity.fixture_path(name)
files[name] = path
observed = _run_oracle(
self,
[
{"name": name, "mode": "resolve", "path": str(path)}
for name, path in files.items()
],
)
for vector in vectors:
name = vector["fixture"]
with self.subTest(fixture=name):
local, expect = observed[name], vector["expect"]
try:
server = _server_evidence(
read_inspection_file(files[name]).metadata
)
except MetadataReadError:
server = unreadable_evidence()
self.assertTrue(local["unreadable"])
self.assertEqual(
_canonical_bytes(local["evidence"]), _canonical_bytes(server)
)
resolution, prompts = (
local["evidence"]["resolution"],
local["evidence"]["prompts"],
)
self.assertEqual(
(
resolution["state"],
resolution["reason"],
resolution["reason_code"],
),
(expect["state"], expect["reason"], expect["reason_code"]),
)
self.assertEqual(
[
parity._role_text(
prompts[role]["status"], prompts[role]["text"]
)
for role in ("positive", "negative")
],
[expect["positive"], expect["negative"]],
)
self.assertEqual(local["facts"], expect["facts"])
self.assertEqual(len(vectors), 28)
def test_local_roles_match_selection_record_sets(self):
answers = self._compare(selection_cases.selection_cases())
reasons = {answer["resolution"]["reason"] for answer in answers.values()}
self.assertTrue(
{
"Every sampler branch uses the same prompt pair",
"Multiple sampler branches",
"Sampler stages have different prompts",
"Cyclic sampler stage ancestry",
"Expected an image link",
"Unsupported sampler stage ancestry",
"Resolution depth limit reached",
"A1111 settings footer is not unique",
"Multiple negative prompt delimiters",
"Negative prompt delimiter is absent",
"Parameters exceed the length limit",
"No A1111 parameters",
}
<= reasons
)
# A stage reaching only previews is dropped: one sampler is named.
self.assertEqual(
answers["select-preview-dropped"]["resolution"]["sampler_id"], "3"
)
def test_local_roles_match_fuzzed_selection_sets(self):
cases = {}
for seed in range(4):
cases.update(selection_cases.selection_fuzz_cases(seed, 150))
answers = self._compare(cases)
reasons = [answer["resolution"]["reason"] for answer in answers.values()]
self.assertGreater(reasons.count("Multiple sampler branches"), 20)
self.assertGreater(reasons.count("Sampler stages have different prompts"), 0)
self.assertGreater(
reasons.count("Every sampler branch uses the same prompt pair"), 0
)
self.assertGreater(reasons.count("A1111 settings footer is not unique"), 0)
def test_local_roles_match_golden_record_sets(self):
answers = self._compare(resolver_cases.golden_cases())
codes = {answer["resolution"]["reason_code"] for answer in answers.values()}
self.assertTrue(
{
"malformed",
"unsupported-node",
"runtime-value",
"composite",
"work-limit",
"depth-limit",
"text-limit",
"graph-ceiling",
"no-sampler",
None,
}
<= codes
)
sources = {
side["source"]
for answer in answers.values()
for side in answer["prompts"].values()
}
self.assertTrue(
{"execution", "graph-inferred", "zeroed-conditioning"} <= sources
)
def test_committed_resolver_vectors_are_current(self):
"""The Node unit test's committed answers are still the server's."""
committed = json.loads(resolver_cases.VECTORS_PATH.read_text(encoding="utf-8"))
self.assertEqual(
committed,
json.loads(json.dumps(resolver_cases.committed_vectors())),
"regenerate with: python3 -m tests.local_resolver_cases --write",
)
def test_local_roles_match_at_every_work_bound(self):
answers = self._compare(resolver_cases.boundary_cases())
for name in ("visits", "depth", "scanned"):
self.assertEqual(answers[f"{name}-edge"]["resolution"]["state"], "resolved")
self.assertIn(
answers[f"{name}-over"]["resolution"]["reason_code"],
("work-limit", "depth-limit"),
)
self.assertEqual(
answers["named-astral"]["resolution"]["node"]["class_type"],
"\U0001f600" * 64,
)
def test_local_json_depth_cap_at_its_boundary(self):
"""Both kinds of deep record at the browser's JSON depth cap (1,000).
Locally a record nested past the cap is invalid JSON: a prompt is
malformed and a workflow is ignored, as the server answers when
``json.loads`` raises RecursionError. The server's own limit depends on
the Python (TESTING.md, "Deep JSON"): about 990 on 3.10 and 3.11, so it
agrees past the cap there; about 10,000 on 3.12 and 3.13 and over
50,000 on 3.14, so it agrees up to the cap there and reads deeper.
"""
cases = resolver_cases.depth_cases()
observed = _run_oracle(
self,
[
{"name": name, "mode": "resolve", "records": records}
for name, _kind, _depth, records in cases
],
)
newer = sys.version_info >= (3, 12)
compared = 0
for name, kind, depth, records in cases:
with self.subTest(case=name):
local = observed[name]["evidence"]["resolution"]
within = depth <= resolver_cases.JSON_MAX_DEPTH
if kind == "prompt":
expected = (
("resolved", None) if within else ("unresolved", "malformed")
)
else:
# A workflow that is read finds the overwritten pair (R1).
expected = (
("unresolved", "malformed") if within else ("resolved", None)
)
self.assertEqual((local["state"], local["reason_code"]), expected)
server = _server_evidence(records)
if server["resolution"]["state"] == local["state"]:
self.assertEqual(
_canonical_bytes(observed[name]["evidence"]),
_canonical_bytes(server),
)
compared += 1
else:
# The documented divergence only: an older server refuses
# a little below the cap, a newer one reads past it.
self.assertEqual(within, not newer)
self.assertGreaterEqual(compared, 2)
def test_local_roles_match_fuzzed_record_sets(self):
cases = {}
for seed in range(4):
cases.update(resolver_cases.fuzz_cases(seed, 150))
answers = self._compare(cases)
states = [answer["resolution"]["state"] for answer in answers.values()]
# Coherent graphs reach real results, not only early refusals.
self.assertGreater(states.count("resolved"), len(states) // 5)
self.assertGreater(states.count("ambiguous"), 0)
self.assertGreater(states.count("unresolved"), len(states) // 10)
self.assertGreaterEqual(
len({answer["resolution"]["reason_code"] for answer in answers.values()}),
5,
)
if __name__ == "__main__":
unittest.main()
+116
View File
@@ -165,6 +165,122 @@ class TestPairScan(AdminAPITestCase):
)
self.assertEqual(self.api.db.get_recent_prompts()["prompts"], [])
def _nan_scanned(self, name, prompt_graph):
"""Scan ``prompt_graph`` as ComfyUI writes it: ``is_changed`` is NaN."""
stamped = {
node_id: {**node, "is_changed": [float("nan")]}
for node_id, node in prompt_graph.items()
}
text = json.dumps(stamped)
self.assertIn("NaN", text)
return self._scanned_text(name, text)
def _scanned_text(self, name, prompt_text):
meta = {"prompt": prompt_text}
return self.api._ingest_scanned_file(self.output_dir / name, meta)
@staticmethod
def _hires_fix_graph(positive):
"""A PromptManager prompt on a base sampler and a hires-fix sampler.
The base latent comes from a custom batch node and the hires pass from
a custom image filter: no sampler branch traces, so the scan links the
prompt the legacy pages show.
"""
return {
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "m.safetensors"},
},
"2": {
"class_type": "MyTriggerWords",
"inputs": {"trigger_words": ["1", 2]},
},
"3": {
"class_type": "PromptManager",
"inputs": {
"text": positive,
"prepend_text": ["2", 0],
"append_text": "",
"clip": ["1", 1],
},
},
"4": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "blur", "clip": ["1", 1]},
},
"5": {"class_type": "MyLatentBatch", "inputs": {"batch_size": 6}},
"6": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["5", 0],
},
},
"7": {
"class_type": "VAEDecode",
"inputs": {"samples": ["6", 0], "vae": ["1", 2]},
},
"8": {"class_type": "MyImageFilter", "inputs": {"images": ["7", 0]}},
"9": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["8", 0], "vae": ["1", 2]},
},
"14": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"denoise": 0.25,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["9", 0],
},
},
"15": {
"class_type": "VAEDecode",
"inputs": {"samples": ["14", 0], "vae": ["1", 2]},
},
"16": {"class_type": "SaveImage", "inputs": {"images": ["15", 0]}},
}
def test_hires_fix_prompt_links_with_or_without_nan_is_changed(self):
# Legacy parity: the legacy pages read NaN as null before parsing, so they
# showed this prompt; the scan left NaN-stamped files in review.
for name, scan in (("plain.png", None), ("stamped.png", "nan")):
with self.subTest(name=name):
g = self._hires_fix_graph("cat on a sofa")
if scan:
status = self._nan_scanned(name, g)
else:
status = self._scanned(name, g)
self.assertIn(status, ("added", "linked"))
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_untraced_single_sampler_links_the_legacy_prompt_with_nan(self):
g = self._hires_fix_graph("cat on a sofa")
for node_id in ("7", "8", "9", "14", "15"):
del g[node_id]
g["16"]["inputs"]["images"] = ["6", 0]
self.assertEqual(self._nan_scanned("single.png", g), "added")
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_nan_inside_prompt_text_is_kept(self):
# Only the NaN number reads as null; legacy's regex also rewrote text.
g = self._hires_fix_graph("NaN cat, value: NaN")
self.assertEqual(self._nan_scanned("text.png", g), "added")
self.assertEqual(self._texts(), ["NaN cat, value: NaN"])
def test_infinity_is_still_unreadable_like_the_legacy_pages(self):
text = json.dumps(self._hires_fix_graph("cat")).replace(
'"seed": 7', '"seed": Infinity'
)
self.assertIn(self._scanned_text("inf.png", text), ("ambiguous", "unresolved"))
self.assertEqual(self._texts(), [])
def test_raw_png_workflow_is_stored_as_object_and_repeat_is_unchanged(self):
path = self.output_dir / "workflow.png"
workflow = {"nodes": [{"id": 1, "type": "PromptManagerPair"}]}
+422
View File
@@ -0,0 +1,422 @@
"""System routes (py/api/system.py): health check job, Overview and statistics.
The health check runs off-loop as a ``read`` registry job of kind ``health``
(one at a time, beside a scan), and the API keeps its last report. The
Overview reads only real values: Library-eligible counts, indexed images, the
last health result or null, no backup until one exists, and the effective
sources and whether they were auto-detected. The missing-thumbnail count is not
part of it: the Overview reads ``/images/thumbnails/status`` on its own (its
walk of every source must not hold the other values back).
"""
import asyncio
import os
import sqlite3
import sys
import threading
from pathlib import Path
from unittest.mock import MagicMock, patch
_mock_server = MagicMock()
_mock_server.PromptServer.instance.routes = MagicMock()
sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.api.jobs import HEALTH_JOB_KIND # noqa: E402
from py.config import GalleryConfig # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
try:
from tests.test_api_admin import AdminAPITestCase # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase # noqa: E402
WAIT = 5
class SystemAPITestCase(AdminAPITestCase):
async def get_application(self):
app = await super().get_application()
# Auto-detection searches the disk; here it finds the fake ComfyUI only.
self.api._find_comfyui_output_dir = lambda: (
str(self.output_dir) if self.output_dir.is_dir() else None
)
return app
def save(self, text, **metadata):
return self.api.db.save_prompt(
text, prompt_hash=generate_prompt_hash(text), **metadata
)
def image(self, prompt_id, name, *, exists=True, folder=None):
path = Path(folder or self.output_dir) / name
if exists:
path.write_bytes(b"png")
return self.api.db.link_image_to_prompt(prompt_id, str(path))
async def json(self, method, url, status=200):
response = await self.client.request(method, url)
self.assertEqual(response.status, status, await response.text())
return await response.json()
async def restore_snapshot(self):
"""Restore the live database from a copy of itself through the legacy route."""
from contextlib import closing
from aiohttp import FormData
snapshot = os.path.join(self.tmpdir, "snapshot.db")
with closing(sqlite3.connect(self.api.db.model.db_path)) as src:
with closing(sqlite3.connect(snapshot)) as dst:
src.backup(dst)
with open(snapshot, "rb") as handle:
upload = handle.read()
form = FormData()
form.add_field("database_file", upload, filename="upload.db")
response = await self.client.request(
"POST", "/prompt_manager/restore", data=form
)
self.assertEqual(response.status, 200, await response.text())
async def run_health(self):
body = await self.json("POST", "/prompt_manager/system/health", 202)
job = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
return body["job"], job
class TestHealthRoute(SystemAPITestCase):
async def test_no_report_before_a_check(self):
body = await self.json("GET", "/prompt_manager/system/health")
self.assertEqual(body, {"success": True, "report": None})
async def test_health_check_runs_as_a_read_job_and_keeps_its_report(self):
prompt = self.save("Images")
self.image(prompt, "kept.png")
self.image(prompt, "gone.png", exists=False)
summary, job = await self.run_health()
self.assertEqual(summary["kind"], HEALTH_JOB_KIND)
self.assertEqual(summary["title"], "Health check")
self.assertEqual(summary["admission"], "read")
self.assertFalse(summary["cancellable"])
self.assertEqual(summary["counters"], {})
self.assertEqual(job.state, "finished")
self.assertEqual(job.progress, 100)
self.assertEqual(job.status_line, "Health check incomplete")
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
self.assertEqual(report["status"], "incomplete")
missing = next(c for c in report["checks"] if c["id"] == "missing_files")
self.assertEqual((missing["status"], missing["count"]), ("issues", 1))
ambiguous = next(c for c in report["checks"] if c["id"] == "ambiguous_metadata")
self.assertEqual(ambiguous["status"], "unavailable")
self.assertNotIn(self.tmpdir, str(report))
async def test_health_check_is_single_flight_and_runs_beside_a_scan(self):
release = threading.Event()
entered = threading.Event()
real = self.api.db.run_health_checks
def gated(*args, **kwargs):
entered.set()
release.wait(WAIT)
return real(*args, **kwargs)
scan_gate = asyncio.Event()
async def scanning(job):
await scan_gate.wait()
scan = self.api._jobs.start(
"scan", "Metadata scan", admission="records", runner=scanning
)
with patch.object(self.api.db, "run_health_checks", side_effect=gated):
first = await self.json("POST", "/prompt_manager/system/health", 202)
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, entered.wait, WAIT)
busy = await self.json("POST", "/prompt_manager/system/health", 409)
self.assertEqual(busy["code"], "busy")
self.assertEqual(
busy["running"], {"id": first["job"]["id"], "title": "Health check"}
)
release.set()
await asyncio.wait_for(self.api._jobs.get(first["job"]["id"]).task, WAIT)
scan_gate.set()
await asyncio.wait_for(scan.task, WAIT)
async def test_failed_check_keeps_the_previous_report(self):
await self.run_health()
before = (await self.json("GET", "/prompt_manager/system/health"))["report"]
with (
patch.object(
self.api.db, "run_health_checks", side_effect=RuntimeError("boom")
),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("POST", "/prompt_manager/system/health", 202)
job = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.state, "failed")
after = (await self.json("GET", "/prompt_manager/system/health"))["report"]
self.assertEqual(after, before)
async def test_absent_configured_source_is_reported_per_source(self):
prompt = self.save("Images")
self.image(prompt, "gone.png", exists=False)
absent = self.comfy_dir / "output" / "unmounted"
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.output_dir),
str(absent),
]
await self.run_health()
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
missing = next(c for c in report["checks"] if c["id"] == "missing_files")
# Per source: the readable output is counted, the absent one is not.
self.assertEqual(missing["status"], "issues")
self.assertEqual(missing["count"], 1)
self.assertTrue(missing["partial"])
self.assertEqual(
[(s["status"], s["count"]) for s in missing["sources"]],
[("checked", 1), ("unavailable", None)],
)
self.assertEqual(report["status"], "incomplete")
async def test_restore_clears_health_report(self):
self.save("Any")
await self.run_health()
self.assertIsNotNone(
(await self.json("GET", "/prompt_manager/system/health"))["report"]
)
await self.restore_snapshot()
body = await self.json("GET", "/prompt_manager/system/health")
self.assertEqual(body, {"success": True, "report": None})
overview = await self.json("GET", "/prompt_manager/system/overview")
self.assertIsNone(overview["health"])
async def test_check_finishing_after_a_restore_is_not_kept(self):
# A check that started on the old database describes it, not the new one.
self.save("Any")
release = threading.Event()
entered = threading.Event()
real = self.api.db.run_health_checks
def gated(*args, **kwargs):
entered.set()
release.wait(WAIT)
return real(*args, **kwargs)
with patch.object(self.api.db, "run_health_checks", side_effect=gated):
body = await self.json("POST", "/prompt_manager/system/health", 202)
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, entered.wait, WAIT)
await self.restore_snapshot()
release.set()
job = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.state, "finished")
body = await self.json("GET", "/prompt_manager/system/health")
self.assertEqual(body, {"success": True, "report": None})
# A check started on the restored database is kept again.
await self.run_health()
self.assertIsNotNone(
(await self.json("GET", "/prompt_manager/system/health"))["report"]
)
async def test_health_sources_resolve_off_the_event_loop(self):
loop_thread = threading.get_ident()
threads = []
real = self.api._health_roots
def recording():
threads.append(threading.get_ident())
return real()
with patch.object(self.api, "_health_roots", side_effect=recording):
await self.run_health()
self.assertEqual(len(threads), 1)
self.assertNotEqual(threads[0], loop_thread)
async def test_health_start_failure_is_an_error_envelope(self):
with (
patch.object(self.api, "_health_roots", side_effect=RuntimeError("secret")),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("POST", "/prompt_manager/system/health", 500)
self.assertEqual(
body,
{
"success": False,
"code": None,
"error": "Failed to start the health check",
},
)
self.assertEqual(self.api._jobs.list(), [])
class TestOverviewRoute(SystemAPITestCase):
async def test_overview_reports_real_counts(self):
self.api.db.save_prompt_pair("Cat", "Blur")
archived = self.api.db.save_prompt_pair("Cat", "Noise")
self.api.db.archive_prompt_pair(archived)
legacy = self.save("Legacy")
self.image(legacy, "a.png")
self.image(legacy, "b.png")
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertTrue(body["success"])
self.assertEqual(
body["library"], {"positive_prompts": 2, "negative_variants": 1}
)
self.assertEqual(body["images"], {"indexed": 2, "rated": 0})
self.assertEqual(
set(body),
{
"success",
"library",
"images",
"health",
"backup",
"needs_review",
"sources",
},
)
async def test_overview_rated_counts_images_of_rated_prompts(self):
rated = self.save("Rated", rating=5)
self.image(rated, "a.png")
self.image(rated, "b.png")
self.image(self.save("Unrated"), "c.png")
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(body["images"], {"indexed": 3, "rated": 2})
async def test_overview_never_fabricates_backup_or_health(self):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertIsNone(body["health"])
self.assertIsNone(body["backup"])
self.assertEqual(body["needs_review"], {"pending": None})
self.save("Any")
await self.run_health()
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["health"],
{"checked_at": report["checked_at"], "status": "incomplete", "issues": 0},
)
self.assertIsNone(body["backup"])
async def test_overview_health_counts_checks_with_issues(self):
prompt = self.save("Images")
self.image(prompt, "gone.png", exists=False)
self.image(prompt, "gone-too.png", exists=False)
await self.run_health()
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(body["health"]["issues"], 1)
async def test_overview_sources_distinguish_auto_detect(self):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["sources"], {"configured": 0, "effective": 1, "auto_detected": True}
)
second = self.output_dir / "archive"
second.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.output_dir),
self._root(second),
]
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["sources"], {"configured": 2, "effective": 2, "auto_detected": False}
)
# Every configured source unavailable: ComfyUI's output is used instead.
GalleryConfig.MONITORING_DIRECTORIES = [str(self.comfy_dir / "absent")]
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(
body["sources"], {"configured": 1, "effective": 1, "auto_detected": True}
)
async def test_overview_never_walks_the_sources_for_thumbnails(self):
(self.output_dir / "one.png").write_bytes(b"png")
with patch.object(
self.api,
"_plan_gallery_thumbnails",
side_effect=AssertionError("walked"),
):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertTrue(body["success"])
self.assertNotIn("thumbnails", body)
async def test_overview_sources_resolve_off_the_event_loop(self):
loop_thread = threading.get_ident()
threads = []
real = self.api._output_dirs_with_origin
def recording():
threads.append(threading.get_ident())
return real()
with patch.object(self.api, "_output_dirs_with_origin", side_effect=recording):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertEqual(body["sources"]["effective"], 1)
self.assertEqual(len(threads), 1)
self.assertNotEqual(threads[0], loop_thread)
async def test_overview_failure_is_an_error_envelope(self):
with (
patch.object(
self.api.db, "library_counts", side_effect=RuntimeError("secret")
),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("GET", "/prompt_manager/system/overview", 500)
self.assertEqual(
body,
{"success": False, "code": None, "error": "Failed to read the overview"},
)
class TestStatisticsRoute(SystemAPITestCase):
async def test_statistics_sum_real_run_counts(self):
a = self.save("Alpha", category="Landscapes", rating=5)
b = self.save("Beta", category="Portraits", rating=2)
self.api.db.record_prompt_use(a, times=3)
self.api.db.record_prompt_use(b)
body = await self.json("GET", "/prompt_manager/system/statistics")
self.assertEqual(
body["by_category"],
[
{"category": "Landscapes", "runs": 3, "average_rating": 5.0},
{"category": "Portraits", "runs": 1, "average_rating": 2.0},
],
)
self.assertEqual(body["totals"], self.api.db.get_statistics())
self.assertTrue(body["success"])
async def test_statistics_lists_rated_images_for_library_totals(self):
# D96: the Statistics dialog's "Library totals" reads the rated count
# with the runs (one read), from the same images the Overview counts.
rated = self.save("Rated", rating=4)
self.image(rated, "a.png")
self.image(self.save("Unrated"), "b.png")
body = await self.json("GET", "/prompt_manager/system/statistics")
self.assertEqual(body["rated_images"], 1)
self.assertEqual(
set(body), {"success", "by_category", "totals", "rated_images"}
)
async def test_statistics_failure_is_an_error_envelope(self):
with (
patch.object(
self.api.db, "usage_by_category", side_effect=RuntimeError("secret")
),
self.assertLogs("prompt_manager.api", "ERROR"),
):
body = await self.json("GET", "/prompt_manager/system/statistics", 500)
self.assertEqual(
body,
{"success": False, "code": None, "error": "Failed to read statistics"},
)
if __name__ == "__main__":
import unittest
unittest.main()
+28
View File
@@ -1,7 +1,9 @@
"""The concept fixture files carry the concept inspector facts (app.js:400-414)."""
import json
import math
import tempfile
import time
import unittest
from pathlib import Path
@@ -15,6 +17,8 @@ except ImportError: # discovered with tests/ as the top-level directory
from tests.browser.workspace_concept_fixture import (
CONCEPT_MODEL,
CONCEPT_PROMPTS,
MAX_EPOCH_SECONDS,
_instant,
write_concept_png,
)
from utils.prompt_inspector import inspect_prompt_evidence, read_inspection_file
@@ -80,5 +84,29 @@ class TestWorkspaceConceptFixture(unittest.TestCase):
self.assertEqual((image["width"], image["height"]), size)
class TestConceptFixtureInstant(unittest.TestCase):
"""One "now" for every relative time the fixture writes (PR #284's midnight)."""
def test_a_chosen_instant_is_kept(self):
self.assertEqual(_instant(1_792_713_598), 1_792_713_598.0)
self.assertEqual(_instant(1_792_713_598.25), 1_792_713_598.25)
def test_the_wall_clock_when_omitted(self):
before = time.time()
now = _instant(None)
self.assertLessEqual(before, now)
self.assertLessEqual(now, time.time())
def test_a_non_number_is_refused(self):
for value in (True, "1792713598", [1]):
with self.subTest(value=value), self.assertRaises(TypeError):
_instant(value)
def test_an_instant_out_of_range_is_refused(self):
for value in (0, -1, MAX_EPOCH_SECONDS, math.inf, math.nan):
with self.subTest(value=value), self.assertRaises(ValueError):
_instant(value)
if __name__ == "__main__":
unittest.main()
+24 -1
View File
@@ -6,7 +6,7 @@ import { once } from 'node:events';
import { readFileSync } from 'node:fs';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { setTimeout as delay } from 'node:timers/promises';
import { startBrowserServer, createServerSupervisor, attachServerStderr } from '../../tools/vendor/start-browser-server.mjs';
import { startBrowserServer, createServerSupervisor, attachServerStderr, forwardPrefixedLines, SLOW_RESET_PREFIX } from '../../tools/vendor/start-browser-server.mjs';
import { workflowWorkers } from '../../tools/vendor/test-workers.mjs';
const fakeServer = fileURLToPath(new URL('./fake-browser-server.mjs', import.meta.url));
@@ -193,6 +193,29 @@ test('a failed test gets the server stderr written during it attached; a passing
assert.equal(String(attached[0].body), 'Traceback (most recent call last): boom\n');
});
test('only complete slow-reset report lines are forwarded, however the chunks split them', () => {
const written = [];
const forward = forwardPrefixedLines(SLOW_RESET_PREFIX, text => written.push(text));
forward(`aiohttp access line\n${SLOW_RESET_PREFIX}POST /__test__/reset still`);
assert.deepEqual(written, [], 'a report line waits for its newline');
forward(` running after 2.000s\nplain stderr\n${SLOW_RESET_PREFIX}Thread 0x1 [asyncio_0]\n${SLOW_RESET_PREFIX.slice(0, 5)}`);
forward(`${SLOW_RESET_PREFIX.slice(5)} File "server.py", line 1\nnot a report ${SLOW_RESET_PREFIX}\n`);
assert.deepEqual(written, [
`${SLOW_RESET_PREFIX}POST /__test__/reset still running after 2.000s\n${SLOW_RESET_PREFIX}Thread 0x1 [asyncio_0]\n`,
`${SLOW_RESET_PREFIX} File "server.py", line 1\n`,
]);
});
test('a slow-reset report on server stderr reaches the worker stderr; other stderr does not', async t => {
const reported = [];
const server = await startBrowserServer({ ...fake(), report: text => reported.push(text) });
t.after(() => server.stop());
await writeStderr(server, 'Traceback: ordinary stderr\n');
await writeStderr(server, `${SLOW_RESET_PREFIX}POST /__test__/reset answered 200 after 16.064s\n`);
assert.deepEqual(reported, [`${SLOW_RESET_PREFIX}POST /__test__/reset answered 200 after 16.064s\n`]);
assert.ok(server.stderrTail().includes('Traceback: ordinary stderr'), 'the stderr tail still keeps everything');
});
test('a server exits with its worker even when the worker is SIGKILLed', async () => {
const pid = await serverOfKilledWorker(`{ command: process.execPath, args: port => [${JSON.stringify(fakeServer)}, '--port', String(port), '--exit-on-stdin-eof'] }`);
await waitFor(() => !alive(pid), `server ${pid} to exit after its worker died`);
+87 -15
View File
@@ -1,9 +1,9 @@
{
"admin.js": {
"lines": 28,
"lines": 31,
"branches": 58,
"functions": 23,
"statements": 28
"functions": 26,
"statements": 31
},
"api-client.js": {
"lines": 98,
@@ -42,10 +42,10 @@
"statements": 98
},
"gallery.js": {
"lines": 39,
"lines": 40,
"branches": 63,
"functions": 43,
"statements": 39
"functions": 44,
"statements": 40
},
"html-escape.js": {
"lines": 98,
@@ -157,7 +157,7 @@
},
"workspace/activity-job-view.mjs": {
"lines": 97,
"branches": 83,
"branches": 86,
"functions": 98,
"statements": 97
},
@@ -169,19 +169,19 @@
},
"workspace/activity-store.mjs": {
"lines": 98,
"branches": 95,
"branches": 96,
"functions": 98,
"statements": 98
},
"workspace/activity.mjs": {
"lines": 95,
"lines": 96,
"branches": 87,
"functions": 98,
"statements": 95
"statements": 96
},
"workspace/file-inspector.mjs": {
"lines": 96,
"branches": 92,
"branches": 93,
"functions": 92,
"statements": 96
},
@@ -229,8 +229,8 @@
},
"workspace/image-tagging.mjs": {
"lines": 97,
"branches": 88,
"functions": 87,
"branches": 90,
"functions": 88,
"statements": 97
},
"workspace/images-category.mjs": {
@@ -253,7 +253,7 @@
},
"workspace/images-selection.mjs": {
"lines": 98,
"branches": 92,
"branches": 93,
"functions": 98,
"statements": 98
},
@@ -263,6 +263,18 @@
"functions": 98,
"statements": 98
},
"workspace/images-tagging-bulk.mjs": {
"lines": 98,
"branches": 93,
"functions": 86,
"statements": 98
},
"workspace/images-tagging-review.mjs": {
"lines": 98,
"branches": 93,
"functions": 98,
"statements": 98
},
"workspace/images-tagging.mjs": {
"lines": 98,
"branches": 98,
@@ -284,7 +296,7 @@
"workspace/inspect.mjs": {
"lines": 98,
"branches": 92,
"functions": 92,
"functions": 93,
"statements": 98
},
"workspace/inspection-state.mjs": {
@@ -329,12 +341,42 @@
"functions": 98,
"statements": 98
},
"workspace/local-parameters.mjs": {
"lines": 98,
"branches": 94,
"functions": 98,
"statements": 98
},
"workspace/local-projection.mjs": {
"lines": 97,
"branches": 96,
"functions": 98,
"statements": 97
},
"workspace/local-resolver-stages.mjs": {
"lines": 94,
"branches": 93,
"functions": 98,
"statements": 94
},
"workspace/local-resolver-support.mjs": {
"lines": 96,
"branches": 80,
"functions": 98,
"statements": 96
},
"workspace/local-resolver.mjs": {
"lines": 96,
"branches": 86,
"functions": 98,
"statements": 96
},
"workspace/local-semantics.mjs": {
"lines": 98,
"branches": 98,
"functions": 98,
"statements": 98
},
"workspace/local-session.mjs": {
"lines": 98,
"branches": 92,
@@ -377,6 +419,12 @@
"functions": 98,
"statements": 98
},
"workspace/py-json.mjs": {
"lines": 97,
"branches": 92,
"functions": 98,
"statements": 97
},
"workspace/recorded-run.mjs": {
"lines": 95,
"branches": 83,
@@ -395,6 +443,30 @@
"functions": 98,
"statements": 98
},
"workspace/system-controls.mjs": {
"lines": 98,
"branches": 87,
"functions": 89,
"statements": 98
},
"workspace/system-overview.mjs": {
"lines": 96,
"branches": 92,
"functions": 98,
"statements": 96
},
"workspace/system-state.mjs": {
"lines": 98,
"branches": 92,
"functions": 92,
"statements": 98
},
"workspace/system.mjs": {
"lines": 93,
"branches": 85,
"functions": 69,
"statements": 93
},
"workspace/theme.mjs": {
"lines": 98,
"branches": 93,
+19
View File
@@ -17,6 +17,22 @@ const HOST = '127.0.0.1';
// tests/browser/server.py prints this once it holds its listening socket. Only a
// complete line counts: a chunk may end mid-number.
const PORT_ANNOUNCEMENT = /^PM_TEST_SERVER_PORT=(\d+)\r?\n/m;
// tests/browser/slow_reset_watch.py starts every line of a slow-reset report with
// this. Those lines go to the worker's stderr, so the job log shows a slow reset
// even when its test passes.
export const SLOW_RESET_PREFIX = 'pm-slow-reset| ';
// A stderr chunk handler that passes every complete line starting with `prefix`
// to `write`. A line split across chunks is held until its newline arrives.
export function forwardPrefixedLines(prefix, write) {
let pending = '';
return chunk => {
const lines = (pending + chunk).split('\n');
pending = lines.pop().slice(-STDERR_TAIL_BYTES);
const matched = lines.filter(line => line.startsWith(prefix));
if (matched.length) write(matched.map(line => `${line}\n`).join(''));
};
}
// The server exits when its stdin closes, so a worker that dies without teardown
// (SIGKILL) leaves no server behind; the launcher holds the pipe's other end.
@@ -40,6 +56,7 @@ export async function startBrowserServer({
cwd = repositoryRoot,
timeoutMs = STARTUP_TIMEOUT_MS,
stderrLogChars = STDERR_LOG_CHARS,
report = chunk => process.stderr.write(chunk),
} = {}) {
if (!Number.isInteger(port) || port < 0 || port > 65535) {
throw new Error(`startBrowserServer needs a port from 0 to 65535, got ${port}`);
@@ -50,7 +67,9 @@ export async function startBrowserServer({
let stderr = '';
let stderrTotal = 0;
child.stderr.setEncoding('utf8');
const forwardReport = forwardPrefixedLines(SLOW_RESET_PREFIX, report);
child.stderr.on('data', chunk => {
forwardReport(chunk);
stderr = (stderr + chunk).slice(-Math.max(stderrLogChars, STDERR_TAIL_BYTES));
stderrTotal += chunk.length;
});
+25
View File
@@ -28,6 +28,7 @@ __all__ = [
"extract_generation_params",
"extract_from_records",
"parse_js_json",
"parse_page_json",
]
PLACEHOLDERS = MappingProxyType(
@@ -953,6 +954,30 @@ def parse_js_json(text):
return None
def _nan_as_null(name):
"""The legacy pages' NaN cleaning; Infinity still throws in ``JSON.parse``."""
if name == "NaN":
return None
raise ValueError(f"{name} is not JSON")
def parse_page_json(text):
"""A metadata record as the legacy gallery and admin pages parsed it.
They replace ``NaN`` with ``null`` and then ``JSON.parse`` (ComfyUI writes
``"is_changed": [NaN]`` for nodes that always re-run, PromptManager among
them). Only the NaN number reads as null here: the pages' regex also
rewrote the word inside strings, which changed prompt text. None where the
parse throws.
"""
if not isinstance(text, str):
return None
try:
return json.loads(text, parse_constant=_nan_as_null, parse_int=_double_integer)
except (ValueError, RecursionError):
return None
def extract_from_records(prompt_text, workflow_text):
"""``extract_generation_params`` for raw ``prompt``/``workflow`` record text.
+2 -1
View File
@@ -20,6 +20,7 @@ from .prompt_pair_resolver import (
_FOOTER,
EMPTY_POSITIVE_REASON,
INSPECTION_PROFILE,
MALFORMED_EVIDENCE_REASON,
ResolverProfile,
resolve_prompt_pair,
)
@@ -548,7 +549,7 @@ def inspect_prompt_evidence(
resolution = {
"state": "unresolved",
"sampler_id": None,
"reason": "Prompt evidence is malformed",
"reason": MALFORMED_EVIDENCE_REASON,
"reason_code": "malformed",
"node": None,
}
+12 -2
View File
@@ -1237,10 +1237,13 @@ class _GraphResolver:
def non_file_outputs(self) -> set:
names = self.semantics.non_file_outputs
# Only a string names a class: a list or object is never looked up (#286).
return {
node_id
for node_id, node in self.graph.items()
if isinstance(node, dict) and node.get("class_type") in names
if isinstance(node, dict)
and isinstance(node.get("class_type"), str)
and node["class_type"] in names
}
def producers_of(self, starts: list, previews: set) -> set:
@@ -1832,6 +1835,8 @@ def _shared_pair(candidates: list) -> Optional[Dict[str, Any]]:
# Bounds for walking a UI workflow's subgraph instances (R1 only).
_MAX_SUBGRAPH_DEPTH = 8
# The inspector's own line for evidence it cannot read (``inspect_prompt_evidence``).
MALFORMED_EVIDENCE_REASON = "Prompt evidence is malformed"
_MAX_WORKFLOW_NODES = 100_000
@@ -1862,7 +1867,9 @@ def _prompt_manager_widgets(workflow: Any) -> Dict[str, str]:
Nodes inside a subgraph run as ``"<instance>:<inner>"`` API IDs (nested
instances chain their IDs); a definition without an instance runs nothing.
The walk is bounded in depth and in nodes visited.
The walk is bounded in depth and in nodes visited. A node whose ``type`` is
a list or an object is skipped: it is neither a subgraph instance nor a
PromptManager node (#286).
"""
nodes = workflow.get("nodes") if isinstance(workflow, dict) else None
if not isinstance(nodes, list):
@@ -1882,6 +1889,9 @@ def _prompt_manager_widgets(workflow: Any) -> Dict[str, str]:
if not isinstance(node_id, (str, int)) or isinstance(node_id, bool):
continue
api_id = f"{prefix}{node_id}"
if isinstance(kind, (list, dict)):
# JSON's unhashable values are never a node type (#286).
continue
if kind in subgraphs and depth < _MAX_SUBGRAPH_DEPTH:
stack.append((subgraphs[kind], api_id + ":", depth + 1))
continue
+46 -7
View File
@@ -25,8 +25,10 @@
this.tagsPage = null;
this.api = ApiClient.createApiClient();
// Search and page loads share one guard: whichever list request is newest wins
this.listRequest = ApiClient.latestOnly();
// The page's lifetime ends when it starts to leave. WebKit cancels the page's reads
// between beforeunload and pagehide; a save's follow-up read started then fails and is
// logged as a page error ("Fetch API cannot load … due to access control checks.").
this.startPageLifetime();
this.infiniteScroll = false; // session value; seeded from settings, toggled in the header
this.infiniteScrollChosen = false;
this.loadingMore = false;
@@ -150,7 +152,18 @@
// Output scan: a server-side job that survives modal dismissal and reloads
this.scanState = ScanProgress.initialState();
this.scanSource = null;
window.addEventListener("beforeunload", (e) => this.warnIfScanRunning(e));
window.addEventListener("beforeunload", (e) => {
this.warnIfScanRunning(e);
// A page that asks to stay (a running scan) keeps its reads until it really goes.
if (!e.defaultPrevented) this.pageLifetime.abort();
});
window.addEventListener("pagehide", () => this.pageLifetime.abort());
// A page restored from the back/forward cache lives again, and so does one whose
// navigation never left (stopped, or answered with a download): its next input revives it.
window.addEventListener("pageshow", (e) => { if (e.persisted) this.revivePageLifetime(); });
for (const type of ["pointerdown", "keydown"]) {
window.addEventListener(type, () => this.revivePageLifetime(), true);
}
this.bindModalEvents();
@@ -319,9 +332,24 @@
}
}
/** A new lifetime once the old one has ended; a living page keeps its list guard. */
revivePageLifetime() {
if (this.pageLifetime.signal.aborted) this.startPageLifetime();
}
/** A fresh page lifetime, and the list guard bound to it. */
startPageLifetime() {
this.pageLifetime = new AbortController();
// Search and page loads share one guard: whichever list request is newest wins
this.listRequest = ApiClient.latestOnly({ lifetime: this.pageLifetime.signal });
}
async loadStatistics() {
// A leaving page starts no read; one in flight is aborted with the page and not reported.
const { signal } = this.pageLifetime;
if (signal.aborted) return;
try {
const response = await ApiClient.fetchUrl("/prompt_manager/stats");
const response = await ApiClient.fetchUrl("/prompt_manager/stats", { signal });
if (response.ok) {
const data = await response.json();
@@ -340,6 +368,7 @@
console.error("Stats API HTTP error:", response.status, response.statusText);
}
} catch (error) {
if (signal.aborted) return;
console.error("Stats error:", error);
// Set some default values if API fails
document.getElementById("totalPrompts").textContent = this.prompts.length || 0;
@@ -1151,6 +1180,11 @@
`Done — ${data.imported} imported, ${data.skipped} skipped`;
this.loadStatistics();
setTimeout(() => this.hideModal("loraImportModal"), 2000);
} else if (data.type === "error") {
// A refusal (another job runs) or a failed import: name it and close the modal.
document.getElementById("loraImportStatus").textContent = `Error: ${data.message}`;
this.showNotification(`LoRA import failed: ${data.message}`, "error");
setTimeout(() => this.hideModal("loraImportModal"), 3000);
}
} catch (e) { /* skip malformed SSE */ }
}
@@ -1202,10 +1236,13 @@
}
this.hideModal("settingsModal");
} else {
throw new Error("Failed to save settings");
// A refusal names its reason (a corrupt config.json, a changed file): show it.
const data = await response.json().catch(() => ({}));
throw new Error(data.error ? `Failed to save settings: ${data.error}` : "Failed to save settings");
}
} catch (error) {
this.showNotification("Failed to save settings", "error");
const reason = error.message.startsWith("Failed to save settings:") ? error.message : "Failed to save settings";
this.showNotification(reason, "error");
}
}
@@ -1819,7 +1856,9 @@
throw new Error(data.error || 'Maintenance failed');
}
} else {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
// A refusal names the job that runs ("Scan for metadata already running").
const data = await response.json().catch(() => ({}));
throw new Error(data.error || `HTTP ${response.status}: ${response.statusText}`);
}
} catch (error) {
console.error('Maintenance error:', error);
+11 -3
View File
@@ -107,20 +107,28 @@
}
/**
* @param {{lifetime?: AbortSignal}} [options] - `lifetime` ends with the page (its pagehide):
* then no call starts a request, the one in flight is aborted, and both resolve to STALE.
* WebKit fails a fetch that a leaving page starts and logs it as a page error.
* @returns {(run: (signal: AbortSignal) => Promise<*>) => Promise<*>} wrapper that
* aborts the previous call's signal and resolves superseded calls to STALE
*/
function latestOnly() {
function latestOnly({ lifetime } = {}) {
let current = null;
lifetime?.addEventListener("abort", () => current?.abort(), { once: true });
return async function (run) {
if (current) current.abort();
if (lifetime?.aborted) {
current = null;
return STALE;
}
const controller = new AbortController();
current = controller;
try {
const result = await run(controller.signal);
return controller === current ? result : STALE;
return controller === current && !lifetime?.aborted ? result : STALE;
} catch (error) {
if (controller !== current) return STALE;
if (controller !== current || lifetime?.aborted) return STALE;
throw error;
}
};
+17 -3
View File
@@ -17,6 +17,14 @@
this.dupResults = null;
this.api = ApiClient.createApiClient();
// The page's lifetime ends when it starts to leave: WebKit fails a fetch a leaving page starts
// ("Fetch API cannot load … due to access control checks.") and logs it as a page error.
this.pageLifetime = new AbortController();
window.addEventListener("beforeunload", () => this.pageLifetime.abort());
window.addEventListener("pagehide", () => this.pageLifetime.abort());
window.addEventListener("pageshow", (e) => {
if (e.persisted && this.pageLifetime.signal.aborted) this.pageLifetime = new AbortController();
});
// Only the newest page request may update the grid
this.imagesRequest = ApiClient.latestOnly();
@@ -2129,8 +2137,11 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
headers: { 'Content-Type': 'application/json' }
});
if (response.ok) {
const data = await response.json();
if (!response.ok) {
// Refused (a running job holds the thumbnails): say why and claim nothing.
const data = await response.json().catch(() => ({}));
this.showNotification(`Thumbnails were not cleared: ${data.error || `HTTP ${response.status}`}`, 'error');
return;
}
} catch (e) {
console.warn('Could not clear server thumbnails:', e);
@@ -2531,8 +2542,11 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
return;
}
// A leaving page starts no read, and one it aborts is not reported.
const signal = this.pageLifetime?.signal;
if (signal?.aborted) return;
try {
const data = await this.api.get('/prompt_manager/images/thumbnails/status');
const data = await this.api.get('/prompt_manager/images/thumbnails/status', { signal });
const total = data.total_images;
const pending = data.pending;
if (!data.success || !Number.isSafeInteger(total) ||
+37 -37
View File
@@ -11,6 +11,9 @@
* filter, and the selected outcome's listed rows (at most 200 per outcome, so "Showing N of M"). Rows under Added and
* Already known open the file in the existing viewer through its listing identity (D57); the others are plain text.
* - Log: `details.term` "Scan log", collapsed, holding the raw timestamped lines in `pre[tabindex=0]`.
* - D89: the tiles, the outcome list, the openable rows and the log and cancel labels are the job kind's (`jobView`):
* an image tagging job shows Tagged, Unchanged, Skipped and Errors, opens Tagged and Unchanged rows, and reads
* "Tagging log" and "Cancel tagging"; another kind keeps the scan's tiles as "Job log" and "Cancel job".
*
* The summary follows the Activity store (one SSE stream per running job); the listed rows and the log are read from
* `GET /prompt_manager/jobs/<id>` at most once a second while the job runs, and once more when it ends.
@@ -19,7 +22,7 @@ import { createButton } from '../shared/toolkit/core.js';
import { pageHeading } from './heading.mjs';
import { showWorkspaceToast } from './toast.mjs';
import { noun } from './plural.mjs';
import { parseJob, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
import { parseJob, jobView, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
import { jobDetail, CANCEL_FAILED_TOAST } from './activity.mjs';
import { downloadJobReport } from './activity-scan.mjs';
import { createMediaViewer } from './media-viewer.mjs';
@@ -36,18 +39,6 @@ export const FILE_GONE = 'This file is no longer in Images. Rescan from Import a
export const OPEN_FAILED = 'The file could not be opened. Try again.';
const LISTING_LIMIT = 5;
/** The counter tiles, in the spec's order and the mockup's tones. */
export const TILES = Object.freeze([
['added', 'Added to Library', 'added'], ['known', 'Already known', ''], ['review', 'Needs review', 'warn'],
['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad'],
]);
/** The outcome list, in the mockup's order: what got added and what went wrong first. */
export const OUTCOMES = Object.freeze([
['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
['no_metadata', 'No usable metadata'], ['known', 'Already known'],
]);
const OPENABLE = new Set(['added', 'known']);
function element(tag, text, className) {
const node = document.createElement(tag);
if (text !== undefined) node.textContent = text;
@@ -103,9 +94,9 @@ export function filterRows(rows, filter) {
return rows.filter(row => `${row.root}/${row.path}`.toLocaleLowerCase().includes(wanted));
}
/** The first outcome with files, so a view opens on what happened; Added when nothing has yet. */
export function defaultOutcome(counters) {
return OUTCOMES.find(([group]) => (counters[group] ?? 0) > 0)?.[0] ?? 'added';
/** The first outcome with files, so a view opens on what happened; the first outcome when nothing has yet. */
export function defaultOutcome(counters, outcomes = JOB_VIEWS.scan.outcomes) {
return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])[0];
}
function section(title, id) {
@@ -132,13 +123,6 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
bar.max = 100;
const stats = element('dl', undefined, 'stat-row');
const tileValues = new Map();
for (const [group, label, tone] of TILES) {
const tile = element('div', undefined, tone ? `stat ${tone}` : 'stat');
const value = element('dd', '0');
tile.append(element('dt', label), value);
tileValues.set(group, value);
stats.append(tile);
}
progressPart.node.append(bar, stats);
const resultsPart = section('Results', 'job-results-title');
@@ -155,15 +139,6 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
list.setAttribute('aria-label', 'Outcomes');
const files = element('div', undefined, 'outcome-files');
const buttons = new Map();
for (const [group, label] of OUTCOMES) {
const button = createButton('');
const count = element('span', '0');
button.append(element('span', label), count);
button.setAttribute('aria-pressed', 'false');
button.count = count;
buttons.set(group, button);
list.append(button);
}
outcomes.append(list, files);
resultsPart.node.append(outcomes);
@@ -181,6 +156,8 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
missing.append(element('p', NOT_FOUND, 'empty-state'), back);
let detail = null;
// The job's kind decides its tiles, outcomes and labels (D89); it is known once the job is.
let view = null;
let selected = null;
let mounted = false;
let lastRead = 0;
@@ -200,13 +177,35 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
const viewer = createMediaViewer({ api, container: main, parentScope: scope, inspectionController: inspection,
sender: createPromptSender({ scope }), onSelect: image => inspection.select(image) });
/** The tiles and the outcome list of the job's kind, built once. */
function layout(kind) {
view = jobView(kind); // D89
for (const [group, label, tone] of view.tiles) {
const tile = element('div', undefined, tone ? `stat ${tone}` : 'stat');
const value = element('dd', '0');
tile.append(element('dt', label), value);
tileValues.set(group, value);
stats.append(tile);
}
for (const [group, label] of view.outcomes) {
const button = createButton('');
const count = element('span', '0');
button.append(element('span', label), count);
button.setAttribute('aria-pressed', 'false');
button.count = count;
buttons.set(group, button);
list.append(button);
}
}
function showBody(job) {
if (mounted) return;
mounted = true;
if (!view) layout(job.kind);
missing.remove();
parts.actions.append(download, cancel);
main.append(progressPart.node, resultsPart.node, log);
selected ??= defaultOutcome(job.counters);
selected ??= defaultOutcome(job.counters, view.outcomes);
}
/** The job is not (or no longer) in this session: say so instead of an empty view. */
@@ -242,7 +241,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
function renderRows(job) {
const rows = detail?.outcomes[selected] ?? [];
const openable = OPENABLE.has(selected);
const openable = view.openable.has(selected);
const wanted = filterRows(rows.map((row, index) => ({ ...row, index })), input.value);
const kept = new Map(wanted.map(row => rowNode(row, row.index, openable)));
rowNodes = kept;
@@ -270,12 +269,13 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
parts.h1.textContent = job.title;
document.title = `${job.title} · Prompt Manager`;
parts.description.textContent = jobDetail(job);
cancel.textContent = scan ? 'Cancel scan' : 'Cancel job';
cancel.textContent = view.cancel;
// A job lost with the server has no report there (review M9).
const lost = job.state === 'lost';
if (lost && document.activeElement === download) parts.h1.focus();
download.disabled = lost;
if (job.state === 'running') {
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel.
if (job.state === 'running' && job.cancellable) {
if (!cancel.isConnected) parts.actions.append(cancel);
} else if (cancel.isConnected) {
// A focused Cancel that goes away hands its focus to the job's heading, never to the page body.
@@ -291,7 +291,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
button.count.textContent = number(job.counters[group] ?? 0);
button.setAttribute('aria-pressed', String(group === selected));
}
logLabel.textContent = `● ${scan ? 'Scan' : 'Job'} log`;
logLabel.textContent = `● ${view.log}`;
const text = detail?.log.length ? detail.log.join('\n') : 'No log lines yet.';
if (pre.textContent !== text) {
// Replacing the text clamps the scroll position, so keep it: pinned readers follow the end, others stay.
+75 -5
View File
@@ -23,10 +23,52 @@ export const LOST_DETAIL = 'This job ended when ComfyUI restarted.';
const SERVER_STATES = new Set(['running', 'finished', 'cancelled', 'failed']);
const FINAL_STATES = new Set(['finished', 'cancelled', 'failed']);
const JOB_ID = /^[0-9a-f]{32}$/;
// The pill's verb for each job kind; a kind without one shows its title.
const PILL_VERBS = Object.freeze({ scan: 'Scanning' });
// The pill's verb for each job kind; a kind without one shows its title (D89: image tagging jobs read "Tagging").
const PILL_VERBS = Object.freeze({ scan: 'Scanning', 'image-tags': 'Tagging' });
const START_TOASTS = Object.freeze({ scan: 'Scan started.' });
/** The scan's job view (D56): its counter tiles `[group, label, tone]` and outcome list in the mockup's orders. */
const SCAN_VIEW = Object.freeze({
tiles: Object.freeze([
['added', 'Added to Library', 'added'], ['known', 'Already known', ''], ['review', 'Needs review', 'warn'],
['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad'],
].map(Object.freeze)),
// What got added and what went wrong first.
outcomes: Object.freeze([
['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
['no_metadata', 'No usable metadata'], ['known', 'Already known'],
].map(Object.freeze)),
// Rows that open the file in the viewer (D57); the others are plain text.
openable: Object.freeze(new Set(['added', 'known'])),
log: 'Scan log',
cancel: 'Cancel scan',
});
/**
* Each job kind's view: its tiles, outcome list, openable rows, log label and cancel label. D89: an image tagging job
* (`image-tags`) counts Tagged (green), Unchanged, Skipped (warning) and Errors (red), lists what got tagged and what
* went wrong first, and opens Tagged and Unchanged rows in the viewer like the scan's (D57).
*/
export const JOB_VIEWS = Object.freeze({
scan: SCAN_VIEW,
'image-tags': Object.freeze({
tiles: Object.freeze([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'],
['errors', 'Errors', 'bad']].map(Object.freeze)),
outcomes: Object.freeze([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'],
['unchanged', 'Unchanged']].map(Object.freeze)),
openable: Object.freeze(new Set(['tagged', 'unchanged'])),
log: 'Tagging log',
cancel: 'Cancel tagging',
}),
});
// Another kind keeps the scan's tiles and outcomes under the generic labels.
const GENERIC_VIEW = Object.freeze({ ...SCAN_VIEW, log: 'Job log', cancel: 'Cancel job' });
/** The view of a job `kind`; an unknown kind gets the generic one. */
export function jobView(kind) {
return Object.hasOwn(JOB_VIEWS, kind) ? JOB_VIEWS[kind] : GENERIC_VIEW;
}
/** The wait before the `attempt`-th re-read (0-based): 1 s, 2 s, 4 s … never more than 30 s. */
export function backoffDelay(attempt) {
const step = Number.isSafeInteger(attempt) && attempt > 0 ? Math.min(attempt, 16) : 0;
@@ -61,6 +103,8 @@ export function parseJob(raw) {
scope: plainObject(raw.scope) ? { ...raw.scope } : {},
listed,
log_length: count(raw.log_length),
// A held legacy-page job cannot be cancelled; an older server sends no key (cancellable then).
cancellable: raw.cancellable !== false,
};
}
@@ -70,20 +114,31 @@ const errorsOf = job => `${number(job.counters.errors ?? 0)} ${noun(job.counters
/** "1,420 added". */
export const addedOf = job => `${number(job.counters.added ?? 0)} added`;
/** The toast when a job ends: "Scan finished · N added · M errors", "Scan cancelled · N added". */
/**
* The toast when a job ends: "Scan finished · N added · M errors", "Scan cancelled · N added"; an image tagging job
* the same way with its own title and count (D89): "Auto-tag images finished · N tagged · M errors".
*/
export function finishToast(job) {
if (job.kind === 'scan') {
if (job.state === 'cancelled') return `Scan cancelled · ${addedOf(job)}`;
if (job.state === 'failed') return `Scan failed · ${addedOf(job)} · ${errorsOf(job)}`;
return `Scan finished · ${addedOf(job)} · ${errorsOf(job)}`;
}
if (job.kind === 'image-tags') {
if (job.state === 'cancelled') return `${job.title} cancelled · ${taggedOf(job)}`;
return `${job.title} ${job.state === 'failed' ? 'failed' : 'finished'} · ${outcomeLine(job)}`;
}
if (job.state === 'cancelled') return `${job.title} was cancelled.`;
if (job.state === 'failed') return `${job.title} failed.`;
return `${job.title} finished.`;
}
// D85: an image tagging job says which of its two modes started.
const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' });
/** The toast after this tab starts a job. */
export function startToast(job) {
if (job.kind === 'image-tags' && IMAGE_TAG_START_TOASTS[job.scope.mode]) return IMAGE_TAG_START_TOASTS[job.scope.mode];
return START_TOASTS[job.kind] ?? `${job.title} started.`;
}
@@ -94,9 +149,14 @@ export function pillParts(job) {
return { label, files, text: label + files };
}
/** The scan's counters as one line: "N added · M errors"; other kinds have none yet. */
/** "12 tagged" (D89). */
export const taggedOf = job => `${number(job.counters.tagged ?? 0)} tagged`;
/** The counters as one line: "N added · M errors" for a scan, "N tagged · M errors" for image tagging (D89). */
export function outcomeLine(job) {
return job.kind === 'scan' ? `${addedOf(job)} · ${errorsOf(job)}` : '';
if (job.kind === 'scan') return `${addedOf(job)} · ${errorsOf(job)}`;
if (job.kind === 'image-tags') return `${taggedOf(job)} · ${errorsOf(job)}`;
return '';
}
function defaultConnect(url) {
@@ -276,6 +336,16 @@ export function createActivityStore({ api, scope, connect = defaultConnect, noti
return { busy: store.running()[0] ?? null, message: error.message };
}
},
/**
* Follow a job another route started (its summary from that route's answer, e.g. the health check's
* `POST /system/health`): it is listed and its stream opened at once, with no read and no start toast.
*/
track(raw) {
const job = parseJob(raw);
if (!job) throw new TypeError('Invalid job');
upsert(job);
return job;
},
/** Ask a running job to stop at its next file boundary; false once it has ended. */
async cancel(id) {
const result = await api.post(`${JOBS_URL}/${encodeURIComponent(id)}/cancel`, {}, { signal: scope.signal });
+21 -8
View File
@@ -11,13 +11,14 @@
* - D54: each `.activity-item` is a real job. Its chip names the state (green "Completed", or blue "Running",
* "Cancelled", "Failed", "Ended"), its line says when and what, and a running job shows its `<progress>` and
* "Cancel job" (no "Simulate completion").
* - D89: an image tagging job's line names its images and, once ended, "N tagged · M errors".
*/
import { createButton } from '../shared/toolkit/core.js';
import { navigateTo } from './state.mjs';
import { pageHeading } from './heading.mjs';
import { showWorkspaceToast } from './toast.mjs';
import { countOf } from './plural.mjs';
import { pillParts, outcomeLine, addedOf, LOST_DETAIL } from './activity-store.mjs';
import { countOf, noun } from './plural.mjs';
import { pillParts, outcomeLine, addedOf, taggedOf, LOST_DETAIL } from './activity-store.mjs';
export const ACTIVITY_DESCRIPTION = 'Scans, tagging, downloads and maintenance share one history.';
export const EMPTY_ACTIVITY = 'No jobs have run since ComfyUI started. Scans and other long operations appear here.';
@@ -44,12 +45,21 @@ export function clockTime(seconds) {
const join = parts => parts.filter(Boolean).join(' · ');
/** What a job covers: "2 output folders" for a scan. */
const whole = value => Number.isSafeInteger(value) && value >= 0;
/** What a job covers: "2 output folders" for a scan, "4 images" for image tagging (D89). */
function scopeLine(job) {
// D89: grouped as the server writes its lines ("19,622 images").
const images = job.scope?.images;
if (job.kind === 'image-tags') return whole(images) ? `${images.toLocaleString('en-US')} ${noun(images, 'image')}` : '';
const folders = job.scope?.output_folders;
return Number.isSafeInteger(folders) && folders >= 0 ? countOf(folders, 'output folder') : '';
return whole(folders) ? countOf(folders, 'output folder') : '';
}
/** What a cancelled job kept: "N added" for a scan, "N tagged" for image tagging (D89). */
const KEPT = Object.freeze({ __proto__: null, scan: addedOf, 'image-tags': taggedOf }); // D89
const keptOf = job => KEPT[job.kind]?.(job) ?? '';
/** The item's line under its title, by state. */
export function jobDetail(job) {
const at = clockTime(job.finished_at);
@@ -58,8 +68,7 @@ export function jobDetail(job) {
}
if (job.state === 'lost') return LOST_DETAIL;
if (job.state === 'cancelled') {
return join([`Cancelled ${at} at a file boundary`, job.kind === 'scan' ? addedOf(job) : '',
'completed work is kept']);
return join([`Cancelled ${at} at a file boundary`, keptOf(job), 'completed work is kept']);
}
if (job.state === 'failed') return join([`Stopped on an error at ${at}`, outcomeLine(job)]);
return join([`Finished ${at}`, outcomeLine(job)]);
@@ -144,11 +153,15 @@ function itemFor(job, scope) {
progress.textContent = `${next.progress}%`;
progress.setAttribute('aria-label', `${next.title} progress`);
if (!progress.isConnected) item.insertBefore(progress, actions.isConnected ? actions : null);
if (!actions.isConnected) item.append(actions);
} else {
progress.remove();
}
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel job.
if (running && next.cancellable) {
if (!actions.isConnected) item.append(actions);
} else if (actions.isConnected) {
// A focused Cancel job that goes away hands its focus to the item's title link, never to the page body.
if (actions.contains(document.activeElement)) link.focus();
progress.remove();
actions.remove();
}
};
+43 -3
View File
@@ -5,6 +5,7 @@ import { createSendButton } from './prompt-sender.mjs';
import { createWorkspaceIcon } from './icons.mjs';
import { navigateTo } from './state.mjs';
import { showWorkspaceToast } from './toast.mjs';
import { openCopyFallback } from './workspace-dialogs.mjs';
const RAW_KEYS = ['prompt', 'workflow', 'parameters', 'comfyassets_prompt_pairs'];
const RAW_LIMIT = 'Raw metadata exceeds the response limit. Download the original file to inspect it externally.';
@@ -26,6 +27,26 @@ const REVEAL_BUSY = 'The folder is already opening.';
const REVEAL_FAILED = 'The folder could not be opened. Try again.';
const FACTS = [['seed', 'Seed'], ['steps', 'Steps'], ['cfg', 'CFG'], ['sampler', 'Sampler'], ['scheduler', 'Scheduler'], ['model', 'Model']];
// D117 (LI-4): the legacy metadata page's COPY ALL (web/metadata.html copyAllData), its lines and placeholders.
const SUMMARY_LINES = [['Checkpoint', 'model'], ['Positive Prompt', 'positive'], ['Negative Prompt', 'negative'], ['Steps', 'steps'],
['CFG Scale', 'cfg'], ['Sampler', 'sampler'], ['Seed', 'seed']];
const SUMMARY_MISSING = {positive:'No prompt found', negative:'No negative prompt found'};
/**
* The one-click settings summary (D117): the legacy COPY ALL text for what this inspector shows. A confirmed prompt
* and a known fact are copied verbatim; anything else, an explicitly empty prompt included, takes the legacy page's
* placeholder, as its parse does (`hasValue` treats '' as absent).
*/
export function settingsSummary(record) {
const roles = promptRoles(record?.prompts);
const value = key => {
if (Object.hasOwn(SUMMARY_MISSING, key)) return roles[key] ? roles[key] : SUMMARY_MISSING[key];
const fact = record?.generation?.[key];
return fact?.status === 'known' && typeof fact.value === 'string' && fact.value !== '' ? fact.value : 'Unknown';
};
return SUMMARY_LINES.map(([label, key]) => `${label}: ${value(key)}`).join('\n');
}
function safeFilename(value) {
return String(value || 'file').replace(/[\\/\x00-\x1f\x7f<>:"|?*]/g, '_').replace(/^\.+|[. ]+$/g, '') || 'file';
}
@@ -175,7 +196,7 @@ function prefixedChip(prefix, text) {
* `variant: 'full'` (the viewer's `#viewer-details`, imageDetails(i, true)) leaves out the cover and adds "File &
* provenance" and "Workflow & raw metadata" with the downloads. `inline: true` means `panel`; neither means `full`.
*/
export function mountFileInspector(container, {api, sender, parentScope, variant, inline = false, promptContext = '', onTagFilter, onRetry, onRefresh, onInteraction = () => {}}) {
export function mountFileInspector(container, {api, sender, parentScope, variant, inline = false, promptContext = '', onTagFilter, onRetry, onRefresh, onInteraction = () => {}, copyFallback = openCopyFallback}) {
if (parentScope.signal.aborted) return {setState() {}, dispose() {}};
const scope = childScope(parentScope);
let contentScope;
@@ -184,7 +205,7 @@ export function mountFileInspector(container, {api, sender, parentScope, variant
scope.listen(container, 'keydown', onInteraction);
scope.listen(container, 'click', onInteraction);
scope.listen(container, 'wheel', onInteraction, {passive:true});
scope.cleanup(() => {contentScope?.dispose(); container.replaceChildren(); container.classList.remove('file-inspector');});
scope.cleanup(() => {contentScope?.dispose(); container.replaceChildren(); container.classList.remove('file-inspector'); container.removeAttribute('aria-busy');});
const full = (variant ?? (inline ? 'panel' : 'full')) === 'full';
function statusView(state, parent) {
@@ -203,6 +224,8 @@ export function mountFileInspector(container, {api, sender, parentScope, variant
if (scope.signal.aborted) return;
contentScope?.dispose(); contentScope = childScope(scope);
container.replaceChildren();
// A local read resolves on this thread in bounded steps; the busy state stays until its answer lands.
if (state.loading) container.setAttribute('aria-busy', 'true'); else container.removeAttribute('aria-busy');
render(state);
}
/**
@@ -244,7 +267,7 @@ export function mountFileInspector(container, {api, sender, parentScope, variant
inner.append(tags);
}
inner.append(...generationFacts(record, 'meta-grid', 'small'));
if (full) inner.append(provenance(record, badge), rawMetadata(record));
if (full) inner.append(summaryButton(record), provenance(record, badge), rawMetadata(record));
const footer = full ? container : el('div', undefined, 'detail-footer'), roles = promptRoles(record.prompts);
if (Object.hasOwn(roles, 'positive') && Object.hasOwn(roles, 'negative')) {
const pair = createButton('', 'send');
@@ -264,6 +287,23 @@ export function mountFileInspector(container, {api, sender, parentScope, variant
}
if (!full && footer.childNodes.length) container.append(footer);
}
/**
* D117 (LI-4): "Copy settings summary" after the six facts, the legacy COPY ALL. It copies as the concept's prompt
* copy does: the shared toast, or the select-to-copy dialog where the clipboard is unavailable or refused.
*/
function summaryButton(record) {
const button = createButton('Copy settings summary', 'settings-summary'), text = settingsSummary(record);
contentScope.listen(button, 'click', async () => {
try {
if (!navigator.clipboard) throw new Error('Clipboard requires a secure context');
await navigator.clipboard.writeText(text);
if (!contentScope.signal.aborted) showWorkspaceToast('Settings summary copied.');
} catch {
if (!contentScope.signal.aborted) copyFallback(text, {scope:contentScope, trigger:button, title:'Copy settings summary'});
}
});
return button;
}
/** The concept's plain cover; the card image opens the viewer (D19). Video keeps its own element. */
function cover(record) {
const {file} = record, video = file?.media_type === 'video';
+116 -14
View File
@@ -128,7 +128,8 @@ export function optionError(model, options) {
return null;
}
function requestOptions(model, options) {
/** The options a request sends for `model`: numbers parsed, only the model's own keys. */
export function requestOptions(model, options) {
const body = {suggestion_limit: numberOf(options.suggestion_limit)};
if (modelOf(model).classifier) {
return {general_threshold: numberOf(options.general_threshold), character_threshold: numberOf(options.character_threshold), ...body};
@@ -177,28 +178,48 @@ function validateReceipt(value) {
return value;
}
/** Thresholds, instructions and suggestion count from the server, else the documented defaults. */
/**
* Thresholds, instructions and suggestion count from the server, else the documented defaults, and the models'
* download status (`models`, null when unknown).
*/
async function readDefaults(api, signal) {
try {
const value = await api.get(MODELS_URL, {signal, cache: 'no-store'});
if (!object(value)) return DEFAULTS;
if (!object(value)) return {defaults: DEFAULTS, models: null};
return {
general_threshold: unit(value.wd14_general_threshold) ? value.wd14_general_threshold : DEFAULTS.general_threshold,
character_threshold: unit(value.wd14_character_threshold) ? value.wd14_character_threshold : DEFAULTS.character_threshold,
instructions: typeof value.default_prompt === 'string' ? value.default_prompt : DEFAULTS.instructions,
suggestion_limit: limit(value.suggestion_limit) ? value.suggestion_limit : DEFAULTS.suggestion_limit,
defaults: {
general_threshold: unit(value.wd14_general_threshold) ? value.wd14_general_threshold : DEFAULTS.general_threshold,
character_threshold: unit(value.wd14_character_threshold) ? value.wd14_character_threshold : DEFAULTS.character_threshold,
instructions: typeof value.default_prompt === 'string' ? value.default_prompt : DEFAULTS.instructions,
suggestion_limit: limit(value.suggestion_limit) ? value.suggestion_limit : DEFAULTS.suggestion_limit,
},
models: object(value.models) ? value.models : null,
};
} catch {
// The defaults keep the dialog usable; Generate reports a real failure itself.
return DEFAULTS;
return {defaults: DEFAULTS, models: null};
}
}
function initialState(source) {
const UNAVAILABLE_HINT = 'Download it from the legacy Auto-tag page.';
/**
* Why `model` cannot run, from the `/autotag/models` status (the server's own `model-unavailable` sentence), else ''
* (downloaded, or the status is unknown: the job then reports a missing model itself).
*/
export function modelUnavailable(models, model) {
const option = modelOf(model);
requireValue(option, 'Unknown tagging model');
const status = object(models) && object(models[model]) ? models[model] : null;
if (!status || status.downloaded !== false) return '';
const name = typeof status.name === 'string' && status.name.trim() ? status.name : option.label.split(' · ')[0];
return `${name} is not downloaded. ${UNAVAILABLE_HINT}`;
}
function initialState(source, preset) {
return {
phase: 'observing', closed: false, source, showScopeNote: source === 'review',
image: null, savedTags: [], tagRevision: 0, currentTags: null, manual: '',
model: MODEL_OPTIONS[0].id, options: {...DEFAULTS}, generated: false, suggestions: [],
model: preset?.model ?? MODEL_OPTIONS[0].id, options: {...DEFAULTS, ...preset?.options}, generated: false, suggestions: [],
suggestionsText: COPY.modelChanged, statusText: '', generateSupported: false, generateReason: null, toast: null,
// Bumped whenever the controller itself rewrites the option fields or the manual draft, never while typing.
optionsRevision: 0, draftRevision: 0,
@@ -219,13 +240,31 @@ function derive(state) {
};
}
export function createImageTaggingController({api, target, source = 'images', onLateSave = () => {}} = {}) {
/** A `{model, options}` preset (the bulk review's choice) when valid, else a TypeError. */
function validatePreset(preset) {
if (preset === null || preset === undefined) return null;
requireValue(object(preset) && modelOf(preset.model) && object(preset.options), 'Invalid tagging preset');
const options = {};
for (const [key, value] of Object.entries(preset.options)) {
requireValue(OPTION_KEYS.has(key), 'Unknown tagging option');
options[key] = value;
}
requireValue(!optionError(preset.model, {...DEFAULTS, ...options}), 'Invalid tagging preset');
return Object.freeze({model: preset.model, options: Object.freeze(options)});
}
/**
* `preset` ({model, options}, D90's review session) starts the dialog on that model, and every observation applies
* those options over the server's defaults.
*/
export function createImageTaggingController({api, target, source = 'images', onLateSave = () => {}, preset = null} = {}) {
requireValue(object(api) && ['get', 'post', 'put', 'del'].every(name => typeof api[name] === 'function'), 'An api client is required');
const imageTarget = validateTarget(target);
requireValue(SOURCES.has(source), 'Unknown tagging source');
requireValue(typeof onLateSave === 'function', 'onLateSave must be a function');
const chosen = validatePreset(preset);
let state = freeze(derive(initialState(source)));
let state = freeze(derive(initialState(source, chosen)));
const listeners = new Set();
let closed = false;
let capture = null;
@@ -297,7 +336,7 @@ export function createImageTaggingController({api, target, source = 'images', on
}
capture = {token: value.capture.token, sha256: value.capture.sha256};
epoch = value.library_epoch;
defaults = await defaultsRead;
({defaults} = await defaultsRead);
if (closed || serial !== openSerial) return;
observedOnce = true;
const {width, height, media_type: mediaType, preview_url: previewUrl, generate} = value.capture;
@@ -311,7 +350,7 @@ export function createImageTaggingController({api, target, source = 'images', on
savedTags: value.tags.slice(), tagRevision: value.tag_revision,
manual: draft === null ? value.tags.join(', ') : changed ? mergeCurrent(value.tags, draft) : draft,
draftRevision: state.draftRevision + 1,
options: {...defaults}, optionsRevision: state.optionsRevision + 1,
options: {...defaults, ...chosen?.options}, optionsRevision: state.optionsRevision + 1,
generateSupported: generate.supported, generateReason: generate.reason,
statusText: changed ? `${COPY.changedElsewhere} ${savedNow(value.tags)}${reason ? ` ${reason}` : ''}` : reason,
});
@@ -482,3 +521,66 @@ export function createImageTaggingController({api, target, source = 'images', on
subscribe, open, setManual, setModel, setOption, generate, remove, save, retry, cancel, dispose,
};
}
/**
* The model and options of the bulk "Auto-tag selected images" dialog (D85): the same fields as the Tag image dialog's
* model panel, without an image. open() reads the server's defaults and the models' download status once; until it
* lands Start waits (`canStart` false). `unavailable` is why the chosen model cannot run ('' when it can), so the
* dialog refuses to start a job that would fail at once.
*/
export function createTaggingOptionsController({api} = {}) {
requireValue(object(api) && typeof api.get === 'function', 'An api client is required');
const listeners = new Set();
let closed = false;
let defaults = DEFAULTS;
let models = null;
let abort = null;
const deriveOptions = value => ({...value, canGenerate: false, unavailable: modelUnavailable(models, value.model),
canStart: value.phase === 'ready' && !modelUnavailable(models, value.model)});
let state = freeze(deriveOptions({phase: 'loading', model: MODEL_OPTIONS[0].id, options: {...DEFAULTS}, optionsRevision: 0}));
function set(patch) {
if (closed) return;
state = freeze(deriveOptions({...state, ...patch}));
for (const listener of [...listeners]) listener(state);
}
async function open() {
if (closed || abort) return;
abort = new AbortController();
const read = await readDefaults(api, abort.signal);
if (closed) return;
({defaults, models} = read);
set({phase: 'ready', options: {...defaults}, optionsRevision: state.optionsRevision + 1});
}
function setModel(id) {
requireValue(modelOf(id), 'Unknown tagging model');
if (closed || id === state.model) return;
// As in the Tag image dialog: the model's fields start from the defaults again; the count is kept (D91).
set({model: id, optionsRevision: state.optionsRevision + 1,
options: {...state.options, general_threshold: defaults.general_threshold,
character_threshold: defaults.character_threshold, instructions: defaults.instructions}});
}
function setOption(key, value) {
requireValue(OPTION_KEYS.has(key), 'Unknown tagging option');
set({options: {...state.options, [key]: value}});
}
/** `{model_type, options}` for a job or a review session, or null while the options are invalid. */
function request() {
if (optionError(state.model, state.options)) return null;
return {model_type: state.model, options: requestOptions(state.model, state.options)};
}
function dispose() {
closed = true;
listeners.clear();
abort?.abort();
}
function subscribe(listener) {
requireValue(typeof listener === 'function', 'A listener function is required');
listeners.add(listener);
return () => listeners.delete(listener);
}
return {
get state() {return state;},
subscribe, open, setModel, setOption, request, dispose,
};
}
+62 -20
View File
@@ -101,10 +101,12 @@ function numberInput(value, {min, max, step}) {
/**
* The concept's `.tag-model-panel` (a1049): `heading`, the Tagging model select, the model's options and Generate.
* It follows `controller` until `scope` is disposed (reused by the bulk Auto-tag dialog). Returns the `element` and
* `requestGenerate()`, Generate's own path (field validity first), for a Retry after a failed Generate.
* It follows `controller` until `scope` is disposed (reused by the bulk Auto-tag dialog, D85, without Generate:
* `generate: false`). Returns the `element`, `validate()` (the fields' native validity, then what the controller would
* refuse; true when valid, else the first invalid field reports) and `requestGenerate()`, Generate's own path (field
* validity first), for a Retry after a failed Generate.
*/
export function renderModelPanel({scope, controller, heading = 'Auto-tag this image'}) {
export function renderModelPanel({scope, controller, heading = 'Auto-tag this image', generate: withGenerate = true}) {
const panel = el('div', undefined, 'tag-model-panel');
const select = el('select');
for (const option of MODEL_OPTIONS) {
@@ -116,7 +118,8 @@ export function renderModelPanel({scope, controller, heading = 'Auto-tag this im
options.id = 'image-tag-model-options';
const generate = createButton('', 'ca-primary');
generate.append(createWorkspaceIcon('tagImage'), ' Generate suggestions');
panel.append(el('h3', heading), field('Tagging model', select, 'image-tag-model'), options, generate);
panel.append(el('h3', heading), field('Tagging model', select, 'image-tag-model'), options);
if (withGenerate) panel.append(generate);
const inputs = new Map();
let shownModel = null, shownOptions = null;
@@ -161,26 +164,28 @@ export function renderModelPanel({scope, controller, heading = 'Auto-tag this im
setDisabled(generate, !state.canGenerate);
}
select.addEventListener('change', () => controller.setModel(select.value));
function requestGenerate() {
function validate() {
const {state} = controller;
if (!state.canGenerate) return;
// The concept's native validity first (a1104–1114), then what the controller would refuse (a blank field).
for (const input of inputs.values()) input.setCustomValidity('');
const invalid = [...inputs.values()].find(input => !input.checkValidity());
if (invalid) { invalid.reportValidity(); return; }
if (invalid) { invalid.reportValidity(); return false; }
const error = optionError(state.model, state.options);
if (error) {
const input = inputs.get(error.field);
input.setCustomValidity(error.message);
input.reportValidity();
return;
return false;
}
controller.generate();
return true;
}
function requestGenerate() {
if (controller.state.canGenerate && validate()) controller.generate();
}
generate.addEventListener('click', () => { if (!disabled(generate)) requestGenerate(); });
scope.cleanup(controller.subscribe(update));
update(controller.state);
return {element: panel, requestGenerate};
return {element: panel, requestGenerate, validate};
}
function previewOf(image, state) {
@@ -192,20 +197,33 @@ function previewOf(image, state) {
/**
* Open "Tag image" for one listed `image` (its tagging target). `source` is the page (`images`, `library`, `review`,
* `tags`); `onSaved(event)` runs after a confirmed save, as every `onImageTagsSaved` listener does.
* `tags`, `bulk`); `onSaved(event)` runs after a confirmed save, as every `onImageTagsSaved` listener does.
*
* D90 (the bulk review session, `images-tagging-review.mjs`): `actions: 'review'` with `position: {index, total}`
* titles the dialog "Tag image · Image <index> of <total>" and makes its actions Stop reviewing, Skip and Approve and
* next (Approve and finish on the last image). `preset` ({model, options}) applies the session's model and options,
* and `autoGenerate` runs Generate as soon as the file is observed. Approve saves as Save does, without the per-image
* toast (the session toasts its count), and calls `onSaved`; Skip writes nothing and calls `onSkip`; Stop reviewing,
* Escape and × write nothing and call `onStop` once. Swapping the next image's dialog in is not a stop.
*/
export function openImageTagging({api, scope, image, source = 'images', trigger = null, onSaved = null}) {
export function openImageTagging({api, scope, image, source = 'images', trigger = null, onSaved = null, preset = null,
autoGenerate = false, actions: mode = 'tag', position = null, onSkip = null, onStop = null}) {
const target = imageTarget(image);
if (!target) throw new TypeError('The image has no tagging target');
const review = mode === 'review';
if (review && !(Number.isSafeInteger(position?.index) && Number.isSafeInteger(position?.total) &&
position.index >= 1 && position.index <= position.total)) throw new TypeError('A review needs its position');
// Set once this dialog hands over to the session (Approve, Skip): its close is then not a Stop.
let advancing = false;
const saved = ({filename: name, tags, toast}) => {
const event = {target, filename: name, tags, source};
announceSaved(event);
if (toast) showWorkspaceToast(toast);
onSaved?.(event);
showWorkspaceToast(toast);
};
// A save that settles after the dialog closed is still reported: its toast, and the pages' refresh (I5).
const onLateSave = outcome => (outcome.saved ? saved(outcome) : showWorkspaceToast(outcome.message));
const controller = createImageTaggingController({api, target, source, onLateSave});
const controller = createImageTaggingController({api, target, source, onLateSave, preset});
const local = createScope();
let open = true;
let completed = false;
@@ -253,9 +271,15 @@ export function openImageTagging({api, scope, image, source = 'images', trigger
// 7. The actions; Retry leads them only while the controller allows it.
const actions = el('div', undefined, 'dialog-actions');
const retry = createButton('Retry');
const cancel = createButton('Cancel');
const cancel = createButton(review ? 'Stop reviewing' : 'Cancel'); // D90
const skip = createButton('Skip');
const primary = createButton('Save manual tags', 'ca-primary');
actions.append(cancel, primary);
const last = review && position.index === position.total;
// D90: Approve with suggestions on screen; after a model change cleared them (M2) Save keeps only the manual field.
const reviewLabel = state => (state.generated ? (last ? 'Approve and finish' : 'Approve and next')
: (last ? 'Save and finish' : 'Save and next'));
if (review) actions.append(cancel, skip, primary);
else actions.append(cancel, primary);
const nodes = [heading];
if (controller.state.showScopeNote) nodes.push(el('p', SCOPE_NOTE, 'tag-scope-note small'));
@@ -301,6 +325,12 @@ export function openImageTagging({api, scope, image, source = 'images', trigger
}
function complete(state) {
completed = true;
if (review) {
// D90: the session swaps the next image in (or ends), so this dialog is not closed here.
advancing = true;
saved({filename: state.image.filename, tags: state.savedTags.slice(), toast: null});
return;
}
closeWorkspaceDialog();
saved({filename: state.image.filename, tags: state.savedTags.slice(), toast: state.toast});
}
@@ -315,8 +345,10 @@ export function openImageTagging({api, scope, image, source = 'images', trigger
if (state.suggestions !== shownRows || state.suggestionsText !== shownText) renderSuggestions(state);
setText(count, state.countText);
setText(status, state.statusText);
setText(primary, state.primaryLabel);
setText(primary, review ? reviewLabel(state) : state.primaryLabel);
setDisabled(primary, !state.canSave);
// D90: while Approve's save is in flight the session cannot be left, so its count includes that save.
if (review) for (const button of [skip, cancel]) setDisabled(button, state.phase === 'saving');
if (state.canRetry !== retry.isConnected) {
if (state.canRetry) actions.prepend(retry); // D116
else {
@@ -327,7 +359,12 @@ export function openImageTagging({api, scope, image, source = 'images', trigger
if (state.phase === 'saved') complete(state);
}
cancel.addEventListener('click', () => closeWorkspaceDialog());
cancel.addEventListener('click', () => { if (!disabled(cancel)) closeWorkspaceDialog(); });
skip.addEventListener('click', () => {
if (!open || advancing || disabled(skip)) return;
advancing = true;
onSkip?.();
});
// Retry after a failed Generate takes Generate's own path, so an invalid field reports its validity (M4).
retry.addEventListener('click', () => (controller.state.phase === 'generate-error' ? requestGenerate() : controller.retry()));
primary.addEventListener('click', () => {
@@ -345,12 +382,17 @@ export function openImageTagging({api, scope, image, source = 'images', trigger
document.getElementById('modal')?.classList.remove('tagging-modal');
local.dispose();
controller.dispose();
if (review && !advancing) onStop?.();
}
openContentDialog({scope, title: 'Tag image', nodes, trigger, onClose: release});
const dialogTitle = review ? `Tag image · Image ${position.index} of ${position.total}` : 'Tag image'; // D90
openContentDialog({scope, title: dialogTitle, nodes, trigger, onClose: release,
canClose: review ? () => controller.state.phase !== 'saving' : null});
document.getElementById('modal')?.classList.add('tagging-modal');
local.cleanup(controller.subscribe(render));
render(controller.state);
controller.open();
controller.open().then(() => {
if (open && autoGenerate && controller.state.canGenerate) controller.generate();
});
return {
close() { if (open) closeWorkspaceDialog(); },
get controller() { return controller; },
+14 -2
View File
@@ -4,6 +4,7 @@ import { inspectPath, toInspectionRecord } from './inspection-state.mjs';
import { BUNDLE_FILENAME, metadataBundle, collectImagesMetadata, exportOutcome } from './images-export.mjs';
import { CATEGORY_DIALOG, CATEGORY_FAILED, CATEGORY_MAX_LENGTH, CATEGORY_NONE_LINKED, categoryOutcome, categoryTargets, updatedCount } from './images-category.mjs';
import { openInputDialog } from './workspace-dialogs.mjs';
import { openAddTags, openAutoTag } from './images-tagging-bulk.mjs';
import { showWorkspaceToast } from './toast.mjs';
const CHECKBOX = '.image-overlays input[type="checkbox"]';
@@ -26,8 +27,11 @@ function el(tag, text, className) {
* - `onCategorySet()` reloads the categories and the listing after a successful Set category (the reload clears the
* selection through `clear`); focus then moves to `focusFallback` if it was on the hidden action.
* - `retain(kept)` drops the checked files a refreshed listing no longer holds (a tag save under a filter, I7).
* - With the Activity store `activity`, Add tags and Auto-tag selected (D85) follow the count, as the concept's Add tags
* leads its actions; `onTagged(keys)` re-reads the files a finished tagging job tagged. The selection stays.
*/
export function mountSelectionBar({ scope, api, grid, getItems, focusFallback, onCategorySet, checked: initial = Object.freeze(new Set()) }) {
export function mountSelectionBar({ scope, api, grid, getItems, focusFallback, onCategorySet, activity = null, onTagged,
checked: initial = Object.freeze(new Set()) }) {
let checked = initial;
let exporting = false, categorizing = false;
const element = el('div');
@@ -40,7 +44,15 @@ export function mountSelectionBar({ scope, api, grid, getItems, focusFallback, o
const clearButton = createButton('Clear');
// Clear hides the bar it sits in, so focus moves to the page heading instead of falling to the page.
scope.listen(clearButton, 'click', () => { clear(); focusFallback.focus({ preventScroll: true }); });
bulk.append(count, setCategory, exportMetadata, clearButton);
bulk.append(count);
if (activity) {
const addTags = createButton('Add tags');
scope.listen(addTags, 'click', () => openAddTags({ scope, store: activity, images: checkedImages(), trigger: addTags, onTagged }));
const autoTag = createButton('Auto-tag selected'); // D85
scope.listen(autoTag, 'click', () => openAutoTag({ api, scope, store: activity, images: checkedImages(), trigger: autoTag, onTagged }));
bulk.append(addTags, autoTag);
}
bulk.append(setCategory, exportMetadata, clearButton);
/** The count updates in place: the same bar and `strong`, its text written only when it changes. */
function render() {
+192
View File
@@ -0,0 +1,192 @@
/**
* The Images selection bar's tagging actions (#249 PR D, Mock inventory M3):
*
* - **Add tags** (the concept's `bulk-tags` simpleInput, app.js a1378–1383): "Tag selected images", "Tags, separated
* by commas", Apply. The names are parsed as the Tag image dialog's manual field parses them, then the `image-tags`
* Activity job adds them to every selected image.
* - **Auto-tag selected** (D85, ruling L1, beyond the concept): "Auto-tag selected images" with the Tag image dialog's
* model panel (its fields show the server's defaults, D91's Suggestions included) and no Generate, then **How to
* apply** (D90, ruling T3-bulk): **Apply automatically** starts the job, **Review each image** starts the browser
* review session (images-tagging-review.mjs). A model that is not downloaded keeps Start disabled and says why.
*
* The toast says the job started (the Activity store's "Tagging started." / "Auto-tagging started."), or why it did
* not: a running record-changing job (named, with its link), a held model, a refused body. Activity shows progress.
* The selection stays either way. When a started job ends having tagged files, `onTagged(keys)` re-reads them.
*/
import { createButton, createScope } from '../shared/toolkit/core.js';
import { createTaggingOptionsController, parseManualTags } from './image-tagging-state.mjs';
import { imageTarget, renderModelPanel } from './image-tagging.mjs';
import { startReviewSession } from './images-tagging-review.mjs';
import { closeWorkspaceDialog, openContentDialog, openInputDialog } from './workspace-dialogs.mjs';
import { busyToast } from './activity-scan.mjs';
import { imageKey } from './images-state.mjs';
import { countOf } from './plural.mjs';
import { showWorkspaceToast } from './toast.mjs';
export const IMAGE_TAG_JOB = 'image-tags';
export const MAX_BULK_IMAGES = 500;
export const TOO_MANY = 'Select at most 500 images.';
export const NO_TAGS = 'Enter at least one tag.'; // D118
export const ADD_DIALOG = Object.freeze({title: 'Tag selected images', label: 'Tags, separated by commas', action: 'Apply'});
export const AUTO_TITLE = 'Auto-tag selected images';
export const APPLY_CHOICES = Object.freeze([
Object.freeze({value: 'auto', label: 'Apply automatically', action: 'Start auto-tagging'}),
Object.freeze({value: 'review', label: 'Review each image', action: 'Start review'}),
]);
// D118: the concept has no copy for these; they follow D55's "The scan could not start. Try again.".
const START_FAILED = Object.freeze({add: 'Tagging could not start. Try again.', auto: 'Auto-tagging could not start. Try again.'});
const FINAL = new Set(['finished', 'cancelled', 'failed']);
function el(tag, text, className) {
const node = document.createElement(tag);
if (text !== undefined) node.textContent = text;
if (className) node.className = className;
return node;
}
/**
* The job targets of the checked `images` in listing order (their reveal identity), or null when more than 500 are
* checked. A file without a tagging identity is left out.
*/
export function bulkTargets(images) {
if (images.length > MAX_BULK_IMAGES) return null;
return images.map(imageTarget).filter(Boolean);
}
/** The Add tags field's names, deduplicated as the Tag image dialog's manual field is. */
export const parseBulkTags = parseManualTags;
// Duck-typed: this ES module cannot import the UMD api-client.js.
const apiMessage = error => (error?.name === 'ApiError' && error.code && error.message ? error.message : null);
/**
* Start one `image-tags` job with `body` through `store.start`. Returns the job, or null after toasting why not: the
* busy answer naming the running job, a coded refusal's own sentence (`model-busy`, a bad body), else a fixed one.
*/
export async function startTagJob({store, scope, body, notify = showWorkspaceToast}) {
try {
const result = await store.start(IMAGE_TAG_JOB, body);
if (scope.signal.aborted) return null;
if (result.busy !== undefined) {
notify(busyToast(result.busy));
return null;
}
return result.job;
} catch (error) {
if (!scope.signal.aborted) notify(apiMessage(error) ?? START_FAILED[body.mode]);
return null;
}
}
/** Call `onEnd(job)` once the job `id` has ended in `store`; returns the unsubscribe. */
export function followJobEnd({store, scope, id, onEnd}) {
let done = false;
const check = () => {
const job = store.find(id);
if (done || !job || !FINAL.has(job.state)) return;
done = true;
stop();
onEnd(job);
};
const stop = store.subscribe(check);
scope.cleanup(stop);
check();
return stop;
}
/**
* Wire one started job: when it ends having tagged files, `onTagged(keys)` gets the image keys it was started for.
*/
function followTagged({store, scope, job, images, onTagged}) {
const keys = new Set(images.map(imageKey));
followJobEnd({store, scope, id: job.id, onEnd: ended => { if ((ended.counters.tagged ?? 0) > 0) onTagged(keys); }});
}
/**
* Add tags: the concept's simpleInput, then the job over `images` (the checked files, taken now). A field that parses
* to no names keeps the dialog and its text, and reports NO_TAGS on the field (D118). `openInput` is the dialog seam.
*/
export function openAddTags({scope, store, images, trigger, onTagged = () => {}, notify = showWorkspaceToast,
openInput = openInputDialog}) {
const targets = bulkTargets(images);
if (!targets) { notify(TOO_MANY); return; }
openInput({scope, trigger, ...ADD_DIALOG, validate: text => (parseBulkTags(text).length ? null : NO_TAGS),
onApply: async text => {
const tags = parseBulkTags(text);
const job = await startTagJob({store, scope, body: {mode: 'add', targets, tags}, notify});
if (job) followTagged({store, scope, job, images, onTagged});
}});
}
/** The D90 How to apply radio group: Apply automatically (checked) or Review each image. */
function applyChoice(onChange) {
const group = el('div', undefined, 'field');
group.setAttribute('role', 'radiogroup');
const caption = el('span', 'How to apply');
caption.id = 'auto-tag-apply-label';
group.setAttribute('aria-labelledby', caption.id);
group.append(caption);
for (const choice of APPLY_CHOICES) {
const label = el('label');
const input = el('input');
Object.assign(input, {type: 'radio', name: 'auto-tag-apply', value: choice.value, checked: choice.value === 'auto'});
input.addEventListener('change', () => { if (input.checked) onChange(choice); });
label.append(input, ` ${choice.label}`);
group.append(label);
}
return group;
}
/**
* Auto-tag selected (D85): the model panel, How to apply (D90), Cancel and Start. Start validates the fields as
* Generate does, then starts the job or the review session over `images` (the checked files, taken now).
*/
export function openAutoTag({api, scope, store, images, trigger, onTagged = () => {}, notify = showWorkspaceToast}) {
const targets = bulkTargets(images);
if (!targets) { notify(TOO_MANY); return; }
const listed = images.filter(imageTarget);
const local = createScope();
const options = createTaggingOptionsController({api});
// D85: the Tag image dialog's model panel without Generate; its Suggestions field is the job's count (D91).
const {element: panel, validate} = renderModelPanel({scope: local, controller: options,
heading: `Auto-tag ${countOf(listed.length, 'image')}`, generate: false}); // D118
let choice = APPLY_CHOICES[0];
const actions = el('div', undefined, 'dialog-actions');
const cancel = createButton('Cancel');
const start = createButton(choice.action, 'ca-primary');
actions.append(cancel, start);
// A model that is not downloaded: the reason, as the Tag image dialog's status line words it.
const status = el('div', undefined, 'small');
status.id = 'tag-generation-status';
status.setAttribute('role', 'status');
const choices = applyChoice(next => { choice = next; start.textContent = next.action; }); // D90
const render = state => {
if (status.textContent !== state.unavailable) status.textContent = state.unavailable;
if (state.canStart) start.removeAttribute('aria-disabled');
else start.setAttribute('aria-disabled', 'true');
};
local.cleanup(options.subscribe(render));
render(options.state);
cancel.addEventListener('click', () => closeWorkspaceDialog());
start.addEventListener('click', () => {
if (start.getAttribute('aria-disabled') === 'true' || !options.state.canStart || !validate()) return;
const request = options.request();
if (!request) return;
if (choice.value === 'review') {
// The first image's dialog replaces this one in the shell (D90).
startReviewSession({api, scope, images: listed, ...request, trigger});
return;
}
closeWorkspaceDialog();
startTagJob({store, scope, body: {mode: 'auto', targets, ...request}, notify}).then(job => {
if (job) followTagged({store, scope, job, images: listed, onTagged});
});
});
openContentDialog({scope, title: AUTO_TITLE, nodes: [panel, choices, status, actions], trigger, onClose: () => {
document.getElementById('modal')?.classList.remove('tagging-modal');
local.dispose();
options.dispose();
}});
document.getElementById('modal')?.classList.add('tagging-modal');
options.open();
}
@@ -0,0 +1,75 @@
/**
* D90 (ruling T3-bulk): "Review each image", the bulk Auto-tag session in the browser. The selected images are walked
* one Tag image dialog at a time (image-tagging.mjs `actions: 'review'`), each opened with the session's model and
* options and Generate already running. Approve saves that image as the dialog's Save does and advances; Skip
* advances without a write; Stop reviewing, Escape and × end the session. Each image is one gated `suggest` call, so
* the session holds no model between images; a refused one (`model-busy`) stays on its image with the dialog's own
* status and Retry until the user retries or skips. Ending toasts "<n> of <M> images tagged."; the selection stays.
*/
import { closeWorkspaceDialog } from './workspace-dialogs.mjs';
import { openImageTagging } from './image-tagging.mjs';
import { showWorkspaceToast } from './toast.mjs';
/** The closing toast: approved saves of the session's images (the plan's copy verbatim). */
export function reviewOutcome(tagged, total) {
return `${tagged} of ${total} images tagged.`;
}
/**
* Walk `images` (listing items with a tagging target, in selection order) with `model_type` and `options`.
* `onSaved(event)` follows every approved save. Returns `{stop()}`, which ends the session (its open dialog closes and
* writes nothing more). `open`, `close` and `notify` are the dialog seams (the defaults in the page).
*/
export function startReviewSession({api, scope, images, model_type: model, options, trigger = null, onSaved = null,
open = openImageTagging, close = closeWorkspaceDialog, notify = showWorkspaceToast}) {
if (!Array.isArray(images) || !images.length) throw new TypeError('A review needs its images');
const total = images.length;
let index = 0, tagged = 0, ended = false, handle = null;
function end() {
if (ended) return;
ended = true;
handle = null;
notify(reviewOutcome(tagged, total));
}
function advance() {
if (ended) return;
index += 1;
if (index >= total) {
// The last dialog handed over (advancing), so closing it is not a Stop.
end();
close();
return;
}
show();
}
function show() {
let current = true;
const settled = () => {
const was = current;
current = false;
return was;
};
handle = open({api, scope, image: images[index], source: 'bulk', trigger, preset: {model, options},
autoGenerate: true, actions: 'review', position: {index: index + 1, total},
onSaved: event => {
// A save that settles after its dialog was left (Skip or Stop while it ran) still counts.
tagged += 1;
onSaved?.(event);
if (settled()) advance();
},
onSkip: () => { if (settled()) advance(); },
onStop: () => { if (settled()) end(); },
});
}
show();
return {
stop() {
if (ended) return;
const open = handle;
end();
open?.close();
},
};
}
+7 -4
View File
@@ -23,10 +23,13 @@ export function openCardTagging({api, scope, image, action, embedded}) {
/**
* Follow confirmed tag saves until `scope` ends. `filtered()` tells whether a Tags filter or a search is active,
* `selection()` is the selected image key, and `onRefreshing(busy)` marks Load more busy while the re-read runs.
* Returns `refresh(keys)`, the same re-read for the files a bulk tagging job tagged (once per job, D85).
*/
export function followImageTagSaves({scope, stream, inspection, filtered, selection, onRefreshing}) {
scope.cleanup(onImageTagsSaved(async ({target}) => {
const key = `${target.root_index}:${target.path}`;
scope.cleanup(onImageTagsSaved(({target}) => refresh(new Set([`${target.root_index}:${target.path}`]))));
return refresh;
async function refresh(keys) {
if (filtered()) {
onRefreshing(true);
try {
@@ -37,6 +40,6 @@ export function followImageTagSaves({scope, stream, inspection, filtered, select
if (!scope.signal.aborted) onRefreshing(false);
}
}
if (!scope.signal.aborted && selection() === key) inspection.refresh();
}));
if (!scope.signal.aborted && keys.has(selection())) inspection.refresh();
}
}

Some files were not shown because too many files have changed in this diff Show More