Compare commits

..
Author SHA1 Message Date
Vito Sansevero 3c4d3b733b Merge remote-tracking branch 'origin/main' into feat/330-s13-pr3 2026-10-28 06:01:30 -07:00
Vito 3f085d0724 ci: full matrix once per PR in the merge queue; Windows/macOS non-blocking (#420) (#421)
* ci: run the full matrix once per PR, in the merge queue (#420)

Pull requests now run a fast subset: Python 3.12 on ubuntu-latest, the
coverage gate, black/flake8, frontend modules, bandit and the Chromium/touch
frontend coverage shards plus merge. The merge queue, pushes to main and
manual runs keep the full matrix: Linux/Windows/macOS x Python 3.12-3.14,
real ComfyUI compatibility and the Firefox/WebKit page workflows.

A dropped matrix leg never reports a check, so the new required "all tests"
gate requires every test.yml job to succeed outside pull requests and real
ComfyUI to be skipped only on pull requests; viewerjs does the same for the
browser shards. branch-protection.json drops the eight legs a pull request
no longer runs and adds "all tests".

* docs(ci): ruleset edit keeps every rule and integration id; find merge-queue runs

* ci: move Windows and macOS out of the blocking path (#420)

The maintainer deferred Windows and macOS to the end of the redesign.
The merge queue now runs Linux on Python 3.12-3.14, real ComfyUI, the
frontend coverage shards and the Firefox/WebKit page workflows; pull
requests keep the Linux 3.12 subset. The six Windows/macOS legs move to
the new portability.yml, which runs on push to main, nightly and on
demand, keeps the 27-minute limit and is never a required check. Red
runs there are fixed at their cause before the cutover (#334).

* docs(ci): required checks live in the ruleset and the classic protection (#420)
2026-10-28 12:44:06 +00:00
Vito Sansevero 7f6bac5392 Merge remote-tracking branch 'origin/main' into feat/330-s13-pr3 2026-10-28 05:05:10 -07:00
Vito e769845417 refactor(admin): open System for settings, maintenance, backup and restore (#330, PR 2) (#407)
* test(legacy): snapshot the routes and the settings response

Pins every registered route and the key paths of GET /prompt_manager/settings
while System S13 (#330) moves the legacy pages' tools to native System. Recorded
on the unchanged main; cutover (#334) removes it.

* test(legacy): prove each removed legacy control has native coverage

TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet).
legacy-coverage.test.mjs fails when a removal cites a capability that is not
covered, copies another native test, lists an id or method twice, or leaves
the id or method on its page. legacy-system-links.spec.mjs holds the System
link table and guards the kept admin flows.

* refactor(admin): open System for Diagnostics and logs

The admin page's Diagnostics and View Logs buttons become links that open
System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs
dialogs and their methods go; TESTING.md lists them against their covered
capabilities (rows B1–B7).

* refactor(admin): open System for settings and keep page options

Settings opens System › Overview in a new tab. The settings dialog becomes Page
options (opened from the list header) with only the Infinite Scrolling default,
and its Save posts {"infinite_scroll": …} alone, so a stale legacy form can no
longer overwrite scan folders or other values edited in System (#330 S13-9).
Scan folders, monitoring status, auto-hide, display mode, worker threads,
LoraManager and the LoRA import leave the page (rows B8–B14).

* refactor(admin): open System for maintenance

Maintenance opens System › Maintenance in a new tab, where each operation runs
as a reviewed Activity job. The one-click legacy Database Maintenance dialog
and its methods go (rows B15–B18); formatFileSize stays for the image modal.

* refactor(admin): open System for backup and restore

Backup DB and Restore DB open System › Backup & export in a new tab, where a
restore is staged, reviewed and run as an Activity job. The one-click Restore
dialog and its methods go (rows B19–B20); the scan dialog's quick backup stays.
The removed browser tests' data checks live on at the route: a new round trip
test restores every prompt with its tags and image links.

* test(legacy): find a removed modal id passed to showModal and settle the links spec

The removal check also finds an id passed as a string to a ...Modal() call and no
longer reads data-id= as id=. The no-request test waits for the page's own
start-up reads to finish, the sort check compares as the server sorts, and the
kept-flow test checks the added prompt shows in the list.

* test(admin): open Page options after the saved default is read

Drops the leftover LoraManager section comment above the Page options save.

* test(legacy): a lazy thumbnail load is not a System link's request
2026-10-28 11:25:20 +00:00
Vito Sansevero eb6b4284f0 feat(gallery): remove Default View Mode from Page options (#330)
System > Appearance's Default layout is the one default-layout control, as
the S13 design's G5 says. The legacy gallery's Page options no longer show
or save a default view and Reset no longer resets it. A stored legacy
defaultViewMode still picks the view the gallery opens in (grid otherwise),
and Save and Reset keep that key for the workspace-preferences migration.
2026-10-28 04:17:38 -07:00
Vito 9a90c7677f fix(exit): stop running jobs, downloads and encoders at Ctrl+C instead of waiting for them (#376) (#417)
* fix(exit): stop running jobs before Python joins the executor threads (#376)

ComfyUI returns from main.py after Ctrl+C without closing its event loop,
and Python joins every executor thread before any atexit callback runs, so
a scan batch, thumbnail batch or model download in flight kept ComfyUI
alive until it ended (on Windows each ffmpeg up to its 30 s timeout).

The package now registers a threading._register_atexit hook (after
concurrent.futures', so it runs before that join) that sets an exit signal
and kills the running encoder processes. Job.cancelled follows the signal,
so every check a job already makes between files stops it at exit;
map_parallel starts no further item, the scan walk stops between folders and
a tagging model download stops before its next chunk. Without the internal
hook exit behaves as before.

* fix(exit): abort Xet downloads and stop LoRA staging at exit (#376)

Review follow-ups: Xet transfers report progress from native code and ignore
the chunk checkpoint, so the exit hook calls huggingface_hub's own
abort_xet_session when it is loaded; an aborted download logs as stopped.
LoRA import staging reads the exit signal itself, so the legacy LoRA scan
stops too. The test's fake encoder outlives the ceiling by less, and
TESTING.md states the download limits of older huggingface_hub.
2026-10-28 10:15:28 +00:00
Vito 65b15bbe75 fix(logs): Clear all logs deletes and empties only what its confirmation showed (#389) (#412)
* fix(logs): Clear all logs deletes and empties only what its confirmation showed (#389)

- /logs/files rows carry an identity string (size:mtime_ns:ino); the
  workspace sends back the active and rotated rows its dialog was built
  from. Under the files and handler locks every listed copy is re-checked
  with _matches_review and the listed active log's inode; any difference
  refuses with nothing changed (409 logs-changed). A copy that appeared
  after the confirmation is kept.
- A second writer of the active log (our flushed stream is behind the
  file's size, or our file is now a rotated copy) refuses first (409
  logs-foreign-writer); each unlink is preceded by a fresh identity check
  and the active log is emptied last.
- An active log with a hard link (st_nlink > 1) is never truncated (409
  logs-hard-link), also on the file-logging-off path.
- The legacy admin page's POST without a body keeps clearing what is
  there, with the foreign-writer and hard-link refusals.
- The browser fixture grows the active log through the handler's stream.

* fix(logs): accept pre-1970 identities, refuse duplicate rows, say the foreign-writer check is best effort (#389)

Review follow-ups: a negative mtime_ns no longer makes every workspace
clear 400; a name listed twice is a malformed list; the confirmed-row cap
is 10000; the hard-link refusal says how to recover; the hard-link and
foreign-append tests assert the kept bytes before the refusal code, so on
the old code they fail on lost data.
2026-10-28 09:06:15 +00:00
Vito 95c4542c6f test(browser): API restores stay unseen by the open page's Activity poll (#413) (#416)
* test(browser): API restores stay unseen by the open page's Activity poll (#413)

Specs that restore through the API while a workspace page is open check
how the page handles a restore it did not see. The page's Activity store
reads the job list every 15 s; on a slow runner (CI run 40646112668, a
16 s restore) that read landed during the restore, the page followed the
job and, as designed (S9), showed the open page again when it finished,
closing the Create tag dialog the vocabulary spec was about to use.

unseenRestore holds the page's job-list reads until the restore has
answered, so they read it as already ended. Used by the vocabulary,
library editor and history specs.

* test(browser): arm the restore gate before the page loads and drain reads in flight (#413)

A job-list read the page sent before the restore could still be answered
while it ran. The gate now routes every job-list read from before the
first navigation and, before a restore, waits for the reads in flight to
be answered. The history and library editor conversions are pre-emptive:
same race, never seen failing.
2026-10-28 08:18:06 +00:00
Vito 534c95248a fix(review): anchor files on their saver in the scan and Needs review (#390) (#414)
* fix(review): anchor files on their saver in the scan and Needs review (#390)

The scan and Apply now resolve with the file's name, as the workspace
inspector does since #383, so the one saver whose filename_prefix names
the file anchors its branch. A file whose branch that anchor resolves is
never queued (from the inspection profile it offers no choice; a
default-ambiguous file asks saver_anchored) and its pending, undecided
items leave on the next scan. REVIEW_RULE_VERSION is folded into
inspection item versions so old items are re-evaluated. No match, several
matches or an unfinished walk anchor nothing. Apply refuses a choice the
anchor contradicts.

* fix(review): refuse a default choice the saver anchor contradicts (#390)

Apply re-resolves a default item with the file's name: when its saver
anchors the branch, no candidate is offered and the choice is refused.
The scan's anchor check skips bytes carrying a decision and remembers
unanchored file states per process, so a rescan does not inspect an
unchanged file again.

* fix(review): queue an anchored file with its saver's pair as the choice (#390)

Maintainer ruling: a file never falls through both the link and the
queue. When only the saver anchor resolves a file the default profile and
the parser floor leave unlinked, the scan queues it with the anchor's pair
as its one candidate (evidence anchor: saver), never linking it. Files
already linked or resolved by the default path are unchanged. Apply
accepts the anchored pair when the bytes now resolve to it, and an
anchored item is a choice the untraced settle keeps.

* fix(review): upgrade old default items of anchored files, tag anchored results (#390)

A pending default item recorded before the saver anchor no longer
shortcuts the rescan when only the anchor resolves the file: the
anchored item supersedes it. Apply accepts a resolved re-resolution only
when it is tagged as the saver anchor's, the anchored pair must carry a
positive text to choose, and limited anchored evidence still counts as a
choice. TESTING.md describes the ruling.
2026-10-28 07:22:48 +00:00
Vito e4dc781c26 fix(images): read a loading collection as loading, and de-flake the JXL fixture and restore specs (#391, #371, #361) (#405)
* fix(images): read a loading collection as loading, never as 0 of 0 images (#391)

A reset published {loading, loaded:0, total:0} and the caption painted it
as "0 of 0 images" until the first page answered, so a fast reader (the
inspector concept spec on WebKit) read an empty result the server never
returned. The count line now says "Loading images…" while a first page is
in flight; the spec waits for the answered count, and a held-listing spec
pins the ordering.

* test(thumbnails): make the no-JPEG-XL-encoder fixture authoritative (#371)

{available:false} stood in only the cached detection, and every settings
read probes a cached None again, finding the host's real encoder. The
fixture now stands in for the detector itself and reset restores it; a
unit test proves the route against a detector that finds an encoder.

* test(system): wait for the restore job's own end before its toast (#361)

The reviewed-restore specs waited for the "Database restore finished"
toast with the 5 s expect, in effect a deadline on the disk-bound publish.
They now wait for the job's progress stream to close, assert the job
finished, and only then check the toast with the default timeout.

* test(images): wait for the search's answered count in the concept comparison (#391)

searchOf took the first changed caption as the search's count, the same
read-before-the-response race as the inspector's tag chip.
2026-10-28 06:16:57 +00:00
Vito d0339128e4 test: isolate API fixture databases from other suites; CI lint pins, failing bandit hook, npm cache (#346, #352, #409) (#411)
* ci: pin black and flake8, let the bandit hook fail, cache npm in the JS job (#352)

The lint job installs black==26.10.1 and flake8==7.4.1 from
requirements-dev.txt (the versions CI resolved on main today), so a new
release cannot turn CI red with no code change.

The local bandit hook ran 'bash -c ... "$@"; true': it could never
fail, and bash bound the first staged file to $0 so it was never
scanned. It now runs 'bandit -lll -q' directly, failing on high
severity exactly like code-quality.yml.

The frontend modules job's setup-node caches npm keyed on
tools/vendor/package-lock.json.

* test(history): wait on the writer's state, not short wall-clock pauses (#352)

Four 0.08 s pauses before asserting that the worker had not probed yet
now wait until the idle worker has run a whole loop pass at the current
manual clock value (three clock reads), so a slow runner cannot end the
pause before the worker decides. A mutation that probes 20 ms early now
fails two tests deterministically.

The real-clock bounded close no longer asserts close(0.02) returned in
under 0.2 s wall time; it asserts the in-flight write is still blocked
and the worker alive, like the output writer test. The manual-clock
bound stays exact. The last 0.1 s non-blocking submit check uses the
1 s NON_BLOCKING_CEILING from #351.

* test(api): give each API fixture database its own temp folder (#346)

Every PromptManagerAPI sweeps its database's folder at startup for the
stages, uploads and safety-backup copies an interrupted restore left.
The API test fixtures created their database directly in the system
temp folder, so any other test process starting an API there (another
worktree's suite, a pre-push hook, an agent's targeted run) deleted
this process's in-flight restore files. The restore, epoch and
retirement tests then failed with 'staged restore changed', 'unable to
open database file' or 503 instead of 409. Busy machines fail because
they are running other suites, not because they are slow: the same
CPU load with a private TMPDIR passes.

private_temp_db() puts the fixture database in a new folder removed at
cleanup (with any safety backups a restore left beside it). A
regression test sweeps a stand-in shared folder in the middle of a
multipart restore, as another suite's startup would.

* test(history): count worker parks, hold saturation blocks until released (#352)

wait_for_worker_pass now counts the idle worker's parks on its wake
event (two parks from now = one whole pass begun after the call)
instead of assuming a fixed number of clock reads per pass, which a
future clock read would have made vacuous. The real-clock bounded
close keeps a loose upper bound (0.02 s + NON_BLOCKING_CEILING).

The two saturation tests (history and output writer) blocked the
worker with release.wait(3) while they queued 1023 more items. On a
loaded runner the fill took longer than 3 s, the worker unblocked on
its own and drained part of the queue (pending 868..1017 != 1024),
and cleanup then reported a leaked worker. The block now holds until
the test releases it; 60 s is only a hang guard, and cleanup always
sets the release. Under one core with 8 busy loops: 8/8 runs failed
before, 6/6 pass after.

* test(maintenance): wait on duplicate jobs with a disk-bound hang guard (#409)

On Windows py3.13 CI (PR #405) a one-file duplicate cleanup missed its
10 s WAIT. The test already waits on the job's own task; the 10 s bound
acted as a deadline on disk-bound work. In the same 50 s window the
following tests' setups took up to 9 s against 0.1-0.6 s around it
while the test bodies stayed fast, which is consistent with a
process-wide disk I/O stall on the runner rather than a lease, handle,
executor or job stall (the log has no thread dump to show more).

tests/disk_waits.py holds the one disk-bound hang guard (90 s per wait,
under the 120 s per-test watchdog), shared with RESTORE_SECONDS (#382).
Waits on find and cleanup job tasks in the duplicate-files tests use
it; gates and server-state polls keep WAIT. The keeper-validation test
turns the safety backup off, since it does not test backups.
2026-10-28 05:18:00 +00:00
Vito Sansevero 9e3c7bedd9 Merge branch 'feat/330-s13-pr2' into feat/330-s13-pr3
# Conflicts:
#	tests/browser/legacy-system-links.spec.mjs
2026-10-27 21:58:05 -07:00
Vito Sansevero 0e5fa52e76 Merge remote-tracking branch 'origin/main' into feat/330-s13-pr2
# Conflicts:
#	TESTING.md
#	tests/browser/legacy-system-links.spec.mjs
#	web/admin.html
#	web/js/admin.js
2026-10-27 21:49:00 -07:00
Vito Sansevero d2da8fc7f5 fix(gallery): let the header buttons wrap on a narrow screen
With Page options beside Settings the header no longer fits a phone's width;
it wrapped off the page and covered the Page options dialog in the touch runs.
2026-10-27 20:57:43 -07:00
Vito ef7e07c765 refactor(admin): open System for Diagnostics and logs (#330, PR 1) (#402)
* test(legacy): snapshot the routes and the settings response

Pins every registered route and the key paths of GET /prompt_manager/settings
while System S13 (#330) moves the legacy pages' tools to native System. Recorded
on the unchanged main; cutover (#334) removes it.

* test(legacy): prove each removed legacy control has native coverage

TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet).
legacy-coverage.test.mjs fails when a removal cites a capability that is not
covered, copies another native test, lists an id or method twice, or leaves
the id or method on its page. legacy-system-links.spec.mjs holds the System
link table and guards the kept admin flows.

* refactor(admin): open System for Diagnostics and logs

The admin page's Diagnostics and View Logs buttons become links that open
System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs
dialogs and their methods go; TESTING.md lists them against their covered
capabilities (rows B1–B7).

* test(legacy): find a removed modal id passed to showModal and settle the links spec

The removal check also finds an id passed as a string to a ...Modal() call and no
longer reads data-id= as id=. The no-request test waits for the page's own
start-up reads to finish, the sort check compares as the server sorts, and the
kept-flow test checks the added prompt shows in the list.
2026-10-28 03:54:55 +00:00
Vito 83fa80c330 fix(restore,threads): a stopped restore waits for its worker; exit-window thread refusals are quiet (#360, #375) (#404)
* fix(restore): a restore cut off by shutdown waits for its worker thread (#360)

A cancelled restore handler or job no longer removes the stage or the
upload, or ends its exclusive registry admission, while the executor
thread still stages or publishes. The cancel waits for the worker; a
publish that finished is adopted before the cancel goes on, and a stage
finished after the cancel is removed.

* fix(threads): a thread or pool refused at interpreter exit is quiet and leaves nothing pending (#375)

During interpreter exit, threading._shutdown shuts every thread pool
before the package teardown runs while our daemon threads keep working.
map_parallel now runs the items a refused pool did not start on the
calling thread, schedule() drops the path and returns False when its
worker cannot start, and the preview queue skips the file. Each
refusal is logged once at DEBUG without a traceback; other start
failures behave as before.

* fix(restore): keep the cancel and the adoption when a stopped restore's publish ends (#360)

A publish that fails after the cancel now ends in CancelledError (after
its prune) instead of a 500, and the restored library is adopted to its
end even when a second cancel arrives. The blocking calls go through
_run_in_executor again, as the executor contract test expects.

Exit-window refusals: the comments state what was measured (a pool is
refused during threading._shutdown; Thread.start only once the
interpreter is finalizing), and the tests assert the one DEBUG line.
2026-10-28 03:48:11 +00:00
Vito Sansevero f78716fc5d test(legacy): a lazy thumbnail load is not a System link's request 2026-10-27 19:58:41 -07:00
Vito Sansevero dfa4b3355f test(legacy): check each System link sends no request in its own test
The no-request check runs inside each link's test, so a slow page no longer
times out the whole table, and it ignores lazy thumbnail loads the layout
starts on its own.
2026-10-27 19:58:41 -07:00
Vito Sansevero 727a8dfd3a fix(gallery): keep the default view in Page options and save viewer choices alone
System › Appearance's Default layout is the workspace's own key, so the legacy
gallery's Default View Mode stays a page option instead of losing its only
control. The viewer's mute and loop buttons remember just that choice instead of
saving the Page options dialog's unfilled inputs over the stored page options.
The pending-thumbnails toast names System › Sources & scans.
2026-10-27 19:51:29 -07:00
Vito Sansevero bd5682e908 docs(system): name System as the home for settings and maintenance
README and TESTING.md send settings, maintenance, backups, diagnostics, the
gallery's tools and auto-tag model downloads to their System section and name
each legacy page's Page options.
2026-10-27 19:34:31 -07:00
Vito Sansevero e2d7cb1719 refactor(autotag): manage tagging models in System
In both auto-tag dialogs a model that is not downloaded shows a Manage in
System link to System › Auto-tagging (a new tab) instead of a Download button,
and the gallery's Unload Model becomes the same link. The legacy download
dialog and the download, cancel and unload methods go (rows B29–B31); the batch
and review stay. A test hook marks a fake model loaded.
2026-10-27 19:33:21 -07:00
Vito Sansevero 6527b6fe1d refactor(gallery): open System for settings and maintenance
The gallery's Settings opens System › Appearance in a new tab. Its settings
dialog becomes Page options (a new header button) with only the page's own
settings: thumbnail startup check, lazy loading, items per page, grid columns,
infinite scrolling, debug mode and API timeout. Save and Reset keep every other
stored key, so the Appearance values System saves survive (S13-13). Generate
Thumbnails, Clear cache, Rescan and Find duplicates leave the page; Page options
links System › Maintenance and Sources & scans (rows B21–B28).
2026-10-27 19:28:49 -07:00
Vito Sansevero 97e8b9b680 test(admin): open Page options after the saved default is read
Drops the leftover LoraManager section comment above the Page options save.
2026-10-27 19:09:21 -07:00
Vito Sansevero 9909bc5faa Merge branch 'feat/330-s13-pr1' into feat/330-s13-pr2 2026-10-27 19:08:01 -07:00
Vito Sansevero 8800defb98 test(legacy): find a removed modal id passed to showModal and settle the links spec
The removal check also finds an id passed as a string to a ...Modal() call and no
longer reads data-id= as id=. The no-request test waits for the page's own
start-up reads to finish, the sort check compares as the server sorts, and the
kept-flow test checks the added prompt shows in the list.
2026-10-27 19:07:44 -07:00
Vito e472a8bf74 fix(system): Tagging selects show their whole chosen option at 390 px (#398) (#399)
* fix(system): Tagging selects show their whole chosen option at 390 px (#398)

The concept's phone cap (120px) cut Existing tags ("Replace after review").
#tag-existing now never shrinks for its row text and keeps its natural width
at phone width, beside the text. The comparison expects selectsFit for every
select, and a new test walks every option of both selects at 390 and 320 px
in both themes. TESTING.md D132 records the decision.

* fix(system): bound Existing tags by its row at phone width (#398 review)
2026-10-28 02:03:22 +00:00
Vito Sansevero 341b813fd7 refactor(admin): open System for backup and restore
Backup DB and Restore DB open System › Backup & export in a new tab, where a
restore is staged, reviewed and run as an Activity job. The one-click Restore
dialog and its methods go (rows B19–B20); the scan dialog's quick backup stays.
The removed browser tests' data checks live on at the route: a new round trip
test restores every prompt with its tags and image links.
2026-10-27 18:49:59 -07:00
Vito Sansevero dae6b3f784 refactor(admin): open System for maintenance
Maintenance opens System › Maintenance in a new tab, where each operation runs
as a reviewed Activity job. The one-click legacy Database Maintenance dialog
and its methods go (rows B15–B18); formatFileSize stays for the image modal.
2026-10-27 18:45:10 -07:00
Vito Sansevero ba2803dbbf refactor(admin): open System for settings and keep page options
Settings opens System › Overview in a new tab. The settings dialog becomes Page
options (opened from the list header) with only the Infinite Scrolling default,
and its Save posts {"infinite_scroll": …} alone, so a stale legacy form can no
longer overwrite scan folders or other values edited in System (#330 S13-9).
Scan folders, monitoring status, auto-hide, display mode, worker threads,
LoraManager and the LoRA import leave the page (rows B8–B14).
2026-10-27 18:44:30 -07:00
Vito Sansevero 0abdcbc814 refactor(admin): open System for Diagnostics and logs
The admin page's Diagnostics and View Logs buttons become links that open
System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs
dialogs and their methods go; TESTING.md lists them against their covered
capabilities (rows B1–B7).
2026-10-27 18:41:38 -07:00
Vito Sansevero c0d27772b4 test(legacy): prove each removed legacy control has native coverage
TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet).
legacy-coverage.test.mjs fails when a removal cites a capability that is not
covered, copies another native test, lists an id or method twice, or leaves
the id or method on its page. legacy-system-links.spec.mjs holds the System
link table and guards the kept admin flows.
2026-10-27 18:18:18 -07:00
Vito Sansevero b26ae112e6 test(legacy): snapshot the routes and the settings response
Pins every registered route and the key paths of GET /prompt_manager/settings
while System S13 (#330) moves the legacy pages' tools to native System. Recorded
on the unchanged main; cutover (#334) removes it.
2026-10-27 18:13:52 -07:00
78 changed files with 4724 additions and 3439 deletions
+1 -8
View File
@@ -3,17 +3,10 @@
"strict": false,
"contexts": [
"py3.12 on ubuntu-latest",
"py3.13 on ubuntu-latest",
"py3.14 on ubuntu-latest",
"py3.12 on windows-latest",
"py3.13 on windows-latest",
"py3.14 on windows-latest",
"py3.12 on macos-latest",
"py3.13 on macos-latest",
"py3.14 on macos-latest",
"coverage gate (85%)",
"black and flake8",
"frontend modules",
"all tests",
"viewerjs",
"real ComfyUI compatibility",
"bandit (high severity)"
+55
View File
@@ -0,0 +1,55 @@
name: Windows and macOS
permissions:
contents: read
# Non-blocking portability runs (#420): never on pull requests or in the merge queue, and no context
# here is a required check. A red run is fixed at its cause before the workspace cutover (#334).
on:
push:
branches: [main]
schedule:
- cron: "17 3 * * *"
workflow_dispatch:
# Every run gets its own group (run_id), so no main commit's run is ever cancelled or dropped as pending.
concurrency:
group: ${{ github.workflow }}-${{ github.run_id }}
cancel-in-progress: false
env:
PYTHONFAULTHANDLER: "1"
# Time limit rule as in test.yml: ceil(1.5 x the slowest of three recent green runs on main) + 3
# minutes, never under 5. Measured 2026-10-26 in test.yml on runs 40257217681 (3be0ada),
# 40240706568 (fb9134f), 40228167328 (f9835aa), before these legs moved here in #420:
# slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
# (macos 6m14s-8m30s, windows 11m42s-15m54s)
jobs:
test:
name: py${{ matrix.python-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 27
strategy:
fail-fast: false
matrix:
os: [windows-latest, macos-latest]
python-version: ["3.12", "3.13", "3.14"]
steps:
- uses: actions/checkout@v7
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Run tests
run: python -m tests.run_suite -v
+48 -3
View File
@@ -28,10 +28,18 @@ env:
# py matrix, slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
# (ubuntu 5m27s-6m27s, macos 6m14s-8m30s, windows 11m42s-15m54s)
# 3.10 and 3.11 left the matrix and 3.14 joined it in #356; re-measure after three green main runs.
# Windows and macOS moved to portability.yml (#420), which keeps this 27-minute limit; this
# Linux-only matrix keeps it too until it is re-measured.
# real ComfyUI compatibility: 13m32s / 8m55s / 11m46s -> 24 min
# coverage gate (85%): 7m26s / 7m22s / 7m54s -> 15 min
# perf budget 30s / 29s / 27s, black and flake8 39s / 37s / 39s,
# frontend modules 23s / 19s / 29s -> 5 min (the floor)
# all tests: a result check without setup, 5 min (the floor)
#
# Pull requests run a fast subset (#420): Python 3.12 on ubuntu-latest, coverage, lint and frontend
# modules. The merge queue (merge_group), pushes to main and manual runs add Python 3.13 and 3.14 on
# Linux and the real ComfyUI job. "all tests" is the required gate that holds the queue to that set.
# Windows and macOS run in portability.yml (push to main, nightly, on demand) and never gate a merge.
jobs:
test:
name: py${{ matrix.python-version }} on ${{ matrix.os }}
@@ -40,8 +48,8 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ["3.12", "3.13", "3.14"]
os: [ubuntu-latest]
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["3.12"]') || fromJSON('["3.12", "3.13", "3.14"]') }}
steps:
- uses: actions/checkout@v7
@@ -63,6 +71,9 @@ jobs:
comfyui:
name: real ComfyUI compatibility
# Merge queue, main and manual runs only. A job skipped by its condition reports Success, so this
# required context never blocks a pull request; "all tests" fails if it is skipped anywhere else.
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
# Chromium and its system libraries come from the pinned Playwright image (tag = @playwright/test in
# tools/vendor/package-lock.json; tests/vendor/browser-image.test.mjs checks it). The runner install
@@ -221,7 +232,10 @@ jobs:
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install black flake8
cache: pip
cache-dependency-path: requirements-dev.txt
# The exact black and flake8 pins from requirements-dev.txt, so a new release cannot turn CI red.
- run: pip install $(grep -E '^(black|flake8)==' requirements-dev.txt)
- run: black --check .
- run: flake8 .
@@ -235,6 +249,8 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
cache-dependency-path: tools/vendor/package-lock.json
- name: Check shipped extension and page modules parse
run: |
@@ -248,3 +264,32 @@ jobs:
- name: Run JS tests
run: node --test tests/js/*.test.js
# The required context "all tests". A dropped matrix leg never reports a check at all, so the eight
# legs that pull requests skip cannot be required by name; this gate requires them instead. Every
# job must succeed; real ComfyUI must be skipped on a pull request and succeed on every other event.
all-tests:
name: all tests
needs: [test, comfyui, coverage, lint, js-test]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Require every test job for this event
env:
EVENT_NAME: ${{ github.event_name }}
TEST_RESULT: ${{ needs.test.result }}
COMFYUI_RESULT: ${{ needs.comfyui.result }}
COVERAGE_RESULT: ${{ needs.coverage.result }}
LINT_RESULT: ${{ needs.lint.result }}
JS_RESULT: ${{ needs.js-test.result }}
run: |
queue_only=success
if [ "$EVENT_NAME" = pull_request ]; then queue_only=skipped; fi
echo "event: $EVENT_NAME (real ComfyUI must be $queue_only)"
echo "test matrix: $TEST_RESULT"
echo "comfyui: $COMFYUI_RESULT"
echo "coverage: $COVERAGE_RESULT"
echo "lint: $LINT_RESULT"
echo "js-test: $JS_RESULT"
test "$TEST_RESULT" = success && test "$COVERAGE_RESULT" = success && test "$LINT_RESULT" = success && test "$JS_RESULT" = success && test "$COMFYUI_RESULT" = "$queue_only"
+10 -3
View File
@@ -165,6 +165,8 @@ jobs:
browser-workflows:
name: page workflows (${{ matrix.browser }}, shard ${{ matrix.shard }}/2)
# Merge queue, main and manual runs only (#420); pull requests run the Chromium/touch coverage shards.
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
# Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
# same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
@@ -219,8 +221,9 @@ jobs:
retention-days: 14
# Branch protection requires the exact context "viewerjs". This job keeps that
# name and fails unless every frontend job above succeeded (every shard, the
# coverage merge and every browser shard), including when one was cancelled or skipped.
# name and fails unless every coverage shard and the coverage merge succeeded and
# the browser shards succeeded, or, on a pull request only, were skipped (#420).
# Any failure, cancellation or other skip fails it.
viewerjs:
needs: [frontend-coverage, frontend-coverage-merge, browser-workflows]
if: always()
@@ -229,11 +232,15 @@ jobs:
steps:
- name: Require every frontend job to succeed
env:
EVENT_NAME: ${{ github.event_name }}
COVERAGE_RESULT: ${{ needs.frontend-coverage.result }}
MERGE_RESULT: ${{ needs.frontend-coverage-merge.result }}
WORKFLOWS_RESULT: ${{ needs.browser-workflows.result }}
run: |
queue_only=success
if [ "$EVENT_NAME" = pull_request ]; then queue_only=skipped; fi
echo "event: $EVENT_NAME (browser-workflows must be $queue_only)"
echo "frontend-coverage: $COVERAGE_RESULT"
echo "frontend-coverage-merge: $MERGE_RESULT"
echo "browser-workflows: $WORKFLOWS_RESULT"
test "$COVERAGE_RESULT" = success && test "$MERGE_RESULT" = success && test "$WORKFLOWS_RESULT" = success
test "$COVERAGE_RESULT" = success && test "$MERGE_RESULT" = success && test "$WORKFLOWS_RESULT" = "$queue_only"
+3 -1
View File
@@ -13,9 +13,11 @@ repos:
language: system
types: [python]
# Fails at the severity CI enforces (code-quality.yml: -lll, high only), never stricter.
# pre-commit appends the staged files, so no shell wrapper can drop the first one.
- id: bandit
name: bandit
entry: bash -c 'bandit "$@" -ll -q 2>&1; true'
entry: bandit -lll -q
language: system
types: [python]
exclude: ^(tests/|reference/)
+29 -38
View File
@@ -56,7 +56,7 @@ Both nodes include the complete PromptManager feature set:
- **📊 Analytics**: Track prompt usage patterns and effectiveness over time
- **🔬 Workflow Analysis**: Extract and analyze ComfyUI workflow data from PNG images
- **📋 Metadata Viewer**: Standalone tool for analyzing ComfyUI-generated images
- **🛠️ System Management**: Built-in diagnostics, backup/restore, and maintenance tools
- **🛠️ System Management**: Built-in diagnostics, backup/restore, and maintenance tools in System
- **🏷️ AI AutoTag**: Automatically tag your image collection using WD14 or JoyCaption vision models
- **🔗 LoRA Manager Integration**: Import LoRA metadata and preview images from [ComfyUI-Lora-Manager](https://github.com/willmiao/ComfyUI-Lora-Manager)
@@ -206,14 +206,11 @@ match Classic and Focused.
- **Responsive Design**: Works on desktop, tablet, and mobile devices
- **Real-time Search**: Instant search results as you type
- **Bulk Operations**: Edit multiple prompts simultaneously with batch tools
- **Settings Panel**: Configure behavior and display options
- **System Management**: Complete toolset for database operations:
- **Settings and System tools live in System**: the dashboard's **Settings** opens System › Overview, **Maintenance** opens System › Maintenance, **Backup DB** and **Restore DB** open System › Backup & export, and **Diagnostics** and **View Logs** open System › Diagnostics & logs, each in a new tab. System reviews every destructive operation before it runs it as an Activity job.
- **Page options**: the list header's **Page options** keeps the prompt list's Infinite Scrolling default
- **Dashboard tools**:
- **🔍 Metadata Viewer**: Launch standalone PNG analysis tool
- **📸 Scan Images**: Bulk scan output directories for ComfyUI images
- **💾 Backup/Restore**: Database backup and recovery operations
- **🔧 Maintenance**: Database optimization and cleanup tools including orphaned prompt removal
- **📋 Logs**: Comprehensive logging system with download capability
- **🔍 Diagnostics**: System health checks and troubleshooting
- **📸 Scan Images**: Bulk scan output directories for ComfyUI images (with a quick backup first)
- **Export Tools**: Download prompts and metadata in various formats
- **Progress Tracking**: Real-time progress indicators for long operations
- **Error Handling**: Graceful error display with recovery suggestions
@@ -333,14 +330,11 @@ The comprehensive gallery system provides complete access to all your ComfyUI ou
#### **⚙️ Gallery Settings**
Configure your viewing experience in the settings panel:
The gallery's **Settings** opens System › Appearance in a new tab: image quality, details on cards, collapsed viewer details, video autoplay/mute/loop, metadata loading and file paths are set there and apply to this page too.
- **Display Settings**: Default view, items per page, grid columns, media info, infinite scrolling
- **Performance Settings**: Image quality, lazy loading
- **Video Settings**: Autoplay, mute by default, loop videos
- **Thumbnail Management**: Generate optimized thumbnails for faster loading (runs on the configured worker threads; a run that stops at the per-run cap tells you how many remain)
- **Find Duplicate Images**: Hashes every file on the worker threads with a progress bar; close the window and it keeps running, results wait in a pill
- **Rescan Output Folder**: Drops the server-side listing cache and rewalks the folders
The gallery's **Page options** keep the page's own settings: the thumbnail check at startup, lazy loading, items per page, grid columns, infinite scrolling, debug mode and API timeout. Reset puts back only these.
Thumbnail generation, the server thumbnail and browser cache clears, refreshing the file listing and finding duplicate images are in System › Sources & scans and System › Maintenance (Page options links both); each runs as an Activity job you can follow after closing the page.
#### **🖼️ Image Viewing Experience**
@@ -398,7 +392,7 @@ Import existing ComfyUI images into your database:
- Any errors or issues encountered
5. **Access imported data** through the normal gallery interface
The scan reads PNG text chunks and A1111-style parameters, and, when `ffprobe` is installed, the prompt ComfyUI's video savers embed in MP4/MOV/WebM files. Files are read on the number of **worker threads** set in Settings → Performance (default: half your cores), which makes a large library several times faster.
The scan reads PNG text chunks and A1111-style parameters, and, when `ffprobe` is installed, the prompt ComfyUI's video savers embed in MP4/MOV/WebM files. Files are read on the number of **worker threads** set in System › Sources & scans (default: half your cores), which makes a large library several times faster.
### 🏷️ AI AutoTag
@@ -414,7 +408,7 @@ Automatically tag your entire image collection using AI vision models:
1. **AutoTag (Batch Mode)**: Tag your entire collection automatically
- Click **"🏷️ AutoTag"** in the admin dashboard
- Choose your model type (FP16 or GGUF)
- Choose your model type (FP16 or GGUF). A model that is not downloaded yet shows **Manage in System**: download (or unload) it in System › Auto-tagging, then reopen the dialog
- Select how to handle already-tagged images:
- **Skip images with existing tags**: Preserve your meticulous manual tagging work
- **Re-tag all images**: Overwrite existing tags with fresh AI analysis
@@ -445,7 +439,7 @@ Adjust the system prompt to match your tagging style:
#### **Requirements**
- Sufficient VRAM for the chosen model (FP16 requires more, GGUF is lighter)
- Models are downloaded automatically on first use
- Models are downloaded in System › Auto-tagging
- Feature requests and improvements are welcome!
### 📁 Folder Filter
@@ -470,18 +464,18 @@ If you use [ComfyUI-Lora-Manager](https://github.com/willmiao/ComfyUI-Lora-Manag
#### **Enabling the Integration**
1. Open the admin dashboard and click **Settings**
2. Scroll to the **Integrations** section — PromptManager will auto-detect if LoRA Manager is installed
1. Open System › Integrations (the admin dashboard's **Settings** opens System)
2. PromptManager auto-detects whether LoRA Manager is installed
![LoRA Enabled](images/pm-lora-enabled.png)
3. Toggle the **LoraManager** switch to enable
4. Optionally enable **Auto-inject trigger words** to automatically append LoRA trigger words when `<lora:name:weight>` is detected in your prompts
5. Click **Save Settings** (requires a ComfyUI restart to take effect)
5. Click **Save preferences** (requires a ComfyUI restart to take effect)
#### **Importing LoRA Data**
Click **Import LoRA Data** in the settings panel to start the import. A progress popup shows real-time status as each LoRA is processed:
Click **Review import** in System › Integrations; Start runs the import as an Activity job:
- Reads LoRA Manager's own settings the way LoRA Manager does: from the folder in `LORA_MANAGER_SETTINGS_DIR`, from the plugin's `settings.json` in portable mode, or from your user config folder, and only the keys that say where its files are
- Finds each LoRA's metadata in both storage modes: beside the model (the default) or in the central sidecar folder, and imports only LoRAs whose model file still exists in a LoRA folder (the review says how many metadata files it skipped)
@@ -502,7 +496,7 @@ A CivitAI API key is **required if your library contains NSFW LoRAs** — CivitA
To add your key:
1. Get your API key from [civitai.com/user/account](https://civitai.com/user/account)
2. Paste it into the **CivitAI API Key** field in the Integrations settings
2. Paste it into the **CivitAI API key** field in System › Integrations
3. Save and re-import to download any previously skipped images
The key is sent only over HTTPS to CivitAI's own hosts. CivitAI serves images through a redirect to its storage host; PromptManager follows such a redirect only over HTTPS to a host on CivitAI's own `civitai.com`, `civitai.red` or `civitai.green` domains, at most three hops, and never sends the key with it. When downloads fail, the import's status line says how many; when none could be downloaded, it starts with a warning. The server log has one warning per failed image with the reason (HTTP status, refused redirect, size cap, unreadable image, timeout or network error).
@@ -517,23 +511,20 @@ The comprehensive web interface provides:
- **Filter Options**: Filter by category, tags, rating, and date
- **Bulk Operations**: Select multiple prompts for batch editing
- **Export Tools**: Download your prompt collection
- **Settings Panel**: Configure auto-save and display options
- **Diagnostics**: System health checks and troubleshooting
- **Settings**: System (new tab) for settings, maintenance, backups and diagnostics; **Page options** for the page's own display settings
### 🔧 Database Maintenance
Keep your database optimized with built-in maintenance tools:
Keep your database optimized in System › Maintenance (the admin dashboard's **Maintenance** opens it in a new tab):
1. **Open the admin dashboard** and click **"🔧 Maintenance"**
2. **Select maintenance operations**:
- **Remove Duplicates**: Find and merge duplicate prompts while preserving images
- **Vacuum Database**: Optimize database file size and performance
- **Clean Orphaned Images**: Remove image records for missing files
- **🚮 Prune Orphaned Prompts**: Remove prompts that have no linked images
- **Check Consistency**: Verify database integrity and relationships
- **Database Statistics**: Generate detailed database reports
3. **Run selected operations** and monitor progress in real-time
4. **Review results** showing detailed information about each operation
- **Duplicate prompt text**: merge duplicate prompts while preserving images
- **Optimize database**: reclaim unused SQLite space
- **Missing-file links**: remove image records for missing files
- **Unused prompt components**: remove prompts nothing uses
- **Run health checks**: verify database integrity and relationships (the Health report)
- **Database size and prompt counts**: System › Diagnostics & logs (Database) and the Overview's Statistics
Each operation opens a review that shows what it will change and the safety backup it takes, then runs as an Activity job.
### Database Location
@@ -835,8 +826,8 @@ Before upgrading or changing this setting:
1. Identify the database your current installation actually uses and confirm its
library contents. Check both the configured path and the prior fallback
`<extension>/prompts.db`; keep track of which existing file you intend to use.
2. Preserve a consistent backup of the active database with the dashboard's
Backup/Restore backup action or SQLite's online backup API. Do not copy only
2. Preserve a consistent backup of the active database with
System › Backup & export's Back up or SQLite's online backup API. Do not copy only
the main file of a live SQLite database: committed data may still be in its
`-wal` file. For a filesystem copy, stop ComfyUI cleanly first and preserve any
remaining `-wal` file together with the main database.
+223 -24
View File
@@ -181,10 +181,12 @@ check, so run it before pushing a new or deleted frontend file instead of
waiting for the `frontend coverage` jobs.
The `viewerjs` check runs on every PR, including backend-only changes. It is an
aggregate job that passes only when every frontend job succeeds: the three
`frontend coverage (shard N/3)` jobs, `frontend coverage (merge)`, and the four
`page workflows (firefox|webkit, shard N/2)` jobs. A failed, cancelled or skipped
job fails it. Every browser job runs with `PM_TEST_WORKERS: 3` (the public
aggregate job. It requires the three `frontend coverage (shard N/3)` jobs and
`frontend coverage (merge)` to succeed on every event. The four
`page workflows (firefox|webkit, shard N/2)` jobs run only in the merge queue,
on `main` and on `workflow_dispatch` (see "Pull requests and the merge queue").
`viewerjs` requires them to be skipped on a pull request and to succeed
everywhere else. Any other failed, cancelled or skipped job fails it. Every browser job runs with `PM_TEST_WORKERS: 3` (the public
runners have 4 vCPUs). Each coverage shard runs `test:coverage:shard -- N/3`
(shard 1 also runs the node units and the viewer suite, shard 3 the vendor pins
and tooling tests) and uploads its raw V8 files as `coverage-raw-N` (kept 3
@@ -271,6 +273,13 @@ gh run view <run id> --json jobs # shows a job that alr
gh run view --job <job id> --log-failed # that job's failed log, at once
```
Python 3.13/3.14, real ComfyUI and Firefox/WebKit failures first show up in the
merge queue, on the merge-group commit rather than the
pushed one. Find that run with `gh run list --event merge_group --workflow test.yml`
(or `viewerjs-check.yml`) and match its head branch
(`gh-readonly-queue/main/pr-<N>-...`). Windows and macOS failures show up only
after the merge, in `gh run list --workflow portability.yml`.
Start fixing as soon as a failed log shows the cause, but push the fix only after
the run for that commit has finished: pull-request runs use `cancel-in-progress`,
so a push cancels the running jobs and can hide a second failure.
@@ -313,6 +322,26 @@ from the tests' package unload (`cleanup_package`), and it removes its own
ComfyUI-Manager restart through `os.execv`, a closed Windows console, or
ComfyUI Desktop stopping its server. SQLite's WAL journal covers those.
Jobs running at exit (#376): before any `atexit` callback, Python joins every
executor thread (`concurrent.futures` registers that join with
`threading._register_atexit`). The package registers its own callback there
(`utils/exit_signal.install`, after `concurrent.futures`, so it runs first):
it sets the exit signal and kills the running encoder processes
(`shutdown_encoders`, the ffmpeg that never sees Ctrl+C on Windows). The call in
flight then stops at its next safe point: `Job.cancelled` is True, so every
check a job already makes between files stops it, `map_parallel` starts no
further item (`ExitRequested`), the scan walk stops between folders, LoRA
import staging stops before its next sidecar (also for the legacy scan) and a
tagging model download stops before its next chunk where huggingface_hub takes
`tqdm_class` (1.x removes its temp file; 0.x `hf_hub_download` does not take it
and finishes the file). Xet transfers ignore that checkpoint, so the hook also
calls hub's own `abort_xet_session` when huggingface_hub is loaded. The
trigger-word cache load (small JSON sidecars) has no checkpoint. No check sits
inside a library write. That hook is
CPython-internal; without it (or when it is refused) `install` returns False and
exit waits for the call in flight, as before. `tests/test_exit_cancel.py` runs a
child process with three such calls running and requires its exit within 15 s.
Stop contract: `PromptTracker.shutdown(timeout)`,
`ImageGenerationHandler.shutdown(timeout)` (restartable),
`ImageGenerationHandler.close(timeout)` (final: `schedule()` refuses and a
@@ -519,29 +548,95 @@ Before changing paths, SQLite, streams, restore or cleanup:
open with `FirstWriterHold`) and give peers that must all succeed the
ordinary busy timeout; cross-thread handoffs wait on events with generous
ceilings, never a short wall clock (#242).
- Run native Windows CI early for these changes and include a Windows impact and
evidence note in the PR. All twelve OS/Python combinations and every applicable
- Run native Windows CI early for these changes. Pull requests and the merge
queue run only Linux, so start Windows and macOS on the branch with
`gh workflow run portability.yml --ref <branch>`.
Include a Windows impact and evidence note in the PR. All six Windows/macOS
combinations in that run, every Linux leg and every applicable
exact-head check must pass before merge. Portable `ntpath`/`PureWindowsPath`
controls do not establish native filesystem, locking or cleanup success.
## Pull requests and the merge queue
Each PR runs the full matrix once, in the merge queue (#420). The event decides
what runs:
| Job | `pull_request` | `merge_group`, push to `main`, `workflow_dispatch` |
| --- | --- | --- |
| `py3.12 on ubuntu-latest` | runs | runs |
| `py3.13 on ubuntu-latest`, `py3.14 on ubuntu-latest` | not created | run |
| `real ComfyUI compatibility` | skipped | runs |
| `coverage gate (85%)`, `black and flake8`, `frontend modules`, `bandit (high severity)` | run | run |
| `frontend coverage (shard N/3)`, `frontend coverage (merge)` (vendor pins and tooling tests on shard 3) | run | run |
| `page workflows (firefox\|webkit, shard N/2)` | skipped | run |
| `all tests`, `viewerjs` (gates) | run | run |
| `perf budget` | skipped | push and dispatch only (not in the queue) |
| `py3.12-3.14 on windows-latest/macos-latest` (`portability.yml`) | not run | push to `main`, nightly, `workflow_dispatch` only; never required |
GitHub has one required set for pull requests and the queue. A job skipped by
its `if:` reports Success. A dropped matrix leg never reports, so it would stay
pending. That is why `all tests` (in `test.yml`) and `viewerjs` carry the
guarantee:
- On a pull request they require the subset to succeed and the queue-only jobs
to be skipped.
- On every other event they require every job to succeed.
So a merge group cannot turn green unless all three Linux Python legs, real
ComfyUI and the four browser shards passed.
Windows and macOS (Python 3.12-3.14) are out of the blocking path for now
(maintainer, 2026-10-28). They run in `portability.yml` on every push to `main`,
nightly and on demand, and no context there is required. A red portability run
is still a failure: fix it at its cause, not by rerunning, and have every one
green before the workspace cutover (#334). `tests/vendor/ci-workflow.test.mjs` pins the
event split and runs both gate scripts for each event.
## Required merge checks
`.github/branch-protection.json` records the required checks: all twelve Python/OS
combinations, Python coverage, formatting, frontend modules, `viewerjs`, `real ComfyUI compatibility`,
and `bandit (high severity)`.
The branch must be current with main, and administrators follow the same checks.
`.github/branch-protection.json` records the required checks:
`py3.12 on ubuntu-latest`, `coverage gate (85%)`, `black and flake8`,
`frontend modules`, `all tests`, `viewerjs`, `real ComfyUI compatibility` and
`bandit (high severity)`. Each one reports on a pull request.
The same set lives in two places, and both must change together: the `main`
ruleset (25960526, which also holds the merge queue) and the classic branch
protection on `main`, which the manifest mirrors. GitHub requires the union of
the two, so a context left in either one blocks every PR that does not report it.
It does not require an additional human approval.
A repository administrator applies the configuration after the new workflow is
passing, then verifies it:
```sh
gh api repos/ComfyAssets/ComfyUI_PromptManager/rulesets/25960526 \
| jq '.rules[] | select(.type == "required_status_checks") | [.parameters.required_status_checks[].context]'
```
A PUT replaces the whole `rules` array, so edit the current ruleset in place.
Keep the `deletion`, `non_fast_forward` and `merge_queue` rules, and give every
context `integration_id` 15368 (GitHub Actions), so no other integration can
post a required check. Save the current ruleset first; it is the rollback:
```sh
gh api repos/ComfyAssets/ComfyUI_PromptManager/rulesets/25960526 > ruleset-before.json
jq '{name, target, enforcement, conditions, bypass_actors, rules: [.rules[]
| if .type == "required_status_checks"
then .parameters.required_status_checks = [$ARGS.positional[] | {context: ., integration_id: 15368}]
else . end]}' ruleset-before.json --args "py3.12 on ubuntu-latest" "..." > ruleset-new.json
gh api --method PUT repos/ComfyAssets/ComfyUI_PromptManager/rulesets/25960526 --input ruleset-new.json
```
Afterwards, read it back. Check that the `merge_queue` rule is still there and
that every context keeps its `integration_id`. Then apply the manifest to the
classic protection and read both back:
```sh
gh api --method PUT repos/ComfyAssets/ComfyUI_PromptManager/branches/main/protection \
--input .github/branch-protection.json
gh api repos/ComfyAssets/ComfyUI_PromptManager/branches/main/protection/required_status_checks
gh api repos/ComfyAssets/ComfyUI_PromptManager/rules/branches/main
```
Update the manifest whenever required job names change. Keep the frontend check
Update the manifest whenever
required job names change. Keep the frontend check
free of path filters so a required check is never silently skipped.
## Unified workspace
@@ -2439,6 +2534,14 @@ production hub `#/overview` is unchanged (D28); its System card opens
the pressed section button and disposes the previous section's scope, so its reads,
listeners and dialogs never outlive it.
System is the home for settings and maintenance (#330 System S13). The legacy pages keep
their prompt list, tags, Scan Images, auto-tag batch and review and gallery browsing; their
Settings, Diagnostics, View Logs, Maintenance, Backup DB, Restore DB, the gallery's
thumbnail, cache, rescan and duplicate tools and the auto-tag model downloads open the
System section that owns them in a new tab (`legacy-system-links.spec.mjs`). Each page
keeps only its own settings in Page options. What left the legacy pages is listed in
"Legacy System UI removed by S13".
Drafts (`system-state.mjs`) live for the page session (Q10). Save writes the dirty
browser fields, then the host fields (`POST /prompt_manager/settings` with
`expected_revision`, #220), then the runtime fields. A failed domain keeps its drafts and
@@ -3983,7 +4086,30 @@ modals use the same routes and keep working.
other: each file's identity is checked just before its unlink.
- **Clear all logs** (`POST /logs/truncate`, also the admin page's Clear) empties the
active log through its handler, deletes the rotated copies (Rule R) and clears the
buffer; other files and links are kept. Response keys are unchanged.
buffer; other files and links are kept. Response keys are unchanged. It deletes or
empties no more than its confirmation showed (#389):
- `/logs/files` rows carry `identity`, the string `"<size>:<mtime_ns>:<ino>"` (a
string: `mtime_ns` and many inodes are beyond a browser's exact integers). The
workspace sends `{"expected": [{name, identity}]}` with the active and rotated rows
its dialog was built from. Under the files lock and every handler lock each listed
rotated copy must still match (`_matches_review`) and the listed active log must be
the same inode, or nothing changes: 409 `logs-changed`. Only listed copies are
deleted; one that appeared after the confirmation is kept. A malformed list is 400.
- Without a body (the legacy admin page, whose confirm lists nothing; also the
workspace when the file list could not be read and the dialog said "deletes its
rotated copies") every rotated copy present at the locked rescan is deleted.
- Either way, before anything changes (best effort, not a cross-process lock: a
write before our own last write, or any while file logging is off here, is not
seen): a second writer of the active log (our
handler's flushed stream is behind the file's size, or its file is now a rotated
copy: another ComfyUI sharing the folder) is 409 `logs-foreign-writer`; an active
log with more than one name (`st_nlink > 1` by `lstat` or the handler's `fstat`,
NTFS included) is 409 `logs-hard-link`, never truncated (replacing it instead would
rename a file our handler holds open, which Windows refuses). Each copy is
`lstat`-checked again just before its unlink; one that changed stops the clear there
and keeps it, the copies after it and the active log, which is emptied last.
- The browser fixture grows the active log through the handler's own stream, since a
second handle appending to it is exactly what Clear refuses.
- **Runtime settings** (`GET/POST /logs/config`): the level (DEBUG…CRITICAL) and console
logging apply until the server restarts; the retention values are shown read-only in
the workspace and the API keeps accepting them.
@@ -4030,6 +4156,9 @@ auto-tag / review", which the Images selection bar owns, not a System section. A
is not `covered` names none (`—`). Test is the exact title of a test that runs, then its
file. Anchors are line numbers on `main` at this PR.
S13 (#330) removed the legacy copy of the `covered` rows the legacy pages linked to System;
"Legacy System UI removed by S13" below lists every removed element against its row.
`tests/vendor/legacy-coverage.test.mjs` fails on a row with any of these problems:
- no owner, or a status outside the three;
- a malformed Native control cell;
@@ -4103,6 +4232,62 @@ list stay unscoped: any row's control proves them.
| API timeout `apiTimeout` | `gallery-settings.js:37` | Legacy gallery (S12-10; read by no request) | kept in legacy | — | defaults are a complete, sane settings object (gallery-settings.test.js) |
| config.json-only keys | `py/config.py:212–234,722–739`; `py/api/diagnostics_report.py:185` | S12 report (read-only) | covered | `diagnostics` · button · "Download diagnostic report" | test_report_lists_config_only_keys (test_api_diagnostics_report.py) |
### Legacy System UI removed by S13
System S13 (#330) points the legacy pages' System buttons at native System (a link that
opens `workspace#/system/<id>` in a new tab) and removes the legacy copy of a capability
only when its row above is `covered`. One row per removed capability: Capability is the
exact Capability cell of its coverage row, Page is `admin` or `gallery`, Removed ids and
Removed methods list what went (`—` for none), and Native test copies that row's Test
cell. An id that stays on a System link is not listed, only its handler. The gallery's
Default View Mode left Page options too: System › Appearance's Default layout is the one
default-layout control. Nothing writes the legacy `defaultViewMode` any more; a stored
value still picks the view the legacy gallery opens in (grid otherwise), and Page options
Save and Reset keep it.
`tests/vendor/legacy-coverage.test.mjs` fails when a row cites a capability that is not
`covered` or copies another test, when an id or method is listed twice for one page, or
when a listed id is still on its page (`id="…"` in the markup or the script, a
`getElementById`/`querySelector('#…')` lookup, an entry of a modal-id list) or a listed
method is still defined or called (`.name(`). A deletion listed nowhere is caught by the
kept-flow tests (`kept admin flows still work` in `legacy-system-links.spec.mjs` and the
legacy page workflows), not by this check.
| Capability | Page | Removed ids | Removed methods | Native test |
| --- | --- | --- | --- | --- |
| Diagnostics run | admin | `#diagnosticsModal`, `#runDiagnosticsBtn`, `#diagnosticsContent` | `showDiagnosticsModal()`, `closeDiagnostics()`, `runDiagnostics()`, `renderDiagnostics()` | Run checks opens the shared Health report (workspace-system-diagnostics.spec.mjs) |
| Synthetic image-link test | admin | `#testImageLinkBtn` | `testImageLink()` | Create test link confirms first and names the prompt (workspace-system-diagnostics.spec.mjs) |
| View logs with level and limit | admin | `#logsModal`, `#logLevel`, `#logLimit`, `#refreshLogsBtn`, `#logsContainer` | `showLogsModal()`, `closeLogs()`, `refreshLogs()`, `displayLogs()` | level filter is minimum-level over all six choices (workspace-system-diagnostics.spec.mjs) |
| Log statistics | admin | `#logStatsTotal`, `#logStatsErrors`, `#logStatsWarnings`, `#logStatsLevel`, `#logStatsSize` | `loadLogStats()`, `formatBytes()` | log statistics show entries, errors, warnings, files and size (workspace-system-diagnostics.spec.mjs) |
| Log files list and download | admin | `#downloadLogsBtn`, `#logFilesList` | `loadLogFiles()`, `downloadLogs()`, `downloadLogFile()` | each listed file downloads by its returned name and a link has no Download (workspace-system-diagnostics.spec.mjs) |
| Clear/truncate logs | admin | `#clearLogsBtn` | `clearLogs()` | Clear all logs states both counts, confirms first and reads everything again (workspace-system-diagnostics.spec.mjs) |
| Set log level (incl. CRITICAL), console logging | admin | `#updateLogConfigBtn`, `#setLogLevel`, `#consoleLogging` | `loadLogConfig()`, `updateLogConfig()` | runtime level (including CRITICAL) and console logging save and say they apply until the server restarts (workspace-system-diagnostics.spec.mjs) |
| Image scan directories add/remove | admin | `#scanPathsList`, `#addScanPathBtn` | `renderScanPaths()`, `addScanPath()`, `removeScanPath()`, `_collectScanPaths()` | Add source validates on the server, then marks the host draft dirty; Save persists (workspace-system-sources.spec.mjs) |
| Currently monitoring list + Refresh | admin | `#refreshMonitoringStatus`, `#monitoringStatus` | `updateMonitoringStatus()` | watching only with a live observer; restart pending otherwise (workspace-system-sources.spec.mjs) |
| Search results auto-hide `result_timeout` | admin | `#resultTimeout` | — | host fields round-trip through /settings with revision (workspace-system.spec.mjs) |
| Web UI display mode | admin | `#webuiDisplayMode` | — | host fields round-trip through /settings with revision (workspace-system.spec.mjs) |
| Worker threads | admin | `#workerThreads`, `#workerThreadsValue` | `renderWorkerThreads()` | worker range max is cpu_count (workspace-system-sources.spec.mjs) |
| LoraManager badge, enable, path, trigger words, CivitAI key, clear key | admin | `#loraDetectionBadge`, `#loraEnabled`, `#loraSettings`, `#loraManagerPath`, `#loraTriggerWords`, `#civitaiApiKey`, `#clearCivitaiKey` | `detectLoraManager()`, `_bindLoraEvents()`, `saveLoraSettings()` | LoRA rows show detected / enabled / cache-loaded separately (workspace-system-integrations.spec.mjs) |
| LoRA data import with progress | admin | `#loraImportModal`, `#loraImportBtn`, `#loraImportStatus`, `#loraImportPercent`, `#loraImportBar`, `#loraImportProcessed`, `#loraImportImported` | `runLoraImport()` | Review import shows replacement, blocked items and network effects with the backup box; Start opens the job (D107) (workspace-system-integrations.spec.mjs) |
| Maintenance cleanup_duplicates, vacuum, cleanup_orphaned_images, prune_orphaned_prompts | admin | `#maintenanceModal`, `#cleanup_duplicates`, `#vacuum`, `#cleanup_orphaned_images`, `#prune_orphaned_prompts`, `#runMaintenanceBtn`, `#maintenanceResults` | `showMaintenanceModal()`, `closeMaintenance()`, `runMaintenance()`, `renderMaintenanceResults()` | Apply reviewed operation closes, toasts, opens the job in Activity (workspace-system-maintenance.spec.mjs) |
| Maintenance check_hash_duplicates, check_consistency | admin | `#check_hash_duplicates`, `#check_consistency` | — | Run health checks opens Health report and updates metric 4 (workspace-system.spec.mjs) |
| Maintenance statistics (database info) | admin | `#statistics` | — | meta-grid shows real database and source status, the pinned Viewer.js version and the host renderers (workspace-system-diagnostics.spec.mjs) |
| Maintenance select all / clear all / run | admin | `#selectAllMaintenanceBtn`, `#clearAllMaintenanceBtn` | `selectAllMaintenance()`, `clearAllMaintenance()` | plan dialog shows the server's scope, protected, backup policy and media lines with the backup box (D100); Cancel writes nothing (workspace-system-maintenance.spec.mjs) |
| Database backup | admin | — | `backupDatabase()` | Back up downloads through the api client and revokes the object URL (D103) (workspace-system-backups.spec.mjs) |
| Restore from file | admin | `#restoreModal`, `#restoreFileInput`, `#confirmRestore`, `#cancelRestore` | `showRestoreModal()`, `confirmRestore()` | Review restore stages the chosen file and shows the real summary with the backup box (D105) (workspace-system-backups.spec.mjs) |
| Gallery thumbnail generation | gallery | `#generateThumbnailsBtn`, `#thumbnailStatus`, `#thumbnailProgress`, `#progressText`, `#progressPercent`, `#cancelThumbnailsBtn`, `#progressBar`, `#progressDetails`, `#progressETA` | `generateThumbnails()`, `generateThumbnailsWithProgress()` | Start closes the dialog, toasts, and opens the job in Activity (workspace-system-sources.spec.mjs) |
| Default view mode, image quality | gallery | `#defaultViewModeSelect`, `#imageQualitySelect` | — | defaults apply to a new visit but not to the current view or a saved view (workspace-system.spec.mjs) |
| Image info on cards; sidebar collapsed | gallery | `#showImageInfoToggle`, `#sidebarCollapsedByDefaultToggle` | — | added rows reach their consumers (workspace-system.spec.mjs) |
| Video autoplay, mute, loop | gallery | `#videoAutoplayToggle`, `#videoMuteToggle`, `#videoLoopToggle` | — | video defaults reach the viewer (workspace-system.spec.mjs) |
| Auto-load metadata, cache metadata, show file paths | gallery | `#autoLoadMetadataToggle`, `#cacheMetadataToggle`, `#showFilePathsToggle` | — | metadata off shows an explicit load and loads once on request (workspace-system.spec.mjs) |
| Browser cache clear | gallery | `#clearCacheBtn` | `clearCache()` | Metadata cache reviews and clears only owned browser entries when there are some (D102) (workspace-system-maintenance.spec.mjs) |
| Clear server thumbnails | gallery | — | — | Thumbnail cache opens the S5 dialog (workspace-system-maintenance.spec.mjs) |
| Refresh file listing | gallery | `#rescanFolderBtn` | `rescanFolder()` | Refresh files calls /gallery/rescan and toasts its counts (workspace-system-sources.spec.mjs) |
| Duplicate image scan, review, delete | gallery | `#scanDuplicatesBtn`, `#dupScanPill`, `#dupScanPillText`, `#duplicatesModal`, `#closeDuplicatesBtn`, `#duplicatesScanStatus`, `#dupProgressText`, `#dupProgressBar`, `#duplicatesContent`, `#duplicatesList`, `#duplicatesFooter`, `#duplicatesCount`, `#selectAllBtn`, `#removeDuplicatesBtn` | `scanDuplicates()`, `attachDuplicateStream()`, `resumeDuplicateScan()`, `applyDuplicateEvent()`, `renderDuplicateProgress()`, `setDuplicateButton()`, `showDuplicatePill()`, `hideDuplicatePill()`, `openDuplicatesModal()`, `showDuplicatesModal()`, `hideDuplicatesModal()`, `displayDuplicates()`, `selectAllDuplicates()`, `removeDuplicates()` | Find duplicates starts a job; its view links Review groups; Keep radios build a duplicate_files plan (D101) (workspace-system-maintenance.spec.mjs) |
| Auto-tag model download | admin | `#autoTagDownloadModal`, `#downloadModelName`, `#downloadStatus`, `#downloadProgressPercent`, `#downloadProgressBar`, `#cancelDownloadBtn` | `downloadModel()`, `cancelDownload()` | Download starts the job and Activity shows it without Cancel (workspace-system-tagging.spec.mjs) |
| Auto-tag model download | gallery | `#autoTagDownloadModal`, `#downloadModelName`, `#downloadStatus`, `#downloadProgressPercent`, `#downloadProgressBar`, `#cancelDownloadBtn` | `downloadModel()`, `cancelDownload()` | Download starts the job and Activity shows it without Cancel (workspace-system-tagging.spec.mjs) |
| Unload model | gallery | — | `unloadModel()` | Unload is enabled only while a model is loaded, and a held gate's refusal names the operation (workspace-system-tagging.spec.mjs) |
### Browser cache ownership (#222)
The legacy gallery's Clear cache used to delete every localStorage key starting `pm_` or
@@ -4120,17 +4305,15 @@ page writes a key that is neither a preference nor inventoried, writes one it ca
CacheStorage or IndexedDB; `the storage scan reports every way of writing browser
storage` pins those patterns. It skips `web/lib` (third-party builds) and the pinned
Toolkit build (`web/js/shared/toolkit`, never hand-edited, verified by `npm run check`;
its storage belongs to the Toolkit repository and a new pin is reviewed on its own). The
gallery's server thumbnail clear stays its own request and runs first: refused or
unreachable, it changes nothing and says so (the old code toasted success when the
request failed); refused browser storage is reported, never hidden
(`tests/js/legacy-refusals.test.js`; the browser case `the gallery cache clear keeps
preferences and other tools caches` in `workflows.spec.mjs`).
its storage belongs to the Toolkit repository and a new pin is reviewed on its own). Since
System S13 (#330) the legacy gallery no longer offers Clear cache: System › Maintenance's
Metadata cache and Thumbnail cache reviews own both parts (removal table above), and
`cache-ownership.js` serves the native review.
The legacy duplicates dialog's View Full now names its source (`?root=<index>`, with the
path URL-encoded) when the gallery's output folder is a listed source; before, the serve
route opened the first source holding a file at that path
(`test_view_full_names_the_gallery_folder_source`).
The duplicates dialog's View Full named its source (`?root=<index>`, with the path
URL-encoded) when the gallery's output folder is a listed source
(`test_view_full_names_the_gallery_folder_source`); S13 removed the legacy dialog, and the
route keeps that behaviour for System › Maintenance's duplicate review.
### Library location contract (#249 System S6b)
@@ -4596,7 +4779,23 @@ files, oldest forgotten first, keyed by 16-byte digests: 8.6 MiB measured at
the cap). A file whose inspection raises is logged, counted in
`review_errors` and reported `unresolved`; the scan goes on. An older version's
pending, undecided item of bytes now untraced leaves the queue
(`settle_stale_review_items`), so an item never stays stuck at 409. The 16-candidate limited
(`settle_stale_review_items`), so an item never stays stuck at 409.
Saver anchor (#390, `tests/test_review_saver_anchor.py`): the scan and Apply
resolve with the file's name as the workspace inspector does, so the one saver
whose `filename_prefix` names the file anchors its branch. A file is never both
unlinked and unqueued (maintainer ruling): when only that anchor resolves it
(the named inspection run resolves, the unnamed one does not) and neither the
default profile nor the parser floor links it, the scan queues it with the
anchor's pair as its one candidate (evidence `anchor: "saver"`; the page
preselects it), never linking it. Such an item counts as a choice for the
untraced settle. Files already linked or resolved by the default path are
unchanged. No match, several matches or an unfinished walk anchor nothing, and
the file is queued as before. `REVIEW_RULE_VERSION` is folded into inspection
item versions, so older inspection items are recorded again once. Apply accepts
the anchored pair and refuses a choice the anchor contradicts. Measured on the
maintainer's 40,890 outputs: the queue is unchanged (249 files); the 242
`gpu_unlock_bench` files the inspector resolves are linked by the parser
floor. The 16-candidate limited
rule and the 8 MiB bound apply unchanged. The queue heals itself: after each batch, the scan removes the pending
items of every listed file (read or skipped as known) that a saved row now
links, and the monitor does the same for each ambiguous output it links
+38
View File
@@ -355,6 +355,44 @@ async def _stop_background_work_on_cleanup(app):
_stop_background_work()
def _kill_encoders_at_exit():
"""Run by the exit hook before Python joins the executor threads (#376):
a JPEG XL or ffmpeg encoder (no console on Windows, so Ctrl+C never
reaches it) is killed instead of being waited for."""
from .py.api.thumbnail_encode import shutdown_encoders
shutdown_encoders()
def _abort_downloads_at_exit():
"""Run by the exit hook (#376): huggingface_hub's Xet transfers ignore
the chunk checkpoint, so they are aborted the way hub aborts on Ctrl+C."""
import sys
if "huggingface_hub" not in sys.modules:
return # no download ever ran: import nothing at exit
from .py.autotag import abort_hub_downloads
abort_hub_downloads()
try:
# Before the executor join: running jobs stop at their next file, item or
# chunk (#376). The teardown below still runs from atexit afterwards.
from .utils import exit_signal as _exit_signal
_exit_signal.install(_kill_encoders_at_exit)
_exit_signal.install(_abort_downloads_at_exit)
except Exception as e:
try:
from .utils.logging_config import get_logger
get_logger("prompt_manager.init").error(
f"Failed to register the exit signal: {e}"
)
except Exception:
pass
try:
atexit.register(_stop_background_work)
from .py.config import server_instance as _teardown_server
+4 -2
View File
@@ -229,13 +229,15 @@ def evidence_offers_choice(evidence: Any) -> bool:
"""
if not isinstance(evidence, dict):
return False
# The saver anchor's one pair (#390) is a suggestion to confirm.
needed = 1 if evidence.get("anchor") == "saver" else MIN_CHOOSABLE
if evidence.get("limited"):
total = evidence.get("candidate_total")
return type(total) is int and total >= MIN_CHOOSABLE
return type(total) is int and total >= needed
candidates = evidence.get("candidates")
if not isinstance(candidates, list):
return False
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
return sum(1 for c in candidates if choosable(c)) >= needed
def evidence_digest(text: str) -> str:
+19 -4
View File
@@ -34,6 +34,11 @@ try:
except ImportError:
from utils.parallel import map_parallel
try:
from ...utils import exit_signal
except ImportError:
from utils import exit_signal
try:
from ...database.pair_ingestion import blank_positive, ingest_resolved_pair
from ...database.run_attribution import (
@@ -118,6 +123,7 @@ def _collect_media_files(output_dirs, extensions):
if not os.path.isdir(root):
continue
for current, dirnames, filenames in os.walk(root, followlinks=False):
exit_signal.checkpoint() # #376: stop between directories at exit
depth = 0 if current == root else len(Path(current).relative_to(root).parts)
dirnames[:] = sorted(
d
@@ -1630,7 +1636,14 @@ class AdminRoutesMixin:
if self._review_file_linked(media_file, roots):
outcome = "unchanged" # already linked: not a review case
else:
self._record_scanned_review(media_file, roots, review)
# The records name the item versions that count; the
# saver anchor (#390) is checked on the captured bytes.
self._record_scanned_review(
media_file,
roots,
review,
scanned=(records.get("canonical"), None),
)
elif untraced and outcome == "unresolved":
# Untraced by the default profile: the inspection profile may
# still offer a choice (queued for review, never linked).
@@ -1691,9 +1704,9 @@ class AdminRoutesMixin:
``"ambiguous"`` (nothing linked it) is a Needs review case. An
ambiguous result with nothing to choose (``reviewable`` is False: fewer
than two candidates carry a prompt) that nothing links returns
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``;
its parsed records go to ``records["canonical"]`` when ``records`` is
a dict.
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
The parsed records of either ambiguous file go to
``records["canonical"]`` when ``records`` is a dict.
"""
if not metadata:
# No prompt metadata (--disable-metadata, bare savers): only the
@@ -1723,6 +1736,8 @@ class AdminRoutesMixin:
records["canonical"] = canonical
elif resolution["state"] == "ambiguous" and ambiguous is not None:
ambiguous.append(media_file)
if records is not None:
records["canonical"] = canonical
if resolution["state"] == "resolved":
result = ingest_resolved_pair(
self.db, str(media_file), canonical, resolution
+73 -3
View File
@@ -483,13 +483,59 @@ class BackupRestoreRoutesMixin:
self.logger.info(f"Removed {len(removed)} old safety backups")
async def _publish(self, model, staged):
"""Publish ``staged``; on a page-copy failure prune and re-raise."""
"""Publish ``staged``; on a page-copy failure prune and re-raise.
A cancel (server stop) waits for the publishing thread (#360): the
caller's cleanup and its registry admission must outlive it. When the
copy then succeeded the library was replaced, so it is adopted
(``_after_publish``, itself run to its end) before the cancel goes
on; a copy that failed is handled as usual, then the cancel goes on.
"""
future = self._in_executor_task(model.publish_staged, staged)
cancelled = await _outlive_cancel(future)
try:
return await self._run_in_executor(model.publish_staged, staged)
backup_path = future.result()
except sqlite3.OperationalError as exc:
if getattr(exc, "_prompt_manager_restore_backup_created", False) is True:
await self._prune_after(model.db_path, failed=True)
if cancelled:
raise asyncio.CancelledError from exc
raise
except Exception as exc:
if cancelled:
raise asyncio.CancelledError from exc
raise
if cancelled:
self.logger.info("The restore finished while the server was stopping")
# A second cancel (the loop's own shutdown) must not cut it short.
adopted = asyncio.ensure_future(self._after_publish(model.db_path))
await _outlive_cancel(adopted)
if not adopted.cancelled() and adopted.exception() is not None:
self.logger.error(
"Could not adopt the restored library",
exc_info=adopted.exception(),
)
raise asyncio.CancelledError
return backup_path
async def _stage_to_end(self, model, temp_path):
"""Stage ``temp_path``; a cancel waits for the worker (#360).
The upload stays until the staging thread is done with it, and a
stage it finished after the cancel is removed before the cancel goes
on.
"""
future = self._in_executor_task(model.stage_restore, temp_path)
if await _outlive_cancel(future):
if not future.cancelled() and future.exception() is None:
await self._discard_held([_Orphan(future.result())])
raise asyncio.CancelledError
return future.result()
def _in_executor_task(self, func, *args):
"""``_run_in_executor(func, *args)`` as a task ``_outlive_cancel`` can
wait for (shielded, the executor call is never cancelled)."""
return asyncio.ensure_future(self._run_in_executor(func, *args))
async def _after_publish(self, db_path):
"""Everything after the copy replaced the library: best effort, logged.
@@ -609,7 +655,7 @@ class BackupRestoreRoutesMixin:
{"success": False, "error": REQUESTS_BUSY}, status=409
)
model = self.db.model
staged = await self._run_in_executor(model.stage_restore, temp_path)
staged = await self._stage_to_end(model, temp_path)
try:
backup_path = await self._publish(model, staged)
finally:
@@ -927,6 +973,30 @@ class BackupRestoreRoutesMixin:
job.update(progress=100, status_line=line)
async def _outlive_cancel(future):
"""Await ``future`` (an executor call's task) to its end, even through
cancels (#360).
A cancelled task cannot stop the worker thread, so the files it uses and
the registry admission it runs under must outlive it: the cancel is held
until the worker returned. Returns whether the task was cancelled
meanwhile; the caller reads the outcome from ``future`` and re-raises
``CancelledError``. A future cancelled itself (an executor shut down) is
not a cancel of the task: ``future.result()`` raises it. Never catches
``GeneratorExit``: a closed loop still closes the coroutine.
"""
cancelled = False
while not future.done():
try:
await asyncio.shield(future)
except asyncio.CancelledError:
if not future.cancelled():
cancelled = True
except Exception:
pass # the caller reads it from the future
return cancelled
class _Orphan:
"""A stage with no store entry, in the shape ``_discard_held`` takes."""
+16 -1
View File
@@ -47,6 +47,11 @@ import time
import uuid
from pathlib import PurePosixPath, PureWindowsPath
try:
from ...utils import exit_signal
except ImportError: # tests import the package top-level
from utils import exit_signal
logger = logging.getLogger(__name__)
# Start and finish order: wall-clock times can tie (coarse Windows clocks).
_sequence = itertools.count()
@@ -195,11 +200,21 @@ class Job:
self.counters = dict.fromkeys(groups, 0)
self.outcomes = {group: [] for group in groups}
self.log = []
self.cancelled = False
self._cancel_requested = False
self.last_event = None
self.task = None
self._subscribers = set()
@property
def cancelled(self):
"""Cancel was requested, or the interpreter is exiting (#376): every
check a job already makes between files also stops it at exit."""
return self._cancel_requested or exit_signal.requested()
@cancelled.setter
def cancelled(self, value):
self._cancel_requested = bool(value)
@property
def record_changing(self):
return self.admission == "records"
+40 -2
View File
@@ -11,6 +11,11 @@ import stat
from aiohttp import web
try:
from ...utils.logging_config import ClearRefused, parse_confirmed
except ImportError: # loaded as the top-level ``py`` package (tests)
from utils.logging_config import ClearRefused, parse_confirmed
# Keys PromptManagerLogger.update_config() understands. Anything else is
# rejected so a client cannot inject arbitrary entries into the logging config.
LOG_CONFIG_KEYS = frozenset(
@@ -263,10 +268,43 @@ class LoggingRoutesMixin:
)
async def truncate_logs(self, request):
"""Truncate all log files."""
"""Clear all logs: what ``{"expected": [...]}`` lists, or all (#389).
``expected`` holds the ``{name, identity}`` rows of ``/logs/files`` the
confirmation showed. Without a body (the legacy admin page, whose
confirmation lists nothing) every rotated copy present now is deleted.
A malformed list is 400; a clear that would empty or delete what was
not confirmed changes nothing and is 409 with the reason's ``code``.
"""
try:
args = ()
if request.can_read_body:
data, error_response = await self._read_json_body(request)
if error_response is not None:
return error_response
if "expected" in data:
try:
parse_confirmed(data["expected"])
except ValueError:
return web.json_response(
{
"success": False,
"error": "expected must list name and identity"
" rows from /logs/files",
},
status=400,
)
args = (data["expected"],)
logger_manager = self._get_logger_manager()
results = await self._run_in_executor(logger_manager.truncate_logs)
try:
results = await self._run_in_executor(
logger_manager.truncate_logs, *args
)
except ClearRefused as refused:
return web.json_response(
{"success": False, "error": str(refused), "code": refused.code},
status=409,
)
return web.json_response(
{
+11 -1
View File
@@ -677,7 +677,14 @@ def _verify_and_read(store, capture, item, root, need_metadata: bool):
facts = store.verify(root, relative)
captured = None
if need_metadata:
captured = captured_review(store, capture, item_profile(item["evidence"]))
# The file's own name, so an inspection item resolves anchored on its
# saver as it was recorded (#390).
captured = captured_review(
store,
capture,
item_profile(item["evidence"]),
relative.rsplit("/", 1)[-1],
)
return relative, facts, captured
@@ -723,6 +730,9 @@ def _candidate_still_offered(resolution: dict, choice: str) -> bool:
that rewords the top-level reason or reorders candidates leaves a
pending item applicable.
"""
if isinstance(resolution, dict) and resolution.get("anchor") == "saver":
# Only the saver anchor resolves the bytes (#390): its pair alone.
resolution = {"state": "ambiguous", "candidates": [resolution]}
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
return False
evidence, _count, limited = review_evidence(resolution)
+3 -2
View File
@@ -88,7 +88,8 @@ class ReviewRecordingMixin:
def _record_scanned_review(self, media_file, roots, review=None, scanned=None):
"""Record one ambiguous, unlinked scanned file (logged and counted).
``scanned`` is the scan's ``(metadata, inspection_resolution)``.
``scanned`` is the scan's ``(metadata, inspection_resolution)``; the
resolution is None when the scan did not run the inspection profile.
"""
try:
outcome = self._record_review_sync(
@@ -199,7 +200,7 @@ class ReviewRecordingMixin:
key = _memory_key(locator)
if state is not None and memory.get(key) == state:
return None
resolution = review_capture.inspection_choice(canonical)
resolution = review_capture.inspection_choice(canonical, media_file.name)
if resolution is not None:
memory.pop(key, None)
return canonical, resolution
+30 -11
View File
@@ -88,9 +88,14 @@ from .thumbnail_encode import shutdown_encoders, writer_settings
from .thumbnail_jobs import ThumbnailOptions
try:
from ...utils.parallel import map_parallel
from ...utils.parallel import map_parallel, note_refusal, refused_at_shutdown
except ImportError: # tests import the package top-level
from utils.parallel import map_parallel
from utils.parallel import map_parallel, note_refusal, refused_at_shutdown
try:
from ...utils.exit_signal import ExitRequested
except ImportError: # tests import the package top-level
from utils.exit_signal import ExitRequested
# The placeholder frame a video without a preview is answered with (JPEG
# bytes, never written to disk); written previews follow the settings.
@@ -410,7 +415,8 @@ class AutoThumbnailer:
def enqueue(self, src, dst, is_video):
"""Queue *dst*'s preview for the worker; False when already queued,
full or closed. Never blocks on a write."""
full, closed or its worker is refused at interpreter exit. Never
blocks on a write."""
key = _dst_key(dst)
with self._changed:
if self._closed or key in self._queued:
@@ -420,34 +426,47 @@ class AutoThumbnailer:
self._queued.add(key)
self._queue.append((key, Path(src), Path(dst), bool(is_video)))
self._busy = True
started = True
if self._start_worker and not self._worker_running:
self._start_worker_locked()
started = self._start_worker_locked()
if not started:
# Refused at interpreter exit: no worker will take it (#375).
self._queue.pop()
self._queued.discard(key)
self._changed.notify_all()
return True
return started
def _start_worker_locked(self):
"""Start the worker (lock held). A start that fails leaves the file
queued and the next enqueue tries again."""
"""Start the worker (lock held); False only when the interpreter is
exiting (#375: logged once at DEBUG, the caller drops the file).
Another start failure leaves the file queued and the next enqueue
tries again."""
worker = threading.Thread(target=self._run, name=WORKER_NAME, daemon=True)
worker.pm_preview_token = _MODULE_TOKEN
worker.pm_preview_close = self.close
self._worker_running = True
try:
worker.start()
except RuntimeError:
except RuntimeError as error:
self._worker_running = False
self._busy = False
if refused_at_shutdown(error):
note_refusal(self._logger, "The automatic preview worker")
return False
self._logger.warning(
"The automatic preview worker could not start; the next output "
"tries again",
exc_info=True,
)
return
return True
self._worker = worker
return True
def _run(self):
try:
self._drain()
except ExitRequested:
pass # #376: the interpreter is exiting; queued files are dropped
except Exception:
self._logger.warning("The automatic preview worker stopped", exc_info=True)
finally:
@@ -790,8 +809,8 @@ class AutoThumbnailMixin:
Returns ``queued``, ``fresh`` (its preview is current) or ``skipped``
(not a media file under a current root, a link, a preview itself,
already queued, or after the app's cleanup). Never writes on the
caller's thread.
already queued, after the app's cleanup, or when its worker is refused
at interpreter exit). Never writes on the caller's thread.
"""
if self._auto_thumbnails_shut:
return "skipped"
+60
View File
@@ -7,6 +7,7 @@ Refactored from standalone_tagger.py for integration with PromptManager API.
import errno
import gc
import inspect
import os
import sys
from pathlib import Path
@@ -32,6 +33,54 @@ try:
except ImportError:
from py.tagging_defaults import DEFAULT_TAG_TEMPLATE
try:
from ..utils import exit_signal
except ImportError:
from utils import exit_signal
def abort_hub_downloads() -> None:
"""Abort huggingface_hub's Xet transfers (run by the exit hook, #376).
Xet downloads report progress from native code, which ignores an
exception raised in ``tqdm.update``; huggingface_hub aborts them itself
on Ctrl+C with ``abort_xet_session``. Nothing happens when
huggingface_hub was never imported or has no Xet support.
"""
if "huggingface_hub" not in sys.modules:
return
try:
from huggingface_hub.utils import _xet
except ImportError:
return
abort = getattr(_xet, "abort_xet_session", None)
if callable(abort):
abort()
def _stop_at_exit(download) -> Dict[str, Any]:
"""Keyword arguments that stop a huggingface_hub download at its next
chunk once the interpreter is exiting (#376), or none when this
huggingface_hub does not take ``tqdm_class``.
huggingface_hub 1.x removes its partial temp file on any failure; older
versions keep their resumable ``.incomplete`` file. The model's own path
is only written by the final rename either way.
"""
try:
if "tqdm_class" not in inspect.signature(download).parameters:
return {}
from huggingface_hub.utils import tqdm as hub_tqdm
except (ImportError, TypeError, ValueError):
return {}
class StopAtExit(hub_tqdm):
def update(self, n=1):
exit_signal.checkpoint()
return super().update(n)
return {"tqdm_class": StopAtExit}
def _autotag_config():
"""The saved defaults; imported late, as config needs ComfyUI's server."""
@@ -397,6 +446,7 @@ class AutoTagService:
filename=filename,
local_dir=str(model_dir),
local_dir_use_symlinks=False,
**_stop_at_exit(hf_hub_download),
)
if progress_callback:
@@ -541,6 +591,13 @@ class AutoTagService:
else:
return self._download_hf_model(progress_callback)
except Exception as e:
if exit_signal.requested():
# ExitRequested between chunks, or the Xet session aborted
# by the exit hook (abort_hub_downloads).
self.logger.info(
f"Model download stopped: ComfyUI is exiting ({model_type})"
)
return False
self.logger.error(f"Download failed: {e}")
if progress_callback:
progress_callback(f"Error: {str(e)}", 0)
@@ -570,6 +627,7 @@ class AutoTagService:
filename=config["filename"],
local_dir=str(gguf_dir),
local_dir_use_symlinks=False,
**_stop_at_exit(hf_hub_download),
)
self.logger.info("GGUF model downloaded")
@@ -587,6 +645,7 @@ class AutoTagService:
filename=config["mmproj_filename"],
local_dir=str(gguf_dir),
local_dir_use_symlinks=False,
**_stop_at_exit(hf_hub_download),
)
self.logger.info("mmproj downloaded")
@@ -613,6 +672,7 @@ class AutoTagService:
repo_id=config["repo"],
local_dir=str(model_dir),
local_dir_use_symlinks=False,
**_stop_at_exit(snapshot_download),
)
self.logger.info("HF model downloaded")
+5 -2
View File
@@ -44,8 +44,10 @@ from .lora_manager_layout import (
)
try:
from ..utils import exit_signal
from ..utils.hashing import generate_prompt_hash
except ImportError: # pragma: no cover - flat test imports
from utils import exit_signal
from utils.hashing import generate_prompt_hash
LORA_CATEGORY = "lora-manager"
@@ -189,14 +191,15 @@ def stage(
downloads its CivitAI example images. A sidecar that cannot be read, or
whose record fails to build, is skipped by its name, as are the
centralized sidecars discovery could not parse. ``cancelled()`` is asked
before each sidecar (raises ImportCancelled); ``on_record(index,
before each sidecar, and so is the exit signal (#376: the legacy scan's
``cancelled`` does not read it); either raises ImportCancelled. ``on_record(index,
record_or_None, staged)`` follows each one.
"""
sidecars = discovery.sidecars
staged = StagedImport(total=len(sidecars))
staged.skipped.extend(discovery.unreadable)
for index, sidecar in enumerate(sidecars):
if cancelled():
if cancelled() or exit_signal.requested():
raise ImportCancelled()
record = None
path = sidecar.metadata_path
+2 -2
View File
@@ -8,8 +8,8 @@ coverage>=7.16.2
numpy>=2.2.6 # WD14 engine tests
# Code quality
black>=26.10.1
flake8>=7.4.1
black==26.10.1
flake8==7.4.1
# Security scanning
bandit[toml]>=1.9.4
+2 -1
View File
@@ -286,7 +286,8 @@ const continuationText=page=>page.locator('.infinite-continuation > p').first().
async function searchOf(page,term) {
const before=await countText(page);
await page.locator('#search').fill(term);
await expect.poll(()=>countText(page)).not.toBe(before);
// Production's caption reads "Loading images…" until the search's listing answers (#391); the concept's is instant.
await expect.poll(async()=>{const text=await countText(page);return text!==before&&text!=='Loading images…';}).toBe(true);
await expect.poll(()=>page.locator(CARDS).count()).toBeLessThan(FIXTURE_IMAGES);
await settle(page);
const result={count:await countText(page),continuation:await continuationText(page),cards:await page.locator(`${CARDS} .image-open`).evaluateAll(nodes=>nodes.map(node=>node.getAttribute('aria-label')))};
+21 -1
View File
@@ -228,7 +228,8 @@ async function productionSendOf(page,context) {
async function tagOf(page) {
const before=await page.locator('#result-count').textContent();
await panelOf(page).locator('.detail-inner > .tags').last().getByRole('button',{name:'cinematic',exact:true}).click();
await expect.poll(()=>page.locator('#result-count').textContent()).not.toBe(before);
// Production's caption reads "Loading images…" until the tag's listing answers (#391); the concept's search is instant.
await expect.poll(async()=>{const text=await page.locator('#result-count').textContent();return text!==before&&text!=='Loading images…';}).toBe(true);
await settle(page);
return {count:await page.locator('#result-count').textContent(),
cards:await page.locator(`${CARDS} .image-open`).evaluateAll(nodes=>nodes.map(node=>node.getAttribute('aria-label')))};
@@ -328,6 +329,25 @@ test('a recipe-linked image without recipe metadata shows its positive prompt me
await expect(panelOf(page).locator('.detail-footer button')).toHaveText(['Send to ComfyUI','Prompt history']);
});
// #391: a tag chip replaces the collection; until its listing answers the caption says it is loading, never that the
// filter matched nothing ("0 of 0 images" was painted before the response and read by a fast reader).
test('a tag chip collection reads as loading, not empty, until its listing answers (#391)',async({page})=>{
await page.setViewportSize({width:1600,height:1100});
await openProductionPanel(page,'dark');
let release,arrived;
const held=new Promise(resolve=>{release=resolve;}),reached=new Promise(resolve=>{arrived=resolve;});
await page.route(url=>url.pathname==='/prompt_manager/images/output'&&url.searchParams.get('tags')==='cinematic',
async route=>{arrived();await held;await route.continue();},{times:1});
const count=page.locator('#result-count');
await panelOf(page).locator('.detail-inner > .tags').last().getByRole('button',{name:'cinematic',exact:true}).click();
await reached;
await expect(count).toHaveText('Loading images…');
await expect(page.locator(CARDS)).toHaveCount(0);
release();
await expect(count).toHaveText(countCopy('8 of 8 images · 2 positive prompts'));
await expect(page.locator(CARDS)).toHaveCount(8);
});
// Task C3 (D88, #249 PR C): the concept's chip row is the image's own tags (`i.tags`, app.js a402). With those names
// saved as the file's current image tags, production draws the row from them (Tags filter buttons in the panel, plain
// in the viewer, the linked positive's equal tags not repeated) and still equals the concept, in both sizes.
+3 -79
View File
@@ -1,11 +1,9 @@
// #328: the legacy admin and gallery auto-tag tools keep working beside System › Auto-tagging. The model download is
// the `model_download` Activity job; the pages' stream attaches to it. No model is ever downloaded: the fixture's
// fake download waits until the test releases it (`/__test__/tagging/download`).
// #328: the legacy admin and gallery auto-tag batch and review keep working beside System › Auto-tagging. Since #330
// the pages download and unload models only through System (legacy-system-links.spec.mjs); a batch never downloads
// one. The fixture's fake download (`/__test__/tagging/download`) is held so nothing is ever fetched.
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
const TITLE = 'Tagging model download · WD14 ViT';
async function post(request, url, data) {
await expect(await request.post(url, data === undefined ? {} : { data })).toBeOK();
}
@@ -33,61 +31,6 @@ test.afterEach(async ({ page, request, browserName }) => {
await post(request, '/__test__/jobs/restart');
});
async function startDownload(page, request) {
await page.locator('#autoTagBtn').click();
const status = page.locator('#wd14VitModelStatus');
await status.getByRole('button', { name: 'Download', exact: true }).click();
const modal = page.locator('#autoTagDownloadModal');
await expect(modal).toBeVisible();
await expect(page.locator('#downloadStatus')).toHaveText('Downloading…');
await expect(page.locator('#downloadProgressBar')).toHaveClass(/animate-pulse/);
await expect(modal).toContainText('The download continues in Activity.');
await expect(page.locator('#cancelDownloadBtn')).toHaveText('Close');
const running = (await downloads(request)).filter(job => job.state === 'running');
expect(running.map(job => [job.title, job.cancellable])).toEqual([[TITLE, false]]);
return { modal, status };
}
test('admin download modal closes on complete and Activity lists the job without Cancel', async ({ page, request }) => {
await page.goto('/prompt_manager/admin');
const { modal, status } = await startDownload(page, request);
await page.goto('/prompt_manager/workspace#/activity');
const item = page.locator('#workspace-main section.surface > .activity-item')
.filter({ has: page.getByRole('link', { name: new RegExp(`${TITLE}$`) }) });
await expect(item.locator('.chip')).toHaveText('Running');
await expect(item.locator('progress')).not.toHaveAttribute('value');
await expect(item.getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
// Back on the admin page a second download of the same model attaches to the running job.
await page.goto('/prompt_manager/admin');
await startDownload(page, request);
await post(request, '/__test__/tagging/download', { mode: 'succeed' });
await expect(modal).toBeHidden();
await expect(status).toContainText('Downloaded');
expect((await downloads(request)).map(job => [job.title, job.state])).toEqual([[TITLE, 'finished']]);
});
test('gallery download modal closes on complete', async ({ page, request }) => {
await page.goto('/prompt_manager/gallery');
const { modal, status } = await startDownload(page, request);
await post(request, '/__test__/tagging/download', { mode: 'succeed' });
await expect(modal).toBeHidden();
await expect(page.locator('#autoTagModal')).toBeVisible();
await expect(status).toContainText('Downloaded');
expect((await downloads(request)).map(job => job.state)).toEqual(['finished']);
});
test('a failed download closes the modal and says so', async ({ page, request }) => {
await page.goto('/prompt_manager/gallery');
const { modal, status } = await startDownload(page, request);
await post(request, '/__test__/tagging/download', { mode: 'fail' });
await expect(modal).toBeHidden();
await expect(page.getByText('Download failed: Download failed')).toBeVisible();
await expect(status.getByRole('button', { name: 'Download', exact: true })).toBeVisible();
expect((await downloads(request)).map(job => job.state)).toEqual(['failed']);
});
// (wd14-vit is not downloaded in these tests: beforeEach marks it so the download tests can offer it.)
// The gallery's batch reader keyed on `data.status` values the server never sends, so its modal never moved or
// closed (#328 Problem 5). It now reads `type` frames as the admin page does.
@@ -171,25 +114,6 @@ test('admin review opens the first image', async ({ page }) => {
await expect(page.getByText('Failed to start review mode')).toHaveCount(0);
});
test('gallery unload works and is refused while the gate is held', async ({ page, request }) => {
await page.goto('/prompt_manager/gallery');
await runGalleryBatch(page); // keeps the model in memory
await page.locator('#autoTagBtn').click();
const unload = page.locator('#unloadModelBtn');
await expect(unload).toBeVisible();
// A held download owns the model gate: Unload is refused with the server's error.
const held = await request.post('/prompt_manager/jobs/model-download', { data: { model_type: 'gguf' } });
expect(held.status()).toBe(202);
await unload.click();
await expect(page.getByText(/^Failed to unload model: A tagging model is busy/)).toBeVisible();
await post(request, '/__test__/tagging/download', { mode: 'succeed' });
await expect.poll(async () => (await downloads(request)).map(job => job.state)).toEqual(['finished']);
await unload.click();
await expect(page.getByText('WD14-SWINV2 model unloaded successfully')).toBeVisible();
});
/** The admin review lists every row, videos too: Skip any image that is not one until the confirmation shows. */
async function untilAutoTagged(page, confirm) {
// Each image's skip check answers before its confirmation or its error shows.
+286
View File
@@ -0,0 +1,286 @@
// System S13 (#330): the legacy admin and gallery pages open native System for their System tools. Each replaced
// control is a plain link (`workspace#/system/<id>`, a new tab), and the legacy flows S13 keeps still work.
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
const PAGES = Object.freeze({ admin: '/prompt_manager/admin', gallery: '/prompt_manager/gallery' });
const LABELS = Object.freeze({
overview: 'Overview',
appearance: 'Appearance',
sources: 'Sources & scans',
integrations: 'Integrations',
tagging: 'Auto-tagging',
maintenance: 'Maintenance',
backups: 'Backup & export',
diagnostics: 'Diagnostics & logs',
});
async function openPageOptions(page) {
await page.locator('#pageOptionsBtn').click();
await expect(page.locator('#settingsModal')).toBeVisible();
}
/** The auto-tag modal with its model rows read from /autotag/models. */
async function openAutoTag(page) {
const read = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/models'));
await page.locator('#autoTagBtn').click();
await read;
await expect(page.locator('#autoTagModal')).toBeVisible();
}
const notDownloaded = (page, request) => post(request, '/__test__/tagging/models', { unavailable: ['gguf'] });
const modelLoaded = (page, request) => post(request, '/__test__/tagging/loaded', { model: 'wd14-vit' });
// One row per System link of the design's §2 map: the page, the control's label, how to find it, the section it opens
// and what a test must arrange before the control shows.
const LINKS = Object.freeze([
{ page: 'admin', label: 'Settings', locate: page => page.locator('#settingsBtn'), section: 'overview' },
{ page: 'admin', label: 'Maintenance', locate: page => page.locator('#maintenanceBtn'), section: 'maintenance' },
{ page: 'admin', label: 'Backup DB', locate: page => page.locator('#backupBtn'), section: 'backups' },
{ page: 'admin', label: 'Restore DB', locate: page => page.locator('#restoreBtn'), section: 'backups' },
{ page: 'gallery', label: 'Settings', locate: page => page.locator('#settingsBtn'), section: 'appearance' },
{ page: 'admin', label: 'Manage in System (model row)', arrange: notDownloaded, reveal: openAutoTag,
locate: page => page.locator('#ggufModelStatus a'), section: 'tagging' },
{ page: 'gallery', label: 'Manage in System (model row)', arrange: notDownloaded, reveal: openAutoTag,
locate: page => page.locator('#ggufModelStatus a'), section: 'tagging' },
{ page: 'gallery', label: 'Manage in System (Unload Model)', arrange: modelLoaded, reveal: openAutoTag,
locate: page => page.locator('#unloadModelBtn'), section: 'tagging' },
{ page: 'gallery', label: 'Page options Maintenance', reveal: openPageOptions,
locate: page => page.locator('#pageOptionsMaintenanceLink'), section: 'maintenance' },
{ page: 'gallery', label: 'Page options Sources & scans', reveal: openPageOptions,
locate: page => page.locator('#pageOptionsSourcesLink'), section: 'sources' },
{ page: 'admin', label: 'Diagnostics', locate: page => page.locator('#diagnosticsBtn'), section: 'diagnostics' },
{ page: 'admin', label: 'View Logs', locate: page => page.locator('#logsBtn'), section: 'diagnostics' },
]);
async function post(request, url, data) {
await expect(await request.post(url, data === undefined ? {} : { data })).toBeOK();
}
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await post(request, '/__test__/reset');
await post(request, '/__test__/system/reset');
await post(request, '/__test__/tagging/reset');
// The 3.2.0 update notice would cover the admin page's toolbar.
await page.addInitScript(() => { localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
page.legacyErrors = [];
page.on('pageerror', error => page.legacyErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await post(request, '/__test__/tagging/reset');
await post(request, '/__test__/system/reset');
expect(page.legacyErrors).toEqual([]);
});
/** Opens the legacy page with the link showing; returns the link. */
async function openLink(page, request, link) {
if (link.arrange) await link.arrange(page, request);
await page.goto(PAGES[link.page]);
if (link.reveal) await link.reveal(page);
const control = link.locate(page);
await expect(control).toBeVisible();
return control;
}
// Each row opens its section in a new tab and sends no request from the legacy page (the design's `a System link sends
// no request from the legacy page`, checked per row so one slow page never times out the whole table).
for (const link of LINKS) {
test(`legacy ${link.page} ${link.label} opens System ${link.section}`, async ({ page, context, request }) => {
// The gallery's thumbnail startup check (K9) reads its status on a timer of its own; it is off here.
await page.addInitScript(() => {
localStorage.setItem('gallerySettings', JSON.stringify({ checkThumbnailsAtStartup: false }));
});
const control = await openLink(page, request, link);
const legacyUrl = page.url();
await expect(control).toHaveAttribute('href', `workspace#/system/${link.section}`);
await expect(control).toHaveAttribute('target', '_blank');
await expect(control).toHaveAttribute('rel', 'noopener');
// The page's own start-up reads (list, film strips, settings) finish first; then the click must add none.
// Lazy thumbnails may still load whenever the layout changes; they are not the click's.
await page.waitForLoadState('networkidle');
const sent = [];
const record = item => { if (item.resourceType() !== 'image') sent.push(`${item.method()} ${item.url()}`); };
page.on('request', record);
const opened = context.waitForEvent('page');
await control.click();
const system = await opened;
await expect(system).toHaveURL(new RegExp(`/prompt_manager/workspace#/system/${link.section}$`));
page.off('request', record);
expect(sent, 'a System link sends no request from the legacy page').toEqual([]);
// System's page heading is "System"; the open section is the pressed button of its section switcher.
const main = system.locator('#workspace-main');
await expect(main.getByRole('heading', { level: 1, name: 'System', exact: true })).toBeVisible();
await expect(main.getByRole('group', { name: 'System sections', exact: true })
.getByRole('button', { name: LABELS[link.section], exact: true })).toHaveAttribute('aria-pressed', 'true');
expect(page.url()).toBe(legacyUrl);
await system.close();
});
}
test('kept admin flows still work', async ({ page, context }) => {
await page.goto(PAGES.admin);
const texts = page.locator('#resultsList .prompt-text');
await expect(texts.first()).toBeVisible();
// K1: sorting reorders the list.
for (const [value, order] of [['text_asc', 1], ['text_desc', -1]]) {
const listed = page.waitForResponse(item => /\/prompt_manager\/(recent|search)\?/.test(item.url()) && item.url().includes(`sort=${value}`));
await page.locator('#sortBy').selectOption(value);
await listed;
await expect.poll(async () => {
const shown = await texts.allTextContents();
// The server sorts case-insensitively (SQLite NOCASE), not by locale.
const key = text => text.toLowerCase();
const sorted = [...shown].sort((a, b) => order * (key(a) < key(b) ? -1 : key(a) > key(b) ? 1 : 0));
return shown.length > 1 && shown.join('\n') === sorted.join('\n');
}).toBe(true);
}
// K3: Add Prompt saves a prompt the list then shows.
await page.locator('#addPromptBtn').click();
await page.locator('#addPromptText').fill('Kept flow lighthouse');
const saved = page.waitForResponse(item => item.request().method() === 'POST' && item.url().endsWith('/prompt_manager/save'));
await page.locator('#saveNewPromptBtn').click();
expect((await saved).ok()).toBe(true);
await expect(page.locator('#addPromptModal')).toBeHidden();
await expect(texts.filter({ hasText: 'Kept flow lighthouse' })).toBeVisible();
// K1: a search narrows the list, and a selection enables the bulk buttons.
const all = await texts.allTextContents();
const word = all[0].trim().split(/\s+/)[0];
const searched = page.waitForResponse(item => item.url().includes('/prompt_manager/search?') && item.url().includes('text='));
await page.locator('#searchText').fill(word);
await page.locator('#searchBtn').click();
await searched;
await expect.poll(async () => {
const shown = await texts.allTextContents();
return shown.length > 0 && shown.length < all.length && shown.every(text => text.toLowerCase().includes(word.toLowerCase()));
}).toBe(true);
await expect(page.locator('#bulkDeleteBtn')).toBeDisabled();
await page.locator('#resultsList .prompt-checkbox').first().check();
for (const id of ['#bulkDeleteBtn', '#bulkTagBtn', '#bulkCategoryBtn']) await expect(page.locator(id)).toBeEnabled();
// K4: Stats reads the statistics again; Export downloads the prompts.
const stats = page.waitForResponse(item => item.url().endsWith('/prompt_manager/stats') && item.ok());
await page.locator('#statsBtn').click();
await stats;
await expect(page.getByText('Stats refreshed successfully!')).toBeVisible();
const download = page.waitForEvent('download');
await page.locator('#exportBtn').click();
expect((await download).suggestedFilename()).toMatch(/^prompts-\d{4}-\d{2}-\d{2}\.json$/);
// K5: Scan Images opens its dialog (with the quick backup) without starting a scan.
await page.locator('#scanBtn').click();
await expect(page.locator('#scanModal')).toBeVisible();
await expect(page.locator('#quickBackupBtn')).toBeVisible();
await page.locator('#cancelScan').click();
await expect(page.locator('#scanModal')).toBeHidden();
// K7: Metadata and Gallery open their pages in a new tab.
for (const [id, path] of [['#metadataBtn', '/prompt_manager/gallery.html'], ['#galleryBtn', '/prompt_manager/gallery']]) {
const opened = context.waitForEvent('page');
await page.locator(id).click();
const tab = await opened;
await tab.waitForLoadState('domcontentloaded');
expect(new URL(tab.url()).pathname).toBe(path);
await tab.close();
}
});
test('Page options keeps the page-only settings admin', async ({ page, request }) => {
const before = (await (await request.get('/prompt_manager/settings')).json()).settings;
await page.goto(PAGES.admin);
await expect(page.locator('#resultsList .prompt-text').first()).toBeVisible();
await page.locator('#pageOptionsBtn').click();
const dialog = page.locator('#settingsModal');
await expect(dialog).toBeVisible();
await expect(dialog.locator('h3')).toHaveText('Page options');
// Exactly the page-only setting (G5): the list's Infinite Scrolling default.
expect(await dialog.locator('input, select, textarea').evaluateAll(fields => fields.map(field => field.id)))
.toEqual(['infiniteScrollDefault']);
await expect(page.locator('#infiniteScrollDefault')).not.toBeChecked();
await page.locator('#infiniteScrollDefault').check();
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith('/prompt_manager/settings'));
await page.locator('#saveSettings').click();
expect((await saved).postDataJSON()).toEqual({ infinite_scroll: true });
await expect(dialog).toBeHidden();
// The list header follows the saved default at once; nothing else changed.
await expect(page.locator('#infiniteScrollSession')).toBeChecked();
const after = (await (await request.get('/prompt_manager/settings')).json()).settings;
expect(after).toEqual({ ...before, infinite_scroll: true });
await page.reload();
// The dialog shows the saved default once the page has read it (the header follows it first).
await expect(page.locator('#infiniteScrollSession')).toBeChecked();
await page.locator('#pageOptionsBtn').click();
await expect(page.locator('#infiniteScrollDefault')).toBeChecked();
});
test('Page options keeps the page-only settings gallery', async ({ page }) => {
// Native Appearance keys and the startup prompt's own keys are in the shared object before the page opens.
const stored = { imageQuality: 'high', showImageInfo: false, videoMute: false,
autoLoadMetadata: false, workspaceLayout: 'masonry', thumbnailPromptDismissed: 'sig', thumbnailsGenerated: true,
checkThumbnailsAtStartup: false, defaultViewMode: 'list' };
await page.addInitScript(value => {
if (!sessionStorage.getItem('seeded')) {
sessionStorage.setItem('seeded', '1');
localStorage.setItem('gallerySettings', value);
}
}, JSON.stringify(stored));
const listed = page.waitForResponse(item => item.url().includes('/prompt_manager/images/output?'));
await page.goto(PAGES.gallery);
await listed;
// With no Page options control, a stored default view still opens the page in it.
await expect(page.locator('#listViewBtn')).toHaveClass(/bg-pm-accent/);
await openPageOptions(page);
const dialog = page.locator('#settingsModal');
await expect(dialog.locator('h3')).toHaveText('Page options');
// Exactly the page-only settings (G5); the default view is System › Appearance's Default layout.
expect(await dialog.locator('input, select, textarea').evaluateAll(fields => fields.map(field => field.id))).toEqual([
'checkThumbnailsAtStartup', 'lazyLoadingToggle', 'defaultLimitSelect', 'gridColumnsSelect',
'infiniteScrollToggle', 'debugModeToggle', 'apiTimeoutInput']);
await expect(dialog.getByText('Default View Mode')).toHaveCount(0);
await page.locator('#defaultLimitSelect').selectOption('50');
await page.locator('#debugModeToggle').check();
await page.locator('#saveSettingsBtn').click();
await expect(dialog).toBeHidden();
const read = () => page.evaluate(() => JSON.parse(localStorage.getItem('gallerySettings')));
let saved = await read();
expect(saved).toMatchObject({ ...stored, defaultLimit: 50, debugMode: true });
// Reset puts back only the page options' defaults; the Appearance, startup-prompt and default-view keys stay.
await openPageOptions(page);
page.once('dialog', dialog => dialog.accept());
await page.locator('#resetSettingsBtn').click();
await expect(page.getByText('Page options reset to defaults')).toBeVisible();
await expect(page.locator('#defaultLimitSelect')).toHaveValue('100');
saved = await read();
expect(saved).toMatchObject({ defaultViewMode: 'list', imageQuality: 'high', showImageInfo: false, videoMute: false,
autoLoadMetadata: false, workspaceLayout: 'masonry', thumbnailPromptDismissed: 'sig', thumbnailsGenerated: true,
defaultLimit: 100, debugMode: false, checkThumbnailsAtStartup: true, lazyLoading: true, gridColumns: 8,
infiniteScroll: false, apiTimeout: 30 });
});
test('the legacy auto-tag modal stays open after Manage in System', async ({ page, context, request }) => {
for (const legacy of ['admin', 'gallery']) {
await post(request, '/__test__/tagging/models', { unavailable: ['gguf'] });
await page.goto(PAGES[legacy]);
await openAutoTag(page);
const link = page.locator('#ggufModelStatus').getByRole('link', { name: 'Manage in System', exact: true });
await expect(link).toHaveAttribute('title', 'Opens System › Auto-tagging in a new tab');
// A downloaded model keeps its status text.
await expect(page.locator('#wd14VitModelStatus')).toHaveText('\u2713 Downloaded');
const opened = context.waitForEvent('page');
await link.click();
const system = await opened;
await expect(system).toHaveURL(/#\/system\/tagging$/);
await expect(page.locator('#autoTagModal')).toBeVisible();
await system.close();
// Downloaded in System meanwhile: reopening the modal reads the models again (CUJ 4).
await post(request, '/__test__/tagging/models', { unavailable: [] });
await page.goto(PAGES[legacy]);
await openAutoTag(page);
await expect(page.locator('#ggufModelStatus')).toHaveText('\u2713 Downloaded');
}
});
+54
View File
@@ -0,0 +1,54 @@
// A restore the open workspace page has not noticed yet (#413).
//
// Specs that restore through the API while a page is open check what the page does with a restore it did not see:
// its next write is refused as "Prompt storage was restored since this dialog opened". But the page's Activity store
// reads the job list every 15 s (activity-store.mjs POLL_MS), and a read the server answers while the restore job runs
// makes the page follow it: when it finishes, the page is shown again from the restored data (S9, index.mjs
// onRestoreFinished), which closes any open dialog. On a slow runner a restore can take longer than the time left to
// the next read (CI run 40646112668: the restore took 16 s), so whether the page saw it depended on timing.
//
// `armUnseenRestores(page)`, before the page loads, passes every job-list read of the page through one route, so it
// knows which reads the server has not answered yet. `unseenRestore(page, restoring)` then closes the gate, waits
// for those reads' answers (sent before the restore began, they cannot list it), runs `restoring()`, and only after
// it has answered lets the held reads go on: they read the restore as already ended, as a tab that read its job list
// just before and just after the restore does. A restore that is never seen running never reloads the page.
const JOBS = '**/prompt_manager/jobs';
const gates = new WeakMap();
/** Route `page`'s job-list reads through the restore gate. Call before the page's first navigation. */
export async function armUnseenRestores(page) {
const gate = { closed: null, inFlight: 0, idle: [] };
gates.set(page, gate);
await page.route(JOBS, async route => {
if (gate.closed) {
await gate.closed;
return route.fallback();
}
gate.inFlight += 1;
try {
const response = await route.fetch();
await route.fulfill({ response }).catch(() => {});
} catch {
await route.abort().catch(() => {});
} finally {
gate.inFlight -= 1;
if (gate.inFlight === 0) for (const resolve of gate.idle.splice(0)) resolve();
}
});
}
/** Run `restoring()` (an API restore) unseen by `page`'s Activity store; resolves with what it resolves with. */
export async function unseenRestore(page, restoring) {
const gate = gates.get(page);
if (!gate) throw new Error('armUnseenRestores(page) must run before the page loads');
let open;
gate.closed = new Promise(resolve => { open = resolve; });
try {
// Reads already on their way were sent before the restore: their answers cannot list it.
if (gate.inFlight > 0) await new Promise(resolve => gate.idle.push(resolve));
return await restoring();
} finally {
gate.closed = null;
open();
}
}
+1 -236
View File
@@ -104,33 +104,6 @@ for (const leave of ['beforeunload', 'pagehide']) {
});
}
// #222: the gallery's Clear cache clears the server's thumbnails by its own request and keeps this browser's
// preferences and other tools' storage; nothing of Prompt Manager's is cached in a browser, and the toast says so.
test('the gallery cache clear keeps preferences and other tools caches', async ({ page }) => {
await page.addInitScript(() => {
if (sessionStorage.getItem('seeded')) return;
sessionStorage.setItem('seeded', '1');
localStorage.setItem('pm_workspace_theme', 'light');
localStorage.setItem('other-tool-setting', 'keep');
localStorage.setItem('pm_other_tool', 'keep');
});
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(2);
await page.evaluate(async () => { await (await caches.open('other-tool-gallery-thumbnails')).put('/x', new Response('x')); });
const clears = [];
page.on('request', req => { if (req.url().includes('/clear-thumbnails')) clears.push(req.method()); });
page.once('dialog', dialog => dialog.accept());
await page.locator('#settingsBtn').click();
await page.locator('#clearCacheBtn').click();
await expect(page.getByText("Thumbnails cleared; original images untouched. Nothing of Prompt Manager's is cached in this browser."))
.toBeVisible();
expect(clears).toEqual(['POST']);
expect(await page.evaluate(async () => ({
theme: localStorage.getItem('pm_workspace_theme'), other: localStorage.getItem('other-tool-setting'),
prefixed: localStorage.getItem('pm_other_tool'), notice: localStorage.getItem('pm_update_notice_dismissed'),
caches: await caches.keys(),
}))).toEqual({ theme: 'light', other: 'keep', prefixed: 'keep', notice: '3.2.0', caches: ['other-tool-gallery-thumbnails'] });
});
// #379: the reviewed tags are that image's own tags; no prompt changes.
test('actual gallery review generates and persists tags for the selected root\'s image', async ({ page, request }) => {
@@ -226,7 +199,7 @@ test('gallery infinite scroll appends a real backend page while retaining existi
await expect(await request.post('/__test__/reset', { data: { extra_images: 108 } })).toBeOK();
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(100);
await page.locator('#settingsBtn').click();
await page.locator('#pageOptionsBtn').click();
await page.locator('#infiniteScrollToggle').check();
await page.locator('#saveSettingsBtn').click();
await expect(page.locator('#settingsModal')).toBeHidden();
@@ -246,37 +219,6 @@ async function promptSnapshot(request) {
})).sort((a, b) => a.id - b.id);
}
async function downloadBackup(page) {
const downloading = page.waitForEvent('download');
await page.locator('#backupBtn').click();
const download = await downloading;
expect(download.suggestedFilename()).toMatch(/^prompts_backup_.*\.db$/);
expect(await download.failure()).toBeNull();
const chunks = [];
for await (const chunk of await download.createReadStream()) chunks.push(chunk);
const buffer = Buffer.concat(chunks);
expect(buffer.subarray(0, 16).toString()).toBe('SQLite format 3\0');
return { name: download.suggestedFilename(), mimeType: 'application/octet-stream', buffer };
}
async function chooseRestore(page, file) {
await page.locator('#restoreBtn').click();
await expect(page.locator('#restoreModal')).toBeVisible();
await page.locator('#restoreFileInput').setInputFiles(file);
}
// Confirms the chosen restore and waits for the server's answer, not for a timer: a restore is disk-bound and can take
// longer than an expect timeout on a slow runner (#359). The modal closes only after that answer.
async function confirmRestoreAndWait(page) {
const restored = page.waitForResponse(response => response.url().endsWith('/prompt_manager/restore'));
await page.locator('#confirmRestore').click();
const response = await restored;
const body = await response.text();
expect(response.status(), body).toBe(200);
expect(JSON.parse(body)).toMatchObject({ success: true, prompt_count: 2, backup_created: expect.any(String) });
await expect(page.locator('#restoreModal')).toBeHidden();
}
function confirmNextDialog(page, accept, message) {
page.once('dialog', async dialog => {
expect(dialog.type()).toBe('confirm');
@@ -292,88 +234,6 @@ async function deletePromptButton(page, text) {
return page.locator(`[data-action="delete"][data-prompt-id="${id}"]`);
}
test('downloaded backup restores deleted prompts, tags, and image links through the actual upload form', async ({ page, request }) => {
await page.goto('/prompt_manager/admin');
await expect(page.locator('.prompt-text')).toHaveCount(2);
const before = await promptSnapshot(request);
expect(before.map(prompt => prompt.tags)).toEqual([['cats'], ['dogs']]);
expect(before.every(prompt => prompt.images.length === 1)).toBe(true);
const backup = await downloadBackup(page);
const button = await deletePromptButton(page, 'Cat portrait');
confirmNextDialog(page, true, 'delete this prompt');
await button.click();
await expect(page.locator('.prompt-text')).toHaveCount(1);
await expect(await request.post('/prompt_manager/save', { data: { text: 'Added after backup', tags: ['temporary'] } })).toBeOK();
expect(await promptSnapshot(request)).not.toEqual(before);
await chooseRestore(page, backup);
confirmNextDialog(page, true, backup.name);
await confirmRestoreAndWait(page);
await expect(page.locator('.prompt-text')).toHaveCount(2);
await page.reload();
await expect(page.locator('.prompt-text').filter({ hasText: 'Cat portrait' })).toBeVisible();
await expect(page.locator('.prompt-text').filter({ hasText: 'Added after backup' })).toHaveCount(0);
expect(await promptSnapshot(request)).toEqual(before);
await page.goto('/prompt_manager/admin#/tags');
await page.locator('[data-tag-name="cats"]').click();
await expect(page.locator('#tagPromptsGrid')).toContainText('Cat portrait');
await page.locator('#tagPromptsGrid img').first().click();
await expect(page.locator('.viewer-canvas img')).toBeVisible();
await expect.poll(() => page.locator('.viewer-canvas img').evaluate(image => image.complete && image.naturalWidth > 0)).toBe(true);
});
test('restore validation and cancellation preserve live data and allow a valid retry', async ({ page, request }) => {
await page.goto('/prompt_manager/admin');
const before = await promptSnapshot(request);
const backup = await downloadBackup(page);
let uploads = 0;
page.on('request', request => { if (request.url().endsWith('/prompt_manager/restore')) uploads++; });
await page.locator('#restoreBtn').click();
await page.locator('#confirmRestore').click();
await expect(page.getByText('Please select a database file to restore', { exact: true })).toBeVisible();
await page.locator('#restoreFileInput').setInputFiles({ ...backup, name: 'backup.txt' });
await page.locator('#confirmRestore').click();
await expect(page.getByText('Please select a valid .db file', { exact: true })).toBeVisible();
expect(uploads).toBe(0);
await page.locator('#restoreFileInput').setInputFiles(backup);
confirmNextDialog(page, false, 'replace your current database');
await page.locator('#confirmRestore').click();
await expect(page.locator('#restoreModal')).toBeVisible();
expect(uploads).toBe(0);
await page.locator('#restoreFileInput').setInputFiles({ name: 'corrupt.db', mimeType: 'application/octet-stream', buffer: Buffer.from('not a SQLite database') });
confirmNextDialog(page, true, 'corrupt.db');
const rejected = page.waitForResponse(response => response.url().endsWith('/prompt_manager/restore'));
await page.locator('#confirmRestore').click();
expect((await rejected).status()).toBe(400);
await expect(page.getByText(/Restore failed:/)).toBeVisible();
await expect(page.locator('#restoreModal')).toBeVisible();
await expect(page.locator('#confirmRestore')).toBeEnabled();
expect(await promptSnapshot(request)).toEqual(before);
await page.locator('#restoreFileInput').setInputFiles(backup);
confirmNextDialog(page, true, backup.name);
await confirmRestoreAndWait(page);
expect(await promptSnapshot(request)).toEqual(before);
});
test('backup and restore recover from interrupted network requests without losing data', async ({ page, request }) => {
await page.goto('/prompt_manager/admin');
const before = await promptSnapshot(request);
await page.route('**/prompt_manager/backup', route => route.abort('failed'), { times: 1 });
await page.locator('#backupBtn').click();
await expect(page.getByText(/Backup failed:/)).toBeVisible();
const backup = await downloadBackup(page);
await chooseRestore(page, backup);
await page.route('**/prompt_manager/restore', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, backup.name);
await page.locator('#confirmRestore').click();
await expect(page.getByText(/Restore failed:/)).toBeVisible();
await expect(page.locator('#confirmRestore')).toBeEnabled();
await expect(page.locator('#restoreModal')).toBeVisible();
expect(await promptSnapshot(request)).toEqual(before);
confirmNextDialog(page, true, backup.name);
await confirmRestoreAndWait(page);
expect(await promptSnapshot(request)).toEqual(before);
});
test('prompt deletion requires confirmation and can retry after a network failure', async ({ page, request }) => {
await page.goto('/prompt_manager/admin');
const before = await promptSnapshot(request);
@@ -401,98 +261,3 @@ test('prompt deletion requires confirmation and can retry after a network failur
await expect(await request.get(url)).toBeOK();
}
});
// The duplicate scan reads the output root only, but the fixture's input root holds a different
// shared/same.png. A bare /images/serve/<path> URL tries every gallery root in turn, so once the
// output copy is deleted it answers 200 with the input copy: check each file in its own root.
const serveIn = (root, path) => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}?root=${root}`;
const rootOf = url => new URL(url, 'http://gallery.test').searchParams.get('root');
// Which copy the scan keeps depends on file times: the fixture stamps shared/same.png with the
// wall clock, and the kernel can stamp its bulk copies a fraction of a millisecond earlier.
// listOrder rewrites the finished scan's groups so a test can pick the order it needs.
async function openDuplicateScan(page, request, listOrder = null) {
await expect(await request.post('/__test__/reset', { data: { extra_images: 2 } })).toBeOK();
if (listOrder) {
await page.route('**/prompt_manager/scan_duplicates/stream', async route => {
const response = await route.fetch();
const body = (await response.text()).replace(/^data: (.*)$/gm, (line, json) => {
const event = JSON.parse(json);
if (event.type !== 'complete') return line;
for (const group of event.duplicates) group.images = listOrder(group.images);
return `data: ${JSON.stringify(event)}`;
});
await route.fulfill({ response, body });
});
}
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(4);
const originals = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
const output = rootOf(originals.find(url => url.includes('/bulk/')));
const input = rootOf(originals.find(url => url.includes('/shared/same.png') && rootOf(url) !== output));
expect(output).not.toBeNull();
expect(input).not.toBeNull();
await page.locator('#settingsBtn').click();
await page.locator('#scanDuplicatesBtn').click();
await expect(page.locator('#duplicatesCount')).toHaveText('1');
const candidates = page.locator('.duplicate-checkbox');
await expect(candidates).toHaveCount(2);
return { originals, output, input, candidates };
}
async function expectOnlySelectedGone(page, request, { originals, output, input }, selected, retained) {
expect((await request.get(serveIn(output, selected))).status()).toBe(404);
await expect(await request.get(serveIn(output, retained))).toBeOK();
await expect(await request.get(serveIn(input, 'shared/same.png'))).toBeOK();
await page.reload();
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(3);
const remaining = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
expect(remaining).toHaveLength(originals.length - 1);
for (const url of remaining) {
expect(originals).toContain(url);
await expect(await request.get(url)).toBeOK();
}
}
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
const scan = await openDuplicateScan(page, request);
const { output, candidates } = scan;
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serveIn(output, selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serveIn(output, selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
test('deleting a duplicate whose path another gallery root also holds removes only the scanned copy', async ({ page, request }) => {
// The order CI saw: a bulk copy stamped first, so shared/same.png is the first Delete candidate.
const ciOrder = ['bulk/0000.png', 'shared/same.png', 'bulk/0001.png'];
const rank = image => ciOrder.indexOf(image.path.replace(/\\/g, '/'));
const scan = await openDuplicateScan(page, request, images => [...images].sort((a, b) => rank(a) - rank(b)));
const selected = await scan.candidates.first().getAttribute('data-image-path');
const retained = await scan.candidates.last().getAttribute('data-image-path');
expect(selected.replace(/\\/g, '/')).toBe('shared/same.png');
await scan.candidates.first().check();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
+8 -7
View File
@@ -1,5 +1,6 @@
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
import {test,expect} from './fixtures.mjs';
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
import {readFile,writeFile} from 'node:fs/promises';
import {createHash} from 'node:crypto';
@@ -16,13 +17,13 @@ const marker=(page,id,anchor='10')=>recorded(page).locator(`button[data-run-id="
const exact=async locator=>locator.evaluate(node=>node.textContent);
test.beforeEach(async({page,request,browserName})=>{
await startCoverage(page,browserName);page.historyErrors=[];page.on('pageerror',error=>page.historyErrors.push(error.message));
await startCoverage(page,browserName);await armUnseenRestores(page);page.historyErrors=[];page.on('pageerror',error=>page.historyErrors.push(error.message));
const reset=await request.post('/__test__/reset',{data:{chronological_history:true}});await expect(reset).toBeOK();page.fixture=await reset.json();
});
test.afterEach(async({page,browserName})=>{await saveCoverage(page,browserName);expect(page.historyErrors).toEqual([]);});
const command=async(request,data)=>{const response=await request.post('/__test__/history/commands',{data});await expect(response).toBeOK();return response.json();};
const restoreSnapshot=async(request,bytes)=>{const response=await request.post('/prompt_manager/restore',{multipart:{database_file:{name:'history.db',mimeType:'application/octet-stream',buffer:bytes}}});await expect(response).toBeOK();expect((await response.json()).success).toBe(true);};
const restoreSnapshot=async(page,request,bytes)=>{const response=await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'history.db',mimeType:'application/octet-stream',buffer:bytes}}}));await expect(response).toBeOK();expect((await response.json()).success).toBe(true);};
const historyRead=async(request,id)=>{const response=await request.get(`/prompt_manager/library/${id}/history`);await expect(response).toBeOK();return response.json();};
const frozenContexts=contexts=>contexts.map(({navigation,positive_prompt_id,negative_prompt_id,pair_id,...frozen})=>frozen);
const immutableReceipts=items=>items.map(({current_candidate,...receipt})=>receipt);
@@ -62,7 +63,7 @@ for(const boundary of ['window','detail','candidate','inspection','media']) test
await command(request,{action:'arm',boundary});const pending=request.get(target);
try {
await expect.poll(async()=> (await(await request.get('/__test__/history/checkpoint')).json()).entered).toBe(true);
await restoreSnapshot(request,bytes);
await restoreSnapshot(page,request,bytes);
} finally {await command(request,{action:'release'});}
const retired=await pending;expect(retired.status()).toBe(409);expect((await retired.json()).code).toBe('history-changed');
for(const url of [`/prompt_manager/library/${f.detail_id}/history?limit=1&cursor=${encodeURIComponent(firstWindow.next_cursor)}`,`${base}?epoch=${epoch}`,`${base}/outputs?epoch=${epoch}`,`${base}/outputs?epoch=${epoch}&cursor=${encodeURIComponent(outputs.next_cursor)}`,media.url,media.thumbnail_url,media.inspection_url]) {
@@ -130,7 +131,7 @@ test('archive delete merge and prune preserve frozen evidence without repairing
test('actual restore with reused numeric owner rejects stale history and offers explicit saved recovery',async({page,request})=>{
const f=page.fixture;await open(page,f.detail_id);const original=await exact(saved(page).getByRole('region',{name:'Positive prompt',exact:true}).locator('p.prompt-text'));
const source=await request.get('/__test__/history/reused-snapshot');await expect(source).toBeOK();await restoreSnapshot(request,await source.body());
const source=await request.get('/__test__/history/reused-snapshot');await expect(source).toBeOK();await restoreSnapshot(page,request,await source.body());
await marker(page,f.runs.shared).click();await expect(dialog(page)).toContainText('Recorded evidence changed');await page.keyboard.press('Escape');
await expect(page.locator('.library-parent-status')).toContainText('changed');expect(await exact(saved(page).getByRole('region',{name:'Positive prompt',exact:true}).locator('p.prompt-text'))).toBe(original);
await expect(recorded(page).getByRole('button',{name:'Reload recorded history',exact:true})).toBeVisible();
@@ -142,12 +143,12 @@ for(const outcome of ['success','error'])test(`delayed actual old browser ${outc
const f=page.fixture;await open(page,f.detail_id);const snapshot=await request.get('/__test__/history/snapshot');const bytes=await snapshot.body();
let entered,release;const started=new Promise(resolve=>{entered=resolve;}),held=new Promise(resolve=>{release=resolve;});
await page.route(new RegExp(`/history/runs/${f.runs.shared}\\?`),async route=>{
if(outcome==='error')await restoreSnapshot(request,bytes);
if(outcome==='error')await restoreSnapshot(page,request,bytes);
const response=await route.fetch();expect(response.status()).toBe(outcome==='success'?200:409);entered();await held;
await route.fulfill({response}).catch(()=>{});
});
try {
await marker(page,f.runs.shared).click();await started;if(outcome==='success')await restoreSnapshot(request,bytes);
await marker(page,f.runs.shared).click();await started;if(outcome==='success')await restoreSnapshot(page,request,bytes);
await page.evaluate(()=>{location.hash='#/overview';});await expect(page.locator('#workspace-main h1')).toHaveText('Overview');release();
await page.unroute(new RegExp(`/history/runs/${f.runs.shared}\\?`));await open(page,f.detail_id);
await expect(recorded(page).locator('button[data-run-id]')).toHaveCount(0);
@@ -258,7 +259,7 @@ test('inspection first discovers a real restore and immediately retires every su
if(changed)window.recordedRecoveryStates.push({receipts:run.querySelectorAll('[data-receipt]').length,active:[...run.querySelectorAll('[data-send-role],[data-copy-role],[data-view-receipt]')].filter(node=>!node.disabled&&node.checkVisibility()).length});
});observer.observe(run,{subtree:true,childList:true,attributes:true,characterData:true});
});
await restoreSnapshot(request,await snapshot.body());
await restoreSnapshot(page,request,await snapshot.body());
} finally {await command(request,{action:'release'});}
const response=await inspectionRead;expect(response.status()).toBe(409);expect((await response.json()).code).toBe('history-changed');
await expect(dialog(page).locator('.recorded-run-status')).toContainText('Recorded evidence changed');
@@ -1,6 +1,7 @@
import {writeFile} from 'node:fs/promises';
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
import {test,expect} from './fixtures.mjs';
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
import {captureCanonicalState} from './history-canonical-comparison.mjs';
import {inventoryOf,mapConcept,diffInventory} from './library-editor-canonical-comparison.mjs';
@@ -13,7 +14,7 @@ const RATINGS=[['5','5 / 5'],['4','4 / 5'],['3','3 / 5'],['2','2 / 5'],['1','1 /
const editorDialog=page=>page.locator('dialog.library-editor');
test.beforeEach(async({page,request,browserName})=>{
await startCoverage(page,browserName);
await startCoverage(page,browserName);await armUnseenRestores(page);
const reset=await request.post('/__test__/library-editor/reset',{data:{}});await expect(reset).toBeOK();page.fixture=await reset.json();
page.editorErrors=[];page.on('pageerror',error=>page.editorErrors.push(error.message));
page.writes=[];page.on('request',item=>{if(item.method()!=='GET')page.writes.push({method:item.method(),url:new URL(item.url()).pathname,body:item.postDataJSON()});});
@@ -260,7 +261,8 @@ test('recovery states render inline and never replay a write',async({page,reques
page.writes.length=0;
dialog=await openEditor(page,{mode:'new'});await ready(dialog);
const snapshot=await request.get('/__test__/library-editor/snapshot');await expect(snapshot).toBeOK();
const restore=await request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:await snapshot.body()}}});
const buffer=await snapshot.body();
const restore=await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer}}}));
await expect(restore).toBeOK();page.writes.length=0;
await field(dialog,'Title').fill('After restore');await field(dialog,'Positive prompt').fill('Restored storage prompt');await button(dialog,'Save').click();
await expect(note(dialog)).toHaveText('Prompt storage was restored since this dialog opened. Nothing was saved.');
@@ -588,7 +590,7 @@ test('saved refresh failure retries only reads',async({page})=>{
test('restored storage after a confirmed save or before an archive offers an explicit Library reload',async({page,request})=>{
const f=page.fixture;const snapshot=await(await request.get('/__test__/library-editor/snapshot')).body();
const restore=()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}});
const restore=()=>unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}}));
await openHistory(page,f.positive_id);
await page.route(`**/prompt_manager/library/${f.positive_id}`,async route=>{
if(route.request().method()!=='PATCH')return route.continue();
@@ -615,7 +617,7 @@ test('restored storage after a confirmed save or before an archive offers an exp
const DISCARDS='Leaving this dialog discards your entries.';
test('restored storage before an Edit or Add negative save offers only the Library reload exit',async({page,request})=>{
const f=page.fixture;const snapshot=await(await request.get('/__test__/library-editor/snapshot')).body();
const restore=()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}});
const restore=()=>unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}}));
await openHistory(page,f.positive_id);
await page.getByRole('button',{name:'Edit metadata',exact:true}).click();let dialog=editorDialog(page);await ready(dialog);
await expect(await restore()).toBeOK();
@@ -47,6 +47,17 @@ async function serverJson(request, url) {
return response.json();
}
/**
* The restore job's own end (#361): its progress stream closes when the job takes its final state, however long the
* disk-bound publish takes (bounded by the test, not by a 5 s expect), and the job finished. The toast the page shows
* for it is then the page's reaction alone, checked with the default timeout.
*/
async function restoreFinished(request, id) {
await expect(await request.get(`/prompt_manager/jobs/${id}/events`)).toBeOK();
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
expect([job.kind, job.state]).toEqual(['restore', 'finished']);
}
async function openBackups(page) {
// A fresh page each time: the drafts live for the page session (Q10), and a hash-only goto would not reload.
if (page.url().endsWith('#/system/backups')) await page.reload();
@@ -216,6 +227,7 @@ test('Restore starts the job, opens Activity; views reload after it finishes', a
const { job } = await (await applied).json();
expect([job.kind, job.title, job.cancellable]).toEqual(['restore', 'Database restore', false]);
await expect(page).toHaveURL(`${workspace}#/activity/${job.id}`);
await restoreFinished(request, job.id);
await expect(toast(page)).toHaveText(/^Database restore finished · Restored \d+ prompts?, /);
// The library is the backup's again, and the page says so without a reload: the sidebar counts were read again.
expect(await libraryTotal(request)).toBe(restoredTotal);
@@ -244,7 +256,8 @@ test('a finished restore shows the open page again from the restored data', asyn
const card = page.locator('#workspace-main article.prompt-card').filter({ hasText: 'S9 prompt the restore removes' });
await expect(card).toHaveCount(1);
await expect(await request.post('/__test__/release')).toBeOK();
await applied;
const { job } = await (await applied).json();
await restoreFinished(request, job.id);
await expect(toast(page)).toHaveText(/^Database restore finished · /);
// The shelf was read again from the restored library, on the same route.
await expect(card).toHaveCount(0);
@@ -382,14 +395,15 @@ test('a failed settings read leaves the policy disabled with the reason', async
await expect(row.locator('input')).toBeDisabled();
});
test('the legacy admin Backup and Restore keep working', async ({ page, request }) => {
test('the legacy Backup and Restore routes keep working and the admin page links to System', async ({ page, request }) => {
const bytes = await backupBytes(request);
const restored = await request.post('/prompt_manager/restore', { multipart: { database_file: {
name: 'prompts_backup.db', mimeType: 'application/octet-stream', buffer: bytes } } });
await expect(restored).toBeOK();
expect((await restored.json()).success).toBe(true);
// #330: the admin page's Backup DB and Restore DB open System › Backup & export.
await page.goto('/prompt_manager/admin');
await expect(page.locator('#restoreFileInput')).toHaveCount(1);
for (const id of ['#backupBtn', '#restoreBtn']) await expect(page.locator(id)).toHaveAttribute('href', 'workspace#/system/backups');
});
/** Start a fake running job (a metadata scan another tab started), which an exclusive restore must wait for. */
@@ -422,6 +436,7 @@ test('a refused restore reopens its review: Restore retries it, Discard releases
await dialog(page).getByRole('button', { name: 'Restore', exact: true }).click();
const { job } = await (await applied).json();
await expect(page).toHaveURL(`${workspace}#/activity/${job.id}`);
await restoreFinished(request, job.id);
await expect(toast(page)).toHaveText(/^Database restore finished · /);
expect(await stagesOf(request)).toBe(0);
});
@@ -516,7 +516,7 @@ test('a rollover after the review deletes nothing and Activity offers Review aga
test('Clear all logs states both counts, confirms first and reads everything again', async ({ page, request }) => {
await seedLogs(request, LOG_FOLDER);
const truncates = [];
page.on('request', item => { if (item.url().endsWith('/prompt_manager/logs/truncate')) truncates.push(item.url()); });
page.on('request', item => { if (item.url().endsWith('/prompt_manager/logs/truncate')) truncates.push(item.postDataJSON()); });
await openDiagnostics(page);
await expect(lines(page)).toContainText('Database backup verified');
const details = await openFiles(page);
@@ -535,6 +535,10 @@ test('Clear all logs states both counts, confirms first and reads everything aga
await dialog(page).getByRole('button', { name: 'Clear', exact: true }).click();
await expect(toast(page)).toHaveText('Logs cleared · 3 rotated copies deleted.');
expect(truncates).toHaveLength(1);
// #389: Clear sends back exactly the files the dialog was built from, each with its identity string.
expect(truncates[0].expected.map(row => row.name).sort()).toEqual(['prompt_manager.log', 'prompt_manager.log.1',
'prompt_manager.log.2', 'prompt_manager.log.3']);
for (const row of truncates[0].expected) expect(row.identity).toMatch(/^\d+:\d+:\d+$/);
// Read again: the viewer, the file line and the list.
await expect(lines(page)).not.toContainText('Database backup verified');
await expect(details.locator(':scope > p.small').first()).toHaveText(/rotated copies: 0$/);
@@ -564,6 +568,23 @@ test('Clear all logs reads the files again before it states what it deletes', as
expect(await names(request)).toEqual(['prompt_manager.log', 'prompt_manager.log.1', 'prompt_manager.log.2', 'prompt_manager.log.3']);
});
test('a rollover while Clear all logs is open deletes nothing and says why (#389)', async ({ page, request }) => {
await seedLogs(request, { rotated: [1024, 1024], active: 100 });
await openDiagnostics(page);
await openFiles(page);
await operation(page, 'Clear all logs').getByRole('button', { name: 'Clear all logs…', exact: true }).click();
await expect(dialog(page).locator('#modal-body > p')).toHaveText(/^Empties prompt_manager\.log, deletes 2 rotated copies /);
// The log rolls over after the confirmation was built: the copies it listed are not the ones there now.
await expect(await request.post('/__test__/logs/rotate')).toBeOK();
const rolled = await names(request);
const answer = page.waitForResponse(response => response.url().endsWith('/prompt_manager/logs/truncate'));
await dialog(page).getByRole('button', { name: 'Clear', exact: true }).click();
expect((await answer).status()).toBe(409);
await expect(toast(page)).toHaveText('The log files changed since you confirmed; nothing was cleared. '
+ 'Read the list again and clear again.');
expect(await names(request)).toEqual(rolled);
});
test('each listed file downloads by its returned name and a link has no Download', async ({ page, request }) => {
await seedLogs(request, LOG_FOLDER);
await openDiagnostics(page);
+6 -4
View File
@@ -1,6 +1,7 @@
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
import {writeFile} from 'node:fs/promises';
import {test,expect} from './fixtures.mjs';
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
import {captureCanonicalState} from './history-canonical-comparison.mjs';
import {pageInventoryOf,inventoryOf,diffInventory,mapVocabularyConcept,mapVocabularyDialog} from './vocabulary-canonical-comparison.mjs';
@@ -40,7 +41,7 @@ async function expectLibrary(page,{tags='',category='',count}) {
let manifest;
test.beforeEach(async({page,request,browserName})=>{
await startCoverage(page,browserName);manifest=await seed(request);
await startCoverage(page,browserName);await armUnseenRestores(page);manifest=await seed(request);
page.vocabularyErrors=[];page.on('pageerror',error=>page.vocabularyErrors.push(error.message));
});
test.afterEach(async({page,browserName})=>{await saveCoverage(page,browserName);expect(page.vocabularyErrors).toEqual([]);});
@@ -264,7 +265,8 @@ const toast=page=>page.locator('#workspace-toast');
const actionNames=scope=>scope.locator('.dialog-actions button').evaluateAll(nodes=>nodes.map(node=>node.textContent.trim()));
const snapshot=async request=>{const response=await request.get('/__test__/vocabulary/snapshot');await expect(response).toBeOK();return response.json();};
const backup=async request=>{const response=await request.get('/__test__/vocabulary/backup');await expect(response).toBeOK();return response.body();};
const restore=async(request,buffer)=>expect(await request.post('/prompt_manager/restore',{multipart:{database_file:{name:'vocabulary.db',mimeType:'application/octet-stream',buffer}}})).toBeOK();
// A restore the open page has not seen, so its dialog stays to report it (#413).
const restore=async(page,request,buffer)=>expect(await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'vocabulary.db',mimeType:'application/octet-stream',buffer}}}))).toBeOK();
const tagNames=rows=>rows.tags.map(([,name])=>name);
function trackWrites(page) {
const writes=[];page.on('request',item=>{if(item.method()!=='GET')writes.push({method:item.method(),url:new URL(item.url()).pathname,body:item.postDataJSON()});});return writes;
@@ -482,7 +484,7 @@ test('conflict and restore states keep the dialog and write nothing',async({page
const before=await snapshot(request);
box=await openDialog(page,main(page).getByRole('button',{name:'+ Create tag',exact:true}),'Create tag');
await restore(request,saved);
await restore(page,request,saved);
await field(box,'Tag name').fill('sunrise');await button(box,'Apply').click();
await expect(note(box)).toHaveText('Prompt storage was restored since this dialog opened. Nothing was changed.');
await expect(button(box,'Apply')).toBeDisabled();
@@ -622,7 +624,7 @@ test('delete confirmation read failure, missing tag and restore keep Delete disa
await expect(button(box,'Delete')).toBeDisabled();await expect(tagRow(page,'calm').locator('td:nth-child(2)')).toHaveText('2');
await button(box,'Cancel').click();
box=await openRename(page,'mist');await restore(request,saved);
box=await openRename(page,'mist');await restore(page,request,saved);
await button(box,'Delete tag…').click();
await expect(note(box)).toHaveText('Prompt storage was restored since this dialog opened. Nothing was changed.');
await expect(button(box,'Delete')).toBeDisabled();expect(await actionNames(box)).toEqual(['Reload','Cancel','Delete']);
+30 -4
View File
@@ -123,6 +123,34 @@ def _touch(path: Path, size: int, moment: datetime.datetime) -> None:
os.utime(path, (stamp, stamp))
def _append_active(manager, log_dir: Path, size: int) -> None:
"""Grow the active log as this process's handler does (#389).
A second handle appending to the file the handler writes is what Clear
all logs refuses as another program writing to it, so the bytes go
through the handler's own stream; with file logging off, to the file.
"""
def same(path) -> bool:
return os.path.normcase(os.path.realpath(path)) == os.path.normcase(
os.path.realpath(log_dir / ACTIVE)
)
for handler in manager._file_handlers():
stream = getattr(handler, "stream", None)
if stream is None or not same(handler.baseFilename):
continue
handler.acquire()
try:
stream.write("x" * size)
stream.flush()
finally:
handler.release()
return
with open(log_dir / ACTIVE, "ab") as handle:
handle.write(b"x" * size)
def _bounded(value, name, top=MAX_SEED_BYTES):
if isinstance(value, bool) or not isinstance(value, int) or not 0 <= value <= top:
raise web.HTTPBadRequest(text=f"{name} must be an integer 0..{top}")
@@ -169,8 +197,7 @@ def register_diagnostics_commands(app, routes, api, root: Path):
for index, (time_text, level, message) in enumerate(CONCEPT_RECORDS):
record = _record(time_text, level, message, index)
manager.add_to_buffer(record, record.getMessage())
with open(log_dir / ACTIVE, "ab") as handle:
handle.write(b"x" * CONCEPT_ACTIVE)
_append_active(manager, log_dir, CONCEPT_ACTIVE)
for number, size in enumerate(CONCEPT_ROTATED, start=1):
moment = ROTATED_TIME - datetime.timedelta(hours=number)
_touch(log_dir / f"{ACTIVE}.{number}", size, moment)
@@ -194,8 +221,7 @@ def register_diagnostics_commands(app, routes, api, root: Path):
manager.add_to_buffer(record, record.getMessage())
active = _bounded(data.get("active", 0), "active")
if active:
with open(log_dir / ACTIVE, "ab") as handle:
handle.write(b"x" * active)
_append_active(manager, log_dir, active)
rotated = data.get("rotated", [])
if not isinstance(rotated, list) or len(rotated) > 20:
raise web.HTTPBadRequest(text="rotated must be a list of sizes")
+14 -2
View File
@@ -35,7 +35,8 @@ read. These per-server hooks only change what a System test needs on top:
previews for Convert outdated;
- ``POST /__test__/system/jxl-encoder`` ``{available}`` (#317) makes the
server read as having (``cjxl``) or lacking a JPEG XL encoder without
probing; reset restores the thumbnail settings and the real detection.
probing, whatever the host has installed (#371: ``false`` stands in for the
detector itself); reset restores the thumbnail settings and the real detection.
- ``POST /__test__/system/duplicate-prompts`` (S7) adds two prompts whose
text differs only in case and spacing (a duplicate prompt text group);
@@ -118,6 +119,11 @@ def register_system_commands(app, routes, api, root: Path):
}
from py.api import thumbnail_encode
real_detect = thumbnail_encode._detect
def _no_encoder():
return None
monitor = getattr(image_monitor, "_monitor_instance", None)
failures: dict[str, int] = {}
@@ -147,6 +153,7 @@ def register_system_commands(app, routes, api, root: Path):
api._health_report = None
for (owner, name), value in host_values.items():
setattr(owner, name, value)
thumbnail_encode._detect = real_detect
thumbnail_encode.reset_encoders()
image_monitor._monitor_instance = monitor
api.__dict__.pop("_library_resolver", None)
@@ -298,7 +305,12 @@ def register_system_commands(app, routes, api, root: Path):
available = data.get("available")
if not isinstance(available, bool):
raise web.HTTPBadRequest()
# Test-only: stand in the cached detection (no probe).
# Test-only: the fixture, not the host, decides (#371). Without an encoder the
# detector itself stands in, so a re-probe (a "none found" is probed again on
# every settings read) cannot find the host's real encoder; with one the cached
# answer stands, and a failed write still probes the real detector as before.
# Reset restores the real detector.
thumbnail_encode._detect = real_detect if available else _no_encoder
with thumbnail_encode._detect_lock:
thumbnail_encode._detected = "cjxl" if available else None
return web.json_response({"success": True})
@@ -247,6 +247,22 @@ def register_tagging_commands(
model.unavailable = set(data["unavailable"])
return web.json_response({"success": True})
@routes.post("/__test__/tagging/loaded")
async def loaded(request: web.Request) -> web.Response:
"""Mark one fake model loaded (``{"model": "<key>"}``), as a batch that
kept it in memory leaves it; ``/autotag/models`` then reports
``model_loaded``. Reset unloads it."""
data = await request.json() if request.can_read_body else None
if (
request.query
or not isinstance(data, dict)
or set(data) != {"model"}
or data["model"] not in MODELS
):
raise web.HTTPBadRequest()
model.loaded = data["model"]
return web.json_response({"success": True})
@routes.post("/__test__/tagging/download")
async def download_mode(request: web.Request) -> web.Response:
"""Switch the model's download to a fake (never a real download).
+16
View File
@@ -0,0 +1,16 @@
"""Hang guards for waits on disk-bound work (tests only).
A test that waits for a job or handoff whose work is file and SQLite I/O
(a safety backup with fsync, a restore, hashing and unlinking files) waits
on that state; the ceiling below only ends a real hang. Hosted Windows
runners stall disk I/O for tens of seconds at a time: a restore took over
10 s (merge queue, #382), and on PR #405 every test setup in a 50 s window
took up to 9 s against 0.1-0.6 s around it while the test bodies stayed
fast, so a one-file duplicate cleanup missed a 10 s bound (#409).
The ceiling is per wait and stays under the suite's 120 s per-test hang
watchdog. A test with several such waits in a row can reach the watchdog
first; either way a real hang fails, the watchdog with every thread's stack.
"""
DISK_BOUND_SECONDS = 90
+16
View File
@@ -113,3 +113,19 @@ by their own suites. The expectations start as the observed baseline. A change
that alters an outcome edits the vector in the same commit.
`tests/js/legacy-metadata-characterization.test.js` pins what
the legacy viewers show on the same files.
# Legacy contract snapshot
`legacy_contract_snapshot.json` records what the legacy admin and gallery pages
talk to while System S13 (#330) moves their tools to native System: every
registered route as `"<METHOD> <path>"` (`routes`) and the key paths of
`GET /prompt_manager/settings` to depth 2 (`settings_keys`; list items give the
union of their keys as `name[].key`). `tests/test_legacy_contract_snapshot.py`
compares a temp-database API against it. Record it again only for an intended
change, and say why in the PR body:
```sh
PM_RECORD_LEGACY_SNAPSHOT=1 python3 -m unittest tests.test_legacy_contract_snapshot
```
Cutover (#334) removes the test and this fixture.
+200
View File
@@ -0,0 +1,200 @@
{
"routes": [
"DELETE /prompt_manager/delete/{prompt_id}",
"DELETE /prompt_manager/image-annotations/capture/{token}",
"DELETE /prompt_manager/images/favorite",
"DELETE /prompt_manager/images/{image_id}",
"DELETE /prompt_manager/pairs/{pair_id}",
"DELETE /prompt_manager/prompts/{prompt_id}/tags",
"DELETE /prompt_manager/restore/preview/{token}",
"DELETE /prompt_manager/saved-views/{view_id}",
"DELETE /prompt_manager/tags/{tag_name}",
"GET /prompt_manager/admin",
"GET /prompt_manager/autotag/models",
"GET /prompt_manager/backup",
"GET /prompt_manager/backup/status",
"GET /prompt_manager/categories",
"GET /prompt_manager/diagnostics",
"GET /prompt_manager/export",
"GET /prompt_manager/gallery",
"GET /prompt_manager/gallery.html",
"GET /prompt_manager/gallery/subfolders",
"GET /prompt_manager/history/runs/{run_id}",
"GET /prompt_manager/history/runs/{run_id}/output/current/inspection",
"GET /prompt_manager/history/runs/{run_id}/output/current/media",
"GET /prompt_manager/history/runs/{run_id}/output/current/thumbnail",
"GET /prompt_manager/history/runs/{run_id}/outputs",
"GET /prompt_manager/image-annotations/capture/{token}",
"GET /prompt_manager/images/all",
"GET /prompt_manager/images/inspect/{image_path}",
"GET /prompt_manager/images/output",
"GET /prompt_manager/images/prompt/{image_path}",
"GET /prompt_manager/images/recent",
"GET /prompt_manager/images/search",
"GET /prompt_manager/images/serve/{filepath}",
"GET /prompt_manager/images/thumbnails/plan",
"GET /prompt_manager/images/thumbnails/status",
"GET /prompt_manager/images/{image_id}/file",
"GET /prompt_manager/jobs",
"GET /prompt_manager/jobs/{job_id}",
"GET /prompt_manager/jobs/{job_id}/events",
"GET /prompt_manager/jobs/{job_id}/report",
"GET /prompt_manager/js/{filepath}",
"GET /prompt_manager/lib/{filepath}",
"GET /prompt_manager/library",
"GET /prompt_manager/library/export",
"GET /prompt_manager/library/options",
"GET /prompt_manager/library/{positive_id}",
"GET /prompt_manager/library/{positive_id}/history",
"GET /prompt_manager/library/{positive_id}/history/pairs/{pair_id}/newest",
"GET /prompt_manager/library/{positive_id}/images",
"GET /prompt_manager/logs",
"GET /prompt_manager/logs/config",
"GET /prompt_manager/logs/download/{filename}",
"GET /prompt_manager/logs/files",
"GET /prompt_manager/logs/stats",
"GET /prompt_manager/lora/detect",
"GET /prompt_manager/lora/status",
"GET /prompt_manager/lora/trigger-words",
"GET /prompt_manager/maintenance/duplicates/{job_id}",
"GET /prompt_manager/pairs",
"GET /prompt_manager/pairs/export",
"GET /prompt_manager/pairs/{pair_id}",
"GET /prompt_manager/prompts/{prompt_id}/images",
"GET /prompt_manager/recent",
"GET /prompt_manager/review/events",
"GET /prompt_manager/review/items",
"GET /prompt_manager/review/items/{item_id}",
"GET /prompt_manager/saved-views",
"GET /prompt_manager/scan/status",
"GET /prompt_manager/scan_duplicates",
"GET /prompt_manager/scan_duplicates/status",
"GET /prompt_manager/scan_output_dir",
"GET /prompt_manager/search",
"GET /prompt_manager/settings",
"GET /prompt_manager/stats",
"GET /prompt_manager/subfolders",
"GET /prompt_manager/system/diagnostics",
"GET /prompt_manager/system/health",
"GET /prompt_manager/system/library",
"GET /prompt_manager/system/overview",
"GET /prompt_manager/system/statistics",
"GET /prompt_manager/tags",
"GET /prompt_manager/tags/filter",
"GET /prompt_manager/tags/stats",
"GET /prompt_manager/tags/suggest",
"GET /prompt_manager/tags/{tag_name}/prompts",
"GET /prompt_manager/test",
"GET /prompt_manager/vocabulary/categories",
"GET /prompt_manager/vocabulary/tags",
"GET /prompt_manager/vocabulary/tags/{tag_id}/impact",
"GET /prompt_manager/web",
"GET /prompt_manager/workspace",
"PATCH /prompt_manager/library/{positive_id}",
"PATCH /prompt_manager/pairs/{pair_id}",
"PATCH /prompt_manager/saved-views/{view_id}",
"PATCH /prompt_manager/vocabulary/tags/{tag_id}",
"POST /prompt_manager/autotag/apply",
"POST /prompt_manager/autotag/batch/preview",
"POST /prompt_manager/autotag/download/{model_type}",
"POST /prompt_manager/autotag/single",
"POST /prompt_manager/autotag/start",
"POST /prompt_manager/autotag/unload",
"POST /prompt_manager/bulk/category",
"POST /prompt_manager/bulk/delete",
"POST /prompt_manager/bulk/tags",
"POST /prompt_manager/cleanup",
"POST /prompt_manager/delete_duplicate_images",
"POST /prompt_manager/diagnostics/test-link",
"POST /prompt_manager/gallery/rescan",
"POST /prompt_manager/image-annotations/observe",
"POST /prompt_manager/image-annotations/suggest",
"POST /prompt_manager/images/clear-thumbnails",
"POST /prompt_manager/images/generate-thumbnails",
"POST /prompt_manager/images/generate-thumbnails/progress",
"POST /prompt_manager/images/link",
"POST /prompt_manager/images/reveal",
"POST /prompt_manager/jobs/autotag",
"POST /prompt_manager/jobs/duplicates",
"POST /prompt_manager/jobs/image-tags",
"POST /prompt_manager/jobs/model-download",
"POST /prompt_manager/jobs/scan",
"POST /prompt_manager/jobs/thumbnails",
"POST /prompt_manager/jobs/{job_id}/cancel",
"POST /prompt_manager/library",
"POST /prompt_manager/library/{positive_id}/variants",
"POST /prompt_manager/logs/config",
"POST /prompt_manager/logs/truncate",
"POST /prompt_manager/lora/enable",
"POST /prompt_manager/lora/import/apply",
"POST /prompt_manager/lora/import/preview",
"POST /prompt_manager/lora/refresh-cache",
"POST /prompt_manager/lora/scan",
"POST /prompt_manager/maintenance",
"POST /prompt_manager/maintenance/apply",
"POST /prompt_manager/maintenance/preview",
"POST /prompt_manager/pairs",
"POST /prompt_manager/prompts/tags",
"POST /prompt_manager/prompts/{prompt_id}/tags",
"POST /prompt_manager/restore",
"POST /prompt_manager/restore/apply",
"POST /prompt_manager/restore/preview",
"POST /prompt_manager/review/items/{item_id}/decision",
"POST /prompt_manager/save",
"POST /prompt_manager/saved-views",
"POST /prompt_manager/scan",
"POST /prompt_manager/scan/cancel",
"POST /prompt_manager/scan_duplicates/stream",
"POST /prompt_manager/settings",
"POST /prompt_manager/settings/validate-source",
"POST /prompt_manager/system/health",
"POST /prompt_manager/system/library/keep",
"POST /prompt_manager/system/library/move",
"POST /prompt_manager/tags/merge",
"POST /prompt_manager/vocabulary/categories/apply",
"POST /prompt_manager/vocabulary/categories/preview",
"POST /prompt_manager/vocabulary/tags",
"POST /prompt_manager/vocabulary/tags/{tag_id}/delete",
"PUT /prompt_manager/image-annotations/tags",
"PUT /prompt_manager/images/favorite",
"PUT /prompt_manager/prompts/{prompt_id}",
"PUT /prompt_manager/prompts/{prompt_id}/rating",
"PUT /prompt_manager/tags/{tag_name}"
],
"settings_keys": [
"monitor",
"monitor.observer_alive",
"monitor.present",
"monitor.scheduled",
"revision",
"settings",
"settings.autotag_character_threshold",
"settings.autotag_existing",
"settings.autotag_general_threshold",
"settings.autotag_model_type",
"settings.autotag_suggestion_limit",
"settings.autotag_template",
"settings.cpu_count",
"settings.gallery_root_path",
"settings.gallery_root_paths",
"settings.infinite_scroll",
"settings.monitored_directories",
"settings.monitoring_enabled",
"settings.result_timeout",
"settings.safety_backup",
"settings.thumbnail_edge",
"settings.thumbnail_format",
"settings.thumbnail_quality",
"settings.webui_display_mode",
"settings.worker_threads",
"sources",
"sources.auto_detected",
"sources.configured",
"sources.effective",
"success",
"thumbnail_encoders",
"thumbnail_encoders.jpeg",
"thumbnail_encoders.jxl",
"thumbnail_encoders.webp"
]
}
+85 -19
View File
@@ -76,35 +76,101 @@ test("the workspace keys have their own defaults and validation", () => {
assert.equal(normalizeGallerySettings({ defaultViewMode: "masonry" }).defaultViewMode, "grid");
});
test("a legacy gallery Save keeps the workspace keys and every other stored key", () => {
// #330 S13: the legacy gallery's Page options edit only the page-only keys (PAGE_OPTION_KEYS); System › Appearance owns the rest, so a Save
// or a Reset from the legacy page keeps every other stored key.
function legacyGallery(stored, controls, extra = {}) {
const fs = require("node:fs");
const path = require("node:path");
const vm = require("node:vm");
const settings = require("../../web/js/gallery-settings.js");
const storage = new Map([["gallerySettings", JSON.stringify({ workspaceLayout: "masonry", workspaceDensity: "compact", someOtherTool: 1 })]]);
const controls = {
lazyLoadingToggle: { checked: true }, imageQualitySelect: { value: "high" }, defaultViewModeSelect: { value: "list" },
defaultLimitSelect: { value: "200" }, gridColumnsSelect: { value: "6" }, showImageInfoToggle: { checked: false },
autoLoadMetadataToggle: { checked: true }, cacheMetadataToggle: { checked: true }, showFilePathsToggle: { checked: false },
sidebarCollapsedByDefaultToggle: { checked: true }, infiniteScrollToggle: { checked: false }, debugModeToggle: { checked: false },
apiTimeoutInput: { value: "30" }, checkThumbnailsAtStartup: { checked: true }, videoAutoplayToggle: { checked: true },
videoMuteToggle: { checked: false }, videoLoopToggle: { checked: true },
};
const storage = new Map([["gallerySettings", JSON.stringify(stored)]]);
const context = vm.createContext({
document: { addEventListener() {}, getElementById: id => controls[id] },
window: {}, console: { debug() {}, warn() {}, error() {} }, GallerySettings: settings,
localStorage: { getItem: key => storage.get(key) ?? null, setItem: (key, value) => storage.set(key, value) },
localStorage: { getItem: key => storage.get(key) ?? null, setItem: (key, value) => storage.set(key, value),
removeItem: key => storage.delete(key) },
...extra,
});
const file = path.join(__dirname, "../../web/js/gallery.js");
const Manager = vm.runInContext(`${fs.readFileSync(file, "utf8")}\nGalleryManager;`, context, { filename: file });
const manager = Object.create(Manager.prototype);
Object.assign(manager, { showNotification() {}, hideSettings() {}, applySettings() {} });
const calls = [];
Object.assign(manager, { showNotification: (message, type) => calls.push([type, message]), hideSettings() {},
applySettings: applied => calls.push(["apply", applied.lazyLoading]) });
return { manager, storage, calls, saved: () => JSON.parse(storage.get("gallerySettings")) };
}
const PAGE_OPTION_CONTROLS = () => ({
checkThumbnailsAtStartup: { checked: false }, lazyLoadingToggle: { checked: false },
defaultLimitSelect: { value: "200" },
gridColumnsSelect: { value: "6" }, infiniteScrollToggle: { checked: true }, debugModeToggle: { checked: true },
apiTimeoutInput: { value: "45" },
});
test("a legacy gallery Save keeps the workspace keys and every other stored key", () => {
const { manager, saved } = legacyGallery({ workspaceLayout: "masonry", workspaceDensity: "compact", someOtherTool: 1,
imageQuality: "high", showImageInfo: false, videoMute: false, thumbnailPromptDismissed: "sig", defaultViewMode: "list" },
PAGE_OPTION_CONTROLS());
manager.saveSettings();
const saved = JSON.parse(storage.get("gallerySettings"));
assert.equal(saved.workspaceLayout, "masonry");
assert.equal(saved.workspaceDensity, "compact");
assert.equal(saved.someOtherTool, 1);
assert.equal(saved.defaultViewMode, "list");
assert.equal(saved.showImageInfo, false);
assert.equal(saved.videoMute, false);
const result = saved();
assert.equal(result.workspaceLayout, "masonry");
assert.equal(result.workspaceDensity, "compact");
assert.equal(result.someOtherTool, 1);
assert.equal(result.imageQuality, "high");
assert.equal(result.showImageInfo, false);
assert.equal(result.videoMute, false);
assert.equal(result.thumbnailPromptDismissed, "sig");
// Page options no longer edit the default view (S13 G5), so a stored one stays.
assert.equal(result.defaultViewMode, "list");
// The page-only keys come from the dialog.
assert.equal(result.lazyLoading, false);
assert.equal(result.debugMode, true);
assert.equal(result.checkThumbnailsAtStartup, false);
assert.equal(result.defaultLimit, 200);
assert.equal(result.gridColumns, 6);
assert.equal(result.infiniteScroll, true);
assert.equal(result.apiTimeout, 45);
});
test("a legacy gallery Reset restores only the page options and keeps every other stored key", () => {
const controls = PAGE_OPTION_CONTROLS();
const { manager, saved, calls } = legacyGallery({ workspaceLayout: "masonry", imageQuality: "high", videoMute: false,
thumbnailPromptDismissed: "sig", thumbnailsGenerated: true, lazyLoading: false, debugMode: true, defaultLimit: 200,
gridColumns: 6, infiniteScroll: true, apiTimeout: 45, checkThumbnailsAtStartup: false, defaultViewMode: "list" }, controls,
{ confirm: message => (calls.push(["confirm", message]), true) });
manager.resetSettings();
const result = saved();
const { DEFAULT_SETTINGS } = require("../../web/js/gallery-settings.js");
for (const key of ["checkThumbnailsAtStartup", "lazyLoading", "defaultLimit", "gridColumns",
"infiniteScroll", "debugMode", "apiTimeout"]) {
assert.equal(result[key], DEFAULT_SETTINGS[key], key);
}
// The default view is not a page option (S13 G5): Reset keeps a stored one.
assert.equal(result.defaultViewMode, "list");
assert.equal(result.workspaceLayout, "masonry");
assert.equal(result.imageQuality, "high");
assert.equal(result.videoMute, false);
assert.equal(result.thumbnailPromptDismissed, "sig");
assert.equal(result.thumbnailsGenerated, true);
// The dialog shows the defaults again and the page applies them.
assert.equal(controls.lazyLoadingToggle.checked, DEFAULT_SETTINGS.lazyLoading);
assert.equal(controls.apiTimeoutInput.value, DEFAULT_SETTINGS.apiTimeout);
assert.deepEqual(calls, [["confirm", "Reset these page options to their defaults?"], ["apply", DEFAULT_SETTINGS.lazyLoading],
["success", "Page options reset to defaults"]]);
});
test("Page options no longer hold the default view; System › Appearance's Default layout is the one control", () => {
const { manager } = legacyGallery({}, {});
assert.deepEqual([...manager.constructor.PAGE_OPTION_KEYS], ["checkThumbnailsAtStartup", "lazyLoading", "defaultLimit",
"gridColumns", "infiniteScroll", "debugMode", "apiTimeout"]);
});
test("a viewer mute or loop click remembers only that key and never reads the Page options dialog", () => {
const { manager, saved, calls } = legacyGallery({ defaultLimit: 200, lazyLoading: false, videoMute: true }, {});
manager.storeSetting("videoMute", false);
const result = saved();
assert.equal(result.videoMute, false);
assert.equal(result.defaultLimit, 200);
assert.equal(result.lazyLoading, false);
assert.deepEqual(calls, []);
});
+15 -12
View File
@@ -112,11 +112,12 @@ test('saving gallery settings retains the remembered thumbnail dismissal', () =>
});
// #249 System S5: the stream's work is now an Activity job that outlives the page's connection, so the gallery's
// Cancel asks the server to stop the job. The stream's first frame, `job {id, started}`, names the job; the gallery
// Cancel asks the server to stop the job. Since #330 the startup dialog is the gallery's only thumbnail stream (System
// › Sources & scans generates the rest). The stream's first frame, `job {id, started}`, names the job; the gallery
// cancels by that id, only a job this stream started, and a Cancel before the id waits for it (review I1).
const JOB = 'b'.repeat(32);
function streamFixture({ modal = false, postFails = false } = {}) {
function streamFixture({ postFails = false } = {}) {
const posts = [];
const warnings = [];
const gets = [];
@@ -146,18 +147,20 @@ function streamFixture({ modal = false, postFails = false } = {}) {
return { success: true };
} } });
const emit = (type, data) => (listeners[type] || []).forEach(handler => handler({ data: JSON.stringify(data) }));
// The dialog's Cancel runs the handler and from then on reads as cancelled, as the startup dialog does.
let cancelled = false;
const ui = { addStatusMessage() {}, progressText: element(), progressPercent: element(), progressBar: element(),
progressDetails: element(), progressETA: element(), isCancelled: () => false,
onCancel: handler => { ui.cancel = handler; } };
const result = modal ? manager.generateThumbnailsWithProgressInModal(ui)
: manager.generateThumbnailsWithProgress({ quality: 'medium' });
progressDetails: element(), progressETA: element(), isCancelled: () => cancelled,
onCancel: handler => { ui.cancel = () => { cancelled = true; handler(); }; } };
const result = manager.generateThumbnailsWithProgressInModal(ui);
const outcome = result.then(value => ({ value }), error => ({ error: error.message }));
const cancel = () => (modal ? ui.cancel() : elements.cancelThumbnailsBtn.click());
const cancel = () => ui.cancel();
const settle = () => new Promise(resolve => setImmediate(resolve));
return { emit, source, posts, gets, warnings, outcome, cancel, settle };
}
const cancelUrl = id => `/prompt_manager/jobs/${id}/cancel`;
const CANCELLED = 'Thumbnail generation cancelled; previews already written are kept';
test('Cancel before the job id arrives cancels that job once it does', async () => {
const f = streamFixture();
@@ -165,8 +168,8 @@ test('Cancel before the job id arrives cancels that job once it does', async ()
await f.settle();
assert.deepEqual(f.posts, []);
assert.equal(f.source.closed, 0); // kept open until the stream names its job
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
f.emit('status', { phase: 'scanning', message: 'Scanning…' });
assert.deepEqual(await f.outcome, { error: CANCELLED });
// The job frame is the stream's first (the dialog closes the stream on any later frame once cancelled).
f.emit('job', { id: JOB, started: true });
await f.settle();
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
@@ -185,7 +188,7 @@ test('Cancel after the id cancels it at once, and a double Cancel posts once', a
await f.settle();
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
assert.equal(f.source.closed, 1);
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
assert.deepEqual(await f.outcome, { error: CANCELLED });
assert.deepEqual(f.gets, []);
});
@@ -216,11 +219,11 @@ test('a cancel post that fails is reported, not thrown', async () => {
await f.settle();
assert.equal(f.posts.length, 1);
assert.equal(f.warnings.length, 1);
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
assert.deepEqual(await f.outcome, { error: CANCELLED });
});
test('the startup dialog cancels its own job by id too', async () => {
const f = streamFixture({ modal: true });
const f = streamFixture();
f.cancel();
await f.settle();
assert.deepEqual(f.posts, []);
+19 -113
View File
@@ -65,109 +65,12 @@ function load(file, className, fetchUrl, extra = {}, dom = elements()) {
const BUSY = 'Scan for metadata already running';
test('gallery cache clear reports a refused server clear and never claims success', async () => {
const calls = [];
const { page, dom, notifications, storage } = load('gallery.js', 'GalleryManager', async url => {
calls.push(url);
return jsonResponse(409, { success: false, error: BUSY });
});
page.getSettings = () => ({ thumbnailsGenerated: true });
dom.getElementById('thumbnailStatus').textContent = 'Generated';
await page.clearCache();
assert.deepEqual(calls, ['/prompt_manager/images/clear-thumbnails']);
assert.deepEqual(notifications, [['error', `Thumbnails were not cleared: ${BUSY}`]]);
assert.equal(dom.nodes.thumbnailStatus.textContent, 'Generated');
assert.equal(storage.get('gallerySettings'), undefined); // thumbnailsGenerated is not reset
});
test('gallery cache clear says what it cleared and keeps preferences and other data (#222)', async () => {
const calls = [];
const { page, dom, notifications, storage } = load('gallery.js', 'GalleryManager', async url => {
calls.push(url);
return jsonResponse(200, { success: true, cleared_count: 3 });
});
page.getSettings = () => ({ thumbnailsGenerated: true });
await page.clearCache();
// The server's thumbnails are cleared by their own explicit call; nothing Prompt Manager owns is cached in this
// browser, so no browser entry is removed and the toast says exactly that.
assert.deepEqual(calls, ['/prompt_manager/images/clear-thumbnails']);
assert.deepEqual(notifications, [['success',
"Thumbnails cleared; original images untouched. Nothing of Prompt Manager's is cached in this browser."]]);
assert.equal(dom.nodes.thumbnailStatus.textContent, 'Not generated');
assert.equal(JSON.parse(storage.get('gallerySettings')).thumbnailsGenerated, false);
assert.equal(storage.get('userData'), 'keep');
assert.equal(storage.get('pm_cached'), '1');
assert.equal(storage.get('pm_workspace_theme'), 'light');
assert.equal(storage.get('pm_update_notice_dismissed'), '3.3.0');
});
test("gallery cache clear never deletes another tool's gallery or thumbnail caches (#222)", async () => {
const names = new Set(['comfy-gallery', 'thumbnail-cache']);
const caches = { keys: async () => [...names], delete: async name => names.delete(name) };
const { page, notifications } = load('gallery.js', 'GalleryManager',
async () => jsonResponse(200, { success: true }), { caches, window: { caches } });
page.getSettings = () => ({});
await page.clearCache();
assert.deepEqual([...names], ['comfy-gallery', 'thumbnail-cache']);
assert.equal(notifications.at(-1)[0], 'success');
});
test('gallery cache clear reports a lost server clear instead of success', async () => {
const { page, dom, notifications, storage } = load('gallery.js', 'GalleryManager', async () => {
throw new TypeError('Failed to fetch');
});
page.getSettings = () => ({ thumbnailsGenerated: true });
dom.getElementById('thumbnailStatus').textContent = 'Generated';
await page.clearCache();
assert.deepEqual(notifications, [['error', 'Thumbnails were not cleared: the server could not be reached.']]);
assert.equal(dom.nodes.thumbnailStatus.textContent, 'Generated');
assert.equal(storage.get('gallerySettings'), undefined);
});
test('gallery cache clear says so when browser storage is denied', async () => {
const denied = { get length() { throw new Error('SecurityError'); }, key() { throw new Error('SecurityError'); },
getItem() { throw new Error('SecurityError'); }, setItem() { throw new Error('SecurityError'); },
removeItem() { throw new Error('SecurityError'); } };
const { page, notifications } = load('gallery.js', 'GalleryManager',
async () => jsonResponse(200, { success: true }), { localStorage: denied });
page.getSettings = () => ({});
await page.clearCache();
assert.deepEqual(notifications, [['warning',
'Thumbnails cleared; original images untouched. This browser refused access to its storage, so nothing there was cleared.']]);
});
test('admin LoRA import shows a refusal frame and closes its modal', async () => {
const { page, dom, timers, notifications } = load('admin.js', 'PromptAdmin',
async () => sseResponse([{ type: 'error', status: BUSY, progress: 0, message: BUSY }]));
page.saveLoraSettings = async () => true;
page.loadStatistics = () => assert.fail('a refused import must not reload statistics');
await page.runLoraImport();
assert.equal(dom.nodes.loraImportStatus.textContent, `Error: ${BUSY}`);
assert.deepEqual(notifications.filter(([type]) => type === 'error'), [['error', `LoRA import failed: ${BUSY}`]]);
assert.equal(timers.length, 1);
timers[0]();
assert.deepEqual(notifications.at(-1), ['hide', 'loraImportModal']);
});
test('admin settings save shows why config.json refused it', async () => {
const message = 'config.json cannot be read, so nothing was saved. Fix or remove the file, then save again.';
const { page, notifications } = load('admin.js', 'PromptAdmin',
async () => jsonResponse(409, { success: false, persisted: false, code: 'config-unreadable', error: message }));
page.settings = {};
page._collectScanPaths = () => [];
page.saveLoraSettings = async () => assert.fail('nothing else is saved after a refusal');
page.setInfiniteScroll = () => assert.fail('nothing changes after a refusal');
await page.saveSettings();
@@ -178,27 +81,30 @@ test('admin settings save keeps its plain message when the reply has no reason',
const { page, notifications } = load('admin.js', 'PromptAdmin', async () => ({ ok: false, status: 500,
json: async () => { throw new SyntaxError('not json'); } }));
page.settings = {};
page._collectScanPaths = () => [];
await page.saveSettings();
assert.deepEqual(notifications, [['error', 'Failed to save settings']]);
});
test('admin maintenance names the job that refused it', async () => {
const { page, dom, notifications } = load('admin.js', 'PromptAdmin',
async () => jsonResponse(409, { success: false, error: BUSY }), {
escapeHtml: text => String(text),
document: { readyState: 'loading', addEventListener() {}, getElementById: id => dom.getElementById(id),
querySelectorAll: selector => (selector === '.maintenance-option:checked' ? [{ id: 'vacuum' }] : []) },
});
// #330 S13-9: the admin page's reduced Page options save sends only the list's scroll default, so a stale legacy form
// can no longer overwrite scan folders or other settings edited in System.
test('admin Page options save posts only infinite_scroll', async () => {
const sent = [];
const { page, dom, notifications } = load('admin.js', 'PromptAdmin', async (url, init) => {
sent.push([url, init.method, JSON.parse(init.body)]);
return jsonResponse(200, { success: true, restart_required: true });
});
page.settings = { infiniteScroll: false };
const scrolled = [];
page.setInfiniteScroll = (on, options) => scrolled.push([on, { ...options }]);
dom.getElementById('infiniteScrollDefault').checked = true;
await page.runMaintenance();
await page.saveSettings();
const results = dom.nodes.maintenanceResults.innerHTML;
assert.match(results, /Maintenance Failed/);
assert.match(results, new RegExp(BUSY));
assert.doesNotMatch(results, /HTTP 409/);
assert.deepEqual(notifications, [['error', '❌ Maintenance failed']]);
assert.equal(dom.nodes.runMaintenanceBtn.disabled, false);
assert.deepEqual(sent, [['/prompt_manager/settings', 'POST', { infinite_scroll: true }]]);
assert.deepEqual(scrolled, [[true, { session: false }]]);
assert.equal(page.settings.infiniteScroll, true);
assert.deepEqual(notifications, [['success', 'Page options saved'], ['hide', 'settingsModal']]);
});
+17
View File
@@ -234,6 +234,23 @@ test('the count line reads N of M images and K positive prompts only when known'
assert.equal(countLine({loaded:1200,total:3400,positiveCount:0}),'1,200 of 3,400 images · 0 positive prompts');
});
test('the count line reads as loading, never as an empty result, until a first page answers (#391)', async () => {
const {countLine,createImageStream}=await source;
assert.equal(countLine({loaded:0,total:0,positiveCount:null,loading:true,error:null}),'Loading images…');
// A failed first page and a settled empty one say what the server said; a later page keeps the loaded count.
assert.equal(countLine({loaded:0,total:0,positiveCount:null,loading:false,error:new Error('x')}),'0 of 0 images');
assert.equal(countLine({loaded:0,total:0,positiveCount:0,loading:false,error:null}),'0 of 0 images · 0 positive prompts');
assert.equal(countLine({loaded:40,total:102,positiveCount:2,loading:true,error:null}),'40 of 102 images · 2 positive prompts');
// While a reset's first page is in flight the line reads as loading (the cleared state before it is replaced in
// the same task, so it is never painted).
let answer;const lines=[];
const stream=createImageStream({api:{get:()=>new Promise(resolve=>{answer=resolve;})},onChange:state=>lines.push(countLine(state))});
const loading=stream.reset({tags:['cinematic']});
assert.equal(lines.at(-1),'Loading images…');
answer({images:[{path:'a.png',root_index:0}],total:1,has_more:false,next_cursor:null,positive_count:1});await loading;
assert.equal(lines.at(-1),'1 of 1 image · 1 positive prompt');
});
test('the caption date names the newest result day under Newest first only', async () => {
const {captionDate}=await source;
const now=new Date(2026,9,17,15,0);
@@ -175,6 +175,28 @@ test('Clear all logs states both counts and its toast says what was done', async
assert.equal(clearToast(undefined), 'Logs cleared · 0 rotated copies deleted.');
});
test('Clear all logs sends back exactly the files its confirmation showed (#389)', async () => {
const { clearBody, clearFailureText, COPY } = await diagnostics;
const files = [
{ filename: 'prompt_manager.log', kind: 'active', identity: '10:1761000000123456789:9007199254740993' },
{ filename: 'prompt_manager.log.1', kind: 'rotated', identity: '20:1761000000123456789:42' },
{ filename: 'prompt_manager.log.4', kind: 'link', identity: '5:1:7' },
{ filename: 'prompt_manager.log.bak', kind: 'other', identity: '1:1:8' },
{ filename: 'prompt_manager.log.2', kind: 'rotated' },
];
// Identities stay strings: mtime_ns and large inodes are not exact JS numbers.
assert.deepEqual(clearBody(files), { expected: [
{ name: 'prompt_manager.log', identity: '10:1761000000123456789:9007199254740993' },
{ name: 'prompt_manager.log.1', identity: '20:1761000000123456789:42' },
] });
assert.deepEqual(clearBody([]), { expected: [] });
assert.deepEqual(clearBody(null), {});
const refused = Object.assign(new Error('The log files changed since you confirmed; nothing was cleared.'), { status: 409 });
assert.equal(clearFailureText(refused), refused.message);
assert.equal(clearFailureText(Object.assign(new Error('boom'), { status: 500 })), COPY.clearFailed);
assert.equal(clearFailureText(undefined), COPY.clearFailed);
});
test('the test-link dialog names the prompt and how Missing-file links removes the record', async () => {
const { testLinkSentence, promptName } = await diagnostics;
assert.equal(testLinkSentence({ id: 812, text: 'Misty harbor' }), 'Writes a placeholder image record '
+24
View File
@@ -0,0 +1,24 @@
"""A temp SQLite database in a folder no other test process shares (#346).
``PromptManagerAPI`` sweeps its database's folder at startup for the stages,
uploads and safety-backup copies an interrupted restore or backup left. A
fixture database directly in the system temp folder made every other test
process that started an API (another worktree's suite, a pre-push hook) delete
this process's in-flight restore files, and left safety backups behind there.
"""
import tempfile
def private_temp_db(case):
"""Return a closed ``NamedTemporaryFile`` handle for an empty ``.db`` file.
The file lives in a new folder that ``case`` removes at cleanup, after its
own teardown has closed and unlinked the database (cleanups run after
``tearDown``), together with anything a restore or backup left next to it.
"""
folder = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
case.addCleanup(folder.cleanup)
handle = tempfile.NamedTemporaryFile(delete=False, suffix=".db", dir=folder.name)
handle.close()
return handle
+2 -3
View File
@@ -8,7 +8,6 @@ without requiring a full ComfyUI server.
import json
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -18,6 +17,7 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from tests.private_temp import private_temp_db
from utils.hashing import generate_prompt_hash
@@ -25,8 +25,7 @@ class APITestCase(AioHTTPTestCase):
"""Base class that stands up a test aiohttp app with PromptManager routes."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
app = web.Application()
routes = web.RouteTableDef()
+2 -2
View File
@@ -35,6 +35,7 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from tests.private_temp import private_temp_db
from py.api.autotag_routes import MODEL_BUSY_ERROR
from py.api.jobs import JobBusy
from utils.hashing import generate_prompt_hash
@@ -143,8 +144,7 @@ class AutotagAPITestCase(AioHTTPTestCase):
fake engine."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
self.tmp_root = Path(tempfile.mkdtemp())
self.output_dir = self.tmp_root / "output"
self.output_dir.mkdir()
+35
View File
@@ -264,6 +264,41 @@ class TestRestoreRoute(BackupRestoreTestCase):
texts = sorted(p["text"] for p in self.api.db.search_prompts())
self.assertEqual(texts, ["restored a", "restored b"])
async def test_restore_round_trip_keeps_tags_and_image_links(self):
"""A downloaded backup brings back every prompt with its tags and images.
The legacy admin page's Restore form left for System (#330); this keeps
its browser test's data check at the route.
"""
db = self.api.db
originals = {}
for text, tags, image in (
("round trip cat", ["cats", "pets"], "cat.png"),
("round trip dog", ["dogs"], "dog.png"),
):
prompt_id = db.save_prompt(
text=text, tags=tags, prompt_hash=generate_prompt_hash(text)
)
path = self._write(image, b"\x89PNG\r\n\x1a\n")
self.assertTrue(db.link_image_to_prompt(str(prompt_id), path))
originals[text] = (prompt_id, sorted(tags), [path])
resp = await self.client.get("/prompt_manager/backup")
self.assertEqual(resp.status, 200)
backup = self._write("downloaded.db", await resp.read())
self.assertTrue(db.delete_prompt(originals["round trip dog"][0]))
resp = await self._restore(backup)
self.assertEqual(resp.status, 200, await resp.text())
restored = {p["text"]: p for p in self.api.db.search_prompts()}
self.assertEqual(sorted(restored), sorted(originals))
for text, (prompt_id, tags, paths) in originals.items():
self.assertEqual(restored[text]["id"], prompt_id)
self.assertEqual(sorted(restored[text]["tags"]), tags)
images = self.api.db.get_prompt_images(str(prompt_id))
self.assertEqual([image["image_path"] for image in images], paths)
async def test_restore_leaves_no_temp_files_behind(self):
upload = self._make_db_file(["restored"])
created = []
+23 -17
View File
@@ -18,15 +18,19 @@ from pathlib import Path
from unittest.mock import patch
try:
from tests.disk_waits import DISK_BOUND_SECONDS
from tests.open_handles import open_handles_under
from tests.test_api_maintenance import ApplyTestCase, _digest
except ImportError: # discovered with tests/ as the top-level directory
from disk_waits import DISK_BOUND_SECONDS
from open_handles import open_handles_under
from test_api_maintenance import ApplyTestCase, _digest
from py.api import duplicate_files # noqa: E402
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
# Gates and server-state polls. Waits on a find or cleanup job's task use
# DISK_BOUND_SECONDS: those jobs hash, back up and unlink files (#409).
WAIT = 10
FIND = "/prompt_manager/jobs/duplicates"
GROUPS = "/prompt_manager/maintenance/duplicates/"
@@ -83,7 +87,7 @@ class DuplicateFilesTestCase(ApplyTestCase):
self.assertEqual(response.status, 202, text)
self.assertNotIn(self.tmpdir, text)
job = self.api._jobs.get(json.loads(text)["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
return job
async def groups(self, job_id, status=200, **query):
@@ -127,7 +131,7 @@ class DuplicateFilesTestCase(ApplyTestCase):
plan = await self.review(job.id, [self.keep_of(group, path)])
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
return plan, cleanup
async def until(self, predicate):
@@ -178,7 +182,7 @@ class TestFindJob(DuplicateFilesTestCase):
gate.set()
body = await (await legacy).text()
job = self.api._jobs.get(summary["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
events = _sse_payloads(body)
done = events[-1]
self.assertEqual(done["type"], "complete")
@@ -241,7 +245,7 @@ class TestFindJob(DuplicateFilesTestCase):
self.assertTrue((await cancel.json())["cancellation_requested"])
gate.set()
job = self.api._jobs.get(job_id)
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
self.assertEqual(job.state, "cancelled")
last = _sse_payloads(await (await stream).text())[-1]
self.assertEqual(last, {"type": "error", "message": "Duplicate scan cancelled"})
@@ -341,6 +345,8 @@ class TestReview(DuplicateFilesTestCase):
self.assertEqual(plan["protected"], {"count": 0, "reasons": []})
async def test_cannot_delete_keeper_or_outside_group(self):
# About which files may go, not the backup: skip its fsync-heavy copy.
PromptManagerConfig.SAFETY_BACKUP = False
self.trio()
self.write("e.png", b"other pair")
self.write("f.png", b"other pair")
@@ -372,7 +378,7 @@ class TestReview(DuplicateFilesTestCase):
self.assertEqual([row["label"] for row in plan["details"]], ["b.png"])
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters["removed"], 1)
self.assertTrue((self.output_dir / "a.png").is_file())
self.assertTrue((self.output_dir / "sub" / "c.png").is_file())
@@ -496,7 +502,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
plan = await self.review(job.id, keep)
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertTrue(kept.is_file())
self.assertFalse(gone.exists())
self.assertFalse(video_thumb.exists())
@@ -508,7 +514,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.write("b.png", b"new bytes") # changed since the review
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters, {"removed": 1, "kept": 2, "errors": 0})
self.assertEqual((self.output_dir / "b.png").read_bytes(), b"new bytes")
self.assertFalse((self.output_dir / "sub" / "c.png").exists())
@@ -522,7 +528,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.write("a.png", b"edited keeper")
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters["removed"], 0)
self.assertTrue((self.output_dir / "b.png").is_file())
self.assertTrue((self.output_dir / "sub" / "c.png").is_file())
@@ -549,7 +555,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.state, "failed")
self.assertEqual(cleanup.counters, {"removed": 1, "kept": 1, "errors": 1})
self.assertEqual(
@@ -584,7 +590,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters, {"removed": 0, "kept": 3, "errors": 1})
self.assertEqual(
cleanup.outcomes["errors"][0]["detail"], duplicate_files.DELETE_FAILED
@@ -619,7 +625,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with context:
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
finally:
if handle is not None: # pragma: no cover - Windows
handle.close()
@@ -642,7 +648,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters["removed"], 2)
self.assertEqual(open_at_unlink, [])
@@ -703,7 +709,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
# Its records moved to the identical kept copy before the unlink failed.
[image] = self.api.db.get_prompt_images(prompt)
self.assertTrue(image["image_path"].endswith("a.png"))
@@ -719,7 +725,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.assertEqual((busy["code"], busy["running"]["id"]), ("busy", find["id"]))
gate.set()
await asyncio.wait_for(
asyncio.shield(self.api._jobs.get(find["id"]).task), WAIT
asyncio.shield(self.api._jobs.get(find["id"]).task), DISK_BOUND_SECONDS
)
# The token survived the refusal; the cleanup holds off a new find.
release = asyncio.Event()
@@ -737,7 +743,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.assertEqual(legacy.status, 409)
release.set()
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
async def test_cleanup_refused_up_front_when_the_backup_does_not_fit(self):
from py.api import maintenance
@@ -965,7 +971,7 @@ class TestReviewHelpers(DuplicateFilesTestCase):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.second_source())]
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.state, "failed")
self.assertEqual(cleanup.last_event["code"], "sources-changed")
self.assertEqual(self.backups(), [])
@@ -985,7 +991,7 @@ class TestReviewHelpers(DuplicateFilesTestCase):
):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.state, "failed")
self.assertEqual(
cleanup.outcomes["errors"][-1]["detail"], duplicate_files.RECORDS_FAILED
+3 -3
View File
@@ -19,17 +19,17 @@ from pathlib import Path
from unittest.mock import patch
try:
from tests.disk_waits import DISK_BOUND_SECONDS
from tests.open_handles import open_handles_under
from tests import test_api_duplicate_files as dup_tests
except ImportError: # discovered with tests/ as the top-level directory
import test_api_duplicate_files as dup_tests
from disk_waits import DISK_BOUND_SECONDS
from open_handles import open_handles_under
from py.api import duplicate_files # noqa: E402
from py.config import GalleryConfig # noqa: E402
WAIT = 10
def _locator(path):
return os.path.normcase(os.path.realpath(str(path)))
@@ -49,7 +49,7 @@ class SafetyTestCase(dup_tests.DuplicateFilesTestCase):
async def apply_plan(self, plan):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
return cleanup
+44
View File
@@ -1169,3 +1169,47 @@ class HistoryHostCleanupTests(unittest.IsolatedAsyncioTestCase):
fence.validate(epoch, generation)
with self.assertRaises(sqlite3.Error):
api._history_fence()
class SharedTempFolderRestoreTests(ImageAPITestCase):
"""#346: a restore survives another suite starting in the shared temp folder.
Every ``PromptManagerAPI`` sweeps its database's folder for stages and
uploads an interrupted restore left. A fixture database directly in the
shared temp folder let any other test process (another worktree's suite,
a pre-push hook) delete this restore's files mid-request. The shared
folder here is a scratch stand-in for the system temp folder, so the test
never sweeps the real one.
"""
async def get_application(self):
shared = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
# Cleanups run last-in first-out: this one after the fixture's own.
self.addCleanup(shared.cleanup)
self.shared_tmp = shared.name
with patch.object(tempfile, "tempdir", shared.name):
return await super().get_application()
async def test_restore_survives_another_suite_sweeping_the_shared_folder(self):
from database import restore_stage
from database.models import PromptModel
self.api.db.save_prompt("kept by the upload")
source = self.comfy_dir / "upload.db"
self.assertTrue(self.api.db.model.backup_database(str(source)))
stage = PromptModel.stage_restore
swept = []
def stage_then_other_suite_starts(model, src_path):
staged = stage(model, src_path)
# What another suite's PromptManagerAPI does at startup.
other = os.path.join(self.shared_tmp, "other-suite.db")
swept.extend(restore_stage.sweep_restore_temps(other))
return staged
form = FormData()
form.add_field("database_file", source.read_bytes(), filename="upload.db")
with patch.object(PromptModel, "stage_restore", stage_then_other_suite_starts):
response = await self.client.post("/prompt_manager/restore", data=form)
self.assertEqual(response.status, 200, await response.text())
self.assertEqual(swept, [], "another suite's sweep reached this restore")
+2 -2
View File
@@ -42,6 +42,7 @@ from py import lora_utils # noqa: E402
from py import lora_manager_layout as layout_module # noqa: E402
from py.api.thumbnail_settings import ThumbnailSettings # noqa: E402
from py.api import PromptManagerAPI # noqa: E402
from tests.private_temp import private_temp_db # noqa: E402
from py.api.prompts import MAX_PAGE_LIMIT # noqa: E402
from py.config import GalleryConfig # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
@@ -60,8 +61,7 @@ class ImageAPITestCase(AioHTTPTestCase):
"""App with PromptManager routes, a temp DB and a temp output directory."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
self._output_tmp = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
# A fake ComfyUI tree: <tmp>/output is the gallery root and <tmp> is the
+69
View File
@@ -415,5 +415,74 @@ class TestApply(LogCleanupTestCase):
self.assertEqual(job.status_line, "The log cleanup failed.")
TRUNCATE = "/prompt_manager/logs/truncate"
FILES = "/prompt_manager/logs/files"
class TestClearAllLogsConfirmed(LogCleanupTestCase):
"""#389: POST /logs/truncate deletes no more than the confirmation listed."""
async def confirmed(self):
response = await self.client.request("GET", FILES)
data = await response.json()
return [
{"name": row["filename"], "identity": row["identity"]}
for row in data["files"]
if row["kind"] in ("active", "rotated")
]
def handler(self):
return next(
h
for h in self.manager.logger.handlers
if isinstance(h, logging.FileHandler)
)
async def test_a_rollover_after_the_confirmation_is_refused(self):
self.write("prompt_manager.log.1", b"a" * 100)
self.write("prompt_manager.log.2", b"b" * 200)
expected = await self.confirmed()
self.app_logger.info("written after the confirmation")
self.handler().doRollover()
before = self.snapshot()
body = await self.post(TRUNCATE, {"expected": expected}, 409)
self.assertEqual(body["success"], False)
self.assertEqual(body["code"], "logs-changed")
self.assertIn("nothing was cleared", body["error"])
self.assertEqual(self.snapshot(), before)
async def test_the_confirmed_files_are_cleared(self):
self.write("prompt_manager.log.1", b"a" * 100)
expected = await self.confirmed()
body = await self.post(TRUNCATE, {"expected": expected}, 200)
self.assertEqual(
sorted(body["results"]["truncated"]),
["prompt_manager.log", "prompt_manager.log.1"],
)
self.assertEqual(self.names(), ["prompt_manager.log"])
async def test_the_legacy_call_without_a_body_still_clears(self):
self.write("prompt_manager.log.1", b"a" * 100)
response = await self.client.request("POST", TRUNCATE)
self.assertEqual(response.status, 200, await response.text())
self.assertEqual(self.names(), ["prompt_manager.log"])
async def test_a_foreign_writer_is_refused_with_a_clear_message(self):
self.write("prompt_manager.log.1", b"a" * 100)
with open(self.path("prompt_manager.log"), "ab") as other:
other.write(b"line from another ComfyUI\n")
body = await self.post(TRUNCATE, {"expected": await self.confirmed()}, 409)
self.assertEqual(body["code"], "logs-foreign-writer")
self.assertIn("Another program is writing", body["error"])
self.assertIn("prompt_manager.log.1", self.names())
async def test_a_malformed_confirmation_is_400(self):
self.write("prompt_manager.log.1", b"a" * 100)
for expected in ("all", [{"name": "prompt_manager.log.1", "identity": 5}]):
with self.subTest(expected=expected):
await self.post(TRUNCATE, {"expected": expected}, 400)
self.assertIn("prompt_manager.log.1", self.names())
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -36,6 +36,7 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from tests.private_temp import private_temp_db
from py import config_store, lora_import, lora_utils
from py.config import IntegrationConfig, PromptManagerConfig
from py.api import settings_routes as settings_module
@@ -57,8 +58,7 @@ class LoraAPITestCase(AioHTTPTestCase):
"""Stands up the PromptManager routes with a temp DB and neutral LoRA config."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
self.tmp_root = Path(tempfile.mkdtemp())
self.custom_nodes = self.tmp_root / "custom_nodes"
self.custom_nodes.mkdir()
+2 -3
View File
@@ -5,7 +5,6 @@ Route tests for the prompt API (py/api/prompts.py).
import json
import os
import sys
import tempfile
import unittest
from unittest.mock import MagicMock, patch
@@ -29,6 +28,7 @@ except ImportError: # discovered with tests/ as the top-level directory
from database.operations import PromptDatabase # noqa: E402
from py.api import PromptManagerAPI # noqa: E402
from tests.private_temp import private_temp_db # noqa: E402
import py.api.prompts as prompts_module # noqa: E402
from py.api.prompts import ( # noqa: E402
BULK_CATEGORY_MAX_PROMPTS,
@@ -43,8 +43,7 @@ class PromptAPITestCase(AioHTTPTestCase):
"""App with PromptManager routes and a temporary SQLite database."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
app = web.Application()
routes = web.RouteTableDef()
+4 -4
View File
@@ -1040,8 +1040,8 @@ class TestApplyRefusals(ReviewRoutesTestCase):
_path, item, detail, observed, cat = await self._opened()
real = review_capture_module.resolve_prompt_pair
def reworded(metadata):
resolution = real(metadata)
def reworded(metadata, **kwargs):
resolution = real(metadata, **kwargs)
return {
**resolution,
"reason": "a reworded reason from a newer resolver",
@@ -1065,8 +1065,8 @@ class TestApplyRefusals(ReviewRoutesTestCase):
_path, item, detail, observed, cat = await self._opened()
real = review_capture_module.resolve_prompt_pair
def without_cat(metadata):
resolution = real(metadata)
def without_cat(metadata, **kwargs):
resolution = real(metadata, **kwargs)
kept = [
c
for c in resolution["candidates"]
+49 -7
View File
@@ -31,6 +31,12 @@ from utils.execution_history_writer import HistoryWriter
# 0.19 s on macOS py3.13 (PR #351).
NON_BLOCKING_CEILING = 1.0
# Saturation tests block the worker while they queue 1023 more items. The
# block must hold until the test releases it (#346/#352): a 3 s timer let a
# loaded runner unblock the worker mid-fill and drain it. This is only a hang
# guard; the test, or cleanup, always sets the release.
SATURATION_HOLD_SECONDS = 60
class ManualClock:
def __init__(self):
@@ -84,6 +90,26 @@ class HistoryWriterTests(unittest.TestCase):
threading.Event().wait(0.005)
self.fail("finite deadline expired waiting for worker behavior")
def wait_for_worker_pass(self, writer):
"""Wait until the idle worker has run one whole loop pass from now.
A worker with no accepted work ends every pass by parking on its wake
event. The first park counted may end a pass already under way; the
second ends a pass that began after this call, so whatever the worker
decides at the current clock value has happened (state, not a
wall-clock pause that a loaded runner can outlast).
"""
wake = writer._wake
park = wake.wait
parks = []
def counted(timeout=None):
parks.append(timeout)
return park(timeout)
with patch.object(wake, "wait", counted):
self.wait_for(lambda: len(parks) >= 2)
def writer(self, factory=None, ready=True):
writer = HistoryWriter(factory or (lambda: self.db), clock=self.clock)
self.writers.append(writer)
@@ -301,7 +327,7 @@ class HistoryWriterTests(unittest.TestCase):
def write(command):
entered.set()
release.wait(3)
release.wait(SATURATION_HOLD_SECONDS)
return original(command)
with patch.object(self.db, "write_execution_start", write):
@@ -348,9 +374,12 @@ class HistoryWriterTests(unittest.TestCase):
first = self.sealed(writer)
charge = first.lease.byte_count
left = threading.Event()
def write(*args, **kwargs):
entered.set()
release.wait(3)
left.set()
return original(*args, **kwargs)
with patch.object(self.db, "write_execution_contexts_for_finish", write):
@@ -396,8 +425,21 @@ class HistoryWriterTests(unittest.TestCase):
before = monotonic()
self.assertFalse(writer.close(0.02))
elapsed = monotonic() - before
# close() waits until its own monotonic deadline, so the lower
# bound holds on any runner.
self.assertGreaterEqual(elapsed, 0.02)
self.assertLess(elapsed, 0.2)
if early_join:
# Manual clock: only the patched join/sleep advance it, so this
# exact bound proves early joins do not spin forever.
self.assertLess(elapsed, 0.2)
else:
# Real clock: a loaded runner may run late, but close() must
# not wait anywhere near the 3 s the blocked write holds.
self.assertLess(elapsed, 0.02 + NON_BLOCKING_CEILING)
# Bounded on the real clock too: close() gave up while the
# in-flight write was still blocked, instead of waiting it out.
self.assertFalse(left.is_set())
self.assertTrue(writer.status().alive)
self.assertEqual(pool.charged_bytes, charge)
replacement = HistoryWriter(lambda: self.db, context_budget=pool)
self.writers.append(replacement)
@@ -771,7 +813,7 @@ class HistoryWriterTests(unittest.TestCase):
self.wait_for(lambda: not writer.status().accepting)
self.assertFalse(writer.submit(fresh))
self.clock.advance(4.99)
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertEqual(probes, [0])
self.clock.advance(0.01)
self.wait_for(lambda: writer.status().accepting)
@@ -794,12 +836,12 @@ class HistoryWriterTests(unittest.TestCase):
writer = self.writer(factory, ready=False)
self.wait_for(lambda: len(calls) == 1)
self.clock.advance(4.99)
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertEqual(calls, [0])
self.clock.advance(0.01)
self.wait_for(lambda: len(calls) == 2)
self.clock.advance(4.99)
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertEqual(calls, [0, 5])
self.clock.advance(0.01)
self.wait_for(lambda: writer.status().accepting)
@@ -853,7 +895,7 @@ class HistoryWriterTests(unittest.TestCase):
)
writer.invalidate()
release.set()
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertFalse(writer.status().accepting)
self.assertIsNone(writer.status().epoch)
self.assertFalse(writer.submit(self.start_command(epoch=stale_epoch)))
@@ -871,7 +913,7 @@ class HistoryWriterTests(unittest.TestCase):
lock.execute("BEGIN IMMEDIATE")
before = time.monotonic()
self.assertTrue(writer.submit(command))
self.assertLess(time.monotonic() - before, 0.1)
self.assertLess(time.monotonic() - before, NON_BLOCKING_CEILING)
self.wait_for(lambda: not writer.status().accepting)
lock.rollback()
self.clock.advance(5)
+11 -4
View File
@@ -18,6 +18,7 @@ from utils.execution_output_capture import seal_outputs
from utils.execution_outputs import snapshot_output_roots
from utils.execution_history_writer import HistoryWriter
from tests.history_waits import drain_writer
from tests.disk_waits import DISK_BOUND_SECONDS
try:
from tests import test_execution_history_writer as fixtures
@@ -28,11 +29,17 @@ except ImportError:
# Upper bound a blocked fake write waits for its release. Tests prove ordering
# with events, so this only caps a failing run and never decides a pass.
BLOCK_SECONDS = 10
# Saturation tests block the worker while they queue 1023 more items. The
# block must hold until the test releases it (#346/#352): a 3 s timer let a
# loaded runner unblock the worker mid-fill and drain it. This is only a hang
# guard; the test, or cleanup, always sets the release.
SATURATION_HOLD_SECONDS = 60
# A restore is a full verify/migrate/backup/replace cycle and is bound by the
# disk: a busy Windows runner took over 10 s (merge queue, #382). Waits that
# span a restore use this ceiling, which stays under the runner's 120 s hang
# watchdog, so a real hang still fails with the thread dump.
RESTORE_SECONDS = 90
# span a restore use the shared disk-bound hang guard (tests/disk_waits.py).
RESTORE_SECONDS = DISK_BOUND_SECONDS
class OutputWriterTests(fixtures.unittest.TestCase):
@@ -591,7 +598,7 @@ class OutputWriterTests(fixtures.unittest.TestCase):
def write(command):
entered.set()
release.wait(3)
release.wait(SATURATION_HOLD_SECONDS)
return original(command)
with patch.object(self.db, "write_execution_start", write):
+463
View File
@@ -0,0 +1,463 @@
"""Work running at Ctrl+C stops at its next safe point (#376).
ComfyUI catches Ctrl+C in ``main.py`` and returns without closing its event
loop. Python then joins every executor thread (``concurrent.futures``'s
``_python_exit``, a ``threading._register_atexit`` callback) before any
``atexit`` callback runs, so a call still running on the default executor
used to keep the process alive until it returned. The package registers its
own ``threading._register_atexit`` hook, which runs before that join: it sets
the exit signal that ``Job.cancelled`` and ``map_parallel`` read, and kills
the encoder processes.
The child-process test starts three long calls on the default executor (a
registry job checking ``job.cancelled`` between files, a ``map_parallel``
batch and an encoder process that ignores the exit, as ffmpeg does on
Windows), returns from its main module and must exit far sooner than any of
them would end.
"""
import json
import os
import subprocess
import sys
import tempfile
import textwrap
import threading
import unittest
from unittest import mock
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
try:
from utils import exit_signal # noqa: E402
except ImportError: # before #376: only the child-process test can run
exit_signal = None
# The parent never waits longer than this for the child (the hang watchdog
# allows 120 s per test). Each fake job alone would run 300 s or more.
CHILD_CEILING_SECONDS = 60
# From the child's return to its last atexit callback: the stopped calls end
# at their next file, item or chunk (the encoder is killed).
EXIT_BOUND_SECONDS = 15
JOB_SECONDS = 600
# The fake encoder outlives the ceiling, yet a failing run leaks it only
# briefly (the parent cannot reach the child's own child to kill it).
ENCODER_SECONDS = 90
REPORT_PREFIX = "PM_EXIT_REPORT "
CHILD = textwrap.dedent("""
import asyncio
import atexit
import contextlib
import importlib
import io
import json
import os
import sys
import threading
import time
root, tmp = sys.argv[1], sys.argv[2]
job_seconds, encoder_seconds = float(sys.argv[3]), int(sys.argv[4])
sys.path.insert(0, root)
state = {"stopped": {}}
def report():
returned = state.get("returned")
print("PM_EXIT_REPORT " + json.dumps({
"exit_seconds": None if returned is None else time.monotonic() - returned,
"stopped": state["stopped"],
}))
sys.stdout.flush()
# Registered first, so it runs after every other atexit callback, and
# every atexit callback runs after the executor threads were joined.
atexit.register(report)
from tests.test_init_package import _import_package, fake_server_module
output = os.path.join(tmp, "output")
with contextlib.redirect_stdout(io.StringIO()):
package, _ = _import_package(
"pm_exit_child",
os.path.join(tmp, "prompts.db"),
output,
fake_server_module(queue_task_done=True),
)
parallel = importlib.import_module("pm_exit_child.utils.parallel")
encode = importlib.import_module("pm_exit_child.py.api.thumbnail_encode")
registry = package.api._jobs
entered = {name: threading.Event() for name in ("job", "batch", "encoder")}
def job_work(job):
# A job's thread work: one "file" at a time, checking its cancel
# between files the way the thumbnail and import jobs do.
entered["job"].set()
deadline = time.monotonic() + job_seconds
while time.monotonic() < deadline:
if job.cancelled:
state["stopped"]["job"] = True
return
time.sleep(0.05)
def batch_item(index):
entered["batch"].set()
time.sleep(0.5)
return index
def batch_work():
try:
parallel.map_parallel(batch_item, range(int(job_seconds)), 2)
except Exception as error:
state["stopped"]["batch"] = type(error).__name__
def encoder_work():
# Stands for ffmpeg on Windows, which runs without a console and
# never sees Ctrl+C: only a kill ends it before its timeout.
cmd = [sys.executable, "-c", "import time; time.sleep(%d)" % encoder_seconds]
try:
encode.run_encoder(
cmd,
timeout=encoder_seconds,
track=lambda _process: entered["encoder"].set(),
)
except Exception as error:
state["stopped"]["encoder"] = type(error).__name__
async def begin():
loop = asyncio.get_running_loop()
async def runner(job):
await loop.run_in_executor(None, job_work, job)
registry.start("exit-test", "Exit test", admission="read", runner=runner)
loop.run_in_executor(None, batch_work)
loop.run_in_executor(None, encoder_work)
for name, event in entered.items():
if not await loop.run_in_executor(None, event.wait, 30):
sys.exit("not started: " + name)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(begin())
state["returned"] = time.monotonic()
# Return with the three calls running and the loop open, as ComfyUI does
# after Ctrl+C.
""")
class TestExitStopsRunningWork(unittest.TestCase):
def test_running_executor_work_stops_at_exit(self):
with tempfile.TemporaryDirectory() as tmp:
os.makedirs(os.path.join(tmp, "output"))
script = os.path.join(tmp, "child.py")
with open(script, "w", encoding="utf-8") as handle:
handle.write(CHILD)
env = dict(os.environ)
env.pop("PROMPT_MANAGER_CONFIG_PATH", None)
env["PROMPT_MANAGER_DATA_DIR"] = tmp
env["PYTHONIOENCODING"] = "utf-8"
try:
completed = subprocess.run(
[
sys.executable,
script,
ROOT,
tmp,
str(JOB_SECONDS),
str(ENCODER_SECONDS),
],
cwd=ROOT,
env=env,
capture_output=True,
timeout=CHILD_CEILING_SECONDS,
encoding="utf-8",
errors="replace",
)
except subprocess.TimeoutExpired as hung:
self.fail(
f"the child did not exit within {CHILD_CEILING_SECONDS} s\n"
f"stdout:\n{hung.stdout}\nstderr:\n{hung.stderr}"
)
detail = f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
self.assertEqual(completed.returncode, 0, detail)
reports = [
line[len(REPORT_PREFIX) :]
for line in completed.stdout.splitlines()
if line.startswith(REPORT_PREFIX)
]
self.assertEqual(len(reports), 1, detail)
report = json.loads(reports[0])
self.assertLess(report["exit_seconds"], EXIT_BOUND_SECONDS, detail)
self.assertEqual(
report["stopped"],
{"job": True, "batch": "ExitRequested", "encoder": "EncoderClosed"},
detail,
)
for text in ("Traceback", "Background work not stopped in time"):
self.assertNotIn(text, completed.stderr, detail)
class _Isolated(unittest.TestCase):
"""Each test starts with the signal clear and no hook installed."""
def setUp(self):
exit_signal.reset()
self.addCleanup(exit_signal.reset)
class TestInstall(_Isolated):
def test_the_hook_is_registered_once_before_the_executor_join(self):
calls = []
with mock.patch.object(
exit_signal.threading,
"_register_atexit",
side_effect=lambda func: calls.append(func),
create=True,
):
self.assertTrue(exit_signal.install())
self.assertTrue(exit_signal.install())
self.assertEqual(len(calls), 1)
self.assertFalse(exit_signal.requested())
calls[0]()
self.assertTrue(exit_signal.requested())
def test_a_missing_internal_hook_degrades_to_the_old_exit(self):
with mock.patch.object(exit_signal, "threading") as fake:
del fake._register_atexit
self.assertFalse(exit_signal.install())
self.assertFalse(exit_signal.requested())
def test_a_refused_registration_degrades_to_the_old_exit(self):
refuse = RuntimeError("can't register atexit after shutdown")
with mock.patch.object(
exit_signal.threading, "_register_atexit", side_effect=refuse, create=True
):
self.assertFalse(exit_signal.install())
def test_a_failing_exit_action_never_escapes_the_hook(self):
# An exception out of a threading atexit callback would abort
# threading._shutdown and skip the executor join itself.
calls = []
def boom():
raise KeyboardInterrupt # a second Ctrl+C
with mock.patch.object(
exit_signal.threading,
"_register_atexit",
side_effect=lambda func: calls.append(func),
create=True,
):
exit_signal.install(boom)
with self.assertLogs(exit_signal.logger, "ERROR"):
calls[0]()
self.assertTrue(exit_signal.requested())
class TestCheckpoints(_Isolated):
def test_checked_runs_until_the_signal_then_raises(self):
seen = []
record = exit_signal.checked(seen.append)
record(1)
exit_signal.request()
with self.assertRaises(exit_signal.ExitRequested):
record(2)
self.assertEqual(seen, [1])
def test_map_parallel_starts_no_item_after_the_signal(self):
from utils.parallel import map_parallel
started = []
gate = threading.Event()
def item(index):
started.append(index)
if index == 0:
exit_signal.request()
gate.set()
return index
with self.assertRaises(exit_signal.ExitRequested):
map_parallel(item, range(50), 1)
self.assertTrue(gate.is_set())
self.assertEqual(started, [0])
def test_job_cancelled_follows_the_signal_and_its_own_cancel(self):
from py.api.jobs import Job
job = Job("scan", "Scan", admission="records")
self.assertFalse(job.cancelled)
job.cancelled = True
self.assertTrue(job.cancelled)
other = Job("scan", "Scan", admission="records", cancellable=False)
exit_signal.request()
self.assertTrue(other.cancelled)
def test_the_scan_walk_stops_between_directories(self):
from py.api.admin import _collect_media_files
with tempfile.TemporaryDirectory() as tmp:
for name in ("a", "b"):
os.makedirs(os.path.join(tmp, name))
with open(os.path.join(tmp, name, "x.png"), "wb") as image:
image.write(b"x")
self.assertEqual(len(_collect_media_files([tmp], [".png"])), 2)
exit_signal.request()
with self.assertRaises(exit_signal.ExitRequested):
_collect_media_files([tmp], [".png"])
class TestLoraImportStopsAtExit(_Isolated):
def test_staging_stops_before_the_next_sidecar_whatever_the_caller_checks(self):
import types
from py import lora_import
discovery = types.SimpleNamespace(
sidecars=[object(), object()], unreadable=[], layout=None
)
exit_signal.request()
with tempfile.TemporaryDirectory() as tmp:
with self.assertRaises(lora_import.ImportCancelled):
lora_import.stage(
discovery,
tmp,
media_extensions=frozenset(),
cancelled=lambda: False, # the legacy scan's own check
)
class _FakeTqdm:
"""huggingface_hub.utils.tqdm as far as a download uses it."""
def __init__(self, *args, **kwargs):
self.n = 0
def update(self, n=1):
self.n += n
def __enter__(self):
return self
def __exit__(self, *exc):
return False
def _fake_hub(chunks_before_exit):
"""A ``huggingface_hub`` whose download writes chunks to a temp file and,
like huggingface_hub 1.x, removes it on any failure before the rename."""
import types
def hf_hub_download(
repo_id, filename, local_dir=None, local_dir_use_symlinks=None, tqdm_class=None
):
final = os.path.join(local_dir, filename)
partial = final + ".abc.incomplete"
bar = (tqdm_class or _FakeTqdm)(total=10)
try:
with open(partial, "wb") as handle:
for index in range(10):
if index == chunks_before_exit:
exit_signal.request()
bar.update(1)
handle.write(b"x")
os.replace(partial, final)
finally:
if os.path.exists(partial):
os.unlink(partial)
return final
hub = types.ModuleType("huggingface_hub")
hub.hf_hub_download = hf_hub_download
hub.snapshot_download = lambda **kwargs: None
utils = types.ModuleType("huggingface_hub.utils")
utils.tqdm = _FakeTqdm
hub.utils = utils
return {"huggingface_hub": hub, "huggingface_hub.utils": utils}
class TestModelDownloadStopsAtExit(_Isolated):
def test_a_download_stops_between_chunks_and_leaves_no_file(self):
from py.autotag import AutoTagService
with tempfile.TemporaryDirectory() as tmp:
service = AutoTagService(models_dir=tmp)
with mock.patch.dict(sys.modules, _fake_hub(chunks_before_exit=3)):
with self.assertLogs(service.logger, "INFO") as logs:
self.assertFalse(service.download_model("wd14-vit"))
folder = os.path.join(tmp, "WD14-ViT")
self.assertEqual(os.listdir(folder), [])
self.assertTrue(any("exiting" in line for line in logs.output), logs.output)
self.assertFalse(any("Download failed" in line for line in logs.output))
def test_a_download_without_the_signal_finishes(self):
from py.autotag import AutoTagService
with tempfile.TemporaryDirectory() as tmp:
service = AutoTagService(models_dir=tmp)
with mock.patch.dict(sys.modules, _fake_hub(chunks_before_exit=None)):
self.assertTrue(service.download_model("wd14-vit"))
folder = os.path.join(tmp, "WD14-ViT")
self.assertEqual(
sorted(os.listdir(folder)), ["model.onnx", "selected_tags.csv"]
)
def test_xet_transfers_are_aborted_the_way_hub_aborts_on_ctrl_c(self):
import types
from py.autotag import abort_hub_downloads
aborted = []
xet = types.ModuleType("huggingface_hub.utils._xet")
xet.abort_xet_session = lambda: aborted.append(True)
hub = _fake_hub(chunks_before_exit=None)
hub["huggingface_hub.utils._xet"] = xet
hub["huggingface_hub.utils"]._xet = xet
with mock.patch.dict(sys.modules, hub):
abort_hub_downloads()
self.assertEqual(aborted, [True])
with mock.patch.dict(sys.modules):
sys.modules.pop("huggingface_hub", None)
abort_hub_downloads() # never imported: nothing to abort
self.assertEqual(aborted, [True])
def test_an_aborted_xet_download_is_logged_as_stopped(self):
from py.autotag import AutoTagService
hub = _fake_hub(chunks_before_exit=None)
def aborted(*args, **kwargs):
exit_signal.request() # the hook ran, then hub raised
raise RuntimeError("xet session aborted")
hub["huggingface_hub"].hf_hub_download = aborted
with tempfile.TemporaryDirectory() as tmp:
service = AutoTagService(models_dir=tmp)
with mock.patch.dict(sys.modules, hub):
with self.assertLogs(service.logger, "INFO") as logs:
self.assertFalse(service.download_model("wd14-vit"))
self.assertTrue(any("exiting" in line for line in logs.output), logs.output)
self.assertFalse(any("Download failed" in line for line in logs.output))
def test_an_old_hub_without_tqdm_class_downloads_as_before(self):
from py.autotag import AutoTagService
hub = _fake_hub(chunks_before_exit=None)
full = hub["huggingface_hub"].hf_hub_download
def old_download(
repo_id, filename, local_dir=None, local_dir_use_symlinks=None
):
return full(repo_id, filename, local_dir, local_dir_use_symlinks)
hub["huggingface_hub"].hf_hub_download = old_download
with tempfile.TemporaryDirectory() as tmp:
service = AutoTagService(models_dir=tmp)
with mock.patch.dict(sys.modules, hub):
self.assertTrue(service.download_model("wd14-vit"))
if __name__ == "__main__":
unittest.main()
+266
View File
@@ -0,0 +1,266 @@
"""A thread or pool refused at interpreter shutdown is quiet (#375).
When the interpreter exits, ``threading._shutdown`` first shuts every
``ThreadPoolExecutor`` (``submit`` then raises "cannot schedule new futures
after interpreter shutdown") and joins the non-daemon threads; only then do
the ``atexit`` callbacks (the package teardown) run. Our daemon threads keep
working meanwhile. ``Thread.start`` is refused ("can't create new thread at
interpreter shutdown") only later, once the interpreter is finalizing; the
unit tests cover that case, the child process the pool case. Such a refusal
is logged once at DEBUG, with no traceback, and leaves nothing pending.
``threading.Thread`` is replaced only in the module under test (a copy of
the ``threading`` namespace), never process-wide. The child-process test runs
the real exit: a daemon thread queues previews while the main thread returns.
"""
import json
import logging
import os
import subprocess
import sys
import tempfile
import textwrap
import threading
import types
import unittest
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from unittest import mock
from py.api import thumbnail_auto
from utils import image_monitor, parallel
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CHILD_CEILING_SECONDS = 60 # the hang watchdog allows 120 s per test
REPORT_PREFIX = "PM_EXIT_REPORT "
THREAD_REFUSAL = "can't create new thread at interpreter shutdown"
POOL_REFUSAL = "cannot schedule new futures after interpreter shutdown"
def _refusing_threading():
"""A ``threading`` namespace whose ``Thread.start`` is refused."""
class RefusedThread(threading.Thread):
def start(self):
raise RuntimeError(THREAD_REFUSAL)
namespace = types.SimpleNamespace(**vars(threading))
namespace.Thread = RefusedThread
return namespace
class _ShutPool(ThreadPoolExecutor):
def submit(self, *args, **kwargs):
raise RuntimeError(POOL_REFUSAL)
class _Capture(logging.Handler):
def __init__(self):
super().__init__(logging.DEBUG)
self.records = []
def emit(self, record):
self.records.append(record)
def _capture(testcase, name):
parallel._noted.clear() # "once" is per process: start each test afresh
logger = logging.getLogger(name)
handler = _Capture()
previous = logger.level
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
def restore():
logger.removeHandler(handler)
logger.setLevel(previous)
testcase.addCleanup(restore)
return handler.records
def _assert_one_debug_line(testcase, records):
"""Exactly one refusal line, at DEBUG and without a traceback."""
refusals = [r for r in records if "interpreter is exiting" in r.getMessage()]
testcase.assertEqual(len(refusals), 1, [r.getMessage() for r in records])
testcase.assertEqual(refusals[0].levelno, logging.DEBUG)
testcase.assertEqual([r for r in records if r.levelno > logging.DEBUG], [])
testcase.assertTrue(all(r.exc_info is None for r in records))
class TestScheduleRefusedAtShutdown(unittest.TestCase):
def setUp(self):
self.handler = image_monitor.ImageGenerationHandler(
mock.MagicMock(), mock.MagicMock()
)
self.addCleanup(self.handler.close)
def test_schedule_returns_false_and_leaves_nothing_pending(self):
records = _capture(self, "prompt_manager.image_monitor")
with mock.patch.object(image_monitor, "threading", _refusing_threading()):
self.assertFalse(self.handler.schedule("/out/a.png"))
self.assertFalse(self.handler.schedule("/out/b.png"))
self.assertEqual(self.handler.pending_count(), 0)
_assert_one_debug_line(self, records)
def test_another_start_failure_still_raises_and_leaves_nothing_pending(self):
class Failing(threading.Thread):
def start(self):
raise RuntimeError("can't start new thread")
namespace = types.SimpleNamespace(**vars(threading))
namespace.Thread = Failing
with mock.patch.object(image_monitor, "threading", namespace):
with self.assertRaises(RuntimeError):
self.handler.schedule("/out/a.png")
self.assertEqual(self.handler.pending_count(), 0)
def test_schedule_before_shutdown_is_unchanged(self):
self.handler.processing_delay = 3600
self.assertTrue(self.handler.schedule("/out/a.png"))
self.assertEqual(self.handler.pending_count(), 1)
class TestPreviewQueueRefusedAtShutdown(unittest.TestCase):
def thumbnailer(self, logger):
made = thumbnail_auto.AutoThumbnailer(
lambda *a, **k: {"action": "generated"}, lambda: True, logger
)
self.addCleanup(made.close)
return made
def test_enqueue_is_refused_quietly_and_drops_the_file(self):
logger = logging.getLogger("prompt_manager.test_exit_window")
records = _capture(self, logger.name)
thumbnailer = self.thumbnailer(logger)
src = Path("/out/a.png")
with mock.patch.object(thumbnail_auto, "threading", _refusing_threading()):
self.assertFalse(thumbnailer.enqueue(src, Path("/out/t/a.webp"), False))
self.assertFalse(thumbnailer.enqueue(src, Path("/out/t/b.webp"), False))
self.assertFalse(thumbnailer._worker_running)
self.assertEqual(len(thumbnailer._queue), 0)
self.assertEqual(thumbnailer._queued, set())
self.assertTrue(thumbnailer.wait_idle(0))
_assert_one_debug_line(self, records)
class TestMapParallelRefusedAtShutdown(unittest.TestCase):
def test_a_shut_pool_runs_the_items_on_the_calling_thread(self):
records = _capture(self, "prompt_manager.parallel")
with mock.patch.object(parallel, "ThreadPoolExecutor", _ShutPool):
self.assertEqual(
parallel.map_parallel(lambda n: n * n, [1, 2, 3], 4), [1, 4, 9]
)
_assert_one_debug_line(self, records)
def test_another_pool_error_still_raises(self):
class Broken(ThreadPoolExecutor):
def submit(self, *args, **kwargs):
raise RuntimeError("something else")
with mock.patch.object(parallel, "ThreadPoolExecutor", Broken):
with self.assertRaises(RuntimeError):
parallel.map_parallel(lambda n: n, [1, 2], 2)
CHILD = textwrap.dedent("""
import threading
# Registered before concurrent.futures is imported, so it runs after the
# pool's own exit hook: from here on, a pool refuses new work.
exiting = threading.Event()
threading._register_atexit(exiting.set)
import json
import logging
import os
import sys
from pathlib import Path
root, tmp = sys.argv[1], sys.argv[2]
sys.path.insert(0, root)
from py.api.thumbnail_auto import AutoThumbnailer
logging.basicConfig(level=logging.WARNING, stream=sys.stderr)
logger = logging.getLogger("prompt_manager.exit_child")
def generate(src, dst, is_video, *, stop, guard, track):
Path(dst).write_bytes(b"preview")
return {"action": "generated"}
thumbnailer = AutoThumbnailer(
generate, lambda: True, logger, workers=lambda: 4
)
queued = threading.Event()
def late_monitor():
exiting.wait()
# One batch of two: the worker maps it on a pool.
with thumbnailer._changed:
for name in ("a", "b"):
src = Path(tmp, name + ".png")
src.write_bytes(b"x")
thumbnailer.enqueue(src, Path(tmp, name + "_thumb.webp"), False)
queued.set()
def holder():
# Non-daemon: the interpreter waits for it, inside the exit window.
exiting.wait()
queued.wait(30)
idle = thumbnailer.wait_idle(30)
written = sorted(
n for n in os.listdir(tmp) if n.endswith("_thumb.webp")
)
print("PM_EXIT_REPORT " + json.dumps({"idle": idle, "written": written}))
sys.stdout.flush()
threading.Thread(target=late_monitor, daemon=True).start()
threading.Thread(target=holder).start()
# Return normally, as ComfyUI does after Ctrl+C.
""")
class TestExitWindowChild(unittest.TestCase):
def test_previews_queued_in_the_exit_window_log_no_traceback(self):
with tempfile.TemporaryDirectory() as tmp:
script = os.path.join(tmp, "child.py")
with open(script, "w", encoding="utf-8") as handle:
handle.write(CHILD)
env = dict(os.environ)
env.pop("PROMPT_MANAGER_CONFIG_PATH", None)
env["PROMPT_MANAGER_DATA_DIR"] = tmp
env["PYTHONIOENCODING"] = "utf-8"
try:
completed = subprocess.run(
[sys.executable, script, ROOT, tmp],
cwd=ROOT,
env=env,
capture_output=True,
timeout=CHILD_CEILING_SECONDS,
encoding="utf-8",
errors="replace",
)
except subprocess.TimeoutExpired as hung:
self.fail(
f"the child did not exit within {CHILD_CEILING_SECONDS} s\n"
f"stdout:\n{hung.stdout}\nstderr:\n{hung.stderr}"
)
detail = f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
self.assertEqual(completed.returncode, 0, detail)
reports = [
line[len(REPORT_PREFIX) :]
for line in completed.stdout.splitlines()
if line.startswith(REPORT_PREFIX)
]
self.assertEqual(len(reports), 1, detail)
report = json.loads(reports[0])
self.assertTrue(report["idle"], detail)
# The batch still ran: on the worker's own thread.
self.assertEqual(report["written"], ["a_thumb.webp", "b_thumb.webp"], detail)
self.assertNotIn("Traceback", completed.stderr, detail)
if __name__ == "__main__":
unittest.main()
+136
View File
@@ -0,0 +1,136 @@
"""The legacy pages' backend contract does not move under System S13 (#330).
S13 points the legacy admin and gallery buttons at native System and removes
their duplicate dialogs, changing no backend file. This snapshot pins what the
legacy pages talk to: every registered route (method and path) and the key
paths of ``GET /prompt_manager/settings``, the one route whose legacy request
body S13 shrinks. Record again only with a reason in the PR body:
PM_RECORD_LEGACY_SNAPSHOT=1 python3 -m unittest tests.test_legacy_contract_snapshot
Cutover (#334) removes this test and its fixture.
"""
import json
import os
import unittest
from pathlib import Path
try:
from tests.test_api_admin import AdminAPITestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase
from py.config import GalleryConfig # noqa: E402
SNAPSHOT = (
Path(__file__).resolve().parent / "fixtures" / "legacy_contract_snapshot.json"
)
RECORD_ENV = "PM_RECORD_LEGACY_SNAPSHOT"
SETTINGS = "/prompt_manager/settings"
DEPTH = 2
CHANGED = (
"the legacy contract changed; if the change is intended (a new route, a new"
" settings key), record it with PM_RECORD_LEGACY_SNAPSHOT=1 and say why in"
" the PR body"
)
def key_paths(value, prefix="", depth=1):
"""The key paths of a JSON object to ``DEPTH`` levels.
``{"a": {"b": 1}}`` gives ``a`` and ``a.b``; the items of a list of
objects give the union of their keys as ``a[].b``.
"""
paths = set()
for key, child in value.items():
path = f"{prefix}{key}"
paths.add(path)
if depth >= DEPTH:
continue
if isinstance(child, dict):
paths |= key_paths(child, f"{path}.", depth + 1)
elif isinstance(child, list):
for item in child:
if isinstance(item, dict):
paths |= key_paths(item, f"{path}[].", depth + 1)
return paths
class TestLegacyContractSnapshot(AdminAPITestCase):
"""Routes and settings keys against tests/fixtures/legacy_contract_snapshot.json.
The base case isolates config.json (``PROMPT_MANAGER_CONFIG_PATH`` in the
temp tree), stubs ComfyUI's ``folder_paths`` and resets the gallery roots;
nothing here reads the maintainer's install.
"""
async def asyncSetUp(self):
await super().asyncSetUp()
self.api._get_all_output_dirs = lambda: [str(self.output_dir)]
GalleryConfig.MONITORING_DIRECTORIES = [str(self.output_dir)]
def _routes(self):
return sorted(
{
f"{route.method} {route.resource.canonical}"
for route in self.app.router.routes()
if route.method != "HEAD"
}
)
async def _settings_keys(self):
resp = await self.client.request("GET", SETTINGS)
body = await resp.json()
self.assertEqual(resp.status, 200, body)
return sorted(key_paths(body))
def _snapshot(self):
with open(SNAPSHOT, encoding="utf-8") as f:
return json.load(f)
def _record(self, key, value):
recorded = self._snapshot() if SNAPSHOT.exists() else {}
recorded[key] = value
with open(SNAPSHOT, "w", encoding="utf-8", newline="\n") as f:
json.dump(recorded, f, indent=2)
f.write("\n")
async def test_routes_match_the_snapshot(self):
routes = self._routes()
if os.environ.get(RECORD_ENV) == "1":
self._record("routes", routes)
self.assertGreater(len(routes), 50)
self.assertEqual(routes, self._snapshot()["routes"], CHANGED)
async def test_settings_response_keeps_its_keys(self):
keys = await self._settings_keys()
if os.environ.get(RECORD_ENV) == "1":
self._record("settings_keys", keys)
self.assertIn("settings.infinite_scroll", keys)
self.assertEqual(keys, self._snapshot()["settings_keys"], CHANGED)
class TestKeyPaths(unittest.TestCase):
def test_depth_and_list_items(self):
value = {
"success": True,
"settings": {"a": 1, "deep": {"hidden": 1}},
"rows": [{"path": "x"}, {"label": "y"}],
}
self.assertEqual(
sorted(key_paths(value)),
[
"rows",
"rows[].label",
"rows[].path",
"settings",
"settings.a",
"settings.deep",
"success",
],
)
if __name__ == "__main__":
unittest.main()
+233
View File
@@ -0,0 +1,233 @@
"""A restore cut off by a server stop waits for its worker thread (#360).
A cancelled handler (or job) task cannot stop the executor thread that
stages or publishes. Its cleanup (the stage, the upload) and the registry's
``restore`` admission must outlive that thread, and the live library ends as
the old or the restored one, never partial, with the safety backup kept.
Every positive wait is on an event. The one bounded wait (``SETTLE``) is a
negative check: the cancelled task must still be pending while its worker is
held.
"""
import asyncio
import os
import shutil
import sqlite3
import threading
from unittest.mock import patch
from database import restore_stage
from database.models import PromptModel
from py.api import admin as admin_mod
from py.api.jobs import RESTORE_JOB_KIND
try:
from tests.test_restore_plans import RestoreTestCase, _texts
except ImportError: # discovered with tests/ as the top-level directory
from test_restore_plans import RestoreTestCase, _texts
WAIT = 60 # a ceiling only a hung test reaches
SETTLE = 0.3 # the negative check's bound: the old code's cleanup takes ms
class _Gate:
"""Hold one blocking call on its worker thread until released."""
def __init__(self, real):
self.real = real
self.entered = threading.Event()
self.release = threading.Event()
self.done = threading.Event()
def method(self):
"""A function to patch onto the class (bound like the real one)."""
def held(*args):
return self(*args)
return held
def __call__(self, *args):
self.entered.set()
self.release.wait(WAIT)
try:
return self.real(*args)
finally:
self.done.set()
class ShutdownRestoreCase(RestoreTestCase):
def gate(self, real):
gate = _Gate(real)
# A failed check never leaves the worker held until WAIT.
self._gates = getattr(self, "_gates", []) + [gate]
self.addCleanup(gate.release.set)
return gate
def stages(self):
"""The stage folders (the upload's prefix starts the same way)."""
return [
name
for name in super().stages()
if os.path.isdir(os.path.join(self.tmpdir, name))
]
def uploads(self):
"""The upload files (not the -wal/-shm its readers may add)."""
return sorted(
name
for name in os.listdir(self.tmpdir)
if name.startswith(restore_stage.UPLOAD_PREFIX) and name.endswith(".tmp")
)
def fake_upload(self, texts):
"""Make ``_receive_restore_upload`` hand over a prepared upload."""
self.upload(texts, "source.db")
source = os.path.join(self.tmpdir, "uploads", "source.db")
target = os.path.join(self.tmpdir, restore_stage.UPLOAD_PREFIX + "x.tmp")
shutil.copyfile(source, target)
async def receive(request, uploads):
uploads.append(target)
return target, None
return patch.object(self.api, "_receive_restore_upload", receive)
async def wait_thread(self, event):
loop = asyncio.get_running_loop()
self.assertTrue(await loop.run_in_executor(None, event.wait, WAIT))
async def assert_still_pending(self, task):
done, _ = await asyncio.wait({task}, timeout=SETTLE)
if done:
for gate in getattr(self, "_gates", ()):
gate.release.set() # let the worker end before teardown
self.fail("the cancelled task ended while its worker still ran")
class TestLegacyRestoreAtShutdown(ShutdownRestoreCase):
async def test_cancel_during_publish_waits_for_the_worker(self):
self.save("old")
old_db = self.api.db
gate = self.gate(PromptModel.publish_staged)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "publish_staged", gate.method()),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(gate.entered)
self.assertEqual(len(self.stages()), 1)
task.cancel()
await self.assert_still_pending(task)
# The worker still has its stage and upload, and runs alone.
self.assertEqual(len(self.stages()), 1)
self.assertEqual(len(self.uploads()), 1)
self.assertIsNotNone(self.api._jobs.active(RESTORE_JOB_KIND))
gate.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertTrue(gate.done.is_set())
self.assertEqual(_texts(self.db_path), ["restored"])
self.assertIsNot(self.api.db, old_db) # the restored library adopted
self.assertEqual(len(self.backups()), 1) # the safety backup is kept
self.assertEqual(self.stages(), [])
self.assertEqual(self.uploads(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
async def test_cancel_during_staging_waits_and_changes_nothing(self):
self.save("old")
gate = self.gate(PromptModel.stage_restore)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "stage_restore", gate.method()),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(gate.entered)
task.cancel()
await self.assert_still_pending(task)
self.assertEqual(len(self.uploads()), 1) # still being read
self.assertIsNotNone(self.api._jobs.active(RESTORE_JOB_KIND))
gate.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertEqual(_texts(self.db_path), ["old"])
self.assertEqual(self.backups(), [])
self.assertEqual(self.stages(), []) # the finished stage is removed
self.assertEqual(self.uploads(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
async def test_a_publish_that_fails_after_the_cancel_still_cancels(self):
self.save("old")
def refuse(model, staged):
raise sqlite3.OperationalError("refused")
gate = self.gate(refuse)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "publish_staged", gate.method()),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(gate.entered)
task.cancel()
await self.assert_still_pending(task)
gate.release.set()
# Not answered as a 500: the cancel goes on.
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertEqual(_texts(self.db_path), ["old"])
self.assertEqual(self.stages(), [])
self.assertEqual(self.uploads(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
async def test_a_second_cancel_still_adopts_the_restored_library(self):
self.save("old")
old_db = self.api.db
publish = self.gate(PromptModel.publish_staged)
prune = self.gate(admin_mod._prune_safety_backups)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "publish_staged", publish.method()),
patch.object(admin_mod, "_prune_safety_backups", prune),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(publish.entered)
task.cancel()
await self.assert_still_pending(task)
publish.release.set()
await self.wait_thread(prune.entered) # adopting the library
task.cancel() # the loop's own shutdown cancels again
await self.assert_still_pending(task)
prune.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertEqual(_texts(self.db_path), ["restored"])
self.assertIsNot(self.api.db, old_db)
self.assertEqual(len(self.backups()), 1)
self.assertEqual(self.stages(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
class TestReviewedRestoreAtShutdown(ShutdownRestoreCase):
async def test_cancelled_job_waits_for_the_publish(self):
self.save("old")
old_db = self.api.db
body = await self.preview(self.upload(["restored"]))
gate = self.gate(PromptModel.publish_staged)
with patch.object(PromptModel, "publish_staged", gate.method()):
started = await self.apply(body["token"])
job = self.api._jobs.get(started["job"]["id"])
await self.wait_thread(gate.entered)
job.task.cancel()
await self.assert_still_pending(job.task)
self.assertEqual(len(self.stages()), 1)
self.assertTrue(job.running) # its exclusive admission holds
gate.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(job.task, WAIT)
self.assertEqual(_texts(self.db_path), ["restored"])
self.assertIsNot(self.api.db, old_db)
self.assertEqual(len(self.backups()), 1)
self.assertEqual(self.stages(), [])
self.assertFalse(job.running)
+27 -17
View File
@@ -298,19 +298,21 @@ class TestQwenEditBatchIsQueued(InspectionTestCase):
class TestDefaultProfileStillDecides(InspectionTestCase):
async def test_ambiguous_file_with_texts_keeps_the_default_profile(self):
"""A default-ambiguous file with a choice never runs the inspection
profile, even when that profile would offer more candidates."""
"""A default-ambiguous file with a choice is recorded from the
default profile, even when the inspection profile would offer more
candidates. That profile only checks the saver anchor (#390): no
saver names this file, so it anchors nothing."""
self._roots()
graph = {
**_prefixed(qwen_edit_graph("make it night"), "q"),
**unlinkable_graph("cat", "dog"),
}
self._write(graph=graph)
self._write("unsaved_00001_.png", graph=graph)
with patch.object(
review_capture,
"resolve_inspection",
side_effect=AssertionError("inspection profile ran"),
"inspection_choice",
side_effect=AssertionError("inspection choices were asked"),
):
await self._scan()
@@ -321,17 +323,24 @@ class TestDefaultProfileStillDecides(InspectionTestCase):
self.assertEqual(item["resolver_version"], review_capture.RESOLVER_VERSION)
self.assertEqual(_texts(evidence), ["cat", "dog"])
async def test_apply_of_a_default_item_never_runs_the_inspection_profile(self):
async def test_apply_of_a_default_item_inspects_only_for_the_anchor(
self,
):
self._roots()
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
await self._scan()
with patch.object(
review_capture,
"resolve_inspection",
side_effect=AssertionError("inspection profile ran"),
with (
_counted(review_capture, "resolve_inspection") as calls,
patch.object(
review_capture,
"inspection_choice",
side_effect=AssertionError("inspection choices were asked"),
),
):
_item, (status, data) = await self._apply("cat")
self.assertEqual(status, 200, data)
# The saver anchor check (#390): with the file's name, nothing matches.
self.assertEqual([c[1] for c in calls], ["two.png"])
self.assertEqual(self._linked_text(self.output_dir / "two.png"), "cat")
@@ -369,15 +378,16 @@ class TestApplyInspectionCandidate(InspectionTestCase):
calls = []
real = review_capture.resolve_inspection
def spy(metadata):
calls.append(metadata)
return real(metadata)
def spy(metadata, file_name=None):
calls.append((metadata, file_name))
return real(metadata, file_name)
with patch.object(review_capture, "resolve_inspection", side_effect=spy):
_item, (status, data) = await self._apply("make it night")
self.assertEqual(status, 200, data)
self.assertEqual(len(calls), 1)
self.assertEqual(calls[0][1], NAME) # anchored on its saver (#390)
async def test_table_change_keeps_the_item_applicable(self):
"""A table change for a class the graph uses moves the item's
@@ -853,8 +863,8 @@ class TestInspectionResolverVersion(InspectionTestCase):
version = _version()
real = review_capture.resolve_inspection
def newer(metadata):
resolution = real(metadata)
def newer(metadata, file_name=None):
resolution = real(metadata, file_name)
return {
**resolution,
"candidates": [
@@ -899,10 +909,10 @@ class TestScanContainment(InspectionTestCase):
self._write("b.png", graph=qwen_edit_graph("snow", "rain"))
real = review_capture.resolve_inspection
def fails_for_a(metadata):
def fails_for_a(metadata, file_name=None):
if "make it night" in json.dumps(metadata):
raise RuntimeError("inspection blew up")
return real(metadata)
return real(metadata, file_name)
with (
patch.object(review_capture, "resolve_inspection", side_effect=fails_for_a),
+333
View File
@@ -0,0 +1,333 @@
"""The scan and Needs review anchor a file on the saver that wrote it (#390).
The workspace inspector (since #383) passes the file's name to the resolver,
which anchors the file on the one saver whose ``filename_prefix`` names it
(``_GraphResolver.saver_stages``). The scan and Needs review use the same
rule. Maintainer ruling: a file never falls through both the link and the
queue, so when only that anchor resolves a file the default profile and the
parser floor do not link, it is queued with the anchor's pair as its one
choice (never linked automatically). No match, several matches or an
unfinished walk anchor nothing, so such a file is queued as before. Apply
re-resolves with the file's name, so a choice the anchor contradicts is
refused as stale evidence.
"""
import json
from unittest.mock import patch
try:
from tests.test_review import unlinkable_graph, write_png
from tests.test_review_inspection import (
PROMPTS,
_counted,
InspectionTestCase,
qwen_edit_graph,
)
except ImportError: # discovered with tests/ as the top-level directory
from test_review import unlinkable_graph, write_png
from test_review_inspection import (
PROMPTS,
_counted,
InspectionTestCase,
qwen_edit_graph,
)
from utils import review_capture
def branch_graph(*prompts, prefixes=None):
"""A Qwen-edit batch whose every SaveImage has its own literal prefix."""
graph = qwen_edit_graph(*prompts)
savers = [n for n in graph.values() if n["class_type"] == "SaveImage"]
names = prefixes or [f"branch_{index}" for index in range(len(savers))]
for saver, prefix in zip(savers, names):
saver["inputs"]["filename_prefix"] = prefix
return graph
def saved_unlinkable_graph(*positives):
"""``unlinkable_graph`` whose branches save under their own prefix.
The default profile finds it ambiguous with a choice and the scan's
browser-parser floor links nothing, so it is queued from the default
profile unless the saver that wrote the file anchors its branch.
"""
graph = unlinkable_graph(*positives)
samplers = [
node_id
for node_id, node in graph.items()
if node["class_type"] == "KSampler" and node_id != "3"
]
for index, sampler in enumerate(samplers):
graph[f"{sampler}1"] = {
"class_type": "VAEDecode",
"inputs": {"samples": [sampler, 0], "vae": ["1", 2]},
}
graph[f"{sampler}2"] = {
"class_type": "SaveImage",
"inputs": {
"images": [f"{sampler}1", 0],
"filename_prefix": f"side_{index}",
},
}
return graph
def _without_name(real):
"""``resolve_inspection`` as it was before #390: the file name ignored."""
def unnamed(metadata, file_name=None):
return real(metadata)
return unnamed
def _only_candidate(test, item):
"""The positive text of an anchored item's one candidate."""
evidence = json.loads(item["evidence"])
test.assertEqual(evidence.get("anchor"), "saver")
test.assertEqual(evidence.get("profile"), "inspection")
(candidate,) = evidence["candidates"]
return candidate["positive"]["text"]
class TestInspectionItemsAnchor(InspectionTestCase):
async def test_file_its_saver_names_is_queued_with_the_anchor_choice(self):
self._roots()
path = self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
job = await self._scan()
(item,) = self._items()
self.assertEqual(_only_candidate(self, item), "add a red hat")
self.assertEqual(item["candidate_count"], 1)
self.assertEqual(
item["resolver_version"],
review_capture.inspection_review_version(
{"prompt": branch_graph(*PROMPTS)}
),
)
(row,) = job["outcomes"]["review"]
self.assertEqual(row["path"], "branch_1_00001_.png")
# Queued, never linked: ingestion stays on the default profile.
self.assertIsNone(self._linked_text(path))
async def test_anchored_item_stays_on_rescan(self):
self._roots()
self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
await self._scan()
(item,) = self._items()
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"]["review_untraced"], 0)
async def test_file_no_saver_names_stays_queued(self):
self._roots()
self._write("elsewhere_00001_.png", graph=branch_graph(*PROMPTS))
await self._scan()
(item,) = self._items()
self.assertEqual(item["candidate_count"], len(PROMPTS))
self.assertNotIn("anchor", json.loads(item["evidence"]))
async def test_file_several_savers_name_stays_queued(self):
self._roots()
graph = branch_graph(*PROMPTS, prefixes=["same", "same", "other"])
self._write("same_00001_.png", graph=graph)
await self._scan()
(item,) = self._items()
self.assertEqual(item["candidate_count"], len(PROMPTS))
async def test_old_rule_item_is_replaced_by_the_anchor_choice(self):
self._roots()
self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
real = review_capture.resolve_inspection
with (
patch.object(review_capture, "REVIEW_RULE_VERSION", 0),
patch.object(
review_capture, "resolve_inspection", side_effect=_without_name(real)
),
):
await self._scan()
(old,) = self._items() # queued under the old rule with every branch
self.assertEqual(old["candidate_count"], len(PROMPTS))
await self._scan("all")
(item,) = self._items()
self.assertNotEqual(item["id"], old["id"])
self.assertEqual(_only_candidate(self, item), "add a red hat")
async def test_apply_links_the_anchor_choice(self):
self._require_routes()
self._roots()
path = self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
await self._scan()
_item, (status, data) = await self._apply("add a red hat")
self.assertEqual(status, 200, data)
self.assertEqual(self._linked_text(path), "add a red hat")
async def test_apply_refuses_an_old_choice_the_anchor_contradicts(self):
self._require_routes()
self._roots()
path = self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
real = review_capture.resolve_inspection
with (
patch.object(review_capture, "REVIEW_RULE_VERSION", 0),
patch.object(
review_capture, "resolve_inspection", side_effect=_without_name(real)
),
):
await self._scan()
item, detail, observed = await self._open()
# branch_1 is the saver of "add a red hat"; choose another branch.
wrong = self._candidate(detail, "make it night")
status, data = await self._decide(
item["id"], self._body(detail, observed, wrong["hash"])
)
self.assertEqual(status, 409, data)
self.assertEqual(data["code"], "review-conflict", data)
self.assertIsNone(self._linked_text(path))
class TestDefaultItemsAnchor(InspectionTestCase):
async def test_anchor_resolves_and_nothing_links_so_it_is_queued(self):
"""Maintainer ruling: never unlinked and unqueued. The anchor resolves
the file, the default profile and the parser floor link nothing, so
it is queued with the anchor's pair as its choice."""
self._roots()
path = self.output_dir / "side_0_00001_.png"
graph = saved_unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self.assertEqual(
review_capture.resolve_prompt_pair({"prompt": graph})["state"], "ambiguous"
)
job = await self._scan()
(item,) = self._items()
self.assertEqual(_only_candidate(self, item), "cat")
(row,) = job["outcomes"]["review"]
self.assertIn("more than one prompt could match", row["detail"])
self.assertIsNone(self._linked_text(path))
async def test_default_ambiguous_file_no_saver_names_stays_queued(self):
self._roots()
write_png(
self.output_dir / "unsaved_00001_.png",
prompt=saved_unlinkable_graph("cat", "dog"),
)
await self._scan()
(item,) = self._items()
evidence = json.loads(item["evidence"])
self.assertNotIn("profile", evidence)
self.assertNotIn("anchor", evidence)
self.assertEqual(item["resolver_version"], review_capture.RESOLVER_VERSION)
async def test_file_the_parser_floor_links_is_not_queued(self):
"""Already linked by the default path: unchanged by the anchor."""
self._roots()
graph = saved_unlinkable_graph("cat", "dog")
del graph["3"] # the parser floor now reads a real sampler
path = self.output_dir / "side_0_00001_.png"
write_png(path, prompt=graph)
await self._scan()
self.assertEqual(self._items(), [])
self.assertIsNotNone(self._linked_text(path))
async def test_apply_refuses_an_old_default_choice_the_anchor_contradicts(self):
self._require_routes()
self._roots()
path = self.output_dir / "side_0_00001_.png"
write_png(path, prompt=saved_unlinkable_graph("cat", "dog"))
with patch.object(review_capture, "_captured_inspection", return_value=None):
await self._scan() # queued from the default profile, as before #390
item, detail, observed = await self._open()
# side_0 is the saver of "cat"; choose the other branch.
wrong = self._candidate(detail, "dog")
status, data = await self._decide(
item["id"], self._body(detail, observed, wrong["hash"])
)
self.assertEqual(status, 409, data)
self.assertEqual(data["code"], "review-conflict", data)
self.assertIsNone(self._linked_text(path))
async def test_old_default_item_is_replaced_by_the_anchor_choice(self):
self._roots()
write_png(
self.output_dir / "side_0_00001_.png",
prompt=saved_unlinkable_graph("cat", "dog"),
)
with patch.object(review_capture, "_captured_inspection", return_value=None):
await self._scan() # queued from the default profile, as before #390
(old,) = self._items()
self.assertEqual(old["resolver_version"], review_capture.RESOLVER_VERSION)
await self._scan("all")
(item,) = self._items()
self.assertNotEqual(item["id"], old["id"])
self.assertEqual(_only_candidate(self, item), "cat")
with _counted(review_capture, "resolve_inspection") as calls:
await self._scan("all") # now queued under its version: no re-read
self.assertEqual(calls, [])
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
async def test_decided_anchored_item_is_kept(self):
self._require_routes()
self._roots()
write_png(
self.output_dir / "side_0_00001_.png",
prompt=saved_unlinkable_graph("cat", "dog"),
)
await self._scan()
item, detail, observed = await self._open()
status, data = await self._decide(
item["id"], self._body(detail, observed, "unresolved")
)
self.assertEqual(status, 200, data)
before = (self._items(), self._decisions())
await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
async def test_apply_accepts_an_old_default_choice_the_anchor_names(self):
self._require_routes()
self._roots()
path = self.output_dir / "side_0_00001_.png"
write_png(path, prompt=saved_unlinkable_graph("cat", "dog"))
with patch.object(review_capture, "_captured_inspection", return_value=None):
await self._scan()
_item, (status, data) = await self._apply("cat")
self.assertEqual(status, 200, data)
self.assertEqual(self._linked_text(path), "cat")
class TestSaverAnchored(InspectionTestCase):
def test_only_the_anchor_counts(self):
anchored = {"prompt": saved_unlinkable_graph("cat", "dog")}
self.assertTrue(review_capture.saver_anchored(anchored, "side_1_00001_.png"))
self.assertFalse(review_capture.saver_anchored(anchored, "nope_00001_.png"))
self.assertFalse(review_capture.saver_anchored(anchored, None))
# One sampler resolves without any name: the anchor decides nothing.
single = {"prompt": branch_graph("make it night")}
self.assertFalse(review_capture.saver_anchored(single, "branch_0_00001_.png"))
+21
View File
@@ -978,6 +978,27 @@ class TestThumbnailOnCapture(AutoThumbnailCase):
(self.output_dir / "thumbnails" / f"{name}_thumb.webp").is_file()
)
async def test_a_worker_refused_at_interpreter_exit_skips_quietly(self):
"""#375: refused once the interpreter exits; nothing stays queued."""
class Refused(threading.Thread):
def start(self):
raise RuntimeError("can't create new thread at interpreter shutdown")
namespace = type(sys)("threading_copy")
namespace.__dict__.update(vars(threading))
namespace.Thread = Refused
thumbnailer = self.api._auto_thumbnails()
path = make_png(self.output_dir / "late.png", size=(32, 32))
with (
patch.object(auto, "threading", namespace),
self.assertNoLogs("prompt_manager.api", "INFO"),
):
self.assertEqual(self.api.queue_capture_thumbnail(str(path)), "skipped")
self.assertFalse(thumbnailer._worker_running)
self.assertEqual(len(thumbnailer._queue), 0)
self.assertTrue(thumbnailer.wait_idle(0))
def test_queue_is_bounded(self):
thumbnailer = auto.AutoThumbnailer(
lambda *a: {"action": "generated"},
+213 -1
View File
@@ -709,7 +709,8 @@ class TestRotatedCopyRule(IsolatedLoggerTestCase):
self.assertNotEqual(link["size"], os.path.getsize(target))
for row in rows.values():
self.assertEqual(
set(row), {"filename", "path", "size", "modified", "is_main", "kind"}
set(row),
{"filename", "path", "size", "modified", "is_main", "kind", "identity"},
)
def test_an_unreadable_timestamp_does_not_break_the_scan(self):
@@ -763,6 +764,217 @@ class TestRotatedCopyRule(IsolatedLoggerTestCase):
self.manager.log_file_entry(name)
class TestClearOnlyWhatWasConfirmed(IsolatedLoggerTestCase):
"""#389: Clear all logs empties or deletes no more than its confirmation showed."""
def setUp(self):
super().setUp()
self.logger.info("active log line") # the active log exists
self.outside = os.path.join(self.tmp.name, "outside")
os.makedirs(self.outside)
def _path(self, name):
return os.path.join(self.log_dir, name)
def _write(self, name, data=b"rotated\n"):
with open(self._path(name), "wb") as handle:
handle.write(data)
def _read(self, path):
with open(path, "rb") as handle:
return handle.read()
def _snapshot(self):
return {name: self._read(self._path(name)) for name in os.listdir(self.log_dir)}
def _assert_unchanged(self, before, growing="prompt_manager.log"):
"""Nothing deleted or emptied (the refusal itself logs to ``growing``)."""
after = self._snapshot()
self.assertEqual(sorted(after), sorted(before))
for name, data in before.items():
if name == growing:
self.assertTrue(after[name].startswith(data), name)
else:
self.assertEqual(after[name], data, name)
def _confirmed(self):
"""What the workspace dialog showed: the listed active and rotated rows."""
return [
{"name": row["filename"], "identity": row["identity"]}
for row in self.manager.get_log_files()
if row["kind"] in ("active", "rotated")
]
def _file_handler(self):
return next(
h
for h in self.manager.logger.handlers
if isinstance(h, logging.FileHandler)
)
def test_listed_identity_is_a_string_that_survives_json_numbers(self):
# mtime_ns (~1.7e18) and many inodes exceed 2**53: as JSON numbers a
# browser would round them and every confirmed clear would be refused.
self._write("prompt_manager.log.1")
for row in self.manager.get_log_files():
with self.subTest(name=row["filename"]):
self.assertIsInstance(row["identity"], str)
self.assertRegex(row["identity"], r"^\d+:-?\d+:\d+$")
def test_a_copy_dated_before_1970_can_be_confirmed(self):
self._write("prompt_manager.log.1")
try:
os.utime(self._path("prompt_manager.log.1"), ns=(0, -(10**9)))
except (OSError, OverflowError, ValueError) as error:
self.skipTest(f"this filesystem cannot store a time before 1970: {error}")
confirmed = self._confirmed()
if not any(":-" in row["identity"] for row in confirmed):
self.skipTest("the filesystem did not keep the negative time")
result = self.manager.truncate_logs(confirmed)
self.assertIn("prompt_manager.log.1", result["truncated"])
def test_a_rollover_after_the_confirmation_deletes_nothing(self):
self._write("prompt_manager.log.1", b"one" * 10)
self._write("prompt_manager.log.2", b"two" * 10)
confirmed = self._confirmed()
self.logger.info("written after the confirmation")
self._file_handler().doRollover() # .2 -> .3, .1 -> .2, log -> .1
before = self._snapshot()
with self.assertRaises(logging_config.ClearRefused) as caught:
self.manager.truncate_logs(confirmed)
self.assertEqual(caught.exception.code, "logs-changed")
self._assert_unchanged(before)
def test_a_copy_that_appeared_after_the_confirmation_is_kept(self):
self._write("prompt_manager.log.1", b"one")
confirmed = self._confirmed()
self._write("prompt_manager.log.5", b"not shown")
result = self.manager.truncate_logs(confirmed)
self.assertEqual(result["errors"], [])
self.assertEqual(
sorted(result["truncated"]), ["prompt_manager.log", "prompt_manager.log.1"]
)
self.assertEqual(
sorted(os.listdir(self.log_dir)),
["prompt_manager.log", "prompt_manager.log.5"],
)
def _clear(self, confirmed=None):
"""Run a clear and return what it raised (None), so the files are
asserted first: on the old code the test fails on lost bytes."""
try:
if confirmed is None:
self.manager.truncate_logs() # the legacy page's call
else:
self.manager.truncate_logs(confirmed)
except Exception as error: # noqa: BLE001 - asserted below
return error
return None
def test_a_foreign_append_to_the_active_log_is_refused(self):
# A second ComfyUI sharing the folder writes the same active log.
self._write("prompt_manager.log.1")
listings = (None, "listed")
for listing in listings:
with self.subTest(listed=listing is not None):
confirmed = self._confirmed() if listing else None
with open(self._path("prompt_manager.log"), "ab") as other:
other.write(b"line from another process\n")
before = self._snapshot()
error = self._clear(confirmed)
self._assert_unchanged(before)
self.assertEqual(getattr(error, "code", None), "logs-foreign-writer")
self.logger.info("our own line") # our stream catches up again
@unittest.skipIf(os.name == "nt", "Windows cannot rename a log that is open")
def test_a_foreign_rotation_of_our_log_is_refused(self):
# Another process rolled the shared log over: our handler still writes
# the file now named .1, and a new active log is the other's.
os.rename(self._path("prompt_manager.log"), self._path("prompt_manager.log.1"))
self._write("prompt_manager.log", b"the other process's log\n")
before = self._snapshot()
with self.assertRaises(logging_config.ClearRefused) as caught:
self.manager.truncate_logs()
self.assertEqual(caught.exception.code, "logs-foreign-writer")
# Our handler still writes the renamed file, so the refusal lands there.
self._assert_unchanged(before, growing="prompt_manager.log.1")
def test_a_copy_that_changes_mid_clear_stops_the_clear(self):
# The identity is checked again just before each unlink: another
# process rotating while we delete stops the clear there.
for n in (1, 2):
self._write(f"prompt_manager.log.{n}", b"r" * n)
confirmed = self._confirmed()
real_unlink = os.unlink
calls = []
def unlink_then_rotate(path, *args, **kwargs):
real_unlink(path, *args, **kwargs)
if not calls:
with open(self._path("prompt_manager.log.2"), "ab") as handle:
handle.write(b"rotated in by another process")
calls.append(os.path.basename(path))
active_before = self._read(self._path("prompt_manager.log"))
with mock.patch.object(logging_config.os, "unlink", unlink_then_rotate):
result = self.manager.truncate_logs(confirmed)
self.assertEqual(calls, ["prompt_manager.log.1"])
self.assertEqual(result["truncated"], ["prompt_manager.log.1"])
self.assertEqual(len(result["errors"]), 1)
self.assertIn("prompt_manager.log.2", result["errors"][0])
self.assertTrue(os.path.exists(self._path("prompt_manager.log.2")))
# The active log is emptied last, so it is kept too.
self.assertTrue(
self._read(self._path("prompt_manager.log")).startswith(active_before)
)
def _require_hard_link(self, source, link):
try:
os.link(source, link)
except (OSError, NotImplementedError, AttributeError) as error:
self.skipTest(f"this filesystem cannot make hard links: {error}")
def test_a_hard_linked_active_log_is_never_emptied(self):
other_name = os.path.join(self.outside, "kept-by-the-user.log")
self._require_hard_link(self._path("prompt_manager.log"), other_name)
kept = self._read(other_name)
self.assertTrue(kept)
for listed in (False, True):
with self.subTest(listed=listed):
error = self._clear(self._confirmed() if listed else None)
self.assertTrue(self._read(other_name).startswith(kept))
self.assertEqual(getattr(error, "code", None), "logs-hard-link")
def test_a_hard_linked_active_log_with_file_logging_off_is_kept(self):
self.manager.update_config({"file_logging": False})
other_name = os.path.join(self.outside, "kept-by-the-user.log")
self._require_hard_link(self._path("prompt_manager.log"), other_name)
kept = self._read(other_name)
error = self._clear()
self.assertEqual(self._read(other_name), kept)
self.assertEqual(getattr(error, "code", None), "logs-hard-link")
def test_a_malformed_confirmation_is_refused_before_anything_changes(self):
self._write("prompt_manager.log.1")
before = self._snapshot()
for expected in (
"prompt_manager.log.1",
[{"name": "prompt_manager.log.1"}],
[{"name": "prompt_manager.log.1", "identity": 7}],
[{"name": "prompt_manager.log.1", "identity": "1:2"}],
[{"name": 1, "identity": "1:2:3"}],
[
{"name": "prompt_manager.log.1", "identity": "1:2:3"},
{"name": "prompt_manager.log.1", "identity": "1:2:3"},
],
["prompt_manager.log.1"],
):
with self.subTest(expected=expected):
with self.assertRaises(ValueError):
self.manager.truncate_logs(expected)
self._assert_unchanged(before)
class TestConfiguration(IsolatedLoggerTestCase):
def test_in_flight_record_cannot_reopen_retired_file_handler(self) -> None:
handler = next(
+58 -2
View File
@@ -1,12 +1,26 @@
"""The System browser hooks only ever reset their own server's config.json (#249 S3)."""
"""The System browser hooks only ever reset their own server's config.json (#249 S3)
and decide JPEG XL availability whatever the host has installed (#371)."""
import asyncio
import os
import sys
import tempfile
import types
import unittest
from pathlib import Path
from unittest import mock
from tests.browser.workspace_system_fixture import register_system_commands
from aiohttp import web
from aiohttp.test_utils import TestClient, TestServer
_mock_server = mock.MagicMock()
_mock_server.PromptServer.instance.routes = mock.MagicMock()
sys.modules.setdefault("server", _mock_server)
from py.api import thumbnail_encode # noqa: E402
from tests.browser.workspace_system_fixture import ( # noqa: E402
register_system_commands,
)
class TestWorkspaceSystemFixture(unittest.TestCase):
@@ -33,6 +47,48 @@ class TestWorkspaceSystemFixture(unittest.TestCase):
register_system_commands(None, None, None, self.root)
self.assertEqual(elsewhere.read_bytes(), b'{"real": true}')
def test_no_encoder_holds_even_when_the_host_has_one(self):
"""``{available: false}`` survives the settings read's re-probe (#371)."""
config = self.root / "config.json"
env = {**os.environ, "PROMPT_MANAGER_CONFIG_PATH": str(config)}
api = types.SimpleNamespace()
app, routes = web.Application(), web.RouteTableDef()
real_detect = thumbnail_encode._detect
with mock.patch.dict(os.environ, env, clear=True):
reset = register_system_commands(app, routes, api, self.root)
app.add_routes(routes)
self.addCleanup(thumbnail_encode.reset_encoders)
self.addCleanup(setattr, thumbnail_encode, "_detect", real_detect)
async def scenario():
async with TestClient(TestServer(app)) as client:
async def post(available):
response = await client.post(
"/__test__/system/jxl-encoder", json={"available": available}
)
self.assertEqual(response.status, 200)
await post(False)
status = thumbnail_encode.encoder_status()["jxl"]
self.assertFalse(status["available"])
self.assertIsNone(
thumbnail_encode.detect_jxl_encoder(recheck_missing=True)
)
await post(True)
status = thumbnail_encode.encoder_status()["jxl"]
self.assertEqual(status["encoder"], "cjxl")
await post(False)
# The host's real detector finds an encoder (Pillow can save JPEG XL).
with mock.patch.object(thumbnail_encode, "_pillow_can_save_jxl", lambda: True):
asyncio.run(scenario())
# Reset gives detection back to the host.
reset()
self.assertIs(thumbnail_encode._detect, real_detect)
status = thumbnail_encode.encoder_status()["jxl"]
self.assertEqual(status["encoder"], "pillow")
if __name__ == "__main__":
unittest.main()
+149 -25
View File
@@ -67,6 +67,45 @@ const browserJobIds = [...allJobs].filter(([, body]) => /npm run --prefix tools\
const mergeJob = () => allJobs.get('frontend-coverage-merge');
const verifyStep = () => steps(mergeJob()).find(step => /name: Verify every shard's raw coverage arrived/.test(step));
// Jobs that run only in the merge queue, on main and on demand (#420): skipped on pull_request.
const QUEUE_ONLY_IF = /^ {4}if: github\.event_name != 'pull_request'$/m;
const EVENTS = ['pull_request', 'merge_group', 'push', 'workflow_dispatch'];
function gateVariables(jobText) {
return new Map([...jobText.matchAll(/^ {10}([A-Z_]+): \$\{\{ needs\.([\w-]+)\.result \}\}$/gm)]
.map(match => [match[2], match[1]]));
}
function runGate(jobText, env) {
const script = runScript(steps(jobText).find(step => /run: \|/.test(step)));
try {
execFileSync('bash', ['--noprofile', '--norc', '-eo', 'pipefail', '-c', script], { env: { PATH: process.env.PATH, ...env }, stdio: 'pipe' });
return true;
} catch {
return false;
}
}
// A gate passes only when every always-on job succeeded and every queue-only job is skipped on a
// pull request and succeeded on every other event. Anything else fails it.
function assertGateBehaviour(jobText, queueOnly) {
assert.match(jobText, /^ {10}EVENT_NAME: \$\{\{ github\.event_name \}\}$/m, 'the gate knows the event');
const variables = gateVariables(jobText);
for (const id of queueOnly) assert.ok(variables.has(id), `${id} reaches the gate`);
for (const event of EVENTS) {
const expected = id => (queueOnly.includes(id) && event === 'pull_request' ? 'skipped' : 'success');
const green = Object.fromEntries([...variables].map(([id, name]) => [name, expected(id)]));
assert.equal(runGate(jobText, { ...green, EVENT_NAME: event }), true, `${event}: the expected results pass`);
for (const [id, name] of variables) {
for (const result of ['success', 'failure', 'cancelled', 'skipped']) {
if (result === expected(id)) continue;
assert.equal(runGate(jobText, { ...green, EVENT_NAME: event, [name]: result }), false,
`${event}: ${id}=${result} must fail the gate`);
}
}
}
}
test('viewerjs keeps its name and needs every frontend job', () => {
const viewerjs = allJobs.get('viewerjs');
assert.ok(viewerjs, 'the required context is the job id viewerjs');
@@ -76,32 +115,17 @@ test('viewerjs keeps its name and needs every frontend job', () => {
assert.deepEqual([...needs].sort(), [...others].sort(), 'viewerjs needs every other job in the workflow');
assert.deepEqual([...needs].sort(), ['browser-workflows', 'frontend-coverage', 'frontend-coverage-merge']);
assert.match(viewerjs, /^ {4}if: always\(\)$/m, 'viewerjs runs (and fails) when a needed job failed or was skipped');
const script = runScript(steps(viewerjs).find(step => /run: \|/.test(step)));
for (const id of needs) {
const variable = new RegExp(`^ {10}([A-Z_]+): \\$\\{\\{ needs\\.${id}\\.result \\}\\}$`, 'm').exec(viewerjs);
assert.ok(variable, `${id}'s result reaches the check`);
assert.match(script, new RegExp(`test "\\$${variable[1]}" = success`), `${id} must be exactly success`);
}
assert.deepEqual([...gateVariables(viewerjs).keys()].sort(), [...needs].sort(), 'every needed result reaches the check');
});
test('viewerjs fails unless every needed job succeeded', () => {
const viewerjs = allJobs.get('viewerjs');
const script = runScript(steps(viewerjs).find(step => /run: \|/.test(step)));
const variables = [...viewerjs.matchAll(/^ {10}([A-Z_]+): \$\{\{ needs\.[\w-]+\.result \}\}$/gm)].map(match => match[1]);
const run = env => {
try {
execFileSync('bash', ['--noprofile', '--norc', '-eo', 'pipefail', '-c', script], { env: { PATH: process.env.PATH, ...env }, stdio: 'pipe' });
return true;
} catch {
return false;
}
};
const allSuccess = Object.fromEntries(variables.map(name => [name, 'success']));
assert.equal(run(allSuccess), true);
for (const name of variables) {
for (const result of ['failure', 'cancelled', 'skipped']) {
assert.equal(run({ ...allSuccess, [name]: result }), false, `${name}=${result} must fail viewerjs`);
}
test('viewerjs requires the browser shards everywhere but on a pull request', () => {
assertGateBehaviour(allJobs.get('viewerjs'), ['browser-workflows']);
});
test('Firefox/WebKit page workflows run only outside pull requests; coverage runs on every event', () => {
assert.match(allJobs.get('browser-workflows'), QUEUE_ONLY_IF);
for (const id of ['frontend-coverage', 'frontend-coverage-merge']) {
assert.doesNotMatch(allJobs.get(id), /^ {4}if:/m, `${id} runs on every event`);
}
});
@@ -224,7 +248,8 @@ test('browser job timeouts keep a real margin over measured runs', () => {
});
// Every job is bounded (#350): a hang ends at a measured limit instead of GitHub's 6-hour default.
const BOUNDED_WORKFLOWS = ['test.yml', 'code-quality.yml', 'viewerjs-check.yml'];
const GATING_WORKFLOWS = ['test.yml', 'code-quality.yml', 'viewerjs-check.yml'];
const BOUNDED_WORKFLOWS = [...GATING_WORKFLOWS, 'portability.yml'];
const readWorkflow = name => readFileSync(new URL(`../../.github/workflows/${name}`, import.meta.url), 'utf8');
test('every job in the CI workflows has a numeric timeout-minutes', () => {
@@ -247,3 +272,102 @@ test('the Python matrix and coverage run the suite through the watchdog runner',
assert.doesNotMatch(found.get(id), /unittest discover/, `${id} no longer runs the bare unittest discover`);
}
});
// CI hygiene (#352): lint tools are pinned, the bandit hook can fail, the JS job caches npm.
const readRepoFile = path => readFileSync(new URL(`../../${path}`, import.meta.url), 'utf8');
test('black and flake8 are pinned exactly and the lint job installs those pins', () => {
const requirements = readRepoFile('requirements-dev.txt');
assert.match(requirements, /^black==\d+\.\d+\.\d+$/m, 'black is pinned exactly');
assert.match(requirements, /^flake8==\d+\.\d+\.\d+$/m, 'flake8 is pinned exactly');
const lint = jobs(readWorkflow('test.yml')).get('lint');
assert.match(lint, /pip install \$\(grep -E '\^\(black\|flake8\)==' requirements-dev\.txt\)/);
assert.doesNotMatch(lint, /pip install black flake8/, 'lint no longer installs the latest releases');
});
test('the frontend modules job caches npm keyed on the vendor lockfile', () => {
const setupNode = steps(jobs(readWorkflow('test.yml')).get('js-test'))
.find(step => /uses: actions\/setup-node@v\d+/.test(step));
assert.match(setupNode, /^ {10}cache: npm$/m);
assert.match(setupNode, /^ {10}cache-dependency-path: tools\/vendor\/package-lock\.json$/m);
});
test('the bandit pre-commit hook fails at the severity CI enforces and passes every file', () => {
const quality = readWorkflow('code-quality.yml');
assert.match(quality, /^ {8}run: bandit -r \. -lll -q --exclude /m, 'CI fails on high severity');
const hook = /- id: bandit\n([\s\S]*?)(?=\n {6}(?:- id:|#)|\n*$)/.exec(readRepoFile('.pre-commit-config.yaml'))?.[1];
assert.ok(hook, 'the bandit hook exists');
assert.match(hook, /^ {8}entry: bandit -lll -q$/m, 'the hook enforces the same -lll level as CI');
assert.doesNotMatch(hook, /true|exit-zero|bash -c/, 'nothing masks the exit code or drops a file');
assert.doesNotMatch(hook, /pass_filenames: false/, 'the hook scans the staged files');
});
// Event split (#420): pull requests run a fast subset, the merge queue and main run everything.
const testJobs = () => jobs(readWorkflow('test.yml'));
test('the Python matrix is Linux 3.12 on a pull request and Linux 3.12-3.14 on every other event', () => {
const body = testJobs().get('test');
assert.deepEqual(matrixList(body, 'os'), ['ubuntu-latest'], 'Windows and macOS never gate a merge');
const match = /^ {8}python-version: \$\{\{ github\.event_name == 'pull_request' && fromJSON\('(\[[^\]]*\])'\) \|\| fromJSON\('(\[[^\]]*\])'\) \}\}$/m.exec(body);
assert.ok(match, 'python-version is chosen by event');
assert.deepEqual({ pr: JSON.parse(match[1]), other: JSON.parse(match[2]) }, { pr: ['3.12'], other: ['3.12', '3.13', '3.14'] });
assert.match(body, /^ {6}fail-fast: false$/m);
assert.match(body, /^ {4}name: py\$\{\{ matrix\.python-version \}\} on \$\{\{ matrix\.os \}\}$/m);
});
test('Windows and macOS run 3.12-3.14 on main, nightly and on demand, never on a PR or in the queue', () => {
const text = readWorkflow('portability.yml');
const triggers = text.slice(text.indexOf('\non:'), text.indexOf('\nconcurrency:'));
for (const event of ['push', 'schedule', 'workflow_dispatch']) assert.match(triggers, new RegExp(`^ {2}${event}:`, 'm'));
for (const event of ['pull_request', 'merge_group']) assert.doesNotMatch(triggers, new RegExp(`^ {2}${event}`, 'm'));
assert.match(triggers, /^ {4}branches: \[main\]$/m, 'push runs only for main');
const body = jobs(text).get('test');
assert.deepEqual(matrixList(body, 'os'), ['windows-latest', 'macos-latest']);
assert.deepEqual(matrixList(body, 'python-version'), ['"3.12"', '"3.13"', '"3.14"']);
assert.match(body, /^ {6}fail-fast: false$/m);
assert.match(body, /^ {8}run: python -m tests\.run_suite -v$/m);
assert.doesNotMatch(text, /cancel-in-progress: true|cancel-in-progress: \$/, 'no portability run is cancelled');
});
test('real ComfyUI runs only outside pull requests; the other test.yml checks run on every event', () => {
const found = testJobs();
assert.match(found.get('comfyui'), QUEUE_ONLY_IF);
for (const id of ['test', 'coverage', 'lint', 'js-test']) {
assert.doesNotMatch(found.get(id), /^ {4}if:/m, `${id} runs on every event`);
}
});
test('all tests needs every gating test.yml job and requires the full set outside pull requests', () => {
const found = testJobs();
const gate = found.get('all-tests');
assert.ok(gate, 'the aggregate gate exists');
assert.match(gate, /^ {4}name: all tests$/m, 'the required context is "all tests"');
assert.match(gate, /^ {4}if: always\(\)$/m);
const gating = [...found.keys()].filter(id => !['all-tests', 'perf-budget'].includes(id));
assert.deepEqual([...needsOf(gate)].sort(), gating.sort(), 'the gate needs every job but the non-gating perf budget');
assert.deepEqual([...gateVariables(gate).keys()].sort(), gating.sort(), 'every needed result reaches the gate');
assertGateBehaviour(gate, ['comfyui']);
});
test('the CI workflows run on pull_request, merge_group and push without path filters', () => {
for (const name of GATING_WORKFLOWS) {
const text = readWorkflow(name);
const triggers = text.slice(text.indexOf('\non:'), text.indexOf('\n', text.indexOf('\non:') + 1) + 400);
for (const event of ['merge_group', 'pull_request', 'push']) {
assert.match(triggers, new RegExp(`^ {2}${event}:`, 'm'), `${name} runs on ${event}`);
}
assert.doesNotMatch(text, /^ {4}paths(-ignore)?:/m, `${name} has no path filter that would leave a required check pending`);
}
assert.doesNotMatch(jobs(readWorkflow('code-quality.yml')).get('security'), /^ {4}if:/m, 'bandit runs on every event');
});
test('every required check reports on a pull request and the gates carry the full matrix', () => {
const contexts = JSON.parse(readRepoFile('.github/branch-protection.json')).required_status_checks.contexts;
assert.deepEqual([...contexts].sort(), [
'all tests', 'bandit (high severity)', 'black and flake8', 'coverage gate (85%)', 'frontend modules',
'py3.12 on ubuntu-latest', 'real ComfyUI compatibility', 'viewerjs',
].sort());
assert.equal(contexts.filter(context => /^py/.test(context)).length, 1,
'a dropped matrix leg never reports on a pull request, so it cannot be required');
assert.ok(!contexts.some(context => /windows|macos/.test(context)), 'Windows and macOS never gate a merge');
});
+98 -1
View File
@@ -1,6 +1,7 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { COLUMNS, CONTROL_EXEMPTIONS, STATUSES, declaredTitles, parseControl, parseCoverageTable, readCoverageTable,
import { COLUMNS, CONTROL_EXEMPTIONS, PAGES, REMOVAL_COLUMNS, STATUSES, declaredTitles, duplicateRemovals, idUses, methodUses,
parseControl, parseCoverageTable, parseRemovalTable, readCoverageTable, readPage, readRemovalTable, removalProblems,
rowProblems } from '../../tools/vendor/legacy-coverage.mjs';
// TESTING.md "Legacy System capability coverage" (#249 System S12, #329): every legacy System capability names an
@@ -114,3 +115,99 @@ test('a covered row names its native control unless it is exempt', () => {
assert.deepEqual([row.Status, row['Native control']], ['covered', '—']);
}
});
// TESTING.md "Legacy System UI removed by S13" (#330): a legacy element is removed only when its capability is
// `covered`, the removal copies that row's native test, and the element is really gone from its page.
const COVERAGE_HEADER = '| Capability | Legacy anchor | Owner | Status | Native control | Test |\n| --- | --- | --- | --- | --- | --- |';
const REMOVAL_HEADER = '| Capability | Page | Removed ids | Removed methods | Native test |\n| --- | --- | --- | --- | --- |';
const FIXTURE_COVERAGE = parseCoverageTable(['### Legacy System capability coverage', '', COVERAGE_HEADER,
'| View logs | `admin.html:1` | S12 Logs | covered | `diagnostics` · combobox · "Level" | level filter works (logs.spec.mjs) |',
'| Infinite scrolling default | `admin.html:2` | Legacy prompt list | kept in legacy | — | scrolls (list-state.test.js) |',
''].join('\n'));
const removalRow = cells => parseRemovalTable(['### Legacy System UI removed by S13', '', REMOVAL_HEADER, `| ${cells} |`, '']
.join('\n'))[0];
const EMPTY_PAGE = () => ({ html: '<button id="kept">Kept</button>', script: 'class Admin {\n kept() {}\n}\n' });
test('every removed legacy element cites a covered capability and its native test', () => {
const coverage = readCoverageTable();
const rows = readRemovalTable();
for (const row of rows) {
const cited = coverage.find(item => item.Capability === row.Capability);
assert.ok(cited, `TESTING.md:${row.line} cites "${row.Capability}"`);
assert.equal(cited.Status, 'covered', `TESTING.md:${row.line} ${row.Capability}`);
assert.equal(row['Native test'], cited.Test, `TESTING.md:${row.line} ${row.Capability}`);
assert.ok(Object.keys(PAGES).includes(row.Page), `TESTING.md:${row.line} page ${row.Page}`);
}
assert.deepEqual(duplicateRemovals(rows), []);
assert.deepEqual(REMOVAL_COLUMNS, ['Capability', 'Page', 'Removed ids', 'Removed methods', 'Native test']);
});
test('a removed id or method is gone from its page', () => {
const coverage = readCoverageTable();
for (const row of readRemovalTable()) {
assert.deepEqual(removalProblems(row, coverage, page => readPage(page)), [], `TESTING.md:${row.line} ${row.Capability}`);
}
// Each way a page can still hold an id or a method is found; a call through another object, or a longer name, is not
// a definition.
const page = {
html: '<div id="logsModal"></div><button onclick="window.admin.closeLogs()">x</button>',
script: [
"document.getElementById('logLevel').value;",
"this.showModal('diagnosticsModal');",
'<span data-id="logsContainer"></span>',
'document.querySelector("#logLimit option");',
"const backdrops = ['settingsModal', 'duplicatesModal'];",
'class Admin {',
' async refreshLogs(level) {',
' }',
'} displayLogs(logs) {',
' }',
'this.loadLogStats();',
'// downloadLogs() {',
'other.formatBytesLater(1);',
].join('\n'),
};
assert.deepEqual(idUses('logsModal', page), ['id="logsModal" in the markup']);
assert.deepEqual(idUses('logLevel', page), ["getElementById('logLevel')"]);
assert.deepEqual(idUses('logLimit', page), ["querySelector('#logLimit')"]);
assert.deepEqual(idUses('duplicatesModal', page), ["'duplicatesModal' in a modal list"]);
assert.deepEqual(idUses('logLevelX', page), []);
assert.deepEqual(idUses('diagnosticsModal', page), ["a …Modal('diagnosticsModal') call"]);
assert.deepEqual(idUses('logsContainer', page), []);
assert.deepEqual(methodUses('closeLogs', page), ['.closeLogs( called in the markup']);
assert.deepEqual(methodUses('refreshLogs', page), ['refreshLogs() defined']);
assert.deepEqual(methodUses('displayLogs', page), ['displayLogs() defined']);
assert.deepEqual(methodUses('loadLogStats', page), ['.loadLogStats( called']);
assert.deepEqual(methodUses('downloadLogs', page), []);
assert.deepEqual(methodUses('formatBytes', page), []);
// One id or method is listed on one row per page.
const twice = [removalRow('View logs | admin | `#logsModal` | — | level filter works (logs.spec.mjs)'),
{ ...removalRow('View logs | admin | `#logsModal` | — | level filter works (logs.spec.mjs)'), line: 9 }];
assert.match(duplicateRemovals(twice).join(), /admin #logsModal is listed on rows/);
assert.deepEqual(removalProblems(removalRow('View logs | admin | `logsModal`, `#ok` | `close` | level filter works (logs.spec.mjs)'),
FIXTURE_COVERAGE, EMPTY_PAGE), ['malformed entry "logsModal"', 'malformed entry "close"']);
});
test('a removal citing a kept-in-legacy capability fails', () => {
const row = removalRow('Infinite scrolling default | admin | `#infiniteScrollDefault` | — | scrolls (list-state.test.js)');
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, EMPTY_PAGE), ['cites a kept in legacy capability']);
const unknown = removalRow('Something else | admin | — | — | scrolls (list-state.test.js)');
assert.deepEqual(removalProblems(unknown, FIXTURE_COVERAGE, EMPTY_PAGE), ['cites no single coverage row "Something else"']);
});
test('a removal whose native test differs fails', () => {
const row = removalRow('View logs | admin | `#logsModal` | `showLogsModal()` | another title (logs.spec.mjs)');
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, EMPTY_PAGE),
['its native test differs from the coverage row\'s "level filter works (logs.spec.mjs)"']);
assert.deepEqual(removalProblems({ ...row, 'Native test': 'level filter works (logs.spec.mjs)' }, FIXTURE_COVERAGE, EMPTY_PAGE), []);
});
test('a removed id still on its page fails', () => {
const row = removalRow('View logs | admin | `#logsModal` | `showLogsModal()` | level filter works (logs.spec.mjs)');
const page = () => ({ html: '<div id="logsModal" class="hidden"></div>', script: 'class Admin {\n showLogsModal() {\n }\n}\n' });
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, page), [
'#logsModal is still on the admin page: id="logsModal" in the markup',
'showLogsModal() is still on the admin page: showLogsModal() defined',
]);
assert.deepEqual(removalProblems({ ...row, Page: 'metadata' }, FIXTURE_COVERAGE, page), ['unknown page "metadata"']);
});
+140 -8
View File
@@ -42,25 +42,30 @@ function cells(line) {
return out;
}
/** The table's rows as objects keyed by COLUMNS, from the section that starts at HEADING. */
export function parseCoverageTable(markdown) {
/** The rows, keyed by `columns`, of the first Markdown table under `heading` (`label` names it in errors). */
function parseTable(markdown, heading, columns, label) {
const lines = markdown.split('\n');
const start = lines.findIndex(line => line.trim() === HEADING);
if (start === -1) throw new Error(`TESTING.md has no "${HEADING}" section`);
const start = lines.findIndex(line => line.trim() === heading);
if (start === -1) throw new Error(`TESTING.md has no "${heading}" section`);
const header = lines.findIndex((line, index) => index > start && line.trim().startsWith('|'));
if (header === -1 || lines.slice(start + 1, header).some(line => /^#{1,3} /.test(line))) {
throw new Error('The coverage section has no table');
throw new Error(`The ${label} section has no table`);
}
const columns = cells(lines[header]);
if (columns.join('|') !== COLUMNS.join('|')) throw new Error(`The coverage table's columns are ${columns.join(', ')}`);
const found = cells(lines[header]);
if (found.join('|') !== columns.join('|')) throw new Error(`The ${label} table's columns are ${found.join(', ')}`);
const rows = [];
for (let index = header + 2; index < lines.length && lines[index].trim().startsWith('|'); index += 1) {
const values = cells(lines[index]);
rows.push({ line: index + 1, ...Object.fromEntries(COLUMNS.map((column, at) => [column, values[at] ?? ''])) });
rows.push({ line: index + 1, ...Object.fromEntries(columns.map((column, at) => [column, values[at] ?? ''])) });
}
return rows;
}
/** The table's rows as objects keyed by COLUMNS, from the section that starts at HEADING. */
export function parseCoverageTable(markdown) {
return parseTable(markdown, HEADING, COLUMNS, 'coverage');
}
/**
* A Native control cell: `` `<section>` · <role> · "<name>" ``, optionally ` · in "<row title>"` when the name is shared
* by several rows of the section → {section, role, name, row}; `—` → null; else it throws.
@@ -213,3 +218,130 @@ export function rowProblems(row, { root = ROOT, locate = file => testFile(file,
export function readCoverageTable(root = ROOT) {
return parseCoverageTable(readFileSync(join(root, 'TESTING.md'), 'utf8'));
}
// ── Legacy System UI removed by S13 (#330) ──────────────────────────────────────────────────────────────────────────
// TESTING.md "Legacy System UI removed by S13": one row per covered capability whose legacy UI S13 removed, with the
// element ids and methods that went. A row must cite a `covered` coverage row and copy its Test cell, and none of its
// ids or methods may still be on its page.
export const REMOVAL_HEADING = '### Legacy System UI removed by S13';
export const REMOVAL_COLUMNS = Object.freeze(['Capability', 'Page', 'Removed ids', 'Removed methods', 'Native test']);
/** Each legacy page's markup and script. */
export const PAGES = Object.freeze({
admin: Object.freeze({ html: 'web/admin.html', script: 'web/js/admin.js' }),
gallery: Object.freeze({ html: 'web/gallery.html', script: 'web/js/gallery.js' }),
});
/** The removal table's rows, from the section that starts at REMOVAL_HEADING. */
export function parseRemovalTable(markdown) {
return parseTable(markdown, REMOVAL_HEADING, REMOVAL_COLUMNS, 'removal');
}
/** The removal table read from TESTING.md. */
export function readRemovalTable(root = ROOT) {
return parseRemovalTable(readFileSync(join(root, 'TESTING.md'), 'utf8'));
}
/** A Removed ids or Removed methods cell → its entries without backticks (`—` → none). */
export function listCell(cell) {
if (cell.trim() === NONE) return [];
return cell.split(',').map(entry => entry.trim().replace(/^`|`$/g, '')).filter(Boolean);
}
/** The removed ids of a row (without `#`) and its malformed entries. */
function removedIds(row) {
const entries = listCell(row['Removed ids']);
return { ids: entries.filter(entry => /^#[\w-]+$/.test(entry)).map(entry => entry.slice(1)),
bad: entries.filter(entry => !/^#[\w-]+$/.test(entry)) };
}
/** The removed methods of a row (without `()`) and its malformed entries. */
function removedMethods(row) {
const entries = listCell(row['Removed methods']);
return { names: entries.filter(entry => /^[A-Za-z_$][\w$]*\(\)$/.test(entry)).map(entry => entry.slice(0, -2)),
bad: entries.filter(entry => !/^[A-Za-z_$][\w$]*\(\)$/.test(entry)) };
}
const escapeRegExp = text => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
/**
* Where `id` is still on a page: `id="<id>"` in its markup or script; in its script a lookup, a `…Modal("<id>")` call
* (`showModal`, `hideModal`) or a modal-list entry.
*/
export function idUses(id, { html, script }) {
const name = escapeRegExp(id);
const code = stripJsComments(script);
const uses = [];
// `id=`, not `data-id=`: the attribute starts after a space or a tag's `<name`.
const attribute = new RegExp(`(?:^|\\s)id=["']${name}["']`, 'm');
if (attribute.test(html)) uses.push(`id="${id}" in the markup`);
if (attribute.test(code)) uses.push(`id="${id}" in the script`);
if (new RegExp(`getElementById\\(\\s*["'\`]${name}["'\`]\\s*\\)`).test(code)) uses.push(`getElementById('${id}')`);
if (new RegExp(`querySelector(?:All)?\\(\\s*["'\`]#${name}\\b`).test(code)) uses.push(`querySelector('#${id}')`);
if (new RegExp(`\\w*Modal\\(\\s*["'\`]${name}["'\`]`).test(code)) uses.push(`a …Modal('${id}') call`);
// A string list of modal ids (the gallery's backdrop list): the entry next to another `…Modal` entry.
for (const list of code.matchAll(/\[[^\[\]]*\]/g)) {
const entry = new RegExp(`["']${name}["']`);
if (entry.test(list[0]) && /["'][\w-]*Modal["']/.test(list[0].replace(entry, ''))) {
uses.push(`'${id}' in a modal list`);
break;
}
}
return uses;
}
/**
* Where method `name` is still on a page: a definition in its script, or a `.<name>(` call (`this.`, `window.admin.`)
* in its script or markup.
*/
export function methodUses(name, { html, script }) {
const method = escapeRegExp(name);
const code = stripJsComments(script);
const uses = [];
if (new RegExp(`(?:^|[\\s;}])(?:async\\s+)?${method}\\s*\\([^)]*\\)\\s*\\{`, 'm').test(code)) uses.push(`${name}() defined`);
if (new RegExp(`\\.${method}\\s*\\(`).test(code)) uses.push(`.${name}( called`);
if (new RegExp(`\\.${method}\\s*\\(`).test(html)) uses.push(`.${name}( called in the markup`);
return uses;
}
/**
* Every problem of one removal row (empty: sound). `coverage` is the coverage table's rows; `readPage(page)` returns
* that page's `{html, script}`.
*/
export function removalProblems(row, coverage, readPage) {
const problems = [];
const cited = coverage.filter(item => item.Capability === row.Capability);
if (cited.length !== 1) problems.push(`cites no single coverage row "${row.Capability}"`);
else if (cited[0].Status !== 'covered') problems.push(`cites a ${cited[0].Status} capability`);
else if (cited[0].Test !== row['Native test']) problems.push(`its native test differs from the coverage row's "${cited[0].Test}"`);
if (!Object.hasOwn(PAGES, row.Page)) {
problems.push(`unknown page "${row.Page}"`);
return problems;
}
const { ids, bad: badIds } = removedIds(row);
const { names, bad: badNames } = removedMethods(row);
for (const entry of [...badIds, ...badNames]) problems.push(`malformed entry "${entry}"`);
const page = readPage(row.Page);
for (const id of ids) for (const use of idUses(id, page)) problems.push(`#${id} is still on the ${row.Page} page: ${use}`);
for (const name of names) for (const use of methodUses(name, page)) problems.push(`${name}() is still on the ${row.Page} page: ${use}`);
return problems;
}
/** Ids and methods listed on more than one row of the same page. */
export function duplicateRemovals(rows) {
const seen = new Map();
const duplicates = [];
for (const row of rows) {
for (const entry of [...listCell(row['Removed ids']), ...listCell(row['Removed methods'])]) {
const key = `${row.Page} ${entry}`;
if (seen.has(key)) duplicates.push(`${key} is listed on rows ${seen.get(key)} and ${row.line}`);
else seen.set(key, row.line);
}
}
return duplicates;
}
/** A page's markup and script from the repository. */
export function readPage(page, root = ROOT) {
const files = PAGES[page];
return { html: readFileSync(join(root, files.html), 'utf8'), script: readFileSync(join(root, files.script), 'utf8') };
}
+110
View File
@@ -0,0 +1,110 @@
"""The process exit signal: running work stops at its next safe point (#376).
ComfyUI catches Ctrl+C in ``main.py`` and returns without closing its event
loop. The interpreter then runs ``threading._shutdown``, which first calls the
``threading._register_atexit`` callbacks, newest first. One of them,
``concurrent.futures``'s ``_python_exit``, joins every executor thread, and
only after that do the ``atexit`` callbacks run (the package teardown). A call
still running on the default executor (a scan batch, a model download, a
thumbnail batch) used to keep the process alive until it returned.
``install()`` registers a callback in that same list after ``concurrent.futures``
did, so it runs before the join. It sets the signal that ``requested()`` reads
(``Job.cancelled`` and ``map_parallel`` read it) and runs ``on_exit``.
``threading._register_atexit`` is CPython-internal. It has the same contract
on 3.12, 3.13 and 3.14. Without it, or when it refuses the registration, exit
behaves as before: the call in flight runs to its end.
"""
import logging
import threading
logger = logging.getLogger("prompt_manager.exit")
_requested = threading.Event()
_install_lock = threading.Lock()
_installed = False
_on_exit = []
class ExitRequested(Exception):
"""The interpreter is exiting: work not yet started is not started."""
def requested():
"""True once the interpreter has started exiting."""
return _requested.is_set()
def request():
"""Set the signal (the exit hook; tests)."""
_requested.set()
def checkpoint():
"""Raise ExitRequested once the interpreter is exiting."""
if _requested.is_set():
raise ExitRequested("ComfyUI is exiting")
def checked(func):
"""``func`` behind a checkpoint: a call that starts after the signal
raises ExitRequested instead of running."""
def run(*args, **kwargs):
checkpoint()
return func(*args, **kwargs)
return run
def _hook():
"""The threading atexit callback. It never raises: an exception here
would end ``threading._shutdown`` before it shuts the executors."""
_requested.set()
for action in tuple(_on_exit):
try:
action()
except (Exception, KeyboardInterrupt):
logger.error("Stopping work at exit failed", exc_info=True)
def install(on_exit=None):
"""Register the exit hook once per process; True when it is registered.
``on_exit`` (optional) runs inside the hook, after the signal is set.
False (logged at DEBUG) when this Python has no ``_register_atexit`` or
refuses it; exit then waits for running work as before.
"""
global _installed
with _install_lock:
if on_exit is not None and on_exit not in _on_exit:
_on_exit.append(on_exit)
if _installed:
return True
# Imported first so its pool join is registered before our hook:
# the callbacks run newest first, so ours runs before that join.
import concurrent.futures.thread # noqa: F401
register = getattr(threading, "_register_atexit", None)
if not callable(register):
logger.debug("threading._register_atexit is missing; exit waits for jobs")
return False
try:
register(_hook)
except RuntimeError as error: # already shutting down
logger.debug(f"Exit hook not registered: {error}")
return False
_installed = True
return True
def reset():
"""Clear the signal and forget the registration (tests only; a hook
already registered stays in ``threading`` and sets the signal at exit)."""
global _installed
with _install_lock:
_requested.clear()
_installed = False
_on_exit.clear()
+13 -3
View File
@@ -35,6 +35,7 @@ from watchdog.events import FileSystemEventHandler
from .hashing import generate_prompt_hash
from .metadata_extractor import ComfyUIMetadataExtractor
from .parallel import note_refusal, refused_at_shutdown
from .logging_config import get_logger
from .prompt_graph import resolve_text, run_prompt_nodes
from .prompt_pair_resolver import resolve_prompt_pair
@@ -194,8 +195,9 @@ class ImageGenerationHandler(FileSystemEventHandler):
and once its size has stopped changing.
Returns:
False when the path is already pending, the pending set is full or
the handler is closed
False when the path is already pending, the pending set is full,
the handler is closed or the interpreter is exiting (its worker
cannot start)
"""
with self._pending_lock:
if self._closed:
@@ -211,7 +213,15 @@ class ImageGenerationHandler(FileSystemEventHandler):
return False
due = time.monotonic() + self.processing_delay
self._pending[image_path] = (prompt_snapshot, due)
self._ensure_worker()
try:
self._ensure_worker()
except RuntimeError as error:
# No worker will ever take it: never leave it pending (#375).
self._pending.pop(image_path, None)
if not refused_at_shutdown(error):
raise
note_refusal(self.logger, "The image worker")
return False
self._work.put(image_path)
return True
+239 -53
View File
@@ -39,6 +39,35 @@ class RotatedChanged(Exception):
self.name = name
# Clear all logs (#389): why it changed nothing, as the API and the pages say it.
CLEAR_CHANGED = (
"The log files changed since you confirmed; nothing was cleared."
" Read the list again and clear again."
)
CLEAR_FOREIGN_WRITER = (
"Another program is writing to prompt_manager.log, such as a second ComfyUI"
" sharing this log folder; nothing was cleared. Stop it and clear again."
)
CLEAR_HARD_LINK = (
"prompt_manager.log has another name (a hard link); emptying it would empty"
" that file too, so nothing was cleared. Remove the other name, then clear"
" again."
)
# A confirmed row's identity: "<size>:<mtime_ns>:<ino>", a string because
# mtime_ns and many inodes are larger than a browser's exact integers.
# mtime_ns is negative for a time before 1970 (a restored archive, say).
_IDENTITY = re.compile(r"([0-9]{1,20}):(-?[0-9]{1,20}):([0-9]{1,40})")
MAX_CONFIRMED_ROWS = 10000
class ClearRefused(Exception):
"""Clear all logs changed nothing; ``code`` names why, the message says it."""
def __init__(self, code: str, message: str):
super().__init__(message)
self.code = code
def _name_key(name: str) -> str:
"""How a name compares: case-folded where the filesystem folds it (Windows)."""
return os.path.normcase(name)
@@ -87,35 +116,37 @@ def scan_log_folder(log_dir) -> List[Dict[str, Any]]:
be read is left out. Never logs: the cleanup calls it while it holds the
file handler's lock.
"""
rows = []
try:
names = [entry.name for entry in os.scandir(log_dir)]
except OSError:
return rows
for name in names:
path = os.path.join(log_dir, name)
try:
st = os.lstat(path)
kind = _kind(name, path, st)
except OSError:
continue
try:
modified = datetime.fromtimestamp(st.st_mtime).isoformat()
except (OSError, OverflowError, ValueError): # a time before 1970 on Windows
modified = ""
rows.append(
{
"name": name,
"kind": kind,
"size": st.st_size,
"mtime_ns": st.st_mtime_ns,
"ino": st.st_ino,
"modified": modified,
"_dir": stat.S_ISDIR(st.st_mode) and kind != "link",
"_regular": stat.S_ISREG(st.st_mode) and kind != "link",
}
)
return rows
return []
rows = (scan_log_entry(log_dir, name) for name in names)
return [row for row in rows if row is not None]
def scan_log_entry(log_dir, name) -> Optional[Dict[str, Any]]:
"""One entry's scan row (``lstat``), or None when it cannot be read."""
path = os.path.join(log_dir, name)
try:
st = os.lstat(path)
kind = _kind(name, path, st)
except OSError:
return None
try:
modified = datetime.fromtimestamp(st.st_mtime).isoformat()
except (OSError, OverflowError, ValueError): # a time before 1970 on Windows
modified = ""
return {
"name": name,
"kind": kind,
"size": st.st_size,
"mtime_ns": st.st_mtime_ns,
"ino": st.st_ino,
"modified": modified,
"_dir": stat.S_ISDIR(st.st_mode) and kind != "link",
"_regular": stat.S_ISREG(st.st_mode) and kind != "link",
"_nlink": st.st_nlink,
}
def public_row(row: Dict[str, Any]) -> Dict[str, Any]:
@@ -127,6 +158,37 @@ def rotated_sort_key(row: Dict[str, Any]) -> int:
return _rotated_number(row["name"]) or 0
def file_identity(row: Dict[str, Any]) -> str:
"""A listed file's identity as the client sends it back: size, mtime_ns, ino."""
return f"{row['size']}:{row['mtime_ns']}:{row['ino']}"
def parse_confirmed(expected) -> List[Dict[str, Any]]:
"""The rows a Clear all logs confirmation showed, as review rows.
``expected`` is a list of ``{name, identity}`` objects from
``/logs/files``; anything else raises ValueError.
"""
if not isinstance(expected, list) or len(expected) > MAX_CONFIRMED_ROWS:
raise ValueError("expected must be a list of listed log files")
rows, seen = [], set()
for item in expected:
if not isinstance(item, dict):
raise ValueError("each expected file must be an object")
name, identity = item.get("name"), item.get("identity")
if not isinstance(name, str) or not isinstance(identity, str):
raise ValueError("each expected file needs a name and an identity")
match = _IDENTITY.fullmatch(identity)
if match is None:
raise ValueError("an expected file's identity is not size:mtime_ns:ino")
size, mtime_ns, ino = (int(part) for part in match.groups())
if _name_key(name) in seen:
raise ValueError("an expected file is listed twice")
seen.add(_name_key(name))
rows.append({"name": name, "size": size, "mtime_ns": mtime_ns, "ino": ino})
return rows
def log_file_entry(log_dir, name) -> Path:
"""The path of a downloadable log file named ``name`` (blocking).
@@ -157,6 +219,11 @@ def _stream_is(stream, ino: int) -> bool:
return not (ino and open_ino and open_ino != ino)
def _stream_ino_is(opened: os.stat_result, ino: int) -> bool:
"""Whether a stream's ``fstat`` is the file with inode ``ino`` (0: unknown)."""
return not (ino and opened.st_ino and opened.st_ino != ino)
def _error_text(error: BaseException) -> str:
"""An OSError's reason and file name, without the folder it is in."""
if isinstance(error, OSError):
@@ -432,6 +499,7 @@ class PromptManagerLogger:
"modified": row["modified"],
"is_main": row["kind"] == "active",
"kind": row["kind"],
"identity": file_identity(row),
}
)
@@ -530,8 +598,14 @@ class PromptManagerLogger:
finally:
handler.release()
if not handled:
flags = os.O_WRONLY | os.O_TRUNC | getattr(os, "O_NOFOLLOW", 0)
os.close(os.open(str(log_file), flags))
flags = os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0)
fd = os.open(str(log_file), flags)
try:
if os.fstat(fd).st_nlink > 1: # emptying it would empty the other
raise ClearRefused("logs-hard-link", CLEAR_HARD_LINK)
os.ftruncate(fd, 0)
finally:
os.close(fd)
def delete_rotated(self, expected: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Delete exactly the reviewed rotated copies, or none of them (blocking).
@@ -582,47 +656,126 @@ class PromptManagerLogger:
)
return {"removed": removed, "failed": failed}
def truncate_logs(self) -> Dict[str, Any]:
"""Empty the active log and delete its rotated copies (Rule R).
def _active_streams(self, active: Optional[Dict[str, Any]]):
"""``fstat`` and offset of every handler stream writing the active name."""
target = os.path.normcase(
os.path.realpath(os.path.join(self.log_dir, ACTIVE_LOG_NAME))
)
for handler in self._file_handlers():
stream = getattr(handler, "stream", None)
base = getattr(handler, "baseFilename", None)
if stream is None or base is None:
continue
if os.path.normcase(os.path.realpath(base)) != target:
continue
try:
stream.flush()
fd = stream.fileno()
yield os.fstat(fd), os.lseek(fd, 0, os.SEEK_CUR)
except (OSError, ValueError, AttributeError):
continue
Every other file in the log folder, and every link, is kept. Also
clears the memory buffer. Runs under ``_files_lock`` and the file
handlers' locks, so no rollover recreates a just-deleted copy, and
logs only once they are released.
def _check_clear_is_safe(self, rows, active) -> None:
"""Refuse a clear that would empty or delete what nobody confirmed.
Called under the files lock and every handler lock, so nothing of this
process writes or rolls over meanwhile; what changes is another
program's doing. Our handler's stream is flushed: a file larger than
the stream's own offset was appended to by someone else, and a stream
whose file is now a rotated copy was rolled over by someone else.
Either is a second writer (another ComfyUI sharing the folder). This
is best effort, not a lock: writes before our own last write, or any
write while this process has file logging off, go unseen. An
active log with more than one name (a hard link) is never emptied: the
other name would be emptied too.
"""
rotated = {row["ino"] for row in rows if row["kind"] == "rotated"}
rotated.discard(0)
nlink = active["_nlink"] if active is not None else 1
for opened, offset in self._active_streams(active):
if (opened.st_ino and opened.st_ino in rotated) or opened.st_size > offset:
raise ClearRefused("logs-foreign-writer", CLEAR_FOREIGN_WRITER)
if active is not None and _stream_ino_is(opened, active["ino"]):
nlink = max(nlink, opened.st_nlink)
if nlink > 1:
raise ClearRefused("logs-hard-link", CLEAR_HARD_LINK)
@staticmethod
def _confirmed_targets(confirmed, current, active) -> List[Dict[str, Any]]:
"""The rotated copies a confirmation listed, each as it was listed.
Refuses (``logs-changed``) when any listed copy is missing or differs
in kind, size, ``mtime_ns`` or inode, or when the listed active log is
now another file (it was rolled over). A copy nobody listed is kept.
"""
targets = []
for reviewed in confirmed:
key = _name_key(reviewed["name"])
row = current.get(key)
if key == _name_key(ACTIVE_LOG_NAME):
ino = reviewed["ino"]
if active is None or (ino and active["ino"] and active["ino"] != ino):
raise ClearRefused("logs-changed", CLEAR_CHANGED)
continue
if _rotated_number(reviewed["name"]) is None:
continue # never deleted, listed or not
if not _matches_review(row, reviewed):
raise ClearRefused("logs-changed", CLEAR_CHANGED)
targets.append(row)
return targets
def truncate_logs(self, expected=None) -> Dict[str, Any]:
"""Empty the active log and delete its rotated copies (Rule R, #389).
``expected`` is what the confirmation showed: ``{name, identity}``
rows of ``/logs/files`` (``parse_confirmed``; ValueError when
malformed). Then only the listed rotated copies are deleted, and only
when each still has the listed size, ``mtime_ns`` and inode and the
active log is still the listed file; otherwise ClearRefused
(``logs-changed``) with nothing changed. Without it (the legacy admin
page, whose confirmation lists nothing) every rotated copy present now
is deleted.
Either way, a second writer of the active log (``logs-foreign-writer``)
or an active log with a hard link (``logs-hard-link``) refuses the
whole clear first. Each copy's identity is checked again just before
its unlink; one that changed (another process rolling over meanwhile)
stops the clear there, keeping it, the rest and the active log, which
is emptied last. Every other file in the log folder, and every link,
is kept. Also clears the memory buffer. Runs under ``_files_lock`` and
the file handlers' locks and logs only once they are released.
Returns:
Dictionary with:
- truncated: List of successfully truncated files
- errors: List of error messages for failed operations
"""
confirmed = None if expected is None else parse_confirmed(expected)
results = {"truncated": [], "errors": []}
refused = None
with self._files_lock:
held = self._acquire_file_handlers()
try:
rows = scan_log_folder(self.log_dir)
active = [row for row in rows if row["kind"] == "active"]
rotated = sorted(
(row for row in rows if row["kind"] == "rotated"),
key=rotated_sort_key,
)
for row in active + rotated:
path = os.path.join(self.log_dir, row["name"])
try:
if row["kind"] == "active":
self._truncate_main_log(Path(path), row["ino"])
else:
os.unlink(path)
results["truncated"].append(row["name"])
except OSError as error:
results["errors"].append(
f"Failed to truncate {row['name']}: {_error_text(error)}"
)
with self._buffer_lock:
self._log_buffer.clear()
current = {_name_key(row["name"]): row for row in rows}
active = next((row for row in rows if row["kind"] == "active"), None)
try:
if confirmed is None:
targets = [row for row in rows if row["kind"] == "rotated"]
else:
targets = self._confirmed_targets(confirmed, current, active)
self._check_clear_is_safe(rows, active)
except ClearRefused as error:
refused = error
else:
self._clear(sorted(targets, key=rotated_sort_key), active, results)
finally:
self._release_handlers(held)
if refused is not None:
self.logger.warning("Clear all logs refused (%s)", refused.code)
raise refused
if results["truncated"]:
self.logger.info("Truncated log files: %s", ", ".join(results["truncated"]))
for error_msg in results["errors"]:
@@ -630,6 +783,39 @@ class PromptManagerLogger:
return results
def _clear(self, targets, active, results) -> None:
"""Delete ``targets`` (each re-checked) then empty ``active`` (locks held)."""
for reviewed in targets:
name = reviewed["name"]
if not _matches_review(scan_log_entry(self.log_dir, name), reviewed):
results["errors"].append(
f"{name} changed while clearing (another program may be writing"
" to this log folder); it, the copies after it and the active"
" log were kept"
)
return
try:
os.unlink(os.path.join(self.log_dir, name))
results["truncated"].append(name)
except OSError as error:
results["errors"].append(
f"Failed to truncate {name}: {_error_text(error)}"
)
if active is not None:
try:
self._truncate_main_log(
Path(self.log_dir, active["name"]), active["ino"]
)
results["truncated"].append(active["name"])
except ClearRefused as error:
results["errors"].append(str(error))
except OSError as error:
results["errors"].append(
f"Failed to truncate {active['name']}: {_error_text(error)}"
)
with self._buffer_lock:
self._log_buffer.clear()
def update_config(self, new_config: Dict[str, Any]):
"""Update logging configuration.
+72 -2
View File
@@ -5,22 +5,92 @@ a CUDA-initialised, multi-threaded process is unsafe and the spawn start
method would re-import ComfyUI's main module in every child. Pillow releases
the GIL while decoding, resizing and encoding, so threads still scale close
to linearly for both jobs.
Interpreter exit (#375): ``threading._shutdown`` shuts every thread pool
(``submit`` then raises "cannot schedule new futures after interpreter
shutdown") before it joins the non-daemon threads and long before the package
teardown runs, while our daemon threads keep working. Measured on 3.12 and
3.14, ``Thread.start`` still succeeds in that window; it is refused ("can't
create new thread at interpreter shutdown") only once the interpreter is
marked finalizing, after the ``atexit`` callbacks. ``refused_at_shutdown``
recognises both; ``note_refusal`` logs one DEBUG line per place, with no
traceback.
"""
import logging
import threading
from concurrent.futures import ThreadPoolExecutor
from .exit_signal import checked
SHUTDOWN_REFUSAL = "interpreter shutdown"
_logger = logging.getLogger("prompt_manager.parallel")
_noted = set()
_noted_lock = threading.Lock()
def refused_at_shutdown(error):
"""Whether ``error`` is a thread or pool start refused at interpreter exit."""
return isinstance(error, RuntimeError) and SHUTDOWN_REFUSAL in str(error)
def note_refusal(logger, what):
"""Log once per ``what``, at DEBUG and without a traceback, that ``what``
was refused because the interpreter is exiting."""
key = (logger.name, what)
with _noted_lock:
if key in _noted:
return
_noted.add(key)
logger.debug(f"{what}: refused, the interpreter is exiting")
def map_parallel(func, items, workers):
"""Apply ``func`` to every item on up to ``workers`` threads, keeping order.
``workers`` of 1 (or fewer) runs on the calling thread with no pool. An
exception in ``func`` propagates to the caller, so callers that must
survive bad input wrap ``func`` themselves.
survive bad input wrap ``func`` themselves. A pool refused at interpreter
exit runs the items it did not start on the calling thread (#375). An item
that has not started when the interpreter starts exiting raises
ExitRequested instead (#376).
"""
func = checked(func)
items = list(items)
workers = max(1, int(workers or 1))
if workers == 1 or len(items) <= 1:
return [func(item) for item in items]
max_workers = min(workers, len(items))
with ThreadPoolExecutor(max_workers, thread_name_prefix="pm-worker") as pool:
return list(pool.map(func, items))
futures = []
try:
for item in items:
futures.append(pool.submit(func, item))
except RuntimeError as error:
if not refused_at_shutdown(error):
raise
note_refusal(_logger, "A worker pool")
# Drop what was queued but not started; it runs here instead.
# (A refused Thread.start leaves its item queued without a future
# here: it is cancelled too, unless a busy pool thread took it in
# between, the one case where an item can run twice.)
pool.shutdown(wait=True, cancel_futures=True)
else:
return [future.result() for future in futures]
return _run_rest(func, items, futures)
def _run_rest(func, items, futures):
"""The pool's finished results, and every item it did not run, run here.
After ``shutdown(cancel_futures=True)`` each submitted future either ran
(its result or exception is kept) or was cancelled before it started.
"""
results = []
for index, item in enumerate(items):
future = futures[index] if index < len(futures) else None
if future is not None and not future.cancelled():
results.append(future.result())
else:
results.append(func(item))
return results
+176 -31
View File
@@ -40,6 +40,19 @@ while it is pending and undecided (never beside a decision, see
profile the item recorded (``captured_review(..., item_profile(evidence))``).
The inspection profile only ever produces review choices; ingestion's own
links stay on the default profile, which never loads the table (D1).
The saver anchor (#390): the inspection profile resolves a file with its
name, as the workspace inspector does since #383, so the one saver whose
``filename_prefix`` names the file anchors its branch. When that anchor alone
resolves a file the default profile left ambiguous and nothing links
(``saver_anchored``), the scan queues it with the anchor's pair as its one
candidate (evidence ``anchor: "saver"``), never linking it: a file never
falls through both the link and the queue (maintainer ruling). No match,
several matches or an unfinished walk anchor nothing and the file is queued
as before. Apply re-resolves with the file's name, so only the anchored
branch can be applied. ``REVIEW_RULE_VERSION`` is folded into inspection
item versions, so inspection items recorded before the anchor are recorded
again by the next scan.
"""
from __future__ import annotations
@@ -104,10 +117,16 @@ MAX_CANDIDATE_DEPTH = 8
# The resolver profile an item's evidence was built with (``profile``).
DEFAULT = "default"
INSPECTION = "inspection"
# The ``anchor`` an item recorded from the saver anchor names (#390).
SAVER_ANCHOR = "saver"
ANCHOR_REASON = "The saver that wrote this file names its branch"
# Inspection item versions: above every plausible ``RESOLVER_VERSION`` and,
# with 40 bits of the folded digest, below 2**53 (exact as a JSON number).
INSPECTION_VERSION_BASE = 1 << 40
_INSPECTION_VERSION_HEX = 10
# How the scan and Apply use the inspection profile, folded into inspection
# item versions. 1: they resolve with the file's name (the saver anchor, #390).
REVIEW_RULE_VERSION = 1
# Above this many distinct node classes the whole table is the fingerprint.
@@ -154,38 +173,92 @@ def inspection_review_version(metadata: Any = None) -> int | None:
"""The ``resolver_version`` of an item recorded from the inspection profile.
Folds the profile with its budgets, ``RESOLVER_VERSION``,
``INSPECTION_RESOLVER_VERSION`` and ``semantics_version(metadata)``, so
a change of either resolver, of the profile's bounds or of the table
about the graph's classes moves it. None when the table cannot be loaded.
``INSPECTION_RESOLVER_VERSION``, ``REVIEW_RULE_VERSION`` and
``semantics_version(metadata)``, so a change of either resolver, of how
review uses the profile, of the profile's bounds or of the table about
the graph's classes moves it. None when the table cannot be loaded.
"""
table = semantics_version(metadata)
if table is None:
return None
profile = repr(astuple(INSPECTION_PROFILE))
seed = f"{profile}:{RESOLVER_VERSION}:{INSPECTION_RESOLVER_VERSION}:{table}"
if REVIEW_RULE_VERSION: # 0 is the seed items had before the rule existed
seed += f":review-rule-{REVIEW_RULE_VERSION}"
folded = hashlib.sha256(seed.encode("utf-8")).hexdigest()
return INSPECTION_VERSION_BASE + int(folded[:_INSPECTION_VERSION_HEX], 16)
def resolve_inspection(metadata: dict) -> dict:
"""``metadata`` resolved with the inspection profile (bounded work)."""
return resolve_prompt_pair(metadata, profile=INSPECTION_PROFILE)
def resolve_inspection(metadata: dict, file_name: str | None = None) -> dict:
"""``metadata`` resolved with the inspection profile (bounded work).
``file_name`` (the file's last path part) anchors the file on the one
saver that wrote it, as in the workspace inspector (#390).
"""
return resolve_prompt_pair(
metadata, profile=INSPECTION_PROFILE, file_name=file_name
)
def inspection_choice(metadata: dict) -> dict | None:
"""The inspection profile's ambiguous result when it offers a choice.
def inspection_choice(metadata: dict, file_name: str | None = None) -> dict | None:
"""The inspection profile's result when the file is a review case.
The scan asks only for a file the default profile found ambiguous
without a choice that nothing links.
without a choice that nothing links. The result is ambiguous with a
choice, or ``resolved`` when ``file_name`` anchors the file on its saver
and only that anchor resolves it (``saver_anchored``): the item then
offers the anchor's pair. None otherwise.
"""
if inspection_review_version(metadata) is None:
return None
resolution = resolve_inspection(metadata)
resolution = resolve_inspection(metadata, file_name)
if resolution.get("state") == "resolved":
return _anchor_of(metadata, file_name, resolution)
if resolution.get("state") != "ambiguous" or not reviewable(resolution):
return None
return resolution
def _anchor_of(metadata: Any, file_name: str | None, named: dict) -> dict | None:
"""``named`` tagged ``anchor: "saver"`` when only the saver anchor
resolves the file, to a pair with a positive text to choose; else None.
``named`` is ``metadata`` resolved with ``file_name``; the profile runs
once more without the name only when it is resolved and choosable.
"""
if not file_name or named.get("state") != "resolved":
return None
try:
offered = choosable(_projection(named))
except (KeyError, TypeError):
return None
if not offered or resolve_inspection(metadata).get("state") == "resolved":
return None
return {**named, "anchor": SAVER_ANCHOR}
def saver_anchored(metadata: Any, file_name: str | None) -> bool:
"""True when the saver that wrote ``file_name`` resolves its branch (#390).
The inspection profile resolves ``metadata`` with the file's name, and
without it the file does not resolve: the anchor alone decides it. A file
the default profile left ambiguous is then queued with the anchor's pair
as its one candidate. No name, an unusable table, no matching saver,
several, or an unfinished walk anchor nothing (False). The profile
without the name runs only when the named run resolved.
"""
return _anchored(metadata, file_name) is not None
def _anchored(metadata: Any, file_name: str | None) -> dict | None:
"""The anchored result ``saver_anchored`` describes, else None."""
if not file_name or not isinstance(metadata, dict):
return None
if inspection_review_version(metadata) is None:
return None
return _anchor_of(metadata, file_name, resolve_inspection(metadata, file_name))
def item_profile(evidence: Any) -> str:
"""The profile an item's stored evidence was built with."""
if isinstance(evidence, dict) and evidence.get("profile") == INSPECTION:
@@ -246,7 +319,9 @@ def record_ambiguous_file(
"""
versions = item_versions(scanned[0] if scanned is not None else None)
spelled_path, queued = _queued_spelling(db, root, relative, versions)
if queued is not None:
if queued is not None and not (
inspection and _default_item_to_anchor(db, queued, scanned, relative)
):
return _already_queued(db, *queued, versions)
epoch = db.get_execution_epoch()
try:
@@ -272,24 +347,24 @@ def record_ambiguous_file(
if resolution is None or resolution.get("state") != "ambiguous":
return "skipped"
profile, version = DEFAULT, RESOLVER_VERSION
if not reviewable(resolution):
version = inspection_review_version(captured[0]) if inspection else None
if version is None:
named = _captured_inspection(captured[0], relative, scanned) if inspection else None
anchored = named is not None and named[0].get("anchor") == SAVER_ANCHOR
if anchored:
# Only the saver anchor resolves it (#390): offer that one pair.
profile, version = INSPECTION, named[1]
resolution = {
"state": "ambiguous",
"reason": ANCHOR_REASON,
"candidates": [named[0]],
}
elif not reviewable(resolution):
if named is None:
return "untraced"
profile = INSPECTION
if (
scanned is not None
and scanned[1] is not None
and _same_records(captured[0], scanned[0])
):
resolution = scanned[1] # the same records: no second run
else:
# Heavier, so only for a file ambiguous with nothing to choose.
resolution = resolve_inspection(captured[0])
profile, (resolution, version) = INSPECTION, named
if resolution.get("state") != "ambiguous" or not reviewable(resolution):
return "untraced"
evidence, count, limited = review_evidence(
resolution, profile, version, captured[0]
resolution, profile, version, captured[0], anchored=anchored
)
try:
inserted = db.record_review_item(
@@ -313,6 +388,60 @@ def record_ambiguous_file(
return "limited" if limited else "recorded"
def _default_item_to_anchor(db, queued: tuple, scanned, relative: str) -> bool:
"""True when the bytes' only pending item is a default item recorded
before #390 and the saver anchor alone resolves the scanned records.
The file is then captured again so the anchored item supersedes that
pending, undecided item. Decided bytes and bytes already queued under
the current inspection version are never re-read.
"""
if scanned is None or scanned[0] is None:
return False
version = inspection_review_version(scanned[0])
if version is None or db.review_pending_at(*queued, version):
return False
if not db.review_pending_at(*queued, RESOLVER_VERSION):
return False
return _anchored(scanned[0], _file_name(relative)) is not None
def _captured_inspection(metadata: dict, relative: str, scanned) -> tuple | None:
"""``(resolution, item_version)`` of the captured records under the
inspection profile, anchored on the file's name; None when the table is
unusable.
A result tagged ``anchor: "saver"`` means the saver anchor alone
resolves the file (#390); any other resolved result is returned as
``unresolved`` (no review case, as before). The scan's result is reused
when the capture holds the same records (``inspection_choice`` applies
the same rule).
"""
version = inspection_review_version(metadata)
if version is None:
return None
if (
scanned is not None
and scanned[1] is not None
and _same_records(metadata, scanned[0])
):
return scanned[1], version # the same records: no second run
name = _file_name(relative)
resolution = resolve_inspection(metadata, name)
if resolution.get("state") == "resolved":
# Resolved without the anchor too: no review case, as before.
resolution = _anchor_of(metadata, name, resolution) or {
**resolution,
"state": "unresolved",
}
return resolution, version
def _file_name(relative: str) -> str:
"""The last part of a ``/``-separated relative path (the saved name)."""
return relative.rsplit("/", 1)[-1]
def _already_queued(db, locator: str, size: int, mtime_ns: int, versions) -> str:
"""``duplicate`` while an existing item is pending, else ``decided``."""
for version in versions:
@@ -354,14 +483,19 @@ def _queued_spelling(db, root, relative: str, versions) -> tuple[str, tuple | No
return spelled, None
def captured_review(store, capture, profile: str = DEFAULT) -> tuple[dict, dict] | None:
def captured_review(
store, capture, profile: str = DEFAULT, file_name: str | None = None
) -> tuple[dict, dict] | None:
"""``(metadata, resolution)`` of the capture's own bytes, else None.
``metadata`` is what the scan hands the ingester: prompt and workflow
JSON-decoded and ``file_info`` (size, dimensions, format) from the
capture. ``resolution`` comes from ``profile`` (``DEFAULT`` or
``INSPECTION``). None when the capture is gone or carries no prompt
metadata.
``INSPECTION``; the latter anchored on ``file_name``, the file's own
name, as the item was recorded). With ``file_name`` (Apply), a default
result its saver anchor alone resolves (``saver_anchored``) is that
anchored result, so only the anchored branch is still offered. None
when the capture is gone or carries no prompt metadata.
"""
if store.acquire(capture.token) is None:
return None
@@ -389,8 +523,15 @@ def captured_review(store, capture, profile: str = DEFAULT) -> tuple[dict, dict]
info["format"] = read.format
canonical["file_info"] = info
if profile == INSPECTION:
return canonical, resolve_inspection(canonical)
return canonical, resolve_prompt_pair(canonical)
named = resolve_inspection(canonical, file_name)
return canonical, _anchor_of(canonical, file_name, named) or named
resolution = resolve_prompt_pair(canonical)
if file_name is not None and resolution.get("state") == "ambiguous":
anchored = _anchored(canonical, file_name)
if anchored is not None:
# Its saver names its branch (#390): only that one is offered.
return canonical, anchored
return canonical, resolution
def reviewable(resolution: Any) -> bool:
@@ -416,12 +557,14 @@ def review_evidence(
profile: str = DEFAULT,
version: int | None = None,
metadata: Any = None,
anchored: bool = False,
) -> tuple[dict, int, bool]:
"""``(evidence, candidate_count, limited)`` for one ambiguous result.
Inspection evidence also names its ``profile`` and ``semantics``, the
table fingerprint of ``metadata``'s graph it was resolved with;
``version`` is the item's version.
``version`` is the item's version. ``anchored`` evidence (the saver
anchor's one pair, #390) also names ``anchor: "saver"``.
"""
base = {
"resolver_version": RESOLVER_VERSION if version is None else version,
@@ -430,6 +573,8 @@ def review_evidence(
if profile == INSPECTION:
base["profile"] = INSPECTION
base["semantics"] = semantics_version(metadata)
if anchored:
base["anchor"] = SAVER_ANCHOR
candidates: list[dict] = []
seen: set[str] = set()
total = 0
+29 -481
View File
@@ -164,26 +164,31 @@
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Export
</button>
<button id="settingsBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
<a id="settingsBtn" href="workspace#/system/overview" target="_blank" rel="noopener"
title="Opens System › Overview in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Settings
</button>
<button id="diagnosticsBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
</a>
<a id="diagnosticsBtn" href="workspace#/system/diagnostics" target="_blank" rel="noopener"
title="Opens System › Diagnostics &amp; logs in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Diagnostics
</button>
<button id="maintenanceBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
</a>
<a id="maintenanceBtn" href="workspace#/system/maintenance" target="_blank" rel="noopener"
title="Opens System › Maintenance in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Maintenance
</button>
<button id="backupBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
</a>
<a id="backupBtn" href="workspace#/system/backups" target="_blank" rel="noopener"
title="Opens System › Backup &amp; export in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Backup DB
</button>
<button id="restoreBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
</a>
<a id="restoreBtn" href="workspace#/system/backups" target="_blank" rel="noopener"
title="Opens System › Backup &amp; export in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Restore DB
</button>
</a>
<button id="scanBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Scan Images
@@ -192,10 +197,11 @@
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Auto Tag
</button>
<button id="logsBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
<a id="logsBtn" href="workspace#/system/diagnostics" target="_blank" rel="noopener"
title="Opens System › Diagnostics &amp; logs in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
View Logs
</button>
</a>
<button id="metadataBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Metadata
@@ -229,11 +235,12 @@
</select>
<span class="text-xs text-pm-secondary">per page</span>
</div>
<!-- Session-only toggle; the saved default lives in Settings -->
<!-- Session-only toggle; the saved default lives in Page options -->
<label class="flex items-center gap-1.5 text-xs text-pm-secondary cursor-pointer" title="Load the next page as you scroll (this session)">
<input type="checkbox" id="infiniteScrollSession" class="w-3.5 h-3.5 rounded border-pm bg-pm-input text-pm-accent focus:ring-pm-accent">
Infinite scroll
</label>
<button id="pageOptionsBtn" type="button" class="text-xs text-pm-secondary hover:text-pm-accent underline">Page options</button>
</div>
<span class="text-xs font-medium text-pm-secondary bg-pm-input px-2 py-0.5 rounded-full" id="resultsCount">Loading...</span>
</div>
@@ -367,61 +374,13 @@
<!-- Settings Modal -->
<div id="settingsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-lg w-full mx-4 border border-pm max-h-[90vh] overflow-y-auto shadow-pm">
<h3 class="text-sm font-semibold text-pm mb-4">Settings</h3>
<h3 class="text-sm font-semibold text-pm mb-4">Page options</h3>
<div class="space-y-4">
<!-- Gallery Settings Section -->
<div class="border-b border-pm pb-3">
<h4 class="text-xs font-semibold text-pm-secondary mb-3 uppercase tracking-wide">Gallery Settings</h4>
<h4 class="text-xs font-semibold text-pm-secondary mb-3 uppercase tracking-wide">Prompt list</h4>
<div class="mb-3">
<label class="block text-xs font-medium text-pm-secondary mb-1">Image Scan Directories</label>
<div id="scanPathsList" class="space-y-2">
<!-- Populated dynamically by JS -->
</div>
<button id="addScanPathBtn" type="button"
class="mt-2 flex items-center gap-1 text-xs text-pm-accent hover:text-pm-accent-hover">
<svg class="w-3.5 h-3.5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 4v16m8-8H4"/>
</svg>
Add Directory
</button>
<p class="text-xs text-pm-muted mt-1">Directories to scan for images. Leave empty to use ComfyUI default output directory.</p>
</div>
<div class="mb-3">
<div class="flex items-center justify-between">
<label class="block text-xs font-medium text-pm-secondary">Currently Monitoring</label>
<button id="refreshMonitoringStatus" class="text-xs text-pm-accent hover:text-pm-accent-hover">Refresh</button>
</div>
<div id="monitoringStatus" class="mt-1.5 px-2.5 py-1.5 bg-pm-input rounded-pm-sm text-xs text-pm-secondary font-mono">
Loading...
</div>
</div>
</div>
<!-- UI Settings Section -->
<div class="border-b border-pm pb-3">
<h4 class="text-xs font-semibold text-pm-secondary mb-3 uppercase tracking-wide">UI Settings</h4>
<div class="mb-3">
<label class="block text-xs font-medium text-pm-secondary mb-1">Search Results Auto-Hide (seconds)</label>
<input type="number" id="resultTimeout" min="0" max="300" value="30"
class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<p class="text-xs text-pm-muted mt-1">Set to 0 to disable auto-hide.</p>
</div>
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Web UI Display Mode</label>
<select id="webuiDisplayMode"
class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="popup">Popup Window (Current)</option>
<option value="newtab">New Tab</option>
</select>
<p class="text-xs text-pm-muted mt-1">Choose how the Web UI opens from ComfyUI nodes.</p>
</div>
<div class="mt-3 flex items-center justify-between">
<div class="flex items-center justify-between">
<div>
<label for="infiniteScrollDefault" class="block text-xs font-medium text-pm-secondary">Infinite Scrolling</label>
<p class="text-xs text-pm-muted">Load the next page of prompts as you scroll. The checkbox in the list header overrides this for the current session.</p>
@@ -429,76 +388,6 @@
<input type="checkbox" id="infiniteScrollDefault" class="w-4 h-4 rounded border-pm bg-pm-input text-pm-accent focus:ring-pm-accent ml-3">
</div>
</div>
<!-- Performance Section -->
<div class="border-b border-pm pb-3">
<h4 class="text-xs font-semibold text-pm-secondary mb-3 uppercase tracking-wide">Performance</h4>
<div>
<div class="flex items-center justify-between mb-1.5">
<label for="workerThreads" class="text-xs font-medium text-pm-secondary">Worker threads</label>
<span id="workerThreadsValue" class="text-pm-accent text-xs font-mono">1 / 1 cores</span>
</div>
<input type="range" id="workerThreads" min="1" max="1" step="1" value="1"
class="w-full h-1.5 bg-pm-primary rounded-lg appearance-none cursor-pointer accent-[var(--pm-accent)]">
<p class="text-xs text-pm-muted mt-1">Threads used by the image rescan and thumbnail generation. More threads finish sooner; fewer leave CPU for ComfyUI while it generates.</p>
</div>
</div>
<!-- Integrations Section -->
<div class="border-b border-pm pb-3">
<h4 class="text-xs font-semibold text-pm-secondary mb-3 uppercase tracking-wide">Integrations</h4>
<div class="mb-3">
<div class="flex items-center justify-between mb-2">
<div class="flex items-center gap-2">
<label class="block text-xs font-medium text-pm-secondary">LoraManager</label>
<span id="loraDetectionBadge" class="text-[10px] px-1.5 py-0.5 rounded-full bg-pm-input text-pm-muted">checking...</span>
</div>
<label class="relative inline-flex items-center cursor-pointer">
<input type="checkbox" id="loraEnabled" class="sr-only peer" disabled>
<div class="w-8 h-4 bg-pm-input rounded-full peer peer-checked:bg-pm-accent transition-colors pointer-events-none after:content-[''] after:absolute after:top-0.5 after:left-0.5 after:bg-white after:rounded-full after:h-3 after:w-3 after:transition-all peer-checked:after:translate-x-4"></div>
</label>
</div>
<div id="loraSettings" class="hidden space-y-2 mt-2 pl-2 border-l-2 border-pm">
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Path</label>
<input type="text" id="loraManagerPath" readonly
class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] text-pm-muted">
</div>
<div class="flex items-center justify-between">
<label class="block text-xs font-medium text-pm-secondary">Auto-inject trigger words</label>
<label class="relative inline-flex items-center cursor-pointer">
<input type="checkbox" id="loraTriggerWords" class="sr-only peer">
<div class="w-8 h-4 bg-pm-input rounded-full peer peer-checked:bg-pm-accent transition-colors pointer-events-none after:content-[''] after:absolute after:top-0.5 after:left-0.5 after:bg-white after:rounded-full after:h-3 after:w-3 after:transition-all peer-checked:after:translate-x-4"></div>
</label>
</div>
<p class="text-xs text-pm-muted">Automatically append trigger words when <code class="text-pm-accent">&lt;lora:name:weight&gt;</code> is detected in prompts.</p>
<div class="mt-2">
<label class="block text-xs font-medium text-pm-secondary mb-1">CivitAI API Key</label>
<input type="password" id="civitaiApiKey" placeholder="Optional — required for NSFW example images"
class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] placeholder:text-pm-muted focus:outline-none">
<p class="text-xs text-pm-muted mt-1">Get your key from <a href="https://civitai.com/user/account" target="_blank" class="text-pm-accent hover:underline">civitai.com/user/account</a></p>
</div>
<label class="mt-1 flex items-center gap-1.5 text-xs text-pm-secondary cursor-pointer">
<input type="checkbox" id="clearCivitaiKey" class="w-3.5 h-3.5 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
<span>Clear the stored key on save</span>
</label>
<button id="loraImportBtn" type="button"
class="mt-1 flex items-center gap-1 text-xs text-pm-accent hover:text-pm-accent-hover">
<svg class="w-3.5 h-3.5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-8l-4-4m0 0L8 8m4-4v12"/>
</svg>
Import LoRA Data
</button>
</div>
</div>
</div>
</div>
<div class="flex justify-end gap-2 mt-4">
@@ -514,32 +403,6 @@
</div>
</div>
<!-- LoRA Import Progress Modal -->
<div id="loraImportModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-md w-full mx-4 border border-pm shadow-pm">
<h3 class="text-sm font-semibold text-pm mb-4">Importing LoRA Data</h3>
<div class="mb-3">
<p class="text-pm-secondary text-xs">Status:</p>
<p id="loraImportStatus" class="text-pm text-xs font-medium truncate">Initializing...</p>
</div>
<div class="mb-3">
<div class="flex justify-between text-xs text-pm-secondary mb-1">
<span>Progress</span>
<span id="loraImportPercent">0%</span>
</div>
<div class="w-full bg-pm-input rounded-full h-1.5">
<div id="loraImportBar" class="bg-pm-accent h-1.5 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<div class="flex justify-between text-[11px] text-pm-muted mt-1">
<span id="loraImportProcessed">0 processed</span>
<span id="loraImportImported">0 imported</span>
</div>
</div>
</div>
</div>
<!-- Bulk Tag Modal -->
<div id="bulkTagModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-md w-full mx-4 border border-pm shadow-pm">
@@ -612,167 +475,6 @@
</div>
</div>
<!-- Database Restore Modal -->
<div id="restoreModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-md w-full mx-4 border border-pm shadow-pm">
<h3 class="text-sm font-semibold text-pm mb-4">Restore Database</h3>
<div class="mb-3">
<div class="bg-pm-warning-tint border border-pm-warning/40 rounded-pm-md p-3 mb-3">
<h4 class="text-pm-warning font-medium text-xs">Warning</h4>
<p class="text-pm text-xs mt-1">This will replace your current database. A backup will be created automatically.</p>
</div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Select Database File (.db)</label>
<input type="file" id="restoreFileInput" accept=".db"
class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] file:mr-3 file:py-1 file:px-3 file:rounded-pm-sm file:border-0 file:text-xs file:font-medium file:bg-pm-accent file:text-pm-accent-fg">
</div>
<div class="flex justify-end gap-2 mt-4">
<button id="cancelRestore"
class="px-3 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Cancel
</button>
<button id="confirmRestore"
class="px-3 py-1.5 bg-pm-warning hover:opacity-80 text-black text-xs font-medium rounded-pm-sm transition-colors">
Restore Database
</button>
</div>
</div>
</div>
<!-- Diagnostics Modal -->
<div id="diagnosticsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
<div class="p-4 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">Gallery System Diagnostics</h3>
<button onclick="window.admin.closeDiagnostics()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
</svg>
</button>
</div>
</div>
<div class="p-4 overflow-y-auto max-h-[calc(90vh-120px)]">
<div class="flex items-center justify-between mb-3">
<button id="runDiagnosticsBtn" class="px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Run Diagnostics
</button>
<button id="testImageLinkBtn" class="px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Test Image Link
</button>
</div>
<div id="diagnosticsContent" class="space-y-3">
<div class="text-center py-6 text-pm-secondary text-xs">
Click "Run Diagnostics" to check the gallery system status
</div>
</div>
</div>
</div>
</div>
<!-- Maintenance Modal -->
<div id="maintenanceModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
<div class="p-4 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">Database Maintenance</h3>
<button onclick="window.admin.closeMaintenance()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
</svg>
</button>
</div>
</div>
<div class="p-4 overflow-y-auto max-h-[calc(90vh-120px)]">
<!-- Maintenance Options -->
<div class="space-y-3 mb-4">
<h4 class="text-xs font-semibold text-pm-secondary uppercase tracking-wide mb-3">Select Maintenance Operations:</h4>
<div class="grid grid-cols-1 md:grid-cols-2 gap-3">
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="cleanup_duplicates" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]" checked>
<div class="flex-1">
<div class="text-pm text-xs font-medium">Remove Duplicates</div>
<div class="text-[11px] text-pm-secondary">Find and remove duplicate prompts based on text content</div>
</div>
</label>
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="vacuum" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]" checked>
<div class="flex-1">
<div class="text-pm text-xs font-medium">Vacuum Database</div>
<div class="text-[11px] text-pm-secondary">Optimize database file size and performance</div>
</div>
</label>
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="cleanup_orphaned_images" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]" checked>
<div class="flex-1">
<div class="text-pm text-xs font-medium">Clean Orphaned Images</div>
<div class="text-[11px] text-pm-secondary">Remove image records for files that no longer exist</div>
</div>
</label>
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="check_hash_duplicates" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]">
<div class="flex-1">
<div class="text-pm text-xs font-medium">Check Hash Duplicates</div>
<div class="text-[11px] text-pm-secondary">Identify prompts with duplicate hash values</div>
</div>
</label>
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="check_consistency" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]">
<div class="flex-1">
<div class="text-pm text-xs font-medium">Consistency Check</div>
<div class="text-[11px] text-pm-secondary">Verify database integrity and data consistency</div>
</div>
</label>
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="statistics" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]">
<div class="flex-1">
<div class="text-pm text-xs font-medium">Database Statistics</div>
<div class="text-[11px] text-pm-secondary">Generate detailed database statistics report</div>
</div>
</label>
<label class="flex items-center gap-2.5 p-3 bg-pm-input rounded-pm-md hover:bg-pm-hover cursor-pointer">
<input type="checkbox" id="prune_orphaned_prompts" class="maintenance-option w-3.5 h-3.5 accent-[var(--pm-accent)]">
<div class="flex-1">
<div class="text-pm text-xs font-medium">Prune Orphaned Prompts</div>
<div class="text-[11px] text-pm-secondary">Remove prompts that have no linked images</div>
</div>
</label>
</div>
</div>
<!-- Action Buttons -->
<div class="flex items-center justify-between mb-4">
<div class="flex gap-2">
<button id="runMaintenanceBtn" class="px-3 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Run Maintenance
</button>
<button id="selectAllMaintenanceBtn" class="px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Select All
</button>
<button id="clearAllMaintenanceBtn" class="px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Clear All
</button>
</div>
</div>
<!-- Results -->
<div id="maintenanceResults" class="space-y-3">
<div class="text-center py-6 text-pm-secondary text-xs">
Click "Run Maintenance" to perform selected operations
</div>
</div>
</div>
</div>
</div>
<!-- Gallery Modal -->
<div id="galleryModal" class="fixed inset-0 bg-black/75 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-6xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
@@ -864,130 +566,6 @@
<span class="inline-block w-1.5 h-1.5 rounded-full bg-pm-accent mr-1.5 align-middle animate-pulse"></span><span id="scanPillText" class="align-middle">Scanning...</span>
</button>
<!-- Logs Modal -->
<div id="logsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-6xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
<div class="p-4 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">System Logs</h3>
<button onclick="window.admin.closeLogs()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
</svg>
</button>
</div>
</div>
<!-- Log Controls -->
<div class="p-4 border-b border-pm">
<div class="grid grid-cols-1 md:grid-cols-5 gap-3 mb-3">
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Log Level</label>
<select id="logLevel" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="">All Levels</option>
<option value="DEBUG">Debug</option>
<option value="INFO">Info</option>
<option value="WARNING">Warning</option>
<option value="ERROR">Error</option>
<option value="CRITICAL">Critical</option>
</select>
</div>
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Limit</label>
<select id="logLimit" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="50">Last 50</option>
<option value="100" selected>Last 100</option>
<option value="200">Last 200</option>
<option value="500">Last 500</option>
</select>
</div>
<div class="flex items-end">
<button id="refreshLogsBtn" class="w-full px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Refresh
</button>
</div>
<div class="flex items-end">
<button id="downloadLogsBtn" class="w-full px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Download
</button>
</div>
<div class="flex items-end">
<button id="clearLogsBtn" class="w-full px-2.5 py-1.5 text-pm-error bg-transparent hover:bg-pm-error-tint border border-pm text-xs font-medium rounded-pm-sm transition-colors">
Clear Logs
</button>
</div>
</div>
<!-- Log Stats -->
<div class="grid grid-cols-2 md:grid-cols-5 gap-3 mb-3">
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Total Logs</div>
<div class="text-sm font-semibold text-pm-accent" id="logStatsTotal">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Errors</div>
<div class="text-sm font-semibold text-pm-error" id="logStatsErrors">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Warnings</div>
<div class="text-sm font-semibold text-pm-warning" id="logStatsWarnings">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Current Level</div>
<div class="text-sm font-semibold text-pm-success" id="logStatsLevel">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Log Size</div>
<div class="text-sm font-semibold text-pm" id="logStatsSize">-</div>
</div>
</div>
<!-- Log Settings -->
<div class="grid grid-cols-1 md:grid-cols-3 gap-3">
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Set Log Level</label>
<select id="setLogLevel" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="DEBUG">Debug</option>
<option value="INFO" selected>Info</option>
<option value="WARNING">Warning</option>
<option value="ERROR">Error</option>
<option value="CRITICAL">Critical</option>
</select>
</div>
<div class="flex items-center">
<label class="flex items-center gap-1.5 text-pm">
<input type="checkbox" id="consoleLogging" checked class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
<span class="text-xs">Console Logging</span>
</label>
</div>
<div class="flex items-end">
<button id="updateLogConfigBtn" class="w-full px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Update Config
</button>
</div>
</div>
</div>
<!-- Log Viewer -->
<div class="flex-1 overflow-hidden">
<div class="h-96 overflow-y-auto bg-pm-primary font-mono text-xs">
<div id="logsContainer" class="p-3 space-y-0.5">
<div class="text-center py-6 text-pm-muted text-xs">
Click "Refresh" to load logs
</div>
</div>
</div>
</div>
<!-- Log Files -->
<div class="p-4 border-t border-pm">
<h4 class="text-xs font-semibold text-pm-secondary uppercase tracking-wide mb-3">Log Files</h4>
<div id="logFilesList" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-2">
</div>
</div>
</div>
</div>
<!-- Add Prompt Modal -->
<div id="addPromptModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-2xl w-full mx-4 border border-pm max-h-[90vh] overflow-y-auto shadow-pm">
@@ -1410,36 +988,6 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
</div>
</div>
<!-- Auto Tag Download Modal -->
<div id="autoTagDownloadModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-md w-full mx-4 border border-pm shadow-pm">
<h3 class="text-sm font-semibold text-pm mb-4">Downloading Model</h3>
<div class="mb-3">
<p class="text-pm-secondary text-xs">Downloading:</p>
<p id="downloadModelName" class="text-pm text-xs font-medium">GGUF Model</p>
</div>
<div class="mb-4">
<div class="flex justify-between text-xs text-pm-secondary mb-1">
<span id="downloadStatus">Preparing...</span>
<span id="downloadProgressPercent">0%</span>
</div>
<div class="w-full bg-pm-input rounded-full h-1.5">
<div id="downloadProgressBar" class="bg-pm-accent h-1.5 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<p class="text-pm-secondary text-xs mt-2">The download continues in Activity.</p>
</div>
<div class="flex justify-center">
<button id="cancelDownloadBtn"
class="px-3 py-1.5 text-pm-error bg-transparent hover:bg-pm-error-tint border border-pm text-xs font-medium rounded-pm-sm transition-colors">
Close
</button>
</div>
</div>
</div>
<script src="js/comfy-metadata.js"></script>
<script src="js/prompt-list-sort.js"></script>
<script src="js/view-router.js"></script>
+27 -249
View File
@@ -8,8 +8,6 @@
<script src="js/html-escape.js"></script>
<script src="js/image-helpers.js"></script>
<script src="js/gallery-settings.js"></script>
<script src="js/cache-ownership.js"></script>
<script src="js/scan-progress.js"></script>
<script src="js/api-client.js"></script>
<script src="js/png-metadata.js"></script>
<script src="js/list-state.js"></script>
@@ -63,7 +61,7 @@
<button id="thumbMdBtn" class="px-2 py-1 bg-pm-accent text-pm-accent-fg text-xs rounded-pm-sm transition-colors" title="Medium thumbnails">M</button>
<button id="thumbLgBtn" class="px-2 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs rounded-pm-sm transition-colors" title="Large thumbnails">L</button>
</div>
<div class="flex items-center gap-2">
<div class="flex flex-wrap items-center gap-2">
<button id="refreshBtn"
class="px-3 py-1 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Refresh
@@ -76,9 +74,14 @@
class="px-3 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
List View
</button>
<button id="settingsBtn"
class="px-3 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
<a id="settingsBtn" href="workspace#/system/appearance" target="_blank" rel="noopener"
title="Opens System › Appearance in a new tab"
class="px-3 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Settings
</a>
<button id="pageOptionsBtn" type="button"
class="px-3 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Page options
</button>
<button id="autoTagBtn"
class="px-3 py-1 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
@@ -176,7 +179,7 @@
<div class="bg-pm-surface rounded-pm-md max-w-2xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm">
<div class="p-3 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">Gallery Settings</h3>
<h3 class="text-sm font-semibold text-pm">Page options</h3>
<button id="closeSettingsBtn" class="p-2 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
@@ -190,25 +193,9 @@
<div class="space-y-3">
<h4 class="text-sm font-semibold text-pm">Performance Settings</h4>
<!-- Thumbnail Generation -->
<div class="bg-pm-input rounded-pm-sm p-3">
<div class="flex items-center justify-between mb-2">
<div>
<h5 class="text-pm text-xs font-medium">Generate Thumbnails</h5>
<p class="text-xs text-pm-secondary">Create optimized thumbnails for faster loading</p>
</div>
<button id="generateThumbnailsBtn" class="px-3 py-1 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs rounded-pm-sm transition-colors">
Generate
</button>
</div>
<div class="text-xs text-pm-muted mb-2">
Status: <span id="thumbnailStatus">Not generated</span>
<div class="text-xs text-pm-muted mt-1">
Thumbnails speed up gallery loading. Original images always used for viewing and metadata.
</div>
</div>
<div class="bg-pm-input rounded-pm-sm p-3 space-y-2">
<!-- Check thumbnails at startup -->
<div class="flex items-center justify-between py-2 border-t border-pm">
<div class="flex items-center justify-between">
<div>
<label class="text-xs text-pm-secondary">Check thumbnails at startup</label>
<p class="text-xs text-pm-muted">Prompt to generate thumbnails if missing when opening gallery</p>
@@ -218,43 +205,9 @@
<div class="w-11 h-6 bg-pm-input peer-focus:outline-none peer-focus:ring-4 peer-focus:ring-pm-accent rounded-full peer peer-checked:after:translate-x-full peer-checked:after:border-white after:content-[''] after:absolute after:top-[2px] after:left-[2px] after:bg-white after:border-pm after:border after:rounded-full after:h-5 after:w-5 after:transition-all peer-checked:bg-pm-accent"></div>
</label>
</div>
<!-- Progress bar (hidden by default) -->
<div id="thumbnailProgress" class="hidden">
<div class="flex items-center justify-between text-xs text-pm-secondary mb-1">
<span id="progressText">Processing...</span>
<div class="flex items-center gap-2">
<span id="progressPercent">0%</span>
<button id="cancelThumbnailsBtn" class="text-pm-error hover:text-pm-error text-xs px-2 py-1 bg-pm-error-tint rounded-pm-sm hidden">
Cancel
</button>
</div>
</div>
<div class="w-full bg-pm-input rounded-full h-2">
<div id="progressBar" class="bg-pm-accent h-2 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<div class="flex items-center justify-between text-xs text-pm-muted mt-1">
<span id="progressDetails">Scanning images...</span>
<span id="progressETA" class="text-pm-muted"></span>
</div>
</div>
</div>
<!-- Image Loading -->
<div class="bg-pm-input rounded-pm-sm p-3">
<h5 class="text-pm text-xs font-medium mb-2">Image Loading</h5>
<div class="space-y-2">
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Lazy Loading</label>
<input type="checkbox" id="lazyLoadingToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Image Quality</label>
<select id="imageQualitySelect" class="px-2 py-1 bg-pm-input border border-pm rounded-pm-sm text-pm text-xs">
<option value="high">High (Original)</option>
<option value="medium" selected>Medium (Optimized)</option>
<option value="low">Low (Fast)</option>
</select>
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Lazy Loading</label>
<input type="checkbox" id="lazyLoadingToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
</div>
</div>
@@ -264,13 +217,6 @@
<h4 class="text-sm font-semibold text-pm">Display Settings</h4>
<div class="bg-pm-input rounded-pm-sm p-3 space-y-2">
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Default View Mode</label>
<select id="defaultViewModeSelect" class="px-2 py-1 bg-pm-input border border-pm rounded-pm-sm text-pm text-xs">
<option value="grid" selected>Grid View</option>
<option value="list">List View</option>
</select>
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Items per Page</label>
<select id="defaultLimitSelect" class="px-2 py-1 bg-pm-input border border-pm rounded-pm-sm text-pm text-xs">
@@ -289,14 +235,6 @@
<option value="10">10 Columns</option>
</select>
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Show Media Info</label>
<input type="checkbox" id="showImageInfoToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Collapse Sidebar by Default</label>
<input type="checkbox" id="sidebarCollapsedByDefaultToggle" class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<div>
<label class="text-xs text-pm-secondary">Infinite Scrolling</label>
@@ -305,78 +243,6 @@
<input type="checkbox" id="infiniteScrollToggle" class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
</div>
<!-- Video Settings -->
<div class="bg-pm-input rounded-pm-sm p-3 space-y-2">
<h5 class="text-pm text-xs font-medium mb-2">Video Settings</h5>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Autoplay Videos</label>
<input type="checkbox" id="videoAutoplayToggle" class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Mute Videos by Default</label>
<input type="checkbox" id="videoMuteToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Loop Videos</label>
<input type="checkbox" id="videoLoopToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
</div>
</div>
<!-- Metadata Settings -->
<div class="space-y-3">
<h4 class="text-sm font-semibold text-pm">Metadata Settings</h4>
<div class="bg-pm-input rounded-pm-sm p-3 space-y-2">
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Auto-load Metadata</label>
<input type="checkbox" id="autoLoadMetadataToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Cache Metadata</label>
<input type="checkbox" id="cacheMetadataToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
<div class="flex items-center justify-between">
<label class="text-xs text-pm-secondary">Show File Paths</label>
<input type="checkbox" id="showFilePathsToggle" checked class="w-4 h-4 text-pm-accent bg-pm-input border-pm rounded focus:ring-pm-accent">
</div>
</div>
</div>
<!-- Maintenance -->
<div class="space-y-3">
<h4 class="text-sm font-semibold text-pm">Maintenance</h4>
<div class="bg-pm-input rounded-pm-sm p-3 space-y-2">
<div class="flex items-center justify-between">
<div>
<h5 class="text-pm text-xs font-medium">Clear Cache</h5>
<p class="text-xs text-pm-secondary">Clear cached metadata and thumbnails</p>
</div>
<button id="clearCacheBtn" class="px-3 py-1 bg-pm-error hover:bg-pm-error text-pm text-xs rounded-pm-sm transition-colors">
Clear
</button>
</div>
<div class="flex items-center justify-between">
<div>
<h5 class="text-pm text-xs font-medium">Rescan Output Folder</h5>
<p class="text-xs text-pm-secondary">Refresh the image list from disk</p>
</div>
<button id="rescanFolderBtn" class="px-3 py-1 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs rounded-pm-sm transition-colors">
Rescan
</button>
</div>
<div class="flex items-center justify-between">
<div>
<h5 class="text-pm text-xs font-medium">Find Duplicate Images</h5>
<p class="text-xs text-pm-secondary">Scan for duplicate images based on file content</p>
</div>
<button id="scanDuplicatesBtn" class="px-3 py-1 bg-pm-warning hover:bg-pm-warning text-pm text-xs rounded-pm-sm transition-colors">
Scan
</button>
</div>
</div>
</div>
<!-- Advanced Settings -->
@@ -395,6 +261,14 @@
</div>
</div>
<p class="text-xs text-pm-secondary">Thumbnails, cache, file listing and duplicates are in System.
<a id="pageOptionsMaintenanceLink" href="workspace#/system/maintenance" target="_blank" rel="noopener"
title="Opens System › Maintenance in a new tab" class="text-pm-accent hover:underline">Maintenance</a>
·
<a id="pageOptionsSourcesLink" href="workspace#/system/sources" target="_blank" rel="noopener"
title="Opens System › Sources &amp; scans in a new tab" class="text-pm-accent hover:underline">Sources &amp; scans</a>
</p>
</div>
<div class="p-3 border-t border-pm flex justify-end gap-2">
<button id="resetSettingsBtn" class="px-3 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
@@ -407,71 +281,6 @@
</div>
</div>
<!-- Minimized duplicate scan: shown while it runs with its modal hidden, or when results are waiting -->
<button id="dupScanPill" type="button" title="Show duplicate scan"
class="hidden fixed bottom-4 right-4 z-40 px-3 py-1.5 bg-pm-surface border border-pm shadow-pm rounded-pm-md text-xs text-pm hover:bg-pm-hover transition-colors">
<span class="inline-block w-1.5 h-1.5 rounded-full bg-pm-accent mr-1.5 align-middle animate-pulse"></span><span id="dupScanPillText" class="align-middle">Scanning...</span>
</button>
<!-- Duplicates Modal -->
<div id="duplicatesModal" class="fixed inset-0 bg-black/50 hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-md max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm">
<div class="p-3 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">Duplicate Images Found</h3>
<button id="closeDuplicatesBtn" class="p-2 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
</svg>
</button>
</div>
</div>
<div class="p-3 overflow-y-auto max-h-[calc(90vh-200px)]">
<div id="duplicatesScanStatus" class="py-6">
<div class="max-w-md mx-auto bg-pm-input rounded-pm-md p-3">
<div class="flex items-center justify-between mb-1.5">
<span class="text-pm text-xs font-medium">Scanning for duplicate images</span>
<span id="dupProgressText" class="text-pm-secondary text-xs">Starting...</span>
</div>
<div class="w-full bg-pm-primary rounded-full h-1.5">
<div id="dupProgressBar" class="bg-pm-accent h-1.5 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<p class="text-[11px] text-pm-muted mt-2">Every file is hashed, so this takes a while on a large library. You can close this window; the scan keeps running and the results will be waiting.</p>
</div>
</div>
<div id="duplicatesContent" class="hidden">
<div class="mb-3 p-3 bg-pm-error-tint border border-pm-accent rounded-pm-sm">
<div class="flex items-start">
<svg class="w-4 h-4 text-pm-error mt-0.5 mr-2 flex-shrink-0" fill="currentColor" viewBox="0 0 20 20">
<path fill-rule="evenodd" d="M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z" clip-rule="evenodd"></path>
</svg>
<div>
<h4 class="text-pm-error text-xs font-medium">Warning: Removing duplicate images cannot be undone!</h4>
<p class="text-pm-secondary text-xs mt-1">When you remove a duplicate image, the selected file will be permanently deleted from disk. Choose carefully which images to keep.</p>
</div>
</div>
</div>
<div id="duplicatesList" class="space-y-3">
<!-- Duplicates will be populated here -->
</div>
</div>
</div>
<div id="duplicatesFooter" class="hidden p-3 border-t border-pm flex justify-between items-center">
<div class="text-xs text-pm-secondary">
<span id="duplicatesCount">0</span> duplicate image groups found
</div>
<div class="flex gap-2">
<button id="selectAllBtn" class="px-3 py-1 bg-pm-input hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Select All to Delete
</button>
<button id="removeDuplicatesBtn" class="px-3 py-1 bg-pm-error hover:bg-pm-error text-pm text-xs font-medium rounded-pm-sm transition-colors">
Delete Selected
</button>
</div>
</div>
</div>
</div>
<!-- Auto Tag Modal -->
<div id="autoTagModal" class="fixed inset-0 bg-black/50 hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-md p-3 max-w-lg w-full mx-4 border border-pm">
@@ -549,9 +358,11 @@
<span class="text-pm-secondary text-xs ml-2">(Faster processing, uses ~4-16GB VRAM)</span>
</div>
</div>
<button id="unloadModelBtn" class="px-3 py-1 bg-pm-error hover:bg-pm-error text-pm text-xs rounded-pm-sm transition-colors hidden">
Unload Model
</button>
<a id="unloadModelBtn" href="workspace#/system/tagging" target="_blank" rel="noopener"
title="Unload the model in System › Auto-tagging (opens a new tab)"
class="px-3 py-1 bg-pm-error hover:bg-pm-error text-pm text-xs rounded-pm-sm transition-colors hidden">
Manage in System
</a>
</label>
<p id="modelLoadedStatus" class="text-pm-muted text-xs mt-1 hidden">
<span class="text-pm-success">●</span> Model currently loaded: <span id="loadedModelType">none</span>
@@ -711,39 +522,6 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
</div>
</div>
<!-- Auto Tag Download Modal -->
<div id="autoTagDownloadModal" class="fixed inset-0 bg-black/50 hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-md p-3 max-w-md w-full mx-4 border border-pm">
<h3 class="text-sm font-semibold text-pm mb-3">Downloading Model</h3>
<!-- Model Name -->
<div class="mb-3">
<p class="text-pm-secondary text-xs">Downloading:</p>
<p id="downloadModelName" class="text-pm text-xs font-medium">GGUF Model</p>
</div>
<!-- Progress Bar -->
<div class="mb-3">
<div class="flex justify-between text-xs text-pm-secondary mb-1">
<span id="downloadStatus">Preparing...</span>
<span id="downloadProgressPercent">0%</span>
</div>
<div class="w-full bg-pm-input rounded-full h-2">
<div id="downloadProgressBar" class="bg-pm-accent h-2 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<p class="text-pm-secondary text-xs mt-2">The download continues in Activity.</p>
</div>
<!-- Cancel Button -->
<div class="flex justify-center">
<button id="cancelDownloadBtn"
class="px-3 py-1 bg-pm-error hover:bg-pm-error text-pm text-xs font-medium rounded-pm-sm transition-colors">
Close
</button>
</div>
</div>
</div>
<!-- Add Prompt Modal -->
<div id="addPromptModal" class="fixed inset-0 bg-black/50 hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-md p-3 max-w-2xl w-full mx-4 border border-pm max-h-[90vh] overflow-y-auto">
+16 -1114
View File
File diff suppressed because it is too large Load Diff
+54 -803
View File
File diff suppressed because it is too large Load Diff
+7 -2
View File
@@ -275,8 +275,13 @@ export function createImageStream({ api, onChange = () => {}, batchSize = 40, re
};
}
/** The concept's count line (app.js a446): "N of M images · K positive prompts", without the count while unknown. */
export function countLine({ loaded, total, positiveCount }) {
/**
* The concept's count line (app.js a446): "N of M images · K positive prompts", without the count while unknown, and
* the page's first "Loading images…" while a first page loads.
*/
export function countLine({ loaded, total, positiveCount, loading = false, error = null }) {
// A first page in flight has no count yet: "0 of 0 images" would claim the filter matched nothing (#391).
if (loading && !loaded && !error) return 'Loading images…';
const images = `${loaded.toLocaleString('en-US')} of ${total.toLocaleString('en-US')} ${noun(total, 'image')}`;
return isCount(positiveCount) ? `${images} · ${positiveCount.toLocaleString('en-US')} ${noun(positiveCount, 'positive prompt')}` : images;
}
+25 -7
View File
@@ -225,6 +225,23 @@ export function clearSentence(files) {
return `Empties ${ACTIVE_LOG}, deletes ${number(rotated.length)} rotated ${noun(rotated.length, 'copy', 'copies')} (${sizeText(size)}) ${tail}`;
}
/**
* #389: the `/logs/truncate` body for the files a confirmation showed. Their active and rotated rows go back with the
* server's `identity`, so the server deletes only those and refuses (409) when any changed meanwhile. A list that could
* not be read (`null`, the general sentence) sends no list: the server then clears what is there, as the sentence says.
*/
export function clearBody(files) {
if (!Array.isArray(files)) return {};
const expected = files.filter(row => (row?.kind === 'active' || row?.kind === 'rotated') && typeof row.filename === 'string'
&& typeof row.identity === 'string').map(row => ({ name: row.filename, identity: row.identity }));
return { expected };
}
/** A refused clear (409) says why in the server's words; any other failure the general line. */
export function clearFailureText(error) {
return error?.status === 409 && typeof error.message === 'string' && error.message ? error.message : COPY.clearFailed;
}
/** CUJ 7: the toast for `/logs/truncate`'s `results` ({truncated: [names], errors: [messages]}). */
export function clearToast(results) {
const truncated = Array.isArray(results?.truncated) ? results.truncated : [];
@@ -469,13 +486,14 @@ function logsSurface({ api, activity, scope, drafts, navigate, notify, latestOnl
const list = el('div', undefined, 'operation-list');
let listed = null;
// The confirmation names what Clear deletes now: the files are read again before it opens (a read that fails
// gives the general sentence, never an old count).
// gives the general sentence, never an old count). Clear sends exactly the files the dialog was built from (#389).
const clear = operationRow({ scope, title: 'Clear all logs', description: COPY.clearDescription, label: 'Clear all logs…',
danger: true, onClick: button => pressing(button, async () => {
await readFiles();
if (scope.signal.aborted) return;
confirmDialog({ scope, trigger: button, title: 'Clear all logs', sentence: clearSentence(listed), label: 'Clear',
danger: true, onConfirm: clearLogs });
const shown = listed;
confirmDialog({ scope, trigger: button, title: 'Clear all logs', sentence: clearSentence(shown), label: 'Clear',
danger: true, onConfirm: () => clearLogs(shown) });
}) });
files.append(el('summary', 'Log files & retention'), line, review, retention, list, clear);
const runtime = runtimeRows({ scope, drafts });
@@ -562,12 +580,12 @@ function logsSurface({ api, activity, scope, drafts, navigate, notify, latestOnl
readConfig();
}
async function clearLogs() {
async function clearLogs(shown) {
let answer;
try {
answer = await api.post(TRUNCATE_URL, {});
} catch {
if (!scope.signal.aborted) notify(COPY.clearFailed);
answer = await api.post(TRUNCATE_URL, clearBody(shown));
} catch (error) {
if (!scope.signal.aborted) notify(clearFailureText(error));
}
if (scope.signal.aborted) return;
if (answer) notify(clearToast(answer.results));