Compare commits

..
Author SHA1 Message Date
Vito Sansevero 54dd5264d9 Merge remote-tracking branch 'origin/main' into fix/391-browser-flakes 2026-10-27 22:37:48 -07:00
Vito d0339128e4 test: isolate API fixture databases from other suites; CI lint pins, failing bandit hook, npm cache (#346, #352, #409) (#411)
* ci: pin black and flake8, let the bandit hook fail, cache npm in the JS job (#352)

The lint job installs black==26.10.1 and flake8==7.4.1 from
requirements-dev.txt (the versions CI resolved on main today), so a new
release cannot turn CI red with no code change.

The local bandit hook ran 'bash -c ... "$@"; true': it could never
fail, and bash bound the first staged file to $0 so it was never
scanned. It now runs 'bandit -lll -q' directly, failing on high
severity exactly like code-quality.yml.

The frontend modules job's setup-node caches npm keyed on
tools/vendor/package-lock.json.

* test(history): wait on the writer's state, not short wall-clock pauses (#352)

Four 0.08 s pauses before asserting that the worker had not probed yet
now wait until the idle worker has run a whole loop pass at the current
manual clock value (three clock reads), so a slow runner cannot end the
pause before the worker decides. A mutation that probes 20 ms early now
fails two tests deterministically.

The real-clock bounded close no longer asserts close(0.02) returned in
under 0.2 s wall time; it asserts the in-flight write is still blocked
and the worker alive, like the output writer test. The manual-clock
bound stays exact. The last 0.1 s non-blocking submit check uses the
1 s NON_BLOCKING_CEILING from #351.

* test(api): give each API fixture database its own temp folder (#346)

Every PromptManagerAPI sweeps its database's folder at startup for the
stages, uploads and safety-backup copies an interrupted restore left.
The API test fixtures created their database directly in the system
temp folder, so any other test process starting an API there (another
worktree's suite, a pre-push hook, an agent's targeted run) deleted
this process's in-flight restore files. The restore, epoch and
retirement tests then failed with 'staged restore changed', 'unable to
open database file' or 503 instead of 409. Busy machines fail because
they are running other suites, not because they are slow: the same
CPU load with a private TMPDIR passes.

private_temp_db() puts the fixture database in a new folder removed at
cleanup (with any safety backups a restore left beside it). A
regression test sweeps a stand-in shared folder in the middle of a
multipart restore, as another suite's startup would.

* test(history): count worker parks, hold saturation blocks until released (#352)

wait_for_worker_pass now counts the idle worker's parks on its wake
event (two parks from now = one whole pass begun after the call)
instead of assuming a fixed number of clock reads per pass, which a
future clock read would have made vacuous. The real-clock bounded
close keeps a loose upper bound (0.02 s + NON_BLOCKING_CEILING).

The two saturation tests (history and output writer) blocked the
worker with release.wait(3) while they queued 1023 more items. On a
loaded runner the fill took longer than 3 s, the worker unblocked on
its own and drained part of the queue (pending 868..1017 != 1024),
and cleanup then reported a leaked worker. The block now holds until
the test releases it; 60 s is only a hang guard, and cleanup always
sets the release. Under one core with 8 busy loops: 8/8 runs failed
before, 6/6 pass after.

* test(maintenance): wait on duplicate jobs with a disk-bound hang guard (#409)

On Windows py3.13 CI (PR #405) a one-file duplicate cleanup missed its
10 s WAIT. The test already waits on the job's own task; the 10 s bound
acted as a deadline on disk-bound work. In the same 50 s window the
following tests' setups took up to 9 s against 0.1-0.6 s around it
while the test bodies stayed fast, which is consistent with a
process-wide disk I/O stall on the runner rather than a lease, handle,
executor or job stall (the log has no thread dump to show more).

tests/disk_waits.py holds the one disk-bound hang guard (90 s per wait,
under the 120 s per-test watchdog), shared with RESTORE_SECONDS (#382).
Waits on find and cleanup job tasks in the duplicate-files tests use
it; gates and server-state polls keep WAIT. The keeper-validation test
turns the safety backup off, since it does not test backups.
2026-10-28 05:18:00 +00:00
Vito ef7e07c765 refactor(admin): open System for Diagnostics and logs (#330, PR 1) (#402)
* test(legacy): snapshot the routes and the settings response

Pins every registered route and the key paths of GET /prompt_manager/settings
while System S13 (#330) moves the legacy pages' tools to native System. Recorded
on the unchanged main; cutover (#334) removes it.

* test(legacy): prove each removed legacy control has native coverage

TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet).
legacy-coverage.test.mjs fails when a removal cites a capability that is not
covered, copies another native test, lists an id or method twice, or leaves
the id or method on its page. legacy-system-links.spec.mjs holds the System
link table and guards the kept admin flows.

* refactor(admin): open System for Diagnostics and logs

The admin page's Diagnostics and View Logs buttons become links that open
System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs
dialogs and their methods go; TESTING.md lists them against their covered
capabilities (rows B1–B7).

* test(legacy): find a removed modal id passed to showModal and settle the links spec

The removal check also finds an id passed as a string to a ...Modal() call and no
longer reads data-id= as id=. The no-request test waits for the page's own
start-up reads to finish, the sort check compares as the server sorts, and the
kept-flow test checks the added prompt shows in the list.
2026-10-28 03:54:55 +00:00
Vito 83fa80c330 fix(restore,threads): a stopped restore waits for its worker; exit-window thread refusals are quiet (#360, #375) (#404)
* fix(restore): a restore cut off by shutdown waits for its worker thread (#360)

A cancelled restore handler or job no longer removes the stage or the
upload, or ends its exclusive registry admission, while the executor
thread still stages or publishes. The cancel waits for the worker; a
publish that finished is adopted before the cancel goes on, and a stage
finished after the cancel is removed.

* fix(threads): a thread or pool refused at interpreter exit is quiet and leaves nothing pending (#375)

During interpreter exit, threading._shutdown shuts every thread pool
before the package teardown runs while our daemon threads keep working.
map_parallel now runs the items a refused pool did not start on the
calling thread, schedule() drops the path and returns False when its
worker cannot start, and the preview queue skips the file. Each
refusal is logged once at DEBUG without a traceback; other start
failures behave as before.

* fix(restore): keep the cancel and the adoption when a stopped restore's publish ends (#360)

A publish that fails after the cancel now ends in CancelledError (after
its prune) instead of a 500, and the restored library is adopted to its
end even when a second cancel arrives. The blocking calls go through
_run_in_executor again, as the executor contract test expects.

Exit-window refusals: the comments state what was measured (a pool is
refused during threading._shutdown; Thread.start only once the
interpreter is finalizing), and the tests assert the one DEBUG line.
2026-10-28 03:48:11 +00:00
Vito e472a8bf74 fix(system): Tagging selects show their whole chosen option at 390 px (#398) (#399)
* fix(system): Tagging selects show their whole chosen option at 390 px (#398)

The concept's phone cap (120px) cut Existing tags ("Replace after review").
#tag-existing now never shrinks for its row text and keeps its natural width
at phone width, beside the text. The comparison expects selectsFit for every
select, and a new test walks every option of both selects at 390 and 320 px
in both themes. TESTING.md D132 records the decision.

* fix(system): bound Existing tags by its row at phone width (#398 review)
2026-10-28 02:03:22 +00:00
Vito Sansevero c4d7dd0c67 test(images): wait for the search's answered count in the concept comparison (#391)
searchOf took the first changed caption as the search's count, the same
read-before-the-response race as the inspector's tag chip.
2026-10-27 18:49:40 -07:00
Vito Sansevero 128b08ff26 test(system): wait for the restore job's own end before its toast (#361)
The reviewed-restore specs waited for the "Database restore finished"
toast with the 5 s expect, in effect a deadline on the disk-bound publish.
They now wait for the job's progress stream to close, assert the job
finished, and only then check the toast with the default timeout.
2026-10-27 18:33:10 -07:00
Vito Sansevero ba22b3721c test(thumbnails): make the no-JPEG-XL-encoder fixture authoritative (#371)
{available:false} stood in only the cached detection, and every settings
read probes a cached None again, finding the host's real encoder. The
fixture now stands in for the detector itself and reset restores it; a
unit test proves the route against a detector that finds an encoder.
2026-10-27 18:27:36 -07:00
Vito Sansevero 7fc36283f8 fix(images): read a loading collection as loading, never as 0 of 0 images (#391)
A reset published {loading, loaded:0, total:0} and the caption painted it
as "0 of 0 images" until the first page answered, so a fast reader (the
inspector concept spec on WebKit) read an empty result the server never
returned. The count line now says "Loading images…" while a first page is
in flight; the spec waits for the answered count, and a held-listing spec
pins the ordering.
2026-10-27 18:27:30 -07:00
Vito 44177974cb fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252) (#397)
* fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252)

A graph where no node takes a positive or negative input fell back to the
pre-redesign _extract_readable_prompt heuristic: the last node with any string
text input won, and with none it read stale UI workflow widgets. It stored LoRA
syntax, display-node reports, install notes, model file names and numbers.

The scan now asks sampler_less_prompt: only the executed API graph counts; a
candidate is a node with a text input that another node consumes; blank and
numeric texts are never prompts; this repository's own positive nodes are
authoritative; one distinct text links, several (or an unreadable wired text
that no candidate builds) stay unlinked and try the recorded-run attribution.
The heuristic and its helpers are removed; A1111 files read the parsed
positive prompt directly.

* fix(scan): refuse loader side outputs, seeded templates and keep A1111 text (#252)

Review findings: an unreadable text link counts as built from a candidate
only through that node's first output, so a LoRA loader's trigger-word output
is unknown, not its text. A node with a seed input holds a template, never the
prompt that ran. A sampler-less graph the rule cannot decide still links the
A1111 positive a metadata saver wrote beside it, as the old scan did. Only
plain numbers are numeric; words such as nan or inf are prompts.

* fix(scan): linear plain-number check for sampler-less texts (#252)

The numeric pattern had overlapping quantifiers that backtrack quadratically
on a long digit run ending in a non-digit; it now has none.
2026-10-28 00:32:30 +00:00
Vito d11b354ed6 fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2) (#394)
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1)

The native "Tag output images" job now runs the image-tags loop over every
still image the output folders list, linked or not: each image gets its own
model tags (at most "Suggestions per image") in its image tag set, and its
bytes are marked auto-tagged in a new media_image_autotags table. No prompt
tag or prompt text is written.

- database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional
  in old backups, refused without the annotation group.
- add_image_tags/replace_image_tags take mark_autotagged; autotag_state and
  autotagged_locators read the marks through the one current-record rule.
- "Skip already tagged" skips images whose current bytes are marked, counted
  in a stat-only pre-pass; hand tags do not count.
- The review counts images and estimates the time (WD14 on CPU or CUDA).
- System > Auto-tagging review rows and copy, D134 and the Auto-tagging
  contract follow.

Part of #379

* fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379)

The job read as ended while it still held the model gate, so a quick
restart could be refused with model-busy. Pre-pass chunks now publish
progress, and a cancel between chunks keeps exact counts.

Part of #379

* fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379)

The live check measured WD14 on CUDA at 0.167 s per image (100 images in
16.7 s with the model load), so the review estimate uses 0.17 s. A finished
or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X
min left"; it now ends with "<verb> <state> · N files processed".

Part of #379

* fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2)

The admin and gallery batch starts the shared Tag output images job
("Tag output images (legacy page)") and relays its progress; closing the
window cancels it. Review asks per image with Skip first and applies by
image and sha256; apply refuses a prompt_id body. The Tag image dialog
marks an image auto-tagged only after Generate suggestions on that
capture. A second legacy batch (also one that wins the start race)
answers JSON 409, and both pages now show that sentence.

* test(tagging): gallery review units apply by image; record each progress change

The gallery review unit tests still pinned the removed prompt_id apply.
The legacy batch spec read the progress text once per observer callback,
so a phase frame and the first image frame landing in one task lost the
phase (WebKit); it now records every mutation record.
2026-10-27 22:40:57 +00:00
Vito 2f8b7cb306 feat(images): select many images: ranges, keys, touch and all N matching (#392) (#393)
The workspace Images page selects like a file manager in Grid, List and
Masonry, across every loaded batch: Shift+click ranges from the anchor,
Ctrl/Cmd+click toggles a card without opening it, Ctrl/Cmd+A selects the
loaded images, Shift+arrows extend, Escape clears, and a touch long-press
starts a selection mode in which taps toggle. A plain click still opens or
inspects.

Once every loaded image is checked and more match, the bar offers
"Select all N matching". Add tags, Auto-tag selected and Set category then
send the listing filter instead of N identities: the image-tags job takes
`filter` in place of `targets` and /bulk/category takes `images_filter`,
both resolved by the listing's own admission code (up to 10,000 images, a
Source bound by its root_scope). Export metadata waits for loaded images.

Recorded as deviation D135; the concept is unchanged.
2026-10-27 22:20:27 +00:00
Vito 37e4e08327 fix(tagging): Images tag search goes through the tag index (#379, PR 3) (#395)
* fix(tagging): Images tag search goes through the tag index (#379, PR 3)

The Images Tags filter and text search no longer stream every image tag
row per request. image_tag_ids reads the names that tag at least one image
(exact under Tags, case-folded substring for q) and image_tag_locators
reads their files by tag_id through idx_media_image_tags_tag. Results are
unchanged: a differential test compares the lookup with the old stream,
and a trace test checks that no statement reads media_image_tags except
by tag id. The search contract in TESTING.md and a browser test (CUJ 8)
follow.

Closes #379

* test(search): pin prompt-only tags and multi-chunk tag lookups (#379)

Reads the tag names and the locators in the same annotation read helper.
2026-10-27 22:07:59 +00:00
69 changed files with 4200 additions and 2094 deletions
+6 -1
View File
@@ -221,7 +221,10 @@ jobs:
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install black flake8
cache: pip
cache-dependency-path: requirements-dev.txt
# The exact black and flake8 pins from requirements-dev.txt, so a new release cannot turn CI red.
- run: pip install $(grep -E '^(black|flake8)==' requirements-dev.txt)
- run: black --check .
- run: flake8 .
@@ -235,6 +238,8 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
cache-dependency-path: tools/vendor/package-lock.json
- name: Check shipped extension and page modules parse
run: |
+3 -1
View File
@@ -13,9 +13,11 @@ repos:
language: system
types: [python]
# Fails at the severity CI enforces (code-quality.yml: -lll, high only), never stricter.
# pre-commit appends the staged files, so no shell wrapper can drop the first one.
- id: bandit
name: bandit
entry: bash -c 'bandit "$@" -ll -q 2>&1; true'
entry: bandit -lll -q
language: system
types: [python]
exclude: ^(tests/|reference/)
+102 -9
View File
@@ -3200,12 +3200,16 @@ the per-section hand-off panel, was retired by S12 (#329): every section is nati
again"; an Unload that found nothing loaded only reads the status again. At phone width
the concept's 120px select cap cuts every engine label, so `#tag-model` alone keeps its
natural width there and its row wraps the select under the text (`workspace.css`
`/* D132 */`); Existing tags stays as the concept draws it, cut at 390 as there. The
comparison maps `selectsFit` only for the engine select and measures each row's
`flexWrap` and whether its control sits under the text. Pinned by `auto-tagging matches
the concept` (both themes and sizes), `auto-tagging at 390 wraps only the engine row`,
`auto-tagging engine select fits at 320 wide` and the mutation check (it includes a
wrapped row) (system-tagging-concept.spec), and the model, Download, Unload and Preview
`/* D132 */`). The concept's cap cuts Existing tags too ("Replace after review"); by the
maintainer's decision (#398) that parity is not kept: `#tag-existing` never shrinks for
its row text and at phone width keeps its natural width beside the text (its row does
not wrap; the text column narrows). 1600 is as the concept draws it. The comparison
expects `selectsFit` true for every select and measures each row's `flexWrap` and
whether its control sits under the text. Pinned by `auto-tagging matches the concept`
(both themes and sizes), `auto-tagging at 390 wraps only the engine row`, `auto-tagging
selects fit every option at 390/320 wide` (every option of both selects, both themes, no
page scroll) and the mutation check (a wrapped row, a squeezed engine and Existing tags
select) (system-tagging-concept.spec), and the model, Download, Unload and Preview
tests of workspace-system-tagging.spec.
- **D133** (Auto-tagging, S11): a "Suggestions per image" row (`#tag-suggestions`,
number 1–100, host key `autotag_suggestion_limit`) follows the two confidence rows:
@@ -3826,7 +3830,9 @@ threshold off that grid is refused ("General-tag confidence must be a multiple o
0.05.", `validate_saved_threshold`; `draftProblem` holds back Preview tagging review and
the batch Review with the same sentence) and falls back on load; an accepted one is
stored exactly on the grid (#369). A per-request threshold keeps its own check: `/image-annotations/suggest` and
`/jobs/autotag` take 0–1; the legacy `/autotag/single` and `/autotag/start` only require a number. `GET /autotag/models` reports the
`/jobs/autotag` take 0–1; the legacy `/autotag/start` passes its options through
`validate_options` and starts the `autotag` job titled "Tag output images (legacy page)";
the legacy `/autotag/single` only requires a number. `GET /autotag/models` reports the
saved values (`wd14_general_threshold`, `wd14_character_threshold`, `suggestion_limit`,
`default_prompt`, `model_type`, `existing`), so the Tag image and Auto-tag selected
dialogs start from them; a preset (the bulk review, Preview tagging review) wins.
@@ -3892,6 +3898,57 @@ images", "Tag output images (legacy page)", "Tagging one image (legacy page)",
is pinned by `tests/test_autotag_schema.py` (forward-only: one empty `WITHOUT ROWID`
table, optional in old backups, refused without the annotation group).
- **The legacy pages (#379, PR 2)** write image tags, never prompt tags.
`POST /autotag/start` (the admin and gallery batch) reads `model_type`, `prompt`,
`skip_tagged` (Skip or Merge), `keep_in_memory` and the thresholds (else the saved
ones), checks them with `validate_options` (WD14: both thresholds; GGUF/HF:
`instructions` = the non-blank `prompt`, else the saved template; the saved
"Suggestions per image") and starts the `autotag` job "Tag output images (legacy
page)" (gate owner `legacy-batch`) through the native start, so the safety backup and
the disk check apply. A second legacy batch answers JSON 409 `{success: false, error:
"Auto-tagging is already running"}` (also when another legacy start wins the race past
that check); `SseStream` hands a non-2xx JSON `error` to `onerror` as `serverMessage`,
which both pages show instead of "Connection lost during auto tagging". Any other refusal (a job in the way: "<title>
already running", `model-unavailable`, `insufficient-space`, `model-busy`, a bad
option) is one `{type: "error", status: "error", message}` frame. The stream relays
the job: the phases as `{type: "progress", progress: 0 | 5 | 8, status: "Taking a
safety backup…" | "Loading model…" | "Listing output images…"}`, each image as
`{type: "progress", progress: 10 + progress × 85 / 100, status, processed, tagged,
skipped}` (at most every 0.5 s; `processed` = Tagged + Unchanged + Errors), and the end
as `complete` (`processed`, `tagged`, `skipped`, `errors`, `skipped_outside_roots: 0`,
`status` "Complete" or "No still images found", `model_status`, `model_loaded`),
`{type: "cancelled", message: "Cancelled. Tags already added were kept."}` or an
`error` frame with the job's status line. A job quiet for 5 s repeats its last frame,
and a closed stream cancels the job (the legacy pages have no other stop).
`POST /autotag/single` takes `existing` (`"skip"` or `"merge"`, default merge). Under
Skip, an image whose current bytes are auto-tagged answers `{success, already_tagged:
true, image_tags, prompt_id, filename}` after one stat, without the model gate.
Otherwise the image is captured and the model runs on the capture under the gate with
the saved cap; the answer adds `tags`, `sha256` (the captured bytes) and `image_tags`,
and `service.custom_prompt` is never set. Refusals: `409 model-unavailable`, `413
too-large`, `404 file-not-found`, `409 file-changed`, `503 busy`, `415 not-a-still`
(the image is marked), `409 model-busy`. `POST /autotag/apply` takes exactly
`{image_id, sha256, tags}` or `{path, root_index?, sha256, tags}` (64 KiB), re-hashes
the file and unions the tags into that image (`add_image_tags`, marked): `{success,
added_tags, total_tags}`; an empty list only marks the image. A body with `prompt_id`
is `400` "This page is out of date. Reload it."; a changed file `409 file-changed`
("The image changed since its tags were generated."); a full set `409 tag-limit`; a
restore `409 storage-changed`. The admin review asks each image with Skip first: an
auto-tagged one shows "Image Already Auto-tagged" / "A tagging model already tagged
this image. Generate new tags?" with its image tags (Skip, Skip All Tagged, Re-tag asks
again with Merge); the gallery review always merges. Apply & Next sends the image and
its `sha256` (linked or not). The batch copy: "Skip images already auto-tagged",
"Auto tagging complete! Tagged N images." ("1 image"), and under both progress bars
"Closing this window stops tagging. Tags already added are kept. Prompt Manager's
Activity page shows the time left.". Pinned by `tests/test_autotag_legacy.py`,
`tests/js/image-helpers.test.js` and `legacy-autotag.spec`.
- **Tag image dialog marks (DEC4).** A save after Generate suggestions ran on that
capture marks the image auto-tagged (also a replayed save, and a saved retry after
`tag-conflict`); a save without Generate does not. The API keeps the suggested capture
tokens; each new one drops the tokens whose capture is gone (`expires_in`, which never
resets a capture's idle clock), a release or a saved receipt drops its own, and
closing the captures clears them (`TestDialogMarks`).
No browser test downloads or runs a model: the injected `TestModel` answers, its
`download_model` raises unless `/__test__/tagging/download {mode: hold | succeed |
fail}` installs a fake that waits on a fixture event, and every tagging reset releases
@@ -4046,6 +4103,33 @@ list stay unscoped: any row's control proves them.
| API timeout `apiTimeout` | `gallery-settings.js:37` | Legacy gallery (S12-10; read by no request) | kept in legacy | — | defaults are a complete, sane settings object (gallery-settings.test.js) |
| config.json-only keys | `py/config.py:212–234,722–739`; `py/api/diagnostics_report.py:185` | S12 report (read-only) | covered | `diagnostics` · button · "Download diagnostic report" | test_report_lists_config_only_keys (test_api_diagnostics_report.py) |
### Legacy System UI removed by S13
System S13 (#330) points the legacy pages' System buttons at native System (a link that
opens `workspace#/system/<id>` in a new tab) and removes the legacy copy of a capability
only when its row above is `covered`. One row per removed capability: Capability is the
exact Capability cell of its coverage row, Page is `admin` or `gallery`, Removed ids and
Removed methods list what went (`—` for none), and Native test copies that row's Test
cell. An id that stays on a System link is not listed, only its handler.
`tests/vendor/legacy-coverage.test.mjs` fails when a row cites a capability that is not
`covered` or copies another test, when an id or method is listed twice for one page, or
when a listed id is still on its page (`id="…"` in the markup or the script, a
`getElementById`/`querySelector('#…')` lookup, an entry of a modal-id list) or a listed
method is still defined or called (`.name(`). A deletion listed nowhere is caught by the
kept-flow tests (`kept admin flows still work` in `legacy-system-links.spec.mjs` and the
legacy page workflows), not by this check.
| Capability | Page | Removed ids | Removed methods | Native test |
| --- | --- | --- | --- | --- |
| Diagnostics run | admin | `#diagnosticsModal`, `#runDiagnosticsBtn`, `#diagnosticsContent` | `showDiagnosticsModal()`, `closeDiagnostics()`, `runDiagnostics()`, `renderDiagnostics()` | Run checks opens the shared Health report (workspace-system-diagnostics.spec.mjs) |
| Synthetic image-link test | admin | `#testImageLinkBtn` | `testImageLink()` | Create test link confirms first and names the prompt (workspace-system-diagnostics.spec.mjs) |
| View logs with level and limit | admin | `#logsModal`, `#logLevel`, `#logLimit`, `#refreshLogsBtn`, `#logsContainer` | `showLogsModal()`, `closeLogs()`, `refreshLogs()`, `displayLogs()` | level filter is minimum-level over all six choices (workspace-system-diagnostics.spec.mjs) |
| Log statistics | admin | `#logStatsTotal`, `#logStatsErrors`, `#logStatsWarnings`, `#logStatsLevel`, `#logStatsSize` | `loadLogStats()`, `formatBytes()` | log statistics show entries, errors, warnings, files and size (workspace-system-diagnostics.spec.mjs) |
| Log files list and download | admin | `#downloadLogsBtn`, `#logFilesList` | `loadLogFiles()`, `downloadLogs()`, `downloadLogFile()` | each listed file downloads by its returned name and a link has no Download (workspace-system-diagnostics.spec.mjs) |
| Clear/truncate logs | admin | `#clearLogsBtn` | `clearLogs()` | Clear all logs states both counts, confirms first and reads everything again (workspace-system-diagnostics.spec.mjs) |
| Set log level (incl. CRITICAL), console logging | admin | `#updateLogConfigBtn`, `#setLogLevel`, `#consoleLogging` | `loadLogConfig()`, `updateLogConfig()` | runtime level (including CRITICAL) and console logging save and say they apply until the server restarts (workspace-system-diagnostics.spec.mjs) |
### Browser cache ownership (#222)
The legacy gallery's Clear cache used to delete every localStorage key starting `pm_` or
@@ -4440,8 +4524,17 @@ impact and command receipts report `image_count` / `images` /
`(size, mtime_ns)` equals its verified fingerprint; an in-place rewrite that
preserves both keeps the old set matching until the file is next observed. The
Images Tags filter (a file matches when every tag is on its linked positive or
its current image tag set) and `q` search stream the tagged records once and
stat only the listed files whose records carry a filter tag or a `q` hit; the
its current image tag set) and `q` search find their candidates through the tag
index (#379), never by streaming every image tag row: `image_tag_ids` reads the
tag names that tag at least one image (an `EXISTS` on `idx_media_image_tags_tag`)
and keeps a name when it equals a Tags filter tag exactly or contains `q`
case-folded; `image_tag_locators` reads the files carrying those ids (`tag_id IN`,
500 ids a chunk). Only the listed ones among them are stat'ed, and only their
current records count; a linked positive's prompt tags still combine with the
image tags under a Tags filter. `test_indexed_candidates_equal_the_full_stream`
compares the lookup with the old full stream (kept in the test as the reference)
and `test_tag_search_reads_image_tags_only_by_tag_id` traces every statement; the
inspector's `image_tags` facet (`current` / `none` / `unavailable`) uses the
inspection's own stable stat (`tests/test_api_workspace_images.py`
`TestWorkspaceImageTags`, `tests/test_api_prompt_inspector.py`). The
+37 -21
View File
@@ -110,12 +110,16 @@ _MARKED_LOCATORS = (
f"SELECT DISTINCT f.locator FROM {AUTOTAG_TABLE} m " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.id=m.byte_id JOIN {FILES_TABLE} f ON f.id=b.file_id"
)
_RECORDS = (
"SELECT f.locator,b.id,b.size,b.mtime_ns,t.name " # nosec B608
f"FROM {FILES_TABLE} f JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"LEFT JOIN {IMAGE_TAGS_TABLE} mit ON mit.byte_id=b.id "
"LEFT JOIN tags t ON t.id=mit.tag_id "
"ORDER BY f.locator,b.first_seen_at,b.id,mit.position"
# Tag search (#379): the names that tag at least one image (the tag index
# answers the EXISTS), then the locators carrying chosen names (by tag_id).
_IMAGE_TAG_NAMES = (
"SELECT t.id,t.name FROM tags t WHERE EXISTS " # nosec B608
f"(SELECT 1 FROM {IMAGE_TAGS_TABLE} mit WHERE mit.tag_id=t.id)"
)
_TAG_LOCATORS = (
f"SELECT DISTINCT f.locator FROM {IMAGE_TAGS_TABLE} mit " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.id=mit.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE mit.tag_id IN ({{marks}})"
)
@@ -516,20 +520,32 @@ class AnnotationOperations:
with self._annotation_read() as conn:
return {row[0] for row in conn.execute(_MARKED_LOCATORS)}
def iter_image_tag_records(self) -> Iterator[tuple[str, int, int, list[str]]]:
"""Stream every byte record as ``(locator, size, mtime_ns, tags)``.
def image_tag_ids(self, wanted, query: str) -> list[int]:
"""Ids of the tag names on at least one image that a search wants.
One SELECT ordered by locator, then by the bytes' first sighting, with
each set's names in position order.
A name counts when it is in ``wanted`` exactly (the Tags filter) or
contains ``query`` case-folded (the text search; ``query`` is
already folded). One read of the names that tag an image.
"""
conn = self.model.get_connection()
current: tuple | None = None
for locator, byte_id, size, mtime_ns, name in conn.execute(_RECORDS):
if current is None or current[0] != byte_id:
if current is not None:
yield current[1]
current = (byte_id, (locator, size, mtime_ns, []))
if name is not None:
current[1][3].append(name)
if current is not None:
yield current[1]
wanted = set(wanted)
with self._annotation_read() as conn:
return [
tag_id
for tag_id, name in conn.execute(_IMAGE_TAG_NAMES)
if name in wanted or (query and query in name.casefold())
]
def image_tag_locators(self, tag_ids) -> set[str]:
"""Locators with any byte record (current or not) carrying one of
``tag_ids``, through the tag index, in chunks of 500 ids."""
ids = list(dict.fromkeys(tag_ids))
found: set[str] = set()
with self._annotation_read() as conn:
for start in range(0, len(ids), _LOOKUP_CHUNK):
chunk = ids[start : start + _LOOKUP_CHUNK]
marks = ",".join("?" * len(chunk))
found.update(
row[0]
for row in conn.execute(_TAG_LOCATORS.format(marks=marks), chunk)
)
return found
-31
View File
@@ -1808,37 +1808,6 @@ class PromptDatabase(
reader = DocumentReader(conn)
return [self._image_row_to_dict(row, reader) for row in cursor]
def autotag_batch_candidates(self) -> List[Dict[str, Any]]:
"""The images a Tag output images batch looks at (#328).
``{id, image_path, prompt_id, prompt_tags}`` for every image with a
stored path that is linked to a prompt, newest first (the
``get_all_images`` order, ties newest row first). Never reads
``workflow_data`` or the other documents: a batch needs none of them.
"""
sql = (
"SELECT gi.id, gi.image_path, gi.prompt_id, "
"(SELECT GROUP_CONCAT(t.name, '|||') FROM prompt_tags pt "
"JOIN tags t ON pt.tag_id = t.id WHERE pt.prompt_id = gi.prompt_id) "
"AS prompt_tags "
"FROM generated_images gi "
"INNER JOIN prompts p ON gi.prompt_id = p.id "
"WHERE gi.image_path IS NOT NULL AND gi.image_path != '' "
"ORDER BY gi.generation_time DESC, gi.id DESC"
)
with self.model.get_connection() as conn:
return [
{
"id": row["id"],
"image_path": row["image_path"],
"prompt_id": row["prompt_id"],
"prompt_tags": (
row["prompt_tags"].split("|||") if row["prompt_tags"] else []
),
}
for row in conn.execute(sql)
]
def get_all_images(
self,
limit: int = 0,
+3 -193
View File
@@ -363,6 +363,9 @@ class PromptManagerAPI(
# Private media captures of the annotation routes (py/api/annotations.py),
# created on first use with their restore subscription.
self._captures = None
# Capture tokens Generate suggestions ran on: a save of one marks the
# image auto-tagged (#379).
self._suggested_tokens = set()
self._captures_unsubscribe = None
# Set by close_annotation_captures (shutdown): no store is recreated.
self._captures_closed = False
@@ -1104,196 +1107,3 @@ class PromptManagerAPI(
result["parameters"][field] = metadata[field]
return result
def _extract_readable_prompt(self, parsed_data):
"""Extract human-readable prompt text from ComfyUI/A1111 data."""
def safe_to_string(value):
if isinstance(value, str):
return value
elif isinstance(value, list):
return " ".join(str(item) for item in value if item)
elif value is not None:
return str(value)
return None
# First check if we already extracted a positive prompt (A1111 format)
if parsed_data.get("positive_prompt"):
return parsed_data["positive_prompt"]
# Check if prompt is already a string
if isinstance(parsed_data.get("prompt"), str):
return parsed_data["prompt"]
# Check if prompt is a simple value that can be converted
if parsed_data.get("prompt") and not isinstance(
parsed_data.get("prompt"), dict
):
return safe_to_string(parsed_data["prompt"])
prompt_data = parsed_data.get("prompt")
if isinstance(prompt_data, dict):
# Use the enhanced logic from parse-metadata.py
positive_prompt = self._extract_positive_prompt_from_comfyui_data(
prompt_data
)
if positive_prompt:
return positive_prompt
# Check workflow data if available
workflow_data = parsed_data.get("workflow")
if isinstance(workflow_data, dict):
positive_prompt = self._extract_positive_prompt_from_comfyui_data(
workflow_data
)
if positive_prompt:
return positive_prompt
# Check parameters for positive prompt
if parsed_data.get("parameters", {}).get("positive"):
return safe_to_string(parsed_data["parameters"]["positive"])
return None
def _get_node_inputs(self, node):
"""Safely get inputs from a node, handling both dict and list formats.
Old format: inputs is a dict with direct key-value pairs
inputs = {"text": "my prompt", "seed": 123}
New format: inputs is a list of connection objects
inputs = [
{"name": "text", "type": "STRING", "link": null,
"widget": {"name": "text"}},
{"name": "clip", "type": "CLIP", "link": 11}
]
"""
if not isinstance(node, dict):
return {}
inputs = node.get("inputs", {})
# If inputs is already a dict, return it
if isinstance(inputs, dict):
return inputs
# If inputs is a list, convert to dict format
if isinstance(inputs, list):
inputs_dict = {}
for input_item in inputs:
if isinstance(input_item, dict) and "name" in input_item:
name = input_item["name"]
inputs_dict[name] = input_item
return inputs_dict
return {}
def _find_text_in_node(self, node):
"""Try to find text content in a node using various strategies.
Handles both old and new workflow formats.
"""
if not isinstance(node, dict):
return None
# Strategy 1: Check normalized inputs for 'text' field
inputs = self._get_node_inputs(node)
if "text" in inputs and isinstance(inputs["text"], str):
return inputs["text"]
# Strategy 2: For text encoder nodes, check widgets_values
class_type = node.get("class_type", node.get("type", ""))
text_encoder_types = [
"CLIPTextEncode",
"CLIPTextEncodeSDXL",
"CLIPTextEncodeSDXLRefiner",
"CLIPTextEncodeFlux",
"PromptManager",
"PromptManagerText",
"BNK_CLIPTextEncoder",
"Text Encoder",
"CLIP Text Encode",
]
if any(
encoder_type.lower() in class_type.lower()
for encoder_type in text_encoder_types
):
widgets_values = node.get("widgets_values", [])
if widgets_values and len(widgets_values) > 0:
if isinstance(widgets_values[0], str) and widgets_values[0].strip():
return widgets_values[0]
return None
def _extract_positive_prompt_from_comfyui_data(self, data):
"""Extract positive prompt from ComfyUI data.
Handles both old and new formats.
"""
if not isinstance(data, dict):
return None
# Build nodes dictionary
nodes_by_id = {}
if "nodes" in data:
# Handle nodes array format
for node in data["nodes"]:
if isinstance(node, dict):
nid = node.get("id")
if nid is not None:
nodes_by_id[nid] = node
else:
# Handle flat dictionary format (node_id -> node_data)
for nid_str, node in data.items():
try:
nid = int(nid_str)
except (ValueError, TypeError):
nid = nid_str
if isinstance(node, dict):
if "id" in node:
nid = node["id"]
nodes_by_id[nid] = node
if not nodes_by_id:
return None
# First, try to find positive/negative connection pattern
pos_id = None
for node in nodes_by_id.values():
if isinstance(node, dict):
inputs = self._get_node_inputs(node)
if "positive" in inputs and "negative" in inputs:
try:
pos_input = inputs["positive"]
if isinstance(pos_input, list) and len(pos_input) > 0:
pos_id = int(pos_input[0])
break
except (ValueError, TypeError, IndexError):
continue
# Get text from the positive node
if pos_id is not None and pos_id in nodes_by_id:
text_val = self._find_text_in_node(nodes_by_id[pos_id])
if text_val:
return text_val
# Fallback: find any text encoder node with text content
text_nodes = []
for node in nodes_by_id.values():
if isinstance(node, dict):
text_val = self._find_text_in_node(node)
if text_val:
# Try to determine if this is positive or negative
node_title = node.get("title", "").lower()
if "neg" not in node_title and "negative" not in node_title:
# Prioritize non-negative prompts
text_nodes.insert(0, text_val)
else:
text_nodes.append(text_val)
# Return the first positive-looking prompt
if text_nodes:
return text_nodes[0]
return None
+19 -8
View File
@@ -47,7 +47,7 @@ try:
extract_generation_params,
parse_page_json,
)
from ...utils.prompt_pair_resolver import resolve_prompt_pair
from ...utils.prompt_pair_resolver import resolve_prompt_pair, sampler_less_prompt
from ...utils.review_capture import reviewable
except ImportError:
from database.pair_ingestion import blank_positive, ingest_resolved_pair
@@ -62,7 +62,7 @@ except ImportError:
extract_generation_params,
parse_page_json,
)
from utils.prompt_pair_resolver import resolve_prompt_pair
from utils.prompt_pair_resolver import resolve_prompt_pair, sampler_less_prompt
from utils.review_capture import reviewable
@@ -1754,13 +1754,24 @@ class AdminRoutesMixin:
canonical.get("parameters"), str
):
return resolution["state"]
elif canonical.get("prompt"):
# No node takes a positive or negative input (a guider or wrapper
# graph): the one wired text, never a guess (#252).
found = sampler_less_prompt(canonical["prompt"])
prompt_text = found["text"]
if prompt_text is None and isinstance(canonical.get("parameters"), str):
# A metadata saver also wrote A1111 text: its positive prompt.
parsed = self._parse_comfyui_prompt(metadata)["positive_prompt"]
prompt_text = parsed.strip() if parsed and parsed.strip() else None
if prompt_text is None:
linked = self._link_recorded_run(
media_file, canonical, detailed, history
)
if linked is not None:
return linked
return "unresolved" if found["state"] == "ambiguous" else "found"
else:
parsed = self._parse_comfyui_prompt(metadata)
if not (parsed.get("prompt") or parsed.get("parameters")):
return None
prompt_text = self._extract_readable_prompt(parsed)
if prompt_text is not None and not isinstance(prompt_text, str):
prompt_text = str(prompt_text)
prompt_text = self._parse_comfyui_prompt(metadata)["positive_prompt"]
if not (prompt_text and prompt_text.strip()):
return "found"
prompt_text = prompt_text.strip()
+20
View File
@@ -296,6 +296,7 @@ class AnnotationRoutesMixin:
self._captures_closed = True
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
store, self._captures = self._captures, None
self._suggested_tokens = set()
if unsubscribe is not None:
unsubscribe()
if store is not None:
@@ -515,6 +516,7 @@ class AnnotationRoutesMixin:
store = self._captures
if store is not None and _TOKEN.fullmatch(token) is not None:
store.release(token)
self._suggested_tokens.discard(token)
return web.json_response(
{"success": True}, headers={"Cache-Control": "no-store"}
)
@@ -564,6 +566,9 @@ class AnnotationRoutesMixin:
return _coded("capture-expired", _OTHER_FILE)
if facts.sha256 != capture.sha256:
return _coded("file-changed")
# A save after Generate suggestions ran on this capture marks the
# image auto-tagged (#379), so a Skip batch leaves the review alone.
suggested = token in self._suggested_tokens
try:
receipt = await self._run_in_executor(
self.db.replace_image_tags,
@@ -574,6 +579,7 @@ class AnnotationRoutesMixin:
expected_epoch=expected_epoch,
expected_tag_revision=revision,
tags=tags,
mark_autotagged=suggested,
)
except AnnotationCommandError as exc:
current = exc.current or {}
@@ -590,10 +596,23 @@ class AnnotationRoutesMixin:
except Exception:
self.logger.exception("Image tag save failed")
return _error(500, "save-failed", "The tags could not be saved.")
# Dropped only after a saved or replayed receipt: a retry after a
# tag-conflict still marks.
self._suggested_tokens.discard(token)
return web.json_response(
{"success": True, **receipt}, headers={"Cache-Control": "no-store"}
)
def _remember_suggested(self, store, token: str) -> None:
"""Note that Generate ran on *token*; forget tokens whose capture is
gone. ``expires_in`` does not reset a capture's idle clock."""
self._suggested_tokens = {
kept
for kept in self._suggested_tokens
if store.expires_in(kept) is not None
}
self._suggested_tokens.add(token)
# -- suggest ---------------------------------------------------------------
async def suggest_image_tags(self, request: web.Request) -> web.Response:
@@ -675,6 +694,7 @@ class AnnotationRoutesMixin:
finally:
if not started:
store.release_use(token)
self._remember_suggested(store, token)
return web.json_response(
{
"success": True,
+520 -409
View File
File diff suppressed because it is too large Load Diff
+73 -3
View File
@@ -483,13 +483,59 @@ class BackupRestoreRoutesMixin:
self.logger.info(f"Removed {len(removed)} old safety backups")
async def _publish(self, model, staged):
"""Publish ``staged``; on a page-copy failure prune and re-raise."""
"""Publish ``staged``; on a page-copy failure prune and re-raise.
A cancel (server stop) waits for the publishing thread (#360): the
caller's cleanup and its registry admission must outlive it. When the
copy then succeeded the library was replaced, so it is adopted
(``_after_publish``, itself run to its end) before the cancel goes
on; a copy that failed is handled as usual, then the cancel goes on.
"""
future = self._in_executor_task(model.publish_staged, staged)
cancelled = await _outlive_cancel(future)
try:
return await self._run_in_executor(model.publish_staged, staged)
backup_path = future.result()
except sqlite3.OperationalError as exc:
if getattr(exc, "_prompt_manager_restore_backup_created", False) is True:
await self._prune_after(model.db_path, failed=True)
if cancelled:
raise asyncio.CancelledError from exc
raise
except Exception as exc:
if cancelled:
raise asyncio.CancelledError from exc
raise
if cancelled:
self.logger.info("The restore finished while the server was stopping")
# A second cancel (the loop's own shutdown) must not cut it short.
adopted = asyncio.ensure_future(self._after_publish(model.db_path))
await _outlive_cancel(adopted)
if not adopted.cancelled() and adopted.exception() is not None:
self.logger.error(
"Could not adopt the restored library",
exc_info=adopted.exception(),
)
raise asyncio.CancelledError
return backup_path
async def _stage_to_end(self, model, temp_path):
"""Stage ``temp_path``; a cancel waits for the worker (#360).
The upload stays until the staging thread is done with it, and a
stage it finished after the cancel is removed before the cancel goes
on.
"""
future = self._in_executor_task(model.stage_restore, temp_path)
if await _outlive_cancel(future):
if not future.cancelled() and future.exception() is None:
await self._discard_held([_Orphan(future.result())])
raise asyncio.CancelledError
return future.result()
def _in_executor_task(self, func, *args):
"""``_run_in_executor(func, *args)`` as a task ``_outlive_cancel`` can
wait for (shielded, the executor call is never cancelled)."""
return asyncio.ensure_future(self._run_in_executor(func, *args))
async def _after_publish(self, db_path):
"""Everything after the copy replaced the library: best effort, logged.
@@ -609,7 +655,7 @@ class BackupRestoreRoutesMixin:
{"success": False, "error": REQUESTS_BUSY}, status=409
)
model = self.db.model
staged = await self._run_in_executor(model.stage_restore, temp_path)
staged = await self._stage_to_end(model, temp_path)
try:
backup_path = await self._publish(model, staged)
finally:
@@ -927,6 +973,30 @@ class BackupRestoreRoutesMixin:
job.update(progress=100, status_line=line)
async def _outlive_cancel(future):
"""Await ``future`` (an executor call's task) to its end, even through
cancels (#360).
A cancelled task cannot stop the worker thread, so the files it uses and
the registry admission it runs under must outlive it: the cancel is held
until the worker returned. Returns whether the task was cancelled
meanwhile; the caller reads the outcome from ``future`` and re-raises
``CancelledError``. A future cancelled itself (an executor shut down) is
not a cancel of the task: ``future.result()`` raises it. Never catches
``GeneratorExit``: a closed loop still closes the coroutine.
"""
cancelled = False
while not future.done():
try:
await asyncio.shield(future)
except asyncio.CancelledError:
if not future.cancelled():
cancelled = True
except Exception:
pass # the caller reads it from the future
return cancelled
class _Orphan:
"""A stage with no store entry, in the shape ``_discard_held`` takes."""
+10 -16
View File
@@ -575,12 +575,6 @@ def _current_fingerprint(path: Path) -> tuple[int, int] | None:
return observed.st_size, observed.st_mtime_ns
def _image_tags_relevant(names: list, wanted: set, query: str) -> bool:
if wanted and not wanted.isdisjoint(names):
return True
return bool(query) and any(query in name.casefold() for name in names)
def _workspace_key(entry: tuple, order: str) -> tuple:
"""The keyset sort key of *entry* (a _Listed or a 4-tuple entry)."""
if not isinstance(entry, _Listed):
@@ -2170,8 +2164,9 @@ class ImageRoutesMixin:
def _image_tag_matches(self, listings: list[tuple], filters: dict) -> dict:
"""Normalized path -> _ImageTagMatch for files whose image tags count.
Streams the tagged byte records once and keeps only listed locators
carrying a filter tag or a ``q`` hit; only those candidates are
Looks the relevant tag names up (exact under Tags, a case-folded
substring for ``q``) and their files through the tag index, keeping
only listed locators; only those candidates are
stat'ed (never every listed file), and a candidate's tags count only
while a byte record is verified at its present ``(size, mtime_ns)``
(spec amendment S1). Under a Tags filter the candidates' linked
@@ -2180,16 +2175,15 @@ class ImageRoutesMixin:
tags, query, category = filters["tags"], filters["query"], filters["category"]
wanted = set(tags)
located = self._workspace_locators(listings)
# The tag index answers which files carry a relevant name (#379); no
# read streams every image tag row.
tag_ids = self.db.image_tag_ids(wanted, query)
candidates = {}
records = self.db.iter_image_tag_records()
try:
for locator, _size, _mtime_ns, names in records:
if tag_ids:
for locator in self.db.image_tag_locators(tag_ids):
entry = located.get(locator)
if entry is not None and locator not in candidates:
if _image_tags_relevant(names, wanted, query):
candidates[locator] = entry
finally:
records.close()
if entry is not None:
candidates[locator] = entry
stats = {}
for locator, entry in candidates.items():
fingerprint = _current_fingerprint(entry.path)
+23 -11
View File
@@ -88,9 +88,9 @@ from .thumbnail_encode import shutdown_encoders, writer_settings
from .thumbnail_jobs import ThumbnailOptions
try:
from ...utils.parallel import map_parallel
from ...utils.parallel import map_parallel, note_refusal, refused_at_shutdown
except ImportError: # tests import the package top-level
from utils.parallel import map_parallel
from utils.parallel import map_parallel, note_refusal, refused_at_shutdown
# The placeholder frame a video without a preview is answered with (JPEG
# bytes, never written to disk); written previews follow the settings.
@@ -410,7 +410,8 @@ class AutoThumbnailer:
def enqueue(self, src, dst, is_video):
"""Queue *dst*'s preview for the worker; False when already queued,
full or closed. Never blocks on a write."""
full, closed or its worker is refused at interpreter exit. Never
blocks on a write."""
key = _dst_key(dst)
with self._changed:
if self._closed or key in self._queued:
@@ -420,30 +421,41 @@ class AutoThumbnailer:
self._queued.add(key)
self._queue.append((key, Path(src), Path(dst), bool(is_video)))
self._busy = True
started = True
if self._start_worker and not self._worker_running:
self._start_worker_locked()
started = self._start_worker_locked()
if not started:
# Refused at interpreter exit: no worker will take it (#375).
self._queue.pop()
self._queued.discard(key)
self._changed.notify_all()
return True
return started
def _start_worker_locked(self):
"""Start the worker (lock held). A start that fails leaves the file
queued and the next enqueue tries again."""
"""Start the worker (lock held); False only when the interpreter is
exiting (#375: logged once at DEBUG, the caller drops the file).
Another start failure leaves the file queued and the next enqueue
tries again."""
worker = threading.Thread(target=self._run, name=WORKER_NAME, daemon=True)
worker.pm_preview_token = _MODULE_TOKEN
worker.pm_preview_close = self.close
self._worker_running = True
try:
worker.start()
except RuntimeError:
except RuntimeError as error:
self._worker_running = False
self._busy = False
if refused_at_shutdown(error):
note_refusal(self._logger, "The automatic preview worker")
return False
self._logger.warning(
"The automatic preview worker could not start; the next output "
"tries again",
exc_info=True,
)
return
return True
self._worker = worker
return True
def _run(self):
try:
@@ -790,8 +802,8 @@ class AutoThumbnailMixin:
Returns ``queued``, ``fresh`` (its preview is current) or ``skipped``
(not a media file under a current root, a link, a preview itself,
already queued, or after the app's cleanup). Never writes on the
caller's thread.
already queued, after the app's cleanup, or when its worker is refused
at interpreter exit). Never writes on the caller's thread.
"""
if self._auto_thumbnails_shut:
return "skipped"
+2 -2
View File
@@ -8,8 +8,8 @@ coverage>=7.16.2
numpy>=2.2.6 # WD14 engine tests
# Code quality
black>=26.10.1
flake8>=7.4.1
black==26.10.1
flake8==7.4.1
# Security scanning
bandit[toml]>=1.9.4
+2 -1
View File
@@ -286,7 +286,8 @@ const continuationText=page=>page.locator('.infinite-continuation > p').first().
async function searchOf(page,term) {
const before=await countText(page);
await page.locator('#search').fill(term);
await expect.poll(()=>countText(page)).not.toBe(before);
// Production's caption reads "Loading images…" until the search's listing answers (#391); the concept's is instant.
await expect.poll(async()=>{const text=await countText(page);return text!==before&&text!=='Loading images…';}).toBe(true);
await expect.poll(()=>page.locator(CARDS).count()).toBeLessThan(FIXTURE_IMAGES);
await settle(page);
const result={count:await countText(page),continuation:await continuationText(page),cards:await page.locator(`${CARDS} .image-open`).evaluateAll(nodes=>nodes.map(node=>node.getAttribute('aria-label')))};
+21 -1
View File
@@ -228,7 +228,8 @@ async function productionSendOf(page,context) {
async function tagOf(page) {
const before=await page.locator('#result-count').textContent();
await panelOf(page).locator('.detail-inner > .tags').last().getByRole('button',{name:'cinematic',exact:true}).click();
await expect.poll(()=>page.locator('#result-count').textContent()).not.toBe(before);
// Production's caption reads "Loading images…" until the tag's listing answers (#391); the concept's search is instant.
await expect.poll(async()=>{const text=await page.locator('#result-count').textContent();return text!==before&&text!=='Loading images…';}).toBe(true);
await settle(page);
return {count:await page.locator('#result-count').textContent(),
cards:await page.locator(`${CARDS} .image-open`).evaluateAll(nodes=>nodes.map(node=>node.getAttribute('aria-label')))};
@@ -328,6 +329,25 @@ test('a recipe-linked image without recipe metadata shows its positive prompt me
await expect(panelOf(page).locator('.detail-footer button')).toHaveText(['Send to ComfyUI','Prompt history']);
});
// #391: a tag chip replaces the collection; until its listing answers the caption says it is loading, never that the
// filter matched nothing ("0 of 0 images" was painted before the response and read by a fast reader).
test('a tag chip collection reads as loading, not empty, until its listing answers (#391)',async({page})=>{
await page.setViewportSize({width:1600,height:1100});
await openProductionPanel(page,'dark');
let release,arrived;
const held=new Promise(resolve=>{release=resolve;}),reached=new Promise(resolve=>{arrived=resolve;});
await page.route(url=>url.pathname==='/prompt_manager/images/output'&&url.searchParams.get('tags')==='cinematic',
async route=>{arrived();await held;await route.continue();},{times:1});
const count=page.locator('#result-count');
await panelOf(page).locator('.detail-inner > .tags').last().getByRole('button',{name:'cinematic',exact:true}).click();
await reached;
await expect(count).toHaveText('Loading images…');
await expect(page.locator(CARDS)).toHaveCount(0);
release();
await expect(count).toHaveText(countCopy('8 of 8 images · 2 positive prompts'));
await expect(page.locator(CARDS)).toHaveCount(8);
});
// Task C3 (D88, #249 PR C): the concept's chip row is the image's own tags (`i.tags`, app.js a402). With those names
// saved as the file's current image tags, production draws the row from them (Tags filter buttons in the panel, plain
// in the viewer, the linked positive's equal tags not repeated) and still equals the concept, in both sizes.
+113 -7
View File
@@ -95,8 +95,14 @@ async function watchProgress(page) {
await page.evaluate(() => {
window.progressSeen = [];
const node = document.getElementById('autoTagProgressPercent');
new MutationObserver(() => window.progressSeen.push(node.textContent))
.observe(node, { childList: true, characterData: true, subtree: true });
// Each change from its own record: frames that land in one task share one
// callback, and the node's text by then is only the last of them.
new MutationObserver(records => {
for (const record of records) {
if (record.type === 'characterData') window.progressSeen.push(record.target.data);
for (const added of record.addedNodes) window.progressSeen.push(added.textContent);
}
}).observe(node, { childList: true, characterData: true, subtree: true });
});
}
@@ -108,25 +114,58 @@ async function runGalleryBatch(page) {
await page.locator('#startAutoTagBtn').click();
await done;
await expect(page.locator('#autoTagProgressModal')).toBeHidden();
await expect(page.getByText(/^Auto tagging complete! Applied tags to \d+ prompts\.$/)).toBeVisible();
// #379: the batch counts images, never prompts.
await expect(page.getByText(/^Auto tagging complete! Tagged \d+ images?\.$/)).toBeVisible();
}
const CLOSE_NOTE = 'Closing this window stops tagging. Tags already added are kept. Prompt Manager\'s Activity page shows the '
+ 'time left.';
async function serverJobs(request) {
const response = await request.get('/prompt_manager/jobs');
await expect(response).toBeOK();
return (await response.json()).jobs;
}
async function runNativeBatch(request) {
const started = await request.post('/prompt_manager/jobs/autotag', { data: { model_type: 'wd14-swinv2', existing: 'skip',
options: { general_threshold: 0.35, character_threshold: 0.85 } } });
expect(started.status()).toBe(202);
const { job } = await started.json();
await expect.poll(async () => (await serverJobs(request)).find(item => item.id === job.id)?.state).toBe('finished');
}
test('gallery batch modal counts progress and closes on complete', async ({ page, request }) => {
await page.goto('/prompt_manager/gallery');
await runGalleryBatch(page);
const seen = await page.evaluate(() => window.progressSeen);
// The fixture's prompts are already tagged, so the batch skips them; its progress frames still move the bar.
expect(seen).toContain('95%');
// The phase frames (model 5%, listing 8%) then the per-image frames (from 10%) move the bar.
expect(seen).toContain('8%');
expect(seen.some(value => parseInt(value, 10) >= 10)).toBe(true);
await expect(page.locator('#autoTagCloseNote')).toHaveText(CLOSE_NOTE);
// The batch ran as the autotag job of the legacy page.
expect((await serverJobs(request)).filter(job => job.kind === 'autotag').map(job => [job.title, job.state]))
.toEqual([['Tag output images (legacy page)', 'finished']]);
expect((await downloads(request)).length).toBe(0);
});
test('admin batch copy names images, not prompts (#379)', async ({ page }) => {
await page.goto('/prompt_manager/admin');
await page.locator('#autoTagBtn').click();
await expect(page.locator('input[name="autoTagMode"][value="skip"] + span')).toHaveText('Skip images already auto-tagged');
await expect(page.getByText('"auto-scanned" is not counted as a real tag')).toHaveCount(0);
await expect(page.locator('#autoTagCloseNote')).toHaveText(CLOSE_NOTE);
await expect(page.locator('#retagConfirmModal h3')).toHaveText('Image Already Auto-tagged');
await expect(page.locator('#retagConfirmModal')).toContainText('A tagging model already tagged this image. Generate new tags?');
});
test('admin review opens the first image', async ({ page }) => {
await page.goto('/prompt_manager/admin');
await page.locator('#autoTagBtn').click();
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
await page.locator('#startReviewBtn').click();
// The first image's prompt is already tagged: confirm a new pass, then the review opens on that image.
await page.locator('#retagConfirmBtn').click();
// #379: no image is auto-tagged yet (prompt tags no longer count), so the review opens on the first image.
await expect(page.locator('#retagConfirmModal')).toBeHidden();
await expect(page.locator('#autoTagReviewModal')).toBeVisible();
await expect(page.locator('#reviewImageName')).not.toHaveText(/^(image_name\.png|Unknown|)$/);
await expect(page.getByText('Failed to start review mode')).toHaveCount(0);
@@ -150,3 +189,70 @@ test('gallery unload works and is refused while the gate is held', async ({ page
await unload.click();
await expect(page.getByText('WD14-SWINV2 model unloaded successfully')).toBeVisible();
});
/** The admin review lists every row, videos too: Skip any image that is not one until the confirmation shows. */
async function untilAutoTagged(page, confirm) {
// Each image's skip check answers before its confirmation or its error shows.
const failed = page.locator('#reviewTagsVisible .text-pm-error');
for (let tries = 0; tries < 5; tries++) {
await expect.poll(async () => (await confirm.isVisible()) || (await failed.count()) > 0).toBe(true);
if (await confirm.isVisible()) return;
const answered = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/single'));
await page.locator('#skipReviewBtn').click();
await answered;
}
await expect(confirm).toBeVisible();
}
// #379: the admin review asks before re-tagging an image a model already tagged, and Apply & Next writes that image's
// tags (its bytes, never its prompt).
test('admin review: an auto-tagged image asks first; Re-tag then Apply & Next tags the image', async ({ page, request }) => {
await runNativeBatch(request);
await page.goto('/prompt_manager/admin');
await page.locator('#autoTagBtn').click();
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
await page.locator('#startReviewBtn').click();
const confirm = page.locator('#retagConfirmModal');
await untilAutoTagged(page, confirm);
await expect(confirm.locator('#retagExistingTags span').first()).toHaveText('sandstone');
await page.locator('#retagConfirmBtn').click();
await expect(page.locator('#autoTagReviewModal')).toBeVisible();
await expect(page.locator('#reviewTagsVisible .tag-chip').first()).toContainText('sandstone');
const applied = page.waitForRequest(item => item.url().endsWith('/prompt_manager/autotag/apply'));
const answer = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/apply'));
await page.locator('#applyReviewBtn').click();
const body = (await applied).postDataJSON();
expect(body.prompt_id).toBeUndefined();
expect(body.sha256).toMatch(/^[0-9a-f]{64}$/);
expect((await answer).status()).toBe(200);
});
test('admin review: Skip All Tagged passes every auto-tagged image', async ({ page, request }) => {
await runNativeBatch(request);
await page.goto('/prompt_manager/admin');
await page.locator('#autoTagBtn').click();
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
await page.locator('#startReviewBtn').click();
const confirm = page.locator('#retagConfirmModal');
await untilAutoTagged(page, confirm);
await page.locator('#retagSkipAllBtn').click();
await expect(page.locator('#retagConfirmModal')).toBeHidden();
await expect(page.getByText('Review complete!')).toBeVisible();
});
test('gallery review: Apply & Next tags the image by its path and bytes', async ({ page }) => {
await page.goto('/prompt_manager/gallery');
await page.locator('#autoTagBtn').click();
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
const suggested = page.waitForRequest(item => item.url().endsWith('/prompt_manager/autotag/single'));
await page.locator('#startReviewBtn').click();
expect((await suggested).postDataJSON().existing).toBe('merge');
await expect(page.locator('#reviewTagsContainer .tag-chip').first()).toContainText('sandstone');
const applied = page.waitForRequest(item => item.url().endsWith('/prompt_manager/autotag/apply'));
const answer = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/apply'));
await page.locator('#applyReviewBtn').click();
const body = (await applied).postDataJSON();
expect(Object.keys(body).sort()).toEqual(['path', 'root_index', 'sha256', 'tags'].filter(key => key in body).sort());
expect(body.prompt_id).toBeUndefined();
expect((await answer).status()).toBe(200);
});
+163
View File
@@ -0,0 +1,163 @@
// System S13 (#330): the legacy admin and gallery pages open native System for their System tools. Each replaced
// control is a plain link (`workspace#/system/<id>`, a new tab), and the legacy flows S13 keeps still work.
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
const PAGES = Object.freeze({ admin: '/prompt_manager/admin', gallery: '/prompt_manager/gallery' });
const LABELS = Object.freeze({
overview: 'Overview',
appearance: 'Appearance',
sources: 'Sources & scans',
integrations: 'Integrations',
tagging: 'Auto-tagging',
maintenance: 'Maintenance',
backups: 'Backup & export',
diagnostics: 'Diagnostics & logs',
});
// One row per System link of the design's §2 map: the page, the control's label, how to find it, the section it opens
// and what a test must arrange before the control shows.
const LINKS = Object.freeze([
{ page: 'admin', label: 'Diagnostics', locate: page => page.locator('#diagnosticsBtn'), section: 'diagnostics' },
{ page: 'admin', label: 'View Logs', locate: page => page.locator('#logsBtn'), section: 'diagnostics' },
]);
async function post(request, url, data) {
await expect(await request.post(url, data === undefined ? {} : { data })).toBeOK();
}
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await post(request, '/__test__/reset');
await post(request, '/__test__/system/reset');
await post(request, '/__test__/tagging/reset');
// The 3.2.0 update notice would cover the admin page's toolbar.
await page.addInitScript(() => { localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
page.legacyErrors = [];
page.on('pageerror', error => page.legacyErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await post(request, '/__test__/tagging/reset');
await post(request, '/__test__/system/reset');
expect(page.legacyErrors).toEqual([]);
});
/** Opens the legacy page with the link showing; returns the link. */
async function openLink(page, request, link) {
if (link.arrange) await link.arrange(page, request);
await page.goto(PAGES[link.page]);
if (link.reveal) await link.reveal(page);
const control = link.locate(page);
await expect(control).toBeVisible();
return control;
}
for (const link of LINKS) {
test(`legacy ${link.page} ${link.label} opens System ${link.section}`, async ({ page, context, request }) => {
const control = await openLink(page, request, link);
const legacyUrl = page.url();
await expect(control).toHaveAttribute('href', `workspace#/system/${link.section}`);
await expect(control).toHaveAttribute('target', '_blank');
await expect(control).toHaveAttribute('rel', 'noopener');
const opened = context.waitForEvent('page');
await control.click();
const system = await opened;
await expect(system).toHaveURL(new RegExp(`/prompt_manager/workspace#/system/${link.section}$`));
// System's page heading is "System"; the open section is the pressed button of its section switcher.
const main = system.locator('#workspace-main');
await expect(main.getByRole('heading', { level: 1, name: 'System', exact: true })).toBeVisible();
await expect(main.getByRole('group', { name: 'System sections', exact: true })
.getByRole('button', { name: LABELS[link.section], exact: true })).toHaveAttribute('aria-pressed', 'true');
expect(page.url()).toBe(legacyUrl);
await system.close();
});
}
test('a System link sends no request from the legacy page', async ({ page, context, request }) => {
for (const link of LINKS) {
const control = await openLink(page, request, link);
// The page's own start-up reads (list, film strips, settings) finish first; then the click must add none.
await page.waitForLoadState('networkidle');
const sent = [];
const record = item => sent.push(`${item.method()} ${item.url()}`);
page.on('request', record);
const opened = context.waitForEvent('page');
await control.click();
const system = await opened;
await system.waitForLoadState('domcontentloaded');
page.off('request', record);
expect(sent, `${link.page} ${link.label}`).toEqual([]);
await system.close();
}
});
test('kept admin flows still work', async ({ page, context }) => {
await page.goto(PAGES.admin);
const texts = page.locator('#resultsList .prompt-text');
await expect(texts.first()).toBeVisible();
// K1: sorting reorders the list.
for (const [value, order] of [['text_asc', 1], ['text_desc', -1]]) {
const listed = page.waitForResponse(item => /\/prompt_manager\/(recent|search)\?/.test(item.url()) && item.url().includes(`sort=${value}`));
await page.locator('#sortBy').selectOption(value);
await listed;
await expect.poll(async () => {
const shown = await texts.allTextContents();
// The server sorts case-insensitively (SQLite NOCASE), not by locale.
const key = text => text.toLowerCase();
const sorted = [...shown].sort((a, b) => order * (key(a) < key(b) ? -1 : key(a) > key(b) ? 1 : 0));
return shown.length > 1 && shown.join('\n') === sorted.join('\n');
}).toBe(true);
}
// K3: Add Prompt saves a prompt the list then shows.
await page.locator('#addPromptBtn').click();
await page.locator('#addPromptText').fill('Kept flow lighthouse');
const saved = page.waitForResponse(item => item.request().method() === 'POST' && item.url().endsWith('/prompt_manager/save'));
await page.locator('#saveNewPromptBtn').click();
expect((await saved).ok()).toBe(true);
await expect(page.locator('#addPromptModal')).toBeHidden();
await expect(texts.filter({ hasText: 'Kept flow lighthouse' })).toBeVisible();
// K1: a search narrows the list, and a selection enables the bulk buttons.
const all = await texts.allTextContents();
const word = all[0].trim().split(/\s+/)[0];
const searched = page.waitForResponse(item => item.url().includes('/prompt_manager/search?') && item.url().includes('text='));
await page.locator('#searchText').fill(word);
await page.locator('#searchBtn').click();
await searched;
await expect.poll(async () => {
const shown = await texts.allTextContents();
return shown.length > 0 && shown.length < all.length && shown.every(text => text.toLowerCase().includes(word.toLowerCase()));
}).toBe(true);
await expect(page.locator('#bulkDeleteBtn')).toBeDisabled();
await page.locator('#resultsList .prompt-checkbox').first().check();
for (const id of ['#bulkDeleteBtn', '#bulkTagBtn', '#bulkCategoryBtn']) await expect(page.locator(id)).toBeEnabled();
// K4: Stats reads the statistics again; Export downloads the prompts.
const stats = page.waitForResponse(item => item.url().endsWith('/prompt_manager/stats') && item.ok());
await page.locator('#statsBtn').click();
await stats;
await expect(page.getByText('Stats refreshed successfully!')).toBeVisible();
const download = page.waitForEvent('download');
await page.locator('#exportBtn').click();
expect((await download).suggestedFilename()).toMatch(/^prompts-\d{4}-\d{2}-\d{2}\.json$/);
// K5: Scan Images opens its dialog (with the quick backup) without starting a scan.
await page.locator('#scanBtn').click();
await expect(page.locator('#scanModal')).toBeVisible();
await expect(page.locator('#quickBackupBtn')).toBeVisible();
await page.locator('#cancelScan').click();
await expect(page.locator('#scanModal')).toBeHidden();
// K7: Metadata and Gallery open their pages in a new tab.
for (const [id, path] of [['#metadataBtn', '/prompt_manager/gallery.html'], ['#galleryBtn', '/prompt_manager/gallery']]) {
const opened = context.waitForEvent('page');
await page.locator(id).click();
const tab = await opened;
await tab.waitForLoadState('domcontentloaded');
expect(new URL(tab.url()).pathname).toBe(path);
await tab.close();
}
});
+30 -15
View File
@@ -127,9 +127,9 @@ function mapTaggingConcept(reference, { template, phone }) {
actions: { ...reference.actions, buttons: [preview, ['Download again', ''], ['Unload', '']] },
// D134: the prompt-tag batch, reviewed first.
operations: [{ tree: ['div', 'div.row-actions'], title: 'Tag output images', description: BATCH, actions: [['Review', '']] }],
// D132: the engine select shows its whole chosen option at every width (at 390 the concept's 120px cap cuts it;
// `#tag-model` alone may grow there). Existing tags keeps the concept's own fit, cut or not.
selectsFit: reference.selectsFit.map((fit, index) => (index === 0 ? true : fit)),
// D132 (#398): every select shows its whole chosen option at every width (at 390 the concept's 120px cap cuts
// them; production lets each grow past it).
selectsFit: reference.selectsFit.map(() => true),
};
}
@@ -168,16 +168,28 @@ for (const theme of THEMES) for (const [width, height] of SIZES) {
});
}
// D132 below the concept's smallest size: on a 320px phone the engine select still shows its whole option.
for (const theme of THEMES) test(`auto-tagging engine select fits at 320 wide ${theme}`, async ({ page }) => {
await page.setViewportSize({ width: 320, height: 700 });
const production = await productionTagging(page, theme);
for (const model of ['hf', 'gguf', 'wd14-swinv2']) {
await page.locator('#workspace-main #tag-model').selectOption(model);
expect((await page.evaluate(measureTagging, '#workspace-main')).selectsFit[0], model).toBe(true);
}
expect(production.pageScroll).toBe(false);
});
// D132 (#398) at phone width and below the concept's smallest size: every option of every select, once chosen, shows
// whole, with no page scroll.
for (const theme of THEMES) for (const width of [390, 320]) {
test(`auto-tagging selects fit every option at ${width} wide ${theme}`, async ({ page }, testInfo) => {
await page.setViewportSize({ width, height: 844 });
await productionTagging(page, theme);
await page.locator('#workspace-main #tag-existing').selectOption('replace_after_review');
await page.locator('#workspace-main section.surface').screenshot({ path: testInfo.outputPath(`production-tagging-selects-${theme}-${width}.png`) });
const selects = page.locator('#workspace-main section.surface > .setting-row > select');
await expect(selects).toHaveCount(2);
for (let index = 0; index < 2; index += 1) {
const select = selects.nth(index);
const values = await select.evaluate(node => [...node.options].map(option => option.value));
for (const value of values) {
await select.selectOption(value);
const measured = await page.evaluate(measureTagging, '#workspace-main');
expect(measured.selectsFit, `${await select.getAttribute('id')}=${value}`).toEqual([true, true]);
expect(measured.pageScroll, value).toBe(false);
}
}
});
}
test('auto-tagging comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
@@ -222,6 +234,9 @@ test('auto-tagging comparison detects mutations', async ({ page, request }) => {
const undoSqueeze = await mutate(page, { css: '#workspace-main #tag-model { max-width: 60px !important; min-width: 0 !important; }' });
expect(await paths()).toEqual(expect.arrayContaining(['selectsFit.0']));
await undoSqueeze();
const undoSqueezeExisting = await mutate(page, { css: '#workspace-main #tag-existing { max-width: 60px !important; min-width: 0 !important; }' });
expect(await paths()).toEqual(expect.arrayContaining(['selectsFit.1']));
await undoSqueezeExisting();
// A row that wraps where the concept's does not.
const undoWrap = await mutate(page, { css: '#workspace-main .setting-row:has(> #tag-existing) { flex-wrap: wrap; }' });
expect(await paths()).toEqual(['rows.1.flexWrap']);
@@ -233,14 +248,14 @@ test('auto-tagging comparison detects mutations', async ({ page, request }) => {
expect(diffInventory(expected, await measure())).toEqual([]);
});
// D132 at phone width, unmapped: only the engine row wraps; Existing tags keeps the concept's cut select.
// D132 at phone width: only the engine row wraps; Existing tags grows beside its text instead (#398).
test('auto-tagging at 390 wraps only the engine row', async ({ page, request }) => {
await page.setViewportSize({ width: 390, height: 844 });
const concept = await conceptTagging(page, 'dark');
const production = await productionTagging(page, 'dark');
expect(diffInventory(mapTaggingConcept(concept, await savedTemplate(request, page)), production)).toEqual([]);
expect(production.rows.map(row => row.flexWrap)).toEqual(['wrap', 'nowrap', 'nowrap', 'nowrap', 'nowrap']);
expect(production.selectsFit[1]).toBe(concept.selectsFit[1]);
expect(production.selectsFit).toEqual([true, true]);
});
/** Runs in the page: the open review dialog: its title, body and the concept's tokens. */
+7 -4
View File
@@ -132,7 +132,8 @@ test('the gallery cache clear keeps preferences and other tools caches', async (
}))).toEqual({ theme: 'light', other: 'keep', prefixed: 'keep', notice: '3.2.0', caches: ['other-tool-gallery-thumbnails'] });
});
test('actual gallery review generates and persists tags for the selected root', async ({ page, request }) => {
// #379: the reviewed tags are that image's own tags; no prompt changes.
test('actual gallery review generates and persists tags for the selected root\'s image', async ({ page, request }) => {
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(2);
await page.locator('#autoTagBtn').click();
@@ -140,13 +141,15 @@ test('actual gallery review generates and persists tags for the selected root',
await page.locator('#startReviewBtn').click();
const generation = await generated;
expect(generation.request().postDataJSON().root_index).toBe(0);
await expect(page.locator('#reviewTagsContainer')).toContainText('generated');
await expect(page.locator('#reviewTagsContainer')).toContainText('sandstone');
const result = page.waitForResponse(response => response.url().endsWith('/prompt_manager/autotag/apply'));
await page.locator('#applyReviewBtn').click();
expect((await (await result).json()).success).toBe(true);
const data = await (await request.get('/prompt_manager/recent?limit=20')).json();
expect(data.results.find(prompt => prompt.text === 'Cat portrait').tags).toContain('generated');
expect(data.results.find(prompt => prompt.text === 'Dog landscape').tags).not.toContain('generated');
expect(data.results.find(prompt => prompt.text === 'Cat portrait').tags).toEqual(['cats']);
expect(data.results.find(prompt => prompt.text === 'Dog landscape').tags).toEqual(['dogs']);
const tagged = await (await request.get('/prompt_manager/images/output?mode=workspace&tags=sandstone')).json();
expect(tagged.images.map(image => `${image.root_index}:${image.path}`)).toEqual(['0:shared/same.png']);
});
test('actual Tags page reloads the selected filter and opens its image viewer', async ({ page }) => {
@@ -47,6 +47,17 @@ async function serverJson(request, url) {
return response.json();
}
/**
* The restore job's own end (#361): its progress stream closes when the job takes its final state, however long the
* disk-bound publish takes (bounded by the test, not by a 5 s expect), and the job finished. The toast the page shows
* for it is then the page's reaction alone, checked with the default timeout.
*/
async function restoreFinished(request, id) {
await expect(await request.get(`/prompt_manager/jobs/${id}/events`)).toBeOK();
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
expect([job.kind, job.state]).toEqual(['restore', 'finished']);
}
async function openBackups(page) {
// A fresh page each time: the drafts live for the page session (Q10), and a hash-only goto would not reload.
if (page.url().endsWith('#/system/backups')) await page.reload();
@@ -216,6 +227,7 @@ test('Restore starts the job, opens Activity; views reload after it finishes', a
const { job } = await (await applied).json();
expect([job.kind, job.title, job.cancellable]).toEqual(['restore', 'Database restore', false]);
await expect(page).toHaveURL(`${workspace}#/activity/${job.id}`);
await restoreFinished(request, job.id);
await expect(toast(page)).toHaveText(/^Database restore finished · Restored \d+ prompts?, /);
// The library is the backup's again, and the page says so without a reload: the sidebar counts were read again.
expect(await libraryTotal(request)).toBe(restoredTotal);
@@ -244,7 +256,8 @@ test('a finished restore shows the open page again from the restored data', asyn
const card = page.locator('#workspace-main article.prompt-card').filter({ hasText: 'S9 prompt the restore removes' });
await expect(card).toHaveCount(1);
await expect(await request.post('/__test__/release')).toBeOK();
await applied;
const { job } = await (await applied).json();
await restoreFinished(request, job.id);
await expect(toast(page)).toHaveText(/^Database restore finished · /);
// The shelf was read again from the restored library, on the same route.
await expect(card).toHaveCount(0);
@@ -422,6 +435,7 @@ test('a refused restore reopens its review: Restore retries it, Discard releases
await dialog(page).getByRole('button', { name: 'Restore', exact: true }).click();
const { job } = await (await applied).json();
await expect(page).toHaveURL(`${workspace}#/activity/${job.id}`);
await restoreFinished(request, job.id);
await expect(toast(page)).toHaveText(/^Database restore finished · /);
expect(await stagesOf(request)).toBe(0);
});
+35
View File
@@ -513,6 +513,41 @@ test('search finds an image by its image tag', async ({ page, request }) => {
await expect(card(page, UNLINKED)).toBeVisible();
});
// #379 CUJ 8: the Tags filter finds images by their own image tags through the tag index; no prompt carries them, and
// selecting a match shows its prompt. A linked prompt's tag still combines with the image tags.
test('the Tags filter finds images by image tags only, and selecting one shows its prompt (#379)', async ({ page, request }) => {
await saveImageTags(request, FIRST, ['beach', '1girl', 'waves']);
await saveImageTags(request, SECOND, ['beach', 'waves']);
await saveImageTags(request, UNLINKED, ['beach', '1girl', 'waves', 'gulls']);
await page.goto(images);
await expect(page.locator('.gallery-body article[data-image-key]')).toHaveCount(manifest.files.length);
const filters = page.locator('#workspace-main search.toolbar').getByRole('button', { name: /^Filters(?: · \d+)?$/ });
const refine = page.getByRole('dialog', { name: 'Refine this view', exact: true });
const applyTags = async value => {
await filters.click();
await refine.getByLabel('Tags', { exact: true }).fill(value);
const listing = page.waitForRequest(item => item.url().includes('/images/output?')
&& new URL(item.url()).searchParams.get('tags') !== null);
await refine.getByRole('button', { name: 'Apply filters', exact: true }).click();
await listing;
};
await applyTags('beach, 1girl, waves');
// Every listed image whose image tags hold all three; SECOND lacks 1girl.
await expect(page.locator('.gallery-body article[data-image-key]')).toHaveCount(2);
await expect(card(page, FIRST)).toBeVisible();
await expect(card(page, UNLINKED)).toBeVisible();
await expect(card(page, SECOND)).toHaveCount(0);
// The newest match is selected; the inspector shows its linked prompt.
await infoButton(page, FIRST).click();
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
await expect(panel(page).locator('.panel-heading h2')).toHaveText(manifest.title);
// The linked positive's own tag combines: SECOND has beach and waves itself and minimal through its prompt.
await applyTags(`beach, waves, ${manifest.prompt_tags[0]}`);
await expect(page.locator('.gallery-body article[data-image-key]')).toHaveCount(2);
await expect(card(page, FIRST)).toBeVisible();
await expect(card(page, SECOND)).toBeVisible();
});
// Task C4 rulings: a save re-reads the selected file's details without a loading flash, and under a Tags filter or a
// search the loaded listing is re-read in place (scroll and selection kept while the file still matches).
const cards = page => page.locator('.gallery-body article[data-image-key]');
+14 -2
View File
@@ -35,7 +35,8 @@ read. These per-server hooks only change what a System test needs on top:
previews for Convert outdated;
- ``POST /__test__/system/jxl-encoder`` ``{available}`` (#317) makes the
server read as having (``cjxl``) or lacking a JPEG XL encoder without
probing; reset restores the thumbnail settings and the real detection.
probing, whatever the host has installed (#371: ``false`` stands in for the
detector itself); reset restores the thumbnail settings and the real detection.
- ``POST /__test__/system/duplicate-prompts`` (S7) adds two prompts whose
text differs only in case and spacing (a duplicate prompt text group);
@@ -118,6 +119,11 @@ def register_system_commands(app, routes, api, root: Path):
}
from py.api import thumbnail_encode
real_detect = thumbnail_encode._detect
def _no_encoder():
return None
monitor = getattr(image_monitor, "_monitor_instance", None)
failures: dict[str, int] = {}
@@ -147,6 +153,7 @@ def register_system_commands(app, routes, api, root: Path):
api._health_report = None
for (owner, name), value in host_values.items():
setattr(owner, name, value)
thumbnail_encode._detect = real_detect
thumbnail_encode.reset_encoders()
image_monitor._monitor_instance = monitor
api.__dict__.pop("_library_resolver", None)
@@ -298,7 +305,12 @@ def register_system_commands(app, routes, api, root: Path):
available = data.get("available")
if not isinstance(available, bool):
raise web.HTTPBadRequest()
# Test-only: stand in the cached detection (no probe).
# Test-only: the fixture, not the host, decides (#371). Without an encoder the
# detector itself stands in, so a re-probe (a "none found" is probed again on
# every settings read) cannot find the host's real encoder; with one the cached
# answer stands, and a failed write still probes the real detector as before.
# Reset restores the real detector.
thumbnail_encode._detect = real_detect if available else _no_encoder
with thumbnail_encode._detect_lock:
thumbnail_encode._detected = "cjxl" if available else None
return web.json_response({"success": True})
+16
View File
@@ -0,0 +1,16 @@
"""Hang guards for waits on disk-bound work (tests only).
A test that waits for a job or handoff whose work is file and SQLite I/O
(a safety backup with fsync, a restore, hashing and unlinking files) waits
on that state; the ceiling below only ends a real hang. Hosted Windows
runners stall disk I/O for tens of seconds at a time: a restore took over
10 s (merge queue, #382), and on PR #405 every test setup in a 50 s window
took up to 9 s against 0.1-0.6 s around it while the test bodies stayed
fast, so a one-file duplicate cleanup missed a 10 s bound (#409).
The ceiling is per wait and stays under the suite's 120 s per-test hang
watchdog. A test with several such waits in a row can reach the watchdog
first; either way a real hang fails, the watchdog with every thread's stack.
"""
DISK_BOUND_SECONDS = 90
+16
View File
@@ -113,3 +113,19 @@ by their own suites. The expectations start as the observed baseline. A change
that alters an outcome edits the vector in the same commit.
`tests/js/legacy-metadata-characterization.test.js` pins what
the legacy viewers show on the same files.
# Legacy contract snapshot
`legacy_contract_snapshot.json` records what the legacy admin and gallery pages
talk to while System S13 (#330) moves their tools to native System: every
registered route as `"<METHOD> <path>"` (`routes`) and the key paths of
`GET /prompt_manager/settings` to depth 2 (`settings_keys`; list items give the
union of their keys as `name[].key`). `tests/test_legacy_contract_snapshot.py`
compares a temp-database API against it. Record it again only for an intended
change, and say why in the PR body:
```sh
PM_RECORD_LEGACY_SNAPSHOT=1 python3 -m unittest tests.test_legacy_contract_snapshot
```
Cutover (#334) removes the test and this fixture.
+200
View File
@@ -0,0 +1,200 @@
{
"routes": [
"DELETE /prompt_manager/delete/{prompt_id}",
"DELETE /prompt_manager/image-annotations/capture/{token}",
"DELETE /prompt_manager/images/favorite",
"DELETE /prompt_manager/images/{image_id}",
"DELETE /prompt_manager/pairs/{pair_id}",
"DELETE /prompt_manager/prompts/{prompt_id}/tags",
"DELETE /prompt_manager/restore/preview/{token}",
"DELETE /prompt_manager/saved-views/{view_id}",
"DELETE /prompt_manager/tags/{tag_name}",
"GET /prompt_manager/admin",
"GET /prompt_manager/autotag/models",
"GET /prompt_manager/backup",
"GET /prompt_manager/backup/status",
"GET /prompt_manager/categories",
"GET /prompt_manager/diagnostics",
"GET /prompt_manager/export",
"GET /prompt_manager/gallery",
"GET /prompt_manager/gallery.html",
"GET /prompt_manager/gallery/subfolders",
"GET /prompt_manager/history/runs/{run_id}",
"GET /prompt_manager/history/runs/{run_id}/output/current/inspection",
"GET /prompt_manager/history/runs/{run_id}/output/current/media",
"GET /prompt_manager/history/runs/{run_id}/output/current/thumbnail",
"GET /prompt_manager/history/runs/{run_id}/outputs",
"GET /prompt_manager/image-annotations/capture/{token}",
"GET /prompt_manager/images/all",
"GET /prompt_manager/images/inspect/{image_path}",
"GET /prompt_manager/images/output",
"GET /prompt_manager/images/prompt/{image_path}",
"GET /prompt_manager/images/recent",
"GET /prompt_manager/images/search",
"GET /prompt_manager/images/serve/{filepath}",
"GET /prompt_manager/images/thumbnails/plan",
"GET /prompt_manager/images/thumbnails/status",
"GET /prompt_manager/images/{image_id}/file",
"GET /prompt_manager/jobs",
"GET /prompt_manager/jobs/{job_id}",
"GET /prompt_manager/jobs/{job_id}/events",
"GET /prompt_manager/jobs/{job_id}/report",
"GET /prompt_manager/js/{filepath}",
"GET /prompt_manager/lib/{filepath}",
"GET /prompt_manager/library",
"GET /prompt_manager/library/export",
"GET /prompt_manager/library/options",
"GET /prompt_manager/library/{positive_id}",
"GET /prompt_manager/library/{positive_id}/history",
"GET /prompt_manager/library/{positive_id}/history/pairs/{pair_id}/newest",
"GET /prompt_manager/library/{positive_id}/images",
"GET /prompt_manager/logs",
"GET /prompt_manager/logs/config",
"GET /prompt_manager/logs/download/{filename}",
"GET /prompt_manager/logs/files",
"GET /prompt_manager/logs/stats",
"GET /prompt_manager/lora/detect",
"GET /prompt_manager/lora/status",
"GET /prompt_manager/lora/trigger-words",
"GET /prompt_manager/maintenance/duplicates/{job_id}",
"GET /prompt_manager/pairs",
"GET /prompt_manager/pairs/export",
"GET /prompt_manager/pairs/{pair_id}",
"GET /prompt_manager/prompts/{prompt_id}/images",
"GET /prompt_manager/recent",
"GET /prompt_manager/review/events",
"GET /prompt_manager/review/items",
"GET /prompt_manager/review/items/{item_id}",
"GET /prompt_manager/saved-views",
"GET /prompt_manager/scan/status",
"GET /prompt_manager/scan_duplicates",
"GET /prompt_manager/scan_duplicates/status",
"GET /prompt_manager/scan_output_dir",
"GET /prompt_manager/search",
"GET /prompt_manager/settings",
"GET /prompt_manager/stats",
"GET /prompt_manager/subfolders",
"GET /prompt_manager/system/diagnostics",
"GET /prompt_manager/system/health",
"GET /prompt_manager/system/library",
"GET /prompt_manager/system/overview",
"GET /prompt_manager/system/statistics",
"GET /prompt_manager/tags",
"GET /prompt_manager/tags/filter",
"GET /prompt_manager/tags/stats",
"GET /prompt_manager/tags/suggest",
"GET /prompt_manager/tags/{tag_name}/prompts",
"GET /prompt_manager/test",
"GET /prompt_manager/vocabulary/categories",
"GET /prompt_manager/vocabulary/tags",
"GET /prompt_manager/vocabulary/tags/{tag_id}/impact",
"GET /prompt_manager/web",
"GET /prompt_manager/workspace",
"PATCH /prompt_manager/library/{positive_id}",
"PATCH /prompt_manager/pairs/{pair_id}",
"PATCH /prompt_manager/saved-views/{view_id}",
"PATCH /prompt_manager/vocabulary/tags/{tag_id}",
"POST /prompt_manager/autotag/apply",
"POST /prompt_manager/autotag/batch/preview",
"POST /prompt_manager/autotag/download/{model_type}",
"POST /prompt_manager/autotag/single",
"POST /prompt_manager/autotag/start",
"POST /prompt_manager/autotag/unload",
"POST /prompt_manager/bulk/category",
"POST /prompt_manager/bulk/delete",
"POST /prompt_manager/bulk/tags",
"POST /prompt_manager/cleanup",
"POST /prompt_manager/delete_duplicate_images",
"POST /prompt_manager/diagnostics/test-link",
"POST /prompt_manager/gallery/rescan",
"POST /prompt_manager/image-annotations/observe",
"POST /prompt_manager/image-annotations/suggest",
"POST /prompt_manager/images/clear-thumbnails",
"POST /prompt_manager/images/generate-thumbnails",
"POST /prompt_manager/images/generate-thumbnails/progress",
"POST /prompt_manager/images/link",
"POST /prompt_manager/images/reveal",
"POST /prompt_manager/jobs/autotag",
"POST /prompt_manager/jobs/duplicates",
"POST /prompt_manager/jobs/image-tags",
"POST /prompt_manager/jobs/model-download",
"POST /prompt_manager/jobs/scan",
"POST /prompt_manager/jobs/thumbnails",
"POST /prompt_manager/jobs/{job_id}/cancel",
"POST /prompt_manager/library",
"POST /prompt_manager/library/{positive_id}/variants",
"POST /prompt_manager/logs/config",
"POST /prompt_manager/logs/truncate",
"POST /prompt_manager/lora/enable",
"POST /prompt_manager/lora/import/apply",
"POST /prompt_manager/lora/import/preview",
"POST /prompt_manager/lora/refresh-cache",
"POST /prompt_manager/lora/scan",
"POST /prompt_manager/maintenance",
"POST /prompt_manager/maintenance/apply",
"POST /prompt_manager/maintenance/preview",
"POST /prompt_manager/pairs",
"POST /prompt_manager/prompts/tags",
"POST /prompt_manager/prompts/{prompt_id}/tags",
"POST /prompt_manager/restore",
"POST /prompt_manager/restore/apply",
"POST /prompt_manager/restore/preview",
"POST /prompt_manager/review/items/{item_id}/decision",
"POST /prompt_manager/save",
"POST /prompt_manager/saved-views",
"POST /prompt_manager/scan",
"POST /prompt_manager/scan/cancel",
"POST /prompt_manager/scan_duplicates/stream",
"POST /prompt_manager/settings",
"POST /prompt_manager/settings/validate-source",
"POST /prompt_manager/system/health",
"POST /prompt_manager/system/library/keep",
"POST /prompt_manager/system/library/move",
"POST /prompt_manager/tags/merge",
"POST /prompt_manager/vocabulary/categories/apply",
"POST /prompt_manager/vocabulary/categories/preview",
"POST /prompt_manager/vocabulary/tags",
"POST /prompt_manager/vocabulary/tags/{tag_id}/delete",
"PUT /prompt_manager/image-annotations/tags",
"PUT /prompt_manager/images/favorite",
"PUT /prompt_manager/prompts/{prompt_id}",
"PUT /prompt_manager/prompts/{prompt_id}/rating",
"PUT /prompt_manager/tags/{tag_name}"
],
"settings_keys": [
"monitor",
"monitor.observer_alive",
"monitor.present",
"monitor.scheduled",
"revision",
"settings",
"settings.autotag_character_threshold",
"settings.autotag_existing",
"settings.autotag_general_threshold",
"settings.autotag_model_type",
"settings.autotag_suggestion_limit",
"settings.autotag_template",
"settings.cpu_count",
"settings.gallery_root_path",
"settings.gallery_root_paths",
"settings.infinite_scroll",
"settings.monitored_directories",
"settings.monitoring_enabled",
"settings.result_timeout",
"settings.safety_backup",
"settings.thumbnail_edge",
"settings.thumbnail_format",
"settings.thumbnail_quality",
"settings.webui_display_mode",
"settings.worker_threads",
"sources",
"sources.auto_detected",
"sources.configured",
"sources.effective",
"success",
"thumbnail_encoders",
"thumbnail_encoders.jpeg",
"thumbnail_encoders.jxl",
"thumbnail_encoders.webp"
]
}
+24 -9
View File
@@ -36,33 +36,48 @@ function fixture() {
const reply = (index, data) => requests[index].resolve({ json: async () => data });
return { manager, requests, reply, notifications };
}
test('gallery review retains root identity and applies tags with the returned prompt ID', async () => {
const SHA = 'a'.repeat(64);
test('gallery review retains root identity and applies the tags to the image by its sha256 (#379)', async () => {
const { manager, requests, reply } = fixture();
const load = manager.loadNextReviewImage();
assert.equal(requests[0].body.root_index, 1);
reply(0, { success: true, prompt_id: 42, tags: ['smile'] });
assert.equal(requests[0].body.existing, 'merge');
reply(0, { success: true, prompt_id: 42, tags: ['smile'], sha256: SHA, image_tags: [] });
await load;
manager.loadNextReviewImage = () => {};
const apply = manager.applyReviewTags();
assert.deepEqual(requests[1].body, { prompt_id: 42, tags: ['smile'] });
assert.equal(requests[1].url, '/prompt_manager/autotag/apply');
assert.deepEqual(requests[1].body, ImageHelpers.autotagApplyBody(
{ id: 'digest', path: 'same.png', root_index: 1 }, SHA, ['smile']));
assert.equal('prompt_id' in requests[1].body, false);
reply(1, { success: true });
await apply;
});
test('unlinked images do not submit an invalid apply request or advance', async () => {
test('an unlinked image applies to the image like a linked one', async () => {
const { manager, requests, reply } = fixture();
const load = manager.loadNextReviewImage();
reply(0, { success: true, prompt_id: null, tags: ['smile'] });
reply(0, { success: true, prompt_id: null, tags: ['smile'], sha256: SHA, image_tags: [] });
await load;
manager.loadNextReviewImage = () => {};
const apply = manager.applyReviewTags();
if (requests[1]) reply(1, { success: false, error: "Unexpected apply" });
assert.equal(requests[1].body.sha256, SHA);
assert.deepEqual(requests[1].body.tags, ['smile']);
reply(1, { success: true });
await apply;
assert.equal(requests.length, 1);
assert.equal(manager.autoTagState.reviewIndex, 0);
});
test('an answer without a captured sha256 never sends an apply', async () => {
const { manager, requests, reply } = fixture();
const load = manager.loadNextReviewImage();
reply(0, { success: true, prompt_id: 42, tags: ['smile'] });
await load;
manager.loadNextReviewImage = () => {};
await manager.applyReviewTags();
assert.equal(requests.filter(r => r.url === '/prompt_manager/autotag/apply').length, 0);
});
test('a failed apply keeps the current image available for retry', async () => {
const { manager, requests, reply } = fixture();
const load = manager.loadNextReviewImage();
reply(0, { success: true, prompt_id: 42, tags: ['smile'] });
reply(0, { success: true, prompt_id: 42, tags: ['smile'], sha256: SHA, image_tags: [] });
await load;
const apply = manager.applyReviewTags();
reply(1, { success: false, error: 'Try again' });
+26
View File
@@ -5,6 +5,8 @@ const {
formatImageCaption,
workflowDocumentHtml,
autotagSingleBody,
autotagApplyBody,
autotagCompleteMessage,
fallbackImageSource,
} = require("../../web/js/image-helpers.js");
@@ -121,3 +123,27 @@ test('metadata lookup preserves encoded filename characters and works behind a p
assert.equal(metadataLookupUrl('/prompt_manager/images/serve/a.png?v=123'), '/prompt_manager/images/prompt/a.png');
assert.equal(metadataLookupUrl('/elsewhere/a.png'), null);
});
test("autotagSingleBody asks to skip auto-tagged images only when told (#379)", () => {
assert.equal(autotagSingleBody({ id: 1 }, WD14).existing, "merge");
assert.equal(autotagSingleBody({ id: 1 }, { ...WD14, existing: "skip" }).existing, "skip");
assert.equal(autotagSingleBody({ id: 1 }, { ...WD14, existing: "odd" }).existing, "merge");
});
test("autotagApplyBody names the image, its bytes and the tags, never a prompt (#379)", () => {
const sha = "a".repeat(64);
assert.deepEqual(autotagApplyBody({ id: 4, prompt_id: 7 }, sha, ["bird"]), { image_id: 4, sha256: sha, tags: ["bird"] });
assert.deepEqual(autotagApplyBody({ id: "digest", path: "sub/a.png", root_index: 1 }, sha, []),
{ path: "sub/a.png", root_index: 1, sha256: sha, tags: [] });
assert.deepEqual(autotagApplyBody({ id: 2 }, undefined, null), { image_id: 2, sha256: "", tags: [] });
const tags = ["x"];
autotagApplyBody({ id: 1 }, sha, tags).tags.push("y");
assert.deepEqual(tags, ["x"]);
});
test("autotagCompleteMessage counts images (#379)", () => {
assert.equal(autotagCompleteMessage(2), "Auto tagging complete! Tagged 2 images.");
assert.equal(autotagCompleteMessage(1), "Auto tagging complete! Tagged 1 image.");
assert.equal(autotagCompleteMessage(0), "Auto tagging complete! Tagged 0 images.");
assert.equal(autotagCompleteMessage(undefined), "Auto tagging complete! Tagged 0 images.");
});
+24
View File
@@ -97,6 +97,30 @@ test("connect reports a non-2xx response through onerror and never onmessage", a
assert.equal(source.readyState, source.CLOSED);
});
test("a non-2xx JSON answer's error sentence reaches onerror as serverMessage", async () => {
let error = null;
const fetchImpl = async () => new Response(
JSON.stringify({ success: false, error: "Auto-tagging is already running" }),
{ status: 409, headers: { "Content-Type": "application/json" } },
);
const source = connect("/x", { fetchImpl });
source.onerror = (err) => { error = err; };
await until(() => error !== null);
assert.match(error.message, /409/);
assert.equal(error.status, 409);
assert.equal(error.serverMessage, "Auto-tagging is already running");
});
test("a non-2xx answer without a JSON error has no serverMessage", async () => {
let error = null;
const fetchImpl = async () => new Response("nope", { status: 500 });
const source = connect("/x", { fetchImpl });
source.onerror = (err) => { error = err; };
await until(() => error !== null);
assert.equal(error.status, 500);
assert.equal(error.serverMessage, undefined);
});
test("connect reports a network failure through onerror", async () => {
let error = null;
const source = connect("/x", { fetchImpl: async () => { throw new TypeError("Failed to fetch"); } });
+17
View File
@@ -234,6 +234,23 @@ test('the count line reads N of M images and K positive prompts only when known'
assert.equal(countLine({loaded:1200,total:3400,positiveCount:0}),'1,200 of 3,400 images · 0 positive prompts');
});
test('the count line reads as loading, never as an empty result, until a first page answers (#391)', async () => {
const {countLine,createImageStream}=await source;
assert.equal(countLine({loaded:0,total:0,positiveCount:null,loading:true,error:null}),'Loading images…');
// A failed first page and a settled empty one say what the server said; a later page keeps the loaded count.
assert.equal(countLine({loaded:0,total:0,positiveCount:null,loading:false,error:new Error('x')}),'0 of 0 images');
assert.equal(countLine({loaded:0,total:0,positiveCount:0,loading:false,error:null}),'0 of 0 images · 0 positive prompts');
assert.equal(countLine({loaded:40,total:102,positiveCount:2,loading:true,error:null}),'40 of 102 images · 2 positive prompts');
// While a reset's first page is in flight the line reads as loading (the cleared state before it is replaced in
// the same task, so it is never painted).
let answer;const lines=[];
const stream=createImageStream({api:{get:()=>new Promise(resolve=>{answer=resolve;})},onChange:state=>lines.push(countLine(state))});
const loading=stream.reset({tags:['cinematic']});
assert.equal(lines.at(-1),'Loading images…');
answer({images:[{path:'a.png',root_index:0}],total:1,has_more:false,next_cursor:null,positive_count:1});await loading;
assert.equal(lines.at(-1),'1 of 1 image · 1 positive prompt');
});
test('the caption date names the newest result day under Newest first only', async () => {
const {captionDate}=await source;
const now=new Date(2026,9,17,15,0);
+24
View File
@@ -0,0 +1,24 @@
"""A temp SQLite database in a folder no other test process shares (#346).
``PromptManagerAPI`` sweeps its database's folder at startup for the stages,
uploads and safety-backup copies an interrupted restore or backup left. A
fixture database directly in the system temp folder made every other test
process that started an API (another worktree's suite, a pre-push hook) delete
this process's in-flight restore files, and left safety backups behind there.
"""
import tempfile
def private_temp_db(case):
"""Return a closed ``NamedTemporaryFile`` handle for an empty ``.db`` file.
The file lives in a new folder that ``case`` removes at cleanup, after its
own teardown has closed and unlinked the database (cleanups run after
``tearDown``), together with anything a restore or backup left next to it.
"""
folder = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
case.addCleanup(folder.cleanup)
handle = tempfile.NamedTemporaryFile(delete=False, suffix=".db", dir=folder.name)
handle.close()
return handle
+21 -7
View File
@@ -430,7 +430,9 @@ class ReplaceImageTagsTests(AnnotationTestCase):
self.assertTrue(
self.db.refresh_image_fingerprint(loc, DIGEST_A, SIZE, MTIME + 1)
)
list(self.db.iter_image_tag_records())
self.assertEqual(
self.db.image_tag_locators(self.db.image_tag_ids(["img"], "")), {loc}
)
self.assertEqual(self.db.get_prompt_by_id(prompt)["tags"], ["prompt-tag"])
@@ -551,14 +553,26 @@ class CurrentImageTagsTests(AnnotationTestCase):
self.assertEqual(current, {locators[0]: ["z", "y"], locators[502]: ["x"]})
lookups = [s for s in statements if "media_annotation_files" in s]
self.assertEqual(len(lookups), 2)
# The tag search finds every record's file by tag id, and the empty
# set's file never.
self.assertEqual(
list(self.db.iter_image_tag_records()),
[
(locators[0], SIZE, MTIME, ["z", "y"]),
(locators[250], SIZE, MTIME + 1, []),
(locators[502], SIZE, MTIME + 2, ["x"]),
],
self.db.image_tag_locators(self.db.image_tag_ids(["z", "x"], "")),
{locators[0], locators[502]},
)
self.assertEqual(
self.db.current_image_tags({locators[250]: (SIZE, MTIME + 1)}),
{locators[250]: []},
)
def test_tag_locators_cover_more_ids_than_one_chunk(self):
# A broad text search can match more than 500 names: every id's file
# comes back across the chunks.
locators = [self.locator(f"t{i:04}.png") for i in range(503)]
for i, loc in enumerate(locators):
self.replace(loc, tags=[f"name_{i:04}"])
ids = self.db.image_tag_ids([], "name_")
self.assertEqual(len(ids), 503)
self.assertEqual(self.db.image_tag_locators(ids), set(locators))
class AutotagMarkTests(AnnotationTestCase):
+2 -3
View File
@@ -8,7 +8,6 @@ without requiring a full ComfyUI server.
import json
import os
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -18,6 +17,7 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from tests.private_temp import private_temp_db
from utils.hashing import generate_prompt_hash
@@ -25,8 +25,7 @@ class APITestCase(AioHTTPTestCase):
"""Base class that stands up a test aiohttp app with PromptManager routes."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
app = web.Application()
routes = web.RouteTableDef()
+13 -21
View File
@@ -846,6 +846,14 @@ class TestSettingsReadMonitorThroughModule(AdminAPITestCase):
self.assertEqual(data["settings"]["gallery_root_paths"], ["output"])
def _wired_prompt(text):
"""A sampler-less graph whose one encoder feeds a guider (#252)."""
return {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": text}},
"2": {"class_type": "BasicGuider", "inputs": {"conditioning": ["1", 0]}},
}
def _raise(*args, **kwargs):
raise RuntimeError("boom")
@@ -2186,11 +2194,7 @@ class TestOutputScanInternals(AdminAPITestCase):
def test_ingest_outcomes(self):
f = Path("gen.png")
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a cat"}}}
)
}
meta = {"prompt": json.dumps(_wired_prompt("a cat"))}
self.assertIsNone(self.api._ingest_scanned_file(f, {}))
self.assertIsNone(self.api._ingest_scanned_file(f, {"prompt": "{}"}))
self.assertEqual(
@@ -2202,31 +2206,19 @@ class TestOutputScanInternals(AdminAPITestCase):
def test_ingest_link_failures_are_tolerated(self):
f = Path("gen.png")
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a dog"}}}
)
}
meta = {"prompt": json.dumps(_wired_prompt("a dog"))}
self.api.db.link_image_to_prompt = _raise
self.assertEqual(self.api._ingest_scanned_file(f, meta), "added")
self.assertEqual(self.api._ingest_scanned_file(f, meta), "found")
def test_ingest_save_failure_counts_as_found(self):
f = Path("gen.png")
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a bird"}}}
)
}
meta = {"prompt": json.dumps(_wired_prompt("a bird"))}
self.api.db.save_prompt = lambda *a, **k: None
self.assertEqual(self.api._ingest_scanned_file(f, meta), "found")
def test_batch_counts_and_per_file_errors(self):
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a fish"}}}
)
}
meta = {"prompt": json.dumps(_wired_prompt("a fish"))}
self.api.db.get_prompt_by_hash = _raise
counts = self.api._ingest_scan_batch_sync(
[(Path("a.png"), meta), (Path("b.png"), {})]
@@ -2915,7 +2907,7 @@ class TestScanReadsVideos(AdminAPITestCase):
shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg not installed"
)
async def test_scan_adds_prompts_found_in_videos(self):
graph = {"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a cat"}}}
graph = _wired_prompt("a cat")
_write_mp4(
self.output_dir / "clip.mp4", json.dumps({"prompt": json.dumps(graph)})
)
+5 -8
View File
@@ -363,23 +363,20 @@ class TestModelGate(SuggestTestCase):
prompt_id = self._save_prompt("batch prompt")
self.api.db.link_image_to_prompt(prompt_id, str(self.output_dir / "sub/a.png"))
started, proceed = threading.Event(), threading.Event()
def slow_generate(image_path, **kwargs):
started.set()
proceed.wait(WAIT)
return ["batch"]
self.tagger.generate_tags = slow_generate
# The legacy batch is the autotag job (#379): it suggests per image.
self.tagger.block = (started, proceed)
batch = asyncio.ensure_future(
self.client.post("/prompt_manager/autotag/start?model_type=wd14-vit")
)
try:
await asyncio.get_running_loop().run_in_executor(None, started.wait, WAIT)
calls = len(self.tagger.calls)
status, body = await self.suggest(self.suggest_body(observed, "gguf"))
self.assertEqual((status, body["code"]), (409, "model-busy"))
self.assertEqual(self.tagger.calls, [])
self.assertEqual(len(self.tagger.calls), calls)
self.assertEqual(self.tagger.loaded, "wd14-vit") # not swapped
finally:
self.tagger.block = None
proceed.set()
response = await batch
await response.read()
+71 -413
View File
@@ -5,6 +5,7 @@ no network request is made.
"""
import asyncio
import hashlib
import json
import os
import shutil
@@ -34,6 +35,8 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from tests.private_temp import private_temp_db
from py.api.autotag_routes import MODEL_BUSY_ERROR
from py.api.jobs import JobBusy
from utils.hashing import generate_prompt_hash
@@ -70,6 +73,7 @@ class FakeAutotagService:
self.status_calls = 0
self.status_threads = []
self.status_paths = {}
self.suggest_calls = 0
def get_models_status(self):
self.status_calls += 1
@@ -115,14 +119,32 @@ class FakeAutotagService:
raise self.generate_error
return list(self.tags)
def generate_tag_suggestions(
self,
image_path,
*,
limit,
prompt=None,
general_threshold=None,
character_threshold=None,
):
"""The suggestion entry point (#379): the fixed tags, at most ``limit``."""
self.suggest_calls += 1
tags = self.generate_tags(
image_path,
prompt=prompt,
general_threshold=general_threshold,
character_threshold=character_threshold,
)
return [(name, None) for name in tags[:limit]]
class AutotagAPITestCase(AioHTTPTestCase):
"""Stands up the PromptManager routes with a temp DB, temp output dir and a
fake engine."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
self.tmp_root = Path(tempfile.mkdtemp())
self.output_dir = self.tmp_root / "output"
self.output_dir.mkdir()
@@ -175,8 +197,17 @@ class AutotagAPITestCase(AioHTTPTestCase):
)
def _make_image(self, directory, name="img.png"):
"""A small, valid image (the legacy review captures and probes it)."""
from PIL import Image
path = directory / name
path.write_bytes(b"\x89PNG\r\n\x1a\n")
path.parent.mkdir(parents=True, exist_ok=True)
shade = sum(name.encode()) % 256
fmt = {".jpg": "JPEG", ".jpeg": "JPEG", ".webp": "WEBP"}.get(
path.suffix.lower(), "PNG"
)
with Image.new("RGB", (4, 4), (shade, 64, 128)) as image:
image.save(path, fmt)
return path
def _link_image(self, prompt_id, path):
@@ -216,10 +247,9 @@ class TestAutotagSingleContainment(AutotagAPITestCase):
self.assertEqual(data["tags"], ["1girl", "smile"])
self.assertEqual(data["prompt_id"], prompt_id)
self.assertEqual(len(self.service.generate_calls), 1)
self.assertEqual(
os.path.realpath(self.service.generate_calls[0][0]),
os.path.realpath(str(image)),
)
# #379: the model reads a private capture of the image, never the file.
self.assertNotIn(str(self.output_dir), self.service.generate_calls[0][0])
self.assertEqual(data["sha256"], hashlib.sha256(image.read_bytes()).hexdigest())
async def test_path_inside_output_dir_is_tagged(self):
image = self._make_image(self.output_dir)
@@ -449,238 +479,6 @@ class TestAutotagSideEffectRoutesArePostOnly(AutotagAPITestCase):
self.assertTrue((await resp.json())["success"])
class TestAutotagBatch(AutotagAPITestCase):
"""Batch tagging over the DB image list with SSE progress reporting."""
def _seed(self):
"""Three prompts: A (one image), B (two images), C (missing file)."""
a = self._save_prompt("prompt a")
b = self._save_prompt("prompt b")
c = self._save_prompt("prompt c")
self._link_image(a, self._make_image(self.output_dir, "a1.png"))
self._link_image(b, self._make_image(self.output_dir, "b1.png"))
self._link_image(b, self._make_image(self.output_dir, "b2.png"))
self._link_image(c, self.output_dir / "missing.png")
return a, b, c
async def _start(self, query="model_type=gguf", body=None):
if body is None:
resp = await self.client.request(
"POST", f"/prompt_manager/autotag/start?{query}"
)
else:
resp = await self._post_json(f"/prompt_manager/autotag/start?{query}", body)
return resp, self._sse_events(await resp.text())
async def test_batch_tags_each_prompt_once_and_reports_counts(self):
a, b, c = self._seed()
resp, events = await self._start()
self.assertEqual(resp.status, 200)
done = events[-1]
self.assertEqual(done["type"], "complete")
self.assertEqual(done["processed"], 2)
self.assertEqual(done["tagged"], 2)
self.assertEqual(done["skipped"], 2)
self.assertEqual(done["errors"], 0)
self.assertEqual(len(self.service.generate_calls), 2)
self.assertEqual(self.api.db.get_prompt_by_id(a)["tags"], ["1girl", "smile"])
self.assertEqual(self.api.db.get_prompt_by_id(b)["tags"], ["1girl", "smile"])
self.assertEqual(self.api.db.get_prompt_by_id(c)["tags"], [])
progress = [e for e in events if e["type"] == "progress"]
self.assertTrue(all(0 <= e["progress"] <= 100 for e in progress))
self.assertTrue(any("processed" in e for e in progress))
async def test_rows_outside_output_dirs_are_skipped_and_counted(self):
inside = self._save_prompt("inside")
outside = self._save_prompt("outside")
self._link_image(inside, self._make_image(self.output_dir, "in.png"))
secret = self._make_image(self.secret_dir, "password.png")
self._link_image(outside, secret)
_, events = await self._start()
done = events[-1]
self.assertEqual(done["type"], "complete")
self.assertEqual(done["processed"], 1)
self.assertEqual(done["tagged"], 1)
self.assertEqual(done["skipped"], 1)
self.assertEqual(done["skipped_outside_roots"], 1)
self.assertEqual(len(self.service.generate_calls), 1)
self.assertEqual(
os.path.realpath(self.service.generate_calls[0][0]),
os.path.realpath(str(self.output_dir / "in.png")),
)
self.assertEqual(self.api.db.get_prompt_by_id(outside)["tags"], [])
progress = [e for e in events if e["type"] == "progress" and "skipped" in e]
self.assertTrue(progress)
self.assertTrue(all("skipped_outside_roots" in e for e in progress))
body = json.dumps(events)
self.assertNotIn(str(self.secret_dir), body)
async def test_second_start_while_running_is_409(self):
self._seed()
started = threading.Event()
release = threading.Event()
def slow_generate(image_path, **kwargs):
started.set()
release.wait(5)
return ["slow"]
self.service.generate_tags = slow_generate
first = asyncio.ensure_future(self._start())
try:
await asyncio.get_event_loop().run_in_executor(None, started.wait, 5)
self.assertTrue(started.is_set())
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start?model_type=gguf"
)
self.assertEqual(resp.status, 409)
self.assertEqual(
await resp.json(),
{"success": False, "error": "Auto-tagging is already running"},
)
finally:
release.set()
resp1, events = await first
self.assertEqual(resp1.status, 200)
self.assertEqual(events[-1]["type"], "complete")
async def test_lock_is_released_after_a_run(self):
self._seed()
resp1, _ = await self._start()
resp2, events = await self._start()
self.assertEqual(resp1.status, 200)
self.assertEqual(resp2.status, 200)
self.assertEqual(events[-1]["type"], "complete")
async def test_lock_is_released_when_the_run_fails(self):
self._seed()
self.service.load_error = RuntimeError("no model")
_, events = await self._start()
self.assertEqual(events[-1]["type"], "error")
self.service.load_error = None
resp, events = await self._start()
self.assertEqual(resp.status, 200)
self.assertEqual(events[-1]["type"], "complete")
async def test_already_tagged_prompts_are_skipped_unless_asked(self):
p = self._save_prompt("tagged", tags=["cat"])
self._link_image(p, self._make_image(self.output_dir, "t.png"))
_, events = await self._start("model_type=gguf&skip_tagged=true")
self.assertEqual(events[-1]["skipped"], 1)
self.assertEqual(self.service.generate_calls, [])
_, events = await self._start("model_type=gguf&skip_tagged=false")
self.assertEqual(events[-1]["tagged"], 1)
self.assertEqual(
self.api.db.get_prompt_by_id(p)["tags"], ["cat", "1girl", "smile"]
)
async def test_prompt_already_holding_the_tags_counts_as_skipped(self):
p = self._save_prompt("same", tags=["1girl", "smile"])
self._link_image(p, self._make_image(self.output_dir, "s.png"))
_, events = await self._start("model_type=gguf&skip_tagged=false")
self.assertEqual(events[-1]["skipped"], 1)
self.assertEqual(events[-1]["tagged"], 0)
async def test_generation_errors_are_counted_and_do_not_abort(self):
self._seed()
self.service.generate_error = RuntimeError("boom")
_, events = await self._start()
self.assertEqual(events[-1]["type"], "complete")
# a1, b1 and b2 each fail: a failed prompt is retried on its next image
self.assertEqual(events[-1]["errors"], 3)
self.assertEqual(events[-1]["tagged"], 0)
async def test_json_body_parameters_are_accepted(self):
self._seed()
_, events = await self._start(
"", {"model_type": "wd14-vit", "general_threshold": 0.5, "prompt": "p"}
)
self.assertEqual(events[-1]["type"], "complete")
self.assertEqual(self.service.load_calls, [("wd14-vit", True)])
self.assertEqual(self.service.generate_calls[0][2], 0.5)
self.assertEqual(self.service.custom_prompt, "p")
async def test_keep_in_memory_false_unloads_after_batch(self):
self._seed()
_, events = await self._start("model_type=gguf&keep_in_memory=false")
self.assertEqual(events[-1]["model_loaded"], False)
self.assertEqual(self.service.unload_calls, 1)
async def test_model_not_downloaded_is_an_error_event(self):
self.service.downloaded = False
_, events = await self._start()
self.assertEqual(events[-1]["type"], "error")
self.assertEqual(self.service.load_calls, [])
async def test_load_failure_is_an_error_event_without_details(self):
self.service.load_error = RuntimeError("/srv/models/secret.gguf is corrupt")
_, events = await self._start()
self.assertEqual(events[-1]["type"], "error")
self.assertNotIn("/srv/models", events[-1]["message"])
async def test_non_numeric_threshold_is_400(self):
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start?general_threshold=abc"
)
self.assertEqual(resp.status, 400)
async def test_invalid_json_body_is_400(self):
resp = await self.client.request(
"POST",
"/prompt_manager/autotag/start",
data="{not json",
headers={"Content-Type": "application/json"},
)
self.assertEqual(resp.status, 400)
async def test_client_disconnect_stops_the_batch_and_unloads(self):
self._seed()
real_write = web.StreamResponse.write
state = {"writes": 0}
async def flaky_write(response, data):
state["writes"] += 1
if b"Processing " in data:
raise ConnectionResetError("Cannot write to closing transport")
return await real_write(response, data)
with patch.object(web.StreamResponse, "write", flaky_write):
try:
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start?keep_in_memory=false"
)
await resp.read()
except Exception:
pass
self.assertEqual(len(self.service.generate_calls), 1)
self.assertEqual(self.service.unload_calls, 1)
class TestAutotagModelsAndDownloadErrors(AutotagAPITestCase):
async def test_models_error_is_500(self):
@@ -834,7 +632,15 @@ class TestAutotagSingleEdgeCases(AutotagAPITestCase):
self.assertEqual(resp.status, 200)
self.assertEqual(self.service.load_calls, [("wd14-vit", True)])
self.assertEqual(self.service.custom_prompt, "terse")
# #379: the request's prompt is for this call only.
self.assertEqual(self.service.custom_prompt, "")
resp = await self._post_json(
"/prompt_manager/autotag/single",
{"path": str(image), "model_type": "gguf", "prompt": "terse"},
)
self.assertEqual(resp.status, 200)
self.assertEqual(self.service.generate_calls[-1][1], "terse")
self.assertEqual(self.service.custom_prompt, "")
async def test_image_id_without_stored_path_is_404(self):
prompt_id = self._save_prompt()
@@ -850,64 +656,6 @@ class TestAutotagSingleEdgeCases(AutotagAPITestCase):
self.assertEqual(resp.status, 404)
class TestApplyAutotag(AutotagAPITestCase):
async def test_missing_prompt_id_is_400(self):
resp = await self._post_json("/prompt_manager/autotag/apply", {"tags": ["a"]})
self.assertEqual(resp.status, 400)
async def test_no_tags_is_a_noop(self):
resp = await self._post_json("/prompt_manager/autotag/apply", {"prompt_id": 1})
data = await resp.json()
self.assertTrue(data["success"])
self.assertEqual(data["message"], "No tags to apply")
async def test_unknown_prompt_is_404(self):
resp = await self._post_json(
"/prompt_manager/autotag/apply", {"prompt_id": 999, "tags": ["a"]}
)
self.assertEqual(resp.status, 404)
async def test_tags_are_merged_without_duplicates(self):
prompt_id = self._save_prompt(tags=["cat"])
resp = await self._post_json(
"/prompt_manager/autotag/apply",
{"prompt_id": prompt_id, "tags": ["cat", "dog"]},
)
data = await resp.json()
self.assertEqual(data["added_tags"], ["dog"])
self.assertEqual(data["total_tags"], 2)
self.assertEqual(
self.api.db.get_prompt_by_id(prompt_id)["tags"], ["cat", "dog"]
)
async def test_string_tags_from_db_are_split(self):
prompt_id = self._save_prompt()
with patch.object(
self.api.db,
"get_prompt_by_id",
return_value={"id": prompt_id, "tags": "cat, dog"},
):
resp = await self._post_json(
"/prompt_manager/autotag/apply",
{"prompt_id": prompt_id, "tags": ["dog", "owl"]},
)
data = await resp.json()
self.assertEqual(data["added_tags"], ["owl"])
self.assertEqual(data["total_tags"], 3)
async def test_db_error_is_500(self):
with patch.object(
self.api.db, "get_prompt_by_id", side_effect=RuntimeError("db")
):
resp = await self._post_json(
"/prompt_manager/autotag/apply", {"prompt_id": 1, "tags": ["a"]}
)
self.assertEqual(resp.status, 500)
class TestAutotagErrorBodiesHideServerPaths(AutotagAPITestCase):
"""Every autotag handler reports exceptions through the shared public helper."""
@@ -932,9 +680,12 @@ class TestAutotagErrorBodiesHideServerPaths(AutotagAPITestCase):
await self._assert_hidden(resp)
async def test_apply_error(self):
with patch.object(self.api.db, "get_prompt_by_id", side_effect=self._locked()):
image = self._make_image(self.output_dir)
digest = hashlib.sha256(image.read_bytes()).hexdigest()
with patch.object(self.api.db, "add_image_tags", side_effect=self._locked()):
resp = await self._post_json(
"/prompt_manager/autotag/apply", {"prompt_id": 1, "tags": ["a"]}
"/prompt_manager/autotag/apply",
{"path": str(image), "sha256": digest, "tags": ["a"]},
)
await self._assert_hidden(resp)
@@ -1008,10 +759,13 @@ class TestLegacyRoutesShareTheModelGate(AutotagAPITestCase):
"POST", "/prompt_manager/autotag/download/gguf"
)
)
await self._busy(
await self.client.request(
"POST", "/prompt_manager/autotag/start?model_type=gguf"
)
# The legacy batch is read as a stream: one error frame (#379).
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start?model_type=gguf"
)
self.assertEqual(
self._sse_events(await resp.text()),
[{"type": "error", "status": "error", "message": MODEL_BUSY_ERROR}],
)
await self._busy(
await self.client.request("POST", "/prompt_manager/autotag/unload")
@@ -1026,7 +780,7 @@ class TestLegacyRoutesShareTheModelGate(AutotagAPITestCase):
self.assertEqual(self.service.unload_calls, 0)
self.assertEqual(self.service.get_loaded_model_type(), "wd14-vit")
# A refused start released the batch lock: the next start runs.
# A refused start left no job behind: the next start runs.
resp = await self.client.request(
"POST", "/prompt_manager/autotag/start?model_type=gguf"
)
@@ -1238,7 +992,7 @@ if __name__ == "__main__":
class TestGalleryReviewContract(AutotagAPITestCase):
async def test_root_relative_gallery_targets_generate_and_apply_to_their_prompt(
async def test_root_relative_gallery_targets_generate_and_apply_to_their_image(
self,
):
second = self.tmp_root / "second-output"
@@ -1255,16 +1009,22 @@ class TestGalleryReviewContract(AutotagAPITestCase):
self.assertEqual(response.status, 200)
data = await response.json()
self.assertEqual(data["prompt_id"], pid)
self.assertEqual(
self.service.generate_calls[-1][0], str((root / "same.png").resolve())
)
response = await self._post_json(
"/prompt_manager/autotag/apply",
{"prompt_id": data["prompt_id"], "tags": data["tags"]},
{
"path": "same.png",
"root_index": index,
"sha256": data["sha256"],
"tags": data["tags"],
},
)
self.assertEqual(response.status, 200)
# The image's own tags; the prompt is never written (#379).
self.assertEqual(self.api.db.get_prompt_by_id(pid)["tags"], [])
locator = os.path.normcase(os.path.realpath(str(root / "same.png")))
self.assertEqual(
set(self.api.db.get_prompt_by_id(pid)["tags"]), set(data["tags"])
self.api.db.read_image_tags(locator, data["sha256"])["tags"],
data["tags"],
)
async def test_gallery_unlinked_image_returns_no_prompt_id(self):
@@ -2058,108 +1818,6 @@ class TestTagOutputImagesJob(BatchJobTestCase):
self.assertTrue(inside)
# The legacy batch's per-image frames over BatchJobTestCase's library, newest
# first, with every throttle check passing (pinned from main's generator).
GOLDEN_LEGACY_ROWS = [
{
"type": "progress",
"progress": 24,
"status": "Processing 1/6...",
"processed": 1,
"tagged": 1,
"skipped": 0,
"skipped_outside_roots": 0,
},
{
"type": "progress",
"progress": 38,
"status": "Skipping 2/6 (outside allowed directories)...",
"processed": 1,
"tagged": 1,
"skipped": 1,
"skipped_outside_roots": 1,
},
{
"type": "progress",
"progress": 52,
"status": "Skipping 3/6 (file missing)...",
"processed": 1,
"tagged": 1,
"skipped": 2,
"skipped_outside_roots": 1,
},
{
"type": "progress",
"progress": 66,
"status": "Processing 4/6...",
"processed": 2,
"tagged": 2,
"skipped": 2,
"skipped_outside_roots": 1,
},
{
"type": "progress",
"progress": 80,
"status": "Skipping 5/6 (prompt already processed)...",
"processed": 2,
"tagged": 2,
"skipped": 3,
"skipped_outside_roots": 1,
},
{
"type": "progress",
"progress": 95,
"status": "Skipping 6/6 (already tagged)...",
"processed": 2,
"tagged": 2,
"skipped": 4,
"skipped_outside_roots": 1,
},
]
class TestLegacyBatchFrames(BatchJobTestCase):
async def test_legacy_batch_events_unchanged(self):
"""Golden frames of the legacy batch over the seeded library."""
self._seed()
ticks = iter(range(1000)) # every throttle check sees a second pass
with patch("py.api.autotag_routes._monotonic", side_effect=lambda: next(ticks)):
resp = await self.client.post(
"/prompt_manager/autotag/start?model_type=wd14-vit&skip_tagged=true"
)
frames = self._sse_events(await resp.text()) # the stream runs here
self.assertEqual(
frames,
[
{"type": "progress", "progress": 0, "status": "Loading model..."},
{
"type": "progress",
"progress": 5,
"status": "Model loaded. Fetching all images from database...",
},
{
"type": "progress",
"progress": 10,
"status": "Found 6 images. Processing...",
},
*GOLDEN_LEGACY_ROWS,
{
"type": "complete",
"progress": 100,
"processed": 2,
"tagged": 2,
"skipped": 4,
"errors": 0,
"skipped_outside_roots": 1,
"status": "Complete",
"model_status": "Model kept in memory",
"model_loaded": True,
},
],
)
class TestTagOutputImagesEdges(BatchJobTestCase):
async def test_cancel_during_backup_loads_no_model(self):
-92
View File
@@ -835,7 +835,6 @@ class TestMetadataParsing(unittest.TestCase):
self.assertEqual(parsed["positive_prompt"], "a cat")
self.assertEqual(parsed["negative_prompt"], "dog")
self.assertEqual(parsed["parameters"]["parameters"], meta["parameters"])
self.assertEqual(self.api._extract_readable_prompt(parsed), "a cat")
def test_parse_comfyui_fields(self):
meta = {
@@ -856,97 +855,6 @@ class TestMetadataParsing(unittest.TestCase):
self.assertEqual(parsed["prompt"], "plain text")
self.assertEqual(parsed["workflow"], {"nodes": []})
def test_extract_readable_prompt_variants(self):
extract = self.api._extract_readable_prompt
self.assertEqual(extract({"prompt": "plain"}), "plain")
self.assertEqual(extract({"prompt": ["a", "", "b"]}), "a b")
self.assertEqual(extract({"prompt": 42}), "42")
self.assertEqual(
extract({"prompt": None, "parameters": {"positive": ["x", "y"]}}), "x y"
)
self.assertIsNone(extract({"prompt": {}, "parameters": {}}))
graph = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "pos"}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "neg"}},
"3": {
"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["2", 0]},
},
}
self.assertEqual(extract({"prompt": graph}), "pos")
self.assertEqual(extract({"prompt": {"x": 1}, "workflow": graph}), "pos")
def test_node_inputs_and_text_lookup(self):
self.assertEqual(self.api._get_node_inputs("nope"), {})
self.assertEqual(self.api._get_node_inputs({"inputs": 5}), {})
self.assertEqual(self.api._get_node_inputs({"inputs": {"a": 1}}), {"a": 1})
listed = self.api._get_node_inputs(
{"inputs": [{"name": "text", "link": None}, {"nope": 1}, "junk"]}
)
self.assertEqual(list(listed), ["text"])
self.assertIsNone(self.api._find_text_in_node(None))
self.assertEqual(self.api._find_text_in_node({"inputs": {"text": "hi"}}), "hi")
self.assertEqual(
self.api._find_text_in_node(
{"type": "CLIPTextEncode", "widgets_values": ["w"]}
),
"w",
)
self.assertIsNone(
self.api._find_text_in_node(
{"type": "CLIPTextEncode", "widgets_values": [" "]}
)
)
self.assertIsNone(self.api._find_text_in_node({"type": "KSampler"}))
def test_positive_prompt_from_nodes_array_and_fallbacks(self):
extract = self.api._extract_positive_prompt_from_comfyui_data
self.assertIsNone(extract("x"))
self.assertIsNone(extract({}))
self.assertIsNone(extract({"nodes": []}))
workflow = {
"nodes": [
{"id": 1, "type": "CLIPTextEncode", "widgets_values": ["pos"]},
{
"id": 2,
"type": "CLIPTextEncode",
"widgets_values": ["neg"],
"title": "Negative",
},
{"id": 3, "type": "KSampler", "inputs": [{"name": "seed", "link": 4}]},
"junk",
]
}
self.assertEqual(extract(workflow), "pos")
only_negative = {
"nodes": [
{
"id": 2,
"type": "CLIPTextEncode",
"widgets_values": ["neg"],
"title": "neg",
},
]
}
self.assertEqual(extract(only_negative), "neg")
bad_link = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "pos"}},
"3": {
"class_type": "KSampler",
"inputs": {"positive": "oops", "negative": []},
},
"4": {
"class_type": "KSampler",
"inputs": {"positive": ["zz", 0], "negative": []},
},
}
self.assertEqual(extract(bad_link), "pos")
if __name__ == "__main__":
unittest.main()
+23 -17
View File
@@ -18,15 +18,19 @@ from pathlib import Path
from unittest.mock import patch
try:
from tests.disk_waits import DISK_BOUND_SECONDS
from tests.open_handles import open_handles_under
from tests.test_api_maintenance import ApplyTestCase, _digest
except ImportError: # discovered with tests/ as the top-level directory
from disk_waits import DISK_BOUND_SECONDS
from open_handles import open_handles_under
from test_api_maintenance import ApplyTestCase, _digest
from py.api import duplicate_files # noqa: E402
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
# Gates and server-state polls. Waits on a find or cleanup job's task use
# DISK_BOUND_SECONDS: those jobs hash, back up and unlink files (#409).
WAIT = 10
FIND = "/prompt_manager/jobs/duplicates"
GROUPS = "/prompt_manager/maintenance/duplicates/"
@@ -83,7 +87,7 @@ class DuplicateFilesTestCase(ApplyTestCase):
self.assertEqual(response.status, 202, text)
self.assertNotIn(self.tmpdir, text)
job = self.api._jobs.get(json.loads(text)["job"]["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
return job
async def groups(self, job_id, status=200, **query):
@@ -127,7 +131,7 @@ class DuplicateFilesTestCase(ApplyTestCase):
plan = await self.review(job.id, [self.keep_of(group, path)])
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
return plan, cleanup
async def until(self, predicate):
@@ -178,7 +182,7 @@ class TestFindJob(DuplicateFilesTestCase):
gate.set()
body = await (await legacy).text()
job = self.api._jobs.get(summary["id"])
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
events = _sse_payloads(body)
done = events[-1]
self.assertEqual(done["type"], "complete")
@@ -241,7 +245,7 @@ class TestFindJob(DuplicateFilesTestCase):
self.assertTrue((await cancel.json())["cancellation_requested"])
gate.set()
job = self.api._jobs.get(job_id)
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
self.assertEqual(job.state, "cancelled")
last = _sse_payloads(await (await stream).text())[-1]
self.assertEqual(last, {"type": "error", "message": "Duplicate scan cancelled"})
@@ -341,6 +345,8 @@ class TestReview(DuplicateFilesTestCase):
self.assertEqual(plan["protected"], {"count": 0, "reasons": []})
async def test_cannot_delete_keeper_or_outside_group(self):
# About which files may go, not the backup: skip its fsync-heavy copy.
PromptManagerConfig.SAFETY_BACKUP = False
self.trio()
self.write("e.png", b"other pair")
self.write("f.png", b"other pair")
@@ -372,7 +378,7 @@ class TestReview(DuplicateFilesTestCase):
self.assertEqual([row["label"] for row in plan["details"]], ["b.png"])
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters["removed"], 1)
self.assertTrue((self.output_dir / "a.png").is_file())
self.assertTrue((self.output_dir / "sub" / "c.png").is_file())
@@ -496,7 +502,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
plan = await self.review(job.id, keep)
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertTrue(kept.is_file())
self.assertFalse(gone.exists())
self.assertFalse(video_thumb.exists())
@@ -508,7 +514,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.write("b.png", b"new bytes") # changed since the review
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters, {"removed": 1, "kept": 2, "errors": 0})
self.assertEqual((self.output_dir / "b.png").read_bytes(), b"new bytes")
self.assertFalse((self.output_dir / "sub" / "c.png").exists())
@@ -522,7 +528,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.write("a.png", b"edited keeper")
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters["removed"], 0)
self.assertTrue((self.output_dir / "b.png").is_file())
self.assertTrue((self.output_dir / "sub" / "c.png").is_file())
@@ -549,7 +555,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.state, "failed")
self.assertEqual(cleanup.counters, {"removed": 1, "kept": 1, "errors": 1})
self.assertEqual(
@@ -584,7 +590,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters, {"removed": 0, "kept": 3, "errors": 1})
self.assertEqual(
cleanup.outcomes["errors"][0]["detail"], duplicate_files.DELETE_FAILED
@@ -619,7 +625,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with context:
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
finally:
if handle is not None: # pragma: no cover - Windows
handle.close()
@@ -642,7 +648,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.counters["removed"], 2)
self.assertEqual(open_at_unlink, [])
@@ -703,7 +709,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
# Its records moved to the identical kept copy before the unlink failed.
[image] = self.api.db.get_prompt_images(prompt)
self.assertTrue(image["image_path"].endswith("a.png"))
@@ -719,7 +725,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.assertEqual((busy["code"], busy["running"]["id"]), ("busy", find["id"]))
gate.set()
await asyncio.wait_for(
asyncio.shield(self.api._jobs.get(find["id"]).task), WAIT
asyncio.shield(self.api._jobs.get(find["id"]).task), DISK_BOUND_SECONDS
)
# The token survived the refusal; the cleanup holds off a new find.
release = asyncio.Event()
@@ -737,7 +743,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
self.assertEqual(legacy.status, 409)
release.set()
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
async def test_cleanup_refused_up_front_when_the_backup_does_not_fit(self):
from py.api import maintenance
@@ -965,7 +971,7 @@ class TestReviewHelpers(DuplicateFilesTestCase):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.second_source())]
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.state, "failed")
self.assertEqual(cleanup.last_event["code"], "sources-changed")
self.assertEqual(self.backups(), [])
@@ -985,7 +991,7 @@ class TestReviewHelpers(DuplicateFilesTestCase):
):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
self.assertEqual(cleanup.state, "failed")
self.assertEqual(
cleanup.outcomes["errors"][-1]["detail"], duplicate_files.RECORDS_FAILED
+3 -3
View File
@@ -19,17 +19,17 @@ from pathlib import Path
from unittest.mock import patch
try:
from tests.disk_waits import DISK_BOUND_SECONDS
from tests.open_handles import open_handles_under
from tests import test_api_duplicate_files as dup_tests
except ImportError: # discovered with tests/ as the top-level directory
import test_api_duplicate_files as dup_tests
from disk_waits import DISK_BOUND_SECONDS
from open_handles import open_handles_under
from py.api import duplicate_files # noqa: E402
from py.config import GalleryConfig # noqa: E402
WAIT = 10
def _locator(path):
return os.path.normcase(os.path.realpath(str(path)))
@@ -49,7 +49,7 @@ class SafetyTestCase(dup_tests.DuplicateFilesTestCase):
async def apply_plan(self, plan):
body = await self.apply(plan["token"])
cleanup = self.api._jobs.get(body["job"]["id"])
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
return cleanup
+44
View File
@@ -1169,3 +1169,47 @@ class HistoryHostCleanupTests(unittest.IsolatedAsyncioTestCase):
fence.validate(epoch, generation)
with self.assertRaises(sqlite3.Error):
api._history_fence()
class SharedTempFolderRestoreTests(ImageAPITestCase):
"""#346: a restore survives another suite starting in the shared temp folder.
Every ``PromptManagerAPI`` sweeps its database's folder for stages and
uploads an interrupted restore left. A fixture database directly in the
shared temp folder let any other test process (another worktree's suite,
a pre-push hook) delete this restore's files mid-request. The shared
folder here is a scratch stand-in for the system temp folder, so the test
never sweeps the real one.
"""
async def get_application(self):
shared = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
# Cleanups run last-in first-out: this one after the fixture's own.
self.addCleanup(shared.cleanup)
self.shared_tmp = shared.name
with patch.object(tempfile, "tempdir", shared.name):
return await super().get_application()
async def test_restore_survives_another_suite_sweeping_the_shared_folder(self):
from database import restore_stage
from database.models import PromptModel
self.api.db.save_prompt("kept by the upload")
source = self.comfy_dir / "upload.db"
self.assertTrue(self.api.db.model.backup_database(str(source)))
stage = PromptModel.stage_restore
swept = []
def stage_then_other_suite_starts(model, src_path):
staged = stage(model, src_path)
# What another suite's PromptManagerAPI does at startup.
other = os.path.join(self.shared_tmp, "other-suite.db")
swept.extend(restore_stage.sweep_restore_temps(other))
return staged
form = FormData()
form.add_field("database_file", source.read_bytes(), filename="upload.db")
with patch.object(PromptModel, "stage_restore", stage_then_other_suite_starts):
response = await self.client.post("/prompt_manager/restore", data=form)
self.assertEqual(response.status, 200, await response.text())
self.assertEqual(swept, [], "another suite's sweep reached this restore")
+2 -2
View File
@@ -42,6 +42,7 @@ from py import lora_utils # noqa: E402
from py import lora_manager_layout as layout_module # noqa: E402
from py.api.thumbnail_settings import ThumbnailSettings # noqa: E402
from py.api import PromptManagerAPI # noqa: E402
from tests.private_temp import private_temp_db # noqa: E402
from py.api.prompts import MAX_PAGE_LIMIT # noqa: E402
from py.config import GalleryConfig # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
@@ -60,8 +61,7 @@ class ImageAPITestCase(AioHTTPTestCase):
"""App with PromptManager routes, a temp DB and a temp output directory."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
self._output_tmp = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
# A fake ComfyUI tree: <tmp>/output is the gallery root and <tmp> is the
+2 -2
View File
@@ -36,6 +36,7 @@ from aiohttp.test_utils import AioHTTPTestCase
from database.operations import PromptDatabase
from py.api import PromptManagerAPI
from tests.private_temp import private_temp_db
from py import config_store, lora_import, lora_utils
from py.config import IntegrationConfig, PromptManagerConfig
from py.api import settings_routes as settings_module
@@ -57,8 +58,7 @@ class LoraAPITestCase(AioHTTPTestCase):
"""Stands up the PromptManager routes with a temp DB and neutral LoRA config."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
self.tmp_root = Path(tempfile.mkdtemp())
self.custom_nodes = self.tmp_root / "custom_nodes"
self.custom_nodes.mkdir()
+2 -3
View File
@@ -5,7 +5,6 @@ Route tests for the prompt API (py/api/prompts.py).
import json
import os
import sys
import tempfile
import unittest
from unittest.mock import MagicMock, patch
@@ -29,6 +28,7 @@ except ImportError: # discovered with tests/ as the top-level directory
from database.operations import PromptDatabase # noqa: E402
from py.api import PromptManagerAPI # noqa: E402
from tests.private_temp import private_temp_db # noqa: E402
import py.api.prompts as prompts_module # noqa: E402
from py.api.prompts import ( # noqa: E402
BULK_CATEGORY_MAX_PROMPTS,
@@ -43,8 +43,7 @@ class PromptAPITestCase(AioHTTPTestCase):
"""App with PromptManager routes and a temporary SQLite database."""
async def get_application(self):
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
self._temp_db.close()
self._temp_db = private_temp_db(self)
app = web.Application()
routes = web.RouteTableDef()
+168 -1
View File
@@ -1831,6 +1831,39 @@ class TestWorkspaceCursorEncoding(unittest.TestCase):
self.assertEqual(_workspace_cursor_key(token, "scope", order), key)
# The pre-#379 candidate read, kept as the differential reference for the
# indexed lookup (`image_tag_ids` + `image_tag_locators`): every byte record
# streamed with its tag names, a locator kept when any record's names are
# relevant (`_image_tags_relevant`).
_REFERENCE_RECORDS = (
"SELECT f.locator,b.id,t.name FROM media_annotation_files f "
"JOIN media_annotation_bytes b ON b.file_id=f.id "
"LEFT JOIN media_image_tags mit ON mit.byte_id=b.id "
"LEFT JOIN tags t ON t.id=mit.tag_id "
"ORDER BY f.locator,b.first_seen_at,b.id,mit.position"
)
def _image_tags_relevant(names, wanted, query):
if wanted and not wanted.isdisjoint(names):
return True
return bool(query) and any(query in name.casefold() for name in names)
def reference_candidates(db, wanted, query):
"""Locators the old full stream admitted for (``wanted``, ``query``)."""
records = {}
for locator, byte_id, name in db.model.get_connection().execute(_REFERENCE_RECORDS):
names = records.setdefault((locator, byte_id), [])
if name is not None:
names.append(name)
return {
locator
for (locator, _byte), names in records.items()
if _image_tags_relevant(names, set(wanted), query)
}
class TestWorkspaceImageTags(ImageAPITestCase):
"""B3: image tags in the Tags filter and search (spec amendment S1)."""
@@ -2038,7 +2071,7 @@ class TestWorkspaceImageTags(ImageAPITestCase):
leak = "no such table: media_annotation_files"
with patch.object(
self.api.db,
"iter_image_tag_records",
"image_tag_locators",
side_effect=sqlite3.OperationalError(leak),
):
# As a failed link read does under a Tags filter: a fixed coded
@@ -2056,6 +2089,140 @@ class TestWorkspaceImageTags(ImageAPITestCase):
self.assertEqual(self.names(body), ["dunes_by_name.png"])
self.assertIn("image tags", "\n".join(logged.output))
def seed_differential(self):
"""Mixed-case and underscore names, a stale record, an unlisted file."""
a = self.media("a.png", mtime=101)
b = self.media("b.png", mtime=102)
c = self.media("c.png", mtime=103)
self.tag(a, ["Beach", "long_hair", "1girl"])
self.tag(b, ["beach", "waves"])
self.tag(c, ["Sunset"])
# c's bytes change: its old record (Sunset) is stale, the new one current.
make_png(c, size=(5, 5))
os.utime(c, (104, 104))
self.tag(c, ["night_sky", "BEACH"])
outside = Path(self._output_tmp.name) / "elsewhere" / "x.png"
outside.parent.mkdir(parents=True, exist_ok=True)
make_png(outside)
self.tag(outside, ["beach", "outside_only"])
async def test_indexed_candidates_equal_the_full_stream(self):
self.seed_differential()
cases = [
(["beach"], ""),
(["Beach"], ""),
(["long_hair", "waves"], ""),
([], "beach"),
([], "hair"),
([], "_"),
([], "sun"),
(["nope"], "zzz"),
(["outside_only"], ""),
(["beach"], "night"),
]
db = self.api.db
for wanted, query in cases:
with self.subTest(wanted=wanted, query=query):
indexed = set(db.image_tag_locators(db.image_tag_ids(wanted, query)))
self.assertEqual(indexed, reference_candidates(db, wanted, query))
# The API answers are the indexed lookup's: exact names under Tags, a
# case-folded substring under q, current records only.
_, body = await self.get(tags="beach")
self.assertEqual(sorted(self.names(body)), ["b.png"])
_, body = await self.get(q="beach")
self.assertEqual(sorted(self.names(body)), ["a.png", "b.png", "c.png"])
_, body = await self.get(q="sunset")
self.assertEqual(self.names(body), [])
async def test_tag_search_reads_image_tags_only_by_tag_id(self):
self.seed_differential()
statements = []
original = self.api.db.model.get_connection
def trace_connection():
connection = original()
connection.set_trace_callback(statements.append)
return connection
with patch.object(
self.api.db.model, "get_connection", side_effect=trace_connection
):
for params in ({"tags": "beach"}, {"q": "hair"}):
response, _ = await self.get(**params)
self.assertEqual(response.status, 200)
reads = [
sql
for sql in statements
if "media_image_tags" in sql and sql.lstrip().upper().startswith("SELECT")
]
self.assertTrue(reads)
for sql in reads:
# Only a tag-id lookup, an EXISTS probe on the tag index, or the
# current records' own tags (byte_id IN) read the junction.
self.assertTrue(
"tag_id IN" in sql or "EXISTS" in sql or "byte_id IN" in sql, sql
)
async def test_the_image_tag_lookup_never_reads_prompt_tags(self):
self.seed_differential()
linked = self.media("linked.png", mtime=105)
prompt = self._save_prompt(
"Prompt with its own tag", tags=["prompt_only_beach"]
)
self.api.db.link_image_to_prompt(prompt, str(linked))
db = self.api.db
# The tag is only on a prompt: the image-tag lookup never yields it,
# under the Tags filter or the text search.
self.assertEqual(db.image_tag_ids(["prompt_only_beach"], ""), [])
self.assertEqual(db.image_tag_ids([], "prompt_only"), [])
self.assertEqual(
db.image_tag_locators(db.image_tag_ids([], "beach")),
reference_candidates(db, [], "beach"),
)
# The search still matches through the linked positive's prompt tags
# (the design keeps that, #379 "Behavior that does not change").
_, body = await self.get(q="prompt_only")
self.assertEqual(self.names(body), ["linked.png"])
@unittest.skipUnless(os.environ.get("PM_PERF") == "1", "timing run (PM_PERF=1)")
async def test_tag_search_timing(self):
"""Linux timing for the PR body: 23,600 records, 10 tags each."""
import hashlib
import statistics
import time
db = self.api.db
epoch = db.get_execution_epoch()
names = [f"name_{n:04d}" for n in range(2000)]
root = os.path.realpath(self.output_dir)
for n in range(23_600):
path = self.output_dir / "perf" / f"{n:05d}.png"
path.parent.mkdir(exist_ok=True)
path.write_bytes(b"x")
os.utime(path, (n + 1, n + 1))
stat = path.stat()
db.replace_image_tags(
locator=os.path.normcase(os.path.join(root, "perf", path.name)),
sha256=hashlib.sha256(str(n).encode()).hexdigest(),
size=stat.st_size,
mtime_ns=stat.st_mtime_ns,
expected_epoch=epoch,
expected_tag_revision=0,
tags=[names[(n * 7 + k) % 2000] for k in range(10)],
)
await self.get(limit=1) # the listing, warm
for label, params in (
("one tag", {"tags": "name_0042"}),
("text", {"q": "e_1"}),
):
times = []
for _ in range(5):
started = time.perf_counter()
response, _ = await self.get(limit=100, **params)
times.append(time.perf_counter() - started)
self.assertEqual(response.status, 200)
print(f"PM_PERF {label}: median {statistics.median(times) * 1000:.0f} ms")
async def test_legacy_output_failure_logs_its_traceback(self):
self.media("legacy.png", mtime=100)
with (
+617
View File
@@ -0,0 +1,617 @@
"""The legacy pages and the Tag image dialog tag images, never prompts (#379, PR 2).
- ``POST /autotag/start`` (legacy admin and gallery batch) starts the
``autotag`` job titled "Tag output images (legacy page)" and relays it as
the legacy frames; closing the stream cancels it.
- ``POST /autotag/single`` suggests on a capture with the saved cap and
answers the captured ``sha256``; ``existing: "skip"`` answers
``already_tagged`` for an auto-tagged image without the model gate.
- ``POST /autotag/apply`` unions into that image's tags and marks it; a
``prompt_id`` body is refused.
- A Tag image dialog save after Generate suggestions marks the image.
No real model runs: the per-image fake from test_autotag_batch answers.
"""
import asyncio
import hashlib
import json
import sqlite3
import threading
from unittest.mock import patch
from aiohttp import web
try:
from test_autotag_batch import BatchTestCase, _absolute, _strings
except ImportError:
from tests.test_autotag_batch import BatchTestCase, _absolute, _strings
from database.execution_history_schema import rotate_history_epoch
from py import model_gate
from py.api import autotag_routes, image_tag_jobs
from py.config import AutotagConfig
START = "/prompt_manager/autotag/start"
SINGLE = "/prompt_manager/autotag/single"
APPLY = "/prompt_manager/autotag/apply"
SUGGEST = "/prompt_manager/image-annotations/suggest"
OBSERVE = "/prompt_manager/image-annotations/observe"
TAGS = "/prompt_manager/image-annotations/tags"
WAIT = 30.0
LEGACY_TITLE = "Tag output images (legacy page)"
def frames_of(text):
return [
json.loads(line[len("data: ") :])
for line in text.splitlines()
if line.startswith("data: ")
]
class LegacyTestCase(BatchTestCase):
async def legacy(self, query="model_type=wd14-vit&skip_tagged=true"):
response = await self.client.post(f"{START}?{query}")
return response, frames_of(await response.text())
def digest(self, path):
return hashlib.sha256((self.output_dir / path).read_bytes()).hexdigest()
async def single(self, body):
response = await self.client.post(SINGLE, json=body)
return response.status, await response.json()
async def apply(self, body):
response = await self.client.post(APPLY, json=body)
return response.status, await response.json()
def snapshot_prompts(self):
conn = sqlite3.connect(self._temp_db.name)
try:
return [
conn.execute(f"SELECT * FROM {t} ORDER BY 1,2").fetchall()
for t in ("prompts", "prompt_tags")
]
finally:
conn.close()
class TestLegacyBatch(LegacyTestCase):
async def test_the_legacy_batch_tags_each_image_and_relays_frames(self):
prompt = self.prompt("one prompt", tags=["beach"])
a = self.image("a.png", (1, 2, 3), sees=["bird"])
b = self.image("b.png", (4, 5, 6), sees=["waves"])
self.link(prompt, a)
self.link(prompt, b)
before = self.snapshot_prompts()
ticks = iter(range(1000))
with patch.object(
autotag_routes, "_monotonic", side_effect=lambda: next(ticks)
):
response, frames = await self.legacy()
self.assertEqual(response.status, 200)
self.assertEqual(self.stored("a.png")["tags"], ["bird"])
self.assertEqual(self.stored("b.png")["tags"], ["waves"])
self.assertEqual(self.snapshot_prompts(), before)
phases = [f["status"] for f in frames if f.get("progress") in (0, 5, 8)]
self.assertEqual(
phases,
sorted(
phases,
key=[
"Taking a safety backup…",
"Loading model…",
"Listing output images…",
].index,
),
)
self.assertIn("Listing output images…", phases)
rows = [f for f in frames if f["type"] == "progress" and "processed" in f]
self.assertTrue(rows)
for row in rows:
self.assertEqual(
set(row),
{"type", "progress", "status", "processed", "tagged", "skipped"},
)
self.assertTrue(10 <= row["progress"] <= 95, row)
self.assertEqual(
frames[-1],
{
"type": "complete",
"progress": 100,
"processed": 2,
"tagged": 2,
"skipped": 0,
"errors": 0,
"skipped_outside_roots": 0,
"status": "Complete",
"model_status": "Model kept in memory",
"model_loaded": True,
},
)
job = self.api._jobs.list()[0]
self.assertEqual(
(job.title, job.kind, job.state), (LEGACY_TITLE, "autotag", "finished")
)
self.assertEqual([s for s in _strings(frames) if _absolute(s)], [])
# Skip: the marked images are skipped before the model runs.
self.tagger.calls.clear()
_, again = await self.legacy()
self.assertEqual(again[-1]["skipped"], 2)
self.assertEqual(again[-1]["processed"], 0)
self.assertEqual(self.tagger.calls, [])
async def test_no_still_images_and_keep_in_memory_false(self):
_, frames = await self.legacy("model_type=wd14-vit&keep_in_memory=false")
self.assertEqual(frames[-1]["status"], "No still images found")
self.assertEqual(frames[-1]["model_loaded"], False)
self.assertEqual(frames[-1]["model_status"], "Model unloaded")
self.assertEqual(self.tagger.unload_calls, 1)
async def test_language_model_options_come_from_the_page_or_the_template(self):
self.image("a.png", (1, 1, 1))
await self.legacy("model_type=gguf&prompt=Tags%20only")
self.assertEqual(self.tagger.calls[-1]["prompt"], "Tags only")
self.assertEqual(self.tagger.calls[-1]["limit"], 10)
self.image("b.png", (2, 2, 2))
await self.relist()
await self.legacy("model_type=gguf&prompt=%20")
self.assertEqual(self.tagger.calls[-1]["prompt"], AutotagConfig.TEMPLATE)
self.assertEqual(self.tagger.custom_prompt, "legacy prompt")
async def test_each_refusal_is_one_error_frame(self):
self.tagger.downloaded["wd14-vit"] = False
_, frames = await self.legacy()
self.assertEqual(len(frames), 1)
self.assertEqual(frames[0]["type"], "error")
self.assertEqual(frames[0]["status"], "error")
self.assertIn("WD14 ViT is not downloaded", frames[0]["message"])
self.tagger.downloaded["wd14-vit"] = True
_, frames = await self.legacy("model_type=wd14-vit&general_threshold=2")
self.assertEqual(
frames,
[
{
"type": "error",
"status": "error",
"message": "general_threshold must be between 0 and 1",
}
],
)
owner = model_gate.new_owner("image-annotations")
self.assertTrue(model_gate.try_acquire(owner))
try:
_, frames = await self.legacy()
finally:
model_gate.release(owner)
self.assertEqual(frames[0]["message"], autotag_routes.MODEL_BUSY_ERROR)
self.assertEqual(self.api._jobs.list(), [])
self.assertIsNone(model_gate.holder())
response = await self.client.post(f"{START}?general_threshold=abc")
self.assertEqual(response.status, 400)
async def test_busy_jobs_refuse_the_legacy_batch(self):
self.image("a.png", (1, 1, 1))
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
status, data = await self.start_batch()
self.assertEqual(status, 202, data)
job = self.api._jobs.get(data["job"]["id"])
try:
await self.wait_for(entered)
_, frames = await self.legacy()
self.assertEqual(
frames,
[
{
"type": "error",
"status": "error",
"message": "Tag output images already running",
}
],
)
finally:
self.tagger.block = None
proceed.set()
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
# A second legacy batch keeps its JSON 409.
self.image("b.png", (2, 2, 2))
await self.relist()
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
first = asyncio.ensure_future(
self.legacy("model_type=wd14-vit&skip_tagged=false")
)
try:
await self.wait_for(entered)
second = await self.client.post(f"{START}?model_type=wd14-vit")
self.assertEqual(second.status, 409)
self.assertEqual(
await second.json(),
{"success": False, "error": "Auto-tagging is already running"},
)
finally:
self.tagger.block = None
proceed.set()
_, frames = await first
self.assertEqual(frames[-1]["type"], "complete")
async def test_a_legacy_start_that_loses_the_race_gets_the_json_409(self):
# Two starts can both pass the title check before either registers its
# job; the loser's busy refusal names the legacy batch and answers the
# same JSON 409 as a second legacy batch.
refusal = image_tag_jobs.Refusal(
409,
"busy",
f"{autotag_routes.LEGACY_BATCH_TITLE} already running",
web.json_response({}, status=409),
)
async def refused(**_kwargs):
return None, refusal
with patch.object(self.api, "_start_autotag", side_effect=refused):
response = await self.client.post(f"{START}?model_type=wd14-vit")
self.assertEqual(response.status, 409)
self.assertEqual(
await response.json(),
{"success": False, "error": "Auto-tagging is already running"},
)
async def test_closing_the_stream_cancels_the_job(self):
for n in range(3):
self.image(f"{n}.png", (n, n, n))
real_write = web.StreamResponse.write
async def flaky_write(response, data):
if b'"processed"' in data:
raise ConnectionResetError("Cannot write to closing transport")
return await real_write(response, data)
with patch.object(web.StreamResponse, "write", flaky_write):
try:
response = await self.client.post(f"{START}?model_type=wd14-vit")
await response.read()
except Exception: # nosec B110 - the client sees the dropped stream
pass
job = self.api._jobs.list()[0]
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
self.assertEqual(job.state, "cancelled")
self.assertLess(job.counters["tagged"], 3)
async def test_a_quiet_job_repeats_its_last_frame(self):
self.image("a.png", (1, 1, 1))
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
with patch.object(autotag_routes, "LEGACY_REPEAT_AFTER", 0.05):
run = asyncio.ensure_future(self.legacy())
await self.wait_for(entered)
await asyncio.sleep(0.3)
self.tagger.block = None
proceed.set()
_, frames = await run
statuses = [f.get("status") for f in frames]
repeated = [s for s in statuses if s and statuses.count(s) > 1]
self.assertTrue(repeated, statuses)
self.assertEqual(frames[-1]["type"], "complete")
async def test_a_cancel_from_activity_ends_with_a_cancelled_frame(self):
for n in range(2):
self.image(f"{n}.png", (n, n, n))
entered, proceed = threading.Event(), threading.Event()
self.tagger.block = (entered, proceed)
run = asyncio.ensure_future(self.legacy())
try:
await self.wait_for(entered)
job = self.api._jobs.active("autotag")
await self.client.post(f"/prompt_manager/jobs/{job.id}/cancel")
finally:
self.tagger.block = None
proceed.set()
_, frames = await run
self.assertEqual(
frames[-1],
{
"type": "cancelled",
"message": "Cancelled. Tags already added were kept.",
},
)
class TestLegacySingleAndApply(LegacyTestCase):
async def test_single_suggests_on_a_capture_and_apply_tags_only_that_image(self):
prompt = self.prompt("shared prompt", tags=["keep"])
a = self.image("a.png", (1, 2, 3), sees=["bird", "sky"])
b = self.image("b.png", (4, 5, 6), sees=["cat"])
image_a = self.link(prompt, a)
self.link(prompt, b)
self.saved_set("b.png", b.read_bytes(), ["hand"])
before = self.snapshot_prompts()
status, data = await self.single(
{"image_id": image_a, "model_type": "wd14-vit"}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["tags"], ["bird", "sky"])
self.assertEqual(data["sha256"], self.digest("a.png"))
self.assertEqual(data["image_tags"], [])
self.assertEqual(data["prompt_id"], prompt)
self.assertEqual(self.tagger.custom_prompt, "legacy prompt")
# Inference read the private capture, not the file.
self.assertNotIn(str(self.output_dir), self.tagger.read_paths[-1])
status, applied = await self.apply(
{"image_id": image_a, "sha256": data["sha256"], "tags": ["bird"]}
)
self.assertEqual(
(status, applied),
(200, {"success": True, "added_tags": ["bird"], "total_tags": 1}),
)
self.assertEqual(self.stored("a.png")["tags"], ["bird"])
self.assertEqual(self.stored("b.png")["tags"], ["hand"])
self.assertEqual(self.snapshot_prompts(), before)
self.assertEqual(self.marked(), {self.locator("a.png")})
async def test_skip_answers_already_tagged_without_the_gate(self):
a = self.image("a.png", (1, 2, 3), sees=["bird"])
await self.run_batch()
self.tagger.calls.clear()
owner = model_gate.new_owner("image-tag-job")
self.assertTrue(model_gate.try_acquire(owner))
try:
status, data = await self.single(
{
"path": "a.png",
"root_index": 0,
"existing": "skip",
"model_type": "wd14-vit",
}
)
finally:
model_gate.release(owner)
self.assertEqual(status, 200, data)
self.assertEqual(data["already_tagged"], True)
self.assertEqual(data["image_tags"], ["bird"])
self.assertEqual(self.tagger.calls, [])
# Hand tags alone do not count; merge always suggests.
b = self.image("b.png", (9, 9, 9))
self.saved_set("b.png", b.read_bytes(), ["hand"])
status, data = await self.single(
{"path": str(b), "existing": "skip", "model_type": "wd14-vit"}
)
self.assertNotIn("already_tagged", data)
self.assertEqual(data["image_tags"], ["hand"])
status, data = await self.single(
{"path": str(a), "existing": "merge", "model_type": "wd14-vit"}
)
self.assertNotIn("already_tagged", data)
async def test_single_uses_the_saved_cap_and_the_refusal_table(self):
self.image("a.png", (1, 2, 3))
AutotagConfig.SUGGESTION_LIMIT = 3
status, data = await self.single(
{"path": "a.png", "root_index": 0, "model_type": "wd14-vit"}
)
self.assertEqual((status, len(data["tags"])), (200, 3))
(self.output_dir / "broken.png").write_bytes(b"not an image")
await self.relist()
status, data = await self.single(
{"path": "broken.png", "root_index": 0, "model_type": "wd14-vit"}
)
self.assertEqual((status, data["code"]), (415, "not-a-still"))
self.assertIn(self.locator("broken.png"), self.marked())
self.tagger.downloaded["gguf"] = False
status, data = await self.single(
{"path": "a.png", "root_index": 0, "model_type": "gguf"}
)
self.assertEqual((status, data["code"]), (409, "model-unavailable"))
self.api._capture_store().max_file = 16
status, data = await self.single(
{"path": "a.png", "root_index": 0, "model_type": "wd14-vit"}
)
self.assertEqual((status, data["code"]), (413, "too-large"))
self.api._capture_store().max_file = 2**20
owner = model_gate.new_owner("image-annotations")
self.assertTrue(model_gate.try_acquire(owner))
try:
status, data = await self.single(
{"path": "a.png", "root_index": 0, "model_type": "wd14-vit"}
)
finally:
model_gate.release(owner)
self.assertEqual((status, data["code"]), (409, "model-busy"))
async def test_apply_body_and_answer_table(self):
a = self.image("a.png", (1, 2, 3))
digest = self.digest("a.png")
status, data = await self.apply({"prompt_id": 1, "tags": ["a"]})
self.assertEqual(
(status, data["error"]), (400, "This page is out of date. Reload it.")
)
status, _ = await self.apply(
{"path": "a.png", "root_index": 0, "sha256": "x", "tags": ["a"]}
)
self.assertEqual(status, 400)
status, _ = await self.apply(
{"path": "a.png", "root_index": 0, "sha256": digest, "tags": ["bad,name"]}
)
self.assertEqual(status, 400)
status, _ = await self.apply({"path": "a.png", "sha256": digest, "tags": []})
self.assertEqual(status, 403) # a bare relative path is not inside a root
# An empty reviewed list only marks the image.
status, data = await self.apply({"path": str(a), "sha256": digest, "tags": []})
self.assertEqual((status, data["added_tags"]), (200, []))
self.assertEqual(self.marked(), {self.locator("a.png")})
# The file changed since its tags were generated.
status, data = await self.apply(
{"path": str(a), "sha256": "0" * 64, "tags": ["x"]}
)
self.assertEqual((status, data["code"]), (409, "file-changed"))
# Too many tags.
full = self.image("full.png", (7, 7, 7))
self.saved_set("full.png", full.read_bytes(), [f"t{i}" for i in range(100)])
status, data = await self.apply(
{"path": str(full), "sha256": self.digest("full.png"), "tags": ["one_more"]}
)
self.assertEqual((status, data["code"]), (409, "tag-limit"))
# Gone.
status, data = await self.apply(
{"path": "gone.png", "root_index": 0, "sha256": digest, "tags": ["x"]}
)
self.assertIn(status, (403, 404))
async def test_apply_after_a_restore_writes_nothing(self):
a = self.image("a.png", (1, 2, 3))
digest = self.digest("a.png")
real = self.api.db.get_execution_epoch
def stale_epoch():
epoch = real()
conn = sqlite3.connect(self._temp_db.name)
try:
rotate_history_epoch(conn)
finally:
conn.close()
return epoch
with patch.object(self.api.db, "get_execution_epoch", side_effect=stale_epoch):
status, data = await self.apply(
{"path": str(a), "sha256": digest, "tags": ["x"]}
)
self.assertEqual((status, data["code"]), (409, "storage-changed"))
self.assertEqual(self.stored("a.png")["tags"], [])
async def test_image_id_and_path_bodies_reach_the_same_bytes(self):
prompt = self.prompt("p")
a = self.image("sub/a.png", (1, 2, 3), sees=["x"])
image_id = self.link(prompt, a)
_, by_id = await self.single({"image_id": image_id, "model_type": "wd14-vit"})
_, by_path = await self.single(
{"path": "sub/a.png", "root_index": 0, "model_type": "wd14-vit"}
)
self.assertEqual(by_id["sha256"], by_path["sha256"])
status, _ = await self.apply(
{"image_id": image_id, "sha256": by_id["sha256"], "tags": ["x"]}
)
self.assertEqual(status, 200)
status, data = await self.apply(
{
"path": "sub/a.png",
"root_index": 0,
"sha256": by_path["sha256"],
"tags": ["y"],
}
)
self.assertEqual((status, data["total_tags"]), (200, 2))
class TestDialogMarks(LegacyTestCase):
"""DEC4: a Tag image dialog save after Generate suggestions marks the image."""
def target(self, path):
return {
"root_index": 0,
"root_scope": self.api._gallery_root_scope(self.output_dir, 0),
"path": path,
}
async def observe(self, path):
response = await self.client.post(OBSERVE, json={"target": self.target(path)})
data = await response.json()
self.assertEqual(response.status, 200, data)
return data
async def suggest(self, observed, path):
response = await self.client.post(
SUGGEST,
json={
"target": self.target(path),
"capture_token": observed["capture"]["token"],
"expected_epoch": observed["library_epoch"],
"model_type": "wd14-vit",
"options": {"general_threshold": 0.35, "character_threshold": 0.85},
},
)
self.assertEqual(response.status, 200, await response.text())
async def save(self, observed, path, tags, revision=None):
response = await self.client.put(
TAGS,
json={
"target": self.target(path),
"capture_token": observed["capture"]["token"],
"sha256": observed["capture"]["sha256"],
"expected_epoch": observed["library_epoch"],
"expected_tag_revision": (
observed["tag_revision"] if revision is None else revision
),
"tags": tags,
},
)
return response.status, await response.json()
async def test_save_after_generate_marks_and_a_skip_batch_leaves_it(self):
self.image("d.png", (1, 2, 3), sees=["bird", "sky"])
observed = await self.observe("d.png")
await self.suggest(observed, "d.png")
status, _ = await self.save(observed, "d.png", ["sky"]) # bird removed
self.assertEqual(status, 200)
self.assertEqual(self.marked(), {self.locator("d.png")})
job = await self.run_batch(existing="skip")
self.assertEqual(self.rows(job, "skipped"), [("d.png", "Already auto-tagged")])
self.assertEqual(self.stored("d.png")["tags"], ["sky"])
async def test_save_without_generate_does_not_mark(self):
self.image("e.png", (1, 2, 3))
observed = await self.observe("e.png")
status, _ = await self.save(observed, "e.png", ["hand"])
self.assertEqual(status, 200)
self.assertEqual(self.marked(), set())
async def test_a_conflict_then_a_saved_retry_marks(self):
f = self.image("f.png", (1, 2, 3))
observed = await self.observe("f.png")
await self.suggest(observed, "f.png")
self.saved_set("f.png", f.read_bytes(), ["other"]) # another writer
status, data = await self.save(observed, "f.png", ["mine"])
self.assertEqual((status, data["code"]), (409, "tag-conflict"))
self.assertEqual(self.marked(), set())
status, _ = await self.save(
observed, "f.png", ["mine"], revision=data["tag_revision"]
)
self.assertEqual(status, 200)
self.assertEqual(self.marked(), {self.locator("f.png")})
async def test_another_capture_does_not_inherit_the_mark(self):
self.image("g.png", (1, 2, 3))
self.image("h.png", (4, 5, 6))
first = await self.observe("g.png")
await self.suggest(first, "g.png")
second = await self.observe("h.png")
status, _ = await self.save(second, "h.png", ["x"])
self.assertEqual(status, 200)
self.assertEqual(self.marked(), set())
self.assertIn(first["capture"]["token"], self.api._suggested_tokens)
async def test_gone_tokens_are_dropped_on_the_next_insert_without_touching_clocks(
self,
):
self.image("i.png", (1, 2, 3))
self.image("j.png", (4, 5, 6))
first = await self.observe("i.png")
await self.suggest(first, "i.png")
store = self.api._capture_store()
store.release(first["capture"]["token"])
second = await self.observe("j.png")
seen = []
real_get = store.get
with patch.object(
store, "get", side_effect=lambda t: seen.append(t) or real_get(t)
):
await self.suggest(second, "j.png")
self.assertEqual(seen, [])
self.assertEqual(self.api._suggested_tokens, {second["capture"]["token"]})
-44
View File
@@ -1398,50 +1398,6 @@ class TestModelHousekeeping(DatabaseTestCase):
self.assertIsNot(self.db.model.get_connection(), failing)
class TestAutotagBatchCandidates(DatabaseTestCase):
"""#328: the batch's candidate read never loads workflow documents."""
def test_autotag_candidates_skip_workflow_data(self):
from utils.hashing import generate_prompt_hash
first = self.db.save_prompt(
text="a", tags=["one", "two"], prompt_hash=generate_prompt_hash("a")
)
second = self.db.save_prompt(text="b", prompt_hash=generate_prompt_hash("b"))
old = self.db.link_image_to_prompt(
first, "/out/a.png", {"workflow": {"big": "x" * 1000}}
)
new = self.db.link_image_to_prompt(second, "/out/b.png")
statements = []
conn = self.db.model.get_connection() # this thread's own connection
conn.set_trace_callback(statements.append)
try:
rows = self.db.autotag_batch_candidates()
finally:
conn.set_trace_callback(None)
self.assertEqual(
rows[0],
{
"id": new,
"image_path": "/out/b.png",
"prompt_id": second,
"prompt_tags": [],
},
)
self.assertEqual(
{**rows[1], "prompt_tags": sorted(rows[1]["prompt_tags"])},
{
"id": old,
"image_path": "/out/a.png",
"prompt_id": first,
"prompt_tags": ["one", "two"],
},
)
self.assertEqual(len(rows), 2)
self.assertTrue(statements)
self.assertFalse(any("workflow_data" in sql for sql in statements))
if __name__ == "__main__":
unittest.main()
+49 -7
View File
@@ -31,6 +31,12 @@ from utils.execution_history_writer import HistoryWriter
# 0.19 s on macOS py3.13 (PR #351).
NON_BLOCKING_CEILING = 1.0
# Saturation tests block the worker while they queue 1023 more items. The
# block must hold until the test releases it (#346/#352): a 3 s timer let a
# loaded runner unblock the worker mid-fill and drain it. This is only a hang
# guard; the test, or cleanup, always sets the release.
SATURATION_HOLD_SECONDS = 60
class ManualClock:
def __init__(self):
@@ -84,6 +90,26 @@ class HistoryWriterTests(unittest.TestCase):
threading.Event().wait(0.005)
self.fail("finite deadline expired waiting for worker behavior")
def wait_for_worker_pass(self, writer):
"""Wait until the idle worker has run one whole loop pass from now.
A worker with no accepted work ends every pass by parking on its wake
event. The first park counted may end a pass already under way; the
second ends a pass that began after this call, so whatever the worker
decides at the current clock value has happened (state, not a
wall-clock pause that a loaded runner can outlast).
"""
wake = writer._wake
park = wake.wait
parks = []
def counted(timeout=None):
parks.append(timeout)
return park(timeout)
with patch.object(wake, "wait", counted):
self.wait_for(lambda: len(parks) >= 2)
def writer(self, factory=None, ready=True):
writer = HistoryWriter(factory or (lambda: self.db), clock=self.clock)
self.writers.append(writer)
@@ -301,7 +327,7 @@ class HistoryWriterTests(unittest.TestCase):
def write(command):
entered.set()
release.wait(3)
release.wait(SATURATION_HOLD_SECONDS)
return original(command)
with patch.object(self.db, "write_execution_start", write):
@@ -348,9 +374,12 @@ class HistoryWriterTests(unittest.TestCase):
first = self.sealed(writer)
charge = first.lease.byte_count
left = threading.Event()
def write(*args, **kwargs):
entered.set()
release.wait(3)
left.set()
return original(*args, **kwargs)
with patch.object(self.db, "write_execution_contexts_for_finish", write):
@@ -396,8 +425,21 @@ class HistoryWriterTests(unittest.TestCase):
before = monotonic()
self.assertFalse(writer.close(0.02))
elapsed = monotonic() - before
# close() waits until its own monotonic deadline, so the lower
# bound holds on any runner.
self.assertGreaterEqual(elapsed, 0.02)
self.assertLess(elapsed, 0.2)
if early_join:
# Manual clock: only the patched join/sleep advance it, so this
# exact bound proves early joins do not spin forever.
self.assertLess(elapsed, 0.2)
else:
# Real clock: a loaded runner may run late, but close() must
# not wait anywhere near the 3 s the blocked write holds.
self.assertLess(elapsed, 0.02 + NON_BLOCKING_CEILING)
# Bounded on the real clock too: close() gave up while the
# in-flight write was still blocked, instead of waiting it out.
self.assertFalse(left.is_set())
self.assertTrue(writer.status().alive)
self.assertEqual(pool.charged_bytes, charge)
replacement = HistoryWriter(lambda: self.db, context_budget=pool)
self.writers.append(replacement)
@@ -771,7 +813,7 @@ class HistoryWriterTests(unittest.TestCase):
self.wait_for(lambda: not writer.status().accepting)
self.assertFalse(writer.submit(fresh))
self.clock.advance(4.99)
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertEqual(probes, [0])
self.clock.advance(0.01)
self.wait_for(lambda: writer.status().accepting)
@@ -794,12 +836,12 @@ class HistoryWriterTests(unittest.TestCase):
writer = self.writer(factory, ready=False)
self.wait_for(lambda: len(calls) == 1)
self.clock.advance(4.99)
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertEqual(calls, [0])
self.clock.advance(0.01)
self.wait_for(lambda: len(calls) == 2)
self.clock.advance(4.99)
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertEqual(calls, [0, 5])
self.clock.advance(0.01)
self.wait_for(lambda: writer.status().accepting)
@@ -853,7 +895,7 @@ class HistoryWriterTests(unittest.TestCase):
)
writer.invalidate()
release.set()
threading.Event().wait(0.08)
self.wait_for_worker_pass(writer)
self.assertFalse(writer.status().accepting)
self.assertIsNone(writer.status().epoch)
self.assertFalse(writer.submit(self.start_command(epoch=stale_epoch)))
@@ -871,7 +913,7 @@ class HistoryWriterTests(unittest.TestCase):
lock.execute("BEGIN IMMEDIATE")
before = time.monotonic()
self.assertTrue(writer.submit(command))
self.assertLess(time.monotonic() - before, 0.1)
self.assertLess(time.monotonic() - before, NON_BLOCKING_CEILING)
self.wait_for(lambda: not writer.status().accepting)
lock.rollback()
self.clock.advance(5)
+11 -4
View File
@@ -18,6 +18,7 @@ from utils.execution_output_capture import seal_outputs
from utils.execution_outputs import snapshot_output_roots
from utils.execution_history_writer import HistoryWriter
from tests.history_waits import drain_writer
from tests.disk_waits import DISK_BOUND_SECONDS
try:
from tests import test_execution_history_writer as fixtures
@@ -28,11 +29,17 @@ except ImportError:
# Upper bound a blocked fake write waits for its release. Tests prove ordering
# with events, so this only caps a failing run and never decides a pass.
BLOCK_SECONDS = 10
# Saturation tests block the worker while they queue 1023 more items. The
# block must hold until the test releases it (#346/#352): a 3 s timer let a
# loaded runner unblock the worker mid-fill and drain it. This is only a hang
# guard; the test, or cleanup, always sets the release.
SATURATION_HOLD_SECONDS = 60
# A restore is a full verify/migrate/backup/replace cycle and is bound by the
# disk: a busy Windows runner took over 10 s (merge queue, #382). Waits that
# span a restore use this ceiling, which stays under the runner's 120 s hang
# watchdog, so a real hang still fails with the thread dump.
RESTORE_SECONDS = 90
# span a restore use the shared disk-bound hang guard (tests/disk_waits.py).
RESTORE_SECONDS = DISK_BOUND_SECONDS
class OutputWriterTests(fixtures.unittest.TestCase):
@@ -591,7 +598,7 @@ class OutputWriterTests(fixtures.unittest.TestCase):
def write(command):
entered.set()
release.wait(3)
release.wait(SATURATION_HOLD_SECONDS)
return original(command)
with patch.object(self.db, "write_execution_start", write):
+266
View File
@@ -0,0 +1,266 @@
"""A thread or pool refused at interpreter shutdown is quiet (#375).
When the interpreter exits, ``threading._shutdown`` first shuts every
``ThreadPoolExecutor`` (``submit`` then raises "cannot schedule new futures
after interpreter shutdown") and joins the non-daemon threads; only then do
the ``atexit`` callbacks (the package teardown) run. Our daemon threads keep
working meanwhile. ``Thread.start`` is refused ("can't create new thread at
interpreter shutdown") only later, once the interpreter is finalizing; the
unit tests cover that case, the child process the pool case. Such a refusal
is logged once at DEBUG, with no traceback, and leaves nothing pending.
``threading.Thread`` is replaced only in the module under test (a copy of
the ``threading`` namespace), never process-wide. The child-process test runs
the real exit: a daemon thread queues previews while the main thread returns.
"""
import json
import logging
import os
import subprocess
import sys
import tempfile
import textwrap
import threading
import types
import unittest
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from unittest import mock
from py.api import thumbnail_auto
from utils import image_monitor, parallel
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CHILD_CEILING_SECONDS = 60 # the hang watchdog allows 120 s per test
REPORT_PREFIX = "PM_EXIT_REPORT "
THREAD_REFUSAL = "can't create new thread at interpreter shutdown"
POOL_REFUSAL = "cannot schedule new futures after interpreter shutdown"
def _refusing_threading():
"""A ``threading`` namespace whose ``Thread.start`` is refused."""
class RefusedThread(threading.Thread):
def start(self):
raise RuntimeError(THREAD_REFUSAL)
namespace = types.SimpleNamespace(**vars(threading))
namespace.Thread = RefusedThread
return namespace
class _ShutPool(ThreadPoolExecutor):
def submit(self, *args, **kwargs):
raise RuntimeError(POOL_REFUSAL)
class _Capture(logging.Handler):
def __init__(self):
super().__init__(logging.DEBUG)
self.records = []
def emit(self, record):
self.records.append(record)
def _capture(testcase, name):
parallel._noted.clear() # "once" is per process: start each test afresh
logger = logging.getLogger(name)
handler = _Capture()
previous = logger.level
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
def restore():
logger.removeHandler(handler)
logger.setLevel(previous)
testcase.addCleanup(restore)
return handler.records
def _assert_one_debug_line(testcase, records):
"""Exactly one refusal line, at DEBUG and without a traceback."""
refusals = [r for r in records if "interpreter is exiting" in r.getMessage()]
testcase.assertEqual(len(refusals), 1, [r.getMessage() for r in records])
testcase.assertEqual(refusals[0].levelno, logging.DEBUG)
testcase.assertEqual([r for r in records if r.levelno > logging.DEBUG], [])
testcase.assertTrue(all(r.exc_info is None for r in records))
class TestScheduleRefusedAtShutdown(unittest.TestCase):
def setUp(self):
self.handler = image_monitor.ImageGenerationHandler(
mock.MagicMock(), mock.MagicMock()
)
self.addCleanup(self.handler.close)
def test_schedule_returns_false_and_leaves_nothing_pending(self):
records = _capture(self, "prompt_manager.image_monitor")
with mock.patch.object(image_monitor, "threading", _refusing_threading()):
self.assertFalse(self.handler.schedule("/out/a.png"))
self.assertFalse(self.handler.schedule("/out/b.png"))
self.assertEqual(self.handler.pending_count(), 0)
_assert_one_debug_line(self, records)
def test_another_start_failure_still_raises_and_leaves_nothing_pending(self):
class Failing(threading.Thread):
def start(self):
raise RuntimeError("can't start new thread")
namespace = types.SimpleNamespace(**vars(threading))
namespace.Thread = Failing
with mock.patch.object(image_monitor, "threading", namespace):
with self.assertRaises(RuntimeError):
self.handler.schedule("/out/a.png")
self.assertEqual(self.handler.pending_count(), 0)
def test_schedule_before_shutdown_is_unchanged(self):
self.handler.processing_delay = 3600
self.assertTrue(self.handler.schedule("/out/a.png"))
self.assertEqual(self.handler.pending_count(), 1)
class TestPreviewQueueRefusedAtShutdown(unittest.TestCase):
def thumbnailer(self, logger):
made = thumbnail_auto.AutoThumbnailer(
lambda *a, **k: {"action": "generated"}, lambda: True, logger
)
self.addCleanup(made.close)
return made
def test_enqueue_is_refused_quietly_and_drops_the_file(self):
logger = logging.getLogger("prompt_manager.test_exit_window")
records = _capture(self, logger.name)
thumbnailer = self.thumbnailer(logger)
src = Path("/out/a.png")
with mock.patch.object(thumbnail_auto, "threading", _refusing_threading()):
self.assertFalse(thumbnailer.enqueue(src, Path("/out/t/a.webp"), False))
self.assertFalse(thumbnailer.enqueue(src, Path("/out/t/b.webp"), False))
self.assertFalse(thumbnailer._worker_running)
self.assertEqual(len(thumbnailer._queue), 0)
self.assertEqual(thumbnailer._queued, set())
self.assertTrue(thumbnailer.wait_idle(0))
_assert_one_debug_line(self, records)
class TestMapParallelRefusedAtShutdown(unittest.TestCase):
def test_a_shut_pool_runs_the_items_on_the_calling_thread(self):
records = _capture(self, "prompt_manager.parallel")
with mock.patch.object(parallel, "ThreadPoolExecutor", _ShutPool):
self.assertEqual(
parallel.map_parallel(lambda n: n * n, [1, 2, 3], 4), [1, 4, 9]
)
_assert_one_debug_line(self, records)
def test_another_pool_error_still_raises(self):
class Broken(ThreadPoolExecutor):
def submit(self, *args, **kwargs):
raise RuntimeError("something else")
with mock.patch.object(parallel, "ThreadPoolExecutor", Broken):
with self.assertRaises(RuntimeError):
parallel.map_parallel(lambda n: n, [1, 2], 2)
CHILD = textwrap.dedent("""
import threading
# Registered before concurrent.futures is imported, so it runs after the
# pool's own exit hook: from here on, a pool refuses new work.
exiting = threading.Event()
threading._register_atexit(exiting.set)
import json
import logging
import os
import sys
from pathlib import Path
root, tmp = sys.argv[1], sys.argv[2]
sys.path.insert(0, root)
from py.api.thumbnail_auto import AutoThumbnailer
logging.basicConfig(level=logging.WARNING, stream=sys.stderr)
logger = logging.getLogger("prompt_manager.exit_child")
def generate(src, dst, is_video, *, stop, guard, track):
Path(dst).write_bytes(b"preview")
return {"action": "generated"}
thumbnailer = AutoThumbnailer(
generate, lambda: True, logger, workers=lambda: 4
)
queued = threading.Event()
def late_monitor():
exiting.wait()
# One batch of two: the worker maps it on a pool.
with thumbnailer._changed:
for name in ("a", "b"):
src = Path(tmp, name + ".png")
src.write_bytes(b"x")
thumbnailer.enqueue(src, Path(tmp, name + "_thumb.webp"), False)
queued.set()
def holder():
# Non-daemon: the interpreter waits for it, inside the exit window.
exiting.wait()
queued.wait(30)
idle = thumbnailer.wait_idle(30)
written = sorted(
n for n in os.listdir(tmp) if n.endswith("_thumb.webp")
)
print("PM_EXIT_REPORT " + json.dumps({"idle": idle, "written": written}))
sys.stdout.flush()
threading.Thread(target=late_monitor, daemon=True).start()
threading.Thread(target=holder).start()
# Return normally, as ComfyUI does after Ctrl+C.
""")
class TestExitWindowChild(unittest.TestCase):
def test_previews_queued_in_the_exit_window_log_no_traceback(self):
with tempfile.TemporaryDirectory() as tmp:
script = os.path.join(tmp, "child.py")
with open(script, "w", encoding="utf-8") as handle:
handle.write(CHILD)
env = dict(os.environ)
env.pop("PROMPT_MANAGER_CONFIG_PATH", None)
env["PROMPT_MANAGER_DATA_DIR"] = tmp
env["PYTHONIOENCODING"] = "utf-8"
try:
completed = subprocess.run(
[sys.executable, script, ROOT, tmp],
cwd=ROOT,
env=env,
capture_output=True,
timeout=CHILD_CEILING_SECONDS,
encoding="utf-8",
errors="replace",
)
except subprocess.TimeoutExpired as hung:
self.fail(
f"the child did not exit within {CHILD_CEILING_SECONDS} s\n"
f"stdout:\n{hung.stdout}\nstderr:\n{hung.stderr}"
)
detail = f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
self.assertEqual(completed.returncode, 0, detail)
reports = [
line[len(REPORT_PREFIX) :]
for line in completed.stdout.splitlines()
if line.startswith(REPORT_PREFIX)
]
self.assertEqual(len(reports), 1, detail)
report = json.loads(reports[0])
self.assertTrue(report["idle"], detail)
# The batch still ran: on the worker's own thread.
self.assertEqual(report["written"], ["a_thumb.webp", "b_thumb.webp"], detail)
self.assertNotIn("Traceback", completed.stderr, detail)
if __name__ == "__main__":
unittest.main()
+4 -2
View File
@@ -535,7 +535,9 @@ class TestHeldJobStates(_LegacyHoldCase):
self.assertIn("event: error", text)
self.assertEqual(await self._listed(), [("thumbnails", "failed")])
async def test_legacy_autotag_that_ends_in_error_is_failed(self):
async def test_legacy_autotag_refused_at_start_leaves_no_job(self):
# #379: the legacy batch is the autotag job; a model that is not
# downloaded refuses the start (one error frame) and registers nothing.
service = MagicMock()
service.get_models_status.return_value = {} # model not downloaded
with patch("py.autotag.get_autotag_service", return_value=service):
@@ -545,7 +547,7 @@ class TestHeldJobStates(_LegacyHoldCase):
text = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn('"type": "error"', text)
self.assertEqual(await self._listed(), [("autotag", "failed")])
self.assertEqual(await self._listed(), [])
if __name__ == "__main__":
+136
View File
@@ -0,0 +1,136 @@
"""The legacy pages' backend contract does not move under System S13 (#330).
S13 points the legacy admin and gallery buttons at native System and removes
their duplicate dialogs, changing no backend file. This snapshot pins what the
legacy pages talk to: every registered route (method and path) and the key
paths of ``GET /prompt_manager/settings``, the one route whose legacy request
body S13 shrinks. Record again only with a reason in the PR body:
PM_RECORD_LEGACY_SNAPSHOT=1 python3 -m unittest tests.test_legacy_contract_snapshot
Cutover (#334) removes this test and its fixture.
"""
import json
import os
import unittest
from pathlib import Path
try:
from tests.test_api_admin import AdminAPITestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase
from py.config import GalleryConfig # noqa: E402
SNAPSHOT = (
Path(__file__).resolve().parent / "fixtures" / "legacy_contract_snapshot.json"
)
RECORD_ENV = "PM_RECORD_LEGACY_SNAPSHOT"
SETTINGS = "/prompt_manager/settings"
DEPTH = 2
CHANGED = (
"the legacy contract changed; if the change is intended (a new route, a new"
" settings key), record it with PM_RECORD_LEGACY_SNAPSHOT=1 and say why in"
" the PR body"
)
def key_paths(value, prefix="", depth=1):
"""The key paths of a JSON object to ``DEPTH`` levels.
``{"a": {"b": 1}}`` gives ``a`` and ``a.b``; the items of a list of
objects give the union of their keys as ``a[].b``.
"""
paths = set()
for key, child in value.items():
path = f"{prefix}{key}"
paths.add(path)
if depth >= DEPTH:
continue
if isinstance(child, dict):
paths |= key_paths(child, f"{path}.", depth + 1)
elif isinstance(child, list):
for item in child:
if isinstance(item, dict):
paths |= key_paths(item, f"{path}[].", depth + 1)
return paths
class TestLegacyContractSnapshot(AdminAPITestCase):
"""Routes and settings keys against tests/fixtures/legacy_contract_snapshot.json.
The base case isolates config.json (``PROMPT_MANAGER_CONFIG_PATH`` in the
temp tree), stubs ComfyUI's ``folder_paths`` and resets the gallery roots;
nothing here reads the maintainer's install.
"""
async def asyncSetUp(self):
await super().asyncSetUp()
self.api._get_all_output_dirs = lambda: [str(self.output_dir)]
GalleryConfig.MONITORING_DIRECTORIES = [str(self.output_dir)]
def _routes(self):
return sorted(
{
f"{route.method} {route.resource.canonical}"
for route in self.app.router.routes()
if route.method != "HEAD"
}
)
async def _settings_keys(self):
resp = await self.client.request("GET", SETTINGS)
body = await resp.json()
self.assertEqual(resp.status, 200, body)
return sorted(key_paths(body))
def _snapshot(self):
with open(SNAPSHOT, encoding="utf-8") as f:
return json.load(f)
def _record(self, key, value):
recorded = self._snapshot() if SNAPSHOT.exists() else {}
recorded[key] = value
with open(SNAPSHOT, "w", encoding="utf-8", newline="\n") as f:
json.dump(recorded, f, indent=2)
f.write("\n")
async def test_routes_match_the_snapshot(self):
routes = self._routes()
if os.environ.get(RECORD_ENV) == "1":
self._record("routes", routes)
self.assertGreater(len(routes), 50)
self.assertEqual(routes, self._snapshot()["routes"], CHANGED)
async def test_settings_response_keeps_its_keys(self):
keys = await self._settings_keys()
if os.environ.get(RECORD_ENV) == "1":
self._record("settings_keys", keys)
self.assertIn("settings.infinite_scroll", keys)
self.assertEqual(keys, self._snapshot()["settings_keys"], CHANGED)
class TestKeyPaths(unittest.TestCase):
def test_depth_and_list_items(self):
value = {
"success": True,
"settings": {"a": 1, "deep": {"hidden": 1}},
"rows": [{"path": "x"}, {"label": "y"}],
}
self.assertEqual(
sorted(key_paths(value)),
[
"rows",
"rows[].label",
"rows[].path",
"settings",
"settings.a",
"settings.deep",
"success",
],
)
if __name__ == "__main__":
unittest.main()
+158 -1
View File
@@ -115,7 +115,12 @@ class TestPairScan(AdminAPITestCase):
self.assertEqual(second["pairs_added"], 0)
def test_legacy_single_prompt_second_pass_not_reported_as_new_link(self):
meta = {"prompt": json.dumps({"1": graph()["1"]})}
# A sampler-less graph: one encoder wired into a guider (#252).
single = {
"1": graph()["1"],
"2": {"class_type": "BasicGuider", "inputs": {"conditioning": ["1", 0]}},
}
meta = {"prompt": json.dumps(single)}
path = self.output_dir / "old.png"
self.assertEqual(self.api._ingest_scanned_file(path, meta), "added")
self.assertEqual(self.api._ingest_scanned_file(path, meta), "unchanged")
@@ -292,3 +297,155 @@ class TestPairScan(AdminAPITestCase):
self.api.db.get_image_prompt_info(str(path))["workflow_data"], workflow
)
self.assertEqual(self.api._ingest_scanned_file(path, metadata), "unchanged")
def flux_graph(text="a red fox", encoder="CLIPTextEncode"):
"""A sampler-less Flux graph: the prompt reaches a guider's conditioning (#252)."""
return {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux.safetensors"}},
"2": {"class_type": "DualCLIPLoader", "inputs": {"clip_name1": "t5"}},
"6": {"class_type": encoder, "inputs": {"text": text, "clip": ["2", 0]}},
"22": {
"class_type": "BasicGuider",
"inputs": {"model": ["1", 0], "conditioning": ["6", 0]},
},
"13": {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["22", 0], "sigmas": ["17", 0]},
},
"17": {"class_type": "BasicScheduler", "inputs": {"steps": 20}},
"9": {"class_type": "SaveImage", "inputs": {"images": ["13", 0]}},
}
class TestSamplerLessScan(AdminAPITestCase):
"""Graphs where no node takes a positive or negative input (#252)."""
def _scanned(self, name, prompt_graph, workflow=None):
meta = {"prompt": json.dumps(prompt_graph)}
if workflow is not None:
meta["workflow"] = json.dumps(workflow)
return self.api._ingest_scanned_file(self.output_dir / name, meta)
def _texts(self):
return [p["text"] for p in self.api.db.get_recent_prompts()["prompts"]]
def test_the_one_wired_prompt_is_linked(self):
self.assertEqual(self._scanned("flux.png", flux_graph()), "added")
self.assertEqual(self._texts(), ["a red fox"])
def test_a_text_node_feeding_the_encoder_is_one_prompt(self):
g = flux_graph(text=["30", 0])
g["30"] = {"class_type": "TextInput", "inputs": {"text": "a red fox"}}
self.assertEqual(self._scanned("textinput.png", g), "added")
self.assertEqual(self._texts(), ["a red fox"])
def test_a_display_node_that_feeds_nothing_is_never_the_prompt(self):
# A real scan stored a "=== GIT CLONE REPORT ===" display text this way.
g = flux_graph()
g["40"] = {
"class_type": "easy showAnything",
"inputs": {"text": "=== GIT CLONE REPORT ===\nTotal repositories: 1"},
}
self.assertEqual(self._scanned("report.png", g), "added")
self.assertEqual(self._texts(), ["a red fox"])
def test_only_unconsumed_texts_store_nothing(self):
g = {
"40": {"class_type": "easy showAnything", "inputs": {"text": "report"}},
"41": {"class_type": "Textbox", "inputs": {"text": "install note"}},
}
self.assertEqual(self._scanned("notes.png", g), "found")
self.assertEqual(self._texts(), [])
def test_two_wired_texts_are_ambiguous_and_stay_unlinked(self):
# A LoRA loader's text input is wired too: the scan stored "<lora:...>".
g = flux_graph()
g["50"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {
"model": ["1", 0],
"clip": ["2", 0],
"text": "<lora:style:1.00>",
},
}
g["22"]["inputs"]["model"] = ["50", 0]
g["6"]["inputs"]["clip"] = ["50", 1]
self.assertEqual(self._scanned("lora.png", g), "unresolved")
self.assertEqual(self._texts(), [])
def test_an_untraceable_wired_text_makes_the_graph_ambiguous(self):
# The encoder's text comes from a node no rule reads: never guess the other.
g = flux_graph(text=["60", 0])
g["60"] = {"class_type": "SomeLLMNode", "inputs": {"seed": 1}}
g["50"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
}
g["22"]["inputs"]["model"] = ["50", 0]
self.assertEqual(self._scanned("llm.png", g), "unresolved")
self.assertEqual(self._texts(), [])
def test_this_repositorys_prompt_node_is_authoritative(self):
g = flux_graph(text="a red fox", encoder="PromptManager")
g["50"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
}
g["22"]["inputs"]["model"] = ["50", 0]
self.assertEqual(self._scanned("own.png", g), "added")
self.assertEqual(self._texts(), ["a red fox"])
def test_a_numeric_text_is_never_stored_as_a_prompt(self):
self.assertEqual(self._scanned("number.png", flux_graph(text="2048")), "found")
self.assertEqual(self._texts(), [])
def test_workflow_widget_values_are_never_read(self):
# The API text is linked from a node no rule reads; the UI workflow
# still holds a stale widget value, which the old scan stored.
g = flux_graph(text=["70", 0])
g["70"] = {"class_type": "PromptBuilder", "inputs": {"prompt": "x", "n": 2}}
workflow = {
"nodes": [
{"id": 6, "type": "CLIPTextEncode", "widgets_values": ["stale text"]}
]
}
self.assertEqual(self._scanned("stale.png", g, workflow), "unresolved")
self.assertEqual(self._texts(), [])
def test_a_lora_loaders_trigger_word_output_is_not_its_text(self):
# The encoder reads the loader's trigger-word output (slot 2), not its
# text input: the prompt is unknown, never the "<lora:...>" syntax.
g = flux_graph(text=["50", 2])
g["50"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
}
g["22"]["inputs"]["model"] = ["50", 0]
self.assertEqual(self._scanned("trigger.png", g), "unresolved")
self.assertEqual(self._texts(), [])
def test_a_seeded_text_node_holds_a_template_not_the_prompt(self):
g = flux_graph(text=["80", 0])
g["80"] = {
"class_type": "RandomPrompt",
"inputs": {"text": "a {red|blue} fox", "seed": 5},
}
self.assertEqual(self._scanned("random.png", g), "unresolved")
self.assertEqual(self._texts(), [])
def test_a1111_parameters_beside_an_undecided_graph_are_read(self):
# Metadata savers write both; the old scan read the A1111 positive.
g = flux_graph()
g["50"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
}
g["22"]["inputs"]["model"] = ["50", 0]
meta = {
"prompt": json.dumps(g),
"parameters": "a red fox\nNegative prompt: blur\nSteps: 20",
}
path = self.output_dir / "saver.png"
self.assertEqual(self.api._ingest_scanned_file(path, meta), "added")
self.assertEqual(self._texts(), ["a red fox"])
+233
View File
@@ -0,0 +1,233 @@
"""A restore cut off by a server stop waits for its worker thread (#360).
A cancelled handler (or job) task cannot stop the executor thread that
stages or publishes. Its cleanup (the stage, the upload) and the registry's
``restore`` admission must outlive that thread, and the live library ends as
the old or the restored one, never partial, with the safety backup kept.
Every positive wait is on an event. The one bounded wait (``SETTLE``) is a
negative check: the cancelled task must still be pending while its worker is
held.
"""
import asyncio
import os
import shutil
import sqlite3
import threading
from unittest.mock import patch
from database import restore_stage
from database.models import PromptModel
from py.api import admin as admin_mod
from py.api.jobs import RESTORE_JOB_KIND
try:
from tests.test_restore_plans import RestoreTestCase, _texts
except ImportError: # discovered with tests/ as the top-level directory
from test_restore_plans import RestoreTestCase, _texts
WAIT = 60 # a ceiling only a hung test reaches
SETTLE = 0.3 # the negative check's bound: the old code's cleanup takes ms
class _Gate:
"""Hold one blocking call on its worker thread until released."""
def __init__(self, real):
self.real = real
self.entered = threading.Event()
self.release = threading.Event()
self.done = threading.Event()
def method(self):
"""A function to patch onto the class (bound like the real one)."""
def held(*args):
return self(*args)
return held
def __call__(self, *args):
self.entered.set()
self.release.wait(WAIT)
try:
return self.real(*args)
finally:
self.done.set()
class ShutdownRestoreCase(RestoreTestCase):
def gate(self, real):
gate = _Gate(real)
# A failed check never leaves the worker held until WAIT.
self._gates = getattr(self, "_gates", []) + [gate]
self.addCleanup(gate.release.set)
return gate
def stages(self):
"""The stage folders (the upload's prefix starts the same way)."""
return [
name
for name in super().stages()
if os.path.isdir(os.path.join(self.tmpdir, name))
]
def uploads(self):
"""The upload files (not the -wal/-shm its readers may add)."""
return sorted(
name
for name in os.listdir(self.tmpdir)
if name.startswith(restore_stage.UPLOAD_PREFIX) and name.endswith(".tmp")
)
def fake_upload(self, texts):
"""Make ``_receive_restore_upload`` hand over a prepared upload."""
self.upload(texts, "source.db")
source = os.path.join(self.tmpdir, "uploads", "source.db")
target = os.path.join(self.tmpdir, restore_stage.UPLOAD_PREFIX + "x.tmp")
shutil.copyfile(source, target)
async def receive(request, uploads):
uploads.append(target)
return target, None
return patch.object(self.api, "_receive_restore_upload", receive)
async def wait_thread(self, event):
loop = asyncio.get_running_loop()
self.assertTrue(await loop.run_in_executor(None, event.wait, WAIT))
async def assert_still_pending(self, task):
done, _ = await asyncio.wait({task}, timeout=SETTLE)
if done:
for gate in getattr(self, "_gates", ()):
gate.release.set() # let the worker end before teardown
self.fail("the cancelled task ended while its worker still ran")
class TestLegacyRestoreAtShutdown(ShutdownRestoreCase):
async def test_cancel_during_publish_waits_for_the_worker(self):
self.save("old")
old_db = self.api.db
gate = self.gate(PromptModel.publish_staged)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "publish_staged", gate.method()),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(gate.entered)
self.assertEqual(len(self.stages()), 1)
task.cancel()
await self.assert_still_pending(task)
# The worker still has its stage and upload, and runs alone.
self.assertEqual(len(self.stages()), 1)
self.assertEqual(len(self.uploads()), 1)
self.assertIsNotNone(self.api._jobs.active(RESTORE_JOB_KIND))
gate.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertTrue(gate.done.is_set())
self.assertEqual(_texts(self.db_path), ["restored"])
self.assertIsNot(self.api.db, old_db) # the restored library adopted
self.assertEqual(len(self.backups()), 1) # the safety backup is kept
self.assertEqual(self.stages(), [])
self.assertEqual(self.uploads(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
async def test_cancel_during_staging_waits_and_changes_nothing(self):
self.save("old")
gate = self.gate(PromptModel.stage_restore)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "stage_restore", gate.method()),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(gate.entered)
task.cancel()
await self.assert_still_pending(task)
self.assertEqual(len(self.uploads()), 1) # still being read
self.assertIsNotNone(self.api._jobs.active(RESTORE_JOB_KIND))
gate.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertEqual(_texts(self.db_path), ["old"])
self.assertEqual(self.backups(), [])
self.assertEqual(self.stages(), []) # the finished stage is removed
self.assertEqual(self.uploads(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
async def test_a_publish_that_fails_after_the_cancel_still_cancels(self):
self.save("old")
def refuse(model, staged):
raise sqlite3.OperationalError("refused")
gate = self.gate(refuse)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "publish_staged", gate.method()),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(gate.entered)
task.cancel()
await self.assert_still_pending(task)
gate.release.set()
# Not answered as a 500: the cancel goes on.
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertEqual(_texts(self.db_path), ["old"])
self.assertEqual(self.stages(), [])
self.assertEqual(self.uploads(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
async def test_a_second_cancel_still_adopts_the_restored_library(self):
self.save("old")
old_db = self.api.db
publish = self.gate(PromptModel.publish_staged)
prune = self.gate(admin_mod._prune_safety_backups)
with (
self.fake_upload(["restored"]),
patch.object(PromptModel, "publish_staged", publish.method()),
patch.object(admin_mod, "_prune_safety_backups", prune),
):
task = asyncio.ensure_future(self.api.restore_database(None))
await self.wait_thread(publish.entered)
task.cancel()
await self.assert_still_pending(task)
publish.release.set()
await self.wait_thread(prune.entered) # adopting the library
task.cancel() # the loop's own shutdown cancels again
await self.assert_still_pending(task)
prune.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(task, WAIT)
self.assertEqual(_texts(self.db_path), ["restored"])
self.assertIsNot(self.api.db, old_db)
self.assertEqual(len(self.backups()), 1)
self.assertEqual(self.stages(), [])
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
class TestReviewedRestoreAtShutdown(ShutdownRestoreCase):
async def test_cancelled_job_waits_for_the_publish(self):
self.save("old")
old_db = self.api.db
body = await self.preview(self.upload(["restored"]))
gate = self.gate(PromptModel.publish_staged)
with patch.object(PromptModel, "publish_staged", gate.method()):
started = await self.apply(body["token"])
job = self.api._jobs.get(started["job"]["id"])
await self.wait_thread(gate.entered)
job.task.cancel()
await self.assert_still_pending(job.task)
self.assertEqual(len(self.stages()), 1)
self.assertTrue(job.running) # its exclusive admission holds
gate.release.set()
with self.assertRaises(asyncio.CancelledError):
await asyncio.wait_for(job.task, WAIT)
self.assertEqual(_texts(self.db_path), ["restored"])
self.assertIsNot(self.api.db, old_db)
self.assertEqual(len(self.backups()), 1)
self.assertEqual(self.stages(), [])
self.assertFalse(job.running)
+112
View File
@@ -0,0 +1,112 @@
"""``sampler_less_prompt``: the one wired text of a graph with no sampler (#252)."""
import time
import unittest
from utils.prompt_pair_resolver import sampler_less_prompt
def _guided(**nodes):
"""``nodes`` plus a guider consuming node ``"1"``."""
graph = {"9": {"class_type": "BasicGuider", "inputs": {"conditioning": ["1", 0]}}}
graph.update(nodes)
return graph
class TestSamplerLessPrompt(unittest.TestCase):
def test_not_a_graph_or_too_large_is_absent(self):
for graph in (None, [], {}, "text"):
self.assertEqual(sampler_less_prompt(graph)["state"], "absent")
big = {str(i): {"class_type": "X", "inputs": {}} for i in range(3)}
self.assertEqual(sampler_less_prompt(big, max_visits=2)["state"], "absent")
def test_one_literal_text(self):
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": " a fox "}}})
self.assertEqual(
sampler_less_prompt(graph),
{"state": "resolved", "text": "a fox", "texts": ["a fox"]},
)
def test_a_number_literal_is_never_a_prompt(self):
for value in (2048, "2048", "1.5", " 7 "):
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": value}}})
self.assertEqual(sampler_less_prompt(graph)["state"], "absent", value)
def test_a_long_digit_run_with_a_letter_is_text_and_fast(self):
text = "9" * 50_000 + "x"
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": text}}})
started = time.monotonic()
self.assertEqual(sampler_less_prompt(graph)["text"], text)
self.assertLess(time.monotonic() - started, 1.0)
def test_number_like_words_are_prompts(self):
for value in ("nan", "inf", "Infinity", "1_000"):
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": value}}})
self.assertEqual(sampler_less_prompt(graph)["text"], value)
def test_an_own_node_with_unreadable_text_beside_another_text_is_ambiguous(self):
graph = _guided(
**{
"1": {"class_type": "PromptManager", "inputs": {"text": ["5", 0]}},
"5": {"class_type": "LLM", "inputs": {"seed": 3}},
"2": {"class_type": "Enc", "inputs": {"text": "<lora:x:1>"}},
"3": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
}
)
self.assertEqual(sampler_less_prompt(graph)["state"], "ambiguous")
def test_unhashable_class_type_is_an_unknown_node_not_a_crash(self):
graph = _guided(**{"1": {"class_type": ["x"], "inputs": {"text": "a fox"}}})
self.assertEqual(sampler_less_prompt(graph)["text"], "a fox")
def test_own_negative_node_is_not_authoritative(self):
graph = _guided(
**{
"1": {"class_type": "Enc", "inputs": {"text": "a fox"}},
"2": {
"class_type": "PromptManagerNegative",
"inputs": {"text": "blur"},
},
"3": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
}
)
found = sampler_less_prompt(graph)
self.assertEqual(found["state"], "ambiguous")
self.assertEqual(found["texts"], ["a fox", "blur"])
def test_own_node_text_wins_over_an_untraceable_text(self):
graph = _guided(
**{
"1": {"class_type": "PromptManagerText", "inputs": {"text": "a fox"}},
"2": {"class_type": "Enc", "inputs": {"text": ["5", 0]}},
"5": {"class_type": "LLM", "inputs": {"seed": 3}},
"6": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
}
)
self.assertEqual(sampler_less_prompt(graph)["text"], "a fox")
def test_two_own_texts_are_ambiguous(self):
graph = _guided(
**{
"1": {"class_type": "PromptManager", "inputs": {"text": "a fox"}},
"2": {"class_type": "PromptManager", "inputs": {"text": "a cat"}},
"3": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
}
)
self.assertEqual(sampler_less_prompt(graph)["state"], "ambiguous")
def test_only_an_untraceable_text_is_ambiguous_with_nothing_to_show(self):
graph = _guided(
**{
"1": {"class_type": "Enc", "inputs": {"text": ["5", 0]}},
"5": {"class_type": "LLM", "inputs": {"seed": 3}},
}
)
self.assertEqual(
sampler_less_prompt(graph),
{"state": "ambiguous", "text": None, "texts": []},
)
if __name__ == "__main__":
unittest.main()
+21
View File
@@ -978,6 +978,27 @@ class TestThumbnailOnCapture(AutoThumbnailCase):
(self.output_dir / "thumbnails" / f"{name}_thumb.webp").is_file()
)
async def test_a_worker_refused_at_interpreter_exit_skips_quietly(self):
"""#375: refused once the interpreter exits; nothing stays queued."""
class Refused(threading.Thread):
def start(self):
raise RuntimeError("can't create new thread at interpreter shutdown")
namespace = type(sys)("threading_copy")
namespace.__dict__.update(vars(threading))
namespace.Thread = Refused
thumbnailer = self.api._auto_thumbnails()
path = make_png(self.output_dir / "late.png", size=(32, 32))
with (
patch.object(auto, "threading", namespace),
self.assertNoLogs("prompt_manager.api", "INFO"),
):
self.assertEqual(self.api.queue_capture_thumbnail(str(path)), "skipped")
self.assertFalse(thumbnailer._worker_running)
self.assertEqual(len(thumbnailer._queue), 0)
self.assertTrue(thumbnailer.wait_idle(0))
def test_queue_is_bounded(self):
thumbnailer = auto.AutoThumbnailer(
lambda *a: {"action": "generated"},
+58 -2
View File
@@ -1,12 +1,26 @@
"""The System browser hooks only ever reset their own server's config.json (#249 S3)."""
"""The System browser hooks only ever reset their own server's config.json (#249 S3)
and decide JPEG XL availability whatever the host has installed (#371)."""
import asyncio
import os
import sys
import tempfile
import types
import unittest
from pathlib import Path
from unittest import mock
from tests.browser.workspace_system_fixture import register_system_commands
from aiohttp import web
from aiohttp.test_utils import TestClient, TestServer
_mock_server = mock.MagicMock()
_mock_server.PromptServer.instance.routes = mock.MagicMock()
sys.modules.setdefault("server", _mock_server)
from py.api import thumbnail_encode # noqa: E402
from tests.browser.workspace_system_fixture import ( # noqa: E402
register_system_commands,
)
class TestWorkspaceSystemFixture(unittest.TestCase):
@@ -33,6 +47,48 @@ class TestWorkspaceSystemFixture(unittest.TestCase):
register_system_commands(None, None, None, self.root)
self.assertEqual(elsewhere.read_bytes(), b'{"real": true}')
def test_no_encoder_holds_even_when_the_host_has_one(self):
"""``{available: false}`` survives the settings read's re-probe (#371)."""
config = self.root / "config.json"
env = {**os.environ, "PROMPT_MANAGER_CONFIG_PATH": str(config)}
api = types.SimpleNamespace()
app, routes = web.Application(), web.RouteTableDef()
real_detect = thumbnail_encode._detect
with mock.patch.dict(os.environ, env, clear=True):
reset = register_system_commands(app, routes, api, self.root)
app.add_routes(routes)
self.addCleanup(thumbnail_encode.reset_encoders)
self.addCleanup(setattr, thumbnail_encode, "_detect", real_detect)
async def scenario():
async with TestClient(TestServer(app)) as client:
async def post(available):
response = await client.post(
"/__test__/system/jxl-encoder", json={"available": available}
)
self.assertEqual(response.status, 200)
await post(False)
status = thumbnail_encode.encoder_status()["jxl"]
self.assertFalse(status["available"])
self.assertIsNone(
thumbnail_encode.detect_jxl_encoder(recheck_missing=True)
)
await post(True)
status = thumbnail_encode.encoder_status()["jxl"]
self.assertEqual(status["encoder"], "cjxl")
await post(False)
# The host's real detector finds an encoder (Pillow can save JPEG XL).
with mock.patch.object(thumbnail_encode, "_pillow_can_save_jxl", lambda: True):
asyncio.run(scenario())
# Reset gives detection back to the host.
reset()
self.assertIs(thumbnail_encode._detect, real_detect)
status = thumbnail_encode.encoder_status()["jxl"]
self.assertEqual(status["encoder"], "pillow")
if __name__ == "__main__":
unittest.main()
+29
View File
@@ -247,3 +247,32 @@ test('the Python matrix and coverage run the suite through the watchdog runner',
assert.doesNotMatch(found.get(id), /unittest discover/, `${id} no longer runs the bare unittest discover`);
}
});
// CI hygiene (#352): lint tools are pinned, the bandit hook can fail, the JS job caches npm.
const readRepoFile = path => readFileSync(new URL(`../../${path}`, import.meta.url), 'utf8');
test('black and flake8 are pinned exactly and the lint job installs those pins', () => {
const requirements = readRepoFile('requirements-dev.txt');
assert.match(requirements, /^black==\d+\.\d+\.\d+$/m, 'black is pinned exactly');
assert.match(requirements, /^flake8==\d+\.\d+\.\d+$/m, 'flake8 is pinned exactly');
const lint = jobs(readWorkflow('test.yml')).get('lint');
assert.match(lint, /pip install \$\(grep -E '\^\(black\|flake8\)==' requirements-dev\.txt\)/);
assert.doesNotMatch(lint, /pip install black flake8/, 'lint no longer installs the latest releases');
});
test('the frontend modules job caches npm keyed on the vendor lockfile', () => {
const setupNode = steps(jobs(readWorkflow('test.yml')).get('js-test'))
.find(step => /uses: actions\/setup-node@v\d+/.test(step));
assert.match(setupNode, /^ {10}cache: npm$/m);
assert.match(setupNode, /^ {10}cache-dependency-path: tools\/vendor\/package-lock\.json$/m);
});
test('the bandit pre-commit hook fails at the severity CI enforces and passes every file', () => {
const quality = readWorkflow('code-quality.yml');
assert.match(quality, /^ {8}run: bandit -r \. -lll -q --exclude /m, 'CI fails on high severity');
const hook = /- id: bandit\n([\s\S]*?)(?=\n {6}(?:- id:|#)|\n*$)/.exec(readRepoFile('.pre-commit-config.yaml'))?.[1];
assert.ok(hook, 'the bandit hook exists');
assert.match(hook, /^ {8}entry: bandit -lll -q$/m, 'the hook enforces the same -lll level as CI');
assert.doesNotMatch(hook, /true|exit-zero|bash -c/, 'nothing masks the exit code or drops a file');
assert.doesNotMatch(hook, /pass_filenames: false/, 'the hook scans the staged files');
});
+98 -1
View File
@@ -1,6 +1,7 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { COLUMNS, CONTROL_EXEMPTIONS, STATUSES, declaredTitles, parseControl, parseCoverageTable, readCoverageTable,
import { COLUMNS, CONTROL_EXEMPTIONS, PAGES, REMOVAL_COLUMNS, STATUSES, declaredTitles, duplicateRemovals, idUses, methodUses,
parseControl, parseCoverageTable, parseRemovalTable, readCoverageTable, readPage, readRemovalTable, removalProblems,
rowProblems } from '../../tools/vendor/legacy-coverage.mjs';
// TESTING.md "Legacy System capability coverage" (#249 System S12, #329): every legacy System capability names an
@@ -114,3 +115,99 @@ test('a covered row names its native control unless it is exempt', () => {
assert.deepEqual([row.Status, row['Native control']], ['covered', '—']);
}
});
// TESTING.md "Legacy System UI removed by S13" (#330): a legacy element is removed only when its capability is
// `covered`, the removal copies that row's native test, and the element is really gone from its page.
const COVERAGE_HEADER = '| Capability | Legacy anchor | Owner | Status | Native control | Test |\n| --- | --- | --- | --- | --- | --- |';
const REMOVAL_HEADER = '| Capability | Page | Removed ids | Removed methods | Native test |\n| --- | --- | --- | --- | --- |';
const FIXTURE_COVERAGE = parseCoverageTable(['### Legacy System capability coverage', '', COVERAGE_HEADER,
'| View logs | `admin.html:1` | S12 Logs | covered | `diagnostics` · combobox · "Level" | level filter works (logs.spec.mjs) |',
'| Infinite scrolling default | `admin.html:2` | Legacy prompt list | kept in legacy | — | scrolls (list-state.test.js) |',
''].join('\n'));
const removalRow = cells => parseRemovalTable(['### Legacy System UI removed by S13', '', REMOVAL_HEADER, `| ${cells} |`, '']
.join('\n'))[0];
const EMPTY_PAGE = () => ({ html: '<button id="kept">Kept</button>', script: 'class Admin {\n kept() {}\n}\n' });
test('every removed legacy element cites a covered capability and its native test', () => {
const coverage = readCoverageTable();
const rows = readRemovalTable();
for (const row of rows) {
const cited = coverage.find(item => item.Capability === row.Capability);
assert.ok(cited, `TESTING.md:${row.line} cites "${row.Capability}"`);
assert.equal(cited.Status, 'covered', `TESTING.md:${row.line} ${row.Capability}`);
assert.equal(row['Native test'], cited.Test, `TESTING.md:${row.line} ${row.Capability}`);
assert.ok(Object.keys(PAGES).includes(row.Page), `TESTING.md:${row.line} page ${row.Page}`);
}
assert.deepEqual(duplicateRemovals(rows), []);
assert.deepEqual(REMOVAL_COLUMNS, ['Capability', 'Page', 'Removed ids', 'Removed methods', 'Native test']);
});
test('a removed id or method is gone from its page', () => {
const coverage = readCoverageTable();
for (const row of readRemovalTable()) {
assert.deepEqual(removalProblems(row, coverage, page => readPage(page)), [], `TESTING.md:${row.line} ${row.Capability}`);
}
// Each way a page can still hold an id or a method is found; a call through another object, or a longer name, is not
// a definition.
const page = {
html: '<div id="logsModal"></div><button onclick="window.admin.closeLogs()">x</button>',
script: [
"document.getElementById('logLevel').value;",
"this.showModal('diagnosticsModal');",
'<span data-id="logsContainer"></span>',
'document.querySelector("#logLimit option");',
"const backdrops = ['settingsModal', 'duplicatesModal'];",
'class Admin {',
' async refreshLogs(level) {',
' }',
'} displayLogs(logs) {',
' }',
'this.loadLogStats();',
'// downloadLogs() {',
'other.formatBytesLater(1);',
].join('\n'),
};
assert.deepEqual(idUses('logsModal', page), ['id="logsModal" in the markup']);
assert.deepEqual(idUses('logLevel', page), ["getElementById('logLevel')"]);
assert.deepEqual(idUses('logLimit', page), ["querySelector('#logLimit')"]);
assert.deepEqual(idUses('duplicatesModal', page), ["'duplicatesModal' in a modal list"]);
assert.deepEqual(idUses('logLevelX', page), []);
assert.deepEqual(idUses('diagnosticsModal', page), ["a …Modal('diagnosticsModal') call"]);
assert.deepEqual(idUses('logsContainer', page), []);
assert.deepEqual(methodUses('closeLogs', page), ['.closeLogs( called in the markup']);
assert.deepEqual(methodUses('refreshLogs', page), ['refreshLogs() defined']);
assert.deepEqual(methodUses('displayLogs', page), ['displayLogs() defined']);
assert.deepEqual(methodUses('loadLogStats', page), ['.loadLogStats( called']);
assert.deepEqual(methodUses('downloadLogs', page), []);
assert.deepEqual(methodUses('formatBytes', page), []);
// One id or method is listed on one row per page.
const twice = [removalRow('View logs | admin | `#logsModal` | — | level filter works (logs.spec.mjs)'),
{ ...removalRow('View logs | admin | `#logsModal` | — | level filter works (logs.spec.mjs)'), line: 9 }];
assert.match(duplicateRemovals(twice).join(), /admin #logsModal is listed on rows/);
assert.deepEqual(removalProblems(removalRow('View logs | admin | `logsModal`, `#ok` | `close` | level filter works (logs.spec.mjs)'),
FIXTURE_COVERAGE, EMPTY_PAGE), ['malformed entry "logsModal"', 'malformed entry "close"']);
});
test('a removal citing a kept-in-legacy capability fails', () => {
const row = removalRow('Infinite scrolling default | admin | `#infiniteScrollDefault` | — | scrolls (list-state.test.js)');
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, EMPTY_PAGE), ['cites a kept in legacy capability']);
const unknown = removalRow('Something else | admin | — | — | scrolls (list-state.test.js)');
assert.deepEqual(removalProblems(unknown, FIXTURE_COVERAGE, EMPTY_PAGE), ['cites no single coverage row "Something else"']);
});
test('a removal whose native test differs fails', () => {
const row = removalRow('View logs | admin | `#logsModal` | `showLogsModal()` | another title (logs.spec.mjs)');
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, EMPTY_PAGE),
['its native test differs from the coverage row\'s "level filter works (logs.spec.mjs)"']);
assert.deepEqual(removalProblems({ ...row, 'Native test': 'level filter works (logs.spec.mjs)' }, FIXTURE_COVERAGE, EMPTY_PAGE), []);
});
test('a removed id still on its page fails', () => {
const row = removalRow('View logs | admin | `#logsModal` | `showLogsModal()` | level filter works (logs.spec.mjs)');
const page = () => ({ html: '<div id="logsModal" class="hidden"></div>', script: 'class Admin {\n showLogsModal() {\n }\n}\n' });
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, page), [
'#logsModal is still on the admin page: id="logsModal" in the markup',
'showLogsModal() is still on the admin page: showLogsModal() defined',
]);
assert.deepEqual(removalProblems({ ...row, Page: 'metadata' }, FIXTURE_COVERAGE, page), ['unknown page "metadata"']);
});
+140 -8
View File
@@ -42,25 +42,30 @@ function cells(line) {
return out;
}
/** The table's rows as objects keyed by COLUMNS, from the section that starts at HEADING. */
export function parseCoverageTable(markdown) {
/** The rows, keyed by `columns`, of the first Markdown table under `heading` (`label` names it in errors). */
function parseTable(markdown, heading, columns, label) {
const lines = markdown.split('\n');
const start = lines.findIndex(line => line.trim() === HEADING);
if (start === -1) throw new Error(`TESTING.md has no "${HEADING}" section`);
const start = lines.findIndex(line => line.trim() === heading);
if (start === -1) throw new Error(`TESTING.md has no "${heading}" section`);
const header = lines.findIndex((line, index) => index > start && line.trim().startsWith('|'));
if (header === -1 || lines.slice(start + 1, header).some(line => /^#{1,3} /.test(line))) {
throw new Error('The coverage section has no table');
throw new Error(`The ${label} section has no table`);
}
const columns = cells(lines[header]);
if (columns.join('|') !== COLUMNS.join('|')) throw new Error(`The coverage table's columns are ${columns.join(', ')}`);
const found = cells(lines[header]);
if (found.join('|') !== columns.join('|')) throw new Error(`The ${label} table's columns are ${found.join(', ')}`);
const rows = [];
for (let index = header + 2; index < lines.length && lines[index].trim().startsWith('|'); index += 1) {
const values = cells(lines[index]);
rows.push({ line: index + 1, ...Object.fromEntries(COLUMNS.map((column, at) => [column, values[at] ?? ''])) });
rows.push({ line: index + 1, ...Object.fromEntries(columns.map((column, at) => [column, values[at] ?? ''])) });
}
return rows;
}
/** The table's rows as objects keyed by COLUMNS, from the section that starts at HEADING. */
export function parseCoverageTable(markdown) {
return parseTable(markdown, HEADING, COLUMNS, 'coverage');
}
/**
* A Native control cell: `` `<section>` · <role> · "<name>" ``, optionally ` · in "<row title>"` when the name is shared
* by several rows of the section → {section, role, name, row}; `—` → null; else it throws.
@@ -213,3 +218,130 @@ export function rowProblems(row, { root = ROOT, locate = file => testFile(file,
export function readCoverageTable(root = ROOT) {
return parseCoverageTable(readFileSync(join(root, 'TESTING.md'), 'utf8'));
}
// ── Legacy System UI removed by S13 (#330) ──────────────────────────────────────────────────────────────────────────
// TESTING.md "Legacy System UI removed by S13": one row per covered capability whose legacy UI S13 removed, with the
// element ids and methods that went. A row must cite a `covered` coverage row and copy its Test cell, and none of its
// ids or methods may still be on its page.
export const REMOVAL_HEADING = '### Legacy System UI removed by S13';
export const REMOVAL_COLUMNS = Object.freeze(['Capability', 'Page', 'Removed ids', 'Removed methods', 'Native test']);
/** Each legacy page's markup and script. */
export const PAGES = Object.freeze({
admin: Object.freeze({ html: 'web/admin.html', script: 'web/js/admin.js' }),
gallery: Object.freeze({ html: 'web/gallery.html', script: 'web/js/gallery.js' }),
});
/** The removal table's rows, from the section that starts at REMOVAL_HEADING. */
export function parseRemovalTable(markdown) {
return parseTable(markdown, REMOVAL_HEADING, REMOVAL_COLUMNS, 'removal');
}
/** The removal table read from TESTING.md. */
export function readRemovalTable(root = ROOT) {
return parseRemovalTable(readFileSync(join(root, 'TESTING.md'), 'utf8'));
}
/** A Removed ids or Removed methods cell → its entries without backticks (`—` → none). */
export function listCell(cell) {
if (cell.trim() === NONE) return [];
return cell.split(',').map(entry => entry.trim().replace(/^`|`$/g, '')).filter(Boolean);
}
/** The removed ids of a row (without `#`) and its malformed entries. */
function removedIds(row) {
const entries = listCell(row['Removed ids']);
return { ids: entries.filter(entry => /^#[\w-]+$/.test(entry)).map(entry => entry.slice(1)),
bad: entries.filter(entry => !/^#[\w-]+$/.test(entry)) };
}
/** The removed methods of a row (without `()`) and its malformed entries. */
function removedMethods(row) {
const entries = listCell(row['Removed methods']);
return { names: entries.filter(entry => /^[A-Za-z_$][\w$]*\(\)$/.test(entry)).map(entry => entry.slice(0, -2)),
bad: entries.filter(entry => !/^[A-Za-z_$][\w$]*\(\)$/.test(entry)) };
}
const escapeRegExp = text => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
/**
* Where `id` is still on a page: `id="<id>"` in its markup or script; in its script a lookup, a `…Modal("<id>")` call
* (`showModal`, `hideModal`) or a modal-list entry.
*/
export function idUses(id, { html, script }) {
const name = escapeRegExp(id);
const code = stripJsComments(script);
const uses = [];
// `id=`, not `data-id=`: the attribute starts after a space or a tag's `<name`.
const attribute = new RegExp(`(?:^|\\s)id=["']${name}["']`, 'm');
if (attribute.test(html)) uses.push(`id="${id}" in the markup`);
if (attribute.test(code)) uses.push(`id="${id}" in the script`);
if (new RegExp(`getElementById\\(\\s*["'\`]${name}["'\`]\\s*\\)`).test(code)) uses.push(`getElementById('${id}')`);
if (new RegExp(`querySelector(?:All)?\\(\\s*["'\`]#${name}\\b`).test(code)) uses.push(`querySelector('#${id}')`);
if (new RegExp(`\\w*Modal\\(\\s*["'\`]${name}["'\`]`).test(code)) uses.push(`a …Modal('${id}') call`);
// A string list of modal ids (the gallery's backdrop list): the entry next to another `…Modal` entry.
for (const list of code.matchAll(/\[[^\[\]]*\]/g)) {
const entry = new RegExp(`["']${name}["']`);
if (entry.test(list[0]) && /["'][\w-]*Modal["']/.test(list[0].replace(entry, ''))) {
uses.push(`'${id}' in a modal list`);
break;
}
}
return uses;
}
/**
* Where method `name` is still on a page: a definition in its script, or a `.<name>(` call (`this.`, `window.admin.`)
* in its script or markup.
*/
export function methodUses(name, { html, script }) {
const method = escapeRegExp(name);
const code = stripJsComments(script);
const uses = [];
if (new RegExp(`(?:^|[\\s;}])(?:async\\s+)?${method}\\s*\\([^)]*\\)\\s*\\{`, 'm').test(code)) uses.push(`${name}() defined`);
if (new RegExp(`\\.${method}\\s*\\(`).test(code)) uses.push(`.${name}( called`);
if (new RegExp(`\\.${method}\\s*\\(`).test(html)) uses.push(`.${name}( called in the markup`);
return uses;
}
/**
* Every problem of one removal row (empty: sound). `coverage` is the coverage table's rows; `readPage(page)` returns
* that page's `{html, script}`.
*/
export function removalProblems(row, coverage, readPage) {
const problems = [];
const cited = coverage.filter(item => item.Capability === row.Capability);
if (cited.length !== 1) problems.push(`cites no single coverage row "${row.Capability}"`);
else if (cited[0].Status !== 'covered') problems.push(`cites a ${cited[0].Status} capability`);
else if (cited[0].Test !== row['Native test']) problems.push(`its native test differs from the coverage row's "${cited[0].Test}"`);
if (!Object.hasOwn(PAGES, row.Page)) {
problems.push(`unknown page "${row.Page}"`);
return problems;
}
const { ids, bad: badIds } = removedIds(row);
const { names, bad: badNames } = removedMethods(row);
for (const entry of [...badIds, ...badNames]) problems.push(`malformed entry "${entry}"`);
const page = readPage(row.Page);
for (const id of ids) for (const use of idUses(id, page)) problems.push(`#${id} is still on the ${row.Page} page: ${use}`);
for (const name of names) for (const use of methodUses(name, page)) problems.push(`${name}() is still on the ${row.Page} page: ${use}`);
return problems;
}
/** Ids and methods listed on more than one row of the same page. */
export function duplicateRemovals(rows) {
const seen = new Map();
const duplicates = [];
for (const row of rows) {
for (const entry of [...listCell(row['Removed ids']), ...listCell(row['Removed methods'])]) {
const key = `${row.Page} ${entry}`;
if (seen.has(key)) duplicates.push(`${key} is listed on rows ${seen.get(key)} and ${row.line}`);
else seen.set(key, row.line);
}
}
return duplicates;
}
/** A page's markup and script from the repository. */
export function readPage(page, root = ROOT) {
const files = PAGES[page];
return { html: readFileSync(join(root, files.html), 'utf8'), script: readFileSync(join(root, files.script), 'utf8') };
}
+13 -3
View File
@@ -35,6 +35,7 @@ from watchdog.events import FileSystemEventHandler
from .hashing import generate_prompt_hash
from .metadata_extractor import ComfyUIMetadataExtractor
from .parallel import note_refusal, refused_at_shutdown
from .logging_config import get_logger
from .prompt_graph import resolve_text, run_prompt_nodes
from .prompt_pair_resolver import resolve_prompt_pair
@@ -194,8 +195,9 @@ class ImageGenerationHandler(FileSystemEventHandler):
and once its size has stopped changing.
Returns:
False when the path is already pending, the pending set is full or
the handler is closed
False when the path is already pending, the pending set is full,
the handler is closed or the interpreter is exiting (its worker
cannot start)
"""
with self._pending_lock:
if self._closed:
@@ -211,7 +213,15 @@ class ImageGenerationHandler(FileSystemEventHandler):
return False
due = time.monotonic() + self.processing_delay
self._pending[image_path] = (prompt_snapshot, due)
self._ensure_worker()
try:
self._ensure_worker()
except RuntimeError as error:
# No worker will ever take it: never leave it pending (#375).
self._pending.pop(image_path, None)
if not refused_at_shutdown(error):
raise
note_refusal(self.logger, "The image worker")
return False
self._work.put(image_path)
return True
+67 -2
View File
@@ -5,17 +5,51 @@ a CUDA-initialised, multi-threaded process is unsafe and the spawn start
method would re-import ComfyUI's main module in every child. Pillow releases
the GIL while decoding, resizing and encoding, so threads still scale close
to linearly for both jobs.
Interpreter exit (#375): ``threading._shutdown`` shuts every thread pool
(``submit`` then raises "cannot schedule new futures after interpreter
shutdown") before it joins the non-daemon threads and long before the package
teardown runs, while our daemon threads keep working. Measured on 3.12 and
3.14, ``Thread.start`` still succeeds in that window; it is refused ("can't
create new thread at interpreter shutdown") only once the interpreter is
marked finalizing, after the ``atexit`` callbacks. ``refused_at_shutdown``
recognises both; ``note_refusal`` logs one DEBUG line per place, with no
traceback.
"""
import logging
import threading
from concurrent.futures import ThreadPoolExecutor
SHUTDOWN_REFUSAL = "interpreter shutdown"
_logger = logging.getLogger("prompt_manager.parallel")
_noted = set()
_noted_lock = threading.Lock()
def refused_at_shutdown(error):
"""Whether ``error`` is a thread or pool start refused at interpreter exit."""
return isinstance(error, RuntimeError) and SHUTDOWN_REFUSAL in str(error)
def note_refusal(logger, what):
"""Log once per ``what``, at DEBUG and without a traceback, that ``what``
was refused because the interpreter is exiting."""
key = (logger.name, what)
with _noted_lock:
if key in _noted:
return
_noted.add(key)
logger.debug(f"{what}: refused, the interpreter is exiting")
def map_parallel(func, items, workers):
"""Apply ``func`` to every item on up to ``workers`` threads, keeping order.
``workers`` of 1 (or fewer) runs on the calling thread with no pool. An
exception in ``func`` propagates to the caller, so callers that must
survive bad input wrap ``func`` themselves.
survive bad input wrap ``func`` themselves. A pool refused at interpreter
exit runs the items it did not start on the calling thread (#375).
"""
items = list(items)
workers = max(1, int(workers or 1))
@@ -23,4 +57,35 @@ def map_parallel(func, items, workers):
return [func(item) for item in items]
max_workers = min(workers, len(items))
with ThreadPoolExecutor(max_workers, thread_name_prefix="pm-worker") as pool:
return list(pool.map(func, items))
futures = []
try:
for item in items:
futures.append(pool.submit(func, item))
except RuntimeError as error:
if not refused_at_shutdown(error):
raise
note_refusal(_logger, "A worker pool")
# Drop what was queued but not started; it runs here instead.
# (A refused Thread.start leaves its item queued without a future
# here: it is cancelled too, unless a busy pool thread took it in
# between, the one case where an item can run twice.)
pool.shutdown(wait=True, cancel_futures=True)
else:
return [future.result() for future in futures]
return _run_rest(func, items, futures)
def _run_rest(func, items, futures):
"""The pool's finished results, and every item it did not run, run here.
After ``shutdown(cancel_futures=True)`` each submitted future either ran
(its result or exception is kept) or was cancelled before it started.
"""
results = []
for index, item in enumerate(items):
future = futures[index] if index < len(futures) else None
if future is not None and not future.cancelled():
results.append(future.result())
else:
results.append(func(item))
return results
+112
View File
@@ -2578,3 +2578,115 @@ def resolve_text_input(
return resolver.text(inputs.get(input_name), "string")[0]
except _Unresolved:
return None
_NUMBER = re.compile(r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?")
def _numeric(text: str) -> bool:
"""Whether ``text`` is a plain number (a widget value such as ``2048``)."""
return _NUMBER.fullmatch(text) is not None
def sampler_less_prompt(graph: Any, *, max_visits: int = 500) -> Dict[str, Any]:
"""The positive prompt of an API graph with no sampler pair, if one (#252).
For graphs ``_no_sampler`` finds ``absent`` (no node takes a ``positive``
or ``negative`` input, such as a Flux guider or a video wrapper). Only the
executed API graph is read, never the UI workflow. A candidate is a node
with a ``text`` input whose output another node consumes: a text that
feeds nothing did not make the image. Its text is traced as a STRING
(``resolve_text_input``); a linked text no rule reads is unknown unless
it is built from a candidate's text (``_covered``). Blank and
numeric texts are never prompts, and a node with a seed input holds a
template, not the prompt (unknown). This repository's own positive-text
nodes are authoritative: when they give a text, only they count.
Returns ``{"state", "text", "texts"}``: ``resolved`` with the one distinct
text; ``ambiguous`` when several texts or an unknown one could be the
prompt (``texts`` lists the readable ones); ``absent`` when none.
"""
absent = {"state": "absent", "text": None, "texts": []}
if not isinstance(graph, dict) or not graph or len(graph) > max_visits:
return absent
nodes = {str(key): value for key, value in graph.items()}
consumed = {
str(value[0])
for node in nodes.values()
if isinstance(node, dict) and isinstance(node.get("inputs"), dict)
for value in node["inputs"].values()
if _link(value)
}
own: list = []
other: list = []
unknown = False
for node_id, node in nodes.items():
inputs = node.get("inputs") if isinstance(node, dict) else None
if node_id not in consumed or not _has_text_input(node):
continue
if _seeded(inputs):
# Generated at run time (a wildcard or a language model): its
# text input is a template, not the prompt that ran.
unknown = True
continue
value = inputs["text"]
text = resolve_text_input(nodes, node_id, "text", max_visits=max_visits)
if text is None:
# A literal that is not a string (a number) is never a prompt.
if _link(value) and not _covered(nodes, value):
unknown = True
continue
text = text.strip()
if not text or _numeric(text):
continue
kind = node.get("class_type")
if isinstance(kind, str) and kind in _OWN_CLASSES and "Negative" not in kind:
own.append(text)
else:
other.append(text)
texts = list(dict.fromkeys(own or other))
if len(texts) == 1 and (own or not unknown):
return {"state": "resolved", "text": texts[0], "texts": texts}
if texts or unknown:
return {"state": "ambiguous", "text": None, "texts": texts}
return absent
def _has_text_input(node: Any) -> bool:
inputs = node.get("inputs") if isinstance(node, dict) else None
return isinstance(inputs, dict) and "text" in inputs
def _seeded(inputs: dict) -> bool:
"""A seed input: the node's string output is a run-time value (the
resolver's ``runtime-value`` rule)."""
return any(isinstance(name, str) and name.endswith("seed") for name in inputs)
def _covered(nodes: dict, link: list) -> bool:
"""Whether an unreadable text link is built from a candidate's text.
True when a node upstream of ``link`` is read through its first output
(slot 0, the legacy encoder rule's text) and has an unseeded ``text``
input: that candidate is read on its own, and whatever else is joined to
it at run time (a caption, a language model) is no stored prompt to
choose. Another output of such a node (a loader's trigger words) is not
its text. False when the text comes only from inputs no candidate holds
(a builder's ``prompt`` field): the prompt is unknown then.
"""
seen, stack = set(), [(str(link[0]), link[1])]
while stack:
node_id, slot = stack.pop()
if (node_id, slot) in seen:
continue
seen.add((node_id, slot))
node = nodes.get(node_id)
inputs = node.get("inputs") if isinstance(node, dict) else None
if not isinstance(inputs, dict):
continue
if "text" in inputs and slot == 0 and not _seeded(inputs):
return True
stack.extend(
(str(value[0]), value[1]) for value in inputs.values() if _link(value)
)
return False
+12 -165
View File
@@ -168,10 +168,11 @@
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Settings
</button>
<button id="diagnosticsBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
<a id="diagnosticsBtn" href="workspace#/system/diagnostics" target="_blank" rel="noopener"
title="Opens System › Diagnostics &amp; logs in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Diagnostics
</button>
</a>
<button id="maintenanceBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Maintenance
@@ -192,10 +193,11 @@
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Auto Tag
</button>
<button id="logsBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
<a id="logsBtn" href="workspace#/system/diagnostics" target="_blank" rel="noopener"
title="Opens System › Diagnostics &amp; logs in a new tab"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
View Logs
</button>
</a>
<button id="metadataBtn"
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Metadata
@@ -641,37 +643,6 @@
</div>
</div>
<!-- Diagnostics Modal -->
<div id="diagnosticsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
<div class="p-4 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">Gallery System Diagnostics</h3>
<button onclick="window.admin.closeDiagnostics()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
</svg>
</button>
</div>
</div>
<div class="p-4 overflow-y-auto max-h-[calc(90vh-120px)]">
<div class="flex items-center justify-between mb-3">
<button id="runDiagnosticsBtn" class="px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Run Diagnostics
</button>
<button id="testImageLinkBtn" class="px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Test Image Link
</button>
</div>
<div id="diagnosticsContent" class="space-y-3">
<div class="text-center py-6 text-pm-secondary text-xs">
Click "Run Diagnostics" to check the gallery system status
</div>
</div>
</div>
</div>
</div>
<!-- Maintenance Modal -->
<div id="maintenanceModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
@@ -864,130 +835,6 @@
<span class="inline-block w-1.5 h-1.5 rounded-full bg-pm-accent mr-1.5 align-middle animate-pulse"></span><span id="scanPillText" class="align-middle">Scanning...</span>
</button>
<!-- Logs Modal -->
<div id="logsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg max-w-6xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
<div class="p-4 border-b border-pm">
<div class="flex items-center justify-between">
<h3 class="text-sm font-semibold text-pm">System Logs</h3>
<button onclick="window.admin.closeLogs()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
</svg>
</button>
</div>
</div>
<!-- Log Controls -->
<div class="p-4 border-b border-pm">
<div class="grid grid-cols-1 md:grid-cols-5 gap-3 mb-3">
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Log Level</label>
<select id="logLevel" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="">All Levels</option>
<option value="DEBUG">Debug</option>
<option value="INFO">Info</option>
<option value="WARNING">Warning</option>
<option value="ERROR">Error</option>
<option value="CRITICAL">Critical</option>
</select>
</div>
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Limit</label>
<select id="logLimit" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="50">Last 50</option>
<option value="100" selected>Last 100</option>
<option value="200">Last 200</option>
<option value="500">Last 500</option>
</select>
</div>
<div class="flex items-end">
<button id="refreshLogsBtn" class="w-full px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Refresh
</button>
</div>
<div class="flex items-end">
<button id="downloadLogsBtn" class="w-full px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
Download
</button>
</div>
<div class="flex items-end">
<button id="clearLogsBtn" class="w-full px-2.5 py-1.5 text-pm-error bg-transparent hover:bg-pm-error-tint border border-pm text-xs font-medium rounded-pm-sm transition-colors">
Clear Logs
</button>
</div>
</div>
<!-- Log Stats -->
<div class="grid grid-cols-2 md:grid-cols-5 gap-3 mb-3">
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Total Logs</div>
<div class="text-sm font-semibold text-pm-accent" id="logStatsTotal">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Errors</div>
<div class="text-sm font-semibold text-pm-error" id="logStatsErrors">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Warnings</div>
<div class="text-sm font-semibold text-pm-warning" id="logStatsWarnings">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Current Level</div>
<div class="text-sm font-semibold text-pm-success" id="logStatsLevel">-</div>
</div>
<div class="bg-pm-input rounded-pm-md p-2">
<div class="text-[11px] text-pm-secondary">Log Size</div>
<div class="text-sm font-semibold text-pm" id="logStatsSize">-</div>
</div>
</div>
<!-- Log Settings -->
<div class="grid grid-cols-1 md:grid-cols-3 gap-3">
<div>
<label class="block text-xs font-medium text-pm-secondary mb-1">Set Log Level</label>
<select id="setLogLevel" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
<option value="DEBUG">Debug</option>
<option value="INFO" selected>Info</option>
<option value="WARNING">Warning</option>
<option value="ERROR">Error</option>
<option value="CRITICAL">Critical</option>
</select>
</div>
<div class="flex items-center">
<label class="flex items-center gap-1.5 text-pm">
<input type="checkbox" id="consoleLogging" checked class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
<span class="text-xs">Console Logging</span>
</label>
</div>
<div class="flex items-end">
<button id="updateLogConfigBtn" class="w-full px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
Update Config
</button>
</div>
</div>
</div>
<!-- Log Viewer -->
<div class="flex-1 overflow-hidden">
<div class="h-96 overflow-y-auto bg-pm-primary font-mono text-xs">
<div id="logsContainer" class="p-3 space-y-0.5">
<div class="text-center py-6 text-pm-muted text-xs">
Click "Refresh" to load logs
</div>
</div>
</div>
</div>
<!-- Log Files -->
<div class="p-4 border-t border-pm">
<h4 class="text-xs font-semibold text-pm-secondary uppercase tracking-wide mb-3">Log Files</h4>
<div id="logFilesList" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-2">
</div>
</div>
</div>
</div>
<!-- Add Prompt Modal -->
<div id="addPromptModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-2xl w-full mx-4 border border-pm max-h-[90vh] overflow-y-auto shadow-pm">
@@ -1187,14 +1034,13 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
<div class="space-y-1.5">
<label class="flex items-center p-2.5 bg-pm-input rounded-pm-md cursor-pointer hover:bg-pm-hover border border-pm">
<input type="radio" name="autoTagMode" value="skip" checked class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
<span class="ml-2.5 text-pm text-xs">Skip images with existing tags</span>
<span class="ml-2.5 text-pm text-xs">Skip images already auto-tagged</span>
</label>
<label class="flex items-center p-2.5 bg-pm-input rounded-pm-md cursor-pointer hover:bg-pm-hover border border-pm">
<input type="radio" name="autoTagMode" value="retag" class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
<span class="ml-2.5 text-pm text-xs">Re-tag all images</span>
</label>
</div>
<p class="text-pm-muted text-[11px] mt-1">Note: "auto-scanned" is not counted as a real tag</p>
</div>
<!-- Action Buttons -->
@@ -1293,6 +1139,7 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
<div class="w-full bg-pm-input rounded-full h-1.5">
<div id="autoTagProgressBar" class="bg-pm-accent h-1.5 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<p id="autoTagCloseNote" class="text-pm-muted text-[11px] mt-1.5">Closing this window stops tagging. Tags already added are kept. Prompt Manager's Activity page shows the time left.</p>
</div>
<div class="grid grid-cols-3 gap-3 mb-4">
@@ -1332,7 +1179,7 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
<!-- Re-tag Confirmation Modal -->
<div id="retagConfirmModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-lg w-full mx-4 border border-pm shadow-pm">
<h3 class="text-sm font-semibold text-pm mb-3">Image Already Tagged</h3>
<h3 class="text-sm font-semibold text-pm mb-3">Image Already Auto-tagged</h3>
<div class="mb-3">
<img id="retagPreviewImage" src="" alt="Preview" class="w-full h-40 object-contain bg-pm-primary rounded-pm-md">
@@ -1345,7 +1192,7 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
</div>
</div>
<p class="text-pm-secondary text-xs mb-4">This image already has tags. Would you like to generate new tags?</p>
<p class="text-pm-secondary text-xs mb-4">A tagging model already tagged this image. Generate new tags?</p>
<div class="flex gap-2 justify-end">
<button id="retagSkipBtn"
+1
View File
@@ -682,6 +682,7 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
<div class="w-full bg-pm-input rounded-full h-2">
<div id="autoTagProgressBar" class="bg-pm-accent h-2 rounded-full transition-all duration-300" style="width: 0%"></div>
</div>
<p id="autoTagCloseNote" class="text-pm-muted text-xs mt-1.5">Closing this window stops tagging. Tags already added are kept. Prompt Manager's Activity page shows the time left.</p>
</div>
<!-- Stats -->
+52 -508
View File
@@ -47,7 +47,6 @@
edit: ({ promptId }) => this.editPrompt(promptId),
delete: ({ promptId }) => this.deletePrompt(promptId),
"open-film": ({ promptId, index }) => this.openFilmStripViewer(promptId, index),
"download-log": ({ filename }) => this.downloadLogFile(filename),
"remove-review-tag": ({ index }) => this.removeReviewTag(index),
"pick-tag-suggestion": ({ tag }) => this.pickTagSuggestion(tag),
};
@@ -128,14 +127,12 @@
document.getElementById("exportBtn").addEventListener("click", () => this.exportPrompts());
document.getElementById("settingsBtn").addEventListener("click", () => this.showSettingsModal());
document.getElementById("statsBtn").addEventListener("click", () => this.showStats());
document.getElementById("diagnosticsBtn").addEventListener("click", () => this.showDiagnosticsModal());
document.getElementById("maintenanceBtn").addEventListener("click", () => this.showMaintenanceModal());
document.getElementById("backupBtn").addEventListener("click", () => this.backupDatabase());
document.getElementById("restoreBtn").addEventListener("click", () => this.showRestoreModal());
document.getElementById("scanBtn").addEventListener("click", () => this.showScanModal());
document.getElementById("autoTagBtn").addEventListener("click", () => this.showAutoTagModal());
document.getElementById("addPromptBtn").addEventListener("click", () => this.showAddPromptModal());
document.getElementById("logsBtn").addEventListener("click", () => this.showLogsModal());
document.getElementById("metadataBtn").addEventListener("click", () => this.openMetadataViewer());
document.getElementById("galleryBtn").addEventListener("click", () => this.openGallery());
@@ -215,10 +212,6 @@
document.getElementById("confirmRestore").addEventListener("click", () => this.confirmRestore());
document.getElementById("cancelRestore").addEventListener("click", () => this.hideModal("restoreModal"));
// Diagnostics modal
document.getElementById("runDiagnosticsBtn").addEventListener("click", () => this.runDiagnostics());
document.getElementById("testImageLinkBtn").addEventListener("click", () => this.testImageLink());
// Maintenance modal
document.getElementById("runMaintenanceBtn").addEventListener("click", () => this.runMaintenance());
document.getElementById("selectAllMaintenanceBtn").addEventListener("click", () => this.selectAllMaintenance());
@@ -230,12 +223,6 @@
document.getElementById("scanPill").addEventListener("click", () => this.showScanModal());
document.getElementById("quickBackupBtn").addEventListener("click", () => this.quickBackup());
// Logs modal
document.getElementById("refreshLogsBtn").addEventListener("click", () => this.refreshLogs());
document.getElementById("downloadLogsBtn").addEventListener("click", () => this.downloadLogs());
document.getElementById("clearLogsBtn").addEventListener("click", () => this.clearLogs());
document.getElementById("updateLogConfigBtn").addEventListener("click", () => this.updateLogConfig());
// Auto Tag modals
document.getElementById("startAutoTagBtn").addEventListener("click", () => this.startAutoTag());
document.getElementById("startReviewBtn").addEventListener("click", () => this.startReview());
@@ -1801,15 +1788,6 @@
return `${size.toFixed(1)} ${units[unitIndex]}`;
}
// Diagnostics functionality
showDiagnosticsModal() {
this.showModal("diagnosticsModal");
}
closeDiagnostics() {
this.hideModal("diagnosticsModal");
}
// Maintenance functionality
showMaintenanceModal() {
this.showModal("maintenanceModal");
@@ -1981,160 +1959,6 @@
resultsContainer.innerHTML = html;
}
async runDiagnostics() {
const content = document.getElementById("diagnosticsContent");
content.innerHTML = `
<div class="text-center py-4">
<div class="w-12 h-12 border-4 border-orange-500/30 border-t-orange-500 rounded-full animate-spin mx-auto mb-4"></div>
<p class="text-pm-secondary">Running diagnostics...</p>
</div>
`;
try {
const response = await ApiClient.fetchUrl('/prompt_manager/diagnostics');
if (response.ok) {
const data = await response.json();
if (data.success) {
this.renderDiagnostics(data.diagnostics);
} else {
throw new Error(data.error || 'Diagnostics failed');
}
} else {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
} catch (error) {
console.error('Diagnostics error:', error);
content.innerHTML = `
<div class="bg-pm-error-tint border border-pm-error rounded-pm-sm p-4">
<h4 class="text-pm-error font-medium mb-2">❌ Diagnostics Failed</h4>
<p class="text-pm-secondary">${escapeHtml(error.message)}</p>
</div>
`;
}
}
renderDiagnostics(diagnostics) {
const content = document.getElementById("diagnosticsContent");
let html = '<div class="space-y-4">';
// Summary
html += '<div class="bg-pm-surface rounded-pm-sm p-4">';
html += '<h4 class="text-pm font-medium mb-3">📋 Diagnostic Summary</h4>';
html += '<div class="grid grid-cols-1 md:grid-cols-2 gap-3">';
for (const [category, result] of Object.entries(diagnostics)) {
const status = result.status === 'ok' ? '✅ PASS' : (result.status === 'warning' ? '⚠️ WARNING' : '❌ FAIL');
const statusColor = result.status === 'ok' ? 'text-pm-success' : (result.status === 'warning' ? 'text-pm-warning' : 'text-pm-error');
html += `
<div class="flex justify-between items-center p-2 bg-pm-input rounded">
<span class="text-pm-secondary capitalize">${escapeHtml(category)}</span>
<span class="${statusColor} font-mono text-sm">${status}</span>
</div>
`;
}
html += '</div></div>';
// Detailed results
for (const [category, result] of Object.entries(diagnostics)) {
const bgColor = result.status === 'ok' ? 'bg-pm-success-tint border-pm-success' :
(result.status === 'warning' ? 'bg-pm-warning/20 border-pm-warning' : 'bg-pm-error-tint border-pm-error');
html += `<div class="${bgColor} border rounded-pm-sm p-4">`;
html += `<h4 class="text-pm font-medium mb-2 capitalize">${escapeHtml(category)}</h4>`;
if (result.message) {
html += `<p class="text-pm-secondary mb-2">${escapeHtml(result.message)}</p>`;
}
// Show specific details based on category
if (category === 'database' && result.status === 'ok') {
html += `<div class="text-sm text-pm-secondary">`;
html += `<p>Prompts: ${escapeHtml(result.prompt_count || 0)}</p>`;
html += `<p>Images table: ${result.has_images_table ? 'Yes' : 'No'}</p>`;
html += `</div>`;
}
if (category === 'images_table' && result.status === 'ok') {
html += `<div class="text-sm text-pm-secondary">`;
html += `<p>Images: ${escapeHtml(result.image_count || 0)}</p>`;
if (result.recent_images && result.recent_images.length > 0) {
html += `<p>Recent images:</p>`;
html += `<ul class="ml-4 list-disc">`;
result.recent_images.slice(0, 3).forEach(img => {
html += `<li>${escapeHtml(img.filename)} → Prompt ${escapeHtml(img.prompt_id)}</li>`;
});
html += `</ul>`;
}
html += `</div>`;
}
if (category === 'comfyui_output' && result.output_dirs) {
html += `<div class="text-sm text-pm-secondary">`;
html += `<p>Output directories found:</p>`;
html += `<ul class="ml-4 list-disc">`;
result.output_dirs.forEach(dir => {
html += `<li>${escapeHtml(dir)}</li>`;
});
html += `</ul>`;
html += `</div>`;
}
if (category === 'dependencies' && result.dependencies) {
html += `<div class="text-sm text-pm-secondary">`;
for (const [dep, available] of Object.entries(result.dependencies)) {
const status = available ? '✅' : '❌';
html += `<p>${status} ${escapeHtml(dep)}</p>`;
}
html += `</div>`;
}
html += '</div>';
}
html += '</div>';
content.innerHTML = html;
}
async testImageLink() {
if (this.prompts.length === 0) {
this.showNotification('No prompts available for testing', 'warning');
return;
}
// Use the first prompt for testing
const testPrompt = this.prompts[0];
try {
const response = await ApiClient.fetchUrl('/prompt_manager/diagnostics/test-link', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
prompt_id: testPrompt.id,
image_path: '/test/fake/image.png'
})
});
if (response.ok) {
const data = await response.json();
if (data.success) {
this.showNotification('✅ Test image link created successfully!', 'success');
// Refresh the gallery to show the test image
setTimeout(() => this.refreshList(), 1000);
} else {
throw new Error(data.result?.message || 'Test failed');
}
} else {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
} catch (error) {
console.error('Test link error:', error);
this.showNotification(`❌ Test failed: ${error.message}`, 'error');
}
}
// Database backup and restore functionality
async backupDatabase() {
try {
@@ -2497,273 +2321,6 @@
this.showNotification(`Scan failed: ${state.message}`, "error");
}
// Logs functionality
showLogsModal() {
this.showModal("logsModal");
this.loadLogStats();
this.loadLogFiles();
this.loadLogConfig();
}
closeLogs() {
this.hideModal("logsModal");
}
async loadLogStats() {
try {
const response = await ApiClient.fetchUrl('/prompt_manager/logs/stats');
const data = await response.json();
if (data.success) {
const stats = data.stats;
document.getElementById("logStatsTotal").textContent = stats.buffer_count || 0;
document.getElementById("logStatsErrors").textContent = stats.level_counts?.ERROR || 0;
document.getElementById("logStatsWarnings").textContent = stats.level_counts?.WARNING || 0;
document.getElementById("logStatsLevel").textContent = stats.current_level || "INFO";
document.getElementById("logStatsSize").textContent = this.formatBytes(stats.total_log_size || 0);
}
} catch (error) {
console.error("Failed to load log stats:", error);
}
}
async loadLogConfig() {
try {
const response = await ApiClient.fetchUrl('/prompt_manager/logs/config');
const data = await response.json();
if (data.success) {
const config = data.config;
document.getElementById("setLogLevel").value = config.level || "INFO";
document.getElementById("consoleLogging").checked = config.console_logging !== false;
}
} catch (error) {
console.error("Failed to load log config:", error);
}
}
async loadLogFiles() {
try {
const response = await ApiClient.fetchUrl('/prompt_manager/logs/files');
const data = await response.json();
if (data.success) {
const container = document.getElementById("logFilesList");
if (data.files.length === 0) {
container.innerHTML = '<div class="col-span-full text-center text-pm-muted">No log files found</div>';
return;
}
container.innerHTML = data.files.map(file => `
<div class="bg-pm-surface rounded-pm-sm p-3">
<div class="flex items-center justify-between mb-2">
<span class="text-sm font-medium text-pm">${escapeHtml(file.filename)}</span>
${file.is_main ? '<span class="bg-pm-accent text-xs px-2 py-1 rounded">Active</span>' : ''}
</div>
<div class="text-xs text-pm-secondary mb-2">
Size: ${this.formatBytes(file.size)} | Modified: ${new Date(file.modified).toLocaleString()}
</div>
<button data-action="download-log" data-filename="${escapeHtml(file.filename)}"
class="w-full px-3 py-1 bg-pm-success hover:bg-pm-success text-pm text-xs rounded transition-colors">
📥 Download
</button>
</div>
`).join('');
}
} catch (error) {
console.error("Failed to load log files:", error);
}
}
async refreshLogs() {
const level = document.getElementById("logLevel").value;
const limit = parseInt(document.getElementById("logLimit").value);
try {
const url = new URL(ApiClient.resolveUrl('/prompt_manager/logs'), window.location.origin);
if (level) url.searchParams.set('level', level);
url.searchParams.set('limit', limit.toString());
const response = await ApiClient.fetchUrl(url);
const data = await response.json();
if (data.success) {
this.displayLogs(data.logs);
this.showNotification(`Loaded ${data.logs.length} log entries`, "success");
} else {
this.showNotification(`Failed to load logs: ${data.error}`, "error");
}
} catch (error) {
console.error("Failed to refresh logs:", error);
this.showNotification("Failed to refresh logs", "error");
}
}
displayLogs(logs) {
const container = document.getElementById("logsContainer");
if (logs.length === 0) {
container.innerHTML = '<div class="text-center py-4 text-pm-muted">No logs found</div>';
return;
}
container.innerHTML = logs.map(log => {
const levelColors = {
DEBUG: 'text-pm-secondary',
INFO: 'text-pm-accent',
WARNING: 'text-pm-warning',
ERROR: 'text-pm-error',
CRITICAL: 'text-pm-error'
};
const levelColor = levelColors[log.level] || 'text-pm-secondary';
const timestamp = new Date(log.timestamp).toLocaleString();
return `
<div class="border-l-2 border-pm pl-3 py-1 hover:bg-pm-surface transition-colors">
<div class="flex items-start space-x-2 text-sm">
<span class="text-pm-muted text-xs font-mono w-24 flex-shrink-0">${escapeHtml(timestamp.split(' ')[1])}</span>
<span class="${levelColor} font-semibold w-16 flex-shrink-0">${escapeHtml(log.level)}</span>
<span class="text-pm-accent text-xs w-32 flex-shrink-0">${escapeHtml(log.logger)}</span>
<span class="text-pm-secondary flex-1">${escapeHtml(log.message)}</span>
</div>
<div class="text-xs text-pm-muted ml-44">
${escapeHtml(log.filename)}:${escapeHtml(log.lineno)}
</div>
</div>
`;
}).join('');
// Auto-scroll to bottom
container.scrollTop = container.scrollHeight;
}
async downloadLogs() {
const level = document.getElementById("logLevel").value;
const limit = parseInt(document.getElementById("logLimit").value);
try {
const url = new URL(ApiClient.resolveUrl('/prompt_manager/logs'), window.location.origin);
if (level) url.searchParams.set('level', level);
url.searchParams.set('limit', limit.toString());
const response = await ApiClient.fetchUrl(url);
const data = await response.json();
if (data.success) {
const logText = data.logs.map(log =>
`${log.timestamp} [${log.level}] ${log.logger}: ${log.message} (${log.filename}:${log.lineno})`
).join('\n');
const blob = new Blob([logText], { type: 'text/plain' });
const url2 = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url2;
a.download = `prompt_manager_logs_${new Date().toISOString().slice(0, 19).replace(/:/g, '-')}.txt`;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url2);
this.showNotification("Logs downloaded successfully", "success");
} else {
this.showNotification(`Failed to download logs: ${data.error}`, "error");
}
} catch (error) {
console.error("Failed to download logs:", error);
this.showNotification("Failed to download logs", "error");
}
}
async downloadLogFile(filename) {
try {
const response = await ApiClient.fetchUrl(`/prompt_manager/logs/download/${encodeURIComponent(filename)}`);
if (response.ok) {
const blob = await response.blob();
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = filename;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
this.showNotification(`Downloaded ${filename}`, "success");
} else {
this.showNotification(`Failed to download ${filename}`, "error");
}
} catch (error) {
console.error("Failed to download log file:", error);
this.showNotification("Failed to download log file", "error");
}
}
async clearLogs() {
if (!confirm("Are you sure you want to clear all log files? This cannot be undone.")) {
return;
}
try {
const response = await ApiClient.fetchUrl('/prompt_manager/logs/truncate', {
method: 'POST'
});
const data = await response.json();
if (data.success) {
this.showNotification(data.message, "success");
this.refreshLogs();
this.loadLogStats();
this.loadLogFiles();
} else {
this.showNotification(`Failed to clear logs: ${data.error}`, "error");
}
} catch (error) {
console.error("Failed to clear logs:", error);
this.showNotification("Failed to clear logs", "error");
}
}
async updateLogConfig() {
const level = document.getElementById("setLogLevel").value;
const consoleLogging = document.getElementById("consoleLogging").checked;
try {
const response = await ApiClient.fetchUrl('/prompt_manager/logs/config', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
level: level,
console_logging: consoleLogging
})
});
const data = await response.json();
if (data.success) {
this.showNotification("Log configuration updated", "success");
this.loadLogStats();
} else {
this.showNotification(`Failed to update config: ${data.error}`, "error");
}
} catch (error) {
console.error("Failed to update log config:", error);
this.showNotification("Failed to update log configuration", "error");
}
}
formatBytes(bytes) {
if (bytes === 0) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
}
// Pagination methods
updatePaginationControls() {
const { currentPage, totalPages, total, limit } = this.pagination;
@@ -3801,7 +3358,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
} else if (data.type === 'complete') {
this.autoTagState.eventSource.close();
this.hideModal("autoTagProgressModal");
this.showNotification(`Auto tagging complete! Applied tags to ${data.tagged || 0} prompts.`, 'success');
this.showNotification(ImageHelpers.autotagCompleteMessage(data.tagged), 'success');
this.refreshList(); // Refresh the prompt list
} else if (data.type === 'error') {
this.autoTagState.eventSource.close();
@@ -3819,7 +3376,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
console.error('EventSource readyState:', this.autoTagState.eventSource.readyState);
this.autoTagState.eventSource.close();
this.hideModal("autoTagProgressModal");
this.showNotification('Connection lost during auto tagging', 'error');
this.showNotification((error && error.serverMessage) || 'Connection lost during auto tagging', 'error');
};
} catch (error) {
console.error('Auto tag error:', error);
@@ -3894,10 +3451,14 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
}
}
// Check if tags array has "real" tags (excluding auto-scanned and prepend:* metadata)
getRealTags(tags) {
if (!tags || !Array.isArray(tags)) return [];
return tags.filter(tag => tag !== 'auto-scanned' && !tag.startsWith('prepend:') && !tag.startsWith('append:'));
/** POST /autotag/single for `image` (#379: `existing` "skip" answers already_tagged for an auto-tagged image). */
async requestReviewTags(image, existing) {
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/single', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(ImageHelpers.autotagSingleBody(image, { ...this.autoTagState, existing }))
});
return response.json();
}
// Show the re-tag confirmation modal
@@ -3949,26 +3510,48 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
// always provides a servable url; never rebuild it from the path.
const imageUrl = this.getImageUrl(image);
// Check if image already has real tags (excluding auto-scanned)
const realTags = this.getRealTags(image.prompt_tags);
this.autoTagState.currentImage = image;
this.autoTagState.currentSha256 = null;
this.autoTagState.tagsExpanded = false; // Reset accordion for new image
if (realTags.length > 0) {
// Skip if "Skip All Tagged" was selected
if (this.autoTagState.skipAllTagged) {
this.autoTagState.reviewIndex++;
return this.loadNextReviewImage();
let data;
try {
// Each image is first asked with "skip": an image a tagging model already tagged answers
// already_tagged (its image tags, never its prompt's) without running the model (#379).
data = await this.requestReviewTags(image, 'skip');
if (data.success && data.already_tagged) {
if (this.autoTagState.skipAllTagged) {
this.autoTagState.reviewIndex++;
return this.loadNextReviewImage();
}
const choice = await this.showRetagConfirmation(image, imageUrl, data.image_tags || []);
if (choice === 'skip' || choice === 'skipAll') {
this.autoTagState.reviewIndex++;
return this.loadNextReviewImage();
}
// choice === 'retag': generate new tags for it
this.showReviewImage(image, imageUrl);
data = await this.requestReviewTags(image, 'merge');
} else {
this.showReviewImage(image, imageUrl);
}
// Show confirmation modal
const choice = await this.showRetagConfirmation(image, imageUrl, realTags);
if (choice === 'skip' || choice === 'skipAll') {
this.autoTagState.reviewIndex++;
return this.loadNextReviewImage();
if (data.success) {
this.autoTagState.currentTags = data.tags;
this.autoTagState.currentSha256 = data.sha256 || null;
this.renderReviewTags();
} else {
document.getElementById('reviewTagsVisible').innerHTML =
`<div class="text-pm-error">Error: ${escapeHtml(data.error)}</div>`;
}
// choice === 'retag' - continue with tagging
} catch (error) {
console.error('Error generating tags:', error);
this.showReviewImage(image, imageUrl);
document.getElementById('reviewTagsVisible').innerHTML =
'<div class="text-pm-error">Failed to generate tags</div>';
}
}
showReviewImage(image, imageUrl) {
document.getElementById('reviewImage').src = imageUrl;
// The record's file name, else the last segment of its path (#328: `filename` was never declared).
const filename = image.filename || String(image.image_path || image.path || '').split(/[\\/]/).pop();
@@ -3977,37 +3560,6 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
document.getElementById('reviewTagsHidden').innerHTML = '';
document.getElementById('reviewTagsAccordion').classList.add('hidden');
document.getElementById('reviewTagsToggle').classList.add('hidden');
// Store prompt_id from database for applying tags later
this.autoTagState.currentPromptId = image.prompt_id;
this.autoTagState.tagsExpanded = false; // Reset accordion for new image
try {
// Review rows come from /images/all (database ids), so this resolves to image_id
const requestBody = ImageHelpers.autotagSingleBody(image, this.autoTagState);
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/single', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(requestBody)
});
const data = await response.json();
if (data.success) {
this.autoTagState.currentTags = data.tags;
// Use prompt_id from database, fallback to API response
this.autoTagState.currentPromptId = image.prompt_id || data.prompt_id;
this.renderReviewTags();
} else {
document.getElementById('reviewTagsVisible').innerHTML =
`<div class="text-pm-error">Error: ${escapeHtml(data.error)}</div>`;
}
} catch (error) {
console.error('Error generating tags:', error);
document.getElementById('reviewTagsVisible').innerHTML =
'<div class="text-pm-error">Failed to generate tags</div>';
}
}
/** /prompt_manager/images/all is capped per call; collect every page. */
@@ -4126,15 +3678,9 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
}
async applyReviewTags() {
if (this.autoTagState.currentTags.length === 0) {
this.skipReviewImage();
return;
}
const promptId = this.autoTagState.currentPromptId;
if (!promptId) {
this.showNotification('No linked prompt found for this image', 'warning');
// #379: the reviewed tags go to this image (linked or not), never its prompt; an empty
// list still marks the image as reviewed by a tagging model.
if (!this.autoTagState.currentSha256) {
this.skipReviewImage();
return;
}
@@ -4143,10 +3689,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/apply', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
prompt_id: promptId,
tags: this.autoTagState.currentTags
})
body: JSON.stringify(ImageHelpers.autotagApplyBody(
this.autoTagState.currentImage, this.autoTagState.currentSha256, this.autoTagState.currentTags))
});
const data = await response.json();
+20 -16
View File
@@ -3560,19 +3560,24 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
} else if (data.type === 'complete') {
this.autoTagState.eventSource.close();
this.hideModal("autoTagProgressModal");
this.showNotification(`Auto tagging complete! Applied tags to ${data.tagged || 0} prompts.`, 'success');
this.showNotification(ImageHelpers.autotagCompleteMessage(data.tagged), 'success');
this.loadImages(); // Refresh the gallery
} else if (data.type === 'error') {
this.autoTagState.eventSource.close();
this.hideModal("autoTagProgressModal");
this.showNotification(`Auto tag error: ${data.message}`, 'error');
} else if (data.type === 'cancelled') {
// Cancelled in Prompt Manager's Activity (#379): as its own cancel.
this.autoTagState.eventSource.close();
this.hideModal("autoTagProgressModal");
this.showNotification('Auto tagging cancelled', 'info');
}
};
this.autoTagState.eventSource.onerror = () => {
this.autoTagState.eventSource.onerror = (error) => {
this.autoTagState.eventSource.close();
this.hideModal("autoTagProgressModal");
this.showNotification('Connection lost during auto tagging', 'error');
this.showNotification((error && error.serverMessage) || 'Connection lost during auto tagging', 'error');
};
} catch (error) {
console.error('Auto tag error:', error);
@@ -3629,6 +3634,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
this.reviewRequest = request;
this.autoTagState.currentTags = [];
this.autoTagState.reviewPromptId = null;
this.autoTagState.reviewImage = null;
this.autoTagState.reviewSha256 = null;
if (this.autoTagState.reviewIndex >= this.autoTagState.reviewImages.length) {
this.hideModal("autoTagReviewModal");
this.showNotification('Review complete!', 'success');
@@ -3643,8 +3650,9 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
document.getElementById('reviewTagsContainer').innerHTML = '<div class="text-pm-secondary">Generating tags...</div>';
try {
// Gallery entries come from /images/output: id is a path digest, so address by path
const requestBody = ImageHelpers.autotagSingleBody(image, this.autoTagState);
// Gallery entries come from /images/output: id is a path digest, so address by path.
// The gallery review always suggests (it never skipped tagged images, #379).
const requestBody = ImageHelpers.autotagSingleBody(image, { ...this.autoTagState, existing: 'merge' });
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/single', {
method: 'POST',
@@ -3657,6 +3665,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
if (this.reviewRequest !== request) return;
if (data.success) {
this.autoTagState.reviewPromptId = data.prompt_id ?? null;
this.autoTagState.reviewImage = image;
this.autoTagState.reviewSha256 = data.sha256 || null;
this.autoTagState.currentTags = data.tags;
this.renderReviewTags();
} else {
@@ -3697,16 +3707,12 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
}
async applyReviewTags() {
if (this.autoTagState.currentTags.length === 0) {
// #379: the reviewed tags go to this image (linked or not), never its prompt; an empty
// list still marks the image as reviewed by a tagging model.
if (!this.autoTagState.reviewSha256) {
this.skipReviewImage();
return;
}
const promptId = this.autoTagState.reviewPromptId;
if (!promptId) {
this.showNotification('This image has no linked prompt. Scan the output folder first.', 'warning');
return;
}
const request = this.reviewRequest;
if (this.reviewApplying === request) return;
this.reviewApplying = request;
@@ -3715,10 +3721,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/apply', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
prompt_id: promptId,
tags: this.autoTagState.currentTags
})
body: JSON.stringify(ImageHelpers.autotagApplyBody(
this.autoTagState.reviewImage, this.autoTagState.reviewSha256, this.autoTagState.currentTags))
});
const data = await response.json();
+44 -10
View File
@@ -70,10 +70,28 @@
* path digest as id, which would 404, so those are addressed by path.
*
* @param {{id?: unknown, path?: unknown}} image
* @param {{modelType: string, generalThreshold?: number, characterThreshold?: number, prompt?: string}} settings
* `existing` (#379): "skip" answers `already_tagged` for an image a model
* already tagged; anything else is "merge" (always suggest).
*
* @param {{id?: unknown, path?: unknown}} image
* @param {{modelType: string, generalThreshold?: number, characterThreshold?: number, prompt?: string, existing?: string}} settings
* @returns {object} JSON-serialisable request body
*/
function autotagSingleBody(image, settings) {
const target = autotagTarget(image);
const existing = settings.existing === "skip" ? "skip" : "merge";
const modelType = String(settings.modelType || "");
const params = modelType.startsWith("wd14")
? {
general_threshold: settings.generalThreshold,
character_threshold: settings.characterThreshold,
}
: { prompt: settings.prompt };
return { ...target, model_type: modelType, existing, ...params };
}
/** The image a legacy single/apply body names: its row id, or its path (and root). */
function autotagTarget(image) {
const entry = image || {};
const target =
typeof entry.path === "string" && entry.path && !Number.isInteger(entry.id)
@@ -82,14 +100,22 @@
if ("path" in target && Number.isInteger(entry.root_index)) {
target.root_index = entry.root_index;
}
const modelType = String(settings.modelType || "");
const params = modelType.startsWith("wd14")
? {
general_threshold: settings.generalThreshold,
character_threshold: settings.characterThreshold,
}
: { prompt: settings.prompt };
return { ...target, model_type: modelType, ...params };
return target;
}
/**
* Body for POST /prompt_manager/autotag/apply (#379): the image, the
* `sha256` /autotag/single answered and the reviewed tags. The tags go to
* that image only, never to its prompt; an empty list marks it reviewed.
*/
function autotagApplyBody(image, sha256, tags) {
return { ...autotagTarget(image), sha256: String(sha256 || ""), tags: Array.isArray(tags) ? [...tags] : [] };
}
/** The legacy batch's completion toast (#379): images, never prompts. */
function autotagCompleteMessage(tagged) {
const count = Number.isSafeInteger(tagged) && tagged > 0 ? tagged : 0;
return `Auto tagging complete! Tagged ${count} ${count === 1 ? "image" : "images"}.`;
}
/**
@@ -122,7 +148,15 @@
}
}
const api = { formatImageCaption, workflowDocumentHtml, autotagSingleBody, fallbackImageSource, metadataLookupUrl };
const api = {
formatImageCaption,
workflowDocumentHtml,
autotagSingleBody,
autotagApplyBody,
autotagCompleteMessage,
fallbackImageSource,
metadataLookupUrl,
};
if (typeof module !== "undefined" && module.exports) module.exports = api;
else root.ImageHelpers = api;
})(typeof window !== "undefined" ? window : globalThis);
+15 -1
View File
@@ -59,6 +59,20 @@
* @returns {{onmessage: Function|null, onerror: Function|null, onclose: Function|null,
* readyState: number, close: Function, CONNECTING: number, OPEN: number, CLOSED: number}}
*/
// A non-2xx answer as an Error carrying its status and, when the body is
// the API's JSON {error}, that sentence as serverMessage for the page.
async function httpError(response) {
const error = new Error(`HTTP ${response.status}`);
error.status = response.status;
try {
const data = await response.json();
if (data && typeof data.error === "string" && data.error) error.serverMessage = data.error;
} catch (_) {
// Not JSON: the status alone.
}
return error;
}
function connect(url, { method = "GET", body, fetchImpl } = {}) {
const doFetch = fetchImpl || ((...args) => globalThis.fetch(...args));
const controller = new AbortController();
@@ -103,7 +117,7 @@
if (body !== undefined) init.body = body;
const requestUrl = root.ApiClient ? root.ApiClient.resolveUrl(url) : url;
const response = await doFetch(requestUrl, init);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
if (!response.ok) throw await httpError(response);
if (source.readyState === CLOSED) return;
source.readyState = OPEN;
+7 -2
View File
@@ -275,8 +275,13 @@ export function createImageStream({ api, onChange = () => {}, batchSize = 40, re
};
}
/** The concept's count line (app.js a446): "N of M images · K positive prompts", without the count while unknown. */
export function countLine({ loaded, total, positiveCount }) {
/**
* The concept's count line (app.js a446): "N of M images · K positive prompts", without the count while unknown, and
* the page's first "Loading images…" while a first page loads.
*/
export function countLine({ loaded, total, positiveCount, loading = false, error = null }) {
// A first page in flight has no count yet: "0 of 0 images" would claim the filter matched nothing (#391).
if (loading && !loaded && !error) return 'Loading images…';
const images = `${loaded.toLocaleString('en-US')} of ${total.toLocaleString('en-US')} ${noun(total, 'image')}`;
return isCount(positiveCount) ? `${images} · ${positiveCount.toLocaleString('en-US')} ${noun(positiveCount, 'positive prompt')}` : images;
}
+4 -1
View File
@@ -433,10 +433,13 @@ pre.prompt-text { font: inherit; font-size: 12px; line-height: 1.75; margin-bloc
}
/* D132: at phone width the concept's 120px cap cuts every engine label ("WD14 SwinV2 · ~400 MB"); the Tagging model
select alone keeps its natural width there (its widest engine label) and, with no room beside the row text, wraps
under it. */
under it. Existing tags (#398) never shrinks for its row text and, past the cap, keeps its natural width beside the
text ("Replace after review"), bounded by its row like the engine select. */
.setting-row #tag-existing { flex-shrink: 0; /* D132 */ }
@media (max-width: 700px) {
#workspace-main .setting-row:has(> #tag-model) { flex-wrap: wrap; /* D132 */ }
#workspace-main .setting-row #tag-model { flex-shrink: 0; max-width: 100% !important; /* D132 */ }
#workspace-main .setting-row #tag-existing { max-width: 100% !important; /* D132 */ }
}
/* D109: System › Diagnostics & logs' added Lines field sits in the concept's toolbar like its Level select, and a value