Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8ddcfe27e1 | ||
|
|
9601837aa9 | ||
|
|
726b7b70da | ||
|
|
af0482a12d | ||
|
|
7c82fdb180 | ||
|
|
1e9ed87454 | ||
|
|
ca8171ec38 | ||
|
|
2ed085f8d7 | ||
|
|
2d0e962f1d | ||
|
|
3f98b57aef | ||
|
|
80da6f125e | ||
|
|
1ba8052cfa | ||
|
|
737d3c8845 | ||
|
|
8bf759d6f8 | ||
|
|
dc8b86751e | ||
|
|
fc3266d455 | ||
|
|
06099579e6 |
@@ -211,3 +211,6 @@ coverage/
|
||||
|
||||
# Disposable upstream checkout for real-host CI
|
||||
.ci-comfyui/
|
||||
|
||||
# Old library files a Move library renamed (never deleted; #249 S6b)
|
||||
*.moved-to-*
|
||||
|
||||
@@ -751,9 +751,51 @@ ComfyUI_PromptManager/
|
||||
|
||||
## Configuration
|
||||
|
||||
### Where your library lives
|
||||
|
||||
New installs keep the library (`prompts.db`, `config.json` with the CivitAI key, the
|
||||
database backups and the logs) in ComfyUI's system user folder
|
||||
`ComfyUI/user/__promptmanager/`. An update, a reinstall or ComfyUI-Manager replacing
|
||||
the node folder no longer touches it, and ComfyUI's `/userdata` web API does not
|
||||
serve `__` system folders. Thumbnails stay in `output/thumbnails`.
|
||||
|
||||
- **Upgrading from an earlier version:** your library stays where it is, in the node
|
||||
folder, until you move it; nothing is moved on its own. The workspace shows a notice
|
||||
above every page and System › Overview a "Library location" row. **Move library**
|
||||
copies the database with SQLite's backup API, verifies the copy (integrity check, every
|
||||
workflow document, the same rows in every table), copies `config.json` and the backups,
|
||||
switches to the new folder without a restart and only then renames the old files
|
||||
`prompts.db.moved-to-__promptmanager` and `config.json.moved-to-__promptmanager` (never
|
||||
deleted; the old backups stay too). If anything fails, the library stays in the node
|
||||
folder, unchanged. **Keep it here** stops the notice; the Overview row stays. Take a
|
||||
backup first (System › Backup & export or the dashboard's Backup) as with any change.
|
||||
- **Your own folder:** set `PROMPT_MANAGER_DATA_DIR` to an absolute folder outside
|
||||
ComfyUI's output, input and temp folders and your image sources. If it is empty and a
|
||||
node-folder library exists, that library stays in use until you move it there.
|
||||
- **An explicit `database.default_path`** in `config.json` and `PROMPT_MANAGER_CONFIG_PATH`
|
||||
keep working as before (described below); such a library is not offered a move.
|
||||
|
||||
### Thumbnails
|
||||
|
||||
Previews in `output/thumbnails` are written as **WebP, quality 80, 512 px** by default
|
||||
(System › Appearance › Thumbnails sets the format, quality and size limit for every
|
||||
preview). Previews from older versions keep their old format (usually PNG) until you
|
||||
run **Convert outdated** (System › Overview or Appearance): it rewrites each one from
|
||||
its original and then removes the old file. Nothing converts on its own.
|
||||
|
||||
**JPEG XL** (smaller still) can be chosen once the server has an encoder. Install one
|
||||
of these, then restart ComfyUI:
|
||||
|
||||
- `pip install pillow-jxl-plugin` in ComfyUI's Python environment (simplest), or
|
||||
- an `ffmpeg` on PATH built with `libjxl`, or
|
||||
- `cjxl` from [libjxl](https://github.com/libjxl/libjxl) on PATH.
|
||||
|
||||
Browsers that cannot show JPEG XL see the original images in the grid instead.
|
||||
|
||||
### Database Settings
|
||||
|
||||
You can customize the database path by creating a `config.json` file in the extension root:
|
||||
You can customize the database path by creating a `config.json` file in the library's
|
||||
folder (the node folder for a library that was not moved):
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -970,6 +1012,10 @@ MIT License - see LICENSE file for details.
|
||||
|
||||
## Changelog
|
||||
|
||||
### Unreleased (Library location)
|
||||
|
||||
- **📁 Library outside the node folder**: new installs keep `prompts.db`, `config.json`, the database backups and the logs in `ComfyUI/user/__promptmanager/`, where an update or reinstall of the node cannot delete them and ComfyUI's `/userdata` API does not reach. An existing library stays in the node folder until you press **Move library** (a verified copy, then a live switch; the old files are renamed `*.moved-to-__promptmanager`, never deleted) or **Keep it here**. `PROMPT_MANAGER_DATA_DIR` names your own folder. After moving, restart ComfyUI once so the logs follow. Going back to an older version after a move: an older version does not look in `user/__promptmanager`, so stop ComfyUI and copy `user/__promptmanager/prompts.db` (with its `-wal` file, if one is left) and `config.json` into the node folder; the `*.moved-to-__promptmanager` files are the library as it was at the move, without later changes.
|
||||
|
||||
### Unreleased (Workflows stored once)
|
||||
|
||||
- **🗜️ Smaller database**: each distinct workflow and prompt graph is stored once, compressed, instead of again for every image. Images recorded by this version use it from the first start; the maintenance job "Store each workflow once" rewrites the existing records and then reclaims the space. A 20,000-image test library went from 1.4 GB to 60 MB. The job always takes a verified backup first: the first time, `prompts.db.pre-workflow-storage_<date time>` beside the database, kept until you delete it; on later runs, a normal pooled safety backup. No image or video file is opened or changed.
|
||||
|
||||
+332
-36
@@ -1932,8 +1932,8 @@ pending count (`GET /prompt_manager/review/items?limit=1`, its `total`).
|
||||
unavailable file's empty media button keeps the concept's 1.33 media area, so its
|
||||
card lines up with the others. A long file name in the panel's `p.small` wraps inside
|
||||
the panel (`overflow-wrap: anywhere`), where the concept's short sample name never
|
||||
overflows. A queue card shows the Images listing's thumbnail when the server knows one
|
||||
(`thumbnail_url`, the same URL the Images page uses), else the original; a re-read
|
||||
overflows. A queue card shows the Images listing's thumbnail (`thumbnail_url`, the
|
||||
same URL the Images page uses; a preview not made yet is made on first view); a re-read
|
||||
rebuilds a card whose media, target or availability changed (a root came or went), so
|
||||
Open and Tag follow it. When an unanswered decision's item leaves the queue in a
|
||||
re-read, the decision was recorded: the page finishes as a recorded decision does (its
|
||||
@@ -2115,10 +2115,11 @@ Sources & scans (`system-sources.mjs`, S4) is the concept's "sources" body on th
|
||||
server's sources. `GET /prompt_manager/settings` reports them (public paths only):
|
||||
`sources.configured` (each `gallery_root_paths` entry with `valid`, `code` and the
|
||||
server's `reason`), `sources.effective` (what is scanned and served, with its
|
||||
`root_index` and `watch`: `watching` only when the image monitor's observer is alive,
|
||||
`root_index`, `watch` (`watching` only when the image monitor's observer is alive,
|
||||
scheduled that folder and monitoring is on, `stopping` when it still watches it but
|
||||
monitoring was switched off, `manual` when monitoring is off or no monitor runs, else
|
||||
`restart_pending`), `sources.auto_detected`, `monitor` and `settings.monitoring_enabled`.
|
||||
`restart_pending`) and `in_comfyui` (its public path is relative to the ComfyUI base)),
|
||||
`sources.auto_detected`, `monitor` and `settings.monitoring_enabled`.
|
||||
`POST /prompt_manager/settings/validate-source {path}` checks one folder and writes
|
||||
nothing. Every caller (validate-source, the settings save, config.json at load) refuses
|
||||
a source through the same checks (`sources.source_problem`): its form first, then
|
||||
@@ -2162,20 +2163,27 @@ A list that cannot be read shows "Sources could not be loaded. Your saved data h
|
||||
changed." and Retry, never an empty list.
|
||||
|
||||
Thumbnail generation is an Activity job (S5, `py/api/thumbnail_jobs.py`). `GET
|
||||
/prompt_manager/images/thumbnails/plan?mode=missing|rebuild&size=128..2048&quality=low|medium|high`
|
||||
counts the work (`targets`, `fresh`, `remaining` past the 10,000-file cap, the source
|
||||
labels) and writes nothing; a value out of range is a 400 `invalid-request` that starts
|
||||
nothing. `POST /prompt_manager/jobs/thumbnails {mode, size, quality}` starts the
|
||||
/prompt_manager/images/thumbnails/plan?mode=missing|rebuild|convert`
|
||||
counts the work (`targets`, `fresh`, `outdated`, `remaining` past the 10,000-file batch cap,
|
||||
the source labels, and the server's `format`, `quality` and `size` it writes with) and
|
||||
writes nothing; an unknown mode is a 400 `invalid-request` that starts nothing. `size`
|
||||
and `quality` an older client sends are ignored (#317). `POST /prompt_manager/jobs/thumbnails {mode}` starts the
|
||||
`thumbnails` job ("Thumbnail generation"): admission `guarded` (never beside a scan or
|
||||
another record-changing job, one at a time, a legacy clear refused while it runs),
|
||||
cancellable at a file boundary (a file not yet started is left alone; written previews
|
||||
stay), outcome groups Generated / Skipped (already fresh, listed without a log line
|
||||
each) / Errors. `size` is the longest edge and `quality` the encoder (low: JPEG 70, no
|
||||
optimize; medium: 85, optimized, what the legacy routes always wrote; high: 95,
|
||||
optimized). Video previews take the same choice: the OpenCV frame and the placeholder
|
||||
are saved with those JPEG settings, and the ffmpeg fallback (no OpenCV) maps them to
|
||||
`-q:v` 8 / 4 / 2 and fits the frame inside the edge keeping its aspect ratio (`-vf
|
||||
scale=E:E:force_original_aspect_ratio=decrease`, no longer a stretched square).
|
||||
each) / Errors. Format, quality and the longest edge are the server's thumbnail
|
||||
settings (#317, below), read once when the job starts and shown in its scope. Video
|
||||
previews take the same settings: the OpenCV frame, ffmpeg's frame (no OpenCV; a PNG on
|
||||
stdout fitted inside the edge keeping its aspect ratio and never upscaled, `-vf
|
||||
scale='min(iw,E)':'min(ih,E)':force_original_aspect_ratio=decrease`) and the
|
||||
placeholder are encoded like an image's preview.
|
||||
MAX_THUMBNAIL_FILES (10,000) bounds one batch, not the job: after each batch the
|
||||
missing, rebuild and convert jobs plan again, leaving out every file a batch already
|
||||
worked on, and go on until nothing is left, the job is cancelled, or a batch made no
|
||||
progress (every file failed: the rest waits for the next run). Progress counts against
|
||||
the overall total and the final line reports every batch (`TestBatchesContinue`,
|
||||
`test_convert_goes_on_past_the_per_batch_cap`). The plan dialog counts all of it.
|
||||
`rebuild` writes fresh previews again, each through the same temp-file
|
||||
rename, so nothing is cleared first and a failed file keeps its old preview (a video
|
||||
whose frame cannot be decoded keeps its preview instead of getting the placeholder);
|
||||
@@ -2183,7 +2191,7 @@ originals are only read (`tests/test_thumbnail_jobs.py`). Cancel is also checked
|
||||
file during the planning walk: a job cancelled there stops the walk at the next file,
|
||||
publishes no `start` frame and ends "Cancelled before any preview was written". Every
|
||||
exit sets a final status line: "N previews written · M errors" (plus "· R more on the
|
||||
next run" past the cap), "No previews to generate · N already up to date" (or "No
|
||||
next run" when a batch made no progress), "No previews to generate · N already up to date" (or "No
|
||||
images or videos to preview"), "Cancelled · N previews written", or the failure; a job
|
||||
with nothing to do never reads "Processing file 0 of 0" (`test_every_exit_sets_a_truthful_final_status_line`,
|
||||
and "a finished thumbnail job with nothing to do says so" in `workspace-activity.spec.mjs`).
|
||||
@@ -2200,8 +2208,9 @@ recording runs beside it instead of being deferred for a whole rebuild
|
||||
all `records` jobs still defer it.
|
||||
|
||||
Deliberate change: the legacy gallery stream `POST /images/generate-thumbnails/progress`
|
||||
now starts this job (missing previews, its quality preset as the size, medium encoding)
|
||||
or attaches to the running one, and streams the job's unchanged `status`/`start`/
|
||||
now starts this job (missing previews with the server's thumbnail settings; its
|
||||
`quality` preset no longer sizes anything, #317 R3) or attaches to the running one (never
|
||||
to a running Convert outdated, which it refuses with its error frame), and streams the job's unchanged `status`/`start`/
|
||||
`progress`/`file_error`/`complete` frames after a first `job {id, started}` frame that
|
||||
names the job and whether this stream started it. The stream reads the registry after
|
||||
its headers are sent, so a job that starts in that gap is attached to, not refused. A
|
||||
@@ -2217,18 +2226,151 @@ generation was cancelled". The blocking
|
||||
`POST /images/generate-thumbnails` and the clear stay legacy holds.
|
||||
|
||||
The workspace starts it from "Thumbnail maintenance" (`thumbnail-review.mjs`, the
|
||||
concept's `thumbnail-review`), opened by Sources' Thumbnail generation → Review options
|
||||
and Overview's Review thumbnails. Scope is Missing thumbnails only (`missing`, the
|
||||
default) or Rebuild all (explicit opt-in) (`rebuild`); Quality is Medium · optimized /
|
||||
High · original / Low · fastest (`medium` / `high` / `low`), preselected from
|
||||
Appearance's Image loading quality (`gallerySettings.imageQuality`, the same values,
|
||||
Q3); Preview size limit is the longest edge, 512 by default, a whole number from 128 to
|
||||
2048. A size outside that range marks the field `aria-invalid`, the plan line says
|
||||
"Enter a size from 128 to 2048 pixels." and nothing is requested; Start reports the
|
||||
field's validity and keeps the dialog open. Start closes the dialog and starts the job
|
||||
through the Activity store, which toasts "Thumbnail generation started."; the job opens
|
||||
concept's `thumbnail-review`), opened by Sources' Thumbnail generation → Review options,
|
||||
Overview's Review thumbnails and Convert outdated, and Appearance's Convert outdated.
|
||||
Scope is Missing thumbnails only (`missing`, the default), Rebuild all (explicit opt-in)
|
||||
(`rebuild`) or Convert outdated (removes older formats) (`convert`, #317). D131: the
|
||||
concept's Quality and Preview size limit fields are replaced by a line naming the
|
||||
server's settings ("Previews are written as WebP · quality 80 · 512 px. Change this in
|
||||
System › Appearance."). Start closes the dialog and starts the job
|
||||
through the Activity store, which toasts "<job title> started."; the job opens
|
||||
in Activity. A start refused because another job holds the slot toasts the D57 busy
|
||||
link toast and stays. The legacy gallery's thumbnail tools are unchanged.
|
||||
link toast and stays. The legacy gallery's thumbnail tools keep working on the same
|
||||
server settings.
|
||||
|
||||
Automatic previews (#249, `py/api/thumbnail_auto.py`). The approved mock has no
|
||||
repeated global thumbnail prompt (`docs/concepts/workspace/README.md`), so the
|
||||
workspace never asks; previews are a cache the server keeps:
|
||||
|
||||
- Every listing gives an image or video a thumbnail URL (`media_thumbnail_url`: Images,
|
||||
saved views and favorites, Library cards and galleries, Needs review, duplicate
|
||||
groups). An existing preview's URL keeps its `v=<mtime>`; a missing one is listed
|
||||
without `v=`.
|
||||
- `GET /images/serve/thumbnails/<rel>_thumb<ext>` for a missing preview is refused as
|
||||
text first (traversal, a backslash, a drive or stream colon, empty or dot segments, so
|
||||
`D:/x`, `\\server\share` and a leading `/` never reach a joined path), then admitted
|
||||
on the event loop (`_PreviewGate`): 3 previews are written at once on the API's own
|
||||
`pm-preview` pool (never the default executor), one flight runs per file (the history
|
||||
route shares the flight key and the admission), and at most 64 requests wait; one
|
||||
more gets its fallback at once without listing a folder. Inside the admission the
|
||||
preview maps back to its media file under the same root (exact names, then one folder
|
||||
listing for a mixed-case extension; a regular file inside the root, never through a
|
||||
link) and is made with the server's thumbnail settings (#317). When it cannot be made an
|
||||
image's original is served and a video's placeholder frame (a video is never sent to
|
||||
an `<img>`), both `Cache-Control: no-store`; cards and the inspector cover also swap a
|
||||
video back to its own element when its preview image fails. Only a file that cannot
|
||||
be decoded is remembered for the session (path, size, mtime; retried once it changes)
|
||||
and counted as `failed` in `/images/thumbnails/status`, which Overview's row adds
|
||||
("N missing · M outdated · K failed"); I/O errors and a preview another program holds
|
||||
open (Windows) are not. A preview folder (or any link segment of it) resolving out of
|
||||
the root is never written. Media inside a nested `thumbnails` folder is listed with no
|
||||
thumbnail and keeps its original. The listing's `thumbnail_ready` says whether the
|
||||
preview file exists (the legacy gallery's dot and "Fast").
|
||||
- The video decoder falls through OpenCV, ffmpeg (a tracked process, killed on
|
||||
shutdown) and the placeholder frame: an OpenCV that cannot open or read a clip (AV1)
|
||||
no longer ends the run.
|
||||
- The image monitor hands every settled output to `queue_capture_thumbnail`, which
|
||||
queues it for one background worker: each wake takes at most Worker threads files and
|
||||
writes them on that many threads; a queued file is not queued twice; the queue holds
|
||||
10,000 and a file left out is made on view.
|
||||
- A finished scan that added files counts the missing previews of the sources it
|
||||
walked: 200 or more start the thumbnail job (missing only, visible and cancellable in
|
||||
Activity, never silent bulk work); fewer go through the worker.
|
||||
- Holds: automatic previews write only `thumbnails/` files and no records, so they run
|
||||
beside a scan, image tagging, the thumbnail job and the Needs review recording (S5
|
||||
M7). They wait (a view gets the original, a queued file is left for its view) while a
|
||||
restore, a duplicate removal, a sources change, the gallery's thumbnail clear or an
|
||||
`exclusive` job runs (`_auto_thumbnail_blocker`), checked when a write starts (after
|
||||
its slot) and again just before its rename.
|
||||
- Shutdown is bounded and logged: the worker stops at a file boundary, ffmpeg is
|
||||
killed, an idle worker leaves (the next output starts a fresh one), the view pool
|
||||
shuts down, and after the app's cleanup a late monitor event
|
||||
starts nothing. A package reload stops the workers the previous import left.
|
||||
- Cards and strips load previews only: Images in every layout and size, Library cards,
|
||||
Needs review cards and the inspector's cover (a video's cover is its preview frame).
|
||||
The media viewer's main media and downloads load the original
|
||||
(`workspace-thumbnails.spec.mjs` records every media request; `tests/test_thumbnail_auto.py`).
|
||||
|
||||
Thumbnail format and settings (#317, `py/api/thumbnail_settings.py`,
|
||||
`py/api/thumbnail_encode.py`, `py/api/thumbnail_convert.py`):
|
||||
|
||||
- Server settings `thumbnail_format` (`webp` default, `jxl`, `jpeg`),
|
||||
`thumbnail_quality` (1–100, default 80) and `thumbnail_edge` (128–2048 px, default
|
||||
512) live in config.json's `thumbnails` section and are saved through `POST
|
||||
/prompt_manager/settings` with its revision; a bad value is a 400 and changes
|
||||
nothing. The legacy `gallery.web_interface.thumbnail_size` (never read; older saves
|
||||
wrote its default 256) is ignored. `GET /prompt_manager/settings` also reports
|
||||
`thumbnail_encoders` (`jxl: {available, encoder, reason}`). One module reads the
|
||||
settings; every writer uses it: automatic previews (capture and first view), the
|
||||
thumbnail job, the history route, and the legacy gallery's stream and blocking
|
||||
generate (`tests/test_thumbnail_format.py`).
|
||||
- Names: `thumbnails/<stem>_thumb.webp|.jxl|.jpg` for images and videos alike. Every
|
||||
lookup (listings, duplicates, enrichment, the autotag scan, the history route) takes
|
||||
the first existing preview the file owns, the configured format first: an older
|
||||
release's `_thumb.png` (or `.jpeg`, `.gif`, the source's own spelling or the
|
||||
upper-case one) keeps showing, and counts as outdated, until it is converted. History's
|
||||
cover uses the same lookup as the Images listing. Two files with one stem in a folder
|
||||
share one preview name, made from the first source in `.png`, `.jpg`, `.jpeg`,
|
||||
`.webp`, `.gif`, then videos. A request for an older name answers the current preview. Previews
|
||||
are served with an explicit `image/webp` / `image/jxl` type (the platform's mimetypes
|
||||
table may not know them) and the same cache headers; `.jxl` is served from
|
||||
`thumbnails/` only.
|
||||
- Every preview is written by `write_thumbnail`: the first frame (animated sources give
|
||||
a still), RGB, fitted inside the edge, encoded into a hidden temp sibling that is
|
||||
closed, verified to decode (a JPEG XL file Pillow cannot read is checked by its
|
||||
signature) and renamed over the preview; a failure keeps the old file. ffmpeg writes
|
||||
with `-update 1`, so a `%` in a name is never a sequence pattern, and reads a video as
|
||||
`file:<path>`. Temp files a crash left (`.<stem>.part-<pid>-<thread><ext>`) are
|
||||
removed by the clear and, when older than two minutes, by a sweep at startup (which
|
||||
also removes one carrying its own PID that predates the process: containers reuse
|
||||
PIDs).
|
||||
- JPEG XL needs no new dependency: Pillow (natively or with `pillow_jxl`, imported when
|
||||
installed), else ffmpeg with `libjxl`, else `cjxl`, detected once. Quality maps to
|
||||
libjxl's distance (`0.1 + (100 − q) × 0.09` from 30 up: q80 → 1.9, q85 → 1.45; never
|
||||
lossless). Without an encoder, saving `jxl` is a 400 `jxl-unavailable` with the
|
||||
reason. An encoder that fails because its binary or module is gone makes that write
|
||||
detect again once: another encoder found is used, none at all writes the file as WebP
|
||||
(logged once) and refreshes the availability Appearance reads; any other failure
|
||||
raises without a probe. "No encoder" is probed again on the settings read, on a
|
||||
`jxl` save and after 60 s, so one slow probe never disables JPEG XL until a restart; if JPEG XL has no encoder, previews are written as WebP
|
||||
and count as the format written. Encoder processes (and ffmpeg's frames) are tracked,
|
||||
bounded by a 30 s timeout and killed at the app's cleanup, after which none starts.
|
||||
- A preview in another format than the one written now is *outdated*.
|
||||
`/images/thumbnails/status` reports `pending` (missing), `outdated` and `ready`; the
|
||||
missing job never writes a second preview beside an outdated one. Convert outdated
|
||||
(`POST /jobs/thumbnails {mode: "convert"}`, "Convert outdated thumbnails") runs under
|
||||
the thumbnail job's kind, guarded admission, worker threads and cancel: for each
|
||||
outdated file it writes the new preview from the source (a video with no decodable
|
||||
frame fails instead of getting the placeholder; a current preview it keeps must
|
||||
decode), then removes that file's old-format previews (regular files in its
|
||||
`thumbnails/` folder that the source owns, the source still present; one already gone
|
||||
counts as removed, and a path that is the new preview under another spelling is never
|
||||
removed). Files sharing a preview name are converted once, as a group, from
|
||||
the first source above, and each file's bytes count once. A failure keeps the old
|
||||
preview; orphans (source gone) and links are never touched. While it runs, a change of
|
||||
the thumbnail format, quality or size limit is a 409 `busy` naming it, and should the
|
||||
format still change underneath, the job stops at the next file without removing
|
||||
anything ("Stopped: the thumbnail format changed …"). Its final line is "N converted · F
|
||||
failed · <before> → <after>"
|
||||
(`tests/test_thumbnail_convert.py`, "Convert outdated replaces older-format previews"
|
||||
in `workspace-system-sources.spec.mjs`). Nothing converts by itself: Appearance's
|
||||
format change only makes previews outdated.
|
||||
- A thumbnail is only a grid or cover placeholder. Inspection refuses any path under a
|
||||
root's `thumbnails/` (403) and always parses the original; the media viewer, its
|
||||
download, History's viewer and sending use the original
|
||||
(`TestThumbnailsAreOnlyPlaceholders`, `the viewer and its download always use the
|
||||
original` in `workspace-media-viewer.test.js`). The inspector cover accepts the
|
||||
listing's preview in any format the file owns (`inspection-state.mjs`).
|
||||
- The workspace decodes a 1×1 embedded JPEG XL once (`thumbnail-support.mjs`); where it
|
||||
fails, cards and the inspector cover load the original for a `.jxl` preview, and any
|
||||
preview `<img>` that fails to load falls back to the original (the legacy gallery
|
||||
already does). A video whose preview it cannot show keeps its own element (Images) or
|
||||
its "Video preview" label (Library), never a broken image. History's cover URL names
|
||||
no format, so the server serves a `.jxl` preview only when the URL carries `jxl=1`
|
||||
(added by the workspace where its probe decoded JPEG XL: Firefox may not list it in
|
||||
Accept; any value but `0` or `1` is a 400) or the request's `Accept` lists
|
||||
`image/jxl`; otherwise the file's preview in another format, else the original
|
||||
image or a video's placeholder. Appearance says when this browser cannot show JPEG XL thumbnails.
|
||||
|
||||
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
|
||||
visit through `latestOnly()` on the section scope; a late answer after the section left
|
||||
@@ -2257,7 +2399,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged
|
||||
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
|
||||
reader can keep it byte-identical (`test_health_check_never_writes`).
|
||||
|
||||
Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D128 assigned):
|
||||
Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D131 assigned):
|
||||
|
||||
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
|
||||
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
|
||||
@@ -2277,8 +2419,11 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
|
||||
"Backup status" reads "No retained backup record." until verified backups exist (S8).
|
||||
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
|
||||
missing thumbnails", "N missing thumbnails" (the pending count of
|
||||
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
|
||||
opens the thumbnail dialog (D99). "One display configuration" says "Grid
|
||||
`/images/thumbnails/status`, followed by " · M outdated" when previews are in an
|
||||
older format and " · K failed" when automatic previews failed this session) or
|
||||
"Thumbnail status unavailable"; Review thumbnails opens the thumbnail dialog (D99),
|
||||
and while previews are outdated a second button, Convert outdated, opens it on that
|
||||
scope (D131). "One display configuration" says "Grid
|
||||
and density settings are shared across views." (no paging). The sources row is
|
||||
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
|
||||
by "M of N configured sources available." or "None configured is available; ComfyUI's
|
||||
@@ -2331,9 +2476,12 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
|
||||
saved yet" with Remove only until Save; a configured source the server cannot use
|
||||
reads "<path> · <server reason>" with Remove only. A line under the list says when the
|
||||
list is not the whole story: "No source is configured. After saving, ComfyUI’s output
|
||||
folder is used." (the last one removed), "No source is configured; ComfyUI’s output
|
||||
folder is used." and "No configured source is available; ComfyUI’s output folder is
|
||||
used." (auto-detected rows, listed without Remove). "restart pending" is the third
|
||||
folder is used." (the last one removed) and "No configured source is available;
|
||||
ComfyUI’s output folder is used." (every configured source unusable). Auto-detected
|
||||
rows are listed without Remove and say they are the default on the row itself:
|
||||
"ComfyUI/output · watching · default" ("ComfyUI/" only when `in_comfyui`; their scan
|
||||
still sends the server's public path), with no line under the list when nothing is
|
||||
configured. "restart pending" is the third
|
||||
watch state. The operation "Refresh files" ("Read every source again to find files
|
||||
added or removed outside ComfyUI.", button "Refresh") closes the surface: it calls the
|
||||
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
|
||||
@@ -2343,8 +2491,8 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
|
||||
maintenance") with production copy. A `p.small[role=status]` line before the actions
|
||||
says what the server's read-only plan (`GET /images/thumbnails/plan`) counts for the
|
||||
chosen scope: "N previews to generate · M already up to date." or, for Rebuild all,
|
||||
"Every preview is regenerated: N files.", with "· R more on the next run" past the
|
||||
per-run cap; "Counting previews…" while it is read and "The previews could not be
|
||||
"Every preview is regenerated: N files.", N counting the files past the per-batch
|
||||
cap too (#317); "Counting previews…" while it is read and "The previews could not be
|
||||
counted." when the read fails (Start still starts). Each scope is read once per
|
||||
dialog, since size and quality change no count. The primary is "Start" (the
|
||||
concept's "Simulate thumbnail job"). Overview's Review thumbnails opens the same
|
||||
@@ -2404,6 +2552,44 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D
|
||||
`Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save
|
||||
refused for one unusable source names it and marks its row`
|
||||
(workspace-system-sources.spec) and the system-state and system-sources node tests.
|
||||
- **D129** (library location, S6b): the concept shows neither where the library lives
|
||||
nor a way to move it. System › Overview's "Library health" gains a "Library location"
|
||||
`.operation` row after Backup status, in that row's own markup: "<path> · <why>",
|
||||
plus "You chose to keep it here." and any `PROMPT_MANAGER_DATA_DIR` problem, with
|
||||
"Move library" for a node-folder library and "Details" (a `dl.meta-grid` dialog)
|
||||
otherwise. While a node-folder library is offered (`location.offer`), the shell shows
|
||||
the concept's `.notice` (the Sources notice's markup: `div > h2 + p`, then
|
||||
`.row-actions` with "Keep it here" and the `ca-primary` "Move library") in
|
||||
`#library-banner` above `main`, in main's own width and padding, with no bottom margin
|
||||
(main's padding spaces it); `workspace.css` carries the `/* D129 */` rules. Move
|
||||
library opens the shared review (From, To, Size, Moves with it, Old files). Pinned by
|
||||
`system shell and overview match the concept` (the mapping inserts the row) and
|
||||
`library notice matches the concept notice` with its mutation check
|
||||
(system-concept.spec), and the D129 tests of workspace-system.spec.
|
||||
- **D130** (Appearance, #317): the concept keeps "Thumbnail quality, size limit" in
|
||||
Appearance (README) but draws no rows for them. Appearance gains a second
|
||||
`section.surface`, "Thumbnails", after "Browsing & display", built from the concept's
|
||||
`.setting-row` markup: "Thumbnail format" (WebP / JPEG XL / JPEG), "Thumbnail
|
||||
quality" (1–100), "Thumbnail size limit" (128–2048 px), all host settings saved with
|
||||
the revision, then the operation "Convert outdated thumbnails". JPEG XL is not in the
|
||||
concept: it is offered only when the server found an encoder, else the option is
|
||||
disabled and the row shows the server's reason ("No JPEG XL encoder found. Install
|
||||
pillow-jxl-plugin …"); a row note also says when this browser cannot show JPEG XL
|
||||
thumbnails. After a save that changed the format, the operation's line reads "Format
|
||||
saved. Existing previews keep their old format until you convert them." Pinned by the
|
||||
appearance comparison (the mapping inserts the surface), `Appearance › Thumbnails
|
||||
saves the server thumbnail settings` (workspace-system.spec) and
|
||||
`workspace-system-thumbnails.test.js`.
|
||||
- **D131** (Thumbnail maintenance and Overview, #317): the dialog's Quality and Preview
|
||||
size limit fields are replaced by a `p.small` after the plan line naming the server's
|
||||
settings ("Previews are written as WebP · quality 80 · 512 px. Change this in System ›
|
||||
Appearance."), since previews follow those settings everywhere; Scope gains "Convert
|
||||
outdated (removes older formats)", whose plan line reads "N previews in an older format
|
||||
to convert." Overview's thumbnail row reads "N missing · M outdated · K failed" and,
|
||||
while previews are outdated, adds a second button, Convert outdated, that opens the
|
||||
dialog on that scope. Pinned by the thumbnail dialog comparison (mapped) and its
|
||||
mutation check (system-concept.spec) and `the thumbnail row counts outdated previews`
|
||||
(workspace-system.spec).
|
||||
- **D124** (inspector): with "Load metadata on selection" off, the Images panel and
|
||||
the viewer's details show "Metadata loads when you ask for it." and a "Load metadata"
|
||||
button instead of reading the file. The concept has no such state, so no concept
|
||||
@@ -2775,6 +2961,116 @@ Restoring a deduped backup on a version before this one is not supported: that
|
||||
version's restore accepts the file and then reads rewritten images without their
|
||||
workflow (README changelog).
|
||||
|
||||
### Library location contract (#249 System S6b)
|
||||
|
||||
`utils/library_location.py` is the one resolver of every data path: the database
|
||||
(`database/operations.py` `_resolve_db_path`, diagnostics), config.json
|
||||
(`PromptManagerConfig.get_config_path`, so the `ConfigStore`), the backups (beside the
|
||||
database), the logs (`utils/logging_config.py`) and the LoRA image cache
|
||||
(`py/lora_utils.py`). Order: a valid `PROMPT_MANAGER_DATA_DIR` (absolute, never inside
|
||||
ComfyUI's output, input or temp folder, a public user folder, an extra gallery root or
|
||||
an image source listed in any config.json the resolver may read; `config_location` and
|
||||
`resolve_library` read those sources the same way, so config.json and the database
|
||||
never split; an invalid one is ignored and reported as a problem code), used when
|
||||
it holds a library or no node-folder library exists; an explicit
|
||||
`database.default_path` other than `prompts.db` (relative to the node folder, as
|
||||
before) and `PROMPT_MANAGER_CONFIG_PATH`, honoured as before; ComfyUI's
|
||||
`user/__promptmanager` (`folder_paths.get_system_user_directory("promptmanager")`, or
|
||||
`get_user_directory()/__promptmanager`) once its marker `library-location.json` says
|
||||
`moved`; a node-folder `prompts.db` that was not moved; else `user/__promptmanager`
|
||||
(new installs; outside ComfyUI the node folder). An empty environment folder never
|
||||
replaces a node-folder library (`data-dir-empty`). Nothing is moved or copied at
|
||||
startup (`tests/test_library_location.py`, `tests/test_library_location_wiring.py`).
|
||||
|
||||
`POST /prompt_manager/system/library/move` (`py/api/library_location.py`,
|
||||
`database/library_move.py`) starts "Move library": kind `library_move`, admission
|
||||
`exclusive`, not cancellable. Refusals answer before any job: 400 `not-movable`,
|
||||
`same-folder`, `target-invalid`; 409 `target-has-library`, `target-conflict`, `busy`;
|
||||
507 `insufficient-space` (the free space of the target's volume, for the database, its
|
||||
backups and config.json plus 16 MiB). The job copies with SQLite's backup API from one
|
||||
read transaction (counts, document digest and `data_version` of that snapshot) into a
|
||||
`.pm-move-*` temp file in the target folder (`journal_mode=DELETE`, fsync, 0600),
|
||||
verifies it (`verify_database_file`, every workflow document, the same rows in every
|
||||
table, the same documents), copies every `prompts.db.backup_*` and
|
||||
`prompts.db.pre-workflow-storage_*` (times kept, size and SHA-256 checked), then, under
|
||||
the old file's `BEGIN IMMEDIATE` and the config lock, rechecks `data_version` (a write
|
||||
meanwhile copies again, up to three times, then `library-busy`), copies config.json
|
||||
(0600, checked), writes the marker `pending`, gives every file its final name with
|
||||
`os.replace` inside the target folder (a different drive is a copy, never a rename),
|
||||
writes the marker `moved` and switches the process (`switch_process`: the resolver's
|
||||
override, `PromptModel.repoint_instances` closes every connection of every model of the
|
||||
old file and points it at the new one, the review and history observers are re-keyed,
|
||||
the API's plans and health report are dropped). Only then are the old `prompts.db` and
|
||||
`config.json` renamed `*.moved-to-<folder>` after a `wal_checkpoint(TRUNCATE)`; a file
|
||||
still open elsewhere is left in place and named (`old-in-use`, `old-locked`); the old
|
||||
backups stay. A failure before the marker says `moved` removes everything it wrote and
|
||||
leaves the old library in use, untouched (`TestFailures`); a crash leaves the old
|
||||
library in use (no or a `pending` marker; the next move replaces its own leftovers) or
|
||||
the new one (`moved`) (`TestCrashWindows`). `GET /prompt_manager/system/library` and the
|
||||
Overview's `location` give `{path, source, movable, keep, offer, target, problem,
|
||||
size_bytes}` with paths relative to ComfyUI; a path in the node folder reads through
|
||||
the node folder's own unresolved place in custom_nodes (`custom_nodes/promptmanager/
|
||||
prompts.db` even when that folder is a symlink to a checkout outside ComfyUI; never
|
||||
absolute, never `..`, else the shared public path, which sources and jobs keep using
|
||||
unchanged); `POST .../keep {keep}` persists
|
||||
`library.keep_in_node_folder` through the `ConfigStore`. Errors are coded and name no
|
||||
path (`tests/test_api_library_location.py`). The real-host `library-location.spec.mjs`
|
||||
(execution suite, its own fresh host on an OS-chosen port, no configuration) proves a new
|
||||
install creates `user/__promptmanager/prompts.db` and its logs, adds nothing but byte
|
||||
code to the node folder, and that `/userdata` serves none of it.
|
||||
|
||||
The switch runs while the copy is write-locked too (a guard transaction on it), so a
|
||||
switch that fails at any point (the resolver, part of the models, the observers) is
|
||||
undone whole: every model points at the old file again, the copy and its marker are
|
||||
removed, the old file's connections the partial switch detached are closed (the models
|
||||
reopen it on next use), and the move fails "… Nothing was moved."
|
||||
(`TestFailedSwitchIsUndone`). A failed switch can stall up to the 30 s busy timeout when
|
||||
a writer is blocked on the copy's guard lock; this is not a deadlock, and the writer
|
||||
then retries on the old file. What
|
||||
follows a complete switch (dropping plans, the health report, the history read fence,
|
||||
which is closed first) is best effort: logged, never a failed move. A thread that
|
||||
opens a connection while the model is being repointed keeps it only if the model's
|
||||
generation did not change meanwhile (`PromptModel.get_connection`).
|
||||
|
||||
Writes racing the switch: a background write that was waiting on the old file's lock
|
||||
when the switch closed its connection fails, and run counting (`count_runs`,
|
||||
`count_pair_runs`) and the image monitor's link (`ImageGenerationHandler.
|
||||
link_image_to_prompt`: the resolved-pair ingest and the fallback or recent-prompt link)
|
||||
run it once more through `PromptDatabase.survive_library_switch` when the model's
|
||||
generation changed during the attempt; the retry opens the new library. The monitor
|
||||
links through `PromptDatabase.link_image_to_prompt_or_raise`, which raises database
|
||||
errors (`link_image_to_prompt` keeps its log-and-return-0 contract for other callers);
|
||||
each attempt is one transaction, so a failed attempt rolls back and a committed one
|
||||
does not retry, and the image is linked once. `TestBackgroundWritesSurviveTheSwitch`
|
||||
races a real UPDATE (run counting) and a real fallback image link against the lock and
|
||||
asserts each lands in the new file only. Any other failure is not retried. Known limit: that
|
||||
blocked writer, and the mover with it, waits out the writer's busy timeout (30 s)
|
||||
before its connection closes: SQLite holds the connection's mutex during the busy
|
||||
wait, so neither a close nor `Connection.interrupt()` ends it sooner (measured), and
|
||||
another thread cannot shorten the timeout of a connection in use. Other writes are
|
||||
unaffected: threads that ask for a connection after the switch get the new file.
|
||||
Writes from the HTTP routes (a prompt saved in the dashboard at that moment) are not
|
||||
retried and report their error.
|
||||
|
||||
A restart after a crash between the switch and the rename finishes the rename: the
|
||||
marker names the old `prompts.db` and `config.json` (`from`, `config_from`), and the API
|
||||
renames them `*.moved-to-<folder>` at startup when they are still there under their
|
||||
own names, never deleting them and leaving a file still in use alone
|
||||
(`TestRestartFinishesTheRename`). A second ComfyUI process using the same node folder
|
||||
(two installs sharing one custom_nodes folder) is not switched: it keeps writing to the
|
||||
old file, renamed or not, and those writes are not in the moved library; a rename while
|
||||
it has the file open is refused as `old-in-use`. The Overview flags a library that stays
|
||||
in the node folder through an explicit `database.default_path` or a node-folder
|
||||
config.json (`inside-node-folder`: a reinstall deletes it, CivitAI key included); it is
|
||||
not offered a move.
|
||||
|
||||
Windows: every connection to the old file is closed before its rename (the source
|
||||
snapshot connection, every model's connections, the checkpoint connection); temp files
|
||||
are closed before `os.replace`; renames stay inside one folder; folder identity compares
|
||||
`(st_dev, st_ino)` and `normcase(realpath)`, so case variants and links of a served
|
||||
folder are refused. The browser test server closes its own log handlers before its temp
|
||||
root (which now holds the logs) is removed.
|
||||
|
||||
## Workspace data fields for the concept pages (#249)
|
||||
|
||||
These fields are additive. The legacy `/prompt_manager/images/output` listing (no
|
||||
|
||||
+17
-1
@@ -87,6 +87,15 @@ try:
|
||||
routes = config.routes
|
||||
api = PromptManagerAPI()
|
||||
api.add_routes(routes)
|
||||
# Temp files of preview writes a crash interrupted (#317), off the
|
||||
# import path: the walk may cover a large thumbnails folder.
|
||||
import threading
|
||||
|
||||
threading.Thread(
|
||||
target=api.sweep_thumbnail_temps,
|
||||
name="PromptManagerPreviewTempSweep",
|
||||
daemon=True,
|
||||
).start()
|
||||
host_app = getattr(config.server_instance, "app", None)
|
||||
if host_app is not None and hasattr(host_app, "on_cleanup"):
|
||||
host_app.on_cleanup.append(api.cleanup_history_reads)
|
||||
@@ -132,11 +141,18 @@ try:
|
||||
_review_api = globals().get("api")
|
||||
if _review_api is not None:
|
||||
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
|
||||
# Every settled output's preview is queued for the API's background
|
||||
# worker (#249): the workspace never asks to generate thumbnails.
|
||||
_global_image_monitor.set_thumbnail_queue(_review_api.queue_capture_thumbnail)
|
||||
_review_host = getattr(config.server_instance, "app", None)
|
||||
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
|
||||
|
||||
async def _stop_review_recording(app, monitor=_global_image_monitor):
|
||||
async def _stop_review_recording(
|
||||
app, monitor=_global_image_monitor, api=_review_api
|
||||
):
|
||||
monitor.set_review_recorder(None)
|
||||
monitor.set_thumbnail_queue(None)
|
||||
api.close_auto_thumbnails(final=True)
|
||||
|
||||
_review_host.on_cleanup.insert(0, _stop_review_recording)
|
||||
|
||||
|
||||
@@ -196,3 +196,14 @@ class ExecutionHistoryOperations:
|
||||
.fetchone()
|
||||
)
|
||||
return dict(row) if row is not None else None
|
||||
|
||||
|
||||
def rekey_history_restore(old_path: str, new_path: str) -> None:
|
||||
"""Move the observers of ``old_path`` to ``new_path`` (Move library)."""
|
||||
old, new = _database_identity(old_path), _database_identity(new_path)
|
||||
if old == new:
|
||||
return
|
||||
with _restore_lock:
|
||||
observers = _restore_observers.pop(old, None)
|
||||
if observers:
|
||||
_restore_observers.setdefault(new, {}).update(observers)
|
||||
|
||||
@@ -0,0 +1,805 @@
|
||||
"""Move library (#249 S6b): copy the node-folder library, verify it, switch.
|
||||
|
||||
``plan_move`` checks a ``utils.library_location`` location can move and
|
||||
sizes the copy; ``move_library`` does it, in this order:
|
||||
|
||||
1. **Disk check** on the target's volume (the copy, the backups and
|
||||
config.json, plus a margin). The source is never renamed across folders,
|
||||
so another drive is simply a copy.
|
||||
2. **Copy** with SQLite's backup API from a private connection holding one
|
||||
read transaction: the per-table counts, the workflow documents' digest
|
||||
and ``PRAGMA data_version`` describe exactly the copied snapshot. The copy
|
||||
is a temp file in the target folder, made self-contained
|
||||
(``journal_mode=DELETE``), fsynced and owner-only.
|
||||
3. **Verify**: ``verify_database_file`` (integrity check, schema, every
|
||||
workflow document decompressed against its hash, #315), the same row
|
||||
count in every table and the same documents.
|
||||
4. **Copy files**: every pooled safety backup and pre-migration backup,
|
||||
checked by size and SHA-256 after the copy (their times are kept, so the
|
||||
pool prunes in the same order).
|
||||
5. **Publish** under the old database's write lock (``BEGIN IMMEDIATE``) and
|
||||
the config.json lock: if any other connection wrote since the snapshot
|
||||
(``data_version``), steps 2–4 run again (``attempts``, then
|
||||
``library-busy``). Else config.json is copied (owner-only, checked), a
|
||||
``pending`` marker is written, the temp files take their final names
|
||||
(``os.replace`` inside the target folder), the marker turns ``moved``
|
||||
and ``switch`` points the process at the new library while the old file
|
||||
is still locked, so no write can land in it.
|
||||
6. **Retire**: the old file's WAL is checkpointed and the old ``prompts.db``
|
||||
and ``config.json`` are renamed ``*.moved-to-<folder>`` (never deleted;
|
||||
the old backups stay where they are). A file still open elsewhere is left
|
||||
in place and reported (``old-in-use``): the marker already makes a restart
|
||||
use the new library.
|
||||
|
||||
Any failure before the marker says ``moved`` removes everything the move
|
||||
wrote in the target folder and leaves the old library in use, untouched.
|
||||
A crash at any point leaves either the old library in use (the marker is
|
||||
absent or ``pending``; the next move replaces its own leftovers) or the new
|
||||
one (``moved``). ``MoveFailed`` carries a code and a message without paths.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import time
|
||||
import types
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Dict, List, Optional, Tuple
|
||||
|
||||
from .models import BUSY_TIMEOUT_SECONDS, PromptModel
|
||||
|
||||
try:
|
||||
from ..utils import library_location as ll
|
||||
from ..utils.logging_config import get_logger
|
||||
except ImportError:
|
||||
from utils import library_location as ll
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("prompt_manager.database.library_move")
|
||||
|
||||
TEMP_PREFIX = ".pm-move-"
|
||||
MOVED_INFIX = ".moved-to-"
|
||||
DISK_MARGIN_BYTES = 16 * 2**20
|
||||
DEFAULT_ATTEMPTS = 3
|
||||
_COPY_CHUNK = 4 * 2**20
|
||||
# Backups that live beside the database and move with it.
|
||||
BACKUP_INFIXES = (".backup_", ".pre-workflow-storage_")
|
||||
|
||||
MESSAGES = {
|
||||
"not-movable": (
|
||||
"This library is not in the node folder, so there is nothing to move."
|
||||
),
|
||||
"same-folder": "The new folder is the node folder itself.",
|
||||
"target-invalid": "The new folder cannot hold the library.",
|
||||
"target-has-library": (
|
||||
"The new folder already holds a Prompt Manager library. Nothing was moved."
|
||||
),
|
||||
"target-conflict": (
|
||||
"The new folder already holds files with the same names. Nothing was moved."
|
||||
),
|
||||
"insufficient-space": (
|
||||
"There is not enough free space for the copy. Nothing was moved."
|
||||
),
|
||||
"copy-failed": "The copy could not be written or did not match. Nothing was moved.",
|
||||
"verify-failed": "The copy did not pass verification. Nothing was moved.",
|
||||
"library-busy": (
|
||||
"Prompt Manager kept writing to the library while it was copied. "
|
||||
"Nothing was moved; try again when no workflow is running."
|
||||
),
|
||||
"switch-failed": (
|
||||
"Prompt Manager could not switch to the verified copy, so the copy was "
|
||||
"removed. Nothing was moved."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
class MoveFailed(Exception):
|
||||
"""A move that did not happen (or did not finish), with a coded reason."""
|
||||
|
||||
def __init__(self, code: str, message: Optional[str] = None):
|
||||
self.code = code
|
||||
self.message = message or MESSAGES.get(code, "The library was not moved.")
|
||||
super().__init__(self.message)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MovePlan:
|
||||
location: "ll.LibraryLocation"
|
||||
db_path: str
|
||||
source_dir: str
|
||||
target_dir: str
|
||||
config_path: Optional[str]
|
||||
backups: Tuple[str, ...]
|
||||
db_bytes: int
|
||||
files_bytes: int
|
||||
|
||||
@property
|
||||
def required_bytes(self) -> int:
|
||||
return self.db_bytes + self.files_bytes + DISK_MARGIN_BYTES
|
||||
|
||||
@property
|
||||
def target_location(self) -> "ll.LibraryLocation":
|
||||
"""Where the library is once moved (an explicit config.json stays)."""
|
||||
source = self.location.target_source or ll.SOURCE_DEFAULT
|
||||
config = (
|
||||
self.location.config_path
|
||||
if self.location.config_explicit
|
||||
else os.path.join(self.target_dir, ll.CONFIG_NAME)
|
||||
)
|
||||
return ll.LibraryLocation(
|
||||
source,
|
||||
os.path.join(self.target_dir, ll.DATABASE_NAME),
|
||||
config,
|
||||
self.target_dir,
|
||||
self.target_dir,
|
||||
None,
|
||||
source,
|
||||
self.location.config_explicit,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MoveResult:
|
||||
location: "ll.LibraryLocation"
|
||||
counts: Dict[str, int]
|
||||
files: List[str]
|
||||
renamed: List[str] = field(default_factory=list)
|
||||
retire_problem: Optional[str] = None
|
||||
|
||||
|
||||
def _size(path: str) -> int:
|
||||
try:
|
||||
return os.path.getsize(path)
|
||||
except OSError:
|
||||
return 0
|
||||
|
||||
|
||||
def _free_bytes(path: str) -> int:
|
||||
return shutil.disk_usage(path).free
|
||||
|
||||
|
||||
def _existing_ancestor(path: str) -> str:
|
||||
current = os.path.abspath(path)
|
||||
while not os.path.isdir(current):
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
break
|
||||
current = parent
|
||||
return current
|
||||
|
||||
|
||||
def backup_names(db_path: str) -> Tuple[str, ...]:
|
||||
"""The pooled and pre-migration backups beside ``db_path`` (names)."""
|
||||
folder = os.path.dirname(os.path.abspath(db_path))
|
||||
base = os.path.basename(db_path)
|
||||
prefixes = tuple(base + infix for infix in BACKUP_INFIXES)
|
||||
try:
|
||||
names = os.listdir(folder)
|
||||
except OSError:
|
||||
return ()
|
||||
return tuple(
|
||||
sorted(
|
||||
name
|
||||
for name in names
|
||||
if name.startswith(prefixes) and os.path.isfile(os.path.join(folder, name))
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def plan_move(location) -> MovePlan:
|
||||
"""Check ``location`` can move to its target; raises MoveFailed."""
|
||||
if not location.movable:
|
||||
raise MoveFailed("not-movable")
|
||||
target = os.path.abspath(location.target_dir)
|
||||
source_dir = os.path.dirname(os.path.abspath(location.db_path))
|
||||
if ll.same_folder(target, source_dir) or os.path.normcase(
|
||||
os.path.realpath(target)
|
||||
) == os.path.normcase(os.path.realpath(source_dir)):
|
||||
raise MoveFailed("same-folder")
|
||||
if os.path.exists(target) and not os.path.isdir(target):
|
||||
raise MoveFailed("target-invalid")
|
||||
backups = backup_names(location.db_path)
|
||||
# config.json moves with a node-folder library; one named by
|
||||
# PROMPT_MANAGER_CONFIG_PATH is the user's and stays where it is.
|
||||
config = None if location.config_explicit else location.config_path
|
||||
config = config if config and os.path.isfile(config) else None
|
||||
if ll.read_marker(target).get("state") != ll.MARKER_PENDING:
|
||||
if os.path.exists(os.path.join(target, ll.DATABASE_NAME)):
|
||||
raise MoveFailed("target-has-library")
|
||||
names = [ll.CONFIG_NAME] if config else []
|
||||
if any(os.path.exists(os.path.join(target, n)) for n in names + list(backups)):
|
||||
raise MoveFailed("target-conflict")
|
||||
db_bytes = _size(location.db_path) + _size(location.db_path + "-wal")
|
||||
files_bytes = sum(_size(os.path.join(source_dir, n)) for n in backups)
|
||||
files_bytes += _size(config) if config else 0
|
||||
return MovePlan(
|
||||
location,
|
||||
os.path.abspath(location.db_path),
|
||||
source_dir,
|
||||
target,
|
||||
config,
|
||||
backups,
|
||||
db_bytes,
|
||||
files_bytes,
|
||||
)
|
||||
|
||||
|
||||
def space_problem(plan: MovePlan) -> Optional[Tuple[int, int]]:
|
||||
"""``(required, free)`` when the target's volume is short, else None."""
|
||||
free = _free_bytes(_existing_ancestor(plan.target_dir))
|
||||
if free < plan.required_bytes:
|
||||
return plan.required_bytes, free
|
||||
return None
|
||||
|
||||
|
||||
# ── copy and verify ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _tables(conn) -> List[str]:
|
||||
return [
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' "
|
||||
"AND name NOT LIKE 'sqlite_%' ORDER BY name"
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _describe(conn) -> Tuple[Dict[str, int], str]:
|
||||
"""Row counts per table and a digest of the workflow documents."""
|
||||
counts = {}
|
||||
for table in _tables(conn):
|
||||
# A name read from sqlite_master, quoted as an identifier.
|
||||
quoted = '"' + table.replace('"', '""') + '"'
|
||||
sql = f"SELECT COUNT(*) FROM {quoted}" # nosec B608
|
||||
counts[table] = conn.execute(sql).fetchone()[0]
|
||||
digest = hashlib.sha256()
|
||||
if "workflow_documents" in counts:
|
||||
for row in conn.execute(
|
||||
"SELECT id, sha256, byte_length FROM workflow_documents ORDER BY id"
|
||||
):
|
||||
digest.update(repr((row[0], bytes(row[1]), row[2])).encode())
|
||||
return counts, digest.hexdigest()
|
||||
|
||||
|
||||
def _fsync_file(path: str) -> None:
|
||||
with open(path, "rb+") as handle:
|
||||
os.fsync(handle.fileno())
|
||||
|
||||
|
||||
def _fsync_dir(path: str) -> None:
|
||||
if os.name != "posix":
|
||||
return
|
||||
fd = os.open(path, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(fd)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def _private(path: str) -> None:
|
||||
if os.name == "posix":
|
||||
os.chmod(path, 0o600)
|
||||
|
||||
|
||||
def _copy_snapshot(source: sqlite3.Connection, temp: str):
|
||||
"""Back up ``source``'s current snapshot to ``temp``; return what it holds."""
|
||||
source.execute("BEGIN")
|
||||
try:
|
||||
version = source.execute("PRAGMA data_version").fetchone()[0]
|
||||
counts, documents = _describe(source)
|
||||
destination = sqlite3.connect(temp)
|
||||
try:
|
||||
source.backup(destination)
|
||||
finally:
|
||||
destination.close()
|
||||
finally:
|
||||
source.execute("COMMIT")
|
||||
with contextlib.closing(sqlite3.connect(temp)) as conn:
|
||||
conn.execute("PRAGMA journal_mode=DELETE").fetchone()
|
||||
_fsync_file(temp)
|
||||
_private(temp)
|
||||
return version, counts, documents
|
||||
|
||||
|
||||
def _verify_copy(temp: str, counts, documents) -> None:
|
||||
ok, reason = PromptModel.verify_database_file(temp)
|
||||
if not ok:
|
||||
logger.error(f"Library move: the copy did not verify: {reason}")
|
||||
detail = reason if reason.startswith("Workflow document") else None
|
||||
raise MoveFailed(
|
||||
"verify-failed",
|
||||
f"{MESSAGES['verify-failed']} {detail}" if detail else None,
|
||||
)
|
||||
uri = Path(os.path.abspath(temp)).as_uri() + "?mode=ro"
|
||||
try:
|
||||
with contextlib.closing(sqlite3.connect(uri, uri=True)) as conn:
|
||||
copied = _describe(conn)
|
||||
except sqlite3.Error as error:
|
||||
logger.error(f"Library move: the copy could not be read: {error}")
|
||||
raise MoveFailed("verify-failed") from error
|
||||
if copied != (counts, documents):
|
||||
logger.error("Library move: the copy's rows or documents differ")
|
||||
raise MoveFailed("verify-failed")
|
||||
|
||||
|
||||
def _hash_copy(source: str, destination: str) -> Tuple[int, str]:
|
||||
"""Copy ``source`` to ``destination`` (fsynced, owner-only, times kept)."""
|
||||
digest = hashlib.sha256()
|
||||
size = 0
|
||||
with open(source, "rb") as reader, open(destination, "wb") as writer:
|
||||
while True:
|
||||
chunk = reader.read(_COPY_CHUNK)
|
||||
if not chunk:
|
||||
break
|
||||
digest.update(chunk)
|
||||
size += len(chunk)
|
||||
writer.write(chunk)
|
||||
writer.flush()
|
||||
os.fsync(writer.fileno())
|
||||
stat = os.stat(source)
|
||||
os.utime(destination, ns=(stat.st_atime_ns, stat.st_mtime_ns))
|
||||
_private(destination)
|
||||
return size, digest.hexdigest()
|
||||
|
||||
|
||||
def _file_digest(path: str) -> Tuple[int, str]:
|
||||
digest = hashlib.sha256()
|
||||
size = 0
|
||||
with open(path, "rb") as reader:
|
||||
while True:
|
||||
chunk = reader.read(_COPY_CHUNK)
|
||||
if not chunk:
|
||||
return size, digest.hexdigest()
|
||||
digest.update(chunk)
|
||||
size += len(chunk)
|
||||
|
||||
|
||||
def _check_copies(copies) -> None:
|
||||
for source, temp, expected in copies:
|
||||
if _file_digest(temp) != expected or _file_digest(source) != expected:
|
||||
logger.error(
|
||||
f"Library move: {os.path.basename(source)} did not copy unchanged"
|
||||
)
|
||||
raise MoveFailed("copy-failed")
|
||||
|
||||
|
||||
def write_marker(
|
||||
directory: str, state: str, source_db: str, source_config: Optional[str] = None
|
||||
) -> str:
|
||||
"""Atomically write the move marker; return its path.
|
||||
|
||||
``from`` and ``config_from`` name the old files, so a restart after a
|
||||
crash between the switch and the rename can finish the rename.
|
||||
"""
|
||||
path = os.path.join(directory, ll.MARKER_NAME)
|
||||
temp = os.path.join(directory, f"{TEMP_PREFIX}{uuid.uuid4().hex}.marker")
|
||||
payload = json.dumps(
|
||||
{
|
||||
"version": 1,
|
||||
"state": state,
|
||||
"from": source_db,
|
||||
"config_from": source_config,
|
||||
"updated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
},
|
||||
indent=2,
|
||||
).encode("utf-8")
|
||||
fd = os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as handle:
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
_private(temp)
|
||||
os.replace(temp, path)
|
||||
except BaseException:
|
||||
_remove(temp)
|
||||
raise
|
||||
_fsync_dir(directory)
|
||||
return path
|
||||
|
||||
|
||||
def _remove(*paths) -> None:
|
||||
for path in paths:
|
||||
try:
|
||||
if path and os.path.lexists(path):
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
logger.warning(f"Library move: could not remove {os.path.basename(path)}")
|
||||
|
||||
|
||||
def sweep_move_temps(directory: str) -> List[str]:
|
||||
"""Remove the temp files an interrupted move left in ``directory``."""
|
||||
removed = []
|
||||
try:
|
||||
names = os.listdir(directory)
|
||||
except OSError:
|
||||
return removed
|
||||
for name in names:
|
||||
if name.startswith(TEMP_PREFIX):
|
||||
_remove(os.path.join(directory, name))
|
||||
removed.append(name)
|
||||
return removed
|
||||
|
||||
|
||||
# ── the move ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _noop(*_args, **_kwargs):
|
||||
return None
|
||||
|
||||
|
||||
def switch_process(old_db: str, location, detached: list) -> None:
|
||||
"""Point this process at ``location``: resolver, every model, observers.
|
||||
|
||||
Adds the old file's connections, detached and still open, to
|
||||
``detached`` as soon as they are detached, so the mover retires them
|
||||
(``PromptModel.retire_detached``) whether or not the rest succeeds.
|
||||
"""
|
||||
from .execution_history import rekey_history_restore
|
||||
from .review_changes import rekey_review_changes
|
||||
|
||||
ll.switch_to(location)
|
||||
detached.extend(PromptModel.repoint_instances(old_db, location.db_path))
|
||||
rekey_review_changes(old_db, location.db_path)
|
||||
rekey_history_restore(old_db, location.db_path)
|
||||
|
||||
|
||||
def unswitch_process(old_db: str, new_db: str) -> list:
|
||||
"""Undo ``switch_process`` (any part of it); returns detached connections."""
|
||||
from .execution_history import rekey_history_restore
|
||||
from .review_changes import rekey_review_changes
|
||||
|
||||
ll.clear_override()
|
||||
detached = PromptModel.repoint_instances(new_db, old_db)
|
||||
rekey_review_changes(new_db, old_db)
|
||||
rekey_history_restore(new_db, old_db)
|
||||
return detached
|
||||
|
||||
|
||||
class _Move:
|
||||
def __init__(self, plan, *, switch, config_lock, progress, checkpoint):
|
||||
self.plan = plan
|
||||
self.switch = switch
|
||||
self.config_lock = config_lock
|
||||
self.progress = progress
|
||||
self.checkpoint = checkpoint
|
||||
self.created_dir = False
|
||||
self.temps: List[str] = []
|
||||
self.published: List[str] = []
|
||||
self.marker: Optional[str] = None
|
||||
self.moved = False
|
||||
# Retired connections still in use; awaited once the locks are free.
|
||||
self.retiring: list = []
|
||||
|
||||
def temp(self, suffix: str) -> str:
|
||||
path = os.path.join(
|
||||
self.plan.target_dir, f"{TEMP_PREFIX}{uuid.uuid4().hex}{suffix}"
|
||||
)
|
||||
self.temps.append(path)
|
||||
return path
|
||||
|
||||
def prepare_target(self) -> None:
|
||||
target = self.plan.target_dir
|
||||
self.created_dir = not os.path.isdir(target)
|
||||
ll.ensure_private_dir(target)
|
||||
sweep_move_temps(target)
|
||||
if ll.read_marker(target).get("state") == ll.MARKER_PENDING:
|
||||
# An earlier move stopped before it switched: its files are ours.
|
||||
for name in (ll.DATABASE_NAME, ll.CONFIG_NAME) + self.plan.backups:
|
||||
_remove(os.path.join(target, name))
|
||||
for suffix in ("-wal", "-shm", "-journal"):
|
||||
_remove(os.path.join(target, ll.DATABASE_NAME + suffix))
|
||||
|
||||
def copy_round(self, source):
|
||||
"""Steps 2–4 once: the verified temp copies and the snapshot version."""
|
||||
self.progress("Copying the database")
|
||||
temp_db = self.temp(".db")
|
||||
version, counts, documents = _copy_snapshot(source, temp_db)
|
||||
self.checkpoint("copied")
|
||||
self.progress("Verifying the copy")
|
||||
_verify_copy(temp_db, counts, documents)
|
||||
self.checkpoint("verified")
|
||||
copies = []
|
||||
for name in self.plan.backups:
|
||||
src = os.path.join(self.plan.source_dir, name)
|
||||
temp = self.temp(".bak")
|
||||
try:
|
||||
expected = _hash_copy(src, temp)
|
||||
except OSError as error:
|
||||
logger.error(f"Library move: copying {name} failed: {error}")
|
||||
raise MoveFailed("copy-failed") from error
|
||||
copies.append((src, temp, expected, name))
|
||||
if copies:
|
||||
self.progress(f"Copied {len(copies)} backup file(s)")
|
||||
self.checkpoint("files")
|
||||
_check_copies([(s, t, e) for s, t, e, _ in copies])
|
||||
return version, counts, temp_db, copies
|
||||
|
||||
def discard_round(self, temp_db, copies) -> None:
|
||||
_remove(temp_db, *[temp for _s, temp, _e, _n in copies])
|
||||
|
||||
def publish(self, temp_db, copies) -> List[str]:
|
||||
"""Under the locks: config.json, marker, final names, marker moved."""
|
||||
target = self.plan.target_dir
|
||||
files = []
|
||||
if self.plan.config_path:
|
||||
temp = self.temp(".config")
|
||||
try:
|
||||
expected = _hash_copy(self.plan.config_path, temp)
|
||||
except OSError as error:
|
||||
raise MoveFailed("copy-failed") from error
|
||||
_check_copies([(self.plan.config_path, temp, expected)])
|
||||
copies = copies + [(self.plan.config_path, temp, expected, ll.CONFIG_NAME)]
|
||||
self.marker = write_marker(
|
||||
target, ll.MARKER_PENDING, self.plan.db_path, self.plan.config_path
|
||||
)
|
||||
self.checkpoint("pending")
|
||||
for _src, temp, _expected, name in copies:
|
||||
final = os.path.join(target, name)
|
||||
os.replace(temp, final)
|
||||
self.published.append(final)
|
||||
files.append(name)
|
||||
final_db = os.path.join(target, ll.DATABASE_NAME)
|
||||
os.replace(temp_db, final_db)
|
||||
self.published.append(final_db)
|
||||
files.append(ll.DATABASE_NAME)
|
||||
_fsync_dir(target)
|
||||
self.checkpoint("published")
|
||||
write_marker(target, ll.MARKER_MOVED, self.plan.db_path, self.plan.config_path)
|
||||
self.moved = True
|
||||
self.checkpoint("marked")
|
||||
return files
|
||||
|
||||
def cleanup(self) -> None:
|
||||
if self.moved:
|
||||
return
|
||||
_remove(*self.temps, *self.published, self.marker)
|
||||
if self.created_dir:
|
||||
try:
|
||||
os.rmdir(self.plan.target_dir)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _unique(path: str) -> str:
|
||||
candidate, number = path, 1
|
||||
while os.path.lexists(candidate):
|
||||
candidate = f"{path}_{number}"
|
||||
number += 1
|
||||
return candidate
|
||||
|
||||
|
||||
def retire_old_files(plan: MovePlan) -> Tuple[List[str], Optional[str]]:
|
||||
"""Rename the old database and config.json; ``(renamed, problem code)``.
|
||||
|
||||
Every connection to the old file must be closed. The WAL is checkpointed
|
||||
first; when it cannot be emptied (another process still has the file
|
||||
open) nothing is renamed (``old-in-use``).
|
||||
"""
|
||||
old = plan.db_path
|
||||
folder = os.path.basename(os.path.normpath(plan.target_dir))
|
||||
try:
|
||||
with contextlib.closing(sqlite3.connect(old, timeout=1)) as conn:
|
||||
busy = conn.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0]
|
||||
except sqlite3.Error as error:
|
||||
logger.warning(
|
||||
f"Library move: the old database could not be checkpointed: {error}"
|
||||
)
|
||||
return [], "old-in-use"
|
||||
if busy or _size(old + "-wal") > 0:
|
||||
return [], "old-in-use"
|
||||
renamed = []
|
||||
try:
|
||||
for path in (old, plan.config_path):
|
||||
if not path or not os.path.isfile(path):
|
||||
continue
|
||||
new_name = _unique(path + MOVED_INFIX + folder)
|
||||
os.rename(path, new_name)
|
||||
renamed.append(os.path.basename(new_name))
|
||||
for suffix in ("-wal", "-shm"):
|
||||
if os.path.isfile(old + suffix) and _size(old + suffix) == 0:
|
||||
os.remove(old + suffix)
|
||||
except OSError as error:
|
||||
logger.warning(f"Library move: the old files could not be renamed: {error}")
|
||||
return renamed, "old-locked"
|
||||
return renamed, None
|
||||
|
||||
|
||||
def finish_interrupted_move(location) -> List[str]:
|
||||
"""Rename old files a crash left between the switch and the rename.
|
||||
|
||||
The marker of the folder in use says ``moved`` and names the old files;
|
||||
an old ``prompts.db`` (and config.json) still there under its own name is
|
||||
renamed ``*.moved-to-<folder>`` now, never deleted. Returns the new
|
||||
names (nothing when there is nothing to finish or the file is in use).
|
||||
"""
|
||||
if location.source not in (ll.SOURCE_DEFAULT, ll.SOURCE_ENVIRONMENT):
|
||||
return []
|
||||
marker = ll.read_marker(location.data_dir)
|
||||
old = marker.get("from")
|
||||
if marker.get("state") != ll.MARKER_MOVED or not isinstance(old, str):
|
||||
return []
|
||||
if not os.path.isfile(old) or os.path.basename(old) != ll.DATABASE_NAME:
|
||||
return []
|
||||
if ll.same_folder(os.path.dirname(old), location.data_dir):
|
||||
return []
|
||||
config = marker.get("config_from")
|
||||
if not (
|
||||
isinstance(config, str)
|
||||
and os.path.basename(config) == ll.CONFIG_NAME
|
||||
and os.path.isfile(config)
|
||||
and os.path.normcase(os.path.abspath(config))
|
||||
!= os.path.normcase(os.path.abspath(location.config_path))
|
||||
):
|
||||
config = None
|
||||
pending = types.SimpleNamespace(
|
||||
db_path=old, config_path=config, target_dir=location.data_dir
|
||||
)
|
||||
renamed, problem = retire_old_files(pending)
|
||||
if problem:
|
||||
logger.warning(f"Library move: the old files are still in use ({problem})")
|
||||
elif renamed:
|
||||
logger.info(f"Library move finished: renamed {', '.join(renamed)}")
|
||||
return renamed
|
||||
|
||||
|
||||
def move_library(
|
||||
plan: MovePlan,
|
||||
*,
|
||||
switch: Callable = switch_process,
|
||||
unswitch: Callable = unswitch_process,
|
||||
after_switch: Callable = _noop,
|
||||
config_lock=None,
|
||||
progress: Callable[[str], None] = _noop,
|
||||
checkpoint: Callable[[str], None] = _noop,
|
||||
attempts: int = DEFAULT_ATTEMPTS,
|
||||
) -> MoveResult:
|
||||
"""Move the library as planned (blocking; see the module docstring).
|
||||
|
||||
``switch(old_db, location, detached)`` points the process at the copy
|
||||
and adds the old connections it detached to ``detached``; if it raises,
|
||||
``unswitch(old_db, new_db)`` undoes it while both files are still
|
||||
locked, every detached connection is closed, the copy is removed and
|
||||
the move fails (``switch-failed``). ``after_switch()`` runs
|
||||
once the switch is final; it is best effort (logged, never fails the move).
|
||||
"""
|
||||
short = space_problem(plan)
|
||||
if short is not None:
|
||||
raise MoveFailed("insufficient-space")
|
||||
lock = config_lock if config_lock is not None else contextlib.nullcontext()
|
||||
move = _Move(
|
||||
plan,
|
||||
switch=switch,
|
||||
config_lock=lock,
|
||||
progress=progress,
|
||||
checkpoint=checkpoint,
|
||||
)
|
||||
move.unswitch = unswitch
|
||||
location = plan.target_location
|
||||
try:
|
||||
move.prepare_target()
|
||||
files, counts = _copy_and_switch(move, location, attempts)
|
||||
except Exception:
|
||||
move.cleanup()
|
||||
raise
|
||||
try:
|
||||
after_switch()
|
||||
except Exception:
|
||||
logger.exception("Library move: a follow-up after the switch failed")
|
||||
from .execution_history import notify_history_restore
|
||||
|
||||
notify_history_restore(location.db_path)
|
||||
progress("Renaming the old files")
|
||||
renamed, problem = retire_old_files(plan)
|
||||
checkpoint("retired")
|
||||
return MoveResult(location, counts, files, renamed, problem)
|
||||
|
||||
|
||||
def _switch_guarded(move: _Move, location) -> list:
|
||||
"""Switch while the copy is locked too; undo it all if any part fails.
|
||||
|
||||
The copy's write lock keeps every repointed thread from writing to it
|
||||
until the switch is whole, so undoing a partial switch loses nothing.
|
||||
Returns the old file's detached connections (still open). On failure
|
||||
every detached connection is retired; those still in use are added to
|
||||
``move.retiring`` for the caller to await once the locks are released.
|
||||
"""
|
||||
plan = move.plan
|
||||
guard = sqlite3.connect(
|
||||
location.db_path, timeout=BUSY_TIMEOUT_SECONDS, isolation_level=None
|
||||
)
|
||||
try:
|
||||
guard.execute("BEGIN IMMEDIATE")
|
||||
detached: list = []
|
||||
try:
|
||||
move.switch(plan.db_path, location, detached)
|
||||
except Exception as error:
|
||||
logger.exception("Library move: switching failed; undoing it")
|
||||
move.moved = False # the copy and its marker are removed
|
||||
# Retire, never wait, here: a connection still in use may be
|
||||
# waiting on the very locks this thread holds.
|
||||
try:
|
||||
undone = move.unswitch(plan.db_path, location.db_path)
|
||||
except Exception:
|
||||
logger.exception("Library move: undoing the switch failed")
|
||||
else:
|
||||
_retire(move, undone)
|
||||
# The old file's connections a partial switch detached: the
|
||||
# models reopen it on next use, so close them now, not at GC.
|
||||
_retire(move, detached)
|
||||
raise MoveFailed("switch-failed") from error
|
||||
move.checkpoint("switched")
|
||||
return detached
|
||||
finally:
|
||||
if guard.in_transaction:
|
||||
guard.execute("ROLLBACK")
|
||||
guard.close()
|
||||
|
||||
|
||||
def _copy_and_switch(move: _Move, location, attempts: int):
|
||||
"""Copy, then switch under both locks; retired connections are awaited.
|
||||
|
||||
A connection that was inside a statement when it was retired is closed
|
||||
by its own thread once the statement ends, which may need the locks
|
||||
released first; the old files are renamed (and a failed copy removed)
|
||||
only after that, since Windows refuses to rename or delete an open file.
|
||||
"""
|
||||
try:
|
||||
return _copy_and_switch_locked(move, location, attempts)
|
||||
finally:
|
||||
try:
|
||||
PromptModel.await_detached(move.retiring, logger)
|
||||
except Exception:
|
||||
logger.exception("Library move: waiting for old connections failed")
|
||||
|
||||
|
||||
def _retire(move: _Move, connections: list) -> None:
|
||||
"""Retire ``connections``, adding those still in use to ``move.retiring``.
|
||||
|
||||
Never raises: a switch that completed must not be reported as failed
|
||||
because one of the old connections could not be retired (logged; the
|
||||
rename then reports the old files as still in use).
|
||||
"""
|
||||
try:
|
||||
move.retiring.extend(PromptModel.retire_detached(connections, logger))
|
||||
except Exception:
|
||||
logger.exception("Library move: retiring the old connections failed")
|
||||
|
||||
|
||||
def _copy_and_switch_locked(move: _Move, location, attempts: int):
|
||||
plan = move.plan
|
||||
for attempt in range(1, attempts + 1):
|
||||
source = sqlite3.connect(
|
||||
plan.db_path, timeout=BUSY_TIMEOUT_SECONDS, isolation_level=None
|
||||
)
|
||||
try:
|
||||
source.execute(f"PRAGMA busy_timeout = {BUSY_TIMEOUT_SECONDS * 1000}")
|
||||
version, counts, temp_db, copies = move.copy_round(source)
|
||||
source.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
if source.execute("PRAGMA data_version").fetchone()[0] != version:
|
||||
move.progress("The library changed during the copy; copying again")
|
||||
move.discard_round(temp_db, copies)
|
||||
continue
|
||||
move.checkpoint("locked")
|
||||
with move.config_lock:
|
||||
files = move.publish(temp_db, copies)
|
||||
detached = _switch_guarded(move, location)
|
||||
# Still holding the old file's lock: a write that was waiting
|
||||
# on it is interrupted and cannot commit, so it fails (and
|
||||
# runs again on the new library) instead of landing in the
|
||||
# old file. Its own thread closes the connection.
|
||||
_retire(move, detached)
|
||||
finally:
|
||||
if source.in_transaction:
|
||||
source.execute("ROLLBACK")
|
||||
return files, counts
|
||||
finally:
|
||||
source.close()
|
||||
raise MoveFailed("library-busy")
|
||||
@@ -0,0 +1,353 @@
|
||||
"""A SQLite connection that is never closed under the thread using it.
|
||||
|
||||
``PromptModel`` gives each thread its own connection, but Move library and
|
||||
``close_all`` must retire every thread's connection from one thread. Closing
|
||||
a ``sqlite3.Connection`` while its owner is inside a statement (Python runs
|
||||
``sqlite3_step`` without the GIL) frees memory the owner then reads: on
|
||||
Windows the process dies with "access violation", elsewhere the owner sees
|
||||
garbage such as a ``MemoryError``.
|
||||
|
||||
``ManagedConnection`` counts the calls its owner is inside. ``retire()`` is
|
||||
safe from any thread and never raises: an idle connection is closed at once;
|
||||
one in use is closed by its owner as the call ends. A retired connection
|
||||
refuses new work (cursors, statements, commits) with the error a closed
|
||||
connection raises.
|
||||
|
||||
What retiring a connection in use does and does not do:
|
||||
|
||||
- It calls ``sqlite3_interrupt`` (thread-safe). That ends a running
|
||||
statement at its next step, but it does not wake a statement that is
|
||||
waiting for another connection's lock: that one waits until the lock is
|
||||
released (or the busy timeout ends it), and only then is it interrupted.
|
||||
An interrupt that arrives between two statements, for example between the
|
||||
implicit ``BEGIN`` and the DML or between ``executemany`` rows, is lost.
|
||||
- Writes are kept out of a file the process has switched away from because
|
||||
a retired connection never commits: legacy-mode DML runs inside a
|
||||
transaction that only ``commit()`` (refused) or ``close()`` (rolls back)
|
||||
ends. A statement that commits by itself is not covered: DDL or a PRAGMA
|
||||
write outside a transaction, or an explicit ``COMMIT`` passed to
|
||||
``execute``/``executescript`` (``executescript`` also commits first, in C,
|
||||
on Python before 3.12). Managed connections must never be used that way
|
||||
for library writes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
import weakref
|
||||
from typing import Iterable, Iterator, List, Optional
|
||||
|
||||
import sqlite3
|
||||
|
||||
CLOSED_MESSAGE = "Cannot operate on a closed database."
|
||||
# Rows fetched per step when a cursor is iterated: the connection is marked
|
||||
# in use once per chunk instead of once per row (iteration costs about what
|
||||
# a plain cursor costs). A loop that stops early leaves up to this many rows
|
||||
# fetched; ``fetchone``/``fetchmany`` after a partial loop skip them.
|
||||
ITER_CHUNK = 128
|
||||
|
||||
|
||||
class ManagedCursor(sqlite3.Cursor):
|
||||
"""A cursor whose calls into SQLite count as its connection being in use."""
|
||||
|
||||
def execute(self, sql, parameters=(), /):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().execute(sql, parameters)
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def executemany(self, sql, seq_of_parameters, /):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().executemany(sql, seq_of_parameters)
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def executescript(self, sql_script, /):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().executescript(sql_script)
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def fetchone(self):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().fetchone()
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def fetchmany(self, *args, **kwargs):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().fetchmany(*args, **kwargs)
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def fetchall(self):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().fetchall()
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def __iter__(self) -> Iterator:
|
||||
return self._chunks()
|
||||
|
||||
def _chunks(self) -> Iterator:
|
||||
while True:
|
||||
rows = self.fetchmany(ITER_CHUNK)
|
||||
if not rows:
|
||||
return
|
||||
yield from rows
|
||||
|
||||
def __next__(self):
|
||||
conn = self.connection
|
||||
conn._enter()
|
||||
try:
|
||||
return super().__next__()
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
def close(self):
|
||||
conn = self.connection
|
||||
try:
|
||||
conn._enter()
|
||||
except sqlite3.ProgrammingError:
|
||||
return super().close() # the connection is closed or closing
|
||||
try:
|
||||
return super().close()
|
||||
finally:
|
||||
conn._leave()
|
||||
|
||||
|
||||
class ManagedConnection(sqlite3.Connection):
|
||||
"""A connection only its owner closes while a call into it is running."""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
# Guards _depth, _retired, _closing and _cursors. Re-entrant so that
|
||||
# code running in the closing thread while it holds the lock (a
|
||||
# cursor() during the cursor-set copy) is refused instead of hanging.
|
||||
self._use_lock = threading.RLock()
|
||||
self._depth = 0
|
||||
self._retired = False
|
||||
self._closing = False
|
||||
self._released = threading.Event()
|
||||
# Python before 3.12 keeps the file open after close() while a cursor
|
||||
# still holds a statement; these are closed first (see _close_now).
|
||||
self._cursors: "weakref.WeakSet[sqlite3.Cursor]" = weakref.WeakSet()
|
||||
self.close_logger: Optional[logging.Logger] = None
|
||||
|
||||
# ── in use ───────────────────────────────────────────────────────
|
||||
|
||||
def _enter(self) -> None:
|
||||
with self._use_lock:
|
||||
if self._retired:
|
||||
raise sqlite3.ProgrammingError(CLOSED_MESSAGE)
|
||||
self._depth += 1
|
||||
|
||||
def _leave(self) -> None:
|
||||
with self._use_lock:
|
||||
self._depth -= 1
|
||||
close_now = self._retired and self._depth == 0
|
||||
if close_now:
|
||||
self._close_now()
|
||||
|
||||
@contextlib.contextmanager
|
||||
def using(self):
|
||||
"""Mark the connection in use for a call this class cannot wrap.
|
||||
|
||||
For example as the *target* of another connection's ``backup()``.
|
||||
"""
|
||||
self._enter()
|
||||
try:
|
||||
yield self
|
||||
finally:
|
||||
self._leave()
|
||||
|
||||
def cursor(self, factory=None):
|
||||
with self._use_lock:
|
||||
if self._retired:
|
||||
raise sqlite3.ProgrammingError(CLOSED_MESSAGE)
|
||||
cursor = super().cursor(factory or ManagedCursor)
|
||||
self._cursors.add(cursor)
|
||||
return cursor
|
||||
|
||||
def execute(self, sql, parameters=(), /):
|
||||
return self.cursor().execute(sql, parameters)
|
||||
|
||||
def executemany(self, sql, seq_of_parameters, /):
|
||||
return self.cursor().executemany(sql, seq_of_parameters)
|
||||
|
||||
def executescript(self, sql_script, /):
|
||||
return self.cursor().executescript(sql_script)
|
||||
|
||||
def commit(self):
|
||||
self._enter()
|
||||
try:
|
||||
return super().commit()
|
||||
finally:
|
||||
self._leave()
|
||||
|
||||
def rollback(self):
|
||||
self._enter() # a closed connection raises, as sqlite3's does
|
||||
try:
|
||||
return super().rollback()
|
||||
finally:
|
||||
self._leave()
|
||||
|
||||
def backup(self, target, **kwargs):
|
||||
self._enter()
|
||||
try:
|
||||
return super().backup(target, **kwargs)
|
||||
finally:
|
||||
self._leave()
|
||||
|
||||
def create_function(self, *args, **kwargs):
|
||||
self._enter()
|
||||
try:
|
||||
return super().create_function(*args, **kwargs)
|
||||
finally:
|
||||
self._leave()
|
||||
|
||||
@property
|
||||
def in_transaction(self) -> bool:
|
||||
self._enter()
|
||||
try:
|
||||
return super().in_transaction
|
||||
finally:
|
||||
self._leave()
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
# The C implementation calls its own commit/rollback, not these.
|
||||
if exc_type is None:
|
||||
try:
|
||||
self.commit()
|
||||
except BaseException:
|
||||
with contextlib.suppress(sqlite3.Error):
|
||||
self.rollback()
|
||||
raise
|
||||
else:
|
||||
self.rollback()
|
||||
return False
|
||||
|
||||
# ── closing ──────────────────────────────────────────────────────
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close now if idle, else as soon as the running call ends.
|
||||
|
||||
Unlike ``sqlite3.Connection.close``, a call from another thread while
|
||||
the owner is inside a call returns before the file is closed. A caller
|
||||
that must know the file is closed (before renaming or deleting it)
|
||||
waits for ``wait_closed`` (``PromptModel.close_detached`` does).
|
||||
"""
|
||||
self.retire()
|
||||
|
||||
def retire(self) -> bool:
|
||||
"""Refuse new work and close; True when the connection is closed already.
|
||||
|
||||
Safe from any thread, and never raises: failures are logged.
|
||||
"""
|
||||
try:
|
||||
with self._use_lock:
|
||||
if not self._retired:
|
||||
self._retired = True
|
||||
if self._depth:
|
||||
# Ends the running statement (see the module docstring
|
||||
# for what it does not end). The owner cannot start
|
||||
# closing the connection while this lock is held.
|
||||
self.interrupt()
|
||||
return False
|
||||
elif self._depth:
|
||||
return False
|
||||
else:
|
||||
return self._released.wait(0)
|
||||
except Exception:
|
||||
self._log_exception("Error retiring database connection")
|
||||
return self._released.wait(0)
|
||||
self._close_now()
|
||||
return True
|
||||
|
||||
def wait_closed(self, timeout: float) -> bool:
|
||||
return self._released.wait(timeout)
|
||||
|
||||
def _close_now(self) -> None:
|
||||
# Only ever called with no call into the connection running, and a
|
||||
# retired connection refuses new calls and cursors: its cursors are
|
||||
# idle, and the set is copied under the lock that guards it.
|
||||
with self._use_lock:
|
||||
if self._closing:
|
||||
return # another thread closes it and sets _released
|
||||
self._closing = True
|
||||
cursors = list(self._cursors)
|
||||
try:
|
||||
for cursor in cursors:
|
||||
with contextlib.suppress(sqlite3.Error):
|
||||
sqlite3.Cursor.close(cursor)
|
||||
super().close()
|
||||
except Exception:
|
||||
self._log_exception("Error closing database connection")
|
||||
finally:
|
||||
self._released.set()
|
||||
|
||||
def _log_exception(self, message: str) -> None:
|
||||
if self.close_logger is not None:
|
||||
self.close_logger.exception(message)
|
||||
|
||||
|
||||
def retire_connections(
|
||||
connections: Iterable[sqlite3.Connection], logger=None
|
||||
) -> List[ManagedConnection]:
|
||||
"""Retire every connection; return those whose owner has yet to close them.
|
||||
|
||||
Safe while holding a lock the owners wait on: nothing here waits for them.
|
||||
One connection failing to retire is logged and never skips the rest.
|
||||
"""
|
||||
pending = []
|
||||
for conn in connections:
|
||||
try:
|
||||
if isinstance(conn, ManagedConnection):
|
||||
if logger is not None:
|
||||
conn.close_logger = logger
|
||||
if not conn.retire():
|
||||
pending.append(conn)
|
||||
else:
|
||||
conn.close()
|
||||
except Exception:
|
||||
if logger is not None:
|
||||
logger.exception("Error closing database connection")
|
||||
return pending
|
||||
|
||||
|
||||
def await_closed(
|
||||
connections: Iterable[ManagedConnection], timeout: float, logger=None
|
||||
) -> bool:
|
||||
"""Wait until the owners closed ``connections`` (retired); False on timeout.
|
||||
|
||||
A connection still open after ``timeout`` (its owner is blocked, for
|
||||
example on another process's lock after a lost interrupt) is logged and
|
||||
left to close when its call ends; a Move library then reports the old
|
||||
files as still in use instead of renaming them.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
still_open = 0
|
||||
for conn in connections:
|
||||
if not conn.wait_closed(max(0.0, deadline - time.monotonic())):
|
||||
still_open += 1
|
||||
if still_open and logger is not None:
|
||||
logger.warning(
|
||||
f"{still_open} database connection(s) still in use after "
|
||||
f"{timeout:g}s; they close when their statement ends"
|
||||
)
|
||||
return not still_open
|
||||
+103
-20
@@ -15,6 +15,11 @@ from contextlib import closing
|
||||
from pathlib import Path
|
||||
from typing import Optional, Set, Tuple
|
||||
|
||||
from .managed_connection import (
|
||||
ManagedConnection,
|
||||
await_closed,
|
||||
retire_connections,
|
||||
)
|
||||
from .library_schema import library_schema_error, migrate_library_state
|
||||
from .image_favorites import image_favorites_schema_error, migrate_image_favorites
|
||||
from .annotation_schema import annotation_schema_error, migrate_annotation_state
|
||||
@@ -609,19 +614,42 @@ class PromptModel:
|
||||
if conn is not None and self._local.generation == self._generation:
|
||||
return conn
|
||||
|
||||
# check_same_thread=False only so close_all can clean up other
|
||||
# threads' connections; each connection is still used by one thread.
|
||||
conn = sqlite3.connect(
|
||||
self.db_path, check_same_thread=False, timeout=BUSY_TIMEOUT_SECONDS
|
||||
)
|
||||
conn.row_factory = sqlite3.Row
|
||||
configure_connection(conn)
|
||||
while True:
|
||||
# The path and generation are read together, and the new
|
||||
# connection is kept only if no Move library repointed the model
|
||||
# meanwhile: otherwise it would write to the old file.
|
||||
with self._conn_lock:
|
||||
path, generation = self.db_path, self._generation
|
||||
# check_same_thread=False only so close_all and Move library can
|
||||
# retire other threads' connections; each connection is still
|
||||
# used by one thread, and ManagedConnection never closes it under
|
||||
# a running statement (see managed_connection).
|
||||
conn = sqlite3.connect(
|
||||
path,
|
||||
check_same_thread=False,
|
||||
timeout=BUSY_TIMEOUT_SECONDS,
|
||||
factory=ManagedConnection,
|
||||
)
|
||||
try:
|
||||
conn.row_factory = sqlite3.Row
|
||||
configure_connection(conn)
|
||||
except BaseException:
|
||||
conn.close()
|
||||
raise
|
||||
with self._conn_lock:
|
||||
if generation == self._generation:
|
||||
self._connections.add(conn)
|
||||
break
|
||||
conn.close()
|
||||
self._local.conn = conn
|
||||
self._local.generation = self._generation
|
||||
with self._conn_lock:
|
||||
self._connections.add(conn)
|
||||
self._local.generation = generation
|
||||
return conn
|
||||
|
||||
@property
|
||||
def generation(self) -> int:
|
||||
"""Bumped whenever every connection is closed or the model repointed."""
|
||||
return self._generation
|
||||
|
||||
@classmethod
|
||||
def close_all_instances(cls) -> None:
|
||||
"""Close every connection of every live model in this process."""
|
||||
@@ -630,22 +658,76 @@ class PromptModel:
|
||||
for model in models:
|
||||
model.close_all()
|
||||
|
||||
@classmethod
|
||||
def repoint_instances(cls, old_path: str, new_path: str) -> list:
|
||||
"""Point every model of ``old_path`` at ``new_path`` (Move library).
|
||||
|
||||
Every thread opens the new file on its next ``get_connection``. The
|
||||
old connections are detached, not closed, and returned: the caller
|
||||
retires them (``retire_detached``) while it still holds the old
|
||||
file's write lock. A write waiting on that lock is then interrupted
|
||||
and can never commit, so it fails (and, through
|
||||
``PromptDatabase.survive_library_switch``, runs again on the new file).
|
||||
"""
|
||||
old_key = os.path.normcase(os.path.realpath(old_path))
|
||||
with cls._instances_lock:
|
||||
models = list(cls._instances)
|
||||
with cls._init_lock:
|
||||
cls._initialized_paths.add(os.path.normcase(os.path.realpath(new_path)))
|
||||
detached = []
|
||||
for model in models:
|
||||
if model._schema_key() == old_key:
|
||||
detached.extend(model._detach_connections(new_path))
|
||||
return detached
|
||||
|
||||
@staticmethod
|
||||
def retire_detached(connections, logger=None) -> list:
|
||||
"""Retire detached connections without waiting for their owners.
|
||||
|
||||
Idle ones close now. One whose owner thread is inside a statement is
|
||||
never closed from here (that frees memory the owner is using: an
|
||||
access violation on Windows); it is interrupted, refuses to commit and
|
||||
is closed by its owner as the statement ends. Returns those still
|
||||
open, for ``await_detached`` once no lock the owners wait on is held.
|
||||
"""
|
||||
return retire_connections(connections, logger)
|
||||
|
||||
@staticmethod
|
||||
def await_detached(connections, logger=None) -> bool:
|
||||
"""Wait until retired connections are closed; False if some are not."""
|
||||
return await_closed(connections, BUSY_TIMEOUT_SECONDS, logger)
|
||||
|
||||
@staticmethod
|
||||
def close_detached(connections, logger=None) -> bool:
|
||||
"""Retire connections ``repoint_instances`` detached and await them.
|
||||
|
||||
Do not call it while holding a lock their owners may wait on; retire
|
||||
them there (``retire_detached``) and await them after releasing it.
|
||||
"""
|
||||
pending = retire_connections(connections, logger)
|
||||
return await_closed(pending, BUSY_TIMEOUT_SECONDS, logger)
|
||||
|
||||
def close_all(self) -> None:
|
||||
"""Close every thread's connection; they reopen lazily on next use.
|
||||
|
||||
Needed before deleting the database file (Windows keeps open files
|
||||
locked) and by tests that ran queries through executor threads.
|
||||
"""
|
||||
self.close_detached(self._detach_connections(), self.logger)
|
||||
|
||||
def _detach_connections(self, new_path: Optional[str] = None) -> list:
|
||||
"""Forget every connection (optionally pointing the model elsewhere).
|
||||
|
||||
Returns them for the caller to close; threads reopen on next use.
|
||||
"""
|
||||
with self._conn_lock:
|
||||
if new_path is not None:
|
||||
self.db_path = new_path
|
||||
connections = list(self._connections)
|
||||
self._connections.clear()
|
||||
self._generation += 1
|
||||
self._local.conn = None
|
||||
for conn in connections:
|
||||
try:
|
||||
conn.close()
|
||||
except sqlite3.Error as e:
|
||||
self.logger.warning(f"Error closing database connection: {e}")
|
||||
return connections
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the calling thread's connection, if it has one.
|
||||
@@ -659,10 +741,7 @@ class PromptModel:
|
||||
self._local.conn = None
|
||||
with self._conn_lock:
|
||||
self._connections.discard(conn)
|
||||
try:
|
||||
conn.close()
|
||||
except sqlite3.Error as e:
|
||||
self.logger.warning(f"Error closing database connection: {e}")
|
||||
self.close_detached([conn], self.logger)
|
||||
|
||||
def _migrate_workflow_name_removal(self, conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
@@ -1168,7 +1247,11 @@ class PromptModel:
|
||||
Path(src_path).resolve().as_uri() + "?mode=ro", uri=True
|
||||
)
|
||||
try:
|
||||
source.backup(self.get_connection())
|
||||
target = self.get_connection()
|
||||
# The target is written under backup's step too: mark it in use
|
||||
# so a concurrent close_all retires it instead of closing it.
|
||||
with target.using():
|
||||
source.backup(target)
|
||||
finally:
|
||||
source.close()
|
||||
|
||||
|
||||
+158
-107
@@ -46,6 +46,7 @@ try:
|
||||
from ..utils.logging_config import get_logger
|
||||
from ..utils.hashing import generate_prompt_hash
|
||||
from ..utils.json_strict import dumps_strict
|
||||
from ..utils import library_location
|
||||
except ImportError:
|
||||
import sys
|
||||
|
||||
@@ -54,6 +55,7 @@ except ImportError:
|
||||
from utils.logging_config import get_logger
|
||||
from utils.hashing import generate_prompt_hash
|
||||
from utils.json_strict import dumps_strict
|
||||
from utils import library_location
|
||||
|
||||
|
||||
# Subquery to fetch tags from junction table, embedded in SELECT statements
|
||||
@@ -99,40 +101,56 @@ def escape_like(term: str) -> str:
|
||||
)
|
||||
|
||||
|
||||
def _resolve_db_path(db_path: Optional[str] = None) -> str:
|
||||
"""Resolve the database path from config, falling back to defaults.
|
||||
def _configured_default_path():
|
||||
"""config.json's ``database.default_path``, or None.
|
||||
|
||||
When no explicit path is given, reads PromptManagerConfig.DEFAULT_DB_PATH
|
||||
(set via config.json). Relative paths are resolved against the extension
|
||||
root directory so the database location is predictable regardless of the
|
||||
process working directory (important for Docker deployments).
|
||||
The package-qualified config wins over a standalone ``py.config`` (a
|
||||
custom-node package uses its own configuration).
|
||||
"""
|
||||
try:
|
||||
try:
|
||||
from ..py.config import PromptManagerConfig
|
||||
except ImportError:
|
||||
from py.config import PromptManagerConfig
|
||||
except Exception:
|
||||
return None
|
||||
return getattr(PromptManagerConfig, "DEFAULT_DB_PATH", None)
|
||||
|
||||
|
||||
def resolve_library_location():
|
||||
"""Where the library lives now (``utils.library_location``, #249 S6b).
|
||||
|
||||
The image sources that rule out a ``PROMPT_MANAGER_DATA_DIR`` are read by
|
||||
the resolver itself, exactly as ``config_location`` reads them.
|
||||
"""
|
||||
return library_location.resolve_library(_configured_default_path())
|
||||
|
||||
|
||||
def _resolve_db_path(db_path: Optional[str] = None) -> str:
|
||||
"""Resolve the database path through the one library resolver.
|
||||
|
||||
With no explicit path, ``utils.library_location.resolve_library`` decides
|
||||
(``PROMPT_MANAGER_DATA_DIR``, an explicit ``database.default_path``, a
|
||||
node-folder library not moved yet, else ComfyUI's ``user/__promptmanager``).
|
||||
An explicit relative path is resolved against the extension root, as
|
||||
before, so the location does not depend on the working directory.
|
||||
|
||||
Args:
|
||||
db_path: Explicit database path, or None to use config/default.
|
||||
db_path: Explicit database path, or None to use the resolver.
|
||||
|
||||
Returns:
|
||||
Absolute path to the SQLite database file.
|
||||
Absolute path to the SQLite database file; its folder exists (a new
|
||||
data folder is created owner-only).
|
||||
"""
|
||||
extension_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
if db_path is None:
|
||||
try:
|
||||
try:
|
||||
from ..py.config import PromptManagerConfig
|
||||
except ImportError:
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
db_path = PromptManagerConfig.DEFAULT_DB_PATH
|
||||
except Exception:
|
||||
db_path = "prompts.db"
|
||||
|
||||
if not os.path.isabs(db_path):
|
||||
db_path = resolve_library_location().db_path
|
||||
elif not os.path.isabs(db_path):
|
||||
extension_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
db_path = os.path.normpath(os.path.join(extension_root, db_path))
|
||||
|
||||
# Ensure parent directory exists for custom paths
|
||||
parent = os.path.dirname(db_path)
|
||||
if parent:
|
||||
os.makedirs(parent, exist_ok=True)
|
||||
library_location.ensure_private_dir(parent)
|
||||
|
||||
return db_path
|
||||
|
||||
@@ -365,6 +383,25 @@ class PromptDatabase(
|
||||
self.model = PromptModel(db_path)
|
||||
self.logger.debug("Database operations initialized successfully")
|
||||
|
||||
def survive_library_switch(self, write):
|
||||
"""Run ``write()``; once more if it failed while the library moved.
|
||||
|
||||
A background write (run counting, the image monitor) that was waiting
|
||||
on the old file when Move library switched fails when its connection
|
||||
is closed. When the model's generation changed during the attempt,
|
||||
the write runs again on a fresh connection, which is the new library;
|
||||
any other failure is raised as it was. ``write`` must open its own
|
||||
transaction (every PromptDatabase method does).
|
||||
"""
|
||||
generation = self.model.generation
|
||||
try:
|
||||
return write()
|
||||
except Exception:
|
||||
if self.model.generation == generation:
|
||||
raise
|
||||
self.logger.info("Running a write again on the moved library")
|
||||
return write()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the calling thread's database connection (see PromptModel.close)."""
|
||||
self.model.close()
|
||||
@@ -1580,103 +1617,117 @@ class PromptDatabase(
|
||||
metadata: Optional metadata about the image
|
||||
|
||||
Returns:
|
||||
int: The ID of the created image record
|
||||
int: The ID of the created image record, or 0 when nothing was
|
||||
linked (invalid or unknown prompt, already linked, or an error,
|
||||
which is logged)
|
||||
"""
|
||||
try:
|
||||
return self.link_image_to_prompt_or_raise(prompt_id, image_path, metadata)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error linking image to prompt {prompt_id}: {e}")
|
||||
return 0
|
||||
|
||||
def link_image_to_prompt_or_raise(
|
||||
self, prompt_id: str, image_path: str, metadata: Optional[Dict[str, Any]] = None
|
||||
) -> int:
|
||||
"""``link_image_to_prompt`` that raises database errors.
|
||||
|
||||
The image monitor links through this so a write that failed because
|
||||
Move library switched reaches ``survive_library_switch`` and runs
|
||||
again on the new library (#249 S6b). Each call is one transaction:
|
||||
a failed attempt rolls back, and a committed one does not raise.
|
||||
Returns 0 for an invalid or unknown prompt or an image already linked.
|
||||
"""
|
||||
# Validate that prompt_id is a valid integer and exists in prompts table
|
||||
try:
|
||||
# Convert prompt_id to integer if it's a string number
|
||||
if isinstance(prompt_id, str) and prompt_id.isdigit():
|
||||
prompt_id_int = int(prompt_id)
|
||||
elif isinstance(prompt_id, int):
|
||||
prompt_id_int = prompt_id
|
||||
# Convert prompt_id to integer if it's a string number
|
||||
if isinstance(prompt_id, str) and prompt_id.isdigit():
|
||||
prompt_id_int = int(prompt_id)
|
||||
elif isinstance(prompt_id, int):
|
||||
prompt_id_int = prompt_id
|
||||
else:
|
||||
# Handle temporary IDs like "temp_123456" - skip linking
|
||||
if isinstance(prompt_id, str) and prompt_id.startswith("temp_"):
|
||||
self.logger.debug(
|
||||
f"Skipping image linking for temporary prompt ID: {prompt_id}"
|
||||
)
|
||||
return 0
|
||||
else:
|
||||
# Handle temporary IDs like "temp_123456" - skip linking
|
||||
if isinstance(prompt_id, str) and prompt_id.startswith("temp_"):
|
||||
self.logger.debug(
|
||||
f"Skipping image linking for temporary prompt ID: {prompt_id}"
|
||||
)
|
||||
return 0
|
||||
else:
|
||||
self.logger.warning(
|
||||
f"Invalid prompt_id format: {prompt_id}, skipping image linking"
|
||||
)
|
||||
return 0
|
||||
|
||||
# Verify the prompt exists in the database
|
||||
with self.model.get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"SELECT id FROM prompts WHERE id = ?", (prompt_id_int,)
|
||||
self.logger.warning(
|
||||
f"Invalid prompt_id format: {prompt_id}, skipping image linking"
|
||||
)
|
||||
if not cursor.fetchone():
|
||||
self.logger.warning(
|
||||
f"Prompt ID {prompt_id_int} not found in database, skipping "
|
||||
"image linking"
|
||||
)
|
||||
return 0
|
||||
return 0
|
||||
|
||||
# Proceed with linking
|
||||
if not isinstance(metadata, dict):
|
||||
metadata = {}
|
||||
filename = os.path.basename(image_path)
|
||||
file_path = normalize_image_path(image_path)
|
||||
file_size, width, height, image_format = _coerce_file_info(metadata)
|
||||
# Each workflow and prompt graph is stored once (new form).
|
||||
documents = stored_values(
|
||||
# Verify the prompt exists in the database
|
||||
with self.model.get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"SELECT id FROM prompts WHERE id = ?", (prompt_id_int,)
|
||||
)
|
||||
if not cursor.fetchone():
|
||||
self.logger.warning(
|
||||
f"Prompt ID {prompt_id_int} not found in database, skipping "
|
||||
"image linking"
|
||||
)
|
||||
return 0
|
||||
|
||||
# Proceed with linking
|
||||
if not isinstance(metadata, dict):
|
||||
metadata = {}
|
||||
filename = os.path.basename(image_path)
|
||||
file_path = normalize_image_path(image_path)
|
||||
file_size, width, height, image_format = _coerce_file_info(metadata)
|
||||
# Each workflow and prompt graph is stored once (new form).
|
||||
documents = stored_values(
|
||||
conn,
|
||||
{
|
||||
"workflow_data": dumps_strict(metadata.get("workflow", {})),
|
||||
"prompt_metadata": dumps_strict(metadata.get("prompt", {})),
|
||||
},
|
||||
)
|
||||
|
||||
# INSERT OR IGNORE skips a file already linked to this prompt
|
||||
# (same prompt_id + normalised full path)
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT OR IGNORE INTO generated_images
|
||||
(prompt_id, image_path, filename, file_path, file_size, width,
|
||||
height, format, workflow_data, prompt_metadata, parameters,
|
||||
workflow_document_id, prompt_document_id)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
prompt_id_int,
|
||||
image_path,
|
||||
filename,
|
||||
file_path,
|
||||
file_size,
|
||||
width,
|
||||
height,
|
||||
image_format,
|
||||
documents["workflow_data"],
|
||||
documents["prompt_metadata"],
|
||||
dumps_strict(metadata.get("parameters", {})),
|
||||
documents["workflow_document_id"],
|
||||
documents["prompt_document_id"],
|
||||
),
|
||||
)
|
||||
if cursor.rowcount == 0:
|
||||
# Already linked: a document made only for this row goes.
|
||||
release_documents(
|
||||
conn,
|
||||
{
|
||||
"workflow_data": dumps_strict(metadata.get("workflow", {})),
|
||||
"prompt_metadata": dumps_strict(metadata.get("prompt", {})),
|
||||
},
|
||||
)
|
||||
|
||||
# INSERT OR IGNORE skips a file already linked to this prompt
|
||||
# (same prompt_id + normalised full path)
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT OR IGNORE INTO generated_images
|
||||
(prompt_id, image_path, filename, file_path, file_size, width,
|
||||
height, format, workflow_data, prompt_metadata, parameters,
|
||||
workflow_document_id, prompt_document_id)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
prompt_id_int,
|
||||
image_path,
|
||||
filename,
|
||||
file_path,
|
||||
file_size,
|
||||
width,
|
||||
height,
|
||||
image_format,
|
||||
documents["workflow_data"],
|
||||
documents["prompt_metadata"],
|
||||
dumps_strict(metadata.get("parameters", {})),
|
||||
documents["workflow_document_id"],
|
||||
documents["prompt_document_id"],
|
||||
),
|
||||
)
|
||||
if cursor.rowcount == 0:
|
||||
# Already linked: a document made only for this row goes.
|
||||
release_documents(
|
||||
conn,
|
||||
(
|
||||
documents["workflow_document_id"],
|
||||
documents["prompt_document_id"],
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
conn.commit()
|
||||
|
||||
if cursor.rowcount == 0:
|
||||
self.logger.debug(
|
||||
f"Image {file_path} already linked to prompt {prompt_id_int}"
|
||||
)
|
||||
return 0
|
||||
if cursor.rowcount == 0:
|
||||
self.logger.debug(
|
||||
f"Image {file_path} already linked to prompt {prompt_id_int}"
|
||||
)
|
||||
return 0
|
||||
|
||||
return cursor.lastrowid
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error linking image to prompt {prompt_id}: {e}")
|
||||
return 0
|
||||
return cursor.lastrowid
|
||||
|
||||
def get_prompt_images(self, prompt_id: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
|
||||
@@ -62,3 +62,14 @@ def notify_review_change(db_path: str) -> None:
|
||||
callback()
|
||||
except Exception:
|
||||
_logger.warning("Needs review change observer failed", exc_info=True)
|
||||
|
||||
|
||||
def rekey_review_changes(old_path: str, new_path: str) -> None:
|
||||
"""Move the observers of ``old_path`` to ``new_path`` (Move library)."""
|
||||
old, new = _identity(old_path), _identity(new_path)
|
||||
if old == new:
|
||||
return
|
||||
with _lock:
|
||||
observers = _observers.pop(old, None)
|
||||
if observers:
|
||||
_observers.setdefault(new, {}).update(observers)
|
||||
|
||||
+25
-7
@@ -9,10 +9,13 @@ Split into domain-specific mixins for maintainability:
|
||||
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
|
||||
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
|
||||
- ThumbnailJobsMixin: thumbnail plan, thumbnail job and the gallery stream on it
|
||||
- ThumbnailConvertMixin: Convert outdated thumbnails (#317), a thumbnail job mode
|
||||
- AutoThumbnailMixin: previews made on first view, on capture and after a scan
|
||||
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
|
||||
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
|
||||
- ReviewEventsMixin: the Needs review change stream an open page re-reads on
|
||||
- SystemRoutesMixin: System health check, Overview and usage statistics
|
||||
- LibraryLocationRoutesMixin: where the library lives, Move library, Keep it here
|
||||
- MaintenanceRoutesMixin: reviewed maintenance plans (preview, apply)
|
||||
- WorkflowStorageRoutesMixin: store each workflow once, then Reclaim space
|
||||
- DuplicateFilesMixin: duplicate image files as find and removal jobs
|
||||
@@ -48,7 +51,10 @@ from .settings_routes import SettingsRoutesMixin
|
||||
from .job_routes import JobRoutesMixin
|
||||
from .image_tag_jobs import ImageTagJobsMixin
|
||||
from .thumbnail_jobs import ThumbnailJobsMixin
|
||||
from .thumbnail_convert import ThumbnailConvertMixin
|
||||
from .thumbnail_auto import AutoThumbnailMixin
|
||||
from .system import SystemRoutesMixin
|
||||
from .library_location import LibraryLocationRoutesMixin
|
||||
from .maintenance import MaintenanceRoutesMixin
|
||||
from .workflow_storage import WorkflowStorageRoutesMixin
|
||||
from .duplicate_files import DuplicateFilesMixin
|
||||
@@ -291,7 +297,10 @@ class PromptManagerAPI(
|
||||
JobRoutesMixin,
|
||||
ImageTagJobsMixin,
|
||||
ThumbnailJobsMixin,
|
||||
ThumbnailConvertMixin,
|
||||
AutoThumbnailMixin,
|
||||
SystemRoutesMixin,
|
||||
LibraryLocationRoutesMixin,
|
||||
MaintenanceRoutesMixin,
|
||||
WorkflowStorageRoutesMixin,
|
||||
DuplicateFilesMixin,
|
||||
@@ -373,6 +382,10 @@ class PromptManagerAPI(
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Could not sweep interrupted backup copies: {e}")
|
||||
|
||||
# A crash between Move library's switch and its rename left the old
|
||||
# node-folder files under their own names (#249 S6b): rename them now.
|
||||
self.finish_interrupted_library_move()
|
||||
|
||||
# Run cleanup on initialization to remove any existing duplicates
|
||||
try:
|
||||
removed = self.db.cleanup_duplicates()
|
||||
@@ -532,6 +545,7 @@ class PromptManagerAPI(
|
||||
self._register_settings_routes(routes)
|
||||
self._register_job_routes(routes)
|
||||
self._register_system_routes(routes)
|
||||
self._register_library_location_routes(routes)
|
||||
self._register_maintenance_routes(routes)
|
||||
self._register_duplicate_routes(routes)
|
||||
self._register_logging_routes(routes)
|
||||
@@ -627,16 +641,20 @@ class PromptManagerAPI(
|
||||
f"{url_quote(rel_path.as_posix(), safe='/')}"
|
||||
)
|
||||
|
||||
# Check for thumbnail
|
||||
rel_no_ext = rel_path.with_suffix("")
|
||||
thumb_rel = (
|
||||
f"thumbnails/{rel_no_ext.as_posix()}_thumb{rel_path.suffix}"
|
||||
)
|
||||
# Any preview the file owns, the current format first (#317)
|
||||
from . import thumbnail_settings
|
||||
from .prompts import thumbnail_url_for
|
||||
|
||||
thumbnail_url = thumbnail_url_for(
|
||||
thumb_rel, output_path / thumb_rel
|
||||
existing = thumbnail_settings.existing_thumbnail(
|
||||
output_path, rel_path
|
||||
)
|
||||
thumbnail_url = None
|
||||
if existing is not None:
|
||||
thumb_rel = (
|
||||
thumbnail_settings.thumbnail_stem_rel(rel_path)
|
||||
+ existing.suffix
|
||||
)
|
||||
thumbnail_url = thumbnail_url_for(thumb_rel, existing)
|
||||
if thumbnail_url:
|
||||
image["thumbnail_url"] = thumbnail_url
|
||||
break # Found matching root, stop searching
|
||||
|
||||
+46
-28
@@ -18,6 +18,7 @@ from pathlib import Path
|
||||
from aiohttp import web
|
||||
|
||||
from .prompts import thumbnail_url_for
|
||||
from . import thumbnail_settings
|
||||
from .jobs import (
|
||||
DEFAULT_SCAN_SCOPE,
|
||||
SCAN_GROUP_NAMES,
|
||||
@@ -146,39 +147,48 @@ def _physical_file_key(path, stat):
|
||||
return ("path", os.path.normcase(os.path.realpath(str(path))))
|
||||
|
||||
|
||||
def _thumbnail_rel_path(rel_path, thumbnail_ext):
|
||||
"""Relative path of the thumbnail the gallery would generate for ``rel_path``."""
|
||||
return f"thumbnails/{rel_path.with_suffix('').as_posix()}_thumb{thumbnail_ext}"
|
||||
|
||||
|
||||
def _media_thumbnail_rel_path(rel_path):
|
||||
"""The thumbnail of a media file: a video's preview is a ``.jpg``."""
|
||||
extension = rel_path.suffix.lower()
|
||||
return _thumbnail_rel_path(
|
||||
rel_path, ".jpg" if extension in VIDEO_EXTENSIONS else rel_path.suffix
|
||||
)
|
||||
def _other_sources(output_path, rel_path):
|
||||
"""Whether another media file shares *rel_path*'s stem in its folder."""
|
||||
folder = Path(output_path) / Path(rel_path).parent
|
||||
stem = Path(rel_path).stem
|
||||
own = Path(rel_path).suffix.lower()
|
||||
for extension in IMAGE_EXTENSIONS + VIDEO_EXTENSIONS:
|
||||
if extension == own:
|
||||
continue
|
||||
for name in (stem + extension, stem + extension.upper()):
|
||||
if (folder / name).is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _remove_thumbnail(output_path, rel_path, logger=None):
|
||||
"""Remove the thumbnail of ``rel_path`` under ``output_path``, if any.
|
||||
"""Remove the previews of ``rel_path`` under ``output_path``, if any.
|
||||
|
||||
Shared by the legacy duplicate delete and the reviewed removal job. A
|
||||
thumbnail that is a link, or one that cannot be removed, is left alone
|
||||
(a stale preview never fails a removal). Returns whether one was removed.
|
||||
Shared by the legacy duplicate delete and the reviewed removal job.
|
||||
Every preview the file owns goes, in any format (#317); a preview name
|
||||
another same-stem file still shares (``x_thumb.webp`` of ``x.png`` and
|
||||
``x.mp4``, ruling R2) stays for it. A preview that is a link, or one that
|
||||
cannot be removed, is left alone (a stale preview never fails a removal).
|
||||
Returns whether one was removed.
|
||||
"""
|
||||
output_path = Path(output_path)
|
||||
thumbnail_path = output_path / _media_thumbnail_rel_path(Path(rel_path))
|
||||
try:
|
||||
if thumbnail_path.is_symlink() or not thumbnail_path.is_file():
|
||||
return False
|
||||
os.remove(thumbnail_path)
|
||||
except OSError as e:
|
||||
rel_path = Path(rel_path)
|
||||
shared = _other_sources(output_path, rel_path)
|
||||
removed = False
|
||||
for thumbnail_path in thumbnail_settings.owned_thumbnails(output_path, rel_path):
|
||||
extension = thumbnail_path.suffix.lower()
|
||||
if shared and extension in thumbnail_settings.WRITTEN_EXTENSIONS:
|
||||
continue
|
||||
try:
|
||||
os.remove(thumbnail_path)
|
||||
except OSError as e:
|
||||
if logger is not None:
|
||||
logger.warning(f"Could not delete thumbnail for {rel_path}: {e}")
|
||||
continue
|
||||
removed = True
|
||||
if logger is not None:
|
||||
logger.warning(f"Could not delete thumbnail for {rel_path}: {e}")
|
||||
return False
|
||||
if logger is not None:
|
||||
logger.debug(f"Deleted associated thumbnail: {thumbnail_path}")
|
||||
return True
|
||||
logger.debug(f"Deleted associated thumbnail: {thumbnail_path}")
|
||||
return removed
|
||||
|
||||
|
||||
async def _read_json(request):
|
||||
@@ -845,8 +855,13 @@ class AdminRoutesMixin:
|
||||
extension = media_path.suffix.lower()
|
||||
is_video = extension in VIDEO_EXTENSIONS
|
||||
|
||||
thumb_rel = _thumbnail_rel_path(rel_path, ".jpg" if is_video else extension)
|
||||
thumbnail_url = thumbnail_url_for(thumb_rel, output_path / thumb_rel)
|
||||
existing = thumbnail_settings.existing_thumbnail(output_path, rel_path)
|
||||
thumbnail_url = None
|
||||
if existing is not None:
|
||||
thumb_rel = (
|
||||
thumbnail_settings.thumbnail_stem_rel(rel_path) + existing.suffix
|
||||
)
|
||||
thumbnail_url = thumbnail_url_for(thumb_rel, existing)
|
||||
|
||||
return {
|
||||
"id": str(hash(str(media_path))),
|
||||
@@ -1504,6 +1519,9 @@ class AdminRoutesMixin:
|
||||
verb = "cancelled" if state == "cancelled" else "finished"
|
||||
job.log_line(f"Scan {verb} · {_plural(job.processed, 'file')} processed")
|
||||
job.finish(state)
|
||||
if state == "finished":
|
||||
# After the scan, so the guarded thumbnail job is admitted (#249).
|
||||
await self._hand_off_scan_thumbnails(job, output_dirs)
|
||||
|
||||
async def _stream_scan_job(self, request, job):
|
||||
"""Stream ``job`` as SSE; stop quietly if the client goes away."""
|
||||
|
||||
@@ -10,6 +10,7 @@ from urllib.parse import quote
|
||||
|
||||
from .jobs import AUTOTAG_BATCH_JOB_KIND
|
||||
from .prompts import thumbnail_url_for
|
||||
from . import thumbnail_settings
|
||||
from .. import model_gate
|
||||
|
||||
from aiohttp import web
|
||||
@@ -792,13 +793,16 @@ class AutotagRoutesMixin:
|
||||
rel_path = image_path.relative_to(output_path)
|
||||
thumbnail_url = None
|
||||
if has_thumbnails:
|
||||
thumb_rel = (
|
||||
f"thumbnails/{rel_path.with_suffix('').as_posix()}"
|
||||
f"_thumb{image_path.suffix}"
|
||||
)
|
||||
thumbnail_url = thumbnail_url_for(
|
||||
thumb_rel, output_path / thumb_rel
|
||||
# Any preview the file owns, the current format first (#317)
|
||||
existing = thumbnail_settings.existing_thumbnail(
|
||||
output_path, rel_path
|
||||
)
|
||||
if existing is not None:
|
||||
thumb_rel = (
|
||||
thumbnail_settings.thumbnail_stem_rel(rel_path)
|
||||
+ existing.suffix
|
||||
)
|
||||
thumbnail_url = thumbnail_url_for(thumb_rel, existing)
|
||||
images.append(
|
||||
{
|
||||
"filename": image_path.name,
|
||||
|
||||
+56
-20
@@ -20,6 +20,7 @@ from .history_current import (
|
||||
_OpenedCurrentMedia,
|
||||
_current_media_response,
|
||||
_prepare_current_media,
|
||||
accepts_jxl,
|
||||
_reauthorize_current_target,
|
||||
candidate_locator,
|
||||
resolve_current_candidate,
|
||||
@@ -142,6 +143,18 @@ def _query(request: web.Request) -> None:
|
||||
raise ValueError("Duplicate history query parameter")
|
||||
|
||||
|
||||
def _jxl_flag(request: web.Request) -> bool:
|
||||
"""The cover's ``jxl`` flag (#317): ``1`` when the workspace decoded a
|
||||
JPEG XL image (a browser may not list ``image/jxl`` in Accept), ``0`` or
|
||||
absent otherwise; anything else is a bad request."""
|
||||
value = request.query.get("jxl")
|
||||
if value is None:
|
||||
return False
|
||||
if value not in ("0", "1"):
|
||||
raise ValueError("Invalid jxl flag")
|
||||
return value == "1"
|
||||
|
||||
|
||||
def _page_limit(request: web.Request) -> int:
|
||||
limit = _decimal(request.query.get("limit", "40"), minimum=1)
|
||||
if limit > 100:
|
||||
@@ -734,7 +747,10 @@ class HistoryRoutesMixin:
|
||||
raise HistoryReadError(422, "response-limit") from None
|
||||
else:
|
||||
prepared = _prepare_current_media(
|
||||
resolved, scope, thumbnail=kind == "thumbnail"
|
||||
resolved,
|
||||
scope,
|
||||
thumbnail=kind == "thumbnail",
|
||||
**media_options,
|
||||
)
|
||||
_check_current(scope, self, fence, epoch, generation)
|
||||
return prepared
|
||||
@@ -761,25 +777,45 @@ class HistoryRoutesMixin:
|
||||
_check_current(scope, self, fence, epoch, generation)
|
||||
raise
|
||||
|
||||
pending = asyncio.create_task(self._run_in_executor(project))
|
||||
try:
|
||||
prepared = await asyncio.shield(pending)
|
||||
except asyncio.CancelledError:
|
||||
pending.add_done_callback(_close_prepared_result)
|
||||
raise
|
||||
except (
|
||||
HistoryReadError,
|
||||
HistoryEpochChanged,
|
||||
sqlite3.Error,
|
||||
OSError,
|
||||
ValueError,
|
||||
):
|
||||
raise
|
||||
except Exception as error:
|
||||
self.logger.warning(
|
||||
"Current history media unavailable (%s)", type(error).__name__
|
||||
)
|
||||
raise _current_unavailable("temporary_unavailable") from None
|
||||
async def execute():
|
||||
pending = asyncio.create_task(self._run_in_executor(project))
|
||||
try:
|
||||
return await asyncio.shield(pending)
|
||||
except asyncio.CancelledError:
|
||||
pending.add_done_callback(_close_prepared_result)
|
||||
raise
|
||||
except (
|
||||
HistoryReadError,
|
||||
HistoryEpochChanged,
|
||||
sqlite3.Error,
|
||||
OSError,
|
||||
ValueError,
|
||||
):
|
||||
raise
|
||||
except Exception as error:
|
||||
self.logger.warning(
|
||||
"Current history media unavailable (%s)", type(error).__name__
|
||||
)
|
||||
raise _current_unavailable("temporary_unavailable") from None
|
||||
|
||||
# A missing preview (#249) is noted on the first read, made through
|
||||
# the serve route's admission and single flight, then read again. A
|
||||
# video whose preview cannot be made is the placeholder frame.
|
||||
needed = []
|
||||
media_options = {}
|
||||
if kind == "thumbnail":
|
||||
media_options = {
|
||||
"ensure_preview": lambda *preview: needed.append(preview),
|
||||
"placeholder": self._video_placeholder_bytes(),
|
||||
"accepts_jxl": _jxl_flag(request) or accepts_jxl(request),
|
||||
}
|
||||
prepared = await execute()
|
||||
if needed:
|
||||
if not isinstance(prepared, bytes):
|
||||
prepared.close()
|
||||
await self._admit_preview(*needed[0])
|
||||
media_options["ensure_preview"] = None
|
||||
prepared = await execute()
|
||||
if isinstance(prepared, bytes):
|
||||
return _success_response(prepared)
|
||||
return await _current_media_response(request, prepared)
|
||||
|
||||
+106
-25
@@ -5,6 +5,7 @@ from dataclasses import dataclass
|
||||
import errno
|
||||
import hashlib
|
||||
import hmac
|
||||
import io
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
@@ -14,7 +15,9 @@ from typing import BinaryIO
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from . import thumbnail_settings
|
||||
from .images import _canonical, _public_path, SERVABLE_EXTENSIONS, VIDEO_EXTENSIONS
|
||||
from .images import existing_media_thumbnail, media_thumbnail_rel
|
||||
|
||||
try:
|
||||
from ...utils.execution_outputs import (
|
||||
@@ -238,14 +241,19 @@ def resolve_current_candidate(
|
||||
|
||||
|
||||
def _open_current_target(
|
||||
target: Path, namespace: Path, scope: CurrentOutputScope
|
||||
target: Path, namespace: Path, scope: CurrentOutputScope, preview: bool = False
|
||||
) -> BinaryIO:
|
||||
"""Open only the canonical target and verify authority against the actual handle."""
|
||||
"""Open only the canonical target and verify authority against the actual handle.
|
||||
|
||||
A *preview* (a file under ``thumbnails/``) may also be JPEG XL (#317)."""
|
||||
if _canonical(target) != target:
|
||||
raise CurrentLocationUnavailable("not_authorized", stale=True)
|
||||
if not _authorized(target, namespace, scope):
|
||||
raise CurrentLocationUnavailable("not_authorized")
|
||||
if target.suffix.lower() not in SERVABLE_EXTENSIONS:
|
||||
suffix = target.suffix.lower()
|
||||
if suffix not in SERVABLE_EXTENSIONS and not (
|
||||
preview and suffix in thumbnail_settings.THUMBNAIL_EXTENSIONS
|
||||
):
|
||||
raise CurrentLocationUnavailable("unsupported_media")
|
||||
flags = (
|
||||
os.O_RDONLY
|
||||
@@ -309,34 +317,107 @@ class _OpenedCurrentMedia:
|
||||
self.handle.close()
|
||||
|
||||
|
||||
def _prepare_current_media(
|
||||
resolved: ResolvedCurrentCandidate, scope: CurrentOutputScope, *, thumbnail: bool
|
||||
) -> _OpenedCurrentMedia:
|
||||
def _preview_writable(root: Path, preview: Path) -> bool:
|
||||
"""A preview may be written: ``thumbnails`` and every link segment of the
|
||||
preview's folder resolve inside *root* (the serve route's rule, #249)."""
|
||||
folder = root / "thumbnails"
|
||||
return _canonical(folder).is_relative_to(_canonical(root)) and _canonical(
|
||||
preview
|
||||
).is_relative_to(_canonical(folder))
|
||||
|
||||
|
||||
def _is_jxl(path) -> bool:
|
||||
return Path(path).suffix.lower() == ".jxl"
|
||||
|
||||
|
||||
def _preview_without_jxl(root: Path, relative: Path) -> Path | None:
|
||||
"""The first existing preview *relative* owns that is not JPEG XL."""
|
||||
return next(
|
||||
(
|
||||
path
|
||||
for path in thumbnail_settings.owned_thumbnails(root, relative)
|
||||
if not _is_jxl(path)
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def accepts_jxl(request) -> bool:
|
||||
"""Whether the request's ``Accept`` lists ``image/jxl``."""
|
||||
accept = request.headers.get("Accept", "")
|
||||
return any(
|
||||
part.split(";", 1)[0].strip().lower() == "image/jxl"
|
||||
for part in accept.split(",")
|
||||
)
|
||||
|
||||
|
||||
def _placeholder_media(placeholder: bytes) -> "_OpenedCurrentMedia":
|
||||
return _OpenedCurrentMedia(io.BytesIO(placeholder), len(placeholder), "image/jpeg")
|
||||
|
||||
|
||||
def _cover_target(resolved, ensure_preview, placeholder, accepts_jxl):
|
||||
"""What the cover opens: a preview path, the placeholder media, or None
|
||||
for the original (see ``_prepare_current_media``)."""
|
||||
target = resolved.canonical_target
|
||||
relative = target.relative_to(resolved.gallery_root)
|
||||
is_video = target.suffix.lower() in VIDEO_EXTENSIONS
|
||||
# #317: any preview the file owns (the current format first); a missing
|
||||
# one is made under the current format's name.
|
||||
preview = existing_media_thumbnail(resolved.gallery_root, relative)
|
||||
if preview is not None and not accepts_jxl and _is_jxl(preview):
|
||||
preview = _preview_without_jxl(resolved.gallery_root, relative)
|
||||
if preview is None:
|
||||
if is_video and placeholder is not None:
|
||||
return _placeholder_media(placeholder)
|
||||
return None
|
||||
return _canonical(preview)
|
||||
if preview is not None:
|
||||
return _canonical(preview)
|
||||
preview = resolved.gallery_root / media_thumbnail_rel(relative)
|
||||
if ensure_preview is not None and _preview_writable(resolved.gallery_root, preview):
|
||||
ensure_preview(target, preview, is_video)
|
||||
if is_video and placeholder is not None:
|
||||
return _placeholder_media(placeholder)
|
||||
return None
|
||||
|
||||
|
||||
def _prepare_current_media(
|
||||
resolved: ResolvedCurrentCandidate,
|
||||
scope: CurrentOutputScope,
|
||||
*,
|
||||
thumbnail: bool,
|
||||
ensure_preview=None,
|
||||
placeholder: bytes | None = None,
|
||||
accepts_jxl: bool = True,
|
||||
) -> _OpenedCurrentMedia:
|
||||
"""Open the current file, or its preview for ``thumbnail``.
|
||||
|
||||
A missing preview that may be written is handed to
|
||||
``ensure_preview(src, dst, is_video)`` (the route makes it and reads
|
||||
again, #249). Without one an image's original is opened and a video is
|
||||
the ``placeholder`` JPEG, never the video inside an ``<img>``.
|
||||
|
||||
The cover's URL names no format, so a JPEG XL preview goes only to a
|
||||
browser whose ``Accept`` lists ``image/jxl`` (*accepts_jxl*, review M3).
|
||||
Another one gets the file's preview in another format when there is one,
|
||||
else the original image or a video's placeholder: never a broken image.
|
||||
"""
|
||||
target = resolved.canonical_target
|
||||
is_preview = False
|
||||
if thumbnail:
|
||||
relative = target.relative_to(resolved.gallery_root)
|
||||
extension = (
|
||||
".jpg"
|
||||
if target.suffix.lower() in VIDEO_EXTENSIONS
|
||||
else target.suffix.lower()
|
||||
)
|
||||
preview = (
|
||||
resolved.gallery_root
|
||||
/ "thumbnails"
|
||||
/ f"{relative.with_suffix('').as_posix()}_thumb{extension}"
|
||||
)
|
||||
try:
|
||||
preview.stat()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
else:
|
||||
target = _canonical(preview)
|
||||
handle = _open_current_target(target, resolved.namespace, scope)
|
||||
chosen = _cover_target(resolved, ensure_preview, placeholder, accepts_jxl)
|
||||
if isinstance(chosen, _OpenedCurrentMedia):
|
||||
return chosen
|
||||
if chosen is not None:
|
||||
target, is_preview = chosen, True
|
||||
handle = _open_current_target(target, resolved.namespace, scope, is_preview)
|
||||
try:
|
||||
return _OpenedCurrentMedia(
|
||||
handle,
|
||||
os.fstat(handle.fileno()).st_size,
|
||||
mimetypes.guess_type(target.name)[0] or "application/octet-stream",
|
||||
thumbnail_settings.content_type(target)
|
||||
or mimetypes.guess_type(target.name)[0]
|
||||
or "application/octet-stream",
|
||||
)
|
||||
except BaseException:
|
||||
handle.close()
|
||||
|
||||
+501
-291
File diff suppressed because it is too large
Load Diff
@@ -515,6 +515,12 @@ class JobRegistry:
|
||||
def get(self, job_id):
|
||||
return self._jobs.get(job_id)
|
||||
|
||||
def running(self):
|
||||
"""The running jobs, oldest first (a snapshot, safe off the event loop)."""
|
||||
jobs = [job for job in tuple(self._jobs.values()) if job.running]
|
||||
jobs.sort(key=lambda job: job._started_seq)
|
||||
return jobs
|
||||
|
||||
def active(self, kind):
|
||||
"""The running job of ``kind``, or None (safe off the event loop)."""
|
||||
for job in tuple(self._jobs.values()):
|
||||
|
||||
@@ -0,0 +1,420 @@
|
||||
"""Library location routes (#249 S6b): where the library lives, Move, Keep.
|
||||
|
||||
- ``GET /prompt_manager/system/library`` → ``{success, location}``: the
|
||||
database's path (relative to ComfyUI, as the statistics show it), why it is
|
||||
there (``source``: ``default``, ``environment``, ``config`` or
|
||||
``node_folder``), whether it can move and where to (``target``), the
|
||||
"Keep it here" choice (``keep``), whether to offer the move (``offer``:
|
||||
movable and not kept), a ``problem`` code and the database's size. The
|
||||
System Overview carries the same object as ``location``.
|
||||
- ``POST /prompt_manager/system/library/move`` → 202 ``{job}``: the
|
||||
``library_move`` job, ``exclusive`` (it runs alone and every other job,
|
||||
legacy-page operations included, is refused while it runs), not
|
||||
cancellable. It copies, verifies and switches the live library
|
||||
(``database/library_move.py``) and ends with one truthful line; a failure
|
||||
leaves the library in the node folder, unchanged. Refused before any job
|
||||
starts: 400 ``not-movable``/``same-folder``/``target-invalid``, 409
|
||||
``target-has-library``/``target-conflict``/``busy``, 507
|
||||
``insufficient-space``.
|
||||
- ``POST /prompt_manager/system/library/keep {keep: bool}`` persists
|
||||
``library.keep_in_node_folder`` through the config store and answers the
|
||||
new location.
|
||||
|
||||
Errors are ``{success:false, code, error}`` and never name a path.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import os
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from .job_routes import busy_response
|
||||
from .jobs import JobBusy
|
||||
|
||||
try:
|
||||
from ...database import library_move as lm
|
||||
from ...database.operations import resolve_library_location
|
||||
from ...utils import library_location as ll
|
||||
except ImportError:
|
||||
from database import library_move as lm
|
||||
from database.operations import resolve_library_location
|
||||
from utils import library_location as ll
|
||||
|
||||
LIBRARY_MOVE_JOB_KIND = "library_move"
|
||||
LIBRARY_MOVE_TITLE = "Move library"
|
||||
LIBRARY_MOVE_GROUPS = ("files", "errors")
|
||||
KEEP_BODY_MAX = 1024
|
||||
|
||||
_REFUSAL_STATUS = {
|
||||
"not-movable": 400,
|
||||
"same-folder": 400,
|
||||
"target-invalid": 400,
|
||||
"target-has-library": 409,
|
||||
"target-conflict": 409,
|
||||
"insufficient-space": 507,
|
||||
}
|
||||
STILL_THERE = "The library is still in the node folder, unchanged."
|
||||
NOT_MOVED = "The library was not moved."
|
||||
OLD_LEFT = {
|
||||
"old-in-use": (
|
||||
"The old database is still open elsewhere, so it was left in the node "
|
||||
"folder; it is no longer used."
|
||||
),
|
||||
"old-locked": (
|
||||
"The old files could not be renamed, so they were left in the node "
|
||||
"folder; they are no longer used."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _failure(message, status, code):
|
||||
return web.json_response(
|
||||
{"success": False, "code": code, "error": message}, status=status
|
||||
)
|
||||
|
||||
|
||||
def _plural(count, one, many=None):
|
||||
return f"{count:,} {one if count == 1 else (many or one + 's')}"
|
||||
|
||||
|
||||
def _same_file(first, second):
|
||||
try:
|
||||
return os.path.samefile(first, second)
|
||||
except OSError:
|
||||
return os.path.normcase(os.path.realpath(first)) == os.path.normcase(
|
||||
os.path.realpath(second)
|
||||
)
|
||||
|
||||
|
||||
CUSTOM_NODES = "custom_nodes"
|
||||
|
||||
|
||||
def _inside(path, folder):
|
||||
"""``path`` relative to ``folder`` by text (normalised spellings), or None."""
|
||||
child, parent = os.path.normcase(path), os.path.normcase(folder)
|
||||
if child == parent:
|
||||
return ""
|
||||
prefix = parent.rstrip(os.sep) + os.sep
|
||||
return path[len(prefix) :] if child.startswith(prefix) else None
|
||||
|
||||
|
||||
def _comfyui_custom_nodes():
|
||||
"""ComfyUI's custom_nodes folders as it lists them (unresolved), or []."""
|
||||
try:
|
||||
from ..config import _import_folder_paths
|
||||
|
||||
getter = getattr(_import_folder_paths(), "get_folder_paths", None)
|
||||
folders = getter(CUSTOM_NODES) if getter else []
|
||||
return [folder for folder in folders or [] if isinstance(folder, str)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def node_folder_public_path(path, extension_root, custom_nodes):
|
||||
"""``custom_nodes/<node folder>/...`` for a path in the node folder, or None.
|
||||
|
||||
The node folder is matched at its own unresolved place in a custom_nodes
|
||||
folder (one of ``custom_nodes``, or the parent of ``extension_root`` when
|
||||
that is named custom_nodes), so a node folder that is a symlink to a
|
||||
checkout outside ComfyUI still reads relative to ComfyUI. ``path`` may be
|
||||
spelled through the link or resolved. Never absolute, never ``..``:
|
||||
anything else is None and the caller uses ``_public_path``.
|
||||
"""
|
||||
if not path or not extension_root:
|
||||
return None
|
||||
try:
|
||||
root = os.path.normpath(os.path.abspath(extension_root))
|
||||
name = os.path.basename(root)
|
||||
folders = [os.path.normpath(os.path.abspath(f)) for f in custom_nodes]
|
||||
parent = os.path.dirname(root)
|
||||
if os.path.normcase(os.path.basename(parent)) == CUSTOM_NODES:
|
||||
folders.append(parent)
|
||||
spellings = [os.path.normpath(os.path.abspath(path))]
|
||||
spellings.append(os.path.realpath(spellings[0]))
|
||||
except (OSError, ValueError, TypeError):
|
||||
return None
|
||||
for folder in folders:
|
||||
node = os.path.join(folder, name)
|
||||
for spelling, node_spelling in zip(spellings, (node, os.path.realpath(node))):
|
||||
rest = _inside(spelling, node_spelling)
|
||||
if rest is None:
|
||||
continue
|
||||
parts = [os.path.basename(folder), name] + (
|
||||
rest.split(os.sep) if rest else []
|
||||
)
|
||||
if all(part and part not in (".", "..") for part in parts):
|
||||
return "/".join(parts)
|
||||
return None
|
||||
|
||||
|
||||
class LibraryLocationRoutesMixin:
|
||||
"""Mixin for the /prompt_manager/system/library endpoints."""
|
||||
|
||||
# Replaced in tests with a resolver over a temp tree.
|
||||
_library_resolver = staticmethod(resolve_library_location)
|
||||
|
||||
def _register_library_location_routes(self, routes):
|
||||
@routes.get("/prompt_manager/system/library")
|
||||
async def library_location_route(request):
|
||||
return await self.get_library_location(request)
|
||||
|
||||
@routes.post("/prompt_manager/system/library/move")
|
||||
async def library_move_route(request):
|
||||
return await self.start_library_move(request)
|
||||
|
||||
@routes.post("/prompt_manager/system/library/keep")
|
||||
async def library_keep_route(request):
|
||||
return await self.keep_library(request)
|
||||
|
||||
# ── location ──────────────────────────────────────────────────────
|
||||
|
||||
def _library_public_path(self, path):
|
||||
"""The library's path for display: through the node folder's place in
|
||||
custom_nodes when it is there (a symlinked node folder included), else
|
||||
``_public_path``. Only the library location uses this; sources and
|
||||
jobs keep ``_public_path``, which their requests round-trip."""
|
||||
shown = node_folder_public_path(
|
||||
path, self._extension_root, _comfyui_custom_nodes()
|
||||
)
|
||||
return shown or self._public_path(path)
|
||||
|
||||
def _live_location(self):
|
||||
"""The resolved location, unless the live database is another file.
|
||||
|
||||
A database the API was handed (tests, the browser fixture server) or
|
||||
one the resolver no longer names is reported as configured and is
|
||||
never moved.
|
||||
"""
|
||||
location = self._library_resolver()
|
||||
live = self.db.model.db_path
|
||||
if _same_file(location.db_path, live):
|
||||
return location
|
||||
return ll.LibraryLocation(
|
||||
ll.SOURCE_CONFIG,
|
||||
live,
|
||||
location.config_path,
|
||||
os.path.dirname(live),
|
||||
config_explicit=location.config_explicit,
|
||||
)
|
||||
|
||||
def _location_payload(self, location):
|
||||
"""The public view of ``location`` (blocking: stats the database)."""
|
||||
from ..config import PromptManagerConfig
|
||||
|
||||
keep = bool(PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER)
|
||||
size = 0
|
||||
try:
|
||||
size = os.path.getsize(location.db_path)
|
||||
except OSError:
|
||||
pass
|
||||
return {
|
||||
"path": self._library_public_path(location.db_path),
|
||||
# Where Move library would put it, and whether config.json goes too.
|
||||
"target_source": location.target_source if location.movable else None,
|
||||
"config_moves": location.movable and not location.config_explicit,
|
||||
"source": location.source,
|
||||
"movable": location.movable,
|
||||
"keep": keep,
|
||||
"offer": location.movable and not keep,
|
||||
"target": (
|
||||
self._library_public_path(location.target_dir)
|
||||
if location.movable
|
||||
else None
|
||||
),
|
||||
"problem": location.problem or self._node_folder_risk(location),
|
||||
"size_bytes": size,
|
||||
}
|
||||
|
||||
# The node folder (tests point it at a temp tree).
|
||||
_extension_root = ll.EXTENSION_ROOT
|
||||
|
||||
def _node_folder_risk(self, location):
|
||||
"""``inside-node-folder`` for a library that stays in the node folder.
|
||||
|
||||
A configured database path, or a config.json the resolver still reads
|
||||
from the node folder, is deleted with the node by a reinstall, and
|
||||
it is never offered a move.
|
||||
"""
|
||||
if location.source == ll.SOURCE_NODE_FOLDER:
|
||||
return None
|
||||
paths = [location.db_path] + (
|
||||
[] if location.config_explicit else [location.config_path]
|
||||
)
|
||||
root = self._extension_root
|
||||
for path in paths:
|
||||
parent = os.path.dirname(os.path.abspath(path))
|
||||
if ll.same_folder(parent, root) or os.path.normcase(
|
||||
os.path.realpath(parent)
|
||||
).startswith(os.path.normcase(os.path.realpath(root)) + os.sep):
|
||||
return "inside-node-folder"
|
||||
return None
|
||||
|
||||
def _library_payload(self):
|
||||
return self._location_payload(self._live_location())
|
||||
|
||||
async def get_library_location(self, request):
|
||||
try:
|
||||
payload = await self._run_in_executor(self._library_payload)
|
||||
except Exception:
|
||||
self.logger.exception("Library location could not be read")
|
||||
return _failure(
|
||||
"The library location could not be read.", 500, "location-failed"
|
||||
)
|
||||
return web.json_response(
|
||||
{"success": True, "location": payload},
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
# ── move ──────────────────────────────────────────────────────────
|
||||
|
||||
def _plan_library_move(self):
|
||||
"""The checked plan (blocking); raises MoveFailed."""
|
||||
plan = lm.plan_move(self._live_location())
|
||||
if lm.space_problem(plan) is not None:
|
||||
raise lm.MoveFailed("insufficient-space")
|
||||
return plan
|
||||
|
||||
async def start_library_move(self, request):
|
||||
blocker = self._jobs.blocker(LIBRARY_MOVE_JOB_KIND, admission="exclusive")
|
||||
if blocker is not None:
|
||||
return busy_response(JobBusy(blocker))
|
||||
try:
|
||||
await self._run_in_executor(self._plan_library_move)
|
||||
except lm.MoveFailed as refused:
|
||||
status = _REFUSAL_STATUS.get(refused.code, 400)
|
||||
return _failure(refused.message, status, refused.code)
|
||||
except Exception:
|
||||
self.logger.exception("Move library could not be planned")
|
||||
return _failure(NOT_MOVED, 500, "move-failed")
|
||||
try:
|
||||
job = self._jobs.start(
|
||||
LIBRARY_MOVE_JOB_KIND,
|
||||
LIBRARY_MOVE_TITLE,
|
||||
admission="exclusive",
|
||||
cancellable=False,
|
||||
scope={},
|
||||
groups=LIBRARY_MOVE_GROUPS,
|
||||
runner=self._run_library_move,
|
||||
)
|
||||
except JobBusy as busy:
|
||||
return busy_response(busy)
|
||||
return web.json_response({"success": True, "job": job.summary()}, status=202)
|
||||
|
||||
def _after_library_switch(self):
|
||||
"""Forget what described the old file (best effort, after the switch).
|
||||
|
||||
Plans, the health report and the history read fence named the old
|
||||
file; the fence is closed before it is dropped.
|
||||
"""
|
||||
self._database_replaced()
|
||||
fence, self._history_read_fence = self._history_read_fence, None
|
||||
if fence is not None:
|
||||
fence.close()
|
||||
|
||||
def finish_interrupted_library_move(self):
|
||||
"""Startup: rename old files a crash left after a move (blocking)."""
|
||||
try:
|
||||
location = self._library_resolver()
|
||||
if _same_file(location.db_path, self.db.model.db_path):
|
||||
return lm.finish_interrupted_move(location)
|
||||
except Exception:
|
||||
self.logger.exception("Could not finish an interrupted library move")
|
||||
return []
|
||||
|
||||
async def _run_library_move(self, job):
|
||||
from ..config_store import _COMMIT_LOCK
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def progress(line):
|
||||
def show():
|
||||
job.log_line(line)
|
||||
job.update(status_line=line)
|
||||
|
||||
loop.call_soon_threadsafe(show)
|
||||
|
||||
job.update(status_line="Checking the new folder")
|
||||
try:
|
||||
plan = await self._run_in_executor(self._plan_library_move)
|
||||
target = self._library_public_path(plan.target_dir)
|
||||
job.log_line(f"Moving the library to {target}")
|
||||
result = await self._run_in_executor(
|
||||
functools.partial(
|
||||
lm.move_library,
|
||||
plan,
|
||||
after_switch=self._after_library_switch,
|
||||
config_lock=_COMMIT_LOCK,
|
||||
progress=progress,
|
||||
)
|
||||
)
|
||||
except lm.MoveFailed as failed:
|
||||
self.logger.error(f"Move library failed: {failed.code}")
|
||||
return self._fail_library_move(
|
||||
job, failed.code, f"{failed.message} {STILL_THERE}"
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Move library failed")
|
||||
return self._fail_library_move(job, None, f"{NOT_MOVED} {STILL_THERE}")
|
||||
await asyncio.sleep(0) # the progress lines come first
|
||||
job.record_counted("files", len(result.files))
|
||||
line = (
|
||||
f"Moved the library to {target}: "
|
||||
f"{_plural(result.counts.get('prompts', 0), 'prompt')} and "
|
||||
f"{_plural(result.counts.get('generated_images', 0), 'image')} verified"
|
||||
)
|
||||
if result.retire_problem:
|
||||
line += ". " + OLD_LEFT.get(result.retire_problem, OLD_LEFT["old-locked"])
|
||||
elif result.renamed:
|
||||
line += (
|
||||
f". The old files were renamed {', '.join(result.renamed)}; "
|
||||
"nothing was deleted"
|
||||
)
|
||||
line += ". Logs move at the next restart."
|
||||
job.log_line(line)
|
||||
job.update(progress=100, status_line=line)
|
||||
|
||||
@staticmethod
|
||||
def _fail_library_move(job, code, message):
|
||||
from .maintenance import _fail_job
|
||||
|
||||
_fail_job(job, code, message)
|
||||
|
||||
# ── keep it here ─────────────────────────────────────────────────
|
||||
|
||||
async def keep_library(self, request):
|
||||
from ..config import PromptManagerConfig
|
||||
from ..config_store import get_config_store
|
||||
|
||||
try:
|
||||
if (request.content_length or 0) > KEEP_BODY_MAX:
|
||||
raise ValueError("too large")
|
||||
data = await request.json()
|
||||
except Exception:
|
||||
data = None
|
||||
if not isinstance(data, dict) or not isinstance(data.get("keep"), bool):
|
||||
return _failure("keep must be true or false.", 400, "invalid-request")
|
||||
keep = data["keep"]
|
||||
|
||||
def mutate(config):
|
||||
library = config.get("library")
|
||||
library = dict(library) if isinstance(library, dict) else {}
|
||||
library["keep_in_node_folder"] = keep
|
||||
config["library"] = library
|
||||
return config
|
||||
|
||||
def publish(_config):
|
||||
PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER = keep
|
||||
|
||||
try:
|
||||
await self._run_in_executor(
|
||||
functools.partial(
|
||||
get_config_store().commit,
|
||||
mutate,
|
||||
expected_revision=None,
|
||||
publish=publish,
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
return self._config_commit_failure(exc)
|
||||
return await self.get_library_location(request)
|
||||
@@ -13,6 +13,7 @@ checks one path and writes nothing.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import functools
|
||||
import os
|
||||
|
||||
from aiohttp import web
|
||||
@@ -23,9 +24,11 @@ from ..config_store import (
|
||||
get_config_store,
|
||||
)
|
||||
from . import sources
|
||||
from . import thumbnail_encode
|
||||
from . import thumbnail_settings
|
||||
from .admin import _current_image_monitor
|
||||
from .job_routes import busy_response
|
||||
from .jobs import JobBusy
|
||||
from .jobs import THUMBNAILS_JOB_KIND, JobBusy
|
||||
|
||||
try:
|
||||
from ...utils.validators import validate_result_timeout, validate_worker_threads
|
||||
@@ -41,7 +44,12 @@ _SETTINGS_FILE_KEYS = {
|
||||
"WORKER_THREADS": ("performance", "worker_threads"),
|
||||
"MONITORING_ENABLED": ("gallery", "monitoring", "enabled"),
|
||||
"SAFETY_BACKUP": ("maintenance", "safety_backup"),
|
||||
"THUMBNAIL_FORMAT": ("thumbnails", "format"),
|
||||
"THUMBNAIL_QUALITY": ("thumbnails", "quality"),
|
||||
"THUMBNAIL_EDGE": ("thumbnails", "edge"),
|
||||
}
|
||||
# The thumbnail settings a running Convert outdated job fixes (#317).
|
||||
_THUMBNAIL_SETTINGS = ("THUMBNAIL_FORMAT", "THUMBNAIL_QUALITY", "THUMBNAIL_EDGE")
|
||||
# Attributes that live on GalleryConfig; the others on PromptManagerConfig.
|
||||
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"})
|
||||
SOURCE_CHANGE_JOB_KIND = "source_change"
|
||||
@@ -176,12 +184,15 @@ class SettingsRoutesMixin:
|
||||
]
|
||||
revision = await self._run_in_executor(self._config_revision)
|
||||
source_status, monitor = await self._run_in_executor(self._source_status)
|
||||
# #317: which thumbnail formats can be written (a cached probe).
|
||||
encoders = await self._run_in_executor(thumbnail_encode.encoder_status)
|
||||
return web.json_response(
|
||||
{
|
||||
"success": True,
|
||||
**revision,
|
||||
"sources": source_status,
|
||||
"monitor": monitor,
|
||||
"thumbnail_encoders": encoders,
|
||||
"settings": {
|
||||
"monitoring_enabled": GalleryConfig.MONITORING_ENABLED,
|
||||
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
|
||||
@@ -189,6 +200,9 @@ class SettingsRoutesMixin:
|
||||
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
|
||||
"worker_threads": PromptManagerConfig.WORKER_THREADS,
|
||||
"safety_backup": PromptManagerConfig.SAFETY_BACKUP,
|
||||
"thumbnail_format": PromptManagerConfig.THUMBNAIL_FORMAT,
|
||||
"thumbnail_quality": PromptManagerConfig.THUMBNAIL_QUALITY,
|
||||
"thumbnail_edge": PromptManagerConfig.THUMBNAIL_EDGE,
|
||||
"cpu_count": PromptManagerConfig.max_worker_threads(),
|
||||
"gallery_root_paths": root_paths,
|
||||
"gallery_root_path": root_paths[0] if root_paths else "",
|
||||
@@ -295,6 +309,26 @@ class SettingsRoutesMixin:
|
||||
return web.json_response(
|
||||
{"success": False, "error": str(ve)}, status=400
|
||||
)
|
||||
if updates.get("THUMBNAIL_FORMAT") == "jxl":
|
||||
# #317: JPEG XL only with an encoder (a cached probe).
|
||||
encoder = await self._run_in_executor(
|
||||
functools.partial(
|
||||
thumbnail_encode.detect_jxl_encoder, recheck_missing=True
|
||||
)
|
||||
)
|
||||
if encoder is None:
|
||||
return web.json_response(
|
||||
{
|
||||
"success": False,
|
||||
"code": "jxl-unavailable",
|
||||
"error": thumbnail_encode.JXL_UNAVAILABLE,
|
||||
},
|
||||
status=400,
|
||||
)
|
||||
|
||||
refused = self._thumbnail_change_refused(updates)
|
||||
if refused is not None:
|
||||
return refused
|
||||
|
||||
# Off the loop: an accepted root is resolved (filesystem access).
|
||||
new_roots, error = await self._run_in_executor(
|
||||
@@ -355,6 +389,33 @@ class SettingsRoutesMixin:
|
||||
status=500,
|
||||
)
|
||||
|
||||
def _thumbnail_change_refused(self, updates):
|
||||
"""409 ``busy`` for a thumbnail format, quality or size change while
|
||||
Convert outdated runs (review I4): the job converts to the settings
|
||||
it started with, and a change under it would make it remove previews
|
||||
in the newly chosen format. Unchanged values pass."""
|
||||
from ..config import PromptManagerConfig
|
||||
|
||||
changed = any(
|
||||
key in updates and updates[key] != getattr(PromptManagerConfig, key)
|
||||
for key in _THUMBNAIL_SETTINGS
|
||||
)
|
||||
job = self._jobs.active(THUMBNAILS_JOB_KIND) if changed else None
|
||||
if job is None or job.scope.get("mode") != "convert":
|
||||
return None
|
||||
return web.json_response(
|
||||
{
|
||||
"success": False,
|
||||
"code": "busy",
|
||||
"error": (
|
||||
f"{job.title} is running. Change the thumbnail settings "
|
||||
"when it finishes, or cancel it first."
|
||||
),
|
||||
"running": {"id": job.id, "title": job.title},
|
||||
},
|
||||
status=409,
|
||||
)
|
||||
|
||||
def _publish_settings(self, updates, new_roots):
|
||||
"""Install committed settings at runtime (inside the config commit).
|
||||
|
||||
@@ -491,6 +552,18 @@ class SettingsRoutesMixin:
|
||||
updates["WORKER_THREADS"] = validate_worker_threads(
|
||||
data["worker_threads"], PromptManagerConfig.max_worker_threads()
|
||||
)
|
||||
if "thumbnail_format" in data:
|
||||
updates["THUMBNAIL_FORMAT"] = thumbnail_settings.validate_format(
|
||||
data["thumbnail_format"]
|
||||
)
|
||||
if "thumbnail_quality" in data:
|
||||
updates["THUMBNAIL_QUALITY"] = thumbnail_settings.validate_quality(
|
||||
data["thumbnail_quality"]
|
||||
)
|
||||
if "thumbnail_edge" in data:
|
||||
updates["THUMBNAIL_EDGE"] = thumbnail_settings.validate_edge(
|
||||
data["thumbnail_edge"]
|
||||
)
|
||||
return updates
|
||||
|
||||
def _parse_gallery_roots(self, data):
|
||||
|
||||
+25
-1
@@ -13,7 +13,9 @@
|
||||
- ``describe_sources`` (blocking: it resolves paths) reports the configured
|
||||
sources with their validity, the effective sources with their listing
|
||||
index and watch state, and the image monitor's state. Every path in it is
|
||||
a public path (``_public_path``), never an absolute one.
|
||||
a public path (``_public_path``), never an absolute one; ``in_comfyui``
|
||||
says whether an effective source's public path is relative to the ComfyUI
|
||||
base (``in_comfyui_base``), so the page can read it as ``ComfyUI/<path>``.
|
||||
|
||||
A source's watch state is ``watching`` when the monitor's observer is alive,
|
||||
scheduled that directory and monitoring is on; ``stopping`` when it still
|
||||
@@ -209,6 +211,27 @@ def watch_state(path, *, present, alive, scheduled_keys, enabled):
|
||||
return "restart_pending"
|
||||
|
||||
|
||||
def in_comfyui_base(path):
|
||||
"""Whether ``_public_path(path)`` is relative to the ComfyUI base directory.
|
||||
|
||||
The base is the first anchor ``_public_path`` tries, so this agrees with
|
||||
it: the same canonical (realpath + normcase) spelling on both sides.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
from ..config import _comfyui_base_directory
|
||||
|
||||
base = _comfyui_base_directory()
|
||||
if not path or not base:
|
||||
return False
|
||||
try:
|
||||
canonical = Path(os.path.normcase(os.path.realpath(str(path))))
|
||||
anchor = Path(os.path.normcase(os.path.realpath(base)))
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
return canonical.is_relative_to(anchor)
|
||||
|
||||
|
||||
def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled):
|
||||
"""``(sources, monitor)`` for GET /settings (blocking; public paths only).
|
||||
|
||||
@@ -233,6 +256,7 @@ def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled
|
||||
scheduled_keys=scheduled_keys,
|
||||
enabled=enabled,
|
||||
),
|
||||
"in_comfyui": in_comfyui_base(path),
|
||||
}
|
||||
for index, path in enumerate(output_dirs)
|
||||
]
|
||||
|
||||
+4
-1
@@ -168,9 +168,10 @@ class SystemRoutesMixin:
|
||||
from ..config import GalleryConfig
|
||||
|
||||
try:
|
||||
(output_dirs, auto_detected), counts = await _gather_settled(
|
||||
(output_dirs, auto_detected), counts, location = await _gather_settled(
|
||||
self._run_in_executor(self._output_dirs_with_origin),
|
||||
self._run_in_executor(self.db.library_counts),
|
||||
self._run_in_executor(self._library_payload),
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("System overview failed")
|
||||
@@ -194,6 +195,8 @@ class SystemRoutesMixin:
|
||||
"effective": len(output_dirs),
|
||||
"auto_detected": auto_detected,
|
||||
},
|
||||
# Where the library lives and why (py/api/library_location.py).
|
||||
"location": location,
|
||||
},
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
@@ -0,0 +1,865 @@
|
||||
"""Automatic thumbnails (#249): previews are a cache the server keeps itself.
|
||||
|
||||
The approved mock has no repeated global thumbnail prompt
|
||||
(``docs/concepts/workspace/README.md``), so the workspace never asks. Every
|
||||
listing gives an image or video a thumbnail URL (``media_thumbnail``), and the
|
||||
previews behind those URLs are made here, in three ways:
|
||||
|
||||
- **On first view.** ``GET /prompt_manager/images/serve/thumbnails/<rel>_thumb
|
||||
<ext>`` for a preview that does not exist yet is admitted on the event loop
|
||||
(``_PreviewGate``): at most ``ON_DEMAND_LIMIT`` previews are written at once,
|
||||
on this API's own small pool (never the default executor), one flight runs
|
||||
per file, and the requests waiting are capped (``MAX_WAITING``); past the
|
||||
cap a request gets its fallback at once. Inside the admission the preview is
|
||||
mapped back to its media file under the same root (the request text is
|
||||
checked first: no traversal, drive, UNC or empty segment; then a regular
|
||||
file inside the root, never through a link), made and served. When it
|
||||
cannot be made an image's original is served and a video's placeholder
|
||||
frame, both ``Cache-Control: no-store``: a video is never sent to an
|
||||
``<img>``. A file that cannot be decoded is remembered for this session
|
||||
(keyed by its path, size and mtime) and counted for the System Overview
|
||||
(``failed`` in ``/images/thumbnails/status``); I/O errors and a preview
|
||||
another program holds open (Windows) are not remembered. The recorded
|
||||
history's thumbnail route goes through the same admission and flight key.
|
||||
- **On capture.** The image monitor hands every new output it settles to
|
||||
``queue_capture_thumbnail``, which queues its preview for one background
|
||||
worker thread (``AutoThumbnailer``). Each wake takes at most Worker threads
|
||||
(``WORKER_THREADS``) files and writes them on that many threads; enqueuing
|
||||
is O(1), so the monitor and ComfyUI's queue never wait. A file already
|
||||
queued is not queued again, the queue is bounded (``MAX_QUEUED``) and a
|
||||
preview left out is made on its first view instead.
|
||||
- **After a scan.** A scan that added files checks the missing previews; from
|
||||
``SCAN_THUMBNAIL_JOB_THRESHOLD`` up it starts the thumbnail Activity job
|
||||
(missing only, visible and cancellable) instead of writing them silently,
|
||||
and fewer go through the worker.
|
||||
|
||||
Every preview is written by ``_generate_one`` with the server's thumbnail
|
||||
settings (format, quality, size limit; ``thumbnail_settings``, #317), into a
|
||||
temp sibling verified and renamed over the old one, so a reader never sees a
|
||||
partial file. A request for an older preview name (``_thumb.png``) is answered
|
||||
with the preview in the current format.
|
||||
|
||||
Holds: automatic previews only write files under ``thumbnails/`` and never
|
||||
touch the database, like the thumbnail job, so they run beside a scan, image
|
||||
tagging and the Needs review recording (S5 M7). They are not written while a
|
||||
job runs that replaces the database or removes media and previews: a restore,
|
||||
a duplicate removal, a sources change, the gallery's thumbnail clear or an
|
||||
``exclusive`` job (``_auto_thumbnail_blocker``). The holds are checked when a
|
||||
write starts (after its slot) and again just before its rename.
|
||||
|
||||
Shutdown: ``close_auto_thumbnails`` stops the worker at a file boundary,
|
||||
kills a running ffmpeg, waits a bounded time (and logs when the wait ran
|
||||
out) and shuts the view pool down; after the app's cleanup
|
||||
(``final=True``) nothing starts a worker again. Importing this module again
|
||||
(a package reload) stops the workers the previous import left.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import collections
|
||||
import functools
|
||||
import io
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from . import thumbnail_settings
|
||||
from .images import (
|
||||
MEDIA_EXTENSIONS,
|
||||
THUMBNAIL_CLEAR_TITLE,
|
||||
VIDEO_EXTENSIONS,
|
||||
_has_traversal,
|
||||
_is_within,
|
||||
_thumbnail_is_fresh,
|
||||
_video_placeholder_image,
|
||||
_worker_threads,
|
||||
media_thumbnail_rel,
|
||||
)
|
||||
from .jobs import (
|
||||
DUPLICATE_CLEANUP_JOB_KIND,
|
||||
RESTORE_JOB_KIND,
|
||||
THUMBNAILS_JOB_KIND,
|
||||
JobBusy,
|
||||
)
|
||||
from .settings_routes import SOURCE_CHANGE_JOB_KIND
|
||||
from .thumbnail_encode import shutdown_encoders, writer_settings
|
||||
from .thumbnail_jobs import ThumbnailOptions
|
||||
|
||||
try:
|
||||
from ...utils.parallel import map_parallel
|
||||
except ImportError: # tests import the package top-level
|
||||
from utils.parallel import map_parallel
|
||||
|
||||
# The placeholder frame a video without a preview is answered with (JPEG
|
||||
# bytes, never written to disk); written previews follow the settings.
|
||||
PLACEHOLDER_SIZE = (thumbnail_settings.DEFAULT_EDGE, thumbnail_settings.DEFAULT_EDGE)
|
||||
PLACEHOLDER_QUALITY = 85
|
||||
# Source extensions in the order a shared preview name looks for them (R2).
|
||||
_SOURCE_ORDER = thumbnail_settings.IMAGE_SOURCE_ORDER + tuple(sorted(VIDEO_EXTENSIONS))
|
||||
# Previews views may write at once (the size of the view pool).
|
||||
ON_DEMAND_LIMIT = 3
|
||||
# View requests that may wait on the gate; one more gets its fallback at once.
|
||||
MAX_WAITING = 64
|
||||
# How long a pool thread waits for the capture worker's flight of the same file.
|
||||
FLIGHT_WAIT_SECONDS = 20
|
||||
# How long close() waits for the worker's batch in hand (ffmpeg's own limit).
|
||||
CLOSE_WAIT_SECONDS = 30
|
||||
# Files the capture worker holds; more are made on their first view.
|
||||
MAX_QUEUED = 10_000
|
||||
# Undecodable files remembered per session (oldest forgotten first).
|
||||
MAX_FAILURES_REMEMBERED = 4096
|
||||
# Missing previews after a scan from which the visible thumbnail job starts:
|
||||
# a fresh install or a newly added source. Fewer go through the worker.
|
||||
SCAN_THUMBNAIL_JOB_THRESHOLD = 200
|
||||
# Jobs beside which no automatic preview is written (see the module docstring).
|
||||
AUTO_REFUSED_KINDS = frozenset(
|
||||
{RESTORE_JOB_KIND, DUPLICATE_CLEANUP_JOB_KIND, SOURCE_CHANGE_JOB_KIND}
|
||||
)
|
||||
WORKER_NAME = "PromptManagerThumbnailWorker"
|
||||
POOL_PREFIX = "pm-preview"
|
||||
# Identifies this import's workers: a reload stops the ones it did not start.
|
||||
_MODULE_TOKEN = object()
|
||||
|
||||
READY, FAILED, BUSY, DEFERRED = "ready", "failed", "busy", "deferred"
|
||||
# What the gate answers a request past MAX_WAITING.
|
||||
OVER = object()
|
||||
|
||||
|
||||
def _fingerprint(src):
|
||||
"""``(path, size, mtime_ns)`` of *src*, or None when it cannot be read."""
|
||||
try:
|
||||
stat = os.stat(src)
|
||||
except OSError:
|
||||
return None
|
||||
return (os.path.normcase(str(src)), stat.st_size, stat.st_mtime_ns)
|
||||
|
||||
|
||||
def flight_key(root, rel_parts):
|
||||
"""The single-flight key of ``root/rel_parts``: its folder resolved, case
|
||||
folded. The serve route, the history route and the worker share it."""
|
||||
path = Path(root).joinpath(*rel_parts)
|
||||
return os.path.normcase(os.path.join(os.path.realpath(path.parent), path.name))
|
||||
|
||||
|
||||
def _dst_key(dst):
|
||||
return flight_key(Path(dst).parent, (Path(dst).name,))
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def video_placeholder_jpeg():
|
||||
"""The placeholder frame as JPEG bytes (a video whose preview is missing)."""
|
||||
buffer = io.BytesIO()
|
||||
_video_placeholder_image(PLACEHOLDER_SIZE).save(
|
||||
buffer, "JPEG", quality=PLACEHOLDER_QUALITY, optimize=True
|
||||
)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def _is_preview_request(filepath):
|
||||
"""Text check only: ``thumbnails/<...>/<name>_thumb<lower-case ext>``.
|
||||
|
||||
Refuses, as text and before any path is joined, what a joined path could
|
||||
take as an anchor: a backslash, a drive or stream colon, and empty, ``.``
|
||||
or ``..`` segments (so a leading ``/`` and ``//server/share`` too).
|
||||
"""
|
||||
if "\\" in filepath or ":" in filepath:
|
||||
return False
|
||||
parts = filepath.split("/")
|
||||
if len(parts) < 2 or parts[0] != "thumbnails":
|
||||
return False
|
||||
if any(part in ("", ".", "..") for part in parts) or "thumbnails" in parts[1:-1]:
|
||||
return False
|
||||
stem, extension = os.path.splitext(parts[-1])
|
||||
return (
|
||||
stem.endswith("_thumb")
|
||||
and len(stem) > len("_thumb")
|
||||
and extension == extension.lower()
|
||||
and extension in thumbnail_settings.THUMBNAIL_EXTENSIONS
|
||||
)
|
||||
|
||||
|
||||
def _regular_file(path):
|
||||
"""True for a regular file that is not a link (one lstat)."""
|
||||
try:
|
||||
return not path.is_symlink() and path.is_file()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _candidates(parts):
|
||||
"""``(folder parts, base, source extensions)`` of a checked preview
|
||||
request: every media extension that owns the requested preview name
|
||||
(``thumbnail_settings.owns``), images first (ruling R2)."""
|
||||
stem, extension = os.path.splitext(parts[-1])
|
||||
wanted = [
|
||||
source
|
||||
for source in _SOURCE_ORDER
|
||||
if source in MEDIA_EXTENSIONS and thumbnail_settings.owns(extension, source)
|
||||
]
|
||||
return parts[1:-1], stem[: -len("_thumb")], wanted
|
||||
|
||||
|
||||
def _current_preview(root, parts):
|
||||
"""The preview *parts* names, in the format written now (#317): a request
|
||||
for an older name (``_thumb.png`` before a conversion) is answered with
|
||||
the current file."""
|
||||
_folder, base, _wanted = _candidates(parts)
|
||||
name = base + thumbnail_settings.THUMB_SUFFIX + writer_settings().extension
|
||||
return Path(root).joinpath(*parts[:-1], name)
|
||||
|
||||
|
||||
def _quick_source(root, parts):
|
||||
"""``(src, is_video)`` by exact name only (no folder listing), or None."""
|
||||
root = Path(root)
|
||||
folder_parts, base, wanted = _candidates(parts)
|
||||
folder = root.joinpath(*folder_parts)
|
||||
for extension in wanted:
|
||||
for name in (base + extension, base + extension.upper()):
|
||||
path = folder / name
|
||||
if _regular_file(path) and _is_within(path, root):
|
||||
return path, extension in VIDEO_EXTENSIONS
|
||||
return None
|
||||
|
||||
|
||||
def _thumbnail_source(root, parts):
|
||||
"""``(src, dst, is_video)`` for the preview *parts* under *root*, or None.
|
||||
|
||||
*parts* passed ``_is_preview_request``. The source has the preview's name
|
||||
without ``_thumb`` and an extension that owns the requested one (any case:
|
||||
exact names first, then one listing of the folder; ruling R4). It must be
|
||||
a regular file inside *root*. ``dst`` is the preview in the format written
|
||||
now (#317), whatever extension was requested; it is None when the
|
||||
preview's folder resolves outside ``root/thumbnails`` (a link): nothing is
|
||||
written there.
|
||||
"""
|
||||
root = Path(root)
|
||||
folder_parts, base, wanted = _candidates(parts)
|
||||
quick = _quick_source(root, parts)
|
||||
if quick is not None:
|
||||
found = quick[0]
|
||||
else:
|
||||
found = _scan_folder(root.joinpath(*folder_parts), base, set(wanted))
|
||||
if found is None or not _is_within(found, root):
|
||||
return None
|
||||
rel = Path(*folder_parts, found.name)
|
||||
dst = root / media_thumbnail_rel(rel)
|
||||
if dst != _current_preview(root, parts):
|
||||
return None
|
||||
if not _is_within(root / "thumbnails", root) or not _is_within(
|
||||
dst, root / "thumbnails"
|
||||
):
|
||||
dst = None
|
||||
return found, dst, found.suffix.lower() in VIDEO_EXTENSIONS
|
||||
|
||||
|
||||
def _scan_folder(folder, base, extensions):
|
||||
"""A mixed-case extension (``.Png``): one listing of *folder*, or None."""
|
||||
try:
|
||||
with os.scandir(folder) as entries:
|
||||
for entry in entries:
|
||||
stem, extension = os.path.splitext(entry.name)
|
||||
if stem == base and extension.lower() in extensions:
|
||||
path = Path(entry.path)
|
||||
if _regular_file(path):
|
||||
return path
|
||||
except OSError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _stop_stale_workers():
|
||||
"""Stop capture workers a previous import of this module started."""
|
||||
for thread in threading.enumerate():
|
||||
token = getattr(thread, "pm_preview_token", None)
|
||||
close = getattr(thread, "pm_preview_close", None)
|
||||
if token is not None and token is not _MODULE_TOKEN and close is not None:
|
||||
close()
|
||||
|
||||
|
||||
class AutoThumbnailer:
|
||||
"""Single-flight preview writes and the capture worker (thread-safe).
|
||||
|
||||
``generate(src, dst, is_video, *, stop, guard, track)`` writes one
|
||||
preview and returns ``_generate_one``'s result dict; ``may_write()``
|
||||
says whether automatic previews may be written now. Views call
|
||||
``ensure`` on the view pool, the monitor calls ``enqueue`` from its
|
||||
thread.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
generate,
|
||||
may_write,
|
||||
logger,
|
||||
*,
|
||||
workers=_worker_threads,
|
||||
max_queued=MAX_QUEUED,
|
||||
start_worker=True,
|
||||
):
|
||||
self._generate = generate
|
||||
self._may_write = may_write
|
||||
self._logger = logger
|
||||
self._workers = workers
|
||||
self._max_queued = max_queued
|
||||
self._start_worker = start_worker
|
||||
self._changed = threading.Condition()
|
||||
self._inflight = {}
|
||||
self._failures = collections.OrderedDict()
|
||||
self._queue = collections.deque()
|
||||
self._queued = set()
|
||||
self._processes = []
|
||||
self._worker = None
|
||||
# True from a worker's start until it leaves (under the lock), so an
|
||||
# idle worker can exit and the next enqueue starts a fresh one.
|
||||
self._worker_running = False
|
||||
self._closed = False
|
||||
self._busy = False
|
||||
self._batch = 0
|
||||
|
||||
@property
|
||||
def failure_count(self):
|
||||
with self._changed:
|
||||
return len(self._failures)
|
||||
|
||||
@property
|
||||
def closed(self):
|
||||
return self._closed
|
||||
|
||||
def _allowed(self):
|
||||
return not self._closed and self._may_write()
|
||||
|
||||
def _track(self, process):
|
||||
"""Keep an ffmpeg process so close() can kill it."""
|
||||
with self._changed:
|
||||
self._processes = [p for p in self._processes if p.poll() is None]
|
||||
self._processes.append(process)
|
||||
closed = self._closed
|
||||
if closed:
|
||||
process.kill()
|
||||
|
||||
# ── one preview ───────────────────────────────────────────────────
|
||||
|
||||
def ensure(self, src, dst, is_video):
|
||||
"""Make *dst* for *src* unless it is fresh; returns the outcome.
|
||||
|
||||
``ready`` (the preview is there), ``failed`` (the file cannot be
|
||||
decoded, remembered), ``busy`` (an I/O error, the file is gone, or
|
||||
another flight did not finish) or ``deferred`` (a hold refuses
|
||||
automatic writes, checked when the write starts and before its
|
||||
rename). Blocking: call it off the event loop.
|
||||
"""
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
return READY
|
||||
fingerprint = _fingerprint(src)
|
||||
if fingerprint is None:
|
||||
return BUSY
|
||||
key = _dst_key(dst)
|
||||
with self._changed:
|
||||
if fingerprint in self._failures:
|
||||
return FAILED
|
||||
flight = self._inflight.get(key)
|
||||
if flight is None:
|
||||
self._inflight[key] = done = threading.Event()
|
||||
if flight is not None:
|
||||
flight.wait(FLIGHT_WAIT_SECONDS)
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
return READY
|
||||
with self._changed:
|
||||
return FAILED if fingerprint in self._failures else BUSY
|
||||
try:
|
||||
return self._lead(src, dst, is_video, fingerprint)
|
||||
finally:
|
||||
with self._changed:
|
||||
self._inflight.pop(key, None)
|
||||
self._changed.notify_all()
|
||||
done.set()
|
||||
|
||||
def _lead(self, src, dst, is_video, fingerprint):
|
||||
if not self._allowed():
|
||||
return DEFERRED
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
return READY
|
||||
result = self._generate(
|
||||
src,
|
||||
dst,
|
||||
is_video,
|
||||
stop=lambda: self._closed,
|
||||
guard=self._allowed,
|
||||
track=self._track,
|
||||
)
|
||||
if result.get("action") in ("generated", "skipped"):
|
||||
return READY
|
||||
if result.get("held"):
|
||||
return DEFERRED
|
||||
if not result.get("decode"):
|
||||
return BUSY # an I/O error says nothing about the file
|
||||
error = result.get("error") or "failed"
|
||||
with self._changed:
|
||||
self._failures[fingerprint] = error
|
||||
while len(self._failures) > MAX_FAILURES_REMEMBERED:
|
||||
self._failures.popitem(last=False)
|
||||
self._logger.info(
|
||||
f"No preview for {os.path.basename(str(src))} ({error}); "
|
||||
"the original is shown"
|
||||
)
|
||||
return FAILED
|
||||
|
||||
# ── the capture worker ────────────────────────────────────────────
|
||||
|
||||
def enqueue(self, src, dst, is_video):
|
||||
"""Queue *dst*'s preview for the worker; False when already queued,
|
||||
full or closed. Never blocks on a write."""
|
||||
key = _dst_key(dst)
|
||||
with self._changed:
|
||||
if self._closed or key in self._queued:
|
||||
return False
|
||||
if len(self._queue) >= self._max_queued:
|
||||
return False
|
||||
self._queued.add(key)
|
||||
self._queue.append((key, Path(src), Path(dst), bool(is_video)))
|
||||
self._busy = True
|
||||
if self._start_worker and not self._worker_running:
|
||||
self._start_worker_locked()
|
||||
self._changed.notify_all()
|
||||
return True
|
||||
|
||||
def _start_worker_locked(self):
|
||||
"""Start the worker (lock held). A start that fails leaves the file
|
||||
queued and the next enqueue tries again."""
|
||||
worker = threading.Thread(target=self._run, name=WORKER_NAME, daemon=True)
|
||||
worker.pm_preview_token = _MODULE_TOKEN
|
||||
worker.pm_preview_close = self.close
|
||||
self._worker_running = True
|
||||
try:
|
||||
worker.start()
|
||||
except RuntimeError:
|
||||
self._worker_running = False
|
||||
self._busy = False
|
||||
self._logger.warning(
|
||||
"The automatic preview worker could not start; the next output "
|
||||
"tries again",
|
||||
exc_info=True,
|
||||
)
|
||||
return
|
||||
self._worker = worker
|
||||
|
||||
def _run(self):
|
||||
try:
|
||||
self._drain()
|
||||
except Exception:
|
||||
self._logger.warning("The automatic preview worker stopped", exc_info=True)
|
||||
finally:
|
||||
# Whatever ended the loop, the next enqueue starts a fresh worker.
|
||||
with self._changed:
|
||||
self._worker_running = False
|
||||
self._busy = False
|
||||
self._batch = 0
|
||||
self._changed.notify_all()
|
||||
|
||||
def _drain(self):
|
||||
while True:
|
||||
with self._changed:
|
||||
if self._closed or not self._queue:
|
||||
# Idle or closed: leave, so no thread outlives its work.
|
||||
self._busy = False
|
||||
self._worker_running = False
|
||||
self._changed.notify_all()
|
||||
return
|
||||
take = min(len(self._queue), max(1, int(self._workers() or 1)))
|
||||
batch = [self._queue.popleft() for _ in range(take)]
|
||||
self._batch = len(batch)
|
||||
self._changed.notify_all()
|
||||
try:
|
||||
map_parallel(self._write_queued, batch, len(batch))
|
||||
finally:
|
||||
with self._changed:
|
||||
self._batch = 0
|
||||
for key, *_rest in batch:
|
||||
self._queued.discard(key)
|
||||
|
||||
def _write_queued(self, item):
|
||||
_key, src, dst, is_video = item
|
||||
try:
|
||||
self.ensure(src, dst, is_video)
|
||||
except Exception: # one file never stops the worker
|
||||
self._logger.warning(
|
||||
f"Automatic preview failed for {src.name}", exc_info=True
|
||||
)
|
||||
|
||||
def close(self):
|
||||
"""Stop the worker at its next file boundary (queued files are
|
||||
dropped), kill a running ffmpeg and wait up to CLOSE_WAIT_SECONDS;
|
||||
a worker still running after that is logged."""
|
||||
with self._changed:
|
||||
self._closed = True
|
||||
self._queue.clear()
|
||||
self._queued.clear()
|
||||
processes, self._processes = self._processes, []
|
||||
self._changed.notify_all()
|
||||
worker = self._worker
|
||||
for process in processes:
|
||||
try:
|
||||
process.kill()
|
||||
except OSError:
|
||||
pass
|
||||
if worker is None or worker is threading.current_thread():
|
||||
return
|
||||
worker.join(CLOSE_WAIT_SECONDS)
|
||||
if worker.is_alive():
|
||||
self._logger.warning(
|
||||
"The automatic preview worker is still running after "
|
||||
f"{CLOSE_WAIT_SECONDS} s; it stops at its next file"
|
||||
)
|
||||
|
||||
# ── waits (tests and shutdown; never timers) ──────────────────────
|
||||
|
||||
def _wait_for(self, predicate, timeout):
|
||||
with self._changed:
|
||||
return self._changed.wait_for(predicate, timeout)
|
||||
|
||||
def wait_idle(self, timeout=None):
|
||||
"""True once the worker has nothing queued or in hand."""
|
||||
return self._wait_for(lambda: not self._busy, timeout)
|
||||
|
||||
def wait_batch(self, size, timeout=None):
|
||||
"""True once the worker holds a batch of *size* files."""
|
||||
return self._wait_for(lambda: self._batch == size, timeout)
|
||||
|
||||
|
||||
class _PreviewGate:
|
||||
"""Admission of view requests on the event loop (#249 I2).
|
||||
|
||||
Waiters are coroutines: at most ``limit`` previews are written at once
|
||||
on the view pool, one flight runs per key, and past ``MAX_WAITING``
|
||||
waiting requests ``run`` answers ``OVER`` at once.
|
||||
"""
|
||||
|
||||
def __init__(self, limit, logger):
|
||||
self.loop = asyncio.get_running_loop()
|
||||
self._slots = asyncio.Semaphore(limit)
|
||||
self._flights = {}
|
||||
self._tasks = set()
|
||||
self._logger = logger
|
||||
self._tick = asyncio.Event()
|
||||
self.waiting = 0
|
||||
|
||||
def _changed(self):
|
||||
tick, self._tick = self._tick, asyncio.Event()
|
||||
tick.set()
|
||||
|
||||
async def settled(self, predicate):
|
||||
"""Return once ``predicate()`` holds (re-checked at each change)."""
|
||||
while not predicate():
|
||||
await self._tick.wait()
|
||||
|
||||
async def run(self, key, work, pool):
|
||||
"""``work()``'s answer for *key* (shared by every request of the
|
||||
flight), None when it raised, or OVER past the cap."""
|
||||
if self.waiting >= MAX_WAITING:
|
||||
return OVER
|
||||
flight = self._flights.get(key)
|
||||
if flight is None:
|
||||
flight = self.loop.create_future()
|
||||
self._flights[key] = flight
|
||||
task = self.loop.create_task(self._lead(key, flight, work, pool))
|
||||
self._tasks.add(task)
|
||||
task.add_done_callback(self._tasks.discard)
|
||||
self.waiting += 1
|
||||
self._changed()
|
||||
try:
|
||||
return await asyncio.shield(flight)
|
||||
finally:
|
||||
self.waiting -= 1
|
||||
self._changed()
|
||||
|
||||
async def _lead(self, key, flight, work, pool):
|
||||
result = None
|
||||
try:
|
||||
async with self._slots:
|
||||
result = await self.loop.run_in_executor(pool, work)
|
||||
except Exception:
|
||||
self._logger.warning("Automatic preview failed", exc_info=True)
|
||||
finally:
|
||||
self._flights.pop(key, None)
|
||||
if not flight.done():
|
||||
flight.set_result(result)
|
||||
|
||||
|
||||
class AutoThumbnailMixin:
|
||||
"""Automatic previews on the API: views, the monitor's queue, scans."""
|
||||
|
||||
_auto_thumbnailer = None
|
||||
_auto_thumbnails_shut = False
|
||||
_preview_gate_obj = None
|
||||
_preview_pool_obj = None
|
||||
_auto_thumbnailer_lock = threading.Lock()
|
||||
|
||||
def _auto_thumbnails(self):
|
||||
"""This API's AutoThumbnailer, created on first use (no worker after
|
||||
the app's cleanup)."""
|
||||
with AutoThumbnailMixin._auto_thumbnailer_lock:
|
||||
if self._auto_thumbnailer is None:
|
||||
self._auto_thumbnailer = AutoThumbnailer(
|
||||
self._write_auto_thumbnail,
|
||||
lambda: self._auto_thumbnail_blocker() is None,
|
||||
self.logger,
|
||||
start_worker=not self._auto_thumbnails_shut,
|
||||
)
|
||||
return self._auto_thumbnailer
|
||||
|
||||
def close_auto_thumbnails(self, final=False):
|
||||
"""Stop the capture worker and the view pool, and forget them.
|
||||
|
||||
Bounded (see AutoThumbnailer.close). ``final`` (the app's cleanup)
|
||||
also keeps a late monitor event from starting a worker again; a
|
||||
test reset passes False and the next use starts fresh ones.
|
||||
"""
|
||||
with AutoThumbnailMixin._auto_thumbnailer_lock:
|
||||
if final:
|
||||
self._auto_thumbnails_shut = True
|
||||
thumbnailer, self._auto_thumbnailer = self._auto_thumbnailer, None
|
||||
pool, self._preview_pool_obj = self._preview_pool_obj, None
|
||||
self._preview_gate_obj = None
|
||||
if thumbnailer is not None:
|
||||
thumbnailer.close()
|
||||
if pool is not None:
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
if final:
|
||||
shutdown_encoders() # kill JPEG XL / ffmpeg encoders still running
|
||||
|
||||
def _auto_thumbnail_failures(self):
|
||||
thumbnailer = self._auto_thumbnailer
|
||||
return 0 if thumbnailer is None else thumbnailer.failure_count
|
||||
|
||||
def _write_auto_thumbnail(self, src, dst, is_video, *, stop, guard, track):
|
||||
return self._generate_one(
|
||||
src,
|
||||
dst,
|
||||
writer_settings(),
|
||||
is_video,
|
||||
stop=stop,
|
||||
guard=guard,
|
||||
track=track,
|
||||
)
|
||||
|
||||
def _auto_thumbnail_blocker(self):
|
||||
"""The running job automatic previews wait for, or None (any thread)."""
|
||||
for job in self._jobs.running():
|
||||
if job.admission == "exclusive" or job.kind in AUTO_REFUSED_KINDS:
|
||||
return job
|
||||
if job.kind == THUMBNAILS_JOB_KIND and job.title == THUMBNAIL_CLEAR_TITLE:
|
||||
return job
|
||||
return None
|
||||
|
||||
# ── the view admission ────────────────────────────────────────────
|
||||
|
||||
def _preview_gate(self):
|
||||
"""The gate of the running loop (a new loop gets a new one)."""
|
||||
gate = self._preview_gate_obj
|
||||
if gate is None or gate.loop is not asyncio.get_running_loop():
|
||||
gate = self._preview_gate_obj = _PreviewGate(ON_DEMAND_LIMIT, self.logger)
|
||||
return gate
|
||||
|
||||
def _preview_pool(self):
|
||||
with AutoThumbnailMixin._auto_thumbnailer_lock:
|
||||
if self._preview_pool_obj is None:
|
||||
self._preview_pool_obj = ThreadPoolExecutor(
|
||||
ON_DEMAND_LIMIT, thread_name_prefix=POOL_PREFIX
|
||||
)
|
||||
return self._preview_pool_obj
|
||||
|
||||
def _preview_pool_threads(self):
|
||||
pool = self._preview_pool_obj
|
||||
return [] if pool is None else list(getattr(pool, "_threads", ()))
|
||||
|
||||
def _video_placeholder_bytes(self):
|
||||
return video_placeholder_jpeg()
|
||||
|
||||
async def _admit_preview(self, src, dst, is_video):
|
||||
"""Make one preview through the view admission (the history route).
|
||||
|
||||
Shares the serve route's flight key; every flight answers the
|
||||
``ensure`` outcome (or None, or OVER), whichever route started it.
|
||||
"""
|
||||
return await self._preview_gate().run(
|
||||
flight_key(Path(dst).parent, (Path(dst).name,)),
|
||||
functools.partial(self._ensure_view, src, dst, is_video),
|
||||
self._preview_pool(),
|
||||
)
|
||||
|
||||
def _ensure_view(self, src, dst, is_video):
|
||||
"""``ensure``, reporting a preview removed under us as busy."""
|
||||
outcome = self._auto_thumbnails().ensure(src, dst, is_video)
|
||||
if outcome == READY and not Path(dst).is_file():
|
||||
return BUSY # removed under us: the fallback this once
|
||||
return outcome
|
||||
|
||||
# ── on first view ─────────────────────────────────────────────────
|
||||
|
||||
def _preview_on_demand(self, root, parts):
|
||||
"""The ``ensure`` outcome for the preview *parts* under *root*
|
||||
(``deferred`` when its folder is linked out of the root), or None
|
||||
when no media file previews as *parts*."""
|
||||
located = _thumbnail_source(root, parts)
|
||||
if located is None:
|
||||
return None
|
||||
src, dst, is_video = located
|
||||
if dst is None:
|
||||
return DEFERRED
|
||||
return self._ensure_view(src, dst, is_video)
|
||||
|
||||
def _preview_response(self, outcome, src, dst, is_video):
|
||||
if outcome == READY and dst is not None:
|
||||
return self._file_response(dst)
|
||||
if is_video:
|
||||
return web.Response(
|
||||
body=video_placeholder_jpeg(),
|
||||
content_type="image/jpeg",
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
response = web.FileResponse(src)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return response
|
||||
|
||||
async def _serve_missing_thumbnail(self, output_dirs, filepath):
|
||||
"""The made preview, or its fallback; None when *filepath* names no
|
||||
preview of a media file in *output_dirs*.
|
||||
|
||||
The flight answers only the outcome (a history flight may be the one
|
||||
joined); the file it previews is derived here: the preview path is
|
||||
the request's own, and a fallback looks its source up again on the
|
||||
view pool (bounded by it, never the default executor).
|
||||
"""
|
||||
if _has_traversal(filepath) or not _is_preview_request(filepath):
|
||||
return None
|
||||
parts = tuple(filepath.split("/"))
|
||||
gate = self._preview_gate()
|
||||
pool = self._preview_pool()
|
||||
for root in output_dirs:
|
||||
root = Path(root)
|
||||
outcome = await gate.run(
|
||||
flight_key(root, parts),
|
||||
functools.partial(self._preview_on_demand, root, parts),
|
||||
pool,
|
||||
)
|
||||
if outcome is None:
|
||||
continue
|
||||
if outcome is OVER:
|
||||
# Past the cap: exact names only, no listing, no write.
|
||||
quick = _quick_source(root, parts)
|
||||
if quick is None:
|
||||
continue
|
||||
return self._preview_response(BUSY, quick[0], None, quick[1])
|
||||
dst = _current_preview(root, parts)
|
||||
if outcome == READY:
|
||||
return self._file_response(dst)
|
||||
located = await gate.loop.run_in_executor(
|
||||
pool, _thumbnail_source, root, parts
|
||||
)
|
||||
if located is None:
|
||||
continue
|
||||
src, _dst, is_video = located
|
||||
return self._preview_response(outcome, src, None, is_video)
|
||||
return None
|
||||
|
||||
# ── on capture ────────────────────────────────────────────────────
|
||||
|
||||
def queue_capture_thumbnail(self, image_path):
|
||||
"""Queue the preview of a new output (the monitor's thread).
|
||||
|
||||
Returns ``queued``, ``fresh`` (its preview is current) or ``skipped``
|
||||
(not a media file under a current root, a link, a preview itself,
|
||||
already queued, or after the app's cleanup). Never writes on the
|
||||
caller's thread.
|
||||
"""
|
||||
if self._auto_thumbnails_shut:
|
||||
return "skipped"
|
||||
located = self._capture_target(image_path)
|
||||
if located is None:
|
||||
return "skipped"
|
||||
src, dst, is_video = located
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
return "fresh"
|
||||
if self._auto_thumbnails().enqueue(src, dst, is_video):
|
||||
return "queued"
|
||||
return "skipped"
|
||||
|
||||
def _capture_target(self, image_path):
|
||||
path = Path(os.path.abspath(str(image_path)))
|
||||
if path.suffix.lower() not in MEDIA_EXTENSIONS or not _regular_file(path):
|
||||
return None
|
||||
folded = os.path.normcase(str(path))
|
||||
for root in self._get_all_output_dirs():
|
||||
root = Path(os.path.abspath(str(root)))
|
||||
base = os.path.normcase(str(root)).rstrip("\\/") + os.sep
|
||||
if not folded.startswith(base) or not _is_within(path, root):
|
||||
continue
|
||||
rel = Path(os.path.relpath(path, root))
|
||||
if "thumbnails" in rel.parts[:-1]:
|
||||
return None
|
||||
dst = root / media_thumbnail_rel(rel)
|
||||
if not _is_within(root / "thumbnails", root) or not _is_within(
|
||||
dst, root / "thumbnails"
|
||||
):
|
||||
return None
|
||||
return path, dst, path.suffix.lower() in VIDEO_EXTENSIONS
|
||||
return None
|
||||
|
||||
# ── after a scan ──────────────────────────────────────────────────
|
||||
|
||||
async def _thumbnails_after_scan(self, job, output_dirs=None):
|
||||
"""Hand a finished scan's missing previews on; returns what it did.
|
||||
|
||||
None (the scan added nothing), ``job`` (the thumbnail job started for
|
||||
``SCAN_THUMBNAIL_JOB_THRESHOLD`` or more missing previews), ``busy``
|
||||
(a job refused it; the previews are made on view), ``queued`` (fewer,
|
||||
through the worker) or ``none`` (nothing missing). It counts the
|
||||
sources the scan walked (``output_dirs``, narrowed to the job's
|
||||
``roots``; every source when None). The server log says which.
|
||||
"""
|
||||
if job.counters.get("added", 0) <= 0:
|
||||
return None
|
||||
if output_dirs is None:
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
roots = job.scope.get("roots")
|
||||
if roots:
|
||||
output_dirs = [output_dirs[i] for i in roots if 0 <= i < len(output_dirs)]
|
||||
plan = await self._run_in_executor(self._plan_gallery_thumbnails, output_dirs)
|
||||
missing = len(plan["targets"]) + plan["remaining"]
|
||||
if missing >= SCAN_THUMBNAIL_JOB_THRESHOLD:
|
||||
options = ThumbnailOptions("missing", writer_settings())
|
||||
try:
|
||||
self._start_thumbnail_job(options)
|
||||
except JobBusy as busy:
|
||||
self.logger.info(
|
||||
f"{missing:,} previews missing after the scan; the thumbnail "
|
||||
f"job did not start while {busy.running.title} runs"
|
||||
)
|
||||
return "busy"
|
||||
self.logger.info(
|
||||
f"{missing:,} previews missing after the scan; thumbnail job started"
|
||||
)
|
||||
return "job"
|
||||
if not missing:
|
||||
return "none"
|
||||
thumbnailer = self._auto_thumbnails()
|
||||
for src, dst, is_video in plan["targets"]:
|
||||
thumbnailer.enqueue(src, dst, is_video)
|
||||
self.logger.info(f"{missing:,} previews missing after the scan; queued")
|
||||
return "queued"
|
||||
|
||||
async def _hand_off_scan_thumbnails(self, job, output_dirs=None):
|
||||
"""``_thumbnails_after_scan`` for the scan runner: never raises."""
|
||||
try:
|
||||
return await self._thumbnails_after_scan(job, output_dirs)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
self.logger.warning("Previews after the scan failed", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
# A package reload re-runs this module: stop the workers the old one left.
|
||||
_stop_stale_workers()
|
||||
@@ -0,0 +1,405 @@
|
||||
"""Convert outdated thumbnails (#317): an explicit, cancellable Activity job.
|
||||
|
||||
``POST /prompt_manager/jobs/thumbnails {mode: "convert"}`` (and
|
||||
``GET /prompt_manager/images/thumbnails/plan?mode=convert`` for its count)
|
||||
starts "Convert outdated thumbnails": the thumbnail job's kind, ``guarded``
|
||||
admission (never beside a record-changing job such as a scan, a restore or
|
||||
a duplicate removal; one thumbnail job at a time), its worker threads and
|
||||
its outcome groups. Nothing converts on its own: changing the format in
|
||||
Appearance only makes previews "outdated".
|
||||
|
||||
A preview is outdated when the file owns one in another format than the one
|
||||
written now (``writer_settings``: a legacy ``_thumb.png``, or a format
|
||||
switched away from). For each such file, at a file boundary (Cancel keeps
|
||||
what was converted):
|
||||
|
||||
1. the new preview is written from the SOURCE media in the configured
|
||||
format (``_generate_one``: atomic, verified to decode; a video whose frame
|
||||
cannot be decoded fails rather than getting the placeholder), unless a
|
||||
current preview at least as new as the source is already there and
|
||||
decodes;
|
||||
2. only then are that file's old-format previews removed: regular files (no
|
||||
links) in its ``thumbnails/`` folder, owned by that source
|
||||
(``thumbnail_settings.owns``), with the source still present. One already
|
||||
gone counts as removed.
|
||||
|
||||
Files sharing a stem in one folder (``x.png`` and ``x.mp4``) share one
|
||||
preview (ruling R2), so they are converted once, as a group: the preview is
|
||||
made from the first source in ``thumbnail_settings.source_rank`` order
|
||||
(``.png``, ``.jpg``, ``.jpeg``, ``.webp``, ``.gif``, then videos), the old
|
||||
previews of every source in the group are removed, and each removed or
|
||||
written file's bytes count once.
|
||||
|
||||
A failure keeps the old preview. Previews whose source is gone (orphans) are
|
||||
never planned, so never touched. The format the job converts to is fixed
|
||||
when it starts; Appearance refuses a thumbnail settings change while it runs
|
||||
(409), and should the format still change underneath, the job stops at the
|
||||
next file without removing anything in another format. The final status line
|
||||
reads "N converted · F failed · <bytes before> → <bytes after>".
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from . import thumbnail_settings
|
||||
from .thumbnail_encode import ThumbnailVerifyError, verify_thumbnail, writer_settings
|
||||
from .admin import _plural, _scan_location, _scan_roots, _scan_status_line
|
||||
from .images import _thumbnail_is_fresh, _worker_threads, _thumbnail_batch_size
|
||||
from .prompts import safe_error_message
|
||||
|
||||
try:
|
||||
from ...utils.parallel import map_parallel
|
||||
except ImportError: # tests import the package top-level
|
||||
from utils.parallel import map_parallel
|
||||
|
||||
CONVERT_JOB_TITLE = "Convert outdated thumbnails"
|
||||
NOTHING_OUTDATED = "No outdated previews to convert"
|
||||
CANCELLED_BEFORE_CONVERTING = "Cancelled before any preview was converted"
|
||||
SOURCE_GONE = "The file is gone; its old preview was kept"
|
||||
OLD_NOT_REMOVED = "Converted; the old preview could not be removed"
|
||||
FORMAT_CHANGED = (
|
||||
"Stopped: the thumbnail format changed while converting · "
|
||||
"{converted} converted. Run Convert outdated again."
|
||||
)
|
||||
_UNITS = ("B", "KB", "MB", "GB", "TB")
|
||||
|
||||
|
||||
def readable_bytes(count):
|
||||
"""``1.5 KB``-style size (whole bytes below 1 KB)."""
|
||||
size = float(count)
|
||||
for unit in _UNITS:
|
||||
if size < 1024 or unit == _UNITS[-1]:
|
||||
return f"{int(size)} B" if unit == "B" else f"{size:.1f} {unit}"
|
||||
size /= 1024
|
||||
return f"{size:.1f} {_UNITS[-1]}" # pragma: no cover - loop always returns
|
||||
|
||||
|
||||
def _old_previews(src, dst, settings):
|
||||
"""``[(path, size)]``: *src*'s previews beside *dst* in another format."""
|
||||
stem = dst.name[: -len(dst.suffix)] if dst.suffix else dst.name
|
||||
old = []
|
||||
for path in thumbnail_settings.owned_in_folder(
|
||||
dst.parent, stem, src.suffix, settings.extension
|
||||
):
|
||||
if path.suffix == settings.extension:
|
||||
continue
|
||||
try:
|
||||
old.append((path, path.stat().st_size))
|
||||
except OSError:
|
||||
continue
|
||||
return old
|
||||
|
||||
|
||||
def _group_targets(targets):
|
||||
"""``[[(src, dst, is_video)]]``: targets sharing a preview name (one
|
||||
folder, one stem) in one group, its first source first (R2)."""
|
||||
groups = {}
|
||||
for target in targets:
|
||||
dst = Path(target[1])
|
||||
key = os.path.normcase(str(dst.with_suffix("")))
|
||||
groups.setdefault(key, []).append(target)
|
||||
ordered = []
|
||||
for group in groups.values():
|
||||
group.sort(
|
||||
key=lambda t: (thumbnail_settings.source_rank(Path(t[0]).suffix), str(t[0]))
|
||||
)
|
||||
ordered.append(group)
|
||||
return ordered
|
||||
|
||||
|
||||
def _same_file(a, b):
|
||||
"""Whether *a* and *b* name one file (re-review N1: on a case-insensitive
|
||||
disk ``x_thumb.JPG`` is the ``x_thumb.jpg`` just written). Without inode
|
||||
numbers (st_ino 0 on some SMB/FAT mounts) the resolved, normcased paths
|
||||
are compared."""
|
||||
try:
|
||||
first, second = os.stat(a), os.stat(b)
|
||||
except OSError:
|
||||
return False
|
||||
if first.st_ino and second.st_ino:
|
||||
return (first.st_dev, first.st_ino) == (second.st_dev, second.st_ino)
|
||||
return os.path.normcase(os.path.realpath(a)) == os.path.normcase(
|
||||
os.path.realpath(b)
|
||||
)
|
||||
|
||||
|
||||
def _format_changed(settings):
|
||||
"""Whether previews are written in another format than *settings* now."""
|
||||
return writer_settings().extension != settings.extension
|
||||
|
||||
|
||||
def _finished_line(totals, remaining):
|
||||
line = (
|
||||
f"{totals['converted']:,} converted · {totals['failed']:,} failed · "
|
||||
f"{readable_bytes(totals['before'])} → {readable_bytes(totals['after'])}"
|
||||
)
|
||||
if remaining:
|
||||
line += f" · {remaining:,} more on the next run"
|
||||
return line
|
||||
|
||||
|
||||
class ThumbnailConvertMixin:
|
||||
"""The Convert outdated job (started through ``_start_thumbnail_job``)."""
|
||||
|
||||
async def _run_convert_job(self, job, options):
|
||||
try:
|
||||
state = await self._convert_job_body(job, options)
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail conversion failed")
|
||||
job.update(status_line="Thumbnail conversion stopped on an internal error.")
|
||||
job.log_line("Conversion failed · internal error")
|
||||
state = "failed"
|
||||
job.finish(state)
|
||||
|
||||
async def _convert_job_body(self, job, options):
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
if not output_dirs:
|
||||
message = "ComfyUI output directory not found"
|
||||
job.update(status_line=message)
|
||||
job.log_line(f"Conversion failed · {message}")
|
||||
return "failed"
|
||||
folders = _plural(len(output_dirs), "output folder")
|
||||
job.scope["output_folders"] = len(output_dirs)
|
||||
job.update(status_line=f"Looking for outdated previews in {folders}")
|
||||
job.log_line(
|
||||
f"Conversion started · {folders} · to {options.settings.describe()}"
|
||||
)
|
||||
plan = await self._plan_conversion(job, options, output_dirs)
|
||||
if job.cancelled:
|
||||
return self._end_conversion(job, 0, CANCELLED_BEFORE_CONVERTING)
|
||||
if not plan["targets"]:
|
||||
job.update(progress=100, status_line=NOTHING_OUTDATED)
|
||||
job.log_line(f"Conversion finished · {NOTHING_OUTDATED.lower()}")
|
||||
return "finished"
|
||||
totals, left = await self._convert_in_batches(job, options, plan, output_dirs)
|
||||
self.invalidate_gallery_cache()
|
||||
if totals["stopped"]:
|
||||
job.update(status_line=FORMAT_CHANGED.format(converted=totals["converted"]))
|
||||
job.log_line(
|
||||
"Conversion stopped · the thumbnail format changed · "
|
||||
f"{_plural(totals['converted'], 'preview')} converted"
|
||||
)
|
||||
return "cancelled"
|
||||
if job.cancelled:
|
||||
return self._end_conversion(
|
||||
job,
|
||||
totals["converted"],
|
||||
f"Cancelled · {_plural(totals['converted'], 'preview')} converted",
|
||||
)
|
||||
line = _finished_line(totals, left)
|
||||
job.update(progress=100, status_line=line)
|
||||
job.log_line(f"Conversion finished · {line}")
|
||||
self.logger.info(f"Thumbnail conversion: {line}")
|
||||
return "finished"
|
||||
|
||||
@staticmethod
|
||||
def _end_conversion(job, converted, status_line):
|
||||
job.update(status_line=status_line)
|
||||
job.log_line(
|
||||
f"Conversion cancelled · {_plural(converted, 'preview')} converted"
|
||||
)
|
||||
return "cancelled"
|
||||
|
||||
async def _plan_conversion(self, job, options, output_dirs, exclude=None):
|
||||
"""One batch's plan (at most MAX_THUMBNAIL_FILES targets), off-loop."""
|
||||
return await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails,
|
||||
output_dirs,
|
||||
"convert",
|
||||
stop=lambda: job.cancelled,
|
||||
settings=options.settings,
|
||||
exclude=exclude,
|
||||
)
|
||||
|
||||
async def _convert_in_batches(self, job, options, plan, output_dirs):
|
||||
"""Convert ``plan``, then plan again after each batch until nothing
|
||||
is left, the job is cancelled or stopped, or a batch converted
|
||||
nothing (files that fail are never planned again). Returns
|
||||
``(totals across batches, files left)``."""
|
||||
totals = {"converted": 0, "failed": 0, "before": 0, "after": 0}
|
||||
totals["stopped"] = False
|
||||
run = {
|
||||
"index": 0,
|
||||
"batches": 0,
|
||||
"total": len(plan["targets"]) + plan["remaining"],
|
||||
}
|
||||
run["start"] = time.monotonic()
|
||||
job.update(
|
||||
total=run["total"], status_line=_scan_status_line(0, run["total"], 0, 0)
|
||||
)
|
||||
attempted = set()
|
||||
while True:
|
||||
before = totals["converted"]
|
||||
await self._convert_targets(
|
||||
job, options, plan["targets"], output_dirs, totals, run
|
||||
)
|
||||
attempted.update(src for src, _dst, _video in plan["targets"])
|
||||
stop = job.cancelled or totals["stopped"]
|
||||
if stop or not plan["remaining"] or totals["converted"] == before:
|
||||
return totals, plan["remaining"]
|
||||
plan = await self._plan_conversion(
|
||||
job, options, output_dirs, exclude=attempted
|
||||
)
|
||||
if job.cancelled:
|
||||
return totals, 0
|
||||
run["total"] = run["index"] + len(plan["targets"]) + plan["remaining"]
|
||||
job.update(total=run["total"])
|
||||
if not plan["targets"]:
|
||||
return totals, plan["remaining"]
|
||||
|
||||
async def _convert_targets(self, job, options, targets, output_dirs, totals, run):
|
||||
"""Convert one batch's *targets* in worker batches of preview groups,
|
||||
recording each file against the job's overall ``run`` counters."""
|
||||
roots = _scan_roots(output_dirs)
|
||||
groups = _group_targets(targets)
|
||||
batch_size = _thumbnail_batch_size(_worker_threads())
|
||||
for batch_start in range(0, len(groups), batch_size):
|
||||
if job.cancelled or totals["stopped"]:
|
||||
break
|
||||
batch = groups[batch_start : batch_start + batch_size]
|
||||
results = await self._run_in_executor(
|
||||
self._convert_batch, batch, options.settings, lambda: job.cancelled
|
||||
)
|
||||
for group, group_results in zip(batch, results):
|
||||
for (src, _dst, _video), result in zip(group, group_results):
|
||||
if result["action"] == "stopped":
|
||||
totals["stopped"] = True
|
||||
if result["action"] in ("cancelled", "stopped"):
|
||||
continue
|
||||
run["index"] += 1
|
||||
index, total = run["index"], max(run["total"], run["index"])
|
||||
self._record_conversion(job, src, result, roots, totals)
|
||||
job.update(processed=index, progress=index * 100 // total)
|
||||
run["batches"] += 1
|
||||
job.update(
|
||||
status_line=_scan_status_line(
|
||||
run["index"],
|
||||
run["total"],
|
||||
run["batches"],
|
||||
time.monotonic() - run["start"],
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0) # let other requests run between batches
|
||||
|
||||
@staticmethod
|
||||
def _record_conversion(job, src, result, roots, totals):
|
||||
path, label, index = _scan_location(src, roots)
|
||||
where = f"{label}/{path}" if label else path
|
||||
row = {"path": path, "root": label, "root_index": index}
|
||||
if result["action"] == "converted":
|
||||
totals["converted"] += 1
|
||||
totals["before"] += result["before"]
|
||||
totals["after"] += result["after"]
|
||||
detail = result["detail"]
|
||||
job.record("generated", {**row, "detail": detail}, f"+ {where} · {detail}")
|
||||
return
|
||||
totals["failed"] += 1
|
||||
detail = result.get("error") or "failed"
|
||||
job.record("errors", {**row, "detail": detail}, f"! {where} · {detail}")
|
||||
|
||||
def _convert_batch(self, groups, settings, stop):
|
||||
"""Convert preview ``groups`` on the configured worker threads
|
||||
(blocking); one result list per group."""
|
||||
convert = functools.partial(self._convert_group, settings=settings, stop=stop)
|
||||
return map_parallel(convert, groups, _worker_threads())
|
||||
|
||||
def _convert_group(self, group, settings, stop):
|
||||
"""Convert the preview a group of same-stem files shares, once.
|
||||
|
||||
The first source writes it; the others are recorded with the same
|
||||
outcome and no bytes of their own."""
|
||||
src, dst, is_video = group[0]
|
||||
siblings = [Path(target[0]) for target in group[1:]]
|
||||
result = self._convert_one(
|
||||
src, dst, is_video, settings, stop=stop, siblings=siblings
|
||||
)
|
||||
if result["action"] != "converted":
|
||||
return [result] * len(group)
|
||||
shared = {**result, "before": 0, "after": 0}
|
||||
shared["detail"] = f"{result['detail']} (shares {Path(src).name}'s preview)"
|
||||
return [result] + [shared] * len(siblings)
|
||||
|
||||
def _convert_one(self, src, dst, is_video, settings, stop=None, siblings=()):
|
||||
"""Convert one file's preview (blocking; never raises).
|
||||
|
||||
``converted`` with the bytes removed and written, ``error`` (the old
|
||||
preview kept), ``cancelled`` (a file boundary) or ``stopped`` (the
|
||||
format changed underneath, review I4). *siblings* share the preview
|
||||
(R2): their old previews go too.
|
||||
"""
|
||||
if stop is not None and stop():
|
||||
return {"action": "cancelled"}
|
||||
if _format_changed(settings):
|
||||
return {"action": "stopped"}
|
||||
dst = Path(dst)
|
||||
try:
|
||||
old = self._old_group_previews([Path(src), *siblings], dst, settings)
|
||||
if not self._current_preview_usable(src, dst, settings):
|
||||
made = self._generate_one(
|
||||
src, dst, settings, is_video, rebuild=True, placeholder=False
|
||||
)
|
||||
if made["action"] != "generated":
|
||||
return {"action": "error", "error": made.get("error")}
|
||||
dst = Path(made["path"])
|
||||
if dst.suffix != settings.extension or not dst.is_file():
|
||||
return {"action": "error", "error": "The new preview was not written"}
|
||||
if not os.path.isfile(src):
|
||||
return {"action": "error", "error": SOURCE_GONE}
|
||||
if _format_changed(settings):
|
||||
return {"action": "stopped"}
|
||||
old = [(path, size) for path, size in old if not _same_file(path, dst)]
|
||||
removed, kept = self._remove_old_previews(old)
|
||||
if kept:
|
||||
return {"action": "error", "error": OLD_NOT_REMOVED}
|
||||
label = thumbnail_settings.FORMAT_LABELS[settings.format]
|
||||
return {
|
||||
"action": "converted",
|
||||
"before": removed,
|
||||
"after": dst.stat().st_size,
|
||||
"detail": f"Converted to {label}",
|
||||
}
|
||||
except Exception as exc:
|
||||
self.logger.warning(f"Conversion failed for {src.name}", exc_info=True)
|
||||
return {"action": "error", "error": safe_error_message(exc)}
|
||||
|
||||
@staticmethod
|
||||
def _old_group_previews(sources, dst, settings):
|
||||
"""``[(path, size)]`` of every source's old previews, each path once."""
|
||||
seen, old = set(), []
|
||||
for source in sources:
|
||||
for path, size in _old_previews(source, dst, settings):
|
||||
key = os.path.normcase(str(path))
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
old.append((path, size))
|
||||
return old
|
||||
|
||||
@staticmethod
|
||||
def _current_preview_usable(src, dst, settings):
|
||||
"""A current preview at least as new as *src* that decodes (M4)."""
|
||||
if not _thumbnail_is_fresh(src, dst):
|
||||
return False
|
||||
try:
|
||||
verify_thumbnail(dst, settings.format)
|
||||
except ThumbnailVerifyError:
|
||||
return False
|
||||
return True
|
||||
|
||||
def _remove_old_previews(self, old):
|
||||
"""``(bytes removed, [kept paths])``; a file already gone counts as
|
||||
removed, with no bytes (another run or a sibling removed it)."""
|
||||
removed, kept = 0, []
|
||||
for path, size in old:
|
||||
try:
|
||||
os.remove(path)
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
except OSError as exc:
|
||||
self.logger.warning(f"Could not remove old preview {path}: {exc}")
|
||||
kept.append(path)
|
||||
continue
|
||||
removed += size
|
||||
return removed, kept
|
||||
@@ -0,0 +1,524 @@
|
||||
"""Thumbnail encoders (#317): every preview file is written here.
|
||||
|
||||
``write_thumbnail(frame, dst, settings)`` encodes a prepared frame
|
||||
(``thumbnail_frame``: the first frame, RGB, fitted inside the edge) in the
|
||||
configured format into a hidden temp sibling, closes it, verifies that it
|
||||
decodes, and renames it over ``dst`` (``os.replace``), so a reader never sees
|
||||
a partial file and a failure keeps the old preview. ``guard()`` is asked just
|
||||
before the rename (automatic previews' holds, #249).
|
||||
|
||||
Formats: WebP (Pillow, ``method=4``) and JPEG (Pillow, ``optimize``) always
|
||||
work. JPEG XL needs no new dependency; the first encoder found is used:
|
||||
|
||||
1. Pillow, natively or through the ``pillow_jxl`` plugin (imported when
|
||||
installed: ``pip install pillow-jxl-plugin``), given the quality itself;
|
||||
2. ffmpeg built with ``libjxl`` (``-distance``), fed a PNG on stdin;
|
||||
3. ``cjxl`` from libjxl (``-d``), fed a temp PNG that is closed first.
|
||||
|
||||
Quality maps to a JPEG XL distance with libjxl's own
|
||||
``JxlEncoderDistanceFromQuality``: ``0.1 + (100 - q) * 0.09`` from 30 up
|
||||
(q80 → d1.9, q85 → d1.45, q90 → d1.0), the quadratic below 30, and q100 →
|
||||
d0.1 (a preview is never lossless). Pillow's plugin applies the same mapping
|
||||
inside libjxl. Detection is cached (``detect_jxl_encoder(refresh=True)``
|
||||
probes again); "none found" is probed again on the settings read
|
||||
(``encoder_status``) and after MISSING_ENCODER_TTL, so one slow probe never
|
||||
turns JPEG XL off until a restart. An encoder that fails because its binary
|
||||
or module is gone makes that write detect again: another encoder found is
|
||||
used for that file, none at all writes it as WebP (logged once) and
|
||||
refreshes what Appearance reads. Any other failure (one bad file) raises
|
||||
without a probe.
|
||||
|
||||
Encoder processes (and ffmpeg's video frames, images.py) go through
|
||||
``run_encoder``: tracked, bounded by a timeout, and killed by
|
||||
``shutdown_encoders()`` at the app's cleanup, after which none starts.
|
||||
"""
|
||||
|
||||
import importlib
|
||||
import io
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess # nosec B404 - fixed argv lists, never a shell
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from . import thumbnail_settings
|
||||
|
||||
try:
|
||||
from ...utils.logging_config import get_logger
|
||||
except ImportError: # tests import the package top-level
|
||||
from utils.logging_config import get_logger
|
||||
|
||||
logger = get_logger("prompt_manager.thumbnails")
|
||||
|
||||
PILLOW_JXL_FORMAT = "JXL"
|
||||
JXL_UNAVAILABLE = (
|
||||
"No JPEG XL encoder found. Install pillow-jxl-plugin "
|
||||
"(pip install pillow-jxl-plugin), ffmpeg built with libjxl, or cjxl from "
|
||||
"libjxl, then restart ComfyUI."
|
||||
)
|
||||
# Seconds an encoder process may run (ffmpeg's video frame shares it).
|
||||
ENCODE_TIMEOUT = 30
|
||||
PROBE_TIMEOUT = 10
|
||||
JXL_EFFORT = 7
|
||||
WEBP_METHOD = 4
|
||||
JXL_SIGNATURES = (b"\xff\x0a", b"\x00\x00\x00\x0cJXL \r\n\x87\n")
|
||||
_NO_WINDOW = getattr(subprocess, "CREATE_NO_WINDOW", 0)
|
||||
|
||||
|
||||
class PreviewWriteHeld(Exception):
|
||||
"""A preview write stopped before its rename: ``guard()`` said no (#249)."""
|
||||
|
||||
|
||||
# The result error of a write held before its rename (never remembered).
|
||||
PREVIEW_HELD = "Preview writes are held while another job runs"
|
||||
|
||||
|
||||
class ThumbnailVerifyError(RuntimeError):
|
||||
"""The encoded preview did not decode; it was never renamed in."""
|
||||
|
||||
|
||||
class EncoderFailed(RuntimeError):
|
||||
"""An encoder process failed or wrote nothing."""
|
||||
|
||||
|
||||
class EncoderClosed(RuntimeError):
|
||||
"""The app is shutting down: no encoder process runs any more."""
|
||||
|
||||
|
||||
# ── temp files ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def temp_sibling(dst, tag="part"):
|
||||
"""Hidden temp path next to *dst* that keeps its extension."""
|
||||
dst = Path(dst)
|
||||
token = f"{os.getpid()}-{threading.get_ident()}"
|
||||
return dst.with_name(f".{dst.stem}.{tag}-{token}{dst.suffix}")
|
||||
|
||||
|
||||
def replace_unless_held(tmp, dst, guard=None):
|
||||
"""Rename *tmp* over *dst*, unless ``guard()`` is false (PreviewWriteHeld)."""
|
||||
if guard is not None and not guard():
|
||||
raise PreviewWriteHeld(PREVIEW_HELD)
|
||||
os.replace(tmp, dst)
|
||||
|
||||
|
||||
# ``temp_sibling``'s names: ``.<stem>.<tag>-<pid>-<thread id><ext>``.
|
||||
_TEMP_NAME = re.compile(r"^\..+\.(?:part|src)-(\d+)-\d+\.[A-Za-z0-9]+$")
|
||||
# A temp at least this old (seconds) outlived its write: no encode runs longer.
|
||||
STRAY_TEMP_AGE = 4 * ENCODE_TIMEOUT
|
||||
# When this process started (this module's import): a temp carrying its PID
|
||||
# that is older was left by an earlier run with the same PID (containers).
|
||||
PROCESS_STARTED = time.time()
|
||||
|
||||
|
||||
def sweep_stray_temps(folder, min_age=None):
|
||||
"""Remove the temp files a crashed write left under *folder* (a preview
|
||||
folder, walked without following links); returns ``(count, bytes)``.
|
||||
|
||||
Only ``temp_sibling`` names are touched, never a link. With *min_age*
|
||||
(seconds, the startup sweep) a temp younger than that, or one this
|
||||
process wrote since it started (it may still be writing it), is left;
|
||||
``min_age=0`` (the clear, which holds every preview write) removes them
|
||||
all.
|
||||
"""
|
||||
min_age = STRAY_TEMP_AGE if min_age is None else min_age
|
||||
now = time.time()
|
||||
own = str(os.getpid())
|
||||
count = size = 0
|
||||
for root, _dirs, files in os.walk(folder):
|
||||
for name in files:
|
||||
match = _TEMP_NAME.match(name)
|
||||
if match is None:
|
||||
continue
|
||||
path = Path(root) / name
|
||||
try:
|
||||
stat = path.lstat()
|
||||
if path.is_symlink() or not path.is_file():
|
||||
continue
|
||||
if min_age and (
|
||||
now - stat.st_mtime < min_age
|
||||
or (match.group(1) == own and stat.st_mtime >= PROCESS_STARTED)
|
||||
):
|
||||
continue
|
||||
path.unlink()
|
||||
except OSError as exc:
|
||||
logger.debug(f"Could not remove stray temp {name}: {exc}")
|
||||
continue
|
||||
count += 1
|
||||
size += stat.st_size
|
||||
return count, size
|
||||
|
||||
|
||||
def _unlink_quietly(path):
|
||||
try:
|
||||
Path(path).unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
except OSError as exc:
|
||||
logger.debug(f"Could not remove temp file {Path(path).name}: {exc}")
|
||||
|
||||
|
||||
# ── encoder processes ────────────────────────────────────────────────
|
||||
|
||||
_process_lock = threading.Lock()
|
||||
_processes = set()
|
||||
_closed = False
|
||||
|
||||
|
||||
def run_encoder(cmd, *, input=None, timeout=ENCODE_TIMEOUT, track=None):
|
||||
"""Run *cmd* (a fixed argv) with *input* on stdin; returns a
|
||||
CompletedProcess with bytes output.
|
||||
|
||||
The process is tracked so ``shutdown_encoders`` can kill it; past
|
||||
*timeout* it is killed and TimeoutExpired raised. After the shutdown
|
||||
nothing starts (EncoderClosed). ``track(process)`` sees it too.
|
||||
"""
|
||||
with _process_lock:
|
||||
if _closed:
|
||||
raise EncoderClosed("Encoders are shut down")
|
||||
process = subprocess.Popen( # nosec B603 - fixed argv, no shell
|
||||
cmd,
|
||||
stdin=subprocess.PIPE if input is not None else subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
creationflags=_NO_WINDOW,
|
||||
)
|
||||
with _process_lock:
|
||||
_processes.add(process)
|
||||
closed = _closed
|
||||
try:
|
||||
if closed:
|
||||
process.kill()
|
||||
if track is not None:
|
||||
track(process)
|
||||
try:
|
||||
out, err = process.communicate(input=input, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.communicate()
|
||||
raise
|
||||
finally:
|
||||
with _process_lock:
|
||||
_processes.discard(process)
|
||||
closed = _closed
|
||||
if closed:
|
||||
raise EncoderClosed("Encoder stopped by the shutdown")
|
||||
return subprocess.CompletedProcess(cmd, process.returncode, out, err)
|
||||
|
||||
|
||||
def shutdown_encoders():
|
||||
"""Kill every running encoder process; none starts afterwards."""
|
||||
global _closed
|
||||
with _process_lock:
|
||||
_closed = True
|
||||
running = list(_processes)
|
||||
for process in running:
|
||||
try:
|
||||
process.kill()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def reset_encoders():
|
||||
"""Allow encoders again and forget the detection (tests, a reload)."""
|
||||
global _closed, _detected, _fallback_logged
|
||||
with _process_lock:
|
||||
_closed = False
|
||||
with _detect_lock:
|
||||
_detected = _UNSET
|
||||
_fallback_logged = False
|
||||
|
||||
|
||||
# ── JPEG XL detection ────────────────────────────────────────────────
|
||||
|
||||
_UNSET = object()
|
||||
_detect_lock = threading.Lock()
|
||||
_detected = _UNSET
|
||||
_detected_at = 0.0
|
||||
# Seconds after which "no JPEG XL encoder" is probed again.
|
||||
MISSING_ENCODER_TTL = 60
|
||||
_fallback_logged = False
|
||||
|
||||
|
||||
def _pillow_jxl_format():
|
||||
"""Pillow's JPEG XL format id when it can save one, else None."""
|
||||
try:
|
||||
native = Image.registered_extensions().get(".jxl")
|
||||
except Exception:
|
||||
native = None
|
||||
if native and native in Image.SAVE:
|
||||
return native
|
||||
return PILLOW_JXL_FORMAT if PILLOW_JXL_FORMAT in Image.SAVE else None
|
||||
|
||||
|
||||
def _pillow_can_save_jxl():
|
||||
if _pillow_jxl_format():
|
||||
return True
|
||||
try:
|
||||
importlib.import_module("pillow_jxl")
|
||||
except Exception:
|
||||
return False
|
||||
return _pillow_jxl_format() is not None
|
||||
|
||||
|
||||
def _pillow_decodes_jxl():
|
||||
try:
|
||||
native = Image.registered_extensions().get(".jxl")
|
||||
except Exception:
|
||||
native = None
|
||||
return bool(native and native in Image.OPEN) or PILLOW_JXL_FORMAT in Image.OPEN
|
||||
|
||||
|
||||
def _ffmpeg_has_libjxl():
|
||||
"""Whether the ffmpeg on PATH lists the ``libjxl`` encoder."""
|
||||
if not shutil.which("ffmpeg"):
|
||||
return False
|
||||
try:
|
||||
done = run_encoder(
|
||||
["ffmpeg", "-hide_banner", "-encoders"], timeout=PROBE_TIMEOUT
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError, EncoderClosed):
|
||||
return False
|
||||
listing = done.stdout
|
||||
if isinstance(listing, bytes):
|
||||
listing = listing.decode("utf-8", "replace")
|
||||
return any(
|
||||
len(line.split()) > 1 and line.split()[1] == "libjxl"
|
||||
for line in listing.splitlines()
|
||||
)
|
||||
|
||||
|
||||
def _detect():
|
||||
if _pillow_can_save_jxl():
|
||||
return "pillow"
|
||||
if _ffmpeg_has_libjxl():
|
||||
return "ffmpeg"
|
||||
if shutil.which("cjxl"):
|
||||
return "cjxl"
|
||||
return None
|
||||
|
||||
|
||||
def detect_jxl_encoder(refresh=False, recheck_missing=False):
|
||||
"""``pillow``, ``ffmpeg``, ``cjxl`` or None (cached; blocking on a probe).
|
||||
|
||||
A cached None is probed again with *recheck_missing* or once it is older
|
||||
than MISSING_ENCODER_TTL."""
|
||||
global _detected, _detected_at
|
||||
with _detect_lock:
|
||||
now = time.monotonic()
|
||||
stale_missing = _detected is None and (
|
||||
recheck_missing or now - _detected_at >= MISSING_ENCODER_TTL
|
||||
)
|
||||
if refresh or _detected is _UNSET or stale_missing:
|
||||
_detected = _detect()
|
||||
_detected_at = now
|
||||
return _detected
|
||||
|
||||
|
||||
def _encoder_gone(encoder):
|
||||
"""Whether *encoder*'s binary or module is no longer there."""
|
||||
if encoder == "pillow":
|
||||
return not _pillow_can_save_jxl()
|
||||
return shutil.which(encoder) is None
|
||||
|
||||
|
||||
def encoder_status():
|
||||
"""Which formats can be written now (``GET /prompt_manager/settings``;
|
||||
a "none found" is probed again here)."""
|
||||
encoder = detect_jxl_encoder(recheck_missing=True)
|
||||
return {
|
||||
"webp": {"available": True},
|
||||
"jpeg": {"available": True},
|
||||
"jxl": {
|
||||
"available": encoder is not None,
|
||||
"encoder": encoder,
|
||||
"reason": None if encoder is not None else JXL_UNAVAILABLE,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def jxl_distance(quality):
|
||||
"""libjxl's distance for a 1–100 quality (q100 gives 0.1, never lossless)."""
|
||||
if quality >= 100:
|
||||
return 0.1
|
||||
if quality >= 30:
|
||||
return 0.1 + (100 - quality) * 0.09
|
||||
return 53.0 / 3000.0 * quality * quality - 23.0 / 20.0 * quality + 25.0
|
||||
|
||||
|
||||
# ── frames and writers ───────────────────────────────────────────────
|
||||
|
||||
|
||||
def thumbnail_frame(img, edge):
|
||||
"""The first frame of *img* as RGB, fitted inside ``edge`` x ``edge``."""
|
||||
try:
|
||||
img.seek(0)
|
||||
except (EOFError, AttributeError, ValueError):
|
||||
pass
|
||||
try:
|
||||
img.draft("RGB", (edge, edge)) # JPEG: decode at a reduced scale
|
||||
except (AttributeError, ValueError, OSError):
|
||||
pass
|
||||
frame = img.convert("RGB") if img.mode != "RGB" else img.copy()
|
||||
frame.thumbnail((edge, edge), Image.Resampling.LANCZOS)
|
||||
return frame
|
||||
|
||||
|
||||
def _png_bytes(img):
|
||||
buffer = io.BytesIO()
|
||||
img.save(buffer, "PNG", compress_level=1)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def _write_webp(img, tmp, settings, track):
|
||||
img.save(tmp, "WEBP", quality=settings.quality, method=WEBP_METHOD)
|
||||
|
||||
|
||||
def _write_jpeg(img, tmp, settings, track):
|
||||
img.save(tmp, "JPEG", quality=settings.quality, optimize=True)
|
||||
|
||||
|
||||
def _check(done, tmp, name):
|
||||
if done.returncode != 0 or not Path(tmp).is_file():
|
||||
detail = (done.stderr or b"")[-300:]
|
||||
if isinstance(detail, bytes):
|
||||
detail = detail.decode("utf-8", "replace")
|
||||
raise EncoderFailed(f"{name} could not encode the preview: {detail.strip()}")
|
||||
|
||||
|
||||
class _EncoderGone(RuntimeError):
|
||||
"""No JPEG XL encoder is left: this file is written as WebP."""
|
||||
|
||||
|
||||
def _write_jxl(img, tmp, settings, track):
|
||||
"""Encode with the detected encoder. One that failed because it is gone
|
||||
makes the write detect again once (another one is used, none raises
|
||||
_EncoderGone); any other failure raises without a probe."""
|
||||
encoder = detect_jxl_encoder()
|
||||
try:
|
||||
_encode_jxl(encoder, img, tmp, settings, track)
|
||||
return
|
||||
except (OSError, EncoderFailed) as exc:
|
||||
if encoder is not None and not _encoder_gone(encoder):
|
||||
raise
|
||||
again = detect_jxl_encoder(refresh=True)
|
||||
if again is None:
|
||||
raise _EncoderGone(str(exc)) from exc
|
||||
if again == encoder:
|
||||
raise
|
||||
logger.info(f"JPEG XL encoder {encoder} failed; using {again}")
|
||||
_unlink_quietly(tmp)
|
||||
_encode_jxl(again, img, tmp, settings, track)
|
||||
|
||||
|
||||
def _encode_jxl(encoder, img, tmp, settings, track):
|
||||
distance = f"{jxl_distance(settings.quality):.2f}"
|
||||
if encoder == "pillow":
|
||||
img.save(
|
||||
tmp,
|
||||
_pillow_jxl_format() or PILLOW_JXL_FORMAT,
|
||||
quality=min(settings.quality, 99),
|
||||
effort=JXL_EFFORT,
|
||||
num_threads=1,
|
||||
)
|
||||
return
|
||||
if encoder == "ffmpeg":
|
||||
cmd = [
|
||||
"ffmpeg", "-v", "error", "-f", "png_pipe", "-i", "-",
|
||||
"-frames:v", "1", "-c:v", "libjxl", "-distance", distance,
|
||||
"-effort", str(JXL_EFFORT), "-f", "image2", "-update", "1",
|
||||
"-y", str(tmp),
|
||||
] # fmt: skip
|
||||
_check(run_encoder(cmd, input=_png_bytes(img), track=track), tmp, "ffmpeg")
|
||||
return
|
||||
if encoder == "cjxl":
|
||||
source = temp_sibling(Path(tmp).with_suffix(".png"), "src")
|
||||
try:
|
||||
with open(source, "wb") as handle:
|
||||
handle.write(_png_bytes(img))
|
||||
cmd = ["cjxl", str(source), str(tmp), "-d", distance, "-e", str(JXL_EFFORT)]
|
||||
_check(run_encoder(cmd, track=track), tmp, "cjxl")
|
||||
finally:
|
||||
_unlink_quietly(source)
|
||||
return
|
||||
raise EncoderFailed(JXL_UNAVAILABLE)
|
||||
|
||||
|
||||
_WRITERS = {"webp": _write_webp, "jpeg": _write_jpeg, "jxl": _write_jxl}
|
||||
|
||||
|
||||
def verify_thumbnail(path, fmt):
|
||||
"""Raise ThumbnailVerifyError unless *path* decodes as a preview.
|
||||
|
||||
A JPEG XL file Pillow cannot read (no plugin) is checked by signature.
|
||||
"""
|
||||
try:
|
||||
if fmt == "jxl" and not _pillow_decodes_jxl():
|
||||
with open(path, "rb") as handle:
|
||||
head = handle.read(12)
|
||||
if not head.startswith(JXL_SIGNATURES):
|
||||
raise ThumbnailVerifyError("The JPEG XL preview has no signature")
|
||||
return
|
||||
with Image.open(path) as written:
|
||||
written.load()
|
||||
except ThumbnailVerifyError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ThumbnailVerifyError(f"The new preview does not decode: {exc}") from exc
|
||||
|
||||
|
||||
def _jxl_fallback():
|
||||
"""Log once that JPEG XL previews are being written as WebP."""
|
||||
global _fallback_logged
|
||||
with _detect_lock:
|
||||
first, _fallback_logged = not _fallback_logged, True
|
||||
if first:
|
||||
logger.warning(
|
||||
"JPEG XL is the thumbnail format but no JPEG XL encoder is available "
|
||||
"any more; previews are written as WebP until one is installed"
|
||||
)
|
||||
|
||||
|
||||
def writer_settings(settings=None):
|
||||
"""The settings a preview is written with now: the configured ones, but
|
||||
WebP while JPEG XL has no encoder (logged once). Its extension names the
|
||||
file, so "outdated" means a preview this server would not write now."""
|
||||
settings = settings or thumbnail_settings.current()
|
||||
if settings.format == "jxl" and detect_jxl_encoder() is None:
|
||||
_jxl_fallback()
|
||||
return replace(settings, format="webp")
|
||||
return settings
|
||||
|
||||
|
||||
def write_thumbnail(frame, dst, settings, *, guard=None, track=None):
|
||||
"""Encode *frame* into *dst* in ``settings``' format; returns the path
|
||||
written (``.webp`` when JPEG XL fell back). Atomic and verified; raises
|
||||
PreviewWriteHeld, ThumbnailVerifyError, EncoderFailed, EncoderClosed or
|
||||
the OSError of the write."""
|
||||
dst = Path(dst)
|
||||
fmt = settings.format
|
||||
if fmt == "jxl" and detect_jxl_encoder() is None:
|
||||
_jxl_fallback()
|
||||
fmt, dst = "webp", dst.with_suffix(".webp")
|
||||
try:
|
||||
return _write_verified(frame, dst, fmt, settings, guard, track)
|
||||
except _EncoderGone:
|
||||
_jxl_fallback()
|
||||
dst = dst.with_suffix(".webp")
|
||||
return _write_verified(frame, dst, "webp", settings, guard, track)
|
||||
|
||||
|
||||
def _write_verified(frame, dst, fmt, settings, guard, track):
|
||||
tmp = temp_sibling(dst)
|
||||
try:
|
||||
_WRITERS[fmt](frame, tmp, settings, track)
|
||||
verify_thumbnail(tmp, fmt)
|
||||
replace_unless_held(tmp, dst, guard)
|
||||
finally:
|
||||
_unlink_quietly(tmp)
|
||||
return dst
|
||||
+143
-118
@@ -1,29 +1,32 @@
|
||||
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
|
||||
"""Thumbnail generation as an Activity job (#249 System S5, Task 12; #317).
|
||||
|
||||
- ``GET /prompt_manager/images/thumbnails/plan?mode=&size=&quality=`` counts
|
||||
what a run would do over every gallery root and writes nothing:
|
||||
``{success, targets, fresh, remaining, roots, mode, size, quality}``.
|
||||
- ``POST /prompt_manager/jobs/thumbnails {mode, size, quality}`` starts the
|
||||
``thumbnails`` job (202 ``{job}``). Its admission class is ``guarded``: it
|
||||
never runs beside a record-changing job, and one thumbnail job runs at a
|
||||
time. It stops at a file boundary when cancelled, keeping the previews it
|
||||
wrote. Its outcome groups are ``generated``, ``skipped`` (already fresh)
|
||||
and ``errors``.
|
||||
- ``GET /prompt_manager/images/thumbnails/plan?mode=`` counts what a run
|
||||
would do over every gallery root and writes nothing: ``{success, targets,
|
||||
fresh, outdated, remaining, roots, mode, format, quality, size}``.
|
||||
- ``POST /prompt_manager/jobs/thumbnails {mode}`` starts the ``thumbnails``
|
||||
job (202 ``{job}``). Its admission class is ``guarded``: it never runs
|
||||
beside a record-changing job, and one thumbnail job runs at a time. It
|
||||
stops at a file boundary when cancelled, keeping the previews it wrote.
|
||||
Its outcome groups are ``generated``, ``skipped`` (already fresh) and
|
||||
``errors``.
|
||||
|
||||
``mode`` is ``missing`` (previews that are absent or older than their file)
|
||||
or ``rebuild`` (every preview again, fresh ones too). ``size`` is the
|
||||
preview's longest edge, 128 to 2048 pixels. ``quality`` picks the encoder
|
||||
settings (``THUMBNAIL_ENCODINGS``). Every preview is written next to the old
|
||||
one and renamed over it, so nothing is cleared first and a failure keeps the
|
||||
old preview; originals are only read.
|
||||
``mode`` is ``missing`` (previews that are absent or older than their file),
|
||||
``rebuild`` (every preview again, fresh ones too) or ``convert`` (Convert
|
||||
outdated: previews in another format, see ``thumbnail_convert.py``). Format, quality and
|
||||
size limit are the server's thumbnail settings (``thumbnail_settings``,
|
||||
#317), read once when the job starts and shown in its scope; a ``size`` or
|
||||
``quality`` an older client sends is ignored. Every preview is written next
|
||||
to the old one, verified and renamed over it, so nothing is cleared first and
|
||||
a failure keeps the old preview; originals are only read.
|
||||
|
||||
The legacy gallery stream ``POST /images/generate-thumbnails/progress``
|
||||
starts this job (its quality preset still sizes the previews) or attaches to
|
||||
the running one. Its first frame, ``job {id, started}``, names that job and
|
||||
whether this stream started it (the gallery cancels by that id, and only a
|
||||
job it started); then come the job's legacy ``status``/``start``/
|
||||
``progress``/``file_error``/``complete`` frames. A client that goes away no
|
||||
longer stops the work: the job outlives the request, as a scan does.
|
||||
starts this job (missing previews, the server's settings; its ``quality``
|
||||
preset is ignored, #317 R3) or attaches to the running one. Its first frame,
|
||||
``job {id, started}``, names that job and whether this stream started it (the
|
||||
gallery cancels by that id, and only a job it started); then come the job's
|
||||
legacy ``status``/``start``/``progress``/``file_error``/``complete`` frames. A
|
||||
client that goes away no longer stops the work: the job outlives the request,
|
||||
as a scan does.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -36,7 +39,6 @@ from aiohttp import web
|
||||
|
||||
from .admin import _plural, _scan_location, _scan_roots, _scan_status_line
|
||||
from .images import (
|
||||
THUMBNAIL_ENCODINGS,
|
||||
_BodyTooLarge,
|
||||
_new_thumbnail_stats,
|
||||
_read_json_body,
|
||||
@@ -48,24 +50,13 @@ from .images import (
|
||||
)
|
||||
from .job_routes import busy_response
|
||||
from .jobs import THUMBNAIL_GROUP_NAMES, THUMBNAILS_JOB_KIND, JobBusy
|
||||
from .thumbnail_convert import CONVERT_JOB_TITLE
|
||||
from .thumbnail_encode import writer_settings
|
||||
from .thumbnail_settings import ThumbnailSettings
|
||||
|
||||
THUMBNAIL_JOB_TITLE = "Thumbnail generation"
|
||||
THUMBNAIL_MODES = ("missing", "rebuild")
|
||||
THUMBNAIL_QUALITIES = tuple(THUMBNAIL_ENCODINGS)
|
||||
MIN_THUMBNAIL_EDGE = 128
|
||||
MAX_THUMBNAIL_EDGE = 2048
|
||||
# The concept's "Preview size limit" default.
|
||||
DEFAULT_THUMBNAIL_EDGE = 512
|
||||
DEFAULT_THUMBNAIL_OPTIONS = {
|
||||
"mode": "missing",
|
||||
"size": DEFAULT_THUMBNAIL_EDGE,
|
||||
"quality": "medium",
|
||||
}
|
||||
_OPTIONS_ERROR = (
|
||||
'mode must be "missing" or "rebuild", quality "low", "medium" or "high", '
|
||||
f"and size a whole number of pixels from {MIN_THUMBNAIL_EDGE} "
|
||||
f"to {MAX_THUMBNAIL_EDGE}"
|
||||
)
|
||||
THUMBNAIL_MODES = ("missing", "rebuild", "convert")
|
||||
_OPTIONS_ERROR = 'mode must be "missing", "rebuild" or "convert"'
|
||||
CANCELLED_ERROR = "Thumbnail generation was cancelled"
|
||||
CANCELLED_BEFORE_WRITING = "Cancelled before any preview was written"
|
||||
NOTHING_TO_PREVIEW = "No images or videos to preview"
|
||||
@@ -80,56 +71,35 @@ _LOG_PREFIXES = {"generated": "+", "errors": "!"}
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ThumbnailOptions:
|
||||
"""One run's choices: the mode, the preview's longest edge and quality."""
|
||||
"""One run's choices: the mode and the thumbnail settings it writes with."""
|
||||
|
||||
mode: str
|
||||
edge: int
|
||||
quality: str
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return (self.edge, self.edge)
|
||||
|
||||
@property
|
||||
def encoding(self):
|
||||
return THUMBNAIL_ENCODINGS[self.quality]
|
||||
settings: ThumbnailSettings
|
||||
|
||||
@property
|
||||
def rebuild(self):
|
||||
return self.mode == "rebuild"
|
||||
|
||||
def scope(self):
|
||||
return {"mode": self.mode, "size": self.edge, "quality": self.quality}
|
||||
|
||||
|
||||
def _edge(value, *, from_query):
|
||||
"""The preview edge as an int, or None when it is not one in range."""
|
||||
if from_query and isinstance(value, str):
|
||||
if not value.isascii() or not value.isdigit() or len(value) > 5:
|
||||
return None
|
||||
value = int(value)
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
return None
|
||||
return value if MIN_THUMBNAIL_EDGE <= value <= MAX_THUMBNAIL_EDGE else None
|
||||
return {
|
||||
"mode": self.mode,
|
||||
"format": self.settings.format,
|
||||
"quality": self.settings.quality,
|
||||
"size": self.settings.edge,
|
||||
}
|
||||
|
||||
|
||||
def thumbnail_options(raw, *, from_query=False):
|
||||
"""``(ThumbnailOptions, None)`` from a body or query, or ``(None, message)``.
|
||||
|
||||
Absent fields take ``DEFAULT_THUMBNAIL_OPTIONS``. Query values are
|
||||
strings, so a query ``size`` may be decimal digits; a JSON ``size`` must
|
||||
be an int.
|
||||
An absent mode is ``missing``; the settings are the server's now.
|
||||
"""
|
||||
if not isinstance(raw, dict) and not hasattr(raw, "get"):
|
||||
return None, _OPTIONS_ERROR
|
||||
mode = raw.get("mode", DEFAULT_THUMBNAIL_OPTIONS["mode"])
|
||||
quality = raw.get("quality", DEFAULT_THUMBNAIL_OPTIONS["quality"])
|
||||
edge = _edge(raw.get("size", DEFAULT_THUMBNAIL_EDGE), from_query=from_query)
|
||||
if mode not in THUMBNAIL_MODES or quality not in THUMBNAIL_QUALITIES:
|
||||
mode = raw.get("mode", "missing")
|
||||
if mode not in THUMBNAIL_MODES:
|
||||
return None, _OPTIONS_ERROR
|
||||
if edge is None:
|
||||
return None, _OPTIONS_ERROR
|
||||
return ThumbnailOptions(mode, edge, quality), None
|
||||
return ThumbnailOptions(mode, writer_settings()), None
|
||||
|
||||
|
||||
def _invalid(message):
|
||||
@@ -168,7 +138,10 @@ class ThumbnailJobsMixin:
|
||||
try:
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails, output_dirs, rebuild=options.rebuild
|
||||
self._plan_gallery_thumbnails,
|
||||
output_dirs,
|
||||
options.mode,
|
||||
settings=options.settings,
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail plan failed")
|
||||
@@ -185,6 +158,7 @@ class ThumbnailJobsMixin:
|
||||
"success": True,
|
||||
"targets": len(plan["targets"]),
|
||||
"fresh": plan["fresh"],
|
||||
"outdated": plan["outdated"],
|
||||
"remaining": plan["remaining"],
|
||||
"roots": [self._public_path(root) for root in output_dirs],
|
||||
**options.scope(),
|
||||
@@ -215,14 +189,20 @@ class ThumbnailJobsMixin:
|
||||
return web.json_response({"success": True, "job": job.summary()}, status=202)
|
||||
|
||||
def _start_thumbnail_job(self, options):
|
||||
"""Register the thumbnail job synchronously; raises JobBusy."""
|
||||
"""Register the thumbnail job synchronously; raises JobBusy.
|
||||
|
||||
``convert`` runs Convert outdated (``thumbnail_convert.py``) under the
|
||||
same kind, admission, cancel and groups.
|
||||
"""
|
||||
convert = options.mode == "convert"
|
||||
runner = self._run_convert_job if convert else self._run_thumbnail_job
|
||||
return self._jobs.start(
|
||||
THUMBNAILS_JOB_KIND,
|
||||
THUMBNAIL_JOB_TITLE,
|
||||
CONVERT_JOB_TITLE if convert else THUMBNAIL_JOB_TITLE,
|
||||
admission="guarded",
|
||||
cancellable=True,
|
||||
scope=options.scope(),
|
||||
runner=functools.partial(self._run_thumbnail_job, options=options),
|
||||
runner=functools.partial(runner, options=options),
|
||||
groups=THUMBNAIL_GROUP_NAMES,
|
||||
)
|
||||
|
||||
@@ -265,7 +245,7 @@ class ThumbnailJobsMixin:
|
||||
job.update(status_line=f"Looking for media files in {folders}")
|
||||
job.log_line(
|
||||
f"Thumbnails started · {folders} · {_MODE_LOG[options.mode]} · "
|
||||
f"{options.edge} px · {options.quality}"
|
||||
f"{options.settings.describe()}"
|
||||
)
|
||||
job.publish(
|
||||
_frame(
|
||||
@@ -273,41 +253,84 @@ class ThumbnailJobsMixin:
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails,
|
||||
output_dirs,
|
||||
rebuild=options.rebuild,
|
||||
stop=lambda: job.cancelled,
|
||||
)
|
||||
plan = await self._plan_thumbnail_batch(job, options, output_dirs)
|
||||
if job.cancelled: # M2: stopped during the walk, before any start frame
|
||||
return self._end_cancelled(job, 0, CANCELLED_BEFORE_WRITING)
|
||||
roots = _scan_roots(output_dirs)
|
||||
self._record_fresh(job, plan, roots)
|
||||
stats, start = await self._generate_thumbnail_targets(job, options, plan, roots)
|
||||
stats, start, left, attempted = await self._generate_in_batches(
|
||||
job, options, plan, roots, output_dirs
|
||||
)
|
||||
self.invalidate_gallery_cache()
|
||||
generated = _plural(stats["generated"], "preview")
|
||||
if job.cancelled:
|
||||
return self._end_cancelled(
|
||||
job, stats["generated"], f"Cancelled · {generated} written"
|
||||
)
|
||||
overall = {"targets": attempted, "skipped": plan["skipped"], "remaining": left}
|
||||
payload = _thumbnail_complete_payload(
|
||||
len(plan["targets"]) + plan["skipped"] + plan["remaining"],
|
||||
len(attempted) + plan["skipped"] + left,
|
||||
stats,
|
||||
start,
|
||||
remaining=plan["remaining"],
|
||||
processed=len(plan["targets"]),
|
||||
remaining=left,
|
||||
processed=len(attempted),
|
||||
)
|
||||
job.update(progress=100, status_line=_finished_line(plan, stats))
|
||||
if plan["remaining"]:
|
||||
job.log_line(
|
||||
f"{plan['remaining']:,} more previews are left for the next run"
|
||||
)
|
||||
job.update(progress=100, status_line=_finished_line(overall, stats))
|
||||
if left:
|
||||
job.log_line(f"{left:,} more previews are left for the next run")
|
||||
errors = _plural(len(stats["errors"]), "error")
|
||||
job.log_line(f"Thumbnails finished · {generated} written · {errors}")
|
||||
self.logger.info(payload["message"])
|
||||
job.publish(_frame("complete", payload))
|
||||
return "finished"
|
||||
|
||||
async def _plan_thumbnail_batch(self, job, options, output_dirs, exclude=None):
|
||||
"""One batch's plan (at most MAX_THUMBNAIL_FILES targets), off-loop."""
|
||||
return await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails,
|
||||
output_dirs,
|
||||
options.mode,
|
||||
stop=lambda: job.cancelled,
|
||||
settings=options.settings,
|
||||
exclude=exclude,
|
||||
)
|
||||
|
||||
async def _generate_in_batches(self, job, options, plan, roots, output_dirs):
|
||||
"""Work through ``plan``, then plan again after each batch until
|
||||
nothing is left, the job is cancelled, or a batch made no progress
|
||||
(every file failed: they are not planned again, and the rest waits
|
||||
for the next run). Files worked on are never planned twice. Returns
|
||||
``(stats, start, files left, files worked on)``; progress counts
|
||||
against the overall total."""
|
||||
run = {
|
||||
"index": 0,
|
||||
"batches": 0,
|
||||
"total": len(plan["targets"]) + plan["remaining"],
|
||||
}
|
||||
stats = _new_thumbnail_stats()
|
||||
stats["skipped"] = plan["skipped"]
|
||||
start = time.monotonic()
|
||||
self._publish_thumbnail_start(job, plan, run["total"])
|
||||
attempted = set()
|
||||
while True:
|
||||
before = stats["generated"] + stats["skipped"]
|
||||
await self._generate_thumbnail_targets(
|
||||
job, options, plan, roots, stats, run, start
|
||||
)
|
||||
attempted.update(src for src, _dst, _video in plan["targets"])
|
||||
progressed = stats["generated"] + stats["skipped"] > before
|
||||
if job.cancelled or not plan["remaining"] or not progressed:
|
||||
return stats, start, plan["remaining"], attempted
|
||||
plan = await self._plan_thumbnail_batch(
|
||||
job, options, output_dirs, exclude=attempted
|
||||
)
|
||||
if job.cancelled:
|
||||
return stats, start, 0, attempted
|
||||
run["total"] = run["index"] + len(plan["targets"]) + plan["remaining"]
|
||||
job.update(total=run["total"])
|
||||
if not plan["targets"]:
|
||||
return stats, start, plan["remaining"], attempted
|
||||
|
||||
@staticmethod
|
||||
def _end_cancelled(job, generated, status_line):
|
||||
"""Close a cancelled job: its status line, log line and error frame."""
|
||||
@@ -335,10 +358,10 @@ class ThumbnailJobsMixin:
|
||||
if plan["skipped"]:
|
||||
job.log_line(f"= {plan['skipped']:,} previews already up to date")
|
||||
|
||||
async def _generate_thumbnail_targets(self, job, options, plan, roots):
|
||||
"""Generate ``plan``'s targets in worker batches, publishing each file."""
|
||||
@staticmethod
|
||||
def _publish_thumbnail_start(job, plan, total):
|
||||
"""The legacy stream's start frame, with the job's overall total."""
|
||||
targets = plan["targets"]
|
||||
total = len(targets)
|
||||
video_count = sum(1 for _, _, is_video in targets if is_video)
|
||||
message = (
|
||||
f"Found {total - video_count} images and {video_count} videos to process"
|
||||
@@ -354,7 +377,7 @@ class ThumbnailJobsMixin:
|
||||
{
|
||||
"total_images": total,
|
||||
"skipped": plan["skipped"],
|
||||
"remaining": plan["remaining"],
|
||||
"remaining": 0,
|
||||
"phase": "processing",
|
||||
"image_count": total - video_count,
|
||||
"video_count": video_count,
|
||||
@@ -362,28 +385,30 @@ class ThumbnailJobsMixin:
|
||||
},
|
||||
)
|
||||
)
|
||||
stats = _new_thumbnail_stats()
|
||||
stats["skipped"] = plan["skipped"]
|
||||
start = time.monotonic()
|
||||
|
||||
async def _generate_thumbnail_targets(
|
||||
self, job, options, plan, roots, stats, run, start
|
||||
):
|
||||
"""Generate one batch (``plan``'s targets) in worker batches,
|
||||
publishing each file against the job's overall ``run`` counters."""
|
||||
targets = plan["targets"]
|
||||
batch_size = _thumbnail_batch_size(_worker_threads())
|
||||
generate = {
|
||||
"encoding": options.encoding,
|
||||
"rebuild": options.rebuild,
|
||||
"stop": lambda: job.cancelled,
|
||||
}
|
||||
index = 0
|
||||
batches = 0
|
||||
for batch_start in range(0, total, batch_size):
|
||||
for batch_start in range(0, len(targets), batch_size):
|
||||
if job.cancelled:
|
||||
break
|
||||
batch = targets[batch_start : batch_start + batch_size]
|
||||
results = await self._run_in_executor(
|
||||
self._generate_batch, batch, options.size, **generate
|
||||
self._generate_batch, batch, options.settings, **generate
|
||||
)
|
||||
for (src, _dst, _video), result in zip(batch, results):
|
||||
if result["action"] == "cancelled":
|
||||
continue
|
||||
index += 1
|
||||
run["index"] += 1
|
||||
index, total = run["index"], max(run["total"], run["index"])
|
||||
_record_thumbnail_result(stats, result)
|
||||
self._record_thumbnail_outcome(job, src, result, roots)
|
||||
job.update(processed=index, progress=index * 100 // total)
|
||||
@@ -400,14 +425,13 @@ class ThumbnailJobsMixin:
|
||||
_thumbnail_progress_payload(index, total, stats, result, start),
|
||||
)
|
||||
)
|
||||
batches += 1
|
||||
run["batches"] += 1
|
||||
job.update(
|
||||
status_line=_scan_status_line(
|
||||
index, total, batches, time.monotonic() - start
|
||||
run["index"], run["total"], run["batches"], time.monotonic() - start
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0) # let other requests run between batches
|
||||
return stats, start
|
||||
|
||||
@staticmethod
|
||||
def _record_thumbnail_outcome(job, src, result, roots):
|
||||
@@ -430,14 +454,13 @@ class ThumbnailJobsMixin:
|
||||
async def generate_thumbnails_with_progress(self, request):
|
||||
"""The gallery's thumbnail stream (POST, server-sent events).
|
||||
|
||||
Starts the thumbnail job with the gallery's quality preset as the
|
||||
preview size (missing previews, medium encoding, as before), or
|
||||
attaches to the running thumbnail job, and streams its frames. A
|
||||
Starts the thumbnail job (missing previews with the server's
|
||||
thumbnail settings; the gallery's ``quality`` preset is ignored, R3),
|
||||
or attaches to the running thumbnail job, and streams its frames. A
|
||||
legacy hold (the blocking generate or clear) or a record-changing
|
||||
job refuses it with the stream's error frame. The job goes on when
|
||||
the client goes away.
|
||||
"""
|
||||
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
|
||||
response = web.StreamResponse(
|
||||
status=200,
|
||||
headers={
|
||||
@@ -449,7 +472,7 @@ class ThumbnailJobsMixin:
|
||||
await response.prepare(request)
|
||||
# M3: read the registry after the last await, so a job that started
|
||||
# while the headers went out is attached to, not refused.
|
||||
job, started, busy = self._legacy_stream_job(thumbnail_size)
|
||||
job, started, busy = self._legacy_stream_job()
|
||||
if busy is not None:
|
||||
message = f"{busy.title} already running"
|
||||
await self._emit_sse(
|
||||
@@ -476,19 +499,21 @@ class ThumbnailJobsMixin:
|
||||
await events.aclose()
|
||||
return response
|
||||
|
||||
def _legacy_stream_job(self, thumbnail_size):
|
||||
def _legacy_stream_job(self):
|
||||
"""``(job, started, busy)`` for the legacy stream, synchronously.
|
||||
|
||||
Attaches to the running cancellable thumbnail job, or starts one with
|
||||
the gallery's preset as the size (missing previews, medium encoding,
|
||||
as the gallery always wrote), or names the job in the way.
|
||||
Attaches to the running cancellable thumbnail job, or starts one for
|
||||
the missing previews with the server's settings, or names the job in
|
||||
the way.
|
||||
"""
|
||||
running = self._jobs.active(THUMBNAILS_JOB_KIND)
|
||||
if running is not None and running.scope.get("mode") == "convert":
|
||||
return None, False, running # no legacy frames to follow (#317)
|
||||
if running is not None and running.cancellable:
|
||||
return running, False, None
|
||||
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
|
||||
if busy is not None:
|
||||
return None, False, busy
|
||||
# blocker() said free and nothing awaited since, so the start is admitted.
|
||||
options = ThumbnailOptions("missing", thumbnail_size[0], "medium")
|
||||
options = ThumbnailOptions("missing", writer_settings())
|
||||
return self._start_thumbnail_job(options), True, None
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Thumbnail settings and names (#317): the one place every preview writer reads.
|
||||
|
||||
Settings (System › Appearance, ``POST /prompt_manager/settings``, config.json
|
||||
``thumbnails``): ``thumbnail_format`` (``webp`` default, ``jxl``, ``jpeg``),
|
||||
``thumbnail_quality`` (1–100, default 80) and ``thumbnail_edge`` (the longest
|
||||
edge, 128–2048 px, default 512). ``current()`` is the snapshot a writer takes
|
||||
once per preview; automatic previews, the Activity job, the history route and
|
||||
the legacy gallery routes all go through it.
|
||||
|
||||
Names: a media file ``<rel>/<stem>.<ext>`` previews as
|
||||
``thumbnails/<rel>/<stem>_thumb<extension of the format>`` (``.webp``,
|
||||
``.jxl``, ``.jpg``), images and videos alike. Two files with one stem in one
|
||||
folder share that name (ruling R2). Older releases kept the source's own
|
||||
extension (``.png``, ``.jpeg``, ``.gif``, ``.webp``, ``.jpg``; ``.jpg`` for a
|
||||
video): those are read, served and converted, never written. A preview with a
|
||||
format extension (``.webp``, ``.jxl``, ``.jpg``) belongs to every same-stem
|
||||
source; a legacy-only one (``.png``, ``.jpeg``, ``.gif``) only to a source
|
||||
with that extension (``owns``, ruling R4). Lookups take the first existing
|
||||
preview the source owns, preferring the configured format.
|
||||
"""
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
FORMATS = ("webp", "jxl", "jpeg")
|
||||
FORMAT_EXTENSIONS = {"webp": ".webp", "jxl": ".jxl", "jpeg": ".jpg"}
|
||||
FORMAT_LABELS = {"webp": "WebP", "jxl": "JPEG XL", "jpeg": "JPEG"}
|
||||
DEFAULT_FORMAT = "webp"
|
||||
DEFAULT_QUALITY = 80
|
||||
DEFAULT_EDGE = 512
|
||||
MIN_QUALITY, MAX_QUALITY = 1, 100
|
||||
MIN_EDGE, MAX_EDGE = 128, 2048
|
||||
|
||||
# Extensions a format writes, then the ones only older releases wrote.
|
||||
WRITTEN_EXTENSIONS = (".webp", ".jxl", ".jpg")
|
||||
LEGACY_EXTENSIONS = (".png", ".jpeg", ".gif")
|
||||
THUMBNAIL_EXTENSIONS = WRITTEN_EXTENSIONS + LEGACY_EXTENSIONS
|
||||
CONTENT_TYPES = {
|
||||
".webp": "image/webp",
|
||||
".jxl": "image/jxl",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".png": "image/png",
|
||||
".gif": "image/gif",
|
||||
}
|
||||
THUMB_SUFFIX = "_thumb"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ThumbnailSettings:
|
||||
"""One preview's settings: the format, the quality and the longest edge."""
|
||||
|
||||
format: str
|
||||
quality: int
|
||||
edge: int
|
||||
|
||||
@property
|
||||
def extension(self):
|
||||
return FORMAT_EXTENSIONS[self.format]
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return (self.edge, self.edge)
|
||||
|
||||
def describe(self):
|
||||
"""``WebP · quality 80 · 512 px`` (log lines and the job's scope)."""
|
||||
return f"{FORMAT_LABELS[self.format]} · quality {self.quality} · {self.edge} px"
|
||||
|
||||
|
||||
def current():
|
||||
"""The configured settings now (read once per preview or per run)."""
|
||||
from ..config import PromptManagerConfig
|
||||
|
||||
fmt = PromptManagerConfig.THUMBNAIL_FORMAT
|
||||
return ThumbnailSettings(
|
||||
fmt if fmt in FORMATS else DEFAULT_FORMAT,
|
||||
int(PromptManagerConfig.THUMBNAIL_QUALITY),
|
||||
int(PromptManagerConfig.THUMBNAIL_EDGE),
|
||||
)
|
||||
|
||||
|
||||
def _whole(value, low, high, message):
|
||||
if type(value) is not int or not low <= value <= high:
|
||||
raise ValueError(message)
|
||||
return value
|
||||
|
||||
|
||||
def validate_format(value):
|
||||
if not isinstance(value, str) or value not in FORMATS:
|
||||
raise ValueError('thumbnail_format must be "webp", "jxl" or "jpeg"')
|
||||
return value
|
||||
|
||||
|
||||
def validate_quality(value):
|
||||
return _whole(
|
||||
value,
|
||||
MIN_QUALITY,
|
||||
MAX_QUALITY,
|
||||
f"thumbnail_quality must be a whole number from {MIN_QUALITY} to {MAX_QUALITY}",
|
||||
)
|
||||
|
||||
|
||||
def validate_edge(value):
|
||||
return _whole(
|
||||
value,
|
||||
MIN_EDGE,
|
||||
MAX_EDGE,
|
||||
"thumbnail_edge must be a whole number of pixels "
|
||||
f"from {MIN_EDGE} to {MAX_EDGE}",
|
||||
)
|
||||
|
||||
|
||||
# ── names ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def thumbnail_stem_rel(rel_path):
|
||||
"""``thumbnails/<rel without extension>_thumb`` (POSIX)."""
|
||||
return f"thumbnails/{Path(rel_path).with_suffix('').as_posix()}{THUMB_SUFFIX}"
|
||||
|
||||
|
||||
def thumbnail_rel(rel_path, extension=None):
|
||||
"""The preview of *rel_path* in *extension* (the configured one when None)."""
|
||||
return thumbnail_stem_rel(rel_path) + (extension or current().extension)
|
||||
|
||||
|
||||
# Ruling R2: files sharing a stem share one preview, made from the first
|
||||
# source in this order: these images, then videos (by extension name).
|
||||
IMAGE_SOURCE_ORDER = (".png", ".jpg", ".jpeg", ".webp", ".gif")
|
||||
|
||||
|
||||
def source_rank(source_extension):
|
||||
"""Sort key of a source extension in R2's order (images first)."""
|
||||
extension = source_extension.lower()
|
||||
if extension in IMAGE_SOURCE_ORDER:
|
||||
return (0, IMAGE_SOURCE_ORDER.index(extension), extension)
|
||||
return (1, 0, extension)
|
||||
|
||||
|
||||
def owns(thumb_extension, source_extension):
|
||||
"""Whether a ``_thumb<thumb_extension>`` preview can belong to a source
|
||||
with *source_extension* (ruling R4)."""
|
||||
thumb_extension = thumb_extension.lower()
|
||||
if thumb_extension in WRITTEN_EXTENSIONS:
|
||||
return True
|
||||
if thumb_extension in LEGACY_EXTENSIONS:
|
||||
return thumb_extension == source_extension.lower()
|
||||
return False
|
||||
|
||||
|
||||
def _legacy_extension(source_extension):
|
||||
from .inspection_service import VIDEO_EXTENSIONS
|
||||
|
||||
source_extension = source_extension.lower()
|
||||
return ".jpg" if source_extension in VIDEO_EXTENSIONS else source_extension
|
||||
|
||||
|
||||
def candidate_extensions(source_extension, preferred=None):
|
||||
"""The preview extensions a source owns, in lookup order: the configured
|
||||
one, the name an older release gave it, then the other written ones."""
|
||||
preferred = preferred or current().extension
|
||||
ordered = [preferred, _legacy_extension(source_extension), *WRITTEN_EXTENSIONS]
|
||||
seen = []
|
||||
for extension in ordered:
|
||||
if extension not in seen and owns(extension, source_extension):
|
||||
seen.append(extension)
|
||||
return tuple(seen)
|
||||
|
||||
|
||||
def owned_extensions(source_extension, preferred=None):
|
||||
"""``candidate_extensions`` plus the other spellings an older release
|
||||
may have given a legacy preview (review M1): the source's own
|
||||
(``X_thumb.JPG`` for ``X.JPG``) and the upper-case legacy one."""
|
||||
extensions = list(candidate_extensions(source_extension, preferred))
|
||||
legacy = _legacy_extension(source_extension)
|
||||
for spelling in (source_extension, legacy.upper()):
|
||||
if spelling not in extensions and owns(spelling, source_extension):
|
||||
extensions.append(spelling)
|
||||
return tuple(extensions)
|
||||
|
||||
|
||||
def _is_preview_file(path):
|
||||
try:
|
||||
return not path.is_symlink() and path.is_file()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def owned_in_folder(folder, stem, source_extension, preferred=None):
|
||||
"""Yield the existing previews ``<folder>/<stem><ext>`` a source with
|
||||
*source_extension* owns, in lookup order (regular files, no links; one
|
||||
file found under two spellings on a case-insensitive disk once)."""
|
||||
seen = set()
|
||||
for extension in owned_extensions(source_extension, preferred):
|
||||
path = Path(folder) / (stem + extension)
|
||||
if not _is_preview_file(path):
|
||||
continue
|
||||
try:
|
||||
stat = path.stat()
|
||||
identity = (stat.st_dev, stat.st_ino) if stat.st_ino else str(path)
|
||||
except OSError:
|
||||
continue
|
||||
if identity in seen:
|
||||
continue # the same file under another case (Windows, macOS)
|
||||
seen.add(identity)
|
||||
yield path
|
||||
|
||||
|
||||
def _owned(root, rel_path, preferred):
|
||||
rel_path = Path(rel_path)
|
||||
stem = Path(root) / thumbnail_stem_rel(rel_path)
|
||||
return owned_in_folder(stem.parent, stem.name, rel_path.suffix, preferred)
|
||||
|
||||
|
||||
def owned_thumbnails(root, rel_path, preferred=None):
|
||||
"""Every existing preview of *rel_path* under *root*, in lookup order."""
|
||||
return list(_owned(root, rel_path, preferred))
|
||||
|
||||
|
||||
def existing_thumbnail(root, rel_path, preferred=None):
|
||||
"""The preview a listing shows for *rel_path*: the first owned one that
|
||||
exists (the configured format first), or None."""
|
||||
return next(_owned(root, rel_path, preferred), None)
|
||||
|
||||
|
||||
def content_type(path):
|
||||
"""The Content-Type of a preview file by its extension, or None."""
|
||||
return CONTENT_TYPES.get(os.path.splitext(str(path))[1].lower())
|
||||
+50
-12
@@ -36,12 +36,14 @@ from typing import Any, Dict, List, Optional, Tuple # noqa: E402
|
||||
# Import logging system
|
||||
try:
|
||||
from ..utils.logging_config import get_logger
|
||||
from ..utils import library_location
|
||||
except ImportError:
|
||||
import sys
|
||||
|
||||
current_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, current_dir)
|
||||
from utils.logging_config import get_logger
|
||||
from utils import library_location
|
||||
|
||||
# Initialize logger for config operations
|
||||
config_logger = get_logger("prompt_manager.config")
|
||||
@@ -200,7 +202,6 @@ class GalleryConfig:
|
||||
AUTO_CLEANUP_MISSING_FILES (bool): Automatically remove missing file records
|
||||
MAX_IMAGE_AGE_DAYS (int): Maximum age in days before cleaning up images
|
||||
IMAGES_PER_PAGE (int): Number of images to display per page in web UI
|
||||
THUMBNAIL_SIZE (int): Size in pixels for generated thumbnails
|
||||
ENABLE_SEARCH (bool): Enable search functionality in web interface
|
||||
ENABLE_METADATA_VIEW (bool): Enable metadata viewing for images
|
||||
MAX_CONCURRENT_PROCESSING (int): Maximum concurrent image processing tasks
|
||||
@@ -225,7 +226,6 @@ class GalleryConfig:
|
||||
|
||||
# Web interface settings
|
||||
IMAGES_PER_PAGE = 20
|
||||
THUMBNAIL_SIZE = 256
|
||||
ENABLE_SEARCH = True
|
||||
ENABLE_METADATA_VIEW = True
|
||||
|
||||
@@ -265,7 +265,6 @@ class GalleryConfig:
|
||||
},
|
||||
"web_interface": {
|
||||
"images_per_page": cls.IMAGES_PER_PAGE,
|
||||
"thumbnail_size": cls.THUMBNAIL_SIZE,
|
||||
"enable_search": cls.ENABLE_SEARCH,
|
||||
"enable_metadata_view": cls.ENABLE_METADATA_VIEW,
|
||||
},
|
||||
@@ -328,8 +327,6 @@ class GalleryConfig:
|
||||
web_interface = new_config.get("web_interface", {})
|
||||
if "images_per_page" in web_interface:
|
||||
cls.IMAGES_PER_PAGE = web_interface["images_per_page"]
|
||||
if "thumbnail_size" in web_interface:
|
||||
cls.THUMBNAIL_SIZE = web_interface["thumbnail_size"]
|
||||
if "enable_search" in web_interface:
|
||||
cls.ENABLE_SEARCH = web_interface["enable_search"]
|
||||
if "enable_metadata_view" in web_interface:
|
||||
@@ -663,6 +660,19 @@ class PromptManagerConfig:
|
||||
# database first (System › Backup & export › Recovery policy, #249 S6).
|
||||
SAFETY_BACKUP = True
|
||||
|
||||
# "Keep it here" (#249 S6b): the user chose to keep a node-folder library
|
||||
# where it is, so the workspace stops offering to move it.
|
||||
KEEP_LIBRARY_IN_NODE_FOLDER = False
|
||||
|
||||
# Thumbnails (#317): every preview writer reads these through
|
||||
# py/api/thumbnail_settings.py (System › Appearance). The legacy
|
||||
# ``gallery.web_interface.thumbnail_size`` was never read and is ignored.
|
||||
THUMBNAIL_FORMATS = ("webp", "jxl", "jpeg")
|
||||
THUMBNAIL_FORMAT = "webp"
|
||||
THUMBNAIL_QUALITY = 80
|
||||
THUMBNAIL_EDGE = 512
|
||||
THUMBNAIL_EDGE_RANGE = (128, 2048)
|
||||
|
||||
@classmethod
|
||||
def max_worker_threads(cls) -> int:
|
||||
"""Largest accepted worker-thread count: the detected core count."""
|
||||
@@ -718,6 +728,12 @@ class PromptManagerConfig:
|
||||
"worker_threads": cls.WORKER_THREADS,
|
||||
},
|
||||
"maintenance": {"safety_backup": cls.SAFETY_BACKUP},
|
||||
"library": {"keep_in_node_folder": cls.KEEP_LIBRARY_IN_NODE_FOLDER},
|
||||
"thumbnails": {
|
||||
"format": cls.THUMBNAIL_FORMAT,
|
||||
"quality": cls.THUMBNAIL_QUALITY,
|
||||
"edge": cls.THUMBNAIL_EDGE,
|
||||
},
|
||||
"gallery": GalleryConfig.get_config(),
|
||||
"integrations": IntegrationConfig.get_config(),
|
||||
"autotag": AutotagConfig.get_config(),
|
||||
@@ -727,14 +743,12 @@ class PromptManagerConfig:
|
||||
def get_config_path(cls) -> str:
|
||||
"""Path of the persisted config.json.
|
||||
|
||||
Honours ``PROMPT_MANAGER_CONFIG_PATH`` when set; otherwise the file
|
||||
lives at the repository root next to ``pyproject.toml``.
|
||||
Honours ``PROMPT_MANAGER_CONFIG_PATH`` when set; otherwise it lives in
|
||||
the library's data folder (``utils.library_location``): the node
|
||||
folder for a library that was not moved, else ComfyUI's
|
||||
``user/__promptmanager``.
|
||||
"""
|
||||
override = os.environ.get(CONFIG_PATH_ENV, "").strip()
|
||||
if override:
|
||||
return override
|
||||
repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
return os.path.join(repo_root, "config.json")
|
||||
return library_location.config_location()[0]
|
||||
|
||||
@classmethod
|
||||
def load_from_file(cls, config_path: Optional[str] = None):
|
||||
@@ -859,6 +873,14 @@ class PromptManagerConfig:
|
||||
):
|
||||
cls.SAFETY_BACKUP = maintenance["safety_backup"]
|
||||
|
||||
library = new_config.get("library", {})
|
||||
if isinstance(library, dict) and isinstance(
|
||||
library.get("keep_in_node_folder"), bool
|
||||
):
|
||||
cls.KEEP_LIBRARY_IN_NODE_FOLDER = library["keep_in_node_folder"]
|
||||
|
||||
cls._update_thumbnails(new_config.get("thumbnails"))
|
||||
|
||||
# Update gallery config
|
||||
if "gallery" in new_config:
|
||||
GalleryConfig.update_config(new_config["gallery"])
|
||||
@@ -870,6 +892,22 @@ class PromptManagerConfig:
|
||||
if "autotag" in new_config:
|
||||
AutotagConfig.update_config(new_config["autotag"])
|
||||
|
||||
@classmethod
|
||||
def _update_thumbnails(cls, section: Any) -> None:
|
||||
"""Apply a ``thumbnails`` section; a hand-edited bad value is ignored."""
|
||||
if not isinstance(section, dict):
|
||||
return
|
||||
fmt = section.get("format")
|
||||
if fmt in cls.THUMBNAIL_FORMATS:
|
||||
cls.THUMBNAIL_FORMAT = fmt
|
||||
quality = section.get("quality")
|
||||
if type(quality) is int and 1 <= quality <= 100:
|
||||
cls.THUMBNAIL_QUALITY = quality
|
||||
edge = section.get("edge")
|
||||
low, high = cls.THUMBNAIL_EDGE_RANGE
|
||||
if type(edge) is int and low <= edge <= high:
|
||||
cls.THUMBNAIL_EDGE = edge
|
||||
|
||||
|
||||
# Load configuration on import
|
||||
try:
|
||||
|
||||
+14
-3
@@ -567,10 +567,21 @@ def download_civitai_images(
|
||||
|
||||
|
||||
def get_lora_image_cache_dir() -> Path:
|
||||
"""Get the directory used to cache downloaded LoRA example images."""
|
||||
# Store in the extension's own directory
|
||||
"""Get the directory used to cache downloaded LoRA example images.
|
||||
|
||||
``<data folder>/data/lora_images`` (``utils.library_location``): the node
|
||||
folder for a library that was not moved, else ComfyUI's
|
||||
``user/__promptmanager``.
|
||||
"""
|
||||
try:
|
||||
from ..utils import library_location
|
||||
except ImportError:
|
||||
from utils import library_location
|
||||
|
||||
ext_root = Path(__file__).resolve().parent.parent
|
||||
cache = ext_root / "data" / "lora_images"
|
||||
cache = Path(
|
||||
library_location.cache_dir("lora_images", extension_root=str(ext_root))
|
||||
)
|
||||
cache.mkdir(parents=True, exist_ok=True)
|
||||
return cache
|
||||
|
||||
|
||||
+22
-1
@@ -3,6 +3,7 @@
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
@@ -355,7 +356,12 @@ def create_app(root: Path) -> web.Application:
|
||||
with Image.new("RGB", (64, 64), (100, 80, 120)) as image:
|
||||
image.save(root / "output" / name, pnginfo=info)
|
||||
os.utime(root / "output" / name, (time.time() - 100, time.time() - 100))
|
||||
api._generate_gallery_thumbnails_sync(api._get_all_output_dirs(), (150, 150))
|
||||
from py.api.thumbnail_settings import ThumbnailSettings
|
||||
|
||||
# 150 px previews in the default format (#317 thumbnail settings).
|
||||
api._generate_gallery_thumbnails_sync(
|
||||
api._get_all_output_dirs(), ThumbnailSettings("webp", 80, 150)
|
||||
)
|
||||
if include_video:
|
||||
# Seed after thumbnail generation to exercise a real unthumbnailed video.
|
||||
video = root / "output" / "clip.webm"
|
||||
@@ -386,6 +392,8 @@ def create_app(root: Path) -> web.Application:
|
||||
async def reset(request: web.Request) -> web.Response:
|
||||
nonlocal history_fixture
|
||||
request_holds.release()
|
||||
# No preview queued by the last test may land in the new fixture.
|
||||
await api._run_in_executor(api.close_auto_thumbnails)
|
||||
library_editor_commands.reset()
|
||||
data = await request.json() if request.can_read_body else {}
|
||||
if not isinstance(data, dict):
|
||||
@@ -654,6 +662,18 @@ def exit_when_stdin_closes(loop: asyncio.AbstractEventLoop) -> threading.Thread:
|
||||
return thread
|
||||
|
||||
|
||||
def close_owned_log_files() -> None:
|
||||
"""Close Prompt Manager's own log handlers before the temp root goes.
|
||||
|
||||
The logs live in the library's data folder (#249 S6b), which for this
|
||||
server is <root>/user/__promptmanager; Windows cannot remove a file
|
||||
this process still has open.
|
||||
"""
|
||||
for handler in logging.getLogger("prompt_manager").handlers[:]:
|
||||
if getattr(handler, "_prompt_manager_owned", False) is True:
|
||||
handler.close()
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
@@ -678,6 +698,7 @@ def main(argv: list[str] | None = None) -> None:
|
||||
finally:
|
||||
# create_app works inside the root; Windows cannot remove a process's cwd.
|
||||
os.chdir(original_cwd)
|
||||
close_owned_log_files()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -181,6 +181,17 @@ function mapOverviewConcept(reference, overview) {
|
||||
// D96: the dialog shows runs and ratings by category and library totals, not prompt, tag or date usage.
|
||||
Statistics: row => ({ ...row, description: 'Runs and ratings by category, with library totals.' }),
|
||||
};
|
||||
const surfaces = reference.columns.surfaces.map(surface => ({ ...surface,
|
||||
rows: surface.rows.map(row => (rows[row.title] ?? (value => value))(row)) }));
|
||||
// D129: "Library location" after Backup status, in the backup row's own markup; the fixture's database is configured
|
||||
// explicitly (config.json), so it does not move and its action is Details.
|
||||
const [health] = surfaces;
|
||||
const backup = health.rows.find(row => row.title === 'Backup status');
|
||||
const location = { ...backup, title: 'Library location',
|
||||
description: `${overview.location.path} · The database path set in config.json.`,
|
||||
action: { ...backup.action, text: 'Details' } };
|
||||
surfaces[0] = { ...health, tree: [...health.tree, 'div.operation'],
|
||||
rows: [...health.rows.slice(0, 2), location, ...health.rows.slice(2)] };
|
||||
return {
|
||||
...reference,
|
||||
// D92: no "Sample configuration" chip.
|
||||
@@ -188,8 +199,7 @@ function mapOverviewConcept(reference, overview) {
|
||||
// D122: the concept's literal label is its prototype action name ("system-tab sections").
|
||||
nav: { ...reference.nav, label: 'System sections' },
|
||||
metrics: { ...reference.metrics, items: reference.metrics.items.map(item => ({ ...item, value: relabel[item.label] })) },
|
||||
columns: { ...reference.columns, surfaces: reference.columns.surfaces.map(surface => ({ ...surface,
|
||||
rows: surface.rows.map(row => (rows[row.title] ?? (value => value))(row)) })) },
|
||||
columns: { ...reference.columns, surfaces },
|
||||
save: { ...reference.save,
|
||||
// D92: truthful status copy, and "Save preferences" is aria-disabled (dimmed) while nothing is unsaved.
|
||||
status: { ...reference.save.status, text: 'No unsaved changes.' },
|
||||
@@ -232,13 +242,21 @@ for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
|
||||
// The recorded differences, by path: D93 the review row (its real count), the backup row, the
|
||||
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
|
||||
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
|
||||
// Save; D122 the nav label; D129 the inserted "Library location" row, which shifts the thumbnail row to the fourth place
|
||||
// (row 2 takes its title, text and action; row 3 and the surface's fifth child are new). (The fixture's first three
|
||||
// metrics are the concept's own counts.)
|
||||
const D129_ROW_ACTION = ['backgroundColor', 'borderTopColor', 'borderTopLeftRadius', 'borderTopWidth', 'className', 'color',
|
||||
'cursor', 'fontSize', 'opacity', 'paddingLeft', 'paddingTop', 'text'].map(key => `columns.surfaces.0.rows.3.action.${key}`);
|
||||
const UNMAPPED_PATHS = [
|
||||
'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
|
||||
'columns.surfaces.0.rows.2.action.text', 'columns.surfaces.0.rows.2.description', 'columns.surfaces.0.rows.2.title',
|
||||
...D129_ROW_ACTION, 'columns.surfaces.0.rows.3.description', 'columns.surfaces.0.rows.3.title',
|
||||
'columns.surfaces.0.rows.3.tree.0', 'columns.surfaces.0.rows.3.tree.1', 'columns.surfaces.0.rows.3.tree.length',
|
||||
'columns.surfaces.0.rows.length', 'columns.surfaces.0.tree.4', 'columns.surfaces.0.tree.length',
|
||||
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
|
||||
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
|
||||
'save.button.cursor', 'save.button.opacity', 'save.status.text',
|
||||
];
|
||||
].sort();
|
||||
|
||||
test('system comparison detects mutations', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
@@ -464,6 +482,8 @@ function mapAppearanceConcept(reference, settings) {
|
||||
control: { ...toggleRow.control, id, checked } }));
|
||||
return {
|
||||
...reference,
|
||||
// #317: the server's thumbnail settings follow in their own surface ("Thumbnails", README: retained in Appearance).
|
||||
layout: [reference.layout[0], 'section.surface', ...reference.layout.slice(1)],
|
||||
// D97: two added toggles and the "Reset display defaults" operation close the surface.
|
||||
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.setting-row', 'div.setting-row', 'div.operation'] },
|
||||
rows: [...reference.rows.map(row => row.control?.id === 'timeout'
|
||||
@@ -512,7 +532,8 @@ test('appearance comparison detects mutations', async ({ page, request }) => {
|
||||
// Unmapped, the concept differs only where D97 and the real timeout are recorded.
|
||||
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
|
||||
.filter((path, index, all) => all.indexOf(path) === index).sort())
|
||||
.toEqual(['operations.0', 'operations.length', 'rows.11', 'rows.12', 'rows.13', 'rows.length', 'surface.tree'].sort());
|
||||
.toEqual(['layout.1', 'layout.2', 'layout.length', 'operations.0', 'operations.length', 'rows.11', 'rows.12', 'rows.13',
|
||||
'rows.length', 'surface.tree'].sort());
|
||||
const measure = () => page.evaluate(measureAppearance, '#workspace-main');
|
||||
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
|
||||
// A removed row.
|
||||
@@ -824,13 +845,16 @@ function measureThumbnailReview() {
|
||||
|
||||
/**
|
||||
* D99: the planned counts line (`p.small[role=status]`) before the actions, from the server's plan for the chosen
|
||||
* options, and the primary "Start" (the concept's "Simulate thumbnail job").
|
||||
* options, and the primary "Start" (the concept's "Simulate thumbnail job"). D131 (#317): the Quality and Preview size
|
||||
* limit fields give way to a second `p.small` naming the server's thumbnail settings, and Scope gains "Convert
|
||||
* outdated (removes older formats)".
|
||||
*/
|
||||
function mapThumbnailConcept(reference, plan) {
|
||||
const [cancel, start] = reference.actions.buttons;
|
||||
const tree = [...reference.tree];
|
||||
tree.splice(tree.indexOf('div.dialog-actions'), 0, 'p.small');
|
||||
return { ...reference, tree,
|
||||
const [scope] = reference.fields;
|
||||
const tree = ['div.field', 'p.small', 'p.small', 'div.dialog-actions'];
|
||||
const fields = [{ ...scope, control: { ...scope.control, options: [...scope.control.options, 'Convert outdated (removes older formats)'] } }];
|
||||
return { ...reference, tree, fields,
|
||||
plan: { ...reference.plan, role: 'status',
|
||||
text: `${number(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ${number(plan.fresh)} already up to date.` },
|
||||
actions: { ...reference.actions, buttons: [cancel, { ...start, text: 'Start' }] } };
|
||||
@@ -856,12 +880,12 @@ async function productionThumbnailReview(page, theme, entry) {
|
||||
await page.locator('#workspace-main').getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
}
|
||||
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await expect(page.locator('dialog#modal[open] #modal-body > p.small')).toHaveText(/ already up to date\.$/);
|
||||
await expect(page.locator('dialog#modal[open] #modal-body > p.small').first()).toHaveText(/ already up to date\.$/);
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureThumbnailReview);
|
||||
}
|
||||
|
||||
const thumbnailPlan = request => serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
|
||||
const thumbnailPlan = request => serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing');
|
||||
|
||||
for (const entry of ['sources', 'overview']) for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
test(`thumbnail dialog matches the concept from ${entry} ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
|
||||
@@ -884,22 +908,106 @@ test('thumbnail dialog comparison detects mutations', async ({ page, request })
|
||||
const expected = mapThumbnailConcept(concept, plan);
|
||||
const production = await productionThumbnailReview(page, 'dark', 'sources');
|
||||
expect(diffInventory(expected, production)).toEqual([]);
|
||||
// Unmapped, the concept differs only where D99 is recorded.
|
||||
expect(diffInventory(concept, production).map(item => item.path).sort())
|
||||
.toEqual(['actions.buttons.1.text', 'plan.role', 'plan.text', 'tree.3', 'tree.4', 'tree.length']);
|
||||
// Unmapped, the concept differs only where D99 and D131 are recorded.
|
||||
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
|
||||
.filter((path, index, all) => all.indexOf(path) === index).sort())
|
||||
.toEqual(['actions.buttons', 'fields.0', 'fields.1', 'fields.2', 'fields.length', 'plan.role', 'plan.text', 'tree.1', 'tree.2']
|
||||
.sort());
|
||||
const paths = async () => diffInventory(expected, await page.evaluate(measureThumbnailReview)).map(item => item.path);
|
||||
// A changed option label, a swapped option order, a swapped action order and a changed style.
|
||||
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium'; });
|
||||
expect(await paths()).toEqual(['fields.1.control.options.0']);
|
||||
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium · optimized'; });
|
||||
await page.evaluate(() => { document.querySelector('#thumb-mode option').textContent = 'Missing only'; });
|
||||
expect(await paths()).toEqual(['fields.0.control.options.0']);
|
||||
await page.evaluate(() => { document.querySelector('#thumb-mode option').textContent = 'Missing thumbnails only'; });
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.0.control.options.0', 'fields.0.control.options.1']));
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); select.selectedIndex = 0; });
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.append(select.firstElementChild); select.selectedIndex = 0; });
|
||||
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
|
||||
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
|
||||
const undoStyle = await mutate(page, { css: 'dialog#modal #thumb-limit { border-color: rgb(255, 0, 0); } dialog#modal #modal-body > p.small { color: rgb(255, 0, 0); }' });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.2.control.borderTopColor', 'plan.color']));
|
||||
const undoStyle = await mutate(page, { css: 'dialog#modal #thumb-mode { border-color: rgb(255, 0, 0); } dialog#modal #modal-body > p.small { color: rgb(255, 0, 0); }' });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.0.control.borderTopColor', 'plan.color']));
|
||||
await undoStyle();
|
||||
expect(diffInventory(expected, await page.evaluate(measureThumbnailReview))).toEqual([]);
|
||||
});
|
||||
|
||||
/**
|
||||
* Runs in the page: one `.notice` (the concept's markup: `div > h2 + p`, then `.row-actions` buttons) with its tokens and
|
||||
* geometry, and its left edge against `edgeSelector` (the content it lines up with).
|
||||
*/
|
||||
function measureNoticeOf([selector, edgeSelector]) {
|
||||
const notice = document.querySelector(selector);
|
||||
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
|
||||
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
|
||||
const text = node => node?.textContent.trim() ?? null;
|
||||
const button = node => ({ text: text(node), className: node.className,
|
||||
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor', 'borderTopLeftRadius') });
|
||||
const edge = document.querySelector(edgeSelector);
|
||||
return {
|
||||
tree: [...notice.children].map(name),
|
||||
text: [...notice.querySelectorAll(':scope > div:not(.row-actions) > *')].map(node => [node.localName, text(node)]),
|
||||
actions: [...notice.querySelectorAll(':scope > .row-actions > button')].map(button),
|
||||
frame: css(notice, 'display', 'alignItems', 'justifyContent', 'columnGap', 'borderTopWidth', 'borderTopColor',
|
||||
'borderTopLeftRadius', 'backgroundColor', 'paddingTop', 'paddingLeft', 'marginBottom', 'flexDirection'),
|
||||
h2: css(notice.querySelector('h2'), 'fontSize', 'fontWeight', 'color'),
|
||||
p: css(notice.querySelector('p'), 'fontSize', 'color', 'marginTop'),
|
||||
rowActions: css(notice.querySelector(':scope > .row-actions'), 'display', 'columnGap', 'flexShrink'),
|
||||
alignedWith: Math.round(notice.getBoundingClientRect().left - edge.getBoundingClientRect().left),
|
||||
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
|
||||
};
|
||||
}
|
||||
|
||||
/** D129: the concept's Sources notice as the library notice: its own copy and actions, no bottom margin (main's padding). */
|
||||
function mapLibraryNotice(reference) {
|
||||
return {
|
||||
...reference,
|
||||
text: [['h2', 'Your library is inside the node folder'], ['p', 'Your library is stored inside the node folder, where an '
|
||||
+ 'update or reinstall can delete it. Move it to ComfyUI\'s user folder: it is copied and verified first, and nothing '
|
||||
+ 'is deleted.']],
|
||||
actions: reference.actions.map((action, index) => ({ ...action, text: ['Keep it here', 'Move library'][index] })),
|
||||
frame: { ...reference.frame, marginBottom: '0px' },
|
||||
};
|
||||
}
|
||||
|
||||
async function conceptNotice(page, theme) {
|
||||
await openConcept(page, 'system', theme);
|
||||
await page.locator('#main .section-nav').getByRole('button', { name: 'Sources & scans', exact: true }).click();
|
||||
await expect(page.locator('#main .settings-layout > .notice')).toBeVisible();
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureNoticeOf, ['#main .settings-layout > .notice', '#main .heading']);
|
||||
}
|
||||
|
||||
async function productionNotice(page, theme) {
|
||||
await openProduction(page, 'system/overview', theme);
|
||||
await expect(page.locator('#library-banner > .notice')).toBeVisible();
|
||||
await expect(page.locator('#workspace-main .summary-metrics')).toBeVisible();
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureNoticeOf, ['#library-banner > .notice', '#workspace-main .heading']);
|
||||
}
|
||||
|
||||
for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
test(`library notice matches the concept notice ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
|
||||
await page.setViewportSize({ width, height });
|
||||
await expect(await request.post('/__test__/system/library', { data: { state: 'node_folder' } })).toBeOK();
|
||||
const concept = await conceptNotice(page, theme);
|
||||
const production = await productionNotice(page, theme);
|
||||
await page.screenshot({ path: testInfo.outputPath(`production-library-notice-${theme}-${width}.png`), fullPage: true });
|
||||
expect(diffInventory(mapLibraryNotice(concept), production)).toEqual([]);
|
||||
expect(production.alignedWith).toBe(0);
|
||||
expect(production.pageScroll).toBe(false);
|
||||
});
|
||||
}
|
||||
|
||||
test('library notice comparison detects mutations', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
await expect(await request.post('/__test__/system/library', { data: { state: 'node_folder' } })).toBeOK();
|
||||
const expected = mapLibraryNotice(await conceptNotice(page, 'dark'));
|
||||
await productionNotice(page, 'dark');
|
||||
const paths = async () => diffInventory(expected,
|
||||
await page.evaluate(measureNoticeOf, ['#library-banner > .notice', '#workspace-main .heading'])).map(item => item.path);
|
||||
expect(await paths()).toEqual([]);
|
||||
const undoToken = await mutate(page, { css: '#library-banner { --soft-blue: rgb(255, 0, 0); }' });
|
||||
expect(await paths()).toEqual(['frame.backgroundColor']);
|
||||
await undoToken();
|
||||
await page.evaluate(() => { document.querySelector('#library-banner .notice h2').textContent = 'Library'; });
|
||||
expect(await paths()).toEqual(['text.0.1']);
|
||||
});
|
||||
|
||||
@@ -19,6 +19,13 @@ test.afterEach(async ({ page, browserName }) => {
|
||||
expect(page.workflowErrors).toEqual([]);
|
||||
});
|
||||
|
||||
// #249 S6b: the legacy admin page says where the library lives (the fixture's database is configured explicitly).
|
||||
test('the admin page names the library location', async ({ page }) => {
|
||||
await page.goto('/prompt_manager/admin');
|
||||
await expect(page.locator('#libraryLocation')).toHaveText('Library: browser.db (set in config.json)');
|
||||
await expect(page.locator('#totalPrompts')).not.toHaveText('-');
|
||||
});
|
||||
|
||||
test('actual gallery form saves a prompt that the admin page can edit and reload', async ({ page, request }) => {
|
||||
await page.goto('/prompt_manager/gallery');
|
||||
await page.locator('#addPromptBtn').click();
|
||||
|
||||
@@ -398,18 +398,20 @@ test('preview uses actual Viewer.js and cleans up when closing or leaving Images
|
||||
test('real video media has a usable card and native playback cleans up on close and navigation', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset', { data: { include_video: true } })).toBeOK();
|
||||
const media = await (await request.get('/prompt_manager/images/output?limit=40')).json();
|
||||
expect(media.images.find(image => image.filename === 'clip.webm').thumbnail_url).toBeNull();
|
||||
// #249: a video with no preview yet still lists its preview URL; the server makes the frame on first view.
|
||||
expect(media.images.find(image => image.filename === 'clip.webm').thumbnail_url).toMatch(/\/thumbnails\/clip_thumb\.webp\?root=0/);
|
||||
await page.goto(workspace);
|
||||
await refine(page, { Media: 'video' });
|
||||
await expect(cards(page)).toHaveCount(1);
|
||||
const card = cards(page).first();
|
||||
await expect(card).toHaveAttribute('data-image-key', '0:clip.webm');
|
||||
await expect(card.locator('img')).toHaveCount(0);
|
||||
await expect(card.getByRole('button', { name: 'Open clip.webm', exact: true })).toBeVisible();
|
||||
// Legacy video art inside the concept card; D11 names the video's own extension.
|
||||
await expect(card.locator('.media-art > .image-open > video')).toHaveCount(1);
|
||||
// The card shows the preview frame, never the video itself; D11 names the video's own extension.
|
||||
await expect(card.locator('video')).toHaveCount(0);
|
||||
await expect(card.locator('.media-art > .image-open > img')).toHaveCount(1);
|
||||
await expect(card.locator('.media-info p')).toHaveText(/(?:^|· )WEBM$/);
|
||||
expect(await card.locator('video').evaluate(node => node.getBoundingClientRect().height)).toBeGreaterThan(20);
|
||||
await expect.poll(() => card.locator('img').evaluate(node => node.complete && node.naturalHeight > 0)).toBe(true);
|
||||
expect(await card.locator('img').evaluate(node => node.getBoundingClientRect().height)).toBeGreaterThan(20);
|
||||
await card.getByRole('button', { name: 'Open clip.webm', exact: true }).click();
|
||||
const video = page.locator('dialog[open] video');
|
||||
await expect(video).toBeVisible();
|
||||
@@ -713,16 +715,45 @@ test('selected layout stays blue with readable text on hover in either theme', a
|
||||
}
|
||||
});
|
||||
|
||||
test('refresh releases the previous inspector video source', async ({ page, request }) => {
|
||||
test('refresh releases the previous inspector cover source', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset', { data: { include_video: true } })).toBeOK();
|
||||
await page.goto(workspace);
|
||||
const card = cards(page).filter({ hasText: 'clip.webm' });
|
||||
await card.getByRole('button', { name: 'Inspect clip.webm', exact: true }).click();
|
||||
// #249: a video's cover is its preview frame, not the video.
|
||||
const cover = details(page).locator('img.detail-cover');
|
||||
await expect(cover).toHaveAttribute('src', /\/thumbnails\/clip_thumb\.webp\?/);
|
||||
await expect(details(page).locator('video')).toHaveCount(0);
|
||||
await cover.evaluate(element => { window.__oldInspectorCover = element; });
|
||||
await rescan(page);
|
||||
await expect.poll(() => page.evaluate(() => !window.__oldInspectorCover.isConnected)).toBe(true);
|
||||
expect(await page.evaluate(() => window.__oldInspectorCover.getAttribute('src'))).toBeNull();
|
||||
});
|
||||
|
||||
/** A listing that names no preview for its videos (the fallback a card and the cover keep for one). */
|
||||
async function videosWithoutPreview(page) {
|
||||
await page.route(url => url.pathname === '/prompt_manager/images/output', async route => {
|
||||
const response = await route.fetch();
|
||||
const body = await response.json();
|
||||
if (Array.isArray(body.images)) body.images = body.images.map(image => image.is_video ? { ...image, thumbnail_url: null } : image);
|
||||
await route.fulfill({ response, json: body });
|
||||
});
|
||||
}
|
||||
|
||||
test('a video listed without a preview keeps its own element, released on refresh', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset', { data: { include_video: true } })).toBeOK();
|
||||
await videosWithoutPreview(page);
|
||||
await page.goto(workspace);
|
||||
const card = cards(page).filter({ hasText: 'clip.webm' });
|
||||
await expect(card.locator('.media-art > .image-open > video')).toHaveCount(1);
|
||||
await expect(card.locator('img')).toHaveCount(0);
|
||||
await card.getByRole('button', { name: 'Inspect clip.webm', exact: true }).click();
|
||||
await expect(details(page).locator('video')).toBeVisible();
|
||||
await details(page).locator('video').evaluate(element => { window.__oldInspectorVideo = element; });
|
||||
await card.locator('video').evaluate(element => { window.__oldCardVideo = element; });
|
||||
await rescan(page);
|
||||
await expect.poll(() => page.evaluate(() => !window.__oldInspectorVideo.isConnected)).toBe(true);
|
||||
expect(await page.evaluate(() => window.__oldInspectorVideo.getAttribute('src'))).toBeNull();
|
||||
await expect.poll(() => page.evaluate(() => !window.__oldInspectorVideo.isConnected && !window.__oldCardVideo.isConnected)).toBe(true);
|
||||
expect(await page.evaluate(() => [window.__oldInspectorVideo.getAttribute('src'), window.__oldCardVideo.getAttribute('src')])).toEqual([null, null]);
|
||||
});
|
||||
|
||||
test('Import Rescan rescans cached files and preserves the collection when rescan fails', async ({ page, request }) => {
|
||||
|
||||
@@ -206,10 +206,9 @@ test('Remove marks the draft; removing the last entry states auto-detect', async
|
||||
expect(await rowsOf(page)).toEqual([['p', 'small', 'No source is configured. After saving, ComfyUI’s output folder is used.']]);
|
||||
await saveButton(page).click();
|
||||
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
|
||||
// Saved: ComfyUI's output folder is detected and listed without Remove.
|
||||
// Saved: ComfyUI's output folder is detected and listed without Remove, its row saying it is the default.
|
||||
await expect.poll(rowsOf.bind(null, page)).toEqual([
|
||||
['ComfyUI output', 'output · manual scan', ['Scan folder']],
|
||||
['p', 'small', 'No source is configured; ComfyUI’s output folder is used.'],
|
||||
['ComfyUI output', 'ComfyUI/output · manual scan · default', ['Scan folder']],
|
||||
]);
|
||||
const { sources } = await serverJson(request, SETTINGS_URL);
|
||||
expect([sources.configured, sources.auto_detected]).toEqual([[], true]);
|
||||
@@ -332,27 +331,25 @@ function thumbnailRequests(page) {
|
||||
const missingLine = plan => `${planCount(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ` +
|
||||
`${planCount(plan.fresh)} already up to date.`;
|
||||
|
||||
test('Review options shows the plan counts for the chosen scope/quality/size before Start', async ({ page, request }) => {
|
||||
const SETTINGS_LINE = 'Previews are written as WebP · quality 80 · 512 px. Change this in System › Appearance.';
|
||||
|
||||
test('Review options shows the plan counts for the chosen scope and the server settings before Start', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
|
||||
// Image loading quality (Appearance, Q3) preselects the dialog's quality.
|
||||
await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('gallerySettings', JSON.stringify({ imageQuality: 'high' })); });
|
||||
const seen = thumbnailRequests(page);
|
||||
const plan = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=missing&size=512&quality=high`);
|
||||
const plan = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=missing`);
|
||||
expect(plan.targets).toBe(3);
|
||||
const review = await openThumbnailReview(page);
|
||||
await expect(review.locator('#modal-body > *')).toHaveCount(5);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
|
||||
await expect(review.locator('#thumb-mode option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)']);
|
||||
await expect(review.locator('#thumb-quality option')).toHaveText(['Medium · optimized', 'High · original', 'Low · fastest']);
|
||||
// D131: Scope, the plan line, the server's settings line, the actions (no quality or size field of its own).
|
||||
await expect(review.locator('#modal-body > *')).toHaveCount(4);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope']);
|
||||
await expect(review.locator('#thumb-mode option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)',
|
||||
'Convert outdated (removes older formats)']);
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
|
||||
await expect(review.locator('#thumb-quality')).toHaveValue('high');
|
||||
await expect(review.locator('#thumb-limit')).toHaveValue('512');
|
||||
await expect(review.locator('#thumb-limit')).toHaveAttribute('min', '128');
|
||||
await expect(review.locator('#thumb-limit')).toHaveAttribute('max', '2048');
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
const [line, settings] = [review.locator('#modal-body > p.small').first(), review.locator('#modal-body > p.small').last()];
|
||||
await expect(line).toHaveAttribute('role', 'status');
|
||||
await expect(line).toHaveText(missingLine(plan));
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'high' }]);
|
||||
await expect(settings).toHaveText(SETTINGS_LINE);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing' }]);
|
||||
await expect(review.locator('.dialog-actions > button')).toHaveText(['Cancel', 'Start']);
|
||||
// Reviewing starts nothing; Cancel closes.
|
||||
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
@@ -361,63 +358,67 @@ test('Review options shows the plan counts for the chosen scope/quality/size bef
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
});
|
||||
|
||||
test('size outside 128–2048 reports validity and requests nothing', async ({ page, request }) => {
|
||||
test('Convert outdated replaces older-format previews from their originals and removes the old files (#317)', async ({ page, request }) => {
|
||||
const { files } = await (await request.post('/__test__/system/legacy-previews', { data: { count: 3 } })).json();
|
||||
const before = await serverJson(request, '/prompt_manager/images/thumbnails/status');
|
||||
expect(before.outdated).toBe(3);
|
||||
const seen = thumbnailRequests(page);
|
||||
const review = await openThumbnailReview(page);
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
const line = review.locator('#modal-body > p.small').first();
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
const size = review.locator('#thumb-limit');
|
||||
for (const value of ['4096', '64', '']) {
|
||||
await size.fill(value);
|
||||
await expect(size).toHaveAttribute('aria-invalid', 'true');
|
||||
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
|
||||
expect(await size.evaluate(input => input.validity.valid)).toBe(false);
|
||||
// Start keeps the dialog open and sends nothing; neither does a scope change.
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
|
||||
await review.locator('#thumb-mode').selectOption('missing');
|
||||
}
|
||||
await review.locator('#thumb-mode').selectOption('convert');
|
||||
await expect(line).toHaveText('3 previews in an older format to convert.');
|
||||
// Choosing the scope converts nothing: only Start does.
|
||||
expect(seen.starts).toEqual([]);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }]);
|
||||
await size.fill('1024');
|
||||
await expect(size).not.toHaveAttribute('aria-invalid');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'convert' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
|
||||
expect(job).toMatchObject({ kind: 'thumbnails', title: 'Convert outdated thumbnails', scope: { mode: 'convert', format: 'webp' } });
|
||||
expect(job.status_line).toMatch(/^3 converted · 0 failed · .+ → .+$/);
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText('Convert outdated thumbnails');
|
||||
const after = await serverJson(request, '/prompt_manager/images/thumbnails/status');
|
||||
expect(after.outdated).toBe(0);
|
||||
// The listing now names the WebP previews; the old PNG previews are gone.
|
||||
for (const file of files) {
|
||||
const name = file.replace(/\.png$/, '');
|
||||
expect((await request.get(`/prompt_manager/images/serve/thumbnails/${name}_thumb.webp?root=0`)).headers()['content-type']).toBe('image/webp');
|
||||
expect((await request.get(`/prompt_manager/images/serve/${file}?root=0`)).ok()).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('Start closes the dialog, toasts, and opens the job in Activity', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
|
||||
const seen = thumbnailRequests(page);
|
||||
const review = await openThumbnailReview(page);
|
||||
await review.locator('#thumb-quality').selectOption('low');
|
||||
await review.locator('#thumb-limit').fill('256');
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
// The toast says it started (or, for a run this quick, that it already finished); Activity shows its progress.
|
||||
await expect(toast(page)).toHaveText(/^(Thumbnail generation started\.|Thumbnails finished · 2 generated · 0 errors)$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'missing', size: 256, quality: 'low' }]);
|
||||
expect(seen.starts).toEqual([{ mode: 'missing' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
|
||||
expect(job).toMatchObject({ kind: 'thumbnails', title: 'Thumbnail generation', scope: { mode: 'missing', size: 256, quality: 'low' } });
|
||||
expect(job).toMatchObject({ kind: 'thumbnails', title: 'Thumbnail generation',
|
||||
scope: { mode: 'missing', format: 'webp', size: 512, quality: 80 } });
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText('Thumbnail generation');
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('Rebuild all is explicit and the plan line says every preview is regenerated', async ({ page, request }) => {
|
||||
const seen = thumbnailRequests(page);
|
||||
const rebuild = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=rebuild&size=512&quality=medium`);
|
||||
const rebuild = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=rebuild`);
|
||||
expect(rebuild.targets).toBeGreaterThan(0);
|
||||
const review = await openThumbnailReview(page);
|
||||
// Missing thumbnails only is the default; rebuilding is the explicit second option.
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
const line = review.locator('#modal-body > p.small').first();
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }, { mode: 'rebuild', size: '512', quality: 'medium' }]);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing' }, { mode: 'rebuild' }]);
|
||||
// Back and forth reads each scope once.
|
||||
await review.locator('#thumb-mode').selectOption('missing');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
@@ -426,7 +427,7 @@ test('Rebuild all is explicit and the plan line says every preview is regenerate
|
||||
expect(seen.plans).toHaveLength(2);
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'rebuild', size: 512, quality: 'medium' }]);
|
||||
expect(seen.starts).toEqual([{ mode: 'rebuild' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
@@ -434,7 +435,7 @@ test('Rebuild all is explicit and the plan line says every preview is regenerate
|
||||
test('a plan that cannot be read says so; Start still starts', async ({ page }) => {
|
||||
await page.route(`**${THUMBNAIL_PLAN_URL}?*`, route => route.fulfill({ status: 500, json: { success: false, error: 'disk' } }));
|
||||
const review = await openThumbnailReview(page);
|
||||
await expect(review.locator('#modal-body > p.small')).toHaveText('The previews could not be counted.');
|
||||
await expect(review.locator('#modal-body > p.small').first()).toHaveText('The previews could not be counted.');
|
||||
await expect(review.getByRole('button', { name: 'Start', exact: true })).not.toHaveAttribute('aria-disabled');
|
||||
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
});
|
||||
|
||||
@@ -68,6 +68,8 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
|
||||
expect(await rowsOf(layout, 'Library health')).toEqual([
|
||||
['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'],
|
||||
['Backup status', 'No retained backup record.', 'View backups'],
|
||||
// D129: the fixture's database is configured explicitly, so it reads as config.json's choice and does not move.
|
||||
['Library location', 'browser.db · The database path set in config.json.', 'Details'],
|
||||
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
|
||||
]);
|
||||
expect(await rowsOf(layout, 'Workspace preferences')).toEqual([
|
||||
@@ -78,6 +80,95 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
|
||||
expect(writes).toEqual([]);
|
||||
});
|
||||
|
||||
// D129 (#249 S6b): where the library lives. The hook makes the fixture's database read as a node-folder library; its
|
||||
// Move library is a stand-in that touches no file (workspace_system_fixture.py).
|
||||
const banner = page => page.locator('#library-banner');
|
||||
const LIBRARY_URL = '/prompt_manager/system/library';
|
||||
|
||||
test('a node-folder library is offered a move above every page; Keep it here stops the offer', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/library', { data: { state: 'node_folder' } })).toBeOK();
|
||||
await page.goto(`${workspace}#/images`);
|
||||
await expect(banner(page).locator('.notice h2')).toHaveText('Your library is inside the node folder');
|
||||
await expect(banner(page).locator('.notice p')).toHaveText(/^Your library is stored inside the node folder, where an update or reinstall can delete it\./);
|
||||
await expect(banner(page).locator('.notice .row-actions button')).toHaveText(['Keep it here', 'Move library']);
|
||||
await expect(banner(page).getByRole('button', { name: 'Move library', exact: true })).toHaveClass(/ca-primary/);
|
||||
await page.goto(`${workspace}#/library`);
|
||||
await expect(banner(page).locator('.notice')).toBeVisible();
|
||||
await banner(page).getByRole('button', { name: 'Keep it here', exact: true }).click();
|
||||
await expect(banner(page)).toBeHidden();
|
||||
const { location } = await serverJson(request, LIBRARY_URL);
|
||||
expect([location.keep, location.offer, location.movable]).toEqual([true, false, true]);
|
||||
await page.reload();
|
||||
const layout = await openOverview(page);
|
||||
await expect(banner(page)).toBeHidden();
|
||||
const row = layout.locator('.operation').filter({ hasText: 'Library location' });
|
||||
await expect(row.locator('p')).toHaveText('browser.db · The node folder, where an update or reinstall can delete it. '
|
||||
+ 'You chose to keep it here.');
|
||||
// The row stays: the move is still one press away.
|
||||
await expect(row.locator('.row-actions button')).toHaveText('Move library');
|
||||
});
|
||||
|
||||
test('Move library is reviewed, runs as an Activity job and ends the offer', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/library', { data: { state: 'node_folder' } })).toBeOK();
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ hasText: 'Library location' });
|
||||
await expect(row.locator('p')).toHaveText('browser.db · The node folder, where an update or reinstall can delete it.');
|
||||
const move = banner(page).getByRole('button', { name: 'Move library', exact: true });
|
||||
await move.click();
|
||||
const dialog = page.locator('dialog#modal[open]');
|
||||
await expect(dialog.locator('#modal-title')).toHaveText('Move library');
|
||||
expect(await dialog.locator('dl.meta-grid > div').evaluateAll(items => items.map(item =>
|
||||
[item.querySelector('dt').textContent, item.querySelector('dd').textContent]))).toEqual([
|
||||
['From', 'browser.db'], ['To', 'user/__promptmanager/prompts.db'], ['Size', expect.stringMatching(/ (KB|MB)$/)],
|
||||
['Moves with it', 'config.json and the database backups'], ['Old files', 'Renamed *.moved-to-…, never deleted'],
|
||||
]);
|
||||
await expect(dialog.locator('.dialog-actions button')).toHaveText(['Cancel', 'Move library']);
|
||||
await dialog.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
await expect(page.locator('dialog#modal[open]')).toHaveCount(0);
|
||||
await expect(move).toBeFocused();
|
||||
await row.getByRole('button', { name: 'Move library', exact: true }).click();
|
||||
await dialog.getByRole('button', { name: 'Move library', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[^/]+$/);
|
||||
await expect(page.locator('#workspace-main')).toContainText(
|
||||
/Moved the library to user\/__promptmanager: \d+ prompts? and 0 images verified\. The old files were renamed browser\.db\.moved-to-__promptmanager; nothing was deleted\./);
|
||||
// The offer ends with the job; the Overview says where the library lives now.
|
||||
await expect(banner(page)).toBeHidden();
|
||||
const after = await openOverview(page);
|
||||
await expect(after.locator('.operation').filter({ hasText: 'Library location' }).locator('p'))
|
||||
.toHaveText('browser.db · ComfyUI\'s user folder, which updates and reinstalls of the node leave alone.');
|
||||
});
|
||||
|
||||
test('a move that fails says so and keeps the offer; a refused start says why', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/library', { data: { state: 'failing' } })).toBeOK();
|
||||
await page.goto(`${workspace}#/images`);
|
||||
await banner(page).getByRole('button', { name: 'Move library', exact: true }).click();
|
||||
await page.locator('dialog#modal[open]').getByRole('button', { name: 'Move library', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[^/]+$/);
|
||||
await expect(page.locator('#workspace-main')).toContainText(
|
||||
'The copy did not pass verification. Nothing was moved. The library is still in the node folder, unchanged.');
|
||||
await expect(banner(page).locator('.notice')).toBeVisible();
|
||||
await expect(await request.post('/__test__/system/fail', { data: { path: `${LIBRARY_URL}/move` } })).toBeOK();
|
||||
await banner(page).getByRole('button', { name: 'Move library', exact: true }).click();
|
||||
await page.locator('dialog#modal[open]').getByRole('button', { name: 'Move library', exact: true }).click();
|
||||
await expect(page.locator('#workspace-toast')).toHaveText('Move library could not start. Try again.');
|
||||
});
|
||||
|
||||
test('a library that does not move shows its details', async ({ page }) => {
|
||||
const layout = await openOverview(page);
|
||||
await expect(banner(page)).toBeHidden();
|
||||
const details = layout.locator('.operation').filter({ hasText: 'Library location' }).getByRole('button', { name: 'Details', exact: true });
|
||||
await details.click();
|
||||
const dialog = page.locator('dialog#modal[open]');
|
||||
await expect(dialog.locator('#modal-title')).toHaveText('Library location');
|
||||
expect(await dialog.locator('dl.meta-grid > div').evaluateAll(items => items.map(item =>
|
||||
[item.querySelector('dt').textContent, item.querySelector('dd').textContent.replace(/[\d.,]+ (KB|MB)$/, 'N')]))).toEqual([
|
||||
['Database', 'browser.db'], ['Why here', 'The database path set in config.json'],
|
||||
['Beside it', 'config.json, the database backups and the logs'], ['Size', 'N'],
|
||||
]);
|
||||
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
|
||||
await expect(details).toBeFocused();
|
||||
});
|
||||
|
||||
test('Run health checks opens Health report and updates metric 4', async ({ page, request }) => {
|
||||
await page.goto(`${workspace}#/system/maintenance`);
|
||||
const layout = layoutOf(page);
|
||||
@@ -144,7 +235,7 @@ test('thumbnail row reflects the status endpoint', async ({ page, request }) =>
|
||||
|
||||
test('Overview Review thumbnails opens the same dialog', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
|
||||
const plan = await serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
|
||||
const plan = await serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing');
|
||||
const starts = [];
|
||||
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/thumbnails')) starts.push(item.postDataJSON()); });
|
||||
const layout = await openOverview(page);
|
||||
@@ -154,8 +245,8 @@ test('Overview Review thumbnails opens the same dialog', async ({ page, request
|
||||
const review = page.locator('dialog#modal[open]');
|
||||
await expect(review.locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await expect(page).toHaveURL(`${workspace}#/system/overview`);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
|
||||
await expect(review.locator('#modal-body > p.small')).toHaveText(`2 previews to generate · ${number(plan.fresh)} already up to date.`);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope']);
|
||||
await expect(review.locator('#modal-body > p.small').first()).toHaveText(`2 previews to generate · ${number(plan.fresh)} already up to date.`);
|
||||
// Escape closes it and focus returns to the row's button.
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(review).toHaveCount(0);
|
||||
@@ -163,11 +254,61 @@ test('Overview Review thumbnails opens the same dialog', async ({ page, request
|
||||
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(starts).toEqual([{ mode: 'missing', size: 512, quality: 'medium' }]);
|
||||
expect(starts).toEqual([{ mode: 'missing' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('the thumbnail row counts outdated previews and offers Convert outdated (#317)', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/legacy-previews', { data: { count: 2 } })).toBeOK();
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
|
||||
await expect(row.locator('h3')).toHaveText('No missing thumbnails · 2 outdated');
|
||||
await expect(row.locator('.row-actions > button')).toHaveText(['Review thumbnails', 'Convert outdated']);
|
||||
await row.getByRole('button', { name: 'Convert outdated', exact: true }).click();
|
||||
const review = page.locator('dialog#modal[open]');
|
||||
await expect(review.locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
// It opens on the Convert outdated scope; nothing starts until Start.
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('convert');
|
||||
await expect(review.locator('#modal-body > p.small').first()).toHaveText('2 previews in an older format to convert.');
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(row.getByRole('button', { name: 'Convert outdated', exact: true })).toBeFocused();
|
||||
});
|
||||
|
||||
test('Appearance › Thumbnails saves the server thumbnail settings and never converts by itself (#317)', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/jxl-encoder', { data: { available: false } })).toBeOK();
|
||||
await expect(await request.post('/__test__/system/legacy-previews', { data: { count: 1 } })).toBeOK();
|
||||
await page.goto(`${workspace}#/system/appearance`);
|
||||
const surface = page.locator('#workspace-main .settings-layout > section.surface').nth(1);
|
||||
await expect(surface.locator(':scope > h2')).toHaveText('Thumbnails');
|
||||
await expect(surface.locator('.setting-row label')).toHaveText(['Thumbnail format', 'Thumbnail quality', 'Thumbnail size limit']);
|
||||
const format = surface.locator('#thumbnail-format');
|
||||
await expect(format).toBeEnabled();
|
||||
await expect(format).toHaveValue('webp');
|
||||
await expect(surface.locator('#thumbnail-quality')).toHaveValue('80');
|
||||
await expect(surface.locator('#thumbnail-edge')).toHaveValue('512');
|
||||
// D130: JPEG XL is offered only with an encoder; without one the row says why, in the server's words.
|
||||
await expect(format.locator('option[value="jxl"]')).toBeDisabled();
|
||||
await expect(surface.locator('.setting-row').first().locator('p.small').first()).toContainText('No JPEG XL encoder found.');
|
||||
await format.selectOption('jpeg');
|
||||
await surface.locator('#thumbnail-quality').fill('65');
|
||||
await surface.locator('#thumbnail-quality').dispatchEvent('change');
|
||||
await page.getByRole('button', { name: 'Save preferences', exact: true }).click();
|
||||
await expect(page.locator('#settings-status')).toHaveText('Saved to the server configuration.');
|
||||
const { settings } = await serverJson(request, '/prompt_manager/settings');
|
||||
expect([settings.thumbnail_format, settings.thumbnail_quality, settings.thumbnail_edge]).toEqual(['jpeg', 65, 512]);
|
||||
// Changing the format converts nothing: the row says so and the old preview is still listed.
|
||||
await expect(surface.locator('.operation p')).toHaveText('Format saved. Existing previews keep their old format until you convert them.');
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
expect((await serverJson(request, '/prompt_manager/images/thumbnails/status')).outdated).toBeGreaterThan(0);
|
||||
// With an encoder JPEG XL can be chosen.
|
||||
await expect(await request.post('/__test__/system/jxl-encoder', { data: { available: true } })).toBeOK();
|
||||
await page.reload();
|
||||
await expect(page.locator('#workspace-main #thumbnail-format option[value="jxl"]')).toBeEnabled();
|
||||
await expect(page.locator('#workspace-main #thumbnail-format')).toHaveValue('jpeg');
|
||||
});
|
||||
|
||||
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
|
||||
const held = async () => (await serverJson(request, '/__test__/hold')).held[THUMBNAILS_URL] ?? 0;
|
||||
await expect(await request.post('/__test__/hold', { data: { path: THUMBNAILS_URL } })).toBeOK();
|
||||
@@ -292,7 +433,7 @@ const panel = page => page.getByRole('complementary', { name: 'Selected image de
|
||||
|
||||
async function openAppearance(page) {
|
||||
await page.goto(`${workspace}#/system/appearance`);
|
||||
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
|
||||
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2').first()).toHaveText('Browsing & display');
|
||||
// The host rows are enabled once the server settings answered.
|
||||
await expect(page.locator('#timeout')).toBeEnabled();
|
||||
}
|
||||
@@ -509,14 +650,16 @@ test('server settings that cannot be read say so in their rows', async ({ page }
|
||||
? route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, error: 'Injected failure' }) })
|
||||
: route.continue()));
|
||||
await page.goto(`${workspace}#/system/appearance`);
|
||||
for (const id of ['open-mode', 'timeout']) {
|
||||
// #317: the Thumbnails rows are server settings too.
|
||||
for (const id of ['open-mode', 'timeout', 'thumbnail-format', 'thumbnail-quality', 'thumbnail-edge']) {
|
||||
const row = page.locator('#workspace-main .setting-row').filter({ has: page.locator(`#${id}`) });
|
||||
await expect(row.locator('p.small')).toHaveText('Server settings could not be read.');
|
||||
await expect(row.locator('p.small').first()).toHaveText('Server settings could not be read.');
|
||||
await expect(page.locator(`#${id}`)).toBeDisabled();
|
||||
}
|
||||
// The browser rows are this browser's, and work as before.
|
||||
await expect(page.locator('#default-view')).toBeEnabled();
|
||||
await expect(page.locator('#workspace-main .setting-row p.small')).toHaveCount(2);
|
||||
await expect(page.locator('#workspace-main .setting-row p.small', { hasText: 'Server settings could not be read.' })).toHaveCount(5);
|
||||
await expect(page.locator('#workspace-main section.surface').first().locator('.setting-row p.small')).toHaveCount(2);
|
||||
});
|
||||
|
||||
test('stale revision shows the conflict status', async ({ page, request }) => {
|
||||
@@ -578,7 +721,7 @@ test('a change another tab saved while System was closed is a conflict on return
|
||||
await other.close();
|
||||
// Back in System: the draft is kept and flagged, and nothing was written over the other tab's value.
|
||||
await page.evaluate(() => { location.hash = '#/system/appearance'; });
|
||||
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
|
||||
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2').first()).toHaveText('Browsing & display');
|
||||
await expect(statusOf(page)).toHaveText('Settings changed elsewhere. Reload this section to see them.');
|
||||
await expect(page.locator('#density')).toHaveAttribute('aria-invalid', 'true');
|
||||
await expect(page.locator('#density')).toHaveValue('compact');
|
||||
|
||||
@@ -602,6 +602,10 @@ test('a save under a Tags filter re-reads the listing: the selection stays while
|
||||
|
||||
test('a save under a search keeps the scroll position and the loaded range', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/tagging/reset', { data: { extra_images: 140 } })).toBeOK();
|
||||
// Listing pages still in flight: one that lands after the scroll below would grow the loaded range.
|
||||
const inFlight = new Set();
|
||||
page.on('request', item => { if (item.url().includes('/images/output?')) inFlight.add(item); });
|
||||
for (const done of ['requestfinished', 'requestfailed']) page.on(done, item => inFlight.delete(item));
|
||||
await page.goto(images);
|
||||
const searched = listingOf(page, query => query.get('q') === 'bulk');
|
||||
await page.getByRole('searchbox', { name: 'Search workspace', exact: true }).fill('bulk');
|
||||
@@ -615,6 +619,9 @@ test('a save under a search keeps the scroll position and the loaded range', asy
|
||||
await expect.poll(async () => { await continuation.scrollIntoViewIfNeeded(); return loadedCount(); }).toBeGreaterThanOrEqual(80);
|
||||
await page.evaluate(() => window.scrollTo(0, Math.round(document.documentElement.scrollHeight / 3)));
|
||||
await expect(continuation).not.toBeInViewport();
|
||||
// A load the continuation scheduled for the next frame has started by now; wait for every page to land.
|
||||
await page.evaluate(() => new Promise(done => requestAnimationFrame(() => requestAnimationFrame(done))));
|
||||
await expect.poll(() => inFlight.size).toBe(0);
|
||||
const loaded = await continuation.textContent();
|
||||
await expect.poll(() => continuation.textContent()).toBe(loaded);
|
||||
const y = await page.evaluate(() => window.scrollY);
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
|
||||
import { test, expect } from './fixtures.mjs';
|
||||
|
||||
/**
|
||||
* #249: cards and strips load previews, never the full file. The approved mock has no repeated thumbnail prompt
|
||||
* (docs/concepts/workspace/README.md), so a preview that was not made yet is made by the server on first view:
|
||||
* the listing still names the `_thumb` URL. Only the media viewer's main image (and downloads) load the original.
|
||||
*/
|
||||
const images = '/prompt_manager/workspace#/images';
|
||||
const library = '/prompt_manager/workspace#/library';
|
||||
const cards = page => page.locator('article[data-image-key]');
|
||||
const details = page => page.getByRole('complementary', { name: 'Selected image details', exact: true });
|
||||
const SERVE = '/prompt_manager/images/serve/';
|
||||
|
||||
/** Every media file the page asks the server for, as `{path, preview}` (path relative to its root). */
|
||||
function recordMedia(page) {
|
||||
const seen = [];
|
||||
page.on('request', request => {
|
||||
const url = new URL(request.url());
|
||||
if (!url.pathname.startsWith(SERVE)) return;
|
||||
const path = decodeURIComponent(url.pathname.slice(SERVE.length));
|
||||
seen.push({ path, preview: /^thumbnails\/.+_thumb\.[a-z0-9]+$/.test(path) });
|
||||
});
|
||||
return seen;
|
||||
}
|
||||
/** Every card image has loaded (or failed): what it asked for is all in `seen`. */
|
||||
const settled = page => page.evaluate(() => Promise.all([...document.querySelectorAll('article img')].map(image => image.decode().catch(() => {}))));
|
||||
const originals = seen => seen.filter(item => !item.preview).map(item => item.path);
|
||||
|
||||
test.beforeEach(async ({ page, browserName }) => {
|
||||
await startCoverage(page, browserName);
|
||||
page.thumbErrors = [];
|
||||
page.on('pageerror', error => page.thumbErrors.push(error.message));
|
||||
});
|
||||
test.afterEach(async ({ page, browserName }) => {
|
||||
await saveCoverage(page, browserName);
|
||||
expect(page.thumbErrors).toEqual([]);
|
||||
});
|
||||
|
||||
test('a page of the Images grid requests previews, never originals, in every layout and size', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset')).toBeOK();
|
||||
// Newest outputs with no preview yet: the live finding (844 such files listed with a null thumbnail).
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 6 } })).toBeOK();
|
||||
const listing = await (await request.get('/prompt_manager/images/output?limit=40')).json();
|
||||
const fresh = listing.images.filter(image => image.path.startsWith('unthumbnailed/'));
|
||||
expect(fresh).toHaveLength(6);
|
||||
for (const image of fresh) expect(image.thumbnail_url).toMatch(/\/thumbnails\/unthumbnailed\/plain_\d{3}_thumb\.webp\?root=0/);
|
||||
const seen = recordMedia(page);
|
||||
await page.goto(images);
|
||||
await expect(cards(page)).toHaveCount(8);
|
||||
await settled(page);
|
||||
expect(originals(seen)).toEqual([]);
|
||||
expect(seen.filter(item => item.path.startsWith('thumbnails/unthumbnailed/'))).toHaveLength(6);
|
||||
// The server made them on that first view.
|
||||
await expect.poll(async () => (await (await request.get('/prompt_manager/images/thumbnails/status')).json()).pending).toBe(0);
|
||||
for (const layout of ['List', 'Masonry', 'Grid']) {
|
||||
await page.getByRole('button', { name: layout, exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: layout, exact: true })).toHaveAttribute('aria-pressed', 'true');
|
||||
for (const density of ['large', 'compact', 'comfortable']) {
|
||||
await page.getByRole('combobox', { name: 'Thumbnail size', exact: true }).selectOption(density);
|
||||
await expect(cards(page)).toHaveCount(8);
|
||||
await settled(page);
|
||||
}
|
||||
}
|
||||
expect(originals(seen)).toEqual([]);
|
||||
});
|
||||
|
||||
test('the inspector cover is the preview and only the media viewer loads the original', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset')).toBeOK();
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK();
|
||||
const seen = recordMedia(page);
|
||||
await page.goto(images);
|
||||
const card = cards(page).filter({ hasText: 'plain_001.png' });
|
||||
await card.getByRole('button', { name: 'Inspect plain_001.png', exact: true }).click();
|
||||
const cover = details(page).locator('img.detail-cover');
|
||||
await expect(cover).toHaveAttribute('src', /\/thumbnails\/unthumbnailed\/plain_001_thumb\.webp\?/);
|
||||
await settled(page);
|
||||
await cover.evaluate(image => image.decode().catch(() => {}));
|
||||
expect(originals(seen)).toEqual([]);
|
||||
await card.getByRole('button', { name: 'Open plain_001.png', exact: true }).click();
|
||||
await expect(page.locator('dialog#viewer-dialog')).toBeVisible();
|
||||
await expect.poll(() => originals(seen)).toEqual(['unthumbnailed/plain_001.png']);
|
||||
await page.getByRole('button', { name: 'Close image viewer', exact: true }).click();
|
||||
});
|
||||
|
||||
test('a video card shows its preview frame instead of loading the video', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset', { data: { include_video: true } })).toBeOK();
|
||||
const seen = recordMedia(page);
|
||||
await page.goto(images);
|
||||
const card = cards(page).filter({ hasText: 'clip.webm' });
|
||||
await expect(card.locator('.media-art > .image-open > img')).toHaveAttribute('src', /\/thumbnails\/clip_thumb\.webp\?/);
|
||||
await expect(card.locator('video')).toHaveCount(0);
|
||||
await settled(page);
|
||||
expect(originals(seen)).toEqual([]);
|
||||
});
|
||||
|
||||
test('Library cards and the shared gallery request previews only', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset', { data: { native_library: true } })).toBeOK();
|
||||
const seen = recordMedia(page);
|
||||
await page.goto(library);
|
||||
const cat = page.locator('article[data-prompt-id]').filter({ has: page.getByText('Cat portrait', { exact: true }) });
|
||||
await expect(cat.locator('.card-art img')).not.toHaveCount(0);
|
||||
await settled(page);
|
||||
expect(seen.length).toBeGreaterThan(0);
|
||||
expect(originals(seen)).toEqual([]);
|
||||
});
|
||||
|
||||
test('a video whose preview cannot be loaded swaps back to its own element', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset', { data: { include_video: true } })).toBeOK();
|
||||
await page.route(url => url.pathname.endsWith('/thumbnails/clip_thumb.webp'), route => route.fulfill({ status: 404, body: '' }));
|
||||
await page.goto(images);
|
||||
const card = cards(page).filter({ hasText: 'clip.webm' });
|
||||
await expect(card.locator('.media-art > .image-open > video')).toHaveCount(1);
|
||||
await expect(card.locator('.media-art > .image-open > img')).toHaveCount(0);
|
||||
await card.getByRole('button', { name: 'Inspect clip.webm', exact: true }).click();
|
||||
await expect(details(page).locator('video.detail-cover')).toHaveCount(1);
|
||||
await expect(details(page).locator('img.detail-cover')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('an image listed without a preview (a nested thumbnails folder) shows its original', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset')).toBeOK();
|
||||
// M10: such files are listed with a null thumbnail_url; the card falls back to the original as before.
|
||||
await page.route(url => url.pathname === '/prompt_manager/images/output', async route => {
|
||||
const response = await route.fetch();
|
||||
const body = await response.json();
|
||||
if (Array.isArray(body.images)) body.images = body.images.map(image => ({ ...image, thumbnail_url: null }));
|
||||
await route.fulfill({ response, json: body });
|
||||
});
|
||||
const seen = recordMedia(page);
|
||||
await page.goto(images);
|
||||
await expect(cards(page)).toHaveCount(2);
|
||||
await settled(page);
|
||||
expect(seen.length).toBeGreaterThan(0);
|
||||
expect(seen.every(item => !item.preview)).toBe(true);
|
||||
});
|
||||
|
||||
test('a JPEG XL thumbnail this browser cannot show loads the original; one it can show is used (#317)', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/reset')).toBeOK();
|
||||
await expect(await request.post('/__test__/system/reset')).toBeOK();
|
||||
try {
|
||||
await expect(await request.post('/__test__/system/jxl-encoder', { data: { available: true } })).toBeOK();
|
||||
const saved = await request.post('/prompt_manager/settings', { data: { thumbnail_format: 'jxl' } });
|
||||
await expect(saved).toBeOK();
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK();
|
||||
const listing = await (await request.get('/prompt_manager/images/output?limit=40')).json();
|
||||
const fresh = listing.images.find(image => image.path === 'unthumbnailed/plain_001.png');
|
||||
expect(fresh.thumbnail_url).toMatch(/\/thumbnails\/unthumbnailed\/plain_001_thumb\.jxl\?root=0/);
|
||||
await page.goto(images);
|
||||
// What this browser does with JPEG XL, decided the way the workspace decides it (an embedded probe).
|
||||
const decodes = await page.evaluate(() => new Promise(resolve => {
|
||||
const probe = new Image();
|
||||
probe.onload = () => resolve(true);
|
||||
probe.onerror = () => resolve(false);
|
||||
probe.src = 'data:image/jxl;base64,/woAkAEAE4gCALQAtZ8gAAAVKqOMG7yc6/nyQ4fFtI3rDG21bWEJY7O9MEhIOGwY4s3xwRCTShIA';
|
||||
}));
|
||||
const card = cards(page).filter({ hasText: 'plain_001.png' });
|
||||
const media = card.locator('.media-art > .image-open > img');
|
||||
// A browser without JPEG XL shows the original; one with it shows the preview, or the original when the
|
||||
// preview cannot be made here (no real encoder in the test server): never a broken image.
|
||||
if (!decodes) await expect(media).toHaveAttribute('src', /\/unthumbnailed\/plain_001\.png\?root=0/);
|
||||
await expect.poll(() => media.evaluate(image => image.complete && image.naturalWidth > 0)).toBe(true);
|
||||
// The inspector cover follows the same rule.
|
||||
await card.getByRole('button', { name: 'Inspect plain_001.png', exact: true }).click();
|
||||
const cover = details(page).locator('img.detail-cover');
|
||||
if (!decodes) await expect(cover).toHaveAttribute('src', /\/unthumbnailed\/plain_001\.png\?/);
|
||||
await expect.poll(() => cover.evaluate(image => image.complete && image.naturalWidth > 0)).toBe(true);
|
||||
} finally {
|
||||
await request.post('/__test__/system/reset');
|
||||
}
|
||||
});
|
||||
@@ -173,7 +173,8 @@ test('visual fixture exposes real stable media and library records', async ({ re
|
||||
await expect(await request.get(image.url)).toBeOK();
|
||||
const thumbnail = await request.get(image.thumbnail_url);
|
||||
await expect(thumbnail).toBeOK();
|
||||
expect(thumbnail.headers()['content-type']).toBe('image/png');
|
||||
// #317: previews are written as WebP by default.
|
||||
expect(thumbnail.headers()['content-type']).toBe('image/webp');
|
||||
const inspection = await (await request.get(`/prompt_manager/images/inspect/${encodeURIComponent(image.path)}?root=${image.root_index}`)).json();
|
||||
expect(inspection.success).toBe(true);
|
||||
expect(inspection.image.path).toBe(image.path);
|
||||
|
||||
@@ -469,7 +469,7 @@ test('System is native: the concept shell, section navigation, sections, hand-of
|
||||
if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible();
|
||||
if (hash === 'sources') await expect(layout.locator('#source-list .operation')).toHaveCount(2);
|
||||
expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
|
||||
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'div.save-row'],
|
||||
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'section.surface', 'div.save-row'],
|
||||
sources: ['div.notice', 'section.surface', 'div.save-row'],
|
||||
maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash]
|
||||
?? ['section.surface handoff', 'div.save-row']);
|
||||
|
||||
@@ -513,7 +513,12 @@ def seed_workspace_concept(
|
||||
tags = [row[0] for row in conn.execute("SELECT name FROM tags ORDER BY id")]
|
||||
if tuple(counts) != (len(CONCEPT_PROMPTS), CONCEPT_RUN_COUNT, CONCEPT_RUN_COUNT):
|
||||
raise RuntimeError(f"Concept fixture run counts are not real: {tuple(counts)}")
|
||||
api._generate_gallery_thumbnails_sync(api._get_all_output_dirs(), (150, 150))
|
||||
from py.api.thumbnail_settings import ThumbnailSettings
|
||||
|
||||
# 150 px previews in the default format (#317 thumbnail settings).
|
||||
api._generate_gallery_thumbnails_sync(
|
||||
api._get_all_output_dirs(), ThumbnailSettings("webp", 80, 150)
|
||||
)
|
||||
api.invalidate_gallery_cache()
|
||||
return {
|
||||
"epoch": epoch,
|
||||
|
||||
@@ -23,6 +23,19 @@ read. These per-server hooks only change what a System test needs on top:
|
||||
image monitor whose observer is ``alive`` (default true) and schedules the
|
||||
named folders of this server's root (``["output"]``), so Sources reads
|
||||
watching, manual scan and restart pending states (S4).
|
||||
- ``POST /__test__/system/library`` ``{state}`` (S6b): ``node_folder`` makes
|
||||
the live database read as a node-folder library that can move to
|
||||
``user/__promptmanager``; Move library then runs a stand-in move that
|
||||
touches no file and ends with the library in the default folder.
|
||||
``failing`` is the same library whose move fails verification. Reset
|
||||
restores the real resolver, the real move and "Keep it here".
|
||||
- ``POST /__test__/system/legacy-previews`` ``{count}`` (#317) writes ``count``
|
||||
PNGs into ``output/legacy`` with an older release's preview beside each
|
||||
(``thumbnails/legacy/<name>_thumb.png``, newer than its file): outdated
|
||||
previews for Convert outdated;
|
||||
- ``POST /__test__/system/jxl-encoder`` ``{available}`` (#317) makes the
|
||||
server read as having (``cjxl``) or lacking a JPEG XL encoder without
|
||||
probing; reset restores the thumbnail settings and the real detection.
|
||||
|
||||
Delays use the shared request holds (``request_hold.py``).
|
||||
"""
|
||||
@@ -54,14 +67,24 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
from py.config import GalleryConfig, PromptManagerConfig
|
||||
from utils import image_monitor
|
||||
|
||||
from database import library_move
|
||||
from utils import library_location
|
||||
|
||||
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
|
||||
real_move = library_move.move_library
|
||||
keep = PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER
|
||||
config_bytes = config_path.read_bytes() if config_path.exists() else None
|
||||
host_values = (
|
||||
PromptManagerConfig.RESULT_TIMEOUT,
|
||||
PromptManagerConfig.WEBUI_DISPLAY_MODE,
|
||||
PromptManagerConfig.WORKER_THREADS,
|
||||
GalleryConfig.MONITORING_ENABLED,
|
||||
PromptManagerConfig.THUMBNAIL_FORMAT,
|
||||
PromptManagerConfig.THUMBNAIL_QUALITY,
|
||||
PromptManagerConfig.THUMBNAIL_EDGE,
|
||||
)
|
||||
from py.api import thumbnail_encode
|
||||
|
||||
monitor = getattr(image_monitor, "_monitor_instance", None)
|
||||
failures: dict[str, int] = {}
|
||||
|
||||
@@ -90,8 +113,15 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
PromptManagerConfig.WEBUI_DISPLAY_MODE,
|
||||
PromptManagerConfig.WORKER_THREADS,
|
||||
GalleryConfig.MONITORING_ENABLED,
|
||||
PromptManagerConfig.THUMBNAIL_FORMAT,
|
||||
PromptManagerConfig.THUMBNAIL_QUALITY,
|
||||
PromptManagerConfig.THUMBNAIL_EDGE,
|
||||
) = host_values
|
||||
thumbnail_encode.reset_encoders()
|
||||
image_monitor._monitor_instance = monitor
|
||||
api.__dict__.pop("_library_resolver", None)
|
||||
library_move.move_library = real_move
|
||||
PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER = keep
|
||||
if config_bytes is None:
|
||||
config_path.unlink(missing_ok=True)
|
||||
else:
|
||||
@@ -172,6 +202,43 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
api.invalidate_gallery_cache()
|
||||
return web.json_response({"success": True})
|
||||
|
||||
@routes.post("/__test__/system/legacy-previews")
|
||||
async def legacy_previews(request):
|
||||
data = await request.json()
|
||||
count = data.get("count")
|
||||
if (
|
||||
isinstance(count, bool)
|
||||
or not isinstance(count, int)
|
||||
or not 1 <= count <= MAX_UNTHUMBNAILED
|
||||
):
|
||||
raise web.HTTPBadRequest()
|
||||
folder = root / "output" / "legacy"
|
||||
previews = root / "output" / "thumbnails" / "legacy"
|
||||
folder.mkdir(exist_ok=True)
|
||||
previews.mkdir(parents=True, exist_ok=True)
|
||||
names = []
|
||||
for number in range(count):
|
||||
name = f"old_{number + 1:03}"
|
||||
with Image.effect_noise((256, 256), 50) as noise:
|
||||
noise.convert("RGB").save(folder / f"{name}.png")
|
||||
noise.convert("RGB").save(previews / f"{name}_thumb.png")
|
||||
later = (folder / f"{name}.png").stat().st_mtime + 5
|
||||
os.utime(previews / f"{name}_thumb.png", (later, later))
|
||||
names.append(f"legacy/{name}.png")
|
||||
api.invalidate_gallery_cache()
|
||||
return web.json_response({"success": True, "files": names})
|
||||
|
||||
@routes.post("/__test__/system/jxl-encoder")
|
||||
async def jxl_encoder(request):
|
||||
data = await request.json()
|
||||
available = data.get("available")
|
||||
if not isinstance(available, bool):
|
||||
raise web.HTTPBadRequest()
|
||||
# Test-only: stand in the cached detection (no probe).
|
||||
with thumbnail_encode._detect_lock:
|
||||
thumbnail_encode._detected = "cjxl" if available else None
|
||||
return web.json_response({"success": True})
|
||||
|
||||
@routes.post("/__test__/system/monitor")
|
||||
async def fake_monitor(request):
|
||||
data = await request.json()
|
||||
@@ -187,6 +254,43 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
)
|
||||
return web.json_response({"success": True})
|
||||
|
||||
@routes.post("/__test__/system/library")
|
||||
async def library_state(request):
|
||||
data = await request.json()
|
||||
state = data.get("state")
|
||||
if state not in ("node_folder", "failing"):
|
||||
raise web.HTTPBadRequest()
|
||||
target = str(root / "user" / "__promptmanager")
|
||||
|
||||
def located(source):
|
||||
return library_location.LibraryLocation(
|
||||
source,
|
||||
api.db.model.db_path,
|
||||
str(config_path),
|
||||
str(root),
|
||||
target if source == library_location.SOURCE_NODE_FOLDER else None,
|
||||
target_source=library_location.SOURCE_DEFAULT,
|
||||
)
|
||||
|
||||
current = {"source": library_location.SOURCE_NODE_FOLDER}
|
||||
api._library_resolver = lambda: located(current["source"])
|
||||
|
||||
def stand_in_move(plan, *, progress, **_kwargs):
|
||||
if state == "failing":
|
||||
raise library_move.MoveFailed("verify-failed")
|
||||
progress("Copying the database")
|
||||
counts = api.db.library_counts()
|
||||
current["source"] = library_location.SOURCE_DEFAULT
|
||||
return library_move.MoveResult(
|
||||
located(library_location.SOURCE_DEFAULT),
|
||||
{"prompts": counts["positive_prompts"], "generated_images": 0},
|
||||
["prompts.db"],
|
||||
["browser.db.moved-to-__promptmanager"],
|
||||
)
|
||||
|
||||
library_move.move_library = stand_in_move
|
||||
return web.json_response({"success": True})
|
||||
|
||||
return reset
|
||||
|
||||
|
||||
|
||||
@@ -148,7 +148,12 @@ def seed_tagging(api: PromptManagerAPI, root: Path, extra_images: int = 0) -> di
|
||||
image_id = db.link_image_to_prompt(positive_id, str(output / relative))
|
||||
if kind == "linked":
|
||||
db.link_image_to_pair(pair_id, image_id, {"source": "fixture"})
|
||||
api._generate_gallery_thumbnails_sync(api._get_all_output_dirs(), (150, 150))
|
||||
from py.api.thumbnail_settings import ThumbnailSettings
|
||||
|
||||
# 150 px previews in the default format (#317 thumbnail settings).
|
||||
api._generate_gallery_thumbnails_sync(
|
||||
api._get_all_output_dirs(), ThumbnailSettings("webp", 80, 150)
|
||||
)
|
||||
api.invalidate_gallery_cache()
|
||||
return {
|
||||
"positive_id": positive_id,
|
||||
|
||||
@@ -164,5 +164,10 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None:
|
||||
path = root / directory / category.lower() / filename
|
||||
_write_art(path, index, positive, negatives[offset % len(negatives)])
|
||||
api.db.link_image_to_prompt(prompt_id, str(path))
|
||||
api._generate_gallery_thumbnails_sync(api._get_all_output_dirs(), (150, 150))
|
||||
from py.api.thumbnail_settings import ThumbnailSettings
|
||||
|
||||
# 150 px previews in the default format (#317 thumbnail settings).
|
||||
api._generate_gallery_thumbnails_sync(
|
||||
api._get_all_output_dirs(), ThumbnailSettings("webp", 80, 150)
|
||||
)
|
||||
api.invalidate_gallery_cache()
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
// #249 S6b: a fresh install keeps its library in ComfyUI's system user folder. A pinned CPU ComfyUI starts with Prompt
|
||||
// Manager and no configuration at all (no config.json, no PROMPT_MANAGER_* paths): the library must be created in
|
||||
// user/__promptmanager, nothing may be written into the node folder, and ComfyUI's /userdata API must not serve it.
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { access, readdir } from 'node:fs/promises';
|
||||
import { createServer } from 'node:net';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { test, expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
|
||||
import { launchSharedHost } from '../../tools/vendor/comfyui-host.mjs';
|
||||
import runtime from '../../tools/vendor/comfyui-runtime.json' with { type: 'json' };
|
||||
|
||||
const repositoryRoot = fileURLToPath(new URL('../../', import.meta.url));
|
||||
|
||||
/** An unused loopback port chosen by the OS (this host shares no port with the other suites). */
|
||||
function freePort() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const probe = createServer();
|
||||
probe.once('error', reject);
|
||||
probe.listen({ host: '127.0.0.1', port: 0, exclusive: true }, () => {
|
||||
const { port } = probe.address();
|
||||
probe.close(error => error ? reject(error) : resolve(port));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
test('a fresh host keeps the library in user/__promptmanager and serves none of it', async ({ playwright }, testInfo) => {
|
||||
test.skip(testInfo.project.name !== 'classic-start', 'one fresh host is enough; the renderer does not matter here');
|
||||
test.setTimeout(180000);
|
||||
const port = await freePort();
|
||||
const host = await launchSharedHost({ checkout: process.env.PM_COMFY_ROOT, repository: repositoryRoot,
|
||||
python: process.env.PM_TEST_PYTHON || 'python', port, pins: runtime, forwardOutput: false,
|
||||
logPath: join(repositoryRoot, 'tools/vendor/test-results/executions/fresh-host.log'), pluginName: 'promptmanager',
|
||||
// A fresh install: nothing written, and no inherited path variable may choose the folder.
|
||||
preparePlugin: async () => ({ PROMPT_MANAGER_CONFIG_PATH: '', PROMPT_MANAGER_DATA_DIR: '',
|
||||
PROMPT_MANAGER_EXTRA_GALLERY_ROOTS: '' }) });
|
||||
const request = await playwright.request.newContext({ baseURL: `http://127.0.0.1:${port}/` });
|
||||
try {
|
||||
await expect.poll(async () => {
|
||||
if (host.child.exitCode !== null) throw new Error('The fresh host exited before it answered');
|
||||
try { return (await request.get('/system_stats')).ok(); } catch { return false; }
|
||||
}, { timeout: 120000 }).toBe(true);
|
||||
expect((await (await request.get('/system_stats')).json()).system.comfyui_version).toBe(runtime.serverVersion);
|
||||
await expect(await request.get('/prompt_manager/stats')).toBeOK();
|
||||
|
||||
const dataFolder = join(host.stateDirectory, 'user', '__promptmanager');
|
||||
await access(join(dataFolder, 'prompts.db'));
|
||||
await access(join(dataFolder, 'logs'));
|
||||
const { location } = await (await request.get('/prompt_manager/system/library')).json();
|
||||
expect(location).toMatchObject({ path: 'user/__promptmanager/prompts.db', source: 'default', movable: false,
|
||||
offer: false, problem: null });
|
||||
|
||||
// Nothing but byte code was added to the node folder: no database, config.json, logs or caches.
|
||||
const installed = join(host.stateDirectory, 'custom_nodes', 'promptmanager');
|
||||
const tracked = new Set(execFileSync('git', ['-C', repositoryRoot, 'ls-files', '-z'], { encoding: 'utf8' })
|
||||
.split('\0').filter(Boolean).map(file => file.split('/')[0]));
|
||||
const added = (await readdir(installed)).filter(name => !tracked.has(name) && name !== '__pycache__');
|
||||
expect(added).toEqual([]);
|
||||
|
||||
// ComfyUI's /userdata serves public users only: the system user's folder is out of reach.
|
||||
for (const path of ['/userdata/..%2F__promptmanager%2Fprompts.db', '/api/userdata/..%2F__promptmanager%2Fprompts.db',
|
||||
'/userdata/%2E%2E%2F__promptmanager%2Fconfig.json', '/userdata/prompts.db']) {
|
||||
const response = await request.get(path);
|
||||
expect(response.status(), path).not.toBe(200);
|
||||
}
|
||||
const listing = await request.get('/userdata?dir=..%2F__promptmanager&recurse=true');
|
||||
expect(listing.status()).not.toBe(200);
|
||||
const asSystemUser = await request.get('/userdata/prompts.db', { headers: { 'comfy-user': '__promptmanager' } });
|
||||
expect(asSystemUser.status()).not.toBe(200);
|
||||
} finally {
|
||||
try { await request.dispose(); } finally { await host.stop(); }
|
||||
}
|
||||
});
|
||||
@@ -39,3 +39,23 @@ test('stale inspection echoes use the canonical recovery vocabulary and identity
|
||||
});
|
||||
}
|
||||
});
|
||||
test('the recorded output inspector cover is its preview, not the original (#249)',()=>{const a=adapter(),target=media('2',1);const record=a.toRecord(envelope(target),target,extractGenerationFacts);assert.deepEqual(record.preview,{url:target.thumbnail_url});assert.notEqual(record.preview.url,record.download.url);});
|
||||
|
||||
// #317 M3 follow-up: Firefox may decode JPEG XL without listing image/jxl in Accept, so the cover URL says so itself.
|
||||
test('the recorded output cover asks for JPEG XL only where this browser decodes it',async()=>{
|
||||
const support=await import('../../web/js/workspace/thumbnail-support.mjs');
|
||||
const a=adapter(),target=media('2',1);
|
||||
const probe=ok=>({createImage:()=>({set src(_){queueMicrotask(()=>ok?this.onload():this.onerror());}})});
|
||||
try {
|
||||
support.resetJxlSupport();
|
||||
assert.equal(a.toRecord(envelope(target),target,extractGenerationFacts).preview.url,target.thumbnail_url,'unknown: no flag');
|
||||
await support.detectJxlSupport(probe(true));
|
||||
const flagged=new URL(a.toRecord(envelope(target),target,extractGenerationFacts).preview.url,'http://localhost/');
|
||||
assert.equal(flagged.searchParams.get('jxl'),'1');
|
||||
assert.equal(flagged.searchParams.getAll('jxl').length,1);
|
||||
assert.equal(`${flagged.pathname}?${new URLSearchParams([...flagged.searchParams].filter(([k])=>k!=='jxl'))}`,target.thumbnail_url);
|
||||
support.resetJxlSupport();
|
||||
await support.detectJxlSupport(probe(false));
|
||||
assert.equal(a.toRecord(envelope(target),target,extractGenerationFacts).preview.url,target.thumbnail_url,'no support: no flag');
|
||||
} finally {support.resetJxlSupport();}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
// #317: a card's media is the listing's thumbnail (WebP, JPEG XL or JPEG), never something the browser cannot show:
|
||||
// a JPEG XL thumbnail in a browser without JPEG XL support loads the original, and a thumbnail that fails to load falls
|
||||
// back to the original. A video whose preview fails still swaps back to its own element (#249).
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mountedDom } from './helpers/mounted-dom.mjs';
|
||||
|
||||
const support = await import('../../web/js/workspace/thumbnail-support.mjs');
|
||||
const { cardMedia } = await import('../../web/js/workspace/image-card.mjs');
|
||||
|
||||
function withDom(run) {
|
||||
const dom = mountedDom();
|
||||
dom.win.ApiClient = { resolveUrl: url => `http://host${url}` };
|
||||
try { return run(dom); } finally { support.resetJxlSupport(); dom.restore(); }
|
||||
}
|
||||
|
||||
const image = (thumbnail, extra = {}) => ({ root_index: 0, path: 'a.png', filename: 'a.png', url: '/serve/a.png?root=0',
|
||||
thumbnail_url: thumbnail, ...extra });
|
||||
|
||||
test('a WebP thumbnail is the card media and falls back to the original when it fails', () => withDom(() => {
|
||||
const node = cardMedia(image('/serve/thumbnails/a_thumb.webp?root=0'));
|
||||
assert.equal(node.tagName, 'img');
|
||||
assert.equal(node.src, 'http://host/serve/thumbnails/a_thumb.webp?root=0');
|
||||
node.fire('error');
|
||||
assert.equal(node.src, 'http://host/serve/a.png?root=0');
|
||||
}));
|
||||
|
||||
test('a JPEG XL thumbnail loads the original where this browser cannot show JPEG XL', async () => {
|
||||
support.resetJxlSupport();
|
||||
await support.detectJxlSupport({ createImage: () => ({ set src(_) { queueMicrotask(() => this.onerror()); } }) });
|
||||
const dom = mountedDom();
|
||||
dom.win.ApiClient = { resolveUrl: url => `http://host${url}` };
|
||||
try {
|
||||
assert.equal(support.jxlSupport(), false);
|
||||
const node = cardMedia(image('/serve/thumbnails/a_thumb.jxl?root=0'));
|
||||
assert.equal(node.src, 'http://host/serve/a.png?root=0');
|
||||
} finally { support.resetJxlSupport(); dom.restore(); }
|
||||
});
|
||||
|
||||
test('a JPEG XL thumbnail is used where this browser shows JPEG XL', async () => {
|
||||
await support.detectJxlSupport({ createImage: () => ({ set src(_) { queueMicrotask(() => this.onload()); } }) });
|
||||
const dom = mountedDom();
|
||||
dom.win.ApiClient = { resolveUrl: url => `http://host${url}` };
|
||||
try {
|
||||
const node = cardMedia(image('/serve/thumbnails/a_thumb.jxl?root=0'));
|
||||
assert.equal(node.src, 'http://host/serve/thumbnails/a_thumb.jxl?root=0');
|
||||
} finally { support.resetJxlSupport(); dom.restore(); }
|
||||
});
|
||||
|
||||
test('a video whose preview fails still swaps back to its own element', () => withDom(dom => {
|
||||
const holder = new dom.Node('div');
|
||||
const node = cardMedia(image('/serve/thumbnails/c_thumb.webp?root=0', { path: 'c.mp4', filename: 'c.mp4', url: '/serve/c.mp4?root=0', is_video: true }));
|
||||
holder.append(node);
|
||||
node.replaceWith = replacement => { holder.children = [replacement]; };
|
||||
node.fire('error');
|
||||
assert.equal(holder.children[0].tagName, 'video');
|
||||
assert.equal(holder.children[0].src, 'http://host/serve/c.mp4?root=0');
|
||||
}));
|
||||
|
||||
test('a video with a JPEG XL thumbnail keeps its own element where JPEG XL cannot be shown (review M2)', async () => {
|
||||
support.resetJxlSupport();
|
||||
await support.detectJxlSupport({ createImage: () => ({ set src(_) { queueMicrotask(() => this.onerror()); } }) });
|
||||
const dom = mountedDom();
|
||||
dom.win.ApiClient = { resolveUrl: url => `http://host${url}` };
|
||||
try {
|
||||
const node = cardMedia(image('/serve/thumbnails/c_thumb.jxl?root=0',
|
||||
{ path: 'c.mp4', filename: 'c.mp4', url: '/serve/c.mp4?root=0', is_video: true }));
|
||||
assert.equal(node.tagName, 'video');
|
||||
assert.equal(node.src, 'http://host/serve/c.mp4?root=0');
|
||||
} finally { support.resetJxlSupport(); dom.restore(); }
|
||||
});
|
||||
@@ -895,3 +895,50 @@ test('cache off never rereads a selection whose source changed, and refresh neve
|
||||
assert.equal(deferred.state.deferred, true);
|
||||
deferred.dispose();
|
||||
});
|
||||
|
||||
test('record carries the listing thumbnail as its preview, never another file (#249)', () => {
|
||||
const toRecord = contract('toInspectionRecord'), entry = image();
|
||||
const thumb = `/prompt_manager/images/serve/thumbnails/folder/a%20%23%25_thumb.png?root=0&root_scope=${scopeA}`;
|
||||
assert.deepEqual(toRecord(envelope(entry), { ...entry, thumbnail_url: thumb }, extractGenerationFacts).preview, { url: thumb });
|
||||
const versioned = `/prompt_manager/images/serve/thumbnails/folder/a%20%23%25_thumb.png?v=12&root=0&root_scope=${scopeA}`;
|
||||
assert.deepEqual(toRecord(envelope(entry), { ...entry, thumbnail_url: versioned }, extractGenerationFacts).preview, { url: versioned });
|
||||
// No thumbnail, or one naming another file, root or origin: no preview (the cover keeps the original).
|
||||
for (const thumbnail_url of [undefined, null, '', thumb.replace('a%20%23%25_thumb', 'b_thumb'), thumb.replace('root=0', 'root=1'),
|
||||
thumb.replace(scopeA, scopeB), `https://other.test${thumb}`, thumb.replace('thumbnails/', ''), `${thumb}#x`]) {
|
||||
assert.equal(toRecord(envelope(entry), { ...entry, thumbnail_url }, extractGenerationFacts).preview, null, String(thumbnail_url));
|
||||
}
|
||||
const video = { root_index: 0, path: 'clip.MP4', filename: 'clip.MP4', url: `/prompt_manager/images/serve/clip.MP4?root=0&root_scope=${scopeA}`, root_scope: scopeA,
|
||||
thumbnail_url: `/prompt_manager/images/serve/thumbnails/clip_thumb.jpg?root=0&root_scope=${scopeA}` };
|
||||
const data = envelope(video); data.file = { ...data.file, media_type: 'video', extension: '.mp4', format: 'MP4' };
|
||||
assert.deepEqual(toRecord(data, video, extractGenerationFacts).preview, { url: video.thumbnail_url });
|
||||
});
|
||||
|
||||
test('the preview may be any thumbnail format the file owns (#317)', () => {
|
||||
const toRecord = contract('toInspectionRecord'), entry = image();
|
||||
const base = '/prompt_manager/images/serve/thumbnails/folder/a%20%23%25_thumb';
|
||||
for (const extension of ['.webp', '.jxl', '.jpg', '.png']) {
|
||||
const thumbnail_url = `${base}${extension}?v=3&root=0&root_scope=${scopeA}`;
|
||||
assert.deepEqual(toRecord(envelope(entry), { ...entry, thumbnail_url }, extractGenerationFacts).preview, { url: thumbnail_url }, extension);
|
||||
}
|
||||
// Not a thumbnail format, upper case, no extension, or a legacy name another file owns: no preview.
|
||||
for (const extension of ['.txt', '.WEBP', '', '.gif', '.jpeg', '.webp.png']) {
|
||||
const thumbnail_url = `${base}${extension}?root=0&root_scope=${scopeA}`;
|
||||
assert.equal(toRecord(envelope(entry), { ...entry, thumbnail_url }, extractGenerationFacts).preview, null, extension);
|
||||
}
|
||||
const video = { root_index: 0, path: 'clip.MP4', filename: 'clip.MP4', url: `/prompt_manager/images/serve/clip.MP4?root=0&root_scope=${scopeA}`, root_scope: scopeA };
|
||||
const data = envelope(video); data.file = { ...data.file, media_type: 'video', extension: '.mp4', format: 'MP4' };
|
||||
for (const [extension, owned] of [['.webp', true], ['.jxl', true], ['.jpg', true], ['.png', false]]) {
|
||||
const thumbnail_url = `/prompt_manager/images/serve/thumbnails/clip_thumb${extension}?root=0&root_scope=${scopeA}`;
|
||||
assert.deepEqual(toRecord(data, { ...video, thumbnail_url }, extractGenerationFacts).preview, owned ? { url: thumbnail_url } : null, extension);
|
||||
}
|
||||
});
|
||||
|
||||
test("names the server quotes (parentheses, '!*') still match their original and preview (#249 M11)", () => {
|
||||
const toRecord = contract('toInspectionRecord');
|
||||
const entry = { root_index: 0, path: "folder/a (1)!'*.png", filename: "a (1)!'*.png", root_scope: scopeA,
|
||||
url: `/prompt_manager/images/serve/folder/a%20%281%29%21%27%2A.png?root=0&root_scope=${scopeA}`,
|
||||
thumbnail_url: `/prompt_manager/images/serve/thumbnails/folder/a%20%281%29%21%27%2A_thumb.png?root=0&root_scope=${scopeA}` };
|
||||
const record = toRecord(envelope(entry), entry, extractGenerationFacts);
|
||||
assert.equal(record.download.url, entry.url);
|
||||
assert.deepEqual(record.preview, { url: entry.thumbnail_url });
|
||||
});
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
// Where the library lives (#249 System S6b, D129): the Overview row's copy, the Move library review and its start,
|
||||
// "Keep it here" and the Move library job's end.
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const locationModule = import('../../web/js/workspace/library-location.mjs');
|
||||
|
||||
const NODE_FOLDER = Object.freeze({
|
||||
path: 'custom_nodes/ComfyUI_PromptManager/prompts.db', source: 'node_folder', movable: true, keep: false, offer: true,
|
||||
target: 'user/__promptmanager', problem: null, size_bytes: 1_700_000_000,
|
||||
target_source: 'default', config_moves: true,
|
||||
});
|
||||
|
||||
function scope() {
|
||||
const controller = new AbortController();
|
||||
return { signal: controller.signal, abort: () => controller.abort(), listen() {}, cleanup() {} };
|
||||
}
|
||||
|
||||
function activityStore({ running = [] } = {}) {
|
||||
const listeners = new Set();
|
||||
let jobs = [];
|
||||
return {
|
||||
tracked: [], loads: 0,
|
||||
track(job) { this.tracked.push(job); jobs = [...jobs, job]; return job; },
|
||||
async load() { this.loads += 1; },
|
||||
running() { return running; },
|
||||
jobs() { return jobs; },
|
||||
set(next) { jobs = next; for (const listener of listeners) listener(); },
|
||||
subscribe(listener) { listeners.add(listener); return () => listeners.delete(listener); },
|
||||
};
|
||||
}
|
||||
|
||||
class ApiError extends Error {
|
||||
constructor(status, message, code) { super(message); this.status = status; this.code = code; }
|
||||
}
|
||||
|
||||
test('the row says where the library is and why, the choice and any problem', async () => {
|
||||
const { locationDescription, SOURCE_TEXT } = await locationModule;
|
||||
assert.equal(locationDescription(NODE_FOLDER),
|
||||
'custom_nodes/ComfyUI_PromptManager/prompts.db · The node folder, where an update or reinstall can delete it.');
|
||||
assert.equal(locationDescription({ ...NODE_FOLDER, keep: true }),
|
||||
`${NODE_FOLDER.path} · ${SOURCE_TEXT.node_folder}. You chose to keep it here.`);
|
||||
assert.equal(locationDescription({ path: 'user/__promptmanager/prompts.db', source: 'default', movable: false, keep: true }),
|
||||
'user/__promptmanager/prompts.db · ComfyUI\'s user folder, which updates and reinstalls of the node leave alone.');
|
||||
assert.match(locationDescription({ ...NODE_FOLDER, problem: 'data-dir-empty' }), /PROMPT_MANAGER_DATA_DIR has no library yet/);
|
||||
assert.match(locationDescription({ path: 'prompts.db', source: 'environment', movable: false }), /PROMPT_MANAGER_DATA_DIR names/);
|
||||
assert.match(locationDescription({ path: 'prompts.db', source: 'config', movable: false }), /config\.json/);
|
||||
});
|
||||
|
||||
test('the notice and the review name the actual target and what moves', async () => {
|
||||
const { bannerText, reviewGrid, locationDescription } = await locationModule;
|
||||
assert.match(bannerText(NODE_FOLDER), /Move it to ComfyUI's user folder: it is copied/);
|
||||
assert.match(bannerText({ ...NODE_FOLDER, target_source: 'environment' }),
|
||||
/Move it to the folder PROMPT_MANAGER_DATA_DIR names: it is copied/);
|
||||
assert.match(bannerText(null), /ComfyUI's user folder/);
|
||||
assert.equal(reviewGrid({ ...NODE_FOLDER, config_moves: false })[3][1],
|
||||
'The database backups (config.json stays where PROMPT_MANAGER_CONFIG_PATH names it)');
|
||||
assert.equal(reviewGrid({ ...NODE_FOLDER, config_moves: true })[3][1], 'config.json and the database backups');
|
||||
assert.match(locationDescription({ path: 'custom_nodes/pm/data/my.db', source: 'config', movable: false,
|
||||
problem: 'inside-node-folder' }), /inside the node folder, where an update or reinstall deletes it/);
|
||||
});
|
||||
|
||||
test('only a well-formed location is shown', async () => {
|
||||
const { isLocation } = await locationModule;
|
||||
assert.equal(isLocation(NODE_FOLDER), true);
|
||||
for (const bad of [null, undefined, 'x', {}, { ...NODE_FOLDER, source: 'elsewhere' }, { ...NODE_FOLDER, path: 3 },
|
||||
{ ...NODE_FOLDER, movable: 'yes' }]) assert.equal(isLocation(bad), false);
|
||||
});
|
||||
|
||||
test('sizes read as the server would write them', async () => {
|
||||
const { sizeText } = await locationModule;
|
||||
assert.equal(sizeText(1_700_000_000), '1.6 GB');
|
||||
assert.equal(sizeText(120 * 2 ** 20), '120.0 MB');
|
||||
assert.equal(sizeText(1500), '2 KB');
|
||||
assert.equal(sizeText(-1), 'Unknown');
|
||||
assert.equal(sizeText(undefined), 'Unknown');
|
||||
});
|
||||
|
||||
test('the review names both folders, the size and what happens to the old files', async () => {
|
||||
const { reviewGrid } = await locationModule;
|
||||
assert.deepEqual(reviewGrid(NODE_FOLDER), [
|
||||
['From', 'custom_nodes/ComfyUI_PromptManager/prompts.db'],
|
||||
['To', 'user/__promptmanager/prompts.db'],
|
||||
['Size', '1.6 GB'],
|
||||
['Moves with it', 'config.json and the database backups'],
|
||||
['Old files', 'Renamed *.moved-to-…, never deleted'],
|
||||
]);
|
||||
});
|
||||
|
||||
test('Move library starts the job, tracks it and opens its view', async () => {
|
||||
const { startLibraryMove, MOVE_URL } = await locationModule;
|
||||
const posts = [];
|
||||
const api = { post: async (url, body) => { posts.push([url, body]); return { job: { id: 'j1', kind: 'library_move' } }; } };
|
||||
const activity = activityStore();
|
||||
const routes = [];
|
||||
const job = await startLibraryMove({ api, activity, scope: scope(), notify: () => assert.fail('no toast'),
|
||||
navigate: hash => routes.push(hash) });
|
||||
assert.equal(job.id, 'j1');
|
||||
assert.deepEqual(posts, [[MOVE_URL, {}]]);
|
||||
assert.deepEqual(activity.tracked.map(item => item.id), ['j1']);
|
||||
assert.deepEqual(routes, ['#/activity/j1']);
|
||||
});
|
||||
|
||||
test('a busy server names the running job; another refusal says the server reason', async () => {
|
||||
const { startLibraryMove, START_FAILED } = await locationModule;
|
||||
const running = { id: 'scan1', title: 'Metadata scan' };
|
||||
const activity = activityStore({ running: [running] });
|
||||
const toasts = [];
|
||||
const busy = { post: async () => { throw new ApiError(409, 'busy', 'busy'); } };
|
||||
assert.equal(await startLibraryMove({ api: busy, activity, scope: scope(), notify: t => toasts.push(t),
|
||||
navigate: () => assert.fail('nothing opens') }), null);
|
||||
assert.equal(activity.loads, 1);
|
||||
assert.deepEqual(toasts.at(-1), ['Another job is running: ', { text: 'Metadata scan', href: '#/activity/scan1' }, '.']);
|
||||
const short = { post: async () => { throw new ApiError(507, 'There is not enough free space for the copy.', 'insufficient-space'); } };
|
||||
await startLibraryMove({ api: short, activity, scope: scope(), notify: t => toasts.push(t), navigate: () => {} });
|
||||
assert.equal(toasts.at(-1), 'There is not enough free space for the copy.');
|
||||
const broken = { post: async () => { throw new TypeError('network'); } };
|
||||
await startLibraryMove({ api: broken, activity, scope: scope(), notify: t => toasts.push(t), navigate: () => {} });
|
||||
assert.equal(toasts.at(-1), START_FAILED);
|
||||
const gone = scope();
|
||||
gone.abort();
|
||||
const before = toasts.length;
|
||||
await startLibraryMove({ api: broken, activity, scope: gone, notify: t => toasts.push(t), navigate: () => {} });
|
||||
assert.equal(toasts.length, before);
|
||||
});
|
||||
|
||||
test('Keep it here posts the choice and resolves the new location, or null with a toast', async () => {
|
||||
const { keepLibrary, KEEP_URL, KEEP_FAILED } = await locationModule;
|
||||
const posts = [];
|
||||
const kept = { ...NODE_FOLDER, keep: true, offer: false };
|
||||
const api = { post: async (url, body) => { posts.push([url, body]); return { location: kept }; } };
|
||||
assert.deepEqual(await keepLibrary({ api, scope: scope() }), kept);
|
||||
assert.deepEqual(posts, [[KEEP_URL, { keep: true }]]);
|
||||
const toasts = [];
|
||||
const failing = { post: async () => { throw new ApiError(500, 'no', 'config-write-failed'); } };
|
||||
assert.equal(await keepLibrary({ api: failing, scope: scope(), notify: t => toasts.push(t) }), null);
|
||||
assert.deepEqual(toasts, [KEEP_FAILED]);
|
||||
const odd = { post: async () => ({ location: { path: 1 } }) };
|
||||
assert.equal(await keepLibrary({ api: odd, scope: scope() }), null);
|
||||
});
|
||||
|
||||
test('the end of a Move library job is reported once; other jobs are ignored', async () => {
|
||||
const { onMoveEnded } = await locationModule;
|
||||
const activity = activityStore();
|
||||
let ended = 0;
|
||||
const unsubscribe = onMoveEnded(activity, () => { ended += 1; });
|
||||
activity.set([{ id: 'm', kind: 'library_move', state: 'running' }, { id: 's', kind: 'scan', state: 'running' }]);
|
||||
assert.equal(ended, 0);
|
||||
activity.set([{ id: 'm', kind: 'library_move', state: 'running' }, { id: 's', kind: 'scan', state: 'finished' }]);
|
||||
assert.equal(ended, 0);
|
||||
activity.set([{ id: 'm', kind: 'library_move', state: 'failed' }]);
|
||||
assert.equal(ended, 1);
|
||||
activity.set([{ id: 'm', kind: 'library_move', state: 'failed' }]);
|
||||
assert.equal(ended, 1);
|
||||
unsubscribe();
|
||||
activity.set([{ id: 'n', kind: 'library_move', state: 'running' }]);
|
||||
activity.set([{ id: 'n', kind: 'library_move', state: 'finished' }]);
|
||||
assert.equal(ended, 1);
|
||||
});
|
||||
@@ -324,3 +324,29 @@ test('server adapters still cache-bust the original on open and retry', async ()
|
||||
} finally { view.dispose(); controller.dispose(); parentScope.dispose(); restoreGlobals(); f.restore(); }
|
||||
}
|
||||
});
|
||||
|
||||
test('the viewer and its download always use the original, never the thumbnail in any format (#317)', async () => {
|
||||
const {rootedFileAdapter, createInspectionController} = await import('../../web/js/workspace/inspection-state.mjs');
|
||||
const scope = 'a'.repeat(64);
|
||||
for (const extension of ['.webp', '.jxl', '.jpg', '.png']) {
|
||||
const item = {root_index:0,path:'folder/a.png',filename:'a.png',root_scope:scope,is_video:false,
|
||||
url:`/prompt_manager/images/serve/folder/a.png?root=0&root_scope=${scope}`,
|
||||
thumbnail_url:`/prompt_manager/images/serve/thumbnails/folder/a_thumb${extension}?v=3&root=0&root_scope=${scope}`};
|
||||
const f = mountedDom(), restoreGlobals = withViewerGlobals(f), parentScope = createScope(), paths = [];
|
||||
const controller = createInspectionController({adapter:rootedFileAdapter,api:{get(url) {paths.push(url);return new Promise(() => {});}}});
|
||||
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter:rootedFileAdapter,
|
||||
onSelect:entry => controller.select(entry),sender:{open(){}}});
|
||||
try {
|
||||
view.open({items:[item],selectedKey:'0:folder/a.png'});
|
||||
// The metadata read is the original's inspection, never a path under thumbnails/.
|
||||
assert.deepEqual(paths, [`/prompt_manager/images/inspect/folder/a.png?root=0&root_scope=${scope}`], extension);
|
||||
const media = f.walk(f.container,n => n.tagName === 'img');
|
||||
assert.equal(media.length, 1, extension);
|
||||
const url = new URL(media[0].src);url.searchParams.delete('_pm_viewer');
|
||||
assert.equal(`${url.pathname}${url.search}`, item.url, extension);
|
||||
assert.doesNotMatch(media[0].src, /thumbnails/, extension);
|
||||
const download = f.walk(f.container,n => n.attributes['aria-label'] === 'Download original')[0];
|
||||
if (download) assert.equal(download.href, item.url, extension);
|
||||
} finally { view.dispose(); controller.dispose(); parentScope.dispose(); restoreGlobals(); f.restore(); }
|
||||
}
|
||||
});
|
||||
|
||||
@@ -59,6 +59,18 @@ test('the thumbnail row is its own read of the thumbnail status', async () => {
|
||||
assert.equal(thumbnailTitle({ pending: 2 }, { effective: 1 }), '2 missing thumbnails');
|
||||
});
|
||||
|
||||
test('the thumbnail row says N missing · M outdated · K failed, and offers Convert outdated only for outdated ones (#317)', async () => {
|
||||
const { thumbnailTitle, convertOffered } = await overviewModule;
|
||||
assert.equal(thumbnailTitle({ pending: 3, outdated: 17500, failed: 2 }), '3 missing thumbnails · 17,500 outdated · 2 failed');
|
||||
assert.equal(thumbnailTitle({ pending: 0, outdated: 1, failed: 0 }), 'No missing thumbnails · 1 outdated');
|
||||
assert.equal(thumbnailTitle({ pending: 0, outdated: 0, failed: 4 }), 'No missing thumbnails · 4 failed');
|
||||
assert.equal(thumbnailTitle({ pending: 1, outdated: 'x' }), '1 missing thumbnail');
|
||||
assert.equal(convertOffered({ pending: 0, outdated: 5 }), true);
|
||||
for (const status of [undefined, null, { pending: 0 }, { outdated: 0 }, { outdated: -1 }, { outdated: '5' }]) {
|
||||
assert.equal(convertOffered(status), false, JSON.stringify(status));
|
||||
}
|
||||
});
|
||||
|
||||
test('an integrity count SQLite stopped at reads as a lower bound', async () => {
|
||||
const { checkChip } = await overviewModule;
|
||||
assert.equal(checkChip({ id: 'integrity', status: 'issues', count: 100, capped: true }), '100+ problems found');
|
||||
@@ -253,3 +265,11 @@ test('waiting stops when its signal aborts', async () => {
|
||||
gone.abort(new Error('already'));
|
||||
await assert.rejects(waitForJob(fakeActivity(), JOB, { signal: gone.signal }), { message: 'already' });
|
||||
});
|
||||
|
||||
test('the thumbnail row counts previews that could not be made (#249)', async () => {
|
||||
const { thumbnailTitle } = await overviewModule;
|
||||
assert.equal(thumbnailTitle({ pending: 12, failed: 2 }), '12 missing thumbnails · 2 failed');
|
||||
assert.equal(thumbnailTitle({ pending: 1, failed: 1 }), '1 missing thumbnail · 1 failed');
|
||||
assert.equal(thumbnailTitle({ pending: 0, failed: 0 }), 'No missing thumbnails');
|
||||
assert.equal(thumbnailTitle({ pending: 3, failed: 'x' }), '3 missing thumbnails');
|
||||
});
|
||||
|
||||
@@ -72,21 +72,35 @@ test('removing the last entry states auto-detect (D98)', async () => {
|
||||
assert.equal(NOTES.removedAll, 'No source is configured. After saving, ComfyUI’s output folder is used.');
|
||||
});
|
||||
|
||||
test('auto-detected sources are listed without Remove, with the line that says how they were chosen', async () => {
|
||||
test('auto-detected sources are listed without Remove and read as the default: "ComfyUI/<path> · <watch> · default"', async () => {
|
||||
const { sourceRows, NOTES } = await sourcesModule;
|
||||
const auto = { configured: [], effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'manual' }], auto_detected: true };
|
||||
const auto = { configured: [], auto_detected: true,
|
||||
effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'watching', in_comfyui: true }] };
|
||||
// The row says it is the default, so no line under the list repeats it; the scan keeps the server's public path.
|
||||
assert.deepEqual(sourceRows({ sources: auto, saved: [], current: [] }), { rows: [
|
||||
{ key: 'auto:0', title: 'ComfyUI output', description: 'output · manual scan', scan: { label: 'Scan folder', root: 0, path: 'output' },
|
||||
remove: null },
|
||||
], note: NOTES.autoDetected });
|
||||
// Every configured source unusable: ComfyUI's output folder is used instead.
|
||||
{ key: 'auto:0', title: 'ComfyUI output', description: 'ComfyUI/output · watching · default',
|
||||
scan: { label: 'Scan changes', root: 0, path: 'output' }, remove: null },
|
||||
], note: null });
|
||||
assert.equal('autoDetected' in NOTES, false);
|
||||
// An output folder outside the ComfyUI base (--output-directory) is shown by its own name.
|
||||
const outside = { ...auto, effective: [{ label: 'Renders', path: 'Renders', root_index: 0, watch: 'manual', in_comfyui: false }] };
|
||||
assert.equal(sourceRows({ sources: outside, saved: [], current: [] }).rows[0].description, 'Renders · manual scan · default');
|
||||
// Every configured source unusable: ComfyUI's output folder is used instead, and the line says why.
|
||||
const fallback = { ...auto, configured: [{ label: 'gone', path: 'gone', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
|
||||
const result = sourceRows({ sources: fallback, saved: ['gone'], current: ['gone'] });
|
||||
assert.deepEqual(result.rows.map(row => [row.title, row.remove]), [['gone', 'gone'], ['ComfyUI output', null]]);
|
||||
assert.deepEqual(result.rows.map(row => [row.title, row.description, row.remove]),
|
||||
[['gone', 'gone · Gallery root does not exist', 'gone'], ['ComfyUI output', 'ComfyUI/output · watching · default', null]]);
|
||||
assert.equal(result.note, NOTES.fallback);
|
||||
assert.equal(sourceRows({ sources: { configured: [], effective: [], auto_detected: false }, saved: [], current: [] }).note, NOTES.none);
|
||||
});
|
||||
|
||||
test('a configured source keeps its plain public path even inside ComfyUI', async () => {
|
||||
const { sourceRows } = await sourcesModule;
|
||||
const sources = { ...STATUS, effective: STATUS.effective.map(source => ({ ...source, in_comfyui: true })) };
|
||||
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
|
||||
assert.deepEqual(rows.map(row => row.description), ['output · watching', 'input · manual scan']);
|
||||
});
|
||||
|
||||
test('a path listed twice in the draft is one row', async () => {
|
||||
const { sourceRows } = await sourcesModule;
|
||||
assert.equal(sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['output', 'output'] }).rows.length, 1);
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
// System › Appearance › Thumbnails (#317): the server's thumbnail format, quality and size limit as host rows of the
|
||||
// one Save row, JPEG XL offered only with an encoder (D130), a note where this browser cannot show JPEG XL, and the
|
||||
// explicit Convert outdated operation (never automatic).
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const thumbnailsModule = import('../../web/js/workspace/system-thumbnails.mjs');
|
||||
const stateModule = import('../../web/js/workspace/system-state.mjs');
|
||||
const coreModule = import('../../web/js/shared/toolkit/core.js');
|
||||
|
||||
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
|
||||
function fakeDOM() {
|
||||
const previous = global.document;
|
||||
const elements = [];
|
||||
class Element {
|
||||
constructor(tag) {
|
||||
this.tagName = tag.toUpperCase();
|
||||
this.children = [];
|
||||
this.attributes = {};
|
||||
this.listeners = [];
|
||||
this.dataset = {};
|
||||
this.className = '';
|
||||
this.id = '';
|
||||
this.value = '';
|
||||
this._text = '';
|
||||
elements.push(this);
|
||||
}
|
||||
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
|
||||
set textContent(text) { this.children = []; this._text = String(text); }
|
||||
append(...nodes) {
|
||||
for (const node of nodes) {
|
||||
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
|
||||
child.parentNode = this;
|
||||
this.children.push(child);
|
||||
}
|
||||
}
|
||||
remove() { if (this.parentNode) this.parentNode.children = this.parentNode.children.filter(child => child !== this); this.parentNode = null; }
|
||||
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
|
||||
setAttribute(name, value) { this.attributes[name] = String(value); }
|
||||
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
|
||||
removeAttribute(name) { delete this.attributes[name]; }
|
||||
addEventListener(type, callback, options = {}) {
|
||||
if (options.signal?.aborted) return;
|
||||
const entry = { type, callback };
|
||||
this.listeners.push(entry);
|
||||
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
|
||||
}
|
||||
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
|
||||
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
|
||||
click() { this.dispatch('click'); }
|
||||
}
|
||||
global.document = {
|
||||
createElement: tag => new Element(tag),
|
||||
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
|
||||
};
|
||||
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
|
||||
return {
|
||||
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
|
||||
find: (root, predicate) => walk(root).find(predicate),
|
||||
tags: node => node.children.map(child => child.tagName.toLowerCase()),
|
||||
restore() { global.document = previous; },
|
||||
};
|
||||
}
|
||||
|
||||
const HOST = { thumbnail_format: 'webp', thumbnail_quality: 80, thumbnail_edge: 512,
|
||||
thumbnail_encoders: { webp: { available: true }, jpeg: { available: true }, jxl: { available: true, encoder: 'cjxl', reason: null } } };
|
||||
const NO_JXL = { webp: { available: true }, jpeg: { available: true },
|
||||
jxl: { available: false, encoder: null, reason: 'No JPEG XL encoder found. Install pillow-jxl-plugin.' } };
|
||||
|
||||
async function mounted({ host = HOST, browserJxl = true, hostFails = false, writeHost = async () => ({ success: true, persisted: true, revision: 'sha256:two' }) } = {}) {
|
||||
const { mountThumbnailSettings } = await thumbnailsModule;
|
||||
const { createSystemDrafts } = await stateModule;
|
||||
const { createScope } = await coreModule;
|
||||
const dom = fakeDOM();
|
||||
let hostValues = null;
|
||||
const drafts = createSystemDrafts({
|
||||
readBrowser: () => ({}), readRuntime: () => ({}),
|
||||
readHost: async () => { if (hostFails) throw new Error('offline'); return { values: hostValues, revision: 'sha256:one' }; },
|
||||
writeBrowser: async () => ({ persisted: true }), writeHost, writeRuntime: async () => ({}),
|
||||
});
|
||||
const scope = createScope();
|
||||
const reviews = [];
|
||||
let answer;
|
||||
const support = () => new Promise(resolve => { answer = resolve; });
|
||||
const surface = mountThumbnailSettings(null, { scope, drafts, api: {}, activity: {}, support,
|
||||
openReview: options => reviews.push(options) });
|
||||
const byId = id => dom.find(surface, node => node.dataset?.setting === id);
|
||||
const notes = () => dom.find(surface, node => node.dataset?.setting === 'thumbnail-format').parentNode.children[0].children.slice(2)
|
||||
.map(node => node.textContent);
|
||||
return { dom, drafts, scope, surface, byId, reviews, notes, browser: () => answer(browserJxl),
|
||||
loadHost: async () => { hostValues = host; await drafts.load('host'); } };
|
||||
}
|
||||
|
||||
test('a Thumbnails surface of three host rows and the Convert outdated operation', async () => {
|
||||
const { dom, surface, scope } = await mounted();
|
||||
try {
|
||||
assert.equal(surface.className, 'surface');
|
||||
assert.equal(surface.children[0].textContent, 'Thumbnails');
|
||||
const rows = surface.children.slice(2);
|
||||
assert.deepEqual(rows.map(row => row.children[0].children[0].textContent),
|
||||
['Thumbnail format', 'Thumbnail quality', 'Thumbnail size limit', 'Convert outdated thumbnails']);
|
||||
assert.deepEqual(rows.map(row => row.className), ['setting-row', 'setting-row', 'setting-row', 'operation']);
|
||||
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
|
||||
});
|
||||
|
||||
test('the rows edit the server settings once they were read', async () => {
|
||||
const { FORMAT_OPTIONS } = await thumbnailsModule;
|
||||
assert.deepEqual(FORMAT_OPTIONS, [['webp', 'WebP'], ['jxl', 'JPEG XL'], ['jpeg', 'JPEG']]);
|
||||
const { dom, drafts, scope, byId, loadHost } = await mounted();
|
||||
try {
|
||||
for (const id of ['thumbnail-format', 'thumbnail-quality', 'thumbnail-edge']) assert.equal(byId(id).disabled, true, id);
|
||||
await loadHost();
|
||||
assert.equal(byId('thumbnail-format').value, 'webp');
|
||||
assert.equal(byId('thumbnail-quality').value, '80');
|
||||
assert.equal(byId('thumbnail-edge').value, '512');
|
||||
assert.equal(byId('thumbnail-format').disabled, false);
|
||||
byId('thumbnail-format').value = 'jpeg'; byId('thumbnail-format').dispatch('change');
|
||||
byId('thumbnail-quality').value = '60'; byId('thumbnail-quality').dispatch('change');
|
||||
byId('thumbnail-edge').value = '300'; byId('thumbnail-edge').dispatch('change');
|
||||
assert.deepEqual(drafts.changes('host'), { thumbnail_format: 'jpeg', thumbnail_quality: 60, thumbnail_edge: 300 });
|
||||
} finally { scope.dispose(); dom.restore(); }
|
||||
});
|
||||
|
||||
test('quality takes 1–100 and the size limit 128–2048 whole numbers only', async () => {
|
||||
const { wholeNumber, QUALITY_LIMIT, EDGE_LIMIT } = await thumbnailsModule;
|
||||
assert.deepEqual(['1', ' 100 ', '0', '101', '80.5', '', 'x'].map(text => wholeNumber(text, QUALITY_LIMIT)), [1, 100, null, null, null, null, null]);
|
||||
assert.deepEqual(['128', '2048', '127', '2049', '1e3'].map(text => wholeNumber(text, EDGE_LIMIT)), [128, 2048, null, null, null]);
|
||||
const { dom, drafts, scope, byId, loadHost } = await mounted();
|
||||
try {
|
||||
await loadHost();
|
||||
byId('thumbnail-quality').value = '0'; byId('thumbnail-quality').dispatch('change');
|
||||
byId('thumbnail-edge').value = '4096'; byId('thumbnail-edge').dispatch('change');
|
||||
assert.deepEqual(drafts.changes('host'), {});
|
||||
assert.deepEqual([...drafts.failed('host')].sort(), ['thumbnail_edge', 'thumbnail_quality']);
|
||||
} finally { scope.dispose(); dom.restore(); }
|
||||
});
|
||||
|
||||
test('JPEG XL is disabled with the server reason when no encoder is installed (D130)', async () => {
|
||||
const { dom, scope, byId, loadHost, notes } = await mounted({ host: { ...HOST, thumbnail_encoders: NO_JXL } });
|
||||
try {
|
||||
await loadHost();
|
||||
const jxl = byId('thumbnail-format').children.find(option => option.value === 'jxl');
|
||||
assert.equal(jxl.disabled, true);
|
||||
assert.deepEqual(notes(), [NO_JXL.jxl.reason]);
|
||||
} finally { scope.dispose(); dom.restore(); }
|
||||
const ready = await mounted();
|
||||
try {
|
||||
await ready.loadHost();
|
||||
assert.equal(ready.byId('thumbnail-format').children.find(option => option.value === 'jxl').disabled, false);
|
||||
assert.deepEqual(ready.notes(), []);
|
||||
} finally { ready.scope.dispose(); ready.dom.restore(); }
|
||||
});
|
||||
|
||||
test('a browser that cannot show JPEG XL says the originals are shown instead', async () => {
|
||||
const { BROWSER_NO_JXL } = await thumbnailsModule;
|
||||
const { dom, scope, loadHost, notes, browser } = await mounted({ browserJxl: false });
|
||||
try {
|
||||
await loadHost();
|
||||
browser();
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
assert.deepEqual(notes(), [BROWSER_NO_JXL]);
|
||||
} finally { scope.dispose(); dom.restore(); }
|
||||
});
|
||||
|
||||
test('Convert outdated opens Thumbnail maintenance on its convert scope; a format save says previews keep theirs', async () => {
|
||||
const { FORMAT_SAVED, CONVERT_DESCRIPTION } = await thumbnailsModule;
|
||||
const { dom, drafts, scope, surface, byId, reviews, loadHost } = await mounted();
|
||||
try {
|
||||
const operation = surface.children.at(-1);
|
||||
assert.equal(operation.children[0].children[1].textContent, CONVERT_DESCRIPTION);
|
||||
dom.find(operation, node => node.tagName === 'BUTTON').click();
|
||||
assert.equal(reviews.length, 1);
|
||||
assert.equal(reviews[0].mode, 'convert');
|
||||
await loadHost();
|
||||
byId('thumbnail-quality').value = '70'; byId('thumbnail-quality').dispatch('change');
|
||||
await drafts.save();
|
||||
assert.equal(operation.children[0].children[1].textContent, CONVERT_DESCRIPTION, 'quality alone converts nothing');
|
||||
byId('thumbnail-format').value = 'jpeg'; byId('thumbnail-format').dispatch('change');
|
||||
await drafts.save();
|
||||
assert.equal(operation.children[0].children[1].textContent, FORMAT_SAVED);
|
||||
assert.equal(reviews.length, 1, 'nothing starts by itself');
|
||||
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
|
||||
});
|
||||
|
||||
test('rows whose server settings could not be read say so, like Appearance’s host rows', async () => {
|
||||
const { HOST_UNAVAILABLE } = await import('../../web/js/workspace/system-appearance.mjs');
|
||||
const { dom, drafts, scope, surface, byId } = await mounted({ hostFails: true });
|
||||
const notes = () => surface.children.slice(2, 5).map(row => row.children[0].children.slice(2).map(note => note.textContent));
|
||||
try {
|
||||
assert.deepEqual(notes(), [[], [], []]);
|
||||
assert.equal(await drafts.reloadHost(), false);
|
||||
assert.deepEqual(notes(), [[HOST_UNAVAILABLE], [HOST_UNAVAILABLE], [HOST_UNAVAILABLE]]);
|
||||
for (const id of ['thumbnail-format', 'thumbnail-quality', 'thumbnail-edge']) assert.equal(byId(id).disabled, true, id);
|
||||
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
|
||||
});
|
||||
@@ -1,35 +1,36 @@
|
||||
// System › thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (app.js 1433–1437,
|
||||
// `thumbnail-review`) on the thumbnail job (S5 Task 12). Pure parts: the option values the job takes, the preview size
|
||||
// check, the Image loading quality preselection (Q3), the D99 plan line and the start outcome.
|
||||
// `thumbnail-review`) on the thumbnail job (S5 Task 12). Pure parts: the scopes the job takes (Convert outdated, #317),
|
||||
// the D131 settings line that replaced its quality and size fields, the D99 plan line and the start outcome.
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const reviewModule = import('../../web/js/workspace/thumbnail-review.mjs');
|
||||
|
||||
test('the concept options map to the job values in the concept order', async () => {
|
||||
const { SCOPE_OPTIONS, QUALITY_OPTIONS, SIZE_LIMIT } = await reviewModule;
|
||||
assert.deepEqual(SCOPE_OPTIONS, [['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]);
|
||||
assert.deepEqual(QUALITY_OPTIONS, [['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]);
|
||||
assert.deepEqual(SIZE_LIMIT, { min: 128, max: 2048, value: 512 });
|
||||
test('the scopes map to the job modes; Convert outdated is the explicit third (#317)', async () => {
|
||||
const { SCOPE_OPTIONS } = await reviewModule;
|
||||
assert.deepEqual(SCOPE_OPTIONS, [['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)'],
|
||||
['convert', 'Convert outdated (removes older formats)']]);
|
||||
});
|
||||
|
||||
test('size outside 128–2048 reports validity: only whole pixels in range are a size', async () => {
|
||||
const { previewSize } = await reviewModule;
|
||||
assert.equal(previewSize('512'), 512);
|
||||
assert.equal(previewSize('128'), 128);
|
||||
assert.equal(previewSize('2048'), 2048);
|
||||
for (const text of ['', ' ', '127', '2049', '4096', '512.5', '-512', 'abc', '1e3', '0x200']) {
|
||||
assert.equal(previewSize(text), null, text);
|
||||
test('D131: the dialog names the server thumbnail settings instead of its own quality and size fields', async () => {
|
||||
const { settingsText, SETTINGS_UNKNOWN } = await reviewModule;
|
||||
assert.equal(settingsText({ format: 'webp', quality: 80, size: 512 }),
|
||||
'Previews are written as WebP · quality 80 · 512 px. Change this in System › Appearance.');
|
||||
assert.equal(settingsText({ format: 'jxl', quality: 90, size: 1024 }),
|
||||
'Previews are written as JPEG XL · quality 90 · 1,024 px. Change this in System › Appearance.');
|
||||
assert.equal(settingsText({ format: 'jpeg', quality: 5, size: 128 }),
|
||||
'Previews are written as JPEG · quality 5 · 128 px. Change this in System › Appearance.');
|
||||
for (const plan of [null, {}, { format: 'png', quality: 80, size: 512 }, { format: 'webp', quality: '80', size: 512 }]) {
|
||||
assert.equal(settingsText(plan), SETTINGS_UNKNOWN);
|
||||
}
|
||||
});
|
||||
|
||||
test('Image loading quality preselects the dialog quality (Q3); anything else is Medium', async () => {
|
||||
const { preselectedQuality } = await reviewModule;
|
||||
assert.equal(preselectedQuality('low'), 'low');
|
||||
assert.equal(preselectedQuality('high'), 'high');
|
||||
assert.equal(preselectedQuality('medium'), 'medium');
|
||||
assert.equal(preselectedQuality(undefined), 'medium');
|
||||
assert.equal(preselectedQuality('original'), 'medium');
|
||||
test('Convert outdated says how many previews in an older format it converts, past the per-batch cap too (#317)', async () => {
|
||||
const { planText } = await reviewModule;
|
||||
assert.equal(planText({ mode: 'convert', targets: 17500, fresh: 2000, outdated: 17500, remaining: 7500 }),
|
||||
'25,000 previews in an older format to convert.');
|
||||
assert.equal(planText({ mode: 'convert', targets: 1, fresh: 0, outdated: 1, remaining: 0 }), '1 preview in an older format to convert.');
|
||||
assert.equal(planText({ mode: 'convert', targets: 0, fresh: 9, outdated: 0, remaining: 0 }), 'No previews in an older format.');
|
||||
});
|
||||
|
||||
test('the plan line counts what a missing-only run generates and what is already up to date (D99)', async () => {
|
||||
@@ -38,7 +39,7 @@ test('the plan line counts what a missing-only run generates and what is already
|
||||
assert.equal(planText({ mode: 'missing', targets: 1, fresh: 1, remaining: 0 }), '1 preview to generate · 1 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 0, fresh: 0, remaining: 0 }), '0 previews to generate · 0 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 10000, fresh: 5, remaining: 2500 }),
|
||||
'10,000 previews to generate · 5 already up to date · 2,500 more on the next run.');
|
||||
'12,500 previews to generate · 5 already up to date.');
|
||||
});
|
||||
|
||||
test('Rebuild all says every preview is regenerated (D99)', async () => {
|
||||
@@ -46,7 +47,7 @@ test('Rebuild all says every preview is regenerated (D99)', async () => {
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 42, fresh: 30, remaining: 0 }), 'Every preview is regenerated: 42 files.');
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 1, fresh: 1, remaining: 0 }), 'Every preview is regenerated: 1 file.');
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 10000, fresh: 9000, remaining: 3 }),
|
||||
'Every preview is regenerated: 10,000 files · 3 more on the next run.');
|
||||
'Every preview is regenerated: 10,003 files.');
|
||||
});
|
||||
|
||||
test('an answer without counts is a failed plan, never zeros', async () => {
|
||||
@@ -64,9 +65,9 @@ test('Start starts the thumbnail job with the chosen options and opens it in Act
|
||||
const calls = [];
|
||||
const navigated = [];
|
||||
const store = { start: async (kind, body) => { calls.push([kind, body]); return { job: { id: 'a'.repeat(32) } }; } };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'rebuild', size: 1024, quality: 'high' },
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'convert' },
|
||||
navigate: hash => navigated.push(hash), notify: () => assert.fail('no toast of its own') });
|
||||
assert.deepEqual(calls, [['thumbnails', { mode: 'rebuild', size: 1024, quality: 'high' }]]);
|
||||
assert.deepEqual(calls, [['thumbnails', { mode: 'convert' }]]);
|
||||
assert.deepEqual(navigated, [`#/activity/${'a'.repeat(32)}`]);
|
||||
});
|
||||
|
||||
@@ -74,7 +75,7 @@ test('a refused start shows the busy link toast and stays', async () => {
|
||||
const { startThumbnails } = await reviewModule;
|
||||
const toasts = [];
|
||||
const store = { start: async () => ({ busy: { id: 'b'.repeat(32), title: 'Scan for metadata' } }) };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing' },
|
||||
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
|
||||
assert.deepEqual(toasts, [['Another job is running: ', { text: 'Scan for metadata', href: `#/activity/${'b'.repeat(32)}` }, '.']]);
|
||||
});
|
||||
@@ -83,11 +84,11 @@ test('a failed start says so; a start after the section left says nothing', asyn
|
||||
const { startThumbnails, START_FAILED } = await reviewModule;
|
||||
const toasts = [];
|
||||
const store = { start: async () => { throw new Error('offline'); } };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing' },
|
||||
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
|
||||
assert.deepEqual(toasts, [START_FAILED]);
|
||||
const gone = new AbortController();
|
||||
gone.abort();
|
||||
await startThumbnails({ store, scope: { signal: gone.signal }, options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
await startThumbnails({ store, scope: { signal: gone.signal }, options: { mode: 'missing' },
|
||||
navigate: () => assert.fail('stays'), notify: () => assert.fail('silent') });
|
||||
});
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
// #317: JPEG XL thumbnails in browsers that cannot decode them. The workspace decodes a tiny embedded JPEG XL once;
|
||||
// where it fails, cards and the inspector cover use the original instead, and any thumbnail <img> that fails to load
|
||||
// falls back to the original. A thumbnail is only a placeholder: the original is what every other view uses.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
const support = await import('../../web/js/workspace/thumbnail-support.mjs');
|
||||
|
||||
/** A fake Image whose decode answers `ok` once its src is set. */
|
||||
function fakeImage(ok, log) {
|
||||
return () => {
|
||||
const image = {};
|
||||
Object.defineProperty(image, 'src', {
|
||||
set(value) { log.push(value); queueMicrotask(() => (ok ? image.onload?.() : image.onerror?.())); },
|
||||
});
|
||||
return image;
|
||||
};
|
||||
}
|
||||
|
||||
test('a .jxl thumbnail URL is recognised by its path, never its query', () => {
|
||||
assert.equal(support.isJxlUrl('/prompt_manager/images/serve/thumbnails/a_thumb.jxl?v=1&root=0'), true);
|
||||
assert.equal(support.isJxlUrl('/prompt_manager/images/serve/thumbnails/a_thumb.JXL'), true);
|
||||
assert.equal(support.isJxlUrl('/prompt_manager/images/serve/thumbnails/a_thumb.webp?x=.jxl'), false);
|
||||
assert.equal(support.isJxlUrl('/x/a.jxl#frag'), true);
|
||||
for (const value of [null, undefined, '', 42]) assert.equal(support.isJxlUrl(value), false);
|
||||
});
|
||||
|
||||
test('decode support is probed once with the embedded image and remembered', async () => {
|
||||
support.resetJxlSupport();
|
||||
const log = [];
|
||||
assert.equal(support.jxlSupport(), undefined, 'unknown before the probe');
|
||||
const first = support.detectJxlSupport({ createImage: fakeImage(true, log) });
|
||||
const again = support.detectJxlSupport({ createImage: fakeImage(false, log) });
|
||||
assert.equal(await first, true);
|
||||
assert.equal(await again, true);
|
||||
assert.equal(log.length, 1, 'one probe');
|
||||
assert.match(log[0], /^data:image\/jxl;base64,\/woA/);
|
||||
assert.equal(support.jxlSupport(), true);
|
||||
support.resetJxlSupport();
|
||||
assert.equal(await support.detectJxlSupport({ createImage: fakeImage(false, log) }), false);
|
||||
assert.equal(support.jxlSupport(), false);
|
||||
support.resetJxlSupport();
|
||||
assert.equal(await support.detectJxlSupport({ createImage: () => { throw new Error('no Image'); } }), false);
|
||||
support.resetJxlSupport();
|
||||
});
|
||||
|
||||
test('cards use the original for a JPEG XL thumbnail this browser cannot show', () => {
|
||||
const image = { url: '/serve/a.png?root=0', thumbnail_url: '/serve/thumbnails/a_thumb.jxl?root=0' };
|
||||
const webp = { ...image, thumbnail_url: '/serve/thumbnails/a_thumb.webp?root=0' };
|
||||
assert.equal(support.thumbnailSource(image, false), image.url);
|
||||
assert.equal(support.thumbnailSource(image, true), image.thumbnail_url);
|
||||
assert.equal(support.thumbnailSource(image, undefined), image.thumbnail_url, 'unknown: try it, the error falls back');
|
||||
assert.equal(support.thumbnailSource(webp, false), webp.thumbnail_url);
|
||||
assert.equal(support.thumbnailSource({ url: image.url }, false), image.url);
|
||||
assert.equal(support.thumbnailSource({ url: image.url, thumbnail_url: null }, true), image.url);
|
||||
});
|
||||
|
||||
test('a thumbnail that fails to load falls back to the original once', () => {
|
||||
const listeners = {};
|
||||
const node = { src: '/serve/thumbnails/a_thumb.jxl',
|
||||
addEventListener(type, handler, options) { listeners[type] = { handler, options }; } };
|
||||
support.fallBackToOriginal(node, '/serve/a.png');
|
||||
assert.equal(listeners.error.options.once, true);
|
||||
listeners.error.handler();
|
||||
assert.equal(node.src, '/serve/a.png');
|
||||
const original = { src: '/serve/a.png', addEventListener: () => assert.fail('no listener for the original itself') };
|
||||
support.fallBackToOriginal(original, '/serve/a.png');
|
||||
const listen = [];
|
||||
const scoped = { src: '/serve/thumbnails/b_thumb.webp', isConnected: true };
|
||||
support.fallBackToOriginal(scoped, '/serve/b.png', (target, type, handler, options) => listen.push([target, type, handler, options]));
|
||||
assert.equal(listen[0][1], 'error');
|
||||
listen[0][2]();
|
||||
assert.equal(scoped.src, '/serve/b.png');
|
||||
});
|
||||
|
||||
test('a video card shows its thumbnail only when this browser can show it (review M2)', () => {
|
||||
const video = { is_video: true, url: '/serve/c.mp4?root=0', thumbnail_url: '/serve/thumbnails/c_thumb.jxl?root=0' };
|
||||
assert.equal(support.showsThumbnail(video, false), false, 'JPEG XL unsupported: the video keeps its own element');
|
||||
assert.equal(support.showsThumbnail(video, true), true);
|
||||
assert.equal(support.showsThumbnail(video, undefined), true, 'unknown: try it, the error swaps');
|
||||
assert.equal(support.showsThumbnail({ ...video, thumbnail_url: '/serve/thumbnails/c_thumb.webp' }, false), true);
|
||||
assert.equal(support.showsThumbnail({ ...video, thumbnail_url: null }, true), false);
|
||||
assert.equal(support.showsThumbnail({ url: '/serve/a.png', thumbnail_url: '/serve/thumbnails/a_thumb.jxl' }, false), true,
|
||||
'an image always shows something: its original');
|
||||
});
|
||||
|
||||
test('a video thumbnail that fails to load is replaced once, never left broken', () => {
|
||||
const listeners = {};
|
||||
let replaced = null;
|
||||
const node = { addEventListener(type, handler, options) { listeners[type] = { handler, options }; },
|
||||
replaceWith(next) { replaced = next; } };
|
||||
support.replaceOnError(node, () => 'label');
|
||||
assert.equal(listeners.error.options.once, true);
|
||||
listeners.error.handler();
|
||||
assert.equal(replaced, 'label');
|
||||
const listen = [];
|
||||
support.replaceOnError(node, () => 'other', (...args) => listen.push(args));
|
||||
assert.equal(listen[0][1], 'error');
|
||||
listen[0][2]();
|
||||
assert.equal(replaced, 'other');
|
||||
});
|
||||
@@ -79,6 +79,8 @@ class AdminAPITestCase(AioHTTPTestCase):
|
||||
|
||||
async def tearDownAsync(self):
|
||||
await super().tearDownAsync() # closes the aiohttp test client
|
||||
# The automatic preview worker writes into the temp tree: stop it first.
|
||||
self.api.close_auto_thumbnails()
|
||||
GalleryConfig.MONITORING_DIRECTORIES = self._orig_gallery_dirs
|
||||
PromptManagerConfig.update_config(self._orig_pm_config)
|
||||
for key, value in self._orig_env.items():
|
||||
@@ -1326,6 +1328,168 @@ class TestSettingsMisc(AdminAPITestCase):
|
||||
self.assertEqual(resp.status, 500)
|
||||
|
||||
|
||||
class TestThumbnailSettings(AdminAPITestCase):
|
||||
"""#317: thumbnail format, quality and size limit are server settings."""
|
||||
|
||||
URL = "/prompt_manager/settings"
|
||||
|
||||
def _encoder(self, name):
|
||||
from py.api import thumbnail_encode
|
||||
|
||||
return patch.object(thumbnail_encode, "detect_jxl_encoder", return_value=name)
|
||||
|
||||
async def test_get_reports_the_settings_and_the_encoders(self):
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
with self._encoder(None):
|
||||
body = await (await self.client.request("GET", self.URL)).json()
|
||||
settings = body["settings"]
|
||||
self.assertEqual(
|
||||
(
|
||||
settings["thumbnail_format"],
|
||||
settings["thumbnail_quality"],
|
||||
settings["thumbnail_edge"],
|
||||
),
|
||||
("webp", 80, 512),
|
||||
)
|
||||
jxl = body["thumbnail_encoders"]["jxl"]
|
||||
self.assertEqual((jxl["available"], jxl["encoder"]), (False, None))
|
||||
self.assertIn("pillow-jxl-plugin", jxl["reason"])
|
||||
with self._encoder("cjxl"):
|
||||
body = await (await self.client.request("GET", self.URL)).json()
|
||||
self.assertEqual(
|
||||
body["thumbnail_encoders"]["jxl"],
|
||||
{"available": True, "encoder": "cjxl", "reason": None},
|
||||
)
|
||||
|
||||
async def test_save_commits_the_thumbnails_section(self):
|
||||
self._write_config({"custom": {"keep": 1}})
|
||||
resp = await self.client.request(
|
||||
"POST",
|
||||
self.URL,
|
||||
json={
|
||||
"thumbnail_format": "jpeg",
|
||||
"thumbnail_quality": 60,
|
||||
"thumbnail_edge": 300,
|
||||
},
|
||||
)
|
||||
self.assertEqual(resp.status, 200)
|
||||
saved = self._read_config()
|
||||
self.assertEqual(
|
||||
saved["thumbnails"], {"format": "jpeg", "quality": 60, "edge": 300}
|
||||
)
|
||||
self.assertEqual(saved["custom"], {"keep": 1})
|
||||
self.assertEqual(
|
||||
(
|
||||
PromptManagerConfig.THUMBNAIL_FORMAT,
|
||||
PromptManagerConfig.THUMBNAIL_QUALITY,
|
||||
PromptManagerConfig.THUMBNAIL_EDGE,
|
||||
),
|
||||
("jpeg", 60, 300),
|
||||
)
|
||||
|
||||
async def test_bad_values_are_refused_and_change_nothing(self):
|
||||
before = PromptManagerConfig.get_config()["thumbnails"]
|
||||
for payload in (
|
||||
{"thumbnail_format": "png"},
|
||||
{"thumbnail_quality": 0},
|
||||
{"thumbnail_quality": "80"},
|
||||
{"thumbnail_edge": 4096},
|
||||
{"thumbnail_edge": True},
|
||||
):
|
||||
resp = await self.client.request("POST", self.URL, json=payload)
|
||||
self.assertEqual(resp.status, 400, payload)
|
||||
self.assertEqual(PromptManagerConfig.get_config()["thumbnails"], before)
|
||||
self.assertFalse(os.path.exists(self.config_path))
|
||||
|
||||
async def test_jpeg_xl_without_an_encoder_is_refused_with_a_code(self):
|
||||
with self._encoder(None):
|
||||
resp = await self.client.request(
|
||||
"POST", self.URL, json={"thumbnail_format": "jxl", "result_timeout": 7}
|
||||
)
|
||||
body = await resp.json()
|
||||
self.assertEqual((resp.status, body["code"]), (400, "jxl-unavailable"))
|
||||
self.assertIn("pillow-jxl-plugin", body["error"])
|
||||
self.assertNotEqual(PromptManagerConfig.THUMBNAIL_FORMAT, "jxl")
|
||||
self.assertNotEqual(PromptManagerConfig.RESULT_TIMEOUT, 7)
|
||||
self.assertFalse(os.path.exists(self.config_path))
|
||||
|
||||
async def test_jpeg_xl_with_an_encoder_is_saved(self):
|
||||
with self._encoder("ffmpeg"):
|
||||
resp = await self.client.request(
|
||||
"POST", self.URL, json={"thumbnail_format": "jxl"}
|
||||
)
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(self._read_config()["thumbnails"]["format"], "jxl")
|
||||
self.assertEqual(PromptManagerConfig.THUMBNAIL_FORMAT, "jxl")
|
||||
|
||||
async def _running_job(self, mode):
|
||||
from py.api.jobs import THUMBNAILS_JOB_KIND
|
||||
from py.api.thumbnail_convert import CONVERT_JOB_TITLE
|
||||
|
||||
gate = asyncio.Event()
|
||||
|
||||
async def runner(job):
|
||||
await gate.wait()
|
||||
job.finish("finished")
|
||||
|
||||
job = self.api._jobs.start(
|
||||
THUMBNAILS_JOB_KIND,
|
||||
CONVERT_JOB_TITLE if mode == "convert" else "Thumbnail generation",
|
||||
admission="guarded",
|
||||
scope={"mode": mode},
|
||||
runner=runner,
|
||||
)
|
||||
|
||||
async def release():
|
||||
gate.set()
|
||||
await asyncio.wait_for(job.task, 5)
|
||||
|
||||
self.addAsyncCleanup(release)
|
||||
return job
|
||||
|
||||
async def test_thumbnail_changes_wait_for_a_running_conversion(self):
|
||||
# Review I4: a convert job's format is fixed; changing the format,
|
||||
# quality or size limit under it is refused with the job named.
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
job = await self._running_job("convert")
|
||||
for change in (
|
||||
{"thumbnail_format": "jpeg"},
|
||||
{"thumbnail_quality": 70},
|
||||
{"thumbnail_edge": 300},
|
||||
):
|
||||
resp = await self.client.request("POST", self.URL, json=change)
|
||||
body = await resp.json()
|
||||
self.assertEqual(resp.status, 409, change)
|
||||
self.assertEqual(body["code"], "busy")
|
||||
self.assertEqual(body["running"]["id"], job.id)
|
||||
self.assertIn("Convert outdated thumbnails", body["error"])
|
||||
self.assertIn("thumbnail", body["error"].lower())
|
||||
self.assertEqual(
|
||||
(
|
||||
PromptManagerConfig.THUMBNAIL_FORMAT,
|
||||
PromptManagerConfig.THUMBNAIL_QUALITY,
|
||||
PromptManagerConfig.THUMBNAIL_EDGE,
|
||||
),
|
||||
("webp", 80, 512),
|
||||
)
|
||||
# The same values, or other settings, still save.
|
||||
resp = await self.client.request(
|
||||
"POST", self.URL, json={"thumbnail_format": "webp", "thumbnail_quality": 80}
|
||||
)
|
||||
self.assertEqual(resp.status, 200)
|
||||
|
||||
async def test_a_missing_previews_job_does_not_block_thumbnail_changes(self):
|
||||
await self._running_job("missing")
|
||||
resp = await self.client.request(
|
||||
"POST", self.URL, json={"thumbnail_format": "jpeg"}
|
||||
)
|
||||
self.assertEqual(resp.status, 200)
|
||||
|
||||
|
||||
class TestDiagnostics(AdminAPITestCase):
|
||||
|
||||
def _install_monitor(self, monitor):
|
||||
|
||||
@@ -131,6 +131,8 @@ class AutotagAPITestCase(AioHTTPTestCase):
|
||||
|
||||
async def tearDownAsync(self):
|
||||
await super().tearDownAsync() # closes the aiohttp test client
|
||||
# The automatic preview worker writes into the temp tree: stop it first.
|
||||
self.api.close_auto_thumbnails()
|
||||
self._service_patch.stop()
|
||||
if model_gate is not None:
|
||||
leaked = model_gate.holder()
|
||||
@@ -1131,10 +1133,17 @@ class TestScanOutputDir(AutotagAPITestCase):
|
||||
self.assertEqual(data["count"], 2)
|
||||
self.assertIs(data["truncated"], False)
|
||||
self.assertIsNone(data["images"][0]["thumbnail_url"])
|
||||
# An older release's preview keeps its spelling; a case-insensitive
|
||||
# disk (Windows, macOS) finds it under the ".jpg" lookup first (#317).
|
||||
# Either way the URL names that file and is served.
|
||||
thumb_url = data["images"][1]["thumbnail_url"]
|
||||
self.assertRegex(
|
||||
data["images"][1]["thumbnail_url"],
|
||||
r"^/prompt_manager/images/serve/thumbnails/sub/b_thumb\.JPG\?v=\d+$",
|
||||
thumb_url,
|
||||
r"^/prompt_manager/images/serve/thumbnails/sub/b_thumb\.(JPG|jpg)\?v=\d+$",
|
||||
)
|
||||
served = await self.client.request("GET", thumb_url)
|
||||
self.assertEqual(served.status, 200)
|
||||
self.assertEqual(await served.read(), b"t")
|
||||
self.assertEqual(
|
||||
data["images"][1]["url"], "/prompt_manager/images/serve/sub/b.JPG"
|
||||
)
|
||||
|
||||
@@ -62,6 +62,8 @@ class BackupRestoreTestCase(AioHTTPTestCase):
|
||||
|
||||
async def tearDownAsync(self):
|
||||
await super().tearDownAsync() # closes the aiohttp test client
|
||||
# The automatic preview worker writes into the temp tree: stop it first.
|
||||
self.api.close_auto_thumbnails()
|
||||
PromptModel.close_all_instances() # restore may have swapped api.db
|
||||
assert_closed(self, self.tmpdir)
|
||||
for name in os.listdir(self.tmpdir):
|
||||
|
||||
@@ -384,6 +384,8 @@ class CoreRoutesTestCase(AioHTTPTestCase):
|
||||
|
||||
async def tearDownAsync(self):
|
||||
await super().tearDownAsync() # closes the aiohttp test client
|
||||
# The automatic preview worker writes into the temp tree: stop it first.
|
||||
self.api.close_auto_thumbnails()
|
||||
self.api.db.close_all()
|
||||
assert_closed(self, self.tmpdir)
|
||||
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
||||
|
||||
@@ -14,6 +14,7 @@ from unittest import skipUnless
|
||||
from unittest.mock import patch
|
||||
|
||||
from aiohttp import web
|
||||
from PIL import Image
|
||||
from aiohttp.test_utils import make_mocked_request
|
||||
|
||||
from tests.test_api_images import ImageAPITestCase, make_png
|
||||
@@ -401,7 +402,7 @@ class HistoryOutputAPITests(ImageAPITestCase):
|
||||
self.assertEqual(response.status, 400)
|
||||
await response.read()
|
||||
|
||||
async def test_media_ranges_head_and_existing_thumbnail_without_generation(self):
|
||||
async def test_media_ranges_head_and_a_preview_made_on_first_view(self):
|
||||
path = self.output_dir / "clip.mp4"
|
||||
payload = bytes(range(256)) * 600
|
||||
path.write_bytes(payload)
|
||||
@@ -430,9 +431,15 @@ class HistoryOutputAPITests(ImageAPITestCase):
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(int(response.headers["Content-Length"]), len(payload))
|
||||
self.assertEqual(await response.read(), b"")
|
||||
# A missing preview is made on first view (#249): an undecodable video
|
||||
# gets the placeholder frame, never the whole file, written in the
|
||||
# thumbnail format (#317, WebP by default).
|
||||
preview = self.output_dir / "thumbnails" / "clip_thumb.webp"
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assertEqual(await response.read(), payload)
|
||||
preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.jpg")
|
||||
self.assertEqual(response.headers["Content-Type"], "image/webp")
|
||||
self.assertEqual(await response.read(), preview.read_bytes())
|
||||
self.assertLess(preview.stat().st_size, len(payload))
|
||||
make_png(preview)
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assertEqual(await response.read(), preview.read_bytes())
|
||||
|
||||
@@ -1111,6 +1118,172 @@ class HistoryOutputAPITests(ImageAPITestCase):
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assert_error(response, await response.json(), 403, "not-authorized")
|
||||
|
||||
async def test_missing_preview_is_never_made_through_a_linked_out_folder(self):
|
||||
original = make_png(self.output_dir / "image.png", size=(64, 64))
|
||||
elsewhere = self.temp_dir / "elsewhere"
|
||||
elsewhere.mkdir()
|
||||
(self.output_dir / "thumbnails").symlink_to(elsewhere, target_is_directory=True)
|
||||
start = self.capture([self.ref()])
|
||||
descriptor = await self.media(start)
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(await response.read(), original.read_bytes())
|
||||
self.assertEqual(os.listdir(elsewhere), [])
|
||||
|
||||
async def test_missing_preview_is_never_made_through_a_linked_sub_folder(self):
|
||||
# M6: every link segment of the preview's folder is checked.
|
||||
original = make_png(self.output_dir / "sub" / "image.png", size=(64, 64))
|
||||
elsewhere = self.temp_dir / "elsewhere"
|
||||
elsewhere.mkdir()
|
||||
(self.output_dir / "thumbnails").mkdir()
|
||||
(self.output_dir / "thumbnails" / "sub").symlink_to(
|
||||
elsewhere, target_is_directory=True
|
||||
)
|
||||
start = self.capture([self.ref(subfolder="sub")])
|
||||
descriptor = await self.media(start)
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(await response.read(), original.read_bytes())
|
||||
self.assertEqual(os.listdir(elsewhere), [])
|
||||
|
||||
async def test_a_video_preview_that_cannot_be_made_is_the_placeholder(self):
|
||||
# I1: never the video bytes inside an <img>.
|
||||
from py.api.jobs import RESTORE_JOB_KIND
|
||||
|
||||
(self.output_dir / "clip.webm").write_bytes(b"AV1 stream no decoder here")
|
||||
start = self.capture([self.ref(filename="clip.webm")])
|
||||
descriptor = await self.media(start)
|
||||
gate = asyncio.Event()
|
||||
|
||||
async def runner(job):
|
||||
await gate.wait()
|
||||
|
||||
job = self.api._jobs.start(
|
||||
RESTORE_JOB_KIND,
|
||||
"restore",
|
||||
admission="records",
|
||||
cancellable=False,
|
||||
runner=runner,
|
||||
)
|
||||
try:
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(response.headers["Content-Type"], "image/jpeg")
|
||||
self.assertTrue((await response.read()).startswith(b"\xff\xd8"))
|
||||
finally:
|
||||
gate.set()
|
||||
await asyncio.wait_for(job.task, 5)
|
||||
self.assertFalse((self.output_dir / "thumbnails").exists())
|
||||
|
||||
async def test_the_cover_is_the_preview_the_images_listing_shows(self):
|
||||
# Review M5: JPEG XL configured with no encoder left writes WebP; the
|
||||
# history cover prefers that WebP, like the Images listing.
|
||||
from py.api import thumbnail_encode
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
make_png(self.output_dir / "image.png")
|
||||
folder = self.output_dir / "thumbnails"
|
||||
folder.mkdir()
|
||||
(folder / "image_thumb.jxl").write_bytes(b"\xff\x0a stale jxl")
|
||||
webp = folder / "image_thumb.webp"
|
||||
Image.new("RGB", (16, 16)).save(webp, "WEBP")
|
||||
start = self.capture([self.ref()])
|
||||
descriptor = await self.media(start)
|
||||
with (
|
||||
patch.object(PromptManagerConfig, "THUMBNAIL_FORMAT", "jxl"),
|
||||
patch.object(thumbnail_encode, "detect_jxl_encoder", return_value=None),
|
||||
):
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
body = await response.read()
|
||||
self.assertEqual(response.headers["Content-Type"], "image/webp")
|
||||
self.assertEqual(body, webp.read_bytes())
|
||||
|
||||
async def _jxl_cover(self, filename, accept):
|
||||
from py.api import thumbnail_encode
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
folder = self.output_dir / "thumbnails"
|
||||
folder.mkdir(exist_ok=True)
|
||||
stem = filename.rsplit(".", 1)[0]
|
||||
jxl = folder / f"{stem}_thumb.jxl"
|
||||
jxl.write_bytes(b"\xff\x0a jxl preview")
|
||||
start = self.capture([self.ref(filename=filename)])
|
||||
descriptor = await self.media(start)
|
||||
headers = {"Accept": accept} if accept else {}
|
||||
with (
|
||||
patch.object(PromptManagerConfig, "THUMBNAIL_FORMAT", "jxl"),
|
||||
patch.object(thumbnail_encode, "detect_jxl_encoder", return_value="cjxl"),
|
||||
):
|
||||
response = await self.current(descriptor["thumbnail_url"], headers=headers)
|
||||
body = await response.read()
|
||||
return response, body, jxl
|
||||
|
||||
async def test_a_jxl_cover_is_served_only_to_a_browser_that_accepts_it(self):
|
||||
# Review M3: History's cover URL names no format, so the server picks:
|
||||
# a browser that does not list image/jxl gets the original image.
|
||||
original = make_png(self.output_dir / "image.png")
|
||||
response, body, jxl = await self._jxl_cover(
|
||||
"image.png", "image/avif,image/webp,image/*,*/*;q=0.8"
|
||||
)
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(body, original.read_bytes())
|
||||
self.assertEqual(response.headers["Content-Type"], "image/png")
|
||||
self.assertTrue(jxl.exists())
|
||||
|
||||
async def test_a_browser_that_lists_image_jxl_gets_the_jxl_cover(self):
|
||||
make_png(self.output_dir / "image.png")
|
||||
response, body, jxl = await self._jxl_cover(
|
||||
"image.png", "image/webp,image/avif,image/jxl,image/*;q=0.8"
|
||||
)
|
||||
self.assertEqual(response.headers["Content-Type"], "image/jxl")
|
||||
self.assertEqual(body, jxl.read_bytes())
|
||||
|
||||
async def test_the_jxl_flag_selects_the_jxl_cover_without_accept(self):
|
||||
# M3 follow-up: a browser that decodes JPEG XL but does not list it in
|
||||
# Accept (Firefox) says so with jxl=1 on the cover URL.
|
||||
make_png(self.output_dir / "image.png")
|
||||
folder = self.output_dir / "thumbnails"
|
||||
folder.mkdir(exist_ok=True)
|
||||
jxl = folder / "image_thumb.jxl"
|
||||
jxl.write_bytes(b"\xff\x0a jxl preview")
|
||||
start = self.capture([self.ref()])
|
||||
descriptor = await self.media(start)
|
||||
from py.api import thumbnail_encode
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
with (
|
||||
patch.object(PromptManagerConfig, "THUMBNAIL_FORMAT", "jxl"),
|
||||
patch.object(thumbnail_encode, "detect_jxl_encoder", return_value="cjxl"),
|
||||
):
|
||||
url = descriptor["thumbnail_url"]
|
||||
response = await self.current(url + "&jxl=1", headers={"Accept": "*/*"})
|
||||
self.assertEqual(response.headers["Content-Type"], "image/jxl")
|
||||
self.assertEqual(response.headers["Cache-Control"], "no-store")
|
||||
self.assertEqual(await response.read(), jxl.read_bytes())
|
||||
response = await self.current(url + "&jxl=0", headers={"Accept": "*/*"})
|
||||
self.assertEqual(response.headers["Content-Type"], "image/png")
|
||||
await response.read()
|
||||
for bad in ("&jxl=true", "&jxl=", "&jxl=1&jxl=1", "&jxl=2"):
|
||||
response = await self.current(url + bad)
|
||||
self.assertEqual(response.status, 400, bad)
|
||||
await response.read()
|
||||
|
||||
async def test_a_video_jxl_cover_is_the_placeholder_without_jxl_support(self):
|
||||
(self.output_dir / "clip.webm").write_bytes(b"AV1 stream no decoder here")
|
||||
response, body, _jxl = await self._jxl_cover("clip.webm", "image/*,*/*")
|
||||
self.assertEqual(response.headers["Content-Type"], "image/jpeg")
|
||||
self.assertTrue(body.startswith(b"\xff\xd8"))
|
||||
|
||||
async def test_a_preview_that_cannot_be_made_serves_the_original(self):
|
||||
broken = self.output_dir / "image.png"
|
||||
broken.write_bytes(b"\x89PNG\r\n\x1a\n not an image")
|
||||
start = self.capture([self.ref()])
|
||||
descriptor = await self.media(start)
|
||||
response = await self.current(descriptor["thumbnail_url"])
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(await response.read(), broken.read_bytes())
|
||||
self.assertFalse((self.output_dir / "thumbnails" / "image_thumb.png").exists())
|
||||
|
||||
async def test_disappearance_before_open_is_missing_and_closes_handle(
|
||||
self,
|
||||
):
|
||||
|
||||
+132
-58
@@ -33,7 +33,8 @@ from aiohttp.test_utils import AioHTTPTestCase # noqa: E402
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
from database.operations import PromptDatabase # noqa: E402
|
||||
import py.api.images as images_module # noqa: E402
|
||||
import py.api.images as images_module
|
||||
from py.api.thumbnail_settings import ThumbnailSettings # noqa: E402
|
||||
from py.api import PromptManagerAPI # noqa: E402
|
||||
from py.api.prompts import MAX_PAGE_LIMIT # noqa: E402
|
||||
from py.config import GalleryConfig # noqa: E402
|
||||
@@ -80,6 +81,8 @@ class ImageAPITestCase(AioHTTPTestCase):
|
||||
|
||||
async def tearDownAsync(self):
|
||||
await super().tearDownAsync() # closes the aiohttp test client
|
||||
# The automatic preview worker writes into the temp tree: stop it first.
|
||||
self.api.close_auto_thumbnails()
|
||||
self._anchor_patch.stop()
|
||||
# Windows refuses to unlink a database that still has open handles.
|
||||
self.api.db.close_all()
|
||||
@@ -395,7 +398,7 @@ class ThumbnailTestCase(ImageAPITestCase):
|
||||
self.api._run_in_executor = spy
|
||||
return calls
|
||||
|
||||
def _thumb_path(self, name, suffix=".png"):
|
||||
def _thumb_path(self, name, suffix=".webp"):
|
||||
return self.output_dir / "thumbnails" / f"{name}_thumb{suffix}"
|
||||
|
||||
|
||||
@@ -494,7 +497,8 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
|
||||
self.assertNotIn("_generate_one", calls)
|
||||
self.assertIn("_plan_gallery_thumbnails", calls)
|
||||
|
||||
async def test_capped_run_reports_remaining_and_the_next_run_finishes(self):
|
||||
async def test_a_run_past_the_batch_cap_goes_on_to_the_end(self):
|
||||
# The cap bounds one batch; the run plans again and finishes (#317).
|
||||
for name in ("a", "b", "c"):
|
||||
make_png(self.output_dir / f"{name}.png", size=(64, 64))
|
||||
|
||||
@@ -507,18 +511,19 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
|
||||
)
|
||||
|
||||
names = [e for e, _ in first]
|
||||
self.assertEqual(names.count("progress"), 2) # only pending files are work
|
||||
self.assertEqual(names.count("progress"), 3) # only pending files are work
|
||||
self.assertEqual(dict(first)["start"]["total_images"], 3)
|
||||
self.assertEqual(
|
||||
[d["total_images"] for e, d in first if e == "progress"], [3] * 3
|
||||
)
|
||||
done = dict(first)["complete"]
|
||||
self.assertEqual(done["count"], 2)
|
||||
self.assertEqual(done["remaining"], 1)
|
||||
self.assertEqual(done["total_images"], 3)
|
||||
self.assertIn("1 more", done["message"])
|
||||
done = dict(second)["complete"]
|
||||
self.assertEqual(done["count"], 1)
|
||||
self.assertEqual(done["skipped"], 2)
|
||||
self.assertEqual(done["count"], 3)
|
||||
self.assertEqual(done["remaining"], 0)
|
||||
self.assertEqual(done["total_images"], 3)
|
||||
self.assertNotIn("more", done["message"])
|
||||
done = dict(second)["complete"]
|
||||
self.assertEqual(done["count"], 0)
|
||||
self.assertEqual(done["skipped"], 3)
|
||||
|
||||
async def test_blocking_endpoint_reports_remaining(self):
|
||||
for name in ("a", "b", "c"):
|
||||
@@ -586,14 +591,15 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
|
||||
events = parse_sse(await resp.text())
|
||||
self.assertEqual(events[-1][0], "error")
|
||||
|
||||
async def test_unknown_quality_falls_back_to_medium(self):
|
||||
make_png(self.output_dir / "one.png", size=(400, 400))
|
||||
async def test_any_quality_preset_uses_the_server_size_limit(self):
|
||||
# #317 R3: the server's thumbnail settings size every preview.
|
||||
make_png(self.output_dir / "one.png", size=(1000, 1000))
|
||||
|
||||
resp = await self.client.request("POST", f"{self.PROGRESS_URL}?quality=huge")
|
||||
await resp.text()
|
||||
|
||||
with Image.open(self._thumb_path("one")) as thumb:
|
||||
self.assertEqual(max(thumb.size), 300)
|
||||
self.assertEqual(max(thumb.size), 512)
|
||||
|
||||
|
||||
class TestGenerateThumbnailsPost(ThumbnailTestCase):
|
||||
@@ -908,7 +914,8 @@ class TestThumbnailFilesAreNeverSeenHalfWritten(ThumbnailTestCase):
|
||||
make_png(self.output_dir / "a.png", size=(64, 64))
|
||||
thumbs = self.output_dir / "thumbnails"
|
||||
thumbs.mkdir()
|
||||
dst = thumbs / "a_thumb.png"
|
||||
dst = thumbs / "a_thumb.webp"
|
||||
settings = ThumbnailSettings("webp", 80, 32)
|
||||
|
||||
def crash_mid_write(self_img, fp, *args, **kwargs):
|
||||
# Simulates an interrupted encoder: bytes on disk, then failure.
|
||||
@@ -918,14 +925,14 @@ class TestThumbnailFilesAreNeverSeenHalfWritten(ThumbnailTestCase):
|
||||
with patch.object(Image.Image, "save", crash_mid_write):
|
||||
with self.assertRaises(OSError):
|
||||
images_module._write_image_thumbnail(
|
||||
self.output_dir / "a.png", dst, (32, 32)
|
||||
self.output_dir / "a.png", dst, settings
|
||||
)
|
||||
self.assertFalse(dst.exists(), "a half-written thumbnail was left in place")
|
||||
self.assertEqual(os.listdir(thumbs), [])
|
||||
|
||||
images_module._write_image_thumbnail(self.output_dir / "a.png", dst, (32, 32))
|
||||
images_module._write_image_thumbnail(self.output_dir / "a.png", dst, settings)
|
||||
self.assertTrue(dst.is_file())
|
||||
self.assertEqual(os.listdir(thumbs), ["a_thumb.png"])
|
||||
self.assertEqual(os.listdir(thumbs), ["a_thumb.webp"])
|
||||
|
||||
def test_thumbnail_url_is_versioned_by_its_mtime(self):
|
||||
import os
|
||||
@@ -933,7 +940,9 @@ class TestThumbnailFilesAreNeverSeenHalfWritten(ThumbnailTestCase):
|
||||
|
||||
make_png(self.output_dir / "a.png", size=(64, 64))
|
||||
self.api._generate_thumbnails_sync(
|
||||
self.output_dir, self.output_dir / "thumbnails", (32, 32)
|
||||
self.output_dir,
|
||||
self.output_dir / "thumbnails",
|
||||
ThumbnailSettings("webp", 80, 32),
|
||||
)
|
||||
thumb = self._thumb_path("a")
|
||||
|
||||
@@ -942,7 +951,7 @@ class TestThumbnailFilesAreNeverSeenHalfWritten(ThumbnailTestCase):
|
||||
)
|
||||
self.assertRegex(
|
||||
entry["thumbnail_url"],
|
||||
r"^/prompt_manager/images/serve/thumbnails/a_thumb\.png\?v=\d+&root=0$",
|
||||
r"^/prompt_manager/images/serve/thumbnails/a_thumb\.webp\?v=\d+&root=0$",
|
||||
)
|
||||
first = int(re.search(r"v=(\d+)", entry["thumbnail_url"]).group(1))
|
||||
self.assertEqual(first, int(thumb.stat().st_mtime))
|
||||
@@ -958,7 +967,9 @@ class TestThumbnailFilesAreNeverSeenHalfWritten(ThumbnailTestCase):
|
||||
async def test_versioned_thumbnail_url_is_served(self):
|
||||
make_png(self.output_dir / "a.png", size=(64, 64))
|
||||
self.api._generate_thumbnails_sync(
|
||||
self.output_dir, self.output_dir / "thumbnails", (32, 32)
|
||||
self.output_dir,
|
||||
self.output_dir / "thumbnails",
|
||||
ThumbnailSettings("webp", 80, 32),
|
||||
)
|
||||
entry = images_module._output_image_entry(
|
||||
self.output_dir / "a.png", self.output_dir, 0
|
||||
@@ -967,7 +978,7 @@ class TestThumbnailFilesAreNeverSeenHalfWritten(ThumbnailTestCase):
|
||||
resp = await self.client.request("GET", entry["thumbnail_url"])
|
||||
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(resp.headers["Content-Type"], "image/png")
|
||||
self.assertEqual(resp.headers["Content-Type"], "image/webp")
|
||||
|
||||
async def test_missing_image_404_is_not_cacheable(self):
|
||||
resp = await self.client.request(
|
||||
@@ -1022,7 +1033,9 @@ class TestMediaScanCaps(ImageAPITestCase):
|
||||
for name in ("a", "b", "c"):
|
||||
make_png(self.output_dir / f"{name}.png")
|
||||
self.api._generate_thumbnails_sync(
|
||||
self.output_dir, self.output_dir / "thumbnails", (64, 64)
|
||||
self.output_dir,
|
||||
self.output_dir / "thumbnails",
|
||||
ThumbnailSettings("webp", 80, 64),
|
||||
)
|
||||
for name in ("d", "e", "f"):
|
||||
make_png(self.output_dir / f"{name}.png")
|
||||
@@ -1081,11 +1094,6 @@ class TestMediaScanCaps(ImageAPITestCase):
|
||||
|
||||
self.assertEqual(targets, [])
|
||||
|
||||
def test_thumbnail_size_ignores_non_string_quality(self):
|
||||
self.assertEqual(self.api._thumbnail_size({"w": 99999}), (300, 300))
|
||||
self.assertEqual(self.api._thumbnail_size(None), (300, 300))
|
||||
self.assertEqual(self.api._thumbnail_size("low"), (150, 150))
|
||||
|
||||
|
||||
class TestJsonBodyCaps(ImageAPITestCase):
|
||||
"""JSON bodies are read in chunks and rejected past MAX_JSON_BODY_BYTES."""
|
||||
@@ -1456,7 +1464,7 @@ class TestGalleryListing(ImageRouteCoverageCase):
|
||||
|
||||
class TestThumbnailUnits(ImageRouteCoverageCase):
|
||||
|
||||
SIZE = (64, 64)
|
||||
SIZE = ThumbnailSettings("webp", 80, 64)
|
||||
|
||||
def test_generate_one_reports_vanished_source(self):
|
||||
result = self.api._generate_one(
|
||||
@@ -1469,11 +1477,9 @@ class TestThumbnailUnits(ImageRouteCoverageCase):
|
||||
src = self.output_dir / "clip.mp4"
|
||||
src.write_bytes(b"\x00")
|
||||
with patch.object(self.api, "_generate_video_thumbnail", return_value=True):
|
||||
ok = self.api._generate_one(src, self._dst("clip", ".jpg"), self.SIZE, True)
|
||||
ok = self.api._generate_one(src, self._dst("clip"), self.SIZE, True)
|
||||
with patch.object(self.api, "_generate_video_thumbnail", return_value=False):
|
||||
bad = self.api._generate_one(
|
||||
src, self._dst("clip", ".jpg"), self.SIZE, True
|
||||
)
|
||||
bad = self.api._generate_one(src, self._dst("clip"), self.SIZE, True)
|
||||
self.assertEqual((ok["action"], ok["type"]), ("generated", "video"))
|
||||
self.assertEqual(bad["action"], "error")
|
||||
|
||||
@@ -1496,7 +1502,7 @@ class TestThumbnailUnits(ImageRouteCoverageCase):
|
||||
)
|
||||
self.assertEqual(result["count"], 2)
|
||||
self.assertTrue(self._dst("alpha").is_file())
|
||||
self.assertTrue(self._dst("photo", ".jpg").is_file())
|
||||
self.assertTrue(self._dst("photo").is_file())
|
||||
|
||||
async def test_emit_sse_swallows_write_errors(self):
|
||||
response = MagicMock()
|
||||
@@ -1519,14 +1525,15 @@ class TestThumbnailUnits(ImageRouteCoverageCase):
|
||||
)
|
||||
self.assertEqual(status, 500)
|
||||
|
||||
def _dst(self, name, suffix=".png"):
|
||||
def _dst(self, name, suffix=".webp"):
|
||||
return self.output_dir / "thumbnails" / f"{name}_thumb{suffix}"
|
||||
|
||||
|
||||
class TestVideoThumbnailFallbacks(ImageRouteCoverageCase):
|
||||
"""_generate_video_thumbnail degrades: cv2 -> ffmpeg -> placeholder."""
|
||||
|
||||
SIZE = (48, 48)
|
||||
# JPEG keeps these checks format-specific; #317 settings carry the edge.
|
||||
SIZE = ThumbnailSettings("jpeg", 85, 48)
|
||||
|
||||
@property
|
||||
def video(self):
|
||||
@@ -1565,39 +1572,73 @@ class TestVideoThumbnailFallbacks(ImageRouteCoverageCase):
|
||||
with Image.open(self.thumb) as img:
|
||||
self.assertEqual(img.format, "JPEG")
|
||||
|
||||
def test_cv2_cannot_open_or_read(self):
|
||||
with patch.dict(sys.modules, {"cv2": self._fake_cv2(opened=False)}):
|
||||
closed = self.api._generate_video_thumbnail(
|
||||
self.video, self.thumb, self.SIZE
|
||||
)
|
||||
with patch.dict(sys.modules, {"cv2": self._fake_cv2(read_ok=False)}):
|
||||
unread = self.api._generate_video_thumbnail(
|
||||
self.video, self.thumb, self.SIZE
|
||||
)
|
||||
self.assertEqual((closed, unread), (False, False))
|
||||
def test_cv2_cannot_open_or_read_falls_through_to_ffmpeg_then_placeholder(self):
|
||||
# #249 I1: an OpenCV that cannot decode the clip (AV1) no longer ends
|
||||
# the run; ffmpeg and then the placeholder are tried.
|
||||
for fake in (self._fake_cv2(opened=False), self._fake_cv2(read_ok=False)):
|
||||
self.thumb.unlink(missing_ok=True)
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": fake}),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("ffmpeg")),
|
||||
):
|
||||
ok = self.api._generate_video_thumbnail(
|
||||
self.video, self.thumb, self.SIZE
|
||||
)
|
||||
self.assertTrue(ok)
|
||||
with Image.open(self.thumb) as img:
|
||||
self.assertEqual((img.format, img.size), ("JPEG", self.SIZE.size))
|
||||
|
||||
def test_cv2_crash_is_contained(self):
|
||||
cv2 = MagicMock()
|
||||
cv2.VideoCapture.side_effect = RuntimeError("driver crash")
|
||||
with patch.dict(sys.modules, {"cv2": cv2}):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": cv2}),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("ffmpeg")),
|
||||
):
|
||||
ok = self.api._generate_video_thumbnail(self.video, self.thumb, self.SIZE)
|
||||
self.assertTrue(ok, "the placeholder is written")
|
||||
|
||||
def test_cv2_failure_keeps_an_existing_preview_on_rebuild(self):
|
||||
make_png(self.thumb)
|
||||
before = self.thumb.read_bytes()
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": self._fake_cv2(opened=False)}),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("ffmpeg")),
|
||||
):
|
||||
ok = self.api._generate_video_thumbnail(
|
||||
self.video, self.thumb, self.SIZE, keep_existing=True
|
||||
)
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual(self.thumb.read_bytes(), before)
|
||||
|
||||
def test_ffmpeg_absent_yields_placeholder(self):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("ffmpeg")),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("ffmpeg")),
|
||||
):
|
||||
ok = self.api._generate_video_thumbnail(self.video, self.thumb, self.SIZE)
|
||||
self.assertTrue(ok)
|
||||
with Image.open(self.thumb) as img:
|
||||
self.assertEqual(img.size, self.SIZE)
|
||||
self.assertEqual(img.size, self.SIZE.size)
|
||||
|
||||
@staticmethod
|
||||
def _process(returncode, stderr=b"", stdout=b""):
|
||||
process = MagicMock(returncode=returncode)
|
||||
process.communicate.return_value = (stdout, stderr)
|
||||
return process
|
||||
|
||||
@staticmethod
|
||||
def _png_frame(size=(96, 64)):
|
||||
import io
|
||||
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", size, (200, 0, 0)).save(buffer, "PNG")
|
||||
return buffer.getvalue()
|
||||
|
||||
def test_ffmpeg_failure_yields_placeholder(self):
|
||||
failed = MagicMock(returncode=1, stderr="no stream")
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", return_value=failed),
|
||||
patch("subprocess.Popen", return_value=self._process(1, b"no stream")),
|
||||
):
|
||||
ok = self.api._generate_video_thumbnail(self.video, self.thumb, self.SIZE)
|
||||
self.assertTrue(ok)
|
||||
@@ -1606,16 +1647,46 @@ class TestVideoThumbnailFallbacks(ImageRouteCoverageCase):
|
||||
def test_ffmpeg_success(self):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", return_value=MagicMock(returncode=0)) as run,
|
||||
patch(
|
||||
"subprocess.Popen",
|
||||
return_value=self._process(0, stdout=self._png_frame()),
|
||||
) as run,
|
||||
):
|
||||
ok = self.api._generate_video_thumbnail(self.video, self.thumb, self.SIZE)
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual(run.call_args.args[0][0], "ffmpeg")
|
||||
with Image.open(self.thumb) as img:
|
||||
# ffmpeg's frame (a PNG on stdout) fitted inside the edge (#317).
|
||||
self.assertEqual((img.format, img.size), ("JPEG", (48, 32)))
|
||||
|
||||
def test_ffmpeg_is_tracked_and_a_hold_before_the_rename_writes_nothing(self):
|
||||
tracked = []
|
||||
|
||||
def writing(cmd, **kwargs):
|
||||
return self._process(0, stdout=self._png_frame())
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.Popen", side_effect=writing),
|
||||
):
|
||||
with self.assertRaises(images_module.PreviewWriteHeld):
|
||||
self.api._generate_video_thumbnail(
|
||||
self.video,
|
||||
self.thumb,
|
||||
self.SIZE,
|
||||
guard=lambda: False,
|
||||
track=tracked.append,
|
||||
)
|
||||
self.assertEqual(len(tracked), 1)
|
||||
self.assertFalse(self.thumb.exists())
|
||||
self.assertEqual(
|
||||
[p.name for p in self.thumb.parent.iterdir() if "part" in p.name], []
|
||||
)
|
||||
|
||||
def test_placeholder_failure_returns_false(self):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError),
|
||||
patch.object(
|
||||
images_module.Image, "new", side_effect=RuntimeError("no PIL")
|
||||
),
|
||||
@@ -2008,7 +2079,7 @@ class TestThumbnailStartupStatus(ThumbnailTestCase):
|
||||
):
|
||||
with self.subTest(route=route):
|
||||
for root in (self.output_dir, second):
|
||||
for thumb in (root / "thumbnails").glob("*_thumb.png"):
|
||||
for thumb in (root / "thumbnails").glob("*_thumb.webp"):
|
||||
thumb.unlink()
|
||||
response = await self.client.post(route, json={})
|
||||
if route == self.PROGRESS_URL:
|
||||
@@ -2016,7 +2087,7 @@ class TestThumbnailStartupStatus(ThumbnailTestCase):
|
||||
else:
|
||||
data = await response.json()
|
||||
self.assertEqual(data["count"], 2)
|
||||
self.assertTrue((second / "thumbnails" / "b_thumb.png").exists())
|
||||
self.assertTrue((second / "thumbnails" / "b_thumb.webp").exists())
|
||||
status = await (await self.client.get(self.STATUS_URL)).json()
|
||||
self.assertEqual(status["pending"], 0)
|
||||
self.assertEqual(status["total_images"], 2)
|
||||
@@ -2036,10 +2107,13 @@ class TestThumbnailStartupStatus(ThumbnailTestCase):
|
||||
make_png(second / "b.png")
|
||||
self.api._get_all_output_dirs = lambda: [self.output_dir, second]
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1):
|
||||
plan = self.api._plan_gallery_thumbnails([self.output_dir, second])
|
||||
response = await self.client.post(self.PROGRESS_URL)
|
||||
data = dict(parse_sse(await response.text()))["complete"]
|
||||
self.assertEqual(data["count"], 1)
|
||||
self.assertEqual(data["remaining"], 1)
|
||||
# One batch holds one file across both roots; the run goes on (#317).
|
||||
self.assertEqual((len(plan["targets"]), plan["remaining"]), (1, 1))
|
||||
self.assertEqual(data["count"], 2)
|
||||
self.assertEqual(data["remaining"], 0)
|
||||
self.assertEqual(data["total_images"], 2)
|
||||
|
||||
async def test_equal_timestamps_do_not_regenerate_existing_thumbnails(self):
|
||||
@@ -2068,7 +2142,7 @@ class TestGalleryRootIdentity(ThumbnailTestCase):
|
||||
root = roots[image["root_index"]]
|
||||
for key, path in (
|
||||
("url", root / "same.png"),
|
||||
("thumbnail_url", root / "thumbnails" / "same_thumb.png"),
|
||||
("thumbnail_url", root / "thumbnails" / "same_thumb.webp"),
|
||||
):
|
||||
response = await self.client.get(image[key])
|
||||
self.assertEqual(response.status, 200)
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
"""Library location routes (#249 S6b): where it lives, Move library, Keep it here.
|
||||
|
||||
The API's database is a node-folder library inside a fake ComfyUI tree; the
|
||||
resolver is pointed at that tree (``api._library_resolver``) so nothing here
|
||||
reads the real environment or the repository's node folder.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
_mock_server = MagicMock()
|
||||
_mock_server.PromptServer.instance.routes = MagicMock()
|
||||
sys.modules.setdefault("server", _mock_server)
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from database import library_move as lm # noqa: E402
|
||||
from database.models import PromptModel # noqa: E402
|
||||
from database.operations import PromptDatabase # noqa: E402
|
||||
from py.config import PromptManagerConfig # noqa: E402
|
||||
from utils import library_location as ll # noqa: E402
|
||||
|
||||
try:
|
||||
from tests.test_api_admin import AdminAPITestCase # noqa: E402
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_admin import AdminAPITestCase # noqa: E402
|
||||
|
||||
WAIT = 10
|
||||
URL = "/prompt_manager/system/library"
|
||||
|
||||
|
||||
class LibraryAPITestCase(AdminAPITestCase):
|
||||
async def get_application(self):
|
||||
app = await super().get_application()
|
||||
# Lower case: a public path is case-folded on Windows (normcase).
|
||||
self.ext = os.path.join(str(self.comfy_dir), "custom_nodes", "pm")
|
||||
os.makedirs(self.ext)
|
||||
self.user = os.path.join(str(self.comfy_dir), "user")
|
||||
os.makedirs(self.user)
|
||||
self.target = os.path.join(self.user, "__promptmanager")
|
||||
sys.modules["folder_paths"].get_user_directory = lambda: self.user
|
||||
previous = self.api.db
|
||||
self.api.db = PromptDatabase(os.path.join(self.ext, "prompts.db"))
|
||||
previous.close_all()
|
||||
self.api.db.save_prompt("a library prompt")
|
||||
self.fp = sys.modules["folder_paths"]
|
||||
self.environ = {}
|
||||
self.api._library_resolver = lambda: ll.resolve_library(
|
||||
None, environ=self.environ, extension_root=self.ext, folder_paths=self.fp
|
||||
)
|
||||
ll.clear_override()
|
||||
self.addCleanup(ll.clear_override)
|
||||
self.addCleanup(PromptModel.close_all_instances)
|
||||
self._keep = PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER
|
||||
self.addCleanup(
|
||||
setattr, PromptManagerConfig, "KEEP_LIBRARY_IN_NODE_FOLDER", self._keep
|
||||
)
|
||||
PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER = False
|
||||
return app
|
||||
|
||||
async def json(self, method, url, status=200, **kwargs):
|
||||
response = await self.client.request(method, url, **kwargs)
|
||||
text = await response.text()
|
||||
self.assertEqual(response.status, status, text)
|
||||
self.assertNotIn(self.tmpdir, text)
|
||||
return json.loads(text)
|
||||
|
||||
async def finished(self, body):
|
||||
job = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
return job
|
||||
|
||||
|
||||
class TestLocationRoute(LibraryAPITestCase):
|
||||
async def test_node_folder_library_offers_the_move(self):
|
||||
body = await self.json("GET", URL)
|
||||
self.assertEqual(
|
||||
body["location"],
|
||||
{
|
||||
"path": "custom_nodes/pm/prompts.db",
|
||||
"source": "node_folder",
|
||||
"movable": True,
|
||||
"keep": False,
|
||||
"offer": True,
|
||||
"target": "user/__promptmanager",
|
||||
"problem": None,
|
||||
"size_bytes": os.path.getsize(self.api.db.model.db_path),
|
||||
"target_source": "default",
|
||||
"config_moves": True,
|
||||
},
|
||||
)
|
||||
|
||||
async def test_overview_carries_the_location(self):
|
||||
body = await self.json("GET", "/prompt_manager/system/overview")
|
||||
self.assertEqual(body["location"]["source"], "node_folder")
|
||||
self.assertTrue(body["location"]["offer"])
|
||||
|
||||
async def test_default_folder_library_offers_nothing(self):
|
||||
self.api._library_resolver = lambda: ll.LibraryLocation(
|
||||
ll.SOURCE_DEFAULT,
|
||||
self.api.db.model.db_path,
|
||||
os.path.join(self.target, "config.json"),
|
||||
self.target,
|
||||
self.target,
|
||||
)
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual((location["source"], location["movable"]), ("default", False))
|
||||
self.assertFalse(location["offer"])
|
||||
|
||||
async def test_a_resolved_file_that_is_not_the_live_database_is_never_moved(self):
|
||||
self.api._library_resolver = lambda: ll.resolve_library(
|
||||
None,
|
||||
environ={},
|
||||
extension_root=os.path.join(self.tmpdir, "elsewhere"),
|
||||
folder_paths=self.fp,
|
||||
)
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual(location["source"], "config")
|
||||
self.assertFalse(location["movable"])
|
||||
refused = await self.json("POST", URL + "/move", 400)
|
||||
self.assertEqual(refused["code"], "not-movable")
|
||||
|
||||
|
||||
class TestLocationDetails(LibraryAPITestCase):
|
||||
async def test_the_target_and_what_moves_follow_the_environment(self):
|
||||
data = os.path.join(self.tmpdir, "pm-data")
|
||||
self.environ = {
|
||||
ll.DATA_DIR_ENV: data,
|
||||
ll.CONFIG_PATH_ENV: os.path.join(self.tmpdir, "mine.json"),
|
||||
}
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual(location["target_source"], "environment")
|
||||
self.assertFalse(location["config_moves"])
|
||||
self.assertEqual(location["problem"], "data-dir-empty")
|
||||
|
||||
async def test_a_configured_library_inside_the_node_folder_is_flagged(self):
|
||||
self.api._library_resolver = lambda: ll.LibraryLocation(
|
||||
ll.SOURCE_CONFIG,
|
||||
self.api.db.model.db_path,
|
||||
os.path.join(self.ext, "config.json"),
|
||||
self.ext,
|
||||
)
|
||||
self.api._extension_root = self.ext
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual(location["problem"], "inside-node-folder")
|
||||
self.assertFalse(location["movable"])
|
||||
|
||||
async def test_a_startup_after_a_crash_finishes_the_rename(self):
|
||||
target_db = os.path.join(self.target, "prompts.db")
|
||||
os.makedirs(self.target)
|
||||
with open(target_db, "wb") as handle:
|
||||
handle.write(b"x")
|
||||
lm.write_marker(self.target, ll.MARKER_MOVED, self.api.db.model.db_path)
|
||||
old = self.api.db.model.db_path
|
||||
self.api.db.model.close_all()
|
||||
moved = ll.LibraryLocation(
|
||||
ll.SOURCE_DEFAULT,
|
||||
target_db,
|
||||
os.path.join(self.target, "config.json"),
|
||||
self.target,
|
||||
self.target,
|
||||
)
|
||||
self.api._library_resolver = lambda: moved
|
||||
with patch.object(self.api.db.model, "db_path", target_db):
|
||||
renamed = self.api.finish_interrupted_library_move()
|
||||
self.assertEqual(renamed, ["prompts.db.moved-to-__promptmanager"])
|
||||
self.assertFalse(os.path.exists(old))
|
||||
|
||||
|
||||
class TestLocationThroughASymlinkedNodeFolder(LibraryAPITestCase):
|
||||
"""custom_nodes/<name> is a symlink to a checkout outside ComfyUI.
|
||||
|
||||
The library location reads relative to ComfyUI through the node folder's
|
||||
own (unresolved) place in custom_nodes, not as a bare file name; the
|
||||
shared ``_public_path`` (sources, jobs) keeps resolving symlinks.
|
||||
"""
|
||||
|
||||
async def get_application(self):
|
||||
app = await super().get_application()
|
||||
self.real = os.path.join(self.tmpdir, "checkout", "ComfyUI_PromptManager")
|
||||
os.makedirs(self.real)
|
||||
self.link = os.path.join(str(self.comfy_dir), "custom_nodes", "promptmanager")
|
||||
try:
|
||||
os.symlink(self.real, self.link, target_is_directory=True)
|
||||
except (OSError, NotImplementedError) as exc:
|
||||
self.skipTest(f"symlinks not permitted here: {exc}")
|
||||
previous = self.api.db
|
||||
self.api.db = PromptDatabase(os.path.join(self.link, "prompts.db"))
|
||||
previous.close_all()
|
||||
self.api.db.save_prompt("a library prompt")
|
||||
self.api._library_resolver = lambda: ll.resolve_library(
|
||||
None, environ=self.environ, extension_root=self.link, folder_paths=self.fp
|
||||
)
|
||||
self.api._extension_root = self.link
|
||||
return app
|
||||
|
||||
async def test_location_names_the_node_folder_in_custom_nodes(self):
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual(location["source"], "node_folder")
|
||||
self.assertEqual(location["path"], "custom_nodes/promptmanager/prompts.db")
|
||||
self.assertEqual(location["target"], "user/__promptmanager")
|
||||
overview = await self.json("GET", "/prompt_manager/system/overview")
|
||||
self.assertEqual(
|
||||
overview["location"]["path"], "custom_nodes/promptmanager/prompts.db"
|
||||
)
|
||||
|
||||
async def test_a_resolved_spelling_of_the_database_reads_the_same(self):
|
||||
from py.api import library_location as routes
|
||||
|
||||
resolved = os.path.join(os.path.realpath(self.real), "prompts.db")
|
||||
self.assertEqual(
|
||||
routes.node_folder_public_path(resolved, self.link, []),
|
||||
"custom_nodes/promptmanager/prompts.db",
|
||||
)
|
||||
|
||||
async def test_the_shared_public_path_is_unchanged(self):
|
||||
self.assertEqual(
|
||||
self.api._public_path(os.path.join(self.link, "prompts.db")), "prompts.db"
|
||||
)
|
||||
|
||||
|
||||
class TestNodeFolderPublicPath(LibraryAPITestCase):
|
||||
def display(self, path, root=None, custom_nodes=()):
|
||||
from py.api import library_location as routes
|
||||
|
||||
return routes.node_folder_public_path(path, root or self.ext, custom_nodes)
|
||||
|
||||
async def test_a_plain_node_folder_reads_relative_to_custom_nodes(self):
|
||||
self.assertEqual(
|
||||
self.display(os.path.join(self.ext, "prompts.db")),
|
||||
"custom_nodes/pm/prompts.db",
|
||||
)
|
||||
self.assertEqual(
|
||||
self.display(os.path.join(self.ext, "data", "prompts.db")),
|
||||
"custom_nodes/pm/data/prompts.db",
|
||||
)
|
||||
|
||||
async def test_a_custom_nodes_entry_from_comfyui_names_the_folder(self):
|
||||
nodes = os.path.join(self.tmpdir, "more_nodes")
|
||||
root = os.path.join(nodes, "pm")
|
||||
self.assertEqual(
|
||||
self.display(os.path.join(root, "prompts.db"), root, [nodes]),
|
||||
"more_nodes/pm/prompts.db",
|
||||
)
|
||||
|
||||
async def test_anything_else_is_not_a_node_folder_path(self):
|
||||
# Outside the node folder, a parent that is no custom_nodes folder, or
|
||||
# a spelling that climbs out: the caller falls back to _public_path.
|
||||
self.assertIsNone(self.display(os.path.join(self.user, "prompts.db")))
|
||||
loose = os.path.join(self.tmpdir, "somewhere", "pm")
|
||||
self.assertIsNone(self.display(os.path.join(loose, "prompts.db"), loose))
|
||||
self.assertIsNone(self.display(os.path.join(self.ext, "..", "prompts.db")))
|
||||
self.assertIsNone(self.display(""))
|
||||
self.assertIsNone(self.display(None))
|
||||
|
||||
async def test_a_library_outside_the_node_folder_keeps_the_public_path(self):
|
||||
self.api._library_resolver = lambda: ll.LibraryLocation(
|
||||
ll.SOURCE_DEFAULT,
|
||||
self.api.db.model.db_path,
|
||||
os.path.join(self.target, "config.json"),
|
||||
self.target,
|
||||
self.target,
|
||||
)
|
||||
self.api._extension_root = os.path.join(self.tmpdir, "elsewhere")
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual(location["path"], "custom_nodes/pm/prompts.db")
|
||||
|
||||
|
||||
class TestMoveRoute(LibraryAPITestCase):
|
||||
async def test_move_runs_as_an_exclusive_job_and_switches_the_live_library(self):
|
||||
body = await self.json("POST", URL + "/move", 202)
|
||||
summary = body["job"]
|
||||
self.assertEqual(summary["kind"], "library_move")
|
||||
self.assertEqual(summary["title"], "Move library")
|
||||
self.assertEqual(summary["admission"], "exclusive")
|
||||
self.assertFalse(summary["cancellable"])
|
||||
job = await self.finished(body)
|
||||
self.assertEqual(job.state, "finished", job.status_line)
|
||||
self.assertEqual(
|
||||
self.api.db.model.db_path, os.path.join(self.target, "prompts.db")
|
||||
)
|
||||
self.assertIn("user/__promptmanager", job.status_line)
|
||||
self.assertIn("1 prompt", job.status_line)
|
||||
self.assertNotIn(self.tmpdir, json.dumps(job.report()))
|
||||
location = (await self.json("GET", URL))["location"]
|
||||
self.assertEqual(location["source"], "default")
|
||||
self.assertEqual(location["path"], "user/__promptmanager/prompts.db")
|
||||
self.assertFalse(location["offer"])
|
||||
prompts = await self.json("GET", "/prompt_manager/stats")
|
||||
self.assertEqual(prompts["stats"]["total_prompts"], 1)
|
||||
self.assertTrue(
|
||||
os.path.isfile(
|
||||
os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")
|
||||
)
|
||||
)
|
||||
|
||||
async def test_refusals_are_coded_without_paths(self):
|
||||
with patch.object(lm, "_free_bytes", return_value=1024):
|
||||
short = await self.json("POST", URL + "/move", 507)
|
||||
self.assertEqual(short["code"], "insufficient-space")
|
||||
os.makedirs(self.target)
|
||||
with open(os.path.join(self.target, "prompts.db"), "wb") as handle:
|
||||
handle.write(b"another library")
|
||||
taken = await self.json("POST", URL + "/move", 409)
|
||||
self.assertEqual(taken["code"], "target-has-library")
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
|
||||
async def test_move_is_refused_while_another_job_runs_and_blocks_others(self):
|
||||
gate = asyncio.Event()
|
||||
|
||||
async def scanning(job):
|
||||
await gate.wait()
|
||||
|
||||
self.api._jobs.start(
|
||||
"scan", "Metadata scan", admission="records", runner=scanning
|
||||
)
|
||||
busy = await self.json("POST", URL + "/move", 409)
|
||||
self.assertEqual(busy["code"], "busy")
|
||||
gate.set()
|
||||
await asyncio.sleep(0)
|
||||
for job in self.api._jobs.list():
|
||||
if job.task is not None:
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
|
||||
release = threading.Event()
|
||||
entered = threading.Event()
|
||||
real = lm.move_library
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(WAIT)
|
||||
return real(*args, **kwargs)
|
||||
|
||||
with patch.object(lm, "move_library", side_effect=gated):
|
||||
body = await self.json("POST", URL + "/move", 202)
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(None, entered.wait, WAIT)
|
||||
refused = await self.json("POST", "/prompt_manager/system/health", 409)
|
||||
self.assertEqual(refused["running"]["title"], "Move library")
|
||||
release.set()
|
||||
job = await self.finished(body)
|
||||
self.assertEqual(job.state, "finished", job.status_line)
|
||||
|
||||
async def test_a_failed_move_ends_truthfully_and_keeps_the_old_library(self):
|
||||
with patch.object(
|
||||
lm, "move_library", side_effect=lm.MoveFailed("verify-failed")
|
||||
):
|
||||
body = await self.json("POST", URL + "/move", 202)
|
||||
job = await self.finished(body)
|
||||
self.assertEqual(job.state, "failed")
|
||||
self.assertIn("did not pass verification", job.status_line)
|
||||
self.assertIn("still in the node folder", job.status_line)
|
||||
self.assertEqual(
|
||||
self.api.db.model.db_path, os.path.join(self.ext, "prompts.db")
|
||||
)
|
||||
with patch.object(lm, "move_library", side_effect=OSError("disk gone")):
|
||||
body = await self.json("POST", URL + "/move", 202)
|
||||
job = await self.finished(body)
|
||||
self.assertEqual(job.state, "failed")
|
||||
self.assertNotIn("disk gone", job.status_line)
|
||||
self.assertIn("still in the node folder", job.status_line)
|
||||
|
||||
async def test_old_file_left_in_place_is_reported(self):
|
||||
real = lm.retire_old_files
|
||||
with patch.object(lm, "retire_old_files", return_value=([], "old-in-use")):
|
||||
body = await self.json("POST", URL + "/move", 202)
|
||||
job = await self.finished(body)
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertIn("left in the node folder", job.status_line)
|
||||
self.assertTrue(callable(real))
|
||||
|
||||
|
||||
class TestKeepRoute(LibraryAPITestCase):
|
||||
async def test_keep_it_here_is_persisted_and_stops_the_offer(self):
|
||||
body = await self.json("POST", URL + "/keep", 200, json={"keep": True})
|
||||
self.assertTrue(body["location"]["keep"])
|
||||
self.assertFalse(body["location"]["offer"])
|
||||
self.assertTrue(body["location"]["movable"])
|
||||
with open(self.config_path, encoding="utf-8") as handle:
|
||||
saved = json.load(handle)
|
||||
self.assertEqual(saved["library"], {"keep_in_node_folder": True})
|
||||
self.assertTrue(PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER)
|
||||
undone = await self.json("POST", URL + "/keep", 200, json={"keep": False})
|
||||
self.assertTrue(undone["location"]["offer"])
|
||||
|
||||
async def test_keep_needs_a_boolean(self):
|
||||
for payload in ({"keep": "yes"}, {}, [True]):
|
||||
bad = await self.json("POST", URL + "/keep", 400, json=payload)
|
||||
self.assertEqual(bad["code"], "invalid-request")
|
||||
self.assertFalse(os.path.exists(self.config_path))
|
||||
|
||||
async def test_unwritable_config_reports_not_saved(self):
|
||||
with patch("py.config_store._write_atomically", side_effect=OSError("ro")):
|
||||
body = await self.json("POST", URL + "/keep", 500, json={"keep": True})
|
||||
self.assertEqual(body["code"], "config-write-failed")
|
||||
self.assertFalse(PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER)
|
||||
|
||||
|
||||
class TestLegacyRoutesDuringAMove(LibraryAPITestCase):
|
||||
async def test_legacy_restore_is_refused_while_the_move_runs(self):
|
||||
from aiohttp import FormData
|
||||
|
||||
release = threading.Event()
|
||||
entered = threading.Event()
|
||||
real = lm.move_library
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(WAIT)
|
||||
return real(*args, **kwargs)
|
||||
|
||||
with patch.object(lm, "move_library", side_effect=gated):
|
||||
body = await self.json("POST", URL + "/move", 202)
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(None, entered.wait, WAIT)
|
||||
form = FormData()
|
||||
form.add_field("database_file", b"SQLite format 3\x00", filename="x.db")
|
||||
response = await self.client.request(
|
||||
"POST", "/prompt_manager/restore", data=form
|
||||
)
|
||||
self.assertEqual(response.status, 409, await response.text())
|
||||
release.set()
|
||||
await self.finished(body)
|
||||
@@ -397,15 +397,21 @@ class TestQueueReads(ReviewRoutesTestCase):
|
||||
self._no_paths()
|
||||
|
||||
async def test_queue_cards_use_the_images_listing_thumbnail(self):
|
||||
# M10: a known thumbnail is the card's media, as on the Images page;
|
||||
# without one the card falls back to the original.
|
||||
# M10: the card's media is the Images listing's thumbnail. Without a
|
||||
# preview yet it is still the thumbnail URL (made on first view, #249).
|
||||
self._require_routes()
|
||||
from py.api.images import _output_image_entry
|
||||
|
||||
path = self._ambiguous("sub/f0.png")
|
||||
self._record(path)
|
||||
item = (await self._list())["items"][0]
|
||||
self.assertIsNone(item["thumbnail_url"])
|
||||
scope = item["target"]["root_scope"]
|
||||
self.assertEqual(
|
||||
item["thumbnail_url"],
|
||||
"/prompt_manager/images/serve/thumbnails/sub/f0_thumb.webp"
|
||||
f"?root=0&root_scope={scope}",
|
||||
)
|
||||
# A legacy preview name is still listed until it is converted (#317).
|
||||
thumb = self.output_dir / "thumbnails" / "sub" / "f0_thumb.png"
|
||||
thumb.parent.mkdir(parents=True)
|
||||
write_png(thumb, prompt=ambiguous_graph("cat", "dog"), color=(1, 2, 3))
|
||||
|
||||
@@ -75,7 +75,8 @@ class TestGalleryConfig(unittest.TestCase):
|
||||
{"web_interface": {"images_per_page": 50, "thumbnail_size": 512}}
|
||||
)
|
||||
self.assertEqual(GalleryConfig.IMAGES_PER_PAGE, 50)
|
||||
self.assertEqual(GalleryConfig.THUMBNAIL_SIZE, 512)
|
||||
# #317 R1: the never-read thumbnail_size is ignored (thumbnails.edge).
|
||||
self.assertFalse(hasattr(GalleryConfig, "THUMBNAIL_SIZE"))
|
||||
|
||||
def test_update_performance(self):
|
||||
GalleryConfig.update_config({"performance": {"max_concurrent_processing": 5}})
|
||||
|
||||
@@ -293,6 +293,8 @@ class TestRestoreFromFile(BackupTestCase):
|
||||
super().backup(target, *args, **kwargs)
|
||||
|
||||
def connect_with_failure(*args, **kwargs) -> sqlite3.Connection:
|
||||
if "factory" in kwargs: # the model's own managed connections
|
||||
return connect(*args, **kwargs)
|
||||
return connect(*args, **kwargs, factory=FailingLiveBackup)
|
||||
|
||||
with patch("database.models.sqlite3.connect", side_effect=connect_with_failure):
|
||||
|
||||
@@ -66,7 +66,7 @@ class TestIngestionKeepsDefaultProfile(unittest.TestCase):
|
||||
handler.link_image_to_prompt("/out/f03.png", {"id": 1}, metadata)
|
||||
self.assertEqual(len(spy.calls), 2)
|
||||
spy.assert_default_and_gated(self)
|
||||
handler.db_manager.link_image_to_prompt.assert_called_once()
|
||||
handler.db_manager.link_image_to_prompt_or_raise.assert_called_once()
|
||||
|
||||
def test_admin_scan_ingestion_uses_the_default_profile(self):
|
||||
from py.api import admin
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
"""Where the library lives (#249 S6b): the one resolver, ``utils/library_location``.
|
||||
|
||||
Every case builds its own extension root (the node folder) and ComfyUI user
|
||||
folder in a temp directory and hands the resolver a fake ``folder_paths``, so
|
||||
no test reads the real environment, the repository's own files or ComfyUI.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from utils import library_location as ll # noqa: E402
|
||||
|
||||
|
||||
def _touch(path, data=b"x"):
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "wb") as handle:
|
||||
handle.write(data)
|
||||
|
||||
|
||||
def _folder_paths(base, *, helper=True):
|
||||
"""A ComfyUI ``folder_paths`` stand-in rooted at ``base``."""
|
||||
module = types.SimpleNamespace(base_path=base)
|
||||
for name in ("output", "input", "temp", "user"):
|
||||
directory = os.path.join(base, name)
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
setattr(module, f"get_{name}_directory", lambda d=directory: d)
|
||||
if helper:
|
||||
module.get_system_user_directory = lambda name="system": os.path.join(
|
||||
base, "user", "__" + name
|
||||
)
|
||||
return module
|
||||
|
||||
|
||||
class _Case(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
self.base = os.path.realpath(self._tmp.name)
|
||||
self.comfy = os.path.join(self.base, "ComfyUI")
|
||||
self.ext = os.path.join(self.comfy, "custom_nodes", "ComfyUI_PromptManager")
|
||||
os.makedirs(self.ext)
|
||||
self.fp = _folder_paths(self.comfy)
|
||||
self.default_dir = os.path.join(self.comfy, "user", "__promptmanager")
|
||||
ll.clear_override()
|
||||
self.addCleanup(ll.clear_override)
|
||||
|
||||
def resolve(self, default_path=None, environ=None, **kwargs):
|
||||
kwargs.setdefault("folder_paths", self.fp)
|
||||
return ll.resolve_library(
|
||||
default_path, environ=environ or {}, extension_root=self.ext, **kwargs
|
||||
)
|
||||
|
||||
def config_location(self, environ=None, **kwargs):
|
||||
kwargs.setdefault("folder_paths", self.fp)
|
||||
return ll.config_location(
|
||||
environ=environ or {}, extension_root=self.ext, **kwargs
|
||||
)
|
||||
|
||||
|
||||
class TestDefaultDataFolder(_Case):
|
||||
def test_uses_comfyui_system_user_helper(self):
|
||||
self.assertEqual(ll.default_data_dir(self.fp), self.default_dir)
|
||||
|
||||
def test_falls_back_to_user_directory_without_helper(self):
|
||||
fp = _folder_paths(self.comfy, helper=False)
|
||||
self.assertEqual(ll.default_data_dir(fp), self.default_dir)
|
||||
|
||||
def test_none_outside_comfyui(self):
|
||||
self.assertIsNone(ll.default_data_dir(None))
|
||||
self.assertIsNone(ll.default_data_dir(types.SimpleNamespace()))
|
||||
|
||||
|
||||
class TestResolutionOrder(_Case):
|
||||
def test_new_install_uses_default_folder_and_creates_nothing_in_node_folder(self):
|
||||
location = self.resolve()
|
||||
self.assertEqual(location.source, ll.SOURCE_DEFAULT)
|
||||
self.assertEqual(location.db_path, os.path.join(self.default_dir, "prompts.db"))
|
||||
self.assertEqual(
|
||||
location.config_path, os.path.join(self.default_dir, "config.json")
|
||||
)
|
||||
self.assertFalse(location.movable)
|
||||
self.assertEqual(os.listdir(self.ext), [])
|
||||
|
||||
def test_legacy_node_folder_library_stays_in_use(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
location = self.resolve()
|
||||
self.assertEqual(location.source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertEqual(location.db_path, os.path.join(self.ext, "prompts.db"))
|
||||
self.assertEqual(location.config_path, os.path.join(self.ext, "config.json"))
|
||||
self.assertEqual(location.data_dir, self.ext)
|
||||
self.assertTrue(location.movable)
|
||||
self.assertEqual(location.target_dir, self.default_dir)
|
||||
|
||||
def test_legacy_wins_over_an_unmarked_default_folder_library(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
_touch(os.path.join(self.default_dir, "prompts.db"))
|
||||
self.assertEqual(self.resolve().source, ll.SOURCE_NODE_FOLDER)
|
||||
|
||||
def test_explicit_default_path_is_honoured_as_today(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
elsewhere = os.path.join(self.base, "elsewhere", "pm.db")
|
||||
location = self.resolve(elsewhere)
|
||||
self.assertEqual(location.source, ll.SOURCE_CONFIG)
|
||||
self.assertEqual(location.db_path, elsewhere)
|
||||
self.assertFalse(location.movable)
|
||||
relative = self.resolve("data/my.db")
|
||||
self.assertEqual(relative.db_path, os.path.join(self.ext, "data", "my.db"))
|
||||
self.assertEqual(relative.source, ll.SOURCE_CONFIG)
|
||||
|
||||
def test_default_setting_value_is_not_explicit(self):
|
||||
for value in (None, "", "prompts.db"):
|
||||
self.assertEqual(self.resolve(value).source, ll.SOURCE_DEFAULT, value)
|
||||
|
||||
def test_environment_folder_wins(self):
|
||||
data = os.path.join(self.base, "pm-data")
|
||||
location = self.resolve(environ={ll.DATA_DIR_ENV: data})
|
||||
self.assertEqual(location.source, ll.SOURCE_ENVIRONMENT)
|
||||
self.assertEqual(location.db_path, os.path.join(data, "prompts.db"))
|
||||
self.assertEqual(location.config_path, os.path.join(data, "config.json"))
|
||||
|
||||
def test_environment_folder_with_a_library_wins_over_legacy(self):
|
||||
data = os.path.join(self.base, "pm-data")
|
||||
_touch(os.path.join(data, "prompts.db"))
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
location = self.resolve(environ={ll.DATA_DIR_ENV: data})
|
||||
self.assertEqual(location.source, ll.SOURCE_ENVIRONMENT)
|
||||
|
||||
def test_empty_environment_folder_never_replaces_a_legacy_library(self):
|
||||
data = os.path.join(self.base, "pm-data")
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
location = self.resolve(environ={ll.DATA_DIR_ENV: data})
|
||||
self.assertEqual(location.source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertEqual(location.db_path, os.path.join(self.ext, "prompts.db"))
|
||||
self.assertEqual(location.problem, "data-dir-empty")
|
||||
self.assertEqual(location.target_dir, data)
|
||||
self.assertTrue(location.movable)
|
||||
|
||||
def test_config_path_environment_is_honoured(self):
|
||||
custom = os.path.join(self.base, "cfg", "custom.json")
|
||||
location = self.resolve(environ={ll.CONFIG_PATH_ENV: custom})
|
||||
self.assertEqual(location.config_path, custom)
|
||||
self.assertEqual(location.source, ll.SOURCE_DEFAULT)
|
||||
|
||||
def test_without_comfyui_the_node_folder_is_used(self):
|
||||
location = self.resolve(folder_paths=None)
|
||||
self.assertEqual(location.source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertEqual(location.db_path, os.path.join(self.ext, "prompts.db"))
|
||||
self.assertFalse(location.movable)
|
||||
|
||||
def test_override_after_a_live_move_wins(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
moved = ll.LibraryLocation(
|
||||
source=ll.SOURCE_DEFAULT,
|
||||
db_path=os.path.join(self.default_dir, "prompts.db"),
|
||||
config_path=os.path.join(self.default_dir, "config.json"),
|
||||
data_dir=self.default_dir,
|
||||
target_dir=self.default_dir,
|
||||
)
|
||||
ll.switch_to(moved)
|
||||
self.assertEqual(self.resolve(), moved)
|
||||
self.assertEqual(self.config_location(), (moved.config_path, moved.data_dir))
|
||||
ll.clear_override()
|
||||
self.assertEqual(self.resolve().source, ll.SOURCE_NODE_FOLDER)
|
||||
|
||||
|
||||
class TestMarker(_Case):
|
||||
def _marker(self, directory, state):
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
with open(os.path.join(directory, ll.MARKER_NAME), "w") as handle:
|
||||
json.dump({"version": 1, "state": state}, handle)
|
||||
|
||||
def test_moved_marker_wins_while_the_old_file_was_not_renamed_yet(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
_touch(os.path.join(self.default_dir, "prompts.db"))
|
||||
self._marker(self.default_dir, "moved")
|
||||
location = self.resolve()
|
||||
self.assertEqual(location.source, ll.SOURCE_DEFAULT)
|
||||
self.assertEqual(location.db_path, os.path.join(self.default_dir, "prompts.db"))
|
||||
self.assertEqual(
|
||||
location.config_path, os.path.join(self.default_dir, "config.json")
|
||||
)
|
||||
|
||||
def test_pending_marker_keeps_the_old_library(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
_touch(os.path.join(self.default_dir, "prompts.db"))
|
||||
self._marker(self.default_dir, "pending")
|
||||
self.assertEqual(self.resolve().source, ll.SOURCE_NODE_FOLDER)
|
||||
data = os.path.join(self.base, "pm-data")
|
||||
_touch(os.path.join(data, "prompts.db"))
|
||||
self._marker(data, "pending")
|
||||
location = self.resolve(environ={ll.DATA_DIR_ENV: data})
|
||||
self.assertEqual(location.source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertEqual(location.problem, "move-unfinished")
|
||||
|
||||
def test_unreadable_marker_is_not_trusted(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
_touch(os.path.join(self.default_dir, "prompts.db"))
|
||||
_touch(os.path.join(self.default_dir, ll.MARKER_NAME), b"{not json")
|
||||
self.assertEqual(self.resolve().source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertEqual(ll.read_marker(self.default_dir), {})
|
||||
|
||||
def test_renamed_old_file_is_never_picked(self):
|
||||
_touch(os.path.join(self.ext, "prompts.db.moved-to-__promptmanager"))
|
||||
_touch(os.path.join(self.default_dir, "prompts.db"))
|
||||
self.assertEqual(self.resolve().source, ll.SOURCE_DEFAULT)
|
||||
|
||||
|
||||
class TestConfigLocation(_Case):
|
||||
def test_legacy_config_without_a_database_keeps_being_read(self):
|
||||
_touch(os.path.join(self.ext, "config.json"), b"{}")
|
||||
path, data_dir = self.config_location()
|
||||
self.assertEqual(path, os.path.join(self.ext, "config.json"))
|
||||
self.assertEqual(data_dir, self.ext)
|
||||
|
||||
def test_legacy_config_alone_never_puts_a_new_database_in_the_node_folder(self):
|
||||
_touch(os.path.join(self.ext, "config.json"), b"{}")
|
||||
location = self.resolve()
|
||||
self.assertEqual(location.db_path, os.path.join(self.default_dir, "prompts.db"))
|
||||
self.assertEqual(location.config_path, os.path.join(self.ext, "config.json"))
|
||||
self.assertFalse(location.movable)
|
||||
|
||||
def test_new_install_reads_config_from_the_data_folder(self):
|
||||
path, data_dir = self.config_location()
|
||||
self.assertEqual(path, os.path.join(self.default_dir, "config.json"))
|
||||
self.assertEqual(data_dir, self.default_dir)
|
||||
|
||||
def test_logs_and_cache_follow_the_data_folder(self):
|
||||
self.assertEqual(
|
||||
ll.logs_dir(environ={}, extension_root=self.ext, folder_paths=self.fp),
|
||||
os.path.join(self.default_dir, "logs"),
|
||||
)
|
||||
_touch(os.path.join(self.ext, "prompts.db"))
|
||||
self.assertEqual(
|
||||
ll.logs_dir(environ={}, extension_root=self.ext, folder_paths=self.fp),
|
||||
os.path.join(self.ext, "logs"),
|
||||
)
|
||||
self.assertEqual(
|
||||
ll.cache_dir(
|
||||
"lora_images", environ={}, extension_root=self.ext, folder_paths=self.fp
|
||||
),
|
||||
os.path.join(self.ext, "data", "lora_images"),
|
||||
)
|
||||
|
||||
|
||||
class TestDataDirValidation(_Case):
|
||||
def problem(self, path, **kwargs):
|
||||
kwargs.setdefault("folder_paths", self.fp)
|
||||
return ll.data_dir_problem(path, **kwargs)
|
||||
|
||||
def test_valid_folder(self):
|
||||
self.assertIsNone(self.problem(os.path.join(self.base, "pm-data")))
|
||||
self.assertIsNone(self.problem(self.default_dir))
|
||||
|
||||
def test_relative_and_empty_paths_are_refused(self):
|
||||
self.assertEqual(self.problem("pm-data"), "data-dir-not-absolute")
|
||||
self.assertEqual(self.problem(""), "data-dir-not-absolute")
|
||||
|
||||
def test_served_folders_are_refused(self):
|
||||
for name in ("output", "input", "temp"):
|
||||
inside = os.path.join(self.comfy, name, "pm")
|
||||
self.assertEqual(self.problem(inside), "data-dir-served", name)
|
||||
self.assertEqual(
|
||||
self.problem(os.path.join(self.comfy, "output")), "data-dir-served"
|
||||
)
|
||||
|
||||
def test_public_user_folder_is_refused_but_system_users_are_fine(self):
|
||||
public = os.path.join(self.comfy, "user", "default", "pm")
|
||||
self.assertEqual(self.problem(public), "data-dir-served")
|
||||
self.assertIsNone(self.problem(os.path.join(self.comfy, "user", "__pm")))
|
||||
|
||||
def test_gallery_source_is_refused(self):
|
||||
source = os.path.join(self.base, "gallery")
|
||||
os.makedirs(source)
|
||||
self.assertEqual(
|
||||
self.problem(os.path.join(source, "pm"), gallery_roots=[source]),
|
||||
"data-dir-served",
|
||||
)
|
||||
self.assertEqual(
|
||||
self.problem(
|
||||
os.path.join(source, "pm"),
|
||||
environ={ll.EXTRA_GALLERY_ROOTS_ENV: source},
|
||||
),
|
||||
"data-dir-served",
|
||||
)
|
||||
|
||||
def test_symlink_into_output_is_refused(self):
|
||||
link = os.path.join(self.base, "link-to-output")
|
||||
try:
|
||||
os.symlink(os.path.join(self.comfy, "output"), link)
|
||||
except (OSError, NotImplementedError):
|
||||
self.skipTest("symlinks are not available")
|
||||
self.assertEqual(self.problem(os.path.join(link, "pm")), "data-dir-served")
|
||||
|
||||
def test_case_variant_spelling_of_output_is_refused_where_the_volume_folds_case(
|
||||
self,
|
||||
):
|
||||
upper = os.path.join(self.comfy, "OUTPUT")
|
||||
if not os.path.isdir(upper):
|
||||
self.skipTest("this volume is case-sensitive")
|
||||
self.assertEqual(self.problem(os.path.join(upper, "pm")), "data-dir-served")
|
||||
|
||||
def test_invalid_environment_folder_is_ignored_with_a_problem(self):
|
||||
inside = os.path.join(self.comfy, "output", "pm")
|
||||
location = self.resolve(environ={ll.DATA_DIR_ENV: inside})
|
||||
self.assertEqual(location.source, ll.SOURCE_DEFAULT)
|
||||
self.assertEqual(location.problem, "data-dir-served")
|
||||
self.assertFalse(os.path.exists(inside))
|
||||
relative = self.resolve(environ={ll.DATA_DIR_ENV: "relative/dir"})
|
||||
self.assertEqual(relative.problem, "data-dir-not-absolute")
|
||||
|
||||
|
||||
class TestOneRuleForConfigAndDatabase(_Case):
|
||||
"""M1: config.json and the database never split over PROMPT_MANAGER_DATA_DIR."""
|
||||
|
||||
def test_a_source_listed_only_in_a_config_file_rules_out_the_folder_for_both(self):
|
||||
source = os.path.join(self.base, "gallery")
|
||||
os.makedirs(source)
|
||||
_touch(
|
||||
os.path.join(self.ext, "config.json"),
|
||||
json.dumps({"gallery": {"monitoring": {"directories": [source]}}}).encode(),
|
||||
)
|
||||
environ = {ll.DATA_DIR_ENV: os.path.join(source, "pm")}
|
||||
location = self.resolve(environ=environ)
|
||||
config_path, data_dir = self.config_location(environ=environ)
|
||||
self.assertEqual(location.problem, "data-dir-served")
|
||||
self.assertEqual(
|
||||
(location.config_path, location.data_dir), (config_path, data_dir)
|
||||
)
|
||||
self.assertNotEqual(location.source, ll.SOURCE_ENVIRONMENT)
|
||||
|
||||
def test_a_relative_source_is_taken_from_comfyui(self):
|
||||
self.fp.base_path = self.comfy
|
||||
os.makedirs(os.path.join(self.comfy, "pictures"))
|
||||
_touch(
|
||||
os.path.join(self.ext, "config.json"),
|
||||
json.dumps(
|
||||
{"gallery": {"monitoring": {"directories": ["pictures"]}}}
|
||||
).encode(),
|
||||
)
|
||||
environ = {ll.DATA_DIR_ENV: os.path.join(self.comfy, "pictures", "pm")}
|
||||
self.assertEqual(self.resolve(environ=environ).problem, "data-dir-served")
|
||||
self.assertEqual(self.config_location(environ=environ)[1], self.ext)
|
||||
|
||||
def test_unreadable_or_odd_config_files_add_no_source(self):
|
||||
for payload in (
|
||||
b"{not json",
|
||||
b"[]",
|
||||
json.dumps({"gallery": {"monitoring": {"directories": "x"}}}).encode(),
|
||||
):
|
||||
_touch(os.path.join(self.ext, "config.json"), payload)
|
||||
environ = {ll.DATA_DIR_ENV: os.path.join(self.base, "pm-data")}
|
||||
self.assertIsNone(self.resolve(environ=environ).problem)
|
||||
|
||||
|
||||
class TestSameFolder(_Case):
|
||||
def test_same_folder_follows_links_and_case(self):
|
||||
self.assertTrue(ll.same_folder(self.ext, self.ext + os.sep))
|
||||
self.assertFalse(ll.same_folder(self.ext, self.default_dir))
|
||||
self.assertFalse(ll.same_folder(self.ext, os.path.join(self.base, "missing")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,169 @@
|
||||
"""Every data path asks the one resolver (#249 S6b).
|
||||
|
||||
The database, the diagnostics database, config.json, the logs and the LoRA
|
||||
image cache all follow ``utils.library_location``: a folder named by
|
||||
``PROMPT_MANAGER_DATA_DIR`` (and, after a live move, the switched location).
|
||||
"""
|
||||
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
import unittest.mock as mock
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
_mock_server = MagicMock()
|
||||
_mock_server.PromptServer.instance.routes = MagicMock()
|
||||
sys.modules.setdefault("server", _mock_server)
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from utils import library_location as ll # noqa: E402
|
||||
|
||||
|
||||
class _EnvCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
base = os.path.realpath(self._tmp.name)
|
||||
self.data = os.path.join(base, "pm-data")
|
||||
patcher = mock.patch.dict(
|
||||
os.environ,
|
||||
{ll.DATA_DIR_ENV: self.data, ll.CONFIG_PATH_ENV: ""},
|
||||
)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
# This test's own ComfyUI folders (the shared stub serves the system
|
||||
# temp folder as "output", which would rightly refuse self.data).
|
||||
comfy = os.path.join(base, "ComfyUI")
|
||||
fp = types.ModuleType("folder_paths")
|
||||
for name in ("output", "input", "temp", "user"):
|
||||
setattr(fp, f"get_{name}_directory", lambda n=name: os.path.join(comfy, n))
|
||||
modules = mock.patch.dict(sys.modules, {"folder_paths": fp})
|
||||
modules.start()
|
||||
self.addCleanup(modules.stop)
|
||||
ll.clear_override()
|
||||
self.addCleanup(ll.clear_override)
|
||||
|
||||
|
||||
class TestDatabasePath(_EnvCase):
|
||||
def test_database_follows_the_data_folder(self):
|
||||
from database.operations import _resolve_db_path
|
||||
|
||||
with mock.patch.dict(sys.modules, {"py.config": None}):
|
||||
path = _resolve_db_path(None)
|
||||
self.assertEqual(path, os.path.join(self.data, "prompts.db"))
|
||||
self.assertTrue(os.path.isdir(self.data))
|
||||
if os.name == "posix":
|
||||
self.assertEqual(os.stat(self.data).st_mode & 0o777, 0o700)
|
||||
|
||||
def test_diagnostics_use_the_same_resolver(self):
|
||||
from database.operations import _resolve_db_path
|
||||
from utils.diagnostics import GalleryDiagnostics
|
||||
|
||||
with mock.patch.dict(sys.modules, {"py.config": None}):
|
||||
self.assertEqual(GalleryDiagnostics._resolve_db_path(), _resolve_db_path())
|
||||
|
||||
def test_switched_location_wins(self):
|
||||
from database.operations import _resolve_db_path
|
||||
|
||||
moved = ll.LibraryLocation(
|
||||
ll.SOURCE_DEFAULT,
|
||||
os.path.join(self.data, "moved", "prompts.db"),
|
||||
os.path.join(self.data, "moved", "config.json"),
|
||||
os.path.join(self.data, "moved"),
|
||||
)
|
||||
ll.switch_to(moved)
|
||||
self.assertEqual(_resolve_db_path(None), moved.db_path)
|
||||
|
||||
|
||||
class TestConfigPath(_EnvCase):
|
||||
def test_config_json_follows_the_data_folder(self):
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
self.assertEqual(
|
||||
PromptManagerConfig.get_config_path(),
|
||||
os.path.join(self.data, "config.json"),
|
||||
)
|
||||
|
||||
def test_explicit_config_path_still_wins(self):
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
custom = os.path.join(self.data, "elsewhere.json")
|
||||
with mock.patch.dict(os.environ, {ll.CONFIG_PATH_ENV: custom}):
|
||||
self.assertEqual(PromptManagerConfig.get_config_path(), custom)
|
||||
|
||||
def test_keep_choice_round_trips(self):
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
original = PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER
|
||||
self.addCleanup(
|
||||
setattr, PromptManagerConfig, "KEEP_LIBRARY_IN_NODE_FOLDER", original
|
||||
)
|
||||
PromptManagerConfig.update_config({"library": {"keep_in_node_folder": True}})
|
||||
self.assertTrue(PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER)
|
||||
self.assertEqual(
|
||||
PromptManagerConfig.get_config()["library"], {"keep_in_node_folder": True}
|
||||
)
|
||||
PromptManagerConfig.update_config({"library": {"keep_in_node_folder": "yes"}})
|
||||
self.assertTrue(PromptManagerConfig.KEEP_LIBRARY_IN_NODE_FOLDER)
|
||||
|
||||
|
||||
class TestLogsAndCache(_EnvCase):
|
||||
def test_logs_go_to_the_data_folder(self):
|
||||
from utils import logging_config
|
||||
from utils.logging_config import PromptManagerLogger
|
||||
|
||||
manager = object.__new__(PromptManagerLogger)
|
||||
self.addCleanup(self._close, manager)
|
||||
manager.__init__()
|
||||
self.assertEqual(manager.log_dir, pathlib.Path(self.data, "logs"))
|
||||
self.assertTrue(manager.log_dir.is_dir())
|
||||
self.assertIs(logging_config.library_location, ll)
|
||||
|
||||
@staticmethod
|
||||
def _close(manager):
|
||||
import logging
|
||||
|
||||
for handler in logging.getLogger("prompt_manager").handlers[:]:
|
||||
if getattr(handler, "_prompt_manager_owned", False):
|
||||
handler.close()
|
||||
try:
|
||||
from utils.logging_config import get_logger_manager
|
||||
|
||||
get_logger_manager()._setup_loggers()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def test_lora_image_cache_follows_the_data_folder(self):
|
||||
from py import lora_utils
|
||||
|
||||
cache = lora_utils.get_lora_image_cache_dir()
|
||||
self.assertEqual(cache, pathlib.Path(self.data, "data", "lora_images"))
|
||||
|
||||
|
||||
class TestNodeFolderUnchangedForLegacyInstalls(unittest.TestCase):
|
||||
"""A node folder that holds the library keeps every path where it was."""
|
||||
|
||||
def test_legacy_paths(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
ext = os.path.join(tmp, "ext")
|
||||
os.makedirs(ext)
|
||||
open(os.path.join(ext, "prompts.db"), "wb").close()
|
||||
fp = types.SimpleNamespace(
|
||||
get_user_directory=lambda: os.path.join(tmp, "u")
|
||||
)
|
||||
location = ll.resolve_library(
|
||||
None, environ={}, extension_root=ext, folder_paths=fp
|
||||
)
|
||||
self.assertEqual(location.db_path, os.path.join(ext, "prompts.db"))
|
||||
self.assertEqual(
|
||||
ll.logs_dir(environ={}, extension_root=ext, folder_paths=fp),
|
||||
os.path.join(ext, "logs"),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,497 @@
|
||||
"""Move library (#249 S6b): copy, verify, switch, then retire the old files.
|
||||
|
||||
Each test builds a node-folder library (a real PromptDatabase with prompts,
|
||||
images and workflow documents, a config.json holding a key, a pooled safety
|
||||
backup and the pre-migration backup) and a ComfyUI user folder in a temp
|
||||
directory, then moves it with ``database.library_move``. Failures and crash
|
||||
windows are injected through the move's ``checkpoint`` hook: an ``Exception``
|
||||
is a failure the move cleans up after, ``SimulatedCrash`` (a BaseException)
|
||||
stops it dead like a killed process.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import types
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from database import library_move as lm # noqa: E402
|
||||
from database.execution_history import ( # noqa: E402
|
||||
notify_history_restore,
|
||||
subscribe_history_restore,
|
||||
)
|
||||
from database.models import PromptModel # noqa: E402
|
||||
from database.operations import PromptDatabase # noqa: E402
|
||||
from database.review_changes import ( # noqa: E402
|
||||
notify_review_change,
|
||||
subscribe_review_changes,
|
||||
)
|
||||
from tests.open_handles import assert_closed, open_handles_under # noqa: E402
|
||||
from utils import library_location as ll # noqa: E402
|
||||
|
||||
KEY = "civitai-secret-key"
|
||||
POOL = "prompts.db.backup_20261001_101010_000001"
|
||||
PRE = "prompts.db.pre-workflow-storage_20261002_101010_000002"
|
||||
|
||||
|
||||
class SimulatedCrash(BaseException):
|
||||
"""The process died here: nothing after this point runs."""
|
||||
|
||||
|
||||
def _digest(path):
|
||||
with open(path, "rb") as handle:
|
||||
return hashlib.sha256(handle.read()).hexdigest()
|
||||
|
||||
|
||||
def _rows(path):
|
||||
"""Every table's rows of ``path`` (read-only), to compare libraries."""
|
||||
uri = "file:" + path.replace(os.sep, "/") + "?mode=ro"
|
||||
conn = sqlite3.connect(uri, uri=True)
|
||||
try:
|
||||
tables = [
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' "
|
||||
"AND name NOT LIKE 'sqlite_%' ORDER BY name"
|
||||
)
|
||||
]
|
||||
return {
|
||||
table: sorted(
|
||||
repr(tuple(row)) for row in conn.execute(f'SELECT * FROM "{table}"')
|
||||
)
|
||||
for table in tables
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _workflow(seed):
|
||||
return {
|
||||
"nodes": [
|
||||
{"id": n, "type": f"Node{seed}", "widgets": [seed, n]} for n in range(30)
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
class _MoveCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.base = os.path.realpath(self._tmp.name)
|
||||
self.comfy = os.path.join(self.base, "ComfyUI")
|
||||
self.ext = os.path.join(self.comfy, "custom_nodes", "ComfyUI_PromptManager")
|
||||
os.makedirs(self.ext)
|
||||
self.fp = types.SimpleNamespace()
|
||||
for name in ("output", "input", "temp", "user"):
|
||||
folder = os.path.join(self.comfy, name)
|
||||
os.makedirs(folder)
|
||||
setattr(self.fp, f"get_{name}_directory", lambda f=folder: f)
|
||||
self.target = os.path.join(self.comfy, "user", "__promptmanager")
|
||||
self.old_db = os.path.join(self.ext, "prompts.db")
|
||||
self.new_db = os.path.join(self.target, "prompts.db")
|
||||
ll.clear_override()
|
||||
self.db = PromptDatabase(self.old_db)
|
||||
for index in range(3):
|
||||
prompt_id = self.db.save_prompt(f"a cat number {index}", category="cats")
|
||||
self.db.link_image_to_prompt(
|
||||
str(prompt_id),
|
||||
os.path.join(self.comfy, "output", f"cat{index}.png"),
|
||||
{"workflow": _workflow(index % 2), "prompt": {"1": {"seed": index}}},
|
||||
)
|
||||
self.db.model.close_all()
|
||||
self.config = os.path.join(self.ext, "config.json")
|
||||
with open(self.config, "w", encoding="utf-8") as handle:
|
||||
json.dump(
|
||||
{"integrations": {"lora_manager": {"civitai_api_key": KEY}}}, handle
|
||||
)
|
||||
for name, stamp in ((POOL, 1_700_000_000), (PRE, 1_700_000_100)):
|
||||
path = os.path.join(self.ext, name)
|
||||
source = sqlite3.connect(self.old_db)
|
||||
copy = sqlite3.connect(path)
|
||||
try:
|
||||
source.backup(copy)
|
||||
finally:
|
||||
copy.close()
|
||||
source.close()
|
||||
os.utime(path, (stamp, stamp))
|
||||
self.before = {
|
||||
"rows": _rows(self.old_db),
|
||||
"config": _digest(self.config),
|
||||
"pool": _digest(os.path.join(self.ext, POOL)),
|
||||
"pre": _digest(os.path.join(self.ext, PRE)),
|
||||
}
|
||||
self.config_lock = threading.RLock()
|
||||
|
||||
def tearDown(self):
|
||||
import gc
|
||||
|
||||
gc.collect()
|
||||
PromptModel.close_all_instances()
|
||||
ll.clear_override()
|
||||
try:
|
||||
assert_closed(self, self.ext, self.target)
|
||||
finally:
|
||||
self._tmp.cleanup()
|
||||
|
||||
def location(self, **kwargs):
|
||||
return ll.resolve_library(
|
||||
None, environ={}, extension_root=self.ext, folder_paths=self.fp, **kwargs
|
||||
)
|
||||
|
||||
def move(self, **kwargs):
|
||||
plan = lm.plan_move(self.location())
|
||||
kwargs.setdefault("config_lock", self.config_lock)
|
||||
return lm.move_library(plan, **kwargs)
|
||||
|
||||
def assert_old_library_untouched(self):
|
||||
self.assertTrue(os.path.isfile(self.old_db))
|
||||
self.assertEqual(_rows(self.old_db), self.before["rows"])
|
||||
self.assertEqual(_digest(self.config), self.before["config"])
|
||||
self.assertEqual(_digest(os.path.join(self.ext, POOL)), self.before["pool"])
|
||||
self.assertEqual(_digest(os.path.join(self.ext, PRE)), self.before["pre"])
|
||||
self.assertEqual(
|
||||
os.path.normcase(self.db.model.db_path), os.path.normcase(self.old_db)
|
||||
)
|
||||
self.assertEqual(self.location().source, ll.SOURCE_NODE_FOLDER)
|
||||
|
||||
def assert_target_clean(self):
|
||||
leftovers = os.listdir(self.target) if os.path.isdir(self.target) else []
|
||||
self.assertEqual(leftovers, [])
|
||||
|
||||
|
||||
class TestMove(_MoveCase):
|
||||
def test_move_copies_verifies_switches_and_retires_the_old_files(self):
|
||||
result = self.move()
|
||||
|
||||
self.assertEqual(_rows(self.new_db), self.before["rows"])
|
||||
self.assertEqual(result.counts["prompts"], 3)
|
||||
self.assertEqual(result.counts["generated_images"], 3)
|
||||
self.assertEqual(
|
||||
result.counts["workflow_documents"], 5
|
||||
) # 2 workflows, 3 graphs
|
||||
self.assertEqual(
|
||||
sorted(result.files), sorted(["config.json", POOL, PRE, "prompts.db"])
|
||||
)
|
||||
self.assertIsNone(result.retire_problem)
|
||||
# The live database object now reads and writes the new file.
|
||||
self.assertEqual(self.db.model.db_path, self.new_db)
|
||||
self.db.save_prompt("written after the move")
|
||||
self.assertEqual(len(_rows(self.new_db)["prompts"]), 4)
|
||||
image = self.db.get_prompt_images(
|
||||
str(self.db.search_prompts(text="number 1")[0]["id"])
|
||||
)[0]
|
||||
self.assertEqual(image["workflow_data"], _workflow(1))
|
||||
# config.json, the pooled backup and the pre-migration backup: same bytes.
|
||||
self.assertEqual(
|
||||
_digest(os.path.join(self.target, "config.json")), self.before["config"]
|
||||
)
|
||||
self.assertEqual(_digest(os.path.join(self.target, POOL)), self.before["pool"])
|
||||
self.assertEqual(_digest(os.path.join(self.target, PRE)), self.before["pre"])
|
||||
self.assertEqual(
|
||||
int(os.path.getmtime(os.path.join(self.target, POOL))), 1_700_000_000
|
||||
)
|
||||
# The old files are renamed, never deleted; the backups stay.
|
||||
renamed = sorted(os.listdir(self.ext))
|
||||
self.assertIn("prompts.db.moved-to-__promptmanager", renamed)
|
||||
self.assertIn("config.json.moved-to-__promptmanager", renamed)
|
||||
self.assertNotIn("prompts.db", renamed)
|
||||
self.assertNotIn("config.json", renamed)
|
||||
self.assertFalse(any(name.endswith(("-wal", "-shm")) for name in renamed))
|
||||
self.assertEqual(
|
||||
_rows(os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")),
|
||||
self.before["rows"],
|
||||
)
|
||||
self.assertEqual(ll.read_marker(self.target)["state"], ll.MARKER_MOVED)
|
||||
|
||||
def test_new_files_are_private(self):
|
||||
if os.name != "posix":
|
||||
self.skipTest("POSIX permissions")
|
||||
self.move()
|
||||
self.assertEqual(os.stat(self.target).st_mode & 0o777, 0o700)
|
||||
for name in ("prompts.db", "config.json", POOL, PRE, ll.MARKER_NAME):
|
||||
mode = os.stat(os.path.join(self.target, name)).st_mode & 0o777
|
||||
self.assertEqual(mode, 0o600, name)
|
||||
|
||||
def test_restart_uses_the_new_library_and_never_the_renamed_file(self):
|
||||
self.move()
|
||||
self.assertEqual(self.location().db_path, self.new_db)
|
||||
ll.clear_override()
|
||||
restarted = self.location()
|
||||
self.assertEqual(restarted.source, ll.SOURCE_DEFAULT)
|
||||
self.assertEqual(restarted.db_path, self.new_db)
|
||||
self.assertEqual(
|
||||
restarted.config_path, os.path.join(self.target, "config.json")
|
||||
)
|
||||
|
||||
def test_every_open_database_object_is_repointed(self):
|
||||
node = PromptDatabase(self.old_db)
|
||||
monitor = PromptDatabase(self.old_db)
|
||||
monitor.get_prompt_by_id(1)
|
||||
other = PromptDatabase(os.path.join(self.base, "other.db"))
|
||||
self.move()
|
||||
self.assertEqual(node.model.db_path, self.new_db)
|
||||
self.assertEqual(monitor.model.db_path, self.new_db)
|
||||
self.assertEqual(monitor.get_prompt_by_id(1)["text"], "a cat number 0")
|
||||
self.assertEqual(other.model.db_path, os.path.join(self.base, "other.db"))
|
||||
assert_closed(
|
||||
self, os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")
|
||||
)
|
||||
|
||||
def test_change_observers_follow_the_library(self):
|
||||
reviews, restores = [], []
|
||||
unsubscribe_review = subscribe_review_changes(
|
||||
self.old_db, lambda: reviews.append(1)
|
||||
)
|
||||
unsubscribe_restore = subscribe_history_restore(
|
||||
self.old_db, lambda: restores.append(1)
|
||||
)
|
||||
self.addCleanup(unsubscribe_review)
|
||||
self.addCleanup(unsubscribe_restore)
|
||||
self.move()
|
||||
self.assertEqual(restores, [1]) # the history readers start a new epoch
|
||||
notify_review_change(self.new_db)
|
||||
notify_history_restore(self.new_db)
|
||||
self.assertEqual(reviews, [1])
|
||||
self.assertEqual(restores, [1, 1])
|
||||
|
||||
def test_files_are_only_renamed_inside_one_folder(self):
|
||||
real_replace, real_rename = os.replace, os.rename
|
||||
moves = []
|
||||
|
||||
def replace(src, dst, *args, **kwargs):
|
||||
moves.append((src, dst))
|
||||
return real_replace(src, dst, *args, **kwargs)
|
||||
|
||||
def rename(src, dst, *args, **kwargs):
|
||||
moves.append((src, dst))
|
||||
return real_rename(src, dst, *args, **kwargs)
|
||||
|
||||
with (
|
||||
mock.patch.object(lm.os, "replace", replace),
|
||||
mock.patch.object(lm.os, "rename", rename),
|
||||
):
|
||||
self.move()
|
||||
self.assertTrue(moves)
|
||||
for src, dst in moves:
|
||||
self.assertEqual(os.path.dirname(src), os.path.dirname(dst), (src, dst))
|
||||
|
||||
def test_a_write_during_the_copy_is_copied_again(self):
|
||||
writes = []
|
||||
|
||||
def checkpoint(step):
|
||||
if step == "files" and not writes:
|
||||
writes.append(PromptDatabase(self.old_db).save_prompt("late write"))
|
||||
|
||||
progress = []
|
||||
self.move(checkpoint=checkpoint, progress=progress.append)
|
||||
self.assertIn("late write", repr(_rows(self.new_db)["prompts"]))
|
||||
self.assertTrue(any("changed" in line for line in progress), progress)
|
||||
|
||||
def test_a_library_that_keeps_changing_is_not_moved(self):
|
||||
def checkpoint(step):
|
||||
if step == "files":
|
||||
PromptDatabase(self.old_db).save_prompt(f"write {os.urandom(4).hex()}")
|
||||
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
self.move(checkpoint=checkpoint, attempts=2)
|
||||
self.assertEqual(caught.exception.code, "library-busy")
|
||||
self.assertEqual(self.location().source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assert_target_clean()
|
||||
|
||||
def test_old_file_still_open_elsewhere_is_left_in_place(self):
|
||||
reader = sqlite3.connect(self.old_db)
|
||||
self.addCleanup(reader.close)
|
||||
reader.execute("BEGIN")
|
||||
reader.execute("SELECT COUNT(*) FROM prompts").fetchone()
|
||||
PromptDatabase(self.old_db).save_prompt("in the wal")
|
||||
result = self.move()
|
||||
self.assertEqual(result.retire_problem, "old-in-use")
|
||||
self.assertTrue(os.path.isfile(self.old_db))
|
||||
self.assertEqual(self.db.model.db_path, self.new_db)
|
||||
reader.rollback()
|
||||
reader.close()
|
||||
ll.clear_override()
|
||||
self.assertEqual(self.location().source, ll.SOURCE_DEFAULT)
|
||||
|
||||
|
||||
class TestFailures(_MoveCase):
|
||||
def _fail_at(self, step):
|
||||
def checkpoint(name):
|
||||
if name == step:
|
||||
raise RuntimeError(f"failure at {step}")
|
||||
|
||||
return checkpoint
|
||||
|
||||
def test_a_failure_at_any_step_leaves_the_old_library_in_use_and_untouched(self):
|
||||
for step in ("copied", "verified", "files", "locked", "pending", "published"):
|
||||
with self.subTest(step=step):
|
||||
with self.assertRaises(RuntimeError):
|
||||
self.move(checkpoint=self._fail_at(step))
|
||||
self.assert_old_library_untouched()
|
||||
self.assert_target_clean()
|
||||
assert_closed(self, self.target)
|
||||
|
||||
def test_a_copy_that_does_not_verify_is_refused(self):
|
||||
def checkpoint(step):
|
||||
if step == "copied":
|
||||
(temp,) = [
|
||||
name for name in os.listdir(self.target) if name.endswith(".db")
|
||||
]
|
||||
with open(os.path.join(self.target, temp), "r+b") as handle:
|
||||
handle.truncate(4096)
|
||||
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
self.move(checkpoint=checkpoint)
|
||||
self.assertEqual(caught.exception.code, "verify-failed")
|
||||
self.assertNotIn(self.base, caught.exception.message)
|
||||
self.assert_old_library_untouched()
|
||||
self.assert_target_clean()
|
||||
|
||||
def test_a_copied_file_that_differs_is_refused(self):
|
||||
def checkpoint(step):
|
||||
if step == "files":
|
||||
for name in os.listdir(self.target):
|
||||
if name.endswith(".bak"):
|
||||
with open(os.path.join(self.target, name), "ab") as handle:
|
||||
handle.write(b"x")
|
||||
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
self.move(checkpoint=checkpoint)
|
||||
self.assertEqual(caught.exception.code, "copy-failed")
|
||||
self.assert_old_library_untouched()
|
||||
self.assert_target_clean()
|
||||
|
||||
def test_not_movable_locations_are_refused(self):
|
||||
os.makedirs(self.target)
|
||||
with open(self.new_db, "wb") as handle:
|
||||
handle.write(b"someone else's library")
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
lm.plan_move(self.location())
|
||||
self.assertEqual(caught.exception.code, "target-has-library")
|
||||
os.remove(self.new_db)
|
||||
default = ll.LibraryLocation(
|
||||
ll.SOURCE_DEFAULT, self.new_db, "c", self.target, self.target
|
||||
)
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
lm.plan_move(default)
|
||||
self.assertEqual(caught.exception.code, "not-movable")
|
||||
|
||||
def test_target_that_is_the_node_folder_is_refused(self):
|
||||
link = os.path.join(self.base, "link-to-node-folder")
|
||||
try:
|
||||
os.symlink(self.ext, link)
|
||||
except (OSError, NotImplementedError):
|
||||
self.skipTest("symlinks are not available")
|
||||
location = ll.LibraryLocation(
|
||||
ll.SOURCE_NODE_FOLDER, self.old_db, self.config, self.ext, link
|
||||
)
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
lm.plan_move(location)
|
||||
self.assertEqual(caught.exception.code, "same-folder")
|
||||
|
||||
def test_disk_check_measures_the_target_volume(self):
|
||||
measured = []
|
||||
|
||||
def free(path):
|
||||
measured.append(path)
|
||||
return 1024
|
||||
|
||||
with mock.patch.object(lm, "_free_bytes", free):
|
||||
plan = lm.plan_move(self.location())
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
lm.move_library(plan, config_lock=self.config_lock)
|
||||
self.assertEqual(caught.exception.code, "insufficient-space")
|
||||
self.assertTrue(measured)
|
||||
for path in measured:
|
||||
self.assertTrue(
|
||||
os.path.normcase(self.comfy) in os.path.normcase(path)
|
||||
and "custom_nodes" not in path,
|
||||
path,
|
||||
)
|
||||
self.assertGreater(plan.required_bytes, os.path.getsize(self.old_db))
|
||||
self.assert_old_library_untouched()
|
||||
|
||||
|
||||
class TestCrashWindows(_MoveCase):
|
||||
"""A crash between steps never loses the library or picks a stale one."""
|
||||
|
||||
def _crash_at(self, step):
|
||||
def checkpoint(name):
|
||||
if name == step:
|
||||
raise SimulatedCrash(step)
|
||||
|
||||
return checkpoint
|
||||
|
||||
def _restart(self):
|
||||
PromptModel.close_all_instances()
|
||||
ll.clear_override()
|
||||
return self.location()
|
||||
|
||||
def test_crash_before_the_switch_keeps_the_old_library_and_a_retry_succeeds(self):
|
||||
for step in ("copied", "files", "pending", "published"):
|
||||
with self.subTest(step=step):
|
||||
with self.assertRaises(SimulatedCrash):
|
||||
self.move(checkpoint=self._crash_at(step))
|
||||
restarted = self._restart()
|
||||
self.assertEqual(restarted.source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertEqual(restarted.db_path, self.old_db)
|
||||
self.assertEqual(_rows(self.old_db), self.before["rows"])
|
||||
result = self.move()
|
||||
self.assertEqual(_rows(self.new_db), self.before["rows"])
|
||||
self.assertIsNone(result.retire_problem)
|
||||
self.assertFalse(
|
||||
[name for name in os.listdir(self.target) if name.startswith(".pm-move-")]
|
||||
)
|
||||
|
||||
def test_crash_after_the_marker_uses_the_new_library(self):
|
||||
for step in ("marked", "switched"):
|
||||
with self.subTest(step=step):
|
||||
with self.assertRaises(SimulatedCrash):
|
||||
self.move(checkpoint=self._crash_at(step))
|
||||
restarted = self._restart()
|
||||
self.assertEqual(restarted.source, ll.SOURCE_DEFAULT)
|
||||
self.assertEqual(restarted.db_path, self.new_db)
|
||||
self.assertEqual(_rows(self.new_db), self.before["rows"])
|
||||
# The old file was not renamed yet and is never picked.
|
||||
self.assertTrue(os.path.isfile(self.old_db))
|
||||
# Reset for the next window: back to the node folder only.
|
||||
for name in os.listdir(self.target):
|
||||
os.remove(os.path.join(self.target, name))
|
||||
|
||||
def test_crash_after_the_rename(self):
|
||||
with self.assertRaises(SimulatedCrash):
|
||||
self.move(checkpoint=self._crash_at("retired"))
|
||||
restarted = self._restart()
|
||||
self.assertEqual(restarted.db_path, self.new_db)
|
||||
self.assertFalse(os.path.exists(self.old_db))
|
||||
|
||||
|
||||
class TestEnvironmentTarget(_MoveCase):
|
||||
def test_move_into_the_environment_folder(self):
|
||||
data = os.path.join(self.base, "pm-data")
|
||||
environ = {ll.DATA_DIR_ENV: data}
|
||||
location = ll.resolve_library(
|
||||
None, environ=environ, extension_root=self.ext, folder_paths=self.fp
|
||||
)
|
||||
self.assertEqual(location.problem, "data-dir-empty")
|
||||
result = lm.move_library(lm.plan_move(location), config_lock=self.config_lock)
|
||||
self.assertEqual(result.location.source, ll.SOURCE_ENVIRONMENT)
|
||||
self.assertEqual(_rows(os.path.join(data, "prompts.db")), self.before["rows"])
|
||||
ll.clear_override()
|
||||
self.assertEqual(
|
||||
ll.resolve_library(
|
||||
None, environ=environ, extension_root=self.ext, folder_paths=self.fp
|
||||
).source,
|
||||
ll.SOURCE_ENVIRONMENT,
|
||||
)
|
||||
self.assertEqual(open_handles_under(self.old_db), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,458 @@
|
||||
"""Move library, final review round (#249 S6b PR #318).
|
||||
|
||||
- I1: a switch that fails at any point is undone while both files are still
|
||||
locked: the copy is removed, the old library stays in use and a restart
|
||||
keeps it. A follow-up after a complete switch only logs.
|
||||
- I2: a background write (run counting, the image monitor) that fails
|
||||
because Move library switched runs once more, on the new library; a write
|
||||
racing the switch on a real connection (a run count, an image link) lands
|
||||
in the new library only.
|
||||
- The switching thread never closes a connection another thread is inside
|
||||
a statement on (a use-after-free in SQLite: "Windows fatal exception:
|
||||
access violation"); it interrupts it and the owner closes it.
|
||||
- M2: the copy is verified through a file URI that survives ``#`` and ``%``.
|
||||
- M6: a restart after a crash between the switch and the rename renames the
|
||||
old files, never deletes them.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import threading
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from database import library_move as lm # noqa: E402
|
||||
from database import models # noqa: E402
|
||||
from database.operations import PromptDatabase # noqa: E402
|
||||
from tests.open_handles import assert_closed # noqa: E402
|
||||
from utils import library_location as ll # noqa: E402
|
||||
from utils import usage_tracking # noqa: E402
|
||||
from utils.hashing import generate_prompt_hash # noqa: E402
|
||||
|
||||
try:
|
||||
from tests.test_library_move import KEY, SimulatedCrash, _MoveCase, _rows
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_library_move import KEY, SimulatedCrash, _MoveCase, _rows
|
||||
|
||||
WAIT = 30
|
||||
|
||||
|
||||
def _run_count(path, text):
|
||||
uri = "file:" + path.replace(os.sep, "/") + "?mode=ro"
|
||||
with contextlib.closing(sqlite3.connect(uri, uri=True)) as conn:
|
||||
row = conn.execute("SELECT run_count FROM prompts WHERE text=?", (text,))
|
||||
return row.fetchone()[0]
|
||||
|
||||
|
||||
class _CrossThreadCloseProbe:
|
||||
"""Record every ``close()`` this thread calls on a connection in use elsewhere.
|
||||
|
||||
The writer marks its connection with ``using`` when its statement starts.
|
||||
Closing that connection from the switching thread while the writer is
|
||||
inside the statement frees memory SQLite is still using; on Windows the
|
||||
process dies with an access violation, on Linux the writer sees garbage.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.in_use = set()
|
||||
self.violations = []
|
||||
|
||||
def using(self, conn):
|
||||
self.in_use.add(conn)
|
||||
|
||||
def _profile(self, _frame, event, arg):
|
||||
if event != "c_call" or getattr(arg, "__name__", "") != "close":
|
||||
return
|
||||
target = getattr(arg, "__self__", None)
|
||||
if isinstance(target, sqlite3.Connection) and target in self.in_use:
|
||||
self.violations.append(threading.current_thread().name)
|
||||
|
||||
@contextlib.contextmanager
|
||||
def watching(self):
|
||||
previous = sys.getprofile()
|
||||
sys.setprofile(self._profile)
|
||||
try:
|
||||
yield self
|
||||
finally:
|
||||
sys.setprofile(previous)
|
||||
|
||||
|
||||
class TestFailedSwitchIsUndone(_MoveCase):
|
||||
"""I1: whatever part of the switch fails, nothing was moved."""
|
||||
|
||||
def _assert_not_moved(self, caught):
|
||||
self.assertEqual(caught.exception.code, "switch-failed")
|
||||
self.assertIn("Nothing was moved", caught.exception.message)
|
||||
self.assertIsNone(ll.current_override())
|
||||
self.assertEqual(self.db.model.db_path, self.old_db)
|
||||
# Writes keep landing in the old library, and a restart keeps it.
|
||||
self.db.save_prompt("written after the failed switch")
|
||||
self.assertIn("written after the failed switch", repr(_rows(self.old_db)))
|
||||
self.assertEqual(self.location().source, ll.SOURCE_NODE_FOLDER)
|
||||
self.assertNotEqual(ll.read_marker(self.target).get("state"), "moved")
|
||||
self.assert_target_clean()
|
||||
self.assertTrue(os.path.isfile(self.config))
|
||||
assert_closed(self, self.target)
|
||||
|
||||
def _failing(self, point):
|
||||
real_switch_to = ll.switch_to
|
||||
real_repoint = models.PromptModel.repoint_instances
|
||||
|
||||
def switch(old_db, location, detached):
|
||||
if point == "before":
|
||||
raise RuntimeError("switch failed before anything")
|
||||
if point == "resolver":
|
||||
real_switch_to(location)
|
||||
raise RuntimeError("switch failed after the resolver")
|
||||
if point == "repoint":
|
||||
real_switch_to(location)
|
||||
detached.extend(real_repoint(old_db, location.db_path))
|
||||
raise RuntimeError("switch failed after the models")
|
||||
lm.switch_process(old_db, location, detached)
|
||||
raise RuntimeError("switch failed at its very end")
|
||||
|
||||
return switch
|
||||
|
||||
def test_a_switch_failing_at_any_point_is_undone(self):
|
||||
node = PromptDatabase(self.old_db)
|
||||
node.get_prompt_by_id(1)
|
||||
for point in ("before", "resolver", "repoint", "complete"):
|
||||
with self.subTest(point=point):
|
||||
with self.assertRaises(lm.MoveFailed) as caught:
|
||||
self.move(switch=self._failing(point))
|
||||
self._assert_not_moved(caught)
|
||||
self.assertEqual(node.model.db_path, self.old_db)
|
||||
self.assertEqual(node.get_prompt_by_id(1)["text"], "a cat number 0")
|
||||
|
||||
def test_a_failed_switch_closes_the_old_connections_it_detached(self):
|
||||
node = PromptDatabase(self.old_db)
|
||||
for point in ("repoint", "complete"):
|
||||
with self.subTest(point=point):
|
||||
before = node.model.get_connection()
|
||||
with self.assertRaises(lm.MoveFailed):
|
||||
self.move(switch=self._failing(point))
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
before.execute("SELECT 1")
|
||||
# The model reopens the old library on next use.
|
||||
self.assertEqual(node.get_prompt_by_id(1)["text"], "a cat number 0")
|
||||
|
||||
def test_a_failing_follow_up_after_the_switch_only_logs(self):
|
||||
def follow_up():
|
||||
raise RuntimeError("the health report could not be dropped")
|
||||
|
||||
with self.assertLogs("prompt_manager.database.library_move", "ERROR"):
|
||||
result = self.move(after_switch=follow_up)
|
||||
self.assertIsNone(result.retire_problem)
|
||||
self.assertEqual(self.db.model.db_path, self.new_db)
|
||||
self.assertTrue(
|
||||
os.path.isfile(
|
||||
os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class _Flaky:
|
||||
"""A write that fails once as if Move library switched while it waited."""
|
||||
|
||||
def __init__(self, db, name):
|
||||
self.db, self.name, self.calls = db, name, 0
|
||||
self.real = getattr(db, name)
|
||||
|
||||
def __call__(self, *args, **kwargs):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
self.db.model.close_all() # what the switch does: a new generation
|
||||
raise sqlite3.ProgrammingError("Cannot operate on a closed database.")
|
||||
return self.real(*args, **kwargs)
|
||||
|
||||
|
||||
class TestBackgroundWritesSurviveTheSwitch(_MoveCase):
|
||||
"""I2: run counting and the image monitor write again on a fresh connection."""
|
||||
|
||||
def test_run_counting_retries_once_after_a_switch(self):
|
||||
db = PromptDatabase(self.old_db)
|
||||
text = "a counted prompt"
|
||||
db.save_prompt(text, prompt_hash=generate_prompt_hash(text))
|
||||
flaky = _Flaky(db, "record_prompt_use")
|
||||
with mock.patch.object(db, "record_prompt_use", flaky):
|
||||
pending = usage_tracking.PendingFirstUse()
|
||||
usage_tracking.count_runs([(generate_prompt_hash(text), text)], db, pending)
|
||||
self.assertEqual(flaky.calls, 2)
|
||||
self.assertEqual(_run_count(self.old_db, text), 1)
|
||||
|
||||
def test_a_failure_without_a_switch_is_not_retried(self):
|
||||
db = PromptDatabase(self.old_db)
|
||||
calls = []
|
||||
|
||||
def broken():
|
||||
calls.append(1)
|
||||
raise sqlite3.OperationalError("disk I/O error")
|
||||
|
||||
with self.assertRaises(sqlite3.OperationalError):
|
||||
db.survive_library_switch(broken)
|
||||
self.assertEqual(calls, [1])
|
||||
|
||||
def test_an_image_link_racing_the_switch_lands_in_the_new_library(self):
|
||||
from utils.image_monitor import ImageGenerationHandler
|
||||
|
||||
self.db.model.close_all()
|
||||
image = os.path.join(self.comfy, "output", "late.png")
|
||||
writer_db = PromptDatabase(self.old_db)
|
||||
handler = ImageGenerationHandler(writer_db, mock.MagicMock())
|
||||
started = threading.Event()
|
||||
probe = _CrossThreadCloseProbe()
|
||||
outcome = {}
|
||||
|
||||
def write():
|
||||
conn = writer_db.model.get_connection()
|
||||
|
||||
def trace(sql):
|
||||
if sql.lstrip().upper().startswith(("BEGIN", "INSERT")):
|
||||
probe.using(conn)
|
||||
started.set()
|
||||
|
||||
conn.set_trace_callback(trace)
|
||||
try:
|
||||
# No metadata: the fallback link, not the resolved pair.
|
||||
handler.link_image_to_prompt(image, {"id": 1}, {})
|
||||
outcome["ok"] = True
|
||||
except Exception as error: # pragma: no cover - reported below
|
||||
outcome["error"] = error
|
||||
finally:
|
||||
writer_db.model.close()
|
||||
|
||||
thread = threading.Thread(target=write)
|
||||
|
||||
def checkpoint(step):
|
||||
if step == "locked":
|
||||
# The old file is write-locked now: start the writer and
|
||||
# wait until its first write is running (it then waits on us).
|
||||
thread.start()
|
||||
self.assertTrue(started.wait(WAIT))
|
||||
|
||||
with probe.watching():
|
||||
result = self.move(checkpoint=checkpoint)
|
||||
thread.join(WAIT)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertEqual(probe.violations, [])
|
||||
self.assertEqual(outcome, {"ok": True})
|
||||
self.assertIsNone(result.retire_problem)
|
||||
moved = [
|
||||
row
|
||||
for row in _rows(self.new_db)["generated_images"]
|
||||
if "late.png" in repr(row)
|
||||
]
|
||||
self.assertEqual(len(moved), 1)
|
||||
renamed = os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")
|
||||
self.assertNotIn("late.png", repr(_rows(renamed).get("generated_images")))
|
||||
|
||||
def test_an_image_that_was_not_linked_is_not_reported_as_linked(self):
|
||||
from utils.image_monitor import ImageGenerationHandler
|
||||
|
||||
db = PromptDatabase(self.old_db)
|
||||
handler = ImageGenerationHandler(db, mock.MagicMock())
|
||||
image = os.path.join(self.comfy, "output", "orphan.png")
|
||||
with mock.patch.object(handler, "logger") as log:
|
||||
handler.link_image_to_prompt(image, {"id": 99999}, {})
|
||||
messages = [str(call) for call in log.debug.call_args_list]
|
||||
self.assertFalse([m for m in messages if "Successfully linked" in m])
|
||||
self.assertNotIn("orphan.png", repr(_rows(self.old_db)["generated_images"]))
|
||||
|
||||
def test_a_write_racing_the_switch_lands_in_the_new_library(self):
|
||||
text = "a racing prompt"
|
||||
self.db.save_prompt(text, prompt_hash=generate_prompt_hash(text))
|
||||
self.db.model.close_all()
|
||||
writer_db = PromptDatabase(self.old_db)
|
||||
thread, started, probe, outcome = self._racing_run_count(writer_db, text)
|
||||
|
||||
def checkpoint(step):
|
||||
if step == "locked":
|
||||
# The old file is write-locked now: start the writer and
|
||||
# wait until its UPDATE is running (it then waits on us).
|
||||
thread.start()
|
||||
self.assertTrue(started.wait(WAIT))
|
||||
|
||||
with probe.watching():
|
||||
result = self.move(checkpoint=checkpoint)
|
||||
# The writer's old connection was closed before the old file was
|
||||
# renamed: Windows refuses to rename an open file.
|
||||
self.assertIsNone(result.retire_problem)
|
||||
thread.join(WAIT)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertEqual(probe.violations, [])
|
||||
self.assertEqual(outcome, {"ok": True})
|
||||
self.assertEqual(_run_count(self.new_db, text), 1)
|
||||
renamed = os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")
|
||||
self.assertEqual(_run_count(renamed, text), 0)
|
||||
assert_closed(self, renamed)
|
||||
|
||||
def test_a_write_racing_a_failed_switch_lands_in_the_old_library(self):
|
||||
text = "a prompt racing a failed switch"
|
||||
self.db.save_prompt(text, prompt_hash=generate_prompt_hash(text))
|
||||
self.db.model.close_all()
|
||||
writer_db = PromptDatabase(self.old_db)
|
||||
thread, started, probe, outcome = self._racing_run_count(writer_db, text)
|
||||
|
||||
def checkpoint(step):
|
||||
if step == "locked":
|
||||
thread.start()
|
||||
self.assertTrue(started.wait(WAIT))
|
||||
|
||||
def switch(old_db, location, detached):
|
||||
lm.switch_process(old_db, location, detached)
|
||||
raise RuntimeError("switch failed at its very end")
|
||||
|
||||
with probe.watching():
|
||||
with self.assertRaises(lm.MoveFailed):
|
||||
self.move(checkpoint=checkpoint, switch=switch)
|
||||
# The copy was removed: no connection was left open on it.
|
||||
self.assert_target_clean()
|
||||
assert_closed(self, self.target)
|
||||
thread.join(WAIT)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertEqual(probe.violations, [])
|
||||
self.assertEqual(outcome, {"ok": True})
|
||||
self.assertEqual(_run_count(self.old_db, text), 1)
|
||||
|
||||
def test_a_retire_failure_after_a_complete_switch_does_not_fail_the_move(self):
|
||||
node = PromptDatabase(self.old_db)
|
||||
node.get_prompt_by_id(1)
|
||||
real = models.PromptModel.retire_detached
|
||||
calls = []
|
||||
|
||||
def failing(connections, logger=None):
|
||||
calls.append(len(connections))
|
||||
real(connections, logger) # closed, but the call still fails
|
||||
raise RuntimeError("retiring failed")
|
||||
|
||||
with mock.patch.object(
|
||||
models.PromptModel, "retire_detached", staticmethod(failing)
|
||||
):
|
||||
with self.assertLogs("prompt_manager.database.library_move", "ERROR"):
|
||||
result = self.move()
|
||||
self.assertEqual(len(calls), 1)
|
||||
self.assertGreater(calls[0], 0)
|
||||
self.assertEqual(node.model.db_path, self.new_db)
|
||||
self.assertEqual(result.location.db_path, self.new_db)
|
||||
self.assertTrue(
|
||||
os.path.isfile(
|
||||
os.path.join(self.ext, "prompts.db.moved-to-__promptmanager")
|
||||
)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _racing_run_count(writer_db, text):
|
||||
"""A run count whose UPDATE marks its connection in use for the probe."""
|
||||
started = threading.Event()
|
||||
probe = _CrossThreadCloseProbe()
|
||||
outcome = {}
|
||||
|
||||
def write():
|
||||
conn = writer_db.model.get_connection()
|
||||
|
||||
def trace(sql):
|
||||
if sql.startswith("UPDATE"):
|
||||
probe.using(conn)
|
||||
started.set()
|
||||
|
||||
conn.set_trace_callback(trace)
|
||||
try:
|
||||
usage_tracking.count_runs(
|
||||
[(generate_prompt_hash(text), text)],
|
||||
writer_db,
|
||||
usage_tracking.PendingFirstUse(),
|
||||
)
|
||||
outcome["ok"] = True
|
||||
except Exception as error: # pragma: no cover - reported below
|
||||
outcome["error"] = error
|
||||
finally:
|
||||
writer_db.model.close()
|
||||
|
||||
return threading.Thread(target=write), started, probe, outcome
|
||||
|
||||
|
||||
class TestVerifyUri(unittest.TestCase):
|
||||
"""M2: a copy whose path holds ``#`` and ``%`` verifies."""
|
||||
|
||||
def test_copy_path_with_uri_characters_verifies(self):
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
folder = os.path.join(tmp, "lib #1 %20")
|
||||
os.makedirs(folder)
|
||||
path = os.path.join(folder, "copy.db")
|
||||
db = PromptDatabase(path)
|
||||
db.save_prompt("a prompt")
|
||||
db.model.close_all()
|
||||
with contextlib.closing(sqlite3.connect(path)) as conn:
|
||||
conn.execute("PRAGMA journal_mode=DELETE")
|
||||
counts, documents = lm._describe(conn)
|
||||
lm._verify_copy(path, counts, documents)
|
||||
models.PromptModel.close_all_instances()
|
||||
|
||||
|
||||
class TestRestartFinishesTheRename(_MoveCase):
|
||||
"""M6: the marker names the old files; a restart renames them."""
|
||||
|
||||
def _crash_after_switch(self):
|
||||
def checkpoint(step):
|
||||
if step == "switched":
|
||||
raise SimulatedCrash(step)
|
||||
|
||||
with self.assertRaises(SimulatedCrash):
|
||||
self.move(checkpoint=checkpoint)
|
||||
models.PromptModel.close_all_instances()
|
||||
ll.clear_override()
|
||||
|
||||
def test_a_restart_renames_the_old_files_never_deletes(self):
|
||||
self._crash_after_switch()
|
||||
self.assertTrue(os.path.isfile(self.old_db))
|
||||
restarted = self.location()
|
||||
self.assertEqual(restarted.db_path, self.new_db)
|
||||
marker = ll.read_marker(self.target)
|
||||
self.assertEqual(
|
||||
(marker["from"], marker["config_from"]), (self.old_db, self.config)
|
||||
)
|
||||
renamed = lm.finish_interrupted_move(restarted)
|
||||
self.assertEqual(
|
||||
sorted(renamed),
|
||||
[
|
||||
"config.json.moved-to-__promptmanager",
|
||||
"prompts.db.moved-to-__promptmanager",
|
||||
],
|
||||
)
|
||||
self.assertFalse(os.path.exists(self.old_db))
|
||||
moved_config = os.path.join(self.ext, "config.json.moved-to-__promptmanager")
|
||||
with open(moved_config, encoding="utf-8") as handle:
|
||||
self.assertEqual(
|
||||
json.load(handle)["integrations"]["lora_manager"]["civitai_api_key"],
|
||||
KEY,
|
||||
)
|
||||
self.assertEqual(lm.finish_interrupted_move(self.location()), [])
|
||||
|
||||
def test_nothing_is_renamed_while_the_old_file_is_in_use(self):
|
||||
self._crash_after_switch()
|
||||
reader = sqlite3.connect(self.old_db)
|
||||
try:
|
||||
reader.execute("BEGIN")
|
||||
reader.execute("SELECT COUNT(*) FROM prompts").fetchone()
|
||||
PromptDatabase(self.old_db).save_prompt("in the wal")
|
||||
self.assertEqual(lm.finish_interrupted_move(self.location()), [])
|
||||
self.assertTrue(os.path.isfile(self.old_db))
|
||||
finally:
|
||||
reader.rollback()
|
||||
reader.close()
|
||||
models.PromptModel.close_all_instances()
|
||||
|
||||
def test_a_node_folder_library_has_nothing_to_finish(self):
|
||||
self.assertEqual(lm.finish_interrupted_move(self.location()), [])
|
||||
self.assertTrue(os.path.isfile(self.old_db))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,332 @@
|
||||
"""ManagedConnection: never closed under the thread using it.
|
||||
|
||||
Closing a sqlite3 connection from one thread while another is inside a
|
||||
statement on it is a use-after-free in SQLite ("Windows fatal exception:
|
||||
access violation" in CI). A retired connection is closed at once when idle,
|
||||
otherwise interrupted and closed by its owner as the call ends; it never
|
||||
commits again.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import weakref
|
||||
from unittest import mock
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from database.managed_connection import ( # noqa: E402
|
||||
ManagedConnection,
|
||||
ManagedCursor,
|
||||
await_closed,
|
||||
retire_connections,
|
||||
)
|
||||
from database.models import PromptModel # noqa: E402
|
||||
from tests.open_handles import assert_closed # noqa: E402
|
||||
|
||||
WAIT = 30
|
||||
|
||||
|
||||
class _Case(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.path = os.path.join(self._tmp.name, "managed.db")
|
||||
with contextlib.closing(sqlite3.connect(self.path)) as conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("CREATE TABLE t(x INTEGER)")
|
||||
conn.execute("INSERT INTO t VALUES (0)")
|
||||
conn.commit()
|
||||
self.opened = []
|
||||
|
||||
def tearDown(self):
|
||||
for conn in self.opened:
|
||||
conn.close()
|
||||
try:
|
||||
assert_closed(self, self._tmp.name)
|
||||
finally:
|
||||
self._tmp.cleanup()
|
||||
|
||||
def connect(self, **kwargs):
|
||||
conn = sqlite3.connect(
|
||||
self.path,
|
||||
check_same_thread=False,
|
||||
timeout=WAIT,
|
||||
factory=ManagedConnection,
|
||||
**kwargs,
|
||||
)
|
||||
self.opened.append(conn)
|
||||
return conn
|
||||
|
||||
def value(self):
|
||||
with contextlib.closing(sqlite3.connect(self.path)) as conn:
|
||||
return conn.execute("SELECT x FROM t").fetchone()[0]
|
||||
|
||||
|
||||
class TestRetire(_Case):
|
||||
def test_an_idle_connection_retired_from_another_thread_closes_at_once(self):
|
||||
conn = self.connect()
|
||||
conn.execute("SELECT 1").fetchone()
|
||||
closed = []
|
||||
thread = threading.Thread(target=lambda: closed.append(conn.retire()))
|
||||
thread.start()
|
||||
thread.join(WAIT)
|
||||
self.assertEqual(closed, [True])
|
||||
self.assertTrue(conn.wait_closed(0))
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
conn.execute("SELECT 1")
|
||||
self.assertTrue(conn.retire()) # retiring again is harmless
|
||||
|
||||
def test_a_connection_in_use_is_closed_by_its_owner_when_the_call_ends(self):
|
||||
conn = self.connect()
|
||||
holder = sqlite3.connect(self.path, isolation_level=None)
|
||||
self.opened.append(holder)
|
||||
holder.execute("BEGIN IMMEDIATE")
|
||||
started = threading.Event()
|
||||
outcome = {}
|
||||
conn.set_trace_callback(
|
||||
lambda sql: started.set() if sql.startswith("UPDATE") else None
|
||||
)
|
||||
|
||||
def write():
|
||||
try:
|
||||
with conn:
|
||||
conn.execute("UPDATE t SET x = 1")
|
||||
outcome["ok"] = True
|
||||
except sqlite3.Error as error:
|
||||
outcome["error"] = error
|
||||
|
||||
thread = threading.Thread(target=write)
|
||||
thread.start()
|
||||
self.assertTrue(started.wait(WAIT))
|
||||
# The owner is waiting on our lock inside its UPDATE: not closed here.
|
||||
self.assertFalse(conn.retire())
|
||||
self.assertFalse(conn.retire())
|
||||
self.assertFalse(conn.wait_closed(0))
|
||||
self.assertEqual(retire_connections([conn]), [conn])
|
||||
holder.execute("ROLLBACK")
|
||||
thread.join(WAIT)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertTrue(conn.wait_closed(WAIT))
|
||||
self.assertIsInstance(outcome.get("error"), sqlite3.Error)
|
||||
self.assertEqual(self.value(), 0)
|
||||
|
||||
def test_a_write_retired_before_its_commit_is_rolled_back(self):
|
||||
conn = self.connect()
|
||||
conn.execute("UPDATE t SET x = 2") # an implicit transaction is open
|
||||
thread = threading.Thread(target=conn.retire)
|
||||
thread.start()
|
||||
thread.join(WAIT)
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
conn.commit()
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
with conn:
|
||||
pass
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
conn.rollback()
|
||||
self.assertEqual(self.value(), 0)
|
||||
|
||||
def test_close_is_retire(self):
|
||||
conn = self.connect()
|
||||
conn.close()
|
||||
self.assertTrue(conn.wait_closed(0))
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
conn.commit()
|
||||
|
||||
def test_plain_connections_are_closed_directly(self):
|
||||
plain = sqlite3.connect(self.path)
|
||||
self.assertEqual(retire_connections([plain]), [])
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
plain.execute("SELECT 1")
|
||||
|
||||
def test_awaiting_a_connection_still_in_use_times_out_with_a_warning(self):
|
||||
conn = self.connect()
|
||||
conn._enter() # its owner is inside a call
|
||||
self.assertEqual(retire_connections([conn]), [conn])
|
||||
logger = mock.Mock()
|
||||
self.assertFalse(await_closed([conn], 0, logger))
|
||||
logger.warning.assert_called_once()
|
||||
conn._leave()
|
||||
self.assertTrue(await_closed([conn], WAIT, logger))
|
||||
|
||||
|
||||
class TestCalls(_Case):
|
||||
def test_every_cursor_call_works_and_is_refused_once_retired(self):
|
||||
conn = self.connect()
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
self.assertIsInstance(cursor, ManagedCursor)
|
||||
conn.executemany("INSERT INTO t VALUES (?)", [(1,), (2,)])
|
||||
conn.executescript("INSERT INTO t VALUES (3);")
|
||||
self.assertEqual(
|
||||
[row["x"] for row in conn.execute("SELECT x FROM t ORDER BY x")],
|
||||
[0, 1, 2, 3],
|
||||
)
|
||||
cursor.execute("SELECT x FROM t ORDER BY x")
|
||||
self.assertEqual(cursor.fetchone()["x"], 0)
|
||||
self.assertEqual([r["x"] for r in cursor.fetchmany(2)], [1, 2])
|
||||
self.assertEqual([r["x"] for r in cursor.fetchall()], [3])
|
||||
cursor.executemany("UPDATE t SET x = x WHERE x = ?", [(1,)])
|
||||
cursor.executescript("SELECT 1;")
|
||||
cursor.close()
|
||||
conn.commit()
|
||||
conn.rollback()
|
||||
with conn:
|
||||
conn.execute("DELETE FROM t WHERE x = 3")
|
||||
with self.assertRaises(ZeroDivisionError):
|
||||
with conn:
|
||||
conn.execute("DELETE FROM t")
|
||||
raise ZeroDivisionError
|
||||
self.assertEqual(self.value(), 0)
|
||||
copy = sqlite3.connect(":memory:")
|
||||
self.opened.append(copy)
|
||||
conn.backup(copy)
|
||||
self.assertEqual(copy.execute("SELECT COUNT(*) FROM t").fetchone()[0], 3)
|
||||
|
||||
reader = conn.execute("SELECT x FROM t")
|
||||
conn.retire()
|
||||
for call in (
|
||||
lambda: reader.fetchone(),
|
||||
lambda: reader.fetchmany(1),
|
||||
lambda: reader.fetchall(),
|
||||
lambda: next(reader),
|
||||
lambda: reader.execute("SELECT 1"),
|
||||
lambda: reader.executemany("SELECT ?", [(1,)]),
|
||||
lambda: reader.executescript("SELECT 1;"),
|
||||
lambda: conn.backup(copy),
|
||||
):
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
call()
|
||||
# As with sqlite3's own cursors (the result varies by Python version).
|
||||
with contextlib.suppress(sqlite3.ProgrammingError):
|
||||
reader.close()
|
||||
|
||||
|
||||
class _CursorOpenedDuringClose(weakref.WeakSet):
|
||||
"""The cursor set of a connection being closed; a cursor opens mid-iteration.
|
||||
|
||||
``hook`` runs after the first cursor is yielded, where a thread switch
|
||||
could let the owner call ``conn.cursor()`` (Python 3.10-3.12 copy a
|
||||
WeakSet one item at a time). It runs in the closing thread itself, so
|
||||
the result does not depend on scheduling.
|
||||
"""
|
||||
|
||||
def __init__(self, items, hook):
|
||||
super().__init__(items)
|
||||
self.hook = hook
|
||||
|
||||
def __iter__(self):
|
||||
hooked = False
|
||||
for item in super().__iter__():
|
||||
yield item
|
||||
if not hooked:
|
||||
hooked = True
|
||||
self.hook()
|
||||
|
||||
|
||||
class TestReviewRound(_Case):
|
||||
"""PR #321 review: I1 (cursor set race), M1 and M2 (unguarded paths)."""
|
||||
|
||||
def test_a_cursor_opened_while_the_connection_closes_cannot_break_retire(self):
|
||||
conn = self.connect()
|
||||
kept = [conn.cursor() for _ in range(3)]
|
||||
refused = []
|
||||
|
||||
def open_cursor():
|
||||
try:
|
||||
kept.append(conn.cursor())
|
||||
except sqlite3.ProgrammingError as error:
|
||||
refused.append(error)
|
||||
|
||||
conn._cursors = _CursorOpenedDuringClose(conn._cursors, open_cursor)
|
||||
self.assertTrue(conn.retire()) # no "Set changed size during iteration"
|
||||
self.assertTrue(conn.wait_closed(0))
|
||||
self.assertEqual(len(refused), 1)
|
||||
self.assertEqual(len(kept), 3)
|
||||
|
||||
def test_a_retired_connection_refuses_new_cursors_and_scripts(self):
|
||||
conn = self.connect()
|
||||
conn._enter() # its owner is inside a call: retired, not yet closed
|
||||
try:
|
||||
self.assertFalse(conn.retire())
|
||||
for call in (
|
||||
conn.cursor,
|
||||
lambda: conn.executescript("CREATE TABLE late(x);"),
|
||||
lambda: conn.in_transaction,
|
||||
lambda: conn.create_function("f", 0, lambda: 1),
|
||||
):
|
||||
with self.subTest(call=call):
|
||||
with self.assertRaises(sqlite3.ProgrammingError):
|
||||
call()
|
||||
finally:
|
||||
conn._leave()
|
||||
self.assertTrue(conn.wait_closed(0))
|
||||
with contextlib.closing(sqlite3.connect(self.path)) as check:
|
||||
tables = check.execute(
|
||||
"SELECT name FROM sqlite_master WHERE name='late'"
|
||||
).fetchall()
|
||||
self.assertEqual(tables, [])
|
||||
|
||||
def test_one_connection_failing_to_retire_does_not_skip_the_rest(self):
|
||||
class Broken(ManagedConnection):
|
||||
def retire(self):
|
||||
raise RuntimeError("retire failed")
|
||||
|
||||
broken = sqlite3.connect(self.path, check_same_thread=False, factory=Broken)
|
||||
try:
|
||||
after = self.connect()
|
||||
logger = mock.Mock()
|
||||
pending = retire_connections([broken, after], logger)
|
||||
self.assertEqual(pending, [])
|
||||
self.assertTrue(after.wait_closed(0))
|
||||
logger.exception.assert_called_once()
|
||||
finally:
|
||||
ManagedConnection.retire(broken)
|
||||
|
||||
def test_a_restore_marks_its_backup_target_in_use(self):
|
||||
model = PromptModel(os.path.join(self._tmp.name, "live.db"))
|
||||
try:
|
||||
self._restore_marks_its_target(model)
|
||||
finally:
|
||||
model.close_all()
|
||||
|
||||
def _restore_marks_its_target(self, model):
|
||||
source_path = os.path.join(self._tmp.name, "source.db")
|
||||
PromptModel(source_path).close_all()
|
||||
target = model.get_connection()
|
||||
depths = []
|
||||
connect = sqlite3.connect
|
||||
|
||||
class Source(sqlite3.Connection):
|
||||
def backup(self, into, *args, **kwargs):
|
||||
depths.append(into._depth)
|
||||
return super().backup(into, *args, **kwargs)
|
||||
|
||||
def connect_source(*args, **kwargs):
|
||||
if "factory" in kwargs:
|
||||
return connect(*args, **kwargs)
|
||||
return connect(*args, **kwargs, factory=Source)
|
||||
|
||||
with mock.patch("database.models.sqlite3.connect", connect_source):
|
||||
model._replace_contents_from(source_path)
|
||||
self.assertEqual(depths, [1])
|
||||
self.assertEqual(target._depth, 0)
|
||||
|
||||
|
||||
class TestModelConnections(_Case):
|
||||
def test_the_model_hands_out_managed_connections_and_closes_them(self):
|
||||
model = PromptModel(os.path.join(self._tmp.name, "model.db"))
|
||||
conn = model.get_connection()
|
||||
self.assertIsInstance(conn, ManagedConnection)
|
||||
model.close()
|
||||
self.assertTrue(conn.wait_closed(0))
|
||||
conn = model.get_connection()
|
||||
model.close_all()
|
||||
self.assertTrue(conn.wait_closed(0))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -150,7 +150,15 @@ class TestSourceStatus(SourcesTestCase):
|
||||
)
|
||||
self.assertEqual(
|
||||
data["sources"]["effective"],
|
||||
[{"label": "a", "path": "output/a", "root_index": 0, "watch": "manual"}],
|
||||
[
|
||||
{
|
||||
"label": "a",
|
||||
"path": "output/a",
|
||||
"root_index": 0,
|
||||
"watch": "manual",
|
||||
"in_comfyui": True,
|
||||
}
|
||||
],
|
||||
)
|
||||
self.assertFalse(data["sources"]["auto_detected"])
|
||||
# The legacy keys are unchanged.
|
||||
@@ -183,10 +191,21 @@ class TestSourceStatus(SourcesTestCase):
|
||||
"path": "output",
|
||||
"root_index": 0,
|
||||
"watch": "manual",
|
||||
"in_comfyui": True,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
def test_a_source_outside_comfyui_is_not_marked_in_comfyui(self):
|
||||
# Sources & scans prefixes "ComfyUI/" only to a path relative to the
|
||||
# ComfyUI base; a folder outside it is shown by its own name.
|
||||
from py.api import sources
|
||||
|
||||
self.assertTrue(sources.in_comfyui_base(str(self.a)))
|
||||
self.assertTrue(sources.in_comfyui_base(str(self.comfy_dir)))
|
||||
self.assertFalse(sources.in_comfyui_base(self.tmpdir))
|
||||
self.assertFalse(sources.in_comfyui_base(os.path.join(self.tmpdir, "Renders")))
|
||||
|
||||
async def test_watching_requires_live_observer_with_root(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
|
||||
|
||||
|
||||
@@ -277,6 +277,7 @@ class TestOverviewRoute(SystemAPITestCase):
|
||||
"backup",
|
||||
"needs_review",
|
||||
"sources",
|
||||
"location",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,436 @@
|
||||
"""Convert outdated thumbnails (#317): an explicit, cancellable Activity job.
|
||||
|
||||
``POST /prompt_manager/jobs/thumbnails {mode: "convert"}`` runs under the
|
||||
thumbnail job's kind, admission (``guarded``) and worker threads. For every
|
||||
media file whose preview is in another format than the one written now, it
|
||||
writes the new preview from the SOURCE (verified, atomic), then removes that
|
||||
file's old-format previews under ``thumbnails/``. A failure keeps the old
|
||||
preview; previews whose source is gone (orphans) are never touched. The final
|
||||
line reports converted / failed and bytes before → after.
|
||||
|
||||
Ordering is driven by events, never by timers.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
try:
|
||||
from tests.test_api_images import ThumbnailTestCase, make_png
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_images import ThumbnailTestCase, make_png
|
||||
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
from py.api import thumbnail_encode # noqa: E402
|
||||
from py.config import PromptManagerConfig # noqa: E402
|
||||
|
||||
JOB = "/prompt_manager/jobs/thumbnails"
|
||||
PLAN = "/prompt_manager/images/thumbnails/plan"
|
||||
STATUS = "/prompt_manager/images/thumbnails/status"
|
||||
WAIT = 10
|
||||
|
||||
|
||||
def big_png(path, size=(512, 512)):
|
||||
"""A noisy PNG: its PNG preview is large, a WebP one much smaller."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
Image.effect_noise(size, 40).convert("RGB").save(path, "PNG")
|
||||
return path
|
||||
|
||||
|
||||
class ConvertCase(ThumbnailTestCase):
|
||||
async def asyncSetUp(self):
|
||||
await super().asyncSetUp()
|
||||
saved = PromptManagerConfig.get_config()["thumbnails"]
|
||||
self.addCleanup(PromptManagerConfig.update_config, {"thumbnails": saved})
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
thumbnail_encode.reset_encoders()
|
||||
self.addCleanup(thumbnail_encode.reset_encoders)
|
||||
self.addCleanup(self.api.close_auto_thumbnails)
|
||||
|
||||
def _legacy(self, name, suffix=".png", source=True):
|
||||
"""A media file (unless ``source`` is False) and its old-format preview."""
|
||||
if source:
|
||||
big_png(self.output_dir / f"{name}.png")
|
||||
preview = big_png(self._thumb_path(name, suffix), size=(256, 256))
|
||||
if source:
|
||||
later = (self.output_dir / f"{name}.png").stat().st_mtime + 5
|
||||
os.utime(preview, (later, later))
|
||||
return preview
|
||||
|
||||
async def _convert(self):
|
||||
resp = await self.client.post(JOB, json={"mode": "convert"})
|
||||
body = await resp.json()
|
||||
self.assertEqual(resp.status, 202, body)
|
||||
job = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(job.task, WAIT)
|
||||
return body["job"], job
|
||||
|
||||
|
||||
class TestConvertOutdated(ConvertCase):
|
||||
async def test_plan_counts_the_outdated_previews(self):
|
||||
self._legacy("a")
|
||||
self._legacy("b")
|
||||
make_png(self.output_dir / "new.png") # missing: not this job's work
|
||||
plan = await (await self.client.get(PLAN, params={"mode": "convert"})).json()
|
||||
self.assertEqual(
|
||||
(plan["mode"], plan["targets"], plan["outdated"]), ("convert", 2, 2)
|
||||
)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
|
||||
async def test_converts_from_the_source_and_removes_the_old_preview(self):
|
||||
old = self._legacy("a")
|
||||
original = (self.output_dir / "a.png").read_bytes()
|
||||
orphan = self._legacy("gone", source=False)
|
||||
make_png(self.output_dir / "new.png")
|
||||
before = old.stat().st_size
|
||||
|
||||
summary, job = await self._convert()
|
||||
|
||||
self.assertEqual(
|
||||
(summary["kind"], summary["admission"], summary["cancellable"]),
|
||||
("thumbnails", "guarded", True),
|
||||
)
|
||||
self.assertEqual(summary["title"], "Convert outdated thumbnails")
|
||||
self.assertEqual(summary["scope"]["mode"], "convert")
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.counters["generated"], 1)
|
||||
self.assertEqual(job.counters["errors"], 0)
|
||||
new = self._thumb_path("a")
|
||||
with Image.open(new) as img:
|
||||
self.assertEqual((img.format, max(img.size)), ("WEBP", 512))
|
||||
self.assertFalse(old.exists(), "the old PNG preview is removed")
|
||||
self.assertEqual((self.output_dir / "a.png").read_bytes(), original)
|
||||
self.assertTrue(orphan.exists(), "an orphan preview is never touched")
|
||||
self.assertFalse(self._thumb_path("new").exists(), "missing is not converted")
|
||||
after = new.stat().st_size
|
||||
self.assertRegex(job.status_line, r"^1 converted · 0 failed · .+ → .+$")
|
||||
self.assertTrue(any("Conversion finished" in line for line in job.log))
|
||||
self.assertLess(after, before)
|
||||
status = await (await self.client.get(STATUS)).json()
|
||||
self.assertEqual((status["outdated"], status["pending"]), (0, 1))
|
||||
|
||||
async def test_a_source_that_fails_keeps_its_old_preview(self):
|
||||
old = self._legacy("bad")
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG broken")
|
||||
later = old.stat().st_mtime + 10
|
||||
os.utime(old, (later, later))
|
||||
keep = old.read_bytes()
|
||||
|
||||
_, job = await self._convert()
|
||||
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(job.counters["generated"], 0)
|
||||
self.assertEqual(old.read_bytes(), keep)
|
||||
self.assertFalse(self._thumb_path("bad").exists())
|
||||
self.assertRegex(job.status_line, r"^0 converted · 1 failed")
|
||||
|
||||
async def test_a_video_without_a_frame_keeps_its_old_preview(self):
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"no decoder")
|
||||
old = big_png(self._thumb_path("clip", ".jpg"), size=(64, 64))
|
||||
keep = old.read_bytes()
|
||||
with (
|
||||
patch.object(self.api, "_opencv_frame", return_value=None),
|
||||
patch.object(self.api, "_ffmpeg_frame", return_value=None),
|
||||
):
|
||||
_, job = await self._convert()
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(old.read_bytes(), keep)
|
||||
self.assertFalse(self._thumb_path("clip").exists(), "never the placeholder")
|
||||
|
||||
async def test_an_old_preview_that_is_a_link_is_left(self):
|
||||
big_png(self.output_dir / "l.png")
|
||||
target = big_png(self.comfy_dir / "elsewhere.png", size=(32, 32))
|
||||
link = self._thumb_path("l", ".png")
|
||||
link.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._make_symlink(target, link)
|
||||
_, job = await self._convert()
|
||||
self.assertTrue(link.is_symlink())
|
||||
self.assertTrue(target.exists())
|
||||
self.assertEqual(job.counters["errors"], 0)
|
||||
|
||||
async def test_a_preview_already_current_only_loses_its_leftovers(self):
|
||||
old = self._legacy("c")
|
||||
current = big_png(self._thumb_path("c"), size=(64, 64))
|
||||
later = old.stat().st_mtime + 5
|
||||
os.utime(current, (later, later))
|
||||
written = current.read_bytes()
|
||||
_, job = await self._convert()
|
||||
self.assertEqual(job.counters["generated"], 1)
|
||||
self.assertEqual(current.read_bytes(), written, "not written again")
|
||||
self.assertFalse(old.exists())
|
||||
|
||||
async def test_cancel_stops_at_a_file_boundary_keeping_what_converted(self):
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
olds = [self._legacy(f"p{i}") for i in range(6)]
|
||||
done = []
|
||||
real = self.api._convert_one
|
||||
job_ids = []
|
||||
|
||||
def counting(*args, **kwargs):
|
||||
result = real(*args, **kwargs)
|
||||
done.append(result)
|
||||
if len(done) == 2:
|
||||
self.assertTrue(self.api._jobs.cancel(job_ids[0]))
|
||||
return result
|
||||
|
||||
self.api._convert_one = counting
|
||||
resp = await self.client.post(JOB, json={"mode": "convert"})
|
||||
job_ids.append((await resp.json())["job"]["id"])
|
||||
job = self.api._jobs.get(job_ids[0])
|
||||
await asyncio.wait_for(job.task, WAIT)
|
||||
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.counters["generated"], 2)
|
||||
converted = [p for p in olds if not p.exists()]
|
||||
self.assertEqual(len(converted), 2)
|
||||
for old in olds:
|
||||
webp = old.with_suffix(".webp")
|
||||
self.assertEqual(webp.exists(), not old.exists(), old.name)
|
||||
self.assertTrue(any("Conversion cancelled" in line for line in job.log))
|
||||
|
||||
async def test_refused_beside_a_running_thumbnail_job_or_scan(self):
|
||||
self._legacy("a")
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.addCleanup(release.set)
|
||||
real = self.api._convert_one
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(WAIT)
|
||||
return real(*args, **kwargs)
|
||||
|
||||
self.api._convert_one = gated
|
||||
resp = await self.client.post(JOB, json={"mode": "convert"})
|
||||
first = (await resp.json())["job"]["id"]
|
||||
loop = asyncio.get_running_loop()
|
||||
self.assertTrue(await loop.run_in_executor(None, entered.wait, WAIT))
|
||||
again = await self.client.post(JOB, json={"mode": "missing"})
|
||||
self.assertEqual(again.status, 409)
|
||||
scan = await self.client.post("/prompt_manager/jobs/scan")
|
||||
self.assertEqual(scan.status, 409)
|
||||
release.set()
|
||||
await asyncio.wait_for(self.api._jobs.get(first).task, WAIT)
|
||||
|
||||
async def test_the_legacy_gallery_stream_never_attaches_to_a_conversion(self):
|
||||
try:
|
||||
from tests.test_api_images import parse_sse
|
||||
except ImportError:
|
||||
from test_api_images import parse_sse
|
||||
|
||||
self._legacy("a")
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.addCleanup(release.set)
|
||||
real = self.api._convert_one
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(WAIT)
|
||||
return real(*args, **kwargs)
|
||||
|
||||
self.api._convert_one = gated
|
||||
resp = await self.client.post(JOB, json={"mode": "convert"})
|
||||
first = (await resp.json())["job"]["id"]
|
||||
loop = asyncio.get_running_loop()
|
||||
self.assertTrue(await loop.run_in_executor(None, entered.wait, WAIT))
|
||||
stream = await self.client.post(self.PROGRESS_URL)
|
||||
events = parse_sse(await stream.text())
|
||||
self.assertEqual(events[-1][0], "error")
|
||||
self.assertIn("Convert outdated thumbnails", events[-1][1]["error"])
|
||||
release.set()
|
||||
await asyncio.wait_for(self.api._jobs.get(first).task, WAIT)
|
||||
|
||||
async def test_nothing_outdated_says_so(self):
|
||||
big_png(self.output_dir / "a.png")
|
||||
_, job = await self._convert()
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.status_line, "No outdated previews to convert")
|
||||
|
||||
async def test_the_source_vanishing_keeps_the_old_preview(self):
|
||||
old = self._legacy("v")
|
||||
real = self.api._generate_one
|
||||
|
||||
def vanish(src, *args, **kwargs):
|
||||
os.remove(src)
|
||||
return real(src, *args, **kwargs)
|
||||
|
||||
self.api._generate_one = vanish
|
||||
_, job = await self._convert()
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertTrue(old.exists(), "a source gone mid-run makes an orphan")
|
||||
|
||||
async def test_same_stem_sources_convert_once_and_count_bytes_once(self):
|
||||
# Review I2 / R2: x.png and x.jpg share x_thumb.webp. The pair is
|
||||
# converted once, from the image first in the source order (.png),
|
||||
# and every removed file's bytes count once.
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 2
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
from py.api.thumbnail_convert import readable_bytes
|
||||
|
||||
png = self.output_dir / "x.png"
|
||||
Image.new("RGB", (300, 300), (220, 20, 20)).save(png)
|
||||
jpg = self.output_dir / "x.jpg"
|
||||
Image.new("RGB", (300, 300), (20, 20, 220)).save(jpg)
|
||||
old_png = big_png(self._thumb_path("x", ".png"), size=(200, 200))
|
||||
old_jpg = self._thumb_path("x", ".jpg")
|
||||
Image.effect_noise((200, 200), 40).convert("RGB").save(old_jpg, "JPEG")
|
||||
later = max(png.stat().st_mtime, jpg.stat().st_mtime) + 5
|
||||
for path in (old_png, old_jpg):
|
||||
os.utime(path, (later, later))
|
||||
before = old_png.stat().st_size + old_jpg.stat().st_size
|
||||
|
||||
_, job = await self._convert()
|
||||
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(
|
||||
(job.counters["generated"], job.counters["errors"]), (2, 0), job.log
|
||||
)
|
||||
new = self._thumb_path("x")
|
||||
self.assertFalse(old_png.exists() or old_jpg.exists())
|
||||
with Image.open(new) as img:
|
||||
red, _green, blue = img.convert("RGB").getpixel((5, 5))
|
||||
self.assertGreater(red, blue, "the image first in the source order wins")
|
||||
self.assertEqual(
|
||||
job.status_line,
|
||||
f"2 converted · 0 failed · {readable_bytes(before)} → "
|
||||
f"{readable_bytes(new.stat().st_size)}",
|
||||
)
|
||||
|
||||
async def test_an_old_preview_already_gone_counts_as_removed(self):
|
||||
old = self._legacy("g")
|
||||
real = self.api._generate_one
|
||||
|
||||
def remove_first(src, *args, **kwargs):
|
||||
os.remove(old)
|
||||
return real(src, *args, **kwargs)
|
||||
|
||||
self.api._generate_one = remove_first
|
||||
_, job = await self._convert()
|
||||
self.assertEqual((job.counters["generated"], job.counters["errors"]), (1, 0))
|
||||
self.assertRegex(job.status_line, r"^1 converted · 0 failed · 0 B → ")
|
||||
|
||||
async def test_a_current_preview_that_does_not_decode_is_written_again(self):
|
||||
# Review M4: a kept current preview is decoded before old ones go.
|
||||
old = self._legacy("m")
|
||||
current = self._thumb_path("m")
|
||||
current.write_bytes(b"RIFF not a webp")
|
||||
later = old.stat().st_mtime + 5
|
||||
os.utime(current, (later, later))
|
||||
_, job = await self._convert()
|
||||
self.assertEqual((job.counters["generated"], job.counters["errors"]), (1, 0))
|
||||
with Image.open(current) as img:
|
||||
self.assertEqual(img.format, "WEBP")
|
||||
self.assertFalse(old.exists())
|
||||
|
||||
async def test_a_format_change_underneath_stops_the_job(self):
|
||||
# Review I4 (defense in depth): the job never removes previews in a
|
||||
# format chosen after it started.
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
olds = [self._legacy(f"f{i}") for i in range(4)]
|
||||
real = self.api._convert_one
|
||||
calls = []
|
||||
|
||||
def switching(*args, **kwargs):
|
||||
result = real(*args, **kwargs)
|
||||
calls.append(result)
|
||||
if len(calls) == 1:
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "jpeg", "quality": 80, "edge": 512}}
|
||||
)
|
||||
jpeg = self._thumb_path("f3", ".jpg")
|
||||
Image.new("RGB", (16, 16)).save(jpeg, "JPEG")
|
||||
return result
|
||||
|
||||
self.api._convert_one = switching
|
||||
_, job = await self._convert()
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertIn("format changed", job.status_line)
|
||||
self.assertEqual(job.counters["generated"], 1)
|
||||
self.assertTrue(self._thumb_path("f3", ".jpg").exists(), "the new format stays")
|
||||
self.assertEqual(sum(1 for p in olds if p.exists()), 3)
|
||||
|
||||
async def test_the_new_preview_is_never_removed_as_an_old_one(self):
|
||||
# Re-review N1: on a case-insensitive disk x_thumb.JPG (an old
|
||||
# spelling) can be the x_thumb.jpg just written. A hard link stands
|
||||
# in for that second name of one file.
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "jpeg", "quality": 80, "edge": 512}}
|
||||
)
|
||||
src = self.output_dir / "x.jpg"
|
||||
Image.new("RGB", (64, 64), (200, 10, 10)).save(src, "JPEG")
|
||||
alias = self._thumb_path("x", ".JPG")
|
||||
alias.parent.mkdir(parents=True, exist_ok=True)
|
||||
Image.new("RGB", (16, 16)).save(alias, "JPEG")
|
||||
later = src.stat().st_mtime + 5
|
||||
os.utime(alias, (later, later))
|
||||
real = self.api._generate_one
|
||||
|
||||
def same_file(source, dst, *args, **kwargs):
|
||||
made = real(source, dst, *args, **kwargs)
|
||||
os.remove(alias)
|
||||
os.link(made["path"], alias)
|
||||
return made
|
||||
|
||||
self.api._generate_one = same_file
|
||||
_, job = await self._convert()
|
||||
self.assertTrue(self._thumb_path("x", ".jpg").exists())
|
||||
self.assertTrue(alias.exists(), "the new preview's other name is kept")
|
||||
self.assertEqual(job.counters["errors"], 0)
|
||||
|
||||
def test_same_file_falls_back_to_the_path_without_inode_numbers(self):
|
||||
from py.api import thumbnail_convert as tc
|
||||
|
||||
a = self.output_dir / "a_thumb.jpg"
|
||||
b = self.output_dir / "b_thumb.jpg"
|
||||
for path in (a, b):
|
||||
path.write_bytes(b"x")
|
||||
real_stat = os.stat
|
||||
|
||||
def no_inode(path, *args, **kwargs):
|
||||
result = real_stat(path, *args, **kwargs)
|
||||
return os.stat_result(
|
||||
(result.st_mode, 0, result.st_dev) + tuple(result)[3:]
|
||||
)
|
||||
|
||||
with patch.object(tc.os, "stat", side_effect=no_inode):
|
||||
self.assertFalse(tc._same_file(a, b))
|
||||
self.assertTrue(tc._same_file(a, a))
|
||||
self.assertTrue(tc._same_file(a, self.output_dir / "." / "a_thumb.jpg"))
|
||||
self.assertFalse(tc._same_file(a, self.output_dir / "gone.jpg"))
|
||||
|
||||
async def test_convert_goes_on_past_the_per_batch_cap(self):
|
||||
import py.api.images as images_module
|
||||
|
||||
olds = [self._legacy(f"c{i}") for i in range(8)]
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
|
||||
_, job = await self._convert()
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.counters["generated"], 8)
|
||||
self.assertFalse(any(old.exists() for old in olds))
|
||||
self.assertEqual(job.summary()["total"], 8)
|
||||
self.assertRegex(job.status_line, r"^8 converted · 0 failed · ")
|
||||
self.assertNotIn("next run", job.status_line)
|
||||
|
||||
def test_byte_sizes_are_readable(self):
|
||||
from py.api.thumbnail_convert import readable_bytes
|
||||
|
||||
self.assertEqual(readable_bytes(0), "0 B")
|
||||
self.assertEqual(readable_bytes(1536), "1.5 KB")
|
||||
self.assertEqual(readable_bytes(5 * 1024 * 1024), "5.0 MB")
|
||||
self.assertEqual(readable_bytes(3 * 1024**3), "3.0 GB")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,525 @@
|
||||
"""Thumbnail encoders (#317): WebP, JPEG and an optional JPEG XL encoder.
|
||||
|
||||
``py/api/thumbnail_encode.py`` writes every preview: the first frame fitted
|
||||
to the edge, encoded in the configured format into a temp sibling that is
|
||||
closed, verified to decode and renamed over the preview. JPEG XL needs no new
|
||||
dependency: Pillow (natively or through ``pillow_jxl``), ffmpeg's ``libjxl``
|
||||
or ``cjxl``, in that order. Encoder processes are tracked, bounded and
|
||||
killed on shutdown.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
sys.modules.setdefault("server", MagicMock())
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
from py.api import thumbnail_encode as te # noqa: E402
|
||||
from py.api.thumbnail_settings import ThumbnailSettings # noqa: E402
|
||||
|
||||
JXL_CODESTREAM = b"\xff\x0a"
|
||||
JXL_CONTAINER = b"\x00\x00\x00\x0cJXL \r\n\x87\n"
|
||||
|
||||
|
||||
def _ffmpeg_has_libjxl():
|
||||
try:
|
||||
return te._ffmpeg_has_libjxl()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class TempCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = Path(os.path.realpath(tempfile.mkdtemp()))
|
||||
self.addCleanup(shutil.rmtree, self.tmp, True)
|
||||
te.reset_encoders()
|
||||
self.addCleanup(te.reset_encoders)
|
||||
|
||||
def _image(self, size=(800, 400), color=(200, 30, 30), mode="RGB"):
|
||||
return Image.new(mode, size, color)
|
||||
|
||||
def _leftovers(self):
|
||||
return [p.name for p in self.tmp.rglob("*") if p.name.startswith(".")]
|
||||
|
||||
|
||||
class TestDistance(unittest.TestCase):
|
||||
def test_quality_maps_to_libjxl_distance(self):
|
||||
self.assertAlmostEqual(te.jxl_distance(80), 1.9)
|
||||
self.assertAlmostEqual(te.jxl_distance(85), 1.45)
|
||||
self.assertAlmostEqual(te.jxl_distance(90), 1.0)
|
||||
self.assertAlmostEqual(te.jxl_distance(30), 6.4)
|
||||
self.assertAlmostEqual(te.jxl_distance(1), 23.8676666, places=4)
|
||||
# A preview is never encoded lossless.
|
||||
self.assertAlmostEqual(te.jxl_distance(100), 0.1)
|
||||
|
||||
|
||||
class TestDetection(TempCase):
|
||||
def test_pillow_native_first(self):
|
||||
with patch.dict(Image.SAVE, {"JXL": object()}):
|
||||
self.assertEqual(te.detect_jxl_encoder(refresh=True), "pillow")
|
||||
|
||||
def test_pillow_plugin_is_imported_when_installed(self):
|
||||
def fake_import(name):
|
||||
if name != "pillow_jxl":
|
||||
raise ImportError(name)
|
||||
Image.SAVE["JXL"] = object()
|
||||
return object()
|
||||
|
||||
self.addCleanup(Image.SAVE.pop, "JXL", None)
|
||||
with patch.object(te.importlib, "import_module", side_effect=fake_import):
|
||||
self.assertEqual(te.detect_jxl_encoder(refresh=True), "pillow")
|
||||
|
||||
def _no_pillow(self):
|
||||
saved = Image.SAVE.pop("JXL", None)
|
||||
if saved is not None:
|
||||
self.addCleanup(Image.SAVE.__setitem__, "JXL", saved)
|
||||
return patch.object(te.importlib, "import_module", side_effect=ImportError)
|
||||
|
||||
def test_ffmpeg_with_libjxl_then_cjxl_then_none(self):
|
||||
with self._no_pillow():
|
||||
with patch.object(te, "_ffmpeg_has_libjxl", return_value=True):
|
||||
self.assertEqual(te.detect_jxl_encoder(refresh=True), "ffmpeg")
|
||||
with (
|
||||
patch.object(te, "_ffmpeg_has_libjxl", return_value=False),
|
||||
patch.object(te.shutil, "which", return_value="/usr/bin/cjxl"),
|
||||
):
|
||||
self.assertEqual(te.detect_jxl_encoder(refresh=True), "cjxl")
|
||||
with (
|
||||
patch.object(te, "_ffmpeg_has_libjxl", return_value=False),
|
||||
patch.object(te.shutil, "which", return_value=None),
|
||||
):
|
||||
self.assertIsNone(te.detect_jxl_encoder(refresh=True))
|
||||
status = te.encoder_status()
|
||||
self.assertEqual(
|
||||
status["jxl"],
|
||||
{"available": False, "encoder": None, "reason": te.JXL_UNAVAILABLE},
|
||||
)
|
||||
self.assertIn("pillow-jxl-plugin", te.JXL_UNAVAILABLE)
|
||||
self.assertTrue(status["webp"]["available"])
|
||||
self.assertTrue(status["jpeg"]["available"])
|
||||
|
||||
def test_detection_is_cached_until_refreshed(self):
|
||||
with (
|
||||
self._no_pillow(),
|
||||
patch.object(te, "_ffmpeg_has_libjxl", return_value=True) as probe,
|
||||
):
|
||||
te.detect_jxl_encoder(refresh=True)
|
||||
te.detect_jxl_encoder()
|
||||
te.encoder_status()
|
||||
self.assertEqual(probe.call_count, 1)
|
||||
|
||||
def test_ffmpeg_probe_reads_the_encoder_list(self):
|
||||
listing = " V....D libjxl libjxl JPEG XL (codec jpegxl)\n"
|
||||
done = subprocess.CompletedProcess([], 0, listing, "")
|
||||
with (
|
||||
patch.object(te, "run_encoder", return_value=done),
|
||||
patch.object(te.shutil, "which", return_value="/usr/bin/ffmpeg"),
|
||||
):
|
||||
self.assertTrue(te._ffmpeg_has_libjxl())
|
||||
other = subprocess.CompletedProcess([], 0, " V..... libwebp\n", "")
|
||||
with (
|
||||
patch.object(te, "run_encoder", return_value=other),
|
||||
patch.object(te.shutil, "which", return_value="/usr/bin/ffmpeg"),
|
||||
):
|
||||
self.assertFalse(te._ffmpeg_has_libjxl())
|
||||
with patch.object(te.shutil, "which", return_value=None):
|
||||
self.assertFalse(te._ffmpeg_has_libjxl())
|
||||
|
||||
|
||||
class TestFrame(TempCase):
|
||||
def test_first_frame_of_an_animation_is_a_still(self):
|
||||
path = self.tmp / "anim.gif"
|
||||
frames = [Image.new("RGB", (64, 64), c) for c in ((255, 0, 0), (0, 0, 255))]
|
||||
frames[0].save(path, save_all=True, append_images=frames[1:], duration=50)
|
||||
with Image.open(path) as img:
|
||||
img.seek(1)
|
||||
frame = te.thumbnail_frame(img, 32)
|
||||
self.assertEqual(frame.mode, "RGB")
|
||||
self.assertEqual(frame.size, (32, 32))
|
||||
red, green, blue = frame.getpixel((16, 16))
|
||||
self.assertGreater(red, 200)
|
||||
self.assertLess(blue, 50)
|
||||
|
||||
def test_modes_without_rgb_are_converted_and_fitted(self):
|
||||
for mode, color in (("RGBA", (1, 2, 3, 0)), ("I;16", 300), ("L", 9)):
|
||||
frame = te.thumbnail_frame(self._image((1000, 250), color, mode), 512)
|
||||
self.assertEqual((frame.mode, frame.size), ("RGB", (512, 128)))
|
||||
|
||||
|
||||
class TestWrite(TempCase):
|
||||
def _settings(self, fmt="webp", quality=80, edge=512):
|
||||
return ThumbnailSettings(fmt, quality, edge)
|
||||
|
||||
def test_webp_and_jpeg_are_written_in_their_format(self):
|
||||
for fmt, name, pil in (("webp", "a.webp", "WEBP"), ("jpeg", "a.jpg", "JPEG")):
|
||||
dst = self.tmp / name
|
||||
written = te.write_thumbnail(self._image(), dst, self._settings(fmt))
|
||||
self.assertEqual(written, dst)
|
||||
with Image.open(dst) as img:
|
||||
self.assertEqual(img.format, pil)
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
def test_quality_changes_the_bytes(self):
|
||||
noisy = Image.effect_noise((256, 256), 64).convert("RGB")
|
||||
low = te.write_thumbnail(
|
||||
noisy, self.tmp / "low.webp", self._settings(quality=10)
|
||||
)
|
||||
high = te.write_thumbnail(
|
||||
noisy, self.tmp / "high.webp", self._settings(quality=95)
|
||||
)
|
||||
self.assertLess(low.stat().st_size, high.stat().st_size)
|
||||
|
||||
def test_guard_false_keeps_the_old_preview(self):
|
||||
dst = self.tmp / "a.webp"
|
||||
dst.write_bytes(b"old")
|
||||
with self.assertRaises(te.PreviewWriteHeld):
|
||||
te.write_thumbnail(
|
||||
self._image(), dst, self._settings(), guard=lambda: False
|
||||
)
|
||||
self.assertEqual(dst.read_bytes(), b"old")
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
def test_a_file_that_does_not_decode_is_never_renamed_in(self):
|
||||
dst = self.tmp / "a.webp"
|
||||
dst.write_bytes(b"old")
|
||||
|
||||
def broken(img, tmp, settings, track):
|
||||
Path(tmp).write_bytes(b"not an image")
|
||||
|
||||
with patch.dict(te._WRITERS, {"webp": broken}):
|
||||
with self.assertRaises(te.ThumbnailVerifyError):
|
||||
te.write_thumbnail(self._image(), dst, self._settings())
|
||||
self.assertEqual(dst.read_bytes(), b"old")
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
def test_jxl_without_an_encoder_falls_back_to_webp_and_logs_once(self):
|
||||
with patch.object(te, "detect_jxl_encoder", return_value=None):
|
||||
with self.assertLogs("prompt_manager.thumbnails", logging.WARNING) as logs:
|
||||
first = te.write_thumbnail(
|
||||
self._image(), self.tmp / "a.jxl", self._settings("jxl")
|
||||
)
|
||||
second = te.write_thumbnail(
|
||||
self._image(), self.tmp / "b.jxl", self._settings("jxl")
|
||||
)
|
||||
self.assertEqual((first.name, second.name), ("a.webp", "b.webp"))
|
||||
self.assertEqual(len([m for m in logs.output if "JPEG XL" in m]), 1)
|
||||
with Image.open(first) as img:
|
||||
self.assertEqual(img.format, "WEBP")
|
||||
|
||||
def _write_jxl(self, encoder):
|
||||
dst = self.tmp / f"{encoder}.jxl"
|
||||
with patch.object(te, "detect_jxl_encoder", return_value=encoder):
|
||||
written = te.write_thumbnail(self._image(), dst, self._settings("jxl"))
|
||||
self.assertEqual(written, dst)
|
||||
head = dst.read_bytes()[:12]
|
||||
self.assertTrue(
|
||||
head.startswith(JXL_CODESTREAM) or head.startswith(JXL_CONTAINER), head
|
||||
)
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
@unittest.skipUnless(_ffmpeg_has_libjxl(), "ffmpeg without libjxl")
|
||||
def test_jxl_through_ffmpeg(self):
|
||||
self._write_jxl("ffmpeg")
|
||||
|
||||
@unittest.skipUnless(shutil.which("cjxl"), "no cjxl")
|
||||
def test_jxl_through_cjxl(self):
|
||||
self._write_jxl("cjxl")
|
||||
|
||||
def test_jxl_through_pillow_passes_the_quality(self):
|
||||
seen = {}
|
||||
|
||||
def save_jxl(im, fp, filename):
|
||||
seen.update(im.encoderinfo)
|
||||
fp.write(JXL_CODESTREAM + b"\x00" * 16)
|
||||
|
||||
Image.register_save("PMTESTJXL", save_jxl)
|
||||
self.addCleanup(Image.SAVE.pop, "PMTESTJXL", None)
|
||||
with (
|
||||
patch.object(te, "PILLOW_JXL_FORMAT", "PMTESTJXL"),
|
||||
patch.object(te, "detect_jxl_encoder", return_value="pillow"),
|
||||
):
|
||||
te.write_thumbnail(self._image(), self.tmp / "p.jxl", self._settings("jxl"))
|
||||
self.assertEqual(seen["quality"], 80)
|
||||
self.assertEqual(seen["num_threads"], 1)
|
||||
|
||||
def test_a_failing_jxl_encoder_raises_and_keeps_nothing(self):
|
||||
failed = subprocess.CompletedProcess([], 1, b"", b"boom")
|
||||
with (
|
||||
patch.object(te, "detect_jxl_encoder", return_value="cjxl"),
|
||||
patch.object(te, "run_encoder", return_value=failed),
|
||||
):
|
||||
with self.assertRaises(te.EncoderFailed):
|
||||
te.write_thumbnail(
|
||||
self._image(), self.tmp / "c.jxl", self._settings("jxl")
|
||||
)
|
||||
self.assertFalse((self.tmp / "c.jxl").exists())
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
def test_verify_accepts_a_jxl_signature_when_pillow_cannot_decode(self):
|
||||
path = self.tmp / "s.jxl"
|
||||
path.write_bytes(JXL_CONTAINER + b"\x00" * 8)
|
||||
with patch.object(te, "_pillow_decodes_jxl", return_value=False):
|
||||
te.verify_thumbnail(path, "jxl")
|
||||
path.write_bytes(b"\x89PNG")
|
||||
with self.assertRaises(te.ThumbnailVerifyError):
|
||||
te.verify_thumbnail(path, "jxl")
|
||||
|
||||
|
||||
class TestFfmpegNames(TempCase):
|
||||
"""Review I1: ffmpeg's image2 muxer reads ``%d`` in an output name as a
|
||||
sequence pattern; ``-update 1`` keeps the temp name literal."""
|
||||
|
||||
def test_the_ffmpeg_command_writes_one_literal_file(self):
|
||||
seen = []
|
||||
|
||||
def fake(cmd, **kwargs):
|
||||
seen.append(cmd)
|
||||
Path(cmd[-1]).write_bytes(JXL_CODESTREAM + b"\x00" * 8)
|
||||
return subprocess.CompletedProcess(cmd, 0, b"", b"")
|
||||
|
||||
with (
|
||||
patch.object(te, "detect_jxl_encoder", return_value="ffmpeg"),
|
||||
patch.object(te, "run_encoder", side_effect=fake),
|
||||
patch.object(te, "_pillow_decodes_jxl", return_value=False),
|
||||
):
|
||||
te.write_thumbnail(
|
||||
Image.new("RGB", (8, 8)),
|
||||
self.tmp / "a_thumb.jxl",
|
||||
ThumbnailSettings("jxl", 80, 64),
|
||||
)
|
||||
cmd = seen[0]
|
||||
self.assertEqual(cmd[cmd.index("-update") + 1], "1")
|
||||
self.assertLess(cmd.index("-update"), len(cmd) - 1)
|
||||
|
||||
@unittest.skipUnless(_ffmpeg_has_libjxl(), "ffmpeg without libjxl")
|
||||
def test_a_percent_in_the_name_is_written_under_that_name(self):
|
||||
dst = self.tmp / "50%done_thumb.jxl"
|
||||
with patch.object(te, "detect_jxl_encoder", return_value="ffmpeg"):
|
||||
written = te.write_thumbnail(
|
||||
Image.new("RGB", (32, 32), (10, 200, 10)),
|
||||
dst,
|
||||
ThumbnailSettings("jxl", 80, 64),
|
||||
)
|
||||
self.assertEqual(written, dst)
|
||||
self.assertTrue(dst.is_file())
|
||||
self.assertEqual(self._leftovers(), [], "no stray sequence-named temp")
|
||||
|
||||
|
||||
class TestStrayTemps(TempCase):
|
||||
"""Review I1: a temp left by a crash (or the old ``%`` bug) is swept."""
|
||||
|
||||
def _touch(self, name, age=3600):
|
||||
path = self.tmp / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(b"x" * 10)
|
||||
when = path.stat().st_mtime - age
|
||||
os.utime(path, (when, when))
|
||||
return path
|
||||
|
||||
def test_only_old_temp_siblings_are_removed(self):
|
||||
strays = [
|
||||
self._touch(".a_thumb.part-12-34.webp"),
|
||||
self._touch(".501one.part-1-2.jxl"),
|
||||
self._touch("sub/.b_thumb.src-5-6.png"),
|
||||
]
|
||||
kept = [
|
||||
self._touch("a_thumb.webp"),
|
||||
self._touch(".hidden"),
|
||||
self._touch("notes.part-1-2.webp"),
|
||||
self._touch(".c_thumb.part-x-2.webp"),
|
||||
self._touch(".young_thumb.part-7-8.webp", age=0),
|
||||
]
|
||||
# A temp of this process written since it started may be in use.
|
||||
own = self._touch(f".mine_thumb.part-{os.getpid()}-9.webp", age=0)
|
||||
count, size = te.sweep_stray_temps(self.tmp, min_age=60)
|
||||
self.assertEqual((count, size), (3, 30))
|
||||
for path in strays:
|
||||
self.assertFalse(path.exists(), path.name)
|
||||
for path in kept + [own]:
|
||||
self.assertTrue(path.exists(), path.name)
|
||||
|
||||
def test_without_an_age_every_temp_goes_but_links_stay(self):
|
||||
stray = self._touch(".a_thumb.part-1-2.webp", age=0)
|
||||
own = self._touch(f".mine_thumb.part-{os.getpid()}-9.webp", age=0)
|
||||
target = self._touch("target.bin")
|
||||
link = self.tmp / ".l_thumb.part-1-2.webp"
|
||||
try:
|
||||
link.symlink_to(target)
|
||||
except (OSError, NotImplementedError):
|
||||
link = None
|
||||
self.assertEqual(te.sweep_stray_temps(self.tmp, min_age=0)[0], 2)
|
||||
self.assertFalse(stray.exists() or own.exists())
|
||||
if link is not None:
|
||||
self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_an_own_pid_temp_older_than_this_process_is_swept(self):
|
||||
# Re-review N3: under stable PIDs (Docker) the previous run's crash
|
||||
# leftovers carry this process's PID; they predate its start.
|
||||
before = te.PROCESS_STARTED - 3600
|
||||
stale = self._touch(f".old_thumb.part-{os.getpid()}-9.webp")
|
||||
os.utime(stale, (before, before))
|
||||
live = self._touch(f".new_thumb.part-{os.getpid()}-9.webp", age=0)
|
||||
self.assertEqual(te.sweep_stray_temps(self.tmp, min_age=60)[0], 1)
|
||||
self.assertFalse(stale.exists())
|
||||
self.assertTrue(live.exists())
|
||||
|
||||
def test_a_missing_folder_sweeps_nothing(self):
|
||||
self.assertEqual(te.sweep_stray_temps(self.tmp / "none"), (0, 0))
|
||||
|
||||
|
||||
class TestEncoderVanishes(TempCase):
|
||||
"""Review I3: an encoder that disappears after detection is detected
|
||||
again once; with none left, that file is written as WebP and the
|
||||
availability Appearance reads is refreshed."""
|
||||
|
||||
def test_a_missing_binary_falls_back_to_webp_and_refreshes(self):
|
||||
found = iter(["cjxl", None, None])
|
||||
with (
|
||||
patch.object(te, "_detect", side_effect=lambda: next(found)),
|
||||
patch.object(te.shutil, "which", return_value=None),
|
||||
patch.object(te, "run_encoder", side_effect=FileNotFoundError("cjxl")),
|
||||
self.assertLogs("prompt_manager.thumbnails", logging.WARNING),
|
||||
):
|
||||
self.assertTrue(te.encoder_status()["jxl"]["available"])
|
||||
written = te.write_thumbnail(
|
||||
Image.new("RGB", (16, 16)),
|
||||
self.tmp / "v_thumb.jxl",
|
||||
ThumbnailSettings("jxl", 80, 64),
|
||||
)
|
||||
status = te.encoder_status()["jxl"]
|
||||
self.assertEqual(written.name, "v_thumb.webp")
|
||||
with Image.open(written) as img:
|
||||
self.assertEqual(img.format, "WEBP")
|
||||
self.assertEqual((status["available"], status["encoder"]), (False, None))
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
def test_another_encoder_found_again_is_used(self):
|
||||
found = iter(["cjxl", "ffmpeg"])
|
||||
used = []
|
||||
|
||||
def fake(cmd, **kwargs):
|
||||
used.append(cmd[0])
|
||||
if cmd[0] == "cjxl":
|
||||
raise FileNotFoundError("cjxl")
|
||||
Path(cmd[-1]).write_bytes(JXL_CODESTREAM + b"\x00" * 8)
|
||||
return subprocess.CompletedProcess(cmd, 0, b"", b"")
|
||||
|
||||
with (
|
||||
patch.object(te, "_detect", side_effect=lambda: next(found)),
|
||||
patch.object(
|
||||
te.shutil,
|
||||
"which",
|
||||
side_effect=lambda name: None if name == "cjxl" else name,
|
||||
),
|
||||
patch.object(te, "run_encoder", side_effect=fake),
|
||||
patch.object(te, "_pillow_decodes_jxl", return_value=False),
|
||||
):
|
||||
written = te.write_thumbnail(
|
||||
Image.new("RGB", (16, 16)),
|
||||
self.tmp / "w_thumb.jxl",
|
||||
ThumbnailSettings("jxl", 80, 64),
|
||||
)
|
||||
self.assertEqual(written.name, "w_thumb.jxl")
|
||||
self.assertEqual(used, ["cjxl", "ffmpeg"])
|
||||
self.assertEqual(self._leftovers(), [])
|
||||
|
||||
def test_a_failure_with_the_encoder_present_does_not_detect_again(self):
|
||||
# Re-review N2: one bad file never runs a probe (which may time out
|
||||
# under load and turn JPEG XL off).
|
||||
failed = subprocess.CompletedProcess([], 1, b"", b"bad input")
|
||||
with (
|
||||
patch.object(te, "_detect", return_value="cjxl") as detect,
|
||||
patch.object(te.shutil, "which", return_value="/usr/bin/cjxl"),
|
||||
patch.object(te, "run_encoder", return_value=failed),
|
||||
):
|
||||
with self.assertRaises(te.EncoderFailed):
|
||||
te.write_thumbnail(
|
||||
Image.new("RGB", (16, 16)),
|
||||
self.tmp / "x_thumb.jxl",
|
||||
ThumbnailSettings("jxl", 80, 64),
|
||||
)
|
||||
self.assertEqual(detect.call_count, 1, "no second probe")
|
||||
|
||||
def test_no_encoder_found_is_checked_again_on_the_settings_read(self):
|
||||
# Re-review N2: a probe that found nothing (a timeout) recovers.
|
||||
found = iter([None, "cjxl"])
|
||||
with patch.object(te, "_detect", side_effect=lambda: next(found)):
|
||||
self.assertIsNone(te.detect_jxl_encoder())
|
||||
self.assertEqual(te.encoder_status()["jxl"]["encoder"], "cjxl")
|
||||
self.assertEqual(te.detect_jxl_encoder(), "cjxl")
|
||||
|
||||
def test_no_encoder_found_is_checked_again_after_a_while(self):
|
||||
found = iter([None, "ffmpeg"])
|
||||
clock = [1000.0]
|
||||
with (
|
||||
patch.object(te, "_detect", side_effect=lambda: next(found)) as detect,
|
||||
patch.object(te.time, "monotonic", side_effect=lambda: clock[0]),
|
||||
):
|
||||
self.assertIsNone(te.detect_jxl_encoder())
|
||||
self.assertIsNone(te.detect_jxl_encoder(), "cached within the TTL")
|
||||
clock[0] += te.MISSING_ENCODER_TTL + 1
|
||||
self.assertEqual(te.detect_jxl_encoder(), "ffmpeg")
|
||||
self.assertEqual(te.detect_jxl_encoder(), "ffmpeg")
|
||||
self.assertEqual(detect.call_count, 2)
|
||||
|
||||
|
||||
class TestEncoderProcesses(TempCase):
|
||||
SLEEPER = [sys.executable, "-c", "import time; time.sleep(60)"]
|
||||
|
||||
def test_a_process_past_its_timeout_is_killed(self):
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
te.run_encoder(self.SLEEPER, timeout=0.5)
|
||||
|
||||
def test_shutdown_kills_running_encoders_and_refuses_new_ones(self):
|
||||
started = threading.Event()
|
||||
tracked = []
|
||||
|
||||
def track(process):
|
||||
tracked.append(process)
|
||||
started.set()
|
||||
|
||||
outcome = {}
|
||||
|
||||
def run():
|
||||
try:
|
||||
outcome["result"] = te.run_encoder(
|
||||
self.SLEEPER, timeout=60, track=track
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - recorded for the assertion
|
||||
outcome["error"] = exc
|
||||
|
||||
worker = threading.Thread(target=run)
|
||||
worker.start()
|
||||
self.assertTrue(started.wait(20))
|
||||
te.shutdown_encoders()
|
||||
worker.join(20)
|
||||
self.assertFalse(worker.is_alive())
|
||||
self.assertIsNotNone(tracked[0].poll())
|
||||
self.assertIsInstance(outcome.get("error"), te.EncoderClosed)
|
||||
with self.assertRaises(te.EncoderClosed):
|
||||
te.run_encoder(self.SLEEPER, timeout=1)
|
||||
|
||||
def test_input_is_piped_and_output_returned(self):
|
||||
echo = [
|
||||
sys.executable,
|
||||
"-c",
|
||||
"import sys; sys.stdout.buffer.write(sys.stdin.buffer.read()[::-1])",
|
||||
]
|
||||
done = te.run_encoder(echo, input=b"abc", timeout=30)
|
||||
self.assertEqual((done.returncode, done.stdout), (0, b"cba"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,444 @@
|
||||
"""Every preview writer and lookup follows the thumbnail settings (#317).
|
||||
|
||||
Automatic previews (capture and first view), the Activity job, the legacy
|
||||
gallery routes, the listings, the history route, duplicates, enrichment and
|
||||
the clear all name previews ``<stem>_thumb.<ext of the configured format>``,
|
||||
write them with the configured quality and size limit, and still find older
|
||||
previews (``.png`` and friends) until they are converted.
|
||||
|
||||
A thumbnail is only a grid or cover placeholder: inspection always reads the
|
||||
original, and never resolves a request to a file under ``thumbnails/``.
|
||||
|
||||
Ordering is driven by events, never by timers.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
try:
|
||||
from tests.test_api_images import ThumbnailTestCase, make_png, parse_sse
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_images import ThumbnailTestCase, make_png, parse_sse
|
||||
|
||||
from PIL import Image, PngImagePlugin # noqa: E402
|
||||
|
||||
import py.api.images as images_module # noqa: E402
|
||||
import py.api.thumbnail_auto as auto # noqa: E402
|
||||
from py.api import thumbnail_encode # noqa: E402
|
||||
from py.api.admin import _remove_thumbnail # noqa: E402
|
||||
from py.config import PromptManagerConfig # noqa: E402
|
||||
|
||||
SERVE = "/prompt_manager/images/serve/"
|
||||
STATUS = "/prompt_manager/images/thumbnails/status"
|
||||
PLAN = "/prompt_manager/images/thumbnails/plan"
|
||||
JOB = "/prompt_manager/jobs/thumbnails"
|
||||
WAIT = 10
|
||||
HAVE_FFMPEG = bool(shutil.which("ffmpeg"))
|
||||
|
||||
|
||||
def graph(positive="portrait", negative="blurry"):
|
||||
return {
|
||||
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": positive}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
|
||||
"3": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["1", 0], "negative": ["2", 0]},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def png_with_prompt(path, size=(64, 64)):
|
||||
info = PngImagePlugin.PngInfo()
|
||||
info.add_text("prompt", json.dumps(graph()))
|
||||
info.add_text("workflow", json.dumps({"nodes": [], "links": []}))
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with Image.new("RGB", size, (9, 90, 200)) as image:
|
||||
image.save(path, pnginfo=info)
|
||||
return path
|
||||
|
||||
|
||||
class FormatCase(ThumbnailTestCase):
|
||||
async def asyncSetUp(self):
|
||||
await super().asyncSetUp()
|
||||
self._orig = PromptManagerConfig.get_config()
|
||||
self.addCleanup(PromptManagerConfig.update_config, self._orig)
|
||||
self._settings("webp", 80, 512)
|
||||
thumbnail_encode.reset_encoders()
|
||||
self.addCleanup(thumbnail_encode.reset_encoders)
|
||||
self.addCleanup(self.api.close_auto_thumbnails)
|
||||
|
||||
def _settings(self, fmt, quality, edge):
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": fmt, "quality": quality, "edge": edge}}
|
||||
)
|
||||
|
||||
def _thumb(self, name, suffix=".webp"):
|
||||
return self.output_dir / "thumbnails" / f"{name}_thumb{suffix}"
|
||||
|
||||
async def _idle(self):
|
||||
loop = asyncio.get_running_loop()
|
||||
thumbnailer = self.api._auto_thumbnails()
|
||||
self.assertTrue(await loop.run_in_executor(None, thumbnailer.wait_idle, WAIT))
|
||||
|
||||
async def _capture(self, path):
|
||||
self.assertEqual(self.api.queue_capture_thumbnail(path), "queued")
|
||||
await self._idle()
|
||||
|
||||
async def _finished(self, job_id):
|
||||
job = self.api._jobs.get(job_id)
|
||||
await asyncio.wait_for(job.task, WAIT)
|
||||
return job
|
||||
|
||||
def _format_of(self, path):
|
||||
with Image.open(path) as img:
|
||||
return img.format, img.size
|
||||
|
||||
|
||||
class TestCaptureAndViewFollowTheSettings(FormatCase):
|
||||
async def test_capture_writes_webp_at_the_size_limit(self):
|
||||
self._settings("webp", 80, 200)
|
||||
src = make_png(self.output_dir / "a.png", size=(800, 400))
|
||||
await self._capture(src)
|
||||
self.assertEqual(self._format_of(self._thumb("a")), ("WEBP", (200, 100)))
|
||||
self.assertFalse(self._thumb("a", ".png").exists())
|
||||
|
||||
async def test_capture_writes_jpeg_when_configured(self):
|
||||
self._settings("jpeg", 50, 300)
|
||||
src = make_png(self.output_dir / "sub" / "b.png", size=(600, 600))
|
||||
await self._capture(src)
|
||||
made = self.output_dir / "thumbnails" / "sub" / "b_thumb.jpg"
|
||||
self.assertEqual(self._format_of(made), ("JPEG", (300, 300)))
|
||||
|
||||
async def test_quality_reaches_the_encoder(self):
|
||||
noisy = Image.effect_noise((400, 400), 80).convert("RGB")
|
||||
for name, quality in (("low", 10), ("high", 95)):
|
||||
noisy.save(self.output_dir / f"{name}.png")
|
||||
self._settings("webp", quality, 512)
|
||||
await self._capture(self.output_dir / f"{name}.png")
|
||||
self.assertLess(
|
||||
self._thumb("low").stat().st_size, self._thumb("high").stat().st_size
|
||||
)
|
||||
|
||||
async def test_an_animated_source_previews_as_a_still(self):
|
||||
path = self.output_dir / "anim.gif"
|
||||
frames = [Image.new("RGB", (64, 64), c) for c in ((250, 0, 0), (0, 0, 250))]
|
||||
frames[0].save(path, save_all=True, append_images=frames[1:], duration=40)
|
||||
await self._capture(path)
|
||||
with Image.open(self._thumb("anim")) as img:
|
||||
self.assertEqual(getattr(img, "n_frames", 1), 1)
|
||||
self.assertGreater(img.convert("RGB").getpixel((32, 32))[0], 200)
|
||||
|
||||
async def test_a_video_preview_is_the_configured_format(self):
|
||||
frame = Image.new("RGB", (640, 360), (0, 200, 0))
|
||||
clip = self.output_dir / "clip.mp4"
|
||||
clip.write_bytes(b"not decoded")
|
||||
with patch.object(self.api, "_opencv_frame", return_value=frame):
|
||||
await self._capture(clip)
|
||||
self.assertEqual(self._format_of(self._thumb("clip"))[0], "WEBP")
|
||||
self.assertFalse(self._thumb("clip", ".jpg").exists())
|
||||
|
||||
@unittest.skipUnless(HAVE_FFMPEG, "no ffmpeg")
|
||||
async def test_ffmpeg_video_frame_is_encoded_in_the_format(self):
|
||||
clip = self.output_dir / "real.mp4"
|
||||
subprocess.run(
|
||||
["ffmpeg", "-v", "quiet", "-f", "lavfi", "-i",
|
||||
"color=c=red:s=320x240:d=2", "-y", str(clip)],
|
||||
check=True, timeout=60,
|
||||
) # fmt: skip
|
||||
with patch.object(self.api, "_opencv_frame", return_value=None):
|
||||
await self._capture(clip)
|
||||
fmt, size = self._format_of(self._thumb("real"))
|
||||
self.assertEqual(fmt, "WEBP")
|
||||
self.assertEqual(size, (320, 240))
|
||||
|
||||
async def test_view_makes_the_preview_and_sends_its_type(self):
|
||||
make_png(self.output_dir / "v.png", size=(64, 64))
|
||||
resp = await self.client.get(SERVE + "thumbnails/v_thumb.webp")
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(resp.headers["Content-Type"], "image/webp")
|
||||
self.assertEqual(resp.headers["Cache-Control"], "public, max-age=3600")
|
||||
self.assertTrue(self._thumb("v").is_file())
|
||||
|
||||
async def test_a_jxl_preview_is_served_as_image_jxl(self):
|
||||
make_png(self.output_dir / "j.png")
|
||||
thumb = self._thumb("j", ".jxl")
|
||||
thumb.parent.mkdir(parents=True, exist_ok=True)
|
||||
thumb.write_bytes(b"\xff\x0a" + b"\x00" * 30)
|
||||
resp = await self.client.get(SERVE + "thumbnails/j_thumb.jxl")
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(resp.headers["Content-Type"], "image/jxl")
|
||||
self.assertEqual(resp.headers["Cache-Control"], "public, max-age=3600")
|
||||
|
||||
async def test_a_converted_away_legacy_url_answers_the_current_preview(self):
|
||||
make_png(self.output_dir / "c.png")
|
||||
await self._capture(self.output_dir / "c.png")
|
||||
resp = await self.client.get(SERVE + "thumbnails/c_thumb.png?v=1")
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(resp.headers["Content-Type"], "image/webp")
|
||||
self.assertFalse(self._thumb("c", ".png").exists())
|
||||
|
||||
async def test_preview_requests_keep_the_text_checks_for_new_extensions(self):
|
||||
accepted = ("thumbnails/a_thumb.jxl", "thumbnails/s/a_thumb.webp")
|
||||
for text in accepted:
|
||||
self.assertTrue(auto._is_preview_request(text), text)
|
||||
refused = (
|
||||
"thumbnails/a_thumb.JXL",
|
||||
"thumbnails\\a_thumb.jxl",
|
||||
"thumbnails/C:a_thumb.webp",
|
||||
"thumbnails//a_thumb.webp",
|
||||
"thumbnails/./a_thumb.jxl",
|
||||
"thumbnails/../a_thumb.jxl",
|
||||
"thumbnails/thumbnails/a_thumb.webp",
|
||||
"thumbnails/a_thumb.txt",
|
||||
"thumbnails/_thumb.webp",
|
||||
)
|
||||
for text in refused:
|
||||
self.assertFalse(auto._is_preview_request(text), text)
|
||||
make_png(self.outside_png())
|
||||
resp = await self.client.get(SERVE + "thumbnails/..%2F..%2Fout_thumb.webp")
|
||||
self.assertIn(resp.status, (403, 404))
|
||||
self.assertFalse((self.comfy_dir / "out_thumb.webp").exists())
|
||||
|
||||
def outside_png(self):
|
||||
return self.comfy_dir / "out.png"
|
||||
|
||||
async def test_jxl_without_an_encoder_writes_webp_once(self):
|
||||
self._settings("jxl", 80, 512)
|
||||
src = make_png(self.output_dir / "x.png", size=(64, 64))
|
||||
calls = []
|
||||
original = self.api._generate_one
|
||||
|
||||
def counting(*args, **kwargs):
|
||||
calls.append(1)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
self.api._generate_one = counting
|
||||
with patch.object(thumbnail_encode, "detect_jxl_encoder", return_value=None):
|
||||
await self._capture(src)
|
||||
self.assertTrue(self._thumb("x").is_file())
|
||||
self.assertFalse(self._thumb("x", ".jxl").exists())
|
||||
self.assertEqual(self.api.queue_capture_thumbnail(src), "fresh")
|
||||
resp = await self.client.get(SERVE + "thumbnails/x_thumb.jxl")
|
||||
self.assertEqual(resp.headers["Content-Type"], "image/webp")
|
||||
self.assertEqual(len(calls), 1)
|
||||
|
||||
|
||||
class TestListingsFindAnyOwnedPreview(FormatCase):
|
||||
def _listed(self, rel):
|
||||
return images_module.media_thumbnail(self.output_dir, Path(rel), 0)
|
||||
|
||||
def test_missing_preview_lists_the_configured_name(self):
|
||||
make_png(self.output_dir / "m.png")
|
||||
url, ready = self._listed("m.png")
|
||||
self.assertEqual(url, SERVE + "thumbnails/m_thumb.webp?root=0")
|
||||
self.assertFalse(ready)
|
||||
|
||||
def test_a_legacy_preview_is_listed_until_a_current_one_exists(self):
|
||||
make_png(self.output_dir / "l.png")
|
||||
make_png(self._thumb("l", ".png"))
|
||||
url, ready = self._listed("l.png")
|
||||
self.assertTrue(url.startswith(SERVE + "thumbnails/l_thumb.png?v="))
|
||||
self.assertTrue(ready)
|
||||
make_png(self._thumb("l"))
|
||||
url, _ready = self._listed("l.png")
|
||||
self.assertTrue(url.startswith(SERVE + "thumbnails/l_thumb.webp?v="))
|
||||
|
||||
def test_a_video_lists_its_legacy_jpg(self):
|
||||
(self.output_dir / "v.mp4").write_bytes(b"x")
|
||||
make_png(self._thumb("v", ".jpg"))
|
||||
url, ready = self._listed("v.mp4")
|
||||
self.assertTrue(url.startswith(SERVE + "thumbnails/v_thumb.jpg?v="))
|
||||
self.assertTrue(ready)
|
||||
|
||||
def test_duplicates_listing_and_enrichment_use_the_existing_preview(self):
|
||||
src = make_png(self.output_dir / "d.png")
|
||||
make_png(self._thumb("d"))
|
||||
described = self.api._describe_media_file(src, self.output_dir, "hash")
|
||||
self.assertTrue(
|
||||
described["thumbnail_url"].startswith(SERVE + "thumbnails/d_thumb.webp")
|
||||
)
|
||||
images = self.api._enrich_images([{"image_path": str(src)}])
|
||||
self.assertTrue(
|
||||
images[0]["thumbnail_url"].startswith(SERVE + "thumbnails/d_thumb.webp")
|
||||
)
|
||||
|
||||
|
||||
class TestPlanStatusAndJobs(FormatCase):
|
||||
async def test_status_counts_missing_outdated_and_ready(self):
|
||||
make_png(self.output_dir / "new.png") # missing
|
||||
make_png(self.output_dir / "old.png")
|
||||
make_png(self._thumb("old", ".png")) # outdated (legacy PNG)
|
||||
make_png(self.output_dir / "cur.png")
|
||||
make_png(self._thumb("cur")) # current
|
||||
body = await (await self.client.get(STATUS)).json()
|
||||
self.assertEqual(
|
||||
(body["pending"], body["outdated"], body["ready"], body["total_images"]),
|
||||
(1, 1, 2, 3),
|
||||
)
|
||||
plan = await (await self.client.get(PLAN, params={"mode": "missing"})).json()
|
||||
self.assertEqual((plan["targets"], plan["fresh"], plan["outdated"]), (1, 1, 1))
|
||||
self.assertEqual(
|
||||
(plan["format"], plan["quality"], plan["size"]), ("webp", 80, 512)
|
||||
)
|
||||
|
||||
async def test_missing_job_writes_the_format_and_leaves_outdated_alone(self):
|
||||
self._settings("webp", 70, 160)
|
||||
make_png(self.output_dir / "n.png", size=(320, 320))
|
||||
make_png(self.output_dir / "o.png")
|
||||
legacy = make_png(self._thumb("o", ".png"))
|
||||
before = legacy.read_bytes()
|
||||
resp = await self.client.post(JOB, json={"mode": "missing"})
|
||||
self.assertEqual(resp.status, 202)
|
||||
job = await self._finished((await resp.json())["job"]["id"])
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.scope["format"], "webp")
|
||||
self.assertEqual(self._format_of(self._thumb("n")), ("WEBP", (160, 160)))
|
||||
self.assertFalse(self._thumb("o").exists())
|
||||
self.assertEqual(legacy.read_bytes(), before)
|
||||
|
||||
async def test_the_legacy_stream_uses_the_server_settings(self):
|
||||
# R3: the gallery's quality preset no longer sizes the preview.
|
||||
self._settings("webp", 80, 200)
|
||||
make_png(self.output_dir / "g.png", size=(800, 800))
|
||||
resp = await self.client.post(self.PROGRESS_URL + "?quality=high")
|
||||
events = dict(parse_sse(await resp.text()))
|
||||
self.assertEqual(events["complete"]["count"], 1)
|
||||
self.assertEqual(self._format_of(self._thumb("g")), ("WEBP", (200, 200)))
|
||||
|
||||
async def test_the_blocking_legacy_generate_uses_the_server_settings(self):
|
||||
self._settings("jpeg", 80, 140)
|
||||
make_png(self.output_dir / "h.png", size=(700, 700))
|
||||
resp = await self.client.post(
|
||||
"/prompt_manager/images/generate-thumbnails", json={"quality": "low"}
|
||||
)
|
||||
self.assertEqual(resp.status, 200)
|
||||
made = self.output_dir / "thumbnails" / "h_thumb.jpg"
|
||||
self.assertEqual(self._format_of(made), ("JPEG", (140, 140)))
|
||||
|
||||
async def test_clear_removes_every_preview_format(self):
|
||||
make_png(self.output_dir / "k.png")
|
||||
for suffix in (".webp", ".png", ".jpg"):
|
||||
make_png(self._thumb("k", suffix))
|
||||
jxl = self._thumb("k", ".jxl")
|
||||
jxl.write_bytes(b"\xff\x0a")
|
||||
resp = await self.client.post("/prompt_manager/images/clear-thumbnails")
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual((await resp.json())["cleared_files"], 4)
|
||||
self.assertTrue((self.output_dir / "k.png").exists())
|
||||
|
||||
|
||||
class TestReviewFixes(FormatCase):
|
||||
async def test_a_legacy_preview_in_another_case_is_outdated(self):
|
||||
# Review M1: X.JPG's older X_thumb.JPG, and y.png's y_thumb.PNG.
|
||||
upper = self.output_dir / "X.JPG"
|
||||
Image.new("RGB", (64, 64)).save(upper, "JPEG")
|
||||
make_png(self.output_dir / "y.png")
|
||||
for name in ("X_thumb.JPG", "y_thumb.PNG"):
|
||||
legacy = self.output_dir / "thumbnails" / name
|
||||
legacy.parent.mkdir(parents=True, exist_ok=True)
|
||||
Image.new("RGB", (32, 32)).save(legacy, "PNG" if "PNG" in name else "JPEG")
|
||||
body = await (await self.client.get(STATUS)).json()
|
||||
self.assertEqual((body["outdated"], body["pending"]), (2, 0))
|
||||
resp = await self.client.post(JOB, json={"mode": "convert"})
|
||||
job = await self._finished((await resp.json())["job"]["id"])
|
||||
self.assertEqual(job.counters["generated"], 2, job.log)
|
||||
names = sorted(os.listdir(self.output_dir / "thumbnails"))
|
||||
self.assertEqual(names, ["X_thumb.webp", "y_thumb.webp"])
|
||||
|
||||
async def test_clear_also_removes_stray_temp_files(self):
|
||||
make_png(self.output_dir / "k.png")
|
||||
make_png(self._thumb("k"))
|
||||
folder = self.output_dir / "thumbnails"
|
||||
for name in (".k_thumb.part-1-2.webp", ".501one.part-3-4.jxl"):
|
||||
(folder / name).write_bytes(b"partial")
|
||||
keep = folder / "notes.txt"
|
||||
keep.write_bytes(b"x")
|
||||
resp = await self.client.post("/prompt_manager/images/clear-thumbnails")
|
||||
self.assertEqual((await resp.json())["cleared_files"], 3)
|
||||
self.assertEqual(os.listdir(folder), ["notes.txt"])
|
||||
|
||||
def test_the_startup_sweep_removes_only_old_temp_files(self):
|
||||
folder = self.output_dir / "thumbnails" / "sub"
|
||||
folder.mkdir(parents=True)
|
||||
old = folder / ".a_thumb.part-1-2.webp"
|
||||
young = folder / ".b_thumb.part-1-2.webp"
|
||||
for path in (old, young):
|
||||
path.write_bytes(b"partial")
|
||||
when = old.stat().st_mtime - 3600
|
||||
os.utime(old, (when, when))
|
||||
self.assertEqual(self.api.sweep_thumbnail_temps(), 1)
|
||||
self.assertFalse(old.exists())
|
||||
self.assertTrue(young.exists())
|
||||
|
||||
|
||||
class TestRemoval(FormatCase):
|
||||
def test_removing_a_source_removes_its_previews_in_every_format(self):
|
||||
make_png(self.output_dir / "r.png")
|
||||
for suffix in (".png", ".webp"):
|
||||
make_png(self._thumb("r", suffix))
|
||||
_remove_thumbnail(self.output_dir, Path("r.png"))
|
||||
self.assertFalse(self._thumb("r", ".png").exists())
|
||||
self.assertFalse(self._thumb("r").exists())
|
||||
|
||||
def test_a_shared_preview_stays_while_another_source_owns_it(self):
|
||||
# R2: r.png and r.mp4 share r_thumb.webp; the PNG's legacy preview is its own.
|
||||
make_png(self.output_dir / "r.png")
|
||||
(self.output_dir / "r.mp4").write_bytes(b"x")
|
||||
make_png(self._thumb("r", ".png"))
|
||||
make_png(self._thumb("r"))
|
||||
_remove_thumbnail(self.output_dir, Path("r.png"))
|
||||
self.assertFalse(self._thumb("r", ".png").exists())
|
||||
self.assertTrue(self._thumb("r").exists())
|
||||
|
||||
|
||||
class TestThumbnailsAreOnlyPlaceholders(FormatCase):
|
||||
"""Maintainer constraint: metadata always comes from the original."""
|
||||
|
||||
async def test_inspection_reads_the_original_not_its_webp_preview(self):
|
||||
png_with_prompt(self.output_dir / "image.png")
|
||||
await self._capture(self.output_dir / "image.png")
|
||||
self.assertTrue(self._thumb("image").is_file())
|
||||
opened = []
|
||||
original_open = Image.open
|
||||
|
||||
def recording(fp, *args, **kwargs):
|
||||
opened.append(str(fp))
|
||||
return original_open(fp, *args, **kwargs)
|
||||
|
||||
with patch.object(Image, "open", recording):
|
||||
resp = await self.client.get(
|
||||
"/prompt_manager/images/inspect/image.png", params={"root": "0"}
|
||||
)
|
||||
body = await resp.json()
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(body["prompts"]["positive"]["text"], "portrait")
|
||||
self.assertEqual(body["prompts"]["negative"]["text"], "blurry")
|
||||
self.assertFalse([p for p in opened if "thumbnails" in p], opened)
|
||||
|
||||
async def test_inspection_never_resolves_a_thumbnail_path(self):
|
||||
png_with_prompt(self.output_dir / "image.png")
|
||||
png_with_prompt(self._thumb("image", ".png"))
|
||||
await self._capture(self.output_dir / "image.png")
|
||||
for rel in (
|
||||
"thumbnails/image_thumb.webp",
|
||||
"thumbnails/image_thumb.png",
|
||||
"Thumbnails/image_thumb.webp",
|
||||
):
|
||||
resp = await self.client.get(
|
||||
f"/prompt_manager/images/inspect/{rel}", params={"root": "0"}
|
||||
)
|
||||
self.assertIn(resp.status, (403, 404), rel)
|
||||
self.assertNotIn("portrait", await resp.text())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+211
-86
@@ -117,6 +117,14 @@ class ThumbnailJobCase(ThumbnailTestCase):
|
||||
self.addCleanup(patcher.stop)
|
||||
return lines
|
||||
|
||||
def _use(self, fmt="webp", quality=80, edge=512):
|
||||
"""Set the server's thumbnail settings for this test (#317)."""
|
||||
saved = PromptManagerConfig.get_config()["thumbnails"]
|
||||
self.addCleanup(PromptManagerConfig.update_config, {"thumbnails": saved})
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": fmt, "quality": quality, "edge": edge}}
|
||||
)
|
||||
|
||||
def _one_worker(self):
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1 # batches of 4
|
||||
@@ -149,13 +157,10 @@ class TestThumbnailPlan(ThumbnailJobCase):
|
||||
os.utime(fresh, (source_time + 10, source_time + 10))
|
||||
before = _tree_state(self.comfy_dir)
|
||||
|
||||
self._use("webp", 80, 512)
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1):
|
||||
missing = await self.client.get(
|
||||
PLAN_URL, params={"mode": "missing", "size": "512", "quality": "high"}
|
||||
)
|
||||
rebuild = await self.client.get(
|
||||
PLAN_URL, params={"mode": "rebuild", "size": "256", "quality": "low"}
|
||||
)
|
||||
missing = await self.client.get(PLAN_URL, params={"mode": "missing"})
|
||||
rebuild = await self.client.get(PLAN_URL, params={"mode": "rebuild"})
|
||||
|
||||
self.assertEqual(missing.status, 200)
|
||||
self.assertEqual(
|
||||
@@ -164,40 +169,38 @@ class TestThumbnailPlan(ThumbnailJobCase):
|
||||
"success": True,
|
||||
"targets": 1,
|
||||
"fresh": 1,
|
||||
"outdated": 0,
|
||||
"remaining": 1,
|
||||
"roots": ["output"],
|
||||
"mode": "missing",
|
||||
"format": "webp",
|
||||
"size": 512,
|
||||
"quality": "high",
|
||||
"quality": 80,
|
||||
},
|
||||
)
|
||||
data = await rebuild.json()
|
||||
# Rebuild regenerates fresh previews too: all three are work, one fits the cap.
|
||||
self.assertEqual((data["targets"], data["fresh"], data["remaining"]), (1, 1, 2))
|
||||
self.assertEqual(
|
||||
(data["mode"], data["size"], data["quality"]), ("rebuild", 256, "low")
|
||||
)
|
||||
self.assertEqual(data["mode"], "rebuild")
|
||||
self.assertEqual(_tree_state(self.comfy_dir), before)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
|
||||
async def test_plan_defaults_to_missing_previews_at_the_concept_size(self):
|
||||
async def test_plan_defaults_to_missing_previews_with_the_settings(self):
|
||||
self._use("webp", 80, 512)
|
||||
make_png(self.output_dir / "a.png")
|
||||
data = await (await self.client.get(PLAN_URL)).json()
|
||||
self.assertEqual(
|
||||
(data["mode"], data["size"], data["quality"], data["targets"]),
|
||||
("missing", 512, "medium", 1),
|
||||
(data["mode"], data["format"], data["size"], data["quality"]),
|
||||
("missing", "webp", 512, 80),
|
||||
)
|
||||
self.assertEqual(data["targets"], 1)
|
||||
|
||||
async def test_size_bounds_validated(self):
|
||||
async def test_mode_validated_and_size_quality_ignored(self):
|
||||
# #317: format, quality and size limit are the server's settings; an
|
||||
# older client's size or quality is ignored, an unknown mode refused.
|
||||
self._use("webp", 80, 512)
|
||||
make_png(self.output_dir / "a.png")
|
||||
for params in (
|
||||
{"size": "127"},
|
||||
{"size": "2049"},
|
||||
{"size": "512.5"},
|
||||
{"size": "big"},
|
||||
{"mode": "all"},
|
||||
{"quality": "ultra"},
|
||||
):
|
||||
for params in ({"mode": "all"}, {"mode": ""}):
|
||||
with self.subTest(params=params):
|
||||
resp = await self.client.get(PLAN_URL, params=params)
|
||||
body = await resp.json()
|
||||
@@ -205,24 +208,14 @@ class TestThumbnailPlan(ThumbnailJobCase):
|
||||
self.assertEqual(
|
||||
(body["success"], body["code"]), (False, "invalid-request")
|
||||
)
|
||||
for body in (
|
||||
{"size": 127},
|
||||
{"size": 2049},
|
||||
{"size": "512"},
|
||||
{"size": 512.0},
|
||||
{"size": True},
|
||||
{"mode": "rebuild-all"},
|
||||
{"quality": None},
|
||||
[],
|
||||
):
|
||||
for body in ({"mode": "rebuild-all"}, {"mode": None}, []):
|
||||
with self.subTest(body=body):
|
||||
resp = await self.client.post(JOB_URL, json=body)
|
||||
self.assertEqual(resp.status, 400)
|
||||
self.assertEqual((await resp.json())["code"], "invalid-request")
|
||||
for edge in (128, 2048):
|
||||
with self.subTest(edge=edge):
|
||||
resp = await self.client.get(PLAN_URL, params={"size": str(edge)})
|
||||
self.assertEqual(resp.status, 200)
|
||||
resp = await self.client.get(PLAN_URL, params={"size": "9999", "quality": "x"})
|
||||
data = await resp.json()
|
||||
self.assertEqual((data["size"], data["quality"]), (512, 80))
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
self.assertFalse(self._thumb_path("a").exists())
|
||||
|
||||
@@ -233,7 +226,8 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
make_png(self.output_dir / "sub" / "b.png", size=(64, 64))
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG not really")
|
||||
|
||||
status, body = await self._start(mode="missing", size=200, quality="medium")
|
||||
self._use("webp", 80, 200)
|
||||
status, body = await self._start(mode="missing", size=999, quality="high")
|
||||
|
||||
self.assertEqual(status, 202)
|
||||
summary = body["job"]
|
||||
@@ -244,7 +238,8 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
self.assertTrue(summary["cancellable"])
|
||||
self.assertEqual(set(summary["counters"]), {"generated", "skipped", "errors"})
|
||||
self.assertEqual(
|
||||
summary["scope"], {"mode": "missing", "size": 200, "quality": "medium"}
|
||||
summary["scope"],
|
||||
{"mode": "missing", "format": "webp", "size": 200, "quality": 80},
|
||||
)
|
||||
job = await self._finished(summary["id"])
|
||||
self.assertEqual(job.state, "finished")
|
||||
@@ -279,7 +274,8 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
async def test_rebuild_never_touches_originals(self):
|
||||
for name in ("a", "b"):
|
||||
make_png(self.output_dir / f"{name}.png", size=(400, 400))
|
||||
_, body = await self._start(size=128)
|
||||
self._use("webp", 80, 128)
|
||||
_, body = await self._start()
|
||||
await self._finished(body["job"]["id"])
|
||||
originals = {
|
||||
name: hashlib.sha256(
|
||||
@@ -288,7 +284,8 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
for name in ("a", "b")
|
||||
}
|
||||
|
||||
_, body = await self._start(mode="rebuild", size=256, quality="high")
|
||||
self._use("webp", 95, 256)
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 0})
|
||||
@@ -315,16 +312,16 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(self._thumb_path("a").read_bytes(), preview)
|
||||
leftovers = [p.name for p in self._thumb_path("a").parent.iterdir()]
|
||||
self.assertEqual(leftovers, ["a_thumb.png"]) # no temp file left
|
||||
self.assertEqual(leftovers, ["a_thumb.webp"]) # no temp file left
|
||||
|
||||
async def test_rebuild_keeps_an_old_video_preview_instead_of_a_placeholder(self):
|
||||
clip = self.output_dir / "clip.mp4"
|
||||
clip.write_bytes(b"\x00")
|
||||
preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.jpg")
|
||||
preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.webp")
|
||||
old = preview.read_bytes()
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
):
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
@@ -339,7 +336,8 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
img.save(self.output_dir / "photo.jpg", "JPEG", quality=95)
|
||||
sizes = {}
|
||||
for quality in ("low", "high"):
|
||||
_, body = await self._start(mode="rebuild", size=300, quality=quality)
|
||||
self._use("jpeg", 40 if quality == "low" else 95, 300)
|
||||
_, body = await self._start(mode="rebuild")
|
||||
await self._finished(body["job"]["id"])
|
||||
thumb = self._thumb_path("photo", ".jpg")
|
||||
with Image.open(thumb) as image:
|
||||
@@ -380,7 +378,7 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
self.assertEqual(len(written), 2)
|
||||
for dst in written:
|
||||
self.assertTrue(dst.is_file())
|
||||
made = sorted((self.output_dir / "thumbnails").glob("*_thumb.png"))
|
||||
made = sorted((self.output_dir / "thumbnails").glob("*_thumb.webp"))
|
||||
self.assertEqual(made, sorted(written))
|
||||
self.assertTrue(any("Thumbnails cancelled" in line for line in job.log))
|
||||
|
||||
@@ -511,16 +509,16 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.addCleanup(release.set)
|
||||
checked = []
|
||||
real_fresh = images_module._thumbnail_is_fresh
|
||||
real_state = images_module._preview_state
|
||||
|
||||
def gated_fresh(src, dst):
|
||||
def gated_state(src, *args):
|
||||
checked.append(src.name)
|
||||
if len(checked) == 1:
|
||||
entered.set()
|
||||
release.wait(10)
|
||||
return real_fresh(src, dst)
|
||||
return real_state(src, *args)
|
||||
|
||||
with patch.object(images_module, "_thumbnail_is_fresh", gated_fresh):
|
||||
with patch.object(images_module, "_preview_state", gated_state):
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
@@ -566,8 +564,8 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "_atomic_save", refuse),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "write_thumbnail", refuse),
|
||||
):
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
@@ -586,49 +584,66 @@ class TestThumbnailJob(ThumbnailJobCase):
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_NOT_WRITABLE)
|
||||
|
||||
async def test_ffmpeg_fallback_keeps_quality_and_aspect_ratio(self):
|
||||
"""M1: without OpenCV, ffmpeg keeps the chosen quality and the aspect ratio."""
|
||||
async def test_ffmpeg_fallback_keeps_the_aspect_ratio_and_the_settings(self):
|
||||
"""M1: without OpenCV, ffmpeg's frame keeps its aspect ratio; #317: it is
|
||||
read as a PNG on stdout and encoded with the server's settings."""
|
||||
import io
|
||||
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
commands = []
|
||||
frame = io.BytesIO()
|
||||
Image.new("RGB", (640, 360), (0, 99, 0)).save(frame, "PNG")
|
||||
|
||||
def fake_ffmpeg(cmd, **_kwargs):
|
||||
commands.append(cmd)
|
||||
Path(cmd[-1]).write_bytes(b"jpeg")
|
||||
return type("Done", (), {"returncode": 0, "stderr": ""})()
|
||||
|
||||
for quality, qscale in (("high", "2"), ("medium", "4"), ("low", "8")):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=fake_ffmpeg),
|
||||
):
|
||||
_, body = await self._start(mode="rebuild", size=640, quality=quality)
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.counters["generated"], 1, quality)
|
||||
cmd = commands[-1]
|
||||
self.assertNotIn("-s", cmd)
|
||||
self.assertEqual(
|
||||
cmd[cmd.index("-vf") + 1],
|
||||
"scale=640:640:force_original_aspect_ratio=decrease",
|
||||
)
|
||||
self.assertEqual(cmd[cmd.index("-q:v") + 1], qscale, quality)
|
||||
|
||||
async def test_the_placeholder_uses_the_chosen_quality(self):
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
saved = []
|
||||
real_save = images_module._atomic_save
|
||||
|
||||
def recording(img, dst, fmt=None, **kwargs):
|
||||
saved.append(kwargs)
|
||||
return real_save(img, dst, fmt, **kwargs)
|
||||
done = type("Done", (), {"returncode": 0})()
|
||||
done.communicate = lambda input=None, timeout=None: (frame.getvalue(), b"")
|
||||
done.poll = lambda: 0
|
||||
done.kill = lambda: None
|
||||
return done
|
||||
|
||||
self._use("webp", 70, 320)
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "_atomic_save", recording),
|
||||
patch("subprocess.Popen", side_effect=fake_ffmpeg),
|
||||
):
|
||||
_, body = await self._start(quality="low")
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.counters["generated"], 1)
|
||||
cmd = commands[-1]
|
||||
self.assertNotIn("-q:v", cmd)
|
||||
self.assertEqual(
|
||||
cmd[cmd.index("-vf") + 1],
|
||||
"scale='min(iw,320)':'min(ih,320)':force_original_aspect_ratio=decrease",
|
||||
)
|
||||
self.assertEqual(cmd[-5:], ["-f", "image2pipe", "-vcodec", "png", "-"])
|
||||
# Review M6: the input is always read as a file, never a protocol URL.
|
||||
self.assertTrue(cmd[cmd.index("-i") + 1].startswith("file:"))
|
||||
with Image.open(self._thumb_path("clip")) as thumb:
|
||||
self.assertEqual((thumb.format, thumb.size), ("WEBP", (320, 180)))
|
||||
|
||||
async def test_the_placeholder_uses_the_settings(self):
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
saved = []
|
||||
real_write = images_module.write_thumbnail
|
||||
|
||||
def recording(frame, dst, settings, **kwargs):
|
||||
saved.append(settings)
|
||||
return real_write(frame, dst, settings, **kwargs)
|
||||
|
||||
self._use("jpeg", 33, 200)
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.Popen", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "write_thumbnail", recording),
|
||||
):
|
||||
_, body = await self._start()
|
||||
await self._finished(body["job"]["id"])
|
||||
self.assertEqual(saved, [images_module.THUMBNAIL_ENCODINGS["low"]])
|
||||
self.assertEqual(
|
||||
[(x.format, x.quality, x.edge) for x in saved], [("jpeg", 33, 200)]
|
||||
)
|
||||
with Image.open(self._thumb_path("clip", ".jpg")) as thumb:
|
||||
self.assertEqual(thumb.size, (200, 200))
|
||||
|
||||
|
||||
# The legacy stream's frames and their keys, as the gallery page reads them.
|
||||
@@ -672,11 +687,119 @@ LEGACY_KEYS = {
|
||||
}
|
||||
|
||||
|
||||
class TestBatchesContinue(ThumbnailJobCase):
|
||||
"""MAX_THUMBNAIL_FILES bounds one batch, not the job: after each batch
|
||||
the job plans again and goes on until nothing is left, it is cancelled,
|
||||
or a batch makes no progress (files that fail every time are never
|
||||
planned again)."""
|
||||
|
||||
def _sources(self, count, broken=False):
|
||||
for i in range(count):
|
||||
path = self.output_dir / f"s{i}.png"
|
||||
if broken:
|
||||
path.write_bytes(b"\x89PNG broken")
|
||||
else:
|
||||
make_png(path, size=(32, 32))
|
||||
|
||||
async def test_one_job_makes_every_preview_past_the_cap(self):
|
||||
self._sources(8)
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
|
||||
status, body = await self._start(mode="missing")
|
||||
self.assertEqual(status, 202)
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.counters["generated"], 8)
|
||||
for i in range(8):
|
||||
self.assertTrue(self._thumb_path(f"s{i}").is_file(), i)
|
||||
summary = job.summary()
|
||||
self.assertEqual((summary["total"], summary["processed"]), (8, 8))
|
||||
self.assertEqual(summary["progress"], 100)
|
||||
self.assertEqual(job.status_line, "8 previews written")
|
||||
|
||||
async def test_rebuild_goes_through_every_file_once(self):
|
||||
self._sources(5)
|
||||
calls = []
|
||||
real = self.api._generate_batch
|
||||
|
||||
def counting(batch, *args, **kwargs):
|
||||
calls.extend(src.name for src, _dst, _video in batch)
|
||||
return real(batch, *args, **kwargs)
|
||||
|
||||
self.api._generate_batch = counting
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 2):
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(sorted(calls), [f"s{i}.png" for i in range(5)])
|
||||
self.assertEqual(job.counters["generated"], 5)
|
||||
|
||||
async def test_cancel_between_batches_stops_the_job(self):
|
||||
self._sources(8)
|
||||
real = self.api._plan_gallery_thumbnails
|
||||
plans = []
|
||||
job_ids = []
|
||||
|
||||
def cancelling(*args, **kwargs):
|
||||
plans.append(1)
|
||||
if len(plans) == 2:
|
||||
self.assertTrue(self.api._jobs.cancel(job_ids[0]))
|
||||
return real(*args, **kwargs)
|
||||
|
||||
self.api._plan_gallery_thumbnails = cancelling
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
|
||||
_, body = await self._start(mode="missing")
|
||||
job_ids.append(body["job"]["id"])
|
||||
job = await self._finished(job_ids[0])
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.counters["generated"], 3)
|
||||
self.assertEqual(len(plans), 2)
|
||||
|
||||
async def test_a_batch_where_everything_fails_stops(self):
|
||||
self._sources(8, broken=True)
|
||||
plans = []
|
||||
real = self.api._plan_gallery_thumbnails
|
||||
|
||||
def counting(*args, **kwargs):
|
||||
plans.append(1)
|
||||
return real(*args, **kwargs)
|
||||
|
||||
self.api._plan_gallery_thumbnails = counting
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 3):
|
||||
_, body = await self._start(mode="missing")
|
||||
job = await asyncio.wait_for(self._finished(body["job"]["id"]), 10)
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(len(plans), 1, "no second plan after a batch with no progress")
|
||||
self.assertEqual(job.counters["errors"], 3)
|
||||
self.assertIn("5 more on the next run", job.status_line)
|
||||
|
||||
async def test_failed_files_are_never_planned_again(self):
|
||||
# One bad file per batch still lets the job go on, without retrying it.
|
||||
for i in range(6):
|
||||
path = self.output_dir / f"m{i}.png"
|
||||
if i in (0, 3):
|
||||
path.write_bytes(b"\x89PNG broken")
|
||||
else:
|
||||
make_png(path, size=(32, 32))
|
||||
seen = []
|
||||
real = self.api._generate_batch
|
||||
|
||||
def counting(batch, *args, **kwargs):
|
||||
seen.extend(src.name for src, _dst, _video in batch)
|
||||
return real(batch, *args, **kwargs)
|
||||
|
||||
self.api._generate_batch = counting
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 2):
|
||||
_, body = await self._start(mode="missing")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(sorted(seen), sorted(f"m{i}.png" for i in range(6)))
|
||||
self.assertEqual((job.counters["generated"], job.counters["errors"]), (4, 2))
|
||||
|
||||
|
||||
class TestLegacyStreamOnTheJob(ThumbnailJobCase):
|
||||
async def test_legacy_progress_events_unchanged_on_success(self):
|
||||
make_png(self.output_dir / "a.png", size=(400, 400))
|
||||
make_png(self.output_dir / "b.png")
|
||||
|
||||
self._use("webp", 80, 200)
|
||||
resp = await self.client.post(f"{self.PROGRESS_URL}?quality=low")
|
||||
events = parse_sse(await resp.text())
|
||||
|
||||
@@ -693,7 +816,8 @@ class TestLegacyStreamOnTheJob(ThumbnailJobCase):
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
with Image.open(self._thumb_path("a")) as thumb:
|
||||
self.assertEqual(max(thumb.size), 150) # the legacy preset still sizes
|
||||
# #317 R3: the server's size limit, not the legacy preset (low = 150).
|
||||
self.assertEqual(max(thumb.size), 200)
|
||||
[job] = self.api._jobs.list()
|
||||
self.assertEqual(
|
||||
(job.kind, job.title, job.state, job.cancellable),
|
||||
@@ -763,7 +887,8 @@ class TestLegacyStreamOnTheJob(ThumbnailJobCase):
|
||||
async def test_legacy_stream_attaches_to_a_running_job(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start(mode="missing", size=256)
|
||||
self._use("webp", 80, 256)
|
||||
_, started = await self._start(mode="missing")
|
||||
await self._wait(entered)
|
||||
|
||||
job = self.api._jobs.get(started["job"]["id"])
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Thumbnail settings and naming (#317): the one module every generator reads.
|
||||
|
||||
``py/api/thumbnail_settings.py`` owns ``thumbnail_format`` / ``thumbnail_quality``
|
||||
/ ``thumbnail_edge`` (``PromptManagerConfig``, config.json ``thumbnails``), the
|
||||
preview's name ``thumbnails/<stem>_thumb.<ext of the format>`` and the lookup
|
||||
that accepts any preview a source owns, preferring the configured format.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
sys.modules.setdefault("server", MagicMock())
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from py.api import thumbnail_settings as ts # noqa: E402
|
||||
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
|
||||
|
||||
|
||||
class ConfigCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._orig = PromptManagerConfig.get_config()
|
||||
self.addCleanup(PromptManagerConfig.update_config, self._orig)
|
||||
|
||||
|
||||
class TestDefaultsAndConfig(ConfigCase):
|
||||
def test_defaults_are_webp_80_512(self):
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
settings = ts.current()
|
||||
self.assertEqual(
|
||||
(settings.format, settings.quality, settings.edge), ("webp", 80, 512)
|
||||
)
|
||||
self.assertEqual(settings.extension, ".webp")
|
||||
self.assertEqual(settings.size, (512, 512))
|
||||
self.assertEqual(
|
||||
(ts.DEFAULT_FORMAT, ts.DEFAULT_QUALITY, ts.DEFAULT_EDGE), ("webp", 80, 512)
|
||||
)
|
||||
|
||||
def test_config_round_trip_in_a_thumbnails_section(self):
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "jpeg", "quality": 55, "edge": 300}}
|
||||
)
|
||||
self.assertEqual(
|
||||
PromptManagerConfig.get_config()["thumbnails"],
|
||||
{"format": "jpeg", "quality": 55, "edge": 300},
|
||||
)
|
||||
self.assertEqual(ts.current().extension, ".jpg")
|
||||
|
||||
def test_hand_edited_bad_values_are_ignored(self):
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
for bad in (
|
||||
{"format": "png"},
|
||||
{"format": 3},
|
||||
{"quality": 0},
|
||||
{"quality": 101},
|
||||
{"quality": True},
|
||||
{"quality": "80"},
|
||||
{"edge": 64},
|
||||
{"edge": 4096},
|
||||
{"edge": 512.5},
|
||||
):
|
||||
PromptManagerConfig.update_config({"thumbnails": bad})
|
||||
self.assertEqual(
|
||||
(ts.current().format, ts.current().quality, ts.current().edge),
|
||||
("webp", 80, 512),
|
||||
)
|
||||
PromptManagerConfig.update_config({"thumbnails": "nonsense"})
|
||||
self.assertEqual(ts.current().format, "webp")
|
||||
|
||||
def test_the_never_read_gallery_thumbnail_size_is_gone(self):
|
||||
# R1: older save_to_file wrote its default 256 into config.json; it
|
||||
# must never size a preview.
|
||||
self.assertFalse(hasattr(GalleryConfig, "THUMBNAIL_SIZE"))
|
||||
PromptManagerConfig.update_config(
|
||||
{"gallery": {"web_interface": {"thumbnail_size": 256}}}
|
||||
)
|
||||
self.assertNotIn("thumbnail_size", GalleryConfig.get_config()["web_interface"])
|
||||
self.assertNotEqual(ts.current().edge, 256)
|
||||
|
||||
def test_load_from_file_reads_the_section(self):
|
||||
tmp = tempfile.mkdtemp()
|
||||
self.addCleanup(shutil.rmtree, tmp, True)
|
||||
path = os.path.join(tmp, "config.json")
|
||||
with open(path, "w") as handle:
|
||||
json.dump({"thumbnails": {"format": "jxl", "quality": 90}}, handle)
|
||||
PromptManagerConfig.load_from_file(path)
|
||||
self.assertEqual((ts.current().format, ts.current().quality), ("jxl", 90))
|
||||
|
||||
|
||||
class TestValidation(unittest.TestCase):
|
||||
def test_format(self):
|
||||
for value in ("webp", "jxl", "jpeg"):
|
||||
self.assertEqual(ts.validate_format(value), value)
|
||||
for value in ("png", "JPEG", "", None, 1):
|
||||
with self.assertRaises(ValueError):
|
||||
ts.validate_format(value)
|
||||
|
||||
def test_quality(self):
|
||||
self.assertEqual(ts.validate_quality(1), 1)
|
||||
self.assertEqual(ts.validate_quality(100), 100)
|
||||
for value in (0, 101, True, 80.0, "80", None):
|
||||
with self.assertRaises(ValueError):
|
||||
ts.validate_quality(value)
|
||||
|
||||
def test_edge(self):
|
||||
self.assertEqual(ts.validate_edge(128), 128)
|
||||
self.assertEqual(ts.validate_edge(2048), 2048)
|
||||
for value in (127, 2049, False, 512.0, "512"):
|
||||
with self.assertRaises(ValueError):
|
||||
ts.validate_edge(value)
|
||||
|
||||
|
||||
class TestNaming(ConfigCase):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
|
||||
def test_images_and_videos_take_the_configured_extension(self):
|
||||
self.assertEqual(
|
||||
ts.thumbnail_rel(Path("sub/a.PNG")), "thumbnails/sub/a_thumb.webp"
|
||||
)
|
||||
self.assertEqual(
|
||||
ts.thumbnail_rel(Path("clip.mp4")), "thumbnails/clip_thumb.webp"
|
||||
)
|
||||
self.assertEqual(
|
||||
ts.thumbnail_rel(Path("a.png"), ".jxl"), "thumbnails/a_thumb.jxl"
|
||||
)
|
||||
|
||||
def test_ownership(self):
|
||||
# R4: legacy-only extensions belong to a source of that extension.
|
||||
self.assertTrue(ts.owns(".png", ".png"))
|
||||
self.assertTrue(ts.owns(".png", ".PNG"))
|
||||
self.assertFalse(ts.owns(".png", ".mp4"))
|
||||
self.assertFalse(ts.owns(".png", ".jpg"))
|
||||
self.assertTrue(ts.owns(".jpeg", ".jpeg"))
|
||||
self.assertTrue(ts.owns(".gif", ".gif"))
|
||||
for ext in (".webp", ".jxl", ".jpg"):
|
||||
for source in (".png", ".mp4", ".gif", ".webp"):
|
||||
self.assertTrue(ts.owns(ext, source))
|
||||
self.assertFalse(ts.owns(".txt", ".png"))
|
||||
|
||||
def test_candidates_prefer_the_configured_format_then_the_legacy_name(self):
|
||||
self.assertEqual(
|
||||
ts.candidate_extensions(".png", ".webp"), (".webp", ".png", ".jxl", ".jpg")
|
||||
)
|
||||
self.assertEqual(
|
||||
ts.candidate_extensions(".mp4", ".jxl"), (".jxl", ".jpg", ".webp")
|
||||
)
|
||||
self.assertEqual(
|
||||
ts.candidate_extensions(".jpg", ".jpg"), (".jpg", ".webp", ".jxl")
|
||||
)
|
||||
|
||||
def test_content_types(self):
|
||||
self.assertEqual(ts.content_type(Path("a_thumb.webp")), "image/webp")
|
||||
self.assertEqual(ts.content_type(Path("a_thumb.jxl")), "image/jxl")
|
||||
self.assertEqual(ts.content_type(Path("a_thumb.jpg")), "image/jpeg")
|
||||
self.assertEqual(ts.content_type(Path("a_thumb.PNG")), "image/png")
|
||||
self.assertEqual(ts.content_type(Path("a_thumb.gif")), "image/gif")
|
||||
self.assertIsNone(ts.content_type(Path("a.txt")))
|
||||
|
||||
|
||||
class TestLookup(ConfigCase):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
PromptManagerConfig.update_config(
|
||||
{"thumbnails": {"format": "webp", "quality": 80, "edge": 512}}
|
||||
)
|
||||
self.root = Path(os.path.realpath(tempfile.mkdtemp()))
|
||||
self.addCleanup(shutil.rmtree, self.root, True)
|
||||
|
||||
def _touch(self, rel):
|
||||
path = self.root / rel
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(b"x")
|
||||
return path
|
||||
|
||||
def test_missing_preview_is_none(self):
|
||||
self.assertIsNone(ts.existing_thumbnail(self.root, Path("a.png")))
|
||||
|
||||
def test_legacy_preview_is_found_and_the_configured_one_preferred(self):
|
||||
legacy = self._touch("thumbnails/sub/a_thumb.png")
|
||||
self.assertEqual(ts.existing_thumbnail(self.root, Path("sub/a.png")), legacy)
|
||||
fresh = self._touch("thumbnails/sub/a_thumb.webp")
|
||||
self.assertEqual(ts.existing_thumbnail(self.root, Path("sub/a.png")), fresh)
|
||||
self.assertEqual(
|
||||
sorted(ts.owned_thumbnails(self.root, Path("sub/a.png"))),
|
||||
sorted([legacy, fresh]),
|
||||
)
|
||||
|
||||
def test_a_preview_owned_by_another_source_is_not_used(self):
|
||||
self._touch("thumbnails/a_thumb.png")
|
||||
self.assertIsNone(ts.existing_thumbnail(self.root, Path("a.mp4")))
|
||||
video = self._touch("thumbnails/a_thumb.jpg")
|
||||
self.assertEqual(ts.existing_thumbnail(self.root, Path("a.mp4")), video)
|
||||
|
||||
def test_a_folder_or_link_is_no_preview(self):
|
||||
(self.root / "thumbnails" / "a_thumb.webp").mkdir(parents=True)
|
||||
self.assertIsNone(ts.existing_thumbnail(self.root, Path("a.png")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Vendored
+3
-1
@@ -25,14 +25,16 @@ test('execution script discovers only the new matrix and preserves all 78 origin
|
||||
assert.match(stock, /Total: 82 tests in 8 files/); // + local Inspect host case (inspect-sender.spec.mjs, 2 projects)
|
||||
assert.doesNotMatch(stock, /execution-history/);
|
||||
assert.doesNotMatch(stock, /execution-outputs/);
|
||||
assert.doesNotMatch(stock, /library-location/); // it starts its own fresh host (#249 S6b)
|
||||
const executions = list('tools/vendor/playwright.execution.config.mjs');
|
||||
assert.match(executions, /\[classic-start\]/);
|
||||
assert.match(executions, /\[nodes2-start\]/);
|
||||
assert.match(executions, /Total: 46 tests in 2 files/);
|
||||
assert.match(executions, /Total: 48 tests in 3 files/); // + the fresh-host library location case
|
||||
for (const project of ['classic-start', 'nodes2-start']) {
|
||||
const cases = executions.split('\n').filter(line => line.includes(`[${project}]`));
|
||||
assert.equal(cases.filter(line => line.includes('execution-history.spec.mjs')).length, 15);
|
||||
assert.equal(cases.filter(line => line.includes('execution-outputs.spec.mjs')).length, 8);
|
||||
assert.equal(cases.filter(line => line.includes('library-location.spec.mjs')).length, 1);
|
||||
}
|
||||
assert.doesNotMatch(executions, /prompt-manager.spec/);
|
||||
});
|
||||
|
||||
Vendored
+1
-1
@@ -19,7 +19,7 @@ const servers = [{
|
||||
|
||||
export default defineConfig({
|
||||
testDir: '../../tests/comfyui',
|
||||
testIgnore: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'],
|
||||
testIgnore: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs', 'library-location.spec.mjs'],
|
||||
outputDir: './test-results/comfyui',
|
||||
timeout: 60000,
|
||||
workers: 1,
|
||||
|
||||
+32
-14
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"admin.js": {
|
||||
"lines": 31,
|
||||
"lines": 32,
|
||||
"branches": 58,
|
||||
"functions": 26,
|
||||
"statements": 31
|
||||
"functions": 27,
|
||||
"statements": 32
|
||||
},
|
||||
"api-client.js": {
|
||||
"lines": 98,
|
||||
@@ -188,7 +188,7 @@
|
||||
"workspace/file-inspector.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 93,
|
||||
"functions": 92,
|
||||
"functions": 93,
|
||||
"statements": 96
|
||||
},
|
||||
"workspace/heading.mjs": {
|
||||
@@ -241,7 +241,7 @@
|
||||
},
|
||||
"workspace/image-tagging.mjs": {
|
||||
"lines": 97,
|
||||
"branches": 90,
|
||||
"branches": 92,
|
||||
"functions": 88,
|
||||
"statements": 97
|
||||
},
|
||||
@@ -335,6 +335,12 @@
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/library-location.mjs": {
|
||||
"lines": 94,
|
||||
"branches": 91,
|
||||
"functions": 93,
|
||||
"statements": 94
|
||||
},
|
||||
"workspace/library-state.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 95,
|
||||
@@ -366,14 +372,14 @@
|
||||
"statements": 97
|
||||
},
|
||||
"workspace/local-resolver-stages.mjs": {
|
||||
"lines": 94,
|
||||
"lines": 95,
|
||||
"branches": 93,
|
||||
"functions": 98,
|
||||
"statements": 94
|
||||
"statements": 95
|
||||
},
|
||||
"workspace/local-resolver-support.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 80,
|
||||
"branches": 81,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
},
|
||||
@@ -420,10 +426,10 @@
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/prompt-blocks.mjs": {
|
||||
"lines": 90,
|
||||
"branches": 92,
|
||||
"lines": 98,
|
||||
"branches": 95,
|
||||
"functions": 98,
|
||||
"statements": 90
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/prompt-inspector.mjs": {
|
||||
"lines": 98,
|
||||
@@ -481,7 +487,7 @@
|
||||
},
|
||||
"workspace/system-controls.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 87,
|
||||
"branches": 89,
|
||||
"functions": 89,
|
||||
"statements": 98
|
||||
},
|
||||
@@ -503,6 +509,12 @@
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/system-thumbnails.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 95,
|
||||
"functions": 83,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/system.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 92,
|
||||
@@ -516,10 +528,16 @@
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/thumbnail-review.mjs": {
|
||||
"lines": 96,
|
||||
"lines": 98,
|
||||
"branches": 93,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/thumbnail-support.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 98,
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/toast.mjs": {
|
||||
"lines": 98,
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ if (host.href !== 'http://127.0.0.1:8196/') throw new Error('Execution tests req
|
||||
|
||||
export default defineConfig({
|
||||
testDir: '../../tests/comfyui',
|
||||
testMatch: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'],
|
||||
testMatch: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs', 'library-location.spec.mjs'],
|
||||
outputDir: './test-results/executions/cases',
|
||||
timeout: 60000,
|
||||
workers: 1,
|
||||
|
||||
+7
-13
@@ -69,8 +69,8 @@ class GalleryDiagnostics:
|
||||
|
||||
Args:
|
||||
db_path: Path to the SQLite database file to diagnose.
|
||||
If None, resolves from config.json or defaults to
|
||||
prompts.db in the extension root.
|
||||
If None, the library's database (see
|
||||
``utils.library_location``).
|
||||
"""
|
||||
if db_path is None:
|
||||
db_path = self._resolve_db_path()
|
||||
@@ -79,18 +79,12 @@ class GalleryDiagnostics:
|
||||
|
||||
@staticmethod
|
||||
def _resolve_db_path() -> str:
|
||||
"""Resolve database path from config, matching PromptDatabase logic."""
|
||||
extension_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
"""The library's database: the same resolver as PromptDatabase."""
|
||||
try:
|
||||
from py.config import PromptManagerConfig
|
||||
|
||||
db_path = PromptManagerConfig.DEFAULT_DB_PATH
|
||||
except Exception:
|
||||
db_path = "prompts.db"
|
||||
|
||||
if not os.path.isabs(db_path):
|
||||
db_path = os.path.normpath(os.path.join(extension_root, db_path))
|
||||
return db_path
|
||||
from ..database.operations import resolve_library_location
|
||||
except ImportError:
|
||||
from database.operations import resolve_library_location
|
||||
return resolve_library_location().db_path
|
||||
|
||||
def run_full_diagnostic(self) -> Dict[str, Any]:
|
||||
"""Run a complete diagnostic check.
|
||||
|
||||
+67
-17
@@ -36,6 +36,7 @@ from .metadata_extractor import ComfyUIMetadataExtractor
|
||||
from .logging_config import get_logger
|
||||
from .prompt_graph import resolve_text, run_prompt_nodes
|
||||
from .prompt_pair_resolver import resolve_prompt_pair
|
||||
from .usage_tracking import survive_switch
|
||||
|
||||
try:
|
||||
from ..database.pair_ingestion import ingest_resolved_pair
|
||||
@@ -79,6 +80,10 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
# ``record_review_file``, wired at start-up): a linked output is
|
||||
# never queued and its pending items leave the queue. None: no pass.
|
||||
self.review_recorder: Optional[Callable[[str], str]] = None
|
||||
# Queues a settled output's preview (the API's
|
||||
# ``queue_capture_thumbnail``, wired at start-up; #249). It only
|
||||
# queues: the preview is written on its own worker. None: no queue.
|
||||
self.thumbnail_queue: Optional[Callable[[str], str]] = None
|
||||
|
||||
# Read from GalleryConfig if available, otherwise use defaults
|
||||
try:
|
||||
@@ -220,10 +225,30 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
)
|
||||
return
|
||||
self.process_new_image(image_path, prompt_snapshot=prompt_snapshot)
|
||||
self._queue_thumbnail(image_path)
|
||||
finally:
|
||||
with self._pending_lock:
|
||||
self._pending.pop(image_path, None)
|
||||
|
||||
def _queue_thumbnail(self, image_path: str) -> None:
|
||||
"""Queue a settled output's preview, best effort (linked or not).
|
||||
|
||||
A failure is logged, never raised: the preview is then made on its
|
||||
first view.
|
||||
"""
|
||||
queue_preview = getattr(self, "thumbnail_queue", None)
|
||||
if queue_preview is None:
|
||||
return
|
||||
try:
|
||||
outcome = queue_preview(image_path)
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
f"Preview of {os.path.basename(image_path)} could not be queued",
|
||||
exc_info=True,
|
||||
)
|
||||
return
|
||||
self.logger.debug(f"Preview {outcome}: {os.path.basename(image_path)}")
|
||||
|
||||
def wait_until_settled(self, image_path: str, stop=None) -> bool:
|
||||
"""True once the file's size is non-zero and unchanged across two reads.
|
||||
|
||||
@@ -563,24 +588,38 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
"""
|
||||
try:
|
||||
resolution = resolve_prompt_pair(metadata)
|
||||
if resolution["state"] == "resolved":
|
||||
positive = resolution["positive"]
|
||||
expected = self.db_manager.get_prompt_by_hash(
|
||||
generate_prompt_hash(positive.get("storage_text", positive["text"]))
|
||||
)
|
||||
if expected and expected["id"] == prompt_context["id"]:
|
||||
ingest_resolved_pair(
|
||||
self.db_manager, image_path, metadata, resolution
|
||||
|
||||
def link():
|
||||
# One write; run again on the moved library when a Move
|
||||
# library switched while it waited (#249 S6b).
|
||||
if resolution["state"] == "resolved":
|
||||
positive = resolution["positive"]
|
||||
expected = self.db_manager.get_prompt_by_hash(
|
||||
generate_prompt_hash(
|
||||
positive.get("storage_text", positive["text"])
|
||||
)
|
||||
)
|
||||
return
|
||||
image_id = self.db_manager.link_image_to_prompt(
|
||||
prompt_id=prompt_context["id"], image_path=image_path, metadata=metadata
|
||||
)
|
||||
fallback_note = " (fallback)" if prompt_context.get("fallback") else ""
|
||||
self.logger.debug(
|
||||
f"Successfully linked image {image_id} to prompt "
|
||||
f"{prompt_context['id']}{fallback_note}"
|
||||
)
|
||||
if expected and expected["id"] == prompt_context["id"]:
|
||||
ingest_resolved_pair(
|
||||
self.db_manager, image_path, metadata, resolution
|
||||
)
|
||||
return None
|
||||
# The raising form: a failed write reaches survive_switch.
|
||||
return self.db_manager.link_image_to_prompt_or_raise(
|
||||
prompt_id=prompt_context["id"],
|
||||
image_path=image_path,
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
image_id = survive_switch(self.db_manager, link)
|
||||
if image_id is None:
|
||||
return
|
||||
if image_id: # 0: nothing new was linked (logged by the database)
|
||||
fallback_note = " (fallback)" if prompt_context.get("fallback") else ""
|
||||
self.logger.debug(
|
||||
f"Successfully linked image {image_id} to prompt "
|
||||
f"{prompt_context['id']}{fallback_note}"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to link image to prompt: {e}")
|
||||
return
|
||||
@@ -636,6 +675,16 @@ class ImageMonitor:
|
||||
self.monitored_directories = []
|
||||
self.logger = get_logger("prompt_manager.image_monitor")
|
||||
self.review_recorder: Optional[Callable[[str], str]] = None
|
||||
self.thumbnail_queue: Optional[Callable[[str], str]] = None
|
||||
|
||||
def set_thumbnail_queue(
|
||||
self, queue_preview: Optional[Callable[[str], str]]
|
||||
) -> None:
|
||||
"""Queue every settled output's preview through ``queue_preview``
|
||||
(None stops it), on the running handler and every later one."""
|
||||
self.thumbnail_queue = queue_preview
|
||||
if self.handler is not None:
|
||||
self.handler.thumbnail_queue = queue_preview
|
||||
|
||||
def set_review_recorder(self, recorder: Optional[Callable[[str], str]]) -> None:
|
||||
"""Record ambiguous outputs through ``recorder`` (None stops recording).
|
||||
@@ -703,6 +752,7 @@ class ImageMonitor:
|
||||
# leaves the monitor stuck "already running" with nothing to stop.
|
||||
handler = ImageGenerationHandler(self.db_manager, self.prompt_tracker)
|
||||
handler.review_recorder = self.review_recorder
|
||||
handler.thumbnail_queue = self.thumbnail_queue
|
||||
observer = Observer()
|
||||
scheduled = []
|
||||
for output_dir in valid_directories:
|
||||
|
||||
@@ -0,0 +1,480 @@
|
||||
"""Where the library lives (#249 S6b): the one resolver for every data path.
|
||||
|
||||
The library is ``prompts.db``, ``config.json`` (it can hold the CivitAI key),
|
||||
the safety backups beside the database, the logs and the LoRA image cache.
|
||||
Inside the node folder an uninstall, a reinstall or a folder-replacing update
|
||||
deletes it, so new installs keep it in ComfyUI's system user folder
|
||||
``user/__promptmanager`` (``folder_paths.get_system_user_directory``), which
|
||||
ComfyUI's ``/userdata`` HTTP API refuses to serve.
|
||||
|
||||
Resolution order (``resolve_library``):
|
||||
|
||||
1. ``PROMPT_MANAGER_DATA_DIR``: an absolute folder outside every folder
|
||||
ComfyUI or Prompt Manager serves. It is used when it holds a library or
|
||||
when no node-folder library exists; an empty one never replaces a
|
||||
node-folder library (that one stays in use, and Move library targets the
|
||||
environment folder). An invalid value is ignored and reported.
|
||||
2. An explicit ``database.default_path`` in config.json, honoured as before
|
||||
(a relative path is relative to the node folder). ``PROMPT_MANAGER_CONFIG_PATH``
|
||||
still names config.json itself.
|
||||
3. The default data folder once a move finished there (its marker says
|
||||
``moved``), even before the old file was renamed.
|
||||
4. A node-folder ``prompts.db`` that was not moved: it stays in use until the
|
||||
user moves it.
|
||||
5. The default data folder (new installs). Outside ComfyUI (no
|
||||
``folder_paths``) the node folder, as before.
|
||||
|
||||
Nothing here moves, copies or creates data: it only reads the file system.
|
||||
After a live move the process uses ``switch_to``'s location until restart.
|
||||
This module imports only the standard library, so the logger can use it.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from typing import Iterable, List, Mapping, Optional, Tuple
|
||||
|
||||
DATA_DIR_ENV = "PROMPT_MANAGER_DATA_DIR"
|
||||
CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH"
|
||||
EXTRA_GALLERY_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
|
||||
SYSTEM_USER_NAME = "promptmanager"
|
||||
SYSTEM_USER_PREFIX = "__"
|
||||
DATABASE_NAME = "prompts.db"
|
||||
CONFIG_NAME = "config.json"
|
||||
MARKER_NAME = "library-location.json"
|
||||
# config.json has always written its default; that value is not a choice.
|
||||
DEFAULT_DB_SETTING = "prompts.db"
|
||||
|
||||
SOURCE_ENVIRONMENT = "environment"
|
||||
SOURCE_CONFIG = "config"
|
||||
SOURCE_NODE_FOLDER = "node_folder"
|
||||
SOURCE_DEFAULT = "default"
|
||||
SOURCES = (SOURCE_ENVIRONMENT, SOURCE_CONFIG, SOURCE_NODE_FOLDER, SOURCE_DEFAULT)
|
||||
|
||||
MARKER_PENDING = "pending"
|
||||
MARKER_MOVED = "moved"
|
||||
|
||||
EXTENSION_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
_UNSET = object()
|
||||
_override_lock = threading.Lock()
|
||||
_override: Optional["LibraryLocation"] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LibraryLocation:
|
||||
"""Where the library is and why.
|
||||
|
||||
``target_dir`` is where Move library would put it (None outside ComfyUI)
|
||||
and ``target_source`` why (the environment folder or the default one);
|
||||
``problem`` is a code for something the user should know (an invalid or
|
||||
empty ``PROMPT_MANAGER_DATA_DIR``, a move that did not finish).
|
||||
``config_explicit``: config.json is named by ``PROMPT_MANAGER_CONFIG_PATH``
|
||||
(it is the user's and never moves).
|
||||
"""
|
||||
|
||||
source: str
|
||||
db_path: str
|
||||
config_path: str
|
||||
data_dir: str
|
||||
target_dir: Optional[str] = None
|
||||
problem: Optional[str] = None
|
||||
target_source: Optional[str] = None
|
||||
config_explicit: bool = False
|
||||
|
||||
@property
|
||||
def movable(self) -> bool:
|
||||
return self.source == SOURCE_NODE_FOLDER and self.target_dir is not None
|
||||
|
||||
|
||||
def import_folder_paths():
|
||||
"""ComfyUI's ``folder_paths`` module, or None outside ComfyUI."""
|
||||
try:
|
||||
import folder_paths
|
||||
except ImportError:
|
||||
return None
|
||||
return folder_paths
|
||||
|
||||
|
||||
def _folder_paths(value):
|
||||
return import_folder_paths() if value is _UNSET else value
|
||||
|
||||
|
||||
def _call(module, name, *args) -> Optional[str]:
|
||||
getter = getattr(module, name, None)
|
||||
if not callable(getter):
|
||||
return None
|
||||
try:
|
||||
value = getter(*args)
|
||||
except Exception: # an older or broken host: treat the folder as unknown
|
||||
return None
|
||||
return str(value) if value else None
|
||||
|
||||
|
||||
def default_data_dir(folder_paths=_UNSET) -> Optional[str]:
|
||||
"""``user/__promptmanager``, or None outside ComfyUI.
|
||||
|
||||
ComfyUI builds without ``get_system_user_directory`` get the same folder
|
||||
joined by hand.
|
||||
"""
|
||||
module = _folder_paths(folder_paths)
|
||||
if module is None:
|
||||
return None
|
||||
folder = _call(module, "get_system_user_directory", SYSTEM_USER_NAME)
|
||||
if folder:
|
||||
return os.path.abspath(folder)
|
||||
user = _call(module, "get_user_directory")
|
||||
if not user:
|
||||
return None
|
||||
return os.path.abspath(os.path.join(user, SYSTEM_USER_PREFIX + SYSTEM_USER_NAME))
|
||||
|
||||
|
||||
def _identity(path: str) -> Optional[Tuple[int, int]]:
|
||||
try:
|
||||
stat = os.stat(path)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return (stat.st_dev, stat.st_ino)
|
||||
|
||||
|
||||
def same_folder(first: str, second: str) -> bool:
|
||||
"""Whether two spellings name one existing folder (links and case folded)."""
|
||||
one, two = _identity(first), _identity(second)
|
||||
if one is None or two is None:
|
||||
return False
|
||||
if one[1] == 0: # no file id on this volume: compare the spelling
|
||||
return _spelling(first) == _spelling(second)
|
||||
return one == two
|
||||
|
||||
|
||||
def _spelling(path: str) -> str:
|
||||
return os.path.normcase(os.path.realpath(path))
|
||||
|
||||
|
||||
def _existing_ancestors(path: str) -> List[str]:
|
||||
"""``path`` (resolved) and its parents that exist, nearest first."""
|
||||
current = os.path.realpath(path)
|
||||
found = []
|
||||
while True:
|
||||
if os.path.isdir(current):
|
||||
found.append(current)
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
return found
|
||||
current = parent
|
||||
|
||||
|
||||
def _inside_any(path: str, folders: Iterable[str]) -> bool:
|
||||
"""Whether ``path`` is one of ``folders`` or below one.
|
||||
|
||||
Compared by spelling (realpath + normcase) and by file identity of every
|
||||
existing ancestor, so symlinks and case-insensitive volumes (Windows,
|
||||
macOS) cannot hide a served folder.
|
||||
"""
|
||||
folders = [folder for folder in folders if folder]
|
||||
if not folders:
|
||||
return False
|
||||
spelling = _spelling(path)
|
||||
sep = os.path.sep
|
||||
for folder in folders:
|
||||
root = _spelling(folder)
|
||||
if spelling == root or spelling.startswith(root.rstrip(sep) + sep):
|
||||
return True
|
||||
identities = {_identity(folder) for folder in folders}
|
||||
identities.discard(None)
|
||||
identities = {item for item in identities if item[1] != 0}
|
||||
return any(_identity(a) in identities for a in _existing_ancestors(path))
|
||||
|
||||
|
||||
def _served_folders(module, environ, gallery_roots) -> List[str]:
|
||||
"""Folders whose files ComfyUI or Prompt Manager hand out over HTTP."""
|
||||
folders = []
|
||||
if module is not None:
|
||||
for name in (
|
||||
"get_output_directory",
|
||||
"get_input_directory",
|
||||
"get_temp_directory",
|
||||
):
|
||||
folder = _call(module, name)
|
||||
if folder:
|
||||
folders.append(folder)
|
||||
extra = (environ or {}).get(EXTRA_GALLERY_ROOTS_ENV, "")
|
||||
folders.extend(part for part in extra.split(os.pathsep) if part.strip())
|
||||
folders.extend(str(root) for root in gallery_roots or () if root)
|
||||
return folders
|
||||
|
||||
|
||||
def _in_public_user_folder(path: str, module) -> bool:
|
||||
"""Whether ``path`` lies in a ``/userdata``-served user folder (not ``__``)."""
|
||||
user = _call(module, "get_user_directory") if module is not None else None
|
||||
if not user:
|
||||
return False
|
||||
if not _inside_any(path, [user]) or same_folder(path, user):
|
||||
return False
|
||||
real_user = _spelling(user)
|
||||
relative = os.path.relpath(_spelling(path), real_user)
|
||||
first = relative.split(os.sep, 1)[0]
|
||||
if first.startswith(os.pardir):
|
||||
# Reached through a link or another spelling: judge by identity.
|
||||
for ancestor in _existing_ancestors(path):
|
||||
if same_folder(os.path.dirname(ancestor), user):
|
||||
first = os.path.basename(ancestor)
|
||||
break
|
||||
else:
|
||||
return False
|
||||
return not first.startswith(SYSTEM_USER_PREFIX)
|
||||
|
||||
|
||||
def data_dir_problem(
|
||||
path, *, folder_paths=_UNSET, environ=None, gallery_roots=()
|
||||
) -> Optional[str]:
|
||||
"""Why ``path`` cannot hold the library, as a code, or None.
|
||||
|
||||
``data-dir-not-absolute``: empty or relative. ``data-dir-served``: inside
|
||||
ComfyUI's output, input or temp folder, a public user folder, an extra
|
||||
gallery root or a gallery source (their files are served over HTTP).
|
||||
"""
|
||||
if not isinstance(path, str) or not path.strip() or not os.path.isabs(path):
|
||||
return "data-dir-not-absolute"
|
||||
module = _folder_paths(folder_paths)
|
||||
if _inside_any(path, _served_folders(module, environ, gallery_roots)):
|
||||
return "data-dir-served"
|
||||
if _in_public_user_folder(path, module):
|
||||
return "data-dir-served"
|
||||
return None
|
||||
|
||||
|
||||
def read_marker(directory: Optional[str]) -> dict:
|
||||
"""The move marker in ``directory``, or ``{}`` when absent or unreadable."""
|
||||
if not directory:
|
||||
return {}
|
||||
try:
|
||||
with open(os.path.join(directory, MARKER_NAME), "r", encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
if not isinstance(data, dict) or data.get("state") not in (
|
||||
MARKER_PENDING,
|
||||
MARKER_MOVED,
|
||||
):
|
||||
return {}
|
||||
return data
|
||||
|
||||
|
||||
def _marker_state(directory) -> Optional[str]:
|
||||
return read_marker(directory).get("state")
|
||||
|
||||
|
||||
def switch_to(location: LibraryLocation) -> None:
|
||||
"""Use ``location`` for the rest of this process (after a live move)."""
|
||||
global _override
|
||||
with _override_lock:
|
||||
_override = location
|
||||
|
||||
|
||||
def clear_override() -> None:
|
||||
"""Forget a live move's location (tests)."""
|
||||
global _override
|
||||
with _override_lock:
|
||||
_override = None
|
||||
|
||||
|
||||
def current_override() -> Optional[LibraryLocation]:
|
||||
with _override_lock:
|
||||
return _override
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Decision:
|
||||
data_dir: str
|
||||
source: str
|
||||
target_dir: Optional[str]
|
||||
problem: Optional[str]
|
||||
db_dir: Optional[str] = None # when the database is not in data_dir
|
||||
target_source: Optional[str] = None
|
||||
|
||||
|
||||
def _configured_sources(module, environ, extension_root, default, raw_env):
|
||||
"""The image sources every config.json this resolver might read lists.
|
||||
|
||||
``config_location`` runs before any config.json is loaded, so it cannot be
|
||||
handed the configured sources; both it and ``resolve_library`` read them
|
||||
here, from each candidate file (the explicit one, the environment
|
||||
folder's, the node folder's and the default folder's), so the two always
|
||||
judge ``PROMPT_MANAGER_DATA_DIR`` alike. Relative sources are taken from
|
||||
ComfyUI's base folder, as the gallery resolves them.
|
||||
"""
|
||||
candidates = [environ.get(CONFIG_PATH_ENV, "").strip()]
|
||||
if raw_env and os.path.isabs(raw_env):
|
||||
candidates.append(os.path.join(raw_env, CONFIG_NAME))
|
||||
candidates.append(os.path.join(extension_root, CONFIG_NAME))
|
||||
if default:
|
||||
candidates.append(os.path.join(default, CONFIG_NAME))
|
||||
base = getattr(module, "base_path", None) if module is not None else None
|
||||
sources = []
|
||||
for candidate in candidates:
|
||||
if not candidate or not os.path.isfile(candidate):
|
||||
continue
|
||||
try:
|
||||
with open(candidate, "r", encoding="utf-8") as handle:
|
||||
data = json.load(handle)
|
||||
directories = data["gallery"]["monitoring"]["directories"]
|
||||
except (OSError, ValueError, KeyError, TypeError):
|
||||
continue
|
||||
if not isinstance(directories, list):
|
||||
continue
|
||||
for directory in directories:
|
||||
if not isinstance(directory, str) or not directory.strip():
|
||||
continue
|
||||
directory = directory.strip()
|
||||
if not os.path.isabs(directory):
|
||||
if not base:
|
||||
continue
|
||||
directory = os.path.join(str(base), directory)
|
||||
sources.append(directory)
|
||||
return sources
|
||||
|
||||
|
||||
def _decide(environ, extension_root, folder_paths, gallery_roots) -> _Decision:
|
||||
"""Which folder holds the library (before an explicit database path)."""
|
||||
module = _folder_paths(folder_paths)
|
||||
environ = os.environ if environ is None else environ
|
||||
legacy_db = os.path.isfile(os.path.join(extension_root, DATABASE_NAME))
|
||||
legacy = legacy_db or os.path.isfile(os.path.join(extension_root, CONFIG_NAME))
|
||||
default = default_data_dir(module)
|
||||
problem = None
|
||||
env_dir = None
|
||||
raw = environ.get(DATA_DIR_ENV, "")
|
||||
if raw.strip():
|
||||
sources = list(gallery_roots or ()) + _configured_sources(
|
||||
module, environ, extension_root, default, raw.strip()
|
||||
)
|
||||
problem = data_dir_problem(
|
||||
raw.strip(),
|
||||
folder_paths=module,
|
||||
environ=environ,
|
||||
gallery_roots=sources,
|
||||
)
|
||||
if problem is None:
|
||||
env_dir = os.path.abspath(raw.strip())
|
||||
target = env_dir or default
|
||||
target_source = SOURCE_ENVIRONMENT if env_dir else SOURCE_DEFAULT
|
||||
|
||||
def decision(data_dir, source, problem=None, db_dir=None):
|
||||
return _Decision(
|
||||
data_dir, source, target, problem, db_dir, target_source if target else None
|
||||
)
|
||||
|
||||
if env_dir is not None:
|
||||
state = _marker_state(env_dir)
|
||||
holds = os.path.isfile(os.path.join(env_dir, DATABASE_NAME))
|
||||
if state != MARKER_PENDING and (holds or not legacy_db):
|
||||
return decision(env_dir, SOURCE_ENVIRONMENT)
|
||||
problem = "move-unfinished" if state == MARKER_PENDING else "data-dir-empty"
|
||||
return decision(extension_root, SOURCE_NODE_FOLDER, problem)
|
||||
if default is None:
|
||||
return decision(extension_root, SOURCE_NODE_FOLDER, problem)
|
||||
if _marker_state(default) == MARKER_MOVED and os.path.isfile(
|
||||
os.path.join(default, DATABASE_NAME)
|
||||
):
|
||||
return decision(default, SOURCE_DEFAULT, problem)
|
||||
if legacy:
|
||||
if _marker_state(default) == MARKER_PENDING:
|
||||
problem = problem or "move-unfinished"
|
||||
if legacy_db:
|
||||
return decision(extension_root, SOURCE_NODE_FOLDER, problem)
|
||||
# Only a node-folder config.json: keep reading it, but a new database
|
||||
# never goes into the node folder.
|
||||
return decision(extension_root, SOURCE_DEFAULT, problem, default)
|
||||
return decision(default, SOURCE_DEFAULT, problem)
|
||||
|
||||
|
||||
def config_location(
|
||||
environ=None, extension_root=None, folder_paths=_UNSET, gallery_roots=()
|
||||
) -> Tuple[str, str]:
|
||||
"""``(config.json path, data folder)``; needs no config value (no cycle)."""
|
||||
override = current_override()
|
||||
if override is not None:
|
||||
return override.config_path, override.data_dir
|
||||
environ = os.environ if environ is None else environ
|
||||
root = extension_root or EXTENSION_ROOT
|
||||
decision = _decide(environ, root, folder_paths, gallery_roots)
|
||||
explicit = environ.get(CONFIG_PATH_ENV, "").strip()
|
||||
config_path = explicit or os.path.join(decision.data_dir, CONFIG_NAME)
|
||||
return config_path, decision.data_dir
|
||||
|
||||
|
||||
def is_explicit_db_setting(value) -> bool:
|
||||
"""Whether config.json's ``database.default_path`` names a chosen file."""
|
||||
return isinstance(value, str) and value.strip() not in ("", DEFAULT_DB_SETTING)
|
||||
|
||||
|
||||
def resolve_library(
|
||||
default_path=None,
|
||||
*,
|
||||
environ: Optional[Mapping[str, str]] = None,
|
||||
extension_root: Optional[str] = None,
|
||||
folder_paths=_UNSET,
|
||||
gallery_roots=(),
|
||||
) -> LibraryLocation:
|
||||
"""The library's location now (the module docstring has the order)."""
|
||||
override = current_override()
|
||||
if override is not None:
|
||||
return override
|
||||
environ = os.environ if environ is None else environ
|
||||
root = extension_root or EXTENSION_ROOT
|
||||
decision = _decide(environ, root, folder_paths, gallery_roots)
|
||||
explicit_config = environ.get(CONFIG_PATH_ENV, "").strip()
|
||||
config_path = explicit_config or os.path.join(decision.data_dir, CONFIG_NAME)
|
||||
common = {
|
||||
"target_source": decision.target_source,
|
||||
"config_explicit": bool(explicit_config),
|
||||
}
|
||||
if decision.source != SOURCE_ENVIRONMENT and is_explicit_db_setting(default_path):
|
||||
db_path = default_path.strip()
|
||||
if not os.path.isabs(db_path):
|
||||
db_path = os.path.normpath(os.path.join(root, db_path))
|
||||
return LibraryLocation(
|
||||
SOURCE_CONFIG,
|
||||
db_path,
|
||||
config_path,
|
||||
decision.data_dir,
|
||||
decision.target_dir,
|
||||
decision.problem,
|
||||
**common,
|
||||
)
|
||||
return LibraryLocation(
|
||||
decision.source,
|
||||
os.path.join(decision.db_dir or decision.data_dir, DATABASE_NAME),
|
||||
config_path,
|
||||
decision.data_dir,
|
||||
decision.target_dir,
|
||||
decision.problem,
|
||||
**common,
|
||||
)
|
||||
|
||||
|
||||
def ensure_private_dir(path: str) -> str:
|
||||
"""Create ``path`` (and parents) if missing; a new folder is owner-only.
|
||||
|
||||
An existing folder is left as it is (its permissions are the user's).
|
||||
"""
|
||||
if not os.path.isdir(path):
|
||||
os.makedirs(path, mode=0o700, exist_ok=True)
|
||||
if os.name == "posix":
|
||||
try:
|
||||
os.chmod(path, 0o700)
|
||||
except OSError:
|
||||
pass
|
||||
return path
|
||||
|
||||
|
||||
def logs_dir(**kwargs) -> str:
|
||||
"""The folder for Prompt Manager's log files."""
|
||||
return os.path.join(config_location(**kwargs)[1], "logs")
|
||||
|
||||
|
||||
def cache_dir(name: str, **kwargs) -> str:
|
||||
"""A re-creatable cache folder, ``<data folder>/data/<name>``."""
|
||||
return os.path.join(config_location(**kwargs)[1], "data", name)
|
||||
@@ -18,6 +18,8 @@ from datetime import datetime
|
||||
from typing import Dict, Any, Optional, List
|
||||
from pathlib import Path
|
||||
|
||||
from . import library_location
|
||||
|
||||
|
||||
class _OwnedRotatingFileHandler(logging.handlers.RotatingFileHandler):
|
||||
"""Keep a retired owned sink closed when an in-flight record resumes."""
|
||||
@@ -65,8 +67,12 @@ class PromptManagerLogger:
|
||||
return
|
||||
|
||||
self._initialized = True
|
||||
self.log_dir = Path(__file__).parent.parent / "logs"
|
||||
self.log_dir.mkdir(exist_ok=True)
|
||||
# The library's data folder (#249 S6b): the node folder for a library
|
||||
# that was not moved, else ComfyUI's user/__promptmanager.
|
||||
self.log_dir = Path(
|
||||
library_location.logs_dir(extension_root=str(Path(__file__).parent.parent))
|
||||
)
|
||||
library_location.ensure_private_dir(str(self.log_dir))
|
||||
|
||||
# Configuration
|
||||
self.config = {
|
||||
|
||||
+29
-8
@@ -187,6 +187,19 @@ def find_pair_id(db: Any, positive: str, negative: str) -> Optional[int]:
|
||||
return row["id"] if row else None
|
||||
|
||||
|
||||
def survive_switch(db: Any, write: Callable[[], Any]) -> Any:
|
||||
"""``write()`` through ``db.survive_library_switch`` when the database has it.
|
||||
|
||||
A write that was waiting on the old file when Move library switched runs
|
||||
again on the new library instead of being lost (#249 S6b). Stand-in
|
||||
databases without the method (tests) just run ``write()``.
|
||||
"""
|
||||
survive = getattr(type(db), "survive_library_switch", None)
|
||||
if survive is None:
|
||||
return write()
|
||||
return survive(db, write)
|
||||
|
||||
|
||||
def count_pair_runs(
|
||||
runs: List[Tuple[str, str, str]],
|
||||
db: Any,
|
||||
@@ -194,20 +207,28 @@ def count_pair_runs(
|
||||
) -> None:
|
||||
"""Count existing recipes at dequeue and defer new recipes until first save."""
|
||||
for key, positive, negative in runs:
|
||||
pair_id = find_pair_id(db, positive, negative)
|
||||
if pair_id:
|
||||
db.record_prompt_pair_use(pair_id)
|
||||
else:
|
||||
|
||||
def count(positive=positive, negative=negative):
|
||||
pair_id = find_pair_id(db, positive, negative)
|
||||
if pair_id:
|
||||
db.record_prompt_pair_use(pair_id)
|
||||
return bool(pair_id)
|
||||
|
||||
if not survive_switch(db, count):
|
||||
pending.add(key)
|
||||
|
||||
|
||||
def count_runs(runs: List[Run], db: Any, pending: PendingFirstUse) -> None:
|
||||
"""Record one use per prompt in the database; remember the rest for first save."""
|
||||
for prompt_hash, _text in runs:
|
||||
existing = db.get_prompt_by_hash(prompt_hash)
|
||||
if existing:
|
||||
db.record_prompt_use(existing["id"])
|
||||
else:
|
||||
|
||||
def count(prompt_hash=prompt_hash):
|
||||
existing = db.get_prompt_by_hash(prompt_hash)
|
||||
if existing:
|
||||
db.record_prompt_use(existing["id"])
|
||||
return bool(existing)
|
||||
|
||||
if not survive_switch(db, count):
|
||||
pending.add(prompt_hash)
|
||||
|
||||
|
||||
|
||||
@@ -79,6 +79,8 @@
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- #249 S6b: where the library (database, config.json, backups) lives and why. -->
|
||||
<p class="text-pm-secondary text-xs mt-2" id="libraryLocation" hidden></p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -297,6 +297,7 @@
|
||||
this.loadTags(),
|
||||
this.loadRecentPrompts(),
|
||||
this.loadSettings(),
|
||||
this.loadLibraryLocation(),
|
||||
]);
|
||||
} catch (error) {
|
||||
this.showNotification("Failed to load initial data", "error");
|
||||
@@ -344,6 +345,29 @@
|
||||
this.listRequest = ApiClient.latestOnly({ lifetime: this.pageLifetime.signal });
|
||||
}
|
||||
|
||||
/** #249 S6b: one line under the statistics saying where the library lives and why. */
|
||||
async loadLibraryLocation() {
|
||||
const { signal } = this.pageLifetime;
|
||||
if (signal.aborted) return;
|
||||
const line = document.getElementById("libraryLocation");
|
||||
const reasons = {
|
||||
default: "ComfyUI's user folder",
|
||||
environment: "set by PROMPT_MANAGER_DATA_DIR",
|
||||
config: "set in config.json",
|
||||
node_folder: "the node folder, where an update or reinstall can delete it; move it from the workspace's System page",
|
||||
};
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl("/prompt_manager/system/library", { signal });
|
||||
const data = response.ok ? await response.json() : null;
|
||||
const location = data && data.location;
|
||||
if (!line || !location || typeof location.path !== "string") return;
|
||||
line.textContent = `Library: ${location.path} (${reasons[location.source] || location.source})`;
|
||||
line.hidden = false;
|
||||
} catch (error) {
|
||||
if (!signal.aborted) console.error("Library location error:", error);
|
||||
}
|
||||
}
|
||||
|
||||
async loadStatistics() {
|
||||
// A leaving page starts no read; one in flight is aborted with the page and not reported.
|
||||
const { signal } = this.pageLifetime;
|
||||
@@ -3418,6 +3442,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
if (image.thumbnail_url) return ApiClient.resolveUrl(image.thumbnail_url);
|
||||
|
||||
// Try to construct thumbnail URL from filename and relative_path
|
||||
// An older release's name: the server answers it with the current preview
|
||||
// (any format the file owns, #317), or makes it on first view.
|
||||
// Thumbnails are stored as: thumbnails/{parent_dir}/{stem}_thumb{ext}
|
||||
// to avoid collisions with same-named files in different directories
|
||||
if (image.filename) {
|
||||
|
||||
+4
-2
@@ -488,7 +488,8 @@
|
||||
const index = from + offset;
|
||||
// Use thumbnail for display if available, fallback to original
|
||||
const displayUrl = image.thumbnail_url || image.url;
|
||||
const hasThumb = !!image.thumbnail_url;
|
||||
// The listing names a preview URL for every file (made on view, #249); the dot means the file exists.
|
||||
const hasThumb = typeof image.thumbnail_ready === 'boolean' ? image.thumbnail_ready : !!image.thumbnail_url;
|
||||
const isVideo = image.is_video || false;
|
||||
const mediaType = image.media_type || 'image';
|
||||
|
||||
@@ -551,7 +552,8 @@
|
||||
const index = from + offset;
|
||||
// Use thumbnail for display if available, fallback to original
|
||||
const displayUrl = image.thumbnail_url || image.url;
|
||||
const hasThumb = !!image.thumbnail_url;
|
||||
// The listing names a preview URL for every file (made on view, #249); the dot means the file exists.
|
||||
const hasThumb = typeof image.thumbnail_ready === 'boolean' ? image.thumbnail_ready : !!image.thumbnail_url;
|
||||
const isVideo = image.is_video || false;
|
||||
const mediaType = image.media_type || 'image';
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import { navigateTo } from './state.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { openCopyFallback } from './workspace-dialogs.mjs';
|
||||
import { displayPreferences } from './preferences.mjs';
|
||||
import { fallBackToOriginal, thumbnailSource } from './thumbnail-support.mjs';
|
||||
|
||||
const RAW_KEYS = ['prompt', 'workflow', 'parameters', 'comfyassets_prompt_pairs'];
|
||||
const RAW_LIMIT = 'Raw metadata exceeds the response limit. Download the original file to inspect it externally.';
|
||||
@@ -318,18 +319,29 @@ export function mountFileInspector(container, {api, sender, parentScope, variant
|
||||
});
|
||||
return button;
|
||||
}
|
||||
/** The concept's plain cover; the card image opens the viewer (D19). Video keeps its own element. */
|
||||
/**
|
||||
* The concept's plain cover; the card image opens the viewer (D19). #249: it is the file's preview (a video's
|
||||
* is a still frame) whenever the listing gave one; without one the original, and a video keeps its own element.
|
||||
* #317: a JPEG XL preview this browser cannot show is the original instead, and an image's preview that fails to
|
||||
* load falls back to the original. Only the cover is a preview: metadata and downloads use the original.
|
||||
*/
|
||||
function cover(record) {
|
||||
const {file} = record, video = file?.media_type === 'video';
|
||||
const node = el(video ? 'video' : 'img', undefined, 'detail-cover'); node.src = record.download.url;
|
||||
if (video) {
|
||||
node.muted = true; node.playsInline = true; node.preload = 'metadata';
|
||||
contentScope.cleanup(() => {node.pause(); node.removeAttribute('src'); node.load();});
|
||||
} else {
|
||||
node.alt = ''; node.decoding = 'async';
|
||||
if (file?.width && file?.height) {node.width = file.width; node.height = file.height;}
|
||||
contentScope.cleanup(() => node.removeAttribute('src'));
|
||||
}
|
||||
const {file} = record, isVideo = file?.media_type === 'video';
|
||||
const preview = record.preview?.url ? thumbnailSource({url: record.download.url, thumbnail_url: record.preview.url}) : null;
|
||||
if (isVideo && (!preview || preview === record.download.url)) return videoCover(record);
|
||||
const node = el('img', undefined, 'detail-cover'); node.src = preview || record.download.url;
|
||||
if (!isVideo) fallBackToOriginal(node, record.download.url, (...args) => contentScope.listen(...args));
|
||||
node.alt = ''; node.decoding = 'async';
|
||||
if (file?.width && file?.height) {node.width = file.width; node.height = file.height;}
|
||||
contentScope.cleanup(() => node.removeAttribute('src'));
|
||||
// #249: a video whose preview cannot be loaded swaps back to its own element.
|
||||
if (isVideo) contentScope.listen(node, 'error', () => { if (node.isConnected) node.replaceWith(videoCover(record)); }, {once:true});
|
||||
return node;
|
||||
}
|
||||
function videoCover(record) {
|
||||
const node = el('video', undefined, 'detail-cover'); node.src = record.download.url;
|
||||
node.muted = true; node.playsInline = true; node.preload = 'metadata';
|
||||
contentScope.cleanup(() => {node.pause(); node.removeAttribute('src'); node.load();});
|
||||
return node;
|
||||
}
|
||||
/** "File & provenance": the file facts, the D21 prompt evidence, the occurrence and the saved association (D66 for a local file). */
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import {projectInspectionPayload} from './inspection-state.mjs';
|
||||
import {jxlSupport} from './thumbnail-support.mjs';
|
||||
|
||||
const object = value => value !== null && typeof value === 'object' && !Array.isArray(value);
|
||||
const uuid = value => typeof value === 'string' && /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$/.test(value);
|
||||
@@ -64,7 +65,12 @@ export function toHistoryInspectionRecord(result, target, extractFacts) {
|
||||
const path = [target.reference.subfolder, target.reference.filename].filter(Boolean).join('/');
|
||||
const identity = {kind:'reported-location', key:target.location_key, path:target.reference.path_style === 'windows' ? path.replaceAll('\\', '/') : path,
|
||||
occurrence:{...target.occurrence}, history_epoch:target.history_epoch, current_scope:target.current_scope, reference:{...target.reference}};
|
||||
return projectInspectionPayload(result, identity, {url:urlFor(target, 'url', 'media'), filename:target.filename}, extractFacts);
|
||||
const record = projectInspectionPayload(result, identity, {url:urlFor(target, 'url', 'media'), filename:target.filename}, extractFacts);
|
||||
// #249: the inspector's cover is the recorded output's preview (made by the server on first view). #317: its URL
|
||||
// names no format, so where this browser decoded JPEG XL it says so (`jxl=1`); the server also reads Accept.
|
||||
const cover = urlFor(target, 'thumbnail_url', 'thumbnail');
|
||||
record.preview = {url:jxlSupport() === true ? `${cover}&jxl=1` : cover};
|
||||
return record;
|
||||
}
|
||||
export const reportedLocationAdapter = Object.freeze({
|
||||
validate,
|
||||
|
||||
@@ -9,6 +9,7 @@ import { cardTitle, cardSubtitle, imageKey, mediaRatio } from './images-state.mj
|
||||
import { createWorkspaceIcon } from './icons.mjs';
|
||||
import { createTagButton } from './image-tagging.mjs';
|
||||
import { paintStar } from './images-favorites.mjs';
|
||||
import { fallBackToOriginal, replaceOnError, showsThumbnail, thumbnailSource } from './thumbnail-support.mjs';
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
@@ -17,19 +18,31 @@ function el(tag, text, className) {
|
||||
return node;
|
||||
}
|
||||
|
||||
/** The card's media: a video's first frame when it has no thumbnail, else the thumbnail or the image. */
|
||||
/** A video's own element (its first frame), for a video without a usable preview. */
|
||||
function videoMedia(image) {
|
||||
const node = el('video');
|
||||
node.src = window.ApiClient.resolveUrl(image.url);
|
||||
node.muted = true; node.playsInline = true; node.preload = 'metadata';
|
||||
return node;
|
||||
}
|
||||
|
||||
/**
|
||||
* The card's media: the thumbnail (a video's is its preview frame), else the image, else a video's own element.
|
||||
* #249: a video whose preview cannot be loaded swaps back to its own element. #317: a JPEG XL thumbnail this browser
|
||||
* cannot show loads the original, and an image's thumbnail that fails to load falls back to the original.
|
||||
*/
|
||||
export function cardMedia(image, lazy = false) {
|
||||
const { resolveUrl } = window.ApiClient;
|
||||
const video = image.is_video && !image.thumbnail_url;
|
||||
const node = el(video ? 'video' : 'img');
|
||||
node.src = resolveUrl(image.thumbnail_url || image.url);
|
||||
if (video) { node.muted = true; node.playsInline = true; node.preload = 'metadata'; }
|
||||
else {
|
||||
// The concept card image: alt is the card title, with the listing's intrinsic size when known.
|
||||
node.alt = cardTitle(image); node.decoding = 'async'; if (lazy) node.loading = 'lazy';
|
||||
if ([image.width, image.height].every(value => Number.isFinite(value) && value > 0)) {
|
||||
node.width = image.width; node.height = image.height;
|
||||
}
|
||||
if (!showsThumbnail(image)) return videoMedia(image);
|
||||
const node = el('img');
|
||||
node.src = resolveUrl(thumbnailSource(image));
|
||||
// replaceWith does nothing for a card no longer in the page.
|
||||
if (image.is_video) replaceOnError(node, () => videoMedia(image));
|
||||
else fallBackToOriginal(node, resolveUrl(image.url));
|
||||
// The concept card image: alt is the card title, with the listing's intrinsic size when known.
|
||||
node.alt = cardTitle(image); node.decoding = 'async'; if (lazy) node.loading = 'lazy';
|
||||
if ([image.width, image.height].every(value => Number.isFinite(value) && value > 0)) {
|
||||
node.width = image.width; node.height = image.height;
|
||||
}
|
||||
return node;
|
||||
}
|
||||
|
||||
@@ -18,6 +18,8 @@ import { mountInspect } from './inspect.mjs';
|
||||
import { mountReview } from './review.mjs';
|
||||
import { createPromptSender } from './prompt-sender.mjs';
|
||||
import { mountSystem } from './system.mjs';
|
||||
import { mountLibraryBanner } from './library-location.mjs';
|
||||
import { detectJxlSupport } from './thumbnail-support.mjs';
|
||||
|
||||
const { createApiClient, resolveUrl } = window.ApiClient;
|
||||
|
||||
@@ -48,6 +50,8 @@ function mountWorkspace() {
|
||||
});
|
||||
// D4: the concept's Theme select; "System" keeps following ComfyUI, then the OS.
|
||||
const theme = mountWorkspaceTheme({ root: document.body, scope });
|
||||
// #317: learn once whether this browser shows JPEG XL thumbnails (cards fall back to originals otherwise).
|
||||
detectJxlSupport();
|
||||
const preference = document.getElementById('theme');
|
||||
preference.value = theme.preference;
|
||||
scope.listen(preference, 'change', () => theme.setPreference(preference.value));
|
||||
@@ -75,6 +79,8 @@ function mountWorkspace() {
|
||||
const activity = createActivityStore({ api, scope, connect: url => new EventSource(resolveUrl(url)) });
|
||||
mountActivityTopbar({ store: activity, scope });
|
||||
activity.load();
|
||||
// D129: a node-folder library is offered a move (or "Keep it here") above every page.
|
||||
mountLibraryBanner(document.getElementById('library-banner'), { api, activity, scope });
|
||||
const sidebar = document.querySelector('aside.sidebar');
|
||||
// The route key a nav link carries in `data-route`: a saved view's is `saved/<id>`.
|
||||
const routeKey = route => route.id === 'saved' ? `saved/${route.savedId}` : route.id;
|
||||
|
||||
@@ -44,6 +44,9 @@ export function inspectPath(image) {
|
||||
return `/prompt_manager/images/inspect/${image.path.split('/').map(encodeURIComponent).join('/')}?${query}`;
|
||||
}
|
||||
|
||||
/** One path segment as the server quotes it (Python `quote(..., safe="/")` also escapes `!'()*`, #249 M11). */
|
||||
const quoteSegment = segment => encodeURIComponent(segment).replace(/[!'()*]/g, c => `%${c.charCodeAt(0).toString(16).toUpperCase()}`);
|
||||
|
||||
function downloadFor(image, required) {
|
||||
try {
|
||||
if (typeof image.url !== 'string' || !image.url) throw new TypeError('Missing original media URL');
|
||||
@@ -55,7 +58,7 @@ function downloadFor(image, required) {
|
||||
const sameOrigin = 'Origin' in globalThis ?
|
||||
globalThis.Origin.from(url).isSameOrigin(globalThis.Origin.from(reference)) :
|
||||
url.origin !== 'null' && url.origin === reference.origin;
|
||||
const expected = new URL(resolveUrl(`/prompt_manager/images/serve/${image.path.split('/').map(encodeURIComponent).join('/')}`), reference);
|
||||
const expected = new URL(resolveUrl(`/prompt_manager/images/serve/${image.path.split('/').map(quoteSegment).join('/')}`), reference);
|
||||
const scope = descriptorScope(image);
|
||||
if (!sameOrigin || !['http:', 'https:'].includes(url.protocol) || url.username || url.password || url.hash ||
|
||||
url.pathname !== expected.pathname || url.searchParams.getAll('root').length !== 1 || url.searchParams.get('root') !== String(image.root_index) ||
|
||||
@@ -68,6 +71,48 @@ function downloadFor(image, required) {
|
||||
}
|
||||
}
|
||||
|
||||
// The server's video extensions (py/api/inspection_service.py VIDEO_EXTENSIONS).
|
||||
const VIDEO_EXTENSIONS = new Set(['.mp4', '.webm', '.avi', '.mov', '.mkv', '.m4v', '.wmv']);
|
||||
// #317: the formats a preview is written in (WebP, JPEG XL, JPEG), for images and videos alike.
|
||||
const WRITTEN_PREVIEW_EXTENSIONS = ['.webp', '.jxl', '.jpg'];
|
||||
|
||||
/**
|
||||
* Every `thumbnails/<path>_thumb<ext>` the server may list as this file's preview (py/api/thumbnail_settings.py
|
||||
* `owns`): a written format, or the name an older release gave it (the source's own extension, `.jpg` for a video).
|
||||
*/
|
||||
function previewPaths(path) {
|
||||
const slash = path.lastIndexOf('/'), dot = path.lastIndexOf('.');
|
||||
const cut = dot > slash ? dot : path.length, spelled = path.slice(cut), extension = spelled.toLowerCase();
|
||||
const stem = `thumbnails/${path.slice(0, cut)}_thumb`;
|
||||
const legacy = VIDEO_EXTENSIONS.has(extension) ? ['.jpg'] : [extension, spelled];
|
||||
return new Set([...WRITTEN_PREVIEW_EXTENSIONS, ...legacy].filter(Boolean).map(ext => stem + ext));
|
||||
}
|
||||
|
||||
/**
|
||||
* #249: the listing's thumbnail of `image` as `{url}` for the inspector's cover, or null. Only the same-origin
|
||||
* serve URL of this file's own preview (any format it owns, #317) under the same root and source scope is used;
|
||||
* anything else keeps the original (a missing preview is made by the server on first view). The cover is only a
|
||||
* placeholder: metadata, downloads and the viewer always use the original.
|
||||
*/
|
||||
function previewFor(image) {
|
||||
try {
|
||||
if (typeof image.thumbnail_url !== 'string' || !image.thumbnail_url) return null;
|
||||
const location = globalThis.location || globalThis.window?.location;
|
||||
const reference = new URL(location?.href || 'http://localhost/');
|
||||
const resolveUrl = globalThis.window?.ApiClient?.resolveUrl || globalThis.ApiClient?.resolveUrl || (url => url);
|
||||
const value = resolveUrl(image.thumbnail_url), url = new URL(value, reference);
|
||||
const expected = new Set([...previewPaths(image.path)].map(preview =>
|
||||
new URL(resolveUrl(`/prompt_manager/images/serve/${preview.split('/').map(quoteSegment).join('/')}`), reference).pathname));
|
||||
const scope = descriptorScope(image);
|
||||
if (url.origin !== reference.origin || url.username || url.password || url.hash || !expected.has(url.pathname) ||
|
||||
url.searchParams.getAll('root').length !== 1 || url.searchParams.get('root') !== String(image.root_index) ||
|
||||
url.searchParams.getAll('root_scope').length > 1 || (scope !== null && url.searchParams.get('root_scope') !== scope)) return null;
|
||||
return { url: value };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function originalMediaDownload(image) {
|
||||
validateIdentity(image);
|
||||
return downloadFor(image, true);
|
||||
@@ -284,16 +329,18 @@ export function toInspectionRecord(result, image, extractFacts = globalThis.Comf
|
||||
if (!legacy && result.inspection_version !== 1) throw new TypeError('Unsupported inspection version');
|
||||
if (!legacy && result.success !== true) throw new TypeError('Invalid inspection success status');
|
||||
if ((!legacy || result.image !== undefined) && (!object(result.image) || result.image.root_index !== image.root_index || result.image.path !== image.path || result.image.filename !== image.path.split('/').at(-1))) throw new TypeError('Mismatched inspection identity');
|
||||
return projectInspectionPayload(result,
|
||||
const record = projectInspectionPayload(result,
|
||||
{ kind: 'rooted-file', key: imageKey(image), root_index: image.root_index, path: image.path, root_scope: scope },
|
||||
downloadFor(image, !legacy || scope !== null), extractFacts, legacy, parser);
|
||||
record.preview = record.download ? previewFor(image) : null;
|
||||
return record;
|
||||
}
|
||||
|
||||
/** Shared bounded payload projection; adapters validate identity before entering here. */
|
||||
export function projectInspectionPayload(result, identity, download, extractFacts = globalThis.ComfyMetadata?.extractGenerationFacts || globalThis.window?.ComfyMetadata?.extractGenerationFacts, legacy = false, parser = comfyMetadata()) {
|
||||
const record = {
|
||||
identity, file: null, prompts: promptsFor(result.prompts), library: null, metadata: null,
|
||||
generation: unavailableFacts(), historyTarget: null, download, imageTags: null,
|
||||
generation: unavailableFacts(), historyTarget: null, download, imageTags: null, preview: null,
|
||||
};
|
||||
if (legacy) return record;
|
||||
record.imageTags = imageTagsFor(result.image_tags);
|
||||
|
||||
@@ -0,0 +1,255 @@
|
||||
/**
|
||||
* Where the library lives (#249 System S6b, D129): the workspace banner, System › Overview's "Library location" row,
|
||||
* the Move library review and its start.
|
||||
*
|
||||
* - The server says where the database is and why (`GET /prompt_manager/system/library` and the Overview's
|
||||
* `location`): `default` (ComfyUI's user folder), `environment` (PROMPT_MANAGER_DATA_DIR), `config`
|
||||
* (config.json's database path) or `node_folder`, where an update or reinstall can delete it.
|
||||
* - D129: while a node-folder library is not kept there, the shell shows the concept's `.notice` (`h2` + `p` +
|
||||
* `.row-actions`) above every page: "Keep it here" saves the choice on the server (the banner stops; the Overview row
|
||||
* stays), "Move library" opens the review. The concept has no such notice; its markup and classes are the Sources
|
||||
* notice's (app.js 680).
|
||||
* - The review is the shared record-changing review (`openReviewDialog`): From, To, Size, what moves with it and what
|
||||
* happens to the old files. Move library starts the server's exclusive "Move library" job, tracks it in Activity and
|
||||
* opens its view; a refusal (another job running, no room, a library already there) says why and nothing starts.
|
||||
* - When a Move library job ends, the banner and the Overview read the location again.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { busyToast } from './activity-scan.mjs';
|
||||
import { closeWorkspaceDialog, openContentDialog, openReviewDialog } from './workspace-dialogs.mjs';
|
||||
import { navigateTo } from './state.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
|
||||
export const LIBRARY_URL = '/prompt_manager/system/library';
|
||||
export const MOVE_URL = `${LIBRARY_URL}/move`;
|
||||
export const KEEP_URL = `${LIBRARY_URL}/keep`;
|
||||
export const MOVE_KIND = 'library_move';
|
||||
|
||||
export const BANNER_TITLE = 'Your library is inside the node folder';
|
||||
const BANNER_LEAD = 'Your library is stored inside the node folder, where an update or reinstall can delete it.';
|
||||
/** Where Move library puts it, by the server's `target_source`. */
|
||||
export const TARGET_TEXT = Object.freeze({
|
||||
default: 'ComfyUI\'s user folder',
|
||||
environment: 'the folder PROMPT_MANAGER_DATA_DIR names',
|
||||
});
|
||||
/** The notice's sentence: it names the actual target. */
|
||||
export function bannerText(location) {
|
||||
const target = TARGET_TEXT[location?.target_source] ?? TARGET_TEXT.default;
|
||||
return `${BANNER_LEAD} Move it to ${target}: it is copied and verified first, and nothing is deleted.`;
|
||||
}
|
||||
export const BANNER_TEXT = bannerText({ target_source: 'default' });
|
||||
export const KEEP_LABEL = 'Keep it here';
|
||||
export const MOVE_LABEL = 'Move library';
|
||||
export const ROW_TITLE = 'Library location';
|
||||
export const START_FAILED = 'Move library could not start. Try again.';
|
||||
export const KEEP_FAILED = 'Your choice could not be saved. Try again.';
|
||||
|
||||
/** Why the library is where it is, by `source`. */
|
||||
export const SOURCE_TEXT = Object.freeze({
|
||||
default: 'ComfyUI\'s user folder, which updates and reinstalls of the node leave alone',
|
||||
environment: 'The folder PROMPT_MANAGER_DATA_DIR names',
|
||||
config: 'The database path set in config.json',
|
||||
node_folder: 'The node folder, where an update or reinstall can delete it',
|
||||
});
|
||||
|
||||
/** A `problem` code as a sentence. */
|
||||
export const PROBLEM_TEXT = Object.freeze({
|
||||
'data-dir-not-absolute': 'PROMPT_MANAGER_DATA_DIR is not an absolute folder, so it was ignored.',
|
||||
'data-dir-served': 'PROMPT_MANAGER_DATA_DIR names a folder whose files are served over the web, so it was ignored.',
|
||||
'data-dir-empty': 'PROMPT_MANAGER_DATA_DIR has no library yet; Move library puts this one there.',
|
||||
'move-unfinished': 'An earlier move did not finish; this library is still the one in use.',
|
||||
'inside-node-folder': 'It is inside the node folder, where an update or reinstall deletes it, config.json and its '
|
||||
+ 'CivitAI key included. Set database.default_path or PROMPT_MANAGER_DATA_DIR to a folder outside it.',
|
||||
});
|
||||
|
||||
const number = value => Number(value).toLocaleString('en-US');
|
||||
|
||||
/** "1.6 GB" or "120.0 MB" or "512 KB". */
|
||||
export function sizeText(bytes) {
|
||||
if (!Number.isFinite(bytes) || bytes < 0) return 'Unknown';
|
||||
if (bytes >= 2 ** 30) return `${(bytes / 2 ** 30).toLocaleString('en-US', { maximumFractionDigits: 1, minimumFractionDigits: 1 })} GB`;
|
||||
if (bytes >= 2 ** 20) return `${(bytes / 2 ** 20).toLocaleString('en-US', { maximumFractionDigits: 1, minimumFractionDigits: 1 })} MB`;
|
||||
return `${number(Math.ceil(bytes / 1024))} KB`;
|
||||
}
|
||||
|
||||
/** Whether `location` is a well-formed server answer. */
|
||||
export function isLocation(location) {
|
||||
return Boolean(location) && typeof location === 'object' && typeof location.path === 'string'
|
||||
&& Object.hasOwn(SOURCE_TEXT, location.source) && typeof location.movable === 'boolean';
|
||||
}
|
||||
|
||||
/** The Overview row's description: the path, why it is there, the user's choice and any problem. */
|
||||
export function locationDescription(location) {
|
||||
const parts = [`${location.path} · ${SOURCE_TEXT[location.source]}.`];
|
||||
if (location.movable && location.keep) parts.push('You chose to keep it here.');
|
||||
const problem = PROBLEM_TEXT[location.problem];
|
||||
if (problem) parts.push(problem);
|
||||
return parts.join(' ');
|
||||
}
|
||||
|
||||
/** The review's `[term, value]` rows. */
|
||||
export function reviewGrid(location) {
|
||||
return [
|
||||
['From', location.path],
|
||||
['To', `${location.target}/prompts.db`],
|
||||
['Size', sizeText(location.size_bytes)],
|
||||
['Moves with it', location.config_moves === false
|
||||
? 'The database backups (config.json stays where PROMPT_MANAGER_CONFIG_PATH names it)'
|
||||
: 'config.json and the database backups'],
|
||||
['Old files', 'Renamed *.moved-to-…, never deleted'],
|
||||
];
|
||||
}
|
||||
|
||||
export const REVIEW_SENTENCE = 'The copy is verified before Prompt Manager switches to it. No other job can run until it '
|
||||
+ 'finishes, and closing this page does not stop it.';
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
if (text !== undefined) node.textContent = text;
|
||||
if (className) node.className = className;
|
||||
return node;
|
||||
}
|
||||
|
||||
/**
|
||||
* Start the Move library job, follow it in Activity and open its view. A busy server toasts the D57 link to the running
|
||||
* job; any other refusal says the server's reason. Nothing is said once `scope` ended.
|
||||
*/
|
||||
export async function startLibraryMove({ api, activity, scope, notify = showWorkspaceToast, navigate = navigateTo }) {
|
||||
try {
|
||||
const { job } = await api.post(MOVE_URL, {}, { signal: scope.signal });
|
||||
if (scope.signal.aborted) return null;
|
||||
const tracked = activity.track(job);
|
||||
navigate(`#/activity/${tracked.id}`);
|
||||
return tracked;
|
||||
} catch (error) {
|
||||
if (scope.signal.aborted) return null;
|
||||
if (error?.status === 409 && error?.code === 'busy') {
|
||||
await activity.load();
|
||||
if (!scope.signal.aborted) notify(busyToast(activity.running()[0] ?? null));
|
||||
return null;
|
||||
}
|
||||
notify(typeof error?.code === 'string' && error.message ? error.message : START_FAILED);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** The Move library review over `location`; Move library starts the job. */
|
||||
export function openMoveReview({ api, activity, scope, location, trigger = null, notify = showWorkspaceToast,
|
||||
navigate = navigateTo }) {
|
||||
return openReviewDialog({ scope, trigger, title: 'Move library', grid: reviewGrid(location), sentence: REVIEW_SENTENCE,
|
||||
cancelLabel: 'Cancel', primary: { label: MOVE_LABEL },
|
||||
onStart: () => { startLibraryMove({ api, activity, scope, notify, navigate }); } });
|
||||
}
|
||||
|
||||
/** Save "Keep it here" (`keep` true) or take it back; resolves the new location, or null when it was not saved. */
|
||||
export async function keepLibrary({ api, scope, keep = true, notify = showWorkspaceToast }) {
|
||||
try {
|
||||
const { location } = await api.post(KEEP_URL, { keep }, { signal: scope.signal });
|
||||
return isLocation(location) ? location : null;
|
||||
} catch (error) {
|
||||
if (!scope.signal.aborted) notify(KEEP_FAILED);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** The details of a library that does not move: where it is, why, and what lives beside it. */
|
||||
export function openLocationDetails({ scope, location, trigger = null }) {
|
||||
const list = el('dl', undefined, 'meta-grid');
|
||||
for (const [term, value] of [['Database', location.path], ['Why here', SOURCE_TEXT[location.source]],
|
||||
['Beside it', 'config.json, the database backups and the logs'], ['Size', sizeText(location.size_bytes)]]) {
|
||||
const entry = el('div');
|
||||
entry.append(el('dt', term), el('dd', value));
|
||||
list.append(entry);
|
||||
}
|
||||
const close = createButton('Close');
|
||||
scope.listen(close, 'click', () => closeWorkspaceDialog());
|
||||
const actions = el('div', undefined, 'dialog-actions');
|
||||
actions.append(close);
|
||||
const nodes = [list];
|
||||
const problem = PROBLEM_TEXT[location.problem];
|
||||
if (problem) nodes.push(el('p', problem, 'small'));
|
||||
nodes.push(actions);
|
||||
openContentDialog({ scope, title: ROW_TITLE, trigger, nodes });
|
||||
}
|
||||
|
||||
/** D129: the Overview's "Library location" `.operation` row; Move library when it can move, else Details. */
|
||||
export function locationRow(location, { api, activity, scope, notify = showWorkspaceToast, navigate = navigateTo }) {
|
||||
const row = el('div', undefined, 'operation');
|
||||
const text = el('div');
|
||||
text.append(el('h3', ROW_TITLE), el('p', locationDescription(location)));
|
||||
const actions = el('div', undefined, 'row-actions');
|
||||
const action = createButton(location.movable ? MOVE_LABEL : 'Details');
|
||||
scope.listen(action, 'click', () => {
|
||||
if (location.movable) openMoveReview({ api, activity, scope, location, trigger: action, notify, navigate });
|
||||
else openLocationDetails({ scope, location, trigger: action });
|
||||
});
|
||||
actions.append(action);
|
||||
row.append(text, actions);
|
||||
return row;
|
||||
}
|
||||
|
||||
/** D129: the concept's `.notice` for a node-folder library: Keep it here and Move library. */
|
||||
export function libraryNotice({ scope, location = null, onKeep, onMove }) {
|
||||
const notice = el('div', undefined, 'notice');
|
||||
const text = el('div');
|
||||
text.append(el('h2', BANNER_TITLE), el('p', bannerText(location)));
|
||||
const keep = createButton(KEEP_LABEL);
|
||||
const move = createButton(MOVE_LABEL, 'ca-primary');
|
||||
scope.listen(keep, 'click', () => onKeep(keep));
|
||||
scope.listen(move, 'click', () => onMove(move));
|
||||
const actions = el('div', undefined, 'row-actions');
|
||||
actions.append(keep, move);
|
||||
notice.append(text, actions);
|
||||
return notice;
|
||||
}
|
||||
|
||||
/** Calls `onEnded()` whenever a Move library job this page sees stops running. Returns the unsubscribe. */
|
||||
export function onMoveEnded(activity, onEnded) {
|
||||
const running = new Set();
|
||||
return activity.subscribe(() => {
|
||||
let ended = false;
|
||||
for (const job of activity.jobs()) {
|
||||
if (job.kind !== MOVE_KIND) continue;
|
||||
if (job.state === 'running') running.add(job.id);
|
||||
else if (running.delete(job.id)) ended = true;
|
||||
}
|
||||
if (ended) onEnded();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Mount the banner in `container` (hidden until the server offers the move). It reads the location once, again when a
|
||||
* Move library job ends, and hides for good this page once the user keeps the library where it is.
|
||||
*/
|
||||
export function mountLibraryBanner(container, { api, activity, scope, notify = showWorkspaceToast, navigate = navigateTo,
|
||||
latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) {
|
||||
const read = latestOnly({ lifetime: scope.signal });
|
||||
const show = location => {
|
||||
container.replaceChildren();
|
||||
container.hidden = !(isLocation(location) && location.offer === true);
|
||||
if (container.hidden) return;
|
||||
container.append(libraryNotice({ scope, location,
|
||||
onKeep: async button => {
|
||||
button.setAttribute('aria-disabled', 'true');
|
||||
const saved = await keepLibrary({ api, scope, notify });
|
||||
if (scope.signal.aborted) return;
|
||||
button.removeAttribute('aria-disabled');
|
||||
if (saved) show(saved);
|
||||
},
|
||||
onMove: button => openMoveReview({ api, activity, scope, location, trigger: button, notify, navigate }) }));
|
||||
};
|
||||
async function load() {
|
||||
try {
|
||||
const answer = await read(signal => api.get(LIBRARY_URL, { signal, cache: 'no-store' }));
|
||||
if (answer === STALE || scope.signal.aborted) return;
|
||||
show(answer.location);
|
||||
} catch {
|
||||
// The banner is advisory: an unreadable location shows nothing (the Overview row reports it).
|
||||
if (!scope.signal.aborted) show(null);
|
||||
}
|
||||
}
|
||||
container.hidden = true;
|
||||
scope.cleanup(onMoveEnded(activity, load));
|
||||
load();
|
||||
return { reload: load };
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import {pageHeading} from './heading.mjs';
|
||||
import {mountToolbar} from './toolbar.mjs';
|
||||
import {libraryFilters,libraryViewState} from './saved-views.mjs';
|
||||
import {openRefineDialog,openContentDialog} from './workspace-dialogs.mjs';
|
||||
import {fallBackToOriginal,replaceOnError,showsThumbnail,thumbnailSource} from './thumbnail-support.mjs';
|
||||
|
||||
let shelfVisit=null;
|
||||
// Every confirmed receipt advances this; a shelf mounted before it never restores its cached stream.
|
||||
@@ -213,9 +214,12 @@ export function mountLibrary(main,api,parentScope,{positiveId,savedState=null,on
|
||||
const art=el('div',undefined,'card-art');art.setAttribute('aria-hidden','true');
|
||||
const previews=item.previews || [];
|
||||
for(const image of previews.slice(0,2)) {
|
||||
if(image.is_video && !image.thumbnail_url) art.append(el('span','Video preview','small'));
|
||||
// #317: a video with no preview, or one this browser cannot show (JPEG XL), keeps its label.
|
||||
if(!showsThumbnail(image)) art.append(el('span','Video preview','small'));
|
||||
else {
|
||||
const img=el('img');img.src=window.ApiClient.resolveUrl(image.thumbnail_url || image.url);img.alt='';img.loading='lazy';img.decoding='async';img.width=image.width || 800;img.height=image.height || 600;art.append(img);
|
||||
// #317: an image's JPEG XL preview this browser cannot show, or one that fails, loads the original;
|
||||
// a video's preview that fails becomes its label, never a broken image.
|
||||
const img=el('img');img.src=window.ApiClient.resolveUrl(thumbnailSource(image));if(image.is_video) replaceOnError(img,()=>el('span','Video preview','small'));else fallBackToOriginal(img,window.ApiClient.resolveUrl(image.url));img.alt='';img.loading='lazy';img.decoding='async';img.width=image.width || 800;img.height=image.height || 600;art.append(img);
|
||||
}
|
||||
}
|
||||
if(!previews.length) art.append(el('span','No available image previews','small'));
|
||||
|
||||
@@ -13,10 +13,14 @@
|
||||
* (`showImageInfo`), "Collapse viewer details by default" (`sidebarCollapsedByDefault`) and the operation "Reset
|
||||
* display defaults", which drafts the gallery's defaults for these browser rows and touches no other key.
|
||||
*
|
||||
* - #317: a second surface, "Thumbnails", holds the server's thumbnail format, quality and size limit
|
||||
* (system-thumbnails.mjs).
|
||||
*
|
||||
* Controls follow the store: a value read later (the host settings, another tab's change) repaints them. A host row
|
||||
* whose value could not be read stays disabled and says so ("Server settings could not be read.").
|
||||
*/
|
||||
import { settingRow, operationRow } from './system-controls.mjs';
|
||||
import { mountThumbnailSettings } from './system-thumbnails.mjs';
|
||||
import { preferenceDefaults } from './preferences.mjs';
|
||||
|
||||
export const RESULT_TIMEOUT_MAX = 300;
|
||||
@@ -98,8 +102,11 @@ function paint(input, control, value) {
|
||||
else if (document.activeElement !== input || control.type === 'select') input.value = String(value);
|
||||
}
|
||||
|
||||
/** Mount Appearance in `container` (System's `.settings-layout`); returns the section's nodes. */
|
||||
export function mountAppearance(container, { scope, drafts }) {
|
||||
/**
|
||||
* Mount Appearance in `container` (System's `.settings-layout`); returns the section's nodes: "Browsing & display",
|
||||
* then the server's thumbnail settings (#317, system-thumbnails.mjs).
|
||||
*/
|
||||
export function mountAppearance(container, { scope, drafts, api, activity }) {
|
||||
const surface = el('section', undefined, 'surface');
|
||||
surface.append(el('h2', 'Browsing & display'), el('p', 'One set of defaults; toolbar changes apply to the current view.'));
|
||||
const controls = [];
|
||||
@@ -139,5 +146,5 @@ export function mountAppearance(container, { scope, drafts }) {
|
||||
};
|
||||
scope.cleanup(drafts.subscribe(repaint));
|
||||
repaint();
|
||||
return [surface];
|
||||
return [surface, mountThumbnailSettings(container, { scope, drafts, api, activity })];
|
||||
}
|
||||
|
||||
@@ -8,6 +8,8 @@
|
||||
* reads 0 before the read answers: a load shows a status line, a failure an alert and Retry. The thumbnail row is
|
||||
* its own read of `/images/thumbnails/status` (a walk of every source): "Checking thumbnails…" until it answers.
|
||||
* D99: its Review thumbnails opens the same "Thumbnail maintenance" dialog as Sources (thumbnail-review.mjs).
|
||||
* #317: it reads "N missing · M outdated · K failed"; while previews are outdated (another format than the
|
||||
* configured one) a second button, "Convert outdated", opens that dialog on its Convert outdated scope (D131).
|
||||
* - D96: the Statistics dialog lists the real runs and average rating per category, then "Library totals".
|
||||
* - D95: the "Health report" lists every check the server ran, Missing files per image source, with a chip per real
|
||||
* result and when it was checked. The check runs from Maintenance's notice ("Run health checks", D123) as a
|
||||
@@ -22,6 +24,7 @@ import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { operationRow } from './system-controls.mjs';
|
||||
import { openThumbnailReview } from './thumbnail-review.mjs';
|
||||
import { isLocation, locationRow, onMoveEnded } from './library-location.mjs';
|
||||
|
||||
export const OVERVIEW_URL = '/prompt_manager/system/overview';
|
||||
export const STATISTICS_URL = '/prompt_manager/system/statistics';
|
||||
@@ -87,8 +90,29 @@ export function thumbnailTitle(status, sources) {
|
||||
if (status === undefined) return THUMBNAILS_LOADING;
|
||||
const missing = status?.pending;
|
||||
if (!Number.isSafeInteger(missing) || sources?.effective === 0) return 'Thumbnail status unavailable';
|
||||
if (missing === 0) return 'No missing thumbnails';
|
||||
return `${number(missing)} missing ${noun(missing, 'thumbnail')}`;
|
||||
const parts = [missing === 0 ? 'No missing thumbnails' : `${number(missing)} missing ${noun(missing, 'thumbnail')}`];
|
||||
// #317: previews in another format than the configured one (Convert outdated replaces them).
|
||||
if (convertOffered(status)) parts.push(`${number(status.outdated)} outdated`);
|
||||
// #249: files whose automatic preview could not be made this session (they show the original).
|
||||
if (Number.isSafeInteger(status.failed) && status.failed > 0) parts.push(`${number(status.failed)} failed`);
|
||||
return parts.join(' · ');
|
||||
}
|
||||
|
||||
/** #317: whether the status counts previews in an older format (the row then offers Convert outdated). */
|
||||
export function convertOffered(status) {
|
||||
return Number.isSafeInteger(status?.outdated) && status.outdated > 0;
|
||||
}
|
||||
|
||||
/** #317: the row's "Convert outdated" button, present only while previews are outdated (D131). */
|
||||
function paintConvert(row, status, { scope, openConvert }) {
|
||||
const actions = row.querySelector('.row-actions');
|
||||
let button = actions.querySelector('[data-convert]');
|
||||
if (!convertOffered(status)) { button?.remove(); return; }
|
||||
if (button) return;
|
||||
button = createButton('Convert outdated');
|
||||
button.dataset.convert = '';
|
||||
scope.listen(button, 'click', () => openConvert(button));
|
||||
actions.append(button);
|
||||
}
|
||||
|
||||
const SOURCES_SHARED = 'Monitoring and enrichment share one source list.';
|
||||
@@ -319,7 +343,7 @@ function surface(heading, rows) {
|
||||
}
|
||||
|
||||
/** The Overview's nodes, and its thumbnail row (painted by its own read). */
|
||||
function overviewNodes(overview, { scope, navigate, openStatistics, openThumbnails }) {
|
||||
function overviewNodes(overview, { api, activity, notify, scope, navigate, openStatistics, openThumbnails }) {
|
||||
const go = section => () => navigate(`#/system/${section}`);
|
||||
const review = operationRow({ scope, title: ambiguousTitle(overview.needs_review),
|
||||
description: Number.isSafeInteger(overview.needs_review?.pending)
|
||||
@@ -332,6 +356,9 @@ function overviewNodes(overview, { scope, navigate, openStatistics, openThumbnai
|
||||
review,
|
||||
operationRow({ scope, title: 'Backup status', description: backupText(overview.backup), label: 'View backups',
|
||||
onClick: go('backups') }),
|
||||
// D129: where the library lives and why; Move library when it is in the node folder.
|
||||
...(isLocation(overview.location)
|
||||
? [locationRow(overview.location, { api, activity, scope, notify, navigate })] : []),
|
||||
// D99: Review thumbnails opens the thumbnail dialog (the concept's row starts a sample job directly).
|
||||
thumbnails,
|
||||
]);
|
||||
@@ -362,8 +389,11 @@ export function mountOverview(container, { api, activity, scope, navigate = navi
|
||||
let thumbnailStatus;
|
||||
let thumbnailRow = null;
|
||||
let sources = null;
|
||||
const openConvert = trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate, mode: 'convert' });
|
||||
const paintThumbnails = () => {
|
||||
if (thumbnailRow) thumbnailRow.querySelector('h3').textContent = thumbnailTitle(thumbnailStatus, sources);
|
||||
if (!thumbnailRow) return;
|
||||
thumbnailRow.querySelector('h3').textContent = thumbnailTitle(thumbnailStatus, sources);
|
||||
paintConvert(thumbnailRow, thumbnailStatus, { scope, openConvert });
|
||||
};
|
||||
|
||||
function swap(nodes) {
|
||||
@@ -394,7 +424,7 @@ export function mountOverview(container, { api, activity, scope, navigate = navi
|
||||
try {
|
||||
const overview = await overviewRead(signal => api.get(OVERVIEW_URL, { signal, cache: 'no-store' }));
|
||||
if (overview === STALE || scope.signal.aborted) return;
|
||||
const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics,
|
||||
const { nodes, thumbnails } = overviewNodes(overview, { api, activity, notify, scope, navigate, openStatistics,
|
||||
openThumbnails: trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate }) });
|
||||
thumbnailRow = thumbnails;
|
||||
sources = overview.sources;
|
||||
@@ -427,6 +457,8 @@ export function mountOverview(container, { api, activity, scope, navigate = navi
|
||||
paintThumbnails();
|
||||
}
|
||||
const first = swap(loading());
|
||||
// D129: a finished Move library changes where the library lives.
|
||||
scope.cleanup(onMoveEnded(activity, load));
|
||||
load();
|
||||
loadThumbnails();
|
||||
return first;
|
||||
|
||||
@@ -18,7 +18,9 @@
|
||||
* starts the thumbnail Activity job.
|
||||
*
|
||||
* D98: per-source Remove; an added source reads "not saved yet" until Save; the line under the list says when
|
||||
* ComfyUI's output folder is used instead; "restart pending" for a source the running monitor does not watch yet; the
|
||||
* ComfyUI's output folder is used instead of configured sources or after an unsaved removal of the last one; with no
|
||||
* source configured, ComfyUI's output folder is the row "ComfyUI/output · watching · default" (`in_comfyui`: its public
|
||||
* path is relative to the ComfyUI base) and its scan keeps the server's public path; "restart pending" for a source the running monitor does not watch yet; the
|
||||
* "Refresh files" operation (`/gallery/rescan`) closes the surface. D127: the add dialog's field is "Folder path",
|
||||
* starts empty and says the server's reason in place; its action is "Add", and Enter adds too.
|
||||
* D128: "watching until restart" for a source the running monitor still watches after monitoring was switched off;
|
||||
@@ -46,11 +48,13 @@ export const WATCH_COPY = Object.freeze({ watching: 'watching', stopping: 'watch
|
||||
// The states whose observer runs now: their scan only has to catch up on changes.
|
||||
const WATCHED = new Set(['watching', 'stopping']);
|
||||
const NOT_SAVED = 'not saved yet';
|
||||
// An auto-detected source (ComfyUI's output folder) says it is the default on its own row.
|
||||
const DEFAULT_SOURCE = 'default';
|
||||
const COMFYUI_PREFIX = 'ComfyUI/';
|
||||
export const DUPLICATE_SOURCE = 'same folder as another source';
|
||||
/** D98: the line under the source list when it does not say everything by itself. */
|
||||
export const NOTES = Object.freeze({
|
||||
removedAll: 'No source is configured. After saving, ComfyUI’s output folder is used.',
|
||||
autoDetected: 'No source is configured; ComfyUI’s output folder is used.',
|
||||
fallback: 'No configured source is available; ComfyUI’s output folder is used.',
|
||||
none: 'No source is available.',
|
||||
});
|
||||
@@ -95,8 +99,11 @@ const folderName = path => path.split('/').filter(Boolean).at(-1) ?? path;
|
||||
|
||||
function effectiveRow(source, remove) {
|
||||
const watched = WATCHED.has(source.watch);
|
||||
return { key: remove ?? `auto:${source.root_index}`, title: source.label,
|
||||
description: `${source.path} · ${WATCH_COPY[source.watch] ?? WATCH_COPY.manual}`,
|
||||
const auto = remove === null;
|
||||
// The default row reads from ComfyUI's own folder ("ComfyUI/output") when its public path is relative to the base.
|
||||
const shown = auto && source.in_comfyui === true ? `${COMFYUI_PREFIX}${source.path}` : source.path;
|
||||
const parts = [shown, WATCH_COPY[source.watch] ?? WATCH_COPY.manual, ...(auto ? [DEFAULT_SOURCE] : [])];
|
||||
return { key: remove ?? `auto:${source.root_index}`, title: source.label, description: parts.join(' · '),
|
||||
scan: { label: watched ? 'Scan changes' : 'Scan folder', root: source.root_index, path: source.path }, remove };
|
||||
}
|
||||
|
||||
@@ -124,7 +131,7 @@ export function sourceRows({ sources, saved, current, labels = addedLabels }) {
|
||||
if (dirty) return { rows, note: roots.length ? null : NOTES.removedAll };
|
||||
if (sources.auto_detected) {
|
||||
rows.push(...effective.map(source => effectiveRow(source, null)));
|
||||
return { rows, note: configured.length ? NOTES.fallback : NOTES.autoDetected };
|
||||
return { rows, note: configured.length ? NOTES.fallback : null };
|
||||
}
|
||||
return { rows, note: rows.length ? null : NOTES.none };
|
||||
}
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* System › Appearance › Thumbnails (#317): the concept README keeps "Thumbnail quality, size limit" as Appearance
|
||||
* settings; they are the server's (`thumbnail_format`, `thumbnail_quality`, `thumbnail_edge`, saved with the settings
|
||||
* revision through the one Save row), so every preview writer follows them. One `section.surface` of the concept's
|
||||
* `.setting-row`s (system-controls.mjs), after "Browsing & display":
|
||||
*
|
||||
* - "Thumbnail format": WebP (default), JPEG XL, JPEG. D130: JPEG XL is not in the concept; it is offered only when
|
||||
* the server found an encoder (`thumbnail_encoders.jxl`), else the option is disabled and the row says why, in the
|
||||
* server's words. A row note also says when this browser cannot show JPEG XL thumbnails (thumbnail-support.mjs):
|
||||
* such a browser shows the originals instead.
|
||||
* - "Thumbnail quality" (1–100) and "Thumbnail size limit" (128–2048 px); a value outside is refused like Appearance's
|
||||
* timeout (no draft, the control highlighted).
|
||||
* - "Convert outdated thumbnails" opens Thumbnail maintenance on its Convert outdated scope; converting is never
|
||||
* automatic. After a save that changed the format, its line says existing previews keep their old format until
|
||||
* converted.
|
||||
*
|
||||
* Host rows stay disabled until the server settings were read, like Appearance's, and say "Server settings could not
|
||||
* be read." while a read failed.
|
||||
*/
|
||||
import { settingRow, operationRow } from './system-controls.mjs';
|
||||
import { detectJxlSupport } from './thumbnail-support.mjs';
|
||||
import { openThumbnailReview } from './thumbnail-review.mjs';
|
||||
import { HOST_UNAVAILABLE } from './system-appearance.mjs';
|
||||
|
||||
export const FORMAT_OPTIONS = Object.freeze([['webp', 'WebP'], ['jxl', 'JPEG XL'], ['jpeg', 'JPEG']]);
|
||||
export const QUALITY_LIMIT = Object.freeze({ min: 1, max: 100 });
|
||||
export const EDGE_LIMIT = Object.freeze({ min: 128, max: 2048 });
|
||||
export const BROWSER_NO_JXL = 'This browser can’t show JPEG XL thumbnails; it shows the originals instead.';
|
||||
export const CONVERT_DESCRIPTION = 'Replace previews in an older format from their originals, then remove the old files.';
|
||||
export const FORMAT_SAVED = 'Format saved. Existing previews keep their old format until you convert them.';
|
||||
|
||||
/** A whole number within `{min, max}` from a field's text, else null. */
|
||||
export function wholeNumber(text, { min, max }) {
|
||||
const value = String(text).trim();
|
||||
if (!/^\d+$/.test(value)) return null;
|
||||
const number = Number(value);
|
||||
return number >= min && number <= max ? number : null;
|
||||
}
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
if (text !== undefined) node.textContent = text;
|
||||
if (className) node.className = className;
|
||||
return node;
|
||||
}
|
||||
|
||||
const ROWS = Object.freeze([
|
||||
['thumbnail-format', 'Thumbnail format', 'WebP is small and fast everywhere. JPEG XL is smaller still where your browser shows it.',
|
||||
'thumbnail_format', { type: 'select', options: FORMAT_OPTIONS }],
|
||||
['thumbnail-quality', 'Thumbnail quality', 'From 1 to 100. 80 keeps previews sharp and small.', 'thumbnail_quality',
|
||||
{ type: 'number', ...QUALITY_LIMIT }],
|
||||
['thumbnail-edge', 'Thumbnail size limit', 'Longest edge in pixels, 128 to 2048. Rebuild all previews to apply a new size.',
|
||||
'thumbnail_edge', { type: 'number', ...EDGE_LIMIT }],
|
||||
]);
|
||||
const LIMITS = Object.freeze({ thumbnail_quality: QUALITY_LIMIT, thumbnail_edge: EDGE_LIMIT });
|
||||
|
||||
/**
|
||||
* Mount the Thumbnails surface; returns it. `support()` resolves whether this browser shows JPEG XL, `openReview`
|
||||
* opens Thumbnail maintenance (`{mode, trigger}`).
|
||||
*/
|
||||
export function mountThumbnailSettings(container, { scope, drafts, api, activity, support = detectJxlSupport,
|
||||
openReview = options => openThumbnailReview({ api, store: activity, scope, ...options }) }) {
|
||||
const surface = el('section', undefined, 'surface');
|
||||
surface.append(el('h2', 'Thumbnails'), el('p', 'Previews in the grid and the inspector cover. Originals are never changed.'));
|
||||
const controls = [];
|
||||
for (const [id, label, description, key, control] of ROWS) {
|
||||
let input;
|
||||
const onChange = value => {
|
||||
if (LIMITS[key]) {
|
||||
value = wholeNumber(input.value, LIMITS[key]);
|
||||
if (value === null) { drafts.reject('host', key); return; }
|
||||
}
|
||||
drafts.set('host', key, value);
|
||||
};
|
||||
const row = settingRow({ id, label, description, control, scope, onChange });
|
||||
input = row.children[1];
|
||||
input.dataset.key = key;
|
||||
surface.append(row);
|
||||
controls.push({ input, key, text: row.children[0] });
|
||||
}
|
||||
const operation = operationRow({ title: 'Convert outdated thumbnails', description: CONVERT_DESCRIPTION,
|
||||
label: 'Convert outdated', scope, onClick: trigger => openReview({ mode: 'convert', trigger }) });
|
||||
surface.append(operation);
|
||||
const operationText = operation.children[0].children[1];
|
||||
|
||||
let browserJxl;
|
||||
let savedFormat;
|
||||
const format = controls[0];
|
||||
const repaint = () => {
|
||||
const encoders = drafts.get('host', 'thumbnail_encoders');
|
||||
for (const { input, key } of controls) {
|
||||
const value = drafts.get('host', key);
|
||||
input.disabled = value === undefined;
|
||||
if (value !== undefined && (document.activeElement !== input || key === 'thumbnail_format')) input.value = String(value);
|
||||
}
|
||||
const jxl = Array.from(format.input.children).find(option => option.value === 'jxl');
|
||||
const unavailable = encoders?.jxl?.available === false;
|
||||
if (jxl) jxl.disabled = unavailable;
|
||||
const failed = drafts.hostState?.() === 'failed';
|
||||
for (const row of controls) {
|
||||
for (const note of row.notes ?? []) note.remove();
|
||||
const lines = [failed && drafts.get('host', row.key) === undefined ? HOST_UNAVAILABLE : null];
|
||||
if (row === format) lines.push(unavailable ? encoders.jxl.reason : null, browserJxl === false ? BROWSER_NO_JXL : null);
|
||||
row.notes = lines.filter(Boolean).map(line => el('p', line, 'small'));
|
||||
row.text.append(...row.notes);
|
||||
}
|
||||
// A format save (not the first read) leaves the existing previews outdated: say so, never convert.
|
||||
const saved = drafts.saved('host').thumbnail_format;
|
||||
if (savedFormat !== undefined && saved !== undefined && saved !== savedFormat) operationText.textContent = FORMAT_SAVED;
|
||||
if (saved !== undefined) savedFormat = saved;
|
||||
};
|
||||
scope.cleanup(drafts.subscribe(repaint));
|
||||
repaint();
|
||||
Promise.resolve(support()).then(value => {
|
||||
if (scope.signal.aborted) return;
|
||||
browserJxl = value;
|
||||
repaint();
|
||||
}, () => {});
|
||||
return surface;
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user