feat(system): thumbnails on Activity (#311)
* feat(system): run thumbnail generation as an Activity job
GET /prompt_manager/images/thumbnails/plan counts what a run would do
(targets, fresh, remaining past the cap, source labels) and writes
nothing. POST /prompt_manager/jobs/thumbnails {mode, size, quality}
starts the "thumbnails" job: admission guarded, cancellable at a file
boundary, outcome groups generated / skipped / errors. size is the
longest edge (128-2048), quality the encoder, and rebuild writes fresh
previews again through the same atomic rename, so a failed file keeps
its old preview and originals are only read.
The legacy gallery stream now starts or attaches to the same job and
streams its unchanged frames. A client that goes away no longer stops
the work; the gallery's Cancel asks the server to cancel the job.
* feat(activity): per-kind job descriptors
One descriptor table (activity-kinds.mjs) decides each job kind's pill
verb, toasts, line, tiles, outcomes, openable rows and labels; the
store re-exports it. The scan and image tagging views are unchanged.
A thumbnail job counts Generated, Skipped and Errors, reads
"Generating thumbnails" in the pill and "Thumbnails finished · N
generated · M errors" when it ends.
Any other kind (the health check, a held legacy-page operation) gets
the generic descriptor with no tiles and no Results, instead of the
scan's zero tiles. A running job without a known total shows an
indeterminate progress bar and no percentage, and only a job the
server can cancel shows Cancel.
* feat(system): review and start thumbnail jobs
Sources' Thumbnail generation -> Review options and Overview's Review
thumbnails open the concept's Thumbnail maintenance dialog: scope, quality
(preselected from Image loading quality) and preview size limit, a plan
line from the read-only thumbnail plan (D99) and Start, which starts the
thumbnails Activity job and opens it. A size outside 128-2048 requests
nothing; a refused start shows the busy link toast.
* fix(system): thumbnail job review fixes (#249 S5)
- The legacy stream's first frame is job {id, started}, sent after the
subscription so no runner frame is missed; it reads the registry after
prepare, so a job started in that gap is attached to, not refused.
- Every exit sets a truthful final status line; nothing to do never
reads "Processing file 0 of 0".
- Cancel is checked per file during the planning walk; a job cancelled
there publishes no start frame.
- The ffmpeg fallback keeps the chosen quality (-q:v) and the aspect
ratio; the placeholder uses the chosen JPEG settings.
- A locked preview on Windows reads "Preview in use by another program;
the old one was kept", not the raw WinError.
- Thumbnail work no longer defers the monitor's Needs review recording:
it writes previews only, never records.
- The disconnect test now proves the job outlives its stream (one worker,
9 files, three batches); the attach test waits on an event.
* fix(gallery): cancel the stream's own thumbnail job by id
The settings panel and the startup dialog cancel the job named by the
stream's job frame, and only one their stream started. A Cancel before
the frame waits for it, a double Cancel posts once, and a job another
tab or System started keeps running. The startup dialog now settles on
Cancel instead of staying on "Cancelling...".
* fix(activity): indeterminate pill bar and the nothing-to-do line
The pill's mini bar sweeps while a job's total is unknown instead of
showing an empty fill. A finished thumbnail job with nothing to do shows
the server's final line. Coverage floors raised for gallery.js and
activity.mjs.
This commit is contained in:
+122
-11
@@ -1353,8 +1353,8 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f
|
||||
the selection bar's tagging shows is the concept's or the plan's (D85, D90, D91).
|
||||
- **D89** (plan D66, Activity for the `image-tags` job): the job's Activity item and its
|
||||
view are the scan's (D54, D56, D57, D59) with the job kind's own counters, from one
|
||||
per-kind table (`JOB_VIEWS` in `activity-store.mjs`; a kind without one keeps the
|
||||
scan's tiles under "Job log" and "Cancel job"). The titles are the server's: "Add
|
||||
per-kind table (`activity-kinds.mjs` since S5, D111; a kind without one gets no
|
||||
tiles, "Job log" and "Cancel job"). The titles are the server's: "Add
|
||||
tags to images" and "Auto-tag images".
|
||||
- The pill's verb is "Tagging" ("Tagging · 41% · 8,240 / 19,622").
|
||||
- The item line and the view's heading line read "Started HH:MM · N images · closing
|
||||
@@ -1387,7 +1387,7 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f
|
||||
finished · N tagged · M errors", "<title> cancelled · N tagged" or "<title> failed ·
|
||||
N tagged · M errors".
|
||||
- Tests: `tests/js/workspace-activity-store.test.js` ("image-tags jobs get their own
|
||||
tiles outcomes and pill verb", "unknown kinds keep the generic view");
|
||||
tiles outcomes and pill verb", "unknown kinds get the generic view without the scan tiles");
|
||||
`tests/browser/workspace-activity.spec.mjs` ("an image tagging job shows Tagged
|
||||
Unchanged Skipped Errors and opens tagged rows", "cancel tagging stops at a file
|
||||
boundary", "the report download names the kind", on fake `image-tags` jobs from
|
||||
@@ -2157,10 +2157,79 @@ keeps the drafts; a save refused for one source says the server's message as the
|
||||
and marks that row. After a save the source list is read again; a monitoring change
|
||||
waits for a restart (the running observer keeps watching until then, and the status
|
||||
says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's
|
||||
Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5).
|
||||
Review options opens the concept's "Thumbnail maintenance" dialog (D99, below).
|
||||
A list that cannot be read shows "Sources could not be loaded. Your saved data has not
|
||||
changed." and Retry, never an empty list.
|
||||
|
||||
Thumbnail generation is an Activity job (S5, `py/api/thumbnail_jobs.py`). `GET
|
||||
/prompt_manager/images/thumbnails/plan?mode=missing|rebuild&size=128..2048&quality=low|medium|high`
|
||||
counts the work (`targets`, `fresh`, `remaining` past the 10,000-file cap, the source
|
||||
labels) and writes nothing; a value out of range is a 400 `invalid-request` that starts
|
||||
nothing. `POST /prompt_manager/jobs/thumbnails {mode, size, quality}` starts the
|
||||
`thumbnails` job ("Thumbnail generation"): admission `guarded` (never beside a scan or
|
||||
another record-changing job, one at a time, a legacy clear refused while it runs),
|
||||
cancellable at a file boundary (a file not yet started is left alone; written previews
|
||||
stay), outcome groups Generated / Skipped (already fresh, listed without a log line
|
||||
each) / Errors. `size` is the longest edge and `quality` the encoder (low: JPEG 70, no
|
||||
optimize; medium: 85, optimized, what the legacy routes always wrote; high: 95,
|
||||
optimized). Video previews take the same choice: the OpenCV frame and the placeholder
|
||||
are saved with those JPEG settings, and the ffmpeg fallback (no OpenCV) maps them to
|
||||
`-q:v` 8 / 4 / 2 and fits the frame inside the edge keeping its aspect ratio (`-vf
|
||||
scale=E:E:force_original_aspect_ratio=decrease`, no longer a stretched square).
|
||||
`rebuild` writes fresh previews again, each through the same temp-file
|
||||
rename, so nothing is cleared first and a failed file keeps its old preview (a video
|
||||
whose frame cannot be decoded keeps its preview instead of getting the placeholder);
|
||||
originals are only read (`tests/test_thumbnail_jobs.py`). Cancel is also checked per
|
||||
file during the planning walk: a job cancelled there stops the walk at the next file,
|
||||
publishes no `start` frame and ends "Cancelled before any preview was written". Every
|
||||
exit sets a final status line: "N previews written · M errors" (plus "· R more on the
|
||||
next run" past the cap), "No previews to generate · N already up to date" (or "No
|
||||
images or videos to preview"), "Cancelled · N previews written", or the failure; a job
|
||||
with nothing to do never reads "Processing file 0 of 0" (`test_every_exit_sets_a_truthful_final_status_line`,
|
||||
and "a finished thumbnail job with nothing to do says so" in `workspace-activity.spec.mjs`).
|
||||
Windows: replacing a preview another process holds open (the gallery serving it, an
|
||||
image viewer) fails with WinError 5 or 32; that file counts as an error reading
|
||||
"Preview in use by another program; the old one was kept" (never the raw "[WinError
|
||||
5]"), and its old preview stays. Any other PermissionError reads "Permission denied
|
||||
writing the preview" (`test_a_locked_preview_is_a_readable_coded_error`).
|
||||
Thumbnail work (the job and the gallery's generate and clear holds) is `guarded`, but it
|
||||
writes only preview files and no records, so the image monitor's Needs review
|
||||
recording runs beside it instead of being deferred for a whole rebuild
|
||||
(`PREVIEW_ONLY_KINDS` in `py/api/review_recording.py`, `blocker(..., compatible=)`;
|
||||
`test_a_running_thumbnail_job_does_not_defer_monitor_recording`). Other `guarded` and
|
||||
all `records` jobs still defer it.
|
||||
|
||||
Deliberate change: the legacy gallery stream `POST /images/generate-thumbnails/progress`
|
||||
now starts this job (missing previews, its quality preset as the size, medium encoding)
|
||||
or attaches to the running one, and streams the job's unchanged `status`/`start`/
|
||||
`progress`/`file_error`/`complete` frames after a first `job {id, started}` frame that
|
||||
names the job and whether this stream started it. The stream reads the registry after
|
||||
its headers are sent, so a job that starts in that gap is attached to, not refused. A
|
||||
client that goes away no longer stops the work (`test_thumbnail_job_survives_client_disconnect`:
|
||||
one worker, 9 files in three batches; the stream sees its client go after the first
|
||||
batch and all 9 previews are written, where a stream that stops on disconnect leaves 1).
|
||||
The gallery's Cancel (its settings panel and the startup dialog) cancels by that id,
|
||||
and only a job its own stream started: a Cancel before the `job` frame keeps the stream
|
||||
open until it arrives, a double Cancel posts once, and a job another tab or System
|
||||
started keeps running while the gallery stops following it (`tests/js/gallery-thumbnails.test.js`).
|
||||
A job cancelled from Activity ends the stream with the error frame "Thumbnail
|
||||
generation was cancelled". The blocking
|
||||
`POST /images/generate-thumbnails` and the clear stay legacy holds.
|
||||
|
||||
The workspace starts it from "Thumbnail maintenance" (`thumbnail-review.mjs`, the
|
||||
concept's `thumbnail-review`), opened by Sources' Thumbnail generation → Review options
|
||||
and Overview's Review thumbnails. Scope is Missing thumbnails only (`missing`, the
|
||||
default) or Rebuild all (explicit opt-in) (`rebuild`); Quality is Medium · optimized /
|
||||
High · original / Low · fastest (`medium` / `high` / `low`), preselected from
|
||||
Appearance's Image loading quality (`gallerySettings.imageQuality`, the same values,
|
||||
Q3); Preview size limit is the longest edge, 512 by default, a whole number from 128 to
|
||||
2048. A size outside that range marks the field `aria-invalid`, the plan line says
|
||||
"Enter a size from 128 to 2048 pixels." and nothing is requested; Start reports the
|
||||
field's validity and keeps the dialog open. Start closes the dialog and starts the job
|
||||
through the Activity store, which toasts "Thumbnail generation started."; the job opens
|
||||
in Activity. A start refused because another job holds the slot toasts the D57 busy
|
||||
link toast and stays. The legacy gallery's thumbnail tools are unchanged.
|
||||
|
||||
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
|
||||
visit through `latestOnly()` on the section scope; a late answer after the section left
|
||||
paints nothing. Until it answers, the section shows "Loading the overview…"; a failure
|
||||
@@ -2188,7 +2257,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged
|
||||
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
|
||||
reader can keep it byte-identical (`test_health_check_never_writes`).
|
||||
|
||||
Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned):
|
||||
Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D128 assigned):
|
||||
|
||||
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
|
||||
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
|
||||
@@ -2209,7 +2278,7 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121
|
||||
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
|
||||
missing thumbnails", "N missing thumbnails" (the pending count of
|
||||
`/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails
|
||||
opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid
|
||||
opens the thumbnail dialog (D99). "One display configuration" says "Grid
|
||||
and density settings are shared across views." (no paging). The sources row is
|
||||
"N output sources" with "Monitoring and enrichment share one source list.", prefixed
|
||||
by "M of N configured sources available." or "None configured is available; ComfyUI's
|
||||
@@ -2270,6 +2339,43 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121
|
||||
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
|
||||
folders." or "Files could not be refreshed. Try again.". The concept's "sample path"
|
||||
note is prototype copy and is dropped.
|
||||
- **D99** (Sources/Overview, S5): the concept's `thumbnail-review` dialog ("Thumbnail
|
||||
maintenance") with production copy. A `p.small[role=status]` line before the actions
|
||||
says what the server's read-only plan (`GET /images/thumbnails/plan`) counts for the
|
||||
chosen scope: "N previews to generate · M already up to date." or, for Rebuild all,
|
||||
"Every preview is regenerated: N files.", with "· R more on the next run" past the
|
||||
per-run cap; "Counting previews…" while it is read and "The previews could not be
|
||||
counted." when the read fails (Start still starts). Each scope is read once per
|
||||
dialog, since size and quality change no count. The primary is "Start" (the
|
||||
concept's "Simulate thumbnail job"). Overview's Review thumbnails opens the same
|
||||
dialog (the concept's row starts a sample "Thumbnail check" job directly). Maintenance's
|
||||
Thumbnail cache row takes the dialog with S7. `thumbnail dialog matches the concept
|
||||
from sources|overview` (system-concept.spec, both themes and sizes) maps the line and
|
||||
"Start"; `thumbnail dialog comparison detects mutations` proves they are the only
|
||||
difference and catches a renamed option, swapped options, swapped actions and a
|
||||
changed style.
|
||||
- **D111** (Activity, S5): each job kind's pill verb, toasts, line, tiles, outcomes,
|
||||
openable rows and labels come from one descriptor table (`activity-kinds.mjs`;
|
||||
`activity-store.mjs` re-exports it). The scan's (D56) and image tagging's (D89) are
|
||||
unchanged. A thumbnail job reads "Generating thumbnails · N% · a / b" in the pill,
|
||||
counts Generated (green), Skipped and Errors (red), lists Generated, Errors, Skipped
|
||||
(Generated and Skipped rows open the viewer), says "Thumbnails finished · N generated ·
|
||||
M errors" ("cancelled · N generated", "failed · …") and keeps "Job log" and "Cancel
|
||||
job". Every other kind (the health check, a kind this page does not know) gets the
|
||||
generic descriptor: title-based toasts and no tiles or Results, so it never shows the
|
||||
scan's zero tiles (S1 review M6, S2 review M3). A running job without a known total
|
||||
has an indeterminate `<progress>` (no `value`) and a pill without a percentage, whose
|
||||
mini bar (`aria-hidden`; the pill's label carries the state) is marked `indeterminate`
|
||||
and sweeps instead of showing an empty fill (a dimmed full bar under reduced motion). Only a
|
||||
running job the server can cancel shows Cancel. Tests:
|
||||
`tests/js/workspace-activity-kinds.test.js`, and in `workspace-activity.spec.mjs` "a
|
||||
thumbnail job shows Generated Skipped Errors, its pill verb and its toasts" and "a held
|
||||
legacy job and the health check show no tiles, no Results and no made-up percentage".
|
||||
- **D112** (Activity, S5): a legacy-page operation held on the registry (not
|
||||
cancellable, no outcome groups) reads as its server title ("Database restore (admin
|
||||
page)", "Thumbnail clear (gallery page)"), with no tiles, no Results, no Cancel, an
|
||||
indeterminate bar while it runs and "<title> finished." when it ends, whatever its
|
||||
kind (the gallery's clear is a `thumbnails` hold).
|
||||
- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the
|
||||
concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's
|
||||
sample 4).
|
||||
@@ -2358,10 +2464,13 @@ and a source row, renames a label and the Add source button and swaps the `--lin
|
||||
token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
|
||||
`{concept,production}-{statistics,health}-{dark,light}.png`,
|
||||
`{concept,production}-appearance-{dark,light}-{1600,390}.png`,
|
||||
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there
|
||||
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`,
|
||||
`concept-thumbnail-review-{dark,light}-{1600,390}.png`,
|
||||
`production-thumbnail-review-{sources,overview}-{dark,light}-{1600,390}.png`) are test artifacts; there
|
||||
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
|
||||
the health check and metric 4, a failed check, the thumbnail row against the status
|
||||
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
|
||||
endpoint (painted after the rest when its read is held, unavailable when it fails), its Review thumbnails
|
||||
opening the thumbnail dialog and starting the job, the review row (Q6), the section links, the Statistics dialog against the
|
||||
route, a load failure with Retry and late answers after leaving, and Appearance: new
|
||||
visits versus the current view and a saved view, the embedded gallery keeping Grid,
|
||||
metadata off and on request, the cache (counted up to a request barrier), hidden paths,
|
||||
@@ -2371,7 +2480,9 @@ the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers
|
||||
notice's Later and Review scan, a scoped scan from one source into Activity, Add source
|
||||
with its server check and Save, a refused folder, Remove down to auto-detect, an absent
|
||||
configured source, the three watch states, the monitoring and worker rows, Refresh
|
||||
files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are
|
||||
files, the thumbnail dialog (its plan line per scope with the preselected quality, an
|
||||
invalid size that requests nothing, Start into Activity, Rebuild all, a failed plan and
|
||||
a refused start), a save refused by a running scan and a failed read. Its hooks are
|
||||
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
|
||||
failures, an absent source, a deleted linked file, unthumbnailed files, a fake image
|
||||
monitor) on the
|
||||
@@ -2386,7 +2497,7 @@ node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-co
|
||||
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \
|
||||
tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \
|
||||
tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \
|
||||
tests/js/workspace-system-sources.test.js
|
||||
tests/js/workspace-system-sources.test.js tests/js/workspace-thumbnail-review.test.js
|
||||
python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v
|
||||
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
|
||||
workspace-system-sources.spec.mjs \
|
||||
@@ -2655,7 +2766,7 @@ pass runs after its own link, best effort: a failure is logged and the link
|
||||
stays. Recording follows the registry's `records`
|
||||
admission (#289): the scan's recording runs under its own `records` job, and
|
||||
the monitor's (one file, not a job) is refused while a `records` or `guarded`
|
||||
job or hold runs. It then records nothing, logs that the file waits for the
|
||||
job or hold runs, except thumbnail work, which writes no records (S5 M7). It then records nothing, logs that the file waits for the
|
||||
next scan, and answers `deferred`; the next "new" scan reads the ambiguous file
|
||||
again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A
|
||||
rescan does not copy or hash an ambiguous file whose current size and
|
||||
|
||||
@@ -8,6 +8,7 @@ Split into domain-specific mixins for maintainability:
|
||||
- AutotagRoutesMixin: auto-tagging model management and tagging
|
||||
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
|
||||
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
|
||||
- ThumbnailJobsMixin: thumbnail plan, thumbnail job and the gallery stream on it
|
||||
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
|
||||
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
|
||||
- ReviewEventsMixin: the Needs review change stream an open page re-reads on
|
||||
@@ -43,6 +44,7 @@ from .admin import AdminRoutesMixin
|
||||
from .settings_routes import SettingsRoutesMixin
|
||||
from .job_routes import JobRoutesMixin
|
||||
from .image_tag_jobs import ImageTagJobsMixin
|
||||
from .thumbnail_jobs import ThumbnailJobsMixin
|
||||
from .system import SystemRoutesMixin
|
||||
from .jobs import JobRegistry
|
||||
from .logging_routes import LoggingRoutesMixin
|
||||
@@ -281,6 +283,7 @@ class PromptManagerAPI(
|
||||
SettingsRoutesMixin,
|
||||
JobRoutesMixin,
|
||||
ImageTagJobsMixin,
|
||||
ThumbnailJobsMixin,
|
||||
SystemRoutesMixin,
|
||||
LoggingRoutesMixin,
|
||||
AutotagRoutesMixin,
|
||||
|
||||
+171
-175
@@ -20,7 +20,7 @@ from typing import Callable, NamedTuple
|
||||
from aiohttp import web
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
from .jobs import THUMBNAILS_JOB_KIND
|
||||
from .jobs import MAX_OUTCOME_ROWS, THUMBNAILS_JOB_KIND
|
||||
from .prompts import bad_request, parse_page_params, publish_image_paths
|
||||
from .prompts import thumbnail_url_for
|
||||
from .prompts import safe_error_message as _safe_error
|
||||
@@ -235,6 +235,8 @@ MEDIA_EXTENSIONS = GALLERY_IMAGE_EXTENSIONS | VIDEO_EXTENSIONS
|
||||
# Caps on request-triggered filesystem work.
|
||||
MAX_SCAN_DEPTH = 12
|
||||
MAX_THUMBNAIL_FILES = 10_000
|
||||
# Already-fresh files a thumbnail job lists under Skipped (its counter stays exact).
|
||||
MAX_FRESH_LISTED = MAX_OUTCOME_ROWS
|
||||
|
||||
|
||||
def _worker_threads():
|
||||
@@ -271,15 +273,52 @@ def _atomic_save(img, dst, fmt=None, **save_kwargs):
|
||||
tmp.unlink()
|
||||
|
||||
|
||||
def _write_image_thumbnail(src, dst, thumbnail_size):
|
||||
"""Resize *src* into *dst* (blocking PIL work), atomically."""
|
||||
# Encoder settings per thumbnail quality (the System thumbnail dialog's
|
||||
# "Medium · optimized", "High · original", "Low · fastest"). The legacy
|
||||
# routes always encode as "medium", which is what they always wrote.
|
||||
THUMBNAIL_ENCODINGS = {
|
||||
"low": {"quality": 70, "optimize": False},
|
||||
"medium": {"quality": 85, "optimize": True},
|
||||
"high": {"quality": 95, "optimize": True},
|
||||
}
|
||||
DEFAULT_THUMBNAIL_ENCODING = THUMBNAIL_ENCODINGS["medium"]
|
||||
# ffmpeg's JPEG scale (-q:v, 2 best to 31 worst) for each encoding's quality.
|
||||
_FFMPEG_QSCALE = {95: 2, 85: 4, 70: 8}
|
||||
|
||||
# A preview that could not be replaced (#249 S5 M6). Windows refuses to rename
|
||||
# over a file another process has open (WinError 5 or 32), such as a preview
|
||||
# the gallery is serving; the old preview stays and the file counts as an error.
|
||||
PREVIEW_IN_USE = "Preview in use by another program; the old one was kept"
|
||||
PREVIEW_NOT_WRITABLE = "Permission denied writing the preview"
|
||||
_WINDOWS_LOCKED = (5, 32) # ERROR_ACCESS_DENIED, ERROR_SHARING_VIOLATION
|
||||
|
||||
|
||||
def _preview_write_error(exc):
|
||||
"""The readable error for a PermissionError while writing a preview."""
|
||||
if getattr(exc, "winerror", None) in _WINDOWS_LOCKED:
|
||||
return PREVIEW_IN_USE
|
||||
return PREVIEW_NOT_WRITABLE
|
||||
|
||||
|
||||
def _ffmpeg_qscale(encoding):
|
||||
"""ffmpeg's ``-q:v`` for a THUMBNAIL_ENCODINGS entry."""
|
||||
return str(_FFMPEG_QSCALE.get(encoding["quality"], 4))
|
||||
|
||||
|
||||
def _write_image_thumbnail(src, dst, thumbnail_size, encoding=None):
|
||||
"""Resize *src* into *dst* (blocking PIL work), atomically.
|
||||
|
||||
``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); a PNG
|
||||
preview keeps only its ``optimize`` flag.
|
||||
"""
|
||||
encoding = encoding or DEFAULT_THUMBNAIL_ENCODING
|
||||
with Image.open(src) as img:
|
||||
if img.mode in ("RGBA", "LA", "P"):
|
||||
img = img.convert("RGB")
|
||||
img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS)
|
||||
save_kwargs = {"quality": 85, "optimize": True}
|
||||
save_kwargs = dict(encoding)
|
||||
if dst.suffix.lower() == ".png":
|
||||
save_kwargs = {"optimize": True}
|
||||
save_kwargs = {"optimize": encoding["optimize"]}
|
||||
_atomic_save(img, dst, **save_kwargs)
|
||||
|
||||
|
||||
@@ -945,6 +984,10 @@ class ImageRoutesMixin:
|
||||
async def thumbnail_status_route(request):
|
||||
return await self.get_thumbnail_status(request)
|
||||
|
||||
@routes.get("/prompt_manager/images/thumbnails/plan")
|
||||
async def thumbnail_plan_route(request):
|
||||
return await self.get_thumbnail_plan(request)
|
||||
|
||||
@routes.post("/prompt_manager/images/generate-thumbnails")
|
||||
async def generate_thumbnails_route(request):
|
||||
return await self.generate_thumbnails(request)
|
||||
@@ -1817,15 +1860,35 @@ class ImageRoutesMixin:
|
||||
self.logger.info(payload["message"])
|
||||
return payload
|
||||
|
||||
def _plan_gallery_thumbnails(self, output_dirs: list[Path]) -> dict:
|
||||
"""Plan every gallery root with one shared per-run generation cap."""
|
||||
combined = {"targets": [], "skipped": 0, "remaining": 0}
|
||||
def _plan_gallery_thumbnails(
|
||||
self, output_dirs: list[Path], rebuild: bool = False, stop=None
|
||||
) -> dict:
|
||||
"""Plan every gallery root with one shared per-run generation cap.
|
||||
|
||||
See ``_plan_thumbnails``; ``fresh_files`` keeps at most
|
||||
MAX_FRESH_LISTED files across the roots. ``stop()`` true ends the walk
|
||||
at the next file (a cancelled job); the partial plan is returned.
|
||||
"""
|
||||
combined = {
|
||||
"targets": [],
|
||||
"skipped": 0,
|
||||
"remaining": 0,
|
||||
"fresh": 0,
|
||||
"fresh_files": [],
|
||||
}
|
||||
for root in output_dirs:
|
||||
if stop is not None and stop():
|
||||
break
|
||||
root = Path(root)
|
||||
plan = self._plan_thumbnails(root, root / "thumbnails")
|
||||
plan = self._plan_thumbnails(
|
||||
root, root / "thumbnails", rebuild=rebuild, stop=stop
|
||||
)
|
||||
available = max(0, MAX_THUMBNAIL_FILES - len(combined["targets"]))
|
||||
combined["targets"].extend(plan["targets"][:available])
|
||||
combined["skipped"] += plan["skipped"]
|
||||
combined["fresh"] += plan["fresh"]
|
||||
room = max(0, MAX_FRESH_LISTED - len(combined["fresh_files"]))
|
||||
combined["fresh_files"].extend(plan["fresh_files"][:room])
|
||||
combined["remaining"] += plan["remaining"] + max(
|
||||
0, len(plan["targets"]) - available
|
||||
)
|
||||
@@ -1858,22 +1921,30 @@ class ImageRoutesMixin:
|
||||
{"success": False, "error": _safe_error(exc)}, status=500
|
||||
)
|
||||
|
||||
def _plan_thumbnails(self, output_path, thumbnails_dir):
|
||||
def _plan_thumbnails(self, output_path, thumbnails_dir, rebuild=False, stop=None):
|
||||
"""Decide which media files need a thumbnail (blocking walk).
|
||||
|
||||
Returns ``{"targets": [(src, dst, is_video)], "skipped": n, "remaining": n}``.
|
||||
Files whose thumbnail is already newer than the source are counted as
|
||||
skipped and never queued, so a large library that is mostly done only
|
||||
pays for the files that changed. The walk is bounded like the gallery
|
||||
listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES, no symlinks) and at most
|
||||
MAX_THUMBNAIL_FILES pending files are queued per run; ``remaining``
|
||||
counts the pending files beyond that cap.
|
||||
Returns ``{"targets": [(src, dst, is_video)], "skipped": n,
|
||||
"remaining": n, "fresh": n, "fresh_files": [src]}``. Files whose
|
||||
thumbnail is already newer than the source are ``fresh``. Without
|
||||
``rebuild`` they are counted as skipped and never queued, so a large
|
||||
library that is mostly done only pays for the files that changed;
|
||||
``fresh_files`` lists the first MAX_FRESH_LISTED of them. With
|
||||
``rebuild`` every file is a target (nothing is skipped). The walk is
|
||||
bounded like the gallery listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES,
|
||||
no symlinks) and at most MAX_THUMBNAIL_FILES targets are queued per
|
||||
run; ``remaining`` counts the targets beyond that cap. ``stop()`` true
|
||||
ends the walk before the next file.
|
||||
"""
|
||||
output_path = Path(output_path)
|
||||
targets = []
|
||||
skipped = 0
|
||||
remaining = 0
|
||||
fresh = 0
|
||||
fresh_files = []
|
||||
for src in _iter_media_files(output_path, MAX_GALLERY_FILES):
|
||||
if stop is not None and stop():
|
||||
break
|
||||
suffix = src.suffix.lower()
|
||||
is_video = suffix in VIDEO_EXTENSIONS
|
||||
rel_no_ext = src.relative_to(output_path).with_suffix("")
|
||||
@@ -1883,8 +1954,12 @@ class ImageRoutesMixin:
|
||||
self.logger.warning(f"Skipping thumbnail outside safe dir: {src.name}")
|
||||
continue
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
skipped += 1
|
||||
continue
|
||||
fresh += 1
|
||||
if not rebuild:
|
||||
skipped += 1
|
||||
if len(fresh_files) < MAX_FRESH_LISTED:
|
||||
fresh_files.append(src)
|
||||
continue
|
||||
if len(targets) >= MAX_THUMBNAIL_FILES:
|
||||
remaining += 1
|
||||
continue
|
||||
@@ -1894,190 +1969,81 @@ class ImageRoutesMixin:
|
||||
f"Thumbnail run capped at {MAX_THUMBNAIL_FILES} new files; "
|
||||
f"{remaining} more will be generated on the next run"
|
||||
)
|
||||
return {"targets": targets, "skipped": skipped, "remaining": remaining}
|
||||
return {
|
||||
"targets": targets,
|
||||
"skipped": skipped,
|
||||
"remaining": remaining,
|
||||
"fresh": fresh,
|
||||
"fresh_files": fresh_files,
|
||||
}
|
||||
|
||||
def _thumbnail_targets(self, output_path, thumbnails_dir):
|
||||
"""Pending (src, dst, is_video) triples; see _plan_thumbnails."""
|
||||
return self._plan_thumbnails(output_path, thumbnails_dir)["targets"]
|
||||
|
||||
def _generate_batch(self, targets, thumbnail_size):
|
||||
def _generate_batch(self, targets, thumbnail_size, **options):
|
||||
"""Create thumbnails for ``targets`` on the configured worker threads.
|
||||
|
||||
Blocking; returns one result dict per target, in order.
|
||||
Blocking; returns one result dict per target, in order. ``options``
|
||||
go to ``_generate_one`` (``encoding``, ``rebuild``, ``stop``).
|
||||
"""
|
||||
|
||||
def one(target):
|
||||
src, dst, is_video = target
|
||||
return self._generate_one(src, dst, thumbnail_size, is_video)
|
||||
return self._generate_one(src, dst, thumbnail_size, is_video, **options)
|
||||
|
||||
return map_parallel(one, targets, _worker_threads())
|
||||
|
||||
def _generate_one(self, src, dst, thumbnail_size, is_video):
|
||||
"""Create one thumbnail (blocking). Never raises; returns a result dict."""
|
||||
def _generate_one(
|
||||
self,
|
||||
src,
|
||||
dst,
|
||||
thumbnail_size,
|
||||
is_video,
|
||||
encoding=None,
|
||||
rebuild=False,
|
||||
stop=None,
|
||||
):
|
||||
"""Create one thumbnail (blocking). Never raises; returns a result dict.
|
||||
|
||||
``stop()`` true before the file starts gives ``action: "cancelled"``
|
||||
and touches nothing (a job's file boundary). ``rebuild`` writes a
|
||||
fresh preview again and never replaces an existing video preview with
|
||||
the placeholder. Every write is atomic, so a failure keeps the old
|
||||
preview.
|
||||
"""
|
||||
result = {
|
||||
"file": src.name,
|
||||
"dir": src.parent.name,
|
||||
"type": "video" if is_video else "image",
|
||||
}
|
||||
try:
|
||||
if stop is not None and stop():
|
||||
return {**result, "action": "cancelled"}
|
||||
if not src.is_file():
|
||||
return {**result, "action": "error", "error": "File no longer exists"}
|
||||
if _thumbnail_is_fresh(src, dst):
|
||||
if not rebuild and _thumbnail_is_fresh(src, dst):
|
||||
return {**result, "action": "skipped"}
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
if is_video:
|
||||
if not self._generate_video_thumbnail(src, dst, thumbnail_size):
|
||||
made = self._generate_video_thumbnail(
|
||||
src, dst, thumbnail_size, encoding, keep_existing=rebuild
|
||||
)
|
||||
if not made:
|
||||
return {**result, "action": "error", "error": "Video decode failed"}
|
||||
else:
|
||||
_write_image_thumbnail(src, dst, thumbnail_size)
|
||||
_write_image_thumbnail(src, dst, thumbnail_size, encoding)
|
||||
return {**result, "action": "generated"}
|
||||
except UnidentifiedImageError:
|
||||
return {**result, "action": "error", "error": "Not a valid image file"}
|
||||
except PermissionError as e:
|
||||
message = _preview_write_error(e)
|
||||
self.logger.warning(f"Thumbnail failed for {src.name}: {message}")
|
||||
return {**result, "action": "error", "error": message}
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Thumbnail failed for {src.name}", exc_info=True)
|
||||
return {**result, "action": "error", "error": _safe_error(e)}
|
||||
|
||||
async def generate_thumbnails_with_progress(self, request):
|
||||
"""Generate thumbnails with Server-Sent Events progress updates (POST).
|
||||
|
||||
``quality`` is read from the query string so a POST-capable SSE
|
||||
client can keep using the same URL shape. Every blocking step (scan,
|
||||
PIL/ffmpeg) runs in the executor; the event loop only writes SSE
|
||||
frames between files.
|
||||
"""
|
||||
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
|
||||
response = web.StreamResponse(
|
||||
status=200,
|
||||
headers={
|
||||
"Content-Type": "text/event-stream",
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
},
|
||||
)
|
||||
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
|
||||
if busy is not None:
|
||||
# The gallery reads this stream: refuse with its error event.
|
||||
await response.prepare(request)
|
||||
message = f"{busy.title} already running"
|
||||
await self._emit_sse(
|
||||
response, "error", {"error": message, "message": message}
|
||||
)
|
||||
return response
|
||||
async with self._jobs.hold(
|
||||
THUMBNAILS_JOB_KIND, THUMBNAIL_GENERATION_TITLE, admission="guarded"
|
||||
) as job:
|
||||
await response.prepare(request)
|
||||
if not await self._thumbnail_progress_stream(response, thumbnail_size):
|
||||
job.finish("failed") # an error event ended the stream
|
||||
return response
|
||||
|
||||
async def _thumbnail_progress_stream(self, response, thumbnail_size) -> bool:
|
||||
"""Stream one generation run; False when it ended with an error event."""
|
||||
try:
|
||||
output_dirs = self._get_all_output_dirs()
|
||||
if not output_dirs:
|
||||
await self._emit_sse(
|
||||
response, "error", {"error": "ComfyUI output directory not found"}
|
||||
)
|
||||
return False
|
||||
|
||||
await self._emit_sse(
|
||||
response,
|
||||
"status",
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails, output_dirs
|
||||
)
|
||||
await self._stream_thumbnails(
|
||||
response,
|
||||
plan["targets"],
|
||||
thumbnail_size,
|
||||
skipped=plan["skipped"],
|
||||
remaining=plan["remaining"],
|
||||
)
|
||||
self.invalidate_gallery_cache()
|
||||
return True
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail generation failed")
|
||||
await self._emit_sse(
|
||||
response,
|
||||
"error",
|
||||
{
|
||||
"error": "An internal error occurred",
|
||||
"message": "Thumbnail generation failed. Check server logs.",
|
||||
},
|
||||
)
|
||||
return False
|
||||
|
||||
async def _stream_thumbnails(
|
||||
self, response, targets, thumbnail_size, skipped=0, remaining=0
|
||||
):
|
||||
"""Process pending *targets* in worker-thread batches, emitting SSE per file.
|
||||
|
||||
``skipped`` files already had fresh thumbnails and ``remaining`` were
|
||||
left for the next run; both only feed the start/complete payloads.
|
||||
"""
|
||||
total = len(targets)
|
||||
video_count = sum(1 for _, _, is_video in targets if is_video)
|
||||
message = (
|
||||
f"Found {total - video_count} images and {video_count} videos to process"
|
||||
)
|
||||
if skipped:
|
||||
message += f" ({skipped} already have thumbnails)"
|
||||
connected = await self._emit_sse(
|
||||
response,
|
||||
"start",
|
||||
{
|
||||
"total_images": total,
|
||||
"skipped": skipped,
|
||||
"remaining": remaining,
|
||||
"phase": "processing",
|
||||
"image_count": total - video_count,
|
||||
"video_count": video_count,
|
||||
"message": message,
|
||||
},
|
||||
)
|
||||
if not connected:
|
||||
return
|
||||
stats = _new_thumbnail_stats()
|
||||
stats["skipped"] = skipped
|
||||
start = _time.monotonic()
|
||||
batch_size = _thumbnail_batch_size(_worker_threads())
|
||||
index = 0
|
||||
for batch_start in range(0, total, batch_size):
|
||||
batch = targets[batch_start : batch_start + batch_size]
|
||||
results = await self._run_in_executor(
|
||||
self._generate_batch, batch, thumbnail_size
|
||||
)
|
||||
for result in results:
|
||||
index += 1
|
||||
_record_thumbnail_result(stats, result)
|
||||
if result["action"] == "error":
|
||||
await self._emit_sse(
|
||||
response,
|
||||
"file_error",
|
||||
{"file": result["file"], "error": result["error"]},
|
||||
)
|
||||
connected = await self._emit_sse(
|
||||
response,
|
||||
"progress",
|
||||
_thumbnail_progress_payload(index, total, stats, result, start),
|
||||
)
|
||||
if not connected:
|
||||
self.logger.info("Thumbnail client disconnected; stopping early")
|
||||
return
|
||||
await asyncio.sleep(0) # let other requests run between batches
|
||||
payload = _thumbnail_complete_payload(
|
||||
total + skipped + remaining,
|
||||
stats,
|
||||
start,
|
||||
remaining=remaining,
|
||||
processed=total,
|
||||
)
|
||||
await self._emit_sse(response, "complete", payload)
|
||||
self.logger.info(payload["message"])
|
||||
|
||||
async def _emit_sse(self, response, event, data) -> bool:
|
||||
"""Write one SSE frame. Returns False once the client has gone away."""
|
||||
try:
|
||||
@@ -2088,8 +2054,25 @@ class ImageRoutesMixin:
|
||||
self.logger.warning(f"Failed to send SSE message: {e}")
|
||||
return False
|
||||
|
||||
def _generate_video_thumbnail(self, video_path, thumbnail_path, thumbnail_size):
|
||||
"""Generate thumbnail from video file. Returns True if successful."""
|
||||
def _generate_video_thumbnail(
|
||||
self,
|
||||
video_path,
|
||||
thumbnail_path,
|
||||
thumbnail_size,
|
||||
encoding=None,
|
||||
keep_existing=False,
|
||||
):
|
||||
"""Generate thumbnail from video file. Returns True if successful.
|
||||
|
||||
``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); it
|
||||
encodes the OpenCV frame, ffmpeg's frame (as ``-q:v``) and the
|
||||
placeholder alike. Both decoders fit the frame inside
|
||||
``thumbnail_size`` keeping its aspect ratio. With ``keep_existing`` an
|
||||
existing preview is kept, and the run fails, when no frame could be
|
||||
decoded, rather than replaced by the placeholder. A PermissionError
|
||||
while replacing the preview propagates (``_generate_one`` names it).
|
||||
"""
|
||||
encoding = encoding or DEFAULT_THUMBNAIL_ENCODING
|
||||
try:
|
||||
# Try using OpenCV first (most reliable)
|
||||
try:
|
||||
@@ -2113,7 +2096,7 @@ class ImageRoutesMixin:
|
||||
|
||||
img = Image.fromarray(frame_rgb)
|
||||
img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS)
|
||||
_atomic_save(img, thumbnail_path, "JPEG", quality=85, optimize=True)
|
||||
_atomic_save(img, thumbnail_path, "JPEG", **encoding)
|
||||
self.logger.debug(
|
||||
f"Generated video thumbnail using OpenCV: {thumbnail_path}"
|
||||
)
|
||||
@@ -2135,8 +2118,11 @@ class ImageRoutesMixin:
|
||||
"00:00:01",
|
||||
"-vframes",
|
||||
"1",
|
||||
"-s",
|
||||
f"{thumbnail_size[0]}x{thumbnail_size[1]}",
|
||||
"-vf",
|
||||
f"scale={thumbnail_size[0]}:{thumbnail_size[1]}"
|
||||
":force_original_aspect_ratio=decrease",
|
||||
"-q:v",
|
||||
_ffmpeg_qscale(encoding),
|
||||
"-y",
|
||||
str(tmp_path),
|
||||
]
|
||||
@@ -2161,6 +2147,12 @@ class ImageRoutesMixin:
|
||||
except (NameError, OSError):
|
||||
pass
|
||||
|
||||
if keep_existing and thumbnail_path.exists():
|
||||
self.logger.warning(
|
||||
f"No frame decoded for {video_path.name}; keeping its preview"
|
||||
)
|
||||
return False
|
||||
|
||||
# Last resort: create a placeholder thumbnail
|
||||
try:
|
||||
from PIL import ImageDraw, ImageFont
|
||||
@@ -2195,18 +2187,22 @@ class ImageRoutesMixin:
|
||||
except (OSError, AttributeError):
|
||||
pass
|
||||
|
||||
_atomic_save(img, thumbnail_path, "JPEG", quality=85)
|
||||
_atomic_save(img, thumbnail_path, "JPEG", **encoding)
|
||||
self.logger.debug(
|
||||
f"Generated placeholder video thumbnail: {thumbnail_path}"
|
||||
)
|
||||
return True
|
||||
|
||||
except PermissionError:
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.warning(
|
||||
f"Failed to create placeholder thumbnail for {video_path}: {e}"
|
||||
)
|
||||
return False
|
||||
|
||||
except PermissionError:
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"Video thumbnail generation failed for {video_path}: {e}"
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
"""Session job routes (Activity): list, inspect, follow and cancel jobs.
|
||||
|
||||
The registry itself is ``py/api/jobs.py``. The scan job's runner is in
|
||||
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``.
|
||||
``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``
|
||||
and the thumbnail job's in ``py/api/thumbnail_jobs.py``.
|
||||
Every response leaves out absolute server paths: job rows already carry
|
||||
root-relative paths and public root labels.
|
||||
"""
|
||||
@@ -129,6 +130,10 @@ class JobRoutesMixin:
|
||||
async def start_image_tag_job_route(request):
|
||||
return await self.start_image_tag_job(request)
|
||||
|
||||
@routes.post("/prompt_manager/jobs/thumbnails")
|
||||
async def start_thumbnail_job_route(request):
|
||||
return await self.start_thumbnail_job(request)
|
||||
|
||||
@routes.get("/prompt_manager/jobs/{job_id}")
|
||||
async def job_detail_route(request):
|
||||
return await self.job_detail(request)
|
||||
|
||||
+45
-7
@@ -72,6 +72,9 @@ SCAN_GROUPS = {
|
||||
IMAGE_TAG_JOB_KIND = "image-tags"
|
||||
IMAGE_TAG_GROUP_NAMES = ("tagged", "unchanged", "skipped", "errors")
|
||||
|
||||
# Thumbnail generation (#249 System S5): previews written, already fresh, failed.
|
||||
THUMBNAIL_GROUP_NAMES = ("generated", "skipped", "errors")
|
||||
|
||||
ADMISSIONS = ("read", "guarded", "records")
|
||||
# What each admission class is refused beside (besides its own kind).
|
||||
_REFUSED_BESIDE = {
|
||||
@@ -247,6 +250,32 @@ class Job:
|
||||
)
|
||||
self.log_line(log_line)
|
||||
|
||||
def record_counted(self, group, count, rows=()):
|
||||
"""Count ``count`` outcomes in ``group`` at once, listing ``rows``.
|
||||
|
||||
For outcomes that need no log line each (a thumbnail job's already
|
||||
fresh previews): the counter is exact, ``rows`` (``record``'s row
|
||||
shape) are listed up to MAX_OUTCOME_ROWS, and nothing is logged.
|
||||
"""
|
||||
if group not in self.counters:
|
||||
raise ValueError(f"Unknown outcome group {group!r} for {self.kind}")
|
||||
self.counters[group] += max(0, int(count))
|
||||
listed = self.outcomes[group]
|
||||
for row in rows:
|
||||
if len(listed) >= MAX_OUTCOME_ROWS:
|
||||
break
|
||||
index = row.get("root_index")
|
||||
if isinstance(index, bool) or not isinstance(index, int) or index < 0:
|
||||
index = None
|
||||
listed.append(
|
||||
{
|
||||
"path": _relative_path(row.get("path")),
|
||||
"root": str(row.get("root") or ""),
|
||||
"root_index": index,
|
||||
"detail": str(row.get("detail") or ""),
|
||||
}
|
||||
)
|
||||
|
||||
def finish(self, state="finished"):
|
||||
"""End the job in ``state`` and release every subscriber.
|
||||
|
||||
@@ -263,18 +292,24 @@ class Job:
|
||||
for queue in list(self._subscribers):
|
||||
queue.put_nowait(None)
|
||||
|
||||
async def events(self):
|
||||
async def events(self, lead=None):
|
||||
"""Yield the latest event, then each later one, until the job finishes.
|
||||
|
||||
``lead``, when given, is yielded first, after the subscription is in
|
||||
place, so nothing published while the consumer sends it is missed.
|
||||
Consume it under ``contextlib.aclosing`` so a consumer that stops early
|
||||
(a closed socket) drops its queue at once instead of at garbage collection.
|
||||
"""
|
||||
queue = asyncio.Queue()
|
||||
self._subscribers.add(queue)
|
||||
# As of the subscription: a later event or the end reaches the queue.
|
||||
snapshot, ended = self.last_event, self.finished
|
||||
try:
|
||||
if self.last_event is not None:
|
||||
yield self.last_event
|
||||
if self.finished:
|
||||
if lead is not None:
|
||||
yield lead
|
||||
if snapshot is not None:
|
||||
yield snapshot
|
||||
if ended:
|
||||
return
|
||||
while True:
|
||||
payload = await queue.get()
|
||||
@@ -425,11 +460,14 @@ class JobRegistry:
|
||||
self._jobs[job.id] = job
|
||||
return job
|
||||
|
||||
def blocker(self, kind, *, admission=None, record_changing=None):
|
||||
def blocker(self, kind, *, admission=None, record_changing=None, compatible=()):
|
||||
"""The running job a new ``kind`` job would be refused for, or None.
|
||||
|
||||
The same kind refuses first; then the oldest running job whose class
|
||||
the new job's admission class may not run beside.
|
||||
the new job's admission class may not run beside. Running jobs of a
|
||||
kind in ``compatible`` never refuse it: the caller knows that work
|
||||
cannot conflict with its own (the monitor's Needs review recording
|
||||
beside thumbnail work, which writes previews and no records).
|
||||
"""
|
||||
same_kind = self.active(kind)
|
||||
if same_kind is not None:
|
||||
@@ -439,7 +477,7 @@ class JobRegistry:
|
||||
running = [job for job in tuple(self._jobs.values()) if job.running]
|
||||
running.sort(key=lambda job: job._started_seq)
|
||||
for job in running:
|
||||
if job.admission in refused:
|
||||
if job.admission in refused and job.kind not in compatible:
|
||||
return job
|
||||
return None
|
||||
|
||||
|
||||
@@ -28,16 +28,23 @@ Recording changes records, so it follows the registry's ``records``
|
||||
admission (#289). The scan records under its own ``records`` job. The
|
||||
monitor's pass lasts one file and is not a job: it is refused while a
|
||||
``records`` or ``guarded`` job runs (``deferred``), and the next scan reads
|
||||
the file again.
|
||||
the file again. Thumbnail work (the job and the gallery's generate and clear
|
||||
holds, kind ``thumbnails``) is ``guarded`` but only writes and removes preview
|
||||
files under ``thumbnails/`` and never touches the database, so it does not
|
||||
defer the recording (#249 S5 M7): a long rebuild no longer holds Needs review
|
||||
back for its whole run.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots
|
||||
from .jobs import THUMBNAILS_JOB_KIND
|
||||
|
||||
# The kind the monitor's recording checks admission under (never registered).
|
||||
REVIEW_RECORDING_KIND = "review_recording"
|
||||
# Guarded kinds that change no records, so the recording may run beside them.
|
||||
PREVIEW_ONLY_KINDS = (THUMBNAILS_JOB_KIND,)
|
||||
# record_ambiguous_file outcomes that leave the file in the pending queue.
|
||||
QUEUED_OUTCOMES = ("recorded", "limited", "duplicate")
|
||||
NEW_OUTCOMES = ("recorded", "limited")
|
||||
@@ -185,11 +192,14 @@ class ReviewRecordingMixin:
|
||||
links it); any other file is recorded when it is ambiguous with a
|
||||
choice, and is ``untraced`` (its pending items without a choice
|
||||
removed) when it is ambiguous without one. While a
|
||||
job runs that a ``records`` job may not run beside, nothing is written
|
||||
and the answer is ``deferred``. Errors propagate: the monitor logs
|
||||
job runs that a ``records`` job may not run beside, other than
|
||||
thumbnail work (``PREVIEW_ONLY_KINDS``), nothing is written and the
|
||||
answer is ``deferred``. Errors propagate: the monitor logs
|
||||
them and keeps its link.
|
||||
"""
|
||||
busy = self._jobs.blocker(REVIEW_RECORDING_KIND, admission="records")
|
||||
busy = self._jobs.blocker(
|
||||
REVIEW_RECORDING_KIND, admission="records", compatible=PREVIEW_ONLY_KINDS
|
||||
)
|
||||
if busy is not None:
|
||||
self.logger.info(
|
||||
f"Needs review was not updated for "
|
||||
|
||||
@@ -0,0 +1,494 @@
|
||||
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
|
||||
|
||||
- ``GET /prompt_manager/images/thumbnails/plan?mode=&size=&quality=`` counts
|
||||
what a run would do over every gallery root and writes nothing:
|
||||
``{success, targets, fresh, remaining, roots, mode, size, quality}``.
|
||||
- ``POST /prompt_manager/jobs/thumbnails {mode, size, quality}`` starts the
|
||||
``thumbnails`` job (202 ``{job}``). Its admission class is ``guarded``: it
|
||||
never runs beside a record-changing job, and one thumbnail job runs at a
|
||||
time. It stops at a file boundary when cancelled, keeping the previews it
|
||||
wrote. Its outcome groups are ``generated``, ``skipped`` (already fresh)
|
||||
and ``errors``.
|
||||
|
||||
``mode`` is ``missing`` (previews that are absent or older than their file)
|
||||
or ``rebuild`` (every preview again, fresh ones too). ``size`` is the
|
||||
preview's longest edge, 128 to 2048 pixels. ``quality`` picks the encoder
|
||||
settings (``THUMBNAIL_ENCODINGS``). Every preview is written next to the old
|
||||
one and renamed over it, so nothing is cleared first and a failure keeps the
|
||||
old preview; originals are only read.
|
||||
|
||||
The legacy gallery stream ``POST /images/generate-thumbnails/progress``
|
||||
starts this job (its quality preset still sizes the previews) or attaches to
|
||||
the running one. Its first frame, ``job {id, started}``, names that job and
|
||||
whether this stream started it (the gallery cancels by that id, and only a
|
||||
job it started); then come the job's legacy ``status``/``start``/
|
||||
``progress``/``file_error``/``complete`` frames. A client that goes away no
|
||||
longer stops the work: the job outlives the request, as a scan does.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from .admin import _plural, _scan_location, _scan_roots, _scan_status_line
|
||||
from .images import (
|
||||
THUMBNAIL_ENCODINGS,
|
||||
_BodyTooLarge,
|
||||
_new_thumbnail_stats,
|
||||
_read_json_body,
|
||||
_record_thumbnail_result,
|
||||
_thumbnail_batch_size,
|
||||
_thumbnail_complete_payload,
|
||||
_thumbnail_progress_payload,
|
||||
_worker_threads,
|
||||
)
|
||||
from .job_routes import busy_response
|
||||
from .jobs import THUMBNAIL_GROUP_NAMES, THUMBNAILS_JOB_KIND, JobBusy
|
||||
|
||||
THUMBNAIL_JOB_TITLE = "Thumbnail generation"
|
||||
THUMBNAIL_MODES = ("missing", "rebuild")
|
||||
THUMBNAIL_QUALITIES = tuple(THUMBNAIL_ENCODINGS)
|
||||
MIN_THUMBNAIL_EDGE = 128
|
||||
MAX_THUMBNAIL_EDGE = 2048
|
||||
# The concept's "Preview size limit" default.
|
||||
DEFAULT_THUMBNAIL_EDGE = 512
|
||||
DEFAULT_THUMBNAIL_OPTIONS = {
|
||||
"mode": "missing",
|
||||
"size": DEFAULT_THUMBNAIL_EDGE,
|
||||
"quality": "medium",
|
||||
}
|
||||
_OPTIONS_ERROR = (
|
||||
'mode must be "missing" or "rebuild", quality "low", "medium" or "high", '
|
||||
f"and size a whole number of pixels from {MIN_THUMBNAIL_EDGE} "
|
||||
f"to {MAX_THUMBNAIL_EDGE}"
|
||||
)
|
||||
CANCELLED_ERROR = "Thumbnail generation was cancelled"
|
||||
CANCELLED_BEFORE_WRITING = "Cancelled before any preview was written"
|
||||
NOTHING_TO_PREVIEW = "No images or videos to preview"
|
||||
_MODE_LOG = {"missing": "missing previews", "rebuild": "every preview"}
|
||||
# Outcome rows: what happened to each file's preview.
|
||||
_DETAILS = {
|
||||
"generated": "Preview written",
|
||||
"skipped": "Preview already up to date",
|
||||
}
|
||||
_LOG_PREFIXES = {"generated": "+", "errors": "!"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ThumbnailOptions:
|
||||
"""One run's choices: the mode, the preview's longest edge and quality."""
|
||||
|
||||
mode: str
|
||||
edge: int
|
||||
quality: str
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return (self.edge, self.edge)
|
||||
|
||||
@property
|
||||
def encoding(self):
|
||||
return THUMBNAIL_ENCODINGS[self.quality]
|
||||
|
||||
@property
|
||||
def rebuild(self):
|
||||
return self.mode == "rebuild"
|
||||
|
||||
def scope(self):
|
||||
return {"mode": self.mode, "size": self.edge, "quality": self.quality}
|
||||
|
||||
|
||||
def _edge(value, *, from_query):
|
||||
"""The preview edge as an int, or None when it is not one in range."""
|
||||
if from_query and isinstance(value, str):
|
||||
if not value.isascii() or not value.isdigit() or len(value) > 5:
|
||||
return None
|
||||
value = int(value)
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
return None
|
||||
return value if MIN_THUMBNAIL_EDGE <= value <= MAX_THUMBNAIL_EDGE else None
|
||||
|
||||
|
||||
def thumbnail_options(raw, *, from_query=False):
|
||||
"""``(ThumbnailOptions, None)`` from a body or query, or ``(None, message)``.
|
||||
|
||||
Absent fields take ``DEFAULT_THUMBNAIL_OPTIONS``. Query values are
|
||||
strings, so a query ``size`` may be decimal digits; a JSON ``size`` must
|
||||
be an int.
|
||||
"""
|
||||
if not isinstance(raw, dict) and not hasattr(raw, "get"):
|
||||
return None, _OPTIONS_ERROR
|
||||
mode = raw.get("mode", DEFAULT_THUMBNAIL_OPTIONS["mode"])
|
||||
quality = raw.get("quality", DEFAULT_THUMBNAIL_OPTIONS["quality"])
|
||||
edge = _edge(raw.get("size", DEFAULT_THUMBNAIL_EDGE), from_query=from_query)
|
||||
if mode not in THUMBNAIL_MODES or quality not in THUMBNAIL_QUALITIES:
|
||||
return None, _OPTIONS_ERROR
|
||||
if edge is None:
|
||||
return None, _OPTIONS_ERROR
|
||||
return ThumbnailOptions(mode, edge, quality), None
|
||||
|
||||
|
||||
def _invalid(message):
|
||||
return web.json_response(
|
||||
{"success": False, "code": "invalid-request", "error": message}, status=400
|
||||
)
|
||||
|
||||
|
||||
def _frame(event, data):
|
||||
"""A legacy stream frame as the job publishes it."""
|
||||
return {"event": event, "data": data}
|
||||
|
||||
|
||||
def _finished_line(plan, stats):
|
||||
"""A finished job's status line: what it wrote, or that it had nothing to do."""
|
||||
if not plan["targets"]:
|
||||
if plan["skipped"]:
|
||||
return f"No previews to generate · {plan['skipped']:,} already up to date"
|
||||
return NOTHING_TO_PREVIEW
|
||||
parts = [f"{_plural(stats['generated'], 'preview')} written"]
|
||||
if stats["errors"]:
|
||||
parts.append(_plural(len(stats["errors"]), "error"))
|
||||
if plan["remaining"]:
|
||||
parts.append(f"{plan['remaining']:,} more on the next run")
|
||||
return " · ".join(parts)
|
||||
|
||||
|
||||
class ThumbnailJobsMixin:
|
||||
"""The thumbnail plan, the thumbnail job and the legacy stream on it."""
|
||||
|
||||
async def get_thumbnail_plan(self, request):
|
||||
"""What a thumbnail run would do (read-only; capped like a run)."""
|
||||
options, error = thumbnail_options(request.query, from_query=True)
|
||||
if error:
|
||||
return _invalid(error)
|
||||
try:
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails, output_dirs, rebuild=options.rebuild
|
||||
)
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail plan failed")
|
||||
return web.json_response(
|
||||
{
|
||||
"success": False,
|
||||
"code": None,
|
||||
"error": "The thumbnail plan could not be read",
|
||||
},
|
||||
status=500,
|
||||
)
|
||||
return web.json_response(
|
||||
{
|
||||
"success": True,
|
||||
"targets": len(plan["targets"]),
|
||||
"fresh": plan["fresh"],
|
||||
"remaining": plan["remaining"],
|
||||
"roots": [self._public_path(root) for root in output_dirs],
|
||||
**options.scope(),
|
||||
},
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
async def start_thumbnail_job(self, request):
|
||||
"""Start the thumbnail job (202), or 409 naming the job in the way."""
|
||||
body = {}
|
||||
if request.can_read_body:
|
||||
try:
|
||||
body = await _read_json_body(request)
|
||||
except _BodyTooLarge:
|
||||
return web.json_response(
|
||||
{"success": False, "code": "too-large", "error": "Body too large"},
|
||||
status=413,
|
||||
)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return _invalid(_OPTIONS_ERROR)
|
||||
options, error = thumbnail_options(body)
|
||||
if error:
|
||||
return _invalid(error)
|
||||
try:
|
||||
job = self._start_thumbnail_job(options)
|
||||
except JobBusy as busy:
|
||||
return busy_response(busy)
|
||||
return web.json_response({"success": True, "job": job.summary()}, status=202)
|
||||
|
||||
def _start_thumbnail_job(self, options):
|
||||
"""Register the thumbnail job synchronously; raises JobBusy."""
|
||||
return self._jobs.start(
|
||||
THUMBNAILS_JOB_KIND,
|
||||
THUMBNAIL_JOB_TITLE,
|
||||
admission="guarded",
|
||||
cancellable=True,
|
||||
scope=options.scope(),
|
||||
runner=functools.partial(self._run_thumbnail_job, options=options),
|
||||
groups=THUMBNAIL_GROUP_NAMES,
|
||||
)
|
||||
|
||||
# ── the job ───────────────────────────────────────────────────────
|
||||
|
||||
async def _run_thumbnail_job(self, job, options):
|
||||
try:
|
||||
state = await self._thumbnail_job_body(job, options)
|
||||
except Exception:
|
||||
self.logger.exception("Thumbnail generation failed")
|
||||
job.update(status_line="Thumbnail generation stopped on an internal error.")
|
||||
job.log_line("Thumbnails failed · internal error")
|
||||
job.publish(
|
||||
_frame(
|
||||
"error",
|
||||
{
|
||||
"error": "An internal error occurred",
|
||||
"message": "Thumbnail generation failed. Check server logs.",
|
||||
},
|
||||
)
|
||||
)
|
||||
state = "failed"
|
||||
job.finish(state)
|
||||
|
||||
async def _thumbnail_job_body(self, job, options):
|
||||
"""Plan, then generate in worker batches; returns the final state.
|
||||
|
||||
Every exit sets the job's final status line (I3): what was written,
|
||||
that there was nothing to do, or what a cancel kept.
|
||||
"""
|
||||
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
|
||||
if not output_dirs:
|
||||
message = "ComfyUI output directory not found"
|
||||
job.update(status_line=message)
|
||||
job.log_line(f"Thumbnails failed · {message}")
|
||||
job.publish(_frame("error", {"error": message}))
|
||||
return "failed"
|
||||
folders = _plural(len(output_dirs), "output folder")
|
||||
job.scope["output_folders"] = len(output_dirs)
|
||||
job.update(status_line=f"Looking for media files in {folders}")
|
||||
job.log_line(
|
||||
f"Thumbnails started · {folders} · {_MODE_LOG[options.mode]} · "
|
||||
f"{options.edge} px · {options.quality}"
|
||||
)
|
||||
job.publish(
|
||||
_frame(
|
||||
"status",
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
)
|
||||
plan = await self._run_in_executor(
|
||||
self._plan_gallery_thumbnails,
|
||||
output_dirs,
|
||||
rebuild=options.rebuild,
|
||||
stop=lambda: job.cancelled,
|
||||
)
|
||||
if job.cancelled: # M2: stopped during the walk, before any start frame
|
||||
return self._end_cancelled(job, 0, CANCELLED_BEFORE_WRITING)
|
||||
roots = _scan_roots(output_dirs)
|
||||
self._record_fresh(job, plan, roots)
|
||||
stats, start = await self._generate_thumbnail_targets(job, options, plan, roots)
|
||||
self.invalidate_gallery_cache()
|
||||
generated = _plural(stats["generated"], "preview")
|
||||
if job.cancelled:
|
||||
return self._end_cancelled(
|
||||
job, stats["generated"], f"Cancelled · {generated} written"
|
||||
)
|
||||
payload = _thumbnail_complete_payload(
|
||||
len(plan["targets"]) + plan["skipped"] + plan["remaining"],
|
||||
stats,
|
||||
start,
|
||||
remaining=plan["remaining"],
|
||||
processed=len(plan["targets"]),
|
||||
)
|
||||
job.update(progress=100, status_line=_finished_line(plan, stats))
|
||||
if plan["remaining"]:
|
||||
job.log_line(
|
||||
f"{plan['remaining']:,} more previews are left for the next run"
|
||||
)
|
||||
errors = _plural(len(stats["errors"]), "error")
|
||||
job.log_line(f"Thumbnails finished · {generated} written · {errors}")
|
||||
self.logger.info(payload["message"])
|
||||
job.publish(_frame("complete", payload))
|
||||
return "finished"
|
||||
|
||||
@staticmethod
|
||||
def _end_cancelled(job, generated, status_line):
|
||||
"""Close a cancelled job: its status line, log line and error frame."""
|
||||
job.update(status_line=status_line)
|
||||
job.log_line(f"Thumbnails cancelled · {_plural(generated, 'preview')} written")
|
||||
job.publish(
|
||||
_frame("error", {"error": CANCELLED_ERROR, "message": CANCELLED_ERROR})
|
||||
)
|
||||
return "cancelled"
|
||||
|
||||
def _record_fresh(self, job, plan, roots):
|
||||
"""List the already fresh previews under Skipped (no log line each)."""
|
||||
rows = []
|
||||
for src in plan["fresh_files"]:
|
||||
path, label, index = _scan_location(src, roots)
|
||||
rows.append(
|
||||
{
|
||||
"path": path,
|
||||
"root": label,
|
||||
"root_index": index,
|
||||
"detail": _DETAILS["skipped"],
|
||||
}
|
||||
)
|
||||
job.record_counted("skipped", plan["skipped"], rows)
|
||||
if plan["skipped"]:
|
||||
job.log_line(f"= {plan['skipped']:,} previews already up to date")
|
||||
|
||||
async def _generate_thumbnail_targets(self, job, options, plan, roots):
|
||||
"""Generate ``plan``'s targets in worker batches, publishing each file."""
|
||||
targets = plan["targets"]
|
||||
total = len(targets)
|
||||
video_count = sum(1 for _, _, is_video in targets if is_video)
|
||||
message = (
|
||||
f"Found {total - video_count} images and {video_count} videos to process"
|
||||
)
|
||||
if plan["skipped"]:
|
||||
message += f" ({plan['skipped']} already have thumbnails)"
|
||||
job.update(total=total)
|
||||
if total: # never "Processing file 0 of 0" (I3)
|
||||
job.update(status_line=_scan_status_line(0, total, 0, 0))
|
||||
job.publish(
|
||||
_frame(
|
||||
"start",
|
||||
{
|
||||
"total_images": total,
|
||||
"skipped": plan["skipped"],
|
||||
"remaining": plan["remaining"],
|
||||
"phase": "processing",
|
||||
"image_count": total - video_count,
|
||||
"video_count": video_count,
|
||||
"message": message,
|
||||
},
|
||||
)
|
||||
)
|
||||
stats = _new_thumbnail_stats()
|
||||
stats["skipped"] = plan["skipped"]
|
||||
start = time.monotonic()
|
||||
batch_size = _thumbnail_batch_size(_worker_threads())
|
||||
generate = {
|
||||
"encoding": options.encoding,
|
||||
"rebuild": options.rebuild,
|
||||
"stop": lambda: job.cancelled,
|
||||
}
|
||||
index = 0
|
||||
batches = 0
|
||||
for batch_start in range(0, total, batch_size):
|
||||
if job.cancelled:
|
||||
break
|
||||
batch = targets[batch_start : batch_start + batch_size]
|
||||
results = await self._run_in_executor(
|
||||
self._generate_batch, batch, options.size, **generate
|
||||
)
|
||||
for (src, _dst, _video), result in zip(batch, results):
|
||||
if result["action"] == "cancelled":
|
||||
continue
|
||||
index += 1
|
||||
_record_thumbnail_result(stats, result)
|
||||
self._record_thumbnail_outcome(job, src, result, roots)
|
||||
job.update(processed=index, progress=index * 100 // total)
|
||||
if result["action"] == "error":
|
||||
job.publish(
|
||||
_frame(
|
||||
"file_error",
|
||||
{"file": result["file"], "error": result["error"]},
|
||||
)
|
||||
)
|
||||
job.publish(
|
||||
_frame(
|
||||
"progress",
|
||||
_thumbnail_progress_payload(index, total, stats, result, start),
|
||||
)
|
||||
)
|
||||
batches += 1
|
||||
job.update(
|
||||
status_line=_scan_status_line(
|
||||
index, total, batches, time.monotonic() - start
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0) # let other requests run between batches
|
||||
return stats, start
|
||||
|
||||
@staticmethod
|
||||
def _record_thumbnail_outcome(job, src, result, roots):
|
||||
path, label, index = _scan_location(src, roots)
|
||||
where = f"{label}/{path}" if label else path
|
||||
if result["action"] == "error":
|
||||
group, detail = "errors", result.get("error") or "failed"
|
||||
elif result["action"] == "skipped":
|
||||
group, detail = "skipped", _DETAILS["skipped"]
|
||||
else:
|
||||
group, detail = "generated", _DETAILS["generated"]
|
||||
row = {"path": path, "root": label, "root_index": index, "detail": detail}
|
||||
if group == "skipped":
|
||||
job.record_counted(group, 1, [row])
|
||||
return
|
||||
job.record(group, row, f"{_LOG_PREFIXES[group]} {where} · {detail}")
|
||||
|
||||
# ── the legacy gallery stream ─────────────────────────────────────
|
||||
|
||||
async def generate_thumbnails_with_progress(self, request):
|
||||
"""The gallery's thumbnail stream (POST, server-sent events).
|
||||
|
||||
Starts the thumbnail job with the gallery's quality preset as the
|
||||
preview size (missing previews, medium encoding, as before), or
|
||||
attaches to the running thumbnail job, and streams its frames. A
|
||||
legacy hold (the blocking generate or clear) or a record-changing
|
||||
job refuses it with the stream's error frame. The job goes on when
|
||||
the client goes away.
|
||||
"""
|
||||
thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium"))
|
||||
response = web.StreamResponse(
|
||||
status=200,
|
||||
headers={
|
||||
"Content-Type": "text/event-stream",
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
},
|
||||
)
|
||||
await response.prepare(request)
|
||||
# M3: read the registry after the last await, so a job that started
|
||||
# while the headers went out is attached to, not refused.
|
||||
job, started, busy = self._legacy_stream_job(thumbnail_size)
|
||||
if busy is not None:
|
||||
message = f"{busy.title} already running"
|
||||
await self._emit_sse(
|
||||
response, "error", {"error": message, "message": message}
|
||||
)
|
||||
return response
|
||||
# No await between the start and the subscription: the lead frame is
|
||||
# sent after it, so the stream misses none of the runner's frames.
|
||||
events = job.events(lead=_frame("job", {"id": job.id, "started": started}))
|
||||
try:
|
||||
async for payload in events:
|
||||
frame = (
|
||||
f"event: {payload['event']}\ndata: {json.dumps(payload['data'])}"
|
||||
)
|
||||
await response.write(f"{frame}\n\n".encode("utf-8"))
|
||||
except Exception as exc: # a write failed: the client went away
|
||||
self.logger.info(
|
||||
f"Thumbnail progress client disconnected ({exc}); "
|
||||
"the thumbnails continue in the background"
|
||||
)
|
||||
finally:
|
||||
# Also when aiohttp cancels the handler on disconnect: the job is
|
||||
# its own task and goes on.
|
||||
await events.aclose()
|
||||
return response
|
||||
|
||||
def _legacy_stream_job(self, thumbnail_size):
|
||||
"""``(job, started, busy)`` for the legacy stream, synchronously.
|
||||
|
||||
Attaches to the running cancellable thumbnail job, or starts one with
|
||||
the gallery's preset as the size (missing previews, medium encoding,
|
||||
as the gallery always wrote), or names the job in the way.
|
||||
"""
|
||||
running = self._jobs.active(THUMBNAILS_JOB_KIND)
|
||||
if running is not None and running.cancellable:
|
||||
return running, False, None
|
||||
busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded")
|
||||
if busy is not None:
|
||||
return None, False, busy
|
||||
# blocker() said free and nothing awaited since, so the start is admitted.
|
||||
options = ThumbnailOptions("missing", thumbnail_size[0], "medium")
|
||||
return self._start_thumbnail_job(options), True, None
|
||||
@@ -775,3 +775,131 @@ test('add source comparison detects mutations', async ({ page, request }) => {
|
||||
await undoStyle();
|
||||
expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]);
|
||||
});
|
||||
|
||||
/**
|
||||
* Runs in the page: the open "Thumbnail maintenance" dialog (the concept's `thumbnail-review`, app.js 1433–1437): its
|
||||
* title, the body's tree, each `.field` (label, control, options, selection, limits), the `p.small` line and the
|
||||
* actions, with the concept's geometry and tokens. The concept has no `p.small`, so a probe one stands in for its style.
|
||||
*/
|
||||
function measureThumbnailReview() {
|
||||
const dialog = document.querySelector('dialog#modal[open]');
|
||||
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
|
||||
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
|
||||
const text = node => node?.textContent.trim() ?? null;
|
||||
const body = dialog.querySelector('#modal-body');
|
||||
const fields = [...body.querySelectorAll(':scope > .field')].map(field => {
|
||||
const label = field.querySelector(':scope > label');
|
||||
const control = field.querySelector(':scope > select, :scope > input');
|
||||
return {
|
||||
tree: [...field.children].map(name),
|
||||
...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom'),
|
||||
label: { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
|
||||
control: { id: control.id, tag: control.localName, type: control.type,
|
||||
options: control.options ? [...control.options].map(option => option.textContent) : null,
|
||||
selected: control.options ? control.selectedIndex : null,
|
||||
value: control.localName === 'input' ? control.value : null,
|
||||
min: control.getAttribute('min'), max: control.getAttribute('max'),
|
||||
...css(control, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
|
||||
};
|
||||
});
|
||||
let line = body.querySelector(':scope > p.small');
|
||||
const probe = !line;
|
||||
if (probe) { line = document.createElement('p'); line.className = 'small'; body.append(line); }
|
||||
const plan = { text: probe ? null : text(line), role: probe ? null : line.getAttribute('role'),
|
||||
...css(line, 'fontSize', 'color', 'marginTop', 'marginBottom') };
|
||||
if (probe) line.remove();
|
||||
const actions = body.querySelector(':scope > .dialog-actions');
|
||||
const rect = dialog.getBoundingClientRect();
|
||||
return {
|
||||
title: text(dialog.querySelector('#modal-title')),
|
||||
tree: [...body.children].map(name),
|
||||
fields,
|
||||
plan,
|
||||
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
|
||||
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
|
||||
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
|
||||
width: Math.round(rect.width),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* D99: the planned counts line (`p.small[role=status]`) before the actions, from the server's plan for the chosen
|
||||
* options, and the primary "Start" (the concept's "Simulate thumbnail job").
|
||||
*/
|
||||
function mapThumbnailConcept(reference, plan) {
|
||||
const [cancel, start] = reference.actions.buttons;
|
||||
const tree = [...reference.tree];
|
||||
tree.splice(tree.indexOf('div.dialog-actions'), 0, 'p.small');
|
||||
return { ...reference, tree,
|
||||
plan: { ...reference.plan, role: 'status',
|
||||
text: `${number(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ${number(plan.fresh)} already up to date.` },
|
||||
actions: { ...reference.actions, buttons: [cancel, { ...start, text: 'Start' }] } };
|
||||
}
|
||||
|
||||
async function conceptThumbnailReview(page, theme) {
|
||||
await conceptSources(page, theme, { notice: false });
|
||||
const row = page.locator('#main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
|
||||
await row.getByRole('button', { name: 'Review options', exact: true }).click();
|
||||
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureThumbnailReview);
|
||||
}
|
||||
|
||||
/** Production's dialog from Sources' Review options or Overview's Review thumbnails, once its plan line answered. */
|
||||
async function productionThumbnailReview(page, theme, entry) {
|
||||
if (entry === 'sources') {
|
||||
await productionSources(page, theme, { notice: false });
|
||||
const row = page.locator('#workspace-main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
|
||||
await row.getByRole('button', { name: 'Review options', exact: true }).click();
|
||||
} else {
|
||||
await openProduction(page, 'system/overview', theme);
|
||||
await page.locator('#workspace-main').getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
}
|
||||
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await expect(page.locator('dialog#modal[open] #modal-body > p.small')).toHaveText(/ already up to date\.$/);
|
||||
await page.mouse.move(0, 0);
|
||||
return page.evaluate(measureThumbnailReview);
|
||||
}
|
||||
|
||||
const thumbnailPlan = request => serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
|
||||
|
||||
for (const entry of ['sources', 'overview']) for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
test(`thumbnail dialog matches the concept from ${entry} ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
|
||||
await page.setViewportSize({ width, height });
|
||||
await watchedSources(request);
|
||||
const plan = await thumbnailPlan(request);
|
||||
const concept = await conceptThumbnailReview(page, theme);
|
||||
await page.screenshot({ path: testInfo.outputPath(`concept-thumbnail-review-${theme}-${width}.png`) });
|
||||
const production = await productionThumbnailReview(page, theme, entry);
|
||||
await page.screenshot({ path: testInfo.outputPath(`production-thumbnail-review-${entry}-${theme}-${width}.png`) });
|
||||
expect(diffInventory(mapThumbnailConcept(concept, plan), production)).toEqual([]);
|
||||
});
|
||||
}
|
||||
|
||||
test('thumbnail dialog comparison detects mutations', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
await watchedSources(request);
|
||||
const plan = await thumbnailPlan(request);
|
||||
const concept = await conceptThumbnailReview(page, 'dark');
|
||||
const expected = mapThumbnailConcept(concept, plan);
|
||||
const production = await productionThumbnailReview(page, 'dark', 'sources');
|
||||
expect(diffInventory(expected, production)).toEqual([]);
|
||||
// Unmapped, the concept differs only where D99 is recorded.
|
||||
expect(diffInventory(concept, production).map(item => item.path).sort())
|
||||
.toEqual(['actions.buttons.1.text', 'plan.role', 'plan.text', 'tree.3', 'tree.4', 'tree.length']);
|
||||
const paths = async () => diffInventory(expected, await page.evaluate(measureThumbnailReview)).map(item => item.path);
|
||||
// A changed option label, a swapped option order, a swapped action order and a changed style.
|
||||
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium'; });
|
||||
expect(await paths()).toEqual(['fields.1.control.options.0']);
|
||||
await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium · optimized'; });
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.0.control.options.0', 'fields.0.control.options.1']));
|
||||
await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); select.selectedIndex = 0; });
|
||||
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
|
||||
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
|
||||
const undoStyle = await mutate(page, { css: 'dialog#modal #thumb-limit { border-color: rgb(255, 0, 0); } dialog#modal #modal-body > p.small { color: rgb(255, 0, 0); }' });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['fields.2.control.borderTopColor', 'plan.color']));
|
||||
await undoStyle();
|
||||
expect(diffInventory(expected, await page.evaluate(measureThumbnailReview))).toEqual([]);
|
||||
});
|
||||
|
||||
@@ -54,8 +54,8 @@ async function step(request, id, data) {
|
||||
await expect(await request.post(`/__test__/jobs/${id}/step`, { data })).toBeOK();
|
||||
}
|
||||
|
||||
async function finish(request, id, state = 'finished') {
|
||||
const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state } });
|
||||
async function finish(request, id, state = 'finished', extra = {}) {
|
||||
const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state, ...extra } });
|
||||
await expect(response).toBeOK();
|
||||
return (await response.json()).job;
|
||||
}
|
||||
@@ -1008,3 +1008,96 @@ test('the report download names the kind', async ({ page, request }) => {
|
||||
expect(report.outcomes.tagged.rows.map(row => [row.path, row.detail])).toEqual([['sub/a.png', 'Added 1 tag']]);
|
||||
await expect(toast(page)).toHaveText('Job report downloaded.');
|
||||
});
|
||||
|
||||
// D111 (System S5): each kind's own descriptor. A thumbnail job counts Generated (green), Skipped and Errors (red),
|
||||
// its pill says "Generating thumbnails" and its toasts "Thumbnails finished · N generated · M errors".
|
||||
const THUMB_JOB = { kind: 'thumbnails', total: 10, output_folders: 2 };
|
||||
|
||||
test('a thumbnail job shows Generated Skipped Errors, its pill verb and its toasts', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request, THUMB_JOB);
|
||||
await step(request, job.id, { processed: 4, records: [
|
||||
{ group: 'generated', path: 'sub/a.png', root: 'output', detail: 'Preview written' },
|
||||
{ group: 'skipped', path: 'b.png', root: 'input', detail: 'Preview already up to date' },
|
||||
{ group: 'errors', path: 'broken.png', root: 'output', detail: 'Not a valid image file' },
|
||||
] });
|
||||
await page.goto(`${workspace}#/images`);
|
||||
await expect(pill(page)).toHaveText('Generating thumbnails · 40% · 4 / 10');
|
||||
await page.goto(`${workspace}#/activity`);
|
||||
await expect(items(page).first().locator('p')).toHaveText(/^Started \d\d:\d\d · 2 output folders · closing this view does not cancel it$/);
|
||||
await expect(items(page).first().locator('progress')).toHaveAttribute('value', '40');
|
||||
await openJob(page, job.id);
|
||||
await expect(heading(page).locator('h1')).toHaveText('Thumbnail generation');
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel job']);
|
||||
expect(await tileValues(page)).toEqual([['stat added', 'Generated', '1'], ['stat', 'Skipped', '1'], ['stat bad', 'Errors', '1']]);
|
||||
await expect(outcomeButtons(page)).toHaveText(['Generated1', 'Errors1', 'Skipped1']);
|
||||
await expect(fileRows(page).first().locator('.path')).toHaveText('sub/a.png');
|
||||
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('button');
|
||||
await outcomeButtons(page).filter({ hasText: 'Errors' }).click();
|
||||
expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('div');
|
||||
await expect(scanLog(page).locator('summary')).toHaveText('● Job log — raw detail, click to show');
|
||||
|
||||
await finish(request, job.id);
|
||||
await expect(toast(page)).toHaveText('Thumbnails finished · 1 generated · 1 error');
|
||||
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d · 1 generated · 1 error$/);
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
|
||||
});
|
||||
|
||||
test('a held legacy job and the health check show no tiles, no Results and no made-up percentage', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const response = await request.post('/__test__/jobs/hold',
|
||||
{ data: { kind: 'thumbnails', title: 'Thumbnail clear (gallery page)', admission: 'guarded' } });
|
||||
await expect(response).toBeOK();
|
||||
const held = (await response.json()).job;
|
||||
await page.goto(`${workspace}#/images`);
|
||||
// D112: the pill names the held operation and shows no percentage; its bar is indeterminate, not an empty fill
|
||||
// (S5 review M5), and stays hidden from assistive tech, which reads the label.
|
||||
await expect(pill(page)).toHaveText('Thumbnail clear (gallery page)');
|
||||
await expect(pill(page)).toHaveAttribute('aria-label', 'Thumbnail clear (gallery page)');
|
||||
await expect(pill(page).locator('.activity-pill-bar')).toHaveClass('activity-pill-bar indeterminate');
|
||||
await expect(pill(page).locator('.activity-pill-bar')).toHaveAttribute('aria-hidden', 'true');
|
||||
expect(await pill(page).locator('.activity-pill-bar i').evaluate(node => node.style.width)).toBe('');
|
||||
await page.goto(`${workspace}#/activity`);
|
||||
const item = items(page).filter({ has: page.getByRole('link', { name: /Thumbnail clear \(gallery page\)$/ }) });
|
||||
await expect(item.locator('p')).toHaveText(/^Started \d\d:\d\d · closing this view does not cancel it$/);
|
||||
await expect(item.locator('progress')).toHaveCount(1);
|
||||
await expect(item.locator('progress')).not.toHaveAttribute('value');
|
||||
await expect(item.getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
|
||||
await item.getByRole('link').click();
|
||||
await expect(heading(page).locator('h1')).toHaveText('Thumbnail clear (gallery page)');
|
||||
await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value');
|
||||
await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0);
|
||||
await expect(resultsSection(page)).toHaveCount(0);
|
||||
await expect(scanLog(page)).toHaveCount(1);
|
||||
await finish(request, held.id);
|
||||
await expect(toast(page)).toHaveText('Thumbnail clear (gallery page) finished.');
|
||||
await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d$/);
|
||||
await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100');
|
||||
|
||||
// The health check (a read job of its own kind): its status line, no tiles, no Cancel.
|
||||
const health = await startJob(request, { kind: 'health', total: 0 });
|
||||
await openJob(page, health.id);
|
||||
await expect(heading(page).locator('h1')).toHaveText('Health check');
|
||||
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Checking the database and image links');
|
||||
await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value');
|
||||
await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0);
|
||||
await expect(resultsSection(page)).toHaveCount(0);
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
|
||||
await expect(pill(page)).toHaveText('Health check');
|
||||
await finish(request, health.id);
|
||||
await expect(toast(page)).toHaveText('Health check finished.');
|
||||
});
|
||||
|
||||
// S5 review I3: a thumbnail job with nothing to generate ends on the server's final line ("No previews to generate · N
|
||||
// already up to date"), never on a stale "Processing file 0 of 0" under the Completed chip.
|
||||
test('a finished thumbnail job with nothing to do says so', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request, { ...THUMB_JOB, total: 0 });
|
||||
await step(request, job.id, { total: 0, processed: 0 }); // the stale line an older server left
|
||||
await openJob(page, job.id);
|
||||
const line = progressSection(page).locator('.panel-heading > span.small');
|
||||
await finish(request, job.id, 'finished', { status_line: 'No previews to generate · 24 already up to date' });
|
||||
await expect(line).toHaveText('No previews to generate · 24 already up to date');
|
||||
await expect(page.getByText('Processing file 0 of 0')).toHaveCount(0);
|
||||
await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100');
|
||||
});
|
||||
|
||||
@@ -304,16 +304,151 @@ test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page
|
||||
await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.');
|
||||
});
|
||||
|
||||
test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => {
|
||||
const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan';
|
||||
const THUMBNAIL_JOB_URL = '/prompt_manager/jobs/thumbnails';
|
||||
const planCount = value => value.toLocaleString('en-US');
|
||||
|
||||
/** Open "Thumbnail maintenance" from Sources' Thumbnail generation → Review options. */
|
||||
async function openThumbnailReview(page) {
|
||||
const layout = await openSources(page);
|
||||
const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
|
||||
await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.');
|
||||
const popup = page.waitForEvent('popup');
|
||||
await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click();
|
||||
const gallery = await popup;
|
||||
expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery');
|
||||
await gallery.close();
|
||||
await expect(statusOf(page)).toHaveText('No unsaved changes.');
|
||||
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
return dialog(page);
|
||||
}
|
||||
|
||||
/** The thumbnail requests this page sends: plan reads (their query) and job starts (their body). */
|
||||
function thumbnailRequests(page) {
|
||||
const seen = { plans: [], starts: [] };
|
||||
page.on('request', item => {
|
||||
const url = new URL(item.url());
|
||||
if (url.pathname === THUMBNAIL_PLAN_URL) seen.plans.push(Object.fromEntries(url.searchParams));
|
||||
if (item.method() === 'POST' && url.pathname === THUMBNAIL_JOB_URL) seen.starts.push(item.postDataJSON());
|
||||
});
|
||||
return seen;
|
||||
}
|
||||
|
||||
const missingLine = plan => `${planCount(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ` +
|
||||
`${planCount(plan.fresh)} already up to date.`;
|
||||
|
||||
test('Review options shows the plan counts for the chosen scope/quality/size before Start', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK();
|
||||
// Image loading quality (Appearance, Q3) preselects the dialog's quality.
|
||||
await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('gallerySettings', JSON.stringify({ imageQuality: 'high' })); });
|
||||
const seen = thumbnailRequests(page);
|
||||
const plan = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=missing&size=512&quality=high`);
|
||||
expect(plan.targets).toBe(3);
|
||||
const review = await openThumbnailReview(page);
|
||||
await expect(review.locator('#modal-body > *')).toHaveCount(5);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
|
||||
await expect(review.locator('#thumb-mode option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)']);
|
||||
await expect(review.locator('#thumb-quality option')).toHaveText(['Medium · optimized', 'High · original', 'Low · fastest']);
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
|
||||
await expect(review.locator('#thumb-quality')).toHaveValue('high');
|
||||
await expect(review.locator('#thumb-limit')).toHaveValue('512');
|
||||
await expect(review.locator('#thumb-limit')).toHaveAttribute('min', '128');
|
||||
await expect(review.locator('#thumb-limit')).toHaveAttribute('max', '2048');
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
await expect(line).toHaveAttribute('role', 'status');
|
||||
await expect(line).toHaveText(missingLine(plan));
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'high' }]);
|
||||
await expect(review.locator('.dialog-actions > button')).toHaveText(['Cancel', 'Start']);
|
||||
// Reviewing starts nothing; Cancel closes.
|
||||
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
expect(seen.starts).toEqual([]);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
});
|
||||
|
||||
test('size outside 128–2048 reports validity and requests nothing', async ({ page, request }) => {
|
||||
const seen = thumbnailRequests(page);
|
||||
const review = await openThumbnailReview(page);
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
const size = review.locator('#thumb-limit');
|
||||
for (const value of ['4096', '64', '']) {
|
||||
await size.fill(value);
|
||||
await expect(size).toHaveAttribute('aria-invalid', 'true');
|
||||
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
|
||||
expect(await size.evaluate(input => input.validity.valid)).toBe(false);
|
||||
// Start keeps the dialog open and sends nothing; neither does a scope change.
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.');
|
||||
await review.locator('#thumb-mode').selectOption('missing');
|
||||
}
|
||||
expect(seen.starts).toEqual([]);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }]);
|
||||
await size.fill('1024');
|
||||
await expect(size).not.toHaveAttribute('aria-invalid');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
|
||||
});
|
||||
|
||||
test('Start closes the dialog, toasts, and opens the job in Activity', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
|
||||
const seen = thumbnailRequests(page);
|
||||
const review = await openThumbnailReview(page);
|
||||
await review.locator('#thumb-quality').selectOption('low');
|
||||
await review.locator('#thumb-limit').fill('256');
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
// The toast says it started (or, for a run this quick, that it already finished); Activity shows its progress.
|
||||
await expect(toast(page)).toHaveText(/^(Thumbnail generation started\.|Thumbnails finished · 2 generated · 0 errors)$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'missing', size: 256, quality: 'low' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
|
||||
expect(job).toMatchObject({ kind: 'thumbnails', title: 'Thumbnail generation', scope: { mode: 'missing', size: 256, quality: 'low' } });
|
||||
await expect(page.locator('#workspace-main h1')).toHaveText('Thumbnail generation');
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('Rebuild all is explicit and the plan line says every preview is regenerated', async ({ page, request }) => {
|
||||
const seen = thumbnailRequests(page);
|
||||
const rebuild = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=rebuild&size=512&quality=medium`);
|
||||
expect(rebuild.targets).toBeGreaterThan(0);
|
||||
const review = await openThumbnailReview(page);
|
||||
// Missing thumbnails only is the default; rebuilding is the explicit second option.
|
||||
await expect(review.locator('#thumb-mode')).toHaveValue('missing');
|
||||
const line = review.locator('#modal-body > p.small');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
|
||||
expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }, { mode: 'rebuild', size: '512', quality: 'medium' }]);
|
||||
// Back and forth reads each scope once.
|
||||
await review.locator('#thumb-mode').selectOption('missing');
|
||||
await expect(line).toHaveText(/ already up to date\.$/);
|
||||
await review.locator('#thumb-mode').selectOption('rebuild');
|
||||
await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`);
|
||||
expect(seen.plans).toHaveLength(2);
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(seen.starts).toEqual([{ mode: 'rebuild', size: 512, quality: 'medium' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('a plan that cannot be read says so; Start still starts', async ({ page }) => {
|
||||
await page.route(`**${THUMBNAIL_PLAN_URL}?*`, route => route.fulfill({ status: 500, json: { success: false, error: 'disk' } }));
|
||||
const review = await openThumbnailReview(page);
|
||||
await expect(review.locator('#modal-body > p.small')).toHaveText('The previews could not be counted.');
|
||||
await expect(review.getByRole('button', { name: 'Start', exact: true })).not.toHaveAttribute('aria-disabled');
|
||||
await review.getByRole('button', { name: 'Cancel', exact: true }).click();
|
||||
});
|
||||
|
||||
test('a refused start shows the busy link toast', async ({ page, request }) => {
|
||||
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
|
||||
const review = await openThumbnailReview(page);
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
|
||||
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
|
||||
await expect(page).toHaveURL(`${workspace}#/system/sources`);
|
||||
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['scan']);
|
||||
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
|
||||
});
|
||||
|
||||
test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => {
|
||||
|
||||
@@ -140,9 +140,32 @@ test('thumbnail row reflects the status endpoint', async ({ page, request }) =>
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
|
||||
await expect(row.locator('h3')).toHaveText('3 missing thumbnails');
|
||||
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources.
|
||||
});
|
||||
|
||||
test('Overview Review thumbnails opens the same dialog', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK();
|
||||
const plan = await serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium');
|
||||
const starts = [];
|
||||
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/thumbnails')) starts.push(item.postDataJSON()); });
|
||||
const layout = await openOverview(page);
|
||||
const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) });
|
||||
await expect(row.locator('h3')).toHaveText('2 missing thumbnails');
|
||||
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
await expect(page).toHaveURL(`${workspace}#/system/sources`);
|
||||
const review = page.locator('dialog#modal[open]');
|
||||
await expect(review.locator('#modal-title')).toHaveText('Thumbnail maintenance');
|
||||
await expect(page).toHaveURL(`${workspace}#/system/overview`);
|
||||
await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']);
|
||||
await expect(review.locator('#modal-body > p.small')).toHaveText(`2 previews to generate · ${number(plan.fresh)} already up to date.`);
|
||||
// Escape closes it and focus returns to the row's button.
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(review).toHaveCount(0);
|
||||
await expect(row.getByRole('button', { name: 'Review thumbnails', exact: true })).toBeFocused();
|
||||
await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click();
|
||||
await review.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
expect(starts).toEqual([{ mode: 'missing', size: 512, quality: 'medium' }]);
|
||||
const id = page.url().split('/').pop();
|
||||
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished');
|
||||
});
|
||||
|
||||
test('the thumbnail row paints when its own status read answers', async ({ page, request }) => {
|
||||
|
||||
@@ -5,18 +5,23 @@ reading files, so a test sets its progress and counters exactly:
|
||||
|
||||
- ``POST /__test__/jobs/start`` starts one (``kind``, ``title``, ``total``,
|
||||
``output_folders``; for ``kind: "image-tags"`` the image tagging job's
|
||||
groups and its ``{mode, images}`` scope instead) and returns its summary,
|
||||
or 409 while one runs;
|
||||
groups and its ``{mode, images}`` scope instead; for ``kind:
|
||||
"thumbnails"`` the thumbnail job's groups, ``guarded`` admission and its
|
||||
``{mode, size, quality, output_folders}`` scope; for ``kind: "health"`` the
|
||||
health check's: no groups, ``read``, not cancellable, with its status
|
||||
line) and returns its summary, or 409 while one runs;
|
||||
- ``POST /__test__/jobs/{id}/step`` records ``records`` (``{group, path,
|
||||
root, root_index, detail}`` file outcomes, counted, listed and logged as the scan
|
||||
records them) and ``bulk`` (``{group, count, root}``: ``count`` numbered
|
||||
rows), sets ``processed`` (and ``total``) and ``counters``, then publishes
|
||||
the progress to every subscriber;
|
||||
- ``POST /__test__/jobs/{id}/finish`` ends it as ``finished``, ``cancelled``
|
||||
or ``failed``; a cancel through the real route ends it ``cancelled``;
|
||||
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``):
|
||||
a running, non-cancellable ``records`` job, as a legacy restore or
|
||||
maintenance request takes, until ``finish`` ends it;
|
||||
or ``failed``, with the runner's final ``status_line`` when given; a cancel
|
||||
through the real route ends it ``cancelled``;
|
||||
- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``,
|
||||
``admission``, default ``records``): a running, non-cancellable job with no
|
||||
outcome groups, as a legacy restore, maintenance or thumbnail clear request
|
||||
takes, until ``finish`` ends it;
|
||||
- ``POST /__test__/jobs/restart`` acts out a ComfyUI restart: the open job
|
||||
streams drop without a final event and the registry starts empty.
|
||||
"""
|
||||
@@ -35,6 +40,9 @@ PREFIXES = {"added": "+", "known": "=", "review": "?", "errors": "!"}
|
||||
PREFIXES["no_metadata"] = "-"
|
||||
# The image tagging job's groups (#249 PR D), logged with the runner's prefixes.
|
||||
PREFIXES.update({"tagged": "+", "unchanged": "=", "skipped": "-"})
|
||||
# The thumbnail job's (#249 System S5).
|
||||
PREFIXES["generated"] = "+"
|
||||
HEALTH_STATUS_LINE = "Checking the database and image links"
|
||||
ROOT_INDEXES = {"output": 0, "input": 1}
|
||||
|
||||
|
||||
@@ -80,9 +88,12 @@ def outcome_records(data):
|
||||
|
||||
def register_activity_commands(app, routes, api):
|
||||
from py.api.jobs import (
|
||||
HEALTH_JOB_KIND,
|
||||
IMAGE_TAG_GROUP_NAMES,
|
||||
IMAGE_TAG_JOB_KIND,
|
||||
SCAN_GROUP_NAMES,
|
||||
THUMBNAIL_GROUP_NAMES,
|
||||
THUMBNAILS_JOB_KIND,
|
||||
JobBusy,
|
||||
JobRegistry,
|
||||
)
|
||||
@@ -138,7 +149,21 @@ def register_activity_commands(app, routes, api):
|
||||
data = await body(request)
|
||||
ending = asyncio.get_running_loop().create_future()
|
||||
kind = str(data.get("kind", "scan"))
|
||||
if kind == IMAGE_TAG_JOB_KIND:
|
||||
admission, cancellable = "records", True
|
||||
if kind == THUMBNAILS_JOB_KIND:
|
||||
title = "Thumbnail generation"
|
||||
scope = {
|
||||
"mode": str(data.get("mode", "missing")),
|
||||
"size": count(data.get("size"), 512),
|
||||
"quality": str(data.get("quality", "medium")),
|
||||
"output_folders": count(data.get("output_folders"), 2),
|
||||
}
|
||||
groups = THUMBNAIL_GROUP_NAMES
|
||||
admission = "guarded"
|
||||
elif kind == HEALTH_JOB_KIND:
|
||||
title, scope, groups = "Health check", {}, ()
|
||||
admission, cancellable = "read", False
|
||||
elif kind == IMAGE_TAG_JOB_KIND:
|
||||
title = "Add tags to images"
|
||||
scope = {
|
||||
"mode": str(data.get("mode", "add")),
|
||||
@@ -162,7 +187,8 @@ def register_activity_commands(app, routes, api):
|
||||
job = api._jobs.start(
|
||||
kind,
|
||||
str(data.get("title", title)),
|
||||
record_changing=True,
|
||||
admission=admission,
|
||||
cancellable=cancellable,
|
||||
scope=scope,
|
||||
runner=fake_runner(ending),
|
||||
groups=groups,
|
||||
@@ -170,6 +196,8 @@ def register_activity_commands(app, routes, api):
|
||||
except JobBusy:
|
||||
raise web.HTTPConflict()
|
||||
job.update(total=count(data.get("total")))
|
||||
if kind == HEALTH_JOB_KIND:
|
||||
job.update(status_line=HEALTH_STATUS_LINE)
|
||||
endings[job.id] = ending
|
||||
return web.json_response({"success": True, "job": job.summary()})
|
||||
|
||||
@@ -184,7 +212,7 @@ def register_activity_commands(app, routes, api):
|
||||
async with api._jobs.hold(
|
||||
str(data.get("kind", "restore")),
|
||||
str(data.get("title", "Database restore (admin page)")),
|
||||
admission="records",
|
||||
admission=str(data.get("admission", "records")),
|
||||
) as job:
|
||||
entered.set_result(job)
|
||||
state = await ending
|
||||
@@ -232,9 +260,12 @@ def register_activity_commands(app, routes, api):
|
||||
@routes.post("/__test__/jobs/{job_id}/finish")
|
||||
async def finish_job(request):
|
||||
job = job_or_404(request)
|
||||
state = (await body(request)).get("state", "finished")
|
||||
data = await body(request)
|
||||
state = data.get("state", "finished")
|
||||
if state not in FINAL_STATES:
|
||||
raise web.HTTPBadRequest()
|
||||
if isinstance(data.get("status_line"), str): # the runner's final line
|
||||
job.update(status_line=data["status_line"])
|
||||
ending = endings[job.id]
|
||||
if not ending.done():
|
||||
ending.set_result(state)
|
||||
|
||||
@@ -110,3 +110,123 @@ test('saving gallery settings retains the remembered thumbnail dismissal', () =>
|
||||
f.manager.saveSettings();
|
||||
assert.equal(JSON.parse(f.storage.get('gallerySettings')).thumbnailPromptDismissed, 'remembered-state');
|
||||
});
|
||||
|
||||
// #249 System S5: the stream's work is now an Activity job that outlives the page's connection, so the gallery's
|
||||
// Cancel asks the server to stop the job. The stream's first frame, `job {id, started}`, names the job; the gallery
|
||||
// cancels by that id, only a job this stream started, and a Cancel before the id waits for it (review I1).
|
||||
const JOB = 'b'.repeat(32);
|
||||
|
||||
function streamFixture({ modal = false, postFails = false } = {}) {
|
||||
const posts = [];
|
||||
const warnings = [];
|
||||
const gets = [];
|
||||
const listeners = {};
|
||||
const source = { closed: 0, readyState: 1, CLOSED: 2, url: null,
|
||||
addEventListener(type, handler) { (listeners[type] ||= []).push(handler); },
|
||||
close() { this.closed += 1; } };
|
||||
const element = () => ({ classList: { add() {}, remove() {} }, style: {}, textContent: '', handlers: {},
|
||||
addEventListener(type, handler) { this.handlers[type] = handler; },
|
||||
removeEventListener(type, handler) { if (this.handlers[type] === handler) delete this.handlers[type]; },
|
||||
click() { this.handlers.click?.(); } });
|
||||
const elements = {};
|
||||
const document = { addEventListener() {}, getElementById: id => elements[id] ||= element(),
|
||||
createElement: () => element(), body: { appendChild() {}, removeChild() {} } };
|
||||
const context = vm.createContext({ document, window: {},
|
||||
console: { debug() {}, warn: (...args) => warnings.push(args), error() {} },
|
||||
setTimeout() {}, localStorage: { setItem() {} },
|
||||
SseStream: { connect: (url, options) => { source.url = url; source.options = options; return source; } } });
|
||||
const file = path.join(__dirname, '../../web/js/gallery.js');
|
||||
const Manager = vm.runInContext(fs.readFileSync(file, 'utf8') + '\nGalleryManager;', context, { filename: file });
|
||||
const manager = Object.create(Manager.prototype);
|
||||
Object.assign(manager, { getSettings: () => ({}),
|
||||
api: { get: async url => { gets.push(url); return { success: true, jobs: [] }; },
|
||||
post: async (url, body) => {
|
||||
posts.push([url, body]);
|
||||
if (postFails) throw new Error('offline');
|
||||
return { success: true };
|
||||
} } });
|
||||
const emit = (type, data) => (listeners[type] || []).forEach(handler => handler({ data: JSON.stringify(data) }));
|
||||
const ui = { addStatusMessage() {}, progressText: element(), progressPercent: element(), progressBar: element(),
|
||||
progressDetails: element(), progressETA: element(), isCancelled: () => false,
|
||||
onCancel: handler => { ui.cancel = handler; } };
|
||||
const result = modal ? manager.generateThumbnailsWithProgressInModal(ui)
|
||||
: manager.generateThumbnailsWithProgress({ quality: 'medium' });
|
||||
const outcome = result.then(value => ({ value }), error => ({ error: error.message }));
|
||||
const cancel = () => (modal ? ui.cancel() : elements.cancelThumbnailsBtn.click());
|
||||
const settle = () => new Promise(resolve => setImmediate(resolve));
|
||||
return { emit, source, posts, gets, warnings, outcome, cancel, settle };
|
||||
}
|
||||
|
||||
const cancelUrl = id => `/prompt_manager/jobs/${id}/cancel`;
|
||||
|
||||
test('Cancel before the job id arrives cancels that job once it does', async () => {
|
||||
const f = streamFixture();
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
assert.equal(f.source.closed, 0); // kept open until the stream names its job
|
||||
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
|
||||
f.emit('status', { phase: 'scanning', message: 'Scanning…' });
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
await f.settle();
|
||||
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
|
||||
assert.equal(f.source.closed, 1);
|
||||
f.emit('job', { id: 'c'.repeat(32), started: true }); // a stray second frame changes nothing
|
||||
await f.settle();
|
||||
assert.equal(f.posts.length, 1);
|
||||
assert.deepEqual(f.gets, []); // never looks a job up by kind
|
||||
});
|
||||
|
||||
test('Cancel after the id cancels it at once, and a double Cancel posts once', async () => {
|
||||
const f = streamFixture();
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
f.cancel();
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
|
||||
assert.equal(f.source.closed, 1);
|
||||
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
|
||||
assert.deepEqual(f.gets, []);
|
||||
});
|
||||
|
||||
test("Cancel leaves another tab's or System's job running and only stops following it", async () => {
|
||||
const f = streamFixture();
|
||||
f.emit('job', { id: JOB, started: false });
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
assert.deepEqual(f.gets, []);
|
||||
assert.equal(f.source.closed, 1);
|
||||
assert.match((await f.outcome).error, /keeps running/);
|
||||
});
|
||||
|
||||
test('a Cancel the stream never names a job for posts nothing and closes when it ends', async () => {
|
||||
const f = streamFixture();
|
||||
f.cancel();
|
||||
f.emit('error', { error: 'Scan for metadata already running' });
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
assert.equal(f.source.closed, 1);
|
||||
});
|
||||
|
||||
test('a cancel post that fails is reported, not thrown', async () => {
|
||||
const f = streamFixture({ postFails: true });
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.equal(f.posts.length, 1);
|
||||
assert.equal(f.warnings.length, 1);
|
||||
assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' });
|
||||
});
|
||||
|
||||
test('the startup dialog cancels its own job by id too', async () => {
|
||||
const f = streamFixture({ modal: true });
|
||||
f.cancel();
|
||||
await f.settle();
|
||||
assert.deepEqual(f.posts, []);
|
||||
f.emit('job', { id: JOB, started: true });
|
||||
await f.settle();
|
||||
assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]]));
|
||||
assert.equal(f.source.closed, 1);
|
||||
assert.match((await f.outcome).error, /cancelled/);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
// Per-kind Activity descriptors (#249 System S5 Task 13, D111): each job kind's pill verb, toasts, line, tiles,
|
||||
// outcomes, openable rows and labels, from one table (web/js/workspace/activity-kinds.mjs).
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const source = import('../../web/js/workspace/activity-kinds.mjs');
|
||||
const store = import('../../web/js/workspace/activity-store.mjs');
|
||||
|
||||
const ID = 'a'.repeat(32);
|
||||
|
||||
function job(overrides = {}) {
|
||||
return { id: ID, kind: 'scan', title: 'Scan for metadata', state: 'running', started_at: 1, finished_at: null,
|
||||
progress: 42, status_line: '', processed: 8240, total: 19622,
|
||||
counters: { added: 1, known: 0, review: 0, no_metadata: 0, errors: 1 }, scope: {}, listed: {}, log_length: 0,
|
||||
cancellable: true, ...overrides };
|
||||
}
|
||||
|
||||
test('scan descriptors are unchanged', async () => {
|
||||
const { kindOf } = await source;
|
||||
const scan = kindOf('scan');
|
||||
assert.equal(scan.pillVerb, 'Scanning');
|
||||
assert.equal(scan.startToast(job()), 'Scan started.');
|
||||
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
|
||||
['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]);
|
||||
assert.deepEqual(scan.outcomes, [['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
|
||||
['no_metadata', 'No usable metadata'], ['known', 'Already known']]);
|
||||
assert.deepEqual([...scan.openable].sort(), ['added', 'known']);
|
||||
assert.deepEqual([scan.log, scan.cancelLabel], ['Scan log', 'Cancel scan']);
|
||||
assert.equal(scan.finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error');
|
||||
assert.equal(scan.finishToast(job({ state: 'cancelled' })), 'Scan cancelled · 1 added');
|
||||
assert.equal(scan.finishToast(job({ state: 'failed' })), 'Scan failed · 1 added · 1 error');
|
||||
assert.equal(scan.line(job()), '1 added · 1 error');
|
||||
assert.equal(scan.kept(job()), '1 added');
|
||||
assert.equal(scan.scope(job({ scope: { output_folders: 2 } })), '2 output folders');
|
||||
// The store keeps answering with the same table.
|
||||
const { jobView, finishToast, startToast, pillParts, outcomeLine } = await store;
|
||||
assert.equal(jobView('scan'), scan);
|
||||
assert.equal(finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error');
|
||||
assert.equal(startToast(job()), 'Scan started.');
|
||||
assert.equal(pillParts(job()).text, 'Scanning · 42% · 8,240 / 19,622');
|
||||
assert.equal(outcomeLine(job()), '1 added · 1 error');
|
||||
});
|
||||
|
||||
test('thumbnail job shows its own tiles and toasts', async () => {
|
||||
const { kindOf, viewOf } = await source;
|
||||
const { finishToast, startToast, pillParts, outcomeLine } = await store;
|
||||
const thumbs = job({ kind: 'thumbnails', title: 'Thumbnail generation', processed: 4, total: 10, progress: 40,
|
||||
counters: { generated: 1420, skipped: 3, errors: 1 }, scope: { mode: 'missing', size: 512, quality: 'medium',
|
||||
output_folders: 2 } });
|
||||
const view = kindOf('thumbnails');
|
||||
assert.equal(viewOf(thumbs), view);
|
||||
assert.deepEqual(view.tiles, [['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]);
|
||||
assert.deepEqual(view.outcomes, [['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]);
|
||||
assert.deepEqual([...view.openable].sort(), ['generated', 'skipped']);
|
||||
assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']);
|
||||
assert.equal(pillParts(thumbs).text, 'Generating thumbnails · 40% · 4 / 10');
|
||||
assert.equal(startToast(thumbs), 'Thumbnail generation started.');
|
||||
assert.equal(finishToast({ ...thumbs, state: 'finished' }), 'Thumbnails finished · 1,420 generated · 1 error');
|
||||
assert.equal(finishToast({ ...thumbs, state: 'failed' }), 'Thumbnails failed · 1,420 generated · 1 error');
|
||||
assert.equal(finishToast({ ...thumbs, state: 'cancelled' }), 'Thumbnails cancelled · 1,420 generated');
|
||||
assert.equal(finishToast({ ...thumbs, counters: { generated: 0, skipped: 0, errors: 2 }, state: 'finished' }),
|
||||
'Thumbnails finished · 0 generated · 2 errors');
|
||||
assert.equal(outcomeLine(thumbs), '1,420 generated · 1 error');
|
||||
assert.equal(view.kept(thumbs), '1,420 generated');
|
||||
assert.equal(view.scope(thumbs), '2 output folders');
|
||||
});
|
||||
|
||||
test('non-cancellable jobs show no Cancel job', async () => {
|
||||
const { canCancel } = await source;
|
||||
assert.equal(canCancel(job()), true);
|
||||
assert.equal(canCancel(job({ cancellable: false })), false);
|
||||
for (const state of ['finished', 'cancelled', 'failed', 'lost']) assert.equal(canCancel(job({ state })), false);
|
||||
// The health check is a read job the server cannot stop.
|
||||
assert.equal(canCancel(job({ kind: 'health', title: 'Health check', cancellable: false, counters: {} })), false);
|
||||
});
|
||||
|
||||
test('indeterminate progress has no value attribute', async () => {
|
||||
const { indeterminate, showProgress } = await source;
|
||||
const { pillParts } = await store;
|
||||
function node() {
|
||||
const attributes = new Map();
|
||||
return { attributes, textContent: '', set value(v) { attributes.set('value', String(v)); },
|
||||
get value() { return Number(attributes.get('value') ?? 0); },
|
||||
setAttribute(key, v) { attributes.set(key, String(v)); }, removeAttribute(key) { attributes.delete(key); } };
|
||||
}
|
||||
// Running with no total yet: the server has not said how much there is, so there is no percentage.
|
||||
const health = job({ kind: 'health', title: 'Health check', progress: 0, processed: 0, total: 0, counters: {},
|
||||
cancellable: false, status_line: 'Checking the database and image links' });
|
||||
assert.equal(indeterminate(health), true);
|
||||
const bar = node();
|
||||
bar.value = 37;
|
||||
showProgress(bar, health);
|
||||
assert.equal(bar.attributes.has('value'), false);
|
||||
assert.equal(bar.textContent, '');
|
||||
assert.equal(pillParts(health).text, 'Health check');
|
||||
assert.deepEqual(pillParts(health), { label: 'Health check', files: '', text: 'Health check' });
|
||||
// A known total is determinate, and so is any ended job.
|
||||
const scan = job();
|
||||
assert.equal(indeterminate(scan), false);
|
||||
showProgress(bar, scan);
|
||||
assert.equal(bar.attributes.get('value'), '42');
|
||||
assert.equal(bar.textContent, '42%');
|
||||
assert.equal(indeterminate({ ...health, state: 'finished', progress: 100 }), false);
|
||||
// A scan before its file count is known is indeterminate too, never "0%".
|
||||
assert.equal(pillParts(job({ total: 0, processed: 0, progress: 0 })).text, 'Scanning');
|
||||
});
|
||||
|
||||
test('held legacy jobs read "(admin page)" with no outcomes', async () => {
|
||||
const { viewOf } = await source;
|
||||
const { jobDetail } = await import('../../web/js/workspace/activity.mjs');
|
||||
const { finishToast, outcomeLine } = await store;
|
||||
// A hold has no outcome groups (counters {}), whatever its kind: thumbnails held by the gallery's clear included.
|
||||
for (const [kind, title] of [['restore', 'Database restore (admin page)'],
|
||||
['thumbnails', 'Thumbnail clear (gallery page)'], ['maintenance', 'Database maintenance (admin page)']]) {
|
||||
const held = job({ kind, title, counters: {}, cancellable: false, progress: 0, processed: 0, total: 0 });
|
||||
const view = viewOf(held);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind);
|
||||
assert.equal(outcomeLine(held), '');
|
||||
assert.match(jobDetail(held), /^Started \d\d:\d\d · closing this view does not cancel it$/);
|
||||
const ended = { ...held, state: 'finished', finished_at: 2 };
|
||||
assert.match(jobDetail(ended), /^Finished \d\d:\d\d$/);
|
||||
assert.equal(finishToast(ended), `${title} finished.`);
|
||||
assert.ok(title.endsWith('page)'));
|
||||
}
|
||||
});
|
||||
|
||||
test('unknown and generic kinds show no tiles of another kind', async () => {
|
||||
const { kindOf, viewOf } = await source;
|
||||
for (const kind of ['maintenance', '', 'constructor', '__proto__', 'health']) {
|
||||
const view = kindOf(kind);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind);
|
||||
assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']);
|
||||
assert.equal(view.pillVerb, null);
|
||||
assert.equal(view.line(job({ kind })), '');
|
||||
assert.equal(view.kept(job({ kind })), '');
|
||||
}
|
||||
const backup = job({ kind: 'backup', title: 'Backup' });
|
||||
assert.equal(kindOf('backup').startToast(backup), 'Backup started.');
|
||||
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'finished' }), 'Backup finished.');
|
||||
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'cancelled' }), 'Backup was cancelled.');
|
||||
assert.equal(kindOf('backup').finishToast({ ...backup, state: 'failed' }), 'Backup failed.');
|
||||
// A kind's tiles are only the groups its job counts: no zero tile for a group the server never reported.
|
||||
const partial = viewOf(job({ counters: { added: 3, errors: 0 } }));
|
||||
assert.deepEqual(partial.tiles, [['added', 'Added to Library', 'added'], ['errors', 'Errors', 'bad']]);
|
||||
assert.deepEqual(partial.outcomes, [['added', 'Added to Library'], ['errors', 'Errors']]);
|
||||
assert.deepEqual([...partial.openable], ['added']);
|
||||
assert.equal(partial.log, 'Scan log');
|
||||
});
|
||||
@@ -289,8 +289,10 @@ test('summaries are validated and the copy follows the spec', async () => {
|
||||
|
||||
const scan = parseJob(summary({ progress: 42, processed: 8240, total: 19622, counters: { added: 1, errors: 1 } }));
|
||||
assert.deepEqual(pillParts(scan), { label: 'Scanning · 42%', files: ' · 8,240 / 19,622', text: 'Scanning · 42% · 8,240 / 19,622' });
|
||||
assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning · 42%');
|
||||
assert.equal(pillParts({ ...scan, kind: 'thumbnails', title: 'Thumbnails' }).text, 'Thumbnails · 42% · 8,240 / 19,622');
|
||||
// D111: without a total the progress is indeterminate, so the pill shows no percentage.
|
||||
assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning');
|
||||
// A kind without a pill verb shows its title (D111: thumbnails have their own, see workspace-activity-kinds.test.js).
|
||||
assert.equal(pillParts({ ...scan, kind: 'backup', title: 'Backup' }).text, 'Backup · 42% · 8,240 / 19,622');
|
||||
assert.equal(finishToast({ ...scan, state: 'finished' }), 'Scan finished · 1 added · 1 error');
|
||||
assert.equal(finishToast({ ...scan, state: 'cancelled' }), 'Scan cancelled · 1 added');
|
||||
assert.equal(finishToast({ ...scan, state: 'failed' }), 'Scan failed · 1 added · 1 error');
|
||||
@@ -332,7 +334,8 @@ test('image-tags jobs get their own tiles outcomes and pill verb', async () => {
|
||||
'Add tags to images finished · 0 tagged · 0 errors');
|
||||
});
|
||||
|
||||
test('unknown kinds keep the generic view', async () => {
|
||||
// D111 (S5): an unknown kind no longer borrows the scan's tiles (S1 review M6, S2 review M3); it has none.
|
||||
test('unknown kinds get the generic view without the scan tiles', async () => {
|
||||
const { jobView } = await source;
|
||||
const scan = jobView('scan');
|
||||
assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
|
||||
@@ -343,7 +346,7 @@ test('unknown kinds keep the generic view', async () => {
|
||||
assert.deepEqual([scan.log, scan.cancel], ['Scan log', 'Cancel scan']);
|
||||
for (const kind of ['maintenance', '', 'constructor', '__proto__']) {
|
||||
const view = jobView(kind);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [scan.tiles, scan.outcomes, [...scan.openable]]);
|
||||
assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []]);
|
||||
assert.deepEqual([view.log, view.cancel], ['Job log', 'Cancel job']);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
// System › thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (app.js 1433–1437,
|
||||
// `thumbnail-review`) on the thumbnail job (S5 Task 12). Pure parts: the option values the job takes, the preview size
|
||||
// check, the Image loading quality preselection (Q3), the D99 plan line and the start outcome.
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const reviewModule = import('../../web/js/workspace/thumbnail-review.mjs');
|
||||
|
||||
test('the concept options map to the job values in the concept order', async () => {
|
||||
const { SCOPE_OPTIONS, QUALITY_OPTIONS, SIZE_LIMIT } = await reviewModule;
|
||||
assert.deepEqual(SCOPE_OPTIONS, [['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]);
|
||||
assert.deepEqual(QUALITY_OPTIONS, [['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]);
|
||||
assert.deepEqual(SIZE_LIMIT, { min: 128, max: 2048, value: 512 });
|
||||
});
|
||||
|
||||
test('size outside 128–2048 reports validity: only whole pixels in range are a size', async () => {
|
||||
const { previewSize } = await reviewModule;
|
||||
assert.equal(previewSize('512'), 512);
|
||||
assert.equal(previewSize('128'), 128);
|
||||
assert.equal(previewSize('2048'), 2048);
|
||||
for (const text of ['', ' ', '127', '2049', '4096', '512.5', '-512', 'abc', '1e3', '0x200']) {
|
||||
assert.equal(previewSize(text), null, text);
|
||||
}
|
||||
});
|
||||
|
||||
test('Image loading quality preselects the dialog quality (Q3); anything else is Medium', async () => {
|
||||
const { preselectedQuality } = await reviewModule;
|
||||
assert.equal(preselectedQuality('low'), 'low');
|
||||
assert.equal(preselectedQuality('high'), 'high');
|
||||
assert.equal(preselectedQuality('medium'), 'medium');
|
||||
assert.equal(preselectedQuality(undefined), 'medium');
|
||||
assert.equal(preselectedQuality('original'), 'medium');
|
||||
});
|
||||
|
||||
test('the plan line counts what a missing-only run generates and what is already up to date (D99)', async () => {
|
||||
const { planText } = await reviewModule;
|
||||
assert.equal(planText({ mode: 'missing', targets: 12, fresh: 30, remaining: 0 }), '12 previews to generate · 30 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 1, fresh: 1, remaining: 0 }), '1 preview to generate · 1 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 0, fresh: 0, remaining: 0 }), '0 previews to generate · 0 already up to date.');
|
||||
assert.equal(planText({ mode: 'missing', targets: 10000, fresh: 5, remaining: 2500 }),
|
||||
'10,000 previews to generate · 5 already up to date · 2,500 more on the next run.');
|
||||
});
|
||||
|
||||
test('Rebuild all says every preview is regenerated (D99)', async () => {
|
||||
const { planText } = await reviewModule;
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 42, fresh: 30, remaining: 0 }), 'Every preview is regenerated: 42 files.');
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 1, fresh: 1, remaining: 0 }), 'Every preview is regenerated: 1 file.');
|
||||
assert.equal(planText({ mode: 'rebuild', targets: 10000, fresh: 9000, remaining: 3 }),
|
||||
'Every preview is regenerated: 10,000 files · 3 more on the next run.');
|
||||
});
|
||||
|
||||
test('an answer without counts is a failed plan, never zeros', async () => {
|
||||
const { planText, PLAN_FAILED } = await reviewModule;
|
||||
assert.equal(planText(null), PLAN_FAILED);
|
||||
assert.equal(planText({ mode: 'missing', targets: 2 }), PLAN_FAILED);
|
||||
assert.equal(planText({ mode: 'missing', targets: -1, fresh: 0, remaining: 0 }), PLAN_FAILED);
|
||||
assert.equal(planText({ mode: 'other', targets: 1, fresh: 0, remaining: 0 }), PLAN_FAILED);
|
||||
});
|
||||
|
||||
const scopeOf = () => ({ signal: new AbortController().signal });
|
||||
|
||||
test('Start starts the thumbnail job with the chosen options and opens it in Activity', async () => {
|
||||
const { startThumbnails } = await reviewModule;
|
||||
const calls = [];
|
||||
const navigated = [];
|
||||
const store = { start: async (kind, body) => { calls.push([kind, body]); return { job: { id: 'a'.repeat(32) } }; } };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'rebuild', size: 1024, quality: 'high' },
|
||||
navigate: hash => navigated.push(hash), notify: () => assert.fail('no toast of its own') });
|
||||
assert.deepEqual(calls, [['thumbnails', { mode: 'rebuild', size: 1024, quality: 'high' }]]);
|
||||
assert.deepEqual(navigated, [`#/activity/${'a'.repeat(32)}`]);
|
||||
});
|
||||
|
||||
test('a refused start shows the busy link toast and stays', async () => {
|
||||
const { startThumbnails } = await reviewModule;
|
||||
const toasts = [];
|
||||
const store = { start: async () => ({ busy: { id: 'b'.repeat(32), title: 'Scan for metadata' } }) };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
|
||||
assert.deepEqual(toasts, [['Another job is running: ', { text: 'Scan for metadata', href: `#/activity/${'b'.repeat(32)}` }, '.']]);
|
||||
});
|
||||
|
||||
test('a failed start says so; a start after the section left says nothing', async () => {
|
||||
const { startThumbnails, START_FAILED } = await reviewModule;
|
||||
const toasts = [];
|
||||
const store = { start: async () => { throw new Error('offline'); } };
|
||||
await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
navigate: () => assert.fail('stays'), notify: message => toasts.push(message) });
|
||||
assert.deepEqual(toasts, [START_FAILED]);
|
||||
const gone = new AbortController();
|
||||
gone.abort();
|
||||
await startThumbnails({ store, scope: { signal: gone.signal }, options: { mode: 'missing', size: 512, quality: 'medium' },
|
||||
navigate: () => assert.fail('stays'), notify: () => assert.fail('silent') });
|
||||
});
|
||||
@@ -413,11 +413,11 @@ class TestThumbnailProgressStream(ThumbnailTestCase):
|
||||
lock = threading.Lock()
|
||||
original = images_module._write_image_thumbnail
|
||||
|
||||
def recording(src, dst, size):
|
||||
def recording(src, dst, size, *args):
|
||||
with lock:
|
||||
seen.add(threading.current_thread().name)
|
||||
time.sleep(0.05)
|
||||
return original(src, dst, size)
|
||||
return original(src, dst, size, *args)
|
||||
|
||||
images_module._write_image_thumbnail = recording
|
||||
self.addCleanup(setattr, images_module, "_write_image_thumbnail", original)
|
||||
@@ -1939,7 +1939,11 @@ class TestGalleryRescan(ImageAPITestCase):
|
||||
|
||||
|
||||
class TestOutputScanHelpers(ImageAPITestCase):
|
||||
"""Stable ids for filesystem entries, no symlink reads, stop on disconnect."""
|
||||
"""Stable ids for filesystem entries and no symlink reads.
|
||||
|
||||
The thumbnail stream no longer stops when its client goes away: see
|
||||
``tests/test_thumbnail_jobs.py`` (the work is a job that outlives it).
|
||||
"""
|
||||
|
||||
def test_output_entry_id_is_a_stable_digest_of_the_path(self):
|
||||
import hashlib
|
||||
@@ -1964,28 +1968,6 @@ class TestOutputScanHelpers(ImageAPITestCase):
|
||||
names = sorted(p.name for p in images_module._iter_media_files(output, 100))
|
||||
self.assertEqual(names, ["real.png"])
|
||||
|
||||
async def test_thumbnail_stream_stops_when_the_client_disconnects(self):
|
||||
generated = []
|
||||
|
||||
def fake_generate(src, dst, size, is_video):
|
||||
generated.append(src)
|
||||
return {
|
||||
"action": "generated",
|
||||
"file": src.name,
|
||||
"dir": src.parent.name,
|
||||
"type": "image",
|
||||
}
|
||||
|
||||
self.api._generate_one = fake_generate
|
||||
|
||||
class GoneResponse:
|
||||
async def write(self, data):
|
||||
raise ConnectionResetError("client went away")
|
||||
|
||||
targets = [(Path(f"/x/{i}.png"), Path(f"/x/t{i}.jpg"), False) for i in range(3)]
|
||||
await self.api._stream_thumbnails(GoneResponse(), targets, (256, 256))
|
||||
self.assertEqual(generated, [])
|
||||
|
||||
|
||||
class TestThumbnailStartupStatus(ThumbnailTestCase):
|
||||
"""Startup counts and generation cover the same configured collection."""
|
||||
|
||||
@@ -101,6 +101,35 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
self.assertEqual(await _collect(job), [{"type": "complete"}])
|
||||
|
||||
async def test_lead_comes_first_and_nothing_published_meanwhile_is_lost(self):
|
||||
"""S5 I1: the ``job`` frame leads, with the subscription already in place."""
|
||||
job = Job("thumbnails", "Thumbnail generation")
|
||||
job.publish({"type": "status"})
|
||||
lead = {"type": "job"}
|
||||
seen = []
|
||||
async with contextlib.aclosing(job.events(lead=lead)) as events:
|
||||
async for payload in events:
|
||||
seen.append(payload)
|
||||
if payload is lead:
|
||||
# Published while the consumer is still sending the lead.
|
||||
job.publish({"type": "progress"})
|
||||
job.publish({"type": "complete"})
|
||||
job.finish()
|
||||
self.assertEqual(
|
||||
seen,
|
||||
[lead, {"type": "status"}, {"type": "progress"}, {"type": "complete"}],
|
||||
)
|
||||
|
||||
async def test_lead_on_a_finished_job_then_its_last_event(self):
|
||||
job = Job("thumbnails", "Thumbnail generation")
|
||||
job.publish({"type": "complete"})
|
||||
job.finish()
|
||||
seen = []
|
||||
async with contextlib.aclosing(job.events(lead={"type": "job"})) as events:
|
||||
async for payload in events:
|
||||
seen.append(payload)
|
||||
self.assertEqual(seen, [{"type": "job"}, {"type": "complete"}])
|
||||
|
||||
async def test_departed_subscriber_is_forgotten(self):
|
||||
job = Job("scan", "Scan")
|
||||
job.publish({"type": "progress", "progress": 1})
|
||||
@@ -151,6 +180,30 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
class TestJobRegistry(unittest.IsolatedAsyncioTestCase):
|
||||
|
||||
async def test_a_compatible_kind_never_blocks(self):
|
||||
"""S5 M7: a caller may name running kinds that cannot conflict with it."""
|
||||
registry = JobRegistry()
|
||||
gate = asyncio.Event()
|
||||
thumbs = registry.start(
|
||||
"thumbnails",
|
||||
"Thumbnail generation",
|
||||
admission="guarded",
|
||||
runner=_gated_runner(gate),
|
||||
)
|
||||
self.assertIs(registry.blocker("probe", admission="records"), thumbs)
|
||||
self.assertIsNone(
|
||||
registry.blocker("probe", admission="records", compatible=("thumbnails",))
|
||||
)
|
||||
backup = registry.start(
|
||||
"backup", "Backup", admission="guarded", runner=_gated_runner(gate)
|
||||
)
|
||||
self.assertIs(
|
||||
registry.blocker("probe", admission="records", compatible=("thumbnails",)),
|
||||
backup,
|
||||
)
|
||||
gate.set()
|
||||
await asyncio.wait_for(asyncio.gather(thumbs.task, backup.task), 2)
|
||||
|
||||
async def test_start_lists_and_finishes_a_job(self):
|
||||
registry = JobRegistry()
|
||||
gate = asyncio.Event()
|
||||
@@ -421,6 +474,28 @@ class TestJobOutcomesAndLog(unittest.TestCase):
|
||||
SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors")
|
||||
)
|
||||
|
||||
def test_record_counted_counts_exactly_lists_capped_and_logs_nothing(self):
|
||||
job = Job("thumbnails", "Thumbnail generation", groups=("skipped",))
|
||||
rows = [
|
||||
{"path": os.path.abspath(f"out{os.sep}f{i}.png"), "root": "output"}
|
||||
for i in range(MAX_OUTCOME_ROWS + 5)
|
||||
]
|
||||
job.record_counted("skipped", 10_000, rows)
|
||||
job.record_counted("skipped", 1, [{"path": "late.png", "root_index": True}])
|
||||
self.assertEqual(job.counters["skipped"], 10_001)
|
||||
self.assertEqual(len(job.outcomes["skipped"]), MAX_OUTCOME_ROWS)
|
||||
self.assertEqual(job.outcomes["skipped"][0]["path"], "f0.png")
|
||||
self.assertIsNone(job.outcomes["skipped"][0]["root_index"])
|
||||
self.assertEqual(job.log, [])
|
||||
with self.assertRaises(ValueError):
|
||||
job.record_counted("bogus", 1)
|
||||
|
||||
def test_thumbnail_job_groups(self):
|
||||
self.assertEqual(
|
||||
jobs_module.THUMBNAIL_GROUP_NAMES, ("generated", "skipped", "errors")
|
||||
)
|
||||
self.assertEqual(jobs_module.THUMBNAILS_JOB_KIND, "thumbnails")
|
||||
|
||||
def test_image_tag_job_kind_and_groups(self):
|
||||
# #249 D1: bulk Add tags / Auto-tag, counter-tile order.
|
||||
self.assertEqual(getattr(jobs_module, "IMAGE_TAG_JOB_KIND", None), "image-tags")
|
||||
|
||||
+40
-2
@@ -859,7 +859,9 @@ class TestMonitorRecording(ReviewTestCase):
|
||||
While a ``records`` or ``guarded`` job runs, the monitor's recording
|
||||
records nothing and answers ``deferred`` (its fallback link stands and
|
||||
a later scan reads the ambiguous file again); a ``read`` job does not
|
||||
refuse it, and it records once the blocking job has ended.
|
||||
refuse it, and it records once the blocking job has ended. Thumbnail
|
||||
work (the job and the gallery's holds) is ``guarded`` but writes only
|
||||
previews, never records, so it does not defer the recording (S5 M7).
|
||||
"""
|
||||
self._require_module()
|
||||
from py.api.jobs import RESTORE_JOB_KIND, THUMBNAILS_JOB_KIND
|
||||
@@ -870,7 +872,7 @@ class TestMonitorRecording(ReviewTestCase):
|
||||
|
||||
for kind, admission in (
|
||||
(RESTORE_JOB_KIND, "records"),
|
||||
(THUMBNAILS_JOB_KIND, "guarded"),
|
||||
("backup", "guarded"),
|
||||
):
|
||||
with self.subTest(admission=admission):
|
||||
async with self.api._jobs.hold(
|
||||
@@ -888,7 +890,43 @@ class TestMonitorRecording(ReviewTestCase):
|
||||
)
|
||||
|
||||
async with self.api._jobs.hold("probe-read", "Read probe", admission="read"):
|
||||
async with self.api._jobs.hold(
|
||||
THUMBNAILS_JOB_KIND,
|
||||
"Thumbnail clear (gallery page)",
|
||||
admission="guarded",
|
||||
):
|
||||
outcome = await asyncio.to_thread(
|
||||
self.api.record_review_file, str(path)
|
||||
)
|
||||
self.assertEqual(outcome, "recorded")
|
||||
self.assertEqual(len(self._items()), 1)
|
||||
|
||||
async def test_a_running_thumbnail_job_does_not_defer_monitor_recording(self):
|
||||
"""S5 M7: a long rebuild never holds Needs review back (no records)."""
|
||||
self._require_module()
|
||||
from py.api.jobs import THUMBNAILS_JOB_KIND
|
||||
|
||||
self._roots()
|
||||
path = self.output_dir / "beside.png"
|
||||
write_png(path, prompt=unlinkable_graph("cat", "dog"))
|
||||
gate = asyncio.Event()
|
||||
|
||||
async def runner(job):
|
||||
await gate.wait()
|
||||
|
||||
job = self.api._jobs.start(
|
||||
THUMBNAILS_JOB_KIND,
|
||||
"Thumbnail generation",
|
||||
admission="guarded",
|
||||
cancellable=True,
|
||||
scope={},
|
||||
runner=runner,
|
||||
)
|
||||
try:
|
||||
outcome = await asyncio.to_thread(self.api.record_review_file, str(path))
|
||||
finally:
|
||||
gate.set()
|
||||
await asyncio.wait_for(job.task, 5)
|
||||
self.assertEqual(outcome, "recorded")
|
||||
self.assertEqual(len(self._items()), 1)
|
||||
|
||||
|
||||
@@ -0,0 +1,846 @@
|
||||
"""Thumbnail generation as an Activity job (#249 System S5, Task 12).
|
||||
|
||||
``GET /prompt_manager/images/thumbnails/plan`` counts what a run would do and
|
||||
writes nothing. ``POST /prompt_manager/jobs/thumbnails`` starts the
|
||||
``thumbnails`` job (admission ``guarded``, cancellable at a file boundary).
|
||||
The legacy gallery stream ``POST /images/generate-thumbnails/progress`` starts
|
||||
or attaches to that job and streams its unchanged events from it, so a client
|
||||
that goes away no longer stops the work.
|
||||
|
||||
Ordering is driven by threading/asyncio events, never by timers.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
try:
|
||||
from tests.test_api_images import ThumbnailTestCase, make_png, parse_sse
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_images import ThumbnailTestCase, make_png, parse_sse
|
||||
|
||||
from PIL import Image # noqa: E402
|
||||
|
||||
import py.api.images as images_module # noqa: E402
|
||||
from py.api.thumbnail_jobs import thumbnail_options # noqa: E402
|
||||
from py.config import PromptManagerConfig # noqa: E402
|
||||
|
||||
PLAN_URL = "/prompt_manager/images/thumbnails/plan"
|
||||
JOB_URL = "/prompt_manager/jobs/thumbnails"
|
||||
|
||||
|
||||
def _tree_state(root):
|
||||
"""{relative path: (size, mtime_ns, sha256)} of every file under ``root``."""
|
||||
state = {}
|
||||
for folder, _dirs, files in os.walk(root):
|
||||
for name in files:
|
||||
path = Path(folder) / name
|
||||
data = path.read_bytes()
|
||||
stat = path.stat()
|
||||
state[path.relative_to(root).as_posix()] = (
|
||||
stat.st_size,
|
||||
stat.st_mtime_ns,
|
||||
hashlib.sha256(data).hexdigest(),
|
||||
)
|
||||
return state
|
||||
|
||||
|
||||
class ThumbnailJobCase(ThumbnailTestCase):
|
||||
"""Helpers: start, gate and wait for thumbnail jobs."""
|
||||
|
||||
async def _start(self, **body):
|
||||
resp = await self.client.post(JOB_URL, json=body)
|
||||
return resp.status, await resp.json()
|
||||
|
||||
async def _finished(self, job_id):
|
||||
job = self.api._jobs.get(job_id)
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
return job
|
||||
|
||||
def _gate_generation(self):
|
||||
"""Block every file in the worker until released; returns (entered, release)."""
|
||||
entered = threading.Event()
|
||||
release = threading.Event()
|
||||
original = self.api._generate_one
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(10)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
self.api._generate_one = gated
|
||||
self.addCleanup(release.set)
|
||||
return entered, release
|
||||
|
||||
async def _wait(self, event):
|
||||
loop = asyncio.get_running_loop()
|
||||
self.assertTrue(await loop.run_in_executor(None, event.wait, 10))
|
||||
|
||||
@staticmethod
|
||||
def _watch_subscribers(job):
|
||||
"""(subscribed, left): threading events for a stream joining or leaving."""
|
||||
subscribed, left = threading.Event(), threading.Event()
|
||||
|
||||
class Watched(set):
|
||||
def add(self, item):
|
||||
super().add(item)
|
||||
subscribed.set()
|
||||
|
||||
def discard(self, item):
|
||||
super().discard(item)
|
||||
left.set()
|
||||
|
||||
job._subscribers = Watched(job._subscribers)
|
||||
return subscribed, left
|
||||
|
||||
def _status_lines(self):
|
||||
"""Every status line any job is given, in order."""
|
||||
from py.api.jobs import Job
|
||||
|
||||
lines = []
|
||||
original = Job.update
|
||||
|
||||
def update(job, **kwargs):
|
||||
if kwargs.get("status_line") is not None:
|
||||
lines.append(kwargs["status_line"])
|
||||
return original(job, **kwargs)
|
||||
|
||||
patcher = patch.object(Job, "update", update)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
return lines
|
||||
|
||||
def _one_worker(self):
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1 # batches of 4
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
|
||||
def _run_scan(self):
|
||||
gate = asyncio.Event()
|
||||
|
||||
async def runner(job):
|
||||
await gate.wait()
|
||||
|
||||
job = self.api._jobs.start(
|
||||
"scan", "Scan for metadata", admission="records", scope={}, runner=runner
|
||||
)
|
||||
|
||||
async def release():
|
||||
gate.set()
|
||||
await asyncio.wait_for(job.task, 5)
|
||||
|
||||
self.addAsyncCleanup(release)
|
||||
return job
|
||||
|
||||
|
||||
class TestThumbnailPlan(ThumbnailJobCase):
|
||||
async def test_plan_is_read_only_and_counts_cap(self):
|
||||
for name in ("a", "b", "c"):
|
||||
make_png(self.output_dir / f"{name}.png", size=(64, 64))
|
||||
fresh = make_png(self._thumb_path("a"))
|
||||
source_time = (self.output_dir / "a.png").stat().st_mtime
|
||||
os.utime(fresh, (source_time + 10, source_time + 10))
|
||||
before = _tree_state(self.comfy_dir)
|
||||
|
||||
with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1):
|
||||
missing = await self.client.get(
|
||||
PLAN_URL, params={"mode": "missing", "size": "512", "quality": "high"}
|
||||
)
|
||||
rebuild = await self.client.get(
|
||||
PLAN_URL, params={"mode": "rebuild", "size": "256", "quality": "low"}
|
||||
)
|
||||
|
||||
self.assertEqual(missing.status, 200)
|
||||
self.assertEqual(
|
||||
await missing.json(),
|
||||
{
|
||||
"success": True,
|
||||
"targets": 1,
|
||||
"fresh": 1,
|
||||
"remaining": 1,
|
||||
"roots": ["output"],
|
||||
"mode": "missing",
|
||||
"size": 512,
|
||||
"quality": "high",
|
||||
},
|
||||
)
|
||||
data = await rebuild.json()
|
||||
# Rebuild regenerates fresh previews too: all three are work, one fits the cap.
|
||||
self.assertEqual((data["targets"], data["fresh"], data["remaining"]), (1, 1, 2))
|
||||
self.assertEqual(
|
||||
(data["mode"], data["size"], data["quality"]), ("rebuild", 256, "low")
|
||||
)
|
||||
self.assertEqual(_tree_state(self.comfy_dir), before)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
|
||||
async def test_plan_defaults_to_missing_previews_at_the_concept_size(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
data = await (await self.client.get(PLAN_URL)).json()
|
||||
self.assertEqual(
|
||||
(data["mode"], data["size"], data["quality"], data["targets"]),
|
||||
("missing", 512, "medium", 1),
|
||||
)
|
||||
|
||||
async def test_size_bounds_validated(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
for params in (
|
||||
{"size": "127"},
|
||||
{"size": "2049"},
|
||||
{"size": "512.5"},
|
||||
{"size": "big"},
|
||||
{"mode": "all"},
|
||||
{"quality": "ultra"},
|
||||
):
|
||||
with self.subTest(params=params):
|
||||
resp = await self.client.get(PLAN_URL, params=params)
|
||||
body = await resp.json()
|
||||
self.assertEqual(resp.status, 400)
|
||||
self.assertEqual(
|
||||
(body["success"], body["code"]), (False, "invalid-request")
|
||||
)
|
||||
for body in (
|
||||
{"size": 127},
|
||||
{"size": 2049},
|
||||
{"size": "512"},
|
||||
{"size": 512.0},
|
||||
{"size": True},
|
||||
{"mode": "rebuild-all"},
|
||||
{"quality": None},
|
||||
[],
|
||||
):
|
||||
with self.subTest(body=body):
|
||||
resp = await self.client.post(JOB_URL, json=body)
|
||||
self.assertEqual(resp.status, 400)
|
||||
self.assertEqual((await resp.json())["code"], "invalid-request")
|
||||
for edge in (128, 2048):
|
||||
with self.subTest(edge=edge):
|
||||
resp = await self.client.get(PLAN_URL, params={"size": str(edge)})
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
self.assertFalse(self._thumb_path("a").exists())
|
||||
|
||||
|
||||
class TestThumbnailJob(ThumbnailJobCase):
|
||||
async def test_job_generates_missing_previews_at_the_chosen_edge(self):
|
||||
make_png(self.output_dir / "a.png", size=(800, 400))
|
||||
make_png(self.output_dir / "sub" / "b.png", size=(64, 64))
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG not really")
|
||||
|
||||
status, body = await self._start(mode="missing", size=200, quality="medium")
|
||||
|
||||
self.assertEqual(status, 202)
|
||||
summary = body["job"]
|
||||
self.assertEqual(
|
||||
(summary["kind"], summary["title"], summary["admission"]),
|
||||
("thumbnails", "Thumbnail generation", "guarded"),
|
||||
)
|
||||
self.assertTrue(summary["cancellable"])
|
||||
self.assertEqual(set(summary["counters"]), {"generated", "skipped", "errors"})
|
||||
self.assertEqual(
|
||||
summary["scope"], {"mode": "missing", "size": 200, "quality": "medium"}
|
||||
)
|
||||
job = await self._finished(summary["id"])
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 1})
|
||||
self.assertEqual((job.processed, job.total, job.progress), (3, 3, 100))
|
||||
self.assertEqual(job.scope["output_folders"], 1)
|
||||
with Image.open(self._thumb_path("a")) as thumb:
|
||||
self.assertEqual(thumb.size, (200, 100))
|
||||
rows = {row["path"]: row for row in job.outcomes["generated"]}
|
||||
self.assertEqual(set(rows), {"a.png", "sub/b.png"})
|
||||
self.assertEqual(
|
||||
(rows["a.png"]["root"], rows["a.png"]["root_index"]), ("output", 0)
|
||||
)
|
||||
self.assertEqual(job.outcomes["errors"][0]["path"], "bad.png")
|
||||
self.assertTrue(any("Thumbnails finished" in line for line in job.log))
|
||||
# Rows and log lines never carry the server's absolute layout.
|
||||
self.assertNotIn(str(self.comfy_dir), repr(job.detail()))
|
||||
|
||||
async def test_fresh_previews_are_counted_as_skipped(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
make_png(self.output_dir / "b.png")
|
||||
fresh = make_png(self._thumb_path("a"))
|
||||
source_time = (self.output_dir / "a.png").stat().st_mtime
|
||||
os.utime(fresh, (source_time + 10, source_time + 10))
|
||||
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters, {"generated": 1, "skipped": 1, "errors": 0})
|
||||
self.assertEqual([r["path"] for r in job.outcomes["skipped"]], ["a.png"])
|
||||
|
||||
async def test_rebuild_never_touches_originals(self):
|
||||
for name in ("a", "b"):
|
||||
make_png(self.output_dir / f"{name}.png", size=(400, 400))
|
||||
_, body = await self._start(size=128)
|
||||
await self._finished(body["job"]["id"])
|
||||
originals = {
|
||||
name: hashlib.sha256(
|
||||
(self.output_dir / f"{name}.png").read_bytes()
|
||||
).hexdigest()
|
||||
for name in ("a", "b")
|
||||
}
|
||||
|
||||
_, body = await self._start(mode="rebuild", size=256, quality="high")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 0})
|
||||
for name in ("a", "b"):
|
||||
digest = hashlib.sha256(
|
||||
(self.output_dir / f"{name}.png").read_bytes()
|
||||
).hexdigest()
|
||||
self.assertEqual(digest, originals[name], name)
|
||||
with Image.open(self._thumb_path(name)) as thumb:
|
||||
self.assertEqual(thumb.size, (256, 256))
|
||||
|
||||
async def test_rebuild_keeps_old_preview_when_generation_fails(self):
|
||||
source = make_png(self.output_dir / "a.png", size=(64, 64))
|
||||
_, body = await self._start()
|
||||
await self._finished(body["job"]["id"])
|
||||
preview = self._thumb_path("a").read_bytes()
|
||||
source.write_bytes(b"\x89PNG broken now")
|
||||
later = self._thumb_path("a").stat().st_mtime + 10
|
||||
os.utime(self._thumb_path("a"), (later, later)) # still fresh
|
||||
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(self._thumb_path("a").read_bytes(), preview)
|
||||
leftovers = [p.name for p in self._thumb_path("a").parent.iterdir()]
|
||||
self.assertEqual(leftovers, ["a_thumb.png"]) # no temp file left
|
||||
|
||||
async def test_rebuild_keeps_an_old_video_preview_instead_of_a_placeholder(self):
|
||||
clip = self.output_dir / "clip.mp4"
|
||||
clip.write_bytes(b"\x00")
|
||||
preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.jpg")
|
||||
old = preview.read_bytes()
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
):
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
self.assertEqual(preview.read_bytes(), old)
|
||||
|
||||
async def test_quality_sets_the_encoder_not_the_edge(self):
|
||||
noise = bytes(
|
||||
(i * 7919 + (i >> 3) * 104729) % 251 for i in range(600 * 600 * 3)
|
||||
)
|
||||
with Image.frombytes("RGB", (600, 600), noise) as img:
|
||||
img.save(self.output_dir / "photo.jpg", "JPEG", quality=95)
|
||||
sizes = {}
|
||||
for quality in ("low", "high"):
|
||||
_, body = await self._start(mode="rebuild", size=300, quality=quality)
|
||||
await self._finished(body["job"]["id"])
|
||||
thumb = self._thumb_path("photo", ".jpg")
|
||||
with Image.open(thumb) as image:
|
||||
self.assertEqual(image.size, (300, 300))
|
||||
sizes[quality] = thumb.stat().st_size
|
||||
self.assertLess(sizes["low"], sizes["high"])
|
||||
|
||||
async def test_no_output_folder_fails_the_job(self):
|
||||
self.api._get_all_output_dirs = lambda: []
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.state, "failed")
|
||||
self.assertEqual(job.status_line, "ComfyUI output directory not found")
|
||||
|
||||
async def test_cancel_stops_at_file_boundary_keeping_written_previews(self):
|
||||
for i in range(6):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
original = PromptManagerConfig.WORKER_THREADS
|
||||
PromptManagerConfig.WORKER_THREADS = 1
|
||||
self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original)
|
||||
written = []
|
||||
real_write = images_module._write_image_thumbnail
|
||||
job_ids = []
|
||||
|
||||
def counting(src, dst, *args, **kwargs):
|
||||
real_write(src, dst, *args, **kwargs)
|
||||
written.append(dst)
|
||||
if len(written) == 2:
|
||||
self.assertTrue(self.api._jobs.cancel(job_ids[0]))
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", counting):
|
||||
_, body = await self._start()
|
||||
job_ids.append(body["job"]["id"])
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.counters["generated"], 2)
|
||||
self.assertEqual(len(written), 2)
|
||||
for dst in written:
|
||||
self.assertTrue(dst.is_file())
|
||||
made = sorted((self.output_dir / "thumbnails").glob("*_thumb.png"))
|
||||
self.assertEqual(made, sorted(written))
|
||||
self.assertTrue(any("Thumbnails cancelled" in line for line in job.log))
|
||||
|
||||
async def test_a_second_thumbnail_job_is_refused_naming_the_first(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, first = await self._start()
|
||||
await self._wait(entered)
|
||||
|
||||
status, body = await self._start(mode="rebuild")
|
||||
|
||||
self.assertEqual((status, body["code"]), (409, "busy"))
|
||||
self.assertEqual(body["running"]["id"], first["job"]["id"])
|
||||
release.set()
|
||||
await self._finished(first["job"]["id"])
|
||||
|
||||
async def test_thumbnail_job_refused_while_scan_runs(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
self._run_scan()
|
||||
|
||||
status, body = await self._start()
|
||||
|
||||
self.assertEqual((status, body["code"]), (409, "busy"))
|
||||
self.assertEqual(body["running"]["title"], "Scan for metadata")
|
||||
self.assertFalse(self._thumb_path("a").exists())
|
||||
|
||||
async def test_scan_refused_while_thumbnail_job_runs(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start()
|
||||
await self._wait(entered)
|
||||
|
||||
resp = await self.client.post("/prompt_manager/jobs/scan")
|
||||
body = await resp.json()
|
||||
|
||||
self.assertEqual((resp.status, body["code"]), (409, "busy"))
|
||||
self.assertEqual(body["running"]["title"], "Thumbnail generation")
|
||||
release.set()
|
||||
await self._finished(started["job"]["id"])
|
||||
|
||||
async def test_clear_refused_while_thumbnail_job_runs(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
kept = make_png(self.output_dir / "thumbnails" / "old_thumb.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start()
|
||||
await self._wait(entered)
|
||||
|
||||
resp = await self.client.post("/prompt_manager/images/clear-thumbnails")
|
||||
|
||||
self.assertEqual(resp.status, 409)
|
||||
self.assertEqual(
|
||||
await resp.json(),
|
||||
{"success": False, "error": "Thumbnail generation already running"},
|
||||
)
|
||||
self.assertTrue(kept.exists())
|
||||
release.set()
|
||||
await self._finished(started["job"]["id"])
|
||||
|
||||
async def test_every_exit_sets_a_truthful_final_status_line(self):
|
||||
"""I3: finished, nothing to do, cancelled and failed each say what happened."""
|
||||
lines = self._status_lines()
|
||||
make_png(self.output_dir / "a.png")
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG nope")
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.status_line, "1 preview written · 1 error")
|
||||
|
||||
# Every preview fresh: the job finishes with nothing to do.
|
||||
source_time = (self.output_dir / "a.png").stat().st_mtime
|
||||
bad_thumb = make_png(self._thumb_path("bad"))
|
||||
for thumb in (self._thumb_path("a"), bad_thumb):
|
||||
os.utime(thumb, (source_time + 10, source_time + 10))
|
||||
os.utime(self.output_dir / "bad.png", (source_time, source_time))
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual((job.state, job.total), ("finished", 0))
|
||||
self.assertEqual(
|
||||
job.status_line, "No previews to generate · 2 already up to date"
|
||||
)
|
||||
self.assertNotIn("Processing file 0 of 0", lines)
|
||||
|
||||
async def test_an_empty_library_says_there_is_nothing_to_preview(self):
|
||||
lines = self._status_lines()
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(
|
||||
(job.state, job.status_line), ("finished", "No images or videos to preview")
|
||||
)
|
||||
self.assertNotIn("Processing file 0 of 0", lines)
|
||||
|
||||
async def test_a_cancelled_job_says_what_it_kept(self):
|
||||
for i in range(6):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
self._one_worker()
|
||||
real_write = images_module._write_image_thumbnail
|
||||
job_ids = []
|
||||
written = []
|
||||
|
||||
def counting(src, dst, *args, **kwargs):
|
||||
real_write(src, dst, *args, **kwargs)
|
||||
written.append(dst)
|
||||
if len(written) == 1:
|
||||
self.api._jobs.cancel(job_ids[0])
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", counting):
|
||||
_, body = await self._start()
|
||||
job_ids.append(body["job"]["id"])
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.status_line, "Cancelled · 1 preview written")
|
||||
|
||||
async def test_an_internal_error_says_the_job_failed(self):
|
||||
def broken(*_args, **_kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
self.api._plan_gallery_thumbnails = broken
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(
|
||||
(job.state, job.status_line),
|
||||
("failed", "Thumbnail generation stopped on an internal error."),
|
||||
)
|
||||
|
||||
async def test_cancel_during_planning_stops_the_walk_and_starts_nothing(self):
|
||||
"""M2: the walk checks Cancel per file; no ``start`` frame is published."""
|
||||
for i in range(10):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.addCleanup(release.set)
|
||||
checked = []
|
||||
real_fresh = images_module._thumbnail_is_fresh
|
||||
|
||||
def gated_fresh(src, dst):
|
||||
checked.append(src.name)
|
||||
if len(checked) == 1:
|
||||
entered.set()
|
||||
release.wait(10)
|
||||
return real_fresh(src, dst)
|
||||
|
||||
with patch.object(images_module, "_thumbnail_is_fresh", gated_fresh):
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
self.assertTrue(self.api._jobs.cancel(job.id))
|
||||
release.set()
|
||||
events = parse_sse(await (await stream).text())
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
|
||||
self.assertEqual(checked, ["p0.png"]) # the walk stopped at the next file
|
||||
self.assertEqual([name for name, _ in events], ["job", "status", "error"])
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertEqual(job.status_line, "Cancelled before any preview was written")
|
||||
self.assertEqual(job.total, 0)
|
||||
self.assertFalse((self.output_dir / "thumbnails").exists())
|
||||
|
||||
async def test_a_locked_preview_is_a_readable_coded_error(self):
|
||||
"""M6: Windows refuses to replace a preview another process has open."""
|
||||
make_png(self.output_dir / "a.png")
|
||||
locked = PermissionError(13, "Access is denied")
|
||||
locked.winerror = 5
|
||||
|
||||
def refuse(*_args, **_kwargs):
|
||||
raise locked
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", refuse):
|
||||
_, body = await self._start(mode="rebuild")
|
||||
job = await self._finished(body["job"]["id"])
|
||||
|
||||
self.assertEqual(job.counters["errors"], 1)
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE)
|
||||
self.assertNotIn("WinError", repr(job.detail()))
|
||||
self.assertTrue(any(images_module.PREVIEW_IN_USE in line for line in job.log))
|
||||
|
||||
async def test_a_locked_video_preview_is_the_same_coded_error(self):
|
||||
clip = self.output_dir / "clip.mp4"
|
||||
clip.write_bytes(b"\x00")
|
||||
locked = PermissionError(13, "Access is denied")
|
||||
locked.winerror = 32
|
||||
|
||||
def refuse(*_args, **_kwargs):
|
||||
raise locked
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "_atomic_save", refuse),
|
||||
):
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE)
|
||||
|
||||
async def test_a_permission_error_off_windows_names_the_permission(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
|
||||
def refuse(*_args, **_kwargs):
|
||||
raise PermissionError(13, "Permission denied")
|
||||
|
||||
with patch.object(images_module, "_write_image_thumbnail", refuse):
|
||||
_, body = await self._start()
|
||||
job = await self._finished(body["job"]["id"])
|
||||
[row] = job.outcomes["errors"]
|
||||
self.assertEqual(row["detail"], images_module.PREVIEW_NOT_WRITABLE)
|
||||
|
||||
async def test_ffmpeg_fallback_keeps_quality_and_aspect_ratio(self):
|
||||
"""M1: without OpenCV, ffmpeg keeps the chosen quality and the aspect ratio."""
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
commands = []
|
||||
|
||||
def fake_ffmpeg(cmd, **_kwargs):
|
||||
commands.append(cmd)
|
||||
Path(cmd[-1]).write_bytes(b"jpeg")
|
||||
return type("Done", (), {"returncode": 0, "stderr": ""})()
|
||||
|
||||
for quality, qscale in (("high", "2"), ("medium", "4"), ("low", "8")):
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=fake_ffmpeg),
|
||||
):
|
||||
_, body = await self._start(mode="rebuild", size=640, quality=quality)
|
||||
job = await self._finished(body["job"]["id"])
|
||||
self.assertEqual(job.counters["generated"], 1, quality)
|
||||
cmd = commands[-1]
|
||||
self.assertNotIn("-s", cmd)
|
||||
self.assertEqual(
|
||||
cmd[cmd.index("-vf") + 1],
|
||||
"scale=640:640:force_original_aspect_ratio=decrease",
|
||||
)
|
||||
self.assertEqual(cmd[cmd.index("-q:v") + 1], qscale, quality)
|
||||
|
||||
async def test_the_placeholder_uses_the_chosen_quality(self):
|
||||
(self.output_dir / "clip.mp4").write_bytes(b"\x00")
|
||||
saved = []
|
||||
real_save = images_module._atomic_save
|
||||
|
||||
def recording(img, dst, fmt=None, **kwargs):
|
||||
saved.append(kwargs)
|
||||
return real_save(img, dst, fmt, **kwargs)
|
||||
|
||||
with (
|
||||
patch.dict(sys.modules, {"cv2": None}),
|
||||
patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")),
|
||||
patch.object(images_module, "_atomic_save", recording),
|
||||
):
|
||||
_, body = await self._start(quality="low")
|
||||
await self._finished(body["job"]["id"])
|
||||
self.assertEqual(saved, [images_module.THUMBNAIL_ENCODINGS["low"]])
|
||||
|
||||
|
||||
# The legacy stream's frames and their keys, as the gallery page reads them.
|
||||
LEGACY_KEYS = {
|
||||
"job": {"id", "started"},
|
||||
"status": {"phase", "message"},
|
||||
"start": {
|
||||
"total_images",
|
||||
"skipped",
|
||||
"remaining",
|
||||
"phase",
|
||||
"image_count",
|
||||
"video_count",
|
||||
"message",
|
||||
},
|
||||
"progress": {
|
||||
"processed",
|
||||
"total_images",
|
||||
"generated",
|
||||
"skipped",
|
||||
"error_count",
|
||||
"percentage",
|
||||
"rate",
|
||||
"eta",
|
||||
"elapsed",
|
||||
"current_file",
|
||||
"file_type",
|
||||
"action",
|
||||
},
|
||||
"complete": {
|
||||
"count",
|
||||
"skipped",
|
||||
"total_images",
|
||||
"remaining",
|
||||
"errors",
|
||||
"error_count",
|
||||
"elapsed_time",
|
||||
"processing_rate",
|
||||
"message",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class TestLegacyStreamOnTheJob(ThumbnailJobCase):
|
||||
async def test_legacy_progress_events_unchanged_on_success(self):
|
||||
make_png(self.output_dir / "a.png", size=(400, 400))
|
||||
make_png(self.output_dir / "b.png")
|
||||
|
||||
resp = await self.client.post(f"{self.PROGRESS_URL}?quality=low")
|
||||
events = parse_sse(await resp.text())
|
||||
|
||||
self.assertEqual(
|
||||
[name for name, _ in events],
|
||||
["job", "status", "start", "progress", "progress", "complete"],
|
||||
)
|
||||
for name, data in events:
|
||||
self.assertEqual(set(data), LEGACY_KEYS[name], name)
|
||||
complete = events[-1][1]
|
||||
self.assertEqual((complete["count"], complete["total_images"]), (2, 2))
|
||||
self.assertEqual(
|
||||
events[1][1],
|
||||
{"phase": "scanning", "message": "Scanning for images and videos..."},
|
||||
)
|
||||
with Image.open(self._thumb_path("a")) as thumb:
|
||||
self.assertEqual(max(thumb.size), 150) # the legacy preset still sizes
|
||||
[job] = self.api._jobs.list()
|
||||
self.assertEqual(
|
||||
(job.kind, job.title, job.state, job.cancellable),
|
||||
("thumbnails", "Thumbnail generation", "finished", True),
|
||||
)
|
||||
self.assertEqual(job.counters["generated"], 2)
|
||||
# I1: the first frame names the job this stream started, for its Cancel.
|
||||
self.assertEqual(events[0][1], {"id": job.id, "started": True})
|
||||
|
||||
async def test_legacy_file_errors_still_stream(self):
|
||||
make_png(self.output_dir / "good.png")
|
||||
(self.output_dir / "bad.png").write_bytes(b"\x89PNG nope")
|
||||
|
||||
resp = await self.client.post(self.PROGRESS_URL)
|
||||
events = parse_sse(await resp.text())
|
||||
|
||||
self.assertEqual(
|
||||
[d for e, d in events if e == "file_error"],
|
||||
[{"file": "bad.png", "error": "Not a valid image file"}],
|
||||
)
|
||||
|
||||
async def test_thumbnail_job_survives_client_disconnect(self):
|
||||
"""I2: the stream sees its client go after batch 1; every batch still runs.
|
||||
|
||||
One worker thread makes batches of 4, so 9 files take three batches. The
|
||||
first file waits until the client is gone; the second batch waits until
|
||||
the stream has seen it go (its subscription dropped) and records whether
|
||||
the job was still running then. A stream that stopped the work on
|
||||
disconnect would leave 4 previews; the job writes all 9.
|
||||
"""
|
||||
for i in range(9):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
self._one_worker()
|
||||
first, closed = threading.Event(), threading.Event()
|
||||
state = {"calls": 0, "left": None, "running_when_left": None}
|
||||
original = self.api._generate_one
|
||||
|
||||
def gated(*args, **kwargs):
|
||||
state["calls"] += 1
|
||||
if state["calls"] == 1:
|
||||
first.set()
|
||||
closed.wait(10)
|
||||
elif state["calls"] == 5: # the second batch
|
||||
state["running_when_left"] = state["left"].wait(10)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
self.api._generate_one = gated
|
||||
self.addCleanup(closed.set)
|
||||
|
||||
resp = await self.client.post(self.PROGRESS_URL)
|
||||
self.assertEqual(resp.status, 200)
|
||||
await self._wait(first)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
_subscribed, state["left"] = self._watch_subscribers(job)
|
||||
resp.close() # the gallery tab went away
|
||||
closed.set()
|
||||
await self._wait(state["left"]) # the server saw the client go
|
||||
self.assertEqual(job.subscriber_count, 0)
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
|
||||
self.assertEqual(job.counters["generated"], 9)
|
||||
self.assertEqual(job.state, "finished")
|
||||
self.assertTrue(state["running_when_left"])
|
||||
for i in range(9):
|
||||
self.assertTrue(self._thumb_path(f"p{i}").is_file())
|
||||
|
||||
async def test_legacy_stream_attaches_to_a_running_job(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
_, started = await self._start(mode="missing", size=256)
|
||||
await self._wait(entered)
|
||||
|
||||
job = self.api._jobs.get(started["job"]["id"])
|
||||
subscribed, _left = self._watch_subscribers(job)
|
||||
attached = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(subscribed) # M4: an event with a ceiling, never a poll
|
||||
release.set()
|
||||
events = parse_sse(await (await attached).text())
|
||||
|
||||
self.assertEqual(events[-1][0], "complete")
|
||||
# I1: the stream names the job it did not start: the gallery never cancels it.
|
||||
self.assertEqual(events[0], ("job", {"id": job.id, "started": False}))
|
||||
# One job: the stream followed the running one with its own options.
|
||||
[listed] = self.api._jobs.list()
|
||||
self.assertIs(listed, job)
|
||||
self.assertEqual(job.scope["size"], 256)
|
||||
|
||||
async def test_cancel_from_activity_ends_the_legacy_stream_with_an_error(self):
|
||||
for i in range(2):
|
||||
make_png(self.output_dir / f"p{i}.png")
|
||||
entered, release = self._gate_generation()
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
job = self.api._jobs.active("thumbnails")
|
||||
self.assertTrue(self.api._jobs.cancel(job.id))
|
||||
release.set()
|
||||
|
||||
events = parse_sse(await (await stream).text())
|
||||
|
||||
self.assertEqual(events[-1][0], "error")
|
||||
self.assertEqual(events[-1][1]["error"], "Thumbnail generation was cancelled")
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
|
||||
async def test_a_job_started_while_the_stream_prepares_is_attached(self):
|
||||
"""M3: a job that starts during ``prepare`` is followed, not refused."""
|
||||
from aiohttp import web
|
||||
|
||||
make_png(self.output_dir / "a.png")
|
||||
entered, release = self._gate_generation()
|
||||
real_prepare = web.StreamResponse.prepare
|
||||
raced = []
|
||||
|
||||
async def prepare(response, request):
|
||||
if not raced and type(response) is web.StreamResponse:
|
||||
options, _ = thumbnail_options({"size": 256})
|
||||
raced.append(self.api._start_thumbnail_job(options))
|
||||
return await real_prepare(response, request)
|
||||
|
||||
with patch.object(web.StreamResponse, "prepare", prepare):
|
||||
stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL))
|
||||
await self._wait(entered)
|
||||
release.set()
|
||||
events = parse_sse(await (await stream).text())
|
||||
|
||||
[job] = raced
|
||||
self.assertEqual(events[0], ("job", {"id": job.id, "started": False}))
|
||||
self.assertEqual(events[-1][0], "complete")
|
||||
self.assertEqual(self.api._jobs.list(), [job])
|
||||
|
||||
async def test_a_refused_stream_names_no_job(self):
|
||||
make_png(self.output_dir / "a.png")
|
||||
self._run_scan()
|
||||
events = parse_sse(await (await self.client.post(self.PROGRESS_URL)).text())
|
||||
self.assertEqual(
|
||||
events,
|
||||
[
|
||||
(
|
||||
"error",
|
||||
{
|
||||
"error": "Scan for metadata already running",
|
||||
"message": "Scan for metadata already running",
|
||||
},
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+17
-5
@@ -42,10 +42,10 @@
|
||||
"statements": 98
|
||||
},
|
||||
"gallery.js": {
|
||||
"lines": 41,
|
||||
"branches": 63,
|
||||
"functions": 44,
|
||||
"statements": 41
|
||||
"lines": 45,
|
||||
"branches": 64,
|
||||
"functions": 48,
|
||||
"statements": 45
|
||||
},
|
||||
"html-escape.js": {
|
||||
"lines": 98,
|
||||
@@ -161,6 +161,12 @@
|
||||
"functions": 98,
|
||||
"statements": 97
|
||||
},
|
||||
"workspace/activity-kinds.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 95,
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/activity-scan.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 78,
|
||||
@@ -175,7 +181,7 @@
|
||||
},
|
||||
"workspace/activity.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 88,
|
||||
"branches": 91,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
},
|
||||
@@ -509,6 +515,12 @@
|
||||
"functions": 98,
|
||||
"statements": 98
|
||||
},
|
||||
"workspace/thumbnail-review.mjs": {
|
||||
"lines": 96,
|
||||
"branches": 93,
|
||||
"functions": 98,
|
||||
"statements": 96
|
||||
},
|
||||
"workspace/toast.mjs": {
|
||||
"lines": 98,
|
||||
"branches": 98,
|
||||
|
||||
+87
-13
@@ -1,4 +1,7 @@
|
||||
class GalleryManager {
|
||||
// A Cancel on a thumbnail job another tab or System started (#249 System S5, review I1).
|
||||
static THUMBNAILS_STOPPED_FOLLOWING = 'Stopped following thumbnail generation started elsewhere; it keeps running in Activity';
|
||||
|
||||
constructor() {
|
||||
this.images = [];
|
||||
this.currentPage = 1;
|
||||
@@ -1911,6 +1914,54 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** Ask the server to stop thumbnail job `jobId` at its next file boundary (#249 System S5). */
|
||||
async cancelThumbnailJob(jobId) {
|
||||
try {
|
||||
await this.api.post(`/prompt_manager/jobs/${encodeURIComponent(jobId)}/cancel`, {});
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.warn('The thumbnail job could not be cancelled:', error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Cancel for one thumbnail stream (#249 System S5, review I1). The stream's first frame, `job {id, started}`,
|
||||
* names the job it follows and whether it started it. `request()` stops that job by its id, and only when this
|
||||
* stream started it: a job another tab or System started keeps running, and the stream just stops following.
|
||||
* A request before the `job` frame keeps the stream open until the frame arrives; `ended()` (the stream closed
|
||||
* or refused without naming a job) leaves nothing to cancel.
|
||||
*/
|
||||
thumbnailCanceller(eventSource) {
|
||||
let job = null;
|
||||
let wanted = false;
|
||||
let done = false;
|
||||
const apply = () => {
|
||||
if (done || !wanted || !job) return;
|
||||
done = true;
|
||||
eventSource.close();
|
||||
if (job.started) this.cancelThumbnailJob(job.id);
|
||||
};
|
||||
return {
|
||||
identify(raw) {
|
||||
if (job) return;
|
||||
let data = null;
|
||||
try { data = JSON.parse(raw); } catch (error) { return; }
|
||||
if (typeof data?.id !== 'string' || !data.id) return;
|
||||
job = { id: data.id, started: data.started === true };
|
||||
apply();
|
||||
},
|
||||
request() { wanted = true; apply(); },
|
||||
/** True once the stream named a job this page did not start. */
|
||||
get following() { return job !== null && !job.started; },
|
||||
ended() {
|
||||
if (done || !wanted || job) return;
|
||||
done = true;
|
||||
eventSource.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async generateThumbnailsWithProgress(options) {
|
||||
const progressBar = document.getElementById('progressBar');
|
||||
const progressText = document.getElementById('progressText');
|
||||
@@ -1932,13 +1983,19 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
let cancelled = false;
|
||||
let resultData = null;
|
||||
const canceller = this.thumbnailCanceller(eventSource);
|
||||
eventSource.addEventListener('job', event => canceller.identify(event.data));
|
||||
|
||||
// Handle cancellation
|
||||
// Handle cancellation. The work is an Activity job that outlives this stream, so closing it no
|
||||
// longer stops anything: ask the server to stop this stream's job by its id (review I1).
|
||||
const cancelHandler = () => {
|
||||
if (cancelled) return;
|
||||
cancelled = true;
|
||||
eventSource.close();
|
||||
cancelBtn.removeEventListener('click', cancelHandler);
|
||||
cancelBtn.classList.add('hidden');
|
||||
reject(new Error('Thumbnail generation cancelled by user'));
|
||||
const following = canceller.following;
|
||||
canceller.request();
|
||||
reject(new Error(following ? GalleryManager.THUMBNAILS_STOPPED_FOLLOWING : 'Thumbnail generation cancelled by user'));
|
||||
};
|
||||
cancelBtn.addEventListener('click', cancelHandler);
|
||||
|
||||
@@ -2013,7 +2070,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Handle completion
|
||||
eventSource.addEventListener('complete', (event) => {
|
||||
if (cancelled) return;
|
||||
if (cancelled) { canceller.ended(); return; }
|
||||
resultData = JSON.parse(event.data);
|
||||
|
||||
// Update final progress
|
||||
@@ -2061,7 +2118,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Handle errors
|
||||
eventSource.addEventListener('error', (event) => {
|
||||
if (cancelled) return;
|
||||
if (cancelled) { canceller.ended(); return; }
|
||||
const data = JSON.parse(event.data);
|
||||
|
||||
// Clean up
|
||||
@@ -2074,7 +2131,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Handle EventSource errors
|
||||
eventSource.onerror = (error) => {
|
||||
if (cancelled) return;
|
||||
if (cancelled) { canceller.ended(); return; }
|
||||
console.error('EventSource error:', error);
|
||||
console.error('EventSource readyState:', eventSource.readyState);
|
||||
|
||||
@@ -2097,6 +2154,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
|
||||
// Stream ended without a complete/error frame (close() does not trigger this)
|
||||
eventSource.onclose = () => {
|
||||
if (cancelled) canceller.ended();
|
||||
if (cancelled || resultData) return;
|
||||
cancelBtn.removeEventListener('click', cancelHandler);
|
||||
cancelBtn.classList.add('hidden');
|
||||
@@ -2692,12 +2750,15 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
statusMessages.scrollTop = statusMessages.scrollHeight;
|
||||
};
|
||||
|
||||
// Handle cancel
|
||||
// Handle cancel: the stream's own job stops at its next file boundary (#249 System S5, review I1)
|
||||
let cancelStream = null;
|
||||
cancelBtn.addEventListener('click', () => {
|
||||
if (isCancelled) return;
|
||||
isCancelled = true;
|
||||
addStatusMessage('Cancellation requested...', 'warning');
|
||||
cancelBtn.disabled = true;
|
||||
cancelBtn.textContent = 'Cancelling...';
|
||||
cancelStream?.();
|
||||
});
|
||||
|
||||
// Handle close (only appears when done)
|
||||
@@ -2721,7 +2782,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
progressDetails,
|
||||
progressETA,
|
||||
addStatusMessage,
|
||||
isCancelled: () => isCancelled
|
||||
isCancelled: () => isCancelled,
|
||||
onCancel: handler => { cancelStream = handler; }
|
||||
});
|
||||
|
||||
// Show completion
|
||||
@@ -2741,10 +2803,14 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('Thumbnail generation failed:', error);
|
||||
addStatusMessage(`Generation failed: ${error.message}`, 'error');
|
||||
progressBar.classList.add('bg-pm-error');
|
||||
progressBar.classList.remove('bg-pm-accent');
|
||||
if (isCancelled) {
|
||||
addStatusMessage(error.message, 'warning');
|
||||
} else {
|
||||
console.error('Thumbnail generation failed:', error);
|
||||
addStatusMessage(`Generation failed: ${error.message}`, 'error');
|
||||
progressBar.classList.add('bg-pm-error');
|
||||
progressBar.classList.remove('bg-pm-accent');
|
||||
}
|
||||
} finally {
|
||||
// Show close button and hide cancel
|
||||
cancelBtn.classList.add('hidden');
|
||||
@@ -2762,7 +2828,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}, 100);
|
||||
}
|
||||
} async generateThumbnailsWithProgressInModal(uiElements) {
|
||||
const { progressText, progressPercent, progressBar, progressDetails, progressETA, addStatusMessage, isCancelled } = uiElements;
|
||||
const { progressText, progressPercent, progressBar, progressDetails, progressETA, addStatusMessage, isCancelled, onCancel } = uiElements;
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
addStatusMessage('Starting thumbnail generation...');
|
||||
@@ -2776,6 +2842,14 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
const eventSource = SseStream.connect(`/prompt_manager/images/generate-thumbnails/progress?quality=${encodeURIComponent(quality)}`, { method: 'POST' });
|
||||
|
||||
let resultData = null;
|
||||
const canceller = this.thumbnailCanceller(eventSource);
|
||||
eventSource.addEventListener('job', event => canceller.identify(event.data));
|
||||
// Cancel stops this stream's job by id, keeps what it wrote, and settles the dialog.
|
||||
onCancel?.(() => {
|
||||
const following = canceller.following;
|
||||
canceller.request();
|
||||
reject(new Error(following ? GalleryManager.THUMBNAILS_STOPPED_FOLLOWING : 'Thumbnail generation cancelled; previews already written are kept'));
|
||||
});
|
||||
|
||||
// Handle scanning/status events
|
||||
eventSource.addEventListener('status', (event) => {
|
||||
|
||||
@@ -14,7 +14,11 @@
|
||||
* - Log: `details.term` "Scan log", collapsed, holding the raw timestamped lines in `pre[tabindex=0]`.
|
||||
* - D89: the tiles, the outcome list, the openable rows and the log and cancel labels are the job kind's (`jobView`):
|
||||
* an image tagging job shows Tagged, Unchanged, Skipped and Errors, opens Tagged and Unchanged rows, and reads
|
||||
* "Tagging log" and "Cancel tagging"; another kind keeps the scan's tiles as "Job log" and "Cancel job".
|
||||
* "Tagging log" and "Cancel tagging".
|
||||
* - D111: the kind's descriptor (activity-kinds.mjs `viewOf`) decides them for every kind: a thumbnail job counts
|
||||
* Generated, Skipped and Errors; a kind without groups (the health check, a held legacy-page operation) shows no
|
||||
* tiles and no Results, never another kind's zeros. A job without a known total has an indeterminate `<progress>`;
|
||||
* only a job the server can stop shows Cancel.
|
||||
*
|
||||
* The summary follows the Activity store (one SSE stream per running job); the listed rows and the log are read from
|
||||
* `GET /prompt_manager/jobs/<id>` at most once a second while the job runs, and once more when it ends.
|
||||
@@ -23,7 +27,8 @@ import { createButton } from '../shared/toolkit/core.js';
|
||||
import { pageHeading } from './heading.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { parseJob, jobView, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { parseJob, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { viewOf, canCancel, showProgress } from './activity-kinds.mjs';
|
||||
import { jobDetail, CANCEL_FAILED_TOAST, createReviewMetadataButton, hasReviewItems } from './activity.mjs';
|
||||
import { downloadJobReport } from './activity-scan.mjs';
|
||||
import { createMediaViewer } from './media-viewer.mjs';
|
||||
@@ -71,7 +76,10 @@ export function parseDetail(raw) {
|
||||
/** The progress line: the server's while running, then what the job got through. */
|
||||
export function progressLine(job) {
|
||||
const files = `${number(job.processed)} of ${number(job.total)} ${noun(job.total, 'file')}`;
|
||||
if (job.state === 'running') return job.status_line || `${job.progress}%`;
|
||||
// D111: a job that never counted files (a held legacy job, the health check) says only what the server says.
|
||||
const counted = job.total > 0;
|
||||
if (job.state === 'running') return job.status_line || (counted ? `${job.progress}%` : '');
|
||||
if (!counted && (job.state === 'finished' || job.state === 'cancelled')) return job.status_line;
|
||||
if (job.state === 'finished') return `${files} processed`;
|
||||
if (job.state === 'cancelled') return `Cancelled after ${files}`;
|
||||
if (job.state === 'lost') return LOST_DETAIL;
|
||||
@@ -97,7 +105,7 @@ export function filterRows(rows, filter) {
|
||||
|
||||
/** The first outcome with files, so a view opens on what happened; the first outcome when nothing has yet. */
|
||||
export function defaultOutcome(counters, outcomes = JOB_VIEWS.scan.outcomes) {
|
||||
return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])[0];
|
||||
return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])?.[0] ?? null;
|
||||
}
|
||||
|
||||
function section(title, id) {
|
||||
@@ -179,9 +187,10 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
const viewer = createMediaViewer({ api, container: main, parentScope: scope, inspectionController: inspection,
|
||||
sender: createPromptSender({ scope }), onSelect: image => inspection.select(image) });
|
||||
|
||||
/** The tiles and the outcome list of the job's kind, built once. */
|
||||
function layout(kind) {
|
||||
view = jobView(kind); // D89
|
||||
/** The tiles and the outcome list of the job's kind, built once (D89, D111: only the groups it counts). */
|
||||
function layout(job) {
|
||||
view = viewOf(job);
|
||||
if (!view.tiles.length) stats.remove();
|
||||
for (const [group, label, tone] of view.tiles) {
|
||||
const tile = element('div', undefined, tone ? `stat ${tone}` : 'stat');
|
||||
const value = element('dd', '0');
|
||||
@@ -203,10 +212,11 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
function showBody(job) {
|
||||
if (mounted) return;
|
||||
mounted = true;
|
||||
if (!view) layout(job.kind);
|
||||
if (!view) layout(job);
|
||||
missing.remove();
|
||||
parts.actions.append(download, cancel);
|
||||
main.append(progressPart.node, resultsPart.node, log);
|
||||
// A kind without outcome groups has no Results to browse (D111).
|
||||
main.append(...(view.outcomes.length ? [progressPart.node, resultsPart.node, log] : [progressPart.node, log]));
|
||||
selected ??= defaultOutcome(job.counters, view.outcomes);
|
||||
}
|
||||
|
||||
@@ -242,6 +252,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
}
|
||||
|
||||
function renderRows(job) {
|
||||
if (!view.outcomes.length) return;
|
||||
const rows = detail?.outcomes[selected] ?? [];
|
||||
const openable = view.openable.has(selected);
|
||||
const wanted = filterRows(rows.map((row, index) => ({ ...row, index })), input.value);
|
||||
@@ -279,7 +290,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
if (hasReviewItems(job)) { if (!review.isConnected) download.after(review); } // D87
|
||||
else review.remove();
|
||||
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel.
|
||||
if (job.state === 'running' && job.cancellable) {
|
||||
if (canCancel(job)) {
|
||||
if (!cancel.isConnected) parts.actions.append(cancel);
|
||||
} else if (cancel.isConnected) {
|
||||
// A focused Cancel that goes away hands its focus to the job's heading, never to the page body.
|
||||
@@ -287,8 +298,8 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
cancel.remove();
|
||||
}
|
||||
status.textContent = progressLine(job);
|
||||
bar.value = job.progress;
|
||||
bar.textContent = `${job.progress}%`;
|
||||
// D111: no value while the total is unknown; a job that ended with nothing counted shows how it ended.
|
||||
showProgress(bar, job.total > 0 || job.state !== 'finished' ? job : { ...job, progress: 100 });
|
||||
bar.setAttribute('aria-label', scan ? 'Scan progress' : `${job.title} progress`);
|
||||
for (const [group, value] of tileValues) value.textContent = number(job.counters[group] ?? 0);
|
||||
for (const [group, button] of buttons) {
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
/**
|
||||
* Per-kind Activity descriptors (#249 System S5, D111): one table that says, for each job kind, what Activity shows.
|
||||
*
|
||||
* `kindOf(kind)` → `{pillVerb, startToast(job), finishToast(job), line(job), kept(job), scope(job), tiles, outcomes,
|
||||
* openable, log, cancelLabel}`:
|
||||
* - `pillVerb`: the pill's verb ("Scanning"); null shows the job's title.
|
||||
* - `startToast` / `finishToast`: what a toast says when this tab starts the job and when it ends.
|
||||
* - `line`: the counters as one line ("N added · M errors"); `kept`: what a cancelled job kept ("N added"); `scope`:
|
||||
* what the job covers ("2 output folders").
|
||||
* - `tiles` (`[group, label, tone]`), `outcomes` (`[group, label]`) and `openable` (rows that open the viewer, D57).
|
||||
* - `log` and `cancelLabel` (`cancel` is the same label under the job view's older name).
|
||||
*
|
||||
* `scan` is exactly the D56 view and `image-tags` the D89 one. `thumbnails` counts Generated (green), Skipped and
|
||||
* Errors (red). Any other kind (the health check, a held legacy-page operation, a kind this page does not know) gets
|
||||
* the generic descriptor: title-based copy and no tiles or outcomes, so it never shows another kind's zeros (S1 review
|
||||
* M6, S2 review M3). `viewOf(job)` keeps only the groups the job itself counts, so a hold (no groups) has none either.
|
||||
*
|
||||
* Progress is indeterminate while a job runs without a known total (`indeterminate`): the `<progress>` has no value
|
||||
* and the pill shows no percentage, never a made-up 0%. A job shows Cancel only while it runs and the server can stop
|
||||
* it (`canCancel`).
|
||||
*/
|
||||
import { noun, countOf } from './plural.mjs';
|
||||
|
||||
// Grouped as the server writes its status lines ("8,240 of 19,622"), whatever the browser's locale.
|
||||
const number = value => value.toLocaleString('en-US');
|
||||
const counted = (job, group, word) => `${number(job.counters?.[group] ?? 0)} ${word}`;
|
||||
const errorsOf = job => `${number(job.counters?.errors ?? 0)} ${noun(job.counters?.errors ?? 0, 'error')}`;
|
||||
const whole = value => Number.isSafeInteger(value) && value >= 0;
|
||||
const tiles = rows => Object.freeze(rows.map(Object.freeze));
|
||||
const groups = (...names) => Object.freeze(new Set(names));
|
||||
|
||||
/** "1,420 added". */
|
||||
export const addedOf = job => counted(job, 'added', 'added');
|
||||
/** "12 tagged" (D89). */
|
||||
export const taggedOf = job => counted(job, 'tagged', 'tagged');
|
||||
/** "3 generated" (D111). */
|
||||
export const generatedOf = job => counted(job, 'generated', 'generated');
|
||||
|
||||
/** "2 output folders": what a scan or a thumbnail job walks. */
|
||||
function foldersOf(job) {
|
||||
const folders = job.scope?.output_folders;
|
||||
return whole(folders) ? countOf(folders, 'output folder') : '';
|
||||
}
|
||||
|
||||
/** "4 images" (D89), grouped as the server writes its lines. */
|
||||
function imagesOf(job) {
|
||||
const images = job.scope?.images;
|
||||
return whole(images) ? `${number(images)} ${noun(images, 'image')}` : '';
|
||||
}
|
||||
|
||||
const titleStarted = job => `${job.title} started.`;
|
||||
|
||||
function titleFinished(job) {
|
||||
if (job.state === 'cancelled') return `${job.title} was cancelled.`;
|
||||
if (job.state === 'failed') return `${job.title} failed.`;
|
||||
return `${job.title} finished.`;
|
||||
}
|
||||
|
||||
/** "<Noun> finished · <line>", "<Noun> cancelled · <kept>", "<Noun> failed · <line>". */
|
||||
const namedFinish = (name, kept, line) => job => {
|
||||
if (job.state === 'cancelled') return `${name(job)} cancelled · ${kept(job)}`;
|
||||
return `${name(job)} ${job.state === 'failed' ? 'failed' : 'finished'} · ${line(job)}`;
|
||||
};
|
||||
|
||||
const nothing = () => '';
|
||||
const GENERIC = Object.freeze({
|
||||
pillVerb: null, startToast: titleStarted, finishToast: titleFinished, line: nothing, kept: nothing,
|
||||
scope: foldersOf, tiles: tiles([]), outcomes: tiles([]), openable: groups(), log: 'Job log', cancelLabel: 'Cancel job',
|
||||
cancel: 'Cancel job',
|
||||
});
|
||||
|
||||
const scanLine = job => `${addedOf(job)} · ${errorsOf(job)}`;
|
||||
const tagLine = job => `${taggedOf(job)} · ${errorsOf(job)}`;
|
||||
const thumbnailLine = job => `${generatedOf(job)} · ${errorsOf(job)}`;
|
||||
// D85: an image tagging job says which of its two modes started.
|
||||
const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' });
|
||||
|
||||
/** Every kind with a descriptor of its own. */
|
||||
export const KINDS = Object.freeze({
|
||||
__proto__: null,
|
||||
// D56: the scan's tiles and outcome list in the mockup's orders; what got added and what went wrong first.
|
||||
scan: Object.freeze({
|
||||
...GENERIC,
|
||||
pillVerb: 'Scanning',
|
||||
startToast: () => 'Scan started.',
|
||||
finishToast: namedFinish(() => 'Scan', addedOf, scanLine),
|
||||
line: scanLine,
|
||||
kept: addedOf,
|
||||
tiles: tiles([['added', 'Added to Library', 'added'], ['known', 'Already known', ''],
|
||||
['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]),
|
||||
outcomes: tiles([['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
|
||||
['no_metadata', 'No usable metadata'], ['known', 'Already known']]),
|
||||
openable: groups('added', 'known'),
|
||||
log: 'Scan log',
|
||||
cancelLabel: 'Cancel scan',
|
||||
cancel: 'Cancel scan',
|
||||
}),
|
||||
// D89: Tagged (green), Unchanged, Skipped (warning) and Errors (red); Tagged and Unchanged rows open the viewer.
|
||||
'image-tags': Object.freeze({
|
||||
...GENERIC,
|
||||
pillVerb: 'Tagging',
|
||||
startToast: job => IMAGE_TAG_START_TOASTS[job.scope?.mode] ?? titleStarted(job),
|
||||
finishToast: namedFinish(job => job.title, taggedOf, tagLine),
|
||||
line: tagLine,
|
||||
kept: taggedOf,
|
||||
scope: imagesOf,
|
||||
tiles: tiles([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'],
|
||||
['errors', 'Errors', 'bad']]),
|
||||
outcomes: tiles([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'], ['unchanged', 'Unchanged']]),
|
||||
openable: groups('tagged', 'unchanged'),
|
||||
log: 'Tagging log',
|
||||
cancelLabel: 'Cancel tagging',
|
||||
cancel: 'Cancel tagging',
|
||||
}),
|
||||
// D111: Generated (green), Skipped (already fresh) and Errors (red); a row with a preview opens the viewer.
|
||||
thumbnails: Object.freeze({
|
||||
...GENERIC,
|
||||
pillVerb: 'Generating thumbnails',
|
||||
finishToast: namedFinish(() => 'Thumbnails', generatedOf, thumbnailLine),
|
||||
line: thumbnailLine,
|
||||
kept: generatedOf,
|
||||
tiles: tiles([['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]),
|
||||
outcomes: tiles([['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]),
|
||||
openable: groups('generated', 'skipped'),
|
||||
}),
|
||||
});
|
||||
|
||||
/** The descriptor of a job `kind`; any kind without one gets the generic descriptor. */
|
||||
export function kindOf(kind) {
|
||||
return (typeof kind === 'string' && KINDS[kind]) || GENERIC;
|
||||
}
|
||||
|
||||
const countsOf = job => job.counters && typeof job.counters === 'object' ? job.counters : {};
|
||||
|
||||
/**
|
||||
* A legacy-page operation held on the registry (D112): not cancellable and counting no outcome groups. Whatever its
|
||||
* kind (the gallery's thumbnail clear is a `thumbnails` hold), it reads as the generic job under its own title.
|
||||
*/
|
||||
export const held = job => job.cancellable === false && Object.keys(countsOf(job)).length === 0;
|
||||
|
||||
/**
|
||||
* The descriptor for one job: its kind's, keeping only the outcome groups the job counts; a held job's is the generic
|
||||
* one.
|
||||
*/
|
||||
export function viewOf(job) {
|
||||
if (held(job)) return GENERIC;
|
||||
const view = kindOf(job.kind);
|
||||
const counts = countsOf(job);
|
||||
const has = ([group]) => Object.hasOwn(counts, group);
|
||||
if (view.tiles.every(has) && view.outcomes.every(has)) return view;
|
||||
return Object.freeze({ ...view, tiles: tiles(view.tiles.filter(has)), outcomes: tiles(view.outcomes.filter(has)),
|
||||
openable: groups(...[...view.openable].filter(group => Object.hasOwn(counts, group))) });
|
||||
}
|
||||
|
||||
/** A running job whose total the server has not reported (yet) has no percentage. */
|
||||
export function indeterminate(job) {
|
||||
return job.state === 'running' && !(job.total > 0);
|
||||
}
|
||||
|
||||
/** Set a `<progress>` to the job's: its value and "N%", or no value at all while indeterminate. */
|
||||
export function showProgress(node, job) {
|
||||
if (indeterminate(job)) {
|
||||
node.removeAttribute('value');
|
||||
node.textContent = '';
|
||||
return;
|
||||
}
|
||||
node.value = job.progress;
|
||||
node.textContent = `${job.progress}%`;
|
||||
}
|
||||
|
||||
/** Truth rule: Cancel only while the job runs and the server can stop it (never on a held legacy job). */
|
||||
export function canCancel(job) {
|
||||
return job.state === 'running' && job.cancellable !== false;
|
||||
}
|
||||
|
||||
/** The pill's text: "Scanning · 42% · 8,240 / 19,622"; the file counts once the total is known, no "%" before. */
|
||||
export function pillParts(job) {
|
||||
const verb = viewOf(job).pillVerb ?? job.title;
|
||||
const label = indeterminate(job) ? verb : `${verb} · ${job.progress}%`;
|
||||
const files = job.total > 0 ? ` · ${number(job.processed)} / ${number(job.total)}` : '';
|
||||
return { label, files, text: label + files };
|
||||
}
|
||||
|
||||
/** The toast after this tab starts a job. */
|
||||
export const startToast = job => viewOf(job).startToast(job);
|
||||
/** The toast when a job ends. */
|
||||
export const finishToast = job => viewOf(job).finishToast(job);
|
||||
/** The counters as one line; '' for a kind without one. */
|
||||
export const outcomeLine = job => viewOf(job).line(job);
|
||||
@@ -12,7 +12,9 @@
|
||||
* the finish toast follows any job this store saw running when it ends.
|
||||
*/
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { KINDS, kindOf, finishToast, startToast } from './activity-kinds.mjs';
|
||||
|
||||
export { addedOf, taggedOf, finishToast, startToast, pillParts, outcomeLine } from './activity-kinds.mjs';
|
||||
|
||||
export const JOBS_URL = '/prompt_manager/jobs';
|
||||
export const BACKOFF_FIRST_MS = 1000;
|
||||
@@ -23,51 +25,13 @@ export const LOST_DETAIL = 'This job ended when ComfyUI restarted.';
|
||||
const SERVER_STATES = new Set(['running', 'finished', 'cancelled', 'failed']);
|
||||
const FINAL_STATES = new Set(['finished', 'cancelled', 'failed']);
|
||||
const JOB_ID = /^[0-9a-f]{32}$/;
|
||||
// The pill's verb for each job kind; a kind without one shows its title (D89: image tagging jobs read "Tagging").
|
||||
const PILL_VERBS = Object.freeze({ scan: 'Scanning', 'image-tags': 'Tagging' });
|
||||
const START_TOASTS = Object.freeze({ scan: 'Scan started.' });
|
||||
|
||||
/** The scan's job view (D56): its counter tiles `[group, label, tone]` and outcome list in the mockup's orders. */
|
||||
const SCAN_VIEW = Object.freeze({
|
||||
tiles: Object.freeze([
|
||||
['added', 'Added to Library', 'added'], ['known', 'Already known', ''], ['review', 'Needs review', 'warn'],
|
||||
['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad'],
|
||||
].map(Object.freeze)),
|
||||
// What got added and what went wrong first.
|
||||
outcomes: Object.freeze([
|
||||
['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'],
|
||||
['no_metadata', 'No usable metadata'], ['known', 'Already known'],
|
||||
].map(Object.freeze)),
|
||||
// Rows that open the file in the viewer (D57); the others are plain text.
|
||||
openable: Object.freeze(new Set(['added', 'known'])),
|
||||
log: 'Scan log',
|
||||
cancel: 'Cancel scan',
|
||||
});
|
||||
|
||||
/**
|
||||
* Each job kind's view: its tiles, outcome list, openable rows, log label and cancel label. D89: an image tagging job
|
||||
* (`image-tags`) counts Tagged (green), Unchanged, Skipped (warning) and Errors (red), lists what got tagged and what
|
||||
* went wrong first, and opens Tagged and Unchanged rows in the viewer like the scan's (D57).
|
||||
* Each job kind's copy and view live in one table (activity-kinds.mjs, D111): the scan's (D56), image tagging's (D89),
|
||||
* thumbnails' and the generic one. The names below are that table's, kept for this store's importers.
|
||||
*/
|
||||
export const JOB_VIEWS = Object.freeze({
|
||||
scan: SCAN_VIEW,
|
||||
'image-tags': Object.freeze({
|
||||
tiles: Object.freeze([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'],
|
||||
['errors', 'Errors', 'bad']].map(Object.freeze)),
|
||||
outcomes: Object.freeze([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'],
|
||||
['unchanged', 'Unchanged']].map(Object.freeze)),
|
||||
openable: Object.freeze(new Set(['tagged', 'unchanged'])),
|
||||
log: 'Tagging log',
|
||||
cancel: 'Cancel tagging',
|
||||
}),
|
||||
});
|
||||
// Another kind keeps the scan's tiles and outcomes under the generic labels.
|
||||
const GENERIC_VIEW = Object.freeze({ ...SCAN_VIEW, log: 'Job log', cancel: 'Cancel job' });
|
||||
|
||||
export const JOB_VIEWS = KINDS;
|
||||
/** The view of a job `kind`; an unknown kind gets the generic one. */
|
||||
export function jobView(kind) {
|
||||
return Object.hasOwn(JOB_VIEWS, kind) ? JOB_VIEWS[kind] : GENERIC_VIEW;
|
||||
}
|
||||
export const jobView = kindOf;
|
||||
|
||||
/** The wait before the `attempt`-th re-read (0-based): 1 s, 2 s, 4 s … never more than 30 s. */
|
||||
export function backoffDelay(attempt) {
|
||||
@@ -108,57 +72,6 @@ export function parseJob(raw) {
|
||||
};
|
||||
}
|
||||
|
||||
// Grouped as the server writes its status lines ("8,240 of 19,622"), whatever the browser's locale.
|
||||
const number = value => value.toLocaleString('en-US');
|
||||
const errorsOf = job => `${number(job.counters.errors ?? 0)} ${noun(job.counters.errors ?? 0, 'error')}`;
|
||||
/** "1,420 added". */
|
||||
export const addedOf = job => `${number(job.counters.added ?? 0)} added`;
|
||||
|
||||
/**
|
||||
* The toast when a job ends: "Scan finished · N added · M errors", "Scan cancelled · N added"; an image tagging job
|
||||
* the same way with its own title and count (D89): "Auto-tag images finished · N tagged · M errors".
|
||||
*/
|
||||
export function finishToast(job) {
|
||||
if (job.kind === 'scan') {
|
||||
if (job.state === 'cancelled') return `Scan cancelled · ${addedOf(job)}`;
|
||||
if (job.state === 'failed') return `Scan failed · ${addedOf(job)} · ${errorsOf(job)}`;
|
||||
return `Scan finished · ${addedOf(job)} · ${errorsOf(job)}`;
|
||||
}
|
||||
if (job.kind === 'image-tags') {
|
||||
if (job.state === 'cancelled') return `${job.title} cancelled · ${taggedOf(job)}`;
|
||||
return `${job.title} ${job.state === 'failed' ? 'failed' : 'finished'} · ${outcomeLine(job)}`;
|
||||
}
|
||||
if (job.state === 'cancelled') return `${job.title} was cancelled.`;
|
||||
if (job.state === 'failed') return `${job.title} failed.`;
|
||||
return `${job.title} finished.`;
|
||||
}
|
||||
|
||||
// D85: an image tagging job says which of its two modes started.
|
||||
const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' });
|
||||
|
||||
/** The toast after this tab starts a job. */
|
||||
export function startToast(job) {
|
||||
if (job.kind === 'image-tags' && IMAGE_TAG_START_TOASTS[job.scope.mode]) return IMAGE_TAG_START_TOASTS[job.scope.mode];
|
||||
return START_TOASTS[job.kind] ?? `${job.title} started.`;
|
||||
}
|
||||
|
||||
/** The pill's text: "Scanning · 42% · 8,240 / 19,622"; the file counts only once the total is known. */
|
||||
export function pillParts(job) {
|
||||
const label = `${PILL_VERBS[job.kind] ?? job.title} · ${job.progress}%`;
|
||||
const files = job.total > 0 ? ` · ${number(job.processed)} / ${number(job.total)}` : '';
|
||||
return { label, files, text: label + files };
|
||||
}
|
||||
|
||||
/** "12 tagged" (D89). */
|
||||
export const taggedOf = job => `${number(job.counters.tagged ?? 0)} tagged`;
|
||||
|
||||
/** The counters as one line: "N added · M errors" for a scan, "N tagged · M errors" for image tagging (D89). */
|
||||
export function outcomeLine(job) {
|
||||
if (job.kind === 'scan') return `${addedOf(job)} · ${errorsOf(job)}`;
|
||||
if (job.kind === 'image-tags') return `${taggedOf(job)} · ${errorsOf(job)}`;
|
||||
return '';
|
||||
}
|
||||
|
||||
function defaultConnect(url) {
|
||||
const resolve = globalThis.ApiClient?.resolveUrl ?? (value => value);
|
||||
return new EventSource(resolve(url));
|
||||
|
||||
@@ -14,13 +14,15 @@
|
||||
* - D87 (plan D64): an ended scan that recorded Needs review items carries the concept's **Review metadata**
|
||||
* (`go-review`, app.js a738) after its line, which opens `#/review`; its chip still names its state (D54).
|
||||
* - D89: an image tagging job's line names its images and, once ended, "N tagged · M errors".
|
||||
* - D111: what a line says (scope, counters, what a cancel kept) is the job kind's descriptor (activity-kinds.mjs); a
|
||||
* job without a known total shows an indeterminate `<progress>`, and only a job the server can stop shows Cancel.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { navigateTo } from './state.mjs';
|
||||
import { pageHeading } from './heading.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { countOf, noun } from './plural.mjs';
|
||||
import { pillParts, outcomeLine, addedOf, taggedOf, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { pillParts, outcomeLine, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { viewOf, canCancel, showProgress, indeterminate } from './activity-kinds.mjs';
|
||||
|
||||
export const ACTIVITY_DESCRIPTION = 'Scans, tagging, downloads and maintenance share one history.';
|
||||
export const EMPTY_ACTIVITY = 'No jobs have run since ComfyUI started. Scans and other long operations appear here.';
|
||||
@@ -60,20 +62,9 @@ export function clockTime(seconds) {
|
||||
|
||||
const join = parts => parts.filter(Boolean).join(' · ');
|
||||
|
||||
const whole = value => Number.isSafeInteger(value) && value >= 0;
|
||||
|
||||
/** What a job covers: "2 output folders" for a scan, "4 images" for image tagging (D89). */
|
||||
function scopeLine(job) {
|
||||
// D89: grouped as the server writes its lines ("19,622 images").
|
||||
const images = job.scope?.images;
|
||||
if (job.kind === 'image-tags') return whole(images) ? `${images.toLocaleString('en-US')} ${noun(images, 'image')}` : '';
|
||||
const folders = job.scope?.output_folders;
|
||||
return whole(folders) ? countOf(folders, 'output folder') : '';
|
||||
}
|
||||
|
||||
/** What a cancelled job kept: "N added" for a scan, "N tagged" for image tagging (D89). */
|
||||
const KEPT = Object.freeze({ __proto__: null, scan: addedOf, 'image-tags': taggedOf }); // D89
|
||||
const keptOf = job => KEPT[job.kind]?.(job) ?? '';
|
||||
/** What a job covers ("2 output folders", "4 images") and what a cancelled one kept ("N added"), by kind (D111). */
|
||||
const scopeLine = job => viewOf(job).scope(job);
|
||||
const keptOf = job => viewOf(job).kept(job);
|
||||
|
||||
/** The item's line under its title, by state. */
|
||||
export function jobDetail(job) {
|
||||
@@ -119,7 +110,10 @@ export function mountActivityTopbar({ store, scope, doc = document }) {
|
||||
const parts = pillParts(job);
|
||||
label.textContent = parts.label;
|
||||
files.textContent = parts.files;
|
||||
fill.style.width = `${job.progress}%`;
|
||||
// D111 (S5 review M5): no total, no fill: the bar sweeps instead of showing an empty determinate 0%.
|
||||
const unknown = indeterminate(job);
|
||||
bar.classList.toggle('indeterminate', unknown);
|
||||
fill.style.width = unknown ? '' : `${job.progress}%`;
|
||||
pill.href = `#/activity/${job.id}`;
|
||||
pill.setAttribute('aria-label', parts.text);
|
||||
};
|
||||
@@ -167,15 +161,14 @@ function itemFor(job, scope) {
|
||||
// parentNode, not isConnected: on the first render the item is not in the page yet, so a job that had already
|
||||
// ended when Activity opened kept its Cancel job (and a running one's progress moved after its actions).
|
||||
if (running) {
|
||||
progress.value = next.progress;
|
||||
progress.textContent = `${next.progress}%`;
|
||||
showProgress(progress, next); // D111: no value while the total is unknown
|
||||
progress.setAttribute('aria-label', `${next.title} progress`);
|
||||
if (!progress.parentNode) item.insertBefore(progress, actions.parentNode ? actions : null);
|
||||
} else {
|
||||
progress.remove();
|
||||
}
|
||||
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel job.
|
||||
if (running && next.cancellable) {
|
||||
if (canCancel(next)) {
|
||||
if (!actions.parentNode) item.append(actions);
|
||||
} else if (actions.parentNode) {
|
||||
// A focused Cancel job that goes away hands its focus to the item's title link, never to the page body.
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
* N issues / Incomplete); the review, backup, thumbnail and sources rows say what the server reports, and nothing
|
||||
* reads 0 before the read answers: a load shows a status line, a failure an alert and Retry. The thumbnail row is
|
||||
* its own read of `/images/thumbnails/status` (a walk of every source): "Checking thumbnails…" until it answers.
|
||||
* D99: its Review thumbnails opens the same "Thumbnail maintenance" dialog as Sources (thumbnail-review.mjs).
|
||||
* - D96: the Statistics dialog lists the real runs and average rating per category, then "Library totals".
|
||||
* - D95: the "Health report" lists every check the server ran, Missing files per image source, with a chip per real
|
||||
* result and when it was checked. The check runs from Maintenance's notice ("Run health checks", D123) as a
|
||||
@@ -20,6 +21,7 @@ import { closeWorkspaceDialog, openContentDialog } from './workspace-dialogs.mjs
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { operationRow } from './system-controls.mjs';
|
||||
import { openThumbnailReview } from './thumbnail-review.mjs';
|
||||
|
||||
export const OVERVIEW_URL = '/prompt_manager/system/overview';
|
||||
export const STATISTICS_URL = '/prompt_manager/system/statistics';
|
||||
@@ -317,7 +319,7 @@ function surface(heading, rows) {
|
||||
}
|
||||
|
||||
/** The Overview's nodes, and its thumbnail row (painted by its own read). */
|
||||
function overviewNodes(overview, { scope, navigate, openStatistics }) {
|
||||
function overviewNodes(overview, { scope, navigate, openStatistics, openThumbnails }) {
|
||||
const go = section => () => navigate(`#/system/${section}`);
|
||||
const review = operationRow({ scope, title: ambiguousTitle(overview.needs_review),
|
||||
description: Number.isSafeInteger(overview.needs_review?.pending)
|
||||
@@ -325,12 +327,12 @@ function overviewNodes(overview, { scope, navigate, openStatistics }) {
|
||||
label: 'Review metadata', onClick: () => navigate('#/review') });
|
||||
const [sourcesTitle, sourcesDescription] = sourcesCopy(overview.sources);
|
||||
const thumbnails = operationRow({ scope, title: THUMBNAILS_LOADING, description: 'Generate only missing previews when necessary.',
|
||||
label: 'Review thumbnails', onClick: go('sources') });
|
||||
label: 'Review thumbnails', onClick: openThumbnails });
|
||||
const health = surface('Library health', [
|
||||
review,
|
||||
operationRow({ scope, title: 'Backup status', description: backupText(overview.backup), label: 'View backups',
|
||||
onClick: go('backups') }),
|
||||
// Until the thumbnail review dialog (S5), Review thumbnails opens Sources, where thumbnail generation lives.
|
||||
// D99: Review thumbnails opens the thumbnail dialog (the concept's row starts a sample job directly).
|
||||
thumbnails,
|
||||
]);
|
||||
const preferences = surface('Workspace preferences', [
|
||||
@@ -350,7 +352,7 @@ function overviewNodes(overview, { scope, navigate, openStatistics }) {
|
||||
* answer replaces in place (the concept's metrics and columns sit directly in `.settings-layout`). A read that a newer
|
||||
* one superseded, or one answering after the section left, paints nothing.
|
||||
*/
|
||||
export function mountOverview(container, { api, scope, navigate = navigateTo, notify = showWorkspaceToast,
|
||||
export function mountOverview(container, { api, activity, scope, navigate = navigateTo, notify = showWorkspaceToast,
|
||||
latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) {
|
||||
const overviewRead = latestOnly({ lifetime: scope.signal });
|
||||
const statisticsRead = latestOnly({ lifetime: scope.signal });
|
||||
@@ -392,7 +394,8 @@ export function mountOverview(container, { api, scope, navigate = navigateTo, no
|
||||
try {
|
||||
const overview = await overviewRead(signal => api.get(OVERVIEW_URL, { signal, cache: 'no-store' }));
|
||||
if (overview === STALE || scope.signal.aborted) return;
|
||||
const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics });
|
||||
const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics,
|
||||
openThumbnails: trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate }) });
|
||||
thumbnailRow = thumbnails;
|
||||
sources = overview.sources;
|
||||
swap(nodes);
|
||||
|
||||
@@ -14,8 +14,8 @@
|
||||
* it to the host draft of `gallery_root_paths`; Remove takes one out. Both wait for the one Save row.
|
||||
* - "Watch for new outputs" (`monitoring_enabled`) and "Worker threads" (`worker_threads`, a range up to the server's
|
||||
* `cpu_count`, D113) are host drafts too. "Enrich existing metadata" reviews a scan of every source.
|
||||
* - "Thumbnail generation → Review options" opens the existing gallery's thumbnail tool until its review dialog (S5),
|
||||
* the D94 hand-off for the one capability not native yet.
|
||||
* - "Thumbnail generation → Review options" opens "Thumbnail maintenance" (thumbnail-review.mjs, S5 Task 14), which
|
||||
* starts the thumbnail Activity job.
|
||||
*
|
||||
* D98: per-source Remove; an added source reads "not saved yet" until Save; the line under the list says when
|
||||
* ComfyUI's output folder is used instead; "restart pending" for a source the running monitor does not watch yet; the
|
||||
@@ -28,6 +28,7 @@
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs';
|
||||
import { openMetadataScan } from './activity-scan.mjs';
|
||||
import { openThumbnailReview } from './thumbnail-review.mjs';
|
||||
import { settingRow, operationRow } from './system-controls.mjs';
|
||||
import { HOST_UNAVAILABLE } from './system-appearance.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
@@ -36,7 +37,6 @@ import { noun } from './plural.mjs';
|
||||
const SETTINGS_URL = '/prompt_manager/settings';
|
||||
const VALIDATE_URL = '/prompt_manager/settings/validate-source';
|
||||
const RESCAN_URL = '/prompt_manager/gallery/rescan';
|
||||
const GALLERY_URL = '/prompt_manager/gallery';
|
||||
const ROOTS_KEY = 'gallery_root_paths';
|
||||
|
||||
export const NOTICE_KEY = 'pm_update_notice_dismissed';
|
||||
@@ -222,8 +222,7 @@ const controlOf = (row, id) => row.querySelector(`[data-setting="${id}"]`);
|
||||
* Activity store (scans start through it), `drafts` the System drafts.
|
||||
*/
|
||||
export function mountSources(container, { api, activity, scope, drafts, notify = showWorkspaceToast, storage = defaultStorage,
|
||||
openWindow = (url) => window.open(url, '_blank', 'noopener'), latestOnly = window.ApiClient.latestOnly,
|
||||
STALE = window.ApiClient.STALE }) {
|
||||
latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) {
|
||||
const statusRead = latestOnly({ lifetime: scope.signal });
|
||||
// undefined while read, null when the read failed, else GET /settings' `sources`.
|
||||
let sources;
|
||||
@@ -270,9 +269,8 @@ export function mountSources(container, { api, activity, scope, drafts, notify =
|
||||
surface.append(monitoring, workers,
|
||||
operationRow({ scope, title: 'Enrich existing metadata', description: 'Resolve both prompts, deduplicate recipes, preserve uncertain links.',
|
||||
label: 'Review scan', onClick: scanAll }),
|
||||
// D94: the gallery keeps thumbnail generation until its review dialog (S5).
|
||||
operationRow({ scope, title: 'Thumbnail generation', description: 'Generate missing previews; rebuilding all is a separate explicit option.',
|
||||
label: 'Review options', onClick: () => openWindow(window.ApiClient.resolveUrl(GALLERY_URL)) }),
|
||||
label: 'Review options', onClick: trigger => openThumbnailReview({ api, store: activity, scope, trigger }) }),
|
||||
// D98: the legacy gallery's Refresh, which rereads every source's file list.
|
||||
operationRow({ scope, title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
|
||||
label: 'Refresh', onClick: refresh }));
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
/**
|
||||
* Thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (`thumbnail-review`, app.js
|
||||
* 1433–1437), opened by Sources' "Thumbnail generation → Review options" and Overview's "Review thumbnails". It starts
|
||||
* the `thumbnails` Activity job (S5 Task 12) with the chosen options.
|
||||
*
|
||||
* - Scope: "Missing thumbnails only" (`missing`, the default) or "Rebuild all (explicit opt-in)" (`rebuild`).
|
||||
* - Quality: "Medium · optimized" / "High · original" / "Low · fastest" (`medium` / `high` / `low`), preselected from
|
||||
* Appearance's Image loading quality (`gallerySettings.imageQuality`, the same three values; Q3).
|
||||
* - Preview size limit: the longest edge, a whole number from 128 to 2048 (concept default 512). A size outside that
|
||||
* marks the field `aria-invalid`, says so in the plan line and sends nothing; Start reports the field's validity.
|
||||
* - D99: a `p.small[role=status]` line before the actions says what the server's plan counts for the chosen scope
|
||||
* (`GET /images/thumbnails/plan`, read-only): "N previews to generate · M already up to date." or, for a rebuild,
|
||||
* "Every preview is regenerated: N files.", plus "· R more on the next run" past the per-run cap. Each scope is read
|
||||
* once per dialog (the size and quality change no count). The primary is "Start" (the concept's "Simulate thumbnail
|
||||
* job"): it closes the dialog and starts the job through the Activity store, which toasts that it started; the job
|
||||
* opens in Activity, where its progress lives. A start refused because another job holds the slot toasts the D57
|
||||
* busy link toast and stays.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs';
|
||||
import { busyToast } from './activity-scan.mjs';
|
||||
import { readPreferences } from './preferences.mjs';
|
||||
import { showWorkspaceToast } from './toast.mjs';
|
||||
import { navigateTo } from './state.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
|
||||
export const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan';
|
||||
export const THUMBNAILS_KIND = 'thumbnails';
|
||||
export const SCOPE_OPTIONS = Object.freeze([['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]);
|
||||
export const QUALITY_OPTIONS = Object.freeze([['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]);
|
||||
export const SIZE_LIMIT = Object.freeze({ min: 128, max: 2048, value: 512 });
|
||||
export const PLAN_LOADING = 'Counting previews…';
|
||||
export const PLAN_FAILED = 'The previews could not be counted.';
|
||||
export const SIZE_INVALID = `Enter a size from ${SIZE_LIMIT.min} to ${SIZE_LIMIT.max} pixels.`;
|
||||
export const START_FAILED = 'Thumbnail generation could not start. Try again.';
|
||||
|
||||
const QUALITIES = new Set(QUALITY_OPTIONS.map(([value]) => value));
|
||||
const count = value => value.toLocaleString('en-US');
|
||||
const isCount = value => Number.isSafeInteger(value) && value >= 0;
|
||||
|
||||
/** The preview size for the field's text: a whole number of pixels in range, else null. */
|
||||
export function previewSize(text) {
|
||||
if (typeof text !== 'string' || !/^\d{1,5}$/.test(text)) return null;
|
||||
const size = Number(text);
|
||||
return size >= SIZE_LIMIT.min && size <= SIZE_LIMIT.max ? size : null;
|
||||
}
|
||||
|
||||
/** Q3: Image loading quality preselects the dialog's quality; anything it does not know is Medium. */
|
||||
export function preselectedQuality(imageQuality) {
|
||||
return QUALITIES.has(imageQuality) ? imageQuality : 'medium';
|
||||
}
|
||||
|
||||
/** D99: the plan line for the server's `{mode, targets, fresh, remaining}`; an answer without counts failed. */
|
||||
export function planText(plan) {
|
||||
if (!plan || !['missing', 'rebuild'].includes(plan.mode) || ![plan.targets, plan.fresh, plan.remaining].every(isCount)) {
|
||||
return PLAN_FAILED;
|
||||
}
|
||||
const more = plan.remaining ? ` · ${count(plan.remaining)} more on the next run` : '';
|
||||
if (plan.mode === 'rebuild') return `Every preview is regenerated: ${count(plan.targets)} ${noun(plan.targets, 'file')}${more}.`;
|
||||
return `${count(plan.targets)} ${noun(plan.targets, 'preview')} to generate · ${count(plan.fresh)} already up to date${more}.`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Start the thumbnail job with `options` (`{mode, size, quality}`) and open its view; a refused or failed start says
|
||||
* why and stays. Nothing is said once the section that started it is gone.
|
||||
*/
|
||||
export async function startThumbnails({ store, scope, options, notify = showWorkspaceToast, navigate = navigateTo }) {
|
||||
try {
|
||||
const result = await store.start(THUMBNAILS_KIND, options);
|
||||
if (scope.signal.aborted) return;
|
||||
if (result.busy !== undefined) {
|
||||
notify(busyToast(result.busy));
|
||||
return;
|
||||
}
|
||||
navigate(`#/activity/${result.job.id}`);
|
||||
} catch {
|
||||
if (!scope.signal.aborted) notify(START_FAILED);
|
||||
}
|
||||
}
|
||||
|
||||
function el(tag, text, className) {
|
||||
const node = document.createElement(tag);
|
||||
if (text !== undefined) node.textContent = text;
|
||||
if (className) node.className = className;
|
||||
return node;
|
||||
}
|
||||
|
||||
function select(options, value) {
|
||||
const node = el('select');
|
||||
node.append(...options.map(([optionValue, label]) => {
|
||||
const option = el('option', label);
|
||||
option.value = optionValue;
|
||||
return option;
|
||||
}));
|
||||
node.value = value;
|
||||
return node;
|
||||
}
|
||||
|
||||
const defaultQuality = () => {
|
||||
try {
|
||||
return readPreferences().values.imageQuality;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Open "Thumbnail maintenance" on the section's `scope`. `store` is the page's Activity store; `trigger` gets focus
|
||||
* back when the dialog closes.
|
||||
*/
|
||||
export function openThumbnailReview({ api, store, scope, trigger = null, quality = defaultQuality(), notify = showWorkspaceToast,
|
||||
navigate = navigateTo }) {
|
||||
const mode = select(SCOPE_OPTIONS, 'missing');
|
||||
const encoder = select(QUALITY_OPTIONS, preselectedQuality(quality));
|
||||
const size = el('input');
|
||||
size.type = 'number';
|
||||
size.value = String(SIZE_LIMIT.value);
|
||||
size.min = String(SIZE_LIMIT.min);
|
||||
size.max = String(SIZE_LIMIT.max);
|
||||
const line = el('p', PLAN_LOADING, 'small');
|
||||
line.setAttribute('role', 'status');
|
||||
const cancel = createButton('Cancel');
|
||||
const start = createButton('Start', 'ca-primary');
|
||||
const actions = el('div', undefined, 'dialog-actions');
|
||||
actions.append(cancel, start);
|
||||
|
||||
// The plan per scope: a promise of its answer (null when it failed); the size and quality change no count.
|
||||
const plans = new Map();
|
||||
let open = true;
|
||||
const chosenSize = () => previewSize(size.value);
|
||||
const validate = () => {
|
||||
const valid = chosenSize() !== null;
|
||||
size.setCustomValidity(valid ? '' : SIZE_INVALID);
|
||||
if (valid) size.removeAttribute('aria-invalid');
|
||||
else size.setAttribute('aria-invalid', 'true');
|
||||
return valid;
|
||||
};
|
||||
const readPlan = chosen => {
|
||||
const query = new URLSearchParams({ mode: chosen, size: String(chosenSize()), quality: encoder.value });
|
||||
return api.get(`${THUMBNAIL_PLAN_URL}?${query}`, { signal: scope.signal, cache: 'no-store' }).catch(() => null);
|
||||
};
|
||||
async function paint() {
|
||||
if (!validate()) { line.textContent = SIZE_INVALID; return; }
|
||||
const chosen = mode.value;
|
||||
if (!plans.has(chosen)) plans.set(chosen, readPlan(chosen));
|
||||
const pending = plans.get(chosen);
|
||||
line.textContent = PLAN_LOADING;
|
||||
const plan = await pending;
|
||||
// A later choice painted its own line meanwhile, or the dialog closed.
|
||||
if (!open || scope.signal.aborted || mode.value !== chosen || !validate()) return;
|
||||
line.textContent = planText(plan);
|
||||
}
|
||||
|
||||
scope.listen(mode, 'change', paint);
|
||||
scope.listen(size, 'input', paint);
|
||||
scope.listen(cancel, 'click', () => closeWorkspaceDialog());
|
||||
scope.listen(start, 'click', () => {
|
||||
if (!validate()) {
|
||||
size.reportValidity();
|
||||
size.focus();
|
||||
return;
|
||||
}
|
||||
const options = { mode: mode.value, size: chosenSize(), quality: encoder.value };
|
||||
closeWorkspaceDialog();
|
||||
startThumbnails({ store, scope, options, notify, navigate });
|
||||
});
|
||||
openContentDialog({ scope, title: 'Thumbnail maintenance', trigger,
|
||||
nodes: [field('Scope', mode, 'thumb-mode'), field('Quality', encoder, 'thumb-quality'),
|
||||
field('Preview size limit', size, 'thumb-limit'), line, actions],
|
||||
onClose: () => { open = false; } });
|
||||
paint();
|
||||
}
|
||||
@@ -24,6 +24,10 @@ main:focus-visible { outline: 2px solid var(--ca-focus); outline-offset: 4px; }
|
||||
.activity-pill:hover { border-color: var(--ca-accent); /* D53 */ }
|
||||
.activity-pill-bar { display: block; inline-size: 70px; block-size: 5px; border-radius: 3px; background: var(--line); overflow: hidden; /* D53 */ }
|
||||
.activity-pill-bar i { display: block; block-size: 100%; inline-size: 0; background: var(--ca-accent); /* D53 */ }
|
||||
/* D111 (S5): a running job without a known total sweeps a short segment; reduced motion shows a dimmed full bar. */
|
||||
.activity-pill-bar.indeterminate i { inline-size: 40%; animation: activity-pill-sweep 1.4s ease-in-out infinite; /* D111 */ }
|
||||
@keyframes activity-pill-sweep { from { transform: translateX(-100%); } to { transform: translateX(250%); } }
|
||||
@media (prefers-reduced-motion: reduce) { .activity-pill-bar.indeterminate i { inline-size: 100%; opacity: .45; animation: none; /* D111 */ } }
|
||||
@media (pointer: coarse) { .activity-pill:not([hidden]) { min-block-size: 40px; /* D53 */ } }
|
||||
@media (max-width: 700px) {
|
||||
/* D52: with the Activity button the full-width row can be wider than a zoomed phone, so its items wrap. */
|
||||
|
||||
Reference in New Issue
Block a user