diff --git a/TESTING.md b/TESTING.md index 93b42d3..7881dc3 100644 --- a/TESTING.md +++ b/TESTING.md @@ -1353,8 +1353,8 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f the selection bar's tagging shows is the concept's or the plan's (D85, D90, D91). - **D89** (plan D66, Activity for the `image-tags` job): the job's Activity item and its view are the scan's (D54, D56, D57, D59) with the job kind's own counters, from one - per-kind table (`JOB_VIEWS` in `activity-store.mjs`; a kind without one keeps the - scan's tiles under "Job log" and "Cancel job"). The titles are the server's: "Add + per-kind table (`activity-kinds.mjs` since S5, D111; a kind without one gets no + tiles, "Job log" and "Cancel job"). The titles are the server's: "Add tags to images" and "Auto-tag images". - The pill's verb is "Tagging" ("Tagging · 41% · 8,240 / 19,622"). - The item line and the view's heading line read "Started HH:MM · N images · closing @@ -1387,7 +1387,7 @@ targets of the longest legal path (1024 code points, even escaped as `\uXXXX`) f finished · N tagged · M errors", " cancelled · N tagged" or "<title> failed · N tagged · M errors". - Tests: `tests/js/workspace-activity-store.test.js` ("image-tags jobs get their own - tiles outcomes and pill verb", "unknown kinds keep the generic view"); + tiles outcomes and pill verb", "unknown kinds get the generic view without the scan tiles"); `tests/browser/workspace-activity.spec.mjs` ("an image tagging job shows Tagged Unchanged Skipped Errors and opens tagged rows", "cancel tagging stops at a file boundary", "the report download names the kind", on fake `image-tags` jobs from @@ -2157,10 +2157,79 @@ keeps the drafts; a save refused for one source says the server's message as the and marks that row. After a save the source list is read again; a monitoring change waits for a restart (the running observer keeps watching until then, and the status says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's -Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5). +Review options opens the concept's "Thumbnail maintenance" dialog (D99, below). A list that cannot be read shows "Sources could not be loaded. Your saved data has not changed." and Retry, never an empty list. +Thumbnail generation is an Activity job (S5, `py/api/thumbnail_jobs.py`). `GET +/prompt_manager/images/thumbnails/plan?mode=missing|rebuild&size=128..2048&quality=low|medium|high` +counts the work (`targets`, `fresh`, `remaining` past the 10,000-file cap, the source +labels) and writes nothing; a value out of range is a 400 `invalid-request` that starts +nothing. `POST /prompt_manager/jobs/thumbnails {mode, size, quality}` starts the +`thumbnails` job ("Thumbnail generation"): admission `guarded` (never beside a scan or +another record-changing job, one at a time, a legacy clear refused while it runs), +cancellable at a file boundary (a file not yet started is left alone; written previews +stay), outcome groups Generated / Skipped (already fresh, listed without a log line +each) / Errors. `size` is the longest edge and `quality` the encoder (low: JPEG 70, no +optimize; medium: 85, optimized, what the legacy routes always wrote; high: 95, +optimized). Video previews take the same choice: the OpenCV frame and the placeholder +are saved with those JPEG settings, and the ffmpeg fallback (no OpenCV) maps them to +`-q:v` 8 / 4 / 2 and fits the frame inside the edge keeping its aspect ratio (`-vf +scale=E:E:force_original_aspect_ratio=decrease`, no longer a stretched square). +`rebuild` writes fresh previews again, each through the same temp-file +rename, so nothing is cleared first and a failed file keeps its old preview (a video +whose frame cannot be decoded keeps its preview instead of getting the placeholder); +originals are only read (`tests/test_thumbnail_jobs.py`). Cancel is also checked per +file during the planning walk: a job cancelled there stops the walk at the next file, +publishes no `start` frame and ends "Cancelled before any preview was written". Every +exit sets a final status line: "N previews written · M errors" (plus "· R more on the +next run" past the cap), "No previews to generate · N already up to date" (or "No +images or videos to preview"), "Cancelled · N previews written", or the failure; a job +with nothing to do never reads "Processing file 0 of 0" (`test_every_exit_sets_a_truthful_final_status_line`, +and "a finished thumbnail job with nothing to do says so" in `workspace-activity.spec.mjs`). +Windows: replacing a preview another process holds open (the gallery serving it, an +image viewer) fails with WinError 5 or 32; that file counts as an error reading +"Preview in use by another program; the old one was kept" (never the raw "[WinError +5]"), and its old preview stays. Any other PermissionError reads "Permission denied +writing the preview" (`test_a_locked_preview_is_a_readable_coded_error`). +Thumbnail work (the job and the gallery's generate and clear holds) is `guarded`, but it +writes only preview files and no records, so the image monitor's Needs review +recording runs beside it instead of being deferred for a whole rebuild +(`PREVIEW_ONLY_KINDS` in `py/api/review_recording.py`, `blocker(..., compatible=)`; +`test_a_running_thumbnail_job_does_not_defer_monitor_recording`). Other `guarded` and +all `records` jobs still defer it. + +Deliberate change: the legacy gallery stream `POST /images/generate-thumbnails/progress` +now starts this job (missing previews, its quality preset as the size, medium encoding) +or attaches to the running one, and streams the job's unchanged `status`/`start`/ +`progress`/`file_error`/`complete` frames after a first `job {id, started}` frame that +names the job and whether this stream started it. The stream reads the registry after +its headers are sent, so a job that starts in that gap is attached to, not refused. A +client that goes away no longer stops the work (`test_thumbnail_job_survives_client_disconnect`: +one worker, 9 files in three batches; the stream sees its client go after the first +batch and all 9 previews are written, where a stream that stops on disconnect leaves 1). +The gallery's Cancel (its settings panel and the startup dialog) cancels by that id, +and only a job its own stream started: a Cancel before the `job` frame keeps the stream +open until it arrives, a double Cancel posts once, and a job another tab or System +started keeps running while the gallery stops following it (`tests/js/gallery-thumbnails.test.js`). +A job cancelled from Activity ends the stream with the error frame "Thumbnail +generation was cancelled". The blocking +`POST /images/generate-thumbnails` and the clear stay legacy holds. + +The workspace starts it from "Thumbnail maintenance" (`thumbnail-review.mjs`, the +concept's `thumbnail-review`), opened by Sources' Thumbnail generation → Review options +and Overview's Review thumbnails. Scope is Missing thumbnails only (`missing`, the +default) or Rebuild all (explicit opt-in) (`rebuild`); Quality is Medium · optimized / +High · original / Low · fastest (`medium` / `high` / `low`), preselected from +Appearance's Image loading quality (`gallerySettings.imageQuality`, the same values, +Q3); Preview size limit is the longest edge, 512 by default, a whole number from 128 to +2048. A size outside that range marks the field `aria-invalid`, the plan line says +"Enter a size from 128 to 2048 pixels." and nothing is requested; Start reports the +field's validity and keeps the dialog open. Start closes the dialog and starts the job +through the Activity store, which toasts "Thumbnail generation started."; the job opens +in Activity. A start refused because another job holds the slot toasts the D57 busy +link toast and stays. The legacy gallery's thumbnail tools are unchanged. + Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per visit through `latestOnly()` on the section scope; a late answer after the section left paints nothing. Until it answers, the section shows "Loading the overview…"; a failure @@ -2188,7 +2257,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged `-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no reader can keep it byte-identical (`test_health_check_never_writes`). -Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned): +Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D128 assigned): - **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server @@ -2209,7 +2278,7 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121 The thumbnail row is "Checking thumbnails…" until its own read answers, then "No missing thumbnails", "N missing thumbnails" (the pending count of `/images/thumbnails/status`) or "Thumbnail status unavailable"; Review thumbnails - opens Sources until the thumbnail dialog (S5). "One display configuration" says "Grid + opens the thumbnail dialog (D99). "One display configuration" says "Grid and density settings are shared across views." (no paging). The sources row is "N output sources" with "Monitoring and enrichment share one source list.", prefixed by "M of N configured sources available." or "None configured is available; ComfyUI's @@ -2270,6 +2339,43 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121 legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output folders." or "Files could not be refreshed. Try again.". The concept's "sample path" note is prototype copy and is dropped. +- **D99** (Sources/Overview, S5): the concept's `thumbnail-review` dialog ("Thumbnail + maintenance") with production copy. A `p.small[role=status]` line before the actions + says what the server's read-only plan (`GET /images/thumbnails/plan`) counts for the + chosen scope: "N previews to generate · M already up to date." or, for Rebuild all, + "Every preview is regenerated: N files.", with "· R more on the next run" past the + per-run cap; "Counting previews…" while it is read and "The previews could not be + counted." when the read fails (Start still starts). Each scope is read once per + dialog, since size and quality change no count. The primary is "Start" (the + concept's "Simulate thumbnail job"). Overview's Review thumbnails opens the same + dialog (the concept's row starts a sample "Thumbnail check" job directly). Maintenance's + Thumbnail cache row takes the dialog with S7. `thumbnail dialog matches the concept + from sources|overview` (system-concept.spec, both themes and sizes) maps the line and + "Start"; `thumbnail dialog comparison detects mutations` proves they are the only + difference and catches a renamed option, swapped options, swapped actions and a + changed style. +- **D111** (Activity, S5): each job kind's pill verb, toasts, line, tiles, outcomes, + openable rows and labels come from one descriptor table (`activity-kinds.mjs`; + `activity-store.mjs` re-exports it). The scan's (D56) and image tagging's (D89) are + unchanged. A thumbnail job reads "Generating thumbnails · N% · a / b" in the pill, + counts Generated (green), Skipped and Errors (red), lists Generated, Errors, Skipped + (Generated and Skipped rows open the viewer), says "Thumbnails finished · N generated · + M errors" ("cancelled · N generated", "failed · …") and keeps "Job log" and "Cancel + job". Every other kind (the health check, a kind this page does not know) gets the + generic descriptor: title-based toasts and no tiles or Results, so it never shows the + scan's zero tiles (S1 review M6, S2 review M3). A running job without a known total + has an indeterminate `<progress>` (no `value`) and a pill without a percentage, whose + mini bar (`aria-hidden`; the pill's label carries the state) is marked `indeterminate` + and sweeps instead of showing an empty fill (a dimmed full bar under reduced motion). Only a + running job the server can cancel shows Cancel. Tests: + `tests/js/workspace-activity-kinds.test.js`, and in `workspace-activity.spec.mjs` "a + thumbnail job shows Generated Skipped Errors, its pill verb and its toasts" and "a held + legacy job and the health check show no tiles, no Results and no made-up percentage". +- **D112** (Activity, S5): a legacy-page operation held on the registry (not + cancellable, no outcome groups) reads as its server title ("Database restore (admin + page)", "Thumbnail clear (gallery page)"), with no tiles, no Results, no Cancel, an + indeterminate bar while it runs and "<title> finished." when it ends, whatever its + kind (the gallery's clear is a `thumbnails` hold). - **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's sample 4). @@ -2358,10 +2464,13 @@ and a source row, renames a label and the Add source button and swaps the `--lin token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`, `{concept,production}-{statistics,health}-{dark,light}.png`, `{concept,production}-appearance-{dark,light}-{1600,390}.png`, -`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there +`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`, +`concept-thumbnail-review-{dark,light}-{1600,390}.png`, +`production-thumbnail-review-{sources,overview}-{dark,light}-{1600,390}.png`) are test artifacts; there are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts, the health check and metric 4, a failed check, the thumbnail row against the status -endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the +endpoint (painted after the rest when its read is held, unavailable when it fails), its Review thumbnails +opening the thumbnail dialog and starting the job, the review row (Q6), the section links, the Statistics dialog against the route, a load failure with Retry and late answers after leaving, and Appearance: new visits versus the current view and a saved view, the embedded gallery keeping Grid, metadata off and on request, the cache (counted up to a request barrier), hidden paths, @@ -2371,7 +2480,9 @@ the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers notice's Later and Review scan, a scoped scan from one source into Activity, Add source with its server check and Save, a refused folder, Remove down to auto-detect, an absent configured source, the three watch states, the monitoring and worker rows, Refresh -files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are +files, the thumbnail dialog (its plan line per scope with the preselected quality, an +invalid size that requests nothing, Start into Activity, Rebuild all, a failed plan and +a refused start), a save refused by a running scan and a failed read. Its hooks are `tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected failures, an absent source, a deleted linked file, unthumbnailed files, a fake image monitor) on the @@ -2386,7 +2497,7 @@ node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-co tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \ tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \ tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \ - tests/js/workspace-system-sources.test.js + tests/js/workspace-system-sources.test.js tests/js/workspace-thumbnail-review.test.js python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \ workspace-system-sources.spec.mjs \ @@ -2655,7 +2766,7 @@ pass runs after its own link, best effort: a failure is logged and the link stays. Recording follows the registry's `records` admission (#289): the scan's recording runs under its own `records` job, and the monitor's (one file, not a job) is refused while a `records` or `guarded` -job or hold runs. It then records nothing, logs that the file waits for the +job or hold runs, except thumbnail work, which writes no records (S5 M7). It then records nothing, logs that the file waits for the next scan, and answers `deferred`; the next "new" scan reads the ambiguous file again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A rescan does not copy or hash an ambiguous file whose current size and diff --git a/py/api/__init__.py b/py/api/__init__.py index 8b12c43..f929b2d 100644 --- a/py/api/__init__.py +++ b/py/api/__init__.py @@ -8,6 +8,7 @@ Split into domain-specific mixins for maintainability: - AutotagRoutesMixin: auto-tagging model management and tagging - JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py) - ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs +- ThumbnailJobsMixin: thumbnail plan, thumbnail job and the gallery stream on it - ReviewRoutesMixin: the Needs review queue, item detail and decisions - ReviewRecordingMixin: Needs review recording for the scan and the monitor - ReviewEventsMixin: the Needs review change stream an open page re-reads on @@ -43,6 +44,7 @@ from .admin import AdminRoutesMixin from .settings_routes import SettingsRoutesMixin from .job_routes import JobRoutesMixin from .image_tag_jobs import ImageTagJobsMixin +from .thumbnail_jobs import ThumbnailJobsMixin from .system import SystemRoutesMixin from .jobs import JobRegistry from .logging_routes import LoggingRoutesMixin @@ -281,6 +283,7 @@ class PromptManagerAPI( SettingsRoutesMixin, JobRoutesMixin, ImageTagJobsMixin, + ThumbnailJobsMixin, SystemRoutesMixin, LoggingRoutesMixin, AutotagRoutesMixin, diff --git a/py/api/images.py b/py/api/images.py index c37a6bb..4ec86a1 100644 --- a/py/api/images.py +++ b/py/api/images.py @@ -20,7 +20,7 @@ from typing import Callable, NamedTuple from aiohttp import web from PIL import Image, UnidentifiedImageError -from .jobs import THUMBNAILS_JOB_KIND +from .jobs import MAX_OUTCOME_ROWS, THUMBNAILS_JOB_KIND from .prompts import bad_request, parse_page_params, publish_image_paths from .prompts import thumbnail_url_for from .prompts import safe_error_message as _safe_error @@ -235,6 +235,8 @@ MEDIA_EXTENSIONS = GALLERY_IMAGE_EXTENSIONS | VIDEO_EXTENSIONS # Caps on request-triggered filesystem work. MAX_SCAN_DEPTH = 12 MAX_THUMBNAIL_FILES = 10_000 +# Already-fresh files a thumbnail job lists under Skipped (its counter stays exact). +MAX_FRESH_LISTED = MAX_OUTCOME_ROWS def _worker_threads(): @@ -271,15 +273,52 @@ def _atomic_save(img, dst, fmt=None, **save_kwargs): tmp.unlink() -def _write_image_thumbnail(src, dst, thumbnail_size): - """Resize *src* into *dst* (blocking PIL work), atomically.""" +# Encoder settings per thumbnail quality (the System thumbnail dialog's +# "Medium · optimized", "High · original", "Low · fastest"). The legacy +# routes always encode as "medium", which is what they always wrote. +THUMBNAIL_ENCODINGS = { + "low": {"quality": 70, "optimize": False}, + "medium": {"quality": 85, "optimize": True}, + "high": {"quality": 95, "optimize": True}, +} +DEFAULT_THUMBNAIL_ENCODING = THUMBNAIL_ENCODINGS["medium"] +# ffmpeg's JPEG scale (-q:v, 2 best to 31 worst) for each encoding's quality. +_FFMPEG_QSCALE = {95: 2, 85: 4, 70: 8} + +# A preview that could not be replaced (#249 S5 M6). Windows refuses to rename +# over a file another process has open (WinError 5 or 32), such as a preview +# the gallery is serving; the old preview stays and the file counts as an error. +PREVIEW_IN_USE = "Preview in use by another program; the old one was kept" +PREVIEW_NOT_WRITABLE = "Permission denied writing the preview" +_WINDOWS_LOCKED = (5, 32) # ERROR_ACCESS_DENIED, ERROR_SHARING_VIOLATION + + +def _preview_write_error(exc): + """The readable error for a PermissionError while writing a preview.""" + if getattr(exc, "winerror", None) in _WINDOWS_LOCKED: + return PREVIEW_IN_USE + return PREVIEW_NOT_WRITABLE + + +def _ffmpeg_qscale(encoding): + """ffmpeg's ``-q:v`` for a THUMBNAIL_ENCODINGS entry.""" + return str(_FFMPEG_QSCALE.get(encoding["quality"], 4)) + + +def _write_image_thumbnail(src, dst, thumbnail_size, encoding=None): + """Resize *src* into *dst* (blocking PIL work), atomically. + + ``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); a PNG + preview keeps only its ``optimize`` flag. + """ + encoding = encoding or DEFAULT_THUMBNAIL_ENCODING with Image.open(src) as img: if img.mode in ("RGBA", "LA", "P"): img = img.convert("RGB") img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS) - save_kwargs = {"quality": 85, "optimize": True} + save_kwargs = dict(encoding) if dst.suffix.lower() == ".png": - save_kwargs = {"optimize": True} + save_kwargs = {"optimize": encoding["optimize"]} _atomic_save(img, dst, **save_kwargs) @@ -945,6 +984,10 @@ class ImageRoutesMixin: async def thumbnail_status_route(request): return await self.get_thumbnail_status(request) + @routes.get("/prompt_manager/images/thumbnails/plan") + async def thumbnail_plan_route(request): + return await self.get_thumbnail_plan(request) + @routes.post("/prompt_manager/images/generate-thumbnails") async def generate_thumbnails_route(request): return await self.generate_thumbnails(request) @@ -1817,15 +1860,35 @@ class ImageRoutesMixin: self.logger.info(payload["message"]) return payload - def _plan_gallery_thumbnails(self, output_dirs: list[Path]) -> dict: - """Plan every gallery root with one shared per-run generation cap.""" - combined = {"targets": [], "skipped": 0, "remaining": 0} + def _plan_gallery_thumbnails( + self, output_dirs: list[Path], rebuild: bool = False, stop=None + ) -> dict: + """Plan every gallery root with one shared per-run generation cap. + + See ``_plan_thumbnails``; ``fresh_files`` keeps at most + MAX_FRESH_LISTED files across the roots. ``stop()`` true ends the walk + at the next file (a cancelled job); the partial plan is returned. + """ + combined = { + "targets": [], + "skipped": 0, + "remaining": 0, + "fresh": 0, + "fresh_files": [], + } for root in output_dirs: + if stop is not None and stop(): + break root = Path(root) - plan = self._plan_thumbnails(root, root / "thumbnails") + plan = self._plan_thumbnails( + root, root / "thumbnails", rebuild=rebuild, stop=stop + ) available = max(0, MAX_THUMBNAIL_FILES - len(combined["targets"])) combined["targets"].extend(plan["targets"][:available]) combined["skipped"] += plan["skipped"] + combined["fresh"] += plan["fresh"] + room = max(0, MAX_FRESH_LISTED - len(combined["fresh_files"])) + combined["fresh_files"].extend(plan["fresh_files"][:room]) combined["remaining"] += plan["remaining"] + max( 0, len(plan["targets"]) - available ) @@ -1858,22 +1921,30 @@ class ImageRoutesMixin: {"success": False, "error": _safe_error(exc)}, status=500 ) - def _plan_thumbnails(self, output_path, thumbnails_dir): + def _plan_thumbnails(self, output_path, thumbnails_dir, rebuild=False, stop=None): """Decide which media files need a thumbnail (blocking walk). - Returns ``{"targets": [(src, dst, is_video)], "skipped": n, "remaining": n}``. - Files whose thumbnail is already newer than the source are counted as - skipped and never queued, so a large library that is mostly done only - pays for the files that changed. The walk is bounded like the gallery - listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES, no symlinks) and at most - MAX_THUMBNAIL_FILES pending files are queued per run; ``remaining`` - counts the pending files beyond that cap. + Returns ``{"targets": [(src, dst, is_video)], "skipped": n, + "remaining": n, "fresh": n, "fresh_files": [src]}``. Files whose + thumbnail is already newer than the source are ``fresh``. Without + ``rebuild`` they are counted as skipped and never queued, so a large + library that is mostly done only pays for the files that changed; + ``fresh_files`` lists the first MAX_FRESH_LISTED of them. With + ``rebuild`` every file is a target (nothing is skipped). The walk is + bounded like the gallery listing (MAX_SCAN_DEPTH, MAX_GALLERY_FILES, + no symlinks) and at most MAX_THUMBNAIL_FILES targets are queued per + run; ``remaining`` counts the targets beyond that cap. ``stop()`` true + ends the walk before the next file. """ output_path = Path(output_path) targets = [] skipped = 0 remaining = 0 + fresh = 0 + fresh_files = [] for src in _iter_media_files(output_path, MAX_GALLERY_FILES): + if stop is not None and stop(): + break suffix = src.suffix.lower() is_video = suffix in VIDEO_EXTENSIONS rel_no_ext = src.relative_to(output_path).with_suffix("") @@ -1883,8 +1954,12 @@ class ImageRoutesMixin: self.logger.warning(f"Skipping thumbnail outside safe dir: {src.name}") continue if _thumbnail_is_fresh(src, dst): - skipped += 1 - continue + fresh += 1 + if not rebuild: + skipped += 1 + if len(fresh_files) < MAX_FRESH_LISTED: + fresh_files.append(src) + continue if len(targets) >= MAX_THUMBNAIL_FILES: remaining += 1 continue @@ -1894,190 +1969,81 @@ class ImageRoutesMixin: f"Thumbnail run capped at {MAX_THUMBNAIL_FILES} new files; " f"{remaining} more will be generated on the next run" ) - return {"targets": targets, "skipped": skipped, "remaining": remaining} + return { + "targets": targets, + "skipped": skipped, + "remaining": remaining, + "fresh": fresh, + "fresh_files": fresh_files, + } def _thumbnail_targets(self, output_path, thumbnails_dir): """Pending (src, dst, is_video) triples; see _plan_thumbnails.""" return self._plan_thumbnails(output_path, thumbnails_dir)["targets"] - def _generate_batch(self, targets, thumbnail_size): + def _generate_batch(self, targets, thumbnail_size, **options): """Create thumbnails for ``targets`` on the configured worker threads. - Blocking; returns one result dict per target, in order. + Blocking; returns one result dict per target, in order. ``options`` + go to ``_generate_one`` (``encoding``, ``rebuild``, ``stop``). """ def one(target): src, dst, is_video = target - return self._generate_one(src, dst, thumbnail_size, is_video) + return self._generate_one(src, dst, thumbnail_size, is_video, **options) return map_parallel(one, targets, _worker_threads()) - def _generate_one(self, src, dst, thumbnail_size, is_video): - """Create one thumbnail (blocking). Never raises; returns a result dict.""" + def _generate_one( + self, + src, + dst, + thumbnail_size, + is_video, + encoding=None, + rebuild=False, + stop=None, + ): + """Create one thumbnail (blocking). Never raises; returns a result dict. + + ``stop()`` true before the file starts gives ``action: "cancelled"`` + and touches nothing (a job's file boundary). ``rebuild`` writes a + fresh preview again and never replaces an existing video preview with + the placeholder. Every write is atomic, so a failure keeps the old + preview. + """ result = { "file": src.name, "dir": src.parent.name, "type": "video" if is_video else "image", } try: + if stop is not None and stop(): + return {**result, "action": "cancelled"} if not src.is_file(): return {**result, "action": "error", "error": "File no longer exists"} - if _thumbnail_is_fresh(src, dst): + if not rebuild and _thumbnail_is_fresh(src, dst): return {**result, "action": "skipped"} dst.parent.mkdir(parents=True, exist_ok=True) if is_video: - if not self._generate_video_thumbnail(src, dst, thumbnail_size): + made = self._generate_video_thumbnail( + src, dst, thumbnail_size, encoding, keep_existing=rebuild + ) + if not made: return {**result, "action": "error", "error": "Video decode failed"} else: - _write_image_thumbnail(src, dst, thumbnail_size) + _write_image_thumbnail(src, dst, thumbnail_size, encoding) return {**result, "action": "generated"} except UnidentifiedImageError: return {**result, "action": "error", "error": "Not a valid image file"} + except PermissionError as e: + message = _preview_write_error(e) + self.logger.warning(f"Thumbnail failed for {src.name}: {message}") + return {**result, "action": "error", "error": message} except Exception as e: self.logger.warning(f"Thumbnail failed for {src.name}", exc_info=True) return {**result, "action": "error", "error": _safe_error(e)} - async def generate_thumbnails_with_progress(self, request): - """Generate thumbnails with Server-Sent Events progress updates (POST). - - ``quality`` is read from the query string so a POST-capable SSE - client can keep using the same URL shape. Every blocking step (scan, - PIL/ffmpeg) runs in the executor; the event loop only writes SSE - frames between files. - """ - thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium")) - response = web.StreamResponse( - status=200, - headers={ - "Content-Type": "text/event-stream", - "Cache-Control": "no-cache", - "Connection": "keep-alive", - }, - ) - busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded") - if busy is not None: - # The gallery reads this stream: refuse with its error event. - await response.prepare(request) - message = f"{busy.title} already running" - await self._emit_sse( - response, "error", {"error": message, "message": message} - ) - return response - async with self._jobs.hold( - THUMBNAILS_JOB_KIND, THUMBNAIL_GENERATION_TITLE, admission="guarded" - ) as job: - await response.prepare(request) - if not await self._thumbnail_progress_stream(response, thumbnail_size): - job.finish("failed") # an error event ended the stream - return response - - async def _thumbnail_progress_stream(self, response, thumbnail_size) -> bool: - """Stream one generation run; False when it ended with an error event.""" - try: - output_dirs = self._get_all_output_dirs() - if not output_dirs: - await self._emit_sse( - response, "error", {"error": "ComfyUI output directory not found"} - ) - return False - - await self._emit_sse( - response, - "status", - {"phase": "scanning", "message": "Scanning for images and videos..."}, - ) - plan = await self._run_in_executor( - self._plan_gallery_thumbnails, output_dirs - ) - await self._stream_thumbnails( - response, - plan["targets"], - thumbnail_size, - skipped=plan["skipped"], - remaining=plan["remaining"], - ) - self.invalidate_gallery_cache() - return True - except Exception: - self.logger.exception("Thumbnail generation failed") - await self._emit_sse( - response, - "error", - { - "error": "An internal error occurred", - "message": "Thumbnail generation failed. Check server logs.", - }, - ) - return False - - async def _stream_thumbnails( - self, response, targets, thumbnail_size, skipped=0, remaining=0 - ): - """Process pending *targets* in worker-thread batches, emitting SSE per file. - - ``skipped`` files already had fresh thumbnails and ``remaining`` were - left for the next run; both only feed the start/complete payloads. - """ - total = len(targets) - video_count = sum(1 for _, _, is_video in targets if is_video) - message = ( - f"Found {total - video_count} images and {video_count} videos to process" - ) - if skipped: - message += f" ({skipped} already have thumbnails)" - connected = await self._emit_sse( - response, - "start", - { - "total_images": total, - "skipped": skipped, - "remaining": remaining, - "phase": "processing", - "image_count": total - video_count, - "video_count": video_count, - "message": message, - }, - ) - if not connected: - return - stats = _new_thumbnail_stats() - stats["skipped"] = skipped - start = _time.monotonic() - batch_size = _thumbnail_batch_size(_worker_threads()) - index = 0 - for batch_start in range(0, total, batch_size): - batch = targets[batch_start : batch_start + batch_size] - results = await self._run_in_executor( - self._generate_batch, batch, thumbnail_size - ) - for result in results: - index += 1 - _record_thumbnail_result(stats, result) - if result["action"] == "error": - await self._emit_sse( - response, - "file_error", - {"file": result["file"], "error": result["error"]}, - ) - connected = await self._emit_sse( - response, - "progress", - _thumbnail_progress_payload(index, total, stats, result, start), - ) - if not connected: - self.logger.info("Thumbnail client disconnected; stopping early") - return - await asyncio.sleep(0) # let other requests run between batches - payload = _thumbnail_complete_payload( - total + skipped + remaining, - stats, - start, - remaining=remaining, - processed=total, - ) - await self._emit_sse(response, "complete", payload) - self.logger.info(payload["message"]) - async def _emit_sse(self, response, event, data) -> bool: """Write one SSE frame. Returns False once the client has gone away.""" try: @@ -2088,8 +2054,25 @@ class ImageRoutesMixin: self.logger.warning(f"Failed to send SSE message: {e}") return False - def _generate_video_thumbnail(self, video_path, thumbnail_path, thumbnail_size): - """Generate thumbnail from video file. Returns True if successful.""" + def _generate_video_thumbnail( + self, + video_path, + thumbnail_path, + thumbnail_size, + encoding=None, + keep_existing=False, + ): + """Generate thumbnail from video file. Returns True if successful. + + ``encoding`` is one of THUMBNAIL_ENCODINGS (medium when None); it + encodes the OpenCV frame, ffmpeg's frame (as ``-q:v``) and the + placeholder alike. Both decoders fit the frame inside + ``thumbnail_size`` keeping its aspect ratio. With ``keep_existing`` an + existing preview is kept, and the run fails, when no frame could be + decoded, rather than replaced by the placeholder. A PermissionError + while replacing the preview propagates (``_generate_one`` names it). + """ + encoding = encoding or DEFAULT_THUMBNAIL_ENCODING try: # Try using OpenCV first (most reliable) try: @@ -2113,7 +2096,7 @@ class ImageRoutesMixin: img = Image.fromarray(frame_rgb) img.thumbnail(thumbnail_size, Image.Resampling.LANCZOS) - _atomic_save(img, thumbnail_path, "JPEG", quality=85, optimize=True) + _atomic_save(img, thumbnail_path, "JPEG", **encoding) self.logger.debug( f"Generated video thumbnail using OpenCV: {thumbnail_path}" ) @@ -2135,8 +2118,11 @@ class ImageRoutesMixin: "00:00:01", "-vframes", "1", - "-s", - f"{thumbnail_size[0]}x{thumbnail_size[1]}", + "-vf", + f"scale={thumbnail_size[0]}:{thumbnail_size[1]}" + ":force_original_aspect_ratio=decrease", + "-q:v", + _ffmpeg_qscale(encoding), "-y", str(tmp_path), ] @@ -2161,6 +2147,12 @@ class ImageRoutesMixin: except (NameError, OSError): pass + if keep_existing and thumbnail_path.exists(): + self.logger.warning( + f"No frame decoded for {video_path.name}; keeping its preview" + ) + return False + # Last resort: create a placeholder thumbnail try: from PIL import ImageDraw, ImageFont @@ -2195,18 +2187,22 @@ class ImageRoutesMixin: except (OSError, AttributeError): pass - _atomic_save(img, thumbnail_path, "JPEG", quality=85) + _atomic_save(img, thumbnail_path, "JPEG", **encoding) self.logger.debug( f"Generated placeholder video thumbnail: {thumbnail_path}" ) return True + except PermissionError: + raise except Exception as e: self.logger.warning( f"Failed to create placeholder thumbnail for {video_path}: {e}" ) return False + except PermissionError: + raise except Exception as e: self.logger.error( f"Video thumbnail generation failed for {video_path}: {e}" diff --git a/py/api/job_routes.py b/py/api/job_routes.py index 83caa85..47056a1 100644 --- a/py/api/job_routes.py +++ b/py/api/job_routes.py @@ -1,7 +1,8 @@ """Session job routes (Activity): list, inspect, follow and cancel jobs. The registry itself is ``py/api/jobs.py``. The scan job's runner is in -``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py``. +``py/api/admin.py``, the image tagging job's in ``py/api/image_tag_jobs.py`` +and the thumbnail job's in ``py/api/thumbnail_jobs.py``. Every response leaves out absolute server paths: job rows already carry root-relative paths and public root labels. """ @@ -129,6 +130,10 @@ class JobRoutesMixin: async def start_image_tag_job_route(request): return await self.start_image_tag_job(request) + @routes.post("/prompt_manager/jobs/thumbnails") + async def start_thumbnail_job_route(request): + return await self.start_thumbnail_job(request) + @routes.get("/prompt_manager/jobs/{job_id}") async def job_detail_route(request): return await self.job_detail(request) diff --git a/py/api/jobs.py b/py/api/jobs.py index b078503..aec92eb 100644 --- a/py/api/jobs.py +++ b/py/api/jobs.py @@ -72,6 +72,9 @@ SCAN_GROUPS = { IMAGE_TAG_JOB_KIND = "image-tags" IMAGE_TAG_GROUP_NAMES = ("tagged", "unchanged", "skipped", "errors") +# Thumbnail generation (#249 System S5): previews written, already fresh, failed. +THUMBNAIL_GROUP_NAMES = ("generated", "skipped", "errors") + ADMISSIONS = ("read", "guarded", "records") # What each admission class is refused beside (besides its own kind). _REFUSED_BESIDE = { @@ -247,6 +250,32 @@ class Job: ) self.log_line(log_line) + def record_counted(self, group, count, rows=()): + """Count ``count`` outcomes in ``group`` at once, listing ``rows``. + + For outcomes that need no log line each (a thumbnail job's already + fresh previews): the counter is exact, ``rows`` (``record``'s row + shape) are listed up to MAX_OUTCOME_ROWS, and nothing is logged. + """ + if group not in self.counters: + raise ValueError(f"Unknown outcome group {group!r} for {self.kind}") + self.counters[group] += max(0, int(count)) + listed = self.outcomes[group] + for row in rows: + if len(listed) >= MAX_OUTCOME_ROWS: + break + index = row.get("root_index") + if isinstance(index, bool) or not isinstance(index, int) or index < 0: + index = None + listed.append( + { + "path": _relative_path(row.get("path")), + "root": str(row.get("root") or ""), + "root_index": index, + "detail": str(row.get("detail") or ""), + } + ) + def finish(self, state="finished"): """End the job in ``state`` and release every subscriber. @@ -263,18 +292,24 @@ class Job: for queue in list(self._subscribers): queue.put_nowait(None) - async def events(self): + async def events(self, lead=None): """Yield the latest event, then each later one, until the job finishes. + ``lead``, when given, is yielded first, after the subscription is in + place, so nothing published while the consumer sends it is missed. Consume it under ``contextlib.aclosing`` so a consumer that stops early (a closed socket) drops its queue at once instead of at garbage collection. """ queue = asyncio.Queue() self._subscribers.add(queue) + # As of the subscription: a later event or the end reaches the queue. + snapshot, ended = self.last_event, self.finished try: - if self.last_event is not None: - yield self.last_event - if self.finished: + if lead is not None: + yield lead + if snapshot is not None: + yield snapshot + if ended: return while True: payload = await queue.get() @@ -425,11 +460,14 @@ class JobRegistry: self._jobs[job.id] = job return job - def blocker(self, kind, *, admission=None, record_changing=None): + def blocker(self, kind, *, admission=None, record_changing=None, compatible=()): """The running job a new ``kind`` job would be refused for, or None. The same kind refuses first; then the oldest running job whose class - the new job's admission class may not run beside. + the new job's admission class may not run beside. Running jobs of a + kind in ``compatible`` never refuse it: the caller knows that work + cannot conflict with its own (the monitor's Needs review recording + beside thumbnail work, which writes previews and no records). """ same_kind = self.active(kind) if same_kind is not None: @@ -439,7 +477,7 @@ class JobRegistry: running = [job for job in tuple(self._jobs.values()) if job.running] running.sort(key=lambda job: job._started_seq) for job in running: - if job.admission in refused: + if job.admission in refused and job.kind not in compatible: return job return None diff --git a/py/api/review_recording.py b/py/api/review_recording.py index 26c2de2..7cb60c6 100644 --- a/py/api/review_recording.py +++ b/py/api/review_recording.py @@ -28,16 +28,23 @@ Recording changes records, so it follows the registry's ``records`` admission (#289). The scan records under its own ``records`` job. The monitor's pass lasts one file and is not a job: it is refused while a ``records`` or ``guarded`` job runs (``deferred``), and the next scan reads -the file again. +the file again. Thumbnail work (the job and the gallery's generate and clear +holds, kind ``thumbnails``) is ``guarded`` but only writes and removes preview +files under ``thumbnails/`` and never touches the database, so it does not +defer the recording (#249 S5 M7): a long rebuild no longer holds Needs review +back for its whole run. """ import os from pathlib import Path from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots +from .jobs import THUMBNAILS_JOB_KIND # The kind the monitor's recording checks admission under (never registered). REVIEW_RECORDING_KIND = "review_recording" +# Guarded kinds that change no records, so the recording may run beside them. +PREVIEW_ONLY_KINDS = (THUMBNAILS_JOB_KIND,) # record_ambiguous_file outcomes that leave the file in the pending queue. QUEUED_OUTCOMES = ("recorded", "limited", "duplicate") NEW_OUTCOMES = ("recorded", "limited") @@ -185,11 +192,14 @@ class ReviewRecordingMixin: links it); any other file is recorded when it is ambiguous with a choice, and is ``untraced`` (its pending items without a choice removed) when it is ambiguous without one. While a - job runs that a ``records`` job may not run beside, nothing is written - and the answer is ``deferred``. Errors propagate: the monitor logs + job runs that a ``records`` job may not run beside, other than + thumbnail work (``PREVIEW_ONLY_KINDS``), nothing is written and the + answer is ``deferred``. Errors propagate: the monitor logs them and keeps its link. """ - busy = self._jobs.blocker(REVIEW_RECORDING_KIND, admission="records") + busy = self._jobs.blocker( + REVIEW_RECORDING_KIND, admission="records", compatible=PREVIEW_ONLY_KINDS + ) if busy is not None: self.logger.info( f"Needs review was not updated for " diff --git a/py/api/thumbnail_jobs.py b/py/api/thumbnail_jobs.py new file mode 100644 index 0000000..9c0d6b6 --- /dev/null +++ b/py/api/thumbnail_jobs.py @@ -0,0 +1,494 @@ +"""Thumbnail generation as an Activity job (#249 System S5, Task 12). + +- ``GET /prompt_manager/images/thumbnails/plan?mode=&size=&quality=`` counts + what a run would do over every gallery root and writes nothing: + ``{success, targets, fresh, remaining, roots, mode, size, quality}``. +- ``POST /prompt_manager/jobs/thumbnails {mode, size, quality}`` starts the + ``thumbnails`` job (202 ``{job}``). Its admission class is ``guarded``: it + never runs beside a record-changing job, and one thumbnail job runs at a + time. It stops at a file boundary when cancelled, keeping the previews it + wrote. Its outcome groups are ``generated``, ``skipped`` (already fresh) + and ``errors``. + +``mode`` is ``missing`` (previews that are absent or older than their file) +or ``rebuild`` (every preview again, fresh ones too). ``size`` is the +preview's longest edge, 128 to 2048 pixels. ``quality`` picks the encoder +settings (``THUMBNAIL_ENCODINGS``). Every preview is written next to the old +one and renamed over it, so nothing is cleared first and a failure keeps the +old preview; originals are only read. + +The legacy gallery stream ``POST /images/generate-thumbnails/progress`` +starts this job (its quality preset still sizes the previews) or attaches to +the running one. Its first frame, ``job {id, started}``, names that job and +whether this stream started it (the gallery cancels by that id, and only a +job it started); then come the job's legacy ``status``/``start``/ +``progress``/``file_error``/``complete`` frames. A client that goes away no +longer stops the work: the job outlives the request, as a scan does. +""" + +import asyncio +import functools +import json +import time +from dataclasses import dataclass + +from aiohttp import web + +from .admin import _plural, _scan_location, _scan_roots, _scan_status_line +from .images import ( + THUMBNAIL_ENCODINGS, + _BodyTooLarge, + _new_thumbnail_stats, + _read_json_body, + _record_thumbnail_result, + _thumbnail_batch_size, + _thumbnail_complete_payload, + _thumbnail_progress_payload, + _worker_threads, +) +from .job_routes import busy_response +from .jobs import THUMBNAIL_GROUP_NAMES, THUMBNAILS_JOB_KIND, JobBusy + +THUMBNAIL_JOB_TITLE = "Thumbnail generation" +THUMBNAIL_MODES = ("missing", "rebuild") +THUMBNAIL_QUALITIES = tuple(THUMBNAIL_ENCODINGS) +MIN_THUMBNAIL_EDGE = 128 +MAX_THUMBNAIL_EDGE = 2048 +# The concept's "Preview size limit" default. +DEFAULT_THUMBNAIL_EDGE = 512 +DEFAULT_THUMBNAIL_OPTIONS = { + "mode": "missing", + "size": DEFAULT_THUMBNAIL_EDGE, + "quality": "medium", +} +_OPTIONS_ERROR = ( + 'mode must be "missing" or "rebuild", quality "low", "medium" or "high", ' + f"and size a whole number of pixels from {MIN_THUMBNAIL_EDGE} " + f"to {MAX_THUMBNAIL_EDGE}" +) +CANCELLED_ERROR = "Thumbnail generation was cancelled" +CANCELLED_BEFORE_WRITING = "Cancelled before any preview was written" +NOTHING_TO_PREVIEW = "No images or videos to preview" +_MODE_LOG = {"missing": "missing previews", "rebuild": "every preview"} +# Outcome rows: what happened to each file's preview. +_DETAILS = { + "generated": "Preview written", + "skipped": "Preview already up to date", +} +_LOG_PREFIXES = {"generated": "+", "errors": "!"} + + +@dataclass(frozen=True) +class ThumbnailOptions: + """One run's choices: the mode, the preview's longest edge and quality.""" + + mode: str + edge: int + quality: str + + @property + def size(self): + return (self.edge, self.edge) + + @property + def encoding(self): + return THUMBNAIL_ENCODINGS[self.quality] + + @property + def rebuild(self): + return self.mode == "rebuild" + + def scope(self): + return {"mode": self.mode, "size": self.edge, "quality": self.quality} + + +def _edge(value, *, from_query): + """The preview edge as an int, or None when it is not one in range.""" + if from_query and isinstance(value, str): + if not value.isascii() or not value.isdigit() or len(value) > 5: + return None + value = int(value) + if isinstance(value, bool) or not isinstance(value, int): + return None + return value if MIN_THUMBNAIL_EDGE <= value <= MAX_THUMBNAIL_EDGE else None + + +def thumbnail_options(raw, *, from_query=False): + """``(ThumbnailOptions, None)`` from a body or query, or ``(None, message)``. + + Absent fields take ``DEFAULT_THUMBNAIL_OPTIONS``. Query values are + strings, so a query ``size`` may be decimal digits; a JSON ``size`` must + be an int. + """ + if not isinstance(raw, dict) and not hasattr(raw, "get"): + return None, _OPTIONS_ERROR + mode = raw.get("mode", DEFAULT_THUMBNAIL_OPTIONS["mode"]) + quality = raw.get("quality", DEFAULT_THUMBNAIL_OPTIONS["quality"]) + edge = _edge(raw.get("size", DEFAULT_THUMBNAIL_EDGE), from_query=from_query) + if mode not in THUMBNAIL_MODES or quality not in THUMBNAIL_QUALITIES: + return None, _OPTIONS_ERROR + if edge is None: + return None, _OPTIONS_ERROR + return ThumbnailOptions(mode, edge, quality), None + + +def _invalid(message): + return web.json_response( + {"success": False, "code": "invalid-request", "error": message}, status=400 + ) + + +def _frame(event, data): + """A legacy stream frame as the job publishes it.""" + return {"event": event, "data": data} + + +def _finished_line(plan, stats): + """A finished job's status line: what it wrote, or that it had nothing to do.""" + if not plan["targets"]: + if plan["skipped"]: + return f"No previews to generate · {plan['skipped']:,} already up to date" + return NOTHING_TO_PREVIEW + parts = [f"{_plural(stats['generated'], 'preview')} written"] + if stats["errors"]: + parts.append(_plural(len(stats["errors"]), "error")) + if plan["remaining"]: + parts.append(f"{plan['remaining']:,} more on the next run") + return " · ".join(parts) + + +class ThumbnailJobsMixin: + """The thumbnail plan, the thumbnail job and the legacy stream on it.""" + + async def get_thumbnail_plan(self, request): + """What a thumbnail run would do (read-only; capped like a run).""" + options, error = thumbnail_options(request.query, from_query=True) + if error: + return _invalid(error) + try: + output_dirs = await self._run_in_executor(self._get_all_output_dirs) + plan = await self._run_in_executor( + self._plan_gallery_thumbnails, output_dirs, rebuild=options.rebuild + ) + except Exception: + self.logger.exception("Thumbnail plan failed") + return web.json_response( + { + "success": False, + "code": None, + "error": "The thumbnail plan could not be read", + }, + status=500, + ) + return web.json_response( + { + "success": True, + "targets": len(plan["targets"]), + "fresh": plan["fresh"], + "remaining": plan["remaining"], + "roots": [self._public_path(root) for root in output_dirs], + **options.scope(), + }, + headers={"Cache-Control": "no-store"}, + ) + + async def start_thumbnail_job(self, request): + """Start the thumbnail job (202), or 409 naming the job in the way.""" + body = {} + if request.can_read_body: + try: + body = await _read_json_body(request) + except _BodyTooLarge: + return web.json_response( + {"success": False, "code": "too-large", "error": "Body too large"}, + status=413, + ) + except (ValueError, UnicodeDecodeError): + return _invalid(_OPTIONS_ERROR) + options, error = thumbnail_options(body) + if error: + return _invalid(error) + try: + job = self._start_thumbnail_job(options) + except JobBusy as busy: + return busy_response(busy) + return web.json_response({"success": True, "job": job.summary()}, status=202) + + def _start_thumbnail_job(self, options): + """Register the thumbnail job synchronously; raises JobBusy.""" + return self._jobs.start( + THUMBNAILS_JOB_KIND, + THUMBNAIL_JOB_TITLE, + admission="guarded", + cancellable=True, + scope=options.scope(), + runner=functools.partial(self._run_thumbnail_job, options=options), + groups=THUMBNAIL_GROUP_NAMES, + ) + + # ── the job ─────────────────────────────────────────────────────── + + async def _run_thumbnail_job(self, job, options): + try: + state = await self._thumbnail_job_body(job, options) + except Exception: + self.logger.exception("Thumbnail generation failed") + job.update(status_line="Thumbnail generation stopped on an internal error.") + job.log_line("Thumbnails failed · internal error") + job.publish( + _frame( + "error", + { + "error": "An internal error occurred", + "message": "Thumbnail generation failed. Check server logs.", + }, + ) + ) + state = "failed" + job.finish(state) + + async def _thumbnail_job_body(self, job, options): + """Plan, then generate in worker batches; returns the final state. + + Every exit sets the job's final status line (I3): what was written, + that there was nothing to do, or what a cancel kept. + """ + output_dirs = await self._run_in_executor(self._get_all_output_dirs) + if not output_dirs: + message = "ComfyUI output directory not found" + job.update(status_line=message) + job.log_line(f"Thumbnails failed · {message}") + job.publish(_frame("error", {"error": message})) + return "failed" + folders = _plural(len(output_dirs), "output folder") + job.scope["output_folders"] = len(output_dirs) + job.update(status_line=f"Looking for media files in {folders}") + job.log_line( + f"Thumbnails started · {folders} · {_MODE_LOG[options.mode]} · " + f"{options.edge} px · {options.quality}" + ) + job.publish( + _frame( + "status", + {"phase": "scanning", "message": "Scanning for images and videos..."}, + ) + ) + plan = await self._run_in_executor( + self._plan_gallery_thumbnails, + output_dirs, + rebuild=options.rebuild, + stop=lambda: job.cancelled, + ) + if job.cancelled: # M2: stopped during the walk, before any start frame + return self._end_cancelled(job, 0, CANCELLED_BEFORE_WRITING) + roots = _scan_roots(output_dirs) + self._record_fresh(job, plan, roots) + stats, start = await self._generate_thumbnail_targets(job, options, plan, roots) + self.invalidate_gallery_cache() + generated = _plural(stats["generated"], "preview") + if job.cancelled: + return self._end_cancelled( + job, stats["generated"], f"Cancelled · {generated} written" + ) + payload = _thumbnail_complete_payload( + len(plan["targets"]) + plan["skipped"] + plan["remaining"], + stats, + start, + remaining=plan["remaining"], + processed=len(plan["targets"]), + ) + job.update(progress=100, status_line=_finished_line(plan, stats)) + if plan["remaining"]: + job.log_line( + f"{plan['remaining']:,} more previews are left for the next run" + ) + errors = _plural(len(stats["errors"]), "error") + job.log_line(f"Thumbnails finished · {generated} written · {errors}") + self.logger.info(payload["message"]) + job.publish(_frame("complete", payload)) + return "finished" + + @staticmethod + def _end_cancelled(job, generated, status_line): + """Close a cancelled job: its status line, log line and error frame.""" + job.update(status_line=status_line) + job.log_line(f"Thumbnails cancelled · {_plural(generated, 'preview')} written") + job.publish( + _frame("error", {"error": CANCELLED_ERROR, "message": CANCELLED_ERROR}) + ) + return "cancelled" + + def _record_fresh(self, job, plan, roots): + """List the already fresh previews under Skipped (no log line each).""" + rows = [] + for src in plan["fresh_files"]: + path, label, index = _scan_location(src, roots) + rows.append( + { + "path": path, + "root": label, + "root_index": index, + "detail": _DETAILS["skipped"], + } + ) + job.record_counted("skipped", plan["skipped"], rows) + if plan["skipped"]: + job.log_line(f"= {plan['skipped']:,} previews already up to date") + + async def _generate_thumbnail_targets(self, job, options, plan, roots): + """Generate ``plan``'s targets in worker batches, publishing each file.""" + targets = plan["targets"] + total = len(targets) + video_count = sum(1 for _, _, is_video in targets if is_video) + message = ( + f"Found {total - video_count} images and {video_count} videos to process" + ) + if plan["skipped"]: + message += f" ({plan['skipped']} already have thumbnails)" + job.update(total=total) + if total: # never "Processing file 0 of 0" (I3) + job.update(status_line=_scan_status_line(0, total, 0, 0)) + job.publish( + _frame( + "start", + { + "total_images": total, + "skipped": plan["skipped"], + "remaining": plan["remaining"], + "phase": "processing", + "image_count": total - video_count, + "video_count": video_count, + "message": message, + }, + ) + ) + stats = _new_thumbnail_stats() + stats["skipped"] = plan["skipped"] + start = time.monotonic() + batch_size = _thumbnail_batch_size(_worker_threads()) + generate = { + "encoding": options.encoding, + "rebuild": options.rebuild, + "stop": lambda: job.cancelled, + } + index = 0 + batches = 0 + for batch_start in range(0, total, batch_size): + if job.cancelled: + break + batch = targets[batch_start : batch_start + batch_size] + results = await self._run_in_executor( + self._generate_batch, batch, options.size, **generate + ) + for (src, _dst, _video), result in zip(batch, results): + if result["action"] == "cancelled": + continue + index += 1 + _record_thumbnail_result(stats, result) + self._record_thumbnail_outcome(job, src, result, roots) + job.update(processed=index, progress=index * 100 // total) + if result["action"] == "error": + job.publish( + _frame( + "file_error", + {"file": result["file"], "error": result["error"]}, + ) + ) + job.publish( + _frame( + "progress", + _thumbnail_progress_payload(index, total, stats, result, start), + ) + ) + batches += 1 + job.update( + status_line=_scan_status_line( + index, total, batches, time.monotonic() - start + ) + ) + await asyncio.sleep(0) # let other requests run between batches + return stats, start + + @staticmethod + def _record_thumbnail_outcome(job, src, result, roots): + path, label, index = _scan_location(src, roots) + where = f"{label}/{path}" if label else path + if result["action"] == "error": + group, detail = "errors", result.get("error") or "failed" + elif result["action"] == "skipped": + group, detail = "skipped", _DETAILS["skipped"] + else: + group, detail = "generated", _DETAILS["generated"] + row = {"path": path, "root": label, "root_index": index, "detail": detail} + if group == "skipped": + job.record_counted(group, 1, [row]) + return + job.record(group, row, f"{_LOG_PREFIXES[group]} {where} · {detail}") + + # ── the legacy gallery stream ───────────────────────────────────── + + async def generate_thumbnails_with_progress(self, request): + """The gallery's thumbnail stream (POST, server-sent events). + + Starts the thumbnail job with the gallery's quality preset as the + preview size (missing previews, medium encoding, as before), or + attaches to the running thumbnail job, and streams its frames. A + legacy hold (the blocking generate or clear) or a record-changing + job refuses it with the stream's error frame. The job goes on when + the client goes away. + """ + thumbnail_size = self._thumbnail_size(request.query.get("quality", "medium")) + response = web.StreamResponse( + status=200, + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + "Connection": "keep-alive", + }, + ) + await response.prepare(request) + # M3: read the registry after the last await, so a job that started + # while the headers went out is attached to, not refused. + job, started, busy = self._legacy_stream_job(thumbnail_size) + if busy is not None: + message = f"{busy.title} already running" + await self._emit_sse( + response, "error", {"error": message, "message": message} + ) + return response + # No await between the start and the subscription: the lead frame is + # sent after it, so the stream misses none of the runner's frames. + events = job.events(lead=_frame("job", {"id": job.id, "started": started})) + try: + async for payload in events: + frame = ( + f"event: {payload['event']}\ndata: {json.dumps(payload['data'])}" + ) + await response.write(f"{frame}\n\n".encode("utf-8")) + except Exception as exc: # a write failed: the client went away + self.logger.info( + f"Thumbnail progress client disconnected ({exc}); " + "the thumbnails continue in the background" + ) + finally: + # Also when aiohttp cancels the handler on disconnect: the job is + # its own task and goes on. + await events.aclose() + return response + + def _legacy_stream_job(self, thumbnail_size): + """``(job, started, busy)`` for the legacy stream, synchronously. + + Attaches to the running cancellable thumbnail job, or starts one with + the gallery's preset as the size (missing previews, medium encoding, + as the gallery always wrote), or names the job in the way. + """ + running = self._jobs.active(THUMBNAILS_JOB_KIND) + if running is not None and running.cancellable: + return running, False, None + busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded") + if busy is not None: + return None, False, busy + # blocker() said free and nothing awaited since, so the start is admitted. + options = ThumbnailOptions("missing", thumbnail_size[0], "medium") + return self._start_thumbnail_job(options), True, None diff --git a/tests/browser/system-concept.spec.mjs b/tests/browser/system-concept.spec.mjs index a0b0ada..f5f9e97 100644 --- a/tests/browser/system-concept.spec.mjs +++ b/tests/browser/system-concept.spec.mjs @@ -775,3 +775,131 @@ test('add source comparison detects mutations', async ({ page, request }) => { await undoStyle(); expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]); }); + +/** + * Runs in the page: the open "Thumbnail maintenance" dialog (the concept's `thumbnail-review`, app.js 1433–1437): its + * title, the body's tree, each `.field` (label, control, options, selection, limits), the `p.small` line and the + * actions, with the concept's geometry and tokens. The concept has no `p.small`, so a probe one stands in for its style. + */ +function measureThumbnailReview() { + const dialog = document.querySelector('dialog#modal[open]'); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + const name = node => [node.localName, ...[...node.classList].sort()].join('.'); + const text = node => node?.textContent.trim() ?? null; + const body = dialog.querySelector('#modal-body'); + const fields = [...body.querySelectorAll(':scope > .field')].map(field => { + const label = field.querySelector(':scope > label'); + const control = field.querySelector(':scope > select, :scope > input'); + return { + tree: [...field.children].map(name), + ...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom'), + label: { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') }, + control: { id: control.id, tag: control.localName, type: control.type, + options: control.options ? [...control.options].map(option => option.textContent) : null, + selected: control.options ? control.selectedIndex : null, + value: control.localName === 'input' ? control.value : null, + min: control.getAttribute('min'), max: control.getAttribute('max'), + ...css(control, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') }, + }; + }); + let line = body.querySelector(':scope > p.small'); + const probe = !line; + if (probe) { line = document.createElement('p'); line.className = 'small'; body.append(line); } + const plan = { text: probe ? null : text(line), role: probe ? null : line.getAttribute('role'), + ...css(line, 'fontSize', 'color', 'marginTop', 'marginBottom') }; + if (probe) line.remove(); + const actions = body.querySelector(':scope > .dialog-actions'); + const rect = dialog.getBoundingClientRect(); + return { + title: text(dialog.querySelector('#modal-title')), + tree: [...body.children].map(name), + fields, + plan, + actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'), + buttons: [...actions.children].map(button => ({ text: text(button), className: button.className, + ...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) }, + width: Math.round(rect.width), + }; +} + +/** + * D99: the planned counts line (`p.small[role=status]`) before the actions, from the server's plan for the chosen + * options, and the primary "Start" (the concept's "Simulate thumbnail job"). + */ +function mapThumbnailConcept(reference, plan) { + const [cancel, start] = reference.actions.buttons; + const tree = [...reference.tree]; + tree.splice(tree.indexOf('div.dialog-actions'), 0, 'p.small'); + return { ...reference, tree, + plan: { ...reference.plan, role: 'status', + text: `${number(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ${number(plan.fresh)} already up to date.` }, + actions: { ...reference.actions, buttons: [cancel, { ...start, text: 'Start' }] } }; +} + +async function conceptThumbnailReview(page, theme) { + await conceptSources(page, theme, { notice: false }); + const row = page.locator('#main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) }); + await row.getByRole('button', { name: 'Review options', exact: true }).click(); + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance'); + await page.mouse.move(0, 0); + return page.evaluate(measureThumbnailReview); +} + +/** Production's dialog from Sources' Review options or Overview's Review thumbnails, once its plan line answered. */ +async function productionThumbnailReview(page, theme, entry) { + if (entry === 'sources') { + await productionSources(page, theme, { notice: false }); + const row = page.locator('#workspace-main .operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) }); + await row.getByRole('button', { name: 'Review options', exact: true }).click(); + } else { + await openProduction(page, 'system/overview', theme); + await page.locator('#workspace-main').getByRole('button', { name: 'Review thumbnails', exact: true }).click(); + } + await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Thumbnail maintenance'); + await expect(page.locator('dialog#modal[open] #modal-body > p.small')).toHaveText(/ already up to date\.$/); + await page.mouse.move(0, 0); + return page.evaluate(measureThumbnailReview); +} + +const thumbnailPlan = request => serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium'); + +for (const entry of ['sources', 'overview']) for (const theme of THEMES) for (const [width, height] of SIZES) { + test(`thumbnail dialog matches the concept from ${entry} ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => { + await page.setViewportSize({ width, height }); + await watchedSources(request); + const plan = await thumbnailPlan(request); + const concept = await conceptThumbnailReview(page, theme); + await page.screenshot({ path: testInfo.outputPath(`concept-thumbnail-review-${theme}-${width}.png`) }); + const production = await productionThumbnailReview(page, theme, entry); + await page.screenshot({ path: testInfo.outputPath(`production-thumbnail-review-${entry}-${theme}-${width}.png`) }); + expect(diffInventory(mapThumbnailConcept(concept, plan), production)).toEqual([]); + }); +} + +test('thumbnail dialog comparison detects mutations', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + await watchedSources(request); + const plan = await thumbnailPlan(request); + const concept = await conceptThumbnailReview(page, 'dark'); + const expected = mapThumbnailConcept(concept, plan); + const production = await productionThumbnailReview(page, 'dark', 'sources'); + expect(diffInventory(expected, production)).toEqual([]); + // Unmapped, the concept differs only where D99 is recorded. + expect(diffInventory(concept, production).map(item => item.path).sort()) + .toEqual(['actions.buttons.1.text', 'plan.role', 'plan.text', 'tree.3', 'tree.4', 'tree.length']); + const paths = async () => diffInventory(expected, await page.evaluate(measureThumbnailReview)).map(item => item.path); + // A changed option label, a swapped option order, a swapped action order and a changed style. + await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium'; }); + expect(await paths()).toEqual(['fields.1.control.options.0']); + await page.evaluate(() => { document.querySelector('#thumb-quality option').textContent = 'Medium · optimized'; }); + await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); }); + expect(await paths()).toEqual(expect.arrayContaining(['fields.0.control.options.0', 'fields.0.control.options.1'])); + await page.evaluate(() => { const select = document.querySelector('#thumb-mode'); select.prepend(select.lastElementChild); select.selectedIndex = 0; }); + await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); }); + expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text'])); + await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); }); + const undoStyle = await mutate(page, { css: 'dialog#modal #thumb-limit { border-color: rgb(255, 0, 0); } dialog#modal #modal-body > p.small { color: rgb(255, 0, 0); }' }); + expect(await paths()).toEqual(expect.arrayContaining(['fields.2.control.borderTopColor', 'plan.color'])); + await undoStyle(); + expect(diffInventory(expected, await page.evaluate(measureThumbnailReview))).toEqual([]); +}); diff --git a/tests/browser/workspace-activity.spec.mjs b/tests/browser/workspace-activity.spec.mjs index 72e95f2..5f16414 100644 --- a/tests/browser/workspace-activity.spec.mjs +++ b/tests/browser/workspace-activity.spec.mjs @@ -54,8 +54,8 @@ async function step(request, id, data) { await expect(await request.post(`/__test__/jobs/${id}/step`, { data })).toBeOK(); } -async function finish(request, id, state = 'finished') { - const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state } }); +async function finish(request, id, state = 'finished', extra = {}) { + const response = await request.post(`/__test__/jobs/${id}/finish`, { data: { state, ...extra } }); await expect(response).toBeOK(); return (await response.json()).job; } @@ -1008,3 +1008,96 @@ test('the report download names the kind', async ({ page, request }) => { expect(report.outcomes.tagged.rows.map(row => [row.path, row.detail])).toEqual([['sub/a.png', 'Added 1 tag']]); await expect(toast(page)).toHaveText('Job report downloaded.'); }); + +// D111 (System S5): each kind's own descriptor. A thumbnail job counts Generated (green), Skipped and Errors (red), +// its pill says "Generating thumbnails" and its toasts "Thumbnails finished · N generated · M errors". +const THUMB_JOB = { kind: 'thumbnails', total: 10, output_folders: 2 }; + +test('a thumbnail job shows Generated Skipped Errors, its pill verb and its toasts', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const job = await startJob(request, THUMB_JOB); + await step(request, job.id, { processed: 4, records: [ + { group: 'generated', path: 'sub/a.png', root: 'output', detail: 'Preview written' }, + { group: 'skipped', path: 'b.png', root: 'input', detail: 'Preview already up to date' }, + { group: 'errors', path: 'broken.png', root: 'output', detail: 'Not a valid image file' }, + ] }); + await page.goto(`${workspace}#/images`); + await expect(pill(page)).toHaveText('Generating thumbnails · 40% · 4 / 10'); + await page.goto(`${workspace}#/activity`); + await expect(items(page).first().locator('p')).toHaveText(/^Started \d\d:\d\d · 2 output folders · closing this view does not cancel it$/); + await expect(items(page).first().locator('progress')).toHaveAttribute('value', '40'); + await openJob(page, job.id); + await expect(heading(page).locator('h1')).toHaveText('Thumbnail generation'); + await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel job']); + expect(await tileValues(page)).toEqual([['stat added', 'Generated', '1'], ['stat', 'Skipped', '1'], ['stat bad', 'Errors', '1']]); + await expect(outcomeButtons(page)).toHaveText(['Generated1', 'Errors1', 'Skipped1']); + await expect(fileRows(page).first().locator('.path')).toHaveText('sub/a.png'); + expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('button'); + await outcomeButtons(page).filter({ hasText: 'Errors' }).click(); + expect(await fileRows(page).first().evaluate(node => node.localName)).toBe('div'); + await expect(scanLog(page).locator('summary')).toHaveText('● Job log — raw detail, click to show'); + + await finish(request, job.id); + await expect(toast(page)).toHaveText('Thumbnails finished · 1 generated · 1 error'); + await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d · 1 generated · 1 error$/); + await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']); +}); + +test('a held legacy job and the health check show no tiles, no Results and no made-up percentage', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const response = await request.post('/__test__/jobs/hold', + { data: { kind: 'thumbnails', title: 'Thumbnail clear (gallery page)', admission: 'guarded' } }); + await expect(response).toBeOK(); + const held = (await response.json()).job; + await page.goto(`${workspace}#/images`); + // D112: the pill names the held operation and shows no percentage; its bar is indeterminate, not an empty fill + // (S5 review M5), and stays hidden from assistive tech, which reads the label. + await expect(pill(page)).toHaveText('Thumbnail clear (gallery page)'); + await expect(pill(page)).toHaveAttribute('aria-label', 'Thumbnail clear (gallery page)'); + await expect(pill(page).locator('.activity-pill-bar')).toHaveClass('activity-pill-bar indeterminate'); + await expect(pill(page).locator('.activity-pill-bar')).toHaveAttribute('aria-hidden', 'true'); + expect(await pill(page).locator('.activity-pill-bar i').evaluate(node => node.style.width)).toBe(''); + await page.goto(`${workspace}#/activity`); + const item = items(page).filter({ has: page.getByRole('link', { name: /Thumbnail clear \(gallery page\)$/ }) }); + await expect(item.locator('p')).toHaveText(/^Started \d\d:\d\d · closing this view does not cancel it$/); + await expect(item.locator('progress')).toHaveCount(1); + await expect(item.locator('progress')).not.toHaveAttribute('value'); + await expect(item.getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0); + await item.getByRole('link').click(); + await expect(heading(page).locator('h1')).toHaveText('Thumbnail clear (gallery page)'); + await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value'); + await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0); + await expect(resultsSection(page)).toHaveCount(0); + await expect(scanLog(page)).toHaveCount(1); + await finish(request, held.id); + await expect(toast(page)).toHaveText('Thumbnail clear (gallery page) finished.'); + await expect(heading(page).locator('p.muted')).toHaveText(/^Finished \d\d:\d\d$/); + await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100'); + + // The health check (a read job of its own kind): its status line, no tiles, no Cancel. + const health = await startJob(request, { kind: 'health', total: 0 }); + await openJob(page, health.id); + await expect(heading(page).locator('h1')).toHaveText('Health check'); + await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Checking the database and image links'); + await expect(progressSection(page).locator('progress')).not.toHaveAttribute('value'); + await expect(progressSection(page).locator('dl.stat-row')).toHaveCount(0); + await expect(resultsSection(page)).toHaveCount(0); + await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']); + await expect(pill(page)).toHaveText('Health check'); + await finish(request, health.id); + await expect(toast(page)).toHaveText('Health check finished.'); +}); + +// S5 review I3: a thumbnail job with nothing to generate ends on the server's final line ("No previews to generate · N +// already up to date"), never on a stale "Processing file 0 of 0" under the Completed chip. +test('a finished thumbnail job with nothing to do says so', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const job = await startJob(request, { ...THUMB_JOB, total: 0 }); + await step(request, job.id, { total: 0, processed: 0 }); // the stale line an older server left + await openJob(page, job.id); + const line = progressSection(page).locator('.panel-heading > span.small'); + await finish(request, job.id, 'finished', { status_line: 'No previews to generate · 24 already up to date' }); + await expect(line).toHaveText('No previews to generate · 24 already up to date'); + await expect(page.getByText('Processing file 0 of 0')).toHaveCount(0); + await expect(progressSection(page).locator('progress')).toHaveAttribute('value', '100'); +}); diff --git a/tests/browser/workspace-system-sources.spec.mjs b/tests/browser/workspace-system-sources.spec.mjs index 5fca0dc..3925001 100644 --- a/tests/browser/workspace-system-sources.spec.mjs +++ b/tests/browser/workspace-system-sources.spec.mjs @@ -304,16 +304,151 @@ test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.'); }); -test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => { +const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan'; +const THUMBNAIL_JOB_URL = '/prompt_manager/jobs/thumbnails'; +const planCount = value => value.toLocaleString('en-US'); + +/** Open "Thumbnail maintenance" from Sources' Thumbnail generation → Review options. */ +async function openThumbnailReview(page) { const layout = await openSources(page); const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) }); await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.'); - const popup = page.waitForEvent('popup'); await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click(); - const gallery = await popup; - expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery'); - await gallery.close(); - await expect(statusOf(page)).toHaveText('No unsaved changes.'); + await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance'); + return dialog(page); +} + +/** The thumbnail requests this page sends: plan reads (their query) and job starts (their body). */ +function thumbnailRequests(page) { + const seen = { plans: [], starts: [] }; + page.on('request', item => { + const url = new URL(item.url()); + if (url.pathname === THUMBNAIL_PLAN_URL) seen.plans.push(Object.fromEntries(url.searchParams)); + if (item.method() === 'POST' && url.pathname === THUMBNAIL_JOB_URL) seen.starts.push(item.postDataJSON()); + }); + return seen; +} + +const missingLine = plan => `${planCount(plan.targets)} ${plan.targets === 1 ? 'preview' : 'previews'} to generate · ` + + `${planCount(plan.fresh)} already up to date.`; + +test('Review options shows the plan counts for the chosen scope/quality/size before Start', async ({ page, request }) => { + await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 3 } })).toBeOK(); + // Image loading quality (Appearance, Q3) preselects the dialog's quality. + await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('gallerySettings', JSON.stringify({ imageQuality: 'high' })); }); + const seen = thumbnailRequests(page); + const plan = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=missing&size=512&quality=high`); + expect(plan.targets).toBe(3); + const review = await openThumbnailReview(page); + await expect(review.locator('#modal-body > *')).toHaveCount(5); + await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']); + await expect(review.locator('#thumb-mode option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)']); + await expect(review.locator('#thumb-quality option')).toHaveText(['Medium · optimized', 'High · original', 'Low · fastest']); + await expect(review.locator('#thumb-mode')).toHaveValue('missing'); + await expect(review.locator('#thumb-quality')).toHaveValue('high'); + await expect(review.locator('#thumb-limit')).toHaveValue('512'); + await expect(review.locator('#thumb-limit')).toHaveAttribute('min', '128'); + await expect(review.locator('#thumb-limit')).toHaveAttribute('max', '2048'); + const line = review.locator('#modal-body > p.small'); + await expect(line).toHaveAttribute('role', 'status'); + await expect(line).toHaveText(missingLine(plan)); + expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'high' }]); + await expect(review.locator('.dialog-actions > button')).toHaveText(['Cancel', 'Start']); + // Reviewing starts nothing; Cancel closes. + await review.getByRole('button', { name: 'Cancel', exact: true }).click(); + await expect(dialog(page)).toHaveCount(0); + expect(seen.starts).toEqual([]); + expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]); +}); + +test('size outside 128–2048 reports validity and requests nothing', async ({ page, request }) => { + const seen = thumbnailRequests(page); + const review = await openThumbnailReview(page); + const line = review.locator('#modal-body > p.small'); + await expect(line).toHaveText(/ already up to date\.$/); + const size = review.locator('#thumb-limit'); + for (const value of ['4096', '64', '']) { + await size.fill(value); + await expect(size).toHaveAttribute('aria-invalid', 'true'); + await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.'); + expect(await size.evaluate(input => input.validity.valid)).toBe(false); + // Start keeps the dialog open and sends nothing; neither does a scope change. + await review.getByRole('button', { name: 'Start', exact: true }).click(); + await expect(dialog(page).locator('#modal-title')).toHaveText('Thumbnail maintenance'); + await review.locator('#thumb-mode').selectOption('rebuild'); + await expect(line).toHaveText('Enter a size from 128 to 2048 pixels.'); + await review.locator('#thumb-mode').selectOption('missing'); + } + expect(seen.starts).toEqual([]); + expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }]); + await size.fill('1024'); + await expect(size).not.toHaveAttribute('aria-invalid'); + await expect(line).toHaveText(/ already up to date\.$/); + expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]); +}); + +test('Start closes the dialog, toasts, and opens the job in Activity', async ({ page, request }) => { + await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK(); + const seen = thumbnailRequests(page); + const review = await openThumbnailReview(page); + await review.locator('#thumb-quality').selectOption('low'); + await review.locator('#thumb-limit').fill('256'); + await review.getByRole('button', { name: 'Start', exact: true }).click(); + await expect(dialog(page)).toHaveCount(0); + await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/); + // The toast says it started (or, for a run this quick, that it already finished); Activity shows its progress. + await expect(toast(page)).toHaveText(/^(Thumbnail generation started\.|Thumbnails finished · 2 generated · 0 errors)$/); + expect(seen.starts).toEqual([{ mode: 'missing', size: 256, quality: 'low' }]); + const id = page.url().split('/').pop(); + const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`); + expect(job).toMatchObject({ kind: 'thumbnails', title: 'Thumbnail generation', scope: { mode: 'missing', size: 256, quality: 'low' } }); + await expect(page.locator('#workspace-main h1')).toHaveText('Thumbnail generation'); + await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished'); +}); + +test('Rebuild all is explicit and the plan line says every preview is regenerated', async ({ page, request }) => { + const seen = thumbnailRequests(page); + const rebuild = await serverJson(request, `${THUMBNAIL_PLAN_URL}?mode=rebuild&size=512&quality=medium`); + expect(rebuild.targets).toBeGreaterThan(0); + const review = await openThumbnailReview(page); + // Missing thumbnails only is the default; rebuilding is the explicit second option. + await expect(review.locator('#thumb-mode')).toHaveValue('missing'); + const line = review.locator('#modal-body > p.small'); + await expect(line).toHaveText(/ already up to date\.$/); + await review.locator('#thumb-mode').selectOption('rebuild'); + await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`); + expect(seen.plans).toEqual([{ mode: 'missing', size: '512', quality: 'medium' }, { mode: 'rebuild', size: '512', quality: 'medium' }]); + // Back and forth reads each scope once. + await review.locator('#thumb-mode').selectOption('missing'); + await expect(line).toHaveText(/ already up to date\.$/); + await review.locator('#thumb-mode').selectOption('rebuild'); + await expect(line).toHaveText(`Every preview is regenerated: ${planCount(rebuild.targets)} files.`); + expect(seen.plans).toHaveLength(2); + await review.getByRole('button', { name: 'Start', exact: true }).click(); + await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/); + expect(seen.starts).toEqual([{ mode: 'rebuild', size: 512, quality: 'medium' }]); + const id = page.url().split('/').pop(); + await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished'); +}); + +test('a plan that cannot be read says so; Start still starts', async ({ page }) => { + await page.route(`**${THUMBNAIL_PLAN_URL}?*`, route => route.fulfill({ status: 500, json: { success: false, error: 'disk' } })); + const review = await openThumbnailReview(page); + await expect(review.locator('#modal-body > p.small')).toHaveText('The previews could not be counted.'); + await expect(review.getByRole('button', { name: 'Start', exact: true })).not.toHaveAttribute('aria-disabled'); + await review.getByRole('button', { name: 'Cancel', exact: true }).click(); +}); + +test('a refused start shows the busy link toast', async ({ page, request }) => { + const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json(); + const review = await openThumbnailReview(page); + await review.getByRole('button', { name: 'Start', exact: true }).click(); + await expect(dialog(page)).toHaveCount(0); + await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.'); + await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`); + await expect(page).toHaveURL(`${workspace}#/system/sources`); + expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['scan']); + await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } }); }); test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => { diff --git a/tests/browser/workspace-system.spec.mjs b/tests/browser/workspace-system.spec.mjs index ac1559f..0080f28 100644 --- a/tests/browser/workspace-system.spec.mjs +++ b/tests/browser/workspace-system.spec.mjs @@ -140,9 +140,32 @@ test('thumbnail row reflects the status endpoint', async ({ page, request }) => const layout = await openOverview(page); const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) }); await expect(row.locator('h3')).toHaveText('3 missing thumbnails'); - // Until the thumbnail review dialog (S5), Review thumbnails opens Sources. +}); + +test('Overview Review thumbnails opens the same dialog', async ({ page, request }) => { + await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 2 } })).toBeOK(); + const plan = await serverJson(request, '/prompt_manager/images/thumbnails/plan?mode=missing&size=512&quality=medium'); + const starts = []; + page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/thumbnails')) starts.push(item.postDataJSON()); }); + const layout = await openOverview(page); + const row = layout.locator('.operation').filter({ has: page.getByRole('button', { name: 'Review thumbnails', exact: true }) }); + await expect(row.locator('h3')).toHaveText('2 missing thumbnails'); await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click(); - await expect(page).toHaveURL(`${workspace}#/system/sources`); + const review = page.locator('dialog#modal[open]'); + await expect(review.locator('#modal-title')).toHaveText('Thumbnail maintenance'); + await expect(page).toHaveURL(`${workspace}#/system/overview`); + await expect(review.locator('.field > label')).toHaveText(['Scope', 'Quality', 'Preview size limit']); + await expect(review.locator('#modal-body > p.small')).toHaveText(`2 previews to generate · ${number(plan.fresh)} already up to date.`); + // Escape closes it and focus returns to the row's button. + await page.keyboard.press('Escape'); + await expect(review).toHaveCount(0); + await expect(row.getByRole('button', { name: 'Review thumbnails', exact: true })).toBeFocused(); + await row.getByRole('button', { name: 'Review thumbnails', exact: true }).click(); + await review.getByRole('button', { name: 'Start', exact: true }).click(); + await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/); + expect(starts).toEqual([{ mode: 'missing', size: 512, quality: 'medium' }]); + const id = page.url().split('/').pop(); + await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.state).toBe('finished'); }); test('the thumbnail row paints when its own status read answers', async ({ page, request }) => { diff --git a/tests/browser/workspace_activity_fixture.py b/tests/browser/workspace_activity_fixture.py index 4d678c1..843b80f 100644 --- a/tests/browser/workspace_activity_fixture.py +++ b/tests/browser/workspace_activity_fixture.py @@ -5,18 +5,23 @@ reading files, so a test sets its progress and counters exactly: - ``POST /__test__/jobs/start`` starts one (``kind``, ``title``, ``total``, ``output_folders``; for ``kind: "image-tags"`` the image tagging job's - groups and its ``{mode, images}`` scope instead) and returns its summary, - or 409 while one runs; + groups and its ``{mode, images}`` scope instead; for ``kind: + "thumbnails"`` the thumbnail job's groups, ``guarded`` admission and its + ``{mode, size, quality, output_folders}`` scope; for ``kind: "health"`` the + health check's: no groups, ``read``, not cancellable, with its status + line) and returns its summary, or 409 while one runs; - ``POST /__test__/jobs/{id}/step`` records ``records`` (``{group, path, root, root_index, detail}`` file outcomes, counted, listed and logged as the scan records them) and ``bulk`` (``{group, count, root}``: ``count`` numbered rows), sets ``processed`` (and ``total``) and ``counters``, then publishes the progress to every subscriber; - ``POST /__test__/jobs/{id}/finish`` ends it as ``finished``, ``cancelled`` - or ``failed``; a cancel through the real route ends it ``cancelled``; -- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``): - a running, non-cancellable ``records`` job, as a legacy restore or - maintenance request takes, until ``finish`` ends it; + or ``failed``, with the runner's final ``status_line`` when given; a cancel + through the real route ends it ``cancelled``; +- ``POST /__test__/jobs/hold`` opens a legacy-page hold (``kind``, ``title``, + ``admission``, default ``records``): a running, non-cancellable job with no + outcome groups, as a legacy restore, maintenance or thumbnail clear request + takes, until ``finish`` ends it; - ``POST /__test__/jobs/restart`` acts out a ComfyUI restart: the open job streams drop without a final event and the registry starts empty. """ @@ -35,6 +40,9 @@ PREFIXES = {"added": "+", "known": "=", "review": "?", "errors": "!"} PREFIXES["no_metadata"] = "-" # The image tagging job's groups (#249 PR D), logged with the runner's prefixes. PREFIXES.update({"tagged": "+", "unchanged": "=", "skipped": "-"}) +# The thumbnail job's (#249 System S5). +PREFIXES["generated"] = "+" +HEALTH_STATUS_LINE = "Checking the database and image links" ROOT_INDEXES = {"output": 0, "input": 1} @@ -80,9 +88,12 @@ def outcome_records(data): def register_activity_commands(app, routes, api): from py.api.jobs import ( + HEALTH_JOB_KIND, IMAGE_TAG_GROUP_NAMES, IMAGE_TAG_JOB_KIND, SCAN_GROUP_NAMES, + THUMBNAIL_GROUP_NAMES, + THUMBNAILS_JOB_KIND, JobBusy, JobRegistry, ) @@ -138,7 +149,21 @@ def register_activity_commands(app, routes, api): data = await body(request) ending = asyncio.get_running_loop().create_future() kind = str(data.get("kind", "scan")) - if kind == IMAGE_TAG_JOB_KIND: + admission, cancellable = "records", True + if kind == THUMBNAILS_JOB_KIND: + title = "Thumbnail generation" + scope = { + "mode": str(data.get("mode", "missing")), + "size": count(data.get("size"), 512), + "quality": str(data.get("quality", "medium")), + "output_folders": count(data.get("output_folders"), 2), + } + groups = THUMBNAIL_GROUP_NAMES + admission = "guarded" + elif kind == HEALTH_JOB_KIND: + title, scope, groups = "Health check", {}, () + admission, cancellable = "read", False + elif kind == IMAGE_TAG_JOB_KIND: title = "Add tags to images" scope = { "mode": str(data.get("mode", "add")), @@ -162,7 +187,8 @@ def register_activity_commands(app, routes, api): job = api._jobs.start( kind, str(data.get("title", title)), - record_changing=True, + admission=admission, + cancellable=cancellable, scope=scope, runner=fake_runner(ending), groups=groups, @@ -170,6 +196,8 @@ def register_activity_commands(app, routes, api): except JobBusy: raise web.HTTPConflict() job.update(total=count(data.get("total"))) + if kind == HEALTH_JOB_KIND: + job.update(status_line=HEALTH_STATUS_LINE) endings[job.id] = ending return web.json_response({"success": True, "job": job.summary()}) @@ -184,7 +212,7 @@ def register_activity_commands(app, routes, api): async with api._jobs.hold( str(data.get("kind", "restore")), str(data.get("title", "Database restore (admin page)")), - admission="records", + admission=str(data.get("admission", "records")), ) as job: entered.set_result(job) state = await ending @@ -232,9 +260,12 @@ def register_activity_commands(app, routes, api): @routes.post("/__test__/jobs/{job_id}/finish") async def finish_job(request): job = job_or_404(request) - state = (await body(request)).get("state", "finished") + data = await body(request) + state = data.get("state", "finished") if state not in FINAL_STATES: raise web.HTTPBadRequest() + if isinstance(data.get("status_line"), str): # the runner's final line + job.update(status_line=data["status_line"]) ending = endings[job.id] if not ending.done(): ending.set_result(state) diff --git a/tests/js/gallery-thumbnails.test.js b/tests/js/gallery-thumbnails.test.js index 5b085f2..cb13516 100644 --- a/tests/js/gallery-thumbnails.test.js +++ b/tests/js/gallery-thumbnails.test.js @@ -110,3 +110,123 @@ test('saving gallery settings retains the remembered thumbnail dismissal', () => f.manager.saveSettings(); assert.equal(JSON.parse(f.storage.get('gallerySettings')).thumbnailPromptDismissed, 'remembered-state'); }); + +// #249 System S5: the stream's work is now an Activity job that outlives the page's connection, so the gallery's +// Cancel asks the server to stop the job. The stream's first frame, `job {id, started}`, names the job; the gallery +// cancels by that id, only a job this stream started, and a Cancel before the id waits for it (review I1). +const JOB = 'b'.repeat(32); + +function streamFixture({ modal = false, postFails = false } = {}) { + const posts = []; + const warnings = []; + const gets = []; + const listeners = {}; + const source = { closed: 0, readyState: 1, CLOSED: 2, url: null, + addEventListener(type, handler) { (listeners[type] ||= []).push(handler); }, + close() { this.closed += 1; } }; + const element = () => ({ classList: { add() {}, remove() {} }, style: {}, textContent: '', handlers: {}, + addEventListener(type, handler) { this.handlers[type] = handler; }, + removeEventListener(type, handler) { if (this.handlers[type] === handler) delete this.handlers[type]; }, + click() { this.handlers.click?.(); } }); + const elements = {}; + const document = { addEventListener() {}, getElementById: id => elements[id] ||= element(), + createElement: () => element(), body: { appendChild() {}, removeChild() {} } }; + const context = vm.createContext({ document, window: {}, + console: { debug() {}, warn: (...args) => warnings.push(args), error() {} }, + setTimeout() {}, localStorage: { setItem() {} }, + SseStream: { connect: (url, options) => { source.url = url; source.options = options; return source; } } }); + const file = path.join(__dirname, '../../web/js/gallery.js'); + const Manager = vm.runInContext(fs.readFileSync(file, 'utf8') + '\nGalleryManager;', context, { filename: file }); + const manager = Object.create(Manager.prototype); + Object.assign(manager, { getSettings: () => ({}), + api: { get: async url => { gets.push(url); return { success: true, jobs: [] }; }, + post: async (url, body) => { + posts.push([url, body]); + if (postFails) throw new Error('offline'); + return { success: true }; + } } }); + const emit = (type, data) => (listeners[type] || []).forEach(handler => handler({ data: JSON.stringify(data) })); + const ui = { addStatusMessage() {}, progressText: element(), progressPercent: element(), progressBar: element(), + progressDetails: element(), progressETA: element(), isCancelled: () => false, + onCancel: handler => { ui.cancel = handler; } }; + const result = modal ? manager.generateThumbnailsWithProgressInModal(ui) + : manager.generateThumbnailsWithProgress({ quality: 'medium' }); + const outcome = result.then(value => ({ value }), error => ({ error: error.message })); + const cancel = () => (modal ? ui.cancel() : elements.cancelThumbnailsBtn.click()); + const settle = () => new Promise(resolve => setImmediate(resolve)); + return { emit, source, posts, gets, warnings, outcome, cancel, settle }; +} + +const cancelUrl = id => `/prompt_manager/jobs/${id}/cancel`; + +test('Cancel before the job id arrives cancels that job once it does', async () => { + const f = streamFixture(); + f.cancel(); + await f.settle(); + assert.deepEqual(f.posts, []); + assert.equal(f.source.closed, 0); // kept open until the stream names its job + assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' }); + f.emit('status', { phase: 'scanning', message: 'Scanning…' }); + f.emit('job', { id: JOB, started: true }); + await f.settle(); + assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]])); + assert.equal(f.source.closed, 1); + f.emit('job', { id: 'c'.repeat(32), started: true }); // a stray second frame changes nothing + await f.settle(); + assert.equal(f.posts.length, 1); + assert.deepEqual(f.gets, []); // never looks a job up by kind +}); + +test('Cancel after the id cancels it at once, and a double Cancel posts once', async () => { + const f = streamFixture(); + f.emit('job', { id: JOB, started: true }); + f.cancel(); + f.cancel(); + await f.settle(); + assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]])); + assert.equal(f.source.closed, 1); + assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' }); + assert.deepEqual(f.gets, []); +}); + +test("Cancel leaves another tab's or System's job running and only stops following it", async () => { + const f = streamFixture(); + f.emit('job', { id: JOB, started: false }); + f.cancel(); + await f.settle(); + assert.deepEqual(f.posts, []); + assert.deepEqual(f.gets, []); + assert.equal(f.source.closed, 1); + assert.match((await f.outcome).error, /keeps running/); +}); + +test('a Cancel the stream never names a job for posts nothing and closes when it ends', async () => { + const f = streamFixture(); + f.cancel(); + f.emit('error', { error: 'Scan for metadata already running' }); + await f.settle(); + assert.deepEqual(f.posts, []); + assert.equal(f.source.closed, 1); +}); + +test('a cancel post that fails is reported, not thrown', async () => { + const f = streamFixture({ postFails: true }); + f.emit('job', { id: JOB, started: true }); + f.cancel(); + await f.settle(); + assert.equal(f.posts.length, 1); + assert.equal(f.warnings.length, 1); + assert.deepEqual(await f.outcome, { error: 'Thumbnail generation cancelled by user' }); +}); + +test('the startup dialog cancels its own job by id too', async () => { + const f = streamFixture({ modal: true }); + f.cancel(); + await f.settle(); + assert.deepEqual(f.posts, []); + f.emit('job', { id: JOB, started: true }); + await f.settle(); + assert.equal(JSON.stringify(f.posts), JSON.stringify([[cancelUrl(JOB), {}]])); + assert.equal(f.source.closed, 1); + assert.match((await f.outcome).error, /cancelled/); +}); diff --git a/tests/js/workspace-activity-kinds.test.js b/tests/js/workspace-activity-kinds.test.js new file mode 100644 index 0000000..5cd4c8c --- /dev/null +++ b/tests/js/workspace-activity-kinds.test.js @@ -0,0 +1,148 @@ +// Per-kind Activity descriptors (#249 System S5 Task 13, D111): each job kind's pill verb, toasts, line, tiles, +// outcomes, openable rows and labels, from one table (web/js/workspace/activity-kinds.mjs). +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const source = import('../../web/js/workspace/activity-kinds.mjs'); +const store = import('../../web/js/workspace/activity-store.mjs'); + +const ID = 'a'.repeat(32); + +function job(overrides = {}) { + return { id: ID, kind: 'scan', title: 'Scan for metadata', state: 'running', started_at: 1, finished_at: null, + progress: 42, status_line: '', processed: 8240, total: 19622, + counters: { added: 1, known: 0, review: 0, no_metadata: 0, errors: 1 }, scope: {}, listed: {}, log_length: 0, + cancellable: true, ...overrides }; +} + +test('scan descriptors are unchanged', async () => { + const { kindOf } = await source; + const scan = kindOf('scan'); + assert.equal(scan.pillVerb, 'Scanning'); + assert.equal(scan.startToast(job()), 'Scan started.'); + assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''], + ['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]); + assert.deepEqual(scan.outcomes, [['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'], + ['no_metadata', 'No usable metadata'], ['known', 'Already known']]); + assert.deepEqual([...scan.openable].sort(), ['added', 'known']); + assert.deepEqual([scan.log, scan.cancelLabel], ['Scan log', 'Cancel scan']); + assert.equal(scan.finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error'); + assert.equal(scan.finishToast(job({ state: 'cancelled' })), 'Scan cancelled · 1 added'); + assert.equal(scan.finishToast(job({ state: 'failed' })), 'Scan failed · 1 added · 1 error'); + assert.equal(scan.line(job()), '1 added · 1 error'); + assert.equal(scan.kept(job()), '1 added'); + assert.equal(scan.scope(job({ scope: { output_folders: 2 } })), '2 output folders'); + // The store keeps answering with the same table. + const { jobView, finishToast, startToast, pillParts, outcomeLine } = await store; + assert.equal(jobView('scan'), scan); + assert.equal(finishToast(job({ state: 'finished' })), 'Scan finished · 1 added · 1 error'); + assert.equal(startToast(job()), 'Scan started.'); + assert.equal(pillParts(job()).text, 'Scanning · 42% · 8,240 / 19,622'); + assert.equal(outcomeLine(job()), '1 added · 1 error'); +}); + +test('thumbnail job shows its own tiles and toasts', async () => { + const { kindOf, viewOf } = await source; + const { finishToast, startToast, pillParts, outcomeLine } = await store; + const thumbs = job({ kind: 'thumbnails', title: 'Thumbnail generation', processed: 4, total: 10, progress: 40, + counters: { generated: 1420, skipped: 3, errors: 1 }, scope: { mode: 'missing', size: 512, quality: 'medium', + output_folders: 2 } }); + const view = kindOf('thumbnails'); + assert.equal(viewOf(thumbs), view); + assert.deepEqual(view.tiles, [['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]); + assert.deepEqual(view.outcomes, [['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]); + assert.deepEqual([...view.openable].sort(), ['generated', 'skipped']); + assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']); + assert.equal(pillParts(thumbs).text, 'Generating thumbnails · 40% · 4 / 10'); + assert.equal(startToast(thumbs), 'Thumbnail generation started.'); + assert.equal(finishToast({ ...thumbs, state: 'finished' }), 'Thumbnails finished · 1,420 generated · 1 error'); + assert.equal(finishToast({ ...thumbs, state: 'failed' }), 'Thumbnails failed · 1,420 generated · 1 error'); + assert.equal(finishToast({ ...thumbs, state: 'cancelled' }), 'Thumbnails cancelled · 1,420 generated'); + assert.equal(finishToast({ ...thumbs, counters: { generated: 0, skipped: 0, errors: 2 }, state: 'finished' }), + 'Thumbnails finished · 0 generated · 2 errors'); + assert.equal(outcomeLine(thumbs), '1,420 generated · 1 error'); + assert.equal(view.kept(thumbs), '1,420 generated'); + assert.equal(view.scope(thumbs), '2 output folders'); +}); + +test('non-cancellable jobs show no Cancel job', async () => { + const { canCancel } = await source; + assert.equal(canCancel(job()), true); + assert.equal(canCancel(job({ cancellable: false })), false); + for (const state of ['finished', 'cancelled', 'failed', 'lost']) assert.equal(canCancel(job({ state })), false); + // The health check is a read job the server cannot stop. + assert.equal(canCancel(job({ kind: 'health', title: 'Health check', cancellable: false, counters: {} })), false); +}); + +test('indeterminate progress has no value attribute', async () => { + const { indeterminate, showProgress } = await source; + const { pillParts } = await store; + function node() { + const attributes = new Map(); + return { attributes, textContent: '', set value(v) { attributes.set('value', String(v)); }, + get value() { return Number(attributes.get('value') ?? 0); }, + setAttribute(key, v) { attributes.set(key, String(v)); }, removeAttribute(key) { attributes.delete(key); } }; + } + // Running with no total yet: the server has not said how much there is, so there is no percentage. + const health = job({ kind: 'health', title: 'Health check', progress: 0, processed: 0, total: 0, counters: {}, + cancellable: false, status_line: 'Checking the database and image links' }); + assert.equal(indeterminate(health), true); + const bar = node(); + bar.value = 37; + showProgress(bar, health); + assert.equal(bar.attributes.has('value'), false); + assert.equal(bar.textContent, ''); + assert.equal(pillParts(health).text, 'Health check'); + assert.deepEqual(pillParts(health), { label: 'Health check', files: '', text: 'Health check' }); + // A known total is determinate, and so is any ended job. + const scan = job(); + assert.equal(indeterminate(scan), false); + showProgress(bar, scan); + assert.equal(bar.attributes.get('value'), '42'); + assert.equal(bar.textContent, '42%'); + assert.equal(indeterminate({ ...health, state: 'finished', progress: 100 }), false); + // A scan before its file count is known is indeterminate too, never "0%". + assert.equal(pillParts(job({ total: 0, processed: 0, progress: 0 })).text, 'Scanning'); +}); + +test('held legacy jobs read "(admin page)" with no outcomes', async () => { + const { viewOf } = await source; + const { jobDetail } = await import('../../web/js/workspace/activity.mjs'); + const { finishToast, outcomeLine } = await store; + // A hold has no outcome groups (counters {}), whatever its kind: thumbnails held by the gallery's clear included. + for (const [kind, title] of [['restore', 'Database restore (admin page)'], + ['thumbnails', 'Thumbnail clear (gallery page)'], ['maintenance', 'Database maintenance (admin page)']]) { + const held = job({ kind, title, counters: {}, cancellable: false, progress: 0, processed: 0, total: 0 }); + const view = viewOf(held); + assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind); + assert.equal(outcomeLine(held), ''); + assert.match(jobDetail(held), /^Started \d\d:\d\d · closing this view does not cancel it$/); + const ended = { ...held, state: 'finished', finished_at: 2 }; + assert.match(jobDetail(ended), /^Finished \d\d:\d\d$/); + assert.equal(finishToast(ended), `${title} finished.`); + assert.ok(title.endsWith('page)')); + } +}); + +test('unknown and generic kinds show no tiles of another kind', async () => { + const { kindOf, viewOf } = await source; + for (const kind of ['maintenance', '', 'constructor', '__proto__', 'health']) { + const view = kindOf(kind); + assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []], kind); + assert.deepEqual([view.log, view.cancelLabel], ['Job log', 'Cancel job']); + assert.equal(view.pillVerb, null); + assert.equal(view.line(job({ kind })), ''); + assert.equal(view.kept(job({ kind })), ''); + } + const backup = job({ kind: 'backup', title: 'Backup' }); + assert.equal(kindOf('backup').startToast(backup), 'Backup started.'); + assert.equal(kindOf('backup').finishToast({ ...backup, state: 'finished' }), 'Backup finished.'); + assert.equal(kindOf('backup').finishToast({ ...backup, state: 'cancelled' }), 'Backup was cancelled.'); + assert.equal(kindOf('backup').finishToast({ ...backup, state: 'failed' }), 'Backup failed.'); + // A kind's tiles are only the groups its job counts: no zero tile for a group the server never reported. + const partial = viewOf(job({ counters: { added: 3, errors: 0 } })); + assert.deepEqual(partial.tiles, [['added', 'Added to Library', 'added'], ['errors', 'Errors', 'bad']]); + assert.deepEqual(partial.outcomes, [['added', 'Added to Library'], ['errors', 'Errors']]); + assert.deepEqual([...partial.openable], ['added']); + assert.equal(partial.log, 'Scan log'); +}); diff --git a/tests/js/workspace-activity-store.test.js b/tests/js/workspace-activity-store.test.js index 82b69da..fc8bfb6 100644 --- a/tests/js/workspace-activity-store.test.js +++ b/tests/js/workspace-activity-store.test.js @@ -289,8 +289,10 @@ test('summaries are validated and the copy follows the spec', async () => { const scan = parseJob(summary({ progress: 42, processed: 8240, total: 19622, counters: { added: 1, errors: 1 } })); assert.deepEqual(pillParts(scan), { label: 'Scanning · 42%', files: ' · 8,240 / 19,622', text: 'Scanning · 42% · 8,240 / 19,622' }); - assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning · 42%'); - assert.equal(pillParts({ ...scan, kind: 'thumbnails', title: 'Thumbnails' }).text, 'Thumbnails · 42% · 8,240 / 19,622'); + // D111: without a total the progress is indeterminate, so the pill shows no percentage. + assert.equal(pillParts({ ...scan, total: 0 }).text, 'Scanning'); + // A kind without a pill verb shows its title (D111: thumbnails have their own, see workspace-activity-kinds.test.js). + assert.equal(pillParts({ ...scan, kind: 'backup', title: 'Backup' }).text, 'Backup · 42% · 8,240 / 19,622'); assert.equal(finishToast({ ...scan, state: 'finished' }), 'Scan finished · 1 added · 1 error'); assert.equal(finishToast({ ...scan, state: 'cancelled' }), 'Scan cancelled · 1 added'); assert.equal(finishToast({ ...scan, state: 'failed' }), 'Scan failed · 1 added · 1 error'); @@ -332,7 +334,8 @@ test('image-tags jobs get their own tiles outcomes and pill verb', async () => { 'Add tags to images finished · 0 tagged · 0 errors'); }); -test('unknown kinds keep the generic view', async () => { +// D111 (S5): an unknown kind no longer borrows the scan's tiles (S1 review M6, S2 review M3); it has none. +test('unknown kinds get the generic view without the scan tiles', async () => { const { jobView } = await source; const scan = jobView('scan'); assert.deepEqual(scan.tiles, [['added', 'Added to Library', 'added'], ['known', 'Already known', ''], @@ -343,7 +346,7 @@ test('unknown kinds keep the generic view', async () => { assert.deepEqual([scan.log, scan.cancel], ['Scan log', 'Cancel scan']); for (const kind of ['maintenance', '', 'constructor', '__proto__']) { const view = jobView(kind); - assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [scan.tiles, scan.outcomes, [...scan.openable]]); + assert.deepEqual([view.tiles, view.outcomes, [...view.openable]], [[], [], []]); assert.deepEqual([view.log, view.cancel], ['Job log', 'Cancel job']); } }); diff --git a/tests/js/workspace-thumbnail-review.test.js b/tests/js/workspace-thumbnail-review.test.js new file mode 100644 index 0000000..eff49e5 --- /dev/null +++ b/tests/js/workspace-thumbnail-review.test.js @@ -0,0 +1,93 @@ +// System › thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (app.js 1433–1437, +// `thumbnail-review`) on the thumbnail job (S5 Task 12). Pure parts: the option values the job takes, the preview size +// check, the Image loading quality preselection (Q3), the D99 plan line and the start outcome. +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const reviewModule = import('../../web/js/workspace/thumbnail-review.mjs'); + +test('the concept options map to the job values in the concept order', async () => { + const { SCOPE_OPTIONS, QUALITY_OPTIONS, SIZE_LIMIT } = await reviewModule; + assert.deepEqual(SCOPE_OPTIONS, [['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]); + assert.deepEqual(QUALITY_OPTIONS, [['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]); + assert.deepEqual(SIZE_LIMIT, { min: 128, max: 2048, value: 512 }); +}); + +test('size outside 128–2048 reports validity: only whole pixels in range are a size', async () => { + const { previewSize } = await reviewModule; + assert.equal(previewSize('512'), 512); + assert.equal(previewSize('128'), 128); + assert.equal(previewSize('2048'), 2048); + for (const text of ['', ' ', '127', '2049', '4096', '512.5', '-512', 'abc', '1e3', '0x200']) { + assert.equal(previewSize(text), null, text); + } +}); + +test('Image loading quality preselects the dialog quality (Q3); anything else is Medium', async () => { + const { preselectedQuality } = await reviewModule; + assert.equal(preselectedQuality('low'), 'low'); + assert.equal(preselectedQuality('high'), 'high'); + assert.equal(preselectedQuality('medium'), 'medium'); + assert.equal(preselectedQuality(undefined), 'medium'); + assert.equal(preselectedQuality('original'), 'medium'); +}); + +test('the plan line counts what a missing-only run generates and what is already up to date (D99)', async () => { + const { planText } = await reviewModule; + assert.equal(planText({ mode: 'missing', targets: 12, fresh: 30, remaining: 0 }), '12 previews to generate · 30 already up to date.'); + assert.equal(planText({ mode: 'missing', targets: 1, fresh: 1, remaining: 0 }), '1 preview to generate · 1 already up to date.'); + assert.equal(planText({ mode: 'missing', targets: 0, fresh: 0, remaining: 0 }), '0 previews to generate · 0 already up to date.'); + assert.equal(planText({ mode: 'missing', targets: 10000, fresh: 5, remaining: 2500 }), + '10,000 previews to generate · 5 already up to date · 2,500 more on the next run.'); +}); + +test('Rebuild all says every preview is regenerated (D99)', async () => { + const { planText } = await reviewModule; + assert.equal(planText({ mode: 'rebuild', targets: 42, fresh: 30, remaining: 0 }), 'Every preview is regenerated: 42 files.'); + assert.equal(planText({ mode: 'rebuild', targets: 1, fresh: 1, remaining: 0 }), 'Every preview is regenerated: 1 file.'); + assert.equal(planText({ mode: 'rebuild', targets: 10000, fresh: 9000, remaining: 3 }), + 'Every preview is regenerated: 10,000 files · 3 more on the next run.'); +}); + +test('an answer without counts is a failed plan, never zeros', async () => { + const { planText, PLAN_FAILED } = await reviewModule; + assert.equal(planText(null), PLAN_FAILED); + assert.equal(planText({ mode: 'missing', targets: 2 }), PLAN_FAILED); + assert.equal(planText({ mode: 'missing', targets: -1, fresh: 0, remaining: 0 }), PLAN_FAILED); + assert.equal(planText({ mode: 'other', targets: 1, fresh: 0, remaining: 0 }), PLAN_FAILED); +}); + +const scopeOf = () => ({ signal: new AbortController().signal }); + +test('Start starts the thumbnail job with the chosen options and opens it in Activity', async () => { + const { startThumbnails } = await reviewModule; + const calls = []; + const navigated = []; + const store = { start: async (kind, body) => { calls.push([kind, body]); return { job: { id: 'a'.repeat(32) } }; } }; + await startThumbnails({ store, scope: scopeOf(), options: { mode: 'rebuild', size: 1024, quality: 'high' }, + navigate: hash => navigated.push(hash), notify: () => assert.fail('no toast of its own') }); + assert.deepEqual(calls, [['thumbnails', { mode: 'rebuild', size: 1024, quality: 'high' }]]); + assert.deepEqual(navigated, [`#/activity/${'a'.repeat(32)}`]); +}); + +test('a refused start shows the busy link toast and stays', async () => { + const { startThumbnails } = await reviewModule; + const toasts = []; + const store = { start: async () => ({ busy: { id: 'b'.repeat(32), title: 'Scan for metadata' } }) }; + await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' }, + navigate: () => assert.fail('stays'), notify: message => toasts.push(message) }); + assert.deepEqual(toasts, [['Another job is running: ', { text: 'Scan for metadata', href: `#/activity/${'b'.repeat(32)}` }, '.']]); +}); + +test('a failed start says so; a start after the section left says nothing', async () => { + const { startThumbnails, START_FAILED } = await reviewModule; + const toasts = []; + const store = { start: async () => { throw new Error('offline'); } }; + await startThumbnails({ store, scope: scopeOf(), options: { mode: 'missing', size: 512, quality: 'medium' }, + navigate: () => assert.fail('stays'), notify: message => toasts.push(message) }); + assert.deepEqual(toasts, [START_FAILED]); + const gone = new AbortController(); + gone.abort(); + await startThumbnails({ store, scope: { signal: gone.signal }, options: { mode: 'missing', size: 512, quality: 'medium' }, + navigate: () => assert.fail('stays'), notify: () => assert.fail('silent') }); +}); diff --git a/tests/test_api_images.py b/tests/test_api_images.py index 8c0361c..afe37cc 100644 --- a/tests/test_api_images.py +++ b/tests/test_api_images.py @@ -413,11 +413,11 @@ class TestThumbnailProgressStream(ThumbnailTestCase): lock = threading.Lock() original = images_module._write_image_thumbnail - def recording(src, dst, size): + def recording(src, dst, size, *args): with lock: seen.add(threading.current_thread().name) time.sleep(0.05) - return original(src, dst, size) + return original(src, dst, size, *args) images_module._write_image_thumbnail = recording self.addCleanup(setattr, images_module, "_write_image_thumbnail", original) @@ -1939,7 +1939,11 @@ class TestGalleryRescan(ImageAPITestCase): class TestOutputScanHelpers(ImageAPITestCase): - """Stable ids for filesystem entries, no symlink reads, stop on disconnect.""" + """Stable ids for filesystem entries and no symlink reads. + + The thumbnail stream no longer stops when its client goes away: see + ``tests/test_thumbnail_jobs.py`` (the work is a job that outlives it). + """ def test_output_entry_id_is_a_stable_digest_of_the_path(self): import hashlib @@ -1964,28 +1968,6 @@ class TestOutputScanHelpers(ImageAPITestCase): names = sorted(p.name for p in images_module._iter_media_files(output, 100)) self.assertEqual(names, ["real.png"]) - async def test_thumbnail_stream_stops_when_the_client_disconnects(self): - generated = [] - - def fake_generate(src, dst, size, is_video): - generated.append(src) - return { - "action": "generated", - "file": src.name, - "dir": src.parent.name, - "type": "image", - } - - self.api._generate_one = fake_generate - - class GoneResponse: - async def write(self, data): - raise ConnectionResetError("client went away") - - targets = [(Path(f"/x/{i}.png"), Path(f"/x/t{i}.jpg"), False) for i in range(3)] - await self.api._stream_thumbnails(GoneResponse(), targets, (256, 256)) - self.assertEqual(generated, []) - class TestThumbnailStartupStatus(ThumbnailTestCase): """Startup counts and generation cover the same configured collection.""" diff --git a/tests/test_jobs.py b/tests/test_jobs.py index 6848fa8..9176ba6 100644 --- a/tests/test_jobs.py +++ b/tests/test_jobs.py @@ -101,6 +101,35 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase): self.assertEqual(await _collect(job), [{"type": "complete"}]) + async def test_lead_comes_first_and_nothing_published_meanwhile_is_lost(self): + """S5 I1: the ``job`` frame leads, with the subscription already in place.""" + job = Job("thumbnails", "Thumbnail generation") + job.publish({"type": "status"}) + lead = {"type": "job"} + seen = [] + async with contextlib.aclosing(job.events(lead=lead)) as events: + async for payload in events: + seen.append(payload) + if payload is lead: + # Published while the consumer is still sending the lead. + job.publish({"type": "progress"}) + job.publish({"type": "complete"}) + job.finish() + self.assertEqual( + seen, + [lead, {"type": "status"}, {"type": "progress"}, {"type": "complete"}], + ) + + async def test_lead_on_a_finished_job_then_its_last_event(self): + job = Job("thumbnails", "Thumbnail generation") + job.publish({"type": "complete"}) + job.finish() + seen = [] + async with contextlib.aclosing(job.events(lead={"type": "job"})) as events: + async for payload in events: + seen.append(payload) + self.assertEqual(seen, [{"type": "job"}, {"type": "complete"}]) + async def test_departed_subscriber_is_forgotten(self): job = Job("scan", "Scan") job.publish({"type": "progress", "progress": 1}) @@ -151,6 +180,30 @@ class TestJobFanOut(unittest.IsolatedAsyncioTestCase): class TestJobRegistry(unittest.IsolatedAsyncioTestCase): + async def test_a_compatible_kind_never_blocks(self): + """S5 M7: a caller may name running kinds that cannot conflict with it.""" + registry = JobRegistry() + gate = asyncio.Event() + thumbs = registry.start( + "thumbnails", + "Thumbnail generation", + admission="guarded", + runner=_gated_runner(gate), + ) + self.assertIs(registry.blocker("probe", admission="records"), thumbs) + self.assertIsNone( + registry.blocker("probe", admission="records", compatible=("thumbnails",)) + ) + backup = registry.start( + "backup", "Backup", admission="guarded", runner=_gated_runner(gate) + ) + self.assertIs( + registry.blocker("probe", admission="records", compatible=("thumbnails",)), + backup, + ) + gate.set() + await asyncio.wait_for(asyncio.gather(thumbs.task, backup.task), 2) + async def test_start_lists_and_finishes_a_job(self): registry = JobRegistry() gate = asyncio.Event() @@ -421,6 +474,28 @@ class TestJobOutcomesAndLog(unittest.TestCase): SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors") ) + def test_record_counted_counts_exactly_lists_capped_and_logs_nothing(self): + job = Job("thumbnails", "Thumbnail generation", groups=("skipped",)) + rows = [ + {"path": os.path.abspath(f"out{os.sep}f{i}.png"), "root": "output"} + for i in range(MAX_OUTCOME_ROWS + 5) + ] + job.record_counted("skipped", 10_000, rows) + job.record_counted("skipped", 1, [{"path": "late.png", "root_index": True}]) + self.assertEqual(job.counters["skipped"], 10_001) + self.assertEqual(len(job.outcomes["skipped"]), MAX_OUTCOME_ROWS) + self.assertEqual(job.outcomes["skipped"][0]["path"], "f0.png") + self.assertIsNone(job.outcomes["skipped"][0]["root_index"]) + self.assertEqual(job.log, []) + with self.assertRaises(ValueError): + job.record_counted("bogus", 1) + + def test_thumbnail_job_groups(self): + self.assertEqual( + jobs_module.THUMBNAIL_GROUP_NAMES, ("generated", "skipped", "errors") + ) + self.assertEqual(jobs_module.THUMBNAILS_JOB_KIND, "thumbnails") + def test_image_tag_job_kind_and_groups(self): # #249 D1: bulk Add tags / Auto-tag, counter-tile order. self.assertEqual(getattr(jobs_module, "IMAGE_TAG_JOB_KIND", None), "image-tags") diff --git a/tests/test_review.py b/tests/test_review.py index 70776c0..ad41001 100644 --- a/tests/test_review.py +++ b/tests/test_review.py @@ -859,7 +859,9 @@ class TestMonitorRecording(ReviewTestCase): While a ``records`` or ``guarded`` job runs, the monitor's recording records nothing and answers ``deferred`` (its fallback link stands and a later scan reads the ambiguous file again); a ``read`` job does not - refuse it, and it records once the blocking job has ended. + refuse it, and it records once the blocking job has ended. Thumbnail + work (the job and the gallery's holds) is ``guarded`` but writes only + previews, never records, so it does not defer the recording (S5 M7). """ self._require_module() from py.api.jobs import RESTORE_JOB_KIND, THUMBNAILS_JOB_KIND @@ -870,7 +872,7 @@ class TestMonitorRecording(ReviewTestCase): for kind, admission in ( (RESTORE_JOB_KIND, "records"), - (THUMBNAILS_JOB_KIND, "guarded"), + ("backup", "guarded"), ): with self.subTest(admission=admission): async with self.api._jobs.hold( @@ -888,7 +890,43 @@ class TestMonitorRecording(ReviewTestCase): ) async with self.api._jobs.hold("probe-read", "Read probe", admission="read"): + async with self.api._jobs.hold( + THUMBNAILS_JOB_KIND, + "Thumbnail clear (gallery page)", + admission="guarded", + ): + outcome = await asyncio.to_thread( + self.api.record_review_file, str(path) + ) + self.assertEqual(outcome, "recorded") + self.assertEqual(len(self._items()), 1) + + async def test_a_running_thumbnail_job_does_not_defer_monitor_recording(self): + """S5 M7: a long rebuild never holds Needs review back (no records).""" + self._require_module() + from py.api.jobs import THUMBNAILS_JOB_KIND + + self._roots() + path = self.output_dir / "beside.png" + write_png(path, prompt=unlinkable_graph("cat", "dog")) + gate = asyncio.Event() + + async def runner(job): + await gate.wait() + + job = self.api._jobs.start( + THUMBNAILS_JOB_KIND, + "Thumbnail generation", + admission="guarded", + cancellable=True, + scope={}, + runner=runner, + ) + try: outcome = await asyncio.to_thread(self.api.record_review_file, str(path)) + finally: + gate.set() + await asyncio.wait_for(job.task, 5) self.assertEqual(outcome, "recorded") self.assertEqual(len(self._items()), 1) diff --git a/tests/test_thumbnail_jobs.py b/tests/test_thumbnail_jobs.py new file mode 100644 index 0000000..9b9d5ba --- /dev/null +++ b/tests/test_thumbnail_jobs.py @@ -0,0 +1,846 @@ +"""Thumbnail generation as an Activity job (#249 System S5, Task 12). + +``GET /prompt_manager/images/thumbnails/plan`` counts what a run would do and +writes nothing. ``POST /prompt_manager/jobs/thumbnails`` starts the +``thumbnails`` job (admission ``guarded``, cancellable at a file boundary). +The legacy gallery stream ``POST /images/generate-thumbnails/progress`` starts +or attaches to that job and streams its unchanged events from it, so a client +that goes away no longer stops the work. + +Ordering is driven by threading/asyncio events, never by timers. +""" + +import asyncio +import hashlib +import os +import sys +import threading +import unittest +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +try: + from tests.test_api_images import ThumbnailTestCase, make_png, parse_sse +except ImportError: # discovered with tests/ as the top-level directory + from test_api_images import ThumbnailTestCase, make_png, parse_sse + +from PIL import Image # noqa: E402 + +import py.api.images as images_module # noqa: E402 +from py.api.thumbnail_jobs import thumbnail_options # noqa: E402 +from py.config import PromptManagerConfig # noqa: E402 + +PLAN_URL = "/prompt_manager/images/thumbnails/plan" +JOB_URL = "/prompt_manager/jobs/thumbnails" + + +def _tree_state(root): + """{relative path: (size, mtime_ns, sha256)} of every file under ``root``.""" + state = {} + for folder, _dirs, files in os.walk(root): + for name in files: + path = Path(folder) / name + data = path.read_bytes() + stat = path.stat() + state[path.relative_to(root).as_posix()] = ( + stat.st_size, + stat.st_mtime_ns, + hashlib.sha256(data).hexdigest(), + ) + return state + + +class ThumbnailJobCase(ThumbnailTestCase): + """Helpers: start, gate and wait for thumbnail jobs.""" + + async def _start(self, **body): + resp = await self.client.post(JOB_URL, json=body) + return resp.status, await resp.json() + + async def _finished(self, job_id): + job = self.api._jobs.get(job_id) + await asyncio.wait_for(job.task, 10) + return job + + def _gate_generation(self): + """Block every file in the worker until released; returns (entered, release).""" + entered = threading.Event() + release = threading.Event() + original = self.api._generate_one + + def gated(*args, **kwargs): + entered.set() + release.wait(10) + return original(*args, **kwargs) + + self.api._generate_one = gated + self.addCleanup(release.set) + return entered, release + + async def _wait(self, event): + loop = asyncio.get_running_loop() + self.assertTrue(await loop.run_in_executor(None, event.wait, 10)) + + @staticmethod + def _watch_subscribers(job): + """(subscribed, left): threading events for a stream joining or leaving.""" + subscribed, left = threading.Event(), threading.Event() + + class Watched(set): + def add(self, item): + super().add(item) + subscribed.set() + + def discard(self, item): + super().discard(item) + left.set() + + job._subscribers = Watched(job._subscribers) + return subscribed, left + + def _status_lines(self): + """Every status line any job is given, in order.""" + from py.api.jobs import Job + + lines = [] + original = Job.update + + def update(job, **kwargs): + if kwargs.get("status_line") is not None: + lines.append(kwargs["status_line"]) + return original(job, **kwargs) + + patcher = patch.object(Job, "update", update) + patcher.start() + self.addCleanup(patcher.stop) + return lines + + def _one_worker(self): + original = PromptManagerConfig.WORKER_THREADS + PromptManagerConfig.WORKER_THREADS = 1 # batches of 4 + self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original) + + def _run_scan(self): + gate = asyncio.Event() + + async def runner(job): + await gate.wait() + + job = self.api._jobs.start( + "scan", "Scan for metadata", admission="records", scope={}, runner=runner + ) + + async def release(): + gate.set() + await asyncio.wait_for(job.task, 5) + + self.addAsyncCleanup(release) + return job + + +class TestThumbnailPlan(ThumbnailJobCase): + async def test_plan_is_read_only_and_counts_cap(self): + for name in ("a", "b", "c"): + make_png(self.output_dir / f"{name}.png", size=(64, 64)) + fresh = make_png(self._thumb_path("a")) + source_time = (self.output_dir / "a.png").stat().st_mtime + os.utime(fresh, (source_time + 10, source_time + 10)) + before = _tree_state(self.comfy_dir) + + with patch.object(images_module, "MAX_THUMBNAIL_FILES", 1): + missing = await self.client.get( + PLAN_URL, params={"mode": "missing", "size": "512", "quality": "high"} + ) + rebuild = await self.client.get( + PLAN_URL, params={"mode": "rebuild", "size": "256", "quality": "low"} + ) + + self.assertEqual(missing.status, 200) + self.assertEqual( + await missing.json(), + { + "success": True, + "targets": 1, + "fresh": 1, + "remaining": 1, + "roots": ["output"], + "mode": "missing", + "size": 512, + "quality": "high", + }, + ) + data = await rebuild.json() + # Rebuild regenerates fresh previews too: all three are work, one fits the cap. + self.assertEqual((data["targets"], data["fresh"], data["remaining"]), (1, 1, 2)) + self.assertEqual( + (data["mode"], data["size"], data["quality"]), ("rebuild", 256, "low") + ) + self.assertEqual(_tree_state(self.comfy_dir), before) + self.assertEqual(self.api._jobs.list(), []) + + async def test_plan_defaults_to_missing_previews_at_the_concept_size(self): + make_png(self.output_dir / "a.png") + data = await (await self.client.get(PLAN_URL)).json() + self.assertEqual( + (data["mode"], data["size"], data["quality"], data["targets"]), + ("missing", 512, "medium", 1), + ) + + async def test_size_bounds_validated(self): + make_png(self.output_dir / "a.png") + for params in ( + {"size": "127"}, + {"size": "2049"}, + {"size": "512.5"}, + {"size": "big"}, + {"mode": "all"}, + {"quality": "ultra"}, + ): + with self.subTest(params=params): + resp = await self.client.get(PLAN_URL, params=params) + body = await resp.json() + self.assertEqual(resp.status, 400) + self.assertEqual( + (body["success"], body["code"]), (False, "invalid-request") + ) + for body in ( + {"size": 127}, + {"size": 2049}, + {"size": "512"}, + {"size": 512.0}, + {"size": True}, + {"mode": "rebuild-all"}, + {"quality": None}, + [], + ): + with self.subTest(body=body): + resp = await self.client.post(JOB_URL, json=body) + self.assertEqual(resp.status, 400) + self.assertEqual((await resp.json())["code"], "invalid-request") + for edge in (128, 2048): + with self.subTest(edge=edge): + resp = await self.client.get(PLAN_URL, params={"size": str(edge)}) + self.assertEqual(resp.status, 200) + self.assertEqual(self.api._jobs.list(), []) + self.assertFalse(self._thumb_path("a").exists()) + + +class TestThumbnailJob(ThumbnailJobCase): + async def test_job_generates_missing_previews_at_the_chosen_edge(self): + make_png(self.output_dir / "a.png", size=(800, 400)) + make_png(self.output_dir / "sub" / "b.png", size=(64, 64)) + (self.output_dir / "bad.png").write_bytes(b"\x89PNG not really") + + status, body = await self._start(mode="missing", size=200, quality="medium") + + self.assertEqual(status, 202) + summary = body["job"] + self.assertEqual( + (summary["kind"], summary["title"], summary["admission"]), + ("thumbnails", "Thumbnail generation", "guarded"), + ) + self.assertTrue(summary["cancellable"]) + self.assertEqual(set(summary["counters"]), {"generated", "skipped", "errors"}) + self.assertEqual( + summary["scope"], {"mode": "missing", "size": 200, "quality": "medium"} + ) + job = await self._finished(summary["id"]) + self.assertEqual(job.state, "finished") + self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 1}) + self.assertEqual((job.processed, job.total, job.progress), (3, 3, 100)) + self.assertEqual(job.scope["output_folders"], 1) + with Image.open(self._thumb_path("a")) as thumb: + self.assertEqual(thumb.size, (200, 100)) + rows = {row["path"]: row for row in job.outcomes["generated"]} + self.assertEqual(set(rows), {"a.png", "sub/b.png"}) + self.assertEqual( + (rows["a.png"]["root"], rows["a.png"]["root_index"]), ("output", 0) + ) + self.assertEqual(job.outcomes["errors"][0]["path"], "bad.png") + self.assertTrue(any("Thumbnails finished" in line for line in job.log)) + # Rows and log lines never carry the server's absolute layout. + self.assertNotIn(str(self.comfy_dir), repr(job.detail())) + + async def test_fresh_previews_are_counted_as_skipped(self): + make_png(self.output_dir / "a.png") + make_png(self.output_dir / "b.png") + fresh = make_png(self._thumb_path("a")) + source_time = (self.output_dir / "a.png").stat().st_mtime + os.utime(fresh, (source_time + 10, source_time + 10)) + + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + + self.assertEqual(job.counters, {"generated": 1, "skipped": 1, "errors": 0}) + self.assertEqual([r["path"] for r in job.outcomes["skipped"]], ["a.png"]) + + async def test_rebuild_never_touches_originals(self): + for name in ("a", "b"): + make_png(self.output_dir / f"{name}.png", size=(400, 400)) + _, body = await self._start(size=128) + await self._finished(body["job"]["id"]) + originals = { + name: hashlib.sha256( + (self.output_dir / f"{name}.png").read_bytes() + ).hexdigest() + for name in ("a", "b") + } + + _, body = await self._start(mode="rebuild", size=256, quality="high") + job = await self._finished(body["job"]["id"]) + + self.assertEqual(job.counters, {"generated": 2, "skipped": 0, "errors": 0}) + for name in ("a", "b"): + digest = hashlib.sha256( + (self.output_dir / f"{name}.png").read_bytes() + ).hexdigest() + self.assertEqual(digest, originals[name], name) + with Image.open(self._thumb_path(name)) as thumb: + self.assertEqual(thumb.size, (256, 256)) + + async def test_rebuild_keeps_old_preview_when_generation_fails(self): + source = make_png(self.output_dir / "a.png", size=(64, 64)) + _, body = await self._start() + await self._finished(body["job"]["id"]) + preview = self._thumb_path("a").read_bytes() + source.write_bytes(b"\x89PNG broken now") + later = self._thumb_path("a").stat().st_mtime + 10 + os.utime(self._thumb_path("a"), (later, later)) # still fresh + + _, body = await self._start(mode="rebuild") + job = await self._finished(body["job"]["id"]) + + self.assertEqual(job.counters["errors"], 1) + self.assertEqual(self._thumb_path("a").read_bytes(), preview) + leftovers = [p.name for p in self._thumb_path("a").parent.iterdir()] + self.assertEqual(leftovers, ["a_thumb.png"]) # no temp file left + + async def test_rebuild_keeps_an_old_video_preview_instead_of_a_placeholder(self): + clip = self.output_dir / "clip.mp4" + clip.write_bytes(b"\x00") + preview = make_png(self.output_dir / "thumbnails" / "clip_thumb.jpg") + old = preview.read_bytes() + with ( + patch.dict(sys.modules, {"cv2": None}), + patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")), + ): + _, body = await self._start(mode="rebuild") + job = await self._finished(body["job"]["id"]) + self.assertEqual(job.counters["errors"], 1) + self.assertEqual(preview.read_bytes(), old) + + async def test_quality_sets_the_encoder_not_the_edge(self): + noise = bytes( + (i * 7919 + (i >> 3) * 104729) % 251 for i in range(600 * 600 * 3) + ) + with Image.frombytes("RGB", (600, 600), noise) as img: + img.save(self.output_dir / "photo.jpg", "JPEG", quality=95) + sizes = {} + for quality in ("low", "high"): + _, body = await self._start(mode="rebuild", size=300, quality=quality) + await self._finished(body["job"]["id"]) + thumb = self._thumb_path("photo", ".jpg") + with Image.open(thumb) as image: + self.assertEqual(image.size, (300, 300)) + sizes[quality] = thumb.stat().st_size + self.assertLess(sizes["low"], sizes["high"]) + + async def test_no_output_folder_fails_the_job(self): + self.api._get_all_output_dirs = lambda: [] + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + self.assertEqual(job.state, "failed") + self.assertEqual(job.status_line, "ComfyUI output directory not found") + + async def test_cancel_stops_at_file_boundary_keeping_written_previews(self): + for i in range(6): + make_png(self.output_dir / f"p{i}.png") + original = PromptManagerConfig.WORKER_THREADS + PromptManagerConfig.WORKER_THREADS = 1 + self.addCleanup(setattr, PromptManagerConfig, "WORKER_THREADS", original) + written = [] + real_write = images_module._write_image_thumbnail + job_ids = [] + + def counting(src, dst, *args, **kwargs): + real_write(src, dst, *args, **kwargs) + written.append(dst) + if len(written) == 2: + self.assertTrue(self.api._jobs.cancel(job_ids[0])) + + with patch.object(images_module, "_write_image_thumbnail", counting): + _, body = await self._start() + job_ids.append(body["job"]["id"]) + job = await self._finished(body["job"]["id"]) + + self.assertEqual(job.state, "cancelled") + self.assertEqual(job.counters["generated"], 2) + self.assertEqual(len(written), 2) + for dst in written: + self.assertTrue(dst.is_file()) + made = sorted((self.output_dir / "thumbnails").glob("*_thumb.png")) + self.assertEqual(made, sorted(written)) + self.assertTrue(any("Thumbnails cancelled" in line for line in job.log)) + + async def test_a_second_thumbnail_job_is_refused_naming_the_first(self): + make_png(self.output_dir / "a.png") + entered, release = self._gate_generation() + _, first = await self._start() + await self._wait(entered) + + status, body = await self._start(mode="rebuild") + + self.assertEqual((status, body["code"]), (409, "busy")) + self.assertEqual(body["running"]["id"], first["job"]["id"]) + release.set() + await self._finished(first["job"]["id"]) + + async def test_thumbnail_job_refused_while_scan_runs(self): + make_png(self.output_dir / "a.png") + self._run_scan() + + status, body = await self._start() + + self.assertEqual((status, body["code"]), (409, "busy")) + self.assertEqual(body["running"]["title"], "Scan for metadata") + self.assertFalse(self._thumb_path("a").exists()) + + async def test_scan_refused_while_thumbnail_job_runs(self): + make_png(self.output_dir / "a.png") + entered, release = self._gate_generation() + _, started = await self._start() + await self._wait(entered) + + resp = await self.client.post("/prompt_manager/jobs/scan") + body = await resp.json() + + self.assertEqual((resp.status, body["code"]), (409, "busy")) + self.assertEqual(body["running"]["title"], "Thumbnail generation") + release.set() + await self._finished(started["job"]["id"]) + + async def test_clear_refused_while_thumbnail_job_runs(self): + make_png(self.output_dir / "a.png") + kept = make_png(self.output_dir / "thumbnails" / "old_thumb.png") + entered, release = self._gate_generation() + _, started = await self._start() + await self._wait(entered) + + resp = await self.client.post("/prompt_manager/images/clear-thumbnails") + + self.assertEqual(resp.status, 409) + self.assertEqual( + await resp.json(), + {"success": False, "error": "Thumbnail generation already running"}, + ) + self.assertTrue(kept.exists()) + release.set() + await self._finished(started["job"]["id"]) + + async def test_every_exit_sets_a_truthful_final_status_line(self): + """I3: finished, nothing to do, cancelled and failed each say what happened.""" + lines = self._status_lines() + make_png(self.output_dir / "a.png") + (self.output_dir / "bad.png").write_bytes(b"\x89PNG nope") + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + self.assertEqual(job.status_line, "1 preview written · 1 error") + + # Every preview fresh: the job finishes with nothing to do. + source_time = (self.output_dir / "a.png").stat().st_mtime + bad_thumb = make_png(self._thumb_path("bad")) + for thumb in (self._thumb_path("a"), bad_thumb): + os.utime(thumb, (source_time + 10, source_time + 10)) + os.utime(self.output_dir / "bad.png", (source_time, source_time)) + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + self.assertEqual((job.state, job.total), ("finished", 0)) + self.assertEqual( + job.status_line, "No previews to generate · 2 already up to date" + ) + self.assertNotIn("Processing file 0 of 0", lines) + + async def test_an_empty_library_says_there_is_nothing_to_preview(self): + lines = self._status_lines() + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + self.assertEqual( + (job.state, job.status_line), ("finished", "No images or videos to preview") + ) + self.assertNotIn("Processing file 0 of 0", lines) + + async def test_a_cancelled_job_says_what_it_kept(self): + for i in range(6): + make_png(self.output_dir / f"p{i}.png") + self._one_worker() + real_write = images_module._write_image_thumbnail + job_ids = [] + written = [] + + def counting(src, dst, *args, **kwargs): + real_write(src, dst, *args, **kwargs) + written.append(dst) + if len(written) == 1: + self.api._jobs.cancel(job_ids[0]) + + with patch.object(images_module, "_write_image_thumbnail", counting): + _, body = await self._start() + job_ids.append(body["job"]["id"]) + job = await self._finished(body["job"]["id"]) + self.assertEqual(job.state, "cancelled") + self.assertEqual(job.status_line, "Cancelled · 1 preview written") + + async def test_an_internal_error_says_the_job_failed(self): + def broken(*_args, **_kwargs): + raise RuntimeError("boom") + + self.api._plan_gallery_thumbnails = broken + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + self.assertEqual( + (job.state, job.status_line), + ("failed", "Thumbnail generation stopped on an internal error."), + ) + + async def test_cancel_during_planning_stops_the_walk_and_starts_nothing(self): + """M2: the walk checks Cancel per file; no ``start`` frame is published.""" + for i in range(10): + make_png(self.output_dir / f"p{i}.png") + entered, release = threading.Event(), threading.Event() + self.addCleanup(release.set) + checked = [] + real_fresh = images_module._thumbnail_is_fresh + + def gated_fresh(src, dst): + checked.append(src.name) + if len(checked) == 1: + entered.set() + release.wait(10) + return real_fresh(src, dst) + + with patch.object(images_module, "_thumbnail_is_fresh", gated_fresh): + stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL)) + await self._wait(entered) + job = self.api._jobs.active("thumbnails") + self.assertTrue(self.api._jobs.cancel(job.id)) + release.set() + events = parse_sse(await (await stream).text()) + await asyncio.wait_for(job.task, 10) + + self.assertEqual(checked, ["p0.png"]) # the walk stopped at the next file + self.assertEqual([name for name, _ in events], ["job", "status", "error"]) + self.assertEqual(job.state, "cancelled") + self.assertEqual(job.status_line, "Cancelled before any preview was written") + self.assertEqual(job.total, 0) + self.assertFalse((self.output_dir / "thumbnails").exists()) + + async def test_a_locked_preview_is_a_readable_coded_error(self): + """M6: Windows refuses to replace a preview another process has open.""" + make_png(self.output_dir / "a.png") + locked = PermissionError(13, "Access is denied") + locked.winerror = 5 + + def refuse(*_args, **_kwargs): + raise locked + + with patch.object(images_module, "_write_image_thumbnail", refuse): + _, body = await self._start(mode="rebuild") + job = await self._finished(body["job"]["id"]) + + self.assertEqual(job.counters["errors"], 1) + [row] = job.outcomes["errors"] + self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE) + self.assertNotIn("WinError", repr(job.detail())) + self.assertTrue(any(images_module.PREVIEW_IN_USE in line for line in job.log)) + + async def test_a_locked_video_preview_is_the_same_coded_error(self): + clip = self.output_dir / "clip.mp4" + clip.write_bytes(b"\x00") + locked = PermissionError(13, "Access is denied") + locked.winerror = 32 + + def refuse(*_args, **_kwargs): + raise locked + + with ( + patch.dict(sys.modules, {"cv2": None}), + patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")), + patch.object(images_module, "_atomic_save", refuse), + ): + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + [row] = job.outcomes["errors"] + self.assertEqual(row["detail"], images_module.PREVIEW_IN_USE) + + async def test_a_permission_error_off_windows_names_the_permission(self): + make_png(self.output_dir / "a.png") + + def refuse(*_args, **_kwargs): + raise PermissionError(13, "Permission denied") + + with patch.object(images_module, "_write_image_thumbnail", refuse): + _, body = await self._start() + job = await self._finished(body["job"]["id"]) + [row] = job.outcomes["errors"] + self.assertEqual(row["detail"], images_module.PREVIEW_NOT_WRITABLE) + + async def test_ffmpeg_fallback_keeps_quality_and_aspect_ratio(self): + """M1: without OpenCV, ffmpeg keeps the chosen quality and the aspect ratio.""" + (self.output_dir / "clip.mp4").write_bytes(b"\x00") + commands = [] + + def fake_ffmpeg(cmd, **_kwargs): + commands.append(cmd) + Path(cmd[-1]).write_bytes(b"jpeg") + return type("Done", (), {"returncode": 0, "stderr": ""})() + + for quality, qscale in (("high", "2"), ("medium", "4"), ("low", "8")): + with ( + patch.dict(sys.modules, {"cv2": None}), + patch("subprocess.run", side_effect=fake_ffmpeg), + ): + _, body = await self._start(mode="rebuild", size=640, quality=quality) + job = await self._finished(body["job"]["id"]) + self.assertEqual(job.counters["generated"], 1, quality) + cmd = commands[-1] + self.assertNotIn("-s", cmd) + self.assertEqual( + cmd[cmd.index("-vf") + 1], + "scale=640:640:force_original_aspect_ratio=decrease", + ) + self.assertEqual(cmd[cmd.index("-q:v") + 1], qscale, quality) + + async def test_the_placeholder_uses_the_chosen_quality(self): + (self.output_dir / "clip.mp4").write_bytes(b"\x00") + saved = [] + real_save = images_module._atomic_save + + def recording(img, dst, fmt=None, **kwargs): + saved.append(kwargs) + return real_save(img, dst, fmt, **kwargs) + + with ( + patch.dict(sys.modules, {"cv2": None}), + patch("subprocess.run", side_effect=FileNotFoundError("no ffmpeg")), + patch.object(images_module, "_atomic_save", recording), + ): + _, body = await self._start(quality="low") + await self._finished(body["job"]["id"]) + self.assertEqual(saved, [images_module.THUMBNAIL_ENCODINGS["low"]]) + + +# The legacy stream's frames and their keys, as the gallery page reads them. +LEGACY_KEYS = { + "job": {"id", "started"}, + "status": {"phase", "message"}, + "start": { + "total_images", + "skipped", + "remaining", + "phase", + "image_count", + "video_count", + "message", + }, + "progress": { + "processed", + "total_images", + "generated", + "skipped", + "error_count", + "percentage", + "rate", + "eta", + "elapsed", + "current_file", + "file_type", + "action", + }, + "complete": { + "count", + "skipped", + "total_images", + "remaining", + "errors", + "error_count", + "elapsed_time", + "processing_rate", + "message", + }, +} + + +class TestLegacyStreamOnTheJob(ThumbnailJobCase): + async def test_legacy_progress_events_unchanged_on_success(self): + make_png(self.output_dir / "a.png", size=(400, 400)) + make_png(self.output_dir / "b.png") + + resp = await self.client.post(f"{self.PROGRESS_URL}?quality=low") + events = parse_sse(await resp.text()) + + self.assertEqual( + [name for name, _ in events], + ["job", "status", "start", "progress", "progress", "complete"], + ) + for name, data in events: + self.assertEqual(set(data), LEGACY_KEYS[name], name) + complete = events[-1][1] + self.assertEqual((complete["count"], complete["total_images"]), (2, 2)) + self.assertEqual( + events[1][1], + {"phase": "scanning", "message": "Scanning for images and videos..."}, + ) + with Image.open(self._thumb_path("a")) as thumb: + self.assertEqual(max(thumb.size), 150) # the legacy preset still sizes + [job] = self.api._jobs.list() + self.assertEqual( + (job.kind, job.title, job.state, job.cancellable), + ("thumbnails", "Thumbnail generation", "finished", True), + ) + self.assertEqual(job.counters["generated"], 2) + # I1: the first frame names the job this stream started, for its Cancel. + self.assertEqual(events[0][1], {"id": job.id, "started": True}) + + async def test_legacy_file_errors_still_stream(self): + make_png(self.output_dir / "good.png") + (self.output_dir / "bad.png").write_bytes(b"\x89PNG nope") + + resp = await self.client.post(self.PROGRESS_URL) + events = parse_sse(await resp.text()) + + self.assertEqual( + [d for e, d in events if e == "file_error"], + [{"file": "bad.png", "error": "Not a valid image file"}], + ) + + async def test_thumbnail_job_survives_client_disconnect(self): + """I2: the stream sees its client go after batch 1; every batch still runs. + + One worker thread makes batches of 4, so 9 files take three batches. The + first file waits until the client is gone; the second batch waits until + the stream has seen it go (its subscription dropped) and records whether + the job was still running then. A stream that stopped the work on + disconnect would leave 4 previews; the job writes all 9. + """ + for i in range(9): + make_png(self.output_dir / f"p{i}.png") + self._one_worker() + first, closed = threading.Event(), threading.Event() + state = {"calls": 0, "left": None, "running_when_left": None} + original = self.api._generate_one + + def gated(*args, **kwargs): + state["calls"] += 1 + if state["calls"] == 1: + first.set() + closed.wait(10) + elif state["calls"] == 5: # the second batch + state["running_when_left"] = state["left"].wait(10) + return original(*args, **kwargs) + + self.api._generate_one = gated + self.addCleanup(closed.set) + + resp = await self.client.post(self.PROGRESS_URL) + self.assertEqual(resp.status, 200) + await self._wait(first) + job = self.api._jobs.active("thumbnails") + _subscribed, state["left"] = self._watch_subscribers(job) + resp.close() # the gallery tab went away + closed.set() + await self._wait(state["left"]) # the server saw the client go + self.assertEqual(job.subscriber_count, 0) + await asyncio.wait_for(job.task, 10) + + self.assertEqual(job.counters["generated"], 9) + self.assertEqual(job.state, "finished") + self.assertTrue(state["running_when_left"]) + for i in range(9): + self.assertTrue(self._thumb_path(f"p{i}").is_file()) + + async def test_legacy_stream_attaches_to_a_running_job(self): + make_png(self.output_dir / "a.png") + entered, release = self._gate_generation() + _, started = await self._start(mode="missing", size=256) + await self._wait(entered) + + job = self.api._jobs.get(started["job"]["id"]) + subscribed, _left = self._watch_subscribers(job) + attached = asyncio.ensure_future(self.client.post(self.PROGRESS_URL)) + await self._wait(subscribed) # M4: an event with a ceiling, never a poll + release.set() + events = parse_sse(await (await attached).text()) + + self.assertEqual(events[-1][0], "complete") + # I1: the stream names the job it did not start: the gallery never cancels it. + self.assertEqual(events[0], ("job", {"id": job.id, "started": False})) + # One job: the stream followed the running one with its own options. + [listed] = self.api._jobs.list() + self.assertIs(listed, job) + self.assertEqual(job.scope["size"], 256) + + async def test_cancel_from_activity_ends_the_legacy_stream_with_an_error(self): + for i in range(2): + make_png(self.output_dir / f"p{i}.png") + entered, release = self._gate_generation() + stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL)) + await self._wait(entered) + job = self.api._jobs.active("thumbnails") + self.assertTrue(self.api._jobs.cancel(job.id)) + release.set() + + events = parse_sse(await (await stream).text()) + + self.assertEqual(events[-1][0], "error") + self.assertEqual(events[-1][1]["error"], "Thumbnail generation was cancelled") + await asyncio.wait_for(job.task, 10) + self.assertEqual(job.state, "cancelled") + + async def test_a_job_started_while_the_stream_prepares_is_attached(self): + """M3: a job that starts during ``prepare`` is followed, not refused.""" + from aiohttp import web + + make_png(self.output_dir / "a.png") + entered, release = self._gate_generation() + real_prepare = web.StreamResponse.prepare + raced = [] + + async def prepare(response, request): + if not raced and type(response) is web.StreamResponse: + options, _ = thumbnail_options({"size": 256}) + raced.append(self.api._start_thumbnail_job(options)) + return await real_prepare(response, request) + + with patch.object(web.StreamResponse, "prepare", prepare): + stream = asyncio.ensure_future(self.client.post(self.PROGRESS_URL)) + await self._wait(entered) + release.set() + events = parse_sse(await (await stream).text()) + + [job] = raced + self.assertEqual(events[0], ("job", {"id": job.id, "started": False})) + self.assertEqual(events[-1][0], "complete") + self.assertEqual(self.api._jobs.list(), [job]) + + async def test_a_refused_stream_names_no_job(self): + make_png(self.output_dir / "a.png") + self._run_scan() + events = parse_sse(await (await self.client.post(self.PROGRESS_URL)).text()) + self.assertEqual( + events, + [ + ( + "error", + { + "error": "Scan for metadata already running", + "message": "Scan for metadata already running", + }, + ) + ], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/vendor/frontend-coverage-baseline.json b/tools/vendor/frontend-coverage-baseline.json index 9d32cdc..9942d6b 100644 --- a/tools/vendor/frontend-coverage-baseline.json +++ b/tools/vendor/frontend-coverage-baseline.json @@ -42,10 +42,10 @@ "statements": 98 }, "gallery.js": { - "lines": 41, - "branches": 63, - "functions": 44, - "statements": 41 + "lines": 45, + "branches": 64, + "functions": 48, + "statements": 45 }, "html-escape.js": { "lines": 98, @@ -161,6 +161,12 @@ "functions": 98, "statements": 97 }, + "workspace/activity-kinds.mjs": { + "lines": 98, + "branches": 95, + "functions": 98, + "statements": 98 + }, "workspace/activity-scan.mjs": { "lines": 96, "branches": 78, @@ -175,7 +181,7 @@ }, "workspace/activity.mjs": { "lines": 96, - "branches": 88, + "branches": 91, "functions": 98, "statements": 96 }, @@ -509,6 +515,12 @@ "functions": 98, "statements": 98 }, + "workspace/thumbnail-review.mjs": { + "lines": 96, + "branches": 93, + "functions": 98, + "statements": 96 + }, "workspace/toast.mjs": { "lines": 98, "branches": 98, diff --git a/web/js/gallery.js b/web/js/gallery.js index cad3c2c..eef4ac0 100644 --- a/web/js/gallery.js +++ b/web/js/gallery.js @@ -1,4 +1,7 @@ class GalleryManager { + // A Cancel on a thumbnail job another tab or System started (#249 System S5, review I1). + static THUMBNAILS_STOPPED_FOLLOWING = 'Stopped following thumbnail generation started elsewhere; it keeps running in Activity'; + constructor() { this.images = []; this.currentPage = 1; @@ -1911,6 +1914,54 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; } } + /** Ask the server to stop thumbnail job `jobId` at its next file boundary (#249 System S5). */ + async cancelThumbnailJob(jobId) { + try { + await this.api.post(`/prompt_manager/jobs/${encodeURIComponent(jobId)}/cancel`, {}); + return true; + } catch (error) { + console.warn('The thumbnail job could not be cancelled:', error); + return false; + } + } + + /** + * Cancel for one thumbnail stream (#249 System S5, review I1). The stream's first frame, `job {id, started}`, + * names the job it follows and whether it started it. `request()` stops that job by its id, and only when this + * stream started it: a job another tab or System started keeps running, and the stream just stops following. + * A request before the `job` frame keeps the stream open until the frame arrives; `ended()` (the stream closed + * or refused without naming a job) leaves nothing to cancel. + */ + thumbnailCanceller(eventSource) { + let job = null; + let wanted = false; + let done = false; + const apply = () => { + if (done || !wanted || !job) return; + done = true; + eventSource.close(); + if (job.started) this.cancelThumbnailJob(job.id); + }; + return { + identify(raw) { + if (job) return; + let data = null; + try { data = JSON.parse(raw); } catch (error) { return; } + if (typeof data?.id !== 'string' || !data.id) return; + job = { id: data.id, started: data.started === true }; + apply(); + }, + request() { wanted = true; apply(); }, + /** True once the stream named a job this page did not start. */ + get following() { return job !== null && !job.started; }, + ended() { + if (done || !wanted || job) return; + done = true; + eventSource.close(); + }, + }; + } + async generateThumbnailsWithProgress(options) { const progressBar = document.getElementById('progressBar'); const progressText = document.getElementById('progressText'); @@ -1932,13 +1983,19 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; let cancelled = false; let resultData = null; + const canceller = this.thumbnailCanceller(eventSource); + eventSource.addEventListener('job', event => canceller.identify(event.data)); - // Handle cancellation + // Handle cancellation. The work is an Activity job that outlives this stream, so closing it no + // longer stops anything: ask the server to stop this stream's job by its id (review I1). const cancelHandler = () => { + if (cancelled) return; cancelled = true; - eventSource.close(); + cancelBtn.removeEventListener('click', cancelHandler); cancelBtn.classList.add('hidden'); - reject(new Error('Thumbnail generation cancelled by user')); + const following = canceller.following; + canceller.request(); + reject(new Error(following ? GalleryManager.THUMBNAILS_STOPPED_FOLLOWING : 'Thumbnail generation cancelled by user')); }; cancelBtn.addEventListener('click', cancelHandler); @@ -2013,7 +2070,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; // Handle completion eventSource.addEventListener('complete', (event) => { - if (cancelled) return; + if (cancelled) { canceller.ended(); return; } resultData = JSON.parse(event.data); // Update final progress @@ -2061,7 +2118,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; // Handle errors eventSource.addEventListener('error', (event) => { - if (cancelled) return; + if (cancelled) { canceller.ended(); return; } const data = JSON.parse(event.data); // Clean up @@ -2074,7 +2131,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; // Handle EventSource errors eventSource.onerror = (error) => { - if (cancelled) return; + if (cancelled) { canceller.ended(); return; } console.error('EventSource error:', error); console.error('EventSource readyState:', eventSource.readyState); @@ -2097,6 +2154,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; // Stream ended without a complete/error frame (close() does not trigger this) eventSource.onclose = () => { + if (cancelled) canceller.ended(); if (cancelled || resultData) return; cancelBtn.removeEventListener('click', cancelHandler); cancelBtn.classList.add('hidden'); @@ -2692,12 +2750,15 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; statusMessages.scrollTop = statusMessages.scrollHeight; }; - // Handle cancel + // Handle cancel: the stream's own job stops at its next file boundary (#249 System S5, review I1) + let cancelStream = null; cancelBtn.addEventListener('click', () => { + if (isCancelled) return; isCancelled = true; addStatusMessage('Cancellation requested...', 'warning'); cancelBtn.disabled = true; cancelBtn.textContent = 'Cancelling...'; + cancelStream?.(); }); // Handle close (only appears when done) @@ -2721,7 +2782,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; progressDetails, progressETA, addStatusMessage, - isCancelled: () => isCancelled + isCancelled: () => isCancelled, + onCancel: handler => { cancelStream = handler; } }); // Show completion @@ -2741,10 +2803,14 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; } } catch (error) { - console.error('Thumbnail generation failed:', error); - addStatusMessage(`Generation failed: ${error.message}`, 'error'); - progressBar.classList.add('bg-pm-error'); - progressBar.classList.remove('bg-pm-accent'); + if (isCancelled) { + addStatusMessage(error.message, 'warning'); + } else { + console.error('Thumbnail generation failed:', error); + addStatusMessage(`Generation failed: ${error.message}`, 'error'); + progressBar.classList.add('bg-pm-error'); + progressBar.classList.remove('bg-pm-accent'); + } } finally { // Show close button and hide cancel cancelBtn.classList.add('hidden'); @@ -2762,7 +2828,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; }, 100); } } async generateThumbnailsWithProgressInModal(uiElements) { - const { progressText, progressPercent, progressBar, progressDetails, progressETA, addStatusMessage, isCancelled } = uiElements; + const { progressText, progressPercent, progressBar, progressDetails, progressETA, addStatusMessage, isCancelled, onCancel } = uiElements; return new Promise((resolve, reject) => { addStatusMessage('Starting thumbnail generation...'); @@ -2776,6 +2842,14 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`; const eventSource = SseStream.connect(`/prompt_manager/images/generate-thumbnails/progress?quality=${encodeURIComponent(quality)}`, { method: 'POST' }); let resultData = null; + const canceller = this.thumbnailCanceller(eventSource); + eventSource.addEventListener('job', event => canceller.identify(event.data)); + // Cancel stops this stream's job by id, keeps what it wrote, and settles the dialog. + onCancel?.(() => { + const following = canceller.following; + canceller.request(); + reject(new Error(following ? GalleryManager.THUMBNAILS_STOPPED_FOLLOWING : 'Thumbnail generation cancelled; previews already written are kept')); + }); // Handle scanning/status events eventSource.addEventListener('status', (event) => { diff --git a/web/js/workspace/activity-job-view.mjs b/web/js/workspace/activity-job-view.mjs index 6d1fc26..f0fb3b1 100644 --- a/web/js/workspace/activity-job-view.mjs +++ b/web/js/workspace/activity-job-view.mjs @@ -14,7 +14,11 @@ * - Log: `details.term` "Scan log", collapsed, holding the raw timestamped lines in `pre[tabindex=0]`. * - D89: the tiles, the outcome list, the openable rows and the log and cancel labels are the job kind's (`jobView`): * an image tagging job shows Tagged, Unchanged, Skipped and Errors, opens Tagged and Unchanged rows, and reads - * "Tagging log" and "Cancel tagging"; another kind keeps the scan's tiles as "Job log" and "Cancel job". + * "Tagging log" and "Cancel tagging". + * - D111: the kind's descriptor (activity-kinds.mjs `viewOf`) decides them for every kind: a thumbnail job counts + * Generated, Skipped and Errors; a kind without groups (the health check, a held legacy-page operation) shows no + * tiles and no Results, never another kind's zeros. A job without a known total has an indeterminate `<progress>`; + * only a job the server can stop shows Cancel. * * The summary follows the Activity store (one SSE stream per running job); the listed rows and the log are read from * `GET /prompt_manager/jobs/<id>` at most once a second while the job runs, and once more when it ends. @@ -23,7 +27,8 @@ import { createButton } from '../shared/toolkit/core.js'; import { pageHeading } from './heading.mjs'; import { showWorkspaceToast } from './toast.mjs'; import { noun } from './plural.mjs'; -import { parseJob, jobView, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs'; +import { parseJob, JOB_VIEWS, JOBS_URL, LOST_DETAIL } from './activity-store.mjs'; +import { viewOf, canCancel, showProgress } from './activity-kinds.mjs'; import { jobDetail, CANCEL_FAILED_TOAST, createReviewMetadataButton, hasReviewItems } from './activity.mjs'; import { downloadJobReport } from './activity-scan.mjs'; import { createMediaViewer } from './media-viewer.mjs'; @@ -71,7 +76,10 @@ export function parseDetail(raw) { /** The progress line: the server's while running, then what the job got through. */ export function progressLine(job) { const files = `${number(job.processed)} of ${number(job.total)} ${noun(job.total, 'file')}`; - if (job.state === 'running') return job.status_line || `${job.progress}%`; + // D111: a job that never counted files (a held legacy job, the health check) says only what the server says. + const counted = job.total > 0; + if (job.state === 'running') return job.status_line || (counted ? `${job.progress}%` : ''); + if (!counted && (job.state === 'finished' || job.state === 'cancelled')) return job.status_line; if (job.state === 'finished') return `${files} processed`; if (job.state === 'cancelled') return `Cancelled after ${files}`; if (job.state === 'lost') return LOST_DETAIL; @@ -97,7 +105,7 @@ export function filterRows(rows, filter) { /** The first outcome with files, so a view opens on what happened; the first outcome when nothing has yet. */ export function defaultOutcome(counters, outcomes = JOB_VIEWS.scan.outcomes) { - return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])[0]; + return (outcomes.find(([group]) => (counters[group] ?? 0) > 0) ?? outcomes[0])?.[0] ?? null; } function section(title, id) { @@ -179,9 +187,10 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { const viewer = createMediaViewer({ api, container: main, parentScope: scope, inspectionController: inspection, sender: createPromptSender({ scope }), onSelect: image => inspection.select(image) }); - /** The tiles and the outcome list of the job's kind, built once. */ - function layout(kind) { - view = jobView(kind); // D89 + /** The tiles and the outcome list of the job's kind, built once (D89, D111: only the groups it counts). */ + function layout(job) { + view = viewOf(job); + if (!view.tiles.length) stats.remove(); for (const [group, label, tone] of view.tiles) { const tile = element('div', undefined, tone ? `stat ${tone}` : 'stat'); const value = element('dd', '0'); @@ -203,10 +212,11 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { function showBody(job) { if (mounted) return; mounted = true; - if (!view) layout(job.kind); + if (!view) layout(job); missing.remove(); parts.actions.append(download, cancel); - main.append(progressPart.node, resultsPart.node, log); + // A kind without outcome groups has no Results to browse (D111). + main.append(...(view.outcomes.length ? [progressPart.node, resultsPart.node, log] : [progressPart.node, log])); selected ??= defaultOutcome(job.counters, view.outcomes); } @@ -242,6 +252,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { } function renderRows(job) { + if (!view.outcomes.length) return; const rows = detail?.outcomes[selected] ?? []; const openable = view.openable.has(selected); const wanted = filterRows(rows.map((row, index) => ({ ...row, index })), input.value); @@ -279,7 +290,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { if (hasReviewItems(job)) { if (!review.isConnected) download.after(review); } // D87 else review.remove(); // Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel. - if (job.state === 'running' && job.cancellable) { + if (canCancel(job)) { if (!cancel.isConnected) parts.actions.append(cancel); } else if (cancel.isConnected) { // A focused Cancel that goes away hands its focus to the job's heading, never to the page body. @@ -287,8 +298,8 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { cancel.remove(); } status.textContent = progressLine(job); - bar.value = job.progress; - bar.textContent = `${job.progress}%`; + // D111: no value while the total is unknown; a job that ended with nothing counted shows how it ended. + showProgress(bar, job.total > 0 || job.state !== 'finished' ? job : { ...job, progress: 100 }); bar.setAttribute('aria-label', scan ? 'Scan progress' : `${job.title} progress`); for (const [group, value] of tileValues) value.textContent = number(job.counters[group] ?? 0); for (const [group, button] of buttons) { diff --git a/web/js/workspace/activity-kinds.mjs b/web/js/workspace/activity-kinds.mjs new file mode 100644 index 0000000..3fd7c75 --- /dev/null +++ b/web/js/workspace/activity-kinds.mjs @@ -0,0 +1,189 @@ +/** + * Per-kind Activity descriptors (#249 System S5, D111): one table that says, for each job kind, what Activity shows. + * + * `kindOf(kind)` → `{pillVerb, startToast(job), finishToast(job), line(job), kept(job), scope(job), tiles, outcomes, + * openable, log, cancelLabel}`: + * - `pillVerb`: the pill's verb ("Scanning"); null shows the job's title. + * - `startToast` / `finishToast`: what a toast says when this tab starts the job and when it ends. + * - `line`: the counters as one line ("N added · M errors"); `kept`: what a cancelled job kept ("N added"); `scope`: + * what the job covers ("2 output folders"). + * - `tiles` (`[group, label, tone]`), `outcomes` (`[group, label]`) and `openable` (rows that open the viewer, D57). + * - `log` and `cancelLabel` (`cancel` is the same label under the job view's older name). + * + * `scan` is exactly the D56 view and `image-tags` the D89 one. `thumbnails` counts Generated (green), Skipped and + * Errors (red). Any other kind (the health check, a held legacy-page operation, a kind this page does not know) gets + * the generic descriptor: title-based copy and no tiles or outcomes, so it never shows another kind's zeros (S1 review + * M6, S2 review M3). `viewOf(job)` keeps only the groups the job itself counts, so a hold (no groups) has none either. + * + * Progress is indeterminate while a job runs without a known total (`indeterminate`): the `<progress>` has no value + * and the pill shows no percentage, never a made-up 0%. A job shows Cancel only while it runs and the server can stop + * it (`canCancel`). + */ +import { noun, countOf } from './plural.mjs'; + +// Grouped as the server writes its status lines ("8,240 of 19,622"), whatever the browser's locale. +const number = value => value.toLocaleString('en-US'); +const counted = (job, group, word) => `${number(job.counters?.[group] ?? 0)} ${word}`; +const errorsOf = job => `${number(job.counters?.errors ?? 0)} ${noun(job.counters?.errors ?? 0, 'error')}`; +const whole = value => Number.isSafeInteger(value) && value >= 0; +const tiles = rows => Object.freeze(rows.map(Object.freeze)); +const groups = (...names) => Object.freeze(new Set(names)); + +/** "1,420 added". */ +export const addedOf = job => counted(job, 'added', 'added'); +/** "12 tagged" (D89). */ +export const taggedOf = job => counted(job, 'tagged', 'tagged'); +/** "3 generated" (D111). */ +export const generatedOf = job => counted(job, 'generated', 'generated'); + +/** "2 output folders": what a scan or a thumbnail job walks. */ +function foldersOf(job) { + const folders = job.scope?.output_folders; + return whole(folders) ? countOf(folders, 'output folder') : ''; +} + +/** "4 images" (D89), grouped as the server writes its lines. */ +function imagesOf(job) { + const images = job.scope?.images; + return whole(images) ? `${number(images)} ${noun(images, 'image')}` : ''; +} + +const titleStarted = job => `${job.title} started.`; + +function titleFinished(job) { + if (job.state === 'cancelled') return `${job.title} was cancelled.`; + if (job.state === 'failed') return `${job.title} failed.`; + return `${job.title} finished.`; +} + +/** "<Noun> finished · <line>", "<Noun> cancelled · <kept>", "<Noun> failed · <line>". */ +const namedFinish = (name, kept, line) => job => { + if (job.state === 'cancelled') return `${name(job)} cancelled · ${kept(job)}`; + return `${name(job)} ${job.state === 'failed' ? 'failed' : 'finished'} · ${line(job)}`; +}; + +const nothing = () => ''; +const GENERIC = Object.freeze({ + pillVerb: null, startToast: titleStarted, finishToast: titleFinished, line: nothing, kept: nothing, + scope: foldersOf, tiles: tiles([]), outcomes: tiles([]), openable: groups(), log: 'Job log', cancelLabel: 'Cancel job', + cancel: 'Cancel job', +}); + +const scanLine = job => `${addedOf(job)} · ${errorsOf(job)}`; +const tagLine = job => `${taggedOf(job)} · ${errorsOf(job)}`; +const thumbnailLine = job => `${generatedOf(job)} · ${errorsOf(job)}`; +// D85: an image tagging job says which of its two modes started. +const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' }); + +/** Every kind with a descriptor of its own. */ +export const KINDS = Object.freeze({ + __proto__: null, + // D56: the scan's tiles and outcome list in the mockup's orders; what got added and what went wrong first. + scan: Object.freeze({ + ...GENERIC, + pillVerb: 'Scanning', + startToast: () => 'Scan started.', + finishToast: namedFinish(() => 'Scan', addedOf, scanLine), + line: scanLine, + kept: addedOf, + tiles: tiles([['added', 'Added to Library', 'added'], ['known', 'Already known', ''], + ['review', 'Needs review', 'warn'], ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad']]), + outcomes: tiles([['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'], + ['no_metadata', 'No usable metadata'], ['known', 'Already known']]), + openable: groups('added', 'known'), + log: 'Scan log', + cancelLabel: 'Cancel scan', + cancel: 'Cancel scan', + }), + // D89: Tagged (green), Unchanged, Skipped (warning) and Errors (red); Tagged and Unchanged rows open the viewer. + 'image-tags': Object.freeze({ + ...GENERIC, + pillVerb: 'Tagging', + startToast: job => IMAGE_TAG_START_TOASTS[job.scope?.mode] ?? titleStarted(job), + finishToast: namedFinish(job => job.title, taggedOf, tagLine), + line: tagLine, + kept: taggedOf, + scope: imagesOf, + tiles: tiles([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'], + ['errors', 'Errors', 'bad']]), + outcomes: tiles([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'], ['unchanged', 'Unchanged']]), + openable: groups('tagged', 'unchanged'), + log: 'Tagging log', + cancelLabel: 'Cancel tagging', + cancel: 'Cancel tagging', + }), + // D111: Generated (green), Skipped (already fresh) and Errors (red); a row with a preview opens the viewer. + thumbnails: Object.freeze({ + ...GENERIC, + pillVerb: 'Generating thumbnails', + finishToast: namedFinish(() => 'Thumbnails', generatedOf, thumbnailLine), + line: thumbnailLine, + kept: generatedOf, + tiles: tiles([['generated', 'Generated', 'added'], ['skipped', 'Skipped', ''], ['errors', 'Errors', 'bad']]), + outcomes: tiles([['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]), + openable: groups('generated', 'skipped'), + }), +}); + +/** The descriptor of a job `kind`; any kind without one gets the generic descriptor. */ +export function kindOf(kind) { + return (typeof kind === 'string' && KINDS[kind]) || GENERIC; +} + +const countsOf = job => job.counters && typeof job.counters === 'object' ? job.counters : {}; + +/** + * A legacy-page operation held on the registry (D112): not cancellable and counting no outcome groups. Whatever its + * kind (the gallery's thumbnail clear is a `thumbnails` hold), it reads as the generic job under its own title. + */ +export const held = job => job.cancellable === false && Object.keys(countsOf(job)).length === 0; + +/** + * The descriptor for one job: its kind's, keeping only the outcome groups the job counts; a held job's is the generic + * one. + */ +export function viewOf(job) { + if (held(job)) return GENERIC; + const view = kindOf(job.kind); + const counts = countsOf(job); + const has = ([group]) => Object.hasOwn(counts, group); + if (view.tiles.every(has) && view.outcomes.every(has)) return view; + return Object.freeze({ ...view, tiles: tiles(view.tiles.filter(has)), outcomes: tiles(view.outcomes.filter(has)), + openable: groups(...[...view.openable].filter(group => Object.hasOwn(counts, group))) }); +} + +/** A running job whose total the server has not reported (yet) has no percentage. */ +export function indeterminate(job) { + return job.state === 'running' && !(job.total > 0); +} + +/** Set a `<progress>` to the job's: its value and "N%", or no value at all while indeterminate. */ +export function showProgress(node, job) { + if (indeterminate(job)) { + node.removeAttribute('value'); + node.textContent = ''; + return; + } + node.value = job.progress; + node.textContent = `${job.progress}%`; +} + +/** Truth rule: Cancel only while the job runs and the server can stop it (never on a held legacy job). */ +export function canCancel(job) { + return job.state === 'running' && job.cancellable !== false; +} + +/** The pill's text: "Scanning · 42% · 8,240 / 19,622"; the file counts once the total is known, no "%" before. */ +export function pillParts(job) { + const verb = viewOf(job).pillVerb ?? job.title; + const label = indeterminate(job) ? verb : `${verb} · ${job.progress}%`; + const files = job.total > 0 ? ` · ${number(job.processed)} / ${number(job.total)}` : ''; + return { label, files, text: label + files }; +} + +/** The toast after this tab starts a job. */ +export const startToast = job => viewOf(job).startToast(job); +/** The toast when a job ends. */ +export const finishToast = job => viewOf(job).finishToast(job); +/** The counters as one line; '' for a kind without one. */ +export const outcomeLine = job => viewOf(job).line(job); diff --git a/web/js/workspace/activity-store.mjs b/web/js/workspace/activity-store.mjs index 0729d44..ca535c4 100644 --- a/web/js/workspace/activity-store.mjs +++ b/web/js/workspace/activity-store.mjs @@ -12,7 +12,9 @@ * the finish toast follows any job this store saw running when it ends. */ import { showWorkspaceToast } from './toast.mjs'; -import { noun } from './plural.mjs'; +import { KINDS, kindOf, finishToast, startToast } from './activity-kinds.mjs'; + +export { addedOf, taggedOf, finishToast, startToast, pillParts, outcomeLine } from './activity-kinds.mjs'; export const JOBS_URL = '/prompt_manager/jobs'; export const BACKOFF_FIRST_MS = 1000; @@ -23,51 +25,13 @@ export const LOST_DETAIL = 'This job ended when ComfyUI restarted.'; const SERVER_STATES = new Set(['running', 'finished', 'cancelled', 'failed']); const FINAL_STATES = new Set(['finished', 'cancelled', 'failed']); const JOB_ID = /^[0-9a-f]{32}$/; -// The pill's verb for each job kind; a kind without one shows its title (D89: image tagging jobs read "Tagging"). -const PILL_VERBS = Object.freeze({ scan: 'Scanning', 'image-tags': 'Tagging' }); -const START_TOASTS = Object.freeze({ scan: 'Scan started.' }); - -/** The scan's job view (D56): its counter tiles `[group, label, tone]` and outcome list in the mockup's orders. */ -const SCAN_VIEW = Object.freeze({ - tiles: Object.freeze([ - ['added', 'Added to Library', 'added'], ['known', 'Already known', ''], ['review', 'Needs review', 'warn'], - ['no_metadata', 'No usable metadata', ''], ['errors', 'Errors', 'bad'], - ].map(Object.freeze)), - // What got added and what went wrong first. - outcomes: Object.freeze([ - ['added', 'Added to Library'], ['review', 'Needs review'], ['errors', 'Errors'], - ['no_metadata', 'No usable metadata'], ['known', 'Already known'], - ].map(Object.freeze)), - // Rows that open the file in the viewer (D57); the others are plain text. - openable: Object.freeze(new Set(['added', 'known'])), - log: 'Scan log', - cancel: 'Cancel scan', -}); - /** - * Each job kind's view: its tiles, outcome list, openable rows, log label and cancel label. D89: an image tagging job - * (`image-tags`) counts Tagged (green), Unchanged, Skipped (warning) and Errors (red), lists what got tagged and what - * went wrong first, and opens Tagged and Unchanged rows in the viewer like the scan's (D57). + * Each job kind's copy and view live in one table (activity-kinds.mjs, D111): the scan's (D56), image tagging's (D89), + * thumbnails' and the generic one. The names below are that table's, kept for this store's importers. */ -export const JOB_VIEWS = Object.freeze({ - scan: SCAN_VIEW, - 'image-tags': Object.freeze({ - tiles: Object.freeze([['tagged', 'Tagged', 'added'], ['unchanged', 'Unchanged', ''], ['skipped', 'Skipped', 'warn'], - ['errors', 'Errors', 'bad']].map(Object.freeze)), - outcomes: Object.freeze([['tagged', 'Tagged'], ['errors', 'Errors'], ['skipped', 'Skipped'], - ['unchanged', 'Unchanged']].map(Object.freeze)), - openable: Object.freeze(new Set(['tagged', 'unchanged'])), - log: 'Tagging log', - cancel: 'Cancel tagging', - }), -}); -// Another kind keeps the scan's tiles and outcomes under the generic labels. -const GENERIC_VIEW = Object.freeze({ ...SCAN_VIEW, log: 'Job log', cancel: 'Cancel job' }); - +export const JOB_VIEWS = KINDS; /** The view of a job `kind`; an unknown kind gets the generic one. */ -export function jobView(kind) { - return Object.hasOwn(JOB_VIEWS, kind) ? JOB_VIEWS[kind] : GENERIC_VIEW; -} +export const jobView = kindOf; /** The wait before the `attempt`-th re-read (0-based): 1 s, 2 s, 4 s … never more than 30 s. */ export function backoffDelay(attempt) { @@ -108,57 +72,6 @@ export function parseJob(raw) { }; } -// Grouped as the server writes its status lines ("8,240 of 19,622"), whatever the browser's locale. -const number = value => value.toLocaleString('en-US'); -const errorsOf = job => `${number(job.counters.errors ?? 0)} ${noun(job.counters.errors ?? 0, 'error')}`; -/** "1,420 added". */ -export const addedOf = job => `${number(job.counters.added ?? 0)} added`; - -/** - * The toast when a job ends: "Scan finished · N added · M errors", "Scan cancelled · N added"; an image tagging job - * the same way with its own title and count (D89): "Auto-tag images finished · N tagged · M errors". - */ -export function finishToast(job) { - if (job.kind === 'scan') { - if (job.state === 'cancelled') return `Scan cancelled · ${addedOf(job)}`; - if (job.state === 'failed') return `Scan failed · ${addedOf(job)} · ${errorsOf(job)}`; - return `Scan finished · ${addedOf(job)} · ${errorsOf(job)}`; - } - if (job.kind === 'image-tags') { - if (job.state === 'cancelled') return `${job.title} cancelled · ${taggedOf(job)}`; - return `${job.title} ${job.state === 'failed' ? 'failed' : 'finished'} · ${outcomeLine(job)}`; - } - if (job.state === 'cancelled') return `${job.title} was cancelled.`; - if (job.state === 'failed') return `${job.title} failed.`; - return `${job.title} finished.`; -} - -// D85: an image tagging job says which of its two modes started. -const IMAGE_TAG_START_TOASTS = Object.freeze({ add: 'Tagging started.', auto: 'Auto-tagging started.' }); - -/** The toast after this tab starts a job. */ -export function startToast(job) { - if (job.kind === 'image-tags' && IMAGE_TAG_START_TOASTS[job.scope.mode]) return IMAGE_TAG_START_TOASTS[job.scope.mode]; - return START_TOASTS[job.kind] ?? `${job.title} started.`; -} - -/** The pill's text: "Scanning · 42% · 8,240 / 19,622"; the file counts only once the total is known. */ -export function pillParts(job) { - const label = `${PILL_VERBS[job.kind] ?? job.title} · ${job.progress}%`; - const files = job.total > 0 ? ` · ${number(job.processed)} / ${number(job.total)}` : ''; - return { label, files, text: label + files }; -} - -/** "12 tagged" (D89). */ -export const taggedOf = job => `${number(job.counters.tagged ?? 0)} tagged`; - -/** The counters as one line: "N added · M errors" for a scan, "N tagged · M errors" for image tagging (D89). */ -export function outcomeLine(job) { - if (job.kind === 'scan') return `${addedOf(job)} · ${errorsOf(job)}`; - if (job.kind === 'image-tags') return `${taggedOf(job)} · ${errorsOf(job)}`; - return ''; -} - function defaultConnect(url) { const resolve = globalThis.ApiClient?.resolveUrl ?? (value => value); return new EventSource(resolve(url)); diff --git a/web/js/workspace/activity.mjs b/web/js/workspace/activity.mjs index a2fc8b9..9595f89 100644 --- a/web/js/workspace/activity.mjs +++ b/web/js/workspace/activity.mjs @@ -14,13 +14,15 @@ * - D87 (plan D64): an ended scan that recorded Needs review items carries the concept's **Review metadata** * (`go-review`, app.js a738) after its line, which opens `#/review`; its chip still names its state (D54). * - D89: an image tagging job's line names its images and, once ended, "N tagged · M errors". + * - D111: what a line says (scope, counters, what a cancel kept) is the job kind's descriptor (activity-kinds.mjs); a + * job without a known total shows an indeterminate `<progress>`, and only a job the server can stop shows Cancel. */ import { createButton } from '../shared/toolkit/core.js'; import { navigateTo } from './state.mjs'; import { pageHeading } from './heading.mjs'; import { showWorkspaceToast } from './toast.mjs'; -import { countOf, noun } from './plural.mjs'; -import { pillParts, outcomeLine, addedOf, taggedOf, LOST_DETAIL } from './activity-store.mjs'; +import { pillParts, outcomeLine, LOST_DETAIL } from './activity-store.mjs'; +import { viewOf, canCancel, showProgress, indeterminate } from './activity-kinds.mjs'; export const ACTIVITY_DESCRIPTION = 'Scans, tagging, downloads and maintenance share one history.'; export const EMPTY_ACTIVITY = 'No jobs have run since ComfyUI started. Scans and other long operations appear here.'; @@ -60,20 +62,9 @@ export function clockTime(seconds) { const join = parts => parts.filter(Boolean).join(' · '); -const whole = value => Number.isSafeInteger(value) && value >= 0; - -/** What a job covers: "2 output folders" for a scan, "4 images" for image tagging (D89). */ -function scopeLine(job) { - // D89: grouped as the server writes its lines ("19,622 images"). - const images = job.scope?.images; - if (job.kind === 'image-tags') return whole(images) ? `${images.toLocaleString('en-US')} ${noun(images, 'image')}` : ''; - const folders = job.scope?.output_folders; - return whole(folders) ? countOf(folders, 'output folder') : ''; -} - -/** What a cancelled job kept: "N added" for a scan, "N tagged" for image tagging (D89). */ -const KEPT = Object.freeze({ __proto__: null, scan: addedOf, 'image-tags': taggedOf }); // D89 -const keptOf = job => KEPT[job.kind]?.(job) ?? ''; +/** What a job covers ("2 output folders", "4 images") and what a cancelled one kept ("N added"), by kind (D111). */ +const scopeLine = job => viewOf(job).scope(job); +const keptOf = job => viewOf(job).kept(job); /** The item's line under its title, by state. */ export function jobDetail(job) { @@ -119,7 +110,10 @@ export function mountActivityTopbar({ store, scope, doc = document }) { const parts = pillParts(job); label.textContent = parts.label; files.textContent = parts.files; - fill.style.width = `${job.progress}%`; + // D111 (S5 review M5): no total, no fill: the bar sweeps instead of showing an empty determinate 0%. + const unknown = indeterminate(job); + bar.classList.toggle('indeterminate', unknown); + fill.style.width = unknown ? '' : `${job.progress}%`; pill.href = `#/activity/${job.id}`; pill.setAttribute('aria-label', parts.text); }; @@ -167,15 +161,14 @@ function itemFor(job, scope) { // parentNode, not isConnected: on the first render the item is not in the page yet, so a job that had already // ended when Activity opened kept its Cancel job (and a running one's progress moved after its actions). if (running) { - progress.value = next.progress; - progress.textContent = `${next.progress}%`; + showProgress(progress, next); // D111: no value while the total is unknown progress.setAttribute('aria-label', `${next.title} progress`); if (!progress.parentNode) item.insertBefore(progress, actions.parentNode ? actions : null); } else { progress.remove(); } // Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel job. - if (running && next.cancellable) { + if (canCancel(next)) { if (!actions.parentNode) item.append(actions); } else if (actions.parentNode) { // A focused Cancel job that goes away hands its focus to the item's title link, never to the page body. diff --git a/web/js/workspace/system-overview.mjs b/web/js/workspace/system-overview.mjs index ce13bc9..8178be2 100644 --- a/web/js/workspace/system-overview.mjs +++ b/web/js/workspace/system-overview.mjs @@ -7,6 +7,7 @@ * N issues / Incomplete); the review, backup, thumbnail and sources rows say what the server reports, and nothing * reads 0 before the read answers: a load shows a status line, a failure an alert and Retry. The thumbnail row is * its own read of `/images/thumbnails/status` (a walk of every source): "Checking thumbnails…" until it answers. + * D99: its Review thumbnails opens the same "Thumbnail maintenance" dialog as Sources (thumbnail-review.mjs). * - D96: the Statistics dialog lists the real runs and average rating per category, then "Library totals". * - D95: the "Health report" lists every check the server ran, Missing files per image source, with a chip per real * result and when it was checked. The check runs from Maintenance's notice ("Run health checks", D123) as a @@ -20,6 +21,7 @@ import { closeWorkspaceDialog, openContentDialog } from './workspace-dialogs.mjs import { showWorkspaceToast } from './toast.mjs'; import { noun } from './plural.mjs'; import { operationRow } from './system-controls.mjs'; +import { openThumbnailReview } from './thumbnail-review.mjs'; export const OVERVIEW_URL = '/prompt_manager/system/overview'; export const STATISTICS_URL = '/prompt_manager/system/statistics'; @@ -317,7 +319,7 @@ function surface(heading, rows) { } /** The Overview's nodes, and its thumbnail row (painted by its own read). */ -function overviewNodes(overview, { scope, navigate, openStatistics }) { +function overviewNodes(overview, { scope, navigate, openStatistics, openThumbnails }) { const go = section => () => navigate(`#/system/${section}`); const review = operationRow({ scope, title: ambiguousTitle(overview.needs_review), description: Number.isSafeInteger(overview.needs_review?.pending) @@ -325,12 +327,12 @@ function overviewNodes(overview, { scope, navigate, openStatistics }) { label: 'Review metadata', onClick: () => navigate('#/review') }); const [sourcesTitle, sourcesDescription] = sourcesCopy(overview.sources); const thumbnails = operationRow({ scope, title: THUMBNAILS_LOADING, description: 'Generate only missing previews when necessary.', - label: 'Review thumbnails', onClick: go('sources') }); + label: 'Review thumbnails', onClick: openThumbnails }); const health = surface('Library health', [ review, operationRow({ scope, title: 'Backup status', description: backupText(overview.backup), label: 'View backups', onClick: go('backups') }), - // Until the thumbnail review dialog (S5), Review thumbnails opens Sources, where thumbnail generation lives. + // D99: Review thumbnails opens the thumbnail dialog (the concept's row starts a sample job directly). thumbnails, ]); const preferences = surface('Workspace preferences', [ @@ -350,7 +352,7 @@ function overviewNodes(overview, { scope, navigate, openStatistics }) { * answer replaces in place (the concept's metrics and columns sit directly in `.settings-layout`). A read that a newer * one superseded, or one answering after the section left, paints nothing. */ -export function mountOverview(container, { api, scope, navigate = navigateTo, notify = showWorkspaceToast, +export function mountOverview(container, { api, activity, scope, navigate = navigateTo, notify = showWorkspaceToast, latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) { const overviewRead = latestOnly({ lifetime: scope.signal }); const statisticsRead = latestOnly({ lifetime: scope.signal }); @@ -392,7 +394,8 @@ export function mountOverview(container, { api, scope, navigate = navigateTo, no try { const overview = await overviewRead(signal => api.get(OVERVIEW_URL, { signal, cache: 'no-store' })); if (overview === STALE || scope.signal.aborted) return; - const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics }); + const { nodes, thumbnails } = overviewNodes(overview, { scope, navigate, openStatistics, + openThumbnails: trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate }) }); thumbnailRow = thumbnails; sources = overview.sources; swap(nodes); diff --git a/web/js/workspace/system-sources.mjs b/web/js/workspace/system-sources.mjs index a204f07..c3b96cc 100644 --- a/web/js/workspace/system-sources.mjs +++ b/web/js/workspace/system-sources.mjs @@ -14,8 +14,8 @@ * it to the host draft of `gallery_root_paths`; Remove takes one out. Both wait for the one Save row. * - "Watch for new outputs" (`monitoring_enabled`) and "Worker threads" (`worker_threads`, a range up to the server's * `cpu_count`, D113) are host drafts too. "Enrich existing metadata" reviews a scan of every source. - * - "Thumbnail generation → Review options" opens the existing gallery's thumbnail tool until its review dialog (S5), - * the D94 hand-off for the one capability not native yet. + * - "Thumbnail generation → Review options" opens "Thumbnail maintenance" (thumbnail-review.mjs, S5 Task 14), which + * starts the thumbnail Activity job. * * D98: per-source Remove; an added source reads "not saved yet" until Save; the line under the list says when * ComfyUI's output folder is used instead; "restart pending" for a source the running monitor does not watch yet; the @@ -28,6 +28,7 @@ import { createButton } from '../shared/toolkit/core.js'; import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs'; import { openMetadataScan } from './activity-scan.mjs'; +import { openThumbnailReview } from './thumbnail-review.mjs'; import { settingRow, operationRow } from './system-controls.mjs'; import { HOST_UNAVAILABLE } from './system-appearance.mjs'; import { showWorkspaceToast } from './toast.mjs'; @@ -36,7 +37,6 @@ import { noun } from './plural.mjs'; const SETTINGS_URL = '/prompt_manager/settings'; const VALIDATE_URL = '/prompt_manager/settings/validate-source'; const RESCAN_URL = '/prompt_manager/gallery/rescan'; -const GALLERY_URL = '/prompt_manager/gallery'; const ROOTS_KEY = 'gallery_root_paths'; export const NOTICE_KEY = 'pm_update_notice_dismissed'; @@ -222,8 +222,7 @@ const controlOf = (row, id) => row.querySelector(`[data-setting="${id}"]`); * Activity store (scans start through it), `drafts` the System drafts. */ export function mountSources(container, { api, activity, scope, drafts, notify = showWorkspaceToast, storage = defaultStorage, - openWindow = (url) => window.open(url, '_blank', 'noopener'), latestOnly = window.ApiClient.latestOnly, - STALE = window.ApiClient.STALE }) { + latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) { const statusRead = latestOnly({ lifetime: scope.signal }); // undefined while read, null when the read failed, else GET /settings' `sources`. let sources; @@ -270,9 +269,8 @@ export function mountSources(container, { api, activity, scope, drafts, notify = surface.append(monitoring, workers, operationRow({ scope, title: 'Enrich existing metadata', description: 'Resolve both prompts, deduplicate recipes, preserve uncertain links.', label: 'Review scan', onClick: scanAll }), - // D94: the gallery keeps thumbnail generation until its review dialog (S5). operationRow({ scope, title: 'Thumbnail generation', description: 'Generate missing previews; rebuilding all is a separate explicit option.', - label: 'Review options', onClick: () => openWindow(window.ApiClient.resolveUrl(GALLERY_URL)) }), + label: 'Review options', onClick: trigger => openThumbnailReview({ api, store: activity, scope, trigger }) }), // D98: the legacy gallery's Refresh, which rereads every source's file list. operationRow({ scope, title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.', label: 'Refresh', onClick: refresh })); diff --git a/web/js/workspace/thumbnail-review.mjs b/web/js/workspace/thumbnail-review.mjs new file mode 100644 index 0000000..e764f56 --- /dev/null +++ b/web/js/workspace/thumbnail-review.mjs @@ -0,0 +1,172 @@ +/** + * Thumbnail review (#249 System S5 Task 14): the concept's "Thumbnail maintenance" dialog (`thumbnail-review`, app.js + * 1433–1437), opened by Sources' "Thumbnail generation → Review options" and Overview's "Review thumbnails". It starts + * the `thumbnails` Activity job (S5 Task 12) with the chosen options. + * + * - Scope: "Missing thumbnails only" (`missing`, the default) or "Rebuild all (explicit opt-in)" (`rebuild`). + * - Quality: "Medium · optimized" / "High · original" / "Low · fastest" (`medium` / `high` / `low`), preselected from + * Appearance's Image loading quality (`gallerySettings.imageQuality`, the same three values; Q3). + * - Preview size limit: the longest edge, a whole number from 128 to 2048 (concept default 512). A size outside that + * marks the field `aria-invalid`, says so in the plan line and sends nothing; Start reports the field's validity. + * - D99: a `p.small[role=status]` line before the actions says what the server's plan counts for the chosen scope + * (`GET /images/thumbnails/plan`, read-only): "N previews to generate · M already up to date." or, for a rebuild, + * "Every preview is regenerated: N files.", plus "· R more on the next run" past the per-run cap. Each scope is read + * once per dialog (the size and quality change no count). The primary is "Start" (the concept's "Simulate thumbnail + * job"): it closes the dialog and starts the job through the Activity store, which toasts that it started; the job + * opens in Activity, where its progress lives. A start refused because another job holds the slot toasts the D57 + * busy link toast and stays. + */ +import { createButton } from '../shared/toolkit/core.js'; +import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs'; +import { busyToast } from './activity-scan.mjs'; +import { readPreferences } from './preferences.mjs'; +import { showWorkspaceToast } from './toast.mjs'; +import { navigateTo } from './state.mjs'; +import { noun } from './plural.mjs'; + +export const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan'; +export const THUMBNAILS_KIND = 'thumbnails'; +export const SCOPE_OPTIONS = Object.freeze([['missing', 'Missing thumbnails only'], ['rebuild', 'Rebuild all (explicit opt-in)']]); +export const QUALITY_OPTIONS = Object.freeze([['medium', 'Medium · optimized'], ['high', 'High · original'], ['low', 'Low · fastest']]); +export const SIZE_LIMIT = Object.freeze({ min: 128, max: 2048, value: 512 }); +export const PLAN_LOADING = 'Counting previews…'; +export const PLAN_FAILED = 'The previews could not be counted.'; +export const SIZE_INVALID = `Enter a size from ${SIZE_LIMIT.min} to ${SIZE_LIMIT.max} pixels.`; +export const START_FAILED = 'Thumbnail generation could not start. Try again.'; + +const QUALITIES = new Set(QUALITY_OPTIONS.map(([value]) => value)); +const count = value => value.toLocaleString('en-US'); +const isCount = value => Number.isSafeInteger(value) && value >= 0; + +/** The preview size for the field's text: a whole number of pixels in range, else null. */ +export function previewSize(text) { + if (typeof text !== 'string' || !/^\d{1,5}$/.test(text)) return null; + const size = Number(text); + return size >= SIZE_LIMIT.min && size <= SIZE_LIMIT.max ? size : null; +} + +/** Q3: Image loading quality preselects the dialog's quality; anything it does not know is Medium. */ +export function preselectedQuality(imageQuality) { + return QUALITIES.has(imageQuality) ? imageQuality : 'medium'; +} + +/** D99: the plan line for the server's `{mode, targets, fresh, remaining}`; an answer without counts failed. */ +export function planText(plan) { + if (!plan || !['missing', 'rebuild'].includes(plan.mode) || ![plan.targets, plan.fresh, plan.remaining].every(isCount)) { + return PLAN_FAILED; + } + const more = plan.remaining ? ` · ${count(plan.remaining)} more on the next run` : ''; + if (plan.mode === 'rebuild') return `Every preview is regenerated: ${count(plan.targets)} ${noun(plan.targets, 'file')}${more}.`; + return `${count(plan.targets)} ${noun(plan.targets, 'preview')} to generate · ${count(plan.fresh)} already up to date${more}.`; +} + +/** + * Start the thumbnail job with `options` (`{mode, size, quality}`) and open its view; a refused or failed start says + * why and stays. Nothing is said once the section that started it is gone. + */ +export async function startThumbnails({ store, scope, options, notify = showWorkspaceToast, navigate = navigateTo }) { + try { + const result = await store.start(THUMBNAILS_KIND, options); + if (scope.signal.aborted) return; + if (result.busy !== undefined) { + notify(busyToast(result.busy)); + return; + } + navigate(`#/activity/${result.job.id}`); + } catch { + if (!scope.signal.aborted) notify(START_FAILED); + } +} + +function el(tag, text, className) { + const node = document.createElement(tag); + if (text !== undefined) node.textContent = text; + if (className) node.className = className; + return node; +} + +function select(options, value) { + const node = el('select'); + node.append(...options.map(([optionValue, label]) => { + const option = el('option', label); + option.value = optionValue; + return option; + })); + node.value = value; + return node; +} + +const defaultQuality = () => { + try { + return readPreferences().values.imageQuality; + } catch { + return undefined; + } +}; + +/** + * Open "Thumbnail maintenance" on the section's `scope`. `store` is the page's Activity store; `trigger` gets focus + * back when the dialog closes. + */ +export function openThumbnailReview({ api, store, scope, trigger = null, quality = defaultQuality(), notify = showWorkspaceToast, + navigate = navigateTo }) { + const mode = select(SCOPE_OPTIONS, 'missing'); + const encoder = select(QUALITY_OPTIONS, preselectedQuality(quality)); + const size = el('input'); + size.type = 'number'; + size.value = String(SIZE_LIMIT.value); + size.min = String(SIZE_LIMIT.min); + size.max = String(SIZE_LIMIT.max); + const line = el('p', PLAN_LOADING, 'small'); + line.setAttribute('role', 'status'); + const cancel = createButton('Cancel'); + const start = createButton('Start', 'ca-primary'); + const actions = el('div', undefined, 'dialog-actions'); + actions.append(cancel, start); + + // The plan per scope: a promise of its answer (null when it failed); the size and quality change no count. + const plans = new Map(); + let open = true; + const chosenSize = () => previewSize(size.value); + const validate = () => { + const valid = chosenSize() !== null; + size.setCustomValidity(valid ? '' : SIZE_INVALID); + if (valid) size.removeAttribute('aria-invalid'); + else size.setAttribute('aria-invalid', 'true'); + return valid; + }; + const readPlan = chosen => { + const query = new URLSearchParams({ mode: chosen, size: String(chosenSize()), quality: encoder.value }); + return api.get(`${THUMBNAIL_PLAN_URL}?${query}`, { signal: scope.signal, cache: 'no-store' }).catch(() => null); + }; + async function paint() { + if (!validate()) { line.textContent = SIZE_INVALID; return; } + const chosen = mode.value; + if (!plans.has(chosen)) plans.set(chosen, readPlan(chosen)); + const pending = plans.get(chosen); + line.textContent = PLAN_LOADING; + const plan = await pending; + // A later choice painted its own line meanwhile, or the dialog closed. + if (!open || scope.signal.aborted || mode.value !== chosen || !validate()) return; + line.textContent = planText(plan); + } + + scope.listen(mode, 'change', paint); + scope.listen(size, 'input', paint); + scope.listen(cancel, 'click', () => closeWorkspaceDialog()); + scope.listen(start, 'click', () => { + if (!validate()) { + size.reportValidity(); + size.focus(); + return; + } + const options = { mode: mode.value, size: chosenSize(), quality: encoder.value }; + closeWorkspaceDialog(); + startThumbnails({ store, scope, options, notify, navigate }); + }); + openContentDialog({ scope, title: 'Thumbnail maintenance', trigger, + nodes: [field('Scope', mode, 'thumb-mode'), field('Quality', encoder, 'thumb-quality'), + field('Preview size limit', size, 'thumb-limit'), line, actions], + onClose: () => { open = false; } }); + paint(); +} diff --git a/web/js/workspace/workspace.css b/web/js/workspace/workspace.css index 3493342..7c3a5af 100644 --- a/web/js/workspace/workspace.css +++ b/web/js/workspace/workspace.css @@ -24,6 +24,10 @@ main:focus-visible { outline: 2px solid var(--ca-focus); outline-offset: 4px; } .activity-pill:hover { border-color: var(--ca-accent); /* D53 */ } .activity-pill-bar { display: block; inline-size: 70px; block-size: 5px; border-radius: 3px; background: var(--line); overflow: hidden; /* D53 */ } .activity-pill-bar i { display: block; block-size: 100%; inline-size: 0; background: var(--ca-accent); /* D53 */ } +/* D111 (S5): a running job without a known total sweeps a short segment; reduced motion shows a dimmed full bar. */ +.activity-pill-bar.indeterminate i { inline-size: 40%; animation: activity-pill-sweep 1.4s ease-in-out infinite; /* D111 */ } +@keyframes activity-pill-sweep { from { transform: translateX(-100%); } to { transform: translateX(250%); } } +@media (prefers-reduced-motion: reduce) { .activity-pill-bar.indeterminate i { inline-size: 100%; opacity: .45; animation: none; /* D111 */ } } @media (pointer: coarse) { .activity-pill:not([hidden]) { min-block-size: 40px; /* D53 */ } } @media (max-width: 700px) { /* D52: with the Activity button the full-width row can be wider than a zoomed phone, so its items wrap. */