diff --git a/TESTING.md b/TESTING.md index ec95197c..7f2bc2d1 100644 --- a/TESTING.md +++ b/TESTING.md @@ -5899,6 +5899,73 @@ are closed before `os.replace`; renames stay inside one folder; folder identity folder are refused. The browser test server closes its own log handlers before its temp root (which now holds the logs) is removed. +### Thumbnail location contract (#320) + +Previews live in each source's `thumbnails/` (the default) or in one folder, `//…`. Every reader, +writer and remover asks `py/api/preview_store.py`; `tests/test_preview_store.py` fails on a `thumbnails` path join +anywhere else. The setting is `thumbnails.location` in config.json, and only the Move thumbnails job writes it: POST +`/settings` answers 409 `location-needs-move`. + +- **Keys and marker.** A key is a slug plus a hash of the source's identity. The folder's + `.promptmanager-thumbnails.json` stores each key with its identity and install. Only a single `-` name + is ever joined onto the folder, and anything else in a marker is dropped. Every write and delete passes + `preview_store.may_write`: below a key folder, with no link, junction or reparse point on the way + (`tests/test_preview_store_folder.py`, including tampered markers). +- **Unavailable.** A folder is unavailable when it is missing, is a link, holds no marker or holds another + folder's marker. A probe checks it every 5 s, and a store call that fails or stalls marks it at once. While it is + unavailable: + - originals are served and nothing is written anywhere; + - generate, convert, clear and prune answer 409 `location-unavailable`; + - the folder is never made again except by Recreate folder. + + The configured location is in force before any request. A bad value is `invalid`, never a fall-back to the + sources. +- **Move thumbnails.** The job is a `thumbnails` job with `scope.mode` `move` (`tests/test_thumbnail_move.py`). It + runs in this order: + 1. The review answers `POST /images/thumbnails/plan`; not enough space answers 507. + 2. A token starts the job. + 3. Automatic previews are held. + 4. Copies run on the job pool in chunks of 2 or 4 files, each with its size checked and its mtime rounded up to + 2 s. + 5. One config commit makes the switch and records the old stores under `cleanup`. The job cannot be cancelled + from here. + 6. The old copies are removed. + + If it fails or is cancelled before the switch, nothing changes. A crash leaves + `/data/thumbnail-move.json`, which the next start undoes. Old copies still held open are removed by the + next start's "Move thumbnails (finishing)" job. Leaving a folder also removes the previews of this install's + sources that are no longer listed (Q320-8). +- **Remove thumbnails of removed sources** (`scope.mode` `prune`). It is offered in Thumbnail maintenance only in + folder mode, and it touches only this install's keys. +- **Page tests.** `tests/browser/workspace-thumbnail-location.spec.mjs` covers the row, both moves, the in-place + refusal, the confirmation, the notice with Recreate and with Use this folder. The legacy gallery runs once in + folder mode (`workflows.spec.mjs`). The hooks are `tests/browser/thumbnail_location_fixture.py`. + +Beyond the concept, against the approved static mock `docs/egm/post-3.4.1/320-mock`: + +- D196: the Thumbnail location row in System › Sources & scans, after Thumbnail generation. + - The status line is "Now: inside each source." or "Now: one folder, .". It does not show the mock's + "(3 sources, 8,512 thumbnails)" counts, because counting means walking every preview on each view. It says + "Using the default folder next to your library." and names old copies that wait to be removed. +- D197: the Move thumbnails review uses the shared review. + - A grid value may be a lead sentence with a list (`[lead, items]`). + - Sources no longer listed appear under "Removed after the move", where the mock has a "Not moved" row + (maintainer answer to Q320-8: remove). + - The Free space line is " free", without naming the disk. +- D198: Activity's view of `mode: move` and `mode: prune`. + - Tiles for a move: Moved, Already there, Left in place, Errors. Tiles for a prune: Removed, Left in place, + Errors. + - The step line under the bar is `#job-phase`. + - Cancel shows only until the switch. +- D199: the unavailable notice on Overview and Sources & scans. + - Recreate folder appears for a missing folder, after a confirmation. + - Use this folder appears for another folder's marker. + - Change location focuses the row; on Overview it opens Sources & scans. +- D200: Thumbnail maintenance's "Remove thumbnails of removed sources" scope, with the plan line and a grid of + key folders. +- D201: Move library's review adds "Thumbnails: Stay in . Move them in System › Sources & scans." in folder + mode. + ### Backup and restore contract (#249 System S8) A restore is staged, then published (`database/restore_stage.py`). The upload is diff --git a/__init__.py b/__init__.py index d95dfff1..3784f414 100644 --- a/__init__.py +++ b/__init__.py @@ -92,10 +92,21 @@ try: routes = config.routes api = PromptManagerAPI() api.add_routes(routes) - # Temp files of preview writes a crash interrupted (#317), off the - # import path: the walk may cover a large thumbnails folder. + + # The thumbnail location's startup check (#320), then the temp files of + # preview writes a crash interrupted (#317), off the import path: the + # walk may cover a large thumbnails folder. + # #320: the configured thumbnail location is in force before any request + # (no I/O); the thread below checks it. + api.use_preview_location() + + def _preview_startup(): + api.start_preview_location() + api.recover_thumbnail_move() # an interrupted move, before the sweep + api.sweep_thumbnail_temps() + threading.Thread( - target=api.sweep_thumbnail_temps, + target=_preview_startup, name="PromptManagerPreviewTempSweep", daemon=True, ).start() @@ -108,6 +119,8 @@ try: # Build each output root's Images listing once the server's loop runs, # so the first open does not wait for the walk (#348). host_app.on_startup.append(api.warm_gallery_listings) + # #320: old thumbnails a move left to remove, once it is checked. + host_app.on_startup.append(api.finish_thumbnail_move) if host_app is not None and hasattr(host_app, "on_shutdown"): # Open Needs review change streams end before the server waits on them. host_app.on_shutdown.append(api.close_review_events) diff --git a/database/library_move.py b/database/library_move.py index c104ac15..b4451a79 100644 --- a/database/library_move.py +++ b/database/library_move.py @@ -78,6 +78,10 @@ MESSAGES = { ), "same-folder": "The new folder is the node folder itself.", "target-invalid": "The new folder cannot hold the library.", + "data-dir-served": ( + "The new folder is inside Prompt Manager's thumbnail folder. Nothing was " + "moved." + ), "target-has-library": ( "The new folder already holds a Prompt Manager library. Nothing was moved." ), diff --git a/py/api/__init__.py b/py/api/__init__.py index b312328e..6997f686 100644 --- a/py/api/__init__.py +++ b/py/api/__init__.py @@ -11,6 +11,7 @@ Split into domain-specific mixins for maintainability: - JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py) - ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs - ThumbnailJobsMixin: thumbnail plan, thumbnail job and the gallery stream on it +- ThumbnailLocationMixin: the thumbnail location check and payloads (#320) - ThumbnailConvertMixin: Convert outdated thumbnails (#317), a thumbnail job mode - AutoThumbnailMixin: previews made on first view, on capture and after a scan - ReviewRoutesMixin: the Needs review queue, item detail and decisions @@ -60,6 +61,11 @@ from .settings_routes import SettingsRoutesMixin from .job_routes import JobRoutesMixin from .image_tag_jobs import ImageTagJobsMixin from .thumbnail_jobs import ThumbnailJobsMixin +from .thumbnail_location import ThumbnailLocationMixin +from .thumbnail_move import ThumbnailMoveMixin +from .thumbnail_move_plan import ThumbnailMovePlanMixin +from .thumbnail_move_recovery import ThumbnailMoveRecoveryMixin +from .thumbnail_prune import ThumbnailPruneMixin from .thumbnail_convert import ThumbnailConvertMixin from .thumbnail_auto import AutoThumbnailMixin from .system import SystemRoutesMixin @@ -338,6 +344,11 @@ class PromptManagerAPI( JobRoutesMixin, ImageTagJobsMixin, ThumbnailJobsMixin, + ThumbnailLocationMixin, + ThumbnailMovePlanMixin, + ThumbnailMoveMixin, + ThumbnailMoveRecoveryMixin, + ThumbnailPruneMixin, ThumbnailConvertMixin, AutoThumbnailMixin, SystemRoutesMixin, @@ -716,6 +727,7 @@ class PromptManagerAPI( # ── Register domain-specific routes from mixins ─────────────── self._register_pair_routes(routes) + self._register_thumbnail_move_routes(routes) self._register_library_routes(routes) self._register_library_delete_routes(routes) self._register_vocabulary_routes(routes) @@ -852,20 +864,11 @@ class PromptManagerAPI( f"{url_quote(rel_path.as_posix(), safe='/')}" ) - # Any preview the file owns, the current format first (#317) - from . import thumbnail_settings - from .prompts import thumbnail_url_for + # Any preview the file owns, the current format first + # (#317); no store I/O here in folder mode (#320). + from .images import listing_thumbnail_url - existing = thumbnail_settings.existing_thumbnail( - output_path, rel_path - ) - thumbnail_url = None - if existing is not None: - thumb_rel = ( - thumbnail_settings.thumbnail_stem_rel(rel_path) - + existing.suffix - ) - thumbnail_url = thumbnail_url_for(thumb_rel, existing) + thumbnail_url = listing_thumbnail_url(output_path, rel_path) if thumbnail_url: image["thumbnail_url"] = thumbnail_url break # Found matching root, stop searching diff --git a/py/api/autotag_routes.py b/py/api/autotag_routes.py index ee676e87..c3962b7e 100644 --- a/py/api/autotag_routes.py +++ b/py/api/autotag_routes.py @@ -19,8 +19,8 @@ from .jobs import ( ) from .maintenance import disk_plan -from .prompts import thumbnail_url_for -from . import thumbnail_settings +from .images import listing_thumbnail_url +from . import preview_store from .. import model_gate try: @@ -1486,23 +1486,21 @@ class AutotagRoutesMixin: files = await self._run_in_executor( self._without_removed, output_path, files ) - has_thumbnails = (output_path / "thumbnails").is_dir() + location = preview_store.current() + if location.mode == preview_store.FOLDER_MODE: + has_thumbnails = location.available # no store I/O here (#320) + else: + store = preview_store.store_dir(output_path) + has_thumbnails = store is not None and store.is_dir() images = [] for image_path in files: rel_path = image_path.relative_to(output_path) thumbnail_url = None if has_thumbnails: - # Any preview the file owns, the current format first (#317) - existing = thumbnail_settings.existing_thumbnail( - output_path, rel_path - ) - if existing is not None: - thumb_rel = ( - thumbnail_settings.thumbnail_stem_rel(rel_path) - + existing.suffix - ) - thumbnail_url = thumbnail_url_for(thumb_rel, existing) + # Any preview the file owns, the current format first + # (#317); in folder mode without store I/O (#320). + thumbnail_url = listing_thumbnail_url(output_path, rel_path) images.append( { "filename": image_path.name, diff --git a/py/api/history_current.py b/py/api/history_current.py index d8c6b7d6..2a0fa58f 100644 --- a/py/api/history_current.py +++ b/py/api/history_current.py @@ -15,10 +15,10 @@ from typing import BinaryIO from aiohttp import web -from . import thumbnail_settings +from . import preview_store, thumbnail_settings from .images import _canonical, _public_path, SERVABLE_EXTENSIONS, VIDEO_EXTENSIONS from .images import in_trash_below_roots -from .images import existing_media_thumbnail, media_thumbnail_rel +from .images import existing_media_thumbnail, media_preview_path try: from ...utils.execution_outputs import ( @@ -244,19 +244,36 @@ def resolve_current_candidate( return _unavailable("temporary_unavailable", error=True) +def _target_authorized(target, namespace, scope, store_root=None) -> bool: + """A preview in the thumbnail folder (#320, folder mode) is authorized by + the store of its source (``preview_store.owns``: inside the key folder, + no link); everything else by the output namespace and the roots.""" + if ( + store_root is not None + and preview_store.current().mode == preview_store.FOLDER_MODE + ): + return preview_store.owns(store_root, target) + return _authorized(target, namespace, scope) + + def _open_current_target( - target: Path, namespace: Path, scope: CurrentOutputScope, preview: bool = False + target: Path, + namespace: Path, + scope: CurrentOutputScope, + preview: bool = False, + store_root: Path | None = None, ) -> BinaryIO: """Open only the canonical target and verify authority against the actual handle. - A *preview* (a file under ``thumbnails/``) may also be JPEG XL (#317).""" + A *preview* (a file under ``thumbnails/``) may also be JPEG XL (#317); in + folder mode it is authorized by the store of *store_root* (#320).""" if _canonical(target) != target: raise CurrentLocationUnavailable("not_authorized", stale=True) if in_trash_below_roots(target, [namespace]): # Also at the open: the cover chooses its preview after the candidate # check, and a preview can be a link into a trash folder (#543). raise CurrentLocationUnavailable("missing") - if not _authorized(target, namespace, scope): + if not _target_authorized(target, namespace, scope, store_root): raise CurrentLocationUnavailable("not_authorized") suffix = target.suffix.lower() if suffix not in SERVABLE_EXTENSIONS and not ( @@ -289,7 +306,7 @@ def _open_current_target( if ( _canonical(target) != target or (observed.st_dev, observed.st_ino) != (current.st_dev, current.st_ino) - or not _authorized(target, namespace, scope) + or not _target_authorized(target, namespace, scope, store_root) ): raise CurrentLocationUnavailable("not_authorized", stale=True) return os.fdopen(fd, "rb") @@ -326,12 +343,10 @@ class _OpenedCurrentMedia: def _preview_writable(root: Path, preview: Path) -> bool: - """A preview may be written: ``thumbnails`` and every link segment of the - preview's folder resolve inside *root* (the serve route's rule, #249).""" - folder = root / "thumbnails" - return _canonical(folder).is_relative_to(_canonical(root)) and _canonical( - preview - ).is_relative_to(_canonical(folder)) + """A preview may be written: the store of *root* and every link segment + of the preview's folder resolve inside it (the serve route's rule, #249; + ``preview_store.owns``).""" + return preview_store.owns(root, preview) def _is_jxl(path) -> bool: @@ -381,8 +396,12 @@ def _cover_target(resolved, ensure_preview, placeholder, accepts_jxl): return _canonical(preview) if preview is not None: return _canonical(preview) - preview = resolved.gallery_root / media_thumbnail_rel(relative) - if ensure_preview is not None and _preview_writable(resolved.gallery_root, preview): + preview = media_preview_path(resolved.gallery_root, relative, create=True) + if ( + ensure_preview is not None + and preview is not None + and _preview_writable(resolved.gallery_root, preview) + ): ensure_preview(target, preview, is_video) if is_video and placeholder is not None: return _placeholder_media(placeholder) @@ -418,7 +437,13 @@ def _prepare_current_media( return chosen if chosen is not None: target, is_preview = chosen, True - handle = _open_current_target(target, resolved.namespace, scope, is_preview) + handle = _open_current_target( + target, + resolved.namespace, + scope, + is_preview, + resolved.gallery_root if is_preview else None, + ) try: return _OpenedCurrentMedia( handle, diff --git a/py/api/images.py b/py/api/images.py index 7d530555..fd66dc97 100644 --- a/py/api/images.py +++ b/py/api/images.py @@ -34,7 +34,8 @@ from .gallery_listing import ( WalkOutcome, walk_root, ) -from . import thumbnail_settings +from . import preview_store, thumbnail_settings +from .thumbnail_location import location_refusal from .thumbnail_encode import ( PREVIEW_HELD, EncoderClosed, @@ -1068,13 +1069,24 @@ def _iter_media_files(output_path, max_files): def media_thumbnail_rel(rel_path, settings=None): - """``thumbnails/_thumb`` (POSIX) for the media file *rel_path*: - the name the preview gets when it is written now (#317), images and - videos alike (``writer_settings``' extension).""" + """``thumbnails/_thumb`` (POSIX URL text) for the media file + *rel_path*: the name the preview gets when it is written now (#317), + images and videos alike (``writer_settings``' extension). Where it is on + disk is ``media_preview_path``.""" settings = settings or writer_settings() return thumbnail_settings.thumbnail_rel(rel_path, settings.extension) +def media_preview_path(output_path, rel_path, settings=None, *, create=False): + """Where the preview of *rel_path* is written now (``preview_store``); + *create* for a writer (a new source gets its key). None when nothing + may be written (the thumbnail folder is unavailable).""" + settings = settings or writer_settings() + return preview_store.preview_path( + output_path, rel_path, settings.extension, create=create + ) + + def existing_media_thumbnail(output_path, rel_path): """The existing preview *rel_path* owns that a listing or cover shows (the format written now first, then older ones), or None. The Images @@ -1084,6 +1096,27 @@ def existing_media_thumbnail(output_path, rel_path): ) +def listing_thumbnail_url(output_path, rel_path): + """The thumbnail URL of an older listing (``_enrich_images``, the + auto-tag scan), or None. These run on the event loop: in folder mode + (#320) the store is never touched there, so the URL names the preview + as it is written now and the serve route finds or makes it off the loop + (None while the thumbnail folder is unavailable).""" + from .prompts import thumbnail_url_for + + location = preview_store.current() + if location.mode == preview_store.FOLDER_MODE: + if not location.available: + return None + quoted = urllib.parse.quote(media_thumbnail_rel(rel_path), safe="/") + return f"/prompt_manager/images/serve/{quoted}" + existing = thumbnail_settings.existing_thumbnail(output_path, rel_path) + if existing is None: + return None + thumb_rel = thumbnail_settings.thumbnail_stem_rel(rel_path) + existing.suffix + return thumbnail_url_for(thumb_rel, existing) + + def media_thumbnail(output_path, rel_path, root_index, *, root_scope=None): """``(thumbnail_url, ready)`` for *rel_path* under *output_path* (a few blocking stats). The Images listing, the Needs review queue cards and the @@ -2798,13 +2831,56 @@ class ImageRoutesMixin: output_dirs[0], filepath, int(root_param), len(outputs) ) + # Preview text is mapped by the resolver (#320); any other text, + # ``Thumbnails/...`` media on a case-sensitive disk included, + # keeps the plain join. + preview_parts = ( + tuple(filepath.split("/")) + if preview_store.is_preview_text(filepath) + else None + ) + location = preview_store.current() + if location.mode == preview_store.FOLDER_MODE: + if preview_parts is None and await self._run_image( + self._in_source_thumbnails, output_dirs, filepath + ): + # Previews live in the thumbnail folder: anything else + # under a source's thumbnails/ is a leftover (#320), in + # any case on a case-insensitive disk. + return _forbidden() + if preview_parts is not None: + # Store I/O never runs on the event loop (a share may + # stall): the lookup runs on the image pool. + found = await self._run_image( + self._find_preview, output_dirs, preview_parts, location + ) + if found == "forbidden": + return _forbidden() + if found is not None: + return self._file_response(found) for output_path in output_dirs: - image_path = output_path / filepath + if ( + preview_parts is not None + and location.mode == preview_store.FOLDER_MODE + ): + break # looked up above + if preview_parts is None: + image_path = output_path / filepath + else: + image_path = preview_store.request_path(output_path, preview_parts) + if image_path is None: + continue if not image_path.exists(): continue - # A symlink that points outside the root is a hard deny, not - # a fall-through to the next root. - if not _is_within(image_path, output_path): + # A symlink that points outside the root (a preview: outside + # the folder holding its store) is a hard deny, not a + # fall-through to the next root. + container = ( + output_path + if preview_parts is None + else preview_store.store_container(output_path) + ) + if not _is_within(image_path, container): return _forbidden() if in_trash_below_roots(image_path, output_dirs): return _file_not_found() @@ -2830,6 +2906,48 @@ class ImageRoutesMixin: {"success": False, "error": _safe_error(e)}, status=500 ) + @staticmethod + def _in_source_thumbnails(output_dirs, filepath): + """Whether *filepath* lies in a source's ``thumbnails`` folder: its + first part is ``thumbnails`` exactly, or another case of it that is + the same folder (a case-insensitive disk). ``Thumbnails/`` media on + a case-sensitive disk is another folder and stays served.""" + first = filepath.split("/", 1)[0] + if first == "thumbnails": + return True + if first.casefold() != "thumbnails": + return False + for output_path in output_dirs: + spelled = output_path / first + cache = preview_store.source_folder(output_path) + try: + if spelled.is_dir() and cache.is_dir() and spelled.samefile(cache): + return True + except OSError: + continue + return False + + @staticmethod + def _find_preview(output_dirs, parts, location): + """The preview file *parts* names in folder mode, ``"forbidden"`` + (it resolves out of its store or through a link) or None (blocking; + the first root whose store holds it, in ``_select_root`` order).""" + for output_path in output_dirs: + image_path = preview_store.request_path(output_path, parts, location) + if image_path is None: + continue + try: + with preview_store.store_io(): + if not os.path.lexists(image_path): + continue + if not preview_store.owns(output_path, image_path, location): + return "forbidden" + if image_path.is_file(): + return image_path + except OSError: + return None + return None + def _is_recorded_media(self, path) -> bool: """Whether the database records ``path`` as a linked image. @@ -2956,6 +3074,9 @@ class ImageRoutesMixin: data, error = await _json_body_or_error(request) if error is not None: return error + refused = location_refusal() + if refused is not None: + return refused return await self._run_held( THUMBNAILS_JOB_KIND, THUMBNAIL_GENERATION_TITLE, @@ -2988,6 +3109,7 @@ class ImageRoutesMixin: """Blocking thumbnail generation loop (run in executor).""" settings = writer_settings(settings) start = _time.monotonic() + thumbnails_dir = preview_store.given_store(output_path, thumbnails_dir) plan = self._plan_thumbnails(output_path, thumbnails_dir, settings=settings) payload = self._execute_thumbnail_plan(plan, settings, start) payload["thumbnails_path"] = _public_path(thumbnails_dir) @@ -3000,7 +3122,8 @@ class ImageRoutesMixin: start = _time.monotonic() plan = self._plan_gallery_thumbnails(output_dirs, settings=settings) payload = self._execute_thumbnail_plan(plan, settings, start) - payload["thumbnails_path"] = _public_path(output_dirs[0] / "thumbnails") + store = preview_store.store_dir(output_dirs[0]) + payload["thumbnails_path"] = None if store is None else _public_path(store) return payload def _execute_thumbnail_plan(self, plan: dict, settings, start: float) -> dict: @@ -3043,9 +3166,12 @@ class ImageRoutesMixin: if stop is not None and stop(): break root = Path(root) + store = preview_store.ensure_store(root) + if store is None: + continue # the thumbnail folder is unavailable plan = self._plan_thumbnails( root, - root / "thumbnails", + store, mode=mode, stop=stop, settings=settings, @@ -3084,6 +3210,8 @@ class ImageRoutesMixin: "format": writer_settings().format, # Files whose automatic preview failed this session (#249). "failed": self._auto_thumbnail_failures(), + # #320: where previews live (no path here, D6A). + "location": self._thumbnail_location_payload(with_path=False), "collection_id": hashlib.sha256(collection.encode()).hexdigest()[ :16 ], @@ -3129,9 +3257,11 @@ class ImageRoutesMixin: """ settings = writer_settings(settings) output_path = Path(output_path) - thumbnails_dir = Path(thumbnails_dir) + thumbnails_dir = preview_store.given_store(output_path, thumbnails_dir) plan = {key: 0 for key in _PLAN_COUNTS} plan.update(targets=[], fresh_files=[]) + if thumbnails_dir is None: + return plan # the thumbnail folder is unavailable folders = {} removed = self._removal_check_under(output_path) for src in _iter_media_files(output_path, MAX_GALLERY_FILES): @@ -3221,6 +3351,9 @@ class ImageRoutesMixin: "dir": src.parent.name, "type": "video" if is_video else "image", } + # Taken when the write starts: a thumbnail location switched since + # then drops the write at its rename (#320). + guard = preview_store.write_guard(guard) dst = Path(dst) if dst.suffix != settings.extension: dst = dst.with_suffix(settings.extension) @@ -3231,7 +3364,7 @@ class ImageRoutesMixin: return {**result, "action": "error", "error": "File no longer exists"} if not rebuild and _thumbnail_is_fresh(src, dst): return {**result, "action": "skipped", "path": str(dst)} - dst.parent.mkdir(parents=True, exist_ok=True) + preview_store.make_parent(dst) if is_video: made = self._generate_video_thumbnail( src, @@ -3436,8 +3569,12 @@ class ImageRoutesMixin: Invalid thumbnail directories return 400; unavailable gallery roots return 404, and unexpected filesystem failures return 500. Holds the - registry as guarded work, like generation. + registry as guarded work, like generation. Refused (409) while the + thumbnail folder is unavailable (#320). """ + refused = location_refusal() + if refused is not None: + return refused return await self._run_held( THUMBNAILS_JOB_KIND, THUMBNAIL_CLEAR_TITLE, @@ -3501,8 +3638,12 @@ class ImageRoutesMixin: self.logger.debug(f"No roots to sweep for preview temp files: {exc}") return 0 for output_path in output_dirs: - folder = Path(output_path) / "thumbnails" - if folder.is_symlink() or not folder.is_dir(): + folder = preview_store.store_dir(output_path) + if ( + folder is None + or preview_store.store_is_link(folder) + or not folder.is_dir() + ): continue removed += sweep_stray_temps(folder)[0] if removed: @@ -3520,13 +3661,16 @@ class ImageRoutesMixin: directories = [] for output_path in output_dirs: output_path = Path(output_path) - thumbnails_dir = output_path / "thumbnails" + thumbnails_dir = preview_store.store_dir(output_path) + if thumbnails_dir is None: + continue if not thumbnails_dir.exists() and not thumbnails_dir.is_symlink(): continue + container = preview_store.store_container(output_path) if ( - thumbnails_dir.is_symlink() - or not _is_within(thumbnails_dir, output_path) - or _canonical(thumbnails_dir) == _canonical(output_path) + preview_store.store_is_link(thumbnails_dir) + or not _is_within(thumbnails_dir, container) + or _canonical(thumbnails_dir) == _canonical(container) ): raise ValueError( "Safety check failed: invalid thumbnails directory path" @@ -3541,7 +3685,15 @@ class ImageRoutesMixin: temps, temp_bytes = sweep_stray_temps(thumbnails_dir, min_age=0) cleared_count += temps cleared_size += temp_bytes - for root, _dirs, files in os.walk(thumbnails_dir): + folder_mode = preview_store.current().mode == preview_store.FOLDER_MODE + for root, dirs, files in os.walk(thumbnails_dir): + if folder_mode: + # Never into a junction or reparse point (#320). + dirs[:] = [ + d + for d in dirs + if not preview_store.store_is_link(Path(root) / d) + ] for name in files: path = Path(root) / name if ( @@ -3549,6 +3701,7 @@ class ImageRoutesMixin: or path.suffix.lower() not in thumbnail_settings.THUMBNAIL_EXTENSIONS or path.is_symlink() + or not preview_store.may_write(path) ): continue try: diff --git a/py/api/library_location.py b/py/api/library_location.py index 17d172c7..30590c53 100644 --- a/py/api/library_location.py +++ b/py/api/library_location.py @@ -50,6 +50,7 @@ _REFUSAL_STATUS = { "not-movable": 400, "same-folder": 400, "target-invalid": 400, + "data-dir-served": 400, "target-has-library": 409, "target-conflict": 409, "insufficient-space": 507, @@ -74,6 +75,24 @@ def _failure(message, status, code): ) +def _thumbnail_folder(): + """The thumbnail folder in use, or None (previews inside each source).""" + from . import preview_store + + setting = preview_store.setting() + return setting.path if setting.mode == preview_store.FOLDER_MODE else None + + +def _inside_thumbnail_folder(path): + """Whether *path* lies in the thumbnail folder in use (#320).""" + from . import preview_store + + setting = preview_store.setting() + if setting.mode != preview_store.FOLDER_MODE or not setting.path: + return False + return ll._inside_any(path, [setting.path]) + + def _plural(count, one, many=None): return f"{count:,} {one if count == 1 else (many or one + 's')}" @@ -225,6 +244,8 @@ class LibraryLocationRoutesMixin: ), "problem": location.problem or self._node_folder_risk(location), "size_bytes": size, + # #320 D201: thumbnails kept in one folder stay there (D8A). + "thumbnails_folder": _thumbnail_folder(), } # The node folder (tests point it at a temp tree). @@ -272,6 +293,8 @@ class LibraryLocationRoutesMixin: def _plan_library_move(self): """The checked plan (blocking); raises MoveFailed.""" plan = lm.plan_move(self._live_location()) + if _inside_thumbnail_folder(plan.target_dir): + raise lm.MoveFailed("data-dir-served") if lm.space_problem(plan) is not None: raise lm.MoveFailed("insufficient-space") return plan diff --git a/py/api/preview_location.py b/py/api/preview_location.py new file mode 100644 index 00000000..862122ce --- /dev/null +++ b/py/api/preview_location.py @@ -0,0 +1,728 @@ +"""The thumbnail location setting (#320): parsing, validation, source keys +and the thumbnail folder's marker. + +``preview_store`` holds the runtime state and maps paths; this module holds +the rules it applies. Nothing here keeps state. + +Schema (config.json ``thumbnails.location``):: + + {"mode": "folder", "path": "D:\\\\Thumbs", "id": "", + "install": "", "cleanup": []} + +The default (and a missing key) is ``{"mode": "source"}``: previews inside +each source. An invalid value never falls back to the sources: it parses as +a folder location with problem ``invalid``, so nothing is written anywhere. + +Layout in folder mode: ``///_thumb``. +A source key is a readable slug plus a short hash of the source's identity +(``output-3f9a2c1b``), assigned once and stored in the marker +``.promptmanager-thumbnails.json`` with the identity it belongs to. +""" + +import errno +import hashlib +import json +import logging +import ntpath +import os +import posixpath +import re +import stat as stat_module +import threading +import time +import unicodedata +import uuid +from contextlib import contextmanager +from dataclasses import dataclass, field + +try: + from ...utils import library_location +except ImportError: # tests import the package top-level + from utils import library_location + +logger = logging.getLogger("prompt_manager.thumbnails") + +SOURCE_MODE = "source" +FOLDER_MODE = "folder" +MARKER_NAME = ".promptmanager-thumbnails.json" +MARKER_VERSION = 1 +# Beside the marker while one process re-reads, changes and replaces it. +MARKER_LOCK_NAME = ".promptmanager-thumbnails.lock" +MARKER_LOCK_TIMEOUT = 1.5 # a writer waits this long (under preview_store.IO_BUDGET) +MARKER_LOCK_STALE = 30.0 # a lock this old was left by a writer that died +MARKER_LOCK_POLL = 0.02 +# A source key: one folder name, a slug and a 8-16 hex hash (``source_key``). +KEY_PATTERN = re.compile(r"^[a-z0-9_-]{1,24}-[0-9a-f]{8,16}$") +# A marker is a few keys; anything larger is not ours. +MAX_MARKER_BYTES = 4 * 2**20 +# OS metadata files a folder may hold and still count as empty. +OS_METADATA = frozenset({"thumbs.db", "desktop.ini", ".ds_store"}) +KEY_HASH_LENGTHS = (8, 12, 16) +SLUG_LENGTH = 24 +# Folder + the longest key (24 + 1 + 16) + this margin must fit (Windows). +WINDOWS_FOLDER_LIMIT = 247 +LONGEST_KEY = SLUG_LENGTH + 1 + KEY_HASH_LENGTHS[-1] +PATH_MARGIN = 40 +# A validation stat on a share that does not answer in time is unavailable. +VALIDATION_TIMEOUT = 5.0 +FILE_ATTRIBUTE_REPARSE_POINT = 0x400 + +_RESERVED = re.compile(r"^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$", re.IGNORECASE) +_BAD_CHARS = re.compile(r'[<>:"|?*\x00-\x1f]') + +# Problem codes, their HTTP status and the text the UI shows (no path). +MESSAGES = { + "location-not-absolute": ( + 400, + "Enter a full folder path, such as D:\\Thumbnails or /mnt/fast/thumbnails.", + ), + "location-device-path": ( + 400, + "Enter the folder as a drive letter or \\\\server\\share path, " + "without \\\\?\\.", + ), + "location-invalid-name": ( + 400, + "This folder name can't be used on Windows. Choose another name.", + ), + "location-too-broad": ( + 400, + "Choose a folder inside a drive or share, not the drive itself.", + ), + "location-served": ( + 400, + "Choose a folder outside your image sources and ComfyUI's output, input " + "and temp folders.", + ), + "location-node-folder": ( + 400, + "Choose a folder outside the Prompt Manager node folder.", + ), + "location-contains-source": ( + 400, + "This folder contains an image source. Choose a folder beside it instead.", + ), + "location-same": (400, "Thumbnails are already stored here."), + "location-inside-current": ( + 400, + "This folder is inside the thumbnail folder in use. Choose a folder " + "outside it.", + ), + "location-parent-missing": ( + 400, + "The folder that would hold it doesn't exist. Check the path and that its " + "drive is connected.", + ), + "location-unavailable": ( + 409, + "This folder can't be reached right now. Check that its drive or share is " + "connected.", + ), + "location-not-empty": ( + 409, + "This folder already holds other files. Choose an empty folder or one " + "Prompt Manager used before.", + ), + "location-not-writable": ( + 409, + "Prompt Manager can't write to this folder. Check its permissions.", + ), + "location-path-too-long": ( + 400, + "This path is too long for Windows. Choose a shorter folder path.", + ), + "insufficient-space": ( + 507, + "There is not enough space for the thumbnails. Free some space or choose " + "another folder.", + ), + "location-needs-move": ( + 409, + "Use Move thumbnails in System › Sources & scans.", + ), +} +HOLDS_THUMBNAILS_REASON = "This folder holds Prompt Manager's thumbnails." + + +def message(code): + """``(status, text)`` of a problem code.""" + return MESSAGES[code] + + +# ── parsing ────────────────────────────────────────────────────────── + + +@dataclass(frozen=True) +class ParsedLocation: + """A ``thumbnails.location`` value, checked as data (no filesystem).""" + + mode: str = SOURCE_MODE + path: str | None = None + id: str | None = None + install: str | None = None + cleanup: tuple = field(default_factory=tuple) + problem: str | None = None + + def to_config(self): + """The config.json form (``invalid`` values are never written).""" + if self.mode == SOURCE_MODE: + value = {"mode": SOURCE_MODE} + else: + value = {"mode": FOLDER_MODE, "path": self.path, "id": self.id} + if self.install: + value["install"] = self.install + value["cleanup"] = list(self.cleanup) + return value + + +def _cleanup(value): + if not isinstance(value, list): + return () + return tuple(item for item in value if isinstance(item, dict)) + + +def parse_location(value, *, flavor=os.path): + """The location a ``thumbnails.location`` value names. + + Missing: inside each source. Anything else that is not a well-formed + location (an unknown mode, a relative path, a missing id) becomes a + folder location with problem ``invalid``: never a silent fall-back. + """ + if value is None: + return ParsedLocation() + if not isinstance(value, dict): + return ParsedLocation(mode=FOLDER_MODE, problem="invalid") + mode = value.get("mode", SOURCE_MODE) + install = value.get("install") if isinstance(value.get("install"), str) else None + cleanup = _cleanup(value.get("cleanup")) + if mode == SOURCE_MODE: + return ParsedLocation(install=install, cleanup=cleanup) + if mode != FOLDER_MODE: + return ParsedLocation(mode=FOLDER_MODE, problem="invalid", cleanup=cleanup) + path, marker_id = value.get("path"), value.get("id") + if ( + not isinstance(path, str) + or form_problem(path, flavor=flavor) is not None + or not isinstance(marker_id, str) + or not marker_id.strip() + ): + return ParsedLocation( + mode=FOLDER_MODE, + path=path if isinstance(path, str) else None, + problem="invalid", + cleanup=cleanup, + ) + return ParsedLocation( + mode=FOLDER_MODE, + path=flavor.abspath(path), + id=marker_id, + install=install, + cleanup=cleanup, + ) + + +# ── form (text only) ───────────────────────────────────────────────── + + +def _is_windows(flavor): + return flavor is ntpath or getattr(flavor, "__name__", "") == "ntpath" + + +def form_problem(path, *, flavor=os.path): + """The text-only problem of *path*, or None (no filesystem access). + + ``\\\\?\\`` and ``\\\\.\\`` prefixes, relative, drive-relative (``C:x``) + and root-relative (``\\x``) spellings, and names Windows cannot hold + (checked on every OS: the folder may be moved to Windows). + """ + if not isinstance(path, str) or not path.strip() or path != path.lstrip(): + return "location-not-absolute" + if path.replace("/", "\\").startswith(("\\\\?\\", "\\\\.\\")): + return "location-device-path" + if _is_windows(flavor): + drive, rest = ntpath.splitdrive(path) + if not drive or not rest.startswith(("\\", "/")): + return "location-not-absolute" + elif not posixpath.isabs(path): + return "location-not-absolute" + drive, rest = flavor.splitdrive(path) + for part in re.split(r"[\\/]+", rest): + if not part: + continue + if ( + _RESERVED.match(part) + or part.endswith((".", " ")) + or _BAD_CHARS.search(part) + ): + return "location-invalid-name" + return None + + +def is_too_broad(path, *, home=None, flavor=os.path): + """A filesystem or share root, or the home folder itself.""" + normalized = flavor.normpath(path) + drive, rest = flavor.splitdrive(normalized) + if rest in ("", "\\", "/"): + return True + home = home if home is not None else os.path.expanduser("~") + return bool(home) and flavor.normcase(normalized) == flavor.normcase( + flavor.normpath(home) + ) + + +def too_long_for_windows(path, long_paths=None): + """Folder + the longest key + a margin over the folder limit (Windows + without long paths; *long_paths* injects the registry answer).""" + if long_paths is None: + long_paths = long_paths_enabled() + if long_paths: + return False + return len(path) + 1 + LONGEST_KEY + PATH_MARGIN > WINDOWS_FOLDER_LIMIT + + +def long_paths_enabled(): + """``LongPathsEnabled`` on Windows (True on other systems).""" + if os.name != "nt": + return True + try: + import winreg + + with winreg.OpenKey( + winreg.HKEY_LOCAL_MACHINE, + r"SYSTEM\CurrentControlSet\Control\FileSystem", + ) as key: + return winreg.QueryValueEx(key, "LongPathsEnabled")[0] == 1 + except OSError: + return False + + +# ── links ──────────────────────────────────────────────────────────── + + +def is_link_like(path): + """A symlink, a junction or any reparse point (one lstat), or missing. + + Raises OSError other than "not found" (an unreachable share). + """ + try: + info = os.lstat(path) + except FileNotFoundError: + return False + if stat_module.S_ISLNK(info.st_mode): + return True + attributes = getattr(info, "st_file_attributes", 0) + if attributes & FILE_ATTRIBUTE_REPARSE_POINT: + return True + is_junction = getattr(os.path, "isjunction", None) + return bool(is_junction and is_junction(path)) + + +# ── source keys ────────────────────────────────────────────────────── + + +def slug(name): + """``NFKD`` to ASCII, lower case, runs outside ``[a-z0-9_]`` as ``-``, + trimmed and cut to 24 characters; empty becomes ``source``.""" + text = unicodedata.normalize("NFKD", str(name)) + text = text.encode("ascii", "ignore").decode("ascii").lower() + text = re.sub(r"[^a-z0-9_]+", "-", text).strip("-") + text = text[:SLUG_LENGTH].strip("-") + return text or "source" + + +def source_key(spelling, identity, taken): + """The key of a source: ``slug(basename) + "-" + sha256(identity)[:n]``. + + *taken* maps existing keys to their identity; a key taken by another + identity grows its hash to 12, then 16 characters. + """ + base = slug(ntpath.basename(str(spelling).rstrip("\\/"))) + digest = hashlib.sha256(str(identity).encode("utf-8")).hexdigest() + for length in KEY_HASH_LENGTHS: + key = f"{base}-{digest[:length]}" + owner = taken.get(key) + if owner is None or owner == identity: + return key + raise ValueError("no free source key") + + +def file_identity(path): + """``[st_dev, st_ino]`` of an existing folder, or None (no file ids).""" + try: + info = os.stat(path) + except (OSError, ValueError): + return None + if not info.st_ino: + return None + return [info.st_dev, info.st_ino] + + +def find_key(keys, spelling, identity, dev_ino): + """The marker key of a source: the same configured spelling, then the + same identity text, then the same file id; or None.""" + keys = {k: v for k, v in keys.items() if valid_key(k) and isinstance(v, dict)} + for key, entry in keys.items(): + if entry.get("path") == spelling: + return key + for key, entry in keys.items(): + if entry.get("identity") == identity: + return key + if dev_ino: + for key, entry in keys.items(): + if entry.get("dev_ino") == list(dev_ino): + return key + return None + + +# ── the marker ─────────────────────────────────────────────────────── + + +def marker_path(folder): + return os.path.join(folder, MARKER_NAME) + + +def read_marker(folder): + """The marker in *folder* as a dict, or None (absent or unreadable as + ours). OSError other than "not found" propagates (an unreachable share).""" + try: + with open(marker_path(folder), "r", encoding="utf-8") as handle: + text = handle.read(MAX_MARKER_BYTES + 1) + except FileNotFoundError: + return None + except (ValueError, UnicodeDecodeError): + return None + if len(text) > MAX_MARKER_BYTES: + return None + try: + data = json.loads(text) + except ValueError: + return None + if not isinstance(data, dict) or not isinstance(data.get("id"), str): + return None + data["keys"] = valid_keys(data.get("keys")) + return data + + +def valid_key(key): + """Whether *key* can name a source's folder: one folder name of the + ``source_key`` form, never a separator, ``..`` or a drive.""" + return isinstance(key, str) and KEY_PATTERN.fullmatch(key) is not None + + +def valid_keys(keys): + """The marker's keys that are safe to join onto the folder; a hand-edited + or foreign entry that is not is dropped (and logged), never followed.""" + if not isinstance(keys, dict): + return {} + kept = {} + for key, entry in keys.items(): + if valid_key(key) and isinstance(entry, dict): + kept[key] = entry + else: + logger.warning( + "Ignoring a thumbnail folder marker entry that is not a source " "key" + ) + return kept + + +@contextmanager +def _marker_lock(folder): + """Hold the folder's marker lock: a file made with ``O_CREAT | O_EXCL`` + (atomic on local disks and SMB/NFS shares, unlike ``flock``), so installs + sharing the folder change the marker one at a time. A lock older than + MARKER_LOCK_STALE was left by a writer that died and is taken over; + waiting longer than MARKER_LOCK_TIMEOUT raises TimeoutError.""" + lock = os.path.join(folder, MARKER_LOCK_NAME) + deadline = time.monotonic() + MARKER_LOCK_TIMEOUT + while True: + try: + handle = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644) + except FileExistsError: + _break_stale_lock(lock) + if time.monotonic() >= deadline: + raise TimeoutError( + errno.ETIMEDOUT, "the thumbnail folder marker is locked", lock + ) + time.sleep(MARKER_LOCK_POLL) + continue + try: + os.write(handle, f"{os.getpid()}\n".encode("ascii")) + finally: + os.close(handle) # closed before any unlink (Windows) + break + try: + yield + finally: + try: + os.unlink(lock) + except FileNotFoundError: + pass # taken over as stale by another writer: nothing to release + + +def _break_stale_lock(lock): + """Remove *lock* when it is older than MARKER_LOCK_STALE (its writer + died); a lock that changed since it was looked at is left alone.""" + try: + before = os.stat(lock) + except FileNotFoundError: + return + if time.time() - before.st_mtime < MARKER_LOCK_STALE: + return + logger.warning(f"Taking over a stale thumbnail folder lock: {lock}") + try: + after = os.stat(lock) + if (after.st_ino, after.st_mtime_ns) == (before.st_ino, before.st_mtime_ns): + os.unlink(lock) + except FileNotFoundError: + pass # another writer took it over first + + +def write_marker(folder, mutate): + """Re-read the marker, apply ``mutate(marker)`` and write it atomically + (a temp sibling renamed over it), all under the folder's marker lock so a + second install sharing the folder never drops this one's change (and + the other way round). Callers hold ``preview_store.write_lock`` (threads + of this process). Returns the marker written; TimeoutError when another + writer holds the lock too long.""" + with _marker_lock(folder): + return _write_marker_locked(folder, mutate) + + +def _write_marker_locked(folder, mutate): + current = read_marker(folder) or {} + marker = { + "version": MARKER_VERSION, + "id": current.get("id"), + "created_by_pm": bool(current.get("created_by_pm", False)), + "keys": dict(current.get("keys") or {}), + } + mutate(marker) + if not marker.get("id"): + raise ValueError("a marker needs an id") + target = marker_path(folder) + temp = os.path.join( + folder, f".{MARKER_NAME}.{os.getpid()}-{threading.get_ident()}.tmp" + ) + try: + with open(temp, "w", encoding="utf-8") as handle: + json.dump(marker, handle, indent=2, sort_keys=True) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temp, target) + finally: + try: + os.unlink(temp) + except OSError: + pass # gone with the rename; a leftover is swept with the temps + return marker + + +def new_id(): + return str(uuid.uuid4()) + + +def init_folder(folder, *, created_by_pm=False): + """The marker id of *folder*: an existing marker is adopted; a folder + without one gets a new marker (new id, no keys).""" + existing = read_marker(folder) + if existing is not None: + return existing["id"] + + def create(marker): + if not marker.get("id"): + marker["id"] = new_id() + marker["created_by_pm"] = created_by_pm + + return write_marker(folder, create)["id"] + + +def folder_state(path, expected_id): + """``(available, problem, marker)`` of a configured thumbnail folder. + + Available: the folder exists, is a directory and not a link, junction or + reparse point, and holds our marker with *expected_id*. Raises OSError + when the folder cannot be reached (an I/O error other than not found). + """ + try: + info = os.lstat(path) + except FileNotFoundError: + return False, "missing", None + except OSError as error: + if error.errno in (errno.ENOENT, errno.ENOTDIR): + return False, "missing", None + raise + if is_link_like(path): + return False, "invalid", None + if not stat_module.S_ISDIR(info.st_mode): + return False, "invalid", None + marker = read_marker(path) + if marker is None: + return False, "missing", None + if marker["id"] != expected_id: + return False, "marker-mismatch", marker + return True, None, marker + + +def holds_only_metadata(path): + """Whether an existing folder is empty but for OS metadata files.""" + with os.scandir(path) as entries: + for entry in entries: + name = entry.name.lower() + if name in OS_METADATA or name.startswith("._"): + continue + return False + return True + + +# ── validation ─────────────────────────────────────────────────────── + + +@dataclass(frozen=True) +class Context: + """What a location is checked against (gathered by the caller).""" + + roots: tuple = () # every image source of every root list + contained: tuple = () # ComfyUI output/input/temp and the library folder + node_folder: str | None = None + default_path: str | None = None + current_path: str | None = None # the folder in use (folder mode) + home: str | None = None + flavor: object = os.path + long_paths: bool | None = None + environ: object = None + # ComfyUI's folder_paths module (None: none; unset: imported). + folder_paths: object = library_location._UNSET + + +def _timed(func, timeout): + """``func()`` on a daemon thread; raises TimeoutError past *timeout*. + A stuck call is left to finish on its own thread.""" + result = {} + done = threading.Event() + + def run(): + try: + result["value"] = func() + except BaseException as error: # _timed() re-raises it in the caller + result["error"] = error + finally: + done.set() + + threading.Thread(target=run, name="pm-preview-stat", daemon=True).start() + if not done.wait(timeout): + raise TimeoutError("the folder did not answer in time") + if "error" in result: + raise result["error"] + return result.get("value") + + +def _inside(path, folders): + return library_location._inside_any(path, folders) + + +def _same(first, second): + return library_location.same_folder(first, second) or os.path.normcase( + os.path.realpath(first) + ) == os.path.normcase(os.path.realpath(second)) + + +def _served(path, context): + environ = os.environ if context.environ is None else context.environ + return ( + library_location.data_dir_problem( + path, + folder_paths=context.folder_paths, + environ=environ, + gallery_roots=list(context.roots), + ) + == "data-dir-served" + ) + + +def _probe_write(path): + """Create and delete a probe file in *path*; False when refused.""" + probe = os.path.join(path, f".pm-probe-{uuid.uuid4().hex[:8]}.tmp") + try: + with open(probe, "wb") as handle: + handle.write(b"") + except OSError: + return False + try: + os.unlink(probe) + except OSError: + return False + return True + + +def location_problem(path, *, purpose, context, timeout=VALIDATION_TIMEOUT): + """Why *path* cannot be the thumbnail folder, as a code, or None. + + *purpose*: ``move`` (every rule), ``startup`` (no ``same``, ``empty``, + ``writable`` or parent rules: availability covers them) or ``recreate`` + (the folder is missing and is made again). + """ + flavor = context.flavor + code = form_problem(path, flavor=flavor) + if code is not None: + return code + path = flavor.abspath(path) + default = bool(context.default_path) and flavor.normcase(path) == flavor.normcase( + flavor.abspath(context.default_path) + ) + if is_too_broad(path, home=context.home, flavor=flavor): + return "location-too-broad" + try: + return _timed( + lambda: _filesystem_problem(path, purpose, context, default), timeout + ) + except (TimeoutError, OSError): + return "location-unavailable" + + +def _filesystem_problem(path, purpose, context, default): + if _served(path, context): + return "location-served" + if context.node_folder and not default and _inside(path, [context.node_folder]): + return "location-node-folder" + inner = list(context.roots) + list(context.contained) + if any(folder and _inside(folder, [path]) for folder in inner): + if not default: + return "location-contains-source" + if ( + purpose == "move" + and context.current_path + and ( + os.path.normcase(path) == os.path.normcase(context.current_path) + or (os.path.isdir(path) and _same(path, context.current_path)) + ) + ): + return "location-same" + if ( + purpose == "move" + and context.current_path + and _inside(path, [context.current_path]) + ): + # The old key folders are removed after the move, and with them a + # target nested inside one of them. + return "location-inside-current" + # Windows, or a test injecting its limit (``long_paths=False``). + if os.name == "nt" or context.long_paths is False: + if too_long_for_windows(path, context.long_paths): + return "location-path-too-long" + if purpose == "startup": + return None + if is_link_like(path): + # Never through a link: the folder itself must be a real folder. + return "location-not-empty" + exists = os.path.isdir(path) + if not exists: + if os.path.lexists(path): + return "location-not-empty" + if not default and not os.path.isdir(os.path.dirname(path)): + return "location-parent-missing" + return None + if read_marker(path) is None and not holds_only_metadata(path): + return "location-not-empty" + if not _probe_write(path): + return "location-not-writable" + return None diff --git a/py/api/preview_store.py b/py/api/preview_store.py new file mode 100644 index 00000000..e3709eac --- /dev/null +++ b/py/api/preview_store.py @@ -0,0 +1,706 @@ +"""Where previews live on disk (#320): the one resolver. + +Every mapping between an image source (a "root"), one of its media files and +that file's preview on disk goes through this module. URL text stays in +``thumbnail_settings`` (``thumbnail_rel``: ``thumbnails/_thumb``, +the shape the serve route, ``inspection-state.mjs`` and ``admin.js`` share); +this module answers where such a preview is on disk. + +The store of a source is where its previews live. With the default location +(mode ``source``, "Inside each source") it is ``/thumbnails``, exactly +the folder every caller joined by hand before, so nothing moves. + +Writes and the switch: every preview writer takes ``write_guard`` when it +starts (it records the location's ``generation``); the final rename runs +under ``write_lock`` and only when the generation is unchanged and no hold +applies (``thumbnail_encode.replace_unless_held``). Changing the location +bumps the generation under the same lock, so no write that began before can +land in the old store afterwards. Nothing reads or commits config while +``write_lock`` is held. The hold check runs under it too, so hold checks take +no lock (the job registry's ``running()`` is a lock-free snapshot) and +``bump_generation`` is never called with a registry or config lock held. + +All functions are blocking: call them off the event loop. +""" + +import errno +import logging +import os +import threading +import time +from contextlib import contextmanager +from dataclasses import dataclass, replace +from pathlib import Path + +from . import preview_location, thumbnail_settings + +try: + from ...utils.media_folders import PREVIEW_FOLDER +except ImportError: # tests import the package top-level + from utils.media_folders import PREVIEW_FOLDER + +logger = logging.getLogger("prompt_manager.thumbnails") + +SOURCE_MODE = preview_location.SOURCE_MODE +FOLDER_MODE = preview_location.FOLDER_MODE +# The probe checks the folder this often, and a check (or a store call) +# taking longer than the budget marks it unreachable. +PROBE_INTERVAL = 5.0 +IO_BUDGET = 2.0 +PROBE_THREAD = "pm-preview-probe" + + +@dataclass(frozen=True) +class Location: + """One snapshot of the thumbnail location (take one per request or job).""" + + mode: str = SOURCE_MODE + path: str | None = None + available: bool = True + problem: str | None = None + generation: int = 0 + + +write_lock = threading.Lock() +_generation = 0 +# The runtime location: the parsed setting, availability and the marker keys +# last read. Swapped as a whole; readers take one snapshot. +_state_lock = threading.Lock() +_parsed = preview_location.ParsedLocation() +_available = True +_problem = None +_keys = {} +_key_cache = {} +_clock = time.monotonic + + +def current(): + """The thumbnail location now.""" + with _state_lock: + parsed, available, problem, generation_now = ( + _parsed, + _available, + _problem, + _generation, + ) + if parsed.mode == SOURCE_MODE: + return Location(generation=generation_now) + return Location(FOLDER_MODE, parsed.path, available, problem, generation_now) + + +def setting(): + """The parsed setting in use (``preview_location.ParsedLocation``).""" + return _parsed + + +def generation(): + return _generation + + +def bump_generation(): + """Invalidate every write that started before (under ``write_lock``).""" + global _generation + with write_lock: + _generation += 1 + return _generation + + +def _canonical(path): + """Resolved and case-folded, so paths compare on every OS.""" + return Path(os.path.normcase(os.path.realpath(str(path)))) + + +def _within(path, folder): + return _canonical(path).is_relative_to(_canonical(folder)) + + +# ── the setting and availability ───────────────────────────────────── + + +def configure(value, *, context=None, check=True): + """Use the ``thumbnails.location`` *value* (config.json) from now on. + + A folder location is validated against *context* (``startup`` rules: a + problem other than an unreachable folder makes it ``invalid``), then + checked once; the probe keeps checking it. Swapped under ``write_lock`` + with a generation bump, so no write that started before lands. + """ + global _parsed, _available, _problem, _generation + parsed = preview_location.parse_location(value) + problem = parsed.problem + if parsed.mode == FOLDER_MODE and problem is None and context is not None: + code = preview_location.location_problem( + parsed.path, purpose="startup", context=context + ) + if code == "location-unavailable": + problem = "unreachable" + elif code is not None: + problem = "invalid" + parsed = preview_location.ParsedLocation( + mode=FOLDER_MODE, + path=parsed.path, + id=parsed.id, + install=parsed.install, + cleanup=parsed.cleanup, + problem="invalid", + ) + with write_lock: + with _state_lock: + _parsed = parsed + _available = parsed.mode == SOURCE_MODE + _problem = problem if parsed.mode == FOLDER_MODE else None + _keys.clear() + _key_cache.clear() + _generation += 1 + if parsed.mode == FOLDER_MODE: + if check and parsed.problem is None: + refresh() + _probe.start() + else: + _probe.stop() + return current() + + +def refresh(): + """Check the folder now (bounded by IO_BUDGET); returns availability.""" + parsed = _parsed + if parsed.mode != FOLDER_MODE or parsed.problem is not None: + return parsed.mode == SOURCE_MODE + try: + available, problem, marker = _probe.check(parsed.path, parsed.id) + except (OSError, TimeoutError, RuntimeError): + available, problem, marker = False, "unreachable", None + _set_state(parsed, available, problem, marker) + return available + + +def _set_keys(parsed, marker): + """Take the keys of a marker just written (availability unchanged).""" + with _state_lock: + if _parsed is not parsed: + return + keys = dict((marker or {}).get("keys") or {}) + if keys != _keys: + _keys.clear() + _keys.update(keys) + _key_cache.clear() + + +def set_cleanup(entries): + """The setting's ``cleanup`` after a removal was committed.""" + global _parsed + with _state_lock: + _parsed = replace(_parsed, cleanup=tuple(entries)) + + +def mark_invalid(): + """The setting cannot be used (a startup check that failed): folder + mode stays in force, unavailable, so nothing is written anywhere.""" + global _parsed, _available, _problem + with _state_lock: + if _parsed.mode != FOLDER_MODE: + _parsed = preview_location.ParsedLocation( + mode=FOLDER_MODE, problem="invalid" + ) + else: + _parsed = replace(_parsed, problem="invalid") + _available, _problem = False, "invalid" + + +def _set_state(parsed, available, problem, marker): + global _available, _problem + with _state_lock: + if _parsed is not parsed: + return # the setting changed meanwhile + _available, _problem = available, problem + keys = dict((marker or {}).get("keys") or {}) + if keys != _keys: + _keys.clear() + _keys.update(keys) + _key_cache.clear() + + +def note_failure(): + """A store call failed or stalled: unreachable until the probe finds the + folder again (requests do no store I/O meanwhile).""" + global _available, _problem + if _parsed.mode != FOLDER_MODE: + return + with _state_lock: + _available, _problem = False, "unreachable" + _probe.wake() + + +@contextmanager +def store_io(): + """Around a store call in folder mode: an I/O error (other than a file + that is not there) or a call over IO_BUDGET marks the folder + unreachable.""" + if _parsed.mode != FOLDER_MODE: + yield + return + started = _clock() + try: + yield + except OSError as error: + if not isinstance(error, (FileNotFoundError, NotADirectoryError)): + note_failure() + raise + finally: + if _clock() - started > IO_BUDGET: + note_failure() + + +class _Probe: + """The background check of the thumbnail folder (one thread, one check + at a time; a stuck check is left to finish on its own thread).""" + + def __init__(self): + self._lock = threading.Lock() + self._wake = threading.Event() + self._stop = threading.Event() + self._thread = None + self._checking = None + + def check(self, path, expected_id): + """``folder_state`` within IO_BUDGET; raises TimeoutError. + + The answer comes from a check that started after this call: one + already running may have looked before what the caller just did, so + it is waited for (never run twice at once; a stuck one times out here) + and followed by a new one.""" + key = (path, expected_id) + with self._lock: + earlier = self._running(key) + pending = None if earlier else self._start_check(path, expected_id) + if earlier: + self._wait(earlier) + with self._lock: + # One another caller started meanwhile is new enough to join. + pending = self._running(key) or self._start_check(path, expected_id) + self._wait(pending) + with self._lock: + if self._checking is pending: + self._checking = None + if "error" in pending[2]: + raise pending[2]["error"] + return pending[2]["value"] + + def _running(self, key): + """The check of *key* still running (lock held), else None.""" + pending = self._checking + if pending is None or pending[0] != key or pending[1].is_set(): + return None + return pending + + @staticmethod + def _wait(pending): + if not pending[1].wait(IO_BUDGET): + raise TimeoutError("the thumbnail folder did not answer in time") + + def _start_check(self, path, expected_id): + """Start one check on its own thread (lock held).""" + result = {} + done = threading.Event() + + def run(): + try: + result["value"] = preview_location.folder_state(path, expected_id) + except BaseException as error: # check() re-raises it in the caller + result["error"] = error + finally: + done.set() + + pending = ((path, expected_id), done, result) + self._checking = pending + threading.Thread(target=run, name=PROBE_THREAD + "-check", daemon=True).start() + return pending + + def start(self): + with self._lock: + if self._thread is not None and self._thread.is_alive(): + return + # Each thread has its own stop: a stopped one never resumes. + stop = self._stop = threading.Event() + thread = threading.Thread( + target=self._run, args=(stop,), name=PROBE_THREAD, daemon=True + ) + try: + thread.start() + except RuntimeError: + return # refused at interpreter exit: no probe + self._thread = thread + + def _run(self, stop): + while not stop.is_set(): + self._wake.wait(PROBE_INTERVAL) + self._wake.clear() + if stop.is_set(): + return + try: + refresh() + except Exception: # one failed check never stops the probe + logger.debug("Thumbnail folder check failed", exc_info=True) + + def wake(self): + self._wake.set() + + def stop(self, timeout=2.0): + """Stop the probe; True when its thread ended.""" + with self._lock: + thread, self._thread = self._thread, None + self._stop.set() + self._wake.set() + if thread is None or thread is threading.current_thread(): + return True + thread.join(timeout) + return not thread.is_alive() + + +_probe = _Probe() + + +def stop_probe(timeout=2.0): + """Stop the folder probe (the package teardown, tests).""" + return _probe.stop(timeout) + + +# ── stores ─────────────────────────────────────────────────────────── + + +def source_folder(root): + """``/thumbnails``: the store in source mode, a leftover in + folder mode.""" + return Path(root) / PREVIEW_FOLDER + + +def store_container(root, location=None): + """The folder a store of *root* must stay inside: the source itself, or + the thumbnail folder in folder mode.""" + location = location or current() + if location.mode == FOLDER_MODE: + return Path(location.path) if location.path else None + return Path(root) + + +def _identity(root): + return os.path.normcase(os.path.realpath(str(root))) + + +def _lookup_key(root): + spelling = str(root) + with _state_lock: + if spelling in _key_cache: + return _key_cache[spelling] + keys = dict(_keys) + key = preview_location.find_key( + keys, + spelling, + _identity(root), + preview_location.file_identity(root), + ) + if key is not None: + with _state_lock: + _key_cache[spelling] = key + return key + + +def store_dir(root, location=None): + """The store of *root*: ``/thumbnails``, or in folder mode + ``/`` (None while the folder is unavailable or before the + source has a key: a reader then finds no preview).""" + location = location or current() + if location.mode == SOURCE_MODE: + return Path(root) / PREVIEW_FOLDER + if not location.available or not location.path: + return None + key = _lookup_key(root) + return None if key is None else _key_folder(location.path, key) + + +def _key_folder(folder, key): + """``/``, only for a key that is one folder name directly + below the thumbnail folder (a marker is data, never a path).""" + if not preview_location.valid_key(key): + return None + store = Path(folder) / key + if store.parent != Path(folder) or store.name != key: + return None + return store + + +def ensure_store(root, location=None, *, create=False): + """``store_dir`` for a writer: in folder mode a source without a key gets + one (recorded in the marker); with *create* its key folder is made (never + the thumbnail folder itself). None while the folder is unavailable, and + for a key folder that is a link, junction or reparse point: nothing is + planned, written or removed through it.""" + location = location or current() + store = store_dir(root, location) + if location.mode == SOURCE_MODE or (store is None and not location.available): + return store + if store is None: + store = _assign_key(root, location) + if store is None: + return None + try: + with store_io(): + linked = preview_location.is_link_like(store) + except OSError: + return None + if linked: + logger.warning(f"Skipping the thumbnail folder {store}: it is a link") + return None + if create: + try: + with store_io(): + os.mkdir(store) + except FileExistsError: + pass # already there: another writer made the key folder + except OSError: + return None + return store + + +def _assign_key(root, location): + parsed = _parsed + if location.path != parsed.path: + return None + spelling = str(root) + identity = _identity(root) + dev_ino = preview_location.file_identity(root) + assigned = {} + + def add(marker): + if marker.get("id") != parsed.id: + raise ValueError("another folder's marker") # marker-mismatch + keys = marker["keys"] + key = preview_location.find_key(keys, spelling, identity, dev_ino) + if key is None: + taken = {k: v.get("identity") for k, v in keys.items()} + key = preview_location.source_key(spelling, identity, taken) + keys[key] = { + "path": spelling, + "identity": identity, + "dev_ino": dev_ino, + "install": parsed.install, + } + assigned["key"] = key + + try: + with write_lock, store_io(): + marker = preview_location.write_marker(location.path, add) + except (OSError, ValueError): + return None + _set_keys(parsed, marker) + return _key_folder(location.path, assigned["key"]) + + +def may_write(path, location=None): + """The one ownership check every preview write and delete passes. + + Source mode: True (the callers' own root rules apply, as before). Folder + mode: *path* lies below a source key folder of the thumbnail folder, + lexically and resolved, and no link, junction or reparse point sits + anywhere from the key folder down to it (the file itself included). + """ + location = location or current() + if location.mode == SOURCE_MODE: + return True + if not location.path: + return False + folder = Path(os.path.abspath(location.path)) + path = Path(os.path.abspath(path)) + try: + relative = path.relative_to(folder) + except ValueError: + return False + if len(relative.parts) < 2 or ".." in relative.parts: + return False + store = _key_folder(folder, relative.parts[0]) + if store is None or not _within(path, folder): + return False + try: + return _no_link_below(store, path) + except OSError: + note_failure() + return False + + +def make_parent(dst, location=None): + """Make the folders a preview *dst* is written into. In folder mode only + folders below the thumbnail folder are made, never the folder itself, + and only while it holds this install's marker (an unplugged drive or an + unmounted mount point raises FileNotFoundError); a *dst* that does not + pass ``may_write`` raises PermissionError.""" + location = location or current() + dst = Path(dst) + if location.mode == SOURCE_MODE or not location.path: + if location.mode != SOURCE_MODE: + raise FileNotFoundError(errno.ENOENT, "thumbnail folder unavailable") + dst.parent.mkdir(parents=True, exist_ok=True) + return + folder = Path(os.path.abspath(location.path)) + with store_io(): + if not location.available or not folder.is_dir(): + raise FileNotFoundError(errno.ENOENT, "thumbnail folder unavailable") + marker = preview_location.read_marker(str(folder)) + if marker is None or marker.get("id") != _parsed.id: + note_failure() + raise FileNotFoundError(errno.ENOENT, "thumbnail folder unavailable") + if not may_write(dst, location): + raise PermissionError(errno.EACCES, "outside the thumbnail folder") + current_folder = folder + for part in Path(os.path.abspath(dst.parent)).relative_to(folder).parts: + current_folder = current_folder / part + try: + os.mkdir(current_folder) + except FileExistsError: + pass # already there: another writer made this folder + if preview_location.is_link_like(current_folder): + raise PermissionError(errno.EACCES, "a link in the thumbnail folder") + + +def store_is_link(store, location=None): + """Whether a store folder is a link (a junction or reparse point too in + folder mode): it is skipped by sweeps and refused by clears.""" + location = location or current() + if location.mode == SOURCE_MODE: + return Path(store).is_symlink() + return preview_location.is_link_like(store) + + +def given_store(root, store): + """A caller's explicit store argument: the default spelling + (``/thumbnails``) means the resolver's store; another folder (only + tests pass one) is used as given.""" + store = Path(store) + if store == Path(root) / PREVIEW_FOLDER: + return store_dir(root) + return store + + +def preview_path(root, media_rel, ext=None, location=None, *, create=False): + """Where the preview of *media_rel* (relative to *root*) is, or is to be + written (*create*: a writer, which gives a new source its key), in *ext* + (the extension written now when None).""" + if create: + store = ensure_store(root, location) + else: + store = store_dir(root, location) + if store is None: + return None + if ext is None: + from .thumbnail_encode import writer_settings + + ext = writer_settings().extension + return store / (thumbnail_settings.preview_stem(media_rel) + ext) + + +def request_path(root, parts, location=None): + """The disk path of the URL parts ``("thumbnails", ...)``, or None when + *parts* is not under an exact ``thumbnails`` first part or the store has + no place for it. The text is checked before (``is_preview_text``).""" + parts = tuple(parts) + if len(parts) < 2 or parts[0] != PREVIEW_FOLDER: + return None + store = store_dir(root, location) + if store is None: + return None + return store.joinpath(*parts[1:]) + + +def _no_link_below(store, path): + """No folder from *store* down to *path* (and *store* itself) is a link, + junction or reparse point (lexical walk, one lstat each).""" + store = Path(os.path.abspath(store)) + path = Path(os.path.abspath(path)) + try: + relative = path.relative_to(store) + except ValueError: + return False + current = store + for part in ("",) + relative.parts: + current = current / part if part else current + if preview_location.is_link_like(current): + return False + return True + + +def owns(root, path, location=None): + """Whether *path* lies inside the store of *root*, resolved: the store + stays inside its container and no link takes *path* out of the store. + In folder mode no link, junction or reparse point may sit anywhere from + the store down to *path*.""" + location = location or current() + store = store_dir(root, location) + if store is None: + return False + container = store_container(root, location) + if not (_within(store, container) and _within(path, store)): + return False + if location.mode == SOURCE_MODE: + return True + return may_write(path, location) + + +# ── preview text ───────────────────────────────────────────────────── + + +def is_preview_text(filepath): + """URL text naming a preview: ``thumbnails/<...>/_thumb``. + + Text only, before any join: no backslash, drive or stream colon, and no + empty, ``.`` or ``..`` segment; ``thumbnails`` exactly as the first part + and nowhere else among the folders; the extension, in any case, one a + preview may carry (``X_thumb.JPG`` of an older release included). + """ + if not filepath or "\\" in filepath or ":" in filepath: + return False + parts = filepath.split("/") + if len(parts) < 2 or parts[0] != PREVIEW_FOLDER: + return False + if any(part in ("", ".", "..") for part in parts): + return False + if PREVIEW_FOLDER in parts[1:-1]: + return False + stem, extension = os.path.splitext(parts[-1]) + return ( + stem.endswith(thumbnail_settings.THUMB_SUFFIX) + and len(stem) > len(thumbnail_settings.THUMB_SUFFIX) + and extension.lower() in thumbnail_settings.THUMBNAIL_EXTENSIONS + ) + + +# ── writes ─────────────────────────────────────────────────────────── + + +class _WriteGuard: + """``guard()`` for one preview write: true while the location has not + changed since the write started and *hold* (if any) allows it.""" + + __slots__ = ("generation", "_hold") + + def __init__(self, hold=None): + self.generation = _generation + self._hold = hold + + def __call__(self): + if self.generation != _generation: + return False + return self._hold is None or bool(self._hold()) + + +def write_guard(hold=None): + """Take when a preview write starts; hand to its final rename.""" + if isinstance(hold, _WriteGuard): + return hold + return _WriteGuard(hold) + + +@contextmanager +def committed(guard=None): + """Around a preview's final rename: holds ``write_lock`` and yields + whether the write may land (``guard()``, re-checked under the lock).""" + with write_lock: + yield guard is None or bool(guard()) diff --git a/py/api/settings_routes.py b/py/api/settings_routes.py index 4cccf0ae..df40d79b 100644 --- a/py/api/settings_routes.py +++ b/py/api/settings_routes.py @@ -30,6 +30,7 @@ from . import thumbnail_settings from .admin import _current_image_monitor from .job_routes import busy_response from .jobs import THUMBNAILS_JOB_KIND, JobBusy +from .thumbnail_location import needs_move_response try: from ...utils.validators import validate_result_timeout, validate_worker_threads @@ -342,6 +343,12 @@ class SettingsRoutesMixin: "sources": source_status, "monitor": monitor, "thumbnail_encoders": encoders, + # #320: the full path only here (D6A). + "thumbnail_location": await self._run_in_executor( + functools.partial( + self._thumbnail_location_payload, with_path=True + ) + ), "settings": { "monitoring_enabled": GalleryConfig.MONITORING_ENABLED, "save_foreign_runs": GalleryConfig.SAVE_FOREIGN_RUNS, @@ -468,6 +475,9 @@ class SettingsRoutesMixin: if error_response is not None: return error_response + if "thumbnail_location" in data: + # #320: only the Move thumbnails job changes the location. + return needs_move_response() expected_revision = data.get("expected_revision") if expected_revision is not None and not isinstance(expected_revision, str): return web.json_response( @@ -576,7 +586,8 @@ class SettingsRoutesMixin: for key in _THUMBNAIL_SETTINGS ) job = self._jobs.active(THUMBNAILS_JOB_KIND) if changed else None - if job is None or job.scope.get("mode") != "convert": + # A move copies previews of the current settings (#320). + if job is None or job.scope.get("mode") not in ("convert", "move"): return None return web.json_response( { diff --git a/py/api/sources.py b/py/api/sources.py index 694780d4..6eaa57f5 100644 --- a/py/api/sources.py +++ b/py/api/sources.py @@ -30,6 +30,8 @@ import ntpath import os import re +from .preview_location import HOLDS_THUMBNAILS_REASON + logger = logging.getLogger("prompt_manager.api") MAX_SOURCE_PATH = 4096 # characters @@ -114,7 +116,26 @@ def source_problem(text): code = source_form_error(text) if code is not None: return code, FORM_REASONS[code] - return GalleryConfig.gallery_root_problem(text) + code, reason = GalleryConfig.gallery_root_problem(text) + if code is None and _holds_thumbnails(GalleryConfig.resolve_gallery_root(text)): + return "holds-thumbnails", HOLDS_THUMBNAILS_REASON + return code, reason + + +def _holds_thumbnails(path): + """Whether *path* is inside or around the thumbnail folder in use (#320): + a source there would list Prompt Manager's own previews.""" + from . import preview_store + + try: + from ...utils.library_location import _inside_any + except ImportError: # tests import the package top-level + from utils.library_location import _inside_any + + setting = preview_store.setting() + if setting.mode != preview_store.FOLDER_MODE or not setting.path: + return False + return _inside_any(path, [setting.path]) or _inside_any(setting.path, [path]) def public_spelling(text): diff --git a/py/api/thumbnail_auto.py b/py/api/thumbnail_auto.py index 4dbb5954..59a80a67 100644 --- a/py/api/thumbnail_auto.py +++ b/py/api/thumbnail_auto.py @@ -67,7 +67,7 @@ from pathlib import Path from aiohttp import web -from . import thumbnail_settings +from . import preview_store, thumbnail_settings from .images import ( in_trash_below_roots, MEDIA_EXTENSIONS, @@ -78,7 +78,7 @@ from .images import ( _thumbnail_is_fresh, _video_placeholder_image, _worker_threads, - media_thumbnail_rel, + media_preview_path, ) from .jobs import ( DUPLICATE_CLEANUP_JOB_KIND, @@ -158,6 +158,36 @@ def _dst_key(dst): return flight_key(Path(dst).parent, (Path(dst).name,)) +class DeferredPreview: + """The preview of *rel* under *root*, resolved when its write starts + (#320): a queued capture follows the thumbnail location at write time, + not the one when it was queued.""" + + __slots__ = ("root", "rel", "strict") + + def __init__(self, root, rel, *, strict=True): + self.root = Path(root) + self.rel = Path(rel) + # A capture refuses a store linked out of its container (the serve + # route's rule); a planned target keeps the plan's rule: inside the + # store, as the thumbnail job writes it. + self.strict = strict + + def resolve(self): + """The preview's path now, or None when nothing may be written.""" + dst = media_preview_path(self.root, self.rel, create=True) + if dst is None: + return None + if self.strict: + return dst if preview_store.owns(self.root, dst) else None + store = preview_store.store_dir(self.root) + return dst if _is_within(dst, store) else None + + +def _resolved(dst): + return dst.resolve() if isinstance(dst, DeferredPreview) else Path(dst) + + @functools.lru_cache(maxsize=1) def video_placeholder_jpeg(): """The placeholder frame as JPEG bytes (a video whose preview is missing).""" @@ -215,10 +245,19 @@ def _candidates(parts): def _current_preview(root, parts): """The preview *parts* names, in the format written now (#317): a request for an older name (``_thumb.png`` before a conversion) is answered with - the current file.""" + the current file. None when the store of *root* has no place for it.""" _folder, base, _wanted = _candidates(parts) name = base + thumbnail_settings.THUMB_SUFFIX + writer_settings().extension - return Path(root).joinpath(*parts[:-1], name) + return preview_store.request_path(root, (*parts[:-1], name)) + + +def _request_key(root, parts): + """The flight key of the preview *parts* under *root*: the key of its + disk path (so the serve and history routes share one flight).""" + path = preview_store.request_path(root, parts) + if path is None: + return flight_key(root, parts) + return _dst_key(path) def _quick_source(root, parts): @@ -259,12 +298,14 @@ def _thumbnail_source(root, parts): if in_trash_below_roots(found, [root]): return None rel = Path(*folder_parts, found.name) - dst = root / media_thumbnail_rel(rel) + dst = media_preview_path(root, rel, create=True) + if dst is None: + # Nowhere to write (the thumbnail folder is unavailable): the + # original is served and nothing is written. + return found, None, found.suffix.lower() in VIDEO_EXTENSIONS if dst != _current_preview(root, parts): return None - if not _is_within(root / "thumbnails", root) or not _is_within( - dst, root / "thumbnails" - ): + if not preview_store.owns(root, dst): dst = None return found, dst, found.suffix.lower() in VIDEO_EXTENSIONS @@ -284,6 +325,21 @@ def _scan_folder(folder, base, extensions): return None +def _deferred(roots, src, dst): + """The ``DeferredPreview`` of a planned ``(src, dst)``: the root whose + store the plan wrote *dst* into (nested roots plan one each); *dst* as + planned when none matches.""" + src = Path(src) + for root in roots: + root = Path(root) + if not src.is_relative_to(root): + continue + rel = src.relative_to(root) + if media_preview_path(root, rel) == Path(dst): + return DeferredPreview(root, rel, strict=False) + return dst + + def _stop_stale_workers(): """Stop capture workers a previous import of this module started.""" for thread in threading.enumerate(): @@ -425,16 +481,23 @@ class AutoThumbnailer: def enqueue(self, src, dst, is_video): """Queue *dst*'s preview for the worker; False when already queued, - full, closed or its worker is refused at interpreter exit. Never - blocks on a write.""" - key = _dst_key(dst) + full, closed, nowhere to write it or its worker is refused at + interpreter exit. Never blocks on a write. *dst* may be a + ``DeferredPreview``: its path is resolved again when the write + starts.""" + now = _resolved(dst) + if now is None: + return False + key = _dst_key(now) + if not isinstance(dst, DeferredPreview): + dst = now with self._changed: if self._closed or key in self._queued: return False if len(self._queue) >= self._max_queued: return False self._queued.add(key) - self._queue.append((key, Path(src), Path(dst), bool(is_video))) + self._queue.append((key, Path(src), dst, bool(is_video))) self._busy = True started = True if self._start_worker and not self._worker_running: @@ -514,6 +577,9 @@ class AutoThumbnailer: def _write_queued(self, item): _key, src, dst, is_video = item try: + dst = _resolved(dst) + if dst is None: + return # nowhere to write it now: made on its first view self.ensure(src, dst, is_video) except Exception: # one file never stops the worker self._logger.warning( @@ -683,6 +749,8 @@ class AutoThumbnailMixin: pool.shutdown(wait=False, cancel_futures=True) if final: shutdown_encoders() # kill JPEG XL / ffmpeg encoders still running + # The thumbnail folder's probe (#320), bounded like the rest. + closed = preview_store.stop_probe() and closed return closed def _auto_thumbnail_failures(self): @@ -707,6 +775,9 @@ class AutoThumbnailMixin: return job if job.kind == THUMBNAILS_JOB_KIND and job.title == THUMBNAIL_CLEAR_TITLE: return job + # Move thumbnails holds automatic previews for its whole run (D2A). + if job.kind == THUMBNAILS_JOB_KIND and job.scope.get("mode") == "move": + return job return None # ── the view admission ──────────────────────────────────────────── @@ -847,7 +918,7 @@ class AutoThumbnailMixin: for root in output_dirs: root = Path(root) outcome = await gate.run( - flight_key(root, parts), + _request_key(root, parts), functools.partial(self._preview_on_demand, root, parts), pool, ) @@ -860,7 +931,7 @@ class AutoThumbnailMixin: continue return self._preview_response(BUSY, quick[0], None, quick[1]) dst = _current_preview(root, parts) - if outcome == READY: + if outcome == READY and dst is not None: return self._file_response(dst) located = await gate.loop.run_in_executor( pool, _thumbnail_source, root, parts @@ -883,17 +954,31 @@ class AutoThumbnailMixin: """ if self._auto_thumbnails_shut: return "skipped" - located = self._capture_target(image_path) + located = self._capture_source(image_path) if located is None: return "skipped" - src, dst, is_video = located + src, root, rel, is_video = located + dst = DeferredPreview(root, rel).resolve() + if dst is None: + return "skipped" if _thumbnail_is_fresh(src, dst): return "fresh" - if self._auto_thumbnails().enqueue(src, dst, is_video): + if self._auto_thumbnails().enqueue(src, DeferredPreview(root, rel), is_video): return "queued" return "skipped" def _capture_target(self, image_path): + """``(src, dst, is_video)`` of a new output's preview now, or None.""" + located = self._capture_source(image_path) + if located is None: + return None + src, root, rel, is_video = located + dst = DeferredPreview(root, rel).resolve() + return None if dst is None else (src, dst, is_video) + + def _capture_source(self, image_path): + """``(src, root, rel, is_video)`` of a media file under a current + root (not a preview, a link or in a trash folder), or None.""" path = Path(os.path.abspath(str(image_path))) if path.suffix.lower() not in MEDIA_EXTENSIONS or not _regular_file(path): return None @@ -906,12 +991,7 @@ class AutoThumbnailMixin: rel = Path(os.path.relpath(path, root)) if "thumbnails" in rel.parts[:-1] or in_trash_below_roots(path, [root]): return None - dst = root / media_thumbnail_rel(rel) - if not _is_within(root / "thumbnails", root) or not _is_within( - dst, root / "thumbnails" - ): - return None - return path, dst, path.suffix.lower() in VIDEO_EXTENSIONS + return path, root, rel, path.suffix.lower() in VIDEO_EXTENSIONS return None # ── after a scan ────────────────────────────────────────────────── @@ -953,7 +1033,7 @@ class AutoThumbnailMixin: return "none" thumbnailer = self._auto_thumbnails() for src, dst, is_video in plan["targets"]: - thumbnailer.enqueue(src, dst, is_video) + thumbnailer.enqueue(src, _deferred(output_dirs, src, dst), is_video) self.logger.info(f"{missing:,} previews missing after the scan; queued") return "queued" diff --git a/py/api/thumbnail_convert.py b/py/api/thumbnail_convert.py index a12d8be1..72434151 100644 --- a/py/api/thumbnail_convert.py +++ b/py/api/thumbnail_convert.py @@ -44,7 +44,7 @@ import os import time from pathlib import Path -from . import thumbnail_settings +from . import preview_store, thumbnail_settings from .thumbnail_encode import ThumbnailVerifyError, verify_thumbnail, writer_settings from .admin import _plural, _scan_location, _scan_roots, _scan_status_line from .images import _thumbnail_is_fresh, _worker_threads, _thumbnail_batch_size @@ -393,6 +393,12 @@ class ThumbnailConvertMixin: removed, with no bytes (another run or a sibling removed it).""" removed, kept = 0, [] for path, size in old: + if not preview_store.may_write(path): + # Folder mode: never a file outside a key folder, or one + # reached through a link (checked at the moment of the delete). + self.logger.warning(f"Not removing {path}: outside the thumbnails") + kept.append(path) + continue try: os.remove(path) except FileNotFoundError: diff --git a/py/api/thumbnail_encode.py b/py/api/thumbnail_encode.py index f7a6ca54..aad912c4 100644 --- a/py/api/thumbnail_encode.py +++ b/py/api/thumbnail_encode.py @@ -46,7 +46,7 @@ from pathlib import Path from PIL import Image -from . import thumbnail_settings +from . import preview_location, preview_store, thumbnail_settings try: from ...utils.logging_config import get_logger @@ -101,14 +101,20 @@ def temp_sibling(dst, tag="part"): def replace_unless_held(tmp, dst, guard=None): - """Rename *tmp* over *dst*, unless ``guard()`` is false (PreviewWriteHeld).""" - if guard is not None and not guard(): - raise PreviewWriteHeld(PREVIEW_HELD) - os.replace(tmp, dst) + """Rename *tmp* over *dst*, unless ``guard()`` is false (PreviewWriteHeld). + + The check and the rename run under ``preview_store.write_lock`` (#320): a + guard from ``preview_store.write_guard`` also refuses a write that started + before the thumbnail location changed.""" + with preview_store.committed(guard) as allowed: + if not allowed: + raise PreviewWriteHeld(PREVIEW_HELD) + os.replace(tmp, dst) # ``temp_sibling``'s names: ``..--``. _TEMP_NAME = re.compile(r"^\..+\.(?:part|src)-(\d+)-\d+\.[A-Za-z0-9]+$") +_MOVE_TEMP_NAME = re.compile(r"\.pm-move-[0-9a-f]{8}\.tmp$") # A temp at least this old (seconds) outlived its write: no encode runs longer. STRAY_TEMP_AGE = 4 * ENCODE_TIMEOUT # When this process started (this module's import): a temp carrying its PID @@ -120,7 +126,10 @@ def sweep_stray_temps(folder, min_age=None): """Remove the temp files a crashed write left under *folder* (a preview folder, walked without following links); returns ``(count, bytes)``. - Only ``temp_sibling`` names are touched, never a link. With *min_age* + Only ``temp_sibling`` names are touched, never a link: a folder that is a + link, junction or reparse point is not walked into (Windows walks into a + junction even without ``followlinks``), and each file passes + ``preview_store.may_write`` before it goes. With *min_age* (seconds, the startup sweep) a temp younger than that, or one this process wrote since it started (it may still be writing it), is left; ``min_age=0`` (the clear, which holds every preview write) removes them @@ -130,19 +139,39 @@ def sweep_stray_temps(folder, min_age=None): now = time.time() own = str(os.getpid()) count = size = 0 - for root, _dirs, files in os.walk(folder): + try: + if preview_location.is_link_like(folder): + return count, size + except OSError as exc: + logger.debug(f"Not sweeping {folder}: {exc}") + return count, size + for root, dirs, files in os.walk(folder): + dirs[:] = [d for d in dirs if not _link_like(Path(root) / d)] for name in files: match = _TEMP_NAME.match(name) - if match is None: + # A move's copy temp (#320): ``.pm-move-<8 hex>.tmp``. + move_temp = match is None and _MOVE_TEMP_NAME.search(name) + if match is None and not move_temp: continue path = Path(root) / name try: stat = path.lstat() - if path.is_symlink() or not path.is_file(): + if ( + path.is_symlink() + or not path.is_file() + or _link_like(path) + or not preview_store.may_write(path) + ): continue if min_age and ( now - stat.st_mtime < min_age - or (match.group(1) == own and stat.st_mtime >= PROCESS_STARTED) + or move_temp + and stat.st_mtime >= PROCESS_STARTED + or ( + match is not None + and match.group(1) == own + and stat.st_mtime >= PROCESS_STARTED + ) ): continue path.unlink() @@ -154,6 +183,14 @@ def sweep_stray_temps(folder, min_age=None): return count, size +def _link_like(path): + """``is_link_like``, with a path that cannot be checked counted as one.""" + try: + return preview_location.is_link_like(path) + except OSError: + return True + + def _unlink_quietly(path): try: Path(path).unlink() diff --git a/py/api/thumbnail_jobs.py b/py/api/thumbnail_jobs.py index 463eb891..7d7c8004 100644 --- a/py/api/thumbnail_jobs.py +++ b/py/api/thumbnail_jobs.py @@ -49,6 +49,7 @@ from .images import ( _worker_threads, ) from .job_routes import busy_response +from .thumbnail_location import location_refusal from .jobs import THUMBNAIL_GROUP_NAMES, THUMBNAILS_JOB_KIND, JobBusy from .thumbnail_convert import CONVERT_JOB_TITLE from .thumbnail_encode import writer_settings @@ -131,7 +132,10 @@ class ThumbnailJobsMixin: """The thumbnail plan, the thumbnail job and the legacy stream on it.""" async def get_thumbnail_plan(self, request): - """What a thumbnail run would do (read-only; capped like a run).""" + """What a thumbnail run would do (read-only; capped like a run). + ``mode=prune`` lists the removed sources' thumbnails (#320).""" + if request.query.get("mode") == "prune": + return await self.get_prune_plan() options, error = thumbnail_options(request.query, from_query=True) if error: return _invalid(error) @@ -179,15 +183,39 @@ class ThumbnailJobsMixin: ) except (ValueError, UnicodeDecodeError): return _invalid(_OPTIONS_ERROR) + if isinstance(body, dict) and body.get("mode") in ("move", "prune"): + return await self._start_location_job(body) options, error = thumbnail_options(body) if error: return _invalid(error) + refused = location_refusal() + if refused is not None: + return refused try: job = self._start_thumbnail_job(options) except JobBusy as busy: return busy_response(busy) return web.json_response({"success": True, "job": job.summary()}, status=202) + async def _start_location_job(self, body): + """Move thumbnails or Remove thumbnails of removed sources (#320).""" + if body["mode"] == "prune": + refused = location_refusal() + if refused is not None: + return refused + try: + if body["mode"] == "move": + started = self.start_thumbnail_move(body) + else: + started = self.start_thumbnail_prune() + except JobBusy as busy: + return busy_response(busy) + if isinstance(started, web.Response): + return started + return web.json_response( + {"success": True, "job": started.summary()}, status=202 + ) + def _start_thumbnail_job(self, options): """Register the thumbnail job synchronously; raises JobBusy. @@ -459,8 +487,12 @@ class ThumbnailJobsMixin: or attaches to the running thumbnail job, and streams its frames. A legacy hold (the blocking generate or clear) or a record-changing job refuses it with the stream's error frame. The job goes on when - the client goes away. + the client goes away. Refused (409) while the thumbnail folder is + unavailable (#320). """ + refused = location_refusal() + if refused is not None: + return refused response = web.StreamResponse( status=200, headers={ @@ -507,8 +539,12 @@ class ThumbnailJobsMixin: the way. """ running = self._jobs.active(THUMBNAILS_JOB_KIND) - if running is not None and running.scope.get("mode") == "convert": - return None, False, running # no legacy frames to follow (#317) + if running is not None and running.scope.get("mode") in ( + "convert", + "move", + "prune", + ): + return None, False, running # no legacy frames to follow (#317, #320) if running is not None and running.cancellable: return running, False, None busy = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded") diff --git a/py/api/thumbnail_location.py b/py/api/thumbnail_location.py new file mode 100644 index 00000000..c9a1a086 --- /dev/null +++ b/py/api/thumbnail_location.py @@ -0,0 +1,162 @@ +"""The thumbnail location on the API (#320): startup, the settings payload +and the refusals while the thumbnail folder is unavailable. + +The setting itself is config.json ``thumbnails.location`` +(``preview_location``); the runtime state is ``preview_store``. POST +``/settings`` never changes it (409 ``location-needs-move``): the Move +thumbnails job is its only writer. +""" + +import os + +from aiohttp import web + +from . import preview_location, preview_store + +try: + from ...utils import library_location +except ImportError: # tests import the package top-level + from utils import library_location + +# The node folder (its web/ is served and a reinstall wipes it). +NODE_FOLDER = os.path.dirname( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +) + + +def default_path(): + """``/data/thumbnails`` (D7B), or None when unknown.""" + try: + return library_location.cache_dir("thumbnails") + except Exception: # no data folder can be decided: no default + return None + + +def location_refusal(location=None): + """The 409 ``location-unavailable`` response while the thumbnail folder + is unavailable, else None (thumbnail generate, convert, clear, prune).""" + location = location or preview_store.current() + if location.mode != preview_store.FOLDER_MODE or location.available: + return None + status, text = preview_location.message("location-unavailable") + return web.json_response( + {"success": False, "code": "location-unavailable", "error": text}, + status=status, + ) + + +def needs_move_response(): + status, text = preview_location.message("location-needs-move") + return web.json_response( + {"success": False, "code": "location-needs-move", "error": text}, + status=status, + ) + + +class ThumbnailLocationMixin: + """Startup check and payloads of the thumbnail location.""" + + def _all_source_roots(self): + """Every image source of every root list: the Images roots, the + configured sources and the extra gallery roots (as strings).""" + from ..config import GalleryConfig, _extra_gallery_roots + + roots = [str(root) for root in self._get_all_output_dirs()] + roots += [str(root) for root in GalleryConfig.MONITORING_DIRECTORIES] + roots += list(_extra_gallery_roots()) + return tuple(dict.fromkeys(root for root in roots if root)) + + def _thumbnail_location_context(self): + """What a thumbnail folder is validated against (blocking).""" + contained = [] + try: + import folder_paths + + for name in ( + "get_output_directory", + "get_input_directory", + "get_temp_directory", + ): + getter = getattr(folder_paths, name, None) + folder = getter() if getter is not None else None + if folder: + contained.append(str(folder)) + except ImportError: + pass # folder_paths exists only inside ComfyUI (tests run without it) + try: + contained.append(library_location.config_location()[1]) + except Exception: # no data folder can be decided + self.logger.debug("No library folder for the thumbnail check") + setting = preview_store.setting() + return preview_location.Context( + roots=self._all_source_roots(), + contained=tuple(contained), + node_folder=NODE_FOLDER, + default_path=default_path(), + current_path=( + setting.path if setting.mode == preview_store.FOLDER_MODE else None + ), + ) + + def use_preview_location(self): + """Put the configured location in force at once, unchecked (the + package's import, before any request): folder mode is unavailable + until ``start_preview_location`` has checked it, so nothing is ever + written to the sources meanwhile. No filesystem access.""" + from ..config import PromptManagerConfig + + try: + return preview_store.configure( + PromptManagerConfig.THUMBNAIL_LOCATION, check=False + ) + except Exception: + self.logger.warning("The thumbnail location is unusable", exc_info=True) + preview_store.mark_invalid() + return preview_store.current() + + def start_preview_location(self): + """Parse and check the configured location once (startup thread); + the probe takes over. Never raises; a check that fails leaves a + folder location unavailable (``invalid``), never the sources.""" + from ..config import PromptManagerConfig + + try: + value = PromptManagerConfig.THUMBNAIL_LOCATION + parsed = preview_location.parse_location(value) + context = None + if parsed.mode == preview_store.FOLDER_MODE: + context = self._thumbnail_location_context() + location = preview_store.configure(value, context=context) + except Exception: + self.logger.warning("The thumbnail location check failed", exc_info=True) + if ( + preview_location.parse_location( + PromptManagerConfig.THUMBNAIL_LOCATION + ).mode + == preview_store.FOLDER_MODE + ): + preview_store.mark_invalid() + return preview_store.current() + if location.mode == preview_store.FOLDER_MODE and not location.available: + self.logger.warning( + "The thumbnail folder is unavailable " + f"({location.problem or 'checking'}): originals are shown and " + "no thumbnails are written until it is back" + ) + return location + + def _thumbnail_location_payload(self, *, with_path): + """``{mode, available, problem}`` plus, for the settings GET only, + ``path`` and ``default_path`` (D6A).""" + location = preview_store.current() + payload = { + "mode": location.mode, + "available": location.available, + "problem": location.problem, + } + if with_path: + payload["path"] = location.path + payload["default_path"] = default_path() + # Old copies a move still has to remove (#320 Part 3). + payload["cleanup_pending"] = len(preview_store.setting().cleanup) + return payload diff --git a/py/api/thumbnail_move.py b/py/api/thumbnail_move.py new file mode 100644 index 00000000..13bfebce --- /dev/null +++ b/py/api/thumbnail_move.py @@ -0,0 +1,579 @@ +"""Move thumbnails (#320 Part 3): the reviewed job that changes where previews +live, the removal of old copies after it, recovery at startup and the +"Remove thumbnails of removed sources" scope (Part 5). + +Kind ``thumbnails`` with ``mode: "move"`` (D1A): admission ``guarded``, so +scans, deletes, tagging, source changes and every other thumbnail job wait +for it, while the monitor's Needs review recording runs beside it. +Automatic previews are held for the whole job (D2A, ``_auto_thumbnail_blocker``). + +Phases: review (``POST /images/thumbnails/plan``, no writes but the +validation probe), start (a plan token), hold (generation bump, journal, +target folder), copy (cancellable at every file boundary, on the job pool +in small chunks, D3A), switch (one config commit sets the location and +records the old stores under ``cleanup``; not cancellable from here), +remove old copies, end. Per-file errors switch anyway (D10A); a folder-wide +error or a cancel before the switch removes what the job wrote and changes +nothing. The move is the only writer of the setting (D4B). + +Moving away from a thumbnail folder also removes the previews of this +install's sources that are no longer listed (Q320-8: remove). +""" + +import functools +import os +import time +from pathlib import Path + +from aiohttp import web + +from . import preview_location, preview_store +from . import thumbnail_move_files as files +from .jobs import THUMBNAILS_JOB_KIND +from .thumbnail_move_plan import ( + MOVE_GROUPS, + MOVE_TITLE, + PLAN_TTL, + _number, + _plural, + _remove, + _roots_revision, + _volume, + _is_network, + _write_json, +) +from . import thumbnail_move_plan as move_plan + +try: + from ...utils import exit_signal + from ...utils.parallel import map_parallel +except ImportError: # tests import the package top-level + from utils import exit_signal + from utils.parallel import map_parallel + + +class ThumbnailMoveMixin: + """Start and run the move job; the removal of old copies.""" + + # ── start ───────────────────────────────────────────────────────── + + def _take_move_plan(self, body): + """The plan a start names, or ``(status, code)``.""" + token = body.get("token") + plan = (self._move_plans or {}).get(token) if isinstance(token, str) else None + if plan is None or time.monotonic() - plan.created > PLAN_TTL: + return 409, "plan-stale" + roots = [str(root) for root in self._get_all_output_dirs()] + if _roots_revision(roots) != plan.revision: + return 409, "plan-stale" + if preview_store.setting() != plan.old: + return 409, "plan-stale" + if plan.confirm_required and body.get("confirmed") is not True: + return 400, "confirmation-required" + return plan + + def start_thumbnail_move(self, body): + """Start the move job (sync, on the loop); raises JobBusy. Returns + the job or a refusal response.""" + if not self._move_recovered: + blocker = self._jobs.active(THUMBNAILS_JOB_KIND) + return web.json_response( + { + "success": False, + "code": "busy", + "error": "Prompt Manager is still checking an earlier move. " + "Try again in a moment.", + "running": None if blocker is None else {"id": blocker.id}, + }, + status=409, + ) + taken = self._take_move_plan(body) + if isinstance(taken, tuple): + status, code = taken + text = { + "plan-stale": "The review is out of date. Review the move again.", + "confirmation-required": "Confirm that Prompt Manager may " + "create its folders there.", + }[code] + return web.json_response( + {"success": False, "code": code, "error": text}, status=status + ) + plan = taken + self._move_plans.pop(plan.token, None) + return self._jobs.start( + THUMBNAILS_JOB_KIND, + MOVE_TITLE, + admission="guarded", + cancellable=True, + scope={ + "mode": "move", + "from": plan.old.mode, + "to": plan.target["mode"], + "to_path": plan.target["path"], + }, + groups=MOVE_GROUPS, + runner=functools.partial(self._run_thumbnail_move, plan=plan), + ) + + # ── the job ─────────────────────────────────────────────────────── + + async def _run_thumbnail_move(self, job, *, plan): + state = {"written": [], "created": [], "created_folder": False} + try: + await self._run_in_executor(self._move_hold, job, plan, state) + except (OSError, ValueError) as error: + await self._run_in_executor(self._move_rollback, plan, state) + self._end_move( + job, "failed", "The new folder could not be made; nothing changed." + ) + self.logger.warning(f"Move thumbnails could not start: {error}") + return + left = [ + { + "path": "/".join(item), + "root": s.label, + "detail": "Not a thumbnail; left in place", + } + for s in plan.sources + if s.scan is not None + for item in s.scan.left + ] + if left: + job.record_counted("left", len(left), left) + items = [ + (source, item) + for source in plan.sources + if source.scan is not None + for item in source.scan.copy + ] + total = len(items) + job.update(total=total, processed=0, progress=0) + chunk = await self._run_in_executor(self._move_chunk, plan) + workers = self._move_workers() + done = 0 + for start in range(0, total, chunk): + if job.cancelled: + await self._move_cancelled(job, plan, state) + return + batch = items[start : start + chunk] + try: + results = await self._run_in_executor( + map_parallel, + functools.partial(self._move_copy_one, state=state), + batch, + workers, + ) + except exit_signal.ExitRequested: + return # exiting: the next start undoes the copies (journal) + except files.FolderGone as error: + self.logger.warning(f"Move thumbnails stopped: {error}") + await self._run_in_executor(self._move_rollback, plan, state) + self._end_move( + job, + "failed", + "The new folder could not be reached or is full; nothing changed.", + ) + return + except Exception: + # Anything else before the switch: undo, never half a move. + self.logger.exception("Move thumbnails failed while copying") + await self._run_in_executor(self._move_rollback, plan, state) + self._end_move(job, "failed", "The copy failed; nothing changed.") + return + for (source, item), (outcome, detail) in zip(batch, results): + row = { + "path": "/".join(item[0]), + "root": source.label, + "detail": detail, + } + job.record( + "moved" if outcome == "moved" else outcome, + row, + f"{outcome}: {source.label} · {'/'.join(item[0])}", + ) + done += len(batch) + label = batch[-1][0].label + job.update( + processed=done, + progress=int(done * 90 / total) if total else 90, + status_line=( + f"Copying {_number(done)} of {_number(total)} previews · {label}" + ), + ) + job.publish({"type": "progress", "summary": job.summary()}) + if job.cancelled: + await self._move_cancelled(job, plan, state) + return + # The switch: not cancellable from here (an exit still stops between + # files and leaves the cleanup to the next start). + job.cancellable = False + job.update(status_line="Switching to the new location", progress=92) + job.publish({"type": "progress", "summary": job.summary()}) + try: + await self._run_in_executor(self._move_switch, plan) + except Exception: + self.logger.exception("Move thumbnails could not switch") + await self._run_in_executor(self._move_rollback, plan, state) + self._end_move( + job, "failed", "The setting could not be saved; nothing changed." + ) + return + await self._remove_old_copies(job, plan) + moved = job.counters.get("moved", 0) + job.counters.get("skipped", 0) + errors = job.counters.get("errors", 0) + where = plan.target["path"] or "inside each source" + parts = [f"{_number(moved)} moved"] + if errors: + parts.append(_plural(errors, "error")) + parts.append( + "old copies removed" + if not job.scope.get("cleanup_left") + else "some old copies are removed at the next start" + ) + self._end_move( + job, "finished", f"Thumbnails moved to {where}: " + " · ".join(parts) + "." + ) + + def _end_move(self, job, state, line): + job.update(status_line=line, progress=100) + job.log_line(line) + job.publish({"type": "complete", "summary": job.summary()}) + job.finish(state) + + def _move_workers(self): + from .images import _worker_threads + + return max(1, int(_worker_threads() or 1)) + + def _move_chunk(self, plan): + """2 files at a time across volumes or to/from a share, else 4, never + more than Worker threads (D3A: the pool keeps its size).""" + sides = [plan.target["path"] or ""] + [s.old_store or "" for s in plan.sources] + sides += [s.new_store for s in plan.sources] + network = any(_is_network(side) for side in sides if side) + volumes = {_volume(side) for side in sides if side} + size = 2 if network or len(volumes) > 1 else 4 + return max(1, min(size, self._move_workers())) + + def _move_hold(self, job, plan, state): + """Phase 3: bump the generation, write the journal, make the target.""" + preview_store.bump_generation() + if plan.to_folder: + folder = plan.target["path"] + if not os.path.isdir(folder): + if plan.target_default: + os.makedirs(folder, exist_ok=True) + else: + os.mkdir(folder) + state["created_folder"] = True + plan.target_id = preview_location.init_folder( + folder, created_by_pm=state["created_folder"] + ) + install = plan.install + + def add_keys(marker): + for source in plan.sources: + if source.key and source.key not in marker["keys"]: + marker["keys"][source.key] = { + "path": source.root, + "identity": os.path.normcase(os.path.realpath(source.root)), + "dev_ino": preview_location.file_identity(source.root), + "install": install, + } + + with preview_store.write_lock: + preview_location.write_marker(folder, add_keys) + _write_json( + move_plan.journal_path(), + { + "state": "copying", + "from": plan.old.to_config(), + "to": plan.target, + "stores": [s.new_store for s in plan.sources], + # Stores that do not exist yet are made by this job only: + # after a crash everything in them goes (B4). + "created_dirs": [ + s.new_store for s in plan.sources if not os.path.isdir(s.new_store) + ], + "created_folder": state["created_folder"], + "install": plan.install, + }, + ) + job.log_line( + f"Move thumbnails started: {plan.old.path or 'inside each source'} → " + f"{plan.target['path'] or 'inside each source'}" + ) + + def _move_copy_one(self, pair, *, state): + """Copy one preview (a pool thread); ``(group, detail)``.""" + source, (relative, size, mtime_ns) = pair + destination = Path(source.new_store).joinpath(*relative) + try: + if not os.path.isdir(destination.parent): + base = Path(source.new_store) + if not base.is_dir(): + if plan_parent_missing(base): + raise files.FolderGone("the new folder is gone") + created = [] + files._make_folders(base.parent, (base.name,), created) + state["created"].extend(created) + created = [] + files._make_folders(base, relative[:-1], created) + state["created"].extend(created) + # Also when the folders already exist (Codex P1 on #592). + files.refuse_links(source.new_store, relative[:-1]) + outcome = files.copy_one( + Path(source.old_store).joinpath(*relative), + destination, + size, + mtime_ns, + state["created"], + ) + except files.CopyFailed as error: + return "errors", str(error) + if outcome == "moved": + state["written"].append(str(destination)) + return "moved", "Copied and checked" + return "skipped", "Already there" + + async def _move_cancelled(self, job, plan, state): + if exit_signal.requested(): + return # the next start removes the partial copies (journal) + await self._run_in_executor(self._move_rollback, plan, state) + self._end_move(job, "cancelled", "Move cancelled: nothing changed.") + + def _move_rollback(self, plan, state): + """Remove what this job wrote (files, folders it made, a folder and + marker it created) and the journal: nothing changed.""" + for path in state["written"]: + files.remove_file(path) + for folder in reversed(state["created"]): + files.remove_empty_folders(folder) + if plan.to_folder and state.get("created_folder"): + folder = plan.target["path"] + try: + names = os.listdir(folder) + except OSError: + names = None + if names is not None and set(names) <= {preview_location.MARKER_NAME}: + _remove(preview_location.marker_path(folder)) + files.remove_empty_folders(folder) + _remove(move_plan.journal_path()) + + def _move_switch(self, plan): + """Phase 5: one config commit sets the location and records the old + stores under ``cleanup``; the runtime follows under the write lock.""" + from ..config import PromptManagerConfig + from ..config_store import get_config_store + + entry = { + "from": plan.old.mode, + "folder": plan.old.path if plan.from_folder else None, + "stores": [s.old_store for s in plan.sources if s.old_store], + "drop_keys": self._old_keys_to_drop(plan), + "install": plan.install, + } + # An older entry never removes a store the move just filled (a move + # back while old copies still waited). + filled = {os.path.normcase(s.new_store) for s in plan.sources} + cleanup = [] + for old_entry in plan.old.cleanup: + stores = [ + store + for store in old_entry.get("stores") or [] + if isinstance(store, str) and os.path.normcase(store) not in filled + ] + folder = old_entry.get("folder") + drop = [ + key + for key in old_entry.get("drop_keys") or [] + if not folder + or os.path.normcase(os.path.join(folder, key)) not in filled + ] + cleanup.append({**old_entry, "stores": stores, "drop_keys": drop}) + cleanup.append(entry) + if plan.to_folder: + value = { + "mode": preview_store.FOLDER_MODE, + "path": plan.target["path"], + "id": plan.target_id, + "install": plan.install, + "cleanup": cleanup, + } + else: + value = { + "mode": preview_store.SOURCE_MODE, + "install": plan.install, + "cleanup": cleanup, + } + + def mutate(config): + section = config.get("thumbnails") + if not isinstance(section, dict): + section = {} + section["location"] = value + config["thumbnails"] = section + return config + + def publish(_config): + PromptManagerConfig.THUMBNAIL_LOCATION = value + preview_store.configure(value, check=False) + + get_config_store().commit(mutate, expected_revision=None, publish=publish) + # The switch is made: what follows never undoes it. + try: + preview_store.refresh() + _remove(move_plan.journal_path()) + except OSError as error: + self.logger.warning( + f"The thumbnail move journal stays until the next start: {error}" + ) + + def _old_keys_to_drop(self, plan): + if not plan.from_folder: + return [] + keys = [s.old_store and os.path.basename(s.old_store) for s in plan.sources] + keys = [key for key in keys if key] + if plan.to_folder and os.path.normcase(plan.old.path or "") == os.path.normcase( + plan.target["path"] or "" + ): + return [] + return keys + [key for key, _store, _scan in plan.unlisted] + + async def _remove_old_copies(self, job, plan): + """Phase 6: run the cleanup the switch recorded (and earlier ones).""" + job.update(status_line="Removing old copies", progress=95) + job.publish({"type": "progress", "summary": job.summary()}) + left = await self._run_in_executor(self._run_cleanup, job) + if left: + job.scope["cleanup_left"] = True + + def _run_cleanup(self, job=None): + """Remove every recorded old store's previews, temps and empty + folders; keep in ``cleanup`` what could not go. Returns how many + entries are left (blocking).""" + setting = preview_store.setting() + remaining = [] + for entry in setting.cleanup: + if not self._cleanup_entry(entry, job): + remaining.append(entry) + if list(setting.cleanup) != remaining: + self._commit_cleanup(remaining) + return len(remaining) + + def _cleanup_entry(self, entry, job): + """One recorded removal; True when it is complete.""" + folder = entry.get("folder") + stores = [ + store for store in entry.get("stores") or [] if isinstance(store, str) + ] + if folder: + if not os.path.isdir(folder): + return False # the old folder is not reachable: kept for later + folder_marker = preview_location.read_marker(folder) or {} + keys = folder_marker.get("keys") or {} + stores += [ + os.path.join(folder, key) + for key in entry.get("drop_keys") or [] + if preview_location.valid_key(key) and key in keys + ] + complete = True + for store in dict.fromkeys(stores): + if not self._remove_store(store, folder, job): + complete = False + if folder and complete: + drop = set(entry.get("drop_keys") or []) + + def forget(marker): + for key in drop: + marker["keys"].pop(key, None) + + try: + with preview_store.write_lock: + marker = preview_location.write_marker(folder, forget) + except (OSError, ValueError): + return False + if not marker["keys"] and marker.get("created_by_pm"): + names = set(os.listdir(folder)) + rest = {n for n in names if not files.is_os_metadata(n)} + if rest <= {preview_location.MARKER_NAME}: + _remove(preview_location.marker_path(folder)) + files.remove_empty_folders(folder) + return complete + + def _remove_store(self, store, folder, job): + """Remove the previews and temps of one old store; True when done.""" + store = Path(store) + if folder: + # A key folder directly in the old thumbnail folder, nothing else. + if not preview_location.valid_key(store.name) or os.path.normcase( + str(store.parent) + ) != os.path.normcase(str(folder)): + return True + elif store.name != "thumbnails": + return True # a hand-edited entry never names another folder + if not store.exists(): + return True + if preview_location.is_link_like(store): + return True # never through a link; nothing of ours there + complete = True + removed = 0 + for current, dirs, names in os.walk(store): + dirs[:] = [ + d for d in dirs if not preview_location.is_link_like(Path(current) / d) + ] + for name in names: + path = Path(current) / name + if files.is_temp_name(name) or files._preview_parts(name): + if preview_location.is_link_like(path): + continue + if exit_signal.requested(): + return False # exiting: the next start finishes it + if files.remove_file(path): + removed += 1 + else: + complete = False + relative = path.relative_to(store).as_posix() + self.logger.warning( + f"Could not remove the old thumbnail {path}; it is " + "removed at the next start" + ) + if job is not None: + # D10A: a removal that failed is an error the job lists. + job.record( + "errors", + { + "path": relative, + "root": store.name, + "detail": "In use by another program; removed " + "at the next start", + }, + f"In use by another program; removed at the next " + f"start: {store.name}/{relative}", + ) + files.remove_empty_folders(store) + if job is not None and removed: + job.log_line(f"Removed {_number(removed)} old files from {store.name}") + return complete + + def _commit_cleanup(self, remaining): + from ..config import PromptManagerConfig + from ..config_store import get_config_store + + def mutate(config): + section = config.get("thumbnails") + if isinstance(section, dict) and isinstance(section.get("location"), dict): + section["location"]["cleanup"] = remaining + return config + + def publish(config): + value = (config.get("thumbnails") or {}).get("location") + PromptManagerConfig.THUMBNAIL_LOCATION = value + preview_store.set_cleanup(remaining) + + get_config_store().commit(mutate, expected_revision=None, publish=publish) + + +def plan_parent_missing(store): + """A key folder's parent (the thumbnail folder) is gone mid-copy.""" + return not Path(store).parent.is_dir() diff --git a/py/api/thumbnail_move_files.py b/py/api/thumbnail_move_files.py new file mode 100644 index 00000000..031820a4 --- /dev/null +++ b/py/api/thumbnail_move_files.py @@ -0,0 +1,348 @@ +"""The file side of Move thumbnails (#320 Part 3): what is copied, how a +copy is made and checked, and how old copies are removed. + +No job, HTTP or config code lives here; ``thumbnail_move`` drives it. + +Copy set (B5): each old store is walked with ``os.scandir``, never through a +symlink, junction or reparse point (counted as "skipped link"). A regular +file is + +- copied when it has the preview grammar (``_thumb``, any written + or legacy extension) and its original still exists in the source; +- removed with the old copies, not copied, when it is an orphan (no + original), lies under a trash folder, or is a temp sibling of a write + (``..part-…``) or of a move (``….pm-move-<8 hex>.tmp``); +- left in place otherwise ("Left in place"). OS metadata files (``Thumbs.db``, + ``desktop.ini``, ``.DS_Store``, ``._*``) are removed only when they alone + keep a preview folder from being removed. + +A copy goes to ``.pm-move-<8 hex>.tmp`` beside its destination, its +size is compared, its mtime is set to the source preview's rounded up to the +next 2-second step (FAT/exFAT keep it fresh, a stale preview stays stale), +then it is renamed into place. Every handle is closed before a rename or an +unlink (Windows). +""" + +import errno +import logging +import os +import re +import secrets +import shutil +import stat as stat_module +import time +from dataclasses import dataclass, field +from pathlib import Path + +from . import preview_location, thumbnail_settings +from .thumbnail_encode import _TEMP_NAME + +try: + from ...utils.media_folders import in_trash_folder +except ImportError: # tests import the package top-level + from utils.media_folders import in_trash_folder + +logger = logging.getLogger("prompt_manager.thumbnails") + +MOVE_TEMP = re.compile(r"\.pm-move-[0-9a-f]{8}\.tmp$") +MOVE_TEMP_SUFFIX = ".pm-move-{token}.tmp" +# The temp name adds this much to a destination path (".pm-move-xxxxxxxx.tmp"). +TEMP_EXTRA = 21 +WINDOWS_FILE_LIMIT = 259 +MTIME_STEP_NS = 2_000_000_000 +# A file another program holds open (Windows) is retried this often. +REMOVE_RETRIES = (0.5, 1.0, 2.0) +_WINDOWS_IN_USE = {32, 33} # ERROR_SHARING_VIOLATION, ERROR_LOCK_VIOLATION + + +def is_os_metadata(name): + folded = name.lower() + return folded in preview_location.OS_METADATA or folded.startswith("._") + + +def is_temp_name(name): + return bool(_TEMP_NAME.match(name) or MOVE_TEMP.search(name)) + + +def _preview_parts(name): + """``(base, extension)`` of a preview file name, or None.""" + stem, extension = os.path.splitext(name) + suffix = thumbnail_settings.THUMB_SUFFIX + if not stem.endswith(suffix) or len(stem) <= len(suffix): + return None + if extension.lower() not in thumbnail_settings.THUMBNAIL_EXTENSIONS: + return None + return stem[: -len(suffix)], extension + + +def _has_original(root, rel_dir, base, extension): + """Whether a media file in ``/`` owns this preview.""" + folder = Path(root).joinpath(*rel_dir) + try: + with os.scandir(folder) as entries: + for entry in entries: + stem, source_ext = os.path.splitext(entry.name) + if stem != base or not source_ext: + continue + if not thumbnail_settings.owns(extension, source_ext): + continue + if entry.is_file(follow_symlinks=False): + return True + except OSError: + return False + return False + + +@dataclass +class StoreScan: + """One old store's files, sorted into what the move does with them.""" + + store: Path + copy: list = field(default_factory=list) # (relative parts, size, mtime_ns) + orphans: list = field(default_factory=list) # relative parts + temps: list = field(default_factory=list) + left: list = field(default_factory=list) + links: int = 0 + bytes: int = 0 + + @property + def removable(self): + return [item[0] for item in self.copy] + self.orphans + self.temps + + +def scan_store(store, root, *, stop=None): + """Sort the files of *store* (the previews of the source *root*). + + A store that does not exist, or is itself a link, scans empty (a link + counts once). + """ + store = Path(store) + scan = StoreScan(store) + try: + if preview_location.is_link_like(store): + scan.links += 1 + return scan + if not store.is_dir(): + return scan + except OSError: + return scan + _walk(scan, store, (), root, stop) + return scan + + +def _walk(scan, folder, rel_dir, root, stop): + try: + with os.scandir(folder) as iterator: + entries = list(iterator) + except OSError: + return + for entry in entries: + if stop is not None and stop(): + return + relative = rel_dir + (entry.name,) + path = Path(entry.path) + try: + if preview_location.is_link_like(path): + scan.links += 1 + continue + info = entry.stat(follow_symlinks=False) + except OSError: + continue + if stat_module.S_ISDIR(info.st_mode): + _walk(scan, path, relative, root, stop) + continue + if not stat_module.S_ISREG(info.st_mode): + scan.left.append(relative) + continue + if is_temp_name(entry.name): + scan.temps.append(relative) + continue + if is_os_metadata(entry.name): + continue # removed only when it alone keeps a folder (rmdir) + parts = _preview_parts(entry.name) + if parts is None: + scan.left.append(relative) + continue + if in_trash_folder("/".join(relative)) or not _has_original( + root, rel_dir, *parts + ): + scan.orphans.append(relative) + continue + scan.copy.append((relative, info.st_size, info.st_mtime_ns)) + scan.bytes += info.st_size + + +# ── copying ────────────────────────────────────────────────────────── + + +def rounded_mtime_ns(mtime_ns): + """*mtime_ns* rounded up to the next 2-second step (FAT/exFAT).""" + return -(-mtime_ns // MTIME_STEP_NS) * MTIME_STEP_NS + + +def too_long(path, long_paths=None): + """A destination (with its temp name) over Windows' file limit.""" + if os.name != "nt" and long_paths is not False: + return False + if long_paths is None: + long_paths = preview_location.long_paths_enabled() + if long_paths: + return False + return len(str(path)) + TEMP_EXTRA > WINDOWS_FILE_LIMIT + + +class CopyFailed(Exception): + """One preview could not be copied (a per-file error, D10A).""" + + +class FolderGone(Exception): + """The destination cannot take writes any more (abort the move).""" + + +def _make_folders(base, relative_dir, created): + """Make ``base/relative_dir`` one folder at a time (*base* must exist); + every folder made is added to *created*.""" + current = Path(base) + for part in relative_dir: + current = current / part + try: + os.mkdir(current) + created.append(str(current)) + except FileExistsError: + pass # already there: made by an earlier copy into this folder + except OSError as error: + if error.errno in (errno.ENOSPC, errno.EDQUOT, errno.EROFS): + raise FolderGone(str(error)) from error + raise CopyFailed( + "Its folder could not be made; made again on view" + ) from error + if preview_location.is_link_like(current): + raise CopyFailed("a link in the destination") + + +def refuse_links(base, relative_dir): + """Raise CopyFailed when *base* or an existing folder on the way to + ``base/relative_dir`` is a link (never copy through one).""" + current = Path(base) + for part in (None, *relative_dir): + if part is not None: + current = current / part + if preview_location.is_link_like(current): + raise CopyFailed("a link in the destination") + + +def copy_one(source, destination, size, mtime_ns, created, *, long_paths=None): + """Copy one preview; returns ``"moved"`` or ``"skipped"`` (already + there). Raises CopyFailed (this file) or FolderGone (the destination).""" + destination = Path(destination) + if too_long(destination, long_paths): + raise CopyFailed("Path too long for Windows; the original is shown") + wanted = rounded_mtime_ns(mtime_ns) + try: + existing = os.stat(destination, follow_symlinks=False) + except FileNotFoundError: + existing = None + except OSError as error: + raise FolderGone(str(error)) from error + if ( + existing is not None + and stat_module.S_ISREG(existing.st_mode) + and existing.st_size == size + and existing.st_mtime_ns == wanted + ): + return "skipped" + temp = destination.with_name( + destination.name + MOVE_TEMP_SUFFIX.format(token=secrets.token_hex(4)) + ) + try: + shutil.copyfile(source, temp) + if os.stat(temp).st_size != size: + raise CopyFailed("The copy did not match; made again on view") + os.utime(temp, ns=(wanted, wanted)) + os.replace(temp, destination) + except CopyFailed: + _unlink(temp) + raise + except FileNotFoundError as error: + _unlink(temp) + if not Path(source).exists(): + raise CopyFailed("Could not be read; made again on view") from error + raise FolderGone(str(error)) from error + except OSError as error: + _unlink(temp) + if error.errno in (errno.ENOSPC, errno.EDQUOT, errno.EROFS): + raise FolderGone(str(error)) from error + raise CopyFailed("Could not be copied; made again on view") from error + return "moved" + + +def _unlink(path): + """Remove a copy's temp file after a failed copy; True when it is gone. + + One that cannot go (a Windows lock, permissions) is logged with its path; + the next start's temp sweep removes it (``.pm-move-*.tmp``).""" + try: + os.unlink(path) + return True + except FileNotFoundError: + return True # never written, or already renamed into place + except OSError as error: + logger.warning(f"Could not remove the move's temp file {path}: {error}") + return False + + +# ── removing ───────────────────────────────────────────────────────── + + +def _in_use(error): + return getattr(error, "winerror", None) in _WINDOWS_IN_USE + + +def remove_file(path, *, sleep=time.sleep): + """Remove one file; a file another program holds open (Windows) is + retried 3 times. Returns True when it is gone.""" + for wait in (0.0,) + REMOVE_RETRIES: + if wait: + sleep(wait) + try: + os.unlink(path) + return True + except FileNotFoundError: + return True + except OSError as error: + if not _in_use(error): + return False + return False + + +def remove_empty_folders(top, *, keep_top=False): + """``rmdir`` the empty folders below *top* (and *top*), bottom-up, never + through a link. A folder holding only OS metadata files loses them first. + Returns the folders that could not be removed.""" + top = Path(top) + left = [] + if not top.is_dir() or preview_location.is_link_like(top): + return left + # Top-down, pruning every link, junction or reparse point, then bottom-up: + # never into a folder outside *top* (Windows junctions included). + folders = [] + for folder, dirs, _files in os.walk(top): + folder = Path(folder) + dirs[:] = [d for d in dirs if not preview_location.is_link_like(folder / d)] + folders.append(folder) + for folder in reversed(folders): + if folder == top and keep_top: + continue + try: + names = os.listdir(folder) + except OSError: + left.append(folder) + continue + if names and all(is_os_metadata(name) for name in names): + for name in names: + remove_file(folder / name) + try: + os.rmdir(folder) + except OSError: + left.append(folder) + return left diff --git a/py/api/thumbnail_move_plan.py b/py/api/thumbnail_move_plan.py new file mode 100644 index 00000000..df70500d --- /dev/null +++ b/py/api/thumbnail_move_plan.py @@ -0,0 +1,492 @@ +"""Move thumbnails, the review (#320 Part 3 phase 1): what a move would copy, +remove and leave, checked before anything is written.""" + +import hashlib +import json +import os +import secrets +import shutil +import time +import uuid +from dataclasses import dataclass, field +from pathlib import Path + +from aiohttp import web + +from . import preview_location, preview_store +from . import thumbnail_move_files as files +from .job_routes import busy_response +from .jobs import THUMBNAILS_JOB_KIND, JobBusy +from .thumbnail_location import default_path + +try: + from ...utils import library_location +except ImportError: # tests import the package top-level + from utils import library_location + +MOVE_TITLE = "Move thumbnails" +FINISHING_TITLE = "Move thumbnails (finishing)" +PRUNE_TITLE = "Remove thumbnails of removed sources" +MOVE_GROUPS = ("moved", "skipped", "left", "errors") +PRUNE_GROUPS = ("removed", "left", "errors") +PLAN_TTL = 600.0 +JOURNAL_NAME = "thumbnail-move.json" +SPACE_MARGIN = 64 * 2**20 +SPACE_SHARE = 0.05 +LISTED = 5 +SENTENCE = ( + "Every thumbnail is copied and checked before Prompt Manager switches. " + "Scans, deletes and tagging wait until it finishes; closing this page does " + "not stop it." +) + + +def _number(value): + return f"{value:,}" + + +def _plural(count, one, many=None): + return f"{_number(count)} {one if count == 1 else (many or one + 's')}" + + +def _size(value): + size = float(value) + for unit in ("B", "KB", "MB", "GB"): + if size < 1024 or unit == "GB": + return f"{size:.1f} {unit}" if unit != "B" else f"{int(size)} B" + size /= 1024 + return f"{size:.1f} GB" + + +def _refusal(code, status=None, **extra): + known_status, text = preview_location.MESSAGES.get(code, (400, code)) + return web.json_response( + {"success": False, "code": code, "error": text, **extra}, + status=status or known_status, + ) + + +def journal_path(): + """``/data/thumbnail-move.json`` (only while a move copies).""" + return library_location.cache_dir(JOURNAL_NAME) + + +def _read_json(path): + try: + with open(path, "r", encoding="utf-8") as handle: + data = json.load(handle) + except (OSError, ValueError): + return None + return data if isinstance(data, dict) else None + + +def _write_json(path, data): + os.makedirs(os.path.dirname(path), exist_ok=True) + temp = f"{path}.{os.getpid()}.tmp" + with open(temp, "w", encoding="utf-8") as handle: + json.dump(data, handle, indent=2) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temp, path) + + +def _remove(path): + try: + os.unlink(path) + except FileNotFoundError: + pass # already gone: removing it is the goal + + +def _is_network(path): + text = str(path) + if text.startswith(("\\\\", "//")): + return True + if os.name == "nt": + try: + import ctypes + + drive = os.path.splitdrive(os.path.abspath(text))[0] + "\\" + return ctypes.windll.kernel32.GetDriveTypeW(drive) == 4 # DRIVE_REMOTE + except (AttributeError, OSError): + return False + return False + + +def _volume(path): + current = Path(os.path.abspath(path)) + while not current.exists() and current.parent != current: + current = current.parent + try: + return os.stat(current).st_dev + except OSError: + return None + + +def _free(path): + current = Path(os.path.abspath(path)) + while not current.exists() and current.parent != current: + current = current.parent + try: + return shutil.disk_usage(current).free + except OSError: + return None + + +# ── the plan ───────────────────────────────────────────────────────── + + +@dataclass +class SourcePlan: + """One listed source: its old store, its new store and what moves.""" + + root: str + label: str + key: str | None + old_store: str | None + new_store: str + scan: files.StoreScan | None + available: bool = True + too_long: list = field(default_factory=list) + + +@dataclass +class MovePlan: + """A reviewed move: from, to, the sources and the removals.""" + + token: str + created: float + revision: str + old: preview_location.ParsedLocation + target: dict # {"mode", "path"} + target_id: str | None + target_exists: bool + target_default: bool + install: str + sources: list + unlisted: list # (key, store path, scan) + needs_bytes: int + confirm_required: bool + + @property + def to_folder(self): + return self.target["mode"] == preview_store.FOLDER_MODE + + @property + def from_folder(self): + return self.old.mode == preview_store.FOLDER_MODE + + +def _roots_revision(roots): + text = json.dumps(sorted(str(root) for root in roots)) + return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16] + + +class ThumbnailMovePlanMixin: + """The move's review and its routes.""" + + _move_plans = None + + # ── routes ──────────────────────────────────────────────────────── + + def _register_thumbnail_move_routes(self, routes): + @routes.post("/prompt_manager/images/thumbnails/plan") + async def thumbnail_move_plan_route(request): + return await self.plan_thumbnail_move(request) + + @routes.post("/prompt_manager/images/thumbnails/location/recreate") + async def thumbnail_location_recreate_route(request): + return await self.recreate_thumbnail_folder(request) + + @routes.post("/prompt_manager/images/thumbnails/location/adopt") + async def thumbnail_location_adopt_route(request): + return await self.adopt_thumbnail_folder(request) + + # ── review ──────────────────────────────────────────────────────── + + async def plan_thumbnail_move(self, request): + try: + body = await request.json() + except (ValueError, UnicodeDecodeError): + body = None + if not isinstance(body, dict) or body.get("mode") != "move": + return web.json_response( + { + "success": False, + "code": "invalid-request", + "error": "mode must be move", + }, + status=400, + ) + target = body.get("location") + if not isinstance(target, dict) or target.get("mode") not in ( + preview_store.SOURCE_MODE, + preview_store.FOLDER_MODE, + ): + return web.json_response( + { + "success": False, + "code": "invalid-request", + "error": "location must name a mode", + }, + status=400, + ) + blocker = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded") + if blocker is not None: + return busy_response(JobBusy(blocker)) + try: + outcome = await self._run_in_executor(self._build_move_plan, target) + except Exception: + self.logger.exception("The thumbnail move could not be planned") + return web.json_response( + { + "success": False, + "code": None, + "error": "The move could not be planned", + }, + status=500, + ) + if isinstance(outcome, str): + return _refusal(outcome) + plan, payload = outcome + if payload.get("insufficient"): + return _refusal( + "insufficient-space", + status=507, + error_detail=payload["insufficient"], + plan=payload, + ) + self._store_move_plan(plan) + return web.json_response({"success": True, **payload}) + + def _store_move_plan(self, plan): + if self._move_plans is None: + self._move_plans = {} + now = time.monotonic() + self._move_plans = { + token: kept + for token, kept in self._move_plans.items() + if now - kept.created < PLAN_TTL + } + self._move_plans[plan.token] = plan + + def _build_move_plan(self, target): + """``(plan, payload)`` or a refusal code (blocking).""" + old = preview_store.setting() + location = preview_store.current() + install = old.install or str(uuid.uuid4()) + target_path = None + default = default_path() + if target["mode"] == preview_store.FOLDER_MODE: + raw = target.get("path") + if not isinstance(raw, str): + return "location-not-absolute" + context = self._thumbnail_location_context() + code = preview_location.location_problem( + raw, purpose="move", context=context + ) + if code is not None: + return code + target_path = os.path.abspath(raw) + elif old.mode == preview_store.SOURCE_MODE and not old.problem: + return "location-same" + target_default = bool(default) and os.path.normcase( + target_path or "" + ) == os.path.normcase(os.path.abspath(default)) + target_exists = bool(target_path) and os.path.isdir(target_path) + target_marker = ( + preview_location.read_marker(target_path) if target_exists else None + ) + roots = [str(root) for root in self._get_all_output_dirs()] + labels = self._source_labels(roots) + old_keys = {} + if old.mode == preview_store.FOLDER_MODE and location.available: + marker = preview_location.read_marker(old.path) or {} + old_keys = marker.get("keys") or {} + new_keys = dict((target_marker or {}).get("keys") or {}) + taken = {key: entry.get("identity") for key, entry in new_keys.items()} + sources = [] + for root in roots: + identity = os.path.normcase(os.path.realpath(root)) + dev_ino = preview_location.file_identity(root) + old_key = None + old_store = None + available = True + if old.mode == preview_store.SOURCE_MODE: + old_store = str(preview_store.source_folder(root)) + elif location.available: + old_key = preview_location.find_key(old_keys, root, identity, dev_ino) + if old_key: + old_store = os.path.join(old.path, old_key) + else: + available = False + key = None + if target_path: + key = old_key or preview_location.find_key( + new_keys, root, identity, dev_ino + ) + if key is None or taken.get(key, identity) != identity: + key = preview_location.source_key(root, identity, taken) + taken[key] = identity + new_store = os.path.join(target_path, key) + else: + new_store = str(preview_store.source_folder(root)) + scan = None + if old_store: + scan = files.scan_store(old_store, root) + plan = SourcePlan( + root, labels.get(root, root), key, old_store, new_store, scan, available + ) + if scan is not None and target_path: + plan.too_long = [ + item[0] + for item in scan.copy + if files.too_long(Path(new_store).joinpath(*item[0])) + ] + sources.append(plan) + unlisted = self._unlisted_keys(old, location, old_keys, install) + needs = sum(source.scan.bytes for source in sources if source.scan is not None) + plan = MovePlan( + token=secrets.token_urlsafe(16), + created=time.monotonic(), + revision=_roots_revision(roots), + old=old, + target={"mode": target["mode"], "path": target_path}, + target_id=(target_marker or {}).get("id"), + target_exists=target_exists, + target_default=target_default, + install=install, + sources=sources, + unlisted=unlisted, + needs_bytes=needs, + confirm_required=bool(target_path) and not target_default, + ) + return plan, self._plan_payload(plan, location) + + def _unlisted_keys(self, old, location, keys, install): + """This install's keys whose source no longer is in any root list + (``(key, store, scan)``), in the folder in use.""" + if old.mode != preview_store.FOLDER_MODE or not location.available: + return [] + listed = { + os.path.normcase(os.path.realpath(root)) + for root in self._all_source_roots() + } + found = [] + for key, entry in sorted(keys.items()): + if not preview_location.valid_key(key) or not isinstance(entry, dict): + continue # a marker is data: never a path out of the folder + if entry.get("install") not in (install, None): + continue + if entry.get("identity") in listed: + continue + store = os.path.join(old.path, key) + found.append( + (key, store, files.scan_store(store, entry.get("path") or store)) + ) + return found + + def _source_labels(self, roots): + from . import sources + + labels = sources._comfyui_labels() + return {root: sources.source_label(root, labels) for root in roots} + + def _plan_payload(self, plan, location): + sources = plan.sources + copies = sum(len(s.scan.copy) for s in sources if s.scan) + orphans = sum(len(s.scan.orphans) for s in sources if s.scan) + temps = sum(len(s.scan.temps) for s in sources if s.scan) + links = sum(s.scan.links for s in sources if s.scan) + left = [ + f"{s.label} · {'/'.join(item)}" + for s in sources + if s.scan + for item in s.scan.left + ] + too_long = [ + f"{s.label} · {'/'.join(item)}" for s in sources for item in s.too_long + ] + unlisted_count = sum( + len(scan.copy) + len(scan.orphans) for _k, _s, scan in plan.unlisted + ) + free = None + insufficient = None + if plan.needs_bytes: + volumes = {} + for source in sources: + if source.scan is None or not source.scan.bytes: + continue + volume = _volume(source.new_store) + entry = volumes.setdefault(volume, [source.new_store, 0]) + entry[1] += source.scan.bytes + for where, needed in volumes.values(): + available = _free(where) + if available is None: + continue + free = available if free is None else min(free, available) + margin = max(SPACE_MARGIN, int(needed * SPACE_SHARE)) + if needed + margin > available: + insufficient = { + "needed": needed + margin, + "free": available, + "text": ( + f"There is not enough space for the thumbnails: " + f"{_size(available)} free, {_size(needed + margin)} " + "needed. Free some space or choose another folder." + ), + } + payload = { + "mode": "move", + "from": { + "mode": plan.old.mode, + "path": plan.old.path, + "available": location.available, + }, + "to": { + "mode": plan.target["mode"], + "path": plan.target["path"], + "exists": plan.target_exists, + "will_create": bool(plan.target["path"]) and not plan.target_exists, + "default": plan.target_default, + }, + "sources": [ + { + "label": s.label, + "key": s.key, + "files": len(s.scan.copy) if s.scan else 0, + "bytes": s.scan.bytes if s.scan else 0, + "too_long": len(s.too_long), + "links": s.scan.links if s.scan else 0, + "orphans": len(s.scan.orphans) if s.scan else 0, + "left_in_place": len(s.scan.left) if s.scan else 0, + "available": s.available, + } + for s in sources + ], + "totals": { + "sources": len(sources), + "files": copies, + "bytes": plan.needs_bytes, + "orphans": orphans, + "temps": temps, + "links": links, + "left_in_place": len(left), + "too_long": len(too_long), + "unlisted_sources": len(plan.unlisted), + "unlisted_files": unlisted_count, + }, + "too_long_listed": too_long[:LISTED], + "left_listed": left[:LISTED], + "unlisted": [ + {"key": key, "files": len(scan.copy) + len(scan.orphans)} + for key, _store, scan in plan.unlisted + ], + "needs_bytes": plan.needs_bytes, + "free_bytes": free, + "confirm_required": plan.confirm_required, + "token": plan.token, + "sentence": SENTENCE, + } + if insufficient: + payload["insufficient"] = insufficient + return payload diff --git a/py/api/thumbnail_move_recovery.py b/py/api/thumbnail_move_recovery.py new file mode 100644 index 00000000..347557d9 --- /dev/null +++ b/py/api/thumbnail_move_recovery.py @@ -0,0 +1,246 @@ +"""Move thumbnails after a restart and the unavailable folder's actions +(#320 Parts 3 and 4): undo an interrupted copy, finish a removal that +waited, Recreate folder and Use this folder.""" + +import os +import threading +from pathlib import Path + +from aiohttp import web + +from . import preview_location, preview_store +from . import thumbnail_move_files as files +from .job_routes import busy_response +from .jobs import THUMBNAILS_JOB_KIND, JobBusy +from .thumbnail_move_plan import ( + FINISHING_TITLE, + MOVE_GROUPS, + _read_json, + _refusal, + _remove, +) +from . import thumbnail_move_plan as move_plan + + +class ThumbnailMoveRecoveryMixin: + """Startup recovery, the finishing job, Recreate and Use this folder.""" + + _move_recover_lock = threading.Lock() + # Per API: set once recover_thumbnail_move has run (a move waits for it). + _move_recovered = False + + # ── startup ─────────────────────────────────────────────────────── + + def recover_thumbnail_move(self): + """Startup (B4): undo an interrupted copy, or forget a journal the + switch already made true. Idempotent; the move start waits for it.""" + with self._move_recover_lock: + if self._move_recovered: + return + try: + self._recover_journal() + except Exception: + self.logger.warning( + "The interrupted thumbnail move check failed", exc_info=True + ) + self._move_recovered = True + + def _recover_journal(self): + path = move_plan.journal_path() + journal = _read_json(path) + if journal is None: + return + setting = preview_store.setting() + target = journal.get("to") or {} + now = setting.path if setting.mode == preview_store.FOLDER_MODE else None + if setting.mode == target.get("mode") and ( + now is None + or os.path.normcase(now) == os.path.normcase(target.get("path") or "") + ): + _remove(path) # the switch happened; only the journal was left + return + for store in journal.get("stores") or []: + if not isinstance(store, str) or not os.path.isdir(store): + continue + for current, dirs, names in os.walk(store): + dirs[:] = [ + d + for d in dirs + if not preview_location.is_link_like(Path(current) / d) + ] + for name in names: + if files.MOVE_TEMP.search(name): + files.remove_file(Path(current) / name) + for folder in reversed(journal.get("created_dirs") or []): + if not isinstance(folder, str) or not os.path.isdir(folder): + continue + for current, dirs, names in os.walk(folder): + dirs[:] = [ + d + for d in dirs + if not preview_location.is_link_like(Path(current) / d) + ] + for name in names: + if files._preview_parts(name) or files.is_temp_name(name): + files.remove_file(Path(current) / name) + files.remove_empty_folders(folder) + folder = target.get("path") + if journal.get("created_folder") and folder and os.path.isdir(folder): + names = set(os.listdir(folder)) + if names <= {preview_location.MARKER_NAME}: + _remove(preview_location.marker_path(folder)) + files.remove_empty_folders(folder) + _remove(path) + self.logger.info("An interrupted thumbnail move was undone; nothing changed") + + def _cleanup_reachable(self, entry): + folder = entry.get("folder") + if folder: + return os.path.isdir(folder) + return True + + async def finish_thumbnail_move(self, _app=None): + """Startup: recover, then start "Move thumbnails (finishing)" when old + copies wait and every one is reachable (on the server's loop).""" + await self._run_in_executor(self.recover_thumbnail_move) + cleanup = list(preview_store.setting().cleanup) + if not cleanup: + return None + reachable = await self._run_in_executor( + lambda: all(self._cleanup_reachable(entry) for entry in cleanup) + ) + if not reachable: + return None + try: + return self._jobs.start( + THUMBNAILS_JOB_KIND, + FINISHING_TITLE, + admission="guarded", + cancellable=False, + scope={"mode": "move", "phase": "remove"}, + groups=MOVE_GROUPS, + runner=self._run_finishing, + ) + except JobBusy: + return None + + async def _run_finishing(self, job): + await self._remove_old_copies(job, None) + left = len(preview_store.setting().cleanup) + self._end_move( + job, + "finished", + ( + "Old thumbnails removed." + if not left + else ( + "Some old thumbnails are still in use; they are removed at " + "the next start." + ) + ), + ) + + # ── Recreate and Use this folder (Part 4) ───────────────────────── + + async def recreate_thumbnail_folder(self, _request): + blocker = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded") + if blocker is not None: + return busy_response(JobBusy(blocker)) + code = await self._run_in_executor(self._recreate_folder) + if code is not None: + status = ( + 409 if code in ("location-has-marker", "location-not-empty") else 400 + ) + if code == "location-has-marker": + return web.json_response( + { + "success": False, + "code": code, + "error": "The folder is back with its thumbnails. Nothing " + "was recreated.", + }, + status=status, + ) + return _refusal(code, status=status) + return web.json_response( + { + "success": True, + "location": self._thumbnail_location_payload(with_path=True), + } + ) + + def _recreate_folder(self): + setting = preview_store.setting() + if setting.mode != preview_store.FOLDER_MODE or not setting.path: + return "location-same" + folder = setting.path + if os.path.isdir(folder): + if preview_location.read_marker(folder) is not None: + return "location-has-marker" + if not preview_location.holds_only_metadata(folder): + return "location-not-empty" + elif not os.path.isdir(os.path.dirname(folder)): + return "location-parent-missing" + else: + try: + os.mkdir(folder) + except OSError: + return "location-not-writable" + try: + marker_id = preview_location.init_folder(folder, created_by_pm=True) + except OSError: + return "location-not-writable" + return self._commit_location_id(marker_id) + + async def adopt_thumbnail_folder(self, _request): + blocker = self._jobs.blocker(THUMBNAILS_JOB_KIND, admission="guarded") + if blocker is not None: + return busy_response(JobBusy(blocker)) + code = await self._run_in_executor(self._adopt_folder) + if code is not None: + return _refusal(code) + return web.json_response( + { + "success": True, + "location": self._thumbnail_location_payload(with_path=True), + } + ) + + def _adopt_folder(self): + setting = preview_store.setting() + if setting.mode != preview_store.FOLDER_MODE or not setting.path: + return "location-same" + try: + marker = preview_location.read_marker(setting.path) + except OSError: + return "location-unavailable" + if marker is None: + return "location-unavailable" + return self._commit_location_id(marker["id"]) + + def _commit_location_id(self, marker_id): + from ..config import PromptManagerConfig + from ..config_store import get_config_store + + value = dict(preview_store.setting().to_config()) + value["id"] = marker_id + + def mutate(config): + section = config.get("thumbnails") + if not isinstance(section, dict): + section = {} + section["location"] = value + config["thumbnails"] = section + return config + + def publish(_config): + PromptManagerConfig.THUMBNAIL_LOCATION = value + preview_store.configure(value, check=False) + + try: + get_config_store().commit(mutate, expected_revision=None, publish=publish) + except Exception: + self.logger.exception("The thumbnail folder could not be saved") + return "location-not-writable" + preview_store.refresh() + return None diff --git a/py/api/thumbnail_prune.py b/py/api/thumbnail_prune.py new file mode 100644 index 00000000..0ffad194 --- /dev/null +++ b/py/api/thumbnail_prune.py @@ -0,0 +1,175 @@ +"""Remove thumbnails of removed sources (#320 Part 5, D200): a Thumbnail +maintenance scope, offered only in folder mode (31A). + +*Removed* means a key of this install in the thumbnail folder's marker whose +identity matches no root in any root list. The job removes the +preview-grammar files and temps of those key folders, ``rmdir``s the empty +folders, drops the keys from the marker and lists anything else as "Left in +place". Kind ``thumbnails``, ``mode: "prune"``. +""" + +import functools +import os +from pathlib import Path + +from aiohttp import web + +from . import preview_location, preview_store +from . import thumbnail_move_files as files +from .jobs import THUMBNAILS_JOB_KIND +from .thumbnail_location import location_refusal +from .thumbnail_move_plan import _number, _plural, _size + +PRUNE_TITLE = "Remove thumbnails of removed sources" +PRUNE_GROUPS = ("removed", "left", "errors") + + +def _prune_line(count, files_count, size): + if not count: + return "No sources were removed: nothing to remove." + return ( + f"{_plural(count, 'source')} no longer listed: " + f"{_plural(files_count, 'thumbnail')} ({_size(size)}) are removed." + ) + + +class ThumbnailPruneMixin: + """The prune plan and job.""" + + def _prune_targets(self): + """``[(key, store, scan)]`` of this install's unlisted keys (blocking).""" + setting = preview_store.setting() + location = preview_store.current() + if setting.mode != preview_store.FOLDER_MODE or not location.available: + return [] + marker = preview_location.read_marker(setting.path) or {} + install = setting.install + return self._unlisted_keys(setting, location, marker.get("keys") or {}, install) + + def _prune_payload(self, targets): + files_count = sum( + len(scan.copy) + len(scan.orphans) for _k, _s, scan in targets + ) + size = sum(scan.bytes for _k, _s, scan in targets) + return { + "success": True, + "mode": "prune", + "sources": [ + { + "key": key, + "files": len(scan.copy) + len(scan.orphans), + "bytes": scan.bytes, + "left_in_place": len(scan.left), + } + for key, _store, scan in targets + ], + "targets": files_count, + "bytes": size, + "line": _prune_line(len(targets), files_count, size), + "folder": preview_store.setting().path, + } + + async def get_prune_plan(self): + refused = location_refusal() + if refused is not None: + return refused + if preview_store.current().mode != preview_store.FOLDER_MODE: + return web.json_response( + { + "success": False, + "code": "invalid-request", + "error": "Thumbnails are kept inside each source: no removed " + "source has thumbnails here.", + }, + status=400, + ) + targets = await self._run_in_executor(self._prune_targets) + return web.json_response( + self._prune_payload(targets), headers={"Cache-Control": "no-store"} + ) + + def start_thumbnail_prune(self): + """Start the prune job (sync, on the loop); raises JobBusy.""" + return self._jobs.start( + THUMBNAILS_JOB_KIND, + PRUNE_TITLE, + admission="guarded", + cancellable=True, + scope={"mode": "prune"}, + groups=PRUNE_GROUPS, + runner=self._run_thumbnail_prune, + ) + + async def _run_thumbnail_prune(self, job): + targets = await self._run_in_executor(self._prune_targets) + job.update(total=len(targets), processed=0) + setting = preview_store.setting() + for done, (key, store, scan) in enumerate(targets, 1): + if job.cancelled: + break + removed, failed = await self._run_in_executor( + functools.partial(self._prune_store, store) + ) + job.record_counted( + "removed", removed, [{"path": key, "detail": "Thumbnails removed"}] + ) + if failed: + job.record_counted("errors", failed) + if scan.left: + job.record_counted( + "left", + len(scan.left), + [{"path": f"{key}/{'/'.join(item)}"} for item in scan.left], + ) + await self._run_in_executor( + functools.partial(self._forget_key, setting.path, key, store) + ) + job.update( + processed=done, + progress=int(done * 100 / len(targets)), + status_line=f"Removing thumbnails of {_number(done)} of " + f"{_number(len(targets))} removed sources", + ) + job.publish({"type": "progress", "summary": job.summary()}) + line = f"{_number(job.counters.get('removed', 0))} thumbnails removed" + if job.counters.get("errors"): + line += f" · {_plural(job.counters['errors'], 'error')}" + job.update(status_line=line, progress=100) + job.log_line(line) + job.finish("cancelled" if job.cancelled else "finished") + + @staticmethod + def _prune_store(store): + """Remove the previews and temps of one key folder; ``(removed, failed)``.""" + removed = failed = 0 + store = Path(store) + if not store.is_dir() or preview_location.is_link_like(store): + return 0, 0 + for current, dirs, names in os.walk(store): + dirs[:] = [ + d for d in dirs if not preview_location.is_link_like(Path(current) / d) + ] + for name in names: + path = Path(current) / name + if not (files.is_temp_name(name) or files._preview_parts(name)): + continue + if preview_location.is_link_like(path): + continue + if files.remove_file(path): + removed += 1 + else: + failed += 1 + files.remove_empty_folders(store) + return removed, failed + + @staticmethod + def _forget_key(folder, key, store): + """Drop *key* from the marker once its folder is gone.""" + if os.path.exists(store): + return + + def forget(marker): + marker["keys"].pop(key, None) + + with preview_store.write_lock: + preview_location.write_marker(folder, forget) diff --git a/py/api/thumbnail_settings.py b/py/api/thumbnail_settings.py index c6bbc5d3..e07d83dd 100644 --- a/py/api/thumbnail_settings.py +++ b/py/api/thumbnail_settings.py @@ -114,9 +114,16 @@ def validate_edge(value): # ── names ──────────────────────────────────────────────────────────── +def preview_stem(rel_path): + """``_thumb`` (POSIX): a preview's name inside the + store of its source (``preview_store``).""" + return f"{Path(rel_path).with_suffix('').as_posix()}{THUMB_SUFFIX}" + + def thumbnail_stem_rel(rel_path): - """``thumbnails/_thumb`` (POSIX).""" - return f"thumbnails/{Path(rel_path).with_suffix('').as_posix()}{THUMB_SUFFIX}" + """``thumbnails/_thumb`` (POSIX): URL text only. + Where the preview is on disk is ``preview_store``'s answer.""" + return f"thumbnails/{preview_stem(rel_path)}" def thumbnail_rel(rel_path, extension=None): @@ -207,9 +214,20 @@ def owned_in_folder(folder, stem, source_extension, preferred=None): def _owned(root, rel_path, preferred): + from . import preview_store + rel_path = Path(rel_path) - stem = Path(root) / thumbnail_stem_rel(rel_path) - return owned_in_folder(stem.parent, stem.name, rel_path.suffix, preferred) + store = preview_store.store_dir(root) + if store is None: + return iter(()) + stem = store / preview_stem(rel_path) + found = owned_in_folder(stem.parent, stem.name, rel_path.suffix, preferred) + if preview_store.current().mode == preview_store.SOURCE_MODE: + return found + # Folder mode: timed, so a stalled share marks the folder unreachable, + # and only previews no link takes out of the key folder (deletes use it). + with preview_store.store_io(): + return iter([path for path in found if preview_store.may_write(path)]) def owned_thumbnails(root, rel_path, preferred=None): diff --git a/py/config.py b/py/config.py index 4af38a5a..91db4dae 100644 --- a/py/config.py +++ b/py/config.py @@ -940,6 +940,8 @@ class PromptManagerConfig: THUMBNAIL_QUALITY = 80 THUMBNAIL_EDGE = 512 THUMBNAIL_EDGE_RANGE = (128, 2048) + # #320 thumbnail location (config.json ``thumbnails.location``), as read. + THUMBNAIL_LOCATION = None @classmethod def max_worker_threads(cls) -> int: @@ -1001,6 +1003,12 @@ class PromptManagerConfig: "format": cls.THUMBNAIL_FORMAT, "quality": cls.THUMBNAIL_QUALITY, "edge": cls.THUMBNAIL_EDGE, + # #320: as read; only the Move thumbnails job writes it. + **( + {"location": cls.THUMBNAIL_LOCATION} + if cls.THUMBNAIL_LOCATION is not None + else {} + ), }, "gallery": GalleryConfig.get_config(), "integrations": IntegrationConfig.get_config(), @@ -1175,6 +1183,11 @@ class PromptManagerConfig: low, high = cls.THUMBNAIL_EDGE_RANGE if type(edge) is int and low <= edge <= high: cls.THUMBNAIL_EDGE = edge + # #320: kept as read (``preview_location.parse_location`` checks it; + # a bad value becomes an unavailable ``invalid`` location, never a + # silent fall-back to the sources). + if "location" in section: + cls.THUMBNAIL_LOCATION = section["location"] # Load configuration on import diff --git a/tests/browser/server.py b/tests/browser/server.py index 5d56771f..3f6fd7e6 100644 --- a/tests/browser/server.py +++ b/tests/browser/server.py @@ -22,6 +22,9 @@ from PIL.PngImagePlugin import PngInfo from tests.browser.folder_trash import close_trash, replace_folder from tests.browser.malloc_tuning import pin_mmap_threshold from tests.browser.request_hold import register_request_holds +from tests.browser.thumbnail_location_fixture import ( + register_thumbnail_location_commands, +) from tests.browser import slow_reset_watch from tests.browser.fast_reset import SEED_ON_LIVE_ENV, ScratchSeeder from tests.browser.workspace_activity_fixture import register_activity_commands @@ -249,6 +252,7 @@ def create_app(root: Path) -> web.Application: library_editor_commands = register_library_editor_commands(routes, api, root) register_vocabulary_commands(routes, api, root) restore_sources = register_source_change_commands(routes, api) + reset_thumbnail_location = register_thumbnail_location_commands(routes, api, root) register_tagging_commands(routes, api, root, model, on_reset=restore_sources) register_review_commands(routes, api, root) register_collections_commands(routes, api, root) @@ -460,6 +464,8 @@ def create_app(root: Path) -> web.Application: restore_sources() # No preview queued by the last test may land in the new fixture. await api._run_in_executor(api.close_auto_thumbnails) + # Previews inside each source again (#320). + await api._run_in_executor(reset_thumbnail_location) library_editor_commands.reset() await api._run_in_executor(reset_logs) data = await request.json() if request.can_read_body else {} @@ -604,6 +610,7 @@ def create_app(root: Path) -> web.Application: history_commands.reset() library_editor_commands.reset() reveal_stub.reset() + await api._run_in_executor(reset_thumbnail_location) history_fixture = None manifest = await api._run_in_executor( seeder.run, diff --git a/tests/browser/system-concept.spec.mjs b/tests/browser/system-concept.spec.mjs index 6018d5ce..33b453bd 100644 --- a/tests/browser/system-concept.spec.mjs +++ b/tests/browser/system-concept.spec.mjs @@ -656,9 +656,10 @@ function mapSourcesConcept(reference, { settings, sources }) { // D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count. return [{ ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } }]; }), - // D158: the inserted row after the first setting row; D98: "Refresh files" closes the surface. + // D158: the inserted row after the first setting row; D196 (#320): the Thumbnail location block after + // "Thumbnail generation"; D98: "Refresh files" closes the surface. surface: { ...reference.surface, tree: [...reference.surface.tree.slice(0, rowAt + 1), 'div.setting-row', - ...reference.surface.tree.slice(rowAt + 1), 'div.operation'] }, + ...reference.surface.tree.slice(rowAt + 1), 'div.setting-block', 'div.operation'] }, operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title, description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }], }; diff --git a/tests/browser/thumbnail_location_fixture.py b/tests/browser/thumbnail_location_fixture.py new file mode 100644 index 00000000..0ee2ed6f --- /dev/null +++ b/tests/browser/thumbnail_location_fixture.py @@ -0,0 +1,170 @@ +"""The thumbnail location on the test server (#320), undone by every reset. + +Folder mode is reachable from config.json and tests only until the Move +thumbnails job lands, so page tests set it here, inside the server's root: + +- ``POST /__test__/thumbnail-location`` ``{"mode": "folder"}`` (or + ``"source"``): previews live in ``/thumbs/`` from now on (a new + folder with Prompt Manager's marker). +- ``GET /__test__/thumbnail-files``: the preview files in the thumbnail + folder and whether any source grew a ``thumbnails/`` folder. +- ``POST /__test__/thumbnail-folder/remove``: the thumbnail folder goes, as + when its drive is unplugged; the server notices it on its next check. +""" + +from __future__ import annotations + +import shutil +from pathlib import Path +from typing import TYPE_CHECKING, Callable + +from aiohttp import web + +if TYPE_CHECKING: + from py.api import PromptManagerAPI + +FOLDER_NAME = "thumbs" + + +def register_thumbnail_location_commands( + routes: web.RouteTableDef, api: PromptManagerAPI, root: Path +) -> Callable[[], None]: + """Register the commands; returns the reset every reset runs.""" + from py.api import preview_location, preview_store + from py.api import thumbnail_location, thumbnail_move_plan + from py.config import PromptManagerConfig + from py.config_store import get_config_store + + folder = root / FOLDER_NAME + default = root / "default-thumbs" + custom = root / "custom-thumbs" # a folder a page types in + journal = root / "data" / "thumbnail-move.json" + # The move's journal and default folder live in this server's root, never + # in the checkout's data folder. + thumbnail_move_plan.journal_path = lambda: str(journal) + thumbnail_move_plan.default_path = lambda: str(default) + thumbnail_location.default_path = lambda: str(default) + api.recover_thumbnail_move() + + def forget_config(config: dict) -> dict: + section = config.get("thumbnails") + if isinstance(section, dict): + section.pop("location", None) + return config + + real_free = thumbnail_move_plan._free + + def reset() -> None: + thumbnail_move_plan._free = real_free + preview_store.configure(None) + PromptManagerConfig.THUMBNAIL_LOCATION = None + store = get_config_store() + config, _revision = store.read() + if ( + isinstance(config.get("thumbnails"), dict) + and "location" in config["thumbnails"] + ): + store.commit(forget_config, expected_revision=None) + for path in (folder, default, custom): + shutil.rmtree(path, ignore_errors=True) + journal.unlink(missing_ok=True) + + def use(mode: str) -> dict: + reset() + if mode == "source": + return {"mode": "source"} + folder.mkdir() + marker_id = preview_location.init_folder(str(folder), created_by_pm=True) + location = preview_store.configure( + { + "mode": "folder", + "path": str(folder), + "id": marker_id, + "install": "browser-tests", + } + ) + return {"mode": location.mode, "available": location.available} + + @routes.post("/__test__/thumbnail-location") + async def set_location(request: web.Request) -> web.Response: + data = await request.json() + mode = data.get("mode") if isinstance(data, dict) else None + if mode not in ("source", "folder"): + raise web.HTTPBadRequest() + return web.json_response(await api._run_in_executor(use, mode)) + + def files() -> dict: + setting = preview_store.setting() + where = Path(setting.path) if setting.path else folder + previews = sorted( + path.relative_to(where).as_posix() + for path in where.rglob("*_thumb.*") + if path.is_file() + ) + sources = [str(path) for path in api._get_all_output_dirs()] + return { + "folder": previews, + "folder_exists": folder.exists() or default.exists() or custom.exists(), + "custom_path": str(custom), + "source_thumbnails": any( + (Path(source) / "thumbnails").exists() for source in sources + ), + } + + @routes.get("/__test__/thumbnail-files") + async def thumbnail_files(_request: web.Request) -> web.Response: + return web.json_response(await api._run_in_executor(files)) + + def remove() -> bool: + setting = preview_store.setting() + shutil.rmtree(setting.path or folder, ignore_errors=True) + return preview_store.refresh() + + @routes.post("/__test__/thumbnail-folder/remove") + async def remove_folder(_request: web.Request) -> web.Response: + available = await api._run_in_executor(remove) + return web.json_response({"available": available}) + + def foreign() -> bool: + """Another install recreated the folder: its marker has another id.""" + setting = preview_store.setting() + + def other(marker: dict) -> None: + marker["id"] = "another-install" + + preview_location.write_marker(setting.path, other) + return preview_store.refresh() + + @routes.post("/__test__/thumbnail-folder/foreign") + async def foreign_folder(_request: web.Request) -> web.Response: + return web.json_response({"available": await api._run_in_executor(foreign)}) + + def unlisted() -> None: + """A source this install no longer lists left its thumbnails here.""" + setting = preview_store.setting() + + def add(marker: dict) -> None: + marker["keys"]["gone-0123abcd"] = { + "path": "/gone", + "identity": "/gone", + "install": setting.install, + } + + preview_location.write_marker(setting.path, add) + gone = Path(setting.path) / "gone-0123abcd" + gone.mkdir() + (gone / "x_thumb.webp").write_bytes(b"preview") + preview_store.refresh() + + @routes.post("/__test__/thumbnail-folder/unlisted") + async def unlisted_source(_request: web.Request) -> web.Response: + await api._run_in_executor(unlisted) + return web.json_response({"success": True}) + + @routes.post("/__test__/thumbnail-space/full") + async def space_full(_request: web.Request) -> web.Response: + """Every target disk reports 1 byte free until the next reset.""" + thumbnail_move_plan._free = lambda _path: 1 + return web.json_response({"success": True}) + + return reset diff --git a/tests/browser/workflows.spec.mjs b/tests/browser/workflows.spec.mjs index 849839b1..5c79ba19 100644 --- a/tests/browser/workflows.spec.mjs +++ b/tests/browser/workflows.spec.mjs @@ -261,3 +261,20 @@ test('prompt deletion requires confirmation and can retry after a network failur await expect(await request.get(url)).toBeOK(); } }); + +// #320: with previews in one folder, the legacy gallery still shows them, made +// and served from that folder (the fixture's own previews inside the sources +// are never served then). +test('the legacy gallery shows thumbnails kept in one folder', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'folder' } })).toBeOK(); + const served = page.waitForResponse(response => + response.url().includes('/prompt_manager/images/serve/thumbnails/') && response.status() === 200); + await page.goto('/prompt_manager/gallery'); + await expect(page.locator('#galleryGrid .image-item')).toHaveCount(2); + const response = await served; // made in the folder before it was served + const name = decodeURIComponent(new URL(response.url()).pathname.split('/serve/thumbnails/')[1]); + const files = await (await request.get('/__test__/thumbnail-files')).json(); + const stem = name.replace(/\.[^.]+$/, ''); + expect(files.folder.some(file => /^[a-z0-9_-]+-[0-9a-f]{8}\//.test(file) + && file.slice(file.indexOf('/') + 1).replace(/\.[^.]+$/, '') === stem)).toBe(true); +}); diff --git a/tests/browser/workspace-thumbnail-location.spec.mjs b/tests/browser/workspace-thumbnail-location.spec.mjs new file mode 100644 index 00000000..06a94d18 --- /dev/null +++ b/tests/browser/workspace-thumbnail-location.spec.mjs @@ -0,0 +1,213 @@ +import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs'; +import { test, expect } from './fixtures.mjs'; +import { seedConcept } from './concept-inventory.mjs'; + +// #320 Thumbnail location (D196–D201, the approved 320 mock) on the concept-shaped real fixture: the row in System › +// Sources & scans reviews and runs Move thumbnails both ways, the review refuses a folder in place, and an unplugged +// thumbnail folder shows the notice whose Recreate folder brings it back. The hooks are thumbnail_location_fixture.py's. +const workspace = '/prompt_manager/workspace'; +const NOTICE_KEY = 'pm_update_notice_dismissed'; + +test.beforeEach(async ({ page, request, browserName }) => { + await startCoverage(page, browserName); + await seedConcept(request); + await expect(await request.post('/__test__/system/reset')).toBeOK(); + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'source' } })).toBeOK(); + page.locationErrors = []; + page.on('pageerror', error => page.locationErrors.push(error.message)); + await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY); +}); + +test.afterEach(async ({ page, request, browserName }) => { + await saveCoverage(page, browserName); + await request.post('/__test__/thumbnail-location', { data: { mode: 'source' } }); + expect(page.locationErrors).toEqual([]); +}); + +const block = page => page.locator('#location-block'); +const dialog = page => page.locator('dialog#modal[open]'); + +async function files(request) { + const response = await request.get('/__test__/thumbnail-files'); + await expect(response).toBeOK(); + return response.json(); +} + +async function openSources(page) { + await page.goto(`${workspace}#/system/sources`); + await expect(block(page).locator('#location-status')).not.toHaveText(''); +} + +/** Review the row's choice and start the move; resolves once its job view says it finished. */ +async function moveFromRow(page) { + const planned = page.waitForResponse(response => response.url().endsWith('/images/thumbnails/plan') + && response.request().method() === 'POST'); + await block(page).getByRole('button', { name: 'Review move…' }).click(); + expect((await planned).status()).toBe(200); + await expect(dialog(page).locator('h2')).toHaveText('Move thumbnails'); + await dialog(page).getByRole('button', { name: 'Move thumbnails' }).click(); + await expect(page.locator('#workspace-main h1')).toHaveText('Move thumbnails'); + await expect(page.locator('#workspace-main')).toContainText('Thumbnails moved to'); +} + +test('the row moves thumbnails into one folder and back', async ({ page, request }) => { + await openSources(page); + const row = block(page); + await expect(row.getByRole('radio', { name: /Inside each source/ })).toBeChecked(); + await expect(row.locator('#location-status')).toHaveText('Now: inside each source.'); + await expect(row.getByRole('button', { name: 'Review move…' })).toHaveAttribute('aria-disabled', 'true'); + await expect(row.locator('#location-path')).toBeDisabled(); + + await row.getByRole('radio', { name: /One folder/ }).check(); + await expect(row.locator('#location-path')).toBeEnabled(); + await expect(row.locator('#location-path')).toHaveValue(/default-thumbs$/); + await expect(row.locator('#location-status')).toHaveText(/^Now: inside each source\. The suggested folder is next to your library/); + await expect(row.getByRole('button', { name: 'Review move…' })).toHaveAttribute('aria-disabled', 'false'); + await moveFromRow(page); + await expect(page.locator('#job-phase')).toHaveText(/^Thumbnails are now read from the new location/); + const moved = await files(request); + expect(moved.folder.length).toBeGreaterThan(0); + expect(moved.folder.every(name => /^[a-z0-9_-]+-[0-9a-f]{8}\//.test(name))).toBe(true); + expect(moved.source_thumbnails).toBe(false); + + // Images shows thumbnails served from the folder. + const served = page.waitForResponse(response => response.url().includes('/images/serve/thumbnails/') + && response.status() === 200); + await page.goto(`${workspace}#/images`); + await served; + + await openSources(page); + await expect(block(page).locator('#location-status')).toHaveText(/^Now: one folder, .*default-thumbs\. Using the default folder/); + await block(page).getByRole('radio', { name: /Inside each source/ }).check(); + await moveFromRow(page); + const back = await files(request); + expect(back.source_thumbnails).toBe(true); + expect(back.folder_exists).toBe(false); +}); + +test('a folder the server refuses is explained under the field', async ({ page }) => { + await openSources(page); + const row = block(page); + await row.getByRole('radio', { name: /One folder/ }).check(); + await row.locator('#location-path').fill('thumbnails'); + await row.getByRole('button', { name: 'Review move…' }).click(); + await expect(row.locator('#location-problem')).toHaveText( + 'Enter a full folder path, such as D:\\Thumbnails or /mnt/fast/thumbnails.'); + await expect(row.locator('#location-path')).toHaveAttribute('aria-invalid', 'true'); + await expect(row.locator('#location-path')).toHaveAttribute('aria-describedby', 'location-problem'); + await expect(dialog(page)).toHaveCount(0); + await row.locator('#location-path').fill('/somewhere/else'); + await expect(row.locator('#location-problem')).toHaveText(''); +}); + +test('an unplugged folder shows the notice and Recreate folder brings it back', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'folder' } })).toBeOK(); + const removed = await request.post('/__test__/thumbnail-folder/remove'); + expect((await removed.json()).available).toBe(false); + await openSources(page); + const notice = page.locator('#unavailable-notice'); + await expect(notice.locator('h2')).toHaveText('The thumbnail folder is unavailable'); + await expect(notice).toContainText('is missing. Originals are shown and no thumbnails are written until it is back.'); + await expect(block(page).locator('#location-status')).toHaveText(/· unavailable\. Originals are shown\.$/); + await notice.getByRole('button', { name: 'Recreate folder' }).click(); + await expect(dialog(page).locator('h2')).toHaveText('Recreate the thumbnail folder?'); + const recreated = page.waitForResponse(response => response.url().endsWith('/location/recreate')); + await dialog(page).getByRole('button', { name: 'Recreate folder' }).click(); + expect((await recreated).status()).toBe(200); + await expect(notice).toHaveCount(0); + await expect(block(page).locator('#location-status')).toHaveText(/^Now: one folder, .*thumbs\.$/); +}); + +test('a folder of your own needs the confirmation before it moves', async ({ page, request }) => { + const custom = (await files(request)).custom_path; + await openSources(page); + const row = block(page); + await row.getByRole('radio', { name: /One folder/ }).check(); + await row.locator('#location-path').fill(custom); + await row.getByRole('button', { name: 'Review move…' }).click(); + const review = dialog(page); + await expect(review.locator('dt')).toContainText(['From', 'To', 'Thumbnails', 'Free space', 'Removed after the move']); + await expect(review.locator('dd').nth(1)).toHaveText(`${custom} (new folder)`); + const move = review.getByRole('button', { name: 'Move thumbnails' }); + await expect(move).toHaveAttribute('aria-disabled', 'true'); + await review.getByRole('checkbox', { name: /may create its own folders/ }).check(); + await expect(move).toHaveAttribute('aria-disabled', 'false'); + await move.click(); + await expect(page.locator('#workspace-main')).toContainText('Thumbnails moved to'); + expect((await files(request)).folder.length).toBeGreaterThan(0); +}); + +test('another folder’s marker offers Use this folder', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'folder' } })).toBeOK(); + await expect(await request.post('/__test__/thumbnail-folder/foreign')).toBeOK(); + await openSources(page); + const notice = page.locator('#unavailable-notice'); + await expect(notice).toContainText('holds another thumbnail folder'); + await notice.getByRole('button', { name: 'Use this folder' }).click(); + await expect(notice).toHaveCount(0); + await expect(block(page).locator('#location-status')).toHaveText(/^Now: one folder, .*thumbs\.$/); +}); + +test('Overview shows the notice and Change location opens the row', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'folder' } })).toBeOK(); + const removed = await request.post('/__test__/thumbnail-folder/remove'); + expect((await removed.json()).available).toBe(false); + await page.goto(`${workspace}#/system/overview`); + const notice = page.locator('#unavailable-notice'); + await expect(notice.locator('h2')).toHaveText('The thumbnail folder is unavailable'); + await notice.getByRole('button', { name: 'Change location' }).click(); + await expect(page).toHaveURL(/#\/system\/sources$/); + const sourcesNotice = page.locator('#unavailable-notice'); + await sourcesNotice.getByRole('button', { name: 'Change location' }).click(); + await expect(block(page).getByRole('radio', { name: /One folder/ })).toBeChecked(); + await expect(block(page).locator('#location-path')).toBeFocused(); +}); + +test('Overview’s Use this folder takes the folder and the notice goes', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'folder' } })).toBeOK(); + await expect(await request.post('/__test__/thumbnail-folder/foreign')).toBeOK(); + await page.goto(`${workspace}#/system/overview`); + const notice = page.locator('#unavailable-notice'); + await expect(notice).toContainText('holds another thumbnail folder'); + const adopted = page.waitForResponse(response => response.url().endsWith('/location/adopt')); + await notice.getByRole('button', { name: 'Use this folder' }).click(); + expect((await adopted).status()).toBe(200); + await expect(notice).toHaveCount(0); +}); + +test('Thumbnail maintenance removes the thumbnails of removed sources in folder mode', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-location', { data: { mode: 'folder' } })).toBeOK(); + await expect(await request.post('/__test__/thumbnail-folder/unlisted')).toBeOK(); + await openSources(page); + const generation = page.locator('#workspace-main .operation').filter({ hasText: 'Thumbnail generation' }); + await generation.getByRole('button', { name: 'Review options' }).click(); + const scope = dialog(page).locator('#thumb-mode'); + await expect(scope.locator('option')).toHaveText(['Missing thumbnails only', 'Rebuild all (explicit opt-in)', + 'Convert outdated (removes older formats)', 'Remove thumbnails of removed sources']); + await scope.selectOption('prune'); + await expect(dialog(page).locator('[role=status]')).toHaveText( + /^1 source no longer listed: 1 thumbnail \(.+\) are removed\.$/); + await expect(dialog(page).locator('dl.meta-grid dt')).toHaveText(['gone-0123abcd']); + await expect(dialog(page)).toContainText('If a source comes back, its thumbnails are made again as you browse.'); + await scope.selectOption('missing'); + await expect(dialog(page).locator('dl.meta-grid')).toBeHidden(); + await scope.selectOption('prune'); + await dialog(page).getByRole('button', { name: 'Start' }).click(); + await expect(page.locator('#workspace-main h1')).toHaveText('Remove thumbnails of removed sources'); + await expect(page.locator('#workspace-main')).toContainText('1 thumbnails removed'); +}); + +test('a disk without the space shows the review but cannot move', async ({ page, request }) => { + await expect(await request.post('/__test__/thumbnail-space/full')).toBeOK(); + await openSources(page); + const row = block(page); + await row.getByRole('radio', { name: /One folder/ }).check(); + const planned = page.waitForResponse(response => response.url().endsWith('/images/thumbnails/plan')); + await row.getByRole('button', { name: 'Review move…' }).click(); + expect((await planned).status()).toBe(507); + const review = dialog(page); + await expect(review.locator('.review-refusal')).toHaveText(/^There is not enough space for the thumbnails: 1 B free/); + await expect(review.getByRole('button', { name: 'Move thumbnails' })).toHaveAttribute('aria-disabled', 'true'); + await review.getByRole('button', { name: 'Cancel' }).click(); + await expect(dialog(page)).toHaveCount(0); +}); diff --git a/tests/fixtures/legacy_contract_snapshot.json b/tests/fixtures/legacy_contract_snapshot.json index 93218b1d..abd9dd84 100644 --- a/tests/fixtures/legacy_contract_snapshot.json +++ b/tests/fixtures/legacy_contract_snapshot.json @@ -126,6 +126,9 @@ "POST /prompt_manager/images/removed/return", "POST /prompt_manager/images/rescope", "POST /prompt_manager/images/reveal", + "POST /prompt_manager/images/thumbnails/location/adopt", + "POST /prompt_manager/images/thumbnails/location/recreate", + "POST /prompt_manager/images/thumbnails/plan", "POST /prompt_manager/jobs/autotag", "POST /prompt_manager/jobs/duplicates", "POST /prompt_manager/jobs/image-tags", @@ -217,6 +220,13 @@ "thumbnail_encoders", "thumbnail_encoders.jpeg", "thumbnail_encoders.jxl", - "thumbnail_encoders.webp" + "thumbnail_encoders.webp", + "thumbnail_location", + "thumbnail_location.available", + "thumbnail_location.cleanup_pending", + "thumbnail_location.default_path", + "thumbnail_location.mode", + "thumbnail_location.path", + "thumbnail_location.problem" ] } diff --git a/tests/js/api-client.test.js b/tests/js/api-client.test.js index 75fc4875..45d5faa6 100644 --- a/tests/js/api-client.test.js +++ b/tests/js/api-client.test.js @@ -371,6 +371,16 @@ test("ApiError keeps the job that applied a used plan (plan-used, System S7)", ( } }); +test("ApiError keeps a move refused for space with its review (#320)", () => { + const plan = { totals: { files: 3 }, insufficient: { text: "Not enough space" } }; + assert.equal(new ApiError(507, "no space", "insufficient-space", null, { plan }).plan, plan); + for (const bad of [null, "x", [1], undefined]) { + assert.equal(new ApiError(507, "no space", "insufficient-space", null, { plan: bad }).plan, null); + } + assert.equal(new ApiError(409, "busy", "busy", null, { plan }).plan, null); + assert.equal(new ApiError(507, "no space", "insufficient-space").plan, null); +}); + test("ApiError keeps a validated running job on busy and a validated source on invalid-source", () => { const body = { running: { id: "a".repeat(32), title: "Scan for metadata" }, source: "output/gone" }; const busy = new ApiError(409, "busy", "busy", null, body); diff --git a/tests/js/workspace-system-thumbnail-location.test.js b/tests/js/workspace-system-thumbnail-location.test.js new file mode 100644 index 00000000..fe93e78b --- /dev/null +++ b/tests/js/workspace-system-thumbnail-location.test.js @@ -0,0 +1,170 @@ +// Thumbnail location (#320, D196–D201): the row's state and status line, the move review's rows, the unavailable +// notice's text, the Activity views of a move and a prune, the prune scope's plan line and the Move library row. +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const locationModule = import('../../web/js/workspace/system-thumbnail-location.mjs'); +const kindsModule = import('../../web/js/workspace/activity-kinds.mjs'); +const reviewModule = import('../../web/js/workspace/thumbnail-review.mjs'); +const libraryModule = import('../../web/js/workspace/library-location.mjs'); + +const SOURCE = Object.freeze({ mode: 'source', path: null, default_path: '/data/thumbnails', available: true, + problem: null, cleanup_pending: 0 }); +const FOLDER = Object.freeze({ mode: 'folder', path: 'E:\\Thumbnails', default_path: '/data/thumbnails', + available: true, problem: null, cleanup_pending: 0 }); + +const PLAN = Object.freeze({ + from: { mode: 'source', path: null, available: true }, + to: { mode: 'folder', path: 'E:\\Thumbnails', exists: true, will_create: false, default: false }, + totals: { sources: 3, files: 8512, bytes: 1.2 * 1024 ** 3, orphans: 41, temps: 2, links: 1, left_in_place: 1, + too_long: 32, unlisted_sources: 0, unlisted_files: 0 }, + too_long_listed: ['Archive 2025 · a_thumb.webp', 'Archive 2025 · b_thumb.webp'], + left_listed: ['Archive 2025 · notes.txt'], + unlisted: [], + free_bytes: 1.6 * 1024 ** 4, + confirm_required: true, + token: 't', +}); + +test('the status line says where previews are now', async () => { + const { statusText } = await locationModule; + assert.equal(statusText(SOURCE), 'Now: inside each source.'); + assert.equal(statusText(FOLDER), 'Now: one folder, E:\\Thumbnails.'); + assert.equal(statusText({ ...FOLDER, path: '/data/thumbnails' }), + 'Now: one folder, /data/thumbnails. Using the default folder next to your library.'); + assert.equal(statusText({ ...FOLDER, available: false, problem: 'missing' }), + 'Now: one folder, E:\\Thumbnails · unavailable. Originals are shown.'); + assert.match(statusText({ ...SOURCE, cleanup_pending: 1 }), /Old thumbnails in 1 place wait to be removed/); + assert.equal(statusText(undefined), ''); + assert.match(statusText(SOURCE, { mode: 'folder', path: '/data/thumbnails' }), + /^Now: inside each source\. The suggested folder is next to your library/); + assert.equal(statusText(SOURCE, { mode: 'folder', path: '/other' }), 'Now: inside each source.'); +}); + +test('Review move… is offered only for another location', async () => { + const { choiceChanged } = await locationModule; + assert.equal(choiceChanged(SOURCE, { mode: 'source' }), false); + assert.equal(choiceChanged(SOURCE, { mode: 'folder', path: ' ' }), false); + assert.equal(choiceChanged(SOURCE, { mode: 'folder', path: '/data/thumbnails' }), true); + assert.equal(choiceChanged(FOLDER, { mode: 'folder', path: 'E:\\Thumbnails' }), false); + assert.equal(choiceChanged(FOLDER, { mode: 'folder', path: 'F:\\Thumbs' }), true); + assert.equal(choiceChanged(FOLDER, { mode: 'source' }), true); + assert.equal(choiceChanged({ ...SOURCE, problem: 'invalid' }, { mode: 'source' }), true); + assert.equal(choiceChanged(undefined, { mode: 'source' }), false); +}); + +test('the review lists what moves, what is removed and what stays', async () => { + const { reviewRows } = await locationModule; + const rows = Object.fromEntries(reviewRows(PLAN)); + assert.equal(rows.From, 'Inside each source (3 sources)'); + assert.equal(rows.To, 'E:\\Thumbnails (empty folder)'); + assert.equal(rows.Thumbnails, '8,512 in 3 sources · 1.2 GB'); + assert.equal(rows['Free space'], '1638.4 GB free'); + assert.match(rows['Too long for Windows'][0], /^32 thumbnails cannot be stored there/); + assert.deepEqual(rows['Too long for Windows'][1], [...PLAN.too_long_listed, 'and 30 more']); + assert.deepEqual(rows['Removed after the move'][1], + ['41 thumbnails whose original is gone', '2 unfinished temporary files']); + assert.equal(rows['Left in place'][0], '1 file that is not a thumbnail:'); + assert.equal(rows['Links skipped'], '1 link inside a thumbnails folder is not followed'); +}); + +test('the review of a move back names the folder and the removed sources', async () => { + const { reviewRows } = await locationModule; + const back = reviewRows({ ...PLAN, from: { mode: 'folder', path: '/data/thumbnails' }, + to: { mode: 'source', path: null }, unlisted: [{ key: 'old-1c22e0a4', files: 240 }], + totals: { sources: 1, files: 1, bytes: 10 }, free_bytes: undefined }); + const rows = Object.fromEntries(back); + assert.equal(rows.From, '/data/thumbnails'); + assert.equal(rows.To, 'Inside each source (1 source)'); + assert.equal(rows['Free space'], undefined); + assert.match(rows['Removed after the move'][0], /The folder itself is removed once empty/); + assert.deepEqual(rows['Removed after the move'][1], + ['1 source no longer listed: 240 thumbnails are removed; they are made again if it returns']); + // Nothing else removed: one full stop, never "created it.." (live test of #592). + const plain = Object.fromEntries(reviewRows({ ...PLAN, from: { mode: 'folder', path: '/data/thumbnails' }, + to: { mode: 'source', path: null }, unlisted: [], totals: { sources: 1, files: 1, bytes: 10 } })); + assert.match(plain['Removed after the move'][0], /Prompt Manager created it\.$/); + const fresh = Object.fromEntries(reviewRows({ ...PLAN, to: { mode: 'folder', path: '/d', default: true, will_create: true } })); + assert.equal(fresh.To, '/d (new, next to your library)'); +}); + +test('the notice says why the folder is unavailable', async () => { + const { noticeText } = await locationModule; + assert.match(noticeText({ ...FOLDER, problem: 'missing' }), /^E:\\Thumbnails is missing\. Originals are shown/); + assert.match(noticeText({ ...FOLDER, problem: 'unreachable' }), /can’t be reached/); + assert.match(noticeText({ ...FOLDER, problem: 'marker-mismatch' }), /holds another thumbnail folder/); + assert.match(noticeText({ ...FOLDER, problem: 'invalid' }), /config\.json can’t be used/); + assert.match(noticeText({ ...FOLDER, problem: null }), /is being checked/); +}); + +test('sizes are written as the server writes them', async () => { + const { sizeText } = await locationModule; + assert.equal(sizeText(512), '512 B'); + assert.equal(sizeText(85.2 * 1024 * 1024), '85.2 MB'); +}); + +test('a move and a prune show their own tiles in Activity', async () => { + const { viewOf, movePhase, finishToast } = await kindsModule; + const move = { kind: 'thumbnails', state: 'running', cancellable: true, scope: { mode: 'move', from: 'source' }, + counters: { moved: 3, skipped: 0, left: 1, errors: 2 }, status_line: '' }; + assert.deepEqual(viewOf(move).tiles.map(([group, label]) => label), ['Moved', 'Already there', 'Left in place', 'Errors']); + assert.equal(viewOf(move).cancelLabel, 'Cancel move'); + assert.match(movePhase(move), /^Step 1 of 3: copying\. Thumbnails are still read from inside each source/); + assert.match(movePhase({ ...move, cancellable: false }), /can no longer be cancelled/); + assert.match(movePhase({ ...move, scope: { mode: 'move', phase: 'remove' } }), /Removing the old copies/); + assert.match(movePhase({ ...move, state: 'finished' }), /1 file that is not a thumbnail was left in place/); + assert.match(movePhase({ ...move, state: 'cancelled' }), /^Nothing changed/); + assert.match(movePhase({ ...move, state: 'finished', counters: { moved: 3, left: 2 } }), + /2 files that are not thumbnails were left in place\.$/); + assert.equal(movePhase({ ...move, state: 'finished', counters: undefined }), 'Thumbnails are now read from the new location.'); + assert.match(movePhase({ ...move, scope: { mode: 'move', from: 'folder' } }), /read from the old folder until the switch/); + const done = { ...move, state: 'finished', status_line: 'Thumbnails moved to E:\\T: 3 moved · old copies removed.' }; + assert.equal(finishToast(done), done.status_line); + const prune = { kind: 'thumbnails', state: 'running', cancellable: true, scope: { mode: 'prune' }, + counters: { removed: 1, left: 0, errors: 0 } }; + assert.deepEqual(viewOf(prune).tiles.map(([, label]) => label), ['Removed', 'Left in place', 'Errors']); + assert.equal(viewOf(move).line(move), ''); + assert.equal(viewOf(done).line(done), done.status_line); + assert.equal(viewOf(move).kept(move), '3 moved'); + assert.equal(finishToast({ ...move, state: 'cancelled', status_line: '' }), viewOf(move).finishToast({ ...move, + state: 'cancelled', status_line: '' })); + const pruned = { ...prune, state: 'finished', status_line: '1 thumbnails removed' }; + assert.equal(viewOf(prune).line(prune), ''); + assert.equal(viewOf(pruned).line(pruned), '1 thumbnails removed'); + assert.equal(viewOf(pruned).line({ ...pruned, status_line: null }), ''); + assert.equal(viewOf(done).line({ ...done, status_line: null }), ''); + assert.equal(finishToast(pruned), '1 thumbnails removed'); + assert.equal(viewOf(prune).kept(prune), '1 removed'); + assert.match(finishToast({ ...prune, state: 'finished', status_line: '', title: 'Remove thumbnails of removed sources' }), + /Remove thumbnails of removed sources/); + const generate = { kind: 'thumbnails', state: 'running', cancellable: true, scope: { mode: 'missing' }, + counters: { generated: 0, skipped: 0, errors: 0 } }; + assert.deepEqual(viewOf(generate).tiles.map(([, label]) => label), ['Generated', 'Skipped', 'Errors']); +}); + +test('the prune scope reads its plan line from the server', async () => { + const { planText, PLAN_FAILED, PRUNE_OPTION } = await reviewModule; + assert.equal(PRUNE_OPTION[1], 'Remove thumbnails of removed sources'); + assert.equal(planText({ mode: 'prune', line: '2 sources no longer listed: 1,240 thumbnails (85.2 MB) are removed.' }), + '2 sources no longer listed: 1,240 thumbnails (85.2 MB) are removed.'); + assert.equal(planText({ mode: 'prune' }), PLAN_FAILED); +}); + +test('the prune review names the folder and each removed source', async () => { + const { pruneText, pruneEntries } = await reviewModule; + assert.match(pruneText({ folder: 'E:\\Thumbs' }), /^Only Prompt Manager’s thumbnails in their folders inside E:\\Thumbs are /); + assert.match(pruneText({}), /inside the thumbnail folder are removed\. If a source comes back/); + assert.deepEqual(pruneEntries({ sources: [{ key: 'gone-0123abcd', files: 1240, bytes: 85.2 * 1024 * 1024 }, + { key: 'one-0123abcd', files: 1, bytes: 512 }] }), + [['gone-0123abcd', '1,240 thumbnails · 85.2 MB'], ['one-0123abcd', '1 thumbnail · 512 B']]); + assert.deepEqual(pruneEntries(null), []); + assert.deepEqual(pruneEntries({ sources: 'x' }), []); +}); + +test('Move library says where thumbnails kept in one folder stay', async () => { + const { reviewGrid } = await libraryModule; + const base = { path: 'p', target: 't', size_bytes: 0, config_moves: true }; + assert.equal(reviewGrid(base).some(([term]) => term === 'Thumbnails'), false); + const rows = Object.fromEntries(reviewGrid({ ...base, thumbnails_folder: '/data/thumbnails' })); + assert.equal(rows.Thumbnails, 'Stay in /data/thumbnails. Move them in System › Sources & scans.'); +}); diff --git a/tests/test_api_library_location.py b/tests/test_api_library_location.py index 7b898aa0..d3edcc6d 100644 --- a/tests/test_api_library_location.py +++ b/tests/test_api_library_location.py @@ -91,6 +91,8 @@ class TestLocationRoute(LibraryAPITestCase): "size_bytes": os.path.getsize(self.api.db.model.db_path), "target_source": "default", "config_moves": True, + # #320 D201: previews kept inside each source (no folder). + "thumbnails_folder": None, }, ) diff --git a/tests/test_api_thumbnail_location.py b/tests/test_api_thumbnail_location.py new file mode 100644 index 00000000..fd78be7c --- /dev/null +++ b/tests/test_api_thumbnail_location.py @@ -0,0 +1,487 @@ +"""The thumbnail location setting (#320 PR 2): parsing, validation, source +keys, the marker, the settings payloads and reverse containment. + +Spelling rules are tested on both path flavors (``ntpath`` and +``posixpath``) as text; filesystem rules run on the native one. +""" + +import json +import ntpath +import os +import posixpath +import re +import sys +import tempfile +import threading +import unittest +from pathlib import Path +from unittest.mock import MagicMock, patch + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +sys.modules.setdefault("server", MagicMock()) + +from py.api import preview_location as pl # noqa: E402 +from py.api import preview_store # noqa: E402 + +REPO = Path(__file__).resolve().parent.parent + + +class TestParse(unittest.TestCase): + def test_missing_or_source_is_inside_each_source(self): + for value in (None, {"mode": "source"}, {}): + with self.subTest(value=value): + parsed = pl.parse_location(value) + self.assertEqual((parsed.mode, parsed.problem), ("source", None)) + + def test_a_folder_location_is_kept_as_spelled(self): + parsed = pl.parse_location( + {"mode": "folder", "path": "D:\\Thumbs", "id": "x", "install": "i"}, + flavor=ntpath, + ) + self.assertEqual( + (parsed.mode, parsed.path, parsed.id, parsed.problem), + ("folder", "D:\\Thumbs", "x", None), + ) + + def test_invalid_values_never_fall_back_to_the_sources(self): + for value in ( + "folder", + {"mode": "elsewhere"}, + {"mode": "folder", "path": "relative", "id": "x"}, + {"mode": "folder", "path": "/abs"}, + {"mode": "folder", "path": "/abs", "id": ""}, + {"mode": "folder", "path": 7, "id": "x"}, + ): + with self.subTest(value=value): + parsed = pl.parse_location(value, flavor=posixpath) + self.assertEqual((parsed.mode, parsed.problem), ("folder", "invalid")) + + +class TestForm(unittest.TestCase): + def test_windows_spellings(self): + cases = { + "D:\\Thumbs": None, + "\\\\server\\share\\thumbs": None, + "C:thumbs": "location-not-absolute", + "\\thumbs": "location-not-absolute", + "thumbs": "location-not-absolute", + "": "location-not-absolute", + " D:\\Thumbs": "location-not-absolute", + "\\\\?\\D:\\Thumbs": "location-device-path", + "\\\\.\\D:\\Thumbs": "location-device-path", + "D:\\CON\\x": "location-invalid-name", + "D:\\com1.txt": "location-invalid-name", + "D:\\thumbs.": "location-invalid-name", + "D:\\thumbs ": "location-invalid-name", + "D:\\a|b": "location-invalid-name", + "D:\\a?b": "location-invalid-name", + "D:\\a\x01b": "location-invalid-name", + } + for path, code in cases.items(): + with self.subTest(path=path): + self.assertEqual(pl.form_problem(path, flavor=ntpath), code) + + def test_posix_spellings(self): + cases = { + "/mnt/fast/thumbs": None, + "mnt/thumbs": "location-not-absolute", + "/mnt/nul": "location-invalid-name", + "/mnt/a:b": "location-invalid-name", + "//?/x": "location-device-path", + } + for path, code in cases.items(): + with self.subTest(path=path): + self.assertEqual(pl.form_problem(path, flavor=posixpath), code) + + def test_roots_and_home_are_too_broad(self): + self.assertTrue(pl.is_too_broad("C:\\", home="C:\\Users\\a", flavor=ntpath)) + self.assertTrue( + pl.is_too_broad("\\\\server\\share", home="C:\\Users\\a", flavor=ntpath) + ) + self.assertTrue( + pl.is_too_broad("C:\\users\\A", home="C:\\Users\\a", flavor=ntpath) + ) + self.assertTrue(pl.is_too_broad("/", home="/home/a", flavor=posixpath)) + self.assertTrue(pl.is_too_broad("/home/a/", home="/home/a", flavor=posixpath)) + self.assertFalse( + pl.is_too_broad("/home/a/thumbs", home="/home/a", flavor=posixpath) + ) + + def test_long_windows_paths_are_refused_without_long_paths(self): + short = "D:\\" + "a" * 100 + long = "D:\\" + "a" * 200 + self.assertFalse(pl.too_long_for_windows(short, long_paths=False)) + self.assertTrue(pl.too_long_for_windows(long, long_paths=False)) + self.assertFalse(pl.too_long_for_windows(long, long_paths=True)) + + +class TestSourceKeys(unittest.TestCase): + def test_slug(self): + self.assertEqual(pl.slug("ComfyUI Output"), "comfyui-output") + self.assertEqual(pl.slug("Ünïcödé_Fôlder"), "unicode_folder") + self.assertEqual(pl.slug(""), "source") + self.assertEqual(pl.slug("!!!"), "source") + self.assertEqual(len(pl.slug("x" * 80)), 24) + + def test_keys_are_a_slug_plus_a_hash(self): + key = pl.source_key("D:\\AI\\Output", "d:\\ai\\output", {}) + self.assertRegex(key, r"^output-[0-9a-f]{8}$") + self.assertRegex(pl.source_key("D:\\", "d:\\", {}), r"^source-[0-9a-f]{8}$") + self.assertRegex( + pl.source_key("\\\\nas\\share\\Out", "\\\\nas\\share\\out", {}), + r"^out-[0-9a-f]{8}$", + ) + + def test_a_taken_key_grows_its_hash(self): + first = pl.source_key("/a/out", "/a/out", {}) + taken = {first: "/someone/else"} + second = pl.source_key("/a/out", "/a/out", taken) + self.assertRegex(second, r"^out-[0-9a-f]{12}$") + taken[second] = "/another" + self.assertRegex( + pl.source_key("/a/out", "/a/out", taken), r"^out-[0-9a-f]{16}$" + ) + self.assertEqual(pl.source_key("/a/out", "/a/out", {first: "/a/out"}), first) + + def test_lookup_by_spelling_identity_then_file_id(self): + keys = { + "out-0123abcd": { + "path": "D:\\Out", + "identity": "d:\\out", + "dev_ino": [1, 2], + }, + "lib-0123abcd": {"path": "E:\\Lib", "identity": "e:\\lib", "dev_ino": None}, + } + self.assertEqual(pl.find_key(keys, "D:\\Out", "x", None), "out-0123abcd") + self.assertEqual(pl.find_key(keys, "Z:\\Out", "d:\\out", None), "out-0123abcd") + # A letter spelling and its UNC spelling: the same file id. + self.assertEqual( + pl.find_key(keys, "\\\\nas\\out", "\\\\nas\\out", (1, 2)), "out-0123abcd" + ) + self.assertIsNone(pl.find_key(keys, "F:\\New", "f:\\new", (9, 9))) + + +class TestMarker(unittest.TestCase): + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.addCleanup(self._tmp.cleanup) + self.folder = self._tmp.name + + def test_init_creates_once_and_adopts_after(self): + first = pl.init_folder(self.folder, created_by_pm=True) + self.assertEqual(pl.init_folder(self.folder), first) + marker = pl.read_marker(self.folder) + self.assertEqual(marker["id"], first) + self.assertTrue(marker["created_by_pm"]) + self.assertEqual(marker["version"], 1) + + def test_writes_merge_what_another_writer_added(self): + pl.init_folder(self.folder) + pl.write_marker( + self.folder, lambda m: m["keys"].update({"a-0123abcd": {"install": "1"}}) + ) + pl.write_marker( + self.folder, lambda m: m["keys"].update({"b-0123abcd": {"install": "2"}}) + ) + self.assertEqual( + sorted(pl.read_marker(self.folder)["keys"]), ["a-0123abcd", "b-0123abcd"] + ) + # No temp file is left beside it. + self.assertEqual(os.listdir(self.folder), [pl.MARKER_NAME]) + + def test_two_processes_adding_keys_at_once_keep_every_key(self): + """Installs sharing one folder (bot review, P2): the read-modify-write + of the marker is locked across processes, so no writer drops another's + key. Each child widens the window by pausing inside ``mutate``.""" + import subprocess + + pl.init_folder(self.folder) + child = ( + "import sys, time\n" + "sys.path.insert(0, sys.argv[1])\n" + "from py.api import preview_location as pl\n" + "folder, tag, start = sys.argv[2], sys.argv[3], float(sys.argv[4])\n" + "while time.time() < start: time.sleep(0.001)\n" + "for n in range(8):\n" + " def add(m, n=n):\n" + " time.sleep(0.01)\n" + " m['keys'][f'{tag}-{n:08x}'] = {'install': tag}\n" + " pl.write_marker(folder, add)\n" + ) + import time + + start = str(time.time() + 1.0) + procs = [ + subprocess.Popen( # nosec B603 - this interpreter, fixed arguments + [sys.executable, "-c", child, str(REPO), self.folder, tag, start] + ) + for tag in ("a", "b", "c") + ] + for proc in procs: + self.assertEqual(proc.wait(timeout=60), 0) + keys = pl.read_marker(self.folder)["keys"] + self.assertEqual(len(keys), 24, sorted(keys)) + self.assertEqual(os.listdir(self.folder), [pl.MARKER_NAME]) + + def test_a_lock_left_by_a_crashed_writer_is_taken_over_when_stale(self): + pl.init_folder(self.folder) + lock = os.path.join(self.folder, pl.MARKER_LOCK_NAME) + with open(lock, "w", encoding="utf-8") as handle: + handle.write("crashed") + old = pl.time.time() - pl.MARKER_LOCK_STALE - 5 + os.utime(lock, (old, old)) + pl.write_marker(self.folder, lambda m: m["keys"].update({"a-0123abcd": {}})) + self.assertIn("a-0123abcd", pl.read_marker(self.folder)["keys"]) + self.assertFalse(os.path.exists(lock)) + + def test_a_held_lock_times_out_without_writing(self): + pl.init_folder(self.folder) + lock = os.path.join(self.folder, pl.MARKER_LOCK_NAME) + with open(lock, "w", encoding="utf-8") as handle: + handle.write("held") + with patch.object(pl, "MARKER_LOCK_TIMEOUT", 0.05): + with self.assertRaises(TimeoutError): + pl.write_marker(self.folder, lambda m: m["keys"].update({"x": {}})) + self.assertTrue(os.path.exists(lock)) # never taken from its holder + self.assertEqual(pl.read_marker(self.folder)["keys"], {}) + + def test_a_folder_without_the_marker_is_missing(self): + self.assertEqual(pl.folder_state(self.folder, "x")[:2], (False, "missing")) + missing = os.path.join(self.folder, "gone") + self.assertEqual(pl.folder_state(missing, "x")[:2], (False, "missing")) + marker_id = pl.init_folder(self.folder) + self.assertEqual(pl.folder_state(self.folder, marker_id)[:2], (True, None)) + self.assertEqual( + pl.folder_state(self.folder, "other")[:2], (False, "marker-mismatch") + ) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + def test_a_linked_folder_is_not_available(self): + real = os.path.join(self.folder, "real") + os.mkdir(real) + marker_id = pl.init_folder(real) + link = os.path.join(self.folder, "link") + try: + os.symlink(real, link, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + self.assertEqual(pl.folder_state(link, marker_id)[:2], (False, "invalid")) + + +class TestValidation(unittest.TestCase): + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.addCleanup(self._tmp.cleanup) + self.base = os.path.realpath(self._tmp.name) + self.source = os.path.join(self.base, "output") + self.node = os.path.join(self.base, "node") + os.mkdir(self.source) + os.mkdir(self.node) + self.context = pl.Context( + roots=(self.source,), + node_folder=self.node, + default_path=os.path.join(self.node, "data", "thumbnails"), + home=os.path.join(self.base, "home"), + environ={}, + long_paths=True, + folder_paths=None, + ) + + def problem(self, path, purpose="move", **changes): + context = self.context + if changes: + context = pl.Context(**{**context.__dict__, **changes}) + return pl.location_problem(path, purpose=purpose, context=context) + + def test_an_empty_new_folder_beside_the_sources_is_accepted(self): + self.assertIsNone(self.problem(os.path.join(self.base, "thumbs"))) + os.mkdir(os.path.join(self.base, "empty")) + Path(self.base, "empty", "Thumbs.db").write_bytes(b"") + self.assertIsNone(self.problem(os.path.join(self.base, "empty"))) + + def test_codes(self): + cases = { + "relative": "location-not-absolute", + os.path.join(self.source, "thumbs"): "location-served", + os.path.join(self.node, "thumbs"): "location-node-folder", + self.base: "location-contains-source", + os.path.join(self.base, "no", "parent"): "location-parent-missing", + } + for path, code in cases.items(): + with self.subTest(path=path): + self.assertEqual(self.problem(path), code) + + def test_the_default_may_sit_in_the_node_folder_and_make_its_parents(self): + self.assertIsNone(self.problem(self.context.default_path)) + + def test_the_folder_in_use_is_the_same(self): + folder = os.path.join(self.base, "thumbs") + os.mkdir(folder) + self.assertEqual(self.problem(folder, current_path=folder), "location-same") + + def test_a_folder_inside_the_folder_in_use_is_refused(self): + # Codex P2 on #592: the old key folders are removed after the move, + # and with them a target nested inside one of them. + folder = os.path.join(self.base, "thumbs") + os.mkdir(folder) + pl.init_folder(folder) + for nested in ( + os.path.join(folder, "output-0123abcd", "nested"), + os.path.join(folder, "beside"), + ): + with self.subTest(nested=nested): + self.assertEqual( + self.problem(nested, current_path=folder), + "location-inside-current", + ) + + def test_a_folder_with_other_files_is_not_empty_unless_it_is_ours(self): + folder = os.path.join(self.base, "full") + os.mkdir(folder) + Path(folder, "photo.jpg").write_bytes(b"x") + self.assertEqual(self.problem(folder), "location-not-empty") + pl.init_folder(folder) + self.assertIsNone(self.problem(folder)) + + @unittest.skipIf(os.name == "nt" or os.geteuid() == 0, "needs POSIX modes") + def test_a_folder_that_refuses_writes(self): + folder = os.path.join(self.base, "ro") + os.mkdir(folder) + os.chmod(folder, 0o500) + self.addCleanup(os.chmod, folder, 0o700) + self.assertEqual(self.problem(folder), "location-not-writable") + + def test_a_stat_that_does_not_answer_is_unavailable(self): + release = threading.Event() + self.addCleanup(release.set) + + def stuck(*_args): + release.wait(10) + + with patch.object(pl, "_filesystem_problem", stuck): + code = pl.location_problem( + os.path.join(self.base, "thumbs"), + purpose="move", + context=self.context, + timeout=0.1, + ) + self.assertEqual(code, "location-unavailable") + + def test_a_long_windows_path(self): + path = os.path.join(self.base, "t" * 220) + self.assertEqual(self.problem(path, long_paths=False), "location-path-too-long") + + def test_every_code_has_a_message_without_a_path(self): + for code, (status, text) in pl.MESSAGES.items(): + with self.subTest(code=code): + self.assertIn(status, (400, 409, 507)) + self.assertTrue(text.endswith(".")) + + +class TestNoLegacySaveCallers(unittest.TestCase): + def test_save_to_file_has_no_callers(self): + offenders = [] + for folder in ("py", "utils", "database"): + for path in (REPO / folder).rglob("*.py"): + for number, line in enumerate( + path.read_text(encoding="utf-8").splitlines(), 1 + ): + code = line.split("#", 1)[0] + if ( + re.search(r"\bsave_to_file\(", code) + and "def save_to_file" not in code + ): + if "PromptManagerConfig.save_to_file('" in line: + continue # its own docstring example + offenders.append(f"{path.relative_to(REPO)}:{number}") + self.assertEqual(offenders, []) + + +try: + from tests.test_api_images import ImageAPITestCase +except ImportError: # discovered with tests/ as the top-level directory + from test_api_images import ImageAPITestCase + + +class TestLocationRoutes(ImageAPITestCase): + async def asyncSetUp(self): + await super().asyncSetUp() + self.addCleanup(preview_store.configure, None) + + def use_folder(self): + folder = self.comfy_dir / "thumbs" + folder.mkdir() + marker_id = pl.init_folder(str(folder), created_by_pm=True) + preview_store.configure( + {"mode": "folder", "path": str(folder), "id": marker_id} + ) + return folder + + async def test_settings_get_names_the_location_with_its_path(self): + folder = self.use_folder() + resp = await self.client.get("/prompt_manager/settings") + location = (await resp.json())["thumbnail_location"] + self.assertEqual(location["mode"], "folder") + self.assertEqual(location["path"], str(folder)) + self.assertTrue(location["available"]) + self.assertIsNone(location["problem"]) + self.assertIn("default_path", location) + + async def test_settings_post_never_changes_the_location(self): + resp = await self.client.post( + "/prompt_manager/settings", + json={"thumbnail_location": {"mode": "folder", "path": "/x"}}, + ) + self.assertEqual(resp.status, 409) + body = await resp.json() + self.assertEqual(body["code"], "location-needs-move") + self.assertNotIn("/x", body["error"]) + + async def test_status_has_the_location_but_no_path(self): + folder = self.use_folder() + resp = await self.client.get("/prompt_manager/images/thumbnails/status") + body = await resp.text() + self.assertNotIn(str(folder), body) + self.assertEqual( + json.loads(body)["location"], + {"mode": "folder", "available": True, "problem": None}, + ) + + async def test_a_source_inside_or_around_the_folder_is_refused(self): + from py.api import sources + + folder = self.use_folder() + inside = folder / "sub" + inside.mkdir() + with patch( + "py.config.GalleryConfig.gallery_root_problem", + classmethod(lambda cls, path: (None, "")), + ): + for path in (inside, folder, self.comfy_dir): + with self.subTest(path=path): + code, reason = sources.source_problem(str(path)) + self.assertEqual(code, "holds-thumbnails") + self.assertEqual(reason, pl.HOLDS_THUMBNAILS_REASON) + self.assertEqual(sources.source_problem(str(self.output_dir))[0], None) + + async def test_a_library_move_into_the_folder_is_refused(self): + from py.api import library_location as routes + + folder = self.use_folder() + self.assertTrue(routes._inside_thumbnail_folder(str(folder / "lib"))) + self.assertFalse(routes._inside_thumbnail_folder(str(self.output_dir))) + + def test_startup_makes_a_location_inside_a_source_invalid(self): + inside = self.output_dir / "thumbs" + inside.mkdir() + marker_id = pl.init_folder(str(inside)) + from py.config import PromptManagerConfig + + value = {"mode": "folder", "path": str(inside), "id": marker_id} + with patch.object(PromptManagerConfig, "THUMBNAIL_LOCATION", value): + location = self.api.start_preview_location() + self.assertEqual((location.available, location.problem), (False, "invalid")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_native_memory.py b/tests/test_native_memory.py index 06a04d0d..88e020a0 100644 --- a/tests/test_native_memory.py +++ b/tests/test_native_memory.py @@ -32,8 +32,12 @@ class ReleaseFreeMemoryTests(unittest.TestCase): def test_musl_does_nothing(self): with ( mock.patch.object(sys, "platform", "linux"), + # Windows has no os.confstr: this test fakes Linux there too. mock.patch.object( - native_memory.os, "confstr", side_effect=ValueError("unknown name") + native_memory.os, + "confstr", + side_effect=ValueError("unknown name"), + create=True, ), mock.patch.object(native_memory.ctypes, "CDLL") as cdll, ): diff --git a/tests/test_preview_probe.py b/tests/test_preview_probe.py new file mode 100644 index 00000000..496a8165 --- /dev/null +++ b/tests/test_preview_probe.py @@ -0,0 +1,93 @@ +"""The thumbnail folder check answers for the folder as it is now (#320). + +A check that started before the caller asked can miss what just happened: #592 +hit it in the merge queue, where a folder deleted right after +``configure`` woke the probe stayed "available" until the next probe. +""" + +import sys +import threading +import unittest +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from py.api import preview_location, preview_store # noqa: E402 + +WAIT = 5 +HERE = (True, None, {"id": "x"}) +GONE = (False, "missing", None) +KEY = ("/thumbs", "x") + + +class TestProbeCheck(unittest.TestCase): + def setUp(self): + self.probe = preview_store._Probe() + + def test_a_finished_check_nobody_collected_is_not_the_answer(self): + with patch.object(preview_location, "folder_state", return_value=HERE): + with self.probe._lock: + _key, done, _result = self.probe._start_check(*KEY) + self.assertTrue(done.wait(WAIT)) + with patch.object(preview_location, "folder_state", return_value=GONE): + self.assertEqual(self.probe.check(*KEY), GONE) + + def test_a_check_already_running_is_followed_by_a_new_one(self): + started = threading.Event() + release = threading.Event() + calls = [] + + def state(_path, _expected_id): + calls.append(threading.current_thread().name) + if len(calls) == 1: + started.set() + self.assertTrue(release.wait(WAIT)) + return HERE + return GONE + + answers = [] + with patch.object(preview_location, "folder_state", state): + with self.probe._lock: + self.probe._start_check(*KEY) + self.assertTrue(started.wait(WAIT)) + caller = threading.Thread( + target=lambda: answers.append(self.probe.check(*KEY)) + ) + caller.start() + release.set() + caller.join(WAIT) + self.assertFalse(caller.is_alive()) + self.assertEqual(answers, [GONE]) + self.assertEqual(len(calls), 2) # one after the other, never two at once + + def test_a_stuck_check_is_never_doubled(self): + release = threading.Event() + self.addCleanup(release.set) + calls = [] + + def state(_path, _expected_id): + calls.append(1) + release.wait(WAIT) + return HERE + + with ( + patch.object(preview_location, "folder_state", state), + patch.object(preview_store, "IO_BUDGET", 0.05), + ): + with self.probe._lock: + self.probe._start_check(*KEY) + with self.assertRaises(TimeoutError): + self.probe.check(*KEY) + self.assertEqual(len(calls), 1) + + def test_an_error_from_the_new_check_reaches_the_caller(self): + with patch.object( + preview_location, "folder_state", side_effect=OSError("share gone") + ): + with self.assertRaises(OSError): + self.probe.check(*KEY) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_preview_store.py b/tests/test_preview_store.py new file mode 100644 index 00000000..960c3431 --- /dev/null +++ b/tests/test_preview_store.py @@ -0,0 +1,457 @@ +"""One resolver for where previews live on disk (#320 PR 1). + +``py/api/preview_store.py`` owns every mapping between an image source, a +media file and its preview on disk. In source mode (the only mode so far) +it returns exactly the paths the code built by hand before +(``/thumbnails/_thumb``), so nothing changes on disk. The +structural test keeps new ``thumbnails`` path joins out of the rest of the +backend. +""" + +import asyncio +import os +import re +import sys +import tempfile +import threading +import unittest +from pathlib import Path +import unittest.mock +from unittest.mock import MagicMock + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +sys.modules.setdefault("server", MagicMock()) + +from py.api import preview_store, thumbnail_settings # noqa: E402 +from py.api.thumbnail_encode import ( # noqa: E402 + PreviewWriteHeld, + replace_unless_held, +) + +REPO = Path(__file__).resolve().parent.parent +WAIT = 10 + + +class SourceModeCase(unittest.TestCase): + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.addCleanup(self._tmp.cleanup) + self.root = Path(self._tmp.name) / "output" + self.root.mkdir() + + +class TestSourceModePaths(SourceModeCase): + def test_default_location_is_inside_each_source(self): + location = preview_store.current() + self.assertEqual(location.mode, preview_store.SOURCE_MODE) + self.assertTrue(location.available) + self.assertIsNone(location.problem) + + def test_store_of_a_source_is_its_thumbnails_folder(self): + self.assertEqual(preview_store.store_dir(self.root), self.root / "thumbnails") + + def test_preview_path_matches_the_old_join(self): + rel = Path("sub", "deep", "a.png") + self.assertEqual( + preview_store.preview_path(self.root, rel, ".webp"), + self.root / "thumbnails/sub/deep/a_thumb.webp", + ) + self.assertEqual( + preview_store.preview_path(self.root, "a.mp4", ".jpg"), + self.root / "thumbnails" / "a_thumb.jpg", + ) + + def test_preview_path_defaults_to_the_written_extension(self): + from py.api.thumbnail_encode import writer_settings + + self.assertEqual( + preview_store.preview_path(self.root, "a.png").suffix, + writer_settings().extension, + ) + + def test_request_path_maps_url_parts_to_the_old_join(self): + parts = ("thumbnails", "sub", "a_thumb.webp") + self.assertEqual( + preview_store.request_path(self.root, parts), self.root.joinpath(*parts) + ) + + def test_request_path_is_none_for_anything_but_preview_parts(self): + self.assertIsNone(preview_store.request_path(self.root, ("sub", "a.png"))) + self.assertIsNone(preview_store.request_path(self.root, ("Thumbnails", "a"))) + self.assertIsNone(preview_store.request_path(self.root, ("thumbnails",))) + + def test_ensure_store_is_the_store(self): + self.assertEqual( + preview_store.ensure_store(self.root), self.root / "thumbnails" + ) + + def test_owns_a_preview_inside_the_store(self): + inside = self.root / "thumbnails" / "sub" / "a_thumb.webp" + self.assertTrue(preview_store.owns(self.root, inside)) + self.assertFalse(preview_store.owns(self.root, self.root / "a.png")) + self.assertFalse( + preview_store.owns(self.root, self.root.parent / "elsewhere.webp") + ) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + def test_owns_refuses_a_store_linked_out_of_the_source(self): + outside = Path(self._tmp.name) / "outside" + outside.mkdir() + try: + (self.root / "thumbnails").symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + self.assertFalse( + preview_store.owns(self.root, self.root / "thumbnails" / "a_thumb.webp") + ) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + def test_owns_refuses_a_preview_folder_linked_out_of_the_store(self): + outside = Path(self._tmp.name) / "outside" + outside.mkdir() + (self.root / "thumbnails").mkdir() + try: + (self.root / "thumbnails" / "sub").symlink_to( + outside, target_is_directory=True + ) + except OSError: + self.skipTest("symlinks not permitted") + self.assertFalse( + preview_store.owns( + self.root, self.root / "thumbnails" / "sub" / "a_thumb.webp" + ) + ) + + def test_store_container_is_the_source(self): + self.assertEqual(preview_store.store_container(self.root), self.root) + + def test_given_store_argument_is_kept_unless_it_is_the_default(self): + default = self.root / "thumbnails" + other = Path(self._tmp.name) / "custom" + self.assertEqual(preview_store.given_store(self.root, default), default) + self.assertEqual(preview_store.given_store(self.root, other), other) + + +class TestPreviewText(unittest.TestCase): + def test_preview_text_is_an_exact_thumbnails_prefix_and_preview_name(self): + for text in ( + "thumbnails/a_thumb.webp", + "thumbnails/sub/a_thumb.jxl", + "thumbnails/a_thumb.JPG", + "thumbnails/a_thumb.Png", + ): + with self.subTest(text=text): + self.assertTrue(preview_store.is_preview_text(text)) + + def test_other_text_is_not_preview_text(self): + for text in ( + "Thumbnails/a_thumb.webp", + "thumbnails/a.png", + "thumbnails/_thumb.webp", + "thumbnails/a_thumb.txt", + "sub/thumbnails/a_thumb.webp", + "thumbnails/../a_thumb.webp", + "thumbnails//a_thumb.webp", + "thumbnails/./a_thumb.webp", + "thumbnails\\a_thumb.webp", + "thumbnails/c:a_thumb.webp", + "thumbnails/sub/thumbnails/a_thumb.webp", + "thumbnails", + "", + ): + with self.subTest(text=text): + self.assertFalse(preview_store.is_preview_text(text)) + + +class TestWritesAndTheSwitch(SourceModeCase): + def _temp_and_dst(self): + folder = self.root / "thumbnails" + folder.mkdir() + tmp = folder / ".a_thumb.part-1-1.webp" + tmp.write_bytes(b"new") + return tmp, folder / "a_thumb.webp" + + def test_a_write_from_the_current_generation_lands(self): + tmp, dst = self._temp_and_dst() + replace_unless_held(tmp, dst, preview_store.write_guard()) + self.assertEqual(dst.read_bytes(), b"new") + + def test_a_write_whose_generation_changed_is_dropped(self): + tmp, dst = self._temp_and_dst() + guard = preview_store.write_guard() + preview_store.bump_generation() + with self.assertRaises(PreviewWriteHeld): + replace_unless_held(tmp, dst, guard) + self.assertFalse(dst.exists()) + + def test_the_hold_check_still_applies(self): + tmp, dst = self._temp_and_dst() + with self.assertRaises(PreviewWriteHeld): + replace_unless_held(tmp, dst, preview_store.write_guard(lambda: False)) + self.assertFalse(dst.exists()) + + def test_the_check_and_the_rename_run_under_the_write_lock(self): + """The hold check and the rename hold ``write_lock``, the lock the + generation bump takes: a switch cannot fall between them.""" + tmp, dst = self._temp_and_dst() + in_check = threading.Event() + release = threading.Event() + seen = {} + + def slow_hold(): + seen["locked"] = preview_store.write_lock.locked() + in_check.set() + release.wait(WAIT) + return True + + guard = preview_store.write_guard(slow_hold) + writer = threading.Thread(target=replace_unless_held, args=(tmp, dst, guard)) + writer.start() + try: + self.assertTrue(in_check.wait(WAIT)) + self.assertTrue(seen["locked"]) + self.assertTrue(preview_store.write_lock.locked()) + finally: + release.set() + writer.join(WAIT) + self.assertFalse(preview_store.write_lock.locked()) + self.assertEqual(dst.read_bytes(), b"new") + bumped = preview_store.bump_generation() + self.assertEqual(preview_store.generation(), bumped) + + +class TestOnlyTheResolverBuildsStorePaths(unittest.TestCase): + """No backend module joins ``thumbnails`` onto a path by hand any more.""" + + ALLOWED = { + Path("py/api/preview_store.py"), + Path("py/api/thumbnail_settings.py"), # URL text + } + # A path join with a literal "thumbnails" operand. + JOIN = re.compile( + r"""/\s*["']thumbnails["']""" + r"""|["']thumbnails["']\s*/""" + r"""|joinpath\([^)]*["']thumbnails["']""" + r"""|os\.path\.join\([^)]*["']thumbnails["']""" + r"""|["']thumbnails/""" + r"""|/\s*PREVIEW_FOLDER|joinpath\([^)]*PREVIEW_FOLDER""" + r"""|Path\([^)]*,\s*["']thumbnails["']""" + ) + + def test_no_thumbnails_join_outside_the_resolver(self): + offenders = [] + files = sorted(REPO.glob("*.py")) + for folder in ("py", "utils", "database"): + files += sorted((REPO / folder).rglob("*.py")) + for path in files: + rel = path.relative_to(REPO) + if rel in self.ALLOWED: + continue + for number, line in enumerate( + path.read_text(encoding="utf-8").splitlines(), 1 + ): + code = line.split("#", 1)[0] + if self.JOIN.search(code): + offenders.append(f"{rel}:{number}: {line.strip()}") + self.assertEqual(offenders, []) + + +class TestSettingsLookupsFollowTheStore(SourceModeCase): + def test_owned_thumbnails_reads_the_store(self): + folder = self.root / "thumbnails" / "sub" + folder.mkdir(parents=True) + (folder / "a_thumb.webp").write_bytes(b"x") + self.assertEqual( + thumbnail_settings.owned_thumbnails(self.root, Path("sub", "a.png")), + [folder / "a_thumb.webp"], + ) + + +try: + from tests.test_thumbnail_auto import AutoThumbnailCase, _entry + from tests.test_api_images import make_png +except ImportError: # discovered with tests/ as the top-level directory + from test_thumbnail_auto import AutoThumbnailCase, _entry + from test_api_images import make_png + + +class RedirectedStoreCase(AutoThumbnailCase): + """Every caller follows the resolver: with the store of the output root + moved to ``/elsewhere/output-key`` (a stand-in for a later mode), + previews are read, written, listed and removed there and only there.""" + + async def asyncSetUp(self): + await super().asyncSetUp() + self.container = self.comfy_dir / "elsewhere" + self.store = self.container / "output-key" + self.store.mkdir(parents=True) + self.old_store = self.output_dir / "thumbnails" + root = self.output_dir + real_store = preview_store.store_dir + real_container = preview_store.store_container + + def store_dir(r, location=None): + return self.store if Path(r) == root else real_store(r, location) + + def store_container(r, location=None): + if Path(r) == root: + return self.container + return real_container(r, location) + + for name, fake in ( + ("store_dir", store_dir), + ("store_container", store_container), + ): + patcher = unittest.mock.patch.object(preview_store, name, fake) + patcher.start() + self.addCleanup(patcher.stop) + + def _preview(self, rel, ext=".webp"): + return self.store / (thumbnail_settings.preview_stem(rel) + ext) + + +class TestEveryCallerFollowsTheStore(RedirectedStoreCase): + async def test_an_existing_preview_is_served_from_the_store(self): + make_png(self.output_dir / "sub" / "a.png", size=(64, 64)) + preview = self._preview("sub/a.png") + make_png(preview, size=(32, 32)) + url = _entry(self.api, self.output_dir, "sub/a.png")["thumbnail_url"] + self.assertIn("v=", url) # the listing found it in the store + resp = await self.client.get(url) + self.assertEqual(resp.status, 200) + self.assertEqual(await resp.read(), preview.read_bytes()) + + async def test_a_legacy_upper_case_preview_is_served_from_the_store(self): + make_png(self.output_dir / "k.JPG", size=(64, 64)) + legacy = self.store / "k_thumb.JPG" + legacy.write_bytes(b"legacy preview") + resp = await self.client.get( + "/prompt_manager/images/serve/thumbnails/k_thumb.JPG?root=0" + ) + self.assertEqual(resp.status, 200) + self.assertEqual(await resp.read(), b"legacy preview") + + async def test_a_missing_preview_is_made_in_the_store_on_first_view(self): + make_png(self.output_dir / "sub" / "b.png", size=(640, 480)) + url = _entry(self.api, self.output_dir, "sub/b.png")["thumbnail_url"] + resp = await self.client.get(url) + self.assertEqual(resp.status, 200) + self.assertTrue(self._preview("sub/b.png").is_file()) + self.assertFalse(self.old_store.exists()) + + async def test_a_leftover_in_the_source_folder_is_not_served(self): + make_png(self.output_dir / "c.png", size=(640, 480)) + leftover = self.old_store / "c_thumb.webp" + leftover.parent.mkdir() + leftover.write_bytes(b"old preview bytes") + resp = await self.client.get( + "/prompt_manager/images/serve/thumbnails/c_thumb.webp?root=0" + ) + self.assertEqual(resp.status, 200) + self.assertNotEqual(await resp.read(), b"old preview bytes") + self.assertTrue(self._preview("c.png").is_file()) + + async def test_capture_writes_into_the_store(self): + path = make_png(self.output_dir / "run" / "d.png", size=(64, 64)) + self.assertEqual(self.api.queue_capture_thumbnail(str(path)), "queued") + await self._idle() + self.assertTrue(self._preview("run/d.png").is_file()) + self.assertFalse(self.old_store.exists()) + + async def test_generate_and_clear_follow_the_store(self): + make_png(self.output_dir / "e.png", size=(64, 64)) + payload = await asyncio.get_running_loop().run_in_executor( + None, self.api._generate_gallery_thumbnails_sync, [self.output_dir] + ) + self.assertTrue(payload["success"]) + self.assertTrue(self._preview("e.png").is_file()) + self.assertFalse(self.old_store.exists()) + self.assertTrue(payload["thumbnails_path"].endswith("output-key")) + cleared, _size = self.api._clear_gallery_thumbnails_sync([self.output_dir]) + self.assertEqual(cleared, 1) + self.assertFalse(self._preview("e.png").exists()) + self.assertTrue(self.store.is_dir()) # the store itself stays + + async def test_removing_a_file_removes_its_preview_in_the_store(self): + from py.api.admin import _remove_thumbnail + + make_png(self.output_dir / "f.png", size=(64, 64)) + preview = self._preview("f.png") + preview.write_bytes(b"x") + self.assertTrue(_remove_thumbnail(self.output_dir, Path("f.png"))) + self.assertFalse(preview.exists()) + + async def test_the_temp_sweep_walks_the_store(self): + stray = self.store / ".g_thumb.part-1-1.webp" + stray.write_bytes(b"x") + old = os.path.getmtime(stray) - 10 * 24 * 3600 + os.utime(stray, (old, old)) + self.assertEqual(self.api.sweep_thumbnail_temps(), 1) + self.assertFalse(stray.exists()) + + async def test_history_cover_writes_into_the_store(self): + from py.api import history_current + + self.assertTrue( + history_current._preview_writable(self.output_dir, self._preview("h.png")) + ) + self.assertFalse( + history_current._preview_writable( + self.output_dir, self.old_store / "h_thumb.webp" + ) + ) + + +class TestCaptureResolvesAtWriteTime(RedirectedStoreCase): + def test_a_queued_capture_follows_a_store_that_moved_before_its_write(self): + from py.api import thumbnail_auto as auto + + written = [] + thumbnailer = auto.AutoThumbnailer( + lambda src, dst, *a, **k: written.append(dst) or {"action": "generated"}, + lambda: True, + self.api.logger, + start_worker=False, + ) + self.addCleanup(thumbnailer.close) + src = make_png(self.output_dir / "i.png", size=(32, 32)) + deferred = auto.DeferredPreview(self.output_dir, Path("i.png")) + self.assertTrue(thumbnailer.enqueue(src, deferred, False)) + moved = self.container / "moved" + moved.mkdir() + self.store = moved # the location changes before the worker runs + thumbnailer._write_queued(thumbnailer._queue.popleft()) + self.assertEqual(written, [moved / "i_thumb.webp"]) + + def test_the_after_scan_queue_maps_each_target_to_its_own_root(self): + from py.api import thumbnail_auto as auto + + nested = self.output_dir / "inner" + nested.mkdir() + src = nested / "j.png" + outer = auto._deferred( + [self.output_dir, nested], src, self._preview("inner/j.png") + ) + inner = auto._deferred( + [self.output_dir, nested], src, nested / "thumbnails" / "j_thumb.webp" + ) + self.assertEqual( + (outer.root, outer.rel), (self.output_dir, Path("inner/j.png")) + ) + self.assertEqual((inner.root, inner.rel), (nested, Path("j.png"))) + stray = self.comfy_dir / "x_thumb.webp" + self.assertEqual(auto._deferred([self.output_dir], src, stray), stray) + + def test_serve_and_history_share_one_flight_key(self): + from py.api import thumbnail_auto as auto + + parts = ("thumbnails", "sub", "a_thumb.webp") + self.assertEqual( + auto._request_key(self.output_dir, parts), + auto._dst_key(self._preview("sub/a.png")), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_preview_store_folder.py b/tests/test_preview_store_folder.py new file mode 100644 index 00000000..a7632fdc --- /dev/null +++ b/tests/test_preview_store_folder.py @@ -0,0 +1,604 @@ +"""One folder mode (#320 PR 2): previews in ``/``. + +Folder mode is reachable from config.json and tests only (D4B). These tests +set it on the resolver directly (``preview_store.configure``) and drive +every route that reads, writes, lists or removes previews: they must touch +only ``/``, never ``/thumbnails``, never through a link, +and nothing at all while the folder is unavailable. Waits are events; no +sleeps. +""" + +import asyncio +import json +import os +import sys +import threading +import unittest +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +try: + from tests.test_thumbnail_auto import AutoThumbnailCase, _entry, WAIT + from tests.test_api_images import make_png +except ImportError: # discovered with tests/ as the top-level directory + from test_thumbnail_auto import AutoThumbnailCase, _entry, WAIT + from test_api_images import make_png + +from py.api import preview_location, preview_store, thumbnail_settings # noqa: E402 + +SERVE = "/prompt_manager/images/serve/" + + +class FolderModeCase(AutoThumbnailCase): + """The output root's previews live in ``/thumbs/``.""" + + async def asyncSetUp(self): + await super().asyncSetUp() + self.folder = self.use_folder(self.comfy_dir / "thumbs") + self.old_store = self.output_dir / "thumbnails" + + def use_folder(self, folder, install="install-a"): + folder.mkdir(exist_ok=True) + marker_id = preview_location.init_folder(str(folder), created_by_pm=True) + self.addCleanup(preview_store.configure, None) + preview_store.configure( + {"mode": "folder", "path": str(folder), "id": marker_id, "install": install} + ) + self.assertTrue(preview_store.current().available) + return folder + + def key_folder(self): + store = preview_store.ensure_store(self.output_dir, create=True) + self.assertIsNotNone(store) + return store + + def preview(self, rel, ext=".webp"): + return self.key_folder() / (thumbnail_settings.preview_stem(rel) + ext) + + def marker(self): + return json.loads( + (self.folder / preview_location.MARKER_NAME).read_text(encoding="utf-8") + ) + + +class TestFolderModeRoutes(FolderModeCase): + async def test_the_key_is_a_readable_name_plus_a_hash_in_the_marker(self): + key = self.key_folder().name + self.assertRegex(key, r"^output-[0-9a-f]{8}$") + entry = self.marker()["keys"][key] + self.assertEqual(entry["path"], str(self.output_dir)) + self.assertEqual(entry["install"], "install-a") + + async def test_another_case_of_the_source_finds_the_same_key(self): + key = self.key_folder() + other = Path(str(self.output_dir).swapcase()) + if not other.is_dir(): + self.skipTest("a case-sensitive volume") + self.assertEqual(preview_store.store_dir(other), key) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_a_linked_spelling_of_the_source_finds_the_same_key(self): + key = self.key_folder() + link = self.comfy_dir / "output-link" + try: + link.symlink_to(self.output_dir, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + self.assertEqual(preview_store.store_dir(link), key) + + async def test_an_existing_preview_is_listed_and_served_from_the_key_folder(self): + make_png(self.output_dir / "sub" / "a.png", size=(64, 64)) + preview = self.preview("sub/a.png") + make_png(preview, size=(32, 32)) + url = _entry(self.api, self.output_dir, "sub/a.png")["thumbnail_url"] + self.assertIn("v=", url) + self.assertTrue(url.startswith(SERVE + "thumbnails/sub/a_thumb.webp")) + resp = await self.client.get(url) + self.assertEqual(resp.status, 200) + self.assertEqual(await resp.read(), preview.read_bytes()) + + async def test_a_missing_preview_is_made_in_the_key_folder(self): + make_png(self.output_dir / "b.png", size=(640, 480)) + url = _entry(self.api, self.output_dir, "b.png")["thumbnail_url"] + resp = await self.client.get(url) + self.assertEqual(resp.status, 200) + self.assertTrue(self.preview("b.png").is_file()) + self.assertFalse(self.old_store.exists()) + + async def test_a_leftover_in_the_source_folder_is_never_served(self): + make_png(self.output_dir / "c.png", size=(640, 480)) + self.old_store.mkdir() + (self.old_store / "c_thumb.webp").write_bytes(b"leftover") + resp = await self.client.get(SERVE + "thumbnails/c_thumb.webp?root=0") + self.assertEqual(resp.status, 200) + self.assertNotEqual(await resp.read(), b"leftover") + + async def test_other_text_under_thumbnails_is_refused(self): + self.old_store.mkdir() + make_png(self.old_store / "notes.png", size=(8, 8)) + resp = await self.client.get(SERVE + "thumbnails/notes.png?root=0") + self.assertEqual(resp.status, 403) + + async def test_traversal_through_thumbnails_is_refused(self): + for text in ( + "thumbnails/../../thumbs/x_thumb.webp", + "thumbnails/..%2F..%2Fx_thumb.webp", + "thumbnails/%2e%2e/x_thumb.webp", + ): + with self.subTest(text=text): + resp = await self.client.get(SERVE + text + "?root=0") + self.assertIn(resp.status, (403, 404)) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_a_link_out_of_the_key_folder_is_never_served_or_written(self): + outside = self.comfy_dir / "outside" + outside.mkdir() + make_png(outside / "d_thumb.webp", size=(8, 8)) + make_png(self.output_dir / "sub" / "d.png", size=(64, 64)) + try: + (self.key_folder() / "sub").symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + resp = await self.client.get(SERVE + "thumbnails/sub/d_thumb.webp?root=0") + self.assertEqual(resp.status, 403) + self.assertFalse(preview_store.owns(self.output_dir, outside / "d_thumb.webp")) + self.assertFalse( + preview_store.owns(self.output_dir, self.key_folder() / "sub" / "x.webp") + ) + + @unittest.skipUnless(os.name == "nt", "junctions are Windows only") + async def test_a_junction_inside_a_store_is_refused(self): + import _winapi + + outside = self.comfy_dir / "outside" + outside.mkdir() + (outside / "e_thumb.webp").write_bytes(b"x") + _winapi.CreateJunction(str(outside), str(self.key_folder() / "sub")) + self.assertTrue(preview_location.is_link_like(self.key_folder() / "sub")) + resp = await self.client.get(SERVE + "thumbnails/sub/e_thumb.webp?root=0") + self.assertEqual(resp.status, 403) + + async def test_capture_writes_into_the_key_folder(self): + path = make_png(self.output_dir / "run" / "f.png", size=(64, 64)) + self.assertEqual(self.api.queue_capture_thumbnail(str(path)), "queued") + await self._idle() + self.assertTrue(self.preview("run/f.png").is_file()) + self.assertFalse(self.old_store.exists()) + + async def test_clear_removes_previews_only(self): + make_png(self.output_dir / "g.png", size=(64, 64)) + payload = await asyncio.get_running_loop().run_in_executor( + None, self.api._generate_gallery_thumbnails_sync, [self.output_dir] + ) + self.assertTrue(payload["success"]) + keep = self.key_folder() / "readme.txt" + keep.write_text("not a preview", encoding="utf-8") + resp = await self.client.post("/prompt_manager/images/clear-thumbnails") + self.assertEqual(resp.status, 200) + self.assertFalse(self.preview("g.png").exists()) + self.assertTrue(keep.exists()) + self.assertTrue((self.folder / preview_location.MARKER_NAME).is_file()) + self.assertTrue(self.output_dir.joinpath("g.png").is_file()) + + async def test_removing_a_file_removes_its_preview_in_the_key_folder(self): + from py.api.admin import _remove_thumbnail + + make_png(self.output_dir / "h.png", size=(64, 64)) + preview = self.preview("h.png") + preview.write_bytes(b"x") + self.assertTrue(_remove_thumbnail(self.output_dir, Path("h.png"))) + self.assertFalse(preview.exists()) + + async def test_history_cover_is_authorized_by_the_store(self): + from py.api import history_current + + preview = self.preview("i.png") + preview.write_bytes(b"x") + target = history_current._canonical(preview) + self.assertTrue( + history_current._target_authorized( + target, self.output_dir, None, self.output_dir + ) + ) + stray = self.comfy_dir / "stray_thumb.webp" + stray.write_bytes(b"x") + self.assertFalse( + history_current._target_authorized( + history_current._canonical(stray), + self.output_dir, + None, + self.output_dir, + ) + ) + + async def test_the_thumbnail_folder_is_never_recreated_by_a_write(self): + make_png(self.output_dir / "j.png", size=(64, 64)) + self.key_folder() + for child in sorted(self.folder.rglob("*"), reverse=True): + child.rmdir() if child.is_dir() else child.unlink() + self.folder.rmdir() + with self.assertRaises(FileNotFoundError): + preview_store.make_parent(self.folder / "output-x" / "sub" / "a.webp") + self.assertFalse(self.folder.exists()) + + +class TestUnavailableFolder(FolderModeCase): + def go_missing(self): + (self.folder / preview_location.MARKER_NAME).unlink() + self.assertFalse(preview_store.refresh()) + location = preview_store.current() + self.assertEqual((location.available, location.problem), (False, "missing")) + + async def test_a_folder_without_its_marker_is_missing_and_writes_nothing(self): + make_png(self.output_dir / "k.png", size=(640, 480)) + self.go_missing() + entry = _entry(self.api, self.output_dir, "k.png") + self.assertFalse(entry.get("thumbnail_ready", False)) + resp = await self.client.get(SERVE + "thumbnails/k_thumb.webp?root=0") + self.assertEqual(resp.status, 200) + self.assertEqual(resp.headers["Cache-Control"], "no-store") + self.assertEqual( + await resp.read(), (self.output_dir / "k.png").read_bytes() + ) # the original + self.assertEqual( + sorted(p.name for p in self.folder.rglob("*") if p.is_file()), [] + ) + self.assertFalse(self.old_store.exists()) + + async def test_thumbnail_work_is_refused_while_unavailable(self): + self.go_missing() + for method, url, body in ( + ("POST", "/prompt_manager/images/generate-thumbnails", {}), + ("POST", "/prompt_manager/images/generate-thumbnails/progress", None), + ("POST", "/prompt_manager/images/clear-thumbnails", None), + ("POST", "/prompt_manager/jobs/thumbnails", {"mode": "missing"}), + ): + with self.subTest(url=url): + resp = await self.client.request(method, url, json=body) + self.assertEqual(resp.status, 409) + self.assertEqual((await resp.json())["code"], "location-unavailable") + + async def test_status_reports_the_location_without_a_path(self): + self.go_missing() + resp = await self.client.get("/prompt_manager/images/thumbnails/status") + location = (await resp.json())["location"] + self.assertEqual( + location, {"mode": "folder", "available": False, "problem": "missing"} + ) + + async def test_another_folders_marker_is_a_mismatch(self): + marker = self.marker() + marker["id"] = "someone-else" + (self.folder / preview_location.MARKER_NAME).write_text( + json.dumps(marker), encoding="utf-8" + ) + self.assertFalse(preview_store.refresh()) + self.assertEqual(preview_store.current().problem, "marker-mismatch") + + async def test_the_temp_sweep_skips_an_unavailable_folder(self): + stray = self.key_folder() / ".l_thumb.part-1-1.webp" + stray.write_bytes(b"x") + old = os.path.getmtime(stray) - 10 * 24 * 3600 + os.utime(stray, (old, old)) + self.go_missing() + self.assertEqual(self.api.sweep_thumbnail_temps(), 0) + self.assertTrue(stray.exists()) + + async def test_a_stalled_store_call_marks_it_unreachable_and_stops_io(self): + make_png(self.output_dir / "m.png", size=(64, 64)) + self.preview("m.png").write_bytes(b"x") + ticks = iter([0.0, preview_store.IO_BUDGET + 1.0]) + with patch.object(preview_store, "_clock", lambda: next(ticks, 99.0)): + _entry(self.api, self.output_dir, "m.png") # one slow lookup + self.assertEqual(preview_store.current().problem, "unreachable") + calls = [] + real = thumbnail_settings.owned_in_folder + with patch.object( + thumbnail_settings, + "owned_in_folder", + lambda *a, **k: calls.append(a) or real(*a, **k), + ): + entry = _entry(self.api, self.output_dir, "m.png") + self.assertEqual(calls, []) # no store I/O while unreachable + self.assertNotIn("v=", entry["thumbnail_url"]) + self.assertTrue(preview_store.refresh()) # the probe finds it again + + async def test_a_check_that_blocks_is_unreachable_within_the_budget(self): + entered = threading.Event() + release = threading.Event() + self.addCleanup(release.set) + + def blocking(path, expected_id): + entered.set() + release.wait(WAIT) + return True, None, None + + with ( + patch.object(preview_store, "IO_BUDGET", 0.2), + patch.object(preview_location, "folder_state", blocking), + ): + self.assertFalse(preview_store.refresh()) + self.assertTrue(entered.is_set()) + self.assertEqual(preview_store.current().problem, "unreachable") + release.set() + + async def test_an_invalid_config_value_writes_nothing_anywhere(self): + preview_store.configure({"mode": "folder", "path": "relative/thumbs"}) + location = preview_store.current() + self.assertEqual((location.available, location.problem), (False, "invalid")) + make_png(self.output_dir / "n.png", size=(640, 480)) + resp = await self.client.get(SERVE + "thumbnails/n_thumb.webp?root=0") + self.assertEqual(resp.status, 200) + self.assertFalse(self.old_store.exists()) + + +class TestSharedFolders(FolderModeCase): + async def test_two_installs_keep_each_others_keys(self): + self.key_folder() + other_root = self.comfy_dir / "other" + other_root.mkdir() + setting = preview_store.setting() + preview_store.configure( + { + "mode": "folder", + "path": setting.path, + "id": setting.id, + "install": "install-b", + } + ) + self.assertIsNotNone(preview_store.ensure_store(other_root)) + keys = self.marker()["keys"] + installs = sorted(entry["install"] for entry in keys.values()) + self.assertEqual(installs, ["install-a", "install-b"]) + + async def test_init_folder_adopts_an_existing_marker(self): + marker_id = self.marker()["id"] + self.assertEqual(preview_location.init_folder(str(self.folder)), marker_id) + + +if __name__ == "__main__": + unittest.main() + + +class TestHostileMarker(FolderModeCase): + """A hand-edited or foreign marker never takes a write or a delete out of + the thumbnail folder (shared and UNC folders can be edited by others).""" + + def plant(self, keys): + marker = self.marker() + marker["keys"] = keys + (self.folder / preview_location.MARKER_NAME).write_text( + json.dumps(marker), encoding="utf-8" + ) + self.assertTrue(preview_store.refresh()) + + def outside_files(self): + return sorted( + p.relative_to(self.comfy_dir).as_posix() + for p in self.comfy_dir.rglob("*") + if p.is_file() + and not p.is_relative_to(self.folder) + and not p.is_relative_to(self.output_dir) + and p.suffix in (".webp", ".jpg", ".png") + ) + + async def test_keys_that_are_not_one_folder_name_are_dropped(self): + entry = {"path": str(self.output_dir), "identity": "x", "install": "i"} + self.plant( + { + "../escape-0123abcd": entry, + str(self.comfy_dir / "abs-0123abcd"): entry, + "a/b-0123abcd": entry, + "a\\b-0123abcd": entry, + "Upper-0123abcd": entry, + "nohash": entry, + "fine-0123abcd": "not a dict", + } + ) + self.assertEqual(preview_location.read_marker(str(self.folder))["keys"], {}) + store = self.key_folder() + self.assertEqual(store.parent, self.folder) + make_png(self.output_dir / "p.png", size=(640, 480)) + resp = await self.client.get(SERVE + "thumbnails/p_thumb.webp?root=0") + self.assertEqual(resp.status, 200) + self.assertTrue((store / "p_thumb.webp").is_file()) + self.assertEqual(self.outside_files(), []) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_a_key_folder_that_is_a_link_is_never_written_or_cleared(self): + outside = self.comfy_dir / "outside" + outside.mkdir() + victim = outside / "q_thumb.webp" + victim.write_bytes(b"keep me") + store = self.key_folder() + store.rmdir() + try: + store.symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + make_png(self.output_dir / "q.png", size=(640, 480)) + from py.api.admin import _remove_thumbnail + + self.assertFalse(_remove_thumbnail(self.output_dir, Path("q.png"))) + resp = await self.client.post("/prompt_manager/images/clear-thumbnails") + self.assertIn(resp.status, (200, 400)) + resp = await self.client.get(SERVE + "thumbnails/q_thumb.webp?root=0") + self.assertIn(resp.status, (200, 403)) + self.assertEqual(victim.read_bytes(), b"keep me") + self.assertEqual(sorted(p.name for p in outside.iterdir()), ["q_thumb.webp"]) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_a_linked_folder_below_a_key_is_never_deleted_through(self): + outside = self.comfy_dir / "outside" + outside.mkdir() + victim = outside / "r_thumb.webp" + victim.write_bytes(b"keep me") + try: + (self.key_folder() / "sub").symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + make_png(self.output_dir / "sub" / "r.png", size=(64, 64)) + from py.api.admin import _remove_thumbnail + + self.assertFalse(_remove_thumbnail(self.output_dir, Path("sub/r.png"))) + resp = await self.client.post("/prompt_manager/images/clear-thumbnails") + self.assertEqual(resp.status, 200) + self.assertEqual(victim.read_bytes(), b"keep me") + + @unittest.skipUnless(os.name == "nt", "junctions are Windows only") + async def test_a_key_folder_that_is_a_junction_is_never_written_or_cleared(self): + import _winapi + + outside = self.comfy_dir / "outside" + outside.mkdir() + victim = outside / "v_thumb.webp" + victim.write_bytes(b"keep me") + store = self.key_folder() + store.rmdir() + _winapi.CreateJunction(str(outside), str(store)) + make_png(self.output_dir / "v.png", size=(640, 480)) + from py.api.admin import _remove_thumbnail + + self.assertFalse(_remove_thumbnail(self.output_dir, Path("v.png"))) + resp = await self.client.post("/prompt_manager/images/clear-thumbnails") + self.assertIn(resp.status, (200, 400)) + resp = await self.client.get(SERVE + "thumbnails/v_thumb.webp?root=0") + self.assertIn(resp.status, (200, 403)) + self.assertEqual(victim.read_bytes(), b"keep me") + self.assertEqual(sorted(p.name for p in outside.iterdir()), ["v_thumb.webp"]) + + def link_key_folder_outside(self): + """The source's key folder replaced by a link to a folder outside.""" + outside = self.comfy_dir / "outside" + outside.mkdir() + store = self.key_folder() + store.rmdir() + try: + store.symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + return outside + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_convert_never_plans_or_deletes_through_a_linked_key_folder(self): + outside = self.link_key_folder_outside() + old = outside / "c_thumb.png" + old.write_bytes(b"keep me") + make_png(self.output_dir / "c.png", size=(640, 480)) + # A linked store is refused before any planning (bot review, P1). + self.assertIsNone(preview_store.ensure_store(self.output_dir)) + plan = self.api._plan_gallery_thumbnails([self.output_dir], mode="convert") + self.assertEqual(plan["targets"], []) + # And every delete passes the ownership check at the moment it runs. + removed, kept = self.api._remove_old_previews([(old, 7)]) + self.assertEqual((removed, kept), (0, [old])) + self.assertEqual(old.read_bytes(), b"keep me") + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_the_temp_sweep_never_walks_into_a_link(self): + from py.api import thumbnail_encode + + outside = self.comfy_dir / "outside" + outside.mkdir() + victim = outside / ".v_thumb.part-1-1.webp" + victim.write_bytes(b"keep me") + os.utime(victim, (1, 1)) + try: + (self.key_folder() / "sub").symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + real_walk = os.walk + + def windows_walk(top, *args, **kwargs): + # Windows walks into a junction even with followlinks=False. + kwargs["followlinks"] = True + return real_walk(top, *args, **kwargs) + + with patch.object(thumbnail_encode.os, "walk", windows_walk): + self.assertEqual(self.api.sweep_thumbnail_temps(), 0) + thumbnail_encode.sweep_stray_temps(self.key_folder(), min_age=0) + resp = await self.client.post("/prompt_manager/images/clear-thumbnails") + self.assertEqual(resp.status, 200) + self.assertEqual(victim.read_bytes(), b"keep me") + + def test_the_temp_sweep_leaves_a_temp_the_store_does_not_own(self): + from py.api import thumbnail_encode + + temp = self.key_folder() / ".w_thumb.part-1-1.webp" + temp.write_bytes(b"x") + with patch.object(preview_store, "may_write", return_value=False): + self.assertEqual( + thumbnail_encode.sweep_stray_temps(self.key_folder(), min_age=0), (0, 0) + ) + self.assertTrue(temp.exists()) + + async def test_a_write_outside_the_folder_is_refused(self): + with self.assertRaises(OSError): + preview_store.make_parent(self.comfy_dir / "elsewhere" / "a_thumb.webp") + self.assertFalse((self.comfy_dir / "elsewhere").exists()) + + async def test_an_unmounted_folder_takes_no_write_before_the_next_check(self): + """An unmounted mount point is an empty folder: the flag may still + say available, but without the marker nothing is made there.""" + store = self.key_folder() + store.rmdir() + (self.folder / preview_location.MARKER_NAME).unlink() + self.assertTrue(preview_store.current().available) # not checked yet + with self.assertRaises(OSError): + preview_store.make_parent(store / "sub" / "s_thumb.webp") + self.assertEqual(list(self.folder.iterdir()), []) + + +class TestStartupNeverFallsBackToTheSources(AutoThumbnailCase): + async def test_a_folder_setting_is_in_force_before_its_check(self): + self.addCleanup(preview_store.configure, None) + folder = self.comfy_dir / "thumbs" + value = {"mode": "folder", "path": str(folder), "id": "x"} + preview_store.configure(value, check=False) + location = preview_store.current() + self.assertEqual((location.mode, location.available), ("folder", False)) + make_png(self.output_dir / "t.png", size=(640, 480)) + resp = await self.client.get(SERVE + "thumbnails/t_thumb.webp?root=0") + self.assertEqual(resp.status, 200) + self.assertFalse((self.output_dir / "thumbnails").exists()) + + async def test_a_failed_startup_check_leaves_the_folder_invalid(self): + from py.config import PromptManagerConfig + + self.addCleanup(preview_store.configure, None) + value = {"mode": "folder", "path": str(self.comfy_dir / "thumbs"), "id": "x"} + with ( + patch.object(PromptManagerConfig, "THUMBNAIL_LOCATION", value), + patch.object( + type(self.api), + "_thumbnail_location_context", + side_effect=RuntimeError("boom"), + ), + ): + self.api.start_preview_location() + location = preview_store.current() + self.assertEqual( + (location.mode, location.available, location.problem), + ("folder", False, "invalid"), + ) + + +class TestNoStoreIoOnTheEventLoop(FolderModeCase): + async def test_legacy_listings_do_not_stat_the_folder(self): + make_png(self.output_dir / "u.png", size=(64, 64)) + self.preview("u.png").write_bytes(b"x") + calls = [] + real = thumbnail_settings.owned_in_folder + with patch.object( + thumbnail_settings, + "owned_in_folder", + lambda *a, **k: calls.append(threading.current_thread().name) + or real(*a, **k), + ): + images = self.api._enrich_images( + [{"image_path": str(self.output_dir / "u.png")}] + ) + self.assertEqual(calls, []) + self.assertEqual(images[0]["thumbnail_url"], SERVE + "thumbnails/u_thumb.webp") diff --git a/tests/test_thumbnail_move.py b/tests/test_thumbnail_move.py new file mode 100644 index 00000000..790afab6 --- /dev/null +++ b/tests/test_thumbnail_move.py @@ -0,0 +1,542 @@ +"""Move thumbnails (#320 PR 3): the reviewed job in every direction, its +failure, cancel and restart paths, Recreate and the removed-sources scope. + +Each test runs against a temp config.json (``AdminAPITestCase``) and a temp +journal; jobs are awaited through their task, never a timer. +""" + +import json +import os +import re +import sys +import unittest +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +try: + from tests.test_api_admin import AdminAPITestCase + from tests.test_api_images import make_png +except ImportError: # discovered with tests/ as the top-level directory + from test_api_admin import AdminAPITestCase + from test_api_images import make_png + +from py.api import preview_location, preview_store # noqa: E402 +from py.api import thumbnail_move_files as files # noqa: E402 +from py.api import thumbnail_move_plan as move_plan # noqa: E402 +from py.config import PromptManagerConfig # noqa: E402 + +REPO = Path(__file__).resolve().parent.parent +PLAN = "/prompt_manager/images/thumbnails/plan" +START = "/prompt_manager/jobs/thumbnails" + + +class MoveCase(AdminAPITestCase): + async def asyncSetUp(self): + await super().asyncSetUp() + self.api._get_all_output_dirs = lambda: [self.output_dir] + self.api.recover_thumbnail_move() + self.journal = Path(self.tmpdir) / "data" / "thumbnail-move.json" + for target in (move_plan,): + patcher = patch.object(target, "journal_path", lambda: str(self.journal)) + patcher.start() + self.addCleanup(patcher.stop) + default = patch( + "py.api.thumbnail_move_plan.default_path", + lambda: str(Path(self.tmpdir) / "default-thumbs"), + ) + default.start() + self.addCleanup(default.stop) + self.addCleanup(preview_store.configure, None) + self.addCleanup(setattr, PromptManagerConfig, "THUMBNAIL_LOCATION", None) + self.old_store = self.output_dir / "thumbnails" + self.folder = Path(self.tmpdir) / "thumbs" + # Two media files with previews, an orphan, a temp and a note. + make_png(self.output_dir / "a.png", size=(8, 8)) + make_png(self.output_dir / "sub" / "b.png", size=(8, 8)) + make_png(self.old_store / "a_thumb.webp", size=(4, 4)) + make_png(self.old_store / "sub" / "b_thumb.webp", size=(4, 4)) + make_png(self.old_store / "gone_thumb.webp", size=(4, 4)) + (self.old_store / ".a_thumb.part-1-1.webp").write_bytes(b"t") + (self.old_store / "notes.txt").write_text("mine", encoding="utf-8") + + async def plan(self, location): + resp = await self.client.post(PLAN, json={"mode": "move", "location": location}) + return resp.status, await resp.json() + + async def move(self, location, confirmed=True): + status, plan = await self.plan(location) + self.assertEqual(status, 200, plan) + resp = await self.client.post( + START, json={"mode": "move", "token": plan["token"], "confirmed": confirmed} + ) + body = await resp.json() + self.assertEqual(resp.status, 202, body) + job = self.api._jobs.get(body["job"]["id"]) + await job.task + return plan, job + + def config(self): + with open(self.config_path, encoding="utf-8") as handle: + return json.load(handle) + + def key_files(self, folder=None): + folder = folder or self.folder + return sorted( + p.relative_to(folder).as_posix() + for p in folder.rglob("*") + if p.is_file() and p.name != preview_location.MARKER_NAME + ) + + +class TestMoveToAFolder(MoveCase): + async def test_review_counts_what_moves_is_removed_and_stays(self): + status, plan = await self.plan({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(status, 200, plan) + totals = plan["totals"] + self.assertEqual( + ( + totals["files"], + totals["orphans"], + totals["temps"], + totals["left_in_place"], + ), + (2, 1, 1, 1), + ) + self.assertTrue(plan["confirm_required"]) + self.assertTrue(plan["to"]["will_create"]) + self.assertFalse(self.folder.exists()) # the review writes nothing + + async def test_a_move_copies_switches_and_removes_old_copies(self): + before = (self.output_dir / "a.png").read_bytes() + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(job.state, "finished", job.log) + self.assertEqual(job.counters["moved"], 2) + keys = self.key_files() + self.assertEqual(len(keys), 2) + self.assertTrue(all(re.match(r"^output-[0-9a-f]{8}/", k) for k in keys)) + location = self.config()["thumbnails"]["location"] + self.assertEqual( + (location["mode"], location["path"]), ("folder", str(self.folder)) + ) + self.assertEqual(location["cleanup"], []) + self.assertEqual(preview_store.current().mode, "folder") + # Old copies, the orphan and the temp went; the note stays, so its + # folder stays; originals are untouched. + self.assertEqual( + sorted(p.name for p in self.old_store.rglob("*")), ["notes.txt"] + ) + self.assertEqual((self.output_dir / "a.png").read_bytes(), before) + self.assertFalse(self.journal.exists()) + copied = next(self.folder.rglob("a_thumb.webp")) + self.assertEqual(copied.stat().st_mtime_ns % files.MTIME_STEP_NS, 0) + + async def test_the_move_back_removes_the_folder_it_made(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + _plan, job = await self.move({"mode": "source"}) + self.assertEqual(job.state, "finished", job.log) + self.assertTrue((self.old_store / "a_thumb.webp").is_file()) + self.assertTrue((self.old_store / "sub" / "b_thumb.webp").is_file()) + self.assertFalse(self.folder.exists()) + self.assertEqual(self.config()["thumbnails"]["location"]["mode"], "source") + + async def test_folder_to_folder_keeps_the_key_names(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + first = self.key_files() + other = Path(self.tmpdir) / "thumbs2" + _plan, job = await self.move({"mode": "folder", "path": str(other)}) + self.assertEqual(job.state, "finished", job.log) + self.assertEqual(self.key_files(other), first) + self.assertFalse(self.folder.exists()) + + async def test_previews_of_sources_no_longer_listed_go_when_leaving(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + setting = preview_store.setting() + + def add(marker): + marker["keys"]["old-0123abcd"] = { + "path": "/gone", + "identity": "/gone", + "install": setting.install, + } + + preview_location.write_marker(str(self.folder), add) + make_png(self.folder / "old-0123abcd" / "x_thumb.webp", size=(4, 4)) + preview_store.refresh() + status, plan = await self.plan({"mode": "source"}) + self.assertEqual(status, 200) + self.assertEqual(plan["totals"]["unlisted_sources"], 1) + _, job = await self.move({"mode": "source"}) + self.assertEqual(job.state, "finished", job.log) + self.assertFalse(self.folder.exists()) + + async def test_confirmation_and_a_stale_token(self): + status, plan = await self.plan({"mode": "folder", "path": str(self.folder)}) + resp = await self.client.post( + START, json={"mode": "move", "token": plan["token"]} + ) + self.assertEqual((await resp.json())["code"], "confirmation-required") + resp = await self.client.post( + START, json={"mode": "move", "token": "nope", "confirmed": True} + ) + self.assertEqual(resp.status, 409) + self.assertEqual((await resp.json())["code"], "plan-stale") + + async def test_a_folder_that_is_too_small_is_refused(self): + with patch.object(move_plan, "_free", lambda _path: 1): + status, body = await self.plan({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(status, 507) + self.assertEqual(body["code"], "insufficient-space") + + async def test_validation_refusals_come_back_in_place(self): + status, body = await self.plan( + {"mode": "folder", "path": str(self.output_dir / "thumbs")} + ) + self.assertEqual((status, body["code"]), (400, "location-served")) + status, body = await self.plan({"mode": "source"}) + self.assertEqual(body["code"], "location-same") + + +class TestFailures(MoveCase): + async def test_a_per_file_error_still_switches_and_removes_its_old_copy(self): + real = files.copy_one + + def failing(source, destination, *args, **kwargs): + if Path(source).name == "a_thumb.webp": + raise files.CopyFailed("Could not be read; made again on view") + return real(source, destination, *args, **kwargs) + + with patch.object(files, "copy_one", failing): + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(job.state, "finished") + self.assertEqual((job.counters["moved"], job.counters["errors"]), (1, 1)) + self.assertFalse((self.old_store / "a_thumb.webp").exists()) + + async def test_a_folder_wide_error_changes_nothing(self): + def gone(*_args, **_kwargs): + raise files.FolderGone("disk full") + + with patch.object(files, "copy_one", gone): + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(job.state, "failed") + self.assertIn("nothing changed", job.status_line) + self.assertFalse(self.folder.exists()) + self.assertTrue((self.old_store / "a_thumb.webp").is_file()) + self.assertEqual(preview_store.current().mode, "source") + self.assertFalse(self.journal.exists()) + + async def test_cancel_before_the_switch_changes_nothing(self): + status, plan = await self.plan({"mode": "folder", "path": str(self.folder)}) + real = self.api._move_copy_one + + def cancel_first(pair, **kwargs): + job = self.api._jobs.active("thumbnails") + job.cancelled = True + return real(pair, **kwargs) + + with patch.object(self.api, "_move_copy_one", cancel_first): + resp = await self.client.post( + START, json={"mode": "move", "token": plan["token"], "confirmed": True} + ) + job = self.api._jobs.get((await resp.json())["job"]["id"]) + await job.task + self.assertEqual(job.state, "cancelled") + self.assertFalse(self.folder.exists()) + self.assertEqual(preview_store.current().mode, "source") + + async def test_an_exit_during_the_copy_leaves_the_journal_for_the_next_start(self): + status, plan = await self.plan({"mode": "folder", "path": str(self.folder)}) + real = self.api._move_copy_one + exiting = {"now": False} + + def exit_after(pair, **kwargs): + result = real(pair, **kwargs) + exiting["now"] = True # the interpreter starts exiting + return result + + from py.api import thumbnail_move + + with ( + patch.object(self.api, "_move_copy_one", exit_after), + patch.object( + thumbnail_move.exit_signal, "requested", lambda: exiting["now"] + ), + ): + resp = await self.client.post( + START, json={"mode": "move", "token": plan["token"], "confirmed": True} + ) + job = self.api._jobs.get((await resp.json())["job"]["id"]) + await job.task + self.assertTrue(self.journal.exists()) + self.assertTrue(self.key_files()) # partial copies were left + # The next start undoes it: copies and the created folder go. + self.api._move_recovered = False + self.api.recover_thumbnail_move() + self.assertFalse(self.journal.exists()) + self.assertFalse(self.folder.exists()) + self.assertTrue((self.old_store / "a_thumb.webp").is_file()) + + +class TestRestart(MoveCase): + async def test_a_journal_left_after_the_switch_is_forgotten(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + self.journal.parent.mkdir(parents=True, exist_ok=True) + self.journal.write_text( + json.dumps( + {"state": "copying", "to": {"mode": "folder", "path": str(self.folder)}} + ), + encoding="utf-8", + ) + self.api._move_recovered = False + self.api.recover_thumbnail_move() + self.assertFalse(self.journal.exists()) + self.assertTrue(self.key_files()) + + async def test_old_copies_that_waited_are_removed_by_the_finishing_job(self): + with patch.object(files, "remove_file", lambda path, **_k: False): + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertTrue(preview_store.setting().cleanup) + self.assertTrue((self.old_store / "a_thumb.webp").exists()) + # Each removal that failed is an error the job counts and lists by path. + rows = job.outcomes["errors"] + self.assertEqual(job.counters["errors"], len(rows)) + self.assertIn("sub/b_thumb.webp", {row["path"] for row in rows}) + self.assertTrue(all("next start" in row["detail"] for row in rows)) + job = await self.api.finish_thumbnail_move() + await job.task + self.assertEqual(job.title, "Move thumbnails (finishing)") + self.assertFalse((self.old_store / "a_thumb.webp").exists()) + self.assertEqual(self.config()["thumbnails"]["location"]["cleanup"], []) + + +class TestAdmission(MoveCase): + async def test_automatic_previews_and_setting_changes_wait_for_the_move(self): + status, plan = await self.plan({"mode": "folder", "path": str(self.folder)}) + seen = {} + real = self.api._move_copy_one + + def observe(pair, **kwargs): + seen["blocker"] = self.api._auto_thumbnail_blocker() + return real(pair, **kwargs) + + with patch.object(self.api, "_move_copy_one", observe): + resp = await self.client.post( + START, json={"mode": "move", "token": plan["token"], "confirmed": True} + ) + job = self.api._jobs.get((await resp.json())["job"]["id"]) + refused = self.api._thumbnail_change_refused({"THUMBNAIL_QUALITY": 5}) + self.assertEqual(refused.status, 409) + await job.task + self.assertIs(seen["blocker"], job) + + def test_the_move_never_trims_memory_and_copies_on_the_job_pool(self): + text = "".join( + (REPO / "py" / "api" / name).read_text(encoding="utf-8") + for name in ("thumbnail_move.py", "thumbnail_prune.py") + ) + self.assertNotIn("_release_job_memory", text) + self.assertIn("map_parallel", (REPO / "py/api/thumbnail_move.py").read_text()) + + +class TestPrune(MoveCase): + async def test_only_this_installs_removed_sources_go(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + install = preview_store.setting().install + + def add(marker): + marker["keys"]["old-0123abcd"] = {"identity": "/gone", "install": install} + marker["keys"]["theirs-0123abcd"] = {"identity": "/x", "install": "other"} + + preview_location.write_marker(str(self.folder), add) + make_png(self.folder / "old-0123abcd" / "x_thumb.webp", size=(4, 4)) + make_png(self.folder / "theirs-0123abcd" / "y_thumb.webp", size=(4, 4)) + preview_store.refresh() + resp = await self.client.get(PLAN + "?mode=prune") + body = await resp.json() + self.assertEqual([s["key"] for s in body["sources"]], ["old-0123abcd"]) + self.assertIn("1 source no longer listed", body["line"]) + resp = await self.client.post(START, json={"mode": "prune"}) + job = self.api._jobs.get((await resp.json())["job"]["id"]) + await job.task + self.assertFalse((self.folder / "old-0123abcd").exists()) + self.assertTrue((self.folder / "theirs-0123abcd" / "y_thumb.webp").exists()) + keys = preview_location.read_marker(str(self.folder))["keys"] + self.assertNotIn("old-0123abcd", keys) + + +class TestRecreate(MoveCase): + async def test_recreate_and_its_refusals(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + url = "/prompt_manager/images/thumbnails/location/recreate" + resp = await self.client.post(url) + self.assertEqual((await resp.json())["code"], "location-has-marker") + for child in sorted(self.folder.rglob("*"), reverse=True): + child.rmdir() if child.is_dir() else child.unlink() + self.folder.rmdir() + self.assertFalse(preview_store.refresh()) + resp = await self.client.post(url) + self.assertEqual(resp.status, 200, await resp.text()) + self.assertTrue(preview_store.current().available) + self.assertNotEqual(self.config()["thumbnails"]["location"]["id"], None) + + async def test_use_this_folder_adopts_another_marker(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + marker = preview_location.read_marker(str(self.folder)) + marker["id"] = "someone-else" + (self.folder / preview_location.MARKER_NAME).write_text( + json.dumps(marker), encoding="utf-8" + ) + self.assertFalse(preview_store.refresh()) + resp = await self.client.post( + "/prompt_manager/images/thumbnails/location/adopt" + ) + self.assertEqual(resp.status, 200, await resp.text()) + self.assertTrue(preview_store.current().available) + self.assertEqual(self.config()["thumbnails"]["location"]["id"], "someone-else") + + +class TestFiles(unittest.TestCase): + def test_mtime_rounds_up_to_two_seconds(self): + self.assertEqual(files.rounded_mtime_ns(0), 0) + self.assertEqual(files.rounded_mtime_ns(1), files.MTIME_STEP_NS) + self.assertEqual( + files.rounded_mtime_ns(files.MTIME_STEP_NS), files.MTIME_STEP_NS + ) + + def test_long_windows_destinations_are_per_file_errors(self): + self.assertTrue(files.too_long("D:\\" + "x" * 240, long_paths=False)) + self.assertFalse(files.too_long("D:\\x", long_paths=False)) + + def test_a_failed_copys_temp_that_cannot_go_is_logged(self): + with patch.object(files.os, "unlink", side_effect=PermissionError("held")): + with self.assertLogs("prompt_manager.thumbnails", "WARNING") as logs: + self.assertFalse(files._unlink("D:/thumbs/.pm-move-1.tmp")) + self.assertIn(".pm-move-1.tmp", logs.output[0]) + with patch.object(files.os, "unlink", side_effect=FileNotFoundError()): + self.assertTrue(files._unlink("gone")) + + def test_a_held_open_file_is_retried_then_left(self): + waits = [] + error = OSError("in use") + error.winerror = 32 + with patch.object(files.os, "unlink", side_effect=error): + self.assertFalse(files.remove_file("x", sleep=waits.append)) + self.assertEqual(waits, [0.5, 1.0, 2.0]) + + +if __name__ == "__main__": + unittest.main() + + +class TestReviewFindings(MoveCase): + async def test_moving_back_never_removes_previews_a_waiting_cleanup_named(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + other = Path(self.tmpdir) / "thumbs2" + # The old copies in the first folder cannot go yet (held open). + with patch.object(files, "remove_file", lambda path, **_k: False): + await self.move({"mode": "folder", "path": str(other)}) + self.assertTrue(preview_store.setting().cleanup) + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(job.state, "finished", job.log) + self.assertEqual(len(self.key_files()), 2) # the previews moved back stay + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + def test_empty_folder_removal_never_walks_through_a_link(self): + outside = Path(self.tmpdir) / "outside" + (outside / "empty").mkdir(parents=True) + (outside / "Thumbs.db").write_bytes(b"x") + store = Path(self.tmpdir) / "store" + store.mkdir() + try: + (store / "link").symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + files.remove_empty_folders(store) + self.assertTrue((outside / "empty").is_dir()) + self.assertTrue((outside / "Thumbs.db").exists()) + + @unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks") + async def test_moving_back_never_copies_through_an_existing_linked_folder(self): + # Codex P1 on #592: an existing linked folder in the destination was + # followed, and a file outside the source was replaced. + await self.move({"mode": "folder", "path": str(self.folder)}) + outside = Path(self.tmpdir) / "outside-p1" + outside.mkdir() + (outside / "b_thumb.webp").write_bytes(b"not ours") + if (self.old_store / "sub").exists(): + (self.old_store / "sub").rmdir() + try: + (self.old_store / "sub").symlink_to(outside, target_is_directory=True) + except OSError: + self.skipTest("symlinks not permitted") + _plan, job = await self.move({"mode": "source", "path": None}) + self.assertEqual((outside / "b_thumb.webp").read_bytes(), b"not ours") + self.assertEqual(job.counters["errors"], 1, job.log) + self.assertTrue((self.old_store / "a_thumb.webp").is_file()) + + @unittest.skipUnless(os.name == "nt", "junctions are Windows only") + def test_empty_folder_removal_never_walks_through_a_junction(self): + import _winapi + + outside = Path(self.tmpdir) / "outside-j" + (outside / "empty").mkdir(parents=True) + (outside / "Thumbs.db").write_bytes(b"x") + store = Path(self.tmpdir) / "store-j" + store.mkdir() + _winapi.CreateJunction(str(outside), str(store / "junction")) + files.remove_empty_folders(store) + self.assertTrue((outside / "empty").is_dir()) + self.assertTrue((outside / "Thumbs.db").exists()) + + async def test_an_unexpected_error_while_copying_changes_nothing(self): + def broken(*_args, **_kwargs): + raise PermissionError("denied") + + with patch.object(files, "_make_folders", broken): + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(job.state, "failed") + self.assertFalse(self.journal.exists()) + self.assertFalse(self.folder.exists()) + self.assertEqual(preview_store.current().mode, "source") + + async def test_a_journal_that_cannot_be_removed_never_undoes_the_switch(self): + from py.api import thumbnail_move + + real = thumbnail_move._remove + + def stuck(path): + if str(path) == str(self.journal): + raise PermissionError("in use") + return real(path) + + with patch.object(thumbnail_move, "_remove", stuck): + _plan, job = await self.move({"mode": "folder", "path": str(self.folder)}) + self.assertEqual(job.state, "finished", job.log) + self.assertEqual(len(self.key_files()), 2) + self.assertEqual(preview_store.current().mode, "folder") + + async def test_a_hostile_key_is_never_pruned(self): + await self.move({"mode": "folder", "path": str(self.folder)}) + victim = Path(self.tmpdir) / "victim" + make_png(victim / "z_thumb.webp", size=(4, 4)) + marker = preview_location.read_marker(str(self.folder)) + marker["keys"]["../victim"] = {"identity": "/gone", "install": marker_install()} + (self.folder / preview_location.MARKER_NAME).write_text( + json.dumps(marker), encoding="utf-8" + ) + preview_store.refresh() + resp = await self.client.post(START, json={"mode": "prune"}) + job = self.api._jobs.get((await resp.json())["job"]["id"]) + await job.task + self.assertTrue((victim / "z_thumb.webp").exists()) + + def test_a_hand_edited_source_store_must_be_a_thumbnails_folder(self): + elsewhere = Path(self.tmpdir) / "photos" + make_png(elsewhere / "keep_thumb.webp", size=(4, 4)) + self.assertTrue(self.api._remove_store(elsewhere, None, None)) + self.assertTrue((elsewhere / "keep_thumb.webp").exists()) + + +def marker_install(): + return preview_store.setting().install diff --git a/tests/trash_route_audit.py b/tests/trash_route_audit.py index 7474fb79..ab5a5186 100644 --- a/tests/trash_route_audit.py +++ b/tests/trash_route_audit.py @@ -15,6 +15,8 @@ KNOWN_FILE_READERS = frozenset( "images.py:os.open(", "__init__.py:Image.open(", "logging_routes.py:os.open(", + # The thumbnail folder's marker lock file (#320), never a media file. + "preview_location.py:os.open(", "thumbnail_auto.py:FileResponse(", "thumbnail_encode.py:Image.open(", } @@ -77,6 +79,7 @@ ROUTE_AUDIT = { "POST /prompt_manager/jobs/autotag", "POST /prompt_manager/jobs/thumbnails", "GET /prompt_manager/images/thumbnails/plan", + "POST /prompt_manager/images/thumbnails/plan", "GET /prompt_manager/images/thumbnails/status", "POST /prompt_manager/images/generate-thumbnails", "POST /prompt_manager/images/generate-thumbnails/progress", @@ -172,6 +175,8 @@ ROUTE_AUDIT = { "POST /prompt_manager/cleanup", "POST /prompt_manager/diagnostics/test-link", "POST /prompt_manager/images/clear-thumbnails", + "POST /prompt_manager/images/thumbnails/location/adopt", + "POST /prompt_manager/images/thumbnails/location/recreate", "POST /prompt_manager/jobs/model-download", "POST /prompt_manager/jobs/{job_id}/cancel", "POST /prompt_manager/library", diff --git a/tools/vendor/frontend-coverage-baseline.json b/tools/vendor/frontend-coverage-baseline.json index dae0d0d9..2e25f8cb 100644 --- a/tools/vendor/frontend-coverage-baseline.json +++ b/tools/vendor/frontend-coverage-baseline.json @@ -653,6 +653,12 @@ "functions": 95, "statements": 98 }, + "workspace/system-thumbnail-location.mjs": { + "lines": 80, + "branches": 65, + "functions": 75, + "statements": 80 + }, "workspace/system-thumbnails.mjs": { "lines": 98, "branches": 95, diff --git a/web/js/api-client.js b/web/js/api-client.js index 009d7cb1..bc3e9523 100644 --- a/web/js/api-client.js +++ b/web/js/api-client.js @@ -62,6 +62,13 @@ (typeof job.title === "string" && job.title.length > 0 && job.title.length <= 200))) { this.job = { id: job.id, title: job.title }; } + // A move refused for space (#320) carries its review, so the page + // can show what would have moved and why it cannot start. + this.plan = null; + if (this.code === "insufficient-space" && current && current.plan && + typeof current.plan === "object" && !Array.isArray(current.plan)) { + this.plan = current.plan; + } this.source = null; if (this.code === "invalid-source" && current && typeof current.source === "string" && current.source.length > 0 && current.source.length <= 4096) { diff --git a/web/js/workspace/activity-job-view.mjs b/web/js/workspace/activity-job-view.mjs index cd2edf01..83cc3b37 100644 --- a/web/js/workspace/activity-job-view.mjs +++ b/web/js/workspace/activity-job-view.mjs @@ -190,7 +190,10 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { bar.max = 100; const stats = element('dl', undefined, 'stat-row'); const tileValues = new Map(); - progressPart.node.append(bar, stats); + // #320 D198: a kind with phases (Move thumbnails) says which one runs and what it means, under the bar. + const phase = element('p', '', 'small'); + phase.id = 'job-phase'; + progressPart.node.append(bar, phase, stats); const resultsPart = section('Results', 'job-results-title'); const search = element('label', undefined, 'search-field'); @@ -248,6 +251,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { function layout(job) { view = viewOf(job); if (!view.tiles.length) stats.remove(); + if (typeof view.phase !== 'function') phase.remove(); for (const [group, label, tone] of view.tiles) { const tile = element('div', undefined, tone ? `stat ${tone}` : 'stat'); const value = element('dd', '0'); @@ -368,6 +372,7 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { cancel.remove(); } status.textContent = progressLine(job); + if (typeof view?.phase === 'function') phase.textContent = view.phase(job); // D111: no value while the total is unknown; a job that ended with nothing counted shows how it ended. showProgress(bar, job.total > 0 || job.state !== 'finished' ? job : { ...job, progress: 100 }); bar.setAttribute('aria-label', scan ? 'Scan progress' : `${job.title} progress`); diff --git a/web/js/workspace/activity-kinds.mjs b/web/js/workspace/activity-kinds.mjs index 18690c73..8cf81377 100644 --- a/web/js/workspace/activity-kinds.mjs +++ b/web/js/workspace/activity-kinds.mjs @@ -51,6 +51,26 @@ export const removedOf = job => counted(job, 'removed', 'removed'); /** "12 imported" (System S10: the LoRA reference import). */ export const importedOf = job => counted(job, 'imported', 'imported'); +/** #320 D198: the step a Move thumbnails job is in, and what it means meanwhile (under the progress bar). */ +export function movePhase(job) { + const from = job.scope?.from === 'folder' ? 'the old folder' : 'inside each source'; + if (job.state === 'running') { + if (job.scope?.phase === 'remove') return 'Removing the old copies a move left. Thumbnails are read from the new location.'; + if (job.cancellable !== false) { + return `Step 1 of 3: copying. Thumbnails are still read from ${from} until the switch. New thumbnails are not ` + + 'written until the move ends; originals are shown instead.'; + } + return 'Step 2 and 3 of 3: switching, then removing the old copies. This can no longer be cancelled.'; + } + if (job.state === 'finished') { + const left = job.counters?.left ?? 0; + return left ? `Thumbnails are now read from the new location. ${number(left)} ${noun(left, 'file')} that ` + + `${left === 1 ? 'is' : 'are'} not ${left === 1 ? 'a thumbnail was' : 'thumbnails were'} left in place.` + : 'Thumbnails are now read from the new location.'; + } + return 'Nothing changed: thumbnails are read from where they were.'; +} + /** "2 output folders": what a scan or a thumbnail job walks. */ function foldersOf(job) { const folders = job.scope?.output_folders; @@ -159,6 +179,32 @@ export const KINDS = Object.freeze({ outcomes: tiles([['generated', 'Generated'], ['errors', 'Errors'], ['skipped', 'Skipped']]), openable: groups('generated', 'skipped'), }), + // #320 D198: Move thumbnails, a `thumbnails` job with `scope.mode` "move": Moved (green), Already there, Left in + // place and Errors (red); Cancel only until the switch (the server says so through `cancellable`). + 'thumbnails:move': Object.freeze({ + ...GENERIC, + pillVerb: 'Moving thumbnails', + finishToast: job => (job.status_line && job.state !== 'running' ? job.status_line : titleFinished(job)), + line: job => (job.state === 'running' ? '' : job.status_line || ''), + kept: job => counted(job, 'moved', 'moved'), + tiles: tiles([['moved', 'Moved', 'added'], ['skipped', 'Already there', ''], ['left', 'Left in place', ''], + ['errors', 'Errors', 'bad']]), + outcomes: tiles([['moved', 'Moved'], ['errors', 'Errors'], ['skipped', 'Already there'], ['left', 'Left in place']]), + log: 'Move log', + phase: movePhase, + cancelLabel: 'Cancel move', + cancel: 'Cancel move', + }), + // #320 D200: Remove thumbnails of removed sources (`scope.mode` "prune"): Removed, Left in place and Errors. + 'thumbnails:prune': Object.freeze({ + ...GENERIC, + pillVerb: 'Removing thumbnails', + finishToast: job => (job.status_line && job.state !== 'running' ? job.status_line : titleFinished(job)), + line: job => (job.state === 'running' ? '' : job.status_line || ''), + kept: removedOf, + tiles: tiles([['removed', 'Removed', 'added'], ['left', 'Left in place', ''], ['errors', 'Errors', 'bad']]), + outcomes: tiles([['removed', 'Removed'], ['errors', 'Errors'], ['left', 'Left in place']]), + }), // System S7: a reviewed maintenance plan's apply, titled after its operation; it runs without a known total. maintenance: Object.freeze({ ...GENERIC, @@ -243,7 +289,9 @@ export const held = job => job.cancellable === false && Object.keys(countsOf(job */ export function viewOf(job) { if (held(job)) return GENERIC; - const view = kindOf(job.kind); + // #320 D198: a thumbnails job chooses its tiles by its scope's mode (move, prune). + const mode = job.kind === 'thumbnails' ? job.scope?.mode : null; + const view = (mode === 'move' || mode === 'prune') ? KINDS[`thumbnails:${mode}`] : kindOf(job.kind); const counts = countsOf(job); const has = ([group]) => Object.hasOwn(counts, group); if (view.tiles.every(has) && view.outcomes.every(has)) return view; diff --git a/web/js/workspace/library-location.mjs b/web/js/workspace/library-location.mjs index 447d5aeb..f3ddde7c 100644 --- a/web/js/workspace/library-location.mjs +++ b/web/js/workspace/library-location.mjs @@ -97,6 +97,9 @@ export function reviewGrid(location) { ? 'The database backups (config.json stays where PROMPT_MANAGER_CONFIG_PATH names it)' : 'config.json and the database backups'], ['Old files', 'Renamed *.moved-to-…, never deleted'], + // #320 D201: previews kept in one folder do not move with the library (D8A). + ...(typeof location.thumbnails_folder === 'string' && location.thumbnails_folder + ? [['Thumbnails', `Stay in ${location.thumbnails_folder}. Move them in System › Sources & scans.`]] : []), ]; } diff --git a/web/js/workspace/system-overview.mjs b/web/js/workspace/system-overview.mjs index 7356213d..d5a7fd8d 100644 --- a/web/js/workspace/system-overview.mjs +++ b/web/js/workspace/system-overview.mjs @@ -28,6 +28,7 @@ import { showWorkspaceToast } from './toast.mjs'; import { noun } from './plural.mjs'; import { operationRow } from './system-controls.mjs'; import { openThumbnailReview } from './thumbnail-review.mjs'; +import { noticeSlot } from './system-thumbnail-location.mjs'; import { isLocation, locationRow, onMoveEnded } from './library-location.mjs'; import { STATUS_LOADING, STATUS_URL, backupStatusText } from './system-backups.mjs'; @@ -406,7 +407,7 @@ function overviewNodes(overview, { api, activity, notify, scope, navigate, openS * `counts` is the shell's sidebar counts (`{get(id), subscribe(listener), freshen(ids)}`, index.mjs): the Images metric * shows its `images` count and follows it, and a visit re-reads it as the workspace Overview does (#555). */ -export function mountOverview(container, { api, activity, scope, counts = null, navigate = navigateTo, +export function mountOverview(container, { api, activity, scope, drafts, counts = null, navigate = navigateTo, notify = showWorkspaceToast, latestOnly = window.ApiClient.latestOnly, STALE = window.ApiClient.STALE }) { const overviewRead = latestOnly({ lifetime: scope.signal }); const statisticsRead = latestOnly({ lifetime: scope.signal }); @@ -466,7 +467,8 @@ export function mountOverview(container, { api, activity, scope, counts = null, if (overview === STALE || scope.signal.aborted) return; const { nodes, thumbnails, backup, metrics } = overviewNodes(overview, { api, activity, notify, scope, navigate, openStatistics, images: imagesCount(), - openThumbnails: trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate }) }); + openThumbnails: trigger => openThumbnailReview({ api, store: activity, scope, trigger, navigate, + location: drafts.get('host', 'thumbnail_location') }) }); imagesValue = metrics.children[2].querySelector('strong'); thumbnailRow = thumbnails; backupRow = backup; @@ -522,5 +524,11 @@ export function mountOverview(container, { api, activity, scope, counts = null, load(); loadThumbnails(); loadBackup(); - return first; + // #320 D199: the unavailable thumbnail folder's notice, first on the page while it lasts. + const unavailable = noticeSlot({ api, scope, notify, location: () => drafts.get('host', 'thumbnail_location'), + onChange: () => navigate('#/system/sources'), onDone: () => drafts.reloadHost() }); + scope.cleanup(drafts.subscribe(() => unavailable.paint())); + // The anchor goes in first, so the notice (inserted after it) leads the section once the anchor is mounted. + unavailable.paint(); + return [unavailable.anchor, ...first]; } diff --git a/web/js/workspace/system-sources.mjs b/web/js/workspace/system-sources.mjs index cd71d330..13d966f4 100644 --- a/web/js/workspace/system-sources.mjs +++ b/web/js/workspace/system-sources.mjs @@ -35,6 +35,7 @@ import { createButton } from '../shared/toolkit/core.js'; import { closeWorkspaceDialog, field, openContentDialog } from './workspace-dialogs.mjs'; import { openMetadataScan } from './activity-scan.mjs'; import { openThumbnailReview } from './thumbnail-review.mjs'; +import { locationBlock, noticeSlot } from './system-thumbnail-location.mjs'; import { settingRow, operationRow } from './system-controls.mjs'; import { HOST_UNAVAILABLE } from './system-appearance.mjs'; import { showWorkspaceToast } from './toast.mjs'; @@ -234,6 +235,19 @@ export function mountSources(container, { api, activity, scope, drafts, notify = nodes.push(notice); } + // #320 D199: the unavailable thumbnail folder's notice, first on the page while it lasts. + const thumbnailLocation = () => drafts.get('host', 'thumbnail_location'); + const location = locationBlock({ api, store: activity, scope, location: thumbnailLocation, notify }); + const unavailable = noticeSlot({ api, scope, location: thumbnailLocation, notify, onChange: () => location.focus(), + onDone: () => drafts.reloadHost?.() }); + const paintLocation = () => { + location.paint(); + unavailable.paint(); + }; + scope.cleanup(drafts.subscribe(paintLocation)); + nodes.push(unavailable.anchor); + paintLocation(); + const surface = el('section', undefined, 'surface'); const heading = el('div', undefined, 'panel-heading'); const addButton = createButton('+ Add source'); @@ -266,7 +280,10 @@ export function mountSources(container, { api, activity, scope, drafts, notify = operationRow({ scope, title: 'Enrich existing metadata', description: 'Resolve both prompts, deduplicate recipes, preserve uncertain links.', label: 'Review scan', onClick: scanAll }), operationRow({ scope, title: 'Thumbnail generation', description: 'Generate missing previews; rebuilding all is a separate explicit option.', - label: 'Review options', onClick: trigger => openThumbnailReview({ api, store: activity, scope, trigger }) }), + label: 'Review options', onClick: trigger => openThumbnailReview({ api, store: activity, scope, trigger, + location: thumbnailLocation() }) }), + // #320 D196: where previews are kept; never a draft (Move thumbnails is its only writer). + location.node, // D98: the legacy gallery's Refresh, which rereads every source's file list. operationRow({ scope, title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.', label: 'Refresh', onClick: refresh })); diff --git a/web/js/workspace/system-thumbnail-location.mjs b/web/js/workspace/system-thumbnail-location.mjs new file mode 100644 index 00000000..0d496085 --- /dev/null +++ b/web/js/workspace/system-thumbnail-location.mjs @@ -0,0 +1,412 @@ +/** + * Thumbnail location (#320, beyond the concept: D196–D199, approved static mock `320-mock`): where previews are kept. + * + * - D196 `locationBlock`: a `.setting-block` in System › Sources & scans after "Thumbnail generation": the radios + * "Inside each source" / "One folder", a "Folder path" field (enabled with One folder, prefilled with the folder in + * use or the server's `default_path`), the status line and "Review move…" (enabled when the choice differs from the + * location in use). Not a host draft: the Save row never saves it; the Move thumbnails job is the only writer (D4B). + * A refusal of the review shows in place under the field (`aria-invalid`), in the server's words. + * - D197 `openMoveReview`: the shared review (Move library's shape) over the server's plan + * (`POST /images/thumbnails/plan`): From, To, Thumbnails, Free space, Too long for Windows, Removed after the move, + * Left in place, Links skipped; a folder that is not the default asks for the confirmation checkbox first. "Move + * thumbnails" starts the job (`POST /jobs/thumbnails {mode: "move", token, confirmed}`) and opens it in Activity. + * - D199 `locationNotice`: while the thumbnail folder is unavailable, a `.notice` on Overview and Sources & scans + * says why, with "Recreate folder" (a missing folder, after a confirmation) or "Use this folder" (another folder's + * marker), and "Change location". + * + * The location comes from GET /settings' `thumbnail_location` (`{mode, path, default_path, available, problem, + * cleanup_pending}`), read with the host settings. + */ +import { createButton } from '../shared/toolkit/core.js'; +import { closeWorkspaceDialog, openContentDialog, openReviewDialog } from './workspace-dialogs.mjs'; +import { busyToast } from './activity-scan.mjs'; +import { showWorkspaceToast } from './toast.mjs'; +import { navigateTo } from './state.mjs'; +import { noun } from './plural.mjs'; + +export const PLAN_URL = '/prompt_manager/images/thumbnails/plan'; +export const RECREATE_URL = '/prompt_manager/images/thumbnails/location/recreate'; +export const ADOPT_URL = '/prompt_manager/images/thumbnails/location/adopt'; +export const LOCATION_HINT = 'Where previews are kept. Originals are never moved. Avoid cloud-synced folders.'; +export const SENTENCE = 'Every thumbnail is copied and checked before Prompt Manager switches. Scans, deletes and ' + + 'tagging wait until it finishes; closing this page does not stop it. Originals are never touched.'; +export const REVIEW_FAILED = 'The move could not be reviewed. Try again.'; +export const START_FAILED = 'Move thumbnails could not start. Try again.'; +export const NOTICE_TITLE = 'The thumbnail folder is unavailable'; +const BACK = 'Originals are shown and no thumbnails are written until it is back.'; +const CHECK = 'If its drive is unplugged, connect it; Prompt Manager checks again every few seconds.'; + +const number = value => Number(value || 0).toLocaleString('en-US'); +const isFolder = location => location?.mode === 'folder'; + +/** "1.2 GB" (binary units, one decimal, as the server writes them). */ +export function sizeText(bytes) { + let size = Number(bytes) || 0; + for (const unit of ['B', 'KB', 'MB', 'GB']) { + if (size < 1024 || unit === 'GB') return unit === 'B' ? `${size} B` : `${size.toFixed(1)} ${unit}`; + size /= 1024; + } + return `${size.toFixed(1)} GB`; +} + +export const SUGGESTED = 'The suggested folder is next to your library; Review move… shows what happens before anything ' + + 'is copied.'; + +/** The status line under the row; `choice` (`{mode, path}`) adds the suggestion while the default folder is chosen. */ +export function statusText(location, choice = null) { + if (!location) return ''; + if (!isFolder(location) && choice?.mode === 'folder' && location.default_path + && choice.path?.trim() === location.default_path) { + return `Now: inside each source. ${SUGGESTED}`; + } + let text = isFolder(location) ? `Now: one folder, ${location.path}` : 'Now: inside each source'; + if (isFolder(location) && location.available === false) return `${text} · unavailable. Originals are shown.`; + text += '.'; + if (isFolder(location) && location.path && location.path === location.default_path) { + text += ' Using the default folder next to your library.'; + } + if (location.cleanup_pending > 0) { + text += ` Old thumbnails in ${number(location.cleanup_pending)} ${noun(location.cleanup_pending, 'place')} wait to ` + + 'be removed until it is connected.'; + } + return text; +} + +/** Whether the row's choice (`{mode, path}`) differs from the location in use (Review move… enabled). */ +export function choiceChanged(location, choice) { + if (!location) return false; + if (choice.mode !== 'folder') return isFolder(location) || Boolean(location.problem); + const path = (choice.path || '').trim(); + return Boolean(path) && (!isFolder(location) || path !== location.path); +} + +/** The D197 review rows (`[term, value]`, a value may be `[lead, items]`) for the server's plan. */ +export function reviewRows(plan) { + const totals = plan.totals ?? {}; + const sources = totals.sources ?? plan.sources?.length ?? 0; + const inside = `Inside each source (${number(sources)} ${noun(sources, 'source')})`; + const from = plan.from?.mode === 'folder' ? plan.from.path : inside; + let to = inside; + if (plan.to?.mode === 'folder') { + const state = plan.to.default ? (plan.to.will_create ? 'new, next to your library' : 'next to your library') + : (plan.to.will_create ? 'new folder' : 'empty folder'); + to = `${plan.to.path} (${state})`; + } + const rows = [['From', from], ['To', to], + ['Thumbnails', `${number(totals.files)} in ${number(sources)} ${noun(sources, 'source')} · ${sizeText(totals.bytes)}`]]; + if (Number.isFinite(plan.free_bytes)) rows.push(['Free space', `${sizeText(plan.free_bytes)} free`]); + if (totals.too_long > 0) { + const more = totals.too_long - (plan.too_long_listed?.length ?? 0); + rows.push(['Too long for Windows', [`${number(totals.too_long)} ${noun(totals.too_long, 'thumbnail')} cannot be ` + + 'stored there. They show the original instead; choose a shorter folder to keep them.', + [...(plan.too_long_listed ?? []), ...(more > 0 ? [`and ${number(more)} more`] : [])]]]); + } + const removed = []; + if (totals.orphans > 0) removed.push(`${number(totals.orphans)} ${noun(totals.orphans, 'thumbnail')} whose original is gone`); + if (totals.temps > 0) removed.push(`${number(totals.temps)} unfinished temporary ${noun(totals.temps, 'file')}`); + for (const source of plan.unlisted ?? []) { + removed.push(`1 source no longer listed: ${number(source.files)} ${noun(source.files, 'thumbnail')} are removed; ` + + 'they are made again if it returns'); + } + const oldCopies = plan.from?.mode === 'folder' + ? 'The thumbnail folders of these sources inside it. The folder itself is removed once empty if Prompt Manager created it' + : 'The old copies in each source’s thumbnails folder'; + rows.push(['Removed after the move', [removed.length ? `${oldCopies}, and:` : `${oldCopies}.`, removed]]); + if (totals.left_in_place > 0) { + rows.push(['Left in place', [`${number(totals.left_in_place)} ${noun(totals.left_in_place, 'file')} that ` + + `${totals.left_in_place === 1 ? 'is' : 'are'} not ${totals.left_in_place === 1 ? 'a thumbnail' : 'thumbnails'}:`, + plan.left_listed ?? []]]); + } + if (totals.links > 0) { + rows.push(['Links skipped', `${number(totals.links)} ${noun(totals.links, 'link')} inside a thumbnails folder ` + + `${totals.links === 1 ? 'is' : 'are'} not followed`]); + } + return rows; +} + +/** The notice text for an unavailable `location` (D199). */ +export function noticeText(location) { + const where = location?.path || 'The thumbnail folder'; + switch (location?.problem) { + case 'missing': return `${where} is missing. ${BACK} ${CHECK}`; + case 'unreachable': return `${where} can’t be reached. ${BACK} ${CHECK}`; + case 'marker-mismatch': return `${where} holds another thumbnail folder. ${BACK} Use it if it is yours, or change ` + + 'the location.'; + case 'invalid': return `The thumbnail location in config.json can’t be used. ${BACK} Change the location.`; + default: return `${where} is being checked. ${BACK}`; + } +} + +function el(tag, text, className) { + const node = document.createElement(tag); + if (text !== undefined) node.textContent = text; + if (className) node.className = className; + return node; +} + +function radio(value, label, hint) { + const node = el('label'); + const input = el('input'); + input.type = 'radio'; + input.name = 'thumb-location'; + input.value = value; + input.id = `loc-${value}`; + node.append(input, ` ${label} `, el('span', hint, 'small')); + return { node, input }; +} + +/** Start the move with the plan's `token`; opens it in Activity, or says why not. */ +export async function startMove({ store, scope, token, confirmed, notify = showWorkspaceToast, navigate = navigateTo, + onStale = null }) { + try { + const result = await store.start('thumbnails', { mode: 'move', token, confirmed }); + if (scope.signal.aborted) return null; + if (result.busy !== undefined) { + notify(busyToast(result.busy)); + return null; + } + navigate(`#/activity/${result.job.id}`); + return result.job; + } catch (error) { + if (scope.signal.aborted) return null; + // The sources or the location changed since the review: review again with what is there now. + if (error?.code === 'plan-stale' && onStale) { + onStale(); + return null; + } + notify(typeof error?.code === 'string' && error.message ? error.message : START_FAILED); + return null; + } +} + +/** Open the D197 review over `plan`; a plan the server refused for space keeps Move thumbnails unavailable. */ +export function openMoveReview({ scope, store, plan, refusal = null, trigger = null, notify, navigate, onStale = null }) { + const options = []; + let box = null; + if (plan.confirm_required && !refusal) { + const label = el('label', undefined, 'review-confirm'); + box = el('input'); + box.type = 'checkbox'; + box.id = 'review-confirmed'; + label.append(box, el('span', `Prompt Manager may create its own folders and a .promptmanager-thumbnails.json file in ${plan.to.path}.`)); + options.push(label); + } + if (refusal) options.push(el('p', refusal, 'small review-refusal')); + const review = openReviewDialog({ scope, trigger, title: 'Move thumbnails', grid: reviewRows(plan), options, + sentence: refusal ? null : SENTENCE, cancelLabel: 'Cancel', primary: { label: 'Move thumbnails' }, + onStart: () => { startMove({ store, scope, token: plan.token, confirmed: Boolean(box?.checked), notify, navigate, + onStale }); } }); + const locked = () => Boolean(refusal) || (box !== null && !box.checked); + review.setPrimary({ label: 'Move thumbnails', disabled: locked() }); + if (box) scope.listen(box, 'change', () => review.setPrimary({ label: 'Move thumbnails', disabled: locked() })); + return review; +} + +/** + * The D196 row; `location()` gives the server's `thumbnail_location` (undefined while read). Returns `{node, paint, + * focus}`; `paint()` follows a new location. + */ +export function locationBlock({ api, store, scope, location, notify = showWorkspaceToast, navigate = navigateTo }) { + const block = el('div', undefined, 'setting-block'); + block.id = 'location-block'; + const text = el('div', undefined, 'setting-block-text'); + const label = el('span', 'Thumbnail location', 'setting-block-label'); + label.id = 'location-label'; + text.append(label, el('p', LOCATION_HINT)); + const choice = el('div', undefined, 'location-choice'); + choice.setAttribute('role', 'radiogroup'); + choice.setAttribute('aria-labelledby', 'location-label'); + const source = radio('source', 'Inside each source', '/thumbnails'); + const folder = radio('folder', 'One folder', 'one subfolder per source'); + choice.append(source.node, folder.node); + const pathField = el('div', undefined, 'location-path field'); + const pathLabel = el('label', 'Folder path'); + pathLabel.htmlFor = 'location-path'; + const input = el('input'); + input.id = 'location-path'; + input.autocomplete = 'off'; + input.spellcheck = false; + const problem = el('p', '', 'small location-problem'); + problem.id = 'location-problem'; + problem.setAttribute('role', 'alert'); + pathField.append(pathLabel, input, problem); + const status = el('p', '', 'small location-status'); + status.id = 'location-status'; + status.setAttribute('role', 'status'); + const actions = el('div', undefined, 'row-actions'); + const review = createButton('Review move…'); + review.id = 'review-move'; + actions.append(review); + block.append(text, choice, pathField, status, actions); + + let shown; + const setProblem = message => { + problem.textContent = message || ''; + pathField.setAttribute('aria-invalid', String(Boolean(message))); + // The field itself carries the state, so a screen reader announces it with the reason. + if (message) { + input.setAttribute('aria-invalid', 'true'); + input.setAttribute('aria-describedby', 'location-problem'); + } else { + input.removeAttribute('aria-invalid'); + input.removeAttribute('aria-describedby'); + } + }; + const current = () => ({ mode: folder.input.checked ? 'folder' : 'source', path: input.value }); + const repaintButton = () => { + const now = location(); + input.disabled = !folder.input.checked || !now; + source.input.disabled = folder.input.disabled = !now; + const changed = choiceChanged(now, current()); + status.textContent = now ? statusText(now, current()) : ''; + review.setAttribute('aria-disabled', String(!changed)); + review.classList.toggle('ca-primary', changed); + }; + const paint = () => { + const now = location(); + if (now && now !== shown) { + shown = now; + (isFolder(now) ? folder : source).input.checked = true; + input.value = (isFolder(now) ? now.path : now.default_path) || ''; + setProblem(null); + } + repaintButton(); + }; + for (const control of [source.input, folder.input]) { + scope.listen(control, 'change', () => { + if (folder.input.checked && !input.value) input.value = location()?.default_path || ''; + setProblem(null); + repaintButton(); + }); + } + scope.listen(input, 'input', () => { setProblem(null); repaintButton(); }); + scope.listen(review, 'click', async () => { + if (review.getAttribute('aria-disabled') === 'true') return; + const chosen = current(); + const body = { mode: 'move', location: chosen.mode === 'folder' ? { mode: 'folder', path: chosen.path.trim() } + : { mode: 'source' } }; + review.setAttribute('aria-disabled', 'true'); + try { + const plan = await api.post(PLAN_URL, body, { signal: scope.signal }); + if (!scope.signal.aborted) { + openMoveReview({ scope, store, plan, trigger: review, notify, navigate, onStale: () => review.click() }); + } + } catch (error) { + if (scope.signal.aborted) return; + if (error?.status === 507 && error?.plan) { + openMoveReview({ scope, store, plan: error.plan, refusal: error.plan.insufficient?.text || error.message, + trigger: review }); + } else if (error?.status === 409 && error?.code === 'busy') { + notify(busyToast(error.running ?? null)); + } else if (typeof error?.code === 'string' && error.code.startsWith('location-') && error.message) { + setProblem(error.message); + input.focus(); + } else { + notify(typeof error?.message === 'string' && error?.code ? error.message : REVIEW_FAILED); + } + } finally { + repaintButton(); + } + }); + paint(); + return { + node: block, + paint, + focus() { + folder.input.checked = true; + repaintButton(); + input.focus(); + block.scrollIntoView?.({ block: 'center' }); + }, + }; +} + +/** Open the Recreate confirmation for `location`; resolves once it closed. */ +export function openRecreate({ api, scope, location, trigger = null, onDone, notify = showWorkspaceToast }) { + const grid = el('dl', undefined, 'meta-grid'); + for (const [term, value] of [['Folder', location.path], ['Made', 'An empty folder with a new .promptmanager-thumbnails.json'], + ['Thumbnails', 'Made again as you browse, or with Thumbnail maintenance']]) { + const entry = el('div'); + entry.append(el('dt', term), el('dd', value)); + grid.append(entry); + } + const why = el('p', '', 'small location-problem'); + const cancel = createButton('Cancel'); + const go = createButton('Recreate folder', 'ca-primary'); + const actions = el('div', undefined, 'dialog-actions'); + actions.append(cancel, go); + scope.listen(cancel, 'click', () => closeWorkspaceDialog()); + scope.listen(go, 'click', async () => { + try { + const result = await api.post(RECREATE_URL, {}, { signal: scope.signal }); + if (scope.signal.aborted) return; + closeWorkspaceDialog(); + onDone?.(result.location); + } catch (error) { + if (scope.signal.aborted) return; + why.textContent = error?.message || 'The folder could not be recreated.'; + if (!error?.message) notify(why.textContent); + } + }); + openContentDialog({ scope, title: 'Recreate the thumbnail folder?', trigger, nodes: [grid, + el('p', 'If the folder is on a drive that is not connected, connect it instead: recreating would put thumbnails on ' + + 'another disk.', 'small'), why, actions] }); +} + +/** + * The D199 notice for `location` (null while it is available): Recreate folder for a missing folder, Use this folder + * for another folder's marker, and Change location (`onChange`). + */ +export function locationNotice({ api, scope, location, onChange, onDone, notify = showWorkspaceToast }) { + if (!isFolder(location) || location.available !== false) return null; + const notice = el('div', undefined, 'notice'); + notice.id = 'unavailable-notice'; + const text = el('div'); + text.append(el('h2', NOTICE_TITLE), el('p', noticeText(location))); + const actions = el('div', undefined, 'row-actions'); + if (location.problem === 'missing') { + const recreate = createButton('Recreate folder'); + scope.listen(recreate, 'click', () => openRecreate({ api, scope, location, trigger: recreate, onDone, notify })); + actions.append(recreate); + } else if (location.problem === 'marker-mismatch') { + const adopt = createButton('Use this folder'); + scope.listen(adopt, 'click', async () => { + try { + const result = await api.post(ADOPT_URL, {}, { signal: scope.signal }); + if (!scope.signal.aborted) onDone?.(result.location); + } catch (error) { + if (!scope.signal.aborted) notify(error?.message || 'The folder could not be used.'); + } + }); + actions.append(adopt); + } + const change = createButton('Change location', 'ca-primary'); + scope.listen(change, 'click', () => onChange?.()); + actions.append(change); + notice.append(text, actions); + return notice; +} + +/** + * Where a section shows the D199 notice: a comment `anchor` (no element, so the concept comparisons see nothing while the + * folder is available) after which `paint()` keeps the notice for `location()`. + */ +export function noticeSlot({ api, scope, location, onChange, onDone, notify = showWorkspaceToast }) { + const anchor = document.createComment('thumbnail-location-notice'); + let shownFor; + let notice = null; + const paint = () => { + const now = location(); + // Not mounted yet: the next paint (a microtask, or the next host read) places it. + if (!anchor.parentNode) { + queueMicrotask(() => { if (anchor.parentNode && !scope.signal.aborted) paint(); }); + return; + } + if (now === shownFor) return; + shownFor = now; + notice?.remove(); + notice = locationNotice({ api, scope, location: now, onChange, onDone, notify }); + if (notice) anchor.after(notice); + }; + return { anchor, paint }; +} diff --git a/web/js/workspace/system.mjs b/web/js/workspace/system.mjs index d769f809..1a5e8129 100644 --- a/web/js/workspace/system.mjs +++ b/web/js/workspace/system.mjs @@ -86,7 +86,9 @@ export function systemDrafts(api) { api.get(LORA_STATUS_URL, { cache: 'no-store' }).catch(() => null)]); // The store reads a missing value as none and a missing revision as unreadable (no host save). #317: which // thumbnail formats the server can write rides along as a read-only value (never drafted or sent). - return { values: { ...result.settings, thumbnail_encoders: result.thumbnail_encoders, ...(lora ? loraValues(lora) : {}) }, + // #320: where previews live rides along the same way (read-only; only Move thumbnails changes it). + return { values: { ...result.settings, thumbnail_encoders: result.thumbnail_encoders, + thumbnail_location: result.thumbnail_location, ...(lora ? loraValues(lora) : {}) }, revision: result.revision }; }, async writeHost(changes, { expected_revision }) { diff --git a/web/js/workspace/thumbnail-review.mjs b/web/js/workspace/thumbnail-review.mjs index 177a261b..6dd3ea0b 100644 --- a/web/js/workspace/thumbnail-review.mjs +++ b/web/js/workspace/thumbnail-review.mjs @@ -23,6 +23,7 @@ import { busyToast } from './activity-scan.mjs'; import { showWorkspaceToast } from './toast.mjs'; import { navigateTo } from './state.mjs'; import { noun } from './plural.mjs'; +import { sizeText } from './system-thumbnail-location.mjs'; export const THUMBNAIL_PLAN_URL = '/prompt_manager/images/thumbnails/plan'; export const THUMBNAILS_KIND = 'thumbnails'; @@ -33,7 +34,9 @@ export const PLAN_FAILED = 'The previews could not be counted.'; export const START_FAILED = 'Thumbnail generation could not start. Try again.'; export const SETTINGS_UNKNOWN = 'Previews follow the thumbnail settings in System › Appearance.'; const FORMAT_LABELS = Object.freeze({ webp: 'WebP', jxl: 'JPEG XL', jpeg: 'JPEG' }); -const MODES = new Set(SCOPE_OPTIONS.map(([value]) => value)); +// #320 D200: offered only while previews are kept in one folder (31A). +export const PRUNE_OPTION = Object.freeze(['prune', 'Remove thumbnails of removed sources']); +const MODES = new Set([...SCOPE_OPTIONS.map(([value]) => value), PRUNE_OPTION[0]]); const count = value => value.toLocaleString('en-US'); const isCount = value => Number.isSafeInteger(value) && value >= 0; @@ -45,8 +48,22 @@ export function settingsText(plan) { return `Previews are written as ${label} · quality ${plan.quality} · ${count(plan.size)} px. Change this in System › Appearance.`; } +/** D200: what a prune removes, in `plan.folder` (the thumbnail folder). */ +export function pruneText(plan) { + return `Only Prompt Manager’s thumbnails in their folders inside ${plan?.folder || 'the thumbnail folder'} are ` + + 'removed. If a source comes back, its thumbnails are made again as you browse.'; +} + +/** D200: one `[key, "N thumbnails · size"]` per removed source of a prune plan. */ +export function pruneEntries(plan) { + const sources = Array.isArray(plan?.sources) ? plan.sources : []; + return sources.map(source => [source.key, `${count(source.files)} ${noun(source.files, 'thumbnail')} · ` + + `${sizeText(source.bytes)}`]); +} + /** D99: the plan line for the server's `{mode, targets, fresh, remaining}`; an answer without counts failed. */ export function planText(plan) { + if (plan?.mode === 'prune') return typeof plan.line === 'string' && plan.line ? plan.line : PLAN_FAILED; if (!plan || !MODES.has(plan.mode) || ![plan.targets, plan.fresh, plan.remaining].every(isCount)) { return PLAN_FAILED; } @@ -101,11 +118,18 @@ function select(options, value) { * Activity store; `trigger` gets focus back when the dialog closes. */ export function openThumbnailReview({ api, store, scope, trigger = null, mode: initial = 'missing', notify = showWorkspaceToast, - navigate = navigateTo }) { - const mode = select(SCOPE_OPTIONS, MODES.has(initial) ? initial : 'missing'); + navigate = navigateTo, location = null }) { + const folderMode = location?.mode === 'folder'; + const options = folderMode ? [...SCOPE_OPTIONS, PRUNE_OPTION] : SCOPE_OPTIONS; + const allowed = new Set(options.map(([value]) => value)); + const mode = select(options, allowed.has(initial) ? initial : 'missing'); const line = el('p', PLAN_LOADING, 'small'); line.setAttribute('role', 'status'); const settings = el('p', SETTINGS_UNKNOWN, 'small'); + // D200: what the prune removes, under the plan line. + const pruneDetail = el('p', '', 'small'); + const pruneGrid = el('dl', undefined, 'meta-grid'); + pruneDetail.hidden = pruneGrid.hidden = true; const cancel = createButton('Cancel'); const start = createButton('Start', 'ca-primary'); const actions = el('div', undefined, 'dialog-actions'); @@ -125,7 +149,20 @@ export function openThumbnailReview({ api, store, scope, trigger = null, mode: i // A later choice painted its own line meanwhile, or the dialog closed. if (!open || scope.signal.aborted || mode.value !== chosen) return; line.textContent = planText(plan); - settings.textContent = settingsText(plan); + const prune = chosen === 'prune'; + settings.hidden = prune; + const entries = pruneEntries(plan); + pruneDetail.hidden = pruneGrid.hidden = !prune || !entries.length; + if (prune) { + pruneDetail.textContent = pruneText(plan); + pruneGrid.replaceChildren(...entries.map(([key, text]) => { + const entry = el('div'); + entry.append(el('dt', key), el('dd', text)); + return entry; + })); + } else { + settings.textContent = settingsText(plan); + } } scope.listen(mode, 'change', paint); @@ -136,7 +173,7 @@ export function openThumbnailReview({ api, store, scope, trigger = null, mode: i startThumbnails({ store, scope, options, notify, navigate }); }); openContentDialog({ scope, title: 'Thumbnail maintenance', trigger, - nodes: [field('Scope', mode, 'thumb-mode'), line, settings, actions], + nodes: [field('Scope', mode, 'thumb-mode'), line, settings, ...(folderMode ? [pruneDetail, pruneGrid] : []), actions], onClose: () => { open = false; } }); paint(); } diff --git a/web/js/workspace/workspace-dialogs.mjs b/web/js/workspace/workspace-dialogs.mjs index c26d82a0..fb3496f1 100644 --- a/web/js/workspace/workspace-dialogs.mjs +++ b/web/js/workspace/workspace-dialogs.mjs @@ -460,9 +460,13 @@ export function backupNotice({ copy, onBackup }) { return notice; } +/** A grid value: text, or (#320) `[lead, items]`, a lead sentence and the list under it. */ +const gridValue = value => typeof value === 'string' || (Array.isArray(value) && value.length === 2 + && typeof value[0] === 'string' && Array.isArray(value[1]) && value[1].every(item => typeof item === 'string')); + function requireGrid(grid) { if (!Array.isArray(grid) || grid.length === 0 || grid.some(row => !Array.isArray(row) || row.length !== 2 || - typeof row[0] !== 'string' || !row[0] || typeof row[1] !== 'string')) { + typeof row[0] !== 'string' || !row[0] || !gridValue(row[1]))) { throw new TypeError('The review needs its [term, value] grid'); } } @@ -495,7 +499,12 @@ export function openReviewDialog({ scope, title, heading = null, grid, options = values.clear(); list.replaceChildren(...rows.map(([term, value]) => { const entry = el('div'); - const dd = el('dd', value); + const dd = el('dd', typeof value === 'string' ? value : value[0]); + if (typeof value !== 'string' && value[1].length) { + const items = el('ul', undefined, 'review-list'); + items.append(...value[1].map(item => el('li', item))); + dd.append(items); + } values.set(term, dd); entry.append(el('dt', term), dd); return entry; diff --git a/web/js/workspace/workspace.css b/web/js/workspace/workspace.css index 8e55b1ee..11053668 100644 --- a/web/js/workspace/workspace.css +++ b/web/js/workspace/workspace.css @@ -619,3 +619,31 @@ pre.prompt-text { font: inherit; font-size: 12px; line-height: 1.75; margin-bloc .review-panel { position: sticky; inset-block-start: 20px; } @media (max-width: 700px) { .review-panel { position: static; } } @media (min-width: 701px) and (max-height: 720px) { .review-panel { max-block-size: calc(100dvh - 40px); overflow: auto; } } + +/* #320 D196: Thumbnail location, a setting wider than a .setting-row column. */ +.setting-block { padding: 17px 0; border-top: 1px solid var(--line); display: grid; gap: 12px; } +.setting-block-text .setting-block-label { font-size: 12px; font-weight: 500; display: block; } +.setting-block-text p { font-size: 11px; color: var(--ca-muted); margin-top: 4px; max-width: 640px; } +.location-choice { display: flex; flex-wrap: wrap; gap: 8px 24px; font-size: 12px; } +.location-choice label { display: inline-flex; gap: 6px; align-items: baseline; } +.location-choice .small { color: var(--ca-muted); } +.location-path { max-width: 640px; margin: 0; } +.location-path input { width: 100%; min-width: 0; font-family: var(--ca-mono, ui-monospace, monospace); font-size: 12px; } +.location-path input:disabled { color: var(--ca-muted); } +.location-path[aria-invalid="true"] input { border-color: var(--red); } +.location-problem { color: var(--red); margin-top: 6px; } +.location-problem:empty { display: none; } +.location-status { margin: 0; overflow-wrap: anywhere; } +.setting-block .row-actions { justify-content: flex-start; } + +/* #320 D197: the move review's lists and confirmation. */ +.review-list { margin: 4px 0 0; padding-inline-start: 18px; font-size: 11px; color: var(--ca-muted); } +.review-list li { overflow-wrap: anywhere; } +.review-confirm { display: flex; gap: 8px; align-items: flex-start; font-size: 12px; margin: 12px 0 0; + padding: 10px 12px; border: 1px solid var(--line); border-radius: 6px; background: var(--ca-field); } +.review-confirm input { margin-top: 2px; flex-shrink: 0; } +.review-refusal { color: var(--red); } + +@media (max-width: 700px) { + .location-choice { flex-direction: column; } +}