15 Commits

Include renames

Author	SHA1	Message	Date
Vito	d868745194	fix(lora): cache LoraManager's layout and close the #355 follow-ups (#362) (#364) ... * feat(lora): cache LoraManager's layout behind file signatures (#362, phase 1) A new py/lora_layout_cache.py keeps one (install, layout) entry and reuses it while the configured path, the plugin settings file's and the settings file's signatures (st_mtime_ns, st_size), the install and every media root's real path are unchanged and the entry is younger than 30 s. A load whose watched file changed within 3 s (git's racy rule), could not be checked, or whose settings were unreadable serves its request only. clear() bumps a generation so a load that started earlier stores nothing. Hits take no lock. Not wired yet; phase 2 puts it behind the image hot paths. * fix(lora): treat an offline Windows share as unvalidated and make cache tests portable (#362) Review findings on phase 1: Windows raises FileNotFoundError with winerror 53/67 for an offline share, which is a temporary failure, not a missing file. Test times are whole milliseconds (NTFS keeps 100 ns steps) and a file system that rounds them skips; cleanup no longer ignores errors; tests added for ValueError, NotADirectoryError, a failed reload, resolve failures and the watched-file set. * perf(lora): serve the image hot paths from the layout cache (#362, phase 2) _lora_image_dirs takes (install, layout) from lora_utils.cached_lora_manager_layout instead of resolving the install and reading LoraManager's settings files on every Library, inspect, reveal and serve request. serve_image builds the allow-list in the executor instead of on the event loop. A finished import (System job or legacy scan, both through replace_lora_imports) clears the cache, so files linked under a folder LoraManager just created are served at once. The integration-off check and the configured-path rule still run before the cache. * fix(lora): fingerprint every scanned sidecar, kept or skipped (#362, phase 3) Discovery keeps every metadata file it saw (scanned_files, resolved and sorted; scanned is now its length). The fingerprint hashes each one's path, size, modification time and whether it was kept, so a skipped sidecar added, removed or edited between the review and Start refuses Start with plan-changed, as the review's counts promised. * fix(lora): warn about refused default LoRA folders and log what failed (#362, phase 4) ComfyUI's default LoRA folders already pass the folder guard; a refused one that exists is now listed in the layout's refused_defaults, and the layout cache warns once per newly refused folder. The cache writes one INFO line when a stored layout differs from the last one logged. BAD_SIDECAR_ROOT and the module docstring name a folder that contains the home folder. A relative preview_url writes a DEBUG line before the suffix search; a non-OSError failure while loading trigger words logs its traceback. Review follow-ups harden two serve tests. * fix(lora): skip a sidecar that is not UTF-8 instead of failing the load (#362) Review follow-up on phase 4: read_lora_metadata caught JSONDecodeError but not UnicodeDecodeError, so one non-UTF-8 sidecar failed the whole trigger-word load (now logged with a traceback every retry) and the import review. It is skipped like any unreadable sidecar. The relative preview_url DEBUG line uses lazy formatting. * docs(testing): the LoRA linked-only gate, layout cache and fingerprint scope (#362, phase 5) TESTING.md's LoRA reference import contract now states what #358 and #362 shipped: the linked-only gate and _media_roots, linked_only media, the cover link order, the guard on ComfyUI's default LoRA folders and refused_defaults, the folder-unreadable code, the trigger cache's catch and 60 s retry, the fingerprint over every scanned sidecar, and the layout cache's key, signatures, racy window, maximum age, root re-check and clears. The targeted command includes tests.test_lora_layout_cache and tests.test_api_images. * fix(lora): a cached layout needs the same home folder; give the rebuild poll its time (#362) Final review follow-ups: the folder guard compares against the home folder, so a cache hit now also needs Path.home() unchanged. TESTING.md says the configured-path rule runs on the cache's result and that a non-UTF-8 sidecar is skipped. CI: 'Rebuild all is explicit...' (webkit) polled the job for the default 5 s while it regenerated 24 previews; the failure screenshot shows 12 of 24 done at the deadline on the busy runner (0.17 s locally). The poll now allows 30 s for that real work.	2026-10-27 05:00:55 -07:00
Vito	80c6ec5340	fix(lora): find LoraManager's sidecars, previews and example images in every storage configuration (#355) (#358) ... * fix(lora): read LoraManager's settings and storage layout the way it does (#355) Add py/lora_manager_layout.py: the settings file in LoraManager's lookup order (environment folder, portable plugin file, user config folder), only the seven keys the import needs, the LoRA roots, the centralized sidecar root, the example-images root and the active library, each folder through one guard. Sidecar discovery for both storage modes and the local preview and example-image lookup live there too. The test runner and the browser test server point LoraManager's settings lookup at a disposable folder, so no test reads the real settings. * fix(lora): import the sidecars and local images LoraManager really uses (#355) The import, the legacy scan, the trigger-word cache and the image server now share LoraManager's layout: sidecars beside the models (alongside) or under the sidecar root (centralized), kept only when their model exists inside a LoRA root. Each LoRA links its LoraManager preview and example images where they are; CivitAI downloads run only for a LoRA without any local image. The review adds skipped, with_local_media and the layout, and offers no token when LoraManager's settings cannot be used; the legacy scan reports the same reason without replacing anything. find_lora_directories becomes default_lora_roots (no scan of the LoraManager install); the preview and example-folder helpers it replaced are removed. The trigger cache never raises into a running node. * fix(lora): never serve a LoraManager folder that holds the home folder (#355) The folder guard also rejects ancestors of the home folder (/home, C:\Users), compared in the realpath + normcase form, so a LoraManager setting cannot widen the image server's allow-list to every user's files. * fix(lora): show linked LoraManager videos as videos (#355) _enrich_images marks every image without is_video by its extension, so an .mp4 preview outside the output folders shows the Video preview label and plays in the inspector. Tests cover serving the sidecar and example-image folders only while the integration is on, and a filesystem root as the sidecar folder adding nothing. * feat(lora): the import review names the LoraManager settings it read (#355) The review gains a first row, LoraManager ("<source> settings · <mode> storage · library <name>", the settings file as its hover title), and its Found row counts metadata files, LoRAs and LoRAs with local images. A plan whose LoraManager settings cannot be used takes the close-only path: "LoraManager settings: not usable: <reason>. Nothing was changed." The browser fixture writes each LoRA's model file and can leave out the local preview, so the download, gate and cancel specs still exercise CivitAI. * docs(lora): describe how the import finds LoraManager's files (#355) * test(lora): set the settings folder relative to the process folder (#355) os.path.relpath fails across Windows drives (the temp folder on C:, the checkout on D:); the test now changes into its temp folder instead. * fix(lora): review findings for the LoraManager layout (#355) - The example-images library subfolder goes through the folder guard like its root, so its canonical (normcased) form matches the media check on Windows for a library name with capitals. - A failed trigger-word load is retried by a running node only after TRIGGER_RETRY_SECONDS, not on every prompt run. - A review whose discovery cannot list a folder says so (folder-unreadable, the folder's name only) instead of a generic error. * test(images): hold the listing clock in the deferred-walk test test_a_deferred_walk_waits_at_least_half_a_second set the previous walk 29.99 s back against a 30 s interval and then read the real clock again: a runner slower than 10 ms between the two lines passed the interval and the walk started at once (failed on macOS 3.13 and Windows 3.11). The module's clock is now held for the mark, so the deferred walk is always the one measured. * test(lora): write the unkept-settings fixture as raw text CodeQL read the dict with a credential-named key flowing into write_text as clear-text storage of a secret. The value is a marker, so the fixture now writes the JSON text directly and names the value MARKER. * fix(lora): link each LoRA's cover as its newest image (#355) Workspace lists show a prompt's media newest link first, so the import links a record's images last to first: the card shows the cover (a still in front of a leading video) the import put first. * fix(lora): show LoRA-linked media on the Library page and in the inspector (#355) Library previews, the positive detail and the per-prompt images page list a prompt's linked media under the output folders plus, while the LoRA integration is on, LoraManager's guarded folders and the download cache (_media_roots; output root indices keep their meaning). That media keeps its rooted identity, so Info and the media viewer open it; the path-based serve route answers for those roots only with a file the database records (either path spelling), never serves or makes a preview there, and output requests never read LoraManager's layout. Such media is marked linked_only: no preview file, no favorite star, no Tag action. Show in folder accepts those roots; the download cache is named, not created. * fix(lora): expose only linked files under LoraManager's folders (#355) Serve, inspect and Show in folder share one gate for a LoRA-linked root (_media_roots past the output folders): a file the database does not record as a linked image is refused with 403, checked by its given and its real path. Before, inspect and reveal read any file under those folders by path. * fix(lora): check the file Show in folder would open under a LoRA root (#355) reveal_image resolved the requested path through the client-path resolver, which can fall back to a path anchor; under a LoRA-linked root the file that would open now passes the same recorded-media gate, so only a linked file is ever revealed there.	2026-10-26 21:21:33 -07:00
Vito	c7ba481d3c	perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) (#353) ... * perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) The image monitor now reports every settled output (linked or not, also when linking raised) with its resolved path through set_output_listener. The API's notify_output_settled hops from the monitor's worker thread onto the server's loop (call_soon_threadsafe) and does nothing before the loop is known or after cleanup. mark_root_dirty matches the path against each walk's resolved_root (normcase(realpath), computed in the executor by walk_root, now its fourth value) and marks the root dirty. A dirty root gets one background walk, no sooner than rewalk_interval = min(15, max(5, 5 x last walk's duration)) after the previous walk started (last_started_at / last_duration are recorded on every walk). _schedule_walk is shared with PR 1's delayed stale refresh: one deferred walk per root, never armed while a walk runs (the walk's end re-checks dirty: the trailing walk), at least 0.5 s away, the earlier deadline wins, and every walk start cancels it. A failed walk clears dirty. Rescan semantics are unchanged. * fix(images): keep removed roots and cancelled walks from walking again (#348, PR 2) Review findings on the monitor-triggered walks: - reset_gallery_roots_cache now retires every listing record: its pending timer is cancelled, dirty is cleared, and no mark, timer or trailing walk starts a walk of it until a reader or the warm-up asks for that root again. Before, a timer or trailing walk pending when the roots changed stored a listing for the removed root, and the monitor (which watches the old roots until restart) kept it walking on every new output. - A walk task cancelled outside cleanup_gallery_listings no longer starts a trailing walk while the loop shuts down. - root_contains ignores a Windows extended-length prefix (\\?\ and \\?\UNC\) on either side, which realpath can keep on a file past MAX_PATH. - The REWALK comment states the ~20% GIL share only for walks up to 3 s. * test(images): prove a cancelled walk starts no trailing walk (#348, PR 2) The previous assertions held even with the trailing walk, because that walk finished before they ran; assert the walk number and the scan count. * perf(images): add a settled output to the listing without a walk (#348, PR 2) Live test: each landing in a 20,000-file folder cost a full walk (~0.4 s of GIL) plus a full workspace rebuild, so Images reads during a run took up to 2.1 s. The monitor's report now patches the stored listing instead. - output_settled (the monitor hop's target) slices the file's part below the root in its own case (relative_part; Windows compares folded), and _run_adds places one root's pending files in one executor batch: _settled_entry applies the walk's filters (suffix, thumbnails, depth, symlink by lstat; macOS on-disk spelling), insert_entry puts the (Path, mtime) entry where the newest-first sort does, add_folder copies the folder sets. Stored only when no walk started and no drop happened meanwhile; walked_at is untouched, so the 30 s refresh still walks. - Anything inexact marks the root dirty and walks as before: a running walk, an unsliceable path, a failed lstat, a tie in mtime, the cap, an unexpected error. Rescan, the refresh, the warm-up and invalidation stay full walks; a walk start forgets the pending adds. - The workspace caches follow a patched listing by entry identity (listing_delta): only added values are built, the newest order places them by bisection, the favorites lookup is copied and extended. Synthetic 20,000 files: per landing 252 ms (walk + full build) -> 18 ms (3 ms add + 15 ms warm build). * fix(images): review fixes for adds without a walk (#348, PR 2) - _newest_step steps the cached newest order only when the per-file cache still holds exactly the values this call built. A stale page read racing a newer warm build (both run in executor threads) stored an order labelled with older values than it held, and the next read inserted the new file a second time. It also sorts in full when a roots reset dropped the per-file values meanwhile. - The stored spelling is checked on Windows too (nt._getfinalpathname: realpath keeps the given case when it cannot open the file), and before the name filters, which compare the walk's stored spelling. macOS opens non-blocking, so a FIFO named like an image never holds the executor. - An output in a folder the walk cannot list (no read permission) is not added, as the walk skips it. - Windows-portable unit tests: POSIX paths in the insert tests, and a str.lower fold for the length guard (Windows' normcase keeps lengths). * test(images): reproduce the newest-order race and fix a Windows clock flake (#348, PR 2) - The race test now has the newer build step from the values the cached order was built from, which is the interleaving that inserted a file twice; it fails without the parts check in _newest_step. - test_a_failed_walk_records_its_timing_and_clears_dirty compared two monotonic readings that Windows' ~15.6 ms clock can make equal; the previous start is moved back instead. - TESTING.md: os.access ignores ACLs on Windows.	2026-10-26 16:14:50 -07:00
Vito	9ac471eb72	perf(images): share and warm the Images listing (#348, PR 1) (#349) ... * perf(images): share and warm one listing per output root (#348) The Images page, its sidebar counts, the subfolder list and the legacy gallery each rebuilt the output listing when the 30 s cache expired, all at once, and the subfolder list computed relative paths on the event loop. Thumbnail writes also threw the listing away. Keep one RootListing per output root (py/api/gallery_listing.py): - one walk per root at a time; concurrent readers wait for it (shielded) - a listing older than 30 s is served at once and refreshed in the background - the walk, the comparison with the previous listing and both folder sets run in the executor; an unchanged walk keeps the same list object - os.scandir probe: a missing root fails the walk and keeps the previous listing (one warning per failure streak) - Rescan waits for a walk started after the click and is a 500 when it fails, even with a previous listing - deletes keep the blocking invalidation; a generation counter that survives drops discards walks that raced it - thumbnail jobs, conversion and the legacy generate/clear routes no longer drop the listing - warm-up on the host's on_startup, cancelled on on_cleanup Adds tools/vendor/perf/images_load_probe.mjs (maintainer tool, not CI) and the TESTING.md section "Images listing cache (#348)". * fix(images): drop listings on the loop when invalidated from a worker thread (#348) A settings save (gallery roots change) and the browser-test seeds call invalidate_gallery_cache() from the executor. Listing records are only touched on the event loop, so an off-loop call now hands the drop to the loop and waits for it, keeping the call synchronous for its callers. The shared-Rescan test waits for the later Rescans to take their walk number instead of sleeping. * test(images): bound the shared-listing test waits (#348) A macOS py3.12 run stopped inside the shared-listing tests and sat until the 6 h job limit with no traceback. It does not reproduce locally (about 300 runs on 3.12 and 3.14, pinned to one core). Requests and background settles in TestSharedListing now give up after 30 s, so a recurrence fails with the stuck await instead of hanging the run. * test(images): let the settle loops yield so a finished walk leaves the set (#348) On CI (ubuntu and macOS py3.12) the runner's watchdog dumped test_a_root_that_recovers_warns_again_on_its_next_failure spinning in _settle: asyncio.gather over tasks that are already done returns without yielding, so the set's discard callback, queued when the walk finished, never ran and the loop never ended (wait_for could not fire either). Delaying discard by one loop turn reproduces it locally; both settle loops now yield one turn after each gather. * test(browser): poll the import review's rows instead of reading them once gridOf reads the dialog's rows once; right after the click the review may not have rendered them yet (firefox CI: received undefined). * perf(images): walk the output with scandir entries and build folders from strings (#348) The live test showed a read during a background refresh taking 2.2 s instead of 0.07 s: the walk held the GIL for about 2 s in the server. py-spy split that walk into two halves: Path.relative_to per file in folder_sets, and Path.stat + Path.is_symlink + Path joins per file in the scan. - _walk_media visits folders in os.walk's order with os.scandir entries: the file type comes from the listing (no lstat per file) and the mtime from entry.stat (one call, none on Windows). _iter_media_files is now a thin wrapper, so its other caller is unchanged. - folder_sets slices the path strings (in the path flavour's separator) instead of calling relative_to, falling back to it for a path that does not start with the root string. On the maintainer's 19,935-file output folder: scan 0.386 s -> 0.112 s, folder sets 0.452 s -> 0.007 s, both results identical to the previous code. A reference copy of the os.walk version checks order, filters, the depth cap, symlinks and the file cap. * perf(images): build the workspace caches behind the warm-up and changed walks (#348) The live test after the faster walk still took 2.9-5.2 s for the first Images open after a restart. py-spy showed the walk was now small; the time went to the workspace's own caches, cold after a restart and built by three parallel first requests at once: the favorites listing (Path.relative_to per file), the per-file values and newest order, and the prompt-link read. - Once the host's startup warm-up has run, a stored walk that changed a listing schedules one executor job that builds what a first page reads, through the request path itself (_select_workspace_page with the default filters, then _workspace_favorite_paths). A walk stored during a build asks for one more build; an unchanged walk (same list kept) builds nothing; a failed build only warns; on_cleanup stops it. Servers that never ran the warm-up (the test servers) start no build. - _favorite_listing slices the path strings as _listed does, falling back to relative_to for a path outside the root string. * perf(images): start a stale listing's refresh walk a second after the read (#348) After the faster walk, a read that found the listing stale still took 0.5-1.0 s against 0.07 s: it was served from the old listing but ran beside the refresh walk it had just started, sharing the GIL. The refresh now starts REFRESH_DELAY (1 s) after that read through a per-root rewalk_handle, so the page's own requests finish first. One pending refresh per root and none while a walk runs; any walk start (a cold read, Rescan, the warm-up) cancels a pending one; on_cleanup cancels them before its first await. * test(browser): accept KB or MB for the picked safety backup's size The restore preview showed '2.5 MB' for the test server's database copy in a CI coverage shard: its size depends on what earlier tests on that server wrote, and the formatter switches to MB at 1 MiB. * fix(images): keep the 3.10-3.13 suffix rule in the scandir walk and harden the warm-up (#348) Review of the walk speedup: - os.path.splitext drops names like ..png that Path.suffix listed on Python 3.10-3.13 (3.14 changed Path.suffix, which hid it from the parity test on a 3.14 venv). _name_suffix spells out the 3.10-3.13 rule; the test now asserts ..png and ...webp are listed and g. is not. - _warm_workspace reads the roots inside its try, so a failing _get_all_output_dirs warns instead of leaving an unretrieved task error. - The folder_sets fallback test now takes the fallback (a Windows case mismatch); the coalescing test pins the walk count.	2026-10-26 11:46:54 -07:00
Vito	2964a3b632	feat(thumbnails): WebP/JPEG XL thumbnails, server-side thumbnail settings and Convert outdated (#319) ... * feat(thumbnails): thumbnail format, quality and size as server settings Thumbnail format (webp default, jxl, jpeg), quality (1-100, default 80) and size limit (128-2048 px, default 512) are server settings in the config.json thumbnails section, saved with the settings revision. The never-read gallery thumbnail_size is gone: older saves wrote its unused default 256 into config.json. One module names previews (<stem>_thumb.webp/.jxl/.jpg) and finds the preview a source owns, preferring the configured format; legacy names are still found. The encoder module writes every preview atomically and verifies it decodes. JPEG XL needs no new dependency: Pillow (natively or with pillow_jxl), ffmpeg with libjxl, or cjxl; without one the settings refuse jxl (400 jxl-unavailable) and GET says why. Encoder processes are tracked, bounded and killed at shutdown. * feat(thumbnails): every preview writer and lookup follows the thumbnail settings Automatic previews (capture and first view), the thumbnail job, the legacy gallery routes and the history route write <stem>_thumb.webp (or .jxl/.jpg) with the configured quality and size limit, for images and videos alike: video frames, ffmpeg's included (a PNG on stdout, never upscaled), and the placeholder are encoded like images. Listings, duplicates, enrichment, the autotag scan and the history route show any preview the file owns, the current format first, so older _thumb.png previews keep working until converted. A request for an older name answers the current preview. Previews are served with an explicit image/webp or image/jxl type and the same cache headers. The plan and status count missing, outdated (another format) and ready previews; the missing job never writes beside an outdated one. Removing a file removes its previews in every format; the clear removes .webp and .jxl too. Encoder processes are killed at shutdown. A preview is only a placeholder: inspection refuses paths under thumbnails/ and always reads the original, and the inspector cover accepts the listing's preview in any format the file owns. * feat(thumbnails): Convert outdated thumbnails as an Activity job POST /jobs/thumbnails {mode: "convert"} starts "Convert outdated thumbnails" under the thumbnail job's kind, guarded admission, worker threads and cancel at a file boundary. For each file whose preview is in another format, it writes the new preview from the source (atomic, verified; a video with no decodable frame fails instead of getting the placeholder), then removes that file's old-format previews in its thumbnails folder. A failure keeps the old preview; orphan previews and links are never touched. The final line reports converted, failed and bytes before and after. The plan counts it with mode=convert; the legacy gallery stream refuses to attach to a running conversion. * feat(thumbnails): thumbnail settings in Appearance and Convert outdated in the workspace System > Appearance gains a Thumbnails surface: format (WebP, JPEG XL, JPEG), quality and size limit, saved to the server with the settings revision. JPEG XL is offered only when the server has an encoder, else the option is disabled with the server's reason (D130); a note says when this browser cannot show JPEG XL. Saving a new format converts nothing: the row then says previews keep their old format. Thumbnail maintenance gains the Convert outdated scope and names the server's settings instead of its own quality and size fields (D131). Overview's thumbnail row reads N missing, M outdated, K failed and offers Convert outdated while previews are outdated. The workspace decodes a tiny JPEG XL once: where it cannot, cards, the Library and the inspector cover load the original for a .jxl preview, and any preview that fails to load falls back to the original. The viewer and downloads keep using the original (regression test). README explains enabling JPEG XL; TESTING.md records D130 and D131. * test(thumbnails): coverage floors for the thumbnail modules and WebP in page workflows The two new frontend modules enter the coverage baseline (98% lines); other floors only rise. The visual fixture's preview is WebP and Appearance has its Thumbnails surface in the System shell check. * fix(thumbnails): say when the thumbnail settings could not be read Like Appearance's other server rows, the Thumbnails rows show 'Server settings could not be read.' while the read failed, instead of only staying disabled. Firefox exposed the gap: its JPEG XL note arrived first and changed the row notes the page workflow counted. * test(thumbnails): accept the legacy preview's spelling a case-insensitive disk finds On Windows and macOS the '.jpg' lookup finds an older release's b_thumb.JPG first, so the listed URL spells '.jpg'; it names the same file there. The test now accepts either spelling and checks the URL serves that preview. * fix(thumbnails): address the final review of the thumbnail format work - ffmpeg writes JPEG XL with -update 1, so a % in a file name is never an image-sequence pattern, and reads videos as file:<path>. The clear and a startup sweep remove temp files that interrupted writes left behind. - Convert outdated converts each group of same-stem files once, from the first source in the R2 order. An old preview that is already gone counts as removed, and each file's bytes count once. A current preview is only kept if it decodes. - An encoder that fails makes the write detect encoders again once. Another encoder is used if one is found; with none, the file is written as WebP and the availability Appearance shows is refreshed. - Changing the thumbnail format, quality or size while Convert outdated runs returns 409 busy. If the format still changes underneath, the job stops without removing anything. - Legacy preview spellings are matched case-insensitively, so X_thumb.JPG counts as outdated. - History's cover uses the same lookup as Images. It serves .jxl only when Accept lists image/jxl, and otherwise the original or the placeholder. - A video whose thumbnail the browser cannot show keeps its own element (Images) or its label (Library). * fix(thumbnails): keep the new preview, probe encoders only when gone, sweep reused PIDs - Convert never removes a path that is the preview it just wrote. It compares the device and inode, or the resolved and normcased paths when the disk has no inode numbers. This covers x_thumb.JPG on a case-insensitive disk. - A JPEG XL encode failure re-runs encoder detection only when the encoder's binary or module is gone. A "no encoder" result is checked again on the settings read, on a jxl save, and after 60 s, so one slow probe no longer turns JPEG XL off until a restart. - The startup sweep now also removes temp files that carry this process's PID but predate its start. Containers reuse PIDs. * fix(history): let the cover ask for JPEG XL when this browser decodes it Firefox can decode JPEG XL without listing image/jxl in Accept. When the workspace's probe decodes JPEG XL, it adds jxl=1 to the History cover URL. The server serves a .jxl preview when that flag is set or when Accept lists image/jxl. In every other case it falls back as before. The flag accepts only 0 or 1, and the response stays no-store. * fix(thumbnails): keep going past the per-batch cap until every preview is made The missing, rebuild and Convert outdated jobs stopped after MAX_THUMBNAIL_FILES (10,000) files and said "N more on the next run", but nothing ever started that next run. The cap now bounds one batch. After each batch the job plans again, leaving out every file it already worked on, and continues until nothing is left, the job is cancelled, or a batch makes no progress. That last case stops on files that fail every time instead of looping over them. Activity progress counts against the overall total, and the final line and the legacy stream's complete frame report every batch. Admission, worker threads, cancellation and the #316 rules are unchanged. The Thumbnail maintenance dialog now counts all the work instead of saying "more on the next run".	2026-10-24 21:00:30 -07:00
Vito	fc3266d455	fix(thumbnails): generate thumbnails automatically and use them everywhere (#316) ... * fix(thumbnails): generate thumbnails automatically and use them everywhere New outputs never got a preview, so the workspace grid loaded full-size originals (844 of 19,907 images on a live install, exactly the newest). - Listings always give images and videos a thumbnail URL; a missing preview is made by the serve route on first view (bounded, single flight per file) and falls back to the original, remembering files that cannot be decoded. - The image monitor queues each settled output's preview on one background worker that follows Worker threads. - A scan that leaves 200 or more previews missing starts the visible thumbnail job; fewer go through the worker. - Automatic previews wait for a restore, a duplicate removal, a sources change, the gallery's clear and exclusive jobs. - The inspector cover uses the preview; the Overview row counts previews that failed. * fix(thumbnails): keep the video fallback tested and stop the preview worker before cleanup A video listed without a preview still renders its own element and is released on refresh. Test teardowns and the browser reset stop the automatic preview worker before the temp tree goes, so no preview lands after cleanup (Windows). TESTING.md documents automatic previews. * fix(thumbnails): admit previews on the loop and never send a video to an img Final review fixes for automatic previews: - A video preview that cannot be made is the placeholder frame (no-store), never the video; OpenCV failures fall through to ffmpeg and the placeholder. Cards and the inspector cover swap back to the video element when the preview image fails. - View requests are admitted on the event loop (3 writers, one flight per file, at most 64 waiting; past that the fallback at once). Writes run on a small dedicated pool, never the default executor. The history route shares the admission, flight key and link checks. - Drive, UNC, empty and dot segments are refused as text; the folder listing runs inside the admission. - Holds are checked after the slot and again before the rename; only decode failures are remembered. - Bounded, logged shutdown: ffmpeg is killed, no worker restarts after cleanup, and a reload stops the old worker. - The listing says whether the preview file exists (thumbnail_ready) for the legacy gallery's dot; media in a nested thumbnails folder keeps its original; file names the server quotes match in the inspector. * fix(thumbnails): cover the image fallback card and document the admission An image listed without a preview (a nested thumbnails folder) still shows its original; TESTING.md describes the view admission, the video placeholder, the holds before the rename and the bounded shutdown. * fix(thumbnails): let an idle preview worker leave and fit case-insensitive filesystems The capture worker exits once its queue is empty, so no thread outlives its work; the next output starts a fresh one. The mixed-case source test expects no folder listing where the exact name already resolves (macOS, Windows). * fix(thumbnails): answer shared flights with the outcome only and restart a failed worker The history and serve routes share a flight key, so a flight now answers only the ensure outcome whichever route started it; the serve route derives the preview path and, for a fallback, looks its source up again on the view pool. A view joining a history flight no longer gets a 500 or an image answered with the video placeholder. The capture worker clears its running state however its loop ends, and a worker that cannot start leaves the file queued for the next enqueue.	2026-10-24 08:35:14 -07:00
Vito	6505d339aa	feat(system): thumbnails on Activity (#311) ... * feat(system): run thumbnail generation as an Activity job GET /prompt_manager/images/thumbnails/plan counts what a run would do (targets, fresh, remaining past the cap, source labels) and writes nothing. POST /prompt_manager/jobs/thumbnails {mode, size, quality} starts the "thumbnails" job: admission guarded, cancellable at a file boundary, outcome groups generated / skipped / errors. size is the longest edge (128-2048), quality the encoder, and rebuild writes fresh previews again through the same atomic rename, so a failed file keeps its old preview and originals are only read. The legacy gallery stream now starts or attaches to the same job and streams its unchanged frames. A client that goes away no longer stops the work; the gallery's Cancel asks the server to cancel the job. * feat(activity): per-kind job descriptors One descriptor table (activity-kinds.mjs) decides each job kind's pill verb, toasts, line, tiles, outcomes, openable rows and labels; the store re-exports it. The scan and image tagging views are unchanged. A thumbnail job counts Generated, Skipped and Errors, reads "Generating thumbnails" in the pill and "Thumbnails finished · N generated · M errors" when it ends. Any other kind (the health check, a held legacy-page operation) gets the generic descriptor with no tiles and no Results, instead of the scan's zero tiles. A running job without a known total shows an indeterminate progress bar and no percentage, and only a job the server can cancel shows Cancel. * feat(system): review and start thumbnail jobs Sources' Thumbnail generation -> Review options and Overview's Review thumbnails open the concept's Thumbnail maintenance dialog: scope, quality (preselected from Image loading quality) and preview size limit, a plan line from the read-only thumbnail plan (D99) and Start, which starts the thumbnails Activity job and opens it. A size outside 128-2048 requests nothing; a refused start shows the busy link toast. * fix(system): thumbnail job review fixes (#249 S5) - The legacy stream's first frame is job {id, started}, sent after the subscription so no runner frame is missed; it reads the registry after prepare, so a job started in that gap is attached to, not refused. - Every exit sets a truthful final status line; nothing to do never reads "Processing file 0 of 0". - Cancel is checked per file during the planning walk; a job cancelled there publishes no start frame. - The ffmpeg fallback keeps the chosen quality (-q:v) and the aspect ratio; the placeholder uses the chosen JPEG settings. - A locked preview on Windows reads "Preview in use by another program; the old one was kept", not the raw WinError. - Thumbnail work no longer defers the monitor's Needs review recording: it writes previews only, never records. - The disconnect test now proves the job outlives its stream (one worker, 9 files, three batches); the attach test waits on an event. * fix(gallery): cancel the stream's own thumbnail job by id The settings panel and the startup dialog cancel the job named by the stream's job frame, and only one their stream started. A Cancel before the frame waits for it, a double Cancel posts once, and a job another tab or System started keeps running. The startup dialog now settles on Cancel instead of staying on "Cancelling...". * fix(activity): indeterminate pill bar and the nothing-to-do line The pill's mini bar sweeps while a job's total is unknown instead of showing an empty fill. A finished thumbnail job with nothing to do shows the server's final line. Coverage floors raised for gallery.js and activity.mjs.	2026-10-23 16:58:06 -07:00
Vito	3b1d00501f	fix(config): serialized config store and registry admission (#289) ... * fix(config): report settings persistence failures (#220) Settings saves now go through one serialized config store (py/config_store.py). A commit re-reads config.json under a process lock, applies the change to a copy and replaces the file atomically (temp file beside it, fsync, close, os.replace), so a failed write leaves the old file in place. POST /prompt_manager/settings commits first and only then publishes the values at runtime. A write failure is 500 config-write-failed, an unparseable config.json is 409 config-unreadable (the file is kept instead of replaced) and a stale expected_revision is 409 settings-changed. A save reports persisted, revision, runtime_applied and restart_required_fields; GET adds the revision. Only submitted keys are written; other sections and the CivitAI key are kept. * fix(lora): report integration save failures (#220) POST /prompt_manager/lora/enable commits integrations.lora_manager through the config store and publishes the runtime values only after config.json holds them. Before, it set the runtime values, rewrote the whole runtime config (dropping sections it did not know) and reported success even when the write had failed. A write failure is 500 config-write-failed and changes nothing at runtime; an unparseable config.json is 409 config-unreadable. The body goes through the capped JSON reader and the flags must be booleans. A trigger word cache that fails to load is reported as persisted:true, cache_loaded:false. The CivitAI key is never echoed. The LoRA tests now write to a temporary config path. * fix(jobs): refuse legacy record-changing operations while a job runs The job registry gains admission classes. A records job is refused while any records or guarded job runs, a guarded job while a records job runs, and read jobs share only the one-job-per-kind rule. record_changing=True stays as the spelling of records. Job summaries add cancellable and admission, and a non-cancellable job ignores cancel. JobRegistry.hold() registers a running, non-cancellable job for a legacy-page operation and ends it when the request does. The legacy restore, maintenance, prompt cleanup, duplicate image delete, LoRA import and batch auto-tag routes hold as records; thumbnail generation (both routes) and thumbnail clear hold as guarded. A maintenance run of only read checks holds as read. So a restore during a scan, or a scan during a legacy restore, is refused with the running job named instead of interleaving; this closes the gap noted on #269. Refusals keep each route's shape: the JSON routes return the legacy 409 "<title> already running", and the thumbnail and LoRA streams send their existing error event. * test(lora): keep the preserved-section fixture free of secret values CodeQL flagged the fixture that writes the secret constant into a test config file as clear-text storage. The test only needs a marker value to prove the stored key survives a save. * fix(config): read under the commit lock and publish saves in write order - ConfigStore.read takes the commit lock, so no read handle is open while config.json is replaced (Windows refuses that replace). - os.replace is retried briefly on a Windows PermissionError (antivirus or indexer handles) before the save is reported as not written. - The temp file is written beside realpath(config.json), so a symlinked config keeps its link and a bind mount fails truthfully. - commit() takes a publish callback that runs under the lock after the write; settings and LoRA saves publish there, so two concurrent saves reach the runtime in write order and the runtime ends on the last one. - The concurrency tests now wait until the second caller is blocked on the lock (tests/lock_probe.py) before releasing the first. * fix(jobs): require an admission class, admit holds on entry, true Activity states - JobRegistry.start raises TypeError when neither admission nor record_changing is given, so a new job never runs as "read" beside a scan by omission. - hold() admits when its block is entered, so a hold that is never entered leaves no running records job behind; callers that answer a refusal in their own way check blocker() first, with no await before the block. - A held SSE operation that ends in an error event (thumbnail generation, legacy batch auto-tag) is listed as failed, and refused input (duplicate delete without paths, an unreadable thumbnail body) is answered before the hold, so it is not listed as a job. - Activity keeps a job's cancellable flag (true when an older server omits it) and shows Cancel only for a running, cancellable job, in Recent work and in the job view. A held legacy job no longer offers a Cancel that does nothing. * fix(admin): legacy pages show why a request was refused A job running in the workspace refuses the legacy record-changing requests, and a corrupt config.json refuses a settings save. The legacy pages now name the reason instead of hiding it: - LoRA import: the busy and failure frames carry status and progress, and the admin modal handles type "error": it shows the message, notifies and closes, instead of staying on "Initializing..." at 0%. - Legacy batch auto-tag: a refusal by another job is a 200 stream with one error frame ({type, status, message}), which both pages already render; a second batch keeps its exact JSON 409. - Gallery cache clear: a refused server clear shows the reason and stops before the success toast and the "Not generated" reset. - Admin settings save and maintenance show the reply's error ("config.json cannot be read...", "Scan for metadata already running") instead of "Failed to save settings" or "HTTP 409: Conflict". - Tests cover the read-only /maintenance branch: a read-only selection runs during a scan as a read hold; a mixed or default selection is refused. * refactor(api): move the settings routes out of admin.py GET and POST /prompt_manager/settings, their validation, the config commit and runtime publication, and the shared commit-failure reply now live in py/api/settings_routes.py as SettingsRoutesMixin, before S3 adds host fields there. No behaviour change: the same routes, payloads and status codes; admin.py drops about 300 lines. * chore(coverage): raise the floors of the frontend files this round touched admin.js, gallery.js, activity.mjs and activity-job-view.mjs, to their measured coverage. No floor is lowered. * fix(admin): the gallery starts no thumbnail status read while it leaves The gallery's startup thumbnail check runs one second after load. When that second ended while the page was leaving (CI, WebKit: the workflow test navigates from the gallery to the admin page), WebKit failed the new read with "Fetch API cannot load ... due to access control checks." and Playwright reported a page error. Like the admin page (#290), the gallery now has a page lifetime that ends on beforeunload and pagehide (and starts again on a back/forward restore). The check starts no read once it has ended and passes it as the read's signal. A page-clock workflow test that leaves before the second ends failed on both events, in Chromium and WebKit, before this change.	2026-10-22 21:39:01 -07:00
Vito	1b8fbdbb75	feat(annotations): image tag routes, model gate and scored suggestions (#278) ... * feat(annotations): add scoped observe, capture preview and tag save routes POST /prompt_manager/image-annotations/observe captures the file named by the listing identity {root_index, root_scope, path} (scope checked against the current roots before any file access) and answers the saved tag set of exactly those bytes, refreshing a matching record's verified fingerprint. GET/DELETE .../capture/{token} serve the private copy (no-store, nosniff, pinned with acquire/release_use while it streams) and release it. PUT .../tags re-hashes the current file and writes only when it still holds the captured bytes, under the storage epoch and tag revision (file-changed, storage-changed, tag-conflict, tag-limit, replayed). Bodies are capped at 64 KiB with unknown fields refused; envelopes never carry server paths or locators. Restores invalidate every capture and the host's cleanup removes the capture directory. * feat(autotag): scored suggestions with a configurable count behind a shared model gate - py/model_gate.py: one process-wide, non-blocking gate for loading, swapping, running, downloading and unloading a tagging model. Worker calls run shielded; a release requested while one still runs takes effect only when it exits (client disconnect keeps the gate). - Legacy /autotag/single, /start, /download/{type} and /unload hold the gate and answer 409 {code: "model-busy"} while another operation holds it; their bodies are otherwise unchanged. /autotag/models gains suggestion_limit. - AutoTagService.generate_tag_suggestions: WD14 (name, probability) best first, GGUF/HF (name, None) in output order, first `limit` distinct case-insensitively; generate_tags keeps its return; the image file is closed before inference returns. - POST /prompt_manager/image-annotations/suggest runs one engine on the pinned capture, with strict per-engine options, never downloads (409 model-unavailable) and never writes. - config.json autotag.defaults.suggestion_limit, integer 1..100, default 10, validated on load. - Capture probe decides Generate from the image header alone, so prompt metadata over the inspection budgets no longer blocks it. * feat(images): match image tags in the Tags filter, search and inspector The workspace Images listing now admits a file under a Tags filter when every requested tag is carried by its linked positive or its current image tag set, and q also matches image tag names (case-folded substring). Image tags count only while a byte record is verified at the file's present (size, mtime_ns) (spec amendment S1): the tagged records are streamed once, only listed files carrying a filter tag or q hit are stat'ed, and their linked positives are read through the file_path index so tags, category, rating and the positive count combine. The rooted inspector envelope gains a validated image_tags facet (current / none / unavailable) read at the inspection's own stable stat. A failed image-tag read fails a Tags filter like a failed link read and degrades a q search to no image tags; plain listings read none. Legacy gallery and search responses are unchanged. * fix(annotations): answer a fixed coded message when the filtered image list fails A failed prompt-link or image-tag read under a Tags, category or rating filter returned the raw exception text (SQL or database messages) in the 500 body. The workspace listing now answers code listing-failed with a fixed sentence and logs the traceback. The legacy output listing and the inspector image-tag read now log with exc_info as well. * fix(annotations): drop suggestion names that Save would reject A JoyCaption part over 50 characters or holding a newline came back as a preselected chip, so the first Save of the dialog failed with a 400 and nothing was stored. generate_tag_suggestions now applies the vocabulary tag-name rules before the case-insensitive dedupe and the limit count, so the requested number of valid chips still comes back. * fix(annotations): stop changing the process warnings filters during capture header_dimensions wrapped Image.open in warnings.catch_warnings with an error filter for DecompressionBombWarning. catch_warnings is not thread-safe, so two overlapping captures could leave that error filter installed for every later Pillow open in the ComfyUI process. Only Pillow's DecompressionBombError is now mapped to HeaderTooLarge; a size that only warns is refused by the existing inference pixel cap. * fix(annotations): free pins on cancellation, serve preview ranges, code every storage error - A preview cancelled while its file opens closes the handle and only then ends the pin, so a doomed capture is never unlinked while open. - An observe cancelled while capturing or reading releases the capture instead of leaving it until the idle expiry. - The model gate takes an on_skipped callback that runs once when the work never starts; Generate passes the capture's release_use, so an executor refusal no longer leaks a pin against the capture budget. - The preview answers a single byte range with 206 and Content-Range (416 for an unsatisfiable or multi-range header), still streamed inside the pin, so Safari and WebKit can play video captures. - _coded maps tag-limit (400), metadata-conflict and vocabulary-conflict, and falls back to 409 with a fixed sentence for any other code instead of raising KeyError. * fix(annotations): deny case and Unicode spellings of a listed file On macOS (case- and normalization-insensitive APFS) a crafted target such as A.PNG or an NFD name opened the listed a.png or NFC file while realpath kept the requested spelling, so the tags were stored under a second locator the Images filter never shows. Capture and verify now require each path part to be spelled as its directory lists it and deny any other case or Unicode spelling as an alias (409 source-changed, like a symlink alias). A missing file with no other spelling keeps its usual outcome. * test(annotations): one shared folder_paths stub for the API test modules test_api_images and test_api_prompts installed a MagicMock folder_paths, so whenever one of them was imported before test_api_autotag the later patch("folder_paths.get_output_directory") in test_api_history_outputs left a mock attribute behind and two test_diagnostics_config tests failed. tests/host_stubs.py now installs one real-module stub, used by test_api_images, test_api_prompts and test_api_autotag (and through test_api_images by the annotation modules, whose copy is gone), so the modules pass in either import order. * fix(annotations): derive the suggestion cap from the image tag set cap py/autotag.py now takes MAX_SUGGESTION_LIMIT from database.annotations MAX_IMAGE_TAGS. The two copies that cannot import it (py/config.py loads before the database, utils/inspection_response.py stays storage-free) are pinned equal by a test, so the 100 limits cannot drift apart. * docs(testing): record capture aliases, preview ranges and the PR B follow-ups TESTING.md now states that case and Unicode spellings are aliases, that the preview serves byte ranges inside the capture pin, where the cancellation tests live, and two tracked follow-ups: the legacy batch's ungated cleanup unload under its own last inference, and the three files over the 800-line ceiling (split py/autotag.py first).	2026-10-22 10:36:58 -07:00
Vito	ecd21165a7	feat(history): add chronological runs and exact output inspection (#227) ... * feat(history): add bounded chronological read projections (#209) * fix(history): validate internal continuation positions (#209) * feat(history): expose fenced history windows and run details (#209) * fix(history): release retired projection connections (#209) * feat(history): resolve reported outputs through current authorization (#209) * fix(history): reject invalid inspection targets (#209) * feat(history): add scoped history state and current-file adapter (#209) * fix(history): preserve slideshow and clear stale reentry actions (#209) * feat(history): render recorded executions and frozen run details (#209) * fix(history): recover receipts and refresh capture evidence (#209) * test(history): verify chronological history across restore and host modes (#209) * fix(history): enforce chronological marker visual parity (#209) * fix(history): resolve final chronological review findings Address #209 recovery, bounded visit restoration, exact receipt availability, and portable regression coverage. * test(history): repair Windows fixtures and review feedback Address #209 and PR #227 native path, cleanup, media and review-comment expectations. Document Windows portability preflight without changing production behavior. * ci: raise the viewerjs job budget for the larger browser suite The cross-browser workflows suite grew from 396 to 496 tests on this branch, which pushed the whole viewerjs job to about 25.5 minutes against the 25 minute cap set in #181. Two exact-head runs on this branch timed out without any test failing: run 39173019283 was killed on test 474 of 496, and run 39175125474 printed "496 passed (12.8m)" and was then cancelled during Playwright shutdown. Raise only the job budget. Browser projects, workers and per-test timeouts are unchanged. Splitting the browsers into parallel jobs is tracked separately.	2026-10-18 17:17:05 -07:00
Vito	d9537a726c	feat(workspace): complete shared inspector and collection viewer (#224) ... * feat(inspector): expose bounded physical file evidence * fix(inspector): distinguish ambiguous saved associations * feat(inspector): derive conservative generation facts * fix(inspector): guard fact ancestry and inspection state * feat(inspector): share complete file details and exports * fix(inspector): preserve fact states and touch targets * feat(viewer): navigate the loaded media collection Refs #216 * fix(viewer): refit after current stage geometry changes Ref #216. Observe the current media stage after readiness and wrapped title changes without rebuilding focused controls or resetting transforms at unchanged dimensions. * feat(viewer): add lifecycle-safe slideshow and media preferences * fix(viewer): resume gallery continuation after close (#216) * fix(images): guard source recovery and wrap viewer actions Clear stale root and folder intent, cancel old pages, and wait for current source discovery before loading a refreshed collection. Preserve safe retry through failed, superseded, and disposed discovery. Wrap narrow viewer footer actions and wait for measured gallery viewport settling in responsive checks. Refs #216; addresses PR #224 review. * test(viewer): await gallery layout after viewport changes Require actual inner, root client, and document widths to agree after each responsive-case viewport and CSS zoom transition. Keep the existing exact frame, overflow, focus, and coarse target assertions unchanged. Observed PR #224 CI initial mobile resize at537px before pending card relayout settled to356px. Refs #216.	2026-10-18 09:41:53 -07:00
Vito	caa9649f32	feat: reuse positive and negative prompt combinations (#195) ... * feat(db): add pair schema and stage restore migrations for #193 * feat(database): persist reusable ordered prompt pairs (#193) * feat(api): expose reusable pairs and atomic metadata enrichment (#193) * feat(metadata): resolve canonical prompt pairs with bounded provenance (#193) * feat(scan): enrich image pairs without guessing or duplicate links (#193) * fix(database): preserve pair tags during global maintenance (#193) * feat(tracking): enrich live image pairs and cancel scans safely (#193) * feat(nodes): encode paired prompts and count cached recipe runs (#193) * fix(database): validate restored pair ownership and provenance (#193) * feat(admin): report pair scan outcomes and cancellation (#193) * feat(ui): add paired prompt editor and explicit recipe library for #193 * fix(autocomplete): include reusable pair metadata vocabulary (#193) * fix(pairs): honor recent sorting and conservative scan outcomes (#193) * fix(ui): protect paired draft saves and clarify negative variants (#193) * fix(metadata): preserve bounded paired execution evidence (#193) * fix(tracking): follow actual pair output roles and prefer execution evidence (#193) * docs(pairs): document live validation and retain renderer screenshots (#193) * test(api): inject disposable databases before API startup (#194) * test(api): canonicalize portable temporary gallery roots (#194) * docs(ui): prototype branching negative variant history (#193) * docs(ui): show selected negative beneath positive in timeline concept (#193) * docs: retain approved timeline for later gallery redesign (#193) * test(db): close restore fixture connections before cleanup (#194) * test(db): reuse existing handle guards in all pair fixtures (#194) * chore: clarify rollback test and intentional exception handling (#193)	2026-10-17 08:00:42 -07:00
Vito	6d9f3fc8fb	fix: preserve gallery image identity and ignore stale metadata (#175) ... * fix: preserve gallery root identity across media and metadata Fixes #173 * fix: ignore stale gallery metadata responses Fixes #174 * test: use platform-valid gallery filenames on Windows	2026-10-16 05:35:03 -07:00
Vito	7b6430e20a	fix: stop misleading thumbnail prompts on gallery startup (#168) ... * fix: count pending thumbnails accurately across gallery roots * fix: clear thumbnails across all configured gallery roots * fix: log skipped thumbnail directory cleanup	2026-10-16 04:56:07 -07:00
Vito	da322f8c94	Hardening: security fixes, queue-time run counting, background jobs, 85%+ test coverage (#153) ... * chore: valid packaging metadata, flake8 config file, ignore local config with secrets pyproject.toml carried requirements.txt comment lines as dependencies, which fails PEP 508 parsing. flake8 never read [tool.flake8] from pyproject, so the settings move to .flake8 and pre-commit runs the full rule set. config.json (holds the CivitAI API key and machine paths) and other local artifacts are now ignored. coverage joins the dev requirements. * ci: test on Linux, Windows and macOS with a coverage gate and lint job * refactor: drop validate_workflow_name, the column it validated no longer exists * chore: keep agent worktrees out of coverage measurement * fix: image routes serve only media files inside allowed directories serve_image and serve_output_image now reject anything whose extension is not in IMAGE_EXTENSIONS, fail closed when no output directory is configured, and answer 403 (not 404) for dot-dot, absolute and symlink-escaping paths. Containment is checked after realpath+normcase so it behaves the same on Windows, macOS and Linux. 500 responses use safe_error_message so OSError text no longer leaks absolute paths. * fix: bound page sizes on list endpoints /search, /recent and /images/recent share parse_page_params: limit is clamped to [1, MAX_PAGE_LIMIT=500], offset to >= 0, and non-integer values answer 400 instead of 500. /search now honours offset and all three return a pagination block with the effective limit and offset. * perf: thumbnail generation no longer blocks the event loop generate_thumbnails_with_progress is split into _thumbnail_targets (scan, executor), _generate_one (one PIL/ffmpeg job, executor) and an async driver that awaits each file via _run_in_executor, emits one progress frame per file and yields between files. Per-file failures are reported as file_error events (error stays fatal for the UI) and the loop continues. The wildcard Access-Control-Allow-Origin header is gone. The blocking POST variant reuses the same helpers. The scan never follows symlinks and is capped by MAX_SCAN_DEPTH / MAX_THUMBNAIL_FILES. * fix: cap every request-driven resource in the image routes - /images/recent, /images/all and /images/output parse limit/offset through parse_page_params on every path (absent, 0, negative, float strings), and the clamped values are what reach the DB. /images/all gets its own MAX_BULK_LIMIT ceiling instead of an unbounded query; /images/recent caps its fetch window at MAX_RECENT_IMAGES_WINDOW. - parse_page_params also clamps offset to MAX_PAGE_OFFSET so values past int64 no longer surface as a 500 from SQLite. - Gallery and thumbnail scans share _iter_media_files: no symlink following, MAX_SCAN_DEPTH, and MAX_GALLERY_FILES / MAX_THUMBNAIL_FILES caps. Thumbnail size comes only from named presets. - JSON bodies on /images/link and /images/generate-thumbnails are read in bounded chunks; over MAX_JSON_BODY_BYTES answers 413, malformed or non-object JSON answers 400. - /images/link only accepts media paths inside the allowed directories (403 otherwise), closing a path-existence oracle. * test: cover image and prompt routes Every route in py/api/images.py and py/api/prompts.py now has a success path and an error path test: save/update/delete/rate/tag/category/bulk endpoints, image list/link/unlink/delete/metadata/scan endpoints, the thumbnail SSE stream, clear-thumbnails, and the video thumbnail chain (fake cv2 success/failure, ffmpeg absent or failing, PIL placeholder). Combined statement+branch coverage: images.py 97.7%, prompts.py 97.4%. * fix: autotag endpoints no longer read arbitrary paths or run on GET * fix: CivitAI key is never returned or sent off-host * fix: autotag/single checks real paths and DB rows against output dirs * fix: CivitAI downloads never follow redirects * refactor: autotag model loading is injectable * test: cover autotag and lora integration * fix: saving LoRA settings without a key keeps the stored CivitAI key * refactor: share the internal error message in autotag routes * fix: never treat negative-only PromptManager nodes as the run prompt run_prompt_nodes fell back to every PromptManager node whenever no positive node was traced, including graphs that do have a sampler and whose only PromptManager node feeds its negative input. Fall back only when the graph has no recognisable sampler (new has_sampler helper). Samplers named like one are recognised even without a model link. Adversarial graphs (cycles, 10k-node fan-out, non-dict inputs) are covered. * fix: do not follow negative text inputs when tracing the positive prompt _TEXT_KEY matched input names such as text_negative, so a styler node feeding a sampler's positive input pulled its negative PromptManager node into the positive trace. Input names matching (^|_)neg(ative)?($|_) are now skipped (new follows_input helper). * fix: count prompt runs when ComfyUI dequeues the prompt, not when it is submitted The on-prompt hook runs before ComfyUI validates the request, so counting there bumped run_count for prompts that were rejected or deleted from the queue. The hook now only resolves the positive PromptManager texts, mints a UUID4 prompt_id when the client sent none, and stashes the hashes under that id (QueuedRuns, bounded to 1000 entries). PromptQueue.get is wrapped idempotently; when a prompt is dequeued for execution its stashed runs are counted (or remembered as pending first use). Servers without a prompt_queue fall back to counting at submit time with one warning. __init__ defines _global_db before the monitoring block and passes a factory that raises RuntimeError while the database is unavailable; the package is now imported under test with server and folder_paths stubbed. * fix: keep batch image queue aligned when unrelated workflows save images The metadata strategy returned None for images whose embedded workflow has no positive PromptManager node, so the handler popped the batch FIFO for them and every later batch image linked to the wrong prompt. It now returns a tri-state: linked (prompt found), foreign (no PromptManager prompt in the workflow, so the image is skipped and no fallback applies) or unknown (only then the queue, snapshot, tracker and recent-prompt fallbacks run). * fix: image monitor validates directories before creating the observer start_monitoring assigned self.observer before checking that any directory existed, so with no valid directory the monitor was stuck "already running" and stop_monitoring raised on join of a never-started observer. Directories are validated first and only a started observer is kept; stop_monitoring is safe to call at any time and always joins. New is_monitoring property. The per-image threading.Timer is replaced by one daemon worker: paths are deduplicated and bounded (MAX_PENDING_IMAGES), processed in creation order (batch images pop the prompt queue in order) after the processing delay and once the file size is unchanged across two reads. * refactor: one prompt hash implementation utils.hashing.generate_prompt_hash is now the only prompt hash: PromptManagerBase._generate_hash delegates to it and the inline hashlib.sha256 copies in the prompt tracker and image monitor are gone. Remove PromptManagerBase._open_web_interface (it targeted a web_interface folder that never existed) and the unused restart_gallery.py script. IS_CHANGED on PromptManager and PromptManagerText now includes category and tags, so editing either re-runs the node and updates the saved metadata. * fix: stop ComfyUI from loading the admin bundle into the canvas page WEB_DIRECTORY = "web" made ComfyUI inject every .js under web/ (admin.js, gallery.js, vendored Tailwind and ViewerJS) into the canvas page. The node extension moves to web/comfy/prompt_manager.js (content unchanged; its ../../scripts/app.js import resolves against the served URL, which is the same) and WEB_DIRECTORY points at that folder only. Admin and gallery assets are still served from web/ by the extension's own routes. tests/test_sort_contract.py still reads web/prompt_manager.js and must be updated to the new path by its owner. * fix: image monitor recent-prompt fallback reads the paginated result get_recent_prompts returns {"prompts": [...]} but _get_fallback_prompt indexed it like a list, so the last-resort link to the most recent prompt always failed silently. Accept both shapes. * test: cover prompt_manager_base, prompt_manager and prompt_search_list Node base: init wiring, metadata update on re-save, run counting, database error paths, tag parsing, search and recent API wrappers, gallery start failure, LoRA trigger-word injection (disabled, missing modules, enabled). PromptManager: missing CLIP error, encoding of the combined text with a fake CLIP, tracking failure tolerance, empty text. PromptManagerText: tracking failure tolerance. PromptSearchList: filters, multipart and LoRA-only skipping, newline collapsing, preview, empty and error results. * fix: diagnostics dependency check survives a watchdog without __version__ check_dependencies read watchdog.__version__ for its log line; recent watchdog releases have no such attribute, so the AttributeError escaped and took the whole diagnostics run down. Read it with a default. Also covers utils/diagnostics.py: database and images-table checks against a real temporary database, file system check, ComfyUI output detection (folder_paths, configured directories, relative fallback, nothing found), dependencies, the full run and the test image link. * test: cover utils/logging_config Isolated logger manager on a temporary log directory: memory buffer order, level filter and 1000-entry cap, rotation and backup listing, tail and whole file reads, path traversal rejection, truncation (including removal errors), level changes across prompt_manager loggers, handler rebuilds. * test: cover utils/comfyui_integration Singleton re-initialisation keeps registrations, latest prompt per thread, cross-thread fallback with the five-minute recency limit, global lookup for the current thread, and cleanup of old registrations. * test: cover __init__ Version parsing from pyproject (and the unknown fallback), and a server whose add_on_prompt_handler raises: nodes and monitoring still come up. * docs: point at the moved canvas extension under web/comfy * fix: give every thread its own sqlite connection PromptModel.get_connection() handed every thread the same check_same_thread=False connection, so a commit or rollback from the watchdog, executor or queue-hook thread could land on another thread's half-done write (sqlite3 raised 'bad parameter or other API misuse' under concurrent saves). Connections now live in threading.local, each with WAL, Row factory, foreign_keys and busy_timeout, and are tracked so they can be closed. PromptModel.close() / PromptDatabase.close() close the calling thread's connection so tests and restores can delete the file on Windows. * fix: saving an existing prompt concurrently returns its id Node execution, the queue hook and the image monitor all do get_prompt_by_hash() then save_prompt(). When two of them race on the same text the second INSERT failed on UNIQUE(hash) and surfaced an IntegrityError. save_prompt now catches it, re-fetches by hash and returns the existing id, so eight threads saving one prompt end with a single row and every record_prompt_use counted. * fix: backups include WAL data and restores are integrity-checked The database runs in WAL mode, so a shutil.copy2 of prompts.db silently dropped every committed row still in prompts.db-wal; the backup download and the pre-restore safety copy both did that. Restore also overwrote the live file with connections open, read the whole upload into memory before checking the 100 MB cap and never ran PRAGMA integrity_check. PromptModel.backup_database now uses the SQLite online backup API, verify_database_file opens a candidate read-only and checks header, integrity and the prompts table, and restore_from_file backs up the live database, closes every thread's connection, copies the file, removes stale -wal/-shm and re-runs migrations. The admin handlers stream the upload to a temp file with an early size cutoff, reject anything verify_database_file refuses with a 400 and keep the {success: ...} JSON shape. The schema-init connection is now closed instead of leaking. * fix: migrations are idempotent and schema init runs once per process Every PromptDatabase() (nodes, image monitor, API, diagnostics) re-ran table creation and all migrations. Schema init is now cached per normalised path (class-level set under a lock) and re-run only when the file disappeared, after a restore, or after reset_schema_cache(). The copy-and-swap migrations drop a stale *_new table left by an interrupted run, execute inside an explicit transaction with PRAGMA foreign_keys = OFF (DROP TABLE prompts used to cascade-delete every generated_images row), roll back cleanly and log and re-raise sqlite3.Error instead of swallowing it. * fix: image uniqueness by full path, escaped LIKE search, no usage stamp on save generated_images was UNIQUE(prompt_id, filename) on the basename, so ComfyUI's same-named files in different date folders were silently ignored by INSERT OR IGNORE, and the lastrowid == 0 branch after it never ran. The table now carries file_path (normcase/normpath/abspath of the image path) with UNIQUE(prompt_id, file_path); the migration collapses rows that normalise to one file and supersedes the old basename migration, which would otherwise undo it on every start. search_prompts and the tag search escaped nothing, so 100% matched '100 percent' and a_b matched aXb; user text now goes through escape_like with ESCAPE. save_prompt no longer stamps last_used_at, the usage backfill runs only on the upgrade that adds the columns, and the Recently Used / Most Used orders put never-run prompts (NULL) after run ones instead of first. * fix: escape LIKE wildcards in image prompt search search_images_by_prompt interpolated the user's text into LIKE unescaped, so '50%' also matched '50 percent'. It now goes through the same escape_like helper as search_prompts. Also drops an unused import and an unused cursor assignment flagged by pyflakes. * test: cover database operations Behaviour tests for export (json/csv/errors), duplicate detection and merging, image metadata storage and lookup, missing-file cleanup, statistics, bulk operations, orphan pruning, consistency checks, subfolder listing, tag queries and management edge cases, legacy tag column parsing, config-driven path resolution, model housekeeping and the TEXT -> INTEGER prompt_id migration. database/operations.py and database/models.py are both above 85% statement+branch coverage. * test: isolate the clock and log buffer from background threads of other tests * fix: gallery roots are restricted to ComfyUI directories A gallery root could be set to '/' (or 'C:\') through POST /prompt_manager/settings because the deny list only matched a few system prefixes. The root was persisted to config.json and fed into the image-serving containment check, which then allowed any file on disk to be read through /prompt_manager/images/serve/. - GalleryConfig.validate_gallery_root() accepts a directory only when its realpath lies inside a ComfyUI directory (folder_paths output/input/ temp/user) or a directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS; filesystem roots, the home directory, files and missing paths are rejected. - save_settings returns 400 and persists nothing when any root fails. - _get_all_output_dirs() filters configured roots through the validator and never falls back to a rejected root. - PROMPT_MANAGER_CONFIG_PATH relocates config.json; load/save default to it. * fix: log config accepts only known keys POST /prompt_manager/logs/config forwarded the whole request body into PromptManagerLogger.update_config(), so any key could be injected into the logging configuration. The handler now accepts only LOG_CONFIG_KEYS (level, max_file_size, backup_count, console_logging, file_logging, buffer_size), type-checks each value (level name, non-negative int, bool) and answers 400 listing unknown keys. Malformed or non-object bodies are 400 instead of 500. * fix: no absolute server paths in API responses Settings, diagnostics, duplicate/output scans and the logging endpoints returned absolute filesystem paths, and several error envelopes echoed str(e) which for OSError carries the full path. - _public_path() renders a path relative to the ComfyUI base directory (folder_paths.base_path, else the output dir's parent) or to the parent of an opted-in extra gallery root, falling back to the basename; _public_error() reduces OSError to strerror + basename. - GET /settings returns gallery roots and monitored directories in that form; POST /settings resolves relative roots against the same anchors (GalleryConfig.resolve_gallery_root) so the GET output round-trips. - Diagnostics output_dirs, monitored_directories and the missing-DB message no longer carry absolute paths; the image monitor is reached through the package-or-top-level import fallback instead of a three-dot relative import that failed outside the package. - Duplicate scan 'path' is now relative to the output root and the delete endpoint accepts relative paths. - /logs/files 'path' and /logs/stats 'log_directory' use the public form; unexpected errors answer with a generic message. * perf: run duplicate and output scans off the event loop The duplicate scan walked the output tree and hashed every file on the aiohttp event loop, and the duplicate delete removed files inline; the output scan ran its per-file database work one executor hop at a time. - _find_duplicate_images_sync / _describe_media_file do the walk and hashing; find_duplicate_images() runs them via _run_in_executor. - _delete_duplicate_images_sync / _delete_one_duplicate do the removal (absolute or output-relative paths) off the loop. - The output scan is split into _collect_output_media_sync, _extract_batch_metadata_sync and _ingest_scan_batch_sync (one executor call per batch for all DB work) with a small SSE generator. - Media collection is shared and de-duplicates by normcase'd path. * refactor: settings reads the image monitor through its module get_settings located the ImageMonitor singleton by scanning sys.modules for any module with 'image_monitor' in its name, a workaround for the package being importable under two identities. It now uses the same try-package / except-top-level import as the rest of the API (_current_image_monitor) and falls back to the configured roots when no monitor is running. * fix: clamp the logs limit and reject non-integers GET /prompt_manager/logs accepted any limit up to 1000 and answered 500 when the value was not an integer. The limit is now clamped to [1, 500] and a non-integer value is a 400. * test: cover core API module and gzip middleware * refactor: split diagnostics, maintenance and HTML serving into helpers - run_diagnostics composes four module-level probes (_diagnose_database, _diagnose_dependencies, _diagnose_output_dirs, _diagnose_image_monitor). - run_maintenance is table-driven: _maintenance_runners maps each operation name to a callable and _run_maintenance_operations reports per-operation success or a public error message. - _read_json answers 400 for malformed or non-object JSON bodies in settings, duplicate delete, test-link and maintenance instead of 500. - test_image_link delegates to _link_test_image. - The four HTML page routes share _serve_html_page. * test: cover admin and core API routes Route tests for stats, cleanup, duplicate scan/delete, settings, diagnostics, test-link, maintenance (every operation, success and failure), the output scan internals, and contract-level tests for backup/restore (status codes and envelope only). * test: make admin, logging and diagnostics tests independent of test order The diagnostics message no longer carries the absolute path, the restore test closes its raw sqlite connections before the live file is replaced, the logs tests declare their own folder_paths base directory, and the isolated logger filters every handler to the test thread. * fix: truncating logs no longer rolls the main log over or pads it with zeros truncate_logs emptied prompt_manager.log through a second open(), leaving the rotating handler's stream at its old offset: the next record was written past the new end (zero padding on POSIX) and tell() still reported the old size, so the file rolled over and resurrected the backup that had just been deleted. It also logged once per file from inside the deletion loop, which could do the same with a small size limit. The main log is now truncated through the handler's own stream and a single summary line is logged afterwards. * fix: HTML escaping covers attribute context escapeHtml now escapes & < > " ' (shared web/js/html-escape.js, loaded by admin, gallery and metadata pages) instead of the textContent/innerHTML round-trip that left quotes intact, so a tag, category, checkpoint or file name containing a quote can no longer break out of a data-*, value or title attribute. Inline onclick handlers that interpolated server data now carry data-action and data-* attributes and are dispatched through one delegated listener per page (web/js/data-actions.js); the img onerror fallback uses data-fallback-src with a capturing error listener. Every remaining template value written into an attribute or text node (prompts, checkpoints, file paths, log lines, directory names, categories, suggestions) is escaped. metadata.html no longer builds an "explorer /select" shell command from the file name (Electron-only branch); the path is copied to the clipboard. * refactor: remove shadowed and dead frontend code admin.js defined parsePNGMetadata, extractComfyUIData, showFullPrompt, showWorkflowData, downloadWorkflowJSON and copyAllMetadata twice; the first definitions were unreachable because the later ones shadow them. updateMetadataPanel, showMetadataError, copyPrompt and tryFallbackMetadata only ever wrote to #metadataContent, an element no page creates, so they are gone too. gallery.js defined escapeHtml and showNotification twice. The admin keydown/resize handlers referenced closeImageViewer, previousImage, nextImage and applyImageSizing, none of which exist, and the imageViewMode state they used; only the Escape-closes-gallery branch remains. Debug console.log calls are removed; the ones that were the only statement of a catch block are console.warn so parse failures stay visible. * fix: stale responses, double submits and leaked listeners web/js/api-client.js wraps fetch with one failure path (ApiError for non-2xx, success:false and non-JSON bodies) and latestOnly(), which aborts the previous in-flight request and resolves superseded calls to STALE. The admin search and recent-page loads share one guard, the gallery page load has its own, so fast typing or clicking can no longer paint an older result over a newer one. In-place prompt editing used a blur listener with {once:true} plus a keydown listener added on every edit, which sent the PUT twice and stacked handlers. The list now has one keydown/focusout pair bound at construction and finishEdit() clears the edit state before touching the element, so a save is sent exactly once. Mutations re-fetch the page currently shown (refreshList) instead of calling search(), and delete, bulk delete, edit, restore and scan no longer reload the whole page. Pagination state is committed only after a successful fetch via the pure ListState.nextPageState(current, response) in both admin and gallery, so a failed or aborted page request leaves the current page intact. The gallery lightbox removed its keydown handlers on a non-existent 'remove' event; a MutationObserver now drops them when the modal leaves the document. web/js/sse-stream.js reads server-sent events over fetch so POST-only progress endpoints can stream. * fix: follow the POST-only autotag streams and the new LoRA and list contracts autotag/download and autotag/start are POST-only now; EventSource can only GET, so both pages stream their progress through SseStream.connect(url, {method: 'POST'}), which keeps the onmessage/onerror/close surface the handlers already use. autotag/single is called with image_id and the re-tag dialog shows the server's filename instead of splitting an absolute path. lora/status reports has_civitai_api_key instead of the key: the field shows a 'key is stored' placeholder, an empty field keeps the key on save and a 'Clear the stored key on save' checkbox sends clear_civitai_api_key. Thumbnail generation reports per-file failures as file_error events; both progress views count them (and show error_count from progress) without closing the stream. The auto-tag review follows pagination.has_more on the capped /images/all endpoint, and /search is paged like /recent so paging in search mode no longer switches the list back to recent prompts. * refactor: one PNG metadata parser admin.js, gallery.js and metadata.html each carried a copy of the PNG chunk walker that reads ComfyUI's tEXt/iTXt entries (admin.js had two until the shadowed one was removed). web/js/png-metadata.js is now the only parser: it checks the signature, bounds-checks every chunk so a truncated file or a bogus length cannot throw, decodes tEXt as Latin-1 and iTXt as UTF-8, skips compressed zTXt/iTXt payloads instead of decoding them as garbage, and stops at IEND. The three page-level parsePNGMetadata functions delegate to it. * fix: tags deep link renders and admin modals close admin.html loaded tags-page.js after admin.js, so opening the page at #/tags found no TagsPageManager and rendered nothing; the script order is now theme-bridge, html-escape, api-client, png-metadata, (list-state, sse-stream, data-actions), viewer, comfy-metadata, prompt-list-sort, view-router, tags-page, admin. Route resolution lives in the pure ViewRouter.resolveView(). The auto-tag modal's close buttons called a bare hideModal(), which only gallery.js defines; they call window.admin.hideModal(). PromptAdmin is constructed from startAdmin() on DOMContentLoaded (or immediately when the document is already parsed) instead of at script evaluation time. * test: cover frontend modules browser-globals.test.js loads every page module through vm with a fake window, the way the pages do, and asserts the global each one exposes (HtmlEscape/escapeHtml, DataActions, ApiClient, ListState, SseStream, PngMetadata, ViewRouter, PromptListSort, ComfyMetadata). sse-stream gains cases for a POST body, missing handlers and close() before or during delivery. Every pure module is above 85% line coverage under node --test --experimental-test-coverage. * fix: page image listings in SQL and drop three unused helpers get_recent_images and search_images_by_prompt take limit/offset so the recent and search image routes no longer over-fetch a 5000-row window and slice it. generate_content_hash, PromptModel.migrate_database and find_duplicates had no callers outside their own tests. The isolated logger tests re-apply their thread filter after update_config rebuilds the handlers. Long lines in the touched database and image modules are wrapped so the full flake8 hook passes. * fix: restore through SQLite's backup API, close_all for shared files, safer output scans restore_from_file copied the uploaded file over the live database and deleted its WAL while other PromptModel instances (nodes, image monitor, queue hook) still held connections, which corrupts on POSIX and fails on Windows. It now copies pages through the online backup API into the live connection so every open handle sees the new content. PromptDatabase.close_all() closes the connections of every thread, which tests need before deleting a database. Output scans skip symlinked files, filesystem entries get a stable digest id instead of a per-process hash(), and thumbnail streaming stops when the client disconnects. * style: wrap long lines in node modules Split tooltip, description and docstring strings that exceed 88 columns using implicit concatenation; string values are unchanged. Drop the unused typing imports from prompt_search_list and mark the deliberately late node imports in __init__ with noqa: E402. * style: wrap long lines in the API layer, autotag and config Split long f-string log messages, response messages, URL builders and docstrings; DEFAULT_PROMPT and SYSTEM_PROMPT are rewritten as implicit concatenations with identical values. Drop the unused shutil import in admin and the unused HFCacheInfo import in autotag; keep the hf_hub_download/snapshot_download availability probe with noqa: F401 and mark the deliberately late imports in config with noqa: E402. * style: wrap long lines and drop unused imports in utils Re-wrap module docstrings, comments and f-string log messages; remove the unused typing.List and PngInfo imports; mark the PIL availability probe in diagnostics with noqa: F401; turn two placeholder-less f-strings into plain strings; strip trailing whitespace inside the diagnostics SQL. * style: wrap long lines in tests Split long docstrings and SQL literals (values unchanged) and drop the unused os import from test_path_security. * chore: flake8 ignores the gitignored standalone tagger script * fix: escape image dimensions and workflow JSON before rendering openGalleryModal interpolated width, height and file_size (stored from a POST body, so possibly strings) straight into data-caption and the card text. showWorkflowData wrote JSON.stringify(workflow) raw into a blank same-origin window, so a workflow containing </pre><script> ran. Both now go through ImageHelpers (new pure module, node:test covered); a string file_size no longer reaches formatFileSize's toFixed either, and a blocked popup shows a notice instead of throwing. * fix: escape remaining server values rendered through innerHTML Maintenance and diagnostics results, review-tag errors, video info and metadata error messages, consistency issues, recent image filenames, output directory names, the prompt rating attribute, gallery media types and duplicate group headers were interpolated unescaped. The copy fallback modal also inlined the prompt text inside <textarea>, which a closing tag could break out of; it now sets textarea.value. * fix: address gallery auto-tag review images by path, not digest id The gallery review flow builds its list from /prompt_manager/images/output, whose id is a 16-hex path digest, but posted it as image_id to autotag/single, so the server looked up a database row and every request 404ed. autotagSingleBody (pure, tested) now sends path for output-folder entries and keeps image_id for database rows, which the admin review flow (/images/all) still uses. * fix: read thumbnail generation progress over POST with SseStream images/generate-thumbnails/progress is becoming POST-only, which the browser EventSource cannot request. Both thumbnail progress consumers in the gallery now use SseStream.connect(url, {method: 'POST'}) exactly as the autotag streams do. SseStream gains addEventListener/removeEventListener for named event: frames (tested) so the status, start, progress, file_error, complete and error listeners stay as they were, and an onclose guard rejects when the stream ends without a terminal frame. * fix: drop the unused copy-text data-action handler Nothing in the admin markup emits data-action="copy-text"; the prompt copy buttons call copyToClipboard directly. * fix(lora): confine LoraManager paths to custom_nodes and hide server paths - is_civitai_url also refuses URLs with userinfo or a non-443 port, so https://evil.com\@civitai.com/x and https://civitai.com:8443/x are rejected. - resolve_lora_manager_path / custom_nodes_directories: lora/enable and lora/scan only accept a LoraManager directory that is a direct child of ComfyUI's custom_nodes (folder_paths base_path or registered custom_nodes folders, else this package's parent) and whose name contains "lora", compared via realpath+normcase; relative public paths resolve against the API path anchors. Anything else is 400, so find_lora_directories can no longer widen the image allow-list with arbitrary trees. - lora/enable persists through PromptManagerConfig.save_to_file(), which honours PROMPT_MANAGER_CONFIG_PATH, instead of a hard-coded config.json. - detect/status/enable return path and detected_path through _public_path; every lora handler reports exceptions via _public_error, and lora/scan errors raised after the SSE stream opened become an error event. - Tests close the database before unlinking it. * fix(autotag): contain batch rows, require media extensions, bound the scan - The batch tagger runs every generated_images row through _resolve_allowed_image_path; rows that exist but lie outside the served output directories are skipped and counted in skipped_outside_roots on progress and complete events, and the engine receives the resolved path. - _resolve_allowed_image_path requires a media extension (.png .jpg .jpeg .webp .gif .bmp .tiff .tif), so autotag/single returns 403 for other files. - autotag/start holds a module-level lock; a concurrent start gets 409 {"success": false, "error": "Auto-tagging is already running"}. - scan_output_dir walks with a 50,000-file and 12-level cap, skips symlinked files and thumbnails, returns path through _public_path, relative_path in POSIX form, and a truncated flag. - Autotag handlers report exceptions via the shared _public_error; the local copy is removed. Tests close the database before unlinking it. * test: close every database before unlinking and anchor public paths in image tests Each owned test that opens a PromptDatabase now calls close_all() before removing the file, and temp directories holding a database use ignore_cleanup_errors, so the suite tears down cleanly on Windows. The image API fixture lays out a fake ComfyUI tree (<tmp>/output) and pins GalleryConfig.path_anchors to it so responses can be checked for root-relative paths. * fix: validate image file_info before it reaches the generated_images columns link_image_to_prompt stored size, dimensions and format straight from the request body, so a string such as '"><img src=x>' ended up in INTEGER columns and was echoed back to the gallery. Size and dimensions are now coerced with int() and bounded (size 0..10^12, width/height 0..65535), format is matched against a whitelist of media formats, and anything else is stored as NULL while the image still links. Non-dict metadata or file_info no longer aborts the link. * fix: clamp paging on tag listings and the page-derived recent offset tags/stats, tags/{tag}/prompts and tags/filter parsed limit/offset with a bare int(), so limit=999999 was passed straight to SQLite and a huge offset raised OverflowError (500). They now share parse_page_params (limit 1..500, offset 0..MAX_PAGE_OFFSET, 400 on non-integers), and /recent clamps the offset it derives from ?page= the same way. * fix: make the thumbnail progress stream POST-only GET /prompt_manager/images/generate-thumbnails/progress wrote thumbnails to disk, so any prefetch, link preview or <img src> could start a full output-directory scan. The route now accepts POST only (GET answers 405); quality still comes from the query string so the SSE client can keep the same URL shape. * fix: describe images relative to the ComfyUI tree in API responses Gallery and prompt responses carried absolute server paths: /images/output entries (path, root_dir), thumbnails_path, the images/prompt lookup, every image list (image_path plus its normalised file_path duplicate) and the preview images attached to prompt listings. They now go through _public_path: /images/output 'path' is relative to its root (what the delete and autotag endpoints resolve against), 'root_dir' and every other image_path are relative to the ComfyUI base, and file_path is dropped. recent/all/search image lists also gain url/thumbnail_url via the same enrichment the prompt-images route already had. Remaining str(e) error bodies in both modules use the basename-only error text. * fix: check thumbnail directory containment with a real path test clear_thumbnails compared resolved paths with str.startswith, so a thumbnails directory that resolved to a sibling with a shared prefix (out vs out2) passed the safety check and was emptied. It now uses the module's _is_within (realpath + normcase + Path.is_relative_to). * fix: decode tag names from the URL only once The tag lookup, rename and delete routes ran urllib.parse.unquote on match_info values that aiohttp had already percent-decoded, so a tag literally named '%41' (requested as %2541) was handled as 'A'. The second decode is dropped. * fix: validate tags on every tag mutation route add_prompt_tag, add_tags_to_prompt, bulk_add_tags, merge_tags and rename_tag accepted whatever the body carried; a non-string tag raised AttributeError (500) and over-long or control-character tags were stored. They now run utils.validators.validate_tags through a shared tags_error helper and answer 400 with the validator's message. Blank entries in add_tags_to_prompt are still skipped rather than rejected. * fix: export every prompt by paging instead of truncating at 10000 /prompt_manager/export fetched a single page of 10000 prompts and silently dropped the rest. It now walks the table in EXPORT_PAGE_SIZE pages (1000) inside one executor job until a short page is returned. * fix: gate the LoraManager serving roots and accept relative link paths A LoraManager path hand-edited into config.json went straight into find_lora_directories and so into the image-serving allow-list. When a path is configured it must now pass lora_utils.resolve_lora_manager_path (a custom_nodes child whose name contains 'lora'); otherwise, or when the resolver is unavailable, no LoRA directories are served. An empty path keeps the custom_nodes-only auto-detection. POST /images/link now also accepts the relative forms the API hands out (root-relative 'sub/a.png' and ComfyUI-relative 'output/sub/a.png'), resolving them against the allowed roots and anchors; '..' segments in a relative path are refused and unknown files are 404. * fix: review flow uses the served image url and the lora-dirs test pins its config Image paths in API responses are now relative to the ComfyUI tree, so the auto-tag review page can no longer derive a serve URL from image_path; the server always provides url. The lora-dirs serving test assumed an empty LoraManager path, which only held in a checkout without a local config.json. * test: stop leaking a mocked utils.comfyui_integration and mock the IS_CHANGED clock test_image_monitor_singleton replaced utils.comfyui_integration in sys.modules with a MagicMock and never restored it, so running it before test_comfyui_integration failed eight tests. The mock was unused; drop it and install the server stub with setdefault in every owned test module. The PromptSearchList IS_CHANGED test slept 10 ms between two time.time() reads; the Windows clock ticks every 15.6 ms, so it was flaky there. Mock the clock with an incrementing counter instead. * fix: resolve_text follows a node shared by two branches The cycle guard used one visited set for the whole walk, so a string node read twice (a diamond, or StringConcatenate with the same source on both inputs) made the second branch None and the prompt text unknowable. Cycle detection is now scoped to the current path; a separate visit counter keeps MAX_VISITS as the bound on total work. * fix: bound resolve_text on diamond chains with memoisation, depth and length caps A per-path visited set alone makes a chain of diamonds (node i reads node i+1 on both inputs) exponential in depth. Each node's text is now memoised so it is expanded once, every expansion is charged to the shared MAX_VISITS budget, chains deeper than MAX_DEPTH (64) are unknowable so recursion stays bounded, and text longer than MAX_TEXT_LENGTH (100k) is unknowable so StringConcatenate doubling cannot exhaust memory. * fix: static routes resolve under web/ with an extension allow-list /prompt_manager/lib/{filepath} and /js/{filepath} only rejected '..' and a leading '/'. On Windows os.path.join(root, 'C:/x') or a leading backslash re-anchors the path, so any file on the host could be read. Requests are now refused when the path is absolute on any platform (slash, backslash, drive letter, UNC prefix), carries an empty, '.' or '..' segment, has an extension outside the allow-list, or resolves (symlinks included) outside the static directory; the resolved real path is served with its MIME type. * feat: capped JSON body reader and single-flight lock for long admin jobs Add _read_json_body(request, max_bytes=1_000_000) to py/api: reads the body in chunks, answers 413 past the cap (declared or streamed) and 400 for invalid or non-object JSON. Every JSON POST in the admin and logging handlers goes through it; it is exposed as PromptManagerAPI._read_json_body for the other route mixins. Add _acquire_long_job(name) / _long_job_busy_response(label) on the API object: one asyncio.Lock per job family, handed out as an async context manager or None when busy, so a handler answers 409 instead of starting a second scan. The duplicate scan and delete share the 'duplicates' family. * fix: harden database restore and keep server paths out of admin responses Restore: verify_database_file and restore_from_file run through _run_in_executor; the upload is also opened read-only with PRAGMA trusted_schema=OFF and refused when sqlite_master lists a trigger or a view (a prompts database has neither, a crafted one could use them to run SQL against the live data); at most MAX_SAFETY_BACKUPS (10) '<db>.backup_*' files are kept next to the live database, oldest first. Responses: backup_created and the maintenance statistics database_path go through _public_path, and backup/restore failure messages through _public_error, so no absolute server path appears in an admin body. * fix: validate hand-edited gallery roots everywhere and bound output scans A root written straight into config.json skipped the containment check the settings endpoint applies. GalleryConfig.update_config (and so load_from_file) now drops entries that fail validate_gallery_root with a warning, _find_comfyui_output_dir validates the configured directory before using it, and ImageMonitor.start_monitoring validates every directory before scheduling the observer. The duplicate cleaner only deletes files with a media extension and never anything under a thumbnails directory. _collect_media_files walks with os.walk(followlinks=False), skips symlinked files and directories, prunes thumbnails, stops MAX_SCAN_DEPTH (12) levels down and after MAX_SCAN_FILES (50000) files. * fix: bound log file size and backup count in the logging config endpoint max_file_size must be between 1 MB and 1024 MB (bytes) and backup_count between 0 and 50; anything else is a 400. Also close every database connection in the logging test teardown. * fix: private config.json, merged settings save, synthetic test-link path PromptManagerConfig.save_to_file and the settings endpoint write config.json through write_private_json: created with mode 0600 (chmod on POSIX, best effort on Windows) because the file can hold an API key. _persist_settings merges web_ui and gallery.monitoring.directories into the existing file instead of overwriting it, so the integrations section and any other key survive a settings save. The test-link diagnostic no longer stores a client-supplied path; it records a marker file name under the output directory. * fix: image worker thread exits on shutdown and restarts cleanly ImageGenerationHandler.shutdown() clears the pending set, sets a stop event the worker's delay and settle waits listen to, and puts a None sentinel on the queue; the worker exits within the current wait. Each worker owns its queue and stop event so a draining worker never swallows an image scheduled after shutdown. ImageMonitor.stop_monitoring calls it. * test: close all database connections before deleting temp files Windows refuses to unlink an open SQLite file and executor threads keep their own connections, so every owned test that creates a PromptDatabase calls close_all() before cleanup; the diagnostics temp dir ignores cleanup errors. * fix: output scan runs as a background job that survives the browser going away Closing the dashboard tab or clicking outside the scan modal raised ClientConnectionResetError out of scan_images and abandoned the scan mid-batch, leaving a half-scanned database. Two overlapping scans could also run at once. - py/api/scan_job.py: ScanJob fans progress out to any number of SSE subscribers (latest snapshot first, then live events) - POST /prompt_manager/scan starts the scan as an asyncio task, or attaches to the one already running; a write to a dead socket ends that stream quietly while the scan continues - GET /prompt_manager/scan/status reports the running flag and last event - dashboard: dismissing the modal while a scan runs minimizes it to a progress pill, the page re-attaches to a running scan after a reload, a lost stream reconnects, and leaving the page mid-scan asks first * feat: worker-thread setting speeds up the output scan and thumbnail generation Both jobs processed one file at a time on a single core. Pillow decodes the whole image to reach trailing PNG text chunks and releases the GIL while it does, so a bounded thread pool scales almost linearly: on a 1 GB sample of 400 PNGs the metadata pass dropped from 18 s to 2.5 s at 8 threads, and 120 thumbnails from 4.6 s to 0.6 s. - utils/parallel.py: map_parallel(func, items, workers) keeps order, runs on the caller's thread when workers is 1, and caps the pool at the item count - PromptManagerConfig.WORKER_THREADS defaults to half the detected cores, is clamped to max_worker_threads() when loaded from config.json, and is persisted under "performance" - settings API returns worker_threads and cpu_count and validates saves; the settings modal gets a "Worker threads" slider capped at the core count - the scan reads each metadata batch on the configured threads; thumbnail generation processes batches of files per executor job, still emitting one progress event per file Threads rather than processes: this runs inside ComfyUI's server, where forking a CUDA-initialised process is unsafe and the spawn start method would re-import ComfyUI's main module in every child. * fix: scan modal ignores backdrop clicks so a focus click cannot dismiss it Only its own buttons dismiss it: Cancel before a scan starts, Hide (minimize to the progress pill) while one runs. * feat: scan reads prompts embedded in videos; unreadable files no longer log errors The output scan handed every .mp4 to Pillow and logged one ERROR per file. ComfyUI's video savers store the queued graph as JSON in the container's comment tag, so those files carry prompts worth linking. - utils/video_metadata.py reads the comment/description tag through ffprobe with a timeout, normalised to the PNG text-chunk shape so the existing parser handles both; without ffprobe videos are skipped and a single INFO notice is logged - _extract_comfyui_metadata routes videos there and logs unreadable images as warnings, since the scan carries on past them Test hygiene: every aiohttp test base class overrode tearDownAsync without calling super(), so no test client was ever closed. That produced thousands of ResourceWarnings, "Unclosed client session" errors from asyncio that broke log assertions, and doubled the suite's run time. * fix: thumbnail runs skip fresh thumbnails before the cap and report what remains The per-run cap of 10,000 applied to the directory walk itself, so a library with more files than that never had thumbnails generated for the rest, and a run that found 9,987 existing thumbnails spent its whole budget on skipping them. - _plan_thumbnails walks the whole tree (bounded like the gallery listing), counts files whose thumbnail is already newer as skipped without queuing them, and caps only the pending files; the leftover count is returned - start and complete payloads carry skipped and remaining; total_images counts every file considered and the processing rate uses the files actually worked on - the gallery page shows a warning naming the pending count instead of a success toast when a run stopped at the cap * feat(gallery): warn with the pending count when a thumbnail run stops at the cap * fix: thumbnails are written atomically and served under mtime-versioned urls Cards showed "Failed to load" for thumbnails that were valid on disk. The browser had fetched them while the thumbnail run was still writing the file and, because image responses are cached for an hour, kept the broken bytes. The card also had no fallback to the original. - image and video thumbnails are written to a hidden temp sibling and renamed into place, so a partial file is never visible at the final path - every thumbnail_url (gallery listing, prompt images, duplicates, autotag listing) carries ?v=<mtime>, so a regenerated thumbnail gets a new URL instead of the cached old one - a 404 from the image serve route is marked no-store; ComfyUI's cache middleware would otherwise let browsers cache the miss for an hour - image-helpers gains fallbackImageSource(): the original is tried once before a card gives up Test hygiene: the autotag tests now stub ComfyUI's server module like the other API test modules, so the module passes when run on its own. * fix(gallery): display settings apply at load, columns work, media info toggles Default View Mode, Items per Page and Grid Columns were only applied when Save was pressed, never when the page loaded; the column setting swapped a Tailwind class that was never compiled (only 3, 6 and 8 exist) and a wider breakpoint class overrode it anyway; Show Media Info was saved and never read. - gallery-settings.js validates what localStorage holds (view mode, limit option, column range, booleans) and falls back to defaults per value - settings are applied in the constructor before the first load; changing the page size reloads, toggling media info re-renders - the column count sets an explicit grid template through a CSS variable that wins on desktop widths - media info lines honour the toggle in grid and list views - image load failures are handled by one delegated listener that tries the original before showing the failed-to-load card * feat(gallery): duplicate scan runs as a background job with progress; infinite scrolling The duplicate scan was one blocking request with a spinner: no progress, a modal that could not be put away, and a result lost if the tab closed. - POST /prompt_manager/scan_duplicates/stream starts the scan as a background job (same ScanJob fan-out as the output scan) or attaches to the running one; GET /prompt_manager/scan_duplicates/status reports it. The job holds the duplicates lock, so deletes are refused while it runs. - hashing runs on the configured worker threads and reports progress about every percent - the gallery modal shows a progress bar, closing it minimizes to a pill that reopens it, results wait in the pill if the modal was closed, and a reload re-attaches to a running scan Infinite scrolling is a new Display Setting (off by default): the next page is appended when the viewport nears the bottom, pagination controls hide, and the load-more rule lives in list-state.js where it is unit tested. Thumb-size buttons and the column setting no longer override each other. * fix(gallery): Rescan Output Folder drops the server listing cache and rewalks The button only re-requested the listing, which the server answers from a 30-second in-memory cache, so a rescan showed exactly what was already on screen. - POST /prompt_manager/gallery/rescan invalidates the gallery cache, rewalks every output root and returns the file and root counts - the button calls it, closes the settings modal, reloads the folder tree and the first page, and reports what it found * feat: tag search box autocompletes, narrowed by the tags already typed - PromptDatabase.suggest_tags(prefix, with_tags, limit): tags starting with the prefix, counted over the prompts that carry every context tag, context tags excluded, ordered by count then name; LIKE wildcards in the prefix are escaped - GET /prompt_manager/tags/suggest?q=&with=&limit= exposes it (limit clamped to 1..50, context capped at 10 tags, registered before the {tag_name} routes) - tag-autocomplete.js splits the comma-separated input into committed tags and the one being typed, builds the query, applies a pick and handles wrap-around keyboard selection - the dashboard Tags box shows suggestions with counts on focus and while typing, arrow keys and Enter/Tab pick, Escape closes, and each pick immediately offers the next tag narrowed by the previous ones * feat(dashboard): infinite scrolling for the prompt list, saved default plus session toggle - PromptManagerConfig.INFINITE_SCROLL (web_ui.infinite_scroll) is reported and saved through the settings API; only booleans are accepted - Settings modal gains an "Infinite Scrolling" default; saving applies it at once - the results header gains an "Infinite scroll" checkbox that overrides the default for the current session without saving - when on, the next page is appended as the list nears its bottom (same load-more rule as the gallery), page controls hide, a short page keeps loading until the list fills, and a refresh after an edit restarts from the top * fix: first CI run on Windows and macOS; tests now enforce closed database files The first push of the branch ran the Tests matrix on all three operating systems. Every Python job failed; the causes were: - numpy missing in CI: the WD14 engine tests import it (requirements-dev) - macOS: temp directories resolve through /private; the admin API tests now use the real path like the gallery roots they compare against - Windows: the startup banner printed an emoji, which raises UnicodeEncodeError on a legacy-code-page console and aborted loading the whole node; the banner is ASCII now and a test loads the package under a cp1252 stdout - Windows: gallery subfolders came back with backslashes, which neither the folder tree nor the search filter understand; they are "/" joined on every platform. A relative DEFAULT_DB_PATH kept its forward slashes after being joined to the extension root; it is normalised now - Windows: sqlite connections opened with `with sqlite3.connect()` are not closed by the context manager and linger until garbage collection, which holds a file lock there. Production sites (vacuum, diagnostics, restore inspection) close explicitly; a restore that swaps the API's database object closes the previous one; PromptModel keeps a registry so a shutdown or a test can close every instance - Windows: served files were still open when their temp directory was removed (tests now read the body), a log download test compared text mode newlines, two tests compared paths with a hard-coded separator tests/open_handles.py makes the Linux run fail the way Windows would: every database test base checks, right before deleting its files, that nothing in the process still holds them open. * fix: Windows case-folded gallery roots in tests, "/" subfolders everywhere, longer busy timeout Second CI run on Windows: - gallery roots are stored resolved and case-folded there; the admin API tests now compare against that form instead of the mixed-case temp path - the subfolder fallback for images outside every root also uses "/", and the test expects it - a relative diagnostics db path is normalised after joining - "database is locked" under many writers on a slow disk: connections now wait up to 30 seconds (connect timeout and busy_timeout), long enough for a scan or thumbnail run while ComfyUI saves images * fix: serve every gallery video format, validate settings before applying, dedupe output dirs Review findings on #153 (Codex): - the image serve route allowed only three video types while the gallery lists seven, so .avi/.mkv/.m4v/.wmv cards opened to a 403; the serve allow-list now covers every gallery video format - a settings payload with one invalid field still applied the valid ones before returning 400; the whole payload is validated first and nothing changes on failure. webui_display_mode is validated too Third Windows CI run: - the same output directory spelled twice (different case or a link) was listed twice in diagnostics and served roots; both dedupe on the resolved, case-folded path - the root-less subfolder test expects the "/" form Code-quality bot: index definitions are (name, target) pairs instead of implicitly concatenated strings; redundant local asyncio imports and two trivial lambdas in test stubs removed. * docs: version 3.3.0, README changelog and refreshed sections Bump to 3.3.0: new features (worker threads, background scans, tag autocomplete, infinite scrolling, video prompt scanning) and API contract changes (POST-only side-effect routes, relative image paths, paginated search) go beyond a patch release. README: changelog entries for 3.2.4 and 3.3.0, refreshed bulk scanning, gallery settings, search, configuration and testing sections, and the roadmap's recently-completed list.	2026-10-08 18:53:11 -07:00