Review findings on #153 (Codex):
- the image serve route allowed only three video types while the gallery
lists seven, so .avi/.mkv/.m4v/.wmv cards opened to a 403; the serve
allow-list now covers every gallery video format
- a settings payload with one invalid field still applied the valid ones
before returning 400; the whole payload is validated first and nothing
changes on failure. webui_display_mode is validated too
Third Windows CI run:
- the same output directory spelled twice (different case or a link) was
listed twice in diagnostics and served roots; both dedupe on the
resolved, case-folded path
- the root-less subfolder test expects the "/" form
Code-quality bot: index definitions are (name, target) pairs instead of
implicitly concatenated strings; redundant local asyncio imports and
two trivial lambdas in test stubs removed.
The first push of the branch ran the Tests matrix on all three operating
systems. Every Python job failed; the causes were:
- numpy missing in CI: the WD14 engine tests import it (requirements-dev)
- macOS: temp directories resolve through /private; the admin API tests
now use the real path like the gallery roots they compare against
- Windows: the startup banner printed an emoji, which raises
UnicodeEncodeError on a legacy-code-page console and aborted loading
the whole node; the banner is ASCII now and a test loads the package
under a cp1252 stdout
- Windows: gallery subfolders came back with backslashes, which neither
the folder tree nor the search filter understand; they are "/" joined
on every platform. A relative DEFAULT_DB_PATH kept its forward slashes
after being joined to the extension root; it is normalised now
- Windows: sqlite connections opened with `with sqlite3.connect()` are
not closed by the context manager and linger until garbage collection,
which holds a file lock there. Production sites (vacuum, diagnostics,
restore inspection) close explicitly; a restore that swaps the API's
database object closes the previous one; PromptModel keeps a registry
so a shutdown or a test can close every instance
- Windows: served files were still open when their temp directory was
removed (tests now read the body), a log download test compared text
mode newlines, two tests compared paths with a hard-coded separator
tests/open_handles.py makes the Linux run fail the way Windows would:
every database test base checks, right before deleting its files, that
nothing in the process still holds them open.
The button only re-requested the listing, which the server answers from a
30-second in-memory cache, so a rescan showed exactly what was already on
screen.
- POST /prompt_manager/gallery/rescan invalidates the gallery cache,
rewalks every output root and returns the file and root counts
- the button calls it, closes the settings modal, reloads the folder tree
and the first page, and reports what it found
Cards showed "Failed to load" for thumbnails that were valid on disk. The
browser had fetched them while the thumbnail run was still writing the
file and, because image responses are cached for an hour, kept the broken
bytes. The card also had no fallback to the original.
- image and video thumbnails are written to a hidden temp sibling and
renamed into place, so a partial file is never visible at the final path
- every thumbnail_url (gallery listing, prompt images, duplicates, autotag
listing) carries ?v=<mtime>, so a regenerated thumbnail gets a new URL
instead of the cached old one
- a 404 from the image serve route is marked no-store; ComfyUI's cache
middleware would otherwise let browsers cache the miss for an hour
- image-helpers gains fallbackImageSource(): the original is tried once
before a card gives up
Test hygiene: the autotag tests now stub ComfyUI's server module like the
other API test modules, so the module passes when run on its own.
The per-run cap of 10,000 applied to the directory walk itself, so a
library with more files than that never had thumbnails generated for the
rest, and a run that found 9,987 existing thumbnails spent its whole
budget on skipping them.
- _plan_thumbnails walks the whole tree (bounded like the gallery listing),
counts files whose thumbnail is already newer as skipped without queuing
them, and caps only the pending files; the leftover count is returned
- start and complete payloads carry skipped and remaining; total_images
counts every file considered and the processing rate uses the files
actually worked on
- the gallery page shows a warning naming the pending count instead of a
success toast when a run stopped at the cap
The output scan handed every .mp4 to Pillow and logged one ERROR per file.
ComfyUI's video savers store the queued graph as JSON in the container's
comment tag, so those files carry prompts worth linking.
- utils/video_metadata.py reads the comment/description tag through
ffprobe with a timeout, normalised to the PNG text-chunk shape so the
existing parser handles both; without ffprobe videos are skipped and a
single INFO notice is logged
- _extract_comfyui_metadata routes videos there and logs unreadable images
as warnings, since the scan carries on past them
Test hygiene: every aiohttp test base class overrode tearDownAsync without
calling super(), so no test client was ever closed. That produced thousands
of ResourceWarnings, "Unclosed client session" errors from asyncio that
broke log assertions, and doubled the suite's run time.
Both jobs processed one file at a time on a single core. Pillow decodes the
whole image to reach trailing PNG text chunks and releases the GIL while it
does, so a bounded thread pool scales almost linearly: on a 1 GB sample of
400 PNGs the metadata pass dropped from 18 s to 2.5 s at 8 threads, and 120
thumbnails from 4.6 s to 0.6 s.
- utils/parallel.py: map_parallel(func, items, workers) keeps order, runs on
the caller's thread when workers is 1, and caps the pool at the item count
- PromptManagerConfig.WORKER_THREADS defaults to half the detected cores,
is clamped to max_worker_threads() when loaded from config.json, and is
persisted under "performance"
- settings API returns worker_threads and cpu_count and validates saves;
the settings modal gets a "Worker threads" slider capped at the core count
- the scan reads each metadata batch on the configured threads; thumbnail
generation processes batches of files per executor job, still emitting
one progress event per file
Threads rather than processes: this runs inside ComfyUI's server, where
forking a CUDA-initialised process is unsafe and the spawn start method
would re-import ComfyUI's main module in every child.
Image paths in API responses are now relative to the ComfyUI tree, so the
auto-tag review page can no longer derive a serve URL from image_path; the
server always provides url. The lora-dirs serving test assumed an empty
LoraManager path, which only held in a checkout without a local config.json.
A LoraManager path hand-edited into config.json went straight into
find_lora_directories and so into the image-serving allow-list. When a
path is configured it must now pass lora_utils.resolve_lora_manager_path
(a custom_nodes child whose name contains 'lora'); otherwise, or when the
resolver is unavailable, no LoRA directories are served. An empty path
keeps the custom_nodes-only auto-detection.
POST /images/link now also accepts the relative forms the API hands out
(root-relative 'sub/a.png' and ComfyUI-relative 'output/sub/a.png'),
resolving them against the allowed roots and anchors; '..' segments in
a relative path are refused and unknown files are 404.
clear_thumbnails compared resolved paths with str.startswith, so a
thumbnails directory that resolved to a sibling with a shared prefix
(out vs out2) passed the safety check and was emptied. It now uses the
module's _is_within (realpath + normcase + Path.is_relative_to).
Gallery and prompt responses carried absolute server paths: /images/output
entries (path, root_dir), thumbnails_path, the images/prompt lookup, every
image list (image_path plus its normalised file_path duplicate) and the
preview images attached to prompt listings. They now go through
_public_path: /images/output 'path' is relative to its root (what the
delete and autotag endpoints resolve against), 'root_dir' and every
other image_path are relative to the ComfyUI base, and file_path is
dropped. recent/all/search image lists also gain url/thumbnail_url via
the same enrichment the prompt-images route already had. Remaining
str(e) error bodies in both modules use the basename-only error text.
GET /prompt_manager/images/generate-thumbnails/progress wrote thumbnails
to disk, so any prefetch, link preview or <img src> could start a full
output-directory scan. The route now accepts POST only (GET answers
405); quality still comes from the query string so the SSE client can
keep the same URL shape.
Each owned test that opens a PromptDatabase now calls close_all() before
removing the file, and temp directories holding a database use
ignore_cleanup_errors, so the suite tears down cleanly on Windows. The
image API fixture lays out a fake ComfyUI tree (<tmp>/output) and pins
GalleryConfig.path_anchors to it so responses can be checked for
root-relative paths.
restore_from_file copied the uploaded file over the live database and deleted
its WAL while other PromptModel instances (nodes, image monitor, queue hook)
still held connections, which corrupts on POSIX and fails on Windows. It now
copies pages through the online backup API into the live connection so every
open handle sees the new content. PromptDatabase.close_all() closes the
connections of every thread, which tests need before deleting a database.
Output scans skip symlinked files, filesystem entries get a stable digest id
instead of a per-process hash(), and thumbnail streaming stops when the
client disconnects.
get_recent_images and search_images_by_prompt take limit/offset so the recent
and search image routes no longer over-fetch a 5000-row window and slice it.
generate_content_hash, PromptModel.migrate_database and find_duplicates had no
callers outside their own tests. The isolated logger tests re-apply their
thread filter after update_config rebuilds the handlers. Long lines in the
touched database and image modules are wrapped so the full flake8 hook passes.
Every route in py/api/images.py and py/api/prompts.py now has a success
path and an error path test: save/update/delete/rate/tag/category/bulk
endpoints, image list/link/unlink/delete/metadata/scan endpoints, the
thumbnail SSE stream, clear-thumbnails, and the video thumbnail chain
(fake cv2 success/failure, ffmpeg absent or failing, PIL placeholder).
Combined statement+branch coverage: images.py 97.7%, prompts.py 97.4%.
- /images/recent, /images/all and /images/output parse limit/offset
through parse_page_params on every path (absent, 0, negative, float
strings), and the clamped values are what reach the DB. /images/all
gets its own MAX_BULK_LIMIT ceiling instead of an unbounded query;
/images/recent caps its fetch window at MAX_RECENT_IMAGES_WINDOW.
- parse_page_params also clamps offset to MAX_PAGE_OFFSET so values past
int64 no longer surface as a 500 from SQLite.
- Gallery and thumbnail scans share _iter_media_files: no symlink
following, MAX_SCAN_DEPTH, and MAX_GALLERY_FILES / MAX_THUMBNAIL_FILES
caps. Thumbnail size comes only from named presets.
- JSON bodies on /images/link and /images/generate-thumbnails are read
in bounded chunks; over MAX_JSON_BODY_BYTES answers 413, malformed or
non-object JSON answers 400.
- /images/link only accepts media paths inside the allowed directories
(403 otherwise), closing a path-existence oracle.
generate_thumbnails_with_progress is split into _thumbnail_targets
(scan, executor), _generate_one (one PIL/ffmpeg job, executor) and an
async driver that awaits each file via _run_in_executor, emits one
progress frame per file and yields between files. Per-file failures
are reported as file_error events (error stays fatal for the UI) and
the loop continues. The wildcard Access-Control-Allow-Origin header is
gone. The blocking POST variant reuses the same helpers. The scan never
follows symlinks and is capped by MAX_SCAN_DEPTH / MAX_THUMBNAIL_FILES.
/search, /recent and /images/recent share parse_page_params: limit is
clamped to [1, MAX_PAGE_LIMIT=500], offset to >= 0, and non-integer
values answer 400 instead of 500. /search now honours offset and all
three return a pagination block with the effective limit and offset.
serve_image and serve_output_image now reject anything whose extension
is not in IMAGE_EXTENSIONS, fail closed when no output directory is
configured, and answer 403 (not 404) for dot-dot, absolute and
symlink-escaping paths. Containment is checked after realpath+normcase
so it behaves the same on Windows, macOS and Linux. 500 responses use
safe_error_message so OSError text no longer leaks absolute paths.
Vito Sansevero