Files
ComfyAssets-ComfyUI_PromptM…/tests/test_api_admin.py
T
Vito da322f8c94 Hardening: security fixes, queue-time run counting, background jobs, 85%+ test coverage (#153)
* chore: valid packaging metadata, flake8 config file, ignore local config with secrets

pyproject.toml carried requirements.txt comment lines as dependencies, which
fails PEP 508 parsing. flake8 never read [tool.flake8] from pyproject, so the
settings move to .flake8 and pre-commit runs the full rule set. config.json
(holds the CivitAI API key and machine paths) and other local artifacts are
now ignored. coverage joins the dev requirements.

* ci: test on Linux, Windows and macOS with a coverage gate and lint job

* refactor: drop validate_workflow_name, the column it validated no longer exists

* chore: keep agent worktrees out of coverage measurement

* fix: image routes serve only media files inside allowed directories

serve_image and serve_output_image now reject anything whose extension
is not in IMAGE_EXTENSIONS, fail closed when no output directory is
configured, and answer 403 (not 404) for dot-dot, absolute and
symlink-escaping paths. Containment is checked after realpath+normcase
so it behaves the same on Windows, macOS and Linux. 500 responses use
safe_error_message so OSError text no longer leaks absolute paths.

* fix: bound page sizes on list endpoints

/search, /recent and /images/recent share parse_page_params: limit is
clamped to [1, MAX_PAGE_LIMIT=500], offset to >= 0, and non-integer
values answer 400 instead of 500. /search now honours offset and all
three return a pagination block with the effective limit and offset.

* perf: thumbnail generation no longer blocks the event loop

generate_thumbnails_with_progress is split into _thumbnail_targets
(scan, executor), _generate_one (one PIL/ffmpeg job, executor) and an
async driver that awaits each file via _run_in_executor, emits one
progress frame per file and yields between files. Per-file failures
are reported as file_error events (error stays fatal for the UI) and
the loop continues. The wildcard Access-Control-Allow-Origin header is
gone. The blocking POST variant reuses the same helpers. The scan never
follows symlinks and is capped by MAX_SCAN_DEPTH / MAX_THUMBNAIL_FILES.

* fix: cap every request-driven resource in the image routes

- /images/recent, /images/all and /images/output parse limit/offset
  through parse_page_params on every path (absent, 0, negative, float
  strings), and the clamped values are what reach the DB. /images/all
  gets its own MAX_BULK_LIMIT ceiling instead of an unbounded query;
  /images/recent caps its fetch window at MAX_RECENT_IMAGES_WINDOW.
- parse_page_params also clamps offset to MAX_PAGE_OFFSET so values past
  int64 no longer surface as a 500 from SQLite.
- Gallery and thumbnail scans share _iter_media_files: no symlink
  following, MAX_SCAN_DEPTH, and MAX_GALLERY_FILES / MAX_THUMBNAIL_FILES
  caps. Thumbnail size comes only from named presets.
- JSON bodies on /images/link and /images/generate-thumbnails are read
  in bounded chunks; over MAX_JSON_BODY_BYTES answers 413, malformed or
  non-object JSON answers 400.
- /images/link only accepts media paths inside the allowed directories
  (403 otherwise), closing a path-existence oracle.

* test: cover image and prompt routes

Every route in py/api/images.py and py/api/prompts.py now has a success
path and an error path test: save/update/delete/rate/tag/category/bulk
endpoints, image list/link/unlink/delete/metadata/scan endpoints, the
thumbnail SSE stream, clear-thumbnails, and the video thumbnail chain
(fake cv2 success/failure, ffmpeg absent or failing, PIL placeholder).
Combined statement+branch coverage: images.py 97.7%, prompts.py 97.4%.

* fix: autotag endpoints no longer read arbitrary paths or run on GET

* fix: CivitAI key is never returned or sent off-host

* fix: autotag/single checks real paths and DB rows against output dirs

* fix: CivitAI downloads never follow redirects

* refactor: autotag model loading is injectable

* test: cover autotag and lora integration

* fix: saving LoRA settings without a key keeps the stored CivitAI key

* refactor: share the internal error message in autotag routes

* fix: never treat negative-only PromptManager nodes as the run prompt

run_prompt_nodes fell back to every PromptManager node whenever no positive
node was traced, including graphs that do have a sampler and whose only
PromptManager node feeds its negative input. Fall back only when the graph
has no recognisable sampler (new has_sampler helper). Samplers named like
one are recognised even without a model link. Adversarial graphs (cycles,
10k-node fan-out, non-dict inputs) are covered.

* fix: do not follow negative text inputs when tracing the positive prompt

_TEXT_KEY matched input names such as text_negative, so a styler node feeding
a sampler's positive input pulled its negative PromptManager node into the
positive trace. Input names matching (^|_)neg(ative)?($|_) are now skipped
(new follows_input helper).

* fix: count prompt runs when ComfyUI dequeues the prompt, not when it is submitted

The on-prompt hook runs before ComfyUI validates the request, so counting
there bumped run_count for prompts that were rejected or deleted from the
queue. The hook now only resolves the positive PromptManager texts, mints a
UUID4 prompt_id when the client sent none, and stashes the hashes under that
id (QueuedRuns, bounded to 1000 entries). PromptQueue.get is wrapped
idempotently; when a prompt is dequeued for execution its stashed runs are
counted (or remembered as pending first use). Servers without a prompt_queue
fall back to counting at submit time with one warning.

__init__ defines _global_db before the monitoring block and passes a factory
that raises RuntimeError while the database is unavailable; the package is
now imported under test with server and folder_paths stubbed.

* fix: keep batch image queue aligned when unrelated workflows save images

The metadata strategy returned None for images whose embedded workflow has
no positive PromptManager node, so the handler popped the batch FIFO for
them and every later batch image linked to the wrong prompt. It now returns
a tri-state: linked (prompt found), foreign (no PromptManager prompt in the
workflow, so the image is skipped and no fallback applies) or unknown (only
then the queue, snapshot, tracker and recent-prompt fallbacks run).

* fix: image monitor validates directories before creating the observer

start_monitoring assigned self.observer before checking that any directory
existed, so with no valid directory the monitor was stuck "already running"
and stop_monitoring raised on join of a never-started observer. Directories
are validated first and only a started observer is kept; stop_monitoring is
safe to call at any time and always joins. New is_monitoring property.

The per-image threading.Timer is replaced by one daemon worker: paths are
deduplicated and bounded (MAX_PENDING_IMAGES), processed in creation order
(batch images pop the prompt queue in order) after the processing delay and
once the file size is unchanged across two reads.

* refactor: one prompt hash implementation

utils.hashing.generate_prompt_hash is now the only prompt hash:
PromptManagerBase._generate_hash delegates to it and the inline
hashlib.sha256 copies in the prompt tracker and image monitor are gone.

Remove PromptManagerBase._open_web_interface (it targeted a web_interface
folder that never existed) and the unused restart_gallery.py script.

IS_CHANGED on PromptManager and PromptManagerText now includes category and
tags, so editing either re-runs the node and updates the saved metadata.

* fix: stop ComfyUI from loading the admin bundle into the canvas page

WEB_DIRECTORY = "web" made ComfyUI inject every .js under web/ (admin.js,
gallery.js, vendored Tailwind and ViewerJS) into the canvas page. The node
extension moves to web/comfy/prompt_manager.js (content unchanged; its
../../scripts/app.js import resolves against the served URL, which is the
same) and WEB_DIRECTORY points at that folder only. Admin and gallery assets
are still served from web/ by the extension's own routes.

tests/test_sort_contract.py still reads web/prompt_manager.js and must be
updated to the new path by its owner.

* fix: image monitor recent-prompt fallback reads the paginated result

get_recent_prompts returns {"prompts": [...]} but _get_fallback_prompt
indexed it like a list, so the last-resort link to the most recent prompt
always failed silently. Accept both shapes.

* test: cover prompt_manager_base, prompt_manager and prompt_search_list

Node base: init wiring, metadata update on re-save, run counting, database
error paths, tag parsing, search and recent API wrappers, gallery start
failure, LoRA trigger-word injection (disabled, missing modules, enabled).
PromptManager: missing CLIP error, encoding of the combined text with a fake
CLIP, tracking failure tolerance, empty text. PromptManagerText: tracking
failure tolerance. PromptSearchList: filters, multipart and LoRA-only
skipping, newline collapsing, preview, empty and error results.

* fix: diagnostics dependency check survives a watchdog without __version__

check_dependencies read watchdog.__version__ for its log line; recent
watchdog releases have no such attribute, so the AttributeError escaped and
took the whole diagnostics run down. Read it with a default.

Also covers utils/diagnostics.py: database and images-table checks against
a real temporary database, file system check, ComfyUI output detection
(folder_paths, configured directories, relative fallback, nothing found),
dependencies, the full run and the test image link.

* test: cover utils/logging_config

Isolated logger manager on a temporary log directory: memory buffer order,
level filter and 1000-entry cap, rotation and backup listing, tail and whole
file reads, path traversal rejection, truncation (including removal
errors), level changes across prompt_manager loggers, handler rebuilds.

* test: cover utils/comfyui_integration

Singleton re-initialisation keeps registrations, latest prompt per thread,
cross-thread fallback with the five-minute recency limit, global lookup for
the current thread, and cleanup of old registrations.

* test: cover __init__

Version parsing from pyproject (and the unknown fallback), and a server
whose add_on_prompt_handler raises: nodes and monitoring still come up.

* docs: point at the moved canvas extension under web/comfy

* fix: give every thread its own sqlite connection

PromptModel.get_connection() handed every thread the same
check_same_thread=False connection, so a commit or rollback from the
watchdog, executor or queue-hook thread could land on another thread's
half-done write (sqlite3 raised 'bad parameter or other API misuse'
under concurrent saves). Connections now live in threading.local, each
with WAL, Row factory, foreign_keys and busy_timeout, and are tracked so
they can be closed. PromptModel.close() / PromptDatabase.close() close
the calling thread's connection so tests and restores can delete the
file on Windows.

* fix: saving an existing prompt concurrently returns its id

Node execution, the queue hook and the image monitor all do
get_prompt_by_hash() then save_prompt(). When two of them race on the
same text the second INSERT failed on UNIQUE(hash) and surfaced an
IntegrityError. save_prompt now catches it, re-fetches by hash and
returns the existing id, so eight threads saving one prompt end with a
single row and every record_prompt_use counted.

* fix: backups include WAL data and restores are integrity-checked

The database runs in WAL mode, so a shutil.copy2 of prompts.db silently
dropped every committed row still in prompts.db-wal; the backup
download and the pre-restore safety copy both did that. Restore also
overwrote the live file with connections open, read the whole upload
into memory before checking the 100 MB cap and never ran
PRAGMA integrity_check.

PromptModel.backup_database now uses the SQLite online backup API,
verify_database_file opens a candidate read-only and checks header,
integrity and the prompts table, and restore_from_file backs up the
live database, closes every thread's connection, copies the file,
removes stale -wal/-shm and re-runs migrations. The admin handlers
stream the upload to a temp file with an early size cutoff, reject
anything verify_database_file refuses with a 400 and keep the
{success: ...} JSON shape. The schema-init connection is now closed
instead of leaking.

* fix: migrations are idempotent and schema init runs once per process

Every PromptDatabase() (nodes, image monitor, API, diagnostics) re-ran
table creation and all migrations. Schema init is now cached per
normalised path (class-level set under a lock) and re-run only when the
file disappeared, after a restore, or after reset_schema_cache().

The copy-and-swap migrations drop a stale *_new table left by an
interrupted run, execute inside an explicit transaction with
PRAGMA foreign_keys = OFF (DROP TABLE prompts used to cascade-delete
every generated_images row), roll back cleanly and log and re-raise
sqlite3.Error instead of swallowing it.

* fix: image uniqueness by full path, escaped LIKE search, no usage stamp on save

generated_images was UNIQUE(prompt_id, filename) on the basename, so
ComfyUI's same-named files in different date folders were silently
ignored by INSERT OR IGNORE, and the lastrowid == 0 branch after it
never ran. The table now carries file_path (normcase/normpath/abspath
of the image path) with UNIQUE(prompt_id, file_path); the migration
collapses rows that normalise to one file and supersedes the old
basename migration, which would otherwise undo it on every start.

search_prompts and the tag search escaped nothing, so 100% matched
'100 percent' and a_b matched aXb; user text now goes through
escape_like with ESCAPE. save_prompt no longer stamps last_used_at, the
usage backfill runs only on the upgrade that adds the columns, and the
Recently Used / Most Used orders put never-run prompts (NULL) after run
ones instead of first.

* fix: escape LIKE wildcards in image prompt search

search_images_by_prompt interpolated the user's text into LIKE
unescaped, so '50%' also matched '50 percent'. It now goes through the
same escape_like helper as search_prompts. Also drops an unused import
and an unused cursor assignment flagged by pyflakes.

* test: cover database operations

Behaviour tests for export (json/csv/errors), duplicate detection and
merging, image metadata storage and lookup, missing-file cleanup,
statistics, bulk operations, orphan pruning, consistency checks,
subfolder listing, tag queries and management edge cases, legacy tag
column parsing, config-driven path resolution, model housekeeping and
the TEXT -> INTEGER prompt_id migration. database/operations.py and
database/models.py are both above 85% statement+branch coverage.

* test: isolate the clock and log buffer from background threads of other tests

* fix: gallery roots are restricted to ComfyUI directories

A gallery root could be set to '/' (or 'C:\') through POST
/prompt_manager/settings because the deny list only matched a few
system prefixes. The root was persisted to config.json and fed into the
image-serving containment check, which then allowed any file on disk to
be read through /prompt_manager/images/serve/.

- GalleryConfig.validate_gallery_root() accepts a directory only when its
  realpath lies inside a ComfyUI directory (folder_paths output/input/
  temp/user) or a directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS;
  filesystem roots, the home directory, files and missing paths are
  rejected.
- save_settings returns 400 and persists nothing when any root fails.
- _get_all_output_dirs() filters configured roots through the validator
  and never falls back to a rejected root.
- PROMPT_MANAGER_CONFIG_PATH relocates config.json; load/save default
  to it.

* fix: log config accepts only known keys

POST /prompt_manager/logs/config forwarded the whole request body into
PromptManagerLogger.update_config(), so any key could be injected into
the logging configuration. The handler now accepts only LOG_CONFIG_KEYS
(level, max_file_size, backup_count, console_logging, file_logging,
buffer_size), type-checks each value (level name, non-negative int,
bool) and answers 400 listing unknown keys. Malformed or non-object
bodies are 400 instead of 500.

* fix: no absolute server paths in API responses

Settings, diagnostics, duplicate/output scans and the logging endpoints
returned absolute filesystem paths, and several error envelopes echoed
str(e) which for OSError carries the full path.

- _public_path() renders a path relative to the ComfyUI base directory
  (folder_paths.base_path, else the output dir's parent) or to the
  parent of an opted-in extra gallery root, falling back to the
  basename; _public_error() reduces OSError to strerror + basename.
- GET /settings returns gallery roots and monitored directories in that
  form; POST /settings resolves relative roots against the same anchors
  (GalleryConfig.resolve_gallery_root) so the GET output round-trips.
- Diagnostics output_dirs, monitored_directories and the missing-DB
  message no longer carry absolute paths; the image monitor is reached
  through the package-or-top-level import fallback instead of a
  three-dot relative import that failed outside the package.
- Duplicate scan 'path' is now relative to the output root and the
  delete endpoint accepts relative paths.
- /logs/files 'path' and /logs/stats 'log_directory' use the public
  form; unexpected errors answer with a generic message.

* perf: run duplicate and output scans off the event loop

The duplicate scan walked the output tree and hashed every file on the
aiohttp event loop, and the duplicate delete removed files inline; the
output scan ran its per-file database work one executor hop at a time.

- _find_duplicate_images_sync / _describe_media_file do the walk and
  hashing; find_duplicate_images() runs them via _run_in_executor.
- _delete_duplicate_images_sync / _delete_one_duplicate do the removal
  (absolute or output-relative paths) off the loop.
- The output scan is split into _collect_output_media_sync,
  _extract_batch_metadata_sync and _ingest_scan_batch_sync (one
  executor call per batch for all DB work) with a small SSE generator.
- Media collection is shared and de-duplicates by normcase'd path.

* refactor: settings reads the image monitor through its module

get_settings located the ImageMonitor singleton by scanning sys.modules
for any module with 'image_monitor' in its name, a workaround for the
package being importable under two identities. It now uses the same
try-package / except-top-level import as the rest of the API
(_current_image_monitor) and falls back to the configured roots when no
monitor is running.

* fix: clamp the logs limit and reject non-integers

GET /prompt_manager/logs accepted any limit up to 1000 and answered 500
when the value was not an integer. The limit is now clamped to [1, 500]
and a non-integer value is a 400.

* test: cover core API module and gzip middleware

* refactor: split diagnostics, maintenance and HTML serving into helpers

- run_diagnostics composes four module-level probes (_diagnose_database,
  _diagnose_dependencies, _diagnose_output_dirs, _diagnose_image_monitor).
- run_maintenance is table-driven: _maintenance_runners maps each
  operation name to a callable and _run_maintenance_operations reports
  per-operation success or a public error message.
- _read_json answers 400 for malformed or non-object JSON bodies in
  settings, duplicate delete, test-link and maintenance instead of 500.
- test_image_link delegates to _link_test_image.
- The four HTML page routes share _serve_html_page.

* test: cover admin and core API routes

Route tests for stats, cleanup, duplicate scan/delete, settings,
diagnostics, test-link, maintenance (every operation, success and
failure), the output scan internals, and contract-level tests for
backup/restore (status codes and envelope only).

* test: make admin, logging and diagnostics tests independent of test order

The diagnostics message no longer carries the absolute path, the restore test
closes its raw sqlite connections before the live file is replaced, the logs
tests declare their own folder_paths base directory, and the isolated logger
filters every handler to the test thread.

* fix: truncating logs no longer rolls the main log over or pads it with zeros

truncate_logs emptied prompt_manager.log through a second open(), leaving the
rotating handler's stream at its old offset: the next record was written past
the new end (zero padding on POSIX) and tell() still reported the old size, so
the file rolled over and resurrected the backup that had just been deleted.
It also logged once per file from inside the deletion loop, which could do the
same with a small size limit. The main log is now truncated through the
handler's own stream and a single summary line is logged afterwards.

* fix: HTML escaping covers attribute context

escapeHtml now escapes & < > " ' (shared web/js/html-escape.js, loaded by
admin, gallery and metadata pages) instead of the textContent/innerHTML
round-trip that left quotes intact, so a tag, category, checkpoint or file
name containing a quote can no longer break out of a data-*, value or title
attribute.

Inline onclick handlers that interpolated server data now carry data-action
and data-* attributes and are dispatched through one delegated listener per
page (web/js/data-actions.js); the img onerror fallback uses
data-fallback-src with a capturing error listener. Every remaining template
value written into an attribute or text node (prompts, checkpoints, file
paths, log lines, directory names, categories, suggestions) is escaped.

metadata.html no longer builds an "explorer /select" shell command from the
file name (Electron-only branch); the path is copied to the clipboard.

* refactor: remove shadowed and dead frontend code

admin.js defined parsePNGMetadata, extractComfyUIData, showFullPrompt,
showWorkflowData, downloadWorkflowJSON and copyAllMetadata twice; the first
definitions were unreachable because the later ones shadow them.
updateMetadataPanel, showMetadataError, copyPrompt and tryFallbackMetadata
only ever wrote to #metadataContent, an element no page creates, so they are
gone too. gallery.js defined escapeHtml and showNotification twice. The admin
keydown/resize handlers referenced closeImageViewer, previousImage, nextImage
and applyImageSizing, none of which exist, and the imageViewMode state they
used; only the Escape-closes-gallery branch remains.

Debug console.log calls are removed; the ones that were the only statement of
a catch block are console.warn so parse failures stay visible.

* fix: stale responses, double submits and leaked listeners

web/js/api-client.js wraps fetch with one failure path (ApiError for non-2xx,
success:false and non-JSON bodies) and latestOnly(), which aborts the previous
in-flight request and resolves superseded calls to STALE. The admin search and
recent-page loads share one guard, the gallery page load has its own, so fast
typing or clicking can no longer paint an older result over a newer one.

In-place prompt editing used a blur listener with {once:true} plus a keydown
listener added on every edit, which sent the PUT twice and stacked handlers.
The list now has one keydown/focusout pair bound at construction and
finishEdit() clears the edit state before touching the element, so a save is
sent exactly once. Mutations re-fetch the page currently shown (refreshList)
instead of calling search(), and delete, bulk delete, edit, restore and scan no
longer reload the whole page.

Pagination state is committed only after a successful fetch via the pure
ListState.nextPageState(current, response) in both admin and gallery, so a
failed or aborted page request leaves the current page intact. The gallery
lightbox removed its keydown handlers on a non-existent 'remove' event; a
MutationObserver now drops them when the modal leaves the document.

web/js/sse-stream.js reads server-sent events over fetch so POST-only progress
endpoints can stream.

* fix: follow the POST-only autotag streams and the new LoRA and list contracts

autotag/download and autotag/start are POST-only now; EventSource can only
GET, so both pages stream their progress through SseStream.connect(url,
{method: 'POST'}), which keeps the onmessage/onerror/close surface the
handlers already use. autotag/single is called with image_id and the re-tag
dialog shows the server's filename instead of splitting an absolute path.

lora/status reports has_civitai_api_key instead of the key: the field shows a
'key is stored' placeholder, an empty field keeps the key on save and a
'Clear the stored key on save' checkbox sends clear_civitai_api_key.

Thumbnail generation reports per-file failures as file_error events; both
progress views count them (and show error_count from progress) without
closing the stream. The auto-tag review follows pagination.has_more on the
capped /images/all endpoint, and /search is paged like /recent so paging in
search mode no longer switches the list back to recent prompts.

* refactor: one PNG metadata parser

admin.js, gallery.js and metadata.html each carried a copy of the PNG chunk
walker that reads ComfyUI's tEXt/iTXt entries (admin.js had two until the
shadowed one was removed). web/js/png-metadata.js is now the only parser:
it checks the signature, bounds-checks every chunk so a truncated file or a
bogus length cannot throw, decodes tEXt as Latin-1 and iTXt as UTF-8, skips
compressed zTXt/iTXt payloads instead of decoding them as garbage, and stops
at IEND. The three page-level parsePNGMetadata functions delegate to it.

* fix: tags deep link renders and admin modals close

admin.html loaded tags-page.js after admin.js, so opening the page at #/tags
found no TagsPageManager and rendered nothing; the script order is now
theme-bridge, html-escape, api-client, png-metadata, (list-state, sse-stream,
data-actions), viewer, comfy-metadata, prompt-list-sort, view-router,
tags-page, admin. Route resolution lives in the pure ViewRouter.resolveView().

The auto-tag modal's close buttons called a bare hideModal(), which only
gallery.js defines; they call window.admin.hideModal(). PromptAdmin is
constructed from startAdmin() on DOMContentLoaded (or immediately when the
document is already parsed) instead of at script evaluation time.

* test: cover frontend modules

browser-globals.test.js loads every page module through vm with a fake
window, the way the pages do, and asserts the global each one exposes
(HtmlEscape/escapeHtml, DataActions, ApiClient, ListState, SseStream,
PngMetadata, ViewRouter, PromptListSort, ComfyMetadata). sse-stream gains
cases for a POST body, missing handlers and close() before or during
delivery. Every pure module is above 85% line coverage under
node --test --experimental-test-coverage.

* fix: page image listings in SQL and drop three unused helpers

get_recent_images and search_images_by_prompt take limit/offset so the recent
and search image routes no longer over-fetch a 5000-row window and slice it.
generate_content_hash, PromptModel.migrate_database and find_duplicates had no
callers outside their own tests. The isolated logger tests re-apply their
thread filter after update_config rebuilds the handlers. Long lines in the
touched database and image modules are wrapped so the full flake8 hook passes.

* fix: restore through SQLite's backup API, close_all for shared files, safer output scans

restore_from_file copied the uploaded file over the live database and deleted
its WAL while other PromptModel instances (nodes, image monitor, queue hook)
still held connections, which corrupts on POSIX and fails on Windows. It now
copies pages through the online backup API into the live connection so every
open handle sees the new content. PromptDatabase.close_all() closes the
connections of every thread, which tests need before deleting a database.
Output scans skip symlinked files, filesystem entries get a stable digest id
instead of a per-process hash(), and thumbnail streaming stops when the
client disconnects.

* style: wrap long lines in node modules

Split tooltip, description and docstring strings that exceed 88 columns
using implicit concatenation; string values are unchanged. Drop the unused
typing imports from prompt_search_list and mark the deliberately late node
imports in __init__ with noqa: E402.

* style: wrap long lines in the API layer, autotag and config

Split long f-string log messages, response messages, URL builders and
docstrings; DEFAULT_PROMPT and SYSTEM_PROMPT are rewritten as implicit
concatenations with identical values. Drop the unused shutil import in
admin and the unused HFCacheInfo import in autotag; keep the
hf_hub_download/snapshot_download availability probe with noqa: F401 and
mark the deliberately late imports in config with noqa: E402.

* style: wrap long lines and drop unused imports in utils

Re-wrap module docstrings, comments and f-string log messages; remove the
unused typing.List and PngInfo imports; mark the PIL availability probe in
diagnostics with noqa: F401; turn two placeholder-less f-strings into plain
strings; strip trailing whitespace inside the diagnostics SQL.

* style: wrap long lines in tests

Split long docstrings and SQL literals (values unchanged) and drop the
unused os import from test_path_security.

* chore: flake8 ignores the gitignored standalone tagger script

* fix: escape image dimensions and workflow JSON before rendering

openGalleryModal interpolated width, height and file_size (stored from a
POST body, so possibly strings) straight into data-caption and the card
text. showWorkflowData wrote JSON.stringify(workflow) raw into a blank
same-origin window, so a workflow containing </pre><script> ran. Both now
go through ImageHelpers (new pure module, node:test covered); a string
file_size no longer reaches formatFileSize's toFixed either, and a blocked
popup shows a notice instead of throwing.

* fix: escape remaining server values rendered through innerHTML

Maintenance and diagnostics results, review-tag errors, video info and
metadata error messages, consistency issues, recent image filenames,
output directory names, the prompt rating attribute, gallery media types
and duplicate group headers were interpolated unescaped. The copy
fallback modal also inlined the prompt text inside <textarea>, which a
closing tag could break out of; it now sets textarea.value.

* fix: address gallery auto-tag review images by path, not digest id

The gallery review flow builds its list from /prompt_manager/images/output,
whose id is a 16-hex path digest, but posted it as image_id to
autotag/single, so the server looked up a database row and every request
404ed. autotagSingleBody (pure, tested) now sends path for output-folder
entries and keeps image_id for database rows, which the admin review flow
(/images/all) still uses.

* fix: read thumbnail generation progress over POST with SseStream

images/generate-thumbnails/progress is becoming POST-only, which the
browser EventSource cannot request. Both thumbnail progress consumers in
the gallery now use SseStream.connect(url, {method: 'POST'}) exactly as
the autotag streams do. SseStream gains addEventListener/removeEventListener
for named event: frames (tested) so the status, start, progress,
file_error, complete and error listeners stay as they were, and an onclose
guard rejects when the stream ends without a terminal frame.

* fix: drop the unused copy-text data-action handler

Nothing in the admin markup emits data-action="copy-text"; the prompt
copy buttons call copyToClipboard directly.

* fix(lora): confine LoraManager paths to custom_nodes and hide server paths

- is_civitai_url also refuses URLs with userinfo or a non-443 port, so
  https://evil.com\@civitai.com/x and https://civitai.com:8443/x are rejected.
- resolve_lora_manager_path / custom_nodes_directories: lora/enable and
  lora/scan only accept a LoraManager directory that is a direct child of
  ComfyUI's custom_nodes (folder_paths base_path or registered custom_nodes
  folders, else this package's parent) and whose name contains "lora",
  compared via realpath+normcase; relative public paths resolve against the
  API path anchors. Anything else is 400, so find_lora_directories can no
  longer widen the image allow-list with arbitrary trees.
- lora/enable persists through PromptManagerConfig.save_to_file(), which
  honours PROMPT_MANAGER_CONFIG_PATH, instead of a hard-coded config.json.
- detect/status/enable return path and detected_path through _public_path;
  every lora handler reports exceptions via _public_error, and lora/scan
  errors raised after the SSE stream opened become an error event.
- Tests close the database before unlinking it.

* fix(autotag): contain batch rows, require media extensions, bound the scan

- The batch tagger runs every generated_images row through
  _resolve_allowed_image_path; rows that exist but lie outside the served
  output directories are skipped and counted in skipped_outside_roots on
  progress and complete events, and the engine receives the resolved path.
- _resolve_allowed_image_path requires a media extension (.png .jpg .jpeg
  .webp .gif .bmp .tiff .tif), so autotag/single returns 403 for other files.
- autotag/start holds a module-level lock; a concurrent start gets 409
  {"success": false, "error": "Auto-tagging is already running"}.
- scan_output_dir walks with a 50,000-file and 12-level cap, skips symlinked
  files and thumbnails, returns path through _public_path, relative_path in
  POSIX form, and a truncated flag.
- Autotag handlers report exceptions via the shared _public_error; the
  local copy is removed. Tests close the database before unlinking it.

* test: close every database before unlinking and anchor public paths in image tests

Each owned test that opens a PromptDatabase now calls close_all() before
removing the file, and temp directories holding a database use
ignore_cleanup_errors, so the suite tears down cleanly on Windows. The
image API fixture lays out a fake ComfyUI tree (<tmp>/output) and pins
GalleryConfig.path_anchors to it so responses can be checked for
root-relative paths.

* fix: validate image file_info before it reaches the generated_images columns

link_image_to_prompt stored size, dimensions and format straight from the
request body, so a string such as '"><img src=x>' ended up in INTEGER
columns and was echoed back to the gallery. Size and dimensions are now
coerced with int() and bounded (size 0..10^12, width/height 0..65535),
format is matched against a whitelist of media formats, and anything
else is stored as NULL while the image still links. Non-dict metadata or
file_info no longer aborts the link.

* fix: clamp paging on tag listings and the page-derived recent offset

tags/stats, tags/{tag}/prompts and tags/filter parsed limit/offset with a
bare int(), so limit=999999 was passed straight to SQLite and a huge
offset raised OverflowError (500). They now share parse_page_params
(limit 1..500, offset 0..MAX_PAGE_OFFSET, 400 on non-integers), and
/recent clamps the offset it derives from ?page= the same way.

* fix: make the thumbnail progress stream POST-only

GET /prompt_manager/images/generate-thumbnails/progress wrote thumbnails
to disk, so any prefetch, link preview or <img src> could start a full
output-directory scan. The route now accepts POST only (GET answers
405); quality still comes from the query string so the SSE client can
keep the same URL shape.

* fix: describe images relative to the ComfyUI tree in API responses

Gallery and prompt responses carried absolute server paths: /images/output
entries (path, root_dir), thumbnails_path, the images/prompt lookup, every
image list (image_path plus its normalised file_path duplicate) and the
preview images attached to prompt listings. They now go through
_public_path: /images/output 'path' is relative to its root (what the
delete and autotag endpoints resolve against), 'root_dir' and every
other image_path are relative to the ComfyUI base, and file_path is
dropped. recent/all/search image lists also gain url/thumbnail_url via
the same enrichment the prompt-images route already had. Remaining
str(e) error bodies in both modules use the basename-only error text.

* fix: check thumbnail directory containment with a real path test

clear_thumbnails compared resolved paths with str.startswith, so a
thumbnails directory that resolved to a sibling with a shared prefix
(out vs out2) passed the safety check and was emptied. It now uses the
module's _is_within (realpath + normcase + Path.is_relative_to).

* fix: decode tag names from the URL only once

The tag lookup, rename and delete routes ran urllib.parse.unquote on
match_info values that aiohttp had already percent-decoded, so a tag
literally named '%41' (requested as %2541) was handled as 'A'. The
second decode is dropped.

* fix: validate tags on every tag mutation route

add_prompt_tag, add_tags_to_prompt, bulk_add_tags, merge_tags and
rename_tag accepted whatever the body carried; a non-string tag raised
AttributeError (500) and over-long or control-character tags were
stored. They now run utils.validators.validate_tags through a shared
tags_error helper and answer 400 with the validator's message. Blank
entries in add_tags_to_prompt are still skipped rather than rejected.

* fix: export every prompt by paging instead of truncating at 10000

/prompt_manager/export fetched a single page of 10000 prompts and
silently dropped the rest. It now walks the table in EXPORT_PAGE_SIZE
pages (1000) inside one executor job until a short page is returned.

* fix: gate the LoraManager serving roots and accept relative link paths

A LoraManager path hand-edited into config.json went straight into
find_lora_directories and so into the image-serving allow-list. When a
path is configured it must now pass lora_utils.resolve_lora_manager_path
(a custom_nodes child whose name contains 'lora'); otherwise, or when the
resolver is unavailable, no LoRA directories are served. An empty path
keeps the custom_nodes-only auto-detection.

POST /images/link now also accepts the relative forms the API hands out
(root-relative 'sub/a.png' and ComfyUI-relative 'output/sub/a.png'),
resolving them against the allowed roots and anchors; '..' segments in
a relative path are refused and unknown files are 404.

* fix: review flow uses the served image url and the lora-dirs test pins its config

Image paths in API responses are now relative to the ComfyUI tree, so the
auto-tag review page can no longer derive a serve URL from image_path; the
server always provides url. The lora-dirs serving test assumed an empty
LoraManager path, which only held in a checkout without a local config.json.

* test: stop leaking a mocked utils.comfyui_integration and mock the IS_CHANGED clock

test_image_monitor_singleton replaced utils.comfyui_integration in
sys.modules with a MagicMock and never restored it, so running it before
test_comfyui_integration failed eight tests. The mock was unused; drop it
and install the server stub with setdefault in every owned test module.

The PromptSearchList IS_CHANGED test slept 10 ms between two time.time()
reads; the Windows clock ticks every 15.6 ms, so it was flaky there. Mock
the clock with an incrementing counter instead.

* fix: resolve_text follows a node shared by two branches

The cycle guard used one visited set for the whole walk, so a string node
read twice (a diamond, or StringConcatenate with the same source on both
inputs) made the second branch None and the prompt text unknowable. Cycle
detection is now scoped to the current path; a separate visit counter
keeps MAX_VISITS as the bound on total work.

* fix: bound resolve_text on diamond chains with memoisation, depth and length caps

A per-path visited set alone makes a chain of diamonds (node i reads node
i+1 on both inputs) exponential in depth. Each node's text is now
memoised so it is expanded once, every expansion is charged to the shared
MAX_VISITS budget, chains deeper than MAX_DEPTH (64) are unknowable so
recursion stays bounded, and text longer than MAX_TEXT_LENGTH (100k) is
unknowable so StringConcatenate doubling cannot exhaust memory.

* fix: static routes resolve under web/ with an extension allow-list

/prompt_manager/lib/{filepath} and /js/{filepath} only rejected '..' and a
leading '/'. On Windows os.path.join(root, 'C:/x') or a leading backslash
re-anchors the path, so any file on the host could be read. Requests are
now refused when the path is absolute on any platform (slash, backslash,
drive letter, UNC prefix), carries an empty, '.' or '..' segment, has an
extension outside the allow-list, or resolves (symlinks included) outside
the static directory; the resolved real path is served with its MIME type.

* feat: capped JSON body reader and single-flight lock for long admin jobs

Add _read_json_body(request, max_bytes=1_000_000) to py/api: reads the
body in chunks, answers 413 past the cap (declared or streamed) and 400
for invalid or non-object JSON. Every JSON POST in the admin and logging
handlers goes through it; it is exposed as PromptManagerAPI._read_json_body
for the other route mixins.

Add _acquire_long_job(name) / _long_job_busy_response(label) on the API
object: one asyncio.Lock per job family, handed out as an async context
manager or None when busy, so a handler answers 409 instead of starting a
second scan. The duplicate scan and delete share the 'duplicates' family.

* fix: harden database restore and keep server paths out of admin responses

Restore: verify_database_file and restore_from_file run through
_run_in_executor; the upload is also opened read-only with
PRAGMA trusted_schema=OFF and refused when sqlite_master lists a trigger
or a view (a prompts database has neither, a crafted one could use them
to run SQL against the live data); at most MAX_SAFETY_BACKUPS (10)
'<db>.backup_*' files are kept next to the live database, oldest first.

Responses: backup_created and the maintenance statistics database_path
go through _public_path, and backup/restore failure messages through
_public_error, so no absolute server path appears in an admin body.

* fix: validate hand-edited gallery roots everywhere and bound output scans

A root written straight into config.json skipped the containment check
the settings endpoint applies. GalleryConfig.update_config (and so
load_from_file) now drops entries that fail validate_gallery_root with a
warning, _find_comfyui_output_dir validates the configured directory
before using it, and ImageMonitor.start_monitoring validates every
directory before scheduling the observer.

The duplicate cleaner only deletes files with a media extension and
never anything under a thumbnails directory. _collect_media_files walks
with os.walk(followlinks=False), skips symlinked files and directories,
prunes thumbnails, stops MAX_SCAN_DEPTH (12) levels down and after
MAX_SCAN_FILES (50000) files.

* fix: bound log file size and backup count in the logging config endpoint

max_file_size must be between 1 MB and 1024 MB (bytes) and backup_count
between 0 and 50; anything else is a 400. Also close every database
connection in the logging test teardown.

* fix: private config.json, merged settings save, synthetic test-link path

PromptManagerConfig.save_to_file and the settings endpoint write
config.json through write_private_json: created with mode 0600 (chmod on
POSIX, best effort on Windows) because the file can hold an API key.
_persist_settings merges web_ui and gallery.monitoring.directories into
the existing file instead of overwriting it, so the integrations section
and any other key survive a settings save.

The test-link diagnostic no longer stores a client-supplied path; it
records a marker file name under the output directory.

* fix: image worker thread exits on shutdown and restarts cleanly

ImageGenerationHandler.shutdown() clears the pending set, sets a stop
event the worker's delay and settle waits listen to, and puts a None
sentinel on the queue; the worker exits within the current wait. Each
worker owns its queue and stop event so a draining worker never swallows
an image scheduled after shutdown. ImageMonitor.stop_monitoring calls it.

* test: close all database connections before deleting temp files

Windows refuses to unlink an open SQLite file and executor threads keep
their own connections, so every owned test that creates a PromptDatabase
calls close_all() before cleanup; the diagnostics temp dir ignores
cleanup errors.

* fix: output scan runs as a background job that survives the browser going away

Closing the dashboard tab or clicking outside the scan modal raised
ClientConnectionResetError out of scan_images and abandoned the scan
mid-batch, leaving a half-scanned database. Two overlapping scans could
also run at once.

- py/api/scan_job.py: ScanJob fans progress out to any number of SSE
  subscribers (latest snapshot first, then live events)
- POST /prompt_manager/scan starts the scan as an asyncio task, or attaches
  to the one already running; a write to a dead socket ends that stream
  quietly while the scan continues
- GET /prompt_manager/scan/status reports the running flag and last event
- dashboard: dismissing the modal while a scan runs minimizes it to a
  progress pill, the page re-attaches to a running scan after a reload,
  a lost stream reconnects, and leaving the page mid-scan asks first

* feat: worker-thread setting speeds up the output scan and thumbnail generation

Both jobs processed one file at a time on a single core. Pillow decodes the
whole image to reach trailing PNG text chunks and releases the GIL while it
does, so a bounded thread pool scales almost linearly: on a 1 GB sample of
400 PNGs the metadata pass dropped from 18 s to 2.5 s at 8 threads, and 120
thumbnails from 4.6 s to 0.6 s.

- utils/parallel.py: map_parallel(func, items, workers) keeps order, runs on
  the caller's thread when workers is 1, and caps the pool at the item count
- PromptManagerConfig.WORKER_THREADS defaults to half the detected cores,
  is clamped to max_worker_threads() when loaded from config.json, and is
  persisted under "performance"
- settings API returns worker_threads and cpu_count and validates saves;
  the settings modal gets a "Worker threads" slider capped at the core count
- the scan reads each metadata batch on the configured threads; thumbnail
  generation processes batches of files per executor job, still emitting
  one progress event per file

Threads rather than processes: this runs inside ComfyUI's server, where
forking a CUDA-initialised process is unsafe and the spawn start method
would re-import ComfyUI's main module in every child.

* fix: scan modal ignores backdrop clicks so a focus click cannot dismiss it

Only its own buttons dismiss it: Cancel before a scan starts, Hide (minimize
to the progress pill) while one runs.

* feat: scan reads prompts embedded in videos; unreadable files no longer log errors

The output scan handed every .mp4 to Pillow and logged one ERROR per file.
ComfyUI's video savers store the queued graph as JSON in the container's
comment tag, so those files carry prompts worth linking.

- utils/video_metadata.py reads the comment/description tag through
  ffprobe with a timeout, normalised to the PNG text-chunk shape so the
  existing parser handles both; without ffprobe videos are skipped and a
  single INFO notice is logged
- _extract_comfyui_metadata routes videos there and logs unreadable images
  as warnings, since the scan carries on past them

Test hygiene: every aiohttp test base class overrode tearDownAsync without
calling super(), so no test client was ever closed. That produced thousands
of ResourceWarnings, "Unclosed client session" errors from asyncio that
broke log assertions, and doubled the suite's run time.

* fix: thumbnail runs skip fresh thumbnails before the cap and report what remains

The per-run cap of 10,000 applied to the directory walk itself, so a
library with more files than that never had thumbnails generated for the
rest, and a run that found 9,987 existing thumbnails spent its whole
budget on skipping them.

- _plan_thumbnails walks the whole tree (bounded like the gallery listing),
  counts files whose thumbnail is already newer as skipped without queuing
  them, and caps only the pending files; the leftover count is returned
- start and complete payloads carry skipped and remaining; total_images
  counts every file considered and the processing rate uses the files
  actually worked on
- the gallery page shows a warning naming the pending count instead of a
  success toast when a run stopped at the cap

* feat(gallery): warn with the pending count when a thumbnail run stops at the cap

* fix: thumbnails are written atomically and served under mtime-versioned urls

Cards showed "Failed to load" for thumbnails that were valid on disk. The
browser had fetched them while the thumbnail run was still writing the
file and, because image responses are cached for an hour, kept the broken
bytes. The card also had no fallback to the original.

- image and video thumbnails are written to a hidden temp sibling and
  renamed into place, so a partial file is never visible at the final path
- every thumbnail_url (gallery listing, prompt images, duplicates, autotag
  listing) carries ?v=<mtime>, so a regenerated thumbnail gets a new URL
  instead of the cached old one
- a 404 from the image serve route is marked no-store; ComfyUI's cache
  middleware would otherwise let browsers cache the miss for an hour
- image-helpers gains fallbackImageSource(): the original is tried once
  before a card gives up

Test hygiene: the autotag tests now stub ComfyUI's server module like the
other API test modules, so the module passes when run on its own.

* fix(gallery): display settings apply at load, columns work, media info toggles

Default View Mode, Items per Page and Grid Columns were only applied when
Save was pressed, never when the page loaded; the column setting swapped a
Tailwind class that was never compiled (only 3, 6 and 8 exist) and a wider
breakpoint class overrode it anyway; Show Media Info was saved and never
read.

- gallery-settings.js validates what localStorage holds (view mode, limit
  option, column range, booleans) and falls back to defaults per value
- settings are applied in the constructor before the first load; changing
  the page size reloads, toggling media info re-renders
- the column count sets an explicit grid template through a CSS variable
  that wins on desktop widths
- media info lines honour the toggle in grid and list views
- image load failures are handled by one delegated listener that tries the
  original before showing the failed-to-load card

* feat(gallery): duplicate scan runs as a background job with progress; infinite scrolling

The duplicate scan was one blocking request with a spinner: no progress,
a modal that could not be put away, and a result lost if the tab closed.

- POST /prompt_manager/scan_duplicates/stream starts the scan as a
  background job (same ScanJob fan-out as the output scan) or attaches to
  the running one; GET /prompt_manager/scan_duplicates/status reports it.
  The job holds the duplicates lock, so deletes are refused while it runs.
- hashing runs on the configured worker threads and reports progress
  about every percent
- the gallery modal shows a progress bar, closing it minimizes to a pill
  that reopens it, results wait in the pill if the modal was closed, and a
  reload re-attaches to a running scan

Infinite scrolling is a new Display Setting (off by default): the next page
is appended when the viewport nears the bottom, pagination controls hide,
and the load-more rule lives in list-state.js where it is unit tested.
Thumb-size buttons and the column setting no longer override each other.

* fix(gallery): Rescan Output Folder drops the server listing cache and rewalks

The button only re-requested the listing, which the server answers from a
30-second in-memory cache, so a rescan showed exactly what was already on
screen.

- POST /prompt_manager/gallery/rescan invalidates the gallery cache,
  rewalks every output root and returns the file and root counts
- the button calls it, closes the settings modal, reloads the folder tree
  and the first page, and reports what it found

* feat: tag search box autocompletes, narrowed by the tags already typed

- PromptDatabase.suggest_tags(prefix, with_tags, limit): tags starting
  with the prefix, counted over the prompts that carry every context tag,
  context tags excluded, ordered by count then name; LIKE wildcards in the
  prefix are escaped
- GET /prompt_manager/tags/suggest?q=&with=&limit= exposes it (limit
  clamped to 1..50, context capped at 10 tags, registered before the
  {tag_name} routes)
- tag-autocomplete.js splits the comma-separated input into committed tags
  and the one being typed, builds the query, applies a pick and handles
  wrap-around keyboard selection
- the dashboard Tags box shows suggestions with counts on focus and while
  typing, arrow keys and Enter/Tab pick, Escape closes, and each pick
  immediately offers the next tag narrowed by the previous ones

* feat(dashboard): infinite scrolling for the prompt list, saved default plus session toggle

- PromptManagerConfig.INFINITE_SCROLL (web_ui.infinite_scroll) is reported
  and saved through the settings API; only booleans are accepted
- Settings modal gains an "Infinite Scrolling" default; saving applies it
  at once
- the results header gains an "Infinite scroll" checkbox that overrides the
  default for the current session without saving
- when on, the next page is appended as the list nears its bottom (same
  load-more rule as the gallery), page controls hide, a short page keeps
  loading until the list fills, and a refresh after an edit restarts from
  the top

* fix: first CI run on Windows and macOS; tests now enforce closed database files

The first push of the branch ran the Tests matrix on all three operating
systems. Every Python job failed; the causes were:

- numpy missing in CI: the WD14 engine tests import it (requirements-dev)
- macOS: temp directories resolve through /private; the admin API tests
  now use the real path like the gallery roots they compare against
- Windows: the startup banner printed an emoji, which raises
  UnicodeEncodeError on a legacy-code-page console and aborted loading
  the whole node; the banner is ASCII now and a test loads the package
  under a cp1252 stdout
- Windows: gallery subfolders came back with backslashes, which neither
  the folder tree nor the search filter understand; they are "/" joined
  on every platform. A relative DEFAULT_DB_PATH kept its forward slashes
  after being joined to the extension root; it is normalised now
- Windows: sqlite connections opened with `with sqlite3.connect()` are
  not closed by the context manager and linger until garbage collection,
  which holds a file lock there. Production sites (vacuum, diagnostics,
  restore inspection) close explicitly; a restore that swaps the API's
  database object closes the previous one; PromptModel keeps a registry
  so a shutdown or a test can close every instance
- Windows: served files were still open when their temp directory was
  removed (tests now read the body), a log download test compared text
  mode newlines, two tests compared paths with a hard-coded separator

tests/open_handles.py makes the Linux run fail the way Windows would:
every database test base checks, right before deleting its files, that
nothing in the process still holds them open.

* fix: Windows case-folded gallery roots in tests, "/" subfolders everywhere, longer busy timeout

Second CI run on Windows:
- gallery roots are stored resolved and case-folded there; the admin API
  tests now compare against that form instead of the mixed-case temp path
- the subfolder fallback for images outside every root also uses "/", and
  the test expects it
- a relative diagnostics db path is normalised after joining
- "database is locked" under many writers on a slow disk: connections now
  wait up to 30 seconds (connect timeout and busy_timeout), long enough for
  a scan or thumbnail run while ComfyUI saves images

* fix: serve every gallery video format, validate settings before applying, dedupe output dirs

Review findings on #153 (Codex):
- the image serve route allowed only three video types while the gallery
  lists seven, so .avi/.mkv/.m4v/.wmv cards opened to a 403; the serve
  allow-list now covers every gallery video format
- a settings payload with one invalid field still applied the valid ones
  before returning 400; the whole payload is validated first and nothing
  changes on failure. webui_display_mode is validated too

Third Windows CI run:
- the same output directory spelled twice (different case or a link) was
  listed twice in diagnostics and served roots; both dedupe on the
  resolved, case-folded path
- the root-less subfolder test expects the "/" form

Code-quality bot: index definitions are (name, target) pairs instead of
implicitly concatenated strings; redundant local asyncio imports and
two trivial lambdas in test stubs removed.

* docs: version 3.3.0, README changelog and refreshed sections

Bump to 3.3.0: new features (worker threads, background scans, tag
autocomplete, infinite scrolling, video prompt scanning) and API contract
changes (POST-only side-effect routes, relative image paths, paginated
search) go beyond a patch release.

README: changelog entries for 3.2.4 and 3.3.0, refreshed bulk scanning,
gallery settings, search, configuration and testing sections, and the
roadmap's recently-completed list.
2026-10-08 18:53:11 -07:00

2011 lines
77 KiB
Python

"""Route tests for the admin API (py/api/admin.py).
Uses aiohttp's test client against a PromptManagerAPI wired to a temporary
database and a stub ``folder_paths`` module, so the tests run without ComfyUI.
"""
import asyncio
import json
import os
import shutil
import sys
import tempfile
import types
import unittest
from pathlib import Path
from unittest.mock import MagicMock
_mock_server = MagicMock()
_mock_server.PromptServer.instance.routes = MagicMock()
sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from database.models import PromptModel # noqa: E402
try:
from tests.open_handles import assert_closed # noqa: E402
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed # noqa: E402
from aiohttp import web # noqa: E402
from aiohttp.test_utils import AioHTTPTestCase # noqa: E402
from database.operations import PromptDatabase # noqa: E402
from py.api import PromptManagerAPI # noqa: E402
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH"
EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
class AdminAPITestCase(AioHTTPTestCase):
"""Test app with PromptManager routes, temp DB and config, fake ComfyUI tree."""
async def get_application(self):
# realpath: macOS hands out /var/... for a /private/var/... directory and
# the gallery roots are stored resolved
self.tmpdir = os.path.realpath(tempfile.mkdtemp())
self.comfy_dir = Path(self.tmpdir) / "ComfyUI"
self.output_dir = self.comfy_dir / "output"
self.output_dir.mkdir(parents=True)
self.outside_dir = Path(self.tmpdir) / "outside"
self.outside_dir.mkdir()
self.config_path = os.path.join(self.tmpdir, "config.json")
self._orig_env = {
key: os.environ.pop(key, None) for key in (CONFIG_PATH_ENV, EXTRA_ROOTS_ENV)
}
os.environ[CONFIG_PATH_ENV] = self.config_path
self._orig_folder_paths = sys.modules.get("folder_paths")
sys.modules["folder_paths"] = types.SimpleNamespace(
base_path=str(self.comfy_dir),
get_output_directory=lambda: str(self.output_dir),
)
self._orig_gallery_dirs = list(GalleryConfig.MONITORING_DIRECTORIES)
self._orig_pm_config = PromptManagerConfig.get_config()
GalleryConfig.MONITORING_DIRECTORIES = []
db_path = os.path.join(self.tmpdir, "prompts.db")
app = web.Application()
routes = web.RouteTableDef()
self.api = PromptManagerAPI()
self.api.db = PromptDatabase(db_path)
self.api.add_routes(routes)
app.router.add_routes(routes)
return app
async def tearDownAsync(self):
await super().tearDownAsync() # closes the aiohttp test client
GalleryConfig.MONITORING_DIRECTORIES = self._orig_gallery_dirs
PromptManagerConfig.update_config(self._orig_pm_config)
for key, value in self._orig_env.items():
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
if self._orig_folder_paths is None:
sys.modules.pop("folder_paths", None)
else:
sys.modules["folder_paths"] = self._orig_folder_paths
PromptModel.close_all_instances() # restore may have swapped api.db
assert_closed(self, self.tmpdir)
shutil.rmtree(self.tmpdir, ignore_errors=True)
# ── helpers ────────────────────────────────────────────────────────
@staticmethod
def _root(path):
"""A gallery root as the config stores it: resolved, case-folded on Windows."""
return GalleryConfig.resolve_gallery_root(str(path))
def _write_config(self, content):
with open(self.config_path, "w") as f:
json.dump(content, f)
def _read_config(self):
with open(self.config_path) as f:
return json.load(f)
class TestSaveSettingsGalleryRoots(AdminAPITestCase):
async def test_rejects_filesystem_root_and_keeps_config(self):
self._write_config({"marker": "untouched"})
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [os.path.abspath(os.sep)]},
)
self.assertEqual(resp.status, 400)
data = await resp.json()
self.assertFalse(data["success"])
self.assertEqual(self._read_config(), {"marker": "untouched"})
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
async def test_rejects_directory_outside_comfyui(self):
self._write_config({"marker": "untouched"})
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [self._root(self.outside_dir)]},
)
self.assertEqual(resp.status, 400)
self.assertEqual(self._read_config(), {"marker": "untouched"})
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
async def test_one_bad_root_rejects_whole_request(self):
good = self.output_dir / "good"
good.mkdir()
self._write_config({"marker": "untouched"})
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [str(good), str(self.outside_dir)]},
)
self.assertEqual(resp.status, 400)
self.assertEqual(self._read_config(), {"marker": "untouched"})
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
async def test_legacy_single_path_rejected_too(self):
self._write_config({"marker": "untouched"})
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_path": os.path.expanduser("~")},
)
self.assertEqual(resp.status, 400)
self.assertEqual(self._read_config(), {"marker": "untouched"})
async def test_accepts_subdir_of_output_and_persists(self):
good = self.output_dir / "renders"
good.mkdir()
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [self._root(good)]},
)
self.assertEqual(resp.status, 200)
data = await resp.json()
self.assertTrue(data["success"])
self.assertTrue(data["restart_required"])
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [self._root(good)])
saved = self._read_config()
self.assertEqual(
saved["gallery"]["monitoring"]["directories"], [self._root(good)]
)
async def test_accepts_env_listed_directory(self):
os.environ[EXTRA_ROOTS_ENV] = str(self.outside_dir)
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [self._root(self.outside_dir)]},
)
self.assertEqual(resp.status, 200)
self.assertEqual(
GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.outside_dir)]
)
async def test_legacy_single_path_accepted(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_path": str(self.output_dir)},
)
self.assertEqual(resp.status, 200)
self.assertEqual(
GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.output_dir)]
)
async def test_empty_paths_clear_roots(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": ["", " "]},
)
self.assertEqual(resp.status, 200)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
self.assertEqual(
self._read_config()["gallery"]["monitoring"]["directories"], []
)
async def test_paths_must_be_a_list(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": str(self.output_dir)},
)
self.assertEqual(resp.status, 400)
async def test_path_entries_must_be_strings(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [42]},
)
self.assertEqual(resp.status, 400)
class FakeMonitor:
def __init__(self, dirs):
self.monitored_directories = list(dirs)
def get_status(self):
return {
"running": True,
"monitored_directories": self.monitored_directories,
"handler_active": True,
"observer_alive": True,
}
class TestNoAbsolutePathsInResponses(AdminAPITestCase):
"""No admin response may reveal the server's absolute filesystem layout."""
def _install_monitor(self, dirs):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = FakeMonitor(dirs)
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
async def _body(self, method, path, **kwargs):
resp = await self.client.request(method, path, **kwargs)
return resp, await resp.text()
async def test_settings_show_roots_relative_to_comfyui(self):
renders = self.output_dir / "renders"
renders.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [str(renders)]
self._install_monitor([str(renders)])
resp, body = await self._body("GET", "/prompt_manager/settings")
self.assertEqual(resp.status, 200)
self.assertNotIn(self.tmpdir, body)
data = json.loads(body)
self.assertEqual(data["settings"]["gallery_root_paths"], ["output/renders"])
self.assertEqual(data["settings"]["gallery_root_path"], "output/renders")
self.assertEqual(data["settings"]["monitored_directories"], ["output/renders"])
async def test_settings_relative_root_round_trips(self):
"""What GET returns can be POSTed back unchanged from any CWD."""
renders = self.output_dir / "renders"
renders.mkdir()
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": ["output/renders"]},
)
self.assertEqual(resp.status, 200)
self.assertEqual(
[os.path.normcase(p) for p in GalleryConfig.MONITORING_DIRECTORIES],
[os.path.normcase(os.path.realpath(renders))],
)
@unittest.skipIf(sys.platform == "win32", "symlinks need privileges on Windows")
async def test_same_output_dir_spelled_twice_is_listed_once(self):
alias = self.comfy_dir / "alias"
os.symlink(self.output_dir, alias, target_is_directory=True)
# Two spellings of one directory (Windows adds case differences too)
GalleryConfig.MONITORING_DIRECTORIES = [str(alias), self._root(self.output_dir)]
self._install_monitor([self._root(self.output_dir)])
resp, body = await self._body("GET", "/prompt_manager/diagnostics")
self.assertEqual(resp.status, 200)
data = json.loads(body)
self.assertEqual(
data["diagnostics"]["comfyui_output"]["output_dirs"], ["output"]
)
self.assertEqual(len(self.api._get_all_output_dirs()), 1)
async def test_diagnostics_paths_are_relative(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
self._install_monitor([self._root(self.output_dir)])
resp, body = await self._body("GET", "/prompt_manager/diagnostics")
self.assertEqual(resp.status, 200)
self.assertNotIn(self.tmpdir, body)
data = json.loads(body)
self.assertEqual(
data["diagnostics"]["comfyui_output"]["output_dirs"], ["output"]
)
self.assertEqual(
data["diagnostics"]["image_monitor"]["monitored_directories"], ["output"]
)
async def test_diagnostics_missing_db_message_has_no_path(self):
self.api.db.model.db_path = os.path.join(self.tmpdir, "missing", "prompts.db")
resp, body = await self._body("GET", "/prompt_manager/diagnostics")
self.assertNotIn(self.tmpdir, body)
data = json.loads(body)
self.assertEqual(data["diagnostics"]["database"]["status"], "error")
self.assertIn("prompts.db", data["diagnostics"]["database"]["message"])
async def test_scan_duplicates_paths_are_relative_to_output(self):
(self.output_dir / "a.png").write_bytes(b"same-bytes")
(self.output_dir / "sub").mkdir()
(self.output_dir / "sub" / "b.png").write_bytes(b"same-bytes")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp, body = await self._body("GET", "/prompt_manager/scan_duplicates")
self.assertEqual(resp.status, 200)
self.assertNotIn(self.tmpdir, body)
data = json.loads(body)
self.assertEqual(len(data["duplicates"]), 1)
paths = sorted(img["path"] for img in data["duplicates"][0]["images"])
self.assertEqual(paths, ["a.png", os.path.join("sub", "b.png")])
async def test_delete_duplicates_accepts_relative_paths(self):
target = self.output_dir / "sub" / "b.png"
target.parent.mkdir()
target.write_bytes(b"bytes")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp, body = await self._body(
"POST",
"/prompt_manager/delete_duplicate_images",
json={"image_paths": [os.path.join("sub", "b.png")]},
)
self.assertEqual(resp.status, 200)
data = json.loads(body)
self.assertEqual(data["deleted_count"], 1)
self.assertFalse(target.exists())
async def test_scan_images_stream_has_no_absolute_paths(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp, body = await self._body("POST", "/prompt_manager/scan")
self.assertEqual(resp.status, 200)
self.assertNotIn(self.tmpdir, body)
self.assertIn('"directories": ["output"]', body)
def _write_png(path, prompt_text=None):
"""Write a tiny PNG, optionally carrying a ComfyUI-style 'prompt' chunk."""
from PIL import Image, PngImagePlugin
img = Image.new("RGB", (2, 2), (255, 0, 0))
info = PngImagePlugin.PngInfo()
if prompt_text is not None:
graph = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": prompt_text}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
"3": {
"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["2", 0]},
},
}
info.add_text("prompt", json.dumps(graph))
img.save(str(path), pnginfo=info)
class TestScansRunOffTheEventLoop(AdminAPITestCase):
"""Blocking filesystem work must go through _run_in_executor."""
def _spy_executor(self):
calls = []
original = self.api._run_in_executor
async def spy(func, *args, **kwargs):
calls.append(getattr(func, "__name__", repr(func)))
return await original(func, *args, **kwargs)
self.api._run_in_executor = spy
return calls
async def test_scan_duplicates_runs_in_executor(self):
(self.output_dir / "a.png").write_bytes(b"dup")
(self.output_dir / "b.png").write_bytes(b"dup")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
calls = self._spy_executor()
resp = await self.client.request("GET", "/prompt_manager/scan_duplicates")
self.assertEqual(resp.status, 200)
self.assertEqual((await resp.json())["duplicates"][0]["count"], 2)
self.assertIn("_find_duplicate_images_sync", calls)
async def test_delete_duplicates_runs_in_executor(self):
(self.output_dir / "a.png").write_bytes(b"dup")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
calls = self._spy_executor()
resp = await self.client.request(
"POST",
"/prompt_manager/delete_duplicate_images",
json={"image_paths": ["a.png"]},
)
self.assertEqual(resp.status, 200)
self.assertEqual((await resp.json())["deleted_count"], 1)
self.assertIn("_delete_duplicate_images_sync", calls)
async def test_output_scan_runs_in_executor(self):
_write_png(self.output_dir / "gen.png", prompt_text="a red square")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
calls = self._spy_executor()
resp = await self.client.request("POST", "/prompt_manager/scan")
body = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn('"type": "complete"', body)
self.assertIn('"added": 1', body)
for name in (
"_collect_output_media_sync",
"_extract_batch_metadata_sync",
"_ingest_scan_batch_sync",
):
self.assertIn(name, calls)
async def test_output_scan_links_existing_prompt_on_second_pass(self):
_write_png(self.output_dir / "gen.png", prompt_text="a red square")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
first = await (await self.client.request("POST", "/prompt_manager/scan")).text()
second = await (
await self.client.request("POST", "/prompt_manager/scan")
).text()
self.assertIn('"added": 1', first)
self.assertIn('"linked": 1', second)
self.assertIn('"added": 0', second)
prompts = self.api.db.get_recent_prompts(limit=10)["prompts"]
self.assertEqual([p["text"] for p in prompts], ["a red square"])
async def test_output_scan_without_directories_reports_error(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.outside_dir)]
sys.modules.pop("folder_paths", None)
self.api._find_comfyui_output_dir = lambda: None
body = await (await self.client.request("POST", "/prompt_manager/scan")).text()
self.assertIn('"type": "error"', body)
class TestDuplicateScanSyncBody(AdminAPITestCase):
"""Behaviour of the synchronous duplicate scan against a real tree."""
def _scan(self):
return self.api._find_duplicate_images_sync(str(self.output_dir))
def test_groups_identical_files_and_sorts_by_mtime(self):
older = self.output_dir / "older.png"
newer = self.output_dir / "sub" / "newer.PNG"
newer.parent.mkdir()
older.write_bytes(b"same")
newer.write_bytes(b"same")
os.utime(older, (1_000_000, 1_000_000))
os.utime(newer, (2_000_000, 2_000_000))
(self.output_dir / "unique.png").write_bytes(b"different")
groups = self._scan()
self.assertEqual(len(groups), 1)
self.assertEqual(groups[0]["count"], 2)
self.assertEqual(
[img["filename"] for img in groups[0]["images"]], ["older.png", "newer.PNG"]
)
self.assertEqual(
groups[0]["images"][1]["path"], os.path.join("sub", "newer.PNG")
)
self.assertEqual(
groups[0]["images"][1]["url"], "/prompt_manager/images/serve/sub/newer.PNG"
)
self.assertNotIn(self.tmpdir, json.dumps(groups))
def test_thumbnails_dir_is_skipped_and_thumbnail_url_reported(self):
(self.output_dir / "a.png").write_bytes(b"same")
(self.output_dir / "b.png").write_bytes(b"same")
thumbs = self.output_dir / "thumbnails"
thumbs.mkdir()
(thumbs / "a_thumb.png").write_bytes(b"same")
(thumbs / "b_thumb.png").write_bytes(b"other")
groups = self._scan()
self.assertEqual(len(groups), 1)
by_name = {img["filename"]: img for img in groups[0]["images"]}
self.assertEqual(set(by_name), {"a.png", "b.png"})
self.assertRegex(
by_name["a.png"]["thumbnail_url"],
r"^/prompt_manager/images/serve/thumbnails/a_thumb\.png\?v=\d+$",
)
self.assertRegex(
by_name["b.png"]["thumbnail_url"],
r"^/prompt_manager/images/serve/thumbnails/b_thumb\.png\?v=\d+$",
)
def test_videos_are_flagged_and_unreadable_entries_skipped(self):
(self.output_dir / "clip.mp4").write_bytes(b"vid")
(self.output_dir / "copy.mp4").write_bytes(b"vid")
(self.output_dir / "not_a_file.png").mkdir() # rglob matches, hashing fails
groups = self._scan()
self.assertEqual(len(groups), 1)
self.assertTrue(all(img["is_video"] for img in groups[0]["images"]))
self.assertTrue(
all(img["media_type"] == "video" for img in groups[0]["images"])
)
def test_missing_output_dir_gives_empty_list(self):
self.assertEqual(
self.api._find_duplicate_images_sync(os.path.join(self.tmpdir, "nope")), []
)
class TestDeleteDuplicatesSyncBody(AdminAPITestCase):
def test_deletes_file_and_thumbnail_inside_output(self):
target = self.output_dir / "sub" / "x.png"
target.parent.mkdir()
target.write_bytes(b"x")
thumb = self.output_dir / "thumbnails" / "sub" / "x_thumb.png"
thumb.parent.mkdir(parents=True)
thumb.write_bytes(b"t")
result = self.api._delete_duplicate_images_sync(
[str(target)], str(self.output_dir)
)
self.assertEqual(result["deleted_count"], 1)
self.assertEqual(result["failed_count"], 0)
self.assertFalse(target.exists())
self.assertFalse(thumb.exists())
def test_refuses_paths_outside_output_and_reports_missing(self):
secret = self.outside_dir / "secret.txt"
secret.write_text("s")
result = self.api._delete_duplicate_images_sync(
[str(secret), "ghost.png"], str(self.output_dir)
)
self.assertEqual(result["deleted_count"], 0)
self.assertEqual(result["failed_count"], 2)
self.assertTrue(secret.exists())
self.assertTrue(any("outside" in f for f in result["failed_files"]))
self.assertTrue(any("not found" in f for f in result["failed_files"]))
def test_without_output_dir_everything_fails(self):
result = self.api._delete_duplicate_images_sync(["a.png"], None)
self.assertEqual(result["failed_count"], 1)
self.assertEqual(result["deleted_count"], 0)
def test_refuses_non_media_files_inside_output(self):
for name in ("notes.txt", "prompts.db", "script.py", "noext"):
(self.output_dir / name).write_bytes(b"x")
result = self.api._delete_duplicate_images_sync(
["notes.txt", "prompts.db", "script.py", "noext"], str(self.output_dir)
)
self.assertEqual(result["deleted_count"], 0)
self.assertEqual(result["failed_count"], 4)
for name in ("notes.txt", "prompts.db", "script.py", "noext"):
self.assertTrue((self.output_dir / name).exists(), name)
self.assertTrue(all("not a media file" in f for f in result["failed_files"]))
def test_refuses_files_under_thumbnails(self):
thumb = self.output_dir / "thumbnails" / "sub" / "x_thumb.png"
thumb.parent.mkdir(parents=True)
thumb.write_bytes(b"t")
result = self.api._delete_duplicate_images_sync(
[os.path.join("thumbnails", "sub", "x_thumb.png")], str(self.output_dir)
)
self.assertEqual(result["deleted_count"], 0)
self.assertTrue(thumb.exists())
self.assertIn("thumbnail", result["failed_files"][0])
def test_every_scanned_media_extension_is_deletable(self):
from py.api.admin import (
DELETABLE_MEDIA_EXTENSIONS,
IMAGE_EXTENSIONS,
VIDEO_EXTENSIONS,
)
for ext in IMAGE_EXTENSIONS + VIDEO_EXTENSIONS + (".tif",):
self.assertIn(ext, DELETABLE_MEDIA_EXTENSIONS)
for ext in (".txt", ".db", ".py", ".json", ".html", ""):
self.assertNotIn(ext, DELETABLE_MEDIA_EXTENSIONS)
(self.output_dir / "a.WEBM").write_bytes(b"v")
result = self.api._delete_duplicate_images_sync(
["a.WEBM"], str(self.output_dir)
)
self.assertEqual(result["deleted_count"], 1)
class TestCollectMediaFiles(AdminAPITestCase):
"""_collect_media_files: bounded walk that never follows symlinked files."""
def _collect(self):
from py.api.admin import (
IMAGE_EXTENSIONS,
VIDEO_EXTENSIONS,
_collect_media_files,
)
found = _collect_media_files(
[self.output_dir], IMAGE_EXTENSIONS + VIDEO_EXTENSIONS
)
return sorted(p.relative_to(self.output_dir).as_posix() for p in found)
def test_matches_case_insensitively_and_skips_thumbnails(self):
(self.output_dir / "a.png").write_bytes(b"x")
(self.output_dir / "b.PNG").write_bytes(b"x")
(self.output_dir / "c.Mp4").write_bytes(b"x")
(self.output_dir / "d.txt").write_bytes(b"x")
(self.output_dir / "sub").mkdir()
(self.output_dir / "sub" / "e.jpg").write_bytes(b"x")
(self.output_dir / "thumbnails").mkdir()
(self.output_dir / "thumbnails" / "a_thumb.png").write_bytes(b"x")
(self.output_dir / "sub" / "thumbnails").mkdir()
(self.output_dir / "sub" / "thumbnails" / "e_thumb.jpg").write_bytes(b"x")
self.assertEqual(self._collect(), ["a.png", "b.PNG", "c.Mp4", "sub/e.jpg"])
def test_symlinked_files_and_directories_are_skipped(self):
secret = Path(self.tmpdir) / "secret.png"
secret.write_bytes(b"s")
secret_dir = Path(self.tmpdir) / "secret_dir"
secret_dir.mkdir()
(secret_dir / "inner.png").write_bytes(b"s")
(self.output_dir / "real.png").write_bytes(b"r")
try:
os.symlink(secret, self.output_dir / "link.png")
os.symlink(secret_dir, self.output_dir / "linkdir")
except (OSError, NotImplementedError):
self.skipTest("symlinks not available")
self.assertEqual(self._collect(), ["real.png"])
def test_depth_is_capped(self):
from py.api.admin import MAX_SCAN_DEPTH
self.assertEqual(MAX_SCAN_DEPTH, 12)
deep = self.output_dir
for level in range(1, MAX_SCAN_DEPTH + 2):
deep = deep / f"d{level}"
deep.mkdir()
(deep / f"f{level}.png").write_bytes(b"x")
found = self._collect()
self.assertIn(
"/".join(f"d{i}" for i in range(1, MAX_SCAN_DEPTH + 1))
+ f"/f{MAX_SCAN_DEPTH}.png",
found,
)
self.assertFalse(
any(name.endswith(f"f{MAX_SCAN_DEPTH + 1}.png") for name in found)
)
self.assertEqual(len(found), MAX_SCAN_DEPTH)
def test_file_count_is_capped(self):
import py.api.admin as admin_module
from unittest.mock import patch
self.assertEqual(admin_module.MAX_SCAN_FILES, 50_000)
for i in range(6):
(self.output_dir / f"{i}.png").write_bytes(b"x")
with patch.object(admin_module, "MAX_SCAN_FILES", 4):
self.assertEqual(len(self._collect()), 4)
def test_missing_root_is_skipped(self):
from py.api.admin import _collect_media_files
self.assertEqual(
_collect_media_files([Path(self.tmpdir) / "nope"], (".png",)), []
)
class TestSettingsReadMonitorThroughModule(AdminAPITestCase):
"""get_settings reaches the image monitor through utils.image_monitor."""
def test_no_sys_modules_scan(self):
import inspect
from py.api.admin import AdminRoutesMixin
source = inspect.getsource(AdminRoutesMixin.get_settings)
self.assertNotIn("sys.modules", source)
async def test_reports_directories_of_running_monitor(self):
import utils.image_monitor as im_mod
from utils.image_monitor import get_image_monitor
orig = im_mod._monitor_instance
im_mod._monitor_instance = None
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
monitor = get_image_monitor(MagicMock(), MagicMock())
monitor.monitored_directories = [self._root(self.output_dir)]
resp = await self.client.request("GET", "/prompt_manager/settings")
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["settings"]["monitored_directories"], ["output"])
async def test_falls_back_to_configured_roots_without_monitor(self):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = None
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp = await self.client.request("GET", "/prompt_manager/settings")
data = await resp.json()
self.assertEqual(data["settings"]["monitored_directories"], ["output"])
self.assertEqual(data["settings"]["gallery_root_paths"], ["output"])
def _raise(*args, **kwargs):
raise RuntimeError("boom")
class TestStatsCleanupAndErrors(AdminAPITestCase):
async def test_stats_success(self):
resp = await self.client.request("GET", "/prompt_manager/stats")
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertIn("total_prompts", data["stats"])
async def test_stats_db_failure_is_500(self):
self.api.db.get_statistics = _raise
resp = await self.client.request("GET", "/prompt_manager/stats")
self.assertEqual(resp.status, 500)
self.assertFalse((await resp.json())["success"])
async def test_cleanup_success(self):
resp = await self.client.request("POST", "/prompt_manager/cleanup")
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["duplicates_removed"], 0)
async def test_cleanup_failure_is_500(self):
self.api.db.cleanup_duplicates = _raise
resp = await self.client.request("POST", "/prompt_manager/cleanup")
self.assertEqual(resp.status, 500)
async def test_scan_duplicates_failure_is_500(self):
self.api._find_comfyui_output_dir = _raise
resp = await self.client.request("GET", "/prompt_manager/scan_duplicates")
body = await resp.text()
self.assertEqual(resp.status, 500)
self.assertNotIn("boom", body)
async def test_scan_duplicates_without_output_dir_is_empty(self):
self.api._find_comfyui_output_dir = lambda: None
resp = await self.client.request("GET", "/prompt_manager/scan_duplicates")
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["duplicates"], [])
class TestDeleteDuplicatesEndpoint(AdminAPITestCase):
async def _post(self, payload):
return await self.client.request(
"POST", "/prompt_manager/delete_duplicate_images", json=payload
)
async def test_empty_list_is_400(self):
resp = await self._post({"image_paths": []})
self.assertEqual(resp.status, 400)
async def test_non_list_is_400(self):
resp = await self._post({"image_paths": "a.png"})
self.assertEqual(resp.status, 400)
async def test_malformed_json_is_400(self):
resp = await self.client.request(
"POST",
"/prompt_manager/delete_duplicate_images",
data=b"{",
headers={"Content-Type": "application/json"},
)
self.assertEqual(resp.status, 400)
async def test_failures_are_reported(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp = await self._post({"image_paths": ["ghost.png"]})
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["failed_count"], 1)
self.assertIn("1 failed", data["message"])
self.assertEqual(len(data["failed_files"]), 1)
async def test_unexpected_error_is_500(self):
self.api._find_comfyui_output_dir = _raise
resp = await self._post({"image_paths": ["a.png"]})
self.assertEqual(resp.status, 500)
def test_thumbnail_removal_failure_does_not_fail_delete(self):
target = self.output_dir / "x.png"
target.write_bytes(b"x")
thumb_dir = self.output_dir / "thumbnails" / "x_thumb.png"
thumb_dir.mkdir(parents=True)
(thumb_dir / "child").write_bytes(b"c") # os.remove on a dir raises OSError
result = self.api._delete_duplicate_images_sync(
[str(target)], str(self.output_dir)
)
self.assertEqual(result["deleted_count"], 1)
self.assertFalse(target.exists())
self.assertTrue(thumb_dir.exists())
def test_unexpected_error_per_file_is_reported_without_path(self):
self.api._delete_one_duplicate = lambda *a: (_ for _ in ()).throw(
PermissionError(13, "Permission denied", os.path.join(self.tmpdir, "x.png"))
)
result = self.api._delete_duplicate_images_sync(["x.png"], str(self.output_dir))
self.assertEqual(result["failed_count"], 1)
self.assertIn("Permission denied", result["failed_files"][0])
self.assertNotIn(self.tmpdir, result["failed_files"][0])
class TestJsonBodyCap(AdminAPITestCase):
"""Every JSON POST in the admin API rejects bodies over the shared cap."""
JSON_POSTS = (
"/prompt_manager/settings",
"/prompt_manager/delete_duplicate_images",
"/prompt_manager/diagnostics/test-link",
"/prompt_manager/maintenance",
)
async def test_oversized_bodies_are_413(self):
body = json.dumps({"pad": "x" * 1_000_100}).encode()
for path in self.JSON_POSTS:
resp = await self.client.request(
"POST", path, data=body, headers={"Content-Type": "application/json"}
)
self.assertEqual(resp.status, 413, path)
self.assertFalse((await resp.json())["success"], path)
class TestDuplicateJobsAreSingleFlight(AdminAPITestCase):
"""A duplicate scan or delete refuses to overlap with a running one."""
def _block_scan(self):
gate = asyncio.Event()
async def slow_scan():
await gate.wait()
return []
self.api.find_duplicate_images = slow_scan
return gate
async def test_second_scan_while_one_runs_is_409(self):
gate = self._block_scan()
first = asyncio.ensure_future(
self.client.request("GET", "/prompt_manager/scan_duplicates")
)
await asyncio.sleep(0.05)
second = await asyncio.wait_for(
self.client.request("GET", "/prompt_manager/scan_duplicates"), 5
)
self.assertEqual(second.status, 409)
data = await second.json()
self.assertFalse(data["success"])
self.assertIn("already running", data["error"])
gate.set()
self.assertEqual((await first).status, 200)
async def test_delete_while_scan_runs_is_409_and_scan_after_is_fine(self):
gate = self._block_scan()
first = asyncio.ensure_future(
self.client.request("GET", "/prompt_manager/scan_duplicates")
)
await asyncio.sleep(0.05)
resp = await asyncio.wait_for(
self.client.request(
"POST",
"/prompt_manager/delete_duplicate_images",
json={"image_paths": ["a.png"]},
),
5,
)
self.assertEqual(resp.status, 409)
gate.set()
await first
resp = await self.client.request("GET", "/prompt_manager/scan_duplicates")
self.assertEqual(resp.status, 200)
async def test_lock_is_released_after_a_failed_scan(self):
self.api._find_comfyui_output_dir = _raise
resp = await self.client.request("GET", "/prompt_manager/scan_duplicates")
self.assertEqual(resp.status, 500)
self.api._find_comfyui_output_dir = lambda: None
resp = await self.client.request("GET", "/prompt_manager/scan_duplicates")
self.assertEqual(resp.status, 200)
class TestSettingsMisc(AdminAPITestCase):
async def test_invalid_payload_changes_nothing(self):
before = (
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
PromptManagerConfig.INFINITE_SCROLL,
list(GalleryConfig.MONITORING_DIRECTORIES),
)
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={
"result_timeout": 42,
"webui_display_mode": "popup",
"worker_threads": 1,
"infinite_scroll": True,
"gallery_root_paths": [os.path.abspath(os.sep)], # rejected
},
)
self.assertEqual(resp.status, 400)
after = (
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
PromptManagerConfig.INFINITE_SCROLL,
list(GalleryConfig.MONITORING_DIRECTORIES),
)
self.assertEqual(after, before)
self.assertFalse(os.path.exists(self.config_path))
async def test_infinite_scroll_default_is_reported_saved_and_validated(self):
data = await (
await self.client.request("GET", "/prompt_manager/settings")
).json()
self.assertIs(data["settings"]["infinite_scroll"], False)
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"infinite_scroll": True}
)
self.assertEqual(resp.status, 200)
self.assertTrue(PromptManagerConfig.INFINITE_SCROLL)
self.assertIs(self._read_config()["web_ui"]["infinite_scroll"], True)
data = await (
await self.client.request("GET", "/prompt_manager/settings")
).json()
self.assertIs(data["settings"]["infinite_scroll"], True)
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"infinite_scroll": "on"}
)
self.assertEqual(resp.status, 400)
self.assertTrue(PromptManagerConfig.INFINITE_SCROLL)
async def test_result_timeout_and_display_mode_saved(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"result_timeout": 9, "webui_display_mode": "popup"},
)
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertFalse(data["restart_required"])
self.assertEqual(PromptManagerConfig.RESULT_TIMEOUT, 9)
self.assertEqual(PromptManagerConfig.WEBUI_DISPLAY_MODE, "popup")
saved = self._read_config()
self.assertEqual(saved["web_ui"]["result_timeout"], 9)
async def test_save_merges_into_existing_config_file(self):
self._write_config(
{
"integrations": {"lora_manager": {"enabled": True, "path": "x"}},
"database": {"default_path": "custom.db"},
"web_ui": {"result_timeout": 1, "show_test_button": True},
}
)
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={
"result_timeout": 7,
"gallery_root_paths": [self._root(self.output_dir)],
},
)
self.assertEqual(resp.status, 200)
saved = self._read_config()
self.assertEqual(saved["integrations"]["lora_manager"]["enabled"], True)
self.assertEqual(saved["integrations"]["lora_manager"]["path"], "x")
self.assertEqual(saved["database"]["default_path"], "custom.db")
self.assertEqual(saved["web_ui"]["result_timeout"], 7)
self.assertEqual(saved["web_ui"]["show_test_button"], True)
self.assertEqual(
saved["gallery"]["monitoring"]["directories"], [self._root(self.output_dir)]
)
async def test_save_replaces_a_corrupt_config_file(self):
with open(self.config_path, "w") as f:
f.write("{not json")
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": 4}
)
self.assertEqual(resp.status, 200)
self.assertEqual(self._read_config()["web_ui"]["result_timeout"], 4)
async def test_saved_config_file_is_private_to_the_user(self):
import stat
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": 4}
)
self.assertEqual(resp.status, 200)
if os.name == "posix":
mode = stat.S_IMODE(os.stat(self.config_path).st_mode)
self.assertEqual(mode, 0o600)
async def test_invalid_result_timeout_is_400(self):
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": "soon"}
)
self.assertEqual(resp.status, 400)
async def test_malformed_json_is_400(self):
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
data=b"{oops",
headers={"Content-Type": "application/json"},
)
self.assertEqual(resp.status, 400)
async def test_unchanged_roots_do_not_require_restart(self):
GalleryConfig.MONITORING_DIRECTORIES = [
os.path.normcase(os.path.realpath(self.output_dir))
]
resp = await self.client.request(
"POST",
"/prompt_manager/settings",
json={"gallery_root_paths": [self._root(self.output_dir)]},
)
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertFalse(data["restart_required"])
async def test_unwritable_config_path_still_succeeds(self):
os.environ[CONFIG_PATH_ENV] = os.path.join(self.tmpdir, "missing", "c.json")
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"result_timeout": 3}
)
self.assertEqual(resp.status, 200)
async def test_save_unexpected_error_is_500(self):
self.api._parse_gallery_roots = _raise
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"gallery_root_paths": []}
)
self.assertEqual(resp.status, 500)
self.assertNotIn("boom", await resp.text())
async def test_get_unexpected_error_is_500(self):
self.api._monitored_directories = _raise
resp = await self.client.request("GET", "/prompt_manager/settings")
self.assertEqual(resp.status, 500)
class TestDiagnostics(AdminAPITestCase):
def _install_monitor(self, monitor):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = monitor
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
async def _diag(self):
resp = await self.client.request("GET", "/prompt_manager/diagnostics")
return resp, (await resp.json())["diagnostics"]
async def test_healthy_report(self):
self.api.db.save_prompt(text="hello", prompt_hash="h1")
self._install_monitor(FakeMonitor([self._root(self.output_dir)]))
resp, diag = await self._diag()
self.assertEqual(resp.status, 200)
self.assertEqual(diag["database"]["status"], "ok")
self.assertEqual(diag["database"]["prompt_count"], 1)
self.assertTrue(diag["database"]["has_images_table"])
self.assertEqual(diag["dependencies"]["status"], "ok")
self.assertEqual(diag["comfyui_output"]["status"], "ok")
self.assertEqual(diag["image_monitor"]["status"], "ok")
async def test_corrupt_database_reported(self):
bad = os.path.join(self.tmpdir, "bad.db")
with open(bad, "wb") as f:
f.write(b"not a database at all, definitely not sqlite")
self.api.db.model.db_path = bad
_, diag = await self._diag()
self.assertEqual(diag["database"]["status"], "error")
self.assertIn("Database error", diag["database"]["message"])
async def test_monitor_not_running_and_no_output_dirs(self):
self._install_monitor(None)
sys.modules.pop("folder_paths", None)
orig_cwd = os.getcwd()
os.chdir(self.outside_dir)
self.addCleanup(os.chdir, orig_cwd)
_, diag = await self._diag()
self.assertEqual(diag["image_monitor"]["status"], "error")
self.assertIn("not initialized", diag["image_monitor"]["message"])
self.assertEqual(diag["comfyui_output"]["status"], "warning")
self.assertEqual(diag["comfyui_output"]["output_dirs"], [])
async def test_relative_output_fallback_without_folder_paths(self):
sys.modules.pop("folder_paths", None)
orig_cwd = os.getcwd()
os.chdir(self.comfy_dir) # has an "output" child
self.addCleanup(os.chdir, orig_cwd)
_, diag = await self._diag()
self.assertEqual(diag["comfyui_output"]["status"], "ok")
self.assertEqual(diag["comfyui_output"]["output_dirs"], ["output"])
async def test_monitor_status_failure_reported(self):
broken = MagicMock()
broken.get_status = _raise
self._install_monitor(broken)
_, diag = await self._diag()
self.assertEqual(diag["image_monitor"]["status"], "error")
self.assertIn("Failed to get monitor status", diag["image_monitor"]["message"])
async def test_unexpected_error_is_500(self):
import py.api.admin as admin_module
orig = admin_module._diagnose_database
admin_module._diagnose_database = _raise
self.addCleanup(setattr, admin_module, "_diagnose_database", orig)
resp = await self.client.request("GET", "/prompt_manager/diagnostics")
self.assertEqual(resp.status, 500)
class TestTestImageLink(AdminAPITestCase):
async def _post(self, payload):
return await self.client.request(
"POST", "/prompt_manager/diagnostics/test-link", json=payload
)
async def test_missing_prompt_id_is_400(self):
resp = await self._post({})
self.assertEqual(resp.status, 400)
async def test_links_test_image(self):
pid = self.api.db.save_prompt(text="hello", prompt_hash="h1")
resp = await self._post({"prompt_id": pid})
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["result"]["status"], "ok")
self.assertIsInstance(data["result"]["image_id"], int)
async def test_client_supplied_path_is_not_stored(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
pid = self.api.db.save_prompt(text="hello", prompt_hash="h1")
evil = os.path.join(self.tmpdir, "etc", "passwd")
resp = await self._post({"prompt_id": pid, "image_path": evil})
data = await resp.json()
self.assertEqual(resp.status, 200, data)
images = self.api.db.get_prompt_images(pid)
self.assertEqual(len(images), 1)
stored = Path(images[0]["image_path"])
self.assertNotEqual(os.path.normcase(str(stored)), os.path.normcase(evil))
self.assertTrue(
stored.resolve().is_relative_to(self.output_dir.resolve()),
stored,
)
self.assertIn("test", stored.name.lower())
self.assertNotIn(self.tmpdir, json.dumps(data))
async def test_without_output_dir_a_relative_marker_is_used(self):
self.api._find_comfyui_output_dir = lambda: None
pid = self.api.db.save_prompt(text="hello", prompt_hash="h1")
resp = await self._post({"prompt_id": pid, "image_path": "/x/y.png"})
self.assertEqual(resp.status, 200)
images = self.api.db.get_prompt_images(pid)
self.assertFalse(os.path.isabs(images[0]["image_path"]))
async def test_db_failure_is_reported_in_result(self):
self.api.db.link_image_to_prompt = _raise
resp = await self._post({"prompt_id": 1})
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertFalse(data["success"])
self.assertEqual(data["result"]["status"], "error")
async def test_malformed_json_is_400(self):
resp = await self.client.request(
"POST",
"/prompt_manager/diagnostics/test-link",
data=b"nope",
headers={"Content-Type": "application/json"},
)
self.assertEqual(resp.status, 400)
class TestMaintenance(AdminAPITestCase):
ALL_OPS = [
"cleanup_duplicates",
"vacuum",
"cleanup_orphaned_images",
"check_hash_duplicates",
"statistics",
"prune_orphaned_prompts",
"check_consistency",
]
async def _post(self, payload=None, **kwargs):
if payload is None:
return await self.client.request(
"POST", "/prompt_manager/maintenance", **kwargs
)
return await self.client.request(
"POST", "/prompt_manager/maintenance", json=payload, **kwargs
)
async def test_default_operations_without_body(self):
resp = await self._post(data=b"", headers={"Content-Type": "text/plain"})
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["operations_completed"], 3)
self.assertTrue(data["all_successful"])
self.assertEqual(
set(data["results"]),
{"cleanup_duplicates", "vacuum", "cleanup_orphaned_images"},
)
async def test_all_operations(self):
self.api.db.save_prompt(text="orphan", prompt_hash="h1")
resp = await self._post({"operations": self.ALL_OPS})
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data["operations_completed"], len(self.ALL_OPS))
self.assertTrue(data["all_successful"], data["results"])
self.assertIn("info", data["results"]["statistics"])
self.assertEqual(data["results"]["check_consistency"]["issues_found"], 0)
async def test_statistics_database_path_is_public(self):
resp = await self._post({"operations": ["statistics"]})
body = await resp.text()
self.assertEqual(resp.status, 200)
self.assertNotIn(self.tmpdir, body)
info = json.loads(body)["results"]["statistics"]["info"]
self.assertEqual(info["database_path"], "prompts.db")
async def test_unknown_operation_does_nothing(self):
resp = await self._post({"operations": ["reboot"]})
data = await resp.json()
self.assertEqual(data["operations_completed"], 0)
self.assertTrue(data["all_successful"])
async def test_each_operation_reports_its_own_failure(self):
self.api.db.cleanup_duplicates = _raise
self.api.db.model.vacuum_database = _raise
self.api.db.cleanup_missing_images = _raise
self.api.db.check_hash_duplicates = _raise
self.api.db.model.get_database_info = _raise
self.api.db.prune_orphaned_prompts = _raise
self.api.db.check_consistency = _raise
resp = await self._post({"operations": self.ALL_OPS})
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertFalse(data["all_successful"])
self.assertEqual(len(data["results"]), len(self.ALL_OPS))
for name in self.ALL_OPS:
self.assertFalse(data["results"][name]["success"], name)
self.assertEqual(data["results"][name]["error"], "boom")
async def test_unexpected_error_is_500(self):
self.api._run_in_executor = _raise
resp = await self._post({"operations": ["vacuum"]})
self.assertEqual(resp.status, 500)
class TestBackupRestoreContract(AdminAPITestCase):
"""Contract tests only: status codes and envelope of backup/restore."""
async def test_backup_downloads_sqlite_file(self):
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
resp = await self.client.request("GET", "/prompt_manager/backup")
body = await resp.read()
self.assertEqual(resp.status, 200)
self.assertIn("attachment", resp.headers["Content-Disposition"])
self.assertTrue(body.startswith(b"SQLite format 3"))
async def test_backup_without_db_file_is_404(self):
self.api.db.model.db_path = os.path.join(self.tmpdir, "nope.db")
resp = await self.client.request("GET", "/prompt_manager/backup")
self.assertEqual(resp.status, 404)
async def _restore(self, field_name, content):
from aiohttp import FormData
form = FormData()
form.add_field(field_name, content, filename="upload.db")
return await self.client.request("POST", "/prompt_manager/restore", data=form)
async def test_restore_wrong_field_is_400(self):
resp = await self._restore("other", b"x")
self.assertEqual(resp.status, 400)
async def test_restore_empty_file_is_400(self):
resp = await self._restore("database_file", b"")
self.assertEqual(resp.status, 400)
async def test_restore_non_sqlite_is_400_and_db_untouched(self):
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
resp = await self._restore("database_file", b"definitely not sqlite data")
self.assertEqual(resp.status, 400)
self.assertIsNotNone(self.api.db.get_prompt_by_hash("h1"))
def _snapshot_upload(self, extra_sql=()):
"""Bytes of a self-contained copy of the live DB (WAL rows included)."""
import sqlite3
from contextlib import closing
snapshot = os.path.join(self.tmpdir, "snapshot.db")
with closing(sqlite3.connect(self.api.db.model.db_path)) as src:
with closing(sqlite3.connect(snapshot)) as dst:
src.backup(dst)
for statement in extra_sql:
dst.execute(statement)
dst.commit()
with open(snapshot, "rb") as f:
return f.read()
async def test_restore_valid_db_succeeds(self):
import sqlite3
from contextlib import closing
self.api.db.save_prompt(text="restored", prompt_hash="h1")
upload = self._snapshot_upload()
resp = await self._restore("database_file", upload)
data = await resp.json()
self.assertEqual(resp.status, 200, data)
self.assertTrue(data["success"])
self.assertEqual(data["prompt_count"], 1)
with closing(sqlite3.connect(self.api.db.model.db_path)) as conn:
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM prompts").fetchone()[0], 1
)
async def test_restore_backup_created_is_a_public_path(self):
self.api.db.save_prompt(text="restored", prompt_hash="h1")
resp = await self._restore("database_file", self._snapshot_upload())
body = await resp.text()
self.assertEqual(resp.status, 200, body)
self.assertNotIn(self.tmpdir, body)
data = json.loads(body)
self.assertTrue(data["backup_created"].startswith("prompts.db.backup_"))
async def test_restore_rejects_database_with_a_trigger(self):
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
upload = self._snapshot_upload(
[
"CREATE TRIGGER evil AFTER INSERT ON prompts "
"BEGIN DELETE FROM prompts; END"
]
)
resp = await self._restore("database_file", upload)
data = await resp.json()
self.assertEqual(resp.status, 400, data)
self.assertIn("trigger", data["error"].lower())
self.assertIsNotNone(self.api.db.get_prompt_by_hash("h1"))
async def test_restore_rejects_database_with_a_view(self):
self.api.db.save_prompt(text="keep me", prompt_hash="h1")
upload = self._snapshot_upload(["CREATE VIEW peek AS SELECT text FROM prompts"])
resp = await self._restore("database_file", upload)
data = await resp.json()
self.assertEqual(resp.status, 400, data)
self.assertIn("view", data["error"].lower())
async def test_restore_verifies_in_the_executor(self):
self.api.db.save_prompt(text="restored", prompt_hash="h1")
upload = self._snapshot_upload()
calls = []
original = self.api._run_in_executor
async def spy(func, *args, **kwargs):
calls.append(getattr(func, "__name__", repr(func)))
return await original(func, *args, **kwargs)
self.api._run_in_executor = spy
resp = await self._restore("database_file", upload)
self.assertEqual(resp.status, 200)
self.assertIn("verify_database_file", calls)
self.assertIn("restore_from_file", calls)
async def test_restore_keeps_at_most_ten_safety_backups(self):
import time
from py.api.admin import MAX_SAFETY_BACKUPS
self.assertEqual(MAX_SAFETY_BACKUPS, 10)
db_path = self.api.db.model.db_path
stale = []
now = time.time()
for i in range(12):
path = f"{db_path}.backup_202001{i + 1:02d}_000000"
with open(path, "wb") as f:
f.write(b"old")
os.utime(path, (now - 10_000 + i, now - 10_000 + i))
stale.append(path)
unrelated = os.path.join(self.tmpdir, "other.db.backup_20200101_000000")
with open(unrelated, "wb") as f:
f.write(b"x")
self.api.db.save_prompt(text="restored", prompt_hash="h1")
resp = await self._restore("database_file", self._snapshot_upload())
self.assertEqual(resp.status, 200)
remaining = sorted(
name
for name in os.listdir(self.tmpdir)
if name.startswith("prompts.db.backup_")
)
self.assertEqual(len(remaining), MAX_SAFETY_BACKUPS)
data = await resp.json()
self.assertIn(data["backup_created"], remaining)
for path in stale[:3]:
self.assertFalse(os.path.exists(path), path)
self.assertTrue(os.path.exists(unrelated))
async def test_restore_failure_body_has_no_absolute_path(self):
self.api.db.save_prompt(text="restored", prompt_hash="h1")
upload = self._snapshot_upload()
def boom(src_path):
raise PermissionError(
13, "Permission denied", os.path.join(self.tmpdir, "prompts.db")
)
self.api.db.model.restore_from_file = boom
resp = await self._restore("database_file", upload)
body = await resp.text()
self.assertEqual(resp.status, 500)
self.assertNotIn(self.tmpdir, body)
self.assertIn("Permission denied", json.loads(body)["error"])
async def test_backup_failure_body_has_no_absolute_path(self):
def boom(path):
raise PermissionError(
13, "Permission denied", os.path.join(self.tmpdir, "x.db")
)
self.api.db.model.backup_database = boom
resp = await self.client.request("GET", "/prompt_manager/backup")
body = await resp.text()
self.assertEqual(resp.status, 500)
self.assertNotIn(self.tmpdir, body)
self.assertIn("Permission denied", json.loads(body)["error"])
class TestOutputScanInternals(AdminAPITestCase):
def test_metadata_extraction_failure_yields_empty(self):
self.api._extract_comfyui_metadata = _raise
results = self.api._extract_batch_metadata_sync([Path("a.png")])
self.assertEqual(results, [(Path("a.png"), {})])
def test_ingest_outcomes(self):
f = Path("gen.png")
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a cat"}}}
)
}
self.assertIsNone(self.api._ingest_scanned_file(f, {}))
self.assertIsNone(self.api._ingest_scanned_file(f, {"prompt": "{}"}))
self.assertEqual(
self.api._ingest_scanned_file(f, {"parameters": " \nNegative prompt: x"}),
"found",
)
self.assertEqual(self.api._ingest_scanned_file(f, meta), "added")
self.assertEqual(self.api._ingest_scanned_file(f, meta), "linked")
def test_ingest_link_failures_are_tolerated(self):
f = Path("gen.png")
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a dog"}}}
)
}
self.api.db.link_image_to_prompt = _raise
self.assertEqual(self.api._ingest_scanned_file(f, meta), "added")
self.assertEqual(self.api._ingest_scanned_file(f, meta), "found")
def test_ingest_save_failure_counts_as_found(self):
f = Path("gen.png")
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a bird"}}}
)
}
self.api.db.save_prompt = lambda *a, **k: None
self.assertEqual(self.api._ingest_scanned_file(f, meta), "found")
def test_batch_counts_and_per_file_errors(self):
meta = {
"prompt": json.dumps(
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a fish"}}}
)
}
self.api.db.get_prompt_by_hash = _raise
counts = self.api._ingest_scan_batch_sync(
[(Path("a.png"), meta), (Path("b.png"), {})]
)
self.assertEqual(counts, {"processed": 2, "found": 0, "added": 0, "linked": 0})
async def test_scan_internal_error_streams_error_event(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
self.api._collect_output_media_sync = _raise
body = await (await self.client.request("POST", "/prompt_manager/scan")).text()
self.assertIn('"type": "error"', body)
self.assertNotIn("boom", body)
class TestOutputScanIsABackgroundJob(AdminAPITestCase):
"""The output scan keeps running when the browser goes away, and a second
request attaches to the running scan instead of starting another one."""
def _gate_collect(self):
"""Block the media collection step until the returned event is set."""
import threading
gate = threading.Event()
calls = []
original = self.api._collect_output_media_sync
def gated(output_dirs):
calls.append(output_dirs)
gate.wait(5)
return original(output_dirs)
self.api._collect_output_media_sync = gated
return gate, calls
async def test_scan_survives_client_disconnect_and_finishes(self):
from unittest.mock import AsyncMock, patch
_write_png(self.output_dir / "gen.png", prompt_text="a red square")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
with self.assertNoLogs("aiohttp.server", level="ERROR"):
with patch.object(
web.StreamResponse,
"write",
AsyncMock(side_effect=ConnectionResetError("closing transport")),
):
resp = await self.client.request("POST", "/prompt_manager/scan")
self.assertEqual(resp.status, 200)
await resp.release()
await asyncio.wait_for(self.api._scan_job.task, 10)
prompts = self.api.db.get_recent_prompts(limit=10)["prompts"]
self.assertEqual([p["text"] for p in prompts], ["a red square"])
self.assertEqual(self.api._scan_job.last_event["type"], "complete")
self.assertFalse(self.api._scan_job.running)
async def test_second_request_attaches_to_the_running_scan(self):
_write_png(self.output_dir / "gen.png", prompt_text="a red square")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
gate, calls = self._gate_collect()
first = asyncio.ensure_future(
self.client.request("POST", "/prompt_manager/scan")
)
await asyncio.sleep(0.05)
second = await asyncio.wait_for(
self.client.request("POST", "/prompt_manager/scan"), 5
)
self.assertEqual(second.status, 200)
self.assertIn("text/event-stream", second.headers["Content-Type"])
gate.set()
first_body = await (await first).text()
second_body = await second.text()
self.assertEqual(len(calls), 1)
self.assertIn('"type": "complete"', first_body)
self.assertIn('"type": "complete"', second_body)
self.assertIn('"added": 1', second_body)
async def test_status_reports_running_scan_and_its_last_event(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
gate, _ = self._gate_collect()
first = asyncio.ensure_future(
self.client.request("POST", "/prompt_manager/scan")
)
await asyncio.sleep(0.05)
resp = await self.client.request("GET", "/prompt_manager/scan/status")
running = await resp.json()
gate.set()
await (await first).text()
resp = await self.client.request("GET", "/prompt_manager/scan/status")
finished = await resp.json()
self.assertEqual(resp.status, 200)
self.assertTrue(running["success"])
self.assertTrue(running["running"])
self.assertEqual(running["last_event"]["type"], "progress")
self.assertFalse(finished["running"])
self.assertEqual(finished["last_event"]["type"], "complete")
async def test_status_before_any_scan_is_idle(self):
resp = await self.client.request("GET", "/prompt_manager/scan/status")
data = await resp.json()
self.assertEqual(resp.status, 200)
self.assertEqual(data, {"success": True, "running": False, "last_event": None})
async def test_scan_after_a_finished_one_starts_fresh(self):
_write_png(self.output_dir / "gen.png", prompt_text="a red square")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
gate, calls = self._gate_collect()
gate.set()
await (await self.client.request("POST", "/prompt_manager/scan")).text()
body = await (await self.client.request("POST", "/prompt_manager/scan")).text()
self.assertEqual(len(calls), 2)
self.assertIn('"linked": 1', body)
async def test_failed_scan_is_not_left_running(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
self.api._collect_output_media_sync = _raise
await (await self.client.request("POST", "/prompt_manager/scan")).text()
data = await (
await self.client.request("GET", "/prompt_manager/scan/status")
).json()
self.assertFalse(data["running"])
self.assertEqual(data["last_event"]["type"], "error")
class TestWorkerThreadsSetting(AdminAPITestCase):
"""The worker-thread count is exposed, validated, persisted and used by the scan."""
async def test_get_settings_reports_worker_threads_and_cpu_count(self):
data = await (
await self.client.request("GET", "/prompt_manager/settings")
).json()
settings = data["settings"]
self.assertEqual(
settings["cpu_count"], PromptManagerConfig.max_worker_threads()
)
self.assertEqual(settings["worker_threads"], PromptManagerConfig.WORKER_THREADS)
self.assertGreaterEqual(settings["worker_threads"], 1)
async def test_save_persists_worker_threads_under_performance(self):
self._write_config({"performance": {"max_search_results": 25}})
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"worker_threads": 1}
)
self.assertEqual(resp.status, 200)
self.assertEqual(PromptManagerConfig.WORKER_THREADS, 1)
saved = self._read_config()
self.assertEqual(saved["performance"]["worker_threads"], 1)
self.assertEqual(saved["performance"]["max_search_results"], 25)
async def test_save_rejects_more_threads_than_cores(self):
before = PromptManagerConfig.WORKER_THREADS
too_many = PromptManagerConfig.max_worker_threads() + 1
resp = await self.client.request(
"POST", "/prompt_manager/settings", json={"worker_threads": too_many}
)
self.assertEqual(resp.status, 400)
self.assertFalse((await resp.json())["success"])
self.assertEqual(PromptManagerConfig.WORKER_THREADS, before)
async def test_scan_reads_metadata_on_several_threads(self):
import threading
import time
for i in range(8):
_write_png(self.output_dir / f"gen{i}.png", prompt_text=f"prompt {i}")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
PromptManagerConfig.WORKER_THREADS = 4
seen = set()
lock = threading.Lock()
original = self.api._extract_comfyui_metadata
def recording(path):
with lock:
seen.add(threading.current_thread().name)
time.sleep(0.05)
return original(path)
self.api._extract_comfyui_metadata = recording
body = await (await self.client.request("POST", "/prompt_manager/scan")).text()
self.assertIn('"added": 8', body)
self.assertGreaterEqual(len(seen), 2)
async def test_scan_with_one_worker_still_completes(self):
_write_png(self.output_dir / "gen.png", prompt_text="solo")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
PromptManagerConfig.WORKER_THREADS = 1
body = await (await self.client.request("POST", "/prompt_manager/scan")).text()
self.assertIn('"added": 1', body)
def _write_mp4(path, comment):
"""Tiny black video with ``comment`` in its container tags (needs ffmpeg)."""
import subprocess
subprocess.run(
[
"ffmpeg",
"-v",
"quiet",
"-f",
"lavfi",
"-i",
"color=c=black:s=16x16:d=0.2",
"-metadata",
f"comment={comment}",
"-y",
str(path),
],
check=True,
timeout=60,
)
class TestScanReadsVideos(AdminAPITestCase):
"""Videos saved by ComfyUI carry the graph in their comment tag."""
def setUp(self):
super().setUp()
from utils import video_metadata
video_metadata.ffprobe_path.cache_clear()
self.addCleanup(video_metadata.ffprobe_path.cache_clear)
@unittest.skipUnless(
shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg not installed"
)
async def test_scan_adds_prompts_found_in_videos(self):
graph = {"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a cat"}}}
_write_mp4(
self.output_dir / "clip.mp4", json.dumps({"prompt": json.dumps(graph)})
)
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
with self.assertNoLogs(level="ERROR"):
body = await (
await self.client.request("POST", "/prompt_manager/scan")
).text()
self.assertIn('"added": 1', body)
prompts = self.api.db.get_recent_prompts(limit=10)["prompts"]
self.assertEqual([p["text"] for p in prompts], ["a cat"])
images = self.api.db.get_prompt_images(prompts[0]["id"])
self.assertTrue(images[0]["image_path"].endswith("clip.mp4"))
async def test_scan_without_ffprobe_skips_videos_quietly(self):
from unittest.mock import patch
from utils import video_metadata
(self.output_dir / "clip.mp4").write_bytes(b"\x00\x00\x00\x18ftypisom")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
with patch.object(video_metadata.shutil, "which", return_value=None):
with self.assertNoLogs("prompt_manager", level="WARNING"):
body = await (
await self.client.request("POST", "/prompt_manager/scan")
).text()
self.assertIn('"processed": 1', body)
self.assertIn('"found": 0', body)
async def test_unreadable_image_is_a_warning_not_an_error(self):
(self.output_dir / "bad.png").write_bytes(b"\x89PNG not really")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
with self.assertLogs(level="WARNING") as cm:
body = await (
await self.client.request("POST", "/prompt_manager/scan")
).text()
self.assertIn('"type": "complete"', body)
self.assertTrue(any("bad.png" in line for line in cm.output))
self.assertEqual([line for line in cm.output if line.startswith("ERROR")], [])
def _sse_payloads(body):
return [
json.loads(line[6:]) for line in body.splitlines() if line.startswith("data: ")
]
class TestDuplicateScanIsABackgroundJob(AdminAPITestCase):
"""The duplicate scan streams progress, survives the client, and is single-flight
with the duplicate delete."""
STREAM = "/prompt_manager/scan_duplicates/stream"
STATUS = "/prompt_manager/scan_duplicates/status"
def _two_dups_one_single(self):
(self.output_dir / "a.png").write_bytes(b"same bytes")
(self.output_dir / "b.png").write_bytes(b"same bytes")
(self.output_dir / "c.png").write_bytes(b"other bytes")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
def _gate_hashing(self):
import threading
gate = threading.Event()
calls = []
original = self.api._calculate_file_hash
def gated(path):
calls.append(path)
gate.wait(5)
return original(path)
self.api._calculate_file_hash = gated
return gate, calls
async def test_stream_reports_progress_then_the_groups(self):
self._two_dups_one_single()
body = await (await self.client.request("POST", self.STREAM)).text()
events = _sse_payloads(body)
types = [e["type"] for e in events]
self.assertIn("progress", types)
self.assertEqual(types[-1], "complete")
done = events[-1]
self.assertEqual(done["processed"], 3)
self.assertEqual(done["found"], 1)
self.assertEqual(done["duplicates"][0]["count"], 2)
self.assertEqual(
sorted(i["filename"] for i in done["duplicates"][0]["images"]),
["a.png", "b.png"],
)
self.assertNotIn(self.tmpdir, body)
async def test_second_request_attaches_and_delete_is_refused_meanwhile(self):
self._two_dups_one_single()
gate, calls = self._gate_hashing()
first = asyncio.ensure_future(self.client.request("POST", self.STREAM))
await asyncio.sleep(0.1)
second = await asyncio.wait_for(self.client.request("POST", self.STREAM), 5)
self.assertEqual(second.status, 200)
delete = await self.client.request(
"POST",
"/prompt_manager/delete_duplicate_images",
json={"image_paths": ["a.png"]},
)
self.assertEqual(delete.status, 409)
running = await (await self.client.request("GET", self.STATUS)).json()
self.assertTrue(running["running"])
gate.set()
first_body = await (await first).text()
second_body = await second.text()
finished = await (await self.client.request("GET", self.STATUS)).json()
self.assertIn('"type": "complete"', first_body)
self.assertIn('"type": "complete"', second_body)
self.assertEqual(len(calls), 3) # one scan, each file hashed once
self.assertFalse(finished["running"])
self.assertEqual(finished["last_event"]["found"], 1)
async def test_status_is_idle_before_any_scan(self):
data = await (await self.client.request("GET", self.STATUS)).json()
self.assertEqual(data, {"success": True, "running": False, "last_event": None})
async def test_failed_scan_streams_an_error_and_releases_the_lock(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
self.api._find_comfyui_output_dir = _raise
body = await (await self.client.request("POST", self.STREAM)).text()
self.api._find_comfyui_output_dir = lambda: str(self.output_dir)
legacy = await self.client.request("GET", "/prompt_manager/scan_duplicates")
self.assertEqual(_sse_payloads(body)[-1]["type"], "error")
self.assertNotIn("boom", body)
self.assertEqual(legacy.status, 200)
def test_hashing_runs_on_worker_threads_and_reports_progress(self):
import threading
import time
for i in range(8):
(self.output_dir / f"f{i}.png").write_bytes(f"bytes {i % 4}".encode())
PromptManagerConfig.WORKER_THREADS = 4
seen = set()
lock = threading.Lock()
original = self.api._calculate_file_hash
def recording(path):
with lock:
seen.add(threading.current_thread().name)
time.sleep(0.05)
return original(path)
self.api._calculate_file_hash = recording
reports = []
groups = self.api._find_duplicate_images_sync(
str(self.output_dir),
progress=lambda done, total: reports.append((done, total)),
)
self.assertEqual(len(groups), 4)
self.assertGreaterEqual(len(seen), 2)
self.assertEqual(reports[-1], (8, 8))
self.assertEqual([t for _, t in reports], [8] * len(reports))
self.assertEqual(sorted(d for d, _ in reports), [d for d, _ in reports])
if __name__ == "__main__":
unittest.main()