The output scan handed every .mp4 to Pillow and logged one ERROR per file.
ComfyUI's video savers store the queued graph as JSON in the container's
comment tag, so those files carry prompts worth linking.
- utils/video_metadata.py reads the comment/description tag through
ffprobe with a timeout, normalised to the PNG text-chunk shape so the
existing parser handles both; without ffprobe videos are skipped and a
single INFO notice is logged
- _extract_comfyui_metadata routes videos there and logs unreadable images
as warnings, since the scan carries on past them
Test hygiene: every aiohttp test base class overrode tearDownAsync without
calling super(), so no test client was ever closed. That produced thousands
of ResourceWarnings, "Unclosed client session" errors from asyncio that
broke log assertions, and doubled the suite's run time.
Both jobs processed one file at a time on a single core. Pillow decodes the
whole image to reach trailing PNG text chunks and releases the GIL while it
does, so a bounded thread pool scales almost linearly: on a 1 GB sample of
400 PNGs the metadata pass dropped from 18 s to 2.5 s at 8 threads, and 120
thumbnails from 4.6 s to 0.6 s.
- utils/parallel.py: map_parallel(func, items, workers) keeps order, runs on
the caller's thread when workers is 1, and caps the pool at the item count
- PromptManagerConfig.WORKER_THREADS defaults to half the detected cores,
is clamped to max_worker_threads() when loaded from config.json, and is
persisted under "performance"
- settings API returns worker_threads and cpu_count and validates saves;
the settings modal gets a "Worker threads" slider capped at the core count
- the scan reads each metadata batch on the configured threads; thumbnail
generation processes batches of files per executor job, still emitting
one progress event per file
Threads rather than processes: this runs inside ComfyUI's server, where
forking a CUDA-initialised process is unsafe and the spawn start method
would re-import ComfyUI's main module in every child.
Closing the dashboard tab or clicking outside the scan modal raised
ClientConnectionResetError out of scan_images and abandoned the scan
mid-batch, leaving a half-scanned database. Two overlapping scans could
also run at once.
- py/api/scan_job.py: ScanJob fans progress out to any number of SSE
subscribers (latest snapshot first, then live events)
- POST /prompt_manager/scan starts the scan as an asyncio task, or attaches
to the one already running; a write to a dead socket ends that stream
quietly while the scan continues
- GET /prompt_manager/scan/status reports the running flag and last event
- dashboard: dismissing the modal while a scan runs minimizes it to a
progress pill, the page re-attaches to a running scan after a reload,
a lost stream reconnects, and leaving the page mid-scan asks first
PromptManagerConfig.save_to_file and the settings endpoint write
config.json through write_private_json: created with mode 0600 (chmod on
POSIX, best effort on Windows) because the file can hold an API key.
_persist_settings merges web_ui and gallery.monitoring.directories into
the existing file instead of overwriting it, so the integrations section
and any other key survive a settings save.
The test-link diagnostic no longer stores a client-supplied path; it
records a marker file name under the output directory.
A root written straight into config.json skipped the containment check
the settings endpoint applies. GalleryConfig.update_config (and so
load_from_file) now drops entries that fail validate_gallery_root with a
warning, _find_comfyui_output_dir validates the configured directory
before using it, and ImageMonitor.start_monitoring validates every
directory before scheduling the observer.
The duplicate cleaner only deletes files with a media extension and
never anything under a thumbnails directory. _collect_media_files walks
with os.walk(followlinks=False), skips symlinked files and directories,
prunes thumbnails, stops MAX_SCAN_DEPTH (12) levels down and after
MAX_SCAN_FILES (50000) files.
Restore: verify_database_file and restore_from_file run through
_run_in_executor; the upload is also opened read-only with
PRAGMA trusted_schema=OFF and refused when sqlite_master lists a trigger
or a view (a prompts database has neither, a crafted one could use them
to run SQL against the live data); at most MAX_SAFETY_BACKUPS (10)
'<db>.backup_*' files are kept next to the live database, oldest first.
Responses: backup_created and the maintenance statistics database_path
go through _public_path, and backup/restore failure messages through
_public_error, so no absolute server path appears in an admin body.
Add _read_json_body(request, max_bytes=1_000_000) to py/api: reads the
body in chunks, answers 413 past the cap (declared or streamed) and 400
for invalid or non-object JSON. Every JSON POST in the admin and logging
handlers goes through it; it is exposed as PromptManagerAPI._read_json_body
for the other route mixins.
Add _acquire_long_job(name) / _long_job_busy_response(label) on the API
object: one asyncio.Lock per job family, handed out as an async context
manager or None when busy, so a handler answers 409 instead of starting a
second scan. The duplicate scan and delete share the 'duplicates' family.
test_image_monitor_singleton replaced utils.comfyui_integration in
sys.modules with a MagicMock and never restored it, so running it before
test_comfyui_integration failed eight tests. The mock was unused; drop it
and install the server stub with setdefault in every owned test module.
The PromptSearchList IS_CHANGED test slept 10 ms between two time.time()
reads; the Windows clock ticks every 15.6 ms, so it was flaky there. Mock
the clock with an incrementing counter instead.
The diagnostics message no longer carries the absolute path, the restore test
closes its raw sqlite connections before the live file is replaced, the logs
tests declare their own folder_paths base directory, and the isolated logger
filters every handler to the test thread.
Route tests for stats, cleanup, duplicate scan/delete, settings,
diagnostics, test-link, maintenance (every operation, success and
failure), the output scan internals, and contract-level tests for
backup/restore (status codes and envelope only).
get_settings located the ImageMonitor singleton by scanning sys.modules
for any module with 'image_monitor' in its name, a workaround for the
package being importable under two identities. It now uses the same
try-package / except-top-level import as the rest of the API
(_current_image_monitor) and falls back to the configured roots when no
monitor is running.
The duplicate scan walked the output tree and hashed every file on the
aiohttp event loop, and the duplicate delete removed files inline; the
output scan ran its per-file database work one executor hop at a time.
- _find_duplicate_images_sync / _describe_media_file do the walk and
hashing; find_duplicate_images() runs them via _run_in_executor.
- _delete_duplicate_images_sync / _delete_one_duplicate do the removal
(absolute or output-relative paths) off the loop.
- The output scan is split into _collect_output_media_sync,
_extract_batch_metadata_sync and _ingest_scan_batch_sync (one
executor call per batch for all DB work) with a small SSE generator.
- Media collection is shared and de-duplicates by normcase'd path.
Settings, diagnostics, duplicate/output scans and the logging endpoints
returned absolute filesystem paths, and several error envelopes echoed
str(e) which for OSError carries the full path.
- _public_path() renders a path relative to the ComfyUI base directory
(folder_paths.base_path, else the output dir's parent) or to the
parent of an opted-in extra gallery root, falling back to the
basename; _public_error() reduces OSError to strerror + basename.
- GET /settings returns gallery roots and monitored directories in that
form; POST /settings resolves relative roots against the same anchors
(GalleryConfig.resolve_gallery_root) so the GET output round-trips.
- Diagnostics output_dirs, monitored_directories and the missing-DB
message no longer carry absolute paths; the image monitor is reached
through the package-or-top-level import fallback instead of a
three-dot relative import that failed outside the package.
- Duplicate scan 'path' is now relative to the output root and the
delete endpoint accepts relative paths.
- /logs/files 'path' and /logs/stats 'log_directory' use the public
form; unexpected errors answer with a generic message.
A gallery root could be set to '/' (or 'C:\') through POST
/prompt_manager/settings because the deny list only matched a few
system prefixes. The root was persisted to config.json and fed into the
image-serving containment check, which then allowed any file on disk to
be read through /prompt_manager/images/serve/.
- GalleryConfig.validate_gallery_root() accepts a directory only when its
realpath lies inside a ComfyUI directory (folder_paths output/input/
temp/user) or a directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS;
filesystem roots, the home directory, files and missing paths are
rejected.
- save_settings returns 400 and persists nothing when any root fails.
- _get_all_output_dirs() filters configured roots through the validator
and never falls back to a rejected root.
- PROMPT_MANAGER_CONFIG_PATH relocates config.json; load/save default
to it.
Vito Sansevero