Commit Graph
64 Commits
Author SHA1 Message Date
Vito Sansevero 08d57b408a feat(dashboard): infinite scrolling for the prompt list, saved default plus session toggle
- PromptManagerConfig.INFINITE_SCROLL (web_ui.infinite_scroll) is reported
  and saved through the settings API; only booleans are accepted
- Settings modal gains an "Infinite Scrolling" default; saving applies it
  at once
- the results header gains an "Infinite scroll" checkbox that overrides the
  default for the current session without saving
- when on, the next page is appended as the list nears its bottom (same
  load-more rule as the gallery), page controls hide, a short page keeps
  loading until the list fills, and a refresh after an edit restarts from
  the top
2026-10-08 18:01:35 -07:00
Vito Sansevero f93412ab90 feat: tag search box autocompletes, narrowed by the tags already typed
- PromptDatabase.suggest_tags(prefix, with_tags, limit): tags starting
  with the prefix, counted over the prompts that carry every context tag,
  context tags excluded, ordered by count then name; LIKE wildcards in the
  prefix are escaped
- GET /prompt_manager/tags/suggest?q=&with=&limit= exposes it (limit
  clamped to 1..50, context capped at 10 tags, registered before the
  {tag_name} routes)
- tag-autocomplete.js splits the comma-separated input into committed tags
  and the one being typed, builds the query, applies a pick and handles
  wrap-around keyboard selection
- the dashboard Tags box shows suggestions with counts on focus and while
  typing, arrow keys and Enter/Tab pick, Escape closes, and each pick
  immediately offers the next tag narrowed by the previous ones
2026-10-08 18:01:35 -07:00
Vito Sansevero c5846306f6 fix(gallery): Rescan Output Folder drops the server listing cache and rewalks
The button only re-requested the listing, which the server answers from a
30-second in-memory cache, so a rescan showed exactly what was already on
screen.

- POST /prompt_manager/gallery/rescan invalidates the gallery cache,
  rewalks every output root and returns the file and root counts
- the button calls it, closes the settings modal, reloads the folder tree
  and the first page, and reports what it found
2026-10-08 18:01:35 -07:00
Vito Sansevero 6e3fa65914 feat(gallery): duplicate scan runs as a background job with progress; infinite scrolling
The duplicate scan was one blocking request with a spinner: no progress,
a modal that could not be put away, and a result lost if the tab closed.

- POST /prompt_manager/scan_duplicates/stream starts the scan as a
  background job (same ScanJob fan-out as the output scan) or attaches to
  the running one; GET /prompt_manager/scan_duplicates/status reports it.
  The job holds the duplicates lock, so deletes are refused while it runs.
- hashing runs on the configured worker threads and reports progress
  about every percent
- the gallery modal shows a progress bar, closing it minimizes to a pill
  that reopens it, results wait in the pill if the modal was closed, and a
  reload re-attaches to a running scan

Infinite scrolling is a new Display Setting (off by default): the next page
is appended when the viewport nears the bottom, pagination controls hide,
and the load-more rule lives in list-state.js where it is unit tested.
Thumb-size buttons and the column setting no longer override each other.
2026-10-08 18:01:35 -07:00
Vito Sansevero 85d1031179 fix: thumbnails are written atomically and served under mtime-versioned urls
Cards showed "Failed to load" for thumbnails that were valid on disk. The
browser had fetched them while the thumbnail run was still writing the
file and, because image responses are cached for an hour, kept the broken
bytes. The card also had no fallback to the original.

- image and video thumbnails are written to a hidden temp sibling and
  renamed into place, so a partial file is never visible at the final path
- every thumbnail_url (gallery listing, prompt images, duplicates, autotag
  listing) carries ?v=<mtime>, so a regenerated thumbnail gets a new URL
  instead of the cached old one
- a 404 from the image serve route is marked no-store; ComfyUI's cache
  middleware would otherwise let browsers cache the miss for an hour
- image-helpers gains fallbackImageSource(): the original is tried once
  before a card gives up

Test hygiene: the autotag tests now stub ComfyUI's server module like the
other API test modules, so the module passes when run on its own.
2026-10-08 18:01:35 -07:00
Vito Sansevero 23c43dd5bf fix: thumbnail runs skip fresh thumbnails before the cap and report what remains
The per-run cap of 10,000 applied to the directory walk itself, so a
library with more files than that never had thumbnails generated for the
rest, and a run that found 9,987 existing thumbnails spent its whole
budget on skipping them.

- _plan_thumbnails walks the whole tree (bounded like the gallery listing),
  counts files whose thumbnail is already newer as skipped without queuing
  them, and caps only the pending files; the leftover count is returned
- start and complete payloads carry skipped and remaining; total_images
  counts every file considered and the processing rate uses the files
  actually worked on
- the gallery page shows a warning naming the pending count instead of a
  success toast when a run stopped at the cap
2026-10-08 18:01:34 -07:00
Vito Sansevero b2ea2597e6 feat: scan reads prompts embedded in videos; unreadable files no longer log errors
The output scan handed every .mp4 to Pillow and logged one ERROR per file.
ComfyUI's video savers store the queued graph as JSON in the container's
comment tag, so those files carry prompts worth linking.

- utils/video_metadata.py reads the comment/description tag through
  ffprobe with a timeout, normalised to the PNG text-chunk shape so the
  existing parser handles both; without ffprobe videos are skipped and a
  single INFO notice is logged
- _extract_comfyui_metadata routes videos there and logs unreadable images
  as warnings, since the scan carries on past them

Test hygiene: every aiohttp test base class overrode tearDownAsync without
calling super(), so no test client was ever closed. That produced thousands
of ResourceWarnings, "Unclosed client session" errors from asyncio that
broke log assertions, and doubled the suite's run time.
2026-10-08 18:01:34 -07:00
Vito Sansevero b98c716b48 feat: worker-thread setting speeds up the output scan and thumbnail generation
Both jobs processed one file at a time on a single core. Pillow decodes the
whole image to reach trailing PNG text chunks and releases the GIL while it
does, so a bounded thread pool scales almost linearly: on a 1 GB sample of
400 PNGs the metadata pass dropped from 18 s to 2.5 s at 8 threads, and 120
thumbnails from 4.6 s to 0.6 s.

- utils/parallel.py: map_parallel(func, items, workers) keeps order, runs on
  the caller's thread when workers is 1, and caps the pool at the item count
- PromptManagerConfig.WORKER_THREADS defaults to half the detected cores,
  is clamped to max_worker_threads() when loaded from config.json, and is
  persisted under "performance"
- settings API returns worker_threads and cpu_count and validates saves;
  the settings modal gets a "Worker threads" slider capped at the core count
- the scan reads each metadata batch on the configured threads; thumbnail
  generation processes batches of files per executor job, still emitting
  one progress event per file

Threads rather than processes: this runs inside ComfyUI's server, where
forking a CUDA-initialised process is unsafe and the spawn start method
would re-import ComfyUI's main module in every child.
2026-10-08 18:01:34 -07:00
Vito Sansevero dc192d5836 fix: output scan runs as a background job that survives the browser going away
Closing the dashboard tab or clicking outside the scan modal raised
ClientConnectionResetError out of scan_images and abandoned the scan
mid-batch, leaving a half-scanned database. Two overlapping scans could
also run at once.

- py/api/scan_job.py: ScanJob fans progress out to any number of SSE
  subscribers (latest snapshot first, then live events)
- POST /prompt_manager/scan starts the scan as an asyncio task, or attaches
  to the one already running; a write to a dead socket ends that stream
  quietly while the scan continues
- GET /prompt_manager/scan/status reports the running flag and last event
- dashboard: dismissing the modal while a scan runs minimizes it to a
  progress pill, the page re-attaches to a running scan after a reload,
  a lost stream reconnects, and leaving the page mid-scan asks first
2026-10-08 18:01:34 -07:00
Vito Sansevero 675168deff fix: private config.json, merged settings save, synthetic test-link path
PromptManagerConfig.save_to_file and the settings endpoint write
config.json through write_private_json: created with mode 0600 (chmod on
POSIX, best effort on Windows) because the file can hold an API key.
_persist_settings merges web_ui and gallery.monitoring.directories into
the existing file instead of overwriting it, so the integrations section
and any other key survive a settings save.

The test-link diagnostic no longer stores a client-supplied path; it
records a marker file name under the output directory.
2026-10-08 18:01:34 -07:00
Vito Sansevero 5dfe7a8409 fix: bound log file size and backup count in the logging config endpoint
max_file_size must be between 1 MB and 1024 MB (bytes) and backup_count
between 0 and 50; anything else is a 400. Also close every database
connection in the logging test teardown.
2026-10-08 18:01:34 -07:00
Vito Sansevero 40dde002a4 fix: validate hand-edited gallery roots everywhere and bound output scans
A root written straight into config.json skipped the containment check
the settings endpoint applies. GalleryConfig.update_config (and so
load_from_file) now drops entries that fail validate_gallery_root with a
warning, _find_comfyui_output_dir validates the configured directory
before using it, and ImageMonitor.start_monitoring validates every
directory before scheduling the observer.

The duplicate cleaner only deletes files with a media extension and
never anything under a thumbnails directory. _collect_media_files walks
with os.walk(followlinks=False), skips symlinked files and directories,
prunes thumbnails, stops MAX_SCAN_DEPTH (12) levels down and after
MAX_SCAN_FILES (50000) files.
2026-10-08 18:01:34 -07:00
Vito Sansevero 2b3cd5cbfb fix: harden database restore and keep server paths out of admin responses
Restore: verify_database_file and restore_from_file run through
_run_in_executor; the upload is also opened read-only with
PRAGMA trusted_schema=OFF and refused when sqlite_master lists a trigger
or a view (a prompts database has neither, a crafted one could use them
to run SQL against the live data); at most MAX_SAFETY_BACKUPS (10)
'<db>.backup_*' files are kept next to the live database, oldest first.

Responses: backup_created and the maintenance statistics database_path
go through _public_path, and backup/restore failure messages through
_public_error, so no absolute server path appears in an admin body.
2026-10-08 18:01:34 -07:00
Vito Sansevero b6f1e0a2c0 feat: capped JSON body reader and single-flight lock for long admin jobs
Add _read_json_body(request, max_bytes=1_000_000) to py/api: reads the
body in chunks, answers 413 past the cap (declared or streamed) and 400
for invalid or non-object JSON. Every JSON POST in the admin and logging
handlers goes through it; it is exposed as PromptManagerAPI._read_json_body
for the other route mixins.

Add _acquire_long_job(name) / _long_job_busy_response(label) on the API
object: one asyncio.Lock per job family, handed out as an async context
manager or None when busy, so a handler answers 409 instead of starting a
second scan. The duplicate scan and delete share the 'duplicates' family.
2026-10-08 18:01:34 -07:00
Vito Sansevero 8413f9e240 fix: static routes resolve under web/ with an extension allow-list
/prompt_manager/lib/{filepath} and /js/{filepath} only rejected '..' and a
leading '/'. On Windows os.path.join(root, 'C:/x') or a leading backslash
re-anchors the path, so any file on the host could be read. Requests are
now refused when the path is absolute on any platform (slash, backslash,
drive letter, UNC prefix), carries an empty, '.' or '..' segment, has an
extension outside the allow-list, or resolves (symlinks included) outside
the static directory; the resolved real path is served with its MIME type.
2026-10-08 18:01:34 -07:00
Vito Sansevero efcbd5a8aa fix: gate the LoraManager serving roots and accept relative link paths
A LoraManager path hand-edited into config.json went straight into
find_lora_directories and so into the image-serving allow-list. When a
path is configured it must now pass lora_utils.resolve_lora_manager_path
(a custom_nodes child whose name contains 'lora'); otherwise, or when the
resolver is unavailable, no LoRA directories are served. An empty path
keeps the custom_nodes-only auto-detection.

POST /images/link now also accepts the relative forms the API hands out
(root-relative 'sub/a.png' and ComfyUI-relative 'output/sub/a.png'),
resolving them against the allowed roots and anchors; '..' segments in
a relative path are refused and unknown files are 404.
2026-10-08 18:01:34 -07:00
Vito Sansevero 2381958fc7 fix: export every prompt by paging instead of truncating at 10000
/prompt_manager/export fetched a single page of 10000 prompts and
silently dropped the rest. It now walks the table in EXPORT_PAGE_SIZE
pages (1000) inside one executor job until a short page is returned.
2026-10-08 18:01:34 -07:00
Vito Sansevero f0e7561fb2 fix: validate tags on every tag mutation route
add_prompt_tag, add_tags_to_prompt, bulk_add_tags, merge_tags and
rename_tag accepted whatever the body carried; a non-string tag raised
AttributeError (500) and over-long or control-character tags were
stored. They now run utils.validators.validate_tags through a shared
tags_error helper and answer 400 with the validator's message. Blank
entries in add_tags_to_prompt are still skipped rather than rejected.
2026-10-08 18:01:34 -07:00
Vito Sansevero 9db5fc6acc fix: decode tag names from the URL only once
The tag lookup, rename and delete routes ran urllib.parse.unquote on
match_info values that aiohttp had already percent-decoded, so a tag
literally named '%41' (requested as %2541) was handled as 'A'. The
second decode is dropped.
2026-10-08 18:01:34 -07:00
Vito Sansevero cd209131ee fix: check thumbnail directory containment with a real path test
clear_thumbnails compared resolved paths with str.startswith, so a
thumbnails directory that resolved to a sibling with a shared prefix
(out vs out2) passed the safety check and was emptied. It now uses the
module's _is_within (realpath + normcase + Path.is_relative_to).
2026-10-08 18:01:34 -07:00
Vito Sansevero e720aeb73a fix: describe images relative to the ComfyUI tree in API responses
Gallery and prompt responses carried absolute server paths: /images/output
entries (path, root_dir), thumbnails_path, the images/prompt lookup, every
image list (image_path plus its normalised file_path duplicate) and the
preview images attached to prompt listings. They now go through
_public_path: /images/output 'path' is relative to its root (what the
delete and autotag endpoints resolve against), 'root_dir' and every
other image_path are relative to the ComfyUI base, and file_path is
dropped. recent/all/search image lists also gain url/thumbnail_url via
the same enrichment the prompt-images route already had. Remaining
str(e) error bodies in both modules use the basename-only error text.
2026-10-08 18:01:34 -07:00
Vito Sansevero 1c37f0ab56 fix: make the thumbnail progress stream POST-only
GET /prompt_manager/images/generate-thumbnails/progress wrote thumbnails
to disk, so any prefetch, link preview or <img src> could start a full
output-directory scan. The route now accepts POST only (GET answers
405); quality still comes from the query string so the SSE client can
keep the same URL shape.
2026-10-08 18:01:34 -07:00
Vito Sansevero 2dbb21edc2 fix: clamp paging on tag listings and the page-derived recent offset
tags/stats, tags/{tag}/prompts and tags/filter parsed limit/offset with a
bare int(), so limit=999999 was passed straight to SQLite and a huge
offset raised OverflowError (500). They now share parse_page_params
(limit 1..500, offset 0..MAX_PAGE_OFFSET, 400 on non-integers), and
/recent clamps the offset it derives from ?page= the same way.
2026-10-08 18:01:34 -07:00
Vito Sansevero f9da6488d6 fix(autotag): contain batch rows, require media extensions, bound the scan
- The batch tagger runs every generated_images row through
  _resolve_allowed_image_path; rows that exist but lie outside the served
  output directories are skipped and counted in skipped_outside_roots on
  progress and complete events, and the engine receives the resolved path.
- _resolve_allowed_image_path requires a media extension (.png .jpg .jpeg
  .webp .gif .bmp .tiff .tif), so autotag/single returns 403 for other files.
- autotag/start holds a module-level lock; a concurrent start gets 409
  {"success": false, "error": "Auto-tagging is already running"}.
- scan_output_dir walks with a 50,000-file and 12-level cap, skips symlinked
  files and thumbnails, returns path through _public_path, relative_path in
  POSIX form, and a truncated flag.
- Autotag handlers report exceptions via the shared _public_error; the
  local copy is removed. Tests close the database before unlinking it.
2026-10-08 18:01:34 -07:00
Vito Sansevero d83383b9e0 fix(lora): confine LoraManager paths to custom_nodes and hide server paths
- is_civitai_url also refuses URLs with userinfo or a non-443 port, so
  https://evil.com\@civitai.com/x and https://civitai.com:8443/x are rejected.
- resolve_lora_manager_path / custom_nodes_directories: lora/enable and
  lora/scan only accept a LoraManager directory that is a direct child of
  ComfyUI's custom_nodes (folder_paths base_path or registered custom_nodes
  folders, else this package's parent) and whose name contains "lora",
  compared via realpath+normcase; relative public paths resolve against the
  API path anchors. Anything else is 400, so find_lora_directories can no
  longer widen the image allow-list with arbitrary trees.
- lora/enable persists through PromptManagerConfig.save_to_file(), which
  honours PROMPT_MANAGER_CONFIG_PATH, instead of a hard-coded config.json.
- detect/status/enable return path and detected_path through _public_path;
  every lora handler reports exceptions via _public_error, and lora/scan
  errors raised after the SSE stream opened become an error event.
- Tests close the database before unlinking it.
2026-10-08 18:01:34 -07:00
Vito Sansevero c8613bdb66 style: wrap long lines in the API layer, autotag and config
Split long f-string log messages, response messages, URL builders and
docstrings; DEFAULT_PROMPT and SYSTEM_PROMPT are rewritten as implicit
concatenations with identical values. Drop the unused shutil import in
admin and the unused HFCacheInfo import in autotag; keep the
hf_hub_download/snapshot_download availability probe with noqa: F401 and
mark the deliberately late imports in config with noqa: E402.
2026-10-08 18:01:34 -07:00
Vito Sansevero 1f422d2c43 fix: restore through SQLite's backup API, close_all for shared files, safer output scans
restore_from_file copied the uploaded file over the live database and deleted
its WAL while other PromptModel instances (nodes, image monitor, queue hook)
still held connections, which corrupts on POSIX and fails on Windows. It now
copies pages through the online backup API into the live connection so every
open handle sees the new content. PromptDatabase.close_all() closes the
connections of every thread, which tests need before deleting a database.
Output scans skip symlinked files, filesystem entries get a stable digest id
instead of a per-process hash(), and thumbnail streaming stops when the
client disconnects.
2026-10-08 18:01:34 -07:00
Vito Sansevero 0393d01746 fix: page image listings in SQL and drop three unused helpers
get_recent_images and search_images_by_prompt take limit/offset so the recent
and search image routes no longer over-fetch a 5000-row window and slice it.
generate_content_hash, PromptModel.migrate_database and find_duplicates had no
callers outside their own tests. The isolated logger tests re-apply their
thread filter after update_config rebuilds the handlers. Long lines in the
touched database and image modules are wrapped so the full flake8 hook passes.
2026-10-08 18:01:34 -07:00
Vito Sansevero 7128957d2c refactor: split diagnostics, maintenance and HTML serving into helpers
- run_diagnostics composes four module-level probes (_diagnose_database,
  _diagnose_dependencies, _diagnose_output_dirs, _diagnose_image_monitor).
- run_maintenance is table-driven: _maintenance_runners maps each
  operation name to a callable and _run_maintenance_operations reports
  per-operation success or a public error message.
- _read_json answers 400 for malformed or non-object JSON bodies in
  settings, duplicate delete, test-link and maintenance instead of 500.
- test_image_link delegates to _link_test_image.
- The four HTML page routes share _serve_html_page.
2026-10-08 18:01:34 -07:00
Vito Sansevero e282f6ea50 fix: clamp the logs limit and reject non-integers
GET /prompt_manager/logs accepted any limit up to 1000 and answered 500
when the value was not an integer. The limit is now clamped to [1, 500]
and a non-integer value is a 400.
2026-10-08 18:01:34 -07:00
Vito Sansevero 879247f1ef refactor: settings reads the image monitor through its module
get_settings located the ImageMonitor singleton by scanning sys.modules
for any module with 'image_monitor' in its name, a workaround for the
package being importable under two identities. It now uses the same
try-package / except-top-level import as the rest of the API
(_current_image_monitor) and falls back to the configured roots when no
monitor is running.
2026-10-08 18:01:34 -07:00
Vito Sansevero 38f4c0af43 perf: run duplicate and output scans off the event loop
The duplicate scan walked the output tree and hashed every file on the
aiohttp event loop, and the duplicate delete removed files inline; the
output scan ran its per-file database work one executor hop at a time.

- _find_duplicate_images_sync / _describe_media_file do the walk and
  hashing; find_duplicate_images() runs them via _run_in_executor.
- _delete_duplicate_images_sync / _delete_one_duplicate do the removal
  (absolute or output-relative paths) off the loop.
- The output scan is split into _collect_output_media_sync,
  _extract_batch_metadata_sync and _ingest_scan_batch_sync (one
  executor call per batch for all DB work) with a small SSE generator.
- Media collection is shared and de-duplicates by normcase'd path.
2026-10-08 18:01:34 -07:00
Vito Sansevero b0e60ec4b8 fix: no absolute server paths in API responses
Settings, diagnostics, duplicate/output scans and the logging endpoints
returned absolute filesystem paths, and several error envelopes echoed
str(e) which for OSError carries the full path.

- _public_path() renders a path relative to the ComfyUI base directory
  (folder_paths.base_path, else the output dir's parent) or to the
  parent of an opted-in extra gallery root, falling back to the
  basename; _public_error() reduces OSError to strerror + basename.
- GET /settings returns gallery roots and monitored directories in that
  form; POST /settings resolves relative roots against the same anchors
  (GalleryConfig.resolve_gallery_root) so the GET output round-trips.
- Diagnostics output_dirs, monitored_directories and the missing-DB
  message no longer carry absolute paths; the image monitor is reached
  through the package-or-top-level import fallback instead of a
  three-dot relative import that failed outside the package.
- Duplicate scan 'path' is now relative to the output root and the
  delete endpoint accepts relative paths.
- /logs/files 'path' and /logs/stats 'log_directory' use the public
  form; unexpected errors answer with a generic message.
2026-10-08 18:01:34 -07:00
Vito Sansevero e7fa993a42 fix: log config accepts only known keys
POST /prompt_manager/logs/config forwarded the whole request body into
PromptManagerLogger.update_config(), so any key could be injected into
the logging configuration. The handler now accepts only LOG_CONFIG_KEYS
(level, max_file_size, backup_count, console_logging, file_logging,
buffer_size), type-checks each value (level name, non-negative int,
bool) and answers 400 listing unknown keys. Malformed or non-object
bodies are 400 instead of 500.
2026-10-08 18:01:34 -07:00
Vito Sansevero c855a1e23b fix: gallery roots are restricted to ComfyUI directories
A gallery root could be set to '/' (or 'C:\') through POST
/prompt_manager/settings because the deny list only matched a few
system prefixes. The root was persisted to config.json and fed into the
image-serving containment check, which then allowed any file on disk to
be read through /prompt_manager/images/serve/.

- GalleryConfig.validate_gallery_root() accepts a directory only when its
  realpath lies inside a ComfyUI directory (folder_paths output/input/
  temp/user) or a directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS;
  filesystem roots, the home directory, files and missing paths are
  rejected.
- save_settings returns 400 and persists nothing when any root fails.
- _get_all_output_dirs() filters configured roots through the validator
  and never falls back to a rejected root.
- PROMPT_MANAGER_CONFIG_PATH relocates config.json; load/save default
  to it.
2026-10-08 18:01:34 -07:00
Vito Sansevero b5720904cd fix: backups include WAL data and restores are integrity-checked
The database runs in WAL mode, so a shutil.copy2 of prompts.db silently
dropped every committed row still in prompts.db-wal; the backup
download and the pre-restore safety copy both did that. Restore also
overwrote the live file with connections open, read the whole upload
into memory before checking the 100 MB cap and never ran
PRAGMA integrity_check.

PromptModel.backup_database now uses the SQLite online backup API,
verify_database_file opens a candidate read-only and checks header,
integrity and the prompts table, and restore_from_file backs up the
live database, closes every thread's connection, copies the file,
removes stale -wal/-shm and re-runs migrations. The admin handlers
stream the upload to a temp file with an early size cutoff, reject
anything verify_database_file refuses with a 400 and keep the
{success: ...} JSON shape. The schema-init connection is now closed
instead of leaking.
2026-10-08 18:01:34 -07:00
Vito Sansevero 2cf6bdd393 refactor: share the internal error message in autotag routes 2026-10-08 18:01:34 -07:00
Vito Sansevero 5c592e4a87 fix: saving LoRA settings without a key keeps the stored CivitAI key 2026-10-08 18:01:34 -07:00
Vito Sansevero a6d4facfbf refactor: autotag model loading is injectable 2026-10-08 18:01:34 -07:00
Vito Sansevero b836e7cef9 fix: autotag/single checks real paths and DB rows against output dirs 2026-10-08 18:01:34 -07:00
Vito Sansevero 22d23020fe fix: CivitAI key is never returned or sent off-host 2026-10-08 18:01:34 -07:00
Vito Sansevero bcb2713baa fix: autotag endpoints no longer read arbitrary paths or run on GET 2026-10-08 18:01:34 -07:00
Vito Sansevero 027fdc09bb fix: cap every request-driven resource in the image routes
- /images/recent, /images/all and /images/output parse limit/offset
  through parse_page_params on every path (absent, 0, negative, float
  strings), and the clamped values are what reach the DB. /images/all
  gets its own MAX_BULK_LIMIT ceiling instead of an unbounded query;
  /images/recent caps its fetch window at MAX_RECENT_IMAGES_WINDOW.
- parse_page_params also clamps offset to MAX_PAGE_OFFSET so values past
  int64 no longer surface as a 500 from SQLite.
- Gallery and thumbnail scans share _iter_media_files: no symlink
  following, MAX_SCAN_DEPTH, and MAX_GALLERY_FILES / MAX_THUMBNAIL_FILES
  caps. Thumbnail size comes only from named presets.
- JSON bodies on /images/link and /images/generate-thumbnails are read
  in bounded chunks; over MAX_JSON_BODY_BYTES answers 413, malformed or
  non-object JSON answers 400.
- /images/link only accepts media paths inside the allowed directories
  (403 otherwise), closing a path-existence oracle.
2026-10-08 18:01:34 -07:00
Vito Sansevero 5fddf56eee perf: thumbnail generation no longer blocks the event loop
generate_thumbnails_with_progress is split into _thumbnail_targets
(scan, executor), _generate_one (one PIL/ffmpeg job, executor) and an
async driver that awaits each file via _run_in_executor, emits one
progress frame per file and yields between files. Per-file failures
are reported as file_error events (error stays fatal for the UI) and
the loop continues. The wildcard Access-Control-Allow-Origin header is
gone. The blocking POST variant reuses the same helpers. The scan never
follows symlinks and is capped by MAX_SCAN_DEPTH / MAX_THUMBNAIL_FILES.
2026-10-08 18:01:34 -07:00
Vito Sansevero 3228824161 fix: bound page sizes on list endpoints
/search, /recent and /images/recent share parse_page_params: limit is
clamped to [1, MAX_PAGE_LIMIT=500], offset to >= 0, and non-integer
values answer 400 instead of 500. /search now honours offset and all
three return a pagination block with the effective limit and offset.
2026-10-08 18:01:34 -07:00
Vito Sansevero 191d02eb6c fix: image routes serve only media files inside allowed directories
serve_image and serve_output_image now reject anything whose extension
is not in IMAGE_EXTENSIONS, fail closed when no output directory is
configured, and answer 403 (not 404) for dot-dot, absolute and
symlink-escaping paths. Containment is checked after realpath+normcase
so it behaves the same on Windows, macOS and Linux. 500 responses use
safe_error_message so OSError text no longer leaks absolute paths.
2026-10-08 18:01:34 -07:00
Vito 6c7e4d5b35 feat: release 3.2.4 — recently used sorting, run counts, positive-prompt image linking (#152)
* chore: start 3.2.4

* feat: track prompt usage (run_count, last_used_at) with server-side sorting

- Migration adds last_used_at and run_count; existing prompts are estimated
  from linked images (count, newest image time, else created_at). The
  backfill only touches NULL rows, so it self-heals rows written after a
  downgrade or an interrupted upgrade.
- Every node run records a use, including re-runs of an existing prompt;
  a tracking failure is logged and never blocks generation.
- search_prompts/get_recent_prompts accept a whitelisted sort key
  (last_used, created, run_count, rating, text); default stays newest first.

Refs #90

* feat: sort param on /recent and /search; microsecond last_used_at

- /prompt_manager/recent and /search accept ?sort= (whitelisted in the DB
  layer; unknown values fall back to newest first).
- last_used_at is written with microseconds from Python so a re-run in the
  same millisecond as another save still sorts first; ordering stays correct
  against millisecond values from the backfill.

Refs #90

* feat: server-side sort in admin UI with Recently Used / Most Used and run badges

- New web/js/prompt-list-sort.js owns the sort options, validation and
  request URLs; admin.js fills the dropdown from it and re-fetches from the
  server on change, so every sort spans all pages (the old client-side sort
  only reordered the current page).
- Default sort is Recently Used; prompt cards show a run-count badge with
  the last-used time on hover.
- The node's Recent button now lists recently used prompts.
- Contract test keeps the UI sort keys and the node's key in sync with the
  server whitelist.

Refs #90

* fix: link images and count runs by the positive prompt, not node execution

With a negative-prompt PromptManager node in the workflow, every executed
PromptManager node pushed onto the image-linking queue, so images could be
linked to the negative prompt or to a previous run. ComfyUI also skips
unchanged nodes, so re-runs of the same prompt were never counted.

- utils/prompt_graph.py finds the PromptManager nodes feeding a sampler's
  positive input (KSampler, CFG/Basic guiders, ControlNet and combine
  pass-throughs, PromptManagerText into encoders), falling back to all
  PromptManager nodes when none can be traced.
- Runs are counted by a ComfyUI on-prompt hook when a workflow is queued;
  prompts not yet saved are counted when the node first saves them.
- Nodes receive the queued graph (hidden PROMPT/UNIQUE_ID). Only the run's
  positive prompt becomes the current prompt; only batch items with linked
  text are pushed onto the FIFO queue.
- The image monitor links from the image's own metadata first, using the
  positive node, and falls back to the queue for batch items.

Refs #90

* fix: address review — count cached runs of linked text, keep repeated pending runs

- Text from known pure string nodes (core PrimitiveString/StringConcatenate,
  WAS Text Concatenate/Multiline, pysssss ShowText) is resolved at queue
  time, so cached re-runs of such prompts are counted and their images link
  from metadata. Unknown nodes are never guessed; the node compares its real
  text with the queue-time resolution and falls back to counting and
  queueing itself on any mismatch.
- A new prompt queued several times before its first run keeps every
  pending run; the first save records all of them.
- Registration flag is a threading.Event; consistent unittest imports;
  logging instead of print in the init fallback.

* refactor: unquote PendingFirstUse annotation so Tuple usage is visible
2026-10-07 13:52:09 -07:00
Vito aa768f57dd fix: release 3.2.3 — metadata parsing, null-safe saves, auto-hide, tag search (#151)
* fix: release 3.2.3 — metadata parsing, null-safe saves, auto-hide, tag search

- fix(#75): stop SaveImage patch from writing the last-run prompt into every
  PromptManager node (negative prompts were saved as copies of the positive)
- fix(#75): replace four duplicated metadata parsers with a shared
  web/js/comfy-metadata.js that traces sampler links (KSampler, CFGGuider /
  SamplerCustomAdvanced, LoRA chains, ControlNet pass-through, any checkpoint
  loader) and recovers per-node text for images saved by older versions
- fix(#143): treat null JSON body fields as empty in prompt API handlers
- fix(#149): keep auto-hide timeout 0 (disabled) instead of falling back to
  the default; validate result_timeout on save
- feat(#150): support tag:name filters in the node search box
- docs(#142): scan dialog mentions A1111/Forge metadata support
- docs(#144): fix ComfyUI-Lora-Manager links
- ci: run JS tests with node --test

* fix: bound metadata graph traversal to prevent tab freeze on crafted PNGs

Link resolution tried every linked input recursively with only a depth cap,
so a PNG whose prompt graph has cycles and fan-out could force ~inputs^25
paths and hang the gallery/metadata page. Each lookup now shares a visited
set (each node expanded once) with a hard cap of 500 visits.

* fix: read the matching output of multi-output settings nodes

KSampler settings linked to a node like SamplerCombo all resolved to its
first value, so steps, cfg and sampler showed the sampler name. Scalar
resolution now prefers the linked node's input with the same field name.

* fix: address review — trust API text, restore workflow-only parsing

- Workflow widget text now only replaces API text for metadata showing the
  pre-3.2.3 corruption signature (2+ PromptManager nodes with identical API
  text but different widgets). Otherwise the API graph wins, so expanded
  dynamic prompts and linked text display what actually ran.
- Images with only a UI workflow are converted to the API graph shape and
  traced the same way: prompts, KSamplerAdvanced, UNET/checkpoint loaders,
  KJNodes Set/Get, Reroute, bypassed and muted nodes.
2026-10-07 11:27:40 -07:00
c01d8e3c12 feat: recursive subfolder filtering + gallery tree navigation (#140)
* feat: recursive subfolder filtering + gallery tree navigation + thumbnail sizing

- Recursive subfolder filtering: selecting an intermediate folder shows all child images
- Gallery tree navigation panel with collapsible folder hierarchy (sticky, resizable)
- Thumbnail size controls (S/M/L) for adjustable grid density
- Fix path collision when multiple gallery roots share overlapping relative paths
- Fix auto-detect mode collapsing nested folders to ambiguous basenames
- Security test suite for path traversal edge cases

Addresses feedback from #126. Fixes code review items P1/P2 from #135.

Co-Authored-By: Kai Bowie <kaigtb@proton.me>

* style: fix Black formatting in test_path_security.py

* feat(gallery): add folder tree sidebar with hierarchical navigation (#126)

- Add collapsible folder tree sidebar to gallery view
- Support hierarchical year/month/date folder navigation
- Include ancestor path expansion for proper tree structure
- Add subfolder filtering to gallery image queries
- Update admin subfolder dropdown with ancestor support
- Fix trailing whitespace in SQL queries

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Kai <noreply@gtb.ai>
Co-authored-by: Kai Bowie <kaigtb@proton.me>
Co-authored-by: Kai <kai@tachi-02.localdomain>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-27 07:58:59 -07:00
Vito 00284d92da feat: LoraManager integration — trigger words & example import (#52) (#136)
* feat: add LoRA utility module for LoraManager integration (#52)

Detection, metadata reading, trigger word cache, and prompt injection
for ComfyUI-Lora-Manager. All functions return empty results when
LoraManager is not installed.

* feat: add IntegrationConfig for opt-in third-party integrations (#52)

* feat(api): add LoraManager integration endpoints (#52)

Detection, enable/disable, scan/import, trigger word lookup,
and cache refresh endpoints under /prompt_manager/lora/*.

* feat: inject LoRA trigger words into prompts at encoding time (#52)

When enabled, scans for <lora:NAME:WEIGHT> tags in prompt text and
appends trigger words from LoraManager metadata. Behind config toggle.

* feat(ui): add LoraManager integration settings and import UI (#52)

Integrations section in settings with auto-detection badge, enable
toggle, trigger word toggle, and Import LoRA Data button.

* fix: detect LoraManager with case-insensitive directory scan (#52)

* fix: simplify LoraManager heuristic to match real install structure (#52)

* fix: find ComfyUI root via folder_paths and handle symlinked installs (#52)

* fix(ui): add pointer-events-none to toggle switch overlays (#52)

The styled div was intercepting clicks meant for the sr-only checkbox
input, preventing toggle switches from being clickable.

* fix: use lora-manager category and tag for imported LoRA data (#52)

* fix: use rglob to find metadata in lora subdirectories (#52)

* feat: discover LoRA dirs from extra_model_paths.yaml and folder_paths (#52)

Only models/loras under the ComfyUI root was checked. Now also parses
extra_model_paths.yaml and uses folder_paths.get_folder_paths('loras')
at runtime to find all configured LoRA directories.

* fix: use example prompts instead of trigger words, serve LoRA preview images (#52)

- Prompt text now uses civitai example prompts (images[].meta.prompt)
  when available, falling back to model name instead of trigger words
- Image serving now allows paths within LoRA directories when the
  integration is enabled, fixing 403 errors on preview images

* fix: handle null civitai field, link all preview images (#52)

- Guard against civitai: null in metadata (was crashing the scan)
- Link all local preview images per LoRA, not just the first
- Add get_civitai_image_urls() for future remote image support

* feat: reimport clears previous lora-manager data first (#52)

Clicking Import LoRA Data now deletes all existing lora-manager
category prompts before scanning, ensuring a clean reimport.

* feat: download civitai example images during LoRA import (#52)

Images from civitai.images[] are downloaded to data/lora_images/ cache
and linked to prompts alongside local preview files. Cached files are
reused on reimport. Image serving allows the cache directory.

* feat: add CivitAI API key setting for authenticated image downloads (#52)

Most civitai example images (especially NSFW) require authentication.
Adds API key field to the Integrations settings panel, passed as
Bearer token when downloading example images.

* feat(ui): add progress modal for LoRA import (#52)

Closes settings modal and shows a dedicated progress modal with
progress bar, status text, and processed/imported counts during
LoRA import. Auto-closes after completion.

* fix: reduce download timeout to 5s, update progress per LoRA (#52)

5K images at 15s timeout was painfully slow. Reduced to 5s fail-fast.
Progress now updates for every LoRA with image count, not every 5th.

* perf: use 512px thumbnails and parallel downloads for LoRA images (#52)

Full-size civitai images averaged 5.6MB each (27GB total for 5K images).
Now requests /width=512/ thumbnails (~50-100KB) and downloads 8 in
parallel per LoRA. Expected speedup: ~100x smaller + 8x parallel.

* fix: resize downloaded images to 512px thumbnails locally (#52)

Civitai CDN returns 401 for /width=N/ thumbnail URLs with API key auth.
Instead, download the original and resize to 512px via PIL before saving.
Reduces disk usage from ~5MB to ~30-50KB per image.

* docs: update README for v3.2.1 LoRA Manager integration (#52)

- Add LoRA Manager integration section with setup guide and CivitAI key docs
- Add folder filter section with rescan note for existing libraries
- Add v3.2.1 changelog entry, split from v3.2.0
- Add WIP notice for LoRA Manager feature
- Update AutoTag section to include WD14 models
- Fix stale references (KikoTextEncode, outdated file structure)
- Remove dated v2 development note
- Add screenshots for settings, integration, and filtered results
- Fix code review items: remove unused constant, add comments to empty excepts
- Bump version to 3.2.1

* test: add unit tests for LoRA integration and folder filter (#52)

- test_lora_utils.py: 32 tests covering trigger word extraction, example
  prompts, image URLs, metadata parsing, cache dir, TriggerWordCache
- test_lora_database.py: 17 tests covering delete_prompts_by_category,
  folder filter search, get_prompt_subfolders, LoRA import workflow
- test_config.py: 7 new IntegrationConfig tests for structure, enable,
  partial update, reset, and roundtrip

* fix: remove unused MagicMock import in test_lora_utils (#52)
2026-04-03 18:26:23 -07:00