Review findings on #153 (Codex):
- the image serve route allowed only three video types while the gallery
lists seven, so .avi/.mkv/.m4v/.wmv cards opened to a 403; the serve
allow-list now covers every gallery video format
- a settings payload with one invalid field still applied the valid ones
before returning 400; the whole payload is validated first and nothing
changes on failure. webui_display_mode is validated too
Third Windows CI run:
- the same output directory spelled twice (different case or a link) was
listed twice in diagnostics and served roots; both dedupe on the
resolved, case-folded path
- the root-less subfolder test expects the "/" form
Code-quality bot: index definitions are (name, target) pairs instead of
implicitly concatenated strings; redundant local asyncio imports and
two trivial lambdas in test stubs removed.
Second CI run on Windows:
- gallery roots are stored resolved and case-folded there; the admin API
tests now compare against that form instead of the mixed-case temp path
- the subfolder fallback for images outside every root also uses "/", and
the test expects it
- a relative diagnostics db path is normalised after joining
- "database is locked" under many writers on a slow disk: connections now
wait up to 30 seconds (connect timeout and busy_timeout), long enough for
a scan or thumbnail run while ComfyUI saves images
The first push of the branch ran the Tests matrix on all three operating
systems. Every Python job failed; the causes were:
- numpy missing in CI: the WD14 engine tests import it (requirements-dev)
- macOS: temp directories resolve through /private; the admin API tests
now use the real path like the gallery roots they compare against
- Windows: the startup banner printed an emoji, which raises
UnicodeEncodeError on a legacy-code-page console and aborted loading
the whole node; the banner is ASCII now and a test loads the package
under a cp1252 stdout
- Windows: gallery subfolders came back with backslashes, which neither
the folder tree nor the search filter understand; they are "/" joined
on every platform. A relative DEFAULT_DB_PATH kept its forward slashes
after being joined to the extension root; it is normalised now
- Windows: sqlite connections opened with `with sqlite3.connect()` are
not closed by the context manager and linger until garbage collection,
which holds a file lock there. Production sites (vacuum, diagnostics,
restore inspection) close explicitly; a restore that swaps the API's
database object closes the previous one; PromptModel keeps a registry
so a shutdown or a test can close every instance
- Windows: served files were still open when their temp directory was
removed (tests now read the body), a log download test compared text
mode newlines, two tests compared paths with a hard-coded separator
tests/open_handles.py makes the Linux run fail the way Windows would:
every database test base checks, right before deleting its files, that
nothing in the process still holds them open.
- PromptDatabase.suggest_tags(prefix, with_tags, limit): tags starting
with the prefix, counted over the prompts that carry every context tag,
context tags excluded, ordered by count then name; LIKE wildcards in the
prefix are escaped
- GET /prompt_manager/tags/suggest?q=&with=&limit= exposes it (limit
clamped to 1..50, context capped at 10 tags, registered before the
{tag_name} routes)
- tag-autocomplete.js splits the comma-separated input into committed tags
and the one being typed, builds the query, applies a pick and handles
wrap-around keyboard selection
- the dashboard Tags box shows suggestions with counts on focus and while
typing, arrow keys and Enter/Tab pick, Escape closes, and each pick
immediately offers the next tag narrowed by the previous ones
link_image_to_prompt stored size, dimensions and format straight from the
request body, so a string such as '"><img src=x>' ended up in INTEGER
columns and was echoed back to the gallery. Size and dimensions are now
coerced with int() and bounded (size 0..10^12, width/height 0..65535),
format is matched against a whitelist of media formats, and anything
else is stored as NULL while the image still links. Non-dict metadata or
file_info no longer aborts the link.
restore_from_file copied the uploaded file over the live database and deleted
its WAL while other PromptModel instances (nodes, image monitor, queue hook)
still held connections, which corrupts on POSIX and fails on Windows. It now
copies pages through the online backup API into the live connection so every
open handle sees the new content. PromptDatabase.close_all() closes the
connections of every thread, which tests need before deleting a database.
Output scans skip symlinked files, filesystem entries get a stable digest id
instead of a per-process hash(), and thumbnail streaming stops when the
client disconnects.
get_recent_images and search_images_by_prompt take limit/offset so the recent
and search image routes no longer over-fetch a 5000-row window and slice it.
generate_content_hash, PromptModel.migrate_database and find_duplicates had no
callers outside their own tests. The isolated logger tests re-apply their
thread filter after update_config rebuilds the handlers. Long lines in the
touched database and image modules are wrapped so the full flake8 hook passes.
search_images_by_prompt interpolated the user's text into LIKE
unescaped, so '50%' also matched '50 percent'. It now goes through the
same escape_like helper as search_prompts. Also drops an unused import
and an unused cursor assignment flagged by pyflakes.
generated_images was UNIQUE(prompt_id, filename) on the basename, so
ComfyUI's same-named files in different date folders were silently
ignored by INSERT OR IGNORE, and the lastrowid == 0 branch after it
never ran. The table now carries file_path (normcase/normpath/abspath
of the image path) with UNIQUE(prompt_id, file_path); the migration
collapses rows that normalise to one file and supersedes the old
basename migration, which would otherwise undo it on every start.
search_prompts and the tag search escaped nothing, so 100% matched
'100 percent' and a_b matched aXb; user text now goes through
escape_like with ESCAPE. save_prompt no longer stamps last_used_at, the
usage backfill runs only on the upgrade that adds the columns, and the
Recently Used / Most Used orders put never-run prompts (NULL) after run
ones instead of first.
Every PromptDatabase() (nodes, image monitor, API, diagnostics) re-ran
table creation and all migrations. Schema init is now cached per
normalised path (class-level set under a lock) and re-run only when the
file disappeared, after a restore, or after reset_schema_cache().
The copy-and-swap migrations drop a stale *_new table left by an
interrupted run, execute inside an explicit transaction with
PRAGMA foreign_keys = OFF (DROP TABLE prompts used to cascade-delete
every generated_images row), roll back cleanly and log and re-raise
sqlite3.Error instead of swallowing it.
The database runs in WAL mode, so a shutil.copy2 of prompts.db silently
dropped every committed row still in prompts.db-wal; the backup
download and the pre-restore safety copy both did that. Restore also
overwrote the live file with connections open, read the whole upload
into memory before checking the 100 MB cap and never ran
PRAGMA integrity_check.
PromptModel.backup_database now uses the SQLite online backup API,
verify_database_file opens a candidate read-only and checks header,
integrity and the prompts table, and restore_from_file backs up the
live database, closes every thread's connection, copies the file,
removes stale -wal/-shm and re-runs migrations. The admin handlers
stream the upload to a temp file with an early size cutoff, reject
anything verify_database_file refuses with a 400 and keep the
{success: ...} JSON shape. The schema-init connection is now closed
instead of leaking.
Node execution, the queue hook and the image monitor all do
get_prompt_by_hash() then save_prompt(). When two of them race on the
same text the second INSERT failed on UNIQUE(hash) and surfaced an
IntegrityError. save_prompt now catches it, re-fetches by hash and
returns the existing id, so eight threads saving one prompt end with a
single row and every record_prompt_use counted.
PromptModel.get_connection() handed every thread the same
check_same_thread=False connection, so a commit or rollback from the
watchdog, executor or queue-hook thread could land on another thread's
half-done write (sqlite3 raised 'bad parameter or other API misuse'
under concurrent saves). Connections now live in threading.local, each
with WAL, Row factory, foreign_keys and busy_timeout, and are tracked so
they can be closed. PromptModel.close() / PromptDatabase.close() close
the calling thread's connection so tests and restores can delete the
file on Windows.
* chore: start 3.2.4
* feat: track prompt usage (run_count, last_used_at) with server-side sorting
- Migration adds last_used_at and run_count; existing prompts are estimated
from linked images (count, newest image time, else created_at). The
backfill only touches NULL rows, so it self-heals rows written after a
downgrade or an interrupted upgrade.
- Every node run records a use, including re-runs of an existing prompt;
a tracking failure is logged and never blocks generation.
- search_prompts/get_recent_prompts accept a whitelisted sort key
(last_used, created, run_count, rating, text); default stays newest first.
Refs #90
* feat: sort param on /recent and /search; microsecond last_used_at
- /prompt_manager/recent and /search accept ?sort= (whitelisted in the DB
layer; unknown values fall back to newest first).
- last_used_at is written with microseconds from Python so a re-run in the
same millisecond as another save still sorts first; ordering stays correct
against millisecond values from the backfill.
Refs #90
* feat: server-side sort in admin UI with Recently Used / Most Used and run badges
- New web/js/prompt-list-sort.js owns the sort options, validation and
request URLs; admin.js fills the dropdown from it and re-fetches from the
server on change, so every sort spans all pages (the old client-side sort
only reordered the current page).
- Default sort is Recently Used; prompt cards show a run-count badge with
the last-used time on hover.
- The node's Recent button now lists recently used prompts.
- Contract test keeps the UI sort keys and the node's key in sync with the
server whitelist.
Refs #90
* fix: link images and count runs by the positive prompt, not node execution
With a negative-prompt PromptManager node in the workflow, every executed
PromptManager node pushed onto the image-linking queue, so images could be
linked to the negative prompt or to a previous run. ComfyUI also skips
unchanged nodes, so re-runs of the same prompt were never counted.
- utils/prompt_graph.py finds the PromptManager nodes feeding a sampler's
positive input (KSampler, CFG/Basic guiders, ControlNet and combine
pass-throughs, PromptManagerText into encoders), falling back to all
PromptManager nodes when none can be traced.
- Runs are counted by a ComfyUI on-prompt hook when a workflow is queued;
prompts not yet saved are counted when the node first saves them.
- Nodes receive the queued graph (hidden PROMPT/UNIQUE_ID). Only the run's
positive prompt becomes the current prompt; only batch items with linked
text are pushed onto the FIFO queue.
- The image monitor links from the image's own metadata first, using the
positive node, and falls back to the queue for batch items.
Refs #90
* fix: address review — count cached runs of linked text, keep repeated pending runs
- Text from known pure string nodes (core PrimitiveString/StringConcatenate,
WAS Text Concatenate/Multiline, pysssss ShowText) is resolved at queue
time, so cached re-runs of such prompts are counted and their images link
from metadata. Unknown nodes are never guessed; the node compares its real
text with the queue-time resolution and falls back to counting and
queueing itself on any mismatch.
- A new prompt queued several times before its first run keeps every
pending run; the first save records all of them.
- Registration flag is a threading.Event; consistent unittest imports;
logging instead of print in the init fallback.
* refactor: unquote PendingFirstUse annotation so Tuple usage is visible
* feat: recursive subfolder filtering + gallery tree navigation + thumbnail sizing
- Recursive subfolder filtering: selecting an intermediate folder shows all child images
- Gallery tree navigation panel with collapsible folder hierarchy (sticky, resizable)
- Thumbnail size controls (S/M/L) for adjustable grid density
- Fix path collision when multiple gallery roots share overlapping relative paths
- Fix auto-detect mode collapsing nested folders to ambiguous basenames
- Security test suite for path traversal edge cases
Addresses feedback from #126. Fixes code review items P1/P2 from #135.
Co-Authored-By: Kai Bowie <kaigtb@proton.me>
* style: fix Black formatting in test_path_security.py
* feat(gallery): add folder tree sidebar with hierarchical navigation (#126)
- Add collapsible folder tree sidebar to gallery view
- Support hierarchical year/month/date folder navigation
- Include ancestor path expansion for proper tree structure
- Add subfolder filtering to gallery image queries
- Update admin subfolder dropdown with ancestor support
- Fix trailing whitespace in SQL queries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Kai <noreply@gtb.ai>
Co-authored-by: Kai Bowie <kaigtb@proton.me>
Co-authored-by: Kai <kai@tachi-02.localdomain>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(search): PromptSearchList crash, filters, and batch output (#52)
- Fix empty list crash (OUTPUT_IS_LIST requires at least one element)
- Add partial tag matching (LIKE instead of exact)
- Add preview and count outputs for batch visibility
- Add skip_multipart filter for Clip_N video prompts
- Add LoRA-only prompt filter
- Collapse newlines for StringOutputList compatibility
* feat(db): add tag_partial parameter for LIKE tag matching (#52)
Allows search_prompts to use LIKE matching for tags instead of exact.
Used by PromptSearchList for discovery-style search.
Existing callers default to exact matching (unchanged behavior).
* feat(tracker): add FIFO prompt queue for batch image linking (#52)
Batch workflows encode all prompts first, then save all images later.
The queue preserves encode order so each image links to the correct
prompt by position (pop on save matches push on encode).
* fix(monitor): multi-strategy image-to-prompt linking (#52)
Previous approach relied on live prompt tracker state, which was
always stale for batch workflows (all images linked to last prompt).
New 5-strategy priority chain:
1. FIFO queue pop (batch-correct by position)
2. Metadata-based DB lookup (extracts prompt from PNG)
3. Prompt snapshot at file-creation time
4. Live prompt tracker
5. Most recent DB prompt fallback
* test: add tests for prompt queue and search filters (#52)
- 6 queue tests: FIFO ordering, batch simulation, exhaustion
- 15 filter tests: newline collapse, Clip_ filter, LoRA-only filter
- Fix _make_tracker helper to include queue fields
* chore: bump version to 3.2.2
* fix: address code review — regex backtracking and queue expiry (#52)
- Replace backtracking-prone LoRA-only regex with subtraction approach
(re.sub removes tags, check if anything remains). Fixes CodeQL alert.
- Add expiry check to pop_next_prompt — skips stale entries to prevent
incorrect linking from accumulated queue entries across sessions.
- Cleanup thread now also prunes expired queue entries.
* feat: add LoRA utility module for LoraManager integration (#52)
Detection, metadata reading, trigger word cache, and prompt injection
for ComfyUI-Lora-Manager. All functions return empty results when
LoraManager is not installed.
* feat: add IntegrationConfig for opt-in third-party integrations (#52)
* feat(api): add LoraManager integration endpoints (#52)
Detection, enable/disable, scan/import, trigger word lookup,
and cache refresh endpoints under /prompt_manager/lora/*.
* feat: inject LoRA trigger words into prompts at encoding time (#52)
When enabled, scans for <lora:NAME:WEIGHT> tags in prompt text and
appends trigger words from LoraManager metadata. Behind config toggle.
* feat(ui): add LoraManager integration settings and import UI (#52)
Integrations section in settings with auto-detection badge, enable
toggle, trigger word toggle, and Import LoRA Data button.
* fix: detect LoraManager with case-insensitive directory scan (#52)
* fix: simplify LoraManager heuristic to match real install structure (#52)
* fix: find ComfyUI root via folder_paths and handle symlinked installs (#52)
* fix(ui): add pointer-events-none to toggle switch overlays (#52)
The styled div was intercepting clicks meant for the sr-only checkbox
input, preventing toggle switches from being clickable.
* fix: use lora-manager category and tag for imported LoRA data (#52)
* fix: use rglob to find metadata in lora subdirectories (#52)
* feat: discover LoRA dirs from extra_model_paths.yaml and folder_paths (#52)
Only models/loras under the ComfyUI root was checked. Now also parses
extra_model_paths.yaml and uses folder_paths.get_folder_paths('loras')
at runtime to find all configured LoRA directories.
* fix: use example prompts instead of trigger words, serve LoRA preview images (#52)
- Prompt text now uses civitai example prompts (images[].meta.prompt)
when available, falling back to model name instead of trigger words
- Image serving now allows paths within LoRA directories when the
integration is enabled, fixing 403 errors on preview images
* fix: handle null civitai field, link all preview images (#52)
- Guard against civitai: null in metadata (was crashing the scan)
- Link all local preview images per LoRA, not just the first
- Add get_civitai_image_urls() for future remote image support
* feat: reimport clears previous lora-manager data first (#52)
Clicking Import LoRA Data now deletes all existing lora-manager
category prompts before scanning, ensuring a clean reimport.
* feat: download civitai example images during LoRA import (#52)
Images from civitai.images[] are downloaded to data/lora_images/ cache
and linked to prompts alongside local preview files. Cached files are
reused on reimport. Image serving allows the cache directory.
* feat: add CivitAI API key setting for authenticated image downloads (#52)
Most civitai example images (especially NSFW) require authentication.
Adds API key field to the Integrations settings panel, passed as
Bearer token when downloading example images.
* feat(ui): add progress modal for LoRA import (#52)
Closes settings modal and shows a dedicated progress modal with
progress bar, status text, and processed/imported counts during
LoRA import. Auto-closes after completion.
* fix: reduce download timeout to 5s, update progress per LoRA (#52)
5K images at 15s timeout was painfully slow. Reduced to 5s fail-fast.
Progress now updates for every LoRA with image count, not every 5th.
* perf: use 512px thumbnails and parallel downloads for LoRA images (#52)
Full-size civitai images averaged 5.6MB each (27GB total for 5K images).
Now requests /width=512/ thumbnails (~50-100KB) and downloads 8 in
parallel per LoRA. Expected speedup: ~100x smaller + 8x parallel.
* fix: resize downloaded images to 512px thumbnails locally (#52)
Civitai CDN returns 401 for /width=N/ thumbnail URLs with API key auth.
Instead, download the original and resize to 512px via PIL before saving.
Reduces disk usage from ~5MB to ~30-50KB per image.
* docs: update README for v3.2.1 LoRA Manager integration (#52)
- Add LoRA Manager integration section with setup guide and CivitAI key docs
- Add folder filter section with rescan note for existing libraries
- Add v3.2.1 changelog entry, split from v3.2.0
- Add WIP notice for LoRA Manager feature
- Update AutoTag section to include WD14 models
- Fix stale references (KikoTextEncode, outdated file structure)
- Remove dated v2 development note
- Add screenshots for settings, integration, and filtered results
- Fix code review items: remove unused constant, add comments to empty excepts
- Bump version to 3.2.1
* test: add unit tests for LoRA integration and folder filter (#52)
- test_lora_utils.py: 32 tests covering trigger word extraction, example
prompts, image URLs, metadata parsing, cache dir, TriggerWordCache
- test_lora_database.py: 17 tests covering delete_prompts_by_category,
folder filter search, get_prompt_subfolders, LoRA import workflow
- test_config.py: 7 new IntegrationConfig tests for structure, enable,
partial update, reset, and roundtrip
* fix: remove unused MagicMock import in test_lora_utils (#52)
* refactor(api): add multi-root output dir infrastructure (#126)
- Change _gallery_cache from None to {} (dict keyed by path string)
- Update invalidate_gallery_cache() to clear dict instead of setting None
- Add _get_all_output_dirs() helper as single source of truth for
resolving configured + fallback output directories
- Update _enrich_prompt_images() to iterate all output roots when
computing relative paths and thumbnail URLs
* feat(api): support multiple gallery scan paths in settings endpoint (#126)
* feat(ui): dynamic multi-path settings with add/remove buttons (#126)
* cleanup: remove dead galleryRootPath assignment in loadSettings
* feat(api): multi-directory gallery scan with subfolder filter and multi-root serve (#126)
* feat(db): add folder filter to search_prompts and get_prompt_subfolders (#126)
* feat(api): add subfolders endpoint and folder filter to prompt search (#126)
* feat(ui): add folder dropdown filter to search bar (#126)
* fix: escape LIKE wildcards in folder filter and remove dead _gallery_cache_time field
* feat(scan): scan all media types across all output dirs with batched extraction (#126)
* chore: gitignore SQLite WAL-mode temp files
* fix: enrich prompt images with URLs and check all output dirs for serving
- Extract _enrich_images() for flat image lists, used by get_prompt_images
- get_prompt_images now adds url/thumbnail_url (fixes stuck "Loading images...")
- serve_image checks all output dirs not just first (fixes 403 on multi-path)
- Fix diagnostics monitor check to read current module attribute, not stale import
* feat(ui): add version-scoped update notice popup for rescan (#126)
* chore: bump version to 3.2.0
* fix: resolve database path from config.json (#119)
PromptDatabase() was hardcoded to "prompts.db" relative to CWD,
ignoring the DEFAULT_DB_PATH setting from config.json. Added
_resolve_db_path() that reads from PromptManagerConfig when available
and resolves relative paths against the extension root directory.
This fixes Docker deployments where the CWD differs from the extension.
* test: add unit tests for database path resolution (#119)
Cover explicit absolute/relative paths, config fallback, relative
config paths resolved to extension root, and PromptDatabase constructor
integration with _resolve_db_path.
* docs: update database path configuration instructions (#119)
Replace outdated advice to edit py/config.py with correct config.json
approach. Add Docker-specific guidance for absolute paths.
* chore: bump version to 3.1.5
* fix: remove unused import and variables from test (#119)
Address code review: remove unused MagicMock import and unused db
local variables in constructor tests.
Phase 5 scalability and polish:
- Add tags + prompt_tags junction tables replacing JSON column
- Migrate existing JSON tags via json_each() on startup
- Rewrite all tag queries to use junction tables (O(1) vs O(n))
- Simplify rename/delete/merge tag operations to single SQL
- Batch-attach preview images to prompt list responses (fix N+1)
- Fix fetchone() fragility under thread contention (defensive defaults)
- Add Google-style docstrings to main node files (prompt_manager.py, prompt_manager_text.py)
- Enhance database module docstrings with detailed parameter and return documentation
- Add comprehensive docstrings to all utils modules with usage examples
- Document API endpoints and configuration classes thoroughly
- Update module-level docstrings in all __init__.py files
- Ensure consistent naming (ComfyUI_PromptManager) across all documentation
- Follow Python best practices for docstring formatting
Fix compatibility issue with older SQLite versions that don't support
ORDER BY clause inside GROUP_CONCAT function.
- Remove ORDER BY from GROUP_CONCAT in SQL queries
- Add GROUP_CONCAT for created_at dates
- Sort IDs manually in Python after fetching results
- Maintain same behavior of keeping oldest prompt as primary
This resolves the startup error:
"sqlite3.OperationalError: near 'ORDER': syntax error"
The fix ensures compatibility with all SQLite versions while preserving
the original functionality of keeping the oldest duplicate.
Vito Sansevero