nodes/ComfyAssets-ComfyUI_PromptManager
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
600 Commits 259 Branches 7 Tags
704db4be8adef93506e184a48901e214c21e56fb
Commit Graph
104 Commits
Author SHA1 Message Date
Vito Sansevero 50989a7bd6 Merge remote-tracking branch 'origin/main' into feat/482-delete-paired 2026-10-29 16:34:20 -07:00
Vito e52cc39cd5 fix(images): background jobs skip files hidden by the removal mark; walkers skip trash folders inside output roots (#501) (#502) 
* fix(images): background jobs skip files hidden by the removal mark; walkers skip trash folders inside output roots (#501)

- Thumbnail jobs and the legacy scan_output_dir listing leave
  library-removed files out (one mark check, mark_hides).
- Missing-file links also forgets removal marks that hide nothing any
  more (file gone from a folder that exists, or changed), only under a
  readable, non-empty source, re-checked at apply.
- The sidebar Images, Favorites and Needs review counts and the
  collection counts are read again when an image_delete job ends.
- utils/media_folders.py: one skip for thumbnails and trash folders
  (.Trash, .Trash-<uid>, .Trashes, $RECYCLE.BIN) used by every walker,
  the image monitor and the delete review.

* test(browser): wait for the answers two specs raced in the merge queue (#501)

- System diagnostics "level filter": wait for each level's own log
  answer, and poll the shown levels; the CRITICAL line is in the
  previous level's answer too, so it proved nothing (firefox).
- Library editor "restored storage after a confirmed save": wait for
  the PATCH answer, which waits for a whole restore in the test's
  route, instead of the 5 s expect default (webkit).
 2026-10-29 22:30:32 +00:00
Vito Sansevero b968730cc4 Merge commit 'f87f2c43' into feat/482-delete-paired 2026-10-29 14:29:54 -07:00
Vito d02a911748 feat(library): delete or archive selected prompts, Show archived prompts and Restore (#443) (#470) 
* feat(library): set the category on many selected prompts (#437)

The Prompt library shelf gets the concept's Images selection checkbox on each
card and its .bulkbar (N selected, Set category, Clear), recorded as D148.
Set category opens the shared one-value dialog and posts
{prompt_ids, category, expected_epoch} to POST /prompt_manager/bulk/category:
the server checks the storage epoch the shelf was read under, then sets the
category on the Library parents in one BEGIN IMMEDIATE transaction, writing
only rows that change. A restored storage answers 409 storage-changed and
writes nothing. The legacy admin and images_filter bodies are unchanged.

This replaces the legacy admin page's bulk Set Category, a cutover
prerequisite (#334).

Closes #437

* feat(library): add tags to many selected prompts and select every loaded prompt (#442)

The Library selection bar (D148) gets Add tags and select-all of the loaded
shelf, the legacy admin page's bulk Add Tags and select-all checkbox.

Add tags posts {prompt_ids, tags, expected_epoch} to
POST /prompt_manager/bulk/tags: in one BEGIN IMMEDIATE transaction the server
checks the storage epoch, normalizes the names as image tags are, gives each
Library parent only the names it lacks (casefold), finds or inserts tag rows
by exact name only when some prompt gets them, and skips prompts that would
pass 20 tags. Prompt tags only. Both bulk/tags forms refuse a non-object body
with 400.

Ctrl/Cmd+A in the grid and "Select all N loaded" in the bar select every
loaded prompt; the offer follows each loaded batch.

Closes #442

* feat(vocabulary): tag search, tag sort and an Untagged prompts entry (#444)

Tags & categories gains a server-side tag search and sort (keyset pages,
cursor scoped to the query) and an "Untagged prompts" row whose Browse opens
the Library with tag_mode=untagged. Refine this view gets Tag match, saved
Library views keep it, and the routes echo the applied query so a stale
server is an error, never an unfiltered list. Recorded as D149.

Refs #444

* feat(library): filter prompts by output folder and browse tags with Any (#444)

Refine this view on the Prompt library gains the concept's Folder (folders
below an output root, read when the dialog opens) and a Tag match select
(All / Any). The listing takes folder and tag_mode, matches folders on the
normalized file_path under every spelling of each output root, echoes both
so a stale server is an error, and saved Library views keep them. Recorded
as D150.

Refs #444

* docs(library): record Add tags and select-all as D151 (#442)

* feat(library): archive paired and delete unpaired selected prompts on the server (#443)

A prompt_archive table marks Library parents hidden from the shelf; older
builds ignore it. New routes /prompt_manager/library/delete/preview,
/library/delete and /library/restore classify a selection (delete,
archive, protected, already archived, not found), execute exactly what was
reviewed under a records hold, take a verified safety backup (disk checked)
before any delete, and restore archived prompts. The shelf, search and the
shelf-facing counts honor the archive (?shelf=active|archived); archiving
repairs a stale prompt hash so scans find the archived row.

* feat(library): Delete… on the selection bar, Show archived prompts and Restore (#443)

The Library bar's Delete… previews the selection and opens the shared
review: what is deleted (in no saved pair), archived (paired: hidden,
restorable) or kept (tagged __protected__), the safety backup and what
cannot be undone. Show archived prompts (N) lists archived prompts with
Restore on their bar; an archived prompt's page shows an Archived notice
with Restore. TESTING.md records D152; the route snapshot gains the three
workspace routes.

* test(library): cover the Delete… review failure and close-only paths; record coverage floors (#443)

* fix(library): let saved-view defaults fill the missing #444 filters

libraryViewState passed `?? 'all'` and `?? ''` fallbacks that
normalizeViewState already applies to an undefined value. The redundant
branches were never taken in the page, which dropped saved-views.mjs below
its 95% branch floor. A unit now covers a shelf without the new filters.

Refs #444

* fix(library): lay out a loaded page of prompts without waiting for an animation frame (#443)

A starved renderer (headless WebKit under CPU load, a background tab) can
hold requestAnimationFrame for seconds. The shelf laid out each loaded page
only in its next frame, so the grid stayed empty after the first load or a
delete's reload while the caption already showed the new count. A loaded
batch is now laid out at once; measurements still flush per frame.

The regression test holds every animation frame and fails before the fix.

* test(saved-views): give the D153 Library view the tag_mode key (#444)

The Library saved-view state now holds tag_mode, and the API requires
every key, so main's new "Highest rated" spec posts it too.

Refs #444

* test(saved-views): give the D153 Library view the folder key (#444)

Refs #444

* fix(library): name the saved pair, not negative variants, as the archive reason (#443)

A prompt in a saved pair with no negative variants was described in the
Delete… review as "1 prompt with negative variants: hidden, restorable".
Paired prompts are archived because of the pair, so the Archive row now
reads "N prompts in a saved pair: hidden, restorable" and the first
sentence explains the pair. Found in the live test of #470.

* fix(library): nest Refine folders by path segment; backslash separates only on Windows (#444)

Live test B1: the Folder list indented a server list sorted as full-path
text, so "-" and "_" sorted before "/" and children landed under the wrong
parent (kiko/char under kiko-v6), with case variants interleaved. The
options are now a tree built by segment: each folder directly under its own
parent, case variants distinct siblings.

On POSIX a folder named "Metadata\Klein..." was listed but filtered to no
prompts, because normalize_folder read "\" as a separator. It now does so
only on Windows (ntpath), as the listing already did.

Refs #444

* fix(library): say why an archived prompt is archived, and name variants only when it has any (#443)

The archived prompt page's notice always said "This prompt and its
negative variants are hidden", also for a saved-pair prompt with none.
It now says the prompt is archived because it is in a saved pair, and
counts its negative variants only when there are any. From the live
test of #470.
 2026-10-29 21:09:22 +00:00
Vito Sansevero f87f2c43d0 Merge remote-tracking branch 'origin/main' into feat/443-bulk-prompt-delete 
# Conflicts:
#	database/models.py
#	py/api/jobs.py
 2026-10-29 13:38:17 -07:00
Vito 66ca988d2a feat(images): delete selected images, from the library or to the trash (#480) (#488) 
* feat(trash): a freedesktop trash backend that renames without replacing or refuses (#480)

* feat(images): remove a file's records and hide it with a removal mark (#480)

Deletes every record of one file under every recorded spelling (image links,
tags, auto-tag marks, pending review items, category, collection memberships,
favorite) in one transaction. A Needs review item that is kept keeps its
annotation rows (ON DELETE RESTRICT). The image_removals table hides an exact
file (locator, size, mtime_ns) until it changes; linking skips a marked file,
and an explicit tag, category, favorite or collection write shows it again.

* feat(images): review and apply Delete images as a reviewed plan job (#480)

POST /prompt_manager/images/delete/preview reviews both modes; the
maintenance apply takes {token, mode}. Library mode removes the records and
writes the removal mark; trash mode moves the file first and removes its
records only after a successful move. The workspace listing, the output scan,
the duplicate find and Tag output images skip marked files.

* feat(images): Delete… in the Images selection bar, inspector and viewer (#480)

The concept's Remove links… place, D159: one review per selection that asks
each time (nothing preselected) whether to remove the files from the library
only or also move them to the trash. The browser test server uses a fake
trash.

* test(images): portable delete tests, the native no-trash proof, D159 in TESTING.md (#480)

* perf(images): cache the removal marks so a listing page reads no extra table (#480)

The marks reload only when this process writes or clears one, or a restore
replaces the database. The route snapshot gains the preview route.

* test(images): the full inspector specs expect D159's Delete… in the panel and the viewer (#480)

* fix(images): the delete review starts nothing without a chosen mode, and dims its primary until then (#480)

An aria-disabled primary ignores every activation, a chosen one starts once,
and the apply helper refuses a missing mode. Tests cover a forced click, a
double click, Enter and Space on the primary, Enter in the dialog and a
submit event, and the server refusing an apply without an exact mode.
 2026-10-29 19:40:26 +00:00
Vito Sansevero e47f05e6be feat(library): delete a prompt in a saved pair permanently, with its own pairs (#482, server) 
The Delete review's preview now marks which archive ids may be deleted
permanently (in a saved pair, and nobody else's negative), with a review
digest of the prompt and its pairs and the counts the review shows. The
write accepts an optional delete_paired_ids with their digests. Each is
deleted with its own saved pairs (image-pair links, then pairs, then the
prompt as #443 deletes it) only while its digest is unchanged. Any drift
archives it and counts it changed. Paired deletes are backed up and
disk-checked like deletes.
 2026-10-29 12:24:57 -07:00
Vito Sansevero 52531a8be4 Merge remote-tracking branch 'origin/main' into feat/443-bulk-prompt-delete 
# Conflicts:
#	database/library.py
#	web/js/workspace/library-state.mjs
#	web/js/workspace/library.mjs
 2026-10-29 07:41:30 -07:00
Vito 0126705045 feat(library): filter prompts by output folder and browse tags with Any (#444, PR 2) (#456) 
* feat(vocabulary): tag search, tag sort and an Untagged prompts entry (#444)

Tags & categories gains a server-side tag search and sort (keyset pages,
cursor scoped to the query) and an "Untagged prompts" row whose Browse opens
the Library with tag_mode=untagged. Refine this view gets Tag match, saved
Library views keep it, and the routes echo the applied query so a stale
server is an error, never an unfiltered list. Recorded as D149.

Refs #444

* feat(library): filter prompts by output folder and browse tags with Any (#444)

Refine this view on the Prompt library gains the concept's Folder (folders
below an output root, read when the dialog opens) and a Tag match select
(All / Any). The listing takes folder and tag_mode, matches folders on the
normalized file_path under every spelling of each output root, echoes both
so a stale server is an error, and saved Library views keep them. Recorded
as D150.

Refs #444

* fix(library): let saved-view defaults fill the missing #444 filters

libraryViewState passed `?? 'all'` and `?? ''` fallbacks that
normalizeViewState already applies to an undefined value. The redundant
branches were never taken in the page, which dropped saved-views.mjs below
its 95% branch floor. A unit now covers a shelf without the new filters.

Refs #444

* test(saved-views): give the D153 Library view the tag_mode key (#444)

The Library saved-view state now holds tag_mode, and the API requires
every key, so main's new "Highest rated" spec posts it too.

Refs #444

* test(saved-views): give the D153 Library view the folder key (#444)

Refs #444

* fix(library): nest Refine folders by path segment; backslash separates only on Windows (#444)

Live test B1: the Folder list indented a server list sorted as full-path
text, so "-" and "_" sorted before "/" and children landed under the wrong
parent (kiko/char under kiko-v6), with case variants interleaved. The
options are now a tree built by segment: each folder directly under its own
parent, case variants distinct siblings.

On POSIX a folder named "Metadata\Klein..." was listed but filtered to no
prompts, because normalize_folder read "\" as a separator. It now does so
only on Windows (ntpath), as the listing already did.

Refs #444
 2026-10-29 14:10:29 +00:00
Vito 89d94cdb2b fix(history): join case-variant raw text to the history of the prompt it counts toward (#460) (#471) 
* fix(history): join outer-whitespace raw text to the history of the prompt it saves (#438)

The Library stores text.strip() while a run keeps its exact raw text. The
history window, run detail scope id, newest pair run and prompt hints now
compare the raw text's stored form (one rule, utils/hashing.py, mirrored in
SQLite with the same code points). Hints also keep matching the exact raw
text, so older rows that kept outer whitespace stay hinted and protected.

* fix(history): join case-variant raw text to the history of the prompt it counts toward (#460)

The prompt hash folds case, so a run of "A Lighthouse" counts toward the
saved "a lighthouse" but never joined its history. History now compares the
hash's own key (prompt_key = strip + str.lower()); SQL calls Python for it
through pm_prompt_key/pm_prompt_hash registered on every managed connection,
since SQLite's lower() folds ASCII only. Hints match by text first (#438) and
fall back to the dedup hash only when no text matches, so no existing hint
becomes ambiguous. A history position from before this change still resumes.

* test(history): a real-host case-variant re-queue joins the prompt's history (#460)
 2026-10-29 13:25:04 +00:00
Vito Sansevero e9d01c917b Merge commit '917aa838cbe1d2f6f2c335885096812a28626cce' into feat/443-bulk-prompt-delete 2026-10-29 05:25:16 -07:00
Vito Sansevero 917aa838cb fix(library): nest Refine folders by path segment; backslash separates only on Windows (#444) 
Live test B1: the Folder list indented a server list sorted as full-path
text, so "-" and "_" sorted before "/" and children landed under the wrong
parent (kiko/char under kiko-v6), with case variants interleaved. The
options are now a tree built by segment: each folder directly under its own
parent, case variants distinct siblings.

On POSIX a folder named "Metadata\Klein..." was listed but filtered to no
prompts, because normalize_folder read "\" as a separator. It now does so
only on Windows (ntpath), as the listing already did.

Refs #444
 2026-10-29 04:36:37 -07:00
Vito 83199518df fix(history): join outer-whitespace raw text to the history of the prompt it saves (#438) (#465) 
The Library stores text.strip() while a run keeps its exact raw text. The
history window, run detail scope id, newest pair run and prompt hints now
compare the raw text's stored form (one rule, utils/hashing.py, mirrored in
SQLite with the same code points). Hints also keep matching the exact raw
text, so older rows that kept outer whitespace stay hinted and protected.
 2026-10-29 11:32:53 +00:00
Vito 5e00ee479f feat(vocabulary): tag search, tag sort and an Untagged prompts entry on Tags & categories (#444, PR 1) (#452) 
* feat(vocabulary): tag search, tag sort and an Untagged prompts entry (#444)

Tags & categories gains a server-side tag search and sort (keyset pages,
cursor scoped to the query) and an "Untagged prompts" row whose Browse opens
the Library with tag_mode=untagged. Refine this view gets Tag match, saved
Library views keep it, and the routes echo the applied query so a stale
server is an error, never an unfiltered list. Recorded as D149.

Refs #444

* test(saved-views): give the D153 Library view the tag_mode key (#444)

The Library saved-view state now holds tag_mode, and the API requires
every key, so main's new "Highest rated" spec posts it too.

Refs #444
 2026-10-29 11:32:48 +00:00
Vito Sansevero 78993dcb4d Merge #456 (feat/444-library-folder-or) into feat/443-bulk-prompt-delete 
Keeps both features: #444's folder filter, tag match (all/any/untagged),
tag search and sort with #443's archive/delete and Show archived shelf.
The ranked tag sort and the Untagged prompts count now hide archived
parents, as the other tag counts and the shelf already do.
 2026-10-29 00:43:07 -07:00
Vito Sansevero a6927489a8 Merge remote-tracking branch 'origin/main' into feat/444-library-folder-or 
# Conflicts:
#	TESTING.md
 2026-10-28 19:17:22 -07:00
Vito Sansevero 53cfd8b035 Merge branch 'feat/444-tags-page-search' into feat/444-library-folder-or 
# Conflicts:
#	TESTING.md
#	web/js/workspace/library.mjs
 2026-10-28 18:41:52 -07:00
Vito 02190d03a8 feat(library): add tags to many selected prompts and select every loaded prompt (#442) (#453) 
* feat(library): set the category on many selected prompts (#437)

The Prompt library shelf gets the concept's Images selection checkbox on each
card and its .bulkbar (N selected, Set category, Clear), recorded as D148.
Set category opens the shared one-value dialog and posts
{prompt_ids, category, expected_epoch} to POST /prompt_manager/bulk/category:
the server checks the storage epoch the shelf was read under, then sets the
category on the Library parents in one BEGIN IMMEDIATE transaction, writing
only rows that change. A restored storage answers 409 storage-changed and
writes nothing. The legacy admin and images_filter bodies are unchanged.

This replaces the legacy admin page's bulk Set Category, a cutover
prerequisite (#334).

Closes #437

* feat(library): add tags to many selected prompts and select every loaded prompt (#442)

The Library selection bar (D148) gets Add tags and select-all of the loaded
shelf, the legacy admin page's bulk Add Tags and select-all checkbox.

Add tags posts {prompt_ids, tags, expected_epoch} to
POST /prompt_manager/bulk/tags: in one BEGIN IMMEDIATE transaction the server
checks the storage epoch, normalizes the names as image tags are, gives each
Library parent only the names it lacks (casefold), finds or inserts tag rows
by exact name only when some prompt gets them, and skips prompts that would
pass 20 tags. Prompt tags only. Both bulk/tags forms refuse a non-object body
with 400.

Ctrl/Cmd+A in the grid and "Select all N loaded" in the bar select every
loaded prompt; the offer follows each loaded batch.

Closes #442

* docs(library): record Add tags and select-all as D151 (#442)
 2026-10-29 01:30:58 +00:00
Vito Sansevero f30db7713d feat(library): archive paired and delete unpaired selected prompts on the server (#443) 
A prompt_archive table marks Library parents hidden from the shelf; older
builds ignore it. New routes /prompt_manager/library/delete/preview,
/library/delete and /library/restore classify a selection (delete,
archive, protected, already archived, not found), execute exactly what was
reviewed under a records hold, take a verified safety backup (disk checked)
before any delete, and restore archived prompts. The shelf, search and the
shelf-facing counts honor the archive (?shelf=active|archived); archiving
repairs a stale prompt hash so scans find the archived row.
 2026-10-28 18:26:27 -07:00
Vito Sansevero 567b06ac48 Merge remote-tracking branch 'origin/main' into feat/444-tags-page-search 
# Conflicts:
#	TESTING.md
#	tests/js/workspace-library-state.test.js
#	web/js/workspace/library-state.mjs
#	web/js/workspace/library.mjs
 2026-10-28 18:06:52 -07:00
Vito Sansevero a543ecff6f Merge remote-tracking branch 'origin/main' into feat/442-bulk-prompt-tags 
# Conflicts:
#	TESTING.md
#	database/library_commands.py
#	py/api/prompts.py
#	tests/browser/library-concept.spec.mjs
#	tests/browser/workspace-library-selection.spec.mjs
#	tests/js/workspace-library-selection.test.js
#	web/js/workspace/library-selection.mjs
#	web/js/workspace/library.mjs
 2026-10-28 17:53:48 -07:00
Vito ac378f07d3 feat(library): set the category on many selected prompts (#437) (#446) 
The Prompt library shelf gets the concept's Images selection checkbox on each
card and its .bulkbar (N selected, Set category, Clear), recorded as D148.
Set category opens the shared one-value dialog and posts
{prompt_ids, category, expected_epoch} to POST /prompt_manager/bulk/category:
the server checks the storage epoch the shelf was read under, then sets the
category on the Library parents in one BEGIN IMMEDIATE transaction, writing
only rows that change. A restored storage answers 409 storage-changed and
writes nothing. The legacy admin and images_filter bodies are unchanged.

This replaces the legacy admin page's bulk Set Category, a cutover
prerequisite (#334).

Closes #437
 2026-10-29 00:31:40 +00:00
Vito Sansevero 9f55f0cc79 Merge feat/444-tags-page-search into feat/444-library-folder-or 
Stack PR 2 on PR 1 so one restart covers both. The shared Library
tag_mode now takes all, any and untagged on the route, the filter, saved
views and Refine's Tag match; untagged still refuses tags and a folder
combines with every mode.

Refs #444
 2026-10-28 17:31:11 -07:00
Vito Sansevero 6372bc7c63 feat(library): filter prompts by output folder and browse tags with Any (#444) 
Refine this view on the Prompt library gains the concept's Folder (folders
below an output root, read when the dialog opens) and a Tag match select
(All / Any). The listing takes folder and tag_mode, matches folders on the
normalized file_path under every spelling of each output root, echoes both
so a stale server is an error, and saved Library views keep them. Recorded
as D150.

Refs #444
 2026-10-28 17:15:25 -07:00
Vito Sansevero d4d99b7e18 feat(vocabulary): tag search, tag sort and an Untagged prompts entry (#444) 
Tags & categories gains a server-side tag search and sort (keyset pages,
cursor scoped to the query) and an "Untagged prompts" row whose Browse opens
the Library with tag_mode=untagged. Refine this view gets Tag match, saved
Library views keep it, and the routes echo the applied query so a stale
server is an error, never an unfiltered list. Recorded as D149.

Refs #444
 2026-10-28 17:07:50 -07:00
Vito Sansevero ebd43a8140 feat(library): add tags to many selected prompts and select every loaded prompt (#442) 
The Library selection bar (D148) gets Add tags and select-all of the loaded
shelf, the legacy admin page's bulk Add Tags and select-all checkbox.

Add tags posts {prompt_ids, tags, expected_epoch} to
POST /prompt_manager/bulk/tags: in one BEGIN IMMEDIATE transaction the server
checks the storage epoch, normalizes the names as image tags are, gives each
Library parent only the names it lacks (casefold), finds or inserts tag rows
by exact name only when some prompt gets them, and skips prompts that would
pass 20 tags. Prompt tags only. Both bulk/tags forms refuse a non-object body
with 400.

Ctrl/Cmd+A in the grid and "Select all N loaded" in the bar select every
loaded prompt; the offer follows each loaded batch.

Closes #442
 2026-10-28 17:04:38 -07:00
Vito Sansevero 72df65113a feat(library): set the category on many selected prompts (#437) 
The Prompt library shelf gets the concept's Images selection checkbox on each
card and its .bulkbar (N selected, Set category, Clear), recorded as D148.
Set category opens the shared one-value dialog and posts
{prompt_ids, category, expected_epoch} to POST /prompt_manager/bulk/category:
the server checks the storage epoch the shelf was read under, then sets the
category on the Library parents in one BEGIN IMMEDIATE transaction, writing
only rows that change. A restored storage answers 409 storage-changed and
writes nothing. The legacy admin and images_filter bodies are unchanged.

This replaces the legacy admin page's bulk Set Category, a cutover
prerequisite (#334).

Closes #437
 2026-10-28 16:17:23 -07:00
Vito 4040007264 fix(workspace): keep open pages valid across a restart and recover a changed source (#430) (#434) 
The gallery root scope and the Images listing cursor were keyed with a
per-process random key, so every ComfyUI restart made an open tab's
scopes answer 409 source-changed (blank Tag image preview, a Generate
that did nothing) and its cursors fail Load more.

- utils/workspace_keys.py: one random secret in workspace-scope.key
  beside the database (owner-only, never config.json or the database),
  two derived keys; per-process fallback with a warning. Move library
  copies it.
- Cursor v2 binds the listing scope in its MAC instead of sending an
  unkeyed hash of the root paths; an unverifiable cursor answers 400
  cursor-stale and Images offers Refresh instead of a dead Retry.
- POST /prompt_manager/images/rescope maps a stale target to its
  current root only when the held scope was issued for it.
- Images finds a renumbered file again from the inspector, the viewer
  and the Tag image dialog (the dialog stays open, draft kept); nothing
  else is kept by an old root_index:path key.
- Tag image dialog: a state that blocks Save shows its reason above the
  actions with Refresh, disabled actions look disabled, and an empty
  preview says "Preview unavailable" (D147).
 2026-10-28 23:06:36 +00:00
Vito 56d0a4904c feat(collections): COLLECTIONS sidebar, collection page and saved views (#331, PR 2) (#410) 
* feat(saved-views): keep a collection in a saved view

Images saved views gain a collection key (#331 D140): empty or a collection
id of 1 to 9 digits. Writes refuse favorites together with a collection;
older rows read an empty collection, and a stored row with both opens the
collection, as the listing does for a URL with both.

Part of #331

* feat(collections): store collections in the library database

* feat(collections): add, remove and rename collections and keep members through duplicate cleanup

* feat(collections): add collection routes and the Images collection filter

* feat(images): store images' own categories in the library database (#332)

One new table, image_categories, and its category index, created empty
by a forward-only migration that reads and writes only its own objects.
Startup checks the structure; restore also checks the values. The store
sets, clears and reads own categories by file key (the deepest output
source holding the file and the path below it), and copies a removed
duplicate's category to a kept copy that has none.

* feat(images): duplicate cleanup keeps images' own categories (#332)

The kept copy's own category wins; a kept copy without one receives the
removed copy's, keyed by the deepest output source holding each file.
The removed copy's row is never deleted, so a give-back or a failed
delete loses nothing. The job log reports how many were kept.

* feat(images): set, remove and list images' own categories (#332)

PUT /prompt_manager/images/category sets the own category of up to 500
listed files, DELETE removes it (listed or not), and
GET /prompt_manager/images/categories lists the values whose file is
listed. Targets are the Images listing identity of an output source,
checked by media_target (shared with Collections). The routes never
touch prompts, recipes or links; POST /prompt_manager/bulk/category is
unchanged.

* fix(images): key symlinked output folders and keep key work off the event loop (#332)

Review fixes: a source matches by its configured spelling too, so a
detected output folder that is a symlink keys its files (routes and
duplicate cleanup). File keys are computed in the executor. A path no
source holds is a plain 403, a lone surrogate a 400. The install repairs
a table that lacks its category index instead of refusing to open.

* fix(collections): address review findings

* feat(images): set or remove a category on all N matching images (#332)

The category routes also accept the Images listing filter of "Select
all N matching" (#392) instead of explicit items, resolved now and
capped at 500 images like an explicit selection.

* feat(collections): browse a collection from the sidebar

The sidebar gains the concept's COLLECTIONS group (D136): one row per
collection with its folder icon, name and listed-member count, without the
PROPOSED badge; the label shows once a collection exists. A row opens
#/collection/<id>, the Images page filtered to that collection (D137): the
collection's name as h1, Collection as breadcrumb and title, Edit to rename
or delete it through the saved views' Edit dialog with collection nouns, and
a note counting members the sources do not list now. An unknown or deleted
collection shows a page that says so with a link to Images. A saved view on
a deleted collection lists nothing and says so (MC6, D140). The list is read
again after a restore and the counts 300 ms after a change.

Part of #331

* test(collections): compare the COLLECTIONS group with the concept

The concept hides COLLECTIONS at 950 px and below, which would leave a phone
no way to open a collection; production shows the rows there in the wrapped
layout of every other sidebar nav (D136). The shell comparison now compares
the group instead of filtering it out: the concept fixture seeds Landscapes
(12) and Experiments (8), the rows compare by name order, and at phone width
the nav compares against the concept's own sidebar-nav layout. The
collection rows are checked at 1600x1100 and 390x844 in both themes.

Part of #331

* test(vendor): count the #332 host case in the execution matrix

image-attribution.spec.mjs gains the image category route case, so the
execution config lists 92 tests (3 attribution cases per project).

* test(browser): install the slideshow clock before wrapping setTimeout

The slideshow fixture wrapped window.setTimeout before page.clock.install(),
so the viewer's timer ran on real time and clock.runFor() did not drive it;
a late WebKit timer failed 'slideshow exposes exact state labels' on CI.

* fix(collections): address review findings

- A list read superseded by a newer one (the 300 ms count read, a restore)
  no longer shows "could not be loaded": the collection and saved-view
  pages wait for the list to settle.
- A listing that finds the open collection gone shows the deleted page once
  the list no longer holds it; when the list read fails or still holds it,
  the page keeps the listing's error and Retry instead of staying blank, and
  each failed read is reported again.
- The missing-members note follows every list read and goes away at zero.
- A rename that settles after its Edit dialog closed leaves any dialog
  opened since alone.
- The deleted-collection, list-failed and saved-view failure pages focus
  their title after a route change, as every page does.

Part of #331

* test(coverage): keep main's floors for files this PR does not touch

coverage:update raised nine floors of untouched files from one run;
metadata-autocomplete.js branches then measured 84.84 against 85.

* test(legacy): record the image category routes in the legacy contract snapshot (#332)

#332 adds PUT/DELETE /prompt_manager/images/category and
GET /prompt_manager/images/categories; the legacy pages' routes are
unchanged.

* test(duplicates): a removed copy with a collection and a category moves both (#331, #332)
 2026-10-28 21:45:09 +00:00
Vito 39877202b2 feat(ingestion): one prompt decision for scan and monitor; scans link each dataset branch's own prompt (#247, PR 1) (#422) 
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386)

When a file's own graph leaves the prompt open, both inspect routes ask
recorded run history about the file (one run reported it and it was written
during that run). The save node that run reported it from anchors the walk
like a file-name match, so dataset branches that share one saver prefix
show their own prompt, labelled "Recorded run". Still open at one sampler
whose PromptManager text was recorded at execution (a language model's
output), the recorded text shows as "Execution recorded". Results the file
resolves alone never change; a failed lookup judges the file alone.

* fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386)

Review follow-ups: the texts PromptManager recorded are looked up at the
sampler history's saver leads to, never at a sampler the unanchored walk
named; a file name naming a different saver than history anchors nothing;
outputs and texts are read in one transaction; "Recorded run" ranks below
every workflow-inferred source; unexpected lookup errors log at error level.

* fix(history): look up recorded runs by history's own spelling of the file (#386)

The history route inspects a case-normalized target on Windows, while
history matches file names exactly, so the occurrence's reference names
the file.

* refactor(inspector): share the inspector's resolve as inspection_resolution

inspect_prompt_evidence now calls inspection_resolution, which returns the
resolution to use, the anchored result and whether the recorded saver chose
it. Ingestion can call the same helper for files the default profile cannot
decide (#247).

* feat(ingestion): one per-file prompt decision for the scan and the monitor

utils/file_prompt.py holds decide_file_prompt, the ladder the output scan
runs today plus the inspector's rule (inspection profile, then the recorded
saver) before the legacy parser, with a SCAN and a MONITOR policy. Junk
text (blank, numbers, LoRA tags only, model file names) is never stored.
graph_prompt_text and parse_comfyui_prompt move here; the parser accepts
parsed metadata (#247).

* feat(ingestion): store a file's prompt in one transaction, one prompt per file

database/scanned_prompts.py links a file to the prompt of one text, creating
it when new, inside one BEGIN IMMEDIATE transaction: a file linked to
another prompt is a conflict, a failed link leaves no prompt behind and two
writers store one prompt. pair_ingestion gains file_link_rows,
link_conflict and Marks/mark_created; ingest_resolved_pair takes optional
marks for the components it creates. recorded_runs and named_prompt in
run_attribution become public (#247).

* feat(scan): decide each scanned file with the shared ladder (#247)

_ingest_scanned_file calls decide_file_prompt (scan policy) and stores the
result: pairs through ingest_resolved_pair and positives through
store_scanned_prompt, both with the scan marks. Files the default profile
leaves undecided now get the inspector's resolve, with the saver recorded
history names, before the legacy parser, so each branch of a shared-prefix
dataset run links its own prompt. Junk text is never stored, a file linked
to another prompt is a conflict instead of a second link, and a failed link
leaves no prompt behind. _BatchHistory reads the stored prompt and the
recorded saver from one recorded_runs pass. _parse_comfyui_prompt calls the
moved parse_comfyui_prompt. TESTING.md documents the decision.

* fix(ingestion): keep the file-name saver anchor out of the ladder (#390)

The #390 ruling queues a file that only its saver's file name resolves for
Needs review, never linking it. inspection_resolution gains name_anchor;
the ladder passes False, so such a file falls to the legacy floor in the
scan (and to Needs review) and is never stored by the monitor, while the
recorded saver still decides. The inspector keeps the name anchor.

* fix(ingestion): no joined PromptManager text from the legacy floor; junk keeps its step's status

The legacy parser returns prepend + text + append, so its answer is refused
when a PromptManager-family node has a prepend or append part (#343). Junk
found at the undecided-graph step keeps that step's unlinked status and
review, so an ambiguous file is still queued for Needs review. The monitor
policy's ambiguous file no longer asks for the recorded run, and the file
size may be read lazily.

* fix(ingestion): keep the legacy answer that is a PromptManager node's own text

The joined-text refusal also refused the #170 parity files whose
PromptManager prepend is a link the browser parser does not follow: its
answer is then the node's literal text, which is what the node stores.
Refuse only an answer that is not such a node's own text.

* fix(scan): one prompt per file for recorded-run links; writes survive a library move

The recorded run's stored prompt now links in one transaction with the file's
link rows, so a file linked to another prompt is a conflict instead of a
second link. The scan's stores run through survive_library_switch. A single
file asks history once (a one-file _BatchHistory), the scan passes
in_output=False and reads the file size only for files with prompt
metadata. A conflict row reads "Kept: already linked to a different
prompt". TESTING.md records the rules.

* fix(ingestion): match a file's link rows under its resolved spelling too

The scan stores realpath spellings and the monitor the watcher's raw path,
so a file reached through a symlinked or junction output folder could get a
second generated_images row. file_link_rows now queries the path and its
realpath (each normalized and raw) and dedupes by row id. TESTING.md notes
the reverse order is matched only when the raw spelling is the realpath.

* fix(ingestion): a switched-off PromptManager node behind a pass-through stores nothing (#90)

The #90 graph walk follows only prompt-like inputs, so a PromptManagerText
with Save to library off reaching the encoder through another input name
(easy showAnything.anything, a custom node's value) was invisible to it,
and the inspector's resolve stored its text. The ladder now checks the
positive every step names: a resolved or inspected positive whose path
passes a node that is off, or a legacy answer that is such a node's own
text, makes the file not_saved.

* fix(ingestion): any switched-off node upstream of a positive stores nothing (#90)

The ladder walks every input upstream of each sampler or guider positive
(not negative, model, clip, latent or image inputs) for PromptManager nodes
whose Save to library is off or wired from an unreadable node, so the
sampler-less step, an off node with a prepend part and a PromptManagerPair
behind a pass-through no longer store its text. The legacy-floor guard reads
a node's resolved text. Every guard reports not_saved for a switch that is
off and not_saved_wired for one it cannot read.

* fix(ingestion): the switch walk skips negative inputs by #90's own rule

The upstream walk treated only names containing "negative" as negative, so
an off node on a neg, neg_text or text_neg input blocked an enabled
positive. prompt_graph gains is_negative_input (the _NEGATIVE_KEY rule, also
used by follows_input) and the walk uses it. The guards accept a node with
null inputs and a side whose provenance is not an object.

* fix(ingestion): the switch walk also skips any input named with "negative"

#90's rule matches only whole underscore-separated words, so negative2,
negativeprompt and NegativePrompt counted as positive inputs and an off
node wired only to one of them blocked the file. The walk skips those too;
follows_input is unchanged.
 2026-10-28 20:45:15 +00:00
Vito 4f37af209c feat(collections): store collections and add their routes and Images filter (#331, PR 1) (#400) 
* feat(collections): store collections in the library database

* feat(collections): add, remove and rename collections and keep members through duplicate cleanup

* feat(collections): add collection routes and the Images collection filter

* feat(images): store images' own categories in the library database (#332)

One new table, image_categories, and its category index, created empty
by a forward-only migration that reads and writes only its own objects.
Startup checks the structure; restore also checks the values. The store
sets, clears and reads own categories by file key (the deepest output
source holding the file and the path below it), and copies a removed
duplicate's category to a kept copy that has none.

* feat(images): duplicate cleanup keeps images' own categories (#332)

The kept copy's own category wins; a kept copy without one receives the
removed copy's, keyed by the deepest output source holding each file.
The removed copy's row is never deleted, so a give-back or a failed
delete loses nothing. The job log reports how many were kept.

* feat(images): set, remove and list images' own categories (#332)

PUT /prompt_manager/images/category sets the own category of up to 500
listed files, DELETE removes it (listed or not), and
GET /prompt_manager/images/categories lists the values whose file is
listed. Targets are the Images listing identity of an output source,
checked by media_target (shared with Collections). The routes never
touch prompts, recipes or links; POST /prompt_manager/bulk/category is
unchanged.

* fix(images): key symlinked output folders and keep key work off the event loop (#332)

Review fixes: a source matches by its configured spelling too, so a
detected output folder that is a symlink keys its files (routes and
duplicate cleanup). File keys are computed in the executor. A path no
source holds is a plain 403, a lone surrogate a 400. The install repairs
a table that lacks its category index instead of refusing to open.

* fix(collections): address review findings

* feat(images): set or remove a category on all N matching images (#332)

The category routes also accept the Images listing filter of "Select
all N matching" (#392) instead of explicit items, resolved now and
capped at 500 images like an explicit selection.

* test(vendor): count the #332 host case in the execution matrix

image-attribution.spec.mjs gains the image category route case, so the
execution config lists 92 tests (3 attribution cases per project).

* test(browser): install the slideshow clock before wrapping setTimeout

The slideshow fixture wrapped window.setTimeout before page.clock.install(),
so the viewer's timer ran on real time and clock.runFor() did not drive it;
a late WebKit timer failed 'slideshow exposes exact state labels' on CI.

* test(legacy): record the image category routes in the legacy contract snapshot (#332)

#332 adds PUT/DELETE /prompt_manager/images/category and
GET /prompt_manager/images/categories; the legacy pages' routes are
unchanged.

* test(duplicates): a removed copy with a collection and a category moves both (#331, #332)
 2026-10-28 18:49:06 +00:00
Vito cdb771d722 feat(images): filter, inspect and count by each image's effective category (#332, PR 2) (#406) 
* feat(images): store images' own categories in the library database (#332)

One new table, image_categories, and its category index, created empty
by a forward-only migration that reads and writes only its own objects.
Startup checks the structure; restore also checks the values. The store
sets, clears and reads own categories by file key (the deepest output
source holding the file and the path below it), and copies a removed
duplicate's category to a kept copy that has none.

* feat(images): duplicate cleanup keeps images' own categories (#332)

The kept copy's own category wins; a kept copy without one receives the
removed copy's, keyed by the deepest output source holding each file.
The removed copy's row is never deleted, so a give-back or a failed
delete loses nothing. The job log reports how many were kept.

* feat(images): set, remove and list images' own categories (#332)

PUT /prompt_manager/images/category sets the own category of up to 500
listed files, DELETE removes it (listed or not), and
GET /prompt_manager/images/categories lists the values whose file is
listed. Targets are the Images listing identity of an output source,
checked by media_target (shared with Collections). The routes never
touch prompts, recipes or links; POST /prompt_manager/bulk/category is
unchanged.

* feat(vocabulary): count and rename image categories (#332)

Tags & categories lists every category value with image_count, the
stored image_categories rows holding it. Values only images hold follow
every prompt-backed value (first_pid at SQLite's INTEGER maximum), so
keyset cursors cross the boundary. /library/options keeps its own query
and never lists image-only values.

Manage category counts image rows (impact.images), treats an image-held
target as existing, renames image rows in the same transaction, allows
renaming an image-only value, and hashes the sorted image rowids into
the impact token so a change between preview and apply is refused with
vocabulary-conflict. Every image-category statement reads
idx_image_categories_category.

* feat(images): the category filter matches each image's effective category (#332)

An image's own category wins; without one it inherits its link's: the
recipe's category when set, else the positive prompt's (MC4, one rule
with the inspector chip). Ambiguous and unlinked images inherit nothing,
so an unlinked image with its own category is now listed. The link read
no longer filters by category in SQL; the page filter, select all N
matching and saved views compare the effective value. A failed
own-category read answers the existing listing-failed 500.

* fix(images): key symlinked output folders and keep key work off the event loop (#332)

Review fixes: a source matches by its configured spelling too, so a
detected output folder that is a symlink keys its files (routes and
duplicate cleanup). File keys are computed in the executor. A path no
source holds is a plain 403, a lone surrogate a 400. The install repairs
a table that lacks its category index instead of refusing to open.

* feat(inspector): show an image's own category in the inspector and History (#332)

The inspector's library facts gain category_scope "image": an image's
own category overrides the inherited one for any link status (linked,
ambiguous, none). Images reads it for output sources only; History reads
it once per inspection inside its snapshot, keyed by the same file key
from the symlink-resolved output folder. A failed read hides the chip
rather than show a possibly wrong value. The chip's hidden prefix names
the origin ("Image category: "); the visible origin text comes in PR 3.

* perf(images): keep the category filter's link read narrow; record D141, D142, D145 (#332)

Under a category the link read keeps only links whose raw recipe or
positive category is the value (Python still decides the inherited
one), the links of images whose own category is the value are read by
path, and plain reads skip the inherited-category work. The own-category
map is keyed by normalized path. TESTING.md records D141, D142 and D145
and amends D13, D36 and D39; the filter budget seeds 2,000 own
categories.

* feat(images): set or remove a category on all N matching images (#332)

The category routes also accept the Images listing filter of "Select
all N matching" (#392) instead of explicit items, resolved now and
capped at 500 images like an explicit selection.

* fix(images): Windows path keys for own categories; plan check for the category read (#332)

Review fixes: on Windows the own-category map keys each file with
normalize_image_path (abspath drops trailing dots and spaces), and the
plan test also covers the category's narrowing clause.

* test(vendor): count the #332 host case in the execution matrix

image-attribution.spec.mjs gains the image category route case, so the
execution config lists 92 tests (3 attribution cases per project).

* test(legacy): record the image category routes in the legacy contract snapshot (#332)

#332 adds PUT/DELETE /prompt_manager/images/category and
GET /prompt_manager/images/categories; the legacy pages' routes are
unchanged.
 2026-10-28 17:19:11 +00:00
Vito 11422d4584 perf(images): read prompt links in one SQLite step (#354) (#418) 
* perf(images): read prompt links in one SQLite step (#354)

get_image_link_matches streamed one sqlite3_step per linked row, and
Python's sqlite3 releases and retakes the GIL around every step. While
ComfyUI generates, each retake waits on the busy thread, so the
first-page link read of the workspace Images page stretched from tens of
milliseconds to seconds.

SQLite now returns the unchanged filter SELECT's rows as one JSON value
(json_group_array), decoded with json.loads: a constant two or three GIL
hand-backs instead of one per row. Nothing is cached. Any
OperationalError from the wrapped statement, or a BLOB file_path or
rating, makes that read stream as before with one warning per process; a
missing JSON function stops the attempt for the rest of the process.

tests/test_database_link_matches.py proves the result equals the
streamed read on every filter combination and on NULL ratings, quoted,
backslashed, non-ASCII and multi-line paths, ambiguous and pair links,
an empty database and both fallbacks; the structural budget test also
checks the wrapped query plan.

* fix(images): keep the maintenance link reader, refine the one-step fallback (#354)

The one-step reader was named _link_rows, which shadowed
MaintenanceOperations._link_rows on PromptDatabase and broke the Missing
links preview and apply; it is now _one_step_link_rows, and a test pins
that the maintenance reader is not shadowed.

An interrupted read (Move library retiring the connection) is raised as
before instead of falling back and spending the one warning; a JSON text
over SQLite's length limit (DataError) streams like the other value
failures. The equivalence test compares value types per path rather than
by row order, and a test checks that the read leaves no WAL snapshot
pinned.

* docs(testing): name every one-step link read fallback (#354)
 2026-10-28 16:00:15 +00:00
Vito b5a56bee0b feat(images): store images' own categories and set them by route (#332, PR 1) (#401) 
* feat(images): store images' own categories in the library database (#332)

One new table, image_categories, and its category index, created empty
by a forward-only migration that reads and writes only its own objects.
Startup checks the structure; restore also checks the values. The store
sets, clears and reads own categories by file key (the deepest output
source holding the file and the path below it), and copies a removed
duplicate's category to a kept copy that has none.

* feat(images): duplicate cleanup keeps images' own categories (#332)

The kept copy's own category wins; a kept copy without one receives the
removed copy's, keyed by the deepest output source holding each file.
The removed copy's row is never deleted, so a give-back or a failed
delete loses nothing. The job log reports how many were kept.

* feat(images): set, remove and list images' own categories (#332)

PUT /prompt_manager/images/category sets the own category of up to 500
listed files, DELETE removes it (listed or not), and
GET /prompt_manager/images/categories lists the values whose file is
listed. Targets are the Images listing identity of an output source,
checked by media_target (shared with Collections). The routes never
touch prompts, recipes or links; POST /prompt_manager/bulk/category is
unchanged.

* fix(images): key symlinked output folders and keep key work off the event loop (#332)

Review fixes: a source matches by its configured spelling too, so a
detected output folder that is a symlink keys its files (routes and
duplicate cleanup). File keys are computed in the executor. A path no
source holds is a plain 403, a lone surrogate a 400. The install repairs
a table that lacks its category index instead of refusing to open.

* feat(images): set or remove a category on all N matching images (#332)

The category routes also accept the Images listing filter of "Select
all N matching" (#392) instead of explicit items, resolved now and
capped at 500 images like an explicit selection.

* test(vendor): count the #332 host case in the execution matrix

image-attribution.spec.mjs gains the image category route case, so the
execution config lists 92 tests (3 attribution cases per project).

* test(legacy): record the image category routes in the legacy contract snapshot (#332)

#332 adds PUT/DELETE /prompt_manager/images/category and
GET /prompt_manager/images/categories; the legacy pages' routes are
unchanged.
 2026-10-28 15:29:41 +00:00
Vito 5583159497 feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) (#396) 
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386)

When a file's own graph leaves the prompt open, both inspect routes ask
recorded run history about the file (one run reported it and it was written
during that run). The save node that run reported it from anchors the walk
like a file-name match, so dataset branches that share one saver prefix
show their own prompt, labelled "Recorded run". Still open at one sampler
whose PromptManager text was recorded at execution (a language model's
output), the recorded text shows as "Execution recorded". Results the file
resolves alone never change; a failed lookup judges the file alone.

* fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386)

Review follow-ups: the texts PromptManager recorded are looked up at the
sampler history's saver leads to, never at a sampler the unanchored walk
named; a file name naming a different saver than history anchors nothing;
outputs and texts are read in one transaction; "Recorded run" ranks below
every workflow-inferred source; unexpected lookup errors log at error level.

* fix(history): look up recorded runs by history's own spelling of the file (#386)

The history route inspects a case-normalized target on Windows, while
history matches file names exactly, so the occurrence's reference names
the file.
 2026-10-28 13:29:29 +00:00
Vito 534c95248a fix(review): anchor files on their saver in the scan and Needs review (#390) (#414) 
* fix(review): anchor files on their saver in the scan and Needs review (#390)

The scan and Apply now resolve with the file's name, as the workspace
inspector does since #383, so the one saver whose filename_prefix names
the file anchors its branch. A file whose branch that anchor resolves is
never queued (from the inspection profile it offers no choice; a
default-ambiguous file asks saver_anchored) and its pending, undecided
items leave on the next scan. REVIEW_RULE_VERSION is folded into
inspection item versions so old items are re-evaluated. No match, several
matches or an unfinished walk anchor nothing. Apply refuses a choice the
anchor contradicts.

* fix(review): refuse a default choice the saver anchor contradicts (#390)

Apply re-resolves a default item with the file's name: when its saver
anchors the branch, no candidate is offered and the choice is refused.
The scan's anchor check skips bytes carrying a decision and remembers
unanchored file states per process, so a rescan does not inspect an
unchanged file again.

* fix(review): queue an anchored file with its saver's pair as the choice (#390)

Maintainer ruling: a file never falls through both the link and the
queue. When only the saver anchor resolves a file the default profile and
the parser floor leave unlinked, the scan queues it with the anchor's pair
as its one candidate (evidence anchor: saver), never linking it. Files
already linked or resolved by the default path are unchanged. Apply
accepts the anchored pair when the bytes now resolve to it, and an
anchored item is a choice the untraced settle keeps.

* fix(review): upgrade old default items of anchored files, tag anchored results (#390)

A pending default item recorded before the saver anchor no longer
shortcuts the rescan when only the anchor resolves the file: the
anchored item supersedes it. Apply accepts a resolved re-resolution only
when it is tagged as the saver anchor's, the anchored pair must carry a
positive text to choose, and limited anchored evidence still counts as a
choice. TESTING.md describes the ruling.
 2026-10-28 07:22:48 +00:00
Vito d11b354ed6 fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2) (#394) 
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1)

The native "Tag output images" job now runs the image-tags loop over every
still image the output folders list, linked or not: each image gets its own
model tags (at most "Suggestions per image") in its image tag set, and its
bytes are marked auto-tagged in a new media_image_autotags table. No prompt
tag or prompt text is written.

- database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional
  in old backups, refused without the annotation group.
- add_image_tags/replace_image_tags take mark_autotagged; autotag_state and
  autotagged_locators read the marks through the one current-record rule.
- "Skip already tagged" skips images whose current bytes are marked, counted
  in a stat-only pre-pass; hand tags do not count.
- The review counts images and estimates the time (WD14 on CPU or CUDA).
- System > Auto-tagging review rows and copy, D134 and the Auto-tagging
  contract follow.

Part of #379

* fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379)

The job read as ended while it still held the model gate, so a quick
restart could be refused with model-busy. Pre-pass chunks now publish
progress, and a cancel between chunks keeps exact counts.

Part of #379

* fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379)

The live check measured WD14 on CUDA at 0.167 s per image (100 images in
16.7 s with the model load), so the review estimate uses 0.17 s. A finished
or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X
min left"; it now ends with "<verb> <state> · N files processed".

Part of #379

* fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2)

The admin and gallery batch starts the shared Tag output images job
("Tag output images (legacy page)") and relays its progress; closing the
window cancels it. Review asks per image with Skip first and applies by
image and sha256; apply refuses a prompt_id body. The Tag image dialog
marks an image auto-tagged only after Generate suggestions on that
capture. A second legacy batch (also one that wins the start race)
answers JSON 409, and both pages now show that sentence.

* test(tagging): gallery review units apply by image; record each progress change

The gallery review unit tests still pinned the removed prompt_id apply.
The legacy batch spec read the progress text once per observer callback,
so a phase frame and the first image frame landing in one task lost the
phase (WebKit); it now records every mutation record.
 2026-10-27 22:40:57 +00:00
Vito 37e4e08327 fix(tagging): Images tag search goes through the tag index (#379, PR 3) (#395) 
* fix(tagging): Images tag search goes through the tag index (#379, PR 3)

The Images Tags filter and text search no longer stream every image tag
row per request. image_tag_ids reads the names that tag at least one image
(exact under Tags, case-folded substring for q) and image_tag_locators
reads their files by tag_id through idx_media_image_tags_tag. Results are
unchanged: a differential test compares the lookup with the old stream,
and a trace test checks that no statement reads media_image_tags except
by tag id. The search contract in TESTING.md and a browser test (CUJ 8)
follow.

Closes #379

* test(search): pin prompt-only tags and multi-chunk tag lookups (#379)

Reads the tag names and the locators in the same annotation read helper.
 2026-10-27 22:07:59 +00:00
Vito d2ea705a44 fix(tagging): Tag output images tags each image, never its prompt (#379, PR 1) (#380) 
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1)

The native "Tag output images" job now runs the image-tags loop over every
still image the output folders list, linked or not: each image gets its own
model tags (at most "Suggestions per image") in its image tag set, and its
bytes are marked auto-tagged in a new media_image_autotags table. No prompt
tag or prompt text is written.

- database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional
  in old backups, refused without the annotation group.
- add_image_tags/replace_image_tags take mark_autotagged; autotag_state and
  autotagged_locators read the marks through the one current-record rule.
- "Skip already tagged" skips images whose current bytes are marked, counted
  in a stat-only pre-pass; hand tags do not count.
- The review counts images and estimates the time (WD14 on CPU or CUDA).
- System > Auto-tagging review rows and copy, D134 and the Auto-tagging
  contract follow.

Part of #379

* fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379)

The job read as ended while it still held the model gate, so a quick
restart could be refused with model-busy. Pre-pass chunks now publish
progress, and a cancel between chunks keeps exact counts.

Part of #379

* fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379)

The live check measured WD14 on CUDA at 0.167 s per image (100 images in
16.7 s with the model load), so the review estimate uses 0.17 s. A finished
or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X
min left"; it now ends with "<verb> <state> · N files processed".

Part of #379

* test(browser): reload the admin page only after the in-place edit saved

The gallery-to-admin workflow pressed Enter and reloaded as soon as the
edited text showed; the page saves after showing it, so a reload that won
the race abandoned the PUT (WebKit, run 40564511000). The test now waits
for the save's answer.

Part of #379

* test(tagging): the batch coverage row names the renamed #379 test
 2026-10-27 20:30:54 +00:00
Vito 4032f4b6ab feat(system): one tagging configuration and Tag output images as a reviewed job (#328, PR 2) (#366) 
* feat(autotag): model status without paths, and a busy answer that names the operation

* feat(autotag): model downloads run as Activity jobs

* fix(autotag): keep each download route's error shape and harden the fake download

* feat(autotag): one server-side tagging configuration

* fix(autotag): the legacy batch and review work again

* feat(autotag): tag output images as a reviewed Activity job

* test(autotag): the legacy batch, review and unload page workflows

* fix(autotag): server-free tagging defaults, cancel before load, and named legacy refusals

* fix(autotag): name only the workspace batch in a legacy refusal frame
 2026-10-27 00:20:13 -07:00
Vito 80c6ec5340 fix(lora): find LoraManager's sidecars, previews and example images in every storage configuration (#355) (#358) 
* fix(lora): read LoraManager's settings and storage layout the way it does (#355)

Add py/lora_manager_layout.py: the settings file in LoraManager's lookup
order (environment folder, portable plugin file, user config folder), only
the seven keys the import needs, the LoRA roots, the centralized sidecar
root, the example-images root and the active library, each folder through
one guard. Sidecar discovery for both storage modes and the local preview
and example-image lookup live there too.

The test runner and the browser test server point LoraManager's settings
lookup at a disposable folder, so no test reads the real settings.

* fix(lora): import the sidecars and local images LoraManager really uses (#355)

The import, the legacy scan, the trigger-word cache and the image server
now share LoraManager's layout: sidecars beside the models (alongside) or
under the sidecar root (centralized), kept only when their model exists
inside a LoRA root. Each LoRA links its LoraManager preview and example
images where they are; CivitAI downloads run only for a LoRA without any
local image. The review adds skipped, with_local_media and the layout, and
offers no token when LoraManager's settings cannot be used; the legacy scan
reports the same reason without replacing anything.

find_lora_directories becomes default_lora_roots (no scan of the
LoraManager install); the preview and example-folder helpers it replaced
are removed. The trigger cache never raises into a running node.

* fix(lora): never serve a LoraManager folder that holds the home folder (#355)

The folder guard also rejects ancestors of the home folder (/home,
C:\Users), compared in the realpath + normcase form, so a LoraManager
setting cannot widen the image server's allow-list to every user's files.

* fix(lora): show linked LoraManager videos as videos (#355)

_enrich_images marks every image without is_video by its extension, so an
.mp4 preview outside the output folders shows the Video preview label and
plays in the inspector. Tests cover serving the sidecar and example-image
folders only while the integration is on, and a filesystem root as the
sidecar folder adding nothing.

* feat(lora): the import review names the LoraManager settings it read (#355)

The review gains a first row, LoraManager ("<source> settings · <mode>
storage · library <name>", the settings file as its hover title), and its
Found row counts metadata files, LoRAs and LoRAs with local images. A plan
whose LoraManager settings cannot be used takes the close-only path:
"LoraManager settings: not usable: <reason>. Nothing was changed." The
browser fixture writes each LoRA's model file and can leave out the local
preview, so the download, gate and cancel specs still exercise CivitAI.

* docs(lora): describe how the import finds LoraManager's files (#355)

* test(lora): set the settings folder relative to the process folder (#355)

os.path.relpath fails across Windows drives (the temp folder on C:, the
checkout on D:); the test now changes into its temp folder instead.

* fix(lora): review findings for the LoraManager layout (#355)

- The example-images library subfolder goes through the folder guard like
  its root, so its canonical (normcased) form matches the media check on
  Windows for a library name with capitals.
- A failed trigger-word load is retried by a running node only after
  TRIGGER_RETRY_SECONDS, not on every prompt run.
- A review whose discovery cannot list a folder says so (folder-unreadable,
  the folder's name only) instead of a generic error.

* test(images): hold the listing clock in the deferred-walk test

test_a_deferred_walk_waits_at_least_half_a_second set the previous walk
29.99 s back against a 30 s interval and then read the real clock again:
a runner slower than 10 ms between the two lines passed the interval and
the walk started at once (failed on macOS 3.13 and Windows 3.11). The
module's clock is now held for the mark, so the deferred walk is always
the one measured.

* test(lora): write the unkept-settings fixture as raw text

CodeQL read the dict with a credential-named key flowing into write_text
as clear-text storage of a secret. The value is a marker, so the fixture
now writes the JSON text directly and names the value MARKER.

* fix(lora): link each LoRA's cover as its newest image (#355)

Workspace lists show a prompt's media newest link first, so the import
links a record's images last to first: the card shows the cover (a still
in front of a leading video) the import put first.

* fix(lora): show LoRA-linked media on the Library page and in the inspector (#355)

Library previews, the positive detail and the per-prompt images page list
a prompt's linked media under the output folders plus, while the LoRA
integration is on, LoraManager's guarded folders and the download cache
(_media_roots; output root indices keep their meaning). That media keeps
its rooted identity, so Info and the media viewer open it; the path-based
serve route answers for those roots only with a file the database records
(either path spelling), never serves or makes a preview there, and output
requests never read LoraManager's layout. Such media is marked
linked_only: no preview file, no favorite star, no Tag action. Show in
folder accepts those roots; the download cache is named, not created.

* fix(lora): expose only linked files under LoraManager's folders (#355)

Serve, inspect and Show in folder share one gate for a LoRA-linked root
(_media_roots past the output folders): a file the database does not
record as a linked image is refused with 403, checked by its given and
its real path. Before, inspect and reveal read any file under those
folders by path.

* fix(lora): check the file Show in folder would open under a LoRA root (#355)

reveal_image resolved the requested path through the client-path
resolver, which can fall back to a path anchor; under a LoRA-linked root
the file that would open now passes the same recorded-media gate, so only
a linked file is ever revealed there.
 2026-10-26 21:21:33 -07:00
Vito 3be0ada6a1 fix(ingestion): link an image only to a prompt that provably produced it (#344) 
* fix(ingestion): link an image only to a prompt that provably produced it

An LLM fed a PromptManagerPair's prepend_text while its text editor was
empty. The node refused to save (an empty positive raised and logged a
warning), and the image monitor then linked the image to a stale tracker
entry or to the newest prompt in the database.

- Nodes: an empty text editor saves and tracks nothing (prepend and
  append text are never the stored prompt), logs at info, and marks the
  node's run key empty so no earlier entry can be used for its images.
- Tracker entries carry a run key: a digest of the prompt node and its
  whole upstream graph, the identity ComfyUI's cache uses. The monitor
  computes the same key from the image's embedded graph and uses a
  queued or snapshot entry only when the keys match (a snapshot also
  needs a known run with one entry, so a batch's last item is never
  applied to every image).
- The recency fallback is gone; anything uncertain stays unlinked and is
  logged at info.
- Recorded execution history attributes a file when exactly one run
  reported it, the file was written during that run, and the run's
  positive contexts name one stored prompt. The monitor and the output
  scan use it as their last certain source.
- The scan and pair ingestion treat a resolved pair with an empty stored
  positive as unresolved instead of raising.
- tools/review_image_links.py reviews selected links by these rules;
  dry run by default, --apply changes only named ids after a backup.

Closes #342

* test(ingestion): real-host attribution case and the #342 testing contract

The managed host disables the image monitor, so the owned fixture pack
gains a route that starts and stops the plugin's monitor singleton for
the attribution case only. The case runs in both renderers: a typed Pair
image links, a Pair image whose positive is only linked prepend text
stays unlinked with nothing stored, an unknown node feeding text links
through its run key, and its cached re-run links again.

* fix(ingestion): take a queued entry only from the run the image appeared in

Dropping a key's older queue entries whenever the node ran again let a
later queued run, executing during the monitor's processing delay, take
the earlier image's entry. The snapshot taken when the image appears now
names the key's latest run, and the queue is popped by key and run:
same-key entries of earlier runs ahead of it are stale, entries of later
runs wait for their own images.

Also keeps the attribution case out of the stock real-host suite.

* test(ingestion): host lookups degrade to no proof

* fix(ingestion): address the #344 review (history re-check, keys, repair)

- Images without an embedded API graph: the scan now asks recorded
  history before skipping a file with no prompt metadata, and the monitor
  gives a keyless unlinked image one history-only re-check when its run's
  outputs are recorded (a seal notification from the outputs write; at
  most 1000 waiting images, each for an hour).
- Run keys: an image is unkeyable when any positive node has no key, so a
  two-pass graph is never linked through the other pass; links follow
  ComfyUI's is_link exactly.
- A failed save marks the node's run key like an empty editor, so an
  older entry of that key can't stand in for this run.
- A placeholder is the latest entry: the image stays unlinked without
  falling through to history, the same rule as a popped placeholder.
- An index on execution_run_outputs(filename, subfolder, root_key,
  path_style) for the history lookup, added by migration and accepted by
  the output schema check (any other index is still refused).
- Repair tool: unlink removes only the link record and only while the
  image is still linked to the prompt it was judged on; every run prints
  what --apply does.
- Real-host case proves the re-run's PromptManager node was cached.

* test(ingestion): real-host bare-output re-check and the review-round contract

A metadata-disabled save, held open behind the fixture gate after
SaveImage, waits unlinked while its run is open and links once the run's
outputs are recorded. The fixture's monitor route lists the waiting
files. TESTING.md covers the history re-check, the read index and what
the repair tool's --apply does.

* fix(ingestion): batch history lookups instead of adding an index

An index on execution_run_outputs would make the library unopenable by
older builds, whose schema check refuses any extra index there. The
migration and the schema-check allowance are reverted, so the history
schema is exactly main's. stored_run_prompts instead looks up a batch of
files with one parameterized filename IN (...) query per chunk of at
most 500 names and matches in memory; the scan computes it once per scan
batch.

* fix(ingestion): address the #344 re-review (seal race, linked roots)

- A sealed history re-check that can't be scheduled yet (the image's own
  pass still pending, or the pending set full) is kept due and scheduled
  when a pending image finishes, instead of being dropped.
- History matches a file through ComfyUI's spelling of its output folder
  first, then with both sides resolved, so a symlinked or junction output
  folder still attributes the scan's resolved paths; waiting files are
  keyed by their resolved path.
- The scan makes its batch history lookup only when a file needs it.
- set_current_prompt pushes to the queue last, so marking a failed
  execution never adds a second queue slot.
- A keyless image outside the output folder doesn't take a waiting slot.
 2026-10-25 19:03:35 -07:00
Vito 8252d9d226 feat(system): Integrations and a staged LoRA import (System S10) (#336) 
* fix(lora): detect LoraManager in ComfyUI's extra custom_nodes folders

Auto-detection looked only in ComfyUI's main custom_nodes folder (#144).
It now also searches every custom_nodes folder ComfyUI registers through
folder_paths, main folder first, skipping folders that cannot be read.

* fix(lora): stage imports before replacing the collection (#221)

The LoRA import deleted the whole lora-manager category first and then
saved each record in its own transaction, so a discovery, download or
database failure left the previous collection gone or partial.

The import now runs in three stages: discover and fingerprint the
metadata files, stage every record (CivitAI downloads go into the image
cache first, counted), then replace the category in one BEGIN IMMEDIATE
transaction (replace_category_prompts). Any failure leaves the previous
prompts, tags and image links exactly as they were. Prompts a saved pair
uses block the replacement; an empty replacement of a non-empty
collection needs a reviewed import.

POST /prompt_manager/lora/import/preview reviews an import without
writing or requesting anything; POST /lora/import/apply {token} runs it
as the Activity job "LoRA reference import" (records admission,
cancellable while it downloads, the safety backup policy before the
replace, refused with plan-changed when the files changed since the
review). The legacy POST /lora/scan runs the same stages with its SSE
event shape.

* feat(lora): the LoRA save takes the settings revision

POST /prompt_manager/lora/enable accepts the optional expected_revision
that GET /settings returns, so the System Save row refuses to write over
a config.json changed elsewhere (settings-changed, nothing written). The
legacy page sends none and still merges into what is on disk.

* feat(system): configure ComfyUI and LoRA integrations natively

System › Integrations replaces its hand-off with the concept's two
surfaces. ComfyUI: only the send destination and renderer the bridge
supports can be chosen; the other options and Return to canvas are
disabled with the reason, and Host compatibility shows the renderer the
open ComfyUI windows report (each bridge destination now carries
renderer: classic or nodes2) (D106). LoRA & resources: detection, the
enabled setting and the loaded trigger words read as separate facts,
the read-only LoraManager path row follows the toggle (D106), and the
CivitAI key is never filled in: typing drafts a new key, blank keeps the
saved one and Clear saved key drafts its removal (D114). They save
through the one Save row (/lora/enable with the settings revision).

LoRA reference data → Review import shows the server's read-only review
(replacement, what saved pairs block, downloads, backup box) and Start
runs the LoRA reference import Activity job (D107), with its own
Activity descriptor and Review again into Integrations when the files or
pairs overtook it.

* test(system): compare Integrations with the concept

system-integrations-concept.spec compares System › Integrations (both
themes, 1600 and 390 wide) with the concept, mapping only D106 and D114,
and the LoRA import review with the concept's record-changing review,
mapping D107; each comparison catches a removed row, a renamed label or
option, a re-enabled unsupported option and a swapped token.

On the real ComfyUI host, each bridge destination reports the renderer
its window draws nodes with, before and after switching it, and System ›
Integrations shows it.

* test(coverage): record the Integrations frontend floors

coverage:update after the full frontend run: system-integrations.mjs at
its measured floor, and the raised maintenance review, job view and
maintenance floors.

* test(system): wait for the renderer line before clicking Integrations rows

CI Firefox (390 wide) clicked the LoraManager toggle while the renderer
check's line above it changed text and moved the rows, so the click did
not reach the switch. The specs now wait for that line to settle (a
visible state) before interacting.

* fix(lora): recheck the files before the replace and stop an abandoned scan

Review fixes for the staged LoRA import (#221):

- The metadata files are listed and fingerprinted again right before the
  replace transaction, so a file changed, removed or added while the
  downloads ran aborts the import with nothing deleted (System job:
  plan-changed, Review again; legacy scan: an error frame in its own
  words). The tests now change real files behind a held download.
- A legacy scan whose page closes stops staging before its next LoRA and
  keeps its records hold until staging has stopped; CivitAI images are
  written to a temp file beside the cache entry and moved in place, so a
  half-written image is never seen as cached.
- Start refuses with plan-changed when the integration was turned off or
  moved to another install since the review.
- A failed job keeps the skipped count and the names of the LoRAs that
  could not be prepared.
- The replace binds the category instead of one parameter per prompt
  (SQLite's variable limit), /lora/status says whether a detected install
  can be used, and the legacy scan logs expected refusals as warnings.

* fix(system): partial saves, key focus and readable selects in Integrations

A save that writes the LoRA part but not the settings part now keeps the
saved part, adopts the new revision and highlights only what failed, so
the next save needs no conflict. The CivitAI key status is a persistent
live region, Clear saved key returns focus to the key field, and the key
draft is dropped when the section is left. The send-destination and
renderer selects keep their width on narrow screens. A detected
LoraManager outside custom_nodes says it cannot be enabled.

* fix(lora): keep the save outcome, honour the umask and sweep stale downloads

Leaving Integrations drops the CivitAI key draft only when a key was
typed, so the last Save's outcome survives and Sources still marks a
refused source. The key status line is rewritten only when it changes.

Cached CivitAI images are opened like a plain save, so they keep the
umask's mode, and each import removes .pm-*.part files older than an
hour from the image cache. A legacy scan stopped because its page left
ends cancelled in Activity, and staging is awaited to the end even if
the handler is cancelled again.

* fix(lora): the import review takes a lease beside a restore

The LoRA import review reads the library, so like the maintenance and
restore previews it now runs under a request lease: it is refused with
409 busy while a restore runs, and a restore waits for it before it
switches the file. Tests pin that a restore is refused while an import
runs and that the review, the reviewed import and the admin page's scan
are refused while a restore runs.
 2026-10-25 10:50:48 -07:00
Vito 28ad3a1a1f feat(system): backup and restore backend (System S8) (#335) 
* refactor(database): stage restores before publishing

restore_from_file is now stage_restore + publish_staged (database/
restore_stage.py). The stage lives in a .pm-restore-* folder in the
library's own folder, is migrated, has damaged workflow documents repaired
from an inline copy or cleared, is verified with every document read and is
bound by its SHA-256. Publishing keeps a unique, standalone, 0600 safety
backup (microsecond names), copies the pages through the backup API as
before and then truncates the WAL, which stayed database-sized after a
restore. The safety backup writer moves to database/safety_backup.py so
the database layer and maintenance share it.

* fix(database): read the verify streams one row at a time

The provenance and review-evidence checks iterated their cursors; on a
managed connection iteration prefetches 128 rows, so up to 128 large values
were held at once while the docstrings promised one. Both now use fetchone.

* feat(backup): review a restore, run it as an Activity job, verify backups

- POST /restore/preview stages an upload in the library's folder and
  answers what it holds beside the live library, writing nothing live;
  DELETE discards it; at most two stages are held (429 too-large).
- POST /restore/apply rechecks the live fingerprint (409 plan-changed) and
  starts "Database restore": exclusive, not cancellable, indeterminate. It
  waits for request leases, publishes the exact staged bytes, swaps the
  database, prunes safety backups and logs counts, repairs and the backup.
- The legacy /restore runs the same stage and publish as an exclusive hold
  with its old response, so a restore and the health check, Move library
  and every other job refuse each other.
- JobRegistry.lease: Needs review decisions and maintenance and restore
  previews are refused while an exclusive job runs, and a restore waits
  for them before it switches the file.
- GET /backup holds a guarded slot, verifies a standalone snapshot before
  streaming it and removes the closed temp file; GET /backup/status lists
  the safety backups on disk and this session's verified download.

* docs(testing): backup and restore contract

* test(system): a restore refuses a running health check

The health route test relied on a legacy restore interleaving with a running
check; the restore is now exclusive and refuses it, naming the check. The
diagnostics backup tests follow the backup handler into
py/api/backup_restore.py.

* fix(backup): drop stale restore previews off the event loop

A restore drops the previews reviewed against the old library; their stage
files are now removed in the executor, with the old database's connections.

* test(backup): wait for the backup hold to end before reading its state

CI (py3.11/3.12 on Linux, every macOS) read the registry right after the
client had the last byte, before the handler left its hold and removed its
temp file: the job was still running. The tests now wait on that state.

* test(images): the scan dialog's Back up now leaves a finished backup hold

A backup download now holds a guarded registry slot while it streams
(#249 S8), so the jobs list holds that finished hold after the dialog's
Back up now; the spec still proves Cancel starts no scan.

* fix(backup): final review round for the restore and backup routes

- I1: once the stage's copy replaced the library the restore succeeded:
  pruning, reopening self.db, dropping old previews and closing the old
  object only log, so the job finishes and the legacy /restore answers 200
  instead of "the library was not changed". The generation is always bumped.
- M1: a backup download holds its guarded slot only while the snapshot is
  written; the stream runs outside it, so a stalled download blocks nothing.
- M2: a restore preview leases only its read of the live library, not the
  upload or the staging; a restore whose drain still times out keeps the
  reviewed stage under its token.
- M3: the reviewed-preview store changes only on the event loop; stage
  folders are removed in the executor.
- M4: a document whose body inflates completely while only its recorded
  hash or only its length disagrees is re-keyed (or merged into the
  document holding that text) instead of dropped.
- M5: a backup download is refused only when the snapshot is not SQLite or
  fails integrity_check; a semantic check failure still downloads, logged,
  sent as X-PromptManager-Verify and recorded as not verified.
- M6: a refused, failed or cancelled preview removes its stage, also when
  the cancel lands while a worker still stages; a timer removes expired
  stages without another request.
- M7, N1: repair reads rows one at a time and counts distinct images.
- N3: the live fingerprint never commits a transaction it did not open.

* test(backup): close the old library the close-failure test leaves open

CI (py3.12 Linux and the coverage gate) found the old object's
executor-thread connections still open at teardown: the test patches its
close_all to fail, and once the object is collected close_all_instances no
longer reaches it. The test now closes it itself.

* fix(backup): re-review round for restore repair, apply and backup cleanup

- R1: when re-keying merges an intact body into a holder whose own body was
  damaged, the rewritten holder leaves the damaged list, so the clear pass
  no longer deletes it and unlinks the images just moved to it.
- R2: a losing duplicate apply removes a stage only when the store still
  held it, never one another apply's job has taken.
- R3: a restore that times out waiting for requests keeps its review only
  while it is unexpired and its slot is free; otherwise the stage goes and
  the job says to try again.
- R4: a backup download cancelled while its snapshot is written removes
  the snapshot once the worker finishes.
 2026-10-25 10:17:07 -07:00
Vito 9601837aa9 fix(system): never close another thread's library connection during a switch (#321) 
* fix(system): never close another thread's library connection during a switch

Move library closed every thread's old connection from the mover thread,
including one whose owner was inside a statement waiting for the old
file's write lock. Closing a sqlite3 connection under a running
sqlite3_step is a use-after-free: Windows CI died with "access violation"
in record_prompt_use, and on Linux the writer got a MemoryError once its
busy timeout ran out.

Managed connections now count the calls their owner is inside.
Retiring one from another thread closes it at once only when idle;
otherwise it is interrupted (sqlite3_interrupt is thread-safe) and its
owner closes it as the call ends. A retired connection refuses new work
and never commits, so a write that waited for the lock is rolled back
and survive_library_switch runs it again on the new library. The mover
retires connections while it holds the locks and waits for the owners
to close them only after releasing them, before the old files are
renamed or a failed copy is removed. Cursors are closed before the
connection so Python 3.10/3.11 release the file handle too.

* test(tagging): wait for in-flight pages before reading the loaded range

The scroll-position test read the loaded count right after scrolling the
continuation out of view, while a listing page the continuation had
already requested could still be in flight. When it landed the count
grew (80 -> 120 in CI) and the stability poll failed. The test now waits
a frame for any scheduled load to start and then for every listing
request to finish before it reads the count.

* fix(system): make retiring a library connection race-free and cheap

Review round for the cross-thread close fix:

- cursor() now records its cursor under the lock _close_now copies the
  cursor set under, and refuses once the connection is retired; a cursor
  opened while the connection closed raised "Set changed size during
  iteration" out of retire() on Python 3.10-3.12. retire() never raises,
  retire_connections keeps going when one connection fails, and the mover
  never reports a completed switch as failed because retiring or awaiting
  old connections did.
- The restore's backup target is marked in use (ManagedConnection.using),
  and in_transaction and create_function are guarded.
- Iterating a cursor marks the connection in use once per fetchmany(128)
  chunk instead of once per row: a 1000-row scan costs 1.10x a plain
  cursor instead of 2.7x (3.14) / 3.8x (3.10).
- Document what an interrupt does not end, the statements a retired
  connection cannot hold back, close() from another thread returning
  before the file is closed, and the await timeout ending as old-in-use.
 2026-10-24 20:39:08 -07:00
Vito 8bf759d6f8 feat(system): keep the library in ComfyUI's user folder (#318) 
* feat(system): resolve every library path in one place

utils/library_location.py decides where the library lives: the
PROMPT_MANAGER_DATA_DIR folder (validated: absolute, never inside a folder
ComfyUI or Prompt Manager serves), an explicit database.default_path as
before, a node-folder library that was not moved, else ComfyUI's
user/__promptmanager. The database, diagnostics, config.json, logs and the
LoRA image cache all ask it, so a new install never writes into the node
folder and an existing one keeps every path until the user moves it.
"Keep it here" is persisted as library.keep_in_node_folder.

* feat(system): move the library as a verified Activity job

POST /prompt_manager/system/library/move starts "Move library", an
exclusive, non-cancellable job (database/library_move.py). It checks the
target volume's free space, copies the database with SQLite's backup API
from one read snapshot, verifies the copy (integrity check, every workflow
document, the same rows in every table), copies the safety and
pre-migration backups and config.json (owner-only, checked by size and
SHA-256), then publishes under the old file's write lock and switches the
live process (every PromptModel, the review and history observers, the
API's plans and health report). The old prompts.db and config.json are
renamed *.moved-to-<folder> only after the switch. A failure removes
everything it wrote and leaves the old library in use; a marker in the new
folder makes a restart after any crash pick the right library.

GET /prompt_manager/system/library and the Overview's "location" say where
the library lives and why; POST .../keep persists "Keep it here".

* feat(system): show where the library lives and offer the move

D129. System > Overview's Library health gains a "Library location" row
(the database path, why it is there, the "Keep it here" choice and any
PROMPT_MANAGER_DATA_DIR problem) with Move library for a node-folder
library and Details otherwise. While a node-folder library is not kept
there, the workspace shows the concept's .notice above every page:
"Keep it here" saves the choice on the server and stops the notice;
"Move library" opens the shared review (From, To, Size, what moves, what
happens to the old files) and starts the exclusive job in Activity. The
notice and the Overview read the location again when the job ends.

The Overview comparison maps the inserted row; a new comparison holds the
notice to the concept's Sources notice (markup, tokens, alignment, both
themes and widths) with a mutation check.

* feat(system): close the browser server's log files before its root goes

The logs now live in the library's data folder, which for the browser test
server is <root>/user/__promptmanager; Windows could not remove the temp
root while the server still held prompt_manager.log open. The API test's
node folder is spelled in lower case because public paths are case-folded
on Windows.

* feat(system): show the library location on the legacy admin page

A line under the admin page's statistics names the database path and why it is there (ComfyUI's user folder, PROMPT_MANAGER_DATA_DIR, config.json, or the node folder with a pointer to Move library), read once from GET /prompt_manager/system/library.

* feat(system): prove a fresh host keeps its library in user/__promptmanager

The execution suite starts one pinned CPU ComfyUI with no configuration at all on an OS-chosen port: the library is created in user/__promptmanager (database and logs), nothing but byte code is added to the node folder, the location route reports the default folder, and ComfyUI's /userdata API serves none of it (traversal spellings, a listing, a system-user header).

* feat(system): document where the library lives and how it moves

README: where the library lives, the upgrade path (Move library / Keep it here), PROMPT_MANAGER_DATA_DIR, explicit paths unchanged, and the way back to an older version. TESTING.md: D129 and the library location contract (resolution order, move steps, crash windows, coded refusals, Windows handling).

* wip(system): library move fix round in progress

* fix(system): retry a fallback image link that races the library switch

The image monitor's fallback and recent-prompt links went through
PromptDatabase.link_image_to_prompt, which logs and returns 0 on any
error, so survive_switch never saw a "database is locked" or closed
connection during Move library and the link was lost in both files.
The monitor now links through link_image_to_prompt_or_raise (one
transaction per attempt); link_image_to_prompt keeps its
log-and-return-0 contract for other callers. A link that returns 0 is
no longer logged as "Successfully linked image 0".

A switch that fails after repointing the models now closes the old
file's detached connections instead of leaving them to GC: switch()
adds them to a list the mover owns.

The mocked retry test is replaced by a real-lock race asserting the
image row lands in the new library only.

* fix(system): say which source is the default and where a symlinked library lives

Sources & scans: with no source configured, ComfyUI's output folder now
reads "ComfyUI/output · watching · default" on its own row instead of a
separate note line. GET /settings marks each effective source with
in_comfyui (its public path is relative to the ComfyUI base) so only
those get the "ComfyUI/" prefix; the scan still sends the server's
public path.

Library location: a library in the node folder reads through the node
folder's unresolved place in custom_nodes, so a node folder symlinked to
a checkout outside ComfyUI shows custom_nodes/promptmanager/prompts.db
instead of prompts.db (System library row, Move review, admin Library
line). Never absolute or "..", else the shared _public_path, which
sources and jobs keep using unchanged.
 2026-10-24 12:54:09 -07:00
Vito 06099579e6 feat(maintenance): store each workflow once (#315) 
* feat(maintenance): reviewed, read-only maintenance plans

POST /prompt_manager/maintenance/preview reviews duplicate_prompts,
optimize, unused_components or missing_links on its own mode=ro snapshot
and answers a Plan (scope, protected reasons, backup policy, media, at most
50 detail rows). The fingerprint, targets and judged sources stay in a
server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB,
never truncated; a restart forgets them), bound to the database it read.

The previews reuse the legacy queries: duplicate groups and the merge are
shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the
#239 protection predicates the plan reports, and missing links are judged
by the health check's link_states. Legacy /maintenance keeps its shape;
vacuum now reports success:false when VACUUM fails, and
cleanup_orphaned_images skips links under unavailable sources.

* feat(maintenance): apply reviewed plans as Activity jobs

POST /prompt_manager/maintenance/apply {token} starts a maintenance job
(records admission, not cancellable, groups changed/protected/errors).
The token is consumed on the loop turn the job starts: a second apply is
409 plan-used naming the job, a busy refusal keeps the token, an unknown
or expired one is 410 plan-expired, and a restore since the review is
409 storage-changed.

With the safety backup policy on (maintenance.safety_backup, default true,
saved through the settings route with the revision), the job first writes
a backup into a temp file beside the database, makes it a standalone
rollback-journal file, verifies it, fsyncs it and renames it onto a unique
<db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file
and nothing changed. The apply then rechecks the reviewed fingerprint
inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of
400 ids, counts changed and protected records, logs the backup by name
only and prunes old safety backups. With the policy off the log and the
status line say "No safety backup".

* feat(maintenance): find and remove duplicate files through Activity

Duplicate image files move onto the job registry:

- POST /prompt_manager/jobs/duplicates finds content-identical media
  files over every effective source as a cancellable read job
  ("Find duplicate image files", group groups_found); the server holds
  the groups per job and GET /prompt_manager/maintenance/duplicates/{id}
  pages them ten at a time.
- The duplicate_files preview binds a removal to the reviewed keepers.
  Every keeper and removal is checked again: containment as text before
  any filesystem access, then not a link, still inside its source, same
  size and hash. The plan lists the database links that move.
- Apply runs duplicate_cleanup (records, not cancellable): the promised
  safety backup, then per file a recheck, the unlink with every handle
  closed and its thumbnail. A changed file is kept, the first failed
  unlink stops the job with a coded message (never the OS text), and only
  removed files have their image links and favorites moved to the kept
  copy.
- The legacy scan, stream, status and delete routes keep their shapes on
  the same registry kinds; the hashing, grouping and thumbnail removal
  are shared with them.

Coordinator rulings on Tasks 15-16:

- Safety backups share the restore pool of 10 `<db>.backup_*` files; the
  pool is now also pruned when an apply fails after its backup.
- Plans report the safety backup's size and the free disk space, and
  apply refuses up front (507 insufficient-space) when the backup or
  VACUUM does not fit, so no job fails halfway for space.

* fix(maintenance): never remove the kept copy; move records before unlink

Final review of #313:

- C1: the duplicate find tells files apart by (st_dev, st_ino), so a
  source listed twice, a symlinked source inside another and a hardlink
  are one file. The preview and the job refuse a removal that
  os.path.samestat matches with its kept copy.
- I1: each file's records move to its verified kept copy in one short
  transaction before that file is unlinked; a failed move deletes
  nothing and a failed unlink leaves the records on the identical kept
  copy. The status line counts files removed and not removed.
- I2: links are looked up under every spelling of the source (the real
  path, the configured and monitored folders, ComfyUI's output).
- I5: image tags join the kept copy as their union and the removed
  copy's pending, undecided Needs review items are settled; the plan
  says so.
- I3: the VACUUM disk check measures SQLite's real temporary folder.
- M1: the token is looked at again with no await before the job starts.
- M2: the backup decision is made once and handed to the job.
- M3: backups are linked into place; startup sweeps interrupted temps.
- M4: Optimize runs alone (new "exclusive" admission) and its plan says
  the database is locked while VACUUM runs.
- M5: the reviewed duplicate merge is strict: collisions and Library
  titles are protected, and errors roll back. The legacy cleanup is
  unchanged.
- M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests.
- M9: the preview binds files by size, mtime and inode without hashing;
  the job hashes right before each unlink.

* fix(maintenance): protect one file reached through two mounts

Re-review of #313, round 2:

- R1: a removal on another device than its kept copy, with the same
  size and mtimes within 2 s, is protected when its inode is equal or its
  path within its source is the same ("May be the same file reached
  through another mount or share"), in the preview and again right before
  each unlink. The job holds the kept copy open during the unlink, so on
  Windows deleting the kept copy itself through another path fails.
- R2: a file's spellings are found by containment: every configured,
  monitored, ComfyUI-output or listed folder whose stat identity is a
  folder above the file. Parent spellings of nested sources and case
  variants match; favorites move under every listed root holding the
  file. Lookups stay index searches.
- m1: a failed unlink's row says the links now name the kept copy.
- m2: a volume with no file ids reads "Can't confirm this is a separate
  file on this drive".

* fix(maintenance): stat duplicate candidates with os.stat

Python 3.10's Path.stat uses the os.stat bound when pathlib was imported,
so the find and the per-file checks could see different stat calls. The
find now calls os.stat directly, like every other check; the mount tests
failed on py3.10 only.

* fix(maintenance): recheck the kept copy right before each unlink

Re-review 2 of #313:

- n1: the cross-mount guard also matches when one path within its source
  ends with the other on a whole folder name (sources listed at different
  depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount).
- n2: after a file's records move and right before its unlink, the held
  kept copy must still be the file at its path (fstat of the hold against
  os.stat) and the cross-mount guard must still hold; otherwise the file
  is kept, its records move back to it, and it counts as not removed.

* test(maintenance): the kept-copy race tests hold on Windows

On Windows the removal job holds the kept copy open without
FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during
the record move is refused. The tests now assert that native outcome:
the kept copy stays and the removal goes ahead.

* feat(database): store each workflow and prompt graph once

generated_images.workflow_data and prompt_metadata repeated the same large
JSON text for every image of a run. Each distinct text now lives once in
workflow_documents, keyed by the sha256 of its exact UTF-8 bytes and
zlib-compressed; image rows reference it through workflow_document_id and
prompt_document_id.

- New writes (link_image_to_prompt, pair ingestion for scan, monitor and
  Needs review) store the new form. Old rows keep their inline text and
  read the same until the migration rewrites them.
- Every reader goes through one accessor (read_stored_text,
  resolve_image_row): gallery and legacy image lists, image by id, prompt
  info for the inspector and metadata page, diagnostics. Payloads keep their
  shape. Inline text wins when present; a document that fails its length or
  hash check reads as unreadable, never as other text.
- No trigger (restore validators refuse them): a document goes in the same
  transaction as its last reference (image and prompt deletes, missing-link
  cleanup, duplicate merges, duplicate-file link merges, ingestion updates),
  and a sweep at startup catches anything a path missed.
- verify_database_file validates the table, the id columns, their foreign
  keys and indexes, and the stored values.

* feat(database): migrate old workflow rows in batches, then reclaim space

The workflow storage migration rewrites old-form image rows (text inline)
to document ids, batch by batch:

- Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or
  16 MB of text, found by a rowid range search and read one row at a time.
- Each text is stored or reused, then read back through the one accessor
  before its row changes; a text that would not come back identical fails
  and rolls back the whole batch.
- A value that is not text stays inline and is counted as protected.
- A crash or a stop leaves committed batches new form and the rest old
  form; both read the same, and a rerun continues. No media file is opened.
- Unreferenced documents are swept at the end.

The maintenance preview gains a read-only workflow_storage plan (records to
rewrite, inline bytes where SQLite has octet_length, the Reclaim space and
older-version notes). reclaim_space sweeps, runs VACUUM and truncates the
WAL so the freed pages go back to the disk.

* feat(maintenance): store each workflow once as an Activity job

The reviewed maintenance plan gains workflow_storage. Applying it starts
"Store each workflow once": kind maintenance, records admission,
cancellable, groups changed/protected/errors.

- The promised verified safety backup comes first; a failed backup ends the
  job with nothing rewritten. The apply's up-front disk check covers it.
- Old-form rows are rewritten batch by batch on the event loop's executor,
  with progress after each committed batch. Cancel stops between batches;
  a failed batch rolls back whole and fails the job with a coded message.
  Either way every record reads as before and the next apply continues.
- When every row was looked at, "Reclaim space" starts: exclusive (runs
  alone), not cancellable. It checks the disk for VACUUM's copies of the
  live pages, then sweeps, runs VACUUM and truncates the WAL. If another job
  holds the registry it does not start, and the finished migration says so
  and names Optimize database.
- No image or video file is opened; tests track every open under the
  output folder and compare the media bytes and mtimes.

* feat(maintenance): document the workflow storage contract

TESTING.md's maintenance contract gains the workflow storage rules: the
documents table and its two forms, the one accessor, garbage in the same
transaction plus the sweeps, the batched migration and Reclaim space, the
media and memory guarantees, and that a deduped backup cannot be restored
into an older version.

A test covers the preview on SQLite before 3.43 (no octet_length, as in
older Windows Python builds): the inline-bytes detail is left out.

* feat(database): keep the migration safe row by row and beside other writers

Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe:

- M1: a batch also ends after 0.5 s, and the loop leaves the write lock
  free for 0.15 s after each committed batch (longer than SQLite's longest
  busy poll), so the image monitor and run counting on another connection
  get in between. RED: a threaded writer with a 2 s busy timeout hit
  "database is locked" (4 of 5 links landed).
- M2: each row runs in its own savepoint. A row that cannot be rewritten
  (undecodable text, a text that would not come back) is rolled back alone,
  stays inline and is listed in errors; the rest of the batch goes on.
- M3: a reused document is decompressed and checked before its id is handed
  out, and repaired from the text (whose hash is its key) when damaged;
  verify_database_file reads every document and refuses a damaged one.
- M5: the two legacy generated_images rebuilds refuse a file that already
  has the document id columns instead of dropping them.
- The preview names the pre-migration backup and says it is always taken.

* feat(maintenance): always back up before storing workflows once

Final review fixes I1, I2 and I3 for the workflow storage dedupe:

- I1: the migration always takes a verified backup, whatever
  maintenance.safety_backup says; the plan reports backup.policy
  "required" and the disk check counts it.
- I2: that backup is prompts.db.pre-workflow-storage_<stamp>, outside the
  pruned prompts.db.backup_* pool and kept until the user removes it; the
  job's last line and the preview name it, and the legacy restore accepts
  it. README's changelog warns that an older version opens and restores a
  rewritten database but shows no workflows, and says how to go back.
- I3: tools/workflow_text_digests.py writes the sha256 of every record's
  stored texts (read-only; inline before the rewrite, through the accessor
  after), so a diff before and after proves byte identity.
- A row the migration could not rewrite is listed in the job's errors and
  named in its last line, and Reclaim space still runs.

* feat(database): fail the batch on system errors and name a damaged document

Re-review fixes R1 and R6 (database side), plus R3's helpers:

- R1: a row is kept inline only for its own cause: text SQLite cannot
  decode, or a text that would not come back. A full disk, an I/O error, a
  malformed database or a bug now fails the batch and rolls it back,
  instead of listing every remaining row as "could not be rewritten".
- R6: verify_database_file names the damaged document and says what to do
  (restore a backup from before the damage, or store an image with the same
  workflow, which repairs it).
- R3: pre_migration_backups lists the kept backups, earliest first; the plan
  says "Nothing to rewrite" and names an existing kept backup, and the older
  version note says new images use the new form from the first start and
  names the earliest backup as the way back.

* feat(maintenance): one kept pre-migration backup and nothing-to-do refusal

Re-review fixes R2, R3, R4, R5 and R6 (job side):

- R3: Apply refuses a plan with nothing to rewrite (409 nothing-to-do, no
  job, no backup). Only the first apply keeps a pre-workflow-storage
  backup; later applies (after a Cancel, a rerun) take a pooled safety
  backup, forced on whatever the policy, so one copy is never pruned.
- R4: a second-connection writer test runs against the live job loop; with
  its pause removed the writer hits "database is locked" (4 of 5 links).
- R5: the digest tool loads only database/workflow_documents.py by path (no
  package start-up, no log handler) and opens a stopped database with
  immutable=1, so it leaves no -wal/-shm; a live one is read through its WAL.
- R6: a backup that fails on a damaged document says which one in the job
  line.
- R2: README's changelog says images recorded by this version use the new
  form from the first start, and the way back is the earliest
  pre-workflow-storage backup (or a pre-update backup if the job never ran).
 2026-10-24 06:50:53 -07:00
Vito f68e92b3b6 feat(system): maintenance backend (#313) 
* feat(maintenance): reviewed, read-only maintenance plans

POST /prompt_manager/maintenance/preview reviews duplicate_prompts,
optimize, unused_components or missing_links on its own mode=ro snapshot
and answers a Plan (scope, protected reasons, backup policy, media, at most
50 detail rows). The fingerprint, targets and judged sources stay in a
server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB,
never truncated; a restart forgets them), bound to the database it read.

The previews reuse the legacy queries: duplicate groups and the merge are
shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the
#239 protection predicates the plan reports, and missing links are judged
by the health check's link_states. Legacy /maintenance keeps its shape;
vacuum now reports success:false when VACUUM fails, and
cleanup_orphaned_images skips links under unavailable sources.

* feat(maintenance): apply reviewed plans as Activity jobs

POST /prompt_manager/maintenance/apply {token} starts a maintenance job
(records admission, not cancellable, groups changed/protected/errors).
The token is consumed on the loop turn the job starts: a second apply is
409 plan-used naming the job, a busy refusal keeps the token, an unknown
or expired one is 410 plan-expired, and a restore since the review is
409 storage-changed.

With the safety backup policy on (maintenance.safety_backup, default true,
saved through the settings route with the revision), the job first writes
a backup into a temp file beside the database, makes it a standalone
rollback-journal file, verifies it, fsyncs it and renames it onto a unique
<db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file
and nothing changed. The apply then rechecks the reviewed fingerprint
inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of
400 ids, counts changed and protected records, logs the backup by name
only and prunes old safety backups. With the policy off the log and the
status line say "No safety backup".

* feat(maintenance): find and remove duplicate files through Activity

Duplicate image files move onto the job registry:

- POST /prompt_manager/jobs/duplicates finds content-identical media
  files over every effective source as a cancellable read job
  ("Find duplicate image files", group groups_found); the server holds
  the groups per job and GET /prompt_manager/maintenance/duplicates/{id}
  pages them ten at a time.
- The duplicate_files preview binds a removal to the reviewed keepers.
  Every keeper and removal is checked again: containment as text before
  any filesystem access, then not a link, still inside its source, same
  size and hash. The plan lists the database links that move.
- Apply runs duplicate_cleanup (records, not cancellable): the promised
  safety backup, then per file a recheck, the unlink with every handle
  closed and its thumbnail. A changed file is kept, the first failed
  unlink stops the job with a coded message (never the OS text), and only
  removed files have their image links and favorites moved to the kept
  copy.
- The legacy scan, stream, status and delete routes keep their shapes on
  the same registry kinds; the hashing, grouping and thumbnail removal
  are shared with them.

Coordinator rulings on Tasks 15-16:

- Safety backups share the restore pool of 10 `<db>.backup_*` files; the
  pool is now also pruned when an apply fails after its backup.
- Plans report the safety backup's size and the free disk space, and
  apply refuses up front (507 insufficient-space) when the backup or
  VACUUM does not fit, so no job fails halfway for space.

* fix(maintenance): never remove the kept copy; move records before unlink

Final review of #313:

- C1: the duplicate find tells files apart by (st_dev, st_ino), so a
  source listed twice, a symlinked source inside another and a hardlink
  are one file. The preview and the job refuse a removal that
  os.path.samestat matches with its kept copy.
- I1: each file's records move to its verified kept copy in one short
  transaction before that file is unlinked; a failed move deletes
  nothing and a failed unlink leaves the records on the identical kept
  copy. The status line counts files removed and not removed.
- I2: links are looked up under every spelling of the source (the real
  path, the configured and monitored folders, ComfyUI's output).
- I5: image tags join the kept copy as their union and the removed
  copy's pending, undecided Needs review items are settled; the plan
  says so.
- I3: the VACUUM disk check measures SQLite's real temporary folder.
- M1: the token is looked at again with no await before the job starts.
- M2: the backup decision is made once and handed to the job.
- M3: backups are linked into place; startup sweeps interrupted temps.
- M4: Optimize runs alone (new "exclusive" admission) and its plan says
  the database is locked while VACUUM runs.
- M5: the reviewed duplicate merge is strict: collisions and Library
  titles are protected, and errors roll back. The legacy cleanup is
  unchanged.
- M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests.
- M9: the preview binds files by size, mtime and inode without hashing;
  the job hashes right before each unlink.

* fix(maintenance): protect one file reached through two mounts

Re-review of #313, round 2:

- R1: a removal on another device than its kept copy, with the same
  size and mtimes within 2 s, is protected when its inode is equal or its
  path within its source is the same ("May be the same file reached
  through another mount or share"), in the preview and again right before
  each unlink. The job holds the kept copy open during the unlink, so on
  Windows deleting the kept copy itself through another path fails.
- R2: a file's spellings are found by containment: every configured,
  monitored, ComfyUI-output or listed folder whose stat identity is a
  folder above the file. Parent spellings of nested sources and case
  variants match; favorites move under every listed root holding the
  file. Lookups stay index searches.
- m1: a failed unlink's row says the links now name the kept copy.
- m2: a volume with no file ids reads "Can't confirm this is a separate
  file on this drive".

* fix(maintenance): stat duplicate candidates with os.stat

Python 3.10's Path.stat uses the os.stat bound when pathlib was imported,
so the find and the per-file checks could see different stat calls. The
find now calls os.stat directly, like every other check; the mount tests
failed on py3.10 only.

* fix(maintenance): recheck the kept copy right before each unlink

Re-review 2 of #313:

- n1: the cross-mount guard also matches when one path within its source
  ends with the other on a whole folder name (sources listed at different
  depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount).
- n2: after a file's records move and right before its unlink, the held
  kept copy must still be the file at its path (fstat of the hold against
  os.stat) and the cross-mount guard must still hold; otherwise the file
  is kept, its records move back to it, and it counts as not removed.

* test(maintenance): the kept-copy race tests hold on Windows

On Windows the removal job holds the kept copy open without
FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during
the record move is refused. The tests now assert that native outcome:
the kept copy stays and the removal goes ahead.

* fix(maintenance): restore a backup of any size the disk has room for

The legacy restore refused the maintainer's own 1.6 GB safety backup with
"File too large. Maximum size is 100MB", a fixed cap since #153. Free
disk space now decides: the upload, its staged copy and the swap must fit,
under a 64 GiB ceiling. A request with a Content-Length that does not fit
is refused before anything is written, one without as soon as its running
size does not, with 507 insufficient-space and a message without paths.
The upload still streams and its temp file is removed on every path.
 2026-10-24 05:28:44 -07:00
Vito 50703c960a feat(review): offer inspection-profile choices for untraceable batches (#312) 
* feat(review): offer inspection-profile choices for untraceable batches

Qwen image-edit batches run several KSamplers, each with its own
TextEncodeQwenImageEdit prompt, all saved under one prefix. The scan's
default profile knows neither the encoder nor the string nodes, so every
candidate was unresolved and the files left Needs review as "could not be
traced", although the inspection profile resolves every text.

When the default profile is ambiguous without a choice and nothing links
the file, the scan now resolves it with the inspection profile and, with at
least two choosable candidates, queues the item from that evidence. The
evidence records profile "inspection" and the semantics table version
(SHA-256 of its canonical JSON); the item's resolver_version folds the
profile, RESOLVER_VERSION and the table version, so a table change is
handled like a resolver change (I2). Apply re-resolves the capture with the
recorded profile, re-checks the chosen candidate hash and links through the
compatible-only ingestion. The inspection profile never links on its own,
ingestion keeps the default profile, and the live monitor never queues.

* feat(review): supersede stale items and inspect each file state once

A semantics table or resolver version change no longer leaves two pending
items for one file. A new item supersedes the other items of the same bytes
when every one is pending and undecided (revision 0, no decision row, the
self-heal's safe delete). Beside a decided or advanced item nothing new is
stored, and the rescan's stat shortcut skips those bytes under any version.

The inspection item version now folds a narrower fingerprint: the table
restricted to the graph's node classes plus its class-independent parts, so
an edit about other classes leaves every item alone.

The inspection profile runs once per file state. The scan resolves the
untraced file at batch level, skips files already queued or decided at their
current size and mtime under their version, and remembers files it found
untraced in this process. The capture reuses the scan's result when it holds
the same records.

* feat(review): contain inspection failures, version the inspection resolver

Final-review fixes for the inspection-profile review choices.

- An inspection failure on one file is logged, counted in review_errors and
  reported unresolved; the scan goes on.
- INSPECTION_RESOLVER_VERSION and the inspection profile's budgets are folded
  into the item version, so an inspection-only resolver change re-inspects
  and supersedes the item. An older version's pending, undecided item of
  bytes now untraced leaves the queue, so an item never stays stuck at 409.
- Fingerprint tests: an edit of the encoder's own entry moves the version,
  an unused dict-keyed class leaves it.
- The untraced memory is a plain dict of 16-byte digests (8.6 MiB measured
  at its 65,536-entry cap).
- Only scans add items: the monitor's pass records nothing and only clears
  the items of outputs it links. Tests and the browser review fixture
  record through the scan's recorder.
- record_review_item documents the byte-record refresh it commits when it
  refuses a new item.

* fix(review): an unusable semantics table never retires recorded items
 2026-10-23 18:07:14 -07:00
Vito b4c7a15045 feat(review): Needs review page, live queue and concept comparison (#288) 
* feat(review): add review queue schema with restore validation

Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.

* feat(review): persist ambiguous sampler evidence from scans and new outputs

The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.

Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.

* feat(review): explicit association decisions with compatible linking

Needs review gains its queue reads and decisions:

- GET /prompt_manager/review/items lists pending items oldest first with
  the exact pending total and an HMAC keyset cursor bound to the storage
  epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
  label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
  for a capture of the item's bytes that the current file still hashes to
  and whose evidence is unchanged, through the ingester's compatible-only
  rules in the same transaction as the decision row; a conflict writes
  nothing. Leave unresolved records the decision and links nothing.

ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.

Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.

* fix(review): build the spelling validation test from native paths

On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.

* fix(review): Apply sees the file under every spelling and rechecks only its choice

Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.

Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.

The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.

* fix(review): validate stored review evidence once per startup

Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.

Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.

* fix(review): keep one capture worker for interactive requests

Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.

* fix(review): refresh the recorded spelling when a file is seen under a new one

A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.

* fix(review): final capture store close and recoverable stale cursors

Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.

A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.

* fix(review): make only the shutdown cleanup final

The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.

* refactor(admin): move Needs review recording into its own mixin

_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.

* fix(review): record the review fix round and the skip's residual risks in TESTING.md

The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.

* feat(review): bind queue cursors to the storage epoch only

A decision or a new item (the monitor records one per generation) no
longer makes an open cursor stale: the (created_at, id) keyset stays
ordered when items leave or are appended. A restore still answers
storage-changed and an unverifiable cursor review-conflict.

* feat(review): Needs review page, SAVED VIEWS row and Activity hook

The concept's Metadata review page at #/review: the notice with the
exact pending count, the queue's media cards (shared with Images through
image-card.mjs), and the Resolve the association panel with the sampler
context, the chosen path's prompt blocks, Leave for later and Apply
association. Apply takes a fresh capture; refusals keep the selection
and explain inline; an unanswered decision is resent once. SAVED VIEWS
lists Needs review after Favorites with its count (retires D50), and an
ended scan that recorded review items offers Review metadata in Activity
(D87). Queue items carry the card's media url.

* feat(review): re-read an open Needs review page when its queue changes

An open Needs review page now follows the queue without polling. A new
server-sent event stream, GET /prompt_manager/review/events, sends
"ready" when it opens and "change" after every committed queue change:
an item the scan or the monitor recorded, a spelling refresh, a decision
from any tab, and a restore. database/review_changes.py notifies after
the commit from whichever thread wrote it. The page re-reads its first
page on each event, keeps the open item, its choice and an unanswered
decision, and re-reads the SAVED VIEWS count with it. When the open item
left the queue, the next one opens with a note. A keepalive comment
every 25 s lets the server drop a closed tab; shutdown ends every stream.

Leaving the page mid-decision now releases that decision's capture.

D-NEW-F1 is renumbered D119 in markers and TESTING.md.

* test(review): Needs review page workflows

The review fixture now seeds a real monitor-fallback conflict (the
monitor pipeline links a new output to the most recent prompt, then
queues it), a limited item and a file under no current root, and can add
an output through the monitor or leave one for a metadata scan.

New workflows: Apply links the chosen pair to that file only; a real
association conflict leaves the links unchanged and explains; a limited
item and an unavailable file can only be left unresolved; a tag save
leaves the item pending; keyboard focus, labels and touch targets; an
open page shows items the monitor and a scan add and a decision made in
another tab; leaving the page releases captures, the change stream and
requests. The faked 409 conflict test is replaced by the real one.

* test(review): cover the card Tag button and raise the review floors

The card's Tag on Needs review now has a workflow, and the page's queue
change hook has no unused default. review.mjs and review-state.mjs
floors rise to their measured coverage; no other floor changes.

* feat(review): keep the loaded depth on live re-reads and read once per visit

A queue change re-read only the first 40 items, so the pages Load more had
shown vanished and the page jumped. Each re-read now reads page after page
until it holds as many items as were shown (a restore met after the first
page reads again from the start), so the cards and the scroll position stay.

The stream's ready made every visit read the queue twice. List reads now
answer the queue's version (library_epoch, review_revision) and ready names
the version read after subscribing; the page re-reads on ready only when the
queue it shows is older. The SAVED VIEWS count follows the page's total, so a
change costs no count read. A stream counts as open only once ready is sent.

* test(review): compare Needs review with the approved concept

review-concept.spec.mjs measures the whole Metadata review page as a tree
in the concept and in production, on the review fixture's new
concept-shaped queue (the concept's first three image names and sizes and
its first prompt). mapConceptReview applies only D29, D119, D3, production's
hooks and accessible names and sample data, and the page then matches at
1600x1100 and 390x844 in both themes. The gallery layout follows the
concept at every breakpoint, the SAVED VIEWS rows match with Needs review
after Favorites, the queue's Tag image dialog matches the needs-review
reference, and a dropped notice border or reordered actions fail.

The comparison found one gap: an unavailable file's card lost its media
area. Its empty media button now keeps the concept's 1.33 ratio (D119).
TESTING.md records the comparison, the extended D119 and a summary of the
image-tagging slice (routes, S1, capture limits, the record-changing job,
the no-download rule, the restart note). A ready whose version cannot be
read re-reads the queue (covered by a page test).

* test(review): give the multi-route shell comparison a fitting time budget

The dispatched CI run failed "shell geometry, tokens and controls match the
concept 390x844 light" on WebKit with a 30 s test timeout (no assertion
failed). Each of these tests makes twelve page loads with captures, hovers
and focus checks: 15-19 s on a CI WebKit runner on main, 22-31 s on this
run's loaded shard. With 2 CPUs and 3 workers it fails locally the same way
(6 of 12 timed out); with a 90 s budget, as the other multi-state concept
comparisons have, all 12 pass.

* fix(review): coalesce queue change events and answer queue thumbnails

- The change stream sends the first notice at once and collapses the
  notices of the next 0.4 s into one trailing change, so a busy scan costs
  an open page one re-read per 0.4 s at most (COALESCE_SECONDS).
- Queue items answer thumbnail_url, the same URL the Images listing builds
  (one shared helper, media_thumbnail_url), or null without a thumbnail.
- The "duplicate and refused decision send nothing" test spies on
  notify_review_change instead of reading the next event.

* fix(review): keep the live queue right through applies, Load more and lost streams

- A change heard while a decision is in flight is owed and read once the
  decision is refused or unanswered (I1).
- Load more has its own read: during a re-read it waits and continues from
  the re-read queue; a re-read started during Load more takes the click
  over, and a failed re-read leaves Load more clickable (I2, M7).
- A stream the browser gave up on (503, proxy error) reopens after 3 s,
  doubling to 30 s, with a quiet "Live updates are paused" line while it is
  down (I3, D-NEW-F2).
- A hidden tab closes its stream and a visible one reopens it; the ready
  version keeps that to at most one read (I4).
- Apply fences on the detail's epoch and a re-read under another epoch reads
  the open item again (M3); leaving during the observe releases its capture
  (M4); an unanswered decision that landed finishes as recorded (M5); cards
  follow a re-read's url, target and availability and show thumbnails
  (M6, M10).
- The unreadable-ready test asserts exactly one re-read (M1); the panel's
  long-file-name wrap carries its D119 marker and TESTING.md entry (M8).

* docs(review): number the live-updates-paused deviation D120

* fix(review): refuse monitor recording beside record-changing jobs

System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.

JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.

* test(backup): wait for the restore, not a 5 s timer, in the thread test

test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.

* fix(activity): drop Cancel job from a job that had ended before Activity opened

Since #289 the Activity list removed an item's actions only when they were
isConnected. On the first render the item is not in the page yet, so a job
that had already ended when Activity opened kept its "Cancel job" (and a
running job's progress was moved after its actions, then back). The list
now asks parentNode instead. The review spec "a scan job with review items
offers Review metadata in Activity" failed on this before the change
(Expected 0 buttons on a finished job, Received 1) and passes after it.

* fix(review): store non-finite evidence as null and count pending items

canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.

* fix(review): show the pending count on the System Overview review row

The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.

* feat(review): open Needs review from the System Overview review row

Review metadata on the Overview's review row now opens #/review, the
concept's go-review, instead of staying aria-disabled.

* fix(system): answer the Overview and Statistics only after every read ended

asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.

* fix(review): queue only ambiguous files that nothing links, and heal the queue

The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).

A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.

The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.

The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.

* fix(review): announce items that leave the queue, and seed only queueable files

#284 now queues a file only when its final outcome is ambiguous and unlinked,
and it removes the pending items of files that are now linked. That removal
is a queue change, so it calls notify_review_change after its commit, like
recording and decisions: an open Needs review page re-reads its queue, and
its count follows.

The review browser fixture wrote graphs that the scan's fallback links, and
it produced the conflict item through the monitor's fallback link. Neither is
queued any more. Its files now carry ComfyUI's Infinity stamp, which the
legacy parser cannot read, so they stay unlinked with the same two
candidates. The conflict item is queued first and its file is linked to
another prompt afterwards. "monitor" additions go through
record_review_file from a worker thread.

* fix(review): never queue an ambiguous file with nothing to choose

The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).

A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".

* test(review): seed the limited item from a graph with more candidates than stored

An A1111 block with two negative lines offers no candidate to choose, so
Needs review no longer queues it (#284). The fixture now records a graph of
17 samplers, which the recorder stores limited.
 2026-10-23 15:02:18 -07:00