ComfyAssets-ComfyUI_PromptManager

nodes/ComfyAssets-ComfyUI_PromptManager

Fork 0

Commit Graph

Author	SHA1	Message	Date
Vito	d2ea705a44	fix(tagging): Tag output images tags each image, never its prompt (#379 , PR 1) (#380 ) * fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1) The native "Tag output images" job now runs the image-tags loop over every still image the output folders list, linked or not: each image gets its own model tags (at most "Suggestions per image") in its image tag set, and its bytes are marked auto-tagged in a new media_image_autotags table. No prompt tag or prompt text is written. - database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional in old backups, refused without the annotation group. - add_image_tags/replace_image_tags take mark_autotagged; autotag_state and autotagged_locators read the marks through the one current-record rule. - "Skip already tagged" skips images whose current bytes are marked, counted in a stat-only pre-pass; hand tags do not count. - The review counts images and estimates the time (WD14 on CPU or CUDA). - System > Auto-tagging review rows and copy, D134 and the Auto-tagging contract follow. Part of #379 * fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379) The job read as ended while it still held the model gate, so a quick restart could be refused with model-busy. Pre-pass chunks now publish progress, and a cancel between chunks keeps exact counts. Part of #379 * fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379) The live check measured WD14 on CUDA at 0.167 s per image (100 images in 16.7 s with the model load), so the review estimate uses 0.17 s. A finished or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X min left"; it now ends with "<verb> <state> · N files processed". Part of #379 * test(browser): reload the admin page only after the in-place edit saved The gallery-to-admin workflow pressed Enter and reloaded as soon as the edited text showed; the page saves after showing it, so a reload that won the race abandoned the PUT (WebKit, run 40564511000). The test now waits for the save's answer. Part of #379 * test(tagging): the batch coverage row names the renamed #379 test	2026-10-27 20:30:54 +00:00
Vito	3f11ea6168	feat(annotations): Needs review queue backend (#284 ) * feat(review): add review queue schema with restore validation Add the Needs review schema group (media_review_metadata, media_review_items, media_review_decisions) beside the image annotation group it cites. review_schema_error validates the catalog, columns, foreign keys, row values and canonical 8 MiB evidence read-only; migrate_review_state installs it in a SAVEPOINT after the annotation group, and verify_database_file requires it under require_library_state. A review group without a valid annotation group is refused. Decisions null their pair and image navigation through indexes when those are deleted. Old backups and annotation-era backups restore and gain the group; browser fixtures clear the new tables on reset. * feat(review): persist ambiguous sampler evidence from scans and new outputs The output scan and the live monitor now queue every file the ingestion resolver finds ambiguous for Needs review. The file is captured, its metadata is read again from the capture and resolved with the default profile, and only an ambiguous result is stored, bound to the capture's annotation byte record and fenced by its storage epoch. Evidence holds one hashed projection per selectable candidate; none, more than 16 or over 8 MiB is a limited item. Recording is idempotent per bytes and evidence and never touches a decision. Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the job scope gains review_items/review_errors and the log a Needs review line. The monitor records after its fallback link, best effort. * feat(review): explicit association decisions with compatible linking Needs review gains its queue reads and decisions: - GET /prompt_manager/review/items lists pending items oldest first with the exact pending total and an HMAC keyset cursor bound to the storage epoch and the review index revision. - GET /prompt_manager/review/items/{id} adds the candidates (role texts, label, selectability), the dependency fingerprint and the digest. - POST /prompt_manager/review/items/{id}/decision applies a candidate only for a capture of the item's bytes that the current file still hashes to and whose evidence is unchanged, through the ingester's compatible-only rules in the same transaction as the decision row; a conflict writes nothing. Leave unresolved records the decision and links nothing. ingest_resolved_pair keeps its contract; its body moves into _ingest_resolved_pair_in, which runs in the caller's transaction. Items keep the file's recorded spelling so targets keep their case on Windows, and a rescan skips the copy and hash of a file whose size and mtime match the bytes an item already cites. * fix(review): build the spelling validation test from native paths On Windows the locator is case-folded and uses backslashes, so the fixed POSIX pair /out/a.png no longer folded to itself and the new spelling check refused the good case. The test now spells a mixed-case native path and folds it with os.path.normcase, and also refuses a locator of another file. * fix(review): Apply sees the file under every spelling and rechecks only its choice Apply now finds the legacy rows of the same physical file however its root is reached. The roots are real paths while the monitor links under the directory it watches, so a symlinked or junctioned root, subst or a mapped drive used to hide the monitor's row and Apply added a second, differently linked row. Same-name rows are resolved to their physical identity (normcase(realpath), else samefile); the dependency fingerprint covers all of them, two rows refuse the choice, one row is enriched under the ingester's compatible-only rules, and a same-name row the request did not classify is a changed association. Apply re-verifies only what the decision depends on: the bytes are still ambiguous and still offer the chosen candidate. A resolver change that rewords the reason or reorders candidates leaves pending items applicable. The resolver version is one constant in prompt_pair_resolver, shared by ingestion provenance and Needs review. The transaction's byte check receives the re-hashed digest (or the body's for Leave unresolved), every Apply refusal branch is tested with zero writes, and the root listing and realpath calls run on a worker thread. * fix(review): validate stored review evidence once per startup Startup used to run the full review validation twice (the early guard and the install) and the annotation group's row checks inside each, so every evidence document was parsed and re-encoded twice. The early guard is now catalog-only (a review group without annotation tables), and the install validates the group once, trusting the annotation group it has just validated. The evidence predicate no longer pattern-scans every document in SQL, because the streamed canonical pass proves it, and the codec is built once. Measured on a database with 40,000 items of 4 KiB evidence: startup open went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen. Restore and verify_database_file keep the full validation. * fix(review): keep one capture worker for interactive requests Needs review recording from the scan and the monitor shared the capture store's two workers with the Tag dialog and Apply, so a busy scan could answer a user's preview or decision with "Too many previews are open". Background captures now never take the last worker or capture slot, and wait up to two seconds for room before counting the file as not added. Interactive captures are admitted or refused at once, as before. * fix(review): refresh the recorded spelling when a file is seen under a new one A case-only rename on Windows or macOS keeps size and mtime_ns, so the rescan skipped the file and the item kept the old spelling. Its target was then formatted from a name the directory no longer lists, the capture refused it, and the item stayed pending with no way to apply it. The skip path and a duplicate insert now record the current spelling for every item at the locator and move the review index revision; item revisions do not change, so an open decision stays valid. Tested under case-folding semantics on every platform. * fix(review): final capture store close and recoverable stale cursors Closing the capture store at shutdown now runs under the store lock and is final: a monitor recording or scan batch still in flight gets a capture error instead of creating a new temp directory and restore subscription after the cleanup. A well-formed queue cursor this process cannot verify (opened before a ComfyUI restart, whose cursor secret is per process) now answers 409 review-conflict with the current total, like any other stale page, instead of an uncoded 400. Malformed cursors stay 400. * fix(review): make only the shutdown cleanup final The previous commit made every close of the capture store final, which also refused the explicit closes that swap in a fresh store. Only the host's on_cleanup hook is final now; an observe after it answers a coded capture error instead of an unhandled 500. * refactor(admin): move Needs review recording into its own mixin _record_scanned_review, _record_review_sync, record_review_file and _log_review_items move unchanged from py/api/admin.py to py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from growing further past the size ceiling. No behaviour change. * fix(review): record the review fix round and the skip's residual risks in TESTING.md The Needs review paragraphs now state the rescan skip's residual risks (an in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted or moved file leaves its pending item counted), the physical identity rule for Apply, the candidate-only recheck and shared resolver version, the spelling refresh, background capture admission, the final shutdown close, the stale-cursor answer and the measured startup validation cost. * fix(review): refuse monitor recording beside record-changing jobs System S1 (#289) gives every record change a registry admission class. The scan already records Needs review items under its own records job. The monitor's recording lasts one file and is not a job, so it now checks the records admission: while a records or guarded job or hold runs it records nothing, logs that the file waits for the next scan, and answers deferred. The fallback link stands and the next "new" scan reads the ambiguous file again, because a fallback link is not a resolved one. JobRegistry.blocker and active iterate a snapshot of the jobs, since the monitor's worker thread now asks them too. * test(backup): wait for the restore, not a 5 s timer, in the thread test test_database_is_usable_from_every_thread_after_restore let its worker wait 5 s for the restore. On a stalled Windows runner (py3.11, where the previous restore test also took 25 s instead of 0.3 s) the restore took longer, the wait timed out, and the worker read the old database before the restore ran. The worker now waits for the event under a 120 s bound that only a hung test reaches, reports a timeout as an error, and the main thread releases it even when the restore raises. With a restore delayed by 6 s the old test fails and the new one passes. * fix(review): store non-finite evidence as null and count pending items canonical_evidence now follows main's non-finite policy (#303): NaN and ±Infinity are written as null instead of raising, so a NaN-stamped graph never turns a review item limited. The System Overview reports the review queue's pending total in needs_review.pending, read in the same snapshot as the Library counts. * fix(review): show the pending count on the System Overview review row The Overview's review row now reads "N ambiguous image links" with the concept's line, from needs_review.pending. Its Review metadata action stays aria-disabled until the Needs review page ships. The system specs and their concept comparison follow the real count. * fix(system): answer the Overview and Statistics only after every read ended asyncio.gather answered at the first failed read while its sibling executor reads were still inside SQLite. In the suite the test teardown then deleted the database under the running read, and py3.10 on Ubuntu segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope). Both routes now wait for all their reads before answering. * fix(review): queue only ambiguous files that nothing links, and heal the queue The maintainer's live scan (19,908 files, scope "new") reported 4,677 files under Needs review, but the queue held 10,232 items: every file the strict resolver found ambiguous was queued, including the ones the legacy browser-parser fallback then linked (all sampled "added" rows) and the ones already linked to a known prompt (135 of 200 sampled "known" rows). A file is now queued only when its final outcome is ambiguous and unlinked: the resolver finds it ambiguous, the fallback links nothing and no saved row links it already. An ambiguous file that a saved row links is reported as unchanged ("Already linked to its prompt"), not as Needs review. The queue heals itself. After each batch, the scan removes the pending items of every listed file (read, or skipped as known) that a saved row now links. The monitor does the same for each ambiguous output it links, and never queues one. Only an undecided item is removed (pending, revision 0, no decision row), so a decision is never deleted or changed and the schema needs no new state. The scan job's scope.review_items now counts this scan's files that are in the pending queue after it, new or already there, so it agrees with the review group's ambiguous rows and with the queue total. review_added and review_settled are new, and the log gains "already in Needs review" and "left Needs review · now linked" lines. A duplicate whose item is decided is reported as "decided" and is not counted. * fix(review): never queue an ambiguous file with nothing to choose The maintainer's live queue held 39 items whose candidates all read "unresolved" ("No positive / No negative"). An ambiguous result is now reviewable only when at least two candidates are resolved with non-blank positive text. With none the file could not be traced; with exactly one, the untraced branches may still have made the file, so that text is not certain either. Both are reported unresolved, never linked and never queued (also the A1111 two-delimiter case, which has no candidates). A rescan removes the pending items of such files whose own evidence offers no choice, with the same safe delete as the linked settle (pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two candidates is kept. The job scope gains review_untraced and the log "= N images left Needs review · nothing to choose". * test(scan): pin only the per-source scope keys beside the review counters * test(system): match the scan scope keys beside the review counters	2026-10-23 14:19:11 -07:00
Vito	b7cf210f0f	feat(annotations): image tag storage, vocabulary participation and capture store (#276 ) * feat(annotations): add image annotation schema with restore validation Install the image annotation group beside image_favorites and saved_views: media_annotation_metadata (one row, schema version 1), media_annotation_files (one private locator per physical file), media_annotation_bytes (one record per observed SHA-256 with the verified size and mtime_ns) and media_image_tags (the ordered tag set of a byte record, joined into the global tags registry). The group migrates in a SAVEPOINT that validates before and after creating it, with no backfill from prompt or pair tags and no filesystem probe. annotation_schema_error reads only: it compares the catalog objects after whitespace compaction, the columns, foreign keys and indexes, and the persisted values (UUIDs, hex digests, bounds, the 4096-character locator cap measured past embedded NULs, exactly one metadata row). verify_database_file checks it with allow_missing=not require_library_state, so old backups restore and a migrated restore stage must carry it. Tests pin the exact DDL, a fresh install and idempotent reopen, a current database without the group gaining it with legacy rows untouched, every partial or malformed shape rejected without repair, install failure rolling back and retrying, a WAL backup/restore round trip, staged-failure immutability of source, live and safety files with every handle closed, and the require_library_state gate. * feat(annotations): persist per-image tag sets in the shared vocabulary Add database/annotations.py (AnnotationOperations, mixed into PromptDatabase): read_image_tags, replace_image_tags and add_image_tags run in one _pair_transaction fenced by the storage epoch, own the byte record by exact locator and digest, always supply UUID ids, rewrite the ordered junction and bump tag_revision plus image_index_revision. Revision mismatches raise AnnotationCommandError("tag-conflict") with the current set; the identical set at the current or previous revision is replayed without writes; a union over 100 names is "tag-limit". normalize_image_tags validates each name with validate_vocabulary_tag_name and folds case-insensitive repeats keeping the first spelling. refresh_image_fingerprint, current_image_tags (chunked, fingerprint matched, newest verification wins) and iter_image_tag_records serve the S1 cheap reads. Vocabulary participation: _move_tag_links also carries media_image_tags, legacy merge_tags moves image-only sources, tag rows gain image_count (files with any byte record carrying the tag), impact gains images and rename/delete receipts gain images_changed. vocabulary-state.mjs accepts the optional counts and names images in the delete text. Browser fixtures clear the three annotation data tables on reset (never the metadata row); TESTING.md documents the schema beside favorites. * feat(annotations): add bounded private media capture store Add utils/media_capture.py: CaptureStore copies one rooted media file into an owner-only prompt-manager-captures-* temp directory, streaming it in 1 MiB chunks while computing the whole-file SHA-256, and compares the opened descriptor with the path (dev, ino where meaningful, size, mtime_ns, ctime_ns) before and after the copy; any disagreement raises CaptureChanged and removes the copy. Symlinked files or directories, root escapes, non-media names and directories raise CaptureDenied; the locator is normcase(join(realpath(root), relative)), the listing's identity, so a symlinked root matches it. Limits are the plan's: 256 MiB per file on stat and stream (CaptureTooLarge), 8 completed captures, 512 MiB reserved and 2 capture/verify workers with nonblocking admission (CaptureCapacity), 900 s idle and 1800 s hard lifetimes on an injected clock. acquire and release_use keep an in-use capture past expiry, release and invalidate_all until its last use ends; close removes only owned files and the owned directory. A header-only probe of the copy marks Generate unsupported above 4096 px per side or 16,777,216 px, for unreadable headers and for video; verify re-hashes the current file without a copy. Every handle closes before unlink. tests/test_media_capture.py covers exact bytes and owner-only modes, alias and escape denial, the symlinked-root locator, change detection during the copy (size, same-size rewrite with restored mtime through the POSIX change time, inode replacement), both size caps, count, byte and worker admission, idle and hard expiry with in-use retention, cleanup scope, handle release at unlink, dimension and video readiness, and the allowlist equality with the inspection service. TESTING.md documents the store and the Windows change-time limit. * test(annotations): wait on the POSIX change time only where it exists The same-size rewrite case waits for the filesystem clock to pass the source's ctime so the detection is deterministic on Linux's coarse timestamps; Windows reports the creation time as st_ctime, where the probe could never advance, so the wait runs only on POSIX. * fix(annotations): compare the change time only between fstat reads Python 3.12+ on Windows reports the change time from stat but the creation time from fstat, so a freshly written file whose creation and write straddle a clock tick failed the pre-read identity check with CaptureChanged. The path-versus-descriptor checks now compare size, mtime_ns and dev/ino; ctime_ns is compared between the fstat before and after the copy, where it still detects a same-size rewrite with a restored mtime on POSIX. The symlinked-root test asserted the locator against the unresolved temp path, which macOS (/var -> /private/var) and Windows (short names) resolve differently; it now compares against the real path. * test(browser): give the inspector evidence matrix the concept comparisons' budget The touch run of 'full inspector attaches matched theme and mobile evidence with reachable content' captures 32 screenshots across four viewports and took 23.5 s, then 31.6 s, in the sharded coverage job, hitting the 30 s default. Like the concept comparison specs it now sets an explicit 120 s timeout; the CI-like pinned run (4 CPUs, 3 workers, repeat-each 3) passes 198/198 with the matrix at 18-20 s on touch. * fix(annotations): compare capture identity like with like and keep paths out of errors Native Windows CI (Python 3.12/3.13) rejected untouched files with CaptureChanged because the path stat and the descriptor fstat disagree there (change time versus creation time, possibly dev/ino encodings). The descriptor is now checked against the path only on the API-stable facts (size, mtime_ns); the full identity (size, mtime_ns, ctime_ns, dev/ino where meaningful) is compared between the fstat before and after the copy and between the path stats before and after. A test injects shifted fstat results to simulate the Windows discrepancy, and a fresh file is captured twice with no platform guard. Error messages carry strerror only, never the path (the path goes to the debug log); a test asserts no CaptureError names the root. Files are unlinked outside the store lock; close() during an in-flight capture lets that worker remove its temp file and then the directory. get() documents that readers of capture.path must acquire/release_use. The test fixture resolves its temp root (macOS /private/var, Windows 8.3 names). TESTING.md states the real identity contract. * fix(annotations): move image revisions when a fingerprint or the vocabulary changes A replayed save that observes a new verified (size, mtime_ns) now bumps image_index_revision, like refresh_image_fingerprint, so caches keyed on it see the record become current again; an identical replay still moves nothing. Vocabulary rename, merge and delete (native and legacy paths) bump the tag_revision of every image set carrying the source tag, in the same transaction and before the links move or cascade, plus one image_index_revision bump when any set was affected. A Tag dialog opened before the change therefore gets tag-conflict with the renamed set as current instead of recreating the old name. * test(annotations): accept change detection wherever fstat exposes a change time Native Windows CI on Python 3.13 detected the same-size, mtime-restored rewrite through the descriptor's change time (3.10-3.12 report the creation time there and complete the torn copy). The test asserted the platform, not the signal: it now accepts CaptureChanged plus temp removal wherever it happens, requires detection on POSIX and on Windows 3.13+, and otherwise requires the digest to describe the copied bytes. A new test injects a drifting fstat change time so the detection path runs on every platform; TESTING.md records the limit as "Windows before Python 3.13". * test(annotations): assert the rewrite outcome, not the platform Native Windows CI detected the same-size, mtime-restored rewrite on 3.12 and missed it on 3.13 in the same run: the path stat's change time (Python 3.12+ fast path) only moves when the clock ticks between the pre-stat and the rewrite, and the tick wait was skipped on Windows. The wait now runs everywhere with fresh probe names (Windows tunnels the creation time of a reused name), and the test asserts the outcome: a detected change leaves no temp file; an undetected one is accepted on Windows only and must produce a digest of the copied bytes; POSIX must detect it. The injected-fstat test keeps the detection path covered on every platform. * fix(images): keep the Filters label node across a listing reload Every listing reload reports the active refinement count, and setActiveFilters rewrote the button's textContent even when the label was unchanged, replacing its text node. WebKit drops a click whose mousedown hit a node that was detached before mouseup, so a Safari user who types a search and clicks Filters inside the 250 ms debounce lost the click, and the webkit page workflows timed out in the source-change recovery tests (1 in ~30 runs of the pinned image; an instrumented probe showed pointerdown/mousedown, a DOM mutation, pointerup/mouseup and no click event). The label is rewritten only when it changes. The toolbar unit test counts textContent writes (1 for an unchanged count before, 0 after), and a workspace-images test pins the text node identity across a search reload: 0/6 before the fix on chromium and webkit, 6/6 after, with the recovery and Refine tests 36/36 in the pinned CI-like run.	2026-10-22 07:19:53 -07:00

Author

SHA1

Message

Date

Vito

d2ea705a44

fix(tagging): Tag output images tags each image, never its prompt (#379 , PR 1) (#380 )

* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1)

The native "Tag output images" job now runs the image-tags loop over every
still image the output folders list, linked or not: each image gets its own
model tags (at most "Suggestions per image") in its image tag set, and its
bytes are marked auto-tagged in a new media_image_autotags table. No prompt
tag or prompt text is written.

- database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional
  in old backups, refused without the annotation group.
- add_image_tags/replace_image_tags take mark_autotagged; autotag_state and
  autotagged_locators read the marks through the one current-record rule.
- "Skip already tagged" skips images whose current bytes are marked, counted
  in a stat-only pre-pass; hand tags do not count.
- The review counts images and estimates the time (WD14 on CPU or CUDA).
- System > Auto-tagging review rows and copy, D134 and the Auto-tagging
  contract follow.

Part of #379

* fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379)

The job read as ended while it still held the model gate, so a quick
restart could be refused with model-busy. Pre-pass chunks now publish
progress, and a cancel between chunks keeps exact counts.

Part of #379

* fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379)

The live check measured WD14 on CUDA at 0.167 s per image (100 images in
16.7 s with the model load), so the review estimate uses 0.17 s. A finished
or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X
min left"; it now ends with "<verb> <state> · N files processed".

Part of #379

* test(browser): reload the admin page only after the in-place edit saved

The gallery-to-admin workflow pressed Enter and reloaded as soon as the
edited text showed; the page saves after showing it, so a reload that won
the race abandoned the PUT (WebKit, run 40564511000). The test now waits
for the save's answer.

Part of #379

* test(tagging): the batch coverage row names the renamed #379 test

2026-10-27 20:30:54 +00:00

Vito

3f11ea6168

feat(annotations): Needs review queue backend (#284 )

* feat(review): add review queue schema with restore validation

Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.

* feat(review): persist ambiguous sampler evidence from scans and new outputs

The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.

Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.

* feat(review): explicit association decisions with compatible linking

Needs review gains its queue reads and decisions:

- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.

ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.

Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.

* fix(review): build the spelling validation test from native paths

On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.

* fix(review): Apply sees the file under every spelling and rechecks only its choice

Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.

Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.

The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.

* fix(review): validate stored review evidence once per startup

Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.

Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.

* fix(review): keep one capture worker for interactive requests

Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.

* fix(review): refresh the recorded spelling when a file is seen under a new one

A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.

* fix(review): final capture store close and recoverable stale cursors

Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.

A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.

* fix(review): make only the shutdown cleanup final

The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.

* refactor(admin): move Needs review recording into its own mixin

_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.

* fix(review): record the review fix round and the skip's residual risks in TESTING.md

The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.

* fix(review): refuse monitor recording beside record-changing jobs

System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.

JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.

* test(backup): wait for the restore, not a 5 s timer, in the thread test

test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.

* fix(review): store non-finite evidence as null and count pending items

canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.

* fix(review): show the pending count on the System Overview review row

The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.

* fix(system): answer the Overview and Statistics only after every read ended

asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.

* fix(review): queue only ambiguous files that nothing links, and heal the queue

The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).

A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.

The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.

The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.

* fix(review): never queue an ambiguous file with nothing to choose

The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).

A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".

* test(scan): pin only the per-source scope keys beside the review counters

* test(system): match the scan scope keys beside the review counters

2026-10-23 14:19:11 -07:00

Vito

b7cf210f0f

feat(annotations): image tag storage, vocabulary participation and capture store (#276 )

* feat(annotations): add image annotation schema with restore validation

Install the image annotation group beside image_favorites and saved_views:
media_annotation_metadata (one row, schema version 1), media_annotation_files
(one private locator per physical file), media_annotation_bytes (one record
per observed SHA-256 with the verified size and mtime_ns) and media_image_tags
(the ordered tag set of a byte record, joined into the global tags registry).

The group migrates in a SAVEPOINT that validates before and after creating
it, with no backfill from prompt or pair tags and no filesystem probe.
annotation_schema_error reads only: it compares the catalog objects after
whitespace compaction, the columns, foreign keys and indexes, and the
persisted values (UUIDs, hex digests, bounds, the 4096-character locator cap
measured past embedded NULs, exactly one metadata row). verify_database_file
checks it with allow_missing=not require_library_state, so old backups
restore and a migrated restore stage must carry it.

Tests pin the exact DDL, a fresh install and idempotent reopen, a current
database without the group gaining it with legacy rows untouched, every
partial or malformed shape rejected without repair, install failure rolling
back and retrying, a WAL backup/restore round trip, staged-failure
immutability of source, live and safety files with every handle closed, and
the require_library_state gate.

* feat(annotations): persist per-image tag sets in the shared vocabulary

Add database/annotations.py (AnnotationOperations, mixed into
PromptDatabase): read_image_tags, replace_image_tags and add_image_tags
run in one _pair_transaction fenced by the storage epoch, own the byte
record by exact locator and digest, always supply UUID ids, rewrite the
ordered junction and bump tag_revision plus image_index_revision.
Revision mismatches raise AnnotationCommandError("tag-conflict") with
the current set; the identical set at the current or previous revision
is replayed without writes; a union over 100 names is "tag-limit".
normalize_image_tags validates each name with validate_vocabulary_tag_name
and folds case-insensitive repeats keeping the first spelling.
refresh_image_fingerprint, current_image_tags (chunked, fingerprint
matched, newest verification wins) and iter_image_tag_records serve the
S1 cheap reads.

Vocabulary participation: _move_tag_links also carries media_image_tags,
legacy merge_tags moves image-only sources, tag rows gain image_count
(files with any byte record carrying the tag), impact gains images and
rename/delete receipts gain images_changed. vocabulary-state.mjs accepts
the optional counts and names images in the delete text.

Browser fixtures clear the three annotation data tables on reset (never
the metadata row); TESTING.md documents the schema beside favorites.

* feat(annotations): add bounded private media capture store

Add utils/media_capture.py: CaptureStore copies one rooted media file
into an owner-only prompt-manager-captures-* temp directory, streaming
it in 1 MiB chunks while computing the whole-file SHA-256, and compares
the opened descriptor with the path (dev, ino where meaningful, size,
mtime_ns, ctime_ns) before and after the copy; any disagreement raises
CaptureChanged and removes the copy. Symlinked files or directories,
root escapes, non-media names and directories raise CaptureDenied; the
locator is normcase(join(realpath(root), relative)), the listing's
identity, so a symlinked root matches it.

Limits are the plan's: 256 MiB per file on stat and stream
(CaptureTooLarge), 8 completed captures, 512 MiB reserved and 2
capture/verify workers with nonblocking admission (CaptureCapacity),
900 s idle and 1800 s hard lifetimes on an injected clock. acquire and
release_use keep an in-use capture past expiry, release and
invalidate_all until its last use ends; close removes only owned files
and the owned directory. A header-only probe of the copy marks Generate
unsupported above 4096 px per side or 16,777,216 px, for unreadable
headers and for video; verify re-hashes the current file without a
copy. Every handle closes before unlink.

tests/test_media_capture.py covers exact bytes and owner-only modes,
alias and escape denial, the symlinked-root locator, change detection
during the copy (size, same-size rewrite with restored mtime through
the POSIX change time, inode replacement), both size caps, count, byte
and worker admission, idle and hard expiry with in-use retention,
cleanup scope, handle release at unlink, dimension and video
readiness, and the allowlist equality with the inspection service.
TESTING.md documents the store and the Windows change-time limit.

* test(annotations): wait on the POSIX change time only where it exists

The same-size rewrite case waits for the filesystem clock to pass the
source's ctime so the detection is deterministic on Linux's coarse
timestamps; Windows reports the creation time as st_ctime, where the
probe could never advance, so the wait runs only on POSIX.

* fix(annotations): compare the change time only between fstat reads

Python 3.12+ on Windows reports the change time from stat but the
creation time from fstat, so a freshly written file whose creation and
write straddle a clock tick failed the pre-read identity check with
CaptureChanged. The path-versus-descriptor checks now compare size,
mtime_ns and dev/ino; ctime_ns is compared between the fstat before
and after the copy, where it still detects a same-size rewrite with a
restored mtime on POSIX.

The symlinked-root test asserted the locator against the unresolved
temp path, which macOS (/var -> /private/var) and Windows (short names)
resolve differently; it now compares against the real path.

* test(browser): give the inspector evidence matrix the concept comparisons' budget

The touch run of 'full inspector attaches matched theme and mobile
evidence with reachable content' captures 32 screenshots across four
viewports and took 23.5 s, then 31.6 s, in the sharded coverage job,
hitting the 30 s default. Like the concept comparison specs it now sets
an explicit 120 s timeout; the CI-like pinned run (4 CPUs, 3 workers,
repeat-each 3) passes 198/198 with the matrix at 18-20 s on touch.

* fix(annotations): compare capture identity like with like and keep paths out of errors

Native Windows CI (Python 3.12/3.13) rejected untouched files with
CaptureChanged because the path stat and the descriptor fstat disagree
there (change time versus creation time, possibly dev/ino encodings).
The descriptor is now checked against the path only on the API-stable
facts (size, mtime_ns); the full identity (size, mtime_ns, ctime_ns,
dev/ino where meaningful) is compared between the fstat before and
after the copy and between the path stats before and after. A test
injects shifted fstat results to simulate the Windows discrepancy, and
a fresh file is captured twice with no platform guard.

Error messages carry strerror only, never the path (the path goes to
the debug log); a test asserts no CaptureError names the root. Files
are unlinked outside the store lock; close() during an in-flight
capture lets that worker remove its temp file and then the directory.
get() documents that readers of capture.path must acquire/release_use.
The test fixture resolves its temp root (macOS /private/var, Windows
8.3 names). TESTING.md states the real identity contract.

* fix(annotations): move image revisions when a fingerprint or the vocabulary changes

A replayed save that observes a new verified (size, mtime_ns) now bumps
image_index_revision, like refresh_image_fingerprint, so caches keyed
on it see the record become current again; an identical replay still
moves nothing.

Vocabulary rename, merge and delete (native and legacy paths) bump the
tag_revision of every image set carrying the source tag, in the same
transaction and before the links move or cascade, plus one
image_index_revision bump when any set was affected. A Tag dialog
opened before the change therefore gets tag-conflict with the renamed
set as current instead of recreating the old name.

* test(annotations): accept change detection wherever fstat exposes a change time

Native Windows CI on Python 3.13 detected the same-size, mtime-restored
rewrite through the descriptor's change time (3.10-3.12 report the
creation time there and complete the torn copy). The test asserted the
platform, not the signal: it now accepts CaptureChanged plus temp
removal wherever it happens, requires detection on POSIX and on
Windows 3.13+, and otherwise requires the digest to describe the
copied bytes. A new test injects a drifting fstat change time so the
detection path runs on every platform; TESTING.md records the limit as
"Windows before Python 3.13".

* test(annotations): assert the rewrite outcome, not the platform

Native Windows CI detected the same-size, mtime-restored rewrite on
3.12 and missed it on 3.13 in the same run: the path stat's change time
(Python 3.12+ fast path) only moves when the clock ticks between the
pre-stat and the rewrite, and the tick wait was skipped on Windows. The
wait now runs everywhere with fresh probe names (Windows tunnels the
creation time of a reused name), and the test asserts the outcome: a
detected change leaves no temp file; an undetected one is accepted on
Windows only and must produce a digest of the copied bytes; POSIX must
detect it. The injected-fstat test keeps the detection path covered on
every platform.

* fix(images): keep the Filters label node across a listing reload

Every listing reload reports the active refinement count, and
setActiveFilters rewrote the button's textContent even when the label
was unchanged, replacing its text node. WebKit drops a click whose
mousedown hit a node that was detached before mouseup, so a Safari
user who types a search and clicks Filters inside the 250 ms debounce
lost the click, and the webkit page workflows timed out in the
source-change recovery tests (1 in ~30 runs of the pinned image; an
instrumented probe showed pointerdown/mousedown, a DOM mutation,
pointerup/mouseup and no click event).

The label is rewritten only when it changes. The toolbar unit test
counts textContent writes (1 for an unchanged count before, 0 after),
and a workspace-images test pins the text node identity across a
search reload: 0/6 before the fix on chromium and webkit, 6/6 after,
with the recovery and Refine tests 36/36 in the pinned CI-like run.

2026-10-22 07:19:53 -07:00