a0fc5171e31dacc786ecd8dcb2ef23bc9fd7ccdc
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
3f11ea6168 |
feat(annotations): Needs review queue backend (#284)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(scan): pin only the per-source scope keys beside the review counters
* test(system): match the scan scope keys beside the review counters
|
||
|
Vito
|
b0a025a326 |
fix(resolver): read NaN as null in the scan's legacy-parser fallback (#304)
ComfyUI stamps "is_changed": [NaN] on nodes that always re-run, the PromptManager node among them. The legacy gallery and admin pages replaced NaN with null before JSON.parse, so they traced these files' prompts. The scan's fallback parsed records strictly, so every NaN-stamped file the resolver could not decide (hires-fix and refiner chains, custom latent nodes) fell to "Needs review" although legacy linked its prompt. parse_page_json parses as the legacy pages did, but only the NaN number reads as null: the word inside prompt text is kept. Infinity still makes the record unreadable, as it did in the pages. |
||
|
Vito
|
a6f6697c50 |
feat(activity): shared job registry, Activity page and Scan for metadata (#269)
* feat(activity): shared session job registry with the scan as its first kind The output scan now runs as the first kind of an in-memory JobRegistry (py/api/jobs.py, replacing py/api/scan_job.py). Each job keeps exact counters, at most 200 outcome rows per group and a log capped at 2,000 lines, with root-relative paths and public root labels only. New routes list, inspect, follow (SSE) and cancel jobs and start the scan (POST /prompt_manager/jobs/scan, 409 busy while another record-changing job runs). The legacy /prompt_manager/scan routes attach to the same job and stream unchanged payloads. * feat(activity): concept Activity page, top-bar button and progress pill The workspace gains the concept's Activity page (renderActivity), its top-bar button and sidebar row, fed by one session job store: a /jobs read, one SSE stream per running job, a 1-30 s capped backoff after a dropped stream and a 15 s list poll while visible. A job the server no longer lists shows as ended, so no pill stays stuck after a restart. D52: the counts show running jobs and hide at 0; the phone top-bar row may wrap. D53: the mockup's progress pill links the running job. D54: real job items with state chips, detail lines and Cancel job. Start and finish toasts follow the spec copy. Fake browser jobs come from tests/browser/workspace_activity_fixture.py. * feat(activity): Scan for metadata with its review dialog and job view Images -> Import keeps Rescan and adds the concept's "Review scan" row as "Scan for metadata". It opens the concept's "Review metadata scan" with the real source count, the fixed "All files" scope, a backup box that downloads the existing database backup, and Cancel / Start scan. Start scan starts the session scan job and opens its view; a start refused by another record-changing job toasts its title as a link. #/activity/<id> is now the approved job-view mockup: the heading with Download report (disabled until Activity-2) and Cancel scan, the Progress surface with the five counter tiles, the Results surface with the outcome list, path filter and capped rows (Added and Already known open the viewer through the listing identity), and the collapsed Scan log. Activity item titles link their views. The scan's progress line groups thousands. Deviations D55-D57 are recorded in TESTING.md. * fix(activity): stop the job store on pagehide and let proxied routes outlive teardown CI's touch coverage run failed "a resolver failure for the ComfyUI link never stops the shell from mounting" with "route.fulfill: Fetch response has been disposed": its trace shows two Library thumbnail requests started by the last route still in the spec's proxy-mount handler while the page fixture was torn down. - The proxy-mount handler ignores only that teardown error, and the workspace and Activity specs unroute with ignoreErrors before teardown. - The Activity store now ends its work on pagehide: the poll and backoff timers stop, the job streams close and a list read in flight is aborted (an older read is also aborted when a newer one starts). A pageshow from the back-forward cache reads the list again. * test(activity): compare the job view log's line height to the hundredth of a pixel WebKit reports the mockup's 11px/1.7 line height as 18.700001px, so the job view comparison failed on WebKit only. The measurement now rounds it, as other engine-dependent values are normalized. * fix(activity): final review round for the job store, job view and scan rows - Store: one failed list read no longer marks running jobs as ended; two in a row, or a good list without them, do (M1). A good read resets the backoff, except for a stream that keeps dropping while the list reads fine (M2). Finished jobs the server pruned are dropped (M3). Numbers group as en-US, like the server's lines (M13). - Job view: a job the store's list no longer has shows the not-found state (M3). Focus moves to the job heading when Cancel scan goes away, and to the item's title link in the list (M4). The detail is read only when its listed rows change or the open log grows, using the summary's new listed and log_length (M12). - Viewer rows open by exact identity: each scan row records its root's listing index, the source must still carry the row's label, and the workspace listing gains an exact `path` filter (M8, M9). - Scan rows name the prompt by its Library title, stored title first (M10). - A toast with a link stays 20 s, and any toast stays while hovered or focused (M14). - A real two-batch scan cancelled after batch 1 keeps partial counts (M7). - TESTING.md: D54, D56 and D57 updated (M5, M6). * test(activity): cover the held and linked toast and a job view pruned by the server Unit tests for the toast's 6.5 s and 20 s timeouts and its hover and focus hold, and a browser test for a finished job's open view after the server stops listing it. Coverage floors are only raised. |
||
|
Vito
|
27b4895795 |
fix(resolver): identical sampler branches resolve to their shared prompt pair (#270)
* fix(resolver): identical sampler branches resolve to their shared prompt pair
A LoRA epoch-compare grid runs one sampler per epoch, every one fed by the
same positive encode and the same ConditioningZeroOut negative, and saves
one concatenated image. `_resolve` returned "Multiple sampler branches"
(ambiguous) whenever more than one sampler was found, even though every
branch used the same pair, so the workspace inspector fell back to the
legacy parser and the image could not be paired.
When every branch resolves to identical texts and stored recipes, the
result is now that pair: state resolved, no sampler named (none alone made
the output), reason "Every sampler branch uses the same prompt pair", the
first branch's sides and provenance, and every branch kept in `candidates`
as evidence. A branch that differs, cannot be read, or is not resolved
keeps the result ambiguous. Both profiles share the rule; the default
profile still cannot read ConditioningZeroOut, so the zero-out grid stays
ambiguous for ingestion and keeps the scan's legacy-parser fallback.
Workspace facts: a resolved result without a named sampler over several
listed samplers merges facts per field, as an ambiguous one does (the
grid's shared seed/steps are known, the per-epoch model is ambiguous).
Tests that encoded the old behavior (identical branches => ambiguous):
- test_prompt_pair_resolver: test_multiple_samplers_are_ambiguous_even_when_texts_match
-> test_multiple_samplers_with_matching_texts_resolve_to_the_shared_pair
(plus a new differing-texts case that stays ambiguous);
test_raw_and_detailed_saves_stay_ambiguous_without_an_anchor (same prompt)
-> ..._of_one_prompt_share_it_without_an_anchor (anchored cases unchanged).
- test_pair_scan: two samplers sharing one prompt now ingest the pair
("cat"/"blur") instead of the positive-only fallback.
- test_api_prompt_inspector, test_inspection_outcomes, test_pair_image_linking:
their "ambiguous" fixtures now use samplers with different prompts.
No parity vector changes (f01/f09 branches are not all resolved).
* fix(inspector): never show the positive as a zeroed negative from the legacy fill
When the server leaves the negative undecided, the workspace fills it from
the legacy browser parser (comfy-metadata.js), which follows
ConditioningZeroOut back to its input and so reads the positive text as
the negative. The workspace now checks the prompt graph first:
- every sampler's negative is a ConditioningZeroOut (Reroutes allowed):
the negative shows the server's explicitly empty zeroed state
({status: resolved, text: "", source: zeroed-conditioning});
- a zero-out elsewhere in a negative path (one branch zeroed, another
encoded; a zero-out combined with other conditioning), or a
workflow-only file holding one: the negative is not filled and keeps
the server's status;
- graphs without a zero-out keep the legacy fill unchanged.
comfy-metadata.js is untouched; the legacy pages keep their reading.
* fix(images): explain why Set category has nothing to set
Set category on a selection where no image is linked to a Library prompt
said only "None of the selected images has a linked prompt." It now says
why and what it means: "The selected images aren't in your Library yet,
so there is no prompt category to set." Unit and page-workflow tests
follow the copy.
* fix(inspector): check only the negative the legacy parse read for a zero-out
Review I1/M2/M3/M5 of the zero-out legacy-fill guard. The guard decided
over every node with a `negative` input, so a graph whose read sampler
encodes "blurry" lost that legacy-filled negative when another branch
zeroed its own (fewer than the legacy pages: below the parity floor).
- comfy-metadata.js gains `legacyNegativeSource`, which returns the graph
and sampler `negative` link that `extractGenerationParams` reads from.
It reuses the parser's own sampler pick (`selectSampler`, factored out
of `fromPromptGraph` without changing any reading). The workspace
checks only that link: a direct zero-out shows the zeroed empty
negative, a zero-out further up the path leaves it unfilled, and
anything else keeps the legacy fill.
- Workflow-only files go through the parser's own workflow conversion,
so bypassed (mode 4) and muted (mode 2) zero-outs and unconnected ones
behave as the legacy reading does. A subgraph instance whose definition
runs a zero-out (directly or nested) leaves the negative unfilled.
- One bounded walk with an index pointer replaces a walk per negative
input with queue.shift().
- An ambiguous fact that the legacy parse knows (a grid's per-branch
model) is still filled, documented: the legacy pages show it and this
page never shows less.
Tests: the reversed mixed graph shows "blurry"; the mixed graph whose
read sampler is zeroed, the zero-out-only grid and a direct workflow
zero-out show the zeroed empty negative; a combined zero-out and a
zero-out subgraph instance stay unfilled; bypassed and unconnected
workflow zero-outs keep the legacy reading. An API-graph Reroute in
front of a zero-out gives the legacy parse no negative, so the server
answer stays (that expectation changed from zeroed).
* fix(images): name the missing link in the Set category toast
Review M4: say that no selected image is linked to a saved prompt, in
the same terms as the partial-success toast and right for one image.
Also review M1: a monitor test that two parallel samplers sharing one
positive and one negative write that pair (the resolver collapse).
|
||
|
Vito
|
36564b4767 |
feat(parse): one generation-parameter parser in Python; scan links what the resolver cannot decide (#253)
* feat(parse): port the browser generation-parameter parser to Python Adds utils/generation_params.py, a line-for-line port of extractGenerationParams in web/js/comfy-metadata.js, the parser behind the gallery, admin and metadata pages. The scan and live monitor can use it so every surface reads prompts the same way. The JavaScript parser is unchanged. The port reproduces the JavaScript value semantics the result depends on (property order, truthiness, String(), trim(), strict equality, Map keys, JSON.parse number rounding, and the inputs on which it throws), and runs its recursive tracing on an explicit stack so crafted graphs cannot exhaust the Python stack. Tests: - behaviour tests for each tracing rule, with JavaScript results as expected values, including the visit-limit boundary; - golden vectors of the JavaScript parser's results for every named case and every PNG fixture, so the comparison runs without Node; - a live comparison against the JavaScript parser when Node is present; - a randomized differential test over 9,000 generated graphs and workflows, coherent and corrupted, including inputs where the JavaScript throws. Five deliberate porting mistakes (trim, truthiness, prepend handling, visit limit, pair slot) each fail the tests. On a 38,602-file output folder, all 18,491 files with a workflow give identical results. * fix(scan): link the traced prompt when the pair resolver cannot decide The output scan skipped every file the pair resolver called ambiguous or unresolved, so workflows where several samplers share one prompt (a base pass, an upscale pass and a detailer) never reached the Library from a scan. It now links the positive prompt the shared parser traces for those files, the prompt the inspector and the legacy pages show. Files the resolver decides are stored as before, and graphs without a recognized sampler keep the legacy extractor (#252). Read-only scans of a 38,602-file output folder into temporary databases: - links: 1,500 before, 7,954 after; none of the 1,500 dropped or changed; - of the added links that differ from the pre-redesign scan, 1,275 replace a number, a LoRA tag, a fragment or a truncated prompt that scan stored. * refactor(parse): return a uniform pair from the SameValueZero key |
||
|
Vito
|
caa9649f32 |
feat: reuse positive and negative prompt combinations (#195)
* feat(db): add pair schema and stage restore migrations for #193 * feat(database): persist reusable ordered prompt pairs (#193) * feat(api): expose reusable pairs and atomic metadata enrichment (#193) * feat(metadata): resolve canonical prompt pairs with bounded provenance (#193) * feat(scan): enrich image pairs without guessing or duplicate links (#193) * fix(database): preserve pair tags during global maintenance (#193) * feat(tracking): enrich live image pairs and cancel scans safely (#193) * feat(nodes): encode paired prompts and count cached recipe runs (#193) * fix(database): validate restored pair ownership and provenance (#193) * feat(admin): report pair scan outcomes and cancellation (#193) * feat(ui): add paired prompt editor and explicit recipe library for #193 * fix(autocomplete): include reusable pair metadata vocabulary (#193) * fix(pairs): honor recent sorting and conservative scan outcomes (#193) * fix(ui): protect paired draft saves and clarify negative variants (#193) * fix(metadata): preserve bounded paired execution evidence (#193) * fix(tracking): follow actual pair output roles and prefer execution evidence (#193) * docs(pairs): document live validation and retain renderer screenshots (#193) * test(api): inject disposable databases before API startup (#194) * test(api): canonicalize portable temporary gallery roots (#194) * docs(ui): prototype branching negative variant history (#193) * docs(ui): show selected negative beneath positive in timeline concept (#193) * docs: retain approved timeline for later gallery redesign (#193) * test(db): close restore fixture connections before cleanup (#194) * test(db): reuse existing handle guards in all pair fixtures (#194) * chore: clarify rollback test and intentional exception handling (#193) |