1881d8f23ce02eaf3fdc2e3a0206eadaa5b0470c
77
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito Sansevero
|
1881d8f23c | Merge remote-tracking branch 'origin/feat/249-needs-review-backend' into feat/249-needs-review-ui | ||
|
Vito Sansevero
|
18d81c9d96 |
fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
|
||
|
Vito Sansevero
|
dac7ac15ba | Merge remote-tracking branch 'origin/feat/249-needs-review-backend' into feat/249-needs-review-ui | ||
|
Vito Sansevero
|
9e125ee4a3 | Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend | ||
|
Vito
|
cbc53ee9f1 |
feat(resolver): trace multi-stage chains through pass-through nodes (#308)
* feat(resolver): trace multi-stage chains through pass-through nodes The inspection resolver (server and local Inspect) now compares a sampler with the earlier sampler its latent continues through image-space pass-through nodes: decode, image filter, model upscale, scale and encode, listed in a new validated `lineage` section of the semantics table. Equal prompts resolve (hires-fix, refiner); different prompts are multistage-differs. A later stage whose positive adds conditioning to the shared base prompt through an additive composite (ConditioningCombine or Concat) keeps the base pair, the text legacy linked (three-stage Flux). Image-edit batches stay ambiguous and run-time text stays untraceable. Ingestion keeps the default profile (D1). A card no longer shows a seed when the inspector names a chain stage whose seed differs from the parser's pick. * feat(resolver): leave out run-time PromptManager parts, labelled Maintainer option A: when a PromptManager prepend_text or append_text is fed directly by a node whose output is decided at run time (LoraManager's TriggerWord Toggle, now a runtime class, or PromptSearchList), the prompt resolves to the node's own text, as legacy showed. The side's source is runtime-parts-omitted and the resolved reason says that text added at run time, such as trigger words, is not included; the prompt block shows that line (D-NEW-R1). The toggle's output is never computed. Literal and traced parts are combined as before; an indirect run-time part, a run-time main text or an unknown feeder keep their outcome. The browser port changes identically. Ingestion is unchanged. * fix(resolver): keep run-time and added-conditioning labels honest A text that left out a run-time PromptManager part is now labelled when any node on its traced path omitted it, so the label survives encoders, concatenations and chained PromptManagers. Comparisons include what a text leaves out, so a partial prompt never equals a complete one: a branch with trigger words and one without stay ambiguous, and such stages are multistage-differs. Only classes listed as run time are left out. The added-conditioning floor labels its positive (D126, "Conditioning added at a later stage is not included."); the run-time line is D125. A card hides its seed only when the stage the inspector names reads a different seed than the parser; an agreeing seed is kept, as on main. The browser port moves its lineage, latent and part helpers into local-resolver-stages.mjs, keeping parity. |
||
|
Vito Sansevero
|
828b5f9db3 |
Merge remote-tracking branch 'origin/feat/249-needs-review-backend' into feat/249-needs-review-ui
# Conflicts: # web/js/workspace/images.mjs |
||
|
Vito Sansevero
|
95f88d3966 |
fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
|
||
|
Vito
|
7c25b351f7 |
test(comfyui): refuse a real-host port another run holds (#310)
The real-host config waited on the /system_stats URL, which any server on
the port answers. When two runs started on 8189 within a few seconds of
each other, the second run's ComfyUI failed to bind ("Port 8189 is already
in use"), but Playwright had already seen the first run's host answer and
ran the second run's tests against it. When the first run finished and
stopped its host, the second run failed mid-test with ECONNREFUSED,
ECONNRESET or ERR_CONNECTION_REFUSED, and passed when rerun alone.
The launcher now refuses a port that is in use, and reports ready only
when /system_stats names its own base directory. The config waits for
that line instead of the URL, and every test checks again that the host
answering the port is the one its run started.
|
||
|
Vito Sansevero
|
9bfb5992b3 |
Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts: # py/api/system.py # tests/test_system_api.py |
||
|
Vito
|
785b744554 |
feat(system): Appearance section and browser display defaults (#307)
* feat(system): share browser display defaults The workspace reads and writes the legacy gallery's gallerySettings object through preferences.mjs: workspaceLayout and workspaceDensity join the shared keys, a write re-reads the stored object and writes only the changed keys, denied storage or a full quota keeps the change for this tab and says so, and storage events report changes from other pages. The legacy gallery Save keeps keys it does not edit, and the viewer reads its video defaults through the shared module. * feat(system): Appearance defaults used by Images and the viewer System > Appearance is the concept's Browsing & display surface on real values: the browser rows edit this browser's display defaults through the shared gallerySettings object, and Open from ComfyUI and the node search auto-hide save to the server settings with their revision. A new Images visit starts from the default layout and size (the current view and saved views keep theirs); metadata off waits for Load metadata; cache off re-reads the shown file; paths off hides the Path row; the viewer reads the video defaults. D97 adds card details, collapsed viewer details and Reset display defaults; storage refused says Saved for this tab only. * fix(system): flag changes made while System was closed and keep embedded galleries fixed Final review fixes for System S3 (Appearance): - I1: the on-demand metadata deviation is D124 (TESTING.md, inspector markers); TESTING.md names the spec that pins it, since no concept comparison renders that state. - I2: System checks the display defaults for conflicts on mount, not only on storage events, and a browser write refuses a draft another page overtook, so a draft is never saved over another tab's value without the conflict status first. New two-tab spec. - I3: the layout and size defaults start the Images page only; embedded galleries (a prompt history's Shared gallery and dialogs) keep Grid/Comfortable. - M1: host rows whose /settings read failed say "Server settings could not be read." - M2: a rejected timeout shows the "Some changes were not saved" status. - M3: the browser loads on mount and on storage events never reject unhandled. - M4: the next accepted write stores the tab-only changes and clears the overlay. - M5: cache off never re-reads a source-changed selection, and refresh() never loads a deferred one. - M6: the System fixture refuses a config.json outside its temp root and restores it with os.replace. - M7: the inspection read counts wait on a request barrier, not frames. - M8 recorded as an open saved-views question in TESTING.md. * fix(system): answer a failed System read only after its other reads finish CI (py3.10 on ubuntu) crashed with a segfault inside SQLite in test_statistics_failure_is_an_error_envelope: the statistics and overview routes used asyncio.gather without return_exceptions, so the 500 answered on the first failure while get_statistics still ran in the executor, and the test tore the database down under it. Both routes now await every read before raising the first failure (failing before, passing after: TestFailedReadsSettleFirst). The frontend coverage merge failed on workspace/activity.mjs branches (86.3 < 87): no test asserted Activity's empty and failed-read states, so their branch was only reached in passing, depending on timing. A spec now asserts both. |
||
|
Vito Sansevero
|
49b5552340 |
feat(review): open Needs review from the System Overview review row
Review metadata on the Overview's review row now opens #/review, the concept's go-review, instead of staying aria-disabled. |
||
|
Vito Sansevero
|
59cdb00f67 | Merge remote-tracking branch 'origin/feat/249-needs-review-backend' into feat/249-needs-review-ui | ||
|
Vito Sansevero
|
765abc6f3e |
Merge remote-tracking branch 'origin/feat/249-needs-review-backend' into feat/249-needs-review-ui
# Conflicts: # TESTING.md # py/api/__init__.py # tests/js/workspace-state.test.js # web/js/workspace/index.mjs # web/js/workspace/state.mjs |
||
|
Vito Sansevero
|
369f38be9a |
fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the concept's line, from needs_review.pending. Its Review metadata action stays aria-disabled until the Needs review page ships. The system specs and their concept comparison follow the real count. |
||
|
Vito Sansevero
|
878612c22d |
Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts: # database/pair_ingestion.py # py/api/__init__.py |
||
|
Vito
|
c86739178d |
feat(system): System page shell, review dialog and Overview (#298)
* feat(system): read-only health check and truthful overview counts POST /prompt_manager/system/health runs the health check off-loop as a read registry job of kind "health" (one at a time, beside a scan, not cancellable, no file outcomes) and keeps the last report; GET returns it or null. The check reads one snapshot on its own mode=ro connection, closed in finally: integrity_check(100), foreign_key_check, stored hashes against utils.hashing, duplicate pair rows, the legacy consistency queries (now shared with check_consistency), and image links whose file is gone, counted only under readable sources. An unavailable source, or the absent Needs-review queue, makes its check unavailable and the report incomplete instead of producing a count. GET /prompt_manager/system/overview returns Library-eligible positive prompts and active negative variants, indexed images, the last health summary or null, backup null, the Needs-review count null, the effective sources with whether they were auto-detected, and the thumbnail plan's missing count. GET /prompt_manager/system/statistics lists runs and average rating per category of Library parents, with the /stats totals. * feat(system): native System route, drafts and shared review dialog - #/system/<section> renders the concept's System shell: section nav, settings layout and Save row with truthful status (D92); each section not delivered yet hands off to its legacy tools (D94). Overview's System card opens #/system/overview (Q12). - A module-level draft store saves browser, then host (expected_revision), then runtime; failed domains stay dirty, a stale revision is a conflict. - System row builders (setting, operation, section nav) on the scope. - openReviewDialog and backupNotice extracted from the scan review, which is now a thin caller with identical markup. - Overview images gain a "rated" count (images whose linked prompt is rated); the missing-files health check reports per source, so partial availability is visible. * feat(system): native Overview with health report and statistics System > Overview renders the concept's metrics and its Library health and Workspace preferences surfaces from GET /system/overview: real counts, "Not checked" until a health check ran, no invented review count, backup or thumbnail status (D93). Explore opens "Usage statistics" with the real runs and ratings per category and the Library totals (D96); the statistics route now also returns rated_images. Maintenance shows the concept's notice with Run health checks above its hand-off: the read-only check runs as a registry read job and opens the "Health report" with a chip per real result, missing files per source (D95). A load failure shows Retry and no zeros; late answers never repaint a newer section. Adds the System concept comparison (shell, Overview, both dialogs, the notice, mutation checks), the System workflow spec and its fixture hooks, and the TESTING.md "Native System" section. * test(system): coverage floors for the native Overview Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes. * fix(system): keep health reports per database and off the event loop - A restore that replaces the database forgets the stored health report, and a check that started before the swap is not kept (database generation stamped at start). - The Overview no longer walks every source for the missing-thumbnail count; the page reads /images/thumbnails/status on its own. - Image sources for the health check and the Overview resolve in the executor; a failure to start the check answers the coded envelope. - Any exception fails only its own check; missing files reads its rows in the snapshot and stats the files after it ended. - The read-only URI keeps Windows UNC paths (empty authority). - An integrity count at SQLite's limit is flagged as capped. * fix(system): thumbnail row read, tracked health job and guarded host save - The Overview's thumbnail row is its own read of the thumbnail status: "Checking thumbnails…" until it answers, unavailable when it fails. - The health job is tracked in the Activity store (new store.track) and followed through its event stream instead of polling every 150 ms. - A host save before a read returned a revision fails without a request; System reads the host values on mount and after a conflict. - "100+ problems found" when the integrity count is capped. - D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone. - The concept spec builds the health report from the concept's own rows and asserts the full unmapped path list; surface tops pin the 18px offset. - TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI and the new behavior. Coverage floors raised for the touched files. * fix(system): make two health tests portable to Windows - Windows forbids "?" in file names, so the URI-character open test uses "#", "%" and a space there. - ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of raising, so the bad-path test raises as POSIX does to reach the missing-files failure path on every platform. |
||
|
Vito
|
b50bf862f0 |
fix(inspect): hold a route change until a starting video settles (#294)
* fix(inspect): keep a replaced video in place until its pipeline has started WebKit (GStreamer) deadlocks the page when a video leaves the page or the viewport before its pipeline has started: the main thread changes the pipeline state and waits for the source's streaming thread, which waits on the main thread in a synchronous bus handler (native stacks of the hung WPE web process in the pinned image). Removing, hiding or merely pushing the video below the fold all hung; none of pause(), removing src, load() or the revoke was the cause on its own. A video replaced while still starting now stays where it is, inert and aria-hidden, with the new preview inserted after it, and the session's revoke of its URL waits. Once it reports metadata or an error (or after 15 s) it is paused, loses its src, is reset with load() and removed, and only then is its URL revoked. Every released video is reset with load(), so no decoder is kept per replaced video. The loadstart replacement spec now runs on WebKit too (15 fresh pages, replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause. * fix(inspect): hold a route change until a starting video settles A route change within the workspace detaches the Inspect page, and WebKit (GStreamer) deadlocks when a video still starting is detached. Route changes at loadstart + 0-7 ms hung 4-5 of 120 fresh pages (14 of 120 with no page code releasing anything, so the detach itself is the trigger). Inspect now exposes leaveGuard(): a promise while a video on the page is still starting (current or replaced). The shell renders the next route once it settles (metadata, error, or the 15 s bound); the latest route wins. Route changes at loadstart + 0-7 ms now hang 0 of 120. Not changed, with numbers in TESTING.md D62: - A replaced video still shows above the new preview until it settles. Same-slot stacking changes the video's own box and hung again (absolute 5/120, absolute + opacity 0 8/120, one-cell grid 4/120). - A real page leave cannot wait. Navigating away hung 8/120 and reload 22/120 even with no page-side release, so it stays a known limitation. |
||
|
Vito Sansevero
|
3e613adba9 |
Merge remote-tracking branch 'origin/feat/249-needs-review-backend' into feat/249-needs-review-ui
# Conflicts: # tests/browser/shell-concept.spec.mjs # tests/browser/workspace_activity_fixture.py # tools/vendor/frontend-coverage-baseline.json # web/js/workspace/activity-job-view.mjs # web/js/workspace/activity.mjs # web/js/workspace/workspace.css |
||
|
Vito Sansevero
|
fcf6445de0 |
fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The scan already records Needs review items under its own records job. The monitor's recording lasts one file and is not a job, so it now checks the records admission: while a records or guarded job or hold runs it records nothing, logs that the file waits for the next scan, and answers deferred. The fallback link stands and the next "new" scan reads the ambiguous file again, because a fallback link is not a resolved one. JobRegistry.blocker and active iterate a snapshot of the jobs, since the monitor's worker thread now asks them too. |
||
|
Vito Sansevero
|
0f3559bedf |
Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend
# Conflicts: # py/api/__init__.py # tests/test_api_core.py |
||
|
Vito
|
4f1c6e236b |
test(browser): report a slow test reset with every thread's stack (#299)
* test(browser): report a slow test reset with every thread's stack On #291 (run 39904288262, WebKit shard 2/2) one /__test__/reset {chronological_history} waited 16 s on its worker's server, with nothing on the server's stderr. The cause did not reproduce: the same worker's file sequence (library, library editor, tagging lifecycle, visual) at 2 CPUs three times, and WebKit shard 2/2 at 2 CPUs, kept every reset under 2 s. A reset still running after 2 s now reports, while it waits, every thread's stack (faulthandler), the default executor's threads and queue, the open connections and WAL size, and on Linux each thread's scheduler state and kernel wait channel. When it ends it reports the total and how long each executor call queued and ran. The lines carry a "pm-slow-reset| " prefix and the worker fixture copies them to the job log, so the next one shows up even in a test that passes. The report never reads another thread's connection. Reading even in_transaction on a connection busy in SQLite blocks on its mutex with the GIL held: a first version of this report froze the whole server until the 30 s busy timeout. A test holds a real write lock behind a reset and checks the server keeps answering while it waits. * test(browser): name each thread in the slow-reset stacks Before Python 3.14 faulthandler prints thread ids only, so the report now lists which name each id belongs to. CI on 3.10 to 3.13 failed the unit test that looks for the blocked thread by name. * test(browser): keep the slow-reset summary on a coarse clock On Windows the report timer can fire a few milliseconds short of the threshold, and the summary was then dropped. A reset that was reported now always gets its summary. The server test uses the plain reset, which the Windows jobs can seed, and puts the server stderr in its failure messages. |
||
|
Vito
|
b2cc489fd4 |
feat(inspect): copy settings summary and local workflow roles (#286)
* feat(inspect): copy settings summary in the full inspector The legacy metadata page's COPY ALL (LI-4) becomes "Copy settings summary" after the six facts of the shared full inspector, so the Images viewer, the recorded history's current file and Inspect all offer it; the Images side panel keeps the concept's footer. The text is the legacy copyAllData template with its placeholders, filled from what the inspector shows. A unit test runs the template read from web/metadata.html on the legacy parse of every parity PNG: 22 of 25 are byte-equal, and the other three differ only by the ZeroOut guard (#249) or facts the legacy parse missed. Copying follows the concept's prompt copy: the toast "Settings summary copied.", or the select-to-copy dialog (now titled per copy) without a clipboard. The concept comparisons map the button as D-NEW-1. * feat(inspect): local execution evidence and prompt tracing A browser port of the server's inspection resolver for one sampler (PR L3 Task 8, LI-1). local-resolver.mjs answers inspect_prompt_evidence's {prompts, resolution} for a local read's records: execution evidence (read_pair_execution and its per-node match), every traced strategy (encoders, strings, concatenations, PromptManager and its historical classes, passthrough, zeroed, selector branches, composites, guiders), the latent stage walk, the R1 overwritten-text guard and the inspection profile's visit, depth, text, path, built and scanned bounds, with the same reason codes and outcome lines. py-json.mjs reads the records as CPython's json.loads does (dict order and last-wins keys, int versus float, BigInt past 2^53, NaN) and measures and strips text as str does. Several sampler candidates, detailer stages and A1111 parameters raise LocalResolverGap until Task 9. Nothing is wired into Inspect yet. Python is the oracle: test_local_inspect_parity compares the port with the server on the parity corpus, 107 named record sets, every work bound at its edge and 600 seeded fuzzed graphs, and requires gaps to be exactly the Task 9 paths. Sixteen porting mutations each fail the suite. The Node unit test replays committed server answers, which the Python suite keeps current. * fix(inspect): a list or object workflow node type answers malformed The R1 overwritten-text check looked every UI workflow node's `type` up among the subgraph IDs. A `type` that is a JSON list or object raised TypeError there, so the inspect route answered 500 "Prompt inspection failed". Such a workflow is now malformed evidence: the resolution is unresolved with reason_code `malformed` and the inspector's own "Prompt evidence is malformed" line, never exception text. The browser port answers the same, and the parity corpus gains the list, object, nested-subgraph and skipped-ID shapes. * fix(inspect): a list or object graph class_type is never a preview output With several sampler candidates the inspector first drops stages whose outputs reach only previews. That pass looked each API graph node's `class_type` up in the preview class set, so a list or object `class_type` raised TypeError and the inspect route answered 500. Only a string names a class now; such a node is an ordinary terminal, as the rest of the resolver already treats it. * feat(inspect): local candidate selection matching the inspector The browser port of the inspection resolver now answers every record set the server does, so no path is left to the server any more: - several sampler candidates, with the identical-branch rule (#270): a compare grid whose branches all resolve to one pair (text and recipe) is resolved, any other mix is ambiguous; - stages whose outputs reach only previews are dropped first; - detailer stages are judged with the samplers in their image ancestry (multistage-differs, cycles, depth and the per-stage visit budgets), and a sampler that only feeds detailers is not a candidate of its own; - A1111 `parameters`, with Python's multiline anchors, `\s` and Unicode `\d`, the bare footer and Template trailers, counted in code points. The parity corpus gains the selection cases and a seeded fuzzer, and every inspection vector now gives the server's prompts, resolution and facts locally. 16 of 17 mutations are caught; the uncaught one (not flattening an ambiguous upstream stage) cannot change any answer. * feat(inspect): local prompts resolved like the server's A file chosen on the Inspect page now shows the prompts and resolution the server's inspector gives the same records: buildLocalPayload takes its roles from the local resolver instead of the interim open roles, so the legacy fill applies exactly where it does for Images. The resolver reads the server's own node table. The page fetches web/js/inspection-semantics.json once through the API client as it mounts, before any selection; the static route serves that one JSON file by exact path (every other JSON stays forbidden). A table that cannot be fetched answers "The inspector's node table is unavailable." as the server does, and a later selection asks again. The Copy settings summary deviation is numbered D117. * fix(inspect): skip a workflow node whose type is a list or object The overwritten-PromptManager walk now skips such a node, on the server and in the browser port, instead of marking the whole file malformed. A prompt that resolves still resolves; the other nodes still feed the check. * fix(inspect): one JSON depth cap for both browser parsers py-json.mjs and local-video.mjs now refuse past the same 1,000 nested arrays or objects, counting every container as CPython's scanner does. Boundary vectors at 999, 1,000 and 1,001 for a deep prompt and a deep workflow run through the Python oracle, and TESTING.md gives the measured per-version server limits and both divergence directions. * fix(inspect): realpath containment for the exact-path node table route inspection-semantics.json now goes through the same realpath check as every other static file, so a symlink out of web/ is refused. The forbidden-path test asserts exactly 403. * fix(inspect): abortable, stepped local reads with a busy state The wait for the node table now ends with the selection's signal. A local read resolves and projects in bounded steps with a turn of the event loop between them, and the inspector carries aria-busy while it loads. A budget test covers maximal 2 MiB graphs and checks the stepped read is byte-identical to one synchronous build. The D117 wrap rule now targets only the summary button, and TESTING.md scopes the restart line to PRs L1 and L2. * fix(inspect): record the two CPython-JSON emulations as a follow-up |
||
|
Vito
|
fd747a0ffc |
fix(inspect): keep a replaced video in place until its pipeline has started (#291)
WebKit (GStreamer) deadlocks the page when a video leaves the page or the viewport before its pipeline has started: the main thread changes the pipeline state and waits for the source's streaming thread, which waits on the main thread in a synchronous bus handler (native stacks of the hung WPE web process in the pinned image). Removing, hiding or merely pushing the video below the fold all hung; none of pause(), removing src, load() or the revoke was the cause on its own. A video replaced while still starting now stays where it is, inert and aria-hidden, with the new preview inserted after it, and the session's revoke of its URL waits. Once it reports metadata or an error (or after 15 s) it is paused, loses its src, is reset with load() and removed, and only then is its URL revoked. Every released video is reset with load(), so no decoder is kept per replaced video. The loadstart replacement spec now runs on WebKit too (15 fresh pages, replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause. |
||
|
Vito Sansevero
|
23887553bf | docs(review): number the live-updates-paused deviation D120 | ||
|
Vito Sansevero
|
807cb32f58 |
fix(review): keep the live queue right through applies, Load more and lost streams
- A change heard while a decision is in flight is owed and read once the decision is refused or unanswered (I1). - Load more has its own read: during a re-read it waits and continues from the re-read queue; a re-read started during Load more takes the click over, and a failed re-read leaves Load more clickable (I2, M7). - A stream the browser gave up on (503, proxy error) reopens after 3 s, doubling to 30 s, with a quiet "Live updates are paused" line while it is down (I3, D-NEW-F2). - A hidden tab closes its stream and a visible one reopens it; the ready version keeps that to at most one read (I4). - Apply fences on the detail's epoch and a re-read under another epoch reads the open item again (M3); leaving during the observe releases its capture (M4); an unanswered decision that landed finishes as recorded (M5); cards follow a re-read's url, target and availability and show thumbnails (M6, M10). - The unreadable-ready test asserts exactly one re-read (M1); the panel's long-file-name wrap carries its D119 marker and TESTING.md entry (M8). |
||
|
Vito Sansevero
|
6fd022cc70 |
test(review): compare Needs review with the approved concept
review-concept.spec.mjs measures the whole Metadata review page as a tree in the concept and in production, on the review fixture's new concept-shaped queue (the concept's first three image names and sizes and its first prompt). mapConceptReview applies only D29, D119, D3, production's hooks and accessible names and sample data, and the page then matches at 1600x1100 and 390x844 in both themes. The gallery layout follows the concept at every breakpoint, the SAVED VIEWS rows match with Needs review after Favorites, the queue's Tag image dialog matches the needs-review reference, and a dropped notice border or reordered actions fail. The comparison found one gap: an unavailable file's card lost its media area. Its empty media button now keeps the concept's 1.33 ratio (D119). TESTING.md records the comparison, the extended D119 and a summary of the image-tagging slice (routes, S1, capture limits, the record-changing job, the no-download rule, the restart note). A ready whose version cannot be read re-reads the queue (covered by a page test). |
||
|
Vito Sansevero
|
9b9a07ccef |
feat(review): keep the loaded depth on live re-reads and read once per visit
A queue change re-read only the first 40 items, so the pages Load more had shown vanished and the page jumped. Each re-read now reads page after page until it holds as many items as were shown (a restore met after the first page reads again from the start), so the cards and the scroll position stay. The stream's ready made every visit read the queue twice. List reads now answer the queue's version (library_epoch, review_revision) and ready names the version read after subscribing; the page re-reads on ready only when the queue it shows is older. The SAVED VIEWS count follows the page's total, so a change costs no count read. A stream counts as open only once ready is sent. |
||
|
Vito Sansevero
|
52c6efcbe4 |
feat(review): re-read an open Needs review page when its queue changes
An open Needs review page now follows the queue without polling. A new server-sent event stream, GET /prompt_manager/review/events, sends "ready" when it opens and "change" after every committed queue change: an item the scan or the monitor recorded, a spelling refresh, a decision from any tab, and a restore. database/review_changes.py notifies after the commit from whichever thread wrote it. The page re-reads its first page on each event, keeps the open item, its choice and an unanswered decision, and re-reads the SAVED VIEWS count with it. When the open item left the queue, the next one opens with a note. A keepalive comment every 25 s lets the server drop a closed tab; shutdown ends every stream. Leaving the page mid-decision now releases that decision's capture. D-NEW-F1 is renumbered D119 in markers and TESTING.md. |
||
|
Vito Sansevero
|
9a7bdb2f09 |
feat(review): Needs review page, SAVED VIEWS row and Activity hook
The concept's Metadata review page at #/review: the notice with the exact pending count, the queue's media cards (shared with Images through image-card.mjs), and the Resolve the association panel with the sampler context, the chosen path's prompt blocks, Leave for later and Apply association. Apply takes a fresh capture; refusals keep the selection and explain inline; an unanswered decision is resent once. SAVED VIEWS lists Needs review after Favorites with its count (retires D50), and an ended scan that recorded review items offers Review metadata in Activity (D87). Queue items carry the card's media url. |
||
|
Vito
|
a8aff3a39d |
feat(annotations): bulk Add tags and Auto-tag selected as Activity jobs (#283)
* feat(activity): image tagging jobs for bulk Add tags and Auto-tag
POST /prompt_manager/jobs/image-tags starts one record-changing
"image-tags" job over 1..500 listed images (202, or the same 409 busy
envelope as /jobs/scan). Add tags hashes each current file and unions a
fixed set into that byte record; Auto-tag captures each still image, asks
one legacy engine for its top N suggestions on the capture (count from
the options or autotag.defaults.suggestion_limit), re-hashes the file and
unions only onto the same bytes. Outcomes are grouped as tagged,
unchanged, skipped (video, too large, file changed, file not found, not a
still image) and errors; cancel stops at a file boundary.
Auto-tag holds the shared model gate for the whole job (409 model-busy at
start when it is held), loads the model once and never downloads one: an
unavailable model fails the job before the first file. The registry gains
blocker() so a start answers busy before taking the gate.
* feat(annotations): Add tags and Auto-tag selected in the selection bar
The Images selection bar gains the concept's Add tags and, beside it,
Auto-tag selected (D85). Add tags starts the image-tags Activity job with
the dialog's comma-separated names; Auto-tag selected reuses the Tag image
model panel (server defaults, Suggestions count) without Generate, refuses
a model that is not downloaded, and either starts the job (Apply
automatically) or walks the selection one Tag image dialog at a time
(Review each image, D90). Busy, model-busy and refused starts toast why;
the selection stays; a finished job's files are read again.
* feat(activity): show image tagging jobs with their own outcomes
Image tagging jobs get their own Activity view (D89): the pill reads
"Tagging", the item and heading line name the images and end with
"N tagged · M errors", and the job view shows Tagged, Unchanged, Skipped
and Errors tiles and outcomes, opens Tagged and Unchanged rows in the
viewer, and reads "Tagging log" and "Cancel tagging". The per-kind table
lives in activity-store.mjs; the scan's view is unchanged.
The selection bar's copy without a concept line is recorded as D117.
* fix(annotations): final review round for bulk image tagging
- Rename the D117 copy deviation to D118 (D117 belongs to another PR).
- The tagging job's progress line gives the scan's "about X min left"
estimate, with each finished file as one batch.
- The start body cap fits 500 targets of the longest legal path, even
escaped; one byte over answers 413.
- A capture store at capacity is retried with a short backoff, then the
file is skipped with a rerun sentence instead of a Tag-dialog error.
- Output roots and their labels are resolved once per job in the
executor, not twice per file on the event loop.
- Review each image holds Stop reviewing, Escape and x while a save is in
flight, and its primary reads "Save and next" when only the manual
field would be saved.
- An Add tags field that parses to no names keeps the dialog and its
text and reports the message on the field.
- Image tagging finish toasts carry the counts ("... finished · N tagged
· M errors").
- Specs: the bulk spec waits for the answered Start before checking that
the invalid press sent nothing; the Activity spec uses a real error row.
|
||
|
Vito Sansevero
|
f00b0232a0 |
feat(review): bind queue cursors to the storage epoch only
A decision or a new item (the monitor records one per generation) no longer makes an open cursor stale: the (created_at, id) keyset stays ordered when items leave or are appended. A restore still answers storage-changed and an unverifiable cursor review-conflict. |
||
|
Vito Sansevero
|
7f1b5b64bd |
fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted or moved file leaves its pending item counted), the physical identity rule for Apply, the candidate-only recheck and shared resolver version, the spelling refresh, background capture admission, the final shutdown close, the stale-cursor answer and the measured startup validation cost. |
||
|
Vito Sansevero
|
3e20c533fa |
feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
|
||
|
Vito Sansevero
|
d25c2cb4f2 | Merge remote-tracking branch 'origin/main' into feat/249-needs-review-backend | ||
|
Vito
|
e29ff4d4b2 |
feat(inspect): the Inspect page for local files (#280)
* feat(inspect): local adapter seams in the shared inspector and viewer
The inspection loader calls adapter.read when an adapter reads by itself, so a
file chosen in this browser never reaches the API. The full inspector shows the
D66 local provenance (no Path row, no Show in folder, local association line),
and the viewer uses adapter.mediaUrl exactly, so a blob: URL is never given the
_pm_viewer cache-buster and Retry media reuses it. Rooted and recorded-history
adapters keep their api.get paths and cache-busted originals.
* feat(inspect): owned local selection session
createLocalSession owns one chosen File at a time: it classifies the bytes,
creates the blob: URL only for a classified image or video, and reads it
through a real inspection controller with the local adapter. Replacement,
dispose, parent abort and pagehide each release the URL, the handle and the
record once; a released handle (and every copy) is foreign to the adapter, and
every await is fenced by a generation counter so a replaced read never
publishes. Empty, multiple and directory drops keep the previous selection.
Baseline entry for local-session.mjs; branch floors raised for the touched
inspection-state.mjs and local-inspection.mjs.
* feat(inspect): native Inspect a file page
#/inspect is now the concept's "Inspect a file" page instead of the legacy
handoff. One chosen or dropped image or video previews inline, and the one
heading action, "Open file metadata", opens the shared viewer with the full
inspector on a record read entirely in this browser. Nothing is uploaded,
imported or stored; only an explicit Send crosses the host bridge.
- inspect.mjs mounts the concept DOM (surface, file-drop, #local-preview)
under the shell heading, owns one local session and the shared viewer and
prompt sender, and releases media before the session revokes its URL on
replacement, route change, page leave and dispose (D60-D65).
- A file that cannot be read after choosing opens the inspector's
unreadable state when the browser declares a supported type, instead of
the "unsupported" copy; the local adapter maps a browser read failure to
an unreadable record.
- The route is native; Overview keeps its card to the legacy metadata page,
which is unchanged.
- workspace-inspect.spec.mjs covers the chooser and viewer rail against the
Images rail for the same bytes, drop, cancel and reselect, unsupported and
unreadable files, video, URL ownership, mid-read replacement, exact send
and the no-request/no-storage rule.
* test(inspect): concept parity and managed-host send for local files
Compare the native Inspect a file page with the immutable concept's
#metadata at 1600x1100 and 390x844 in both themes (initial, selected,
viewer-open), mapping D60-D66 and pinning the concept's CSS values and
the chooser's displayed file name, with a mutation check. Send a local
PNG (explicitly empty negative) and core-tags.webm to a real Prompt
Manager (+/-) node across Classic and Nodes 2.0 switches and reloads;
a connected input rejects the send unchanged. Document D60-D66, the
legacy-floor dispositions and the commands in TESTING.md.
* fix(inspect): reset a released preview video in the next task
On WebKit (GStreamer) a video replaced while its pipeline was still
starting blocked the page: pause, src removal and load() in one task
never returned, so the next chooser selection hung (CI page workflows
webkit shard 2/2, "URLs and media end with their owning session").
The src is still dropped before the URL is revoked; only the load()
reset moves to the next task. A fresh-page regression replaces the
first video as soon as it shows, fifteen times.
* fix(inspect): never reset a preview video that is still starting
Deferring load() to the next task was not enough: CI on
|
||
|
Vito Sansevero
|
f3366a13f8 |
feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion resolver finds ambiguous for Needs review. The file is captured, its metadata is read again from the capture and resolved with the default profile, and only an ambiguous result is stored, bound to the capture's annotation byte record and fenced by its storage epoch. Evidence holds one hashed projection per selectable candidate; none, more than 16 or over 8 MiB is a limited item. Recording is idempotent per bytes and evidence and never touches a decision. Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the job scope gains review_items/review_errors and the log a Needs review line. The monitor records after its fallback link, best effort. |
||
|
Vito Sansevero
|
de6af76064 |
feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata, media_review_items, media_review_decisions) beside the image annotation group it cites. review_schema_error validates the catalog, columns, foreign keys, row values and canonical 8 MiB evidence read-only; migrate_review_state installs it in a SAVEPOINT after the annotation group, and verify_database_file requires it under require_library_state. A review group without a valid annotation group is refused. Decisions null their pair and image navigation through indexes when those are deleted. Old backups and annotation-era backups restore and gain the group; browser fixtures clear the new tables on reset. |
||
|
Vito
|
195a1d33ac |
feat(annotations): Tag image dialog, card Tag button and inspector tag chips (#281)
* feat(annotations): image tagging controller with staged suggestions Add the UI-agnostic Tag image dialog controller (#249 C1): observe the file into a capture, prefill the saved tags and the server's tagging defaults, stage the top suggestions as removable chips, and save manual tags plus the remaining chips with case-insensitive dedupe (manual spelling first). Every read is serial-guarded; saves are sent once and a lost answer is resent identically only by Retry. Conflict, storage, file and capture outcomes keep the draft and gate Save on a reopen. ApiError now carries a validated copy of a tag-conflict body's current tag set so the dialog can show it. * feat(annotations): Tag image dialog beside Info on every image card Every Images and Library-embedded gallery card shows the concept's Tag after Info. Tag opens the concept's Tag image dialog (image-tagging.mjs over the C1 controller) in the shared modal shell with the sticky tagging-modal footer: capture preview, manual tags, the four legacy engines with real defaults, explicit Generate, removable suggestion chips (D83), legacy model labels (D84), a Suggestions count (D91) and Retry where the controller allows it. The capture is released on close, route change and pagehide. The Tags page gains the concept's Auto-tag images for the image selected in Images (aria-disabled without one, D115) and real Images counts that follow every save. Retires D9, #245 D2 and #245 D3. The toast now uses the mock's copy verbatim ("tags" even for one), C1's D60/D61/D68 markers are renumbered to D83/D84/D91, and at extreme zoom a card's info row stacks (and wraps Info and Tag) when its row is narrower than the narrowest bounded card, list rows included (D31). * feat(annotations): show image tags in the inspector and viewer The inspector parses the B3 envelope's image_tags facet (strictly: exact keys, current/none/unavailable, at most 100 vocabulary names) and draws one chip row after the prompt blocks: the file's current image tags first, then D21's linked tags not already shown (case-insensitive). In the Images panel image tags are Tags filter buttons and linked tags keep D21's rule; in the viewer and wherever no Tags filter exists every chip is plain. none and unavailable add no chips (D88, plan D65; D21 amended). The tagging fixture reset now drops leftover captures, so a long run of tagging tests in one worker stays below the capture store's limit. * fix(annotations): re-read a tagged file in place after a save After a tag save the selected file's inspection is re-read quietly: the panel and the viewer keep the current details until the new answer lands, with no "Loading file details…" in between. While a Tags filter or a search is active, Images re-reads its loaded range in place (same query, from the first page up to the loaded count, published once), keeping the cards, the scroll position and the selection while the selected file still matches; when it no longer matches, the panel follows the collection. * fix(annotations): give suggestion chips the concept chip corner The D83 chips took the later `.tag-suggestion` row rule's 5px corner; the concept `.chip` has 4px. The concept comparison of the Tag dialog found it. * test(annotations): compare the Tag dialog with the approved concept The Tag image dialog's inventory (order, text, field attributes, tokens, preview box, chips, status, sticky actions) must equal the concept's at 1600x1100 and 390x844, dark and light, opened, generated with WD14 and with a language model, for an empty result and in the review scope, and the card's Info and Tag group must equal the concept's. Production runs on the concept fixture with the concept image's tags saved and the fake model answering the concept's own sample; only D29, D83, D84 and D91 are mapped, and a mutation check shows the comparison notices a change. The D3 footer check again asserts a strictly shorter footer, with short seedless subtitles that stay on one line beside Info and Tag. * fix(annotations): final review round for the Tag image dialog - Option fields show the server defaults that Generate sends: the controller bumps an options revision on open and on a model change, and the panel re-renders only then (I1). - A tag conflict shows the current set at once ("Saved now: ...") and puts it ahead of the draft in the manual field, so the next Save sends what is shown (I2). - The opening sentence is the concept's on-screen copy; "Remove anything you do not want to add." leads the chips only while chips exist; the comparison's opened mapping is gone (I3). - ApiClient gains an additive keepalive option; the capture release and the Save PUT use it, and pagehide closes the dialog (I4). - Closing during a save keeps the capture until the PUT settles, then releases it and still reports the outcome (toast, page refresh) (I5). - A refresh asked during a page load is pending and runs once it settles; re-reads use pages up to 200 files; a failed re-read toasts; Load more waits for a refresh (I6, M2, M3). - The selection bar drops checked files a refreshed result no longer lists (I7). - D116 records Retry; after a failed Generate it takes Generate's own path, and after a changed storage, file or expired capture it observes again keeping the draft (I9, M4, M5). - A save made from Tags marks a filtered Images visit stale (M6). - images.mjs drops under 800 lines: the tagging wiring moves to images-tagging.mjs and the favorites star to images-favorites.mjs (unchanged), and reset() reuses dropSelection() (M1). - Restore the activity-scan.mjs coverage floors; drop the duplicate paired captures test (I8, M9). * fix(annotations): coverage floors for the Tag dialog round Floors for the new images-favorites.mjs and images-tagging.mjs, and the measured raises of image-tagging-state.mjs and image-tagging.mjs. activity-scan.mjs keeps main's floors. |
||
|
Vito
|
1b8fbdbb75 |
feat(annotations): image tag routes, model gate and scored suggestions (#278)
* feat(annotations): add scoped observe, capture preview and tag save routes
POST /prompt_manager/image-annotations/observe captures the file named by
the listing identity {root_index, root_scope, path} (scope checked against
the current roots before any file access) and answers the saved tag set of
exactly those bytes, refreshing a matching record's verified fingerprint.
GET/DELETE .../capture/{token} serve the private copy (no-store, nosniff,
pinned with acquire/release_use while it streams) and release it.
PUT .../tags re-hashes the current file and writes only when it still
holds the captured bytes, under the storage epoch and tag revision
(file-changed, storage-changed, tag-conflict, tag-limit, replayed).
Bodies are capped at 64 KiB with unknown fields refused; envelopes never
carry server paths or locators. Restores invalidate every capture and the
host's cleanup removes the capture directory.
* feat(autotag): scored suggestions with a configurable count behind a shared model gate
- py/model_gate.py: one process-wide, non-blocking gate for loading,
swapping, running, downloading and unloading a tagging model. Worker
calls run shielded; a release requested while one still runs takes
effect only when it exits (client disconnect keeps the gate).
- Legacy /autotag/single, /start, /download/{type} and /unload hold the
gate and answer 409 {code: "model-busy"} while another operation holds
it; their bodies are otherwise unchanged. /autotag/models gains
suggestion_limit.
- AutoTagService.generate_tag_suggestions: WD14 (name, probability) best
first, GGUF/HF (name, None) in output order, first `limit` distinct
case-insensitively; generate_tags keeps its return; the image file is
closed before inference returns.
- POST /prompt_manager/image-annotations/suggest runs one engine on the
pinned capture, with strict per-engine options, never downloads
(409 model-unavailable) and never writes.
- config.json autotag.defaults.suggestion_limit, integer 1..100, default
10, validated on load.
- Capture probe decides Generate from the image header alone, so prompt
metadata over the inspection budgets no longer blocks it.
* feat(images): match image tags in the Tags filter, search and inspector
The workspace Images listing now admits a file under a Tags filter when
every requested tag is carried by its linked positive or its current image
tag set, and q also matches image tag names (case-folded substring). Image
tags count only while a byte record is verified at the file's present
(size, mtime_ns) (spec amendment S1): the tagged records are streamed
once, only listed files carrying a filter tag or q hit are stat'ed, and
their linked positives are read through the file_path index so tags,
category, rating and the positive count combine.
The rooted inspector envelope gains a validated image_tags facet
(current / none / unavailable) read at the inspection's own stable stat.
A failed image-tag read fails a Tags filter like a failed link read and
degrades a q search to no image tags; plain listings read none. Legacy
gallery and search responses are unchanged.
* fix(annotations): answer a fixed coded message when the filtered image list fails
A failed prompt-link or image-tag read under a Tags, category or rating
filter returned the raw exception text (SQL or database messages) in the
500 body. The workspace listing now answers code listing-failed with a
fixed sentence and logs the traceback. The legacy output listing and the
inspector image-tag read now log with exc_info as well.
* fix(annotations): drop suggestion names that Save would reject
A JoyCaption part over 50 characters or holding a newline came back as a
preselected chip, so the first Save of the dialog failed with a 400 and
nothing was stored. generate_tag_suggestions now applies the vocabulary
tag-name rules before the case-insensitive dedupe and the limit count, so
the requested number of valid chips still comes back.
* fix(annotations): stop changing the process warnings filters during capture
header_dimensions wrapped Image.open in warnings.catch_warnings with an
error filter for DecompressionBombWarning. catch_warnings is not
thread-safe, so two overlapping captures could leave that error filter
installed for every later Pillow open in the ComfyUI process. Only
Pillow's DecompressionBombError is now mapped to HeaderTooLarge; a size
that only warns is refused by the existing inference pixel cap.
* fix(annotations): free pins on cancellation, serve preview ranges, code every storage error
- A preview cancelled while its file opens closes the handle and only
then ends the pin, so a doomed capture is never unlinked while open.
- An observe cancelled while capturing or reading releases the capture
instead of leaving it until the idle expiry.
- The model gate takes an on_skipped callback that runs once when the
work never starts; Generate passes the capture's release_use, so an
executor refusal no longer leaks a pin against the capture budget.
- The preview answers a single byte range with 206 and Content-Range
(416 for an unsatisfiable or multi-range header), still streamed inside
the pin, so Safari and WebKit can play video captures.
- _coded maps tag-limit (400), metadata-conflict and vocabulary-conflict,
and falls back to 409 with a fixed sentence for any other code instead
of raising KeyError.
* fix(annotations): deny case and Unicode spellings of a listed file
On macOS (case- and normalization-insensitive APFS) a crafted target such
as A.PNG or an NFD name opened the listed a.png or NFC file while
realpath kept the requested spelling, so the tags were stored under a
second locator the Images filter never shows. Capture and verify now
require each path part to be spelled as its directory lists it and deny
any other case or Unicode spelling as an alias (409 source-changed, like
a symlink alias). A missing file with no other spelling keeps its usual
outcome.
* test(annotations): one shared folder_paths stub for the API test modules
test_api_images and test_api_prompts installed a MagicMock folder_paths,
so whenever one of them was imported before test_api_autotag the later
patch("folder_paths.get_output_directory") in test_api_history_outputs
left a mock attribute behind and two test_diagnostics_config tests
failed. tests/host_stubs.py now installs one real-module stub, used by
test_api_images, test_api_prompts and test_api_autotag (and through
test_api_images by the annotation modules, whose copy is gone), so the
modules pass in either import order.
* fix(annotations): derive the suggestion cap from the image tag set cap
py/autotag.py now takes MAX_SUGGESTION_LIMIT from database.annotations
MAX_IMAGE_TAGS. The two copies that cannot import it (py/config.py loads
before the database, utils/inspection_response.py stays storage-free) are
pinned equal by a test, so the 100 limits cannot drift apart.
* docs(testing): record capture aliases, preview ranges and the PR B follow-ups
TESTING.md now states that case and Unicode spellings are aliases, that
the preview serves byte ranges inside the capture pin, where the
cancellation tests live, and two tracked follow-ups: the legacy batch's
ungated cleanup unload under its own last inference, and the three files
over the 800-line ceiling (split py/autotag.py first).
|
||
|
Vito
|
0095db8cbc |
feat(inspect): local evidence readers for images and videos (#277)
* feat(inspect): bounded browser PNG reader matching the server's
The Inspect page will read a chosen file entirely in the browser, and the
record it shows must equal what Images shows for the same bytes. This adds
to web/js/png-metadata.js (loaded by the legacy pages as before, with
parsePngTextChunks unchanged):
- readPngMetadataFile(file, {signal, prefix, yieldControl, decompress}):
a port of utils/prompt_inspector.py's _png_metadata, _PngRecords,
_png_text, _inflate and _batch_normalized. It slices only chunk headers,
80-byte keyword prefixes and the retained prompt, workflow, parameters
and comfyassets_prompt_pairs records, CRC-checks what it keeps, applies
the same budgets (2 MiB shared role budget encoded and decoded, 4 MiB
workflow that becomes limited, 8 MiB and 4 MiB duplicate budgets, 4 MiB
duplicate decoded parameters, 10,000 chunks, the batch-index rule for
parameters copies) and compares duplicate chunks byte for byte. Every
range is checked against file.size before slicing and charged against a
19,794,389-byte ceiling, so a multi-gigabyte PNG costs only its metadata.
Compressed text inflates through DecompressionStream with one 256-byte
write in flight and the output capped at the record's allowance; an
AbortSignal rejects the read with an AbortError and leaves no unhandled
rejection; yieldControl runs about every 8 ms of work.
- classifyPrefix(bytes): png, jpeg, webp, iso-bmff, matroska, avi, asf or
null from the first 32 bytes.
- readImageHeader(file, kind): JPEG (SOFn walk as Pillow) and WebP (VP8,
VP8L, VP8X) dimensions from at most 64 KiB.
tests/test_local_inspect_parity.py drives tests/js/support/local-inspect-
oracle.mjs over the committed and test-time inspection parity corpus plus
compressed, oversized, Latin-1, batch-duplicate and all-four-record shapes,
and asserts records, record formats, dimensions and readability equal
read_inspection_file on each file; the two JPEG/WebP fixtures are checked
against Pillow. tests/js/png-local-reader.test.js pins each server rule,
the sparse 3 GiB read, the prefix reuse, inflation back-pressure and abort.
tests/fixtures/local-inspect/tiny.{jpg,webp} are CC0 FFmpeg test patterns.
* feat(inspect): bounded browser video tag readers
Add web/js/workspace/local-video.mjs: ISO-BMFF (MP4/MOV/M4V) and Matroska
(MKV/WebM) walkers that read only the container tags the server asks ffprobe
for, slicing box and element headers, carrier names and retained values within
the server's budgets and never a byte inside mdat or a Cluster; a port of
utils/video_metadata.select_video_carrier with the record budgets of
read_inspection_file; and a byte-exact json.loads/json.dumps round trip so
envelope objects reserialize as the server spells them.
Every naming rule mirrors ffprobe 9.0.2 on hand-built files: legacy ilst atoms,
mdta keys, joined repeats, data types, Matroska language and default handling,
and Tags behind a Cluster found only through a SeekHead.
Tests: node units over byte-exact container builders and the committed
fixtures; a Python parity test comparing the local carrier with the server's
selection from the same tag maps (mocked probe) and from the same files (real
ffprobe, including 1 MiB + 2.5 MiB per-key records); an ffprobe
characterization of the recorded fixture maps.
* feat(inspect): server-identical local projection and payload
Port utils/inspection_response.py to the browser: a strict parse with Python's
Decimal spelling, duplicate-key rejection, NaN/Infinity as null and integers
past 2^53-1 as text; the prompt, workflow and carrier projection with the
depth-64 and 20,000-node ceilings; _metadata; and the 8 MiB omission steps
(workflow graph, then prompt graph, then all raw) measured as UTF-8 bytes.
local-inspection.mjs builds the inspection_version 1 wire for one chosen File
from the Task 1 and Task 2 readers (observation selected-local-file, interim
unresolved roles, unavailable library), fits it to the response budget, and
exposes localFileAdapter for the shared controller, inspector and viewer:
handles are session-owned by a private token, so a tampered or hand-made
descriptor is rejected and blob: URLs are never resolved through the API
client. projectInspectionPayload admits selected-local-file only for a
local-file identity.
A Python oracle test compares projectMetadata with _metadata byte for byte over
the parity PNG corpus, the committed video fixtures and tag maps, 75 committed
adversarial record sets (projection-cases.json) and node-ceiling sets built at
test time.
* test(inspect): import oracle modules by file URL for Windows
The local-inspect oracle imported product modules with path.join(...), which
Node parses as a URL scheme on Windows (D:\... -> ERR_UNSUPPORTED_ESM_URL_SCHEME),
failing every tests/test_local_inspect_parity.py oracle test on the Windows CI
job. Every dynamic import in tests/js/support now goes through moduleUrl, a
pathToFileURL wrapper, and a node unit proves the Windows form on any host and
scans the oracles so a path import cannot return.
* fix(inspect): read video tags as ffprobe reports them
Same bytes now give the server's record in five shapes where the local
reader was stricter or kept different tags (PR #277 final review I1, I2,
M1, M2, M4):
- Invalid UTF-8 in a tag value is replaced as ffprobe's default
string_validation=replace does (av_utf8_decode, one U+FFFD per failing
decode step), instead of failing the read. Key names and TagNames that
do not decode are no carrier and are skipped.
- An ilst item whose keys index is 0 or out of range is skipped, as
ffmpeg does, instead of failing the read.
- Only the first keys box is read, as ffmpeg does.
- A first per-key workflow over 4 MiB keeps its place in the carrier
choice as an unread marker, so the server's carrier (per-key, workflow
limited) is chosen; when the server's first probe would overflow, the
re-probe leaves it out and its own bound applies.
- An envelope integer of more than 4300 digits fails the Python json
port, so that envelope is skipped as CPython's json.loads skips it.
Each shape was checked against ffprobe 9.0.2 and read_inspection_file;
the real-ffprobe parity test now builds them, and a 600-case random-byte
comparison of the replacement against ffprobe matched.
* perf(inspect): measure the local response size natively and yield between steps
fitInspectionPayload counted the UTF-8 size of an 8-12 MB wire with a
per-character loop and ran every omission step on the main thread in one
turn (PR #277 final review M7). The size is now counted by TextEncoder
encodeInto through one bounded scratch buffer, and each re-measure
follows a yieldControl turn, with an abort signal checked between steps.
The fit is async; the fitted payload is the same as before.
* fix(inspect): hold video tags as ffmpeg's dictionary does, decode Mac Roman text
Three more same-bytes divergences, found while verifying the review
fixes against ffprobe 9.0.2 and read_inspection_file:
- ffmpeg keeps one tag per ASCII-folded name under the spelling set
last (ISO-BMFF joins the values with ";", Matroska keeps the last), so
PROMPT and Prompt are one record on the server; the local reader kept
both and failed the read as a duplicate. A legacy comment atom and a
later COMMENT key likewise leave no exact-case envelope.
- ffprobe's format_tags filter folds ASCII case only, so a name that is
"workflow" only under Unicode case rules is never reported; the local
reader took it.
- ISO-BMFF data types 0 and 3 are Mac Roman text to ffmpeg, written into
twice the input size (a long value is cut); the local reader decoded
type 0 as UTF-8 and skipped type 3.
The real-ffprobe parity test builds each shape.
* docs(testing): list the local Inspect divergences; raise touched coverage floors
TESTING.md now tracks the local Inspect (#249 PR L1) divergences that
remain after the review fixes, each checked on a host with ffprobe 9.0.2:
media kind by bytes versus extension (parity is for the same bytes with a
matching extension), JPEG dimensions past the 64 KiB header read, classic
QuickTime udta text atoms, raw versus escaped tag bounds, the 10,000
element ceiling, unknown EBML DocTypes, the JSON depth approximation and
the C _decimal assumption.
coverage:update raised the branch floors of the three local Inspect
modules this PR touches; the unrelated activity-scan raise it proposed is
left at main's values.
|
||
|
Vito
|
b7cf210f0f |
feat(annotations): image tag storage, vocabulary participation and capture store (#276)
* feat(annotations): add image annotation schema with restore validation
Install the image annotation group beside image_favorites and saved_views:
media_annotation_metadata (one row, schema version 1), media_annotation_files
(one private locator per physical file), media_annotation_bytes (one record
per observed SHA-256 with the verified size and mtime_ns) and media_image_tags
(the ordered tag set of a byte record, joined into the global tags registry).
The group migrates in a SAVEPOINT that validates before and after creating
it, with no backfill from prompt or pair tags and no filesystem probe.
annotation_schema_error reads only: it compares the catalog objects after
whitespace compaction, the columns, foreign keys and indexes, and the
persisted values (UUIDs, hex digests, bounds, the 4096-character locator cap
measured past embedded NULs, exactly one metadata row). verify_database_file
checks it with allow_missing=not require_library_state, so old backups
restore and a migrated restore stage must carry it.
Tests pin the exact DDL, a fresh install and idempotent reopen, a current
database without the group gaining it with legacy rows untouched, every
partial or malformed shape rejected without repair, install failure rolling
back and retrying, a WAL backup/restore round trip, staged-failure
immutability of source, live and safety files with every handle closed, and
the require_library_state gate.
* feat(annotations): persist per-image tag sets in the shared vocabulary
Add database/annotations.py (AnnotationOperations, mixed into
PromptDatabase): read_image_tags, replace_image_tags and add_image_tags
run in one _pair_transaction fenced by the storage epoch, own the byte
record by exact locator and digest, always supply UUID ids, rewrite the
ordered junction and bump tag_revision plus image_index_revision.
Revision mismatches raise AnnotationCommandError("tag-conflict") with
the current set; the identical set at the current or previous revision
is replayed without writes; a union over 100 names is "tag-limit".
normalize_image_tags validates each name with validate_vocabulary_tag_name
and folds case-insensitive repeats keeping the first spelling.
refresh_image_fingerprint, current_image_tags (chunked, fingerprint
matched, newest verification wins) and iter_image_tag_records serve the
S1 cheap reads.
Vocabulary participation: _move_tag_links also carries media_image_tags,
legacy merge_tags moves image-only sources, tag rows gain image_count
(files with any byte record carrying the tag), impact gains images and
rename/delete receipts gain images_changed. vocabulary-state.mjs accepts
the optional counts and names images in the delete text.
Browser fixtures clear the three annotation data tables on reset (never
the metadata row); TESTING.md documents the schema beside favorites.
* feat(annotations): add bounded private media capture store
Add utils/media_capture.py: CaptureStore copies one rooted media file
into an owner-only prompt-manager-captures-* temp directory, streaming
it in 1 MiB chunks while computing the whole-file SHA-256, and compares
the opened descriptor with the path (dev, ino where meaningful, size,
mtime_ns, ctime_ns) before and after the copy; any disagreement raises
CaptureChanged and removes the copy. Symlinked files or directories,
root escapes, non-media names and directories raise CaptureDenied; the
locator is normcase(join(realpath(root), relative)), the listing's
identity, so a symlinked root matches it.
Limits are the plan's: 256 MiB per file on stat and stream
(CaptureTooLarge), 8 completed captures, 512 MiB reserved and 2
capture/verify workers with nonblocking admission (CaptureCapacity),
900 s idle and 1800 s hard lifetimes on an injected clock. acquire and
release_use keep an in-use capture past expiry, release and
invalidate_all until its last use ends; close removes only owned files
and the owned directory. A header-only probe of the copy marks Generate
unsupported above 4096 px per side or 16,777,216 px, for unreadable
headers and for video; verify re-hashes the current file without a
copy. Every handle closes before unlink.
tests/test_media_capture.py covers exact bytes and owner-only modes,
alias and escape denial, the symlinked-root locator, change detection
during the copy (size, same-size rewrite with restored mtime through
the POSIX change time, inode replacement), both size caps, count, byte
and worker admission, idle and hard expiry with in-use retention,
cleanup scope, handle release at unlink, dimension and video
readiness, and the allowlist equality with the inspection service.
TESTING.md documents the store and the Windows change-time limit.
* test(annotations): wait on the POSIX change time only where it exists
The same-size rewrite case waits for the filesystem clock to pass the
source's ctime so the detection is deterministic on Linux's coarse
timestamps; Windows reports the creation time as st_ctime, where the
probe could never advance, so the wait runs only on POSIX.
* fix(annotations): compare the change time only between fstat reads
Python 3.12+ on Windows reports the change time from stat but the
creation time from fstat, so a freshly written file whose creation and
write straddle a clock tick failed the pre-read identity check with
CaptureChanged. The path-versus-descriptor checks now compare size,
mtime_ns and dev/ino; ctime_ns is compared between the fstat before
and after the copy, where it still detects a same-size rewrite with a
restored mtime on POSIX.
The symlinked-root test asserted the locator against the unresolved
temp path, which macOS (/var -> /private/var) and Windows (short names)
resolve differently; it now compares against the real path.
* test(browser): give the inspector evidence matrix the concept comparisons' budget
The touch run of 'full inspector attaches matched theme and mobile
evidence with reachable content' captures 32 screenshots across four
viewports and took 23.5 s, then 31.6 s, in the sharded coverage job,
hitting the 30 s default. Like the concept comparison specs it now sets
an explicit 120 s timeout; the CI-like pinned run (4 CPUs, 3 workers,
repeat-each 3) passes 198/198 with the matrix at 18-20 s on touch.
* fix(annotations): compare capture identity like with like and keep paths out of errors
Native Windows CI (Python 3.12/3.13) rejected untouched files with
CaptureChanged because the path stat and the descriptor fstat disagree
there (change time versus creation time, possibly dev/ino encodings).
The descriptor is now checked against the path only on the API-stable
facts (size, mtime_ns); the full identity (size, mtime_ns, ctime_ns,
dev/ino where meaningful) is compared between the fstat before and
after the copy and between the path stats before and after. A test
injects shifted fstat results to simulate the Windows discrepancy, and
a fresh file is captured twice with no platform guard.
Error messages carry strerror only, never the path (the path goes to
the debug log); a test asserts no CaptureError names the root. Files
are unlinked outside the store lock; close() during an in-flight
capture lets that worker remove its temp file and then the directory.
get() documents that readers of capture.path must acquire/release_use.
The test fixture resolves its temp root (macOS /private/var, Windows
8.3 names). TESTING.md states the real identity contract.
* fix(annotations): move image revisions when a fingerprint or the vocabulary changes
A replayed save that observes a new verified (size, mtime_ns) now bumps
image_index_revision, like refresh_image_fingerprint, so caches keyed
on it see the record become current again; an identical replay still
moves nothing.
Vocabulary rename, merge and delete (native and legacy paths) bump the
tag_revision of every image set carrying the source tag, in the same
transaction and before the links move or cascade, plus one
image_index_revision bump when any set was affected. A Tag dialog
opened before the change therefore gets tag-conflict with the renamed
set as current instead of recreating the old name.
* test(annotations): accept change detection wherever fstat exposes a change time
Native Windows CI on Python 3.13 detected the same-size, mtime-restored
rewrite through the descriptor's change time (3.10-3.12 report the
creation time there and complete the torn copy). The test asserted the
platform, not the signal: it now accepts CaptureChanged plus temp
removal wherever it happens, requires detection on POSIX and on
Windows 3.13+, and otherwise requires the digest to describe the
copied bytes. A new test injects a drifting fstat change time so the
detection path runs on every platform; TESTING.md records the limit as
"Windows before Python 3.13".
* test(annotations): assert the rewrite outcome, not the platform
Native Windows CI detected the same-size, mtime-restored rewrite on
3.12 and missed it on 3.13 in the same run: the path stat's change time
(Python 3.12+ fast path) only moves when the clock ticks between the
pre-stat and the rewrite, and the tick wait was skipped on Windows. The
wait now runs everywhere with fresh probe names (Windows tunnels the
creation time of a reused name), and the test asserts the outcome: a
detected change leaves no temp file; an undetected one is accepted on
Windows only and must produce a digest of the copied bytes; POSIX must
detect it. The injected-fstat test keeps the detection path covered on
every platform.
* fix(images): keep the Filters label node across a listing reload
Every listing reload reports the active refinement count, and
setActiveFilters rewrote the button's textContent even when the label
was unchanged, replacing its text node. WebKit drops a click whose
mousedown hit a node that was detached before mouseup, so a Safari
user who types a search and clicks Filters inside the 250 ms debounce
lost the click, and the webkit page workflows timed out in the
source-change recovery tests (1 in ~30 runs of the pinned image; an
instrumented probe showed pointerdown/mousedown, a DOM mutation,
pointerup/mouseup and no click event).
The label is rewritten only when it changes. The toolbar unit test
counts textContent writes (1 for an unchanged count before, 0 after),
and a workspace-images test pins the text node identity across a
search reload: 0/6 before the fix on chromium and webkit, 6/6 after,
with the recovery and Refine tests 36/36 in the pinned CI-like run.
|
||
|
Vito
|
bc14f29db7 |
fix(inspect): video metadata carrier parity between Images and Inspect (#275)
* fix(video): select one prompt carrier from per-key or comment tags Current ComfyUI savers write one container tag per key (prompt, workflow, parameters, comfyassets_prompt_pairs); only the old VideoHelperSuite comment envelope was read. select_video_carrier picks exactly one carrier in the order per-key tags, comment envelope, description: the first with a role key wins, else the first with a workflow, and carriers never merge. Matroska upper-cased tag names fold on our side; differing fold-duplicates fail the read like duplicate PNG keys, identical ones are one record. Per-key values stay original-text; envelope objects are reserialized. * fix(inspect): read per-key video prompt tags The inspector asked ffprobe only for comment and description, so videos written by core SaveVideo/SaveWEBM and current VideoHelperSuite (one tag per key) were inspected as having no metadata. It now asks for prompt,workflow,parameters,comfyassets_prompt_pairs,comment,description and hands the tags to the shared carrier selector. ffprobe stdout is bounded at the role and workflow budgets plus 64 KiB of envelope; past that it probes once more without the workflow tag within the role budget and marks the per-key workflow limited, while an over-budget envelope still fails the read. Differing fold-duplicate tags fail like duplicate PNG keys. * fix(video): scan videos written by current ComfyUI savers The output scan and live monitor read container tags through metadata_from_tags, which now delegates to the shared carrier selector, so per-key prompt, workflow, parameters and comfyassets_prompt_pairs tags (including Matroska upper-cased names) are ingested with the default resolver profile exactly like the same graph in a PNG. Fold-duplicate tags and null markers are skipped quietly. Pins the behavior with a mocked ffprobe scan test and real MP4/MKV/WebM round trips when ffmpeg is present. * docs(testing): document video carrier selection Record the shared video carrier order (per-key tags, comment envelope, description), the fold-duplicate rule, the 6 MiB + 64 KiB probe bound with the role-only re-probe and limited workflow, and that scans read the same carrier as the inspector. * fix(inspect): size video probe bounds for JSON escaping, exact-case envelopes ffprobe JSON-escapes tag text, so bounds measured on its stdout could mark a video workflow limited, or fail a role record, that the identical PNG reads within its decoded budget. The first probe now allows twice the role and workflow budgets plus 64 KiB (12 MiB + 64 KiB) and the role-only re-probe twice the role budget plus 64 KiB, leaving the decoded per-record budgets as the only gate; a reader-level test reads near-budget quote-dense records as PNG and as video and asserts identical metadata and formats. Envelope keys (comment, description) are matched exactly as the legacy readers did; only the four per-key carriers fold Matroska case. Fixtures can write the legacy ilst comment atom (metadata_tags=False), and the docs now say the admin scan reads the carrier while the live monitor still processes images only, with a follow-up recorded in TESTING.md. |
||
|
Vito
|
f95f66d76a |
ci(browser): run page workflows with parallel workers and CI shards (#274)
* test(browser): one test server per Playwright worker The page workflows started one global webServer and ran with one worker. Each worker now owns its own tests/browser/server.py on an OS-chosen port, with its own temp root and database, through a shared `test` in tests/browser/fixtures.mjs. baseURL, and so page, request and browser.newContext, point at that worker's server. A server that dies fails the running test with its stderr tail and the next test gets a fresh one. The worker count comes from PM_TEST_WORKERS, else 1 under CI, else a quarter of the CPUs up to 8. Chromium workflows in the pinned image: 510 tests in 606 s with one worker on main, 511 (one new isolation spec) in 119 s with eight. * test(browser): test servers bind an OS-chosen port themselves tests/browser/server.py accepts --port 0: it binds 127.0.0.1:0 before building the app, holds that socket for its whole life and prints PM_TEST_SERVER_PORT=<n> to stdout. startBrowserServer defaults to port 0 and reads the announced port, so no port is reserved and released before the server binds and two launchers can no longer race for one. The "address already in use" retry and free-port.mjs are gone; --port N still works and is checked against the announcement. * ci(coverage): shard and merge modes for the frontend coverage run frontend-coverage.mjs now plans its steps from the command line. No arguments still runs everything, then reports and checks. --shard I/N runs one slice of the chromium+touch workflows into coverage/js/raw without wiping it or reporting; shard 1 also runs the node units and the viewer suite. --merge-only reports and checks over raw coverage the shards wrote, with --update-baseline as before, and refuses an empty or missing raw dir. npm scripts: test:coverage:shard and coverage:merge. * ci(viewerjs): run browser suites with workers and shards Frontend coverage runs as three shards (PM_TEST_WORKERS=3 each) that upload their raw V8 files as coverage-raw-N; a merge job downloads them, fails unless all three arrived with raw files and no clashing names, then runs coverage:merge. Firefox and WebKit page workflows run as two shards each with three workers. The required viewerjs context keeps its name and now needs the shards, the merge and the browser shards, failing unless every one succeeded. * ci(viewerjs): cache npm downloads and size timeouts from the first sharded run The first sharded run lost a webkit shard to its 12-minute limit after all 267 tests passed, and npm ci stalled for 5 minutes in six of eight jobs. Each npm ci now restores npm's download cache keyed on the vendor lockfile, and browser job timeouts are the measured times plus at least 50%, never under 20 minutes. The workflow comment records the measurements. * test(browser): wait for rendering frames, not timers or luck, under load Two Library workflows failed in CI's webkit shard 2/2 (run 39714840809) under parallel load: - "library infinite loading ..." counted cards right after the last page hid "Load more". That render queues the window update for the next frame, so a slow frame left the 120 cards of the all-rows layout in place (received 120, expected < 80). It now waits for the next frame, which runs after every callback the page queued. - "row measurement preserves anchor ..." settled with a 60 ms timer. A scroll event is delivered in a rendering frame, so on a starved renderer the timer ended before the scroll's frame and no card was in view (null anchor). settleLibrary now waits for two consecutive native frames with no Library frame queued, checked inside the frame callback. Before/after: a scratch copy that delays every frame by 120 ms (as a starved runner) failed the old helpers 19 of 40 runs, at the CI lines, and passed the new ones 40 of 40 on chromium and webkit. Throttled to 1 CPU with 4 workers, the original spec failed 1 of 42 for each test. The Tags viewer workflow closed the viewer as soon as its image was visible, before ViewerJS rendered the title, so CI's coverage never ran the page's title callback (tags-page.js functions 26/46 = 56.52%, one function above the floor of 56; locally 27/46). The test now asserts the title names the shown file, which always runs the callback. * test(browser): stable under parallel workers Document how to run the browser suites in parallel and sharded, with the measured times, and what keeps specs stable beside each other: wait for a rendering frame, a response or a visible state, never for time. With |
||
|
Vito
|
2cecfcf0ac |
feat(activity): scan new and unresolved files by default, and download a job report (#273)
* feat(activity): scan new and unresolved files by default, all files on request
Scan for metadata now skips, before reading any metadata, the files the
Library already links with a resolved link (one generated_images row with
exactly one pair link), found with one streamed query that keeps only the
scanned paths and compares paths with os.path.normcase on both sides.
Skipped files count as Already known ("Skipped · already linked").
Unlinked, fallback-linked, ambiguous and unresolved files are read again.
POST /prompt_manager/jobs/scan takes {scope: "new" | "all"} (default
"new", anything else is a 400). The review dialog shows the concept's
"New and unresolved files" and adds an "All files" switch (D58). The
legacy /prompt_manager/scan route keeps scanning all files with unchanged
SSE payloads.
* feat(activity): download a job report
GET /prompt_manager/jobs/{id}/report returns the summary, counters, every
outcome group's listed rows with truncated flags, the scope and the log as a
JSON attachment (prompt-manager-<kind>-<YYYYMMDD-HHMM>.json), with no absolute
paths. The job view's Download report saves it through the api-client URL, and
the scan log opens at its newest line and follows new lines only while the
reader is at the bottom.
* fix(activity): re-read known files whose size changed, and review follow-ups
The new scope now skips a known, resolved file only while its stored
generated_images.file_size equals its size on disk; a NULL stored size is
re-read. A file replaced at the same path is read again and its conflict
shows under Needs review. The scan stores each file's size when it ingests
it, and both sides of the known-set match go through normalize_image_path.
A failed known-set read logs and falls back to reading every file, and an
empty request body asks for the default scope. The All files switch is
described by its hint, a lost job's Download report is disabled, the report
toast is skipped once the view is gone, and the scan log opens at its
newest line every time. Fixtures use the real scope shape.
* test(activity): wait for the job view's settled detail read, not a fixed delay
The 200-row cap test now starts counting detail reads once one read has
returned the full listing and none is in flight. The job view re-reads after
an in-flight read only when the summary still lists more than that read
returned, so no redundant read is left due after opening.
|
||
|
Vito
|
a6f6697c50 |
feat(activity): shared job registry, Activity page and Scan for metadata (#269)
* feat(activity): shared session job registry with the scan as its first kind The output scan now runs as the first kind of an in-memory JobRegistry (py/api/jobs.py, replacing py/api/scan_job.py). Each job keeps exact counters, at most 200 outcome rows per group and a log capped at 2,000 lines, with root-relative paths and public root labels only. New routes list, inspect, follow (SSE) and cancel jobs and start the scan (POST /prompt_manager/jobs/scan, 409 busy while another record-changing job runs). The legacy /prompt_manager/scan routes attach to the same job and stream unchanged payloads. * feat(activity): concept Activity page, top-bar button and progress pill The workspace gains the concept's Activity page (renderActivity), its top-bar button and sidebar row, fed by one session job store: a /jobs read, one SSE stream per running job, a 1-30 s capped backoff after a dropped stream and a 15 s list poll while visible. A job the server no longer lists shows as ended, so no pill stays stuck after a restart. D52: the counts show running jobs and hide at 0; the phone top-bar row may wrap. D53: the mockup's progress pill links the running job. D54: real job items with state chips, detail lines and Cancel job. Start and finish toasts follow the spec copy. Fake browser jobs come from tests/browser/workspace_activity_fixture.py. * feat(activity): Scan for metadata with its review dialog and job view Images -> Import keeps Rescan and adds the concept's "Review scan" row as "Scan for metadata". It opens the concept's "Review metadata scan" with the real source count, the fixed "All files" scope, a backup box that downloads the existing database backup, and Cancel / Start scan. Start scan starts the session scan job and opens its view; a start refused by another record-changing job toasts its title as a link. #/activity/<id> is now the approved job-view mockup: the heading with Download report (disabled until Activity-2) and Cancel scan, the Progress surface with the five counter tiles, the Results surface with the outcome list, path filter and capped rows (Added and Already known open the viewer through the listing identity), and the collapsed Scan log. Activity item titles link their views. The scan's progress line groups thousands. Deviations D55-D57 are recorded in TESTING.md. * fix(activity): stop the job store on pagehide and let proxied routes outlive teardown CI's touch coverage run failed "a resolver failure for the ComfyUI link never stops the shell from mounting" with "route.fulfill: Fetch response has been disposed": its trace shows two Library thumbnail requests started by the last route still in the spec's proxy-mount handler while the page fixture was torn down. - The proxy-mount handler ignores only that teardown error, and the workspace and Activity specs unroute with ignoreErrors before teardown. - The Activity store now ends its work on pagehide: the poll and backoff timers stop, the job streams close and a list read in flight is aborted (an older read is also aborted when a newer one starts). A pageshow from the back-forward cache reads the list again. * test(activity): compare the job view log's line height to the hundredth of a pixel WebKit reports the mockup's 11px/1.7 line height as 18.700001px, so the job view comparison failed on WebKit only. The measurement now rounds it, as other engine-dependent values are normalized. * fix(activity): final review round for the job store, job view and scan rows - Store: one failed list read no longer marks running jobs as ended; two in a row, or a good list without them, do (M1). A good read resets the backoff, except for a stream that keeps dropping while the list reads fine (M2). Finished jobs the server pruned are dropped (M3). Numbers group as en-US, like the server's lines (M13). - Job view: a job the store's list no longer has shows the not-found state (M3). Focus moves to the job heading when Cancel scan goes away, and to the item's title link in the list (M4). The detail is read only when its listed rows change or the open log grows, using the summary's new listed and log_length (M12). - Viewer rows open by exact identity: each scan row records its root's listing index, the source must still carry the row's label, and the workspace listing gains an exact `path` filter (M8, M9). - Scan rows name the prompt by its Library title, stored title first (M10). - A toast with a link stays 20 s, and any toast stays while hovered or focused (M14). - A real two-batch scan cancelled after batch 1 keeps partial counts (M7). - TESTING.md: D54, D56 and D57 updated (M5, M6). * test(activity): cover the held and linked toast and a job view pruned by the server Unit tests for the toast's 6.5 s and 20 s timeouts and its hover and focus hold, and a browser test for a finished job's open view after the server stops listing it. Coverage floors are only raised. |
||
|
Vito
|
6e9d8f81fd |
ci(coverage): fast check that every frontend file has a coverage baseline (#271)
The frontend coverage job only reports a missing baseline entry after its full browser run. coverage:baseline-check compares the files the gate measures (same discoverSources) with frontend-coverage-baseline.json in milliseconds and also flags entries for files that no longer exist. npm test runs the same check. |
||
|
Vito
|
e9996996e2 |
feat(views): saved views and Favorites on Images and Library (#268)
* feat(views): saved-views store and API
Add a saved_views table (page images|library, trimmed 1-100 character
name, whitelisted JSON state of at most 4 KB, position, timestamps)
installed through the same validated-savepoint migration as image
favorites. It is optional in old backups and required in a migrated
restore stage; restore rejects a tampered definition, extra indexes and
any row that would fail write validation.
Routes under /prompt_manager/saved-views: GET lists views by position,
POST {page,name,state} returns 201 {success, view}, PATCH /{id} {name}
renames and DELETE /{id} removes, both 404 for an unknown id. The state
must hold exactly the page's toolbar and Refine-this-view keys (Images:
q, category, sort, source, folder, media, tags; Library: q, category,
sort, tags) with their types and sort/media values; anything else is a
400.
Refs #249
* feat(views): Save view, the SAVED VIEWS sidebar and opening a view
The shared toolbar gains the concept's Save view between sort and Filters
on Images and the Prompt library. It opens the concept's simpleInput
("Save this view", "View name", default "Search: <q>" or "My workspace
view", Cancel / Save); an empty name is "Saved view". A save posts the
page's full whitelisted state (toolbar value plus Refine filters) and
toasts "Saved view added to the sidebar.".
The shell adds the concept's SAVED VIEWS label and nav#saved-nav, loaded
with the shell and re-read after every save. Rows use the concept's
links() markup (search icon, name, no count) and link #/saved/<id>.
Opening one mounts its page with the saved state through the page's own
filters, so the toolbar and "Filters · N" (D44) show it; the h1,
breadcrumb and title are the view's name. A removed Source or Folder is
dropped to its default after discovery, so no listing read is rejected;
an unknown id re-reads the list once, then opens Images; a failed list
read offers Retry.
D16 is retired: images-concept and library-concept no longer map Save
view out. New deviations: D49 (no PROPOSED badge) and D50 (no Needs
review row); shell-concept maps Favorites out until Task 3 and compares
a saved row against the concept's. Browser fixtures now wipe
saved_views.
* feat(views): Edit rename/delete and the Favorites saved view
- Favorites is the first SAVED VIEWS row (star icon, name, real count).
The count is the Favorites listing's own limit=1 total, follows the
nav-count rules and is re-read after every confirmed star or unstar.
- #/favorites opens Images filtered to favorites. The workspace listing
takes favorites=1: each selected root's favorites are read through the
image_favorites primary key and matched against the cached scan with the
toggle's case folding; a failed read is an error, never an unfiltered
page. The cursor scope includes the flag.
- Saved Images state gains a required favorites boolean, so Save view on
Favorites reopens the favorites.
- An open saved view's heading leads with Edit: "Edit saved view" with
View name, Delete... (danger), Cancel and Save. Delete... confirms in the
same dialog (never window.confirm). Toasts "Saved view renamed." and
"Saved view deleted."; a delete returns to the view's plain page.
- shell-concept no longer maps Favorites out (Needs review stays, D50).
Recorded as D51 in TESTING.md.
* fix(views): open stale saved views safely and finish the review fixes
Startup and restore now check only the saved_views structure (table,
columns, types, sizes, a JSON object state) and never the value rules,
so a row saved under older rules no longer stops the database from
opening. Reads normalize each view: missing keys take their defaults,
unknown keys are dropped, invalid values default and the name is made
displayable. POST and PATCH still validate strictly.
Only request validation (SavedViewInvalid) maps to 400; database and
internal errors are a logged generic 500. A saved view no longer
replaces the plain Images or Library return state, a toolbar change
while sources load keeps a saved Source and Folder, a state over 4 KB
is refused in the browser with its own toast, the delete confirmation
skips its failure toast after the page is gone, the Edit dialog reuses
the shared field helper, and quick star toggles coalesce into one
Favorites count read.
* test(views): cover leaving a pending saved view and add its coverage floor
A saved view left while the list or a rename is still pending leaves
the next page alone; the shell drops two unreachable heading guards.
workspace/saved-views.mjs gets its frontend coverage baseline entry;
no existing floor changes.
* fix(views): a corrupt saved-view state no longer blocks startup
At startup a saved_views row whose state is not a JSON object is
logged as a warning by id only and left in place. The listing and
rename skip it, so its #/saved/<id> link behaves like a missing view.
Restore stays strict and still rejects a backup holding such a row.
|
||
|
Vito
|
762669758b |
feat(history): concept gallery cards, history comparison and CSS retirement (#266)
* feat(history): concept shared gallery The history page's gallery is the concept's section.history-gallery with its panel heading (Shared gallery, <name> · same positive prompt) and the Images media cards, followed by the recorded executions. A saved variant with no images shows the concept's empty copy. Recorded as D47. * feat(history): drop the shared gallery's Linked images caption (D47 ruling) The concept's "Shared gallery" heading is the gallery's only heading: the embedded collection's "Linked images" heading and "N loaded · N linked images" count repeated it. mountImages gains caption: false and a focusTarget; only the history page's shared gallery passes them, so the Images page and the All images and Repeat run dialogs keep their caption. The layout row and the selected-image side panel stay (TESTING.md D47). * test(history): concept comparison for prompt history history-concept.spec.mjs compares the Prompt history with the concept's #prompt/alpine on the concept fixture at 1600x1100 and 390x844, both themes and motion settings, on Chromium and touch, with its behaviors in both documents and a mutation check. inventoryOf gains a history section; mapHistoryConcept applies D24, D35, D45, D46, D47 and the approved Show archived button, laying the concept out at runtime. Corrections it demonstrated: each compared negative mounts in a display: contents child of the concept's .compare-item (D25), and the concept fixture now links images oldest first and counts one pair use per image, so newest-first reads and lane run counts match the concept. history-canonical-comparison.mjs uses the concept's selectors for production and drops its closed polish entries. * feat(history): drop the Linked images caption from the history dialogs (D47 ruling) The maintainer extended the D47 trim to the history page's two media dialogs. All images for this positive and Repeat run are named by their own dialog title, as in the concept (app.js a1282-1304), so their embedded collection no longer shows the "Linked images" heading or the "N loaded · N linked images" count. A stranded focus returns to the dialog's close button. The Images page keeps its caption. The history concept comparison now records each dialog's caption count in both documents, and TESTING.md D47 covers the dialogs. * refactor(workspace): retire library.css and the interim page guards Every page is rebuilt, so the rules that waited for PRs D, E and F go. library.css is deleted with its link; the overlay is now the only stylesheet after concept.css. Deleted because the concept already draws them: the .library-grid tracks and breakpoints, the history .row-actions flex row and button size, summary cursors, h2 margins, and the sender and Recorded run dialogs' colors and border. Deleted because they had no effect: min-inline-size on block children, a .recorded-run-dialog pre limit, the general .prompt-inspector margin (now scoped to the Recorded run dialog, the only mount point that is not display: contents) and a 700px padding a later rule overrode. The duplicated sender and Recorded run dialog guards are merged into one rule each. Kept with a marker: windowed Library rows (D3), the shelf states (D40), the history panel's wrap, archived-lane and coarse-target rules (D45), and under the new D48 the production surfaces the concept never shows: recorded executions and the Recorded run dialog (#209, D26), the Archive negative confirmation (#235) and the prompt sender. They now use the concept's --green and --red tokens instead of private copies with the same values. The workspace browser specs on Chromium and the concept specs on touch pass unchanged. TESTING.md records D48 and closes the role-block gap. * refactor(workspace): drop the dead embedded caption and tighten the review guards - images.mjs: an embedded collection never has a caption (D47), so the caption option, its "Linked images" heading, the embedded count string and the heading arm of the focus fallback are removed; callers no longer pass caption:false. Behavior is unchanged. - workspace.css: delete the never-applied .recorded-run-dialog::backdrop rule (the concept's .ca-tools dialog::backdrop always won) and note that the D3 row's min-inline-size: 0 is load-bearing for the concept's <=700px 1fr track. - concept-css test: the waiting-rule guard is case-insensitive and is proven against the exact retired comments and the kept provenance labels. - history-concept spec: function replacement for the singular run label, and the DOM mutation removes Compare negatives by role. - D47 markers sit where the caption trim happens. |
||
|
Vito
|
1ff2ddce05 |
fix(inspector): fall back to the positive's category and tags for recipe-linked images (#267)
* fix(inspector): fall back to the positive's category and tags for recipe-linked images Maintainer ruling D39 (2026-10-21): a recipe-linked image shows the recipe's own category and tags when set, otherwise the linked positive prompt's, each field on its own and without merging tag sets. Set category (#259) and the Images category and Tags filters work on the positive, so the panel now agrees with them. The inspection library replaces metadata_scope with category_scope and tags_scope so the category chip's hidden scope prefix (D21) and the tag chips' filter-button rule (#260 D2) follow the value actually shown. History's media inspector uses the same shared rule. The concept fixture's positive_links workaround is gone: the comparison now matches the concept without it. * test(inspector): assert the D39 fallback tag chips in their saved order * fix(inspector): treat Unicode whitespace as a blank History recipe category * test(inspector): cover mixed-scope panels, invalid library scopes and the skipped positive tags query |
||
|
Vito
|
10a4847a6b |
feat(history): concept heading, negative variants and compare (#265)
* feat(history): concept heading and summary The Prompt history page takes the approved mock's heading and summary surface (docs/concepts/workspace/app.js renderHistory a591-598, #249 F1): - Heading description is "Prompt history · one positive, one shared gallery"; the actions stay "← Prompt library", "Edit metadata" and button.send "Send to ComfyUI" (aria-label "Send prompt pair to ComfyUI: <title>"), on one row at 1600. Send is disabled only while the selected pair's roles are loading (D24). - The summary is article.surface > span.chip category, the positive and selected negative prompt blocks, then .card-meta "N tracked runs · N images · N negative variants". Tracked runs come from the recorded history window's recorded_run_count (hidden until a read completes, kept while a refresh is in flight); images and variants from the saved positive's image_count and variant_count. - The summary's copy/send actions are named after the prompt, as in the concept ("Copy positive prompt: <title>"); mountPromptBlocks gains setTitle so an in-place metadata edit renames them without a remount. - mountRecordedHistory gains an optional onRecordedCount callback. Selector mapping: - .library-surface (history summary) -> .library-history article.surface - .library-surface .library-chip -> article.surface > span.chip - "Copy|Send <role> prompt[ to ComfyUI]" in "Saved prompt texts" -> "...: <title>" - heading p.muted "One positive prompt. Reusable negative variants. One shared gallery." -> "Prompt history · one positive, one shared gallery" - Unchanged: region "Saved prompt texts", .library-variants, .library-shared-gallery, .recorded-history (still .library-surface). Tests: workspace-history "history heading actions stay on one row at 1600 and are enabled after load" and "summary meta uses recorded runs" (both failed before), a prompt-blocks setTitle unit, and the mapped selectors in workspace-visual, workspace-library-editor and the canonical comparison. * feat(history): concept negative variants panel The history's Negative variants panel is the concept's (app.js a598, style.css 1059-1154): section.surface.timeline-panel with the panel heading (h2, p.small "Choose a red dot to change the negative and filter this gallery."), row actions Compare negatives (aria-pressed; its section is Task F3), + Add negative and the approved Show archived; a .timeline of .positive-lane and .variant-lane rows (--lane-offset 38 + 47i) with the "−" .variant-dot, "<name> · N runs" and a .run-dot when the pair's run_count > 1; and .timeline-lower with the legend and All images for this positive. D26: All images for this positive opens the concept dialog with the embedded, bounded (Load more) media gallery instead of filtering the shared gallery. The run dot opens Recorded run for the pair's newest recorded execution, else a Repeat run dialog with the pair's newest linked image. Backing data: - library variants carry prompt_pairs.run_count; - GET /prompt_manager/library/{positive_id}/history/pairs/{pair_id}/newest reads the newest run in the positive's exact raw-text scope whose current-epoch saved-pair hint still resolves to that pair (the get_history_run hint rule), inside the fenced history snapshot. - workspace-dialogs gains openContentDialog (onClose releases the body). - mountRecordedHistory returns openRun. D45 (new deviations): - a dot is named "Select negative <title>" when no other loaded lane has that title, else "Select negative variant N", so names stay unique; - the bounded "N of M saved negative variants" count stays a visually hidden status; "Saved combinations, not an execution timeline." is gone; - archived lanes (#235) are concept lanes marked .library-variant-archived at .6 emphasis with their Restore button; coarse dots keep a 47px target; the heading, legend row and All images button wrap at 390px and 200% zoom. Retired CSS: .library-branches, .library-positive-lane, .library-variant-lane, .library-variant-dot, .library-variants, .library-panel-heading, .library-variant-footer, .library-variant-summary. Selector mapping: - .library-variants -> .library-history .timeline-panel (region "Saved negative variants" unchanged) - .library-branches -> .timeline; .library-positive-lane -> .positive-lane - .library-variant-lane -> .variant-lane; .library-variant-dot -> .variant-dot - .library-variant-archived and .library-variant-restore unchanged - .library-variant-footer / button:last-child -> .timeline-lower / button - .library-variants > p.muted -> .timeline-panel .panel-heading .small - "Select negative variant N" -> "Select negative <title>" for titled pairs (history, native library, inspector, visual and real-host fixtures); untitled pairs unchanged - lane text "· N linked image(s)" -> "· N run(s)" - All images for this positive: aria-pressed filter -> dialog "All images for this positive"; tests that used it for the whole gallery select the lane linking the image, or read the dialog - variant panel row actions gain Compare negatives first Tests: new workspace-history "variants panel matches the concept", "run dot appears only for pairs with more than one run and opens its run", "all images dialog lists every image for the positive"; new workspace-library-editor "show archived and restore still work" (all four failed before the change). Python: variants carry run_count, newest pair run (DB and route). Unit: content dialog. The history fixture counts two canonical pairs and links one image to Clean geometry. * feat(history): compare negative variants Compare negatives now opens the concept's comparison (app.js a599-616, style.css 1155-1179) as a section.surface right after the timeline panel: h2 "Compare negative variants", p "Baseline compared with <name>. Choose another red dot to change the comparison.", a .compare-grid of two .compare-item negative blocks (baseline, selected), p.small "Added terms", then span.diff-added "+ term" per added term or p.small "No added terms.". Choosing another red dot updates the same section in place; pressing Compare negatives again removes it. D25: the baseline is the oldest active variant; with no selection the selected side falls back to it, as the concept's v does. The pure addedTerms(base, selected) in history-state.mjs is the concept's rule: comma split, trim, drop empty terms, exact case-sensitive difference that keeps the selected order and repeats. D46 (new deviations): - lanes are newest first, so the baseline is the lowest-id active variant, not the first lane; while the variant page has more, the oldest is not loaded and the section says "Load more negative variants to reach the baseline." instead of guessing; - the two blocks are named "Baseline" and "Selected" (regions and copy/send actions), so no name repeats on the page; - Compare negatives is disabled when there is no active variant. Selector mapping: none changed; the section is new (.library-history .timeline-panel + section.surface). Tests: unit "added terms split, trim and exact-diff like the concept"; workspace-history "compare toggles, follows the selected dot and stacks at 390" and "compare waits for the page holding the oldest active negative" (all failed before the change). * test(history): guard the newest-pair route and record D45/D46 TESTING.md lists the Prompt history page's spec decisions (D24-D26) and its new deviations D45 and D46 beside D23 and D40-D44, notes under D46 that production opens on the newest lane while the concept opens on its baseline, adds GET /prompt_manager/library/{positive_id}/history/pairs/ {pair_id}/newest to the history route list with its guards, and documents the detail variants' run_count. Tests: the newest-pair read skips a stale reference_epoch row, stops at 16 stale-hint candidates (17 newer stale rows hide an older valid run and report {run: null}), and rejects a negative-only or deleted positive; the route answers 404 for an ineligible positive, 503 storage-unavailable for storage failures without leaking details, and 409 history-changed for a restore during the snapshot. * fix(history): run-dot touch target, unique dot names and dialog cleanup - D45: under pointer: coarse the 12px run dot gets a 44px ::after target, and while a run dot shows the lane text wraps within 139px before it. - D45: dot names compare trimmed titles and never use a title that reads like a fallback "variant N", so names stay unique. - All images and Repeat run share one dialog-scope slot: each open replaces the previous scope instead of adding a cleanup per open. - A run-dot lookup that settles after the view went stale opens nothing. - Retire the unused .library-main .library-chip rule and the stale "Saved combinations" comparison note. Tests (workspace-history, Chromium and touch): the coarse run-dot target and lane width; a failed newest lookup shows the toast and Repeat run; Recorded run that cannot open (recorded history failed) falls back to Repeat run; an archived pair's Repeat run reads archive=archived; Compare negatives is disabled with no active variant; dot names stay unique. * fix(workspace): one inert handle for aborted history and prompt-block mounts An aborted parent no longer gets a separate never-called stub per method: mountRecordedHistory returns its hoisted openRun (it answers false before reading later state), and mountPromptBlocks returns one shared no-op for update, setState, setTitle and dispose. This keeps the history.mjs and prompt-blocks.mjs function-coverage floors (98) without lowering them. Tests: test:coverage (Chromium and touch, 922 workflows) passes for all 61 files; it failed before on history.mjs 94.44 and prompt-blocks.mjs 93.75 function coverage. |