3be0ada6a129cbdd7e72193986b5ccdb2f86fc26
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
8252d9d226 |
feat(system): Integrations and a staged LoRA import (System S10) (#336)
* fix(lora): detect LoraManager in ComfyUI's extra custom_nodes folders Auto-detection looked only in ComfyUI's main custom_nodes folder (#144). It now also searches every custom_nodes folder ComfyUI registers through folder_paths, main folder first, skipping folders that cannot be read. * fix(lora): stage imports before replacing the collection (#221) The LoRA import deleted the whole lora-manager category first and then saved each record in its own transaction, so a discovery, download or database failure left the previous collection gone or partial. The import now runs in three stages: discover and fingerprint the metadata files, stage every record (CivitAI downloads go into the image cache first, counted), then replace the category in one BEGIN IMMEDIATE transaction (replace_category_prompts). Any failure leaves the previous prompts, tags and image links exactly as they were. Prompts a saved pair uses block the replacement; an empty replacement of a non-empty collection needs a reviewed import. POST /prompt_manager/lora/import/preview reviews an import without writing or requesting anything; POST /lora/import/apply {token} runs it as the Activity job "LoRA reference import" (records admission, cancellable while it downloads, the safety backup policy before the replace, refused with plan-changed when the files changed since the review). The legacy POST /lora/scan runs the same stages with its SSE event shape. * feat(lora): the LoRA save takes the settings revision POST /prompt_manager/lora/enable accepts the optional expected_revision that GET /settings returns, so the System Save row refuses to write over a config.json changed elsewhere (settings-changed, nothing written). The legacy page sends none and still merges into what is on disk. * feat(system): configure ComfyUI and LoRA integrations natively System › Integrations replaces its hand-off with the concept's two surfaces. ComfyUI: only the send destination and renderer the bridge supports can be chosen; the other options and Return to canvas are disabled with the reason, and Host compatibility shows the renderer the open ComfyUI windows report (each bridge destination now carries renderer: classic or nodes2) (D106). LoRA & resources: detection, the enabled setting and the loaded trigger words read as separate facts, the read-only LoraManager path row follows the toggle (D106), and the CivitAI key is never filled in: typing drafts a new key, blank keeps the saved one and Clear saved key drafts its removal (D114). They save through the one Save row (/lora/enable with the settings revision). LoRA reference data → Review import shows the server's read-only review (replacement, what saved pairs block, downloads, backup box) and Start runs the LoRA reference import Activity job (D107), with its own Activity descriptor and Review again into Integrations when the files or pairs overtook it. * test(system): compare Integrations with the concept system-integrations-concept.spec compares System › Integrations (both themes, 1600 and 390 wide) with the concept, mapping only D106 and D114, and the LoRA import review with the concept's record-changing review, mapping D107; each comparison catches a removed row, a renamed label or option, a re-enabled unsupported option and a swapped token. On the real ComfyUI host, each bridge destination reports the renderer its window draws nodes with, before and after switching it, and System › Integrations shows it. * test(coverage): record the Integrations frontend floors coverage:update after the full frontend run: system-integrations.mjs at its measured floor, and the raised maintenance review, job view and maintenance floors. * test(system): wait for the renderer line before clicking Integrations rows CI Firefox (390 wide) clicked the LoraManager toggle while the renderer check's line above it changed text and moved the rows, so the click did not reach the switch. The specs now wait for that line to settle (a visible state) before interacting. * fix(lora): recheck the files before the replace and stop an abandoned scan Review fixes for the staged LoRA import (#221): - The metadata files are listed and fingerprinted again right before the replace transaction, so a file changed, removed or added while the downloads ran aborts the import with nothing deleted (System job: plan-changed, Review again; legacy scan: an error frame in its own words). The tests now change real files behind a held download. - A legacy scan whose page closes stops staging before its next LoRA and keeps its records hold until staging has stopped; CivitAI images are written to a temp file beside the cache entry and moved in place, so a half-written image is never seen as cached. - Start refuses with plan-changed when the integration was turned off or moved to another install since the review. - A failed job keeps the skipped count and the names of the LoRAs that could not be prepared. - The replace binds the category instead of one parameter per prompt (SQLite's variable limit), /lora/status says whether a detected install can be used, and the legacy scan logs expected refusals as warnings. * fix(system): partial saves, key focus and readable selects in Integrations A save that writes the LoRA part but not the settings part now keeps the saved part, adopts the new revision and highlights only what failed, so the next save needs no conflict. The CivitAI key status is a persistent live region, Clear saved key returns focus to the key field, and the key draft is dropped when the section is left. The send-destination and renderer selects keep their width on narrow screens. A detected LoraManager outside custom_nodes says it cannot be enabled. * fix(lora): keep the save outcome, honour the umask and sweep stale downloads Leaving Integrations drops the CivitAI key draft only when a key was typed, so the last Save's outcome survives and Sources still marks a refused source. The key status line is rewritten only when it changes. Cached CivitAI images are opened like a plain save, so they keep the umask's mode, and each import removes .pm-*.part files older than an hour from the image cache. A legacy scan stopped because its page left ends cancelled in Activity, and staging is awaited to the end even if the handler is cancelled again. * fix(lora): the import review takes a lease beside a restore The LoRA import review reads the library, so like the maintenance and restore previews it now runs under a request lease: it is refused with 409 busy while a restore runs, and a restore waits for it before it switches the file. Tests pin that a restore is refused while an import runs and that the review, the reviewed import and the admin page's scan are refused while a restore runs. |
||
|
Vito
|
2964a3b632 |
feat(thumbnails): WebP/JPEG XL thumbnails, server-side thumbnail settings and Convert outdated (#319)
* feat(thumbnails): thumbnail format, quality and size as server settings
Thumbnail format (webp default, jxl, jpeg), quality (1-100, default 80)
and size limit (128-2048 px, default 512) are server settings in the
config.json thumbnails section, saved with the settings revision. The
never-read gallery thumbnail_size is gone: older saves wrote its unused
default 256 into config.json.
One module names previews (<stem>_thumb.webp/.jxl/.jpg) and finds the
preview a source owns, preferring the configured format; legacy names
are still found. The encoder module writes every preview atomically and
verifies it decodes. JPEG XL needs no new dependency: Pillow (natively
or with pillow_jxl), ffmpeg with libjxl, or cjxl; without one the
settings refuse jxl (400 jxl-unavailable) and GET says why. Encoder
processes are tracked, bounded and killed at shutdown.
* feat(thumbnails): every preview writer and lookup follows the thumbnail settings
Automatic previews (capture and first view), the thumbnail job, the
legacy gallery routes and the history route write <stem>_thumb.webp
(or .jxl/.jpg) with the configured quality and size limit, for images
and videos alike: video frames, ffmpeg's included (a PNG on stdout,
never upscaled), and the placeholder are encoded like images.
Listings, duplicates, enrichment, the autotag scan and the history
route show any preview the file owns, the current format first, so
older _thumb.png previews keep working until converted. A request for
an older name answers the current preview. Previews are served with
an explicit image/webp or image/jxl type and the same cache headers.
The plan and status count missing, outdated (another format) and
ready previews; the missing job never writes beside an outdated one.
Removing a file removes its previews in every format; the clear
removes .webp and .jxl too. Encoder processes are killed at shutdown.
A preview is only a placeholder: inspection refuses paths under
thumbnails/ and always reads the original, and the inspector cover
accepts the listing's preview in any format the file owns.
* feat(thumbnails): Convert outdated thumbnails as an Activity job
POST /jobs/thumbnails {mode: "convert"} starts "Convert outdated
thumbnails" under the thumbnail job's kind, guarded admission, worker
threads and cancel at a file boundary. For each file whose preview is
in another format, it writes the new preview from the source (atomic,
verified; a video with no decodable frame fails instead of getting the
placeholder), then removes that file's old-format previews in its
thumbnails folder. A failure keeps the old preview; orphan previews
and links are never touched. The final line reports converted, failed
and bytes before and after. The plan counts it with mode=convert; the
legacy gallery stream refuses to attach to a running conversion.
* feat(thumbnails): thumbnail settings in Appearance and Convert outdated in the workspace
System > Appearance gains a Thumbnails surface: format (WebP, JPEG XL,
JPEG), quality and size limit, saved to the server with the settings
revision. JPEG XL is offered only when the server has an encoder, else
the option is disabled with the server's reason (D130); a note says
when this browser cannot show JPEG XL. Saving a new format converts
nothing: the row then says previews keep their old format.
Thumbnail maintenance gains the Convert outdated scope and names the
server's settings instead of its own quality and size fields (D131).
Overview's thumbnail row reads N missing, M outdated, K failed and
offers Convert outdated while previews are outdated.
The workspace decodes a tiny JPEG XL once: where it cannot, cards, the
Library and the inspector cover load the original for a .jxl preview,
and any preview that fails to load falls back to the original. The
viewer and downloads keep using the original (regression test).
README explains enabling JPEG XL; TESTING.md records D130 and D131.
* test(thumbnails): coverage floors for the thumbnail modules and WebP in page workflows
The two new frontend modules enter the coverage baseline (98% lines);
other floors only rise. The visual fixture's preview is WebP and
Appearance has its Thumbnails surface in the System shell check.
* fix(thumbnails): say when the thumbnail settings could not be read
Like Appearance's other server rows, the Thumbnails rows show 'Server
settings could not be read.' while the read failed, instead of only
staying disabled. Firefox exposed the gap: its JPEG XL note arrived
first and changed the row notes the page workflow counted.
* test(thumbnails): accept the legacy preview's spelling a case-insensitive disk finds
On Windows and macOS the '.jpg' lookup finds an older release's
b_thumb.JPG first, so the listed URL spells '.jpg'; it names the same
file there. The test now accepts either spelling and checks the URL
serves that preview.
* fix(thumbnails): address the final review of the thumbnail format work
- ffmpeg writes JPEG XL with -update 1, so a % in a file name is never an
image-sequence pattern, and reads videos as file:<path>. The clear and a
startup sweep remove temp files that interrupted writes left behind.
- Convert outdated converts each group of same-stem files once, from the
first source in the R2 order. An old preview that is already gone counts
as removed, and each file's bytes count once. A current preview is only
kept if it decodes.
- An encoder that fails makes the write detect encoders again once. Another
encoder is used if one is found; with none, the file is written as WebP
and the availability Appearance shows is refreshed.
- Changing the thumbnail format, quality or size while Convert outdated runs
returns 409 busy. If the format still changes underneath, the job stops
without removing anything.
- Legacy preview spellings are matched case-insensitively, so X_thumb.JPG
counts as outdated.
- History's cover uses the same lookup as Images. It serves .jxl only when
Accept lists image/jxl, and otherwise the original or the placeholder.
- A video whose thumbnail the browser cannot show keeps its own element
(Images) or its label (Library).
* fix(thumbnails): keep the new preview, probe encoders only when gone, sweep reused PIDs
- Convert never removes a path that is the preview it just wrote. It
compares the device and inode, or the resolved and normcased paths when
the disk has no inode numbers. This covers x_thumb.JPG on a
case-insensitive disk.
- A JPEG XL encode failure re-runs encoder detection only when the
encoder's binary or module is gone. A "no encoder" result is checked
again on the settings read, on a jxl save, and after 60 s, so one slow
probe no longer turns JPEG XL off until a restart.
- The startup sweep now also removes temp files that carry this process's
PID but predate its start. Containers reuse PIDs.
* fix(history): let the cover ask for JPEG XL when this browser decodes it
Firefox can decode JPEG XL without listing image/jxl in Accept. When the
workspace's probe decodes JPEG XL, it adds jxl=1 to the History cover URL.
The server serves a .jxl preview when that flag is set or when Accept
lists image/jxl. In every other case it falls back as before. The flag
accepts only 0 or 1, and the response stays no-store.
* fix(thumbnails): keep going past the per-batch cap until every preview is made
The missing, rebuild and Convert outdated jobs stopped after
MAX_THUMBNAIL_FILES (10,000) files and said "N more on the next run", but
nothing ever started that next run. The cap now bounds one batch. After
each batch the job plans again, leaving out every file it already worked
on, and continues until nothing is left, the job is cancelled, or a batch
makes no progress. That last case stops on files that fail every time
instead of looping over them.
Activity progress counts against the overall total, and the final line and
the legacy stream's complete frame report every batch. Admission, worker
threads, cancellation and the #316 rules are unchanged. The Thumbnail
maintenance dialog now counts all the work instead of saying "more on the
next run".
|
||
|
Vito
|
8bf759d6f8 |
feat(system): keep the library in ComfyUI's user folder (#318)
* feat(system): resolve every library path in one place utils/library_location.py decides where the library lives: the PROMPT_MANAGER_DATA_DIR folder (validated: absolute, never inside a folder ComfyUI or Prompt Manager serves), an explicit database.default_path as before, a node-folder library that was not moved, else ComfyUI's user/__promptmanager. The database, diagnostics, config.json, logs and the LoRA image cache all ask it, so a new install never writes into the node folder and an existing one keeps every path until the user moves it. "Keep it here" is persisted as library.keep_in_node_folder. * feat(system): move the library as a verified Activity job POST /prompt_manager/system/library/move starts "Move library", an exclusive, non-cancellable job (database/library_move.py). It checks the target volume's free space, copies the database with SQLite's backup API from one read snapshot, verifies the copy (integrity check, every workflow document, the same rows in every table), copies the safety and pre-migration backups and config.json (owner-only, checked by size and SHA-256), then publishes under the old file's write lock and switches the live process (every PromptModel, the review and history observers, the API's plans and health report). The old prompts.db and config.json are renamed *.moved-to-<folder> only after the switch. A failure removes everything it wrote and leaves the old library in use; a marker in the new folder makes a restart after any crash pick the right library. GET /prompt_manager/system/library and the Overview's "location" say where the library lives and why; POST .../keep persists "Keep it here". * feat(system): show where the library lives and offer the move D129. System > Overview's Library health gains a "Library location" row (the database path, why it is there, the "Keep it here" choice and any PROMPT_MANAGER_DATA_DIR problem) with Move library for a node-folder library and Details otherwise. While a node-folder library is not kept there, the workspace shows the concept's .notice above every page: "Keep it here" saves the choice on the server and stops the notice; "Move library" opens the shared review (From, To, Size, what moves, what happens to the old files) and starts the exclusive job in Activity. The notice and the Overview read the location again when the job ends. The Overview comparison maps the inserted row; a new comparison holds the notice to the concept's Sources notice (markup, tokens, alignment, both themes and widths) with a mutation check. * feat(system): close the browser server's log files before its root goes The logs now live in the library's data folder, which for the browser test server is <root>/user/__promptmanager; Windows could not remove the temp root while the server still held prompt_manager.log open. The API test's node folder is spelled in lower case because public paths are case-folded on Windows. * feat(system): show the library location on the legacy admin page A line under the admin page's statistics names the database path and why it is there (ComfyUI's user folder, PROMPT_MANAGER_DATA_DIR, config.json, or the node folder with a pointer to Move library), read once from GET /prompt_manager/system/library. * feat(system): prove a fresh host keeps its library in user/__promptmanager The execution suite starts one pinned CPU ComfyUI with no configuration at all on an OS-chosen port: the library is created in user/__promptmanager (database and logs), nothing but byte code is added to the node folder, the location route reports the default folder, and ComfyUI's /userdata API serves none of it (traversal spellings, a listing, a system-user header). * feat(system): document where the library lives and how it moves README: where the library lives, the upgrade path (Move library / Keep it here), PROMPT_MANAGER_DATA_DIR, explicit paths unchanged, and the way back to an older version. TESTING.md: D129 and the library location contract (resolution order, move steps, crash windows, coded refusals, Windows handling). * wip(system): library move fix round in progress * fix(system): retry a fallback image link that races the library switch The image monitor's fallback and recent-prompt links went through PromptDatabase.link_image_to_prompt, which logs and returns 0 on any error, so survive_switch never saw a "database is locked" or closed connection during Move library and the link was lost in both files. The monitor now links through link_image_to_prompt_or_raise (one transaction per attempt); link_image_to_prompt keeps its log-and-return-0 contract for other callers. A link that returns 0 is no longer logged as "Successfully linked image 0". A switch that fails after repointing the models now closes the old file's detached connections instead of leaving them to GC: switch() adds them to a list the mover owns. The mocked retry test is replaced by a real-lock race asserting the image row lands in the new library only. * fix(system): say which source is the default and where a symlinked library lives Sources & scans: with no source configured, ComfyUI's output folder now reads "ComfyUI/output · watching · default" on its own row instead of a separate note line. GET /settings marks each effective source with in_comfyui (its public path is relative to the ComfyUI base) so only those get the "ComfyUI/" prefix; the scan still sends the server's public path. Library location: a library in the node folder reads through the node folder's unresolved place in custom_nodes, so a node folder symlinked to a checkout outside ComfyUI shows custom_nodes/promptmanager/prompts.db instead of prompts.db (System library row, Move review, admin Library line). Never absolute or "..", else the shared _public_path, which sources and jobs keep using unchanged. |
||
|
Vito
|
fc3266d455 |
fix(thumbnails): generate thumbnails automatically and use them everywhere (#316)
* fix(thumbnails): generate thumbnails automatically and use them everywhere New outputs never got a preview, so the workspace grid loaded full-size originals (844 of 19,907 images on a live install, exactly the newest). - Listings always give images and videos a thumbnail URL; a missing preview is made by the serve route on first view (bounded, single flight per file) and falls back to the original, remembering files that cannot be decoded. - The image monitor queues each settled output's preview on one background worker that follows Worker threads. - A scan that leaves 200 or more previews missing starts the visible thumbnail job; fewer go through the worker. - Automatic previews wait for a restore, a duplicate removal, a sources change, the gallery's clear and exclusive jobs. - The inspector cover uses the preview; the Overview row counts previews that failed. * fix(thumbnails): keep the video fallback tested and stop the preview worker before cleanup A video listed without a preview still renders its own element and is released on refresh. Test teardowns and the browser reset stop the automatic preview worker before the temp tree goes, so no preview lands after cleanup (Windows). TESTING.md documents automatic previews. * fix(thumbnails): admit previews on the loop and never send a video to an img Final review fixes for automatic previews: - A video preview that cannot be made is the placeholder frame (no-store), never the video; OpenCV failures fall through to ffmpeg and the placeholder. Cards and the inspector cover swap back to the video element when the preview image fails. - View requests are admitted on the event loop (3 writers, one flight per file, at most 64 waiting; past that the fallback at once). Writes run on a small dedicated pool, never the default executor. The history route shares the admission, flight key and link checks. - Drive, UNC, empty and dot segments are refused as text; the folder listing runs inside the admission. - Holds are checked after the slot and again before the rename; only decode failures are remembered. - Bounded, logged shutdown: ffmpeg is killed, no worker restarts after cleanup, and a reload stops the old worker. - The listing says whether the preview file exists (thumbnail_ready) for the legacy gallery's dot; media in a nested thumbnails folder keeps its original; file names the server quotes match in the inspector. * fix(thumbnails): cover the image fallback card and document the admission An image listed without a preview (a nested thumbnails folder) still shows its original; TESTING.md describes the view admission, the video placeholder, the holds before the rename and the bounded shutdown. * fix(thumbnails): let an idle preview worker leave and fit case-insensitive filesystems The capture worker exits once its queue is empty, so no thread outlives its work; the next output starts a fresh one. The mixed-case source test expects no folder listing where the exact name already resolves (macOS, Windows). * fix(thumbnails): answer shared flights with the outcome only and restart a failed worker The history and serve routes share a flight key, so a flight now answers only the ensure outcome whichever route started it; the serve route derives the preview path and, for a fallback, looks its source up again on the view pool. A view joining a history flight no longer gets a 500 or an image answered with the video placeholder. The capture worker clears its running state however its loop ends, and a worker that cannot start leaves the file queued for the next enqueue. |
||
|
Vito
|
b4c7a15045 |
feat(review): Needs review page, live queue and concept comparison (#288)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* feat(review): bind queue cursors to the storage epoch only
A decision or a new item (the monitor records one per generation) no
longer makes an open cursor stale: the (created_at, id) keyset stays
ordered when items leave or are appended. A restore still answers
storage-changed and an unverifiable cursor review-conflict.
* feat(review): Needs review page, SAVED VIEWS row and Activity hook
The concept's Metadata review page at #/review: the notice with the
exact pending count, the queue's media cards (shared with Images through
image-card.mjs), and the Resolve the association panel with the sampler
context, the chosen path's prompt blocks, Leave for later and Apply
association. Apply takes a fresh capture; refusals keep the selection
and explain inline; an unanswered decision is resent once. SAVED VIEWS
lists Needs review after Favorites with its count (retires D50), and an
ended scan that recorded review items offers Review metadata in Activity
(D87). Queue items carry the card's media url.
* feat(review): re-read an open Needs review page when its queue changes
An open Needs review page now follows the queue without polling. A new
server-sent event stream, GET /prompt_manager/review/events, sends
"ready" when it opens and "change" after every committed queue change:
an item the scan or the monitor recorded, a spelling refresh, a decision
from any tab, and a restore. database/review_changes.py notifies after
the commit from whichever thread wrote it. The page re-reads its first
page on each event, keeps the open item, its choice and an unanswered
decision, and re-reads the SAVED VIEWS count with it. When the open item
left the queue, the next one opens with a note. A keepalive comment
every 25 s lets the server drop a closed tab; shutdown ends every stream.
Leaving the page mid-decision now releases that decision's capture.
D-NEW-F1 is renumbered D119 in markers and TESTING.md.
* test(review): Needs review page workflows
The review fixture now seeds a real monitor-fallback conflict (the
monitor pipeline links a new output to the most recent prompt, then
queues it), a limited item and a file under no current root, and can add
an output through the monitor or leave one for a metadata scan.
New workflows: Apply links the chosen pair to that file only; a real
association conflict leaves the links unchanged and explains; a limited
item and an unavailable file can only be left unresolved; a tag save
leaves the item pending; keyboard focus, labels and touch targets; an
open page shows items the monitor and a scan add and a decision made in
another tab; leaving the page releases captures, the change stream and
requests. The faked 409 conflict test is replaced by the real one.
* test(review): cover the card Tag button and raise the review floors
The card's Tag on Needs review now has a workflow, and the page's queue
change hook has no unused default. review.mjs and review-state.mjs
floors rise to their measured coverage; no other floor changes.
* feat(review): keep the loaded depth on live re-reads and read once per visit
A queue change re-read only the first 40 items, so the pages Load more had
shown vanished and the page jumped. Each re-read now reads page after page
until it holds as many items as were shown (a restore met after the first
page reads again from the start), so the cards and the scroll position stay.
The stream's ready made every visit read the queue twice. List reads now
answer the queue's version (library_epoch, review_revision) and ready names
the version read after subscribing; the page re-reads on ready only when the
queue it shows is older. The SAVED VIEWS count follows the page's total, so a
change costs no count read. A stream counts as open only once ready is sent.
* test(review): compare Needs review with the approved concept
review-concept.spec.mjs measures the whole Metadata review page as a tree
in the concept and in production, on the review fixture's new
concept-shaped queue (the concept's first three image names and sizes and
its first prompt). mapConceptReview applies only D29, D119, D3, production's
hooks and accessible names and sample data, and the page then matches at
1600x1100 and 390x844 in both themes. The gallery layout follows the
concept at every breakpoint, the SAVED VIEWS rows match with Needs review
after Favorites, the queue's Tag image dialog matches the needs-review
reference, and a dropped notice border or reordered actions fail.
The comparison found one gap: an unavailable file's card lost its media
area. Its empty media button now keeps the concept's 1.33 ratio (D119).
TESTING.md records the comparison, the extended D119 and a summary of the
image-tagging slice (routes, S1, capture limits, the record-changing job,
the no-download rule, the restart note). A ready whose version cannot be
read re-reads the queue (covered by a page test).
* test(review): give the multi-route shell comparison a fitting time budget
The dispatched CI run failed "shell geometry, tokens and controls match the
concept 390x844 light" on WebKit with a 30 s test timeout (no assertion
failed). Each of these tests makes twelve page loads with captures, hovers
and focus checks: 15-19 s on a CI WebKit runner on main, 22-31 s on this
run's loaded shard. With 2 CPUs and 3 workers it fails locally the same way
(6 of 12 timed out); with a 90 s budget, as the other multi-state concept
comparisons have, all 12 pass.
* fix(review): coalesce queue change events and answer queue thumbnails
- The change stream sends the first notice at once and collapses the
notices of the next 0.4 s into one trailing change, so a busy scan costs
an open page one re-read per 0.4 s at most (COALESCE_SECONDS).
- Queue items answer thumbnail_url, the same URL the Images listing builds
(one shared helper, media_thumbnail_url), or null without a thumbnail.
- The "duplicate and refused decision send nothing" test spies on
notify_review_change instead of reading the next event.
* fix(review): keep the live queue right through applies, Load more and lost streams
- A change heard while a decision is in flight is owed and read once the
decision is refused or unanswered (I1).
- Load more has its own read: during a re-read it waits and continues from
the re-read queue; a re-read started during Load more takes the click
over, and a failed re-read leaves Load more clickable (I2, M7).
- A stream the browser gave up on (503, proxy error) reopens after 3 s,
doubling to 30 s, with a quiet "Live updates are paused" line while it is
down (I3, D-NEW-F2).
- A hidden tab closes its stream and a visible one reopens it; the ready
version keeps that to at most one read (I4).
- Apply fences on the detail's epoch and a re-read under another epoch reads
the open item again (M3); leaving during the observe releases its capture
(M4); an unanswered decision that landed finishes as recorded (M5); cards
follow a re-read's url, target and availability and show thumbnails
(M6, M10).
- The unreadable-ready test asserts exactly one re-read (M1); the panel's
long-file-name wrap carries its D119 marker and TESTING.md entry (M8).
* docs(review): number the live-updates-paused deviation D120
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(activity): drop Cancel job from a job that had ended before Activity opened
Since #289 the Activity list removed an item's actions only when they were
isConnected. On the first render the item is not in the page yet, so a job
that had already ended when Activity opened kept its "Cancel job" (and a
running job's progress was moved after its actions, then back). The list
now asks parentNode instead. The review spec "a scan job with review items
offers Review metadata in Activity" failed on this before the change
(Expected 0 buttons on a finished job, Received 1) and passes after it.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* feat(review): open Needs review from the System Overview review row
Review metadata on the Overview's review row now opens #/review, the
concept's go-review, instead of staying aria-disabled.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): announce items that leave the queue, and seed only queueable files
#284 now queues a file only when its final outcome is ambiguous and unlinked,
and it removes the pending items of files that are now linked. That removal
is a queue change, so it calls notify_review_change after its commit, like
recording and decisions: an open Needs review page re-reads its queue, and
its count follows.
The review browser fixture wrote graphs that the scan's fallback links, and
it produced the conflict item through the monitor's fallback link. Neither is
queued any more. Its files now carry ComfyUI's Infinity stamp, which the
legacy parser cannot read, so they stay unlinked with the same two
candidates. The conflict item is queued first and its file is linked to
another prompt afterwards. "monitor" additions go through
record_review_file from a worker thread.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(review): seed the limited item from a graph with more candidates than stored
An A1111 block with two negative lines offers no candidate to choose, so
Needs review no longer queues it (#284). The fixture now records a graph of
17 samplers, which the recorder stores limited.
|
||
|
Vito
|
3f11ea6168 |
feat(annotations): Needs review queue backend (#284)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(scan): pin only the per-source scope keys beside the review counters
* test(system): match the scan scope keys beside the review counters
|
||
|
Vito
|
c86739178d |
feat(system): System page shell, review dialog and Overview (#298)
* feat(system): read-only health check and truthful overview counts POST /prompt_manager/system/health runs the health check off-loop as a read registry job of kind "health" (one at a time, beside a scan, not cancellable, no file outcomes) and keeps the last report; GET returns it or null. The check reads one snapshot on its own mode=ro connection, closed in finally: integrity_check(100), foreign_key_check, stored hashes against utils.hashing, duplicate pair rows, the legacy consistency queries (now shared with check_consistency), and image links whose file is gone, counted only under readable sources. An unavailable source, or the absent Needs-review queue, makes its check unavailable and the report incomplete instead of producing a count. GET /prompt_manager/system/overview returns Library-eligible positive prompts and active negative variants, indexed images, the last health summary or null, backup null, the Needs-review count null, the effective sources with whether they were auto-detected, and the thumbnail plan's missing count. GET /prompt_manager/system/statistics lists runs and average rating per category of Library parents, with the /stats totals. * feat(system): native System route, drafts and shared review dialog - #/system/<section> renders the concept's System shell: section nav, settings layout and Save row with truthful status (D92); each section not delivered yet hands off to its legacy tools (D94). Overview's System card opens #/system/overview (Q12). - A module-level draft store saves browser, then host (expected_revision), then runtime; failed domains stay dirty, a stale revision is a conflict. - System row builders (setting, operation, section nav) on the scope. - openReviewDialog and backupNotice extracted from the scan review, which is now a thin caller with identical markup. - Overview images gain a "rated" count (images whose linked prompt is rated); the missing-files health check reports per source, so partial availability is visible. * feat(system): native Overview with health report and statistics System > Overview renders the concept's metrics and its Library health and Workspace preferences surfaces from GET /system/overview: real counts, "Not checked" until a health check ran, no invented review count, backup or thumbnail status (D93). Explore opens "Usage statistics" with the real runs and ratings per category and the Library totals (D96); the statistics route now also returns rated_images. Maintenance shows the concept's notice with Run health checks above its hand-off: the read-only check runs as a registry read job and opens the "Health report" with a chip per real result, missing files per source (D95). A load failure shows Retry and no zeros; late answers never repaint a newer section. Adds the System concept comparison (shell, Overview, both dialogs, the notice, mutation checks), the System workflow spec and its fixture hooks, and the TESTING.md "Native System" section. * test(system): coverage floors for the native Overview Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes. * fix(system): keep health reports per database and off the event loop - A restore that replaces the database forgets the stored health report, and a check that started before the swap is not kept (database generation stamped at start). - The Overview no longer walks every source for the missing-thumbnail count; the page reads /images/thumbnails/status on its own. - Image sources for the health check and the Overview resolve in the executor; a failure to start the check answers the coded envelope. - Any exception fails only its own check; missing files reads its rows in the snapshot and stats the files after it ended. - The read-only URI keeps Windows UNC paths (empty authority). - An integrity count at SQLite's limit is flagged as capped. * fix(system): thumbnail row read, tracked health job and guarded host save - The Overview's thumbnail row is its own read of the thumbnail status: "Checking thumbnails…" until it answers, unavailable when it fails. - The health job is tracked in the Activity store (new store.track) and followed through its event stream instead of polling every 150 ms. - A host save before a read returned a revision fails without a request; System reads the host values on mount and after a conflict. - "100+ problems found" when the integrity count is capped. - D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone. - The concept spec builds the health report from the concept's own rows and asserts the full unmapped path list; surface tops pin the 18px offset. - TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI and the new behavior. Coverage floors raised for the touched files. * fix(system): make two health tests portable to Windows - Windows forbids "?" in file names, so the URI-character open test uses "#", "%" and a space there. - ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of raising, so the bad-path test raises as POSIX does to reach the missing-files failure path on every platform. |
||
|
Vito
|
4f1c6e236b |
test(browser): report a slow test reset with every thread's stack (#299)
* test(browser): report a slow test reset with every thread's stack On #291 (run 39904288262, WebKit shard 2/2) one /__test__/reset {chronological_history} waited 16 s on its worker's server, with nothing on the server's stderr. The cause did not reproduce: the same worker's file sequence (library, library editor, tagging lifecycle, visual) at 2 CPUs three times, and WebKit shard 2/2 at 2 CPUs, kept every reset under 2 s. A reset still running after 2 s now reports, while it waits, every thread's stack (faulthandler), the default executor's threads and queue, the open connections and WAL size, and on Linux each thread's scheduler state and kernel wait channel. When it ends it reports the total and how long each executor call queued and ran. The lines carry a "pm-slow-reset| " prefix and the worker fixture copies them to the job log, so the next one shows up even in a test that passes. The report never reads another thread's connection. Reading even in_transaction on a connection busy in SQLite blocks on its mutex with the GIL held: a first version of this report froze the whole server until the 30 s busy timeout. A test holds a real write lock behind a reset and checks the server keeps answering while it waits. * test(browser): name each thread in the slow-reset stacks Before Python 3.14 faulthandler prints thread ids only, so the report now lists which name each id belongs to. CI on 3.10 to 3.13 failed the unit test that looks for the blocked thread by name. * test(browser): keep the slow-reset summary on a coarse clock On Windows the report timer can fire a few milliseconds short of the threshold, and the summary was then dropped. A reset that was reported now always gets its summary. The server test uses the plain reset, which the Windows jobs can seed, and puts the server stderr in its failure messages. |
||
|
Vito
|
08151ca93d |
test(library): read one "now" in the fixture and the page across midnight (#287)
* test(library): read one "now" in the fixture and the page across midnight The concept fixture recorded each prompt's last use with the database's wall clock at seeding, while the page computed "today" from its own clock at render. A run seeded at 23:59:5x UTC and rendered after midnight read "Used yesterday" against the concept's "Used today" (PR #284, frontend coverage shard 2/3). The concept fixture now takes one instant (optional "now" through the concept reset hook, the wall clock otherwise) for every last use, mtime and run time it writes and returns it as manifest.now; seedConceptAt fixes the browser's Date at that instant. The Library and Images concept specs seed through it, and the two workspace specs that assert "Used today" and the "Today" caption fix the page clock at the time they patch or seed. A new test starts the browser at 23:59:58 UTC, seeds at that instant and passes midnight before either document opens. * test(activity): judge the lost-server backoff by its gaps, not a timed window "lost server clears the pill and marks jobs ended" counted failed job-list re-reads in a 3.5 s waitForTimeout window that started once the test saw the ended job. Re-reads back off 1 s, 2 s, then 4 s, so the third lands 4 s after the second. A runner that noticed the ended job more than 0.5 s late moved the window over the third read (firefox shard 1/2 on #287: 3 reads, expected at most 2). The test now waits for the third failed read and asserts the gaps between reads: at least 2 s, then at least 4 s, with 100 ms allowed for the request being reported late. * test(activity): replace the spec's remaining timer waits with gaps and a run-out clock Three checks in workspace-activity.spec.mjs waited on wall-clock timers: - "lost server": waitForTimeout(2500), then at most 2 list reads. The read after the one that found the restart must be the periodic poll, so the test now waits for it and asserts the gap is at least POLL_MS (less 100 ms for late request reporting). Any extra read lands in that gap. - "200-row cap note" (review M12): waitForTimeout(1500), then no detail read. The page's timers are now Playwright's clock. The test runs one read-throttle interval out, so any read the counter update scheduled is sent before the log opens. Once the log's read has answered, exactly one read must be listed. - "busy start" (review M14): waitForTimeout(7000), then the link toast is still visible. The test now runs the page's clock 7 s on, which fires a 6.5 s clear however slowly the runner got there. RED, each against a temporary product change: POLL_MS 3000 (gap 3004 ms, expected at least 14900), detail re-read on new log lines while the log is closed (an unexpected read), LINK_TOAST_MS 6500 (toast link gone), and BACKOFF_FIRST_MS 200 for the earlier backoff check (gap 407 ms, expected at least 1900). |
||
|
Vito
|
5123654b0d |
fix(admin): start no list or stats read while the admin page leaves (#290)
On WebKit, a reload cancels the page's reads between beforeunload and
pagehide. After an in-place edit, finishEdit awaits loadStatistics, which
swallows its cancelled read, and then calls refreshList. That /recent read
starts while the page is leaving, and WebKit fails it with "Fetch API cannot
load ... due to access control checks." Playwright reports that console
error as a page error. This failed workflows.spec on WebKit in CI.
It is not an unhandled rejection: the page catches the failure. The read
should never start.
The admin page now has a lifetime (AbortController) that ends on
beforeunload, unless a running scan asks to stay, and on pagehide.
- ApiClient.latestOnly({lifetime}) starts no request once the lifetime has
ended, aborts the one in flight, and resolves both to STALE.
- loadStatistics skips and silences a read that ended with the page.
- Real errors still propagate and show as before.
- A page restored from the back/forward cache, or one whose navigation never
left, gets a new lifetime on pageshow or its next pointer or key input.
Tests:
- The test server can hold a real request at the server (request_hold.py),
because a page.route hold is cancelled differently by WebKit.
- A WebKit regression test holds /prompt_manager/stats after an in-place
edit and reloads. It failed 5/5 before the fix with the CI message and
passes 5/5 after it.
- A test checks that a called-off leave reads the list again.
- api-client unit tests cover the lifetime.
|
||
|
Vito
|
e29ff4d4b2 |
feat(inspect): the Inspect page for local files (#280)
* feat(inspect): local adapter seams in the shared inspector and viewer
The inspection loader calls adapter.read when an adapter reads by itself, so a
file chosen in this browser never reaches the API. The full inspector shows the
D66 local provenance (no Path row, no Show in folder, local association line),
and the viewer uses adapter.mediaUrl exactly, so a blob: URL is never given the
_pm_viewer cache-buster and Retry media reuses it. Rooted and recorded-history
adapters keep their api.get paths and cache-busted originals.
* feat(inspect): owned local selection session
createLocalSession owns one chosen File at a time: it classifies the bytes,
creates the blob: URL only for a classified image or video, and reads it
through a real inspection controller with the local adapter. Replacement,
dispose, parent abort and pagehide each release the URL, the handle and the
record once; a released handle (and every copy) is foreign to the adapter, and
every await is fenced by a generation counter so a replaced read never
publishes. Empty, multiple and directory drops keep the previous selection.
Baseline entry for local-session.mjs; branch floors raised for the touched
inspection-state.mjs and local-inspection.mjs.
* feat(inspect): native Inspect a file page
#/inspect is now the concept's "Inspect a file" page instead of the legacy
handoff. One chosen or dropped image or video previews inline, and the one
heading action, "Open file metadata", opens the shared viewer with the full
inspector on a record read entirely in this browser. Nothing is uploaded,
imported or stored; only an explicit Send crosses the host bridge.
- inspect.mjs mounts the concept DOM (surface, file-drop, #local-preview)
under the shell heading, owns one local session and the shared viewer and
prompt sender, and releases media before the session revokes its URL on
replacement, route change, page leave and dispose (D60-D65).
- A file that cannot be read after choosing opens the inspector's
unreadable state when the browser declares a supported type, instead of
the "unsupported" copy; the local adapter maps a browser read failure to
an unreadable record.
- The route is native; Overview keeps its card to the legacy metadata page,
which is unchanged.
- workspace-inspect.spec.mjs covers the chooser and viewer rail against the
Images rail for the same bytes, drop, cancel and reselect, unsupported and
unreadable files, video, URL ownership, mid-read replacement, exact send
and the no-request/no-storage rule.
* test(inspect): concept parity and managed-host send for local files
Compare the native Inspect a file page with the immutable concept's
#metadata at 1600x1100 and 390x844 in both themes (initial, selected,
viewer-open), mapping D60-D66 and pinning the concept's CSS values and
the chooser's displayed file name, with a mutation check. Send a local
PNG (explicitly empty negative) and core-tags.webm to a real Prompt
Manager (+/-) node across Classic and Nodes 2.0 switches and reloads;
a connected input rejects the send unchanged. Document D60-D66, the
legacy-floor dispositions and the commands in TESTING.md.
* fix(inspect): reset a released preview video in the next task
On WebKit (GStreamer) a video replaced while its pipeline was still
starting blocked the page: pause, src removal and load() in one task
never returned, so the next chooser selection hung (CI page workflows
webkit shard 2/2, "URLs and media end with their owning session").
The src is still dropped before the URL is revoked; only the load()
reset moves to the next task. A fresh-page regression replaces the
first video as soon as it shows, fifteen times.
* fix(inspect): never reset a preview video that is still starting
Deferring load() to the next task was not enough: CI on
|
||
|
Vito
|
195a1d33ac |
feat(annotations): Tag image dialog, card Tag button and inspector tag chips (#281)
* feat(annotations): image tagging controller with staged suggestions Add the UI-agnostic Tag image dialog controller (#249 C1): observe the file into a capture, prefill the saved tags and the server's tagging defaults, stage the top suggestions as removable chips, and save manual tags plus the remaining chips with case-insensitive dedupe (manual spelling first). Every read is serial-guarded; saves are sent once and a lost answer is resent identically only by Retry. Conflict, storage, file and capture outcomes keep the draft and gate Save on a reopen. ApiError now carries a validated copy of a tag-conflict body's current tag set so the dialog can show it. * feat(annotations): Tag image dialog beside Info on every image card Every Images and Library-embedded gallery card shows the concept's Tag after Info. Tag opens the concept's Tag image dialog (image-tagging.mjs over the C1 controller) in the shared modal shell with the sticky tagging-modal footer: capture preview, manual tags, the four legacy engines with real defaults, explicit Generate, removable suggestion chips (D83), legacy model labels (D84), a Suggestions count (D91) and Retry where the controller allows it. The capture is released on close, route change and pagehide. The Tags page gains the concept's Auto-tag images for the image selected in Images (aria-disabled without one, D115) and real Images counts that follow every save. Retires D9, #245 D2 and #245 D3. The toast now uses the mock's copy verbatim ("tags" even for one), C1's D60/D61/D68 markers are renumbered to D83/D84/D91, and at extreme zoom a card's info row stacks (and wraps Info and Tag) when its row is narrower than the narrowest bounded card, list rows included (D31). * feat(annotations): show image tags in the inspector and viewer The inspector parses the B3 envelope's image_tags facet (strictly: exact keys, current/none/unavailable, at most 100 vocabulary names) and draws one chip row after the prompt blocks: the file's current image tags first, then D21's linked tags not already shown (case-insensitive). In the Images panel image tags are Tags filter buttons and linked tags keep D21's rule; in the viewer and wherever no Tags filter exists every chip is plain. none and unavailable add no chips (D88, plan D65; D21 amended). The tagging fixture reset now drops leftover captures, so a long run of tagging tests in one worker stays below the capture store's limit. * fix(annotations): re-read a tagged file in place after a save After a tag save the selected file's inspection is re-read quietly: the panel and the viewer keep the current details until the new answer lands, with no "Loading file details…" in between. While a Tags filter or a search is active, Images re-reads its loaded range in place (same query, from the first page up to the loaded count, published once), keeping the cards, the scroll position and the selection while the selected file still matches; when it no longer matches, the panel follows the collection. * fix(annotations): give suggestion chips the concept chip corner The D83 chips took the later `.tag-suggestion` row rule's 5px corner; the concept `.chip` has 4px. The concept comparison of the Tag dialog found it. * test(annotations): compare the Tag dialog with the approved concept The Tag image dialog's inventory (order, text, field attributes, tokens, preview box, chips, status, sticky actions) must equal the concept's at 1600x1100 and 390x844, dark and light, opened, generated with WD14 and with a language model, for an empty result and in the review scope, and the card's Info and Tag group must equal the concept's. Production runs on the concept fixture with the concept image's tags saved and the fake model answering the concept's own sample; only D29, D83, D84 and D91 are mapped, and a mutation check shows the comparison notices a change. The D3 footer check again asserts a strictly shorter footer, with short seedless subtitles that stay on one line beside Info and Tag. * fix(annotations): final review round for the Tag image dialog - Option fields show the server defaults that Generate sends: the controller bumps an options revision on open and on a model change, and the panel re-renders only then (I1). - A tag conflict shows the current set at once ("Saved now: ...") and puts it ahead of the draft in the manual field, so the next Save sends what is shown (I2). - The opening sentence is the concept's on-screen copy; "Remove anything you do not want to add." leads the chips only while chips exist; the comparison's opened mapping is gone (I3). - ApiClient gains an additive keepalive option; the capture release and the Save PUT use it, and pagehide closes the dialog (I4). - Closing during a save keeps the capture until the PUT settles, then releases it and still reports the outcome (toast, page refresh) (I5). - A refresh asked during a page load is pending and runs once it settles; re-reads use pages up to 200 files; a failed re-read toasts; Load more waits for a refresh (I6, M2, M3). - The selection bar drops checked files a refreshed result no longer lists (I7). - D116 records Retry; after a failed Generate it takes Generate's own path, and after a changed storage, file or expired capture it observes again keeping the draft (I9, M4, M5). - A save made from Tags marks a filtered Images visit stale (M6). - images.mjs drops under 800 lines: the tagging wiring moves to images-tagging.mjs and the favorites star to images-favorites.mjs (unchanged), and reset() reuses dropSelection() (M1). - Restore the activity-scan.mjs coverage floors; drop the duplicate paired captures test (I8, M9). * fix(annotations): coverage floors for the Tag dialog round Floors for the new images-favorites.mjs and images-tagging.mjs, and the measured raises of image-tagging-state.mjs and image-tagging.mjs. activity-scan.mjs keeps main's floors. |
||
|
Vito
|
1b8fbdbb75 |
feat(annotations): image tag routes, model gate and scored suggestions (#278)
* feat(annotations): add scoped observe, capture preview and tag save routes
POST /prompt_manager/image-annotations/observe captures the file named by
the listing identity {root_index, root_scope, path} (scope checked against
the current roots before any file access) and answers the saved tag set of
exactly those bytes, refreshing a matching record's verified fingerprint.
GET/DELETE .../capture/{token} serve the private copy (no-store, nosniff,
pinned with acquire/release_use while it streams) and release it.
PUT .../tags re-hashes the current file and writes only when it still
holds the captured bytes, under the storage epoch and tag revision
(file-changed, storage-changed, tag-conflict, tag-limit, replayed).
Bodies are capped at 64 KiB with unknown fields refused; envelopes never
carry server paths or locators. Restores invalidate every capture and the
host's cleanup removes the capture directory.
* feat(autotag): scored suggestions with a configurable count behind a shared model gate
- py/model_gate.py: one process-wide, non-blocking gate for loading,
swapping, running, downloading and unloading a tagging model. Worker
calls run shielded; a release requested while one still runs takes
effect only when it exits (client disconnect keeps the gate).
- Legacy /autotag/single, /start, /download/{type} and /unload hold the
gate and answer 409 {code: "model-busy"} while another operation holds
it; their bodies are otherwise unchanged. /autotag/models gains
suggestion_limit.
- AutoTagService.generate_tag_suggestions: WD14 (name, probability) best
first, GGUF/HF (name, None) in output order, first `limit` distinct
case-insensitively; generate_tags keeps its return; the image file is
closed before inference returns.
- POST /prompt_manager/image-annotations/suggest runs one engine on the
pinned capture, with strict per-engine options, never downloads
(409 model-unavailable) and never writes.
- config.json autotag.defaults.suggestion_limit, integer 1..100, default
10, validated on load.
- Capture probe decides Generate from the image header alone, so prompt
metadata over the inspection budgets no longer blocks it.
* feat(images): match image tags in the Tags filter, search and inspector
The workspace Images listing now admits a file under a Tags filter when
every requested tag is carried by its linked positive or its current image
tag set, and q also matches image tag names (case-folded substring). Image
tags count only while a byte record is verified at the file's present
(size, mtime_ns) (spec amendment S1): the tagged records are streamed
once, only listed files carrying a filter tag or q hit are stat'ed, and
their linked positives are read through the file_path index so tags,
category, rating and the positive count combine.
The rooted inspector envelope gains a validated image_tags facet
(current / none / unavailable) read at the inspection's own stable stat.
A failed image-tag read fails a Tags filter like a failed link read and
degrades a q search to no image tags; plain listings read none. Legacy
gallery and search responses are unchanged.
* fix(annotations): answer a fixed coded message when the filtered image list fails
A failed prompt-link or image-tag read under a Tags, category or rating
filter returned the raw exception text (SQL or database messages) in the
500 body. The workspace listing now answers code listing-failed with a
fixed sentence and logs the traceback. The legacy output listing and the
inspector image-tag read now log with exc_info as well.
* fix(annotations): drop suggestion names that Save would reject
A JoyCaption part over 50 characters or holding a newline came back as a
preselected chip, so the first Save of the dialog failed with a 400 and
nothing was stored. generate_tag_suggestions now applies the vocabulary
tag-name rules before the case-insensitive dedupe and the limit count, so
the requested number of valid chips still comes back.
* fix(annotations): stop changing the process warnings filters during capture
header_dimensions wrapped Image.open in warnings.catch_warnings with an
error filter for DecompressionBombWarning. catch_warnings is not
thread-safe, so two overlapping captures could leave that error filter
installed for every later Pillow open in the ComfyUI process. Only
Pillow's DecompressionBombError is now mapped to HeaderTooLarge; a size
that only warns is refused by the existing inference pixel cap.
* fix(annotations): free pins on cancellation, serve preview ranges, code every storage error
- A preview cancelled while its file opens closes the handle and only
then ends the pin, so a doomed capture is never unlinked while open.
- An observe cancelled while capturing or reading releases the capture
instead of leaving it until the idle expiry.
- The model gate takes an on_skipped callback that runs once when the
work never starts; Generate passes the capture's release_use, so an
executor refusal no longer leaks a pin against the capture budget.
- The preview answers a single byte range with 206 and Content-Range
(416 for an unsatisfiable or multi-range header), still streamed inside
the pin, so Safari and WebKit can play video captures.
- _coded maps tag-limit (400), metadata-conflict and vocabulary-conflict,
and falls back to 409 with a fixed sentence for any other code instead
of raising KeyError.
* fix(annotations): deny case and Unicode spellings of a listed file
On macOS (case- and normalization-insensitive APFS) a crafted target such
as A.PNG or an NFD name opened the listed a.png or NFC file while
realpath kept the requested spelling, so the tags were stored under a
second locator the Images filter never shows. Capture and verify now
require each path part to be spelled as its directory lists it and deny
any other case or Unicode spelling as an alias (409 source-changed, like
a symlink alias). A missing file with no other spelling keeps its usual
outcome.
* test(annotations): one shared folder_paths stub for the API test modules
test_api_images and test_api_prompts installed a MagicMock folder_paths,
so whenever one of them was imported before test_api_autotag the later
patch("folder_paths.get_output_directory") in test_api_history_outputs
left a mock attribute behind and two test_diagnostics_config tests
failed. tests/host_stubs.py now installs one real-module stub, used by
test_api_images, test_api_prompts and test_api_autotag (and through
test_api_images by the annotation modules, whose copy is gone), so the
modules pass in either import order.
* fix(annotations): derive the suggestion cap from the image tag set cap
py/autotag.py now takes MAX_SUGGESTION_LIMIT from database.annotations
MAX_IMAGE_TAGS. The two copies that cannot import it (py/config.py loads
before the database, utils/inspection_response.py stays storage-free) are
pinned equal by a test, so the 100 limits cannot drift apart.
* docs(testing): record capture aliases, preview ranges and the PR B follow-ups
TESTING.md now states that case and Unicode spellings are aliases, that
the preview serves byte ranges inside the capture pin, where the
cancellation tests live, and two tracked follow-ups: the legacy batch's
ungated cleanup unload under its own last inference, and the three files
over the 800-line ceiling (split py/autotag.py first).
|
||
|
Vito
|
b7cf210f0f |
feat(annotations): image tag storage, vocabulary participation and capture store (#276)
* feat(annotations): add image annotation schema with restore validation
Install the image annotation group beside image_favorites and saved_views:
media_annotation_metadata (one row, schema version 1), media_annotation_files
(one private locator per physical file), media_annotation_bytes (one record
per observed SHA-256 with the verified size and mtime_ns) and media_image_tags
(the ordered tag set of a byte record, joined into the global tags registry).
The group migrates in a SAVEPOINT that validates before and after creating
it, with no backfill from prompt or pair tags and no filesystem probe.
annotation_schema_error reads only: it compares the catalog objects after
whitespace compaction, the columns, foreign keys and indexes, and the
persisted values (UUIDs, hex digests, bounds, the 4096-character locator cap
measured past embedded NULs, exactly one metadata row). verify_database_file
checks it with allow_missing=not require_library_state, so old backups
restore and a migrated restore stage must carry it.
Tests pin the exact DDL, a fresh install and idempotent reopen, a current
database without the group gaining it with legacy rows untouched, every
partial or malformed shape rejected without repair, install failure rolling
back and retrying, a WAL backup/restore round trip, staged-failure
immutability of source, live and safety files with every handle closed, and
the require_library_state gate.
* feat(annotations): persist per-image tag sets in the shared vocabulary
Add database/annotations.py (AnnotationOperations, mixed into
PromptDatabase): read_image_tags, replace_image_tags and add_image_tags
run in one _pair_transaction fenced by the storage epoch, own the byte
record by exact locator and digest, always supply UUID ids, rewrite the
ordered junction and bump tag_revision plus image_index_revision.
Revision mismatches raise AnnotationCommandError("tag-conflict") with
the current set; the identical set at the current or previous revision
is replayed without writes; a union over 100 names is "tag-limit".
normalize_image_tags validates each name with validate_vocabulary_tag_name
and folds case-insensitive repeats keeping the first spelling.
refresh_image_fingerprint, current_image_tags (chunked, fingerprint
matched, newest verification wins) and iter_image_tag_records serve the
S1 cheap reads.
Vocabulary participation: _move_tag_links also carries media_image_tags,
legacy merge_tags moves image-only sources, tag rows gain image_count
(files with any byte record carrying the tag), impact gains images and
rename/delete receipts gain images_changed. vocabulary-state.mjs accepts
the optional counts and names images in the delete text.
Browser fixtures clear the three annotation data tables on reset (never
the metadata row); TESTING.md documents the schema beside favorites.
* feat(annotations): add bounded private media capture store
Add utils/media_capture.py: CaptureStore copies one rooted media file
into an owner-only prompt-manager-captures-* temp directory, streaming
it in 1 MiB chunks while computing the whole-file SHA-256, and compares
the opened descriptor with the path (dev, ino where meaningful, size,
mtime_ns, ctime_ns) before and after the copy; any disagreement raises
CaptureChanged and removes the copy. Symlinked files or directories,
root escapes, non-media names and directories raise CaptureDenied; the
locator is normcase(join(realpath(root), relative)), the listing's
identity, so a symlinked root matches it.
Limits are the plan's: 256 MiB per file on stat and stream
(CaptureTooLarge), 8 completed captures, 512 MiB reserved and 2
capture/verify workers with nonblocking admission (CaptureCapacity),
900 s idle and 1800 s hard lifetimes on an injected clock. acquire and
release_use keep an in-use capture past expiry, release and
invalidate_all until its last use ends; close removes only owned files
and the owned directory. A header-only probe of the copy marks Generate
unsupported above 4096 px per side or 16,777,216 px, for unreadable
headers and for video; verify re-hashes the current file without a
copy. Every handle closes before unlink.
tests/test_media_capture.py covers exact bytes and owner-only modes,
alias and escape denial, the symlinked-root locator, change detection
during the copy (size, same-size rewrite with restored mtime through
the POSIX change time, inode replacement), both size caps, count, byte
and worker admission, idle and hard expiry with in-use retention,
cleanup scope, handle release at unlink, dimension and video
readiness, and the allowlist equality with the inspection service.
TESTING.md documents the store and the Windows change-time limit.
* test(annotations): wait on the POSIX change time only where it exists
The same-size rewrite case waits for the filesystem clock to pass the
source's ctime so the detection is deterministic on Linux's coarse
timestamps; Windows reports the creation time as st_ctime, where the
probe could never advance, so the wait runs only on POSIX.
* fix(annotations): compare the change time only between fstat reads
Python 3.12+ on Windows reports the change time from stat but the
creation time from fstat, so a freshly written file whose creation and
write straddle a clock tick failed the pre-read identity check with
CaptureChanged. The path-versus-descriptor checks now compare size,
mtime_ns and dev/ino; ctime_ns is compared between the fstat before
and after the copy, where it still detects a same-size rewrite with a
restored mtime on POSIX.
The symlinked-root test asserted the locator against the unresolved
temp path, which macOS (/var -> /private/var) and Windows (short names)
resolve differently; it now compares against the real path.
* test(browser): give the inspector evidence matrix the concept comparisons' budget
The touch run of 'full inspector attaches matched theme and mobile
evidence with reachable content' captures 32 screenshots across four
viewports and took 23.5 s, then 31.6 s, in the sharded coverage job,
hitting the 30 s default. Like the concept comparison specs it now sets
an explicit 120 s timeout; the CI-like pinned run (4 CPUs, 3 workers,
repeat-each 3) passes 198/198 with the matrix at 18-20 s on touch.
* fix(annotations): compare capture identity like with like and keep paths out of errors
Native Windows CI (Python 3.12/3.13) rejected untouched files with
CaptureChanged because the path stat and the descriptor fstat disagree
there (change time versus creation time, possibly dev/ino encodings).
The descriptor is now checked against the path only on the API-stable
facts (size, mtime_ns); the full identity (size, mtime_ns, ctime_ns,
dev/ino where meaningful) is compared between the fstat before and
after the copy and between the path stats before and after. A test
injects shifted fstat results to simulate the Windows discrepancy, and
a fresh file is captured twice with no platform guard.
Error messages carry strerror only, never the path (the path goes to
the debug log); a test asserts no CaptureError names the root. Files
are unlinked outside the store lock; close() during an in-flight
capture lets that worker remove its temp file and then the directory.
get() documents that readers of capture.path must acquire/release_use.
The test fixture resolves its temp root (macOS /private/var, Windows
8.3 names). TESTING.md states the real identity contract.
* fix(annotations): move image revisions when a fingerprint or the vocabulary changes
A replayed save that observes a new verified (size, mtime_ns) now bumps
image_index_revision, like refresh_image_fingerprint, so caches keyed
on it see the record become current again; an identical replay still
moves nothing.
Vocabulary rename, merge and delete (native and legacy paths) bump the
tag_revision of every image set carrying the source tag, in the same
transaction and before the links move or cascade, plus one
image_index_revision bump when any set was affected. A Tag dialog
opened before the change therefore gets tag-conflict with the renamed
set as current instead of recreating the old name.
* test(annotations): accept change detection wherever fstat exposes a change time
Native Windows CI on Python 3.13 detected the same-size, mtime-restored
rewrite through the descriptor's change time (3.10-3.12 report the
creation time there and complete the torn copy). The test asserted the
platform, not the signal: it now accepts CaptureChanged plus temp
removal wherever it happens, requires detection on POSIX and on
Windows 3.13+, and otherwise requires the digest to describe the
copied bytes. A new test injects a drifting fstat change time so the
detection path runs on every platform; TESTING.md records the limit as
"Windows before Python 3.13".
* test(annotations): assert the rewrite outcome, not the platform
Native Windows CI detected the same-size, mtime-restored rewrite on
3.12 and missed it on 3.13 in the same run: the path stat's change time
(Python 3.12+ fast path) only moves when the clock ticks between the
pre-stat and the rewrite, and the tick wait was skipped on Windows. The
wait now runs everywhere with fresh probe names (Windows tunnels the
creation time of a reused name), and the test asserts the outcome: a
detected change leaves no temp file; an undetected one is accepted on
Windows only and must produce a digest of the copied bytes; POSIX must
detect it. The injected-fstat test keeps the detection path covered on
every platform.
* fix(images): keep the Filters label node across a listing reload
Every listing reload reports the active refinement count, and
setActiveFilters rewrote the button's textContent even when the label
was unchanged, replacing its text node. WebKit drops a click whose
mousedown hit a node that was detached before mouseup, so a Safari
user who types a search and clicks Filters inside the 250 ms debounce
lost the click, and the webkit page workflows timed out in the
source-change recovery tests (1 in ~30 runs of the pinned image; an
instrumented probe showed pointerdown/mousedown, a DOM mutation,
pointerup/mouseup and no click event).
The label is rewritten only when it changes. The toolbar unit test
counts textContent writes (1 for an unchanged count before, 0 after),
and a workspace-images test pins the text node identity across a
search reload: 0/6 before the fix on chromium and webkit, 6/6 after,
with the recovery and Refine tests 36/36 in the pinned CI-like run.
|
||
|
Vito
|
f95f66d76a |
ci(browser): run page workflows with parallel workers and CI shards (#274)
* test(browser): one test server per Playwright worker The page workflows started one global webServer and ran with one worker. Each worker now owns its own tests/browser/server.py on an OS-chosen port, with its own temp root and database, through a shared `test` in tests/browser/fixtures.mjs. baseURL, and so page, request and browser.newContext, point at that worker's server. A server that dies fails the running test with its stderr tail and the next test gets a fresh one. The worker count comes from PM_TEST_WORKERS, else 1 under CI, else a quarter of the CPUs up to 8. Chromium workflows in the pinned image: 510 tests in 606 s with one worker on main, 511 (one new isolation spec) in 119 s with eight. * test(browser): test servers bind an OS-chosen port themselves tests/browser/server.py accepts --port 0: it binds 127.0.0.1:0 before building the app, holds that socket for its whole life and prints PM_TEST_SERVER_PORT=<n> to stdout. startBrowserServer defaults to port 0 and reads the announced port, so no port is reserved and released before the server binds and two launchers can no longer race for one. The "address already in use" retry and free-port.mjs are gone; --port N still works and is checked against the announcement. * ci(coverage): shard and merge modes for the frontend coverage run frontend-coverage.mjs now plans its steps from the command line. No arguments still runs everything, then reports and checks. --shard I/N runs one slice of the chromium+touch workflows into coverage/js/raw without wiping it or reporting; shard 1 also runs the node units and the viewer suite. --merge-only reports and checks over raw coverage the shards wrote, with --update-baseline as before, and refuses an empty or missing raw dir. npm scripts: test:coverage:shard and coverage:merge. * ci(viewerjs): run browser suites with workers and shards Frontend coverage runs as three shards (PM_TEST_WORKERS=3 each) that upload their raw V8 files as coverage-raw-N; a merge job downloads them, fails unless all three arrived with raw files and no clashing names, then runs coverage:merge. Firefox and WebKit page workflows run as two shards each with three workers. The required viewerjs context keeps its name and now needs the shards, the merge and the browser shards, failing unless every one succeeded. * ci(viewerjs): cache npm downloads and size timeouts from the first sharded run The first sharded run lost a webkit shard to its 12-minute limit after all 267 tests passed, and npm ci stalled for 5 minutes in six of eight jobs. Each npm ci now restores npm's download cache keyed on the vendor lockfile, and browser job timeouts are the measured times plus at least 50%, never under 20 minutes. The workflow comment records the measurements. * test(browser): wait for rendering frames, not timers or luck, under load Two Library workflows failed in CI's webkit shard 2/2 (run 39714840809) under parallel load: - "library infinite loading ..." counted cards right after the last page hid "Load more". That render queues the window update for the next frame, so a slow frame left the 120 cards of the all-rows layout in place (received 120, expected < 80). It now waits for the next frame, which runs after every callback the page queued. - "row measurement preserves anchor ..." settled with a 60 ms timer. A scroll event is delivered in a rendering frame, so on a starved renderer the timer ended before the scroll's frame and no card was in view (null anchor). settleLibrary now waits for two consecutive native frames with no Library frame queued, checked inside the frame callback. Before/after: a scratch copy that delays every frame by 120 ms (as a starved runner) failed the old helpers 19 of 40 runs, at the CI lines, and passed the new ones 40 of 40 on chromium and webkit. Throttled to 1 CPU with 4 workers, the original spec failed 1 of 42 for each test. The Tags viewer workflow closed the viewer as soon as its image was visible, before ViewerJS rendered the title, so CI's coverage never ran the page's title callback (tags-page.js functions 26/46 = 56.52%, one function above the floor of 56; locally 27/46). The test now asserts the title names the shown file, which always runs the callback. * test(browser): stable under parallel workers Document how to run the browser suites in parallel and sharded, with the measured times, and what keeps specs stable beside each other: wait for a rendering frame, a response or a visible state, never for time. With |
||
|
Vito
|
a6f6697c50 |
feat(activity): shared job registry, Activity page and Scan for metadata (#269)
* feat(activity): shared session job registry with the scan as its first kind The output scan now runs as the first kind of an in-memory JobRegistry (py/api/jobs.py, replacing py/api/scan_job.py). Each job keeps exact counters, at most 200 outcome rows per group and a log capped at 2,000 lines, with root-relative paths and public root labels only. New routes list, inspect, follow (SSE) and cancel jobs and start the scan (POST /prompt_manager/jobs/scan, 409 busy while another record-changing job runs). The legacy /prompt_manager/scan routes attach to the same job and stream unchanged payloads. * feat(activity): concept Activity page, top-bar button and progress pill The workspace gains the concept's Activity page (renderActivity), its top-bar button and sidebar row, fed by one session job store: a /jobs read, one SSE stream per running job, a 1-30 s capped backoff after a dropped stream and a 15 s list poll while visible. A job the server no longer lists shows as ended, so no pill stays stuck after a restart. D52: the counts show running jobs and hide at 0; the phone top-bar row may wrap. D53: the mockup's progress pill links the running job. D54: real job items with state chips, detail lines and Cancel job. Start and finish toasts follow the spec copy. Fake browser jobs come from tests/browser/workspace_activity_fixture.py. * feat(activity): Scan for metadata with its review dialog and job view Images -> Import keeps Rescan and adds the concept's "Review scan" row as "Scan for metadata". It opens the concept's "Review metadata scan" with the real source count, the fixed "All files" scope, a backup box that downloads the existing database backup, and Cancel / Start scan. Start scan starts the session scan job and opens its view; a start refused by another record-changing job toasts its title as a link. #/activity/<id> is now the approved job-view mockup: the heading with Download report (disabled until Activity-2) and Cancel scan, the Progress surface with the five counter tiles, the Results surface with the outcome list, path filter and capped rows (Added and Already known open the viewer through the listing identity), and the collapsed Scan log. Activity item titles link their views. The scan's progress line groups thousands. Deviations D55-D57 are recorded in TESTING.md. * fix(activity): stop the job store on pagehide and let proxied routes outlive teardown CI's touch coverage run failed "a resolver failure for the ComfyUI link never stops the shell from mounting" with "route.fulfill: Fetch response has been disposed": its trace shows two Library thumbnail requests started by the last route still in the spec's proxy-mount handler while the page fixture was torn down. - The proxy-mount handler ignores only that teardown error, and the workspace and Activity specs unroute with ignoreErrors before teardown. - The Activity store now ends its work on pagehide: the poll and backoff timers stop, the job streams close and a list read in flight is aborted (an older read is also aborted when a newer one starts). A pageshow from the back-forward cache reads the list again. * test(activity): compare the job view log's line height to the hundredth of a pixel WebKit reports the mockup's 11px/1.7 line height as 18.700001px, so the job view comparison failed on WebKit only. The measurement now rounds it, as other engine-dependent values are normalized. * fix(activity): final review round for the job store, job view and scan rows - Store: one failed list read no longer marks running jobs as ended; two in a row, or a good list without them, do (M1). A good read resets the backoff, except for a stream that keeps dropping while the list reads fine (M2). Finished jobs the server pruned are dropped (M3). Numbers group as en-US, like the server's lines (M13). - Job view: a job the store's list no longer has shows the not-found state (M3). Focus moves to the job heading when Cancel scan goes away, and to the item's title link in the list (M4). The detail is read only when its listed rows change or the open log grows, using the summary's new listed and log_length (M12). - Viewer rows open by exact identity: each scan row records its root's listing index, the source must still carry the row's label, and the workspace listing gains an exact `path` filter (M8, M9). - Scan rows name the prompt by its Library title, stored title first (M10). - A toast with a link stays 20 s, and any toast stays while hovered or focused (M14). - A real two-batch scan cancelled after batch 1 keeps partial counts (M7). - TESTING.md: D54, D56 and D57 updated (M5, M6). * test(activity): cover the held and linked toast and a job view pruned by the server Unit tests for the toast's 6.5 s and 20 s timeouts and its hover and focus hold, and a browser test for a finished job's open view after the server stops listing it. Coverage floors are only raised. |
||
|
Vito
|
e9996996e2 |
feat(views): saved views and Favorites on Images and Library (#268)
* feat(views): saved-views store and API
Add a saved_views table (page images|library, trimmed 1-100 character
name, whitelisted JSON state of at most 4 KB, position, timestamps)
installed through the same validated-savepoint migration as image
favorites. It is optional in old backups and required in a migrated
restore stage; restore rejects a tampered definition, extra indexes and
any row that would fail write validation.
Routes under /prompt_manager/saved-views: GET lists views by position,
POST {page,name,state} returns 201 {success, view}, PATCH /{id} {name}
renames and DELETE /{id} removes, both 404 for an unknown id. The state
must hold exactly the page's toolbar and Refine-this-view keys (Images:
q, category, sort, source, folder, media, tags; Library: q, category,
sort, tags) with their types and sort/media values; anything else is a
400.
Refs #249
* feat(views): Save view, the SAVED VIEWS sidebar and opening a view
The shared toolbar gains the concept's Save view between sort and Filters
on Images and the Prompt library. It opens the concept's simpleInput
("Save this view", "View name", default "Search: <q>" or "My workspace
view", Cancel / Save); an empty name is "Saved view". A save posts the
page's full whitelisted state (toolbar value plus Refine filters) and
toasts "Saved view added to the sidebar.".
The shell adds the concept's SAVED VIEWS label and nav#saved-nav, loaded
with the shell and re-read after every save. Rows use the concept's
links() markup (search icon, name, no count) and link #/saved/<id>.
Opening one mounts its page with the saved state through the page's own
filters, so the toolbar and "Filters · N" (D44) show it; the h1,
breadcrumb and title are the view's name. A removed Source or Folder is
dropped to its default after discovery, so no listing read is rejected;
an unknown id re-reads the list once, then opens Images; a failed list
read offers Retry.
D16 is retired: images-concept and library-concept no longer map Save
view out. New deviations: D49 (no PROPOSED badge) and D50 (no Needs
review row); shell-concept maps Favorites out until Task 3 and compares
a saved row against the concept's. Browser fixtures now wipe
saved_views.
* feat(views): Edit rename/delete and the Favorites saved view
- Favorites is the first SAVED VIEWS row (star icon, name, real count).
The count is the Favorites listing's own limit=1 total, follows the
nav-count rules and is re-read after every confirmed star or unstar.
- #/favorites opens Images filtered to favorites. The workspace listing
takes favorites=1: each selected root's favorites are read through the
image_favorites primary key and matched against the cached scan with the
toggle's case folding; a failed read is an error, never an unfiltered
page. The cursor scope includes the flag.
- Saved Images state gains a required favorites boolean, so Save view on
Favorites reopens the favorites.
- An open saved view's heading leads with Edit: "Edit saved view" with
View name, Delete... (danger), Cancel and Save. Delete... confirms in the
same dialog (never window.confirm). Toasts "Saved view renamed." and
"Saved view deleted."; a delete returns to the view's plain page.
- shell-concept no longer maps Favorites out (Needs review stays, D50).
Recorded as D51 in TESTING.md.
* fix(views): open stale saved views safely and finish the review fixes
Startup and restore now check only the saved_views structure (table,
columns, types, sizes, a JSON object state) and never the value rules,
so a row saved under older rules no longer stops the database from
opening. Reads normalize each view: missing keys take their defaults,
unknown keys are dropped, invalid values default and the name is made
displayable. POST and PATCH still validate strictly.
Only request validation (SavedViewInvalid) maps to 400; database and
internal errors are a logged generic 500. A saved view no longer
replaces the plain Images or Library return state, a toolbar change
while sources load keeps a saved Source and Folder, a state over 4 KB
is refused in the browser with its own toast, the delete confirmation
skips its failure toast after the page is gone, the Edit dialog reuses
the shared field helper, and quick star toggles coalesce into one
Favorites count read.
* test(views): cover leaving a pending saved view and add its coverage floor
A saved view left while the list or a rename is still pending leaves
the next page alone; the shell drops two unreachable heading guards.
workspace/saved-views.mjs gets its frontend coverage baseline entry;
no existing floor changes.
* fix(views): a corrupt saved-view state no longer blocks startup
At startup a saved_views row whose state is not a JSON object is
logged as a warning by id only and left in place. The listing and
rename skip it, so its #/saved/<id> link behaves like a missing view.
Restore stays strict and still rejects a backup holding such a row.
|
||
|
Vito
|
1ff2ddce05 |
fix(inspector): fall back to the positive's category and tags for recipe-linked images (#267)
* fix(inspector): fall back to the positive's category and tags for recipe-linked images Maintainer ruling D39 (2026-10-21): a recipe-linked image shows the recipe's own category and tags when set, otherwise the linked positive prompt's, each field on its own and without merging tag sets. Set category (#259) and the Images category and Tags filters work on the positive, so the panel now agrees with them. The inspection library replaces metadata_scope with category_scope and tags_scope so the category chip's hidden scope prefix (D21) and the tag chips' filter-button rule (#260 D2) follow the value actually shown. History's media inspector uses the same shared rule. The concept fixture's positive_links workaround is gone: the comparison now matches the concept without it. * test(inspector): assert the D39 fallback tag chips in their saved order * fix(inspector): treat Unicode whitespace as a blank History recipe category * test(inspector): cover mixed-scope panels, invalid library scopes and the skipped positive tags query |
||
|
Vito
|
3476f8ede9 |
feat(inspector): concept comparison, Show in folder and toast placement (#264)
* test(inspector): concept comparison for the panel and viewer details Compare the Images side panel and the viewer's full details with the concept (1600x1100 and 390x844, both themes): structure, tokens, geometry and controls, plus copy, the acknowledged pair send, the tag chip filter and Prompt history. mapInspectorConcept applies only D3, D20, D21 and D22; a mutation test shows the comparison notices a changed role label and a removed footer. The comparison showed the role copy and send names lacked the concept's ": <title>" subject, so the inspector now passes its title to the prompt blocks. It also showed that a recipe-linked image shows its recipe's own category and tags (#224), which the concept fixture's recipes lack; that is recorded as D38, and the fixture's new positive_links option links the compared image to its positive alone. * docs(inspector): renumber the recipe-category decision to D39 * feat(images): local-only Show in folder route POST /prompt_manager/images/reveal takes the listing identity {root_index, root_scope, path} and opens this computer's file manager at that file. It answers 403 not-local unless the client is loopback, no proxy header is present and Host/Origin name this computer; the scope is checked with _gallery_scope_error (409) and the path resolves only inside that root to a regular media file. Openers run as argv lists with an absolute executable, no shell and DEVNULL streams (D38). * feat(inspector): Show in folder after the File & provenance path The viewer's File & provenance offers "Show in folder" right after the Path value for a listed file only, POSTs its listing identity and toasts the outcome (opened, not-local, unavailable or failed). The browser test server stubs the opener; TESTING.md records the decision as D38. * fix(images): keep the discovery alert clear of the Rescan toast A rescan that leaves the Images discovery alert visible could show its fixed bottom toast over the alert on a short window. The concept's toast keeps its placement; the production-only alert now scrolls into view only as far as needed, with a 96px block-end scroll margin (D42). * docs(inspector): concept panel and viewer details * fix(images): open Windows folders through the shell API, not an Explorer command line Explorer splits its own command line on commas, so a gallery path such as output\a,\Windows\System32\calc.exe,b\x.png escaped the roots through [explorer, "/select,", path] (security review H1). Windows now selects the file with ILCreateFromPathW and SHOpenFolderAndSelectItems, and on any shell failure falls back to os.startfile on the folder. No process is started. * fix(images): re-check, isolate and reap the Show in folder opener - re-check that the real path is still a regular file inside its root right before launching; a changed target answers 404 (review L4) - run openers on a dedicated single worker, not ComfyUI's default executor, and make dbus-send wait for the FileManager1 reply (--print-reply --reply-timeout=2000) so a missing service falls back to xdg-open (L1) - reap launched processes on a daemon thread, give them a minimal environment with a fixed system PATH, and use /usr/bin/open on macOS and dbus-send/xdg-open resolved only on that PATH (L5) * fix(images): accept Show in folder only as a same-origin JSON request, one at a time - require an Origin equal to the request's own scheme://host:port and, when sent, Sec-Fetch-Site: same-origin; refuse every forwarding-style header (Via, X-Forwarded-*, X-Original-*, *-Client-IP, CF-Connecting-IP and the rest) as remote (security review M1, L2) - answer a non-JSON body with 415, so a cross-origin request always needs a CORS preflight (L2) - single flight plus a 1 s cooldown per process: 429 code busy (L1) - path resolution errors such as ENAMETOOLONG are a JSON 404, and neither that nor a start failure logs the client's path at error level (L3) * fix(inspector): say the folder is already opening on a busy Show in folder A 429 busy from the reveal route now toasts "The folder is already opening." The browser workflow checks that the reveal POST carries the page's own Origin and an application/json body, and TESTING.md D38 records the tightened route, the toast table and the accepted residual risk of a header-less local proxy. * test(inspector): compare the viewer details under a coarse pointer and prove its mutations The touch project measured production's D34 40px disclosure targets against the concept's fine-pointer summaries. The concept's plain-chip copy now also carries the recorded D34 coarse rule, so both documents are measured under the same emulation. The open File & provenance is compared too (D38 Show in folder and D20 Download original mapped), and the mutation check now removes the viewer's pair actions and Show in folder. * refactor(inspector): attach Show in folder to the Path value it follows The button was placed on the second fact row by position; the Path dd is now built by the caller and kept, so reordering the facts cannot move it. * test(inspector): let the uncertain-file send check fail again Inspector send buttons are named "... to ComfyUI: <title>", so the anchored regex could never match and the no-send assertion always passed. * fix(images): run the Windows folder fallback inside COM and keep desktop session variables Show in folder's os.startfile fallback now runs inside the same CoInitializeEx/CoUninitialize bracket as the shell selection, as ShellExecute callers should. The Linux opener keeps XAUTHORITY, DESKTOP_SESSION, XDG_SESSION_DESKTOP and XDG_CONFIG_HOME, so a file manager can reach the X11 display under GDM or SDDM and honour the user's mimeapps.list. A Windows-only test calls the real ILCreateFromPathW and ILFree on a temp file and pairs CoInitializeEx with CoUninitialize through the module's helper, without SHOpenFolderAndSelectItems, so no window opens on CI. * fix(images): scroll to the rescan's discovery alert only when the toast band covers it D42 scrolled the alert into view after every rescan, so a user scrolled down the grid was sent back to the top. It now scrolls only when the alert intersects the toast's band (its scroll-margin-block-end). * refactor(inspector): name prompt actions as PR #263 does The title option keeps its name, default and output ("Copy positive prompt: <title>") and now uses #263's actionName helper; the region-name arrow no longer shadows the title option. * test(inspector): the comparison opens Filters by its D44 active-count name After a tag chip applies its filter, the shared toolbar's Filters button reads "Filters · 1" (D44, #263), so the comparison opens it by that name. * fix(images): POSIX opener paths stay absolute under Python 3.13 on Windows Python 3.13 on Windows no longer counts "/usr/bin/x" as absolute, so the faked-Linux reveal tests failed on windows-latest py3.13 (the Linux and macOS openers are POSIX paths whatever host evaluates them). Use posixpath.isabs for those lookups. |
||
|
Vito
|
86d82186ee |
feat(images): concept Images page — toolbar, cards, star, viewer, compared against the mock (#258)
* feat(workspace): shared concept toolbar and dialogs
Add the concept's shared page toolbar (search.toolbar with the search
field, category filter, sort and Filters) and its dialogs in one modal
shell: "Refine this view" (Source / Folder / Media / Tags, Apply filters),
"Import & inspect" with only the real operations (Open inspector, Rescan),
and the "Copy prompt" select-to-copy fallback. The sort lists the
concept's three options first (D14); there is no Save view (D16). Every
dialog returns focus to its trigger on close or Escape.
Refs #249
* fix(workspace): bind shared dialogs to the page and its route
- Drop the invented Refine note; every field is real (D14).
- Ignore a late close event once the shell was reopened or replaced, so
an action that opens the next dialog keeps it open.
- Dialog openers require the page scope; disposing it closes and removes
the shell unless a newer page's dialog is open. Every route change
closes the shared dialog before the next page mounts.
- Clear filters also reports category '' so the page clears the toolbar
category, as the concept does.
- The toolbar requires its scope and falls back to the first sort for an
unknown value; dialog callbacks are validated up front.
Refs #249
* feat(images): concept toolbar, filter row and caption
The Images page now renders the approved mock's heading, shared toolbar,
filter row, section caption and infinite continuation (#249, Task C2).
- Heading: "Your generations, their prompts, and the next idea." with
Import (upload icon) before New prompt (primary).
- Toolbar (shared mountToolbar): search over filenames, linked prompt
text, titles and tags; category (the Library's categories); sort
Recently added / Name A–Z / Highest rated, then Oldest first and
Name Z–A (D14).
- Filter row: Grid/List/Masonry segmented control, Thumbnail size,
the scope chip ("All output folders" or "<source> · <folder>") and
"Group by positive prompt →".
- Caption: "N of M images · K positive prompts" from positive_count; a
plain cursor page keeps the first page's count and an unknown count
hides the "· K positive prompts" part. The date span shows
"<Today|Yesterday|weekday> · <Month D>" of the newest loaded result
under a date sort and is hidden otherwise (D17).
- Continuation: "N of M images · more load as you scroll" with Load
more, the concept's end copy, Retry for a failed page, and the
concept's "K more images loaded. N of M." toast.
- The listing sends category, tags and sort=rating (PR B params).
Old control -> new control:
- "Search filenames" field -> toolbar search "Search workspace"
(placeholder "Search prompts, tags, filenames…"), also matching
prompt text and tags
- Source select -> Filters -> "Refine this view" Source
- Folder select -> Refine Folder (all sources' folders and ancestors)
- Media select -> Refine Media
- (new) Tags -> Refine Tags; Category -> toolbar category select
- Sort select (Newest/Oldest/Filename A–Z/Z–A) -> toolbar sort
(Recently added/Oldest first/Name A–Z/Name Z–A, plus Highest rated)
- Layout buttons -> .segmented Grid/List/Masonry in the filter row
- Thumbnail size field -> label.small + select#image-size
- "Group by positive prompt" link -> button.chip "Group by positive
prompt →" (#/library)
- Refresh -> Import -> "Scan an output folder" -> Rescan (D15)
- "Current gallery" link -> removed; Overview's tool cards link the
legacy gallery (D29)
- "All images" h2 + count -> .section-caption #result-count
- .image-continuation -> .infinite-continuation (Retry kept)
Legacy behavior kept: video items, keyboard grid navigation, scroll and
filter restoration, bounded DOM, continuation retry, source-change
recovery (now disabling the toolbar, Filters and Import). The Filters
and Import dialogs close on route change and browser Back.
The shared toolbar (C1) called .some() on select.options, which is an
HTMLOptionsCollection in a browser (only the unit tests' fake DOM gave
an array); setValue and setCategories now spread it first. The layout
segmented control may wrap at extreme zoom (D31) instead of scrolling
the 390px page sideways.
The retired .image-filters, .image-view-controls, .image-toolbar,
.image-layouts, .image-collection, .image-continuation and duplicated
.image-workspace grid rules are gone from images.css.
* test(workspace): model select.options as a browser collection
The toolbar units' fake DOM returned select.options as an Array, so the
toolbar's .some() call passed there while it threw in every browser
(fixed in
|
||
|
Vito
|
9d398f9862 |
feat(workspace): concept shell — sidebar, topbar, heading and footer on the approved mock's CSS (#256)
* build(workspace): vendor the approved concept stylesheet with a drift check * refactor(workspace): adopt the concept stylesheet as the workspace base * feat(workspace): concept sidebar with real navigation counts * feat(workspace): concept topbar with Theme and ComfyUI link * refactor(workspace): one concept heading and footer for every route * fix(workspace): keep the canvas link on the proxy mount A resolver failure no longer stops the shell from mounting: the ComfyUI link falls back to the root read from the page's own mount path (never another origin). Tests cover a non-standard /ComfyBackendDirect mount, including that no request escapes it. * test(workspace): concept fixture and shell comparison * test(workspace): concept fixture carries the concept inspector facts * fix(workspace): match the concept brand name and compare shell hover states * test(workspace): shell comparison mutation check and docs Add mutate() to the shared concept inventory and a shell mutation check: a nav-row min-height rule and a removed .breadcrumbs each make the comparison fail with the changed path. Harden the resolver-failure guard test with an injection count, make the overlay lint fail when workspace.css is missing, mark the library.css .row-actions leftovers for PR E, and document the concept CSS base, the overlay lint, the comparison method and the recorded shell deviations in TESTING.md. * fix(workspace): focus the page title on a route change A sidebar or history navigation now focuses #page-title, as the concept does on every hashchange (app.js 1168), so a screen reader announces the page heading instead of the main region. The first render leaves focus alone and the skip link still focuses main. The heading module's comment now describes what the shell does. * fix(workspace): give the Overview nav entry its own home glyph Overview reused the concept's Activity pulse byte for byte, so once the Activity row lands two entries would share one icon. Overview now has an outline home glyph. The shell comparison takes it from the D28 ruling instead of the concept's activity path, and the icon test checks that every approved glyph is distinct. * test(workspace): accept only #249 decision markers and state the lint scope The overlay lint accepted any D-number, so another issue's ruling such as "D5 (#245)" could mark a concept override. A marker now has to be a D-number with no other issue cited; the vocabulary rule that cited #245 reads "#245 decision 5". TESTING.md now states that the lint covers workspace.css only, lists the unmarked overrides still in images.css, and says PRs C and D run the lint over each page sheet they keep. * perf(workspace): re-read only Library and Tags counts after a Library write A confirmed Library or vocabulary write cannot change the Images count, but it re-read all three, including the Images output listing. It now re-reads only the Library and Tags counts. Each count has its own read, so a write never cancels an Images read that is still in flight. |
||
|
Vito
|
cbf46a7573 |
feat(vocabulary): native Tags & categories screen (#245)
* feat(vocabulary): add workspace vocabulary state * feat(vocabulary): render the native Tags & categories screen * feat(vocabulary): add guarded vocabulary dialogs * fix(vocabulary): recover unconfirmed deletes and list every category in Manage * refactor(vocabulary): use the shared plural helper * test(vocabulary): compare the native screen with the approved concept * fix(vocabulary): wrap long tag names so row actions stay visible on narrow screens * docs(vocabulary): document the native vocabulary screen and gates Also aligns the shell link focus-ring offset with the concept (3px) and makes the chrome comparison poll the document title. * fix(vocabulary): refresh lists when a write lands after its dialog closes A rename, delete, create or category apply that commits after its dialog was disposed (Chromium force-closing the modal on a repeated Escape, or Back while the write is in flight) returned before onInvalidate, so the lists were never re-read and a cached Library shelf kept showing the old names. Like the recipe editor, a disposed command now still reports any outcome that may have changed storage (receipt, conflict, restore, unconfirmed) without rendering; only a failed write stays silent. * fix(vocabulary): give the Manage categories datalist its own id The proposed-name datalist reused the recipe editor's `categories` id. It was safe only because routes dispose each other's dialogs; it is now `vocabulary-category-options`. * docs(vocabulary): correct vocabulary testing notes Name the Tag column and the Create tag heading action, say that a failed Load more moves focus to Retry, place bulk actions on the admin page, describe late writes after a closed dialog, Browse's exact filter and the full deviation list, and use a test port that is not the live concept server's. * test(vocabulary): raise the vocabulary dialog branch floor coverage:update after the late-write and datalist fixes: workspace/vocabulary-dialog.mjs branches 87 -> 88. No floor is lowered. * fix(vocabulary): keep 390px row actions visible regardless of font metrics The one-column Tags table reserved a fixed 6em for the tag name. With CI's stock DejaVu Sans fallback (about 9% wider than the pinned image's) Tag + Prompts + Images + stacked actions needed 351px of a 336px wrap, so Browse and Rename / merge were pushed out of view on every row. At 700px and below the Tag column now has no fixed minimum and the actions ask for no more than their stacked width, so all remaining space goes to the tag name: short names stay on one line wherever the concept's own table fits, and a wider font wraps the name instead of clipping an action. A new test widens the metrics with letter spacing and fails without this change in every engine. * ci: run browser jobs in the pinned Playwright image Concept comparisons measure geometry, which follows the fonts a browser falls back to. The concept names Inter, which neither environment ships: the pinned image falls back to WenQuanYi Zen Hei, a stock ubuntu-latest runner to DejaVu Sans. Under DejaVu the concept's own Tags table needs more than a 390px viewport (its page scrolls sideways and, under touch emulation, the widened layout viewport resizes its dialogs), so the 390px comparisons failed in CI while passing locally. Both browser jobs now run in mcr.microsoft.com/playwright:v1.64.1-noble, the image local runs use, instead of installing browsers on the runner. A tooling test keeps the tag equal to @playwright/test in the lockfile. * fix(vocabulary): ignore a category preview edited while it is pending Editing the Category or proposed name while a slow Preview was in flight was a no-op in the saving phase, so the late response still became an applicable preview: the form showed the new values while Apply would submit the old source and target. An edit during a pending preview now marks it stale. Its late result returns the dialog to idle (Preview again) instead of showing an impact or offering Apply; a plain failure of a stale preview is not shown either, while a restore or conflict still is, since it describes storage. The inputs stay editable during the request, as in the recipe editor, which disables only its actions while saving. * ci: use the image's Python and Node in containerized browser jobs Inside mcr.microsoft.com/playwright:v1.64.1-noble, actions/setup-python failed before any test ran: `spawn /__t/Python/3.12.15/x64/bin/pip ENOENT`. The toolcache builds come from the runner, and their entry points name host interpreter paths that do not exist in the container. The jobs now use the image's Python 3.12 and Node 24. The image has no ensurepip and the job runs as uid 1001 without root, so a checksum-pinned pip wheel bootstraps a user-space venv for requirements-dev.txt; a step checks that the image Node is 24. The tooling test forbids setup-python and setup-node in these jobs and requires the pinned bootstrap. |
||
|
Vito
|
8bcde4fbe4 |
feat(library): native recipe editor UI, pair archive browsing and export (#235)
* feat(library): manage native editor drafts and recovery * fix(library): fence pair archive retries against restored storage * feat(library): add native recipe editor dialogs * fix(library): read archive counts through the editor controller * feat(library): reconcile editor saves and archived pairs Wire the concept's heading actions (Library Export and + New prompt, Images New prompt, history ← Prompt library, Edit metadata and pair Send to ComfyUI) and the Negative variants + Add negative and Show archived controls. Saves navigate or reconcile in place through createLibraryDetail.accept, invalidate cached shelf state by write generation, browse and restore archived pairs, and offer an explicit Library reload when storage was restored. * fix(library): refresh archived pairs and align editor outcomes * test(library): enforce approved native editor presentation * test(library): verify editor host and coverage compatibility Adds the rejected connected-target host assertion, pins the stock host listing at 80 cases, documents the native editor, and keeps Save/Archive focus after a failed write. * docs(testing): correct host case totals and cover archive focus * fix(library): fence editor reloads after restore and settle outcome copy |
||
|
Vito
|
ecd21165a7 |
feat(history): add chronological runs and exact output inspection (#227)
* feat(history): add bounded chronological read projections (#209) * fix(history): validate internal continuation positions (#209) * feat(history): expose fenced history windows and run details (#209) * fix(history): release retired projection connections (#209) * feat(history): resolve reported outputs through current authorization (#209) * fix(history): reject invalid inspection targets (#209) * feat(history): add scoped history state and current-file adapter (#209) * fix(history): preserve slideshow and clear stale reentry actions (#209) * feat(history): render recorded executions and frozen run details (#209) * fix(history): recover receipts and refresh capture evidence (#209) * test(history): verify chronological history across restore and host modes (#209) * fix(history): enforce chronological marker visual parity (#209) * fix(history): resolve final chronological review findings Address #209 recovery, bounded visit restoration, exact receipt availability, and portable regression coverage. * test(history): repair Windows fixtures and review feedback Address #209 and PR #227 native path, cleanup, media and review-comment expectations. Document Windows portability preflight without changing production behavior. * ci: raise the viewerjs job budget for the larger browser suite The cross-browser workflows suite grew from 396 to 496 tests on this branch, which pushed the whole viewerjs job to about 25.5 minutes against the 25 minute cap set in #181. Two exact-head runs on this branch timed out without any test failing: run 39173019283 was killed on test 474 of 496, and run 39175125474 printed "496 passed (12.8m)" and was then cancelled during Playwright shutdown. Raise only the job budget. Browser projects, workers and per-test timeouts are unchanged. Splitting the browsers into parallel jobs is tracked separately. |
||
|
Vito
|
d9537a726c |
feat(workspace): complete shared inspector and collection viewer (#224)
* feat(inspector): expose bounded physical file evidence * fix(inspector): distinguish ambiguous saved associations * feat(inspector): derive conservative generation facts * fix(inspector): guard fact ancestry and inspection state * feat(inspector): share complete file details and exports * fix(inspector): preserve fact states and touch targets * feat(viewer): navigate the loaded media collection Refs #216 * fix(viewer): refit after current stage geometry changes Ref #216. Observe the current media stage after readiness and wrapped title changes without rebuilding focused controls or resetting transforms at unchanged dimensions. * feat(viewer): add lifecycle-safe slideshow and media preferences * fix(viewer): resume gallery continuation after close (#216) * fix(images): guard source recovery and wrap viewer actions Clear stale root and folder intent, cancel old pages, and wait for current source discovery before loading a refreshed collection. Preserve safe retry through failed, superseded, and disposed discovery. Wrap narrow viewer footer actions and wait for measured gallery viewport settling in responsive checks. Refs #216; addresses PR #224 review. * test(viewer): await gallery layout after viewport changes Require actual inner, root client, and document widths to agree after each responsive-case viewport and CSS zoom transition. Keep the existing exact frame, overflow, focus, and coarse target assertions unchanged. Observed PR #224 CI initial mobile resize at537px before pending card relayout settled to356px. Refs #216. |
||
|
Vito
|
e7cfcc2bfb |
style(workspace): align browsing with the approved concept (#219)
* style(workspace): align approved shell and visual fixtures * test(workflows): preserve evidence across browser projects * style(workspace): match approved image browsing geometry * style(workspace): match natural library card layout * fix(workspace): retain wheel motion during row correction * fix(workspace): contain zoomed controls and image metadata (#215) |
||
|
Vito
|
fdba6fd012 |
feat(workspace): group saved negatives in the native Prompt Library (#210)
* feat(workspace): group saved negative variants in native library (#208) * fix(library): preserve full name order and batch previews (#208) |
||
|
Vito
|
dc345671ae |
feat(workspace): inspect and send positive and negative prompts (#206)
* feat(workspace): inspect and send positive and negative prompts * docs(workspace): capture complete inspector and viewer previews * docs(inspector): explain the subprocess cleanup race * fix(workspace): defer observed gallery layout writes (#207) * fix(inspector): keep path-only prompt associations unverified |
||
|
Vito
|
590aa2cd0e |
feat(workspace): add native Images browsing and infinite scroll (#201)
* feat(workspace): add native image browsing and continuous loading * test(workspace): assert empty collections through their status * fix(workspace): rescan refreshed images and expose ancestor folders |
||
|
Vito
|
caa9649f32 |
feat: reuse positive and negative prompt combinations (#195)
* feat(db): add pair schema and stage restore migrations for #193 * feat(database): persist reusable ordered prompt pairs (#193) * feat(api): expose reusable pairs and atomic metadata enrichment (#193) * feat(metadata): resolve canonical prompt pairs with bounded provenance (#193) * feat(scan): enrich image pairs without guessing or duplicate links (#193) * fix(database): preserve pair tags during global maintenance (#193) * feat(tracking): enrich live image pairs and cancel scans safely (#193) * feat(nodes): encode paired prompts and count cached recipe runs (#193) * fix(database): validate restored pair ownership and provenance (#193) * feat(admin): report pair scan outcomes and cancellation (#193) * feat(ui): add paired prompt editor and explicit recipe library for #193 * fix(autocomplete): include reusable pair metadata vocabulary (#193) * fix(pairs): honor recent sorting and conservative scan outcomes (#193) * fix(ui): protect paired draft saves and clarify negative variants (#193) * fix(metadata): preserve bounded paired execution evidence (#193) * fix(tracking): follow actual pair output roles and prefer execution evidence (#193) * docs(pairs): document live validation and retain renderer screenshots (#193) * test(api): inject disposable databases before API startup (#194) * test(api): canonicalize portable temporary gallery roots (#194) * docs(ui): prototype branching negative variant history (#193) * docs(ui): show selected negative beneath positive in timeline concept (#193) * docs: retain approved timeline for later gallery redesign (#193) * test(db): close restore fixture connections before cleanup (#194) * test(db): reuse existing handle guards in all pair fixtures (#194) * chore: clarify rollback test and intentional exception handling (#193) |
||
|
Vito
|
f1b1d4731b | test: exercise real page workflows and enforce frontend coverage (#181) |