3be0ada6a129cbdd7e72193986b5ccdb2f86fc26
67
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
3be0ada6a1 |
fix(ingestion): link an image only to a prompt that provably produced it (#344)
* fix(ingestion): link an image only to a prompt that provably produced it An LLM fed a PromptManagerPair's prepend_text while its text editor was empty. The node refused to save (an empty positive raised and logged a warning), and the image monitor then linked the image to a stale tracker entry or to the newest prompt in the database. - Nodes: an empty text editor saves and tracks nothing (prepend and append text are never the stored prompt), logs at info, and marks the node's run key empty so no earlier entry can be used for its images. - Tracker entries carry a run key: a digest of the prompt node and its whole upstream graph, the identity ComfyUI's cache uses. The monitor computes the same key from the image's embedded graph and uses a queued or snapshot entry only when the keys match (a snapshot also needs a known run with one entry, so a batch's last item is never applied to every image). - The recency fallback is gone; anything uncertain stays unlinked and is logged at info. - Recorded execution history attributes a file when exactly one run reported it, the file was written during that run, and the run's positive contexts name one stored prompt. The monitor and the output scan use it as their last certain source. - The scan and pair ingestion treat a resolved pair with an empty stored positive as unresolved instead of raising. - tools/review_image_links.py reviews selected links by these rules; dry run by default, --apply changes only named ids after a backup. Closes #342 * test(ingestion): real-host attribution case and the #342 testing contract The managed host disables the image monitor, so the owned fixture pack gains a route that starts and stops the plugin's monitor singleton for the attribution case only. The case runs in both renderers: a typed Pair image links, a Pair image whose positive is only linked prepend text stays unlinked with nothing stored, an unknown node feeding text links through its run key, and its cached re-run links again. * fix(ingestion): take a queued entry only from the run the image appeared in Dropping a key's older queue entries whenever the node ran again let a later queued run, executing during the monitor's processing delay, take the earlier image's entry. The snapshot taken when the image appears now names the key's latest run, and the queue is popped by key and run: same-key entries of earlier runs ahead of it are stale, entries of later runs wait for their own images. Also keeps the attribution case out of the stock real-host suite. * test(ingestion): host lookups degrade to no proof * fix(ingestion): address the #344 review (history re-check, keys, repair) - Images without an embedded API graph: the scan now asks recorded history before skipping a file with no prompt metadata, and the monitor gives a keyless unlinked image one history-only re-check when its run's outputs are recorded (a seal notification from the outputs write; at most 1000 waiting images, each for an hour). - Run keys: an image is unkeyable when any positive node has no key, so a two-pass graph is never linked through the other pass; links follow ComfyUI's is_link exactly. - A failed save marks the node's run key like an empty editor, so an older entry of that key can't stand in for this run. - A placeholder is the latest entry: the image stays unlinked without falling through to history, the same rule as a popped placeholder. - An index on execution_run_outputs(filename, subfolder, root_key, path_style) for the history lookup, added by migration and accepted by the output schema check (any other index is still refused). - Repair tool: unlink removes only the link record and only while the image is still linked to the prompt it was judged on; every run prints what --apply does. - Real-host case proves the re-run's PromptManager node was cached. * test(ingestion): real-host bare-output re-check and the review-round contract A metadata-disabled save, held open behind the fixture gate after SaveImage, waits unlinked while its run is open and links once the run's outputs are recorded. The fixture's monitor route lists the waiting files. TESTING.md covers the history re-check, the read index and what the repair tool's --apply does. * fix(ingestion): batch history lookups instead of adding an index An index on execution_run_outputs would make the library unopenable by older builds, whose schema check refuses any extra index there. The migration and the schema-check allowance are reverted, so the history schema is exactly main's. stored_run_prompts instead looks up a batch of files with one parameterized filename IN (...) query per chunk of at most 500 names and matches in memory; the scan computes it once per scan batch. * fix(ingestion): address the #344 re-review (seal race, linked roots) - A sealed history re-check that can't be scheduled yet (the image's own pass still pending, or the pending set full) is kept due and scheduled when a pending image finishes, instead of being dropped. - History matches a file through ComfyUI's spelling of its output folder first, then with both sides resolved, so a symlinked or junction output folder still attributes the scan's resolved paths; waiting files are keyed by their resolved path. - The scan makes its batch history lookup only when a file needs it. - set_current_prompt pushes to the queue last, so marking a failed execution never adds a second queue slot. - A keyless image outside the output folder doesn't take a waiting slot. |
||
|
Vito
|
48b1292115 |
feat(nodes): let a link drive the prompt text of every PromptManager node (#345)
* feat(nodes): let a link drive the prompt text of every PromptManager node The Focused, Split and Pair editors hide their widgets, and ComfyUI only draws a widget's input socket while the widget is visible. `text` (and the Pair's `negative_text`) therefore had no socket, and a STRING link dropped on the editor fell through to the first free STRING input, `prepend_text`, which is how #342's workflow was wired. - Detach those inputs from their hidden widgets on create and after a workflow load, so they are ordinary sockets in Classic and Nodes 2.0. The schema, widget order and saved values are unchanged. - A STRING link dropped on the editor body fills the main prompt, then the negative, before the frontend's own choice. - Each editor says when its text comes from a link; the typed text is kept for disconnect. The Classic nodes now save the typed text under a link instead of null, and "use prompt" no longer overwrites it while linked. - Real-host tests: mouse drags in both renderers, a workflow saved before this change, and executions proving the linked text is saved, counted, recorded in history and linked to the run's image. Closes #343 * test(coverage): floor for the linked-text module * test(nodes): PromptSearchList text takes a dragged link like a core widget * fix(nodes): linked text review round for #343 - A legacy frontend Primitive linked to an editor's text or the Pair's negative now runs with its value: the graphToPrompt wrapper and the queue hook fill it in (through virtual reroutes), in new and pre-#343 workflows. - Moving a linked text and releasing it on the editor keeps it on `text`: a link being moved counts as free, as the frontend counts it. - A link dropped on a prompt box (or its link note) fills that box's input, so the Pair's Negative prompt box takes `negative_text` (Classic through getInputOnPos, Nodes 2.0 through the box's slot key). - Only exact STRING links are steered; wildcards keep the frontend choice. - A linked Classic text box gives its row back and gets it back on disconnect; the PromptManagerText note uses the theme colour. - The metadata parsers map positional widget values through the widget table when an editor's widget-bound inputs do not line up with it. - Real-host cases for the legacy Primitive (new and saved workflows, a real Run), the moved link, the Negative box, paste/clone/undo and Split. * fix(nodes): resolve a legacy Primitive's text like the host, once per run (#343) - Inputs still bound to their widget (Classic PromptManager/PromptManagerText) are left as the host resolved them; the injection no longer overwrites them. - For the editors' detached text inputs, the Primitive's value goes through its "Run widget replace on values" replacement and the hidden widget's own serializer, so {a|b} wildcards and %date% tokens resolve as on main. - Each queued output is filled once (graphToPrompt wrapper, or the queue hook for an output that did not come through it): a wildcard is drawn once. - Editors inside subgraphs are found under their <subgraph>:<inner> id. - Real-host runPrompt installs its request route before the run starts. - Shared parser case pins the widget-table fallback for a core KSampler. * fix(nodes): pin the Classic wildcard behaviour and match queued ids by type (#343) - Classic PromptManager/PromptManagerText expose _pmTextSerializerReady, settled once their own text serializer is installed; the real-host specs wait on it and assert the Classic text is queued as written, as on main. - The Primitive fill only touches an output entry whose class_type matches the graph node at that id. - TESTING.md notes the host's harmless "Invalid widget" warning per Run. |
||
|
Vito
|
fb9134f081 |
feat(system): Backup & export (System S9) (#339)
* feat(system): back up, restore and export natively System > Backup & export replaces its hand-off with the concept's section on the S8 backend: Back up downloads the verified copy and the status line shows only what GET /backup/status reports (D103); Review restore stages a chosen file and shows the server's review in the shared review dialog, Cancel discards the stage and Restore runs the Database restore job, after which the open views reload (D105); Export JSON plus the components and saved-pairs exports (D103); Import / inspect opens the shared Import & inspect dialog; the safety-backup policy saves through the Save row and scheduled backups offer Manual only (D104). Overview's Backup status row reads the same status. Refs #326 * test(system): compare Backup & export with the concept system-backups-concept.spec compares the section, the Restore preview and the Import & inspect dialog with the concept in both themes at 1600 and 390 wide, mapping only D103, D104, D105 (and D15/D55 for the shared Import dialog), and proves a renamed label, a removed row, a lost danger mark and swapped tokens are noticed. A review's grid values now wrap anywhere (D105), so a long backup file name keeps the concept's two equal columns at phone width. Refs #326 * docs(testing): Backup & export contract (D103-D105) Refs #326 * test(coverage): raise the frontend floors S9 measured system-backups.mjs gets its measured floor in place of the provisional one; admin.js, activity-scan.mjs and maintenance-review.mjs branches go up. Refs #326 * fix(system): keep a refused or handed-back restore review usable A Restore refused as busy now opens the same review again with Discard / Restore, and a restore job whose leases did not drain hands its review back (scope.review_kept) to the page that started it, which reopens it; every other path that abandons a token after Restore sends its DELETE, so held reviews can no longer lock restore out. The kept-review line says where the review is, and a full review store says what to do (I1). applyPlan resolves with its outcome and its job tracking now has failing-before proof (N5). Also: the safety-backup toggle says what the policy does (D104, M2); the restore picker has no type filter so safety backups can be chosen (M3); "Export all prompts JSON" names what /export holds (N1); small snapshots show KB (N3); the duplicated holding and save helpers are shared (N4); a dismissal test covers Cancel, x, Escape, a replacing dialog and leaving (M1). Refs #326 * test(coverage): raise the system-backups branch floor Refs #326 |
||
|
Vito
|
f9835aa4d6 |
fix(lora): allow civitai.red/.green and log why a CivitAI download failed (#340) (#341)
civitai.red and civitai.green (and their subdomains) are now allowed redirect targets under the civitai.com rules: HTTPS only, default port, no credentials, matched on the parsed host name, at most 3 hops, never with the key. Their www/api/image hosts are also allowed starting hosts, whose first request carries the key as image.civitai.com does. Each failed download now logs one warning with the URL without query or credentials and a short reason: HTTP status, refused redirect, hop limit, size cap, decode or save error, timeout or network error. |
||
|
Vito
|
8252d9d226 |
feat(system): Integrations and a staged LoRA import (System S10) (#336)
* fix(lora): detect LoraManager in ComfyUI's extra custom_nodes folders Auto-detection looked only in ComfyUI's main custom_nodes folder (#144). It now also searches every custom_nodes folder ComfyUI registers through folder_paths, main folder first, skipping folders that cannot be read. * fix(lora): stage imports before replacing the collection (#221) The LoRA import deleted the whole lora-manager category first and then saved each record in its own transaction, so a discovery, download or database failure left the previous collection gone or partial. The import now runs in three stages: discover and fingerprint the metadata files, stage every record (CivitAI downloads go into the image cache first, counted), then replace the category in one BEGIN IMMEDIATE transaction (replace_category_prompts). Any failure leaves the previous prompts, tags and image links exactly as they were. Prompts a saved pair uses block the replacement; an empty replacement of a non-empty collection needs a reviewed import. POST /prompt_manager/lora/import/preview reviews an import without writing or requesting anything; POST /lora/import/apply {token} runs it as the Activity job "LoRA reference import" (records admission, cancellable while it downloads, the safety backup policy before the replace, refused with plan-changed when the files changed since the review). The legacy POST /lora/scan runs the same stages with its SSE event shape. * feat(lora): the LoRA save takes the settings revision POST /prompt_manager/lora/enable accepts the optional expected_revision that GET /settings returns, so the System Save row refuses to write over a config.json changed elsewhere (settings-changed, nothing written). The legacy page sends none and still merges into what is on disk. * feat(system): configure ComfyUI and LoRA integrations natively System › Integrations replaces its hand-off with the concept's two surfaces. ComfyUI: only the send destination and renderer the bridge supports can be chosen; the other options and Return to canvas are disabled with the reason, and Host compatibility shows the renderer the open ComfyUI windows report (each bridge destination now carries renderer: classic or nodes2) (D106). LoRA & resources: detection, the enabled setting and the loaded trigger words read as separate facts, the read-only LoraManager path row follows the toggle (D106), and the CivitAI key is never filled in: typing drafts a new key, blank keeps the saved one and Clear saved key drafts its removal (D114). They save through the one Save row (/lora/enable with the settings revision). LoRA reference data → Review import shows the server's read-only review (replacement, what saved pairs block, downloads, backup box) and Start runs the LoRA reference import Activity job (D107), with its own Activity descriptor and Review again into Integrations when the files or pairs overtook it. * test(system): compare Integrations with the concept system-integrations-concept.spec compares System › Integrations (both themes, 1600 and 390 wide) with the concept, mapping only D106 and D114, and the LoRA import review with the concept's record-changing review, mapping D107; each comparison catches a removed row, a renamed label or option, a re-enabled unsupported option and a swapped token. On the real ComfyUI host, each bridge destination reports the renderer its window draws nodes with, before and after switching it, and System › Integrations shows it. * test(coverage): record the Integrations frontend floors coverage:update after the full frontend run: system-integrations.mjs at its measured floor, and the raised maintenance review, job view and maintenance floors. * test(system): wait for the renderer line before clicking Integrations rows CI Firefox (390 wide) clicked the LoraManager toggle while the renderer check's line above it changed text and moved the rows, so the click did not reach the switch. The specs now wait for that line to settle (a visible state) before interacting. * fix(lora): recheck the files before the replace and stop an abandoned scan Review fixes for the staged LoRA import (#221): - The metadata files are listed and fingerprinted again right before the replace transaction, so a file changed, removed or added while the downloads ran aborts the import with nothing deleted (System job: plan-changed, Review again; legacy scan: an error frame in its own words). The tests now change real files behind a held download. - A legacy scan whose page closes stops staging before its next LoRA and keeps its records hold until staging has stopped; CivitAI images are written to a temp file beside the cache entry and moved in place, so a half-written image is never seen as cached. - Start refuses with plan-changed when the integration was turned off or moved to another install since the review. - A failed job keeps the skipped count and the names of the LoRAs that could not be prepared. - The replace binds the category instead of one parameter per prompt (SQLite's variable limit), /lora/status says whether a detected install can be used, and the legacy scan logs expected refusals as warnings. * fix(system): partial saves, key focus and readable selects in Integrations A save that writes the LoRA part but not the settings part now keeps the saved part, adopts the new revision and highlights only what failed, so the next save needs no conflict. The CivitAI key status is a persistent live region, Clear saved key returns focus to the key field, and the key draft is dropped when the section is left. The send-destination and renderer selects keep their width on narrow screens. A detected LoraManager outside custom_nodes says it cannot be enabled. * fix(lora): keep the save outcome, honour the umask and sweep stale downloads Leaving Integrations drops the CivitAI key draft only when a key was typed, so the last Save's outcome survives and Sources still marks a refused source. The key status line is rewritten only when it changes. Cached CivitAI images are opened like a plain save, so they keep the umask's mode, and each import removes .pm-*.part files older than an hour from the image cache. A legacy scan stopped because its page left ends cancelled in Activity, and staging is awaited to the end even if the handler is cancelled again. * fix(lora): the import review takes a lease beside a restore The LoRA import review reads the library, so like the maintenance and restore previews it now runs under a request lease: it is refused with 409 busy while a restore runs, and a restore waits for it before it switches the file. Tests pin that a restore is refused while an import runs and that the review, the reviewed import and the admin page's scan are refused while a restore runs. |
||
|
Vito
|
28ad3a1a1f |
feat(system): backup and restore backend (System S8) (#335)
* refactor(database): stage restores before publishing restore_from_file is now stage_restore + publish_staged (database/ restore_stage.py). The stage lives in a .pm-restore-* folder in the library's own folder, is migrated, has damaged workflow documents repaired from an inline copy or cleared, is verified with every document read and is bound by its SHA-256. Publishing keeps a unique, standalone, 0600 safety backup (microsecond names), copies the pages through the backup API as before and then truncates the WAL, which stayed database-sized after a restore. The safety backup writer moves to database/safety_backup.py so the database layer and maintenance share it. * fix(database): read the verify streams one row at a time The provenance and review-evidence checks iterated their cursors; on a managed connection iteration prefetches 128 rows, so up to 128 large values were held at once while the docstrings promised one. Both now use fetchone. * feat(backup): review a restore, run it as an Activity job, verify backups - POST /restore/preview stages an upload in the library's folder and answers what it holds beside the live library, writing nothing live; DELETE discards it; at most two stages are held (429 too-large). - POST /restore/apply rechecks the live fingerprint (409 plan-changed) and starts "Database restore": exclusive, not cancellable, indeterminate. It waits for request leases, publishes the exact staged bytes, swaps the database, prunes safety backups and logs counts, repairs and the backup. - The legacy /restore runs the same stage and publish as an exclusive hold with its old response, so a restore and the health check, Move library and every other job refuse each other. - JobRegistry.lease: Needs review decisions and maintenance and restore previews are refused while an exclusive job runs, and a restore waits for them before it switches the file. - GET /backup holds a guarded slot, verifies a standalone snapshot before streaming it and removes the closed temp file; GET /backup/status lists the safety backups on disk and this session's verified download. * docs(testing): backup and restore contract * test(system): a restore refuses a running health check The health route test relied on a legacy restore interleaving with a running check; the restore is now exclusive and refuses it, naming the check. The diagnostics backup tests follow the backup handler into py/api/backup_restore.py. * fix(backup): drop stale restore previews off the event loop A restore drops the previews reviewed against the old library; their stage files are now removed in the executor, with the old database's connections. * test(backup): wait for the backup hold to end before reading its state CI (py3.11/3.12 on Linux, every macOS) read the registry right after the client had the last byte, before the handler left its hold and removed its temp file: the job was still running. The tests now wait on that state. * test(images): the scan dialog's Back up now leaves a finished backup hold A backup download now holds a guarded registry slot while it streams (#249 S8), so the jobs list holds that finished hold after the dialog's Back up now; the spec still proves Cancel starts no scan. * fix(backup): final review round for the restore and backup routes - I1: once the stage's copy replaced the library the restore succeeded: pruning, reopening self.db, dropping old previews and closing the old object only log, so the job finishes and the legacy /restore answers 200 instead of "the library was not changed". The generation is always bumped. - M1: a backup download holds its guarded slot only while the snapshot is written; the stream runs outside it, so a stalled download blocks nothing. - M2: a restore preview leases only its read of the live library, not the upload or the staging; a restore whose drain still times out keeps the reviewed stage under its token. - M3: the reviewed-preview store changes only on the event loop; stage folders are removed in the executor. - M4: a document whose body inflates completely while only its recorded hash or only its length disagrees is re-keyed (or merged into the document holding that text) instead of dropped. - M5: a backup download is refused only when the snapshot is not SQLite or fails integrity_check; a semantic check failure still downloads, logged, sent as X-PromptManager-Verify and recorded as not verified. - M6: a refused, failed or cancelled preview removes its stage, also when the cancel lands while a worker still stages; a timer removes expired stages without another request. - M7, N1: repair reads rows one at a time and counts distinct images. - N3: the live fingerprint never commits a transaction it did not open. * test(backup): close the old library the close-failure test leaves open CI (py3.12 Linux and the coverage gate) found the old object's executor-thread connections still open at teardown: the test patches its close_all to fail, and once the object is collected close_all_instances no longer reaches it. The test now closes it itself. * fix(backup): re-review round for restore repair, apply and backup cleanup - R1: when re-keying merges an intact body into a holder whose own body was damaged, the rewritten holder leaves the damaged list, so the clear pass no longer deletes it and unlinks the images just moved to it. - R2: a losing duplicate apply removes a stage only when the store still held it, never one another apply's job has taken. - R3: a restore that times out waiting for requests keeps its review only while it is unexpired and its slot is free; otherwise the stage goes and the job says to try again. - R4: a backup download cancelled while its snapshot is written removes the snapshot once the worker finishes. |
||
|
Vito
|
9ab25f9aaa |
feat(system): Maintenance UI (System S7) (#322)
* fix(gallery): clear only Prompt Manager cache entries (#222) The legacy gallery's Clear cache deleted every localStorage key starting pm_ or promptManager_ (the workspace theme and the dismissed update notice among them) and every CacheStorage cache whose name held gallery or thumbnail, which on ComfyUI's shared origin can belong to another tool. web/js/cache-ownership.js names what a clear may remove from an inventory of the real writers: every key Prompt Manager writes is a preference and no page opens CacheStorage or IndexedDB, so nothing is removed and the toast says so. A test fails when a new storage writer is not classified. The server thumbnail clear stays its own request; a refused or unreachable server changes nothing and is reported, and refused browser storage is reported as such. * feat(system): run reviewed maintenance natively System › Maintenance is native: the concept's notice and its Database and Images & cache surfaces replace the hand-off. - Integrity & relationship checks runs the health check; the report's Review missing links reviews a fresh Missing-file links plan. - Review plan shows the server's read-only plan (D100): scope, protected, backup policy and media lines, the listed records, how long the review holds and the backup box; Apply reviewed operation sends only its token and opens the job in Activity. busy names the running job, plan-used links the job that applied it, an expired or replaced review is reviewed again, and a lost answer says Outcome not confirmed and points to Activity. - Find duplicates starts the search; its view offers Review groups (D101): ten groups at a time, Keep radios per copy, each keeper sent as {group_id, root_index, path}; the dialog says History keeps the removed paths in its run records. - Thumbnail cache opens the thumbnail dialog; Metadata cache is disabled with its reason while nothing is owned (D102, Q5). - Activity descriptors for maintenance, duplicates and duplicate_cleanup. - The legacy duplicates dialog's View Full names its source (?root=). * test(system): compare Maintenance with the concept system-maintenance-concept.spec compares System › Maintenance, the plan dialog of each operation and the duplicates dialog with the concept in both themes and both viewports, mapped only by D100–D102, and proves a mutated row, label, danger mark or token is noticed. The duplicates dialog's Review duplicate cleanup is the concept's plain button. * test(system): Maintenance no longer hands off; raise coverage floors The System shell test expected Maintenance's D94 hand-off; the section is native now. coverage:update sets the new modules' floors from the measured run (cache-ownership 94, maintenance-review 94, system-maintenance 92 lines) and raises the floors other files passed. * test(admin): close the preview worker after the last job in teardown A scan publishes its last event, which ends the stream the test reads, before it hands its missing previews to the automatic worker. That hand-off can run after tearDownAsync closed the worker, so it starts a new one whose .part files were still open when the final drain checked for open handles (py3.10 on ubuntu, test_scan_reads_metadata_on_several_threads). The drain now closes the worker for good once every job has ended. * fix(system): address the S7 final review - I1: an apply answer is a refusal only when our server sent it (a known code, or a 4xx whose body says success:false); a proxy's 5xx, an unreadable 2xx or a dropped connection says Outcome not confirmed and reads Activity again. ApiError gains 'refused'; the server's own pre-start failure is coded 'apply-failed'. - M1: the lost-response test lets the apply run, then drops its answer. - M2: a failed apply records its code in the job's scope; Activity offers Review again for an overtaken plan (Maintenance for a duplicate cleanup whose sources changed). - M3: a failed duplicate cleanup toasts the server's reason. - M4: a review button holds while its preview is read (one preview per double press). - M5: the storage guard also catches bracket and property assignment and CacheStorage/IndexedDB opens; why the Toolkit build is skipped is written down. - M6: a refused storage listing says so on the Metadata cache row. - M7, M8: D100 text corrected; workflow_storage has its title. - M9: a busy start names the job the server named. * test: follow #319 in the Maintenance tests - The thumbnail dialog has only Scope since #319. - The admin teardown refuses new preview workers on its own API instead of closing with final=True, which since #319 also shuts the process-wide encoders that later test modules use. * fix(system): keep the server's search-again line on a duplicate review A duplicate cleanup review the server refuses with 409 sources-changed showed the generic "could not be reviewed" message; it now shows the server's "The sources changed since the search. Search for duplicates again." * fix(system): a plan with nothing to do offers only Close A reviewed maintenance plan whose scope is 0 records ("No duplicate prompt text") offered Apply reviewed operation, which started an empty job. It now shows its rows, "There is nothing to apply; nothing was changed." and a Close button. * fix(system): wrap long copy paths in the duplicates review A Keep label is the copy's source and path, and dataset folders often have long names with no space to break at, so the review scrolled sideways. The label now wraps anywhere inside its compare item (D101). |
||
|
Vito
|
2964a3b632 |
feat(thumbnails): WebP/JPEG XL thumbnails, server-side thumbnail settings and Convert outdated (#319)
* feat(thumbnails): thumbnail format, quality and size as server settings
Thumbnail format (webp default, jxl, jpeg), quality (1-100, default 80)
and size limit (128-2048 px, default 512) are server settings in the
config.json thumbnails section, saved with the settings revision. The
never-read gallery thumbnail_size is gone: older saves wrote its unused
default 256 into config.json.
One module names previews (<stem>_thumb.webp/.jxl/.jpg) and finds the
preview a source owns, preferring the configured format; legacy names
are still found. The encoder module writes every preview atomically and
verifies it decodes. JPEG XL needs no new dependency: Pillow (natively
or with pillow_jxl), ffmpeg with libjxl, or cjxl; without one the
settings refuse jxl (400 jxl-unavailable) and GET says why. Encoder
processes are tracked, bounded and killed at shutdown.
* feat(thumbnails): every preview writer and lookup follows the thumbnail settings
Automatic previews (capture and first view), the thumbnail job, the
legacy gallery routes and the history route write <stem>_thumb.webp
(or .jxl/.jpg) with the configured quality and size limit, for images
and videos alike: video frames, ffmpeg's included (a PNG on stdout,
never upscaled), and the placeholder are encoded like images.
Listings, duplicates, enrichment, the autotag scan and the history
route show any preview the file owns, the current format first, so
older _thumb.png previews keep working until converted. A request for
an older name answers the current preview. Previews are served with
an explicit image/webp or image/jxl type and the same cache headers.
The plan and status count missing, outdated (another format) and
ready previews; the missing job never writes beside an outdated one.
Removing a file removes its previews in every format; the clear
removes .webp and .jxl too. Encoder processes are killed at shutdown.
A preview is only a placeholder: inspection refuses paths under
thumbnails/ and always reads the original, and the inspector cover
accepts the listing's preview in any format the file owns.
* feat(thumbnails): Convert outdated thumbnails as an Activity job
POST /jobs/thumbnails {mode: "convert"} starts "Convert outdated
thumbnails" under the thumbnail job's kind, guarded admission, worker
threads and cancel at a file boundary. For each file whose preview is
in another format, it writes the new preview from the source (atomic,
verified; a video with no decodable frame fails instead of getting the
placeholder), then removes that file's old-format previews in its
thumbnails folder. A failure keeps the old preview; orphan previews
and links are never touched. The final line reports converted, failed
and bytes before and after. The plan counts it with mode=convert; the
legacy gallery stream refuses to attach to a running conversion.
* feat(thumbnails): thumbnail settings in Appearance and Convert outdated in the workspace
System > Appearance gains a Thumbnails surface: format (WebP, JPEG XL,
JPEG), quality and size limit, saved to the server with the settings
revision. JPEG XL is offered only when the server has an encoder, else
the option is disabled with the server's reason (D130); a note says
when this browser cannot show JPEG XL. Saving a new format converts
nothing: the row then says previews keep their old format.
Thumbnail maintenance gains the Convert outdated scope and names the
server's settings instead of its own quality and size fields (D131).
Overview's thumbnail row reads N missing, M outdated, K failed and
offers Convert outdated while previews are outdated.
The workspace decodes a tiny JPEG XL once: where it cannot, cards, the
Library and the inspector cover load the original for a .jxl preview,
and any preview that fails to load falls back to the original. The
viewer and downloads keep using the original (regression test).
README explains enabling JPEG XL; TESTING.md records D130 and D131.
* test(thumbnails): coverage floors for the thumbnail modules and WebP in page workflows
The two new frontend modules enter the coverage baseline (98% lines);
other floors only rise. The visual fixture's preview is WebP and
Appearance has its Thumbnails surface in the System shell check.
* fix(thumbnails): say when the thumbnail settings could not be read
Like Appearance's other server rows, the Thumbnails rows show 'Server
settings could not be read.' while the read failed, instead of only
staying disabled. Firefox exposed the gap: its JPEG XL note arrived
first and changed the row notes the page workflow counted.
* test(thumbnails): accept the legacy preview's spelling a case-insensitive disk finds
On Windows and macOS the '.jpg' lookup finds an older release's
b_thumb.JPG first, so the listed URL spells '.jpg'; it names the same
file there. The test now accepts either spelling and checks the URL
serves that preview.
* fix(thumbnails): address the final review of the thumbnail format work
- ffmpeg writes JPEG XL with -update 1, so a % in a file name is never an
image-sequence pattern, and reads videos as file:<path>. The clear and a
startup sweep remove temp files that interrupted writes left behind.
- Convert outdated converts each group of same-stem files once, from the
first source in the R2 order. An old preview that is already gone counts
as removed, and each file's bytes count once. A current preview is only
kept if it decodes.
- An encoder that fails makes the write detect encoders again once. Another
encoder is used if one is found; with none, the file is written as WebP
and the availability Appearance shows is refreshed.
- Changing the thumbnail format, quality or size while Convert outdated runs
returns 409 busy. If the format still changes underneath, the job stops
without removing anything.
- Legacy preview spellings are matched case-insensitively, so X_thumb.JPG
counts as outdated.
- History's cover uses the same lookup as Images. It serves .jxl only when
Accept lists image/jxl, and otherwise the original or the placeholder.
- A video whose thumbnail the browser cannot show keeps its own element
(Images) or its label (Library).
* fix(thumbnails): keep the new preview, probe encoders only when gone, sweep reused PIDs
- Convert never removes a path that is the preview it just wrote. It
compares the device and inode, or the resolved and normcased paths when
the disk has no inode numbers. This covers x_thumb.JPG on a
case-insensitive disk.
- A JPEG XL encode failure re-runs encoder detection only when the
encoder's binary or module is gone. A "no encoder" result is checked
again on the settings read, on a jxl save, and after 60 s, so one slow
probe no longer turns JPEG XL off until a restart.
- The startup sweep now also removes temp files that carry this process's
PID but predate its start. Containers reuse PIDs.
* fix(history): let the cover ask for JPEG XL when this browser decodes it
Firefox can decode JPEG XL without listing image/jxl in Accept. When the
workspace's probe decodes JPEG XL, it adds jxl=1 to the History cover URL.
The server serves a .jxl preview when that flag is set or when Accept
lists image/jxl. In every other case it falls back as before. The flag
accepts only 0 or 1, and the response stays no-store.
* fix(thumbnails): keep going past the per-batch cap until every preview is made
The missing, rebuild and Convert outdated jobs stopped after
MAX_THUMBNAIL_FILES (10,000) files and said "N more on the next run", but
nothing ever started that next run. The cap now bounds one batch. After
each batch the job plans again, leaving out every file it already worked
on, and continues until nothing is left, the job is cancelled, or a batch
makes no progress. That last case stops on files that fail every time
instead of looping over them.
Activity progress counts against the overall total, and the final line and
the legacy stream's complete frame report every batch. Admission, worker
threads, cancellation and the #316 rules are unchanged. The Thumbnail
maintenance dialog now counts all the work instead of saying "more on the
next run".
|
||
|
Vito
|
8bf759d6f8 |
feat(system): keep the library in ComfyUI's user folder (#318)
* feat(system): resolve every library path in one place utils/library_location.py decides where the library lives: the PROMPT_MANAGER_DATA_DIR folder (validated: absolute, never inside a folder ComfyUI or Prompt Manager serves), an explicit database.default_path as before, a node-folder library that was not moved, else ComfyUI's user/__promptmanager. The database, diagnostics, config.json, logs and the LoRA image cache all ask it, so a new install never writes into the node folder and an existing one keeps every path until the user moves it. "Keep it here" is persisted as library.keep_in_node_folder. * feat(system): move the library as a verified Activity job POST /prompt_manager/system/library/move starts "Move library", an exclusive, non-cancellable job (database/library_move.py). It checks the target volume's free space, copies the database with SQLite's backup API from one read snapshot, verifies the copy (integrity check, every workflow document, the same rows in every table), copies the safety and pre-migration backups and config.json (owner-only, checked by size and SHA-256), then publishes under the old file's write lock and switches the live process (every PromptModel, the review and history observers, the API's plans and health report). The old prompts.db and config.json are renamed *.moved-to-<folder> only after the switch. A failure removes everything it wrote and leaves the old library in use; a marker in the new folder makes a restart after any crash pick the right library. GET /prompt_manager/system/library and the Overview's "location" say where the library lives and why; POST .../keep persists "Keep it here". * feat(system): show where the library lives and offer the move D129. System > Overview's Library health gains a "Library location" row (the database path, why it is there, the "Keep it here" choice and any PROMPT_MANAGER_DATA_DIR problem) with Move library for a node-folder library and Details otherwise. While a node-folder library is not kept there, the workspace shows the concept's .notice above every page: "Keep it here" saves the choice on the server and stops the notice; "Move library" opens the shared review (From, To, Size, what moves, what happens to the old files) and starts the exclusive job in Activity. The notice and the Overview read the location again when the job ends. The Overview comparison maps the inserted row; a new comparison holds the notice to the concept's Sources notice (markup, tokens, alignment, both themes and widths) with a mutation check. * feat(system): close the browser server's log files before its root goes The logs now live in the library's data folder, which for the browser test server is <root>/user/__promptmanager; Windows could not remove the temp root while the server still held prompt_manager.log open. The API test's node folder is spelled in lower case because public paths are case-folded on Windows. * feat(system): show the library location on the legacy admin page A line under the admin page's statistics names the database path and why it is there (ComfyUI's user folder, PROMPT_MANAGER_DATA_DIR, config.json, or the node folder with a pointer to Move library), read once from GET /prompt_manager/system/library. * feat(system): prove a fresh host keeps its library in user/__promptmanager The execution suite starts one pinned CPU ComfyUI with no configuration at all on an OS-chosen port: the library is created in user/__promptmanager (database and logs), nothing but byte code is added to the node folder, the location route reports the default folder, and ComfyUI's /userdata API serves none of it (traversal spellings, a listing, a system-user header). * feat(system): document where the library lives and how it moves README: where the library lives, the upgrade path (Move library / Keep it here), PROMPT_MANAGER_DATA_DIR, explicit paths unchanged, and the way back to an older version. TESTING.md: D129 and the library location contract (resolution order, move steps, crash windows, coded refusals, Windows handling). * wip(system): library move fix round in progress * fix(system): retry a fallback image link that races the library switch The image monitor's fallback and recent-prompt links went through PromptDatabase.link_image_to_prompt, which logs and returns 0 on any error, so survive_switch never saw a "database is locked" or closed connection during Move library and the link was lost in both files. The monitor now links through link_image_to_prompt_or_raise (one transaction per attempt); link_image_to_prompt keeps its log-and-return-0 contract for other callers. A link that returns 0 is no longer logged as "Successfully linked image 0". A switch that fails after repointing the models now closes the old file's detached connections instead of leaving them to GC: switch() adds them to a list the mover owns. The mocked retry test is replaced by a real-lock race asserting the image row lands in the new library only. * fix(system): say which source is the default and where a symlinked library lives Sources & scans: with no source configured, ComfyUI's output folder now reads "ComfyUI/output · watching · default" on its own row instead of a separate note line. GET /settings marks each effective source with in_comfyui (its public path is relative to the ComfyUI base) so only those get the "ComfyUI/" prefix; the scan still sends the server's public path. Library location: a library in the node folder reads through the node folder's unresolved place in custom_nodes, so a node folder symlinked to a checkout outside ComfyUI shows custom_nodes/promptmanager/prompts.db instead of prompts.db (System library row, Move review, admin Library line). Never absolute or "..", else the shared _public_path, which sources and jobs keep using unchanged. |
||
|
Vito
|
fc3266d455 |
fix(thumbnails): generate thumbnails automatically and use them everywhere (#316)
* fix(thumbnails): generate thumbnails automatically and use them everywhere New outputs never got a preview, so the workspace grid loaded full-size originals (844 of 19,907 images on a live install, exactly the newest). - Listings always give images and videos a thumbnail URL; a missing preview is made by the serve route on first view (bounded, single flight per file) and falls back to the original, remembering files that cannot be decoded. - The image monitor queues each settled output's preview on one background worker that follows Worker threads. - A scan that leaves 200 or more previews missing starts the visible thumbnail job; fewer go through the worker. - Automatic previews wait for a restore, a duplicate removal, a sources change, the gallery's clear and exclusive jobs. - The inspector cover uses the preview; the Overview row counts previews that failed. * fix(thumbnails): keep the video fallback tested and stop the preview worker before cleanup A video listed without a preview still renders its own element and is released on refresh. Test teardowns and the browser reset stop the automatic preview worker before the temp tree goes, so no preview lands after cleanup (Windows). TESTING.md documents automatic previews. * fix(thumbnails): admit previews on the loop and never send a video to an img Final review fixes for automatic previews: - A video preview that cannot be made is the placeholder frame (no-store), never the video; OpenCV failures fall through to ffmpeg and the placeholder. Cards and the inspector cover swap back to the video element when the preview image fails. - View requests are admitted on the event loop (3 writers, one flight per file, at most 64 waiting; past that the fallback at once). Writes run on a small dedicated pool, never the default executor. The history route shares the admission, flight key and link checks. - Drive, UNC, empty and dot segments are refused as text; the folder listing runs inside the admission. - Holds are checked after the slot and again before the rename; only decode failures are remembered. - Bounded, logged shutdown: ffmpeg is killed, no worker restarts after cleanup, and a reload stops the old worker. - The listing says whether the preview file exists (thumbnail_ready) for the legacy gallery's dot; media in a nested thumbnails folder keeps its original; file names the server quotes match in the inspector. * fix(thumbnails): cover the image fallback card and document the admission An image listed without a preview (a nested thumbnails folder) still shows its original; TESTING.md describes the view admission, the video placeholder, the holds before the rename and the bounded shutdown. * fix(thumbnails): let an idle preview worker leave and fit case-insensitive filesystems The capture worker exits once its queue is empty, so no thread outlives its work; the next output starts a fresh one. The mixed-case source test expects no folder listing where the exact name already resolves (macOS, Windows). * fix(thumbnails): answer shared flights with the outcome only and restart a failed worker The history and serve routes share a flight key, so a flight now answers only the ensure outcome whichever route started it; the serve route derives the preview path and, for a fallback, looks its source up again on the view pool. A view joining a history flight no longer gets a 500 or an image answered with the video placeholder. The capture worker clears its running state however its loop ends, and a worker that cannot start leaves the file queued for the next enqueue. |
||
|
Vito
|
06099579e6 |
feat(maintenance): store each workflow once (#315)
* feat(maintenance): reviewed, read-only maintenance plans POST /prompt_manager/maintenance/preview reviews duplicate_prompts, optimize, unused_components or missing_links on its own mode=ro snapshot and answers a Plan (scope, protected reasons, backup policy, media, at most 50 detail rows). The fingerprint, targets and judged sources stay in a server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB, never truncated; a restart forgets them), bound to the database it read. The previews reuse the legacy queries: duplicate groups and the merge are shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the #239 protection predicates the plan reports, and missing links are judged by the health check's link_states. Legacy /maintenance keeps its shape; vacuum now reports success:false when VACUUM fails, and cleanup_orphaned_images skips links under unavailable sources. * feat(maintenance): apply reviewed plans as Activity jobs POST /prompt_manager/maintenance/apply {token} starts a maintenance job (records admission, not cancellable, groups changed/protected/errors). The token is consumed on the loop turn the job starts: a second apply is 409 plan-used naming the job, a busy refusal keeps the token, an unknown or expired one is 410 plan-expired, and a restore since the review is 409 storage-changed. With the safety backup policy on (maintenance.safety_backup, default true, saved through the settings route with the revision), the job first writes a backup into a temp file beside the database, makes it a standalone rollback-journal file, verifies it, fsyncs it and renames it onto a unique <db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file and nothing changed. The apply then rechecks the reviewed fingerprint inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of 400 ids, counts changed and protected records, logs the backup by name only and prunes old safety backups. With the policy off the log and the status line say "No safety backup". * feat(maintenance): find and remove duplicate files through Activity Duplicate image files move onto the job registry: - POST /prompt_manager/jobs/duplicates finds content-identical media files over every effective source as a cancellable read job ("Find duplicate image files", group groups_found); the server holds the groups per job and GET /prompt_manager/maintenance/duplicates/{id} pages them ten at a time. - The duplicate_files preview binds a removal to the reviewed keepers. Every keeper and removal is checked again: containment as text before any filesystem access, then not a link, still inside its source, same size and hash. The plan lists the database links that move. - Apply runs duplicate_cleanup (records, not cancellable): the promised safety backup, then per file a recheck, the unlink with every handle closed and its thumbnail. A changed file is kept, the first failed unlink stops the job with a coded message (never the OS text), and only removed files have their image links and favorites moved to the kept copy. - The legacy scan, stream, status and delete routes keep their shapes on the same registry kinds; the hashing, grouping and thumbnail removal are shared with them. Coordinator rulings on Tasks 15-16: - Safety backups share the restore pool of 10 `<db>.backup_*` files; the pool is now also pruned when an apply fails after its backup. - Plans report the safety backup's size and the free disk space, and apply refuses up front (507 insufficient-space) when the backup or VACUUM does not fit, so no job fails halfway for space. * fix(maintenance): never remove the kept copy; move records before unlink Final review of #313: - C1: the duplicate find tells files apart by (st_dev, st_ino), so a source listed twice, a symlinked source inside another and a hardlink are one file. The preview and the job refuse a removal that os.path.samestat matches with its kept copy. - I1: each file's records move to its verified kept copy in one short transaction before that file is unlinked; a failed move deletes nothing and a failed unlink leaves the records on the identical kept copy. The status line counts files removed and not removed. - I2: links are looked up under every spelling of the source (the real path, the configured and monitored folders, ComfyUI's output). - I5: image tags join the kept copy as their union and the removed copy's pending, undecided Needs review items are settled; the plan says so. - I3: the VACUUM disk check measures SQLite's real temporary folder. - M1: the token is looked at again with no await before the job starts. - M2: the backup decision is made once and handed to the job. - M3: backups are linked into place; startup sweeps interrupted temps. - M4: Optimize runs alone (new "exclusive" admission) and its plan says the database is locked while VACUUM runs. - M5: the reviewed duplicate merge is strict: collisions and Library titles are protected, and errors roll back. The legacy cleanup is unchanged. - M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests. - M9: the preview binds files by size, mtime and inode without hashing; the job hashes right before each unlink. * fix(maintenance): protect one file reached through two mounts Re-review of #313, round 2: - R1: a removal on another device than its kept copy, with the same size and mtimes within 2 s, is protected when its inode is equal or its path within its source is the same ("May be the same file reached through another mount or share"), in the preview and again right before each unlink. The job holds the kept copy open during the unlink, so on Windows deleting the kept copy itself through another path fails. - R2: a file's spellings are found by containment: every configured, monitored, ComfyUI-output or listed folder whose stat identity is a folder above the file. Parent spellings of nested sources and case variants match; favorites move under every listed root holding the file. Lookups stay index searches. - m1: a failed unlink's row says the links now name the kept copy. - m2: a volume with no file ids reads "Can't confirm this is a separate file on this drive". * fix(maintenance): stat duplicate candidates with os.stat Python 3.10's Path.stat uses the os.stat bound when pathlib was imported, so the find and the per-file checks could see different stat calls. The find now calls os.stat directly, like every other check; the mount tests failed on py3.10 only. * fix(maintenance): recheck the kept copy right before each unlink Re-review 2 of #313: - n1: the cross-mount guard also matches when one path within its source ends with the other on a whole folder name (sources listed at different depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount). - n2: after a file's records move and right before its unlink, the held kept copy must still be the file at its path (fstat of the hold against os.stat) and the cross-mount guard must still hold; otherwise the file is kept, its records move back to it, and it counts as not removed. * test(maintenance): the kept-copy race tests hold on Windows On Windows the removal job holds the kept copy open without FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during the record move is refused. The tests now assert that native outcome: the kept copy stays and the removal goes ahead. * feat(database): store each workflow and prompt graph once generated_images.workflow_data and prompt_metadata repeated the same large JSON text for every image of a run. Each distinct text now lives once in workflow_documents, keyed by the sha256 of its exact UTF-8 bytes and zlib-compressed; image rows reference it through workflow_document_id and prompt_document_id. - New writes (link_image_to_prompt, pair ingestion for scan, monitor and Needs review) store the new form. Old rows keep their inline text and read the same until the migration rewrites them. - Every reader goes through one accessor (read_stored_text, resolve_image_row): gallery and legacy image lists, image by id, prompt info for the inspector and metadata page, diagnostics. Payloads keep their shape. Inline text wins when present; a document that fails its length or hash check reads as unreadable, never as other text. - No trigger (restore validators refuse them): a document goes in the same transaction as its last reference (image and prompt deletes, missing-link cleanup, duplicate merges, duplicate-file link merges, ingestion updates), and a sweep at startup catches anything a path missed. - verify_database_file validates the table, the id columns, their foreign keys and indexes, and the stored values. * feat(database): migrate old workflow rows in batches, then reclaim space The workflow storage migration rewrites old-form image rows (text inline) to document ids, batch by batch: - Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or 16 MB of text, found by a rowid range search and read one row at a time. - Each text is stored or reused, then read back through the one accessor before its row changes; a text that would not come back identical fails and rolls back the whole batch. - A value that is not text stays inline and is counted as protected. - A crash or a stop leaves committed batches new form and the rest old form; both read the same, and a rerun continues. No media file is opened. - Unreferenced documents are swept at the end. The maintenance preview gains a read-only workflow_storage plan (records to rewrite, inline bytes where SQLite has octet_length, the Reclaim space and older-version notes). reclaim_space sweeps, runs VACUUM and truncates the WAL so the freed pages go back to the disk. * feat(maintenance): store each workflow once as an Activity job The reviewed maintenance plan gains workflow_storage. Applying it starts "Store each workflow once": kind maintenance, records admission, cancellable, groups changed/protected/errors. - The promised verified safety backup comes first; a failed backup ends the job with nothing rewritten. The apply's up-front disk check covers it. - Old-form rows are rewritten batch by batch on the event loop's executor, with progress after each committed batch. Cancel stops between batches; a failed batch rolls back whole and fails the job with a coded message. Either way every record reads as before and the next apply continues. - When every row was looked at, "Reclaim space" starts: exclusive (runs alone), not cancellable. It checks the disk for VACUUM's copies of the live pages, then sweeps, runs VACUUM and truncates the WAL. If another job holds the registry it does not start, and the finished migration says so and names Optimize database. - No image or video file is opened; tests track every open under the output folder and compare the media bytes and mtimes. * feat(maintenance): document the workflow storage contract TESTING.md's maintenance contract gains the workflow storage rules: the documents table and its two forms, the one accessor, garbage in the same transaction plus the sweeps, the batched migration and Reclaim space, the media and memory guarantees, and that a deduped backup cannot be restored into an older version. A test covers the preview on SQLite before 3.43 (no octet_length, as in older Windows Python builds): the inline-bytes detail is left out. * feat(database): keep the migration safe row by row and beside other writers Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe: - M1: a batch also ends after 0.5 s, and the loop leaves the write lock free for 0.15 s after each committed batch (longer than SQLite's longest busy poll), so the image monitor and run counting on another connection get in between. RED: a threaded writer with a 2 s busy timeout hit "database is locked" (4 of 5 links landed). - M2: each row runs in its own savepoint. A row that cannot be rewritten (undecodable text, a text that would not come back) is rolled back alone, stays inline and is listed in errors; the rest of the batch goes on. - M3: a reused document is decompressed and checked before its id is handed out, and repaired from the text (whose hash is its key) when damaged; verify_database_file reads every document and refuses a damaged one. - M5: the two legacy generated_images rebuilds refuse a file that already has the document id columns instead of dropping them. - The preview names the pre-migration backup and says it is always taken. * feat(maintenance): always back up before storing workflows once Final review fixes I1, I2 and I3 for the workflow storage dedupe: - I1: the migration always takes a verified backup, whatever maintenance.safety_backup says; the plan reports backup.policy "required" and the disk check counts it. - I2: that backup is prompts.db.pre-workflow-storage_<stamp>, outside the pruned prompts.db.backup_* pool and kept until the user removes it; the job's last line and the preview name it, and the legacy restore accepts it. README's changelog warns that an older version opens and restores a rewritten database but shows no workflows, and says how to go back. - I3: tools/workflow_text_digests.py writes the sha256 of every record's stored texts (read-only; inline before the rewrite, through the accessor after), so a diff before and after proves byte identity. - A row the migration could not rewrite is listed in the job's errors and named in its last line, and Reclaim space still runs. * feat(database): fail the batch on system errors and name a damaged document Re-review fixes R1 and R6 (database side), plus R3's helpers: - R1: a row is kept inline only for its own cause: text SQLite cannot decode, or a text that would not come back. A full disk, an I/O error, a malformed database or a bug now fails the batch and rolls it back, instead of listing every remaining row as "could not be rewritten". - R6: verify_database_file names the damaged document and says what to do (restore a backup from before the damage, or store an image with the same workflow, which repairs it). - R3: pre_migration_backups lists the kept backups, earliest first; the plan says "Nothing to rewrite" and names an existing kept backup, and the older version note says new images use the new form from the first start and names the earliest backup as the way back. * feat(maintenance): one kept pre-migration backup and nothing-to-do refusal Re-review fixes R2, R3, R4, R5 and R6 (job side): - R3: Apply refuses a plan with nothing to rewrite (409 nothing-to-do, no job, no backup). Only the first apply keeps a pre-workflow-storage backup; later applies (after a Cancel, a rerun) take a pooled safety backup, forced on whatever the policy, so one copy is never pruned. - R4: a second-connection writer test runs against the live job loop; with its pause removed the writer hits "database is locked" (4 of 5 links). - R5: the digest tool loads only database/workflow_documents.py by path (no package start-up, no log handler) and opens a stopped database with immutable=1, so it leaves no -wal/-shm; a live one is read through its WAL. - R6: a backup that fails on a damaged document says which one in the job line. - R2: README's changelog says images recorded by this version use the new form from the first start, and the way back is the earliest pre-workflow-storage backup (or a pre-update backup if the job never ran). |
||
|
Vito
|
f68e92b3b6 |
feat(system): maintenance backend (#313)
* feat(maintenance): reviewed, read-only maintenance plans POST /prompt_manager/maintenance/preview reviews duplicate_prompts, optimize, unused_components or missing_links on its own mode=ro snapshot and answers a Plan (scope, protected reasons, backup policy, media, at most 50 detail rows). The fingerprint, targets and judged sources stay in a server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB, never truncated; a restart forgets them), bound to the database it read. The previews reuse the legacy queries: duplicate groups and the merge are shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the #239 protection predicates the plan reports, and missing links are judged by the health check's link_states. Legacy /maintenance keeps its shape; vacuum now reports success:false when VACUUM fails, and cleanup_orphaned_images skips links under unavailable sources. * feat(maintenance): apply reviewed plans as Activity jobs POST /prompt_manager/maintenance/apply {token} starts a maintenance job (records admission, not cancellable, groups changed/protected/errors). The token is consumed on the loop turn the job starts: a second apply is 409 plan-used naming the job, a busy refusal keeps the token, an unknown or expired one is 410 plan-expired, and a restore since the review is 409 storage-changed. With the safety backup policy on (maintenance.safety_backup, default true, saved through the settings route with the revision), the job first writes a backup into a temp file beside the database, makes it a standalone rollback-journal file, verifies it, fsyncs it and renames it onto a unique <db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file and nothing changed. The apply then rechecks the reviewed fingerprint inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of 400 ids, counts changed and protected records, logs the backup by name only and prunes old safety backups. With the policy off the log and the status line say "No safety backup". * feat(maintenance): find and remove duplicate files through Activity Duplicate image files move onto the job registry: - POST /prompt_manager/jobs/duplicates finds content-identical media files over every effective source as a cancellable read job ("Find duplicate image files", group groups_found); the server holds the groups per job and GET /prompt_manager/maintenance/duplicates/{id} pages them ten at a time. - The duplicate_files preview binds a removal to the reviewed keepers. Every keeper and removal is checked again: containment as text before any filesystem access, then not a link, still inside its source, same size and hash. The plan lists the database links that move. - Apply runs duplicate_cleanup (records, not cancellable): the promised safety backup, then per file a recheck, the unlink with every handle closed and its thumbnail. A changed file is kept, the first failed unlink stops the job with a coded message (never the OS text), and only removed files have their image links and favorites moved to the kept copy. - The legacy scan, stream, status and delete routes keep their shapes on the same registry kinds; the hashing, grouping and thumbnail removal are shared with them. Coordinator rulings on Tasks 15-16: - Safety backups share the restore pool of 10 `<db>.backup_*` files; the pool is now also pruned when an apply fails after its backup. - Plans report the safety backup's size and the free disk space, and apply refuses up front (507 insufficient-space) when the backup or VACUUM does not fit, so no job fails halfway for space. * fix(maintenance): never remove the kept copy; move records before unlink Final review of #313: - C1: the duplicate find tells files apart by (st_dev, st_ino), so a source listed twice, a symlinked source inside another and a hardlink are one file. The preview and the job refuse a removal that os.path.samestat matches with its kept copy. - I1: each file's records move to its verified kept copy in one short transaction before that file is unlinked; a failed move deletes nothing and a failed unlink leaves the records on the identical kept copy. The status line counts files removed and not removed. - I2: links are looked up under every spelling of the source (the real path, the configured and monitored folders, ComfyUI's output). - I5: image tags join the kept copy as their union and the removed copy's pending, undecided Needs review items are settled; the plan says so. - I3: the VACUUM disk check measures SQLite's real temporary folder. - M1: the token is looked at again with no await before the job starts. - M2: the backup decision is made once and handed to the job. - M3: backups are linked into place; startup sweeps interrupted temps. - M4: Optimize runs alone (new "exclusive" admission) and its plan says the database is locked while VACUUM runs. - M5: the reviewed duplicate merge is strict: collisions and Library titles are protected, and errors roll back. The legacy cleanup is unchanged. - M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests. - M9: the preview binds files by size, mtime and inode without hashing; the job hashes right before each unlink. * fix(maintenance): protect one file reached through two mounts Re-review of #313, round 2: - R1: a removal on another device than its kept copy, with the same size and mtimes within 2 s, is protected when its inode is equal or its path within its source is the same ("May be the same file reached through another mount or share"), in the preview and again right before each unlink. The job holds the kept copy open during the unlink, so on Windows deleting the kept copy itself through another path fails. - R2: a file's spellings are found by containment: every configured, monitored, ComfyUI-output or listed folder whose stat identity is a folder above the file. Parent spellings of nested sources and case variants match; favorites move under every listed root holding the file. Lookups stay index searches. - m1: a failed unlink's row says the links now name the kept copy. - m2: a volume with no file ids reads "Can't confirm this is a separate file on this drive". * fix(maintenance): stat duplicate candidates with os.stat Python 3.10's Path.stat uses the os.stat bound when pathlib was imported, so the find and the per-file checks could see different stat calls. The find now calls os.stat directly, like every other check; the mount tests failed on py3.10 only. * fix(maintenance): recheck the kept copy right before each unlink Re-review 2 of #313: - n1: the cross-mount guard also matches when one path within its source ends with the other on a whole folder name (sources listed at different depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount). - n2: after a file's records move and right before its unlink, the held kept copy must still be the file at its path (fstat of the hold against os.stat) and the cross-mount guard must still hold; otherwise the file is kept, its records move back to it, and it counts as not removed. * test(maintenance): the kept-copy race tests hold on Windows On Windows the removal job holds the kept copy open without FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during the record move is refused. The tests now assert that native outcome: the kept copy stays and the removal goes ahead. * fix(maintenance): restore a backup of any size the disk has room for The legacy restore refused the maintainer's own 1.6 GB safety backup with "File too large. Maximum size is 100MB", a fixed cap since #153. Free disk space now decides: the upload, its staged copy and the swap must fit, under a 64 GiB ceiling. A request with a Content-Length that does not fit is refused before anything is written, one without as soon as its running size does not, with 507 insufficient-space and a message without paths. The upload still streams and its temp file is removed on every path. |
||
|
Vito
|
50703c960a |
feat(review): offer inspection-profile choices for untraceable batches (#312)
* feat(review): offer inspection-profile choices for untraceable batches Qwen image-edit batches run several KSamplers, each with its own TextEncodeQwenImageEdit prompt, all saved under one prefix. The scan's default profile knows neither the encoder nor the string nodes, so every candidate was unresolved and the files left Needs review as "could not be traced", although the inspection profile resolves every text. When the default profile is ambiguous without a choice and nothing links the file, the scan now resolves it with the inspection profile and, with at least two choosable candidates, queues the item from that evidence. The evidence records profile "inspection" and the semantics table version (SHA-256 of its canonical JSON); the item's resolver_version folds the profile, RESOLVER_VERSION and the table version, so a table change is handled like a resolver change (I2). Apply re-resolves the capture with the recorded profile, re-checks the chosen candidate hash and links through the compatible-only ingestion. The inspection profile never links on its own, ingestion keeps the default profile, and the live monitor never queues. * feat(review): supersede stale items and inspect each file state once A semantics table or resolver version change no longer leaves two pending items for one file. A new item supersedes the other items of the same bytes when every one is pending and undecided (revision 0, no decision row, the self-heal's safe delete). Beside a decided or advanced item nothing new is stored, and the rescan's stat shortcut skips those bytes under any version. The inspection item version now folds a narrower fingerprint: the table restricted to the graph's node classes plus its class-independent parts, so an edit about other classes leaves every item alone. The inspection profile runs once per file state. The scan resolves the untraced file at batch level, skips files already queued or decided at their current size and mtime under their version, and remembers files it found untraced in this process. The capture reuses the scan's result when it holds the same records. * feat(review): contain inspection failures, version the inspection resolver Final-review fixes for the inspection-profile review choices. - An inspection failure on one file is logged, counted in review_errors and reported unresolved; the scan goes on. - INSPECTION_RESOLVER_VERSION and the inspection profile's budgets are folded into the item version, so an inspection-only resolver change re-inspects and supersedes the item. An older version's pending, undecided item of bytes now untraced leaves the queue, so an item never stays stuck at 409. - Fingerprint tests: an edit of the encoder's own entry moves the version, an unused dict-keyed class leaves it. - The untraced memory is a plain dict of 16-byte digests (8.6 MiB measured at its 65,536-entry cap). - Only scans add items: the monitor's pass records nothing and only clears the items of outputs it links. Tests and the browser review fixture record through the scan's recorder. - record_review_item documents the byte-record refresh it commits when it refuses a new item. * fix(review): an unusable semantics table never retires recorded items |
||
|
Vito
|
6505d339aa |
feat(system): thumbnails on Activity (#311)
* feat(system): run thumbnail generation as an Activity job
GET /prompt_manager/images/thumbnails/plan counts what a run would do
(targets, fresh, remaining past the cap, source labels) and writes
nothing. POST /prompt_manager/jobs/thumbnails {mode, size, quality}
starts the "thumbnails" job: admission guarded, cancellable at a file
boundary, outcome groups generated / skipped / errors. size is the
longest edge (128-2048), quality the encoder, and rebuild writes fresh
previews again through the same atomic rename, so a failed file keeps
its old preview and originals are only read.
The legacy gallery stream now starts or attaches to the same job and
streams its unchanged frames. A client that goes away no longer stops
the work; the gallery's Cancel asks the server to cancel the job.
* feat(activity): per-kind job descriptors
One descriptor table (activity-kinds.mjs) decides each job kind's pill
verb, toasts, line, tiles, outcomes, openable rows and labels; the
store re-exports it. The scan and image tagging views are unchanged.
A thumbnail job counts Generated, Skipped and Errors, reads
"Generating thumbnails" in the pill and "Thumbnails finished · N
generated · M errors" when it ends.
Any other kind (the health check, a held legacy-page operation) gets
the generic descriptor with no tiles and no Results, instead of the
scan's zero tiles. A running job without a known total shows an
indeterminate progress bar and no percentage, and only a job the
server can cancel shows Cancel.
* feat(system): review and start thumbnail jobs
Sources' Thumbnail generation -> Review options and Overview's Review
thumbnails open the concept's Thumbnail maintenance dialog: scope, quality
(preselected from Image loading quality) and preview size limit, a plan
line from the read-only thumbnail plan (D99) and Start, which starts the
thumbnails Activity job and opens it. A size outside 128-2048 requests
nothing; a refused start shows the busy link toast.
* fix(system): thumbnail job review fixes (#249 S5)
- The legacy stream's first frame is job {id, started}, sent after the
subscription so no runner frame is missed; it reads the registry after
prepare, so a job started in that gap is attached to, not refused.
- Every exit sets a truthful final status line; nothing to do never
reads "Processing file 0 of 0".
- Cancel is checked per file during the planning walk; a job cancelled
there publishes no start frame.
- The ffmpeg fallback keeps the chosen quality (-q:v) and the aspect
ratio; the placeholder uses the chosen JPEG settings.
- A locked preview on Windows reads "Preview in use by another program;
the old one was kept", not the raw WinError.
- Thumbnail work no longer defers the monitor's Needs review recording:
it writes previews only, never records.
- The disconnect test now proves the job outlives its stream (one worker,
9 files, three batches); the attach test waits on an event.
* fix(gallery): cancel the stream's own thumbnail job by id
The settings panel and the startup dialog cancel the job named by the
stream's job frame, and only one their stream started. A Cancel before
the frame waits for it, a double Cancel posts once, and a job another
tab or System started keeps running. The startup dialog now settles on
Cancel instead of staying on "Cancelling...".
* fix(activity): indeterminate pill bar and the nothing-to-do line
The pill's mini bar sweeps while a job's total is unknown instead of
showing an empty fill. A finished thumbnail job with nothing to do shows
the server's final line. Coverage floors raised for gallery.js and
activity.mjs.
|
||
|
Vito
|
b4c7a15045 |
feat(review): Needs review page, live queue and concept comparison (#288)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* feat(review): bind queue cursors to the storage epoch only
A decision or a new item (the monitor records one per generation) no
longer makes an open cursor stale: the (created_at, id) keyset stays
ordered when items leave or are appended. A restore still answers
storage-changed and an unverifiable cursor review-conflict.
* feat(review): Needs review page, SAVED VIEWS row and Activity hook
The concept's Metadata review page at #/review: the notice with the
exact pending count, the queue's media cards (shared with Images through
image-card.mjs), and the Resolve the association panel with the sampler
context, the chosen path's prompt blocks, Leave for later and Apply
association. Apply takes a fresh capture; refusals keep the selection
and explain inline; an unanswered decision is resent once. SAVED VIEWS
lists Needs review after Favorites with its count (retires D50), and an
ended scan that recorded review items offers Review metadata in Activity
(D87). Queue items carry the card's media url.
* feat(review): re-read an open Needs review page when its queue changes
An open Needs review page now follows the queue without polling. A new
server-sent event stream, GET /prompt_manager/review/events, sends
"ready" when it opens and "change" after every committed queue change:
an item the scan or the monitor recorded, a spelling refresh, a decision
from any tab, and a restore. database/review_changes.py notifies after
the commit from whichever thread wrote it. The page re-reads its first
page on each event, keeps the open item, its choice and an unanswered
decision, and re-reads the SAVED VIEWS count with it. When the open item
left the queue, the next one opens with a note. A keepalive comment
every 25 s lets the server drop a closed tab; shutdown ends every stream.
Leaving the page mid-decision now releases that decision's capture.
D-NEW-F1 is renumbered D119 in markers and TESTING.md.
* test(review): Needs review page workflows
The review fixture now seeds a real monitor-fallback conflict (the
monitor pipeline links a new output to the most recent prompt, then
queues it), a limited item and a file under no current root, and can add
an output through the monitor or leave one for a metadata scan.
New workflows: Apply links the chosen pair to that file only; a real
association conflict leaves the links unchanged and explains; a limited
item and an unavailable file can only be left unresolved; a tag save
leaves the item pending; keyboard focus, labels and touch targets; an
open page shows items the monitor and a scan add and a decision made in
another tab; leaving the page releases captures, the change stream and
requests. The faked 409 conflict test is replaced by the real one.
* test(review): cover the card Tag button and raise the review floors
The card's Tag on Needs review now has a workflow, and the page's queue
change hook has no unused default. review.mjs and review-state.mjs
floors rise to their measured coverage; no other floor changes.
* feat(review): keep the loaded depth on live re-reads and read once per visit
A queue change re-read only the first 40 items, so the pages Load more had
shown vanished and the page jumped. Each re-read now reads page after page
until it holds as many items as were shown (a restore met after the first
page reads again from the start), so the cards and the scroll position stay.
The stream's ready made every visit read the queue twice. List reads now
answer the queue's version (library_epoch, review_revision) and ready names
the version read after subscribing; the page re-reads on ready only when the
queue it shows is older. The SAVED VIEWS count follows the page's total, so a
change costs no count read. A stream counts as open only once ready is sent.
* test(review): compare Needs review with the approved concept
review-concept.spec.mjs measures the whole Metadata review page as a tree
in the concept and in production, on the review fixture's new
concept-shaped queue (the concept's first three image names and sizes and
its first prompt). mapConceptReview applies only D29, D119, D3, production's
hooks and accessible names and sample data, and the page then matches at
1600x1100 and 390x844 in both themes. The gallery layout follows the
concept at every breakpoint, the SAVED VIEWS rows match with Needs review
after Favorites, the queue's Tag image dialog matches the needs-review
reference, and a dropped notice border or reordered actions fail.
The comparison found one gap: an unavailable file's card lost its media
area. Its empty media button now keeps the concept's 1.33 ratio (D119).
TESTING.md records the comparison, the extended D119 and a summary of the
image-tagging slice (routes, S1, capture limits, the record-changing job,
the no-download rule, the restart note). A ready whose version cannot be
read re-reads the queue (covered by a page test).
* test(review): give the multi-route shell comparison a fitting time budget
The dispatched CI run failed "shell geometry, tokens and controls match the
concept 390x844 light" on WebKit with a 30 s test timeout (no assertion
failed). Each of these tests makes twelve page loads with captures, hovers
and focus checks: 15-19 s on a CI WebKit runner on main, 22-31 s on this
run's loaded shard. With 2 CPUs and 3 workers it fails locally the same way
(6 of 12 timed out); with a 90 s budget, as the other multi-state concept
comparisons have, all 12 pass.
* fix(review): coalesce queue change events and answer queue thumbnails
- The change stream sends the first notice at once and collapses the
notices of the next 0.4 s into one trailing change, so a busy scan costs
an open page one re-read per 0.4 s at most (COALESCE_SECONDS).
- Queue items answer thumbnail_url, the same URL the Images listing builds
(one shared helper, media_thumbnail_url), or null without a thumbnail.
- The "duplicate and refused decision send nothing" test spies on
notify_review_change instead of reading the next event.
* fix(review): keep the live queue right through applies, Load more and lost streams
- A change heard while a decision is in flight is owed and read once the
decision is refused or unanswered (I1).
- Load more has its own read: during a re-read it waits and continues from
the re-read queue; a re-read started during Load more takes the click
over, and a failed re-read leaves Load more clickable (I2, M7).
- A stream the browser gave up on (503, proxy error) reopens after 3 s,
doubling to 30 s, with a quiet "Live updates are paused" line while it is
down (I3, D-NEW-F2).
- A hidden tab closes its stream and a visible one reopens it; the ready
version keeps that to at most one read (I4).
- Apply fences on the detail's epoch and a re-read under another epoch reads
the open item again (M3); leaving during the observe releases its capture
(M4); an unanswered decision that landed finishes as recorded (M5); cards
follow a re-read's url, target and availability and show thumbnails
(M6, M10).
- The unreadable-ready test asserts exactly one re-read (M1); the panel's
long-file-name wrap carries its D119 marker and TESTING.md entry (M8).
* docs(review): number the live-updates-paused deviation D120
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(activity): drop Cancel job from a job that had ended before Activity opened
Since #289 the Activity list removed an item's actions only when they were
isConnected. On the first render the item is not in the page yet, so a job
that had already ended when Activity opened kept its "Cancel job" (and a
running job's progress was moved after its actions, then back). The list
now asks parentNode instead. The review spec "a scan job with review items
offers Review metadata in Activity" failed on this before the change
(Expected 0 buttons on a finished job, Received 1) and passes after it.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* feat(review): open Needs review from the System Overview review row
Review metadata on the Overview's review row now opens #/review, the
concept's go-review, instead of staying aria-disabled.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): announce items that leave the queue, and seed only queueable files
#284 now queues a file only when its final outcome is ambiguous and unlinked,
and it removes the pending items of files that are now linked. That removal
is a queue change, so it calls notify_review_change after its commit, like
recording and decisions: an open Needs review page re-reads its queue, and
its count follows.
The review browser fixture wrote graphs that the scan's fallback links, and
it produced the conflict item through the monitor's fallback link. Neither is
queued any more. Its files now carry ComfyUI's Infinity stamp, which the
legacy parser cannot read, so they stay unlinked with the same two
candidates. The conflict item is queued first and its file is linked to
another prompt afterwards. "monitor" additions go through
record_review_file from a worker thread.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(review): seed the limited item from a graph with more candidates than stored
An A1111 block with two negative lines offers no candidate to choose, so
Needs review no longer queues it (#284). The fixture now records a graph of
17 samplers, which the recorder stores limited.
|
||
|
Vito
|
3f11ea6168 |
feat(annotations): Needs review queue backend (#284)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(scan): pin only the per-source scope keys beside the review counters
* test(system): match the scan scope keys beside the review counters
|
||
|
Vito
|
69de20a427 |
feat(system): Sources & scans (#309)
* feat(system): report configured, effective and watched sources
GET /prompt_manager/settings adds settings.monitoring_enabled, a sources
block (configured roots with valid/code/reason, effective roots with their
listing index and watch state, auto_detected) and the monitor state
(present, observer_alive, scheduled). Every path is a public path.
A source is "watching" only while a live observer scheduled it, "manual"
when monitoring is off or the process has no monitor, and otherwise
"restart_pending".
POST accepts monitoring_enabled as a strict bool, saved under
gallery.monitoring.enabled; restart_required_fields names it when the
running observer does not already match. A save that changes the sources
runs as a "Source change" records hold, so it is refused with 409 busy
while a scan runs. Case variants of one folder are saved once.
POST /prompt_manager/settings/validate-source {path} checks a path
lexically (no "..", no control characters, native path forms only) and
then against the gallery-root rules, answering a code and reason, and
writes nothing.
* fix(system): no existence oracle in source validation
A gallery root's containment in the allowed ComfyUI folders is now
decided before its existence and type, so a path outside them answers
"outside-allowed" whether or not it exists. validate-source also folds
the filesystem-root and home-directory codes into "outside-allowed", with
one reason, and logs the detail server-side only. Paths inside the
allowed folders still say "not-found" or "not-directory". The legacy
settings save shares the same check.
* feat(scan): scan one selected source
POST /prompt_manager/jobs/scan accepts roots: indexes into the effective
sources, strict non-negative ints (no bools or floats), deduplicated, at
most 64, non-empty when present. An index past the sources is a 400
"unknown-source" and starts nothing. The job walks only the selected
sources, resolved when it was requested, and keeps each file's listing
root_index. Its scope records files, roots (null for every source) and
output_folders, the number of folders walked, which the log's first line
counts. The legacy /scan route still scans every source, and attaching to
a scoped job keeps its payload keys.
* feat(system): manage sources and start scans natively
System > Sources & scans renders the concept's section on the server's
sources: one row per effective source with its watch state and a scan
scoped to that source, the 3.2.0 update notice with the legacy dismissal,
Add source checked by validate-source, Remove, monitoring and worker
threads as host drafts behind the one Save row, and Refresh files.
A save refused by a running scan names the job. Thumbnail generation
hands off to the legacy gallery until S5.
Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md.
* fix(security): decide source containment as text before any filesystem access
Every caller (validate-source, POST /settings, config.json at load) now
refuses a source through one check, sources.source_problem: the form
check first, then GalleryConfig.gallery_root_problem, which compares the
normalised spelling with the allowed folders before anything resolves,
stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a
device or any outside folder is refused with no SMB request, no
executor stall and no existence or timing oracle; only then is the path
resolved and contained again, which still catches a symlink out. The
filesystem root and the home directory answer outside-allowed with the
same reason as any outside path, so no caller learns the home path.
Also:
- POST /settings runs the form checks and parses the roots off the loop;
an unusable source is a coded 400 invalid-source naming its public
path (as text) and its index.
- ComfyUI folders outside the base directory are path anchors, so their
sources round-trip and never collide by folder name.
- A per-source scan sends each index with its public path; a stale or
shifted index is 409 sources-changed and starts nothing.
- A source still watched after monitoring was switched off is
"stopping".
- Roots are compared as folders, not spellings, so an unchanged legacy
resend takes no source-change hold.
- Tests: no realpath/stat/lstat of an outside path (native and Windows
forms with ntpath fakes), the alike answers with /, ~ and a link to ~,
an event instead of a timer wait, and a teardown that drains jobs
before the temporary database goes.
* fix(system): say what Sources saved, refused and still watches
- A per-source scan sends the source's public path; when the sources
changed since they were read the toast says the server's reason and
the list and saved values are read again.
- "watching until restart" for a source the running monitor still
watches after monitoring was saved off, and the status adds "Restart
ComfyUI to apply: Watch for new outputs." from restart_required_fields.
- A save refused for one unusable source marks that row and the status
is the server's message naming it.
- A valid configured source the effective list holds under another
spelling reads "same folder as another source".
- The busy toast names the job the 409 named (ApiError keeps a
validated running job and refused source), falling back to the list.
- The Add source dialog's listeners live on the page scope and Enter
adds; its markup is compared with the concept (D127).
- TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the
Sources contract describes the lexical containment, invalid-source and
sources-changed.
* fix(security): guard the Windows form probes on a token of their own
The no-filesystem guard matched "C:", which on a Windows runner is also
the temporary ComfyUI tree the save may resolve for its trusted first
entry. The probe paths now carry a token no trusted directory holds.
* docs(system): number the Sources states deviation D128
|
||
|
Vito
|
cbc53ee9f1 |
feat(resolver): trace multi-stage chains through pass-through nodes (#308)
* feat(resolver): trace multi-stage chains through pass-through nodes The inspection resolver (server and local Inspect) now compares a sampler with the earlier sampler its latent continues through image-space pass-through nodes: decode, image filter, model upscale, scale and encode, listed in a new validated `lineage` section of the semantics table. Equal prompts resolve (hires-fix, refiner); different prompts are multistage-differs. A later stage whose positive adds conditioning to the shared base prompt through an additive composite (ConditioningCombine or Concat) keeps the base pair, the text legacy linked (three-stage Flux). Image-edit batches stay ambiguous and run-time text stays untraceable. Ingestion keeps the default profile (D1). A card no longer shows a seed when the inspector names a chain stage whose seed differs from the parser's pick. * feat(resolver): leave out run-time PromptManager parts, labelled Maintainer option A: when a PromptManager prepend_text or append_text is fed directly by a node whose output is decided at run time (LoraManager's TriggerWord Toggle, now a runtime class, or PromptSearchList), the prompt resolves to the node's own text, as legacy showed. The side's source is runtime-parts-omitted and the resolved reason says that text added at run time, such as trigger words, is not included; the prompt block shows that line (D-NEW-R1). The toggle's output is never computed. Literal and traced parts are combined as before; an indirect run-time part, a run-time main text or an unknown feeder keep their outcome. The browser port changes identically. Ingestion is unchanged. * fix(resolver): keep run-time and added-conditioning labels honest A text that left out a run-time PromptManager part is now labelled when any node on its traced path omitted it, so the label survives encoders, concatenations and chained PromptManagers. Comparisons include what a text leaves out, so a partial prompt never equals a complete one: a branch with trigger words and one without stay ambiguous, and such stages are multistage-differs. Only classes listed as run time are left out. The added-conditioning floor labels its positive (D126, "Conditioning added at a later stage is not included."); the run-time line is D125. A card hides its seed only when the stage the inspector names reads a different seed than the parser; an agreeing seed is kept, as on main. The browser port moves its lineage, latent and part helpers into local-resolver-stages.mjs, keeping parity. |
||
|
Vito
|
7c25b351f7 |
test(comfyui): refuse a real-host port another run holds (#310)
The real-host config waited on the /system_stats URL, which any server on
the port answers. When two runs started on 8189 within a few seconds of
each other, the second run's ComfyUI failed to bind ("Port 8189 is already
in use"), but Playwright had already seen the first run's host answer and
ran the second run's tests against it. When the first run finished and
stopped its host, the second run failed mid-test with ECONNREFUSED,
ECONNRESET or ERR_CONNECTION_REFUSED, and passed when rerun alone.
The launcher now refuses a port that is in use, and reports ready only
when /system_stats names its own base directory. The config waits for
that line instead of the URL, and every test checks again that the host
answering the port is the one its run started.
|
||
|
Vito
|
785b744554 |
feat(system): Appearance section and browser display defaults (#307)
* feat(system): share browser display defaults The workspace reads and writes the legacy gallery's gallerySettings object through preferences.mjs: workspaceLayout and workspaceDensity join the shared keys, a write re-reads the stored object and writes only the changed keys, denied storage or a full quota keeps the change for this tab and says so, and storage events report changes from other pages. The legacy gallery Save keeps keys it does not edit, and the viewer reads its video defaults through the shared module. * feat(system): Appearance defaults used by Images and the viewer System > Appearance is the concept's Browsing & display surface on real values: the browser rows edit this browser's display defaults through the shared gallerySettings object, and Open from ComfyUI and the node search auto-hide save to the server settings with their revision. A new Images visit starts from the default layout and size (the current view and saved views keep theirs); metadata off waits for Load metadata; cache off re-reads the shown file; paths off hides the Path row; the viewer reads the video defaults. D97 adds card details, collapsed viewer details and Reset display defaults; storage refused says Saved for this tab only. * fix(system): flag changes made while System was closed and keep embedded galleries fixed Final review fixes for System S3 (Appearance): - I1: the on-demand metadata deviation is D124 (TESTING.md, inspector markers); TESTING.md names the spec that pins it, since no concept comparison renders that state. - I2: System checks the display defaults for conflicts on mount, not only on storage events, and a browser write refuses a draft another page overtook, so a draft is never saved over another tab's value without the conflict status first. New two-tab spec. - I3: the layout and size defaults start the Images page only; embedded galleries (a prompt history's Shared gallery and dialogs) keep Grid/Comfortable. - M1: host rows whose /settings read failed say "Server settings could not be read." - M2: a rejected timeout shows the "Some changes were not saved" status. - M3: the browser loads on mount and on storage events never reject unhandled. - M4: the next accepted write stores the tab-only changes and clears the overlay. - M5: cache off never re-reads a source-changed selection, and refresh() never loads a deferred one. - M6: the System fixture refuses a config.json outside its temp root and restores it with os.replace. - M7: the inspection read counts wait on a request barrier, not frames. - M8 recorded as an open saved-views question in TESTING.md. * fix(system): answer a failed System read only after its other reads finish CI (py3.10 on ubuntu) crashed with a segfault inside SQLite in test_statistics_failure_is_an_error_envelope: the statistics and overview routes used asyncio.gather without return_exceptions, so the 500 answered on the first failure while get_statistics still ran in the executor, and the test tore the database down under it. Both routes now await every read before raising the first failure (failing before, passing after: TestFailedReadsSettleFirst). The frontend coverage merge failed on workspace/activity.mjs branches (86.3 < 87): no test asserted Activity's empty and failed-read states, so their branch was only reached in passing, depending on timing. A spec now asserts both. |
||
|
Vito
|
c86739178d |
feat(system): System page shell, review dialog and Overview (#298)
* feat(system): read-only health check and truthful overview counts POST /prompt_manager/system/health runs the health check off-loop as a read registry job of kind "health" (one at a time, beside a scan, not cancellable, no file outcomes) and keeps the last report; GET returns it or null. The check reads one snapshot on its own mode=ro connection, closed in finally: integrity_check(100), foreign_key_check, stored hashes against utils.hashing, duplicate pair rows, the legacy consistency queries (now shared with check_consistency), and image links whose file is gone, counted only under readable sources. An unavailable source, or the absent Needs-review queue, makes its check unavailable and the report incomplete instead of producing a count. GET /prompt_manager/system/overview returns Library-eligible positive prompts and active negative variants, indexed images, the last health summary or null, backup null, the Needs-review count null, the effective sources with whether they were auto-detected, and the thumbnail plan's missing count. GET /prompt_manager/system/statistics lists runs and average rating per category of Library parents, with the /stats totals. * feat(system): native System route, drafts and shared review dialog - #/system/<section> renders the concept's System shell: section nav, settings layout and Save row with truthful status (D92); each section not delivered yet hands off to its legacy tools (D94). Overview's System card opens #/system/overview (Q12). - A module-level draft store saves browser, then host (expected_revision), then runtime; failed domains stay dirty, a stale revision is a conflict. - System row builders (setting, operation, section nav) on the scope. - openReviewDialog and backupNotice extracted from the scan review, which is now a thin caller with identical markup. - Overview images gain a "rated" count (images whose linked prompt is rated); the missing-files health check reports per source, so partial availability is visible. * feat(system): native Overview with health report and statistics System > Overview renders the concept's metrics and its Library health and Workspace preferences surfaces from GET /system/overview: real counts, "Not checked" until a health check ran, no invented review count, backup or thumbnail status (D93). Explore opens "Usage statistics" with the real runs and ratings per category and the Library totals (D96); the statistics route now also returns rated_images. Maintenance shows the concept's notice with Run health checks above its hand-off: the read-only check runs as a registry read job and opens the "Health report" with a chip per real result, missing files per source (D95). A load failure shows Retry and no zeros; late answers never repaint a newer section. Adds the System concept comparison (shell, Overview, both dialogs, the notice, mutation checks), the System workflow spec and its fixture hooks, and the TESTING.md "Native System" section. * test(system): coverage floors for the native Overview Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes. * fix(system): keep health reports per database and off the event loop - A restore that replaces the database forgets the stored health report, and a check that started before the swap is not kept (database generation stamped at start). - The Overview no longer walks every source for the missing-thumbnail count; the page reads /images/thumbnails/status on its own. - Image sources for the health check and the Overview resolve in the executor; a failure to start the check answers the coded envelope. - Any exception fails only its own check; missing files reads its rows in the snapshot and stats the files after it ended. - The read-only URI keeps Windows UNC paths (empty authority). - An integrity count at SQLite's limit is flagged as capped. * fix(system): thumbnail row read, tracked health job and guarded host save - The Overview's thumbnail row is its own read of the thumbnail status: "Checking thumbnails…" until it answers, unavailable when it fails. - The health job is tracked in the Activity store (new store.track) and followed through its event stream instead of polling every 150 ms. - A host save before a read returned a revision fails without a request; System reads the host values on mount and after a conflict. - "100+ problems found" when the integrity count is capped. - D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone. - The concept spec builds the health report from the concept's own rows and asserts the full unmapped path list; surface tops pin the 18px offset. - TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI and the new behavior. Coverage floors raised for the touched files. * fix(system): make two health tests portable to Windows - Windows forbids "?" in file names, so the URI-character open test uses "#", "%" and a space there. - ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of raising, so the bad-path test raises as POSIX does to reach the missing-files failure path on every platform. |
||
|
Vito
|
b50bf862f0 |
fix(inspect): hold a route change until a starting video settles (#294)
* fix(inspect): keep a replaced video in place until its pipeline has started WebKit (GStreamer) deadlocks the page when a video leaves the page or the viewport before its pipeline has started: the main thread changes the pipeline state and waits for the source's streaming thread, which waits on the main thread in a synchronous bus handler (native stacks of the hung WPE web process in the pinned image). Removing, hiding or merely pushing the video below the fold all hung; none of pause(), removing src, load() or the revoke was the cause on its own. A video replaced while still starting now stays where it is, inert and aria-hidden, with the new preview inserted after it, and the session's revoke of its URL waits. Once it reports metadata or an error (or after 15 s) it is paused, loses its src, is reset with load() and removed, and only then is its URL revoked. Every released video is reset with load(), so no decoder is kept per replaced video. The loadstart replacement spec now runs on WebKit too (15 fresh pages, replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause. * fix(inspect): hold a route change until a starting video settles A route change within the workspace detaches the Inspect page, and WebKit (GStreamer) deadlocks when a video still starting is detached. Route changes at loadstart + 0-7 ms hung 4-5 of 120 fresh pages (14 of 120 with no page code releasing anything, so the detach itself is the trigger). Inspect now exposes leaveGuard(): a promise while a video on the page is still starting (current or replaced). The shell renders the next route once it settles (metadata, error, or the 15 s bound); the latest route wins. Route changes at loadstart + 0-7 ms now hang 0 of 120. Not changed, with numbers in TESTING.md D62: - A replaced video still shows above the new preview until it settles. Same-slot stacking changes the video's own box and hung again (absolute 5/120, absolute + opacity 0 8/120, one-cell grid 4/120). - A real page leave cannot wait. Navigating away hung 8/120 and reload 22/120 even with no page-side release, so it stays a known limitation. |
||
|
Vito
|
4f1c6e236b |
test(browser): report a slow test reset with every thread's stack (#299)
* test(browser): report a slow test reset with every thread's stack On #291 (run 39904288262, WebKit shard 2/2) one /__test__/reset {chronological_history} waited 16 s on its worker's server, with nothing on the server's stderr. The cause did not reproduce: the same worker's file sequence (library, library editor, tagging lifecycle, visual) at 2 CPUs three times, and WebKit shard 2/2 at 2 CPUs, kept every reset under 2 s. A reset still running after 2 s now reports, while it waits, every thread's stack (faulthandler), the default executor's threads and queue, the open connections and WAL size, and on Linux each thread's scheduler state and kernel wait channel. When it ends it reports the total and how long each executor call queued and ran. The lines carry a "pm-slow-reset| " prefix and the worker fixture copies them to the job log, so the next one shows up even in a test that passes. The report never reads another thread's connection. Reading even in_transaction on a connection busy in SQLite blocks on its mutex with the GIL held: a first version of this report froze the whole server until the 30 s busy timeout. A test holds a real write lock behind a reset and checks the server keeps answering while it waits. * test(browser): name each thread in the slow-reset stacks Before Python 3.14 faulthandler prints thread ids only, so the report now lists which name each id belongs to. CI on 3.10 to 3.13 failed the unit test that looks for the blocked thread by name. * test(browser): keep the slow-reset summary on a coarse clock On Windows the report timer can fire a few milliseconds short of the threshold, and the summary was then dropped. A reset that was reported now always gets its summary. The server test uses the plain reset, which the Windows jobs can seed, and puts the server stderr in its failure messages. |
||
|
Vito
|
b2cc489fd4 |
feat(inspect): copy settings summary and local workflow roles (#286)
* feat(inspect): copy settings summary in the full inspector The legacy metadata page's COPY ALL (LI-4) becomes "Copy settings summary" after the six facts of the shared full inspector, so the Images viewer, the recorded history's current file and Inspect all offer it; the Images side panel keeps the concept's footer. The text is the legacy copyAllData template with its placeholders, filled from what the inspector shows. A unit test runs the template read from web/metadata.html on the legacy parse of every parity PNG: 22 of 25 are byte-equal, and the other three differ only by the ZeroOut guard (#249) or facts the legacy parse missed. Copying follows the concept's prompt copy: the toast "Settings summary copied.", or the select-to-copy dialog (now titled per copy) without a clipboard. The concept comparisons map the button as D-NEW-1. * feat(inspect): local execution evidence and prompt tracing A browser port of the server's inspection resolver for one sampler (PR L3 Task 8, LI-1). local-resolver.mjs answers inspect_prompt_evidence's {prompts, resolution} for a local read's records: execution evidence (read_pair_execution and its per-node match), every traced strategy (encoders, strings, concatenations, PromptManager and its historical classes, passthrough, zeroed, selector branches, composites, guiders), the latent stage walk, the R1 overwritten-text guard and the inspection profile's visit, depth, text, path, built and scanned bounds, with the same reason codes and outcome lines. py-json.mjs reads the records as CPython's json.loads does (dict order and last-wins keys, int versus float, BigInt past 2^53, NaN) and measures and strips text as str does. Several sampler candidates, detailer stages and A1111 parameters raise LocalResolverGap until Task 9. Nothing is wired into Inspect yet. Python is the oracle: test_local_inspect_parity compares the port with the server on the parity corpus, 107 named record sets, every work bound at its edge and 600 seeded fuzzed graphs, and requires gaps to be exactly the Task 9 paths. Sixteen porting mutations each fail the suite. The Node unit test replays committed server answers, which the Python suite keeps current. * fix(inspect): a list or object workflow node type answers malformed The R1 overwritten-text check looked every UI workflow node's `type` up among the subgraph IDs. A `type` that is a JSON list or object raised TypeError there, so the inspect route answered 500 "Prompt inspection failed". Such a workflow is now malformed evidence: the resolution is unresolved with reason_code `malformed` and the inspector's own "Prompt evidence is malformed" line, never exception text. The browser port answers the same, and the parity corpus gains the list, object, nested-subgraph and skipped-ID shapes. * fix(inspect): a list or object graph class_type is never a preview output With several sampler candidates the inspector first drops stages whose outputs reach only previews. That pass looked each API graph node's `class_type` up in the preview class set, so a list or object `class_type` raised TypeError and the inspect route answered 500. Only a string names a class now; such a node is an ordinary terminal, as the rest of the resolver already treats it. * feat(inspect): local candidate selection matching the inspector The browser port of the inspection resolver now answers every record set the server does, so no path is left to the server any more: - several sampler candidates, with the identical-branch rule (#270): a compare grid whose branches all resolve to one pair (text and recipe) is resolved, any other mix is ambiguous; - stages whose outputs reach only previews are dropped first; - detailer stages are judged with the samplers in their image ancestry (multistage-differs, cycles, depth and the per-stage visit budgets), and a sampler that only feeds detailers is not a candidate of its own; - A1111 `parameters`, with Python's multiline anchors, `\s` and Unicode `\d`, the bare footer and Template trailers, counted in code points. The parity corpus gains the selection cases and a seeded fuzzer, and every inspection vector now gives the server's prompts, resolution and facts locally. 16 of 17 mutations are caught; the uncaught one (not flattening an ambiguous upstream stage) cannot change any answer. * feat(inspect): local prompts resolved like the server's A file chosen on the Inspect page now shows the prompts and resolution the server's inspector gives the same records: buildLocalPayload takes its roles from the local resolver instead of the interim open roles, so the legacy fill applies exactly where it does for Images. The resolver reads the server's own node table. The page fetches web/js/inspection-semantics.json once through the API client as it mounts, before any selection; the static route serves that one JSON file by exact path (every other JSON stays forbidden). A table that cannot be fetched answers "The inspector's node table is unavailable." as the server does, and a later selection asks again. The Copy settings summary deviation is numbered D117. * fix(inspect): skip a workflow node whose type is a list or object The overwritten-PromptManager walk now skips such a node, on the server and in the browser port, instead of marking the whole file malformed. A prompt that resolves still resolves; the other nodes still feed the check. * fix(inspect): one JSON depth cap for both browser parsers py-json.mjs and local-video.mjs now refuse past the same 1,000 nested arrays or objects, counting every container as CPython's scanner does. Boundary vectors at 999, 1,000 and 1,001 for a deep prompt and a deep workflow run through the Python oracle, and TESTING.md gives the measured per-version server limits and both divergence directions. * fix(inspect): realpath containment for the exact-path node table route inspection-semantics.json now goes through the same realpath check as every other static file, so a symlink out of web/ is refused. The forbidden-path test asserts exactly 403. * fix(inspect): abortable, stepped local reads with a busy state The wait for the node table now ends with the selection's signal. A local read resolves and projects in bounded steps with a turn of the event loop between them, and the inspector carries aria-busy while it loads. A budget test covers maximal 2 MiB graphs and checks the stepped read is byte-identical to one synchronous build. The D117 wrap rule now targets only the summary button, and TESTING.md scopes the restart line to PRs L1 and L2. * fix(inspect): record the two CPython-JSON emulations as a follow-up |
||
|
Vito
|
fd747a0ffc |
fix(inspect): keep a replaced video in place until its pipeline has started (#291)
WebKit (GStreamer) deadlocks the page when a video leaves the page or the viewport before its pipeline has started: the main thread changes the pipeline state and waits for the source's streaming thread, which waits on the main thread in a synchronous bus handler (native stacks of the hung WPE web process in the pinned image). Removing, hiding or merely pushing the video below the fold all hung; none of pause(), removing src, load() or the revoke was the cause on its own. A video replaced while still starting now stays where it is, inert and aria-hidden, with the new preview inserted after it, and the session's revoke of its URL waits. Once it reports metadata or an error (or after 15 s) it is paused, loses its src, is reset with load() and removed, and only then is its URL revoked. Every released video is reset with load(), so no decoder is kept per replaced video. The loadstart replacement spec now runs on WebKit too (15 fresh pages, replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause. |
||
|
Vito
|
a8aff3a39d |
feat(annotations): bulk Add tags and Auto-tag selected as Activity jobs (#283)
* feat(activity): image tagging jobs for bulk Add tags and Auto-tag
POST /prompt_manager/jobs/image-tags starts one record-changing
"image-tags" job over 1..500 listed images (202, or the same 409 busy
envelope as /jobs/scan). Add tags hashes each current file and unions a
fixed set into that byte record; Auto-tag captures each still image, asks
one legacy engine for its top N suggestions on the capture (count from
the options or autotag.defaults.suggestion_limit), re-hashes the file and
unions only onto the same bytes. Outcomes are grouped as tagged,
unchanged, skipped (video, too large, file changed, file not found, not a
still image) and errors; cancel stops at a file boundary.
Auto-tag holds the shared model gate for the whole job (409 model-busy at
start when it is held), loads the model once and never downloads one: an
unavailable model fails the job before the first file. The registry gains
blocker() so a start answers busy before taking the gate.
* feat(annotations): Add tags and Auto-tag selected in the selection bar
The Images selection bar gains the concept's Add tags and, beside it,
Auto-tag selected (D85). Add tags starts the image-tags Activity job with
the dialog's comma-separated names; Auto-tag selected reuses the Tag image
model panel (server defaults, Suggestions count) without Generate, refuses
a model that is not downloaded, and either starts the job (Apply
automatically) or walks the selection one Tag image dialog at a time
(Review each image, D90). Busy, model-busy and refused starts toast why;
the selection stays; a finished job's files are read again.
* feat(activity): show image tagging jobs with their own outcomes
Image tagging jobs get their own Activity view (D89): the pill reads
"Tagging", the item and heading line name the images and end with
"N tagged · M errors", and the job view shows Tagged, Unchanged, Skipped
and Errors tiles and outcomes, opens Tagged and Unchanged rows in the
viewer, and reads "Tagging log" and "Cancel tagging". The per-kind table
lives in activity-store.mjs; the scan's view is unchanged.
The selection bar's copy without a concept line is recorded as D117.
* fix(annotations): final review round for bulk image tagging
- Rename the D117 copy deviation to D118 (D117 belongs to another PR).
- The tagging job's progress line gives the scan's "about X min left"
estimate, with each finished file as one batch.
- The start body cap fits 500 targets of the longest legal path, even
escaped; one byte over answers 413.
- A capture store at capacity is retried with a short backoff, then the
file is skipped with a rerun sentence instead of a Tag-dialog error.
- Output roots and their labels are resolved once per job in the
executor, not twice per file on the event loop.
- Review each image holds Stop reviewing, Escape and x while a save is in
flight, and its primary reads "Save and next" when only the manual
field would be saved.
- An Add tags field that parses to no names keeps the dialog and its
text and reports the message on the field.
- Image tagging finish toasts carry the counts ("... finished · N tagged
· M errors").
- Specs: the bulk spec waits for the answered Start before checking that
the invalid press sent nothing; the Activity spec uses a real error row.
|
||
|
Vito
|
e29ff4d4b2 |
feat(inspect): the Inspect page for local files (#280)
* feat(inspect): local adapter seams in the shared inspector and viewer
The inspection loader calls adapter.read when an adapter reads by itself, so a
file chosen in this browser never reaches the API. The full inspector shows the
D66 local provenance (no Path row, no Show in folder, local association line),
and the viewer uses adapter.mediaUrl exactly, so a blob: URL is never given the
_pm_viewer cache-buster and Retry media reuses it. Rooted and recorded-history
adapters keep their api.get paths and cache-busted originals.
* feat(inspect): owned local selection session
createLocalSession owns one chosen File at a time: it classifies the bytes,
creates the blob: URL only for a classified image or video, and reads it
through a real inspection controller with the local adapter. Replacement,
dispose, parent abort and pagehide each release the URL, the handle and the
record once; a released handle (and every copy) is foreign to the adapter, and
every await is fenced by a generation counter so a replaced read never
publishes. Empty, multiple and directory drops keep the previous selection.
Baseline entry for local-session.mjs; branch floors raised for the touched
inspection-state.mjs and local-inspection.mjs.
* feat(inspect): native Inspect a file page
#/inspect is now the concept's "Inspect a file" page instead of the legacy
handoff. One chosen or dropped image or video previews inline, and the one
heading action, "Open file metadata", opens the shared viewer with the full
inspector on a record read entirely in this browser. Nothing is uploaded,
imported or stored; only an explicit Send crosses the host bridge.
- inspect.mjs mounts the concept DOM (surface, file-drop, #local-preview)
under the shell heading, owns one local session and the shared viewer and
prompt sender, and releases media before the session revokes its URL on
replacement, route change, page leave and dispose (D60-D65).
- A file that cannot be read after choosing opens the inspector's
unreadable state when the browser declares a supported type, instead of
the "unsupported" copy; the local adapter maps a browser read failure to
an unreadable record.
- The route is native; Overview keeps its card to the legacy metadata page,
which is unchanged.
- workspace-inspect.spec.mjs covers the chooser and viewer rail against the
Images rail for the same bytes, drop, cancel and reselect, unsupported and
unreadable files, video, URL ownership, mid-read replacement, exact send
and the no-request/no-storage rule.
* test(inspect): concept parity and managed-host send for local files
Compare the native Inspect a file page with the immutable concept's
#metadata at 1600x1100 and 390x844 in both themes (initial, selected,
viewer-open), mapping D60-D66 and pinning the concept's CSS values and
the chooser's displayed file name, with a mutation check. Send a local
PNG (explicitly empty negative) and core-tags.webm to a real Prompt
Manager (+/-) node across Classic and Nodes 2.0 switches and reloads;
a connected input rejects the send unchanged. Document D60-D66, the
legacy-floor dispositions and the commands in TESTING.md.
* fix(inspect): reset a released preview video in the next task
On WebKit (GStreamer) a video replaced while its pipeline was still
starting blocked the page: pause, src removal and load() in one task
never returned, so the next chooser selection hung (CI page workflows
webkit shard 2/2, "URLs and media end with their owning session").
The src is still dropped before the URL is revoked; only the load()
reset moves to the next task. A fresh-page regression replaces the
first video as soon as it shows, fifteen times.
* fix(inspect): never reset a preview video that is still starting
Deferring load() to the next task was not enough: CI on
|
||
|
Vito
|
195a1d33ac |
feat(annotations): Tag image dialog, card Tag button and inspector tag chips (#281)
* feat(annotations): image tagging controller with staged suggestions Add the UI-agnostic Tag image dialog controller (#249 C1): observe the file into a capture, prefill the saved tags and the server's tagging defaults, stage the top suggestions as removable chips, and save manual tags plus the remaining chips with case-insensitive dedupe (manual spelling first). Every read is serial-guarded; saves are sent once and a lost answer is resent identically only by Retry. Conflict, storage, file and capture outcomes keep the draft and gate Save on a reopen. ApiError now carries a validated copy of a tag-conflict body's current tag set so the dialog can show it. * feat(annotations): Tag image dialog beside Info on every image card Every Images and Library-embedded gallery card shows the concept's Tag after Info. Tag opens the concept's Tag image dialog (image-tagging.mjs over the C1 controller) in the shared modal shell with the sticky tagging-modal footer: capture preview, manual tags, the four legacy engines with real defaults, explicit Generate, removable suggestion chips (D83), legacy model labels (D84), a Suggestions count (D91) and Retry where the controller allows it. The capture is released on close, route change and pagehide. The Tags page gains the concept's Auto-tag images for the image selected in Images (aria-disabled without one, D115) and real Images counts that follow every save. Retires D9, #245 D2 and #245 D3. The toast now uses the mock's copy verbatim ("tags" even for one), C1's D60/D61/D68 markers are renumbered to D83/D84/D91, and at extreme zoom a card's info row stacks (and wraps Info and Tag) when its row is narrower than the narrowest bounded card, list rows included (D31). * feat(annotations): show image tags in the inspector and viewer The inspector parses the B3 envelope's image_tags facet (strictly: exact keys, current/none/unavailable, at most 100 vocabulary names) and draws one chip row after the prompt blocks: the file's current image tags first, then D21's linked tags not already shown (case-insensitive). In the Images panel image tags are Tags filter buttons and linked tags keep D21's rule; in the viewer and wherever no Tags filter exists every chip is plain. none and unavailable add no chips (D88, plan D65; D21 amended). The tagging fixture reset now drops leftover captures, so a long run of tagging tests in one worker stays below the capture store's limit. * fix(annotations): re-read a tagged file in place after a save After a tag save the selected file's inspection is re-read quietly: the panel and the viewer keep the current details until the new answer lands, with no "Loading file details…" in between. While a Tags filter or a search is active, Images re-reads its loaded range in place (same query, from the first page up to the loaded count, published once), keeping the cards, the scroll position and the selection while the selected file still matches; when it no longer matches, the panel follows the collection. * fix(annotations): give suggestion chips the concept chip corner The D83 chips took the later `.tag-suggestion` row rule's 5px corner; the concept `.chip` has 4px. The concept comparison of the Tag dialog found it. * test(annotations): compare the Tag dialog with the approved concept The Tag image dialog's inventory (order, text, field attributes, tokens, preview box, chips, status, sticky actions) must equal the concept's at 1600x1100 and 390x844, dark and light, opened, generated with WD14 and with a language model, for an empty result and in the review scope, and the card's Info and Tag group must equal the concept's. Production runs on the concept fixture with the concept image's tags saved and the fake model answering the concept's own sample; only D29, D83, D84 and D91 are mapped, and a mutation check shows the comparison notices a change. The D3 footer check again asserts a strictly shorter footer, with short seedless subtitles that stay on one line beside Info and Tag. * fix(annotations): final review round for the Tag image dialog - Option fields show the server defaults that Generate sends: the controller bumps an options revision on open and on a model change, and the panel re-renders only then (I1). - A tag conflict shows the current set at once ("Saved now: ...") and puts it ahead of the draft in the manual field, so the next Save sends what is shown (I2). - The opening sentence is the concept's on-screen copy; "Remove anything you do not want to add." leads the chips only while chips exist; the comparison's opened mapping is gone (I3). - ApiClient gains an additive keepalive option; the capture release and the Save PUT use it, and pagehide closes the dialog (I4). - Closing during a save keeps the capture until the PUT settles, then releases it and still reports the outcome (toast, page refresh) (I5). - A refresh asked during a page load is pending and runs once it settles; re-reads use pages up to 200 files; a failed re-read toasts; Load more waits for a refresh (I6, M2, M3). - The selection bar drops checked files a refreshed result no longer lists (I7). - D116 records Retry; after a failed Generate it takes Generate's own path, and after a changed storage, file or expired capture it observes again keeping the draft (I9, M4, M5). - A save made from Tags marks a filtered Images visit stale (M6). - images.mjs drops under 800 lines: the tagging wiring moves to images-tagging.mjs and the favorites star to images-favorites.mjs (unchanged), and reset() reuses dropSelection() (M1). - Restore the activity-scan.mjs coverage floors; drop the duplicate paired captures test (I8, M9). * fix(annotations): coverage floors for the Tag dialog round Floors for the new images-favorites.mjs and images-tagging.mjs, and the measured raises of image-tagging-state.mjs and image-tagging.mjs. activity-scan.mjs keeps main's floors. |
||
|
Vito
|
1b8fbdbb75 |
feat(annotations): image tag routes, model gate and scored suggestions (#278)
* feat(annotations): add scoped observe, capture preview and tag save routes
POST /prompt_manager/image-annotations/observe captures the file named by
the listing identity {root_index, root_scope, path} (scope checked against
the current roots before any file access) and answers the saved tag set of
exactly those bytes, refreshing a matching record's verified fingerprint.
GET/DELETE .../capture/{token} serve the private copy (no-store, nosniff,
pinned with acquire/release_use while it streams) and release it.
PUT .../tags re-hashes the current file and writes only when it still
holds the captured bytes, under the storage epoch and tag revision
(file-changed, storage-changed, tag-conflict, tag-limit, replayed).
Bodies are capped at 64 KiB with unknown fields refused; envelopes never
carry server paths or locators. Restores invalidate every capture and the
host's cleanup removes the capture directory.
* feat(autotag): scored suggestions with a configurable count behind a shared model gate
- py/model_gate.py: one process-wide, non-blocking gate for loading,
swapping, running, downloading and unloading a tagging model. Worker
calls run shielded; a release requested while one still runs takes
effect only when it exits (client disconnect keeps the gate).
- Legacy /autotag/single, /start, /download/{type} and /unload hold the
gate and answer 409 {code: "model-busy"} while another operation holds
it; their bodies are otherwise unchanged. /autotag/models gains
suggestion_limit.
- AutoTagService.generate_tag_suggestions: WD14 (name, probability) best
first, GGUF/HF (name, None) in output order, first `limit` distinct
case-insensitively; generate_tags keeps its return; the image file is
closed before inference returns.
- POST /prompt_manager/image-annotations/suggest runs one engine on the
pinned capture, with strict per-engine options, never downloads
(409 model-unavailable) and never writes.
- config.json autotag.defaults.suggestion_limit, integer 1..100, default
10, validated on load.
- Capture probe decides Generate from the image header alone, so prompt
metadata over the inspection budgets no longer blocks it.
* feat(images): match image tags in the Tags filter, search and inspector
The workspace Images listing now admits a file under a Tags filter when
every requested tag is carried by its linked positive or its current image
tag set, and q also matches image tag names (case-folded substring). Image
tags count only while a byte record is verified at the file's present
(size, mtime_ns) (spec amendment S1): the tagged records are streamed
once, only listed files carrying a filter tag or q hit are stat'ed, and
their linked positives are read through the file_path index so tags,
category, rating and the positive count combine.
The rooted inspector envelope gains a validated image_tags facet
(current / none / unavailable) read at the inspection's own stable stat.
A failed image-tag read fails a Tags filter like a failed link read and
degrades a q search to no image tags; plain listings read none. Legacy
gallery and search responses are unchanged.
* fix(annotations): answer a fixed coded message when the filtered image list fails
A failed prompt-link or image-tag read under a Tags, category or rating
filter returned the raw exception text (SQL or database messages) in the
500 body. The workspace listing now answers code listing-failed with a
fixed sentence and logs the traceback. The legacy output listing and the
inspector image-tag read now log with exc_info as well.
* fix(annotations): drop suggestion names that Save would reject
A JoyCaption part over 50 characters or holding a newline came back as a
preselected chip, so the first Save of the dialog failed with a 400 and
nothing was stored. generate_tag_suggestions now applies the vocabulary
tag-name rules before the case-insensitive dedupe and the limit count, so
the requested number of valid chips still comes back.
* fix(annotations): stop changing the process warnings filters during capture
header_dimensions wrapped Image.open in warnings.catch_warnings with an
error filter for DecompressionBombWarning. catch_warnings is not
thread-safe, so two overlapping captures could leave that error filter
installed for every later Pillow open in the ComfyUI process. Only
Pillow's DecompressionBombError is now mapped to HeaderTooLarge; a size
that only warns is refused by the existing inference pixel cap.
* fix(annotations): free pins on cancellation, serve preview ranges, code every storage error
- A preview cancelled while its file opens closes the handle and only
then ends the pin, so a doomed capture is never unlinked while open.
- An observe cancelled while capturing or reading releases the capture
instead of leaving it until the idle expiry.
- The model gate takes an on_skipped callback that runs once when the
work never starts; Generate passes the capture's release_use, so an
executor refusal no longer leaks a pin against the capture budget.
- The preview answers a single byte range with 206 and Content-Range
(416 for an unsatisfiable or multi-range header), still streamed inside
the pin, so Safari and WebKit can play video captures.
- _coded maps tag-limit (400), metadata-conflict and vocabulary-conflict,
and falls back to 409 with a fixed sentence for any other code instead
of raising KeyError.
* fix(annotations): deny case and Unicode spellings of a listed file
On macOS (case- and normalization-insensitive APFS) a crafted target such
as A.PNG or an NFD name opened the listed a.png or NFC file while
realpath kept the requested spelling, so the tags were stored under a
second locator the Images filter never shows. Capture and verify now
require each path part to be spelled as its directory lists it and deny
any other case or Unicode spelling as an alias (409 source-changed, like
a symlink alias). A missing file with no other spelling keeps its usual
outcome.
* test(annotations): one shared folder_paths stub for the API test modules
test_api_images and test_api_prompts installed a MagicMock folder_paths,
so whenever one of them was imported before test_api_autotag the later
patch("folder_paths.get_output_directory") in test_api_history_outputs
left a mock attribute behind and two test_diagnostics_config tests
failed. tests/host_stubs.py now installs one real-module stub, used by
test_api_images, test_api_prompts and test_api_autotag (and through
test_api_images by the annotation modules, whose copy is gone), so the
modules pass in either import order.
* fix(annotations): derive the suggestion cap from the image tag set cap
py/autotag.py now takes MAX_SUGGESTION_LIMIT from database.annotations
MAX_IMAGE_TAGS. The two copies that cannot import it (py/config.py loads
before the database, utils/inspection_response.py stays storage-free) are
pinned equal by a test, so the 100 limits cannot drift apart.
* docs(testing): record capture aliases, preview ranges and the PR B follow-ups
TESTING.md now states that case and Unicode spellings are aliases, that
the preview serves byte ranges inside the capture pin, where the
cancellation tests live, and two tracked follow-ups: the legacy batch's
ungated cleanup unload under its own last inference, and the three files
over the 800-line ceiling (split py/autotag.py first).
|
||
|
Vito
|
0095db8cbc |
feat(inspect): local evidence readers for images and videos (#277)
* feat(inspect): bounded browser PNG reader matching the server's
The Inspect page will read a chosen file entirely in the browser, and the
record it shows must equal what Images shows for the same bytes. This adds
to web/js/png-metadata.js (loaded by the legacy pages as before, with
parsePngTextChunks unchanged):
- readPngMetadataFile(file, {signal, prefix, yieldControl, decompress}):
a port of utils/prompt_inspector.py's _png_metadata, _PngRecords,
_png_text, _inflate and _batch_normalized. It slices only chunk headers,
80-byte keyword prefixes and the retained prompt, workflow, parameters
and comfyassets_prompt_pairs records, CRC-checks what it keeps, applies
the same budgets (2 MiB shared role budget encoded and decoded, 4 MiB
workflow that becomes limited, 8 MiB and 4 MiB duplicate budgets, 4 MiB
duplicate decoded parameters, 10,000 chunks, the batch-index rule for
parameters copies) and compares duplicate chunks byte for byte. Every
range is checked against file.size before slicing and charged against a
19,794,389-byte ceiling, so a multi-gigabyte PNG costs only its metadata.
Compressed text inflates through DecompressionStream with one 256-byte
write in flight and the output capped at the record's allowance; an
AbortSignal rejects the read with an AbortError and leaves no unhandled
rejection; yieldControl runs about every 8 ms of work.
- classifyPrefix(bytes): png, jpeg, webp, iso-bmff, matroska, avi, asf or
null from the first 32 bytes.
- readImageHeader(file, kind): JPEG (SOFn walk as Pillow) and WebP (VP8,
VP8L, VP8X) dimensions from at most 64 KiB.
tests/test_local_inspect_parity.py drives tests/js/support/local-inspect-
oracle.mjs over the committed and test-time inspection parity corpus plus
compressed, oversized, Latin-1, batch-duplicate and all-four-record shapes,
and asserts records, record formats, dimensions and readability equal
read_inspection_file on each file; the two JPEG/WebP fixtures are checked
against Pillow. tests/js/png-local-reader.test.js pins each server rule,
the sparse 3 GiB read, the prefix reuse, inflation back-pressure and abort.
tests/fixtures/local-inspect/tiny.{jpg,webp} are CC0 FFmpeg test patterns.
* feat(inspect): bounded browser video tag readers
Add web/js/workspace/local-video.mjs: ISO-BMFF (MP4/MOV/M4V) and Matroska
(MKV/WebM) walkers that read only the container tags the server asks ffprobe
for, slicing box and element headers, carrier names and retained values within
the server's budgets and never a byte inside mdat or a Cluster; a port of
utils/video_metadata.select_video_carrier with the record budgets of
read_inspection_file; and a byte-exact json.loads/json.dumps round trip so
envelope objects reserialize as the server spells them.
Every naming rule mirrors ffprobe 9.0.2 on hand-built files: legacy ilst atoms,
mdta keys, joined repeats, data types, Matroska language and default handling,
and Tags behind a Cluster found only through a SeekHead.
Tests: node units over byte-exact container builders and the committed
fixtures; a Python parity test comparing the local carrier with the server's
selection from the same tag maps (mocked probe) and from the same files (real
ffprobe, including 1 MiB + 2.5 MiB per-key records); an ffprobe
characterization of the recorded fixture maps.
* feat(inspect): server-identical local projection and payload
Port utils/inspection_response.py to the browser: a strict parse with Python's
Decimal spelling, duplicate-key rejection, NaN/Infinity as null and integers
past 2^53-1 as text; the prompt, workflow and carrier projection with the
depth-64 and 20,000-node ceilings; _metadata; and the 8 MiB omission steps
(workflow graph, then prompt graph, then all raw) measured as UTF-8 bytes.
local-inspection.mjs builds the inspection_version 1 wire for one chosen File
from the Task 1 and Task 2 readers (observation selected-local-file, interim
unresolved roles, unavailable library), fits it to the response budget, and
exposes localFileAdapter for the shared controller, inspector and viewer:
handles are session-owned by a private token, so a tampered or hand-made
descriptor is rejected and blob: URLs are never resolved through the API
client. projectInspectionPayload admits selected-local-file only for a
local-file identity.
A Python oracle test compares projectMetadata with _metadata byte for byte over
the parity PNG corpus, the committed video fixtures and tag maps, 75 committed
adversarial record sets (projection-cases.json) and node-ceiling sets built at
test time.
* test(inspect): import oracle modules by file URL for Windows
The local-inspect oracle imported product modules with path.join(...), which
Node parses as a URL scheme on Windows (D:\... -> ERR_UNSUPPORTED_ESM_URL_SCHEME),
failing every tests/test_local_inspect_parity.py oracle test on the Windows CI
job. Every dynamic import in tests/js/support now goes through moduleUrl, a
pathToFileURL wrapper, and a node unit proves the Windows form on any host and
scans the oracles so a path import cannot return.
* fix(inspect): read video tags as ffprobe reports them
Same bytes now give the server's record in five shapes where the local
reader was stricter or kept different tags (PR #277 final review I1, I2,
M1, M2, M4):
- Invalid UTF-8 in a tag value is replaced as ffprobe's default
string_validation=replace does (av_utf8_decode, one U+FFFD per failing
decode step), instead of failing the read. Key names and TagNames that
do not decode are no carrier and are skipped.
- An ilst item whose keys index is 0 or out of range is skipped, as
ffmpeg does, instead of failing the read.
- Only the first keys box is read, as ffmpeg does.
- A first per-key workflow over 4 MiB keeps its place in the carrier
choice as an unread marker, so the server's carrier (per-key, workflow
limited) is chosen; when the server's first probe would overflow, the
re-probe leaves it out and its own bound applies.
- An envelope integer of more than 4300 digits fails the Python json
port, so that envelope is skipped as CPython's json.loads skips it.
Each shape was checked against ffprobe 9.0.2 and read_inspection_file;
the real-ffprobe parity test now builds them, and a 600-case random-byte
comparison of the replacement against ffprobe matched.
* perf(inspect): measure the local response size natively and yield between steps
fitInspectionPayload counted the UTF-8 size of an 8-12 MB wire with a
per-character loop and ran every omission step on the main thread in one
turn (PR #277 final review M7). The size is now counted by TextEncoder
encodeInto through one bounded scratch buffer, and each re-measure
follows a yieldControl turn, with an abort signal checked between steps.
The fit is async; the fitted payload is the same as before.
* fix(inspect): hold video tags as ffmpeg's dictionary does, decode Mac Roman text
Three more same-bytes divergences, found while verifying the review
fixes against ffprobe 9.0.2 and read_inspection_file:
- ffmpeg keeps one tag per ASCII-folded name under the spelling set
last (ISO-BMFF joins the values with ";", Matroska keeps the last), so
PROMPT and Prompt are one record on the server; the local reader kept
both and failed the read as a duplicate. A legacy comment atom and a
later COMMENT key likewise leave no exact-case envelope.
- ffprobe's format_tags filter folds ASCII case only, so a name that is
"workflow" only under Unicode case rules is never reported; the local
reader took it.
- ISO-BMFF data types 0 and 3 are Mac Roman text to ffmpeg, written into
twice the input size (a long value is cut); the local reader decoded
type 0 as UTF-8 and skipped type 3.
The real-ffprobe parity test builds each shape.
* docs(testing): list the local Inspect divergences; raise touched coverage floors
TESTING.md now tracks the local Inspect (#249 PR L1) divergences that
remain after the review fixes, each checked on a host with ffprobe 9.0.2:
media kind by bytes versus extension (parity is for the same bytes with a
matching extension), JPEG dimensions past the 64 KiB header read, classic
QuickTime udta text atoms, raw versus escaped tag bounds, the 10,000
element ceiling, unknown EBML DocTypes, the JSON depth approximation and
the C _decimal assumption.
coverage:update raised the branch floors of the three local Inspect
modules this PR touches; the unrelated activity-scan raise it proposed is
left at main's values.
|
||
|
Vito
|
b7cf210f0f |
feat(annotations): image tag storage, vocabulary participation and capture store (#276)
* feat(annotations): add image annotation schema with restore validation
Install the image annotation group beside image_favorites and saved_views:
media_annotation_metadata (one row, schema version 1), media_annotation_files
(one private locator per physical file), media_annotation_bytes (one record
per observed SHA-256 with the verified size and mtime_ns) and media_image_tags
(the ordered tag set of a byte record, joined into the global tags registry).
The group migrates in a SAVEPOINT that validates before and after creating
it, with no backfill from prompt or pair tags and no filesystem probe.
annotation_schema_error reads only: it compares the catalog objects after
whitespace compaction, the columns, foreign keys and indexes, and the
persisted values (UUIDs, hex digests, bounds, the 4096-character locator cap
measured past embedded NULs, exactly one metadata row). verify_database_file
checks it with allow_missing=not require_library_state, so old backups
restore and a migrated restore stage must carry it.
Tests pin the exact DDL, a fresh install and idempotent reopen, a current
database without the group gaining it with legacy rows untouched, every
partial or malformed shape rejected without repair, install failure rolling
back and retrying, a WAL backup/restore round trip, staged-failure
immutability of source, live and safety files with every handle closed, and
the require_library_state gate.
* feat(annotations): persist per-image tag sets in the shared vocabulary
Add database/annotations.py (AnnotationOperations, mixed into
PromptDatabase): read_image_tags, replace_image_tags and add_image_tags
run in one _pair_transaction fenced by the storage epoch, own the byte
record by exact locator and digest, always supply UUID ids, rewrite the
ordered junction and bump tag_revision plus image_index_revision.
Revision mismatches raise AnnotationCommandError("tag-conflict") with
the current set; the identical set at the current or previous revision
is replayed without writes; a union over 100 names is "tag-limit".
normalize_image_tags validates each name with validate_vocabulary_tag_name
and folds case-insensitive repeats keeping the first spelling.
refresh_image_fingerprint, current_image_tags (chunked, fingerprint
matched, newest verification wins) and iter_image_tag_records serve the
S1 cheap reads.
Vocabulary participation: _move_tag_links also carries media_image_tags,
legacy merge_tags moves image-only sources, tag rows gain image_count
(files with any byte record carrying the tag), impact gains images and
rename/delete receipts gain images_changed. vocabulary-state.mjs accepts
the optional counts and names images in the delete text.
Browser fixtures clear the three annotation data tables on reset (never
the metadata row); TESTING.md documents the schema beside favorites.
* feat(annotations): add bounded private media capture store
Add utils/media_capture.py: CaptureStore copies one rooted media file
into an owner-only prompt-manager-captures-* temp directory, streaming
it in 1 MiB chunks while computing the whole-file SHA-256, and compares
the opened descriptor with the path (dev, ino where meaningful, size,
mtime_ns, ctime_ns) before and after the copy; any disagreement raises
CaptureChanged and removes the copy. Symlinked files or directories,
root escapes, non-media names and directories raise CaptureDenied; the
locator is normcase(join(realpath(root), relative)), the listing's
identity, so a symlinked root matches it.
Limits are the plan's: 256 MiB per file on stat and stream
(CaptureTooLarge), 8 completed captures, 512 MiB reserved and 2
capture/verify workers with nonblocking admission (CaptureCapacity),
900 s idle and 1800 s hard lifetimes on an injected clock. acquire and
release_use keep an in-use capture past expiry, release and
invalidate_all until its last use ends; close removes only owned files
and the owned directory. A header-only probe of the copy marks Generate
unsupported above 4096 px per side or 16,777,216 px, for unreadable
headers and for video; verify re-hashes the current file without a
copy. Every handle closes before unlink.
tests/test_media_capture.py covers exact bytes and owner-only modes,
alias and escape denial, the symlinked-root locator, change detection
during the copy (size, same-size rewrite with restored mtime through
the POSIX change time, inode replacement), both size caps, count, byte
and worker admission, idle and hard expiry with in-use retention,
cleanup scope, handle release at unlink, dimension and video
readiness, and the allowlist equality with the inspection service.
TESTING.md documents the store and the Windows change-time limit.
* test(annotations): wait on the POSIX change time only where it exists
The same-size rewrite case waits for the filesystem clock to pass the
source's ctime so the detection is deterministic on Linux's coarse
timestamps; Windows reports the creation time as st_ctime, where the
probe could never advance, so the wait runs only on POSIX.
* fix(annotations): compare the change time only between fstat reads
Python 3.12+ on Windows reports the change time from stat but the
creation time from fstat, so a freshly written file whose creation and
write straddle a clock tick failed the pre-read identity check with
CaptureChanged. The path-versus-descriptor checks now compare size,
mtime_ns and dev/ino; ctime_ns is compared between the fstat before
and after the copy, where it still detects a same-size rewrite with a
restored mtime on POSIX.
The symlinked-root test asserted the locator against the unresolved
temp path, which macOS (/var -> /private/var) and Windows (short names)
resolve differently; it now compares against the real path.
* test(browser): give the inspector evidence matrix the concept comparisons' budget
The touch run of 'full inspector attaches matched theme and mobile
evidence with reachable content' captures 32 screenshots across four
viewports and took 23.5 s, then 31.6 s, in the sharded coverage job,
hitting the 30 s default. Like the concept comparison specs it now sets
an explicit 120 s timeout; the CI-like pinned run (4 CPUs, 3 workers,
repeat-each 3) passes 198/198 with the matrix at 18-20 s on touch.
* fix(annotations): compare capture identity like with like and keep paths out of errors
Native Windows CI (Python 3.12/3.13) rejected untouched files with
CaptureChanged because the path stat and the descriptor fstat disagree
there (change time versus creation time, possibly dev/ino encodings).
The descriptor is now checked against the path only on the API-stable
facts (size, mtime_ns); the full identity (size, mtime_ns, ctime_ns,
dev/ino where meaningful) is compared between the fstat before and
after the copy and between the path stats before and after. A test
injects shifted fstat results to simulate the Windows discrepancy, and
a fresh file is captured twice with no platform guard.
Error messages carry strerror only, never the path (the path goes to
the debug log); a test asserts no CaptureError names the root. Files
are unlinked outside the store lock; close() during an in-flight
capture lets that worker remove its temp file and then the directory.
get() documents that readers of capture.path must acquire/release_use.
The test fixture resolves its temp root (macOS /private/var, Windows
8.3 names). TESTING.md states the real identity contract.
* fix(annotations): move image revisions when a fingerprint or the vocabulary changes
A replayed save that observes a new verified (size, mtime_ns) now bumps
image_index_revision, like refresh_image_fingerprint, so caches keyed
on it see the record become current again; an identical replay still
moves nothing.
Vocabulary rename, merge and delete (native and legacy paths) bump the
tag_revision of every image set carrying the source tag, in the same
transaction and before the links move or cascade, plus one
image_index_revision bump when any set was affected. A Tag dialog
opened before the change therefore gets tag-conflict with the renamed
set as current instead of recreating the old name.
* test(annotations): accept change detection wherever fstat exposes a change time
Native Windows CI on Python 3.13 detected the same-size, mtime-restored
rewrite through the descriptor's change time (3.10-3.12 report the
creation time there and complete the torn copy). The test asserted the
platform, not the signal: it now accepts CaptureChanged plus temp
removal wherever it happens, requires detection on POSIX and on
Windows 3.13+, and otherwise requires the digest to describe the
copied bytes. A new test injects a drifting fstat change time so the
detection path runs on every platform; TESTING.md records the limit as
"Windows before Python 3.13".
* test(annotations): assert the rewrite outcome, not the platform
Native Windows CI detected the same-size, mtime-restored rewrite on
3.12 and missed it on 3.13 in the same run: the path stat's change time
(Python 3.12+ fast path) only moves when the clock ticks between the
pre-stat and the rewrite, and the tick wait was skipped on Windows. The
wait now runs everywhere with fresh probe names (Windows tunnels the
creation time of a reused name), and the test asserts the outcome: a
detected change leaves no temp file; an undetected one is accepted on
Windows only and must produce a digest of the copied bytes; POSIX must
detect it. The injected-fstat test keeps the detection path covered on
every platform.
* fix(images): keep the Filters label node across a listing reload
Every listing reload reports the active refinement count, and
setActiveFilters rewrote the button's textContent even when the label
was unchanged, replacing its text node. WebKit drops a click whose
mousedown hit a node that was detached before mouseup, so a Safari
user who types a search and clicks Filters inside the 250 ms debounce
lost the click, and the webkit page workflows timed out in the
source-change recovery tests (1 in ~30 runs of the pinned image; an
instrumented probe showed pointerdown/mousedown, a DOM mutation,
pointerup/mouseup and no click event).
The label is rewritten only when it changes. The toolbar unit test
counts textContent writes (1 for an unchanged count before, 0 after),
and a workspace-images test pins the text node identity across a
search reload: 0/6 before the fix on chromium and webkit, 6/6 after,
with the recovery and Refine tests 36/36 in the pinned CI-like run.
|
||
|
Vito
|
bc14f29db7 |
fix(inspect): video metadata carrier parity between Images and Inspect (#275)
* fix(video): select one prompt carrier from per-key or comment tags Current ComfyUI savers write one container tag per key (prompt, workflow, parameters, comfyassets_prompt_pairs); only the old VideoHelperSuite comment envelope was read. select_video_carrier picks exactly one carrier in the order per-key tags, comment envelope, description: the first with a role key wins, else the first with a workflow, and carriers never merge. Matroska upper-cased tag names fold on our side; differing fold-duplicates fail the read like duplicate PNG keys, identical ones are one record. Per-key values stay original-text; envelope objects are reserialized. * fix(inspect): read per-key video prompt tags The inspector asked ffprobe only for comment and description, so videos written by core SaveVideo/SaveWEBM and current VideoHelperSuite (one tag per key) were inspected as having no metadata. It now asks for prompt,workflow,parameters,comfyassets_prompt_pairs,comment,description and hands the tags to the shared carrier selector. ffprobe stdout is bounded at the role and workflow budgets plus 64 KiB of envelope; past that it probes once more without the workflow tag within the role budget and marks the per-key workflow limited, while an over-budget envelope still fails the read. Differing fold-duplicate tags fail like duplicate PNG keys. * fix(video): scan videos written by current ComfyUI savers The output scan and live monitor read container tags through metadata_from_tags, which now delegates to the shared carrier selector, so per-key prompt, workflow, parameters and comfyassets_prompt_pairs tags (including Matroska upper-cased names) are ingested with the default resolver profile exactly like the same graph in a PNG. Fold-duplicate tags and null markers are skipped quietly. Pins the behavior with a mocked ffprobe scan test and real MP4/MKV/WebM round trips when ffmpeg is present. * docs(testing): document video carrier selection Record the shared video carrier order (per-key tags, comment envelope, description), the fold-duplicate rule, the 6 MiB + 64 KiB probe bound with the role-only re-probe and limited workflow, and that scans read the same carrier as the inspector. * fix(inspect): size video probe bounds for JSON escaping, exact-case envelopes ffprobe JSON-escapes tag text, so bounds measured on its stdout could mark a video workflow limited, or fail a role record, that the identical PNG reads within its decoded budget. The first probe now allows twice the role and workflow budgets plus 64 KiB (12 MiB + 64 KiB) and the role-only re-probe twice the role budget plus 64 KiB, leaving the decoded per-record budgets as the only gate; a reader-level test reads near-budget quote-dense records as PNG and as video and asserts identical metadata and formats. Envelope keys (comment, description) are matched exactly as the legacy readers did; only the four per-key carriers fold Matroska case. Fixtures can write the legacy ilst comment atom (metadata_tags=False), and the docs now say the admin scan reads the carrier while the live monitor still processes images only, with a follow-up recorded in TESTING.md. |
||
|
Vito
|
f95f66d76a |
ci(browser): run page workflows with parallel workers and CI shards (#274)
* test(browser): one test server per Playwright worker The page workflows started one global webServer and ran with one worker. Each worker now owns its own tests/browser/server.py on an OS-chosen port, with its own temp root and database, through a shared `test` in tests/browser/fixtures.mjs. baseURL, and so page, request and browser.newContext, point at that worker's server. A server that dies fails the running test with its stderr tail and the next test gets a fresh one. The worker count comes from PM_TEST_WORKERS, else 1 under CI, else a quarter of the CPUs up to 8. Chromium workflows in the pinned image: 510 tests in 606 s with one worker on main, 511 (one new isolation spec) in 119 s with eight. * test(browser): test servers bind an OS-chosen port themselves tests/browser/server.py accepts --port 0: it binds 127.0.0.1:0 before building the app, holds that socket for its whole life and prints PM_TEST_SERVER_PORT=<n> to stdout. startBrowserServer defaults to port 0 and reads the announced port, so no port is reserved and released before the server binds and two launchers can no longer race for one. The "address already in use" retry and free-port.mjs are gone; --port N still works and is checked against the announcement. * ci(coverage): shard and merge modes for the frontend coverage run frontend-coverage.mjs now plans its steps from the command line. No arguments still runs everything, then reports and checks. --shard I/N runs one slice of the chromium+touch workflows into coverage/js/raw without wiping it or reporting; shard 1 also runs the node units and the viewer suite. --merge-only reports and checks over raw coverage the shards wrote, with --update-baseline as before, and refuses an empty or missing raw dir. npm scripts: test:coverage:shard and coverage:merge. * ci(viewerjs): run browser suites with workers and shards Frontend coverage runs as three shards (PM_TEST_WORKERS=3 each) that upload their raw V8 files as coverage-raw-N; a merge job downloads them, fails unless all three arrived with raw files and no clashing names, then runs coverage:merge. Firefox and WebKit page workflows run as two shards each with three workers. The required viewerjs context keeps its name and now needs the shards, the merge and the browser shards, failing unless every one succeeded. * ci(viewerjs): cache npm downloads and size timeouts from the first sharded run The first sharded run lost a webkit shard to its 12-minute limit after all 267 tests passed, and npm ci stalled for 5 minutes in six of eight jobs. Each npm ci now restores npm's download cache keyed on the vendor lockfile, and browser job timeouts are the measured times plus at least 50%, never under 20 minutes. The workflow comment records the measurements. * test(browser): wait for rendering frames, not timers or luck, under load Two Library workflows failed in CI's webkit shard 2/2 (run 39714840809) under parallel load: - "library infinite loading ..." counted cards right after the last page hid "Load more". That render queues the window update for the next frame, so a slow frame left the 120 cards of the all-rows layout in place (received 120, expected < 80). It now waits for the next frame, which runs after every callback the page queued. - "row measurement preserves anchor ..." settled with a 60 ms timer. A scroll event is delivered in a rendering frame, so on a starved renderer the timer ended before the scroll's frame and no card was in view (null anchor). settleLibrary now waits for two consecutive native frames with no Library frame queued, checked inside the frame callback. Before/after: a scratch copy that delays every frame by 120 ms (as a starved runner) failed the old helpers 19 of 40 runs, at the CI lines, and passed the new ones 40 of 40 on chromium and webkit. Throttled to 1 CPU with 4 workers, the original spec failed 1 of 42 for each test. The Tags viewer workflow closed the viewer as soon as its image was visible, before ViewerJS rendered the title, so CI's coverage never ran the page's title callback (tags-page.js functions 26/46 = 56.52%, one function above the floor of 56; locally 27/46). The test now asserts the title names the shown file, which always runs the callback. * test(browser): stable under parallel workers Document how to run the browser suites in parallel and sharded, with the measured times, and what keeps specs stable beside each other: wait for a rendering frame, a response or a visible state, never for time. With |
||
|
Vito
|
2cecfcf0ac |
feat(activity): scan new and unresolved files by default, and download a job report (#273)
* feat(activity): scan new and unresolved files by default, all files on request
Scan for metadata now skips, before reading any metadata, the files the
Library already links with a resolved link (one generated_images row with
exactly one pair link), found with one streamed query that keeps only the
scanned paths and compares paths with os.path.normcase on both sides.
Skipped files count as Already known ("Skipped · already linked").
Unlinked, fallback-linked, ambiguous and unresolved files are read again.
POST /prompt_manager/jobs/scan takes {scope: "new" | "all"} (default
"new", anything else is a 400). The review dialog shows the concept's
"New and unresolved files" and adds an "All files" switch (D58). The
legacy /prompt_manager/scan route keeps scanning all files with unchanged
SSE payloads.
* feat(activity): download a job report
GET /prompt_manager/jobs/{id}/report returns the summary, counters, every
outcome group's listed rows with truncated flags, the scope and the log as a
JSON attachment (prompt-manager-<kind>-<YYYYMMDD-HHMM>.json), with no absolute
paths. The job view's Download report saves it through the api-client URL, and
the scan log opens at its newest line and follows new lines only while the
reader is at the bottom.
* fix(activity): re-read known files whose size changed, and review follow-ups
The new scope now skips a known, resolved file only while its stored
generated_images.file_size equals its size on disk; a NULL stored size is
re-read. A file replaced at the same path is read again and its conflict
shows under Needs review. The scan stores each file's size when it ingests
it, and both sides of the known-set match go through normalize_image_path.
A failed known-set read logs and falls back to reading every file, and an
empty request body asks for the default scope. The All files switch is
described by its hint, a lost job's Download report is disabled, the report
toast is skipped once the view is gone, and the scan log opens at its
newest line every time. Fixtures use the real scope shape.
* test(activity): wait for the job view's settled detail read, not a fixed delay
The 200-row cap test now starts counting detail reads once one read has
returned the full listing and none is in flight. The job view re-reads after
an in-flight read only when the summary still lists more than that read
returned, so no redundant read is left due after opening.
|
||
|
Vito
|
a6f6697c50 |
feat(activity): shared job registry, Activity page and Scan for metadata (#269)
* feat(activity): shared session job registry with the scan as its first kind The output scan now runs as the first kind of an in-memory JobRegistry (py/api/jobs.py, replacing py/api/scan_job.py). Each job keeps exact counters, at most 200 outcome rows per group and a log capped at 2,000 lines, with root-relative paths and public root labels only. New routes list, inspect, follow (SSE) and cancel jobs and start the scan (POST /prompt_manager/jobs/scan, 409 busy while another record-changing job runs). The legacy /prompt_manager/scan routes attach to the same job and stream unchanged payloads. * feat(activity): concept Activity page, top-bar button and progress pill The workspace gains the concept's Activity page (renderActivity), its top-bar button and sidebar row, fed by one session job store: a /jobs read, one SSE stream per running job, a 1-30 s capped backoff after a dropped stream and a 15 s list poll while visible. A job the server no longer lists shows as ended, so no pill stays stuck after a restart. D52: the counts show running jobs and hide at 0; the phone top-bar row may wrap. D53: the mockup's progress pill links the running job. D54: real job items with state chips, detail lines and Cancel job. Start and finish toasts follow the spec copy. Fake browser jobs come from tests/browser/workspace_activity_fixture.py. * feat(activity): Scan for metadata with its review dialog and job view Images -> Import keeps Rescan and adds the concept's "Review scan" row as "Scan for metadata". It opens the concept's "Review metadata scan" with the real source count, the fixed "All files" scope, a backup box that downloads the existing database backup, and Cancel / Start scan. Start scan starts the session scan job and opens its view; a start refused by another record-changing job toasts its title as a link. #/activity/<id> is now the approved job-view mockup: the heading with Download report (disabled until Activity-2) and Cancel scan, the Progress surface with the five counter tiles, the Results surface with the outcome list, path filter and capped rows (Added and Already known open the viewer through the listing identity), and the collapsed Scan log. Activity item titles link their views. The scan's progress line groups thousands. Deviations D55-D57 are recorded in TESTING.md. * fix(activity): stop the job store on pagehide and let proxied routes outlive teardown CI's touch coverage run failed "a resolver failure for the ComfyUI link never stops the shell from mounting" with "route.fulfill: Fetch response has been disposed": its trace shows two Library thumbnail requests started by the last route still in the spec's proxy-mount handler while the page fixture was torn down. - The proxy-mount handler ignores only that teardown error, and the workspace and Activity specs unroute with ignoreErrors before teardown. - The Activity store now ends its work on pagehide: the poll and backoff timers stop, the job streams close and a list read in flight is aborted (an older read is also aborted when a newer one starts). A pageshow from the back-forward cache reads the list again. * test(activity): compare the job view log's line height to the hundredth of a pixel WebKit reports the mockup's 11px/1.7 line height as 18.700001px, so the job view comparison failed on WebKit only. The measurement now rounds it, as other engine-dependent values are normalized. * fix(activity): final review round for the job store, job view and scan rows - Store: one failed list read no longer marks running jobs as ended; two in a row, or a good list without them, do (M1). A good read resets the backoff, except for a stream that keeps dropping while the list reads fine (M2). Finished jobs the server pruned are dropped (M3). Numbers group as en-US, like the server's lines (M13). - Job view: a job the store's list no longer has shows the not-found state (M3). Focus moves to the job heading when Cancel scan goes away, and to the item's title link in the list (M4). The detail is read only when its listed rows change or the open log grows, using the summary's new listed and log_length (M12). - Viewer rows open by exact identity: each scan row records its root's listing index, the source must still carry the row's label, and the workspace listing gains an exact `path` filter (M8, M9). - Scan rows name the prompt by its Library title, stored title first (M10). - A toast with a link stays 20 s, and any toast stays while hovered or focused (M14). - A real two-batch scan cancelled after batch 1 keeps partial counts (M7). - TESTING.md: D54, D56 and D57 updated (M5, M6). * test(activity): cover the held and linked toast and a job view pruned by the server Unit tests for the toast's 6.5 s and 20 s timeouts and its hover and focus hold, and a browser test for a finished job's open view after the server stops listing it. Coverage floors are only raised. |
||
|
Vito
|
6e9d8f81fd |
ci(coverage): fast check that every frontend file has a coverage baseline (#271)
The frontend coverage job only reports a missing baseline entry after its full browser run. coverage:baseline-check compares the files the gate measures (same discoverSources) with frontend-coverage-baseline.json in milliseconds and also flags entries for files that no longer exist. npm test runs the same check. |
||
|
Vito
|
e9996996e2 |
feat(views): saved views and Favorites on Images and Library (#268)
* feat(views): saved-views store and API
Add a saved_views table (page images|library, trimmed 1-100 character
name, whitelisted JSON state of at most 4 KB, position, timestamps)
installed through the same validated-savepoint migration as image
favorites. It is optional in old backups and required in a migrated
restore stage; restore rejects a tampered definition, extra indexes and
any row that would fail write validation.
Routes under /prompt_manager/saved-views: GET lists views by position,
POST {page,name,state} returns 201 {success, view}, PATCH /{id} {name}
renames and DELETE /{id} removes, both 404 for an unknown id. The state
must hold exactly the page's toolbar and Refine-this-view keys (Images:
q, category, sort, source, folder, media, tags; Library: q, category,
sort, tags) with their types and sort/media values; anything else is a
400.
Refs #249
* feat(views): Save view, the SAVED VIEWS sidebar and opening a view
The shared toolbar gains the concept's Save view between sort and Filters
on Images and the Prompt library. It opens the concept's simpleInput
("Save this view", "View name", default "Search: <q>" or "My workspace
view", Cancel / Save); an empty name is "Saved view". A save posts the
page's full whitelisted state (toolbar value plus Refine filters) and
toasts "Saved view added to the sidebar.".
The shell adds the concept's SAVED VIEWS label and nav#saved-nav, loaded
with the shell and re-read after every save. Rows use the concept's
links() markup (search icon, name, no count) and link #/saved/<id>.
Opening one mounts its page with the saved state through the page's own
filters, so the toolbar and "Filters · N" (D44) show it; the h1,
breadcrumb and title are the view's name. A removed Source or Folder is
dropped to its default after discovery, so no listing read is rejected;
an unknown id re-reads the list once, then opens Images; a failed list
read offers Retry.
D16 is retired: images-concept and library-concept no longer map Save
view out. New deviations: D49 (no PROPOSED badge) and D50 (no Needs
review row); shell-concept maps Favorites out until Task 3 and compares
a saved row against the concept's. Browser fixtures now wipe
saved_views.
* feat(views): Edit rename/delete and the Favorites saved view
- Favorites is the first SAVED VIEWS row (star icon, name, real count).
The count is the Favorites listing's own limit=1 total, follows the
nav-count rules and is re-read after every confirmed star or unstar.
- #/favorites opens Images filtered to favorites. The workspace listing
takes favorites=1: each selected root's favorites are read through the
image_favorites primary key and matched against the cached scan with the
toggle's case folding; a failed read is an error, never an unfiltered
page. The cursor scope includes the flag.
- Saved Images state gains a required favorites boolean, so Save view on
Favorites reopens the favorites.
- An open saved view's heading leads with Edit: "Edit saved view" with
View name, Delete... (danger), Cancel and Save. Delete... confirms in the
same dialog (never window.confirm). Toasts "Saved view renamed." and
"Saved view deleted."; a delete returns to the view's plain page.
- shell-concept no longer maps Favorites out (Needs review stays, D50).
Recorded as D51 in TESTING.md.
* fix(views): open stale saved views safely and finish the review fixes
Startup and restore now check only the saved_views structure (table,
columns, types, sizes, a JSON object state) and never the value rules,
so a row saved under older rules no longer stops the database from
opening. Reads normalize each view: missing keys take their defaults,
unknown keys are dropped, invalid values default and the name is made
displayable. POST and PATCH still validate strictly.
Only request validation (SavedViewInvalid) maps to 400; database and
internal errors are a logged generic 500. A saved view no longer
replaces the plain Images or Library return state, a toolbar change
while sources load keeps a saved Source and Folder, a state over 4 KB
is refused in the browser with its own toast, the delete confirmation
skips its failure toast after the page is gone, the Edit dialog reuses
the shared field helper, and quick star toggles coalesce into one
Favorites count read.
* test(views): cover leaving a pending saved view and add its coverage floor
A saved view left while the list or a rename is still pending leaves
the next page alone; the shell drops two unreachable heading guards.
workspace/saved-views.mjs gets its frontend coverage baseline entry;
no existing floor changes.
* fix(views): a corrupt saved-view state no longer blocks startup
At startup a saved_views row whose state is not a JSON object is
logged as a warning by id only and left in place. The listing and
rename skip it, so its #/saved/<id> link behaves like a missing view.
Restore stays strict and still rejects a backup holding such a row.
|
||
|
Vito
|
762669758b |
feat(history): concept gallery cards, history comparison and CSS retirement (#266)
* feat(history): concept shared gallery The history page's gallery is the concept's section.history-gallery with its panel heading (Shared gallery, <name> · same positive prompt) and the Images media cards, followed by the recorded executions. A saved variant with no images shows the concept's empty copy. Recorded as D47. * feat(history): drop the shared gallery's Linked images caption (D47 ruling) The concept's "Shared gallery" heading is the gallery's only heading: the embedded collection's "Linked images" heading and "N loaded · N linked images" count repeated it. mountImages gains caption: false and a focusTarget; only the history page's shared gallery passes them, so the Images page and the All images and Repeat run dialogs keep their caption. The layout row and the selected-image side panel stay (TESTING.md D47). * test(history): concept comparison for prompt history history-concept.spec.mjs compares the Prompt history with the concept's #prompt/alpine on the concept fixture at 1600x1100 and 390x844, both themes and motion settings, on Chromium and touch, with its behaviors in both documents and a mutation check. inventoryOf gains a history section; mapHistoryConcept applies D24, D35, D45, D46, D47 and the approved Show archived button, laying the concept out at runtime. Corrections it demonstrated: each compared negative mounts in a display: contents child of the concept's .compare-item (D25), and the concept fixture now links images oldest first and counts one pair use per image, so newest-first reads and lane run counts match the concept. history-canonical-comparison.mjs uses the concept's selectors for production and drops its closed polish entries. * feat(history): drop the Linked images caption from the history dialogs (D47 ruling) The maintainer extended the D47 trim to the history page's two media dialogs. All images for this positive and Repeat run are named by their own dialog title, as in the concept (app.js a1282-1304), so their embedded collection no longer shows the "Linked images" heading or the "N loaded · N linked images" count. A stranded focus returns to the dialog's close button. The Images page keeps its caption. The history concept comparison now records each dialog's caption count in both documents, and TESTING.md D47 covers the dialogs. * refactor(workspace): retire library.css and the interim page guards Every page is rebuilt, so the rules that waited for PRs D, E and F go. library.css is deleted with its link; the overlay is now the only stylesheet after concept.css. Deleted because the concept already draws them: the .library-grid tracks and breakpoints, the history .row-actions flex row and button size, summary cursors, h2 margins, and the sender and Recorded run dialogs' colors and border. Deleted because they had no effect: min-inline-size on block children, a .recorded-run-dialog pre limit, the general .prompt-inspector margin (now scoped to the Recorded run dialog, the only mount point that is not display: contents) and a 700px padding a later rule overrode. The duplicated sender and Recorded run dialog guards are merged into one rule each. Kept with a marker: windowed Library rows (D3), the shelf states (D40), the history panel's wrap, archived-lane and coarse-target rules (D45), and under the new D48 the production surfaces the concept never shows: recorded executions and the Recorded run dialog (#209, D26), the Archive negative confirmation (#235) and the prompt sender. They now use the concept's --green and --red tokens instead of private copies with the same values. The workspace browser specs on Chromium and the concept specs on touch pass unchanged. TESTING.md records D48 and closes the role-block gap. * refactor(workspace): drop the dead embedded caption and tighten the review guards - images.mjs: an embedded collection never has a caption (D47), so the caption option, its "Linked images" heading, the embedded count string and the heading arm of the focus fallback are removed; callers no longer pass caption:false. Behavior is unchanged. - workspace.css: delete the never-applied .recorded-run-dialog::backdrop rule (the concept's .ca-tools dialog::backdrop always won) and note that the D3 row's min-inline-size: 0 is load-bearing for the concept's <=700px 1fr track. - concept-css test: the waiting-rule guard is case-insensitive and is proven against the exact retired comments and the kept provenance labels. - history-concept spec: function replacement for the singular run label, and the DOM mutation removes Compare negatives by role. - D47 markers sit where the caption trim happens. |
||
|
Vito
|
1ff2ddce05 |
fix(inspector): fall back to the positive's category and tags for recipe-linked images (#267)
* fix(inspector): fall back to the positive's category and tags for recipe-linked images Maintainer ruling D39 (2026-10-21): a recipe-linked image shows the recipe's own category and tags when set, otherwise the linked positive prompt's, each field on its own and without merging tag sets. Set category (#259) and the Images category and Tags filters work on the positive, so the panel now agrees with them. The inspection library replaces metadata_scope with category_scope and tags_scope so the category chip's hidden scope prefix (D21) and the tag chips' filter-button rule (#260 D2) follow the value actually shown. History's media inspector uses the same shared rule. The concept fixture's positive_links workaround is gone: the comparison now matches the concept without it. * test(inspector): assert the D39 fallback tag chips in their saved order * fix(inspector): treat Unicode whitespace as a blank History recipe category * test(inspector): cover mixed-scope panels, invalid library scopes and the skipped positive tags query |
||
|
Vito
|
10a4847a6b |
feat(history): concept heading, negative variants and compare (#265)
* feat(history): concept heading and summary The Prompt history page takes the approved mock's heading and summary surface (docs/concepts/workspace/app.js renderHistory a591-598, #249 F1): - Heading description is "Prompt history · one positive, one shared gallery"; the actions stay "← Prompt library", "Edit metadata" and button.send "Send to ComfyUI" (aria-label "Send prompt pair to ComfyUI: <title>"), on one row at 1600. Send is disabled only while the selected pair's roles are loading (D24). - The summary is article.surface > span.chip category, the positive and selected negative prompt blocks, then .card-meta "N tracked runs · N images · N negative variants". Tracked runs come from the recorded history window's recorded_run_count (hidden until a read completes, kept while a refresh is in flight); images and variants from the saved positive's image_count and variant_count. - The summary's copy/send actions are named after the prompt, as in the concept ("Copy positive prompt: <title>"); mountPromptBlocks gains setTitle so an in-place metadata edit renames them without a remount. - mountRecordedHistory gains an optional onRecordedCount callback. Selector mapping: - .library-surface (history summary) -> .library-history article.surface - .library-surface .library-chip -> article.surface > span.chip - "Copy|Send <role> prompt[ to ComfyUI]" in "Saved prompt texts" -> "...: <title>" - heading p.muted "One positive prompt. Reusable negative variants. One shared gallery." -> "Prompt history · one positive, one shared gallery" - Unchanged: region "Saved prompt texts", .library-variants, .library-shared-gallery, .recorded-history (still .library-surface). Tests: workspace-history "history heading actions stay on one row at 1600 and are enabled after load" and "summary meta uses recorded runs" (both failed before), a prompt-blocks setTitle unit, and the mapped selectors in workspace-visual, workspace-library-editor and the canonical comparison. * feat(history): concept negative variants panel The history's Negative variants panel is the concept's (app.js a598, style.css 1059-1154): section.surface.timeline-panel with the panel heading (h2, p.small "Choose a red dot to change the negative and filter this gallery."), row actions Compare negatives (aria-pressed; its section is Task F3), + Add negative and the approved Show archived; a .timeline of .positive-lane and .variant-lane rows (--lane-offset 38 + 47i) with the "−" .variant-dot, "<name> · N runs" and a .run-dot when the pair's run_count > 1; and .timeline-lower with the legend and All images for this positive. D26: All images for this positive opens the concept dialog with the embedded, bounded (Load more) media gallery instead of filtering the shared gallery. The run dot opens Recorded run for the pair's newest recorded execution, else a Repeat run dialog with the pair's newest linked image. Backing data: - library variants carry prompt_pairs.run_count; - GET /prompt_manager/library/{positive_id}/history/pairs/{pair_id}/newest reads the newest run in the positive's exact raw-text scope whose current-epoch saved-pair hint still resolves to that pair (the get_history_run hint rule), inside the fenced history snapshot. - workspace-dialogs gains openContentDialog (onClose releases the body). - mountRecordedHistory returns openRun. D45 (new deviations): - a dot is named "Select negative <title>" when no other loaded lane has that title, else "Select negative variant N", so names stay unique; - the bounded "N of M saved negative variants" count stays a visually hidden status; "Saved combinations, not an execution timeline." is gone; - archived lanes (#235) are concept lanes marked .library-variant-archived at .6 emphasis with their Restore button; coarse dots keep a 47px target; the heading, legend row and All images button wrap at 390px and 200% zoom. Retired CSS: .library-branches, .library-positive-lane, .library-variant-lane, .library-variant-dot, .library-variants, .library-panel-heading, .library-variant-footer, .library-variant-summary. Selector mapping: - .library-variants -> .library-history .timeline-panel (region "Saved negative variants" unchanged) - .library-branches -> .timeline; .library-positive-lane -> .positive-lane - .library-variant-lane -> .variant-lane; .library-variant-dot -> .variant-dot - .library-variant-archived and .library-variant-restore unchanged - .library-variant-footer / button:last-child -> .timeline-lower / button - .library-variants > p.muted -> .timeline-panel .panel-heading .small - "Select negative variant N" -> "Select negative <title>" for titled pairs (history, native library, inspector, visual and real-host fixtures); untitled pairs unchanged - lane text "· N linked image(s)" -> "· N run(s)" - All images for this positive: aria-pressed filter -> dialog "All images for this positive"; tests that used it for the whole gallery select the lane linking the image, or read the dialog - variant panel row actions gain Compare negatives first Tests: new workspace-history "variants panel matches the concept", "run dot appears only for pairs with more than one run and opens its run", "all images dialog lists every image for the positive"; new workspace-library-editor "show archived and restore still work" (all four failed before the change). Python: variants carry run_count, newest pair run (DB and route). Unit: content dialog. The history fixture counts two canonical pairs and links one image to Clean geometry. * feat(history): compare negative variants Compare negatives now opens the concept's comparison (app.js a599-616, style.css 1155-1179) as a section.surface right after the timeline panel: h2 "Compare negative variants", p "Baseline compared with <name>. Choose another red dot to change the comparison.", a .compare-grid of two .compare-item negative blocks (baseline, selected), p.small "Added terms", then span.diff-added "+ term" per added term or p.small "No added terms.". Choosing another red dot updates the same section in place; pressing Compare negatives again removes it. D25: the baseline is the oldest active variant; with no selection the selected side falls back to it, as the concept's v does. The pure addedTerms(base, selected) in history-state.mjs is the concept's rule: comma split, trim, drop empty terms, exact case-sensitive difference that keeps the selected order and repeats. D46 (new deviations): - lanes are newest first, so the baseline is the lowest-id active variant, not the first lane; while the variant page has more, the oldest is not loaded and the section says "Load more negative variants to reach the baseline." instead of guessing; - the two blocks are named "Baseline" and "Selected" (regions and copy/send actions), so no name repeats on the page; - Compare negatives is disabled when there is no active variant. Selector mapping: none changed; the section is new (.library-history .timeline-panel + section.surface). Tests: unit "added terms split, trim and exact-diff like the concept"; workspace-history "compare toggles, follows the selected dot and stacks at 390" and "compare waits for the page holding the oldest active negative" (all failed before the change). * test(history): guard the newest-pair route and record D45/D46 TESTING.md lists the Prompt history page's spec decisions (D24-D26) and its new deviations D45 and D46 beside D23 and D40-D44, notes under D46 that production opens on the newest lane while the concept opens on its baseline, adds GET /prompt_manager/library/{positive_id}/history/pairs/ {pair_id}/newest to the history route list with its guards, and documents the detail variants' run_count. Tests: the newest-pair read skips a stale reference_epoch row, stops at 16 stale-hint candidates (17 newer stale rows hide an older valid run and report {run: null}), and rejects a negative-only or deleted positive; the route answers 404 for an ineligible positive, 503 storage-unavailable for storage failures without leaking details, and 409 history-changed for a restore during the snapshot. * fix(history): run-dot touch target, unique dot names and dialog cleanup - D45: under pointer: coarse the 12px run dot gets a 44px ::after target, and while a run dot shows the lane text wraps within 139px before it. - D45: dot names compare trimmed titles and never use a title that reads like a fallback "variant N", so names stay unique. - All images and Repeat run share one dialog-scope slot: each open replaces the previous scope instead of adding a cleanup per open. - A run-dot lookup that settles after the view went stale opens nothing. - Retire the unused .library-main .library-chip rule and the stale "Saved combinations" comparison note. Tests (workspace-history, Chromium and touch): the coarse run-dot target and lane width; a failed newest lookup shows the toast and Repeat run; Recorded run that cannot open (recorded history failed) falls back to Repeat run; an archived pair's Repeat run reads archive=archived; Compare negatives is disabled with no active variant; dot names stay unique. * fix(workspace): one inert handle for aborted history and prompt-block mounts An aborted parent no longer gets a separate never-called stub per method: mountRecordedHistory returns its hoisted openRun (it answers false before reading later state), and mountPromptBlocks returns one shared no-op for update, setState, setTitle and dispose. This keeps the history.mjs and prompt-blocks.mjs function-coverage floors (98) without lowering them. Tests: test:coverage (Chromium and touch, 922 workflows) passes for all 61 files; it failed before on history.mjs 94.44 and prompt-blocks.mjs 93.75 function coverage. |
||
|
Vito
|
3476f8ede9 |
feat(inspector): concept comparison, Show in folder and toast placement (#264)
* test(inspector): concept comparison for the panel and viewer details Compare the Images side panel and the viewer's full details with the concept (1600x1100 and 390x844, both themes): structure, tokens, geometry and controls, plus copy, the acknowledged pair send, the tag chip filter and Prompt history. mapInspectorConcept applies only D3, D20, D21 and D22; a mutation test shows the comparison notices a changed role label and a removed footer. The comparison showed the role copy and send names lacked the concept's ": <title>" subject, so the inspector now passes its title to the prompt blocks. It also showed that a recipe-linked image shows its recipe's own category and tags (#224), which the concept fixture's recipes lack; that is recorded as D38, and the fixture's new positive_links option links the compared image to its positive alone. * docs(inspector): renumber the recipe-category decision to D39 * feat(images): local-only Show in folder route POST /prompt_manager/images/reveal takes the listing identity {root_index, root_scope, path} and opens this computer's file manager at that file. It answers 403 not-local unless the client is loopback, no proxy header is present and Host/Origin name this computer; the scope is checked with _gallery_scope_error (409) and the path resolves only inside that root to a regular media file. Openers run as argv lists with an absolute executable, no shell and DEVNULL streams (D38). * feat(inspector): Show in folder after the File & provenance path The viewer's File & provenance offers "Show in folder" right after the Path value for a listed file only, POSTs its listing identity and toasts the outcome (opened, not-local, unavailable or failed). The browser test server stubs the opener; TESTING.md records the decision as D38. * fix(images): keep the discovery alert clear of the Rescan toast A rescan that leaves the Images discovery alert visible could show its fixed bottom toast over the alert on a short window. The concept's toast keeps its placement; the production-only alert now scrolls into view only as far as needed, with a 96px block-end scroll margin (D42). * docs(inspector): concept panel and viewer details * fix(images): open Windows folders through the shell API, not an Explorer command line Explorer splits its own command line on commas, so a gallery path such as output\a,\Windows\System32\calc.exe,b\x.png escaped the roots through [explorer, "/select,", path] (security review H1). Windows now selects the file with ILCreateFromPathW and SHOpenFolderAndSelectItems, and on any shell failure falls back to os.startfile on the folder. No process is started. * fix(images): re-check, isolate and reap the Show in folder opener - re-check that the real path is still a regular file inside its root right before launching; a changed target answers 404 (review L4) - run openers on a dedicated single worker, not ComfyUI's default executor, and make dbus-send wait for the FileManager1 reply (--print-reply --reply-timeout=2000) so a missing service falls back to xdg-open (L1) - reap launched processes on a daemon thread, give them a minimal environment with a fixed system PATH, and use /usr/bin/open on macOS and dbus-send/xdg-open resolved only on that PATH (L5) * fix(images): accept Show in folder only as a same-origin JSON request, one at a time - require an Origin equal to the request's own scheme://host:port and, when sent, Sec-Fetch-Site: same-origin; refuse every forwarding-style header (Via, X-Forwarded-*, X-Original-*, *-Client-IP, CF-Connecting-IP and the rest) as remote (security review M1, L2) - answer a non-JSON body with 415, so a cross-origin request always needs a CORS preflight (L2) - single flight plus a 1 s cooldown per process: 429 code busy (L1) - path resolution errors such as ENAMETOOLONG are a JSON 404, and neither that nor a start failure logs the client's path at error level (L3) * fix(inspector): say the folder is already opening on a busy Show in folder A 429 busy from the reveal route now toasts "The folder is already opening." The browser workflow checks that the reveal POST carries the page's own Origin and an application/json body, and TESTING.md D38 records the tightened route, the toast table and the accepted residual risk of a header-less local proxy. * test(inspector): compare the viewer details under a coarse pointer and prove its mutations The touch project measured production's D34 40px disclosure targets against the concept's fine-pointer summaries. The concept's plain-chip copy now also carries the recorded D34 coarse rule, so both documents are measured under the same emulation. The open File & provenance is compared too (D38 Show in folder and D20 Download original mapped), and the mutation check now removes the viewer's pair actions and Show in folder. * refactor(inspector): attach Show in folder to the Path value it follows The button was placed on the second fact row by position; the Path dd is now built by the caller and kept, so reordering the facts cannot move it. * test(inspector): let the uncertain-file send check fail again Inspector send buttons are named "... to ComfyUI: <title>", so the anchored regex could never match and the no-send assertion always passed. * fix(images): run the Windows folder fallback inside COM and keep desktop session variables Show in folder's os.startfile fallback now runs inside the same CoInitializeEx/CoUninitialize bracket as the shell selection, as ShellExecute callers should. The Linux opener keeps XAUTHORITY, DESKTOP_SESSION, XDG_SESSION_DESKTOP and XDG_CONFIG_HOME, so a file manager can reach the X11 display under GDM or SDDM and honour the user's mimeapps.list. A Windows-only test calls the real ILCreateFromPathW and ILFree on a temp file and pairs CoInitializeEx with CoUninitialize through the module's helper, without SHOpenFolderAndSelectItems, so no window opens on CI. * fix(images): scroll to the rescan's discovery alert only when the toast band covers it D42 scrolled the alert into view after every rescan, so a user scrolled down the grid was sent back to the top. It now scrolls only when the alert intersects the toast's band (its scroll-margin-block-end). * refactor(inspector): name prompt actions as PR #263 does The title option keeps its name, default and output ("Copy positive prompt: <title>") and now uses #263's actionName helper; the region-name arrow no longer shadows the title option. * test(inspector): the comparison opens Filters by its D44 active-count name After a tag chip applies its filter, the shared toolbar's Filters button reads "Filters · 1" (D44, #263), so the comparison opens it by that name. * fix(images): POSIX opener paths stay absolute under Python 3.13 on Windows Python 3.13 on Windows no longer counts "/usr/bin/x" as absolute, so the faked-Linux reveal tests failed on windows-latest py3.13 (the Linux and macOS openers are POSIX paths whatever host evaluates them). Use posixpath.isabs for those lookups. |
||
|
Vito
|
0a7b0238ca |
feat(library): concept prompt library cards, toolbar and caption (#263)
* feat(library): concept toolbar and caption
The Prompt library shelf uses the shared concept toolbar (search, category,
sort, Filters): sorts are Recently added / Name A-Z / Highest rated, then
Oldest first; the Tags filter moves to "Refine this view" (D14) and Refresh
library is removed (D15), with Retry kept and re-reading failed categories.
The caption is the concept's .section-caption, "N positive prompts · M
negative variants" over the filtered set from variant_total.
Library-embedded media cards keep the favorite star working: the Library
gallery read already reports favorite (one batched lookup per page, now
covered by a test), and star answers are delivered to every mounted embedded
gallery, so a star toggles, repaints and survives a reload. The checkbox and
selection bar stay Images-only.
* feat(library): concept prompt cards with real runs and last use
The Prompt library shelf renders the concept's article.prompt-card:
.card-art (up to two previews), .card-title with a title button and the
category chip, the positive prompt block, .card-meta ("− N negative
variants", "N runs", "N images"), .tags, and .card-footer with the rating
and last-use line and the positive send plus Explore history buttons.
D23: "N runs" is the parent's run_count; the footer reads Used today /
yesterday / N days ago / <Mon D> in the viewer's local calendar days, or
"Not used yet"; stars carry a visually hidden "Rated N out of 5" and an
unrated parent shows no stars. The positive is clamped to two lines while
copy and send use the full text. Long titles wrap inside the title row
and the category chip never breaks mid-word. Card actions are named after
the prompt ("Copy positive prompt: <title>") through a new title option on
the shared prompt blocks. The production .library-card CSS is removed and
the row estimates follow the clamped card height.
The shelf error now reads "Prompt library could not be loaded. Retry."
(Refresh is gone). TESTING.md records D23, the production-only shelf copy
(D40) and the concept search placeholder kept on the Library (D41).
* test(library): concept comparison for the shelf
library-concept.spec.mjs compares the Prompt library shelf with the
immutable concept on the concept-shaped fixture: heading, toolbar,
caption, grid and the six cards (art, title row, prompt block, meta,
tags, footer) at 1600x1100 and 390x844 in both themes and motion
settings, the 3-column grid at 1800x1100, and the behaviors in both
documents (title and Explore history open history, the footer send
carries the positive only, copy toasts "Prompt copied.", the "neon"
search and the Landscapes category). A mutation test proves the
comparison notices a footer padding change and a removed variants chip.
mapLibraryConcept applies only recorded deviations: D14, D16, D23 (title
min-width, singular variant chip, spoken rating, the two-line clamp the
concept's samples exceed at 390px), D35, D36, D40 and the new D42 (tag
chips in the store's alphabetical order). Each re-laid-out part is
checked against the concept page with that deviation applied at runtime.
Corrections the comparison demonstrated:
- the fixture creates the concept's prompts last to first, so Recently
added lists them in the concept's order;
- the shelf no longer ends with the legacy "Current prompt library"
link; Overview's legacy cards keep it, as with "Current gallery".
The shared inventory gains the Library section, and reflow/withoutItems
move into concept-inventory.mjs for both comparison specs.
* test(workspace): reach the legacy Library from Overview, not the shelf
The shelf no longer links the legacy Library (#249 E3). The shell
destinations test now expects no legacy link on Prompt library and
asserts Overview's "Open existing prompt library" card still links
/prompt_manager/admin and that the page loads.
* feat(library): Filters shows its active count; final review follow-ups
- D44: the shared toolbar's Filters button reads "Filters · N" while N
"Refine this view" filters are active (Library: Tags; Images: Source,
Folder, Media, Tags), so a Browse-set Tags filter is visible on the
shelf. The text is the accessible name. Unit and browser tests cover
Browse -> "Filters · 1" -> Clear -> "Filters" and the Images counts.
- Renumber the alphabetical tag chips deviation D42 -> D43 (D42 is #264's).
- The caption count is a role=status region written only when it changes
and kept shown while the shelf reloads.
- usedLabel names the year for a use in another calendar year (D23).
- prompt-blocks: same `title` option API as #264 (`subject` suffix); the
region-name helper's parameter no longer shadows it.
- Scope library.css .row-actions rules to the history panels; fix stale
CSS and spec comments.
- Retry re-reading a failed category list and the no-op Apply/search
paths are covered by a browser test.
- TESTING.md: D23 amendments (year, button vs link trade-off), D44, the
legacy Library link wording.
|
||
|
Vito
|
00bda8f442 |
feat(inspector): concept prompt blocks, side panel and viewer details (#260)
* feat(workspace): concept prompt blocks Shared prompt blocks (Images panel, viewer details, Library cards and saved texts, recorded runs) now render the approved mock's markup (app.js promptBlock a298-302, concept.css s550-588), #249 D22: - section.prompt-section.<role>[aria-label="<Role> prompt"] > .prompt-heading (span.role-label "+ Positive prompt" / "- Negative prompt"; .prompt-actions: copy button.icon-btn "Copy <role> prompt", then send button.send.icon-btn "Send <role> prompt to ComfyUI") + p.prompt-text. - No per-block source line; a file's evidence moves to the panel badge (D2) and "File & provenance" (D3). Recorded runs keep every field's source in "Frozen field provenance". - Empty text reads "No negative prompt (explicitly empty)" and still sends an explicit clear. Absent, ambiguous and unresolved roles render p.prompt-text.muted with the existing status strings and no actions; unknown role names render nothing and cannot be sent or copied. - Copy shows the shared toast "Prompt copied."; without a clipboard it opens the concept's "Copy prompt" select-to-copy dialog. - The data-send-role / data-copy-role contract is unchanged. The D22 interim overrides (26px actions, prompt-text margins, copy status, source line) are retired so the concept's rules apply; the action buttons measure the mock's 31x31. pre.prompt-text styling stays only for the Library editor's Archive dialog and frozen field provenance, which keep their own markup until their page rebuilds. Test selector mapping: .prompt-block / .prompt-positive -> .prompt-section(.positive) .prompt-block-heading h3 -> .prompt-heading span.role-label region pre / pre.prompt-text -> p.prompt-text "Empty negative prompt" -> "No negative prompt (explicitly empty)" "Send <role> prompt" -> "Send <role> prompt to ComfyUI" .prompt-source / copy status line -> removed (toast or Copy prompt dialog) copy button 30x26 -> 31x31 (measured in the mock) * feat(inspector): concept side panel The Images side panel is the concept's aside#image-inspector.detail-panel in the order of app.js imageDetails: plain cover, heading with the format chip, filename, the D21 evidence badge and category chip, prompt blocks, tag chips, the six facts and the footer (Send to ComfyUI pair, Prompt history). Provenance, raw records and downloads stay in the viewer's full details. Positive-scope tag chips set the Images Tags filter to that tag. * fix(inspector): mark the side panel's overlay rules with their D2 decision The overlay lint (tests/vendor/concept-css.test.mjs) requires a /* D<n> */ marker on every workspace.css rule that overrides a concept selector; the three .detail-panel rules from D2 lacked one. * feat(inspector): full details in the viewer The viewer's #viewer-details renders the concept's imageDetails(i, true): the side panel content without the cover in a class-less wrapper, then "File & provenance" and "Workflow & raw metadata" (pre[tabindex=0] > code and the two downloads with their unavailable reasons), then the pair send ("Send to ComfyUI", labelled "Send prompt pair to ComfyUI: <title>") and Prompt history buttons. One renderer now serves both variants. Provenance wording (maintainer, 2026-10-20): the six per-fact "<Fact>: Configuration evidence · <source>" lines and FACT_SOURCES are gone; the concept's "Prompt evidence: <badge>" line carries the D21 badge in lower case and is left out without a badge. The honest facts stay (current-file note, Source, Path, Size, Dimensions, Format, Media type, Modified, occurrence, association, Download original). An unverified format reads "PNG extension", no stray leading dot. inspector.css and images.css are deleted; only rules with a live consumer moved to the overlay under #249 D3 (inspector wrapping, the inert aria-disabled download, the narrow Images container, the Library and recorded-run prompt blocks, the Archive negative's prompt-block, the prompt sender) plus a D20 wrap rule for the viewer's buttons at 200% zoom. D1 review follow-ups: library.css no longer restyles recorded-run p.prompt-text.muted (m2); the copy fallback has a seam and unit tests for a missing clipboard and a refused write (m3); a stale test title is renamed (m4); the fallback trigger falls back to the same-role button when the clicked one was detached by a re-render (m7); TESTING.md records the D20 and D22 deviations (m1). Selector mapping in tests: .file-inspector-inner -> .panel-heading, .generation-facts -> .meta-grid (first), .file-facts -> details .meta-grid, .file-name -> #viewer-details > div > p.small, .file-inspector-footer -> #viewer-details, viewer "Send prompt pair" -> /^Send prompt pair to ComfyUI: /. * test(comfyui): real-host specs read the concept prompt blocks library-sender: an explicitly empty saved negative now reads the concept's "No negative prompt (explicitly empty)" (D22). The prompt-sender spec's panel and viewer selectors (complementary "Selected image details", the card image opening the viewer, .meta-grid facts, the labelled pair send) moved with the D3 commit. * test(comfyui): the frozen run's prompt regions read the concept prompt text * fix(inspector): coarse viewer-detail targets and Prompt history on the shown route - Prompt history closes the viewer and, when its target is the route already shown, re-renders it as the concept's go() does (navigateTo plus a router refresh event), instead of doing nothing. - The viewer details' disclosure summaries and "Download original" keep 40px targets on coarse pointers (#249 D34), as TESTING.md promises. - Drop the dead `.prompt-positive` rules and merge the two `pre.prompt-text` rules. - TESTING.md records the side panel's D20/D21 deviations. |
||
|
Vito
|
2b8126b7be |
feat(images): card selection checkbox and selection bar (Set category, Export metadata) (#259)
* feat(images): card selection checkbox and the concept selection bar
Every Images card's overlay starts with the concept's "Select <filename>"
checkbox before the star. Checked image keys form the page's selection,
kept across Load more, infinite scroll and bounded re-mounting and cleared
whenever the collection is replaced. The #bulkbar between the filter row
and the gallery shows "N selected" with Set category, Export metadata and
Clear (the first two are wired by the next tasks); Clear empties it.
The interim D8 star rule is removed, and the concept comparison now
measures the overlay with its checkbox and the bar with one card checked.
* feat(images): export the selected images' metadata as one JSON bundle
The selection bar's Export metadata reads each checked image's inspection,
at most four at a time, and downloads prompt-manager-images-metadata.json:
format, version 1, exported_at and, in listing order, the same envelope the
inspector's "Download metadata JSON" saves (one shared builder,
metadataEnvelope). The button is aria-disabled and ignores a second press
while it runs, the selection is kept, and the toast reports the exported
count, the images that could not be read, or that none could (no download).
Also: the bar and its count are built only on the Images page, the
Library-embedded gallery is asserted to have no checkbox or bar, a sort
change is asserted to clear the selection, the checkbox label test asserts
the toggle and an unchanged selection and inspector, and TESTING.md records
that a return to Images keeps the selection while a reload starts empty.
* feat(images): set the category of the selected images' linked prompts
The selection bar's Set category opens the concept's simpleInput dialog
("Categorize selected images", one Category field, Cancel / Apply) and
posts the selection's distinct linked positive ids to
/prompt_manager/bulk/category. Images have no category of their own; the
listing's category filter matches the linked positive's, so that is what
changes. With no linked image checked nothing is posted and a toast says
so; on success the toast counts the prompts and any skipped images, then
the categories and the listing reload (clearing the selection); a failure
keeps the selection. The button ignores a second press while it runs.
The bulk category route now rejects anything but a non-empty list of at
most 1000 positive integer ids (deduplicated in order) and validates the
category with validate_category; bad JSON or a non-object body is a 400.
The response shape and the legacy admin request are unchanged.
* test(images): zoomed card keyboard order follows the concept's checkbox before the star
The card's Tab order is image, Select checkbox, star, Inspect (concept app.js:398), both ways; List hides the overlays as the concept does, so there the image is followed by Inspect.
* fix(api): answer 400 for bulk category ids beyond SQLite's INTEGER range
An id of 2**63 or more passed the parser and made sqlite3 raise OverflowError, which the route turned into a 500.
* refactor(images): move the selection bar into images-selection.mjs and harden its actions
- images.mjs is back under 800 lines; the bar, its checked set, Export metadata and Set category live in images-selection.mjs.
- Busy flags release in finally; the dead guards are gone and the category targets are taken when the modal dialog opens.
- The category write no longer carries the page signal; leaving Images mid-write drops the saved visit so the return reloads.
- The categories and the listing reload together after Set category.
- The Category field stops at the server's 100 characters; the server's updated_count is a tested pure helper.
- New browser tests: leaving mid-export and mid-write, Blob URL revoke, updated_count 0 and a 400 rejection.
- Additive coverage baseline entries for images-category, images-export and images-selection.
- TESTING.md D8 records the dialog deviations and these rulings.
|
||
|
Vito
|
86d82186ee |
feat(images): concept Images page — toolbar, cards, star, viewer, compared against the mock (#258)
* feat(workspace): shared concept toolbar and dialogs
Add the concept's shared page toolbar (search.toolbar with the search
field, category filter, sort and Filters) and its dialogs in one modal
shell: "Refine this view" (Source / Folder / Media / Tags, Apply filters),
"Import & inspect" with only the real operations (Open inspector, Rescan),
and the "Copy prompt" select-to-copy fallback. The sort lists the
concept's three options first (D14); there is no Save view (D16). Every
dialog returns focus to its trigger on close or Escape.
Refs #249
* fix(workspace): bind shared dialogs to the page and its route
- Drop the invented Refine note; every field is real (D14).
- Ignore a late close event once the shell was reopened or replaced, so
an action that opens the next dialog keeps it open.
- Dialog openers require the page scope; disposing it closes and removes
the shell unless a newer page's dialog is open. Every route change
closes the shared dialog before the next page mounts.
- Clear filters also reports category '' so the page clears the toolbar
category, as the concept does.
- The toolbar requires its scope and falls back to the first sort for an
unknown value; dialog callbacks are validated up front.
Refs #249
* feat(images): concept toolbar, filter row and caption
The Images page now renders the approved mock's heading, shared toolbar,
filter row, section caption and infinite continuation (#249, Task C2).
- Heading: "Your generations, their prompts, and the next idea." with
Import (upload icon) before New prompt (primary).
- Toolbar (shared mountToolbar): search over filenames, linked prompt
text, titles and tags; category (the Library's categories); sort
Recently added / Name A–Z / Highest rated, then Oldest first and
Name Z–A (D14).
- Filter row: Grid/List/Masonry segmented control, Thumbnail size,
the scope chip ("All output folders" or "<source> · <folder>") and
"Group by positive prompt →".
- Caption: "N of M images · K positive prompts" from positive_count; a
plain cursor page keeps the first page's count and an unknown count
hides the "· K positive prompts" part. The date span shows
"<Today|Yesterday|weekday> · <Month D>" of the newest loaded result
under a date sort and is hidden otherwise (D17).
- Continuation: "N of M images · more load as you scroll" with Load
more, the concept's end copy, Retry for a failed page, and the
concept's "K more images loaded. N of M." toast.
- The listing sends category, tags and sort=rating (PR B params).
Old control -> new control:
- "Search filenames" field -> toolbar search "Search workspace"
(placeholder "Search prompts, tags, filenames…"), also matching
prompt text and tags
- Source select -> Filters -> "Refine this view" Source
- Folder select -> Refine Folder (all sources' folders and ancestors)
- Media select -> Refine Media
- (new) Tags -> Refine Tags; Category -> toolbar category select
- Sort select (Newest/Oldest/Filename A–Z/Z–A) -> toolbar sort
(Recently added/Oldest first/Name A–Z/Name Z–A, plus Highest rated)
- Layout buttons -> .segmented Grid/List/Masonry in the filter row
- Thumbnail size field -> label.small + select#image-size
- "Group by positive prompt" link -> button.chip "Group by positive
prompt →" (#/library)
- Refresh -> Import -> "Scan an output folder" -> Rescan (D15)
- "Current gallery" link -> removed; Overview's tool cards link the
legacy gallery (D29)
- "All images" h2 + count -> .section-caption #result-count
- .image-continuation -> .infinite-continuation (Retry kept)
Legacy behavior kept: video items, keyboard grid navigation, scroll and
filter restoration, bounded DOM, continuation retry, source-change
recovery (now disabling the toolbar, Filters and Import). The Filters
and Import dialogs close on route change and browser Back.
The shared toolbar (C1) called .some() on select.options, which is an
HTMLOptionsCollection in a browser (only the unit tests' fake DOM gave
an array); setValue and setCategories now spread it first. The layout
segmented control may wrap at extreme zoom (D31) instead of scrolling
the 390px page sideways.
The retired .image-filters, .image-view-controls, .image-toolbar,
.image-layouts, .image-collection, .image-continuation and duplicated
.image-workspace grid rules are gone from images.css.
* test(workspace): model select.options as a browser collection
The toolbar units' fake DOM returned select.options as an Array, so the
toolbar's .some() call passed there while it threw in every browser
(fixed in
|
||
|
Vito
|
6c82098207 |
feat(images): data for the concept cards — linked titles, seed, search and filters, counts, favorites (#257)
* feat(images): link summary on workspace image items
* fix(images): carry title source and cut first-line titles like the Library
Summaries gain title_source (stored or text). The inspector always shows a
stored title and confirms only a first-line title against the file, so a
stored title equal to the first line is no longer withheld. A link-title read
failure no longer marks a readable link unavailable. First-line titles over
80 characters end with an ellipsis, as the Library's title() does. The card
gate for unconfirmed first-line titles is pinned as an expected failure for B2.
* feat(images): unambiguous seed on workspace items
Workspace items gain `seed`, read with the shared generation-parameter
parser from the prompt/workflow records the header open already loaded
(each at most 2 MiB; no extra file reads, videos give None). A
placeholder or non-integer seed gives None. JSON integers beyond 2**53
are recovered exactly, so the card prints the digits the inspector shows.
The same parse confirms first-line card titles: a `text` title shows only
when the file's parsed positive gives the same title. One helper applies
that rule on the card and in the inspector, which now reads the positive
from its own inspection records with the same parser.
* fix(images): a deeply nested record no longer fails the workspace listing
The Python parser recurses over nested arrays where JavaScript String()
does not, so a crafted 2 KB PNG turned every workspace page holding it
(and its inspect route) into a 500. Card facts now treat RecursionError
like any record the parser cannot use: no seed and no confirmed title.
Also: Library previews skip the card parse they never show; the parse
runs after the file handle closes; the exact-seed read uses the one JSON
reader (parse_js_json(..., exact_integers=True)) and the public
number_string; record size is decided from the text length unless only
the UTF-8 byte count can tell.
* feat(images): search and filter workspace images by linked prompts
The workspace listing now takes category (exact), tags (comma list, all
required) and sort=rating, and q also matches the linked positive's text,
Library title, pair title and tag names. One bounded, parameterized SELECT
(get_image_link_matches / get_image_paths_matching) returns normalized
paths of unambiguously linked images; the listing is intersected with it.
Unlinked and ambiguous files are excluded by a category or tag filter and
sort last by rating, with a newest-first tie-break. The cursor scope covers
the new filters and sort; unknown values are still rejected.
The derived listing values and sort orders are cached per gallery scan so a
50,000-file listing with 20,000 links pages well under 150 ms at p95.
* test(images): count the filter budget everywhere, time it in one job
The 150 ms wall-clock budget failed under the CI coverage tracer, so it
is split. test_filter_work_is_bounded runs on every job and under
coverage: one filter SELECT per request, the ambiguity probe is an index
search (EXPLAIN QUERY PLAN), the per-scan values are built on the cold
request only, and a cursor page calls the sort key at most
ceil(log2 n)+3 times. test_filter_budget runs only with
PROMPTMANAGER_PERF_BUDGET=1 and no tracer, asserts the best of three
rounds' p95, and has its own non-required "perf budget" job. TESTING.md
records the convention.
* fix(images): rating sort works however many link rows the database holds
The filter SELECT counted every generated_images row against a 500,000
row cap and materialized all of them first, so a large history made
sort=rating answer 400 with nothing to narrow. The rows are now streamed
and only paths in the listing's normalized set are kept, so memory is
bounded by the listing; the cap and LinkFilterTooBroad are gone.
* fix(images): rebuild workspace listing values only where the scan changed
A TTL rescan that finds the same files reuses the previous values and
sort orders; one that finds a few changes rebuilds only those files.
Gallery invalidation and a gallery-roots change drop the derived values,
so old roots no longer hold memory. category is trimmed like tags.
* fix(images): a monitor invalidation rebuilds only the new files' values
The image monitor invalidates the gallery whenever it sees a new file,
and that dropped every derived workspace value, so the next request paid
a full cold build. Invalidation now keeps the per-file values and drops
only the sort orders; the rescan reuses every unchanged file and builds
only new or changed ones. A gallery-roots change still drops everything.
* feat(library): expose run count, last use and variant totals
* fix(library): an out-of-range last use reads as unknown, not a failed list
* perf(images): read positive counts only on a plain listing's first page
* feat(images): persist image favorites
Favorites are stored in image_favorites keyed by the gallery root label
and the relative path, set and cleared through PUT/DELETE
/prompt_manager/images/favorite with a {root_index, path} body validated
against the rooted listing. Workspace image cards carry favorite: bool.
The table is created by migration, travels with backups, is optional in
old backups and required after a restore's migration.
* fix(images): key favorites by the real root and case-normalized path
Folder names are not unique across gallery roots, so two roots shown as
"output" shared each other's stars. Favorites are now keyed by the
root's real, case-normalized path (server-side only) and a
case-normalized relative path, so a Windows case-only rename keeps its
star while distinct Linux files stay distinct. Toggles look the file up
in a per-scan dictionary instead of walking the listing.
* fix(images): favorites lookup follows monitor invalidation and the stored key bound
A monitor invalidation now also drops the favorites lookup, which held the
previous scan list until the next toggle. PUT refuses a favorite whose root
or relative key would exceed the schema's 4096-character bound, so the live
database keeps passing startup and restore validation.
* docs(testing): workspace data fields for the concept pages
* fix(images): card facts read the inspector's projected graphs
The inspector hands its browser parser only graphs the inspection
projection parsed (strict JSON, depth 64, node ceiling); a deeper record
is "limited" there, so the legacy fill never runs and no seed shows. The
card parsed the raw records instead and showed a seed the inspector did
not, e.g. for a 900-deep class_type beside a normal sampler.
project_graph() is now the one projection step, used by the inspection
response and by the card. The card runs the shared parser on the
projected graphs, the same values the browser receives: big integers
arrive as their exact digits, so the separate exact-integer re-parse and
the parse_js_json flag it needed are gone.
* fix(images): send the card seed as a decimal string
JSON.parse rounds a number past 2**53, so a listing seed such as
9223372036854788153 reached the page as 9223372036854788000 while the
inspector showed the file's digits. Every non-null card seed is now its
decimal string.
The Node parity harness now JSON.parses the real listing response and
compares String(seed) with the seed the inspector shows, including a
seed past 2**53. TESTING.md describes the string contract and drops the
removed exact re-read.
* fix(images): bound digit-string seeds and isolate each card's parse
A seed recorded as a 5000-digit string reached int(), which raises
ValueError past Python's integer string limit, and the listing answered
500. A digit-string seed now has at most 20 digits (2**64 fits), and the
conversion falls back to no seed on ValueError.
Each card's parse is also guarded in the page builder: an unexpected
error in one file's records logs a warning and gives that card no seed
and no first-line title, and the rest of the page still lists.
* fix(images): filename search survives a failed prompt-link read
On main a workspace q matched only file paths, so search worked with a
broken database. Since search also matches linked prompts, any q turned
a failed link read into a 500. A q without category, tags or the rating
sort now falls back to filename matches with a null positive_count and
logs the failure with its traceback; those linked filters still fail.
* fix(images): card reads PNG records with the inspector's reader
The card read PNG text through Pillow's header info, where a later
duplicate prompt chunk silently wins and text after IDAT is missed. The
inspector's bounded reader refuses conflicting duplicates and reads text
after IDAT, so a file with two different prompt chunks showed a card
seed the inspector refused.
PNG cards now read their records with read_inspection_file, the reader
the inspector uses. When it refuses a file the card keeps its
dimensions from Pillow and shows no seed. Other formats still use
Pillow's header info, as the inspector's header read does.
* perf(images): read each card's header facts once per gallery scan
Every page parsed each card's prompt and workflow records again, about
3.4 ms per 50 KB record set and 25 ms per 400 KB workflow, inside
ComfyUI's server process, on every scroll-back and refresh.
Dimensions, the seed and the parsed positive's display title are now
cached per root by (path, mtime), beside the scan's per-file values and
invalidated the same way: a rescan keeps unchanged files and prunes the
rest, a roots change drops them all, and only files of the current scan
are stored. An unreadable header is not stored. Links and favorites
are still read on every page.
* ci: run the perf budget on main pushes and on demand only
The wall-clock budget has under 2x margin on shared ubuntu runners, and
a red exact-head check blocks a merge that must never be rerun to green.
The job now runs on each push to main and on workflow_dispatch, not on
pull requests. TESTING.md says a red run on main means fix the code.
* fix(library): cut display titles at 80 UTF-16 units, never inside a pair
The server counted code points while the Library's JavaScript counts UTF-16
units, so a first line of emoji kept 80 emoji on cards but showed 40 in the
Library. Both now cut at 80 UTF-16 units and drop a character that would
straddle the limit instead of leaving half a surrogate pair.
|
||
|
Vito
|
9d398f9862 |
feat(workspace): concept shell — sidebar, topbar, heading and footer on the approved mock's CSS (#256)
* build(workspace): vendor the approved concept stylesheet with a drift check * refactor(workspace): adopt the concept stylesheet as the workspace base * feat(workspace): concept sidebar with real navigation counts * feat(workspace): concept topbar with Theme and ComfyUI link * refactor(workspace): one concept heading and footer for every route * fix(workspace): keep the canvas link on the proxy mount A resolver failure no longer stops the shell from mounting: the ComfyUI link falls back to the root read from the page's own mount path (never another origin). Tests cover a non-standard /ComfyBackendDirect mount, including that no request escapes it. * test(workspace): concept fixture and shell comparison * test(workspace): concept fixture carries the concept inspector facts * fix(workspace): match the concept brand name and compare shell hover states * test(workspace): shell comparison mutation check and docs Add mutate() to the shared concept inventory and a shell mutation check: a nav-row min-height rule and a removed .breadcrumbs each make the comparison fail with the changed path. Harden the resolver-failure guard test with an injection count, make the overlay lint fail when workspace.css is missing, mark the library.css .row-actions leftovers for PR E, and document the concept CSS base, the overlay lint, the comparison method and the recorded shell deviations in TESTING.md. * fix(workspace): focus the page title on a route change A sidebar or history navigation now focuses #page-title, as the concept does on every hashchange (app.js 1168), so a screen reader announces the page heading instead of the main region. The first render leaves focus alone and the skip link still focuses main. The heading module's comment now describes what the shell does. * fix(workspace): give the Overview nav entry its own home glyph Overview reused the concept's Activity pulse byte for byte, so once the Activity row lands two entries would share one icon. Overview now has an outline home glyph. The shell comparison takes it from the D28 ruling instead of the concept's activity path, and the icon test checks that every approved glyph is distinct. * test(workspace): accept only #249 decision markers and state the lint scope The overlay lint accepted any D-number, so another issue's ruling such as "D5 (#245)" could mark a concept override. A marker now has to be a D-number with no other issue cited; the vocabulary rule that cited #245 reads "#245 decision 5". TESTING.md now states that the lint covers workspace.css only, lists the unmarked overrides still in images.css, and says PRs C and D run the lint over each page sheet they keep. * perf(workspace): re-read only Library and Tags counts after a Library write A confirmed Library or vocabulary write cannot change the Images count, but it re-read all three, including the Images output listing. It now re-reads only the Library and Tags counts. Each count has its own read, so a write never cancels an Images read that is still in flight. |
||
|
Vito
|
6d2213ead5 |
feat(inspect): declarative node support matrix (#250)
* feat(inspect): add a declarative inspection node semantics table
* fix(inspect): bound and isolate the semantics table
Bound the semantics loader: at most 1 MiB is read, containers may nest
at most eight levels, and a RecursionError from the JSON parser becomes
a SemanticsError instead of escaping.
The table is no longer loaded at import. get_semantics() reads it on
first use and caches only a valid result, and the resolver imports it
lazily, only for a profile with semantics on. The default ingestion
profile never imports the module, so a missing or corrupt table cannot
break scan, monitor or admin import. Inspection of a prompt graph with
an unusable table is unresolved with the reason "Inspection node
semantics are unavailable" and an error log, rather than a crash.
INSPECTION_PROFILE now has semantics on; no strategy consumes the table
yet, so resolution is unchanged.
* feat(inspect): resolve guider-based samplers
Under INSPECTION_PROFILE the resolver discovers samplers from the
semantics table and follows SamplerCustomAdvanced through its guider
link (Reroutes allowed) to a listed guider:
- CFGGuider: positive and negative, so the pair resolves.
- BasicGuider: positive only. The state is "absent" with the reason
"Guider has no negative input", the existing missing-negative path.
- Any other guider, including DualCFGGuider (D17), is unsupported-node
and names that guider.
Guider hops draw visits, scan and depth from the existing budgets. A
SamplerCustomAdvanced upstream in latent ancestry is an earlier stage,
so the result stays multistage-ambiguous. Positive and negative
provenance record guider_id. FaceDetailer (detailer-stage) is not a
candidate yet: deciding it needs output ancestry. DEFAULT_PROFILE never
loads the table and is unchanged.
Vector changes (inspection profile only, default vectors unchanged):
- f07b: no-sampler -> resolved (CFGGuider).
- f17: no-sampler -> unsupported-node (FluxGuidance, Task 2.3).
- f01b: no-sampler -> unsupported-node (EmptyFlux2LatentImage latent,
Task 2.5).
- f01: no-sampler -> ambiguous (24 SamplerCustomAdvanced branches).
The no-sampler tests (resolver and inspect route) now use a sampler
class the inspector does not know, because f17 has a supported one.
* feat(inspect): trace conditioning through supported pass-through and selector nodes
* feat(inspect): treat detailers as sampler stages
* fix(inspect): choose selectors that share one condition together
* fix(inspect): bound detailer stage chains by the depth limit
* feat(inspect): support common text encoders and string sources
* fix(inspect): keep pair execution evidence behind unreadable string sources
* feat(inspect): accept any non-sampler latent origin
* fix(inspect): queue each node once in breadth-first ancestry walks
The latent and detailer image walks queued every link before checking
whether its node was already seen, and popped from the list head. A dense
shared fan-in grew the queue far past the visit budget before any visit
was charged. Both walks now use a deque and enqueue each node at most
once, charging every examined edge as scanned work; outcomes are
unchanged.
* fix(inspect): treat unlisted pipe and embed samplers as possible stages
The latent and detailer ancestry walks only flagged unlisted nodes with a
linked latent_image, guider, positive or negative input, so easy-use pipe
samplers, Impact pipe detailers and embed-driven video samplers were walked
through and the final sampler's prompt resolved silently. A separate
stage-shaped input set (adding pipes, embeds and custom-sampler parts)
now makes such nodes unsupported-node, named.
* fix(inspect): walk each sampler's latent ancestry once per resolution
Every selector branch run and candidate re-walked the same latent
ancestry, so conditioning-carrying origins such as WanImageToVideo turned
a three-selector graph from ambiguous into work-limit. The walk outcome is
now cached per sampler; branch runs of one candidate pay for it once and
other candidates are charged its visit cost, as the text memo does. Walks
that ran out of work are not cached. Every link input is still walked.
* fix(inspect): treat unlisted model-driven samplers as possible stages
Unlisted samplers that link a model with conditioning or a latent under
other input names (XlabsSampler, generic video samplers) were still walked
through, so the final sampler's prompt resolved silently. An unlisted node
with a linked model plus conditioning, neg_conditioning, samples or latent
is now unsupported-node, named; model patches such as LoraLoader and
ModelSampling nodes are not.
* feat(inspect): compute facts from the resolved sampler
* fix(inspect): merge facts across sampler branches when ambiguous
* fix(inspect): merge facts over unresolved stages and share stage inputs via the table
* feat(inspect): explain why a prompt could not be resolved
* feat(inspect): port legacy parser rules the new resolver missed (PromptManager)
R1: before 3.2.3 a save patch overwrote every PromptManager node's saved
text with the last-run node's prompt. The legacy viewer recognizes the
signature (comfy-metadata.js createContext: two or more PromptManager
workflow nodes with one API text and different widget values). Inspection
now refuses those nodes as malformed instead of resolving to another
node's prompt; the inspector hands the parsed workflow to the resolver for
this check only. Lands before the new pass-throughs (R6) that would
otherwise make these graphs resolve to the wrong text.
R7: this repository's node classes up to 07a5af3 (PromptManagerV2,
Positive, Negative, Detailer, V2Text, PositiveText, NegativeText) resolve
with their own outputs and their exact strip and ", " join.
The default (ingestion) profile ignores both.
* feat(inspect): port legacy parser rules the new resolver missed (A1111)
R3: an accumulating batch saver repeats the A1111 parameters record once
per batch item, identical except "Batch index: N". The bounded PNG reader
now keeps the last copy (Pillow's rule, which #165 relied on) instead of
calling the file unreadable. Any other difference, including a missing or
second batch index, is still a conflict, and decoding the copies draws from
the duplicate role budget.
R4: parameters text that starts with "Negative prompt:" has an empty
positive (#165), instead of showing the delimiter line as the positive.
R5: the settings footer may be followed by sd-dynamic-prompts' one-line
"Template:" and "Negative Template:" records, and may be "Steps: N" alone
when there is no full footer.
R10: commas inside A1111's quoted settings values no longer split them, so
an "ADetailer prompt" with a comma keeps the other facts known.
The facts' A1111 reader mirrors R4, R5 and R10; the default (ingestion)
profile keeps the historical split. Also from the Task 2.7 review: a guard
test fails on any uncoded unresolved reason without a user line (M1), and
the encoder bounds role lines to the client's 512 code units (M4).
* feat(inspect): port legacy parser rules the new resolver missed (installed nodes)
R6 and R9 for nodes whose sources are installed beside ComfyUI, each
entry's input names and output slots read from the node's code (the task
report lists file and line):
- detailer stages UltimateSDUpscale (image), UltimateSDUpscaleNoUpscale
(upscaled_image) and DetailerForEach (image), judged with their upstream
samplers like FaceDetailer;
- pass-throughs ControlNetInpaintingAliMamaApply and LTXVConditioning
(positive 0, negative 1) and FluxKontextMultiReferenceLatentMethod;
- encoder Text to Conditioning, strings TextInput and DisplayText;
- facts model pass-throughs Lora Loader (LoraManager), Power Lora Loader
(rgthree), ModelPatchTorchSettings, CFGNorm, PerturbedAttentionGuidance.
Impact Pack encodes a detailer's non-empty wildcard instead of its positive
input, so FaceDetailer and DetailerForEach now list it as a prompt
override: a literal wildcard is decided at run time and a linked one names
its source, instead of resolving to the positive input. The overwritten
PromptManager refusal (R1) still holds behind the new pass-throughs. The
default profile does not read the table.
* feat(inspect): port legacy parser rules the new resolver missed (upstream nodes)
R6 and R9 for nodes not installed beside ComfyUI here. Their input names
and output slots come from each pack's source at the commit the saved
workflows record in the node's cnr_id and ver properties (the task report
lists repository, commit, file and line):
- samplers ClownsharKSampler_Beta (RES4LYF), KSampler (Efficient) and
ttN KSampler_v2 (latent input "latent");
- pass-throughs SeedVarianceEnhancer and ApplyInstantIDAdvanced (positive
1, negative 2; output 0 is the model);
- selector Switch conditioning [Crystools];
- encoder smZ CLIPTextEncode: its text when with_SDXL is false; when true
the connected CLIP decides at run time which inputs are read, so the
prompt is known only when text, text_g and text_l agree;
- strings ttN text, CR Text, CR Prompt Text and TextBoxMira;
- facts model pass-through CR Apply LoRA Stack.
Inputs that replace what a sampler reads are not followed: RES4LYF options
(may carry a guider, and with sigmas replace the settings), an Efficiency
script and a ttN XY plot leave the prompt unsupported and the facts
unresolved. The overwritten PromptManager refusal (R1) still holds behind
SeedVarianceEnhancer. The default profile does not read the table.
* test(inspect): cover batch copies past the role budget and orphan widgets
* fix(inspect): match the batch index only in the settings footer
Review I-1: duplicate A1111 parameters copies were compared after blanking
"Batch index: N" anywhere in the record, so copies whose prompts differ
only by that phrase were accepted. The field is now blanked only inside
the record's one "Steps: N, ..." footer line, which must hold it exactly
once; any other difference stays a conflict.
Review M-4: the inspector parses the workflow record only when the API
graph can show the overwritten-PromptManager signature (a PromptManager
class, or a node with PromptManager's search_text input).
* fix(inspect): refuse overwritten PromptManager text inside subgraphs
Controller ruling on review M-1: PromptManager nodes inside subgraphs run
as "<instance>:<inner>" API IDs, nested instances chaining their IDs. The
overwritten-text check now walks subgraph instances (bounded in depth and
nodes) and reads a text widget stored by name, so these nodes are refused
like top-level ones, anchored or not. A definition without an instance
runs nothing and is ignored.
Review M-5: the historical adapters skip an explicit null prepend or append,
as the 07a5af3 code's "if prepend_text:" did.
* test(inspect): import unittest one way in the inspection suites
Code scanning flagged the mixed 'import unittest' and 'from unittest import
mock' forms. Use 'import unittest' plus 'import unittest.mock' everywhere.
* fix(history): name the recorded run's current-file prompts apart from its frozen fields
Now that the inspector resolves more saved files, the Recorded run dialog can
hold two regions named "Positive prompt" (and two of each Copy/Send action):
the frozen run field and the inline current file's prompt. Assistive
technology could not tell them apart, and the real-host chronology test read
two <pre> elements.
Prompt blocks take an optional context label. The inline current file inside
a recorded run names its regions "Current file positive prompt" and its
actions "Copy/Send current file positive prompt"; the visible headings stay
as in the concept. The real-host test now also scopes its frozen-field reads
to the "Frozen run prompt texts" section, and a disposable-server workflow
pins the distinct names with a resolved current file.
* fix(inspect): drop preview-only stages from unanchored candidates
A stage whose outputs reach only non-file outputs (PreviewImage and the other
non_file_outputs) cannot have written a saved file, yet it stayed an
unanchored candidate, so a preview-only FaceDetailer made the saved branch
ambiguous (f01b).
The inspection profile now drops such stages before detailer absorption,
using two linear backward passes over the graph (charged like the consumer
map; a failed pass keeps every stage). A stage without consumers, or one that
also reaches any other terminal, is kept, and nothing is dropped when every
stage is preview-only. Raw plus detailed double saves both write files and
stay ambiguous. A file in the host temp folder may itself be a preview, so it
is inspected with PREVIEW_INSPECTION_PROFILE, which keeps those stages.
Workspace facts follow the same choice: an absent or unresolved result that
names a sampler merges facts across that sampler's stages only, so the
dropped detailer's settings no longer blur the saved file's facts.
f01b now resolves (absent negative, guider has none); its vector is updated
and the expected failure removed. The default profile is unchanged:
differential against
|
||
|
Vito
|
cbf46a7573 |
feat(vocabulary): native Tags & categories screen (#245)
* feat(vocabulary): add workspace vocabulary state * feat(vocabulary): render the native Tags & categories screen * feat(vocabulary): add guarded vocabulary dialogs * fix(vocabulary): recover unconfirmed deletes and list every category in Manage * refactor(vocabulary): use the shared plural helper * test(vocabulary): compare the native screen with the approved concept * fix(vocabulary): wrap long tag names so row actions stay visible on narrow screens * docs(vocabulary): document the native vocabulary screen and gates Also aligns the shell link focus-ring offset with the concept (3px) and makes the chrome comparison poll the document title. * fix(vocabulary): refresh lists when a write lands after its dialog closes A rename, delete, create or category apply that commits after its dialog was disposed (Chromium force-closing the modal on a repeated Escape, or Back while the write is in flight) returned before onInvalidate, so the lists were never re-read and a cached Library shelf kept showing the old names. Like the recipe editor, a disposed command now still reports any outcome that may have changed storage (receipt, conflict, restore, unconfirmed) without rendering; only a failed write stays silent. * fix(vocabulary): give the Manage categories datalist its own id The proposed-name datalist reused the recipe editor's `categories` id. It was safe only because routes dispose each other's dialogs; it is now `vocabulary-category-options`. * docs(vocabulary): correct vocabulary testing notes Name the Tag column and the Create tag heading action, say that a failed Load more moves focus to Retry, place bulk actions on the admin page, describe late writes after a closed dialog, Browse's exact filter and the full deviation list, and use a test port that is not the live concept server's. * test(vocabulary): raise the vocabulary dialog branch floor coverage:update after the late-write and datalist fixes: workspace/vocabulary-dialog.mjs branches 87 -> 88. No floor is lowered. * fix(vocabulary): keep 390px row actions visible regardless of font metrics The one-column Tags table reserved a fixed 6em for the tag name. With CI's stock DejaVu Sans fallback (about 9% wider than the pinned image's) Tag + Prompts + Images + stacked actions needed 351px of a 336px wrap, so Browse and Rename / merge were pushed out of view on every row. At 700px and below the Tag column now has no fixed minimum and the actions ask for no more than their stacked width, so all remaining space goes to the tag name: short names stay on one line wherever the concept's own table fits, and a wider font wraps the name instead of clipping an action. A new test widens the metrics with letter spacing and fails without this change in every engine. * ci: run browser jobs in the pinned Playwright image Concept comparisons measure geometry, which follows the fonts a browser falls back to. The concept names Inter, which neither environment ships: the pinned image falls back to WenQuanYi Zen Hei, a stock ubuntu-latest runner to DejaVu Sans. Under DejaVu the concept's own Tags table needs more than a 390px viewport (its page scrolls sideways and, under touch emulation, the widened layout viewport resizes its dialogs), so the 390px comparisons failed in CI while passing locally. Both browser jobs now run in mcr.microsoft.com/playwright:v1.64.1-noble, the image local runs use, instead of installing browsers on the runner. A tooling test keeps the tag equal to @playwright/test in the lockfile. * fix(vocabulary): ignore a category preview edited while it is pending Editing the Category or proposed name while a slow Preview was in flight was a no-op in the saving phase, so the late response still became an applicable preview: the form showed the new values while Apply would submit the old source and target. An edit during a pending preview now marks it stale. Its late result returns the dialog to idle (Preview again) instead of showing an impact or offering Apply; a plain failure of a stale preview is not shown either, while a restore or conflict still is, since it describes storage. The inputs stay editable during the request, as in the recipe editor, which disables only its actions while saving. * ci: use the image's Python and Node in containerized browser jobs Inside mcr.microsoft.com/playwright:v1.64.1-noble, actions/setup-python failed before any test ran: `spawn /__t/Python/3.12.15/x64/bin/pip ENOENT`. The toolcache builds come from the runner, and their entry points name host interpreter paths that do not exist in the container. The jobs now use the image's Python 3.12 and Node 24. The image has no ensurepip and the job runs as uid 1001 without root, so a checksum-pinned pip wheel bootstraps a user-space venv for requirements-dev.txt; a step checks that the image Node is 24. The tooling test forbids setup-python and setup-node in these jobs and requires the pinned bootstrap. |
||
|
Vito
|
1ba9e22824 |
fix(database): stop SQLite write-lock starvation under concurrent saves (#246)
* test(database): require identical WAL/NORMAL settings on managed connections (#242) Concurrent writers failed with 'database is locked' on Windows runners despite the 30 s busy timeout: every WAL commit flushed to disk while it held the write lock, so polling waiters starved. These tests pin the shared per-connection settings, including synchronous=NORMAL in WAL mode. * fix(database): stop WAL commits from holding the write lock across a flush (#242) Every managed connection now goes through configure_connection: WAL, foreign keys, the 30 s busy timeout and, when WAL is active, synchronous=NORMAL. Under FULL each commit flushed to disk while it held SQLite's only write lock; on slow Windows disks a burst of writers (a scan beside ComfyUI saves, or the concurrency tests) starved a polling waiter past the busy timeout and it failed with 'database is locked'. With a 60 ms emulated flush on Linux, 8 threads x 50 save+use calls went from 49.6 s with a 32.1 s worst call (and 'database is locked' in the unit test) to 0.7 s with a 0.4 s worst call. NORMAL in WAL mode keeps the database consistent after an application crash; only an OS crash or power loss can drop the most recent commits. In-memory or non-WAL databases keep SQLite's FULL default. * test(history): hold the first concurrent writer open past the bounded wait (#242) The concurrent replay tests expected every peer to succeed, but each peer only waits 250 ms for BEGIN IMMEDIATE by design. On a slow Windows runner three queued commits exceeded that and a peer raised 'database is locked'. Holding the first writer's transaction open for 500 ms makes the same failure deterministic on any host (20/20 locally). * test(history): let concurrent replay peers queue like ordinary writers (#242) History writes deliberately give up after a bounded 250 ms wait so they never stall ComfyUI; the existing bounded-timeout test still proves that. The concurrent replay tests assert that every peer succeeds, which is a serialization and first-capture property, so they now raise the budget to the shared 30 s busy timeout while the first writer is held open. The budget is one named constant, EXECUTION_BUSY_TIMEOUT_MS, shared by the transaction helper and the background writer. Before: 20/20 'database is locked' with the held writer. After: 0/20. * test(history): wait on restore handoff events, not a 5 s clock (#242) The retired-owner projection tests park a history read at a barrier while the test performs a full multipart restore, then release it. The barrier gave up after 5 s and raised inside the route handler, so on a slow Windows runner (a 694 s suite run) aiohttp answered with its text/plain 500 and the test reported 'history route missing'. Every wait already ends on an event; the ceiling now only stops a broken test from hanging (60 s). With restore slowed by 5.5 s locally, all 8 subtests failed with text/plain before and pass after. * docs(testing): add the concurrent SQLite test rule to the Windows preflight (#242) |