d2ea705a44a2164fffb503cff0192a734d9afa90
81
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
d2ea705a44 |
fix(tagging): Tag output images tags each image, never its prompt (#379, PR 1) (#380)
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1) The native "Tag output images" job now runs the image-tags loop over every still image the output folders list, linked or not: each image gets its own model tags (at most "Suggestions per image") in its image tag set, and its bytes are marked auto-tagged in a new media_image_autotags table. No prompt tag or prompt text is written. - database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional in old backups, refused without the annotation group. - add_image_tags/replace_image_tags take mark_autotagged; autotag_state and autotagged_locators read the marks through the one current-record rule. - "Skip already tagged" skips images whose current bytes are marked, counted in a stat-only pre-pass; hand tags do not count. - The review counts images and estimates the time (WD14 on CPU or CUDA). - System > Auto-tagging review rows and copy, D134 and the Auto-tagging contract follow. Part of #379 * fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379) The job read as ended while it still held the model gate, so a quick restart could be refused with model-busy. Pre-pass chunks now publish progress, and a cancel between chunks keeps exact counts. Part of #379 * fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379) The live check measured WD14 on CUDA at 0.167 s per image (100 images in 16.7 s with the model load), so the review estimate uses 0.17 s. A finished or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X min left"; it now ends with "<verb> <state> · N files processed". Part of #379 * test(browser): reload the admin page only after the in-place edit saved The gallery-to-admin workflow pressed Enter and reloaded as soon as the edited text showed; the page saves after showing it, so a reload that won the race abandoned the PUT (WebKit, run 40564511000). The test now waits for the save's answer. Part of #379 * test(tagging): the batch coverage row names the renamed #379 test |
||
|
Vito
|
52a00370ea |
test(system): prove Diagnostics & logs against the concept and every legacy System capability's owner (#329, PR 3) (#387)
* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * test(system): compare Diagnostics & logs with the concept System › Diagnostics & logs and its log cleanup review are compared with the concept (app.js 694–695 and maintenancePlan("rotated logs")) in both themes at 1600x1100 and 390x844, with D109 and D110 (and D100 for the review) as the only mapped differences, a 320-wide select check and a mutation check. The fixture gains a concept seed mode (the concept's four log lines, a 1.4 MB active log with three rotated copies, console logging off), and the plan dialog's measureReview and mapPlanConcept move unchanged into concept-inventory.mjs for Maintenance and this review. The comparison found that the Database line read a report key that does not exist (database_info.size_bytes) and so always showed "0 bytes"; it now reads database_size_bytes, the key get_database_info() returns. Part of #329 * test(system): prove every legacy System capability has an owner TESTING.md gains "Legacy System capability coverage": every System capability of the legacy admin and gallery pages with its anchor on main, owner, status (covered, kept in legacy or approved removal), native control and the exact title of the test that proves it. tests/vendor/legacy-coverage.test.mjs (with tools/vendor/legacy-coverage.mjs) fails on a row with no owner, an unknown status, a malformed native control, a control on a row that is not covered, or a test title its file does not declare; a title only in a comment does not count. workspace-system.spec's "every covered legacy System capability has its native control" opens each covered row's section and finds the control by role and accessible name. Versus the design's table: the anchors are re-checked against main, "Run health checks" is on Maintenance (D123), not Overview, Diagnostics run is proven by "Run checks opens the shared Health report", and the export rows repeat their test title instead of "(same as above)". Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 * test(system): require a running test and a control for every covered capability Review fixes for the legacy coverage proof: - A title declared only with test.skip, .fixme or .fail no longer proves a row (.only and .slow still do). - Every covered row names its native control, except the rows in CONTROL_EXEMPTIONS, each with its reason (today only the gallery per-image auto-tag row, owned by the Images selection bar). The browser test asserts that the covered rows without a control are exactly the exemptions. - JavaScript comments are stripped across the whole source, wherever a block comment starts, skipping strings, template literals and regular expressions. A Python def test_ counts only at function or method indentation and outside comments and docstrings. - A control name several rows share ("Review plan", "Review options", "Refresh") is looked up inside the row the table names (· in "<row>"). - The diagnostics comparison records each select's chosen option, so a wrong default level is caught. Part of #329 |
||
|
Vito
|
bdc38422a9 |
feat(system): Diagnostics & logs natively, reviewed log cleanup and Clear all logs (#329, PR 2) (#385)
* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 |
||
|
Vito
|
a0fc5171e3 |
fix(inspector): pass text through unknown nodes instead of refusing them (#381) (#383)
* fix(inspector): pass text through unknown nodes instead of refusing them (#381) The inspector refused any node class missing from the semantics table, so a PromptManagerPair behind an rgthree Any Switch showed "Prompt uses a node the inspector does not support". The inspection resolver (Python and browser) now reads through unknown nodes: a text input is the text, positive/negative route by output slot, a conditioning input is followed alone, otherwise every linked input except core type inputs (pass_through.skip_inputs). PromptManager nodes are authoritative; several different texts become one candidate each ("Several prompts reach the sampler"). A seeded text node with no conditioning input is generated at run time, and a recorded PromptManager execution is not checked against text that passed through an unknown node. INSPECTION_RESOLVER_VERSION goes to 2; ingestion keeps the default profile. A legacy parity gate records every committed graph whose inspection positive matches the legacy parser's; no graph may leave it. * fix(inspector): keep recorded pair text, string-only seed rule, role inputs and unique candidates (#381) Review of the pass-through: a walk through an unknown node never overrides a PromptManagerPair's execution record (any outcome but work-limit is unverifiable while the record is checked); the run-time seed rule reads only string outputs; a node with positive/negative inputs follows only the role of the requested slot, nothing when it is unlinked; one text offered twice is one candidate. * fix(inspector): anchor on the saver that wrote the file and resolve stages on shared links (#381) An inspected file's name now anchors the inspection on the one saver whose filename_prefix (literal or traced) makes the file's stem <prefix>_<digits>[_]; the samplers in that saver's image ancestry are the candidates. No match, several matches or an unreadable walk anchor nothing, so the file stays "several possible prompts". In a latent walk, an earlier sampler reading the same conditioning links as the later one is the same prompt, not an unattributed stage. Python and the browser resolver behave the same; the legacy parity gate states each graph that differs from legacy on purpose. * fix(inspector): skip only plain first stages on shared links; an unfinished prefix scan anchors nothing (#381) Review of the saver anchor: a same-links earlier stage is skipped only when it is not a detailer, sets no prompt override and continues no other sampler, so an earlier prompt or override behind it is never hidden. A work limit while tracing a saver prefix ends the match scan without an anchor instead of counting that saver as no match. * test(inspector): cover every plain-first-stage outcome on both resolvers (#381) The frontend coverage gate found local-resolver-stages.mjs branches at 92.44% (< 93). Each way an earlier stage stops being a plain first stage (no latent, literal, missing or cyclic latent, a sampler-shaped node, a detailer) is now a resolver test and an anchor-first-stage-* parity vector. |
||
|
Vito
|
d868745194 |
fix(lora): cache LoraManager's layout and close the #355 follow-ups (#362) (#364)
* feat(lora): cache LoraManager's layout behind file signatures (#362, phase 1) A new py/lora_layout_cache.py keeps one (install, layout) entry and reuses it while the configured path, the plugin settings file's and the settings file's signatures (st_mtime_ns, st_size), the install and every media root's real path are unchanged and the entry is younger than 30 s. A load whose watched file changed within 3 s (git's racy rule), could not be checked, or whose settings were unreadable serves its request only. clear() bumps a generation so a load that started earlier stores nothing. Hits take no lock. Not wired yet; phase 2 puts it behind the image hot paths. * fix(lora): treat an offline Windows share as unvalidated and make cache tests portable (#362) Review findings on phase 1: Windows raises FileNotFoundError with winerror 53/67 for an offline share, which is a temporary failure, not a missing file. Test times are whole milliseconds (NTFS keeps 100 ns steps) and a file system that rounds them skips; cleanup no longer ignores errors; tests added for ValueError, NotADirectoryError, a failed reload, resolve failures and the watched-file set. * perf(lora): serve the image hot paths from the layout cache (#362, phase 2) _lora_image_dirs takes (install, layout) from lora_utils.cached_lora_manager_layout instead of resolving the install and reading LoraManager's settings files on every Library, inspect, reveal and serve request. serve_image builds the allow-list in the executor instead of on the event loop. A finished import (System job or legacy scan, both through replace_lora_imports) clears the cache, so files linked under a folder LoraManager just created are served at once. The integration-off check and the configured-path rule still run before the cache. * fix(lora): fingerprint every scanned sidecar, kept or skipped (#362, phase 3) Discovery keeps every metadata file it saw (scanned_files, resolved and sorted; scanned is now its length). The fingerprint hashes each one's path, size, modification time and whether it was kept, so a skipped sidecar added, removed or edited between the review and Start refuses Start with plan-changed, as the review's counts promised. * fix(lora): warn about refused default LoRA folders and log what failed (#362, phase 4) ComfyUI's default LoRA folders already pass the folder guard; a refused one that exists is now listed in the layout's refused_defaults, and the layout cache warns once per newly refused folder. The cache writes one INFO line when a stored layout differs from the last one logged. BAD_SIDECAR_ROOT and the module docstring name a folder that contains the home folder. A relative preview_url writes a DEBUG line before the suffix search; a non-OSError failure while loading trigger words logs its traceback. Review follow-ups harden two serve tests. * fix(lora): skip a sidecar that is not UTF-8 instead of failing the load (#362) Review follow-up on phase 4: read_lora_metadata caught JSONDecodeError but not UnicodeDecodeError, so one non-UTF-8 sidecar failed the whole trigger-word load (now logged with a traceback every retry) and the import review. It is skipped like any unreadable sidecar. The relative preview_url DEBUG line uses lazy formatting. * docs(testing): the LoRA linked-only gate, layout cache and fingerprint scope (#362, phase 5) TESTING.md's LoRA reference import contract now states what #358 and #362 shipped: the linked-only gate and _media_roots, linked_only media, the cover link order, the guard on ComfyUI's default LoRA folders and refused_defaults, the folder-unreadable code, the trigger cache's catch and 60 s retry, the fingerprint over every scanned sidecar, and the layout cache's key, signatures, racy window, maximum age, root re-check and clears. The targeted command includes tests.test_lora_layout_cache and tests.test_api_images. * fix(lora): a cached layout needs the same home folder; give the rebuild poll its time (#362) Final review follow-ups: the folder guard compares against the home folder, so a cache hit now also needs Path.home() unchanged. TESTING.md says the configured-path rule runs on the cache's result and that a non-UTF-8 sidecar is skipped. CI: 'Rebuild all is explicit...' (webkit) polled the job for the default 5 s while it regenerated 24 previews; the failure screenshot shows 12 of 24 done at the deadline on the busy runner (0.17 s locally). The poll now allows 30 s for that real work. |
||
|
Vito
|
1bb3678f32 |
fix(threads): describe the teardown as built, warn only for a live observer, and keep an interrupted teardown's result (#377) (#378)
- _stop_background_work pre-fills the started parts with False and stores the result and removes its atexit entry in a finally:, so a Ctrl+C between steps no longer leaves the result unset and the callback registered. - The "watchdog observer still running" warning moves into _stop_observer: only for an observer still alive after its join, naming the join's own timeout (not after observer.stop() raised, and not the full deadline). - Tests: the plain-teardown tests share one package; two teardown tests run with a live preview worker held in a fake encode (ffmpeg killed, step gets min(CLOSE_WAIT_SECONDS, time left), thumbnails False while held and True when the kill releases it); an interrupt-between-steps test; two warning tests. - TESTING.md and the ledger fixture comment: the refusal comes first, the image worker and the observer share one 5 s deadline, and what an interrupt between steps does. Closes #377 |
||
|
Vito
|
2f67b98287 |
fix(tagging): save WD14 confidences only on the 0.05 grid every page shows (#369) (#373)
* fix(tagging): save WD14 confidences only on the 0.05 grid every page shows (#369) A saved threshold off the 0.05 step (for example 0.37) blocked Generate and Start in the workspace Tag image dialogs (native step mismatch) and made the legacy sliders send 0.35. The server now refuses such a save with a message that names the setting, a stored one falls back on load, and an accepted one is stored exactly on the grid. System > Auto-tagging refuses the same values before Preview tagging review or the batch Review, and hands thresholds on snapped to the grid. Per-request thresholds keep their own checks. * test(browser): restore the safety backup policy on system reset The Recovery policy test in workspace-system-backups saves safety_backup false at runtime, and /__test__/system/reset restored config.json but not PromptManagerConfig.SAFETY_BACKUP. A later spec on the same server (Tag output images review) then saw "No safety backup (the policy is off)". The new #369 test shifted the CI shards so the two met on one worker. |
||
|
Vito
|
f3b9f61f11 |
fix(threads): stop background work at exit, on cleanup and on the tests' unload (#301) (#374)
* fix(threads): stop background work at exit, on cleanup and on the tests' unload (#301) - __init__.py: _stop_background_work() detaches the monitor's callbacks, stops the monitor, closes the preview worker and shuts the prompt cleanup thread down against one 10 s deadline, once. It runs from atexit (ComfyUI never runs aiohttp's cleanup hooks), from on_cleanup where a host runs it, and from the tests' package unload; it removes its own atexit callback. Registered after the history hook, so at exit it runs before the history dispose. - AutoThumbnailer.close/close_auto_thumbnails take a timeout and return whether the worker ended. - Tests: teardown result, once-only, on_cleanup, atexit order, a failing step or a second Ctrl+C never skipping the tracker, one shared deadline, three repeated loads leaving no thread, and a child process that exits normally with the image worker alive. * fix(threads): refuse new previews before the monitor stops; isolate the exit child (#301) Review findings: - The teardown makes the API refuse new previews before it detaches the monitor's callbacks: an image in hand that already read the preview queue no longer starts a preview worker during exit (Python 3.12+ refuses that thread and logs a traceback). New refuse_auto_thumbnails(). - The teardown imports its bounds inside each step and falls back to the standard logger, so a failing import never skips the other steps. - The exit child keeps its library and log in its own folder (PROMPT_MANAGER_DATA_DIR), reports watchdog's threads too, fails with the child's output on a hang, and rejects "Error while stopping". - TESTING.md: "Background threads at exit". |
||
|
Vito
|
ae755d2ce0 |
feat(system): Auto-tagging screen, native in System (#328, PR 3) (#368)
* feat(system): configure auto-tagging natively * test(system): compare Auto-tagging with the concept against the real backend |
||
|
Vito
|
fd949b870c |
test(admin): wait for the restore response, not a timer (#359) (#363)
The legacy restore specs "restore validation and cancellation preserve live data and allow a valid retry" and "backup and restore recover from interrupted network requests without losing data" ended with a 5 s toBeHidden on the restore modal. The modal closes only after the server answers, so that was a 5 s deadline on a disk-bound restore, which WebKit CI runners sometimes exceed. Both specs, and the downloaded-backup spec that already waited on the response, now share confirmRestoreAndWait: it waits for the POST /prompt_manager/restore response, asserts a 200 (showing the body on failure) and the success body, then checks the modal closes. TESTING.md explains how to reproduce disk-bound timeouts with --device-write-iops in the pinned image. |
||
|
Vito
|
80c6ec5340 |
fix(lora): find LoraManager's sidecars, previews and example images in every storage configuration (#355) (#358)
* fix(lora): read LoraManager's settings and storage layout the way it does (#355) Add py/lora_manager_layout.py: the settings file in LoraManager's lookup order (environment folder, portable plugin file, user config folder), only the seven keys the import needs, the LoRA roots, the centralized sidecar root, the example-images root and the active library, each folder through one guard. Sidecar discovery for both storage modes and the local preview and example-image lookup live there too. The test runner and the browser test server point LoraManager's settings lookup at a disposable folder, so no test reads the real settings. * fix(lora): import the sidecars and local images LoraManager really uses (#355) The import, the legacy scan, the trigger-word cache and the image server now share LoraManager's layout: sidecars beside the models (alongside) or under the sidecar root (centralized), kept only when their model exists inside a LoRA root. Each LoRA links its LoraManager preview and example images where they are; CivitAI downloads run only for a LoRA without any local image. The review adds skipped, with_local_media and the layout, and offers no token when LoraManager's settings cannot be used; the legacy scan reports the same reason without replacing anything. find_lora_directories becomes default_lora_roots (no scan of the LoraManager install); the preview and example-folder helpers it replaced are removed. The trigger cache never raises into a running node. * fix(lora): never serve a LoraManager folder that holds the home folder (#355) The folder guard also rejects ancestors of the home folder (/home, C:\Users), compared in the realpath + normcase form, so a LoraManager setting cannot widen the image server's allow-list to every user's files. * fix(lora): show linked LoraManager videos as videos (#355) _enrich_images marks every image without is_video by its extension, so an .mp4 preview outside the output folders shows the Video preview label and plays in the inspector. Tests cover serving the sidecar and example-image folders only while the integration is on, and a filesystem root as the sidecar folder adding nothing. * feat(lora): the import review names the LoraManager settings it read (#355) The review gains a first row, LoraManager ("<source> settings · <mode> storage · library <name>", the settings file as its hover title), and its Found row counts metadata files, LoRAs and LoRAs with local images. A plan whose LoraManager settings cannot be used takes the close-only path: "LoraManager settings: not usable: <reason>. Nothing was changed." The browser fixture writes each LoRA's model file and can leave out the local preview, so the download, gate and cancel specs still exercise CivitAI. * docs(lora): describe how the import finds LoraManager's files (#355) * test(lora): set the settings folder relative to the process folder (#355) os.path.relpath fails across Windows drives (the temp folder on C:, the checkout on D:); the test now changes into its temp folder instead. * fix(lora): review findings for the LoraManager layout (#355) - The example-images library subfolder goes through the folder guard like its root, so its canonical (normcased) form matches the media check on Windows for a library name with capitals. - A failed trigger-word load is retried by a running node only after TRIGGER_RETRY_SECONDS, not on every prompt run. - A review whose discovery cannot list a folder says so (folder-unreadable, the folder's name only) instead of a generic error. * test(images): hold the listing clock in the deferred-walk test test_a_deferred_walk_waits_at_least_half_a_second set the previous walk 29.99 s back against a 30 s interval and then read the real clock again: a runner slower than 10 ms between the two lines passed the interval and the walk started at once (failed on macOS 3.13 and Windows 3.11). The module's clock is now held for the mark, so the deferred walk is always the one measured. * test(lora): write the unkept-settings fixture as raw text CodeQL read the dict with a credential-named key flowing into write_text as clear-text storage of a secret. The value is a marker, so the fixture now writes the JSON text directly and names the value MARKER. * fix(lora): link each LoRA's cover as its newest image (#355) Workspace lists show a prompt's media newest link first, so the import links a record's images last to first: the card shows the cover (a still in front of a leading video) the import put first. * fix(lora): show LoRA-linked media on the Library page and in the inspector (#355) Library previews, the positive detail and the per-prompt images page list a prompt's linked media under the output folders plus, while the LoRA integration is on, LoraManager's guarded folders and the download cache (_media_roots; output root indices keep their meaning). That media keeps its rooted identity, so Info and the media viewer open it; the path-based serve route answers for those roots only with a file the database records (either path spelling), never serves or makes a preview there, and output requests never read LoraManager's layout. Such media is marked linked_only: no preview file, no favorite star, no Tag action. Show in folder accepts those roots; the download cache is named, not created. * fix(lora): expose only linked files under LoraManager's folders (#355) Serve, inspect and Show in folder share one gate for a LoRA-linked root (_media_roots past the output folders): a file the database does not record as a linked image is refused with 403, checked by its given and its real path. Before, inspect and reveal read any file under those folders by path. * fix(lora): check the file Show in folder would open under a LoRA root (#355) reveal_image resolved the requested path through the client-path resolver, which can fall back to a path anchor; under a LoRA-linked root the file that would open now passes the same recorded-media gate, so only a linked file is ever revealed there. |
||
|
Vito
|
c7ba481d3c |
perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) (#353)
* perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) The image monitor now reports every settled output (linked or not, also when linking raised) with its resolved path through set_output_listener. The API's notify_output_settled hops from the monitor's worker thread onto the server's loop (call_soon_threadsafe) and does nothing before the loop is known or after cleanup. mark_root_dirty matches the path against each walk's resolved_root (normcase(realpath), computed in the executor by walk_root, now its fourth value) and marks the root dirty. A dirty root gets one background walk, no sooner than rewalk_interval = min(15, max(5, 5 x last walk's duration)) after the previous walk started (last_started_at / last_duration are recorded on every walk). _schedule_walk is shared with PR 1's delayed stale refresh: one deferred walk per root, never armed while a walk runs (the walk's end re-checks dirty: the trailing walk), at least 0.5 s away, the earlier deadline wins, and every walk start cancels it. A failed walk clears dirty. Rescan semantics are unchanged. * fix(images): keep removed roots and cancelled walks from walking again (#348, PR 2) Review findings on the monitor-triggered walks: - reset_gallery_roots_cache now retires every listing record: its pending timer is cancelled, dirty is cleared, and no mark, timer or trailing walk starts a walk of it until a reader or the warm-up asks for that root again. Before, a timer or trailing walk pending when the roots changed stored a listing for the removed root, and the monitor (which watches the old roots until restart) kept it walking on every new output. - A walk task cancelled outside cleanup_gallery_listings no longer starts a trailing walk while the loop shuts down. - root_contains ignores a Windows extended-length prefix (\\?\ and \\?\UNC\) on either side, which realpath can keep on a file past MAX_PATH. - The REWALK comment states the ~20% GIL share only for walks up to 3 s. * test(images): prove a cancelled walk starts no trailing walk (#348, PR 2) The previous assertions held even with the trailing walk, because that walk finished before they ran; assert the walk number and the scan count. * perf(images): add a settled output to the listing without a walk (#348, PR 2) Live test: each landing in a 20,000-file folder cost a full walk (~0.4 s of GIL) plus a full workspace rebuild, so Images reads during a run took up to 2.1 s. The monitor's report now patches the stored listing instead. - output_settled (the monitor hop's target) slices the file's part below the root in its own case (relative_part; Windows compares folded), and _run_adds places one root's pending files in one executor batch: _settled_entry applies the walk's filters (suffix, thumbnails, depth, symlink by lstat; macOS on-disk spelling), insert_entry puts the (Path, mtime) entry where the newest-first sort does, add_folder copies the folder sets. Stored only when no walk started and no drop happened meanwhile; walked_at is untouched, so the 30 s refresh still walks. - Anything inexact marks the root dirty and walks as before: a running walk, an unsliceable path, a failed lstat, a tie in mtime, the cap, an unexpected error. Rescan, the refresh, the warm-up and invalidation stay full walks; a walk start forgets the pending adds. - The workspace caches follow a patched listing by entry identity (listing_delta): only added values are built, the newest order places them by bisection, the favorites lookup is copied and extended. Synthetic 20,000 files: per landing 252 ms (walk + full build) -> 18 ms (3 ms add + 15 ms warm build). * fix(images): review fixes for adds without a walk (#348, PR 2) - _newest_step steps the cached newest order only when the per-file cache still holds exactly the values this call built. A stale page read racing a newer warm build (both run in executor threads) stored an order labelled with older values than it held, and the next read inserted the new file a second time. It also sorts in full when a roots reset dropped the per-file values meanwhile. - The stored spelling is checked on Windows too (nt._getfinalpathname: realpath keeps the given case when it cannot open the file), and before the name filters, which compare the walk's stored spelling. macOS opens non-blocking, so a FIFO named like an image never holds the executor. - An output in a folder the walk cannot list (no read permission) is not added, as the walk skips it. - Windows-portable unit tests: POSIX paths in the insert tests, and a str.lower fold for the length guard (Windows' normcase keeps lengths). * test(images): reproduce the newest-order race and fix a Windows clock flake (#348, PR 2) - The race test now has the newer build step from the values the cached order was built from, which is the interleaving that inserted a file twice; it fails without the parts check in _newest_step. - test_a_failed_walk_records_its_timing_and_clears_dirty compared two monotonic readings that Windows' ~15.6 ms clock can make equal; the previous start is moved back instead. - TESTING.md: os.access ignores ACLs on Windows. |
||
|
Vito
|
9ac471eb72 |
perf(images): share and warm the Images listing (#348, PR 1) (#349)
* perf(images): share and warm one listing per output root (#348) The Images page, its sidebar counts, the subfolder list and the legacy gallery each rebuilt the output listing when the 30 s cache expired, all at once, and the subfolder list computed relative paths on the event loop. Thumbnail writes also threw the listing away. Keep one RootListing per output root (py/api/gallery_listing.py): - one walk per root at a time; concurrent readers wait for it (shielded) - a listing older than 30 s is served at once and refreshed in the background - the walk, the comparison with the previous listing and both folder sets run in the executor; an unchanged walk keeps the same list object - os.scandir probe: a missing root fails the walk and keeps the previous listing (one warning per failure streak) - Rescan waits for a walk started after the click and is a 500 when it fails, even with a previous listing - deletes keep the blocking invalidation; a generation counter that survives drops discards walks that raced it - thumbnail jobs, conversion and the legacy generate/clear routes no longer drop the listing - warm-up on the host's on_startup, cancelled on on_cleanup Adds tools/vendor/perf/images_load_probe.mjs (maintainer tool, not CI) and the TESTING.md section "Images listing cache (#348)". * fix(images): drop listings on the loop when invalidated from a worker thread (#348) A settings save (gallery roots change) and the browser-test seeds call invalidate_gallery_cache() from the executor. Listing records are only touched on the event loop, so an off-loop call now hands the drop to the loop and waits for it, keeping the call synchronous for its callers. The shared-Rescan test waits for the later Rescans to take their walk number instead of sleeping. * test(images): bound the shared-listing test waits (#348) A macOS py3.12 run stopped inside the shared-listing tests and sat until the 6 h job limit with no traceback. It does not reproduce locally (about 300 runs on 3.12 and 3.14, pinned to one core). Requests and background settles in TestSharedListing now give up after 30 s, so a recurrence fails with the stuck await instead of hanging the run. * test(images): let the settle loops yield so a finished walk leaves the set (#348) On CI (ubuntu and macOS py3.12) the runner's watchdog dumped test_a_root_that_recovers_warns_again_on_its_next_failure spinning in _settle: asyncio.gather over tasks that are already done returns without yielding, so the set's discard callback, queued when the walk finished, never ran and the loop never ended (wait_for could not fire either). Delaying discard by one loop turn reproduces it locally; both settle loops now yield one turn after each gather. * test(browser): poll the import review's rows instead of reading them once gridOf reads the dialog's rows once; right after the click the review may not have rendered them yet (firefox CI: received undefined). * perf(images): walk the output with scandir entries and build folders from strings (#348) The live test showed a read during a background refresh taking 2.2 s instead of 0.07 s: the walk held the GIL for about 2 s in the server. py-spy split that walk into two halves: Path.relative_to per file in folder_sets, and Path.stat + Path.is_symlink + Path joins per file in the scan. - _walk_media visits folders in os.walk's order with os.scandir entries: the file type comes from the listing (no lstat per file) and the mtime from entry.stat (one call, none on Windows). _iter_media_files is now a thin wrapper, so its other caller is unchanged. - folder_sets slices the path strings (in the path flavour's separator) instead of calling relative_to, falling back to it for a path that does not start with the root string. On the maintainer's 19,935-file output folder: scan 0.386 s -> 0.112 s, folder sets 0.452 s -> 0.007 s, both results identical to the previous code. A reference copy of the os.walk version checks order, filters, the depth cap, symlinks and the file cap. * perf(images): build the workspace caches behind the warm-up and changed walks (#348) The live test after the faster walk still took 2.9-5.2 s for the first Images open after a restart. py-spy showed the walk was now small; the time went to the workspace's own caches, cold after a restart and built by three parallel first requests at once: the favorites listing (Path.relative_to per file), the per-file values and newest order, and the prompt-link read. - Once the host's startup warm-up has run, a stored walk that changed a listing schedules one executor job that builds what a first page reads, through the request path itself (_select_workspace_page with the default filters, then _workspace_favorite_paths). A walk stored during a build asks for one more build; an unchanged walk (same list kept) builds nothing; a failed build only warns; on_cleanup stops it. Servers that never ran the warm-up (the test servers) start no build. - _favorite_listing slices the path strings as _listed does, falling back to relative_to for a path outside the root string. * perf(images): start a stale listing's refresh walk a second after the read (#348) After the faster walk, a read that found the listing stale still took 0.5-1.0 s against 0.07 s: it was served from the old listing but ran beside the refresh walk it had just started, sharing the GIL. The refresh now starts REFRESH_DELAY (1 s) after that read through a per-root rewalk_handle, so the page's own requests finish first. One pending refresh per root and none while a walk runs; any walk start (a cold read, Rescan, the warm-up) cancels a pending one; on_cleanup cancels them before its first await. * test(browser): accept KB or MB for the picked safety backup's size The restore preview showed '2.5 MB' for the test server's database copy in a CI coverage shard: its size depends on what earlier tests on that server wrote, and the formatter switches to MB at 1 MiB. * fix(images): keep the 3.10-3.13 suffix rule in the scandir walk and harden the warm-up (#348) Review of the walk speedup: - os.path.splitext drops names like ..png that Path.suffix listed on Python 3.10-3.13 (3.14 changed Path.suffix, which hid it from the parity test on a 3.14 venv). _name_suffix spells out the 3.10-3.13 rule; the test now asserts ..png and ...webp are listed and g. is not. - _warm_workspace reads the roots inside its try, so a failing _get_all_output_dirs warns instead of leaving an unretrieved task error. - The folder_sets fallback test now takes the fallback (a Windows case mismatch); the coalescing test pins the walk count. |
||
|
Vito
|
5772de8d27 |
ci: bound every job and make hangs name themselves (#350) (#351)
* test: run the suite through a watchdog that names a hang (#350) tests/run_suite.py discovers the same tests as unittest discover -s tests, with the same sys.path and exit codes (5 when none ran), writes 'start: <test id>' to stderr before each test and arms faulthandler.dump_traceback_later: 120 s per test (PM_TEST_HANG_LIMIT) and 120 s after the run (PM_TEST_SHUTDOWN_LIMIT), never cancelled. A hang in a test, its teardown, its event-loop shutdown or interpreter exit prints every thread's stack and exits non-zero instead of running to the job limit. The pre-push hook now runs it too. * ci: bound every job and run the Python suite through the watchdog (#350) Every job in test.yml and code-quality.yml gets timeout-minutes = ceil(1.5 x slowest of three green main runs) + 3, min 5, with the runs and times written above jobs:. The matrix and the coverage gate run python -m tests.run_suite. ci-workflow.test.mjs checks every job in the three CI workflows has a limit and that the matrix and coverage use the runner. Job names are unchanged. * docs: document the test runner, time limits and following a push (#350) * fix(test): keep unittest's warning filters and tighten the runner's edges (#350) Review follow-ups: the runner now shows warnings as unittest.main does (DeprecationWarning and ResourceWarning were hidden), rejects non-ASCII digit limits with the one-line exit-2 error, and starts each 'start:' line on its own line at default verbosity. The docs now say that class and module fixtures count toward the test before them, that the shutdown limit covers interpreter exit, and that exit 5 on zero tests is 3.12+ behaviour applied on every version. * test: decode the runner's stderr as UTF-8 on Windows too (#350) A child Python writes a piped stderr in the ANSI codepage on Windows, so the non-ASCII limit value in the usage-error test came back as U+FFFD. The helper now sets PYTHONIOENCODING=utf-8 for the child. * test(history): give the writer's non-blocking checks a 1 s ceiling (#350) py3.13 on macos-latest failed on PR #351 with writer.close(0) taking 0.186 s against a 0.1 s ceiling while another thread held the commit. The blocked path these checks rule out waits up to 3 s (release.wait(3)), so a 1 s ceiling still tells returning at once from waiting for the release, without failing on a loaded shared runner (TESTING.md, #242). * test(restore): wait for the cancelled preview's worker before teardown (#350) py3.13 on ubuntu-latest failed on PR #351 (run 40334794505) with 'Files still open at teardown: prompts.db, prompts.db-shm' in test_cancel_while_reading_the_live_library. Cancelling the preview cannot stop its executor thread, which goes on to open the live library; the test returned without waiting, so that open could land between teardown's close_all_instances() and assert_closed(). The test now waits for the worker, as test_cancel_while_staging already does. |
||
|
Vito
|
3be0ada6a1 |
fix(ingestion): link an image only to a prompt that provably produced it (#344)
* fix(ingestion): link an image only to a prompt that provably produced it An LLM fed a PromptManagerPair's prepend_text while its text editor was empty. The node refused to save (an empty positive raised and logged a warning), and the image monitor then linked the image to a stale tracker entry or to the newest prompt in the database. - Nodes: an empty text editor saves and tracks nothing (prepend and append text are never the stored prompt), logs at info, and marks the node's run key empty so no earlier entry can be used for its images. - Tracker entries carry a run key: a digest of the prompt node and its whole upstream graph, the identity ComfyUI's cache uses. The monitor computes the same key from the image's embedded graph and uses a queued or snapshot entry only when the keys match (a snapshot also needs a known run with one entry, so a batch's last item is never applied to every image). - The recency fallback is gone; anything uncertain stays unlinked and is logged at info. - Recorded execution history attributes a file when exactly one run reported it, the file was written during that run, and the run's positive contexts name one stored prompt. The monitor and the output scan use it as their last certain source. - The scan and pair ingestion treat a resolved pair with an empty stored positive as unresolved instead of raising. - tools/review_image_links.py reviews selected links by these rules; dry run by default, --apply changes only named ids after a backup. Closes #342 * test(ingestion): real-host attribution case and the #342 testing contract The managed host disables the image monitor, so the owned fixture pack gains a route that starts and stops the plugin's monitor singleton for the attribution case only. The case runs in both renderers: a typed Pair image links, a Pair image whose positive is only linked prepend text stays unlinked with nothing stored, an unknown node feeding text links through its run key, and its cached re-run links again. * fix(ingestion): take a queued entry only from the run the image appeared in Dropping a key's older queue entries whenever the node ran again let a later queued run, executing during the monitor's processing delay, take the earlier image's entry. The snapshot taken when the image appears now names the key's latest run, and the queue is popped by key and run: same-key entries of earlier runs ahead of it are stale, entries of later runs wait for their own images. Also keeps the attribution case out of the stock real-host suite. * test(ingestion): host lookups degrade to no proof * fix(ingestion): address the #344 review (history re-check, keys, repair) - Images without an embedded API graph: the scan now asks recorded history before skipping a file with no prompt metadata, and the monitor gives a keyless unlinked image one history-only re-check when its run's outputs are recorded (a seal notification from the outputs write; at most 1000 waiting images, each for an hour). - Run keys: an image is unkeyable when any positive node has no key, so a two-pass graph is never linked through the other pass; links follow ComfyUI's is_link exactly. - A failed save marks the node's run key like an empty editor, so an older entry of that key can't stand in for this run. - A placeholder is the latest entry: the image stays unlinked without falling through to history, the same rule as a popped placeholder. - An index on execution_run_outputs(filename, subfolder, root_key, path_style) for the history lookup, added by migration and accepted by the output schema check (any other index is still refused). - Repair tool: unlink removes only the link record and only while the image is still linked to the prompt it was judged on; every run prints what --apply does. - Real-host case proves the re-run's PromptManager node was cached. * test(ingestion): real-host bare-output re-check and the review-round contract A metadata-disabled save, held open behind the fixture gate after SaveImage, waits unlinked while its run is open and links once the run's outputs are recorded. The fixture's monitor route lists the waiting files. TESTING.md covers the history re-check, the read index and what the repair tool's --apply does. * fix(ingestion): batch history lookups instead of adding an index An index on execution_run_outputs would make the library unopenable by older builds, whose schema check refuses any extra index there. The migration and the schema-check allowance are reverted, so the history schema is exactly main's. stored_run_prompts instead looks up a batch of files with one parameterized filename IN (...) query per chunk of at most 500 names and matches in memory; the scan computes it once per scan batch. * fix(ingestion): address the #344 re-review (seal race, linked roots) - A sealed history re-check that can't be scheduled yet (the image's own pass still pending, or the pending set full) is kept due and scheduled when a pending image finishes, instead of being dropped. - History matches a file through ComfyUI's spelling of its output folder first, then with both sides resolved, so a symlinked or junction output folder still attributes the scan's resolved paths; waiting files are keyed by their resolved path. - The scan makes its batch history lookup only when a file needs it. - set_current_prompt pushes to the queue last, so marking a failed execution never adds a second queue slot. - A keyless image outside the output folder doesn't take a waiting slot. |
||
|
Vito
|
48b1292115 |
feat(nodes): let a link drive the prompt text of every PromptManager node (#345)
* feat(nodes): let a link drive the prompt text of every PromptManager node The Focused, Split and Pair editors hide their widgets, and ComfyUI only draws a widget's input socket while the widget is visible. `text` (and the Pair's `negative_text`) therefore had no socket, and a STRING link dropped on the editor fell through to the first free STRING input, `prepend_text`, which is how #342's workflow was wired. - Detach those inputs from their hidden widgets on create and after a workflow load, so they are ordinary sockets in Classic and Nodes 2.0. The schema, widget order and saved values are unchanged. - A STRING link dropped on the editor body fills the main prompt, then the negative, before the frontend's own choice. - Each editor says when its text comes from a link; the typed text is kept for disconnect. The Classic nodes now save the typed text under a link instead of null, and "use prompt" no longer overwrites it while linked. - Real-host tests: mouse drags in both renderers, a workflow saved before this change, and executions proving the linked text is saved, counted, recorded in history and linked to the run's image. Closes #343 * test(coverage): floor for the linked-text module * test(nodes): PromptSearchList text takes a dragged link like a core widget * fix(nodes): linked text review round for #343 - A legacy frontend Primitive linked to an editor's text or the Pair's negative now runs with its value: the graphToPrompt wrapper and the queue hook fill it in (through virtual reroutes), in new and pre-#343 workflows. - Moving a linked text and releasing it on the editor keeps it on `text`: a link being moved counts as free, as the frontend counts it. - A link dropped on a prompt box (or its link note) fills that box's input, so the Pair's Negative prompt box takes `negative_text` (Classic through getInputOnPos, Nodes 2.0 through the box's slot key). - Only exact STRING links are steered; wildcards keep the frontend choice. - A linked Classic text box gives its row back and gets it back on disconnect; the PromptManagerText note uses the theme colour. - The metadata parsers map positional widget values through the widget table when an editor's widget-bound inputs do not line up with it. - Real-host cases for the legacy Primitive (new and saved workflows, a real Run), the moved link, the Negative box, paste/clone/undo and Split. * fix(nodes): resolve a legacy Primitive's text like the host, once per run (#343) - Inputs still bound to their widget (Classic PromptManager/PromptManagerText) are left as the host resolved them; the injection no longer overwrites them. - For the editors' detached text inputs, the Primitive's value goes through its "Run widget replace on values" replacement and the hidden widget's own serializer, so {a|b} wildcards and %date% tokens resolve as on main. - Each queued output is filled once (graphToPrompt wrapper, or the queue hook for an output that did not come through it): a wildcard is drawn once. - Editors inside subgraphs are found under their <subgraph>:<inner> id. - Real-host runPrompt installs its request route before the run starts. - Shared parser case pins the widget-table fallback for a core KSampler. * fix(nodes): pin the Classic wildcard behaviour and match queued ids by type (#343) - Classic PromptManager/PromptManagerText expose _pmTextSerializerReady, settled once their own text serializer is installed; the real-host specs wait on it and assert the Classic text is queued as written, as on main. - The Primitive fill only touches an output entry whose class_type matches the graph node at that id. - TESTING.md notes the host's harmless "Invalid widget" warning per Run. |
||
|
Vito
|
fb9134f081 |
feat(system): Backup & export (System S9) (#339)
* feat(system): back up, restore and export natively System > Backup & export replaces its hand-off with the concept's section on the S8 backend: Back up downloads the verified copy and the status line shows only what GET /backup/status reports (D103); Review restore stages a chosen file and shows the server's review in the shared review dialog, Cancel discards the stage and Restore runs the Database restore job, after which the open views reload (D105); Export JSON plus the components and saved-pairs exports (D103); Import / inspect opens the shared Import & inspect dialog; the safety-backup policy saves through the Save row and scheduled backups offer Manual only (D104). Overview's Backup status row reads the same status. Refs #326 * test(system): compare Backup & export with the concept system-backups-concept.spec compares the section, the Restore preview and the Import & inspect dialog with the concept in both themes at 1600 and 390 wide, mapping only D103, D104, D105 (and D15/D55 for the shared Import dialog), and proves a renamed label, a removed row, a lost danger mark and swapped tokens are noticed. A review's grid values now wrap anywhere (D105), so a long backup file name keeps the concept's two equal columns at phone width. Refs #326 * docs(testing): Backup & export contract (D103-D105) Refs #326 * test(coverage): raise the frontend floors S9 measured system-backups.mjs gets its measured floor in place of the provisional one; admin.js, activity-scan.mjs and maintenance-review.mjs branches go up. Refs #326 * fix(system): keep a refused or handed-back restore review usable A Restore refused as busy now opens the same review again with Discard / Restore, and a restore job whose leases did not drain hands its review back (scope.review_kept) to the page that started it, which reopens it; every other path that abandons a token after Restore sends its DELETE, so held reviews can no longer lock restore out. The kept-review line says where the review is, and a full review store says what to do (I1). applyPlan resolves with its outcome and its job tracking now has failing-before proof (N5). Also: the safety-backup toggle says what the policy does (D104, M2); the restore picker has no type filter so safety backups can be chosen (M3); "Export all prompts JSON" names what /export holds (N1); small snapshots show KB (N3); the duplicated holding and save helpers are shared (N4); a dismissal test covers Cancel, x, Escape, a replacing dialog and leaving (M1). Refs #326 * test(coverage): raise the system-backups branch floor Refs #326 |
||
|
Vito
|
f9835aa4d6 |
fix(lora): allow civitai.red/.green and log why a CivitAI download failed (#340) (#341)
civitai.red and civitai.green (and their subdomains) are now allowed redirect targets under the civitai.com rules: HTTPS only, default port, no credentials, matched on the parsed host name, at most 3 hops, never with the key. Their www/api/image hosts are also allowed starting hosts, whose first request carries the key as image.civitai.com does. Each failed download now logs one warning with the URL without query or credentials and a short reason: HTTP status, refused redirect, hop limit, size cap, decode or save error, timeout or network error. |
||
|
Vito
|
8252d9d226 |
feat(system): Integrations and a staged LoRA import (System S10) (#336)
* fix(lora): detect LoraManager in ComfyUI's extra custom_nodes folders Auto-detection looked only in ComfyUI's main custom_nodes folder (#144). It now also searches every custom_nodes folder ComfyUI registers through folder_paths, main folder first, skipping folders that cannot be read. * fix(lora): stage imports before replacing the collection (#221) The LoRA import deleted the whole lora-manager category first and then saved each record in its own transaction, so a discovery, download or database failure left the previous collection gone or partial. The import now runs in three stages: discover and fingerprint the metadata files, stage every record (CivitAI downloads go into the image cache first, counted), then replace the category in one BEGIN IMMEDIATE transaction (replace_category_prompts). Any failure leaves the previous prompts, tags and image links exactly as they were. Prompts a saved pair uses block the replacement; an empty replacement of a non-empty collection needs a reviewed import. POST /prompt_manager/lora/import/preview reviews an import without writing or requesting anything; POST /lora/import/apply {token} runs it as the Activity job "LoRA reference import" (records admission, cancellable while it downloads, the safety backup policy before the replace, refused with plan-changed when the files changed since the review). The legacy POST /lora/scan runs the same stages with its SSE event shape. * feat(lora): the LoRA save takes the settings revision POST /prompt_manager/lora/enable accepts the optional expected_revision that GET /settings returns, so the System Save row refuses to write over a config.json changed elsewhere (settings-changed, nothing written). The legacy page sends none and still merges into what is on disk. * feat(system): configure ComfyUI and LoRA integrations natively System › Integrations replaces its hand-off with the concept's two surfaces. ComfyUI: only the send destination and renderer the bridge supports can be chosen; the other options and Return to canvas are disabled with the reason, and Host compatibility shows the renderer the open ComfyUI windows report (each bridge destination now carries renderer: classic or nodes2) (D106). LoRA & resources: detection, the enabled setting and the loaded trigger words read as separate facts, the read-only LoraManager path row follows the toggle (D106), and the CivitAI key is never filled in: typing drafts a new key, blank keeps the saved one and Clear saved key drafts its removal (D114). They save through the one Save row (/lora/enable with the settings revision). LoRA reference data → Review import shows the server's read-only review (replacement, what saved pairs block, downloads, backup box) and Start runs the LoRA reference import Activity job (D107), with its own Activity descriptor and Review again into Integrations when the files or pairs overtook it. * test(system): compare Integrations with the concept system-integrations-concept.spec compares System › Integrations (both themes, 1600 and 390 wide) with the concept, mapping only D106 and D114, and the LoRA import review with the concept's record-changing review, mapping D107; each comparison catches a removed row, a renamed label or option, a re-enabled unsupported option and a swapped token. On the real ComfyUI host, each bridge destination reports the renderer its window draws nodes with, before and after switching it, and System › Integrations shows it. * test(coverage): record the Integrations frontend floors coverage:update after the full frontend run: system-integrations.mjs at its measured floor, and the raised maintenance review, job view and maintenance floors. * test(system): wait for the renderer line before clicking Integrations rows CI Firefox (390 wide) clicked the LoraManager toggle while the renderer check's line above it changed text and moved the rows, so the click did not reach the switch. The specs now wait for that line to settle (a visible state) before interacting. * fix(lora): recheck the files before the replace and stop an abandoned scan Review fixes for the staged LoRA import (#221): - The metadata files are listed and fingerprinted again right before the replace transaction, so a file changed, removed or added while the downloads ran aborts the import with nothing deleted (System job: plan-changed, Review again; legacy scan: an error frame in its own words). The tests now change real files behind a held download. - A legacy scan whose page closes stops staging before its next LoRA and keeps its records hold until staging has stopped; CivitAI images are written to a temp file beside the cache entry and moved in place, so a half-written image is never seen as cached. - Start refuses with plan-changed when the integration was turned off or moved to another install since the review. - A failed job keeps the skipped count and the names of the LoRAs that could not be prepared. - The replace binds the category instead of one parameter per prompt (SQLite's variable limit), /lora/status says whether a detected install can be used, and the legacy scan logs expected refusals as warnings. * fix(system): partial saves, key focus and readable selects in Integrations A save that writes the LoRA part but not the settings part now keeps the saved part, adopts the new revision and highlights only what failed, so the next save needs no conflict. The CivitAI key status is a persistent live region, Clear saved key returns focus to the key field, and the key draft is dropped when the section is left. The send-destination and renderer selects keep their width on narrow screens. A detected LoraManager outside custom_nodes says it cannot be enabled. * fix(lora): keep the save outcome, honour the umask and sweep stale downloads Leaving Integrations drops the CivitAI key draft only when a key was typed, so the last Save's outcome survives and Sources still marks a refused source. The key status line is rewritten only when it changes. Cached CivitAI images are opened like a plain save, so they keep the umask's mode, and each import removes .pm-*.part files older than an hour from the image cache. A legacy scan stopped because its page left ends cancelled in Activity, and staging is awaited to the end even if the handler is cancelled again. * fix(lora): the import review takes a lease beside a restore The LoRA import review reads the library, so like the maintenance and restore previews it now runs under a request lease: it is refused with 409 busy while a restore runs, and a restore waits for it before it switches the file. Tests pin that a restore is refused while an import runs and that the review, the reviewed import and the admin page's scan are refused while a restore runs. |
||
|
Vito
|
28ad3a1a1f |
feat(system): backup and restore backend (System S8) (#335)
* refactor(database): stage restores before publishing restore_from_file is now stage_restore + publish_staged (database/ restore_stage.py). The stage lives in a .pm-restore-* folder in the library's own folder, is migrated, has damaged workflow documents repaired from an inline copy or cleared, is verified with every document read and is bound by its SHA-256. Publishing keeps a unique, standalone, 0600 safety backup (microsecond names), copies the pages through the backup API as before and then truncates the WAL, which stayed database-sized after a restore. The safety backup writer moves to database/safety_backup.py so the database layer and maintenance share it. * fix(database): read the verify streams one row at a time The provenance and review-evidence checks iterated their cursors; on a managed connection iteration prefetches 128 rows, so up to 128 large values were held at once while the docstrings promised one. Both now use fetchone. * feat(backup): review a restore, run it as an Activity job, verify backups - POST /restore/preview stages an upload in the library's folder and answers what it holds beside the live library, writing nothing live; DELETE discards it; at most two stages are held (429 too-large). - POST /restore/apply rechecks the live fingerprint (409 plan-changed) and starts "Database restore": exclusive, not cancellable, indeterminate. It waits for request leases, publishes the exact staged bytes, swaps the database, prunes safety backups and logs counts, repairs and the backup. - The legacy /restore runs the same stage and publish as an exclusive hold with its old response, so a restore and the health check, Move library and every other job refuse each other. - JobRegistry.lease: Needs review decisions and maintenance and restore previews are refused while an exclusive job runs, and a restore waits for them before it switches the file. - GET /backup holds a guarded slot, verifies a standalone snapshot before streaming it and removes the closed temp file; GET /backup/status lists the safety backups on disk and this session's verified download. * docs(testing): backup and restore contract * test(system): a restore refuses a running health check The health route test relied on a legacy restore interleaving with a running check; the restore is now exclusive and refuses it, naming the check. The diagnostics backup tests follow the backup handler into py/api/backup_restore.py. * fix(backup): drop stale restore previews off the event loop A restore drops the previews reviewed against the old library; their stage files are now removed in the executor, with the old database's connections. * test(backup): wait for the backup hold to end before reading its state CI (py3.11/3.12 on Linux, every macOS) read the registry right after the client had the last byte, before the handler left its hold and removed its temp file: the job was still running. The tests now wait on that state. * test(images): the scan dialog's Back up now leaves a finished backup hold A backup download now holds a guarded registry slot while it streams (#249 S8), so the jobs list holds that finished hold after the dialog's Back up now; the spec still proves Cancel starts no scan. * fix(backup): final review round for the restore and backup routes - I1: once the stage's copy replaced the library the restore succeeded: pruning, reopening self.db, dropping old previews and closing the old object only log, so the job finishes and the legacy /restore answers 200 instead of "the library was not changed". The generation is always bumped. - M1: a backup download holds its guarded slot only while the snapshot is written; the stream runs outside it, so a stalled download blocks nothing. - M2: a restore preview leases only its read of the live library, not the upload or the staging; a restore whose drain still times out keeps the reviewed stage under its token. - M3: the reviewed-preview store changes only on the event loop; stage folders are removed in the executor. - M4: a document whose body inflates completely while only its recorded hash or only its length disagrees is re-keyed (or merged into the document holding that text) instead of dropped. - M5: a backup download is refused only when the snapshot is not SQLite or fails integrity_check; a semantic check failure still downloads, logged, sent as X-PromptManager-Verify and recorded as not verified. - M6: a refused, failed or cancelled preview removes its stage, also when the cancel lands while a worker still stages; a timer removes expired stages without another request. - M7, N1: repair reads rows one at a time and counts distinct images. - N3: the live fingerprint never commits a transaction it did not open. * test(backup): close the old library the close-failure test leaves open CI (py3.12 Linux and the coverage gate) found the old object's executor-thread connections still open at teardown: the test patches its close_all to fail, and once the object is collected close_all_instances no longer reaches it. The test now closes it itself. * fix(backup): re-review round for restore repair, apply and backup cleanup - R1: when re-keying merges an intact body into a holder whose own body was damaged, the rewritten holder leaves the damaged list, so the clear pass no longer deletes it and unlinks the images just moved to it. - R2: a losing duplicate apply removes a stage only when the store still held it, never one another apply's job has taken. - R3: a restore that times out waiting for requests keeps its review only while it is unexpired and its slot is free; otherwise the stage goes and the job says to try again. - R4: a backup download cancelled while its snapshot is written removes the snapshot once the worker finishes. |
||
|
Vito
|
9ab25f9aaa |
feat(system): Maintenance UI (System S7) (#322)
* fix(gallery): clear only Prompt Manager cache entries (#222) The legacy gallery's Clear cache deleted every localStorage key starting pm_ or promptManager_ (the workspace theme and the dismissed update notice among them) and every CacheStorage cache whose name held gallery or thumbnail, which on ComfyUI's shared origin can belong to another tool. web/js/cache-ownership.js names what a clear may remove from an inventory of the real writers: every key Prompt Manager writes is a preference and no page opens CacheStorage or IndexedDB, so nothing is removed and the toast says so. A test fails when a new storage writer is not classified. The server thumbnail clear stays its own request; a refused or unreachable server changes nothing and is reported, and refused browser storage is reported as such. * feat(system): run reviewed maintenance natively System › Maintenance is native: the concept's notice and its Database and Images & cache surfaces replace the hand-off. - Integrity & relationship checks runs the health check; the report's Review missing links reviews a fresh Missing-file links plan. - Review plan shows the server's read-only plan (D100): scope, protected, backup policy and media lines, the listed records, how long the review holds and the backup box; Apply reviewed operation sends only its token and opens the job in Activity. busy names the running job, plan-used links the job that applied it, an expired or replaced review is reviewed again, and a lost answer says Outcome not confirmed and points to Activity. - Find duplicates starts the search; its view offers Review groups (D101): ten groups at a time, Keep radios per copy, each keeper sent as {group_id, root_index, path}; the dialog says History keeps the removed paths in its run records. - Thumbnail cache opens the thumbnail dialog; Metadata cache is disabled with its reason while nothing is owned (D102, Q5). - Activity descriptors for maintenance, duplicates and duplicate_cleanup. - The legacy duplicates dialog's View Full names its source (?root=). * test(system): compare Maintenance with the concept system-maintenance-concept.spec compares System › Maintenance, the plan dialog of each operation and the duplicates dialog with the concept in both themes and both viewports, mapped only by D100–D102, and proves a mutated row, label, danger mark or token is noticed. The duplicates dialog's Review duplicate cleanup is the concept's plain button. * test(system): Maintenance no longer hands off; raise coverage floors The System shell test expected Maintenance's D94 hand-off; the section is native now. coverage:update sets the new modules' floors from the measured run (cache-ownership 94, maintenance-review 94, system-maintenance 92 lines) and raises the floors other files passed. * test(admin): close the preview worker after the last job in teardown A scan publishes its last event, which ends the stream the test reads, before it hands its missing previews to the automatic worker. That hand-off can run after tearDownAsync closed the worker, so it starts a new one whose .part files were still open when the final drain checked for open handles (py3.10 on ubuntu, test_scan_reads_metadata_on_several_threads). The drain now closes the worker for good once every job has ended. * fix(system): address the S7 final review - I1: an apply answer is a refusal only when our server sent it (a known code, or a 4xx whose body says success:false); a proxy's 5xx, an unreadable 2xx or a dropped connection says Outcome not confirmed and reads Activity again. ApiError gains 'refused'; the server's own pre-start failure is coded 'apply-failed'. - M1: the lost-response test lets the apply run, then drops its answer. - M2: a failed apply records its code in the job's scope; Activity offers Review again for an overtaken plan (Maintenance for a duplicate cleanup whose sources changed). - M3: a failed duplicate cleanup toasts the server's reason. - M4: a review button holds while its preview is read (one preview per double press). - M5: the storage guard also catches bracket and property assignment and CacheStorage/IndexedDB opens; why the Toolkit build is skipped is written down. - M6: a refused storage listing says so on the Metadata cache row. - M7, M8: D100 text corrected; workflow_storage has its title. - M9: a busy start names the job the server named. * test: follow #319 in the Maintenance tests - The thumbnail dialog has only Scope since #319. - The admin teardown refuses new preview workers on its own API instead of closing with final=True, which since #319 also shuts the process-wide encoders that later test modules use. * fix(system): keep the server's search-again line on a duplicate review A duplicate cleanup review the server refuses with 409 sources-changed showed the generic "could not be reviewed" message; it now shows the server's "The sources changed since the search. Search for duplicates again." * fix(system): a plan with nothing to do offers only Close A reviewed maintenance plan whose scope is 0 records ("No duplicate prompt text") offered Apply reviewed operation, which started an empty job. It now shows its rows, "There is nothing to apply; nothing was changed." and a Close button. * fix(system): wrap long copy paths in the duplicates review A Keep label is the copy's source and path, and dataset folders often have long names with no space to break at, so the review scrolled sideways. The label now wraps anywhere inside its compare item (D101). |
||
|
Vito
|
2964a3b632 |
feat(thumbnails): WebP/JPEG XL thumbnails, server-side thumbnail settings and Convert outdated (#319)
* feat(thumbnails): thumbnail format, quality and size as server settings
Thumbnail format (webp default, jxl, jpeg), quality (1-100, default 80)
and size limit (128-2048 px, default 512) are server settings in the
config.json thumbnails section, saved with the settings revision. The
never-read gallery thumbnail_size is gone: older saves wrote its unused
default 256 into config.json.
One module names previews (<stem>_thumb.webp/.jxl/.jpg) and finds the
preview a source owns, preferring the configured format; legacy names
are still found. The encoder module writes every preview atomically and
verifies it decodes. JPEG XL needs no new dependency: Pillow (natively
or with pillow_jxl), ffmpeg with libjxl, or cjxl; without one the
settings refuse jxl (400 jxl-unavailable) and GET says why. Encoder
processes are tracked, bounded and killed at shutdown.
* feat(thumbnails): every preview writer and lookup follows the thumbnail settings
Automatic previews (capture and first view), the thumbnail job, the
legacy gallery routes and the history route write <stem>_thumb.webp
(or .jxl/.jpg) with the configured quality and size limit, for images
and videos alike: video frames, ffmpeg's included (a PNG on stdout,
never upscaled), and the placeholder are encoded like images.
Listings, duplicates, enrichment, the autotag scan and the history
route show any preview the file owns, the current format first, so
older _thumb.png previews keep working until converted. A request for
an older name answers the current preview. Previews are served with
an explicit image/webp or image/jxl type and the same cache headers.
The plan and status count missing, outdated (another format) and
ready previews; the missing job never writes beside an outdated one.
Removing a file removes its previews in every format; the clear
removes .webp and .jxl too. Encoder processes are killed at shutdown.
A preview is only a placeholder: inspection refuses paths under
thumbnails/ and always reads the original, and the inspector cover
accepts the listing's preview in any format the file owns.
* feat(thumbnails): Convert outdated thumbnails as an Activity job
POST /jobs/thumbnails {mode: "convert"} starts "Convert outdated
thumbnails" under the thumbnail job's kind, guarded admission, worker
threads and cancel at a file boundary. For each file whose preview is
in another format, it writes the new preview from the source (atomic,
verified; a video with no decodable frame fails instead of getting the
placeholder), then removes that file's old-format previews in its
thumbnails folder. A failure keeps the old preview; orphan previews
and links are never touched. The final line reports converted, failed
and bytes before and after. The plan counts it with mode=convert; the
legacy gallery stream refuses to attach to a running conversion.
* feat(thumbnails): thumbnail settings in Appearance and Convert outdated in the workspace
System > Appearance gains a Thumbnails surface: format (WebP, JPEG XL,
JPEG), quality and size limit, saved to the server with the settings
revision. JPEG XL is offered only when the server has an encoder, else
the option is disabled with the server's reason (D130); a note says
when this browser cannot show JPEG XL. Saving a new format converts
nothing: the row then says previews keep their old format.
Thumbnail maintenance gains the Convert outdated scope and names the
server's settings instead of its own quality and size fields (D131).
Overview's thumbnail row reads N missing, M outdated, K failed and
offers Convert outdated while previews are outdated.
The workspace decodes a tiny JPEG XL once: where it cannot, cards, the
Library and the inspector cover load the original for a .jxl preview,
and any preview that fails to load falls back to the original. The
viewer and downloads keep using the original (regression test).
README explains enabling JPEG XL; TESTING.md records D130 and D131.
* test(thumbnails): coverage floors for the thumbnail modules and WebP in page workflows
The two new frontend modules enter the coverage baseline (98% lines);
other floors only rise. The visual fixture's preview is WebP and
Appearance has its Thumbnails surface in the System shell check.
* fix(thumbnails): say when the thumbnail settings could not be read
Like Appearance's other server rows, the Thumbnails rows show 'Server
settings could not be read.' while the read failed, instead of only
staying disabled. Firefox exposed the gap: its JPEG XL note arrived
first and changed the row notes the page workflow counted.
* test(thumbnails): accept the legacy preview's spelling a case-insensitive disk finds
On Windows and macOS the '.jpg' lookup finds an older release's
b_thumb.JPG first, so the listed URL spells '.jpg'; it names the same
file there. The test now accepts either spelling and checks the URL
serves that preview.
* fix(thumbnails): address the final review of the thumbnail format work
- ffmpeg writes JPEG XL with -update 1, so a % in a file name is never an
image-sequence pattern, and reads videos as file:<path>. The clear and a
startup sweep remove temp files that interrupted writes left behind.
- Convert outdated converts each group of same-stem files once, from the
first source in the R2 order. An old preview that is already gone counts
as removed, and each file's bytes count once. A current preview is only
kept if it decodes.
- An encoder that fails makes the write detect encoders again once. Another
encoder is used if one is found; with none, the file is written as WebP
and the availability Appearance shows is refreshed.
- Changing the thumbnail format, quality or size while Convert outdated runs
returns 409 busy. If the format still changes underneath, the job stops
without removing anything.
- Legacy preview spellings are matched case-insensitively, so X_thumb.JPG
counts as outdated.
- History's cover uses the same lookup as Images. It serves .jxl only when
Accept lists image/jxl, and otherwise the original or the placeholder.
- A video whose thumbnail the browser cannot show keeps its own element
(Images) or its label (Library).
* fix(thumbnails): keep the new preview, probe encoders only when gone, sweep reused PIDs
- Convert never removes a path that is the preview it just wrote. It
compares the device and inode, or the resolved and normcased paths when
the disk has no inode numbers. This covers x_thumb.JPG on a
case-insensitive disk.
- A JPEG XL encode failure re-runs encoder detection only when the
encoder's binary or module is gone. A "no encoder" result is checked
again on the settings read, on a jxl save, and after 60 s, so one slow
probe no longer turns JPEG XL off until a restart.
- The startup sweep now also removes temp files that carry this process's
PID but predate its start. Containers reuse PIDs.
* fix(history): let the cover ask for JPEG XL when this browser decodes it
Firefox can decode JPEG XL without listing image/jxl in Accept. When the
workspace's probe decodes JPEG XL, it adds jxl=1 to the History cover URL.
The server serves a .jxl preview when that flag is set or when Accept
lists image/jxl. In every other case it falls back as before. The flag
accepts only 0 or 1, and the response stays no-store.
* fix(thumbnails): keep going past the per-batch cap until every preview is made
The missing, rebuild and Convert outdated jobs stopped after
MAX_THUMBNAIL_FILES (10,000) files and said "N more on the next run", but
nothing ever started that next run. The cap now bounds one batch. After
each batch the job plans again, leaving out every file it already worked
on, and continues until nothing is left, the job is cancelled, or a batch
makes no progress. That last case stops on files that fail every time
instead of looping over them.
Activity progress counts against the overall total, and the final line and
the legacy stream's complete frame report every batch. Admission, worker
threads, cancellation and the #316 rules are unchanged. The Thumbnail
maintenance dialog now counts all the work instead of saying "more on the
next run".
|
||
|
Vito
|
8bf759d6f8 |
feat(system): keep the library in ComfyUI's user folder (#318)
* feat(system): resolve every library path in one place utils/library_location.py decides where the library lives: the PROMPT_MANAGER_DATA_DIR folder (validated: absolute, never inside a folder ComfyUI or Prompt Manager serves), an explicit database.default_path as before, a node-folder library that was not moved, else ComfyUI's user/__promptmanager. The database, diagnostics, config.json, logs and the LoRA image cache all ask it, so a new install never writes into the node folder and an existing one keeps every path until the user moves it. "Keep it here" is persisted as library.keep_in_node_folder. * feat(system): move the library as a verified Activity job POST /prompt_manager/system/library/move starts "Move library", an exclusive, non-cancellable job (database/library_move.py). It checks the target volume's free space, copies the database with SQLite's backup API from one read snapshot, verifies the copy (integrity check, every workflow document, the same rows in every table), copies the safety and pre-migration backups and config.json (owner-only, checked by size and SHA-256), then publishes under the old file's write lock and switches the live process (every PromptModel, the review and history observers, the API's plans and health report). The old prompts.db and config.json are renamed *.moved-to-<folder> only after the switch. A failure removes everything it wrote and leaves the old library in use; a marker in the new folder makes a restart after any crash pick the right library. GET /prompt_manager/system/library and the Overview's "location" say where the library lives and why; POST .../keep persists "Keep it here". * feat(system): show where the library lives and offer the move D129. System > Overview's Library health gains a "Library location" row (the database path, why it is there, the "Keep it here" choice and any PROMPT_MANAGER_DATA_DIR problem) with Move library for a node-folder library and Details otherwise. While a node-folder library is not kept there, the workspace shows the concept's .notice above every page: "Keep it here" saves the choice on the server and stops the notice; "Move library" opens the shared review (From, To, Size, what moves, what happens to the old files) and starts the exclusive job in Activity. The notice and the Overview read the location again when the job ends. The Overview comparison maps the inserted row; a new comparison holds the notice to the concept's Sources notice (markup, tokens, alignment, both themes and widths) with a mutation check. * feat(system): close the browser server's log files before its root goes The logs now live in the library's data folder, which for the browser test server is <root>/user/__promptmanager; Windows could not remove the temp root while the server still held prompt_manager.log open. The API test's node folder is spelled in lower case because public paths are case-folded on Windows. * feat(system): show the library location on the legacy admin page A line under the admin page's statistics names the database path and why it is there (ComfyUI's user folder, PROMPT_MANAGER_DATA_DIR, config.json, or the node folder with a pointer to Move library), read once from GET /prompt_manager/system/library. * feat(system): prove a fresh host keeps its library in user/__promptmanager The execution suite starts one pinned CPU ComfyUI with no configuration at all on an OS-chosen port: the library is created in user/__promptmanager (database and logs), nothing but byte code is added to the node folder, the location route reports the default folder, and ComfyUI's /userdata API serves none of it (traversal spellings, a listing, a system-user header). * feat(system): document where the library lives and how it moves README: where the library lives, the upgrade path (Move library / Keep it here), PROMPT_MANAGER_DATA_DIR, explicit paths unchanged, and the way back to an older version. TESTING.md: D129 and the library location contract (resolution order, move steps, crash windows, coded refusals, Windows handling). * wip(system): library move fix round in progress * fix(system): retry a fallback image link that races the library switch The image monitor's fallback and recent-prompt links went through PromptDatabase.link_image_to_prompt, which logs and returns 0 on any error, so survive_switch never saw a "database is locked" or closed connection during Move library and the link was lost in both files. The monitor now links through link_image_to_prompt_or_raise (one transaction per attempt); link_image_to_prompt keeps its log-and-return-0 contract for other callers. A link that returns 0 is no longer logged as "Successfully linked image 0". A switch that fails after repointing the models now closes the old file's detached connections instead of leaving them to GC: switch() adds them to a list the mover owns. The mocked retry test is replaced by a real-lock race asserting the image row lands in the new library only. * fix(system): say which source is the default and where a symlinked library lives Sources & scans: with no source configured, ComfyUI's output folder now reads "ComfyUI/output · watching · default" on its own row instead of a separate note line. GET /settings marks each effective source with in_comfyui (its public path is relative to the ComfyUI base) so only those get the "ComfyUI/" prefix; the scan still sends the server's public path. Library location: a library in the node folder reads through the node folder's unresolved place in custom_nodes, so a node folder symlinked to a checkout outside ComfyUI shows custom_nodes/promptmanager/prompts.db instead of prompts.db (System library row, Move review, admin Library line). Never absolute or "..", else the shared _public_path, which sources and jobs keep using unchanged. |
||
|
Vito
|
fc3266d455 |
fix(thumbnails): generate thumbnails automatically and use them everywhere (#316)
* fix(thumbnails): generate thumbnails automatically and use them everywhere New outputs never got a preview, so the workspace grid loaded full-size originals (844 of 19,907 images on a live install, exactly the newest). - Listings always give images and videos a thumbnail URL; a missing preview is made by the serve route on first view (bounded, single flight per file) and falls back to the original, remembering files that cannot be decoded. - The image monitor queues each settled output's preview on one background worker that follows Worker threads. - A scan that leaves 200 or more previews missing starts the visible thumbnail job; fewer go through the worker. - Automatic previews wait for a restore, a duplicate removal, a sources change, the gallery's clear and exclusive jobs. - The inspector cover uses the preview; the Overview row counts previews that failed. * fix(thumbnails): keep the video fallback tested and stop the preview worker before cleanup A video listed without a preview still renders its own element and is released on refresh. Test teardowns and the browser reset stop the automatic preview worker before the temp tree goes, so no preview lands after cleanup (Windows). TESTING.md documents automatic previews. * fix(thumbnails): admit previews on the loop and never send a video to an img Final review fixes for automatic previews: - A video preview that cannot be made is the placeholder frame (no-store), never the video; OpenCV failures fall through to ffmpeg and the placeholder. Cards and the inspector cover swap back to the video element when the preview image fails. - View requests are admitted on the event loop (3 writers, one flight per file, at most 64 waiting; past that the fallback at once). Writes run on a small dedicated pool, never the default executor. The history route shares the admission, flight key and link checks. - Drive, UNC, empty and dot segments are refused as text; the folder listing runs inside the admission. - Holds are checked after the slot and again before the rename; only decode failures are remembered. - Bounded, logged shutdown: ffmpeg is killed, no worker restarts after cleanup, and a reload stops the old worker. - The listing says whether the preview file exists (thumbnail_ready) for the legacy gallery's dot; media in a nested thumbnails folder keeps its original; file names the server quotes match in the inspector. * fix(thumbnails): cover the image fallback card and document the admission An image listed without a preview (a nested thumbnails folder) still shows its original; TESTING.md describes the view admission, the video placeholder, the holds before the rename and the bounded shutdown. * fix(thumbnails): let an idle preview worker leave and fit case-insensitive filesystems The capture worker exits once its queue is empty, so no thread outlives its work; the next output starts a fresh one. The mixed-case source test expects no folder listing where the exact name already resolves (macOS, Windows). * fix(thumbnails): answer shared flights with the outcome only and restart a failed worker The history and serve routes share a flight key, so a flight now answers only the ensure outcome whichever route started it; the serve route derives the preview path and, for a fallback, looks its source up again on the view pool. A view joining a history flight no longer gets a 500 or an image answered with the video placeholder. The capture worker clears its running state however its loop ends, and a worker that cannot start leaves the file queued for the next enqueue. |
||
|
Vito
|
06099579e6 |
feat(maintenance): store each workflow once (#315)
* feat(maintenance): reviewed, read-only maintenance plans POST /prompt_manager/maintenance/preview reviews duplicate_prompts, optimize, unused_components or missing_links on its own mode=ro snapshot and answers a Plan (scope, protected reasons, backup policy, media, at most 50 detail rows). The fingerprint, targets and judged sources stay in a server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB, never truncated; a restart forgets them), bound to the database it read. The previews reuse the legacy queries: duplicate groups and the merge are shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the #239 protection predicates the plan reports, and missing links are judged by the health check's link_states. Legacy /maintenance keeps its shape; vacuum now reports success:false when VACUUM fails, and cleanup_orphaned_images skips links under unavailable sources. * feat(maintenance): apply reviewed plans as Activity jobs POST /prompt_manager/maintenance/apply {token} starts a maintenance job (records admission, not cancellable, groups changed/protected/errors). The token is consumed on the loop turn the job starts: a second apply is 409 plan-used naming the job, a busy refusal keeps the token, an unknown or expired one is 410 plan-expired, and a restore since the review is 409 storage-changed. With the safety backup policy on (maintenance.safety_backup, default true, saved through the settings route with the revision), the job first writes a backup into a temp file beside the database, makes it a standalone rollback-journal file, verifies it, fsyncs it and renames it onto a unique <db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file and nothing changed. The apply then rechecks the reviewed fingerprint inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of 400 ids, counts changed and protected records, logs the backup by name only and prunes old safety backups. With the policy off the log and the status line say "No safety backup". * feat(maintenance): find and remove duplicate files through Activity Duplicate image files move onto the job registry: - POST /prompt_manager/jobs/duplicates finds content-identical media files over every effective source as a cancellable read job ("Find duplicate image files", group groups_found); the server holds the groups per job and GET /prompt_manager/maintenance/duplicates/{id} pages them ten at a time. - The duplicate_files preview binds a removal to the reviewed keepers. Every keeper and removal is checked again: containment as text before any filesystem access, then not a link, still inside its source, same size and hash. The plan lists the database links that move. - Apply runs duplicate_cleanup (records, not cancellable): the promised safety backup, then per file a recheck, the unlink with every handle closed and its thumbnail. A changed file is kept, the first failed unlink stops the job with a coded message (never the OS text), and only removed files have their image links and favorites moved to the kept copy. - The legacy scan, stream, status and delete routes keep their shapes on the same registry kinds; the hashing, grouping and thumbnail removal are shared with them. Coordinator rulings on Tasks 15-16: - Safety backups share the restore pool of 10 `<db>.backup_*` files; the pool is now also pruned when an apply fails after its backup. - Plans report the safety backup's size and the free disk space, and apply refuses up front (507 insufficient-space) when the backup or VACUUM does not fit, so no job fails halfway for space. * fix(maintenance): never remove the kept copy; move records before unlink Final review of #313: - C1: the duplicate find tells files apart by (st_dev, st_ino), so a source listed twice, a symlinked source inside another and a hardlink are one file. The preview and the job refuse a removal that os.path.samestat matches with its kept copy. - I1: each file's records move to its verified kept copy in one short transaction before that file is unlinked; a failed move deletes nothing and a failed unlink leaves the records on the identical kept copy. The status line counts files removed and not removed. - I2: links are looked up under every spelling of the source (the real path, the configured and monitored folders, ComfyUI's output). - I5: image tags join the kept copy as their union and the removed copy's pending, undecided Needs review items are settled; the plan says so. - I3: the VACUUM disk check measures SQLite's real temporary folder. - M1: the token is looked at again with no await before the job starts. - M2: the backup decision is made once and handed to the job. - M3: backups are linked into place; startup sweeps interrupted temps. - M4: Optimize runs alone (new "exclusive" admission) and its plan says the database is locked while VACUUM runs. - M5: the reviewed duplicate merge is strict: collisions and Library titles are protected, and errors roll back. The legacy cleanup is unchanged. - M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests. - M9: the preview binds files by size, mtime and inode without hashing; the job hashes right before each unlink. * fix(maintenance): protect one file reached through two mounts Re-review of #313, round 2: - R1: a removal on another device than its kept copy, with the same size and mtimes within 2 s, is protected when its inode is equal or its path within its source is the same ("May be the same file reached through another mount or share"), in the preview and again right before each unlink. The job holds the kept copy open during the unlink, so on Windows deleting the kept copy itself through another path fails. - R2: a file's spellings are found by containment: every configured, monitored, ComfyUI-output or listed folder whose stat identity is a folder above the file. Parent spellings of nested sources and case variants match; favorites move under every listed root holding the file. Lookups stay index searches. - m1: a failed unlink's row says the links now name the kept copy. - m2: a volume with no file ids reads "Can't confirm this is a separate file on this drive". * fix(maintenance): stat duplicate candidates with os.stat Python 3.10's Path.stat uses the os.stat bound when pathlib was imported, so the find and the per-file checks could see different stat calls. The find now calls os.stat directly, like every other check; the mount tests failed on py3.10 only. * fix(maintenance): recheck the kept copy right before each unlink Re-review 2 of #313: - n1: the cross-mount guard also matches when one path within its source ends with the other on a whole folder name (sources listed at different depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount). - n2: after a file's records move and right before its unlink, the held kept copy must still be the file at its path (fstat of the hold against os.stat) and the cross-mount guard must still hold; otherwise the file is kept, its records move back to it, and it counts as not removed. * test(maintenance): the kept-copy race tests hold on Windows On Windows the removal job holds the kept copy open without FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during the record move is refused. The tests now assert that native outcome: the kept copy stays and the removal goes ahead. * feat(database): store each workflow and prompt graph once generated_images.workflow_data and prompt_metadata repeated the same large JSON text for every image of a run. Each distinct text now lives once in workflow_documents, keyed by the sha256 of its exact UTF-8 bytes and zlib-compressed; image rows reference it through workflow_document_id and prompt_document_id. - New writes (link_image_to_prompt, pair ingestion for scan, monitor and Needs review) store the new form. Old rows keep their inline text and read the same until the migration rewrites them. - Every reader goes through one accessor (read_stored_text, resolve_image_row): gallery and legacy image lists, image by id, prompt info for the inspector and metadata page, diagnostics. Payloads keep their shape. Inline text wins when present; a document that fails its length or hash check reads as unreadable, never as other text. - No trigger (restore validators refuse them): a document goes in the same transaction as its last reference (image and prompt deletes, missing-link cleanup, duplicate merges, duplicate-file link merges, ingestion updates), and a sweep at startup catches anything a path missed. - verify_database_file validates the table, the id columns, their foreign keys and indexes, and the stored values. * feat(database): migrate old workflow rows in batches, then reclaim space The workflow storage migration rewrites old-form image rows (text inline) to document ids, batch by batch: - Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or 16 MB of text, found by a rowid range search and read one row at a time. - Each text is stored or reused, then read back through the one accessor before its row changes; a text that would not come back identical fails and rolls back the whole batch. - A value that is not text stays inline and is counted as protected. - A crash or a stop leaves committed batches new form and the rest old form; both read the same, and a rerun continues. No media file is opened. - Unreferenced documents are swept at the end. The maintenance preview gains a read-only workflow_storage plan (records to rewrite, inline bytes where SQLite has octet_length, the Reclaim space and older-version notes). reclaim_space sweeps, runs VACUUM and truncates the WAL so the freed pages go back to the disk. * feat(maintenance): store each workflow once as an Activity job The reviewed maintenance plan gains workflow_storage. Applying it starts "Store each workflow once": kind maintenance, records admission, cancellable, groups changed/protected/errors. - The promised verified safety backup comes first; a failed backup ends the job with nothing rewritten. The apply's up-front disk check covers it. - Old-form rows are rewritten batch by batch on the event loop's executor, with progress after each committed batch. Cancel stops between batches; a failed batch rolls back whole and fails the job with a coded message. Either way every record reads as before and the next apply continues. - When every row was looked at, "Reclaim space" starts: exclusive (runs alone), not cancellable. It checks the disk for VACUUM's copies of the live pages, then sweeps, runs VACUUM and truncates the WAL. If another job holds the registry it does not start, and the finished migration says so and names Optimize database. - No image or video file is opened; tests track every open under the output folder and compare the media bytes and mtimes. * feat(maintenance): document the workflow storage contract TESTING.md's maintenance contract gains the workflow storage rules: the documents table and its two forms, the one accessor, garbage in the same transaction plus the sweeps, the batched migration and Reclaim space, the media and memory guarantees, and that a deduped backup cannot be restored into an older version. A test covers the preview on SQLite before 3.43 (no octet_length, as in older Windows Python builds): the inline-bytes detail is left out. * feat(database): keep the migration safe row by row and beside other writers Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe: - M1: a batch also ends after 0.5 s, and the loop leaves the write lock free for 0.15 s after each committed batch (longer than SQLite's longest busy poll), so the image monitor and run counting on another connection get in between. RED: a threaded writer with a 2 s busy timeout hit "database is locked" (4 of 5 links landed). - M2: each row runs in its own savepoint. A row that cannot be rewritten (undecodable text, a text that would not come back) is rolled back alone, stays inline and is listed in errors; the rest of the batch goes on. - M3: a reused document is decompressed and checked before its id is handed out, and repaired from the text (whose hash is its key) when damaged; verify_database_file reads every document and refuses a damaged one. - M5: the two legacy generated_images rebuilds refuse a file that already has the document id columns instead of dropping them. - The preview names the pre-migration backup and says it is always taken. * feat(maintenance): always back up before storing workflows once Final review fixes I1, I2 and I3 for the workflow storage dedupe: - I1: the migration always takes a verified backup, whatever maintenance.safety_backup says; the plan reports backup.policy "required" and the disk check counts it. - I2: that backup is prompts.db.pre-workflow-storage_<stamp>, outside the pruned prompts.db.backup_* pool and kept until the user removes it; the job's last line and the preview name it, and the legacy restore accepts it. README's changelog warns that an older version opens and restores a rewritten database but shows no workflows, and says how to go back. - I3: tools/workflow_text_digests.py writes the sha256 of every record's stored texts (read-only; inline before the rewrite, through the accessor after), so a diff before and after proves byte identity. - A row the migration could not rewrite is listed in the job's errors and named in its last line, and Reclaim space still runs. * feat(database): fail the batch on system errors and name a damaged document Re-review fixes R1 and R6 (database side), plus R3's helpers: - R1: a row is kept inline only for its own cause: text SQLite cannot decode, or a text that would not come back. A full disk, an I/O error, a malformed database or a bug now fails the batch and rolls it back, instead of listing every remaining row as "could not be rewritten". - R6: verify_database_file names the damaged document and says what to do (restore a backup from before the damage, or store an image with the same workflow, which repairs it). - R3: pre_migration_backups lists the kept backups, earliest first; the plan says "Nothing to rewrite" and names an existing kept backup, and the older version note says new images use the new form from the first start and names the earliest backup as the way back. * feat(maintenance): one kept pre-migration backup and nothing-to-do refusal Re-review fixes R2, R3, R4, R5 and R6 (job side): - R3: Apply refuses a plan with nothing to rewrite (409 nothing-to-do, no job, no backup). Only the first apply keeps a pre-workflow-storage backup; later applies (after a Cancel, a rerun) take a pooled safety backup, forced on whatever the policy, so one copy is never pruned. - R4: a second-connection writer test runs against the live job loop; with its pause removed the writer hits "database is locked" (4 of 5 links). - R5: the digest tool loads only database/workflow_documents.py by path (no package start-up, no log handler) and opens a stopped database with immutable=1, so it leaves no -wal/-shm; a live one is read through its WAL. - R6: a backup that fails on a damaged document says which one in the job line. - R2: README's changelog says images recorded by this version use the new form from the first start, and the way back is the earliest pre-workflow-storage backup (or a pre-update backup if the job never ran). |
||
|
Vito
|
f68e92b3b6 |
feat(system): maintenance backend (#313)
* feat(maintenance): reviewed, read-only maintenance plans POST /prompt_manager/maintenance/preview reviews duplicate_prompts, optimize, unused_components or missing_links on its own mode=ro snapshot and answers a Plan (scope, protected reasons, backup policy, media, at most 50 detail rows). The fingerprint, targets and judged sources stay in a server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB, never truncated; a restart forgets them), bound to the database it read. The previews reuse the legacy queries: duplicate groups and the merge are shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the #239 protection predicates the plan reports, and missing links are judged by the health check's link_states. Legacy /maintenance keeps its shape; vacuum now reports success:false when VACUUM fails, and cleanup_orphaned_images skips links under unavailable sources. * feat(maintenance): apply reviewed plans as Activity jobs POST /prompt_manager/maintenance/apply {token} starts a maintenance job (records admission, not cancellable, groups changed/protected/errors). The token is consumed on the loop turn the job starts: a second apply is 409 plan-used naming the job, a busy refusal keeps the token, an unknown or expired one is 410 plan-expired, and a restore since the review is 409 storage-changed. With the safety backup policy on (maintenance.safety_backup, default true, saved through the settings route with the revision), the job first writes a backup into a temp file beside the database, makes it a standalone rollback-journal file, verifies it, fsyncs it and renames it onto a unique <db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file and nothing changed. The apply then rechecks the reviewed fingerprint inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of 400 ids, counts changed and protected records, logs the backup by name only and prunes old safety backups. With the policy off the log and the status line say "No safety backup". * feat(maintenance): find and remove duplicate files through Activity Duplicate image files move onto the job registry: - POST /prompt_manager/jobs/duplicates finds content-identical media files over every effective source as a cancellable read job ("Find duplicate image files", group groups_found); the server holds the groups per job and GET /prompt_manager/maintenance/duplicates/{id} pages them ten at a time. - The duplicate_files preview binds a removal to the reviewed keepers. Every keeper and removal is checked again: containment as text before any filesystem access, then not a link, still inside its source, same size and hash. The plan lists the database links that move. - Apply runs duplicate_cleanup (records, not cancellable): the promised safety backup, then per file a recheck, the unlink with every handle closed and its thumbnail. A changed file is kept, the first failed unlink stops the job with a coded message (never the OS text), and only removed files have their image links and favorites moved to the kept copy. - The legacy scan, stream, status and delete routes keep their shapes on the same registry kinds; the hashing, grouping and thumbnail removal are shared with them. Coordinator rulings on Tasks 15-16: - Safety backups share the restore pool of 10 `<db>.backup_*` files; the pool is now also pruned when an apply fails after its backup. - Plans report the safety backup's size and the free disk space, and apply refuses up front (507 insufficient-space) when the backup or VACUUM does not fit, so no job fails halfway for space. * fix(maintenance): never remove the kept copy; move records before unlink Final review of #313: - C1: the duplicate find tells files apart by (st_dev, st_ino), so a source listed twice, a symlinked source inside another and a hardlink are one file. The preview and the job refuse a removal that os.path.samestat matches with its kept copy. - I1: each file's records move to its verified kept copy in one short transaction before that file is unlinked; a failed move deletes nothing and a failed unlink leaves the records on the identical kept copy. The status line counts files removed and not removed. - I2: links are looked up under every spelling of the source (the real path, the configured and monitored folders, ComfyUI's output). - I5: image tags join the kept copy as their union and the removed copy's pending, undecided Needs review items are settled; the plan says so. - I3: the VACUUM disk check measures SQLite's real temporary folder. - M1: the token is looked at again with no await before the job starts. - M2: the backup decision is made once and handed to the job. - M3: backups are linked into place; startup sweeps interrupted temps. - M4: Optimize runs alone (new "exclusive" admission) and its plan says the database is locked while VACUUM runs. - M5: the reviewed duplicate merge is strict: collisions and Library titles are protected, and errors roll back. The legacy cleanup is unchanged. - M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests. - M9: the preview binds files by size, mtime and inode without hashing; the job hashes right before each unlink. * fix(maintenance): protect one file reached through two mounts Re-review of #313, round 2: - R1: a removal on another device than its kept copy, with the same size and mtimes within 2 s, is protected when its inode is equal or its path within its source is the same ("May be the same file reached through another mount or share"), in the preview and again right before each unlink. The job holds the kept copy open during the unlink, so on Windows deleting the kept copy itself through another path fails. - R2: a file's spellings are found by containment: every configured, monitored, ComfyUI-output or listed folder whose stat identity is a folder above the file. Parent spellings of nested sources and case variants match; favorites move under every listed root holding the file. Lookups stay index searches. - m1: a failed unlink's row says the links now name the kept copy. - m2: a volume with no file ids reads "Can't confirm this is a separate file on this drive". * fix(maintenance): stat duplicate candidates with os.stat Python 3.10's Path.stat uses the os.stat bound when pathlib was imported, so the find and the per-file checks could see different stat calls. The find now calls os.stat directly, like every other check; the mount tests failed on py3.10 only. * fix(maintenance): recheck the kept copy right before each unlink Re-review 2 of #313: - n1: the cross-mount guard also matches when one path within its source ends with the other on a whole folder name (sources listed at different depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount). - n2: after a file's records move and right before its unlink, the held kept copy must still be the file at its path (fstat of the hold against os.stat) and the cross-mount guard must still hold; otherwise the file is kept, its records move back to it, and it counts as not removed. * test(maintenance): the kept-copy race tests hold on Windows On Windows the removal job holds the kept copy open without FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during the record move is refused. The tests now assert that native outcome: the kept copy stays and the removal goes ahead. * fix(maintenance): restore a backup of any size the disk has room for The legacy restore refused the maintainer's own 1.6 GB safety backup with "File too large. Maximum size is 100MB", a fixed cap since #153. Free disk space now decides: the upload, its staged copy and the swap must fit, under a 64 GiB ceiling. A request with a Content-Length that does not fit is refused before anything is written, one without as soon as its running size does not, with 507 insufficient-space and a message without paths. The upload still streams and its temp file is removed on every path. |
||
|
Vito
|
50703c960a |
feat(review): offer inspection-profile choices for untraceable batches (#312)
* feat(review): offer inspection-profile choices for untraceable batches Qwen image-edit batches run several KSamplers, each with its own TextEncodeQwenImageEdit prompt, all saved under one prefix. The scan's default profile knows neither the encoder nor the string nodes, so every candidate was unresolved and the files left Needs review as "could not be traced", although the inspection profile resolves every text. When the default profile is ambiguous without a choice and nothing links the file, the scan now resolves it with the inspection profile and, with at least two choosable candidates, queues the item from that evidence. The evidence records profile "inspection" and the semantics table version (SHA-256 of its canonical JSON); the item's resolver_version folds the profile, RESOLVER_VERSION and the table version, so a table change is handled like a resolver change (I2). Apply re-resolves the capture with the recorded profile, re-checks the chosen candidate hash and links through the compatible-only ingestion. The inspection profile never links on its own, ingestion keeps the default profile, and the live monitor never queues. * feat(review): supersede stale items and inspect each file state once A semantics table or resolver version change no longer leaves two pending items for one file. A new item supersedes the other items of the same bytes when every one is pending and undecided (revision 0, no decision row, the self-heal's safe delete). Beside a decided or advanced item nothing new is stored, and the rescan's stat shortcut skips those bytes under any version. The inspection item version now folds a narrower fingerprint: the table restricted to the graph's node classes plus its class-independent parts, so an edit about other classes leaves every item alone. The inspection profile runs once per file state. The scan resolves the untraced file at batch level, skips files already queued or decided at their current size and mtime under their version, and remembers files it found untraced in this process. The capture reuses the scan's result when it holds the same records. * feat(review): contain inspection failures, version the inspection resolver Final-review fixes for the inspection-profile review choices. - An inspection failure on one file is logged, counted in review_errors and reported unresolved; the scan goes on. - INSPECTION_RESOLVER_VERSION and the inspection profile's budgets are folded into the item version, so an inspection-only resolver change re-inspects and supersedes the item. An older version's pending, undecided item of bytes now untraced leaves the queue, so an item never stays stuck at 409. - Fingerprint tests: an edit of the encoder's own entry moves the version, an unused dict-keyed class leaves it. - The untraced memory is a plain dict of 16-byte digests (8.6 MiB measured at its 65,536-entry cap). - Only scans add items: the monitor's pass records nothing and only clears the items of outputs it links. Tests and the browser review fixture record through the scan's recorder. - record_review_item documents the byte-record refresh it commits when it refuses a new item. * fix(review): an unusable semantics table never retires recorded items |
||
|
Vito
|
6505d339aa |
feat(system): thumbnails on Activity (#311)
* feat(system): run thumbnail generation as an Activity job
GET /prompt_manager/images/thumbnails/plan counts what a run would do
(targets, fresh, remaining past the cap, source labels) and writes
nothing. POST /prompt_manager/jobs/thumbnails {mode, size, quality}
starts the "thumbnails" job: admission guarded, cancellable at a file
boundary, outcome groups generated / skipped / errors. size is the
longest edge (128-2048), quality the encoder, and rebuild writes fresh
previews again through the same atomic rename, so a failed file keeps
its old preview and originals are only read.
The legacy gallery stream now starts or attaches to the same job and
streams its unchanged frames. A client that goes away no longer stops
the work; the gallery's Cancel asks the server to cancel the job.
* feat(activity): per-kind job descriptors
One descriptor table (activity-kinds.mjs) decides each job kind's pill
verb, toasts, line, tiles, outcomes, openable rows and labels; the
store re-exports it. The scan and image tagging views are unchanged.
A thumbnail job counts Generated, Skipped and Errors, reads
"Generating thumbnails" in the pill and "Thumbnails finished · N
generated · M errors" when it ends.
Any other kind (the health check, a held legacy-page operation) gets
the generic descriptor with no tiles and no Results, instead of the
scan's zero tiles. A running job without a known total shows an
indeterminate progress bar and no percentage, and only a job the
server can cancel shows Cancel.
* feat(system): review and start thumbnail jobs
Sources' Thumbnail generation -> Review options and Overview's Review
thumbnails open the concept's Thumbnail maintenance dialog: scope, quality
(preselected from Image loading quality) and preview size limit, a plan
line from the read-only thumbnail plan (D99) and Start, which starts the
thumbnails Activity job and opens it. A size outside 128-2048 requests
nothing; a refused start shows the busy link toast.
* fix(system): thumbnail job review fixes (#249 S5)
- The legacy stream's first frame is job {id, started}, sent after the
subscription so no runner frame is missed; it reads the registry after
prepare, so a job started in that gap is attached to, not refused.
- Every exit sets a truthful final status line; nothing to do never
reads "Processing file 0 of 0".
- Cancel is checked per file during the planning walk; a job cancelled
there publishes no start frame.
- The ffmpeg fallback keeps the chosen quality (-q:v) and the aspect
ratio; the placeholder uses the chosen JPEG settings.
- A locked preview on Windows reads "Preview in use by another program;
the old one was kept", not the raw WinError.
- Thumbnail work no longer defers the monitor's Needs review recording:
it writes previews only, never records.
- The disconnect test now proves the job outlives its stream (one worker,
9 files, three batches); the attach test waits on an event.
* fix(gallery): cancel the stream's own thumbnail job by id
The settings panel and the startup dialog cancel the job named by the
stream's job frame, and only one their stream started. A Cancel before
the frame waits for it, a double Cancel posts once, and a job another
tab or System started keeps running. The startup dialog now settles on
Cancel instead of staying on "Cancelling...".
* fix(activity): indeterminate pill bar and the nothing-to-do line
The pill's mini bar sweeps while a job's total is unknown instead of
showing an empty fill. A finished thumbnail job with nothing to do shows
the server's final line. Coverage floors raised for gallery.js and
activity.mjs.
|
||
|
Vito
|
b4c7a15045 |
feat(review): Needs review page, live queue and concept comparison (#288)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* feat(review): bind queue cursors to the storage epoch only
A decision or a new item (the monitor records one per generation) no
longer makes an open cursor stale: the (created_at, id) keyset stays
ordered when items leave or are appended. A restore still answers
storage-changed and an unverifiable cursor review-conflict.
* feat(review): Needs review page, SAVED VIEWS row and Activity hook
The concept's Metadata review page at #/review: the notice with the
exact pending count, the queue's media cards (shared with Images through
image-card.mjs), and the Resolve the association panel with the sampler
context, the chosen path's prompt blocks, Leave for later and Apply
association. Apply takes a fresh capture; refusals keep the selection
and explain inline; an unanswered decision is resent once. SAVED VIEWS
lists Needs review after Favorites with its count (retires D50), and an
ended scan that recorded review items offers Review metadata in Activity
(D87). Queue items carry the card's media url.
* feat(review): re-read an open Needs review page when its queue changes
An open Needs review page now follows the queue without polling. A new
server-sent event stream, GET /prompt_manager/review/events, sends
"ready" when it opens and "change" after every committed queue change:
an item the scan or the monitor recorded, a spelling refresh, a decision
from any tab, and a restore. database/review_changes.py notifies after
the commit from whichever thread wrote it. The page re-reads its first
page on each event, keeps the open item, its choice and an unanswered
decision, and re-reads the SAVED VIEWS count with it. When the open item
left the queue, the next one opens with a note. A keepalive comment
every 25 s lets the server drop a closed tab; shutdown ends every stream.
Leaving the page mid-decision now releases that decision's capture.
D-NEW-F1 is renumbered D119 in markers and TESTING.md.
* test(review): Needs review page workflows
The review fixture now seeds a real monitor-fallback conflict (the
monitor pipeline links a new output to the most recent prompt, then
queues it), a limited item and a file under no current root, and can add
an output through the monitor or leave one for a metadata scan.
New workflows: Apply links the chosen pair to that file only; a real
association conflict leaves the links unchanged and explains; a limited
item and an unavailable file can only be left unresolved; a tag save
leaves the item pending; keyboard focus, labels and touch targets; an
open page shows items the monitor and a scan add and a decision made in
another tab; leaving the page releases captures, the change stream and
requests. The faked 409 conflict test is replaced by the real one.
* test(review): cover the card Tag button and raise the review floors
The card's Tag on Needs review now has a workflow, and the page's queue
change hook has no unused default. review.mjs and review-state.mjs
floors rise to their measured coverage; no other floor changes.
* feat(review): keep the loaded depth on live re-reads and read once per visit
A queue change re-read only the first 40 items, so the pages Load more had
shown vanished and the page jumped. Each re-read now reads page after page
until it holds as many items as were shown (a restore met after the first
page reads again from the start), so the cards and the scroll position stay.
The stream's ready made every visit read the queue twice. List reads now
answer the queue's version (library_epoch, review_revision) and ready names
the version read after subscribing; the page re-reads on ready only when the
queue it shows is older. The SAVED VIEWS count follows the page's total, so a
change costs no count read. A stream counts as open only once ready is sent.
* test(review): compare Needs review with the approved concept
review-concept.spec.mjs measures the whole Metadata review page as a tree
in the concept and in production, on the review fixture's new
concept-shaped queue (the concept's first three image names and sizes and
its first prompt). mapConceptReview applies only D29, D119, D3, production's
hooks and accessible names and sample data, and the page then matches at
1600x1100 and 390x844 in both themes. The gallery layout follows the
concept at every breakpoint, the SAVED VIEWS rows match with Needs review
after Favorites, the queue's Tag image dialog matches the needs-review
reference, and a dropped notice border or reordered actions fail.
The comparison found one gap: an unavailable file's card lost its media
area. Its empty media button now keeps the concept's 1.33 ratio (D119).
TESTING.md records the comparison, the extended D119 and a summary of the
image-tagging slice (routes, S1, capture limits, the record-changing job,
the no-download rule, the restart note). A ready whose version cannot be
read re-reads the queue (covered by a page test).
* test(review): give the multi-route shell comparison a fitting time budget
The dispatched CI run failed "shell geometry, tokens and controls match the
concept 390x844 light" on WebKit with a 30 s test timeout (no assertion
failed). Each of these tests makes twelve page loads with captures, hovers
and focus checks: 15-19 s on a CI WebKit runner on main, 22-31 s on this
run's loaded shard. With 2 CPUs and 3 workers it fails locally the same way
(6 of 12 timed out); with a 90 s budget, as the other multi-state concept
comparisons have, all 12 pass.
* fix(review): coalesce queue change events and answer queue thumbnails
- The change stream sends the first notice at once and collapses the
notices of the next 0.4 s into one trailing change, so a busy scan costs
an open page one re-read per 0.4 s at most (COALESCE_SECONDS).
- Queue items answer thumbnail_url, the same URL the Images listing builds
(one shared helper, media_thumbnail_url), or null without a thumbnail.
- The "duplicate and refused decision send nothing" test spies on
notify_review_change instead of reading the next event.
* fix(review): keep the live queue right through applies, Load more and lost streams
- A change heard while a decision is in flight is owed and read once the
decision is refused or unanswered (I1).
- Load more has its own read: during a re-read it waits and continues from
the re-read queue; a re-read started during Load more takes the click
over, and a failed re-read leaves Load more clickable (I2, M7).
- A stream the browser gave up on (503, proxy error) reopens after 3 s,
doubling to 30 s, with a quiet "Live updates are paused" line while it is
down (I3, D-NEW-F2).
- A hidden tab closes its stream and a visible one reopens it; the ready
version keeps that to at most one read (I4).
- Apply fences on the detail's epoch and a re-read under another epoch reads
the open item again (M3); leaving during the observe releases its capture
(M4); an unanswered decision that landed finishes as recorded (M5); cards
follow a re-read's url, target and availability and show thumbnails
(M6, M10).
- The unreadable-ready test asserts exactly one re-read (M1); the panel's
long-file-name wrap carries its D119 marker and TESTING.md entry (M8).
* docs(review): number the live-updates-paused deviation D120
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(activity): drop Cancel job from a job that had ended before Activity opened
Since #289 the Activity list removed an item's actions only when they were
isConnected. On the first render the item is not in the page yet, so a job
that had already ended when Activity opened kept its "Cancel job" (and a
running job's progress was moved after its actions, then back). The list
now asks parentNode instead. The review spec "a scan job with review items
offers Review metadata in Activity" failed on this before the change
(Expected 0 buttons on a finished job, Received 1) and passes after it.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* feat(review): open Needs review from the System Overview review row
Review metadata on the Overview's review row now opens #/review, the
concept's go-review, instead of staying aria-disabled.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): announce items that leave the queue, and seed only queueable files
#284 now queues a file only when its final outcome is ambiguous and unlinked,
and it removes the pending items of files that are now linked. That removal
is a queue change, so it calls notify_review_change after its commit, like
recording and decisions: an open Needs review page re-reads its queue, and
its count follows.
The review browser fixture wrote graphs that the scan's fallback links, and
it produced the conflict item through the monitor's fallback link. Neither is
queued any more. Its files now carry ComfyUI's Infinity stamp, which the
legacy parser cannot read, so they stay unlinked with the same two
candidates. The conflict item is queued first and its file is linked to
another prompt afterwards. "monitor" additions go through
record_review_file from a worker thread.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(review): seed the limited item from a graph with more candidates than stored
An A1111 block with two negative lines offers no candidate to choose, so
Needs review no longer queues it (#284). The fixture now records a graph of
17 samplers, which the recorder stores limited.
|
||
|
Vito
|
3f11ea6168 |
feat(annotations): Needs review queue backend (#284)
* feat(review): add review queue schema with restore validation
Add the Needs review schema group (media_review_metadata,
media_review_items, media_review_decisions) beside the image annotation
group it cites. review_schema_error validates the catalog, columns,
foreign keys, row values and canonical 8 MiB evidence read-only;
migrate_review_state installs it in a SAVEPOINT after the annotation
group, and verify_database_file requires it under require_library_state.
A review group without a valid annotation group is refused. Decisions
null their pair and image navigation through indexes when those are
deleted. Old backups and annotation-era backups restore and gain the
group; browser fixtures clear the new tables on reset.
* feat(review): persist ambiguous sampler evidence from scans and new outputs
The output scan and the live monitor now queue every file the ingestion
resolver finds ambiguous for Needs review. The file is captured, its metadata
is read again from the capture and resolved with the default profile, and only
an ambiguous result is stored, bound to the capture's annotation byte record
and fenced by its storage epoch. Evidence holds one hashed projection per
selectable candidate; none, more than 16 or over 8 MiB is a limited item.
Recording is idempotent per bytes and evidence and never touches a decision.
Scan statuses, counters, outcome rows and legacy SSE events are unchanged; the
job scope gains review_items/review_errors and the log a Needs review line.
The monitor records after its fallback link, best effort.
* feat(review): explicit association decisions with compatible linking
Needs review gains its queue reads and decisions:
- GET /prompt_manager/review/items lists pending items oldest first with
the exact pending total and an HMAC keyset cursor bound to the storage
epoch and the review index revision.
- GET /prompt_manager/review/items/{id} adds the candidates (role texts,
label, selectability), the dependency fingerprint and the digest.
- POST /prompt_manager/review/items/{id}/decision applies a candidate only
for a capture of the item's bytes that the current file still hashes to
and whose evidence is unchanged, through the ingester's compatible-only
rules in the same transaction as the decision row; a conflict writes
nothing. Leave unresolved records the decision and links nothing.
ingest_resolved_pair keeps its contract; its body moves into
_ingest_resolved_pair_in, which runs in the caller's transaction.
Items keep the file's recorded spelling so targets keep their case on
Windows, and a rescan skips the copy and hash of a file whose size and
mtime match the bytes an item already cites.
* fix(review): build the spelling validation test from native paths
On Windows the locator is case-folded and uses backslashes, so the fixed
POSIX pair /out/a.png no longer folded to itself and the new spelling
check refused the good case. The test now spells a mixed-case native path
and folds it with os.path.normcase, and also refuses a locator of another
file.
* fix(review): Apply sees the file under every spelling and rechecks only its choice
Apply now finds the legacy rows of the same physical file however its root
is reached. The roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, subst or a mapped
drive used to hide the monitor's row and Apply added a second, differently
linked row. Same-name rows are resolved to their physical identity
(normcase(realpath), else samefile); the dependency fingerprint covers all
of them, two rows refuse the choice, one row is enriched under the
ingester's compatible-only rules, and a same-name row the request did not
classify is a changed association.
Apply re-verifies only what the decision depends on: the bytes are still
ambiguous and still offer the chosen candidate. A resolver change that
rewords the reason or reorders candidates leaves pending items applicable.
The resolver version is one constant in prompt_pair_resolver, shared by
ingestion provenance and Needs review.
The transaction's byte check receives the re-hashed digest (or the body's
for Leave unresolved), every Apply refusal branch is tested with zero
writes, and the root listing and realpath calls run on a worker thread.
* fix(review): validate stored review evidence once per startup
Startup used to run the full review validation twice (the early guard and
the install) and the annotation group's row checks inside each, so every
evidence document was parsed and re-encoded twice. The early guard is now
catalog-only (a review group without annotation tables), and the install
validates the group once, trusting the annotation group it has just
validated. The evidence predicate no longer pattern-scans every document in
SQL, because the streamed canonical pass proves it, and the codec is built
once.
Measured on a database with 40,000 items of 4 KiB evidence: startup open
went from 7.7 s to 2.5 s. A test pins one evidence pass per reopen.
Restore and verify_database_file keep the full validation.
* fix(review): keep one capture worker for interactive requests
Needs review recording from the scan and the monitor shared the capture
store's two workers with the Tag dialog and Apply, so a busy scan could
answer a user's preview or decision with "Too many previews are open".
Background captures now never take the last worker or capture slot, and
wait up to two seconds for room before counting the file as not added.
Interactive captures are admitted or refused at once, as before.
* fix(review): refresh the recorded spelling when a file is seen under a new one
A case-only rename on Windows or macOS keeps size and mtime_ns, so the
rescan skipped the file and the item kept the old spelling. Its target was
then formatted from a name the directory no longer lists, the capture
refused it, and the item stayed pending with no way to apply it. The skip
path and a duplicate insert now record the current spelling for every item
at the locator and move the review index revision; item revisions do not
change, so an open decision stays valid. Tested under case-folding
semantics on every platform.
* fix(review): final capture store close and recoverable stale cursors
Closing the capture store at shutdown now runs under the store lock and is
final: a monitor recording or scan batch still in flight gets a capture
error instead of creating a new temp directory and restore subscription
after the cleanup.
A well-formed queue cursor this process cannot verify (opened before a
ComfyUI restart, whose cursor secret is per process) now answers 409
review-conflict with the current total, like any other stale page, instead
of an uncoded 400. Malformed cursors stay 400.
* fix(review): make only the shutdown cleanup final
The previous commit made every close of the capture store final, which
also refused the explicit closes that swap in a fresh store. Only the host's
on_cleanup hook is final now; an observe after it answers a coded capture
error instead of an unhandled 500.
* refactor(admin): move Needs review recording into its own mixin
_record_scanned_review, _record_review_sync, record_review_file and
_log_review_items move unchanged from py/api/admin.py to
py/api/review_recording.py (ReviewRecordingMixin), keeping admin.py from
growing further past the size ceiling. No behaviour change.
* fix(review): record the review fix round and the skip's residual risks in TESTING.md
The Needs review paragraphs now state the rescan skip's residual risks (an
in-place rewrite that keeps size and mtime_ns is never re-queued; a deleted
or moved file leaves its pending item counted), the physical identity rule
for Apply, the candidate-only recheck and shared resolver version, the
spelling refresh, background capture admission, the final shutdown close,
the stale-cursor answer and the measured startup validation cost.
* fix(review): refuse monitor recording beside record-changing jobs
System S1 (#289) gives every record change a registry admission class. The
scan already records Needs review items under its own records job. The
monitor's recording lasts one file and is not a job, so it now checks the
records admission: while a records or guarded job or hold runs it records
nothing, logs that the file waits for the next scan, and answers deferred.
The fallback link stands and the next "new" scan reads the ambiguous file
again, because a fallback link is not a resolved one.
JobRegistry.blocker and active iterate a snapshot of the jobs, since the
monitor's worker thread now asks them too.
* test(backup): wait for the restore, not a 5 s timer, in the thread test
test_database_is_usable_from_every_thread_after_restore let its worker
wait 5 s for the restore. On a stalled Windows runner (py3.11, where the
previous restore test also took 25 s instead of 0.3 s) the restore took
longer, the wait timed out, and the worker read the old database before
the restore ran. The worker now waits for the event under a 120 s bound
that only a hung test reaches, reports a timeout as an error, and the
main thread releases it even when the restore raises. With a restore
delayed by 6 s the old test fails and the new one passes.
* fix(review): store non-finite evidence as null and count pending items
canonical_evidence now follows main's non-finite policy (#303): NaN and
±Infinity are written as null instead of raising, so a NaN-stamped graph
never turns a review item limited. The System Overview reports the review
queue's pending total in needs_review.pending, read in the same snapshot
as the Library counts.
* fix(review): show the pending count on the System Overview review row
The Overview's review row now reads "N ambiguous image links" with the
concept's line, from needs_review.pending. Its Review metadata action
stays aria-disabled until the Needs review page ships. The system specs
and their concept comparison follow the real count.
* fix(system): answer the Overview and Statistics only after every read ended
asyncio.gather answered at the first failed read while its sibling
executor reads were still inside SQLite. In the suite the test teardown
then deleted the database under the running read, and py3.10 on Ubuntu
segfaulted in get_statistics (test_statistics_failure_is_an_error_envelope).
Both routes now wait for all their reads before answering.
* fix(review): queue only ambiguous files that nothing links, and heal the queue
The maintainer's live scan (19,908 files, scope "new") reported 4,677 files
under Needs review, but the queue held 10,232 items: every file the strict
resolver found ambiguous was queued, including the ones the legacy
browser-parser fallback then linked (all sampled "added" rows) and the ones
already linked to a known prompt (135 of 200 sampled "known" rows).
A file is now queued only when its final outcome is ambiguous and unlinked:
the resolver finds it ambiguous, the fallback links nothing and no saved row
links it already. An ambiguous file that a saved row links is reported as
unchanged ("Already linked to its prompt"), not as Needs review.
The queue heals itself. After each batch, the scan removes the pending items
of every listed file (read, or skipped as known) that a saved row now links.
The monitor does the same for each ambiguous output it links, and never
queues one. Only an undecided item is removed (pending, revision 0, no
decision row), so a decision is never deleted or changed and the schema
needs no new state.
The scan job's scope.review_items now counts this scan's files that are in
the pending queue after it, new or already there, so it agrees with the
review group's ambiguous rows and with the queue total. review_added and
review_settled are new, and the log gains "already in Needs review" and
"left Needs review · now linked" lines. A duplicate whose item is decided is
reported as "decided" and is not counted.
* fix(review): never queue an ambiguous file with nothing to choose
The maintainer's live queue held 39 items whose candidates all read
"unresolved" ("No positive / No negative"). An ambiguous result is now
reviewable only when at least two candidates are resolved with non-blank
positive text. With none the file could not be traced; with exactly one,
the untraced branches may still have made the file, so that text is not
certain either. Both are reported unresolved, never linked and never
queued (also the A1111 two-delimiter case, which has no candidates).
A rescan removes the pending items of such files whose own evidence
offers no choice, with the same safe delete as the linked settle
(pending, revision 0, no decision row). A limited item (over 16 candidates) that holds at least two
candidates is kept. The job scope gains review_untraced and the log
"= N images left Needs review · nothing to choose".
* test(scan): pin only the per-source scope keys beside the review counters
* test(system): match the scan scope keys beside the review counters
|
||
|
Vito
|
69de20a427 |
feat(system): Sources & scans (#309)
* feat(system): report configured, effective and watched sources
GET /prompt_manager/settings adds settings.monitoring_enabled, a sources
block (configured roots with valid/code/reason, effective roots with their
listing index and watch state, auto_detected) and the monitor state
(present, observer_alive, scheduled). Every path is a public path.
A source is "watching" only while a live observer scheduled it, "manual"
when monitoring is off or the process has no monitor, and otherwise
"restart_pending".
POST accepts monitoring_enabled as a strict bool, saved under
gallery.monitoring.enabled; restart_required_fields names it when the
running observer does not already match. A save that changes the sources
runs as a "Source change" records hold, so it is refused with 409 busy
while a scan runs. Case variants of one folder are saved once.
POST /prompt_manager/settings/validate-source {path} checks a path
lexically (no "..", no control characters, native path forms only) and
then against the gallery-root rules, answering a code and reason, and
writes nothing.
* fix(system): no existence oracle in source validation
A gallery root's containment in the allowed ComfyUI folders is now
decided before its existence and type, so a path outside them answers
"outside-allowed" whether or not it exists. validate-source also folds
the filesystem-root and home-directory codes into "outside-allowed", with
one reason, and logs the detail server-side only. Paths inside the
allowed folders still say "not-found" or "not-directory". The legacy
settings save shares the same check.
* feat(scan): scan one selected source
POST /prompt_manager/jobs/scan accepts roots: indexes into the effective
sources, strict non-negative ints (no bools or floats), deduplicated, at
most 64, non-empty when present. An index past the sources is a 400
"unknown-source" and starts nothing. The job walks only the selected
sources, resolved when it was requested, and keeps each file's listing
root_index. Its scope records files, roots (null for every source) and
output_folders, the number of folders walked, which the log's first line
counts. The legacy /scan route still scans every source, and attaching to
a scoped job keeps its payload keys.
* feat(system): manage sources and start scans natively
System > Sources & scans renders the concept's section on the server's
sources: one row per effective source with its watch state and a scan
scoped to that source, the 3.2.0 update notice with the legacy dismissal,
Add source checked by validate-source, Remove, monitoring and worker
threads as host drafts behind the one Save row, and Refresh files.
A save refused by a running scan names the job. Thumbnail generation
hands off to the legacy gallery until S5.
Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md.
* fix(security): decide source containment as text before any filesystem access
Every caller (validate-source, POST /settings, config.json at load) now
refuses a source through one check, sources.source_problem: the form
check first, then GalleryConfig.gallery_root_problem, which compares the
normalised spelling with the allowed folders before anything resolves,
stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a
device or any outside folder is refused with no SMB request, no
executor stall and no existence or timing oracle; only then is the path
resolved and contained again, which still catches a symlink out. The
filesystem root and the home directory answer outside-allowed with the
same reason as any outside path, so no caller learns the home path.
Also:
- POST /settings runs the form checks and parses the roots off the loop;
an unusable source is a coded 400 invalid-source naming its public
path (as text) and its index.
- ComfyUI folders outside the base directory are path anchors, so their
sources round-trip and never collide by folder name.
- A per-source scan sends each index with its public path; a stale or
shifted index is 409 sources-changed and starts nothing.
- A source still watched after monitoring was switched off is
"stopping".
- Roots are compared as folders, not spellings, so an unchanged legacy
resend takes no source-change hold.
- Tests: no realpath/stat/lstat of an outside path (native and Windows
forms with ntpath fakes), the alike answers with /, ~ and a link to ~,
an event instead of a timer wait, and a teardown that drains jobs
before the temporary database goes.
* fix(system): say what Sources saved, refused and still watches
- A per-source scan sends the source's public path; when the sources
changed since they were read the toast says the server's reason and
the list and saved values are read again.
- "watching until restart" for a source the running monitor still
watches after monitoring was saved off, and the status adds "Restart
ComfyUI to apply: Watch for new outputs." from restart_required_fields.
- A save refused for one unusable source marks that row and the status
is the server's message naming it.
- A valid configured source the effective list holds under another
spelling reads "same folder as another source".
- The busy toast names the job the 409 named (ApiError keeps a
validated running job and refused source), falling back to the list.
- The Add source dialog's listeners live on the page scope and Enter
adds; its markup is compared with the concept (D127).
- TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the
Sources contract describes the lexical containment, invalid-source and
sources-changed.
* fix(security): guard the Windows form probes on a token of their own
The no-filesystem guard matched "C:", which on a Windows runner is also
the temporary ComfyUI tree the save may resolve for its trusted first
entry. The probe paths now carry a token no trusted directory holds.
* docs(system): number the Sources states deviation D128
|
||
|
Vito
|
cbc53ee9f1 |
feat(resolver): trace multi-stage chains through pass-through nodes (#308)
* feat(resolver): trace multi-stage chains through pass-through nodes The inspection resolver (server and local Inspect) now compares a sampler with the earlier sampler its latent continues through image-space pass-through nodes: decode, image filter, model upscale, scale and encode, listed in a new validated `lineage` section of the semantics table. Equal prompts resolve (hires-fix, refiner); different prompts are multistage-differs. A later stage whose positive adds conditioning to the shared base prompt through an additive composite (ConditioningCombine or Concat) keeps the base pair, the text legacy linked (three-stage Flux). Image-edit batches stay ambiguous and run-time text stays untraceable. Ingestion keeps the default profile (D1). A card no longer shows a seed when the inspector names a chain stage whose seed differs from the parser's pick. * feat(resolver): leave out run-time PromptManager parts, labelled Maintainer option A: when a PromptManager prepend_text or append_text is fed directly by a node whose output is decided at run time (LoraManager's TriggerWord Toggle, now a runtime class, or PromptSearchList), the prompt resolves to the node's own text, as legacy showed. The side's source is runtime-parts-omitted and the resolved reason says that text added at run time, such as trigger words, is not included; the prompt block shows that line (D-NEW-R1). The toggle's output is never computed. Literal and traced parts are combined as before; an indirect run-time part, a run-time main text or an unknown feeder keep their outcome. The browser port changes identically. Ingestion is unchanged. * fix(resolver): keep run-time and added-conditioning labels honest A text that left out a run-time PromptManager part is now labelled when any node on its traced path omitted it, so the label survives encoders, concatenations and chained PromptManagers. Comparisons include what a text leaves out, so a partial prompt never equals a complete one: a branch with trigger words and one without stay ambiguous, and such stages are multistage-differs. Only classes listed as run time are left out. The added-conditioning floor labels its positive (D126, "Conditioning added at a later stage is not included."); the run-time line is D125. A card hides its seed only when the stage the inspector names reads a different seed than the parser; an agreeing seed is kept, as on main. The browser port moves its lineage, latent and part helpers into local-resolver-stages.mjs, keeping parity. |
||
|
Vito
|
7c25b351f7 |
test(comfyui): refuse a real-host port another run holds (#310)
The real-host config waited on the /system_stats URL, which any server on
the port answers. When two runs started on 8189 within a few seconds of
each other, the second run's ComfyUI failed to bind ("Port 8189 is already
in use"), but Playwright had already seen the first run's host answer and
ran the second run's tests against it. When the first run finished and
stopped its host, the second run failed mid-test with ECONNREFUSED,
ECONNRESET or ERR_CONNECTION_REFUSED, and passed when rerun alone.
The launcher now refuses a port that is in use, and reports ready only
when /system_stats names its own base directory. The config waits for
that line instead of the URL, and every test checks again that the host
answering the port is the one its run started.
|
||
|
Vito
|
785b744554 |
feat(system): Appearance section and browser display defaults (#307)
* feat(system): share browser display defaults The workspace reads and writes the legacy gallery's gallerySettings object through preferences.mjs: workspaceLayout and workspaceDensity join the shared keys, a write re-reads the stored object and writes only the changed keys, denied storage or a full quota keeps the change for this tab and says so, and storage events report changes from other pages. The legacy gallery Save keeps keys it does not edit, and the viewer reads its video defaults through the shared module. * feat(system): Appearance defaults used by Images and the viewer System > Appearance is the concept's Browsing & display surface on real values: the browser rows edit this browser's display defaults through the shared gallerySettings object, and Open from ComfyUI and the node search auto-hide save to the server settings with their revision. A new Images visit starts from the default layout and size (the current view and saved views keep theirs); metadata off waits for Load metadata; cache off re-reads the shown file; paths off hides the Path row; the viewer reads the video defaults. D97 adds card details, collapsed viewer details and Reset display defaults; storage refused says Saved for this tab only. * fix(system): flag changes made while System was closed and keep embedded galleries fixed Final review fixes for System S3 (Appearance): - I1: the on-demand metadata deviation is D124 (TESTING.md, inspector markers); TESTING.md names the spec that pins it, since no concept comparison renders that state. - I2: System checks the display defaults for conflicts on mount, not only on storage events, and a browser write refuses a draft another page overtook, so a draft is never saved over another tab's value without the conflict status first. New two-tab spec. - I3: the layout and size defaults start the Images page only; embedded galleries (a prompt history's Shared gallery and dialogs) keep Grid/Comfortable. - M1: host rows whose /settings read failed say "Server settings could not be read." - M2: a rejected timeout shows the "Some changes were not saved" status. - M3: the browser loads on mount and on storage events never reject unhandled. - M4: the next accepted write stores the tab-only changes and clears the overlay. - M5: cache off never re-reads a source-changed selection, and refresh() never loads a deferred one. - M6: the System fixture refuses a config.json outside its temp root and restores it with os.replace. - M7: the inspection read counts wait on a request barrier, not frames. - M8 recorded as an open saved-views question in TESTING.md. * fix(system): answer a failed System read only after its other reads finish CI (py3.10 on ubuntu) crashed with a segfault inside SQLite in test_statistics_failure_is_an_error_envelope: the statistics and overview routes used asyncio.gather without return_exceptions, so the 500 answered on the first failure while get_statistics still ran in the executor, and the test tore the database down under it. Both routes now await every read before raising the first failure (failing before, passing after: TestFailedReadsSettleFirst). The frontend coverage merge failed on workspace/activity.mjs branches (86.3 < 87): no test asserted Activity's empty and failed-read states, so their branch was only reached in passing, depending on timing. A spec now asserts both. |
||
|
Vito
|
c86739178d |
feat(system): System page shell, review dialog and Overview (#298)
* feat(system): read-only health check and truthful overview counts POST /prompt_manager/system/health runs the health check off-loop as a read registry job of kind "health" (one at a time, beside a scan, not cancellable, no file outcomes) and keeps the last report; GET returns it or null. The check reads one snapshot on its own mode=ro connection, closed in finally: integrity_check(100), foreign_key_check, stored hashes against utils.hashing, duplicate pair rows, the legacy consistency queries (now shared with check_consistency), and image links whose file is gone, counted only under readable sources. An unavailable source, or the absent Needs-review queue, makes its check unavailable and the report incomplete instead of producing a count. GET /prompt_manager/system/overview returns Library-eligible positive prompts and active negative variants, indexed images, the last health summary or null, backup null, the Needs-review count null, the effective sources with whether they were auto-detected, and the thumbnail plan's missing count. GET /prompt_manager/system/statistics lists runs and average rating per category of Library parents, with the /stats totals. * feat(system): native System route, drafts and shared review dialog - #/system/<section> renders the concept's System shell: section nav, settings layout and Save row with truthful status (D92); each section not delivered yet hands off to its legacy tools (D94). Overview's System card opens #/system/overview (Q12). - A module-level draft store saves browser, then host (expected_revision), then runtime; failed domains stay dirty, a stale revision is a conflict. - System row builders (setting, operation, section nav) on the scope. - openReviewDialog and backupNotice extracted from the scan review, which is now a thin caller with identical markup. - Overview images gain a "rated" count (images whose linked prompt is rated); the missing-files health check reports per source, so partial availability is visible. * feat(system): native Overview with health report and statistics System > Overview renders the concept's metrics and its Library health and Workspace preferences surfaces from GET /system/overview: real counts, "Not checked" until a health check ran, no invented review count, backup or thumbnail status (D93). Explore opens "Usage statistics" with the real runs and ratings per category and the Library totals (D96); the statistics route now also returns rated_images. Maintenance shows the concept's notice with Run health checks above its hand-off: the read-only check runs as a registry read job and opens the "Health report" with a chip per real result, missing files per source (D95). A load failure shows Retry and no zeros; late answers never repaint a newer section. Adds the System concept comparison (shell, Overview, both dialogs, the notice, mutation checks), the System workflow spec and its fixture hooks, and the TESTING.md "Native System" section. * test(system): coverage floors for the native Overview Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes. * fix(system): keep health reports per database and off the event loop - A restore that replaces the database forgets the stored health report, and a check that started before the swap is not kept (database generation stamped at start). - The Overview no longer walks every source for the missing-thumbnail count; the page reads /images/thumbnails/status on its own. - Image sources for the health check and the Overview resolve in the executor; a failure to start the check answers the coded envelope. - Any exception fails only its own check; missing files reads its rows in the snapshot and stats the files after it ended. - The read-only URI keeps Windows UNC paths (empty authority). - An integrity count at SQLite's limit is flagged as capped. * fix(system): thumbnail row read, tracked health job and guarded host save - The Overview's thumbnail row is its own read of the thumbnail status: "Checking thumbnails…" until it answers, unavailable when it fails. - The health job is tracked in the Activity store (new store.track) and followed through its event stream instead of polling every 150 ms. - A host save before a read returned a revision fails without a request; System reads the host values on mount and after a conflict. - "100+ problems found" when the integrity count is capped. - D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone. - The concept spec builds the health report from the concept's own rows and asserts the full unmapped path list; surface tops pin the 18px offset. - TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI and the new behavior. Coverage floors raised for the touched files. * fix(system): make two health tests portable to Windows - Windows forbids "?" in file names, so the URI-character open test uses "#", "%" and a space there. - ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of raising, so the bad-path test raises as POSIX does to reach the missing-files failure path on every platform. |
||
|
Vito
|
b50bf862f0 |
fix(inspect): hold a route change until a starting video settles (#294)
* fix(inspect): keep a replaced video in place until its pipeline has started WebKit (GStreamer) deadlocks the page when a video leaves the page or the viewport before its pipeline has started: the main thread changes the pipeline state and waits for the source's streaming thread, which waits on the main thread in a synchronous bus handler (native stacks of the hung WPE web process in the pinned image). Removing, hiding or merely pushing the video below the fold all hung; none of pause(), removing src, load() or the revoke was the cause on its own. A video replaced while still starting now stays where it is, inert and aria-hidden, with the new preview inserted after it, and the session's revoke of its URL waits. Once it reports metadata or an error (or after 15 s) it is paused, loses its src, is reset with load() and removed, and only then is its URL revoked. Every released video is reset with load(), so no decoder is kept per replaced video. The loadstart replacement spec now runs on WebKit too (15 fresh pages, replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause. * fix(inspect): hold a route change until a starting video settles A route change within the workspace detaches the Inspect page, and WebKit (GStreamer) deadlocks when a video still starting is detached. Route changes at loadstart + 0-7 ms hung 4-5 of 120 fresh pages (14 of 120 with no page code releasing anything, so the detach itself is the trigger). Inspect now exposes leaveGuard(): a promise while a video on the page is still starting (current or replaced). The shell renders the next route once it settles (metadata, error, or the 15 s bound); the latest route wins. Route changes at loadstart + 0-7 ms now hang 0 of 120. Not changed, with numbers in TESTING.md D62: - A replaced video still shows above the new preview until it settles. Same-slot stacking changes the video's own box and hung again (absolute 5/120, absolute + opacity 0 8/120, one-cell grid 4/120). - A real page leave cannot wait. Navigating away hung 8/120 and reload 22/120 even with no page-side release, so it stays a known limitation. |
||
|
Vito
|
4f1c6e236b |
test(browser): report a slow test reset with every thread's stack (#299)
* test(browser): report a slow test reset with every thread's stack On #291 (run 39904288262, WebKit shard 2/2) one /__test__/reset {chronological_history} waited 16 s on its worker's server, with nothing on the server's stderr. The cause did not reproduce: the same worker's file sequence (library, library editor, tagging lifecycle, visual) at 2 CPUs three times, and WebKit shard 2/2 at 2 CPUs, kept every reset under 2 s. A reset still running after 2 s now reports, while it waits, every thread's stack (faulthandler), the default executor's threads and queue, the open connections and WAL size, and on Linux each thread's scheduler state and kernel wait channel. When it ends it reports the total and how long each executor call queued and ran. The lines carry a "pm-slow-reset| " prefix and the worker fixture copies them to the job log, so the next one shows up even in a test that passes. The report never reads another thread's connection. Reading even in_transaction on a connection busy in SQLite blocks on its mutex with the GIL held: a first version of this report froze the whole server until the 30 s busy timeout. A test holds a real write lock behind a reset and checks the server keeps answering while it waits. * test(browser): name each thread in the slow-reset stacks Before Python 3.14 faulthandler prints thread ids only, so the report now lists which name each id belongs to. CI on 3.10 to 3.13 failed the unit test that looks for the blocked thread by name. * test(browser): keep the slow-reset summary on a coarse clock On Windows the report timer can fire a few milliseconds short of the threshold, and the summary was then dropped. A reset that was reported now always gets its summary. The server test uses the plain reset, which the Windows jobs can seed, and puts the server stderr in its failure messages. |
||
|
Vito
|
b2cc489fd4 |
feat(inspect): copy settings summary and local workflow roles (#286)
* feat(inspect): copy settings summary in the full inspector The legacy metadata page's COPY ALL (LI-4) becomes "Copy settings summary" after the six facts of the shared full inspector, so the Images viewer, the recorded history's current file and Inspect all offer it; the Images side panel keeps the concept's footer. The text is the legacy copyAllData template with its placeholders, filled from what the inspector shows. A unit test runs the template read from web/metadata.html on the legacy parse of every parity PNG: 22 of 25 are byte-equal, and the other three differ only by the ZeroOut guard (#249) or facts the legacy parse missed. Copying follows the concept's prompt copy: the toast "Settings summary copied.", or the select-to-copy dialog (now titled per copy) without a clipboard. The concept comparisons map the button as D-NEW-1. * feat(inspect): local execution evidence and prompt tracing A browser port of the server's inspection resolver for one sampler (PR L3 Task 8, LI-1). local-resolver.mjs answers inspect_prompt_evidence's {prompts, resolution} for a local read's records: execution evidence (read_pair_execution and its per-node match), every traced strategy (encoders, strings, concatenations, PromptManager and its historical classes, passthrough, zeroed, selector branches, composites, guiders), the latent stage walk, the R1 overwritten-text guard and the inspection profile's visit, depth, text, path, built and scanned bounds, with the same reason codes and outcome lines. py-json.mjs reads the records as CPython's json.loads does (dict order and last-wins keys, int versus float, BigInt past 2^53, NaN) and measures and strips text as str does. Several sampler candidates, detailer stages and A1111 parameters raise LocalResolverGap until Task 9. Nothing is wired into Inspect yet. Python is the oracle: test_local_inspect_parity compares the port with the server on the parity corpus, 107 named record sets, every work bound at its edge and 600 seeded fuzzed graphs, and requires gaps to be exactly the Task 9 paths. Sixteen porting mutations each fail the suite. The Node unit test replays committed server answers, which the Python suite keeps current. * fix(inspect): a list or object workflow node type answers malformed The R1 overwritten-text check looked every UI workflow node's `type` up among the subgraph IDs. A `type` that is a JSON list or object raised TypeError there, so the inspect route answered 500 "Prompt inspection failed". Such a workflow is now malformed evidence: the resolution is unresolved with reason_code `malformed` and the inspector's own "Prompt evidence is malformed" line, never exception text. The browser port answers the same, and the parity corpus gains the list, object, nested-subgraph and skipped-ID shapes. * fix(inspect): a list or object graph class_type is never a preview output With several sampler candidates the inspector first drops stages whose outputs reach only previews. That pass looked each API graph node's `class_type` up in the preview class set, so a list or object `class_type` raised TypeError and the inspect route answered 500. Only a string names a class now; such a node is an ordinary terminal, as the rest of the resolver already treats it. * feat(inspect): local candidate selection matching the inspector The browser port of the inspection resolver now answers every record set the server does, so no path is left to the server any more: - several sampler candidates, with the identical-branch rule (#270): a compare grid whose branches all resolve to one pair (text and recipe) is resolved, any other mix is ambiguous; - stages whose outputs reach only previews are dropped first; - detailer stages are judged with the samplers in their image ancestry (multistage-differs, cycles, depth and the per-stage visit budgets), and a sampler that only feeds detailers is not a candidate of its own; - A1111 `parameters`, with Python's multiline anchors, `\s` and Unicode `\d`, the bare footer and Template trailers, counted in code points. The parity corpus gains the selection cases and a seeded fuzzer, and every inspection vector now gives the server's prompts, resolution and facts locally. 16 of 17 mutations are caught; the uncaught one (not flattening an ambiguous upstream stage) cannot change any answer. * feat(inspect): local prompts resolved like the server's A file chosen on the Inspect page now shows the prompts and resolution the server's inspector gives the same records: buildLocalPayload takes its roles from the local resolver instead of the interim open roles, so the legacy fill applies exactly where it does for Images. The resolver reads the server's own node table. The page fetches web/js/inspection-semantics.json once through the API client as it mounts, before any selection; the static route serves that one JSON file by exact path (every other JSON stays forbidden). A table that cannot be fetched answers "The inspector's node table is unavailable." as the server does, and a later selection asks again. The Copy settings summary deviation is numbered D117. * fix(inspect): skip a workflow node whose type is a list or object The overwritten-PromptManager walk now skips such a node, on the server and in the browser port, instead of marking the whole file malformed. A prompt that resolves still resolves; the other nodes still feed the check. * fix(inspect): one JSON depth cap for both browser parsers py-json.mjs and local-video.mjs now refuse past the same 1,000 nested arrays or objects, counting every container as CPython's scanner does. Boundary vectors at 999, 1,000 and 1,001 for a deep prompt and a deep workflow run through the Python oracle, and TESTING.md gives the measured per-version server limits and both divergence directions. * fix(inspect): realpath containment for the exact-path node table route inspection-semantics.json now goes through the same realpath check as every other static file, so a symlink out of web/ is refused. The forbidden-path test asserts exactly 403. * fix(inspect): abortable, stepped local reads with a busy state The wait for the node table now ends with the selection's signal. A local read resolves and projects in bounded steps with a turn of the event loop between them, and the inspector carries aria-busy while it loads. A budget test covers maximal 2 MiB graphs and checks the stepped read is byte-identical to one synchronous build. The D117 wrap rule now targets only the summary button, and TESTING.md scopes the restart line to PRs L1 and L2. * fix(inspect): record the two CPython-JSON emulations as a follow-up |
||
|
Vito
|
fd747a0ffc |
fix(inspect): keep a replaced video in place until its pipeline has started (#291)
WebKit (GStreamer) deadlocks the page when a video leaves the page or the viewport before its pipeline has started: the main thread changes the pipeline state and waits for the source's streaming thread, which waits on the main thread in a synchronous bus handler (native stacks of the hung WPE web process in the pinned image). Removing, hiding or merely pushing the video below the fold all hung; none of pause(), removing src, load() or the revoke was the cause on its own. A video replaced while still starting now stays where it is, inert and aria-hidden, with the new preview inserted after it, and the session's revoke of its URL waits. Once it reports metadata or an error (or after 15 s) it is paused, loses its src, is reset with load() and removed, and only then is its URL revoked. Every released video is reset with load(), so no decoder is kept per replaced video. The loadstart replacement spec now runs on WebKit too (15 fresh pages, replaced at loadstart + 0-4 ms). TESTING.md D62 records the cause. |
||
|
Vito
|
a8aff3a39d |
feat(annotations): bulk Add tags and Auto-tag selected as Activity jobs (#283)
* feat(activity): image tagging jobs for bulk Add tags and Auto-tag
POST /prompt_manager/jobs/image-tags starts one record-changing
"image-tags" job over 1..500 listed images (202, or the same 409 busy
envelope as /jobs/scan). Add tags hashes each current file and unions a
fixed set into that byte record; Auto-tag captures each still image, asks
one legacy engine for its top N suggestions on the capture (count from
the options or autotag.defaults.suggestion_limit), re-hashes the file and
unions only onto the same bytes. Outcomes are grouped as tagged,
unchanged, skipped (video, too large, file changed, file not found, not a
still image) and errors; cancel stops at a file boundary.
Auto-tag holds the shared model gate for the whole job (409 model-busy at
start when it is held), loads the model once and never downloads one: an
unavailable model fails the job before the first file. The registry gains
blocker() so a start answers busy before taking the gate.
* feat(annotations): Add tags and Auto-tag selected in the selection bar
The Images selection bar gains the concept's Add tags and, beside it,
Auto-tag selected (D85). Add tags starts the image-tags Activity job with
the dialog's comma-separated names; Auto-tag selected reuses the Tag image
model panel (server defaults, Suggestions count) without Generate, refuses
a model that is not downloaded, and either starts the job (Apply
automatically) or walks the selection one Tag image dialog at a time
(Review each image, D90). Busy, model-busy and refused starts toast why;
the selection stays; a finished job's files are read again.
* feat(activity): show image tagging jobs with their own outcomes
Image tagging jobs get their own Activity view (D89): the pill reads
"Tagging", the item and heading line name the images and end with
"N tagged · M errors", and the job view shows Tagged, Unchanged, Skipped
and Errors tiles and outcomes, opens Tagged and Unchanged rows in the
viewer, and reads "Tagging log" and "Cancel tagging". The per-kind table
lives in activity-store.mjs; the scan's view is unchanged.
The selection bar's copy without a concept line is recorded as D117.
* fix(annotations): final review round for bulk image tagging
- Rename the D117 copy deviation to D118 (D117 belongs to another PR).
- The tagging job's progress line gives the scan's "about X min left"
estimate, with each finished file as one batch.
- The start body cap fits 500 targets of the longest legal path, even
escaped; one byte over answers 413.
- A capture store at capacity is retried with a short backoff, then the
file is skipped with a rerun sentence instead of a Tag-dialog error.
- Output roots and their labels are resolved once per job in the
executor, not twice per file on the event loop.
- Review each image holds Stop reviewing, Escape and x while a save is in
flight, and its primary reads "Save and next" when only the manual
field would be saved.
- An Add tags field that parses to no names keeps the dialog and its
text and reports the message on the field.
- Image tagging finish toasts carry the counts ("... finished · N tagged
· M errors").
- Specs: the bulk spec waits for the answered Start before checking that
the invalid press sent nothing; the Activity spec uses a real error row.
|
||
|
Vito
|
e29ff4d4b2 |
feat(inspect): the Inspect page for local files (#280)
* feat(inspect): local adapter seams in the shared inspector and viewer
The inspection loader calls adapter.read when an adapter reads by itself, so a
file chosen in this browser never reaches the API. The full inspector shows the
D66 local provenance (no Path row, no Show in folder, local association line),
and the viewer uses adapter.mediaUrl exactly, so a blob: URL is never given the
_pm_viewer cache-buster and Retry media reuses it. Rooted and recorded-history
adapters keep their api.get paths and cache-busted originals.
* feat(inspect): owned local selection session
createLocalSession owns one chosen File at a time: it classifies the bytes,
creates the blob: URL only for a classified image or video, and reads it
through a real inspection controller with the local adapter. Replacement,
dispose, parent abort and pagehide each release the URL, the handle and the
record once; a released handle (and every copy) is foreign to the adapter, and
every await is fenced by a generation counter so a replaced read never
publishes. Empty, multiple and directory drops keep the previous selection.
Baseline entry for local-session.mjs; branch floors raised for the touched
inspection-state.mjs and local-inspection.mjs.
* feat(inspect): native Inspect a file page
#/inspect is now the concept's "Inspect a file" page instead of the legacy
handoff. One chosen or dropped image or video previews inline, and the one
heading action, "Open file metadata", opens the shared viewer with the full
inspector on a record read entirely in this browser. Nothing is uploaded,
imported or stored; only an explicit Send crosses the host bridge.
- inspect.mjs mounts the concept DOM (surface, file-drop, #local-preview)
under the shell heading, owns one local session and the shared viewer and
prompt sender, and releases media before the session revokes its URL on
replacement, route change, page leave and dispose (D60-D65).
- A file that cannot be read after choosing opens the inspector's
unreadable state when the browser declares a supported type, instead of
the "unsupported" copy; the local adapter maps a browser read failure to
an unreadable record.
- The route is native; Overview keeps its card to the legacy metadata page,
which is unchanged.
- workspace-inspect.spec.mjs covers the chooser and viewer rail against the
Images rail for the same bytes, drop, cancel and reselect, unsupported and
unreadable files, video, URL ownership, mid-read replacement, exact send
and the no-request/no-storage rule.
* test(inspect): concept parity and managed-host send for local files
Compare the native Inspect a file page with the immutable concept's
#metadata at 1600x1100 and 390x844 in both themes (initial, selected,
viewer-open), mapping D60-D66 and pinning the concept's CSS values and
the chooser's displayed file name, with a mutation check. Send a local
PNG (explicitly empty negative) and core-tags.webm to a real Prompt
Manager (+/-) node across Classic and Nodes 2.0 switches and reloads;
a connected input rejects the send unchanged. Document D60-D66, the
legacy-floor dispositions and the commands in TESTING.md.
* fix(inspect): reset a released preview video in the next task
On WebKit (GStreamer) a video replaced while its pipeline was still
starting blocked the page: pause, src removal and load() in one task
never returned, so the next chooser selection hung (CI page workflows
webkit shard 2/2, "URLs and media end with their owning session").
The src is still dropped before the URL is revoked; only the load()
reset moves to the next task. A fresh-page regression replaces the
first video as soon as it shows, fifteen times.
* fix(inspect): never reset a preview video that is still starting
Deferring load() to the next task was not enough: CI on
|
||
|
Vito
|
195a1d33ac |
feat(annotations): Tag image dialog, card Tag button and inspector tag chips (#281)
* feat(annotations): image tagging controller with staged suggestions Add the UI-agnostic Tag image dialog controller (#249 C1): observe the file into a capture, prefill the saved tags and the server's tagging defaults, stage the top suggestions as removable chips, and save manual tags plus the remaining chips with case-insensitive dedupe (manual spelling first). Every read is serial-guarded; saves are sent once and a lost answer is resent identically only by Retry. Conflict, storage, file and capture outcomes keep the draft and gate Save on a reopen. ApiError now carries a validated copy of a tag-conflict body's current tag set so the dialog can show it. * feat(annotations): Tag image dialog beside Info on every image card Every Images and Library-embedded gallery card shows the concept's Tag after Info. Tag opens the concept's Tag image dialog (image-tagging.mjs over the C1 controller) in the shared modal shell with the sticky tagging-modal footer: capture preview, manual tags, the four legacy engines with real defaults, explicit Generate, removable suggestion chips (D83), legacy model labels (D84), a Suggestions count (D91) and Retry where the controller allows it. The capture is released on close, route change and pagehide. The Tags page gains the concept's Auto-tag images for the image selected in Images (aria-disabled without one, D115) and real Images counts that follow every save. Retires D9, #245 D2 and #245 D3. The toast now uses the mock's copy verbatim ("tags" even for one), C1's D60/D61/D68 markers are renumbered to D83/D84/D91, and at extreme zoom a card's info row stacks (and wraps Info and Tag) when its row is narrower than the narrowest bounded card, list rows included (D31). * feat(annotations): show image tags in the inspector and viewer The inspector parses the B3 envelope's image_tags facet (strictly: exact keys, current/none/unavailable, at most 100 vocabulary names) and draws one chip row after the prompt blocks: the file's current image tags first, then D21's linked tags not already shown (case-insensitive). In the Images panel image tags are Tags filter buttons and linked tags keep D21's rule; in the viewer and wherever no Tags filter exists every chip is plain. none and unavailable add no chips (D88, plan D65; D21 amended). The tagging fixture reset now drops leftover captures, so a long run of tagging tests in one worker stays below the capture store's limit. * fix(annotations): re-read a tagged file in place after a save After a tag save the selected file's inspection is re-read quietly: the panel and the viewer keep the current details until the new answer lands, with no "Loading file details…" in between. While a Tags filter or a search is active, Images re-reads its loaded range in place (same query, from the first page up to the loaded count, published once), keeping the cards, the scroll position and the selection while the selected file still matches; when it no longer matches, the panel follows the collection. * fix(annotations): give suggestion chips the concept chip corner The D83 chips took the later `.tag-suggestion` row rule's 5px corner; the concept `.chip` has 4px. The concept comparison of the Tag dialog found it. * test(annotations): compare the Tag dialog with the approved concept The Tag image dialog's inventory (order, text, field attributes, tokens, preview box, chips, status, sticky actions) must equal the concept's at 1600x1100 and 390x844, dark and light, opened, generated with WD14 and with a language model, for an empty result and in the review scope, and the card's Info and Tag group must equal the concept's. Production runs on the concept fixture with the concept image's tags saved and the fake model answering the concept's own sample; only D29, D83, D84 and D91 are mapped, and a mutation check shows the comparison notices a change. The D3 footer check again asserts a strictly shorter footer, with short seedless subtitles that stay on one line beside Info and Tag. * fix(annotations): final review round for the Tag image dialog - Option fields show the server defaults that Generate sends: the controller bumps an options revision on open and on a model change, and the panel re-renders only then (I1). - A tag conflict shows the current set at once ("Saved now: ...") and puts it ahead of the draft in the manual field, so the next Save sends what is shown (I2). - The opening sentence is the concept's on-screen copy; "Remove anything you do not want to add." leads the chips only while chips exist; the comparison's opened mapping is gone (I3). - ApiClient gains an additive keepalive option; the capture release and the Save PUT use it, and pagehide closes the dialog (I4). - Closing during a save keeps the capture until the PUT settles, then releases it and still reports the outcome (toast, page refresh) (I5). - A refresh asked during a page load is pending and runs once it settles; re-reads use pages up to 200 files; a failed re-read toasts; Load more waits for a refresh (I6, M2, M3). - The selection bar drops checked files a refreshed result no longer lists (I7). - D116 records Retry; after a failed Generate it takes Generate's own path, and after a changed storage, file or expired capture it observes again keeping the draft (I9, M4, M5). - A save made from Tags marks a filtered Images visit stale (M6). - images.mjs drops under 800 lines: the tagging wiring moves to images-tagging.mjs and the favorites star to images-favorites.mjs (unchanged), and reset() reuses dropSelection() (M1). - Restore the activity-scan.mjs coverage floors; drop the duplicate paired captures test (I8, M9). * fix(annotations): coverage floors for the Tag dialog round Floors for the new images-favorites.mjs and images-tagging.mjs, and the measured raises of image-tagging-state.mjs and image-tagging.mjs. activity-scan.mjs keeps main's floors. |
||
|
Vito
|
1b8fbdbb75 |
feat(annotations): image tag routes, model gate and scored suggestions (#278)
* feat(annotations): add scoped observe, capture preview and tag save routes
POST /prompt_manager/image-annotations/observe captures the file named by
the listing identity {root_index, root_scope, path} (scope checked against
the current roots before any file access) and answers the saved tag set of
exactly those bytes, refreshing a matching record's verified fingerprint.
GET/DELETE .../capture/{token} serve the private copy (no-store, nosniff,
pinned with acquire/release_use while it streams) and release it.
PUT .../tags re-hashes the current file and writes only when it still
holds the captured bytes, under the storage epoch and tag revision
(file-changed, storage-changed, tag-conflict, tag-limit, replayed).
Bodies are capped at 64 KiB with unknown fields refused; envelopes never
carry server paths or locators. Restores invalidate every capture and the
host's cleanup removes the capture directory.
* feat(autotag): scored suggestions with a configurable count behind a shared model gate
- py/model_gate.py: one process-wide, non-blocking gate for loading,
swapping, running, downloading and unloading a tagging model. Worker
calls run shielded; a release requested while one still runs takes
effect only when it exits (client disconnect keeps the gate).
- Legacy /autotag/single, /start, /download/{type} and /unload hold the
gate and answer 409 {code: "model-busy"} while another operation holds
it; their bodies are otherwise unchanged. /autotag/models gains
suggestion_limit.
- AutoTagService.generate_tag_suggestions: WD14 (name, probability) best
first, GGUF/HF (name, None) in output order, first `limit` distinct
case-insensitively; generate_tags keeps its return; the image file is
closed before inference returns.
- POST /prompt_manager/image-annotations/suggest runs one engine on the
pinned capture, with strict per-engine options, never downloads
(409 model-unavailable) and never writes.
- config.json autotag.defaults.suggestion_limit, integer 1..100, default
10, validated on load.
- Capture probe decides Generate from the image header alone, so prompt
metadata over the inspection budgets no longer blocks it.
* feat(images): match image tags in the Tags filter, search and inspector
The workspace Images listing now admits a file under a Tags filter when
every requested tag is carried by its linked positive or its current image
tag set, and q also matches image tag names (case-folded substring). Image
tags count only while a byte record is verified at the file's present
(size, mtime_ns) (spec amendment S1): the tagged records are streamed
once, only listed files carrying a filter tag or q hit are stat'ed, and
their linked positives are read through the file_path index so tags,
category, rating and the positive count combine.
The rooted inspector envelope gains a validated image_tags facet
(current / none / unavailable) read at the inspection's own stable stat.
A failed image-tag read fails a Tags filter like a failed link read and
degrades a q search to no image tags; plain listings read none. Legacy
gallery and search responses are unchanged.
* fix(annotations): answer a fixed coded message when the filtered image list fails
A failed prompt-link or image-tag read under a Tags, category or rating
filter returned the raw exception text (SQL or database messages) in the
500 body. The workspace listing now answers code listing-failed with a
fixed sentence and logs the traceback. The legacy output listing and the
inspector image-tag read now log with exc_info as well.
* fix(annotations): drop suggestion names that Save would reject
A JoyCaption part over 50 characters or holding a newline came back as a
preselected chip, so the first Save of the dialog failed with a 400 and
nothing was stored. generate_tag_suggestions now applies the vocabulary
tag-name rules before the case-insensitive dedupe and the limit count, so
the requested number of valid chips still comes back.
* fix(annotations): stop changing the process warnings filters during capture
header_dimensions wrapped Image.open in warnings.catch_warnings with an
error filter for DecompressionBombWarning. catch_warnings is not
thread-safe, so two overlapping captures could leave that error filter
installed for every later Pillow open in the ComfyUI process. Only
Pillow's DecompressionBombError is now mapped to HeaderTooLarge; a size
that only warns is refused by the existing inference pixel cap.
* fix(annotations): free pins on cancellation, serve preview ranges, code every storage error
- A preview cancelled while its file opens closes the handle and only
then ends the pin, so a doomed capture is never unlinked while open.
- An observe cancelled while capturing or reading releases the capture
instead of leaving it until the idle expiry.
- The model gate takes an on_skipped callback that runs once when the
work never starts; Generate passes the capture's release_use, so an
executor refusal no longer leaks a pin against the capture budget.
- The preview answers a single byte range with 206 and Content-Range
(416 for an unsatisfiable or multi-range header), still streamed inside
the pin, so Safari and WebKit can play video captures.
- _coded maps tag-limit (400), metadata-conflict and vocabulary-conflict,
and falls back to 409 with a fixed sentence for any other code instead
of raising KeyError.
* fix(annotations): deny case and Unicode spellings of a listed file
On macOS (case- and normalization-insensitive APFS) a crafted target such
as A.PNG or an NFD name opened the listed a.png or NFC file while
realpath kept the requested spelling, so the tags were stored under a
second locator the Images filter never shows. Capture and verify now
require each path part to be spelled as its directory lists it and deny
any other case or Unicode spelling as an alias (409 source-changed, like
a symlink alias). A missing file with no other spelling keeps its usual
outcome.
* test(annotations): one shared folder_paths stub for the API test modules
test_api_images and test_api_prompts installed a MagicMock folder_paths,
so whenever one of them was imported before test_api_autotag the later
patch("folder_paths.get_output_directory") in test_api_history_outputs
left a mock attribute behind and two test_diagnostics_config tests
failed. tests/host_stubs.py now installs one real-module stub, used by
test_api_images, test_api_prompts and test_api_autotag (and through
test_api_images by the annotation modules, whose copy is gone), so the
modules pass in either import order.
* fix(annotations): derive the suggestion cap from the image tag set cap
py/autotag.py now takes MAX_SUGGESTION_LIMIT from database.annotations
MAX_IMAGE_TAGS. The two copies that cannot import it (py/config.py loads
before the database, utils/inspection_response.py stays storage-free) are
pinned equal by a test, so the 100 limits cannot drift apart.
* docs(testing): record capture aliases, preview ranges and the PR B follow-ups
TESTING.md now states that case and Unicode spellings are aliases, that
the preview serves byte ranges inside the capture pin, where the
cancellation tests live, and two tracked follow-ups: the legacy batch's
ungated cleanup unload under its own last inference, and the three files
over the 800-line ceiling (split py/autotag.py first).
|
||
|
Vito
|
0095db8cbc |
feat(inspect): local evidence readers for images and videos (#277)
* feat(inspect): bounded browser PNG reader matching the server's
The Inspect page will read a chosen file entirely in the browser, and the
record it shows must equal what Images shows for the same bytes. This adds
to web/js/png-metadata.js (loaded by the legacy pages as before, with
parsePngTextChunks unchanged):
- readPngMetadataFile(file, {signal, prefix, yieldControl, decompress}):
a port of utils/prompt_inspector.py's _png_metadata, _PngRecords,
_png_text, _inflate and _batch_normalized. It slices only chunk headers,
80-byte keyword prefixes and the retained prompt, workflow, parameters
and comfyassets_prompt_pairs records, CRC-checks what it keeps, applies
the same budgets (2 MiB shared role budget encoded and decoded, 4 MiB
workflow that becomes limited, 8 MiB and 4 MiB duplicate budgets, 4 MiB
duplicate decoded parameters, 10,000 chunks, the batch-index rule for
parameters copies) and compares duplicate chunks byte for byte. Every
range is checked against file.size before slicing and charged against a
19,794,389-byte ceiling, so a multi-gigabyte PNG costs only its metadata.
Compressed text inflates through DecompressionStream with one 256-byte
write in flight and the output capped at the record's allowance; an
AbortSignal rejects the read with an AbortError and leaves no unhandled
rejection; yieldControl runs about every 8 ms of work.
- classifyPrefix(bytes): png, jpeg, webp, iso-bmff, matroska, avi, asf or
null from the first 32 bytes.
- readImageHeader(file, kind): JPEG (SOFn walk as Pillow) and WebP (VP8,
VP8L, VP8X) dimensions from at most 64 KiB.
tests/test_local_inspect_parity.py drives tests/js/support/local-inspect-
oracle.mjs over the committed and test-time inspection parity corpus plus
compressed, oversized, Latin-1, batch-duplicate and all-four-record shapes,
and asserts records, record formats, dimensions and readability equal
read_inspection_file on each file; the two JPEG/WebP fixtures are checked
against Pillow. tests/js/png-local-reader.test.js pins each server rule,
the sparse 3 GiB read, the prefix reuse, inflation back-pressure and abort.
tests/fixtures/local-inspect/tiny.{jpg,webp} are CC0 FFmpeg test patterns.
* feat(inspect): bounded browser video tag readers
Add web/js/workspace/local-video.mjs: ISO-BMFF (MP4/MOV/M4V) and Matroska
(MKV/WebM) walkers that read only the container tags the server asks ffprobe
for, slicing box and element headers, carrier names and retained values within
the server's budgets and never a byte inside mdat or a Cluster; a port of
utils/video_metadata.select_video_carrier with the record budgets of
read_inspection_file; and a byte-exact json.loads/json.dumps round trip so
envelope objects reserialize as the server spells them.
Every naming rule mirrors ffprobe 9.0.2 on hand-built files: legacy ilst atoms,
mdta keys, joined repeats, data types, Matroska language and default handling,
and Tags behind a Cluster found only through a SeekHead.
Tests: node units over byte-exact container builders and the committed
fixtures; a Python parity test comparing the local carrier with the server's
selection from the same tag maps (mocked probe) and from the same files (real
ffprobe, including 1 MiB + 2.5 MiB per-key records); an ffprobe
characterization of the recorded fixture maps.
* feat(inspect): server-identical local projection and payload
Port utils/inspection_response.py to the browser: a strict parse with Python's
Decimal spelling, duplicate-key rejection, NaN/Infinity as null and integers
past 2^53-1 as text; the prompt, workflow and carrier projection with the
depth-64 and 20,000-node ceilings; _metadata; and the 8 MiB omission steps
(workflow graph, then prompt graph, then all raw) measured as UTF-8 bytes.
local-inspection.mjs builds the inspection_version 1 wire for one chosen File
from the Task 1 and Task 2 readers (observation selected-local-file, interim
unresolved roles, unavailable library), fits it to the response budget, and
exposes localFileAdapter for the shared controller, inspector and viewer:
handles are session-owned by a private token, so a tampered or hand-made
descriptor is rejected and blob: URLs are never resolved through the API
client. projectInspectionPayload admits selected-local-file only for a
local-file identity.
A Python oracle test compares projectMetadata with _metadata byte for byte over
the parity PNG corpus, the committed video fixtures and tag maps, 75 committed
adversarial record sets (projection-cases.json) and node-ceiling sets built at
test time.
* test(inspect): import oracle modules by file URL for Windows
The local-inspect oracle imported product modules with path.join(...), which
Node parses as a URL scheme on Windows (D:\... -> ERR_UNSUPPORTED_ESM_URL_SCHEME),
failing every tests/test_local_inspect_parity.py oracle test on the Windows CI
job. Every dynamic import in tests/js/support now goes through moduleUrl, a
pathToFileURL wrapper, and a node unit proves the Windows form on any host and
scans the oracles so a path import cannot return.
* fix(inspect): read video tags as ffprobe reports them
Same bytes now give the server's record in five shapes where the local
reader was stricter or kept different tags (PR #277 final review I1, I2,
M1, M2, M4):
- Invalid UTF-8 in a tag value is replaced as ffprobe's default
string_validation=replace does (av_utf8_decode, one U+FFFD per failing
decode step), instead of failing the read. Key names and TagNames that
do not decode are no carrier and are skipped.
- An ilst item whose keys index is 0 or out of range is skipped, as
ffmpeg does, instead of failing the read.
- Only the first keys box is read, as ffmpeg does.
- A first per-key workflow over 4 MiB keeps its place in the carrier
choice as an unread marker, so the server's carrier (per-key, workflow
limited) is chosen; when the server's first probe would overflow, the
re-probe leaves it out and its own bound applies.
- An envelope integer of more than 4300 digits fails the Python json
port, so that envelope is skipped as CPython's json.loads skips it.
Each shape was checked against ffprobe 9.0.2 and read_inspection_file;
the real-ffprobe parity test now builds them, and a 600-case random-byte
comparison of the replacement against ffprobe matched.
* perf(inspect): measure the local response size natively and yield between steps
fitInspectionPayload counted the UTF-8 size of an 8-12 MB wire with a
per-character loop and ran every omission step on the main thread in one
turn (PR #277 final review M7). The size is now counted by TextEncoder
encodeInto through one bounded scratch buffer, and each re-measure
follows a yieldControl turn, with an abort signal checked between steps.
The fit is async; the fitted payload is the same as before.
* fix(inspect): hold video tags as ffmpeg's dictionary does, decode Mac Roman text
Three more same-bytes divergences, found while verifying the review
fixes against ffprobe 9.0.2 and read_inspection_file:
- ffmpeg keeps one tag per ASCII-folded name under the spelling set
last (ISO-BMFF joins the values with ";", Matroska keeps the last), so
PROMPT and Prompt are one record on the server; the local reader kept
both and failed the read as a duplicate. A legacy comment atom and a
later COMMENT key likewise leave no exact-case envelope.
- ffprobe's format_tags filter folds ASCII case only, so a name that is
"workflow" only under Unicode case rules is never reported; the local
reader took it.
- ISO-BMFF data types 0 and 3 are Mac Roman text to ffmpeg, written into
twice the input size (a long value is cut); the local reader decoded
type 0 as UTF-8 and skipped type 3.
The real-ffprobe parity test builds each shape.
* docs(testing): list the local Inspect divergences; raise touched coverage floors
TESTING.md now tracks the local Inspect (#249 PR L1) divergences that
remain after the review fixes, each checked on a host with ffprobe 9.0.2:
media kind by bytes versus extension (parity is for the same bytes with a
matching extension), JPEG dimensions past the 64 KiB header read, classic
QuickTime udta text atoms, raw versus escaped tag bounds, the 10,000
element ceiling, unknown EBML DocTypes, the JSON depth approximation and
the C _decimal assumption.
coverage:update raised the branch floors of the three local Inspect
modules this PR touches; the unrelated activity-scan raise it proposed is
left at main's values.
|
||
|
Vito
|
b7cf210f0f |
feat(annotations): image tag storage, vocabulary participation and capture store (#276)
* feat(annotations): add image annotation schema with restore validation
Install the image annotation group beside image_favorites and saved_views:
media_annotation_metadata (one row, schema version 1), media_annotation_files
(one private locator per physical file), media_annotation_bytes (one record
per observed SHA-256 with the verified size and mtime_ns) and media_image_tags
(the ordered tag set of a byte record, joined into the global tags registry).
The group migrates in a SAVEPOINT that validates before and after creating
it, with no backfill from prompt or pair tags and no filesystem probe.
annotation_schema_error reads only: it compares the catalog objects after
whitespace compaction, the columns, foreign keys and indexes, and the
persisted values (UUIDs, hex digests, bounds, the 4096-character locator cap
measured past embedded NULs, exactly one metadata row). verify_database_file
checks it with allow_missing=not require_library_state, so old backups
restore and a migrated restore stage must carry it.
Tests pin the exact DDL, a fresh install and idempotent reopen, a current
database without the group gaining it with legacy rows untouched, every
partial or malformed shape rejected without repair, install failure rolling
back and retrying, a WAL backup/restore round trip, staged-failure
immutability of source, live and safety files with every handle closed, and
the require_library_state gate.
* feat(annotations): persist per-image tag sets in the shared vocabulary
Add database/annotations.py (AnnotationOperations, mixed into
PromptDatabase): read_image_tags, replace_image_tags and add_image_tags
run in one _pair_transaction fenced by the storage epoch, own the byte
record by exact locator and digest, always supply UUID ids, rewrite the
ordered junction and bump tag_revision plus image_index_revision.
Revision mismatches raise AnnotationCommandError("tag-conflict") with
the current set; the identical set at the current or previous revision
is replayed without writes; a union over 100 names is "tag-limit".
normalize_image_tags validates each name with validate_vocabulary_tag_name
and folds case-insensitive repeats keeping the first spelling.
refresh_image_fingerprint, current_image_tags (chunked, fingerprint
matched, newest verification wins) and iter_image_tag_records serve the
S1 cheap reads.
Vocabulary participation: _move_tag_links also carries media_image_tags,
legacy merge_tags moves image-only sources, tag rows gain image_count
(files with any byte record carrying the tag), impact gains images and
rename/delete receipts gain images_changed. vocabulary-state.mjs accepts
the optional counts and names images in the delete text.
Browser fixtures clear the three annotation data tables on reset (never
the metadata row); TESTING.md documents the schema beside favorites.
* feat(annotations): add bounded private media capture store
Add utils/media_capture.py: CaptureStore copies one rooted media file
into an owner-only prompt-manager-captures-* temp directory, streaming
it in 1 MiB chunks while computing the whole-file SHA-256, and compares
the opened descriptor with the path (dev, ino where meaningful, size,
mtime_ns, ctime_ns) before and after the copy; any disagreement raises
CaptureChanged and removes the copy. Symlinked files or directories,
root escapes, non-media names and directories raise CaptureDenied; the
locator is normcase(join(realpath(root), relative)), the listing's
identity, so a symlinked root matches it.
Limits are the plan's: 256 MiB per file on stat and stream
(CaptureTooLarge), 8 completed captures, 512 MiB reserved and 2
capture/verify workers with nonblocking admission (CaptureCapacity),
900 s idle and 1800 s hard lifetimes on an injected clock. acquire and
release_use keep an in-use capture past expiry, release and
invalidate_all until its last use ends; close removes only owned files
and the owned directory. A header-only probe of the copy marks Generate
unsupported above 4096 px per side or 16,777,216 px, for unreadable
headers and for video; verify re-hashes the current file without a
copy. Every handle closes before unlink.
tests/test_media_capture.py covers exact bytes and owner-only modes,
alias and escape denial, the symlinked-root locator, change detection
during the copy (size, same-size rewrite with restored mtime through
the POSIX change time, inode replacement), both size caps, count, byte
and worker admission, idle and hard expiry with in-use retention,
cleanup scope, handle release at unlink, dimension and video
readiness, and the allowlist equality with the inspection service.
TESTING.md documents the store and the Windows change-time limit.
* test(annotations): wait on the POSIX change time only where it exists
The same-size rewrite case waits for the filesystem clock to pass the
source's ctime so the detection is deterministic on Linux's coarse
timestamps; Windows reports the creation time as st_ctime, where the
probe could never advance, so the wait runs only on POSIX.
* fix(annotations): compare the change time only between fstat reads
Python 3.12+ on Windows reports the change time from stat but the
creation time from fstat, so a freshly written file whose creation and
write straddle a clock tick failed the pre-read identity check with
CaptureChanged. The path-versus-descriptor checks now compare size,
mtime_ns and dev/ino; ctime_ns is compared between the fstat before
and after the copy, where it still detects a same-size rewrite with a
restored mtime on POSIX.
The symlinked-root test asserted the locator against the unresolved
temp path, which macOS (/var -> /private/var) and Windows (short names)
resolve differently; it now compares against the real path.
* test(browser): give the inspector evidence matrix the concept comparisons' budget
The touch run of 'full inspector attaches matched theme and mobile
evidence with reachable content' captures 32 screenshots across four
viewports and took 23.5 s, then 31.6 s, in the sharded coverage job,
hitting the 30 s default. Like the concept comparison specs it now sets
an explicit 120 s timeout; the CI-like pinned run (4 CPUs, 3 workers,
repeat-each 3) passes 198/198 with the matrix at 18-20 s on touch.
* fix(annotations): compare capture identity like with like and keep paths out of errors
Native Windows CI (Python 3.12/3.13) rejected untouched files with
CaptureChanged because the path stat and the descriptor fstat disagree
there (change time versus creation time, possibly dev/ino encodings).
The descriptor is now checked against the path only on the API-stable
facts (size, mtime_ns); the full identity (size, mtime_ns, ctime_ns,
dev/ino where meaningful) is compared between the fstat before and
after the copy and between the path stats before and after. A test
injects shifted fstat results to simulate the Windows discrepancy, and
a fresh file is captured twice with no platform guard.
Error messages carry strerror only, never the path (the path goes to
the debug log); a test asserts no CaptureError names the root. Files
are unlinked outside the store lock; close() during an in-flight
capture lets that worker remove its temp file and then the directory.
get() documents that readers of capture.path must acquire/release_use.
The test fixture resolves its temp root (macOS /private/var, Windows
8.3 names). TESTING.md states the real identity contract.
* fix(annotations): move image revisions when a fingerprint or the vocabulary changes
A replayed save that observes a new verified (size, mtime_ns) now bumps
image_index_revision, like refresh_image_fingerprint, so caches keyed
on it see the record become current again; an identical replay still
moves nothing.
Vocabulary rename, merge and delete (native and legacy paths) bump the
tag_revision of every image set carrying the source tag, in the same
transaction and before the links move or cascade, plus one
image_index_revision bump when any set was affected. A Tag dialog
opened before the change therefore gets tag-conflict with the renamed
set as current instead of recreating the old name.
* test(annotations): accept change detection wherever fstat exposes a change time
Native Windows CI on Python 3.13 detected the same-size, mtime-restored
rewrite through the descriptor's change time (3.10-3.12 report the
creation time there and complete the torn copy). The test asserted the
platform, not the signal: it now accepts CaptureChanged plus temp
removal wherever it happens, requires detection on POSIX and on
Windows 3.13+, and otherwise requires the digest to describe the
copied bytes. A new test injects a drifting fstat change time so the
detection path runs on every platform; TESTING.md records the limit as
"Windows before Python 3.13".
* test(annotations): assert the rewrite outcome, not the platform
Native Windows CI detected the same-size, mtime-restored rewrite on
3.12 and missed it on 3.13 in the same run: the path stat's change time
(Python 3.12+ fast path) only moves when the clock ticks between the
pre-stat and the rewrite, and the tick wait was skipped on Windows. The
wait now runs everywhere with fresh probe names (Windows tunnels the
creation time of a reused name), and the test asserts the outcome: a
detected change leaves no temp file; an undetected one is accepted on
Windows only and must produce a digest of the copied bytes; POSIX must
detect it. The injected-fstat test keeps the detection path covered on
every platform.
* fix(images): keep the Filters label node across a listing reload
Every listing reload reports the active refinement count, and
setActiveFilters rewrote the button's textContent even when the label
was unchanged, replacing its text node. WebKit drops a click whose
mousedown hit a node that was detached before mouseup, so a Safari
user who types a search and clicks Filters inside the 250 ms debounce
lost the click, and the webkit page workflows timed out in the
source-change recovery tests (1 in ~30 runs of the pinned image; an
instrumented probe showed pointerdown/mousedown, a DOM mutation,
pointerup/mouseup and no click event).
The label is rewritten only when it changes. The toolbar unit test
counts textContent writes (1 for an unchanged count before, 0 after),
and a workspace-images test pins the text node identity across a
search reload: 0/6 before the fix on chromium and webkit, 6/6 after,
with the recovery and Refine tests 36/36 in the pinned CI-like run.
|
||
|
Vito
|
bc14f29db7 |
fix(inspect): video metadata carrier parity between Images and Inspect (#275)
* fix(video): select one prompt carrier from per-key or comment tags Current ComfyUI savers write one container tag per key (prompt, workflow, parameters, comfyassets_prompt_pairs); only the old VideoHelperSuite comment envelope was read. select_video_carrier picks exactly one carrier in the order per-key tags, comment envelope, description: the first with a role key wins, else the first with a workflow, and carriers never merge. Matroska upper-cased tag names fold on our side; differing fold-duplicates fail the read like duplicate PNG keys, identical ones are one record. Per-key values stay original-text; envelope objects are reserialized. * fix(inspect): read per-key video prompt tags The inspector asked ffprobe only for comment and description, so videos written by core SaveVideo/SaveWEBM and current VideoHelperSuite (one tag per key) were inspected as having no metadata. It now asks for prompt,workflow,parameters,comfyassets_prompt_pairs,comment,description and hands the tags to the shared carrier selector. ffprobe stdout is bounded at the role and workflow budgets plus 64 KiB of envelope; past that it probes once more without the workflow tag within the role budget and marks the per-key workflow limited, while an over-budget envelope still fails the read. Differing fold-duplicate tags fail like duplicate PNG keys. * fix(video): scan videos written by current ComfyUI savers The output scan and live monitor read container tags through metadata_from_tags, which now delegates to the shared carrier selector, so per-key prompt, workflow, parameters and comfyassets_prompt_pairs tags (including Matroska upper-cased names) are ingested with the default resolver profile exactly like the same graph in a PNG. Fold-duplicate tags and null markers are skipped quietly. Pins the behavior with a mocked ffprobe scan test and real MP4/MKV/WebM round trips when ffmpeg is present. * docs(testing): document video carrier selection Record the shared video carrier order (per-key tags, comment envelope, description), the fold-duplicate rule, the 6 MiB + 64 KiB probe bound with the role-only re-probe and limited workflow, and that scans read the same carrier as the inspector. * fix(inspect): size video probe bounds for JSON escaping, exact-case envelopes ffprobe JSON-escapes tag text, so bounds measured on its stdout could mark a video workflow limited, or fail a role record, that the identical PNG reads within its decoded budget. The first probe now allows twice the role and workflow budgets plus 64 KiB (12 MiB + 64 KiB) and the role-only re-probe twice the role budget plus 64 KiB, leaving the decoded per-record budgets as the only gate; a reader-level test reads near-budget quote-dense records as PNG and as video and asserts identical metadata and formats. Envelope keys (comment, description) are matched exactly as the legacy readers did; only the four per-key carriers fold Matroska case. Fixtures can write the legacy ilst comment atom (metadata_tags=False), and the docs now say the admin scan reads the carrier while the live monitor still processes images only, with a follow-up recorded in TESTING.md. |
||
|
Vito
|
f95f66d76a |
ci(browser): run page workflows with parallel workers and CI shards (#274)
* test(browser): one test server per Playwright worker The page workflows started one global webServer and ran with one worker. Each worker now owns its own tests/browser/server.py on an OS-chosen port, with its own temp root and database, through a shared `test` in tests/browser/fixtures.mjs. baseURL, and so page, request and browser.newContext, point at that worker's server. A server that dies fails the running test with its stderr tail and the next test gets a fresh one. The worker count comes from PM_TEST_WORKERS, else 1 under CI, else a quarter of the CPUs up to 8. Chromium workflows in the pinned image: 510 tests in 606 s with one worker on main, 511 (one new isolation spec) in 119 s with eight. * test(browser): test servers bind an OS-chosen port themselves tests/browser/server.py accepts --port 0: it binds 127.0.0.1:0 before building the app, holds that socket for its whole life and prints PM_TEST_SERVER_PORT=<n> to stdout. startBrowserServer defaults to port 0 and reads the announced port, so no port is reserved and released before the server binds and two launchers can no longer race for one. The "address already in use" retry and free-port.mjs are gone; --port N still works and is checked against the announcement. * ci(coverage): shard and merge modes for the frontend coverage run frontend-coverage.mjs now plans its steps from the command line. No arguments still runs everything, then reports and checks. --shard I/N runs one slice of the chromium+touch workflows into coverage/js/raw without wiping it or reporting; shard 1 also runs the node units and the viewer suite. --merge-only reports and checks over raw coverage the shards wrote, with --update-baseline as before, and refuses an empty or missing raw dir. npm scripts: test:coverage:shard and coverage:merge. * ci(viewerjs): run browser suites with workers and shards Frontend coverage runs as three shards (PM_TEST_WORKERS=3 each) that upload their raw V8 files as coverage-raw-N; a merge job downloads them, fails unless all three arrived with raw files and no clashing names, then runs coverage:merge. Firefox and WebKit page workflows run as two shards each with three workers. The required viewerjs context keeps its name and now needs the shards, the merge and the browser shards, failing unless every one succeeded. * ci(viewerjs): cache npm downloads and size timeouts from the first sharded run The first sharded run lost a webkit shard to its 12-minute limit after all 267 tests passed, and npm ci stalled for 5 minutes in six of eight jobs. Each npm ci now restores npm's download cache keyed on the vendor lockfile, and browser job timeouts are the measured times plus at least 50%, never under 20 minutes. The workflow comment records the measurements. * test(browser): wait for rendering frames, not timers or luck, under load Two Library workflows failed in CI's webkit shard 2/2 (run 39714840809) under parallel load: - "library infinite loading ..." counted cards right after the last page hid "Load more". That render queues the window update for the next frame, so a slow frame left the 120 cards of the all-rows layout in place (received 120, expected < 80). It now waits for the next frame, which runs after every callback the page queued. - "row measurement preserves anchor ..." settled with a 60 ms timer. A scroll event is delivered in a rendering frame, so on a starved renderer the timer ended before the scroll's frame and no card was in view (null anchor). settleLibrary now waits for two consecutive native frames with no Library frame queued, checked inside the frame callback. Before/after: a scratch copy that delays every frame by 120 ms (as a starved runner) failed the old helpers 19 of 40 runs, at the CI lines, and passed the new ones 40 of 40 on chromium and webkit. Throttled to 1 CPU with 4 workers, the original spec failed 1 of 42 for each test. The Tags viewer workflow closed the viewer as soon as its image was visible, before ViewerJS rendered the title, so CI's coverage never ran the page's title callback (tags-page.js functions 26/46 = 56.52%, one function above the floor of 56; locally 27/46). The test now asserts the title names the shown file, which always runs the callback. * test(browser): stable under parallel workers Document how to run the browser suites in parallel and sharded, with the measured times, and what keeps specs stable beside each other: wait for a rendering frame, a response or a visible state, never for time. With |
||
|
Vito
|
2cecfcf0ac |
feat(activity): scan new and unresolved files by default, and download a job report (#273)
* feat(activity): scan new and unresolved files by default, all files on request
Scan for metadata now skips, before reading any metadata, the files the
Library already links with a resolved link (one generated_images row with
exactly one pair link), found with one streamed query that keeps only the
scanned paths and compares paths with os.path.normcase on both sides.
Skipped files count as Already known ("Skipped · already linked").
Unlinked, fallback-linked, ambiguous and unresolved files are read again.
POST /prompt_manager/jobs/scan takes {scope: "new" | "all"} (default
"new", anything else is a 400). The review dialog shows the concept's
"New and unresolved files" and adds an "All files" switch (D58). The
legacy /prompt_manager/scan route keeps scanning all files with unchanged
SSE payloads.
* feat(activity): download a job report
GET /prompt_manager/jobs/{id}/report returns the summary, counters, every
outcome group's listed rows with truncated flags, the scope and the log as a
JSON attachment (prompt-manager-<kind>-<YYYYMMDD-HHMM>.json), with no absolute
paths. The job view's Download report saves it through the api-client URL, and
the scan log opens at its newest line and follows new lines only while the
reader is at the bottom.
* fix(activity): re-read known files whose size changed, and review follow-ups
The new scope now skips a known, resolved file only while its stored
generated_images.file_size equals its size on disk; a NULL stored size is
re-read. A file replaced at the same path is read again and its conflict
shows under Needs review. The scan stores each file's size when it ingests
it, and both sides of the known-set match go through normalize_image_path.
A failed known-set read logs and falls back to reading every file, and an
empty request body asks for the default scope. The All files switch is
described by its hint, a lost job's Download report is disabled, the report
toast is skipped once the view is gone, and the scan log opens at its
newest line every time. Fixtures use the real scope shape.
* test(activity): wait for the job view's settled detail read, not a fixed delay
The 200-row cap test now starts counting detail reads once one read has
returned the full listing and none is in flight. The job view re-reads after
an in-flight read only when the summary still lists more than that read
returned, so no redundant read is left due after opening.
|
||
|
Vito
|
a6f6697c50 |
feat(activity): shared job registry, Activity page and Scan for metadata (#269)
* feat(activity): shared session job registry with the scan as its first kind The output scan now runs as the first kind of an in-memory JobRegistry (py/api/jobs.py, replacing py/api/scan_job.py). Each job keeps exact counters, at most 200 outcome rows per group and a log capped at 2,000 lines, with root-relative paths and public root labels only. New routes list, inspect, follow (SSE) and cancel jobs and start the scan (POST /prompt_manager/jobs/scan, 409 busy while another record-changing job runs). The legacy /prompt_manager/scan routes attach to the same job and stream unchanged payloads. * feat(activity): concept Activity page, top-bar button and progress pill The workspace gains the concept's Activity page (renderActivity), its top-bar button and sidebar row, fed by one session job store: a /jobs read, one SSE stream per running job, a 1-30 s capped backoff after a dropped stream and a 15 s list poll while visible. A job the server no longer lists shows as ended, so no pill stays stuck after a restart. D52: the counts show running jobs and hide at 0; the phone top-bar row may wrap. D53: the mockup's progress pill links the running job. D54: real job items with state chips, detail lines and Cancel job. Start and finish toasts follow the spec copy. Fake browser jobs come from tests/browser/workspace_activity_fixture.py. * feat(activity): Scan for metadata with its review dialog and job view Images -> Import keeps Rescan and adds the concept's "Review scan" row as "Scan for metadata". It opens the concept's "Review metadata scan" with the real source count, the fixed "All files" scope, a backup box that downloads the existing database backup, and Cancel / Start scan. Start scan starts the session scan job and opens its view; a start refused by another record-changing job toasts its title as a link. #/activity/<id> is now the approved job-view mockup: the heading with Download report (disabled until Activity-2) and Cancel scan, the Progress surface with the five counter tiles, the Results surface with the outcome list, path filter and capped rows (Added and Already known open the viewer through the listing identity), and the collapsed Scan log. Activity item titles link their views. The scan's progress line groups thousands. Deviations D55-D57 are recorded in TESTING.md. * fix(activity): stop the job store on pagehide and let proxied routes outlive teardown CI's touch coverage run failed "a resolver failure for the ComfyUI link never stops the shell from mounting" with "route.fulfill: Fetch response has been disposed": its trace shows two Library thumbnail requests started by the last route still in the spec's proxy-mount handler while the page fixture was torn down. - The proxy-mount handler ignores only that teardown error, and the workspace and Activity specs unroute with ignoreErrors before teardown. - The Activity store now ends its work on pagehide: the poll and backoff timers stop, the job streams close and a list read in flight is aborted (an older read is also aborted when a newer one starts). A pageshow from the back-forward cache reads the list again. * test(activity): compare the job view log's line height to the hundredth of a pixel WebKit reports the mockup's 11px/1.7 line height as 18.700001px, so the job view comparison failed on WebKit only. The measurement now rounds it, as other engine-dependent values are normalized. * fix(activity): final review round for the job store, job view and scan rows - Store: one failed list read no longer marks running jobs as ended; two in a row, or a good list without them, do (M1). A good read resets the backoff, except for a stream that keeps dropping while the list reads fine (M2). Finished jobs the server pruned are dropped (M3). Numbers group as en-US, like the server's lines (M13). - Job view: a job the store's list no longer has shows the not-found state (M3). Focus moves to the job heading when Cancel scan goes away, and to the item's title link in the list (M4). The detail is read only when its listed rows change or the open log grows, using the summary's new listed and log_length (M12). - Viewer rows open by exact identity: each scan row records its root's listing index, the source must still carry the row's label, and the workspace listing gains an exact `path` filter (M8, M9). - Scan rows name the prompt by its Library title, stored title first (M10). - A toast with a link stays 20 s, and any toast stays while hovered or focused (M14). - A real two-batch scan cancelled after batch 1 keeps partial counts (M7). - TESTING.md: D54, D56 and D57 updated (M5, M6). * test(activity): cover the held and linked toast and a job view pruned by the server Unit tests for the toast's 6.5 s and 20 s timeouts and its hover and focus hold, and a browser test for a finished job's open view after the server stops listing it. Coverage floors are only raised. |
||
|
Vito
|
6e9d8f81fd |
ci(coverage): fast check that every frontend file has a coverage baseline (#271)
The frontend coverage job only reports a missing baseline entry after its full browser run. coverage:baseline-check compares the files the gate measures (same discoverSources) with frontend-coverage-baseline.json in milliseconds and also flags entries for files that no longer exist. npm test runs the same check. |