6b568de09bb307e4ec873789606aa65726cbfb33
512
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito Sansevero
|
6b568de09b |
feat(tools): reviewed repair of the foreign dataset links (#247)
tools/repair_foreign_links.py judges every link row of a file that only foreign runs reported with the shared ladder (scan policy, recorded saver). The dry run judges a copy, writes a JSON plan and a Markdown list, and rehearses the apply on that copy to audit the prompts that would lose every link (deleting them is off unless chosen). --apply refuses while ComfyUI runs, takes a backup, re-judges every row inside one BEGIN IMMEDIATE transaction, creates the targets with the scan's marks, relinks and drops pair links, keeps UNIQUE(prompt_id, file_path), and rolls back with exit 2 on any drift. |
||
|
Vito Sansevero
|
b107de70f8 |
fix(scan): one prompt per file for recorded-run links; writes survive a library move
The recorded run's stored prompt now links in one transaction with the file's link rows, so a file linked to another prompt is a conflict instead of a second link. The scan's stores run through survive_library_switch. A single file asks history once (a one-file _BatchHistory), the scan passes in_output=False and reads the file size only for files with prompt metadata. A conflict row reads "Kept: already linked to a different prompt". TESTING.md records the rules. |
||
|
Vito Sansevero
|
a6d634509b |
fix(ingestion): keep the legacy answer that is a PromptManager node's own text
The joined-text refusal also refused the #170 parity files whose PromptManager prepend is a link the browser parser does not follow: its answer is then the node's literal text, which is what the node stores. Refuse only an answer that is not such a node's own text. |
||
|
Vito Sansevero
|
e137270f45 |
fix(ingestion): no joined PromptManager text from the legacy floor; junk keeps its step's status
The legacy parser returns prepend + text + append, so its answer is refused when a PromptManager-family node has a prepend or append part (#343). Junk found at the undecided-graph step keeps that step's unlinked status and review, so an ambiguous file is still queued for Needs review. The monitor policy's ambiguous file no longer asks for the recorded run, and the file size may be read lazily. |
||
|
Vito Sansevero
|
4b4fa7f8ff |
fix(ingestion): keep the file-name saver anchor out of the ladder (#390)
The #390 ruling queues a file that only its saver's file name resolves for Needs review, never linking it. inspection_resolution gains name_anchor; the ladder passes False, so such a file falls to the legacy floor in the scan (and to Needs review) and is never stored by the monitor, while the recorded saver still decides. The inspector keeps the name anchor. |
||
|
Vito Sansevero
|
4a90a5dd83 |
feat(scan): decide each scanned file with the shared ladder (#247)
_ingest_scanned_file calls decide_file_prompt (scan policy) and stores the result: pairs through ingest_resolved_pair and positives through store_scanned_prompt, both with the scan marks. Files the default profile leaves undecided now get the inspector's resolve, with the saver recorded history names, before the legacy parser, so each branch of a shared-prefix dataset run links its own prompt. Junk text is never stored, a file linked to another prompt is a conflict instead of a second link, and a failed link leaves no prompt behind. _BatchHistory reads the stored prompt and the recorded saver from one recorded_runs pass. _parse_comfyui_prompt calls the moved parse_comfyui_prompt. TESTING.md documents the decision. |
||
|
Vito Sansevero
|
e3191853c8 |
feat(ingestion): store a file's prompt in one transaction, one prompt per file
database/scanned_prompts.py links a file to the prompt of one text, creating it when new, inside one BEGIN IMMEDIATE transaction: a file linked to another prompt is a conflict, a failed link leaves no prompt behind and two writers store one prompt. pair_ingestion gains file_link_rows, link_conflict and Marks/mark_created; ingest_resolved_pair takes optional marks for the components it creates. recorded_runs and named_prompt in run_attribution become public (#247). |
||
|
Vito Sansevero
|
9bf8362d22 |
feat(ingestion): one per-file prompt decision for the scan and the monitor
utils/file_prompt.py holds decide_file_prompt, the ladder the output scan runs today plus the inspector's rule (inspection profile, then the recorded saver) before the legacy parser, with a SCAN and a MONITOR policy. Junk text (blank, numbers, LoRA tags only, model file names) is never stored. graph_prompt_text and parse_comfyui_prompt move here; the parser accepts parsed metadata (#247). |
||
|
Vito Sansevero
|
a90d251f39 |
refactor(inspector): share the inspector's resolve as inspection_resolution
inspect_prompt_evidence now calls inspection_resolution, which returns the resolution to use, the anchored result and whether the recorded saver chose it. Ingestion can call the same helper for files the default profile cannot decide (#247). |
||
|
Vito Sansevero
|
a8e56f1812 | Merge remote-tracking branch 'origin/main' into feat/247-foreign-workflows | ||
|
Vito
|
e769845417 |
refactor(admin): open System for settings, maintenance, backup and restore (#330, PR 2) (#407)
* test(legacy): snapshot the routes and the settings response Pins every registered route and the key paths of GET /prompt_manager/settings while System S13 (#330) moves the legacy pages' tools to native System. Recorded on the unchanged main; cutover (#334) removes it. * test(legacy): prove each removed legacy control has native coverage TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet). legacy-coverage.test.mjs fails when a removal cites a capability that is not covered, copies another native test, lists an id or method twice, or leaves the id or method on its page. legacy-system-links.spec.mjs holds the System link table and guards the kept admin flows. * refactor(admin): open System for Diagnostics and logs The admin page's Diagnostics and View Logs buttons become links that open System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs dialogs and their methods go; TESTING.md lists them against their covered capabilities (rows B1–B7). * refactor(admin): open System for settings and keep page options Settings opens System › Overview in a new tab. The settings dialog becomes Page options (opened from the list header) with only the Infinite Scrolling default, and its Save posts {"infinite_scroll": …} alone, so a stale legacy form can no longer overwrite scan folders or other values edited in System (#330 S13-9). Scan folders, monitoring status, auto-hide, display mode, worker threads, LoraManager and the LoRA import leave the page (rows B8–B14). * refactor(admin): open System for maintenance Maintenance opens System › Maintenance in a new tab, where each operation runs as a reviewed Activity job. The one-click legacy Database Maintenance dialog and its methods go (rows B15–B18); formatFileSize stays for the image modal. * refactor(admin): open System for backup and restore Backup DB and Restore DB open System › Backup & export in a new tab, where a restore is staged, reviewed and run as an Activity job. The one-click Restore dialog and its methods go (rows B19–B20); the scan dialog's quick backup stays. The removed browser tests' data checks live on at the route: a new round trip test restores every prompt with its tags and image links. * test(legacy): find a removed modal id passed to showModal and settle the links spec The removal check also finds an id passed as a string to a ...Modal() call and no longer reads data-id= as id=. The no-request test waits for the page's own start-up reads to finish, the sort check compares as the server sorts, and the kept-flow test checks the added prompt shows in the list. * test(admin): open Page options after the saved default is read Drops the leftover LoraManager section comment above the Page options save. * test(legacy): a lazy thumbnail load is not a System link's request |
||
|
Vito Sansevero
|
3bba78d0eb | Merge remote-tracking branch 'origin/main' into feat/386-recorded-attribution | ||
|
Vito Sansevero
|
2d7e138624 |
fix(history): look up recorded runs by history's own spelling of the file (#386)
The history route inspects a case-normalized target on Windows, while history matches file names exactly, so the occurrence's reference names the file. |
||
|
Vito
|
9a90c7677f |
fix(exit): stop running jobs, downloads and encoders at Ctrl+C instead of waiting for them (#376) (#417)
* fix(exit): stop running jobs before Python joins the executor threads (#376) ComfyUI returns from main.py after Ctrl+C without closing its event loop, and Python joins every executor thread before any atexit callback runs, so a scan batch, thumbnail batch or model download in flight kept ComfyUI alive until it ended (on Windows each ffmpeg up to its 30 s timeout). The package now registers a threading._register_atexit hook (after concurrent.futures', so it runs before that join) that sets an exit signal and kills the running encoder processes. Job.cancelled follows the signal, so every check a job already makes between files stops it at exit; map_parallel starts no further item, the scan walk stops between folders and a tagging model download stops before its next chunk. Without the internal hook exit behaves as before. * fix(exit): abort Xet downloads and stop LoRA staging at exit (#376) Review follow-ups: Xet transfers report progress from native code and ignore the chunk checkpoint, so the exit hook calls huggingface_hub's own abort_xet_session when it is loaded; an aborted download logs as stopped. LoRA import staging reads the exit signal itself, so the legacy LoRA scan stops too. The test's fake encoder outlives the ceiling by less, and TESTING.md states the download limits of older huggingface_hub. |
||
|
Vito
|
65b15bbe75 |
fix(logs): Clear all logs deletes and empties only what its confirmation showed (#389) (#412)
* fix(logs): Clear all logs deletes and empties only what its confirmation showed (#389) - /logs/files rows carry an identity string (size:mtime_ns:ino); the workspace sends back the active and rotated rows its dialog was built from. Under the files and handler locks every listed copy is re-checked with _matches_review and the listed active log's inode; any difference refuses with nothing changed (409 logs-changed). A copy that appeared after the confirmation is kept. - A second writer of the active log (our flushed stream is behind the file's size, or our file is now a rotated copy) refuses first (409 logs-foreign-writer); each unlink is preceded by a fresh identity check and the active log is emptied last. - An active log with a hard link (st_nlink > 1) is never truncated (409 logs-hard-link), also on the file-logging-off path. - The legacy admin page's POST without a body keeps clearing what is there, with the foreign-writer and hard-link refusals. - The browser fixture grows the active log through the handler's stream. * fix(logs): accept pre-1970 identities, refuse duplicate rows, say the foreign-writer check is best effort (#389) Review follow-ups: a negative mtime_ns no longer makes every workspace clear 400; a name listed twice is a malformed list; the confirmed-row cap is 10000; the hard-link refusal says how to recover; the hard-link and foreign-append tests assert the kept bytes before the refusal code, so on the old code they fail on lost data. |
||
|
Vito
|
95c4542c6f |
test(browser): API restores stay unseen by the open page's Activity poll (#413) (#416)
* test(browser): API restores stay unseen by the open page's Activity poll (#413) Specs that restore through the API while a workspace page is open check how the page handles a restore it did not see. The page's Activity store reads the job list every 15 s; on a slow runner (CI run 40646112668, a 16 s restore) that read landed during the restore, the page followed the job and, as designed (S9), showed the open page again when it finished, closing the Create tag dialog the vocabulary spec was about to use. unseenRestore holds the page's job-list reads until the restore has answered, so they read it as already ended. Used by the vocabulary, library editor and history specs. * test(browser): arm the restore gate before the page loads and drain reads in flight (#413) A job-list read the page sent before the restore could still be answered while it ran. The gate now routes every job-list read from before the first navigation and, before a restore, waits for the reads in flight to be answered. The history and library editor conversions are pre-emptive: same race, never seen failing. |
||
|
Vito
|
534c95248a |
fix(review): anchor files on their saver in the scan and Needs review (#390) (#414)
* fix(review): anchor files on their saver in the scan and Needs review (#390) The scan and Apply now resolve with the file's name, as the workspace inspector does since #383, so the one saver whose filename_prefix names the file anchors its branch. A file whose branch that anchor resolves is never queued (from the inspection profile it offers no choice; a default-ambiguous file asks saver_anchored) and its pending, undecided items leave on the next scan. REVIEW_RULE_VERSION is folded into inspection item versions so old items are re-evaluated. No match, several matches or an unfinished walk anchor nothing. Apply refuses a choice the anchor contradicts. * fix(review): refuse a default choice the saver anchor contradicts (#390) Apply re-resolves a default item with the file's name: when its saver anchors the branch, no candidate is offered and the choice is refused. The scan's anchor check skips bytes carrying a decision and remembers unanchored file states per process, so a rescan does not inspect an unchanged file again. * fix(review): queue an anchored file with its saver's pair as the choice (#390) Maintainer ruling: a file never falls through both the link and the queue. When only the saver anchor resolves a file the default profile and the parser floor leave unlinked, the scan queues it with the anchor's pair as its one candidate (evidence anchor: saver), never linking it. Files already linked or resolved by the default path are unchanged. Apply accepts the anchored pair when the bytes now resolve to it, and an anchored item is a choice the untraced settle keeps. * fix(review): upgrade old default items of anchored files, tag anchored results (#390) A pending default item recorded before the saver anchor no longer shortcuts the rescan when only the anchor resolves the file: the anchored item supersedes it. Apply accepts a resolved re-resolution only when it is tagged as the saver anchor's, the anchored pair must carry a positive text to choose, and limited anchored evidence still counts as a choice. TESTING.md describes the ruling. |
||
|
Vito
|
e4dc781c26 |
fix(images): read a loading collection as loading, and de-flake the JXL fixture and restore specs (#391, #371, #361) (#405)
* fix(images): read a loading collection as loading, never as 0 of 0 images (#391) A reset published {loading, loaded:0, total:0} and the caption painted it as "0 of 0 images" until the first page answered, so a fast reader (the inspector concept spec on WebKit) read an empty result the server never returned. The count line now says "Loading images…" while a first page is in flight; the spec waits for the answered count, and a held-listing spec pins the ordering. * test(thumbnails): make the no-JPEG-XL-encoder fixture authoritative (#371) {available:false} stood in only the cached detection, and every settings read probes a cached None again, finding the host's real encoder. The fixture now stands in for the detector itself and reset restores it; a unit test proves the route against a detector that finds an encoder. * test(system): wait for the restore job's own end before its toast (#361) The reviewed-restore specs waited for the "Database restore finished" toast with the 5 s expect, in effect a deadline on the disk-bound publish. They now wait for the job's progress stream to close, assert the job finished, and only then check the toast with the default timeout. * test(images): wait for the search's answered count in the concept comparison (#391) searchOf took the first changed caption as the search's count, the same read-before-the-response race as the inspector's tag chip. |
||
|
Vito
|
d0339128e4 |
test: isolate API fixture databases from other suites; CI lint pins, failing bandit hook, npm cache (#346, #352, #409) (#411)
* ci: pin black and flake8, let the bandit hook fail, cache npm in the JS job (#352) The lint job installs black==26.10.1 and flake8==7.4.1 from requirements-dev.txt (the versions CI resolved on main today), so a new release cannot turn CI red with no code change. The local bandit hook ran 'bash -c ... "$@"; true': it could never fail, and bash bound the first staged file to $0 so it was never scanned. It now runs 'bandit -lll -q' directly, failing on high severity exactly like code-quality.yml. The frontend modules job's setup-node caches npm keyed on tools/vendor/package-lock.json. * test(history): wait on the writer's state, not short wall-clock pauses (#352) Four 0.08 s pauses before asserting that the worker had not probed yet now wait until the idle worker has run a whole loop pass at the current manual clock value (three clock reads), so a slow runner cannot end the pause before the worker decides. A mutation that probes 20 ms early now fails two tests deterministically. The real-clock bounded close no longer asserts close(0.02) returned in under 0.2 s wall time; it asserts the in-flight write is still blocked and the worker alive, like the output writer test. The manual-clock bound stays exact. The last 0.1 s non-blocking submit check uses the 1 s NON_BLOCKING_CEILING from #351. * test(api): give each API fixture database its own temp folder (#346) Every PromptManagerAPI sweeps its database's folder at startup for the stages, uploads and safety-backup copies an interrupted restore left. The API test fixtures created their database directly in the system temp folder, so any other test process starting an API there (another worktree's suite, a pre-push hook, an agent's targeted run) deleted this process's in-flight restore files. The restore, epoch and retirement tests then failed with 'staged restore changed', 'unable to open database file' or 503 instead of 409. Busy machines fail because they are running other suites, not because they are slow: the same CPU load with a private TMPDIR passes. private_temp_db() puts the fixture database in a new folder removed at cleanup (with any safety backups a restore left beside it). A regression test sweeps a stand-in shared folder in the middle of a multipart restore, as another suite's startup would. * test(history): count worker parks, hold saturation blocks until released (#352) wait_for_worker_pass now counts the idle worker's parks on its wake event (two parks from now = one whole pass begun after the call) instead of assuming a fixed number of clock reads per pass, which a future clock read would have made vacuous. The real-clock bounded close keeps a loose upper bound (0.02 s + NON_BLOCKING_CEILING). The two saturation tests (history and output writer) blocked the worker with release.wait(3) while they queued 1023 more items. On a loaded runner the fill took longer than 3 s, the worker unblocked on its own and drained part of the queue (pending 868..1017 != 1024), and cleanup then reported a leaked worker. The block now holds until the test releases it; 60 s is only a hang guard, and cleanup always sets the release. Under one core with 8 busy loops: 8/8 runs failed before, 6/6 pass after. * test(maintenance): wait on duplicate jobs with a disk-bound hang guard (#409) On Windows py3.13 CI (PR #405) a one-file duplicate cleanup missed its 10 s WAIT. The test already waits on the job's own task; the 10 s bound acted as a deadline on disk-bound work. In the same 50 s window the following tests' setups took up to 9 s against 0.1-0.6 s around it while the test bodies stayed fast, which is consistent with a process-wide disk I/O stall on the runner rather than a lease, handle, executor or job stall (the log has no thread dump to show more). tests/disk_waits.py holds the one disk-bound hang guard (90 s per wait, under the 120 s per-test watchdog), shared with RESTORE_SECONDS (#382). Waits on find and cleanup job tasks in the duplicate-files tests use it; gates and server-state polls keep WAIT. The keeper-validation test turns the safety backup off, since it does not test backups. |
||
|
Vito
|
ef7e07c765 |
refactor(admin): open System for Diagnostics and logs (#330, PR 1) (#402)
* test(legacy): snapshot the routes and the settings response Pins every registered route and the key paths of GET /prompt_manager/settings while System S13 (#330) moves the legacy pages' tools to native System. Recorded on the unchanged main; cutover (#334) removes it. * test(legacy): prove each removed legacy control has native coverage TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet). legacy-coverage.test.mjs fails when a removal cites a capability that is not covered, copies another native test, lists an id or method twice, or leaves the id or method on its page. legacy-system-links.spec.mjs holds the System link table and guards the kept admin flows. * refactor(admin): open System for Diagnostics and logs The admin page's Diagnostics and View Logs buttons become links that open System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs dialogs and their methods go; TESTING.md lists them against their covered capabilities (rows B1–B7). * test(legacy): find a removed modal id passed to showModal and settle the links spec The removal check also finds an id passed as a string to a ...Modal() call and no longer reads data-id= as id=. The no-request test waits for the page's own start-up reads to finish, the sort check compares as the server sorts, and the kept-flow test checks the added prompt shows in the list. |
||
|
Vito
|
83fa80c330 |
fix(restore,threads): a stopped restore waits for its worker; exit-window thread refusals are quiet (#360, #375) (#404)
* fix(restore): a restore cut off by shutdown waits for its worker thread (#360) A cancelled restore handler or job no longer removes the stage or the upload, or ends its exclusive registry admission, while the executor thread still stages or publishes. The cancel waits for the worker; a publish that finished is adopted before the cancel goes on, and a stage finished after the cancel is removed. * fix(threads): a thread or pool refused at interpreter exit is quiet and leaves nothing pending (#375) During interpreter exit, threading._shutdown shuts every thread pool before the package teardown runs while our daemon threads keep working. map_parallel now runs the items a refused pool did not start on the calling thread, schedule() drops the path and returns False when its worker cannot start, and the preview queue skips the file. Each refusal is logged once at DEBUG without a traceback; other start failures behave as before. * fix(restore): keep the cancel and the adoption when a stopped restore's publish ends (#360) A publish that fails after the cancel now ends in CancelledError (after its prune) instead of a 500, and the restored library is adopted to its end even when a second cancel arrives. The blocking calls go through _run_in_executor again, as the executor contract test expects. Exit-window refusals: the comments state what was measured (a pool is refused during threading._shutdown; Thread.start only once the interpreter is finalizing), and the tests assert the one DEBUG line. |
||
|
Vito
|
e472a8bf74 |
fix(system): Tagging selects show their whole chosen option at 390 px (#398) (#399)
* fix(system): Tagging selects show their whole chosen option at 390 px (#398) The concept's phone cap (120px) cut Existing tags ("Replace after review"). #tag-existing now never shrinks for its row text and keeps its natural width at phone width, beside the text. The comparison expects selectsFit for every select, and a new test walks every option of both selects at 390 and 320 px in both themes. TESTING.md D132 records the decision. * fix(system): bound Existing tags by its row at phone width (#398 review) |
||
|
Vito
|
44177974cb |
fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252) (#397)
* fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252) A graph where no node takes a positive or negative input fell back to the pre-redesign _extract_readable_prompt heuristic: the last node with any string text input won, and with none it read stale UI workflow widgets. It stored LoRA syntax, display-node reports, install notes, model file names and numbers. The scan now asks sampler_less_prompt: only the executed API graph counts; a candidate is a node with a text input that another node consumes; blank and numeric texts are never prompts; this repository's own positive nodes are authoritative; one distinct text links, several (or an unreadable wired text that no candidate builds) stay unlinked and try the recorded-run attribution. The heuristic and its helpers are removed; A1111 files read the parsed positive prompt directly. * fix(scan): refuse loader side outputs, seeded templates and keep A1111 text (#252) Review findings: an unreadable text link counts as built from a candidate only through that node's first output, so a LoRA loader's trigger-word output is unknown, not its text. A node with a seed input holds a template, never the prompt that ran. A sampler-less graph the rule cannot decide still links the A1111 positive a metadata saver wrote beside it, as the old scan did. Only plain numbers are numeric; words such as nan or inf are prompts. * fix(scan): linear plain-number check for sampler-less texts (#252) The numeric pattern had overlapping quantifiers that backtrack quadratically on a long digit run ending in a non-digit; it now has none. |
||
|
Vito Sansevero
|
e34c1e2351 |
fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386)
Review follow-ups: the texts PromptManager recorded are looked up at the sampler history's saver leads to, never at a sampler the unanchored walk named; a file name naming a different saver than history anchors nothing; outputs and texts are read in one transaction; "Recorded run" ranks below every workflow-inferred source; unexpected lookup errors log at error level. |
||
|
Vito Sansevero
|
68443551b2 |
feat(inspector): use recorded run history when the graph alone can't name the prompt (#386)
When a file's own graph leaves the prompt open, both inspect routes ask recorded run history about the file (one run reported it and it was written during that run). The save node that run reported it from anchors the walk like a file-name match, so dataset branches that share one saver prefix show their own prompt, labelled "Recorded run". Still open at one sampler whose PromptManager text was recorded at execution (a language model's output), the recorded text shows as "Execution recorded". Results the file resolves alone never change; a failed lookup judges the file alone. |
||
|
Vito
|
d11b354ed6 |
fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2) (#394)
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1) The native "Tag output images" job now runs the image-tags loop over every still image the output folders list, linked or not: each image gets its own model tags (at most "Suggestions per image") in its image tag set, and its bytes are marked auto-tagged in a new media_image_autotags table. No prompt tag or prompt text is written. - database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional in old backups, refused without the annotation group. - add_image_tags/replace_image_tags take mark_autotagged; autotag_state and autotagged_locators read the marks through the one current-record rule. - "Skip already tagged" skips images whose current bytes are marked, counted in a stat-only pre-pass; hand tags do not count. - The review counts images and estimates the time (WD14 on CPU or CUDA). - System > Auto-tagging review rows and copy, D134 and the Auto-tagging contract follow. Part of #379 * fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379) The job read as ended while it still held the model gate, so a quick restart could be refused with model-busy. Pre-pass chunks now publish progress, and a cancel between chunks keeps exact counts. Part of #379 * fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379) The live check measured WD14 on CUDA at 0.167 s per image (100 images in 16.7 s with the model load), so the review estimate uses 0.17 s. A finished or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X min left"; it now ends with "<verb> <state> · N files processed". Part of #379 * fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2) The admin and gallery batch starts the shared Tag output images job ("Tag output images (legacy page)") and relays its progress; closing the window cancels it. Review asks per image with Skip first and applies by image and sha256; apply refuses a prompt_id body. The Tag image dialog marks an image auto-tagged only after Generate suggestions on that capture. A second legacy batch (also one that wins the start race) answers JSON 409, and both pages now show that sentence. * test(tagging): gallery review units apply by image; record each progress change The gallery review unit tests still pinned the removed prompt_id apply. The legacy batch spec read the progress text once per observer callback, so a phase frame and the first image frame landing in one task lost the phase (WebKit); it now records every mutation record. |
||
|
Vito
|
2f8b7cb306 |
feat(images): select many images: ranges, keys, touch and all N matching (#392) (#393)
The workspace Images page selects like a file manager in Grid, List and Masonry, across every loaded batch: Shift+click ranges from the anchor, Ctrl/Cmd+click toggles a card without opening it, Ctrl/Cmd+A selects the loaded images, Shift+arrows extend, Escape clears, and a touch long-press starts a selection mode in which taps toggle. A plain click still opens or inspects. Once every loaded image is checked and more match, the bar offers "Select all N matching". Add tags, Auto-tag selected and Set category then send the listing filter instead of N identities: the image-tags job takes `filter` in place of `targets` and /bulk/category takes `images_filter`, both resolved by the listing's own admission code (up to 10,000 images, a Source bound by its root_scope). Export metadata waits for loaded images. Recorded as deviation D135; the concept is unchanged. |
||
|
Vito
|
37e4e08327 |
fix(tagging): Images tag search goes through the tag index (#379, PR 3) (#395)
* fix(tagging): Images tag search goes through the tag index (#379, PR 3) The Images Tags filter and text search no longer stream every image tag row per request. image_tag_ids reads the names that tag at least one image (exact under Tags, case-folded substring for q) and image_tag_locators reads their files by tag_id through idx_media_image_tags_tag. Results are unchanged: a differential test compares the lookup with the old stream, and a trace test checks that no statement reads media_image_tags except by tag id. The search contract in TESTING.md and a browser test (CUJ 8) follow. Closes #379 * test(search): pin prompt-only tags and multi-chunk tag lookups (#379) Reads the tag names and the locators in the same annotation read helper. |
||
|
Vito
|
d2ea705a44 |
fix(tagging): Tag output images tags each image, never its prompt (#379, PR 1) (#380)
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1) The native "Tag output images" job now runs the image-tags loop over every still image the output folders list, linked or not: each image gets its own model tags (at most "Suggestions per image") in its image tag set, and its bytes are marked auto-tagged in a new media_image_autotags table. No prompt tag or prompt text is written. - database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional in old backups, refused without the annotation group. - add_image_tags/replace_image_tags take mark_autotagged; autotag_state and autotagged_locators read the marks through the one current-record rule. - "Skip already tagged" skips images whose current bytes are marked, counted in a stat-only pre-pass; hand tags do not count. - The review counts images and estimates the time (WD14 on CPU or CUDA). - System > Auto-tagging review rows and copy, D134 and the Auto-tagging contract follow. Part of #379 * fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379) The job read as ended while it still held the model gate, so a quick restart could be refused with model-busy. Pre-pass chunks now publish progress, and a cancel between chunks keeps exact counts. Part of #379 * fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379) The live check measured WD14 on CUDA at 0.167 s per image (100 images in 16.7 s with the model load), so the review estimate uses 0.17 s. A finished or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X min left"; it now ends with "<verb> <state> · N files processed". Part of #379 * test(browser): reload the admin page only after the in-place edit saved The gallery-to-admin workflow pressed Enter and reloaded as soon as the edited text showed; the page saves after showing it, so a reload that won the race abandoned the PUT (WebKit, run 40564511000). The test now waits for the save's answer. Part of #379 * test(tagging): the batch coverage row names the renamed #379 test |
||
|
Vito
|
52a00370ea |
test(system): prove Diagnostics & logs against the concept and every legacy System capability's owner (#329, PR 3) (#387)
* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * test(system): compare Diagnostics & logs with the concept System › Diagnostics & logs and its log cleanup review are compared with the concept (app.js 694–695 and maintenancePlan("rotated logs")) in both themes at 1600x1100 and 390x844, with D109 and D110 (and D100 for the review) as the only mapped differences, a 320-wide select check and a mutation check. The fixture gains a concept seed mode (the concept's four log lines, a 1.4 MB active log with three rotated copies, console logging off), and the plan dialog's measureReview and mapPlanConcept move unchanged into concept-inventory.mjs for Maintenance and this review. The comparison found that the Database line read a report key that does not exist (database_info.size_bytes) and so always showed "0 bytes"; it now reads database_size_bytes, the key get_database_info() returns. Part of #329 * test(system): prove every legacy System capability has an owner TESTING.md gains "Legacy System capability coverage": every System capability of the legacy admin and gallery pages with its anchor on main, owner, status (covered, kept in legacy or approved removal), native control and the exact title of the test that proves it. tests/vendor/legacy-coverage.test.mjs (with tools/vendor/legacy-coverage.mjs) fails on a row with no owner, an unknown status, a malformed native control, a control on a row that is not covered, or a test title its file does not declare; a title only in a comment does not count. workspace-system.spec's "every covered legacy System capability has its native control" opens each covered row's section and finds the control by role and accessible name. Versus the design's table: the anchors are re-checked against main, "Run health checks" is on Maintenance (D123), not Overview, Diagnostics run is proven by "Run checks opens the shared Health report", and the export rows repeat their test title instead of "(same as above)". Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 * test(system): require a running test and a control for every covered capability Review fixes for the legacy coverage proof: - A title declared only with test.skip, .fixme or .fail no longer proves a row (.only and .slow still do). - Every covered row names its native control, except the rows in CONTROL_EXEMPTIONS, each with its reason (today only the gallery per-image auto-tag row, owned by the Images selection bar). The browser test asserts that the covered rows without a control are exactly the exemptions. - JavaScript comments are stripped across the whole source, wherever a block comment starts, skipping strings, template literals and regular expressions. A Python def test_ counts only at function or method indentation and outside comments and docstrings. - A control name several rows share ("Review plan", "Review options", "Refresh") is looked up inside the row the table names (· in "<row>"). - The diagnostics comparison records each select's chosen option, so a wrong default level is caught. Part of #329 |
||
|
Vito
|
bdc38422a9 |
feat(system): Diagnostics & logs natively, reviewed log cleanup and Clear all logs (#329, PR 2) (#385)
* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 |
||
|
Vito
|
a985af506b |
test(browser): seed resets on a scratch copy and publish them in one commit (#384) (#388)
* test(browser): seed resets on a scratch copy and publish them in one commit (#384) The concept and plain resets made hundreds of committed transactions on the live database; on a slow CI disk they queued behind WAL checkpoint syncs and took 30+ s. Each reset now copies the live database into a scratch database (backup API, no syncs), runs the unchanged seeder against it through an API proxy, and replaces the rows of the tables the seed changed in one live transaction. A commit by another live connection meanwhile fails the reset. tests/browser/reset_equivalence.py proves every reset kind leaves the same rows, files and response as the old path (PM_TEST_SEED_ON_LIVE=1). * test(browser): empty media folders by a rename, delete them in the background (#384) With the database commits gone, throttled resets still stalled in rmtree: unlinking the last fixture's files waits for their pending writeback. Each reseed now moves the old output and input folders into <root>/reset-trash and makes empty ones; a background thread deletes the trash, and the server's cleanup stops it and removes the rest before the temp root goes. * fix(test): recheck the live data version under the publish's write lock (#384) A writer whose transaction was open while the seed ran committed before BEGIN IMMEDIATE returned and its rows were overwritten. The publish now rechecks data_version under the lock, and restores foreign_keys and detaches the scratch database on every error path. * test(browser): keep the reset scratch database in /dev/shm when there is one (#384) Throttled runs still waited on the runner's disk while creating and rewriting the scratch database, whose pages were under writeback. * test(browser): keep the scratch on disk when /dev/shm is short, delete in place after cleanup (#384) The largest scratch copy is about 8 MB; Docker's default /dev/shm is 64 MB and shared with the browsers. A reset that empties a folder after the server's cleanup closed its trash deletes in place instead of starting a thread nobody stops. * fix(test): drop the gallery caches only after the reset's rows are published (#384) The seeders drop the Images listing at their end. Run on the scratch copy, that drop came before the publish, so a background walk in between could cache per-file values built from the live database's old rows: CI's inspector tag chip found 0 of 0 images. The proxy now records the drop and the seeder makes it after the publish, as the old reset did. * fix(test): remove /dev/shm reset scratch left by killed browser servers (#384) A server killed outright never runs its cleanup, and tmpfs keeps the scratch folder in memory until reboot: the Python suite's server tests left one per killed server. Scratch folders now carry the owner's pid, and each new one first removes those whose process is gone (POSIX only). |
||
|
Vito
|
a0fc5171e3 |
fix(inspector): pass text through unknown nodes instead of refusing them (#381) (#383)
* fix(inspector): pass text through unknown nodes instead of refusing them (#381) The inspector refused any node class missing from the semantics table, so a PromptManagerPair behind an rgthree Any Switch showed "Prompt uses a node the inspector does not support". The inspection resolver (Python and browser) now reads through unknown nodes: a text input is the text, positive/negative route by output slot, a conditioning input is followed alone, otherwise every linked input except core type inputs (pass_through.skip_inputs). PromptManager nodes are authoritative; several different texts become one candidate each ("Several prompts reach the sampler"). A seeded text node with no conditioning input is generated at run time, and a recorded PromptManager execution is not checked against text that passed through an unknown node. INSPECTION_RESOLVER_VERSION goes to 2; ingestion keeps the default profile. A legacy parity gate records every committed graph whose inspection positive matches the legacy parser's; no graph may leave it. * fix(inspector): keep recorded pair text, string-only seed rule, role inputs and unique candidates (#381) Review of the pass-through: a walk through an unknown node never overrides a PromptManagerPair's execution record (any outcome but work-limit is unverifiable while the record is checked); the run-time seed rule reads only string outputs; a node with positive/negative inputs follows only the role of the requested slot, nothing when it is unlinked; one text offered twice is one candidate. * fix(inspector): anchor on the saver that wrote the file and resolve stages on shared links (#381) An inspected file's name now anchors the inspection on the one saver whose filename_prefix (literal or traced) makes the file's stem <prefix>_<digits>[_]; the samplers in that saver's image ancestry are the candidates. No match, several matches or an unreadable walk anchor nothing, so the file stays "several possible prompts". In a latent walk, an earlier sampler reading the same conditioning links as the later one is the same prompt, not an unattributed stage. Python and the browser resolver behave the same; the legacy parity gate states each graph that differs from legacy on purpose. * fix(inspector): skip only plain first stages on shared links; an unfinished prefix scan anchors nothing (#381) Review of the saver anchor: a same-links earlier stage is skipped only when it is not a detailer, sets no prompt override and continues no other sampler, so an earlier prompt or override behind it is never hidden. A work limit while tracing a saver prefix ends the match scan without an anchor instead of counting that saver as no match. * test(inspector): cover every plain-first-stage outcome on both resolvers (#381) The frontend coverage gate found local-resolver-stages.mjs branches at 92.44% (< 93). Each way an earlier stage stops being a plain first stage (no latent, literal, missing or cyclic latent, a sampler-shaped node, a detailer) is now a resolver test and an anchor-first-stage-* parity vector. |
||
|
Vito
|
0652beba90 |
ci: run the required workflows for the merge queue (#382)
* ci: run the required workflows for the merge queue main no longer requires a branch to be up to date before it merges, so a PR no longer re-runs CI after every other merge. The Tests, Code Quality and Viewer.js workflows also run on merge_group, so the merge queue can test each PR against the latest main once, in order. The tracked protection record now says strict: false. * test(history): give waits that span a database restore their own ceiling The merge queue's Windows py3.12 job failed test_restore_or_generation_rotation_between_transactions_discards_evidence (restore=True): the restore it waits for, a full verify/migrate/backup/ replace cycle, took over the 10 s BLOCK_SECONDS on a busy runner. The two waits that span a restore now use RESTORE_SECONDS (90 s), which stays under the suite's 120 s hang watchdog, so a real hang still fails with its thread dump. Behavioral waits keep BLOCK_SECONDS. * test(history): drain the writer before every bounded close or dispose The merge queue's Windows py3.12 job failed test_saturated_start_cannot_capture_outputs_or_retain_previous_roots: dispose(2) ran while the writer still worked through 1,024 queued commands on a busy runner. Like #370, the test now drains first; the 2 s bound is not what it tests. Four more closes that follow a released block get the same drain, so the next slow runner can't fail them one at a time. * ci(comfyui): run the real-host job in the pinned Playwright image The real ComfyUI compatibility job installed Chromium with `playwright install --with-deps`, which pulls the browser's system packages from Ubuntu's apt mirror on every run. Three runs (#349 run 40367579020, #378, #382 job 122287230985: "Fetched 19.7 MB in 12min 23s (26.5 kB/s)") spent 11-12 minutes there and were cancelled at the 24-minute limit after all 89 tests had passed. The job now runs inside mcr.microsoft.com/playwright:v1.64.1-noble, the same image as the frontend coverage and page workflow jobs, which ships Chromium and its libraries, so the job has no apt step at all. The whole job moves rather than only the browser part: the Playwright suites launch the managed ComfyUI host themselves (PM_TEST_PYTHON), so splitting the browser off would mean running the host on the runner and the browser in a container across a network boundary, a larger change than moving one job whose host is plain Python and pip wheels. Python follows the other image jobs: the image's own Python 3.12 with a user-space venv bootstrapped from a checksum-pinned pip wheel (toolcache setup-python builds do not run in the container). A step fails the job when the image Python differs from pythonVersion in tools/vendor/comfyui-runtime.json, so the pinned host stays honest; torch, torchvision and ComfyUI's requirements install from the same pins as before. pip and npm download caches replace setup-python's and setup-node's. tests/vendor/browser-image.test.mjs now checks this job like the other browser jobs: image tag equals @playwright/test in the lockfile, no browser install or apt, no setup-python/setup-node, checksum-pinned pip, Node 24, and the runtime Python check. The 24-minute limit is unchanged. |
||
|
Vito
|
f35e010f79 |
feat(system): log cleanup rule, reviewed rotated-log cleanup and diagnostic report (#329, PR 1) (#367)
* fix(logs): delete and download only rotated copies under one rule Rule R (#329): a file is a rotated copy only when it is a direct entry of the log folder named prompt_manager.log.<n> (n >= 1, no leading zero, compared after normcase) and a regular file that is not a symlink or a junction. Everything else is never deleted. - Clear (POST /logs/truncate) empties the active log and deletes only rotated copies; a .bak, a .1.gz, any other file and every link are kept. It runs off the event loop under one files lock and the file handler's lock, so no rollover renames a file mid-way, and logs only afterwards. - delete_rotated(expected) deletes exactly the reviewed copies, or nothing when one of them changed (size, mtime_ns, inode, missing, now a link). - The download serves only a listed regular, non-link file, opened without following links and read off the loop; a link or folder is a 404. - /logs/files rows gain kind; files and stats read links with lstat and stats no longer count a link's target. - The config update runs off the loop and swaps handlers under the files lock. * feat(logs): review and apply rotated-log cleanup as a maintenance plan POST /maintenance/preview {operation: "rotated_logs", options: {}} lists the rotated copies Rule R allows, with size and time, and what it keeps (the active log, links, other files); it writes nothing and answers the Plan shape with basenames only, backup policy none and no disk key. POST /maintenance/apply starts the "Rotated log copies" maintenance job (admission read: runs beside a scan, refused beside another maintenance job or an exclusive one). It deletes exactly the reviewed copies, or nothing and fails plan-changed when any of them changed after the review; a copy that cannot be deleted becomes an errors row. The token is consumed on the loop turn the job starts. (#329) * feat(diagnostics): add a shareable report without secrets or paths GET /system/diagnostics builds the report on the server, off the event loop and under a registry lease (409 busy while a restore runs): the project and Viewer.js versions, Python and the platform, the four legacy checks (one helper, legacy_checks, that GET /diagnostics also calls), public database info, source counts, the log configuration and statistics, the last health summary and the config.json-only keys from an explicit list. The CivitAI key is reported only as has_civitai_api_key; every path is relative to ComfyUI or reads "outside ComfyUI". A scrub replaces anything still absolute, the key or the home folder and counts it in redacted (0 for a correct build). The version parser moves unchanged to utils/version.py. (#329) * fix(logs): never empty a renamed log or serve a swapped file; keep root anchors private Review findings on #329 PR 1: - Clear empties the active log through the handler only when the handler's open stream is the listed file (same inode); a log the user renamed while ComfyUI runs is no longer emptied. - A download is refused when the opened file is not the one listed (inode), or has another name (a hard link). - Windows reparse points that name another file (the name-surrogate bit: symlinks, junctions, WSL links) are links under Rule R; cloud placeholders stay regular files. - The report's paths use the closest anchor that is not a filesystem root, and the scrub also catches the home folder without its root. * fix(diagnostics): never anchor a report path above the home folder; serve only regular log files Review findings on #329 PR 1: - report_path never uses a filesystem root or a strict ancestor of the home folder (/home, C:\Users) as its anchor, so a path inside home needs an anchor at or below it and otherwise reads "outside ComfyUI"; no path starts with a user name. The scrub matches the home folder only as whole path parts (in full, without its root, and its name). - A download serves only a regular file (a FIFO, socket or device is a 404) and opens it non-blocking. - One unreadable timestamp no longer breaks the log listing, stats, download, review or Clear. - The report tests read a temporary log folder, never the process's own. * fix(diagnostics): match the bare home folder name only inside paths A home folder named like a word (/home/info) no longer removes the log level INFO from the report; the name still goes when it is a part of a path. Documents the accepted residual risk of another user's home. (#329) |
||
|
Vito
|
d868745194 |
fix(lora): cache LoraManager's layout and close the #355 follow-ups (#362) (#364)
* feat(lora): cache LoraManager's layout behind file signatures (#362, phase 1) A new py/lora_layout_cache.py keeps one (install, layout) entry and reuses it while the configured path, the plugin settings file's and the settings file's signatures (st_mtime_ns, st_size), the install and every media root's real path are unchanged and the entry is younger than 30 s. A load whose watched file changed within 3 s (git's racy rule), could not be checked, or whose settings were unreadable serves its request only. clear() bumps a generation so a load that started earlier stores nothing. Hits take no lock. Not wired yet; phase 2 puts it behind the image hot paths. * fix(lora): treat an offline Windows share as unvalidated and make cache tests portable (#362) Review findings on phase 1: Windows raises FileNotFoundError with winerror 53/67 for an offline share, which is a temporary failure, not a missing file. Test times are whole milliseconds (NTFS keeps 100 ns steps) and a file system that rounds them skips; cleanup no longer ignores errors; tests added for ValueError, NotADirectoryError, a failed reload, resolve failures and the watched-file set. * perf(lora): serve the image hot paths from the layout cache (#362, phase 2) _lora_image_dirs takes (install, layout) from lora_utils.cached_lora_manager_layout instead of resolving the install and reading LoraManager's settings files on every Library, inspect, reveal and serve request. serve_image builds the allow-list in the executor instead of on the event loop. A finished import (System job or legacy scan, both through replace_lora_imports) clears the cache, so files linked under a folder LoraManager just created are served at once. The integration-off check and the configured-path rule still run before the cache. * fix(lora): fingerprint every scanned sidecar, kept or skipped (#362, phase 3) Discovery keeps every metadata file it saw (scanned_files, resolved and sorted; scanned is now its length). The fingerprint hashes each one's path, size, modification time and whether it was kept, so a skipped sidecar added, removed or edited between the review and Start refuses Start with plan-changed, as the review's counts promised. * fix(lora): warn about refused default LoRA folders and log what failed (#362, phase 4) ComfyUI's default LoRA folders already pass the folder guard; a refused one that exists is now listed in the layout's refused_defaults, and the layout cache warns once per newly refused folder. The cache writes one INFO line when a stored layout differs from the last one logged. BAD_SIDECAR_ROOT and the module docstring name a folder that contains the home folder. A relative preview_url writes a DEBUG line before the suffix search; a non-OSError failure while loading trigger words logs its traceback. Review follow-ups harden two serve tests. * fix(lora): skip a sidecar that is not UTF-8 instead of failing the load (#362) Review follow-up on phase 4: read_lora_metadata caught JSONDecodeError but not UnicodeDecodeError, so one non-UTF-8 sidecar failed the whole trigger-word load (now logged with a traceback every retry) and the import review. It is skipped like any unreadable sidecar. The relative preview_url DEBUG line uses lazy formatting. * docs(testing): the LoRA linked-only gate, layout cache and fingerprint scope (#362, phase 5) TESTING.md's LoRA reference import contract now states what #358 and #362 shipped: the linked-only gate and _media_roots, linked_only media, the cover link order, the guard on ComfyUI's default LoRA folders and refused_defaults, the folder-unreadable code, the trigger cache's catch and 60 s retry, the fingerprint over every scanned sidecar, and the layout cache's key, signatures, racy window, maximum age, root re-check and clears. The targeted command includes tests.test_lora_layout_cache and tests.test_api_images. * fix(lora): a cached layout needs the same home folder; give the rebuild poll its time (#362) Final review follow-ups: the folder guard compares against the home folder, so a cache hit now also needs Path.home() unchanged. TESTING.md says the configured-path rule runs on the cache's result and that a non-UTF-8 sidecar is skipped. CI: 'Rebuild all is explicit...' (webkit) polled the job for the default 5 s while it regenerated 24 previews; the failure screenshot shows 12 of 24 done at the deadline on the busy runner (0.17 s locally). The poll now allows 30 s for that real work. |
||
|
Vito
|
1bb3678f32 |
fix(threads): describe the teardown as built, warn only for a live observer, and keep an interrupted teardown's result (#377) (#378)
- _stop_background_work pre-fills the started parts with False and stores the result and removes its atexit entry in a finally:, so a Ctrl+C between steps no longer leaves the result unset and the callback registered. - The "watchdog observer still running" warning moves into _stop_observer: only for an observer still alive after its join, naming the join's own timeout (not after observer.stop() raised, and not the full deadline). - Tests: the plain-teardown tests share one package; two teardown tests run with a live preview worker held in a fake encode (ffmpeg killed, step gets min(CLOSE_WAIT_SECONDS, time left), thumbnails False while held and True when the kill releases it); an interrupt-between-steps test; two warning tests. - TESTING.md and the ledger fixture comment: the refusal comes first, the image worker and the observer share one 5 s deadline, and what an interrupt between steps does. Closes #377 |
||
|
Vito
|
2f67b98287 |
fix(tagging): save WD14 confidences only on the 0.05 grid every page shows (#369) (#373)
* fix(tagging): save WD14 confidences only on the 0.05 grid every page shows (#369) A saved threshold off the 0.05 step (for example 0.37) blocked Generate and Start in the workspace Tag image dialogs (native step mismatch) and made the legacy sliders send 0.35. The server now refuses such a save with a message that names the setting, a stored one falls back on load, and an accepted one is stored exactly on the grid. System > Auto-tagging refuses the same values before Preview tagging review or the batch Review, and hands thresholds on snapped to the grid. Per-request thresholds keep their own checks. * test(browser): restore the safety backup policy on system reset The Recovery policy test in workspace-system-backups saves safety_backup false at runtime, and /__test__/system/reset restored config.json but not PromptManagerConfig.SAFETY_BACKUP. A later spec on the same server (Tag output images review) then saw "No safety backup (the policy is off)". The new #369 test shifted the CI shards so the two met on one worker. |
||
|
Vito
|
f3b9f61f11 |
fix(threads): stop background work at exit, on cleanup and on the tests' unload (#301) (#374)
* fix(threads): stop background work at exit, on cleanup and on the tests' unload (#301) - __init__.py: _stop_background_work() detaches the monitor's callbacks, stops the monitor, closes the preview worker and shuts the prompt cleanup thread down against one 10 s deadline, once. It runs from atexit (ComfyUI never runs aiohttp's cleanup hooks), from on_cleanup where a host runs it, and from the tests' package unload; it removes its own atexit callback. Registered after the history hook, so at exit it runs before the history dispose. - AutoThumbnailer.close/close_auto_thumbnails take a timeout and return whether the worker ended. - Tests: teardown result, once-only, on_cleanup, atexit order, a failing step or a second Ctrl+C never skipping the tracker, one shared deadline, three repeated loads leaving no thread, and a child process that exits normally with the image worker alive. * fix(threads): refuse new previews before the monitor stops; isolate the exit child (#301) Review findings: - The teardown makes the API refuse new previews before it detaches the monitor's callbacks: an image in hand that already read the preview queue no longer starts a preview worker during exit (Python 3.12+ refuses that thread and logs a traceback). New refuse_auto_thumbnails(). - The teardown imports its bounds inside each step and falls back to the standard logger, so a failing import never skips the other steps. - The exit child keeps its library and log in its own folder (PROMPT_MANAGER_DATA_DIR), reports watchdog's threads too, fails with the child's output on a hang, and rejects "Error while stopping". - TESTING.md: "Background threads at exit". |
||
|
Vito
|
b1e7abd380 |
fix(threads): stop and join the prompt cleanup thread, the image worker and the observer (#302, #370) (#372)
* fix(threads): stop and join the prompt cleanup thread, the image worker and the observer (#302, #370) - PromptTracker: the cleanup thread waits on a stop event instead of time.sleep; new shutdown(timeout=2) joins it and returns whether it ended. - ImageGenerationHandler.shutdown(timeout=5) now joins the worker and returns a bool; new close() is final: schedule() refuses and a sealed run's history re-check is dropped, so no worker restarts afterwards. - ImageMonitor.stop_monitoring(timeout=5) closes the handler first, then stops and joins the watchdog observer within one deadline. - Every test that builds a real tracker, handler or monitor ends with its final stop and an assertion, before its folder and databases go. - #370: the wrapper/session history test drains before dispose(); the held-write bound test uses 30 s ceilings instead of tight wall clocks. * test(history): drain the writer before asserting a bounded close it does not test (#370) * fix(threads): a later stop waits again for a worker or observer an earlier stop left running (#302) Review findings on #372: - ImageGenerationHandler keeps the workers a stop signalled that had not ended; every later shutdown/close joins them again, so a second call no longer returns True while the first worker still runs. - ImageMonitor.stop_monitoring keeps an observer or handler it left running and stops and joins it again on the next call. - A closed handler takes no awaiting history path, and a re-check that raced close() is not kept due. - Tests: second stops of a busy worker and a live observer, the closed handler's re-check paths, the handle test asserts the worker ended and stops before it asserts, and a no-new-thread check instead of set equality. * test(browser): the System reset restores every settings value, safety_backup included The frontend coverage shard of #372 failed in workspace-system-tagging: the review read "No safety backup (the policy is off)". The backups spec saves safety_backup=false, and /__test__/system/reset restored config.json but not PromptManagerConfig.SAFETY_BACKUP (nor INFINITE_SCROLL), so a later spec on the same worker server saw the policy off. The reset now captures and restores every non-tagging value POST /settings can change. |
||
|
Vito
|
ae755d2ce0 |
feat(system): Auto-tagging screen, native in System (#328, PR 3) (#368)
* feat(system): configure auto-tagging natively * test(system): compare Auto-tagging with the concept against the real backend |
||
|
Vito
|
4032f4b6ab |
feat(system): one tagging configuration and Tag output images as a reviewed job (#328, PR 2) (#366)
* feat(autotag): model status without paths, and a busy answer that names the operation * feat(autotag): model downloads run as Activity jobs * fix(autotag): keep each download route's error shape and harden the fake download * feat(autotag): one server-side tagging configuration * fix(autotag): the legacy batch and review work again * feat(autotag): tag output images as a reviewed Activity job * test(autotag): the legacy batch, review and unload page workflows * fix(autotag): server-free tagging defaults, cancel before load, and named legacy refusals * fix(autotag): name only the workspace batch in a legacy refusal frame |
||
|
Vito
|
88fa13689a |
feat(system): model status, a named model gate and downloads as Activity jobs (#328, PR 1) (#365)
* feat(autotag): model status without paths, and a busy answer that names the operation * feat(autotag): model downloads run as Activity jobs * fix(autotag): keep each download route's error shape and harden the fake download |
||
|
Vito
|
cfe47d3cb0 |
ci: support and test Python 3.12-3.14 only (#356) (#357)
* ci: test Python 3.11-3.14 and require Python 3.11 (#356) Python 3.10 reaches end of life in October 2026 and no PromptManager users run it; 3.14 is current and was untested. The test matrix runs 3.11-3.14 on Linux, Windows and macOS, bandit runs on 3.11, and requires-python is >=3.11 so the package no longer claims a version CI does not test. The packaging test drops its 3.10-only tomllib fallback. The required checks follow the matrix (py3.10 -> py3.14). * ci: narrow Python support to 3.12-3.14 (#356) ComfyUI recommends 3.13, names 3.12 as the fallback and 3.14 as working (its README); PromptManager supports exactly those. requires-python is >=3.12, the test matrix and the required checks drop 3.11, and bandit runs on 3.12. |
||
|
Vito
|
fd949b870c |
test(admin): wait for the restore response, not a timer (#359) (#363)
The legacy restore specs "restore validation and cancellation preserve live data and allow a valid retry" and "backup and restore recover from interrupted network requests without losing data" ended with a 5 s toBeHidden on the restore modal. The modal closes only after the server answers, so that was a 5 s deadline on a disk-bound restore, which WebKit CI runners sometimes exceed. Both specs, and the downloaded-backup spec that already waited on the response, now share confirmRestoreAndWait: it waits for the POST /prompt_manager/restore response, asserts a 200 (showing the body on failure) and the success body, then checks the modal closes. TESTING.md explains how to reproduce disk-bound timeouts with --device-write-iops in the pinned image. |
||
|
Vito
|
80c6ec5340 |
fix(lora): find LoraManager's sidecars, previews and example images in every storage configuration (#355) (#358)
* fix(lora): read LoraManager's settings and storage layout the way it does (#355) Add py/lora_manager_layout.py: the settings file in LoraManager's lookup order (environment folder, portable plugin file, user config folder), only the seven keys the import needs, the LoRA roots, the centralized sidecar root, the example-images root and the active library, each folder through one guard. Sidecar discovery for both storage modes and the local preview and example-image lookup live there too. The test runner and the browser test server point LoraManager's settings lookup at a disposable folder, so no test reads the real settings. * fix(lora): import the sidecars and local images LoraManager really uses (#355) The import, the legacy scan, the trigger-word cache and the image server now share LoraManager's layout: sidecars beside the models (alongside) or under the sidecar root (centralized), kept only when their model exists inside a LoRA root. Each LoRA links its LoraManager preview and example images where they are; CivitAI downloads run only for a LoRA without any local image. The review adds skipped, with_local_media and the layout, and offers no token when LoraManager's settings cannot be used; the legacy scan reports the same reason without replacing anything. find_lora_directories becomes default_lora_roots (no scan of the LoraManager install); the preview and example-folder helpers it replaced are removed. The trigger cache never raises into a running node. * fix(lora): never serve a LoraManager folder that holds the home folder (#355) The folder guard also rejects ancestors of the home folder (/home, C:\Users), compared in the realpath + normcase form, so a LoraManager setting cannot widen the image server's allow-list to every user's files. * fix(lora): show linked LoraManager videos as videos (#355) _enrich_images marks every image without is_video by its extension, so an .mp4 preview outside the output folders shows the Video preview label and plays in the inspector. Tests cover serving the sidecar and example-image folders only while the integration is on, and a filesystem root as the sidecar folder adding nothing. * feat(lora): the import review names the LoraManager settings it read (#355) The review gains a first row, LoraManager ("<source> settings · <mode> storage · library <name>", the settings file as its hover title), and its Found row counts metadata files, LoRAs and LoRAs with local images. A plan whose LoraManager settings cannot be used takes the close-only path: "LoraManager settings: not usable: <reason>. Nothing was changed." The browser fixture writes each LoRA's model file and can leave out the local preview, so the download, gate and cancel specs still exercise CivitAI. * docs(lora): describe how the import finds LoraManager's files (#355) * test(lora): set the settings folder relative to the process folder (#355) os.path.relpath fails across Windows drives (the temp folder on C:, the checkout on D:); the test now changes into its temp folder instead. * fix(lora): review findings for the LoraManager layout (#355) - The example-images library subfolder goes through the folder guard like its root, so its canonical (normcased) form matches the media check on Windows for a library name with capitals. - A failed trigger-word load is retried by a running node only after TRIGGER_RETRY_SECONDS, not on every prompt run. - A review whose discovery cannot list a folder says so (folder-unreadable, the folder's name only) instead of a generic error. * test(images): hold the listing clock in the deferred-walk test test_a_deferred_walk_waits_at_least_half_a_second set the previous walk 29.99 s back against a 30 s interval and then read the real clock again: a runner slower than 10 ms between the two lines passed the interval and the walk started at once (failed on macOS 3.13 and Windows 3.11). The module's clock is now held for the mark, so the deferred walk is always the one measured. * test(lora): write the unkept-settings fixture as raw text CodeQL read the dict with a credential-named key flowing into write_text as clear-text storage of a secret. The value is a marker, so the fixture now writes the JSON text directly and names the value MARKER. * fix(lora): link each LoRA's cover as its newest image (#355) Workspace lists show a prompt's media newest link first, so the import links a record's images last to first: the card shows the cover (a still in front of a leading video) the import put first. * fix(lora): show LoRA-linked media on the Library page and in the inspector (#355) Library previews, the positive detail and the per-prompt images page list a prompt's linked media under the output folders plus, while the LoRA integration is on, LoraManager's guarded folders and the download cache (_media_roots; output root indices keep their meaning). That media keeps its rooted identity, so Info and the media viewer open it; the path-based serve route answers for those roots only with a file the database records (either path spelling), never serves or makes a preview there, and output requests never read LoraManager's layout. Such media is marked linked_only: no preview file, no favorite star, no Tag action. Show in folder accepts those roots; the download cache is named, not created. * fix(lora): expose only linked files under LoraManager's folders (#355) Serve, inspect and Show in folder share one gate for a LoRA-linked root (_media_roots past the output folders): a file the database does not record as a linked image is refused with 403, checked by its given and its real path. Before, inspect and reveal read any file under those folders by path. * fix(lora): check the file Show in folder would open under a LoRA root (#355) reveal_image resolved the requested path through the client-path resolver, which can fall back to a path anchor; under a LoRA-linked root the file that would open now passes the same recorded-media gate, so only a linked file is ever revealed there. |
||
|
Vito
|
c7ba481d3c |
perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) (#353)
* perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) The image monitor now reports every settled output (linked or not, also when linking raised) with its resolved path through set_output_listener. The API's notify_output_settled hops from the monitor's worker thread onto the server's loop (call_soon_threadsafe) and does nothing before the loop is known or after cleanup. mark_root_dirty matches the path against each walk's resolved_root (normcase(realpath), computed in the executor by walk_root, now its fourth value) and marks the root dirty. A dirty root gets one background walk, no sooner than rewalk_interval = min(15, max(5, 5 x last walk's duration)) after the previous walk started (last_started_at / last_duration are recorded on every walk). _schedule_walk is shared with PR 1's delayed stale refresh: one deferred walk per root, never armed while a walk runs (the walk's end re-checks dirty: the trailing walk), at least 0.5 s away, the earlier deadline wins, and every walk start cancels it. A failed walk clears dirty. Rescan semantics are unchanged. * fix(images): keep removed roots and cancelled walks from walking again (#348, PR 2) Review findings on the monitor-triggered walks: - reset_gallery_roots_cache now retires every listing record: its pending timer is cancelled, dirty is cleared, and no mark, timer or trailing walk starts a walk of it until a reader or the warm-up asks for that root again. Before, a timer or trailing walk pending when the roots changed stored a listing for the removed root, and the monitor (which watches the old roots until restart) kept it walking on every new output. - A walk task cancelled outside cleanup_gallery_listings no longer starts a trailing walk while the loop shuts down. - root_contains ignores a Windows extended-length prefix (\\?\ and \\?\UNC\) on either side, which realpath can keep on a file past MAX_PATH. - The REWALK comment states the ~20% GIL share only for walks up to 3 s. * test(images): prove a cancelled walk starts no trailing walk (#348, PR 2) The previous assertions held even with the trailing walk, because that walk finished before they ran; assert the walk number and the scan count. * perf(images): add a settled output to the listing without a walk (#348, PR 2) Live test: each landing in a 20,000-file folder cost a full walk (~0.4 s of GIL) plus a full workspace rebuild, so Images reads during a run took up to 2.1 s. The monitor's report now patches the stored listing instead. - output_settled (the monitor hop's target) slices the file's part below the root in its own case (relative_part; Windows compares folded), and _run_adds places one root's pending files in one executor batch: _settled_entry applies the walk's filters (suffix, thumbnails, depth, symlink by lstat; macOS on-disk spelling), insert_entry puts the (Path, mtime) entry where the newest-first sort does, add_folder copies the folder sets. Stored only when no walk started and no drop happened meanwhile; walked_at is untouched, so the 30 s refresh still walks. - Anything inexact marks the root dirty and walks as before: a running walk, an unsliceable path, a failed lstat, a tie in mtime, the cap, an unexpected error. Rescan, the refresh, the warm-up and invalidation stay full walks; a walk start forgets the pending adds. - The workspace caches follow a patched listing by entry identity (listing_delta): only added values are built, the newest order places them by bisection, the favorites lookup is copied and extended. Synthetic 20,000 files: per landing 252 ms (walk + full build) -> 18 ms (3 ms add + 15 ms warm build). * fix(images): review fixes for adds without a walk (#348, PR 2) - _newest_step steps the cached newest order only when the per-file cache still holds exactly the values this call built. A stale page read racing a newer warm build (both run in executor threads) stored an order labelled with older values than it held, and the next read inserted the new file a second time. It also sorts in full when a roots reset dropped the per-file values meanwhile. - The stored spelling is checked on Windows too (nt._getfinalpathname: realpath keeps the given case when it cannot open the file), and before the name filters, which compare the walk's stored spelling. macOS opens non-blocking, so a FIFO named like an image never holds the executor. - An output in a folder the walk cannot list (no read permission) is not added, as the walk skips it. - Windows-portable unit tests: POSIX paths in the insert tests, and a str.lower fold for the length guard (Windows' normcase keeps lengths). * test(images): reproduce the newest-order race and fix a Windows clock flake (#348, PR 2) - The race test now has the newer build step from the values the cached order was built from, which is the interleaving that inserted a file twice; it fails without the parts check in _newest_step. - test_a_failed_walk_records_its_timing_and_clears_dirty compared two monotonic readings that Windows' ~15.6 ms clock can make equal; the previous start is moved back instead. - TESTING.md: os.access ignores ACLs on Windows. |
||
|
Vito
|
9ac471eb72 |
perf(images): share and warm the Images listing (#348, PR 1) (#349)
* perf(images): share and warm one listing per output root (#348) The Images page, its sidebar counts, the subfolder list and the legacy gallery each rebuilt the output listing when the 30 s cache expired, all at once, and the subfolder list computed relative paths on the event loop. Thumbnail writes also threw the listing away. Keep one RootListing per output root (py/api/gallery_listing.py): - one walk per root at a time; concurrent readers wait for it (shielded) - a listing older than 30 s is served at once and refreshed in the background - the walk, the comparison with the previous listing and both folder sets run in the executor; an unchanged walk keeps the same list object - os.scandir probe: a missing root fails the walk and keeps the previous listing (one warning per failure streak) - Rescan waits for a walk started after the click and is a 500 when it fails, even with a previous listing - deletes keep the blocking invalidation; a generation counter that survives drops discards walks that raced it - thumbnail jobs, conversion and the legacy generate/clear routes no longer drop the listing - warm-up on the host's on_startup, cancelled on on_cleanup Adds tools/vendor/perf/images_load_probe.mjs (maintainer tool, not CI) and the TESTING.md section "Images listing cache (#348)". * fix(images): drop listings on the loop when invalidated from a worker thread (#348) A settings save (gallery roots change) and the browser-test seeds call invalidate_gallery_cache() from the executor. Listing records are only touched on the event loop, so an off-loop call now hands the drop to the loop and waits for it, keeping the call synchronous for its callers. The shared-Rescan test waits for the later Rescans to take their walk number instead of sleeping. * test(images): bound the shared-listing test waits (#348) A macOS py3.12 run stopped inside the shared-listing tests and sat until the 6 h job limit with no traceback. It does not reproduce locally (about 300 runs on 3.12 and 3.14, pinned to one core). Requests and background settles in TestSharedListing now give up after 30 s, so a recurrence fails with the stuck await instead of hanging the run. * test(images): let the settle loops yield so a finished walk leaves the set (#348) On CI (ubuntu and macOS py3.12) the runner's watchdog dumped test_a_root_that_recovers_warns_again_on_its_next_failure spinning in _settle: asyncio.gather over tasks that are already done returns without yielding, so the set's discard callback, queued when the walk finished, never ran and the loop never ended (wait_for could not fire either). Delaying discard by one loop turn reproduces it locally; both settle loops now yield one turn after each gather. * test(browser): poll the import review's rows instead of reading them once gridOf reads the dialog's rows once; right after the click the review may not have rendered them yet (firefox CI: received undefined). * perf(images): walk the output with scandir entries and build folders from strings (#348) The live test showed a read during a background refresh taking 2.2 s instead of 0.07 s: the walk held the GIL for about 2 s in the server. py-spy split that walk into two halves: Path.relative_to per file in folder_sets, and Path.stat + Path.is_symlink + Path joins per file in the scan. - _walk_media visits folders in os.walk's order with os.scandir entries: the file type comes from the listing (no lstat per file) and the mtime from entry.stat (one call, none on Windows). _iter_media_files is now a thin wrapper, so its other caller is unchanged. - folder_sets slices the path strings (in the path flavour's separator) instead of calling relative_to, falling back to it for a path that does not start with the root string. On the maintainer's 19,935-file output folder: scan 0.386 s -> 0.112 s, folder sets 0.452 s -> 0.007 s, both results identical to the previous code. A reference copy of the os.walk version checks order, filters, the depth cap, symlinks and the file cap. * perf(images): build the workspace caches behind the warm-up and changed walks (#348) The live test after the faster walk still took 2.9-5.2 s for the first Images open after a restart. py-spy showed the walk was now small; the time went to the workspace's own caches, cold after a restart and built by three parallel first requests at once: the favorites listing (Path.relative_to per file), the per-file values and newest order, and the prompt-link read. - Once the host's startup warm-up has run, a stored walk that changed a listing schedules one executor job that builds what a first page reads, through the request path itself (_select_workspace_page with the default filters, then _workspace_favorite_paths). A walk stored during a build asks for one more build; an unchanged walk (same list kept) builds nothing; a failed build only warns; on_cleanup stops it. Servers that never ran the warm-up (the test servers) start no build. - _favorite_listing slices the path strings as _listed does, falling back to relative_to for a path outside the root string. * perf(images): start a stale listing's refresh walk a second after the read (#348) After the faster walk, a read that found the listing stale still took 0.5-1.0 s against 0.07 s: it was served from the old listing but ran beside the refresh walk it had just started, sharing the GIL. The refresh now starts REFRESH_DELAY (1 s) after that read through a per-root rewalk_handle, so the page's own requests finish first. One pending refresh per root and none while a walk runs; any walk start (a cold read, Rescan, the warm-up) cancels a pending one; on_cleanup cancels them before its first await. * test(browser): accept KB or MB for the picked safety backup's size The restore preview showed '2.5 MB' for the test server's database copy in a CI coverage shard: its size depends on what earlier tests on that server wrote, and the formatter switches to MB at 1 MiB. * fix(images): keep the 3.10-3.13 suffix rule in the scandir walk and harden the warm-up (#348) Review of the walk speedup: - os.path.splitext drops names like ..png that Path.suffix listed on Python 3.10-3.13 (3.14 changed Path.suffix, which hid it from the parity test on a 3.14 venv). _name_suffix spells out the 3.10-3.13 rule; the test now asserts ..png and ...webp are listed and g. is not. - _warm_workspace reads the roots inside its try, so a failing _get_all_output_dirs warns instead of leaving an unretrieved task error. - The folder_sets fallback test now takes the fallback (a Windows case mismatch); the coalescing test pins the walk count. |
||
|
Vito
|
5772de8d27 |
ci: bound every job and make hangs name themselves (#350) (#351)
* test: run the suite through a watchdog that names a hang (#350) tests/run_suite.py discovers the same tests as unittest discover -s tests, with the same sys.path and exit codes (5 when none ran), writes 'start: <test id>' to stderr before each test and arms faulthandler.dump_traceback_later: 120 s per test (PM_TEST_HANG_LIMIT) and 120 s after the run (PM_TEST_SHUTDOWN_LIMIT), never cancelled. A hang in a test, its teardown, its event-loop shutdown or interpreter exit prints every thread's stack and exits non-zero instead of running to the job limit. The pre-push hook now runs it too. * ci: bound every job and run the Python suite through the watchdog (#350) Every job in test.yml and code-quality.yml gets timeout-minutes = ceil(1.5 x slowest of three green main runs) + 3, min 5, with the runs and times written above jobs:. The matrix and the coverage gate run python -m tests.run_suite. ci-workflow.test.mjs checks every job in the three CI workflows has a limit and that the matrix and coverage use the runner. Job names are unchanged. * docs: document the test runner, time limits and following a push (#350) * fix(test): keep unittest's warning filters and tighten the runner's edges (#350) Review follow-ups: the runner now shows warnings as unittest.main does (DeprecationWarning and ResourceWarning were hidden), rejects non-ASCII digit limits with the one-line exit-2 error, and starts each 'start:' line on its own line at default verbosity. The docs now say that class and module fixtures count toward the test before them, that the shutdown limit covers interpreter exit, and that exit 5 on zero tests is 3.12+ behaviour applied on every version. * test: decode the runner's stderr as UTF-8 on Windows too (#350) A child Python writes a piped stderr in the ANSI codepage on Windows, so the non-ASCII limit value in the usage-error test came back as U+FFFD. The helper now sets PYTHONIOENCODING=utf-8 for the child. * test(history): give the writer's non-blocking checks a 1 s ceiling (#350) py3.13 on macos-latest failed on PR #351 with writer.close(0) taking 0.186 s against a 0.1 s ceiling while another thread held the commit. The blocked path these checks rule out waits up to 3 s (release.wait(3)), so a 1 s ceiling still tells returning at once from waiting for the release, without failing on a loaded shared runner (TESTING.md, #242). * test(restore): wait for the cancelled preview's worker before teardown (#350) py3.13 on ubuntu-latest failed on PR #351 (run 40334794505) with 'Files still open at teardown: prompts.db, prompts.db-shm' in test_cancel_while_reading_the_live_library. Cancelling the preview cannot stop its executor thread, which goes on to open the live library; the test returned without waiting, so that open could land between teardown's close_all_instances() and assert_closed(). The test now waits for the worker, as test_cancel_while_staging already does. |
||
|
Vito
|
3be0ada6a1 |
fix(ingestion): link an image only to a prompt that provably produced it (#344)
* fix(ingestion): link an image only to a prompt that provably produced it An LLM fed a PromptManagerPair's prepend_text while its text editor was empty. The node refused to save (an empty positive raised and logged a warning), and the image monitor then linked the image to a stale tracker entry or to the newest prompt in the database. - Nodes: an empty text editor saves and tracks nothing (prepend and append text are never the stored prompt), logs at info, and marks the node's run key empty so no earlier entry can be used for its images. - Tracker entries carry a run key: a digest of the prompt node and its whole upstream graph, the identity ComfyUI's cache uses. The monitor computes the same key from the image's embedded graph and uses a queued or snapshot entry only when the keys match (a snapshot also needs a known run with one entry, so a batch's last item is never applied to every image). - The recency fallback is gone; anything uncertain stays unlinked and is logged at info. - Recorded execution history attributes a file when exactly one run reported it, the file was written during that run, and the run's positive contexts name one stored prompt. The monitor and the output scan use it as their last certain source. - The scan and pair ingestion treat a resolved pair with an empty stored positive as unresolved instead of raising. - tools/review_image_links.py reviews selected links by these rules; dry run by default, --apply changes only named ids after a backup. Closes #342 * test(ingestion): real-host attribution case and the #342 testing contract The managed host disables the image monitor, so the owned fixture pack gains a route that starts and stops the plugin's monitor singleton for the attribution case only. The case runs in both renderers: a typed Pair image links, a Pair image whose positive is only linked prepend text stays unlinked with nothing stored, an unknown node feeding text links through its run key, and its cached re-run links again. * fix(ingestion): take a queued entry only from the run the image appeared in Dropping a key's older queue entries whenever the node ran again let a later queued run, executing during the monitor's processing delay, take the earlier image's entry. The snapshot taken when the image appears now names the key's latest run, and the queue is popped by key and run: same-key entries of earlier runs ahead of it are stale, entries of later runs wait for their own images. Also keeps the attribution case out of the stock real-host suite. * test(ingestion): host lookups degrade to no proof * fix(ingestion): address the #344 review (history re-check, keys, repair) - Images without an embedded API graph: the scan now asks recorded history before skipping a file with no prompt metadata, and the monitor gives a keyless unlinked image one history-only re-check when its run's outputs are recorded (a seal notification from the outputs write; at most 1000 waiting images, each for an hour). - Run keys: an image is unkeyable when any positive node has no key, so a two-pass graph is never linked through the other pass; links follow ComfyUI's is_link exactly. - A failed save marks the node's run key like an empty editor, so an older entry of that key can't stand in for this run. - A placeholder is the latest entry: the image stays unlinked without falling through to history, the same rule as a popped placeholder. - An index on execution_run_outputs(filename, subfolder, root_key, path_style) for the history lookup, added by migration and accepted by the output schema check (any other index is still refused). - Repair tool: unlink removes only the link record and only while the image is still linked to the prompt it was judged on; every run prints what --apply does. - Real-host case proves the re-run's PromptManager node was cached. * test(ingestion): real-host bare-output re-check and the review-round contract A metadata-disabled save, held open behind the fixture gate after SaveImage, waits unlinked while its run is open and links once the run's outputs are recorded. The fixture's monitor route lists the waiting files. TESTING.md covers the history re-check, the read index and what the repair tool's --apply does. * fix(ingestion): batch history lookups instead of adding an index An index on execution_run_outputs would make the library unopenable by older builds, whose schema check refuses any extra index there. The migration and the schema-check allowance are reverted, so the history schema is exactly main's. stored_run_prompts instead looks up a batch of files with one parameterized filename IN (...) query per chunk of at most 500 names and matches in memory; the scan computes it once per scan batch. * fix(ingestion): address the #344 re-review (seal race, linked roots) - A sealed history re-check that can't be scheduled yet (the image's own pass still pending, or the pending set full) is kept due and scheduled when a pending image finishes, instead of being dropped. - History matches a file through ComfyUI's spelling of its output folder first, then with both sides resolved, so a symlinked or junction output folder still attributes the scan's resolved paths; waiting files are keyed by their resolved path. - The scan makes its batch history lookup only when a file needs it. - set_current_prompt pushes to the queue last, so marking a failed execution never adds a second queue slot. - A keyless image outside the output folder doesn't take a waiting slot. |