56d0a4904ca2dc43cbc60c35b5de1b95218fbd3e
179
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
56d0a4904c |
feat(collections): COLLECTIONS sidebar, collection page and saved views (#331, PR 2) (#410)
* feat(saved-views): keep a collection in a saved view Images saved views gain a collection key (#331 D140): empty or a collection id of 1 to 9 digits. Writes refuse favorites together with a collection; older rows read an empty collection, and a stored row with both opens the collection, as the listing does for a URL with both. Part of #331 * feat(collections): store collections in the library database * feat(collections): add, remove and rename collections and keep members through duplicate cleanup * feat(collections): add collection routes and the Images collection filter * feat(images): store images' own categories in the library database (#332) One new table, image_categories, and its category index, created empty by a forward-only migration that reads and writes only its own objects. Startup checks the structure; restore also checks the values. The store sets, clears and reads own categories by file key (the deepest output source holding the file and the path below it), and copies a removed duplicate's category to a kept copy that has none. * feat(images): duplicate cleanup keeps images' own categories (#332) The kept copy's own category wins; a kept copy without one receives the removed copy's, keyed by the deepest output source holding each file. The removed copy's row is never deleted, so a give-back or a failed delete loses nothing. The job log reports how many were kept. * feat(images): set, remove and list images' own categories (#332) PUT /prompt_manager/images/category sets the own category of up to 500 listed files, DELETE removes it (listed or not), and GET /prompt_manager/images/categories lists the values whose file is listed. Targets are the Images listing identity of an output source, checked by media_target (shared with Collections). The routes never touch prompts, recipes or links; POST /prompt_manager/bulk/category is unchanged. * fix(images): key symlinked output folders and keep key work off the event loop (#332) Review fixes: a source matches by its configured spelling too, so a detected output folder that is a symlink keys its files (routes and duplicate cleanup). File keys are computed in the executor. A path no source holds is a plain 403, a lone surrogate a 400. The install repairs a table that lacks its category index instead of refusing to open. * fix(collections): address review findings * feat(images): set or remove a category on all N matching images (#332) The category routes also accept the Images listing filter of "Select all N matching" (#392) instead of explicit items, resolved now and capped at 500 images like an explicit selection. * feat(collections): browse a collection from the sidebar The sidebar gains the concept's COLLECTIONS group (D136): one row per collection with its folder icon, name and listed-member count, without the PROPOSED badge; the label shows once a collection exists. A row opens #/collection/<id>, the Images page filtered to that collection (D137): the collection's name as h1, Collection as breadcrumb and title, Edit to rename or delete it through the saved views' Edit dialog with collection nouns, and a note counting members the sources do not list now. An unknown or deleted collection shows a page that says so with a link to Images. A saved view on a deleted collection lists nothing and says so (MC6, D140). The list is read again after a restore and the counts 300 ms after a change. Part of #331 * test(collections): compare the COLLECTIONS group with the concept The concept hides COLLECTIONS at 950 px and below, which would leave a phone no way to open a collection; production shows the rows there in the wrapped layout of every other sidebar nav (D136). The shell comparison now compares the group instead of filtering it out: the concept fixture seeds Landscapes (12) and Experiments (8), the rows compare by name order, and at phone width the nav compares against the concept's own sidebar-nav layout. The collection rows are checked at 1600x1100 and 390x844 in both themes. Part of #331 * test(vendor): count the #332 host case in the execution matrix image-attribution.spec.mjs gains the image category route case, so the execution config lists 92 tests (3 attribution cases per project). * test(browser): install the slideshow clock before wrapping setTimeout The slideshow fixture wrapped window.setTimeout before page.clock.install(), so the viewer's timer ran on real time and clock.runFor() did not drive it; a late WebKit timer failed 'slideshow exposes exact state labels' on CI. * fix(collections): address review findings - A list read superseded by a newer one (the 300 ms count read, a restore) no longer shows "could not be loaded": the collection and saved-view pages wait for the list to settle. - A listing that finds the open collection gone shows the deleted page once the list no longer holds it; when the list read fails or still holds it, the page keeps the listing's error and Retry instead of staying blank, and each failed read is reported again. - The missing-members note follows every list read and goes away at zero. - A rename that settles after its Edit dialog closed leaves any dialog opened since alone. - The deleted-collection, list-failed and saved-view failure pages focus their title after a route change, as every page does. Part of #331 * test(coverage): keep main's floors for files this PR does not touch coverage:update raised nine floors of untouched files from one run; metadata-autocomplete.js branches then measured 84.84 against 85. * test(legacy): record the image category routes in the legacy contract snapshot (#332) #332 adds PUT/DELETE /prompt_manager/images/category and GET /prompt_manager/images/categories; the legacy pages' routes are unchanged. * test(duplicates): a removed copy with a collection and a category moves both (#331, #332) |
||
|
Vito
|
e86c0dcf05 |
fix(ci): write browser coverage without each script's source (#432) (#433)
Every raw browser coverage file repeated the full source of each mapped script, about 84% of its bytes, and the frontend coverage merge ran out of Node's heap at 1,884 files. c8 reads the source from the mapped file on disk, so the entries keep only the URL, script id and functions. |
||
|
Vito
|
39877202b2 |
feat(ingestion): one prompt decision for scan and monitor; scans link each dataset branch's own prompt (#247, PR 1) (#422)
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) When a file's own graph leaves the prompt open, both inspect routes ask recorded run history about the file (one run reported it and it was written during that run). The save node that run reported it from anchors the walk like a file-name match, so dataset branches that share one saver prefix show their own prompt, labelled "Recorded run". Still open at one sampler whose PromptManager text was recorded at execution (a language model's output), the recorded text shows as "Execution recorded". Results the file resolves alone never change; a failed lookup judges the file alone. * fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386) Review follow-ups: the texts PromptManager recorded are looked up at the sampler history's saver leads to, never at a sampler the unanchored walk named; a file name naming a different saver than history anchors nothing; outputs and texts are read in one transaction; "Recorded run" ranks below every workflow-inferred source; unexpected lookup errors log at error level. * fix(history): look up recorded runs by history's own spelling of the file (#386) The history route inspects a case-normalized target on Windows, while history matches file names exactly, so the occurrence's reference names the file. * refactor(inspector): share the inspector's resolve as inspection_resolution inspect_prompt_evidence now calls inspection_resolution, which returns the resolution to use, the anchored result and whether the recorded saver chose it. Ingestion can call the same helper for files the default profile cannot decide (#247). * feat(ingestion): one per-file prompt decision for the scan and the monitor utils/file_prompt.py holds decide_file_prompt, the ladder the output scan runs today plus the inspector's rule (inspection profile, then the recorded saver) before the legacy parser, with a SCAN and a MONITOR policy. Junk text (blank, numbers, LoRA tags only, model file names) is never stored. graph_prompt_text and parse_comfyui_prompt move here; the parser accepts parsed metadata (#247). * feat(ingestion): store a file's prompt in one transaction, one prompt per file database/scanned_prompts.py links a file to the prompt of one text, creating it when new, inside one BEGIN IMMEDIATE transaction: a file linked to another prompt is a conflict, a failed link leaves no prompt behind and two writers store one prompt. pair_ingestion gains file_link_rows, link_conflict and Marks/mark_created; ingest_resolved_pair takes optional marks for the components it creates. recorded_runs and named_prompt in run_attribution become public (#247). * feat(scan): decide each scanned file with the shared ladder (#247) _ingest_scanned_file calls decide_file_prompt (scan policy) and stores the result: pairs through ingest_resolved_pair and positives through store_scanned_prompt, both with the scan marks. Files the default profile leaves undecided now get the inspector's resolve, with the saver recorded history names, before the legacy parser, so each branch of a shared-prefix dataset run links its own prompt. Junk text is never stored, a file linked to another prompt is a conflict instead of a second link, and a failed link leaves no prompt behind. _BatchHistory reads the stored prompt and the recorded saver from one recorded_runs pass. _parse_comfyui_prompt calls the moved parse_comfyui_prompt. TESTING.md documents the decision. * fix(ingestion): keep the file-name saver anchor out of the ladder (#390) The #390 ruling queues a file that only its saver's file name resolves for Needs review, never linking it. inspection_resolution gains name_anchor; the ladder passes False, so such a file falls to the legacy floor in the scan (and to Needs review) and is never stored by the monitor, while the recorded saver still decides. The inspector keeps the name anchor. * fix(ingestion): no joined PromptManager text from the legacy floor; junk keeps its step's status The legacy parser returns prepend + text + append, so its answer is refused when a PromptManager-family node has a prepend or append part (#343). Junk found at the undecided-graph step keeps that step's unlinked status and review, so an ambiguous file is still queued for Needs review. The monitor policy's ambiguous file no longer asks for the recorded run, and the file size may be read lazily. * fix(ingestion): keep the legacy answer that is a PromptManager node's own text The joined-text refusal also refused the #170 parity files whose PromptManager prepend is a link the browser parser does not follow: its answer is then the node's literal text, which is what the node stores. Refuse only an answer that is not such a node's own text. * fix(scan): one prompt per file for recorded-run links; writes survive a library move The recorded run's stored prompt now links in one transaction with the file's link rows, so a file linked to another prompt is a conflict instead of a second link. The scan's stores run through survive_library_switch. A single file asks history once (a one-file _BatchHistory), the scan passes in_output=False and reads the file size only for files with prompt metadata. A conflict row reads "Kept: already linked to a different prompt". TESTING.md records the rules. * fix(ingestion): match a file's link rows under its resolved spelling too The scan stores realpath spellings and the monitor the watcher's raw path, so a file reached through a symlinked or junction output folder could get a second generated_images row. file_link_rows now queries the path and its realpath (each normalized and raw) and dedupes by row id. TESTING.md notes the reverse order is matched only when the raw spelling is the realpath. * fix(ingestion): a switched-off PromptManager node behind a pass-through stores nothing (#90) The #90 graph walk follows only prompt-like inputs, so a PromptManagerText with Save to library off reaching the encoder through another input name (easy showAnything.anything, a custom node's value) was invisible to it, and the inspector's resolve stored its text. The ladder now checks the positive every step names: a resolved or inspected positive whose path passes a node that is off, or a legacy answer that is such a node's own text, makes the file not_saved. * fix(ingestion): any switched-off node upstream of a positive stores nothing (#90) The ladder walks every input upstream of each sampler or guider positive (not negative, model, clip, latent or image inputs) for PromptManager nodes whose Save to library is off or wired from an unreadable node, so the sampler-less step, an off node with a prepend part and a PromptManagerPair behind a pass-through no longer store its text. The legacy-floor guard reads a node's resolved text. Every guard reports not_saved for a switch that is off and not_saved_wired for one it cannot read. * fix(ingestion): the switch walk skips negative inputs by #90's own rule The upstream walk treated only names containing "negative" as negative, so an off node on a neg, neg_text or text_neg input blocked an enabled positive. prompt_graph gains is_negative_input (the _NEGATIVE_KEY rule, also used by follows_input) and the walk uses it. The guards accept a node with null inputs and a side whose provenance is not an object. * fix(ingestion): the switch walk also skips any input named with "negative" #90's rule matches only whole underscore-separated words, so negative2, negativeprompt and NegativePrompt counted as positive inputs and an off node wired only to one of them blocked the file. The walk skips those too; follows_input is unchanged. |
||
|
Vito
|
41027aa1b1 |
fix(autotag): report a missing backend package instead of offering the model as ready (#426)
Model status now says, per backend, which Python packages cannot be imported (importlib.util.find_spec, nothing imported): llama-cpp-python for GGUF, torch/transformers/accelerate/bitsandbytes for HuggingFace, onnxruntime for WD14, with the pip install command and a note (the CUDA CMAKE_ARGS for llama-cpp-python, onnxruntime-gpu for WD14). Nothing is installed. load_model refuses such a model before the loader with MissingBackendPackages (one WARNING line, no traceback); a ModuleNotFoundError from a loader becomes the same refusal. The suggest route, the legacy single and batch routes and both tagging jobs answer 409 model-unavailable with that sentence. System > Auto-tagging reads "Downloaded. Needs llama-cpp-python: pip install llama-cpp-python. ..." instead of "Downloaded. Not loaded.", Review explains instead of reviewing, and the bulk tagging dialog keeps Start disabled with the sentence. "downloaded" keeps its files-only meaning, so the legacy pages are unchanged. Closes #424 |
||
|
Vito
|
c7375573ed |
feat(images): set an image's own category, show its origin and count image categories (#332, PR 3) (#428)
* feat(images): store images' own categories in the library database (#332) One new table, image_categories, and its category index, created empty by a forward-only migration that reads and writes only its own objects. Startup checks the structure; restore also checks the values. The store sets, clears and reads own categories by file key (the deepest output source holding the file and the path below it), and copies a removed duplicate's category to a kept copy that has none. * feat(images): duplicate cleanup keeps images' own categories (#332) The kept copy's own category wins; a kept copy without one receives the removed copy's, keyed by the deepest output source holding each file. The removed copy's row is never deleted, so a give-back or a failed delete loses nothing. The job log reports how many were kept. * feat(images): set, remove and list images' own categories (#332) PUT /prompt_manager/images/category sets the own category of up to 500 listed files, DELETE removes it (listed or not), and GET /prompt_manager/images/categories lists the values whose file is listed. Targets are the Images listing identity of an output source, checked by media_target (shared with Collections). The routes never touch prompts, recipes or links; POST /prompt_manager/bulk/category is unchanged. * feat(vocabulary): count and rename image categories (#332) Tags & categories lists every category value with image_count, the stored image_categories rows holding it. Values only images hold follow every prompt-backed value (first_pid at SQLite's INTEGER maximum), so keyset cursors cross the boundary. /library/options keeps its own query and never lists image-only values. Manage category counts image rows (impact.images), treats an image-held target as existing, renames image rows in the same transaction, allows renaming an image-only value, and hashes the sorted image rowids into the impact token so a change between preview and apply is refused with vocabulary-conflict. Every image-category statement reads idx_image_categories_category. * feat(images): the category filter matches each image's effective category (#332) An image's own category wins; without one it inherits its link's: the recipe's category when set, else the positive prompt's (MC4, one rule with the inspector chip). Ambiguous and unlinked images inherit nothing, so an unlinked image with its own category is now listed. The link read no longer filters by category in SQL; the page filter, select all N matching and saved views compare the effective value. A failed own-category read answers the existing listing-failed 500. * fix(images): key symlinked output folders and keep key work off the event loop (#332) Review fixes: a source matches by its configured spelling too, so a detected output folder that is a symlink keys its files (routes and duplicate cleanup). File keys are computed in the executor. A path no source holds is a plain 403, a lone surrogate a 400. The install repairs a table that lacks its category index instead of refusing to open. * feat(inspector): show an image's own category in the inspector and History (#332) The inspector's library facts gain category_scope "image": an image's own category overrides the inherited one for any link status (linked, ambiguous, none). Images reads it for output sources only; History reads it once per inspection inside its snapshot, keyed by the same file key from the symlink-resolved output folder. A failed read hides the chip rather than show a possibly wrong value. The chip's hidden prefix names the origin ("Image category: "); the visible origin text comes in PR 3. * perf(images): keep the category filter's link read narrow; record D141, D142, D145 (#332) Under a category the link read keeps only links whose raw recipe or positive category is the value (Python still decides the inherited one), the links of images whose own category is the value are read by path, and plain reads skip the inherited-category work. The own-category map is keyed by normalized path. TESTING.md records D141, D142 and D145 and amends D13, D36 and D39; the filter budget seeds 2,000 own categories. * feat(images): set or remove a category on all N matching images (#332) The category routes also accept the Images listing filter of "Select all N matching" (#392) instead of explicit items, resolved now and capped at 500 images like an explicit selection. * fix(images): Windows path keys for own categories; plan check for the category read (#332) Review fixes: on Windows the own-category map keys each file with normalize_image_path (abspath drops trailing dots and spaces), and the plan test also covers the category's narrowing clause. * feat(images): Set category sets each image's own category; the chip names its origin (#332) The selection bar's Set category writes the selected images' own category (linked or not, up to 500, or all N matching up to 500), with the toolbar categories as suggestions; Use prompt category removes it after a confirmation that states the count. It never writes a prompt (#259's prompt action is retired from Images; the Library editor and the legacy admin Set Category still change prompt categories). The toolbar lists the listed images' own categories beside the prompt and recipe ones. The inspector chip shows its origin: "· this image", "· prompt" or "· recipe". Tags & categories counts images and opens Images for a category only images use. * test(images): compare the category chip's visible origin against the concept (#332) D146: the inspector comparison measures each chip's span.chip-source and lays the concept's own chip out with the recorded rule and origin added at runtime (withChipSource), in the side panel and the viewer's details. The D39 recipe case asserts the visible " · prompt"; the mutation check removes the origin. * test(images): compare Set category and the category options against the concept (#332) With one image's own category set through the real route, the toolbar's #category-filter options are compared with D141 mapped (the image category sorted in with the Library's, D36), and the Set category dialog against the concept's simpleInput laid out as recorded at runtime: D140 (empty field, datalist#workspace-input-suggestions of the toolbar's categories, Use prompt category between Cancel and Apply) and D8 (Apply). Unmapped, the dialog differs only in those parts. * test(history): compare the History media inspector's category chip against the concept (#332) D145: with an own category set on the baseline's newest output, the Recorded run viewer's chip is compared with the concept viewer's category chip laid out with the D146 origin and the own category at runtime; D21 maps its hidden "Image category: " prefix. Unmapped, the two differ only in the text, the prefix, the width and span.chip-source. * test(images): Set category sets each image's own category in page workflows (#332) Replace the #259 prompt-category tests with CUJ 1, 2, 3, 8 and 10, the suggestions datalist, the 500-image limit, the source-changed path, D142 and coded refusals; Images never posts /prompt_manager/bulk/category. * test(inspector): the category chip names its origin in the panel, viewer and History (#332) CUJ 4 and 5: "· this image", "· prompt" and "· recipe" chips, an unlinked file's own category, and History's current-file inspector. * test(vocabulary): category rename covers image categories; image-only rows browse Images (#332) CUJ 6 and 7 with own categories set through the API on two output files. * test(vocabulary): compare image-category rows and the Manage category preview against the concept (#332) The vocabulary fixture seeds own image categories on request (image_categories: Landscapes on two images, Portraits on one image only). mapVocabularyConcept maps D143 ("N positive prompts · M images", image-only rows before Manage category), a model checked against the concept page with those rows added at runtime, and mapVocabularyDialog takes the preview sentence, so the Manage category preview is compared with D9 and D144's images. The default fixture and mapping are unchanged. * test(vocabulary): park the pointer before measuring the category rows (#332) Firefox kept the pointer where the concept's Review was pressed, over the production Portraits row's Browse, so that button was measured hovered. * docs(testing): record D140, D143, D144 and D146 and the image category UI (#332) D140 (Set category writes images' own categories), D146 (the chip's visible origin) beside D141, D142 and D145; D143 and D144 in the vocabulary section. D8, D21 (the D20/D21 side panel) and D39 amended; the D8, the "all N matching" and the Native vocabulary UI paragraphs name the Prompt library editor and the legacy admin "Set Category" for prompt categories; the #332 section lists the interface and its comparisons. * test(vendor): count the #332 host case in the execution matrix image-attribution.spec.mjs gains the image category route case, so the execution config lists 92 tests (3 attribution cases per project). * fix(images): focus after a changed source; Browse is a fresh Images visit (#332) Review fixes: a 409 source-changed reload moves focus to the heading as a success does; a selection the listing no longer holds says none could be found instead of sending nothing; a Tags & categories Browse opens a plain Images visit filtered by the category (remembered and with the Appearance defaults) rather than a saved-view visit. * test(legacy): record the image category routes in the legacy contract snapshot (#332) #332 adds PUT/DELETE /prompt_manager/images/category and GET /prompt_manager/images/categories; the legacy pages' routes are unchanged. * refactor(images): take the selection bar's categories from its one caller (#332) The getCategories default was never used (images.mjs always passes the toolbar's categories) and left images-selection.mjs function coverage at 97.05 (floor 98). |
||
|
Vito
|
4f37af209c |
feat(collections): store collections and add their routes and Images filter (#331, PR 1) (#400)
* feat(collections): store collections in the library database * feat(collections): add, remove and rename collections and keep members through duplicate cleanup * feat(collections): add collection routes and the Images collection filter * feat(images): store images' own categories in the library database (#332) One new table, image_categories, and its category index, created empty by a forward-only migration that reads and writes only its own objects. Startup checks the structure; restore also checks the values. The store sets, clears and reads own categories by file key (the deepest output source holding the file and the path below it), and copies a removed duplicate's category to a kept copy that has none. * feat(images): duplicate cleanup keeps images' own categories (#332) The kept copy's own category wins; a kept copy without one receives the removed copy's, keyed by the deepest output source holding each file. The removed copy's row is never deleted, so a give-back or a failed delete loses nothing. The job log reports how many were kept. * feat(images): set, remove and list images' own categories (#332) PUT /prompt_manager/images/category sets the own category of up to 500 listed files, DELETE removes it (listed or not), and GET /prompt_manager/images/categories lists the values whose file is listed. Targets are the Images listing identity of an output source, checked by media_target (shared with Collections). The routes never touch prompts, recipes or links; POST /prompt_manager/bulk/category is unchanged. * fix(images): key symlinked output folders and keep key work off the event loop (#332) Review fixes: a source matches by its configured spelling too, so a detected output folder that is a symlink keys its files (routes and duplicate cleanup). File keys are computed in the executor. A path no source holds is a plain 403, a lone surrogate a 400. The install repairs a table that lacks its category index instead of refusing to open. * fix(collections): address review findings * feat(images): set or remove a category on all N matching images (#332) The category routes also accept the Images listing filter of "Select all N matching" (#392) instead of explicit items, resolved now and capped at 500 images like an explicit selection. * test(vendor): count the #332 host case in the execution matrix image-attribution.spec.mjs gains the image category route case, so the execution config lists 92 tests (3 attribution cases per project). * test(browser): install the slideshow clock before wrapping setTimeout The slideshow fixture wrapped window.setTimeout before page.clock.install(), so the viewer's timer ran on real time and clock.runFor() did not drive it; a late WebKit timer failed 'slideshow exposes exact state labels' on CI. * test(legacy): record the image category routes in the legacy contract snapshot (#332) #332 adds PUT/DELETE /prompt_manager/images/category and GET /prompt_manager/images/categories; the legacy pages' routes are unchanged. * test(duplicates): a removed copy with a collection and a category moves both (#331, #332) |
||
|
Vito
|
3ebd5e4166 |
ci(browser): a hung browser step fails with its running tests named and its traces kept (#425) (#429)
* ci(browser): stop a hung browser run before the job limit and name it A browser job killed at its 20-minute limit keeps no log and skips its upload steps (merge-queue run 40723857746, firefox shard 2/2), so nothing showed which test hung. - Each browser test step gets timeout-minutes 18, below the job's 20, so the step fails and the always() uploads still run. - Under CI, Playwright's globalTimeout stops the run first: PM_TEST_GLOBAL_TIMEOUT_MINUTES 17 for page workflows, 15 for coverage workflows, a fixed 3 for the viewer suite. Locally there is no limit unless the variable is set. Each test keeps the 30 s default. - running-tests-reporter.mjs prints the tests still running when the run ends; Playwright's own reporters cannot, since those tests never end. - A blob report goes to tools/vendor/test-results/blob/, inside the uploaded traces path. - ci-workflow.test.mjs pins globalTimeout < step limit < job limit and the always() uploads; no existing limit is raised. Closes #425 * ci(browser): rely on the step limit, report the running tests on SIGTERM Maintainer ruling on #425: no Playwright globalTimeout. The 18-minute step limit under the unchanged 20-minute job limit stays, as do the always() uploads and the CI blob report. - Drop globalTimeout and PM_TEST_GLOBAL_TIMEOUT_MINUTES with its helper and tests. - A step that reaches its limit gets SIGTERM, which ended Playwright with no report and no blob. The running-tests reporter now catches the first SIGTERM, prints the tests still running and sends Playwright its own SIGINT, so the run ends interrupted and the blob report is written. A second SIGTERM kills as usual. - Test paths are relative to the suite's rootDir instead of the cwd, so `npm run --prefix` no longer prints ../../ prefixes. - The workflow comment and TESTING.md say the guarantee assumes an npm cache hit and that coverage shard 1 is tight in its 18-minute step. * ci(browser): forward SIGTERM and SIGINT from frontend-coverage.mjs to its child In the coverage job npm signals only frontend-coverage.mjs. It blocked in spawnSync with no listener, so Playwright never saw a step timeout. It now runs each step's child with async spawn, in its own process group, and forwards SIGTERM and SIGINT to the running child. Step order, exit codes and the "Command failed (N)" message are unchanged. Locally, in the pinned image, `timeout -s TERM` on test:coverage printed the still-running tests and wrote both blob reports. TESTING.md records that a SIGTERM to npm alone stops at npm's `sh -c`, and that delivery through `docker exec` in CI is not yet observed. Refs #425 * ci(browser): load the running-tests reporter by absolute path A config imported from another directory (the workflow-artifacts test copies it to a temp root) resolved './running-tests-reporter.mjs' against that root under CI and failed to load it. Resolve it from this module instead. |
||
|
Vito
|
d9f168c317 |
feat(nodes): per-node Save to library switch (#90) (#427)
* feat(nodes): per-node Save to library switch (#90) Every PromptManager node (Classic, Text, Focused, Split, Pair) gets an optional save_to_library BOOLEAN, default on and declared last. Off, the node still returns its normal outputs but PromptManager ignores the run: - the node saves, counts, tracks and records nothing; - the queue hook reads the switch from the queued graph (literal, or a linked PrimitiveBoolean), so a cached node is not counted either; a switch wired from an unreadable node is left to the node itself; - an opted-out node is no run prompt, so no tracker entry, run key, embedded-metadata match or history attribution links its images, and a run with any opted-out positive node links none of its files; - the scan reports such files as not_saved (never stored or queued for Needs review), and a file whose only opted-out node is its negative is stored positive-only, without the pair; - execution history records no prompt text for an opted-out role. PromptSearchList only reads the library and gets no switch. The switch takes the widget index older workflows used for the editor's own DOM slot (''), so a configure step turns any non-boolean value on: workflows saved before the switch load with it on and no earlier widget value moves. The Focused, Split and Pair editors draw it as a "Save to library" checkbox in their toolbar; Classic and Text show the native toggle. * fix(nodes): Save to library review fixes (#90) - A switch wired from a node the graph cannot read and off at run time marks the node's run key, so an earlier run's tracker entry never links this run's images. - The scan reports such files as not_saved_wired ("only the run itself can link it") instead of not_saved. - Workflow-only metadata ignores muted and bypassed PromptManager nodes and reads nodes inside subgraph definitions. - The Focused, Split and Pair footer says when the switch comes from a link. * fix(usage): count cached re-runs of runtime-linked prompt text (#347) A PromptManager-family node whose text comes from a runtime link (PreviewAny, an LLM, a list node) counted only the job that executed it; later jobs served from ComfyUI's output cache counted nothing. Each execution now records a receipt (digests of its STRING outputs and hashes of the text it stores) per run key and queue item. On execution_cached the receipt whose digests equal the cache entry's text is replayed once for the job, with the node's role and what the queue hook already saw recomputed from the job's own graph. The executing node and the replay count through one shared own_counts. Unmatched cache entries count nothing, as before; a node that executes after its replay in the same job skips its own count. No schema change, no backfill. * fix(usage): never replay a node twice in one job; dedupe receipts once complete (#347) A repeated execution_cached id or message for the same job now counts the node once. Receipts replace equal older ones only when their job finishes (or the next job starts), so a list still recording [A] of [A, B] no longer drops an older [A] receipt. Node-level tests check the receipt each execution records. * test(coverage): baseline for the Save to library module (#90) * test(usage): disabled node then cached jobs counts 0 (#90, #347) A PromptManager node with Save to library off records no cached-run receipt and is no run prompt in the replayed job's graph, so ComfyUI serving it from its cache on later jobs counts nothing. The prompt is saved first by an enabled run, so a replayed count would show. Covers PromptManager, Focused, Split, Text and the Pair. * fix(nodes): hide the editor's native widgets through one helper (#90) The Save to library switch copied the text fallbacks' zero-height computeSize arrow, and neither copy was ever called, so focused-node.js function coverage fell to 87.5 (floor 90). One hideNativeWidget helper now hides both, and the spec checks that the hidden switch and a text fallback take no node height. |
||
|
Vito
|
cdb771d722 |
feat(images): filter, inspect and count by each image's effective category (#332, PR 2) (#406)
* feat(images): store images' own categories in the library database (#332) One new table, image_categories, and its category index, created empty by a forward-only migration that reads and writes only its own objects. Startup checks the structure; restore also checks the values. The store sets, clears and reads own categories by file key (the deepest output source holding the file and the path below it), and copies a removed duplicate's category to a kept copy that has none. * feat(images): duplicate cleanup keeps images' own categories (#332) The kept copy's own category wins; a kept copy without one receives the removed copy's, keyed by the deepest output source holding each file. The removed copy's row is never deleted, so a give-back or a failed delete loses nothing. The job log reports how many were kept. * feat(images): set, remove and list images' own categories (#332) PUT /prompt_manager/images/category sets the own category of up to 500 listed files, DELETE removes it (listed or not), and GET /prompt_manager/images/categories lists the values whose file is listed. Targets are the Images listing identity of an output source, checked by media_target (shared with Collections). The routes never touch prompts, recipes or links; POST /prompt_manager/bulk/category is unchanged. * feat(vocabulary): count and rename image categories (#332) Tags & categories lists every category value with image_count, the stored image_categories rows holding it. Values only images hold follow every prompt-backed value (first_pid at SQLite's INTEGER maximum), so keyset cursors cross the boundary. /library/options keeps its own query and never lists image-only values. Manage category counts image rows (impact.images), treats an image-held target as existing, renames image rows in the same transaction, allows renaming an image-only value, and hashes the sorted image rowids into the impact token so a change between preview and apply is refused with vocabulary-conflict. Every image-category statement reads idx_image_categories_category. * feat(images): the category filter matches each image's effective category (#332) An image's own category wins; without one it inherits its link's: the recipe's category when set, else the positive prompt's (MC4, one rule with the inspector chip). Ambiguous and unlinked images inherit nothing, so an unlinked image with its own category is now listed. The link read no longer filters by category in SQL; the page filter, select all N matching and saved views compare the effective value. A failed own-category read answers the existing listing-failed 500. * fix(images): key symlinked output folders and keep key work off the event loop (#332) Review fixes: a source matches by its configured spelling too, so a detected output folder that is a symlink keys its files (routes and duplicate cleanup). File keys are computed in the executor. A path no source holds is a plain 403, a lone surrogate a 400. The install repairs a table that lacks its category index instead of refusing to open. * feat(inspector): show an image's own category in the inspector and History (#332) The inspector's library facts gain category_scope "image": an image's own category overrides the inherited one for any link status (linked, ambiguous, none). Images reads it for output sources only; History reads it once per inspection inside its snapshot, keyed by the same file key from the symlink-resolved output folder. A failed read hides the chip rather than show a possibly wrong value. The chip's hidden prefix names the origin ("Image category: "); the visible origin text comes in PR 3. * perf(images): keep the category filter's link read narrow; record D141, D142, D145 (#332) Under a category the link read keeps only links whose raw recipe or positive category is the value (Python still decides the inherited one), the links of images whose own category is the value are read by path, and plain reads skip the inherited-category work. The own-category map is keyed by normalized path. TESTING.md records D141, D142 and D145 and amends D13, D36 and D39; the filter budget seeds 2,000 own categories. * feat(images): set or remove a category on all N matching images (#332) The category routes also accept the Images listing filter of "Select all N matching" (#392) instead of explicit items, resolved now and capped at 500 images like an explicit selection. * fix(images): Windows path keys for own categories; plan check for the category read (#332) Review fixes: on Windows the own-category map keys each file with normalize_image_path (abspath drops trailing dots and spaces), and the plan test also covers the category's narrowing clause. * test(vendor): count the #332 host case in the execution matrix image-attribution.spec.mjs gains the image category route case, so the execution config lists 92 tests (3 attribution cases per project). * test(legacy): record the image category routes in the legacy contract snapshot (#332) #332 adds PUT/DELETE /prompt_manager/images/category and GET /prompt_manager/images/categories; the legacy pages' routes are unchanged. |
||
|
Vito
|
55fa2c4870 |
fix(runs): count cached PromptManager nodes fed by a runtime text link on every job (#347) (#423)
* fix(usage): count cached re-runs of runtime-linked prompt text (#347) A PromptManager-family node whose text comes from a runtime link (PreviewAny, an LLM, a list node) counted only the job that executed it; later jobs served from ComfyUI's output cache counted nothing. Each execution now records a receipt (digests of its STRING outputs and hashes of the text it stores) per run key and queue item. On execution_cached the receipt whose digests equal the cache entry's text is replayed once for the job, with the node's role and what the queue hook already saw recomputed from the job's own graph. The executing node and the replay count through one shared own_counts. Unmatched cache entries count nothing, as before; a node that executes after its replay in the same job skips its own count. No schema change, no backfill. * fix(usage): never replay a node twice in one job; dedupe receipts once complete (#347) A repeated execution_cached id or message for the same job now counts the node once. Receipts replace equal older ones only when their job finishes (or the next job starts), so a list still recording [A] of [A, B] no longer drops an older [A] receipt. Node-level tests check the receipt each execution records. * test(comfyui): count #347's cached-run cases in the execution suite pin (#347) The pin listed 19 execution-text-link cases per project; #347 adds five node kinds and eight one-renderer cases, so the suite lists 32 per project and 118 in total with #332's image-attribution case from main. |
||
|
Vito
|
11422d4584 |
perf(images): read prompt links in one SQLite step (#354) (#418)
* perf(images): read prompt links in one SQLite step (#354) get_image_link_matches streamed one sqlite3_step per linked row, and Python's sqlite3 releases and retakes the GIL around every step. While ComfyUI generates, each retake waits on the busy thread, so the first-page link read of the workspace Images page stretched from tens of milliseconds to seconds. SQLite now returns the unchanged filter SELECT's rows as one JSON value (json_group_array), decoded with json.loads: a constant two or three GIL hand-backs instead of one per row. Nothing is cached. Any OperationalError from the wrapped statement, or a BLOB file_path or rating, makes that read stream as before with one warning per process; a missing JSON function stops the attempt for the rest of the process. tests/test_database_link_matches.py proves the result equals the streamed read on every filter combination and on NULL ratings, quoted, backslashed, non-ASCII and multi-line paths, ambiguous and pair links, an empty database and both fallbacks; the structural budget test also checks the wrapped query plan. * fix(images): keep the maintenance link reader, refine the one-step fallback (#354) The one-step reader was named _link_rows, which shadowed MaintenanceOperations._link_rows on PromptDatabase and broke the Missing links preview and apply; it is now _one_step_link_rows, and a test pins that the maintenance reader is not shadowed. An interrupted read (Move library retiring the connection) is raised as before instead of falling back and spending the one warning; a JSON text over SQLite's length limit (DataError) streams like the other value failures. The equivalence test compares value types per path rather than by row order, and a test checks that the read leaves no WAL snapshot pinned. * docs(testing): name every one-step link read fallback (#354) |
||
|
Vito
|
b5a56bee0b |
feat(images): store images' own categories and set them by route (#332, PR 1) (#401)
* feat(images): store images' own categories in the library database (#332) One new table, image_categories, and its category index, created empty by a forward-only migration that reads and writes only its own objects. Startup checks the structure; restore also checks the values. The store sets, clears and reads own categories by file key (the deepest output source holding the file and the path below it), and copies a removed duplicate's category to a kept copy that has none. * feat(images): duplicate cleanup keeps images' own categories (#332) The kept copy's own category wins; a kept copy without one receives the removed copy's, keyed by the deepest output source holding each file. The removed copy's row is never deleted, so a give-back or a failed delete loses nothing. The job log reports how many were kept. * feat(images): set, remove and list images' own categories (#332) PUT /prompt_manager/images/category sets the own category of up to 500 listed files, DELETE removes it (listed or not), and GET /prompt_manager/images/categories lists the values whose file is listed. Targets are the Images listing identity of an output source, checked by media_target (shared with Collections). The routes never touch prompts, recipes or links; POST /prompt_manager/bulk/category is unchanged. * fix(images): key symlinked output folders and keep key work off the event loop (#332) Review fixes: a source matches by its configured spelling too, so a detected output folder that is a symlink keys its files (routes and duplicate cleanup). File keys are computed in the executor. A path no source holds is a plain 403, a lone surrogate a 400. The install repairs a table that lacks its category index instead of refusing to open. * feat(images): set or remove a category on all N matching images (#332) The category routes also accept the Images listing filter of "Select all N matching" (#392) instead of explicit items, resolved now and capped at 500 images like an explicit selection. * test(vendor): count the #332 host case in the execution matrix image-attribution.spec.mjs gains the image category route case, so the execution config lists 92 tests (3 attribution cases per project). * test(legacy): record the image category routes in the legacy contract snapshot (#332) #332 adds PUT/DELETE /prompt_manager/images/category and GET /prompt_manager/images/categories; the legacy pages' routes are unchanged. |
||
|
Vito
|
95586e6395 |
refactor(gallery): open System for gallery settings, maintenance and tagging models (#330, PR 3) (#408)
* test(legacy): snapshot the routes and the settings response Pins every registered route and the key paths of GET /prompt_manager/settings while System S13 (#330) moves the legacy pages' tools to native System. Recorded on the unchanged main; cutover (#334) removes it. * test(legacy): prove each removed legacy control has native coverage TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet). legacy-coverage.test.mjs fails when a removal cites a capability that is not covered, copies another native test, lists an id or method twice, or leaves the id or method on its page. legacy-system-links.spec.mjs holds the System link table and guards the kept admin flows. * refactor(admin): open System for Diagnostics and logs The admin page's Diagnostics and View Logs buttons become links that open System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs dialogs and their methods go; TESTING.md lists them against their covered capabilities (rows B1–B7). * refactor(admin): open System for settings and keep page options Settings opens System › Overview in a new tab. The settings dialog becomes Page options (opened from the list header) with only the Infinite Scrolling default, and its Save posts {"infinite_scroll": …} alone, so a stale legacy form can no longer overwrite scan folders or other values edited in System (#330 S13-9). Scan folders, monitoring status, auto-hide, display mode, worker threads, LoraManager and the LoRA import leave the page (rows B8–B14). * refactor(admin): open System for maintenance Maintenance opens System › Maintenance in a new tab, where each operation runs as a reviewed Activity job. The one-click legacy Database Maintenance dialog and its methods go (rows B15–B18); formatFileSize stays for the image modal. * refactor(admin): open System for backup and restore Backup DB and Restore DB open System › Backup & export in a new tab, where a restore is staged, reviewed and run as an Activity job. The one-click Restore dialog and its methods go (rows B19–B20); the scan dialog's quick backup stays. The removed browser tests' data checks live on at the route: a new round trip test restores every prompt with its tags and image links. * test(legacy): find a removed modal id passed to showModal and settle the links spec The removal check also finds an id passed as a string to a ...Modal() call and no longer reads data-id= as id=. The no-request test waits for the page's own start-up reads to finish, the sort check compares as the server sorts, and the kept-flow test checks the added prompt shows in the list. * test(admin): open Page options after the saved default is read Drops the leftover LoraManager section comment above the Page options save. * refactor(gallery): open System for settings and maintenance The gallery's Settings opens System › Appearance in a new tab. Its settings dialog becomes Page options (a new header button) with only the page's own settings: thumbnail startup check, lazy loading, items per page, grid columns, infinite scrolling, debug mode and API timeout. Save and Reset keep every other stored key, so the Appearance values System saves survive (S13-13). Generate Thumbnails, Clear cache, Rescan and Find duplicates leave the page; Page options links System › Maintenance and Sources & scans (rows B21–B28). * refactor(autotag): manage tagging models in System In both auto-tag dialogs a model that is not downloaded shows a Manage in System link to System › Auto-tagging (a new tab) instead of a Download button, and the gallery's Unload Model becomes the same link. The legacy download dialog and the download, cancel and unload methods go (rows B29–B31); the batch and review stay. A test hook marks a fake model loaded. * docs(system): name System as the home for settings and maintenance README and TESTING.md send settings, maintenance, backups, diagnostics, the gallery's tools and auto-tag model downloads to their System section and name each legacy page's Page options. * fix(gallery): keep the default view in Page options and save viewer choices alone System › Appearance's Default layout is the workspace's own key, so the legacy gallery's Default View Mode stays a page option instead of losing its only control. The viewer's mute and loop buttons remember just that choice instead of saving the Page options dialog's unfilled inputs over the stored page options. The pending-thumbnails toast names System › Sources & scans. * test(legacy): check each System link sends no request in its own test The no-request check runs inside each link's test, so a slow page no longer times out the whole table, and it ignores lazy thumbnail loads the layout starts on its own. * test(legacy): a lazy thumbnail load is not a System link's request * fix(gallery): let the header buttons wrap on a narrow screen With Page options beside Settings the header no longer fits a phone's width; it wrapped off the page and covered the Page options dialog in the touch runs. * feat(gallery): remove Default View Mode from Page options (#330) System > Appearance's Default layout is the one default-layout control, as the S13 design's G5 says. The legacy gallery's Page options no longer show or save a default view and Reset no longer resets it. A stored legacy defaultViewMode still picks the view the gallery opens in (grid otherwise), and Save and Reset keep that key for the workspace-preferences migration. |
||
|
Vito
|
5583159497 |
feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) (#396)
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) When a file's own graph leaves the prompt open, both inspect routes ask recorded run history about the file (one run reported it and it was written during that run). The save node that run reported it from anchors the walk like a file-name match, so dataset branches that share one saver prefix show their own prompt, labelled "Recorded run". Still open at one sampler whose PromptManager text was recorded at execution (a language model's output), the recorded text shows as "Execution recorded". Results the file resolves alone never change; a failed lookup judges the file alone. * fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386) Review follow-ups: the texts PromptManager recorded are looked up at the sampler history's saver leads to, never at a sampler the unanchored walk named; a file name naming a different saver than history anchors nothing; outputs and texts are read in one transaction; "Recorded run" ranks below every workflow-inferred source; unexpected lookup errors log at error level. * fix(history): look up recorded runs by history's own spelling of the file (#386) The history route inspects a case-normalized target on Windows, while history matches file names exactly, so the occurrence's reference names the file. |
||
|
Vito
|
3f085d0724 |
ci: full matrix once per PR in the merge queue; Windows/macOS non-blocking (#420) (#421)
* ci: run the full matrix once per PR, in the merge queue (#420) Pull requests now run a fast subset: Python 3.12 on ubuntu-latest, the coverage gate, black/flake8, frontend modules, bandit and the Chromium/touch frontend coverage shards plus merge. The merge queue, pushes to main and manual runs keep the full matrix: Linux/Windows/macOS x Python 3.12-3.14, real ComfyUI compatibility and the Firefox/WebKit page workflows. A dropped matrix leg never reports a check, so the new required "all tests" gate requires every test.yml job to succeed outside pull requests and real ComfyUI to be skipped only on pull requests; viewerjs does the same for the browser shards. branch-protection.json drops the eight legs a pull request no longer runs and adds "all tests". * docs(ci): ruleset edit keeps every rule and integration id; find merge-queue runs * ci: move Windows and macOS out of the blocking path (#420) The maintainer deferred Windows and macOS to the end of the redesign. The merge queue now runs Linux on Python 3.12-3.14, real ComfyUI, the frontend coverage shards and the Firefox/WebKit page workflows; pull requests keep the Linux 3.12 subset. The six Windows/macOS legs move to the new portability.yml, which runs on push to main, nightly and on demand, keeps the 27-minute limit and is never a required check. Red runs there are fixed at their cause before the cutover (#334). * docs(ci): required checks live in the ruleset and the classic protection (#420) |
||
|
Vito
|
e769845417 |
refactor(admin): open System for settings, maintenance, backup and restore (#330, PR 2) (#407)
* test(legacy): snapshot the routes and the settings response Pins every registered route and the key paths of GET /prompt_manager/settings while System S13 (#330) moves the legacy pages' tools to native System. Recorded on the unchanged main; cutover (#334) removes it. * test(legacy): prove each removed legacy control has native coverage TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet). legacy-coverage.test.mjs fails when a removal cites a capability that is not covered, copies another native test, lists an id or method twice, or leaves the id or method on its page. legacy-system-links.spec.mjs holds the System link table and guards the kept admin flows. * refactor(admin): open System for Diagnostics and logs The admin page's Diagnostics and View Logs buttons become links that open System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs dialogs and their methods go; TESTING.md lists them against their covered capabilities (rows B1–B7). * refactor(admin): open System for settings and keep page options Settings opens System › Overview in a new tab. The settings dialog becomes Page options (opened from the list header) with only the Infinite Scrolling default, and its Save posts {"infinite_scroll": …} alone, so a stale legacy form can no longer overwrite scan folders or other values edited in System (#330 S13-9). Scan folders, monitoring status, auto-hide, display mode, worker threads, LoraManager and the LoRA import leave the page (rows B8–B14). * refactor(admin): open System for maintenance Maintenance opens System › Maintenance in a new tab, where each operation runs as a reviewed Activity job. The one-click legacy Database Maintenance dialog and its methods go (rows B15–B18); formatFileSize stays for the image modal. * refactor(admin): open System for backup and restore Backup DB and Restore DB open System › Backup & export in a new tab, where a restore is staged, reviewed and run as an Activity job. The one-click Restore dialog and its methods go (rows B19–B20); the scan dialog's quick backup stays. The removed browser tests' data checks live on at the route: a new round trip test restores every prompt with its tags and image links. * test(legacy): find a removed modal id passed to showModal and settle the links spec The removal check also finds an id passed as a string to a ...Modal() call and no longer reads data-id= as id=. The no-request test waits for the page's own start-up reads to finish, the sort check compares as the server sorts, and the kept-flow test checks the added prompt shows in the list. * test(admin): open Page options after the saved default is read Drops the leftover LoraManager section comment above the Page options save. * test(legacy): a lazy thumbnail load is not a System link's request |
||
|
Vito
|
9a90c7677f |
fix(exit): stop running jobs, downloads and encoders at Ctrl+C instead of waiting for them (#376) (#417)
* fix(exit): stop running jobs before Python joins the executor threads (#376) ComfyUI returns from main.py after Ctrl+C without closing its event loop, and Python joins every executor thread before any atexit callback runs, so a scan batch, thumbnail batch or model download in flight kept ComfyUI alive until it ended (on Windows each ffmpeg up to its 30 s timeout). The package now registers a threading._register_atexit hook (after concurrent.futures', so it runs before that join) that sets an exit signal and kills the running encoder processes. Job.cancelled follows the signal, so every check a job already makes between files stops it at exit; map_parallel starts no further item, the scan walk stops between folders and a tagging model download stops before its next chunk. Without the internal hook exit behaves as before. * fix(exit): abort Xet downloads and stop LoRA staging at exit (#376) Review follow-ups: Xet transfers report progress from native code and ignore the chunk checkpoint, so the exit hook calls huggingface_hub's own abort_xet_session when it is loaded; an aborted download logs as stopped. LoRA import staging reads the exit signal itself, so the legacy LoRA scan stops too. The test's fake encoder outlives the ceiling by less, and TESTING.md states the download limits of older huggingface_hub. |
||
|
Vito
|
65b15bbe75 |
fix(logs): Clear all logs deletes and empties only what its confirmation showed (#389) (#412)
* fix(logs): Clear all logs deletes and empties only what its confirmation showed (#389) - /logs/files rows carry an identity string (size:mtime_ns:ino); the workspace sends back the active and rotated rows its dialog was built from. Under the files and handler locks every listed copy is re-checked with _matches_review and the listed active log's inode; any difference refuses with nothing changed (409 logs-changed). A copy that appeared after the confirmation is kept. - A second writer of the active log (our flushed stream is behind the file's size, or our file is now a rotated copy) refuses first (409 logs-foreign-writer); each unlink is preceded by a fresh identity check and the active log is emptied last. - An active log with a hard link (st_nlink > 1) is never truncated (409 logs-hard-link), also on the file-logging-off path. - The legacy admin page's POST without a body keeps clearing what is there, with the foreign-writer and hard-link refusals. - The browser fixture grows the active log through the handler's stream. * fix(logs): accept pre-1970 identities, refuse duplicate rows, say the foreign-writer check is best effort (#389) Review follow-ups: a negative mtime_ns no longer makes every workspace clear 400; a name listed twice is a malformed list; the confirmed-row cap is 10000; the hard-link refusal says how to recover; the hard-link and foreign-append tests assert the kept bytes before the refusal code, so on the old code they fail on lost data. |
||
|
Vito
|
95c4542c6f |
test(browser): API restores stay unseen by the open page's Activity poll (#413) (#416)
* test(browser): API restores stay unseen by the open page's Activity poll (#413) Specs that restore through the API while a workspace page is open check how the page handles a restore it did not see. The page's Activity store reads the job list every 15 s; on a slow runner (CI run 40646112668, a 16 s restore) that read landed during the restore, the page followed the job and, as designed (S9), showed the open page again when it finished, closing the Create tag dialog the vocabulary spec was about to use. unseenRestore holds the page's job-list reads until the restore has answered, so they read it as already ended. Used by the vocabulary, library editor and history specs. * test(browser): arm the restore gate before the page loads and drain reads in flight (#413) A job-list read the page sent before the restore could still be answered while it ran. The gate now routes every job-list read from before the first navigation and, before a restore, waits for the reads in flight to be answered. The history and library editor conversions are pre-emptive: same race, never seen failing. |
||
|
Vito
|
534c95248a |
fix(review): anchor files on their saver in the scan and Needs review (#390) (#414)
* fix(review): anchor files on their saver in the scan and Needs review (#390) The scan and Apply now resolve with the file's name, as the workspace inspector does since #383, so the one saver whose filename_prefix names the file anchors its branch. A file whose branch that anchor resolves is never queued (from the inspection profile it offers no choice; a default-ambiguous file asks saver_anchored) and its pending, undecided items leave on the next scan. REVIEW_RULE_VERSION is folded into inspection item versions so old items are re-evaluated. No match, several matches or an unfinished walk anchor nothing. Apply refuses a choice the anchor contradicts. * fix(review): refuse a default choice the saver anchor contradicts (#390) Apply re-resolves a default item with the file's name: when its saver anchors the branch, no candidate is offered and the choice is refused. The scan's anchor check skips bytes carrying a decision and remembers unanchored file states per process, so a rescan does not inspect an unchanged file again. * fix(review): queue an anchored file with its saver's pair as the choice (#390) Maintainer ruling: a file never falls through both the link and the queue. When only the saver anchor resolves a file the default profile and the parser floor leave unlinked, the scan queues it with the anchor's pair as its one candidate (evidence anchor: saver), never linking it. Files already linked or resolved by the default path are unchanged. Apply accepts the anchored pair when the bytes now resolve to it, and an anchored item is a choice the untraced settle keeps. * fix(review): upgrade old default items of anchored files, tag anchored results (#390) A pending default item recorded before the saver anchor no longer shortcuts the rescan when only the anchor resolves the file: the anchored item supersedes it. Apply accepts a resolved re-resolution only when it is tagged as the saver anchor's, the anchored pair must carry a positive text to choose, and limited anchored evidence still counts as a choice. TESTING.md describes the ruling. |
||
|
Vito
|
e4dc781c26 |
fix(images): read a loading collection as loading, and de-flake the JXL fixture and restore specs (#391, #371, #361) (#405)
* fix(images): read a loading collection as loading, never as 0 of 0 images (#391) A reset published {loading, loaded:0, total:0} and the caption painted it as "0 of 0 images" until the first page answered, so a fast reader (the inspector concept spec on WebKit) read an empty result the server never returned. The count line now says "Loading images…" while a first page is in flight; the spec waits for the answered count, and a held-listing spec pins the ordering. * test(thumbnails): make the no-JPEG-XL-encoder fixture authoritative (#371) {available:false} stood in only the cached detection, and every settings read probes a cached None again, finding the host's real encoder. The fixture now stands in for the detector itself and reset restores it; a unit test proves the route against a detector that finds an encoder. * test(system): wait for the restore job's own end before its toast (#361) The reviewed-restore specs waited for the "Database restore finished" toast with the 5 s expect, in effect a deadline on the disk-bound publish. They now wait for the job's progress stream to close, assert the job finished, and only then check the toast with the default timeout. * test(images): wait for the search's answered count in the concept comparison (#391) searchOf took the first changed caption as the search's count, the same read-before-the-response race as the inspector's tag chip. |
||
|
Vito
|
d0339128e4 |
test: isolate API fixture databases from other suites; CI lint pins, failing bandit hook, npm cache (#346, #352, #409) (#411)
* ci: pin black and flake8, let the bandit hook fail, cache npm in the JS job (#352) The lint job installs black==26.10.1 and flake8==7.4.1 from requirements-dev.txt (the versions CI resolved on main today), so a new release cannot turn CI red with no code change. The local bandit hook ran 'bash -c ... "$@"; true': it could never fail, and bash bound the first staged file to $0 so it was never scanned. It now runs 'bandit -lll -q' directly, failing on high severity exactly like code-quality.yml. The frontend modules job's setup-node caches npm keyed on tools/vendor/package-lock.json. * test(history): wait on the writer's state, not short wall-clock pauses (#352) Four 0.08 s pauses before asserting that the worker had not probed yet now wait until the idle worker has run a whole loop pass at the current manual clock value (three clock reads), so a slow runner cannot end the pause before the worker decides. A mutation that probes 20 ms early now fails two tests deterministically. The real-clock bounded close no longer asserts close(0.02) returned in under 0.2 s wall time; it asserts the in-flight write is still blocked and the worker alive, like the output writer test. The manual-clock bound stays exact. The last 0.1 s non-blocking submit check uses the 1 s NON_BLOCKING_CEILING from #351. * test(api): give each API fixture database its own temp folder (#346) Every PromptManagerAPI sweeps its database's folder at startup for the stages, uploads and safety-backup copies an interrupted restore left. The API test fixtures created their database directly in the system temp folder, so any other test process starting an API there (another worktree's suite, a pre-push hook, an agent's targeted run) deleted this process's in-flight restore files. The restore, epoch and retirement tests then failed with 'staged restore changed', 'unable to open database file' or 503 instead of 409. Busy machines fail because they are running other suites, not because they are slow: the same CPU load with a private TMPDIR passes. private_temp_db() puts the fixture database in a new folder removed at cleanup (with any safety backups a restore left beside it). A regression test sweeps a stand-in shared folder in the middle of a multipart restore, as another suite's startup would. * test(history): count worker parks, hold saturation blocks until released (#352) wait_for_worker_pass now counts the idle worker's parks on its wake event (two parks from now = one whole pass begun after the call) instead of assuming a fixed number of clock reads per pass, which a future clock read would have made vacuous. The real-clock bounded close keeps a loose upper bound (0.02 s + NON_BLOCKING_CEILING). The two saturation tests (history and output writer) blocked the worker with release.wait(3) while they queued 1023 more items. On a loaded runner the fill took longer than 3 s, the worker unblocked on its own and drained part of the queue (pending 868..1017 != 1024), and cleanup then reported a leaked worker. The block now holds until the test releases it; 60 s is only a hang guard, and cleanup always sets the release. Under one core with 8 busy loops: 8/8 runs failed before, 6/6 pass after. * test(maintenance): wait on duplicate jobs with a disk-bound hang guard (#409) On Windows py3.13 CI (PR #405) a one-file duplicate cleanup missed its 10 s WAIT. The test already waits on the job's own task; the 10 s bound acted as a deadline on disk-bound work. In the same 50 s window the following tests' setups took up to 9 s against 0.1-0.6 s around it while the test bodies stayed fast, which is consistent with a process-wide disk I/O stall on the runner rather than a lease, handle, executor or job stall (the log has no thread dump to show more). tests/disk_waits.py holds the one disk-bound hang guard (90 s per wait, under the 120 s per-test watchdog), shared with RESTORE_SECONDS (#382). Waits on find and cleanup job tasks in the duplicate-files tests use it; gates and server-state polls keep WAIT. The keeper-validation test turns the safety backup off, since it does not test backups. |
||
|
Vito
|
ef7e07c765 |
refactor(admin): open System for Diagnostics and logs (#330, PR 1) (#402)
* test(legacy): snapshot the routes and the settings response Pins every registered route and the key paths of GET /prompt_manager/settings while System S13 (#330) moves the legacy pages' tools to native System. Recorded on the unchanged main; cutover (#334) removes it. * test(legacy): prove each removed legacy control has native coverage TESTING.md gains the "Legacy System UI removed by S13" table (no rows yet). legacy-coverage.test.mjs fails when a removal cites a capability that is not covered, copies another native test, lists an id or method twice, or leaves the id or method on its page. legacy-system-links.spec.mjs holds the System link table and guards the kept admin flows. * refactor(admin): open System for Diagnostics and logs The admin page's Diagnostics and View Logs buttons become links that open System › Diagnostics & logs in a new tab. The legacy Diagnostics and System Logs dialogs and their methods go; TESTING.md lists them against their covered capabilities (rows B1–B7). * test(legacy): find a removed modal id passed to showModal and settle the links spec The removal check also finds an id passed as a string to a ...Modal() call and no longer reads data-id= as id=. The no-request test waits for the page's own start-up reads to finish, the sort check compares as the server sorts, and the kept-flow test checks the added prompt shows in the list. |
||
|
Vito
|
83fa80c330 |
fix(restore,threads): a stopped restore waits for its worker; exit-window thread refusals are quiet (#360, #375) (#404)
* fix(restore): a restore cut off by shutdown waits for its worker thread (#360) A cancelled restore handler or job no longer removes the stage or the upload, or ends its exclusive registry admission, while the executor thread still stages or publishes. The cancel waits for the worker; a publish that finished is adopted before the cancel goes on, and a stage finished after the cancel is removed. * fix(threads): a thread or pool refused at interpreter exit is quiet and leaves nothing pending (#375) During interpreter exit, threading._shutdown shuts every thread pool before the package teardown runs while our daemon threads keep working. map_parallel now runs the items a refused pool did not start on the calling thread, schedule() drops the path and returns False when its worker cannot start, and the preview queue skips the file. Each refusal is logged once at DEBUG without a traceback; other start failures behave as before. * fix(restore): keep the cancel and the adoption when a stopped restore's publish ends (#360) A publish that fails after the cancel now ends in CancelledError (after its prune) instead of a 500, and the restored library is adopted to its end even when a second cancel arrives. The blocking calls go through _run_in_executor again, as the executor contract test expects. Exit-window refusals: the comments state what was measured (a pool is refused during threading._shutdown; Thread.start only once the interpreter is finalizing), and the tests assert the one DEBUG line. |
||
|
Vito
|
e472a8bf74 |
fix(system): Tagging selects show their whole chosen option at 390 px (#398) (#399)
* fix(system): Tagging selects show their whole chosen option at 390 px (#398) The concept's phone cap (120px) cut Existing tags ("Replace after review"). #tag-existing now never shrinks for its row text and keeps its natural width at phone width, beside the text. The comparison expects selectsFit for every select, and a new test walks every option of both selects at 390 and 320 px in both themes. TESTING.md D132 records the decision. * fix(system): bound Existing tags by its row at phone width (#398 review) |
||
|
Vito
|
44177974cb |
fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252) (#397)
* fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252) A graph where no node takes a positive or negative input fell back to the pre-redesign _extract_readable_prompt heuristic: the last node with any string text input won, and with none it read stale UI workflow widgets. It stored LoRA syntax, display-node reports, install notes, model file names and numbers. The scan now asks sampler_less_prompt: only the executed API graph counts; a candidate is a node with a text input that another node consumes; blank and numeric texts are never prompts; this repository's own positive nodes are authoritative; one distinct text links, several (or an unreadable wired text that no candidate builds) stay unlinked and try the recorded-run attribution. The heuristic and its helpers are removed; A1111 files read the parsed positive prompt directly. * fix(scan): refuse loader side outputs, seeded templates and keep A1111 text (#252) Review findings: an unreadable text link counts as built from a candidate only through that node's first output, so a LoRA loader's trigger-word output is unknown, not its text. A node with a seed input holds a template, never the prompt that ran. A sampler-less graph the rule cannot decide still links the A1111 positive a metadata saver wrote beside it, as the old scan did. Only plain numbers are numeric; words such as nan or inf are prompts. * fix(scan): linear plain-number check for sampler-less texts (#252) The numeric pattern had overlapping quantifiers that backtrack quadratically on a long digit run ending in a non-digit; it now has none. |
||
|
Vito
|
d11b354ed6 |
fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2) (#394)
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1) The native "Tag output images" job now runs the image-tags loop over every still image the output folders list, linked or not: each image gets its own model tags (at most "Suggestions per image") in its image tag set, and its bytes are marked auto-tagged in a new media_image_autotags table. No prompt tag or prompt text is written. - database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional in old backups, refused without the annotation group. - add_image_tags/replace_image_tags take mark_autotagged; autotag_state and autotagged_locators read the marks through the one current-record rule. - "Skip already tagged" skips images whose current bytes are marked, counted in a stat-only pre-pass; hand tags do not count. - The review counts images and estimates the time (WD14 on CPU or CUDA). - System > Auto-tagging review rows and copy, D134 and the Auto-tagging contract follow. Part of #379 * fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379) The job read as ended while it still held the model gate, so a quick restart could be refused with model-busy. Pre-pass chunks now publish progress, and a cancel between chunks keeps exact counts. Part of #379 * fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379) The live check measured WD14 on CUDA at 0.167 s per image (100 images in 16.7 s with the model load), so the review estimate uses 0.17 s. A finished or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X min left"; it now ends with "<verb> <state> · N files processed". Part of #379 * fix(tagging): legacy pages tag each image, not its prompt (#379, PR 2) The admin and gallery batch starts the shared Tag output images job ("Tag output images (legacy page)") and relays its progress; closing the window cancels it. Review asks per image with Skip first and applies by image and sha256; apply refuses a prompt_id body. The Tag image dialog marks an image auto-tagged only after Generate suggestions on that capture. A second legacy batch (also one that wins the start race) answers JSON 409, and both pages now show that sentence. * test(tagging): gallery review units apply by image; record each progress change The gallery review unit tests still pinned the removed prompt_id apply. The legacy batch spec read the progress text once per observer callback, so a phase frame and the first image frame landing in one task lost the phase (WebKit); it now records every mutation record. |
||
|
Vito
|
2f8b7cb306 |
feat(images): select many images: ranges, keys, touch and all N matching (#392) (#393)
The workspace Images page selects like a file manager in Grid, List and Masonry, across every loaded batch: Shift+click ranges from the anchor, Ctrl/Cmd+click toggles a card without opening it, Ctrl/Cmd+A selects the loaded images, Shift+arrows extend, Escape clears, and a touch long-press starts a selection mode in which taps toggle. A plain click still opens or inspects. Once every loaded image is checked and more match, the bar offers "Select all N matching". Add tags, Auto-tag selected and Set category then send the listing filter instead of N identities: the image-tags job takes `filter` in place of `targets` and /bulk/category takes `images_filter`, both resolved by the listing's own admission code (up to 10,000 images, a Source bound by its root_scope). Export metadata waits for loaded images. Recorded as deviation D135; the concept is unchanged. |
||
|
Vito
|
37e4e08327 |
fix(tagging): Images tag search goes through the tag index (#379, PR 3) (#395)
* fix(tagging): Images tag search goes through the tag index (#379, PR 3) The Images Tags filter and text search no longer stream every image tag row per request. image_tag_ids reads the names that tag at least one image (exact under Tags, case-folded substring for q) and image_tag_locators reads their files by tag_id through idx_media_image_tags_tag. Results are unchanged: a differential test compares the lookup with the old stream, and a trace test checks that no statement reads media_image_tags except by tag id. The search contract in TESTING.md and a browser test (CUJ 8) follow. Closes #379 * test(search): pin prompt-only tags and multi-chunk tag lookups (#379) Reads the tag names and the locators in the same annotation read helper. |
||
|
Vito
|
d2ea705a44 |
fix(tagging): Tag output images tags each image, never its prompt (#379, PR 1) (#380)
* fix(tagging): Tag output images tags each image, not its prompt (#379, PR 1) The native "Tag output images" job now runs the image-tags loop over every still image the output folders list, linked or not: each image gets its own model tags (at most "Suggestions per image") in its image tag set, and its bytes are marked auto-tagged in a new media_image_autotags table. No prompt tag or prompt text is written. - database/autotag_schema.py: one forward-only WITHOUT ROWID table, optional in old backups, refused without the annotation group. - add_image_tags/replace_image_tags take mark_autotagged; autotag_state and autotagged_locators read the marks through the one current-record rule. - "Skip already tagged" skips images whose current bytes are marked, counted in a stat-only pre-pass; hand tags do not count. - The review counts images and estimates the time (WD14 on CPU or CUDA). - System > Auto-tagging review rows and copy, D134 and the Auto-tagging contract follow. Part of #379 * fix(tagging): end the batch job after its cleanup and report pre-pass progress (#379) The job read as ended while it still held the model gate, so a quick restart could be refused with model-busy. Pre-pass chunks now publish progress, and a cancel between chunks keeps exact counts. Part of #379 * fix(tagging): measured GPU rate, and an ended image-tags job shows no time left (#379) The live check measured WD14 on CUDA at 0.167 s per image (100 images in 16.7 s with the model load), so the review estimate uses 0.17 s. A finished or cancelled Auto-tag/Add tags job kept "Processing file N of N · about X min left"; it now ends with "<verb> <state> · N files processed". Part of #379 * test(browser): reload the admin page only after the in-place edit saved The gallery-to-admin workflow pressed Enter and reloaded as soon as the edited text showed; the page saves after showing it, so a reload that won the race abandoned the PUT (WebKit, run 40564511000). The test now waits for the save's answer. Part of #379 * test(tagging): the batch coverage row names the renamed #379 test |
||
|
Vito
|
52a00370ea |
test(system): prove Diagnostics & logs against the concept and every legacy System capability's owner (#329, PR 3) (#387)
* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * test(system): compare Diagnostics & logs with the concept System › Diagnostics & logs and its log cleanup review are compared with the concept (app.js 694–695 and maintenancePlan("rotated logs")) in both themes at 1600x1100 and 390x844, with D109 and D110 (and D100 for the review) as the only mapped differences, a 320-wide select check and a mutation check. The fixture gains a concept seed mode (the concept's four log lines, a 1.4 MB active log with three rotated copies, console logging off), and the plan dialog's measureReview and mapPlanConcept move unchanged into concept-inventory.mjs for Maintenance and this review. The comparison found that the Database line read a report key that does not exist (database_info.size_bytes) and so always showed "0 bytes"; it now reads database_size_bytes, the key get_database_info() returns. Part of #329 * test(system): prove every legacy System capability has an owner TESTING.md gains "Legacy System capability coverage": every System capability of the legacy admin and gallery pages with its anchor on main, owner, status (covered, kept in legacy or approved removal), native control and the exact title of the test that proves it. tests/vendor/legacy-coverage.test.mjs (with tools/vendor/legacy-coverage.mjs) fails on a row with no owner, an unknown status, a malformed native control, a control on a row that is not covered, or a test title its file does not declare; a title only in a comment does not count. workspace-system.spec's "every covered legacy System capability has its native control" opens each covered row's section and finds the control by role and accessible name. Versus the design's table: the anchors are re-checked against main, "Run health checks" is on Maintenance (D123), not Overview, Diagnostics run is proven by "Run checks opens the shared Health report", and the export rows repeat their test title instead of "(same as above)". Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 * test(system): require a running test and a control for every covered capability Review fixes for the legacy coverage proof: - A title declared only with test.skip, .fixme or .fail no longer proves a row (.only and .slow still do). - Every covered row names its native control, except the rows in CONTROL_EXEMPTIONS, each with its reason (today only the gallery per-image auto-tag row, owned by the Images selection bar). The browser test asserts that the covered rows without a control are exactly the exemptions. - JavaScript comments are stripped across the whole source, wherever a block comment starts, skipping strings, template literals and regular expressions. A Python def test_ counts only at function or method indentation and outside comments and docstrings. - A control name several rows share ("Review plan", "Review options", "Refresh") is looked up inside the row the table names (· in "<row>"). - The diagnostics comparison records each select's chosen option, so a wrong default level is caught. Part of #329 |
||
|
Vito
|
bdc38422a9 |
feat(system): Diagnostics & logs natively, reviewed log cleanup and Clear all logs (#329, PR 2) (#385)
* fix(logs): declare the compressed length so a log download finishes The gzip middleware compressed a text body but kept the handler's declared Content-Length (the uncompressed size), so a browser waited for bytes that never came: downloading any log over 1 KB hung, in the workspace and on the admin page. The middleware now drops a declared length before compressing. Part of #329 * feat(system): show Diagnostics & logs natively and retire the hand-off panel System › Diagnostics & logs is native (D109, D110): the concept's Diagnostics surface with the live database, source, Viewer.js and renderer lines, Run checks (the shared Health report), the diagnostic report, Inspect and the confirmed test link; and the Logs surface with the minimum-level viewer, Lines 1-500, statistics, the log download, the runtime level and console setting saved by Save preferences, the log files with their downloads, Review log cleanup (the rotated_logs maintenance plan, no backup box, Listed files) and a confirmed Clear all logs. D94's hand-off panel, SECTION_HANDOFFS and the undelivered runtime adapter are gone. The browser server logs into its own temporary root. Part of #329 * fix(system): confirm Clear all logs on a fresh file list and record coverage Review fixes for the Diagnostics & logs section: - Clear all logs reads the log files again before it confirms, so the dialog names the copies it deletes now (a list it cannot read gives the general sentence, never an old count); Refresh reads the files too. - Runtime values that cannot be read stay disabled and say so. - Create test link holds its button until the write answered, so a second press cannot write a second record. - The Lines field names its range. - The browser server builds its logger in its own root, so no log file is ever opened in another folder; the fixture closes every directory scan and replaces a link instead of writing through it. - system.mjs drops the unused section fallback; frontend coverage floors recorded with coverage:update (system-diagnostics.mjs 97.5% lines). Part of #329 * fix(system): read the database size the report sends The Database line read database_info.size_bytes, a key the report does not have (get_database_info answers database_size_bytes), so it always read "Readable · 0 bytes · N images". The browser test now requires a non-zero size. Found by PR 3's concept comparison. Part of #329 |
||
|
Vito
|
a985af506b |
test(browser): seed resets on a scratch copy and publish them in one commit (#384) (#388)
* test(browser): seed resets on a scratch copy and publish them in one commit (#384) The concept and plain resets made hundreds of committed transactions on the live database; on a slow CI disk they queued behind WAL checkpoint syncs and took 30+ s. Each reset now copies the live database into a scratch database (backup API, no syncs), runs the unchanged seeder against it through an API proxy, and replaces the rows of the tables the seed changed in one live transaction. A commit by another live connection meanwhile fails the reset. tests/browser/reset_equivalence.py proves every reset kind leaves the same rows, files and response as the old path (PM_TEST_SEED_ON_LIVE=1). * test(browser): empty media folders by a rename, delete them in the background (#384) With the database commits gone, throttled resets still stalled in rmtree: unlinking the last fixture's files waits for their pending writeback. Each reseed now moves the old output and input folders into <root>/reset-trash and makes empty ones; a background thread deletes the trash, and the server's cleanup stops it and removes the rest before the temp root goes. * fix(test): recheck the live data version under the publish's write lock (#384) A writer whose transaction was open while the seed ran committed before BEGIN IMMEDIATE returned and its rows were overwritten. The publish now rechecks data_version under the lock, and restores foreign_keys and detaches the scratch database on every error path. * test(browser): keep the reset scratch database in /dev/shm when there is one (#384) Throttled runs still waited on the runner's disk while creating and rewriting the scratch database, whose pages were under writeback. * test(browser): keep the scratch on disk when /dev/shm is short, delete in place after cleanup (#384) The largest scratch copy is about 8 MB; Docker's default /dev/shm is 64 MB and shared with the browsers. A reset that empties a folder after the server's cleanup closed its trash deletes in place instead of starting a thread nobody stops. * fix(test): drop the gallery caches only after the reset's rows are published (#384) The seeders drop the Images listing at their end. Run on the scratch copy, that drop came before the publish, so a background walk in between could cache per-file values built from the live database's old rows: CI's inspector tag chip found 0 of 0 images. The proxy now records the drop and the seeder makes it after the publish, as the old reset did. * fix(test): remove /dev/shm reset scratch left by killed browser servers (#384) A server killed outright never runs its cleanup, and tmpfs keeps the scratch folder in memory until reboot: the Python suite's server tests left one per killed server. Scratch folders now carry the owner's pid, and each new one first removes those whose process is gone (POSIX only). |
||
|
Vito
|
a0fc5171e3 |
fix(inspector): pass text through unknown nodes instead of refusing them (#381) (#383)
* fix(inspector): pass text through unknown nodes instead of refusing them (#381) The inspector refused any node class missing from the semantics table, so a PromptManagerPair behind an rgthree Any Switch showed "Prompt uses a node the inspector does not support". The inspection resolver (Python and browser) now reads through unknown nodes: a text input is the text, positive/negative route by output slot, a conditioning input is followed alone, otherwise every linked input except core type inputs (pass_through.skip_inputs). PromptManager nodes are authoritative; several different texts become one candidate each ("Several prompts reach the sampler"). A seeded text node with no conditioning input is generated at run time, and a recorded PromptManager execution is not checked against text that passed through an unknown node. INSPECTION_RESOLVER_VERSION goes to 2; ingestion keeps the default profile. A legacy parity gate records every committed graph whose inspection positive matches the legacy parser's; no graph may leave it. * fix(inspector): keep recorded pair text, string-only seed rule, role inputs and unique candidates (#381) Review of the pass-through: a walk through an unknown node never overrides a PromptManagerPair's execution record (any outcome but work-limit is unverifiable while the record is checked); the run-time seed rule reads only string outputs; a node with positive/negative inputs follows only the role of the requested slot, nothing when it is unlinked; one text offered twice is one candidate. * fix(inspector): anchor on the saver that wrote the file and resolve stages on shared links (#381) An inspected file's name now anchors the inspection on the one saver whose filename_prefix (literal or traced) makes the file's stem <prefix>_<digits>[_]; the samplers in that saver's image ancestry are the candidates. No match, several matches or an unreadable walk anchor nothing, so the file stays "several possible prompts". In a latent walk, an earlier sampler reading the same conditioning links as the later one is the same prompt, not an unattributed stage. Python and the browser resolver behave the same; the legacy parity gate states each graph that differs from legacy on purpose. * fix(inspector): skip only plain first stages on shared links; an unfinished prefix scan anchors nothing (#381) Review of the saver anchor: a same-links earlier stage is skipped only when it is not a detailer, sets no prompt override and continues no other sampler, so an earlier prompt or override behind it is never hidden. A work limit while tracing a saver prefix ends the match scan without an anchor instead of counting that saver as no match. * test(inspector): cover every plain-first-stage outcome on both resolvers (#381) The frontend coverage gate found local-resolver-stages.mjs branches at 92.44% (< 93). Each way an earlier stage stops being a plain first stage (no latent, literal, missing or cyclic latent, a sampler-shaped node, a detailer) is now a resolver test and an anchor-first-stage-* parity vector. |
||
|
Vito
|
0652beba90 |
ci: run the required workflows for the merge queue (#382)
* ci: run the required workflows for the merge queue main no longer requires a branch to be up to date before it merges, so a PR no longer re-runs CI after every other merge. The Tests, Code Quality and Viewer.js workflows also run on merge_group, so the merge queue can test each PR against the latest main once, in order. The tracked protection record now says strict: false. * test(history): give waits that span a database restore their own ceiling The merge queue's Windows py3.12 job failed test_restore_or_generation_rotation_between_transactions_discards_evidence (restore=True): the restore it waits for, a full verify/migrate/backup/ replace cycle, took over the 10 s BLOCK_SECONDS on a busy runner. The two waits that span a restore now use RESTORE_SECONDS (90 s), which stays under the suite's 120 s hang watchdog, so a real hang still fails with its thread dump. Behavioral waits keep BLOCK_SECONDS. * test(history): drain the writer before every bounded close or dispose The merge queue's Windows py3.12 job failed test_saturated_start_cannot_capture_outputs_or_retain_previous_roots: dispose(2) ran while the writer still worked through 1,024 queued commands on a busy runner. Like #370, the test now drains first; the 2 s bound is not what it tests. Four more closes that follow a released block get the same drain, so the next slow runner can't fail them one at a time. * ci(comfyui): run the real-host job in the pinned Playwright image The real ComfyUI compatibility job installed Chromium with `playwright install --with-deps`, which pulls the browser's system packages from Ubuntu's apt mirror on every run. Three runs (#349 run 40367579020, #378, #382 job 122287230985: "Fetched 19.7 MB in 12min 23s (26.5 kB/s)") spent 11-12 minutes there and were cancelled at the 24-minute limit after all 89 tests had passed. The job now runs inside mcr.microsoft.com/playwright:v1.64.1-noble, the same image as the frontend coverage and page workflow jobs, which ships Chromium and its libraries, so the job has no apt step at all. The whole job moves rather than only the browser part: the Playwright suites launch the managed ComfyUI host themselves (PM_TEST_PYTHON), so splitting the browser off would mean running the host on the runner and the browser in a container across a network boundary, a larger change than moving one job whose host is plain Python and pip wheels. Python follows the other image jobs: the image's own Python 3.12 with a user-space venv bootstrapped from a checksum-pinned pip wheel (toolcache setup-python builds do not run in the container). A step fails the job when the image Python differs from pythonVersion in tools/vendor/comfyui-runtime.json, so the pinned host stays honest; torch, torchvision and ComfyUI's requirements install from the same pins as before. pip and npm download caches replace setup-python's and setup-node's. tests/vendor/browser-image.test.mjs now checks this job like the other browser jobs: image tag equals @playwright/test in the lockfile, no browser install or apt, no setup-python/setup-node, checksum-pinned pip, Node 24, and the runtime Python check. The 24-minute limit is unchanged. |
||
|
Vito
|
f35e010f79 |
feat(system): log cleanup rule, reviewed rotated-log cleanup and diagnostic report (#329, PR 1) (#367)
* fix(logs): delete and download only rotated copies under one rule Rule R (#329): a file is a rotated copy only when it is a direct entry of the log folder named prompt_manager.log.<n> (n >= 1, no leading zero, compared after normcase) and a regular file that is not a symlink or a junction. Everything else is never deleted. - Clear (POST /logs/truncate) empties the active log and deletes only rotated copies; a .bak, a .1.gz, any other file and every link are kept. It runs off the event loop under one files lock and the file handler's lock, so no rollover renames a file mid-way, and logs only afterwards. - delete_rotated(expected) deletes exactly the reviewed copies, or nothing when one of them changed (size, mtime_ns, inode, missing, now a link). - The download serves only a listed regular, non-link file, opened without following links and read off the loop; a link or folder is a 404. - /logs/files rows gain kind; files and stats read links with lstat and stats no longer count a link's target. - The config update runs off the loop and swaps handlers under the files lock. * feat(logs): review and apply rotated-log cleanup as a maintenance plan POST /maintenance/preview {operation: "rotated_logs", options: {}} lists the rotated copies Rule R allows, with size and time, and what it keeps (the active log, links, other files); it writes nothing and answers the Plan shape with basenames only, backup policy none and no disk key. POST /maintenance/apply starts the "Rotated log copies" maintenance job (admission read: runs beside a scan, refused beside another maintenance job or an exclusive one). It deletes exactly the reviewed copies, or nothing and fails plan-changed when any of them changed after the review; a copy that cannot be deleted becomes an errors row. The token is consumed on the loop turn the job starts. (#329) * feat(diagnostics): add a shareable report without secrets or paths GET /system/diagnostics builds the report on the server, off the event loop and under a registry lease (409 busy while a restore runs): the project and Viewer.js versions, Python and the platform, the four legacy checks (one helper, legacy_checks, that GET /diagnostics also calls), public database info, source counts, the log configuration and statistics, the last health summary and the config.json-only keys from an explicit list. The CivitAI key is reported only as has_civitai_api_key; every path is relative to ComfyUI or reads "outside ComfyUI". A scrub replaces anything still absolute, the key or the home folder and counts it in redacted (0 for a correct build). The version parser moves unchanged to utils/version.py. (#329) * fix(logs): never empty a renamed log or serve a swapped file; keep root anchors private Review findings on #329 PR 1: - Clear empties the active log through the handler only when the handler's open stream is the listed file (same inode); a log the user renamed while ComfyUI runs is no longer emptied. - A download is refused when the opened file is not the one listed (inode), or has another name (a hard link). - Windows reparse points that name another file (the name-surrogate bit: symlinks, junctions, WSL links) are links under Rule R; cloud placeholders stay regular files. - The report's paths use the closest anchor that is not a filesystem root, and the scrub also catches the home folder without its root. * fix(diagnostics): never anchor a report path above the home folder; serve only regular log files Review findings on #329 PR 1: - report_path never uses a filesystem root or a strict ancestor of the home folder (/home, C:\Users) as its anchor, so a path inside home needs an anchor at or below it and otherwise reads "outside ComfyUI"; no path starts with a user name. The scrub matches the home folder only as whole path parts (in full, without its root, and its name). - A download serves only a regular file (a FIFO, socket or device is a 404) and opens it non-blocking. - One unreadable timestamp no longer breaks the log listing, stats, download, review or Clear. - The report tests read a temporary log folder, never the process's own. * fix(diagnostics): match the bare home folder name only inside paths A home folder named like a word (/home/info) no longer removes the log level INFO from the report; the name still goes when it is a part of a path. Documents the accepted residual risk of another user's home. (#329) |
||
|
Vito
|
d868745194 |
fix(lora): cache LoraManager's layout and close the #355 follow-ups (#362) (#364)
* feat(lora): cache LoraManager's layout behind file signatures (#362, phase 1) A new py/lora_layout_cache.py keeps one (install, layout) entry and reuses it while the configured path, the plugin settings file's and the settings file's signatures (st_mtime_ns, st_size), the install and every media root's real path are unchanged and the entry is younger than 30 s. A load whose watched file changed within 3 s (git's racy rule), could not be checked, or whose settings were unreadable serves its request only. clear() bumps a generation so a load that started earlier stores nothing. Hits take no lock. Not wired yet; phase 2 puts it behind the image hot paths. * fix(lora): treat an offline Windows share as unvalidated and make cache tests portable (#362) Review findings on phase 1: Windows raises FileNotFoundError with winerror 53/67 for an offline share, which is a temporary failure, not a missing file. Test times are whole milliseconds (NTFS keeps 100 ns steps) and a file system that rounds them skips; cleanup no longer ignores errors; tests added for ValueError, NotADirectoryError, a failed reload, resolve failures and the watched-file set. * perf(lora): serve the image hot paths from the layout cache (#362, phase 2) _lora_image_dirs takes (install, layout) from lora_utils.cached_lora_manager_layout instead of resolving the install and reading LoraManager's settings files on every Library, inspect, reveal and serve request. serve_image builds the allow-list in the executor instead of on the event loop. A finished import (System job or legacy scan, both through replace_lora_imports) clears the cache, so files linked under a folder LoraManager just created are served at once. The integration-off check and the configured-path rule still run before the cache. * fix(lora): fingerprint every scanned sidecar, kept or skipped (#362, phase 3) Discovery keeps every metadata file it saw (scanned_files, resolved and sorted; scanned is now its length). The fingerprint hashes each one's path, size, modification time and whether it was kept, so a skipped sidecar added, removed or edited between the review and Start refuses Start with plan-changed, as the review's counts promised. * fix(lora): warn about refused default LoRA folders and log what failed (#362, phase 4) ComfyUI's default LoRA folders already pass the folder guard; a refused one that exists is now listed in the layout's refused_defaults, and the layout cache warns once per newly refused folder. The cache writes one INFO line when a stored layout differs from the last one logged. BAD_SIDECAR_ROOT and the module docstring name a folder that contains the home folder. A relative preview_url writes a DEBUG line before the suffix search; a non-OSError failure while loading trigger words logs its traceback. Review follow-ups harden two serve tests. * fix(lora): skip a sidecar that is not UTF-8 instead of failing the load (#362) Review follow-up on phase 4: read_lora_metadata caught JSONDecodeError but not UnicodeDecodeError, so one non-UTF-8 sidecar failed the whole trigger-word load (now logged with a traceback every retry) and the import review. It is skipped like any unreadable sidecar. The relative preview_url DEBUG line uses lazy formatting. * docs(testing): the LoRA linked-only gate, layout cache and fingerprint scope (#362, phase 5) TESTING.md's LoRA reference import contract now states what #358 and #362 shipped: the linked-only gate and _media_roots, linked_only media, the cover link order, the guard on ComfyUI's default LoRA folders and refused_defaults, the folder-unreadable code, the trigger cache's catch and 60 s retry, the fingerprint over every scanned sidecar, and the layout cache's key, signatures, racy window, maximum age, root re-check and clears. The targeted command includes tests.test_lora_layout_cache and tests.test_api_images. * fix(lora): a cached layout needs the same home folder; give the rebuild poll its time (#362) Final review follow-ups: the folder guard compares against the home folder, so a cache hit now also needs Path.home() unchanged. TESTING.md says the configured-path rule runs on the cache's result and that a non-UTF-8 sidecar is skipped. CI: 'Rebuild all is explicit...' (webkit) polled the job for the default 5 s while it regenerated 24 previews; the failure screenshot shows 12 of 24 done at the deadline on the busy runner (0.17 s locally). The poll now allows 30 s for that real work. |
||
|
Vito
|
1bb3678f32 |
fix(threads): describe the teardown as built, warn only for a live observer, and keep an interrupted teardown's result (#377) (#378)
- _stop_background_work pre-fills the started parts with False and stores the result and removes its atexit entry in a finally:, so a Ctrl+C between steps no longer leaves the result unset and the callback registered. - The "watchdog observer still running" warning moves into _stop_observer: only for an observer still alive after its join, naming the join's own timeout (not after observer.stop() raised, and not the full deadline). - Tests: the plain-teardown tests share one package; two teardown tests run with a live preview worker held in a fake encode (ffmpeg killed, step gets min(CLOSE_WAIT_SECONDS, time left), thumbnails False while held and True when the kill releases it); an interrupt-between-steps test; two warning tests. - TESTING.md and the ledger fixture comment: the refusal comes first, the image worker and the observer share one 5 s deadline, and what an interrupt between steps does. Closes #377 |
||
|
Vito
|
2f67b98287 |
fix(tagging): save WD14 confidences only on the 0.05 grid every page shows (#369) (#373)
* fix(tagging): save WD14 confidences only on the 0.05 grid every page shows (#369) A saved threshold off the 0.05 step (for example 0.37) blocked Generate and Start in the workspace Tag image dialogs (native step mismatch) and made the legacy sliders send 0.35. The server now refuses such a save with a message that names the setting, a stored one falls back on load, and an accepted one is stored exactly on the grid. System > Auto-tagging refuses the same values before Preview tagging review or the batch Review, and hands thresholds on snapped to the grid. Per-request thresholds keep their own checks. * test(browser): restore the safety backup policy on system reset The Recovery policy test in workspace-system-backups saves safety_backup false at runtime, and /__test__/system/reset restored config.json but not PromptManagerConfig.SAFETY_BACKUP. A later spec on the same server (Tag output images review) then saw "No safety backup (the policy is off)". The new #369 test shifted the CI shards so the two met on one worker. |
||
|
Vito
|
f3b9f61f11 |
fix(threads): stop background work at exit, on cleanup and on the tests' unload (#301) (#374)
* fix(threads): stop background work at exit, on cleanup and on the tests' unload (#301) - __init__.py: _stop_background_work() detaches the monitor's callbacks, stops the monitor, closes the preview worker and shuts the prompt cleanup thread down against one 10 s deadline, once. It runs from atexit (ComfyUI never runs aiohttp's cleanup hooks), from on_cleanup where a host runs it, and from the tests' package unload; it removes its own atexit callback. Registered after the history hook, so at exit it runs before the history dispose. - AutoThumbnailer.close/close_auto_thumbnails take a timeout and return whether the worker ended. - Tests: teardown result, once-only, on_cleanup, atexit order, a failing step or a second Ctrl+C never skipping the tracker, one shared deadline, three repeated loads leaving no thread, and a child process that exits normally with the image worker alive. * fix(threads): refuse new previews before the monitor stops; isolate the exit child (#301) Review findings: - The teardown makes the API refuse new previews before it detaches the monitor's callbacks: an image in hand that already read the preview queue no longer starts a preview worker during exit (Python 3.12+ refuses that thread and logs a traceback). New refuse_auto_thumbnails(). - The teardown imports its bounds inside each step and falls back to the standard logger, so a failing import never skips the other steps. - The exit child keeps its library and log in its own folder (PROMPT_MANAGER_DATA_DIR), reports watchdog's threads too, fails with the child's output on a hang, and rejects "Error while stopping". - TESTING.md: "Background threads at exit". |
||
|
Vito
|
b1e7abd380 |
fix(threads): stop and join the prompt cleanup thread, the image worker and the observer (#302, #370) (#372)
* fix(threads): stop and join the prompt cleanup thread, the image worker and the observer (#302, #370) - PromptTracker: the cleanup thread waits on a stop event instead of time.sleep; new shutdown(timeout=2) joins it and returns whether it ended. - ImageGenerationHandler.shutdown(timeout=5) now joins the worker and returns a bool; new close() is final: schedule() refuses and a sealed run's history re-check is dropped, so no worker restarts afterwards. - ImageMonitor.stop_monitoring(timeout=5) closes the handler first, then stops and joins the watchdog observer within one deadline. - Every test that builds a real tracker, handler or monitor ends with its final stop and an assertion, before its folder and databases go. - #370: the wrapper/session history test drains before dispose(); the held-write bound test uses 30 s ceilings instead of tight wall clocks. * test(history): drain the writer before asserting a bounded close it does not test (#370) * fix(threads): a later stop waits again for a worker or observer an earlier stop left running (#302) Review findings on #372: - ImageGenerationHandler keeps the workers a stop signalled that had not ended; every later shutdown/close joins them again, so a second call no longer returns True while the first worker still runs. - ImageMonitor.stop_monitoring keeps an observer or handler it left running and stops and joins it again on the next call. - A closed handler takes no awaiting history path, and a re-check that raced close() is not kept due. - Tests: second stops of a busy worker and a live observer, the closed handler's re-check paths, the handle test asserts the worker ended and stops before it asserts, and a no-new-thread check instead of set equality. * test(browser): the System reset restores every settings value, safety_backup included The frontend coverage shard of #372 failed in workspace-system-tagging: the review read "No safety backup (the policy is off)". The backups spec saves safety_backup=false, and /__test__/system/reset restored config.json but not PromptManagerConfig.SAFETY_BACKUP (nor INFINITE_SCROLL), so a later spec on the same worker server saw the policy off. The reset now captures and restores every non-tagging value POST /settings can change. |
||
|
Vito
|
ae755d2ce0 |
feat(system): Auto-tagging screen, native in System (#328, PR 3) (#368)
* feat(system): configure auto-tagging natively * test(system): compare Auto-tagging with the concept against the real backend |
||
|
Vito
|
4032f4b6ab |
feat(system): one tagging configuration and Tag output images as a reviewed job (#328, PR 2) (#366)
* feat(autotag): model status without paths, and a busy answer that names the operation * feat(autotag): model downloads run as Activity jobs * fix(autotag): keep each download route's error shape and harden the fake download * feat(autotag): one server-side tagging configuration * fix(autotag): the legacy batch and review work again * feat(autotag): tag output images as a reviewed Activity job * test(autotag): the legacy batch, review and unload page workflows * fix(autotag): server-free tagging defaults, cancel before load, and named legacy refusals * fix(autotag): name only the workspace batch in a legacy refusal frame |
||
|
Vito
|
88fa13689a |
feat(system): model status, a named model gate and downloads as Activity jobs (#328, PR 1) (#365)
* feat(autotag): model status without paths, and a busy answer that names the operation * feat(autotag): model downloads run as Activity jobs * fix(autotag): keep each download route's error shape and harden the fake download |
||
|
Vito
|
cfe47d3cb0 |
ci: support and test Python 3.12-3.14 only (#356) (#357)
* ci: test Python 3.11-3.14 and require Python 3.11 (#356) Python 3.10 reaches end of life in October 2026 and no PromptManager users run it; 3.14 is current and was untested. The test matrix runs 3.11-3.14 on Linux, Windows and macOS, bandit runs on 3.11, and requires-python is >=3.11 so the package no longer claims a version CI does not test. The packaging test drops its 3.10-only tomllib fallback. The required checks follow the matrix (py3.10 -> py3.14). * ci: narrow Python support to 3.12-3.14 (#356) ComfyUI recommends 3.13, names 3.12 as the fallback and 3.14 as working (its README); PromptManager supports exactly those. requires-python is >=3.12, the test matrix and the required checks drop 3.11, and bandit runs on 3.12. |
||
|
Vito
|
fd949b870c |
test(admin): wait for the restore response, not a timer (#359) (#363)
The legacy restore specs "restore validation and cancellation preserve live data and allow a valid retry" and "backup and restore recover from interrupted network requests without losing data" ended with a 5 s toBeHidden on the restore modal. The modal closes only after the server answers, so that was a 5 s deadline on a disk-bound restore, which WebKit CI runners sometimes exceed. Both specs, and the downloaded-backup spec that already waited on the response, now share confirmRestoreAndWait: it waits for the POST /prompt_manager/restore response, asserts a 200 (showing the body on failure) and the success body, then checks the modal closes. TESTING.md explains how to reproduce disk-bound timeouts with --device-write-iops in the pinned image. |
||
|
Vito
|
80c6ec5340 |
fix(lora): find LoraManager's sidecars, previews and example images in every storage configuration (#355) (#358)
* fix(lora): read LoraManager's settings and storage layout the way it does (#355) Add py/lora_manager_layout.py: the settings file in LoraManager's lookup order (environment folder, portable plugin file, user config folder), only the seven keys the import needs, the LoRA roots, the centralized sidecar root, the example-images root and the active library, each folder through one guard. Sidecar discovery for both storage modes and the local preview and example-image lookup live there too. The test runner and the browser test server point LoraManager's settings lookup at a disposable folder, so no test reads the real settings. * fix(lora): import the sidecars and local images LoraManager really uses (#355) The import, the legacy scan, the trigger-word cache and the image server now share LoraManager's layout: sidecars beside the models (alongside) or under the sidecar root (centralized), kept only when their model exists inside a LoRA root. Each LoRA links its LoraManager preview and example images where they are; CivitAI downloads run only for a LoRA without any local image. The review adds skipped, with_local_media and the layout, and offers no token when LoraManager's settings cannot be used; the legacy scan reports the same reason without replacing anything. find_lora_directories becomes default_lora_roots (no scan of the LoraManager install); the preview and example-folder helpers it replaced are removed. The trigger cache never raises into a running node. * fix(lora): never serve a LoraManager folder that holds the home folder (#355) The folder guard also rejects ancestors of the home folder (/home, C:\Users), compared in the realpath + normcase form, so a LoraManager setting cannot widen the image server's allow-list to every user's files. * fix(lora): show linked LoraManager videos as videos (#355) _enrich_images marks every image without is_video by its extension, so an .mp4 preview outside the output folders shows the Video preview label and plays in the inspector. Tests cover serving the sidecar and example-image folders only while the integration is on, and a filesystem root as the sidecar folder adding nothing. * feat(lora): the import review names the LoraManager settings it read (#355) The review gains a first row, LoraManager ("<source> settings · <mode> storage · library <name>", the settings file as its hover title), and its Found row counts metadata files, LoRAs and LoRAs with local images. A plan whose LoraManager settings cannot be used takes the close-only path: "LoraManager settings: not usable: <reason>. Nothing was changed." The browser fixture writes each LoRA's model file and can leave out the local preview, so the download, gate and cancel specs still exercise CivitAI. * docs(lora): describe how the import finds LoraManager's files (#355) * test(lora): set the settings folder relative to the process folder (#355) os.path.relpath fails across Windows drives (the temp folder on C:, the checkout on D:); the test now changes into its temp folder instead. * fix(lora): review findings for the LoraManager layout (#355) - The example-images library subfolder goes through the folder guard like its root, so its canonical (normcased) form matches the media check on Windows for a library name with capitals. - A failed trigger-word load is retried by a running node only after TRIGGER_RETRY_SECONDS, not on every prompt run. - A review whose discovery cannot list a folder says so (folder-unreadable, the folder's name only) instead of a generic error. * test(images): hold the listing clock in the deferred-walk test test_a_deferred_walk_waits_at_least_half_a_second set the previous walk 29.99 s back against a 30 s interval and then read the real clock again: a runner slower than 10 ms between the two lines passed the interval and the walk started at once (failed on macOS 3.13 and Windows 3.11). The module's clock is now held for the mark, so the deferred walk is always the one measured. * test(lora): write the unkept-settings fixture as raw text CodeQL read the dict with a credential-named key flowing into write_text as clear-text storage of a secret. The value is a marker, so the fixture now writes the JSON text directly and names the value MARKER. * fix(lora): link each LoRA's cover as its newest image (#355) Workspace lists show a prompt's media newest link first, so the import links a record's images last to first: the card shows the cover (a still in front of a leading video) the import put first. * fix(lora): show LoRA-linked media on the Library page and in the inspector (#355) Library previews, the positive detail and the per-prompt images page list a prompt's linked media under the output folders plus, while the LoRA integration is on, LoraManager's guarded folders and the download cache (_media_roots; output root indices keep their meaning). That media keeps its rooted identity, so Info and the media viewer open it; the path-based serve route answers for those roots only with a file the database records (either path spelling), never serves or makes a preview there, and output requests never read LoraManager's layout. Such media is marked linked_only: no preview file, no favorite star, no Tag action. Show in folder accepts those roots; the download cache is named, not created. * fix(lora): expose only linked files under LoraManager's folders (#355) Serve, inspect and Show in folder share one gate for a LoRA-linked root (_media_roots past the output folders): a file the database does not record as a linked image is refused with 403, checked by its given and its real path. Before, inspect and reveal read any file under those folders by path. * fix(lora): check the file Show in folder would open under a LoRA root (#355) reveal_image resolved the requested path through the client-path resolver, which can fall back to a path anchor; under a LoRA-linked root the file that would open now passes the same recorded-media gate, so only a linked file is ever revealed there. |
||
|
Vito
|
c7ba481d3c |
perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) (#353)
* perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) The image monitor now reports every settled output (linked or not, also when linking raised) with its resolved path through set_output_listener. The API's notify_output_settled hops from the monitor's worker thread onto the server's loop (call_soon_threadsafe) and does nothing before the loop is known or after cleanup. mark_root_dirty matches the path against each walk's resolved_root (normcase(realpath), computed in the executor by walk_root, now its fourth value) and marks the root dirty. A dirty root gets one background walk, no sooner than rewalk_interval = min(15, max(5, 5 x last walk's duration)) after the previous walk started (last_started_at / last_duration are recorded on every walk). _schedule_walk is shared with PR 1's delayed stale refresh: one deferred walk per root, never armed while a walk runs (the walk's end re-checks dirty: the trailing walk), at least 0.5 s away, the earlier deadline wins, and every walk start cancels it. A failed walk clears dirty. Rescan semantics are unchanged. * fix(images): keep removed roots and cancelled walks from walking again (#348, PR 2) Review findings on the monitor-triggered walks: - reset_gallery_roots_cache now retires every listing record: its pending timer is cancelled, dirty is cleared, and no mark, timer or trailing walk starts a walk of it until a reader or the warm-up asks for that root again. Before, a timer or trailing walk pending when the roots changed stored a listing for the removed root, and the monitor (which watches the old roots until restart) kept it walking on every new output. - A walk task cancelled outside cleanup_gallery_listings no longer starts a trailing walk while the loop shuts down. - root_contains ignores a Windows extended-length prefix (\\?\ and \\?\UNC\) on either side, which realpath can keep on a file past MAX_PATH. - The REWALK comment states the ~20% GIL share only for walks up to 3 s. * test(images): prove a cancelled walk starts no trailing walk (#348, PR 2) The previous assertions held even with the trailing walk, because that walk finished before they ran; assert the walk number and the scan count. * perf(images): add a settled output to the listing without a walk (#348, PR 2) Live test: each landing in a 20,000-file folder cost a full walk (~0.4 s of GIL) plus a full workspace rebuild, so Images reads during a run took up to 2.1 s. The monitor's report now patches the stored listing instead. - output_settled (the monitor hop's target) slices the file's part below the root in its own case (relative_part; Windows compares folded), and _run_adds places one root's pending files in one executor batch: _settled_entry applies the walk's filters (suffix, thumbnails, depth, symlink by lstat; macOS on-disk spelling), insert_entry puts the (Path, mtime) entry where the newest-first sort does, add_folder copies the folder sets. Stored only when no walk started and no drop happened meanwhile; walked_at is untouched, so the 30 s refresh still walks. - Anything inexact marks the root dirty and walks as before: a running walk, an unsliceable path, a failed lstat, a tie in mtime, the cap, an unexpected error. Rescan, the refresh, the warm-up and invalidation stay full walks; a walk start forgets the pending adds. - The workspace caches follow a patched listing by entry identity (listing_delta): only added values are built, the newest order places them by bisection, the favorites lookup is copied and extended. Synthetic 20,000 files: per landing 252 ms (walk + full build) -> 18 ms (3 ms add + 15 ms warm build). * fix(images): review fixes for adds without a walk (#348, PR 2) - _newest_step steps the cached newest order only when the per-file cache still holds exactly the values this call built. A stale page read racing a newer warm build (both run in executor threads) stored an order labelled with older values than it held, and the next read inserted the new file a second time. It also sorts in full when a roots reset dropped the per-file values meanwhile. - The stored spelling is checked on Windows too (nt._getfinalpathname: realpath keeps the given case when it cannot open the file), and before the name filters, which compare the walk's stored spelling. macOS opens non-blocking, so a FIFO named like an image never holds the executor. - An output in a folder the walk cannot list (no read permission) is not added, as the walk skips it. - Windows-portable unit tests: POSIX paths in the insert tests, and a str.lower fold for the length guard (Windows' normcase keeps lengths). * test(images): reproduce the newest-order race and fix a Windows clock flake (#348, PR 2) - The race test now has the newer build step from the values the cached order was built from, which is the interleaving that inserted a file twice; it fails without the parts check in _newest_step. - test_a_failed_walk_records_its_timing_and_clears_dirty compared two monotonic readings that Windows' ~15.6 ms clock can make equal; the previous start is moved back instead. - TESTING.md: os.access ignores ACLs on Windows. |
||
|
Vito
|
9ac471eb72 |
perf(images): share and warm the Images listing (#348, PR 1) (#349)
* perf(images): share and warm one listing per output root (#348) The Images page, its sidebar counts, the subfolder list and the legacy gallery each rebuilt the output listing when the 30 s cache expired, all at once, and the subfolder list computed relative paths on the event loop. Thumbnail writes also threw the listing away. Keep one RootListing per output root (py/api/gallery_listing.py): - one walk per root at a time; concurrent readers wait for it (shielded) - a listing older than 30 s is served at once and refreshed in the background - the walk, the comparison with the previous listing and both folder sets run in the executor; an unchanged walk keeps the same list object - os.scandir probe: a missing root fails the walk and keeps the previous listing (one warning per failure streak) - Rescan waits for a walk started after the click and is a 500 when it fails, even with a previous listing - deletes keep the blocking invalidation; a generation counter that survives drops discards walks that raced it - thumbnail jobs, conversion and the legacy generate/clear routes no longer drop the listing - warm-up on the host's on_startup, cancelled on on_cleanup Adds tools/vendor/perf/images_load_probe.mjs (maintainer tool, not CI) and the TESTING.md section "Images listing cache (#348)". * fix(images): drop listings on the loop when invalidated from a worker thread (#348) A settings save (gallery roots change) and the browser-test seeds call invalidate_gallery_cache() from the executor. Listing records are only touched on the event loop, so an off-loop call now hands the drop to the loop and waits for it, keeping the call synchronous for its callers. The shared-Rescan test waits for the later Rescans to take their walk number instead of sleeping. * test(images): bound the shared-listing test waits (#348) A macOS py3.12 run stopped inside the shared-listing tests and sat until the 6 h job limit with no traceback. It does not reproduce locally (about 300 runs on 3.12 and 3.14, pinned to one core). Requests and background settles in TestSharedListing now give up after 30 s, so a recurrence fails with the stuck await instead of hanging the run. * test(images): let the settle loops yield so a finished walk leaves the set (#348) On CI (ubuntu and macOS py3.12) the runner's watchdog dumped test_a_root_that_recovers_warns_again_on_its_next_failure spinning in _settle: asyncio.gather over tasks that are already done returns without yielding, so the set's discard callback, queued when the walk finished, never ran and the loop never ended (wait_for could not fire either). Delaying discard by one loop turn reproduces it locally; both settle loops now yield one turn after each gather. * test(browser): poll the import review's rows instead of reading them once gridOf reads the dialog's rows once; right after the click the review may not have rendered them yet (firefox CI: received undefined). * perf(images): walk the output with scandir entries and build folders from strings (#348) The live test showed a read during a background refresh taking 2.2 s instead of 0.07 s: the walk held the GIL for about 2 s in the server. py-spy split that walk into two halves: Path.relative_to per file in folder_sets, and Path.stat + Path.is_symlink + Path joins per file in the scan. - _walk_media visits folders in os.walk's order with os.scandir entries: the file type comes from the listing (no lstat per file) and the mtime from entry.stat (one call, none on Windows). _iter_media_files is now a thin wrapper, so its other caller is unchanged. - folder_sets slices the path strings (in the path flavour's separator) instead of calling relative_to, falling back to it for a path that does not start with the root string. On the maintainer's 19,935-file output folder: scan 0.386 s -> 0.112 s, folder sets 0.452 s -> 0.007 s, both results identical to the previous code. A reference copy of the os.walk version checks order, filters, the depth cap, symlinks and the file cap. * perf(images): build the workspace caches behind the warm-up and changed walks (#348) The live test after the faster walk still took 2.9-5.2 s for the first Images open after a restart. py-spy showed the walk was now small; the time went to the workspace's own caches, cold after a restart and built by three parallel first requests at once: the favorites listing (Path.relative_to per file), the per-file values and newest order, and the prompt-link read. - Once the host's startup warm-up has run, a stored walk that changed a listing schedules one executor job that builds what a first page reads, through the request path itself (_select_workspace_page with the default filters, then _workspace_favorite_paths). A walk stored during a build asks for one more build; an unchanged walk (same list kept) builds nothing; a failed build only warns; on_cleanup stops it. Servers that never ran the warm-up (the test servers) start no build. - _favorite_listing slices the path strings as _listed does, falling back to relative_to for a path outside the root string. * perf(images): start a stale listing's refresh walk a second after the read (#348) After the faster walk, a read that found the listing stale still took 0.5-1.0 s against 0.07 s: it was served from the old listing but ran beside the refresh walk it had just started, sharing the GIL. The refresh now starts REFRESH_DELAY (1 s) after that read through a per-root rewalk_handle, so the page's own requests finish first. One pending refresh per root and none while a walk runs; any walk start (a cold read, Rescan, the warm-up) cancels a pending one; on_cleanup cancels them before its first await. * test(browser): accept KB or MB for the picked safety backup's size The restore preview showed '2.5 MB' for the test server's database copy in a CI coverage shard: its size depends on what earlier tests on that server wrote, and the formatter switches to MB at 1 MiB. * fix(images): keep the 3.10-3.13 suffix rule in the scandir walk and harden the warm-up (#348) Review of the walk speedup: - os.path.splitext drops names like ..png that Path.suffix listed on Python 3.10-3.13 (3.14 changed Path.suffix, which hid it from the parity test on a 3.14 venv). _name_suffix spells out the 3.10-3.13 rule; the test now asserts ..png and ...webp are listed and g. is not. - _warm_workspace reads the roots inside its try, so a failing _get_all_output_dirs warns instead of leaving an unretrieved task error. - The folder_sets fallback test now takes the fallback (a Windows case mismatch); the coalescing test pins the walk count. |
||
|
Vito
|
5772de8d27 |
ci: bound every job and make hangs name themselves (#350) (#351)
* test: run the suite through a watchdog that names a hang (#350) tests/run_suite.py discovers the same tests as unittest discover -s tests, with the same sys.path and exit codes (5 when none ran), writes 'start: <test id>' to stderr before each test and arms faulthandler.dump_traceback_later: 120 s per test (PM_TEST_HANG_LIMIT) and 120 s after the run (PM_TEST_SHUTDOWN_LIMIT), never cancelled. A hang in a test, its teardown, its event-loop shutdown or interpreter exit prints every thread's stack and exits non-zero instead of running to the job limit. The pre-push hook now runs it too. * ci: bound every job and run the Python suite through the watchdog (#350) Every job in test.yml and code-quality.yml gets timeout-minutes = ceil(1.5 x slowest of three green main runs) + 3, min 5, with the runs and times written above jobs:. The matrix and the coverage gate run python -m tests.run_suite. ci-workflow.test.mjs checks every job in the three CI workflows has a limit and that the matrix and coverage use the runner. Job names are unchanged. * docs: document the test runner, time limits and following a push (#350) * fix(test): keep unittest's warning filters and tighten the runner's edges (#350) Review follow-ups: the runner now shows warnings as unittest.main does (DeprecationWarning and ResourceWarning were hidden), rejects non-ASCII digit limits with the one-line exit-2 error, and starts each 'start:' line on its own line at default verbosity. The docs now say that class and module fixtures count toward the test before them, that the shutdown limit covers interpreter exit, and that exit 5 on zero tests is 3.12+ behaviour applied on every version. * test: decode the runner's stderr as UTF-8 on Windows too (#350) A child Python writes a piped stderr in the ANSI codepage on Windows, so the non-ASCII limit value in the usage-error test came back as U+FFFD. The helper now sets PYTHONIOENCODING=utf-8 for the child. * test(history): give the writer's non-blocking checks a 1 s ceiling (#350) py3.13 on macos-latest failed on PR #351 with writer.close(0) taking 0.186 s against a 0.1 s ceiling while another thread held the commit. The blocked path these checks rule out waits up to 3 s (release.wait(3)), so a 1 s ceiling still tells returning at once from waiting for the release, without failing on a loaded shared runner (TESTING.md, #242). * test(restore): wait for the cancelled preview's worker before teardown (#350) py3.13 on ubuntu-latest failed on PR #351 (run 40334794505) with 'Files still open at teardown: prompts.db, prompts.db-shm' in test_cancel_while_reading_the_live_library. Cancelling the preview cannot stop its executor thread, which goes on to open the live library; the test returned without waiting, so that open could land between teardown's close_all_instances() and assert_closed(). The test now waits for the worker, as test_cancel_while_staging already does. |