b264dac2e8d1bea9ff7ef7cd6c61d3c19145dc98
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
8870496ea3 |
feat(monitor): store certain prompts from workflows without a PromptManager node, with a System switch (#247, PR 2) (#467)
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) When a file's own graph leaves the prompt open, both inspect routes ask recorded run history about the file (one run reported it and it was written during that run). The save node that run reported it from anchors the walk like a file-name match, so dataset branches that share one saver prefix show their own prompt, labelled "Recorded run". Still open at one sampler whose PromptManager text was recorded at execution (a language model's output), the recorded text shows as "Execution recorded". Results the file resolves alone never change; a failed lookup judges the file alone. * fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386) Review follow-ups: the texts PromptManager recorded are looked up at the sampler history's saver leads to, never at a sampler the unanchored walk named; a file name naming a different saver than history anchors nothing; outputs and texts are read in one transaction; "Recorded run" ranks below every workflow-inferred source; unexpected lookup errors log at error level. * fix(history): look up recorded runs by history's own spelling of the file (#386) The history route inspects a case-normalized target on Windows, while history matches file names exactly, so the occurrence's reference names the file. * refactor(inspector): share the inspector's resolve as inspection_resolution inspect_prompt_evidence now calls inspection_resolution, which returns the resolution to use, the anchored result and whether the recorded saver chose it. Ingestion can call the same helper for files the default profile cannot decide (#247). * feat(ingestion): one per-file prompt decision for the scan and the monitor utils/file_prompt.py holds decide_file_prompt, the ladder the output scan runs today plus the inspector's rule (inspection profile, then the recorded saver) before the legacy parser, with a SCAN and a MONITOR policy. Junk text (blank, numbers, LoRA tags only, model file names) is never stored. graph_prompt_text and parse_comfyui_prompt move here; the parser accepts parsed metadata (#247). * feat(ingestion): store a file's prompt in one transaction, one prompt per file database/scanned_prompts.py links a file to the prompt of one text, creating it when new, inside one BEGIN IMMEDIATE transaction: a file linked to another prompt is a conflict, a failed link leaves no prompt behind and two writers store one prompt. pair_ingestion gains file_link_rows, link_conflict and Marks/mark_created; ingest_resolved_pair takes optional marks for the components it creates. recorded_runs and named_prompt in run_attribution become public (#247). * feat(scan): decide each scanned file with the shared ladder (#247) _ingest_scanned_file calls decide_file_prompt (scan policy) and stores the result: pairs through ingest_resolved_pair and positives through store_scanned_prompt, both with the scan marks. Files the default profile leaves undecided now get the inspector's resolve, with the saver recorded history names, before the legacy parser, so each branch of a shared-prefix dataset run links its own prompt. Junk text is never stored, a file linked to another prompt is a conflict instead of a second link, and a failed link leaves no prompt behind. _BatchHistory reads the stored prompt and the recorded saver from one recorded_runs pass. _parse_comfyui_prompt calls the moved parse_comfyui_prompt. TESTING.md documents the decision. * fix(ingestion): keep the file-name saver anchor out of the ladder (#390) The #390 ruling queues a file that only its saver's file name resolves for Needs review, never linking it. inspection_resolution gains name_anchor; the ladder passes False, so such a file falls to the legacy floor in the scan (and to Needs review) and is never stored by the monitor, while the recorded saver still decides. The inspector keeps the name anchor. * feat(config): gallery.monitoring.save_foreign_runs, default on (#247) * fix(ingestion): no joined PromptManager text from the legacy floor; junk keeps its step's status The legacy parser returns prepend + text + append, so its answer is refused when a PromptManager-family node has a prepend or append part (#343). Junk found at the undecided-graph step keeps that step's unlinked status and review, so an ambiguous file is still queued for Needs review. The monitor policy's ambiguous file no longer asks for the recorded run, and the file size may be read lazily. * fix(ingestion): keep the legacy answer that is a PromptManager node's own text The joined-text refusal also refused the #170 parity files whose PromptManager prepend is a link the browser parser does not follow: its answer is then the node's literal text, which is what the node stores. Refuse only an answer that is not such a node's own text. * feat(monitor): store the certain prompt of a foreign workflow's output (#247) A foreign output (no positive PromptManager node) is decided by the shared ladder with the MONITOR policy. Certain files are stored at settle with RUN_MARKS through survive_switch; files only recorded history can decide wait in their own bounded set and are re-checked once after the run's outputs are sealed, with the sealed reference. The setting GalleryConfig.SAVE_FOREIGN_RUNS is read for every file. * fix(scan): one prompt per file for recorded-run links; writes survive a library move The recorded run's stored prompt now links in one transaction with the file's link rows, so a file linked to another prompt is a conflict instead of a second link. The scan's stores run through survive_library_switch. A single file asks history once (a one-file _BatchHistory), the scan passes in_output=False and reads the file size only for files with prompt metadata. A conflict row reads "Kept: already linked to a different prompt". TESTING.md records the rules. * feat(settings): save_foreign_runs through POST /settings, applied live (#247) GET /settings reports settings.save_foreign_runs; a save takes a strict bool, writes gallery.monitoring.save_foreign_runs and publishes it to GalleryConfig in the commit, with no restart field. The diagnostics report lists it. The legacy contract snapshot gains the added key (an addition, no legacy key moves). * feat(system): Save prompts from other workflows in Sources & scans (#247, D147) A host setting row below Watch for new outputs drafts save_foreign_runs and saves with the one Save row; the server applies it at once, so no restart is named. While monitoring is drafted off the row is disabled and says it applies while Watch for new outputs is on. The concept comparison maps the inserted row and its mutation check covers it. * test(comfyui): a workflow of ComfyUI's own nodes creates and links its prompt (#247) The real-host attribution spec gains the foreign case (CLIPTextEncode, KSampler, SaveImage): its prompt is created and linked at settle and a second run links to the same prompt. The fixture route lists the waiting foreign files. TESTING.md records the monitor's foreign path and D147. * test(vendor): the execution matrix counts the #247 real-host case * fix(monitor): read a foreign file's size only when the ladder needs it (#247) Uses PR 1's lazy file size and pins that a foreign file linked to another prompt keeps its one link on the pair and the positive store paths. * fix(ingestion): match a file's link rows under its resolved spelling too The scan stores realpath spellings and the monitor the watcher's raw path, so a file reached through a symlinked or junction output folder could get a second generated_images row. file_link_rows now queries the path and its realpath (each normalized and raw) and dedupes by row id. TESTING.md notes the reverse order is matched only when the raw spelling is the realpath. * fix(monitor): wait for the run unless history answered; monitoring off stops foreign writes (#247) Review fixes: at settle a certain answer ends the wait only when the lookup found a record or the answer is the inspector's or the recorded saver's; the legacy floor waits for the re-check. A record without a certain prompt ends the wait at once. Foreign writes need both save_foreign_runs and monitoring_enabled, read again just before the store. Dropped foreign files are logged outside the lock (evictions as evictions, due ones too), and a drained re-check never drops a newer reference. * fix(system): the foreign row can be set while the monitor still watches; its note describes it (#247, D147) Review fixes: while monitoring is saved off but a source still reads watching until restart (D128), the row stays enabled with its note; the note is the switch's aria-describedby. The browser spec restores save_foreign_runs on the shared server. TESTING.md moves D147 into order and records the monitor's review rules. * test(comfyui): each branch of a shared-prefix foreign run links to its own prompt (#247 CUJ 1) * fix(ingestion): a switched-off PromptManager node behind a pass-through stores nothing (#90) The #90 graph walk follows only prompt-like inputs, so a PromptManagerText with Save to library off reaching the encoder through another input name (easy showAnything.anything, a custom node's value) was invisible to it, and the inspector's resolve stored its text. The ladder now checks the positive every step names: a resolved or inspected positive whose path passes a node that is off, or a legacy answer that is such a node's own text, makes the file not_saved. * fix(ingestion): any switched-off node upstream of a positive stores nothing (#90) The ladder walks every input upstream of each sampler or guider positive (not negative, model, clip, latent or image inputs) for PromptManager nodes whose Save to library is off or wired from an unreadable node, so the sampler-less step, an off node with a prepend part and a PromptManagerPair behind a pass-through no longer store its text. The legacy-floor guard reads a node's resolved text. Every guard reports not_saved for a switch that is off and not_saved_wired for one it cannot read. * fix(ingestion): the switch walk skips negative inputs by #90's own rule The upstream walk treated only names containing "negative" as negative, so an off node on a neg, neg_text or text_neg input blocked an enabled positive. prompt_graph gains is_negative_input (the _NEGATIVE_KEY rule, also used by follows_input) and the walk uses it. The guards accept a node with null inputs and a side whose provenance is not an object. * fix(ingestion): the switch walk also skips any input named with "negative" #90's rule matches only whole underscore-separated words, so negative2, negativeprompt and NegativePrompt counted as positive inputs and an off node wired only to one of them blocked the file. The walk skips those too; follows_input is unchanged. * fix(monitor): a foreign not_saved decision logs its skip and never waits (#247, #90) The foreign path stored nothing for a ladder not_saved decision, but logged it as "No certain prompt" and, when only the legacy answer was a switched-off node's text, kept the file waiting for a seal that could only store nothing. Such a decision now ends the wait at settle, and both settle and the re-check log the opted-out path's own line (a wired switch says so). Tests cover an only-PromptManager node switched off (opted out, not foreign), a switched-off PromptManagerText behind a pass-through into a core CLIPTextEncode, a wired switch, a negative-only switched-off node, recorded history holding the off text, and foreign saving off. * fix(monitor): an undecided not_saved waits for its run's record (#247, #90) A not_saved from the ladder's undecided step is not final before a record exists: the recorded saver may still resolve another branch. The settle lookup now keeps such a file waiting; with the record it stores nothing and logs the opted-out line. An early not_saved (a switched-off node upstream of any sampler's positive) never reaches the lookup and stays final. Tests: the undecided case (literal and wired switch) waits, then skips; a switched-off node feeding one branch of a shared-prefix run never waits and stores nothing, not even the branch the recorded saver names. * docs(system): renumber the foreign-runs switch deviation to D158 (#247) #430 took D147 on main; the Save prompts from other workflows row is D158. * test(monitor): a switched-off node in the recorded saver's branch is never foreign (#247, #90) PR 3's repair skips a file whose recorded saver's branch reads a switched-off PromptManager node. On the monitor the same shape never reaches the foreign path: a literal switch is the opted-out path's and a wired one the node's own run's. Nothing is stored at settle or after the run, even from a recorded run holding the off text. |
||
|
Vito
|
55fa2c4870 |
fix(runs): count cached PromptManager nodes fed by a runtime text link on every job (#347) (#423)
* fix(usage): count cached re-runs of runtime-linked prompt text (#347) A PromptManager-family node whose text comes from a runtime link (PreviewAny, an LLM, a list node) counted only the job that executed it; later jobs served from ComfyUI's output cache counted nothing. Each execution now records a receipt (digests of its STRING outputs and hashes of the text it stores) per run key and queue item. On execution_cached the receipt whose digests equal the cache entry's text is replayed once for the job, with the node's role and what the queue hook already saw recomputed from the job's own graph. The executing node and the replay count through one shared own_counts. Unmatched cache entries count nothing, as before; a node that executes after its replay in the same job skips its own count. No schema change, no backfill. * fix(usage): never replay a node twice in one job; dedupe receipts once complete (#347) A repeated execution_cached id or message for the same job now counts the node once. Receipts replace equal older ones only when their job finishes (or the next job starts), so a list still recording [A] of [A, B] no longer drops an older [A] receipt. Node-level tests check the receipt each execution records. * test(comfyui): count #347's cached-run cases in the execution suite pin (#347) The pin listed 19 execution-text-link cases per project; #347 adds five node kinds and eight one-renderer cases, so the suite lists 32 per project and 118 in total with #332's image-attribution case from main. |
||
|
Vito
|
1bb3678f32 |
fix(threads): describe the teardown as built, warn only for a live observer, and keep an interrupted teardown's result (#377) (#378)
- _stop_background_work pre-fills the started parts with False and stores the result and removes its atexit entry in a finally:, so a Ctrl+C between steps no longer leaves the result unset and the callback registered. - The "watchdog observer still running" warning moves into _stop_observer: only for an observer still alive after its join, naming the join's own timeout (not after observer.stop() raised, and not the full deadline). - Tests: the plain-teardown tests share one package; two teardown tests run with a live preview worker held in a fake encode (ffmpeg killed, step gets min(CLOSE_WAIT_SECONDS, time left), thumbnails False while held and True when the kill releases it); an interrupt-between-steps test; two warning tests. - TESTING.md and the ledger fixture comment: the refusal comes first, the image worker and the observer share one 5 s deadline, and what an interrupt between steps does. Closes #377 |
||
|
Vito
|
b1e7abd380 |
fix(threads): stop and join the prompt cleanup thread, the image worker and the observer (#302, #370) (#372)
* fix(threads): stop and join the prompt cleanup thread, the image worker and the observer (#302, #370) - PromptTracker: the cleanup thread waits on a stop event instead of time.sleep; new shutdown(timeout=2) joins it and returns whether it ended. - ImageGenerationHandler.shutdown(timeout=5) now joins the worker and returns a bool; new close() is final: schedule() refuses and a sealed run's history re-check is dropped, so no worker restarts afterwards. - ImageMonitor.stop_monitoring(timeout=5) closes the handler first, then stops and joins the watchdog observer within one deadline. - Every test that builds a real tracker, handler or monitor ends with its final stop and an assertion, before its folder and databases go. - #370: the wrapper/session history test drains before dispose(); the held-write bound test uses 30 s ceilings instead of tight wall clocks. * test(history): drain the writer before asserting a bounded close it does not test (#370) * fix(threads): a later stop waits again for a worker or observer an earlier stop left running (#302) Review findings on #372: - ImageGenerationHandler keeps the workers a stop signalled that had not ended; every later shutdown/close joins them again, so a second call no longer returns True while the first worker still runs. - ImageMonitor.stop_monitoring keeps an observer or handler it left running and stops and joins it again on the next call. - A closed handler takes no awaiting history path, and a re-check that raced close() is not kept due. - Tests: second stops of a busy worker and a live observer, the closed handler's re-check paths, the handle test asserts the worker ended and stops before it asserts, and a no-new-thread check instead of set equality. * test(browser): the System reset restores every settings value, safety_backup included The frontend coverage shard of #372 failed in workspace-system-tagging: the review read "No safety backup (the policy is off)". The backups spec saves safety_backup=false, and /__test__/system/reset restored config.json but not PromptManagerConfig.SAFETY_BACKUP (nor INFINITE_SCROLL), so a later spec on the same worker server saw the policy off. The reset now captures and restores every non-tagging value POST /settings can change. |
||
|
Vito
|
3be0ada6a1 |
fix(ingestion): link an image only to a prompt that provably produced it (#344)
* fix(ingestion): link an image only to a prompt that provably produced it An LLM fed a PromptManagerPair's prepend_text while its text editor was empty. The node refused to save (an empty positive raised and logged a warning), and the image monitor then linked the image to a stale tracker entry or to the newest prompt in the database. - Nodes: an empty text editor saves and tracks nothing (prepend and append text are never the stored prompt), logs at info, and marks the node's run key empty so no earlier entry can be used for its images. - Tracker entries carry a run key: a digest of the prompt node and its whole upstream graph, the identity ComfyUI's cache uses. The monitor computes the same key from the image's embedded graph and uses a queued or snapshot entry only when the keys match (a snapshot also needs a known run with one entry, so a batch's last item is never applied to every image). - The recency fallback is gone; anything uncertain stays unlinked and is logged at info. - Recorded execution history attributes a file when exactly one run reported it, the file was written during that run, and the run's positive contexts name one stored prompt. The monitor and the output scan use it as their last certain source. - The scan and pair ingestion treat a resolved pair with an empty stored positive as unresolved instead of raising. - tools/review_image_links.py reviews selected links by these rules; dry run by default, --apply changes only named ids after a backup. Closes #342 * test(ingestion): real-host attribution case and the #342 testing contract The managed host disables the image monitor, so the owned fixture pack gains a route that starts and stops the plugin's monitor singleton for the attribution case only. The case runs in both renderers: a typed Pair image links, a Pair image whose positive is only linked prepend text stays unlinked with nothing stored, an unknown node feeding text links through its run key, and its cached re-run links again. * fix(ingestion): take a queued entry only from the run the image appeared in Dropping a key's older queue entries whenever the node ran again let a later queued run, executing during the monitor's processing delay, take the earlier image's entry. The snapshot taken when the image appears now names the key's latest run, and the queue is popped by key and run: same-key entries of earlier runs ahead of it are stale, entries of later runs wait for their own images. Also keeps the attribution case out of the stock real-host suite. * test(ingestion): host lookups degrade to no proof * fix(ingestion): address the #344 review (history re-check, keys, repair) - Images without an embedded API graph: the scan now asks recorded history before skipping a file with no prompt metadata, and the monitor gives a keyless unlinked image one history-only re-check when its run's outputs are recorded (a seal notification from the outputs write; at most 1000 waiting images, each for an hour). - Run keys: an image is unkeyable when any positive node has no key, so a two-pass graph is never linked through the other pass; links follow ComfyUI's is_link exactly. - A failed save marks the node's run key like an empty editor, so an older entry of that key can't stand in for this run. - A placeholder is the latest entry: the image stays unlinked without falling through to history, the same rule as a popped placeholder. - An index on execution_run_outputs(filename, subfolder, root_key, path_style) for the history lookup, added by migration and accepted by the output schema check (any other index is still refused). - Repair tool: unlink removes only the link record and only while the image is still linked to the prompt it was judged on; every run prints what --apply does. - Real-host case proves the re-run's PromptManager node was cached. * test(ingestion): real-host bare-output re-check and the review-round contract A metadata-disabled save, held open behind the fixture gate after SaveImage, waits unlinked while its run is open and links once the run's outputs are recorded. The fixture's monitor route lists the waiting files. TESTING.md covers the history re-check, the read index and what the repair tool's --apply does. * fix(ingestion): batch history lookups instead of adding an index An index on execution_run_outputs would make the library unopenable by older builds, whose schema check refuses any extra index there. The migration and the schema-check allowance are reverted, so the history schema is exactly main's. stored_run_prompts instead looks up a batch of files with one parameterized filename IN (...) query per chunk of at most 500 names and matches in memory; the scan computes it once per scan batch. * fix(ingestion): address the #344 re-review (seal race, linked roots) - A sealed history re-check that can't be scheduled yet (the image's own pass still pending, or the pending set full) is kept due and scheduled when a pending image finishes, instead of being dropped. - History matches a file through ComfyUI's spelling of its output folder first, then with both sides resolved, so a symlinked or junction output folder still attributes the scan's resolved paths; waiting files are keyed by their resolved path. - The scan makes its batch history lookup only when a file needs it. - set_current_prompt pushes to the queue last, so marking a failed execution never adds a second queue slot. - A keyless image outside the output folder doesn't take a waiting slot. |
||
|
Vito
|
48b1292115 |
feat(nodes): let a link drive the prompt text of every PromptManager node (#345)
* feat(nodes): let a link drive the prompt text of every PromptManager node The Focused, Split and Pair editors hide their widgets, and ComfyUI only draws a widget's input socket while the widget is visible. `text` (and the Pair's `negative_text`) therefore had no socket, and a STRING link dropped on the editor fell through to the first free STRING input, `prepend_text`, which is how #342's workflow was wired. - Detach those inputs from their hidden widgets on create and after a workflow load, so they are ordinary sockets in Classic and Nodes 2.0. The schema, widget order and saved values are unchanged. - A STRING link dropped on the editor body fills the main prompt, then the negative, before the frontend's own choice. - Each editor says when its text comes from a link; the typed text is kept for disconnect. The Classic nodes now save the typed text under a link instead of null, and "use prompt" no longer overwrites it while linked. - Real-host tests: mouse drags in both renderers, a workflow saved before this change, and executions proving the linked text is saved, counted, recorded in history and linked to the run's image. Closes #343 * test(coverage): floor for the linked-text module * test(nodes): PromptSearchList text takes a dragged link like a core widget * fix(nodes): linked text review round for #343 - A legacy frontend Primitive linked to an editor's text or the Pair's negative now runs with its value: the graphToPrompt wrapper and the queue hook fill it in (through virtual reroutes), in new and pre-#343 workflows. - Moving a linked text and releasing it on the editor keeps it on `text`: a link being moved counts as free, as the frontend counts it. - A link dropped on a prompt box (or its link note) fills that box's input, so the Pair's Negative prompt box takes `negative_text` (Classic through getInputOnPos, Nodes 2.0 through the box's slot key). - Only exact STRING links are steered; wildcards keep the frontend choice. - A linked Classic text box gives its row back and gets it back on disconnect; the PromptManagerText note uses the theme colour. - The metadata parsers map positional widget values through the widget table when an editor's widget-bound inputs do not line up with it. - Real-host cases for the legacy Primitive (new and saved workflows, a real Run), the moved link, the Negative box, paste/clone/undo and Split. * fix(nodes): resolve a legacy Primitive's text like the host, once per run (#343) - Inputs still bound to their widget (Classic PromptManager/PromptManagerText) are left as the host resolved them; the injection no longer overwrites them. - For the editors' detached text inputs, the Primitive's value goes through its "Run widget replace on values" replacement and the hidden widget's own serializer, so {a|b} wildcards and %date% tokens resolve as on main. - Each queued output is filled once (graphToPrompt wrapper, or the queue hook for an output that did not come through it): a wildcard is drawn once. - Editors inside subgraphs are found under their <subgraph>:<inner> id. - Real-host runPrompt installs its request route before the run starts. - Shared parser case pins the widget-table fallback for a core KSampler. * fix(nodes): pin the Classic wildcard behaviour and match queued ids by type (#343) - Classic PromptManager/PromptManagerText expose _pmTextSerializerReady, settled once their own text serializer is installed; the real-host specs wait on it and assert the Classic text is queued as written, as on main. - The Primitive fill only touches an output entry whose class_type matches the graph node at that id. - TESTING.md notes the host's harmless "Invalid widget" warning per Run. |
||
|
Vito
|
4ef236aaf7 |
feat(history): persist exact run output references (#214)
* feat(history): define bounded output reference contracts * feat(database): migrate execution output evidence to schema v3 * feat(history): persist immutable run output references * test(history): isolate mocked node gallery monitor * feat(history): capture bounded output descriptors with shared leases * fix(history): authenticate payload owners and release failed captures * feat(history): attach output evidence to exact finalized attempts * test(history): verify run output evidence on real ComfyUI * fix(history): release idle worker payload references * docs(history): clarify defensive lease cleanup * test(history): align output fixtures with native path styles * test(history): separate restore completion from writer drain * test(history): bound batch drains by progress and total time |
||
|
Vito
|
bae542c9ac |
feat(history): persist exact execution prompt contexts (#213)
* feat(history): define exact bounded execution contexts * fix(history): retain observable sources and validate evidence * feat(history): migrate execution contexts to schema v2 * feat(history): persist immutable execution context snapshots * fix(history): preserve package identity for execution contexts * feat(history): bound context payloads beside lifecycle finalization * fix(history): fence context commits and resolve stored run identity * fix(history): bound persisted identifier bytes before restore reads * feat(history): bind exact runtime context to execution attempts * test(history): discover owner invalidation callback regression * feat(history): capture cached prompt strings without mutating host caches * test(history): verify durable contexts on real ComfyUI hosts * fix(history): scope transforms to captured source paths * test(ci): update execution context tooling fixtures (#209) * fix(history): harden context validation and worker fixtures (#209) * fix(history): honor close deadlines after early joins (#209) |
||
|
Vito
|
763e6b37b1 |
feat(history): record durable ComfyUI execution attempts (#212)
* feat(history): normalize bounded execution evidence (#209) * fix(history): bound lifecycle kind checks before hashing (#209) * feat(history): persist execution attempts with restore fencing (#209) * test(api): bootstrap standalone backup restore fixtures (#209) * feat(history): isolate bounded asynchronous ledger writes (#209) * feat(history): observe actual queue attempts without altering execution (#209) * fix(database): honor package-scoped configuration (#211) * test(history): exercise real cached and headless executions (#209) * fix(history): reject unsupported uniqueness and isolate package fixtures (#209) * test(history): support Python 3.10 package fixture cleanup (#209) |