b264dac2e8d1bea9ff7ef7cd6c61d3c19145dc98
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vito
|
ae3e191990 |
feat(workspace): show the one-time older-images notice in the shell (#529)
* feat(workspace): show the one-time older-images notice in the shell After the switch everyone lands on Library, so someone who has not dismissed the notice might never be prompted. It now shows above every workspace page until dismissed, sharing pm_update_notice_dismissed with System > Sources and the admin modal. Browser specs start with it dismissed. Closes #527 * test(workspace): turn the older-images notice on where specs expect it Two System specs read the Sources notice from empty storage, and the new spec read a job list earlier specs had filled. |
||
|
Vito
|
8870496ea3 |
feat(monitor): store certain prompts from workflows without a PromptManager node, with a System switch (#247, PR 2) (#467)
* feat(inspector): use recorded run history when the graph alone can't name the prompt (#386) When a file's own graph leaves the prompt open, both inspect routes ask recorded run history about the file (one run reported it and it was written during that run). The save node that run reported it from anchors the walk like a file-name match, so dataset branches that share one saver prefix show their own prompt, labelled "Recorded run". Still open at one sampler whose PromptManager text was recorded at execution (a language model's output), the recorded text shows as "Execution recorded". Results the file resolves alone never change; a failed lookup judges the file alone. * fix(inspector): recorded texts follow history's saver; conflicts anchor nothing (#386) Review follow-ups: the texts PromptManager recorded are looked up at the sampler history's saver leads to, never at a sampler the unanchored walk named; a file name naming a different saver than history anchors nothing; outputs and texts are read in one transaction; "Recorded run" ranks below every workflow-inferred source; unexpected lookup errors log at error level. * fix(history): look up recorded runs by history's own spelling of the file (#386) The history route inspects a case-normalized target on Windows, while history matches file names exactly, so the occurrence's reference names the file. * refactor(inspector): share the inspector's resolve as inspection_resolution inspect_prompt_evidence now calls inspection_resolution, which returns the resolution to use, the anchored result and whether the recorded saver chose it. Ingestion can call the same helper for files the default profile cannot decide (#247). * feat(ingestion): one per-file prompt decision for the scan and the monitor utils/file_prompt.py holds decide_file_prompt, the ladder the output scan runs today plus the inspector's rule (inspection profile, then the recorded saver) before the legacy parser, with a SCAN and a MONITOR policy. Junk text (blank, numbers, LoRA tags only, model file names) is never stored. graph_prompt_text and parse_comfyui_prompt move here; the parser accepts parsed metadata (#247). * feat(ingestion): store a file's prompt in one transaction, one prompt per file database/scanned_prompts.py links a file to the prompt of one text, creating it when new, inside one BEGIN IMMEDIATE transaction: a file linked to another prompt is a conflict, a failed link leaves no prompt behind and two writers store one prompt. pair_ingestion gains file_link_rows, link_conflict and Marks/mark_created; ingest_resolved_pair takes optional marks for the components it creates. recorded_runs and named_prompt in run_attribution become public (#247). * feat(scan): decide each scanned file with the shared ladder (#247) _ingest_scanned_file calls decide_file_prompt (scan policy) and stores the result: pairs through ingest_resolved_pair and positives through store_scanned_prompt, both with the scan marks. Files the default profile leaves undecided now get the inspector's resolve, with the saver recorded history names, before the legacy parser, so each branch of a shared-prefix dataset run links its own prompt. Junk text is never stored, a file linked to another prompt is a conflict instead of a second link, and a failed link leaves no prompt behind. _BatchHistory reads the stored prompt and the recorded saver from one recorded_runs pass. _parse_comfyui_prompt calls the moved parse_comfyui_prompt. TESTING.md documents the decision. * fix(ingestion): keep the file-name saver anchor out of the ladder (#390) The #390 ruling queues a file that only its saver's file name resolves for Needs review, never linking it. inspection_resolution gains name_anchor; the ladder passes False, so such a file falls to the legacy floor in the scan (and to Needs review) and is never stored by the monitor, while the recorded saver still decides. The inspector keeps the name anchor. * feat(config): gallery.monitoring.save_foreign_runs, default on (#247) * fix(ingestion): no joined PromptManager text from the legacy floor; junk keeps its step's status The legacy parser returns prepend + text + append, so its answer is refused when a PromptManager-family node has a prepend or append part (#343). Junk found at the undecided-graph step keeps that step's unlinked status and review, so an ambiguous file is still queued for Needs review. The monitor policy's ambiguous file no longer asks for the recorded run, and the file size may be read lazily. * fix(ingestion): keep the legacy answer that is a PromptManager node's own text The joined-text refusal also refused the #170 parity files whose PromptManager prepend is a link the browser parser does not follow: its answer is then the node's literal text, which is what the node stores. Refuse only an answer that is not such a node's own text. * feat(monitor): store the certain prompt of a foreign workflow's output (#247) A foreign output (no positive PromptManager node) is decided by the shared ladder with the MONITOR policy. Certain files are stored at settle with RUN_MARKS through survive_switch; files only recorded history can decide wait in their own bounded set and are re-checked once after the run's outputs are sealed, with the sealed reference. The setting GalleryConfig.SAVE_FOREIGN_RUNS is read for every file. * fix(scan): one prompt per file for recorded-run links; writes survive a library move The recorded run's stored prompt now links in one transaction with the file's link rows, so a file linked to another prompt is a conflict instead of a second link. The scan's stores run through survive_library_switch. A single file asks history once (a one-file _BatchHistory), the scan passes in_output=False and reads the file size only for files with prompt metadata. A conflict row reads "Kept: already linked to a different prompt". TESTING.md records the rules. * feat(settings): save_foreign_runs through POST /settings, applied live (#247) GET /settings reports settings.save_foreign_runs; a save takes a strict bool, writes gallery.monitoring.save_foreign_runs and publishes it to GalleryConfig in the commit, with no restart field. The diagnostics report lists it. The legacy contract snapshot gains the added key (an addition, no legacy key moves). * feat(system): Save prompts from other workflows in Sources & scans (#247, D147) A host setting row below Watch for new outputs drafts save_foreign_runs and saves with the one Save row; the server applies it at once, so no restart is named. While monitoring is drafted off the row is disabled and says it applies while Watch for new outputs is on. The concept comparison maps the inserted row and its mutation check covers it. * test(comfyui): a workflow of ComfyUI's own nodes creates and links its prompt (#247) The real-host attribution spec gains the foreign case (CLIPTextEncode, KSampler, SaveImage): its prompt is created and linked at settle and a second run links to the same prompt. The fixture route lists the waiting foreign files. TESTING.md records the monitor's foreign path and D147. * test(vendor): the execution matrix counts the #247 real-host case * fix(monitor): read a foreign file's size only when the ladder needs it (#247) Uses PR 1's lazy file size and pins that a foreign file linked to another prompt keeps its one link on the pair and the positive store paths. * fix(ingestion): match a file's link rows under its resolved spelling too The scan stores realpath spellings and the monitor the watcher's raw path, so a file reached through a symlinked or junction output folder could get a second generated_images row. file_link_rows now queries the path and its realpath (each normalized and raw) and dedupes by row id. TESTING.md notes the reverse order is matched only when the raw spelling is the realpath. * fix(monitor): wait for the run unless history answered; monitoring off stops foreign writes (#247) Review fixes: at settle a certain answer ends the wait only when the lookup found a record or the answer is the inspector's or the recorded saver's; the legacy floor waits for the re-check. A record without a certain prompt ends the wait at once. Foreign writes need both save_foreign_runs and monitoring_enabled, read again just before the store. Dropped foreign files are logged outside the lock (evictions as evictions, due ones too), and a drained re-check never drops a newer reference. * fix(system): the foreign row can be set while the monitor still watches; its note describes it (#247, D147) Review fixes: while monitoring is saved off but a source still reads watching until restart (D128), the row stays enabled with its note; the note is the switch's aria-describedby. The browser spec restores save_foreign_runs on the shared server. TESTING.md moves D147 into order and records the monitor's review rules. * test(comfyui): each branch of a shared-prefix foreign run links to its own prompt (#247 CUJ 1) * fix(ingestion): a switched-off PromptManager node behind a pass-through stores nothing (#90) The #90 graph walk follows only prompt-like inputs, so a PromptManagerText with Save to library off reaching the encoder through another input name (easy showAnything.anything, a custom node's value) was invisible to it, and the inspector's resolve stored its text. The ladder now checks the positive every step names: a resolved or inspected positive whose path passes a node that is off, or a legacy answer that is such a node's own text, makes the file not_saved. * fix(ingestion): any switched-off node upstream of a positive stores nothing (#90) The ladder walks every input upstream of each sampler or guider positive (not negative, model, clip, latent or image inputs) for PromptManager nodes whose Save to library is off or wired from an unreadable node, so the sampler-less step, an off node with a prepend part and a PromptManagerPair behind a pass-through no longer store its text. The legacy-floor guard reads a node's resolved text. Every guard reports not_saved for a switch that is off and not_saved_wired for one it cannot read. * fix(ingestion): the switch walk skips negative inputs by #90's own rule The upstream walk treated only names containing "negative" as negative, so an off node on a neg, neg_text or text_neg input blocked an enabled positive. prompt_graph gains is_negative_input (the _NEGATIVE_KEY rule, also used by follows_input) and the walk uses it. The guards accept a node with null inputs and a side whose provenance is not an object. * fix(ingestion): the switch walk also skips any input named with "negative" #90's rule matches only whole underscore-separated words, so negative2, negativeprompt and NegativePrompt counted as positive inputs and an off node wired only to one of them blocked the file. The walk skips those too; follows_input is unchanged. * fix(monitor): a foreign not_saved decision logs its skip and never waits (#247, #90) The foreign path stored nothing for a ladder not_saved decision, but logged it as "No certain prompt" and, when only the legacy answer was a switched-off node's text, kept the file waiting for a seal that could only store nothing. Such a decision now ends the wait at settle, and both settle and the re-check log the opted-out path's own line (a wired switch says so). Tests cover an only-PromptManager node switched off (opted out, not foreign), a switched-off PromptManagerText behind a pass-through into a core CLIPTextEncode, a wired switch, a negative-only switched-off node, recorded history holding the off text, and foreign saving off. * fix(monitor): an undecided not_saved waits for its run's record (#247, #90) A not_saved from the ladder's undecided step is not final before a record exists: the recorded saver may still resolve another branch. The settle lookup now keeps such a file waiting; with the record it stores nothing and logs the opted-out line. An early not_saved (a switched-off node upstream of any sampler's positive) never reaches the lookup and stays final. Tests: the undecided case (literal and wired switch) waits, then skips; a switched-off node feeding one branch of a shared-prefix run never waits and stores nothing, not even the branch the recorded saver names. * docs(system): renumber the foreign-runs switch deviation to D158 (#247) #430 took D147 on main; the Save prompts from other workflows row is D158. * test(monitor): a switched-off node in the recorded saver's branch is never foreign (#247, #90) PR 3's repair skips a file whose recorded saver's branch reads a switched-off PromptManager node. On the monitor the same shape never reaches the foreign path: a literal switch is the opted-out path's and a wired one the node's own run's. Nothing is stored at settle or after the run, even from a recorded run holding the off text. |
||
|
Vito
|
8bf759d6f8 |
feat(system): keep the library in ComfyUI's user folder (#318)
* feat(system): resolve every library path in one place utils/library_location.py decides where the library lives: the PROMPT_MANAGER_DATA_DIR folder (validated: absolute, never inside a folder ComfyUI or Prompt Manager serves), an explicit database.default_path as before, a node-folder library that was not moved, else ComfyUI's user/__promptmanager. The database, diagnostics, config.json, logs and the LoRA image cache all ask it, so a new install never writes into the node folder and an existing one keeps every path until the user moves it. "Keep it here" is persisted as library.keep_in_node_folder. * feat(system): move the library as a verified Activity job POST /prompt_manager/system/library/move starts "Move library", an exclusive, non-cancellable job (database/library_move.py). It checks the target volume's free space, copies the database with SQLite's backup API from one read snapshot, verifies the copy (integrity check, every workflow document, the same rows in every table), copies the safety and pre-migration backups and config.json (owner-only, checked by size and SHA-256), then publishes under the old file's write lock and switches the live process (every PromptModel, the review and history observers, the API's plans and health report). The old prompts.db and config.json are renamed *.moved-to-<folder> only after the switch. A failure removes everything it wrote and leaves the old library in use; a marker in the new folder makes a restart after any crash pick the right library. GET /prompt_manager/system/library and the Overview's "location" say where the library lives and why; POST .../keep persists "Keep it here". * feat(system): show where the library lives and offer the move D129. System > Overview's Library health gains a "Library location" row (the database path, why it is there, the "Keep it here" choice and any PROMPT_MANAGER_DATA_DIR problem) with Move library for a node-folder library and Details otherwise. While a node-folder library is not kept there, the workspace shows the concept's .notice above every page: "Keep it here" saves the choice on the server and stops the notice; "Move library" opens the shared review (From, To, Size, what moves, what happens to the old files) and starts the exclusive job in Activity. The notice and the Overview read the location again when the job ends. The Overview comparison maps the inserted row; a new comparison holds the notice to the concept's Sources notice (markup, tokens, alignment, both themes and widths) with a mutation check. * feat(system): close the browser server's log files before its root goes The logs now live in the library's data folder, which for the browser test server is <root>/user/__promptmanager; Windows could not remove the temp root while the server still held prompt_manager.log open. The API test's node folder is spelled in lower case because public paths are case-folded on Windows. * feat(system): show the library location on the legacy admin page A line under the admin page's statistics names the database path and why it is there (ComfyUI's user folder, PROMPT_MANAGER_DATA_DIR, config.json, or the node folder with a pointer to Move library), read once from GET /prompt_manager/system/library. * feat(system): prove a fresh host keeps its library in user/__promptmanager The execution suite starts one pinned CPU ComfyUI with no configuration at all on an OS-chosen port: the library is created in user/__promptmanager (database and logs), nothing but byte code is added to the node folder, the location route reports the default folder, and ComfyUI's /userdata API serves none of it (traversal spellings, a listing, a system-user header). * feat(system): document where the library lives and how it moves README: where the library lives, the upgrade path (Move library / Keep it here), PROMPT_MANAGER_DATA_DIR, explicit paths unchanged, and the way back to an older version. TESTING.md: D129 and the library location contract (resolution order, move steps, crash windows, coded refusals, Windows handling). * wip(system): library move fix round in progress * fix(system): retry a fallback image link that races the library switch The image monitor's fallback and recent-prompt links went through PromptDatabase.link_image_to_prompt, which logs and returns 0 on any error, so survive_switch never saw a "database is locked" or closed connection during Move library and the link was lost in both files. The monitor now links through link_image_to_prompt_or_raise (one transaction per attempt); link_image_to_prompt keeps its log-and-return-0 contract for other callers. A link that returns 0 is no longer logged as "Successfully linked image 0". A switch that fails after repointing the models now closes the old file's detached connections instead of leaving them to GC: switch() adds them to a list the mover owns. The mocked retry test is replaced by a real-lock race asserting the image row lands in the new library only. * fix(system): say which source is the default and where a symlinked library lives Sources & scans: with no source configured, ComfyUI's output folder now reads "ComfyUI/output · watching · default" on its own row instead of a separate note line. GET /settings marks each effective source with in_comfyui (its public path is relative to the ComfyUI base) so only those get the "ComfyUI/" prefix; the scan still sends the server's public path. Library location: a library in the node folder reads through the node folder's unresolved place in custom_nodes, so a node folder symlinked to a checkout outside ComfyUI shows custom_nodes/promptmanager/prompts.db instead of prompts.db (System library row, Move review, admin Library line). Never absolute or "..", else the shared _public_path, which sources and jobs keep using unchanged. |
||
|
Vito
|
69de20a427 |
feat(system): Sources & scans (#309)
* feat(system): report configured, effective and watched sources
GET /prompt_manager/settings adds settings.monitoring_enabled, a sources
block (configured roots with valid/code/reason, effective roots with their
listing index and watch state, auto_detected) and the monitor state
(present, observer_alive, scheduled). Every path is a public path.
A source is "watching" only while a live observer scheduled it, "manual"
when monitoring is off or the process has no monitor, and otherwise
"restart_pending".
POST accepts monitoring_enabled as a strict bool, saved under
gallery.monitoring.enabled; restart_required_fields names it when the
running observer does not already match. A save that changes the sources
runs as a "Source change" records hold, so it is refused with 409 busy
while a scan runs. Case variants of one folder are saved once.
POST /prompt_manager/settings/validate-source {path} checks a path
lexically (no "..", no control characters, native path forms only) and
then against the gallery-root rules, answering a code and reason, and
writes nothing.
* fix(system): no existence oracle in source validation
A gallery root's containment in the allowed ComfyUI folders is now
decided before its existence and type, so a path outside them answers
"outside-allowed" whether or not it exists. validate-source also folds
the filesystem-root and home-directory codes into "outside-allowed", with
one reason, and logs the detail server-side only. Paths inside the
allowed folders still say "not-found" or "not-directory". The legacy
settings save shares the same check.
* feat(scan): scan one selected source
POST /prompt_manager/jobs/scan accepts roots: indexes into the effective
sources, strict non-negative ints (no bools or floats), deduplicated, at
most 64, non-empty when present. An index past the sources is a 400
"unknown-source" and starts nothing. The job walks only the selected
sources, resolved when it was requested, and keeps each file's listing
root_index. Its scope records files, roots (null for every source) and
output_folders, the number of folders walked, which the log's first line
counts. The legacy /scan route still scans every source, and attaching to
a scoped job keeps its payload keys.
* feat(system): manage sources and start scans natively
System > Sources & scans renders the concept's section on the server's
sources: one row per effective source with its watch state and a scan
scoped to that source, the 3.2.0 update notice with the legacy dismissal,
Add source checked by validate-source, Remove, monitoring and worker
threads as host drafts behind the one Save row, and Refresh files.
A save refused by a running scan names the job. Thumbnail generation
hands off to the legacy gallery until S5.
Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md.
* fix(security): decide source containment as text before any filesystem access
Every caller (validate-source, POST /settings, config.json at load) now
refuses a source through one check, sources.source_problem: the form
check first, then GalleryConfig.gallery_root_problem, which compares the
normalised spelling with the allowed folders before anything resolves,
stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a
device or any outside folder is refused with no SMB request, no
executor stall and no existence or timing oracle; only then is the path
resolved and contained again, which still catches a symlink out. The
filesystem root and the home directory answer outside-allowed with the
same reason as any outside path, so no caller learns the home path.
Also:
- POST /settings runs the form checks and parses the roots off the loop;
an unusable source is a coded 400 invalid-source naming its public
path (as text) and its index.
- ComfyUI folders outside the base directory are path anchors, so their
sources round-trip and never collide by folder name.
- A per-source scan sends each index with its public path; a stale or
shifted index is 409 sources-changed and starts nothing.
- A source still watched after monitoring was switched off is
"stopping".
- Roots are compared as folders, not spellings, so an unchanged legacy
resend takes no source-change hold.
- Tests: no realpath/stat/lstat of an outside path (native and Windows
forms with ntpath fakes), the alike answers with /, ~ and a link to ~,
an event instead of a timer wait, and a teardown that drains jobs
before the temporary database goes.
* fix(system): say what Sources saved, refused and still watches
- A per-source scan sends the source's public path; when the sources
changed since they were read the toast says the server's reason and
the list and saved values are read again.
- "watching until restart" for a source the running monitor still
watches after monitoring was saved off, and the status adds "Restart
ComfyUI to apply: Watch for new outputs." from restart_required_fields.
- A save refused for one unusable source marks that row and the status
is the server's message naming it.
- A valid configured source the effective list holds under another
spelling reads "same folder as another source".
- The busy toast names the job the 409 named (ApiError keeps a
validated running job and refused source), falling back to the list.
- The Add source dialog's listeners live on the page scope and Enter
adds; its markup is compared with the concept (D127).
- TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the
Sources contract describes the lexical containment, invalid-source and
sources-changed.
* fix(security): guard the Windows form probes on a token of their own
The no-filesystem guard matched "C:", which on a Windows runner is also
the temporary ComfyUI tree the save may resolve for its trusted first
entry. The probe paths now carry a token no trusted directory holds.
* docs(system): number the Sources states deviation D128
|