From 3f028c054b82130e2c28fd17a01664b17e97e852 Mon Sep 17 00:00:00 2001 From: Vito Sansevero Date: Sat, 24 Oct 2026 08:23:42 -0700 Subject: [PATCH] feat(system): document where the library lives and how it moves README: where the library lives, the upgrade path (Move library / Keep it here), PROMPT_MANAGER_DATA_DIR, explicit paths unchanged, and the way back to an older version. TESTING.md: D129 and the library location contract (resolution order, move steps, crash windows, coded refusals, Windows handling). --- README.md | 31 +++++++++++++++++++++- TESTING.md | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 104 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b60585ed..ed28d927 100644 --- a/README.md +++ b/README.md @@ -751,9 +751,34 @@ ComfyUI_PromptManager/ ## Configuration +### Where your library lives + +New installs keep the library (`prompts.db`, `config.json` with the CivitAI key, the +database backups and the logs) in ComfyUI's system user folder +`ComfyUI/user/__promptmanager/`. An update, a reinstall or ComfyUI-Manager replacing +the node folder no longer touches it, and ComfyUI's `/userdata` web API does not +serve `__` system folders. Thumbnails stay in `output/thumbnails`. + +- **Upgrading from an earlier version:** your library stays where it is, in the node + folder, until you move it; nothing is moved on its own. The workspace shows a notice + above every page and System › Overview a "Library location" row. **Move library** + copies the database with SQLite's backup API, verifies the copy (integrity check, every + workflow document, the same rows in every table), copies `config.json` and the backups, + switches to the new folder without a restart and only then renames the old files + `prompts.db.moved-to-__promptmanager` and `config.json.moved-to-__promptmanager` (never + deleted; the old backups stay too). If anything fails, the library stays in the node + folder, unchanged. **Keep it here** stops the notice; the Overview row stays. Take a + backup first (System › Backup & export or the dashboard's Backup) as with any change. +- **Your own folder:** set `PROMPT_MANAGER_DATA_DIR` to an absolute folder outside + ComfyUI's output, input and temp folders and your image sources. If it is empty and a + node-folder library exists, that library stays in use until you move it there. +- **An explicit `database.default_path`** in `config.json` and `PROMPT_MANAGER_CONFIG_PATH` + keep working as before (described below); such a library is not offered a move. + ### Database Settings -You can customize the database path by creating a `config.json` file in the extension root: +You can customize the database path by creating a `config.json` file in the library's +folder (the node folder for a library that was not moved): ```json { @@ -970,6 +995,10 @@ MIT License - see LICENSE file for details. ## Changelog +### Unreleased (Library location) + +- **📁 Library outside the node folder**: new installs keep `prompts.db`, `config.json`, the database backups and the logs in `ComfyUI/user/__promptmanager/`, where an update or reinstall of the node cannot delete them and ComfyUI's `/userdata` API does not reach. An existing library stays in the node folder until you press **Move library** (a verified copy, then a live switch; the old files are renamed `*.moved-to-__promptmanager`, never deleted) or **Keep it here**. `PROMPT_MANAGER_DATA_DIR` names your own folder. After moving, restart ComfyUI once so the logs follow. Going back to an older version after a move: an older version does not look in `user/__promptmanager`, so stop ComfyUI and copy `user/__promptmanager/prompts.db` (with its `-wal` file, if one is left) and `config.json` into the node folder; the `*.moved-to-__promptmanager` files are the library as it was at the move, without later changes. + ### Unreleased (Workflows stored once) - **🗜️ Smaller database**: each distinct workflow and prompt graph is stored once, compressed, instead of again for every image. Images recorded by this version use it from the first start; the maintenance job "Store each workflow once" rewrites the existing records and then reclaims the space. A 20,000-image test library went from 1.4 GB to 60 MB. The job always takes a verified backup first: the first time, `prompts.db.pre-workflow-storage_` beside the database, kept until you delete it; on later runs, a normal pooled safety backup. No image or video file is opened or changed. diff --git a/TESTING.md b/TESTING.md index 213d3ebe..fba44a24 100644 --- a/TESTING.md +++ b/TESTING.md @@ -2257,7 +2257,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged `-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no reader can keep it byte-identical (`test_health_check_never_writes`). -Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D128 assigned): +Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D113; D121–D129 assigned): - **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server @@ -2404,6 +2404,20 @@ Recorded deviations (plan D60–D67, D79–D81 renumbered D92–D99 and D111–D `Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save refused for one unusable source names it and marks its row` (workspace-system-sources.spec) and the system-state and system-sources node tests. +- **D129** (library location, S6b): the concept shows neither where the library lives + nor a way to move it. System › Overview's "Library health" gains a "Library location" + `.operation` row after Backup status, in that row's own markup: " · ", + plus "You chose to keep it here." and any `PROMPT_MANAGER_DATA_DIR` problem, with + "Move library" for a node-folder library and "Details" (a `dl.meta-grid` dialog) + otherwise. While a node-folder library is offered (`location.offer`), the shell shows + the concept's `.notice` (the Sources notice's markup: `div > h2 + p`, then + `.row-actions` with "Keep it here" and the `ca-primary` "Move library") in + `#library-banner` above `main`, in main's own width and padding, with no bottom margin + (main's padding spaces it); `workspace.css` carries the `/* D129 */` rules. Move + library opens the shared review (From, To, Size, Moves with it, Old files). Pinned by + `system shell and overview match the concept` (the mapping inserts the row) and + `library notice matches the concept notice` with its mutation check + (system-concept.spec), and the D129 tests of workspace-system.spec. - **D124** (inspector): with "Load metadata on selection" off, the Images panel and the viewer's details show "Metadata loads when you ask for it." and a "Load metadata" button instead of reading the file. The concept has no such state, so no concept @@ -2775,6 +2789,65 @@ Restoring a deduped backup on a version before this one is not supported: that version's restore accepts the file and then reads rewritten images without their workflow (README changelog). +### Library location contract (#249 System S6b) + +`utils/library_location.py` is the one resolver of every data path: the database +(`database/operations.py` `_resolve_db_path`, diagnostics), config.json +(`PromptManagerConfig.get_config_path`, so the `ConfigStore`), the backups (beside the +database), the logs (`utils/logging_config.py`) and the LoRA image cache +(`py/lora_utils.py`). Order: a valid `PROMPT_MANAGER_DATA_DIR` (absolute, never inside +ComfyUI's output, input or temp folder, a public user folder, an extra gallery root or +a gallery source; an invalid one is ignored and reported as a problem code), used when +it holds a library or no node-folder library exists; an explicit +`database.default_path` other than `prompts.db` (relative to the node folder, as +before) and `PROMPT_MANAGER_CONFIG_PATH`, honoured as before; ComfyUI's +`user/__promptmanager` (`folder_paths.get_system_user_directory("promptmanager")`, or +`get_user_directory()/__promptmanager`) once its marker `library-location.json` says +`moved`; a node-folder `prompts.db` that was not moved; else `user/__promptmanager` +(new installs; outside ComfyUI the node folder). An empty environment folder never +replaces a node-folder library (`data-dir-empty`). Nothing is moved or copied at +startup (`tests/test_library_location.py`, `tests/test_library_location_wiring.py`). + +`POST /prompt_manager/system/library/move` (`py/api/library_location.py`, +`database/library_move.py`) starts "Move library": kind `library_move`, admission +`exclusive`, not cancellable. Refusals answer before any job: 400 `not-movable`, +`same-folder`, `target-invalid`; 409 `target-has-library`, `target-conflict`, `busy`; +507 `insufficient-space` (the free space of the target's volume, for the database, its +backups and config.json plus 16 MiB). The job copies with SQLite's backup API from one +read transaction (counts, document digest and `data_version` of that snapshot) into a +`.pm-move-*` temp file in the target folder (`journal_mode=DELETE`, fsync, 0600), +verifies it (`verify_database_file`, every workflow document, the same rows in every +table, the same documents), copies every `prompts.db.backup_*` and +`prompts.db.pre-workflow-storage_*` (times kept, size and SHA-256 checked), then, under +the old file's `BEGIN IMMEDIATE` and the config lock, rechecks `data_version` (a write +meanwhile copies again, up to three times, then `library-busy`), copies config.json +(0600, checked), writes the marker `pending`, gives every file its final name with +`os.replace` inside the target folder (a different drive is a copy, never a rename), +writes the marker `moved` and switches the process (`switch_process`: the resolver's +override, `PromptModel.repoint_instances` closes every connection of every model of the +old file and points it at the new one, the review and history observers are re-keyed, +the API's plans and health report are dropped). Only then are the old `prompts.db` and +`config.json` renamed `*.moved-to-` after a `wal_checkpoint(TRUNCATE)`; a file +still open elsewhere is left in place and named (`old-in-use`, `old-locked`); the old +backups stay. A failure before the marker says `moved` removes everything it wrote and +leaves the old library in use, untouched (`TestFailures`); a crash leaves the old +library in use (no or a `pending` marker; the next move replaces its own leftovers) or +the new one (`moved`) (`TestCrashWindows`). `GET /prompt_manager/system/library` and the +Overview's `location` give `{path, source, movable, keep, offer, target, problem, +size_bytes}` with paths relative to ComfyUI; `POST .../keep {keep}` persists +`library.keep_in_node_folder` through the `ConfigStore`. Errors are coded and name no +path (`tests/test_api_library_location.py`). The real-host `library-location.spec.mjs` +(execution suite, its own fresh host on an OS-chosen port, no configuration) proves a new +install creates `user/__promptmanager/prompts.db` and its logs, adds nothing but byte +code to the node folder, and that `/userdata` serves none of it. + +Windows: every connection to the old file is closed before its rename (the source +snapshot connection, every model's connections, the checkpoint connection); temp files +are closed before `os.replace`; renames stay inside one folder; folder identity compares +`(st_dev, st_ino)` and `normcase(realpath)`, so case variants and links of a served +folder are refused. The browser test server closes its own log handlers before its temp +root (which now holds the logs) is removed. + ## Workspace data fields for the concept pages (#249) These fields are additive. The legacy `/prompt_manager/images/output` listing (no