* feat(maintenance): reviewed, read-only maintenance plans
POST /prompt_manager/maintenance/preview reviews duplicate_prompts,
optimize, unused_components or missing_links on its own mode=ro snapshot
and answers a Plan (scope, protected reasons, backup policy, media, at most
50 detail rows). The fingerprint, targets and judged sources stay in a
server-side PlanStore under a single-use token (15 min, 8 plans, 16 MiB,
never truncated; a restart forgets them), bound to the database it read.
The previews reuse the legacy queries: duplicate groups and the merge are
shared with cleanup_duplicates, ORPHANED_PROMPTS_SQL is built from the
#239 protection predicates the plan reports, and missing links are judged
by the health check's link_states. Legacy /maintenance keeps its shape;
vacuum now reports success:false when VACUUM fails, and
cleanup_orphaned_images skips links under unavailable sources.
* feat(maintenance): apply reviewed plans as Activity jobs
POST /prompt_manager/maintenance/apply {token} starts a maintenance job
(records admission, not cancellable, groups changed/protected/errors).
The token is consumed on the loop turn the job starts: a second apply is
409 plan-used naming the job, a busy refusal keeps the token, an unknown
or expired one is 410 plan-expired, and a restore since the review is
409 storage-changed.
With the safety backup policy on (maintenance.safety_backup, default true,
saved through the settings route with the revision), the job first writes
a backup into a temp file beside the database, makes it a standalone
rollback-journal file, verifies it, fsyncs it and renames it onto a unique
<db>.backup_<YYYYmmdd_HHMMSS_ffffff>[_n] name; any failure leaves no file
and nothing changed. The apply then rechecks the reviewed fingerprint
inside BEGIN IMMEDIATE (plan-changed writes nothing), deletes in batches of
400 ids, counts changed and protected records, logs the backup by name
only and prunes old safety backups. With the policy off the log and the
status line say "No safety backup".
* feat(maintenance): find and remove duplicate files through Activity
Duplicate image files move onto the job registry:
- POST /prompt_manager/jobs/duplicates finds content-identical media
files over every effective source as a cancellable read job
("Find duplicate image files", group groups_found); the server holds
the groups per job and GET /prompt_manager/maintenance/duplicates/{id}
pages them ten at a time.
- The duplicate_files preview binds a removal to the reviewed keepers.
Every keeper and removal is checked again: containment as text before
any filesystem access, then not a link, still inside its source, same
size and hash. The plan lists the database links that move.
- Apply runs duplicate_cleanup (records, not cancellable): the promised
safety backup, then per file a recheck, the unlink with every handle
closed and its thumbnail. A changed file is kept, the first failed
unlink stops the job with a coded message (never the OS text), and only
removed files have their image links and favorites moved to the kept
copy.
- The legacy scan, stream, status and delete routes keep their shapes on
the same registry kinds; the hashing, grouping and thumbnail removal
are shared with them.
Coordinator rulings on Tasks 15-16:
- Safety backups share the restore pool of 10 `<db>.backup_*` files; the
pool is now also pruned when an apply fails after its backup.
- Plans report the safety backup's size and the free disk space, and
apply refuses up front (507 insufficient-space) when the backup or
VACUUM does not fit, so no job fails halfway for space.
* fix(maintenance): never remove the kept copy; move records before unlink
Final review of #313:
- C1: the duplicate find tells files apart by (st_dev, st_ino), so a
source listed twice, a symlinked source inside another and a hardlink
are one file. The preview and the job refuse a removal that
os.path.samestat matches with its kept copy.
- I1: each file's records move to its verified kept copy in one short
transaction before that file is unlinked; a failed move deletes
nothing and a failed unlink leaves the records on the identical kept
copy. The status line counts files removed and not removed.
- I2: links are looked up under every spelling of the source (the real
path, the configured and monitored folders, ComfyUI's output).
- I5: image tags join the kept copy as their union and the removed
copy's pending, undecided Needs review items are settled; the plan
says so.
- I3: the VACUUM disk check measures SQLite's real temporary folder.
- M1: the token is looked at again with no await before the job starts.
- M2: the backup decision is made once and handed to the job.
- M3: backups are linked into place; startup sweeps interrupted temps.
- M4: Optimize runs alone (new "exclusive" admission) and its plan says
the database is locked while VACUUM runs.
- M5: the reviewed duplicate merge is strict: collisions and Library
titles are protected, and errors roll back. The legacy cleanup is
unchanged.
- M7: 2,000 keep entries within a 1 MiB body. M8: event waits in tests.
- M9: the preview binds files by size, mtime and inode without hashing;
the job hashes right before each unlink.
* fix(maintenance): protect one file reached through two mounts
Re-review of #313, round 2:
- R1: a removal on another device than its kept copy, with the same
size and mtimes within 2 s, is protected when its inode is equal or its
path within its source is the same ("May be the same file reached
through another mount or share"), in the preview and again right before
each unlink. The job holds the kept copy open during the unlink, so on
Windows deleting the kept copy itself through another path fails.
- R2: a file's spellings are found by containment: every configured,
monitored, ComfyUI-output or listed folder whose stat identity is a
folder above the file. Parent spellings of nested sources and case
variants match; favorites move under every listed root holding the
file. Lookups stay index searches.
- m1: a failed unlink's row says the links now name the kept copy.
- m2: a volume with no file ids reads "Can't confirm this is a separate
file on this drive".
* fix(maintenance): stat duplicate candidates with os.stat
Python 3.10's Path.stat uses the os.stat bound when pathlib was imported,
so the find and the per-file checks could see different stat calls. The
find now calls os.stat directly, like every other check; the mount tests
failed on py3.10 only.
* fix(maintenance): recheck the kept copy right before each unlink
Re-review 2 of #313:
- n1: the cross-mount guard also matches when one path within its source
ends with the other on a whole folder name (sources listed at different
depths, e.g. 2024/x.png and x.png on a FUSE or noserverino mount).
- n2: after a file's records move and right before its unlink, the held
kept copy must still be the file at its path (fstat of the hold against
os.stat) and the cross-mount guard must still hold; otherwise the file
is kept, its records move back to it, and it counts as not removed.
* test(maintenance): the kept-copy race tests hold on Windows
On Windows the removal job holds the kept copy open without
FILE_SHARE_DELETE, so the tests' attempt to delete or replace it during
the record move is refused. The tests now assert that native outcome:
the kept copy stays and the removal goes ahead.
* feat(database): store each workflow and prompt graph once
generated_images.workflow_data and prompt_metadata repeated the same large
JSON text for every image of a run. Each distinct text now lives once in
workflow_documents, keyed by the sha256 of its exact UTF-8 bytes and
zlib-compressed; image rows reference it through workflow_document_id and
prompt_document_id.
- New writes (link_image_to_prompt, pair ingestion for scan, monitor and
Needs review) store the new form. Old rows keep their inline text and
read the same until the migration rewrites them.
- Every reader goes through one accessor (read_stored_text,
resolve_image_row): gallery and legacy image lists, image by id, prompt
info for the inspector and metadata page, diagnostics. Payloads keep their
shape. Inline text wins when present; a document that fails its length or
hash check reads as unreadable, never as other text.
- No trigger (restore validators refuse them): a document goes in the same
transaction as its last reference (image and prompt deletes, missing-link
cleanup, duplicate merges, duplicate-file link merges, ingestion updates),
and a sweep at startup catches anything a path missed.
- verify_database_file validates the table, the id columns, their foreign
keys and indexes, and the stored values.
* feat(database): migrate old workflow rows in batches, then reclaim space
The workflow storage migration rewrites old-form image rows (text inline)
to document ids, batch by batch:
- Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or
16 MB of text, found by a rowid range search and read one row at a time.
- Each text is stored or reused, then read back through the one accessor
before its row changes; a text that would not come back identical fails
and rolls back the whole batch.
- A value that is not text stays inline and is counted as protected.
- A crash or a stop leaves committed batches new form and the rest old
form; both read the same, and a rerun continues. No media file is opened.
- Unreferenced documents are swept at the end.
The maintenance preview gains a read-only workflow_storage plan (records to
rewrite, inline bytes where SQLite has octet_length, the Reclaim space and
older-version notes). reclaim_space sweeps, runs VACUUM and truncates the
WAL so the freed pages go back to the disk.
* feat(maintenance): store each workflow once as an Activity job
The reviewed maintenance plan gains workflow_storage. Applying it starts
"Store each workflow once": kind maintenance, records admission,
cancellable, groups changed/protected/errors.
- The promised verified safety backup comes first; a failed backup ends the
job with nothing rewritten. The apply's up-front disk check covers it.
- Old-form rows are rewritten batch by batch on the event loop's executor,
with progress after each committed batch. Cancel stops between batches;
a failed batch rolls back whole and fails the job with a coded message.
Either way every record reads as before and the next apply continues.
- When every row was looked at, "Reclaim space" starts: exclusive (runs
alone), not cancellable. It checks the disk for VACUUM's copies of the
live pages, then sweeps, runs VACUUM and truncates the WAL. If another job
holds the registry it does not start, and the finished migration says so
and names Optimize database.
- No image or video file is opened; tests track every open under the
output folder and compare the media bytes and mtimes.
* feat(maintenance): document the workflow storage contract
TESTING.md's maintenance contract gains the workflow storage rules: the
documents table and its two forms, the one accessor, garbage in the same
transaction plus the sweeps, the batched migration and Reclaim space, the
media and memory guarantees, and that a deduped backup cannot be restored
into an older version.
A test covers the preview on SQLite before 3.43 (no octet_length, as in
older Windows Python builds): the inline-bytes detail is left out.
* feat(database): keep the migration safe row by row and beside other writers
Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe:
- M1: a batch also ends after 0.5 s, and the loop leaves the write lock
free for 0.15 s after each committed batch (longer than SQLite's longest
busy poll), so the image monitor and run counting on another connection
get in between. RED: a threaded writer with a 2 s busy timeout hit
"database is locked" (4 of 5 links landed).
- M2: each row runs in its own savepoint. A row that cannot be rewritten
(undecodable text, a text that would not come back) is rolled back alone,
stays inline and is listed in errors; the rest of the batch goes on.
- M3: a reused document is decompressed and checked before its id is handed
out, and repaired from the text (whose hash is its key) when damaged;
verify_database_file reads every document and refuses a damaged one.
- M5: the two legacy generated_images rebuilds refuse a file that already
has the document id columns instead of dropping them.
- The preview names the pre-migration backup and says it is always taken.
* feat(maintenance): always back up before storing workflows once
Final review fixes I1, I2 and I3 for the workflow storage dedupe:
- I1: the migration always takes a verified backup, whatever
maintenance.safety_backup says; the plan reports backup.policy
"required" and the disk check counts it.
- I2: that backup is prompts.db.pre-workflow-storage_<stamp>, outside the
pruned prompts.db.backup_* pool and kept until the user removes it; the
job's last line and the preview name it, and the legacy restore accepts
it. README's changelog warns that an older version opens and restores a
rewritten database but shows no workflows, and says how to go back.
- I3: tools/workflow_text_digests.py writes the sha256 of every record's
stored texts (read-only; inline before the rewrite, through the accessor
after), so a diff before and after proves byte identity.
- A row the migration could not rewrite is listed in the job's errors and
named in its last line, and Reclaim space still runs.
* feat(database): fail the batch on system errors and name a damaged document
Re-review fixes R1 and R6 (database side), plus R3's helpers:
- R1: a row is kept inline only for its own cause: text SQLite cannot
decode, or a text that would not come back. A full disk, an I/O error, a
malformed database or a bug now fails the batch and rolls it back,
instead of listing every remaining row as "could not be rewritten".
- R6: verify_database_file names the damaged document and says what to do
(restore a backup from before the damage, or store an image with the same
workflow, which repairs it).
- R3: pre_migration_backups lists the kept backups, earliest first; the plan
says "Nothing to rewrite" and names an existing kept backup, and the older
version note says new images use the new form from the first start and
names the earliest backup as the way back.
* feat(maintenance): one kept pre-migration backup and nothing-to-do refusal
Re-review fixes R2, R3, R4, R5 and R6 (job side):
- R3: Apply refuses a plan with nothing to rewrite (409 nothing-to-do, no
job, no backup). Only the first apply keeps a pre-workflow-storage
backup; later applies (after a Cancel, a rerun) take a pooled safety
backup, forced on whatever the policy, so one copy is never pruned.
- R4: a second-connection writer test runs against the live job loop; with
its pause removed the writer hits "database is locked" (4 of 5 links).
- R5: the digest tool loads only database/workflow_documents.py by path (no
package start-up, no log handler) and opens a stopped database with
immutable=1, so it leaves no -wal/-shm; a live one is read through its WAL.
- R6: a backup that fails on a damaged document says which one in the job
line.
- R2: README's changelog says images recorded by this version use the new
form from the first start, and the way back is the earliest
pre-workflow-storage backup (or a pre-update backup if the job never ran).
Vito
06099579e6