Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe:
- M1: a batch also ends after 0.5 s, and the loop leaves the write lock
free for 0.15 s after each committed batch (longer than SQLite's longest
busy poll), so the image monitor and run counting on another connection
get in between. RED: a threaded writer with a 2 s busy timeout hit
"database is locked" (4 of 5 links landed).
- M2: each row runs in its own savepoint. A row that cannot be rewritten
(undecodable text, a text that would not come back) is rolled back alone,
stays inline and is listed in errors; the rest of the batch goes on.
- M3: a reused document is decompressed and checked before its id is handed
out, and repaired from the text (whose hash is its key) when damaged;
verify_database_file reads every document and refuses a damaged one.
- M5: the two legacy generated_images rebuilds refuse a file that already
has the document id columns instead of dropping them.
- The preview names the pre-migration backup and says it is always taken.
The reviewed maintenance plan gains workflow_storage. Applying it starts
"Store each workflow once": kind maintenance, records admission,
cancellable, groups changed/protected/errors.
- The promised verified safety backup comes first; a failed backup ends the
job with nothing rewritten. The apply's up-front disk check covers it.
- Old-form rows are rewritten batch by batch on the event loop's executor,
with progress after each committed batch. Cancel stops between batches;
a failed batch rolls back whole and fails the job with a coded message.
Either way every record reads as before and the next apply continues.
- When every row was looked at, "Reclaim space" starts: exclusive (runs
alone), not cancellable. It checks the disk for VACUUM's copies of the
live pages, then sweeps, runs VACUUM and truncates the WAL. If another job
holds the registry it does not start, and the finished migration says so
and names Optimize database.
- No image or video file is opened; tests track every open under the
output folder and compare the media bytes and mtimes.
generated_images.workflow_data and prompt_metadata repeated the same large
JSON text for every image of a run. Each distinct text now lives once in
workflow_documents, keyed by the sha256 of its exact UTF-8 bytes and
zlib-compressed; image rows reference it through workflow_document_id and
prompt_document_id.
- New writes (link_image_to_prompt, pair ingestion for scan, monitor and
Needs review) store the new form. Old rows keep their inline text and
read the same until the migration rewrites them.
- Every reader goes through one accessor (read_stored_text,
resolve_image_row): gallery and legacy image lists, image by id, prompt
info for the inspector and metadata page, diagnostics. Payloads keep their
shape. Inline text wins when present; a document that fails its length or
hash check reads as unreadable, never as other text.
- No trigger (restore validators refuse them): a document goes in the same
transaction as its last reference (image and prompt deletes, missing-link
cleanup, duplicate merges, duplicate-file link merges, ingestion updates),
and a sweep at startup catches anything a path missed.
- verify_database_file validates the table, the id columns, their foreign
keys and indexes, and the stored values.
Vito Sansevero