Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe:
- M1: a batch also ends after 0.5 s, and the loop leaves the write lock
free for 0.15 s after each committed batch (longer than SQLite's longest
busy poll), so the image monitor and run counting on another connection
get in between. RED: a threaded writer with a 2 s busy timeout hit
"database is locked" (4 of 5 links landed).
- M2: each row runs in its own savepoint. A row that cannot be rewritten
(undecodable text, a text that would not come back) is rolled back alone,
stays inline and is listed in errors; the rest of the batch goes on.
- M3: a reused document is decompressed and checked before its id is handed
out, and repaired from the text (whose hash is its key) when damaged;
verify_database_file reads every document and refuses a damaged one.
- M5: the two legacy generated_images rebuilds refuse a file that already
has the document id columns instead of dropping them.
- The preview names the pre-migration backup and says it is always taken.
The reviewed maintenance plan gains workflow_storage. Applying it starts
"Store each workflow once": kind maintenance, records admission,
cancellable, groups changed/protected/errors.
- The promised verified safety backup comes first; a failed backup ends the
job with nothing rewritten. The apply's up-front disk check covers it.
- Old-form rows are rewritten batch by batch on the event loop's executor,
with progress after each committed batch. Cancel stops between batches;
a failed batch rolls back whole and fails the job with a coded message.
Either way every record reads as before and the next apply continues.
- When every row was looked at, "Reclaim space" starts: exclusive (runs
alone), not cancellable. It checks the disk for VACUUM's copies of the
live pages, then sweeps, runs VACUUM and truncates the WAL. If another job
holds the registry it does not start, and the finished migration says so
and names Optimize database.
- No image or video file is opened; tests track every open under the
output folder and compare the media bytes and mtimes.
The workflow storage migration rewrites old-form image rows (text inline)
to document ids, batch by batch:
- Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or
16 MB of text, found by a rowid range search and read one row at a time.
- Each text is stored or reused, then read back through the one accessor
before its row changes; a text that would not come back identical fails
and rolls back the whole batch.
- A value that is not text stays inline and is counted as protected.
- A crash or a stop leaves committed batches new form and the rest old
form; both read the same, and a rerun continues. No media file is opened.
- Unreferenced documents are swept at the end.
The maintenance preview gains a read-only workflow_storage plan (records to
rewrite, inline bytes where SQLite has octet_length, the Reclaim space and
older-version notes). reclaim_space sweeps, runs VACUUM and truncates the
WAL so the freed pages go back to the disk.
generated_images.workflow_data and prompt_metadata repeated the same large
JSON text for every image of a run. Each distinct text now lives once in
workflow_documents, keyed by the sha256 of its exact UTF-8 bytes and
zlib-compressed; image rows reference it through workflow_document_id and
prompt_document_id.
- New writes (link_image_to_prompt, pair ingestion for scan, monitor and
Needs review) store the new form. Old rows keep their inline text and
read the same until the migration rewrites them.
- Every reader goes through one accessor (read_stored_text,
resolve_image_row): gallery and legacy image lists, image by id, prompt
info for the inspector and metadata page, diagnostics. Payloads keep their
shape. Inline text wins when present; a document that fails its length or
hash check reads as unreadable, never as other text.
- No trigger (restore validators refuse them): a document goes in the same
transaction as its last reference (image and prompt deletes, missing-link
cleanup, duplicate merges, duplicate-file link merges, ingestion updates),
and a sweep at startup catches anything a path missed.
- verify_database_file validates the table, the id columns, their foreign
keys and indexes, and the stored values.
Vito Sansevero