Final review fixes M1, M2, M3 and M5 for the workflow storage dedupe:
- M1: a batch also ends after 0.5 s, and the loop leaves the write lock
free for 0.15 s after each committed batch (longer than SQLite's longest
busy poll), so the image monitor and run counting on another connection
get in between. RED: a threaded writer with a 2 s busy timeout hit
"database is locked" (4 of 5 links landed).
- M2: each row runs in its own savepoint. A row that cannot be rewritten
(undecodable text, a text that would not come back) is rolled back alone,
stays inline and is listed in errors; the rest of the batch goes on.
- M3: a reused document is decompressed and checked before its id is handed
out, and repaired from the text (whose hash is its key) when damaged;
verify_database_file reads every document and refuses a damaged one.
- M5: the two legacy generated_images rebuilds refuse a file that already
has the document id columns instead of dropping them.
- The preview names the pre-migration backup and says it is always taken.
TESTING.md's maintenance contract gains the workflow storage rules: the
documents table and its two forms, the one accessor, garbage in the same
transaction plus the sweeps, the batched migration and Reclaim space, the
media and memory guarantees, and that a deduped backup cannot be restored
into an older version.
A test covers the preview on SQLite before 3.43 (no octet_length, as in
older Windows Python builds): the inline-bytes detail is left out.
The workflow storage migration rewrites old-form image rows (text inline)
to document ids, batch by batch:
- Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or
16 MB of text, found by a rowid range search and read one row at a time.
- Each text is stored or reused, then read back through the one accessor
before its row changes; a text that would not come back identical fails
and rolls back the whole batch.
- A value that is not text stays inline and is counted as protected.
- A crash or a stop leaves committed batches new form and the rest old
form; both read the same, and a rerun continues. No media file is opened.
- Unreferenced documents are swept at the end.
The maintenance preview gains a read-only workflow_storage plan (records to
rewrite, inline bytes where SQLite has octet_length, the Reclaim space and
older-version notes). reclaim_space sweeps, runs VACUUM and truncates the
WAL so the freed pages go back to the disk.
Vito Sansevero