feat(maintenance): document the workflow storage contract

TESTING.md's maintenance contract gains the workflow storage rules: the
documents table and its two forms, the one accessor, garbage in the same
transaction plus the sweeps, the batched migration and Reclaim space, the
media and memory guarantees, and that a deduped backup cannot be restored
into an older version.

A test covers the preview on SQLite before 3.43 (no octet_length, as in
older Windows Python builds): the inline-bytes detail is left out.
This commit is contained in:
Vito Sansevero
2026-10-24 04:51:59 -07:00
parent 0ad791a6f4
commit 3a309d3e27
2 changed files with 52 additions and 0 deletions
+43
View File
@@ -2695,6 +2695,49 @@ runs: every per-prompt protection and fingerprint read is an index search, the
duplicate groups and the link rows are one scan each, and a delete batch (400 ids)
searches by rowid with its cascades by index.
**Workflow storage** (`database/workflow_documents.py`, `database/workflow_storage.py`,
`py/api/workflow_storage.py`; `tests/test_workflow_documents.py`,
`tests/test_workflow_migration.py`, `tests/test_api_workflow_storage.py`). Each distinct
workflow and prompt graph text is one `workflow_documents` row: `sha256` of its exact
UTF-8 bytes (unique), `byte_length` and `data` = `zlib.compress` of those bytes. Image rows
reference it through `workflow_document_id` / `prompt_document_id` (foreign keys, one
index each). The text is stored and hashed exactly as written, never parsed and
re-serialized. Two forms are readable forever: old (text inline, no id) and new (inline
NULL, id set); inline text wins when both are present. Every reader goes through
`read_stored_text` / `resolve_image_row` (image lists, image by id, prompt info for the
inspector and metadata page, diagnostics, pair ingestion's comparison), which also drops
the id keys so payloads keep their shape; a document that fails its length or hash check
reads as unreadable (None), never as other text. Round trips are proven byte for byte
over edge texts (spacing, key order, duplicate keys, number spellings, `NaN`/`Infinity`
tokens, NUL, BOM, astral and combining characters, 500-deep nesting, a 1.2 MB+
workflow, `""` and NULL) and seeded random text and workflows
(`TestRoundTrip`). New writes (`link_image_to_prompt`, pair ingestion for scan, monitor
and Needs review) store the new form. No trigger is installed (restore validators refuse
them): a document goes in the transaction that drops its last reference (image and
prompt deletes, missing-link cleanup, duplicate merges, duplicate-file link merges,
ingestion updates: `TestGarbage`, `TestMoreGarbagePaths`), and startup, the migration's
end and Reclaim space sweep anything a path missed. `verify_database_file` checks the
table definition, the id columns, their foreign keys and indexes, the stored values and
dangling ids (`TestSchemaErrors`); a pre-dedupe file is accepted and migrated on open
and on restore. The `workflow_storage` maintenance plan counts the old-form records
(one pass over record headers) and its apply starts "Store each workflow once"
(`records`, cancellable): the verified safety backup, then batches of at most 200 rows or
16 MB of text, each one `BEGIN IMMEDIATE` transaction found by a rowid range search and
read one row at a time; each text is read back through the accessor before its row
changes, and a mismatch or any error rolls the batch back whole. A crash or Cancel leaves
committed batches new form and the rest old form, and the next apply continues
(`test_a_crash_mid_migration_leaves_every_row_readable_and_resumes`,
`test_cancel_stops_after_the_current_batch_and_a_rerun_continues`). Values that are not
text stay inline (protected). "Reclaim space" follows when every row was looked at
(`exclusive`, not cancellable): a disk check for VACUUM's copies of the live pages,
then sweep, VACUUM and `wal_checkpoint(TRUNCATE)`; when another job holds the registry it
does not start and the finished migration names Optimize database. No image or video
file is opened (`test_the_jobs_never_open_a_media_file` tracks every Python open under
the output folder and compares bytes and mtimes). Memory stays bounded by one row plus
the batch's distinct documents (`test_memory_stays_bounded_by_one_row`, tracemalloc).
Restoring a deduped backup on a version before this one is not supported (that version
reads rewritten images without their workflow).
## Workspace data fields for the concept pages (#249)
These fields are additive. The legacy `/prompt_manager/images/output` listing (no
+9
View File
@@ -354,6 +354,15 @@ class TestPreview(MigrationTestCase):
self.assertIn("Older versions", labels)
self.assertIn("Reclaim space", labels)
def test_preview_without_octet_length_omits_the_inline_bytes(self):
# SQLite before 3.43 (older Windows Python builds) has no octet_length.
self.library(count=4)
with patch.object(ws.sqlite3, "sqlite_version_info", (3, 37, 2)):
draft = self.db.preview_maintenance("workflow_storage")
labels = [row["label"] for row in draft.details]
self.assertNotIn("Inline text", labels)
self.assertEqual(draft.scope["count"], 4)
def test_preview_of_a_migrated_library_has_nothing_to_do(self):
self.library(count=5)
self.migrate()