feat(maintenance): document the workflow storage contract
TESTING.md's maintenance contract gains the workflow storage rules: the documents table and its two forms, the one accessor, garbage in the same transaction plus the sweeps, the batched migration and Reclaim space, the media and memory guarantees, and that a deduped backup cannot be restored into an older version. A test covers the preview on SQLite before 3.43 (no octet_length, as in older Windows Python builds): the inline-bytes detail is left out.
This commit is contained in:
+43
@@ -2695,6 +2695,49 @@ runs: every per-prompt protection and fingerprint read is an index search, the
|
||||
duplicate groups and the link rows are one scan each, and a delete batch (400 ids)
|
||||
searches by rowid with its cascades by index.
|
||||
|
||||
**Workflow storage** (`database/workflow_documents.py`, `database/workflow_storage.py`,
|
||||
`py/api/workflow_storage.py`; `tests/test_workflow_documents.py`,
|
||||
`tests/test_workflow_migration.py`, `tests/test_api_workflow_storage.py`). Each distinct
|
||||
workflow and prompt graph text is one `workflow_documents` row: `sha256` of its exact
|
||||
UTF-8 bytes (unique), `byte_length` and `data` = `zlib.compress` of those bytes. Image rows
|
||||
reference it through `workflow_document_id` / `prompt_document_id` (foreign keys, one
|
||||
index each). The text is stored and hashed exactly as written, never parsed and
|
||||
re-serialized. Two forms are readable forever: old (text inline, no id) and new (inline
|
||||
NULL, id set); inline text wins when both are present. Every reader goes through
|
||||
`read_stored_text` / `resolve_image_row` (image lists, image by id, prompt info for the
|
||||
inspector and metadata page, diagnostics, pair ingestion's comparison), which also drops
|
||||
the id keys so payloads keep their shape; a document that fails its length or hash check
|
||||
reads as unreadable (None), never as other text. Round trips are proven byte for byte
|
||||
over edge texts (spacing, key order, duplicate keys, number spellings, `NaN`/`Infinity`
|
||||
tokens, NUL, BOM, astral and combining characters, 500-deep nesting, a 1.2 MB+
|
||||
workflow, `""` and NULL) and seeded random text and workflows
|
||||
(`TestRoundTrip`). New writes (`link_image_to_prompt`, pair ingestion for scan, monitor
|
||||
and Needs review) store the new form. No trigger is installed (restore validators refuse
|
||||
them): a document goes in the transaction that drops its last reference (image and
|
||||
prompt deletes, missing-link cleanup, duplicate merges, duplicate-file link merges,
|
||||
ingestion updates: `TestGarbage`, `TestMoreGarbagePaths`), and startup, the migration's
|
||||
end and Reclaim space sweep anything a path missed. `verify_database_file` checks the
|
||||
table definition, the id columns, their foreign keys and indexes, the stored values and
|
||||
dangling ids (`TestSchemaErrors`); a pre-dedupe file is accepted and migrated on open
|
||||
and on restore. The `workflow_storage` maintenance plan counts the old-form records
|
||||
(one pass over record headers) and its apply starts "Store each workflow once"
|
||||
(`records`, cancellable): the verified safety backup, then batches of at most 200 rows or
|
||||
16 MB of text, each one `BEGIN IMMEDIATE` transaction found by a rowid range search and
|
||||
read one row at a time; each text is read back through the accessor before its row
|
||||
changes, and a mismatch or any error rolls the batch back whole. A crash or Cancel leaves
|
||||
committed batches new form and the rest old form, and the next apply continues
|
||||
(`test_a_crash_mid_migration_leaves_every_row_readable_and_resumes`,
|
||||
`test_cancel_stops_after_the_current_batch_and_a_rerun_continues`). Values that are not
|
||||
text stay inline (protected). "Reclaim space" follows when every row was looked at
|
||||
(`exclusive`, not cancellable): a disk check for VACUUM's copies of the live pages,
|
||||
then sweep, VACUUM and `wal_checkpoint(TRUNCATE)`; when another job holds the registry it
|
||||
does not start and the finished migration names Optimize database. No image or video
|
||||
file is opened (`test_the_jobs_never_open_a_media_file` tracks every Python open under
|
||||
the output folder and compares bytes and mtimes). Memory stays bounded by one row plus
|
||||
the batch's distinct documents (`test_memory_stays_bounded_by_one_row`, tracemalloc).
|
||||
Restoring a deduped backup on a version before this one is not supported (that version
|
||||
reads rewritten images without their workflow).
|
||||
|
||||
## Workspace data fields for the concept pages (#249)
|
||||
|
||||
These fields are additive. The legacy `/prompt_manager/images/output` listing (no
|
||||
|
||||
@@ -354,6 +354,15 @@ class TestPreview(MigrationTestCase):
|
||||
self.assertIn("Older versions", labels)
|
||||
self.assertIn("Reclaim space", labels)
|
||||
|
||||
def test_preview_without_octet_length_omits_the_inline_bytes(self):
|
||||
# SQLite before 3.43 (older Windows Python builds) has no octet_length.
|
||||
self.library(count=4)
|
||||
with patch.object(ws.sqlite3, "sqlite_version_info", (3, 37, 2)):
|
||||
draft = self.db.preview_maintenance("workflow_storage")
|
||||
labels = [row["label"] for row in draft.details]
|
||||
self.assertNotIn("Inline text", labels)
|
||||
self.assertEqual(draft.scope["count"], 4)
|
||||
|
||||
def test_preview_of_a_migrated_library_has_nothing_to_do(self):
|
||||
self.library(count=5)
|
||||
self.migrate()
|
||||
|
||||
Reference in New Issue
Block a user