diff --git a/TESTING.md b/TESTING.md index 4be830a1..dbd959bb 100644 --- a/TESTING.md +++ b/TESTING.md @@ -2695,6 +2695,49 @@ runs: every per-prompt protection and fingerprint read is an index search, the duplicate groups and the link rows are one scan each, and a delete batch (400 ids) searches by rowid with its cascades by index. +**Workflow storage** (`database/workflow_documents.py`, `database/workflow_storage.py`, +`py/api/workflow_storage.py`; `tests/test_workflow_documents.py`, +`tests/test_workflow_migration.py`, `tests/test_api_workflow_storage.py`). Each distinct +workflow and prompt graph text is one `workflow_documents` row: `sha256` of its exact +UTF-8 bytes (unique), `byte_length` and `data` = `zlib.compress` of those bytes. Image rows +reference it through `workflow_document_id` / `prompt_document_id` (foreign keys, one +index each). The text is stored and hashed exactly as written, never parsed and +re-serialized. Two forms are readable forever: old (text inline, no id) and new (inline +NULL, id set); inline text wins when both are present. Every reader goes through +`read_stored_text` / `resolve_image_row` (image lists, image by id, prompt info for the +inspector and metadata page, diagnostics, pair ingestion's comparison), which also drops +the id keys so payloads keep their shape; a document that fails its length or hash check +reads as unreadable (None), never as other text. Round trips are proven byte for byte +over edge texts (spacing, key order, duplicate keys, number spellings, `NaN`/`Infinity` +tokens, NUL, BOM, astral and combining characters, 500-deep nesting, a 1.2 MB+ +workflow, `""` and NULL) and seeded random text and workflows +(`TestRoundTrip`). New writes (`link_image_to_prompt`, pair ingestion for scan, monitor +and Needs review) store the new form. No trigger is installed (restore validators refuse +them): a document goes in the transaction that drops its last reference (image and +prompt deletes, missing-link cleanup, duplicate merges, duplicate-file link merges, +ingestion updates: `TestGarbage`, `TestMoreGarbagePaths`), and startup, the migration's +end and Reclaim space sweep anything a path missed. `verify_database_file` checks the +table definition, the id columns, their foreign keys and indexes, the stored values and +dangling ids (`TestSchemaErrors`); a pre-dedupe file is accepted and migrated on open +and on restore. The `workflow_storage` maintenance plan counts the old-form records +(one pass over record headers) and its apply starts "Store each workflow once" +(`records`, cancellable): the verified safety backup, then batches of at most 200 rows or +16 MB of text, each one `BEGIN IMMEDIATE` transaction found by a rowid range search and +read one row at a time; each text is read back through the accessor before its row +changes, and a mismatch or any error rolls the batch back whole. A crash or Cancel leaves +committed batches new form and the rest old form, and the next apply continues +(`test_a_crash_mid_migration_leaves_every_row_readable_and_resumes`, +`test_cancel_stops_after_the_current_batch_and_a_rerun_continues`). Values that are not +text stay inline (protected). "Reclaim space" follows when every row was looked at +(`exclusive`, not cancellable): a disk check for VACUUM's copies of the live pages, +then sweep, VACUUM and `wal_checkpoint(TRUNCATE)`; when another job holds the registry it +does not start and the finished migration names Optimize database. No image or video +file is opened (`test_the_jobs_never_open_a_media_file` tracks every Python open under +the output folder and compares bytes and mtimes). Memory stays bounded by one row plus +the batch's distinct documents (`test_memory_stays_bounded_by_one_row`, tracemalloc). +Restoring a deduped backup on a version before this one is not supported (that version +reads rewritten images without their workflow). + ## Workspace data fields for the concept pages (#249) These fields are additive. The legacy `/prompt_manager/images/output` listing (no diff --git a/tests/test_workflow_migration.py b/tests/test_workflow_migration.py index 32d51df0..51f8d922 100644 --- a/tests/test_workflow_migration.py +++ b/tests/test_workflow_migration.py @@ -354,6 +354,15 @@ class TestPreview(MigrationTestCase): self.assertIn("Older versions", labels) self.assertIn("Reclaim space", labels) + def test_preview_without_octet_length_omits_the_inline_bytes(self): + # SQLite before 3.43 (older Windows Python builds) has no octet_length. + self.library(count=4) + with patch.object(ws.sqlite3, "sqlite_version_info", (3, 37, 2)): + draft = self.db.preview_maintenance("workflow_storage") + labels = [row["label"] for row in draft.details] + self.assertNotIn("Inline text", labels) + self.assertEqual(draft.scope["count"], 4) + def test_preview_of_a_migrated_library_has_nothing_to_do(self): self.library(count=5) self.migrate()