feat(database): migrate old workflow rows in batches, then reclaim space

The workflow storage migration rewrites old-form image rows (text inline)
to document ids, batch by batch:

- Each batch is one BEGIN IMMEDIATE transaction of at most 200 rows or
  16 MB of text, found by a rowid range search and read one row at a time.
- Each text is stored or reused, then read back through the one accessor
  before its row changes; a text that would not come back identical fails
  and rolls back the whole batch.
- A value that is not text stays inline and is counted as protected.
- A crash or a stop leaves committed batches new form and the rest old
  form; both read the same, and a rerun continues. No media file is opened.
- Unreferenced documents are swept at the end.

The maintenance preview gains a read-only workflow_storage plan (records to
rewrite, inline bytes where SQLite has octet_length, the Reclaim space and
older-version notes). reclaim_space sweeps, runs VACUUM and truncates the
WAL so the freed pages go back to the disk.
This commit is contained in:
Vito Sansevero
2026-10-24 04:34:09 -07:00
parent b75c371430
commit 02b0e6813f
5 changed files with 789 additions and 4 deletions
+2 -3
View File
@@ -11,6 +11,7 @@ reads both forms, and gives back the exact text that was stored.
import json
import os
import random
import shutil
import sqlite3
import sys
import tempfile
@@ -134,9 +135,7 @@ class DocumentsTestCase(unittest.TestCase):
PromptModel.close_all_instances()
PromptModel.reset_schema_cache()
assert_closed(self, self.tmpdir)
for name in os.listdir(self.tmpdir):
os.unlink(os.path.join(self.tmpdir, name))
os.rmdir(self.tmpdir)
shutil.rmtree(self.tmpdir)
def conn(self):
return self.db.model.get_connection()