fix(system): Tagging selects show their whole chosen option at 390 px (#398 ) (#399 )

* fix(system): Tagging selects show their whole chosen option at 390 px (#398) The concept's phone cap (120px) cut Existing tags ("Replace after review"). #tag-existing now never shrinks for its row text and keeps its natural width at phone width, beside the text. The comparison expects selectsFit for every select, and a new test walks every option of both selects at 390 and 320 px in both themes. TESTING.md D132 records the decision. * fix(system): bound Existing tags by its row at phone width (#398 review)
fix(scan): read sampler-less graphs from the one wired text, retire the heuristic (#252 ) (#397 )
2026-10-28 02:03:22 +00:00 · 2026-10-28 00:32:30 +00:00 · 2026-10-27 22:40:57 +00:00 · 2026-10-27 22:20:27 +00:00 · 2026-10-27 22:07:59 +00:00 · 2026-10-27 20:30:54 +00:00
851 changed files with 295462 additions and 13954 deletions
@@ -0,0 +1,66 @@
 # Coverage configuration for pytest-cov
 [run]
 source = .
 omit = 
    */tests/*
    */venv/*
    */env/*
    */__pycache__/*
    */migrations/*
    */conftest.py
    */setup.py
    */manage.py
    */.tox/*
    */node_modules/*
    */static/*
    */media/*
    */htmlcov/*
    */.claude/*
    restart_gallery.py
    standalone_tagger.py
 branch = true
 parallel = true
 [report]
 # Regexes for lines to exclude from consideration
 exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover
    # Don't complain about missing debug-only code:
    def __repr__
    if self\.debug
    # Don't complain if tests don't hit defensive assertion code:
    raise AssertionError
    raise NotImplementedError
    # Don't complain if non-runnable code isn't run:
    if 0:
    if __name__ == .__main__.:
    # Don't complain about abstract methods, they aren't run:
    @(abc\.)?abstractmethod
    # Don't complain about type checking imports
    if TYPE_CHECKING:
    # Don't complain about logger calls
    \.logger\.(debug|info|warning|error|critical)
 ignore_errors = true
 show_missing = true
 skip_covered = false
 precision = 2
 [html]
 directory = htmlcov
 title = ComfyUI PromptManager Test Coverage Report
 [xml]
 output = coverage.xml
 [json]
 output = coverage.json
 pretty_print = true
@@ -0,0 +1,6 @@
 [flake8]
 max-line-length = 88
 extend-ignore = E203, W503
 exclude = reference/, .git, __pycache__, web/, venv/, htmlcov/, build/, dist/, .claude/, docs/, standalone_tagger.py
 per-file-ignores =
    tests/*:E402
@@ -0,0 +1,27 @@
 {
  "required_status_checks": {
    "strict": false,
    "contexts": [
      "py3.12 on ubuntu-latest",
      "py3.13 on ubuntu-latest",
      "py3.14 on ubuntu-latest",
      "py3.12 on windows-latest",
      "py3.13 on windows-latest",
      "py3.14 on windows-latest",
      "py3.12 on macos-latest",
      "py3.13 on macos-latest",
      "py3.14 on macos-latest",
      "coverage gate (85%)",
      "black and flake8",
      "frontend modules",
      "viewerjs",
      "real ComfyUI compatibility",
      "bandit (high severity)"
    ]
  },
  "enforce_admins": true,
  "required_pull_request_reviews": null,
  "restrictions": null,
  "allow_force_pushes": false,
  "allow_deletions": false
 }
@@ -0,0 +1,60 @@
 version: 2
 updates:
  # Python dependencies (requirements.txt / setup.cfg / pyproject.toml)
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "sunday"
      time: "06:00"
      timezone: "America/Los_Angeles"
    open-pull-requests-limit: 10
    labels:
      - "dependencies"
      - "security"
    commit-message:
      prefix: "deps(pip)"
      include: "scope"
    rebase-strategy: "auto"
    # Optional: group non-breaking updates to reduce PR noise
    groups:
      python-minor-and-patch:
        applies-to: "version-updates"
        update-types:
          - "minor"
          - "patch"
  # GitHub Actions workflow dependencies
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "sunday"
      time: "06:15"
      timezone: "America/Los_Angeles"
    open-pull-requests-limit: 10
    labels:
      - "dependencies"
    commit-message:
      prefix: "deps(actions)"
      include: "scope"
    rebase-strategy: "auto"
    groups:
      actions-minor-and-patch:
        applies-to: "version-updates"
        update-types:
          - "minor"
          - "patch"
  # Maintainer browser test dependencies; Viewer.js itself has an asset-aware updater.
  - package-ecosystem: "npm"
    directory: "/tools/vendor"
    schedule:
      interval: "weekly"
    allow:
      - dependency-name: "@playwright/test"
    labels:
      - "dependencies"
    commit-message:
      prefix: "deps(vendor-tools)"
@@ -0,0 +1,43 @@
 name: Code Quality
 permissions:
  contents: read
 on:
  merge_group:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 # Superseded PR pushes cancel their older run. Every main push gets its own
 # group (run_id), so no main commit's run is ever cancelled or dropped as pending.
 concurrency:
  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 jobs:
  security:
    name: bandit (high severity)
    runs-on: ubuntu-latest
    # ceil(1.5 x slowest of three green main runs) + 3, never under 5 (test.yml): runs 40257217665,
    # 40254137380, 40240706583 took 23s / 16s / 23s -> 5 min.
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v7
      - name: Set up Python 3.12
        uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - name: Install bandit
        run: pip install bandit[toml]
      # Medium findings are reported without failing until they are triaged.
      # High severity is a hard gate.
      - name: Security report (medium and above, informational)
        run: bandit -r . -ll -q --exclude ./tests,./reference,./venv,./tools/vendor/node_modules --exit-zero
      - name: Security scan (high severity gate)
        run: bandit -r . -lll -q --exclude ./tests,./reference,./venv,./tools/vendor/node_modules
@@ -18,7 +18,7 @@ jobs:
    if: ${{ github.repository_owner == 'ComfyAssets' }}
    steps:
      - name: Check out code
-        uses: actions/checkout@v4
+        uses: actions/checkout@v7
        with:
          submodules: true
      - name: Publish Custom Node
@@ -0,0 +1,250 @@
 name: Tests
 permissions:
  contents: read
 on:
  merge_group:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  workflow_dispatch:
 # Superseded PR pushes cancel their older run. Every main push gets its own
 # group (run_id), so no main commit's run is ever cancelled or dropped as pending.
 concurrency:
  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 env:
  PYTHONFAULTHANDLER: "1"
 # Every job has a time limit: ceil(1.5 x the slowest of three recent green runs on main) + 3
 # minutes, never under 5. The 3 minutes leave room for the Python runner's watchdog (120 s, see
 # tests/run_suite.py) to print its stack dump before GitHub cancels the job. A limit changes only
 # with new measurements written here (TESTING.md "Time limits and hangs").
 # Measured 2026-10-26 on runs 40257217681 (3be0ada), 40240706568 (fb9134f), 40228167328 (f9835aa):
 #   py matrix, slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
 #     (ubuntu 5m27s-6m27s, macos 6m14s-8m30s, windows 11m42s-15m54s)
 #     3.10 and 3.11 left the matrix and 3.14 joined it in #356; re-measure after three green main runs.
 #   real ComfyUI compatibility: 13m32s / 8m55s / 11m46s -> 24 min
 #   coverage gate (85%): 7m26s / 7m22s / 7m54s -> 15 min
 #   perf budget 30s / 29s / 27s, black and flake8 39s / 37s / 39s,
 #   frontend modules 23s / 19s / 29s -> 5 min (the floor)
 jobs:
  test:
    name: py${{ matrix.python-version }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    timeout-minutes: 27
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        python-version: ["3.12", "3.13", "3.14"]
    steps:
      - uses: actions/checkout@v7
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v7
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
          cache-dependency-path: requirements-dev.txt
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-dev.txt
      - name: Run tests
        run: python -m tests.run_suite -v
  comfyui:
    name: real ComfyUI compatibility
    runs-on: ubuntu-latest
    # Chromium and its system libraries come from the pinned Playwright image (tag = @playwright/test in
    # tools/vendor/package-lock.json; tests/vendor/browser-image.test.mjs checks it). The runner install
    # pulled them from Ubuntu's apt mirror on every run, and three runs (#349, #378, #382) crawled at
    # ~26 kB/s for 11-12 minutes until the job hit its limit after every test had passed.
    container:
      image: mcr.microsoft.com/playwright:v1.64.1-noble
      options: --user 1001
    timeout-minutes: 24
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - name: Read the reviewed runtime pins
        id: runtime
        run: |
          node --input-type=module <<'JS' >> "$GITHUB_OUTPUT"
          import runtime from './tools/vendor/comfyui-runtime.json' with { type: 'json' };
          for (const [key, value] of Object.entries(runtime)) console.log(`${key}=${value}`);
          import crossPack from './tools/vendor/cross-pack-runtime.json' with { type: 'json' };
          console.log(`kikoRepository=${crossPack.repository}`);
          console.log(`kikoRevision=${crossPack.revision}`);
          JS
      - name: Check out pinned ComfyUI
        uses: actions/checkout@v7
        with:
          repository: ${{ steps.runtime.outputs.repository }}
          ref: ${{ steps.runtime.outputs.revision }}
          path: .ci-comfyui
          persist-credentials: false
      - name: Check out pinned KikoTools for compatibility
        uses: actions/checkout@v7
        with:
          repository: ${{ steps.runtime.outputs.kikoRepository }}
          ref: ${{ steps.runtime.outputs.kikoRevision }}
          path: .ci-kikotools
          persist-credentials: false
      # The image's own Python and Node 24: setup-python/setup-node install the runner's toolcache builds,
      # whose entry points name host paths that do not exist in the container. The image has no ensurepip
      # and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv. The image
      # Python must be the reviewed host Python, so a pin or image bump that disagrees fails here.
      - name: Python environment on the reviewed host Python
        env:
          PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
          PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
          PINNED_PYTHON: ${{ steps.runtime.outputs.pythonVersion }}
        run: |
          image="$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])')"
          [ "$image" = "$PINNED_PYTHON" ] || { echo "::error::image Python $image is not the pinned $PINNED_PYTHON"; exit 1; }
          venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
          python3 -m venv --without-pip "$venv"
          curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
          echo "$PIP_SHA256  $wheel" | sha256sum -c -
          "$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
          echo "$venv/bin" >> "$GITHUB_PATH"
      - run: python --version && node --version && node --version | grep -q '^v24\.'
      # pip's download cache (torch alone is ~200 MB); inside the container ~ is the job's HOME (/github/home).
      - name: Cache pip downloads
        uses: actions/cache@v6
        with:
          path: ~/.cache/pip
          key: pip-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('requirements-dev.txt', '.ci-comfyui/requirements.txt', 'tools/vendor/comfyui-runtime.json') }}
      - name: Install CPU runtime and test dependencies
        run: |
          pip install "torch==${{ steps.runtime.outputs.torchVersion }}" "torchvision==${{ steps.runtime.outputs.torchvisionVersion }}" --index-url https://download.pytorch.org/whl/cpu
          pip install -r .ci-comfyui/requirements.txt -r requirements-dev.txt "torch==${{ steps.runtime.outputs.torchVersion }}" "torchvision==${{ steps.runtime.outputs.torchvisionVersion }}"
      # npm's download cache, keyed on the lockfile. node_modules itself is still built fresh by npm ci.
      - name: Cache npm downloads
        uses: actions/cache@v6
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
      - run: npm ci --prefix tools/vendor --ignore-scripts
      - name: Classic and Nodes 2.0 startup and compatibility
        env:
          PM_COMFY_ROOT: ${{ github.workspace }}/.ci-comfyui
          PM_TEST_PYTHON: python
        run: npm run --prefix tools/vendor test:comfyui
      - name: Real cached and headless execution history
        env:
          PM_COMFY_ROOT: ${{ github.workspace }}/.ci-comfyui
          PM_TEST_PYTHON: python
        run: npm run --prefix tools/vendor test:executions
      - name: Independent and combined pack compatibility
        env:
          PM_COMFY_ROOT: ${{ github.workspace }}/.ci-comfyui
          PM_KIKO_ROOT: ${{ github.workspace }}/.ci-kikotools
          PM_TEST_PYTHON: python
        run: npm run --prefix tools/vendor test:compat
      - name: Upload real-host reports and startup logs
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: real-comfyui-reports
          path: tools/vendor/test-results/
          if-no-files-found: ignore
          retention-days: 14
  coverage:
    name: coverage gate (85%)
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: requirements-dev.txt
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-dev.txt
      - name: Measure coverage
        run: |
          python -m coverage run -m tests.run_suite
          python -m coverage combine
          python -m coverage report --fail-under=85
  perf-budget:
    # The reference wall-clock measurement (TESTING.md "Performance budgets").
    # Shared-runner timing varies, so it never gates a pull request: it runs on
    # each push to main and on demand. A red run on main means fix the code.
    name: perf budget
    if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
          cache: pip
          cache-dependency-path: requirements-dev.txt
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-dev.txt
      - name: Workspace filter budget
        env:
          PROMPTMANAGER_PERF_BUDGET: "1"
        run: python -m unittest -v tests.test_api_workspace_images.TestWorkspaceImages.test_filter_budget
  lint:
    name: black and flake8
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install black flake8
      - run: black --check .
      - run: flake8 .
  js-test:
    name: frontend modules
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: "24"
      - name: Check shipped extension and page modules parse
        run: |
          shopt -s globstar nullglob
          for f in web/comfy/**/*.js web/comfy/**/*.mjs web/js/**/*.js web/js/**/*.mjs; do
            node --input-type=module --check < "$f" || exit 1
          done
      - name: Install pinned shared test contracts
        run: npm ci --prefix tools/vendor --ignore-scripts
      - name: Run JS tests
        run: node --test tests/js/*.test.js
@@ -0,0 +1,239 @@
 name: Viewer.js vendor and compatibility
 on:
  merge_group:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  workflow_dispatch:
 permissions:
  contents: read
 # Superseded PR pushes cancel their older run. Every main push gets its own
 # group (run_id), so no main commit's run is ever cancelled or dropped as pending.
 concurrency:
  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 # Timeouts are the measured job time plus at least 50%, and never under 20 minutes for a browser job,
 # because runner speed varies. Before sharding (main 2cecfcf, one worker): frontend coverage 32m10s,
 # page workflows firefox 18m59s, webkit 19m26s. First sharded run (PR #274 run 39712909047, 3 workers,
 # no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s
 # (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests
 # 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in
 # 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39725176320 at cbc8a61, npm ci 0-2s):
 # coverage shards 7m34s/6m04s/5m22s (tests 6m38s/4m53s/4m04s), merge 54s, firefox 7m57s/5m21s (tests
 # 7m08s/4m34s), webkit 8m16s/5m03s (tests 7m26s/3m58s); a cache miss adds up to 5m to npm ci, which
 # the 20-minute limits absorb.
 jobs:
  frontend-coverage:
    name: frontend coverage (shard ${{ matrix.shard }}/3)
    runs-on: ubuntu-latest
    # Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
    # same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
    # tests/vendor/browser-image.test.mjs checks it). Browsers and fonts come from the image.
    container:
      image: mcr.microsoft.com/playwright:v1.64.1-noble
      options: --user 1001
    timeout-minutes: 20
    # Public-repo ubuntu-latest runners have 4 vCPUs: three Playwright workers, each with its own test
    # server, leave one CPU for the servers and the runner.
    env:
      PM_TEST_WORKERS: 3
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    steps:
      - uses: actions/checkout@v7
      # The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
      # builds, whose entry points name host paths that do not exist in the container. The image has no
      # ensurepip and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv.
      - name: Python environment
        env:
          PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
          PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
        run: |
          venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
          python3 -m venv --without-pip "$venv"
          curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
          echo "$PIP_SHA256  $wheel" | sha256sum -c -
          "$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
          "$venv/bin/python" -m pip install --no-cache-dir -r requirements-dev.txt
          echo "$venv/bin" >> "$GITHUB_PATH"
      - run: python --version && node --version && node --version | grep -q '^v24\.'
      # npm's download cache, keyed on the lockfile; inside the container ~ is the job's HOME (/github/home),
      # which npm also uses. node_modules itself is still built fresh by npm ci.
      - name: Cache npm downloads
        uses: actions/cache@v6
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
      - run: npm ci --prefix tools/vendor --ignore-scripts
      # Shard 1 also runs the node units and the viewer suite, so the vendor pins and the tooling tests
      # run on the lightest shard instead.
      - if: matrix.shard == 3
        run: npm run --prefix tools/vendor check
      - if: matrix.shard == 3
        run: npm run --prefix tools/vendor test
      - name: Page workflows with coverage, shard ${{ matrix.shard }}/3
        run: npm run --prefix tools/vendor test:coverage:shard -- ${{ matrix.shard }}/3
      - name: Upload raw coverage
        uses: actions/upload-artifact@v7
        with:
          name: coverage-raw-${{ matrix.shard }}
          path: coverage/js/raw/
          if-no-files-found: error
          retention-days: 3
      - name: Upload Chromium and touch traces
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: frontend-traces-shard-${{ matrix.shard }}
          path: tools/vendor/test-results/
          retention-days: 14
  frontend-coverage-merge:
    name: frontend coverage (merge)
    needs: frontend-coverage
    runs-on: ubuntu-latest
    # Same image as the shards: the raw files name sources by the container's workspace path.
    container:
      image: mcr.microsoft.com/playwright:v1.64.1-noble
      options: --user 1001
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
      - run: node --version && node --version | grep -q '^v24\.'
      # npm's download cache, keyed on the lockfile; inside the container ~ is the job's HOME (/github/home),
      # which npm also uses. node_modules itself is still built fresh by npm ci.
      - name: Cache npm downloads
        uses: actions/cache@v6
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
      - run: npm ci --prefix tools/vendor --ignore-scripts
      # Each artifact lands in its own coverage-raw-N directory, so the next step can count them.
      - uses: actions/download-artifact@v8
        with:
          pattern: coverage-raw-*
          path: ${{ runner.temp }}/coverage-raw
      # coverage:merge only checks that raw files exist; merging two of three shards would report
      # partial coverage, so a missing or empty shard artifact fails here.
      - name: Verify every shard's raw coverage arrived
        shell: bash
        env:
          STAGED_RAW: ${{ runner.temp }}/coverage-raw
          EXPECTED_SHARDS: 3
        run: |
          found=0
          for shard in $(seq 1 "$EXPECTED_SHARDS"); do
            directory="$STAGED_RAW/coverage-raw-$shard"
            count=0
            if [ -d "$directory" ]; then count=$(find "$directory" -maxdepth 1 -type f -name '*.json' | wc -l); fi
            echo "coverage-raw-$shard: $count raw files"
            if [ "$count" -gt 0 ]; then found=$((found + 1)); fi
          done
          echo "shard artifacts with raw coverage: $found of $EXPECTED_SHARDS"
          if [ "$found" -ne "$EXPECTED_SHARDS" ]; then
            echo "::error::Expected raw coverage from $EXPECTED_SHARDS shards, got $found of $EXPECTED_SHARDS"
            exit 1
          fi
          staged=$(find "$STAGED_RAW" -mindepth 2 -maxdepth 2 -type f -name '*.json' | wc -l)
          mkdir -p coverage/js/raw
          # cp -n never overwrites; a clash shows up as fewer merged files than staged ones.
          find "$STAGED_RAW" -mindepth 2 -maxdepth 2 -type f -name '*.json' -exec cp -n -t coverage/js/raw {} + || true
          merged=$(find coverage/js/raw -maxdepth 1 -type f -name '*.json' | wc -l)
          if [ "$merged" -ne "$staged" ]; then
            echo "::error::Shards wrote clashing raw file names: $staged staged, $merged merged"
            exit 1
          fi
          echo "merged $merged raw files"
      - name: Frontend coverage report and baseline check
        run: npm run --prefix tools/vendor coverage:merge
      - name: Upload frontend coverage report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: frontend-test-reports
          path: |
            coverage/js/
            !coverage/js/raw/**
          retention-days: 14
  browser-workflows:
    name: page workflows (${{ matrix.browser }}, shard ${{ matrix.shard }}/2)
    runs-on: ubuntu-latest
    # Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
    # same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
    # tests/vendor/browser-image.test.mjs checks it). Browsers and fonts come from the image.
    container:
      image: mcr.microsoft.com/playwright:v1.64.1-noble
      options: --user 1001
    timeout-minutes: 20
    # Public-repo ubuntu-latest runners have 4 vCPUs: three Playwright workers, each with its own test
    # server, leave one CPU for the servers and the runner.
    env:
      PM_TEST_WORKERS: 3
    strategy:
      fail-fast: false
      matrix:
        browser: [firefox, webkit]
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v7
      # The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
      # builds, whose entry points name host paths that do not exist in the container. The image has no
      # ensurepip and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv.
      - name: Python environment
        env:
          PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
          PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
        run: |
          venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
          python3 -m venv --without-pip "$venv"
          curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
          echo "$PIP_SHA256  $wheel" | sha256sum -c -
          "$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
          "$venv/bin/python" -m pip install --no-cache-dir -r requirements-dev.txt
          echo "$venv/bin" >> "$GITHUB_PATH"
      - run: python --version && node --version && node --version | grep -q '^v24\.'
      # npm's download cache, keyed on the lockfile; inside the container ~ is the job's HOME (/github/home),
      # which npm also uses. node_modules itself is still built fresh by npm ci.
      - name: Cache npm downloads
        uses: actions/cache@v6
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
      - run: npm ci --prefix tools/vendor --ignore-scripts
      - name: Page workflows in ${{ matrix.browser }}, shard ${{ matrix.shard }}/2
        run: npm run --prefix tools/vendor test:workflows -- --project ${{ matrix.browser }} --shard ${{ matrix.shard }}/2
      - name: Upload browser traces
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: browser-traces-${{ matrix.browser }}-shard-${{ matrix.shard }}
          path: tools/vendor/test-results/
          retention-days: 14
  # Branch protection requires the exact context "viewerjs". This job keeps that
  # name and fails unless every frontend job above succeeded (every shard, the
  # coverage merge and every browser shard), including when one was cancelled or skipped.
  viewerjs:
    needs: [frontend-coverage, frontend-coverage-merge, browser-workflows]
    if: always()
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Require every frontend job to succeed
        env:
          COVERAGE_RESULT: ${{ needs.frontend-coverage.result }}
          MERGE_RESULT: ${{ needs.frontend-coverage-merge.result }}
          WORKFLOWS_RESULT: ${{ needs.browser-workflows.result }}
        run: |
          echo "frontend-coverage: $COVERAGE_RESULT"
          echo "frontend-coverage-merge: $MERGE_RESULT"
          echo "browser-workflows: $WORKFLOWS_RESULT"
          test "$COVERAGE_RESULT" = success && test "$MERGE_RESULT" = success && test "$WORKFLOWS_RESULT" = success
@@ -0,0 +1,83 @@
 name: Update vendored Viewer.js
 on:
  schedule:
    - cron: '30 7 * * *'
  workflow_dispatch:
    inputs:
      version:
        description: 'Stable version to vendor; leave blank for the latest npm release'
        required: false
        type: string
 permissions:
  contents: read
 concurrency:
  group: viewerjs-update
  cancel-in-progress: false
 jobs:
  update:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
      actions: write
    env:
      GH_TOKEN: ${{ github.token }}
    steps:
      - uses: actions/checkout@v7
        with:
          ref: ${{ github.event.repository.default_branch }}
      - uses: actions/setup-node@v7
        with:
          node-version: '24'
          cache: npm
          cache-dependency-path: tools/vendor/package-lock.json
      - run: npm ci --prefix tools/vendor --ignore-scripts
      - name: Update the dependency and regenerate assets
        env:
          VIEWER_VERSION: ${{ inputs.version }}
        run: |
          if [ -n "$VIEWER_VERSION" ]; then
            npm run --prefix tools/vendor update -- --version "$VIEWER_VERSION"
          else
            npm run --prefix tools/vendor update
          fi
      - name: Detect changes
        id: changes
        run: |
          if git diff --quiet -- tools/vendor/package.json tools/vendor/package-lock.json web/lib/viewerjs; then
            echo 'changed=false' >> "$GITHUB_OUTPUT"
          else
            echo 'changed=true' >> "$GITHUB_OUTPUT"
          fi
      - if: steps.changes.outputs.changed == 'true'
        run: |
          npm run --prefix tools/vendor check
          npm run --prefix tools/vendor test
          node --test tests/js/*.test.js
          npx --prefix tools/vendor playwright install --with-deps chromium
          npm run --prefix tools/vendor test:browser
      - name: Open the update PR
        if: steps.changes.outputs.changed == 'true'
        env:
          BASE_BRANCH: ${{ github.event.repository.default_branch }}
        run: |
          version=$(node -p "require('./web/lib/viewerjs/provenance.json').version")
          branch="deps/viewerjs-$version"
          existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --state all --head "$branch" --json number --jq 'length')
          if [ "$existing" != '0' ]; then
            echo "A PR already exists for Viewer.js $version; leaving it for review."
            exit 0
          fi
          remote=$(git ls-remote origin "refs/heads/$branch")
          if [ -z "$remote" ]; then
            git switch -c "$branch"
            git add tools/vendor/package.json tools/vendor/package-lock.json web/lib/viewerjs
            git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' commit -m "deps: update vendored Viewer.js to $version"
            git push origin "HEAD:refs/heads/$branch"
          fi
          gh pr create --repo "$GITHUB_REPOSITORY" --base "$BASE_BRANCH" --head "$branch" --title "deps: update vendored Viewer.js to $version" --body-file tools/vendor/update-pr.md
          gh workflow run viewerjs-check.yml --repo "$GITHUB_REPOSITORY" --ref "$branch"
@@ -1,3 +1,29 @@
 # Tailwind standalone CLI (downloaded by `make css-setup`)
 tailwindcss
 # AI tooling / local config
 .claude/
 .playwright-mcp/
 .serena/
 AGENTS.md
 CLAUDE.md
 # Local files
 # Local PromptManager state (config.json holds API keys and machine paths)
 config.json
 resume.sh
 *.backup_*
 promptmanager_logs/
 custom_data/
 data/
 docs/
 logs/
 standalone_tagger.py
 prompts.db
 prompts.db-shm
 prompts.db-wal
 prompts.db-journal
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -16,6 +42,8 @@ eggs/
 .eggs/
 lib/
 lib64/
 !web/lib/
 !web/lib/**
 parts/
 sdist/
 var/
@@ -172,3 +200,17 @@ cython_debug/
 # PyPI configuration file
 .pypirc
 # Maintainer-only JavaScript tooling and browser test artifacts
 node_modules/
 test-results/
 playwright-report/
 # Combined frontend coverage reports
 coverage/
 # Disposable upstream checkout for real-host CI
 .ci-comfyui/
 # Old library files a Move library renamed (never deleted; #249 S6b)
 *.moved-to-*
@@ -0,0 +1,28 @@
 repos:
  - repo: local
    hooks:
      - id: black
        name: black
        entry: black --line-length=88 --target-version=py310
        language: system
        types: [python]
      - id: flake8
        name: flake8
        entry: flake8 --show-source
        language: system
        types: [python]
      - id: bandit
        name: bandit
        entry: bash -c 'bandit "$@" -ll -q 2>&1; true'
        language: system
        types: [python]
        exclude: ^(tests/|reference/)
      - id: tests
        name: tests
        entry: python -m tests.run_suite -q
        language: system
        pass_filenames: false
        stages: [pre-push]
@@ -0,0 +1,43 @@
 TAILWIND_VERSION := 4.1.18
 TAILWIND_CLI := ./tailwindcss
 UNAME_S := $(shell uname -s)
 UNAME_M := $(shell uname -m)
 ifeq ($(UNAME_S),Darwin)
  ifeq ($(UNAME_M),arm64)
    TAILWIND_PLATFORM := tailwindcss-macos-arm64
  else
    TAILWIND_PLATFORM := tailwindcss-macos-x64
  endif
 else
  ifeq ($(UNAME_M),aarch64)
    TAILWIND_PLATFORM := tailwindcss-linux-arm64
  else
    TAILWIND_PLATFORM := tailwindcss-linux-x64
  endif
 endif
 TAILWIND_URL := https://github.com/tailwindlabs/tailwindcss/releases/download/v$(TAILWIND_VERSION)/$(TAILWIND_PLATFORM)
 .PHONY: css css-watch css-setup clean-css
 ## Download Tailwind standalone CLI (dev only, not committed)
 css-setup:
 	@if [ ! -f $(TAILWIND_CLI) ]; then \
 		echo "Downloading Tailwind CSS v$(TAILWIND_VERSION) CLI..."; \
 		curl -sLo $(TAILWIND_CLI) $(TAILWIND_URL) && chmod +x $(TAILWIND_CLI); \
 	else \
 		echo "Tailwind CLI already present"; \
 	fi
 ## Rebuild compiled CSS (run after changing Tailwind classes in HTML/JS)
 css: css-setup
 	$(TAILWIND_CLI) -i web/lib/tailwind/input.css -o web/lib/tailwind/styles.css --minify
 ## Watch mode for development
 css-watch: css-setup
 	$(TAILWIND_CLI) -i web/lib/tailwind/input.css -o web/lib/tailwind/styles.css --watch
 ## Remove downloaded CLI binary
 clean-css:
 	rm -f $(TAILWIND_CLI)
@@ -1,48 +1,12 @@
 # ComfyUI Prompt Manager
-A comprehensive ComfyUI custom node that extends the standard text encoder with persistent prompt storage, advanced search capabilities, automatic image gallery system, and powerful ComfyUI workflow metadata analysis using SQLite.
+![License](https://img.shields.io/badge/license-MIT-blue.svg)
 ![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)
 ![ComfyUI](https://img.shields.io/badge/ComfyUI-compatible-green.svg)
 ![Tests](https://github.com/ComfyAssets/ComfyUI_PromptManager/workflows/Tests/badge.svg)
 ![Code Quality](https://github.com/ComfyAssets/ComfyUI_PromptManager/workflows/Code%20Quality/badge.svg)
-# 🚀 Prompt Manager v2 Rewrite  
+A comprehensive ComfyUI custom node that extends the standard text encoder with persistent prompt storage, advanced search capabilities, automatic image gallery system, folder-based organization, LoRA Manager integration, and powerful ComfyUI workflow metadata analysis using SQLite.
 We’ve been hard at work on a **complete rewrite** of Prompt Manager.  
 The new version brings over all features from v1, introduces a dark theme (v2 look), and adds **negative prompt support**.  
 ## ✨ What’s New
 - Unified dark theme UI (v2)  
 - Table view (for power users) and Card view (for quick browsing)  
 - Negative prompt field with clear highlighting  
 - Streamlined toolbar & filters  
 ---
 ## 📊 Progress  
 **Rewrite completion status:**  
 - [█████████░░] 85% done  
 ✅ Core features migrated  
 ✅ Dark UI overhaul  
 ✅ Table + Card views  
 ✅ Negative prompt handling  
 ⬜ API integration & real data wiring  
 ⬜ Community feedback + polish pass  
 ---
 ## 🧪 Call for Beta Testers  
 We’d love your help testing Prompt Manager v2 before release!  
 - Try the new UI locally  
 - Report bugs, UI quirks, or feature requests  
 - Share feedback on usability & performance  
 👉 Open an [Issue](../../issues) with feedback or join the discussion in [Discussions](../../discussions).  
 ---
 💡 Thanks to everyone who has supported this project—your input drives its evolution!  
 ## Overview
@@ -67,12 +31,24 @@ A text-focused variant that outputs raw STRING for maximum flexibility:
 - **🔗 Text Concatenation**: Built-in prepend/append functionality for dynamic text assembly
 - **⚡ Lightweight Processing**: No CLIP encoding overhead for text-only workflows
 ### **PromptSearchList** (Batch Processing Node)
 A search node that outputs prompts as a list for batch processing workflows:
 - **🔍 Database Search**: Search prompts by text, category, tags, and rating
 - **📋 List Output**: Uses `OUTPUT_IS_LIST` for native batch processing support
 - **🔗 Batch Compatible**: Connect directly to batch processing nodes like String OutputList
 - **⚡ Read-Only**: Lightweight search operation without database writes
 This node enables powerful batch workflows by allowing you to search your prompt database and process multiple prompts in a single execution.
 ### **Shared Features**
 Both nodes include the complete PromptManager feature set:
 - **💾 Persistent Storage**: Automatically saves all prompts to a local SQLite database
 - **🔍 Advanced Search**: Query past prompts with text search, category filtering, and metadata
 - **📁 Folder Filter**: Browse and filter prompts by output subdirectory
 - **🖼️ Automatic Image Gallery**: Automatically links generated images to their prompts
 - **🏷️ Rich Metadata**: Add categories, tags, ratings, notes, and workflow names to prompts
 - **🚫 Duplicate Prevention**: Uses SHA256 hashing to detect and prevent duplicate storage
@@ -81,6 +57,8 @@ Both nodes include the complete PromptManager feature set:
 - **🔬 Workflow Analysis**: Extract and analyze ComfyUI workflow data from PNG images
 - **📋 Metadata Viewer**: Standalone tool for analyzing ComfyUI-generated images
 - **🛠️ System Management**: Built-in diagnostics, backup/restore, and maintenance tools
 - **🏷️ AI AutoTag**: Automatically tag your image collection using WD14 or JoyCaption vision models
 - **🔗 LoRA Manager Integration**: Import LoRA metadata and preview images from [ComfyUI-Lora-Manager](https://github.com/willmiao/ComfyUI-Lora-Manager)
 ![Image Gallery](images/pm-02.png)
@@ -124,6 +102,7 @@ The comprehensive gallery system provides complete management of all ComfyUI out
 - **Performance Optimized**: Efficient scanning with thumbnail generation and lazy loading
 #### **🎬 Full Video Support**
 - **Native Video Playback**: HTML5 video player with custom controls for all supported formats
 - **Video Thumbnails**: Automatic thumbnail generation with OpenCV, ffmpeg, or placeholder fallback
 - **Playback Controls**: Autoplay, mute, loop settings with persistent preferences
@@ -132,6 +111,7 @@ The comprehensive gallery system provides complete management of all ComfyUI out
 - **Mixed Media Gallery**: Videos and images integrated seamlessly in the same interface
 #### **🔍 Professional Image Viewer (ViewerJS Integration)**
 - **Full-Featured Viewing**: Zoom, rotation, slideshow modes with professional controls
 - **Advanced Navigation**: Smooth transitions, thumbnail strip, and keyboard shortcuts
 - **Zoom Controls**: Mouse wheel zoom, drag to pan, and touch gesture support
@@ -150,6 +130,7 @@ The comprehensive gallery system provides complete management of all ComfyUI out
 - **Cross-Platform Support**: Works on desktop, tablet, and mobile devices
 #### **⚡ Performance Features**
 - **Smart Thumbnail System**: Generates optimized thumbnails for faster loading
 - **Lazy Loading**: Load content as needed for smooth browsing of large galleries
 - **Grid & List Views**: Choose between compact grid or detailed list display
@@ -158,6 +139,35 @@ The comprehensive gallery system provides complete management of all ComfyUI out
 _Powered by [ViewerJS](https://github.com/fengyuanchen/viewerjs) for professional image viewing capabilities._
 Viewer.js is vendored locally. See [Viewer.js maintenance](tools/vendor/README.md)
 for reproducible updates, verification, and automated update PRs.
 Prompt Manager appears as **Prompt Manager (Classic)** in the Add Node menu.
 It keeps the existing layout and behavior while using shared
 [ComfyAssets Toolkit](https://github.com/ComfyAssets/ComfyAssets-Toolkit) controls
 that follow ComfyUI's light/dark theme. Existing workflows keep the same internal
 node identifier and sockets. The reusable architecture is inspired by
 [Pixaroma](https://github.com/pixaroma/ComfyUI-Pixaroma); no implementation or assets
 were copied. See the [toolkit maintenance notes](tools/vendor/README.md#shared-comfyassets-browser-controls).
 **Prompt Manager (Focused)** adds an editor-first layout with collapsible
 composition and metadata fields. Open **Browse library** to search saved prompts,
 filter by saved category and comma-separated tags, preview a selection, then choose
 **Use prompt**; **Load metadata** controls whether
 category and tags come with it. Metadata fields suggest saved categories and tags;
 choose with the mouse or arrow keys and Enter. Tag selections append `, ` for the
 next tag; category remains a single value. New values can still be typed freely.
 Prepend and append STRING sockets remain visible
 when the fields are collapsed, so LoRA trigger outputs can connect directly.
 Encoding, storage, and CONDITIONING/text outputs match Classic. Both layouts
 support ComfyUI Classic and Nodes 2.0.
 **Prompt Manager (Split)** keeps the same editor and library visible together.
 Wide nodes place the library beside the editor; narrow nodes stack the panels.
 It shares Focused's category/tag autocomplete, search filters, preview-before-use,
 workflow persistence, and connectable prepend/append inputs. Encoding and storage
 match Classic and Focused.
 ### 🔬 ComfyUI Workflow Analysis
 - **PNG Metadata Parsing**: Advanced PNG chunk analysis to extract embedded ComfyUI data
@@ -183,7 +193,7 @@ _Powered by [ViewerJS](https://github.com/fengyuanchen/viewerjs) for professiona
 - **Full-text search** across all stored prompts
 - **Category filtering** for organized prompt collections
- **Tag-based search** with support for multiple tags
+- **Tag-based search** with support for multiple tags, with autocomplete that narrows to tags seen together with the ones already typed
 - **Rating filters** to find your best prompts
 - **Date range filtering** for temporal searches
 - **Recent prompts** quick access
@@ -279,6 +289,20 @@ For text-only workflows or when you need STRING output:
   - **Append Text**: Text added after your main prompt
 4. **Connect the STRING output** to any node that accepts text input
 #### Using PromptSearchList (Batch Processing)
 For batch processing workflows that need multiple prompts:
 1. **Add the PromptSearchList node** to your workflow
 2. **Configure search filters**:
   - **text**: Search for prompts containing this text
   - **category**: Filter by specific category
   - **tags**: Comma-separated tags to filter by (e.g., "anime, detailed")
   - **min_rating**: Minimum rating (0-5) to include
   - **limit**: Maximum number of results (1-1000, default 50)
 3. **Connect the prompts output** to batch-compatible nodes like String OutputList
 4. **Each matching prompt** will be processed as a separate batch item
 #### Shared Metadata Options
 Both nodes support the same metadata fields:
@@ -296,22 +320,30 @@ Both nodes will automatically save prompts to the database and link any generate
 The comprehensive gallery system provides complete access to all your ComfyUI output:
 #### **📁 Accessing the Gallery**
 1. **Open the admin interface** at `http://localhost:8188/prompt_manager/admin`
 2. **Click the "🖼️ Gallery" button** to open the output-wide gallery
 3. **Browse all content** from your ComfyUI output directory
 #### **🎬 Video & Image Support**
 - **Mixed Media Viewing**: Images and videos appear together in the same gallery
 - **Video Thumbnails**: Videos show with play button overlays and 🎬 icons
 - **Smart Filtering**: Filter by media type (All, Images, Videos) using the dropdown
 #### **⚙️ Gallery Settings**
 Configure your viewing experience in the settings panel:
- **Performance Settings**: Grid columns, image quality, lazy loading
+
 - **Display Settings**: Default view, items per page, grid columns, media info, infinite scrolling
 - **Performance Settings**: Image quality, lazy loading
 - **Video Settings**: Autoplay, mute by default, loop videos
- **Thumbnail Management**: Generate optimized thumbnails for faster loading
+- **Thumbnail Management**: Generate optimized thumbnails for faster loading (runs on the configured worker threads; a run that stops at the per-run cap tells you how many remain)
 - **Find Duplicate Images**: Hashes every file on the worker threads with a progress bar; close the window and it keeps running, results wait in a pill
 - **Rescan Output Folder**: Drops the server-side listing cache and rewalks the folders
 #### **🖼️ Image Viewing Experience**
 1. **Click any image** to open the professional viewer with:
   - **Full-screen viewing** with zoom, rotation, and slideshow controls
   - **Integrated metadata panel** showing workflow data and generation parameters
@@ -323,6 +355,7 @@ Configure your viewing experience in the settings panel:
   - **Space** for slideshow mode
 #### **🎬 Video Viewing Experience**
 1. **Click any video** to open the video player with:
   - **HTML5 video controls** with play, pause, seek, and fullscreen
   - **Custom control buttons** for mute/unmute and loop toggle
@@ -331,6 +364,7 @@ Configure your viewing experience in the settings panel:
 2. **Playback settings** are remembered across sessions
 #### **🔍 Advanced Features**
 - **Real-time Search**: Search across filenames and metadata
 - **View Modes**: Switch between grid and list views
 - **Sorting Options**: Sort by date, size, or filename
@@ -356,14 +390,125 @@ Use the standalone metadata viewer to analyze any ComfyUI-generated PNG:
 Import existing ComfyUI images into your database:
 1. **Open admin dashboard** and click **"📸 Scan Images"**
-2. **Start scan** to analyze your entire output directory
+2. **Start scan** to analyze every configured output directory
-3. **Monitor progress** with real-time status updates
+3. **Monitor progress** with real-time status updates. The scan runs on the server: closing the window only minimizes it to a progress pill, and reloading the page re-attaches to a running scan
 4. **Review results** showing:
   - Images found and processed
   - Prompts extracted and linked
   - Any errors or issues encountered
 5. **Access imported data** through the normal gallery interface
 The scan reads PNG text chunks and A1111-style parameters, and, when `ffprobe` is installed, the prompt ComfyUI's video savers embed in MP4/MOV/WebM files. Files are read on the number of **worker threads** set in Settings → Performance (default: half your cores), which makes a large library several times faster.
 ### 🏷️ AI AutoTag
 Automatically tag your entire image collection using AI vision models:
 #### **Model Options**
 - **WD14 SwinV2 / WD14 ViT**: Fast ONNX-based classifiers (~400MB) that output Danbooru tags with confidence scores — no prompt needed, adjustable confidence thresholds
 - **JoyCaption Beta One FP16**: Full precision model for highest quality tagging (requires more VRAM)
 - **JoyCaption Beta One GGUF (FP8)**: Quantized model for lower VRAM usage with minimal quality loss
 #### **Two Tagging Modes**
 1. **AutoTag (Batch Mode)**: Tag your entire collection automatically
   - Click **"🏷️ AutoTag"** in the admin dashboard
   - Choose your model type (FP16 or GGUF)
   - Select how to handle already-tagged images:
     - **Skip images with existing tags**: Preserve your meticulous manual tagging work
     - **Re-tag all images**: Overwrite existing tags with fresh AI analysis
   - Customize the system prompt to match your collection style
   - Monitor real-time progress as images are processed
 2. **Review Mode**: Tag images one-by-one with approval
   - Review each image and its AI-generated tags before applying
   - Edit, add, or remove tags before saving
   - Skip images you don't want to tag
   - Perfect for curating high-quality tag assignments
 #### **Handling Existing Tags**
 For users who have meticulously tagged their collections from the start:
 - AutoTag respects your existing work with the **"Skip tagged"** option
 - **"auto-scanned"** placeholder tags don't count as real tags
 - Choose to re-tag specific images while preserving others
 - Review mode asks before overwriting on each image
 #### **Customization**
 Adjust the system prompt to match your tagging style:
 - Focus on specific attributes (style, mood, composition, subjects)
 - Match your existing tag vocabulary
 - Optimize for your collection's theme (anime, photography, landscapes, etc.)
 #### **Requirements**
 - Sufficient VRAM for the chosen model (FP16 requires more, GGUF is lighter)
 - Models are downloaded automatically on first use
 - Feature requests and improvements are welcome!
 ### 📁 Folder Filter
 ![Folder Filter](images/pm-lora-import.png)
 Organize and filter your prompt library by output subdirectory. The dashboard search panel includes a **Folder** dropdown that lists all subdirectories where your images are stored.
 - **Automatic Detection**: Folders are extracted from your image paths — no configuration needed
 - **Multi-Directory Support**: Works across all configured gallery scan directories
 - **Quick Filtering**: Select a folder from the dropdown to instantly filter prompts to that subdirectory
 > **Note:** If you have an existing prompt library from before the folder feature was added, you will need to **rescan your image library** (click **Scan Images** in the admin dashboard) to populate folder data for your existing prompts.
 ### 🔗 LoRA Manager Integration
 ![LoRA Settings](images/pm-settings-integrations.png)
 If you use [ComfyUI-Lora-Manager](https://github.com/willmiao/ComfyUI-Lora-Manager), PromptManager can import your LoRA metadata, trigger words, and example images directly into your prompt database. This lets you search, tag, and browse your LoRA collection alongside your regular prompts.
 > **WIP:** LoRA Manager support is a work in progress — this was a highly requested feature. Please [open issues](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues) for any bugs or feature requests.
 #### **Enabling the Integration**
 1. Open the admin dashboard and click **Settings**
 2. Scroll to the **Integrations** section — PromptManager will auto-detect if LoRA Manager is installed
 ![LoRA Enabled](images/pm-lora-enabled.png)
 3. Toggle the **LoraManager** switch to enable
 4. Optionally enable **Auto-inject trigger words** to automatically append LoRA trigger words when `<lora:name:weight>` is detected in your prompts
 5. Click **Save Settings** (requires a ComfyUI restart to take effect)
 #### **Importing LoRA Data**
 Click **Import LoRA Data** in the settings panel to start the import. A progress popup shows real-time status as each LoRA is processed:
 - Reads LoRA Manager's own settings the way LoRA Manager does: from the folder in `LORA_MANAGER_SETTINGS_DIR`, from the plugin's `settings.json` in portable mode, or from your user config folder, and only the keys that say where its files are
 - Finds each LoRA's metadata in both storage modes: beside the model (the default) or in the central sidecar folder, and imports only LoRAs whose model file still exists in a LoRA folder (the review says how many metadata files it skipped)
 - Links LoRA Manager's preview (including `.preview.webp` and video previews) and every downloaded example image and video where they are, without copying them
 - Downloads example images from CivitAI as 512px thumbnails only for LoRAs that have no local image
 - Creates a prompt entry for each LoRA with trigger words, example prompts, and preview images
 - All imported LoRAs are tagged with `lora-manager` category for easy filtering
 The System › Integrations review names the settings it read (source, storage mode and LoRA Manager library) and counts metadata files, LoRAs and LoRAs with local images. If LoRA Manager's settings cannot be used, it says why and nothing is changed.
 ![LoRA Results](images/pm-lora-filtered.png)
 After import, filter by the `lora-manager` category to browse your LoRA collection.
 #### **CivitAI API Key (Optional)**
 A CivitAI API key is **required if your library contains NSFW LoRAs** — CivitAI blocks unauthenticated access to NSFW preview images. Without a key, only SFW preview images are downloaded.
 To add your key:
 1. Get your API key from [civitai.com/user/account](https://civitai.com/user/account)
 2. Paste it into the **CivitAI API Key** field in the Integrations settings
 3. Save and re-import to download any previously skipped images
 The key is sent only over HTTPS to CivitAI's own hosts. CivitAI serves images through a redirect to its storage host; PromptManager follows such a redirect only over HTTPS to a host on CivitAI's own `civitai.com`, `civitai.red` or `civitai.green` domains, at most three hops, and never sends the key with it. When downloads fail, the import's status line says how many; when none could be downloaded, it starts with a warning. The server log has one warning per failed image with the reason (HTTP status, refused redirect, size cap, unreadable image, timeout or network error).
 > **Note:** Re-importing is safe — PromptManager skips LoRAs that were already imported. To do a fresh import, the previous `lora-manager` data is cleared automatically before re-scanning.
 ### 🌐 Web Interface Features
 The comprehensive web interface provides:
@@ -392,7 +537,7 @@ Keep your database optimized with built-in maintenance tools:
 ### Database Location
-By default, the database is saved as `example_prompts.db` in the node directory. This file contains all your prompts and linked images and can be backed up or shared.
+By default, the database is saved as `prompts.db` in the extension directory. It contains your prompt library and image links. See the [database path compatibility note](#database-path-compatibility-211) before upgrading a deployment with a custom path.
 ### Searching Prompts
@@ -471,9 +616,29 @@ PromptManagerText → Text Processor → Style Applicator → Final Text Node
 "cyberpunk city" → style processing → "neon-lit cyberpunk metropolis at night"
 ```
 ### PromptSearchList (Batch Processing) Examples
 #### Example 6: Batch Processing with Tags
 ```
 Tags: "anime, detailed"
 Limit: 10
 → Outputs: List of up to 10 prompts with both "anime" and "detailed" tags
 → Each prompt is processed as a separate batch item
 ```
 #### Example 7: Category-Based Batch
 ```
 Category: "portraits"
 Min Rating: 4
 Limit: 20
 → Outputs: List of up to 20 highly-rated portrait prompts for batch generation
 ```
 ### Traditional Examples
-#### Example 6: Abstract Art
+#### Example 8: Abstract Art
 ```
 Input: "Swirling colors in an abstract geometric pattern"
@@ -524,10 +689,12 @@ CREATE TABLE generated_images (
 - **`prompt_manager.py`** - Main ComfyUI node implementation (CLIP encoding)
 - **`prompt_manager_text.py`** - Text-only node implementation (STRING output)
 - **`prompt_search_list.py`** - Batch search node implementation (LIST output)
 - **`database/models.py`** - Database schema and connection management
 - **`database/operations.py`** - CRUD operations and search functionality
- **`py/api.py`** - Web API endpoints for the interface
+- **`py/api/`** - Web API route modules (prompts, images, tags, lora integration, etc.)
 - **`py/config.py`** - Configuration management
 - **`py/lora_utils.py`** - LoRA Manager integration utilities
 - **`utils/hashing.py`** - SHA256 hashing for deduplication
 - **`utils/validators.py`** - Input validation and sanitization
 - **`utils/image_monitor.py`** - Automatic image detection system
@@ -537,7 +704,9 @@ CREATE TABLE generated_images (
 - **`utils/diagnostics.py`** - System diagnostics and health checks
 - **`web/admin.html`** - Advanced admin dashboard with metadata panel
 - **`web/index.html`** - Simple web interface
- **`web/prompt_manager.js`** - JavaScript functionality
+- **`web/comfy/prompt_manager.js`** - ComfyUI canvas extension (the only file ComfyUI loads)
 - **`web/js/admin.js`**, **`web/js/gallery.js`** - Dashboard and gallery JavaScript
 - **`web/js/tags-page.js`** - Tag management JavaScript
 - **`web/metadata.html`** - Standalone PNG metadata viewer
 ### File Structure
@@ -547,14 +716,16 @@ ComfyUI_PromptManager/
 ├── __init__.py                    # Node registration
 ├── prompt_manager.py             # Main node implementation (CLIP encoding)
 ├── prompt_manager_text.py        # Text-only node implementation (STRING output)
 ├── prompt_search_list.py         # Batch search node implementation (LIST output)
 ├── database/
 │   ├── __init__.py
 │   ├── models.py                 # Database schema
 │   └── operations.py             # Database operations
 ├── py/
 │   ├── __init__.py
-│   ├── api.py                    # Web API endpoints
+│   ├── api/                      # Web API route modules
-│   └── config.py                 # Configuration
+│   ├── config.py                 # Configuration
 │   └── lora_utils.py             # LoRA Manager integration
 ├── utils/
 │   ├── __init__.py
 │   ├── hashing.py               # Hashing utilities
@@ -566,9 +737,15 @@ ComfyUI_PromptManager/
 │   └── diagnostics.py           # System diagnostics
 ├── web/
 │   ├── admin.html               # Advanced admin dashboard
 │   ├── gallery.html             # Image gallery
 │   ├── index.html               # Simple web interface
 │   ├── metadata.html            # Standalone metadata viewer
-│   └── prompt_manager.js        # JavaScript functionality
+│   ├── comfy/
 │   │   └── prompt_manager.js    # ComfyUI canvas extension (WEB_DIRECTORY)
 │   └── js/
 │       ├── admin.js             # Dashboard JavaScript
 │       ├── gallery.js           # Gallery JavaScript
 │       └── tags-page.js         # Tag management JavaScript
 ├── tests/
 │   ├── __init__.py
 │   └── test_basic.py            # Test suite
@@ -580,15 +757,96 @@ ComfyUI_PromptManager/
 ## Configuration
 ### Where your library lives
 New installs keep the library (`prompts.db`, `config.json` with the CivitAI key, the
 database backups and the logs) in ComfyUI's system user folder
 `ComfyUI/user/__promptmanager/`. An update, a reinstall or ComfyUI-Manager replacing
 the node folder no longer touches it, and ComfyUI's `/userdata` web API does not
 serve `__` system folders. Thumbnails stay in `output/thumbnails`.
 - **Upgrading from an earlier version:** your library stays where it is, in the node
  folder, until you move it; nothing is moved on its own. The workspace shows a notice
  above every page and System › Overview a "Library location" row. **Move library**
  copies the database with SQLite's backup API, verifies the copy (integrity check, every
  workflow document, the same rows in every table), copies `config.json` and the backups,
  switches to the new folder without a restart and only then renames the old files
  `prompts.db.moved-to-__promptmanager` and `config.json.moved-to-__promptmanager` (never
  deleted; the old backups stay too). If anything fails, the library stays in the node
  folder, unchanged. **Keep it here** stops the notice; the Overview row stays. Take a
  backup first (System › Backup & export or the dashboard's Backup) as with any change.
 - **Your own folder:** set `PROMPT_MANAGER_DATA_DIR` to an absolute folder outside
  ComfyUI's output, input and temp folders and your image sources. If it is empty and a
  node-folder library exists, that library stays in use until you move it there.
 - **An explicit `database.default_path`** in `config.json` and `PROMPT_MANAGER_CONFIG_PATH`
  keep working as before (described below); such a library is not offered a move.
 ### Thumbnails
 Previews in `output/thumbnails` are written as **WebP, quality 80, 512 px** by default
 (System › Appearance › Thumbnails sets the format, quality and size limit for every
 preview). Previews from older versions keep their old format (usually PNG) until you
 run **Convert outdated** (System › Overview or Appearance): it rewrites each one from
 its original and then removes the old file. Nothing converts on its own.
 **JPEG XL** (smaller still) can be chosen once the server has an encoder. Install one
 of these, then restart ComfyUI:
 - `pip install pillow-jxl-plugin` in ComfyUI's Python environment (simplest), or
 - an `ffmpeg` on PATH built with `libjxl`, or
 - `cjxl` from [libjxl](https://github.com/libjxl/libjxl) on PATH.
 Browsers that cannot show JPEG XL see the original images in the grid instead.
 ### Database Settings
-You can customize the database path by modifying the configuration:
+You can customize the database path by creating a `config.json` file in the library's
 folder (the node folder for a library that was not moved):
-```python
+```json
-# In py/config.py
+{
-DATABASE_PATH = "custom_path/prompts.db"
+  "database": {
    "default_path": "/path/to/your/prompts.db"
  }
 }
 ```
 Relative paths are resolved against the extension directory. Absolute paths are used as-is, which is recommended for Docker deployments where the working directory may not persist:
 ```json
 {
  "database": {
    "default_path": "/data/custom_nodes/ComfyUI_PromptManager/prompts.db"
  }
 }
 ```
 ### Database Path Compatibility (#211)
 PromptManager now honors the `database.default_path` setting in `config.json`
 when loaded as a ComfyUI custom node. Relative paths resolve against the extension
 directory, not ComfyUI's working directory; absolute paths are used as-is. Earlier
 versions could ignore this setting and use `<extension>/prompts.db` instead.
 Honoring a different configured path can therefore show an empty or different
 library. PromptManager does not automatically move or merge database files.
 Before upgrading or changing this setting:
 1. Identify the database your current installation actually uses and confirm its
   library contents. Check both the configured path and the prior fallback
   `<extension>/prompts.db`; keep track of which existing file you intend to use.
 2. Preserve a consistent backup of the active database with the dashboard's
   Backup/Restore backup action or SQLite's online backup API. Do not copy only
   the main file of a live SQLite database: committed data may still be in its
   `-wal` file. For a filesystem copy, stop ComfyUI cleanly first and preserve any
   remaining `-wal` file together with the main database.
 3. Set `database.default_path` to the existing intended database file. An absolute
   path makes the selection explicit; do not move or overwrite either database
   while choosing the path.
 4. Restart ComfyUI, then verify that the expected library appears and the active
   database path resolves to the file you selected. Keep the backup until both
   checks succeed.
 ### Gallery & Monitoring Settings
 Configure the automatic image detection and gallery system:
@@ -617,11 +875,15 @@ Configure the web interface behavior:
 # Web UI settings (PromptManagerConfig class)
 RESULT_TIMEOUT = 5  # Seconds to auto-hide results in ComfyUI node
 WEBUI_DISPLAY_MODE = 'newtab'  # 'popup' or 'newtab' for Web UI button
 INFINITE_SCROLL = False  # Dashboard list loads the next page on scroll by default
 SHOW_TEST_BUTTON = False  # Show API test button in node UI
 ```
 All of these are editable in the dashboard Settings modal and saved to `config.json` (which is private to the installation and holds the CivitAI key, so it is never committed).
 ### Performance Tuning
 - **Worker threads** (Settings → Performance): threads used by the image rescan, thumbnail generation and duplicate detection. Default is half the detected cores, maximum is the core count. Pillow releases the GIL while decoding, so the scan and thumbnails scale almost linearly; keep a few cores free while ComfyUI is generating.
 - The database automatically creates indexes for optimal search performance
 - Regular `VACUUM` operations keep the database optimized
 - Consider backing up the database periodically
@@ -656,7 +918,7 @@ print(f"Average rating: {info['average_rating']}")
 # Create backup
 db.model.backup_database("backup_prompts.db")
-# The database file can be copied directly for backup
+# The online backup includes committed data still in the WAL.
 ```
 ## Development
@@ -664,10 +926,15 @@ db.model.backup_database("backup_prompts.db")
 ### Running Tests
 ```bash
-cd KikoTextEncode
+cd ComfyUI_PromptManager
-python -m pytest tests/ -v
+pip install -r requirements-dev.txt
 python -m tests.run_suite                      # Python (what CI runs: unittest discovery plus a hang watchdog)
 node --test tests/js/*.test.js                 # frontend pure modules
 python -m coverage run -m tests.run_suite && python -m coverage combine && python -m coverage report
 ```
 CI runs the suite on Linux, Windows and macOS for Python 3.12 to 3.14 with an 85% coverage gate. `python -m unittest discover -s tests` still works; see `TESTING.md` ("Time limits and hangs") for the watchdog. Tests that delete a database file first check that nothing in the process still holds it open (`tests/open_handles.py`), so a connection leak fails on Linux the way it would on Windows.
 ### Code Style
 The project follows PEP 8 guidelines with:
@@ -709,7 +976,7 @@ The project follows PEP 8 guidelines with:
 For debugging, you can enable verbose logging in the node:
 ```python
-# Add to kiko_text_encode.py
+# Add to prompt_manager.py
 import logging
 logging.basicConfig(level=logging.DEBUG)
 ```
@@ -726,15 +993,19 @@ MIT License - see LICENSE file for details.
 ## Roadmap
-### Completed in v3.0.0
+### Recently Completed
- **✅ PNG Metadata Analysis**: Complete ComfyUI workflow extraction from images
+- **✅ Security & Reliability Hardening (3.3.0)**: validated gallery roots, contained file serving, safe backups and restores, background scans, 85%+ test coverage on all three operating systems
- **✅ Standalone Metadata Viewer**: Dedicated tool for analyzing any ComfyUI image
+- **✅ Worker Threads**: scan, thumbnails and duplicate detection run on a configurable thread pool
- **✅ Advanced Admin Dashboard**: Comprehensive management interface with modern UI
+- **✅ Tag Autocomplete & Infinite Scrolling**: faster browsing in the dashboard and gallery
- **✅ Integrated Metadata Panel**: Real-time workflow analysis in image viewer
+- **✅ Queue-Time Run Counting (3.2.4)**: run counts and image linking follow the ComfyUI queue
- **✅ Bulk Image Scanning**: Mass import of existing ComfyUI images
+- **✅ LoRA Manager Integration**: Import LoRA metadata, trigger words, and preview images
- **✅ System Management Tools**: Backup, restore, diagnostics, and maintenance
+- **✅ Folder Filter**: Browse and filter prompts by output subdirectory
- **✅ Enhanced Error Handling**: Robust PNG parsing with NaN value cleaning
+- **✅ Multi-Directory Gallery**: Scan multiple output directories simultaneously
 - **✅ WD14 Tagger**: Fast ONNX-based auto-tagging with Danbooru tags
 - **✅ Tailwind v4 Migration**: ComfyUI theme token system for consistent styling
 - **✅ Tag Management Page**: Dedicated page for tag search, rename, merge, and delete
 - **✅ Junction Table Tags**: Normalized tag storage with proper foreign keys
 ### Planned Features
@@ -742,20 +1013,117 @@ MIT License - see LICENSE file for details.
 - **🤝 Collaboration**: Share prompt collections with other users
 - **🧠 AI Suggestions**: Recommend similar prompts based on metadata analysis
 - **📈 Advanced Analytics**: Detailed usage statistics and trends with workflow insights
 - **🔌 Plugin System**: Support for third-party extensions and custom analyzers
 - **🎨 Enhanced Batch Processing**: Advanced bulk operations with metadata editing
 - **🔄 Workflow Templates**: Save and reuse common workflow patterns
 - **📊 Visual Analytics**: Charts and graphs for prompt effectiveness analysis
 ### Integration Ideas
 - **Auto-tagging**: Use AI to automatically categorize prompts
 - **Workflow linking**: Connect prompts to specific workflow templates
 - **Image analysis**: Analyze generated images to improve suggestions
 - **Version control**: Track prompt iterations and effectiveness
 ## Changelog
 ### Unreleased (Library location)
 - **📁 Library outside the node folder**: new installs keep `prompts.db`, `config.json`, the database backups and the logs in `ComfyUI/user/__promptmanager/`, where an update or reinstall of the node cannot delete them and ComfyUI's `/userdata` API does not reach. An existing library stays in the node folder until you press **Move library** (a verified copy, then a live switch; the old files are renamed `*.moved-to-__promptmanager`, never deleted) or **Keep it here**. `PROMPT_MANAGER_DATA_DIR` names your own folder. After moving, restart ComfyUI once so the logs follow. Going back to an older version after a move: an older version does not look in `user/__promptmanager`, so stop ComfyUI and copy `user/__promptmanager/prompts.db` (with its `-wal` file, if one is left) and `config.json` into the node folder; the `*.moved-to-__promptmanager` files are the library as it was at the move, without later changes.
 ### Unreleased (Workflows stored once)
 - **🗜️ Smaller database**: each distinct workflow and prompt graph is stored once, compressed, instead of again for every image. Images recorded by this version use it from the first start; the maintenance job "Store each workflow once" rewrites the existing records and then reclaims the space. A 20,000-image test library went from 1.4 GB to 60 MB. The job always takes a verified backup first: the first time, `prompts.db.pre-workflow-storage_<date time>` beside the database, kept until you delete it; on later runs, a normal pooled safety backup. No image or video file is opened or changed.
 - **⚠️ Downgrading**: images recorded by this version (and, after the job, all images) are stored in the new form. An older Prompt Manager version still opens and even restores such a database, but shows those images without workflows or prompt graphs until you update again. To go back to an older version, stop ComfyUI, delete `prompts.db-wal` and `prompts.db-shm`, and copy the **earliest** `prompts.db.pre-workflow-storage_<date time>` over `prompts.db`, or, if the job never ran, a backup taken before this update (records added since that backup are lost). Otherwise stay on this version.
 ### v3.3.1 (Proxy Routing Fix)
 - **🌐 Proxy support**: the dashboard, gallery, tag page and metadata viewer now work when ComfyUI is reached through a proxy mount such as SwarmUI's `/ComfyBackendDirect/`. Scripts, images, API calls, progress streams and gallery folder filters stay under the mount, and the canvas node's buttons use ComfyUI's `fileURL()`. Direct ComfyUI URLs are unchanged. Thanks to @ghosttigerllc-bit ([#164](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/164))
 - **📦 Dependencies**: the runtime minimums stay at `Pillow>=8.0.0` and `watchdog>=2.1.0`, so updating Prompt Manager never forces upgrades in the shared ComfyUI environment
 ### v3.3.0 (Hardening, Background Jobs & Worker Threads)
 - **🔒 Security**: gallery roots restricted to ComfyUI directories and validated wherever they are read; media-only, fail-closed image serving; Windows path escape in static routes closed; quote-safe HTML escaping with inline handlers moved to data attributes; the CivitAI key is never returned and only sent to HTTPS CivitAI hosts; auto-tag paths contained; side-effect routes are POST-only; backups include the WAL and restores are size-capped, integrity-checked and applied through SQLite's online backup API
 - **🧵 Worker Threads**: a Settings slider (default half the cores) used by the image rescan, thumbnail generation and duplicate detection. Measured: a 1 GB sample scanned in 2.5 s instead of 18 s at 8 threads
 - **🔁 Background Jobs**: the image rescan and the duplicate scan run on the server, stream progress, survive a closed tab, minimize to a progress pill and re-attach after a reload
 - **🎬 Video Prompts**: the scan reads the prompt ComfyUI's video savers embed in MP4/MOV/WebM containers (needs `ffprobe`)
 - **🖼️ Thumbnails**: written atomically and served under versioned URLs so a regenerated thumbnail never shows a stale cached copy; the per-run cap applies only to thumbnails that still need generating and the gallery says how many remain
 - **🏷️ Tag Autocomplete**: the dashboard tag box suggests tags, narrowed to those that co-occur with the tags already typed
 - **♾️ Infinite Scrolling**: optional in the gallery and the dashboard list (saved default plus a session toggle)
 - **⚙️ Gallery Settings** now apply at load; grid columns, media info and the Rescan button work as labelled
 - **🐛 Correctness**: run counts increment when ComfyUI dequeues a prompt; negative prompts are never traced; per-thread SQLite connections and idempotent migrations (an old rebuild could delete every image row); `truncate_logs` no longer rolls the log over; the ComfyUI canvas loads only the canvas extension; Windows console banners are ASCII-safe; subfolders use `/` on every platform
 - **🧪 Tests & CI**: 1,460 Python and 142 JavaScript tests (from 510 and 26), coverage 96% with an 85% gate, CI on Linux, Windows and macOS for Python 3.10–3.12, flake8 clean
 - **⚠️ API changes**: image paths in responses are relative to the ComfyUI tree (use `url`); `autotag/download`, `autotag/start` and the thumbnail progress stream are POST-only; `lora/status` returns `has_civitai_api_key` instead of the key; `/search` is paginated
 ### v3.2.4 (Recently Used Sorting & Queue-Time Run Counts)
 - **🕒 Recently Used Sort**: sort the dashboard by last use, most used or rating, server-side across pages
 - **🔢 Run Counts**: `run_count` and `last_used_at` are recorded through the ComfyUI queue hook, so cached nodes still count
 - **🔗 Positive-Prompt Image Linking**: generated images link to the positive prompt traced through the queued graph
 ### v3.2.1 (LoRA Manager Integration)
 - **🔗 LoRA Manager Integration**: Import LoRA metadata, trigger words, and CivitAI example images from [ComfyUI-Lora-Manager](https://github.com/willmiao/ComfyUI-Lora-Manager) into your prompt database
 - **💉 Auto-Inject Trigger Words**: Optionally append LoRA trigger words when `<lora:name:weight>` is detected in prompts during encoding
 - **🔑 CivitAI API Key Support**: Authenticate with CivitAI to download NSFW preview images
 - **📥 Import Progress Modal**: Real-time SSE streaming progress during LoRA import with per-model status
 > **Note:** LoRA Manager support is a WIP — this was a highly requested feature. Please [open issues](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues) for any bugs or feature requests.
 ### v3.2.0 (Folder Filter & QoL Improvements)
 - **📁 Folder Filter**: New folder dropdown in the search panel to filter prompts by output subdirectory
 - **📂 Multi-Directory Gallery Scan**: Configure multiple output directories and browse them all in one gallery
 - **🖼️ Filmstrip Prompt Display**: Show prompt text and copy button in the filmstrip image viewer
 - **🖱️ Click-to-Close Viewer**: Click outside the image viewer to close it
 - **🐛 Bug Fixes**: Fixed database path config, UnboundLocalError in text inputs, node caching skip, diagnostics singleton lifecycle
 ### v3.1.0 (WD14 Tagger, Tailwind v4 & Major Refactors)
 - **🏷️ WD14 Tagger Support**: Added WD14 SwinV2 and WD14 ViT as new auto-tag model options — fast ONNX-based classifiers (~400MB) that output Danbooru tags with confidence scores, no prompt needed
 - **🎚️ Threshold Controls**: Adjustable general tag (default 0.35) and character tag (default 0.85) confidence thresholds for WD14 models via UI sliders
 - **🎨 Tailwind v4 Migration**: Complete frontend redesign with ComfyUI theme token system (`--pm-*` design tokens) that adapts to ComfyUI's color palette
 - **🏗️ API Refactor**: Split monolithic `api.py` into domain-specific route modules for better maintainability
 - **🗃️ Junction Table Tags**: Migrated tag storage from JSON column to normalized junction tables with proper foreign keys
 - **📄 Tags Page**: Dedicated tag management page with search, rename, merge, and delete operations
 - **🔍 Load More Button**: Progressive prompt loading instead of traditional pagination
 - **🔒 Security Hardening**: Resolved 15 GitHub code scanning alerts, prevented stack trace exposure in SSE errors
 - **🧪 Test Coverage**: Added comprehensive unit tests for validators, hashing, metadata, config, and prompt tracker; API integration tests; database tests with CI pipeline
 - **⚙️ CI/CD Improvements**: Added code quality scanning, security scanning, Dependabot, and pre-commit hooks with Black formatting
 - **🐛 Bug Fixes**: Fixed custom image scan directory config (#108), gallery crash on thumbnail linking, broken imports after API refactor, widget value sync issues, and prepend/append metadata tag filtering in auto-tagger
 ### v3.0.26 (Top Bar Button)
 - **🔘 Top Bar PM Button**: Added a "PM" button to the ComfyUI top bar for quick access to the admin dashboard
 - **🚀 No Node Required**: Open PromptManager without having to add the node to your workflow first
 - **🎨 Teal Button Design**: Distinctive teal-colored button positioned before the settings gear
 - **🔄 Frontend Compatibility**: Works with both new ComfyUI frontend (v1.3.1+) and legacy versions
 ### v3.0.25 (Film Strip & QoL Improvements)
 - **🎞️ Film Strip Preview**: Backported from V2 - thumbnail strip under each prompt showing linked images
 - **🖼️ Quick Image Viewer**: Click any filmstrip thumbnail to open full-size image viewer
 - **📸 Smart Thumbnails**: Automatically uses generated thumbnails for faster loading, falls back to full images
 - **⬆️ Auto-Scroll on Pagination**: Page navigation now smoothly scrolls to top after loading
 ### v3.0.24 (Manual Prompt Creation)
 - **➕ Add Prompt Button**: New "Add Prompt" button in the admin dashboard Bulk Actions bar
 - **📝 Direct Prompt Creation**: Create prompts directly from the UI without generating images
 - **🛡️ Protection from Pruning**: New prompts are automatically protected from orphan cleanup by default
 - **🏷️ Full Metadata Support**: Set category, tags, rating, and notes when creating prompts
 - **🔒 Orphan Protection Toggle**: Option to protect or unprotect prompts from maintenance cleanup
 ### v3.0.23 (AI AutoTag Release)
 - **🏷️ AI AutoTag**: Automatically tag your entire image collection using JoyCaption vision models
 - **🤖 Dual Model Support**: Choose between JoyCaption Beta One FP16 (high quality) or GGUF FP8 (lower VRAM)
 - **📋 Two Tagging Modes**: Batch mode for bulk tagging, Review mode for one-by-one approval
 - **⚙️ Smart Tag Handling**: Skip already-tagged images or re-tag all - respects your existing work
 - **✏️ Customizable Prompts**: Adjust the system prompt to match your collection style and tag vocabulary
 - **🔄 Real-time Progress**: SSE-based streaming progress updates during batch operations
 - **📦 Tag Accordion UI**: Collapsible tag display shows first row with expandable section for large tag sets
 - **🎯 Review Confirmation**: Prompts before overwriting existing tags in review mode
 ### v3.0.10 (Batch Processing Node)
 - **📋 PromptSearchList Node**: New node that searches the prompt database and outputs results as a list
 - **🔗 Batch Processing Support**: Uses `OUTPUT_IS_LIST` for native ComfyUI batch processing compatibility
 - **🔍 Full Search Capabilities**: Search by text, category, tags, minimum rating, and result limit
 - **⚡ Lightweight Operation**: Read-only search with no database writes
 - **🎯 Workflow Integration**: Connect directly to batch processing nodes like String OutputList
 ### v3.0.9 (Output-Wide Gallery & Video Support)
 - **📁 Output-Wide Gallery Mode**: Complete gallery redesign to show ALL content from ComfyUI output directory
@@ -0,0 +1,64 @@
 # 🔐 Security Policy
 ## Supported Versions
 Security updates are provided for the actively maintained versions of
 **ComfyUI_PromptManager**.
 | Version         | Supported |
 |-----------------|-----------|
 | `main` / latest | ✅        |
 | Older releases  | ❌        |
 Users are encouraged to stay up to date with the latest version.
 ---
 ## Reporting a Vulnerability
 For most security-related issues, **please open a GitHub issue** in this
 repository.
 When opening an issue:
 - Clearly describe the problem
 - Include steps to reproduce
 - Mention the affected version or commit
 - Avoid posting secrets, tokens, or private data
 ### Sensitive Issues
 If you believe the issue:
 - Could enable remote code execution
 - Exposes credentials or private data
 - Is actively exploitable in the wild
 Please use **GitHub Security Advisories** instead:
 - Go to the **Security** tab
 - Click **Report a vulnerability**
 ---
 ## Response Expectations
 - Issues are typically reviewed within **a few days**
 - Valid issues will be labeled and tracked publicly
 - Fixes may be discussed openly and merged transparently
 ---
 ## Scope
 This policy applies to:
 - Code contained in this repository
 - Default configurations and documented usage
 It does not cover:
 - Third-party nodes, models, or extensions
 - User workflows, prompts, or local environment issues
 ---
 ## Thank You
 Community reports and transparency help keep the project healthy.
 Thanks for contributing responsibly.
@@ -1,57 +1,81 @@
 """
-ComfyUI_PromptManager: A comprehensive ComfyUI custom node that extends text encoding 
+ComfyUI_PromptManager: A comprehensive ComfyUI custom node that extends text encoding
-with persistent prompt storage, advanced search capabilities, and an automatic image 
+with persistent prompt storage, advanced search capabilities, and an automatic image
 gallery system.
-This module provides two main node types:
+This module provides prompt node types:
 - PromptManager: CLIP encoding node that outputs CONDITIONING
- PromptManagerText: Text-only node that outputs STRING with prepend/append functionality
+- PromptManagerFocused: Editor-first layout using the same encoding contract
 - PromptManagerSplit: Side-by-side editor and library using the same contract
 - PromptManagerText: Text-only node that outputs STRING with prepend/append
  functionality
-Both nodes share the same database backend for persistent prompt storage and include
+The nodes share the same database backend for persistent prompt storage and include
 an automatic image gallery system that monitors ComfyUI output directories and links
 generated images to their source prompts.
 Features:
 - SQLite-based persistent prompt storage with deduplication
- Advanced search and categorization system  
+- Advanced search and categorization system
 - Real-time image gallery with metadata extraction
 - Web-based admin dashboard for prompt management
 - Comprehensive logging and diagnostics
 """
-import re
+import atexit
 import threading
 import time
 from pathlib import Path
 def get_version():
-    """Parse version from pyproject.toml"""
+    """Parse version from pyproject.toml (``utils/version.py``)."""
    from .utils.version import project_version
    try:
-        pyproject_path = Path(__file__).parent / "pyproject.toml"
+        root = Path(__file__).parent
        if pyproject_path.exists():
            content = pyproject_path.read_text()
            match = re.search(r'version\s*=\s*["\']([^"\']+)["\']', content)
            if match:
                return match.group(1)
    except Exception:
-        pass
+        return "unknown"
-    return "unknown"
+    return project_version(root)
-from .prompt_manager import PromptManager
+from .prompt_manager import (  # noqa: E402
-from .prompt_manager_text import PromptManagerText
+    PromptManager,
    PromptManagerFocused,
    PromptManagerSplit,
 )
 from .prompt_manager_pair import PromptManagerPair  # noqa: E402
 from .prompt_manager_text import PromptManagerText  # noqa: E402
 from .prompt_search_list import PromptSearchList  # noqa: E402
 NODE_CLASS_MAPPINGS = {
    "PromptManager": PromptManager,
    "PromptManagerFocused": PromptManagerFocused,
    "PromptManagerSplit": PromptManagerSplit,
    "PromptManagerPair": PromptManagerPair,
    "PromptManagerText": PromptManagerText,
    "PromptSearchList": PromptSearchList,
 }
 NODE_DISPLAY_NAME_MAPPINGS = {
-    "PromptManager": "Prompt Manager",
+    "PromptManager": "Prompt Manager (Classic)",
    "PromptManagerFocused": "Prompt Manager (Focused)",
    "PromptManagerSplit": "Prompt Manager (Split)",
    "PromptManagerPair": "Prompt Manager (+/-)",
    "PromptManagerText": "Prompt Manager Text",
    "PromptSearchList": "Prompt Search List",
 }
-# Define path to web directory for UI components
+# Canvas extension only. ComfyUI loads every .js under this directory into the
-WEB_DIRECTORY = "web"
+# canvas page, so the admin/gallery bundles and vendored libraries under web/
 # are served by our own routes instead (see py/api).
 WEB_DIRECTORY = "web/comfy"
 # Parts the package teardown stops (``_stop_background_work``). Each stays None
 # when its start-up block below fails, and the teardown skips it.
 api = None
 _global_prompt_tracker = None
 _global_image_monitor = None
 # Add API routes (same pattern as ComfyUI_Assets)
 try:
@@ -68,6 +92,25 @@ try:
    routes = config.routes
    api = PromptManagerAPI()
    api.add_routes(routes)
    # Temp files of preview writes a crash interrupted (#317), off the
    # import path: the walk may cover a large thumbnails folder.
    threading.Thread(
        target=api.sweep_thumbnail_temps,
        name="PromptManagerPreviewTempSweep",
        daemon=True,
    ).start()
    host_app = getattr(config.server_instance, "app", None)
    if host_app is not None and hasattr(host_app, "on_cleanup"):
        host_app.on_cleanup.append(api.cleanup_history_reads)
        host_app.on_cleanup.append(api.cleanup_annotation_captures)
        host_app.on_cleanup.append(api.cleanup_gallery_listings)
    if host_app is not None and hasattr(host_app, "on_startup"):
        # Build each output root's Images listing once the server's loop runs,
        # so the first open does not wait for the walk (#348).
        host_app.on_startup.append(api.warm_gallery_listings)
    if host_app is not None and hasattr(host_app, "on_shutdown"):
        # Open Needs review change streams end before the server waits on them.
        host_app.on_shutdown.append(api.close_review_events)
 except Exception as e:
    # Log to internal logging system without console spam
@@ -76,7 +119,257 @@ except Exception as e:
        logger = get_logger("prompt_manager.init")
        logger.error(f"Failed to register API routes: {e}")
-    except:
+    except Exception:
        pass
 # Shared database, set up with the monitoring system below. Stays None when that
 # fails so the queue hook can report the database as unavailable instead of crashing.
 _global_db = None
 # Start image monitoring globally at module import time
 # This ensures images are linked regardless of which node is used in the workflow
 try:
    from .database.operations import PromptDatabase
    from .utils.image_monitor import get_image_monitor
    from .utils.prompt_tracker import get_prompt_tracker
    from .utils.logging_config import get_logger
    _init_logger = get_logger("prompt_manager.init")
    _init_logger.info("Starting global image monitoring system...")
    # Initialize database and get singleton instances
    _global_db = PromptDatabase()
    _global_prompt_tracker = get_prompt_tracker(_global_db)
    _global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
    # The Needs review pass for ambiguous new outputs runs through the API's
    # capture store (the one the scan and the Tag dialog use): a linked output
    # is never queued and its pending items leave the queue. The package
    # teardown detaches it before that store is closed at shutdown.
    if api is not None:
        _global_image_monitor.set_review_recorder(api.record_review_file)
        # Every settled output's preview is queued for the API's background
        # worker (#249): the workspace never asks to generate thumbnails.
        _global_image_monitor.set_thumbnail_queue(api.queue_capture_thumbnail)
        # Every settled output walks its Images listing again (#348): the
        # API hops from the monitor's thread onto the server's loop.
        _global_image_monitor.set_output_listener(api.notify_output_settled)
    # Start monitoring
    _global_image_monitor.start_monitoring()
    _init_logger.info("Global image monitoring system started successfully")
 except Exception as e:
    try:
        from .utils.logging_config import get_logger
        _init_logger = get_logger("prompt_manager.init")
        _init_logger.error(f"Failed to start image monitoring: {e}")
    except Exception:
        print(f"[ComfyUI-PromptManager] Warning: Failed to start image monitoring: {e}")
 def _db_factory():
    """The shared database for the queue hook; called lazily per request."""
    if _global_db is None:
        raise RuntimeError("PromptManager database unavailable")
    return _global_db
 # Count prompt runs when ComfyUI dequeues workflows (it skips unchanged nodes, so
 # node execution alone misses re-runs). Separate from monitoring so either can fail.
 try:
    from .py.config import server_instance
    from .utils.usage_tracking import register_queue_hook
    register_queue_hook(server_instance, _db_factory)
 except Exception as e:
    try:
        from .utils.logging_config import get_logger
        get_logger("prompt_manager.init").error(f"Failed to register usage hook: {e}")
    except Exception:
        import logging
        logging.getLogger("prompt_manager.init").error(
            f"Failed to register usage hook: {e}"
        )
 def _history_db_factory() -> "PromptDatabase":
    """Initialize history on its worker even if monitoring initialization failed."""
    if _global_db is not None:
        return _global_db
    from .database.operations import PromptDatabase
    return PromptDatabase()
 _history_registration = None
 try:
    from .py.config import server_instance
    from .utils.execution_history import register_history_hook
    _history_registration = register_history_hook(server_instance, _history_db_factory)
 except Exception:
    try:
        from .utils.logging_config import get_logger
        get_logger("prompt_manager.init").error("Failed to register execution history")
    except Exception:
        import logging
        logging.getLogger("prompt_manager.init").error(
            "Failed to register execution history"
        )
 # ── Package teardown (#301, #302) ────────────────────────────────────────────
 # ComfyUI never runs aiohttp's cleanup hooks (it never calls
 # ``runner.cleanup()``), so ``atexit`` is the live host's path: Ctrl+C, and a
 # normal exit. It is registered after the history hook, so at exit (newest
 # first) it runs before the history dispose. Hosts that do run ``on_cleanup``
 # (our test servers) run it there, and the tests' package unload calls it.
 TEARDOWN_SECONDS = 10.0
 _clock = time.monotonic
 _background_work_lock = threading.Lock()
 _background_work_result = None
 def _teardown_step(name, stop, logger):
    """Run one teardown step; a failure (even Ctrl+C) is logged and is False."""
    try:
        return bool(stop())
    except (Exception, KeyboardInterrupt):
        logger.error(f"Background work: stopping {name} failed", exc_info=True)
        return False
 def _teardown_logger():
    """PromptManager's init logger; the standard one if that cannot load."""
    try:
        from .utils.logging_config import get_logger
        return get_logger("prompt_manager.init")
    except Exception:
        import logging
        return logging.getLogger("prompt_manager.init")
 def _stop_background_work():
    """Stop the image monitor, the preview worker and the prompt cleanup
    thread against one TEARDOWN_SECONDS deadline, once.
    Returns ``{"monitor", "thumbnails", "tracker"}`` booleans (True: it
    stopped), leaving out parts that never started. Later calls return a
    copy of the first result. A Ctrl+C inside a step is logged and the
    step is False; one between steps is re-raised after the result (parts
    not reached are False) is stored and the ``atexit`` entry removed.
    """
    global _background_work_result
    with _background_work_lock:
        if _background_work_result is not None:
            return dict(_background_work_result)
        monitor, thumbnails, tracker = (
            _global_image_monitor,
            api,
            _global_prompt_tracker,
        )
        # A part whose global is set has started; False until its step says
        # otherwise, so an interrupt between steps leaves an honest result.
        started = (
            ("monitor", monitor),
            ("thumbnails", thumbnails),
            ("tracker", tracker),
        )
        result = {name: False for name, part in started if part is not None}
        try:
            logger = _teardown_logger()
            began = _clock()
            deadline = began + TEARDOWN_SECONDS
            def left(own_bound):
                return max(0.0, min(own_bound, deadline - _clock()))
            if monitor is not None:
                def stop_monitor():
                    from .utils.image_monitor import OBSERVER_JOIN_SECONDS
                    # Callbacks first: an image in hand queues no preview into a
                    # closing preview worker. The worker may have read the queue
                    # callback already, so the API refuses new previews too (at
                    # exit, Python 3.12+ refuses to start their worker thread).
                    if thumbnails is not None:
                        thumbnails.refuse_auto_thumbnails()
                    monitor.set_review_recorder(None)
                    monitor.set_thumbnail_queue(None)
                    monitor.set_output_listener(None)
                    return monitor.stop_monitoring(left(OBSERVER_JOIN_SECONDS))
                result["monitor"] = _teardown_step(
                    "the image monitor", stop_monitor, logger
                )
            if thumbnails is not None:
                def close_previews():
                    from .py.api.thumbnail_auto import CLOSE_WAIT_SECONDS
                    return thumbnails.close_auto_thumbnails(
                        final=True, timeout=left(CLOSE_WAIT_SECONDS)
                    )
                result["thumbnails"] = _teardown_step(
                    "the preview worker", close_previews, logger
                )
            if tracker is not None:
                def stop_tracker():
                    from .utils.prompt_tracker import TRACKER_JOIN_SECONDS
                    return tracker.shutdown(left(TRACKER_JOIN_SECONDS))
                result["tracker"] = _teardown_step(
                    "the prompt cleanup thread", stop_tracker, logger
                )
            elapsed = _clock() - began
            failed = [name for name, ok in result.items() if not ok]
            if failed:
                logger.warning(
                    f"Background work not stopped in time ({elapsed:.1f} s): "
                    + ", ".join(failed)
                )
            else:
                parts = " ".join(f"{name}=ok" for name in result)
                logger.info(f"Background work stopped in {elapsed:.1f} s: {parts}")
        finally:
            _background_work_result = result
            atexit.unregister(_stop_background_work)
        return dict(result)
 async def _stop_background_work_on_cleanup(app):
    """aiohttp ``on_cleanup``: no request is served then, so it runs inline."""
    _stop_background_work()
 try:
    atexit.register(_stop_background_work)
    from .py.config import server_instance as _teardown_server
    _teardown_host = getattr(_teardown_server, "app", None)
    if _teardown_host is not None and hasattr(_teardown_host, "on_cleanup"):
        _teardown_host.on_cleanup.insert(0, _stop_background_work_on_cleanup)
 except Exception as e:
    try:
        from .utils.logging_config import get_logger
        get_logger("prompt_manager.init").error(
            f"Failed to register the background work teardown: {e}"
        )
    except Exception:
        pass
 __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
@@ -85,6 +378,8 @@ __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
 print()
 print(f"\033[94m[ComfyUI-PromptManager] Version:\033[0m {get_version()}")
 for node_key, display_name in NODE_DISPLAY_NAME_MAPPINGS.items():
-    print(f"🫶 \033[94mLoaded:\033[0m {display_name}")
+    # ASCII only: a Windows console using a legacy code page cannot print emoji,
    # and a UnicodeEncodeError here would abort loading the whole node.
    print(f"  \033[94mLoaded:\033[0m {display_name}")
 print(f"\033[94mTotal: {len(NODE_CLASS_MAPPINGS)} tools loaded\033[0m")
 print()
@@ -8,4 +8,4 @@ search capabilities, metadata management, and image tracking functionality.
 from .operations import PromptDatabase
 from .models import PromptModel
-__all__ = ["PromptDatabase", "PromptModel"]
+__all__ = ["PromptDatabase", "PromptModel"]
@@ -0,0 +1,290 @@
 """Image annotation schema group (#249): file locators, byte records, image tags.
 The group installs like ``image_favorites`` and ``saved_views``: optional in old
 backups, required once a database is current. It owns
 - ``media_annotation_metadata``: one row, the group's schema version and the
  image index revision;
 - ``media_annotation_files``: one private canonical locator per physical file
  (``normcase(join(realpath(root), relative))``, server-side only);
 - ``media_annotation_bytes``: one record per observed whole-file SHA-256 of a
  file, with the size and ``mtime_ns`` verified when those bytes were last seen
  and the tag set's revision;
 - ``media_image_tags``: the ordered image tag set of one byte record, joined
  into the global ``tags`` registry so vocabulary maintenance carries image
  links like prompt and pair links.
 Installation never backfills anything from prompt or pair tags and never
 touches the filesystem. Validation reads only: the catalog objects are compared
 after whitespace compaction, columns, foreign keys and indexes through the
 PRAGMAs, and persisted values in single bounded SQL passes.
 """
 import re
 import sqlite3
 from typing import Optional
 from .image_favorites import MAX_KEY_LENGTH
 SCHEMA_VERSION = 1
 METADATA_TABLE = "media_annotation_metadata"
 FILES_TABLE = "media_annotation_files"
 BYTES_TABLE = "media_annotation_bytes"
 IMAGE_TAGS_TABLE = "media_image_tags"
 TABLES = (METADATA_TABLE, FILES_TABLE, BYTES_TABLE, IMAGE_TAGS_TABLE)
 MAX_LOCATOR_LENGTH = MAX_KEY_LENGTH
 _TIMESTAMP_BYTES = 64
 _TABLE_DDL = {
    METADATA_TABLE: (
        f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
        "CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
        f"CHECK(schema_version={SCHEMA_VERSION}), "
        "image_index_revision INTEGER NOT NULL DEFAULT 0 "
        "CHECK(image_index_revision>=0))"
    ),
    FILES_TABLE: (
        f"CREATE TABLE {FILES_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
        "locator TEXT NOT NULL UNIQUE, created_at TEXT NOT NULL)"
    ),
    BYTES_TABLE: (
        f"CREATE TABLE {BYTES_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
        f"file_id TEXT NOT NULL REFERENCES {FILES_TABLE}(id) ON DELETE CASCADE, "
        "sha256 TEXT NOT NULL CHECK(length(sha256)=64 AND sha256 NOT GLOB "
        "'*[^0-9a-f]*'), size INTEGER NOT NULL CHECK(size>=0), "
        "mtime_ns INTEGER NOT NULL, verified_at TEXT NOT NULL, "
        "first_seen_at TEXT NOT NULL, tag_revision INTEGER NOT NULL DEFAULT 0 "
        "CHECK(tag_revision>=0), UNIQUE(file_id, sha256))"
    ),
    IMAGE_TAGS_TABLE: (
        f"CREATE TABLE {IMAGE_TAGS_TABLE} (byte_id TEXT NOT NULL REFERENCES "
        f"{BYTES_TABLE}(id) ON DELETE CASCADE, tag_id INTEGER NOT NULL "
        "REFERENCES tags(id) ON DELETE CASCADE, position INTEGER NOT NULL "
        "CHECK(position>=0), PRIMARY KEY(byte_id, tag_id))"
    ),
 }
 _INDEX_DDL = {
    "idx_media_annotation_bytes_fingerprint": (
        "CREATE INDEX idx_media_annotation_bytes_fingerprint ON "
        f"{BYTES_TABLE}(file_id, size, mtime_ns)"
    ),
    "idx_media_image_tags_tag": (
        f"CREATE INDEX idx_media_image_tags_tag ON {IMAGE_TAGS_TABLE}(tag_id)"
    ),
    "idx_media_image_tags_position": (
        "CREATE UNIQUE INDEX idx_media_image_tags_position ON "
        f"{IMAGE_TAGS_TABLE}(byte_id, position)"
    ),
 }
 # SQLite names the indexes behind TEXT primary keys and UNIQUE constraints
 # ``sqlite_autoindex_<table>_<n>``; their catalog rows carry no SQL.
 _AUTOINDEXES = {FILES_TABLE: 2, BYTES_TABLE: 2, IMAGE_TAGS_TABLE: 1}
 # (name, declared type, not null, default, primary key position)
 _COLUMNS = {
    METADATA_TABLE: (
        ("singleton_id", "INTEGER", 0, None, 1),
        ("schema_version", "INTEGER", 1, None, 0),
        ("image_index_revision", "INTEGER", 1, "0", 0),
    ),
    FILES_TABLE: (
        ("id", "TEXT", 0, None, 1),
        ("locator", "TEXT", 1, None, 0),
        ("created_at", "TEXT", 1, None, 0),
    ),
    BYTES_TABLE: (
        ("id", "TEXT", 0, None, 1),
        ("file_id", "TEXT", 1, None, 0),
        ("sha256", "TEXT", 1, None, 0),
        ("size", "INTEGER", 1, None, 0),
        ("mtime_ns", "INTEGER", 1, None, 0),
        ("verified_at", "TEXT", 1, None, 0),
        ("first_seen_at", "TEXT", 1, None, 0),
        ("tag_revision", "INTEGER", 1, "0", 0),
    ),
    IMAGE_TAGS_TABLE: (
        ("byte_id", "TEXT", 1, None, 1),
        ("tag_id", "INTEGER", 1, None, 2),
        ("position", "INTEGER", 1, None, 0),
    ),
 }
 # (column, parent table, parent column, on delete)
 _FOREIGN_KEYS = {
    METADATA_TABLE: set(),
    FILES_TABLE: set(),
    BYTES_TABLE: {("file_id", FILES_TABLE, "id", "CASCADE")},
    IMAGE_TAGS_TABLE: {
        ("byte_id", BYTES_TABLE, "id", "CASCADE"),
        ("tag_id", "tags", "id", "CASCADE"),
    },
 }
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql).lower()
 def _uuid(column: str) -> str:
    """SQL: *column* is a lowercase hyphenated 36-character UUID string."""
    return (
        f"typeof({column}) = 'text' AND length({column}) = 36 "
        f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
        f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
        f"AND length(replace({column}, '-', '')) = 32 "
        f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
    )
 def _timestamp(column: str) -> str:
    return (
        f"typeof({column}) = 'text' "
        f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
    )
 def _non_negative(column: str) -> str:
    return f"typeof({column}) = 'integer' AND {column} >= 0"
 # Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
 _ROW_PREDICATES = {
    FILES_TABLE: " AND ".join(
        (
            _uuid("id"),
            "typeof(locator) = 'text'",
            # UTF-8 is at most 4 bytes per character; the decoded length of
            # the few candidates above the byte bound is checked in Python.
            f"length(CAST(locator AS BLOB)) <= {MAX_LOCATOR_LENGTH * 4}",
            _timestamp("created_at"),
        )
    ),
    BYTES_TABLE: " AND ".join(
        (
            _uuid("id"),
            _uuid("file_id"),
            "typeof(sha256) = 'text' AND length(sha256) = 64 "
            "AND sha256 NOT GLOB '*[^0-9a-f]*'",
            _non_negative("size"),
            "typeof(mtime_ns) = 'integer'",
            _timestamp("verified_at"),
            _timestamp("first_seen_at"),
            _non_negative("tag_revision"),
        )
    ),
    IMAGE_TAGS_TABLE: " AND ".join(
        (_uuid("byte_id"), "typeof(tag_id) = 'integer'", _non_negative("position"))
    ),
 }
 def _expected_catalog() -> set:
    objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
    objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
    objects |= {
        ("index", f"sqlite_autoindex_{table}_{n}", "")
        for table, count in _AUTOINDEXES.items()
        for n in range(1, count + 1)
    }
    return objects
 def _catalog(conn: sqlite3.Connection) -> list:
    names = TABLES + tuple(_INDEX_DDL)
    marks = ",".join("?" * len(names))
    table_marks = ",".join("?" * len(TABLES))
    return conn.execute(  # only "?" placeholders are interpolated
        "SELECT type,name,sql FROM sqlite_master "
        f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})",  # nosec B608
        names + TABLES,
    ).fetchall()
 def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
    for table, expected in _COLUMNS.items():
        columns = tuple(
            (row[1], row[2].upper(), row[3], row[4], row[5])
            for row in conn.execute(f"PRAGMA table_info({table})")
        )
        if columns != expected:
            return f"Image annotations table {table} has invalid columns"
        foreign_keys = {
            (row[3], row[2], row[4], row[6])
            for row in conn.execute(f"PRAGMA foreign_key_list({table})")
        }
        if foreign_keys != _FOREIGN_KEYS[table]:
            return f"Image annotations table {table} has invalid foreign keys"
        if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
            return f"Image annotations table {table} contains foreign key violations"
    return None
 def _value_error(conn: sqlite3.Connection) -> Optional[str]:
    # Only this module's constants are interpolated into the SQL below.
    rows = conn.execute(
        "SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' "  # nosec B608
        "AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
        f"AND schema_version = {SCHEMA_VERSION} "
        f"AND {_non_negative('image_index_revision')}) FROM {METADATA_TABLE}"
    ).fetchone()
    if rows[0] != 1 or rows[1] != 1:
        return "Image annotations metadata is invalid"
    for table, predicate in _ROW_PREDICATES.items():
        if conn.execute(
            f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"  # nosec B608
        ).fetchone():
            return f"Image annotations table {table} contains invalid values"
    # length() stops at an embedded NUL, so the SQL pass bounds bytes and the
    # few candidates over the character bound are measured decoded here.
    for (raw,) in conn.execute(
        f"SELECT CAST(locator AS BLOB) FROM {FILES_TABLE} "  # nosec B608
        f"WHERE length(CAST(locator AS BLOB)) > {MAX_LOCATOR_LENGTH}"
    ):
        if len(bytes(raw).decode("utf-8", "replace")) > MAX_LOCATOR_LENGTH:
            return f"Image annotations table {FILES_TABLE} contains invalid values"
    return None
 def annotation_schema_error(
    conn: sqlite3.Connection, *, allow_missing: bool = True
 ) -> Optional[str]:
    """Reject partial or unsupported annotation schemas and invalid rows.
    Reads only: nothing is ever repaired. A database without any of the
    group's objects is acceptable for backups that predate it unless
    ``allow_missing`` is false.
    """
    objects = _catalog(conn)
    if not objects:
        return None if allow_missing else "Missing image annotation schema"
    present = {name for kind, name, _sql in objects if kind == "table"}
    if present != set(TABLES):
        return "Incomplete image annotation schema"
    actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
    if actual != _expected_catalog():
        return "Image annotations has an invalid schema definition"
    return _structure_error(conn) or _value_error(conn)
 def migrate_annotation_state(conn: sqlite3.Connection) -> None:
    """Install the group atomically; validate before and after creating it."""
    conn.execute("SAVEPOINT media_annotation_schema")
    try:
        error = annotation_schema_error(conn)
        if error:
            raise sqlite3.DatabaseError(error)
        for sql in _TABLE_DDL.values():
            conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        for sql in _INDEX_DDL.values():
            conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
        conn.execute(
            f"INSERT OR IGNORE INTO {METADATA_TABLE} "
            "(singleton_id, schema_version, image_index_revision) VALUES (1, ?, 0)",
            (SCHEMA_VERSION,),
        )
        error = annotation_schema_error(conn, allow_missing=False)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute("RELEASE media_annotation_schema")
    except BaseException:
        conn.execute("ROLLBACK TO media_annotation_schema")
        conn.execute("RELEASE media_annotation_schema")
        raise
@@ -0,0 +1,551 @@
 """Per-image tag sets (#249): byte-exact storage in the shared vocabulary.
 A tag set belongs to one byte record (``media_annotation_bytes``): the
 whole-file SHA-256 of one physical file, owned by the file's private locator
 (``media_annotation_files``). Callers pass verified facts (digest, size,
 ``mtime_ns``); nothing here reads the filesystem or resolves an owner through
 prompt links, so tagging one image never changes another image, a prompt or a
 pair.
 Commands run in one ``_pair_transaction`` fenced by the storage epoch, like the
 native Library and vocabulary commands. A save is a full replacement guarded
 by the set's ``tag_revision``; the identical set sent again (a lost-response
 retry, or the current set at its own revision) answers ``replayed: true`` and
 writes nothing. ``add_image_tags`` unions into the existing set, keeping the
 stored spelling. Names enter the global ``tags`` registry exactly as spelled,
 so vocabulary rename, merge and delete carry image links with prompt and pair
 links (``PromptDatabase._move_tag_links``).
 Cheap reads (spec amendment S1) treat a byte record as *current* only while
 the file's present ``(size, mtime_ns)`` equals the record's verified
 fingerprint; a file without a current record shows no image tags.
 A write made from a tagging model's result passes ``mark_autotagged`` (#379):
 the byte record is then marked auto-tagged (``media_image_autotags``) in the
 same transaction, so a mark never exists without the write it records. Hand
 tags never mark.
 """
 import re
 import sqlite3
 from contextlib import contextmanager
 from typing import Iterator
 from uuid import uuid4
 from .annotation_schema import (
    BYTES_TABLE,
    FILES_TABLE,
    IMAGE_TAGS_TABLE,
    MAX_LOCATOR_LENGTH,
    METADATA_TABLE,
 )
 from .autotag_schema import AUTOTAG_TABLE
 from .library_commands import _UUID, LibraryCommandError, _validate_epoch
 from .models import utc_now_iso
 from .vocabulary import validate_vocabulary_tag_name
 MAX_IMAGE_TAGS = 100
 _LOOKUP_CHUNK = 500
 _DIGEST = re.compile(r"[0-9a-f]{64}")
 _MAX_INT = 2**63 - 1
 _MIN_INT = -(2**63)
 # Only this module's table constants are interpolated into the SQL below.
 _OWNER = f"SELECT id FROM {FILES_TABLE} WHERE locator=?"  # nosec B608
 _INSERT_OWNER = (
    f"INSERT INTO {FILES_TABLE}(id,locator,created_at) VALUES(?,?,?)"  # nosec B608
 )
 _BYTE = (
    f"SELECT id,tag_revision,size,mtime_ns FROM {BYTES_TABLE} "  # nosec B608
    "WHERE file_id=? AND sha256=?"
 )
 _INSERT_BYTE = (
    f"INSERT INTO {BYTES_TABLE}(id,file_id,sha256,size,mtime_ns,"  # nosec B608
    "verified_at,first_seen_at,tag_revision) VALUES(?,?,?,?,?,?,?,0)"
 )
 _REFRESH_BYTE = (
    f"UPDATE {BYTES_TABLE} SET size=?,mtime_ns=?,verified_at=? WHERE id=?"  # nosec B608
 )
 _REFRESH_BY_LOCATOR = (
    f"UPDATE {BYTES_TABLE} SET size=?,mtime_ns=?,verified_at=? "  # nosec B608
    f"WHERE sha256=? AND file_id=({_OWNER})"
 )
 _BYTE_TAGS = (
    f"SELECT t.name FROM {IMAGE_TAGS_TABLE} mit "  # nosec B608
    "JOIN tags t ON t.id=mit.tag_id WHERE mit.byte_id=? ORDER BY mit.position"
 )
 _CLEAR_TAGS = f"DELETE FROM {IMAGE_TAGS_TABLE} WHERE byte_id=?"  # nosec B608
 _INSERT_TAG = (
    f"INSERT INTO {IMAGE_TAGS_TABLE}(byte_id,tag_id,position) "  # nosec B608
    "VALUES(?,?,?)"
 )
 _BUMP_REVISION = (
    f"UPDATE {BYTES_TABLE} SET tag_revision=tag_revision+1 WHERE id=?"  # nosec B608
 )
 _REVISION = f"SELECT tag_revision FROM {BYTES_TABLE} WHERE id=?"  # nosec B608
 _BUMP_INDEX = (
    f"UPDATE {METADATA_TABLE} SET "  # nosec B608
    "image_index_revision=image_index_revision+1 WHERE singleton_id=1"
 )
 _READ = (
    "SELECT b.id,b.tag_revision,"  # nosec B608
    f"(SELECT COUNT(*) FROM {BYTES_TABLE} o WHERE o.file_id=f.id AND o.sha256!=?) "
    f"FROM {FILES_TABLE} f LEFT JOIN {BYTES_TABLE} b "
    "ON b.file_id=f.id AND b.sha256=? WHERE f.locator=?"
 )
 _FINGERPRINTS = (
    "SELECT f.locator,b.id,b.size,b.mtime_ns,b.verified_at "  # nosec B608
    f"FROM {FILES_TABLE} f JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    "WHERE f.locator IN ({marks})"
 )
 _TAGS_OF = (
    f"SELECT mit.byte_id,t.name FROM {IMAGE_TAGS_TABLE} mit "  # nosec B608
    "JOIN tags t ON t.id=mit.tag_id WHERE mit.byte_id IN ({marks}) "
    "ORDER BY mit.byte_id,mit.position"
 )
 _MARK = f"INSERT OR IGNORE INTO {AUTOTAG_TABLE}(byte_id) VALUES(?)"  # nosec B608
 _MARKED_OF = (
    f"SELECT byte_id FROM {AUTOTAG_TABLE} WHERE byte_id IN ({{marks}})"  # nosec B608
 )
 _MARKED_LOCATORS = (
    f"SELECT DISTINCT f.locator FROM {AUTOTAG_TABLE} m "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.id=m.byte_id JOIN {FILES_TABLE} f ON f.id=b.file_id"
 )
 # Tag search (#379): the names that tag at least one image (the tag index
 # answers the EXISTS), then the locators carrying chosen names (by tag_id).
 _IMAGE_TAG_NAMES = (
    "SELECT t.id,t.name FROM tags t WHERE EXISTS "  # nosec B608
    f"(SELECT 1 FROM {IMAGE_TAGS_TABLE} mit WHERE mit.tag_id=t.id)"
 )
 _TAG_LOCATORS = (
    f"SELECT DISTINCT f.locator FROM {IMAGE_TAGS_TABLE} mit "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.id=mit.byte_id "
    f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE mit.tag_id IN ({{marks}})"
 )
 class AnnotationCommandError(LibraryCommandError):
    """An image tag command lost a race or hit the set's bound.
    ``code`` is ``tag-conflict`` (the set changed since it was read) or
    ``tag-limit`` (the union would exceed ``MAX_IMAGE_TAGS``); ``current`` is
    the stored set ``{sha256, tags, tag_revision}`` read inside the
    transaction, so the caller can show it.
    """
    def __init__(self, code: str, epoch: str, current: dict | None = None):
        super().__init__(code, epoch)
        self.current = current
 class ImageTagLimitError(ValueError):
    """More than ``MAX_IMAGE_TAGS`` distinct names in one set."""
 def normalize_image_tags(names: object) -> list[str]:
    """Trim and validate each name, fold case-insensitive repeats, cap at 100.
    The first occurrence keeps its spelling and position. Raises ``ValueError``
    for anything but a list of valid vocabulary names, and its subclass
    ``ImageTagLimitError`` for more than ``MAX_IMAGE_TAGS`` distinct names.
    """
    if not isinstance(names, list):
        raise ValueError("tags must be a list of strings")
    result: list[str] = []
    seen: set[str] = set()
    for value in names:
        name = validate_vocabulary_tag_name(value)
        key = name.casefold()
        if key in seen:
            continue
        seen.add(key)
        result.append(name)
    if len(result) > MAX_IMAGE_TAGS:
        raise ImageTagLimitError(f"An image can carry at most {MAX_IMAGE_TAGS} tags")
    return result
 def _validate_locator(locator: object) -> None:
    if not isinstance(locator, str) or not 0 < len(locator) <= MAX_LOCATOR_LENGTH:
        raise ValueError(
            f"locator must be a nonblank path of at most {MAX_LOCATOR_LENGTH} chars"
        )
 def _validate_digest(sha256: object) -> None:
    if not isinstance(sha256, str) or not _DIGEST.fullmatch(sha256):
        raise ValueError("sha256 must be 64 lowercase hexadecimal characters")
 def _validate_int(value: object, name: str, minimum: int) -> None:
    if type(value) is not int or not minimum <= value <= _MAX_INT:
        raise ValueError(f"{name} must be an integer not below {minimum}")
 def _validate_target(locator: object, sha256: object, size: object, mtime_ns: object):
    _validate_locator(locator)
    _validate_digest(sha256)
    _validate_int(size, "size", 0)
    _validate_int(mtime_ns, "mtime_ns", _MIN_INT)
 def _validate_fingerprint(locator: object, fingerprint: object) -> None:
    _validate_locator(locator)
    if (
        not isinstance(fingerprint, tuple)
        or len(fingerprint) != 2
        or any(type(value) is not int for value in fingerprint)
    ):
        raise ValueError("stats must map locators to (size, mtime_ns) integer pairs")
 def _new_id() -> str:
    """A canonical UUID; the schema's CHECK lets NULL through, this does not."""
    value = str(uuid4())
    if not _UUID.fullmatch(value):
        raise ValueError("Annotation ids must be canonical UUIDs")
    return value
 def _receipt(epoch: str, sha256: str, tags: list, revision: int, replayed: bool):
    return {
        "library_epoch": epoch,
        "sha256": sha256,
        "tags": list(tags),
        "tag_revision": revision,
        "replayed": replayed,
    }
 class AnnotationOperations:
    """PromptDatabase mixin: image tag sets, fingerprints and cheap reads.
    ``_`` helpers take the open ``_pair_transaction`` connection.
    """
    @contextmanager
    def _annotation_read(self) -> Iterator[sqlite3.Connection]:
        """One deferred read transaction, so paired SELECTs see one snapshot."""
        conn = self.model.get_connection()
        if conn.in_transaction:
            raise sqlite3.ProgrammingError(
                "Annotation reads require their own snapshot"
            )
        with conn:
            conn.execute("BEGIN")
            yield conn
    @staticmethod
    def _annotation_owner(conn: sqlite3.Connection, locator: str, now: str) -> str:
        row = conn.execute(_OWNER, (locator,)).fetchone()
        if row is not None:
            return row[0]
        file_id = _new_id()
        conn.execute(_INSERT_OWNER, (file_id, locator, now))
        return file_id
    @staticmethod
    def _annotation_byte(
        conn: sqlite3.Connection,
        file_id: str,
        sha256: str,
        size: int,
        mtime_ns: int,
        now: str,
    ) -> tuple[str, int, bool, bool]:
        """The byte record for these bytes, refreshed or created.
        Returns ``(id, tag_revision, created, refingerprinted)``; the last is
        True when an existing record's verified ``(size, mtime_ns)`` changed,
        which moves its currency for cheap reads (the caller bumps the index
        revision when nothing else does).
        """
        row = conn.execute(_BYTE, (file_id, sha256)).fetchone()
        if row is None:
            byte_id = _new_id()
            conn.execute(
                _INSERT_BYTE, (byte_id, file_id, sha256, size, mtime_ns, now, now)
            )
            return byte_id, 0, True, False
        conn.execute(_REFRESH_BYTE, (size, mtime_ns, now, row[0]))
        return row[0], row[1], False, (row[2], row[3]) != (size, mtime_ns)
    @staticmethod
    def _byte_tags(conn: sqlite3.Connection, byte_id: str) -> list[str]:
        return [row[0] for row in conn.execute(_BYTE_TAGS, (byte_id,))]
    def _write_image_tags(
        self, conn: sqlite3.Connection, byte_id: str, tags: list[str]
    ) -> int:
        """Rewrite the junction in order, bump both revisions; return the new one."""
        tag_ids = self._ensure_tags(conn, tags)
        conn.execute(_CLEAR_TAGS, (byte_id,))
        conn.executemany(
            _INSERT_TAG,
            [(byte_id, tag_ids[name], position) for position, name in enumerate(tags)],
        )
        conn.execute(_BUMP_REVISION, (byte_id,))
        conn.execute(_BUMP_INDEX)
        return conn.execute(_REVISION, (byte_id,)).fetchone()[0]
    def _current_set(self, conn, sha256, byte_id, revision) -> dict:
        return {
            "sha256": sha256,
            "tags": self._byte_tags(conn, byte_id),
            "tag_revision": revision,
        }
    def read_image_tags(self, locator: str, sha256: str) -> dict:
        """The stored set of these exact bytes, or an empty set at revision 0.
        ``prior_count`` is the number of other byte records at the locator
        (earlier or later bytes of the same file).
        """
        _validate_locator(locator)
        _validate_digest(sha256)
        with self._annotation_read() as conn:
            row = conn.execute(_READ, (sha256, sha256, locator)).fetchone()
            if row is None:
                return {"tags": [], "tag_revision": 0, "prior_count": 0}
            tags = self._byte_tags(conn, row[0]) if row[0] is not None else []
            return {"tags": tags, "tag_revision": row[1] or 0, "prior_count": row[2]}
    def replace_image_tags(
        self,
        *,
        locator: str,
        sha256: str,
        size: int,
        mtime_ns: int,
        expected_epoch: str,
        expected_tag_revision: int,
        tags: list,
        mark_autotagged: bool = False,
    ) -> dict:
        """Replace the set of these exact bytes, guarded by epoch and revision.
        Creates the owner and byte record when the bytes are new (an explicit
        empty set is a real save). The identical set at the current revision,
        or one behind it (a lost-response retry), is ``replayed`` and writes
        no tag; any other revision mismatch is ``tag-conflict`` with the
        current set and no writes. ``mark_autotagged`` marks the bytes
        auto-tagged on a saved or replayed receipt (the Tag image dialog's
        save after Generate suggestions).
        """
        _validate_target(locator, sha256, size, mtime_ns)
        _validate_epoch(expected_epoch)
        _validate_int(expected_tag_revision, "expected_tag_revision", 0)
        tags = normalize_image_tags(tags)
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            now = utc_now_iso()
            file_id = self._annotation_owner(conn, locator, now)
            byte_id, revision, created, refingerprinted = self._annotation_byte(
                conn, file_id, sha256, size, mtime_ns, now
            )
            current = self._byte_tags(conn, byte_id)
            if (
                not created
                and current == tags
                and expected_tag_revision in (revision, revision - 1)
            ):
                if refingerprinted:
                    conn.execute(_BUMP_INDEX)
                if mark_autotagged:
                    conn.execute(_MARK, (byte_id,))
                return _receipt(epoch, sha256, current, revision, True)
            if expected_tag_revision != revision:
                raise AnnotationCommandError(
                    "tag-conflict",
                    epoch,
                    {"sha256": sha256, "tags": current, "tag_revision": revision},
                )
            revision = self._write_image_tags(conn, byte_id, tags)
            if mark_autotagged:
                conn.execute(_MARK, (byte_id,))
            return _receipt(epoch, sha256, tags, revision, False)
    def add_image_tags(
        self,
        *,
        locator: str,
        sha256: str,
        size: int,
        mtime_ns: int,
        expected_epoch: str,
        tags: list,
        mark_autotagged: bool = False,
    ) -> dict:
        """Union ``tags`` into the set of these exact bytes, keeping stored spelling.
        ``added`` counts the names that were new (case-insensitively). A union
        over ``MAX_IMAGE_TAGS`` is ``tag-limit`` with the current set and no
        writes; a union that adds nothing to an existing record is ``replayed``.
        ``mark_autotagged`` (a tagging model's result) also marks the bytes
        auto-tagged, replayed or not; only such a call may pass no tags (the
        model found none: the image is marked and its set is unchanged).
        """
        _validate_target(locator, sha256, size, mtime_ns)
        _validate_epoch(expected_epoch)
        tags = normalize_image_tags(tags)
        if not tags and not mark_autotagged:
            raise ValueError("tags must name at least one tag")
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            now = utc_now_iso()
            file_id = self._annotation_owner(conn, locator, now)
            byte_id, revision, created, refingerprinted = self._annotation_byte(
                conn, file_id, sha256, size, mtime_ns, now
            )
            current = self._byte_tags(conn, byte_id)
            present = {name.casefold() for name in current}
            added = [name for name in tags if name.casefold() not in present]
            merged = current + added
            if len(merged) > MAX_IMAGE_TAGS:
                raise AnnotationCommandError(
                    "tag-limit",
                    epoch,
                    {"sha256": sha256, "tags": current, "tag_revision": revision},
                )
            if mark_autotagged:
                conn.execute(_MARK, (byte_id,))
            if not created and not added:
                if refingerprinted:
                    conn.execute(_BUMP_INDEX)
                return {**_receipt(epoch, sha256, current, revision, True), "added": 0}
            revision = self._write_image_tags(conn, byte_id, merged)
            return {
                **_receipt(epoch, sha256, merged, revision, False),
                "added": len(added),
            }
    def refresh_image_fingerprint(
        self, locator: str, sha256: str, size: int, mtime_ns: int
    ) -> bool:
        """Record that these exact bytes were just seen with this stat.
        True when a byte record was refreshed; nothing is created and no tag
        or tag revision changes. The image index revision moves because the
        record's currency for cheap reads may have changed.
        """
        _validate_target(locator, sha256, size, mtime_ns)
        with self.model.get_connection() as conn:
            cursor = conn.execute(
                _REFRESH_BY_LOCATOR, (size, mtime_ns, utc_now_iso(), sha256, locator)
            )
            if cursor.rowcount > 0:
                conn.execute(_BUMP_INDEX)
            return cursor.rowcount > 0
    @staticmethod
    def _validated_stats(stats: object) -> dict:
        if not isinstance(stats, dict):
            raise ValueError(
                "stats must map locators to (size, mtime_ns) integer pairs"
            )
        for locator, fingerprint in stats.items():
            _validate_fingerprint(locator, fingerprint)
        return stats
    @staticmethod
    def _current_byte_records(conn: sqlite3.Connection, stats: dict) -> dict:
        """``{locator: byte_id}`` of each locator's current byte record.
        The one rule every cheap read uses: a record is current while its
        verified ``(size, mtime_ns)`` equals the file's present fingerprint
        in ``stats``; the most recently verified wins (ties on ``(verified_at,
        byte_id)``). A locator without a current record is left out. Lookups
        go in chunks of 500 locators.
        """
        locators = list(stats)
        chosen: dict[str, tuple[tuple[str, str], str]] = {}
        for start in range(0, len(locators), _LOOKUP_CHUNK):
            chunk = locators[start : start + _LOOKUP_CHUNK]
            marks = ",".join("?" * len(chunk))
            for locator, byte_id, size, mtime_ns, verified_at in conn.execute(
                _FINGERPRINTS.format(marks=marks), chunk
            ):
                if (size, mtime_ns) != stats[locator]:
                    continue
                key = (verified_at, byte_id)
                if locator not in chosen or key > chosen[locator][0]:
                    chosen[locator] = (key, byte_id)
        return {locator: byte_id for locator, (_key, byte_id) in chosen.items()}
    def current_image_tags(self, stats: dict) -> dict:
        """Image tags of the files whose current ``(size, mtime_ns)`` is verified.
        ``stats`` maps locators to their present fingerprint. A locator is
        answered (possibly with an empty set) only when a byte record carries
        exactly that verified fingerprint; the most recently verified record
        wins when several do. Lookups go in chunks of 500 locators.
        """
        stats = self._validated_stats(stats)
        with self._annotation_read() as conn:
            current = self._current_byte_records(conn, stats)
            owners = {byte_id: locator for locator, byte_id in current.items()}
            result: dict[str, list[str]] = {locator: [] for locator in current}
            byte_ids = list(owners)
            for start in range(0, len(byte_ids), _LOOKUP_CHUNK):
                chunk = byte_ids[start : start + _LOOKUP_CHUNK]
                marks = ",".join("?" * len(chunk))
                for byte_id, name in conn.execute(_TAGS_OF.format(marks=marks), chunk):
                    result[owners[byte_id]].append(name)
        return result
    def autotag_state(self, stats: dict) -> dict:
        """``{locator: bool}``: is each file's current byte record auto-tagged?
        ``stats`` as for :meth:`current_image_tags`, whose rule picks the
        current record; a locator without one is False. One read snapshot.
        """
        stats = self._validated_stats(stats)
        with self._annotation_read() as conn:
            current = self._current_byte_records(conn, stats)
            byte_ids = list(dict.fromkeys(current.values()))
            marked: set[str] = set()
            for start in range(0, len(byte_ids), _LOOKUP_CHUNK):
                chunk = byte_ids[start : start + _LOOKUP_CHUNK]
                marks = ",".join("?" * len(chunk))
                marked.update(
                    row[0]
                    for row in conn.execute(_MARKED_OF.format(marks=marks), chunk)
                )
        return {locator: current.get(locator) in marked for locator in stats}
    def autotagged_locators(self) -> set:
        """Locators with at least one auto-tagged byte record, current or not.
        No fingerprint is checked (the Tag output images review's estimate).
        """
        with self._annotation_read() as conn:
            return {row[0] for row in conn.execute(_MARKED_LOCATORS)}
    def image_tag_ids(self, wanted, query: str) -> list[int]:
        """Ids of the tag names on at least one image that a search wants.
        A name counts when it is in ``wanted`` exactly (the Tags filter) or
        contains ``query`` case-folded (the text search; ``query`` is
        already folded). One read of the names that tag an image.
        """
        wanted = set(wanted)
        with self._annotation_read() as conn:
            return [
                tag_id
                for tag_id, name in conn.execute(_IMAGE_TAG_NAMES)
                if name in wanted or (query and query in name.casefold())
            ]
    def image_tag_locators(self, tag_ids) -> set[str]:
        """Locators with any byte record (current or not) carrying one of
        ``tag_ids``, through the tag index, in chunks of 500 ids."""
        ids = list(dict.fromkeys(tag_ids))
        found: set[str] = set()
        with self._annotation_read() as conn:
            for start in range(0, len(ids), _LOOKUP_CHUNK):
                chunk = ids[start : start + _LOOKUP_CHUNK]
                marks = ",".join("?" * len(chunk))
                found.update(
                    row[0]
                    for row in conn.execute(_TAG_LOCATORS.format(marks=marks), chunk)
                )
        return found
@@ -0,0 +1,140 @@
 """Auto-tagged marks (#379): which image bytes a tagging model has handled.
 One table, ``media_image_autotags``: one row per byte record of the image
 annotation group (``media_annotation_bytes``) means "a tagging model looked at
 these exact bytes and its result was stored", or that the bytes were found to
 be untaggable. "Skip already tagged" skips an image whose current byte record
 is marked; hand tags never mark. Rows are written only by the image tag store
 (``database/annotations.py``), in the transaction of the tag write they
 record.
 The group installs like ``image_favorites``: optional in old backups,
 required once a database is current, and never backfilled. Like the Needs
 review group it is valid only beside a valid image annotation group, because
 every mark cites one byte record (``ON DELETE CASCADE``). Validation reads
 only: the catalog compared after whitespace compaction, the columns and the
 foreign key through the PRAGMAs, then one bounded value pass and the foreign
 key check.
 """
 import re
 import sqlite3
 from typing import Optional
 from .annotation_schema import BYTES_TABLE
 from .annotation_schema import TABLES as ANNOTATION_TABLES
 from .annotation_schema import _catalog as _annotation_catalog
 from .annotation_schema import _uuid, annotation_schema_error
 AUTOTAG_TABLE = "media_image_autotags"
 _DDL = (
    f"CREATE TABLE {AUTOTAG_TABLE} (byte_id TEXT PRIMARY KEY "
    f"REFERENCES {BYTES_TABLE}(id) ON DELETE CASCADE) WITHOUT ROWID"
 )
 # (cid, name, declared type, not null, default, primary key position): the
 # key of a WITHOUT ROWID table is implicitly NOT NULL.
 _COLUMNS = [(0, "byte_id", "TEXT", 1, None, 1)]
 # (column, parent table, parent column, on delete)
 _FOREIGN_KEYS = {("byte_id", BYTES_TABLE, "id", "CASCADE")}
 NEEDS_ANNOTATIONS = "Image auto-tag marks need the image annotation tables"
 INVALID_SCHEMA = "Image auto-tag marks have an invalid schema"
 INVALID_VALUES = "Image auto-tag marks contain invalid values"
 MISSING = "Missing image auto-tag marks schema"
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql).lower()
 def _catalog(conn: sqlite3.Connection) -> list:
    return conn.execute(
        "SELECT type,name,sql FROM sqlite_master WHERE name=? OR tbl_name=?",
        (AUTOTAG_TABLE, AUTOTAG_TABLE),
    ).fetchall()
 def autotag_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
    """Catalog only: the marks table present without its annotation tables.
    Startup asks this before the annotation group installs, which would make
    a marks table copied without it look valid. It reads no rows.
    """
    if not _catalog(conn):
        return None
    present = {
        name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
    }
    if present != set(ANNOTATION_TABLES):
        return NEEDS_ANNOTATIONS
    return None
 def autotag_schema_error(
    conn: sqlite3.Connection,
    *,
    allow_missing: bool = True,
    annotations_valid: bool = False,
 ) -> Optional[str]:
    """Reject an unsupported marks table or invalid marks; reads only.
    A database without the table is acceptable for backups that predate it
    unless ``allow_missing`` is false. A present table requires a present,
    valid image annotation group; ``annotations_valid`` says the caller has
    just validated it (startup's annotation install), so only its presence is
    checked again.
    """
    objects = _catalog(conn)
    if not objects:
        return None if allow_missing else MISSING
    if annotations_valid:
        if autotag_without_annotations_error(conn):
            return NEEDS_ANNOTATIONS
    elif annotation_schema_error(conn, allow_missing=False):
        return NEEDS_ANNOTATIONS
    actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
    if actual != {("table", AUTOTAG_TABLE, _compact(_DDL))}:
        return INVALID_SCHEMA
    columns = [
        (row[0], row[1], row[2].upper(), row[3], row[4], row[5])
        for row in conn.execute(f"PRAGMA table_info({AUTOTAG_TABLE})")
    ]
    if columns != _COLUMNS:
        return INVALID_SCHEMA
    foreign_keys = {
        (row[3], row[2], row[4], row[6])
        for row in conn.execute(f"PRAGMA foreign_key_list({AUTOTAG_TABLE})")
    }
    if foreign_keys != _FOREIGN_KEYS:
        return INVALID_SCHEMA
    # Only this module's constants are interpolated into the SQL below.
    if conn.execute(
        f"SELECT 1 FROM {AUTOTAG_TABLE} "  # nosec B608
        f"WHERE NOT ({_uuid('byte_id')}) LIMIT 1"
    ).fetchone():
        return INVALID_VALUES
    if conn.execute(f"PRAGMA foreign_key_check({AUTOTAG_TABLE})").fetchone():
        return INVALID_VALUES
    return None
 def migrate_autotag_state(conn: sqlite3.Connection) -> None:
    """Install the table atomically; validate before and after creating it.
    Runs after ``migrate_annotation_state`` (which has just validated the
    annotation group). Creates one empty table: no row is read or copied.
    """
    conn.execute("SAVEPOINT media_autotag_schema")
    try:
        error = autotag_schema_error(conn, annotations_valid=True)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        error = autotag_schema_error(conn, allow_missing=False, annotations_valid=True)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute("RELEASE media_autotag_schema")
    except BaseException:
        conn.execute("ROLLBACK TO media_autotag_schema")
        conn.execute("RELEASE media_autotag_schema")
        raise
@@ -0,0 +1,201 @@
 """Atomic replacement of one imported category (#221, #249 System S10).
 The LoRA reference import used to delete its whole category and then save
 each record in its own transaction, so a failure part way left the previous
 collection deleted and the new one partial. ``replace_category_prompts``
 does both in one ``BEGIN IMMEDIATE`` transaction: the category's current
 prompts (with their tags and image links) are replaced by the staged
 records, or, on any error, nothing changes.
 A staged record is ``{text, hash, tags, images}``. A record whose hash a
 prompt outside the category already has links its images to that prompt
 (the legacy import did the same); a second record with the hash of an
 earlier one links its images to that one. Neither is a new prompt.
 """
 from typing import Any, Dict, List, Optional, Sequence
 try:
    from .maintenance import PlanChanged
    from .pairs import PairDependencyError
    from .workflow_documents import documents_of_rows, release_documents
 except ImportError:  # pragma: no cover - flat test imports
    from database.maintenance import PlanChanged
    from database.pairs import PairDependencyError
    from database.workflow_documents import documents_of_rows, release_documents
 class EmptyReplacement(ValueError):
    """The replacement has no record and would remove every current import."""
 # The category's prompt ids, as a subquery (one bound parameter).
 _IN_CATEGORY = "SELECT id FROM prompts WHERE category = ?"
 # Each record's outcome and the result counter it adds to.
 _COUNTERS = {"imported": "imported", "linked": "linked", "duplicate": "duplicates"}
 def _placeholders(values: Sequence[Any]) -> str:
    return ",".join("?" * len(values))
 class CategoryImportOperations:
    """Read the state of an imported category and replace it atomically."""
    def category_import_state(self, category: str) -> Dict[str, Any]:
        """``{ids, paired}``: the category's prompt ids (ascending) and how
        many of them a saved pair uses (those block a replacement)."""
        with self.model.get_connection() as conn:
            ids = [
                row[0]
                for row in conn.execute(
                    "SELECT id FROM prompts WHERE category = ? ORDER BY id",
                    (category,),
                ).fetchall()
            ]
            paired = conn.execute(
                "SELECT COUNT(*) FROM prompts WHERE category = ? AND id IN "
                "(SELECT positive_prompt_id FROM prompt_pairs UNION "
                "SELECT negative_prompt_id FROM prompt_pairs)",
                (category,),
            ).fetchone()[0]
        return {"ids": ids, "paired": paired}
    def existing_prompt_hashes(
        self, hashes: Sequence[str], exclude_category: str
    ) -> set:
        """The given hashes a prompt outside ``exclude_category`` already has."""
        found = set()
        wanted = list(dict.fromkeys(hashes))
        with self.model.get_connection() as conn:
            for start in range(0, len(wanted), 500):
                chunk = wanted[start : start + 500]
                rows = conn.execute(
                    f"SELECT hash FROM prompts WHERE hash IN ({_placeholders(chunk)}) "
                    "AND (category IS NULL OR category != ?)",
                    [*chunk, exclude_category],
                ).fetchall()
                found.update(row[0] for row in rows)
        return found
    def replace_category_prompts(
        self,
        category: str,
        records: List[Dict[str, Any]],
        *,
        expected_ids: Optional[List[int]] = None,
        allow_empty: bool = False,
    ) -> Dict[str, Any]:
        """Replace every prompt of ``category`` with ``records`` in one transaction.
        Raises (and changes nothing): ``PlanChanged`` when ``expected_ids`` is
        given and the category's prompt ids differ from it;
        ``PairDependencyError`` when a saved pair uses one of its prompts;
        ``EmptyReplacement`` when ``records`` is empty while the category has
        prompts, unless ``allow_empty`` (an empty replacement that was
        reviewed). Any other error rolls everything back too.
        Returns ``{removed, imported, linked, duplicates, images, rows}``:
        prompts removed and created, records linked to an existing prompt or
        to an earlier record with the same text, image links added, and per
        record ``{name, outcome}`` (``imported``/``linked``/``duplicate``).
        """
        with self._pair_transaction() as conn:
            ids = [
                row[0]
                for row in conn.execute(
                    "SELECT id FROM prompts WHERE category = ? ORDER BY id",
                    (category,),
                ).fetchall()
            ]
            if expected_ids is not None and ids != list(expected_ids):
                raise PlanChanged("The imported prompts changed since the review")
            if not records and ids and not allow_empty:
                raise EmptyReplacement(
                    "Nothing to import; the previous imports were kept"
                )
            self._ensure_category_unpaired(conn, category)
            removed = self._delete_category_rows(conn, category, ids)
            result = {
                "removed": removed,
                "imported": 0,
                "linked": 0,
                "duplicates": 0,
                "images": 0,
                "rows": [],
            }
            created = {}
            for record in records:
                self._import_record(conn, category, record, created, result)
            return result
    def _delete_category_rows(self, conn, category: str, ids: List[int]) -> int:
        """Delete the category's prompts with their image links and tags.
        No commit. The rows are selected by category (just verified against
        ``ids`` under the write lock), so no statement binds one parameter per
        prompt and SQLite's variable limit never caps a collection.
        """
        if not ids:
            return 0
        documents = documents_of_rows(conn, "prompt_id", ids)  # chunked
        conn.execute(
            f"DELETE FROM generated_images WHERE prompt_id IN ({_IN_CATEGORY})",
            (category,),
        )
        release_documents(conn, documents)
        conn.execute(
            f"DELETE FROM prompt_tags WHERE prompt_id IN ({_IN_CATEGORY})",
            (category,),
        )
        return conn.execute(
            "DELETE FROM prompts WHERE category = ?", (category,)
        ).rowcount
    @staticmethod
    def _ensure_category_unpaired(conn, category: str) -> None:
        """Raise PairDependencyError when a saved pair uses a category prompt."""
        row = conn.execute(
            f"SELECT id FROM prompt_pairs WHERE positive_prompt_id IN ({_IN_CATEGORY}) "
            f"OR negative_prompt_id IN ({_IN_CATEGORY}) LIMIT 1",
            (category, category),
        ).fetchone()
        if row:
            raise PairDependencyError(
                f"Prompt belongs to saved pair {row[0]}; delete the pair first "
                "or save changed text as a new prompt"
            )
    def _import_record(self, conn, category, record, created, result) -> None:
        """Save one staged record on ``conn``, or link it to the prompt it repeats."""
        prompt_hash = record["hash"]
        name = record.get("name") or record["text"]
        if prompt_hash in created:
            prompt_id, outcome = created[prompt_hash], "duplicate"
        else:
            row = conn.execute(
                "SELECT id FROM prompts WHERE hash = ?", (prompt_hash,)
            ).fetchone()
            if row is not None:
                prompt_id, outcome = row[0], "linked"
            else:
                prompt_id = self._insert_prompt(
                    conn,
                    record["text"],
                    category,
                    list(record.get("tags") or []),
                    None,
                    None,
                    prompt_hash,
                )
                outcome = "imported"
            created[prompt_hash] = prompt_id
        result[_COUNTERS[outcome]] += 1
        # Linked last to first: every workspace list shows a prompt's media
        # newest link first, so the record's first image (the LoRA cover,
        # #355) is the one a Library card shows.
        for image in reversed(record.get("images") or []):
            if self._insert_image_link(conn, prompt_id, image):
                result["images"] += 1
        result["rows"].append({"name": name, "outcome": outcome})
@@ -0,0 +1,215 @@
 """Records of duplicate image files: count them and move them (#249 S6, T17).
 A removed copy's records move to its verified kept copy before the copy is
 unlinked (``py/api/duplicate_files.py``), one file per short ``BEGIN
 IMMEDIATE`` transaction, so no row ever names a deleted file:
 - its image records (``generated_images``), found under every spelling the
  file is linked by. A record moves to the kept copy, unless its prompt
  already links the kept copy: then its saved-pair links join that record
  and it is deleted;
 - its favorite;
 - its image tags (``media_image_tags`` of its byte record), joined onto the
  kept copy's byte record as their union, in order;
 - its pending Needs review items, settled: only an item still ``pending`` at
  revision 0 with no decision row is removed (``review._SETTLE_ITEM``), so a
  decided item is never touched.
 Every statement is an index search (see ``FILE_LINK_SQL``).
 """
 import os
 from .annotation_schema import BYTES_TABLE, FILES_TABLE
 from .annotations import _BUMP_INDEX
 from .models import normalize_image_path, utc_now_iso
 from .review import _BUMP_REVIEW_INDEX, _PENDING_OF_LOCATOR, _SETTLE_ITEM
 from .review_changes import notify_review_change
 from .workflow_documents import documents_of_rows, release_documents
 LINKS_BY_FILE_SQL = "SELECT id, prompt_id FROM generated_images WHERE file_path = ?"
 LEGACY_LINKS_SQL = (
    "SELECT id, prompt_id FROM generated_images "
    "WHERE image_path = ? AND file_path IS NULL"
 )
 KEEPER_LINK_SQL = (
    "SELECT id FROM generated_images WHERE prompt_id = ? AND file_path = ?"
 )
 MOVE_PAIR_LINKS_SQL = (
    "UPDATE OR IGNORE image_prompt_pairs SET image_id = ? WHERE image_id = ?"
 )
 MOVE_LINK_SQL = (
    "UPDATE generated_images SET image_path = ?, filename = ?, file_path = ? "
    "WHERE id = ?"
 )
 DELETE_LINK_SQL = "DELETE FROM generated_images WHERE id = ?"
 FAVORITE_SQL = (
    "SELECT created_at FROM image_favorites WHERE root_key = ? AND relative_path = ?"
 )
 INSERT_FAVORITE_SQL = "INSERT OR IGNORE INTO image_favorites VALUES (?, ?, ?)"
 DELETE_FAVORITE_SQL = (
    "DELETE FROM image_favorites WHERE root_key = ? AND relative_path = ?"
 )
 COPY_BYTE_SQL = (
    f"SELECT b.id FROM {FILES_TABLE} f JOIN {BYTES_TABLE} b "  # nosec B608
    "ON b.file_id=f.id WHERE f.locator=? AND b.sha256=?"
 )
 COPY_TAG_COUNT_SQL = (
    "SELECT COUNT(*) FROM media_image_tags WHERE byte_id=?"  # nosec B608
 )
 PENDING_COUNT_SQL = (
    f"SELECT COUNT(*) FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    "JOIN media_review_items i ON i.byte_id=b.id "
    "WHERE f.locator=? AND +i.state='pending' AND i.revision=0 "
    "AND NOT EXISTS (SELECT 1 FROM media_review_decisions d WHERE d.item_id=i.id)"
 )
 FILE_LINK_SQL = (
    LINKS_BY_FILE_SQL,
    LEGACY_LINKS_SQL,
    KEEPER_LINK_SQL,
    MOVE_PAIR_LINKS_SQL,
    MOVE_LINK_SQL,
    DELETE_LINK_SQL,
    FAVORITE_SQL,
    DELETE_FAVORITE_SQL,
    COPY_BYTE_SQL,
    COPY_TAG_COUNT_SQL,
    PENDING_COUNT_SQL,
 )
 def _file_link_rows(conn, spelled):
    """``(id, prompt_id)`` of the image records linking one spelling."""
    rows = list(conn.execute(LINKS_BY_FILE_SQL, (normalize_image_path(spelled),)))
    rows += list(conn.execute(LEGACY_LINKS_SQL, (spelled,)))
    return [tuple(row) for row in rows]
 def _copy_byte(conn, locator, sha256):
    row = conn.execute(COPY_BYTE_SQL, (locator, sha256)).fetchone()
    return row[0] if row else None
 class DuplicateCopyOperations:
    """PromptDatabase mixin: the records of duplicate copies."""
    def count_copy_records(self, copies) -> dict:
        """``{path: {links, tags, review}}`` for each copy (one read snapshot).
        ``copies`` maps each path to ``{spellings, locator, sha256}``:
        ``links`` counts the image records under any spelling, ``tags`` the
        image tags of its bytes and ``review`` its pending, undecided Needs
        review items.
        """
        from .maintenance import _read_snapshot
        def read(conn):
            found = {}
            for path, copy in copies.items():
                ids = {
                    row[0]
                    for spelled in copy["spellings"]
                    for row in _file_link_rows(conn, spelled)
                }
                byte_id = _copy_byte(conn, copy["locator"], copy["sha256"])
                tags = (
                    conn.execute(COPY_TAG_COUNT_SQL, (byte_id,)).fetchone()[0]
                    if byte_id
                    else 0
                )
                review = conn.execute(PENDING_COUNT_SQL, (copy["locator"],))
                found[path] = {
                    "links": len(ids),
                    "tags": tags,
                    "review": review.fetchone()[0],
                }
            return found
        return _read_snapshot(self.model.db_path, read)
    def move_copy_records(self, move) -> dict:
        """Move one removed copy's records to its kept copy (one transaction).
        ``move``: ``removed`` and ``kept`` (the spellings each file is linked
        by), ``targets`` (``{removed spelling: kept spelling}``, the spelling
        a moved record takes; else ``kept_path``), ``favorites`` (``(root key,
        removed key, kept root key, kept key)`` per listed root holding the
        file), ``removed_locator``,
        ``kept_locator``, ``sha256`` and the kept copy's ``kept_size`` and
        ``kept_mtime_ns``. Returns ``{moved, merged, favorites, tags,
        review}``.
        """
        counts = dict.fromkeys(("moved", "merged", "favorites", "tags", "review"), 0)
        kept_keys = [normalize_image_path(path) for path in move["kept"]]
        with self._pair_transaction() as conn:
            for spelled in dict.fromkeys(move["removed"]):
                target = move["targets"].get(spelled, move["kept_path"])
                keys = list(dict.fromkeys(kept_keys + [normalize_image_path(target)]))
                for image_id, prompt_id in _file_link_rows(conn, spelled):
                    existing = None
                    for key in keys:
                        existing = conn.execute(
                            KEEPER_LINK_SQL, (prompt_id, key)
                        ).fetchone()
                        if existing is not None:
                            break
                    if existing is not None:
                        conn.execute(MOVE_PAIR_LINKS_SQL, (existing[0], image_id))
                        documents = documents_of_rows(conn, "id", [image_id])
                        conn.execute(DELETE_LINK_SQL, (image_id,))
                        release_documents(conn, documents)
                        counts["merged"] += 1
                        continue
                    conn.execute(
                        MOVE_LINK_SQL,
                        (
                            target,
                            os.path.basename(target),
                            normalize_image_path(target),
                            image_id,
                        ),
                    )
                    counts["moved"] += 1
            for favorite in move.get("favorites", ()):
                counts["favorites"] += _move_favorite(conn, *favorite)
            counts["tags"] = self._move_copy_tags(conn, move)
            for (item_id,) in conn.execute(
                _PENDING_OF_LOCATOR, (move["removed_locator"],)
            ).fetchall():
                counts["review"] += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
            if counts["review"]:
                conn.execute(_BUMP_REVIEW_INDEX)
        if counts["review"]:
            notify_review_change(self.model.db_path)
        return counts
    def _move_copy_tags(self, conn, move):
        """Join the removed copy's image tags onto the kept copy's; the count added."""
        removed = _copy_byte(conn, move["removed_locator"], move["sha256"])
        if removed is None:
            return 0
        tags = self._byte_tags(conn, removed)
        if not tags:
            return 0
        now = utc_now_iso()
        file_id = self._annotation_owner(conn, move["kept_locator"], now)
        byte_id, _revision, created, moved = self._annotation_byte(
            conn, file_id, move["sha256"], move["kept_size"], move["kept_mtime_ns"], now
        )
        current = self._byte_tags(conn, byte_id)
        union = current + [tag for tag in tags if tag not in current]
        if union != current:
            self._write_image_tags(conn, byte_id, union)
        elif created or moved:
            conn.execute(_BUMP_INDEX)
        return len(union) - len(current)
 def _move_favorite(conn, root_key, removed_key, kept_root, kept_key):
    row = conn.execute(FAVORITE_SQL, (root_key, removed_key)).fetchone()
    if row is None:
        return 0
    conn.execute(INSERT_FAVORITE_SQL, (kept_root, kept_key, row[0]))
    conn.execute(DELETE_FAVORITE_SQL, (root_key, removed_key))
    return 1
@@ -0,0 +1,406 @@
 """Owned execution-context v1 tables within execution-history schema v2.
 Content IDs hash exact scalar values; evidence JSON contains metadata only.
 All validation scans use cursors and bounded per-row payloads.
 """
 import json
 import sqlite3
 from .execution_history_schema import _compact, _uuid_check
 try:
    from ..utils.execution_contexts import (
        BUNDLE_STATES,
        CAPTURE_REASONS,
        EVIDENCE_SOURCES,
        UNKNOWN_REASONS,
        MAX_CONTEXT_BYTES,
        ContextBundle,
        PromptSnapshot,
        RecipeContext,
        TextEvidence,
        validate_bundle,
        content_digest,
        text_size,
    )
 except ImportError:
    from utils.execution_contexts import (
        BUNDLE_STATES,
        CAPTURE_REASONS,
        EVIDENCE_SOURCES,
        UNKNOWN_REASONS,
        MAX_CONTEXT_BYTES,
        ContextBundle,
        PromptSnapshot,
        RecipeContext,
        TextEvidence,
        validate_bundle,
        content_digest,
        text_size,
    )
 FIELDS = ("raw", "effective", "prepend", "append")
 TABLE_COLUMNS = {
    "execution_context_texts": (
        ("text_id", "TEXT", 1, 1),
        ("text_value", "TEXT", 1, 0),
    ),
    "execution_prompt_snapshots": (
        ("snapshot_id", "TEXT", 1, 1),
        *((f"{field}_text_id", "TEXT", 0, 0) for field in FIELDS),
    ),
    "execution_contexts": (
        ("context_id", "TEXT", 1, 1),
        ("positive_snapshot_id", "TEXT", 0, 0),
        ("negative_snapshot_id", "TEXT", 0, 0),
    ),
    "execution_run_contexts": (
        ("run_id", "TEXT", 1, 1),
        ("anchor_id", "TEXT", 1, 2),
        ("context_id", "TEXT", 1, 0),
        ("evidence_json", "TEXT", 1, 0),
        ("positive_prompt_id", "INTEGER", 0, 0),
        ("negative_prompt_id", "INTEGER", 0, 0),
        ("pair_id", "INTEGER", 0, 0),
        ("reference_epoch", "TEXT", 0, 0),
        ("capture_version", "INTEGER", 1, 0),
    ),
    "execution_context_capture": (
        ("run_id", "TEXT", 1, 1),
        ("state", "TEXT", 1, 0),
        ("reason", "TEXT", 1, 0),
        ("format_version", "INTEGER", 1, 0),
    ),
 }
 def _digest_check(column: str, nullable: bool = False) -> str:
    check = (
        f"typeof({column}) = 'text' AND length({column}) = 64 "
        f"AND length(CAST({column} AS BLOB)) = 64 "
        f"AND {column} NOT GLOB '*[^0-9a-f]*'"
    )
    return f"{column} IS NULL OR ({check})" if nullable else check
 def _bounded_uuid_check(column: str) -> str:
    return f"{_uuid_check(column)} AND length(CAST({column} AS BLOB)) = 36"
 def _enum_check(column: str, values: frozenset[str]) -> str:
    return (
        f"typeof({column}) = 'text' AND {column} IN ("
        + ",".join("'" + value + "'" for value in sorted(values))
        + ")"
    )
 TABLE_CHECKS = {
    "execution_context_texts": (
        _digest_check("text_id"),
        "typeof(text_value) = 'text' AND length(CAST(text_value AS BLOB)) <= 65536",
    ),
    "execution_prompt_snapshots": (
        _digest_check("snapshot_id"),
        *(_digest_check(f"{field}_text_id", True) for field in FIELDS),
    ),
    "execution_contexts": (
        _digest_check("context_id"),
        _digest_check("positive_snapshot_id", True),
        _digest_check("negative_snapshot_id", True),
        "positive_snapshot_id IS NOT NULL OR negative_snapshot_id IS NOT NULL",
    ),
    "execution_run_contexts": (
        _bounded_uuid_check("run_id"),
        "typeof(anchor_id) = 'text' AND length(anchor_id) BETWEEN 1 AND 128 "
        "AND length(CAST(anchor_id AS BLOB)) <= 512",
        _digest_check("context_id"),
        "typeof(evidence_json) = 'text' AND length(CAST(evidence_json AS "
        "BLOB)) <= 8192",
        *(
            f"{column} IS NULL OR (typeof({column}) = 'integer' AND {column} > 0)"
            for column in ("positive_prompt_id", "negative_prompt_id", "pair_id")
        ),
        f"reference_epoch IS NULL OR ({_bounded_uuid_check('reference_epoch')})",
        "typeof(capture_version) = 'integer' AND capture_version = 1",
    ),
    "execution_context_capture": (
        _bounded_uuid_check("run_id"),
        _enum_check("state", BUNDLE_STATES),
        _enum_check("reason", CAPTURE_REASONS),
        "length(CAST(reason AS BLOB)) <= 64",
        "typeof(format_version) = 'integer' AND format_version = 1",
    ),
 }
 # (source column, destination table, destination column); actions always NO ACTION.
 TABLE_FOREIGN_KEYS = {
    "execution_context_texts": (),
    "execution_prompt_snapshots": tuple(
        (f"{field}_text_id", "execution_context_texts", "text_id") for field in FIELDS
    ),
    "execution_contexts": tuple(
        (f"{role}_snapshot_id", "execution_prompt_snapshots", "snapshot_id")
        for role in ("positive", "negative")
    ),
    "execution_run_contexts": (
        ("run_id", "execution_runs", "run_id"),
        ("context_id", "execution_contexts", "context_id"),
    ),
    "execution_context_capture": (("run_id", "execution_runs", "run_id"),),
 }
 # Only SQLite-generated primary-key indexes are unique. No secondary indexes required.
 def install_context_schema(conn: sqlite3.Connection) -> None:
    """Install tables inside the caller's transaction, without committing."""
    for table, columns in TABLE_COLUMNS.items():
        definitions = [
            f"{name} {kind}{' NOT NULL' if required else ''}"
            for name, kind, required, _ in columns
        ]
        primary = [name for name, _, _, order in columns if order]
        definitions.append(f"PRIMARY KEY ({', '.join(primary)})")
        definitions.extend(f"CHECK({check})" for check in TABLE_CHECKS[table])
        definitions.extend(
            f"FOREIGN KEY ({column}) REFERENCES {target}({key}) "
            "ON UPDATE NO ACTION ON DELETE NO ACTION"
            for column, target, key in TABLE_FOREIGN_KEYS[table]
        )
        conn.execute(f"CREATE TABLE {table} ({', '.join(definitions)})")
 def _unique_object(pairs: list[tuple[str, object]]) -> dict:
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("Duplicate evidence key")
        result[key] = value
    return result
 def _snapshot_values(conn: sqlite3.Connection, snapshot_id: str) -> tuple:
    row = conn.execute(
        "SELECT raw_text_id,effective_text_id,prepend_text_id,append_text_id "
        "FROM execution_prompt_snapshots WHERE snapshot_id=?",
        (snapshot_id,),
    ).fetchone()
    if row is None:
        raise ValueError("Missing snapshot")
    values = []
    for text_id in row:
        if text_id is None:
            values.append(None)
        else:
            text = conn.execute(
                "SELECT text_value FROM execution_context_texts WHERE text_id=?",
                (text_id,),
            ).fetchone()
            if text is None:
                raise ValueError("Missing context text")
            text_size(text[0])
            values.append(text[0])
    return tuple(values)
 def _validate_evidence(
    conn: sqlite3.Connection, anchor_id: str, context_id: str, encoded: str
 ) -> tuple[RecipeContext, bool]:
    text_size(encoded, 8192)
    evidence = json.loads(encoded, object_pairs_hook=_unique_object)
    if (
        type(evidence) is not dict
        or set(evidence)
        != {"format_version", "role_provenance", "positive", "negative"}
        or type(evidence["format_version"]) is not int
        or evidence["format_version"] != 1
        or evidence["role_provenance"] != "inferred"
    ):
        raise ValueError("Invalid context evidence")
    row = conn.execute(
        "SELECT positive_snapshot_id,negative_snapshot_id FROM "
        "execution_contexts WHERE context_id=?",
        (context_id,),
    ).fetchone()
    if row is None:
        raise ValueError("Missing context")
    complete = True
    snapshots = []
    for role, snapshot_id in zip(("positive", "negative"), row):
        metadata = evidence[role]
        if snapshot_id is None:
            if metadata is not None:
                raise ValueError("Unexpected role evidence")
            complete = False
            snapshots.append(None)
            continue
        values = _snapshot_values(conn, snapshot_id)
        if type(metadata) is not dict or set(metadata) != set(FIELDS):
            raise ValueError("Invalid snapshot evidence")
        fields = []
        for field, value in zip(FIELDS, values):
            item = metadata[field]
            if type(item) is not dict or set(item) != {
                "provenance",
                "source",
                "reason",
            }:
                raise ValueError("Invalid field evidence")
            provenance, source, reason = (
                item["provenance"],
                item["source"],
                item["reason"],
            )
            if (
                type(provenance) is not str
                or type(source) is not str
                or source not in EVIDENCE_SOURCES.get(provenance, ())
            ):
                raise ValueError("Invalid evidence source")
            if value is None:
                complete = False
                if (
                    provenance != "unknown"
                    or type(reason) is not str
                    or reason not in UNKNOWN_REASONS
                ):
                    raise ValueError("Unknown evidence requires reason")
            elif provenance == "unknown" or reason is not None:
                raise ValueError("Known evidence metadata mismatch")
            fields.append(TextEvidence(value, provenance, source, reason))
        snapshots.append(PromptSnapshot(*fields))
    return RecipeContext(anchor_id, *snapshots, evidence["role_provenance"]), complete
 def context_schema_error(conn: sqlite3.Connection) -> str | None:
    """Validate exact owned schema, bounded evidence, fingerprints and links."""
    if conn.execute(
        "SELECT 1 FROM sqlite_master WHERE type IN ('trigger','view') LIMIT 1"
    ).fetchone():
        return "Unsupported execution context trigger/view"
    tables = {
        row[0]: row[1]
        for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
        if row[0] in TABLE_COLUMNS
    }
    if tables.keys() != TABLE_COLUMNS.keys():
        return "Incomplete execution context schema"
    for table, expected in TABLE_COLUMNS.items():
        columns = tuple(
            (row[1], row[2].upper(), row[3], row[5])
            for row in conn.execute(f"PRAGMA table_xinfo({table})")
        )
        if columns != expected:
            return f"{table} has invalid execution context columns"
        sql = _compact(tables[table])
        checks = TABLE_CHECKS[table]
        if sql.count("check(") != len(checks) or any(
            f"check({_compact(check)})" not in sql for check in checks
        ):
            return f"{table} has invalid execution context checks"
        foreign = tuple(
            sorted(
                (row[3], row[2], row[4], row[5], row[6], row[7])
                for row in conn.execute(f"PRAGMA foreign_key_list({table})")
            )
        )
        expected_foreign = tuple(
            sorted(
                (*key, "NO ACTION", "NO ACTION", "NONE")
                for key in TABLE_FOREIGN_KEYS[table]
            )
        )
        if foreign != expected_foreign:
            return f"{table} has invalid execution context foreign keys"
        for index in conn.execute(f"PRAGMA index_list({table})"):
            if not index[2]:
                continue
            name = index[1].replace('"', '""')
            keys = tuple(
                row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
            )
            primary = tuple(column[0] for column in expected if column[3])
            if (
                index[3] != "pk"
                or index[4]
                or tuple(key[2] for key in keys) != primary
                or any(key[3] or key[4] != "BINARY" for key in keys)
            ):
                return f"{table} has unexpected execution context uniqueness"
        predicate = " AND ".join(f"({check})" for check in checks)
        if conn.execute(
            f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
        ).fetchone():
            return f"{table} contains invalid execution context records"
        if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
            return f"{table} contains orphan execution context records"
    if conn.execute(
        "SELECT 1 FROM execution_run_contexts GROUP BY run_id HAVING "
        "COUNT(*) > 16 LIMIT 1"
    ).fetchone():
        return "Too many execution contexts per run"
    try:
        for text_id, text in conn.execute(
            "SELECT text_id,text_value FROM execution_context_texts"
        ):
            text_size(text)
            if content_digest("text", (text,)) != text_id:
                raise ValueError("Invalid text fingerprint")
        for (snapshot_id,) in conn.execute(
            "SELECT snapshot_id FROM execution_prompt_snapshots"
        ):
            if (
                content_digest("snapshot", _snapshot_values(conn, snapshot_id))
                != snapshot_id
            ):
                raise ValueError("Invalid snapshot fingerprint")
        for context_id, positive, negative in conn.execute(
            "SELECT context_id,positive_snapshot_id,negative_snapshot_id FROM "
            "execution_contexts"
        ):
            if content_digest("context", (positive, negative)) != context_id:
                raise ValueError("Invalid context fingerprint")
            payload_size = 0
            for snapshot_id in (positive, negative):
                if snapshot_id is not None:
                    payload_size += sum(
                        text_size(value)
                        for value in _snapshot_values(conn, snapshot_id)
                        if value is not None
                    )
            if payload_size > MAX_CONTEXT_BYTES:
                raise ValueError("Oversized detached context")
        previous_run = None
        contexts = ()
        for run_id, anchor_id, context_id, evidence in conn.execute(
            "SELECT run_id,anchor_id,context_id,evidence_json FROM "
            "execution_run_contexts ORDER BY run_id,anchor_id"
        ):
            if run_id != previous_run:
                contexts = ()
                previous_run = run_id
            context, complete = _validate_evidence(
                conn, anchor_id, context_id, evidence
            )
            status = conn.execute(
                "SELECT state,reason FROM execution_context_capture WHERE run_id=?",
                (run_id,),
            ).fetchone()
            if status is None:
                raise ValueError("Context link has no capture header")
            state, reason = status
            if state == "complete" and not complete:
                raise ValueError("Incomplete evidence marked complete")
            candidate = (*contexts, context)
            validate_bundle(ContextBundle(candidate, state, reason))
            contexts = candidate
        if conn.execute(
            "SELECT 1 FROM execution_context_capture c WHERE state='complete' "
            "AND NOT EXISTS (SELECT 1 FROM execution_run_contexts r WHERE "
            "r.run_id=c.run_id) LIMIT 1"
        ).fetchone():
            raise ValueError("Complete capture has no contexts")
    except (ValueError, TypeError, RecursionError, OverflowError) as exc:
        return f"Invalid execution context content: {type(exc).__name__}"
    return None
@@ -0,0 +1,411 @@
 """Immutable execution capture and exact, epoch-fenced library navigation hints."""
 from contextlib import ExitStack
 import json
 import sqlite3
 from typing import Callable, ContextManager, Iterator
 try:
    from ..utils.execution_history_events import RunFinish, encode_command
    from ..utils.execution_contexts import (
        CAPTURE_REASONS,
        ContextBundle,
        content_digest,
        validate_bundle,
    )
 except ImportError:
    from utils.execution_history_events import RunFinish, encode_command
    from utils.execution_contexts import (
        CAPTURE_REASONS,
        ContextBundle,
        content_digest,
        validate_bundle,
    )
 FIELDS = ("raw", "effective", "prepend", "append")
 ROLES = ("positive", "negative")
 def _prepare(bundle: ContextBundle) -> list[dict]:
    validate_bundle(bundle)
    if bundle.reason not in CAPTURE_REASONS:
        raise ValueError("Invalid capture reason")
    prepared = []
    complete = bool(bundle.contexts)
    for context in bundle.contexts:
        evidence = {"format_version": 1, "role_provenance": context.role_provenance}
        snapshots = []
        for role in ROLES:
            snapshot = getattr(context, role)
            if snapshot is None:
                snapshots.append(None)
                evidence[role] = None
                complete = False
                continue
            values = tuple(getattr(snapshot, field).value for field in FIELDS)
            complete = complete and all(value is not None for value in values)
            snapshots.append((content_digest("snapshot", values), values))
            evidence[role] = {
                field: {
                    "provenance": getattr(snapshot, field).provenance,
                    "source": getattr(snapshot, field).source,
                    "reason": getattr(snapshot, field).reason,
                }
                for field in FIELDS
            }
        if all(snapshot is None for snapshot in snapshots):
            raise ValueError("Context must contain a snapshot")
        context_id = content_digest(
            "context", tuple(s[0] if s else None for s in snapshots)
        )
        encoded = json.dumps(
            evidence,
            ensure_ascii=False,
            allow_nan=False,
            separators=(",", ":"),
            sort_keys=True,
        )
        if len(encoded.encode("utf-8")) > 8192:
            raise ValueError("Oversized context evidence")
        prepared.append(
            {
                "anchor_id": context.anchor_id,
                "context_id": context_id,
                "snapshots": snapshots,
                "evidence_json": encoded,
            }
        )
    if bundle.state == "complete" and not complete:
        raise ValueError("Incomplete scalar capture")
    return sorted(prepared, key=lambda item: item["anchor_id"])
 def _existing_row(
    conn: sqlite3.Connection, table: str, columns: tuple[str, ...], values: tuple
 ) -> bool:
    row = conn.execute(
        f"SELECT {','.join(columns[1:])} FROM {table} WHERE {columns[0]}=?",
        (values[0],),
    ).fetchone()
    if row is None:
        return False
    if tuple(row) != values[1:]:
        raise ValueError("Execution context digest collision")
    return True
 def _content_rows(item: dict) -> Iterator[tuple[str, tuple[str, ...], tuple]]:
    snapshot_ids = []
    for snapshot in item["snapshots"]:
        if snapshot is None:
            snapshot_ids.append(None)
            continue
        snapshot_id, values = snapshot
        text_ids = []
        for value in values:
            text_id = None if value is None else content_digest("text", (value,))
            if value is not None:
                yield "execution_context_texts", ("text_id", "text_value"), (
                    text_id,
                    value,
                )
            text_ids.append(text_id)
        yield "execution_prompt_snapshots", (
            "snapshot_id",
            *(f"{field}_text_id" for field in FIELDS),
        ), (snapshot_id, *text_ids)
        snapshot_ids.append(snapshot_id)
    yield "execution_contexts", (
        "context_id",
        "positive_snapshot_id",
        "negative_snapshot_id",
    ), (item["context_id"], *snapshot_ids)
 def _insert_exact(
    conn: sqlite3.Connection, table: str, columns: tuple[str, ...], values: tuple
 ) -> None:
    if not _existing_row(conn, table, columns, values):
        names = ",".join(columns)
        placeholders = ",".join("?" for _ in columns)
        conn.execute(f"INSERT INTO {table} ({names}) VALUES ({placeholders})", values)
 def _role_hints(role: str) -> str:
    return (
        f"SELECT r.{role}_prompt_id, tx.text_value "
        "FROM execution_run_contexts r "
        "JOIN execution_contexts c ON c.context_id = r.context_id "
        f"JOIN execution_prompt_snapshots s ON s.snapshot_id = c.{role}_snapshot_id "
        "JOIN execution_context_texts tx ON tx.text_id = s.raw_text_id "
        f"WHERE r.{role}_prompt_id IS NOT NULL AND r.reference_epoch = ("
        "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id = 1)"
    )
 # Set form of the read-time prompt-hint rule in get_execution_contexts():
 # a stored role hint is live only in the current storage epoch, and only when
 # its exact raw snapshot text matches exactly one prompt, which is the hinted
 # one (_prompt_hint). Evaluated once per statement, never per prompt row.
 # Defines the CTE ``live_hint_prompts(prompt_id)``; prefix it with WITH.
 LIVE_HINT_CTES = (
    "stored_hints(prompt_id, raw) AS ("
    + _role_hints("positive")
    + " UNION "
    + _role_hints("negative")
    + "), unique_hint_texts(prompt_id, raw) AS ("
    "SELECT MIN(id), text COLLATE BINARY FROM prompts "
    "WHERE text COLLATE BINARY IN (SELECT raw FROM stored_hints) "
    "GROUP BY text COLLATE BINARY HAVING COUNT(*) = 1"
    "), live_hint_prompts(prompt_id) AS ("
    "SELECT h.prompt_id FROM stored_hints h JOIN unique_hint_texts u "
    "ON u.prompt_id = h.prompt_id AND u.raw = h.raw COLLATE BINARY)"
 )
 LIVE_HINT_PROMPT_IDS_SQL = (
    "WITH " + LIVE_HINT_CTES + " SELECT DISTINCT prompt_id FROM live_hint_prompts"
 )
 def _prompt_hint(conn: sqlite3.Connection, raw: str | None) -> int | None:
    if raw is None:
        return None
    rows = conn.execute(
        "SELECT id FROM prompts WHERE text=? COLLATE BINARY LIMIT 2", (raw,)
    ).fetchall()
    return rows[0][0] if len(rows) == 1 else None
 def _hints(
    conn: sqlite3.Connection, positive: str | None, negative: str | None
 ) -> tuple:
    pos_id, neg_id = _prompt_hint(conn, positive), _prompt_hint(conn, negative)
    pair_id = None
    negative_ambiguous = (
        negative is not None
        and neg_id is None
        and conn.execute(
            "SELECT 1 FROM prompts WHERE text=? COLLATE BINARY LIMIT 1", (negative,)
        ).fetchone()
        is not None
    )
    if pos_id is not None and negative is not None and not negative_ambiguous:
        rows = conn.execute(
            "SELECT pp.id FROM prompt_pairs pp LEFT JOIN prompts n ON "
            "n.id=pp.negative_prompt_id "
            "WHERE pp.positive_prompt_id=? AND ((pp.negative_prompt_id IS NULL AND "
            "?='') "
            "OR (pp.negative_prompt_id=? AND n.text=? COLLATE BINARY)) LIMIT 2",
            (pos_id, negative, neg_id, negative),
        ).fetchall()
        if len(rows) == 1:
            pair_id = rows[0][0]
    return pos_id, neg_id, pair_id
 def _values(conn: sqlite3.Connection, snapshot_id: str | None) -> dict | None:
    if snapshot_id is None:
        return None
    row = conn.execute(
        "SELECT r.text_value,e.text_value,p.text_value,a.text_value "
        "FROM execution_prompt_snapshots s "
        "LEFT JOIN execution_context_texts r ON r.text_id=s.raw_text_id "
        "LEFT JOIN execution_context_texts e ON e.text_id=s.effective_text_id "
        "LEFT JOIN execution_context_texts p ON p.text_id=s.prepend_text_id "
        "LEFT JOIN execution_context_texts a ON a.text_id=s.append_text_id "
        "WHERE s.snapshot_id=?",
        (snapshot_id,),
    ).fetchone()
    if row is None:
        raise sqlite3.DatabaseError("Missing execution context snapshot")
    return dict(zip(FIELDS, row))
 def _write_capture(
    conn: sqlite3.Connection,
    run_id: str,
    expected_epoch: str,
    bundle: ContextBundle,
    prepared: list[dict],
 ) -> bool:
    capture = conn.execute(
        "SELECT state,reason,format_version FROM execution_context_capture "
        "WHERE run_id=?",
        (run_id,),
    ).fetchone()
    if capture is not None:
        for item in prepared:
            for table, columns, values in _content_rows(item):
                _existing_row(conn, table, columns, values)
        links = conn.execute(
            "SELECT anchor_id,context_id,evidence_json FROM "
            "execution_run_contexts WHERE run_id=? ORDER BY anchor_id",
            (run_id,),
        ).fetchall()
        expected = [
            (item["anchor_id"], item["context_id"], item["evidence_json"])
            for item in prepared
        ]
        # Compare JSON objects so serialization order never creates a conflict.
        original = [(row[0], row[1], json.loads(row[2])) for row in links]
        proposed = [
            (anchor, context, json.loads(evidence))
            for anchor, context, evidence in expected
        ]
        if capture[0] != "conflicting" and (
            tuple(capture) != (bundle.state, bundle.reason, bundle.format_version)
            or original != proposed
        ):
            conn.execute(
                "UPDATE execution_context_capture SET "
                "state='conflicting',reason='conflicting' WHERE run_id=?",
                (run_id,),
            )
        return True
    for item in prepared:
        for table, columns, values in _content_rows(item):
            _insert_exact(conn, table, columns, values)
        raw = [snapshot[1][0] if snapshot else None for snapshot in item["snapshots"]]
        positive_id, negative_id, pair_id = _hints(conn, *raw)
        conn.execute(
            "INSERT INTO "
            "execution_run_contexts(run_id,anchor_id,context_id,evidence_json,"
            "positive_prompt_id,negative_prompt_id,pair_id,reference_epoch,"
            "capture_version) VALUES(?,?,?,?,?,?,?,?,1)",
            (
                run_id,
                item["anchor_id"],
                item["context_id"],
                item["evidence_json"],
                positive_id,
                negative_id,
                pair_id,
                expected_epoch,
            ),
        )
    conn.execute(
        "INSERT INTO "
        "execution_context_capture(run_id,state,reason,format_version) "
        "VALUES(?,?,?,?)",
        (run_id, bundle.state, bundle.reason, bundle.format_version),
    )
    return True
 class ExecutionContextOperations:
    """Compose with ExecutionHistoryOperations on PromptDatabase."""
    def write_execution_contexts(
        self, run_id: str, expected_epoch: str, bundle: ContextBundle
    ) -> bool:
        prepared = _prepare(bundle)
        with self._execution_transaction(expected_epoch) as conn:
            if (
                conn.execute(
                    "SELECT 1 FROM execution_runs WHERE run_id=? AND storage_epoch=?",
                    (run_id, expected_epoch),
                ).fetchone()
                is None
            ):
                return False
            return _write_capture(conn, run_id, expected_epoch, bundle, prepared)
    def write_execution_contexts_for_finish(
        self,
        command: RunFinish,
        bundle: ContextBundle,
        *,
        generation_guard: Callable[[], ContextManager[bool]] | None = None,
    ) -> bool:
        encode_command(command)
        if type(command) is not RunFinish:
            raise ValueError("Expected execution finish command")
        prepared = _prepare(bundle)
        # Enter the guard after BEGIN, and keep it until COMMIT/ROLLBACK exits.
        with ExitStack() as guards:
            with self._execution_transaction(command.storage_epoch) as conn:
                if generation_guard is not None and not guards.enter_context(
                    generation_guard()
                ):
                    return False
                row = conn.execute(
                    "SELECT run_id FROM execution_runs WHERE session_id=? "
                    "AND queue_item_id=? AND storage_epoch=?",
                    (command.session_id, command.queue_item_id, command.storage_epoch),
                ).fetchone()
                if row is None:
                    return False
                return _write_capture(
                    conn, row[0], command.storage_epoch, bundle, prepared
                )
    def get_execution_contexts(self, run_id: str) -> dict:
        """Return first-capture scalars; hints are None unless currently valid.
        reference_epoch remains historical evidence even when hints are invalidated.
        Reads never rebind a first-capture reference or alter stored capture data.
        """
        conn = self.model.get_connection()
        own_transaction = not conn.in_transaction
        if own_transaction:
            conn.execute("BEGIN")
        try:
            capture = conn.execute(
                "SELECT state,reason,format_version FROM execution_context_capture "
                "WHERE run_id=?",
                (run_id,),
            ).fetchone()
            if capture is None:
                return {
                    "state": "not_captured",
                    "reason": "missing",
                    "format_version": None,
                    "contexts": [],
                }
            epoch = self.get_execution_epoch()
            contexts = []
            rows = conn.execute(
                "SELECT r.*,c.positive_snapshot_id,c.negative_snapshot_id FROM "
                "execution_run_contexts r JOIN execution_contexts c ON "
                "c.context_id=r.context_id WHERE r.run_id=? ORDER BY r.anchor_id",
                (run_id,),
            )
            for row in rows:
                positive = _values(conn, row["positive_snapshot_id"])
                negative = _values(conn, row["negative_snapshot_id"])
                hints = (None, None, None)
                if row["reference_epoch"] == epoch:
                    current = _hints(
                        conn,
                        positive["raw"] if positive else None,
                        negative["raw"] if negative else None,
                    )
                    hints = tuple(
                        row[key] if row[key] == value else None
                        for key, value in zip(
                            ("positive_prompt_id", "negative_prompt_id", "pair_id"),
                            current,
                        )
                    )
                contexts.append(
                    {
                        "anchor_id": row["anchor_id"],
                        "context_id": row["context_id"],
                        "positive": positive,
                        "negative": negative,
                        "evidence": json.loads(row["evidence_json"]),
                        "positive_prompt_id": hints[0],
                        "negative_prompt_id": hints[1],
                        "pair_id": hints[2],
                        "reference_epoch": row["reference_epoch"],
                    }
                )
            return {
                "state": capture[0],
                "reason": capture[1],
                "format_version": capture[2],
                "contexts": contexts,
            }
        finally:
            if own_transaction:
                conn.rollback()
@@ -0,0 +1,209 @@
 """Transactional attempt writes and process-local restore invalidation."""
 import os
 import sqlite3
 import threading
 from contextlib import contextmanager
 from typing import Callable, Iterator
 try:
    from ..utils.execution_history_events import (
        RunFinish,
        RunStart,
        classify_evidence,
        encode_command,
    )
    from ..utils.logging_config import get_logger
 except ImportError:
    from utils.execution_history_events import (
        RunFinish,
        RunStart,
        classify_evidence,
        encode_command,
    )
    from utils.logging_config import get_logger
 _logger = get_logger("prompt_manager.database.execution_history")
 # Bounded wait for a history write lock. History is best effort and must
 # never stall ComfyUI's queue: a write that cannot start in time raises
 # "database is locked" and the writer records a gap instead of waiting.
 EXECUTION_BUSY_TIMEOUT_MS = 250
 _restore_lock = threading.Lock()
 _restore_observers: dict[str, dict[object, Callable[[], None]]] = {}
 def _database_identity(db_path: str) -> str:
    return os.path.normcase(os.path.realpath(db_path))
 def subscribe_history_restore(
    db_path: str, callback: Callable[[], None]
 ) -> Callable[[], None]:
    """Observe successful restores of the canonical database path."""
    identity = _database_identity(db_path)
    token = object()
    with _restore_lock:
        _restore_observers.setdefault(identity, {})[token] = callback
    def unsubscribe() -> None:
        with _restore_lock:
            observers = _restore_observers.get(identity)
            if observers is not None:
                observers.pop(token, None)
                if not observers:
                    _restore_observers.pop(identity, None)
    return unsubscribe
 def notify_history_restore(db_path: str) -> None:
    """Notify outside the registry lock; observers cannot fail a restore."""
    with _restore_lock:
        callbacks = tuple(
            _restore_observers.get(_database_identity(db_path), {}).values()
        )
    for callback in callbacks:
        try:
            callback()
        except Exception:
            _logger.warning("Execution history restore observer failed", exc_info=True)
 class HistoryEpochChanged(Exception):
    """A command belongs to storage replaced by a completed restore."""
    def __init__(self, storage_epoch: str):
        self.storage_epoch = storage_epoch
        super().__init__("Execution history storage epoch changed")
 class ExecutionHistoryOperations:
    """Mixin using the calling thread's existing managed model connection."""
    def get_execution_epoch(self) -> str:
        row = (
            self.model.get_connection()
            .execute(
                "SELECT storage_epoch FROM execution_history_metadata "
                "WHERE singleton_id=1"
            )
            .fetchone()
        )
        if row is None:
            raise sqlite3.DatabaseError("Missing execution history metadata")
        return row[0]
    @contextmanager
    def _execution_transaction(
        self, expected_epoch: str
    ) -> Iterator[sqlite3.Connection]:
        conn = self.model.get_connection()
        if conn.in_transaction:
            raise sqlite3.ProgrammingError(
                "Execution history requires its own transaction"
            )
        timeout = conn.execute("PRAGMA busy_timeout").fetchone()[0]
        conn.execute(f"PRAGMA busy_timeout={int(EXECUTION_BUSY_TIMEOUT_MS)}")
        try:
            conn.execute("BEGIN IMMEDIATE")
            try:
                epoch = self.get_execution_epoch()
                if epoch != expected_epoch:
                    raise HistoryEpochChanged(epoch)
                yield conn
                conn.commit()
            except BaseException:
                conn.rollback()
                raise
        finally:
            conn.execute(f"PRAGMA busy_timeout={timeout}")
    def write_execution_start(self, command: RunStart) -> bool:
        encode_command(command)
        if type(command) is not RunStart:
            raise ValueError("Expected execution start command")
        with self._execution_transaction(command.storage_epoch) as conn:
            conn.execute(
                "INSERT INTO execution_runs "
                "(run_id,session_id,queue_item_id,job_id,storage_epoch,record_version,"
                "observed_dequeue_at,outcome,diagnostics_omitted) "
                "VALUES(?,?,?,?,?,1,?,'observed',0) "
                "ON CONFLICT(session_id,queue_item_id) DO NOTHING",
                (
                    command.run_id,
                    command.session_id,
                    command.queue_item_id,
                    command.job_id,
                    command.storage_epoch,
                    command.observed_dequeue_at,
                ),
            )
        return True
    def write_execution_finish(self, command: RunFinish) -> bool:
        encode_command(command)
        if type(command) is not RunFinish:
            raise ValueError("Expected execution finish command")
        with self._execution_transaction(command.storage_epoch) as conn:
            row = conn.execute(
                "SELECT * FROM execution_runs WHERE session_id=? AND queue_item_id=? "
                "AND storage_epoch=?",
                (command.session_id, command.queue_item_id, command.storage_epoch),
            ).fetchone()
            if row is None:
                return False
            completion = classify_evidence(row["job_id"], command.evidence)
            if row["outcome"] == "unknown":
                return True
            if row["outcome"] != "observed":
                contradictory = completion.reason == "conflicting_terminal" or (
                    completion.outcome in ("succeeded", "failed", "interrupted")
                    and completion.outcome != row["outcome"]
                )
                if contradictory:
                    conn.execute(
                        "UPDATE execution_runs SET outcome='unknown',"
                        "reason='conflicting_terminal' WHERE run_id=?",
                        (row["run_id"],),
                    )
                return True
            conn.execute(
                "UPDATE execution_runs SET observed_callback_at=?,"
                "executor_started_at=?,"
                "terminal_event_at=?,outcome=?,reason=?,diagnostics_omitted=?,"
                "node_id=?,"
                "node_type=?,exception_type=? WHERE run_id=?",
                (
                    command.observed_callback_at,
                    completion.executor_started_at,
                    completion.terminal_event_at,
                    completion.outcome,
                    completion.reason,
                    int(completion.diagnostics_omitted),
                    completion.node_id,
                    completion.node_type,
                    completion.exception_type,
                    row["run_id"],
                ),
            )
        return True
    def get_execution_run(self, run_id: str) -> dict | None:
        row = (
            self.model.get_connection()
            .execute("SELECT * FROM execution_runs WHERE run_id=?", (run_id,))
            .fetchone()
        )
        return dict(row) if row is not None else None
 def rekey_history_restore(old_path: str, new_path: str) -> None:
    """Move the observers of ``old_path`` to ``new_path`` (Move library)."""
    old, new = _database_identity(old_path), _database_identity(new_path)
    if old == new:
        return
    with _restore_lock:
        observers = _restore_observers.pop(old, None)
        if observers:
            _restore_observers.setdefault(new, {}).update(observers)
@@ -0,0 +1,349 @@
 """Versioned additive schema for observed execution attempts."""
 import re
 import sqlite3
 from uuid import uuid4
 SCHEMA_VERSION = 3
 def _uuid_check(column: str) -> str:
    return (
        f"typeof({column}) = 'text' AND length({column}) = 36 "
        f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
        f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
        f"AND length(replace({column}, '-', '')) = 32 "
        f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
    )
 def _integer_check(column: str, *, nullable: bool = False) -> str:
    expression = f"typeof({column}) = 'integer' AND {column} >= 0"
    return f"{column} IS NULL OR ({expression})" if nullable else expression
 _METADATA_CHECKS = (
    "singleton_id = 1",
    "schema_version = 3",
    _uuid_check("storage_epoch"),
 )
 _RUN_CHECKS = (
    *(
        _uuid_check(column)
        for column in ("run_id", "session_id", "job_id", "storage_epoch")
    ),
    _integer_check("queue_item_id"),
    "record_version = 1",
    _integer_check("observed_dequeue_at"),
    *(
        _integer_check(column, nullable=True)
        for column in (
            "observed_callback_at",
            "executor_started_at",
            "terminal_event_at",
        )
    ),
    "outcome IN ('observed', 'succeeded', 'failed', 'interrupted', 'unknown')",
    "(outcome = 'observed' AND reason IS NULL AND "
    "observed_callback_at IS NULL) OR (outcome != 'observed' AND "
    "reason IS NOT NULL AND observed_callback_at IS NOT NULL AND "
    "((outcome = 'succeeded' AND reason = 'terminal_success') OR "
    "(outcome = 'failed' AND reason = 'terminal_error') OR (outcome = "
    "'interrupted' AND reason = 'terminal_interrupted') OR (outcome = "
    "'unknown' AND reason IN ('missing_terminal', "
    "'conflicting_terminal', 'invalid_evidence', 'evidence_limit'))))",
    "diagnostics_omitted IN (0, 1)",
    *(
        f"{column} IS NULL OR (typeof({column}) = 'text' "
        f"AND length({column}) <= {limit})"
        for column, limit in (
            ("node_id", 128),
            ("node_type", 128),
            ("exception_type", 256),
        )
    ),
 )
 _TABLE_COLUMNS = {
    "execution_history_metadata": (
        ("singleton_id", "INTEGER", 1, 1),
        ("schema_version", "INTEGER", 1, 0),
        ("storage_epoch", "TEXT", 1, 0),
    ),
    "execution_runs": (
        ("run_id", "TEXT", 1, 1),
        ("session_id", "TEXT", 1, 0),
        ("queue_item_id", "INTEGER", 1, 0),
        ("job_id", "TEXT", 1, 0),
        ("storage_epoch", "TEXT", 1, 0),
        ("record_version", "INTEGER", 1, 0),
        ("observed_dequeue_at", "INTEGER", 1, 0),
        ("observed_callback_at", "INTEGER", 0, 0),
        ("executor_started_at", "INTEGER", 0, 0),
        ("terminal_event_at", "INTEGER", 0, 0),
        ("outcome", "TEXT", 1, 0),
        ("reason", "TEXT", 0, 0),
        ("diagnostics_omitted", "INTEGER", 1, 0),
        ("node_id", "TEXT", 0, 0),
        ("node_type", "TEXT", 0, 0),
        ("exception_type", "TEXT", 0, 0),
    ),
 }
 _TABLE_CHECKS = {
    "execution_history_metadata": _METADATA_CHECKS,
    "execution_runs": _RUN_CHECKS,
 }
 # Row-only guards preserve the supported v1 DDL fingerprint while bounding reads.
 _ROW_BYTE_CHECKS = {
    "execution_history_metadata": ("length(CAST(storage_epoch AS BLOB)) = 36",),
    "execution_runs": (
        *(
            f"length(CAST({column} AS BLOB)) = 36"
            for column in (
                "run_id",
                "session_id",
                "job_id",
                "storage_epoch",
            )
        ),
        *(
            f"{column} IS NULL OR length(CAST({column} AS BLOB)) <= {limit}"
            for column, limit in (
                ("node_id", 512),
                ("node_type", 512),
                ("exception_type", 1024),
            )
        ),
    ),
 }
 _INDEXES = {
    "idx_execution_runs_identity": (True, ("session_id", "queue_item_id")),
    "idx_execution_runs_observed": (False, ("observed_dequeue_at", "run_id")),
 }
 def _compact(sql: str) -> str:
    return "".join(
        token if token.startswith("'") else re.sub(r"\s+", "", token).lower()
        for token in re.findall(r"'(?:''|[^'])*'|[^']+", sql)
    )
 def history_schema_error(
    conn: sqlite3.Connection, *, allow_missing: bool = True
 ) -> str | None:
    """Reject partial/unsupported history schemas and invalid persisted records."""
    tables = {
        row[0]: row[1]
        for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
        if row[0] in _TABLE_COLUMNS
    }
    from .execution_context_schema import TABLE_COLUMNS, context_schema_error
    from .execution_output_schema import (
        TABLE_COLUMNS as OUTPUT_COLUMNS,
        output_schema_error,
    )
    output_present = any(
        row[0] in OUTPUT_COLUMNS
        for row in conn.execute("SELECT name FROM sqlite_master")
    )
    if not tables and output_present:
        return "Execution outputs require execution history"
    context_present = any(
        row[0] in TABLE_COLUMNS
        for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'")
    )
    if not tables and context_present:
        return "Execution contexts require execution history"
    if not tables:
        return None if allow_missing else "Missing execution history schema"
    if tables.keys() != _TABLE_COLUMNS.keys():
        return "Incomplete execution history schema"
    metadata = conn.execute(
        "SELECT CASE WHEN typeof(schema_version) = 'integer' "
        "THEN schema_version ELSE NULL END FROM execution_history_metadata LIMIT 2"
    ).fetchall()
    if len(metadata) != 1 or len(metadata[0]) != 1 or metadata[0][0] not in (1, 2, 3):
        return "Unsupported execution history metadata/version"
    version = metadata[0][0]
    if version < 3 and output_present:
        return "Unexpected execution outputs for history v1/v2"
    if version == 1 and context_present:
        return "Unexpected execution contexts for history v1"
    if not allow_missing and version != SCHEMA_VERSION:
        return "Execution history migration required"
    for table, expected in _TABLE_COLUMNS.items():
        columns = tuple(
            (row[1], row[2].upper(), row[3], row[5])
            for row in conn.execute(f"PRAGMA table_info({table})")
        )
        if columns != expected:
            return f"{table} has invalid execution history columns"
        sql = _compact(tables[table])
        checks = _TABLE_CHECKS[table]
        if table == "execution_history_metadata":
            checks = tuple(
                check.replace("schema_version = 3", f"schema_version = {version}")
                for check in checks
            )
        if sql.count("check(") != len(checks) or any(
            f"check({_compact(check)})" not in sql for check in checks
        ):
            return f"{table} has invalid execution history checks"
        if conn.execute(f"PRAGMA foreign_key_list({table})").fetchone():
            return f"{table} has unexpected execution history foreign keys"
        for index in conn.execute(f"PRAGMA index_list({table})"):
            if not index[2]:
                continue
            if table == "execution_runs" and index[1] == "idx_execution_runs_identity":
                continue
            name = index[1].replace('"', '""')
            keys = tuple(
                row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
            )
            primary = tuple(column[0] for column in expected if column[3])
            if (
                index[3] != "pk"
                or index[4]
                or tuple(key[2] for key in keys) != primary
                or any(key[3] or key[4] != "BINARY" for key in keys)
            ):
                return f"{table} has unexpected execution history uniqueness"
        predicate = " AND ".join(
            f"({check})" for check in (*checks, *_ROW_BYTE_CHECKS[table])
        )
        if conn.execute(
            f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
        ).fetchone():
            return f"{table} contains invalid execution history records"
    metadata = conn.execute(
        "SELECT singleton_id,schema_version,storage_epoch FROM "
        "execution_history_metadata"
    ).fetchall()
    if len(metadata) != 1 or metadata[0][0] != 1 or metadata[0][1] != version:
        return "Unsupported execution history metadata/version"
    indexes = {row[1]: row for row in conn.execute("PRAGMA index_list(execution_runs)")}
    for name, (unique, columns) in _INDEXES.items():
        row = indexes.get(name)
        if row is None or bool(row[2]) != unique or row[4]:
            return f"Missing or invalid execution history index {name}"
        actual = tuple(conn.execute(f"PRAGMA index_xinfo({name})"))
        keys = tuple(item for item in actual if item[5])
        if tuple(item[2] for item in keys) != columns or any(
            item[3] or item[4] != "BINARY" for item in keys
        ):
            return f"Invalid execution history index {name}"
    if version >= 2:
        error = context_schema_error(conn)
        if error:
            return error
    return output_schema_error(conn) if version == 3 else None
 def install_context_schema(conn: sqlite3.Connection) -> None:
    from .execution_context_schema import install_context_schema as install
    install(conn)
 def install_output_schema(conn: sqlite3.Connection) -> None:
    from .execution_output_schema import install_output_schema as install
    install(conn)
 def migrate_execution_history(conn: sqlite3.Connection) -> None:
    """Install history, contexts and outputs atomically, without backfill."""
    conn.execute("SAVEPOINT execution_history_schema")
    try:
        error = history_schema_error(conn)
        if error:
            raise sqlite3.DatabaseError(error)
        present = conn.execute(
            "SELECT 1 FROM sqlite_master WHERE name IN "
            "('execution_runs','execution_history_metadata') LIMIT 1"
        ).fetchone()
        if not present:
            for table, columns in _TABLE_COLUMNS.items():
                definitions = [
                    f"{name} {kind}{' NOT NULL' if required else ''}"
                    f"{' PRIMARY KEY' if primary else ''}"
                    for name, kind, required, primary in columns
                ]
                definitions.extend(f"CHECK({check})" for check in _TABLE_CHECKS[table])
                conn.execute(f"CREATE TABLE {table} ({', '.join(definitions)})")
            for name, (unique, columns) in _INDEXES.items():
                conn.execute(
                    f"CREATE {'UNIQUE ' if unique else ''}INDEX {name} "
                    f"ON execution_runs({', '.join(columns)})"
                )
            conn.execute(
                "INSERT INTO execution_history_metadata VALUES(1,?,?)",
                (SCHEMA_VERSION, str(uuid4())),
            )
            install_context_schema(conn)
            install_output_schema(conn)
        else:
            version, epoch = conn.execute(
                "SELECT schema_version,storage_epoch FROM execution_history_metadata"
            ).fetchone()
            if version < SCHEMA_VERSION:
                conn.execute(
                    "ALTER TABLE execution_history_metadata RENAME TO "
                    "execution_history_metadata_old"
                )
                definitions = [
                    f"{name} {kind}{' NOT NULL' if required else ''}"
                    f"{' PRIMARY KEY' if primary else ''}"
                    for name, kind, required, primary in _TABLE_COLUMNS[
                        "execution_history_metadata"
                    ]
                ]
                definitions.extend(f"CHECK({check})" for check in _METADATA_CHECKS)
                conn.execute(
                    "CREATE TABLE execution_history_metadata "
                    f"({', '.join(definitions)})"
                )
                conn.execute(
                    "INSERT INTO execution_history_metadata VALUES(1,?,?)",
                    (SCHEMA_VERSION, epoch),
                )
                conn.execute("DROP TABLE execution_history_metadata_old")
                if version == 1:
                    install_context_schema(conn)
                install_output_schema(conn)
        # Read projections need reverse content lookup and the complete run order.
        for table, columns in (
            ("execution_prompt_snapshots", "raw_text_id,snapshot_id"),
            ("execution_contexts", "positive_snapshot_id,context_id"),
            ("execution_run_contexts", "context_id,run_id,anchor_id"),
            ("execution_runs", "observed_dequeue_at,session_id,queue_item_id,run_id"),
        ):
            conn.execute(
                f"CREATE INDEX IF NOT EXISTS idx_history_read_{table} "
                f"ON {table}({columns})"
            )
        error = history_schema_error(conn, allow_missing=False)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute("RELEASE execution_history_schema")
    except BaseException:
        conn.execute("ROLLBACK TO execution_history_schema")
        conn.execute("RELEASE execution_history_schema")
        raise
 def rotate_history_epoch(conn: sqlite3.Connection) -> str:
    """Publish a fresh staging epoch without rewriting restored run epochs."""
    epoch = str(uuid4())
    with conn:
        cursor = conn.execute(
            "UPDATE execution_history_metadata SET storage_epoch=? WHERE "
            "singleton_id=1",
            (epoch,),
        )
        if cursor.rowcount != 1:
            raise sqlite3.DatabaseError("Missing execution history metadata")
    return epoch
@@ -0,0 +1,218 @@
 """Owned schema and bounded restore validation for frozen output locators."""
 import sqlite3
 from .execution_history_schema import _compact, _uuid_check
 try:
    from ..utils.execution_outputs import (
        BUNDLE_OVERHEAD,
        MAX_BUNDLE_BYTES,
        MAX_IMAGE_POSITIONS,
        MAX_REFERENCE_BYTES,
        REFERENCE_OVERHEAD,
        STATE_REASONS,
        OutputBundle,
        OutputReference,
        validate_output_bundle,
    )
 except ImportError:
    from utils.execution_outputs import (
        BUNDLE_OVERHEAD,
        MAX_BUNDLE_BYTES,
        MAX_IMAGE_POSITIONS,
        MAX_REFERENCE_BYTES,
        REFERENCE_OVERHEAD,
        STATE_REASONS,
        OutputBundle,
        OutputReference,
        validate_output_bundle,
    )
 TABLE_COLUMNS = {
    "execution_output_capture": (
        ("run_id", "TEXT", 1, 1),
        ("state", "TEXT", 1, 0),
        ("reason", "TEXT", 1, 0),
        ("format_version", "INTEGER", 1, 0),
        ("source", "TEXT", 1, 0),
    ),
    "execution_run_outputs": (
        ("run_id", "TEXT", 1, 1),
        ("output_node_id", "TEXT", 1, 2),
        ("image_ordinal", "INTEGER", 1, 3),
        ("folder_type", "TEXT", 1, 0),
        ("subfolder", "TEXT", 1, 0),
        ("filename", "TEXT", 1, 0),
        ("path_style", "TEXT", 1, 0),
        ("root_key", "TEXT", 0, 0),
    ),
 }
 TABLE_FOREIGN_KEYS = {
    "execution_output_capture": (("run_id", "execution_runs", "run_id"),),
    "execution_run_outputs": (("run_id", "execution_output_capture", "run_id"),),
 }
 def _text_check(column: str, maximum: int, *, characters: int | None = None) -> str:
    return (
        f"typeof({column}) = 'text' AND length({column}) <= "
        f"{maximum if characters is None else characters} "
        f"AND length(CAST({column} AS BLOB)) <= {maximum} "
        f"AND instr(CAST({column} AS BLOB), x'00') = 0"
    )
 def _enum_check(column: str, values: tuple | frozenset) -> str:
    return (
        f"{_text_check(column, 32)} AND {column} IN ("
        + ",".join("'" + value + "'" for value in sorted(values))
        + ")"
    )
 _RUN_ID_CHECK = f"{_uuid_check('run_id')} AND {_text_check('run_id', 36)}"
 TABLE_CHECKS = {
    "execution_output_capture": (
        _RUN_ID_CHECK,
        _enum_check("state", tuple(STATE_REASONS)),
        _enum_check("reason", frozenset.union(*STATE_REASONS.values())),
        "typeof(format_version) = 'integer' AND format_version = 1",
        f"{_text_check('source', 18)} AND source = 'host_final_history'",
    ),
    "execution_run_outputs": (
        _RUN_ID_CHECK,
        f"{_text_check('output_node_id', 512, characters=128)} "
        "AND length(output_node_id) >= 1",
        "typeof(image_ordinal) = 'integer' AND image_ordinal BETWEEN 0 AND 255",
        _enum_check("folder_type", ("output", "temp")),
        _text_check("subfolder", 2048),
        f"{_text_check('filename', 1024)} AND length(filename) >= 1",
        _enum_check("path_style", ("posix", "windows")),
        f"root_key IS NULL OR ({_text_check('root_key', 64)} "
        "AND length(root_key) = 64 AND root_key NOT GLOB '*[^0-9a-f]*')",
    ),
 }
 def _table_sql(table: str) -> str:
    columns = TABLE_COLUMNS[table]
    definitions = [
        f"{name} {kind}{' NOT NULL' if required else ''}"
        for name, kind, required, _ in columns
    ]
    primary = [name for name, _, _, order in columns if order]
    definitions.append(f"PRIMARY KEY ({', '.join(primary)})")
    definitions.extend(f"CHECK({check})" for check in TABLE_CHECKS[table])
    definitions.extend(
        f"FOREIGN KEY ({column}) REFERENCES {target}({key}) "
        "ON UPDATE NO ACTION ON DELETE NO ACTION"
        for column, target, key in TABLE_FOREIGN_KEYS[table]
    )
    return f"CREATE TABLE {table} ({', '.join(definitions)})"
 def install_output_schema(conn: sqlite3.Connection) -> None:
    """Install the two owned tables within the caller's migration transaction."""
    for table in TABLE_COLUMNS:
        conn.execute(_table_sql(table))
 def output_schema_error(conn: sqlite3.Connection) -> str | None:
    """Reject altered schemas and invalid rows before returning bounded DTOs."""
    if conn.execute(
        "SELECT 1 FROM sqlite_master WHERE type IN ('trigger','view') LIMIT 1"
    ).fetchone():
        return "Unsupported execution output trigger/view"
    tables = {
        row[0]: row[1]
        for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
        if row[0] in TABLE_COLUMNS
    }
    if tables.keys() != TABLE_COLUMNS.keys():
        return "Incomplete execution output schema"
    for table, expected in TABLE_COLUMNS.items():
        columns = tuple(
            (row[1], row[2].upper(), row[3], row[5])
            for row in conn.execute(f"PRAGMA table_xinfo({table})")
        )
        if columns != expected or _compact(tables[table]) != _compact(
            _table_sql(table)
        ):
            return f"{table} has invalid execution output schema"
        foreign = tuple(
            sorted(
                (row[3], row[2], row[4], row[5], row[6], row[7])
                for row in conn.execute(f"PRAGMA foreign_key_list({table})")
            )
        )
        if foreign != tuple(
            (*key, "NO ACTION", "NO ACTION", "NONE")
            for key in TABLE_FOREIGN_KEYS[table]
        ):
            return f"{table} has invalid execution output foreign keys"
        for index in conn.execute(f"PRAGMA index_list({table})"):
            name = index[1].replace('"', '""')
            keys = tuple(
                row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
            )
            if (
                not index[2]
                or index[3] != "pk"
                or index[4]
                or tuple(key[2] for key in keys)
                != tuple(column[0] for column in expected if column[3])
                or any(key[3] or key[4] != "BINARY" for key in keys)
            ):
                return f"{table} has unexpected execution output index/uniqueness"
        # Run every scalar guard in SQLite before any evidence text is fetched.
        predicate = " AND ".join(f"({check})" for check in TABLE_CHECKS[table])
        if conn.execute(
            f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
        ).fetchone():
            return f"{table} contains invalid execution output records"
        if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
            return f"{table} contains orphan execution output records"
    reference_bytes = f"{REFERENCE_OVERHEAD} + " + " + ".join(
        f"coalesce(length(CAST({column} AS BLOB)), 0)"
        for column in (
            "output_node_id",
            "folder_type",
            "subfolder",
            "filename",
            "path_style",
            "root_key",
        )
    )
    if conn.execute(
        "SELECT 1 FROM execution_run_outputs "
        f"WHERE ({reference_bytes}) > {MAX_REFERENCE_BYTES} LIMIT 1"
    ).fetchone():
        return "Oversized execution output reference"
    if conn.execute(
        "SELECT 1 FROM execution_run_outputs GROUP BY run_id "
        f"HAVING COUNT(*) > {MAX_IMAGE_POSITIONS} OR "
        f"{BUNDLE_OVERHEAD} + SUM({reference_bytes}) > {MAX_BUNDLE_BYTES} LIMIT 1"
    ).fetchone():
        return "Oversized execution output bundle"
    try:
        for run_id, state, reason, version, source in conn.execute(
            "SELECT run_id,state,reason,format_version,source "
            "FROM execution_output_capture"
        ):
            references = tuple(
                OutputReference(*row)
                for row in conn.execute(
                    "SELECT output_node_id,image_ordinal,folder_type,subfolder,"
                    "filename,path_style,root_key FROM execution_run_outputs "
                    "WHERE run_id=? ORDER BY output_node_id,image_ordinal",
                    (run_id,),
                )
            )
            validate_output_bundle(
                OutputBundle(references, state, reason, version, source)
            )
    except (ValueError, TypeError, OverflowError, sqlite3.Error) as exc:
        return f"Invalid execution output content: {type(exc).__name__}"
    return None
@@ -0,0 +1,159 @@
 """Immutable run-owned output references, independent of mutable galleries."""
 import sqlite3
 from contextlib import ExitStack
 from typing import Callable, ContextManager
 from .run_attribution import notify_outputs_sealed
 try:
    from ..utils.execution_history_events import RunFinish, encode_command
    from ..utils.execution_outputs import OutputBundle, validate_output_bundle
 except ImportError:
    from utils.execution_history_events import RunFinish, encode_command
    from utils.execution_outputs import OutputBundle, validate_output_bundle
 _REFERENCE_FIELDS = (
    "output_node_id",
    "image_ordinal",
    "folder_type",
    "subfolder",
    "filename",
    "path_style",
    "root_key",
 )
 _REFERENCE_COLUMNS = ",".join(_REFERENCE_FIELDS)
 def _capture(conn: sqlite3.Connection, run_id: str) -> sqlite3.Row | None:
    capture = conn.execute(
        "SELECT state,reason,format_version,source FROM execution_output_capture "
        "WHERE run_id=?",
        (run_id,),
    ).fetchone()
    # Legacy callers may disable FK enforcement; never adopt orphan evidence.
    if capture is None:
        if conn.execute(
            "SELECT 1 FROM execution_run_outputs WHERE run_id=? LIMIT 1", (run_id,)
        ).fetchone():
            raise sqlite3.DatabaseError("Output references have no capture header")
    elif (
        conn.execute(
            "SELECT 1 FROM execution_runs WHERE run_id=?", (run_id,)
        ).fetchone()
        is None
    ):
        raise sqlite3.DatabaseError("Output capture has no execution parent")
    return capture
 def _references(conn: sqlite3.Connection, run_id: str) -> list[tuple]:
    return [
        tuple(row)
        for row in conn.execute(
            f"SELECT {_REFERENCE_COLUMNS} FROM execution_run_outputs "
            "WHERE run_id=? ORDER BY output_node_id,image_ordinal",
            (run_id,),
        )
    ]
 class ExecutionOutputOperations:
    """Compose with ExecutionHistoryOperations on PromptDatabase."""
    def write_execution_outputs_for_finish(
        self,
        command: RunFinish,
        bundle: OutputBundle,
        *,
        generation_guard: Callable[[], ContextManager[bool]] | None = None,
    ) -> bool:
        encode_command(command)
        if type(command) is not RunFinish:
            raise ValueError("Expected execution finish command")
        validate_output_bundle(bundle)
        header = (bundle.state, bundle.reason, bundle.format_version, bundle.source)
        prepared = [
            tuple(getattr(reference, field) for field in _REFERENCE_FIELDS)
            for reference in bundle.references
        ]
        # Enter after BEGIN and retain the guard until COMMIT/ROLLBACK finishes.
        with ExitStack() as guards:
            with self._execution_transaction(command.storage_epoch) as conn:
                if generation_guard is not None and not guards.enter_context(
                    generation_guard()
                ):
                    return False
                parent = conn.execute(
                    "SELECT run_id FROM execution_runs WHERE session_id=? "
                    "AND queue_item_id=? AND storage_epoch=?",
                    (command.session_id, command.queue_item_id, command.storage_epoch),
                ).fetchone()
                if parent is None:
                    return False
                run_id = parent[0]
                capture = _capture(conn, run_id)
                if capture is not None:
                    if capture[0] != "conflicting" and (
                        tuple(capture) != header
                        or _references(conn, run_id) != prepared
                    ):
                        conn.execute(
                            "UPDATE execution_output_capture SET "
                            "state='conflicting',reason='conflicting' WHERE run_id=?",
                            (run_id,),
                        )
                    return True
                conn.execute(
                    "INSERT INTO execution_output_capture "
                    "(run_id,state,reason,format_version,source) VALUES(?,?,?,?,?)",
                    (run_id, *header),
                )
                conn.executemany(
                    f"INSERT INTO execution_run_outputs (run_id,{_REFERENCE_COLUMNS}) "
                    "VALUES(?,?,?,?,?,?,?,?)",
                    [(run_id, *reference) for reference in prepared],
                )
        # Committed: images waiting for this run may now be attributed (#342)
        notify_outputs_sealed(
            run_id, [dict(zip(_REFERENCE_FIELDS, row)) for row in prepared]
        )
        return True
    def get_execution_outputs(self, run_id: str) -> dict:
        """Read frozen evidence in one snapshot without resolving its locators."""
        conn = self.model.get_connection()
        own_transaction = not conn.in_transaction
        if own_transaction:
            conn.execute("BEGIN")
        try:
            capture = _capture(conn, run_id)
            if capture is None:
                return {
                    "state": "not_captured",
                    "reason": None,
                    "format_version": None,
                    "source": None,
                    "references": [],
                }
            references = []
            for row in _references(conn, run_id):
                reference = dict(zip(_REFERENCE_FIELDS, row))
                reference.update(
                    file_identity="unverified",
                    newly_generated="unknown",
                    context_membership="unknown",
                    transient=reference["folder_type"] == "temp",
                )
                references.append(reference)
            return {
                "state": capture[0],
                "reason": capture[1],
                "format_version": capture[2],
                "source": capture[3],
                "references": references,
            }
        finally:
            if own_transaction:
                conn.rollback()
@@ -0,0 +1,380 @@
 """The read-only health check behind System › Overview and Maintenance.
 ``run_health_checks`` opens its own ``mode=ro`` connection, reads one
 snapshot and closes the connection in ``finally``; it never writes. Each
 check reports ``passed``, ``issues`` (with an exact ``count``; the
 integrity check stops at ``INTEGRITY_LIMIT`` problems and then says
 ``capped``),
 ``unavailable`` (it could not be checked here, so ``count`` is None, or
 ``missing_files`` checked only some sources and found nothing) or ``failed``
 (it raised; the other checks still run). ``missing_files`` also reports each
 source and whether the check was ``partial``; it reads the image rows inside
 the snapshot and checks their files after the snapshot ended, so a slow source
 never holds the WAL. At most ``MAX_DETAIL`` detail strings are returned per
 check, and none of them carries a server path.
 """
 import datetime
 import logging
 import ntpath
 import os
 import sqlite3
 from pathlib import PurePosixPath, PureWindowsPath
 from urllib.parse import quote
 try:
    from ..utils.hashing import generate_prompt_hash
    from .models import normalize_image_path
 except ImportError:
    from utils.hashing import generate_prompt_hash
    from database.models import normalize_image_path
 logger = logging.getLogger("prompt_manager.database.health")
 MAX_DETAIL = 10  # detail strings returned per check
 INTEGRITY_LIMIT = 100  # PRAGMA integrity_check(N): problems SQLite reports
 BATCH_ROWS = 100  # image rows read per fetch
 REVIEW_UNAVAILABLE = "Needs review is not available yet."
 CHECK_FAILED = "The check could not finish."
 # Check ids and the labels of the concept's health report, in its order;
 # pair uniqueness and consistency are the added rows (D63, now D95).
 CHECKS = (
    ("integrity", "Integrity check"),
    ("foreign_keys", "Foreign keys & pair references"),
    ("canonical_hashes", "Canonical hashes"),
    ("pair_uniqueness", "Pair uniqueness"),
    ("missing_files", "Missing files"),
    ("ambiguous_metadata", "Ambiguous metadata"),
    ("consistency", "Consistency"),
 )
 _DATABASE_CHECKS = (
    "integrity",
    "foreign_keys",
    "canonical_hashes",
    "pair_uniqueness",
    "missing_files",
    "consistency",
 )
 def iter_consistency_issues(conn):
    """The legacy consistency issues: tag and image rows of a missing prompt."""
    for ref in conn.execute(
        "SELECT pt.prompt_id FROM prompt_tags pt "
        "LEFT JOIN prompts p ON pt.prompt_id = p.id WHERE p.id IS NULL"
    ):
        yield f"prompt_tags entry references non-existent prompt {ref[0]}"
    for ref in conn.execute(
        "SELECT gi.id, gi.prompt_id FROM generated_images gi "
        "LEFT JOIN prompts p ON gi.prompt_id = p.id WHERE p.id IS NULL"
    ):
        yield f"Image {ref[0]} references non-existent prompt {ref[1]}"
 def _result(count, detail, *, status=None):
    return {
        "status": status or ("issues" if count else "passed"),
        "count": count,
        "detail": detail[:MAX_DETAIL],
    }
 def _counted(lines):
    """(count, first MAX_DETAIL lines) of an iterable, consumed once."""
    count, detail = 0, []
    for line in lines:
        count += 1
        if len(detail) < MAX_DETAIL:
            detail.append(line)
    return _result(count, detail)
 def readonly_uri(db_path, *, windows=None):
    """The ``mode=ro`` SQLite URI of *db_path*, on this platform or the given one.
    pathlib spells a UNC path ``\\\\server\\share\\x.db`` as
    ``file://server/share/x.db``, a remote authority SQLite refuses to open.
    With an empty authority (``file:////server/share/x.db``) SQLite hands
    Windows the ``//server/share`` path, which it opens as the UNC path.
    """
    if windows is None:
        windows = os.name == "nt"
        db_path = os.path.abspath(db_path)
    if not windows:
        return PurePosixPath(db_path).as_uri() + "?mode=ro"
    path = PureWindowsPath(db_path)
    if ntpath.splitdrive(str(path))[0].startswith("\\\\"):
        return "file:" + quote("//" + path.as_posix()) + "?mode=ro"
    return path.as_uri() + "?mode=ro"
 def _integrity(conn, _roots):
    rows = [
        row[0] for row in conn.execute(f"PRAGMA integrity_check({INTEGRITY_LIMIT})")
    ]
    if rows == ["ok"]:
        return {**_result(0, []), "capped": False}
    # SQLite stops listing at the limit: there may be more problems than that.
    return {**_result(len(rows), rows), "capped": len(rows) >= INTEGRITY_LIMIT}
 def _foreign_keys(conn, _roots):
    return _counted(
        f"{table} row {rowid} references a missing {parent} row"
        for table, rowid, parent, _fkid in conn.execute("PRAGMA foreign_key_check")
    )
 def _prompt_hash(text):
    return generate_prompt_hash(text) if isinstance(text, str) else None
 def _canonical_hashes(conn, _roots):
    conn.create_function("pm_prompt_hash", 1, _prompt_hash, deterministic=True)
    return _counted(
        f"Prompt {row[0]}: the stored hash does not match its text"
        for row in conn.execute(
            "SELECT id FROM prompts WHERE hash IS NOT pm_prompt_hash(text) ORDER BY id"
        )
    )
 def _pair_uniqueness(conn, _roots):
    groups = conn.execute(
        "SELECT positive_prompt_id,negative_prompt_id,COUNT(*) FROM prompt_pairs "
        "GROUP BY positive_prompt_id,negative_prompt_id HAVING COUNT(*)>1 "
        "ORDER BY positive_prompt_id,negative_prompt_id"
    ).fetchall()
    detail = [
        f"Positive prompt {positive} with "
        + ("no negative" if negative is None else f"negative {negative}")
        + f": {rows} pair rows"
        for positive, negative, rows in groups
    ]
    # The surplus rows: what keeping one row per pair would leave out.
    return _result(sum(rows - 1 for _p, _n, rows in groups), detail)
 def _consistency(conn, _roots):
    return _counted(iter_consistency_issues(conn))
 def _readable(root):
    return os.path.isdir(root) and os.access(root, os.R_OK | os.X_OK)
 def _root_prefix(root):
    return normalize_image_path(root).rstrip("\\/") + os.sep
 def _source_name(root, index):
    """A source's folder name for the report (never its server path)."""
    return os.path.basename(str(root).rstrip("\\/")) or f"Source {index + 1}"
 def _owner(path, prefixes):
    """Index of the longest source prefix *path* lies under, or None."""
    owner, longest = None, -1
    for index, prefix in enumerate(prefixes):
        if path.startswith(prefix) and len(prefix) > longest:
            owner, longest = index, len(prefix)
    return owner
 def _image_owner(image_path, file_path, prefixes):
    spelled = file_path or normalize_image_path(image_path)
    owner = _owner(spelled, prefixes)
    if owner is None:
        owner = _owner(normalize_image_path(os.path.realpath(image_path)), prefixes)
    return owner
 def readable_roots(roots):
    """Whether each source can be read now (an unmounted drive cannot)."""
    return [_readable(root) for root in roots]
 def link_states(rows, roots, readable):
    """``(row, owner, state)`` for each image row ``(id, image_path, filename,
    file_path)``: ``owner`` is the index of the source it lies under (None
    outside every source) and ``state`` is ``missing``, ``present``,
    ``unavailable`` (its source cannot be read, so the file is not looked at)
    or ``outside``. ``readable`` is ``readable_roots(roots)``. The health
    check's ``missing_files`` and the maintenance missing-link plan both
    judge links this way, so their counts agree.
    """
    prefixes = [_root_prefix(root) for root in roots]
    for row in rows:
        owner = _image_owner(row[1], row[3], prefixes)
        if owner is None:
            yield row, None, "outside"
        elif not readable[owner]:
            yield row, owner, "unavailable"
        elif os.path.exists(row[1]):
            yield row, owner, "present"
        else:
            yield row, owner, "missing"
 def _unavailable_line(unavailable, *, partial):
    noun = "source is" if unavailable == 1 else "sources are"
    effect = "its files were" if unavailable == 1 else "their files were"
    if not partial:
        effect = "missing files were"
    return f"{unavailable} image {noun} unavailable, so {effect} not counted."
 def _missing_files(conn, roots):
    """Links whose file is gone, counted per source (only readable ones).
    Each source reports ``checked`` with its own count, or ``unavailable``
    (an unmounted drive) with no count: its links are never turned into a
    count, while the readable sources still are. The check's ``count`` is the
    readable sources' total (None when none is readable) and ``partial`` says
    some sources were not checked. Links outside every source are named in the
    detail, not counted.
    Inside the snapshot this only reads the image rows; it returns the step
    that checks their files, which runs after the snapshot ended.
    """
    if not roots:
        result = _result(None, ["No image source is available."], status="unavailable")
        return {**result, "partial": False, "sources": []}
    rows = []
    cursor = conn.execute(
        "SELECT id,image_path,filename,file_path FROM generated_images ORDER BY id"
    )
    while batch := cursor.fetchmany(BATCH_ROWS):
        rows.extend(batch)
    return lambda: _check_files(rows, roots)
 def _check_files(rows, roots):
    """The ``missing_files`` result of the image *rows*, from the file system."""
    readable = readable_roots(roots)
    sources = [
        {
            "name": _source_name(root, index),
            "status": "checked" if ok else "unavailable",
            "count": 0 if ok else None,
        }
        for index, (root, ok) in enumerate(zip(roots, readable))
    ]
    unavailable = readable.count(False)
    if not any(readable):
        result = _result(
            None, [_unavailable_line(unavailable, partial=False)], status="unavailable"
        )
        return {**result, "partial": False, "sources": sources}
    outside, missing = 0, []
    for row, owner, state in link_states(rows, roots, readable):
        if state == "outside":
            outside += 1
        elif state == "missing":
            image_id, image_path, filename, _file_path = row
            sources[owner]["count"] += 1
            if len(missing) < MAX_DETAIL:
                name = filename or os.path.basename(image_path)
                missing.append(f"Image {image_id}: {name} is missing")
    count = sum(source["count"] or 0 for source in sources)
    detail = [_unavailable_line(unavailable, partial=True)] if unavailable else []
    detail += missing
    if outside:
        links = "link" if outside == 1 else "links"
        verb = "was" if outside == 1 else "were"
        detail.append(
            f"{outside} {links} outside the image sources {verb} not checked."
        )
    status = None if count or not unavailable else "unavailable"
    result = _result(count, detail, status=status)
    return {**result, "partial": bool(unavailable), "sources": sources}
 def _ambiguous(review_count):
    if review_count is None:
        return _result(None, [REVIEW_UNAVAILABLE], status="unavailable")
    return _result(review_count, [])
 def overall_status(checks) -> str:
    """``passed`` only when every check passed; any check not run is incomplete."""
    statuses = {check["status"] for check in checks}
    if statuses & {"unavailable", "failed"} or any(
        check.get("partial") for check in checks
    ):
        return "incomplete"
    return "issues" if "issues" in statuses else "passed"
 def _failed():
    return _result(None, [CHECK_FAILED], status="failed")
 def _run_check(check_id, step, *args):
    """One check's result, or None (reported as failed) when it raised."""
    try:
        return step(*args)
    except Exception:  # any error fails this check only; the others still run
        logger.exception(f"Health check {check_id} failed")
        return None
 def _run_database_checks(db_path, roots):
    """Each database check's result (None when it raised), from one snapshot.
    A check may return a step instead of its result: the step runs after the
    snapshot ended and the connection closed (``missing_files``' file checks).
    """
    results = dict.fromkeys(_DATABASE_CHECKS)
    try:
        conn = sqlite3.connect(readonly_uri(db_path), uri=True)
    except sqlite3.Error:
        logger.exception("Health check could not open the database")
        return results
    try:
        conn.execute("BEGIN")  # one read snapshot for every check
        for check_id in _DATABASE_CHECKS:
            runner = globals()["_" + check_id]  # looked up per run (tests patch it)
            results[check_id] = _run_check(check_id, runner, conn, roots)
    except sqlite3.Error:
        logger.exception("Health check could not read the database")
    finally:
        conn.close()
    for check_id, result in results.items():
        if callable(result):
            results[check_id] = _run_check(check_id, result)
    return results
 class HealthOperations:
    """Mixin for PromptDatabase: the read-only System health check."""
    def run_health_checks(self, roots, *, review_count=None) -> dict:
        """Run every check and return a ``HealthReport``.
        Args:
            roots: The image source directories to judge missing files under:
                the effective sources plus any configured source that is not
                available (reported as an unavailable source of ``missing_files``).
            review_count: The Needs-review pending count, or None while no
                review queue exists (``ambiguous_metadata`` is then unavailable).
        """
        if review_count is not None and (
            type(review_count) is not int or review_count < 0
        ):
            raise ValueError("review_count must be a non-negative integer or None")
        roots = [str(root) for root in roots]
        database = _run_database_checks(self.model.db_path, roots)
        checks = []
        for check_id, label in CHECKS:
            if check_id == "ambiguous_metadata":
                result = _ambiguous(review_count)
            else:
                result = database[check_id] or _failed()
            checks.append({"id": check_id, "label": label, **result})
        return {
            "success": True,
            "checked_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
            "status": overall_status(checks),
            "checks": checks,
        }
@@ -0,0 +1,140 @@
 """Bounded exact-path saved display within the caller's history snapshot."""
 import os
 import sqlite3
 try:
    from ..utils.inspection_response import (
        MAX_LIBRARY_BYTES,
        _library_projection,
        library_metadata,
    )
 except ImportError:
    from utils.inspection_response import (
        MAX_LIBRARY_BYTES,
        _library_projection,
        library_metadata,
    )
 def _display_bytes(value: bytes | None) -> str | None:
    if value is None:
        return None
    return value.decode("utf-8", "replace")
 def _tags(
    conn: sqlite3.Connection, owner: int, *, pair: bool
 ) -> tuple[list[str], bool]:
    table, column = (
        ("prompt_pair_tags", "pair_id") if pair else ("prompt_tags", "prompt_id")
    )
    tags = []
    used = 0
    cursor = conn.execute(
        f"SELECT substr(CAST(t.name AS BLOB),1,?) FROM {table} pt "
        f"JOIN tags t ON t.id=pt.tag_id WHERE pt.{column}=? LIMIT 129",
        (MAX_LIBRARY_BYTES + 1, owner),
    )
    for row in cursor:
        used += len(row[0])
        if used > MAX_LIBRARY_BYTES or len(tags) == 128:
            return [], True
        tags.append(_display_bytes(row[0]))
    return tags, False
 def _saved_projection(conn: sqlite3.Connection, row: sqlite3.Row) -> tuple[dict, dict]:
    saved = {"prompt_id": row["prompt_id"]}
    pairs = list(
        conn.execute(
            "SELECT pp.id,pp.positive_prompt_id,"
            "substr(CAST(pp.title AS BLOB),1,?) AS title,"
            "substr(CAST(pp.category AS BLOB),1,?) AS category "
            "FROM image_prompt_pairs ip JOIN prompt_pairs pp ON pp.id=ip.pair_id "
            "WHERE ip.image_id=? LIMIT 2",
            (MAX_LIBRARY_BYTES + 1, MAX_LIBRARY_BYTES + 1, row["id"]),
        )
    )
    if len(pairs) > 1:
        return saved, _library_projection({"status": "limited"})
    pair = pairs[0] if pairs else None
    title = pair["title"] if pair is not None else None
    recipe_tags, limited = (
        _tags(conn, pair["id"], pair=True) if pair is not None else ([], False)
    )
    positive_tags = []
    if not recipe_tags and not limited:
        # D39: only read the positive's tags when they are what the panel shows.
        positive_tags, limited = _tags(conn, row["prompt_id"], pair=False)
    metadata = library_metadata(
        row["category"],
        positive_tags,
        (pair["category"], recipe_tags) if pair is not None else None,
    )
    limited = limited or any(
        value is not None and len(value) > MAX_LIBRARY_BYTES
        for value in (title, metadata["category"])
    )
    if pair is not None:
        saved["pair"] = {
            "id": pair["id"],
            "positive_prompt_id": pair["positive_prompt_id"],
        }
    library = _library_projection(
        {
            "status": "linked",
            "positive_id": row["prompt_id"],
            "pair_id": pair["id"] if pair is not None else None,
            "title": _display_bytes(title),
            **metadata,
            "category": _display_bytes(metadata["category"]),
        }
    )
    if limited:
        library = _library_projection({"status": "limited", "tags": []})
    return saved, library
 def read_history_gallery(
    conn: sqlite3.Connection, paths: tuple[str, ...]
 ) -> dict[str, dict]:
    """Read at most two rows/path and two pair associations/unique row; no writes."""
    if not conn.in_transaction:
        raise sqlite3.ProgrammingError("Saved history display requires caller snapshot")
    if (
        type(paths) is not tuple
        or len(paths) > 100
        or any(
            type(path) is not str
            or not os.path.isabs(path)
            or path != os.path.normcase(os.path.normpath(path))
            or "\0" in path
            for path in paths
        )
    ):
        raise ValueError("Invalid history gallery paths")
    result = {}
    for path in dict.fromkeys(paths):
        rows = list(
            conn.execute(
                "SELECT gi.id,gi.prompt_id,"
                "substr(CAST(p.category AS BLOB),1,?) AS category "
                "FROM generated_images gi JOIN prompts p ON p.id=gi.prompt_id "
                "WHERE gi.file_path=? OR gi.image_path=? LIMIT 2",
                (MAX_LIBRARY_BYTES + 1, path, path),
            )
        )
        match = "none" if not rows else "unique" if len(rows) == 1 else "ambiguous"
        saved, library = None, _library_projection(
            {"status": "none" if not rows else "ambiguous"}
        )
        if match == "unique":
            saved, library = _saved_projection(conn, rows[0])
        result[path] = {
            "gallery_match": match,
            "gallery_image_id": rows[0]["id"] if match == "unique" else None,
            "saved": saved,
            "library": library,
        }
    return result
@@ -0,0 +1,585 @@
 """Snapshot-owned, bounded projections of immutable execution evidence."""
 import codecs
 from contextlib import contextmanager
 from contextvars import ContextVar
 from dataclasses import dataclass
 import hashlib
 import json
 import re
 import sqlite3
 from typing import Iterator
 from .execution_contexts import _hints, _values, content_digest
 from .execution_history import HistoryEpochChanged
 from .execution_outputs import _capture, _REFERENCE_COLUMNS
 from .library import _ELIGIBLE
 RunKey = tuple[int, str, int, str]
 class HistoryCaptureChanged(ValueError):
    """A continuation no longer describes the stored output capture header."""
@dataclass(frozen=True)
 class WindowPosition:
    raw_text_id: str
    upper: RunKey
    before: RunKey
@dataclass(frozen=True)
 class OutputPosition:
    capture_fingerprint: str
    after: tuple[str, int]
 _DIGEST = re.compile(r"[0-9a-f]{64}")
 _UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
 def _valid_digest(value: object) -> bool:
    return type(value) is str and _DIGEST.fullmatch(value) is not None
 def _valid_run_key(value: object) -> bool:
    return (
        type(value) is tuple
        and len(value) == 4
        and all(type(value[i]) is int and 0 <= value[i] <= 2**63 - 1 for i in (0, 2))
        and all(
            type(value[i]) is str and _UUID.fullmatch(value[i]) is not None
            for i in (1, 3)
        )
    )
 def _valid_coordinate(value: object) -> bool:
    if type(value) is not tuple or len(value) != 2:
        return False
    node, ordinal = value
    if not (
        type(node) is str
        and 1 <= len(node) <= 128
        and "\0" not in node
        and type(ordinal) is int
        and 0 <= ordinal <= 255
    ):
        return False
    try:
        return len(node.encode("utf-8")) <= 512
    except UnicodeError:
        return False
 def _validate_window_position(position: WindowPosition | None) -> None:
    if position is not None and not (
        type(position) is WindowPosition
        and _valid_digest(position.raw_text_id)
        and _valid_run_key(position.upper)
        and _valid_run_key(position.before)
    ):
        raise ValueError("Invalid history window position")
 def _validate_output_position(position: OutputPosition | None) -> None:
    if position is not None and not (
        type(position) is OutputPosition
        and _valid_digest(position.capture_fingerprint)
        and _valid_coordinate(position.after)
    ):
        raise ValueError("Invalid history output position")
 _snapshots: ContextVar[tuple] = ContextVar("history_read_snapshots", default=())
 _KEY = "(r.observed_dequeue_at,r.session_id,r.queue_item_id,r.run_id)"
 _ORDER = (
    "r.observed_dequeue_at DESC,r.session_id COLLATE BINARY DESC,"
    "r.queue_item_id DESC,r.run_id COLLATE BINARY DESC"
 )
 # Stale-hint rows a newest-pair lookup inspects before it reports none.
 _PAIR_RUN_CANDIDATES = 16
 _MATCHING = (
    "SELECT DISTINCT rc.run_id FROM execution_prompt_snapshots s "
    "JOIN execution_contexts c ON c.positive_snapshot_id=s.snapshot_id "
    "JOIN execution_run_contexts rc ON rc.context_id=c.context_id "
    "WHERE s.raw_text_id=?"
 )
 def _limit(limit: int) -> None:
    if type(limit) is not int or not 1 <= limit <= 100:
        raise ValueError("History limit must be an integer from 1 to 100")
 def _key(row: dict) -> RunKey:
    return tuple(
        row[name]
        for name in ("observed_dequeue_at", "session_id", "queue_item_id", "run_id")
    )
 def _context_capture(conn: sqlite3.Connection, run_id: str) -> dict:
    row = conn.execute(
        "SELECT state,reason,format_version FROM execution_context_capture "
        "WHERE run_id=?",
        (run_id,),
    ).fetchone()
    return (
        dict(row)
        if row is not None
        else {"state": "not_captured", "reason": "missing", "format_version": None}
    )
 def _output_header(conn: sqlite3.Connection, run_id: str) -> tuple[dict, int | None]:
    row = _capture(conn, run_id)
    if row is None:
        return (
            dict(state="not_captured", reason=None, format_version=None, source=None),
            None,
        )
    count = conn.execute(
        "SELECT COUNT(*) FROM execution_run_outputs WHERE run_id=?", (run_id,)
    ).fetchone()[0]
    return dict(row), count
 def _list_status(capture: dict) -> str:
    if capture["state"] == "not_captured":
        return "not_captured"
    return "complete" if capture["state"] == "complete" else "incomplete"
 def _output_summary(conn: sqlite3.Connection, run_id: str) -> dict:
    capture, count = _output_header(conn, run_id)
    return dict(
        capture, retained_occurrence_count=count, list_status=_list_status(capture)
    )
 def _fingerprint(capture: dict, count: int | None) -> str:
    return hashlib.sha256(
        json.dumps(
            [
                1,
                *(capture[k] for k in ("state", "reason", "format_version", "source")),
                count,
            ],
            ensure_ascii=False,
            allow_nan=False,
            separators=(",", ":"),
        ).encode("utf-8")
    ).hexdigest()
 def _receipt(run_id: str, row: sqlite3.Row) -> dict:
    return {
        "occurrence": {
            "run_id": run_id,
            "output_node_id": row["output_node_id"],
            "image_ordinal": row["image_ordinal"],
        },
        "reference": {
            key: row[key]
            for key in (
                "folder_type",
                "subfolder",
                "filename",
                "path_style",
                "root_key",
            )
        },
        "file_identity": "unverified",
        "newly_generated": "unknown",
        "context_membership": "unknown",
        "transient": row["folder_type"] == "temp",
    }
 class HistoryReadOperations:
    """Read evidence only inside an explicitly owned, epoch-first snapshot."""
    @contextmanager
    def recorded_history_snapshot(
        self, expected_epoch: str | None = None
    ) -> Iterator[str]:
        conn = self.model.get_connection()
        if conn.in_transaction:
            raise sqlite3.ProgrammingError(
                "History reads require their own transaction"
            )
        conn.execute("BEGIN")
        token = _snapshots.set((*_snapshots.get(), conn))
        try:
            epoch = self.get_execution_epoch()
            if expected_epoch is not None and epoch != expected_epoch:
                raise HistoryEpochChanged(epoch)
            yield epoch
        finally:
            try:
                conn.rollback()
            finally:
                _snapshots.reset(token)
    def _history_connection(self) -> sqlite3.Connection:
        conn = self.model.get_connection()
        if not conn.in_transaction or conn not in _snapshots.get():
            raise sqlite3.ProgrammingError(
                "History read requires recorded_history_snapshot"
            )
        return conn
    def list_history_window(
        self,
        positive_id: int,
        *,
        limit: int = 40,
        after: WindowPosition | None = None,
        restart: bool = False,
    ) -> dict | None:
        conn = self._history_connection()
        _limit(limit)
        _validate_window_position(after)
        if type(restart) is not bool or (
            restart and (after is None or after.before != after.upper)
        ):
            raise ValueError("Invalid history restart position")
        saved = conn.execute(
            "SELECT CASE WHEN typeof(p.text)='text' AND "
            "length(CAST(p.text AS BLOB))<=65536 THEN p.text ELSE NULL END AS text "
            f"FROM prompts p WHERE p.id=? AND ({_ELIGIBLE})",
            (positive_id,),
        ).fetchone()
        if saved is None:
            return None
        text = saved["text"]
        raw_id = None if text is None else content_digest("text", (text,))
        if after is not None and after.raw_text_id != raw_id:
            raise ValueError("History position text scope changed")
        group = {
            "positive_id": positive_id,
            "raw_text_id": raw_id,
            "relation": "exact_frozen_raw_text",
            "scope_status": "uncapturable_text" if text is None else "no_matches",
        }
        result = dict(
            group=group,
            recorded_run_count=0,
            items=[],
            context_summaries=[],
            has_more=False,
            next_position=None,
            resume_position=None,
        )
        if (
            raw_id is None
            or conn.execute(
                "SELECT 1 FROM execution_context_texts WHERE text_id=? "
                "AND text_value=? COLLATE BINARY",
                (raw_id, text),
            ).fetchone()
            is None
        ):
            return result
        where = f"r.run_id IN ({_MATCHING})"
        params = [raw_id]
        if after is not None:
            where += f" AND {_KEY}<=(?,?,?,?)"
            params.extend(after.upper)
        count = conn.execute(
            f"SELECT COUNT(*) FROM execution_runs r WHERE {where}", params
        ).fetchone()[0]
        result["recorded_run_count"] = count
        group["scope_status"] = "matched" if count else "no_matches"
        if after is not None and not restart:
            where += f" AND {_KEY}<(?,?,?,?)"
            params.extend(after.before)
        rows = [
            dict(row)
            for row in conn.execute(
                f"SELECT r.* FROM execution_runs r WHERE {where} "
                f"ORDER BY {_ORDER} LIMIT ?",
                (*params, limit + 1),
            )
        ]
        more = len(rows) > limit
        rows = rows[:limit]
        if not rows:
            return result
        upper = after.upper if after is not None else _key(rows[0])
        result["resume_position"] = WindowPosition(raw_id, upper, upper)
        if more:
            result["next_position"] = WindowPosition(raw_id, upper, _key(rows[-1]))
        result["has_more"] = more
        rows.reverse()
        by_run = {row["run_id"]: row for row in rows}
        for row in rows:
            row["diagnostics_available"] = any(
                row[field] is not None
                for field in ("node_id", "node_type", "exception_type")
            )
            for field in ("node_id", "node_type", "exception_type"):
                del row[field]
            row["diagnostics_omitted"] = bool(row["diagnostics_omitted"])
            row["context_capture"] = _context_capture(conn, row["run_id"])
            row["output_summary"] = _output_summary(conn, row["run_id"])
            row["matching_anchors"] = []
        placeholders = ",".join("?" for _ in rows)
        anchors = conn.execute(
            "SELECT rc.run_id,rc.anchor_id,rc.context_id,rc.evidence_json,"
            "c.positive_snapshot_id,c.negative_snapshot_id,"
            "substr(CAST(t.text_value AS BLOB),1,512) AS prefix,"
            "length(CAST(t.text_value AS BLOB)) AS byte_length "
            "FROM execution_run_contexts rc JOIN execution_contexts c "
            "ON c.context_id=rc.context_id JOIN execution_prompt_snapshots p "
            "ON p.snapshot_id=c.positive_snapshot_id "
            "LEFT JOIN execution_prompt_snapshots n "
            "ON n.snapshot_id=c.negative_snapshot_id "
            "LEFT JOIN execution_context_texts t ON t.text_id=n.raw_text_id "
            f"WHERE rc.run_id IN ({placeholders}) AND p.raw_text_id=? "
            "ORDER BY rc.anchor_id COLLATE BINARY,rc.run_id COLLATE BINARY",
            (*by_run, raw_id),
        )
        summaries = {}
        evidence_by_context = {}
        for anchor in anchors:
            reference = {
                key: anchor[key]
                for key in (
                    "anchor_id",
                    "context_id",
                    "positive_snapshot_id",
                    "negative_snapshot_id",
                )
            }
            by_run[anchor["run_id"]]["matching_anchors"].append(reference)
            context_id = anchor["context_id"]
            if context_id not in summaries:
                prefix = b"" if anchor["byte_length"] == 0 else anchor["prefix"]
                preview = (
                    None
                    if prefix is None
                    else codecs.getincrementaldecoder("utf-8")().decode(
                        prefix, final=False
                    )[:100]
                )
                evidence_by_context[context_id] = []
                summaries[context_id] = {
                    key: anchor[key]
                    for key in (
                        "context_id",
                        "positive_snapshot_id",
                        "negative_snapshot_id",
                    )
                }
                summaries[context_id]["negative_raw"] = {
                    "state": (
                        "unknown"
                        if preview is None
                        else "explicit_empty" if not preview else "known"
                    ),
                    "preview": preview,
                    "truncated": (
                        False
                        if preview is None
                        else len(preview.encode("utf-8")) < anchor["byte_length"]
                    ),
                    "evidence": evidence_by_context[context_id],
                }
            negative = json.loads(anchor["evidence_json"])["negative"]
            evidence = (
                negative["raw"]
                if negative is not None
                else dict(provenance="unknown", source="unknown", reason="missing")
            )
            evidence_by_context[context_id].append(
                {
                    "run_id": anchor["run_id"],
                    "anchor_id": anchor["anchor_id"],
                    **evidence,
                }
            )
        # Context order follows its first chronological matching run, then anchor.
        ordered = {}
        for row in rows:
            for anchor in row["matching_anchors"]:
                ordered.setdefault(
                    anchor["context_id"], summaries[anchor["context_id"]]
                )
        result["items"] = rows
        result["context_summaries"] = list(ordered.values())
        return result
    def get_newest_pair_run(self, positive_id: int, pair_id: int) -> dict | None:
        """The newest matching run whose current saved-pair hint names `pair_id`.
        None when the positive is not an eligible saved prompt or the pair is
        not one of its pairs; ``{"run": None}`` when no recorded run names it.
        Only a hint captured in this epoch and still resolving to the same pair
        counts, as in ``get_history_run``.
        """
        conn = self._history_connection()
        saved = conn.execute(
            "SELECT CASE WHEN typeof(p.text)='text' AND "
            "length(CAST(p.text AS BLOB))<=65536 THEN p.text ELSE NULL END AS text "
            f"FROM prompts p WHERE p.id=? AND ({_ELIGIBLE})",
            (positive_id,),
        ).fetchone()
        if saved is None or (
            conn.execute(
                "SELECT 1 FROM prompt_pairs WHERE id=? AND positive_prompt_id=?",
                (pair_id, positive_id),
            ).fetchone()
            is None
        ):
            return None
        if saved["text"] is None:
            return {"run": None}
        epoch = self.get_execution_epoch()
        candidates = conn.execute(
            "SELECT r.run_id,rc.anchor_id,rc.context_id,"
            "c.positive_snapshot_id,c.negative_snapshot_id "
            "FROM execution_runs r JOIN execution_run_contexts rc "
            "ON rc.run_id=r.run_id JOIN execution_contexts c "
            "ON c.context_id=rc.context_id JOIN execution_prompt_snapshots s "
            "ON s.snapshot_id=c.positive_snapshot_id "
            "WHERE s.raw_text_id=? AND rc.pair_id=? AND rc.reference_epoch=? "
            f"ORDER BY {_ORDER},rc.anchor_id COLLATE BINARY LIMIT ?",
            (
                content_digest("text", (saved["text"],)),
                pair_id,
                epoch,
                _PAIR_RUN_CANDIDATES,
            ),
        ).fetchall()
        for row in candidates:
            positive = _values(conn, row["positive_snapshot_id"])
            negative = _values(conn, row["negative_snapshot_id"])
            if positive is None or positive["raw"] != saved["text"]:
                continue
            hints = _hints(conn, positive["raw"], negative["raw"] if negative else None)
            if hints[2] == pair_id:
                return {
                    "run": {
                        key: row[key] for key in ("run_id", "anchor_id", "context_id")
                    }
                }
        return {"run": None}
    def get_history_run(self, run_id: str) -> dict | None:
        conn = self._history_connection()
        run = self.get_execution_run(run_id)
        if run is None:
            return None
        capture = self.get_execution_contexts(run_id)
        snapshots = {
            row["anchor_id"]: row
            for row in conn.execute(
                "SELECT rc.anchor_id,c.positive_snapshot_id,c.negative_snapshot_id,"
                "p.raw_text_id AS positive_raw_text_id "
                "FROM execution_run_contexts rc JOIN execution_contexts c "
                "ON c.context_id=rc.context_id LEFT JOIN execution_prompt_snapshots p "
                "ON p.snapshot_id=c.positive_snapshot_id WHERE rc.run_id=?",
                (run_id,),
            )
        }
        contexts = capture.pop("contexts")
        for context in contexts:
            row = snapshots[context["anchor_id"]]
            context.update(
                {
                    key: row[key]
                    for key in (
                        "positive_snapshot_id",
                        "negative_snapshot_id",
                        "positive_raw_text_id",
                    )
                }
            )
            pair = context["pair_id"]
            context["navigation"] = {
                "positive_available": context["positive_prompt_id"] is not None
                and conn.execute(
                    f"SELECT 1 FROM prompts p WHERE p.id=? AND ({_ELIGIBLE})",
                    (context["positive_prompt_id"],),
                ).fetchone()
                is not None,
                "negative_available": context["negative_prompt_id"] is not None,
                "pair_available": pair is not None
                and conn.execute(
                    "SELECT 1 FROM prompt_pairs WHERE id=? AND archived=0", (pair,)
                ).fetchone()
                is not None,
            }
        return dict(
            run=run,
            context_capture=capture,
            contexts=contexts,
            output_summary=_output_summary(conn, run_id),
        )
    def list_history_outputs(
        self,
        run_id: str,
        *,
        limit: int = 40,
        after: OutputPosition | None = None,
    ) -> dict | None:
        conn = self._history_connection()
        _limit(limit)
        _validate_output_position(after)
        # Validate header/orphans even when the requested parent does not exist.
        capture, count = _output_header(conn, run_id)
        if self.get_execution_run(run_id) is None:
            return None
        fingerprint = _fingerprint(capture, count)
        if after is not None and after.capture_fingerprint != fingerprint:
            raise HistoryCaptureChanged("History output capture changed")
        where = "run_id=?"
        params = [run_id]
        if after is not None:
            where += " AND (output_node_id COLLATE BINARY,image_ordinal)>(?,?)"
            params.extend(after.after)
        rows = list(
            conn.execute(
                f"SELECT {_REFERENCE_COLUMNS} FROM execution_run_outputs WHERE {where} "
                "ORDER BY output_node_id COLLATE BINARY,image_ordinal LIMIT ?",
                (*params, limit + 1),
            )
        )
        more = len(rows) > limit
        rows = rows[:limit]
        position = (
            OutputPosition(
                fingerprint, (rows[-1]["output_node_id"], rows[-1]["image_ordinal"])
            )
            if more
            else None
        )
        return dict(
            capture=capture,
            retained_occurrence_count=count,
            list_status=_list_status(capture),
            capture_fingerprint=fingerprint,
            items=[_receipt(run_id, row) for row in rows],
            has_more=more,
            next_position=position,
        )
    def get_history_occurrence(
        self, run_id: str, node: str, ordinal: int
    ) -> dict | None:
        conn = self._history_connection()
        capture, _ = _output_header(conn, run_id)
        row = conn.execute(
            f"SELECT {_REFERENCE_COLUMNS} FROM execution_run_outputs "
            "WHERE run_id=? AND output_node_id=? COLLATE BINARY AND image_ordinal=?",
            (run_id, node, ordinal),
        ).fetchone()
        return (
            dict(capture=capture, receipt=_receipt(run_id, row))
            if row is not None
            else None
        )
@@ -0,0 +1,152 @@
 """Persisted image favorites, keyed by gallery root + relative path.
 The root key is the root's real, case-normalized path and the relative key is
 case-normalized the same way (see ``py/api/images.py``); both stay
 server-side. Folder names alone are not unique across roots, so a favorite
 follows the directory itself when the configured roots are reordered.
 Optional in old backups, required once a database is current (see
 ``library_schema.py``).
 """
 import re
 import sqlite3
 from typing import Iterable
 TABLE = "image_favorites"
 MAX_KEY_LENGTH = 4096
 _LOOKUP_CHUNK = 500
 _DDL = (
    f"CREATE TABLE {TABLE} (root_key TEXT NOT NULL, relative_path TEXT NOT NULL, "
    "created_at TEXT NOT NULL, PRIMARY KEY(root_key, relative_path))"
 )
 _AUTOINDEX = f"sqlite_autoindex_{TABLE}_1"
 _COLUMNS = (
    ("root_key", "TEXT", 1, 1),
    ("relative_path", "TEXT", 1, 2),
    ("created_at", "TEXT", 1, 0),
 )
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql).lower()
 def image_favorites_schema_error(
    conn: sqlite3.Connection, *, allow_missing: bool = True
 ) -> str | None:
    """Reject unsupported favorites schemas and invalid persisted values.
    Reads only. A missing table is acceptable for backups that predate it
    unless ``allow_missing`` is false.
    """
    objects = conn.execute(
        "SELECT type,name,sql FROM sqlite_master WHERE name=? "
        "OR (type IN ('index','trigger') AND tbl_name=?)",
        (TABLE, TABLE),
    ).fetchall()
    if not objects:
        return None if allow_missing else "Missing image favorites schema"
    tables = [sql for kind, _name, sql in objects if kind == "table"]
    others = {(kind, name, sql) for kind, name, sql in objects if kind != "table"}
    if len(tables) != 1 or others != {("index", _AUTOINDEX, None)}:
        return "Image favorites name is reserved by an unsupported schema object"
    if _compact(tables[0] or "") != _compact(_DDL):
        return "Image favorites has an invalid table definition"
    columns = tuple(
        (row[1], row[2].upper(), row[3], row[5])
        for row in conn.execute(f"PRAGMA table_info({TABLE})")
    )
    if columns != _COLUMNS:
        return "Image favorites has invalid columns"
    # Bytes bound characters from above (UTF-8 is at most 4 bytes each).
    limit = MAX_KEY_LENGTH * 4
    if conn.execute(
        f"SELECT 1 FROM {TABLE} WHERE typeof(root_key) != 'text' "
        "OR typeof(relative_path) != 'text' OR typeof(created_at) != 'text' "
        f"OR length(CAST(root_key AS BLOB)) > {limit} "
        f"OR length(CAST(relative_path AS BLOB)) > {limit} "
        "OR length(CAST(created_at AS BLOB)) > 64 LIMIT 1"
    ).fetchone():
        return "Image favorites contains invalid values"
    for root, path in conn.execute(
        f"SELECT CAST(root_key AS BLOB), CAST(relative_path AS BLOB) FROM {TABLE} "
        f"WHERE length(CAST(root_key AS BLOB)) > {MAX_KEY_LENGTH} "
        f"OR length(CAST(relative_path AS BLOB)) > {MAX_KEY_LENGTH}"
    ):
        for raw in (root, path):
            if len(bytes(raw).decode("utf-8", "replace")) > MAX_KEY_LENGTH:
                return "Image favorites contains invalid values"
    return None
 def migrate_image_favorites(conn: sqlite3.Connection) -> None:
    """Install the table atomically; validate before and after creating it."""
    conn.execute("SAVEPOINT image_favorites_schema")
    try:
        error = image_favorites_schema_error(conn)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        error = image_favorites_schema_error(conn, allow_missing=False)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute("RELEASE image_favorites_schema")
    except BaseException:
        conn.execute("ROLLBACK TO image_favorites_schema")
        conn.execute("RELEASE image_favorites_schema")
        raise
 class ImageFavoriteOperations:
    """PromptDatabase mixin: set, clear and look up image favorites."""
    def set_image_favorite(self, root_key: str, relative_path: str) -> bool:
        """Mark one file as a favorite; repeating it is a no-op. Returns True."""
        from .models import utc_now_iso  # models imports this module's schema
        with self.model.get_connection() as conn:
            conn.execute(
                "INSERT OR IGNORE INTO image_favorites VALUES (?, ?, ?)",
                (root_key, relative_path, utc_now_iso()),
            )
        return True
    def clear_image_favorite(self, root_key: str, relative_path: str) -> bool:
        """Remove one favorite; True when a row was removed."""
        with self.model.get_connection() as conn:
            cursor = conn.execute(
                "DELETE FROM image_favorites WHERE root_key=? AND relative_path=?",
                (root_key, relative_path),
            )
        return cursor.rowcount > 0
    def get_image_favorites(
        self, root_key: str, relative_paths: Iterable[str]
    ) -> set[str]:
        """The subset of *relative_paths* favorited under *root_key*."""
        wanted = list(dict.fromkeys(relative_paths))
        found: set[str] = set()
        conn = self.model.get_connection()
        for start in range(0, len(wanted), _LOOKUP_CHUNK):
            chunk = wanted[start : start + _LOOKUP_CHUNK]
            marks = ",".join("?" * len(chunk))
            rows = conn.execute(  # only "?" placeholders are interpolated
                "SELECT relative_path FROM image_favorites WHERE root_key=? "
                f"AND relative_path IN ({marks})",  # nosec B608
                (root_key, *chunk),
            )
            found.update(path for (path,) in rows)
        return found
    def get_root_favorites(self, root_key: str) -> set[str]:
        """Every relative key favorited under *root_key* (a primary-key range).
        The Favorites listing (``favorites=1``) intersects these with the
        root's scan, so its size follows the root's favorites, not its files.
        """
        conn = self.model.get_connection()
        rows = conn.execute(
            "SELECT relative_path FROM image_favorites WHERE root_key=?",
            (root_key,),
        )
        return {path for (path,) in rows}
@@ -0,0 +1,684 @@
 """Read-only positive-parent views over existing prompt and pair tables."""
 import datetime
 import sqlite3
 from collections.abc import Iterator
 from contextlib import contextmanager
 from .library_tokens import library_edit_token
 _ELIGIBLE = """(EXISTS(SELECT 1 FROM prompt_pairs ep WHERE ep.positive_prompt_id=p.id)
 OR EXISTS(SELECT 1 FROM generated_images eg WHERE eg.prompt_id=p.id)
 OR NOT EXISTS(SELECT 1 FROM prompt_pairs en WHERE en.negative_prompt_id=p.id)
 OR EXISTS(SELECT 1 FROM prompt_library_state es WHERE es.prompt_id=p.id))"""
 _FROM = "prompts p LEFT JOIN prompt_library_state ls ON ls.prompt_id=p.id"
 # The characters JavaScript's String.prototype.trim() removes, so a title the
 # display treats as blank also falls back to the prompt text for name sorting.
 _TRIM_CODES = (9, 10, 11, 12, 13, 32, 160, 5760, *range(8192, 8203))
 _TRIM_CODES += (8232, 8233, 8239, 8287, 12288, 65279)
 _BLANK = f"char({','.join(map(str, _TRIM_CODES))})"
 _JS_TRIM = "".join(map(chr, _TRIM_CODES))
 # The Library's display title: a stored title, else the prompt's first line.
 DISPLAY_TITLE_LIMIT = 80
 def _utf16_units(text: str) -> int:
    """JavaScript ``length``: characters beyond the BMP count as two units."""
    return len(text) + sum(1 for ch in text if ord(ch) > 0xFFFF)
 def _utf16_prefix(text: str, units: int) -> str:
    """The longest prefix within ``units`` UTF-16 units, never splitting a pair."""
    used = 0
    for index, ch in enumerate(text):
        used += 2 if ord(ch) > 0xFFFF else 1
        if used > units:
            return text[:index]
    return text
 def titled(stored_titles: tuple, text: object) -> tuple[str | None, str | None]:
    """Return ``(title, source)``: a stored title, else the text's first line.
    ``source`` is ``"stored"`` for the first non-blank stored title (blank as
    in JavaScript's trim, like the Library view), ``"text"`` for the first line,
    and None when both are empty. The first line is trimmed and, past
    DISPLAY_TITLE_LIMIT UTF-16 units, cut with an ellipsis like the Library's
    ``libraryTitle()`` (never inside a surrogate pair).
    """
    for title in stored_titles:
        if isinstance(title, str) and title.strip(_JS_TRIM):
            return title, "stored"
    if not isinstance(text, str):
        return None, None
    first = text.split("\n")[0].strip(_JS_TRIM)
    if _utf16_units(first) > DISPLAY_TITLE_LIMIT:
        first = _utf16_prefix(first, DISPLAY_TITLE_LIMIT) + "\u2026"
    return (first, "text") if first else (None, None)
 def display_title(stored_titles: tuple, text: object) -> str | None:
    """The Library display title of ``titled``, without its source."""
    return titled(stored_titles, text)[0]
 def utc_iso(value: object) -> str | None:
    """A stored timestamp as UTC ISO text, or None when absent or unreadable.
    Usage writes are already UTC ISO; a naive value (SQLite's CURRENT_TIMESTAMP
    form) is read as UTC, and an offset is converted to UTC. One bad row must
    not fail a whole Library read, so a value past the datetime range after
    conversion (OverflowError) is unreadable too. Python 3.11+ fromisoformat
    accepts more ISO forms than 3.10 (e.g. a 2-digit fraction); every form the
    app writes (utc_now_iso, the 3.2.4 backfill, CURRENT_TIMESTAMP, Z) parses
    the same on both, so only hand-edited values can differ by version.
    """
    if not isinstance(value, str):
        return None
    text = value.strip()
    if text.endswith(("Z", "z")):
        text = text[:-1] + "+00:00"
    try:
        stamp = datetime.datetime.fromisoformat(text)
        if stamp.tzinfo is None:
            stamp = stamp.replace(tzinfo=datetime.timezone.utc)
        return stamp.astimezone(datetime.timezone.utc).isoformat()
    except (ValueError, OverflowError):
        return None
 _NAME = (
    f"LOWER(CASE WHEN TRIM(COALESCE(ls.title,''),{_BLANK})!='' "
    "THEN ls.title ELSE p.text END)"
 )
 _ORDERS = {
    "newest": ("COALESCE(julianday(p.created_at),0)", "DESC"),
    "oldest": ("COALESCE(julianday(p.created_at),0)", "ASC"),
    "name": (_NAME, "ASC"),
    "rating": ("COALESCE(p.rating,0)", "DESC"),
 }
 # Pair archive scopes for variant and image reads, as conditions on alias pp.
 ARCHIVE_SCOPES = {
    "active": " AND pp.archived=0",
    "archived": " AND pp.archived=1",
    "all": "",
 }
 def _escape(value: str) -> str:
    return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
 def _limit(value: int) -> None:
    if type(value) is not int or not 1 <= value <= 200:
        raise ValueError("limit must be between 1 and 200")
 def _archive(value: str) -> str:
    """Return the pair-archive condition for one scope (alias ``pp``)."""
    if not isinstance(value, str) or value not in ARCHIVE_SCOPES:
        raise ValueError("archive must be one of active, archived or all")
    return ARCHIVE_SCOPES[value]
 def _filter(text: str, category: str | None, tags: list[str]) -> tuple[str, list]:
    if not isinstance(text, str) or len(text) > 500:
        raise ValueError("Search text must be a string of at most 500 characters")
    if category is not None and (not isinstance(category, str) or len(category) > 200):
        raise ValueError("Invalid library category")
    if (
        not isinstance(tags, list)
        or len(tags) > 30
        or any(not isinstance(tag, str) or not tag or len(tag) > 200 for tag in tags)
    ):
        raise ValueError("Invalid library tags")
    parent, pair, parent_params, pair_params = [], ["pp.archived=0"], [], []
    if text:
        parent.append("(p.text LIKE ? ESCAPE '\\' OR ls.title LIKE ? ESCAPE '\\')")
        parent_params.extend([f"%{_escape(text)}%"] * 2)
        pair.append(
            "(p.text LIKE ? ESCAPE '\\' OR ls.title LIKE ? ESCAPE '\\' "
            "OR n.text LIKE ? ESCAPE '\\' OR pp.title LIKE ? ESCAPE '\\')"
        )
        pair_params.extend([f"%{_escape(text)}%"] * 4)
    if category is not None:
        parent.append("p.category=?")
        pair.append("pp.category=?")
        parent_params.append(category)
        pair_params.append(category)
    for tag in tags:
        parent.append(
            "EXISTS(SELECT 1 FROM prompt_tags pt JOIN tags t ON t.id=pt.tag_id "
            "WHERE pt.prompt_id=p.id AND t.name=?)"
        )
        pair.append(
            "EXISTS(SELECT 1 FROM prompt_pair_tags pt JOIN tags t ON t.id=pt.tag_id "
            "WHERE pt.pair_id=pp.id AND t.name=?)"
        )
        parent_params.append(tag)
        pair_params.append(tag)
    condition = (
        "("
        + (" AND ".join(parent) or "1=1")
        + (
            ") OR EXISTS(SELECT 1 FROM prompt_pairs pp "
            "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
            "WHERE pp.positive_prompt_id=p.id AND "
        )
        + " AND ".join(pair)
        + ")"
    )
    return f"({_ELIGIBLE}) AND ({condition})", [*parent_params, *pair_params]
 class LibraryOperations:
    """Group reusable negative variants without changing legacy prompt queries.
    Each public read runs inside ``library_snapshot``; the ``_``-prefixed
    helpers take that snapshot's connection and epoch so routes can share it.
    """
    @contextmanager
    def library_snapshot(
        self, *, include_ceiling: bool = False
    ) -> Iterator[tuple[sqlite3.Connection, str, int | None]]:
        """Hold one read transaction, reading the epoch (and ceiling) first."""
        conn = self.model.get_connection()
        if conn.in_transaction:
            raise sqlite3.ProgrammingError("Library reads require their own snapshot")
        conn.execute("BEGIN")
        try:
            row = conn.execute(
                "SELECT storage_epoch,"
                + (
                    "(SELECT COALESCE(MAX(id),0) FROM prompts)"
                    if include_ceiling
                    else "NULL"
                )
                + " FROM execution_history_metadata WHERE singleton_id=1"
            ).fetchone()
            if row is None:
                raise sqlite3.DatabaseError("Missing execution history metadata")
            yield conn, row[0], row[1]
        finally:
            if conn.in_transaction:
                conn.rollback()
    def _library_parents(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        where: str,
        params: list,
        suffix: str = "",
        sort_expression: str | None = None,
    ) -> list[dict]:
        rows = conn.execute(
            "SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes,p.created_at,"
            "p.run_count,p.last_used_at,"
            + (f"{sort_expression} AS _sort," if sort_expression else "")
            + "(SELECT COUNT(*) FROM prompt_pairs v WHERE v.positive_prompt_id=p.id "
            "AND v.archived=0) AS variant_count,"
            "(SELECT COUNT(*) FROM generated_images g "
            "WHERE g.prompt_id=p.id) AS image_count "
            "FROM " + _FROM + " WHERE " + where + suffix,
            params,
        ).fetchall()
        result = [{**dict(row), "tags": []} for row in rows]
        if result:
            by_id = {row["id"]: row for row in result}
            marks = ",".join("?" for _ in result)
            for row in conn.execute(
                "SELECT pt.prompt_id,t.name FROM prompt_tags pt "
                "JOIN tags t ON t.id=pt.tag_id "
                f"WHERE pt.prompt_id IN ({marks}) ORDER BY t.name",
                list(by_id),
            ):
                by_id[row["prompt_id"]]["tags"].append(row["name"])
        for row in result:
            # Usage is shown, never compared: the token hashes owner fields only.
            row["run_count"] = row["run_count"] if type(row["run_count"]) is int else 0
            row["last_used_at"] = utc_iso(row["last_used_at"])
            row["edit_token"] = library_edit_token(epoch, row)
        return result
    def _list_library(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        ceiling: int,
        *,
        text: str = "",
        category: str | None = None,
        tags: list[str] | None = None,
        sort: str = "newest",
        limit: int = 40,
        after: dict | None = None,
    ) -> dict:
        _limit(limit)
        if sort not in _ORDERS:
            raise ValueError("Unknown library sort order")
        where, params = _filter(text, category, [] if tags is None else tags)
        if after is not None:
            if (
                not isinstance(after, dict)
                or set(after) != {"ceiling", "value", "id"}
                or type(after["ceiling"]) is not int
                or type(after["id"]) is not int
            ):
                raise ValueError("Invalid library cursor")
            ceiling = after["ceiling"]
        where += " AND p.id<=?"
        params.append(ceiling)
        # One SELECT counts the filtered parents and their active variants.
        total, variant_total = conn.execute(
            "SELECT COUNT(*),COALESCE(SUM((SELECT COUNT(*) FROM prompt_pairs v "
            "WHERE v.positive_prompt_id=p.id AND v.archived=0)),0) FROM "
            + _FROM
            + " WHERE "
            + where,
            params,
        ).fetchone()
        expression, direction = _ORDERS[sort]
        if after is not None:
            operator = ">" if direction == "ASC" else "<"
            where += (
                f" AND ({expression}{operator}? OR "
                f"({expression}=? AND p.id{operator}?))"
            )
            params.extend([after["value"], after["value"], after["id"]])
        rows = self._library_parents(
            conn,
            epoch,
            where,
            [*params, limit + 1],
            f" ORDER BY {expression} {direction},p.id {direction} LIMIT ?",
            sort_expression=expression,
        )
        has_more = len(rows) > limit
        rows = rows[:limit]
        next_key = None
        if has_more:
            next_key = {
                "ceiling": ceiling,
                "value": rows[-1]["_sort"],
                "id": rows[-1]["id"],
            }
        for row in rows:
            row.pop("_sort")
        return {
            "library_epoch": epoch,
            "items": rows,
            "total": total,
            "variant_total": variant_total,
            "has_more": has_more,
            "next_key": next_key,
        }
    def list_library(
        self,
        *,
        text: str = "",
        category: str | None = None,
        tags: list[str] | None = None,
        sort: str = "newest",
        limit: int = 40,
        after: dict | None = None,
    ) -> dict:
        with self.library_snapshot(include_ceiling=True) as (conn, epoch, ceiling):
            return self._list_library(
                conn,
                epoch,
                ceiling,
                text=text,
                category=category,
                tags=tags,
                sort=sort,
                limit=limit,
                after=after,
            )
    def _is_library_positive(self, conn: sqlite3.Connection, positive_id: int) -> bool:
        return (
            conn.execute(
                "SELECT 1 FROM prompts p WHERE p.id=? AND " + _ELIGIBLE, (positive_id,)
            ).fetchone()
            is not None
        )
    def _get_library_positive(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        positive_id: int,
        *,
        variant_limit: int = 40,
        variant_after: int | None = None,
        archive: str = "active",
    ) -> dict | None:
        _limit(variant_limit)
        scope = _archive(archive)
        if variant_after is not None and (
            type(variant_after) is not int or variant_after <= 0
        ):
            raise ValueError("Invalid variant cursor")
        rows = self._library_parents(
            conn, epoch, _ELIGIBLE + " AND p.id=?", [positive_id]
        )
        if not rows:
            return None
        positive = rows[0]
        positive["archived_variant_count"] = conn.execute(
            "SELECT COUNT(*) FROM prompt_pairs "
            "WHERE positive_prompt_id=? AND archived=1",
            (positive_id,),
        ).fetchone()[0]
        variant_total = {
            "active": positive["variant_count"],
            "archived": positive["archived_variant_count"],
            "all": positive["variant_count"] + positive["archived_variant_count"],
        }[archive]
        where = "pp.positive_prompt_id=?" + scope
        params = [positive_id]
        if variant_after is not None:
            where += " AND pp.id<?"
            params.append(variant_after)
        variants = [
            dict(row)
            for row in conn.execute(
                "SELECT pp.id,COALESCE(n.text,'') AS negative_text,"
                "pp.category,pp.title,pp.archived,"
                "pp.negative_prompt_id IS NULL AS explicit_empty,pp.run_count,"
                "(SELECT COUNT(DISTINCT image_id) FROM image_prompt_pairs ip "
                "WHERE ip.pair_id=pp.id) AS image_count "
                "FROM prompt_pairs pp "
                "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
                "WHERE " + where + " ORDER BY pp.id DESC LIMIT ?",
                [*params, variant_limit + 1],
            )
        ]
        variants_has_more = len(variants) > variant_limit
        variants = variants[:variant_limit]
        by_id = {row["id"]: row for row in variants}
        for row in variants:
            row["tags"] = []
            row["archived"] = bool(row["archived"])
            row["explicit_empty"] = bool(row["explicit_empty"])
            # The history lane's "N runs" (#249): queue-hook pair uses, 0 when unset.
            row["run_count"] = row["run_count"] if type(row["run_count"]) is int else 0
        if by_id:
            marks = ",".join("?" for _ in by_id)
            for row in conn.execute(
                "SELECT pt.pair_id,t.name FROM prompt_pair_tags pt "
                "JOIN tags t ON t.id=pt.tag_id "
                f"WHERE pt.pair_id IN ({marks}) ORDER BY t.name",
                list(by_id),
            ):
                by_id[row["pair_id"]]["tags"].append(row["name"])
        unassigned = conn.execute(
            "SELECT COUNT(*) FROM generated_images g WHERE g.prompt_id=? "
            "AND NOT EXISTS(SELECT 1 FROM image_prompt_pairs ip "
            "WHERE ip.image_id=g.id)",
            (positive_id,),
        ).fetchone()[0]
        return {
            "library_epoch": epoch,
            "positive": positive,
            "variants": variants,
            "unassigned_image_count": unassigned,
            "variant_total": variant_total,
            "variants_has_more": variants_has_more,
            "variants_next_key": variants[-1]["id"] if variants_has_more else None,
        }
    def get_library_positive(
        self,
        positive_id: int,
        *,
        variant_limit: int = 40,
        variant_after: int | None = None,
        archive: str = "active",
    ) -> dict | None:
        with self.library_snapshot() as (conn, epoch, _ceiling):
            return self._get_library_positive(
                conn,
                epoch,
                positive_id,
                variant_limit=variant_limit,
                variant_after=variant_after,
                archive=archive,
            )
    def _get_library_options(self, conn: sqlite3.Connection, epoch: str) -> dict:
        rows = conn.execute(
            "SELECT category FROM (SELECT p.category AS category "
            "FROM prompts p WHERE "
            + _ELIGIBLE
            + (" UNION SELECT category FROM prompt_pairs " "WHERE archived=0) ")
            + "WHERE category IS NOT NULL AND TRIM(category)!='' "
            "ORDER BY category COLLATE NOCASE,category"
        ).fetchall()
        return {
            "library_epoch": epoch,
            "categories": [row["category"] for row in rows],
        }
    def get_library_options(self) -> dict:
        with self.library_snapshot() as (conn, epoch, _ceiling):
            return self._get_library_options(conn, epoch)
    def get_library_previews(self, positive_ids: list[int]) -> dict[int, list[dict]]:
        """Read at most two links per parent in one bounded page query."""
        if (
            not isinstance(positive_ids, list)
            or len(positive_ids) > 200
            or any(type(value) is not int or value <= 0 for value in positive_ids)
        ):
            raise ValueError("Invalid preview parent IDs")
        if not positive_ids:
            return {}
        result = {value: [] for value in positive_ids}
        marks = ",".join("?" for _ in positive_ids)
        rows = self.model.get_connection().execute(
            "WITH ranked AS (SELECT id,prompt_id,image_path,ROW_NUMBER() OVER "
            "(PARTITION BY prompt_id ORDER BY id DESC) AS position "
            f"FROM generated_images WHERE prompt_id IN ({marks})) "
            "SELECT id,prompt_id,image_path FROM ranked WHERE position<=2 "
            "ORDER BY prompt_id,id DESC",
            positive_ids,
        )
        for row in rows:
            result[row["prompt_id"]].append(dict(row))
        return result
    def _list_library_images(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        positive_id: int,
        *,
        pair_id: int | None = None,
        limit: int = 40,
        after: int | None = None,
        archive: str = "active",
    ) -> dict:
        _limit(limit)
        scope = _archive(archive)
        if after is not None and (type(after) is not int or after <= 0):
            raise ValueError("Invalid image cursor")
        if (
            pair_id is not None
            and not conn.execute(
                "SELECT 1 FROM prompt_pairs pp WHERE pp.id=? "
                "AND pp.positive_prompt_id=?" + scope,
                (pair_id, positive_id),
            ).fetchone()
        ):
            if conn.execute(
                "SELECT 1 FROM prompt_pairs WHERE id=? AND positive_prompt_id=?",
                (pair_id, positive_id),
            ).fetchone():
                raise ValueError("Variant is not in the requested archive scope")
            raise ValueError("Variant does not belong to this positive prompt")
        where, params = "g.prompt_id=?", [positive_id]
        if pair_id is not None:
            where += (
                " AND EXISTS(SELECT 1 FROM image_prompt_pairs ip "
                "WHERE ip.image_id=g.id AND ip.pair_id=?)"
            )
            params.append(pair_id)
        total = conn.execute(
            "SELECT COUNT(*) FROM generated_images g WHERE " + where, params
        ).fetchone()[0]
        if after is not None:
            where += " AND g.id<?"
            params.append(after)
        rows = conn.execute(
            "SELECT g.id,g.image_path FROM generated_images g WHERE "
            + where
            + " ORDER BY g.id DESC LIMIT ?",
            [*params, limit + 1],
        ).fetchall()
        return {
            "library_epoch": epoch,
            "images": [dict(row) for row in rows[:limit]],
            "total": total,
            "has_more": len(rows) > limit,
            "next_key": rows[limit - 1]["id"] if len(rows) > limit else None,
        }
    def list_library_images(
        self,
        positive_id: int,
        *,
        pair_id: int | None = None,
        limit: int = 40,
        after: int | None = None,
        archive: str = "active",
    ) -> dict:
        with self.library_snapshot() as (conn, epoch, _ceiling):
            return self._list_library_images(
                conn,
                epoch,
                positive_id,
                pair_id=pair_id,
                limit=limit,
                after=after,
                archive=archive,
            )
    def export_library(self) -> dict:
        """Read every portable owner and pair from one snapshot, writing nothing.
        Local IDs order the rows internally and never appear in the document.
        """
        with self.library_snapshot() as (conn, _epoch, _ceiling):
            owners = {}
            order = []
            for row in conn.execute(
                "SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes FROM "
                + _FROM
                + " WHERE "
                + _ELIGIBLE
                + " ORDER BY p.id"
            ):
                owners[row["id"]] = {
                    "positive_text": row["text"],
                    "title": row["title"],
                    "category": row["category"],
                    "tags": [],
                    "rating": row["rating"],
                    "notes": row["notes"],
                    "variants": [],
                }
                order.append(row["id"])
            for row in conn.execute(
                "SELECT pt.prompt_id,t.name FROM prompt_tags pt "
                "JOIN tags t ON t.id=pt.tag_id ORDER BY pt.prompt_id,t.name"
            ):
                if row["prompt_id"] in owners:
                    owners[row["prompt_id"]]["tags"].append(row["name"])
            variants = {}
            for row in conn.execute(
                "SELECT pp.id,pp.positive_prompt_id,"
                "COALESCE(n.text,'') AS negative_text,"
                "pp.title,pp.category,pp.rating,pp.notes,pp.archived "
                "FROM prompt_pairs pp "
                "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
                "ORDER BY pp.id"
            ):
                variant = {
                    "negative_text": row["negative_text"],
                    "title": row["title"],
                    "category": row["category"],
                    "tags": [],
                    "rating": row["rating"],
                    "notes": row["notes"],
                    "archived": bool(row["archived"]),
                }
                variants[row["id"]] = variant
                owners[row["positive_prompt_id"]]["variants"].append(variant)
            for row in conn.execute(
                "SELECT pt.pair_id,t.name FROM prompt_pair_tags pt "
                "JOIN tags t ON t.id=pt.tag_id ORDER BY pt.pair_id,t.name"
            ):
                variants[row["pair_id"]]["tags"].append(row["name"])
            return {
                "schema_version": 1,
                "kind": "prompt-manager-library",
                "positives": [owners[key] for key in order],
            }
    def library_counts(self) -> dict:
        """System › Overview counts from one snapshot, writing nothing.
        ``positive_prompts`` and ``negative_variants`` are the unfiltered
        Library's ``total`` and ``variant_total``: eligible parents and active
        pairs (every pair's positive is eligible). ``images`` counts indexed
        ``generated_images`` rows, not files on disk. An image has no rating
        of its own (it shows its linked prompt's), so ``rated_images`` counts
        the indexed images whose linked prompt is rated. ``review_pending`` is
        the Needs review queue's pending total.
        """
        with self.library_snapshot() as (conn, _epoch, _ceiling):
            row = conn.execute(
                "SELECT (SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE + "),"
                "(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
                "(SELECT COUNT(*) FROM generated_images),"
                "(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
                "ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL),"
                "(SELECT COUNT(*) FROM media_review_items WHERE state='pending')"
            ).fetchone()
        return {
            "positive_prompts": row[0],
            "negative_variants": row[1],
            "images": row[2],
            "rated_images": row[3],
            "review_pending": row[4],
        }
    def usage_by_category(self) -> list[dict]:
        """Runs and average rating of the Library's parents, per category.
        Only eligible parents count, so a negative-only prompt's runs (which
        repeat its pairs' runs) are never added twice. A blank category groups
        with no category as ``None``. Unrated parents are left out of the
        average, which is ``None`` when no parent is rated. Most runs first.
        """
        with self.library_snapshot() as (conn, _epoch, _ceiling):
            rows = conn.execute(
                "SELECT * FROM (SELECT NULLIF(TRIM(p.category),'') AS category,"
                "COALESCE(SUM(p.run_count),0) AS runs,AVG(p.rating) AS rating "
                "FROM prompts p WHERE " + _ELIGIBLE + " GROUP BY 1) "
                "ORDER BY runs DESC,category IS NULL,category"
            ).fetchall()
        return [
            {
                "category": row["category"],
                "runs": row["runs"],
                "average_rating": (
                    None if row["rating"] is None else round(row["rating"], 2)
                ),
            }
            for row in rows
        ]
@@ -0,0 +1,310 @@
 """Transactional native Library commands guarded by epoch and edit token."""
 import re
 import sqlite3
 from .library import _ELIGIBLE
 from .models import utc_now_iso
 try:
    from ..utils.validators import (
        validate_category,
        validate_prompt_text,
        validate_tags,
    )
 except ImportError:
    from utils.validators import validate_category, validate_prompt_text, validate_tags
 METADATA_FIELDS = frozenset({"title", "category", "tags", "rating", "notes"})
 MAX_TEXT_FIELD = 10000
 _UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
 _TOKEN = re.compile(r"[0-9a-f]{64}")
 # Parent metadata stored on the prompt row; title lives in prompt_library_state.
 _PROMPT_COLUMNS = ("category", "rating", "notes")
 class LibraryCommandError(Exception):
    """A native command lost a race against storage or another writer.
    ``code`` is a wire code (``metadata-conflict``, ``storage-changed`` or
    ``vocabulary-conflict`` for a tag or category that changed since it was read),
    ``epoch`` the storage epoch read inside the transaction and ``positive``
    the current owner record when it may be disclosed.
    """
    def __init__(self, code: str, epoch: str, positive: dict | None = None):
        super().__init__(code)
        self.code = code
        self.epoch = epoch
        self.positive = positive
 def validate_library_metadata(metadata: dict, *, require_title: bool = False) -> None:
    """Reject anything but a nonempty allowlisted parent metadata object."""
    if not isinstance(metadata, dict) or not metadata:
        raise ValueError("metadata must be a nonempty object")
    unknown = set(metadata) - METADATA_FIELDS
    if unknown:
        raise ValueError("Unknown metadata fields: " + ", ".join(sorted(unknown)))
    if require_title and (
        not isinstance(metadata.get("title"), str) or not metadata["title"].strip()
    ):
        raise ValueError("title is required")
    for name in ("title", "notes"):
        value = metadata.get(name)
        if value is not None and not isinstance(value, str):
            raise ValueError(f"{name} must be a string or null")
        if value is not None and len(value) > MAX_TEXT_FIELD:
            raise ValueError(f"{name} is too long")
    if metadata.get("category") is not None:
        validate_category(metadata["category"])
    if "tags" in metadata:
        if not isinstance(metadata["tags"], list):
            raise ValueError("tags must be a list of strings")
        validate_tags(metadata["tags"])
    rating = metadata.get("rating")
    if rating is not None and (type(rating) is not int or not 1 <= rating <= 5):
        raise ValueError("rating must be an integer between 1 and 5 or null")
 def normalize_library_metadata(metadata: dict) -> dict:
    """Return validated metadata with category trimmed and blank stored as NULL."""
    if "category" not in metadata:
        return metadata
    category = metadata["category"]
    return {**metadata, "category": (category or "").strip() or None}
 def normalize_library_tags(tags: list[str]) -> list[str]:
    """Trim and deduplicate validated tags, keeping first-seen order."""
    return list(dict.fromkeys(tag.strip() for tag in tags))
 def _validate_epoch(expected_epoch: str) -> None:
    if not isinstance(expected_epoch, str) or not _UUID.fullmatch(expected_epoch):
        raise ValueError("expected_epoch must be a canonical UUID")
 def _validate_guards(positive_id: int, expected_epoch: str, if_match: str) -> None:
    if type(positive_id) is not int or not 1 <= positive_id <= 2**63 - 1:
        raise ValueError("Invalid positive prompt ID")
    _validate_epoch(expected_epoch)
    if not isinstance(if_match, str) or not _TOKEN.fullmatch(if_match):
        raise ValueError("if_match must be 64 lowercase hexadecimal characters")
 def _validate_negative_text(negative_text: str) -> None:
    """Require an explicit string; blank deliberately means the empty negative."""
    if not isinstance(negative_text, str):
        raise ValueError("negative_text must be explicit; use an empty string for none")
    if negative_text.strip():
        validate_prompt_text(negative_text)
 class LibraryCommandOperations:
    """Connection-level parent commands mixed into PromptDatabase."""
    def _library_epoch(self, conn: sqlite3.Connection, expected_epoch: str) -> str:
        row = conn.execute(
            "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
        ).fetchone()
        if row is None:
            raise sqlite3.DatabaseError("Missing execution history metadata")
        if row[0] != expected_epoch:
            raise LibraryCommandError("storage-changed", row[0])
        return row[0]
    def _library_parent(
        self, conn: sqlite3.Connection, epoch: str, positive_id: int
    ) -> dict | None:
        rows = self._library_parents(
            conn, epoch, _ELIGIBLE + " AND p.id=?", [positive_id]
        )
        return rows[0] if rows else None
    def _ensure_library_parent(self, conn: sqlite3.Connection, prompt_id: int) -> None:
        """Register explicit parent intent once; an existing row is never touched."""
        if conn.execute(
            "SELECT 1 FROM prompt_library_state WHERE prompt_id=?", (prompt_id,)
        ).fetchone():
            return
        conn.execute(
            "INSERT INTO prompt_library_state(prompt_id,title) VALUES(?,NULL)",
            (prompt_id,),
        )
    def _apply_library_metadata(
        self, conn: sqlite3.Connection, current: dict, metadata: dict
    ) -> None:
        """Write only fields that differ; stamp ``prompts.updated_at`` once."""
        prompt_id = current["id"]
        changed = False
        if "title" in metadata and metadata["title"] != current["title"]:
            conn.execute(
                "UPDATE prompt_library_state SET title=? WHERE prompt_id=?",
                (metadata["title"], prompt_id),
            )
            changed = True
        columns = {
            name: metadata[name]
            for name in _PROMPT_COLUMNS
            if name in metadata and metadata[name] != current[name]
        }
        if "tags" in metadata:
            tags = normalize_library_tags(metadata["tags"])
            if set(tags) != set(current["tags"]):
                self._sync_prompt_tags(conn, prompt_id, tags)
                changed = True
        if columns or changed:
            columns["updated_at"] = utc_now_iso()
            conn.execute(
                "UPDATE prompts SET "
                + ",".join(f"{name}=?" for name in columns)
                + " WHERE id=?",
                [*columns.values(), prompt_id],
            )
    def update_library_parent(
        self,
        positive_id: int,
        *,
        expected_epoch: str,
        if_match: str,
        metadata: dict,
    ) -> dict | None:
        """Apply explicit parent metadata if the epoch and edit token still match.
        Returns ``{library_epoch, positive}`` read before commit, or None when
        the positive is missing or not a Library parent. Raises
        LibraryCommandError for a rotated epoch or a stale token.
        """
        _validate_guards(positive_id, expected_epoch, if_match)
        validate_library_metadata(metadata)
        metadata = normalize_library_metadata(metadata)
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            current = self._library_parent(conn, epoch, positive_id)
            if current is None:
                return None
            if current["edit_token"] != if_match:
                raise LibraryCommandError("metadata-conflict", epoch, current)
            self._ensure_library_parent(conn, positive_id)
            self._apply_library_metadata(conn, current, metadata)
            positive = self._library_parent(conn, epoch, positive_id)
            if positive is None:
                raise sqlite3.DatabaseError("Library parent vanished during update")
            return {"library_epoch": epoch, "positive": positive}
    def _recipe_negative(
        self, conn: sqlite3.Connection, negative_text: str
    ) -> int | None:
        if not negative_text.strip():
            return None
        return self._pair_component_outcome(conn, negative_text)[0]
    def _recipe_receipt(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        positive_id: int,
        pair_id: int,
        *,
        positive_created: bool,
        pair_created: bool,
    ) -> dict:
        """Read both owners inside the write transaction for the receipt."""
        positive = self._library_parent(conn, epoch, positive_id)
        pairs = self._read_pair_rows(conn, "pp.id = ?", [pair_id])
        if positive is None or not pairs:
            raise sqlite3.DatabaseError("Library recipe vanished during save")
        return {
            "library_epoch": epoch,
            "positive": positive,
            "pair": pairs[0],
            "positive_created": positive_created,
            "pair_created": pair_created,
            # Parent metadata is applied only to a newly inserted positive.
            "parent_metadata_applied": positive_created,
        }
    def create_library_recipe(
        self,
        positive_text: str,
        negative_text: str,
        *,
        expected_epoch: str,
        parent_metadata: dict,
    ) -> dict:
        """Create or reuse a positive and its ordered pair in one transaction.
        Submitted parent metadata applies only when the positive component is
        newly inserted; a reused component keeps its own metadata. New pairs get
        empty pair metadata. Reused pairs keep theirs, including archive state.
        Every success registers the positive as a Library parent.
        """
        _validate_epoch(expected_epoch)
        validate_prompt_text(positive_text)
        _validate_negative_text(negative_text)
        validate_library_metadata(parent_metadata, require_title=True)
        parent_metadata = normalize_library_metadata(parent_metadata)
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            positive_id, positive_created = self._pair_component_outcome(
                conn, positive_text
            )
            negative_id = self._recipe_negative(conn, negative_text)
            pair_id, pair_created = self._save_pair_components(
                conn, positive_id, negative_id
            )
            self._ensure_library_parent(conn, positive_id)
            if positive_created:
                current = self._library_parent(conn, epoch, positive_id)
                if current is None:
                    raise sqlite3.DatabaseError("New Library parent is not eligible")
                self._apply_library_metadata(conn, current, parent_metadata)
            return self._recipe_receipt(
                conn,
                epoch,
                positive_id,
                pair_id,
                positive_created=positive_created,
                pair_created=pair_created,
            )
    def add_library_negative(
        self,
        positive_id: int,
        negative_text: str,
        *,
        expected_epoch: str,
        if_match: str,
    ) -> dict | None:
        """Create or reuse a pair under the exact stored positive.
        The stored positive ID and text are used as they are; only the negative
        is resolved. Returns None when the positive is missing or not a Library
        parent, and raises LibraryCommandError for a rotated epoch or a stale
        token, in both cases before any insertion.
        """
        _validate_guards(positive_id, expected_epoch, if_match)
        _validate_negative_text(negative_text)
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            current = self._library_parent(conn, epoch, positive_id)
            if current is None:
                return None
            if current["edit_token"] != if_match:
                raise LibraryCommandError("metadata-conflict", epoch, current)
            negative_id = self._recipe_negative(conn, negative_text)
            pair_id, pair_created = self._save_pair_components(
                conn, positive_id, negative_id
            )
            self._ensure_library_parent(conn, positive_id)
            return self._recipe_receipt(
                conn,
                epoch,
                positive_id,
                pair_id,
                positive_created=False,
                pair_created=pair_created,
            )
@@ -0,0 +1,805 @@
 """Move library (#249 S6b): copy the node-folder library, verify it, switch.
 ``plan_move`` checks a ``utils.library_location`` location can move and
 sizes the copy; ``move_library`` does it, in this order:
 1. **Disk check** on the target's volume (the copy, the backups and
   config.json, plus a margin). The source is never renamed across folders,
   so another drive is simply a copy.
 2. **Copy** with SQLite's backup API from a private connection holding one
   read transaction: the per-table counts, the workflow documents' digest
   and ``PRAGMA data_version`` describe exactly the copied snapshot. The copy
   is a temp file in the target folder, made self-contained
   (``journal_mode=DELETE``), fsynced and owner-only.
 3. **Verify**: ``verify_database_file`` (integrity check, schema, every
   workflow document decompressed against its hash, #315), the same row
   count in every table and the same documents.
 4. **Copy files**: every pooled safety backup and pre-migration backup,
   checked by size and SHA-256 after the copy (their times are kept, so the
   pool prunes in the same order).
 5. **Publish** under the old database's write lock (``BEGIN IMMEDIATE``) and
   the config.json lock: if any other connection wrote since the snapshot
   (``data_version``), steps 2–4 run again (``attempts``, then
   ``library-busy``). Else config.json is copied (owner-only, checked), a
   ``pending`` marker is written, the temp files take their final names
   (``os.replace`` inside the target folder), the marker turns ``moved``
   and ``switch`` points the process at the new library while the old file
   is still locked, so no write can land in it.
 6. **Retire**: the old file's WAL is checkpointed and the old ``prompts.db``
   and ``config.json`` are renamed ``*.moved-to-<folder>`` (never deleted;
   the old backups stay where they are). A file still open elsewhere is left
   in place and reported (``old-in-use``): the marker already makes a restart
   use the new library.
 Any failure before the marker says ``moved`` removes everything the move
 wrote in the target folder and leaves the old library in use, untouched.
 A crash at any point leaves either the old library in use (the marker is
 absent or ``pending``; the next move replaces its own leftovers) or the new
 one (``moved``). ``MoveFailed`` carries a code and a message without paths.
 """
 import contextlib
 import hashlib
 import json
 import os
 import shutil
 import sqlite3
 import time
 import types
 import uuid
 from pathlib import Path
 from dataclasses import dataclass, field
 from typing import Callable, Dict, List, Optional, Tuple
 from .models import BUSY_TIMEOUT_SECONDS, PromptModel
 try:
    from ..utils import library_location as ll
    from ..utils.logging_config import get_logger
 except ImportError:
    from utils import library_location as ll
    from utils.logging_config import get_logger
 logger = get_logger("prompt_manager.database.library_move")
 TEMP_PREFIX = ".pm-move-"
 MOVED_INFIX = ".moved-to-"
 DISK_MARGIN_BYTES = 16 * 2**20
 DEFAULT_ATTEMPTS = 3
 _COPY_CHUNK = 4 * 2**20
 # Backups that live beside the database and move with it.
 BACKUP_INFIXES = (".backup_", ".pre-workflow-storage_")
 MESSAGES = {
    "not-movable": (
        "This library is not in the node folder, so there is nothing to move."
    ),
    "same-folder": "The new folder is the node folder itself.",
    "target-invalid": "The new folder cannot hold the library.",
    "target-has-library": (
        "The new folder already holds a Prompt Manager library. Nothing was moved."
    ),
    "target-conflict": (
        "The new folder already holds files with the same names. Nothing was moved."
    ),
    "insufficient-space": (
        "There is not enough free space for the copy. Nothing was moved."
    ),
    "copy-failed": "The copy could not be written or did not match. Nothing was moved.",
    "verify-failed": "The copy did not pass verification. Nothing was moved.",
    "library-busy": (
        "Prompt Manager kept writing to the library while it was copied. "
        "Nothing was moved; try again when no workflow is running."
    ),
    "switch-failed": (
        "Prompt Manager could not switch to the verified copy, so the copy was "
        "removed. Nothing was moved."
    ),
 }
 class MoveFailed(Exception):
    """A move that did not happen (or did not finish), with a coded reason."""
    def __init__(self, code: str, message: Optional[str] = None):
        self.code = code
        self.message = message or MESSAGES.get(code, "The library was not moved.")
        super().__init__(self.message)
@dataclass(frozen=True)
 class MovePlan:
    location: "ll.LibraryLocation"
    db_path: str
    source_dir: str
    target_dir: str
    config_path: Optional[str]
    backups: Tuple[str, ...]
    db_bytes: int
    files_bytes: int
    @property
    def required_bytes(self) -> int:
        return self.db_bytes + self.files_bytes + DISK_MARGIN_BYTES
    @property
    def target_location(self) -> "ll.LibraryLocation":
        """Where the library is once moved (an explicit config.json stays)."""
        source = self.location.target_source or ll.SOURCE_DEFAULT
        config = (
            self.location.config_path
            if self.location.config_explicit
            else os.path.join(self.target_dir, ll.CONFIG_NAME)
        )
        return ll.LibraryLocation(
            source,
            os.path.join(self.target_dir, ll.DATABASE_NAME),
            config,
            self.target_dir,
            self.target_dir,
            None,
            source,
            self.location.config_explicit,
        )
@dataclass
 class MoveResult:
    location: "ll.LibraryLocation"
    counts: Dict[str, int]
    files: List[str]
    renamed: List[str] = field(default_factory=list)
    retire_problem: Optional[str] = None
 def _size(path: str) -> int:
    try:
        return os.path.getsize(path)
    except OSError:
        return 0
 def _free_bytes(path: str) -> int:
    return shutil.disk_usage(path).free
 def _existing_ancestor(path: str) -> str:
    current = os.path.abspath(path)
    while not os.path.isdir(current):
        parent = os.path.dirname(current)
        if parent == current:
            break
        current = parent
    return current
 def backup_names(db_path: str) -> Tuple[str, ...]:
    """The pooled and pre-migration backups beside ``db_path`` (names)."""
    folder = os.path.dirname(os.path.abspath(db_path))
    base = os.path.basename(db_path)
    prefixes = tuple(base + infix for infix in BACKUP_INFIXES)
    try:
        names = os.listdir(folder)
    except OSError:
        return ()
    return tuple(
        sorted(
            name
            for name in names
            if name.startswith(prefixes) and os.path.isfile(os.path.join(folder, name))
        )
    )
 def plan_move(location) -> MovePlan:
    """Check ``location`` can move to its target; raises MoveFailed."""
    if not location.movable:
        raise MoveFailed("not-movable")
    target = os.path.abspath(location.target_dir)
    source_dir = os.path.dirname(os.path.abspath(location.db_path))
    if ll.same_folder(target, source_dir) or os.path.normcase(
        os.path.realpath(target)
    ) == os.path.normcase(os.path.realpath(source_dir)):
        raise MoveFailed("same-folder")
    if os.path.exists(target) and not os.path.isdir(target):
        raise MoveFailed("target-invalid")
    backups = backup_names(location.db_path)
    # config.json moves with a node-folder library; one named by
    # PROMPT_MANAGER_CONFIG_PATH is the user's and stays where it is.
    config = None if location.config_explicit else location.config_path
    config = config if config and os.path.isfile(config) else None
    if ll.read_marker(target).get("state") != ll.MARKER_PENDING:
        if os.path.exists(os.path.join(target, ll.DATABASE_NAME)):
            raise MoveFailed("target-has-library")
        names = [ll.CONFIG_NAME] if config else []
        if any(os.path.exists(os.path.join(target, n)) for n in names + list(backups)):
            raise MoveFailed("target-conflict")
    db_bytes = _size(location.db_path) + _size(location.db_path + "-wal")
    files_bytes = sum(_size(os.path.join(source_dir, n)) for n in backups)
    files_bytes += _size(config) if config else 0
    return MovePlan(
        location,
        os.path.abspath(location.db_path),
        source_dir,
        target,
        config,
        backups,
        db_bytes,
        files_bytes,
    )
 def space_problem(plan: MovePlan) -> Optional[Tuple[int, int]]:
    """``(required, free)`` when the target's volume is short, else None."""
    free = _free_bytes(_existing_ancestor(plan.target_dir))
    if free < plan.required_bytes:
        return plan.required_bytes, free
    return None
 # ── copy and verify ───────────────────────────────────────────────────
 def _tables(conn) -> List[str]:
    return [
        row[0]
        for row in conn.execute(
            "SELECT name FROM sqlite_master WHERE type='table' "
            "AND name NOT LIKE 'sqlite_%' ORDER BY name"
        )
    ]
 def _describe(conn) -> Tuple[Dict[str, int], str]:
    """Row counts per table and a digest of the workflow documents."""
    counts = {}
    for table in _tables(conn):
        # A name read from sqlite_master, quoted as an identifier.
        quoted = '"' + table.replace('"', '""') + '"'
        sql = f"SELECT COUNT(*) FROM {quoted}"  # nosec B608
        counts[table] = conn.execute(sql).fetchone()[0]
    digest = hashlib.sha256()
    if "workflow_documents" in counts:
        for row in conn.execute(
            "SELECT id, sha256, byte_length FROM workflow_documents ORDER BY id"
        ):
            digest.update(repr((row[0], bytes(row[1]), row[2])).encode())
    return counts, digest.hexdigest()
 def _fsync_file(path: str) -> None:
    with open(path, "rb+") as handle:
        os.fsync(handle.fileno())
 def _fsync_dir(path: str) -> None:
    if os.name != "posix":
        return
    fd = os.open(path, os.O_RDONLY)
    try:
        os.fsync(fd)
    finally:
        os.close(fd)
 def _private(path: str) -> None:
    if os.name == "posix":
        os.chmod(path, 0o600)
 def _copy_snapshot(source: sqlite3.Connection, temp: str):
    """Back up ``source``'s current snapshot to ``temp``; return what it holds."""
    source.execute("BEGIN")
    try:
        version = source.execute("PRAGMA data_version").fetchone()[0]
        counts, documents = _describe(source)
        destination = sqlite3.connect(temp)
        try:
            source.backup(destination)
        finally:
            destination.close()
    finally:
        source.execute("COMMIT")
    with contextlib.closing(sqlite3.connect(temp)) as conn:
        conn.execute("PRAGMA journal_mode=DELETE").fetchone()
    _fsync_file(temp)
    _private(temp)
    return version, counts, documents
 def _verify_copy(temp: str, counts, documents) -> None:
    ok, reason = PromptModel.verify_database_file(temp)
    if not ok:
        logger.error(f"Library move: the copy did not verify: {reason}")
        detail = reason if reason.startswith("Workflow document") else None
        raise MoveFailed(
            "verify-failed",
            f"{MESSAGES['verify-failed']} {detail}" if detail else None,
        )
    uri = Path(os.path.abspath(temp)).as_uri() + "?mode=ro"
    try:
        with contextlib.closing(sqlite3.connect(uri, uri=True)) as conn:
            copied = _describe(conn)
    except sqlite3.Error as error:
        logger.error(f"Library move: the copy could not be read: {error}")
        raise MoveFailed("verify-failed") from error
    if copied != (counts, documents):
        logger.error("Library move: the copy's rows or documents differ")
        raise MoveFailed("verify-failed")
 def _hash_copy(source: str, destination: str) -> Tuple[int, str]:
    """Copy ``source`` to ``destination`` (fsynced, owner-only, times kept)."""
    digest = hashlib.sha256()
    size = 0
    with open(source, "rb") as reader, open(destination, "wb") as writer:
        while True:
            chunk = reader.read(_COPY_CHUNK)
            if not chunk:
                break
            digest.update(chunk)
            size += len(chunk)
            writer.write(chunk)
        writer.flush()
        os.fsync(writer.fileno())
    stat = os.stat(source)
    os.utime(destination, ns=(stat.st_atime_ns, stat.st_mtime_ns))
    _private(destination)
    return size, digest.hexdigest()
 def _file_digest(path: str) -> Tuple[int, str]:
    digest = hashlib.sha256()
    size = 0
    with open(path, "rb") as reader:
        while True:
            chunk = reader.read(_COPY_CHUNK)
            if not chunk:
                return size, digest.hexdigest()
            digest.update(chunk)
            size += len(chunk)
 def _check_copies(copies) -> None:
    for source, temp, expected in copies:
        if _file_digest(temp) != expected or _file_digest(source) != expected:
            logger.error(
                f"Library move: {os.path.basename(source)} did not copy unchanged"
            )
            raise MoveFailed("copy-failed")
 def write_marker(
    directory: str, state: str, source_db: str, source_config: Optional[str] = None
 ) -> str:
    """Atomically write the move marker; return its path.
    ``from`` and ``config_from`` name the old files, so a restart after a
    crash between the switch and the rename can finish the rename.
    """
    path = os.path.join(directory, ll.MARKER_NAME)
    temp = os.path.join(directory, f"{TEMP_PREFIX}{uuid.uuid4().hex}.marker")
    payload = json.dumps(
        {
            "version": 1,
            "state": state,
            "from": source_db,
            "config_from": source_config,
            "updated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        },
        indent=2,
    ).encode("utf-8")
    fd = os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    try:
        with os.fdopen(fd, "wb") as handle:
            handle.write(payload)
            handle.flush()
            os.fsync(handle.fileno())
        _private(temp)
        os.replace(temp, path)
    except BaseException:
        _remove(temp)
        raise
    _fsync_dir(directory)
    return path
 def _remove(*paths) -> None:
    for path in paths:
        try:
            if path and os.path.lexists(path):
                os.remove(path)
        except OSError:
            logger.warning(f"Library move: could not remove {os.path.basename(path)}")
 def sweep_move_temps(directory: str) -> List[str]:
    """Remove the temp files an interrupted move left in ``directory``."""
    removed = []
    try:
        names = os.listdir(directory)
    except OSError:
        return removed
    for name in names:
        if name.startswith(TEMP_PREFIX):
            _remove(os.path.join(directory, name))
            removed.append(name)
    return removed
 # ── the move ─────────────────────────────────────────────────────────
 def _noop(*_args, **_kwargs):
    return None
 def switch_process(old_db: str, location, detached: list) -> None:
    """Point this process at ``location``: resolver, every model, observers.
    Adds the old file's connections, detached and still open, to
    ``detached`` as soon as they are detached, so the mover retires them
    (``PromptModel.retire_detached``) whether or not the rest succeeds.
    """
    from .execution_history import rekey_history_restore
    from .review_changes import rekey_review_changes
    ll.switch_to(location)
    detached.extend(PromptModel.repoint_instances(old_db, location.db_path))
    rekey_review_changes(old_db, location.db_path)
    rekey_history_restore(old_db, location.db_path)
 def unswitch_process(old_db: str, new_db: str) -> list:
    """Undo ``switch_process`` (any part of it); returns detached connections."""
    from .execution_history import rekey_history_restore
    from .review_changes import rekey_review_changes
    ll.clear_override()
    detached = PromptModel.repoint_instances(new_db, old_db)
    rekey_review_changes(new_db, old_db)
    rekey_history_restore(new_db, old_db)
    return detached
 class _Move:
    def __init__(self, plan, *, switch, config_lock, progress, checkpoint):
        self.plan = plan
        self.switch = switch
        self.config_lock = config_lock
        self.progress = progress
        self.checkpoint = checkpoint
        self.created_dir = False
        self.temps: List[str] = []
        self.published: List[str] = []
        self.marker: Optional[str] = None
        self.moved = False
        # Retired connections still in use; awaited once the locks are free.
        self.retiring: list = []
    def temp(self, suffix: str) -> str:
        path = os.path.join(
            self.plan.target_dir, f"{TEMP_PREFIX}{uuid.uuid4().hex}{suffix}"
        )
        self.temps.append(path)
        return path
    def prepare_target(self) -> None:
        target = self.plan.target_dir
        self.created_dir = not os.path.isdir(target)
        ll.ensure_private_dir(target)
        sweep_move_temps(target)
        if ll.read_marker(target).get("state") == ll.MARKER_PENDING:
            # An earlier move stopped before it switched: its files are ours.
            for name in (ll.DATABASE_NAME, ll.CONFIG_NAME) + self.plan.backups:
                _remove(os.path.join(target, name))
            for suffix in ("-wal", "-shm", "-journal"):
                _remove(os.path.join(target, ll.DATABASE_NAME + suffix))
    def copy_round(self, source):
        """Steps 2–4 once: the verified temp copies and the snapshot version."""
        self.progress("Copying the database")
        temp_db = self.temp(".db")
        version, counts, documents = _copy_snapshot(source, temp_db)
        self.checkpoint("copied")
        self.progress("Verifying the copy")
        _verify_copy(temp_db, counts, documents)
        self.checkpoint("verified")
        copies = []
        for name in self.plan.backups:
            src = os.path.join(self.plan.source_dir, name)
            temp = self.temp(".bak")
            try:
                expected = _hash_copy(src, temp)
            except OSError as error:
                logger.error(f"Library move: copying {name} failed: {error}")
                raise MoveFailed("copy-failed") from error
            copies.append((src, temp, expected, name))
        if copies:
            self.progress(f"Copied {len(copies)} backup file(s)")
        self.checkpoint("files")
        _check_copies([(s, t, e) for s, t, e, _ in copies])
        return version, counts, temp_db, copies
    def discard_round(self, temp_db, copies) -> None:
        _remove(temp_db, *[temp for _s, temp, _e, _n in copies])
    def publish(self, temp_db, copies) -> List[str]:
        """Under the locks: config.json, marker, final names, marker moved."""
        target = self.plan.target_dir
        files = []
        if self.plan.config_path:
            temp = self.temp(".config")
            try:
                expected = _hash_copy(self.plan.config_path, temp)
            except OSError as error:
                raise MoveFailed("copy-failed") from error
            _check_copies([(self.plan.config_path, temp, expected)])
            copies = copies + [(self.plan.config_path, temp, expected, ll.CONFIG_NAME)]
        self.marker = write_marker(
            target, ll.MARKER_PENDING, self.plan.db_path, self.plan.config_path
        )
        self.checkpoint("pending")
        for _src, temp, _expected, name in copies:
            final = os.path.join(target, name)
            os.replace(temp, final)
            self.published.append(final)
            files.append(name)
        final_db = os.path.join(target, ll.DATABASE_NAME)
        os.replace(temp_db, final_db)
        self.published.append(final_db)
        files.append(ll.DATABASE_NAME)
        _fsync_dir(target)
        self.checkpoint("published")
        write_marker(target, ll.MARKER_MOVED, self.plan.db_path, self.plan.config_path)
        self.moved = True
        self.checkpoint("marked")
        return files
    def cleanup(self) -> None:
        if self.moved:
            return
        _remove(*self.temps, *self.published, self.marker)
        if self.created_dir:
            try:
                os.rmdir(self.plan.target_dir)
            except OSError:
                pass
 def _unique(path: str) -> str:
    candidate, number = path, 1
    while os.path.lexists(candidate):
        candidate = f"{path}_{number}"
        number += 1
    return candidate
 def retire_old_files(plan: MovePlan) -> Tuple[List[str], Optional[str]]:
    """Rename the old database and config.json; ``(renamed, problem code)``.
    Every connection to the old file must be closed. The WAL is checkpointed
    first; when it cannot be emptied (another process still has the file
    open) nothing is renamed (``old-in-use``).
    """
    old = plan.db_path
    folder = os.path.basename(os.path.normpath(plan.target_dir))
    try:
        with contextlib.closing(sqlite3.connect(old, timeout=1)) as conn:
            busy = conn.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0]
    except sqlite3.Error as error:
        logger.warning(
            f"Library move: the old database could not be checkpointed: {error}"
        )
        return [], "old-in-use"
    if busy or _size(old + "-wal") > 0:
        return [], "old-in-use"
    renamed = []
    try:
        for path in (old, plan.config_path):
            if not path or not os.path.isfile(path):
                continue
            new_name = _unique(path + MOVED_INFIX + folder)
            os.rename(path, new_name)
            renamed.append(os.path.basename(new_name))
        for suffix in ("-wal", "-shm"):
            if os.path.isfile(old + suffix) and _size(old + suffix) == 0:
                os.remove(old + suffix)
    except OSError as error:
        logger.warning(f"Library move: the old files could not be renamed: {error}")
        return renamed, "old-locked"
    return renamed, None
 def finish_interrupted_move(location) -> List[str]:
    """Rename old files a crash left between the switch and the rename.
    The marker of the folder in use says ``moved`` and names the old files;
    an old ``prompts.db`` (and config.json) still there under its own name is
    renamed ``*.moved-to-<folder>`` now, never deleted. Returns the new
    names (nothing when there is nothing to finish or the file is in use).
    """
    if location.source not in (ll.SOURCE_DEFAULT, ll.SOURCE_ENVIRONMENT):
        return []
    marker = ll.read_marker(location.data_dir)
    old = marker.get("from")
    if marker.get("state") != ll.MARKER_MOVED or not isinstance(old, str):
        return []
    if not os.path.isfile(old) or os.path.basename(old) != ll.DATABASE_NAME:
        return []
    if ll.same_folder(os.path.dirname(old), location.data_dir):
        return []
    config = marker.get("config_from")
    if not (
        isinstance(config, str)
        and os.path.basename(config) == ll.CONFIG_NAME
        and os.path.isfile(config)
        and os.path.normcase(os.path.abspath(config))
        != os.path.normcase(os.path.abspath(location.config_path))
    ):
        config = None
    pending = types.SimpleNamespace(
        db_path=old, config_path=config, target_dir=location.data_dir
    )
    renamed, problem = retire_old_files(pending)
    if problem:
        logger.warning(f"Library move: the old files are still in use ({problem})")
    elif renamed:
        logger.info(f"Library move finished: renamed {', '.join(renamed)}")
    return renamed
 def move_library(
    plan: MovePlan,
    *,
    switch: Callable = switch_process,
    unswitch: Callable = unswitch_process,
    after_switch: Callable = _noop,
    config_lock=None,
    progress: Callable[[str], None] = _noop,
    checkpoint: Callable[[str], None] = _noop,
    attempts: int = DEFAULT_ATTEMPTS,
 ) -> MoveResult:
    """Move the library as planned (blocking; see the module docstring).
    ``switch(old_db, location, detached)`` points the process at the copy
    and adds the old connections it detached to ``detached``; if it raises,
    ``unswitch(old_db, new_db)`` undoes it while both files are still
    locked, every detached connection is closed, the copy is removed and
    the move fails (``switch-failed``). ``after_switch()`` runs
    once the switch is final; it is best effort (logged, never fails the move).
    """
    short = space_problem(plan)
    if short is not None:
        raise MoveFailed("insufficient-space")
    lock = config_lock if config_lock is not None else contextlib.nullcontext()
    move = _Move(
        plan,
        switch=switch,
        config_lock=lock,
        progress=progress,
        checkpoint=checkpoint,
    )
    move.unswitch = unswitch
    location = plan.target_location
    try:
        move.prepare_target()
        files, counts = _copy_and_switch(move, location, attempts)
    except Exception:
        move.cleanup()
        raise
    try:
        after_switch()
    except Exception:
        logger.exception("Library move: a follow-up after the switch failed")
    from .execution_history import notify_history_restore
    notify_history_restore(location.db_path)
    progress("Renaming the old files")
    renamed, problem = retire_old_files(plan)
    checkpoint("retired")
    return MoveResult(location, counts, files, renamed, problem)
 def _switch_guarded(move: _Move, location) -> list:
    """Switch while the copy is locked too; undo it all if any part fails.
    The copy's write lock keeps every repointed thread from writing to it
    until the switch is whole, so undoing a partial switch loses nothing.
    Returns the old file's detached connections (still open). On failure
    every detached connection is retired; those still in use are added to
    ``move.retiring`` for the caller to await once the locks are released.
    """
    plan = move.plan
    guard = sqlite3.connect(
        location.db_path, timeout=BUSY_TIMEOUT_SECONDS, isolation_level=None
    )
    try:
        guard.execute("BEGIN IMMEDIATE")
        detached: list = []
        try:
            move.switch(plan.db_path, location, detached)
        except Exception as error:
            logger.exception("Library move: switching failed; undoing it")
            move.moved = False  # the copy and its marker are removed
            # Retire, never wait, here: a connection still in use may be
            # waiting on the very locks this thread holds.
            try:
                undone = move.unswitch(plan.db_path, location.db_path)
            except Exception:
                logger.exception("Library move: undoing the switch failed")
            else:
                _retire(move, undone)
            # The old file's connections a partial switch detached: the
            # models reopen it on next use, so close them now, not at GC.
            _retire(move, detached)
            raise MoveFailed("switch-failed") from error
        move.checkpoint("switched")
        return detached
    finally:
        if guard.in_transaction:
            guard.execute("ROLLBACK")
        guard.close()
 def _copy_and_switch(move: _Move, location, attempts: int):
    """Copy, then switch under both locks; retired connections are awaited.
    A connection that was inside a statement when it was retired is closed
    by its own thread once the statement ends, which may need the locks
    released first; the old files are renamed (and a failed copy removed)
    only after that, since Windows refuses to rename or delete an open file.
    """
    try:
        return _copy_and_switch_locked(move, location, attempts)
    finally:
        try:
            PromptModel.await_detached(move.retiring, logger)
        except Exception:
            logger.exception("Library move: waiting for old connections failed")
 def _retire(move: _Move, connections: list) -> None:
    """Retire ``connections``, adding those still in use to ``move.retiring``.
    Never raises: a switch that completed must not be reported as failed
    because one of the old connections could not be retired (logged; the
    rename then reports the old files as still in use).
    """
    try:
        move.retiring.extend(PromptModel.retire_detached(connections, logger))
    except Exception:
        logger.exception("Library move: retiring the old connections failed")
 def _copy_and_switch_locked(move: _Move, location, attempts: int):
    plan = move.plan
    for attempt in range(1, attempts + 1):
        source = sqlite3.connect(
            plan.db_path, timeout=BUSY_TIMEOUT_SECONDS, isolation_level=None
        )
        try:
            source.execute(f"PRAGMA busy_timeout = {BUSY_TIMEOUT_SECONDS * 1000}")
            version, counts, temp_db, copies = move.copy_round(source)
            source.execute("BEGIN IMMEDIATE")
            try:
                if source.execute("PRAGMA data_version").fetchone()[0] != version:
                    move.progress("The library changed during the copy; copying again")
                    move.discard_round(temp_db, copies)
                    continue
                move.checkpoint("locked")
                with move.config_lock:
                    files = move.publish(temp_db, copies)
                    detached = _switch_guarded(move, location)
                # Still holding the old file's lock: a write that was waiting
                # on it is interrupted and cannot commit, so it fails (and
                # runs again on the new library) instead of landing in the
                # old file. Its own thread closes the connection.
                _retire(move, detached)
            finally:
                if source.in_transaction:
                    source.execute("ROLLBACK")
            return files, counts
        finally:
            source.close()
    raise MoveFailed("library-busy")
@@ -0,0 +1,86 @@
 """Optional-old, required-current schema for Library parent state."""
 import re
 import sqlite3
 TABLE = "prompt_library_state"
 MAX_TITLE_LENGTH = 10_000
 _DDL = (
    f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, title TEXT, "
    "FOREIGN KEY(prompt_id) REFERENCES prompts(id) ON DELETE CASCADE)"
 )
 _COLUMNS = (("prompt_id", "INTEGER", 0, 1), ("title", "TEXT", 0, 0))
 _FOREIGN_KEYS = {("prompt_id", "prompts", "id", "CASCADE")}
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql).lower()
 def library_schema_error(
    conn: sqlite3.Connection, *, allow_missing: bool = True
 ) -> str | None:
    """Reject unsupported Library state schemas and invalid persisted values.
    Reads only: nothing is ever repaired. A missing table is acceptable for
    backups that predate it unless ``allow_missing`` is false.
    """
    objects = conn.execute(
        "SELECT type,sql FROM sqlite_master WHERE name=? "
        "OR (type IN ('index','trigger') AND tbl_name=?)",
        (TABLE, TABLE),
    ).fetchall()
    if not objects:
        return None if allow_missing else "Missing Library state schema"
    tables = [sql for kind, sql in objects if kind == "table"]
    if len(tables) != 1 or len(objects) != 1:
        return "Library state name is reserved by an unsupported schema object"
    if _compact(tables[0] or "") != _compact(_DDL):
        return "Library state has an invalid table definition"
    columns = tuple(
        (row[1], row[2].upper(), row[3], row[5])
        for row in conn.execute(f"PRAGMA table_info({TABLE})")
    )
    if columns != _COLUMNS:
        return "Library state has invalid columns"
    foreign_keys = {
        (row[3], row[2], row[4], row[6])
        for row in conn.execute(f"PRAGMA foreign_key_list({TABLE})")
    }
    if foreign_keys != _FOREIGN_KEYS:
        return "Library state has invalid foreign key definitions"
    if conn.execute(f"PRAGMA foreign_key_check({TABLE})").fetchone():
        return "Library state contains foreign key violations"
    # length() stops at an embedded NUL, so bound by bytes in SQL (characters
    # never outnumber bytes, and UTF-8 uses at most 4 bytes per character) and
    # count the few remaining candidates as decoded characters.
    if conn.execute(
        f"SELECT 1 FROM {TABLE} WHERE typeof(title) NOT IN ('null','text') "
        f"OR length(CAST(title AS BLOB)) > {MAX_TITLE_LENGTH * 4} LIMIT 1"
    ).fetchone():
        return "Library state contains invalid titles"
    for (raw,) in conn.execute(
        f"SELECT CAST(title AS BLOB) FROM {TABLE} "
        f"WHERE length(CAST(title AS BLOB)) > {MAX_TITLE_LENGTH}"
    ):
        if len(bytes(raw).decode("utf-8", "replace")) > MAX_TITLE_LENGTH:
            return "Library state contains invalid titles"
    return None
 def migrate_library_state(conn: sqlite3.Connection) -> None:
    """Install the table atomically; validate before and after creating it."""
    conn.execute("SAVEPOINT prompt_library_state_schema")
    try:
        error = library_schema_error(conn)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        error = library_schema_error(conn, allow_missing=False)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute("RELEASE prompt_library_state_schema")
    except BaseException:
        conn.execute("ROLLBACK TO prompt_library_state_schema")
        conn.execute("RELEASE prompt_library_state_schema")
        raise
@@ -0,0 +1,37 @@
 """Narrow comparison tokens for a Library parent's own metadata projection."""
 import hashlib
 import json
 _DOMAIN = "prompt-manager-library-parent:v1\n"
 def canonical_token(domain: str, preimage: dict) -> str:
    """SHA-256 of ``domain`` plus the canonical JSON of ``preimage``."""
    payload = json.dumps(
        preimage,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        allow_nan=False,
    )
    return hashlib.sha256((domain + payload).encode("utf-8")).hexdigest()
 def library_edit_token(epoch: str, positive: dict) -> str:
    """Hash the parent's owner fields under the current storage epoch.
    Timestamps, counts, pairs and previews are excluded, and an absent state
    row projects the same null title as a default one.
    """
    preimage = {
        "epoch": epoch,
        "id": positive["id"],
        "text": positive["text"],
        "title": positive["title"],
        "category": positive["category"],
        "notes": positive["notes"],
        "rating": positive["rating"],
        "tags": sorted(set(positive["tags"])),
    }
    return canonical_token(_DOMAIN, preimage)
@@ -0,0 +1,740 @@
 """Reviewed maintenance plans: read-only previews and their exact apply (#249 S6).
 Four operations, each the native owner of a legacy ``/maintenance`` one and
 built on the same queries, so a preview counts what the legacy cleanup would
 change:
 - ``duplicate_prompts``: prompts whose text is equal ignoring case and outer
  spaces (``DUPLICATE_PROMPTS_SQL``, legacy ``cleanup_duplicates``); each
  group keeps its oldest prompt, which takes over the others' images and
  metadata. A group with a member in a saved pair is kept whole.
 - ``optimize``: VACUUM (legacy ``vacuum``).
 - ``unused_components``: prompts nothing references (``ORPHANED_PROMPTS_SQL``,
  legacy ``prune_orphaned_prompts``, with the #239 protections).
 - ``missing_links``: image rows whose file is gone (legacy
  ``cleanup_orphaned_images``), counted only under a source that can be read
  now, as the health check's ``missing_files`` judges them
  (``health.link_states``). It deletes database rows only, never a file.
 ``preview_maintenance`` reads one snapshot on its own ``mode=ro`` connection
 (``health.readonly_uri``) and closes it in ``finally``; files are looked at
 after the snapshot ended. It returns a ``MaintenanceDraft``: the public plan
 fields, the targets an apply needs and the fingerprint of exactly what was
 reviewed (ids, text hashes, metadata, tags, Library state and pair refs of
 every prompt involved, or each link's row and source). The fingerprint never
 leaves the server.
 ``apply_maintenance`` recomputes that fingerprint on the write connection
 inside ``BEGIN IMMEDIATE`` and raises ``PlanChanged`` before writing anything
 when it differs; otherwise it applies the reviewed targets in batches of
 ``ID_BATCH`` in that one transaction. VACUUM cannot run in a transaction, so
 ``optimize`` binds to its database only (the caller checks that).
 """
 import hashlib
 import json
 import os
 import shutil
 import sqlite3
 from dataclasses import dataclass, field
 from pathlib import Path
 from .execution_contexts import LIVE_HINT_CTES
 from .health import BATCH_ROWS, link_states, readable_roots, readonly_uri
 from .pairs import PairDependencyError
 from .workflow_documents import documents_of_rows, release_documents
 MAINTENANCE_OPERATIONS = (
    "duplicate_prompts",
    "optimize",
    "unused_components",
    "missing_links",
    # Applied as its own batched job (database/workflow_storage.py).
    "workflow_storage",
 )
 MAX_DETAILS = 50  # detail rows a plan lists
 MAX_RESULT_ROWS = 200  # changed records an apply describes (the job lists 200)
 EXCERPT_CHARS = 80
 # Ids bound per statement: the pair-reference read binds each id twice, which
 # stays under SQLite's historical 999-variable limit.
 ID_BATCH = 400
 # Why a prompt no image links to is kept (#239): predicates on ``p``, in the
 # order the plan names them. ORPHANED_PROMPTS_SQL is built from them, so the
 # reasons a plan lists are exactly the rules the prune applies.
 PROTECTIONS = (
    (
        "EXISTS (SELECT 1 FROM prompt_pairs pp "
        "WHERE pp.positive_prompt_id = p.id OR pp.negative_prompt_id = p.id)",
        "in saved pairs",
        "in a saved pair",
    ),
    (
        "EXISTS (SELECT 1 FROM prompt_tags pt JOIN tags t ON pt.tag_id = t.id "
        "WHERE pt.prompt_id = p.id AND t.name = '__protected__')",
        "tagged __protected__",
        "tagged __protected__",
    ),
    (
        "EXISTS (SELECT 1 FROM prompt_library_state ls WHERE ls.prompt_id = p.id)",
        "with Library state (a title or an explicit parent)",
        "with Library state (a title or an explicit parent)",
    ),
    (
        "p.id IN (SELECT prompt_id FROM live_hint_prompts)",
        "used by recorded runs",
        "used by a recorded run",
    ),
 )
 # Exempt references: images, pairs, the __protected__ tag, Library parent
 # state, and live recorded-run hints (computed once, see LIVE_HINT_CTES).
 ORPHANED_PROMPTS_SQL = (
    "WITH "
    + LIVE_HINT_CTES
    + " SELECT p.id FROM prompts p"
    + " LEFT JOIN generated_images gi ON p.id = gi.prompt_id"
    + " WHERE gi.prompt_id IS NULL"
    + "".join(f" AND NOT ({predicate})" for predicate, _many, _one in PROTECTIONS)
 )
 # The prompts no image links to, with which protections keep each one.
 _UNLINKED_PROTECTIONS_SQL = (
    "WITH "
    + LIVE_HINT_CTES
    + " SELECT "
    + ", ".join(
        f"COALESCE(SUM({predicate}), 0)" for predicate, _many, _one in PROTECTIONS
    )
    + ", COALESCE(SUM("
    + " OR ".join(f"({predicate})" for predicate, _many, _one in PROTECTIONS)
    + "), 0) FROM prompts p WHERE NOT EXISTS "
    "(SELECT 1 FROM generated_images gi WHERE gi.prompt_id = p.id)"
 )
 # Duplicate prompt groups, as the legacy cleanup finds them (case-insensitive
 # text; GROUP_CONCAT order is sorted by created_at afterwards).
 DUPLICATE_PROMPTS_SQL = """
    SELECT LOWER(TRIM(text)) as normalized_text, COUNT(*) as count,
           GROUP_CONCAT(id) as ids,
           GROUP_CONCAT(created_at) as created_dates
    FROM prompts
    GROUP BY LOWER(TRIM(text))
    HAVING COUNT(*) > 1
 """
 _LINK_ROWS_SQL = (
    "SELECT id, image_path, filename, file_path FROM generated_images ORDER BY id"
 )
 # What a prompt's fingerprint covers, per batch of ids.
 _PROMPT_STATE_SQL = (
    "SELECT id, hash, text, category, rating, notes, created_at, updated_at "
    "FROM prompts WHERE id IN ({marks}) ORDER BY id",
    "SELECT prompt_id, tag_id FROM prompt_tags WHERE prompt_id IN ({marks}) "
    "ORDER BY prompt_id, tag_id",
    "SELECT * FROM prompt_library_state WHERE prompt_id IN ({marks}) "
    "ORDER BY prompt_id",
    "SELECT id, positive_prompt_id, negative_prompt_id FROM prompt_pairs "
    "WHERE positive_prompt_id IN ({marks}) OR negative_prompt_id IN ({marks}) "
    "ORDER BY id",
 )
 NO_FILES_DELETED = "No image file is deleted."
 LINKS_ONLY = "Database links only; no image file is deleted."
 UNUSED_DETAIL = "No image, saved pair, Library state or recorded run uses it"
 # Why a duplicate group is kept whole (``_duplicate_conflict``).
 CONFLICT_REASONS = {
    "collision": ("group links one image twice", "groups link one image twice"),
    "title": (
        "group has its own Library title",
        "groups have their own Library titles",
    ),
 }
 CONFLICT_DETAILS = {
    "collision": "Kept: two copies link the same image",
    "title": "Kept: a copy has its own Library title",
 }
 OPTIMIZE_LOCK_NOTE = (
    "Runs alone: the database is locked while it runs, and on a large library "
    "this takes a while. Prompt runs counted and images captured meanwhile "
    "wait up to 30 seconds each; anything that waits longer is not recorded."
 )
 class PlanChanged(Exception):
    """What a plan was reviewed on changed before it was applied."""
 class MergeConflict(Exception):
    """A strict duplicate merge would drop an image link or a Library title."""
@dataclass
 class MaintenanceDraft:
    """A preview: the public plan fields plus what only the server keeps."""
    operation: str
    fingerprint: str
    targets: list
    scope: dict
    protected: dict
    media: dict
    details: list = field(default_factory=list)
    def public(self):
        """The plan fields a response may carry (no fingerprint, no targets)."""
        return {
            "operation": self.operation,
            "scope": dict(self.scope),
            "protected": {
                "count": self.protected["count"],
                "reasons": list(self.protected["reasons"]),
            },
            "media": dict(self.media),
            "details": [dict(row) for row in self.details],
        }
 def _plural(count, one, many=None):
    return f"{count:,} {one if count == 1 else (many or one + 's')}"
 def _excerpt(text):
    text = " ".join(str(text or "").split())
    if len(text) <= EXCERPT_CHARS:
        return text
    return text[: EXCERPT_CHARS - 1].rstrip() + "…"
 def _megabytes(size):
    return f"{size / 2**20:,.1f} MB"
 class _Fingerprint:
    """A streaming sha256 over JSON-encoded parts."""
    def __init__(self, operation):
        self._hash = hashlib.sha256()
        self.add(operation)
    def add(self, part):
        encoded = json.dumps(part, separators=(",", ":"), default=str)
        self._hash.update(encoded.encode("utf-8") + b"\n")
    def hexdigest(self):
        return "sha256:" + self._hash.hexdigest()
 def _batches(ids):
    ids = list(ids)
    for start in range(0, len(ids), ID_BATCH):
        yield ids[start : start + ID_BATCH]
 def _marks(batch):
    return ",".join("?" for _ in batch)
 def _add_prompt_state(fingerprint, conn, ids):
    """Fold every reviewed fact about the prompts ``ids`` into ``fingerprint``."""
    for batch in _batches(sorted(ids)):
        marks = _marks(batch)
        for sql in _PROMPT_STATE_SQL:
            params = batch * sql.count("{marks}")
            for row in conn.execute(sql.format(marks=marks), params):
                fingerprint.add(list(row))
 def _texts(conn, ids, limit=MAX_DETAILS):
    """``{id: text}`` of the first ``limit`` prompts, for detail rows."""
    ids = list(ids)[:limit]
    if not ids:
        return {}
    rows = conn.execute(
        f"SELECT id, text FROM prompts WHERE id IN ({_marks(ids)})", ids
    )
    return {row[0]: row[1] for row in rows}
 def _orphaned_ids(conn):
    return sorted(row[0] for row in conn.execute(ORPHANED_PROMPTS_SQL))
 def _delete_ids(conn, table, ids):
    """Delete ``table`` rows by id in batches; returns the rows removed.
    The workflow documents only those rows referenced go with them (a
    prompt's images go by cascade), in the same transaction.
    """
    if table not in ("prompts", "generated_images"):
        raise ValueError(f"Unsupported table {table!r}")
    removed = 0
    for batch in _batches(ids):
        documents = documents_of_rows(
            conn, "id" if table == "generated_images" else "prompt_id", batch
        )
        cursor = conn.execute(
            f"DELETE FROM {table} WHERE id IN ({_marks(batch)})",  # nosec B608
            batch,
        )
        removed += cursor.rowcount
        release_documents(conn, documents)
    return removed
 def _read_snapshot(db_path, read):
    """``read(conn)`` on one ``mode=ro`` snapshot; the connection always closes."""
    conn = sqlite3.connect(readonly_uri(db_path), uri=True)
    try:
        conn.row_factory = sqlite3.Row
        conn.execute("BEGIN")  # one read snapshot for the whole plan
        return read(conn)
    finally:
        conn.close()
 def _relative(image_path, root):
    try:
        return Path(os.path.relpath(image_path, root)).as_posix()
    except ValueError:  # another drive (Windows)
        return os.path.basename(image_path)
 def _source_name(root, index):
    return os.path.basename(str(root).rstrip("\\/")) or f"Source {index + 1}"
 class MaintenanceOperations:
    """Mixin for PromptDatabase: maintenance previews and their apply."""
    # ── shared with the legacy cleanup ───────────────────────────────
    def _duplicate_groups(self, conn):
        """``(groups, paired)``: duplicate id groups, oldest first.
        ``paired`` are the groups with a member in a saved pair, which the
        cleanup keeps whole.
        """
        groups, paired = [], []
        for duplicate in conn.execute(DUPLICATE_PROMPTS_SQL).fetchall():
            ids = duplicate["ids"].split(",")
            created_dates = duplicate["created_dates"].split(",")
            # Sort IDs by created_at date to keep the oldest
            id_date_pairs = list(zip(ids, created_dates))
            id_date_pairs.sort(key=lambda x: x[1])
            sorted_ids = [int(pair[0]) for pair in id_date_pairs]
            try:
                self._ensure_prompts_unpaired(conn, sorted_ids)
            except PairDependencyError:
                paired.append(sorted_ids)
                continue
            groups.append(sorted_ids)
        return groups, paired
    @staticmethod
    def _duplicate_conflict(conn, sorted_ids):
        """Why merging the group would lose something, or None.
        ``collision``: two members link the same file, so moving the images
        to the oldest prompt would hit ``UNIQUE(prompt_id, file_path)``.
        ``title``: a member that would be deleted has its own Library title.
        """
        marks = _marks(sorted_ids)
        collision = conn.execute(
            "SELECT 1 FROM generated_images WHERE prompt_id IN "  # nosec B608
            f"({marks}) AND file_path IS NOT NULL GROUP BY file_path "
            "HAVING COUNT(DISTINCT prompt_id) > 1 LIMIT 1",
            sorted_ids,
        ).fetchone()
        if collision:
            return "collision"
        duplicates = sorted_ids[1:]
        titled = conn.execute(
            "SELECT 1 FROM prompt_library_state WHERE prompt_id IN "  # nosec B608
            f"({_marks(duplicates)}) AND trim(coalesce(title, '')) != '' LIMIT 1",
            duplicates,
        ).fetchone()
        return "title" if titled else None
    def _merge_duplicate_group(self, conn, sorted_ids, strict=False):
        """Keep the oldest prompt of a group; returns (removed, images moved).
        ``strict`` (the reviewed apply): a merge that would drop an image
        link or a Library title raises MergeConflict, and every helper error
        propagates, so the transaction rolls back. The legacy cleanup keeps
        its old, lenient behaviour.
        """
        primary_id, duplicate_ids = sorted_ids[0], sorted_ids[1:]
        if strict:
            conflict = self._duplicate_conflict(conn, sorted_ids)
            if conflict:
                raise MergeConflict(conflict)
        self.logger.debug(
            f"Merging duplicates: keeping {primary_id}, removing {duplicate_ids}"
        )
        primary = conn.execute(
            "SELECT * FROM prompts WHERE id = ?", (primary_id,)
        ).fetchone()
        if not primary:
            return 0, 0
        merged_metadata = self._merge_duplicate_metadata(
            conn, primary_id, duplicate_ids, strict=strict
        )
        moved = self._transfer_images_to_primary(
            conn, primary_id, duplicate_ids, strict=strict
        )
        if merged_metadata:
            self._update_primary_with_merged_metadata(
                conn, primary_id, merged_metadata, strict=strict
            )
        removed = 0
        # Images the lenient merge could not move go with their prompt.
        documents = documents_of_rows(conn, "prompt_id", duplicate_ids)
        for duplicate_id in duplicate_ids:
            conn.execute("DELETE FROM prompts WHERE id = ?", (duplicate_id,))
            removed += 1
            self.logger.debug(f"Removed duplicate prompt {duplicate_id}")
        release_documents(conn, documents)
        return removed, moved
    @staticmethod
    def _missing_link_ids(rows, roots):
        """Ids of the ``rows`` whose file is gone under a readable source."""
        readable = readable_roots(roots)
        return [
            row[0]
            for row, _owner, state in link_states(rows, roots, readable)
            if state == "missing"
        ]
    # ── previews ──────────────────────────────────────────────────────
    def preview_maintenance(self, operation, roots=()) -> MaintenanceDraft:
        """The reviewed plan of ``operation``; writes nothing (blocking).
        ``roots`` are the sources missing links are judged under (the health
        check's: the effective sources plus configured ones that are gone).
        """
        if operation not in MAINTENANCE_OPERATIONS:
            raise ValueError(f"Unknown maintenance operation {operation!r}")
        roots = [str(root) for root in roots]
        if operation == "optimize":
            return self._plan_optimize()
        if operation == "workflow_storage":
            return self._plan_workflow_storage()
        if operation == "missing_links":
            rows = _read_snapshot(self.model.db_path, self._link_rows)
            return self._plan_missing_links(rows, roots)
        reader = (
            self._plan_duplicate_prompts
            if operation == "duplicate_prompts"
            else self._plan_unused_components
        )
        return _read_snapshot(self.model.db_path, reader)
    @staticmethod
    def _link_rows(conn):
        rows = []
        cursor = conn.execute(_LINK_ROWS_SQL)
        while batch := cursor.fetchmany(BATCH_ROWS):
            rows.extend(tuple(row) for row in batch)
        return rows
    def _duplicates_fingerprint(self, conn):
        """``(groups, paired, conflicted, fingerprint)`` of the duplicate merge.
        ``conflicted`` holds ``(ids, reason)`` of the groups a merge would
        lose an image link or a Library title in (``_duplicate_conflict``);
        the plan protects them.
        """
        candidates, paired = self._duplicate_groups(conn)
        groups, conflicted = [], []
        for ids in candidates:
            reason = self._duplicate_conflict(conn, ids)
            if reason:
                conflicted.append((ids, reason))
            else:
                groups.append(ids)
        fingerprint = _Fingerprint("duplicate_prompts")
        fingerprint.add({"groups": groups, "paired": paired, "conflicted": conflicted})
        _add_prompt_state(fingerprint, conn, [i for group in groups for i in group])
        return groups, paired, conflicted, fingerprint.hexdigest()
    def _plan_duplicate_prompts(self, conn):
        groups, paired, conflicted, fingerprint = self._duplicates_fingerprint(conn)
        removed = sum(len(group) - 1 for group in groups)
        held = [ids for ids, _reason in conflicted]
        texts = _texts(conn, [group[0] for group in (groups + paired + held)])
        details = (
            [
                {
                    "label": _excerpt(texts.get(group[0])),
                    "detail": f"Keeps the oldest of {len(group)} copies",
                }
                for group in groups
            ]
            + [
                {
                    "label": _excerpt(texts.get(group[0])),
                    "detail": "Kept: belongs to a saved pair",
                }
                for group in paired
            ]
            + [
                {
                    "label": _excerpt(texts.get(ids[0])),
                    "detail": CONFLICT_DETAILS[reason],
                }
                for ids, reason in conflicted
            ]
        )
        if groups:
            label = (
                f"{_plural(removed, 'duplicate prompt')} in "
                f"{_plural(len(groups), 'group')}"
            )
        else:
            label = "No duplicate prompt text"
        reasons = []
        if len(paired) == 1:
            reasons.append("1 group belongs to a saved pair")
        elif paired:
            reasons.append(f"{len(paired):,} groups belong to saved pairs")
        for reason, (one, many) in CONFLICT_REASONS.items():
            count = sum(1 for _ids, why in conflicted if why == reason)
            if count:
                reasons.append(f"{count:,} {one if count == 1 else many}")
        return MaintenanceDraft(
            operation="duplicate_prompts",
            fingerprint=fingerprint,
            targets=groups,
            scope={"count": removed, "label": label},
            protected={
                "count": sum(len(group) for group in paired + held),
                "reasons": reasons,
            },
            media={"deletes_files": False, "label": NO_FILES_DELETED},
            details=details[:MAX_DETAILS],
        )
    def _unused_fingerprint(self, conn):
        ids = _orphaned_ids(conn)
        fingerprint = _Fingerprint("unused_components")
        fingerprint.add(ids)
        _add_prompt_state(fingerprint, conn, ids)
        return ids, fingerprint.hexdigest()
    def _plan_unused_components(self, conn):
        ids, fingerprint = self._unused_fingerprint(conn)
        *by_reason, kept = conn.execute(_UNLINKED_PROTECTIONS_SQL).fetchone()
        reasons = [
            f"{count:,} {many if count != 1 else one}"
            for count, (_predicate, many, one) in zip(by_reason, PROTECTIONS)
            if count
        ]
        texts = _texts(conn, ids)
        count = len(ids)
        return MaintenanceDraft(
            operation="unused_components",
            fingerprint=fingerprint,
            targets=ids,
            scope={
                "count": count,
                "label": (
                    _plural(count, "unused prompt component")
                    if count
                    else "No unused prompt components"
                ),
            },
            protected={"count": kept, "reasons": reasons},
            media={"deletes_files": False, "label": NO_FILES_DELETED},
            details=[
                {"label": _excerpt(texts.get(prompt_id)), "detail": UNUSED_DETAIL}
                for prompt_id in ids[:MAX_DETAILS]
            ],
        )
    @staticmethod
    def _links_fingerprint(candidates):
        fingerprint = _Fingerprint("missing_links")
        for row, owner in candidates:
            fingerprint.add([row[0], row[1], row[3], owner])
        return fingerprint.hexdigest()
    def _plan_missing_links(self, rows, roots):
        readable = readable_roots(roots)
        candidates, unavailable, outside = [], 0, 0
        for row, owner, state in link_states(rows, roots, readable):
            if state == "missing":
                candidates.append((row, owner))
            elif state == "unavailable":
                unavailable += 1
            elif state == "outside":
                outside += 1
        reasons = []
        if unavailable:
            reasons.append(
                _plural(
                    unavailable,
                    "link under an unavailable source",
                    "links under unavailable sources",
                )
            )
        if outside:
            reasons.append(
                _plural(
                    outside,
                    "link outside the image sources",
                    "links outside the image sources",
                )
            )
        count = len(candidates)
        return MaintenanceDraft(
            operation="missing_links",
            fingerprint=self._links_fingerprint(candidates),
            targets=[row[0] for row, _owner in candidates],
            scope={
                "count": count,
                "label": (
                    _plural(count, "link to a missing file", "links to missing files")
                    if count
                    else "No links to missing files"
                ),
            },
            protected={"count": unavailable + outside, "reasons": reasons},
            media={"deletes_files": False, "label": LINKS_ONLY},
            details=[
                {
                    "label": _relative(row[1], roots[owner]),
                    "detail": (
                        f"{_source_name(roots[owner], owner)} · the file is missing"
                    ),
                }
                for row, owner in candidates[:MAX_DETAILS]
            ],
        )
    def _plan_optimize(self):
        def read(conn):
            return [
                conn.execute(f"PRAGMA {name}").fetchone()[0]
                for name in ("page_size", "page_count", "freelist_count")
            ]
        page_size, page_count, free_pages = _read_snapshot(self.model.db_path, read)
        db_path = self.model.db_path
        wal = db_path + "-wal"
        size = page_size * page_count
        wal_size = os.path.getsize(wal) if os.path.exists(wal) else 0
        free = shutil.disk_usage(os.path.dirname(os.path.abspath(db_path))).free
        # VACUUM writes the rebuilt database through the WAL beside it.
        return MaintenanceDraft(
            operation="optimize",
            fingerprint=_Fingerprint("optimize").hexdigest(),
            targets=[],
            scope={"count": 1, "label": "The whole database"},
            protected={"count": 0, "reasons": []},
            media={"deletes_files": False, "label": NO_FILES_DELETED},
            details=[
                {
                    "label": "Database file",
                    "detail": f"{_megabytes(size)}, plus {_megabytes(wal_size)} "
                    "not yet checkpointed",
                },
                {
                    "label": "Free pages",
                    "detail": f"About {_megabytes(free_pages * page_size)} "
                    "can be reclaimed",
                },
                {
                    "label": "Lock",
                    "detail": OPTIMIZE_LOCK_NOTE,
                },
                {
                    "label": "Free space needed",
                    "detail": f"About {_megabytes(size)} beside the database and "
                    f"{_megabytes(size)} in SQLite's temporary folder; "
                    f"{_megabytes(free)} available beside the database",
                },
            ],
        )
    # ── apply ─────────────────────────────────────────────────────────
    def apply_maintenance(self, operation, fingerprint, targets, roots=()) -> dict:
        """Apply a reviewed plan exactly, or raise PlanChanged (blocking).
        Inside ``BEGIN IMMEDIATE`` the fingerprint is computed again on the
        write connection; when it differs nothing is written. ``targets`` and
        ``roots`` are the plan's. Returns ``{changed, rows}``: how many
        records changed and up to ``MAX_RESULT_ROWS`` descriptions of them
        (``{prompt_id[, kept_id]}`` or ``{image_path}``).
        """
        if operation not in MAINTENANCE_OPERATIONS or operation == "workflow_storage":
            raise ValueError(f"Unknown maintenance operation {operation!r}")
        if operation == "optimize":
            # VACUUM cannot run in a transaction; the plan binds the database.
            if fingerprint != _Fingerprint("optimize").hexdigest():
                raise PlanChanged("optimize")
            self.model.vacuum_database()
            return {"changed": 1, "rows": []}
        appliers = {
            "duplicate_prompts": self._apply_duplicate_prompts,
            "unused_components": self._apply_unused_components,
            "missing_links": self._apply_missing_links,
        }
        with self._pair_transaction() as conn:
            return appliers[operation](conn, fingerprint, targets, roots)
    def _apply_duplicate_prompts(self, conn, fingerprint, targets, _roots):
        groups, _paired, _conflicted, current = self._duplicates_fingerprint(conn)
        if current != fingerprint or groups != targets:
            raise PlanChanged("duplicate_prompts")
        changed, rows = 0, []
        for sorted_ids in groups:
            removed, _moved = self._merge_duplicate_group(conn, sorted_ids, strict=True)
            changed += removed
            rows.extend(
                {"prompt_id": duplicate_id, "kept_id": sorted_ids[0]}
                for duplicate_id in sorted_ids[1:]
            )
        return {"changed": changed, "rows": rows[:MAX_RESULT_ROWS]}
    def _apply_unused_components(self, conn, fingerprint, targets, _roots):
        ids, current = self._unused_fingerprint(conn)
        if current != fingerprint or ids != targets:
            raise PlanChanged("unused_components")
        texts = _texts(conn, ids, limit=MAX_RESULT_ROWS)
        changed = _delete_ids(conn, "prompts", ids)
        rows = [
            {"prompt_id": prompt_id, "text": _excerpt(texts.get(prompt_id))}
            for prompt_id in ids[:MAX_RESULT_ROWS]
        ]
        return {"changed": changed, "rows": rows}
    def _apply_missing_links(self, conn, fingerprint, targets, roots):
        rows = []
        for batch in _batches(sorted(targets)):
            rows.extend(
                tuple(row)
                for row in conn.execute(
                    "SELECT id, image_path, filename, file_path FROM generated_images "
                    f"WHERE id IN ({_marks(batch)}) ORDER BY id",  # nosec B608
                    batch,
                )
            )
        readable = readable_roots(roots)
        candidates = []
        for row, owner, state in link_states(rows, roots, readable):
            if state != "missing":  # back, or its source went away
                raise PlanChanged("missing_links")
            candidates.append((row, owner))
        if self._links_fingerprint(candidates) != fingerprint:
            raise PlanChanged("missing_links")
        changed = _delete_ids(conn, "generated_images", [row[0] for row in rows])
        return {
            "changed": changed,
            "rows": [{"image_path": row[1]} for row in rows[:MAX_RESULT_ROWS]],
        }
@@ -0,0 +1,353 @@
 """A SQLite connection that is never closed under the thread using it.
 ``PromptModel`` gives each thread its own connection, but Move library and
 ``close_all`` must retire every thread's connection from one thread. Closing
 a ``sqlite3.Connection`` while its owner is inside a statement (Python runs
 ``sqlite3_step`` without the GIL) frees memory the owner then reads: on
 Windows the process dies with "access violation", elsewhere the owner sees
 garbage such as a ``MemoryError``.
 ``ManagedConnection`` counts the calls its owner is inside. ``retire()`` is
 safe from any thread and never raises: an idle connection is closed at once;
 one in use is closed by its owner as the call ends. A retired connection
 refuses new work (cursors, statements, commits) with the error a closed
 connection raises.
 What retiring a connection in use does and does not do:
 - It calls ``sqlite3_interrupt`` (thread-safe). That ends a running
  statement at its next step, but it does not wake a statement that is
  waiting for another connection's lock: that one waits until the lock is
  released (or the busy timeout ends it), and only then is it interrupted.
  An interrupt that arrives between two statements, for example between the
  implicit ``BEGIN`` and the DML or between ``executemany`` rows, is lost.
 - Writes are kept out of a file the process has switched away from because
  a retired connection never commits: legacy-mode DML runs inside a
  transaction that only ``commit()`` (refused) or ``close()`` (rolls back)
  ends. A statement that commits by itself is not covered: DDL or a PRAGMA
  write outside a transaction, or an explicit ``COMMIT`` passed to
  ``execute``/``executescript`` (``executescript`` also commits first, in C,
  on Python before 3.12). Managed connections must never be used that way
  for library writes.
 """
 from __future__ import annotations
 import contextlib
 import logging
 import threading
 import time
 import weakref
 from typing import Iterable, Iterator, List, Optional
 import sqlite3
 CLOSED_MESSAGE = "Cannot operate on a closed database."
 # Rows fetched per step when a cursor is iterated: the connection is marked
 # in use once per chunk instead of once per row (iteration costs about what
 # a plain cursor costs). A loop that stops early leaves up to this many rows
 # fetched; ``fetchone``/``fetchmany`` after a partial loop skip them.
 ITER_CHUNK = 128
 class ManagedCursor(sqlite3.Cursor):
    """A cursor whose calls into SQLite count as its connection being in use."""
    def execute(self, sql, parameters=(), /):
        conn = self.connection
        conn._enter()
        try:
            return super().execute(sql, parameters)
        finally:
            conn._leave()
    def executemany(self, sql, seq_of_parameters, /):
        conn = self.connection
        conn._enter()
        try:
            return super().executemany(sql, seq_of_parameters)
        finally:
            conn._leave()
    def executescript(self, sql_script, /):
        conn = self.connection
        conn._enter()
        try:
            return super().executescript(sql_script)
        finally:
            conn._leave()
    def fetchone(self):
        conn = self.connection
        conn._enter()
        try:
            return super().fetchone()
        finally:
            conn._leave()
    def fetchmany(self, *args, **kwargs):
        conn = self.connection
        conn._enter()
        try:
            return super().fetchmany(*args, **kwargs)
        finally:
            conn._leave()
    def fetchall(self):
        conn = self.connection
        conn._enter()
        try:
            return super().fetchall()
        finally:
            conn._leave()
    def __iter__(self) -> Iterator:
        return self._chunks()
    def _chunks(self) -> Iterator:
        while True:
            rows = self.fetchmany(ITER_CHUNK)
            if not rows:
                return
            yield from rows
    def __next__(self):
        conn = self.connection
        conn._enter()
        try:
            return super().__next__()
        finally:
            conn._leave()
    def close(self):
        conn = self.connection
        try:
            conn._enter()
        except sqlite3.ProgrammingError:
            return super().close()  # the connection is closed or closing
        try:
            return super().close()
        finally:
            conn._leave()
 class ManagedConnection(sqlite3.Connection):
    """A connection only its owner closes while a call into it is running."""
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # Guards _depth, _retired, _closing and _cursors. Re-entrant so that
        # code running in the closing thread while it holds the lock (a
        # cursor() during the cursor-set copy) is refused instead of hanging.
        self._use_lock = threading.RLock()
        self._depth = 0
        self._retired = False
        self._closing = False
        self._released = threading.Event()
        # Python before 3.12 keeps the file open after close() while a cursor
        # still holds a statement; these are closed first (see _close_now).
        self._cursors: "weakref.WeakSet[sqlite3.Cursor]" = weakref.WeakSet()
        self.close_logger: Optional[logging.Logger] = None
    # ── in use ───────────────────────────────────────────────────────
    def _enter(self) -> None:
        with self._use_lock:
            if self._retired:
                raise sqlite3.ProgrammingError(CLOSED_MESSAGE)
            self._depth += 1
    def _leave(self) -> None:
        with self._use_lock:
            self._depth -= 1
            close_now = self._retired and self._depth == 0
        if close_now:
            self._close_now()
    @contextlib.contextmanager
    def using(self):
        """Mark the connection in use for a call this class cannot wrap.
        For example as the *target* of another connection's ``backup()``.
        """
        self._enter()
        try:
            yield self
        finally:
            self._leave()
    def cursor(self, factory=None):
        with self._use_lock:
            if self._retired:
                raise sqlite3.ProgrammingError(CLOSED_MESSAGE)
            cursor = super().cursor(factory or ManagedCursor)
            self._cursors.add(cursor)
        return cursor
    def execute(self, sql, parameters=(), /):
        return self.cursor().execute(sql, parameters)
    def executemany(self, sql, seq_of_parameters, /):
        return self.cursor().executemany(sql, seq_of_parameters)
    def executescript(self, sql_script, /):
        return self.cursor().executescript(sql_script)
    def commit(self):
        self._enter()
        try:
            return super().commit()
        finally:
            self._leave()
    def rollback(self):
        self._enter()  # a closed connection raises, as sqlite3's does
        try:
            return super().rollback()
        finally:
            self._leave()
    def backup(self, target, **kwargs):
        self._enter()
        try:
            return super().backup(target, **kwargs)
        finally:
            self._leave()
    def create_function(self, *args, **kwargs):
        self._enter()
        try:
            return super().create_function(*args, **kwargs)
        finally:
            self._leave()
    @property
    def in_transaction(self) -> bool:
        self._enter()
        try:
            return super().in_transaction
        finally:
            self._leave()
    def __exit__(self, exc_type, exc_value, traceback):
        # The C implementation calls its own commit/rollback, not these.
        if exc_type is None:
            try:
                self.commit()
            except BaseException:
                with contextlib.suppress(sqlite3.Error):
                    self.rollback()
                raise
        else:
            self.rollback()
        return False
    # ── closing ──────────────────────────────────────────────────────
    def close(self) -> None:
        """Close now if idle, else as soon as the running call ends.
        Unlike ``sqlite3.Connection.close``, a call from another thread while
        the owner is inside a call returns before the file is closed. A caller
        that must know the file is closed (before renaming or deleting it)
        waits for ``wait_closed`` (``PromptModel.close_detached`` does).
        """
        self.retire()
    def retire(self) -> bool:
        """Refuse new work and close; True when the connection is closed already.
        Safe from any thread, and never raises: failures are logged.
        """
        try:
            with self._use_lock:
                if not self._retired:
                    self._retired = True
                    if self._depth:
                        # Ends the running statement (see the module docstring
                        # for what it does not end). The owner cannot start
                        # closing the connection while this lock is held.
                        self.interrupt()
                        return False
                elif self._depth:
                    return False
                else:
                    return self._released.wait(0)
        except Exception:
            self._log_exception("Error retiring database connection")
            return self._released.wait(0)
        self._close_now()
        return True
    def wait_closed(self, timeout: float) -> bool:
        return self._released.wait(timeout)
    def _close_now(self) -> None:
        # Only ever called with no call into the connection running, and a
        # retired connection refuses new calls and cursors: its cursors are
        # idle, and the set is copied under the lock that guards it.
        with self._use_lock:
            if self._closing:
                return  # another thread closes it and sets _released
            self._closing = True
            cursors = list(self._cursors)
        try:
            for cursor in cursors:
                with contextlib.suppress(sqlite3.Error):
                    sqlite3.Cursor.close(cursor)
            super().close()
        except Exception:
            self._log_exception("Error closing database connection")
        finally:
            self._released.set()
    def _log_exception(self, message: str) -> None:
        if self.close_logger is not None:
            self.close_logger.exception(message)
 def retire_connections(
    connections: Iterable[sqlite3.Connection], logger=None
 ) -> List[ManagedConnection]:
    """Retire every connection; return those whose owner has yet to close them.
    Safe while holding a lock the owners wait on: nothing here waits for them.
    One connection failing to retire is logged and never skips the rest.
    """
    pending = []
    for conn in connections:
        try:
            if isinstance(conn, ManagedConnection):
                if logger is not None:
                    conn.close_logger = logger
                if not conn.retire():
                    pending.append(conn)
            else:
                conn.close()
        except Exception:
            if logger is not None:
                logger.exception("Error closing database connection")
    return pending
 def await_closed(
    connections: Iterable[ManagedConnection], timeout: float, logger=None
 ) -> bool:
    """Wait until the owners closed ``connections`` (retired); False on timeout.
    A connection still open after ``timeout`` (its owner is blocked, for
    example on another process's lock after a lost interrupt) is logged and
    left to close when its call ends; a Move library then reports the old
    files as still in use instead of renaming them.
    """
    deadline = time.monotonic() + timeout
    still_open = 0
    for conn in connections:
        if not conn.wait_closed(max(0.0, deadline - time.monotonic())):
            still_open += 1
    if still_open and logger is not None:
        logger.warning(
            f"{still_open} database connection(s) still in use after "
            f"{timeout:g}s; they close when their statement ends"
        )
    return not still_open
@@ -0,0 +1,212 @@
 """Atomic metadata enrichment for a single known generation context."""
 import os
 from typing import Any, Dict
 from .models import normalize_image_path
 from .operations import _coerce_file_info
 from .workflow_documents import (
    TEXT_COLUMNS,
    DocumentReader,
    release_documents,
    resolve_image_row,
    stored_values,
 )
 try:
    from ..utils.hashing import generate_prompt_hash
    from ..utils.json_strict import dumps_strict
    from ..utils.prompt_pair_resolver import RESOLVER_VERSION
 except ImportError:
    from utils.hashing import generate_prompt_hash
    from utils.json_strict import dumps_strict
    from utils.prompt_pair_resolver import RESOLVER_VERSION
 def blank_positive(resolution: Dict[str, Any]) -> bool:
    """A resolved pair whose stored positive text is empty (#342).
    A PromptManager node whose editor is empty encodes only its prepend and
    append text, which is never stored as a prompt: such a result names no
    prompt to link.
    """
    if resolution.get("state") != "resolved":
        return False
    positive = resolution.get("positive") or {}
    text = positive.get("storage_text", positive.get("text"))
    return not (isinstance(text, str) and text.strip())
 def ingest_resolved_pair(
    db: Any, image_path: str, metadata: Dict[str, Any], resolution: Dict[str, Any]
 ) -> Dict[str, Any]:
    state = resolution.get("state", "unresolved")
    if state == "resolved" and blank_positive(resolution):
        state = "unresolved"
    if state != "resolved":
        return {"status": state, "pairs_added": 0, "links_added": 0}
    with db._pair_transaction() as conn:
        return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution)
 def _conflict() -> Dict[str, Any]:
    return {"status": "conflict", "pairs_added": 0, "links_added": 0}
 def _ingest_resolved_pair_in(
    conn: Any,
    db: Any,
    image_path: str,
    metadata: Dict[str, Any],
    resolution: Dict[str, Any],
    *,
    provenance: Dict[str, Any] | None = None,
 ) -> Dict[str, Any]:
    """``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``.
    The compatible-only rules are the public ingester's: an image with more
    than one exact row, another positive, or a prior pair with another
    negative is a ``conflict`` and nothing is written. The caller owns the
    transaction, so its own later failure rolls the link back too.
    ``provenance`` replaces the link's default resolver provenance (Needs
    review records the user's selection there).
    """
    positive, negative = resolution["positive"], resolution["negative"]
    positive_text = positive.get("storage_text", positive["text"])
    negative_text = negative.get("storage_text", negative["text"])
    normalized = normalize_image_path(image_path)
    if provenance is None:
        provenance = {
            "resolver_version": RESOLVER_VERSION,
            "positive": positive,
            "negative": negative,
        }
    encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":"))
    images = conn.execute(
        "SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
        "JOIN prompts p ON p.id=gi.prompt_id "
        "WHERE gi.file_path=? OR gi.image_path=?",
        (normalized, image_path),
    ).fetchall()
    if len(images) > 1 or (
        images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
    ):
        return _conflict()
    existing = images[0] if images else None
    if existing:
        prior = conn.execute(
            "SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
            "JOIN prompt_pairs pp ON pp.id=ip.pair_id "
            "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
            (existing["id"],),
        ).fetchall()
        expected = (
            generate_prompt_hash(negative_text) if negative_text.strip() else None
        )
        if any(row["negative_hash"] != expected for row in prior):
            return _conflict()
    prior_pair = conn.execute(
        "SELECT pp.id FROM prompt_pairs pp "
        "JOIN prompts p ON p.id=pp.positive_prompt_id "
        "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
        "WHERE p.hash=? AND n.hash IS ?",
        (
            generate_prompt_hash(positive_text),
            generate_prompt_hash(negative_text) if negative_text.strip() else None,
        ),
    ).fetchone()
    pair_id = db._save_pair(conn, positive_text, negative_text)
    pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone()
    pairs_added = int(prior_pair is None)
    file_size, width, height, image_format = _coerce_file_info(metadata)
    # ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
    snapshots = {
        column: dumps_strict(metadata[key], sort_keys=True)
        for column, key in (
            ("workflow_data", "workflow"),
            ("prompt_metadata", "prompt"),
            ("parameters", "parameters"),
        )
        if key in metadata
    }
    info = {
        "file_size": file_size,
        "width": width,
        "height": height,
        "format": image_format,
    }
    snapshots.update({key: value for key, value in info.items() if value is not None})
    changed = False
    if existing:
        image_id = existing["id"]
        # Compared as stored text, whichever form the row is in.
        current = resolve_image_row(DocumentReader(conn), existing)
        updates = {
            key: value for key, value in snapshots.items() if current[key] != value
        }
        if updates:
            replaced = [
                existing[reference]
                for column, reference in TEXT_COLUMNS
                if column in updates
            ]
            updates = _document_form(conn, updates)
            conn.execute(
                "UPDATE generated_images SET "
                + ",".join(f"{key}=?" for key in updates)
                + " WHERE id=?",
                [*updates.values(), image_id],
            )
            release_documents(conn, replaced)
            changed = True
    else:
        stored = _document_form(conn, snapshots)
        columns = ["prompt_id", "image_path", "filename", "file_path", *stored]
        values = [
            pair["positive_prompt_id"],
            image_path,
            os.path.basename(image_path),
            normalized,
            *stored.values(),
        ]
        image_id = conn.execute(
            "INSERT INTO generated_images("
            + ",".join(columns)
            + ") VALUES("
            + ",".join("?" for _ in columns)
            + ")",
            values,
        ).lastrowid
    linked = conn.execute(
        "SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
        "AND image_id=? AND provenance=?",
        (pair_id, image_id, encoded),
    ).fetchone()
    db._link_image_to_pair(conn, pair_id, image_id, provenance)
    status = (
        "added"
        if existing is None
        else "enriched" if changed or not linked else "unchanged"
    )
    return {
        "status": status,
        "pairs_added": pairs_added,
        "links_added": int(not linked),
        "pair_id": pair_id,
        "image_id": image_id,
    }
 def _document_form(conn: Any, values: Dict[str, Any]) -> Dict[str, Any]:
    """``values`` with the workflow and prompt graph texts stored once.
    Each of those columns becomes NULL beside its document id (new form);
    every other column is unchanged.
    """
    texts = {
        column: values[column]
        for column, _reference in TEXT_COLUMNS
        if column in values
    }
    rest = {key: value for key, value in values.items() if key not in texts}
    return {**rest, **stored_values(conn, texts)}
@@ -0,0 +1,433 @@
 """Ordered reusable prompt pairs and their generation provenance."""
 import json
 import sqlite3
 from contextlib import contextmanager
 from typing import Any, Dict, Iterator, List, Optional
 from .models import pair_provenance_error, utc_now_iso
 try:
    from ..utils.hashing import generate_prompt_hash
 except ImportError:
    from utils.hashing import generate_prompt_hash
 class PairDependencyError(ValueError):
    """An operation would change or remove content used by a saved pair."""
 def _validate_metadata(
    title: Optional[str],
    category: Optional[str],
    tags: Optional[List[str]],
    rating: Optional[int],
    notes: Optional[str],
 ) -> Optional[List[str]]:
    for name, value in (("title", title), ("category", category), ("notes", notes)):
        if value is not None and not isinstance(value, str):
            raise ValueError(f"{name} must be a string")
    if rating is not None and (type(rating) is not int or not 1 <= rating <= 5):
        raise ValueError("Rating must be between 1 and 5")
    if tags is None:
        return None
    if not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags):
        raise ValueError("tags must be a list of strings")
    return list(dict.fromkeys(tag.strip() for tag in tags if tag.strip()))
 class PairOperations:
    """Connection-level pair operations mixed into PromptDatabase."""
    @contextmanager
    def _pair_transaction(self) -> Iterator[sqlite3.Connection]:
        conn = self.model.get_connection()
        if conn.in_transaction:
            raise RuntimeError("Pair operations require their own transaction")
        with conn:
            conn.execute("BEGIN IMMEDIATE")
            yield conn
    def _ensure_prompts_unpaired(
        self, conn: sqlite3.Connection, prompt_ids: List[int]
    ) -> None:
        if not prompt_ids:
            return
        placeholders = ",".join("?" for _ in prompt_ids)
        row = conn.execute(
            f"SELECT id FROM prompt_pairs WHERE positive_prompt_id IN ({placeholders}) "
            f"OR negative_prompt_id IN ({placeholders}) LIMIT 1",
            [*prompt_ids, *prompt_ids],
        ).fetchone()
        if row:
            raise PairDependencyError(
                f"Prompt belongs to saved pair {row['id']}; delete the pair first "
                "or save changed text as a new prompt"
            )
    def _pair_component(self, conn: sqlite3.Connection, text: str) -> int:
        return self._pair_component_outcome(conn, text)[0]
    def _pair_component_outcome(
        self, conn: sqlite3.Connection, text: str
    ) -> tuple[int, bool]:
        """Resolve a component by its dedup hash; report whether it was inserted.
        The first stored spelling wins: the hash ignores outer whitespace and
        case, internal whitespace stays significant. ``created`` comes from the
        INSERT result under the caller's held write transaction.
        """
        prompt_hash = generate_prompt_hash(text)
        now = utc_now_iso()
        created = (
            conn.execute(
                "INSERT INTO prompts(text, hash, created_at, updated_at) "
                "VALUES(?,?,?,?) ON CONFLICT(hash) DO NOTHING",
                (text.strip(), prompt_hash, now, now),
            ).rowcount
            == 1
        )
        row = conn.execute(
            "SELECT id FROM prompts WHERE hash = ?", (prompt_hash,)
        ).fetchone()
        return row["id"], created
    def _sync_pair_tags(
        self, conn: sqlite3.Connection, pair_id: int, tags: List[str]
    ) -> None:
        conn.execute("DELETE FROM prompt_pair_tags WHERE pair_id = ?", (pair_id,))
        for tag_id in self._ensure_tags(conn, tags).values():
            conn.execute(
                "INSERT INTO prompt_pair_tags(pair_id,tag_id) VALUES(?,?)",
                (pair_id, tag_id),
            )
    def save_prompt_pair(
        self,
        positive_text: str,
        negative_text: str = "",
        *,
        title: Optional[str] = None,
        category: Optional[str] = None,
        tags: Optional[List[str]] = None,
        rating: Optional[int] = None,
        notes: Optional[str] = None,
    ) -> int:
        """Save an ordered pair atomically; repeat saves preserve its metadata."""
        with self._pair_transaction() as conn:
            return self._save_pair(
                conn,
                positive_text,
                negative_text,
                title=title,
                category=category,
                tags=tags,
                rating=rating,
                notes=notes,
            )
    def _save_pair(
        self,
        conn: sqlite3.Connection,
        positive_text: str,
        negative_text: str = "",
        *,
        title: Optional[str] = None,
        category: Optional[str] = None,
        tags: Optional[List[str]] = None,
        rating: Optional[int] = None,
        notes: Optional[str] = None,
    ) -> int:
        """Save an ordered pair atomically; repeat saves preserve its metadata."""
        if not isinstance(positive_text, str) or not positive_text.strip():
            raise ValueError("Positive prompt text cannot be empty")
        if not isinstance(negative_text, str):
            raise ValueError("Negative prompt text must be a string; use '' for empty")
        # Validate metadata before any component insertion.
        _validate_metadata(title, category, tags, rating, notes)
        positive = self._pair_component(conn, positive_text)
        negative = (
            self._pair_component(conn, negative_text) if negative_text.strip() else None
        )
        return self._save_pair_components(
            conn,
            positive,
            negative,
            title=title,
            category=category,
            tags=tags,
            rating=rating,
            notes=notes,
        )[0]
    def _save_pair_components(
        self,
        conn: sqlite3.Connection,
        positive_id: int,
        negative_id: Optional[int],
        *,
        title: Optional[str] = None,
        category: Optional[str] = None,
        tags: Optional[List[str]] = None,
        rating: Optional[int] = None,
        notes: Optional[str] = None,
    ) -> tuple[int, bool]:
        """Create or reuse the ordered pair of existing component IDs.
        Returns ``(pair_id, created)``. A reused pair keeps all of its metadata
        and archive state; metadata applies only to a newly inserted pair.
        """
        tags = _validate_metadata(title, category, tags, rating, notes)
        existing = conn.execute(
            "SELECT id FROM prompt_pairs WHERE positive_prompt_id = ? "
            "AND negative_prompt_id IS ?",
            (positive_id, negative_id),
        ).fetchone()
        if existing:
            return existing["id"], False
        now = utc_now_iso()
        pair_id = conn.execute(
            "INSERT INTO prompt_pairs(positive_prompt_id,negative_prompt_id,title,"
            "category,rating,notes,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?)",
            (positive_id, negative_id, title, category, rating, notes, now, now),
        ).lastrowid
        if tags:
            self._sync_pair_tags(conn, pair_id, tags)
        return pair_id, True
    def _read_pair_rows(
        self, conn: sqlite3.Connection, where: str, params: List[Any], suffix: str = ""
    ) -> List[Dict[str, Any]]:
        rows = conn.execute(
            "SELECT pp.*, p.text AS positive_text, "
            "COALESCE(n.text,'') AS negative_text,"
            "(SELECT COUNT(DISTINCT image_id) FROM image_prompt_pairs ip "
            "WHERE ip.pair_id=pp.id) AS image_count FROM prompt_pairs pp "
            "JOIN prompts p ON p.id=pp.positive_prompt_id "
            "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE " + where + suffix,
            params,
        ).fetchall()
        pairs = [dict(row) for row in rows]
        if not pairs:
            return pairs
        by_id = {pair["id"]: pair for pair in pairs}
        for pair in pairs:
            pair["archived"] = bool(pair["archived"])
            pair["tags"] = []
        placeholders = ",".join("?" for _ in by_id)
        for row in conn.execute(
            "SELECT pt.pair_id,t.name FROM prompt_pair_tags pt JOIN tags t ON "
            f"t.id=pt.tag_id WHERE pt.pair_id IN ({placeholders}) ORDER BY t.name",
            list(by_id),
        ):
            by_id[row["pair_id"]]["tags"].append(row["name"])
        return pairs
    def get_prompt_pair(self, pair_id: int) -> Optional[Dict[str, Any]]:
        """Return components, pair metadata and image count, including archives."""
        rows = self._read_pair_rows(self.model.get_connection(), "pp.id = ?", [pair_id])
        return rows[0] if rows else None
    def list_prompt_pairs(
        self,
        *,
        text: Optional[str] = None,
        category: Optional[str] = None,
        tags: Optional[List[str]] = None,
        include_archived: bool = False,
        limit: int = 50,
        offset: int = 0,
        sort: str = "newest",
    ) -> Dict[str, Any]:
        """List pairs with all-tag filtering and stable paginated ordering."""
        orders = {
            "newest": "pp.id DESC",
            "last_used_desc": "julianday(pp.last_used_at) DESC, pp.id DESC",
        }
        if sort not in orders:
            raise ValueError("Unknown pair sort order")
        if type(limit) is not int or not 1 <= limit <= 500:
            raise ValueError("limit must be between 1 and 500")
        if type(offset) is not int or offset < 0:
            raise ValueError("offset must be a nonnegative integer")
        tags = _validate_metadata(None, category, tags, None, None)
        if text is not None and not isinstance(text, str):
            raise ValueError("text must be a string")
        clauses, params = ["1=1"], []
        if not include_archived:
            clauses.append("pp.archived=0")
        if category is not None:
            clauses.append("pp.category = ?")
            params.append(category)
        if text:
            clauses.append("(p.text LIKE ? OR n.text LIKE ? OR pp.title LIKE ?)")
            params.extend([f"%{text}%"] * 3)
        for tag in tags or []:
            clauses.append(
                "EXISTS(SELECT 1 FROM prompt_pair_tags pt JOIN tags t "
                "ON t.id=pt.tag_id WHERE pt.pair_id=pp.id AND t.name=?)"
            )
            params.append(tag)
        where = " AND ".join(clauses)
        conn = self.model.get_connection()
        total = conn.execute(
            "SELECT COUNT(*) FROM prompt_pairs pp "
            "JOIN prompts p ON p.id=pp.positive_prompt_id "
            "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE " + where,
            params,
        ).fetchone()[0]
        pairs = self._read_pair_rows(
            conn,
            where,
            [*params, limit, offset],
            f" ORDER BY {orders[sort]} LIMIT ? OFFSET ?",
        )
        return {
            "pairs": pairs,
            "total": total,
            "limit": limit,
            "offset": offset,
            "has_more": offset + len(pairs) < total,
        }
    def update_prompt_pair_metadata(
        self,
        pair_id: int,
        *,
        title: Optional[str] = None,
        category: Optional[str] = None,
        tags: Optional[List[str]] = None,
        rating: Optional[int] = None,
        notes: Optional[str] = None,
    ) -> bool:
        """Update pair metadata explicitly; None preserves and empty values clear."""
        tags = _validate_metadata(title, category, tags, rating, notes)
        changes = {
            key: value
            for key, value in (
                ("title", title),
                ("category", category),
                ("rating", rating),
                ("notes", notes),
            )
            if value is not None
        }
        if not changes and tags is None:
            return False
        with self._pair_transaction() as conn:
            if not conn.execute(
                "SELECT 1 FROM prompt_pairs WHERE id=?", (pair_id,)
            ).fetchone():
                return False
            changes["updated_at"] = utc_now_iso()
            conn.execute(
                "UPDATE prompt_pairs SET "
                + ",".join(f"{key}=?" for key in changes)
                + " WHERE id=?",
                [*changes.values(), pair_id],
            )
            if tags is not None:
                self._sync_pair_tags(conn, pair_id, tags)
            return True
    def archive_prompt_pair(self, pair_id: int, archived: bool = True) -> bool:
        """Hide a reusable pair without changing components or generation history."""
        if type(archived) is not bool:
            raise ValueError("archived must be a boolean")
        with self._pair_transaction() as conn:
            return (
                conn.execute(
                    "UPDATE prompt_pairs SET archived=?,updated_at=? WHERE id=?",
                    (int(archived), utc_now_iso(), pair_id),
                ).rowcount
                > 0
            )
    def delete_prompt_pair(self, pair_id: int) -> bool:
        """Delete unused pair metadata; generated pairs must be archived instead."""
        with self._pair_transaction() as conn:
            if conn.execute(
                "SELECT 1 FROM image_prompt_pairs WHERE pair_id=? LIMIT 1", (pair_id,)
            ).fetchone():
                raise PairDependencyError(
                    "Pair has generated images; archive it instead"
                )
            return (
                conn.execute("DELETE FROM prompt_pairs WHERE id=?", (pair_id,)).rowcount
                > 0
            )
    def record_prompt_pair_use(self, pair_id: int, times: int = 1) -> bool:
        """Count pair uses independently of standalone prompt usage."""
        if type(times) is not int or times < 1:
            raise ValueError("times must be a positive integer")
        with self._pair_transaction() as conn:
            return (
                conn.execute(
                    "UPDATE prompt_pairs SET run_count=run_count+?,last_used_at=? "
                    "WHERE id=?",
                    (times, utc_now_iso(), pair_id),
                ).rowcount
                > 0
            )
    def link_image_to_pair(
        self, pair_id: int, image_id: int, provenance: Dict[str, Any]
    ) -> int:
        """Associate an image with immutable run provenance, idempotently."""
        with self._pair_transaction() as conn:
            return self._link_image_to_pair(conn, pair_id, image_id, provenance)
    def _link_image_to_pair(
        self,
        conn: sqlite3.Connection,
        pair_id: int,
        image_id: int,
        provenance: Dict[str, Any],
    ) -> int:
        """Associate an image with immutable run provenance, idempotently."""
        if not isinstance(provenance, dict):
            raise ValueError("provenance must be an object")
        try:
            encoded = json.dumps(
                provenance, sort_keys=True, separators=(",", ":"), allow_nan=False
            )
        except (TypeError, ValueError, RecursionError) as error:
            raise ValueError("provenance must contain JSON values") from error
        error = pair_provenance_error(encoded)
        if error:
            raise ValueError(error)
        pair = conn.execute(
            "SELECT positive_prompt_id FROM prompt_pairs WHERE id=?", (pair_id,)
        ).fetchone()
        image = conn.execute(
            "SELECT prompt_id FROM generated_images WHERE id=?", (image_id,)
        ).fetchone()
        if not pair or not image:
            raise ValueError("Pair and image must exist")
        if pair["positive_prompt_id"] != image["prompt_id"]:
            raise ValueError("Image must belong to the pair's positive prompt")
        conn.execute(
            "INSERT INTO image_prompt_pairs(pair_id,image_id,provenance,created_at) "
            "VALUES(?,?,?,?) ON CONFLICT(pair_id,image_id,provenance) DO NOTHING",
            (pair_id, image_id, encoded, utc_now_iso()),
        )
        return conn.execute(
            "SELECT id FROM image_prompt_pairs "
            "WHERE pair_id=? AND image_id=? AND provenance=?",
            (pair_id, image_id, encoded),
        ).fetchone()["id"]
    def get_image_prompt_pairs(self, image_id: int) -> List[Dict[str, Any]]:
        """Return every recorded pair association with decoded provenance."""
        rows = (
            self.model.get_connection()
            .execute(
                "SELECT * FROM image_prompt_pairs WHERE image_id=? ORDER BY id",
                (image_id,),
            )
            .fetchall()
        )
        return [
            {**dict(row), "provenance": json.loads(row["provenance"])} for row in rows
        ]
@@ -0,0 +1,365 @@
 """A restore in two steps: stage the upload, then publish it (#249 System S8).
 **Stage** (``stage_restore``) never touches the live database or the source.
 In a new folder ``.pm-restore-*`` in the library's own folder (#318: never
 the system temp folder, which may be another disk), it:
 1. checks the source (``verify_database_file`` without reading every
   workflow document: a damaged one is repaired below, not refused);
 2. copies it with SQLite's backup API (a rollback-journal file and a WAL file
   alike) and opens the copy as a ``PromptModel``, which migrates an older
   schema;
 3. repairs damaged workflow documents (``repair_damaged_documents``),
   rotates the history epoch and counts what the copy holds;
 4. makes the copy one self-contained file and verifies it fully (every
   current table, every document);
 5. records the copy's SHA-256, so publishing binds exactly these bytes.
 **Publish** (``publish_staged``) re-hashes the stage (``StageChanged`` if it
 differs), keeps the current database as a uniquely named, standalone safety
 backup beside it (``database/safety_backup.py``; not verified, because the
 database being replaced may be the damaged one), copies the stage over the
 live pages through the backup API (no file is swapped under open handles; an
 ``OperationalError`` there leaves the live data intact, #226), then
 checkpoints the WAL with TRUNCATE so the copied pages do not leave a
 database-sized ``-wal`` behind. A reader that keeps a snapshot past the busy
 wait is logged, not fatal: the restore itself is done.
 Every handle on the stage is closed before its folder is removed (Windows).
 ``sweep_restore_temps`` removes what an interrupted restore left (startup).
 """
 import hashlib
 import logging
 import os
 import sqlite3
 import tempfile
 from contextlib import closing
 from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Dict, Optional
 from . import workflow_documents as wd
 from .execution_history_schema import rotate_history_epoch
 from .safety_backup import take_safety_backup
 logger = logging.getLogger("prompt_manager.database.restore_stage")
 STAGE_PREFIX = ".pm-restore-"
 UPLOAD_PREFIX = ".pm-restore-upload-"
 STAGED_NAME = "staged.db"
 # How long the post-restore TRUNCATE checkpoint waits for readers to leave.
 CHECKPOINT_BUSY_MS = 30_000
 _HASH_CHUNK = 1024 * 1024
 SUMMARY_KEYS = (
    "prompts",
    "positives",
    "pairs",
    "archived_pairs",
    "image_links",
    "runs",
    "saved_views",
 )
 class StageChanged(ValueError):
    """The staged file is no longer the bytes that were reviewed."""
@dataclass
 class StagedRestore:
    """A verified, migrated copy of an upload, ready to publish."""
    path: str
    digest: str
    size: int
    summary: Dict[str, int]
    schema: Dict[str, bool]
    documents: Dict[str, int] = field(
        default_factory=lambda: {"repaired": 0, "cleared": 0, "images": 0}
    )
    @property
    def directory(self) -> str:
        return os.path.dirname(self.path)
    def discard(self) -> None:
        """Remove the stage folder (idempotent; its handles are all closed)."""
        remove_stage_folder(self.directory)
 def remove_stage_folder(directory: str) -> None:
    """Remove a stage folder and its files, logging what cannot be removed."""
    if not directory or not os.path.isdir(directory):
        return
    for name in os.listdir(directory):
        try:
            os.remove(os.path.join(directory, name))
        except OSError as error:
            logger.warning(f"Could not remove restore stage file {name}: {error}")
    try:
        os.rmdir(directory)
    except OSError as error:
        logger.warning(f"Could not remove the restore stage folder: {error}")
 def file_digest(path: str) -> str:
    """``sha256:<hex>`` of the file's bytes (blocking)."""
    digest = hashlib.sha256()
    with open(path, "rb") as handle:
        for chunk in iter(lambda: handle.read(_HASH_CHUNK), b""):
            digest.update(chunk)
    return "sha256:" + digest.hexdigest()
 def _readonly(path: str) -> sqlite3.Connection:
    return sqlite3.connect(Path(path).resolve().as_uri() + "?mode=ro", uri=True)
 def _schema(path: str) -> list:
    """The file's schema objects, to tell whether staging migrated it."""
    with closing(_readonly(path)) as conn:
        return sorted(
            conn.execute(
                "SELECT type, name, tbl_name, sql FROM sqlite_master "
                "WHERE name NOT LIKE 'sqlite_%'"
            ).fetchall(),
            key=repr,
        )
 def _count(conn: sqlite3.Connection, table: str, where: str = "") -> int:
    exists = conn.execute(
        "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
    ).fetchone()
    if not exists:
        return 0
    # ``table`` and ``where`` are constants of library_summary, never input.
    return conn.execute(f"SELECT COUNT(*) FROM {table} {where}").fetchone()[0]  # nosec
 def library_summary(conn: sqlite3.Connection) -> Dict[str, int]:
    """What a library holds, for a restore's review (reads only).
    ``positives`` are the Library's prompts (``library_counts``'s
    ``positive_prompts``); ``pairs`` the active saved pairs and
    ``archived_pairs`` the archived ones; ``image_links`` the indexed image
    rows; ``runs`` the recorded executions.
    """
    from .library import _ELIGIBLE
    has_state = conn.execute(
        "SELECT 1 FROM sqlite_master WHERE type='table' AND "
        "name='prompt_library_state'"
    ).fetchone()
    has_pairs = conn.execute(
        "SELECT 1 FROM sqlite_master WHERE type='table' AND name='prompt_pairs'"
    ).fetchone()
    if has_state and has_pairs:
        positives = conn.execute(
            "SELECT COUNT(*) FROM prompts p WHERE " + _ELIGIBLE
        ).fetchone()[0]
    else:
        positives = _count(conn, "prompts")
    return {
        "prompts": _count(conn, "prompts"),
        "positives": positives,
        "pairs": _count(conn, "prompt_pairs", "WHERE archived=0"),
        "archived_pairs": _count(conn, "prompt_pairs", "WHERE archived<>0"),
        "image_links": _count(conn, "generated_images"),
        "runs": _count(conn, "execution_runs"),
        "saved_views": _count(conn, "saved_views"),
    }
 # The newest row of each table a restore would overwrite: with the counts,
 # a write after the review (a new run, image, prompt or edit) changes it.
 _FINGERPRINT_SQL = (
    ("prompts", "SELECT MAX(id), MAX(updated_at), MAX(last_used_at) FROM prompts"),
    ("generated_images", "SELECT MAX(id) FROM generated_images"),
    ("prompt_pairs", "SELECT MAX(id), MAX(updated_at) FROM prompt_pairs"),
    ("execution_runs", "SELECT COUNT(*), MAX(rowid) FROM execution_runs"),
    ("saved_views", "SELECT MAX(rowid) FROM saved_views"),
 )
 def live_fingerprint(conn: sqlite3.Connection):
    """``(summary, fingerprint)`` of a live library, read in one snapshot.
    The fingerprint changes when rows are added to or removed from the
    tables a restore replaces, or when a prompt, pair, run or saved view is
    the newest edited (counts plus the newest row and edit of each table);
    the apply refuses with ``plan-changed`` then. It does not see every
    edit (a tag or favorite change on an older row); those are kept in the
    restore's safety backup. A read transaction the caller already has is
    used and left open; one opened here is ended here.
    """
    import json
    opened = not conn.in_transaction
    if opened:
        conn.execute("BEGIN")  # one read snapshot for every count
    try:
        summary = library_summary(conn)
        newest = []
        for table, sql in _FINGERPRINT_SQL:
            try:
                newest.append([table, list(conn.execute(sql).fetchone())])
            except sqlite3.OperationalError:
                newest.append([table, None])  # no such table or column
    finally:
        if opened:
            conn.rollback()  # it only read
    text = json.dumps([summary, newest], sort_keys=True, default=str)
    return summary, "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest()
 def _copy(src_path: str, staged_path: str) -> None:
    source = _readonly(src_path)
    try:
        with closing(sqlite3.connect(staged_path)) as destination:
            source.backup(destination)
    finally:
        source.close()
 def _standalone(path: str) -> None:
    with closing(sqlite3.connect(path)) as conn:
        conn.execute("PRAGMA journal_mode=DELETE").fetchone()
 def stage_restore(model_cls, db_path: str, src_path: str) -> StagedRestore:
    """Stage ``src_path`` beside ``db_path`` (blocking); see the module docstring.
    Raises ValueError with a readable reason when the source or the migrated
    stage does not verify; the stage folder is then already removed.
    """
    ok, reason = model_cls.verify_database_file(src_path, documents=False)
    if not ok:
        raise ValueError(reason)
    before = _schema(src_path)
    with closing(_readonly(src_path)) as conn:
        library_state = bool(
            conn.execute(
                "SELECT 1 FROM sqlite_master WHERE type='table' AND "
                "name='prompt_library_state'"
            ).fetchone()
        )
    folder = os.path.dirname(os.path.abspath(db_path))
    directory = tempfile.mkdtemp(prefix=STAGE_PREFIX, dir=folder)
    staged_path = os.path.join(directory, STAGED_NAME)
    staged_key = os.path.normcase(os.path.realpath(staged_path))
    try:
        _copy(src_path, staged_path)
        staged = None
        try:
            staged = model_cls(staged_path)
            conn = staged.get_connection()
            documents = wd.repair_damaged_documents(conn)
            rotate_history_epoch(conn)
            summary = library_summary(conn)
        finally:
            if staged is not None:
                staged.close_all()
        _standalone(staged_path)
        ok, reason = model_cls.verify_database_file(
            staged_path, require_library_state=True
        )
        if not ok:
            raise ValueError(reason)
        result = StagedRestore(
            path=staged_path,
            digest=file_digest(staged_path),
            size=os.path.getsize(staged_path),
            summary=summary,
            schema={
                "migrated": _schema(staged_path) != before,
                "library_state": library_state,
            },
            documents=documents,
        )
    except BaseException:
        remove_stage_folder(directory)
        raise
    finally:
        with model_cls._init_lock:
            model_cls._initialized_paths.discard(staged_key)
    return result
 def checkpoint_truncate(conn: sqlite3.Connection) -> bool:
    """Checkpoint the WAL and truncate it to 0 bytes; False when a reader held it."""
    previous = conn.execute("PRAGMA busy_timeout").fetchone()[0]
    try:
        conn.execute(f"PRAGMA busy_timeout = {int(CHECKPOINT_BUSY_MS)}")
        busy = conn.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0]
    finally:
        conn.execute(f"PRAGMA busy_timeout = {int(previous)}")
    return not busy
 def publish_staged(model, staged: StagedRestore) -> str:
    """Publish ``staged`` over ``model``'s database; return the safety backup path.
    "" when there was no database to back up. Raises StageChanged before
    anything is written when the stage's bytes changed since staging, and
    SafetyBackupFailed when the current database could not be kept. An
    ``OperationalError`` from the page copy propagates with
    ``_prompt_manager_restore_backup_created`` set; the live data is intact.
    """
    if not os.path.isfile(staged.path) or file_digest(staged.path) != staged.digest:
        raise StageChanged("The staged restore changed since it was reviewed")
    backup_path = ""
    if os.path.exists(model.db_path):
        backup_path = take_safety_backup(model, verify=False)
        logger.info(f"Current database backed up to: {os.path.basename(backup_path)}")
    try:
        model._replace_contents_from(staged.path)
    except sqlite3.OperationalError as exc:
        exc._prompt_manager_restore_backup_created = bool(backup_path)
        raise
    with model._init_lock:
        model._initialized_paths.add(model._schema_key())
    _truncate_wal(model)
    return backup_path
 def _truncate_wal(model) -> Optional[bool]:
    """TRUNCATE-checkpoint the live WAL after a publish (never raises)."""
    try:
        conn = model.get_connection()
        with conn.using():
            truncated = checkpoint_truncate(conn)
    except sqlite3.Error as error:
        logger.warning(f"The database log could not be truncated: {error}")
        return None
    if not truncated:
        logger.warning(
            "The database log could not be truncated after the restore: a "
            "reader kept it open. It shrinks at the next checkpoint with no "
            "reader."
        )
    return truncated
 def sweep_restore_temps(db_path: str) -> list:
    """Remove the stages and uploads an interrupted restore left (startup)."""
    folder = os.path.dirname(os.path.abspath(db_path))
    removed = []
    try:
        names = os.listdir(folder)
    except OSError:
        return removed
    for name in names:
        path = os.path.join(folder, name)
        if name.startswith(UPLOAD_PREFIX) and os.path.isfile(path):
            try:
                os.remove(path)
                removed.append(name)
            except OSError:
                continue
        elif name.startswith(STAGE_PREFIX) and os.path.isdir(path):
            remove_stage_folder(path)
            if not os.path.exists(path):
                removed.append(name)
    return removed
@@ -0,0 +1,648 @@
 """Needs review items (#249): persist and read capture-bound ambiguous evidence.
 An item is one ambiguous resolver result for one annotation byte record (the
 whole-file SHA-256 of one physical file at one locator), keyed by
 ``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the
 same evidence for the same bytes is a duplicate and never touches an existing
 item, its state, revision or decisions. Changed bytes are a new byte record
 and therefore a new pending item; the old one stays.
 Callers pass verified facts (the capture's digest, size and ``mtime_ns``);
 nothing here reads the filesystem. Writes run in one ``_pair_transaction``
 fenced by the storage epoch the capture was taken under.
 """
 import hashlib
 import json
 import os
 import re
 import sqlite3
 from typing import Iterator
 try:
    from ..utils.hashing import generate_prompt_hash
 except ImportError:
    from utils.hashing import generate_prompt_hash
 from .annotation_schema import (
    BYTES_TABLE,
    FILES_TABLE,
    METADATA_TABLE as ANNOTATION_METADATA_TABLE,
 )
 from .annotations import _new_id, _validate_locator, _validate_target
 from .models import normalize_image_path, utc_now_iso
 from .review_changes import notify_review_change
 from .review_schema import (
    DECISIONS_TABLE,
    ITEMS_TABLE,
    MAX_CANDIDATES,
    MAX_SPELLED_PATH_BYTES,
    METADATA_TABLE,
    canonical_evidence,
    evidence_digest,
    evidence_offers_choice,
 )
 # Only this module's table constants are interpolated into the SQL below.
 _INSERT_ITEM = (
    f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path,"  # nosec B608
    "resolver_version,evidence_hash,evidence,candidate_count,limited,state,"
    "revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)"
 )
 _BUMP_REVIEW_INDEX = (
    f"UPDATE {METADATA_TABLE} SET "  # nosec B608
    "review_index_revision=review_index_revision+1 WHERE singleton_id=1"
 )
 # A later spelling of the same locator (a case-only rename on a case-folding
 # filesystem) replaces the recorded one, so targets keep formatting.
 _REFRESH_SPELLING = (
    f"UPDATE {ITEMS_TABLE} SET spelled_path=? "  # nosec B608
    f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b "
    f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)"
 )
 _REVIEW_INDEX = (
    f"SELECT review_index_revision FROM {METADATA_TABLE} "  # nosec B608
    "WHERE singleton_id=1"
 )
 _BUMP_IMAGE_INDEX = (
    f"UPDATE {ANNOTATION_METADATA_TABLE} SET "  # nosec B608
    "image_index_revision=image_index_revision+1 WHERE singleton_id=1"
 )
 _ITEMS_OF_LOCATOR = (
    "SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash,"  # nosec B608
    "i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at,"
    f"i.updated_at,b.sha256 FROM {FILES_TABLE} f "
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? ORDER BY i.created_at,i.id"
 )
 _QUEUED = (
    f"SELECT 1 FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
    "LIMIT 1"
 )
 _PENDING_AT = (
    f"SELECT 1 FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
    "AND i.state='pending' LIMIT 1"
 )
 # Bytes verified at this stat whose item carries a decision (any version).
 _DECIDED_AT = (
    f"SELECT 1 FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? AND b.size=? AND b.mtime_ns=? "
    "AND (i.state<>'pending' OR i.revision<>0 OR EXISTS (SELECT 1 FROM "
    f"{DECISIONS_TABLE} d WHERE d.item_id=i.id)) LIMIT 1"
 )
 # The other items of one byte record (another version or evidence), with
 # whether each may be superseded: pending, revision 0, no decision row.
 _OTHER_ITEMS_OF_BYTES = (
    "SELECT i.id, (i.state='pending' AND i.revision=0 AND NOT EXISTS "  # nosec B608
    f"(SELECT 1 FROM {DECISIONS_TABLE} d WHERE d.item_id=i.id)) "
    f"FROM {ITEMS_TABLE} i WHERE i.byte_id=? "
    "AND NOT (i.resolver_version=? AND i.evidence_hash=?)"
 )
 # Pending items of the bytes verified at one stat, with their versions.
 _PENDING_AT_STAT = (
    f"SELECT i.id,i.resolver_version FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.state='pending'"
 )
 # Pending items of one file: the candidates a link settles.
 _PENDING_OF_LOCATOR = (
    f"SELECT i.id FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? AND i.state='pending'"
 )
 # Pending items of one file with their evidence: an untraced file's settle.
 _PENDING_EVIDENCE_OF_LOCATOR = (
    f"SELECT i.id,i.evidence FROM {FILES_TABLE} f "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
    f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
    "WHERE f.locator=? AND i.state='pending'"
 )
 # Only an undecided item goes: a decision row (RESTRICT) is never deleted.
 _SETTLE_ITEM = (
    f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' "  # nosec B608
    f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d "
    f"WHERE d.item_id={ITEMS_TABLE}.id)"
 )
 _EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
 _PENDING_COUNT = (
    f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'"  # nosec B608
 )
 _QUEUE_COLUMNS = (
    "id",
    "state",
    "revision",
    "created_at",
    "candidate_count",
    "limited",
    "spelled_path",
    "locator",
 )
 _QUEUE_SELECT = (
    "SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count,"  # nosec B608
    f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i "
    f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
    f"JOIN {FILES_TABLE} f ON f.id=b.file_id "
 )
 _QUEUE_FIRST = _QUEUE_SELECT + (
    "WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?"
 )
 _QUEUE_AFTER = _QUEUE_SELECT + (
    "WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) "
    "ORDER BY i.created_at,i.id LIMIT ?"
 )
 _DETAIL_COLUMNS = _QUEUE_COLUMNS + (
    "byte_id",
    "resolver_version",
    "evidence_hash",
    "evidence",
    "updated_at",
    "sha256",
 )
 _DETAIL = (
    "SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count,"  # nosec B608
    "i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version,"
    "i.evidence_hash,i.evidence,i.updated_at,b.sha256 "
    f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
    f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?"
 )
 # The legacy rows and pair links the ingester's compatibility rules read.
 _DEPENDENT_IMAGES = (
    "SELECT gi.id,p.hash,gi.image_path FROM generated_images gi "
    "JOIN prompts p ON p.id=gi.prompt_id "
    "WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id"
 )
 _DEPENDENT_LINKS = (
    "SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip "
    "JOIN prompt_pairs pp ON pp.id=ip.pair_id "
    "LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
    "WHERE ip.image_id=? ORDER BY ip.pair_id"
 )
 # Rows whose spelling ends with the file's name: the candidates for another
 # spelling of the same physical file (the link summaries' filename fallback).
 _SAME_NAME_ROWS = (
    "SELECT id,image_path FROM generated_images "
    "WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id"
 )
 _TITLES = (
    "SELECT p.hash,s.title FROM prompts p "
    "JOIN prompt_library_state s ON s.prompt_id=p.id "
    "WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id"
 )
 _ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
 _ITEM_COLUMNS = (
    "id",
    "byte_id",
    "resolver_version",
    "evidence_hash",
    "evidence",
    "candidate_count",
    "limited",
    "state",
    "revision",
    "created_at",
    "updated_at",
    "sha256",
 )
 def _validate_spelling(spelled_path: object, locator: str) -> None:
    if (
        not isinstance(spelled_path, str)
        or not spelled_path
        or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES
        or os.path.normcase(spelled_path) != locator
    ):
        raise ValueError("spelled_path must be the file's spelling of its locator")
 def dependent_images(conn: sqlite3.Connection, spellings) -> list:
    """``(id, image_path, positive_hash)`` of every legacy row of the file.
    ``spellings`` are the paths the file is linked under (the Apply path and
    every alias of the same physical file); each is matched as the ingester
    matches one path. Rows are distinct and ordered by id.
    """
    found = {}
    for spelled in dict.fromkeys(spellings):
        for row in conn.execute(
            _DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled)
        ):
            found[row[0]] = (row[0], row[2], row[1])
    return [found[key] for key in sorted(found)]
 def same_name_rows(conn: sqlite3.Connection, name: str) -> list:
    """``(id, image_path)`` of rows whose spelling may end with ``name``."""
    # Imported here: operations composes this module's mixin.
    from .operations import escape_like
    return [
        (row[0], row[1])
        for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",))
    ]
 def dependency_fingerprint(
    conn: sqlite3.Connection, image_path: str, aliases=()
 ) -> str:
    """SHA-256 of what Apply's compatibility depends on for ``image_path``.
    The exact legacy rows of the file under ``image_path`` and every alias
    spelling of it, with their positive hashes, and each row's pair links
    with the negative hash, read as the ingester reads them. Unrelated prompt
    changes (tags, titles) do not move it.
    """
    rows = []
    for image_id, _spelled, positive_hash in dependent_images(
        conn, (image_path, *aliases)
    ):
        links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall()
        rows.append([image_id, positive_hash, [list(link) for link in links]])
    text = canonical_evidence({"v": 1, "images": rows})
    return hashlib.sha256(text.encode("utf-8")).hexdigest()
 class ReviewOperations:
    """PromptDatabase mixin: insert and read Needs review items."""
    def record_review_item(
        self,
        *,
        locator: str,
        spelled_path: str,
        sha256: str,
        size: int,
        mtime_ns: int,
        epoch: str,
        resolver_version: int,
        evidence: dict,
        candidate_count: int,
        limited: bool,
    ) -> bool:
        """Store one ambiguous result for these exact bytes; True when new.
        The byte record is created (or its verified fingerprint refreshed)
        like an observe. ``spelled_path`` is the file's own spelling (it must
        fold to ``locator``); targets are formatted from it. Raises
        ``LibraryCommandError("storage-changed")`` when the storage epoch is
        no longer ``epoch`` (a restore since the capture) and ``ValueError``
        for invalid facts or evidence.
        One file never holds two pending items for the same bytes: the new
        item supersedes the byte record's other items (an older resolver or
        semantics version) when every one is pending and undecided (the safe
        delete of the self-heal); when one carries a decision or a later
        revision, it is kept and nothing new is stored (False).
        """
        _validate_target(locator, sha256, size, mtime_ns)
        _validate_spelling(spelled_path, locator)
        if type(resolver_version) is not int or resolver_version < 1:
            raise ValueError("resolver_version must be a positive integer")
        if (
            type(candidate_count) is not int
            or not 0 <= candidate_count <= MAX_CANDIDATES
        ):
            raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}")
        if not isinstance(limited, bool) or (limited and candidate_count):
            raise ValueError("A limited item carries no candidates")
        text = canonical_evidence(evidence)
        evidence_hash = evidence_digest(text)
        with self._pair_transaction() as conn:
            self._library_epoch(conn, epoch)
            now = utc_now_iso()
            file_id = self._annotation_owner(conn, locator, now)
            byte_id, _revision, created, moved = self._annotation_byte(
                conn, file_id, sha256, size, mtime_ns, now
            )
            if created or moved:
                conn.execute(_BUMP_IMAGE_INDEX)
            # The byte record is created or refreshed even when the item is
            # refused below (an observe would do the same); that commits
            # with the image index bump, and the review queue is unchanged.
            others = conn.execute(
                _OTHER_ITEMS_OF_BYTES, (byte_id, resolver_version, evidence_hash)
            ).fetchall()
            if any(not removable for _id, removable in others):
                return False
            superseded = sum(
                conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
                for item_id, _removable in others
            )
            cursor = conn.execute(
                _INSERT_ITEM,
                (
                    _new_id(),
                    byte_id,
                    spelled_path,
                    resolver_version,
                    evidence_hash,
                    text,
                    candidate_count,
                    int(limited),
                    now,
                    now,
                ),
            )
            inserted = cursor.rowcount > 0
            refreshed = _refresh_spelling(conn, locator, spelled_path)
            if inserted or refreshed or superseded:
                conn.execute(_BUMP_REVIEW_INDEX)
        if inserted or refreshed or superseded:
            notify_review_change(self.model.db_path)
        return inserted
    def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool:
        """Record ``spelled_path`` as the spelling of every item at ``locator``.
        The rescan's skip calls it for a file it does not copy again: a
        case-only rename keeps size and ``mtime_ns``, and a target formatted
        from the old spelling would be refused. True when a spelling changed
        (the review index revision moves; item revisions do not, so an open
        decision stays valid).
        """
        _validate_locator(locator)
        _validate_spelling(spelled_path, locator)
        with self._pair_transaction() as conn:
            refreshed = _refresh_spelling(conn, locator, spelled_path)
            if refreshed:
                conn.execute(_BUMP_REVIEW_INDEX)
        if refreshed:
            notify_review_change(self.model.db_path)
        return refreshed
    def review_index_revision(self) -> int:
        """The queue's revision (moved by every queue change)."""
        row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone()
        if row is None:
            raise sqlite3.DatabaseError("Missing review queue metadata")
        return row[0]
    def review_queue_version(self) -> dict:
        """``{library_epoch, review_revision}`` of the queue, from one snapshot.
        The pair a list read answers: equal pairs mean the same queue (a
        restore moves the epoch, every queue change moves the revision).
        """
        with self._annotation_read() as conn:
            epoch = conn.execute(_EPOCH).fetchone()
            revision = conn.execute(_REVIEW_INDEX).fetchone()
        if epoch is None or revision is None:
            raise sqlite3.DatabaseError("Missing review queue metadata")
        return {"library_epoch": epoch[0], "review_revision": revision[0]}
    def review_items_for(self, locator: str) -> list:
        """Every item of the file at ``locator``, oldest first, evidence parsed."""
        _validate_locator(locator)
        return list(self._iter_review_items(locator))
    def _iter_review_items(self, locator: str) -> Iterator[dict]:
        conn = self.model.get_connection()
        for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)):
            item = dict(zip(_ITEM_COLUMNS, tuple(row)))
            item["evidence"] = json.loads(item["evidence"])
            item["limited"] = bool(item["limited"])
            yield item
    # -- queue reads (#249 E3) -------------------------------------------------
    def review_fingerprint_queued(
        self, locator: str, size: int, mtime_ns: int, resolver_version: int
    ) -> bool:
        """True when an item cites a byte record verified at this exact stat.
        A rescan uses it to skip the copy and hash of a file it already
        queued: the same ``(size, mtime_ns)`` the bytes were verified with.
        """
        row = (
            self.model.get_connection()
            .execute(_QUEUED, (locator, size, mtime_ns, resolver_version))
            .fetchone()
        )
        return row is not None
    def review_decided_at(self, locator: str, size: int, mtime_ns: int) -> bool:
        """True when bytes verified at this stat carry a decided item.
        Any version: a newer resolver or table never adds an item beside a
        decision, so the rescan need not copy those bytes again.
        """
        row = (
            self.model.get_connection()
            .execute(_DECIDED_AT, (locator, size, mtime_ns))
            .fetchone()
        )
        return row is not None
    def review_pending_at(
        self, locator: str, size: int, mtime_ns: int, resolver_version: int
    ) -> bool:
        """True when a *pending* item cites bytes verified at this exact stat.
        ``review_fingerprint_queued`` also matches decided items; a file whose
        only item is decided is not in the queue.
        """
        row = (
            self.model.get_connection()
            .execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version))
            .fetchone()
        )
        return row is not None
    def review_file_linked(self, spellings) -> bool:
        """True when a saved row links the file under any of ``spellings``."""
        with self._annotation_read() as conn:
            return bool(dependent_images(conn, spellings))
    def settle_linked_review_items(self, files) -> int:
        """Remove the pending, undecided items of files that are now linked.
        ``files`` holds ``(locator, spellings)`` per file. An item leaves the
        queue only when it is still ``pending`` at revision 0 with no decision
        row and a saved row links its file under one of ``spellings``; a
        decided item is never touched. Files without a pending item cost one
        indexed read and open no write transaction. Returns how many items
        were removed (the review index revision moves when any was).
        """
        files = [(locator, tuple(spellings)) for locator, spellings in files]
        for locator, _spellings in files:
            _validate_locator(locator)
        conn = self.model.get_connection()
        candidates = [
            (locator, spellings)
            for locator, spellings in files
            if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
        ]
        if not candidates:
            return 0
        removed = 0
        with self._pair_transaction() as conn:
            for locator, spellings in candidates:
                if not dependent_images(conn, spellings):
                    continue
                for (item_id,) in conn.execute(
                    _PENDING_OF_LOCATOR, (locator,)
                ).fetchall():
                    removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
            if removed:
                conn.execute(_BUMP_REVIEW_INDEX)
        if removed:
            notify_review_change(self.model.db_path)
        return removed
    def settle_stale_review_items(
        self, locator: str, size: int, mtime_ns: int, versions
    ) -> int:
        """Remove pending, undecided items of the bytes verified at this stat
        whose ``resolver_version`` is not one of ``versions``.
        The caller found these bytes untraced under the current versions, so
        an older version's item could only refuse Apply. The same safe delete
        as the other settles; one indexed read when nothing is pending.
        """
        _validate_locator(locator)
        current = set(versions)
        conn = self.model.get_connection()
        stale = [
            item_id
            for item_id, version in conn.execute(
                _PENDING_AT_STAT, (locator, size, mtime_ns)
            )
            if version not in current
        ]
        if not stale:
            return 0
        removed = 0
        with self._pair_transaction() as conn:
            for item_id in stale:
                removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
            if removed:
                conn.execute(_BUMP_REVIEW_INDEX)
        if removed:
            notify_review_change(self.model.db_path)
        return removed
    def settle_untraced_review_items(self, locators) -> int:
        """Remove the pending, undecided items that offer nothing to choose.
        ``locators`` are files the caller found ambiguous without a choice
        (fewer than two candidates carry a prompt). Of their items, one leaves
        the queue only when its own stored evidence offers no choice either
        (``evidence_offers_choice``) and it is still ``pending`` at revision 0
        with no decision row, the same safe delete as
        ``settle_linked_review_items``. Files without a pending item cost one
        indexed read and open no write transaction. Returns how many items
        were removed (the review index revision moves when any was).
        """
        locators = list(locators)
        for locator in locators:
            _validate_locator(locator)
        conn = self.model.get_connection()
        pending = [
            locator
            for locator in locators
            if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
        ]
        if not pending:
            return 0
        removed = 0
        with self._pair_transaction() as conn:
            for locator in pending:
                rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall()
                for item_id, evidence in rows:
                    if evidence_offers_choice(json.loads(evidence)):
                        continue
                    removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
            if removed:
                conn.execute(_BUMP_REVIEW_INDEX)
        return removed
    def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict:
        """One page of pending items, oldest first, from one snapshot.
        ``after`` is the ``(created_at, id)`` keyset of the last item already
        shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is
        the exact pending count and ``items`` carry their ``spelled_path`` and
        ``locator`` (server-side only).
        """
        if type(limit) is not int or limit < 1:
            raise ValueError("limit must be a positive integer")
        with self._annotation_read() as conn:
            epoch = conn.execute(_EPOCH).fetchone()
            revision = conn.execute(_REVIEW_INDEX).fetchone()
            if epoch is None or revision is None:
                raise sqlite3.DatabaseError("Missing review queue metadata")
            total = conn.execute(_PENDING_COUNT).fetchone()[0]
            if after is None:
                rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall()
            else:
                rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall()
        items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]]
        return {
            "epoch": epoch[0],
            "revision": revision[0],
            "total": total,
            "items": items,
            "more": len(rows) > limit,
        }
    def review_item(self, item_id: str) -> dict | None:
        """One item with its parsed evidence and the pending total, or None."""
        if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None:
            return None
        with self._annotation_read() as conn:
            epoch = conn.execute(_EPOCH).fetchone()
            if epoch is None:
                raise sqlite3.DatabaseError("Missing execution history metadata")
            row = conn.execute(_DETAIL, (item_id,)).fetchone()
            if row is None:
                return None
            item = _queue_row(row, _DETAIL_COLUMNS)
            item["evidence"] = json.loads(item["evidence"])
            item["epoch"] = epoch[0]
            item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0]
        return item
    def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str:
        with self._annotation_read() as conn:
            return dependency_fingerprint(conn, image_path, aliases)
    def review_same_name_rows(self, name: str) -> list:
        """``(id, image_path)`` of legacy rows that may spell a file ``name``."""
        with self._annotation_read() as conn:
            return same_name_rows(conn, name)
    def stored_prompt_titles(self, texts) -> dict:
        """Library titles of saved prompts by text (dedup hash), when set."""
        hashes = {generate_prompt_hash(text): text for text in texts if text.strip()}
        if not hashes:
            return {}
        marks = ",".join("?" * len(hashes))
        found: dict[str, tuple] = {}
        conn = self.model.get_connection()
        for prompt_hash, title in conn.execute(
            _TITLES.format(marks=marks), list(hashes)
        ):
            text = hashes[prompt_hash]
            found[text] = found.get(text, ()) + (title,)
        return found
 def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool:
    return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0
 def _queue_row(row, columns) -> dict:
    item = dict(zip(columns, tuple(row)))
    item["limited"] = bool(item["limited"])
    return item
@@ -0,0 +1,75 @@
 """Needs review queue change notices (#249 PR F2).
 Every committed change to the pending queue (an item recorded by the scan or
 the monitor, a spelling refresh, a decision) calls ``notify_review_change``
 after its transaction, from whichever thread wrote it. The review event
 stream (``py/api/review_events.py``) subscribes, so an open Needs review page
 re-reads its queue when it changes instead of polling. Observers are keyed by
 the database file, like ``subscribe_history_restore``; an observer that fails
 is logged and never fails the write.
 """
 import os
 import threading
 from typing import Callable
 try:
    from ..utils.logging_config import get_logger
 except ImportError:
    from utils.logging_config import get_logger
 _logger = get_logger("prompt_manager.database.review_changes")
 _lock = threading.Lock()
 _observers: dict[str, dict[object, Callable[[], None]]] = {}
 def _identity(db_path: str) -> str:
    return os.path.normcase(os.path.realpath(db_path))
 def subscribe_review_changes(
    db_path: str, callback: Callable[[], None]
 ) -> Callable[[], None]:
    """Call ``callback()`` after each committed queue change; return the unsubscribe."""
    identity = _identity(db_path)
    token = object()
    with _lock:
        _observers.setdefault(identity, {})[token] = callback
    def unsubscribe() -> None:
        with _lock:
            observers = _observers.get(identity)
            if observers is not None:
                observers.pop(token, None)
                if not observers:
                    _observers.pop(identity, None)
    return unsubscribe
 def observer_count(db_path: str) -> int:
    """How many observers watch ``db_path`` (tests and diagnostics)."""
    with _lock:
        return len(_observers.get(_identity(db_path), {}))
 def notify_review_change(db_path: str) -> None:
    """Tell every observer of ``db_path``; call it after the commit."""
    with _lock:
        callbacks = tuple(_observers.get(_identity(db_path), {}).values())
    for callback in callbacks:
        try:
            callback()
        except Exception:
            _logger.warning("Needs review change observer failed", exc_info=True)
 def rekey_review_changes(old_path: str, new_path: str) -> None:
    """Move the observers of ``old_path`` to ``new_path`` (Move library)."""
    old, new = _identity(old_path), _identity(new_path)
    if old == new:
        return
    with _lock:
        observers = _observers.pop(old, None)
        if observers:
            _observers.setdefault(new, {}).update(observers)
@@ -0,0 +1,291 @@
 """Needs review decisions (#249 E3): explicit, auditable and compatible-only.
 A decision is one row of ``media_review_decisions`` for one item revision.
 ``selected`` links the chosen candidate's pair to the file through the public
 ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same
 ``_pair_transaction`` as the decision row, the item's new state and revision
 and the review index revision; any conflict or later failure rolls all of it
 back. ``unresolved`` records the decision and links nothing. No decision
 counts a run, touches usage or calls an execution writer.
 Callers verify the file first (the capture must be the item's bytes and the
 current file must still hash to them); nothing here reads the filesystem.
 """
 import os
 import sqlite3
 from .annotation_schema import BYTES_TABLE
 from .annotations import _new_id
 from .library_commands import LibraryCommandError
 from .models import utc_now_iso
 from .review import (
    _BUMP_REVIEW_INDEX,
    _DETAIL,
    _DETAIL_COLUMNS,
    _PENDING_COUNT,
    _queue_row,
    dependency_fingerprint,
    dependent_images,
    same_name_rows,
 )
 from .review_changes import notify_review_change
 from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence
 DECISION_KIND = "user-selection"
 _INSERT_DECISION = (
    f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision,"  # nosec B608
    "created_at,outcome,candidate_hash,evidence,pair_id,image_id) "
    "VALUES(?,?,?,?,?,?,?,?,?)"
 )
 _ADVANCE_ITEM = (
    f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1,"  # nosec B608
    "updated_at=? WHERE id=? AND revision=?"
 )
 _ITEM_BYTES = (
    f"SELECT b.sha256 FROM {ITEMS_TABLE} i "  # nosec B608
    f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?"
 )
 class ReviewCommandError(LibraryCommandError):
    """A decision lost a race or would rewrite an association.
    ``code`` is ``review-conflict`` (the item was decided or changed since it
    was read), ``association-conflict`` (the choice is not compatible with
    the file's saved links, or those links changed), ``file-changed`` (the
    item is not about these bytes) or ``review-missing``. ``item`` is the
    current item read inside the transaction, ``total`` the pending count and
    ``fingerprint`` the current dependency fingerprint, when known;
    ``changed`` marks an association conflict caused by links that changed
    since the reviewer read them (rather than an incompatible choice).
    """
    def __init__(
        self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False
    ):
        super().__init__(code, epoch)
        self.item = item
        self.total = total
        self.fingerprint = fingerprint
        self.changed = changed
        # The public view of ``item``, set by the API when it formats one.
        self.view = None
 class _Rollback(Exception):
    """Raised inside the transaction so the context manager rolls back."""
    def __init__(self, error: ReviewCommandError):
        super().__init__(error.code)
        self.error = error
 class ReviewDecisionOperations:
    """PromptDatabase mixin: record one decision on one Needs review item."""
    def decide_review_item(
        self,
        *,
        item_id: str,
        expected_epoch: str,
        expected_revision: int,
        sha256: str,
        outcome: str,
        candidate: dict | None = None,
        image_path: str | None = None,
        metadata: dict | None = None,
        expected_fingerprint: str | None = None,
        aliases: tuple = (),
        known_rows: frozenset | None = None,
    ) -> dict:
        """Record the decision; return ``{item, total, pair}``.
        ``selected`` needs the evidence ``candidate`` (with its ``hash``), the
        ``image_path`` the scan and the monitor link under, the captured
        ``metadata`` and the ``expected_fingerprint`` the reviewer saw.
        ``aliases`` are the other spellings legacy rows of the same physical
        file use (a symlinked, junctioned or mapped root) and ``known_rows``
        the ids of every same-name row the caller classified; a same-name row
        it did not see is a changed association.
        Raises ``LibraryCommandError("storage-changed")`` after a restore and
        :class:`ReviewCommandError` for every refused decision, with zero
        writes either way.
        """
        if outcome not in ("selected", "unresolved"):
            raise ValueError("outcome must be selected or unresolved")
        if outcome == "selected" and (
            not isinstance(candidate, dict) or not image_path
        ):
            raise ValueError("A selected decision needs a candidate and a file")
        try:
            with self._pair_transaction() as conn:
                decided = self._decide_in(
                    conn,
                    item_id=item_id,
                    expected_epoch=expected_epoch,
                    expected_revision=expected_revision,
                    sha256=sha256,
                    outcome=outcome,
                    candidate=candidate,
                    image_path=image_path,
                    metadata=metadata or {},
                    expected_fingerprint=expected_fingerprint,
                    aliases=tuple(aliases),
                    known_rows=known_rows,
                )
        except _Rollback as rollback:
            raise rollback.error from None
        # After the commit: open Needs review pages re-read their queue.
        notify_review_change(self.model.db_path)
        return decided
    def _decide_in(self, conn: sqlite3.Connection, **command) -> dict:
        epoch = self._library_epoch(conn, command["expected_epoch"])
        item_id = command["item_id"]
        current = _current_item(conn, item_id)
        if current is None:
            raise _Rollback(ReviewCommandError("review-missing", epoch))
        total = conn.execute(_PENDING_COUNT).fetchone()[0]
        if (
            current["state"] != "pending"
            or current["revision"] != command["expected_revision"]
        ):
            raise _Rollback(
                ReviewCommandError("review-conflict", epoch, item=current, total=total)
            )
        if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]:
            raise _Rollback(ReviewCommandError("file-changed", epoch))
        decision_id = _new_id()
        pair = None
        candidate = command["candidate"] if command["outcome"] == "selected" else None
        if candidate is not None:
            pair = self._apply_candidate(
                conn, epoch, current, decision_id, total, **command
            )
        evidence = {
            "kind": DECISION_KIND,
            "decision_id": decision_id,
            "outcome": command["outcome"],
            "reviewed_digest": command["sha256"],
            "item_evidence_hash": current["evidence_hash"],
            "resolver_version": current["resolver_version"],
            "candidate": candidate,
            "dependency_fingerprint": command["expected_fingerprint"],
            "pair": pair,
        }
        now = utc_now_iso()
        self._insert_review_decision(
            conn,
            (
                decision_id,
                item_id,
                current["revision"] + 1,
                now,
                command["outcome"],
                candidate["hash"] if candidate is not None else None,
                canonical_evidence(evidence),
                pair["id"] if pair else None,
                pair["image_id"] if pair else None,
            ),
        )
        conn.execute(
            _ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"])
        )
        conn.execute(_BUMP_REVIEW_INDEX)
        return {
            "item": _current_item(conn, item_id),
            "total": conn.execute(_PENDING_COUNT).fetchone()[0],
            "pair": pair,
            "decision_id": decision_id,
        }
    def _apply_candidate(
        self, conn, epoch, current, decision_id, total, **command
    ) -> dict:
        """Link the candidate's pair, compatible-only; raise on any conflict."""
        # Imported here: pair_ingestion imports operations, which composes this.
        from .pair_ingestion import _ingest_resolved_pair_in
        image_path = command["image_path"]
        aliases = command["aliases"]
        fingerprint = dependency_fingerprint(conn, image_path, aliases)
        known = command["known_rows"]
        unseen = known is not None and any(
            row_id not in known
            for row_id, _path in same_name_rows(conn, os.path.basename(image_path))
        )
        if unseen or fingerprint != command["expected_fingerprint"]:
            raise _Rollback(
                ReviewCommandError(
                    "association-conflict",
                    epoch,
                    item=current,
                    total=total,
                    fingerprint=fingerprint,
                    changed=True,
                )
            )
        rows = dependent_images(conn, (image_path, *aliases))
        if len(rows) > 1:
            # One file linked under several spellings: no single row to enrich.
            raise _Rollback(
                ReviewCommandError(
                    "association-conflict",
                    epoch,
                    item=current,
                    total=total,
                    fingerprint=fingerprint,
                )
            )
        if rows:
            # Enrich the one row of this file under the spelling it was
            # linked with, so the ingester's compatibility rules see it.
            image_path = rows[0][1]
        candidate = command["candidate"]
        positive, negative = candidate["positive"], candidate["negative"]
        provenance = {
            "kind": DECISION_KIND,
            "decision_id": decision_id,
            "reviewed_digest": command["sha256"],
            "candidate_hash": candidate["hash"],
            "resolver_version": current["resolver_version"],
            "positive": positive,
            "negative": negative,
        }
        result = _ingest_resolved_pair_in(
            conn,
            self,
            image_path,
            command["metadata"],
            {"state": "resolved", "positive": positive, "negative": negative},
            provenance=provenance,
        )
        if result["status"] == "conflict":
            raise _Rollback(
                ReviewCommandError(
                    "association-conflict",
                    epoch,
                    item=current,
                    total=total,
                    fingerprint=fingerprint,
                )
            )
        return {
            "id": result["pair_id"],
            "image_id": result["image_id"],
            "status": result["status"],
        }
    @staticmethod
    def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None:
        conn.execute(_INSERT_DECISION, values)
 def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None:
    row = conn.execute(_DETAIL, (item_id,)).fetchone()
    if row is None:
        return None
    item = _queue_row(row, _DETAIL_COLUMNS)
    item.pop("evidence")
    return item
@@ -0,0 +1,513 @@
 """Needs review schema group (#249): capture-bound ambiguous items and decisions.
 The group installs like ``image_annotations``: optional in old backups,
 required once a database is current, and valid only beside a valid image
 annotation group, because every item cites one annotation byte record. It owns
 - ``media_review_metadata``: one row, the group's schema version and the
  review index revision (moved by every queue change);
 - ``media_review_items``: one ambiguous resolver result for one byte record,
  keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's
  spelling when it was recorded (the locator is case-folded on Windows, a
  target is not), its canonical evidence, candidate count, ``limited`` flag,
  state and revision;
 - ``media_review_decisions``: the auditable decision history of an item, one
  row per decision revision, with optional navigation to the pair and image
  the decision linked (nulled, never cascaded, when those are deleted).
 Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted
 keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound
 agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is
 the SHA-256 of the item's evidence text.
 Installation never backfills anything and never touches the filesystem.
 Validation reads only: catalog objects compared after whitespace compaction,
 columns and foreign keys through the PRAGMAs, persisted values in single
 bounded SQL passes, then one streamed pass that re-canonicalizes each evidence
 document (one row in memory at a time).
 """
 import hashlib
 import json
 import re
 import sqlite3
 from typing import Any, Optional
 from .annotation_schema import BYTES_TABLE
 from .annotation_schema import TABLES as ANNOTATION_TABLES
 from .annotation_schema import _catalog as _annotation_catalog
 from .annotation_schema import annotation_schema_error
 try:
    from ..utils.json_strict import dumps_strict
 except ImportError:
    from utils.json_strict import dumps_strict
 SCHEMA_VERSION = 1
 METADATA_TABLE = "media_review_metadata"
 ITEMS_TABLE = "media_review_items"
 DECISIONS_TABLE = "media_review_decisions"
 TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE)
 MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
 MAX_CANDIDATES = 16
 # The file's own spelling when it was recorded (bytes of UTF-8 text). Locators
 # are case-folded on Windows, so a target is formatted from this instead.
 MAX_SPELLED_PATH_BYTES = 16384
 ITEM_STATES = ("pending", "selected", "unresolved")
 DECISION_OUTCOMES = ("selected", "unresolved")
 _TIMESTAMP_BYTES = 64
 _HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'"
 _EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})"
 _TABLE_DDL = {
    METADATA_TABLE: (
        f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
        "CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
        f"CHECK(schema_version={SCHEMA_VERSION}), "
        "review_index_revision INTEGER NOT NULL DEFAULT 0 "
        "CHECK(review_index_revision>=0))"
    ),
    ITEMS_TABLE: (
        f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
        f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, "
        "spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) "
        f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), "
        "resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL "
        f"CHECK({_HEX64.format(c='evidence_hash')}), "
        f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
        "candidate_count INTEGER NOT NULL "
        f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), "
        "limited INTEGER NOT NULL CHECK(limited IN (0,1)), "
        "state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), "
        "revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, "
        "updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
    ),
    DECISIONS_TABLE: (
        f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY "
        "CHECK(length(id)=36), item_id TEXT NOT NULL "
        f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, "
        "decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
        "outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
        "candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
        f"({_HEX64.format(c='candidate_hash')})), "
        f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
        "pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
        "image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
        "UNIQUE(item_id, decision_revision), "
        "CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
    ),
 }
 _INDEX_DDL = {
    # The queue reads pending items oldest first with a stable tiebreak.
    "idx_media_review_items_queue": (
        "CREATE INDEX idx_media_review_items_queue ON "
        f"{ITEMS_TABLE}(state, created_at, id)"
    ),
    # Deleting a pair or an image nulls decision navigation by index, not scan.
    "idx_media_review_decisions_pair": (
        "CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)"
    ),
    "idx_media_review_decisions_image": (
        "CREATE INDEX idx_media_review_decisions_image ON "
        f"{DECISIONS_TABLE}(image_id)"
    ),
 }
 # ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key.
 _AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2}
 # (name, declared type, not null, default, primary key position)
 _COLUMNS = {
    METADATA_TABLE: (
        ("singleton_id", "INTEGER", 0, None, 1),
        ("schema_version", "INTEGER", 1, None, 0),
        ("review_index_revision", "INTEGER", 1, "0", 0),
    ),
    ITEMS_TABLE: (
        ("id", "TEXT", 0, None, 1),
        ("byte_id", "TEXT", 1, None, 0),
        ("spelled_path", "TEXT", 1, None, 0),
        ("resolver_version", "INTEGER", 1, None, 0),
        ("evidence_hash", "TEXT", 1, None, 0),
        ("evidence", "TEXT", 1, None, 0),
        ("candidate_count", "INTEGER", 1, None, 0),
        ("limited", "INTEGER", 1, None, 0),
        ("state", "TEXT", 1, None, 0),
        ("revision", "INTEGER", 1, None, 0),
        ("created_at", "TEXT", 1, None, 0),
        ("updated_at", "TEXT", 1, None, 0),
    ),
    DECISIONS_TABLE: (
        ("id", "TEXT", 0, None, 1),
        ("item_id", "TEXT", 1, None, 0),
        ("decision_revision", "INTEGER", 1, None, 0),
        ("created_at", "TEXT", 1, None, 0),
        ("outcome", "TEXT", 1, None, 0),
        ("candidate_hash", "TEXT", 0, None, 0),
        ("evidence", "TEXT", 1, None, 0),
        ("pair_id", "INTEGER", 0, None, 0),
        ("image_id", "INTEGER", 0, None, 0),
    ),
 }
 # (column, parent table, parent column, on delete)
 _FOREIGN_KEYS = {
    METADATA_TABLE: set(),
    ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")},
    DECISIONS_TABLE: {
        ("item_id", ITEMS_TABLE, "id", "RESTRICT"),
        ("pair_id", "prompt_pairs", "id", "SET NULL"),
        ("image_id", "generated_images", "id", "SET NULL"),
    },
 }
 class _NonCanonical(ValueError):
    pass
 def _reject_constant(name: str) -> Any:
    raise _NonCanonical(f"JSON constant {name} is not allowed")
 # Built once: validation re-encodes every stored document at startup.
 _ENCODER = json.JSONEncoder(
    sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False
 )
 _DECODER = json.JSONDecoder(parse_constant=_reject_constant)
 def canonical_evidence(value: Any) -> str:
    """Serialize one evidence object canonically; ``ValueError`` when invalid.
    The text is a JSON object with sorted keys, no whitespace, ASCII only and
    no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number
    (ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's
    policy everywhere (``utils.json_strict.dumps_strict``).
    """
    if not isinstance(value, dict):
        raise ValueError("Review evidence must be a JSON object")
    try:
        try:
            text = _ENCODER.encode(value)
        except ValueError as error:
            if "Out of range float" not in str(error):
                raise
            text = dumps_strict(
                value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
            )
    except (TypeError, RecursionError) as error:
        raise ValueError(f"Review evidence is not serializable: {error}") from None
    if len(text) > MAX_EVIDENCE_BYTES:
        raise ValueError("Review evidence is larger than 8 MiB")
    return text
 def choosable(candidate: Any) -> bool:
    """True when a candidate offers a prompt to choose: it is ``resolved`` and
    its positive side carries non-blank text. The page disables any other
    candidate, so it is never something to choose.
    """
    if not isinstance(candidate, dict) or candidate.get("state") != "resolved":
        return False
    positive = candidate.get("positive")
    if not isinstance(positive, dict):
        return False
    text = positive.get("text")
    return isinstance(text, str) and bool(text.strip())
 # An item is worth a decision only when at least this many candidates offer
 # a prompt; with fewer there is nothing to choose between (maintainer test of
 # #284/#288: 39 items whose candidates were all "unresolved").
 MIN_CHOOSABLE = 2
 def evidence_offers_choice(evidence: Any) -> bool:
    """False when stored evidence provably offers nothing to choose.
    Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A
    limited item stores none, only ``candidate_total``; it may offer a choice
    (more than ``MAX_CANDIDATES`` prompts) unless that total is too small.
    """
    if not isinstance(evidence, dict):
        return False
    if evidence.get("limited"):
        total = evidence.get("candidate_total")
        return type(total) is int and total >= MIN_CHOOSABLE
    candidates = evidence.get("candidates")
    if not isinstance(candidates, list):
        return False
    return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
 def evidence_digest(text: str) -> str:
    """SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical)."""
    return hashlib.sha256(text.encode("utf-8")).hexdigest()
 def _is_canonical(text: Any) -> bool:
    if not isinstance(text, str):
        return False
    try:
        value = _DECODER.decode(text)
        return canonical_evidence(value) == text
    except (ValueError, RecursionError):
        return False
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql).lower()
 def _uuid(column: str) -> str:
    """SQL: *column* is a lowercase hyphenated 36-character UUID string."""
    return (
        f"typeof({column}) = 'text' AND length({column}) = 36 "
        f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
        f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
        f"AND length(replace({column}, '-', '')) = 32 "
        f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
    )
 def _timestamp(column: str) -> str:
    return (
        f"typeof({column}) = 'text' "
        f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
    )
 def _non_negative(column: str) -> str:
    return f"typeof({column}) = 'integer' AND {column} >= 0"
 def _hex64(column: str) -> str:
    return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column)
 def _evidence(column: str) -> str:
    # Type and size only: the streamed pass below proves the text canonical
    # (which implies printable ASCII), so a pattern scan here would read every
    # document a second time.
    return (
        f"typeof({column}) = 'text' "
        f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}"
    )
 def _nullable_integer(column: str) -> str:
    return f"({column} IS NULL OR typeof({column}) = 'integer')"
 def _in(column: str, values: tuple) -> str:
    return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})"
 # Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
 _ROW_PREDICATES = {
    ITEMS_TABLE: " AND ".join(
        (
            _uuid("id"),
            _uuid("byte_id"),
            "typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) "
            f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}",
            "typeof(resolver_version) = 'integer'",
            _hex64("evidence_hash"),
            _evidence("evidence"),
            _non_negative("candidate_count"),
            f"candidate_count <= {MAX_CANDIDATES}",
            "typeof(limited) = 'integer' AND limited IN (0, 1)",
            _in("state", ITEM_STATES),
            _non_negative("revision"),
            _timestamp("created_at"),
            _timestamp("updated_at"),
        )
    ),
    DECISIONS_TABLE: " AND ".join(
        (
            _uuid("id"),
            _uuid("item_id"),
            _non_negative("decision_revision"),
            _timestamp("created_at"),
            _in("outcome", DECISION_OUTCOMES),
            "(CASE WHEN outcome = 'selected' THEN "
            f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)",
            _evidence("evidence"),
            _nullable_integer("pair_id"),
            _nullable_integer("image_id"),
        )
    ),
 }
 def _expected_catalog() -> set:
    objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
    objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
    objects |= {
        ("index", f"sqlite_autoindex_{table}_{n}", "")
        for table, count in _AUTOINDEXES.items()
        for n in range(1, count + 1)
    }
    return objects
 def _catalog(conn: sqlite3.Connection) -> list:
    names = TABLES + tuple(_INDEX_DDL)
    marks = ",".join("?" * len(names))
    table_marks = ",".join("?" * len(TABLES))
    return conn.execute(  # only "?" placeholders are interpolated
        "SELECT type,name,sql FROM sqlite_master "
        f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})",  # nosec B608
        names + TABLES,
    ).fetchall()
 def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
    for table, expected in _COLUMNS.items():
        columns = tuple(
            (row[1], row[2].upper(), row[3], row[4], row[5])
            for row in conn.execute(f"PRAGMA table_info({table})")
        )
        if columns != expected:
            return f"Review queue table {table} has invalid columns"
        foreign_keys = {
            (row[3], row[2], row[4], row[6])
            for row in conn.execute(f"PRAGMA foreign_key_list({table})")
        }
        if foreign_keys != _FOREIGN_KEYS[table]:
            return f"Review queue table {table} has invalid foreign keys"
        if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
            return f"Review queue table {table} contains foreign key violations"
    return None
 def _evidence_error(conn: sqlite3.Connection) -> Optional[str]:
    """Stream every evidence document once; one row is held at a time.
    Rows are read with ``fetchone``: iterating a managed connection's cursor
    would prefetch ``ITER_CHUNK`` (128) rows (``database/managed_connection.py``).
    """
    items = conn.execute(
        f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}"  # nosec B608
    )
    while (row := items.fetchone()) is not None:
        evidence_hash, text = row[0], row[1]
        if not _is_canonical(text) or evidence_digest(text) != evidence_hash:
            return f"Review queue table {ITEMS_TABLE} contains invalid evidence"
    decisions = conn.execute(f"SELECT evidence FROM {DECISIONS_TABLE}")  # nosec B608
    while (row := decisions.fetchone()) is not None:
        text = row[0]
        if not _is_canonical(text):
            return f"Review queue table {DECISIONS_TABLE} contains invalid evidence"
    return None
 def _value_error(conn: sqlite3.Connection) -> Optional[str]:
    # Only this module's constants are interpolated into the SQL below.
    rows = conn.execute(
        "SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' "  # nosec B608
        "AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
        f"AND schema_version = {SCHEMA_VERSION} "
        f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}"
    ).fetchone()
    if rows[0] != 1 or rows[1] != 1:
        return "Review queue metadata is invalid"
    for table, predicate in _ROW_PREDICATES.items():
        if conn.execute(
            f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"  # nosec B608
        ).fetchone():
            return f"Review queue table {table} contains invalid values"
    return _evidence_error(conn)
 def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
    """Catalog only: a review group present without its annotation tables.
    Startup asks this before the annotation group installs, which would make
    a review group copied without it look valid. It reads no rows; the full
    validation runs once in :func:`migrate_review_state`.
    """
    if not _catalog(conn):
        return None
    present = {
        name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
    }
    if present != set(ANNOTATION_TABLES):
        return (
            "Review queue schema requires a valid image annotation schema: "
            "Missing image annotation schema"
        )
    return None
 def review_schema_error(
    conn: sqlite3.Connection,
    *,
    allow_missing: bool = True,
    annotations_valid: bool = False,
 ) -> Optional[str]:
    """Reject partial or unsupported review schemas and invalid rows.
    Reads only: nothing is ever repaired. A database without any of the
    group's objects is acceptable for backups that predate it unless
    ``allow_missing`` is false. A present group requires a present, valid
    image annotation group; ``annotations_valid`` says the caller has just
    validated it (startup's annotation install), so only its presence is
    checked again.
    """
    objects = _catalog(conn)
    if not objects:
        return None if allow_missing else "Missing review queue schema"
    if annotations_valid:
        annotation_error = review_without_annotations_error(conn)
    else:
        annotation_error = annotation_schema_error(conn, allow_missing=False)
        if annotation_error:
            annotation_error = (
                "Review queue schema requires a valid image annotation schema: "
                + annotation_error
            )
    if annotation_error:
        return annotation_error
    present = {name for kind, name, _sql in objects if kind == "table"}
    if present != set(TABLES):
        return "Incomplete review queue schema"
    actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
    if actual != _expected_catalog():
        return "Review queue has an invalid schema definition"
    return _structure_error(conn) or _value_error(conn)
 def migrate_review_state(conn: sqlite3.Connection) -> None:
    """Install the group atomically; validate before and after creating it.
    Runs after ``migrate_annotation_state`` (which has just validated the
    annotation group): the group is only valid beside the annotation group it
    cites. A present group is validated once, evidence included.
    """
    conn.execute("SAVEPOINT media_review_schema")
    try:
        installed = bool(_catalog(conn))
        error = review_schema_error(conn, annotations_valid=True)
        if error:
            raise sqlite3.DatabaseError(error)
        if installed:
            # Already complete and valid: nothing to create, and the evidence
            # pass is the expensive part of validation, so it runs once.
            conn.execute("RELEASE media_review_schema")
            return
        for sql in _TABLE_DDL.values():
            conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        for sql in _INDEX_DDL.values():
            conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
        conn.execute(
            f"INSERT OR IGNORE INTO {METADATA_TABLE} "
            "(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)",
            (SCHEMA_VERSION,),
        )
        error = review_schema_error(conn, allow_missing=False, annotations_valid=True)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute("RELEASE media_review_schema")
    except BaseException:
        conn.execute("ROLLBACK TO media_review_schema")
        conn.execute("RELEASE media_review_schema")
        raise
@@ -0,0 +1,244 @@
 """Attribute an output file to a stored prompt through recorded history (#342).
 Execution history records, per run, the files ComfyUI reported
 (``execution_run_outputs``) and the PromptManager texts the run used
 (``execution_run_contexts``, whose ``positive_prompt_id`` hint is set only
 while the raw text matches exactly one stored prompt). A file is attributed
 only when all of this holds:
 - exactly one recorded run reported the file's output location (same output
  root, subfolder and name);
 - the file was last written while that run was executing, from its dequeue
  to its terminal event (else the finish callback): a later run that history
  did not record may have reused the name;
 - every positive context of that run names the same stored prompt.
 Anything else returns None: the caller leaves the file unlinked. Read-only.
 Lookups are batched (``stored_run_prompts``): one parameterized query per
 chunk of at most ``LOOKUP_CHUNK`` distinct file names, matched in memory. The
 history schema is left exactly as it is (no extra index), so older builds can
 still open the library.
 """
 import logging
 import os
 import sqlite3
 import threading
 import weakref
 from typing import Any, Callable, Dict, Iterable, List, Optional
 try:
    from ..utils.execution_outputs import snapshot_output_roots
    from ..utils.library_location import import_folder_paths
 except ImportError:
    from utils.execution_outputs import snapshot_output_roots
    from utils.library_location import import_folder_paths
 # File times may trail or lead the recorded run times a little (coarse file
 # system clocks, the write finishing just after the host reported it).
 MTIME_SLACK_MS = 2000
 # Distinct file names per lookup query: well under SQLite's bound-parameter
 # limit (999 on old builds), so a scan batch costs one query per chunk.
 LOOKUP_CHUNK = 500
 _RUNS_FOR_NAMES = (
    "SELECT DISTINCT o.filename, o.subfolder, o.root_key, o.path_style, o.run_id, "
    "r.observed_dequeue_at, COALESCE(r.terminal_event_at, r.observed_callback_at) "
    "FROM execution_run_outputs o JOIN execution_runs r ON r.run_id = o.run_id "
    "WHERE o.folder_type = 'output' AND o.filename IN ({marks})"
 )
 def host_output_directory() -> Optional[str]:
    """ComfyUI's output directory, or None outside ComfyUI."""
    folder_paths = import_folder_paths()
    if folder_paths is None:
        return None
    try:
        value = folder_paths.get_output_directory()
    except Exception:
        return None
    return value if isinstance(value, str) and value else None
 def _inside(image_path: str, output_root: str) -> Optional[str]:
    """``image_path`` relative to ``output_root``, or None when outside it."""
    try:
        relative = os.path.relpath(image_path, output_root)
    except ValueError:  # another drive on Windows
        return None
    if os.path.isabs(relative) or relative == os.pardir:
        return None
    if relative.startswith(os.pardir + os.sep):
        return None
    return relative
 def _resolved(path: str) -> str:
    try:
        return os.path.realpath(path)
    except (OSError, ValueError):
        return os.path.abspath(path)
 def _location(image_path: str, output_root: str, path_style: str):
    """(root key, subfolder, filename) as history records them, or None.
    The root key is always the host's own spelling of its output folder,
    because history recorded that one. The file may be spelled through the
    same folder or through its resolved path (a symlinked or junction output
    folder, as scan roots are): the lexical spelling is tried first, then
    both sides resolved.
    """
    root_key = snapshot_output_roots(path_style, output_root, output_root).output_key
    if root_key is None:
        return None
    relative = _inside(os.path.abspath(image_path), os.path.abspath(output_root))
    if relative is None:
        relative = _inside(_resolved(image_path), _resolved(output_root))
    if relative is None:
        return None
    subfolder, filename = os.path.split(relative)
    if not filename:
        return None
    return root_key, subfolder, filename
 def in_output_folder(image_path: str, output_root: Optional[str]) -> bool:
    """Whether history could ever attribute ``image_path`` (inside the host's
    output folder, in either spelling)."""
    if not image_path or not output_root:
        return False
    style = "windows" if os.name == "nt" else "posix"
    return _location(image_path, output_root, style) is not None
 def _written_during(image_path: str, started: Any, finished: Any) -> bool:
    if not isinstance(started, int) or not isinstance(finished, int):
        return False
    try:
        written = os.stat(image_path).st_mtime_ns // 1_000_000
    except OSError:
        return False
    return started - MTIME_SLACK_MS <= written <= finished + MTIME_SLACK_MS
 def stored_run_prompt(
    db: Any,
    image_path: str,
    output_root: Optional[str],
    path_style: Optional[str] = None,
 ) -> Optional[int]:
    """The stored prompt id history attributes to ``image_path``, else None."""
    if not image_path:
        return None
    return stored_run_prompts(db, [image_path], output_root, path_style).get(image_path)
 def _runs_by_location(db: Any, names: List[str]) -> Optional[Dict[tuple, dict]]:
    """``{(filename, subfolder, root_key, style): {run_id: (start, end)}}``.
    One query per ``LOOKUP_CHUNK`` names; None without history tables.
    """
    found: Dict[tuple, dict] = {}
    conn = db.model.get_connection()
    for start in range(0, len(names), LOOKUP_CHUNK):
        chunk = names[start : start + LOOKUP_CHUNK]
        query = _RUNS_FOR_NAMES.format(marks=",".join("?" for _ in chunk))
        try:
            rows = conn.execute(query, chunk).fetchall()  # nosec B608
        except sqlite3.Error:
            return None  # no history tables (a library without history)
        for name, subfolder, root_key, style, run_id, began, ended in rows:
            runs = found.setdefault((name, subfolder, root_key, style), {})
            runs[run_id] = (began, ended)
    return found
 def _named_prompt(db: Any, run_id: str) -> Optional[int]:
    contexts = db.get_execution_contexts(run_id).get("contexts") or []
    named = {
        context.get("positive_prompt_id")
        for context in contexts
        if context.get("positive") is not None
    }
    if len(named) != 1:
        return None
    (prompt_id,) = named
    return prompt_id if isinstance(prompt_id, int) else None
 def stored_run_prompts(
    db: Any,
    image_paths: Iterable[str],
    output_root: Optional[str],
    path_style: Optional[str] = None,
 ) -> Dict[str, Optional[int]]:
    """``{path: stored prompt id or None}`` for many files in few queries.
    The output lookup is one query per ``LOOKUP_CHUNK`` distinct names; each
    run that alone reported a file is then read once for its contexts.
    """
    paths = list(dict.fromkeys(path for path in image_paths if path))
    result: Dict[str, Optional[int]] = dict.fromkeys(paths)
    if not paths or not output_root:
        return result
    style = path_style or ("windows" if os.name == "nt" else "posix")
    located = {}
    for path in paths:
        location = _location(path, output_root, style)
        if location is not None:
            located[path] = location
    if not located:
        return result
    names = sorted({filename for _key, _sub, filename in located.values()})
    runs = _runs_by_location(db, names)
    if not runs:
        return result
    prompts: Dict[str, Optional[int]] = {}
    for path, (root_key, subfolder, filename) in located.items():
        reported = runs.get((filename, subfolder, root_key, style), {})
        if len(reported) != 1:
            continue
        ((run_id, (began, ended)),) = reported.items()
        if not _written_during(path, began, ended):
            continue
        if run_id not in prompts:
            prompts[run_id] = _named_prompt(db, run_id)
        result[path] = prompts[run_id]
    return result
 _LOGGER = logging.getLogger("prompt_manager.run_attribution")
 _SEALED_LISTENERS: List[weakref.WeakMethod] = []
 _SEALED_LOCK = threading.Lock()
 def subscribe_outputs_sealed(method: Callable[[str, List[dict]], None]) -> None:
    """Call bound ``method(run_id, references)`` after a run's outputs are recorded.
    Held weakly: a listener that is garbage collected is dropped.
    """
    with _SEALED_LOCK:
        _SEALED_LISTENERS.append(weakref.WeakMethod(method))
 def notify_outputs_sealed(run_id: str, references: Iterable[dict]) -> None:
    """Tell listeners a run's output references were just committed.
    A listener failure is logged and never reaches the history writer.
    """
    references = list(references)
    with _SEALED_LOCK:
        alive = [ref for ref in _SEALED_LISTENERS if ref() is not None]
        _SEALED_LISTENERS[:] = alive
        listeners = [ref() for ref in alive]
    for listener in listeners:
        if listener is None:
            continue
        try:
            listener(run_id, references)
        except Exception:
            _LOGGER.warning("Output seal listener failed", exc_info=True)
@@ -0,0 +1,185 @@
 """Safety backups beside the live database: one writer for every caller.
 A safety backup is ``<db>.<label>_<YYYYmmdd_HHMMSS_ffffff>[_n]`` in the
 library's own folder. It is written by SQLite's backup API into a temp file
 beside the database (``SAFETY_TEMP_PREFIX``, swept at startup after a crash),
 made one self-contained file (rollback journal), optionally verified, fsynced,
 made private (``0o600`` on POSIX) and only then given its unique name, so a
 backup name never carries a half-written file. The names are unique within
 one second (microseconds, then a counter): on Windows two backups in the same
 second must not overwrite each other.
 Maintenance applies take a verified backup (``py/api/maintenance.py``); a
 restore takes an unverified one, because the database it replaces may be the
 damaged one the user is recovering from (``database/restore_stage.py``).
 """
 import datetime
 import os
 import sqlite3
 import tempfile
 from contextlib import closing
 SAFETY_TEMP_PREFIX = ".pm-safety-"
 _NAME_ATTEMPTS = 1000
 class SafetyBackupFailed(Exception):
    """The safety backup could not be written, verified or put in place.
    ``reason`` is the verify reason when the copy did not verify.
    """
    def __init__(self, message, reason=None):
        super().__init__(message)
        self.reason = reason
 def _now():
    return datetime.datetime.now()
 def remove_quietly(*paths):
    """Remove each existing path, ignoring errors (cleanup after a failure)."""
    for path in paths:
        try:
            if path and os.path.exists(path):
                os.remove(path)
        except OSError:
            pass
 def backup_names(db_path, label="backup"):
    """Candidate names, unique within one second (microseconds, then ``_n``)."""
    base = f"{db_path}.{label}_{_now().strftime('%Y%m%d_%H%M%S_%f')}"
    for attempt in range(_NAME_ATTEMPTS):
        yield base if attempt == 0 else f"{base}_{attempt}"
 def reserve_backup_name(db_path, label="backup"):
    """Create and return an unused ``<db>.<label>_<stamp>[_n]`` (O_EXCL)."""
    for candidate in backup_names(db_path, label):
        try:
            fd = os.open(candidate, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
        except FileExistsError:
            continue
        os.close(fd)
        return candidate
    raise SafetyBackupFailed("No free backup name")
 def place_backup(temp, db_path, label="backup"):
    """Give the finished ``temp`` its unique backup name; return that path.
    ``os.link`` is atomic and refuses an existing name, so no empty or half
    file ever carries a backup name, even across a crash. A filesystem
    without hard links falls back to reserving the name (``O_EXCL``) and
    ``os.replace``. The temp file is removed by the caller.
    """
    for candidate in backup_names(db_path, label):
        try:
            os.link(temp, candidate)
        except FileExistsError:
            continue
        except (OSError, NotImplementedError):
            break  # no hard links here
        else:
            return candidate
    else:
        raise SafetyBackupFailed("No free backup name")
    final = reserve_backup_name(db_path, label)
    try:
        os.replace(temp, final)
    except OSError:
        remove_quietly(final)
        raise
    return final
 def sweep_safety_temps(db_path):
    """Remove the temp copies an interrupted safety backup left (blocking).
    Called once at startup, before any job can take a backup. Returns the
    names removed.
    """
    folder = os.path.dirname(os.path.abspath(db_path))
    removed = []
    try:
        names = os.listdir(folder)
    except OSError:
        return removed
    for name in names:
        if not (name.startswith(SAFETY_TEMP_PREFIX) and ".tmp" in name):
            continue
        try:
            os.remove(os.path.join(folder, name))
            removed.append(name)
        except OSError:
            continue
    return removed
 def make_standalone(path):
    """Make the copy one self-contained file (rollback journal, not WAL).
    The backup API copies the live database's WAL flag; a WAL file opened
    read-only later (a verify, a restore preview) would leave ``-wal`` and
    ``-shm`` sidecars beside it.
    """
    try:
        with closing(sqlite3.connect(path)) as conn:
            conn.execute("PRAGMA journal_mode=DELETE").fetchone()
    except sqlite3.Error as exc:
        raise SafetyBackupFailed("The backup could not be finished") from exc
 def fsync_file(path):
    with open(path, "rb+") as handle:
        os.fsync(handle.fileno())
 def safety_temp(db_path):
    """A new, closed temp file beside the database (``SAFETY_TEMP_PREFIX``)."""
    fd, temp = tempfile.mkstemp(
        dir=os.path.dirname(os.path.abspath(db_path)),
        prefix=SAFETY_TEMP_PREFIX,
        suffix=".tmp",
    )
    os.close(fd)
    return temp
 def take_safety_backup(model, label="backup", *, verify=True):
    """Write (and by default verify) a safety backup; return its path (blocking).
    The copy is made in a temp file beside the database and only renamed
    onto its final, unique name ``<db>.<label>_<stamp>`` once it is complete
    (and verified), so a backup file on disk is always a whole one. ``label``
    "backup" joins the pruned pool; another label (the workflow storage
    migration's ``pre-workflow-storage``) is kept until the user removes it.
    Raises SafetyBackupFailed, leaving no file.
    """
    db_path = os.path.abspath(model.db_path)
    temp = safety_temp(db_path)
    final = None
    try:
        if not model.backup_database(temp):
            raise SafetyBackupFailed("The backup could not be written")
        make_standalone(temp)
        if verify:
            ok, reason = model.verify_database_file(temp)
            if not ok:
                raise SafetyBackupFailed("The backup did not verify", reason)
        fsync_file(temp)
        if os.name != "nt":
            os.chmod(temp, 0o600)
        final = place_backup(temp, db_path, label)
        return final
    except OSError as exc:
        remove_quietly(final)
        raise SafetyBackupFailed("The backup could not be put in place") from exc
    except SafetyBackupFailed:
        remove_quietly(final)
        raise
    finally:
        remove_quietly(temp, temp + "-journal", temp + "-wal", temp + "-shm")
@@ -0,0 +1,334 @@
 """Persisted saved views (D16): a named Images or Library toolbar state.
 A view's state holds exactly its page's toolbar values and its Refine-this-view
 refinements, whitelisted per page and stored as compact JSON of at most 4 KB.
 Writes (the API's POST and PATCH) validate strictly. Startup and restore check
 only the structure (the table, its columns and each row's types, sizes and JSON
 object), never the value rules: a row saved under older rules must not stop the
 database from opening. Startup is more tolerant still: a row whose state is not
 a JSON object is logged by id, left in place and skipped by every read, while
 restore rejects such a backup. Reads normalize instead, so a stale row opens with its
 usable values and defaults for the rest. Optional in old backups, required once
 a database is current (see ``library_schema.py``).
 """
 import json
 import logging
 import re
 import sqlite3
 from typing import Optional
 # The models logger's parent ("prompt_manager.database.models" logs there too).
 _LOGGER = logging.getLogger("prompt_manager.database.saved_views")
 TABLE = "saved_views"
 PAGES = ("images", "library")
 # The schema check's literal SQL repeats the name and state caps (names in
 # bytes, at most 4 per character); keep them in step.
 MAX_NAME_LENGTH = 100
 MAX_STATE_BYTES = 4096
 MAX_TAGS = 20
 MAX_TAG_LENGTH = 50
 _IMAGE_SORTS = ("newest", "oldest", "name", "name-desc", "rating")
 _LIBRARY_SORTS = ("newest", "oldest", "name", "rating")
 _MEDIA = ("all", "image", "video")
 _CONTROL = re.compile(r"[\x00-\x1f\x7f]")
 _SOURCE = re.compile(r"all|0|[1-9][0-9]{0,8}")
 FALLBACK_NAME = "Saved view"
 class SavedViewInvalid(ValueError):
    """A request's page, name or state breaks the write rules (a 400)."""
 # (kind, limit-or-choices, default) per state key; "text" values are
 # length-capped. The defaults match the client's (saved-views.mjs FIELDS).
 _STATE_FIELDS = {
    "images": {
        "q": ("text", 500, ""),
        "category": ("text", 100, ""),
        "sort": ("choice", _IMAGE_SORTS, "newest"),
        "source": ("source", None, "all"),
        "folder": ("text", 1024, ""),
        "media": ("choice", _MEDIA, "all"),
        "tags": ("tags", None, ()),
        # The Favorites page's filter (#249 D51): a view saved there reopens it.
        "favorites": ("bool", None, False),
    },
    "library": {
        "q": ("text", 500, ""),
        "category": ("text", 100, ""),
        "sort": ("choice", _LIBRARY_SORTS, "newest"),
        "tags": ("tags", None, ()),
    },
 }
 _DDL = (
    f"CREATE TABLE {TABLE} (id INTEGER PRIMARY KEY AUTOINCREMENT, "
    "page TEXT NOT NULL CHECK(page IN ('images','library')), "
    "name TEXT NOT NULL, state TEXT NOT NULL, position INTEGER NOT NULL, "
    "created_at TEXT NOT NULL, updated_at TEXT NOT NULL)"
 )
 _COLUMNS = (
    ("id", "INTEGER", 0, 1),
    ("page", "TEXT", 1, 0),
    ("name", "TEXT", 1, 0),
    ("state", "TEXT", 1, 0),
    ("position", "INTEGER", 1, 0),
    ("created_at", "TEXT", 1, 0),
    ("updated_at", "TEXT", 1, 0),
 )
 _SELECT = "SELECT id, page, name, state FROM saved_views"
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql).lower()
 def validate_name(name: object) -> str:
    """The trimmed name, 1 to 100 characters without control characters."""
    if not isinstance(name, str):
        raise SavedViewInvalid("Name must be a string")
    trimmed = name.strip()
    if not 1 <= len(trimmed) <= MAX_NAME_LENGTH:
        raise SavedViewInvalid(f"Name must be 1 to {MAX_NAME_LENGTH} characters")
    if _CONTROL.search(trimmed):
        raise SavedViewInvalid("Name contains control characters")
    return trimmed
 def _validate_value(key: str, kind: str, limit, value: object) -> None:
    """Raise ``SavedViewInvalid`` when *value* breaks *key*'s write rule."""
    if kind == "tags":
        if not isinstance(value, list) or len(value) > MAX_TAGS:
            raise SavedViewInvalid(
                f"State '{key}' must be a list of at most {MAX_TAGS}"
            )
        for tag in value:
            if (
                not isinstance(tag, str)
                or not tag.strip()
                or len(tag) > MAX_TAG_LENGTH
                or "," in tag
                or _CONTROL.search(tag)
            ):
                raise SavedViewInvalid(f"State '{key}' holds an invalid tag")
        return
    if kind == "bool":
        if not isinstance(value, bool):
            raise SavedViewInvalid(f"State '{key}' must be true or false")
        return
    if not isinstance(value, str):
        raise SavedViewInvalid(f"State '{key}' must be a string")
    if kind == "choice" and value not in limit:
        raise SavedViewInvalid(f"State '{key}' is not a supported value")
    if kind == "source" and not _SOURCE.fullmatch(value):
        raise SavedViewInvalid(f"State '{key}' must be 'all' or a source index")
    if kind == "text" and (len(value) > limit or _CONTROL.search(value)):
        raise SavedViewInvalid(f"State '{key}' is too long or has control characters")
 def encode_state(page: object, state: object) -> str:
    """Validate *state* for *page* and return its compact JSON (at most 4 KB)."""
    if page not in PAGES:
        raise SavedViewInvalid("Page must be 'images' or 'library'")
    fields = _STATE_FIELDS[page]
    if not isinstance(state, dict):
        raise SavedViewInvalid("State must be an object")
    if set(state) != set(fields):
        raise SavedViewInvalid(
            f"State for {page} must contain exactly " + ", ".join(sorted(fields))
        )
    for key, (kind, limit, _default) in fields.items():
        _validate_value(key, kind, limit, state[key])
    encoded = json.dumps(
        {key: state[key] for key in fields}, ensure_ascii=False, separators=(",", ":")
    )
    if len(encoded.encode("utf-8")) > MAX_STATE_BYTES:
        raise SavedViewInvalid("State must encode to at most 4 KB")
    return encoded
 def _decode_object(raw: object) -> Optional[dict]:
    """The JSON object stored in *raw*, or None when it holds no object."""
    if isinstance(raw, dict):
        return raw
    if not isinstance(raw, str):
        return None
    try:
        decoded = json.loads(raw)
    except (ValueError, RecursionError):
        return None
    return decoded if isinstance(decoded, dict) else None
 def normalize_state(page: str, raw: object) -> dict:
    """The page's full state from a stored row, never raising.
    Every key of the page's current whitelist is present: a stored value that
    passes today's write rule is kept, anything else (missing, invalid, or a
    row that is not a JSON object) takes the key's default, and keys the
    whitelist no longer has are dropped. An unknown page has no keys.
    """
    fields = _STATE_FIELDS.get(page, {})
    stored = _decode_object(raw) or {}
    state = {}
    for key, (kind, limit, default) in fields.items():
        value = stored.get(key, default)
        try:
            _validate_value(key, kind, limit, value)
        except SavedViewInvalid:
            value = default
        state[key] = list(value) if kind == "tags" else value
    return state
 def normalize_name(name: object) -> str:
    """A stored name made displayable: trimmed, capped, never empty."""
    text = _CONTROL.sub("", name) if isinstance(name, str) else ""
    trimmed = text.strip()[:MAX_NAME_LENGTH].strip()
    return trimmed or FALLBACK_NAME
 def saved_views_schema_error(
    conn: sqlite3.Connection,
    *,
    allow_missing: bool = True,
    check_states: bool = True,
 ) -> Optional[str]:
    """Reject unsupported saved-views schemas and structurally invalid rows.
    Reads only. Checks the table, its columns, each row's types and sizes and
    that each state is a JSON object; never the value rules, which change
    between versions and are normalized on read instead. A missing table is
    acceptable for backups that predate it unless ``allow_missing`` is false.
    ``check_states=False`` (live startup) skips the JSON-object check; such
    rows are reported by ``unusable_view_ids`` and skipped on read.
    """
    objects = conn.execute(
        "SELECT type,name,sql FROM sqlite_master WHERE name=? "
        "OR (type IN ('index','trigger') AND tbl_name=?)",
        (TABLE, TABLE),
    ).fetchall()
    if not objects:
        return None if allow_missing else "Missing saved views schema"
    tables = [sql for kind, _name, sql in objects if kind == "table"]
    if len(tables) != 1 or len(objects) != 1:
        return "Saved views name is reserved by an unsupported schema object"
    if _compact(tables[0] or "") != _compact(_DDL):
        return "Saved views has an invalid table definition"
    columns = tuple(
        (row[1], row[2].upper(), row[3], row[5])
        for row in conn.execute(f"PRAGMA table_info({TABLE})")
    )
    if columns != _COLUMNS:
        return "Saved views has invalid columns"
    # Literal SQL: the table name and limits are constants of this module.
    if conn.execute(
        "SELECT 1 FROM saved_views WHERE typeof(page) != 'text' "
        "OR page NOT IN ('images','library') "
        "OR typeof(name) != 'text' OR typeof(state) != 'text' "
        "OR typeof(position) != 'integer' OR typeof(created_at) != 'text' "
        "OR typeof(updated_at) != 'text' "
        "OR length(CAST(name AS BLOB)) > 400 "
        "OR length(CAST(state AS BLOB)) > 4096 "
        "OR length(CAST(created_at AS BLOB)) > 64 "
        "OR length(CAST(updated_at AS BLOB)) > 64 LIMIT 1"
    ).fetchone():
        return "Saved views contains invalid values"
    if check_states and unusable_view_ids(conn):
        return "Saved views contains invalid values"
    return None
 def unusable_view_ids(conn: sqlite3.Connection) -> list[int]:
    """Ids of rows whose state is not a JSON object (never listed)."""
    return [
        view_id
        for view_id, state in conn.execute("SELECT id, state FROM saved_views")
        if _decode_object(state) is None
    ]
 def migrate_saved_views(conn: sqlite3.Connection) -> None:
    """Install the table atomically; validate before and after creating it."""
    conn.execute("SAVEPOINT saved_views_schema")
    try:
        error = saved_views_schema_error(conn, check_states=False)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        error = saved_views_schema_error(conn, allow_missing=False, check_states=False)
        if error:
            raise sqlite3.DatabaseError(error)
        # A corrupt state never stops the live database from opening: the row
        # stays (no data is destroyed) and reads skip it. Only its id is logged.
        for view_id in unusable_view_ids(conn):
            _LOGGER.warning(
                "Saved view id %d has an unreadable state and is hidden", view_id
            )
        conn.execute("RELEASE saved_views_schema")
    except BaseException:
        conn.execute("ROLLBACK TO saved_views_schema")
        conn.execute("RELEASE saved_views_schema")
        raise
 def _view(row) -> dict:
    view_id, page, name, state = row
    return {
        "id": view_id,
        "page": page,
        "name": normalize_name(name),
        "state": normalize_state(page, state),
    }
 class SavedViewOperations:
    """PromptDatabase mixin: list, create, rename and delete saved views."""
    def list_saved_views(self) -> list[dict]:
        """Every view, in list order (position, then creation)."""
        rows = self.model.get_connection().execute(_SELECT + " ORDER BY position, id")
        # A row whose state is no JSON object (logged at startup) is never listed.
        return [_view(row) for row in rows if _decode_object(row[3]) is not None]
    def create_saved_view(self, page: object, name: object, state: object) -> dict:
        """Validate and append one view at the end of the list."""
        from .models import utc_now_iso  # models imports this module's schema
        encoded = encode_state(page, state)
        trimmed = validate_name(name)
        now = utc_now_iso()
        with self.model.get_connection() as conn:
            cursor = conn.execute(
                "INSERT INTO saved_views (page, name, state, position, "
                "created_at, updated_at) SELECT ?, ?, ?, "
                "COALESCE(MAX(position), -1) + 1, ?, ? FROM saved_views",
                (page, trimmed, encoded, now, now),
            )
            row = conn.execute(
                _SELECT + " WHERE id = ?", (cursor.lastrowid,)
            ).fetchone()
        return _view(row)
    def rename_saved_view(self, view_id: int, name: object) -> Optional[dict]:
        """The renamed view, or None when no view has *view_id*."""
        from .models import utc_now_iso
        trimmed = validate_name(name)
        with self.model.get_connection() as conn:
            conn.execute(
                "UPDATE saved_views SET name = ?, updated_at = ? WHERE id = ?",
                (trimmed, utc_now_iso(), view_id),
            )
            row = conn.execute(_SELECT + " WHERE id = ?", (view_id,)).fetchone()
        # A hidden (unreadable) view is not found, as in the listing.
        return _view(row) if row and _decode_object(row[3]) is not None else None
    def delete_saved_view(self, view_id: int) -> bool:
        """True when a view was removed."""
        with self.model.get_connection() as conn:
            cursor = conn.execute("DELETE FROM saved_views WHERE id = ?", (view_id,))
        return cursor.rowcount > 0
@@ -0,0 +1,461 @@
 """Bounded tag and category reads, and guarded tag commands.
 A count is the number of eligible Library parents that carry the value
 directly or through one active saved pair, which is exactly what
 ``list_library(tags=[name])`` / ``list_library(category=value)`` return.
 A tag row's ``image_count`` is the number of files (private locators) with any
 byte record carrying the tag (#249), so a file counts once however many of its
 byte versions carry it, and equal bytes at two paths count twice.
 Tag commands run in one ``_pair_transaction``: the storage epoch is checked
 first, then the exact ``{id, name}`` identity, and the receipt is read on the
 same connection before commit. Owner rows and their timestamps never change.
 Category consolidation is previewed in one ``library_snapshot`` and applied in
 one ``_pair_transaction`` that recomputes the owner projection and requires the
 preview's ``impact_token``; only rows carrying the exact source change.
 """
 import sqlite3
 from .library import _ELIGIBLE, _limit
 from .library_commands import _TOKEN, LibraryCommandError, _validate_epoch
 from .library_tokens import canonical_token
 from .models import utc_now_iso
 try:
    from ..utils.validators import validate_category, validate_tags
 except ImportError:
    from utils.validators import validate_category, validate_tags
 _CATEGORY_DOMAIN = "prompt-manager-vocabulary-category:v1\n"
 # Files carrying a tag through any of their byte records (#249 image tags).
 _IMAGE_FILES = (
    "SELECT mit.tag_id,COUNT(DISTINCT b.file_id) FROM media_image_tags mit "
    "JOIN media_annotation_bytes b ON b.id=mit.byte_id"
 )
 def _tag_rows(page: str) -> str:
    """Count rows for the registry rows selected by ``page`` (id, name)."""
    # Eligible parent components carrying a tag in the page, plus the
    # positive parent of every active pair carrying it (always eligible).
    return (
        "WITH page AS (" + page + "),"
        "owners(tag_id,pid) AS ("
        "SELECT pt.tag_id,p.id FROM prompts p "
        "JOIN prompt_tags pt ON pt.prompt_id=p.id "
        "WHERE pt.tag_id IN (SELECT id FROM page) AND " + _ELIGIBLE + " "
        "UNION SELECT ppt.tag_id,pp.positive_prompt_id FROM prompt_pairs pp "
        "JOIN prompt_pair_tags ppt ON ppt.pair_id=pp.id "
        "WHERE pp.archived=0 AND ppt.tag_id IN (SELECT id FROM page)),"
        "images(tag_id,n) AS (" + _IMAGE_FILES + " "
        "WHERE mit.tag_id IN (SELECT id FROM page) GROUP BY mit.tag_id) "
        "SELECT page.id,page.name,COUNT(DISTINCT owners.pid) AS prompt_count,"
        "COALESCE(MAX(images.n),0) AS image_count "
        "FROM page LEFT JOIN owners ON owners.tag_id=page.id "
        "LEFT JOIN images ON images.tag_id=page.id "
        "GROUP BY page.id ORDER BY page.id"
    )
 _TAG_PAGE = _tag_rows("SELECT id,name FROM tags WHERE id>? ORDER BY id LIMIT ?")
 _TAG_ONE = _tag_rows("SELECT id,name FROM tags WHERE id=?")
 # Every owner kind a tag can sit on, for the delete confirmation.
 _TAG_IMPACT = (
    "SELECT "
    "(SELECT COUNT(*) FROM prompts p JOIN prompt_tags pt ON pt.prompt_id=p.id "
    "WHERE pt.tag_id=:tag AND " + _ELIGIBLE + ") AS positive_prompts,"
    "(SELECT COUNT(*) FROM prompts p JOIN prompt_tags pt ON pt.prompt_id=p.id "
    "WHERE pt.tag_id=:tag AND NOT " + _ELIGIBLE + ") AS other_prompts,"
    "(SELECT COUNT(*) FROM prompt_pairs pp JOIN prompt_pair_tags ppt "
    "ON ppt.pair_id=pp.id WHERE ppt.tag_id=:tag AND pp.archived=0) "
    "AS negative_variants,"
    "(SELECT COUNT(*) FROM prompt_pairs pp JOIN prompt_pair_tags ppt "
    "ON ppt.pair_id=pp.id WHERE ppt.tag_id=:tag AND pp.archived!=0) "
    "AS archived_negative_variants,"
    "(SELECT COUNT(DISTINCT b.file_id) FROM media_image_tags mit "
    "JOIN media_annotation_bytes b ON b.id=mit.byte_id WHERE mit.tag_id=:tag) "
    "AS images"
 )
 _MAX_ID = 2**63 - 1
 def validate_vocabulary_tag_name(value: object) -> str:
    """Return the trimmed name, applying tag rules plus the native comma ban."""
    if not isinstance(value, str):
        raise ValueError("Tag name must be a string")
    validate_tags([value])
    name = value.strip()
    if "," in name:
        raise ValueError("Tag names cannot contain commas")
    return name
 def _validate_tag_id(tag_id: object) -> None:
    if type(tag_id) is not int or not 1 <= tag_id <= _MAX_ID:
        raise ValueError("Invalid tag ID")
 # The /library/options universe: nonblank values on eligible parents or on
 # active pairs, grouped exactly (whitespace and case variants stay distinct).
 _CATEGORY_GROUPS = (
    "WITH owners(value,pid) AS ("
    "SELECT p.category,p.id FROM prompts p WHERE p.category IS NOT NULL "
    "AND TRIM(p.category)!='' AND " + _ELIGIBLE + " "
    "UNION SELECT pp.category,pp.positive_prompt_id FROM prompt_pairs pp "
    "WHERE pp.archived=0 AND pp.category IS NOT NULL AND TRIM(pp.category)!=''),"
    "grouped AS (SELECT value,MIN(pid) AS first_pid,"
    "COUNT(DISTINCT pid) AS prompt_count FROM owners GROUP BY value) "
 )
 # Every owner of one exact category value, in id order, with the flag the
 # impact counts split on.
 _CATEGORY_COMPONENTS = (
    "SELECT p.id," + _ELIGIBLE + " AS eligible FROM prompts p "
    "WHERE p.category=? ORDER BY p.id"
 )
 _CATEGORY_PAIRS = "SELECT id,archived FROM prompt_pairs WHERE category=? ORDER BY id"
 _CATEGORY_EXISTS = (
    "SELECT EXISTS(SELECT 1 FROM prompts WHERE category=:v)"
    " OR EXISTS(SELECT 1 FROM prompt_pairs WHERE category=:v)"
 )
 # Pairs after components, so a failing pair update rolls both back.
 _CATEGORY_UPDATES = (
    "UPDATE prompts SET category=?,updated_at=? WHERE category=?",
    "UPDATE prompt_pairs SET category=?,updated_at=? WHERE category=?",
 )
 def category_impact_token(
    epoch: str,
    source: str,
    target: str,
    target_exists: bool,
    components: list,
    pairs: list,
 ) -> str:
    """Hash a category projection like ``library_edit_token`` (no secret)."""
    preimage = {
        "epoch": epoch,
        "source": source,
        "target": target,
        "target_exists": bool(target_exists),
        "components": sorted([int(i), bool(flag)] for i, flag in components),
        "pairs": sorted([int(i), bool(flag)] for i, flag in pairs),
    }
    return canonical_token(_CATEGORY_DOMAIN, preimage)
 def _validate_category_change(source: object, target: object) -> str:
    """Check the exact source and return the trimmed, different target."""
    if not isinstance(source, str) or not source.strip():
        raise ValueError("source must be a nonblank category value")
    if not isinstance(target, str):
        raise ValueError("Category must be a string")
    validate_category(target)
    target = target.strip()
    if not target:
        raise ValueError("New category name cannot be blank")
    if target == source:
        raise ValueError("New category name must differ from the current one")
    return target
 def _validate_impact_token(impact_token: object) -> None:
    if not isinstance(impact_token, str) or not _TOKEN.fullmatch(impact_token):
        raise ValueError("impact_token must be 64 lowercase hexadecimal characters")
 def _tag_after(after: object) -> int:
    if after is None:
        return 0
    if type(after) is not int or after <= 0:
        raise ValueError("Invalid vocabulary cursor")
    return after
 def _category_after(after: object) -> tuple[int, str] | None:
    if after is None:
        return None
    if (
        not isinstance(after, list)
        or len(after) != 2
        or type(after[0]) is not int
        or not isinstance(after[1], str)
    ):
        raise ValueError("Invalid vocabulary cursor")
    return after[0], after[1]
 class VocabularyOperations:
    """Vocabulary reads and tag commands.
    ``_`` helpers take an open ``library_snapshot`` or ``_pair_transaction``
    connection.
    """
    def _vocabulary_tag(self, conn: sqlite3.Connection, tag_id: int) -> dict | None:
        row = conn.execute(_TAG_ONE, (tag_id,)).fetchone()
        return dict(row) if row else None
    def _vocabulary_identity(
        self, conn: sqlite3.Connection, epoch: str, tag_id: int, name: str
    ) -> dict:
        """The stored tag, or vocabulary-conflict when it is gone or renamed."""
        tag = self._vocabulary_tag(conn, tag_id)
        if tag is None or tag["name"] != name:
            raise LibraryCommandError("vocabulary-conflict", epoch)
        return tag
    @staticmethod
    def _tag_owner_counts(conn: sqlite3.Connection, tag_id: int) -> dict:
        components = conn.execute(
            "SELECT COUNT(DISTINCT prompt_id) FROM prompt_tags WHERE tag_id=?",
            (tag_id,),
        ).fetchone()[0]
        pairs = conn.execute(
            "SELECT COUNT(DISTINCT pair_id) FROM prompt_pair_tags WHERE tag_id=?",
            (tag_id,),
        ).fetchone()[0]
        images = conn.execute(
            "SELECT COUNT(DISTINCT b.file_id) FROM media_image_tags mit "
            "JOIN media_annotation_bytes b ON b.id=mit.byte_id WHERE mit.tag_id=?",
            (tag_id,),
        ).fetchone()[0]
        return {
            "components_changed": components,
            "pairs_changed": pairs,
            "images_changed": images,
        }
    def create_vocabulary_tag(self, name: str, *, expected_epoch: str) -> dict:
        """Insert an unused registry row, or reuse the exact existing name."""
        _validate_epoch(expected_epoch)
        name = validate_vocabulary_tag_name(name)
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            row = conn.execute("SELECT id FROM tags WHERE name=?", (name,)).fetchone()
            created = row is None
            if created:
                tag_id = conn.execute(
                    "INSERT INTO tags(name) VALUES(?)", (name,)
                ).lastrowid
            else:
                tag_id = row[0]
            return {
                "library_epoch": epoch,
                "created": created,
                "tag": self._vocabulary_tag(conn, tag_id),
            }
    def rename_vocabulary_tag(
        self, tag_id: int, *, expected_epoch: str, name: str, new_name: str
    ) -> dict:
        """Rename ``{tag_id, name}``, merging into an existing exact ``new_name``.
        Links move across components and all pairs, archived included; an
        owner already carrying the target keeps one link.
        """
        _validate_tag_id(tag_id)
        _validate_epoch(expected_epoch)
        if not isinstance(name, str):
            raise ValueError("name must be the tag's current name")
        new_name = validate_vocabulary_tag_name(new_name)
        if new_name == name:
            raise ValueError("New name must differ from the current name")
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            self._vocabulary_identity(conn, epoch, tag_id, name)
            counts = self._tag_owner_counts(conn, tag_id)
            target = conn.execute(
                "SELECT id FROM tags WHERE name=?", (new_name,)
            ).fetchone()
            if target is None:
                self._bump_image_tag_bytes(conn, tag_id)
                conn.execute("UPDATE tags SET name=? WHERE id=?", (new_name, tag_id))
                result_id = tag_id
            else:
                result_id = target[0]
                self._move_tag_links(conn, tag_id, result_id)
                conn.execute("DELETE FROM tags WHERE id=?", (tag_id,))
            return {
                "library_epoch": epoch,
                "merged": target is not None,
                "tag": self._vocabulary_tag(conn, result_id),
                **counts,
            }
    def delete_vocabulary_tag(
        self, tag_id: int, *, expected_epoch: str, name: str
    ) -> dict:
        """Remove the registry row ``{tag_id, name}``; its links cascade."""
        _validate_tag_id(tag_id)
        _validate_epoch(expected_epoch)
        if not isinstance(name, str):
            raise ValueError("name must be the tag's current name")
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            self._vocabulary_identity(conn, epoch, tag_id, name)
            counts = self._tag_owner_counts(conn, tag_id)
            self._bump_image_tag_bytes(conn, tag_id)
            conn.execute("DELETE FROM tags WHERE id=?", (tag_id,))
            return {"library_epoch": epoch, "deleted": True, **counts}
    def vocabulary_tag_impact(self, tag_id: int) -> dict | None:
        """Owner counts by kind for one tag, or None when it does not exist."""
        _validate_tag_id(tag_id)
        with self.library_snapshot() as (conn, epoch, _ceiling):
            if not conn.execute("SELECT 1 FROM tags WHERE id=?", (tag_id,)).fetchone():
                return None
            row = conn.execute(_TAG_IMPACT, {"tag": tag_id}).fetchone()
            return {"library_epoch": epoch, "impact": dict(row)}
    @staticmethod
    def _category_projection(
        conn: sqlite3.Connection, epoch: str, source: str, target: str
    ) -> dict:
        """Owners of ``source``, impact counts and the token over them."""
        components = [
            [row[0], bool(row[1])]
            for row in conn.execute(_CATEGORY_COMPONENTS, (source,))
        ]
        pairs = [
            [row[0], bool(row[1])] for row in conn.execute(_CATEGORY_PAIRS, (source,))
        ]
        if not components and not pairs:
            raise LibraryCommandError("vocabulary-conflict", epoch)
        target_exists = bool(
            conn.execute(_CATEGORY_EXISTS, {"v": target}).fetchone()[0]
        )
        eligible = sum(flag for _id, flag in components)
        archived = sum(flag for _id, flag in pairs)
        return {
            "target_exists": target_exists,
            "impact": {
                "positive_prompts": eligible,
                "other_prompts": len(components) - eligible,
                "negative_variants": len(pairs) - archived,
                "archived_negative_variants": archived,
            },
            "impact_token": category_impact_token(
                epoch, source, target, target_exists, components, pairs
            ),
        }
    def preview_vocabulary_category(
        self, *, expected_epoch: str, source: str, target: str
    ) -> dict:
        """Read-only impact of renaming the exact ``source`` to ``target``."""
        _validate_epoch(expected_epoch)
        target = _validate_category_change(source, target)
        with self.library_snapshot() as (conn, epoch, _ceiling):
            if epoch != expected_epoch:
                raise LibraryCommandError("storage-changed", epoch)
            projection = self._category_projection(conn, epoch, source, target)
            return {
                "library_epoch": epoch,
                "source": source,
                "target": target,
                **projection,
            }
    def apply_vocabulary_category(
        self, *, expected_epoch: str, source: str, target: str, impact_token: str
    ) -> dict:
        """Set the exact ``source`` to ``target`` on every component and pair.
        The recomputed projection must still match the preview's token; a
        mismatch is ``vocabulary-conflict`` with nothing written.
        """
        _validate_epoch(expected_epoch)
        target = _validate_category_change(source, target)
        _validate_impact_token(impact_token)
        with self._pair_transaction() as conn:
            epoch = self._library_epoch(conn, expected_epoch)
            projection = self._category_projection(conn, epoch, source, target)
            if projection["impact_token"] != impact_token:
                raise LibraryCommandError("vocabulary-conflict", epoch)
            stamp = utc_now_iso()
            for statement in _CATEGORY_UPDATES:
                conn.execute(statement, (target, stamp, source))
            return {
                "library_epoch": epoch,
                "source": source,
                "target": target,
                "consolidated": projection["target_exists"],
                "impact": projection["impact"],
            }
    def _vocabulary_tags(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        *,
        limit: int = 100,
        after: int | None = None,
    ) -> dict:
        _limit(limit)
        start = _tag_after(after)
        total = conn.execute("SELECT COUNT(*) FROM tags").fetchone()[0]
        rows = [dict(row) for row in conn.execute(_TAG_PAGE, (start, limit + 1))]
        has_more = len(rows) > limit
        rows = rows[:limit]
        return {
            "library_epoch": epoch,
            "tags": rows,
            "total": total,
            "has_more": has_more,
            "next_key": rows[-1]["id"] if has_more else None,
        }
    def _vocabulary_categories(
        self,
        conn: sqlite3.Connection,
        epoch: str,
        *,
        limit: int = 100,
        after: list | None = None,
    ) -> dict:
        _limit(limit)
        key = _category_after(after)
        total = conn.execute(
            _CATEGORY_GROUPS + "SELECT COUNT(*) FROM grouped"
        ).fetchone()[0]
        where, params = "", []
        if key is not None:
            where = " WHERE first_pid>? OR (first_pid=? AND value>?)"
            params = [key[0], key[0], key[1]]
        rows = conn.execute(
            _CATEGORY_GROUPS
            + "SELECT value,first_pid,prompt_count FROM grouped"
            + where
            + " ORDER BY first_pid,value LIMIT ?",
            [*params, limit + 1],
        ).fetchall()
        has_more = len(rows) > limit
        rows = rows[:limit]
        return {
            "library_epoch": epoch,
            "categories": [
                {"value": row["value"], "prompt_count": row["prompt_count"]}
                for row in rows
            ],
            "total": total,
            "has_more": has_more,
            "next_key": (
                [rows[-1]["first_pid"], rows[-1]["value"]] if has_more else None
            ),
        }
    def list_vocabulary_tags(self, *, limit: int = 100, after: int | None = None):
        with self.library_snapshot() as (conn, epoch, _ceiling):
            return self._vocabulary_tags(conn, epoch, limit=limit, after=after)
    def list_vocabulary_categories(
        self, *, limit: int = 100, after: list | None = None
    ) -> dict:
        with self.library_snapshot() as (conn, epoch, _ceiling):
            return self._vocabulary_categories(conn, epoch, limit=limit, after=after)
@@ -0,0 +1,652 @@
 """Each workflow and prompt graph stored once, compressed (#249 S6 carry-over).
 ``generated_images.workflow_data`` (the UI workflow) and ``prompt_metadata``
 (the API prompt graph) used to hold the same large JSON text again for every
 image of a run. Now each distinct text is one ``workflow_documents`` row:
 - ``sha256``: the SHA-256 of the text's exact UTF-8 bytes (unique key);
 - ``byte_length``: the length of those bytes;
 - ``data``: ``zlib.compress`` of those bytes.
 An image row references its texts through ``workflow_document_id`` and
 ``prompt_document_id``. Two forms coexist until the migration has rewritten
 every row, and both stay readable forever:
 - old form: the text inline in ``workflow_data`` / ``prompt_metadata``, no id;
 - new form: the inline column NULL and the document id set.
 The text is stored and hashed exactly as written, never parsed and
 re-serialized, because a JSON round trip can change bytes (key order,
 spacing, number spelling, duplicate keys, ComfyUI's ``NaN`` tokens).
 ``read_stored_text`` is the one decoder. Inline text, when present, wins (a
 row an older app version rewrote inline stays right); otherwise the document
 is read by its rowid, decompressed within its recorded length and checked
 against its length and hash. A document that fails a check reads as
 unreadable (None, logged), never as other text. ``resolve_image_row`` puts
 the texts back into a ``generated_images`` row and drops the id keys, so
 every payload keeps its shape.
 No trigger: every restore validator refuses triggers in a restored file. A
 document is removed in code: ``release_documents`` in the same transaction
 that drops a row's last reference, and ``sweep_documents`` (at startup, after
 the migration and before Reclaim space) for anything a path missed.
 """
 import hashlib
 import logging
 import re
 import sqlite3
 import zlib
 from collections import OrderedDict
 from typing import Any, Dict, Iterable, Optional, Set
 logger = logging.getLogger("prompt_manager.database.workflow_documents")
 TABLE = "workflow_documents"
 # (inline text column, document id column) on generated_images.
 TEXT_COLUMNS = (
    ("workflow_data", "workflow_document_id"),
    ("prompt_metadata", "prompt_document_id"),
 )
 DOCUMENT_ID_COLUMNS = tuple(reference for _column, reference in TEXT_COLUMNS)
 INDEXES = {
    "workflow_document_id": "idx_image_workflow_document",
    "prompt_document_id": "idx_image_prompt_document",
 }
 COMPRESSION_LEVEL = 6
 # A text larger than this stays inline: a document is decompressed in one
 # piece, and its recorded length bounds that (a restored file cannot make a
 # small row expand without limit).
 MAX_DOCUMENT_BYTES = 256 * 2**20
 ID_BATCH = 400  # ids bound per statement (under SQLite's old 999 limit)
 _DDL = (
    f"CREATE TABLE {TABLE} (id INTEGER PRIMARY KEY, sha256 BLOB NOT NULL UNIQUE, "
    "byte_length INTEGER NOT NULL, data BLOB NOT NULL)"
 )
 _AUTOINDEX = f"sqlite_autoindex_{TABLE}_1"
 _REFERENCE_TYPE = f"INTEGER REFERENCES {TABLE}(id)"
 _UNREFERENCED = (
    "NOT EXISTS (SELECT 1 FROM generated_images g "
    "WHERE g.workflow_document_id=workflow_documents.id) "
    "AND NOT EXISTS (SELECT 1 FROM generated_images g "
    "WHERE g.prompt_document_id=workflow_documents.id)"
 )
 LOOKUP_SQL = (
    "SELECT id, sha256, byte_length, data FROM workflow_documents WHERE sha256=?"
 )
 REPAIR_SQL = "UPDATE workflow_documents SET byte_length=?, data=? WHERE id=?"
 ALL_DOCUMENTS_SQL = "SELECT sha256, byte_length, data, id FROM workflow_documents"
 READ_SQL = "SELECT sha256, byte_length, data FROM workflow_documents WHERE id=?"
 INSERT_SQL = (
    "INSERT INTO workflow_documents (sha256, byte_length, data) VALUES (?, ?, ?)"
 )
 # Built from constants only; the ids are bound as "?" marks.
 RELEASE_SQL = "DELETE FROM workflow_documents WHERE id IN ({marks}) AND "
 RELEASE_SQL += _UNREFERENCED
 # The sweep walks the ids through the small unique index, not the blobs.
 SWEEP_SQL = (
    "DELETE FROM workflow_documents WHERE id IN (SELECT id FROM workflow_documents "
    "INDEXED BY sqlite_autoindex_workflow_documents_1 WHERE "
 )
 SWEEP_SQL += _UNREFERENCED + ")"
 IMAGE_DOCUMENTS_SQL = (
    "SELECT workflow_document_id, prompt_document_id FROM generated_images WHERE "
 )
 # Statements whose query plans tests/test_workflow_documents.py records.
 QUERY_PLAN_STATEMENTS = {
    "lookup by hash": (LOOKUP_SQL, (b"\0" * 32,)),
    "read by id": (READ_SQL, (1,)),
    "release": (RELEASE_SQL.format(marks="?,?"), (1, 2)),
    "sweep": (SWEEP_SQL, ()),
    "documents of images": (IMAGE_DOCUMENTS_SQL + "id IN (?,?)", (1, 2)),
    "documents of prompts": (IMAGE_DOCUMENTS_SQL + "prompt_id IN (?,?)", (1, 2)),
 }
 class DocumentMismatch(Exception):
    """A document would not give back exactly the text it was made from."""
 def _compact(sql: str) -> str:
    return re.sub(r"\s+", "", sql or "").lower()
 # ── schema ────────────────────────────────────────────────────────────
 def _reference_columns(conn: sqlite3.Connection) -> Dict[str, tuple]:
    return {
        row[1]: tuple(row)
        for row in conn.execute("PRAGMA table_info(generated_images)")
        if row[1] in DOCUMENT_ID_COLUMNS
    }
 def _reference_error(conn: sqlite3.Connection, columns: Dict[str, tuple]):
    for name, info in columns.items():
        # (cid, name, type, notnull, default, pk)
        if (str(info[2]).upper(), info[3], info[4], info[5]) != ("INTEGER", 0, None, 0):
            return "Workflow document references have an invalid column"
    keys = {
        (row[3], row[2], row[4])
        for row in conn.execute("PRAGMA foreign_key_list(generated_images)")
        if row[3] in DOCUMENT_ID_COLUMNS
    }
    if keys != {(name, TABLE, "id") for name in DOCUMENT_ID_COLUMNS}:
        return "Workflow document references have invalid foreign keys"
    for column, index in INDEXES.items():
        info = conn.execute(f'PRAGMA index_info("{index}")').fetchall()
        owner = conn.execute(
            "SELECT tbl_name FROM sqlite_master WHERE type='index' AND name=?",
            (index,),
        ).fetchone()
        if [row[2] for row in info] != [column] or not owner:
            return "Workflow document references are missing their index"
        if owner[0] != "generated_images":
            return "Workflow document references are missing their index"
    return None
 def _value_error(conn: sqlite3.Connection):
    if conn.execute(
        "SELECT 1 FROM workflow_documents WHERE typeof(sha256)!='blob' "
        "OR length(sha256)!=32 OR typeof(byte_length)!='integer' OR byte_length<0 "
        "OR byte_length>? OR typeof(data)!='blob' LIMIT 1",
        (MAX_DOCUMENT_BYTES,),
    ).fetchone():
        return "Workflow documents contain invalid values"
    for column in DOCUMENT_ID_COLUMNS:
        # typeof() reads no overflow page; a dangling id is a rowid probe.
        # ``column`` is one of DOCUMENT_ID_COLUMNS, never caller data.
        sql = (
            f"SELECT 1 FROM generated_images g WHERE typeof(g.{column}) "  # nosec B608
            f"NOT IN ('null','integer') OR (g.{column} IS NOT NULL AND NOT EXISTS "
            f"(SELECT 1 FROM workflow_documents d WHERE d.id=g.{column})) LIMIT 1"
        )
        if conn.execute(sql).fetchone():
            return "Workflow document references point at missing documents"
    return None
 def damaged_message(document_id: int) -> str:
    """The verify reason for a damaged document: which one, and what to do."""
    return (
        f"Workflow document {document_id} is damaged: its text no longer matches "
        "its checksum, and the images that use it show no workflow. To fix it, "
        "restore a backup from before the damage, or store a new image with the "
        "same workflow (that repairs it)."
    )
 def _damaged_document(conn: sqlite3.Connection) -> Optional[int]:
    """The first document failing its length or hash check, or None."""
    for row in conn.execute(ALL_DOCUMENTS_SQL):
        if _decode(row[:3], row[3], quiet=True) is None:
            return row[3]
    return None
 def workflow_documents_schema_error(
    conn: sqlite3.Connection, *, allow_missing: bool = True, documents: bool = False
 ) -> Optional[str]:
    """Reject an unsupported document schema or invalid stored values (reads only).
    A database from before the dedupe has neither the table nor the id
    columns; that is acceptable unless ``allow_missing`` is false. Half of
    the schema never is.
    """
    objects = conn.execute(
        "SELECT type,name,sql FROM sqlite_master WHERE name=? "
        "OR (type IN ('index','trigger') AND tbl_name=?)",
        (TABLE, TABLE),
    ).fetchall()
    has_images = conn.execute(
        "SELECT 1 FROM sqlite_master WHERE type='table' AND name='generated_images'"
    ).fetchone()
    columns = _reference_columns(conn) if has_images else {}
    if not objects and not columns:
        if allow_missing or not has_images:
            return None
        return "Missing workflow document schema"
    if not objects or set(columns) != set(DOCUMENT_ID_COLUMNS):
        return "Incomplete workflow document schema"
    tables = [sql for kind, _name, sql in objects if kind == "table"]
    others = {(kind, name, sql) for kind, name, sql in objects if kind != "table"}
    if len(tables) != 1 or others != {("index", _AUTOINDEX, None)}:
        return "Workflow documents name is reserved by an unsupported schema object"
    # The exact definition fixes the columns, their types and constraints.
    if _compact(tables[0]) != _compact(_DDL):
        return "Workflow documents have an invalid table definition"
    error = _reference_error(conn, columns) or _value_error(conn)
    if error is None and documents:
        # Restore and backup verification only: every document is read.
        damaged = _damaged_document(conn)
        if damaged is not None:
            return damaged_message(damaged)
    return error
 def migrate_workflow_documents(conn: sqlite3.Connection) -> None:
    """Install the table, the id columns and their indexes atomically.
    Runs after every older migration (the table rebuilds only run on
    databases without ``file_path``, before this). Validates before and
    after, then sweeps documents no row references.
    """
    conn.execute("SAVEPOINT workflow_documents_schema")
    try:
        error = workflow_documents_schema_error(conn)
        if error:
            raise sqlite3.DatabaseError(error)
        conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
        present = _reference_columns(conn)
        for column in DOCUMENT_ID_COLUMNS:
            if column not in present:
                conn.execute(
                    f"ALTER TABLE generated_images ADD COLUMN {column} "
                    f"{_REFERENCE_TYPE}"
                )
        for column, index in INDEXES.items():
            conn.execute(
                f"CREATE INDEX IF NOT EXISTS {index} ON generated_images({column})"
            )
        error = workflow_documents_schema_error(conn, allow_missing=False)
        if error:
            raise sqlite3.DatabaseError(error)
        sweep_documents(conn)
        conn.execute("RELEASE workflow_documents_schema")
    except BaseException:
        conn.execute("ROLLBACK TO workflow_documents_schema")
        conn.execute("RELEASE workflow_documents_schema")
        raise
 # ── write ─────────────────────────────────────────────────────────────
 def storable(text: Any) -> bool:
    """Whether ``text`` can be a document (text within MAX_DOCUMENT_BYTES)."""
    if not isinstance(text, str):
        return False
    # UTF-8 takes at most 4 bytes a character: most texts need no encoding.
    if len(text) * 4 <= MAX_DOCUMENT_BYTES:
        return True
    return len(text.encode("utf-8")) <= MAX_DOCUMENT_BYTES
 def store_text(
    conn: sqlite3.Connection, text: Optional[str], verified: Optional[set] = None
 ) -> Optional[int]:
    """The id of the document holding exactly ``text``; None for None.
    Reuses the document with the same hash, or inserts a new one whose
    compressed bytes were checked to decompress to ``text``'s bytes. A reused
    document is decompressed and checked first; a damaged one (a restored
    file's corruption under a valid hash) is repaired from ``text``, whose
    hash is its key, so every row that references it reads right again.
    ``verified``, when given, holds the ids already checked (a migration
    batch), which are then not read again. Opens a write transaction first
    when none is open, so a concurrent ``release_documents`` cannot remove a
    reused document before the caller references it. Raises TypeError for a
    non-text value and ValueError for a text over MAX_DOCUMENT_BYTES.
    """
    if text is None:
        return None
    if not isinstance(text, str):
        raise TypeError("Only text is stored as a workflow document")
    raw = text.encode("utf-8")
    if len(raw) > MAX_DOCUMENT_BYTES:
        raise ValueError("Text is too large for a workflow document")
    digest = hashlib.sha256(raw).digest()
    if not conn.in_transaction:
        conn.execute("BEGIN IMMEDIATE")
    row = conn.execute(LOOKUP_SQL, (digest,)).fetchone()
    if row is not None and verified is not None and row[0] in verified:
        return row[0]
    data = zlib.compress(raw, COMPRESSION_LEVEL)
    if zlib.decompress(data) != raw:  # pragma: no cover - zlib is exact
        raise DocumentMismatch("Compression did not round-trip")
    if row is None:
        document_id = conn.execute(INSERT_SQL, (digest, len(raw), data)).lastrowid
    else:
        document_id = row[0]
        if _decode(row[1:], document_id, quiet=True) != text:
            logger.warning(f"Workflow document {document_id} was damaged; repaired")
            conn.execute(REPAIR_SQL, (len(raw), data, document_id))
    if verified is not None:
        verified.add(document_id)
    return document_id
 def stored_values(
    conn: sqlite3.Connection, texts: Dict[str, Optional[str]]
 ) -> Dict[str, Any]:
    """New-form column values for ``{text column: text}`` (writers).
    Each text becomes ``{column: None, id column: document id}``; None
    stays ``{column: None, id column: None}``; a value that cannot be a
    document (not text, or too large) stays inline with no id.
    """
    references = dict(TEXT_COLUMNS)
    values: Dict[str, Any] = {}
    for column, text in texts.items():
        reference = references[column]
        if text is None or not storable(text):
            values[column], values[reference] = text, None
        else:
            values[column], values[reference] = None, store_text(conn, text)
    return values
 # ── read ──────────────────────────────────────────────────────────────
 def _decode(row, document_id, quiet=False) -> Optional[str]:
    """``(sha256, byte_length, data)`` as checked text, or None (logged)."""
    if row is None:
        logger.warning(f"Workflow document {document_id} is missing")
        return None
    digest, length, data = bytes(row[0]), row[1], row[2]
    try:
        if not isinstance(length, int) or not 0 <= length <= MAX_DOCUMENT_BYTES:
            raise ValueError("invalid length")
        inflater = zlib.decompressobj()
        raw = inflater.decompress(bytes(data), length + 1)
        if len(raw) != length or inflater.unconsumed_tail or not inflater.eof:
            raise ValueError("length differs")
        if hashlib.sha256(raw).digest() != digest:
            raise ValueError("hash differs")
        return raw.decode("utf-8")
    except (zlib.error, ValueError, TypeError) as error:
        if not quiet:
            logger.warning(f"Workflow document {document_id} is unreadable: {error}")
        return None
 def read_stored_text(
    conn: sqlite3.Connection, inline: Any, document_id: Optional[int]
 ) -> Any:
    """The stored text of one column: inline when present, else the document.
    Returns None when there is neither, or when the document is missing or
    fails its checks (logged). An inline value is returned unchanged.
    """
    if inline is not None:
        return inline
    if document_id is None:
        return None
    return _decode(conn.execute(READ_SQL, (document_id,)).fetchone(), document_id)
 class DocumentReader:
    """``read_stored_text`` with a small memo for list reads on one connection."""
    MEMO = 32
    def __init__(self, conn: sqlite3.Connection):
        self.conn = conn
        self._memo: "OrderedDict[int, Optional[str]]" = OrderedDict()
    def text(self, inline: Any, document_id: Optional[int]) -> Any:
        if inline is not None or document_id is None:
            return read_stored_text(self.conn, inline, document_id)
        if document_id in self._memo:
            self._memo.move_to_end(document_id)
            return self._memo[document_id]
        text = read_stored_text(self.conn, None, document_id)
        self._memo[document_id] = text
        if len(self._memo) > self.MEMO:
            self._memo.popitem(last=False)
        return text
 def resolve_image_row(reader: DocumentReader, row) -> Dict[str, Any]:
    """A ``generated_images`` row as a dict with its texts read back.
    The id columns are dropped, so a payload built from it has the shape it
    had before the dedupe. Columns the row did not select stay absent.
    """
    data = dict(row)
    for column, reference in TEXT_COLUMNS:
        has_reference = reference in data
        document_id = data.pop(reference, None)
        if column in data or has_reference:
            data[column] = reader.text(data.get(column), document_id)
    return data
 # ── garbage ───────────────────────────────────────────────────────────
 def _chunks(values):
    values = list(values)
    for start in range(0, len(values), ID_BATCH):
        yield values[start : start + ID_BATCH]
 def image_documents(conn: sqlite3.Connection, where: str, params) -> Set[int]:
    """The document ids the image rows matching ``where`` reference.
    ``where`` is a fixed condition written in this package (``id IN (...)``,
    ``prompt_id IN (...)``), never caller data; values go in ``params``.
    """
    ids: Set[int] = set()
    sql = IMAGE_DOCUMENTS_SQL + where
    for row in conn.execute(sql, list(params)):  # nosec B608
        ids.update(value for value in row if value is not None)
    return ids
 def documents_of_rows(
    conn: sqlite3.Connection, column: str, values: Iterable[Any]
 ) -> Set[int]:
    """Document ids of the image rows whose ``column`` is in ``values``.
    ``column`` is ``id`` or ``prompt_id`` (the rows a prompt delete cascades to).
    """
    if column not in ("id", "prompt_id"):
        raise ValueError(f"Unsupported column {column!r}")
    ids: Set[int] = set()
    for chunk in _chunks(values):
        marks = ",".join("?" for _ in chunk)
        ids |= image_documents(conn, f"{column} IN ({marks})", chunk)
    return ids
 def release_documents(conn: sqlite3.Connection, ids: Iterable[Optional[int]]) -> int:
    """Delete the documents in ``ids`` that no image row references now.
    Runs in the caller's transaction, right after the delete or update that
    dropped the references. Returns how many documents were removed.
    """
    removed = 0
    for chunk in _chunks(sorted({i for i in ids if i is not None})):
        marks = ",".join("?" for _ in chunk)
        # Only "?" marks are interpolated.
        sql = RELEASE_SQL.format(marks=marks)
        removed += conn.execute(sql, chunk).rowcount  # nosec B608
    return removed
 def sweep_documents(conn: sqlite3.Connection) -> int:
    """Delete every document no image row references; returns the count."""
    return conn.execute(SWEEP_SQL).rowcount
 # ── repair (a restore's stage) ────────────────────────────────────────
 INLINE_TEXTS_SQL = (
    "SELECT workflow_data, prompt_metadata FROM generated_images "
    "WHERE workflow_data IS NOT NULL OR prompt_metadata IS NOT NULL"
 )
 REKEY_SQL = "UPDATE workflow_documents SET sha256=?, byte_length=? WHERE id=?"
 def _rows(conn: sqlite3.Connection, sql: str, params=()):
    """Each row of ``sql``, read one at a time.
    Iterating a managed connection's cursor prefetches ``ITER_CHUNK`` (128)
    rows, and a document or inline text can be megabytes.
    """
    cursor = conn.execute(sql, params)
    while (row := cursor.fetchone()) is not None:
        yield row
 def _inflated(data) -> Optional[bytes]:
    """The complete text bytes ``data`` inflates to, or None.
    Complete means the zlib stream ends (its Adler-32 checked) within
    MAX_DOCUMENT_BYTES with nothing after it: the body is intact even when
    the row's recorded hash or length is not.
    """
    try:
        inflater = zlib.decompressobj()
        raw = inflater.decompress(bytes(data), MAX_DOCUMENT_BYTES + 1)
        if not inflater.eof or inflater.unconsumed_tail or inflater.unused_data:
            return None
        if len(raw) > MAX_DOCUMENT_BYTES:
            return None
        raw.decode("utf-8")
        return raw
    except (zlib.error, TypeError, UnicodeDecodeError):
        return None
 def _one_column_off(row, raw: bytes) -> bool:
    """Whether only the recorded hash or only the recorded length is wrong.
    The two columns check the body independently. When the body inflates
    completely and one of them still agrees with it, that column vouches for
    the body and the other one is the damaged value. When both disagree, the
    body itself is the suspect (another valid stream), so it is not trusted.
    """
    hash_ok = hashlib.sha256(raw).digest() == bytes(row[0])
    length_ok = row[1] == len(raw)
    return hash_ok != length_ok
 def _inline_copies(conn: sqlite3.Connection, wanted: Dict[bytes, int]):
    """``{document id: text}`` for each wanted hash an inline row still holds."""
    found: Dict[int, str] = {}
    for row in _rows(conn, INLINE_TEXTS_SQL):
        for text in row:
            if not isinstance(text, str) or not storable(text):
                continue
            document_id = wanted.get(hashlib.sha256(text.encode("utf-8")).digest())
            if document_id is not None and document_id not in found:
                found[document_id] = text
        if len(found) == len(wanted):
            break
    return found
 def _rekey(conn: sqlite3.Connection, document_id: int, raw: bytes) -> Optional[int]:
    """Give an intact body its true key, or merge it into the document that has it.
    Returns the holder's id when its own body was damaged and has just been
    rewritten from ``raw`` (it is repaired, never to be cleared), else None.
    """
    digest = hashlib.sha256(raw).digest()
    holder = conn.execute(LOOKUP_SQL, (digest,)).fetchone()
    if holder is None or holder[0] == document_id:
        conn.execute(REKEY_SQL, (digest, len(raw), document_id))
        return None
    for column in DOCUMENT_ID_COLUMNS:
        # ``column`` is one of DOCUMENT_ID_COLUMNS, never caller data.
        conn.execute(
            f"UPDATE generated_images SET {column}=? WHERE {column}=?",  # nosec B608
            (holder[0], document_id),
        )
    conn.execute("DELETE FROM workflow_documents WHERE id=?", (document_id,))
    if _decode(holder[1:], holder[0], quiet=True) is None:
        conn.execute(
            REPAIR_SQL, (len(raw), zlib.compress(raw, COMPRESSION_LEVEL), holder[0])
        )
        return holder[0]
    return None
 def repair_damaged_documents(conn: sqlite3.Connection) -> Dict[str, int]:
    """Repair or clear every document failing its checks; commit (writes).
    For a restore's stage only, never the live library. In order:
    1. A document whose body inflates completely and whose recorded hash or
       length (not both) disagrees is re-keyed from its text, or merged
       into the document that already holds that text (``_one_column_off``).
    2. A damaged body whose exact text an inline row still holds (the same
       SHA-256, its key) is stored again from that text.
    3. Any other damaged document is unreadable for good: its images'
       references are cleared (they already showed no workflow) and the
       document is removed, so the stage verifies.
    Returns ``{repaired, cleared, images}``: documents repaired (1 and 2),
    documents cleared and the distinct image rows whose reference was
    cleared. Rows are read one at a time.
    """
    damaged: Dict[bytes, int] = {}
    intact: Dict[int, bytes] = {}
    for row in _rows(conn, ALL_DOCUMENTS_SQL):
        if _decode(row[:3], row[3], quiet=True) is None:
            raw = _inflated(row[2])
            if raw is not None and _one_column_off(row, raw):
                intact[row[3]] = raw
            else:
                damaged[bytes(row[0])] = row[3]
    result = {"repaired": 0, "cleared": 0, "images": 0}
    if not damaged and not intact:
        return result
    with conn:
        healed = []
        for document_id, raw in intact.items():
            holder = _rekey(conn, document_id, raw)
            if holder is not None and holder in damaged.values():
                # Rewritten from the merged body: repaired, not to be cleared
                # (it now also holds the merged images' references).
                damaged = {k: v for k, v in damaged.items() if v != holder}
                healed.append(holder)
        copies = _inline_copies(conn, damaged) if damaged else {}
        for document_id, text in copies.items():
            raw = text.encode("utf-8")
            data = zlib.compress(raw, COMPRESSION_LEVEL)
            conn.execute(REPAIR_SQL, (len(raw), data, document_id))
        lost = sorted(set(damaged.values()) - set(copies))
        images: Set[int] = set()
        for chunk in _chunks(lost):
            marks = ",".join("?" * len(chunk))
            # ``marks`` is only "?" placeholders; the ids are bound.
            for (image_id,) in _rows(
                conn,
                "SELECT id FROM generated_images WHERE "  # nosec B608
                f"workflow_document_id IN ({marks}) OR prompt_document_id IN ({marks})",
                chunk + chunk,
            ):
                images.add(image_id)
            for column in DOCUMENT_ID_COLUMNS:
                conn.execute(
                    f"UPDATE generated_images SET {column}=NULL "  # nosec B608
                    f"WHERE {column} IN ({marks})",
                    chunk,
                )
            conn.execute(
                f"DELETE FROM workflow_documents WHERE id IN ({marks})",  # nosec B608
                chunk,
            )
    for document_id in list(intact) + healed + list(copies):
        logger.warning(f"Workflow document {document_id} was damaged; repaired")
    if lost:
        logger.warning(
            f"{len(lost)} damaged workflow document(s) could not be repaired; "
            f"{len(images)} image(s) show no workflow"
        )
    result.update(
        repaired=len(intact) + len(healed) + len(copies),
        cleared=len(lost),
        images=len(images),
    )
    return result
@@ -0,0 +1,455 @@
 """The workflow storage migration and Reclaim space (#249 S6 carry-over).
 ``migrate_workflow_storage`` rewrites old-form image rows (workflow and
 prompt graph text inline) to the new form (a ``workflow_documents`` id,
 ``database/workflow_documents.py``) in batches:
 - each batch is one ``BEGIN IMMEDIATE`` transaction over at most
  ``BATCH_ROWS`` rows or about ``BATCH_BYTES`` of text, found by a rowid
  range search (``NEXT_IDS_SQL``) and read one row at a time, so memory is
  bounded by one row plus the batch's distinct documents;
 - each text is stored (or its document reused) and read back through the
  one accessor before its row changes; a text that would not come back
  identical fails the batch (``DocumentMismatch``) and rolls it back whole;
 - a value that is not text (or is over ``MAX_DOCUMENT_BYTES``) stays inline
  and is counted as protected;
 - each row runs in its own savepoint: one that cannot be rewritten (text
  SQLite cannot decode, a text that would not come back) stays inline and
  is listed, and the rest of the batch goes on;
 - a batch ends after ``BATCH_SECONDS``, and the write lock is then left
  free for ``BATCH_PAUSE``, so other writers are never starved.
 A crash or a stop between batches leaves committed batches in the new form
 and the rest in the old form; both read the same, and a rerun continues
 from the first old-form row. Only database rows change: no image or video
 file is opened.
 ``reclaim_space`` is the VACUUM afterwards: it sweeps unreferenced
 documents, rebuilds the file and checkpoints the WAL with TRUNCATE, so the
 freed pages go back to the disk.
 """
 import logging
 import os
 import sqlite3
 import time
 from contextlib import closing
 from .workflow_documents import (
    TEXT_COLUMNS,
    DocumentMismatch,
    read_stored_text,
    release_documents,
    storable,
    store_text,
    sweep_documents,
 )
 OPERATION = "workflow_storage"
 # The first apply's backup: ``<db>.pre-workflow-storage_<stamp>``, outside the
 # pruned ``<db>.backup_*`` pool. Only one is ever made; later applies take a
 # pooled safety backup.
 PRE_MIGRATION_LABEL = "pre-workflow-storage"
 BATCH_ROWS = 200
 BATCH_BYTES = 16 * 2**20  # characters of text rewritten per transaction
 # A batch holds the write lock: it ends after this long, and the loop then
 # leaves the lock free for BATCH_PAUSE, longer than SQLite's longest busy
 # poll (100 ms), so the image monitor and the run counting get in between.
 BATCH_SECONDS = 0.5
 BATCH_PAUSE = 0.15
 def _row_error(error) -> bool:
    """Whether ``error`` is one row's own cause, which keeps only that row inline.
    Text SQLite cannot decode for Python, and a text that would not come
    back identical. Anything else (a full disk, an I/O error, a malformed
    database, a bug) fails the whole batch. The decode error is matched by
    its message: ``sqlite_errorcode`` needs Python 3.11.
    """
    if isinstance(error, (DocumentMismatch, UnicodeError)):
        return True
    return isinstance(error, sqlite3.OperationalError) and str(error).startswith(
        "Could not decode"
    )
 logger = logging.getLogger("prompt_manager.database.workflow_storage")
 _clock = time.monotonic
 # typeof() reads a column's type from the record header: no overflow page.
 NEXT_IDS_SQL = (
    "SELECT id FROM generated_images WHERE id>? AND "
    "(typeof(workflow_data)!='null' OR typeof(prompt_metadata)!='null') "
    "ORDER BY id LIMIT ?"
 )
 ROW_SQL = (
    "SELECT workflow_data, prompt_metadata, workflow_document_id, "
    "prompt_document_id FROM generated_images WHERE id=?"
 )
 UPDATE_SQL = (
    "UPDATE generated_images SET {assignments} WHERE id=?"  # columns are constants
 )
 # One pass over the record headers: rows with inline text, and inline
 # values that are not text (they stay inline).
 INLINE_COUNTS_SQL = (
    "SELECT "
    "SUM(typeof(workflow_data)='text' OR typeof(prompt_metadata)='text'), "
    "SUM(typeof(workflow_data) NOT IN ('null','text')) "
    "+ SUM(typeof(prompt_metadata) NOT IN ('null','text')) "
    "FROM generated_images"
 )
 # octet_length() (SQLite 3.43+) reads no overflow page either.
 INLINE_BYTES_SQL = (
    "SELECT SUM(octet_length(workflow_data)), SUM(octet_length(prompt_metadata)) "
    "FROM generated_images"
 )
 QUERY_PLAN_STATEMENTS = {
    "next old-form rows": (NEXT_IDS_SQL, (0, BATCH_ROWS)),
    "one row": (ROW_SQL, (1,)),
 }
 def _rewrite_row(conn, image_id, row, verified, checked):
    """Store the row's inline texts as documents; returns (changed, protected, size).
    ``verified`` maps document ids already read back in this batch to their
    text, and ``checked`` holds the reused ids ``store_text`` already
    checked, so a document shared by many rows is decompressed once.
    """
    changes, replaced = {}, []
    protected = size = 0
    for index, (column, reference) in enumerate(TEXT_COLUMNS):
        value, current = row[index], row[index + len(TEXT_COLUMNS)]
        if value is None:
            continue
        if not storable(value):
            protected += 1
            continue
        document_id = store_text(conn, value, checked)
        if document_id not in verified:
            verified[document_id] = read_stored_text(conn, None, document_id)
        if verified[document_id] != value:
            raise DocumentMismatch(f"Image {image_id}: {column} would change")
        changes[column], changes[reference] = None, document_id
        if current is not None and current != document_id:
            replaced.append(current)
        size += len(value)
    if changes:
        assignments = ",".join(f"{key}=?" for key in changes)
        conn.execute(
            UPDATE_SQL.format(assignments=assignments),
            [*changes.values(), image_id],
        )
    release_documents(conn, replaced)
    return bool(changes), protected, size
 def migrate_batch(
    conn,
    after_id,
    max_rows=BATCH_ROWS,
    max_bytes=BATCH_BYTES,
    max_seconds=BATCH_SECONDS,
 ):
    """Rewrite the next batch after ``after_id`` in one transaction.
    The batch ends after ``max_rows`` rows, about ``max_bytes`` of text or
    ``max_seconds``, whichever comes first. Each row runs in its own
    savepoint: a row that cannot be rewritten (``_row_error``) is rolled
    back alone, stays inline and is listed in ``errors``; anything else rolls
    the whole batch back and propagates. Returns ``{last_id, rows,
    protected, errors, done}``: the last row looked at, rows rewritten,
    inline values kept (not text), the ids kept by an error and whether
    there was nothing left to look at.
    """
    if conn.in_transaction:
        raise RuntimeError("The migration needs its own transaction")
    started = _clock()
    conn.execute("BEGIN IMMEDIATE")
    try:
        ids = [row[0] for row in conn.execute(NEXT_IDS_SQL, (after_id, max_rows))]
        result = {
            "last_id": after_id,
            "rows": 0,
            "protected": 0,
            "errors": [],
            "done": not ids,
        }
        verified, checked, size = {}, set(), 0
        for image_id in ids:
            conn.execute("SAVEPOINT workflow_row")
            try:
                row = conn.execute(ROW_SQL, (image_id,)).fetchone()
                changed, protected, row_size = _rewrite_row(
                    conn, image_id, row, verified, checked
                )
            except Exception as error:
                if not _row_error(error):
                    raise
                conn.execute("ROLLBACK TO workflow_row")
                conn.execute("RELEASE workflow_row")
                verified.clear()  # the row's documents were rolled back too
                checked.clear()
                logger.warning(f"Image record {image_id} kept inline: {error}")
                result["errors"].append(image_id)
                changed, protected, row_size = False, 0, 0
            else:
                conn.execute("RELEASE workflow_row")
            result["rows"] += int(changed)
            result["protected"] += protected
            result["last_id"] = image_id
            size += row_size
            if size >= max_bytes or _clock() - started >= max_seconds:
                break
        conn.commit()
        return result
    except BaseException:
        conn.rollback()
        raise
 def pre_migration_backups(db_path):
    """The kept pre-migration backups beside ``db_path``, earliest first."""
    db_path = os.path.abspath(db_path)
    prefix = os.path.basename(db_path) + "." + PRE_MIGRATION_LABEL + "_"
    folder = os.path.dirname(db_path)
    try:
        names = os.listdir(folder)
    except OSError:
        return []
    return sorted(
        name
        for name in names
        if name.startswith(prefix) and os.path.isfile(os.path.join(folder, name))
    )
 def _file_bytes(db_path):
    return sum(
        os.path.getsize(path)
        for path in (db_path, db_path + "-wal")
        if os.path.exists(path)
    )
 class WorkflowStorageOperations:
    """PromptDatabase mixin: the workflow storage migration and Reclaim space."""
    def migrate_workflow_storage(
        self,
        *,
        on_batch=None,
        should_stop=None,
        max_rows=BATCH_ROWS,
        max_bytes=BATCH_BYTES,
    ) -> dict:
        """Rewrite every old-form row, batch by batch (blocking).
        ``on_batch(progress)`` runs after each committed batch with the
        running totals ``{rows, protected, batches, last_id}``;
        ``should_stop()`` is asked before each batch. Each committed batch is
        followed by ``BATCH_PAUSE`` with the write lock free. Returns
        ``{rows, protected, batches, stopped, errors, documents_removed}``
        (``errors``: the ids of rows kept inline by an error). Unreferenced
        documents are swept once every row was looked at.
        """
        totals = {"rows": 0, "protected": 0, "batches": 0, "last_id": 0}
        errors = []
        stopped = False
        while True:
            if should_stop is not None and should_stop():
                stopped = True
                break
            batch = self.migrate_workflow_batch(totals["last_id"], max_rows, max_bytes)
            if batch["done"]:
                break
            time.sleep(BATCH_PAUSE)  # leave the write lock to other writers
            errors += batch["errors"]
            totals["rows"] += batch["rows"]
            totals["protected"] += batch["protected"]
            totals["batches"] += 1
            totals["last_id"] = batch["last_id"]
            if on_batch is not None:
                on_batch(dict(totals))
        removed = 0 if stopped else self.sweep_workflow_documents()
        return {
            "rows": totals["rows"],
            "protected": totals["protected"],
            "batches": totals["batches"],
            "stopped": stopped,
            "errors": errors,
            "documents_removed": removed,
        }
    def migrate_workflow_batch(
        self, after_id, max_rows=BATCH_ROWS, max_bytes=BATCH_BYTES
    ) -> dict:
        """One batch on the calling thread's connection (``migrate_batch``)."""
        return migrate_batch(self.model.get_connection(), after_id, max_rows, max_bytes)
    def sweep_workflow_documents(self) -> int:
        """Remove every document no image row references (one transaction)."""
        conn = self.model.get_connection()
        with conn:
            return sweep_documents(conn)
    def workflow_storage_status(self) -> dict:
        """``workflow_storage_counts`` on one read-only snapshot (blocking)."""
        from .maintenance import _read_snapshot
        return _read_snapshot(self.model.db_path, self.workflow_storage_counts)
    def workflow_storage_counts(self, conn) -> dict:
        """``{rows, protected, documents, inline_bytes}`` on ``conn`` (reads only).
        ``inline_bytes`` is None where SQLite has no ``octet_length``.
        """
        rows, protected = conn.execute(INLINE_COUNTS_SQL).fetchone()
        documents = conn.execute("SELECT COUNT(*) FROM workflow_documents").fetchone()
        inline_bytes = None
        if sqlite3.sqlite_version_info >= (3, 43, 0):
            row = conn.execute(INLINE_BYTES_SQL).fetchone()
            inline_bytes = sum(value or 0 for value in row)
        return {
            "rows": rows or 0,
            "protected": protected or 0,
            "documents": documents[0],
            "inline_bytes": inline_bytes,
        }
    def reclaim_space(self) -> dict:
        """Sweep, VACUUM and truncate the WAL (blocking); file bytes before/after.
        The caller holds the registry's ``exclusive`` slot. Raises whatever
        stopped VACUUM; the database is then unchanged.
        """
        db_path = self.model.db_path
        removed = self.sweep_workflow_documents()
        before = _file_bytes(db_path)
        self.model.vacuum_database()
        with closing(sqlite3.connect(db_path)) as checkpoint:
            busy = checkpoint.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0]
        return {
            "before_bytes": before,
            "after_bytes": _file_bytes(db_path),
            "documents_removed": removed,
            "checkpoint_busy": bool(busy),
        }
    # ── preview ───────────────────────────────────────────────────────
    def _plan_workflow_storage(self):
        """The reviewed plan of the migration (one read-only snapshot)."""
        from .maintenance import (
            MaintenanceDraft,
            _Fingerprint,
            _megabytes,
            _plural,
            _read_snapshot,
        )
        def read(conn):
            sizes = [
                conn.execute(f"PRAGMA {name}").fetchone()[0]
                for name in ("page_size", "page_count", "freelist_count")
            ]
            return sizes, self.workflow_storage_counts(conn)
        (page_size, page_count, free_pages), counts = _read_snapshot(
            self.model.db_path, read
        )
        size = page_size * page_count
        details = [
            {
                "label": "Database file",
                "detail": f"{_megabytes(size)}, of which "
                f"{_megabytes(free_pages * page_size)} is free pages",
            },
            {
                "label": "Image records",
                "detail": f"{_plural(counts['rows'], 'record')} keep their workflow "
                "or prompt graph inline; each distinct text is stored once, "
                "compressed",
            },
        ]
        if not counts["rows"]:
            details[1]["detail"] = NOTHING_TO_DO
        if counts["inline_bytes"] is not None:
            details.append(
                {
                    "label": "Inline text",
                    "detail": f"{_megabytes(counts['inline_bytes'])} now",
                }
            )
        if counts["documents"]:
            details.append(
                {
                    "label": "Already stored once",
                    "detail": _plural(counts["documents"], "workflow document"),
                }
            )
        name = os.path.basename(self.model.db_path)
        kept = pre_migration_backups(self.model.db_path)
        backup = (
            POOLED_BACKUP_NOTE.format(
                name=f"{name}.backup_<date and time>", kept=kept[0]
            )
            if kept
            else BACKUP_NOTE.format(
                name=f"{name}.{PRE_MIGRATION_LABEL}_<date and time>"
            )
        )
        details += [
            {"label": "Backup", "detail": backup},
            {"label": "Reclaim space", "detail": RECLAIM_NOTE},
            {"label": "Older versions", "detail": OLDER_VERSIONS_NOTE},
        ]
        protected = counts["protected"]
        return MaintenanceDraft(
            operation=OPERATION,
            fingerprint=_Fingerprint(OPERATION).hexdigest(),
            targets=[],
            scope={
                "count": counts["rows"],
                "label": (
                    _plural(counts["rows"], "image record") + " to rewrite"
                    if counts["rows"]
                    else "Nothing to rewrite"
                ),
            },
            protected={
                "count": protected,
                "reasons": (
                    [f"{_plural(protected, 'stored value')} not text; kept as is"]
                    if protected
                    else []
                ),
            },
            media={"deletes_files": False, "label": MEDIA_NOTE},
            details=details,
        )
 MEDIA_NOTE = "No image or video file is opened or changed; only the database."
 RECLAIM_NOTE = (
    "When every record is rewritten, Reclaim space runs alone to give the freed "
    "space back to the disk (VACUUM). The database is locked while it runs."
 )
 BACKUP_NOTE = (
    "Always taken first, whatever the safety backup setting: {name} beside "
    "the database, verified, and kept until you remove it (it is never pruned)."
 )
 POOLED_BACKUP_NOTE = (
    "A safety backup is taken first, whatever the safety backup setting: {name} "
    "among the pooled backups; the pre-migration backup {kept} is already kept."
 )
 NOTHING_TO_DO = "Every image record already stores its workflow once."
 OLDER_VERSIONS_NOTE = (
    "This version stores the workflows of images it records in the new form, "
    "whether or not this job runs, and this job rewrites the rest. An older "
    "Prompt Manager version shows those images without their workflow until you "
    "update again. The way back to an older version is the earliest "
    "pre-workflow-storage backup."
 )
@@ -0,0 +1,85 @@
 # Application Redesign Development Plan
 The maintainer approved the [interactive concept](concepts/workspace/README.md)
 on 2026-10-17. It is a design reference with simulated operations, not production
 code. This plan proposes implementation order and acceptance gates.
 ## Delivery status
 - Foundation merged in [#198](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/198).
 - [#200](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/200) delivers native Images browsing, physical-file details, infinite scroll, Grid/List/Masonry, Viewer.js and video playback. The workspace opens on Images; original application entry points remain available.
 - Shared role-aware sending and the physical-file prompt inspector merged in [#203](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/203) and [#206](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/206).
 - The native Prompt Library merged in [#210](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/210) for [#208](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/208): one positive parent, saved negative variants, full selected text and a shared filtered image gallery. It does not invent chronological runs from counters or scan dates.
 - [Durable history storage](prompt-history-storage.md) records job attempts and outcomes from [#212](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/212), frozen prompt contexts from [#213](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/213), and observed run-owned output locators from [#214](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/214). Those locators do not verify file bytes or establish gallery-image associations. That association contract and the chronological execution timeline remain pending under [#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209).
 - [#215](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/215) brings the shell, Images and natural-height Prompt Library presentation into the approved geometry and themes while retaining their existing operations. [#216](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/216) adds the shared current-file inspector with exact generation facts, scoped provenance, raw/download evidence and loaded-collection viewer controls, including a slideshow that pauses at native video. This delivery does not establish whole-application parity, change the default entry point or constitute a release. Recorded chronological run/context browsing continues under [#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209); image tagging/review, System consolidation and remaining organization capabilities continue under #196.
 - Local Inspect (#249, PRs L1–L2) replaces the `#/inspect` handoff with the concept's native "Inspect a file" page: one local image or video previews inline and opens the shared viewer and full inspector on a record read entirely in the browser, with nothing uploaded, imported or stored. The legacy metadata page (`web/metadata.html`, served at `/prompt_manager/gallery.html`) stays reachable from Overview; its capabilities map to the shared inspector, except its one-click COPY ALL summary, whose ruled replacement ("Copy settings summary") is pending. Recorded deviations D60–D66 are in [TESTING.md](../TESTING.md). Local roles equal to the server's resolver follow in PR L3.
 ## Scope and compatibility
 Rebuild the standalone application around one shared shell. Preserve existing
 node schemas, saved workflows, backend capabilities, pinned Viewer.js and the
 ComfyAssets toolkit. Keep existing application entry points usable during the
 transition. Do not change the running maintainer instance during automated tests.
 Use the concept's feature inventory as a parity checklist. Each row needs an
 implementation issue, observable acceptance criteria and automated test coverage.
 Record any intentional behavior change explicitly; no silent feature removal.
 ## Proposed delivery sequence
 Use a tracking issue and small issue-linked branches and PRs. Dependencies may
 require splitting these milestones further.
 | Milestone | Deliverable and acceptance focus |
 | --- | --- |
 | 1. Foundation | Production shell, routing, shared theme and disposable page-test harness. Verify startup, navigation, dark/light changes, cleanup and existing entry points. Establish discovered-test and coverage baselines before changes. |
 | 2. Image workspace | Real API-backed filters, Grid/List/Masonry, readable thumbnails, selection and shared Viewer.js inspector. Continuous infinite scroll only, with accessible Load more fallback, stable ordering, retry and restored browsing context. Bound retained DOM and test large datasets. |
 | 3. Prompt library and history | Real positive/negative associations, selected full negative, run history and shared image filtering. Design any missing execution-history persistence and migration before wiring the timeline; never invent historical runs from incomplete metadata. |
 | 4. Send to ComfyUI | Shared paper-plane actions wherever prompt text appears; role-specific and pair sends, destination preview, acknowledgement and failure recovery. Cover explicit empty versus unknown negatives and connected inputs. Never queue or replace a workflow. |
 | 5. Editing and review | Prompt editing, vocabulary management, per-image manual/auto-tag approval and Needs review. Preserve cancellation semantics and distinguish tag approval from prompt-association decisions. |
 | 6. System and maintenance | Settings, sources/scans, integrations, jobs, diagnostics, backup/restore and maintenance impact previews. Cover existing capabilities before retiring their old controls. |
 | 7. Additional organization | Favorites independent of ratings, collections, saved views, Activity, scheduled backups and portable import previews. Specify persistence, validation and recovery separately for each capability. |
 | 8. Cutover | Complete the parity checklist, migration and upgrade validation, accessibility/performance checks, documentation and targeted maintainer live testing. Switch the default only after these pass; preserve compatible URLs. |
 ## TDD and test layers
 For each behavior, first write and run a meaningful failing test. Implement the
 smallest working change, verify it passes, then refactor with the suite green.
 Include failing-before/passing-after evidence in the PR.
 - **Unit:** filtering and selection state, cursor handling, prompt roles,
  formatting, validation, navigation and lifecycle cleanup.
 - **Integration:** real HTTP handlers and disposable SQLite/filesystem fixtures;
  persistence, migration from supported old schemas, idempotent scans, restore,
  rollback and reference-safe deletion. Mock inference and external services.
 - **Smoke:** shipped production pages start and navigate with real fixture APIs;
  test populated, empty and error states, assets and unexpected console failures.
 - **Browser workflows:** production HTML/JS across Chromium, Firefox, WebKit and
  touch; editing/reload, infinite append, viewer cleanup, tagging approval,
  keyboard access, delayed responses, cancellation and retries.
 - **Host integration:** pinned disposable CPU ComfyUI, Classic and Nodes 2.0,
  switching both ways and workflow reload for host-facing changes. Exercise
  independent and combined pack installations for shared bridge/toolkit changes.
 Retain existing test discovery and coverage gates. Carry forward database
 injection and open-handle checks into new fixtures. Add new frontend modules to
 coverage measurement and its baseline; do not lower floors or replace existing
 regression coverage with mockup-only tests.
 ## GitHub and release gates
 Extend existing required workflows: the Python 3.10–3.13 matrix on Linux,
 Windows and macOS; 85% Python coverage; formatting; frontend units and per-file
 coverage floors; vendor checks; browser workflows; real ComfyUI compatibility.
 Publish traces, screenshots, logs and coverage artifacts on failures. See
 [TESTING.md](../TESTING.md) for commands and fixture isolation requirements.
 Every PR must map acceptance criteria to tests, report relevant checks and include
 screenshots for visual changes. All required checks must pass before merging.
 Use maintainer live review at meaningful visual and host-integration milestones;
 capture discovered regressions in automated tests. Notify the maintainer when
 Python changes require a ComfyUI restart.
 Keep version/publication changes separate from intermediate UI work: the existing
 registry workflow publishes on main/master changes to `pyproject.toml` or manual
 dispatch. Do not trigger a release merely to preview a redesign milestone.
@@ -0,0 +1,138 @@
 # Prompt Manager application redesign concept
 Status: visual concept approved by the maintainer, 2026-10-17. Production
 implementation is progressing in tested slices; see the [development plan](../../application-redesign.md).
 The production workspace now has native Images browsing; this concept remains the
 approved target for the complete application.
 ## Review the mockup
 Run `python tools/preview-prompt-timeline.py --bind 0.0.0.0 --port 8767` and
 open `/workspace/`. The original approved timeline remains available at `/`.
 The server exposes an explicit allowlist of concept files and pinned assets.
 All examples are synthetic; there are no application API calls, database access,
 real model downloads, or ComfyUI handoffs. Local file inspection previews an
 image in the browser without uploading it. Exports contain sample data only.
 The sidebar opens Images, Prompt library, Tags & categories, Inspect a file,
 Activity, System and Design review. System includes Overview, Appearance,
 Sources & scans, Integrations, Auto-tagging, Maintenance, Backup & export, and
 Diagnostics & logs. The in-app Design review contains the feature map below.
 ## Proposed direction
 One application shell replaces separate Admin, Gallery and Metadata shells.
 Use the approved shared ComfyAssets theme: ComfyUI dark/light, blue primary
 actions, green positive roles, red negative roles and errors. Preserve existing
 capabilities and services while consolidating controls and clarifying their scope.
 Browsing is image-led with readable thumbnails and one metadata inspector.
 Prompt library groups by positive; negative variants and runs share its gallery.
 The full selected negative appears directly beneath the positive in history.
 Favorites express preference independently of the existing quality rating.
 Collections organize projects without duplicating recipe text; saved views keep
 search/filter context. New capabilities are proposals, not implemented features.
 Maintenance moves out of the browsing toolbar. Checks, impact preview, backups,
 and apply steps have separate responsibilities. Distinguish prompt text reuse
 from image-file duplicates, and metadata enrichment from thumbnail generation.
 No repeated global thumbnail-generation prompt on opening the gallery.
 ## Send to ComfyUI
 Use the same original paper-plane outline, blue accent and accessible role label
 on every positive/negative prompt block, including image details and the viewer.
 A pair action sends both. Preview destination and exact fields before applying;
 positive-only and negative-only actions leave the other role untouched. An
 explicitly empty negative means clear the target field; unknown metadata must
 never be interpreted as empty. Sending text neither queues execution nor
 replaces a workflow.
 Production needs a shared host bridge with versioned payloads, origin validation,
 destination discovery, capability checks, disconnected-host handling, linked-input
 checks and acknowledgements. Preview actual target values and report success only
 after acknowledgement. Default to asking for a destination; never guess between
 multiple nodes or positive and negative encoder inputs. Verify Classic and Nodes
 2.0 with both packs installed independently and together.
 ## Existing feature inventory
 | Current feature | Proposed home / behavior | Disposition |
 | --- | --- | --- |
 | Admin prompt cards / per-prompt gallery | Prompt library + one prompt history workspace | Retained, rebuilt |
 | Standalone image gallery / Admin image modal | One Images workspace + shared Viewer.js viewer | Consolidated |
 | Video playback, autoplay / mute / loop, viewer slideshow and transforms | Shared media viewer; playback controls in Appearance; same image navigation | Retained |
 | Metadata page / viewer sidebar / full-text popups | One reusable metadata inspector; inline complete prompt text | Consolidated |
 | Prompt add, edit, rating, notes, copy, delete | Recipe editor, shared details, contextual bulk actions; reference-safe archive | Retained, clarified |
 | Categories, tag rename / merge, autocomplete | Tags & categories; same vocabulary in every editor | Retained |
 | Grid/list, thumbnail size, columns, pages/infinite scroll | Infinite scroll only; one Appearance configuration and contextual layout / density overrides | Simplified |
 | Scan, rescan, monitoring directories, worker threads | Sources & scans; explicit metadata enrichment vs thumbnail work | Consolidated |
 | Ollama, HuggingFace, WD14 variants, thresholds, review | One Auto-tagging configuration and review flow | Retained |
 | Thumbnail quality, size limit, metadata cache, paths | Appearance + cache controls in Maintenance | Retained |
 | Prompt dedup, hash checks, consistency, unused prompts | Maintenance plan with dependency checks and impact preview | Retained, safer flow |
 | Image dedup, orphan image links, browser cache | Separate reviewed operations in Maintenance | Retained |
 | Database backup / restore, JSON export | Backup & export; recovery and portable recipes clearly distinguished | Retained |
 | Diagnostics, image-link checks, logs, rotation, log levels | System → Diagnostics & logs | Retained |
 | LoRA detection/import, trigger injection, CivitAI key | System → Integrations; positive-only triggers | Retained |
 | Statistics / counts | System overview + usage breakdown; context counts in the workspace | Retained |
 | Send prompts to ComfyUI from every prompt surface | Role-aware paper-plane action + destination / overwrite preview | New proposal |
 | Reusable negatives / approved branch timeline | Prompt history with selected negative directly beneath positive | Approved concept |
 | Favorites, saved searches, project collections | Sidebar views; collections do not duplicate recipe text | New proposal |
 | Ambiguous metadata review, source provenance, run comparisons | Needs review queue; recorded vs inferred evidence; pair/run selection | New / expanded |
 | Unified jobs, cancellation, resumable progress visibility | Activity center; cancellation distinguished from hiding progress | New proposal |
 | Scheduled backups, pre-change safety snapshots | Explicit recovery policy after storage configuration | New proposal |
 ## Implementation boundaries and test requirements
 Keep Viewer.js 1.15.2 and the shared toolkit as reproducible pinned artifacts;
 this mock uses their actual checked-in files without modifying them. Preserve
 legacy routes during the transition, current node layouts/schemas, existing
 controllers where reusable, and all existing test/coverage gates.
 The timeline needs persisted execution records to represent real runs. Saved
 views, collections, metadata review resolution, activity history, scheduled
 backups, portable JSON import, and the ComfyUI bridge need separate implementation
 and persistence work. Preserve source evidence and distinguish recorded from
 inferred metadata. Existing schema migration is automatic; old image enrichment
 remains an optional explicit scan. The one-time upgrade notice belongs in the
 new shell, with Scan and Later choices.
 Implement in reviewed slices: shared application shell and inspector; reusable
 prompt blocks/send bridge; image workspace; grouped library/history; settings and
 maintenance; optional organization and workflow improvements. Write regressions
 first for production behavior. Retain Python, frontend unit, Chromium/Firefox/
 WebKit/touch and managed Classic/Nodes 2.0 tests. Add behavior tests for shared
 filters, navigation state, viewer cleanup, interrupted jobs, dependency-safe
 maintenance, backup/restore, paired metadata and correct send destinations.
 Screenshots in `screenshots/` record the reviewed starting point. The approved
 standalone timeline in `../prompt-variant-timeline.html` remains unchanged.
 ## Review polish
 History actions use a shared branch icon. Images can switch among Grid, List and
 Masonry; Masonry preserves each image's proportions and keeps DOM order while
 packing cards with a responsive layout. It does not require experimental native
 masonry support. Resize observers are disconnected when leaving the view.
 Every image card includes a tagging action beside Info, including cards in Needs
 review. Choose WD14 SwinV2, WD14 ViT, Ollama or HuggingFace; adjust thresholds or
 instructions, generate sample suggestions, edit tags manually, and approve only
 the desired suggestions. Changes are staged until approval, apply to that image
 alone and do not resolve uncertain prompt associations. Changing models discards
 previous suggestions. Cancel preserves existing tags. All inference is simulated.
 ## Continuous browsing decision
 The redesign uses infinite scroll only. Remove page numbers, previous/next page
 buttons, paged/infinite mode selection and results-per-page controls. Keep request
 batching internal. The mock appends 12 images at a time in Grid, List and Masonry;
 it preserves existing card nodes, focus and selection while appending. It also
 retains the loaded range when switching layouts or revisiting a search. A
 keyboard-accessible Load more action provides an alternate trigger for the same
 continuous stream, not pagination. Exhausted and empty results are explicit.
 Production will need bounded rendering, stable cursors, scroll restoration and
 loading/error/retry behavior for large collections. Existing backend pagination
 can continue providing internal request batches; no production tests or routes
 are removed as part of this prototype.
@@ -0,0 +1,105 @@
 <!doctype html>
 <html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width,initial-scale=1" />
    <meta name="referrer" content="no-referrer" />
    <title>Images · Prompt Manager concept</title>
    <link rel="stylesheet" href="/toolkit/controls.css" />
    <link rel="stylesheet" href="/vendor/viewer.css" />
    <link rel="stylesheet" href="/workspace/style.css" />
    <script src="/vendor/viewer.js" defer></script>
    <script src="/workspace/app.js" type="module"></script>
  </head>
  <body class="ca-tools">
    <a class="skip" href="#main">Skip to workspace</a>
    <div class="app-shell">
      <aside class="sidebar">
        <a class="brand" href="#images"
          ><span class="brand-mark">P<span>±</span></span
          ><span>Prompt Manager<small>COMFYASSETS</small></span></a
        >
        <div class="sidebar-label">WORKSPACE</div>
        <nav id="primary-nav" aria-label="Workspace"></nav>
        <div class="sidebar-label">
          SAVED VIEWS <span class="new-label">PROPOSED</span>
        </div>
        <nav id="saved-nav" aria-label="Saved views"></nav>
        <div class="sidebar-label">
          COLLECTIONS <span class="new-label">PROPOSED</span>
        </div>
        <nav id="collections-nav" aria-label="Collections"></nav>
        <div class="sidebar-bottom">
          <nav id="system-nav" aria-label="Tools and system"></nav>
          <div class="connection">
            <i></i>
            <div>
              ComfyUI workspace<small
                >Demo connection · Classic / Nodes 2.0</small
              >
            </div>
          </div>
        </div>
      </aside>
      <div class="workspace">
        <header class="topbar">
          <div class="breadcrumbs">
            ComfyAssets <span>/</span> <span id="breadcrumb">Images</span>
          </div>
          <div class="top-actions">
            <span class="concept-badge">Interactive concept</span
            ><button id="activity-open" type="button">
              Activity <span class="count" id="activity-count">2</span></button
            ><label class="theme-label" for="theme"
              >Theme
              <select id="theme" aria-label="Theme">
                <option value="dark">Dark</option>
                <option value="light">Light</option>
                <option value="system">System</option>
              </select></label
            ><a class="canvas-link" href="#send"
              >ComfyUI <span aria-hidden="true">↗</span></a
            >
          </div>
        </header>
        <main id="main" tabindex="-1"></main>
        <footer class="app-footer">
          <span
            >Design prototype · sample images and data · actions stay in this
            tab</span
          ><button type="button" id="review-link">What changes & why</button>
        </footer>
      </div>
    </div>
    <div id="toast" class="toast" role="status" aria-atomic="true"></div>
    <dialog id="modal" aria-labelledby="modal-title">
      <div class="dialog-header">
        <h2 id="modal-title"></h2>
        <button type="button" id="modal-close" aria-label="Close dialog">
          ×
        </button>
      </div>
      <div id="modal-body"></div>
    </dialog>
    <dialog id="viewer-dialog" aria-labelledby="viewer-title">
      <div class="dialog-header">
        <div>
          <h2 id="viewer-title">Image details</h2>
          <p class="muted small">
            One viewer · image, prompts, generation data, workflow
          </p>
        </div>
        <button type="button" id="viewer-close" aria-label="Close image viewer">
          ×
        </button>
      </div>
      <div class="viewer-layout">
        <div class="viewer-left">
          <div id="viewer-stage"></div>
          <div class="viewer-toolbar" id="viewer-toolbar"></div>
        </div>
        <aside id="viewer-details" aria-label="Image metadata"></aside>
      </div>
    </dialog>
  </body>
 </html>
@@ -0,0 +1,84 @@
 # Durable Prompt History
 Status: the durable job ledger, immutable prompt contexts and run-owned output
 references are implemented for
 [#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Gallery
 image associations and the chronological timeline remain pending; #209 stays open.
 ## Recorded jobs
 New jobs observed at the ComfyUI queue's dequeue boundary receive an immutable
 run identity and a host-session/queue-item identity. The public ComfyUI job UUID
 is a correlation field: submitting it twice records two attempts. Rejected graphs
 and jobs removed from the queue before dequeue create no execution rows. Cached
 jobs are recorded even when no PromptManager node runs, and headless jobs do not
 require a browser or WebSocket client.
 The additive `execution_runs` table stores dequeue time, observed callback time,
 optional executor event times and an explicit outcome: `observed`, `succeeded`,
 `failed`, `interrupted` or `unknown`. Matching stored terminal events determine
 outcomes. Missing or conflicting terminal evidence remains unknown; a process
 crash can leave an observed row without an invented completion time. Bounded
 safe diagnostics retain node ID/type and exception type, never graph text,
 arbitrary extra data, credentials, exception messages, traceback, inputs or
 arbitrary output payloads. A bounded background writer keeps storage failures from
 changing host execution and reports capture gaps rather than inventing rows.
 Existing databases begin with zero execution rows. Migration and backup restore
 preserve prompts, pairs, image links and aggregate counters; storage epochs keep
 stale queued writes from being replayed into restored data. There is no
 historical backfill from counters, scan dates, image timestamps or saved recipes.
 [Testing](../TESTING.md#durable-execution-ledger) describes the real CPU host
 matrix, historical failure proof, privacy checks and cleanup. Python deployment
 requires restarting ComfyUI.
 ## What the existing UI proves
 `prompts.run_count` and `prompt_pairs.run_count` remain aggregate usage counters,
 not completed-run counts. `last_used_at` is the latest aggregate update; legacy
 migration may have estimated counts from linked images. `generated_images`
 identifies linked files; image counts and insertion times are not run counts or
 execution times. Image/pair association creation time can reflect an old-image
 scan. A saved pair's creation time is its recipe date.
 The native Prompt Library shows saved negative variants, linked images and
 aggregate uses. It does not expose the ledger as chronological dots or repeat-run
 branches. `PromptTracker` identities and PNG recipe evidence remain separate
 from durable host job identity.
 ## Immutable contexts and output references
 Prompt contexts retain bounded raw/effective positive, negative and composition
 snapshots with nullable recipe navigation hints and explicit evidence provenance.
 Unknown text remains SQL NULL; a known empty negative remains empty text. Cached
 evidence belongs to its exact cache provider and attempt, rather than inheriting
 unrelated recent prompts. Deletion and restore can remove navigation hints without
 rewriting historical snapshots or rebinding them to current library rows.
 Schema v3 adds `execution_output_capture` and `execution_run_outputs` without
 backfilling older runs or modifying gallery image links. At final queue completion,
 the writer attaches bounded output evidence from that attempt's final host history
 to its immutable run ID. The internal `get_execution_outputs(run_id)` reader
 returns state, reason, format version, source and ordered references. A complete
 capture with `none_reported` means the final host history reported no images;
 `not_captured`, partial, unavailable and conflicting evidence remain explicit.
 Failed and interrupted jobs can still have complete evidence for images reported
 before termination. Running jobs do not gain a final capture merely because an
 image node finished.
 Each reference records output node ID, image ordinal, folder type, subfolder,
 filename, path style and an opaque configured-root key. A cached repeat can retain
 the same locators under a new run. Locators do not verify file bytes, continued
 existence, a newly generated file or membership in any prompt context: the reader
 reports file identity as `unverified`, generation and context membership as
 `unknown`, and temp outputs as transient. Capture never reads, resolves or serves
 the files and adds no public route. PNG metadata is not required. Same-path file
 replacement and existing gallery provenance retain their separate semantics.
 ## Remaining history work
 Define a deduplicated run/gallery-image relationship that preserves existing image
 provenance and same-path replacement protection. Add a chronological timeline only
 after its navigation and evidence semantics can honestly represent prompt-level
 execution chronology, batched outputs and repeat-run branches.
@@ -0,0 +1,40 @@
 # Prompt inspection and sending
 The opt-in workspace at `/prompt_manager/workspace#/images` shows full positive
 and negative prompts in both Image details and the full-image preview. Existing
 gallery and metadata tools remain available during the application rebuild.
 Each prompt shows its evidence source: recorded execution, workflow inference,
 or embedded generation parameters. Path-only saved-library associations are
 marked unverified and cannot be sent: a filename alone cannot prove that an image
 has not been replaced. Missing, ambiguous, and unreadable prompts remain visible
 as distinct states. An explicitly
 empty negative is a resolved value and can clear a destination's negative field.
 Use the blue paper-plane action beside a prompt, or **Send prompt pair**. Keep a
 ComfyUI tab open under the same origin and deployment prefix. Choose a window and
 compatible Prompt Manager node, select **Preview changes**, review the complete
 before/after text and preserved prepend/append composition, then **Apply to
 ComfyUI**. The dialog reports success only after a matching acknowledgement.
 It does not load workflows or queue generation. If confirmation is lost, inspect
 ComfyUI and preview again before another attempt. Closing the dialog cannot undo
 an update already sent.
 ## Read-only API
 `GET /prompt_manager/images/inspect/{relative_path}?root={root_index}` requires
 an explicit configured root, including when two roots contain the same filename.
 The response contains `image` identity and `prompts.positive` / `prompts.negative`.
 Each role has `status` (`resolved`, `absent`, `unresolved`, or `ambiguous`), `text`
 (string only when resolved, otherwise null), and `source`. Inspection does not
 index files, alter saved prompts, run scans, or decode image pixels. PNG text and
 video probe output are bounded; unsupported or malformed evidence remains
 unresolved rather than becoming sendable placeholder text. Videos are read from
 one container carrier: the per-key tags written by core SaveVideo/SaveWEBM and
 current VideoHelperSuite (`prompt`, `workflow`, `parameters`,
 `comfyassets_prompt_pairs`), else the old VideoHelperSuite `comment` envelope,
 else `description`. The admin scan reads the same carrier, so Images and
 ingestion agree on a video's prompts (see TESTING.md for the order and budgets;
 the live monitor processes images only).
 Python route changes require restarting ComfyUI. Browser assets require a refresh.
@@ -0,0 +1,49 @@
 # Prompt sending bridge
 Issue #202 establishes the transport and host adapter before the redesigned
 inspector displays prompt text. Gallery paper-plane controls and normalized
 metadata follow in a separate slice; existing node layouts are unchanged.
 ## Shared transport
 PromptManager consumes the exact Toolkit 0.2.0 artifact recorded in
 `web/js/shared/toolkit/provenance.json`. The shared module has no ComfyUI imports.
 Its client discovers explicit host sessions and compatible destinations, requests
 a preview, and applies that preview using a one-use, client-bound, expiring ticket.
 Success requires an acknowledgement with verified destination values.
 BroadcastChannel connects independently opened tabs on the same origin and storage
 partition. Deployment prefixes have separate channel namespaces. Exact origin,
 protocol and peer checks reject unrelated messages. Same-origin scripts are
 trusted; this transport is not authentication against other scripts on that origin.
 Unsupported messaging is reported without breaking existing ComfyUI nodes.
 An apply timeout or abort after posting means the outcome is unknown. It cannot
 prove the host cancelled the write. The sender must inspect a fresh destination
 preview before deciding to send again; automatic retry must not overwrite edits.
 ## PromptManager adapter
 Classic, Text, Focused and Split accept positive text; +/- accepts either role or
 both. Plain CLIP encoders are not assigned roles by title or position. An omitted
 role remains untouched; an explicitly empty string clears only the selected
 field. Unknown/ambiguous metadata must not become an empty string.
 Previews show original and requested text plus positive prepend/append composition.
 Connected composition is marked unknown rather than guessing upstream values.
 Connected destination fields reject writes. Applying rechecks graph, node, widgets,
 values and connections. Ordinary serialization cache synchronization is accepted
 only when it preserves live values. Both requested roles are checked before writing.
 Widget values, visible inputs, serialization and composed previews are synchronized.
 Callback failures restore local fields; arbitrary external callback side effects
 cannot be undone. Sending never queues execution or loads/replaces a workflow.
 ## Lifecycle and validation
 Page hiding closes the host and invalidates tickets; restored pages mount one fresh
 host. Client disposal clears pending timers/listeners and closes its channel.
 Shared protocol contracts run against the actual bundled file, alongside adapter
 units, cross-browser tab workflows and real Classic/Nodes 2.0 scenarios. Pinned
 PromptManager-only, KikoTools-only and combined disposable hosts verify coexistence
 and workflow reload without modifying either live installation.
@@ -0,0 +1,111 @@
 # Reusable positive and negative prompts
 Tracking: [#193](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/193).
 This work builds on the merged Split node in PR #192. Keep the feature unmerged
 until maintainer live testing is complete.
 ## Storage contract
 Keep existing prompt IDs, text hashes, metadata, image links and node schemas.
 A component can serve either role. Ordered combinations refer to shared prompt
 rows; an explicitly empty negative uses a null reference. Missing or ambiguous
 metadata does not mean an empty negative. Pair metadata belongs to the
 combination; saving another combination does not overwrite component metadata.
 Existing trim/lowercase hash normalization remains unchanged.
 Saved components are immutable while referenced. To change text, create or reuse
 a component and save a new combination. Archive combinations that have generation
 history. Pruning and legacy editing/deletion must respect these dependencies.
 Historical workflow snapshots remain intact, including their original text.
 ## Migration and recovery
 Create pair tables after existing legacy rebuild migrations, in an atomic,
 repeatable migration. Never scan media during schema initialization. Test fresh,
 legacy and current databases, repeated startup, injected failures and retries.
 Preserve old backup compatibility. Migrate and validate a disposable backup copy
 before replacing live database pages. Reject broken pair schemas, foreign keys,
 triggers and views. Use SQLite backup APIs so committed WAL data is included.
 ## Metadata and scanning
 Resolve positive and negative from the same sampler/output context. Track both
 node ID and output slot; bound graph traversal and text construction. Unknown
 conditioning operations and conflicting sampler candidates remain unresolved.
 Never guess that the first encoder or most recent negative belongs to an image.
 Import high-confidence pairs atomically with their image association and source
 provenance. Repeated scans must not duplicate components, pairs or image links,
 regenerate thumbnails, or increment generation run counts. Preserve existing
 conflicting/manual associations for review. Report new, enriched, unchanged,
 ambiguous, unresolved and failed results separately.
 ## Live node
 Use the shared toolkit and Split foundation. Positive and negative editors have
 equal size. Only positive receives prepend, append and automatic LoRA triggers.
 Keep upstream connections authoritative. Library selection previews both fields
 before an explicit load. Saving text changes creates another reusable combination.
 Verify Classic and Nodes 2.0 startup, switching both directions, reload, caching,
 linked inputs, popup behavior and lifecycle cleanup before maintainer testing.
 ## Delivery gate
 Keep all existing coverage floors and test suites. Record regression failures
 before implementing each behavior. Tests use disposable state and the managed
 CPU host; never mutate the maintainer's running ComfyUI or personal database.
 Python changes require a ComfyUI restart. Live approval is required before merge.
 ## Live testing
 1. Download a database backup from Admin before the first restart. The additive
   schema migration runs on startup; existing workflows and prompt IDs stay valid.
 2. Restart ComfyUI, reload the browser, and add **Prompt Manager (+/-)** from
   **ComfyAssets / Prompts**. Connect CLIP and the two conditioning outputs to the
   sampler's positive and negative inputs.
 3. Enter both texts, save a pair, then keep positive unchanged and change negative.
   Recent should show both combinations with distinct negative excerpts. Selecting
   one previews both; **Use pair** loads both together.
 4. Test category/tag autocomplete and metadata opt-in, resizing, light/dark,
   switching Classic / Nodes 2.0 both ways, and workflow reload. Connected fields
   stay authoritative; run the node to save connected metadata values.
 5. Queue the same recipe twice and change only negative once. Usage should count
   dequeued jobs even when ComfyUI caches the node. Prepend, append and LoRA triggers
   affect positive only. Gallery text is labelled as recorded during execution or
   recovered from workflow metadata.
 6. Run an output metadata scan to enrich existing images. Repeat it: pairs and
   links must remain deduplicated, with unchanged results rather than new links.
   Conflicting or ambiguous files need review and are preserved.
 Fresh executions embed bounded text evidence in PNG metadata. A cached node may
 not emit fresh evidence, so its graph-derived text is explicitly marked inferred.
 Multiple distinct batch records without an image anchor remain unresolved.
 Rescanning after interruption is safe through idempotent per-file writes; this
 version does not persist a filesystem scan cursor across server restarts.
 The API can request cancellation with `POST /prompt_manager/scan/cancel`; the
 existing dashboard Hide button continues to hide progress without cancelling.
 Pair APIs are under `/prompt_manager/pairs`: POST saves a combination, GET lists
 and filters, GET/PATCH/DELETE `/{id}` reads, edits metadata or removes an unused
 pair. Changing text creates another combination. Null metadata updates preserve
 existing values; empty strings/lists clear supported fields. Archive a pair with
 `PATCH {"archived":true}`. GET `/export` provides versioned JSON containing both
 texts and pair metadata; this is an export format, not a JSON import endpoint.
 ## Gallery redesign concept retained
 The maintainer approved the interactive
 [negative variant timeline concept](concepts/prompt-variant-timeline.html) on
 2026-10-17. Preserve it for the later gallery redesign; it is not production UI
 and does not expand the current node work.
 Keep one positive parent and one shared gallery, with red branches for reusable
 negative variants and smaller dots for repeat runs. Show the selected negative
 in full directly beneath the unchanged positive; selecting another variant or
 run updates that text and filters the shared gallery. Comparison shows both
 negatives. Distinguish explicitly empty negatives from an all-variants view and
 saved variants from executed runs. Production history needs persisted execution
 records; the prototype's timeline uses illustrative sample data.
 The Admin upgrade/scan notice remains a separate follow-up: explain automatic
 schema migration and offer an optional metadata enrichment scan or Later.
@@ -0,0 +1,13 @@
 # Native Library visual evidence
 Captured from the production workspace using disposable synthetic browser-test
 fixtures. No personal images, library, or running ComfyUI state was accessed.
 Solid-color media are intentional test images.
 - `library-dark.png`: grouped positive shelf, dark appearance.
 - `library-light.png`: the same shelf, light appearance.
 - `history-selected-negative.png`: selected negative directly below its positive,
  saved-variant branches and the filtered shared gallery.
 - `history-mobile.png`: the same history at a 390-pixel viewport.
 These screenshots document implemented behavior, rather than a design mockup.
--- a/Show More
+++ b/Show More