Compare commits

..
Author SHA1 Message Date
Vito Sansevero c1b5fd3fc0 Merge remote-tracking branch 'origin/main' into feat/444-library-folder-or
# Conflicts:
#	TESTING.md
#	database/library.py
#	database/saved_views.py
#	tests/browser/workspace-library.spec.mjs
#	tests/browser/workspace-saved-views.spec.mjs
#	tests/js/workspace-library-state.test.js
#	tests/js/workspace-saved-views.test.js
#	tests/test_saved_views.py
#	web/js/workspace/library-state.mjs
#	web/js/workspace/library.mjs
#	web/js/workspace/saved-views.mjs
#	web/js/workspace/workspace-dialogs.mjs
2026-10-29 05:26:43 -07:00
Vito Sansevero 917aa838cb fix(library): nest Refine folders by path segment; backslash separates only on Windows (#444)
Live test B1: the Folder list indented a server list sorted as full-path
text, so "-" and "_" sorted before "/" and children landed under the wrong
parent (kiko/char under kiko-v6), with case variants interleaved. The
options are now a tree built by segment: each folder directly under its own
parent, case variants distinct siblings.

On POSIX a folder named "Metadata\Klein..." was listed but filtered to no
prompts, because normalize_folder read "\" as a separator. It now does so
only on Windows (ntpath), as the listing already did.

Refs #444
2026-10-29 04:36:37 -07:00
Vito Sansevero c36ea7a0fb test(saved-views): give the D153 Library view the folder key (#444)
Refs #444
2026-10-29 00:06:10 -07:00
Vito Sansevero 1153de26fc Merge branch 'feat/444-tags-page-search' into feat/444-library-folder-or 2026-10-29 00:06:05 -07:00
Vito Sansevero e14ec03945 test(saved-views): give the D153 Library view the tag_mode key (#444)
The Library saved-view state now holds tag_mode, and the API requires
every key, so main's new "Highest rated" spec posts it too.

Refs #444
2026-10-29 00:05:59 -07:00
Vito Sansevero 990ebee79b Merge branch 'feat/444-tags-page-search' into feat/444-library-folder-or
# Conflicts:
#	TESTING.md
#	tests/js/workspace-saved-views.test.js
#	web/js/workspace/library-state.mjs
#	web/js/workspace/library.mjs
#	web/js/workspace/saved-views.mjs
2026-10-29 00:01:02 -07:00
Vito Sansevero d44cd198c4 Merge remote-tracking branch 'origin/main' into feat/444-tags-page-search
# Conflicts:
#	TESTING.md
#	tests/js/workspace-saved-views.test.js
#	web/js/workspace/library-state.mjs
#	web/js/workspace/library.mjs
#	web/js/workspace/saved-views.mjs
2026-10-29 00:00:16 -07:00
Vito Sansevero 07c16dca58 fix(library): let saved-view defaults fill the missing #444 filters
libraryViewState passed `?? 'all'` and `?? ''` fallbacks that
normalizeViewState already applies to an undefined value. The redundant
branches were never taken in the page, which dropped saved-views.mjs below
its 95% branch floor. A unit now covers a shelf without the new filters.

Refs #444
2026-10-28 20:00:49 -07:00
Vito Sansevero a6927489a8 Merge remote-tracking branch 'origin/main' into feat/444-library-folder-or
# Conflicts:
#	TESTING.md
2026-10-28 19:17:22 -07:00
Vito Sansevero 53cfd8b035 Merge branch 'feat/444-tags-page-search' into feat/444-library-folder-or
# Conflicts:
#	TESTING.md
#	web/js/workspace/library.mjs
2026-10-28 18:41:52 -07:00
Vito Sansevero 567b06ac48 Merge remote-tracking branch 'origin/main' into feat/444-tags-page-search
# Conflicts:
#	TESTING.md
#	tests/js/workspace-library-state.test.js
#	web/js/workspace/library-state.mjs
#	web/js/workspace/library.mjs
2026-10-28 18:06:52 -07:00
Vito Sansevero 4888427a1e Merge remote-tracking branch 'origin/main' into feat/444-tags-page-search 2026-10-28 17:31:47 -07:00
Vito Sansevero 9f55f0cc79 Merge feat/444-tags-page-search into feat/444-library-folder-or
Stack PR 2 on PR 1 so one restart covers both. The shared Library
tag_mode now takes all, any and untagged on the route, the filter, saved
views and Refine's Tag match; untagged still refuses tags and a folder
combines with every mode.

Refs #444
2026-10-28 17:31:11 -07:00
Vito Sansevero 6372bc7c63 feat(library): filter prompts by output folder and browse tags with Any (#444)
Refine this view on the Prompt library gains the concept's Folder (folders
below an output root, read when the dialog opens) and a Tag match select
(All / Any). The listing takes folder and tag_mode, matches folders on the
normalized file_path under every spelling of each output root, echoes both
so a stale server is an error, and saved Library views keep them. Recorded
as D150.

Refs #444
2026-10-28 17:15:25 -07:00
Vito Sansevero d4d99b7e18 feat(vocabulary): tag search, tag sort and an Untagged prompts entry (#444)
Tags & categories gains a server-side tag search and sort (keyset pages,
cursor scoped to the query) and an "Untagged prompts" row whose Browse opens
the Library with tag_mode=untagged. Refine this view gets Tag match, saved
Library views keep it, and the routes echo the applied query so a stale
server is an error, never an unfiltered list. Recorded as D149.

Refs #444
2026-10-28 17:07:50 -07:00
58 changed files with 231 additions and 4773 deletions
+5 -28
View File
@@ -21,16 +21,15 @@ env:
PYTHONFAULTHANDLER: "1"
# Time limit rule as in test.yml: ceil(1.5 x the slowest of three recent green runs on main) + 3
# minutes, never under 5, timed as whole-job wall time. One matrix job, so the slowest leg of each
# run counts (tests/vendor/ci-workflow.test.mjs recomputes it, #493).
# Measured 2026-10-29 on main runs 40935337099 (dd247e3), 40938407252 (dd247e3), 40941240624 (dd247e3):
# slowest leg: 7m24s (py3.12 windows) / 7m31s (py3.13 windows) / 7m32s (py3.14 windows) -> 15 min
# (windows 4m56s-7m32s, macos 4m39s-6m14s)
# minutes, never under 5. Measured 2026-10-26 in test.yml on runs 40257217681 (3be0ada),
# 40240706568 (fb9134f), 40228167328 (f9835aa), before these legs moved here in #420:
# slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
# (macos 6m14s-8m30s, windows 11m42s-15m54s)
jobs:
test:
name: py${{ matrix.python-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 15
timeout-minutes: 27
strategy:
fail-fast: false
matrix:
@@ -52,27 +51,5 @@ jobs:
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
# The runner's TEMP is on C:, the OS disk, where a flush took about 6 ms
# and a test database's create-to-delete about 111 ms; on the D: work
# disk the same took 0.5 ms and 23 ms (macOS: 19 ms). Tests write their
# databases under TEMP, so it moves to a folder on D: (measured in #487,
# run 40915866444). The runner's TEMP was an 8.3 short name
# (C:\Users\RUNNER~1\...), so the new one is spelled by its short name too
# and paths still have to be resolved. D: makes no short names by default
# (run 40922502209), so they are switched on for it first.
- name: Keep test files on the work disk (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: |
fsutil 8dot3name set D: 0
$folder = New-Item -ItemType Directory -Force -Path (Join-Path $env:RUNNER_TEMP "PromptManagerTests")
$short = (New-Object -ComObject Scripting.FileSystemObject).GetFolder($folder.FullName).ShortPath
Write-Output "test TEMP: $short (long name $($folder.FullName))"
if ($short -eq $folder.FullName) { Write-Output "::warning::D: gave the test TEMP no 8.3 short name" }
"TEMP=$short" >> $env:GITHUB_ENV
"TMP=$short" >> $env:GITHUB_ENV
- name: Run tests
env:
PM_TEST_SLOWEST: "40"
run: python -m tests.run_suite -v
+2 -2
View File
@@ -28,8 +28,8 @@ env:
# py matrix, slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
# (ubuntu 5m27s-6m27s, macos 6m14s-8m30s, windows 11m42s-15m54s)
# 3.10 and 3.11 left the matrix and 3.14 joined it in #356; re-measure after three green main runs.
# Windows and macOS moved to portability.yml (#420), which has its own measured limit (#493);
# this Linux-only matrix keeps 27 minutes until it is re-measured.
# Windows and macOS moved to portability.yml (#420), which keeps this 27-minute limit; this
# Linux-only matrix keeps it too until it is re-measured.
# real ComfyUI compatibility: 13m32s / 8m55s / 11m46s -> 24 min
# coverage gate (85%): 7m26s / 7m22s / 7m54s -> 15 min
# perf budget 30s / 29s / 27s, black and flake8 39s / 37s / 39s,
+4 -9
View File
@@ -36,11 +36,6 @@ concurrency:
# 12m53s, node units and the viewer suite before 11.8m of workflows), so coverage is the tight one
# in its 18 minutes. This holds only when the npm cache hits: a cold cache adds up to 5m to npm ci,
# and then the job limit can still end the job before the step limit does.
# #473: firefox/webkit shard 1/2 reached its 18-minute step limit (queue run 40866252975: webkit 622 tests
# passed in 17m56s, 48.5 worker-minutes vs 28.0 in shard 2/2; recent shard 1/2 jobs 13.7m-18.9m, shard 2/2
# 7.3m-12.2m), because --shard splits by test count and the slow concept specs sort into shard 1. The page
# workflows now run in three shards per browser with the limits unchanged; the per-shard times are
# re-measured after three green main runs.
jobs:
frontend-coverage:
name: frontend coverage (shard ${{ matrix.shard }}/3)
@@ -180,7 +175,7 @@ jobs:
retention-days: 14
browser-workflows:
name: page workflows (${{ matrix.browser }}, shard ${{ matrix.shard }}/3)
name: page workflows (${{ matrix.browser }}, shard ${{ matrix.shard }}/2)
# Merge queue, main and manual runs only (#420); pull requests run the Chromium/touch coverage shards.
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
@@ -199,7 +194,7 @@ jobs:
fail-fast: false
matrix:
browser: [firefox, webkit]
shard: [1, 2, 3]
shard: [1, 2]
steps:
- uses: actions/checkout@v7
# The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
@@ -226,9 +221,9 @@ jobs:
path: ~/.npm
key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
- run: npm ci --prefix tools/vendor --ignore-scripts
- name: Page workflows in ${{ matrix.browser }}, shard ${{ matrix.shard }}/3
- name: Page workflows in ${{ matrix.browser }}, shard ${{ matrix.shard }}/2
timeout-minutes: 18
run: npm run --prefix tools/vendor test:workflows -- --project ${{ matrix.browser }} --shard ${{ matrix.shard }}/3
run: npm run --prefix tools/vendor test:workflows -- --project ${{ matrix.browser }} --shard ${{ matrix.shard }}/2
# Traces and the blob report (test-results/blob/) of a failed, timed-out or cancelled step.
- name: Upload browser traces
if: always()
+10 -119
View File
@@ -251,29 +251,12 @@ the process exits non-zero.
stacks.
- When the watchdog fires under coverage, that run's coverage data is lost; the
job fails anyway and the dump is what matters.
- `PM_TEST_SLOWEST=N` (off by default) lists the N slowest tests after the
summary, each timed from its `start:` line to the next one like the hang
limit. The Windows and macOS jobs (`portability.yml`) set it to 40, so every
run shows where its time went (#487). The list prints only when the run
ends: a job GitHub cancels or a watchdog exit prints none, so read the
timestamps between `start:` lines instead.
- On the Windows runners `TEMP` sits on the OS disk C:, where a flush took about
6 ms and a test database's create-to-delete about 111 ms (D: 0.5 ms and 23 ms,
macOS 19 ms). `portability.yml` points `TEMP`/`TMP` at a folder under
`RUNNER_TEMP` on D: before the suite. It switches 8.3 short names on for D:
and spells the folder by its short name, as the runner's own TEMP was
(`C:\Users\RUNNER~1\...`), logs it, and warns when there is none.
A test that waits on a fixed busy timeout, a network name (`\\server\share`:
use `\\localhost\<missing share>`) or a default TLS context (it reads the
Windows certificate store) costs seconds there; the slowest list shows it.
Every job in `test.yml`, `code-quality.yml`, `viewerjs-check.yml` and
`portability.yml` has `timeout-minutes` (`tests/vendor/ci-workflow.test.mjs`
checks it). The rule is ceil(1.5 x the slowest of three recent green runs on
`main`) + 3 minutes, never under 5, timed as whole-job wall time; the three run
ids, their times and the resulting limits are written in a comment above
`jobs:`. For `portability.yml` the tooling test recomputes the limit from that
comment (#493); its one matrix job takes the slowest Windows or macOS leg. The 3 minutes leave room for one watchdog firing, so
Every job in `test.yml`, `code-quality.yml` and `viewerjs-check.yml` has
`timeout-minutes` (`tests/vendor/ci-workflow.test.mjs` checks it). The rule is
ceil(1.5 x the slowest of three recent green runs on `main`) + 3 minutes, never
under 5; the three run ids, their times and the resulting limits are written
in a comment above `jobs:`. The 3 minutes leave room for one watchdog firing, so
the dump prints before GitHub cancels the job. A job that hits its limit, or a
watchdog dump, is a failure like any other: read the dump, find the blocking
call, fix it. Never raise a limit or rerun to get green; a limit changes only
@@ -1847,9 +1830,8 @@ Inspect plan keeps D60–D66 (plan D60 → D83, D61 → D84, D65 → D88, D68
blocks' context ("Current file image tags" in History). An empty source draws no
section. Repeats are removed within a source only, so a linked tag that is also an
image tag shows in both sections and each count is true. D21's filter rules are
unchanged. History's current-file read carries the same `image_tags` facet as the Images
read (#457), so History's inline current file and its viewer show the image tags section
too (`test_current_inspection_carries_the_files_image_tags`). `tests/browser/inspector-concept.spec.mjs` wraps the
unchanged. History's current-file read carries no `image_tags` facet (D88), so there
only the linked section shows. `tests/browser/inspector-concept.spec.mjs` wraps the
concept's row in the same sections at runtime (`withTagSections`, the D154 rule
injected) and compares the measured sections (frame, heading, label, row); a
mutation of the label color, the row margin or the top rule fails it. Browser tests:
@@ -2886,12 +2868,6 @@ mapped differences"):
change within the workspace waits the same way: the shell renders the next route once a
video still starting on Inspect settles (`leaveGuard`; the latest route wins). Route
changes at `loadstart` + 0–7 ms hung 4–5 of 120 pages before and 0 of 120 after.
The shared viewer's own video gets the same guard (#483): a new selection while the
viewer D155 opened is still starting its video leaves the viewer open, untouched, until
that video reports metadata or an error (or 15 s), then closes it; the URL is revoked
after that, and a viewer asked for meanwhile (the new file's D155 open) opens then. The
merge queue hung once without it (run 40903238387: the PNG selection 12 ms after the
inline video showed, with the viewer's new video still starting).
A real page leave cannot wait. With no page code releasing anything, navigating
away hung 8 of 120 pages and reload 22 of 120 (3 and 8 of 120 with the page's own
release), so that WebKit teardown stays a known limitation; dispose still releases at
@@ -3323,8 +3299,7 @@ until this browser dismissed the 3.2.0 update notice (`pm_update_notice_dismisse
the legacy admin page's key); Later dismisses it without scanning, and Review scan
dismisses it as the legacy Scan does. "+ Add source" asks validate-source, then adds
the folder to the host draft of `gallery_root_paths`; Remove takes one out; "Watch for
new outputs" (`monitoring_enabled`), "Save prompts from other workflows"
(`save_foreign_runs`, D158) and Worker threads (`worker_threads`) are host
new outputs" (`monitoring_enabled`) and Worker threads (`worker_threads`) are host
drafts too, and Save writes them with the revision. A save the server refuses because a
scan runs toasts "Another job is running: <title>." with a link to that job (D57) and
keeps the drafts; a save refused for one source says the server's message as the status
@@ -3573,7 +3548,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
reader can keep it byte-identical (`test_health_check_never_writes`).
Recorded deviations (plan D60–D75, D79–D82 renumbered D92–D107, D109–D110 and D111–D114; D121–D134 and D158 assigned). D94,
Recorded deviations (plan D60–D75, D79–D82 renumbered D92–D107, D109–D110 and D111–D114; D121–D134 assigned). D94,
the per-section hand-off panel, was retired by S12 (#329): every section is native.
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
@@ -4093,26 +4068,6 @@ the per-section hand-off panel, was retired by S12 (#329): every section is nati
Cancel-only review matches the concept review shell` (system-tagging-concept.spec) and
the batch tests of workspace-system-tagging.spec; the mutation check renames each grid
label in turn.
- **D158** (Sources, #247 D-Q4): the concept has no switch for prompts from workflows
without a PromptManager node; the maintainer chose a visible switch over a config key.
A `.setting-row` in the concept's toggle markup follows "Watch for new outputs":
"Save prompts from other workflows" (`#foreign-runs`, host key `save_foreign_runs`,
config.json `gallery.monitoring.save_foreign_runs`, default on), "New outputs of
workflows without a PromptManager node add their prompt to the library when it is
certain.". It is saved with the revision like the other host rows and applied at once
(the monitor reads it for every file), so a save names no restart. While "Watch for
new outputs" is drafted off, a `p.small` under the description says "Applies while
Watch for new outputs is on." (the switch's `aria-describedby`; the value is kept) and
the switch is disabled, except while a source still reads "watching until restart"
(D128): the running observer still delivers files then, so the value can be set. The
server stores no foreign prompt while `monitoring_enabled` is saved off, whatever the
observer does.
Pinned by `sources matches the concept` (the mapping inserts the row after the
monitoring row), `sources comparison detects mutations` (a missing row and a changed
label are caught), `D158: Save prompts from other workflows saves save_foreign_runs
live; off while monitoring is off (#247)` (keyboard on desktop, a tap on touch) and
`D158: saved off while the monitor still watches until a restart, the foreign row can
still be set (D128)` (workspace-system-sources.spec) and the system-sources node test.
- **D109** (Logs, S12, #329): the concept's Logs surface on the live server. The toolbar
keeps "Level", Refresh and the download (renamed "Download log": the active log,
`/logs/download/prompt_manager.log`, through the api client), with all six minimum
@@ -6040,70 +5995,6 @@ Tests: `tests/test_file_prompt.py` (one test per ladder step and junk class),
saver, a legacy-linked file left as it is, `_BatchHistory.recorded`) and
`tests/test_recorded_saver_anchor.py` `TestInspectionResolution`.
**In the monitor.** An output whose workflow has no positive PromptManager node
(`_find_prompt_from_metadata` says `foreign`) used to be logged and skipped. It now
goes through the same ladder with the `MONITOR` policy (`utils/foreign_runs.py`, the
handler's `_ingest_foreign`, `_recheck_foreign`, `_store_foreign`):
- **At settle.** The ladder runs without history. Certain (`pair` or `positive`): the
prompt is stored and the file linked at once. `history` (an undecided graph inside
ComfyUI's output folder): the file is registered in its own waiting set, then the
ladder runs again at once with the file's recorded output, because the run may have
been sealed before the file settled. When that lookup finds a record, its answer is
final: certain is stored, anything else is logged "No certain prompt for <file>;
left unlinked" and the file stops waiting (no seal is coming). When it finds none,
only the inspector's or the recorded saver's answer is stored; anything else (the
legacy floor included) keeps waiting for the re-check. Anything else at the first
pass is logged "No certain prompt for <file>; left unlinked".
- **After the run ends.** When the run's outputs are sealed, each waiting foreign file
gets one re-check (`ForeignRecheck(reference)`) with the sealed output reference,
so history is asked with the spelling it recorded (a Windows folder whose case
differs from the disk still links). Not certain: "No certain prompt for <file>
after its run ended; left unlinked". An ambiguous inspector answer is never given
the legacy floor in the monitor (D-Q3); an untraceable one still is.
- **Waiting sets.** `_awaiting_foreign` and `_foreign_due` sit beside the keyless sets
with the same bounds (1000 files, one hour) but are separate, so a large foreign run
never evicts a waiting keyless file. An expired entry is logged "No recorded run
for <file>; left unlinked", an evicted one (waiting or due) "Too many files wait for
their run; <file> left unlinked", both after the lock is released. A due re-check
is dropped only when it was scheduled, never a newer reference for the same file. `awaiting_foreign()` lists them;
`shutdown()` and `close()` clear them, and a closed handler re-checks nothing.
- **Store.** One write through `survive_switch` (rerun after a Move library): a pair
through `ingest_resolved_pair(..., marks=RUN_MARKS)`, a positive through
`store_scanned_prompt_or_raise(..., RUN_MARKS)`. A created prompt carries category
`scanned`, tag `auto-scanned` and the note `Saved from a workflow run: <file>`; a
file linked to another prompt is `conflict` and keeps its link. Logged at info
"Foreign run: <status> prompt <id> for <file> (<source>)"; a failure is logged and
the file stays unlinked.
- **Never.** No tracker FIFO entry is read or consumed, nothing is counted (run count
stays 0, no `last_used_at`), and nothing is queued for Needs review.
- **The setting.** `GalleryConfig.SAVE_FOREIGN_RUNS` (`gallery.monitoring.
save_foreign_runs`, default true, only a bool applies) is read for every file, at
settle, at the re-check and again just before the store, together with
`MONITORING_ENABLED`: either saved off stops foreign writes at once (the running
observer watches until a restart, D128); off logs "No PromptManager prompt in
workflow, skipping image" as before. System › Sources & scans "Save prompts from other workflows" (D158)
saves it through `POST /prompt_manager/settings` with no restart field; the
diagnostics report lists it.
Tests: `tests/test_run_attribution.py` `TestMonitorForeignRuns` (dataset branches
linked after the seal, a seal before settle, a name two runs reported, the setting
off at settle and at the re-check, a plain workflow linked at settle with no run
count and the tracker queue untouched, the sealed reference's spelling, the separate
bound, expiry, a seal while the pending set is full, shutdown, a failed store, no
record at settle keeps waiting, a record without an answer ends the wait, monitoring
saved off, the setting read before the store, a seal between registering and the
settle lookup, a newer due reference kept, evictions logged, one link per file),
`tests/test_image_linking.py` `TestForeignWorkflows`, `tests/test_image_monitor_singleton.py`
(close drops a foreign re-check), `tests/test_config.py`,
`tests/test_settings_sources.py` `TestSaveForeignRunsSetting`, and the real-host
`tests/comfyui/image-attribution.spec.mjs` case "an output of a workflow without a
PromptManager node creates and links its prompt; a second run reuses it" (ComfyUI's
own `CLIPTextEncode`, `KSampler` and `SaveImage`), and "each branch of a
shared-prefix workflow without a PromptManager node links to its own prompt after the
run" (two branches, one save prefix: the recorded saver names each file's branch).
The fixture route's GET also lists `foreign`, the waiting foreign files.
## Linked prompt text (#343)
A link drives `text` on every PromptManager node, and `negative_text` on the Pair,
@@ -6384,7 +6275,7 @@ The versioned public reads are:
| `GET /prompt_manager/library/{positive_id}/history/pairs/{pair_id}/newest` | The pair's newest recorded run for the history run dot (#249 D26): `{run: {run_id, anchor_id, context_id}}` or `{run: null}` |
| `GET /prompt_manager/history/runs/{run_id}` | Frozen fields, per-field provenance, capture and outcome for a known UUID |
| `GET /prompt_manager/history/runs/{run_id}/outputs` | Ordered host-reported occurrences, including repeated locations |
| `GET /prompt_manager/history/runs/{run_id}/output/current/{media,thumbnail,inspection}` | Authorized current file at that reported location, using the returned epoch/scope URL; `inspection` carries the shared inspector envelope including the file's `image_tags` facet (#457) |
| `GET /prompt_manager/history/runs/{run_id}/output/current/{media,thumbnail,inspection}` | Authorized current file at that reported location, using the returned epoch/scope URL |
The newest-pair read takes no query. It answers 404 when the positive is not an
eligible saved prompt or the pair is not one of its pairs. It walks the positive's exact
+26 -58
View File
@@ -7,11 +7,6 @@ from typing import Callable, ContextManager, Iterator
try:
from ..utils.execution_history_events import RunFinish, encode_command
from ..utils.hashing import (
generate_prompt_hash,
stored_prompt_text,
stored_prompt_text_sql,
)
from ..utils.execution_contexts import (
CAPTURE_REASONS,
ContextBundle,
@@ -20,11 +15,6 @@ try:
)
except ImportError:
from utils.execution_history_events import RunFinish, encode_command
from utils.hashing import (
generate_prompt_hash,
stored_prompt_text,
stored_prompt_text_sql,
)
from utils.execution_contexts import (
CAPTURE_REASONS,
ContextBundle,
@@ -144,8 +134,7 @@ def _insert_exact(
def _role_hints(role: str) -> str:
return (
f"SELECT r.{role}_prompt_id, tx.text_value, "
f"{stored_prompt_text_sql('tx.text_value')} "
f"SELECT r.{role}_prompt_id, tx.text_value "
"FROM execution_run_contexts r "
"JOIN execution_contexts c ON c.context_id = r.context_id "
f"JOIN execution_prompt_snapshots s ON s.snapshot_id = c.{role}_snapshot_id "
@@ -157,78 +146,57 @@ def _role_hints(role: str) -> str:
# Set form of the read-time prompt-hint rule in get_execution_contexts():
# a stored role hint is live only in the current storage epoch, and only when
# the hinted prompt is the one match of _prompt_hint: by text (the raw
# snapshot text or its stored form, outer whitespace stripped, #438), or, only
# when no prompt has either text, by dedup hash (case folded too, #460).
# The hash is a fallback, so a hint #438 kept is never made ambiguous.
# Needs register_prompt_functions (pm_prompt_hash). The hint rows are the
# outer loop (CROSS JOIN), so each prompts lookup is by index.
# its exact raw snapshot text matches exactly one prompt, which is the hinted
# one (_prompt_hint). Evaluated once per statement, never per prompt row.
# Defines the CTE ``live_hint_prompts(prompt_id)``; prefix it with WITH.
LIVE_HINT_CTES = (
"stored_hints(prompt_id, raw, stored) AS ("
"stored_hints(prompt_id, raw) AS ("
+ _role_hints("positive")
+ " UNION "
+ _role_hints("negative")
+ "), hint_matches(prompt_id, raw, stored, match_id) AS ("
"SELECT h.prompt_id, h.raw, h.stored, p.id FROM stored_hints h "
"CROSS JOIN prompts p ON p.text COLLATE BINARY IN (h.raw, h.stored) "
"UNION ALL "
"SELECT h.prompt_id, h.raw, h.stored, p.id FROM stored_hints h "
"CROSS JOIN prompts p ON p.hash = pm_prompt_hash(h.raw) "
"WHERE NOT EXISTS (SELECT 1 FROM prompts q "
"WHERE q.text COLLATE BINARY IN (h.raw, h.stored))"
+ "), unique_hint_texts(prompt_id, raw) AS ("
"SELECT MIN(id), text COLLATE BINARY FROM prompts "
"WHERE text COLLATE BINARY IN (SELECT raw FROM stored_hints) "
"GROUP BY text COLLATE BINARY HAVING COUNT(*) = 1"
"), live_hint_prompts(prompt_id) AS ("
"SELECT prompt_id FROM hint_matches GROUP BY prompt_id, raw, stored "
"HAVING COUNT(*) = 1 AND MIN(match_id) = prompt_id)"
"SELECT h.prompt_id FROM stored_hints h JOIN unique_hint_texts u "
"ON u.prompt_id = h.prompt_id AND u.raw = h.raw COLLATE BINARY)"
)
LIVE_HINT_PROMPT_IDS_SQL = (
"WITH " + LIVE_HINT_CTES + " SELECT DISTINCT prompt_id FROM live_hint_prompts"
)
_BY_TEXT = "SELECT id FROM prompts WHERE text COLLATE BINARY IN (?,?) LIMIT 2"
def _text_matches(conn: sqlite3.Connection, raw: str) -> list:
return [row[0] for row in conn.execute(_BY_TEXT, (raw, stored_prompt_text(raw)))]
def _prompt_hint(conn: sqlite3.Connection, raw: str | None) -> int | None:
"""The one prompt that stores ``raw`` (#438), else the one it counts toward.
By text first: ``raw`` or ``raw.strip()`` (the exact form keeps hints of
older rows that kept outer whitespace). Only when no prompt has either
text, by the dedup hash, which also folds case (#460).
"""
if raw is None:
return None
matches = _text_matches(conn, raw)
if not matches:
matches = [
row[0]
for row in conn.execute(
"SELECT id FROM prompts WHERE hash=? LIMIT 2",
(generate_prompt_hash(raw),),
)
]
return matches[0] if len(matches) == 1 else None
rows = conn.execute(
"SELECT id FROM prompts WHERE text=? COLLATE BINARY LIMIT 2", (raw,)
).fetchall()
return rows[0][0] if len(rows) == 1 else None
def _hints(
conn: sqlite3.Connection, positive: str | None, negative: str | None
) -> tuple:
pos_id, neg_id = _prompt_hint(conn, positive), _prompt_hint(conn, negative)
stored = None if negative is None else stored_prompt_text(negative)
pair_id = None
# A known negative that names no prompt because several share its text
negative_ambiguous = (
stored is not None and neg_id is None and bool(_text_matches(conn, negative))
negative is not None
and neg_id is None
and conn.execute(
"SELECT 1 FROM prompts WHERE text=? COLLATE BINARY LIMIT 1", (negative,)
).fetchone()
is not None
)
if pos_id is not None and stored is not None and not negative_ambiguous:
if pos_id is not None and negative is not None and not negative_ambiguous:
rows = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"SELECT pp.id FROM prompt_pairs pp LEFT JOIN prompts n ON "
"n.id=pp.negative_prompt_id "
"WHERE pp.positive_prompt_id=? AND ((pp.negative_prompt_id IS NULL AND "
"?='') OR pp.negative_prompt_id=?) LIMIT 2",
(pos_id, stored, neg_id),
"?='') "
"OR (pp.negative_prompt_id=? AND n.text=? COLLATE BINARY)) LIMIT 2",
(pos_id, negative, neg_id, negative),
).fetchall()
if len(rows) == 1:
pair_id = rows[0][0]
+7 -3
View File
@@ -23,10 +23,10 @@ from pathlib import PurePosixPath, PureWindowsPath
from urllib.parse import quote
try:
from ..utils.hashing import register_prompt_functions
from ..utils.hashing import generate_prompt_hash
from .models import normalize_image_path
except ImportError:
from utils.hashing import register_prompt_functions
from utils.hashing import generate_prompt_hash
from database.models import normalize_image_path
logger = logging.getLogger("prompt_manager.database.health")
@@ -125,8 +125,12 @@ def _foreign_keys(conn, _roots):
)
def _prompt_hash(text):
return generate_prompt_hash(text) if isinstance(text, str) else None
def _canonical_hashes(conn, _roots):
register_prompt_functions(conn) # pm_prompt_hash, shared with history (#460)
conn.create_function("pm_prompt_hash", 1, _prompt_hash, deterministic=True)
return _counted(
f"Prompt {row[0]}: the stored hash does not match its text"
for row in conn.execute(
+28 -66
View File
@@ -15,11 +15,6 @@ from .execution_history import HistoryEpochChanged
from .execution_outputs import _capture, _REFERENCE_COLUMNS
from .library import _ELIGIBLE
try:
from ..utils.hashing import prompt_key, stored_prompt_text
except ImportError:
from utils.hashing import prompt_key, stored_prompt_text
RunKey = tuple[int, str, int, str]
@@ -105,43 +100,14 @@ _ORDER = (
)
# Stale-hint rows a newest-pair lookup inspects before it reports none.
_PAIR_RUN_CANDIDATES = 16
# The recorded texts of one scope, as a JSON array parameter (_scope_text_ids).
_IN_SCOPE = "IN (SELECT value FROM json_each(?))"
_MATCHING = (
"SELECT DISTINCT rc.run_id FROM execution_prompt_snapshots s "
"JOIN execution_contexts c ON c.positive_snapshot_id=s.snapshot_id "
"JOIN execution_run_contexts rc ON rc.context_id=c.context_id "
f"WHERE s.raw_text_id {_IN_SCOPE}"
"WHERE s.raw_text_id=?"
)
def _scope_text_ids(conn: sqlite3.Connection, key: str) -> str:
"""Every recorded raw text whose prompt key is ``key``, as a JSON array.
A run keeps its exact raw text, while the Library saves and counts it as
the prompt with its key (outer whitespace #438, case #460). The values are
compared, never only their digests; ``pm_prompt_key`` is Python's key.
"""
return json.dumps(
[
row[0]
for row in conn.execute(
# Only snapshot raw texts: effective and affix texts are skipped
"SELECT t.text_id FROM (SELECT DISTINCT raw_text_id AS text_id "
"FROM execution_prompt_snapshots) s CROSS JOIN "
"execution_context_texts t WHERE t.text_id=s.text_id AND "
"pm_prompt_key(t.text_value)=? COLLATE BINARY",
(key,),
)
]
)
def _scope_text_id(raw: str | None) -> str | None:
"""The scope digest of a raw text: that of its prompt key."""
return None if raw is None else content_digest("text", (prompt_key(raw),))
def _limit(limit: int) -> None:
if type(limit) is not int or not 1 <= limit <= 100:
raise ValueError("History limit must be an integer from 1 to 100")
@@ -289,15 +255,8 @@ class HistoryReadOperations:
if saved is None:
return None
text = saved["text"]
raw_id = _scope_text_id(text)
# A position from before #460 names the scope by the stripped text
# alone; it is the same scope, so a page open across the upgrade resumes.
legacy_id = (
None
if text is None
else content_digest("text", (stored_prompt_text(text),))
)
if after is not None and after.raw_text_id not in (raw_id, legacy_id):
raw_id = None if text is None else content_digest("text", (text,))
if after is not None and after.raw_text_id != raw_id:
raise ValueError("History position text scope changed")
group = {
"positive_id": positive_id,
@@ -314,11 +273,18 @@ class HistoryReadOperations:
next_position=None,
resume_position=None,
)
if raw_id is None:
if (
raw_id is None
or conn.execute(
"SELECT 1 FROM execution_context_texts WHERE text_id=? "
"AND text_value=? COLLATE BINARY",
(raw_id, text),
).fetchone()
is None
):
return result
scope = _scope_text_ids(conn, prompt_key(text))
where = f"r.run_id IN ({_MATCHING})"
params = [scope]
params = [raw_id]
if after is not None:
where += f" AND {_KEY}<=(?,?,?,?)"
params.extend(after.upper)
@@ -372,9 +338,9 @@ class HistoryReadOperations:
"LEFT JOIN execution_prompt_snapshots n "
"ON n.snapshot_id=c.negative_snapshot_id "
"LEFT JOIN execution_context_texts t ON t.text_id=n.raw_text_id "
f"WHERE rc.run_id IN ({placeholders}) AND p.raw_text_id {_IN_SCOPE} "
f"WHERE rc.run_id IN ({placeholders}) AND p.raw_text_id=? "
"ORDER BY rc.anchor_id COLLATE BINARY,rc.run_id COLLATE BINARY",
(*by_run, scope),
(*by_run, raw_id),
)
summaries = {}
evidence_by_context = {}
@@ -472,7 +438,6 @@ class HistoryReadOperations:
if saved["text"] is None:
return {"run": None}
epoch = self.get_execution_epoch()
key = prompt_key(saved["text"])
candidates = conn.execute(
"SELECT r.run_id,rc.anchor_id,rc.context_id,"
"c.positive_snapshot_id,c.negative_snapshot_id "
@@ -480,11 +445,10 @@ class HistoryReadOperations:
"ON rc.run_id=r.run_id JOIN execution_contexts c "
"ON c.context_id=rc.context_id JOIN execution_prompt_snapshots s "
"ON s.snapshot_id=c.positive_snapshot_id "
f"WHERE s.raw_text_id {_IN_SCOPE} AND rc.pair_id=? "
"AND rc.reference_epoch=? "
"WHERE s.raw_text_id=? AND rc.pair_id=? AND rc.reference_epoch=? "
f"ORDER BY {_ORDER},rc.anchor_id COLLATE BINARY LIMIT ?",
(
_scope_text_ids(conn, key),
content_digest("text", (saved["text"],)),
pair_id,
epoch,
_PAIR_RUN_CANDIDATES,
@@ -493,11 +457,7 @@ class HistoryReadOperations:
for row in candidates:
positive = _values(conn, row["positive_snapshot_id"])
negative = _values(conn, row["negative_snapshot_id"])
if (
positive is None
or positive["raw"] is None
or prompt_key(positive["raw"]) != key
):
if positive is None or positive["raw"] != saved["text"]:
continue
hints = _hints(conn, positive["raw"], negative["raw"] if negative else None)
if hints[2] == pair_id:
@@ -517,9 +477,11 @@ class HistoryReadOperations:
snapshots = {
row["anchor_id"]: row
for row in conn.execute(
"SELECT rc.anchor_id,c.positive_snapshot_id,c.negative_snapshot_id "
"SELECT rc.anchor_id,c.positive_snapshot_id,c.negative_snapshot_id,"
"p.raw_text_id AS positive_raw_text_id "
"FROM execution_run_contexts rc JOIN execution_contexts c "
"ON c.context_id=rc.context_id WHERE rc.run_id=?",
"ON c.context_id=rc.context_id LEFT JOIN execution_prompt_snapshots p "
"ON p.snapshot_id=c.positive_snapshot_id WHERE rc.run_id=?",
(run_id,),
)
}
@@ -529,12 +491,12 @@ class HistoryReadOperations:
context.update(
{
key: row[key]
for key in ("positive_snapshot_id", "negative_snapshot_id")
},
# The history scope this context belongs to (#438)
positive_raw_text_id=_scope_text_id(
(context["positive"] or {}).get("raw")
),
for key in (
"positive_snapshot_id",
"negative_snapshot_id",
"positive_raw_text_id",
)
}
)
pair = context["pair_id"]
context["navigation"] = {
-6
View File
@@ -44,11 +44,6 @@ from .health import BATCH_ROWS, link_states, readable_roots, readonly_uri
from .pairs import PairDependencyError
from .workflow_documents import documents_of_rows, release_documents
try:
from ..utils.hashing import register_prompt_functions
except ImportError:
from utils.hashing import register_prompt_functions
MAINTENANCE_OPERATIONS = (
"duplicate_prompts",
"optimize",
@@ -294,7 +289,6 @@ def _read_snapshot(db_path, read):
conn = sqlite3.connect(readonly_uri(db_path), uri=True)
try:
conn.row_factory = sqlite3.Row
register_prompt_functions(conn) # LIVE_HINT_CTES calls them (#460)
conn.execute("BEGIN") # one read snapshot for the whole plan
return read(conn)
finally:
-8
View File
@@ -38,11 +38,6 @@ from .review_schema import (
review_without_annotations_error,
)
from .saved_views import migrate_saved_views, saved_views_schema_error
try:
from ..utils.hashing import register_prompt_functions
except ImportError:
from utils.hashing import register_prompt_functions
from .image_categories import image_categories_schema_error, migrate_image_categories
from .workflow_documents import (
DOCUMENT_ID_COLUMNS,
@@ -158,15 +153,12 @@ def configure_connection(conn: sqlite3.Connection) -> None:
(common on Windows) let other writers' polling busy handlers starve past
BUSY_TIMEOUT_SECONDS and fail with "database is locked" (#242). Without
WAL (for example an in-memory database) SQLite's FULL default is kept.
It also registers the prompt-key SQL functions (register_prompt_functions).
"""
journal_mode = conn.execute("PRAGMA journal_mode = WAL").fetchone()[0]
conn.execute("PRAGMA foreign_keys = ON")
conn.execute(f"PRAGMA busy_timeout = {BUSY_TIMEOUT_SECONDS * 1000}")
if str(journal_mode).lower() == "wal":
conn.execute("PRAGMA synchronous = NORMAL")
# History reads and the live-hint rule call Python's prompt key (#460)
register_prompt_functions(conn)
class PromptModel:
-2
View File
@@ -213,8 +213,6 @@ def public_config():
"max_image_age_days": GalleryConfig.MAX_IMAGE_AGE_DAYS,
"max_concurrent_processing": GalleryConfig.MAX_CONCURRENT_PROCESSING,
"metadata_extraction_timeout": GalleryConfig.METADATA_EXTRACTION_TIMEOUT,
# config.json gallery.monitoring.save_foreign_runs (#247)
"save_foreign_runs": GalleryConfig.SAVE_FOREIGN_RUNS,
},
"integrations": {
"lora_manager_enabled": bool(IntegrationConfig.LORA_MANAGER_ENABLED),
+1 -13
View File
@@ -27,11 +27,7 @@ from .history_current import (
resolve_current_candidate,
snapshot_current_output_scope,
)
from .inspection_service import (
CurrentFileChanged,
image_tags_reader,
inspect_current_file,
)
from .inspection_service import CurrentFileChanged, inspect_current_file
from .images import image_category_key
try:
@@ -782,14 +778,6 @@ class HistoryRoutesMixin:
),
own,
),
# The file's own image tags, keyed as the Tag dialog
# keys them: the normcased realpath (#457).
image_tags=image_tags_reader(
db,
str(resolved.canonical_target),
self.logger,
close=True,
),
recorded=recorded_output_lookup(
db, close=True, reference=receipt["reference"]
),
+21 -3
View File
@@ -55,7 +55,6 @@ from .inspection_service import (
IMAGE_EXTENSIONS as _IMAGE_EXTENSIONS,
SERVABLE_EXTENSIONS,
VIDEO_EXTENSIONS,
image_tags_reader,
inspect_current_file,
)
@@ -3963,9 +3962,28 @@ class ImageRoutesMixin:
)
def _inspection_image_tags(self, root: Path, parts: tuple) -> Callable:
"""The inspector's ``image_tags`` reader for one rooted file (S1)."""
"""The inspector's ``image_tags`` reader for one rooted file (S1).
The returned function takes the inspection's own stable stat and
answers ``current`` only for a byte record verified at exactly that
``(size, mtime_ns)``; a failed read is ``unavailable``, never an error.
"""
locator = os.path.normcase(os.path.join(os.path.realpath(root), *parts))
return image_tags_reader(self.db, locator, self.logger)
def read(size: int, mtime_ns: int) -> dict:
try:
found = self.db.current_image_tags({locator: (size, mtime_ns)})
except Exception:
self.logger.warning(
"Inspection image tags could not be read", exc_info=True
)
return {"state": "unavailable", "tags": []}
tags = found.get(locator)
if tags is None:
return {"state": "none", "tags": []}
return {"state": "current", "tags": list(tags)}
return read
def _read_own_category(self, sources, path):
"""One file's own category, None, or OWN_CATEGORY_UNREAD (blocking).
-29
View File
@@ -93,35 +93,6 @@ def _in_host_temp(path: Path) -> bool:
return False
def image_tags_reader(
db: Any, locator: str, logger: Any, *, close: bool = False
) -> Callable[[int, int], dict]:
"""The inspector's ``image_tags`` reader for one file *locator* (S1).
The returned function takes the inspection's own stable stat and answers
``current`` only for a byte record verified at exactly that
``(size, mtime_ns)``; a failed read is ``unavailable``, never an error.
``close`` closes the calling thread's connection afterwards, for routes
that close theirs after reading (History).
"""
def read(size: int, mtime_ns: int) -> dict:
try:
found = db.current_image_tags({locator: (size, mtime_ns)})
except Exception:
logger.warning("Inspection image tags could not be read", exc_info=True)
return {"state": "unavailable", "tags": []}
finally:
if close:
db.model.close()
tags = found.get(locator)
if tags is None:
return {"state": "none", "tags": []}
return {"state": "current", "tags": list(tags)}
return read
def inspect_current_file(
path: Path,
*,
+2 -9
View File
@@ -8,8 +8,7 @@ is shared with the LoRA integration's save.
GET also reports the image sources and the monitor (``sources.py``). A save
that changes the sources runs as a ``source_change`` registry hold, so it is
refused while a record-changing job (a scan) runs. ``monitoring_enabled``
takes effect at the next restart; ``save_foreign_runs`` (#247) at once, since
the monitor reads it for every file. ``POST /prompt_manager/settings/validate-source``
takes effect at the next restart. ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing.
"""
@@ -44,7 +43,6 @@ _SETTINGS_FILE_KEYS = {
"INFINITE_SCROLL": ("web_ui", "infinite_scroll"),
"WORKER_THREADS": ("performance", "worker_threads"),
"MONITORING_ENABLED": ("gallery", "monitoring", "enabled"),
"SAVE_FOREIGN_RUNS": ("gallery", "monitoring", "save_foreign_runs"),
"SAFETY_BACKUP": ("maintenance", "safety_backup"),
"THUMBNAIL_FORMAT": ("thumbnails", "format"),
"THUMBNAIL_QUALITY": ("thumbnails", "quality"),
@@ -60,7 +58,7 @@ _SETTINGS_FILE_KEYS = {
# The thumbnail settings a running Convert outdated job fixes (#317).
_THUMBNAIL_SETTINGS = ("THUMBNAIL_FORMAT", "THUMBNAIL_QUALITY", "THUMBNAIL_EDGE")
# Attributes that live on GalleryConfig; the others on PromptManagerConfig.
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED", "SAVE_FOREIGN_RUNS"})
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"})
# Attributes that live on AutotagConfig, named without the AUTOTAG_ prefix.
_AUTOTAG_SETTINGS = frozenset(
name for name in _SETTINGS_FILE_KEYS if name.startswith("AUTOTAG_")
@@ -266,7 +264,6 @@ class SettingsRoutesMixin:
"thumbnail_encoders": encoders,
"settings": {
"monitoring_enabled": GalleryConfig.MONITORING_ENABLED,
"save_foreign_runs": GalleryConfig.SAVE_FOREIGN_RUNS,
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
@@ -629,10 +626,6 @@ class SettingsRoutesMixin:
if not isinstance(data["monitoring_enabled"], bool):
raise ValueError("monitoring_enabled must be true or false")
updates["MONITORING_ENABLED"] = data["monitoring_enabled"]
if "save_foreign_runs" in data:
if not isinstance(data["save_foreign_runs"], bool):
raise ValueError("save_foreign_runs must be true or false")
updates["SAVE_FOREIGN_RUNS"] = data["save_foreign_runs"]
if "safety_backup" in data:
if not isinstance(data["safety_backup"], bool):
raise ValueError("safety_backup must be true or false")
-13
View File
@@ -194,8 +194,6 @@ class GalleryConfig:
Attributes:
MONITORING_ENABLED (bool): Enable/disable automatic image monitoring
SAVE_FOREIGN_RUNS (bool): The monitor stores the certain prompt of an
output whose workflow has no PromptManager node (#247)
MONITORING_DIRECTORIES (List[str]): Directories to monitor for new images
SUPPORTED_EXTENSIONS (List[str]): Image file extensions to process
PROCESSING_DELAY (float): Delay in seconds before processing new files
@@ -212,8 +210,6 @@ class GalleryConfig:
# Image monitoring settings
MONITORING_ENABLED = True
# Read by the monitor for every file, so a save applies without a restart
SAVE_FOREIGN_RUNS = True
MONITORING_DIRECTORIES = [] # Auto-detect if empty
SUPPORTED_EXTENSIONS = [".png", ".jpg", ".jpeg", ".webp", ".gif"]
PROCESSING_DELAY = 2.0 # Seconds to wait before processing new files
@@ -255,7 +251,6 @@ class GalleryConfig:
return {
"monitoring": {
"enabled": cls.MONITORING_ENABLED,
"save_foreign_runs": cls.SAVE_FOREIGN_RUNS,
"directories": cls.MONITORING_DIRECTORIES,
"extensions": cls.SUPPORTED_EXTENSIONS,
"processing_delay": cls.PROCESSING_DELAY,
@@ -308,14 +303,6 @@ class GalleryConfig:
config_logger.warning(
"Ignoring gallery monitoring enabled: expected true or false"
)
if "save_foreign_runs" in monitoring:
if isinstance(monitoring["save_foreign_runs"], bool):
cls.SAVE_FOREIGN_RUNS = monitoring["save_foreign_runs"]
else:
config_logger.warning(
"Ignoring gallery monitoring save_foreign_runs: expected true "
"or false"
)
if "directories" in monitoring:
directories = cls._validated_directories(monitoring["directories"])
if directories is not None:
+7 -29
View File
@@ -618,21 +618,10 @@ const WATCHED = new Set(['watching', 'stopping']);
const REFRESH_FILES = { title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
label: 'Refresh' };
// D158 (#247): the row production adds directly below "Watch for new outputs", in that row's markup.
const FOREIGN_RUNS = { id: 'foreign-runs', label: 'Save prompts from other workflows',
description: 'New outputs of workflows without a PromptManager node add their prompt to the library when it is certain.' };
/** D158: the concept's "Watch for new outputs" row as the inserted "Save prompts from other workflows" row. */
function foreignRunsRow(monitoringRow, settings) {
return { ...monitoringRow, text: [['label', FOREIGN_RUNS.label, FOREIGN_RUNS.id], ['p', FOREIGN_RUNS.description, null]],
control: { ...monitoringRow.control, id: FOREIGN_RUNS.id, checked: settings.save_foreign_runs } };
}
/** The concept's Sources with the recorded deviations applied, given the server's real settings and sources. */
function mapSourcesConcept(reference, { settings, sources }) {
const [sample] = reference.sources;
const scan = sample.actions[0];
const rowAt = reference.surface.tree.indexOf('div.setting-row');
return {
...reference,
// The real sources replace the two samples (without the "sample path" note). D98: each configured one has Remove,
@@ -640,15 +629,13 @@ function mapSourcesConcept(reference, { settings, sources }) {
sources: sources.effective.map(source => ({ ...sample, title: source.label,
description: `${source.path} · ${WATCH_TEXT[source.watch]}`,
actions: [{ ...scan, text: WATCHED.has(source.watch) ? 'Scan changes' : 'Scan folder' }, { ...scan, text: 'Remove' }] })),
rows: reference.rows.flatMap(row => {
if (row.control.id === 'monitoring') return [{ ...row, control: { ...row.control, checked: settings.monitoring_enabled } },
foreignRunsRow(row, settings)];
rows: reference.rows.map(row => {
if (row.control.id === 'monitoring') return { ...row, control: { ...row.control, checked: settings.monitoring_enabled } };
// D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count.
return [{ ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } }];
return { ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } };
}),
// D158: the inserted row after the first setting row; D98: "Refresh files" closes the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree.slice(0, rowAt + 1), 'div.setting-row',
...reference.surface.tree.slice(rowAt + 1), 'div.operation'] },
// D98: "Refresh files" closes the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.operation'] },
operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title,
description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }],
};
@@ -702,12 +689,10 @@ test('sources comparison detects mutations', async ({ page, request }) => {
const expected = mapSourcesConcept(concept, server);
const production = await productionSources(page, 'dark', { notice: true });
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only in the real sources, the D98 operation, the D158 row (which moves the D113
// range to the third row) and the D113 range.
// Unmapped, the concept differs only in the real sources, the D98 operation and the D113 range.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.2', 'operations.length', 'rows.1', 'rows.2', 'rows.length', 'sources.0', 'sources.1',
'surface.tree'].sort());
.toEqual(['operations.2', 'operations.length', 'rows.1', 'sources.0', 'sources.1', 'surface.tree'].sort());
const measure = () => page.evaluate(measureSources, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed notice and a removed source row.
@@ -721,13 +706,6 @@ test('sources comparison detects mutations', async ({ page, request }) => {
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for outputs'; });
expect(await paths()).toEqual(['rows.0.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for new outputs'; });
// D158: the inserted row is compared too: a missing one and a changed label are caught.
const undoForeign = await mutate(page, { remove: ['#workspace-main .setting-row:has(#foreign-runs)'] });
expect(await paths()).toEqual(expect.arrayContaining(['rows.length', 'surface.tree.length']));
await undoForeign();
await page.evaluate(() => { document.querySelector('#workspace-main label[for="foreign-runs"]').textContent = 'Save other prompts'; });
expect(await paths()).toEqual(['rows.1.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="foreign-runs"]').textContent = 'Save prompts from other workflows'; });
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = 'Add source'; });
expect(await paths()).toEqual(['heading.action.text']);
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = '+ Add source'; });
+10 -26
View File
@@ -1,7 +1,6 @@
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
import {test,expect} from './fixtures.mjs';
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
import {saveImageTags} from './concept-inventory.mjs';
import {readFile,writeFile} from 'node:fs/promises';
import {createHash} from 'node:crypto';
@@ -496,38 +495,23 @@ test('the inline current file names its prompts apart from the frozen run fields
expect(await exact(dialog(page).getByRole('region',{name:'Positive prompt',exact:true}).locator('p.prompt-text'))).toBe(before);
});
// #449 (D154), #457: History's inline current file draws the file's own image tags in the shared labelled section, named
// with its context so they stay apart from the frozen run's fields. The tags are saved on the real file through the Tag
// routes and come from History's own current-file read (its `image_tags` facet), as Images reads them.
test('the inline current file labels its image tags after its negative prompt',async({page,request})=>{
const reads=[];
// #449 (D154): History's inline current file draws tags in the shared labelled sections, named with its context so they
// stay apart from the frozen run's fields. The server sends History no image_tags facet today (D88), so the read is
// given one here to prove the shared render in History.
test('the inline current file labels its image tags after its negative prompt',async({page})=>{
await page.route(url=>url.pathname.endsWith('/inspection'),async route=>{const response=await route.fetch();const body=await response.json();
reads.push(body.image_tags);
body.prompts={positive:{status:'resolved',text:'current file positive',source:'graph-inferred'},negative:{status:'resolved',text:'current file negative',source:'graph-inferred'}};
body.image_tags={state:'current',tags:['history_tag']};
await route.fulfill({response,json:body});});
await open(page,page.fixture.detail_id);await marker(page,page.fixture.runs.shared).click();await expect(dialog(page).locator('[data-receipt]')).toHaveCount(5);
await dialog(page).getByRole('button',{name:'Raw',exact:true}).click();
await dialog(page).locator('[data-receipt]').first().getByRole('button',{name:'View current file',exact:true}).click();
await expect(page.locator('dialog#viewer-dialog')).toBeVisible();await page.keyboard.press('Escape');await expect(dialog(page)).toBeVisible();
const current=dialog(page).getByRole('region',{name:'Current reported file',exact:true});
const tags=current.getByRole('region',{name:'Current file image tags',exact:true});
const showCurrent=async()=>{
await open(page,page.fixture.detail_id);await marker(page,page.fixture.runs.shared).click();await expect(dialog(page).locator('[data-receipt]')).toHaveCount(5);
await dialog(page).getByRole('button',{name:'Raw',exact:true}).click();
await dialog(page).locator('[data-receipt]').first().getByRole('button',{name:'View current file',exact:true}).click();
await expect(page.locator('dialog#viewer-dialog')).toBeVisible();await page.keyboard.press('Escape');await expect(dialog(page)).toBeVisible();
await expect(current.getByRole('region',{name:'Current file negative prompt',exact:true})).toBeVisible();
};
// An untagged file draws no image tags section.
await showCurrent();await expect(tags).toHaveCount(0);expect(reads.at(-1)).toEqual({state:'none',tags:[]});
await dialog(page).getByRole('button',{name:'Close recorded run',exact:true}).click();
expect(await saveImageTags(request,'history/current.png',['history_tag','dusk'])).toEqual(['history_tag','dusk']);
await showCurrent();expect(reads.at(-1)).toEqual({state:'current',tags:['history_tag','dusk']});
await expect(tags.locator('.role-label')).toHaveText('Image tags · 2');await expect(tags.locator('.tags > .chip')).toHaveText(['history_tag','dusk']);
await expect(tags.locator('.role-label')).toHaveText('Image tags · 1');await expect(tags.locator('.tags > .chip')).toHaveText(['history_tag']);
const negative=current.getByRole('region',{name:'Current file negative prompt',exact:true});
expect(await negative.evaluate((node,section)=>({after:Boolean(node.compareDocumentPosition(section)&Node.DOCUMENT_POSITION_FOLLOWING),inside:node.contains(section)}),await tags.elementHandle())).toEqual({after:true,inside:false});
// The viewer opened from History shares the inspection: the same section, plain chips (no Tags filter there, D88).
await dialog(page).locator('[data-receipt]').first().getByRole('button',{name:'View current file',exact:true}).click();
const viewerTags=page.locator('dialog#viewer-dialog #viewer-details section.tag-section.image-tags');
await expect(viewerTags.locator('.role-label')).toHaveText('Image tags · 2');
await expect(viewerTags.locator('span.chip')).toHaveText(['history_tag','dusk']);await expect(viewerTags.locator('button')).toHaveCount(0);
await page.keyboard.press('Escape');await expect(dialog(page)).toBeVisible();
});
// #332 CUJ 5 (MC8, §5.3): History's current-file inspector reads the file's own category with the Images key rule, so
-141
View File
@@ -704,147 +704,6 @@ test('a video replaced while still starting is reset once it settles', async ({
}
});
test('a new selection while the viewer\'s own video is still starting leaves it until it settles (#483)', async ({ context, browserName }) => {
test.setTimeout(180000);
// D155 opens the viewer once the inline preview has settled; the viewer then starts a video of its own. A new
// selection from that video's own `loadstart` (readyState HAVE_NOTHING), or a few milliseconds after it, must not
// reset it or take it off the page while it starts (WebKit, GStreamer: TESTING.md D62; merge-queue run
// 40903238387). The viewer stays open until it reports metadata or an error, then closes; the URL goes last.
for (let attempt = 0; attempt < 10; attempt++) {
const page = await context.newPage();
try {
if (attempt === 0) await startCoverage(page, browserName);
await page.addInitScript(() => {
const load = HTMLMediaElement.prototype.load;
HTMLMediaElement.prototype.load = function () { if (!this.hasAttribute('src')) { this.__resets = (this.__resets ?? 0) + 1; this.__resetReadyState ??= this.readyState; } return load.call(this); };
const revoke = URL.revokeObjectURL;
URL.revokeObjectURL = url => {
const starting = window.__starting;
if (url === starting?.url) starting.revoked = { src: starting.node.hasAttribute('src'), connected: starting.node.isConnected, resets: starting.node.__resets ?? 0 };
return revoke(url);
};
});
await openInspect(page);
await page.evaluate(() => {
const input = document.querySelector('#local-file');
new MutationObserver((records, observer) => {
const video = document.querySelector('#viewer-dialog video');
if (!video) return;
observer.disconnect();
const replace = () => {
window.__starting = { node: video, url: video.getAttribute('src'), readyState: video.readyState };
input.files = window.__pngFiles; input.dispatchEvent(new Event('change', { bubbles: true }));
window.__starting.replaced = { connected: video.isConnected, src: video.hasAttribute('src'), resets: video.__resets ?? 0 };
};
video.addEventListener('loadstart', () => { if (window.__delay) setTimeout(replace, window.__delay); else replace(); }, { once: true });
}).observe(document.body, { childList: true, subtree: true });
});
const png = parityPng('f02.png');
await page.evaluate(bytes => {
const transfer = new DataTransfer();
transfer.items.add(new File([new Uint8Array(bytes)], 'f02.png', { type: 'image/png' }));
window.__pngFiles = transfer.files;
}, [...png.buffer]);
await page.evaluate(delay => { window.__delay = delay; }, attempt % 5);
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(preview(page).locator('img.local-file-preview')).toBeVisible({ timeout: 8000 });
await expect.poll(() => page.evaluate(() => window.__starting?.revoked ?? null)).toEqual({ src: false, connected: false, resets: 1 });
const { readyState, replaced } = await page.evaluate(() => ({ readyState: window.__starting.readyState, replaced: window.__starting.replaced }));
if (attempt % 5 === 0) expect(readyState).toBe(0);
// Starting when replaced: still on the page with its URL, never reset; settled: released at once.
expect(replaced).toEqual(readyState === 0 ? { connected: true, src: true, resets: 0 } : { connected: false, src: false, resets: 1 });
expect(await page.evaluate(() => window.__starting.node.__resets)).toBe(1);
// Its one reset came after it had metadata, never while it was starting.
expect(await page.evaluate(() => window.__starting.node.__resetReadyState)).toBeGreaterThanOrEqual(1);
// The new file's viewer opens after the old one closed, and the page still answers.
await expect(viewer(page).getByRole('img', { name: 'f02.png', exact: true })).toBeVisible({ timeout: 8000 });
await expect(viewer(page).locator('video')).toHaveCount(0);
if (attempt === 0) await saveCoverage(page, browserName);
} finally {
await page.close();
}
}
});
/** #483: holds the viewer's video in HAVE_NOTHING (its `loadedmetadata` never fires) so a hold can be driven by hand. */
async function stallViewerVideo(page) {
await page.addInitScript(() => {
const describe = Object.getOwnPropertyDescriptor(HTMLMediaElement.prototype, 'readyState');
Object.defineProperty(HTMLMediaElement.prototype, 'readyState', { configurable: true, get() {
return this.closest?.('#viewer-dialog') && window.__stallViewer ? 0 : describe.get.call(this);
} });
window.addEventListener('loadedmetadata', event => {
if (window.__stallViewer && event.target.closest?.('#viewer-dialog')) event.stopImmediatePropagation();
}, true);
});
}
test('a held viewer closed by hand, or replaced twice, opens only the latest file\'s viewer at once (#483)', async ({ page }) => {
await stallViewerVideo(page);
await openInspect(page);
await page.evaluate(() => { window.__stallViewer = true; });
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(viewer(page).locator('video')).toBeAttached();
// Two selections while the viewer's video is (made to look) still starting: the viewer stays, untouched.
await chooser(page).setInputFiles(parityPng('f02.png', 'first.png'));
await chooser(page).setInputFiles(parityPng('f02.png', 'second.png'));
await expect(preview(page).locator('img.local-file-preview')).toBeVisible();
await expect(viewer(page).locator('video')).toBeAttached();
// Open file metadata pressed meanwhile (the page behind the viewer is inert, so by script): the latest request.
await action(page).evaluate(node => node.click());
// Closing it by hand ends the hold at once: the latest file's viewer opens, once, well before D62's 15 s.
await page.keyboard.press('Escape');
await expect(viewer(page).getByRole('img', { name: 'second.png', exact: true })).toBeVisible({ timeout: 5000 });
await expect(viewer(page).getByRole('img', { name: 'first.png', exact: true })).toHaveCount(0);
await expect(viewer(page).locator('video')).toHaveCount(0);
await expect(page.locator('dialog[open]')).toHaveCount(1);
});
test('a held viewer revokes its URL only once it is gone, drops a stale open, and ends on page leave (#483)', async ({ page }) => {
await stallViewerVideo(page);
await openInspect(page);
await trackReleases(page);
await page.evaluate(() => { window.__stallViewer = true; });
const videoUrl = () => page.evaluate(() => document.querySelector('#viewer-dialog video')?.getAttribute('src') ?? null);
const released = url => page.evaluate(url => window.__released.filter(item => item.url === url), url);
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(viewer(page).locator('video')).toBeAttached();
const first = await videoUrl();
await chooser(page).setInputFiles(parityPng('f02.png', 'held.png'));
await expect(preview(page).locator('img.local-file-preview')).toBeVisible();
// The session asked to revoke the video's URL; it waits while the held viewer still plays it.
expect(await released(first)).toEqual([]);
// An Open asked for during the hold, then an AVI (which never opens the viewer by itself): the open for the earlier
// file is stale and is dropped when the hold ends.
await action(page).evaluate(node => node.click());
const avi = { name: 'clip.avi', mimeType: 'video/x-msvideo', buffer: Buffer.concat([Buffer.from('RIFF'), Buffer.alloc(4), Buffer.from('AVI LIST'), Buffer.alloc(64)]) };
await chooser(page).setInputFiles(avi);
await expect(page.locator('#workspace-toast')).toHaveText(TOAST);
await page.keyboard.press('Escape');
await expect(page.locator('dialog[open]')).toHaveCount(0);
await expect.poll(() => released(first)).toEqual([{ url: first, live: 0 }]);
// Page leave while held: the viewer goes and the deferred URL is revoked with nothing holding it.
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(viewer(page).locator('video')).toBeAttached();
const second = await videoUrl();
await chooser(page).setInputFiles(parityPng('f02.png', 'leave.png'));
await expect(viewer(page).locator('video')).toBeAttached();
expect(await released(second)).toEqual([]);
await page.evaluate(() => window.dispatchEvent(new PageTransitionEvent('pagehide', { persisted: true })));
await expect(page.locator('dialog[open]')).toHaveCount(0);
await expect.poll(() => released(second)).toEqual([{ url: second, live: 0 }]);
// Another route while held: the page's dispose ends the hold and revokes the URL with nothing holding it.
await page.evaluate(() => window.dispatchEvent(new PageTransitionEvent('pageshow', { persisted: true })));
await choose(page, localFixture('core-tags.webm', 'video/webm'));
await expect(viewer(page).locator('video')).toBeAttached();
const third = await videoUrl();
await chooser(page).setInputFiles(parityPng('f02.png', 'route.png'));
await expect(viewer(page).locator('video')).toBeAttached();
await page.evaluate(() => { window.location.hash = '#/library'; });
await expect(main(page).getByRole('heading', { level: 1, name: 'Prompt library', exact: true })).toBeVisible();
await expect.poll(() => released(third)).toEqual([{ url: third, live: 0 }]);
});
test('leaving for another route while a video starts waits for it to settle', async ({ context, browserName }) => {
test.setTimeout(180000);
// A WebKit (GStreamer) video detached before its pipeline had started deadlocked the page (TESTING.md D62); a
+7 -16
View File
@@ -37,15 +37,6 @@ test.afterEach(async ({ page, request, browserName }) => {
await expect(await request.post('/__test__/tagging/reset', { data: {} })).toBeOK();
});
async function openSettled(page, url) {
// The page reads its selected file's details as soon as it lists. A source change before that read is answered
// makes the page find the file again on its own, so the test's old-numbering locators are gone before they are
// used (#484): every test changes sources only after the read was answered.
const inspected = page.waitForResponse(item => new URL(item.url()).pathname.startsWith('/prompt_manager/images/inspect/') &&
item.request().method() === 'GET');
await page.goto(url);
expect((await inspected).status()).toBe(200);
}
async function openTag(page, path = FIRST, root = 0) {
const observed = answered(page, '/image-annotations/observe');
await tagButton(page, path, root).click();
@@ -62,7 +53,7 @@ async function savedTags(request, path, root) {
}
test('renumbered sources under an open Tag dialog: it finds the file again, keeps the draft and Generate works', async ({ page, request }) => {
await openSettled(page, images);
await page.goto(images);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
expect((await openTag(page)).status()).toBe(200);
await manual(page).fill('kept_draft');
@@ -100,7 +91,7 @@ test('renumbered sources under an open Tag dialog: it finds the file again, keep
});
test('the inspector finds a renumbered file again; checked files from the old numbering are cleared', async ({ page, request }) => {
await openSettled(page, images);
await page.goto(images);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
await card(page, FIRST).getByRole('checkbox', { name: /^Select / }).check();
await expect(page.locator('.bulkbar')).toBeVisible();
@@ -122,7 +113,7 @@ test('the inspector finds a renumbered file again; checked files from the old nu
});
test('a scope that cannot be mapped: Generate and Save are visibly disabled, the reason and Refresh sit beside them', async ({ page }) => {
await openSettled(page, images);
await page.goto(images);
await expect(card(page, FIRST)).toBeVisible();
await expect(await page.request.post('/__test__/scope-key/rotate')).toBeOK();
const rescoped = answered(page, '/images/rescope');
@@ -166,7 +157,7 @@ test('a scope that cannot be mapped: Generate and Save are visibly disabled, the
});
test('an unmappable inspector read keeps its visible Refresh, and Refresh recovers the panel', async ({ page }) => {
await openSettled(page, images);
await page.goto(images);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
await expect(await page.request.post('/__test__/scope-key/rotate')).toBeOK();
const rescoped = answered(page, '/images/rescope');
@@ -182,7 +173,7 @@ test('an unmappable inspector read keeps its visible Refresh, and Refresh recove
test('a Load more cursor the server cannot verify offers Refresh instead of a dead Retry', async ({ page, request }) => {
await expect(await request.post('/__test__/tagging/reset', { data: { extra_images: 60 } })).toBeOK();
await openSettled(page, images);
await page.goto(images);
await expect(cards(page).first()).toBeVisible();
await expect(page.locator('#gallery-continuation')).toContainText('more load as you scroll');
await expect(await request.post('/__test__/scope-key/rotate')).toBeOK();
@@ -200,7 +191,7 @@ test('a Load more cursor the server cannot verify offers Refresh instead of a de
});
test('a viewer whose media source was renumbered closes onto the file, kept selected', async ({ page, request }) => {
await openSettled(page, images);
await page.goto(images);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
await expect(await request.post('/__test__/sources/reverse')).toBeOK();
const rescoped = answered(page, '/images/rescope');
@@ -226,7 +217,7 @@ test('a collection page finds a renumbered member again and stays on its members
await expect(added).toBeOK();
const collections = (await (await request.get('/prompt_manager/collections')).json()).collections;
const { id } = collections.find(item => item.name === name);
await openSettled(page, `/prompt_manager/workspace#/collection/${id}`);
await page.goto(`/prompt_manager/workspace#/collection/${id}`);
await expect(cards(page)).toHaveCount(1);
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
await expect(await request.post('/__test__/sources/reverse')).toBeOK();
@@ -268,71 +268,6 @@ test('Watch for new outputs saves monitoring_enabled; a restart applies it', asy
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · watching']);
});
test('D158: Save prompts from other workflows saves save_foreign_runs live; off while monitoring is off (#247)', async ({ page, request }, testInfo) => {
try {
const layout = await openSources(page);
const rows = layout.locator('section.surface > .setting-row');
await expect(rows.locator(':scope > div > label')).toHaveText(['Watch for new outputs', 'Save prompts from other workflows', 'Worker threads']);
const foreign = layout.getByRole('switch', { name: 'Save prompts from other workflows', exact: true });
const foreignRow = rows.filter({ has: page.locator('#foreign-runs') });
await expect(foreignRow.locator(':scope > div > p')).toHaveText('New outputs of workflows without a PromptManager node add their prompt to the library when it is certain.');
await expect(foreign).toBeChecked();
await expect(foreign).toBeEnabled();
// Keyboard on desktop, a tap on the touch project.
if (testInfo.project.name === 'touch') await foreign.tap();
else {
await foreign.focus();
await page.keyboard.press('Space');
}
await expect(foreign).not.toBeChecked();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const response = await saved;
expect(response.request().postDataJSON()).toEqual(expect.objectContaining({ save_foreign_runs: false }));
expect((await response.json()).restart_required_fields).toEqual([]);
// Applied live: the monitor reads it for every file, so no restart is named.
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.save_foreign_runs).toBe(false);
// With Watch for new outputs drafted off nothing is watched: the row is disabled and its note says why.
const monitoring = layout.getByRole('switch', { name: 'Watch for new outputs', exact: true });
await monitoring.uncheck();
await expect(foreign).toBeDisabled();
const note = foreignRow.locator('p.small');
await expect(note).toHaveText('Applies while Watch for new outputs is on.');
await expect(foreign).toHaveAttribute('aria-describedby', await note.getAttribute('id'));
await monitoring.check();
await expect(foreign).toBeEnabled();
await expect(note).toHaveCount(0);
await expect(foreign).not.toHaveAttribute('aria-describedby', /./);
await expect(statusOf(page)).toHaveText('No unsaved changes.');
} finally {
// The server is shared by the spec's tests: put the default back whatever happened.
await request.post(SETTINGS_URL, { data: { save_foreign_runs: true } });
}
});
test('D158: saved off while the monitor still watches until a restart, the foreign row can still be set (D128)', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'] } })).toBeOK();
try {
const layout = await openSources(page);
await layout.getByRole('switch', { name: 'Watch for new outputs', exact: true }).uncheck();
await saveButton(page).click();
await expect(sourceList(page).locator('.operation p').first()).toHaveText('output · watching until restart');
const foreign = layout.getByRole('switch', { name: 'Save prompts from other workflows', exact: true });
await expect(foreign).toBeEnabled();
await expect(layout.locator('.setting-row').filter({ has: page.locator('#foreign-runs') }).locator('p.small'))
.toHaveText('Applies while Watch for new outputs is on.');
await foreign.uncheck();
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
expect((await saved).postDataJSON()).toEqual({ save_foreign_runs: false, expected_revision: expect.any(String) });
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
} finally {
await request.post(SETTINGS_URL, { data: { save_foreign_runs: true } });
}
});
test('worker range max is cpu_count', async ({ page, request }) => {
const { settings } = await serverJson(request, SETTINGS_URL);
const layout = await openSources(page);
-15
View File
@@ -309,21 +309,6 @@ test('fresh registered PM variants preserve exact raw whitespace and known empty
expect(value.negative.raw).toBe('');
expect(value.negative.effective).toBe('');
expect(value.evidence.positive.raw).toMatchObject({ provenance: 'recorded', source: 'runtime_return' });
// #438: the Library stores the stripped text; the run is in that prompt's history
expect(value.positive_prompt_id, 'the run names the prompt the Library saved').toEqual(expect.any(Number));
const runId = (await readExecutionRuns(execution.databasePath, id)).rows[0].run_id;
const response = await execution.request.get(`/prompt_manager/library/${value.positive_prompt_id}/history`);expect(response.ok()).toBe(true);
expect((await response.json()).items.map(item => item.run_id)).toContain(runId);
if (kind !== 'PromptManager') continue;
// #460: an upper-case re-queue counts toward the same prompt and joins its history
const upper = await submit(execution.request, recipeGraph({ kind, text: `${kind.toUpperCase()} E\u0301` }));
await history(execution.request, upper);
const variant = (await capture(execution, upper)).contexts[0];
expect(variant.positive.raw).toBe(`${kind.toUpperCase()} E\u0301`);
expect(variant.positive_prompt_id, 'the case variant names the same prompt').toBe(value.positive_prompt_id);
const variantRun = (await readExecutionRuns(execution.databasePath, upper)).rows[0].run_id;
const scoped = await execution.request.get(`/prompt_manager/library/${value.positive_prompt_id}/history`);
expect((await scoped.json()).items.map(item => item.run_id)).toContain(variantRun);
}
});
+2 -9
View File
@@ -334,15 +334,8 @@ async def image_monitor(request: web.Request) -> web.Response:
@PromptServer.instance.routes.get("/pm_execution_fixture/monitor")
async def image_monitor_state(request: web.Request) -> web.Response:
"""File names of images waiting for their run's recorded outputs: keyless
ones (``awaiting``) and foreign ones (``foreign``, #247)."""
"""File names of keyless images waiting for their run's recorded outputs."""
monitor = _plugin_module("utils.image_monitor")._monitor_instance
handler = getattr(monitor, "handler", None)
waiting = handler.awaiting_history() if handler is not None else []
foreign = handler.awaiting_foreign() if handler is not None else []
return web.json_response(
{
"awaiting": [os.path.basename(p) for p in waiting],
"foreign": [os.path.basename(p) for p in foreign],
}
)
return web.json_response({"awaiting": [os.path.basename(p) for p in waiting]})
+1 -64
View File
@@ -1,7 +1,6 @@
// #342 on the real pinned host: the image monitor links an output only to a prompt
// that provably produced it. An unknown node (PMHistoryCached) stands in for the
// maintainer's LLM; KSampler is the fixture's deterministic stand-in. #247: an output
// of a workflow without a PromptManager node gets its own prompt.
// maintainer's LLM; KSampler is the fixture's deterministic stand-in.
import { randomUUID } from 'node:crypto';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
@@ -142,68 +141,6 @@ test('an output saved without metadata links once its recorded run names a store
}
});
// #247: no PromptManager node at all, only ComfyUI's own text encoders, sampler and
// saver (the fixture's inputs and decoder stand in for the model files).
function foreignGraph({ text, seed = 1, prefix }) {
return {
'10': { class_type: 'PMHistoryInputs', inputs: {} },
'20': { class_type: 'CLIPTextEncode', inputs: { text, clip: ['10', 0] } },
'30': { class_type: 'CLIPTextEncode', inputs: { text: '', clip: ['10', 0] } },
'40': { class_type: 'KSampler', inputs: { model: ['10', 1], latent_image: ['10', 2], positive: ['20', 0],
negative: ['30', 0], seed, steps: 1, cfg: 1, sampler_name: 'euler', scheduler: 'normal', denoise: 1 } },
'60': { class_type: 'PMHistoryImages', inputs: { value: 64, batch: 1, latent: ['40', 0] } },
'70': { class_type: 'SaveImage', inputs: { images: ['60', 0], filename_prefix: prefix } },
};
}
test('an output of a workflow without a PromptManager node creates and links its prompt; a second run reuses it', async ({ execution }, testInfo) => {
const text = `foreign lighthouse at dusk ${testInfo.project.name} ${randomUUID()}`;
const first = await saved(execution.request, foreignGraph({ text, prefix: 'foreign/run' }));
await linkedTo(execution, first, text);
// Only the seed changes: a new file, the same text, so the same stored prompt.
const second = await saved(execution.request, foreignGraph({ text, seed: 2, prefix: 'foreign/run' }));
await linkedTo(execution, second, text);
const state = await readLinks(execution.databasePath, [first, second], [text]);
expect(state).toEqual({ links: { [first]: text, [second]: text }, stored: { [text]: true } });
// Decided from the graph at settle: nothing waits for the run's recorded outputs.
expect((await (await execution.request.get('/pm_execution_fixture/monitor')).json()).foreign).toEqual([]);
});
// CUJ 1: two branches saved under one prefix, so neither the graph nor the file name
// names a file's branch; the run's recorded saver does, once its outputs are sealed.
function sharedPrefixGraph(texts) {
const nodes = {
'10': { class_type: 'PMHistoryInputs', inputs: {} },
'30': { class_type: 'CLIPTextEncode', inputs: { text: '', clip: ['10', 0] } },
};
texts.forEach((text, index) => {
const base = (index + 1) * 100;
nodes[String(base)] = { class_type: 'CLIPTextEncode', inputs: { text, clip: ['10', 0] } };
nodes[String(base + 1)] = { class_type: 'KSampler', inputs: { model: ['10', 1], latent_image: ['10', 2],
positive: [String(base), 0], negative: ['30', 0], seed: index + 1, steps: 1, cfg: 1, sampler_name: 'euler',
scheduler: 'normal', denoise: 1 } };
nodes[String(base + 2)] = { class_type: 'PMHistoryImages', inputs: { value: 64, batch: 1, latent: [String(base + 1), 0] } };
nodes[String(base + 3)] = { class_type: 'SaveImage', inputs: { images: [String(base + 2), 0], filename_prefix: 'foreign/dataset' } };
});
return nodes;
}
test('each branch of a shared-prefix workflow without a PromptManager node links to its own prompt after the run', async ({ execution }, testInfo) => {
const tag = `${testInfo.project.name} ${randomUUID()}`;
const texts = [`dataset branch one harbor ${tag}`, `dataset branch two meadow ${tag}`];
const id = randomUUID();
const response = await execution.request.post('/prompt', { data: { prompt: sharedPrefixGraph(texts), prompt_id: id } });
expect(response.ok()).toBe(true);
let item;
await expect.poll(async () => {
item = (await (await execution.request.get(`/history/${id}`)).json())[id];
return item?.status?.messages?.some(([kind]) => kind === 'execution_success') || false;
}, { timeout: 15000 }).toBe(true);
const files = ['103', '203'].map(node => item.outputs[node].images[0].filename);
for (const [index, file] of files.entries()) await linkedTo(execution, file, texts[index]);
expect((await (await execution.request.get('/pm_execution_fixture/monitor')).json()).foreign).toEqual([]);
});
// #332 on the real host: the migration ran at startup and the image category routes
// set, list and remove an own category on a generated output.
test('an output gets its own image category and loses it again', async ({ execution }, testInfo) => {
-1
View File
@@ -192,7 +192,6 @@
"settings.monitoring_enabled",
"settings.result_timeout",
"settings.safety_backup",
"settings.save_foreign_runs",
"settings.thumbnail_edge",
"settings.thumbnail_format",
"settings.thumbnail_quality",
-17
View File
@@ -157,20 +157,3 @@ test('a valid configured source missing from the effective list is another spell
assert.deepEqual(sourceRows({ sources, saved, current: saved }).rows.at(-1), { key: 'output/.', title: 'output',
description: 'output/. · same folder as another source', scan: null, remove: 'output/.' });
});
test('D158: "Save prompts from other workflows" drafts save_foreign_runs, disabled while monitoring is off (#247)', async () => {
const { FOREIGN_RUNS, foreignRunsState } = await sourcesModule;
assert.deepEqual(FOREIGN_RUNS, { id: 'foreign-runs', key: 'save_foreign_runs', label: 'Save prompts from other workflows',
description: 'New outputs of workflows without a PromptManager node add their prompt to the library when it is certain.',
off: 'Applies while Watch for new outputs is on.' });
assert.deepEqual(foreignRunsState({ value: true, monitoring: true }), { disabled: false, checked: true, note: null });
assert.deepEqual(foreignRunsState({ value: false, monitoring: true }), { disabled: false, checked: false, note: null });
// Monitoring drafted or saved off: nothing is watched, so the row says why it is disabled and keeps its value.
assert.deepEqual(foreignRunsState({ value: true, monitoring: false }), { disabled: true, checked: true, note: FOREIGN_RUNS.off });
// Not read yet (or the host read failed): disabled without a value, the host rows' own note applies.
assert.deepEqual(foreignRunsState({ value: undefined, monitoring: undefined }), { disabled: true, checked: null, note: null });
assert.deepEqual(foreignRunsState({ value: undefined, monitoring: false }), { disabled: true, checked: null, note: null });
// Saved off while the running monitor still watches until the restart (D128 `stopping`): it can still be set.
assert.deepEqual(foreignRunsState({ value: true, monitoring: false, stopping: true }),
{ disabled: false, checked: true, note: FOREIGN_RUNS.off });
});
+2 -44
View File
@@ -28,10 +28,6 @@ stack shows which). A dump after the summary is a hang at interpreter exit
A limit is a measurement, not a dial: change it only with new measurements
written next to it (TESTING.md "Time limits and hangs").
PM_TEST_SLOWEST=N (opt-in, #487) lists the N slowest tests after the summary.
A test's time runs from its ``start:`` line to the next one, so it includes
the class and module fixtures in between, like the hang limit.
"""
import argparse
@@ -40,7 +36,6 @@ import os
import shutil
import sys
import tempfile
import time
import unittest
from pathlib import Path
@@ -68,57 +63,23 @@ def read_limit(name, default):
return int(text)
class TestClock:
"""Seconds from each test's start to the next start (or the run's end)."""
def __init__(self, clock=time.perf_counter):
self._clock = clock
self._current = None
self._started = 0.0
self.durations = []
def start(self, test_id):
self.stop()
self._current = test_id
self._started = self._clock()
def stop(self):
if self._current is not None:
self.durations.append((self._clock() - self._started, self._current))
self._current = None
def report(self, count):
"""Return the lines that list the ``count`` slowest tests."""
slowest = sorted(self.durations, key=lambda item: item[0], reverse=True)
lines = [f"slowest {min(count, len(slowest))} of {len(slowest)} tests:"]
lines += [f"{seconds:8.2f}s {test_id}" for seconds, test_id in slowest[:count]]
return lines
def make_result_class(stream, per_test_limit, shutdown_limit, slowest=0):
def make_result_class(stream, per_test_limit, shutdown_limit):
"""Build a TextTestResult that names each test and re-arms the watchdog."""
def arm(seconds):
faulthandler.dump_traceback_later(seconds, exit=True, file=stream)
clock = TestClock()
class WatchdogResult(unittest.TextTestResult):
def startTest(self, test):
# At default verbosity the progress dots share a line; start ours.
prefix = "\n" if self.dots else ""
stream.write(f"{prefix}start: {test.id()}\n")
stream.flush()
clock.start(test.id())
super().startTest(test)
arm(per_test_limit)
def stopTestRun(self):
clock.stop()
super().stopTestRun()
if slowest:
stream.write("\n".join(["", *clock.report(slowest), ""]))
stream.flush()
arm(shutdown_limit)
return WatchdogResult
@@ -177,7 +138,6 @@ def main(argv=None):
try:
per_test_limit = read_limit("PM_TEST_HANG_LIMIT", DEFAULT_PER_TEST_LIMIT)
shutdown_limit = read_limit("PM_TEST_SHUTDOWN_LIMIT", DEFAULT_SHUTDOWN_LIMIT)
slowest = read_limit("PM_TEST_SLOWEST", 0)
except LimitError as error:
stream.write(f"{error}\n")
return USAGE_EXIT
@@ -192,9 +152,7 @@ def main(argv=None):
verbosity=args.verbosity,
# As unittest.main does: show warnings unless -W/PYTHONWARNINGS set them.
warnings=None if sys.warnoptions else "default",
resultclass=make_result_class(
stream, per_test_limit, shutdown_limit, slowest
),
resultclass=make_result_class(stream, per_test_limit, shutdown_limit),
)
result = runner.run(suite)
finally:
+1 -4
View File
@@ -45,9 +45,7 @@ SEEDED_PATHS = (
"/srv/someone/renders",
"C:\\Users\\someone\\Documents\\renders",
"D:/Users/someone/prompts.db",
# A share on this machine: on Windows the report resolves every source,
# and an unknown server name made that wait about 2.9 s (#487).
"\\\\localhost\\pm-no-such-share\\someone\\renders",
"\\\\nas\\share\\someone\\renders",
)
_WINDOWS_ABSOLUTE = re.compile(r"^[A-Za-z]:[\\/]")
TOP_KEYS = {
@@ -87,7 +85,6 @@ CONFIG = {
"max_image_age_days",
"max_concurrent_processing",
"metadata_extraction_timeout",
"save_foreign_runs",
},
"integrations": {"lora_manager_enabled", "has_civitai_api_key"},
}
+3 -25
View File
@@ -18,14 +18,11 @@ from unittest.mock import patch
from aiohttp import FormData, web
from utils.hashing import generate_prompt_hash
try:
from test_api_images import ImageAPITestCase
except ImportError:
from tests.test_api_images import ImageAPITestCase
from database import restore_stage
from database.operations import PromptDatabase
from py.api.library import _encode, _scope
from tests.test_execution_context_database import bundle, snapshot
@@ -44,21 +41,6 @@ OTHER_SESSION = "aaaaaaaa-3333-4333-8333-333333333333"
MISSING_RUN = "bbbbbbbb-4444-4444-8444-444444444444"
@contextmanager
def no_wal_truncate_wait():
"""Restore with the restore notification held back, as these tests need.
The read under test keeps its snapshot open across the restore, so the
publish's WAL truncate can never succeed; without the short wait it would
sit out the full CHECKPOINT_BUSY_MS (30 s) each time (#487).
"""
with (
patch("database.execution_history.notify_history_restore"),
patch.object(restore_stage, "CHECKPOINT_BUSY_MS", 50),
):
yield
class HistoryAPITests(ImageAPITestCase):
async def tearDownAsync(self):
if hasattr(self.api, "close_history_reads"):
@@ -209,11 +191,7 @@ class HistoryAPITests(ImageAPITestCase):
context["evidence"]["positive"]["raw"]["provenance"], "recorded"
)
with self.api.db.model.get_connection() as conn:
# An edit changes the text and its dedup hash together (#460)
conn.execute(
"UPDATE prompts SET text='edited',hash=? WHERE id=?",
(generate_prompt_hash("edited"), positive),
)
conn.execute("UPDATE prompts SET text='edited' WHERE id=?", (positive,))
_, changed = await self.detail(start.run_id)
self.assertEqual(changed["contexts"][0]["positive"], context["positive"])
self.assertFalse(changed["contexts"][0]["navigation"]["positive_available"])
@@ -362,7 +340,7 @@ class HistoryAPITests(ImageAPITestCase):
def during_snapshot(*args, **kwargs):
result = original(*args, **kwargs)
with no_wal_truncate_wait():
with patch("database.execution_history.notify_history_restore"):
writer.model.restore_from_file(source)
return result
@@ -906,7 +884,7 @@ class HistoryAPITests(ImageAPITestCase):
def during_snapshot(*args, **kwargs):
result = original(*args, **kwargs)
with no_wal_truncate_wait():
with patch("database.execution_history.notify_history_restore"):
writer.model.restore_from_file(source)
return result
-116
View File
@@ -1015,122 +1015,6 @@ class HistoryOutputAPITests(ImageAPITestCase):
response = await self.current(descriptor["inspection_url"])
self.assert_error(response, await response.json(), 422, "response-limit")
async def tag_image(self, relative, tags):
"""Save *tags* on Images root 0's *relative* file through the Tag routes.
The real observe and save routes store the locator the Tag dialog
stores (``media_capture._resolve``), so History must find it (#457).
"""
self.addCleanup(self.api.close_annotation_captures)
roots = self.api._get_all_output_dirs()
target = {
"root_index": 0,
"root_scope": self.api._gallery_root_scope(roots[0], 0),
"path": relative,
}
response = await self.client.post(
"/prompt_manager/image-annotations/observe", json={"target": target}
)
observed = await response.json()
self.assertEqual(response.status, 200, observed)
token = observed["capture"]["token"]
try:
response = await self.client.put(
"/prompt_manager/image-annotations/tags",
json={
"target": target,
"capture_token": token,
"sha256": observed["capture"]["sha256"],
"expected_epoch": observed["library_epoch"],
"expected_tag_revision": observed["tag_revision"],
"tags": list(tags),
},
)
saved = await response.json()
self.assertEqual(response.status, 200, saved)
finally:
await self.client.delete(
f"/prompt_manager/image-annotations/capture/{token}"
)
return saved["tags"]
async def test_current_inspection_carries_the_files_image_tags(self):
"""#457: History reads the file's own image tags, as Images does (S1)."""
sub = self.output_dir / "batch"
sub.mkdir()
tagged = png(sub / "tagged.png", prompt=graph("Tagged file"))
png(self.output_dir / "plain.png", prompt=graph("Plain file"))
self.assertEqual(
await self.tag_image("batch/tagged.png", ["dunes", "warm_light"]),
["dunes", "warm_light"],
)
start = self.capture(
[
self.ref("1", subfolder="batch", filename="tagged.png"),
self.ref("2", filename="plain.png"),
]
)
_, page = await self.outputs(start)
urls = [
item["current_candidate"]["media"]["inspection_url"]
for item in page["items"]
]
bodies = []
for url in urls:
response = await self.current(url)
body = await response.json()
self.assertEqual(response.status, 200, body)
bodies.append(body)
self.assertEqual(
bodies[0].get("image_tags"),
{"state": "current", "tags": ["dunes", "warm_light"]},
)
# Image tags are the file's own search metadata, never prompt tags.
self.assertEqual(bodies[0]["prompts"]["positive"]["text"], "Tagged file")
self.assertEqual(bodies[1].get("image_tags"), {"state": "none", "tags": []})
# New bytes at the same path: the tags belong to the old byte record.
stat = tagged.stat()
os.utime(tagged, ns=(stat.st_atime_ns, stat.st_mtime_ns + 10**9))
response = await self.current(urls[0])
body = await response.json()
self.assertEqual(response.status, 200, body)
self.assertEqual(body["image_tags"], {"state": "none", "tags": []})
async def test_current_inspection_finds_image_tags_under_a_symlinked_output(self):
"""An output folder reached through a link (another drive) still matches."""
link = self.comfy_dir / "linked_output"
self._make_symlink(self.output_dir, link, target_is_directory=True)
self.api._get_all_output_dirs = lambda: [link]
self.host_output = str(link)
png(self.output_dir / "image.png", prompt=graph("Linked folder"))
self.assertEqual(await self.tag_image("image.png", ["dunes"]), ["dunes"])
start = self.capture([self.ref()])
descriptor = await self.media(start)
response = await self.current(descriptor["inspection_url"])
body = await response.json()
self.assertEqual(response.status, 200, body)
self.assertEqual(body["image_tags"], {"state": "current", "tags": ["dunes"]})
async def test_current_inspection_image_tag_failure_is_unavailable(self):
png(self.output_dir / "image.png", prompt=graph("File evidence"))
await self.tag_image("image.png", ["dunes"])
start = self.capture([self.ref()])
descriptor = await self.media(start)
with (
self.assertLogs(self.api.logger, "WARNING") as logs,
patch.object(
type(self.api.db),
"current_image_tags",
side_effect=sqlite3.OperationalError("locked"),
),
):
response = await self.current(descriptor["inspection_url"])
body = await response.json()
self.assertEqual(response.status, 200, body)
self.assertEqual(body["image_tags"], {"state": "unavailable", "tags": []})
self.assertEqual(body["prompts"]["positive"]["text"], "File evidence")
self.assertNotIn(str(self.output_dir), "\n".join(logs.output))
async def test_optional_candidate_failure_preserves_every_occurrence_without_urls(
self,
):
+8 -36
View File
@@ -35,7 +35,6 @@ from aiohttp import web # noqa: E402
from aiohttp.test_utils import AioHTTPTestCase # noqa: E402
from PIL import Image # noqa: E402
from database.models import PromptModel # noqa: E402
from database.operations import PromptDatabase # noqa: E402
import py.api.images as images_module
from py.api import gallery_listing # noqa: E402
@@ -93,41 +92,14 @@ class ImageAPITestCase(AioHTTPTestCase):
self._anchor_patch.stop()
# Windows refuses to unlink a database that still has open handles.
self.api.db.close_all()
try:
leaked = self._close_leaked_connections()
finally:
for path in (
self._temp_db.name,
self._temp_db.name + "-wal",
self._temp_db.name + "-shm",
):
if os.path.exists(path):
os.unlink(path)
self._output_tmp.cleanup()
if leaked:
raise AssertionError(
f"{leaked} connection(s) to the test database were still open at "
"teardown; Windows cannot unlink it (WinError 32). Close every "
"extra PromptDatabase before tearDownAsync, not in a cleanup."
)
def _close_leaked_connections(self):
"""Close and count connections other models still hold on the temp DB.
Linux unlinks an open SQLite file, Windows does not; counting here makes
a handle left open past teardown fail on every OS (#475).
"""
key = os.path.normcase(os.path.realpath(self._temp_db.name))
with PromptModel._instances_lock:
models = list(PromptModel._instances)
leaked = 0
for model in models:
if model._schema_key() != key:
continue
with model._conn_lock:
leaked += len(model._connections)
model.close_all()
return leaked
for path in (
self._temp_db.name,
self._temp_db.name + "-wal",
self._temp_db.name + "-shm",
):
if os.path.exists(path):
os.unlink(path)
self._output_tmp.cleanup()
# -- helpers ---------------------------------------------------------
+3 -22
View File
@@ -38,25 +38,11 @@ def _payload(token):
class TestRestartKeepsTheOpenPage(ImageAPITestCase):
async def get_application(self):
self._restarts = []
app = await super().get_application()
for index in range(3):
make_png(self.output_dir / f"image_{index}.png")
return app
async def tearDownAsync(self):
# Before the base teardown unlinks the database: cleanups run after
# tearDownAsync, and Windows cannot unlink a file still open (#475).
errors = []
while self._restarts:
try:
await self._restarts.pop()()
except Exception as error: # every closer runs; the first error wins
errors.append(error)
await super().tearDownAsync()
if errors:
raise errors[0]
async def first_page(self, client=None, **params):
response = await (client or self.client).get(
LISTING, params={"mode": "workspace", "limit": "1", **params}
@@ -81,7 +67,7 @@ class TestRestartKeepsTheOpenPage(ImageAPITestCase):
api.close_auto_thumbnails()
api.db.close_all()
self._restarts.append(close)
self.addAsyncCleanup(close)
return api, client
async def test_a_restart_keeps_every_held_scope_and_cursor_valid(self):
@@ -166,13 +152,8 @@ class TestRestartKeepsTheOpenPage(ImageAPITestCase):
other_db = self.comfy_dir / "elsewhere" / "prompts.db"
other_db.parent.mkdir()
api = PromptManagerAPI(db=PromptDatabase(str(other_db)))
async def close():
api.close_auto_thumbnails()
api.db.close_all()
# Closed before teardown removes the folder that holds its database.
self._restarts.append(close)
self.addCleanup(api.db.close_all)
self.addCleanup(api.close_auto_thumbnails)
self.assertNotEqual(
api._gallery_root_scope(self.output_dir, 0), page["images"][0]["root_scope"]
)
-22
View File
@@ -59,28 +59,6 @@ class TestGalleryConfig(unittest.TestCase):
self.assertFalse(GalleryConfig.MONITORING_ENABLED)
self.assertEqual(GalleryConfig.PROCESSING_DELAY, 5.0)
def test_save_foreign_runs_defaults_on_and_only_booleans_apply(self):
"""#247: gallery.monitoring.save_foreign_runs, default true."""
self.assertIs(
GalleryConfig.get_config()["monitoring"]["save_foreign_runs"], True
)
GalleryConfig.update_config({"monitoring": {"save_foreign_runs": False}})
self.assertIs(GalleryConfig.SAVE_FOREIGN_RUNS, False)
for junk in ("true", 1, None):
with self.subTest(value=junk):
with self.assertLogs("prompt_manager.config", "WARNING") as logs:
GalleryConfig.update_config(
{"monitoring": {"save_foreign_runs": junk}}
)
self.assertIn(
"Ignoring gallery monitoring save_foreign_runs: expected true "
"or false",
"\n".join(logs.output),
)
self.assertIs(GalleryConfig.SAVE_FOREIGN_RUNS, False)
GalleryConfig.update_config({"monitoring": {"save_foreign_runs": True}})
self.assertIs(GalleryConfig.SAVE_FOREIGN_RUNS, True)
def test_update_tracking(self):
GalleryConfig.update_config({"tracking": {"prompt_timeout": 300}})
self.assertEqual(GalleryConfig.PROMPT_TIMEOUT, 300)
+2 -133
View File
@@ -15,7 +15,6 @@ from database import execution_contexts
from database.execution_history import HistoryEpochChanged
from database.execution_context_schema import context_schema_error
from database.execution_contexts import LIVE_HINT_PROMPT_IDS_SQL
from utils.hashing import generate_prompt_hash
from database.execution_history_schema import rotate_history_epoch
from utils.execution_contexts import (
ContextBundle,
@@ -467,11 +466,7 @@ class ContextDatabaseTests(unittest.TestCase):
with conn:
conn.execute("DELETE FROM prompts WHERE id=3")
self.assertEqual(assert_same(), {1, 2, 4})
# A case-only edit keeps the prompt's hash, so the run still counts
# toward it (#460); another text drops the hint.
db.update_prompt_text(4, "Other")
self.assertEqual(assert_same(), {1, 2, 4})
db.update_prompt_text(4, "another")
self.assertEqual(assert_same(), {1, 2})
with conn:
conn.execute("INSERT INTO prompts(id,text) VALUES(5,'none')")
@@ -479,131 +474,6 @@ class ContextDatabaseTests(unittest.TestCase):
rotate_history_epoch(conn)
self.assertEqual(assert_same(), set())
def test_hints_resolve_the_stored_form_of_outer_whitespace_raw_text(self):
# #438: the Library stores text.strip(); the raw run text keeps its spaces.
db = self.database()
conn = db.model.get_connection()
with conn:
conn.execute(
"INSERT INTO prompts(id,text) VALUES(1,'positive'),(2,'negative')"
)
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(7,1,2)"
)
start = self.capture(db, 0, bundle(" negative\n", positive="positive\n\t"))
context = db.get_execution_contexts(start.run_id)["contexts"][0]
self.assertEqual(context["positive"]["raw"], "positive\n\t")
self.assertEqual(
(
context["positive_prompt_id"],
context["negative_prompt_id"],
context["pair_id"],
),
(1, 2, 7),
)
live = {row[0] for row in conn.execute(LIVE_HINT_PROMPT_IDS_SQL)}
self.assertEqual(live, {1, 2})
stored = conn.execute(
"SELECT positive_prompt_id FROM execution_run_contexts WHERE run_id=?",
(start.run_id,),
).fetchone()[0]
self.assertEqual(stored, 1)
# A legacy row that kept outer whitespace keeps its exact-text hint and
# with it the recorded-run protection from pruning.
with conn:
conn.execute("INSERT INTO prompts(id,text) VALUES(3,' legacy ')")
legacy = self.capture(db, 1, bundle("negative", positive=" legacy "))
hinted = db.get_execution_contexts(legacy.run_id)["contexts"][0]
self.assertEqual(hinted["positive_prompt_id"], 3)
live = {row[0] for row in conn.execute(LIVE_HINT_PROMPT_IDS_SQL)}
self.assertEqual(live, {1, 2, 3})
self.assertEqual(db.prune_orphaned_prompts(), 0)
# Both forms present make the hint ambiguous, in both rule forms.
with conn:
conn.execute("INSERT INTO prompts(id,text) VALUES(4,'legacy')")
hinted = db.get_execution_contexts(legacy.run_id)["contexts"][0]
self.assertIsNone(hinted["positive_prompt_id"])
live = {row[0] for row in conn.execute(LIVE_HINT_PROMPT_IDS_SQL)}
self.assertEqual(live, {1, 2})
def test_hints_follow_the_hash_for_case_variant_raw_text(self):
# #460: the run names the prompt the Library counted it toward.
db = self.database()
first = db.save_prompt(
"\u00e9t\u00e9 positive",
prompt_hash=generate_prompt_hash("\u00e9t\u00e9 positive"),
)
second = db.save_prompt(
"negative \u03c3", prompt_hash=generate_prompt_hash("negative \u03c3")
)
with db.model.get_connection() as conn:
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(7,?,?)",
(first, second),
)
start = self.capture(
db, 0, bundle("NEGATIVE \u03a3", positive="\u00c9T\u00c9 Positive")
)
context = db.get_execution_contexts(start.run_id)["contexts"][0]
self.assertEqual(
(
context["positive_prompt_id"],
context["negative_prompt_id"],
context["pair_id"],
),
(first, second, 7),
)
conn = db.model.get_connection()
live = {row[0] for row in conn.execute(LIVE_HINT_PROMPT_IDS_SQL)}
self.assertEqual(live, {first, second})
# A prompt that only shares SQLite's ASCII lower() is not a match
other = db.save_prompt("other", prompt_hash=generate_prompt_hash("other"))
start = self.capture(db, 1, bundle("x", positive="\u00d6THER"))
context = db.get_execution_contexts(start.run_id)["contexts"][0]
self.assertIsNone(context["positive_prompt_id"])
self.assertNotIn(
other, {row[0] for row in conn.execute(LIVE_HINT_PROMPT_IDS_SQL)}
)
def test_hash_is_only_a_fallback_so_a_text_hint_stays_unambiguous(self):
# #460 review: a hash match must never make a #438 text hint ambiguous,
# or its prompt would lose the recorded-run prune protection.
db = self.database()
conn = db.model.get_connection()
with conn:
# A legacy row without a hash, and a case variant with the hash
conn.execute("INSERT INTO prompts(id,text) VALUES(1,'a lighthouse')")
conn.execute(
"INSERT INTO prompts(id,text,hash) VALUES(2,'A Lighthouse',?)",
(generate_prompt_hash("a lighthouse"),),
)
exact = self.capture(db, 0, bundle("x", positive="a lighthouse"))
variant = self.capture(db, 1, bundle("x", positive="A LIGHTHOUSE"))
hints = [
db.get_execution_contexts(start.run_id)["contexts"][0]["positive_prompt_id"]
for start in (exact, variant)
]
self.assertEqual(hints, [1, 2])
live = {row[0] for row in conn.execute(LIVE_HINT_PROMPT_IDS_SQL)}
self.assertEqual(live, {1, 2})
self.assertEqual(db.prune_orphaned_prompts(), 0)
def test_live_hint_prompt_lookups_use_indexes(self):
db = self.database()
conn = db.model.get_connection()
plan = [
row[3]
for row in conn.execute("EXPLAIN QUERY PLAN " + LIVE_HINT_PROMPT_IDS_SQL)
]
prompt_steps = [
step for step in plan if " p " in f" {step} " or "TABLE p" in step
]
self.assertTrue(prompt_steps, plan)
self.assertEqual([step for step in plan if step.startswith("SCAN p")], [])
self.assertEqual([step for step in plan if step.startswith("SCAN q")], [])
def test_hints_require_exact_unique_raw_text_and_never_refresh(self):
db = self.database()
conn = db.model.get_connection()
@@ -634,13 +504,12 @@ class ContextDatabaseTests(unittest.TestCase):
db.get_execution_contexts(fresh.run_id)["contexts"][0]["positive_prompt_id"]
)
def test_null_negative_pair_only_matches_a_known_stored_empty(self):
def test_null_negative_pair_only_matches_known_exact_empty(self):
db = self.database()
with db.model.get_connection() as conn:
conn.execute("INSERT INTO prompts(id,text) VALUES(1,'positive')")
conn.execute("INSERT INTO prompt_pairs(id,positive_prompt_id) VALUES(7,1)")
# A whitespace-only negative is the stored empty one (#438)
for item, raw, expected in ((0, "", 7), (1, " \n", 7), (2, None, None)):
for item, raw, expected in ((0, "", 7), (1, " ", None), (2, None, None)):
candidate = bundle(negative=raw or "")
if raw is None:
ctx = candidate.contexts[0]
-40
View File
@@ -19,7 +19,6 @@ from unittest.mock import patch
from tests.history_waits import drain_writer
from database import PromptDatabase
from utils import usage_tracking
from utils.hashing import generate_prompt_hash
from utils.execution_history_writer import HistoryWriter
try:
@@ -917,42 +916,3 @@ class CachedRuntimeHistoryTests(HooksCase):
from database.run_attribution import named_prompt
self.assertIsNone(named_prompt(self.db, self.rows()[1]["run_id"]))
class CachedRuntimeCaseHistoryTests(CachedRuntimeHistoryTests):
"""Raw text that differs only in case joins the saved prompt's history (#460).
The hash folds case, so the run counts toward the lower-case prompt saved
first; the run keeps its exact raw spelling.
"""
TEXT = "\u00c9 LIGHTHOUSE the LLM wrote"
def setUp(self):
super().setUp()
self.db.delete_prompt(self.prompt_id)
lower = self.TEXT.lower()
self.prompt_id = self.db.save_prompt(
text=lower, prompt_hash=generate_prompt_hash(lower)
)
def test_the_library_counts_the_case_variant_as_this_prompt(self):
self.assertEqual(
self.db.get_prompt_by_hash(generate_prompt_hash(self.TEXT))["id"],
self.prompt_id,
)
class CachedRuntimeWhitespaceHistoryTests(CachedRuntimeHistoryTests):
"""Raw text with outer whitespace joins the history of the prompt it saves (#438).
The Library stores ``"a lighthouse the LLM wrote"``; the run keeps the exact
frozen raw text, newline included, yet is a tracked run of that prompt.
"""
TEXT = "\u3000 a lighthouse the LLM wrote\n"
def test_the_library_stores_the_stripped_text(self):
self.assertEqual(
self.db.get_prompt_by_id(self.prompt_id)["text"], self.TEXT.strip()
)
-330
View File
@@ -1,330 +0,0 @@
"""#247 PR 2 x #90: the monitor's foreign path never stores switched-off text.
A file whose run's prompt has Save to library off is the opted-out path's,
not the foreign path's; a foreign graph whose ladder decision is
``not_saved`` (a switched-off node the #90 walk does not reach) stores
nothing at settle, at the re-check after the seal, or from recorded history,
and leaves no wait behind.
"""
import os
import unittest
from tests import saver_anchor_graphs as shapes
from tests import test_run_attribution as attribution
from tests import test_save_to_library as save_tests
from tests.test_pair_scan import sampled_graph
from utils.foreign_runs import ForeignRecheck
from utils.hashing import generate_prompt_hash
from py.config import GalleryConfig # after tests: it stubs the host
LOGGER = "prompt_manager.image_monitor"
OFF_TEXT = "SECRET OFF TEXT"
SKIPPED = "Save to library is off for this run's prompt, skipping image"
# The class dict, not the class: a TestCase bound here would run again
Fixture = vars(attribution.TestMonitorForeignRuns)
def off_behind(middle="MyNode", key="value"):
"""A switched-off PromptManagerText reaching a core CLIPTextEncode through
``middle``'s ``key`` input: the #90 walk does not follow ``key``."""
graph = save_tests.run_graph()
graph["1"] = {
"class_type": "PromptManagerText",
"inputs": {"text": OFF_TEXT, "save_to_library": False},
}
graph["7"] = {"class_type": middle, "inputs": {key: ["1", 0]}}
graph["8"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["7", 0], "clip": ["4", 1]},
}
graph["3"]["inputs"]["positive"] = ["8", 0]
return graph
def off_text_unwired():
"""Only history can decide; the legacy answer is a dangling switched-off
node's own text (the ladder says ``not_saved`` once it may ask history)."""
graph = sampled_graph("unused", "blurry")
graph["1"]["inputs"]["text"] = ["30", 0]
graph["30"] = {"class_type": "VendorPromptMaker", "inputs": {"prompt": OFF_TEXT}}
graph["40"] = {
"class_type": "PromptManagerText",
"inputs": {"text": OFF_TEXT, "save_to_library": False},
}
return graph
class TestForeignPathKeepsSaveToLibrary(attribution.RecordedRuns, unittest.TestCase):
setUp = Fixture["setUp"]
close_databases = Fixture["close_databases"]
wait_for = Fixture["wait_for"]
image = Fixture["image"]
linked_text = Fixture["linked_text"]
key = Fixture["key"]
prompt_count = Fixture["prompt_count"]
settled = Fixture["settled"]
def reference(self, path):
return {
"folder_type": "output",
"subfolder": "dataset",
"filename": os.path.basename(path),
}
def seal(self, path, positives=(), nodes=("9",)):
"""Record one run reporting ``path`` (a new run id for every call)."""
self.runs = getattr(self, "runs", -1) + 1
self.record_run(
self.db,
self.output,
[("dataset", os.path.basename(path))],
list(positives),
item=self.runs,
nodes=list(nodes),
)
def recheck(self, path):
"""The re-check the seal schedules, run through the monitor's entry."""
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(path, ForeignRecheck(self.reference(path)))
return "\n".join(logs.output)
def assert_nothing_stored(self, path):
self.assertIsNone(self.linked_text(path))
self.assertEqual(self.prompt_count(), 0)
self.assertIsNone(self.db.get_prompt_by_hash(generate_prompt_hash(OFF_TEXT)))
self.assertEqual(self.handler.awaiting_foreign(), [])
self.assertEqual(list(self.handler._foreign_due), [])
def test_q1_the_only_prompt_manager_node_off_is_opted_out_not_foreign(self):
graph = save_tests.run_graph(save=False)
# The switched-off positive is the only PromptManager node
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": "blurry"}}
sampler_less = {
"1": {
"class_type": "PromptManagerText",
"inputs": {"text": OFF_TEXT, "save_to_library": False},
}
}
for name, graph in (("sampled", graph), ("sampler-less", sampler_less)):
with self.subTest(name=name):
self.metadata = {"prompt": graph}
state, _ = self.handler._find_prompt_from_metadata(self.metadata)
self.assertEqual(state, "opted_out")
path = self.image(f"IMG_GEN_{name}_00001_.png")
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(path)
self.assertIn(SKIPPED, "\n".join(logs.output))
self.seal(path)
self.assertTrue(self.settled())
self.assertIn(SKIPPED, self.recheck(path))
self.assert_nothing_stored(path)
def test_q2_an_off_text_node_behind_a_pass_through_stores_nothing(self):
for middle, key in (("MyNode", "value"), ("easy showAnything", "anything")):
with self.subTest(middle=middle):
self.metadata = {"prompt": off_behind(middle, key)}
state, _ = self.handler._find_prompt_from_metadata(self.metadata)
self.assertEqual(state, "foreign")
path = self.image(f"IMG_GEN_{key}_00001_.png")
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(path)
self.assertIn(SKIPPED, "\n".join(logs.output))
self.assert_nothing_stored(path)
self.seal(path)
self.assertTrue(self.settled())
self.assertIn(SKIPPED, self.recheck(path))
self.assert_nothing_stored(path)
def test_q2_a_wired_switch_behind_a_pass_through_stores_nothing(self):
graph = off_behind()
graph["1"]["inputs"]["save_to_library"] = ["20", 0]
graph["20"] = {"class_type": "SomeBoolNode", "inputs": {}}
self.metadata = {"prompt": graph}
path = self.image("IMG_GEN_wired_00001_.png")
wired = "Save to library is wired from a node the image cannot show"
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(path)
self.assertIn(wired, "\n".join(logs.output))
self.seal(path)
self.assertTrue(self.settled())
self.assertIn(wired, self.recheck(path))
self.assert_nothing_stored(path)
def test_q3_an_undecided_not_saved_waits_for_the_record_then_skips(self):
"""No record yet: a not_saved of the undecided step is not final (a
recorded saver may resolve another branch), so the file waits for its
run; the decision with the record stores nothing and logs the skip."""
wired = off_text_unwired()
wired["40"]["inputs"]["save_to_library"] = ["20", 0]
wired["20"] = {"class_type": "SomeBoolNode", "inputs": {}}
cases = (
("off", off_text_unwired(), SKIPPED),
("wired", wired, "Save to library is wired from a node"),
)
for name, graph, line in cases:
with self.subTest(name=name):
self.metadata = {"prompt": graph}
path = self.image(f"vendor_{name}_00001_.png")
self.handler.process_new_image(path)
self.assertIsNone(self.linked_text(path))
self.assertEqual(self.handler.awaiting_foreign(), [self.key(path)])
with self.assertLogs(LOGGER, "INFO") as logs:
self.seal(path)
self.assertTrue(
self.wait_for(lambda: not self.handler.awaiting_foreign())
)
self.assertTrue(self.settled())
self.assertIn(line, "\n".join(logs.output))
self.assert_nothing_stored(path)
def test_q3_an_early_not_saved_never_waits_even_with_a_saver_branch(self):
"""Review: a switched-off node feeding one branch's positive is the
ladder's early, final not_saved (PR 1's #90 rule covers every
sampler's positive), so the file never waits and nothing is stored,
not even the plain branch the recorded saver names."""
graph = shapes.shared_prefix()
graph["20"]["inputs"]["text"] = ["50", 0]
graph["50"] = {
"class_type": "PromptManagerText",
"inputs": {"text": OFF_TEXT, "save_to_library": False},
}
self.metadata = {"prompt": graph}
path = self.image("IMG_GEN_00001_.png")
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(path)
self.assertIn(SKIPPED, "\n".join(logs.output))
self.assert_nothing_stored(path)
self.seal(path, nodes=("13",))
self.assertTrue(self.settled())
self.assertIn(SKIPPED, self.recheck(path))
self.assert_nothing_stored(path)
self.assertIsNone(
self.db.get_prompt_by_hash(generate_prompt_hash(shapes.BRANCH_TEXTS[0]))
)
def test_q3_an_off_node_in_the_recorded_saver_branch_is_never_foreign(self):
"""PR 3's parity case: a switched-off node feeds the very branch the
recorded saver names (saver "23" is branch 2). A literal switch is the
opted-out path's; a wired one is the node's own run's (#90), never the
foreign path's. Nothing is stored at settle or after the run, and no
file waits for a foreign re-check."""
cases = (
("off", False, "opted_out", SKIPPED),
("wired", ["52", 0], "unknown", "No certain prompt for"),
)
for name, save, state, line in cases:
with self.subTest(name=name):
graph = shapes.shared_prefix()
graph["50"] = {
"class_type": "PromptManagerText",
"inputs": {"text": OFF_TEXT, "save_to_library": save},
}
graph["20"]["inputs"]["text"] = ["50", 0]
if isinstance(save, list):
graph["52"] = {"class_type": "SomeBoolNode", "inputs": {}}
self.metadata = {"prompt": graph}
found, _ = self.handler._find_prompt_from_metadata(self.metadata)
self.assertEqual(found, state)
path = self.image(f"IMG_GEN_{name}_00001_.png")
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(path)
self.assertIn(line, "\n".join(logs.output))
self.assert_nothing_stored(path)
self.seal(path, positives=[OFF_TEXT], nodes=("23",))
self.assertTrue(self.settled())
self.handler.process_new_image(path)
self.assert_nothing_stored(path)
self.assertIsNone(
self.db.get_prompt_by_hash(
generate_prompt_hash(shapes.BRANCH_TEXTS[1])
)
)
def test_q4_an_off_node_on_the_negative_only_still_links_the_positive(self):
positive = "a lighthouse on a cliff at dusk"
plain = sampled_graph(positive, "unused")
plain["2"] = {
"class_type": "PromptManager",
"inputs": {"text": OFF_TEXT, "save_to_library": False},
}
behind = sampled_graph(positive, "unused")
behind["2"]["inputs"]["text"] = ["7", 0]
behind["7"] = {
"class_type": "PromptManagerText",
"inputs": {"text": OFF_TEXT, "save_to_library": False},
}
for name, graph in (("direct", plain), ("behind an encoder", behind)):
with self.subTest(name=name):
self.metadata = {"prompt": graph}
state, _ = self.handler._find_prompt_from_metadata(self.metadata)
self.assertEqual(state, "foreign")
path = self.image(f"plain_{len(name)}_00001_.png")
self.handler.process_new_image(path)
self.assertEqual(self.linked_text(path), positive)
self.assertIsNone(
self.db.get_prompt_by_hash(generate_prompt_hash(OFF_TEXT))
)
pairs = self.db.list_prompt_pairs()["pairs"]
self.assertEqual(pairs, [])
self.assertEqual(self.handler.awaiting_foreign(), [])
def test_q5_recorded_history_never_brings_back_switched_off_text(self):
"""Even a recorded run whose contexts hold the off text (an older
history row) cannot name it, at settle or at the re-check."""
cases = {"pass-through": off_behind(), "dangling": off_text_unwired()}
for name, graph in cases.items():
with self.subTest(name=name):
self.metadata = {"prompt": graph}
path = self.image(f"IMG_GEN_{len(name)}_00001_.png")
self.seal(path, positives=[OFF_TEXT], nodes=("9",))
self.handler.process_new_image(path)
self.assert_nothing_stored(path)
self.assertIn(SKIPPED, self.recheck(path))
self.assert_nothing_stored(path)
def test_q6_with_foreign_saving_off_save_to_library_is_unchanged(self):
GalleryConfig.SAVE_FOREIGN_RUNS = False
opted_out = save_tests.run_graph(save=False)
self.metadata = {"prompt": opted_out}
skipped = self.image("IMG_GEN_off_00001_.png")
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(skipped)
self.assertIn(SKIPPED, "\n".join(logs.output))
self.metadata = {"prompt": off_behind()}
foreign = self.image("IMG_GEN_foreign_00001_.png")
with self.assertLogs(LOGGER, "INFO") as logs:
self.handler.process_new_image(foreign)
self.assertIn("No PromptManager prompt in workflow", "\n".join(logs.output))
for path in (skipped, foreign):
self.assert_nothing_stored(path)
# A switched-on PromptManager positive still links by its metadata
stored = self.db.save_prompt(
text=save_tests.POSITIVE,
prompt_hash=generate_prompt_hash(save_tests.POSITIVE),
)
self.metadata = {"prompt": save_tests.run_graph(save=True)}
on = self.image("IMG_GEN_on_00001_.png")
self.handler.process_new_image(on)
self.assertEqual(self.linked_text(on), save_tests.POSITIVE)
self.assertEqual(
self.db.get_prompt_by_hash(generate_prompt_hash(save_tests.POSITIVE))["id"],
stored,
)
def test_the_shared_prefix_dataset_still_links_after_the_seal(self):
"""Regression guard (passes before the fix): the wait still works."""
self.metadata = {"prompt": shapes.shared_prefix()}
path = self.image("IMG_GEN_00001_.png")
self.handler.process_new_image(path)
self.assertEqual(self.handler.awaiting_foreign(), [self.key(path)])
self.seal(path, nodes=("13",))
self.assertTrue(self.wait_for(lambda: self.linked_text(path)))
self.assertTrue(self.settled())
self.assertEqual(self.linked_text(path), shapes.BRANCH_TEXTS[0])
if __name__ == "__main__":
unittest.main()
-96
View File
@@ -16,11 +16,6 @@ from tests.background_stops import stop_tracker # noqa: E402
from utils.hashing import (
generate_prompt_hash,
is_duplicate_prompt,
STRIPPED_CODE_POINTS,
prompt_key,
register_prompt_functions,
stored_prompt_text,
stored_prompt_text_sql,
)
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -120,96 +115,5 @@ class TestGeneratePromptHashEdgeCases(unittest.TestCase):
self.assertEqual(len(h), 64)
class TestStoredPromptText(unittest.TestCase):
"""SQLite strips exactly the characters Python's str.strip() does (#438)."""
WHITESPACE = "".join(c for c in map(chr, range(0x110000)) if c.isspace())
def sql_strip(self, value):
import sqlite3
conn = sqlite3.connect(":memory:")
try:
return conn.execute(
f"SELECT {stored_prompt_text_sql('?')}", (value,)
).fetchone()[0]
finally:
conn.close()
def test_code_points_are_exactly_python_whitespace(self):
self.assertEqual("".join(map(chr, STRIPPED_CODE_POINTS)), self.WHITESPACE)
def test_python_rule_is_the_hash_strip(self):
self.assertEqual(stored_prompt_text(" a lighthouse\n"), "a lighthouse")
self.assertEqual(
generate_prompt_hash(stored_prompt_text(" A b\u3000")),
generate_prompt_hash(" A b\u3000"),
)
def test_sql_strips_every_python_whitespace_character(self):
for char in self.WHITESPACE:
with self.subTest(code=hex(ord(char))):
value = f"{char}a{char}b{char}"
self.assertEqual(self.sql_strip(value), stored_prompt_text(value))
mixed = self.WHITESPACE + "a lighthouse é" + self.WHITESPACE
self.assertEqual(self.sql_strip(mixed), "a lighthouse é")
def test_sql_keeps_what_python_keeps(self):
for value in ("\u200ba\u200b", "\ufeffa", "a\u0301", "", "x"):
with self.subTest(value=repr(value)):
self.assertEqual(self.sql_strip(value), stored_prompt_text(value))
class TestPromptKey(unittest.TestCase):
"""History compares the hash's own key; SQL calls Python for it (#460)."""
CASES = (
"A Lighthouse",
" \u00c9t\u00c9\n",
"\u03a3\u03a3",
"\u0130stanbul",
"\u1e9e",
"\u212a",
"ok",
)
def test_key_is_the_hash_normalization(self):
import hashlib
for value in self.CASES:
with self.subTest(value=repr(value)):
self.assertEqual(
generate_prompt_hash(value),
hashlib.sha256(prompt_key(value).encode("utf-8")).hexdigest(),
)
self.assertEqual(prompt_key(" A Lighthouse\n"), "a lighthouse")
def test_sql_functions_give_pythons_result_where_sql_lower_does_not(self):
import sqlite3
conn = sqlite3.connect(":memory:")
try:
register_prompt_functions(conn)
for value in self.CASES:
with self.subTest(value=repr(value)):
key, digest = conn.execute(
"SELECT pm_prompt_key(?), pm_prompt_hash(?)", (value, value)
).fetchone()
self.assertEqual(key, prompt_key(value))
self.assertEqual(digest, generate_prompt_hash(value))
self.assertEqual(
conn.execute(
"SELECT pm_prompt_key(1), pm_prompt_hash(NULL)"
).fetchone(),
(None, None),
)
# SQLite's own lower() folds ASCII only
self.assertNotEqual(
conn.execute("SELECT lower(?)", ("\u00c9",)).fetchone()[0], "\u00e9"
)
finally:
conn.close()
if __name__ == "__main__":
unittest.main()
+1 -7
View File
@@ -7,7 +7,6 @@ from contextlib import closing
from pathlib import Path
from unittest.mock import patch
from database import restore_stage
from database.execution_history import HistoryEpochChanged, notify_history_restore
from tests import test_execution_history_database as history_tests
@@ -42,12 +41,7 @@ class HistoryReadFenceTests(unittest.TestCase):
generation = fence.begin()
with reader.recorded_history_snapshot() as epoch:
original = reader.get_history_run(start.run_id)
# The open snapshot keeps the WAL from being truncated; do not
# sit out the full 30 s checkpoint wait for it (#487).
with (
patch("database.execution_history.notify_history_restore"),
patch.object(restore_stage, "CHECKPOINT_BUSY_MS", 50),
):
with patch("database.execution_history.notify_history_restore"):
writer.model.restore_from_file(source)
self.assertEqual(reader.get_execution_epoch(), epoch)
self.assertEqual(original["run"]["storage_epoch"], epoch)
+5 -135
View File
@@ -9,15 +9,9 @@ from unittest.mock import patch
from database.execution_history import HistoryEpochChanged
from database.execution_history_schema import history_schema_error
from utils.execution_contexts import (
ContextBundle,
RecipeContext,
TextEvidence,
content_digest,
)
from utils.execution_contexts import ContextBundle, RecipeContext, TextEvidence
from utils.execution_outputs import OutputBundle
from tests.test_execution_context_database import bundle, snapshot
from utils.hashing import generate_prompt_hash
from tests.test_execution_output_database import bundle as outputs
from tests import test_execution_history_database as history_tests
@@ -34,10 +28,6 @@ class HistoryReadTests(unittest.TestCase):
conn.execute("INSERT INTO prompts(id,text) VALUES(?,?)", (identity, text))
return identity
def hashed(self, db, text):
"""A prompt saved as the nodes save it: with its dedup hash."""
return db.save_prompt(text, prompt_hash=generate_prompt_hash(text))
def capture(self, db, item=0, context=None, output=None, time=100):
start = self.start(db, item, time)
db.write_execution_start(start)
@@ -156,11 +146,10 @@ class HistoryReadTests(unittest.TestCase):
)
self.require_reads(db)
with db.recorded_history_snapshot():
# Outer whitespace (#438) and case (#460) join one scope;
# Unicode normalization stays distinct.
expected = (5, 5, 5, 5, 1, 1, 2, 2)
for i, count in enumerate(expected, 1):
self.assertEqual(db.list_history_window(i)["recorded_run_count"], count)
for i in range(1, len(values) + 1):
self.assertEqual(
db.list_history_window(i)["recorded_run_count"], 2 if i == 1 else 1
)
self.assertIsNone(db.list_history_window(100))
self.assertIsNone(db.list_history_window(101))
page = db.list_history_window(99)
@@ -1000,122 +989,3 @@ class HistoryReadTests(unittest.TestCase):
db.delete_prompt(saved["positive_prompt_id"])
with db.recorded_history_snapshot():
self.assertIsNone(db.get_newest_pair_run(saved["positive_prompt_id"], pair))
def test_outer_whitespace_raw_text_joins_the_stored_prompt_scope(self):
# #438: the Library stores text.strip(); the run keeps its raw text.
db = self.database()
positive = db.save_prompt("a lighthouse")
raws = ("a lighthouse\n", "a lighthouse", "  a lighthouse \t")
starts = [
self.capture(db, i, bundle(positive=raw), time=100 + i)
for i, raw in enumerate(raws)
]
self.saved(db, "a lighthouse at dusk", 50)
self.capture(db, 9, bundle(positive="a lighthouse at dusk\n"), time=200)
with db.recorded_history_snapshot():
page = db.list_history_window(positive)
first = db.list_history_window(positive, limit=2)
rest = db.list_history_window(
positive, limit=2, after=first["next_position"]
)
details = [db.get_history_run(start.run_id) for start in starts]
self.assertEqual(page["recorded_run_count"], 3)
self.assertEqual(page["group"]["scope_status"], "matched")
self.assertEqual(
page["group"]["raw_text_id"], content_digest("text", ("a lighthouse",))
)
self.assertEqual(
[r["run_id"] for r in page["items"]], [s.run_id for s in starts]
)
self.assertTrue(all(r["matching_anchors"] for r in page["items"]))
self.assertEqual(len(page["context_summaries"]), 3)
self.assertEqual(
[r["run_id"] for r in first["items"] + rest["items"]],
[s.run_id for s in starts[1:]] + [starts[0].run_id],
)
for raw, detail in zip(raws, details):
context = detail["contexts"][0]
self.assertEqual(context["positive"]["raw"], raw)
self.assertEqual(
context["positive_raw_text_id"], page["group"]["raw_text_id"]
)
self.assertEqual(context["positive_prompt_id"], positive)
self.assertTrue(context["navigation"]["positive_available"])
def test_newest_pair_run_follows_outer_whitespace_raw_text(self):
db = self.database()
pair = db.save_prompt_pair("a lighthouse", "fog")
positive = db.get_prompt_pair(pair)["positive_prompt_id"]
newest = self.capture(
db, 0, bundle(" fog\n", positive="a lighthouse\n"), time=100
)
with db.recorded_history_snapshot():
found = db.get_newest_pair_run(positive, pair)
self.assertEqual(found["run"]["run_id"], newest.run_id)
def test_case_variant_raw_text_joins_the_saved_prompt_scope(self):
# #460: the hash folds case, so these runs count toward this prompt.
db = self.database()
positive = self.hashed(db, "\u00e9t\u00e9 lighthouse \u03c3")
raws = (
"\u00c9T\u00c9 Lighthouse \u03a3\n",
"\u00e9t\u00e9 lighthouse \u03c3",
" \u00c9t\u00e9 LIGHTHOUSE \u03a3",
)
starts = [
self.capture(db, i, bundle(positive=raw), time=100 + i)
for i, raw in enumerate(raws)
]
# Unicode normalization and inner text stay distinct
self.capture(db, 7, bundle(positive="e\u0301te\u0301 lighthouse \u03c3"))
self.capture(db, 8, bundle(positive="\u00e9t\u00e9 lighthouse \u03c3"))
dotted = self.hashed(db, "\u0130stanbul")
dotted_run = self.capture(db, 9, bundle(positive="\u0130STANBUL"), time=300)
with db.recorded_history_snapshot():
page = db.list_history_window(positive)
details = [db.get_history_run(start.run_id) for start in starts]
turkish = db.list_history_window(dotted)
self.assertEqual(page["recorded_run_count"], 3)
self.assertEqual(
[r["run_id"] for r in page["items"]], [s.run_id for s in starts]
)
for raw, detail in zip(raws, details):
context = detail["contexts"][0]
self.assertEqual(context["positive"]["raw"], raw)
self.assertEqual(
context["positive_raw_text_id"], page["group"]["raw_text_id"]
)
self.assertEqual(context["positive_prompt_id"], positive)
self.assertEqual([r["run_id"] for r in turkish["items"]], [dotted_run.run_id])
def test_newest_pair_run_follows_case_variant_raw_text(self):
db = self.database()
pair = db.save_prompt_pair("a lighthouse", "fog")
positive = db.get_prompt_pair(pair)["positive_prompt_id"]
newest = self.capture(db, 0, bundle("FOG", positive="A Lighthouse"), time=100)
with db.recorded_history_snapshot():
found = db.get_newest_pair_run(positive, pair)
self.assertEqual(found["run"]["run_id"], newest.run_id)
def test_a_position_from_before_460_resumes_the_same_scope(self):
db = self.database()
positive = self.hashed(db, "A Lighthouse")
for i in range(3):
self.capture(db, i, bundle(positive="a lighthouse"), time=100 + i)
with db.recorded_history_snapshot():
first = db.list_history_window(positive, limit=2)
position = first["next_position"]
legacy = replace(
position, raw_text_id=content_digest("text", ("A Lighthouse",))
)
rest = db.list_history_window(positive, limit=2, after=legacy)
with self.assertRaises(ValueError):
db.list_history_window(
positive,
after=replace(position, raw_text_id=content_digest("text", ("x",))),
)
self.assertEqual(first["recorded_run_count"], 3)
self.assertEqual(len(rest["items"]), 1)
self.assertEqual(
rest["group"]["raw_text_id"], content_digest("text", ("a lighthouse",))
)
+1 -24
View File
@@ -172,40 +172,17 @@ class TestForeignWorkflows(LinkingTestCase):
self.tracker.set_current_prompt(text, {"prompt_id": prompt_id})
return prompt_id
def _linked_text(self):
row = (
self.db.model.get_connection()
.execute(
"SELECT p.text FROM generated_images g "
"JOIN prompts p ON p.id = g.prompt_id WHERE g.image_path = ?",
(self.image,),
)
.fetchone()
)
return row[0] if row else None
def test_foreign_image_leaves_the_batch_queue_aligned(self):
"""#247: it links to its own prompt and never takes a queue entry."""
first = self._queue_batch_item("item one")
self.assertIsNone(self._process({"prompt": foreign_graph()}))
self.assertEqual(self.linked, [])
self.assertEqual(self.tracker.pop_next_prompt()["id"], first)
self.assertIsNone(self.tracker.pop_next_prompt())
self.assertEqual(self._linked_text(), "a cat")
prompt = self.db.get_prompt_by_hash(generate_prompt_hash("a cat"))
self.assertEqual(prompt["run_count"] or 0, 0)
def test_a_foreign_image_is_linked_only_by_the_ladder(self):
def test_foreign_image_is_not_linked_by_the_later_fallbacks_either(self):
self._queue_batch_item("item one") # live tracker and recent prompt exist
untraceable = foreign_graph()
untraceable["9"]["inputs"]["text"] = ["30", 0]
untraceable["30"] = {"class_type": "VendorPromptMaker", "inputs": {}}
self.assertIsNone(self._process({"prompt": untraceable}))
self.assertEqual(self.linked, [])
self.assertIsNone(self._linked_text())
self.assertIsNone(self._process({"prompt": foreign_graph()}))
self.assertEqual(self.linked, [])
self.assertEqual(self._linked_text(), "a cat")
def test_negative_only_workflow_is_foreign(self):
self._save("blurry")
-5
View File
@@ -597,11 +597,6 @@ class TestHandlerScheduling(unittest.TestCase):
self.handler._schedule_recheck(path)
self.assertIsNone(self.handler._worker)
self.assertEqual(list(self.handler._history_due), [])
# #247: a foreign file's sealed re-check is dropped the same way
self.handler._await_foreign(path)
self.assertEqual(self.handler.awaiting_foreign(), [])
self.handler._schedule_recheck(path, {"filename": "late.png"})
self.assertEqual(list(self.handler._foreign_due), [])
self.assertEqual(self._live_workers() - before, set())
def test_a_sealed_run_starts_a_worker_on_an_open_handler(self):
-30
View File
@@ -887,36 +887,6 @@ class TestImageTagsFacet(unittest.TestCase):
self.assertNotIn("image_tags", json.loads(encode(inspection({}))))
self.assertNotIn("image_tags", self.encoded(None))
def history_encoded(self, facet):
from utils.inspection_response import encode_history_inspection_response
body = encode_history_inspection_response(
current_output={"history_epoch": "epoch"},
file={"observation": "current-file", "size": 100},
prompts=PROMPTS,
resolution={"state": "resolved", "sampler_id": "3", "reason": ""},
inspection=inspection({"prompt": graph_text()}),
library={"status": "none"},
image_tags=facet,
)
return json.loads(body)
def test_history_envelope_validates_image_tags_like_the_rooted_one(self):
"""#457: History's current-file read carries the same facet."""
for facet in (
{"state": "current", "tags": ["dunes", "warm_light"]},
{"state": "none", "tags": []},
{"state": "unavailable", "tags": []},
{"state": "current", "tags": ["a,b"]},
{"state": "current", "tags": [], "locator": "/abs/path.png"},
):
with self.subTest(facet=facet):
self.assertEqual(
self.history_encoded(facet)["image_tags"],
self.encoded(facet)["image_tags"],
)
self.assertNotIn("image_tags", self.history_encoded(None))
class TestVideoProbeBoundsMatchPngBudgets(unittest.TestCase):
"""ffprobe escapes tag text; the stdout bounds must never gate a record
+5 -21
View File
@@ -14,7 +14,6 @@ import sys
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -754,17 +753,6 @@ class LocalMediaTests(LayoutTestCase):
self.assertEqual(layout_module.VIDEO_SUFFIXES, frozenset(VIDEO_EXTENSIONS))
# Windows path rules without the filesystem. On Windows ntpath.realpath asks
# the network for \\server\share and waits seconds for a host that does not
# exist (#487); elsewhere ntpath.realpath already is ntpath.abspath.
LEXICAL_NT = SimpleNamespace(
normcase=ntpath.normcase,
realpath=ntpath.abspath,
dirname=ntpath.dirname,
sep=ntpath.sep,
)
class FolderGuardTests(unittest.TestCase):
"""Portable path-flavor controls (TESTING.md Windows portability preflight)."""
@@ -774,7 +762,7 @@ class FolderGuardTests(unittest.TestCase):
def guard_nt(self, path, home="C:\\Users\\Vito"):
return layout_module._guard_folder(
path, home, pathmod=LEXICAL_NT, isdir=lambda _p: True
path, home, pathmod=ntpath, isdir=lambda _p: True
)
def test_filesystem_roots_and_home_are_rejected(self):
@@ -844,12 +832,8 @@ class FolderGuardTests(unittest.TestCase):
def test_missing_folders_add_nothing(self):
self.assertIsNone(layout_module._guard_folder(str(self.tmp / "x"), "/home"))
# A missing share on this machine: an unknown server name made
# Windows wait about 2.8 s for the network (#487).
self.assertIsNone(
layout_module._guard_folder(
"\\\\localhost\\pm-no-such-share\\loras", str(self.tmp)
)
layout_module._guard_folder("\\\\server\\share\\loras", str(self.tmp))
)
self.assertEqual(
layout_module._guard_folder(str(self.tmp), "/nonexistent-home"),
@@ -859,17 +843,17 @@ class FolderGuardTests(unittest.TestCase):
def test_windows_model_path_matches_a_lower_case_root(self):
self.assertTrue(
layout_module._inside(
"C:/Models/Loras/A.safetensors", ("c:\\models\\loras",), LEXICAL_NT
"C:/Models/Loras/A.safetensors", ("c:\\models\\loras",), ntpath
)
)
self.assertFalse(
layout_module._inside(
"C:/Models/LorasX/A.safetensors", ("c:\\models\\loras",), LEXICAL_NT
"C:/Models/LorasX/A.safetensors", ("c:\\models\\loras",), ntpath
)
)
self.assertFalse(
layout_module._inside(
"\\\\other\\share\\A.safetensors", ("c:\\models\\loras",), LEXICAL_NT
"\\\\other\\share\\A.safetensors", ("c:\\models\\loras",), ntpath
)
)
+2 -12
View File
@@ -9,7 +9,6 @@ import email
import json
import os
import shutil
import ssl
import sys
import tempfile
import threading
@@ -670,15 +669,6 @@ class TestStaleDownloadSweep(unittest.TestCase):
self.assertTrue(old.exists())
def _unused_tls_context():
"""A TLS context for the fake handlers, which never open a connection.
HTTPSHandler() builds a default context, and on Windows that reads the
system certificate store: about 0.14 s per handler (#487).
"""
return ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
class _FakeHTTPSHandler(urllib.request.HTTPSHandler):
"""Serves canned responses through a real urllib opener, recording requests.
@@ -687,7 +677,7 @@ class _FakeHTTPSHandler(urllib.request.HTTPSHandler):
"""
def __init__(self, redirect_to):
super().__init__(context=_unused_tls_context())
super().__init__()
self.redirect_to = redirect_to
self.requests = []
@@ -755,7 +745,7 @@ class _ChainHTTPSHandler(urllib.request.HTTPSHandler):
urllib opener, recording every request; an unknown URL answers 404."""
def __init__(self, routes):
super().__init__(context=_unused_tls_context())
super().__init__()
self.routes = routes
self.requests = []
-951
View File
@@ -1,951 +0,0 @@
"""tools/repair_foreign_links.py: the reviewed repair of the #247 dataset links.
The fixture is a library whose foreign (no PromptManager node) dataset run
wrote shared-prefix files that the old scan linked to the wrong branch text.
Recorded history names each file's saver, so the ladder (``recorded-run``)
names the right branch.
"""
import contextlib
import hashlib
import io
import json
import os
import shutil
import sqlite3
import tempfile
import unittest
import unittest.mock
from pathlib import Path
from PIL import Image
from PIL.PngImagePlugin import PngInfo
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
from database.models import PromptModel
from database.operations import PromptDatabase
from tests import saver_anchor_graphs as shapes
from tests.test_run_attribution import RecordedRuns
from tools import repair_foreign_links as tool
from utils.file_prompt import Decision
from utils.hashing import generate_prompt_hash
TEXTS = shapes.BRANCH_TEXTS
NAME = "IMG_GEN_{:05}_.png"
def png(path, graph):
info = PngInfo()
info.add_text("prompt", json.dumps(graph))
Image.new("RGB", (4, 4)).save(path, pnginfo=info)
def sha(text):
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def file_sha(path):
return hashlib.sha256(Path(path).read_bytes()).hexdigest()
def dump(path):
"""The database's logical content (independent of WAL checkpoints)."""
conn = sqlite3.connect(path)
try:
return list(conn.iterdump())
finally:
conn.close()
def not_running(_comfyui_dir, _port):
return []
def no_handles(_database):
return []
class RepairFixture(RecordedRuns, unittest.TestCase):
"""Rows: A relink to a new prompt, B relink to a stored prompt (with a
pair link), C keep, D file gone, E two savers (skip), F the UNIQUE case
(a wrong row beside a correct row), H relink from a rated prompt, G a
PromptManager run (not a candidate)."""
def setUp(self):
self.tmp = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmp, True)
self.addCleanup(assert_closed, self, self.tmp)
self.addCleanup(PromptModel.reset_schema_cache)
self.output = os.path.join(self.tmp, "output")
os.makedirs(os.path.join(self.output, "dataset"))
self.db_path = os.path.join(self.tmp, "prompts.db")
self.plan_path = os.path.join(self.tmp, "plan.json")
self.backups = os.path.join(self.tmp, "backups")
os.makedirs(self.backups)
self.comfy = os.path.join(self.tmp, "comfy")
os.makedirs(self.comfy)
Path(self.comfy, "main.py").write_text("# ComfyUI\n", encoding="utf-8")
self.files = {}
for key, number in zip("ABCDEFHG", range(1, 9)):
path = os.path.join(self.output, "dataset", NAME.format(number))
png(path, shapes.shared_prefix())
self.files[key] = path
db = PromptDatabase(self.db_path)
try:
self.build(db)
finally:
db.close_all()
os.remove(self.files["D"])
def build(self, db):
save = lambda text, **kw: db.save_prompt( # noqa: E731
text=text, prompt_hash=generate_prompt_hash(text), **kw
)
self.p0 = save(TEXTS[0], category="scanned", tags=["auto-scanned"])
self.p2 = save(TEXTS[2], category="scanned", tags=["auto-scanned"])
self.orphan = save(
"an old guess",
category="scanned",
tags=["auto-scanned"],
notes="Auto-scanned from IMG_GEN_00001_.png",
)
self.rated = save("a rated guess", rating=5)
link = lambda prompt, key: db.link_image_to_prompt( # noqa: E731
prompt, self.files[key], metadata={}
)
self.rows = {
"A": link(self.orphan, "A"),
"B": link(self.p0, "B"),
"C": link(self.p0, "C"),
"D": link(self.p0, "D"),
"E": link(self.p0, "E"),
"F": link(self.p0, "F"),
"F_ok": link(self.p2, "F"),
"H": link(self.rated, "H"),
"G": link(self.p0, "G"),
}
conn = db.model.get_connection()
with conn:
pair = conn.execute(
"INSERT INTO prompt_pairs(positive_prompt_id) VALUES(?)", (self.p0,)
).lastrowid
conn.execute(
"INSERT INTO image_prompt_pairs(image_id, pair_id, provenance) "
"VALUES(?,?,'{}')",
(self.rows["B"], pair),
)
name = lambda key: ("dataset", os.path.basename(self.files[key])) # noqa
# Savers of the shared-prefix graph: "13", "23", "33", "43" (branches 1-4)
self.record_run(
db,
self.output,
[name(k) for k in "CDEAEBFH"],
[],
nodes=["13", "13", "13", "23", "23", "33", "33", "43"],
)
self.record_run(db, self.output, [name("G")], [TEXTS[0]], item=1, nodes=["23"])
def dry_run(self, *args):
out = io.StringIO()
code = tool.main(
[
self.db_path,
"--output-dir",
self.output,
"--plan",
self.plan_path,
*args,
],
out=out,
)
return code, out.getvalue()
def apply(self, *args, probe=not_running, handles=no_handles):
out = io.StringIO()
code = tool.main(
[
self.db_path,
"--apply",
self.plan_path,
"--comfyui-dir",
os.path.join(self.tmp, "comfy"),
"--backup-dir",
self.backups,
*args,
],
out=out,
probe=probe,
handles=handles,
)
return code, out.getvalue()
def plan(self):
with open(self.plan_path, encoding="utf-8") as stream:
return json.load(stream)
def write_plan(self, plan):
with open(self.plan_path, "w", encoding="utf-8") as stream:
json.dump(plan, stream)
def query(self, sql, args=()):
conn = sqlite3.connect(self.db_path)
try:
return conn.execute(sql, args).fetchall()
finally:
conn.close()
def query_write(self, sql, args):
conn = sqlite3.connect(self.db_path)
try:
with conn:
conn.execute(sql, args)
finally:
conn.close()
def links(self):
return dict(self.query("SELECT id, prompt_id FROM generated_images"))
class DryRunTests(RepairFixture):
def test_plan_names_relink_keep_and_skip_rows(self):
code, out = self.dry_run()
self.assertEqual(code, 0, out)
items = {item["row_id"]: item for item in self.plan()["rows"]}
rows = self.rows
self.assertNotIn(rows["G"], items, "a PromptManager run is not foreign")
actions = {key: items[rows[key]]["action"] for key in rows if key != "G"}
self.assertEqual(
actions,
{
"A": "relink",
"B": "relink",
"C": "keep",
"D": "skip",
"E": "skip",
"F": "relink",
"F_ok": "keep",
"H": "relink",
},
)
a = items[rows["A"]]
self.assertEqual(a["current_prompt_id"], self.orphan)
self.assertEqual(a["current_text_sha256"], sha("an old guess"))
self.assertEqual(a["target_text"], TEXTS[1])
self.assertEqual(a["target_hash"], generate_prompt_hash(TEXTS[1]))
self.assertEqual(a["target_prompt_id"], "new")
self.assertEqual(a["file"], self.files["A"])
self.assertEqual(items[rows["B"]]["target_prompt_id"], self.p2)
self.assertEqual(items[rows["H"]]["target_prompt_id"], "new")
self.assertEqual(items[rows["D"]]["reason"], "file gone")
self.assertEqual(
items[rows["E"]]["reason"], "not decided by the recorded saver"
)
self.assertIn("relink: 4", out)
self.assertIn("keep: 2", out)
self.assertIn("skip: 2", out)
def test_orphan_audit_lists_the_prompts_losing_every_link(self):
self.dry_run()
plan = self.plan()
orphans = {o["id"]: o for o in plan["orphans"]}
self.assertEqual(set(orphans), {self.orphan, self.rated})
self.assertEqual(orphans[self.orphan]["blockers"], [])
self.assertTrue(orphans[self.orphan]["safe"])
self.assertEqual(orphans[self.rated]["blockers"], ["usage or rating"])
self.assertFalse(orphans[self.rated]["safe"])
self.assertFalse(plan["delete_orphans"], "off by default")
self.dry_run("--delete-orphans")
self.assertTrue(self.plan()["delete_orphans"])
def test_dry_run_writes_nothing_and_writes_the_markdown_list(self):
before = file_sha(self.db_path)
listing = sorted(os.listdir(self.tmp))
code, out = self.dry_run()
self.assertEqual(code, 0)
self.assertEqual(file_sha(self.db_path), before)
self.assertIn("Dry run: nothing was changed.", out)
# A read-only reader of a WAL library may leave its empty side files
sidecars = {"prompts.db-shm", "prompts.db-wal"}
self.assertEqual(
sorted(set(os.listdir(self.tmp)) - sidecars),
sorted(set(listing + ["plan.json", "plan.list.md"]) - sidecars),
)
wal = Path(self.tmp, "prompts.db-wal")
self.assertEqual(wal.stat().st_size if wal.exists() else 0, 0)
markdown = Path(self.tmp, "plan.list.md").read_text(encoding="utf-8")
self.assertIn(f"| {self.rows['A']} |", markdown)
long_path = os.path.join("dataset", NAME.format(1))
self.assertIn(f"| {long_path} |", markdown, "file paths are never cut")
self.assertIn(generate_prompt_hash(TEXTS[1])[:12], markdown)
self.assertIn("new", markdown)
self.assertIn("would lose every link", markdown)
class DryRunReviewTests(RepairFixture):
def test_plan_records_its_library_and_fingerprint(self):
self.dry_run()
plan = self.plan()
self.assertEqual(
plan["source_db"], os.path.normcase(os.path.realpath(self.db_path))
)
self.assertEqual(plan["max_row_id"], max(self.links()))
def test_a_markdown_plan_path_is_refused(self):
out = io.StringIO()
with (
self.assertRaises(SystemExit),
contextlib.redirect_stderr(io.StringIO()),
):
tool.main(
[
self.db_path,
"--output-dir",
self.output,
"--plan",
os.path.join(self.tmp, "plan.md"),
],
out=out,
)
self.assertFalse(os.path.exists(os.path.join(self.tmp, "plan.md")))
def test_keep_compares_the_linked_text_not_the_stored_hash(self):
self.query_write("UPDATE prompts SET hash='stale' WHERE id=?", (self.p0,))
self.dry_run()
items = {item["row_id"]: item for item in self.plan()["rows"]}
self.assertEqual(items[self.rows["C"]]["action"], "keep")
def test_delete_orphans_is_rehearsed_with_the_deletes(self):
code, out = self.dry_run("--delete-orphans")
self.assertEqual(code, 0, out)
simulated = self.plan()["simulated"]
self.assertEqual(simulated["result"]["orphans_deleted"], 1)
self.assertEqual(
simulated["after"]["prompts"], simulated["before"]["prompts"] + 2 - 1
)
def test_a_file_on_another_drive_is_listed_by_its_full_path(self):
self.dry_run()
plan = self.plan()
with unittest.mock.patch.object(
tool.os.path, "relpath", side_effect=ValueError("other drive")
):
listing = tool.markdown(plan, {})
self.assertIn(self.files["A"], listing)
class ApplyTests(RepairFixture):
def setUp(self):
super().setUp()
code, out = self.dry_run()
self.assertEqual(code, 0, out)
def test_apply_relinks_creates_targets_and_drops_pair_links(self):
code, out = self.apply()
self.assertEqual(code, 0, out)
links = self.links()
new = dict(self.query("SELECT hash, id FROM prompts"))
target_a = new[generate_prompt_hash(TEXTS[1])]
target_h = new[generate_prompt_hash(TEXTS[3])]
self.assertEqual(links[self.rows["A"]], target_a)
self.assertEqual(links[self.rows["B"]], self.p2)
self.assertEqual(links[self.rows["H"]], target_h)
self.assertEqual(links[self.rows["C"]], self.p0, "keep")
self.assertEqual(links[self.rows["E"]], self.p0, "skip")
self.assertEqual(links[self.rows["G"]], self.p0, "not foreign")
self.assertEqual(
self.query(
"SELECT COUNT(*) FROM image_prompt_pairs WHERE image_id=?",
(self.rows["B"],),
),
[(0,)],
)
category, notes = self.query(
"SELECT category, notes FROM prompts WHERE id=?", (target_a,)
)[0]
self.assertEqual(category, "scanned")
self.assertEqual(
notes, f"Auto-scanned from {os.path.basename(self.files['A'])}"
)
tags = self.query(
"SELECT t.name FROM prompt_tags pt JOIN tags t ON t.id=pt.tag_id "
"WHERE pt.prompt_id=?",
(target_a,),
)
self.assertEqual(tags, [("auto-scanned",)])
self.assertIn("integrity_check", out)
self.assertIn("foreign_key_check", out)
self.assertRegex(out, r"integrity_check\s+ok\s+ok")
# Orphans are kept unless the plan chose to delete them
self.assertTrue(self.query("SELECT 1 FROM prompts WHERE id=?", (self.orphan,)))
def test_unique_case_deletes_the_wrong_row(self):
code, out = self.apply()
self.assertEqual(code, 0, out)
links = self.links()
self.assertNotIn(self.rows["F"], links)
self.assertEqual(links[self.rows["F_ok"]], self.p2)
rows = self.query(
"SELECT prompt_id FROM generated_images WHERE image_path=?",
(self.files["F"],),
)
self.assertEqual(rows, [(self.p2,)])
def test_backup_is_taken_before_writing(self):
before = self.links()
code, out = self.apply()
self.assertEqual(code, 0, out)
names = os.listdir(self.backups)
self.assertEqual(len(names), 1)
self.assertRegex(names[0], r"^prompts\.db\.pre-247-repair_\d{8}_\d{6}$")
backup = os.path.join(self.backups, names[0])
conn = sqlite3.connect(backup)
try:
saved = dict(conn.execute("SELECT id, prompt_id FROM generated_images"))
finally:
conn.close()
self.assertEqual(saved, before)
self.assertIn(f"Backup: {backup}", out)
def test_backup_beside_the_database_by_default(self):
out = io.StringIO()
code = tool.main(
[
self.db_path,
"--apply",
self.plan_path,
"--comfyui-dir",
os.path.join(self.tmp, "comfy"),
],
out=out,
probe=not_running,
handles=no_handles,
)
self.assertEqual(code, 0, out.getvalue())
self.assertTrue(
any(
n.startswith("prompts.db.pre-247-repair_") for n in os.listdir(self.tmp)
)
)
def test_refuses_while_comfyui_runs(self):
before = file_sha(self.db_path)
code, out = self.apply(probe=lambda _d, _p: ["process 42 (main.py)"])
self.assertEqual(code, 2)
self.assertIn("REFUSED", out)
self.assertIn("process 42", out)
self.assertEqual(os.listdir(self.backups), [], "no backup either")
self.assertEqual(file_sha(self.db_path), before)
def test_rehearsal_skips_the_running_check_on_a_copy_only(self):
code, out = self.apply("--rehearsal", probe=lambda _d, _p: ["port 8188"])
self.assertEqual(code, 0, out)
self.assertIn("rehearsal", out)
# The live library of that ComfyUI folder: --rehearsal is refused
live_dir = os.path.join(self.tmp, "comfy", "user", "__promptmanager")
os.makedirs(live_dir)
live = os.path.join(live_dir, "prompts.db")
shutil.copy(self.db_path, live)
out = io.StringIO()
code = tool.main(
[
live,
"--apply",
self.plan_path,
"--comfyui-dir",
os.path.join(self.tmp, "comfy"),
"--rehearsal",
],
out=out,
probe=lambda _d, _p: [],
handles=no_handles,
)
self.assertEqual(code, 2)
self.assertIn("--rehearsal", out.getvalue())
def test_drift_refuses_and_changes_nothing(self):
cases = {
"row moved": lambda: self.query_write(
"UPDATE generated_images SET prompt_id=? WHERE id=?",
(self.p2, self.rows["H"]),
),
"text changed": lambda: self.query_write(
"UPDATE prompts SET text='an older guess' WHERE id=?", (self.orphan,)
),
"file changed": lambda: png(self.files["B"], shapes.branch_bench()),
"target stored since": lambda: self.query_write(
"INSERT INTO prompts(text, hash) VALUES(?,?)",
(TEXTS[1], generate_prompt_hash(TEXTS[1])),
),
}
pristine = os.path.join(self.tmp, "pristine.db")
shutil.copy(self.db_path, pristine)
image = Path(self.files["B"]).read_bytes()
for name, drift in cases.items():
with self.subTest(name):
shutil.copy(pristine, self.db_path)
Path(self.files["B"]).write_bytes(image)
shutil.rmtree(self.backups)
os.makedirs(self.backups)
drift()
before = dump(self.db_path)
code, out = self.apply()
self.assertEqual(code, 2, out)
self.assertIn("REFUSED (rolled back, nothing changed)", out)
self.assertEqual(dump(self.db_path), before)
def test_late_failure_rolls_back_every_write(self):
plan = self.plan()
plan["delete_orphans"] = True
self.write_plan(plan)
# The safe orphan gained a user tag after the dry run
self.query_write("INSERT INTO tags(name) VALUES('mine')", ())
self.query_write(
"INSERT INTO prompt_tags(prompt_id, tag_id) "
"SELECT ?, id FROM tags WHERE name='mine'",
(self.orphan,),
)
before = self.links()
prompts = self.query("SELECT COUNT(*) FROM prompts")
code, out = self.apply()
self.assertEqual(code, 2, out)
self.assertIn(f"prompt {self.orphan}", out)
self.assertIn("user tags", out)
self.assertEqual(self.links(), before, "the relinks were rolled back")
self.assertEqual(self.query("SELECT COUNT(*) FROM prompts"), prompts)
def test_orphans_are_deleted_only_when_chosen_and_unreferenced(self):
plan = self.plan()
plan["delete_orphans"] = True
self.write_plan(plan)
code, out = self.apply()
self.assertEqual(code, 0, out)
self.assertFalse(self.query("SELECT 1 FROM prompts WHERE id=?", (self.orphan,)))
self.assertTrue(
self.query("SELECT 1 FROM prompts WHERE id=?", (self.rated,)),
"a prompt with a rating is never deleted",
)
self.assertIn("orphans_deleted", out)
def test_an_applied_plan_refuses_a_second_time(self):
self.assertEqual(self.apply()[0], 0)
before = dump(self.db_path)
code, out = self.apply()
self.assertEqual(code, 2)
self.assertIn("REFUSED", out)
self.assertEqual(dump(self.db_path), before)
class ApplySafetyTests(RepairFixture):
def setUp(self):
super().setUp()
code, out = self.dry_run()
self.assertEqual(code, 0, out)
def assert_unchanged(self, before, code, out):
self.assertEqual(code, 2, out)
self.assertEqual(dump(self.db_path), before)
def test_refuses_when_another_process_has_the_library_open(self):
before = dump(self.db_path)
held = lambda _db: ["process 7 has prompts.db-wal open"] # noqa: E731
for args in ((), ("--rehearsal",)):
with self.subTest(args):
code, out = self.apply(*args, handles=held)
self.assert_unchanged(before, code, out)
self.assertIn("process 7", out)
self.assertEqual(os.listdir(self.backups), [])
def test_the_running_check_runs_again_just_before_writing(self):
calls = []
def late(_d, _p):
calls.append(1)
return [] if len(calls) == 1 else ["something listens on port 8188"]
before = dump(self.db_path)
code, out = self.apply(probe=late)
self.assert_unchanged(before, code, out)
self.assertEqual(len(calls), 2)
self.assertIn("rolled back", out)
self.assertIn("Backup:", out)
def test_without_proc_only_an_explicit_stop_lets_it_through(self):
blind = lambda _db: [tool.NO_PROC] # noqa: E731
before = dump(self.db_path)
code, out = self.apply(handles=blind)
self.assert_unchanged(before, code, out)
self.assertIn("--i-stopped-comfyui", out)
code, out = self.apply("--i-stopped-comfyui", handles=blind)
self.assertEqual(code, 0, out)
def test_a_folder_without_main_py_is_refused(self):
os.remove(os.path.join(self.comfy, "main.py"))
before = dump(self.db_path)
code, out = self.apply()
self.assert_unchanged(before, code, out)
self.assertIn("main.py", out)
def test_a_plan_for_another_library_is_refused(self):
other = os.path.join(self.tmp, "other.db")
shutil.copy(self.db_path, other)
out = io.StringIO()
code = tool.main(
[other, "--apply", self.plan_path, "--comfyui-dir", self.comfy],
out=out,
probe=not_running,
handles=no_handles,
)
self.assertEqual(code, 2)
self.assertIn("made for", out.getvalue())
def test_opening_the_library_must_not_change_its_schema(self):
self.query_write("DROP INDEX idx_prompt_pairs_positive_only", ())
PromptModel.reset_schema_cache()
code, out = self.apply()
self.assertEqual(code, 2, out)
self.assertIn("schema", out)
self.assertIn("live code version", out)
def test_an_existing_backup_file_is_refused(self):
stamp = "20261028_000000"
Path(self.backups, f"prompts.db.pre-247-repair_{stamp}").write_bytes(b"x")
before = dump(self.db_path)
with unittest.mock.patch.object(tool, "_stamp", return_value=stamp):
code, out = self.apply()
self.assert_unchanged(before, code, out)
self.assertIn("already exists", out)
def test_a_backup_that_does_not_match_is_refused(self):
real = tool.backup_database
def short(source, target):
real(source, target)
conn = sqlite3.connect(target)
try:
with conn:
conn.execute("DELETE FROM image_prompt_pairs")
finally:
conn.close()
before = dump(self.db_path)
with unittest.mock.patch.object(tool, "backup_database", short):
code, out = self.apply()
self.assert_unchanged(before, code, out)
self.assertIn("backup", out)
def test_a_changed_target_text_is_refused(self):
plan = self.plan()
for item in plan["rows"]:
if item["row_id"] == self.rows["A"]:
item["target_text"] = "something else"
self.write_plan(plan)
before = dump(self.db_path)
code, out = self.apply()
self.assert_unchanged(before, code, out)
self.assertIn("target_text", out)
def test_an_orphan_gone_or_changed_at_apply_is_refused(self):
plan = self.plan()
plan["delete_orphans"] = True
plan["orphans"][0]["text_sha256"] = "0" * 64
self.write_plan(plan)
before = dump(self.db_path)
code, out = self.apply()
self.assert_unchanged(before, code, out)
self.assertIn("gone or text changed", out)
def test_any_error_rolls_back_and_names_the_backup(self):
before = dump(self.db_path)
with unittest.mock.patch.object(
tool, "_relink", side_effect=RuntimeError("disk on fire")
):
code, out = self.apply()
self.assert_unchanged(before, code, out)
self.assertIn("rolled back", out)
self.assertIn("disk on fire", out)
self.assertIn(self.backups, out)
def test_a_failed_integrity_check_after_writing_exits_1(self):
real = tool.counts
seen = []
def counts(path, readonly=True):
found = real(path, readonly)
if path == self.db_path:
seen.append(path)
if len(seen) > 1:
return {**found, "integrity_check": "corrupt"}
return found
with unittest.mock.patch.object(tool, "counts", counts):
code, out = self.apply()
self.assertEqual(code, 1, out)
self.assertIn("restore", out)
class BlockerTests(RepairFixture):
"""Each reference that keeps a prompt from being deleted."""
def test_every_blocker(self):
pid = self.orphan
cases = {
"linked images": None,
"saved pairs": "INSERT INTO prompt_pairs(positive_prompt_id) VALUES(:id)",
"recorded runs": "UPDATE execution_run_contexts "
"SET positive_prompt_id=:id",
"library state": "INSERT INTO prompt_library_state(prompt_id) "
"VALUES(:id)",
"user tags": "INSERT INTO prompt_tags(prompt_id, tag_id) "
"SELECT :id, id FROM tags WHERE name='mine'",
"usage or rating": "UPDATE prompts SET run_count=3 WHERE id=:id",
"category": "UPDATE prompts SET category='portraits' WHERE id=:id",
"notes": "UPDATE prompts SET notes='mine' WHERE id=:id",
}
conn = sqlite3.connect(self.db_path)
try:
conn.execute("INSERT INTO tags(name) VALUES('mine')")
conn.commit()
for name, sql in cases.items():
with self.subTest(name):
conn.execute("SAVEPOINT blocker")
if sql is None:
unused = {pid}
else:
conn.execute(
"DELETE FROM generated_images WHERE prompt_id=?", (pid,)
)
conn.execute(sql, {"id": pid})
unused = {pid}
found = tool.blockers(conn, pid, unused)
self.assertTrue(any(item.startswith(name) for item in found), found)
conn.execute("ROLLBACK TO blocker")
conn.execute("RELEASE blocker")
conn.execute("DELETE FROM generated_images WHERE prompt_id=?", (pid,))
self.assertEqual(tool.blockers(conn, pid, {pid}), [])
self.assertEqual(
tool.blockers(conn, pid, set()), ["not unused by the app's rule"]
)
conn.rollback()
finally:
conn.close()
def _off_text_node(text, save=False):
return {
"class_type": "PromptManagerText",
"inputs": {"text": text, "save_to_library": save},
}
def _encode_from(source):
return {
"class_type": "CLIPTextEncode",
"inputs": {"text": source, "clip": ["1", 1]},
}
def off_behind_pass_through(branch=2):
"""The dataset graph with ``branch``'s text coming from a switched-off
PromptManagerText through a pass-through the #90 walk does not follow."""
graph = shapes.shared_prefix()
graph["50"] = _off_text_node(TEXTS[branch - 1])
graph["51"] = {"class_type": "MyNode", "inputs": {"value": ["50", 0]}}
graph[str(branch * 10)] = _encode_from(["51", 0])
return graph
def off_target(branch=2, save=False):
"""``branch``'s text straight from a PromptManagerText whose switch is
``save`` (False, or a link to a node the graph cannot read)."""
graph = shapes.shared_prefix()
graph["50"] = _off_text_node(TEXTS[branch - 1], save)
graph[str(branch * 10)] = _encode_from(["50", 0])
if isinstance(save, list):
graph["52"] = {"class_type": "SomeBoolNode", "inputs": {}}
return graph
def off_negative_only():
"""Every branch's negative from a switched-off PromptManagerText."""
graph = shapes.shared_prefix()
graph["50"] = _off_text_node(shapes.NEGATIVE)
graph["3"] = _encode_from(["50", 0])
return graph
class SaveToLibraryTests(RecordedRuns, unittest.TestCase):
"""#90: the repair never relinks a file to a switched-off node's text.
The plan decides each file through ``decide_file_prompt``; every answer
but a positive from the recorded saver or the inspector is a skip, so a
``not_saved`` or ``not_saved_wired`` file is never written. Each file is
reported by a run without contexts (as a cached or unrecorded run is),
so only the ladder can stop it. Saver "23" is branch 2.
"""
def setUp(self):
self.tmp = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmp, True)
self.addCleanup(assert_closed, self, self.tmp)
self.addCleanup(PromptModel.reset_schema_cache)
self.output = os.path.join(self.tmp, "output")
os.makedirs(os.path.join(self.output, "dataset"))
self.db_path = os.path.join(self.tmp, "prompts.db")
def plan_for(self, graph):
path = os.path.join(self.output, "dataset", NAME.format(1))
png(path, graph)
db = PromptDatabase(self.db_path)
try:
wrong = db.save_prompt(
text=TEXTS[0], prompt_hash=generate_prompt_hash(TEXTS[0])
)
db.link_image_to_prompt(wrong, path, metadata={})
self.record_run(
db, self.output, [("dataset", NAME.format(1))], [], nodes=["23"]
)
plan = tool.build_plan(db, self.output, self.db_path)
finally:
db.close_all()
(item,) = plan["rows"]
return item
def test_an_off_node_behind_a_pass_through_is_skipped(self):
item = self.plan_for(off_behind_pass_through())
self.assertEqual(item["action"], "skip")
self.assertEqual(item["decision"], "not_saved/none")
self.assertNotIn("target_text", item)
def test_an_off_node_in_the_recorded_run_target_is_skipped(self):
cases = {
"off": (False, "not_saved/none"),
"wired": (["52", 0], "not_saved_wired/none"),
}
for name, (save, decision) in cases.items():
with self.subTest(name):
if os.path.exists(self.db_path):
os.remove(self.db_path)
item = self.plan_for(off_target(save=save))
self.assertEqual(item["action"], "skip")
self.assertEqual(item["decision"], decision)
def test_an_off_node_on_another_branch_still_skips_the_file(self):
"""The ladder judges the file's whole graph: any switched-off text
upstream of a positive makes the file not saved."""
item = self.plan_for(off_target(branch=3))
self.assertEqual(item["action"], "skip")
self.assertEqual(item["decision"], "not_saved/none")
def test_an_off_node_wired_only_to_the_negative_still_repairs(self):
item = self.plan_for(off_negative_only())
self.assertEqual(item["action"], "relink")
self.assertEqual(item["source"], "recorded-run")
self.assertEqual(item["target_text"], TEXTS[1])
def test_a_switched_on_node_in_the_target_still_repairs(self):
"""Regression guard: a switch that is on is no reason to skip."""
item = self.plan_for(off_target(save=True))
self.assertEqual(item["action"], "relink")
self.assertEqual(item["target_text"], TEXTS[1])
class ActionTests(unittest.TestCase):
"""Only the recorded saver relinks; a certain step naming the linked
prompt keeps it; the legacy floor and the rest are skipped."""
def test_action_by_decision(self):
linked = generate_prompt_hash("linked")
cases = [
(Decision("positive", "other", source="recorded-run"), "relink"),
(Decision("positive", "linked", source="recorded-run"), "keep"),
(Decision("positive", "linked", source="inspector"), "keep"),
(Decision("positive", "other", source="inspector"), "skip"),
(Decision("positive", "linked", source="legacy-parser"), "skip"),
(Decision("pair", "linked", source="resolver"), "skip"),
(Decision("never"), "skip"),
(Decision("not_saved"), "skip"),
(Decision("not_saved_wired"), "skip"),
(Decision("positive", "other", source="opted-out-negative"), "skip"),
]
for decision, action in cases:
with self.subTest(decision.source, text=decision.text):
self.assertEqual(tool.action_of(decision, linked), action)
class CommandLineTests(unittest.TestCase):
def test_running_probe_reports_a_listening_port(self):
import socket
with socket.socket() as server:
server.bind(("127.0.0.1", 0))
server.listen(1)
port = server.getsockname()[1]
found = tool.comfyui_running("/nonexistent/comfy", port)
self.assertTrue(any(str(port) in item for item in found))
def test_running_probe_is_quiet_when_nothing_runs(self):
import socket
with socket.socket() as probe:
probe.bind(("127.0.0.1", 0))
port = probe.getsockname()[1]
self.assertEqual(tool.comfyui_running("/nonexistent/comfy", port), [])
def fake_proc(self, tmp):
proc = Path(tmp, "proc")
db = Path(tmp, "prompts.db")
db.write_bytes(b"")
Path(tmp, "prompts.db-wal").write_bytes(b"")
comfy = Path(tmp, "comfy")
comfy.mkdir()
holder = proc / "125" / "fd"
holder.mkdir(parents=True)
os.symlink(Path(tmp, "prompts.db-wal"), holder / "3")
(proc / "125" / "cmdline").write_bytes(b"python\0main.py\0")
os.symlink(comfy, proc / "125" / "cwd")
(proc / "123").mkdir() # unreadable: no fd, cmdline or cwd
(proc / "self").mkdir() # not a pid
return proc, db, comfy
@unittest.skipIf(os.name == "nt", "symlinks need privileges on Windows")
def test_proc_scan_finds_holders_and_skips_unreadable_entries(self):
with tempfile.TemporaryDirectory() as tmp:
proc, db, comfy = self.fake_proc(tmp)
found = tool.open_by_others(str(db), proc=str(proc), own_pid=1)
self.assertEqual(len(found), 1)
self.assertIn("125", found[0])
self.assertEqual(
tool.open_by_others(str(db), proc=str(proc), own_pid=125), []
)
running = tool.comfyui_running(str(comfy), 1, proc=str(proc))
self.assertTrue(any("125" in item for item in running), running)
def test_without_proc_the_scan_says_it_cannot_tell(self):
self.assertEqual(
tool.open_by_others("/x/prompts.db", proc="/nonexistent/proc"),
[tool.NO_PROC],
)
def test_missing_database_is_an_argument_error(self):
with self.assertRaises(SystemExit), contextlib.redirect_stderr(io.StringIO()):
tool.main(
["/nonexistent/prompts.db", "--output-dir", "/tmp"], out=io.StringIO()
)
def test_apply_needs_the_comfyui_folder(self):
with tempfile.TemporaryDirectory() as tmp:
db = os.path.join(tmp, "prompts.db")
Path(db).write_bytes(b"")
with (
self.assertRaises(SystemExit),
contextlib.redirect_stderr(io.StringIO()),
):
tool.main([db, "--apply", "plan.json"], out=io.StringIO())
if __name__ == "__main__":
unittest.main()
-482
View File
@@ -33,7 +33,6 @@ from utils.execution_outputs import (
snapshot_output_roots,
)
from utils.hashing import generate_prompt_hash
from py.config import GalleryConfig
from utils.image_monitor import ImageGenerationHandler
STYLE = "windows" if os.name == "nt" else "posix"
@@ -772,487 +771,6 @@ class TestMonitorHistoryRecheck(RecordedRuns, unittest.TestCase):
run_attribution.notify_outputs_sealed("run", []) # no dead callback raises
def folder_is_case_sensitive(folder):
"""True when ``folder``'s volume tells names apart by case: the existing
folder spelled with swapped case is not found (#476)."""
head, name = os.path.split(folder)
return not os.path.exists(os.path.join(head, name.swapcase()))
class TestMonitorForeignRuns(RecordedRuns, unittest.TestCase):
"""#247: outputs of workflows without a PromptManager node, in the monitor."""
def setUp(self):
history_tests.HistoryDatabaseTests.setUp(self)
self.db = history_tests.HistoryDatabaseTests.database(self)
self.output = str(Path(self.temp.name) / "output")
os.makedirs(os.path.join(self.output, "dataset"))
self.tracker = make_tracker(self.db)
self.handler = ImageGenerationHandler(self.db, self.tracker)
self.handler.processing_delay = 0
self.handler.settle_interval = 0.01
self.addCleanup(close_handler, self, self.handler)
self.metadata = {"prompt": shapes.shared_prefix()}
saved = GalleryConfig.SAVE_FOREIGN_RUNS
self.addCleanup(setattr, GalleryConfig, "SAVE_FOREIGN_RUNS", saved)
GalleryConfig.SAVE_FOREIGN_RUNS = True
patches = (
mock.patch(
"utils.image_monitor.host_output_directory", return_value=self.output
),
mock.patch(
"database.run_attribution.host_output_directory",
return_value=self.output,
),
mock.patch.object(
self.handler.metadata_extractor,
"extract_metadata",
side_effect=lambda path: self.metadata,
),
)
for patch in patches:
patch.start()
self.addCleanup(patch.stop)
close_databases = history_tests.HistoryDatabaseTests.close_databases
wait_for = TestMonitorHistoryRecheck.wait_for
def image(self, name, subfolder="dataset"):
path = os.path.join(self.output, subfolder, name)
Path(path).write_bytes(b"png")
return path
def linked_text(self, path):
row = (
self.db.model.get_connection()
.execute(
"SELECT p.text FROM generated_images g "
"JOIN prompts p ON p.id = g.prompt_id WHERE g.image_path = ?",
(str(path),),
)
.fetchone()
)
return row[0] if row else None
def key(self, path):
return self.handler._awaiting_key(path)
def prompt_count(self):
connection = self.db.model.get_connection()
return connection.execute("SELECT COUNT(*) FROM prompts").fetchone()[0]
def settled(self):
return self.wait_for(lambda: self.handler.pending_count() == 0)
def test_each_dataset_file_links_to_its_branch_once_the_run_ends(self):
"""CUJ 1: unlinked while the run is open, its own branch after the seal."""
paths = [self.image(f"IMG_GEN_0000{n}_.png") for n in (1, 2)]
for path in paths:
self.handler.process_new_image(path)
self.assertEqual([self.linked_text(path) for path in paths], [None, None])
self.assertEqual(
self.handler.awaiting_foreign(), [self.key(path) for path in paths]
)
self.assertEqual(self.handler.awaiting_history(), [])
self.record_run(
self.db,
self.output,
[("dataset", os.path.basename(path)) for path in paths],
[],
nodes=["13", "23"],
)
self.assertTrue(self.wait_for(lambda: all(map(self.linked_text, paths))))
self.assertTrue(self.settled())
self.assertEqual(
[self.linked_text(path) for path in paths], list(shapes.BRANCH_TEXTS[:2])
)
self.assertEqual(self.handler.awaiting_foreign(), [])
prompt = self.db.get_prompt_by_hash(
generate_prompt_hash(shapes.BRANCH_TEXTS[0])
)
self.assertEqual(prompt["category"], "scanned")
self.assertIn("Saved from a workflow run: IMG_GEN_00001_.png", prompt["notes"])
def test_a_run_sealed_before_the_file_settles_links_it_at_once(self):
path = self.image("IMG_GEN_00003_.png")
self.record_run(
self.db, self.output, [("dataset", "IMG_GEN_00003_.png")], [], nodes=["33"]
)
self.handler.process_new_image(path)
self.assertEqual(self.linked_text(path), shapes.BRANCH_TEXTS[2])
self.assertEqual(self.handler.awaiting_foreign(), [])
def test_a_name_two_runs_reported_stays_unlinked(self):
"""CUJ 3: the file was overwritten; no run proves its saver."""
path = self.image("IMG_GEN_00004_.png")
reported = [("dataset", "IMG_GEN_00004_.png")]
self.record_run(self.db, self.output, reported, [], nodes=["13"])
self.record_run(self.db, self.output, reported, [], item=1, nodes=["23"])
self.handler.process_new_image(path)
self.assertIsNone(self.linked_text(path))
reference = {"folder_type": "output", "subfolder": "dataset"}
reference["filename"] = "IMG_GEN_00004_.png"
with self.assertLogs("prompt_manager.image_monitor", "INFO") as logs:
self.handler._recheck_foreign(path, reference)
self.assertIn(
"No certain prompt for IMG_GEN_00004_.png after its run ended; "
"left unlinked",
"\n".join(logs.output),
)
self.assertIsNone(self.linked_text(path))
self.assertEqual(self.prompt_count(), 0)
def test_the_setting_off_stores_nothing_at_settle_or_after_the_run(self):
"""CUJ 4: read for every file, so it applies without a restart."""
GalleryConfig.SAVE_FOREIGN_RUNS = False
skipped = self.image("IMG_GEN_00005_.png")
with self.assertLogs("prompt_manager.image_monitor", "INFO") as logs:
self.handler.process_new_image(skipped)
self.assertIn(
"No PromptManager prompt in workflow, skipping image",
"\n".join(logs.output),
)
self.assertEqual(self.handler.awaiting_foreign(), [])
GalleryConfig.SAVE_FOREIGN_RUNS = True
waiting = self.image("IMG_GEN_00006_.png")
self.handler.process_new_image(waiting)
self.assertEqual(self.handler.awaiting_foreign(), [self.key(waiting)])
GalleryConfig.SAVE_FOREIGN_RUNS = False
self.record_run(
self.db, self.output, [("dataset", "IMG_GEN_00006_.png")], [], nodes=["13"]
)
self.assertTrue(self.wait_for(lambda: not self.handler.awaiting_foreign()))
self.assertTrue(self.settled())
self.assertEqual(
[self.linked_text(p) for p in (skipped, waiting)], [None, None]
)
self.assertEqual(self.prompt_count(), 0)
def test_a_plain_foreign_workflow_links_at_settle_and_counts_no_run(self):
"""CUJ 2: the default profile decides; the tracker queue is untouched."""
text = "a lighthouse on a cliff at dusk"
self.metadata = {"prompt": sampled_graph(text, "blurry")}
queued = self.db.save_prompt(
text="queued item", prompt_hash=generate_prompt_hash("queued item")
)
self.tracker.set_current_prompt("queued item", {"prompt_id": queued})
first, second = self.image("plain_00001_.png"), self.image("plain_00002_.png")
self.handler.process_new_image(first)
self.handler.process_new_image(second)
self.assertEqual(
[self.linked_text(first), self.linked_text(second)], [text, text]
)
prompt = self.db.get_prompt_by_hash(generate_prompt_hash(text))
self.assertEqual(prompt["run_count"] or 0, 0)
self.assertIsNone(prompt.get("last_used_at"))
self.assertEqual(self.tracker.pop_next_prompt()["id"], queued)
self.assertEqual(self.handler.awaiting_foreign(), [])
def test_the_sealed_reference_spells_the_file_as_history_recorded_it(self):
"""Windows: the watcher's spelling differs in case from the run's report.
On a case-sensitive folder the two spellings are two files and the
folded key joins them; on a case-insensitive one (Windows, default
macOS) they already name one file, so only history's spelling and
the watcher's differ (#476)."""
seen = os.path.join(self.output, "dataset", "img_gen_00007_.png")
Path(seen).write_bytes(b"png")
recorded = os.path.join(self.output, "Dataset", "IMG_GEN_00007_.png")
self.assertNotEqual(os.path.basename(recorded), os.path.basename(seen))
if folder_is_case_sensitive(os.path.join(self.output, "dataset")):
Path(self.output, "Dataset").mkdir()
self.image("IMG_GEN_00007_.png", subfolder="Dataset")
else:
self.assertTrue(os.path.exists(recorded))
folded = staticmethod(lambda path: os.path.realpath(path).lower())
with mock.patch.object(ImageGenerationHandler, "_awaiting_key", folded):
self.handler.process_new_image(seen)
self.assertIsNone(self.linked_text(seen))
self.record_run(
self.db,
self.output,
[("Dataset", os.path.basename(recorded))],
[],
nodes=["43"],
)
self.assertTrue(self.wait_for(lambda: self.linked_text(seen)))
self.assertTrue(self.settled())
self.assertEqual(self.linked_text(seen), shapes.BRANCH_TEXTS[3])
def test_a_full_foreign_set_never_evicts_a_waiting_keyless_file(self):
from utils import image_monitor
keyless = self.image("bare_00001_.png", subfolder="")
with mock.patch.object(
self.handler.metadata_extractor, "extract_metadata", return_value=None
):
self.handler.process_new_image(keyless)
paths = [self.image(f"IMG_GEN_1000{n}_.png") for n in range(3)]
with (
mock.patch.object(image_monitor, "MAX_AWAITING_HISTORY", 2),
self.assertLogs("prompt_manager.image_monitor", "INFO") as logs,
):
for path in paths:
self.handler.process_new_image(path)
self.assertEqual(
self.handler.awaiting_foreign(), [self.key(p) for p in paths[1:]]
)
self.assertEqual(self.handler.awaiting_history(), [self.key(keyless)])
self.assertIn(
"Too many files wait for their run; IMG_GEN_10000_.png left unlinked",
"\n".join(logs.output),
)
def test_an_expired_foreign_entry_is_logged_and_left_unlinked(self):
from utils import image_monitor
path = self.image("IMG_GEN_00008_.png")
self.handler.process_new_image(path)
later = time.monotonic() + image_monitor.AWAIT_HISTORY_SECONDS + 1
with (
mock.patch.object(image_monitor.time, "monotonic", return_value=later),
self.assertLogs("prompt_manager.image_monitor", "INFO") as logs,
):
self.assertEqual(self.handler.awaiting_foreign(), [])
self.assertIn(
"No recorded run for IMG_GEN_00008_.png; left unlinked",
"\n".join(logs.output),
)
self.assertIsNone(self.linked_text(path))
def test_a_seal_while_the_pending_set_is_full_is_rechecked_later(self):
from utils import image_monitor
path = self.image("IMG_GEN_00009_.png")
self.handler.process_new_image(path)
blocker = self.image("blocker.png", subfolder="")
release = threading.Event()
original = self.handler.process_new_image
def slow(image_path, prompt_snapshot=None):
if image_path == blocker:
release.wait(5)
return None
return original(image_path, prompt_snapshot=prompt_snapshot)
self.handler.process_new_image = slow
with mock.patch.object(image_monitor, "MAX_PENDING_IMAGES", 1):
self.assertTrue(self.handler.schedule(blocker))
self.assertTrue(self.wait_for(lambda: self.handler.pending_count() == 1))
self.record_run(
self.db,
self.output,
[("dataset", "IMG_GEN_00009_.png")],
[],
nodes=["13"],
)
self.assertEqual(list(self.handler._foreign_due), [path])
release.set()
self.assertTrue(self.wait_for(lambda: self.linked_text(path)))
self.assertTrue(self.settled())
self.assertEqual(self.linked_text(path), shapes.BRANCH_TEXTS[0])
def test_shutdown_forgets_waiting_and_due_foreign_files(self):
path = self.image("IMG_GEN_00010_.png")
self.handler.process_new_image(path)
self.handler._foreign_due[path] = {"filename": "IMG_GEN_00010_.png"}
self.handler.shutdown()
self.assertEqual(self.handler.awaiting_foreign(), [])
self.assertEqual(list(self.handler._foreign_due), [])
def link_count(self, path):
connection = self.db.model.get_connection()
return connection.execute(
"SELECT COUNT(*) FROM generated_images WHERE image_path = ?", (str(path),)
).fetchone()[0]
def test_a_file_linked_to_another_prompt_never_gets_a_second_link(self):
"""One prompt per file: the pair path and the positive path both refuse."""
other = self.db.save_prompt(
"an earlier prompt", generate_prompt_hash("an earlier prompt")
)
plain = self.image("plain_00004_.png")
self.db.link_image_to_prompt(other, plain)
self.metadata = {"prompt": sampled_graph("a quiet harbor", "blurry")}
with self.assertLogs("prompt_manager.image_monitor", "INFO") as logs:
self.handler.process_new_image(plain)
self.assertIn("Foreign run: conflict", "\n".join(logs.output))
dataset = self.image("IMG_GEN_00011_.png")
self.db.link_image_to_prompt(other, dataset)
self.metadata = {"prompt": shapes.shared_prefix()}
self.record_run(
self.db, self.output, [("dataset", "IMG_GEN_00011_.png")], [], nodes=["13"]
)
self.handler.process_new_image(dataset)
for path in (plain, dataset):
self.assertEqual(self.linked_text(path), "an earlier prompt")
self.assertEqual(self.link_count(path), 1)
for text in ("a quiet harbor", shapes.BRANCH_TEXTS[0]):
self.assertIsNone(self.db.get_prompt_by_hash(generate_prompt_hash(text)))
def test_the_file_size_is_read_only_when_the_ladder_needs_it(self):
"""PR 1's lazy size: a foreign file decided without a store reads none."""
# Foreign (a core encoder only) and no API graph: never, at step 3
encoder = {"type": "CLIPTextEncode", "inputs": [], "widgets_values": ["x"]}
self.metadata = {"text_encoder_nodes": [encoder]}
path = self.image("bare_00002_.png", subfolder="")
with mock.patch("utils.foreign_runs.os.path.getsize") as size:
self.handler.process_new_image(path)
size.assert_not_called()
def untraceable(self):
"""Unresolved and only the legacy parser names a text (an unknown node)."""
graph = sampled_graph("unused", "blurry")
graph["1"]["inputs"]["text"] = ["30", 0]
graph["30"] = {
"class_type": "VendorPromptMaker",
"inputs": {"prompt": "a misty valley at dawn"},
}
return {"prompt": graph}
def test_no_record_at_settle_keeps_the_file_waiting_for_the_legacy_floor(self):
"""Review: a lookup that found no record is not the run's answer yet."""
self.metadata = self.untraceable()
path = self.image("vendor_00001_.png")
self.handler.process_new_image(path)
self.assertIsNone(self.linked_text(path))
self.assertEqual(self.handler.awaiting_foreign(), [self.key(path)])
self.record_run(self.db, self.output, [("dataset", "vendor_00001_.png")], [])
self.assertTrue(self.wait_for(lambda: self.linked_text(path)))
self.assertTrue(self.settled())
self.assertEqual(self.linked_text(path), "a misty valley at dawn")
def test_a_record_found_at_settle_without_a_certain_prompt_ends_the_wait(self):
"""Review: sealed before settle, still ambiguous: no seal is coming."""
path = self.image("IMG_GEN_00012_.png")
# One run reported it from two savers: a record, but no saver
reported = [("dataset", "IMG_GEN_00012_.png")] * 2
self.record_run(self.db, self.output, reported, [], nodes=["13", "23"])
with self.assertLogs("prompt_manager.image_monitor", "INFO") as logs:
self.handler.process_new_image(path)
self.assertIn(
"No certain prompt for IMG_GEN_00012_.png; left unlinked",
"\n".join(logs.output),
)
self.assertEqual(self.handler.awaiting_foreign(), [])
self.assertIsNone(self.linked_text(path))
def test_monitoring_saved_off_stops_foreign_writes_at_settle_and_recheck(self):
"""Review: the observer runs until a restart; nothing is written."""
enabled = GalleryConfig.MONITORING_ENABLED
self.addCleanup(setattr, GalleryConfig, "MONITORING_ENABLED", enabled)
GalleryConfig.MONITORING_ENABLED = False
self.metadata = {"prompt": sampled_graph("a quiet harbor", "blurry")}
plain = self.image("plain_00005_.png")
self.handler.process_new_image(plain)
self.assertIsNone(self.linked_text(plain))
GalleryConfig.MONITORING_ENABLED = True
self.metadata = {"prompt": shapes.shared_prefix()}
waiting = self.image("IMG_GEN_00013_.png")
self.handler.process_new_image(waiting)
GalleryConfig.MONITORING_ENABLED = False
reference = {"folder_type": "output", "subfolder": "dataset"}
reference["filename"] = "IMG_GEN_00013_.png"
self.record_run(
self.db, self.output, [("dataset", reference["filename"])], [], nodes=["13"]
)
self.assertTrue(self.wait_for(lambda: not self.handler.awaiting_foreign()))
self.assertTrue(self.settled())
self.handler._recheck_foreign(waiting, reference)
self.assertIsNone(self.linked_text(waiting))
self.assertEqual(self.prompt_count(), 0)
def test_the_setting_is_read_again_just_before_the_store(self):
self.metadata = {"prompt": sampled_graph("a quiet harbor", "blurry")}
path = self.image("plain_00006_.png")
real = __import__("utils.foreign_runs", fromlist=["x"]).decide_foreign
def decide_then_switch_off(*args, **kwargs):
decision = real(*args, **kwargs)
GalleryConfig.SAVE_FOREIGN_RUNS = False
return decision
with mock.patch(
"utils.image_monitor.decide_foreign", side_effect=decide_then_switch_off
):
self.handler.process_new_image(path)
self.assertIsNone(self.linked_text(path))
self.assertEqual(self.prompt_count(), 0)
def test_a_seal_between_registering_and_the_settle_lookup_links_once(self):
"""Regression guard (passes before the review fixes): one link only."""
path = self.image("IMG_GEN_00014_.png")
real = __import__("utils.foreign_runs", fromlist=["x"]).history_lookup
def sealed_first(db, image_path, reference=None):
lookup = real(db, image_path, reference)
def run_then_look():
if reference is None: # the settle lookup: the run seals now
self.record_run(
self.db,
self.output,
[("dataset", "IMG_GEN_00014_.png")],
[],
nodes=["13"],
)
return lookup()
return run_then_look
with mock.patch("utils.image_monitor.history_lookup", side_effect=sealed_first):
self.assertTrue(self.handler.schedule(path))
self.assertTrue(self.wait_for(lambda: self.linked_text(path)))
self.assertTrue(self.settled())
self.assertEqual(self.linked_text(path), shapes.BRANCH_TEXTS[0])
self.assertEqual(self.link_count(path), 1)
self.assertEqual(list(self.handler._foreign_due), [])
def test_a_drained_recheck_never_drops_a_newer_reference(self):
path = self.image("IMG_GEN_00015_.png")
older, newer = {"filename": "old"}, {"filename": "new"}
self.handler._foreign_due[path] = older
real = self.handler.schedule
def schedule(image_path, snapshot=None):
# The run is sealed again while the older one is scheduled
self.handler._foreign_due[image_path] = newer
return real(image_path, snapshot)
self.handler.schedule = schedule
with mock.patch.object(self.handler, "process_new_image"):
self.handler._drain_history_due()
self.assertIs(self.handler._foreign_due[path], newer)
def test_an_evicted_due_recheck_is_logged(self):
from utils import image_monitor
self.handler.schedule = lambda *args, **kwargs: False
with (
mock.patch.object(image_monitor, "MAX_AWAITING_HISTORY", 1),
self.assertLogs("prompt_manager.image_monitor", "INFO") as logs,
):
for name in ("a_00001_.png", "b_00001_.png"):
self.handler._schedule_recheck(self.image(name), {"filename": name})
self.assertIn(
"Too many files wait for their run; a_00001_.png left unlinked",
"\n".join(logs.output),
)
def test_a_failed_store_is_logged_and_leaves_the_file_unlinked(self):
self.metadata = {"prompt": sampled_graph("a quiet harbor", "blurry")}
path = self.image("plain_00003_.png")
with (
mock.patch(
"utils.foreign_runs.ingest_resolved_pair",
side_effect=sqlite3.OperationalError("locked"),
),
self.assertLogs("prompt_manager.image_monitor", "ERROR"),
):
self.handler.process_new_image(path)
self.assertIsNone(self.linked_text(path))
class TestBoundedLookup(RecordedRuns, unittest.TestCase):
"""M5: history lookups are batched, and the history schema is unchanged."""
+2 -53
View File
@@ -18,13 +18,11 @@ from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
SUBPROCESS_TIMEOUT = 60
WATCHDOG_ENV = ("PM_TEST_HANG_LIMIT", "PM_TEST_SHUTDOWN_LIMIT")
# Integer settings the runner reads; a bad value is a usage error.
RUNNER_ENV = (*WATCHDOG_ENV, "PM_TEST_SLOWEST")
def run_suite(folder, extra_env=None, *args, verbosity=("-v",)):
"""Run the runner on ``folder`` and return the completed process."""
env = {k: v for k, v in os.environ.items() if k not in RUNNER_ENV}
env = {k: v for k, v in os.environ.items() if k not in WATCHDOG_ENV}
# Windows writes a piped stderr in the ANSI codepage; ask for the UTF-8
# this helper decodes, so non-ASCII values and paths survive.
env["PYTHONIOENCODING"] = "utf-8"
@@ -271,57 +269,8 @@ class RunSuiteTests(unittest.TestCase):
self.assertIn("ok", result.stderr, detail)
self.assertIn("Timeout (", result.stderr, detail)
def test_slowest_tests_are_listed_when_asked(self):
self.write(
"test_probe_slowest.py",
"""
import time
import unittest
class Probe(unittest.TestCase):
def test_fast(self):
pass
def test_slow(self):
time.sleep(0.3)
def test_slower(self):
time.sleep(0.6)
""",
)
result = run_suite(self.folder, {"PM_TEST_SLOWEST": "2"})
self.assertEqual(result.returncode, 0, result.stderr)
report = result.stderr.split("slowest 2 of 3 tests:\n", 1)
self.assertEqual(len(report), 2, result.stderr)
lines = [line.split() for line in report[1].strip().splitlines()[:2]]
self.assertEqual(
[line[1] for line in lines],
[
"test_probe_slowest.Probe.test_slower",
"test_probe_slowest.Probe.test_slow",
],
result.stderr,
)
self.assertGreaterEqual(float(lines[0][0].rstrip("s")), 0.6)
self.assertNotIn("test_fast", report[1])
def test_slowest_tests_are_not_listed_by_default(self):
self.write(
"test_probe_quiet.py",
"""
import unittest
class Probe(unittest.TestCase):
def test_ok(self):
pass
""",
)
result = run_suite(self.folder)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertNotIn("slowest", result.stderr)
def test_invalid_limit_is_a_usage_error(self):
for name in RUNNER_ENV:
for name in WATCHDOG_ENV:
for value in ("abc", "0", "-3", "1.5", "\u00b2"):
with self.subTest(name=name, value=value):
result = run_suite(self.folder, {name: value})
-37
View File
@@ -323,43 +323,6 @@ class TestMonitoringSetting(SourcesTestCase):
self.assertIs(GalleryConfig.MONITORING_ENABLED, False)
class TestSaveForeignRunsSetting(SourcesTestCase):
"""#247 D-Q4: "Save prompts from other workflows", applied live."""
async def asyncSetUp(self):
await super().asyncSetUp()
saved = GalleryConfig.SAVE_FOREIGN_RUNS
self.addCleanup(setattr, GalleryConfig, "SAVE_FOREIGN_RUNS", saved)
GalleryConfig.SAVE_FOREIGN_RUNS = True
async def test_read_and_write_published_live_without_a_restart(self):
self.assertIs((await self._get())["settings"]["save_foreign_runs"], True)
self._install_monitor(FakeMonitor([self.output_dir]))
status, data = await self._post(
"/prompt_manager/settings", {"save_foreign_runs": False}
)
self.assertEqual(status, 200, data)
self.assertTrue(data["persisted"])
self.assertEqual(data["restart_required_fields"], [])
saved = self._read_config()
self.assertIs(saved["gallery"]["monitoring"]["save_foreign_runs"], False)
self.assertIs(GalleryConfig.SAVE_FOREIGN_RUNS, False)
self.assertIs((await self._get())["settings"]["save_foreign_runs"], False)
async def test_only_true_or_false_is_accepted(self):
self._write_config({"custom": {"keep": True}})
before = self._config_bytes()
for value in ("false", 0, None, []):
with self.subTest(value=value):
status, data = await self._post(
"/prompt_manager/settings", {"save_foreign_runs": value}
)
self.assertEqual(status, 400, data)
self.assertIn("save_foreign_runs must be true or false", data["error"])
self.assertEqual(self._config_bytes(), before)
self.assertIs(GalleryConfig.SAVE_FOREIGN_RUNS, True)
class TestSourceChangeHold(SourcesTestCase):
async def test_root_save_refused_while_scan_job_runs(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
+3 -27
View File
@@ -212,15 +212,11 @@ test('every browser job sets PM_TEST_WORKERS and runs in the pinned image', () =
assert.match(mergeJob(), PINNED_IMAGE, 'the merge sees the same workspace paths as the shards');
});
// #473: two shards split by test count put the slow concept specs in shard 1, which reached the 18-minute step limit.
test('page workflows run each browser in three shards', () => {
test('page workflows run each browser in two shards', () => {
const workflows = allJobs.get('browser-workflows');
assert.deepEqual(matrixList(workflows, 'browser'), ['firefox', 'webkit']);
assert.deepEqual(matrixList(workflows, 'shard'), ['1', '2', '3']);
assert.match(workflows, /test:workflows -- --project \$\{\{ matrix\.browser \}\} --shard \$\{\{ matrix\.shard \}\}\/3/);
assert.match(workflows, /^ {4}name: page workflows \(\$\{\{ matrix\.browser \}\}, shard \$\{\{ matrix\.shard \}\}\/3\)$/m);
assert.match(workflows, /name: Page workflows in \$\{\{ matrix\.browser \}\}, shard \$\{\{ matrix\.shard \}\}\/3$/m);
assert.doesNotMatch(workflows, /\/2\b/, 'no step still names two shards');
assert.deepEqual(matrixList(workflows, 'shard'), ['1', '2']);
assert.match(workflows, /test:workflows -- --project \$\{\{ matrix\.browser \}\} --shard \$\{\{ matrix\.shard \}\}\/2/);
assert.match(workflows, /^ {6}fail-fast: false$/m);
assert.match(workflows, /name: browser-traces-\$\{\{ matrix\.browser \}\}-shard-\$\{\{ matrix\.shard \}\}/);
});
@@ -353,26 +349,6 @@ test('Windows and macOS run 3.12-3.14 on main, nightly and on demand, never on a
assert.doesNotMatch(text, /cancel-in-progress: true|cancel-in-progress: \$/, 'no portability run is cancelled');
});
// #493: the portability limit follows the rule in its comment: ceil(1.5 x the slowest of three
// green main runs) + 3 minutes, never under 5, with those runs and times written above jobs:.
// The times are whole-job wall times. That the runs were green and on main is the author's word.
test('the portability limit is computed from three measured green main runs', () => {
const text = readWorkflow('portability.yml');
const comment = text.slice(0, text.indexOf('\njobs:'));
const runs = [...comment.matchAll(/^# Measured \d{4}-\d{2}-\d{2} on main runs (\d+) \([0-9a-f]{7}\), (\d+) \([0-9a-f]{7}\), (\d+) \([0-9a-f]{7}\):$/gm)];
assert.equal(runs.length, 1, 'the comment names the three runs it measured, once');
assert.equal(new Set(runs[0].slice(1)).size, 3, 'three different runs');
const time = '(\\d+)m([0-5]\\d)s \\(([^)]+)\\)';
const slowest = new RegExp(`^#\\s+slowest leg: ${time} / ${time} / ${time} -> (\\d+) min$`, 'm').exec(comment);
assert.ok(slowest, 'the comment gives the slowest leg of each run and the limit');
const seconds = [1, 4, 7].map(i => Number(slowest[i]) * 60 + Number(slowest[i + 1]));
const expected = Math.max(5, Math.ceil(1.5 * Math.max(...seconds) / 60) + 3);
assert.equal(Number(slowest[10]), expected, 'the written limit follows the rule');
assert.equal(comment.match(/-> \d+ min/g).length, 1, 'no older measurement stays beside it');
const limit = Number(/^ {4}timeout-minutes: (\d+)\s*$/m.exec(jobs(text).get('test'))[1]);
assert.equal(limit, expected, 'the job uses the written limit');
});
test('real ComfyUI runs only outside pull requests; the other test.yml checks run on every event', () => {
const found = testJobs();
assert.match(found.get('comfyui'), QUEUE_ONLY_IF);
+3 -3
View File
@@ -36,14 +36,14 @@ test('execution script discovers only the new matrix and preserves all 78 origin
// + the fresh-host library location case, + #342 image attribution, + #343 linked text,
// + #332 image category routes on a generated output (image-attribution.spec.mjs, 2 projects),
// + #347 cached runtime-linked runs: five node kinds and eight one-renderer cases (execution-text-link.spec.mjs, 2 projects),
// + #90 Save to library off (five node types), + #247 foreign workflows (image-attribution.spec.mjs, 2 tests x 2 projects)
assert.match(executions, /Total: 132 tests in 5 files/);
// + #90 Save to library off (five node types)
assert.match(executions, /Total: 128 tests in 5 files/);
for (const project of ['classic-start', 'nodes2-start']) {
const cases = executions.split('\n').filter(line => line.includes(`[${project}]`));
assert.equal(cases.filter(line => line.includes('execution-history.spec.mjs')).length, 15);
assert.equal(cases.filter(line => line.includes('execution-outputs.spec.mjs')).length, 8);
assert.equal(cases.filter(line => line.includes('library-location.spec.mjs')).length, 1);
assert.equal(cases.filter(line => line.includes('image-attribution.spec.mjs')).length, 5);
assert.equal(cases.filter(line => line.includes('image-attribution.spec.mjs')).length, 3);
assert.equal(cases.filter(line => line.includes('execution-text-link.spec.mjs')).length, 37);
}
assert.doesNotMatch(executions, /prompt-manager.spec/);
-81
View File
@@ -1,81 +0,0 @@
"""Is ComfyUI stopped? The checks a library repair runs before writing (#247).
``comfyui_running`` looks for ``main.py`` processes of one ComfyUI folder
and a listener on its port. ``open_by_others`` lists the processes that hold
the library file, its ``-wal`` or its ``-shm`` open. Both read ``/proc``
(Linux); without it ``open_by_others`` answers ``[NO_PROC]``: it cannot
tell, and the caller refuses unless the user says ComfyUI is stopped.
Entries that cannot be read (another user's process, one that just exited)
are skipped.
"""
import os
import socket
NO_PROC = (
"cannot list the processes that hold the library open (no /proc): stop "
"ComfyUI and pass --i-stopped-comfyui"
)
def _same(path):
return os.path.normcase(os.path.realpath(path))
def _pids(proc, own_pid=None):
try:
names = os.listdir(proc)
except OSError:
return []
return [name for name in names if name.isdigit() and name != str(own_pid)]
def comfyui_running(comfyui_dir, port, proc="/proc"):
"""``main.py`` processes running from ``comfyui_dir`` and a listener on
``port`` (127.0.0.1), each as one line; empty when none is found."""
found = []
folder = _same(comfyui_dir)
for pid in _pids(proc):
try:
with open(os.path.join(proc, pid, "cmdline"), "rb") as stream:
parts = stream.read().split(b"\0")
cwd = _same(os.readlink(os.path.join(proc, pid, "cwd")))
except OSError:
continue
if any(part.endswith(b"main.py") for part in parts) and (
cwd == folder or any(folder.encode() in part for part in parts)
):
found.append(f"process {pid} (main.py in {cwd})")
with socket.socket() as probe:
probe.settimeout(1)
if probe.connect_ex(("127.0.0.1", port)) == 0:
found.append(f"something listens on port {port}")
return found
def open_by_others(database, proc="/proc", own_pid=None):
"""Processes other than ``own_pid`` (default: this one) holding
``database``, its ``-wal`` or ``-shm`` open; ``[NO_PROC]`` without /proc."""
if not os.path.isdir(proc):
return [NO_PROC]
own_pid = os.getpid() if own_pid is None else own_pid
targets = {_same(database + suffix): suffix for suffix in ("", "-wal", "-shm")}
found = []
for pid in _pids(proc, own_pid):
fd_dir = os.path.join(proc, pid, "fd")
try:
fds = os.listdir(fd_dir)
except OSError:
continue
held = set()
for fd in fds:
try:
target = os.readlink(os.path.join(fd_dir, fd))
except OSError:
continue
target = target.replace(" (deleted)", "")
if os.path.isabs(target) and _same(target) in targets:
held.add(os.path.basename(database) + targets[_same(target)])
if held:
found.append(f"process {pid} has {', '.join(sorted(held))} open")
return found
-776
View File
@@ -1,776 +0,0 @@
"""Repair the links the old scan gave foreign dataset files (#247), on review.
Before #247 the output scan linked each file of a foreign run (a workflow
with no PromptManager node) through the legacy parser, so every file of a
shared-prefix dataset run got the first branch's text. Recorded history
knows which save node wrote each file; the shared ladder
(``utils.file_prompt.decide_file_prompt``, scan policy) uses it
(``recorded-run``) to name the right branch. This tool judges every
``generated_images`` row of a file whose recording runs all have no
PromptManager context:
``relink`` the ladder names a positive by the recorded saver, and its text
is not the linked prompt's
``keep`` the recorded saver or the inspector names the linked prompt
``skip`` anything else (file gone, no single recorded saver, ambiguous,
the legacy floor or another step decided)
Dry run (the default) judges a copy of the database (SQLite backup from a
read-only connection) and writes a JSON plan and a Markdown list beside it
(``<plan stem>.list.md``). It also rehearses the apply on a second copy, to
list the prompts that would lose every link with their remaining
references. Deleting those is OFF unless the dry run is given
``--delete-orphans``; the choice is recorded in the plan and rehearsed on a
third copy. The library is never written.
``--apply <plan.json>`` (ComfyUI stopped), only on the library the plan was
made from (its real path, recorded in the plan):
1. refuses unless ``--comfyui-dir`` holds ``main.py``; refuses while a
``main.py`` runs from it, anything listens on ``--port``, or any other
process holds the library, its ``-wal`` or ``-shm`` open
(``tools/comfyui_stopped.py``; without /proc only with
``--i-stopped-comfyui``). ``--rehearsal`` skips the process and port
check, never the open-file check, and never for that folder's library;
2. saves ``<db>.pre-247-repair_<stamp>`` (or into ``--backup-dir``), never
over an existing file, and refuses unless the backup passes
``integrity_check`` with the library's counts. Opening the library must
not change its schema (a newer checkout's migration): run the tool from a
checkout at the live code version. The running check runs once more just
before writing;
3. in ONE ``BEGIN IMMEDIATE`` transaction: re-judges every planned row and
refuses on any drift (row moved, prompt text changed, file changed, target
stored since), creates the target prompts that do not exist (category
``scanned``, tag ``auto-scanned``, note ``Auto-scanned from <first file>``),
points each row at its target and drops its pair links (they named the old
positive). A file that already has a row for the target keeps that row and
loses the wrong one (UNIQUE(prompt_id, file_path)). Orphans are deleted only
when the plan chose it and each is still unreferenced.
Any failed check rolls everything back and exits 2. Before/after table
counts, ``integrity_check`` and ``foreign_key_check`` are printed.
python3 tools/repair_foreign_links.py <db> --output-dir <ComfyUI output> \\
--plan repair_plan.json
python3 tools/repair_foreign_links.py <db> --apply repair_plan.json \\
--comfyui-dir <ComfyUI folder>
Run from a checkout at the live code version (after the merge: main) with
its venv.
"""
import argparse
import datetime
import hashlib
import json
import logging
import os
import sqlite3
import sys
import tempfile
from pathlib import Path
from types import SimpleNamespace
_ROOT = Path(__file__).resolve().parents[1]
if str(_ROOT) not in sys.path:
sys.path.insert(0, str(_ROOT))
from database.maintenance import ORPHANED_PROMPTS_SQL, _delete_ids # noqa: E402
from database.operations import PromptDatabase # noqa: E402
from database.pair_ingestion import ( # noqa: E402
SCAN_MARKS,
file_link_rows,
mark_created,
)
from database.run_attribution import recorded_output # noqa: E402
from py.api import PromptManagerAPI # noqa: E402
from tools.comfyui_stopped import ( # noqa: E402
NO_PROC,
comfyui_running,
open_by_others,
)
from tools.review_image_links import backup_database # noqa: E402
from utils.execution_outputs import snapshot_output_roots # noqa: E402
from utils.file_prompt import POSITIVE, Policy, decide_file_prompt # noqa: E402
from utils.hashing import generate_prompt_hash # noqa: E402
PLAN_VERSION = 1
COMFY_PORT = 8188
LIVE_LIBRARY = ("user", "__promptmanager", "prompts.db")
EXCERPT = 80
COUNTED = (
"prompts",
"generated_images",
"prompt_tags",
"image_prompt_pairs",
"workflow_documents",
"prompt_pairs",
"prompt_library_state",
"execution_run_contexts",
"tags",
)
# The ladder steps whose answer is certain enough to keep a link.
KEEP_SOURCES = ("recorded-run", "inspector")
# What must be unchanged since the dry run for a row to be written.
DRIFT_KEYS = (
"action",
"file",
"current_prompt_id",
"current_text_sha256",
"file_size",
"file_mtime_ns",
"target_text",
"target_hash",
"target_prompt_id",
)
# Locations reported only by runs without any PromptManager context.
_FOREIGN_LOCATIONS = (
"SELECT o.subfolder, o.filename FROM execution_run_outputs o "
"WHERE o.folder_type = 'output' AND o.root_key = ? AND o.path_style = ? "
"GROUP BY o.subfolder, o.filename "
"HAVING SUM(EXISTS (SELECT 1 FROM execution_run_contexts c "
"WHERE c.run_id = o.run_id)) = 0 "
"ORDER BY o.subfolder, o.filename"
)
class Refused(Exception):
"""A check failed: nothing is written."""
def sha256(text):
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def metadata_reader():
"""The scan's metadata extraction (PNG text chunks, ffprobe for videos)."""
api = SimpleNamespace(logger=logging.getLogger("prompt_manager.repair_247"))
return lambda path: PromptManagerAPI._extract_comfyui_metadata(api, path) or {}
def foreign_rows(db, output_dir):
"""The link rows of every file only foreign runs reported, by id."""
style = "windows" if os.name == "nt" else "posix"
root_key = snapshot_output_roots(style, output_dir, output_dir).output_key
conn = db.model.get_connection()
rows = {}
for subfolder, filename in conn.execute(_FOREIGN_LOCATIONS, (root_key, style)):
path = os.path.join(output_dir, subfolder, filename)
rows.update((row["id"], row) for row in file_link_rows(conn, path))
return [rows[key] for key in sorted(rows)]
class Judge:
"""One file's decision per path, for all of its rows."""
def __init__(self, db, output_dir, read=None):
self.db = db
self.output_dir = output_dir
self.read = read or metadata_reader()
self.decisions = {}
def decide(self, path, size):
if path not in self.decisions:
self.decisions[path] = decide_file_prompt(
self.read(path),
file_name=os.path.basename(path),
file_size=size,
in_output=True,
lookup=lambda: recorded_output(self.db, path, self.output_dir),
policy=Policy.SCAN,
)
return self.decisions[path]
def row(self, row):
"""The plan entry of one ``generated_images`` row."""
conn = self.db.model.get_connection()
path = row["image_path"]
(text,) = conn.execute(
"SELECT text FROM prompts WHERE id = ?", (row["prompt_id"],)
).fetchone()
prompt_hash = generate_prompt_hash(text)
item = {
"row_id": row["id"],
"file": path,
"current_prompt_id": row["prompt_id"],
"current_text_sha256": sha256(text),
}
try:
stat = os.stat(path)
except OSError:
return {**item, "action": "skip", "reason": "file gone"}
item.update(file_size=stat.st_size, file_mtime_ns=stat.st_mtime_ns)
decision = self.decide(path, stat.st_size)
action = action_of(decision, prompt_hash)
if action == "skip":
return {
**item,
"action": "skip",
"reason": "not decided by the recorded saver",
"decision": f"{decision.kind}/{decision.source}",
}
target_hash = generate_prompt_hash(decision.text)
stored = conn.execute(
"SELECT id FROM prompts WHERE hash = ?", (target_hash,)
).fetchone()
item.update(
target_text=decision.text,
target_hash=target_hash,
target_prompt_id=stored[0] if stored else "new",
action=action,
source=decision.source,
)
return item
def action_of(decision, linked_hash):
"""``relink``, ``keep`` or ``skip`` for a row linked to ``linked_hash``.
Only the recorded saver (``recorded-run``) names a new target. A certain
step (the recorded saver or the inspector) naming the linked prompt keeps
it; the legacy floor and every other answer are skipped.
"""
if decision.kind != POSITIVE or decision.source not in KEEP_SOURCES:
return "skip"
if generate_prompt_hash(decision.text) == linked_hash:
return "keep"
return "relink" if decision.source == "recorded-run" else "skip"
def library_identity(path):
"""How a plan names its library: the case-normalized real path."""
return os.path.normcase(os.path.realpath(path))
def build_plan(db, output_dir, source_db, read=None):
"""The plan for ``db`` (a copy of ``source_db``)."""
judge = Judge(db, output_dir, read)
conn = db.model.get_connection()
(max_row_id,) = conn.execute("SELECT MAX(id) FROM generated_images").fetchone()
return {
"version": PLAN_VERSION,
"created": datetime.datetime.now().isoformat(timespec="seconds"),
"source_db": library_identity(source_db),
"max_row_id": max_row_id,
"output_dir": output_dir,
"rows": [judge.row(row) for row in foreign_rows(db, output_dir)],
"orphans": [],
"delete_orphans": False,
}
def blockers(conn, prompt_id, unused):
"""Why ``prompt_id`` must not be deleted (empty: nothing references it)."""
found = []
values = {"id": prompt_id, "tag": SCAN_MARKS.tags[0]}
checks = {
"linked images": "SELECT COUNT(*) FROM generated_images WHERE prompt_id=:id",
"saved pairs": "SELECT COUNT(*) FROM prompt_pairs "
"WHERE positive_prompt_id=:id OR negative_prompt_id=:id",
"recorded runs": "SELECT COUNT(*) FROM execution_run_contexts "
"WHERE positive_prompt_id=:id OR negative_prompt_id=:id",
"library state": "SELECT COUNT(*) FROM prompt_library_state "
"WHERE prompt_id=:id",
"user tags": "SELECT COUNT(*) FROM prompt_tags pt JOIN tags t "
"ON t.id=pt.tag_id WHERE pt.prompt_id=:id AND t.name != :tag",
}
for name, sql in checks.items():
count = conn.execute(sql, values).fetchone()[0]
if count:
found.append(f"{name}: {count}")
run_count, last_used, rating, category, notes = conn.execute(
"SELECT run_count, last_used_at, rating, category, notes FROM prompts "
"WHERE id=?",
(prompt_id,),
).fetchone()
if run_count or last_used or rating:
found.append("usage or rating")
if category not in (None, SCAN_MARKS.category):
found.append(f"category {category}")
if notes and not notes.startswith("Auto-scanned from "):
found.append("notes")
if prompt_id not in unused:
found.append("not unused by the app's rule")
return found
def _rejudge(conn, judge, plan):
for item in plan["rows"]:
if item["action"] == "skip":
continue
row = conn.execute(
"SELECT * FROM generated_images WHERE id = ?", (item["row_id"],)
).fetchone()
if row is None:
raise Refused(f"row {item['row_id']}: gone")
if generate_prompt_hash(item.get("target_text") or "") != item.get(
"target_hash"
):
raise Refused(f"row {item['row_id']}: target_text does not match its hash")
fresh = judge.row(row)
for key in DRIFT_KEYS:
if fresh.get(key) != item.get(key):
raise Refused(
f"row {item['row_id']}: {key} changed "
f"({item.get(key)!r} -> {fresh.get(key)!r})"
)
def _targets(conn, db, relinks):
"""``{target hash: prompt id}``, creating each "new" target once."""
targets = {}
for item in relinks:
digest = item["target_hash"]
if digest in targets:
continue
if item["target_prompt_id"] != "new":
targets[digest] = item["target_prompt_id"]
continue
prompt_id, created = db._pair_component_outcome(conn, item["target_text"])
if not created:
raise Refused(f"row {item['row_id']}: target prompt already stored")
mark_created(conn, db, prompt_id, SCAN_MARKS, os.path.basename(item["file"]))
targets[digest] = prompt_id
return targets
def _relink(conn, item, target):
"""Point the row at ``target``; True when the wrong row was deleted instead."""
row_id = item["row_id"]
correct = [
row
for row in file_link_rows(conn, item["file"])
if row["prompt_id"] == target and row["id"] != row_id
]
if correct: # UNIQUE(prompt_id, file_path): the correct row stays alone
if _delete_ids(conn, "generated_images", [row_id]) != 1:
raise Refused(f"row {row_id}: delete touched no row")
return True
# A pair link records the old positive: it goes with the old link
conn.execute("DELETE FROM image_prompt_pairs WHERE image_id = ?", (row_id,))
cursor = conn.execute(
"UPDATE generated_images SET prompt_id = ? WHERE id = ? AND prompt_id = ?",
(target, row_id, item["current_prompt_id"]),
)
if cursor.rowcount != 1:
raise Refused(f"row {row_id}: relink touched {cursor.rowcount} rows")
return False
def _audit(conn, relinks):
"""The prompts the relinks leave without any link, with their blockers."""
sources = {}
for item in relinks:
sources.setdefault(item["current_prompt_id"], []).append(item)
unused = {row[0] for row in conn.execute(ORPHANED_PROMPTS_SQL)}
audit = []
for prompt_id in sorted(sources):
if conn.execute(
"SELECT 1 FROM generated_images WHERE prompt_id = ?", (prompt_id,)
).fetchone():
continue
(text,) = conn.execute(
"SELECT text FROM prompts WHERE id = ?", (prompt_id,)
).fetchone()
found = blockers(conn, prompt_id, unused)
audit.append(
{
"id": prompt_id,
"text_excerpt": " ".join(text.split())[:EXCERPT],
"text_sha256": sha256(text),
"images_before": len(sources[prompt_id]),
"blockers": found,
"safe": not found,
}
)
return audit
def _delete_orphans(conn, plan):
chosen = [orphan for orphan in plan["orphans"] if orphan["safe"]]
unused = {row[0] for row in conn.execute(ORPHANED_PROMPTS_SQL)}
for orphan in chosen:
row = conn.execute(
"SELECT text FROM prompts WHERE id = ?", (orphan["id"],)
).fetchone()
if row is None or sha256(row[0]) != orphan["text_sha256"]:
raise Refused(f"prompt {orphan['id']}: gone or text changed")
found = blockers(conn, orphan["id"], unused)
if found:
raise Refused(f"prompt {orphan['id']}: {', '.join(found)}")
ids = [orphan["id"] for orphan in chosen]
if _delete_ids(conn, "prompts", ids) != len(ids):
raise Refused("an orphan prompt was not deleted")
return len(ids)
def apply_plan(db, plan, read=None, before_write=None):
"""Write ``plan`` in one transaction; returns ``(result, orphan audit)``.
``before_write()`` runs just before ``BEGIN IMMEDIATE`` (the running
check). Raises ``Refused`` (and rolls back) on any drift or failed check.
"""
judge = Judge(db, plan["output_dir"], read)
relinks = [item for item in plan["rows"] if item["action"] == "relink"]
if before_write is not None:
before_write()
with db._pair_transaction() as conn:
_rejudge(conn, judge, plan)
targets = _targets(conn, db, relinks)
dropped = sum(
_relink(conn, item, targets[item["target_hash"]]) for item in relinks
)
audit = _audit(conn, relinks)
deleted = _delete_orphans(conn, plan) if plan.get("delete_orphans") else 0
created = sum(1 for item in relinks if item["target_prompt_id"] == "new")
result = {
"relinked": len(relinks) - dropped,
"duplicates_dropped": dropped,
"prompts_created": len(
{i["target_hash"] for i in relinks if i["target_prompt_id"] == "new"}
),
"rows_to_new_prompts": created,
"kept": sum(1 for item in plan["rows"] if item["action"] == "keep"),
"orphans_deleted": deleted,
}
return result, audit
def counts(path, readonly=True):
"""Table counts plus the integrity and foreign-key checks.
Read-only for a library; the tool's own backup is opened read-write so
a WAL file leaves no side files behind once it closes.
"""
uri = Path(os.path.abspath(path)).as_uri() + ("?mode=ro" if readonly else "")
conn = sqlite3.connect(uri, uri=True)
try:
found = {
table: conn.execute(f'SELECT COUNT(*) FROM "{table}"').fetchone()[0]
for table in COUNTED # nosec B608 - fixed table names
}
found["integrity_check"] = conn.execute("PRAGMA integrity_check").fetchone()[0]
found["foreign_key_check"] = len(
conn.execute("PRAGMA foreign_key_check").fetchall()
)
return found
finally:
conn.close()
def _print_counts(before, after, out):
out.write(f"{'':24}{'before':>12}{'after':>12}\n")
for key in before:
out.write(f"{key:24}{before[key]!s:>12}{after[key]!s:>12}\n")
def _summary(plan):
actions = [item["action"] for item in plan["rows"]]
return " ".join(
f"{name}: {actions.count(name)}" for name in ("relink", "keep", "skip")
)
def _cell(text, limit=EXCERPT):
"""One table cell: whitespace folded, cut to ``limit`` (None: whole)."""
folded = " ".join(str(text).split())
return (folded if limit is None else folded[:limit]).replace("|", "\\|")
def _relative(path, root):
try:
return os.path.relpath(path, root)
except ValueError: # another drive on Windows
return path
def markdown(plan, prompt_texts):
"""The maintainer's list of the plan."""
root = plan["output_dir"]
rel = lambda path: _relative(path, root) # noqa: E731
rows = plan["rows"]
lines = [
"# #247 repair plan (dry run)",
"",
f"Created {plan['created']}; output folder `{root}`. {_summary(plan)}.",
"",
f"## Relink ({sum(i['action'] == 'relink' for i in rows)})",
"",
"| row | file | linked prompt | linked text | target | target hash "
"| target text |",
"|---:|---|---:|---|---|---|---|",
]
for item in (i for i in rows if i["action"] == "relink"):
lines.append(
f"| {item['row_id']} | {_cell(rel(item['file']), None)} | "
f"{item['current_prompt_id']} | "
f"{_cell(prompt_texts.get(item['current_prompt_id'], ''))} | "
f"{item['target_prompt_id']} | `{item['target_hash'][:12]}` | "
f"{_cell(item['target_text'])} |"
)
lines += ["", f"## Keep ({sum(i['action'] == 'keep' for i in rows)})", ""]
lines += ["| row | file | prompt |", "|---:|---|---:|"]
for item in (i for i in rows if i["action"] == "keep"):
lines.append(
f"| {item['row_id']} | {_cell(rel(item['file']), None)} | "
f"{item['current_prompt_id']} |"
)
lines += ["", f"## Skip ({sum(i['action'] == 'skip' for i in rows)})", ""]
lines += ["| row | file | prompt | reason |", "|---:|---|---:|---|"]
for item in (i for i in rows if i["action"] == "skip"):
reason = item["reason"] + (
f" ({item['decision']})" if item.get("decision") else ""
)
lines.append(
f"| {item['row_id']} | {_cell(rel(item['file']), None)} | "
f"{item['current_prompt_id']} | {reason} |"
)
choice = "ON (--delete-orphans)" if plan["delete_orphans"] else "OFF"
lines += [
"",
f"## Prompts that would lose every link ({len(plan['orphans'])})",
"",
f"Deleting the safe ones: {choice}.",
"",
"| id | text | images before | blockers | safe to delete |",
"|---:|---|---:|---|---|",
]
for orphan in plan["orphans"]:
lines.append(
f"| {orphan['id']} | {_cell(orphan['text_excerpt'])} | "
f"{orphan['images_before']} | {', '.join(orphan['blockers']) or 'none'} | "
f"{'yes' if orphan['safe'] else 'no'} |"
)
simulated = plan.get("simulated")
if simulated:
lines += ["", "## Rehearsed on the dry run's copy", ""]
lines += ["| table | before | after |", "|---|---:|---:|"]
for key, value in simulated["before"].items():
lines.append(f"| {key} | {value} | {simulated['after'][key]} |")
return "\n".join(lines) + "\n"
def _texts(db):
return dict(
db.model.get_connection().execute(
"SELECT id, text FROM prompts WHERE id IN (SELECT prompt_id "
"FROM generated_images)"
)
)
def _rehearse(source, scratch, name, plan):
"""``apply_plan`` on a fresh backup of ``source``: (counts, result, audit)."""
copy = os.path.join(scratch, name)
backup_database(source, copy)
db = PromptDatabase(copy)
try:
before = counts(copy)
result, audit = apply_plan(db, plan)
finally:
db.close_all()
return {"before": before, "after": counts(copy), "result": result}, audit
def dry_run(args, out):
with tempfile.TemporaryDirectory() as scratch:
copy = os.path.join(scratch, "judge.db")
backup_database(args.database, copy)
db = PromptDatabase(copy)
try:
plan = build_plan(db, os.path.abspath(args.output_dir), args.database)
texts = _texts(db)
finally:
db.close_all()
simulated, audit = _rehearse(args.database, scratch, "relinks.db", plan)
plan.update(orphans=audit, delete_orphans=bool(args.delete_orphans))
if plan["delete_orphans"]:
# The plan as the apply will write it, deletes included
simulated, _audit = _rehearse(args.database, scratch, "deletes.db", plan)
plan["simulated"] = simulated
plan_path = Path(args.plan)
plan_path.write_text(
json.dumps(plan, indent=1, ensure_ascii=False) + "\n", encoding="utf-8"
)
list_path = plan_path.with_name(plan_path.stem + ".list.md")
list_path.write_text(markdown(plan, texts), encoding="utf-8")
out.write(f"{_summary(plan)}\n")
out.write(
f"Prompts that would lose every link: {len(audit)} "
f"({sum(o['safe'] for o in audit)} safe to delete; deleting "
f"{'ON' if plan['delete_orphans'] else 'OFF'})\n"
)
out.write(f"Plan: {plan_path}\nList: {list_path}\n")
out.write("Dry run: nothing was changed.\n")
return 0
def _is_live(database, comfyui_dir):
live = os.path.join(comfyui_dir, *LIVE_LIBRARY)
return library_identity(database) == library_identity(live)
def _stamp():
return datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
def schema_of(path):
"""``sqlite_master`` and ``user_version`` from a read-only connection."""
uri = Path(os.path.abspath(path)).as_uri() + "?mode=ro"
conn = sqlite3.connect(uri, uri=True)
try:
master = conn.execute(
"SELECT type, name, tbl_name, sql FROM sqlite_master ORDER BY type, name"
).fetchall()
return master, conn.execute("PRAGMA user_version").fetchone()[0]
finally:
conn.close()
def running_reasons(args, probe, handles):
"""Why writing must not start now (empty: go ahead).
Another process holding the library open always counts, also in a
rehearsal; ComfyUI's processes and port count unless rehearsing.
"""
held = [
reason
for reason in handles(args.database)
if not (reason == NO_PROC and args.i_stopped_comfyui)
]
return held + ([] if args.rehearsal else probe(args.comfyui_dir, args.port))
def _preflight(args, plan, probe, handles):
"""The refusals before any backup or write, as a message (None: go on)."""
if plan.get("version") != PLAN_VERSION:
return "not a plan of this tool"
if args.rehearsal and _is_live(args.database, args.comfyui_dir):
return f"--rehearsal is for a copy; this is the library of {args.comfyui_dir}"
if plan.get("source_db") != library_identity(args.database):
return f"this plan was made for {plan.get('source_db')}"
if not os.path.isfile(os.path.join(args.comfyui_dir, "main.py")):
return f"no main.py in {args.comfyui_dir}: not a ComfyUI folder"
reasons = running_reasons(args, probe, handles)
if reasons:
return "ComfyUI may be running: " + "; ".join(reasons)
return None
def _backup(args, before):
"""The verified backup's path; raises ``Refused``."""
folder = args.backup_dir or os.path.dirname(os.path.abspath(args.database))
name = f"{os.path.basename(args.database)}.pre-247-repair_{_stamp()}"
backup = os.path.join(folder, name)
if os.path.exists(backup):
raise Refused(f"the backup {backup} already exists")
backup_database(args.database, backup)
saved = counts(backup, readonly=False)
if saved != before or saved["integrity_check"] != "ok":
raise Refused(f"the backup {backup} does not match the library")
return backup
def _open_unchanged(path):
"""``PromptDatabase(path)``; raises ``Refused`` when opening it changed
its schema (a migration of a newer checkout, committed on open)."""
schema = schema_of(path)
db = PromptDatabase(path)
db.model.get_connection()
if schema_of(path) != schema:
db.close_all()
raise Refused(
"opening the library with this checkout changed its schema (that "
"change is committed); run the tool from a checkout at the live "
"code version"
)
return db
def apply(args, out, probe, handles):
with open(args.apply, encoding="utf-8") as stream:
plan = json.load(stream)
refusal = _preflight(args, plan, probe, handles)
if refusal:
out.write(f"REFUSED: {refusal}\n")
return 2
if args.rehearsal:
out.write("(rehearsal on a copy: ComfyUI's process and port not checked)\n")
before = counts(args.database)
try:
backup = _backup(args, before)
except Refused as error:
out.write(f"REFUSED: {error}\n")
return 2
out.write(f"Backup: {backup}\n")
def still_stopped():
reasons = running_reasons(args, probe, handles)
if reasons:
raise Refused("ComfyUI may be running: " + "; ".join(reasons))
try:
db = _open_unchanged(args.database)
except Refused as error:
out.write(f"REFUSED: {error}. Backup: {backup}\n")
return 2
try:
result, _audit = apply_plan(db, plan, before_write=still_stopped)
except Exception as error:
out.write(
f"REFUSED (rolled back, nothing changed): {error}. Backup: {backup}\n"
)
return 2
finally:
db.close_all()
after = counts(args.database)
_print_counts(before, after, out)
out.write(f"Applied: {json.dumps(result, sort_keys=True)}\n")
if after["integrity_check"] != "ok" or after["foreign_key_check"]:
out.write(f"WARNING: integrity or foreign-key check failed; restore {backup}\n")
return 1
return 0
def main(argv=None, out=sys.stdout, probe=comfyui_running, handles=open_by_others):
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
parser.add_argument("database", help="the library database (prompts.db)")
parser.add_argument("--output-dir", help="ComfyUI's output folder (dry run)")
parser.add_argument(
"--plan", default="repair_foreign_links_plan.json", help="plan to write"
)
parser.add_argument(
"--delete-orphans",
action="store_true",
help="record in the plan that the safe orphan prompts are deleted",
)
parser.add_argument("--apply", metavar="PLAN", help="write this reviewed plan")
parser.add_argument("--comfyui-dir", help="the ComfyUI folder (apply)")
parser.add_argument("--backup-dir", help="where the backup goes (apply)")
parser.add_argument("--port", type=int, default=COMFY_PORT)
parser.add_argument(
"--rehearsal",
action="store_true",
help="apply to a copy while ComfyUI runs (never its own library)",
)
parser.add_argument(
"--i-stopped-comfyui",
action="store_true",
help="without /proc: say ComfyUI is stopped (the port is still checked)",
)
args = parser.parse_args(argv)
if not os.path.isfile(args.database):
parser.error(f"no database at {args.database}")
if args.apply:
if not args.comfyui_dir:
parser.error("--apply needs --comfyui-dir (the running check)")
return apply(args, out, probe, handles)
if not args.output_dir:
parser.error("the dry run needs --output-dir")
if Path(args.plan).suffix.lower() == ".md":
parser.error("--plan is the JSON plan; the list is written beside it")
return dry_run(args, out)
if __name__ == "__main__":
sys.exit(main())
-142
View File
@@ -1,142 +0,0 @@
"""The monitor's path for outputs of workflows without a PromptManager node (#247).
A foreign output (no positive PromptManager node in its graph) is decided by
the ladder the output scan uses (``utils.file_prompt.decide_file_prompt``)
with the ``MONITOR`` policy: certain files are stored at once, files only
recorded history can decide wait for their run's outputs to be sealed, and
everything else stays unlinked. This module holds the parts that do not
need the monitor's waiting sets: the setting, the decision call and the
store. The waiting and the re-check stay on ``ImageGenerationHandler``.
"""
import os
from dataclasses import dataclass
from typing import Any, Callable, Mapping, Optional, Tuple
from .file_prompt import (
NOT_SAVED,
NOT_SAVED_WIRED,
PAIR,
POSITIVE,
Decision,
Policy,
decide_file_prompt,
)
from .hashing import generate_prompt_hash
from .usage_tracking import survive_switch
try:
from ..database.pair_ingestion import RUN_MARKS, ingest_resolved_pair
from ..database.run_attribution import recorded_output_lookup
from ..database.scanned_prompts import store_scanned_prompt_or_raise
except ImportError:
from database.pair_ingestion import RUN_MARKS, ingest_resolved_pair
from database.run_attribution import recorded_output_lookup
from database.scanned_prompts import store_scanned_prompt_or_raise
# Decisions that name a prompt to store
CERTAIN = (PAIR, POSITIVE)
# Decisions of a run whose prompt has Save to library off, or wired from a
# node the file cannot show (#90): final, nothing is stored, nothing waits
NOT_SAVED_KINDS = (NOT_SAVED, NOT_SAVED_WIRED)
# Sources certain without a recorded run: the graph alone, or its recorded saver
ANCHORED_SOURCES = ("inspector", "recorded-run")
@dataclass(frozen=True)
class ForeignRecheck:
"""``schedule`` snapshot: the one re-check of a foreign output after its
run's outputs were sealed. ``reference`` is the sealed output reference,
the spelling history recorded (``recorded_output_lookup(reference=)``)."""
reference: Mapping[str, Any]
def save_foreign_runs() -> bool:
"""Whether a foreign output may be stored now, read for every file.
``GalleryConfig.SAVE_FOREIGN_RUNS`` and ``MONITORING_ENABLED``: a
monitoring switch saved off stops foreign writes at once, although the
running observer watches until a restart. True when the configuration
cannot be imported (the defaults).
"""
try:
from ..py.config import GalleryConfig
except ImportError:
try:
from py.config import GalleryConfig
except ImportError:
return True
return (
GalleryConfig.SAVE_FOREIGN_RUNS is not False
and GalleryConfig.MONITORING_ENABLED is not False
)
def history_lookup(
db: Any, image_path: str, reference: Optional[Mapping[str, Any]] = None
) -> Callable[[], Any]:
"""The ladder's ``lookup`` for one file: its ``RecordedOutput`` or None.
With ``reference`` the file is spelled as history recorded it. Failures
are logged by ``recorded_output_lookup`` and name nothing.
"""
lookup = recorded_output_lookup(
db, reference=dict(reference) if reference is not None else None
)
return lambda: lookup(image_path)
def decide_foreign(
image_path: str,
metadata: Any,
*,
in_output: bool,
lookup: Optional[Callable[[], Any]],
) -> Decision:
"""The ``MONITOR`` ladder for one foreign output (the size is read only
when the ladder needs it)."""
def file_size() -> Optional[int]:
try:
return os.path.getsize(image_path)
except OSError:
return None
return decide_file_prompt(
metadata,
file_name=os.path.basename(image_path),
file_size=file_size,
in_output=in_output,
lookup=lookup,
policy=Policy.MONITOR,
)
def store_foreign(
db: Any, image_path: str, decision: Decision
) -> Tuple[str, Optional[int]]:
"""Store a certain decision: ``(status, prompt id)``.
One write through ``survive_switch``, so it runs again on the moved
library when a Move library switched while it waited (#249 S6b). Created
prompts carry ``RUN_MARKS``. A file linked to another prompt is a
``conflict`` and nothing is written. Database errors are raised.
"""
def store():
if decision.kind == PAIR:
result = ingest_resolved_pair(
db, image_path, decision.canonical, decision.resolution, marks=RUN_MARKS
)
return result["status"], result.get("image_id")
return store_scanned_prompt_or_raise(
db, image_path, decision.canonical, decision.text, RUN_MARKS
)
status, linked = survive_switch(db, store)
if decision.kind != PAIR or linked is None:
return status, linked
# The pair ingester reports the image row; its prompt is the positive's.
prompt = db.get_prompt_by_hash(generate_prompt_hash(decision.text))
return status, prompt["id"] if prompt else None
+4 -65
View File
@@ -58,72 +58,11 @@ def generate_prompt_hash(text: str) -> str:
if not isinstance(text, str):
raise TypeError("Text must be a string")
return hashlib.sha256(prompt_key(text).encode("utf-8")).hexdigest()
# Normalize the text by stripping whitespace and converting to lowercase
# for consistent hashing regardless of minor formatting differences
normalized_text = text.strip().lower()
def stored_prompt_text(text: str) -> str:
"""The text the Library stores for raw node text: outer whitespace removed.
Every prompt writer stores ``text.strip()`` and the hash above strips too,
so this is the form recorded raw run text is compared in (#438). Case and
inner whitespace stay significant.
"""
return text.strip()
# The code points str.strip() removes (every str.isspace() character), for
# SQLite's char(); tests/test_hashing.py pins it to the running Python.
STRIPPED_CODE_POINTS = (
*range(0x09, 0x0E),
*range(0x1C, 0x21),
0x85,
0xA0,
0x1680,
*range(0x2000, 0x200B),
0x2028,
0x2029,
0x202F,
0x205F,
0x3000,
)
_STRIP_CODES = ",".join(map(str, STRIPPED_CODE_POINTS))
def prompt_key(text: str) -> str:
"""The dedup key: the stored text with case folded by ``str.lower()``.
Two texts with one key are one prompt (the hash is sha256 of the key), so
a run belongs to the history of the prompt with its key (#460).
"""
return stored_prompt_text(text).lower()
def _sql_key(value):
return prompt_key(value) if isinstance(value, str) else None
def _sql_hash(value):
return generate_prompt_hash(value) if isinstance(value, str) else None
def register_prompt_functions(conn) -> None:
"""Give ``conn`` ``pm_prompt_key(x)`` and ``pm_prompt_hash(x)`` (#460).
SQLite's lower() folds ASCII only; these call Python, so SQL gets exactly
``prompt_key`` / ``generate_prompt_hash`` (NULL for a non-text value).
Never used in schema objects, so the file stays readable without them.
"""
conn.create_function("pm_prompt_key", 1, _sql_key, deterministic=True)
conn.create_function("pm_prompt_hash", 1, _sql_hash, deterministic=True)
def stored_prompt_text_sql(expression: str) -> str:
"""``stored_prompt_text`` as an SQLite expression over ``expression``.
SQLite's one-argument trim() removes spaces only; trim(X, Y) removes every
character of Y, multi-byte ones included, so it strips what Python does.
"""
return f"trim({expression}, char({_STRIP_CODES}))"
return hashlib.sha256(normalized_text.encode("utf-8")).hexdigest()
def is_duplicate_prompt(text1: str, text2: str, threshold: float = 0.95) -> bool:
+41 -201
View File
@@ -33,17 +33,6 @@ from typing import Callable, Optional, Dict, Any, List, Tuple
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
from .file_prompt import HISTORY, NOT_SAVED, NOT_SAVED_WIRED
from .foreign_runs import (
ANCHORED_SOURCES,
CERTAIN,
NOT_SAVED_KINDS,
ForeignRecheck,
decide_foreign,
history_lookup,
save_foreign_runs,
store_foreign,
)
from .hashing import generate_prompt_hash
from .metadata_extractor import ComfyUIMetadataExtractor
from .parallel import note_refusal, refused_at_shutdown
@@ -164,11 +153,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
# Sealed re-checks ``schedule`` refused (the path still pending, or the
# pending set full): retried each time a pending image finishes.
self._history_due: "OrderedDict[str, None]" = OrderedDict()
# The same two sets for foreign outputs (#247), apart so a large
# foreign run never evicts a waiting keyless file; the due set keeps
# each file's sealed output reference.
self._awaiting_foreign: "OrderedDict[str, Tuple[str, float]]" = OrderedDict()
self._foreign_due: "OrderedDict[str, dict]" = OrderedDict()
self._awaiting_lock = threading.Lock()
self._sealed_subscribed = False
# Set by ``close``: no new work, no new worker, ever (#302).
@@ -269,8 +253,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
with self._awaiting_lock:
self._awaiting.clear()
self._history_due.clear()
self._awaiting_foreign.clear()
self._foreign_due.clear()
with self._pending_lock:
self._pending.clear()
worker, self._worker = self._worker, None
@@ -456,9 +438,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
if prompt_snapshot is HISTORY_ONLY:
self._recheck_history(image_path)
return
if isinstance(prompt_snapshot, ForeignRecheck):
self._recheck_foreign(image_path, prompt_snapshot.reference)
return
self.logger.info(f"Processing image: {image_path}")
if not os.path.exists(image_path):
@@ -478,12 +457,17 @@ class ImageGenerationHandler(FileSystemEventHandler):
# negative-prompt PromptManager nodes in the workflow.
state, current_prompt = self._find_prompt_from_metadata(metadata)
if state == "opted_out":
self._log_not_saved(image_path)
self.logger.info(
"Save to library is off for this run's prompt, skipping image: "
f"{image_path}"
)
return
if state == "foreign":
# Another workflow saved this image: never a batch queue entry
# of ours; its own graph may name its prompt (#247).
self._ingest_foreign(image_path, metadata)
# Another workflow saved this image: it belongs to no prompt of
# ours, and must not consume a batch queue entry.
self.logger.info(
f"No PromptManager prompt in workflow, skipping image: {image_path}"
)
return
if current_prompt:
self.logger.info(
@@ -605,55 +589,36 @@ class ImageGenerationHandler(FileSystemEventHandler):
resolved = os.path.abspath(image_path)
return os.path.normcase(resolved)
def _prune_awaiting(self, store=None) -> List[Tuple[str, bool]]:
"""Drop expired and surplus entries of ``store`` (the keyless set by
default; ``_awaiting_lock`` held). Returns the dropped foreign files
as ``(path, evicted)`` for ``_log_dropped`` after the lock."""
store = self._awaiting if store is None else store
def _prune_awaiting(self) -> None:
"""Drop expired and surplus entries (``_awaiting_lock`` held)."""
now = time.monotonic()
expired = [k for k, (_p, due) in store.items() if due <= now]
dropped = [(store.pop(key)[0], False) for key in expired]
while len(store) > MAX_AWAITING_HISTORY:
dropped.append((store.popitem(last=False)[1][0], True))
return dropped if store is self._awaiting_foreign else []
for key in [k for k, (_p, due) in self._awaiting.items() if due <= now]:
del self._awaiting[key]
while len(self._awaiting) > MAX_AWAITING_HISTORY:
self._awaiting.popitem(last=False)
def _log_dropped(self, dropped: List[Tuple[str, bool]]) -> None:
"""Log foreign files that stopped waiting (outside ``_awaiting_lock``)."""
for image_path, evicted in dropped:
name = os.path.basename(image_path)
if evicted:
self.logger.info(
f"Too many files wait for their run; {name} left unlinked"
)
else:
self.logger.info(f"No recorded run for {name}; left unlinked")
def _await_history(self, image_path: str, store=None) -> None:
def _await_history(self, image_path: str) -> None:
"""Re-check ``image_path`` once, history only, when its run is sealed.
``store`` is the waiting set (``_awaiting_foreign`` for a foreign
output). A closed handler re-checks nothing.
A closed handler re-checks nothing.
"""
if self._closed:
return
if not self._sealed_subscribed:
subscribe_outputs_sealed(self._on_outputs_sealed)
self._sealed_subscribed = True
store = self._awaiting if store is None else store
key = self._awaiting_key(image_path)
with self._awaiting_lock:
store.pop(key, None)
store[key] = (image_path, time.monotonic() + AWAIT_HISTORY_SECONDS)
dropped = self._prune_awaiting(store)
self._log_dropped(dropped)
self._awaiting.pop(key, None)
self._awaiting[key] = (
image_path,
time.monotonic() + AWAIT_HISTORY_SECONDS,
)
self._prune_awaiting()
def _await_foreign(self, image_path: str) -> None:
self._await_history(image_path, self._awaiting_foreign)
def _stop_awaiting(self, image_path: str, store=None) -> None:
store = self._awaiting if store is None else store
def _stop_awaiting(self, image_path: str) -> None:
with self._awaiting_lock:
store.pop(self._awaiting_key(image_path), None)
self._awaiting.pop(self._awaiting_key(image_path), None)
def awaiting_history(self) -> List[str]:
"""Normalized paths still waiting for their run's recorded outputs."""
@@ -661,14 +626,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
self._prune_awaiting()
return list(self._awaiting)
def awaiting_foreign(self) -> List[str]:
"""Normalized paths of foreign outputs waiting for their run (#247)."""
with self._awaiting_lock:
dropped = self._prune_awaiting(self._awaiting_foreign)
waiting = list(self._awaiting_foreign)
self._log_dropped(dropped)
return waiting
def _on_outputs_sealed(self, run_id: str, references: List[dict]) -> None:
"""A run's outputs were recorded: re-check its waiting images once.
@@ -680,7 +637,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
due = []
with self._awaiting_lock:
self._prune_awaiting()
dropped = self._prune_awaiting(self._awaiting_foreign)
for reference in references:
if reference.get("folder_type") != "output":
continue
@@ -691,53 +647,39 @@ class ImageGenerationHandler(FileSystemEventHandler):
)
waiting = self._awaiting.pop(key, None)
if waiting is not None:
due.append((waiting[0], None))
waiting = self._awaiting_foreign.pop(key, None)
if waiting is not None:
due.append((waiting[0], dict(reference)))
self._log_dropped(dropped)
for image_path, reference in due:
self._schedule_recheck(image_path, reference)
due.append(waiting[0])
for image_path in due:
self._schedule_recheck(image_path)
@staticmethod
def _recheck_snapshot(reference):
return HISTORY_ONLY if reference is None else ForeignRecheck(reference)
def _schedule_recheck(self, image_path: str, reference=None) -> None:
"""Schedule the history-only re-check (with the sealed ``reference``:
a foreign output's), or keep it due for later.
def _schedule_recheck(self, image_path: str) -> None:
"""Schedule the history-only re-check, or keep it due for later.
A closed handler drops it.
"""
if self._closed:
return
if self.schedule(image_path, self._recheck_snapshot(reference)):
if self.schedule(image_path, HISTORY_ONLY):
return
with self._awaiting_lock:
# ``close`` sets the flag before ``shutdown`` clears this set, so
# a close that raced the refusal above is seen here.
if self._closed:
return
due = self._history_due if reference is None else self._foreign_due
due.pop(image_path, None)
due[image_path] = reference
evicted = []
while len(due) > MAX_AWAITING_HISTORY:
evicted.append(due.popitem(last=False)[0])
if reference is not None:
self._log_dropped([(path, True) for path in evicted])
self._history_due.pop(image_path, None)
self._history_due[image_path] = None
while len(self._history_due) > MAX_AWAITING_HISTORY:
self._history_due.popitem(last=False)
def _drain_history_due(self) -> None:
"""Schedule due re-checks now that a pending slot or path is free."""
with self._awaiting_lock:
due = [(self._history_due, p, None) for p in self._history_due]
due += [(self._foreign_due, *item) for item in self._foreign_due.items()]
for store, image_path, reference in due:
if self.schedule(image_path, self._recheck_snapshot(reference)):
if not self._history_due:
return
due = list(self._history_due)
for image_path in due:
if self.schedule(image_path, HISTORY_ONLY):
with self._awaiting_lock:
# A newer seal may have replaced it meanwhile: keep that one
if image_path in store and store[image_path] is reference:
del store[image_path]
self._history_due.pop(image_path, None)
def _recheck_history(self, image_path: str) -> None:
"""The one history-only re-check of a keyless image (#342)."""
@@ -761,108 +703,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
metadata or {"file_info": self.get_basic_file_info(image_path)},
)
def _ingest_foreign(self, image_path: str, metadata) -> None:
"""A foreign output at settle: store its certain prompt, or wait for
its run's recorded outputs when only history can decide (#247)."""
name = os.path.basename(image_path)
if not save_foreign_runs():
self.logger.info(
f"No PromptManager prompt in workflow, skipping image: {image_path}"
)
return
in_output = in_output_folder(image_path, host_output_directory())
decision = decide_foreign(
image_path, metadata, in_output=in_output, lookup=None
)
if decision.kind == HISTORY:
decision = self._settle_lookup(image_path, metadata)
if decision is None:
self.logger.debug(f"{name} waits for its run to end")
return
self._stop_awaiting(image_path, self._awaiting_foreign)
if decision.kind in NOT_SAVED_KINDS:
self._log_not_saved(image_path, decision.kind)
elif decision.kind in CERTAIN:
self._store_foreign(image_path, decision)
else:
self.logger.info(f"No certain prompt for {name}; left unlinked")
def _log_not_saved(self, image_path: str, kind: str = NOT_SAVED) -> None:
"""The log line of a file whose run's prompt does not save (#90)."""
if kind == NOT_SAVED_WIRED:
reason = "is wired from a node the image cannot show"
else:
reason = "is off"
self.logger.info(
f"Save to library {reason} for this run's prompt, skipping image: "
f"{image_path}"
)
def _settle_lookup(self, image_path: str, metadata):
"""The settle-time history lookup of a file only history can decide.
Registered first, so a run sealed before this lookup is not missed.
Returns None to keep waiting: no record yet, and the answer is not
the inspector's or the recorded saver's (the legacy floor waits for
the re-check). With a record, any decision ends the wait. A
``not_saved`` here is the undecided step's (an early one never asks
history, so it never reaches this lookup): a recorded saver may still
resolve another branch, so without a record it waits too (#90).
"""
self._await_foreign(image_path)
found = []
lookup = history_lookup(self.db_manager, image_path)
def recorded():
found.append(lookup())
return found[-1]
decision = decide_foreign(image_path, metadata, in_output=True, lookup=recorded)
if any(record is not None for record in found):
return decision
if decision.kind in CERTAIN and decision.source in ANCHORED_SOURCES:
return decision
return None
def _recheck_foreign(self, image_path: str, reference) -> None:
"""The one re-check of a foreign output after its run ended (#247)."""
name = os.path.basename(image_path)
if not os.path.exists(image_path):
return
if not save_foreign_runs():
self.logger.info(f"Saving prompts from other workflows is off: {name}")
return
try:
metadata = self.metadata_extractor.extract_metadata(image_path)
except Exception as error:
self.logger.warning(f"Metadata of {name} could not be read: {error}")
return
lookup = history_lookup(self.db_manager, image_path, reference)
decision = decide_foreign(image_path, metadata, in_output=True, lookup=lookup)
if decision.kind in NOT_SAVED_KINDS:
self._log_not_saved(image_path, decision.kind)
return
if decision.kind in CERTAIN:
self._store_foreign(image_path, decision)
return
self.logger.info(
f"No certain prompt for {name} after its run ended; left unlinked"
)
def _store_foreign(self, image_path: str, decision) -> None:
name = os.path.basename(image_path)
if not save_foreign_runs(): # switched off while this file was decided
self.logger.info(f"Saving prompts from other workflows is off: {name}")
return
try:
status, prompt_id = store_foreign(self.db_manager, image_path, decision)
except Exception as error:
self.logger.error(f"Failed to store the prompt of {name}: {error}")
return
self.logger.info(
f"Foreign run: {status} prompt {prompt_id} for {name} ({decision.source})"
)
def _history_prompt(self, image_path: str) -> Optional[int]:
"""The stored prompt recorded history attributes to this file, if one."""
try:
+1 -6
View File
@@ -548,12 +548,8 @@ def encode_history_inspection_response(
resolution: dict,
inspection: "InspectionRead | None",
library: dict,
image_tags: dict | None = None,
) -> bytes:
"""Use occurrence identity with the same current-file evidence projection.
*image_tags*, when given, adds the validated ``image_tags`` facet (#457).
"""
"""Use occurrence identity with the same current-file evidence projection."""
return _encode_inspection(
identity={
"history_version": 1,
@@ -565,5 +561,4 @@ def encode_history_inspection_response(
resolution=resolution,
inspection=inspection,
library=library,
image_tags=image_tags,
)
+1 -1
View File
@@ -173,7 +173,7 @@ export function mountRecordedHistory(container, {api, positiveId, parentScope, s
}
count.textContent = window.group ? `${countOf(window.recorded_run_count, 'distinct recorded run')} as of this read · ${window.items.length} loaded positions` : '';
const group = window.group;
scopeText.textContent = group?.scope_status === 'uncapturable_text' ? 'This saved positive cannot be matched within frozen-text capture limits.' : 'Runs are grouped by frozen positive raw text that is this saved text apart from outer whitespace, independently of saved pairs. Effective text, prepend/append composition and evidence can differ. Reported output locations do not establish context membership or historical file identity.';
scopeText.textContent = group?.scope_status === 'uncapturable_text' ? 'This saved positive cannot be matched within frozen-text capture limits.' : 'Runs are grouped by exact frozen positive raw text, independently of saved pairs. Effective text, prepend/append composition and evidence can differ. Reported output locations do not establish context membership or historical file identity.';
status.setAttribute('role', window.error || next.run.error ? 'alert' : 'status');
status.textContent = next.validating ? 'Revalidating recorded history…' : window.loading ? 'Loading recorded history…' : window.error ? 'Recorded history could not be loaded.' : next.refreshRequired ? 'Recorded evidence changed. Reload before using these actions.' : !items.length ? 'No matching recorded executions.' : window.has_more ? 'Newest bounded window · displayed in chronological order.' : 'All matching recorded runs in this window are loaded.';
if (next.run.error && !next.selection.run_id && !next.refreshRequired) status.textContent += ' The previous recorded selection could not be restored.';
+8 -49
View File
@@ -115,39 +115,9 @@ export function mountInspect(main, { parentScope, api, sender }) {
retiring.delete(url);
entry.waiting.abort(); clearTimeout(entry.timer);
releaseMedia(entry.preview);
if (entry.revoke) revokeObjectURL(url);
if (entry.revoke) URL.revokeObjectURL(url);
entry.done();
}
// #483 (D62): the viewer's own video, when a new selection closes the viewer while that video is still starting.
// The viewer stays open until the video reports metadata or an error (or D62's 15 s), then closes; its URL, which
// the session asked to revoke, goes after that. A viewer asked for meanwhile opens then: only the latest request,
// and only if its selection is still current (an automatic open waits for its own inline video again).
let held = null;
function closeViewer() {
const node = document.querySelector('#viewer-dialog video');
if (!node || !isStarting(node)) { viewer.close(); return; }
const waiting = new AbortController();
held = { url: node.getAttribute('src'), revoke: false, waiting, pending: null, timer: setTimeout(finishHeld, STARTING_VIDEO_MS) };
for (const type of ['loadedmetadata', 'error']) node.addEventListener(type, finishHeld, { once: true, signal: waiting.signal });
}
function endHeld() {
const entry = held;
if (!entry) return null;
held = null;
entry.waiting.abort(); clearTimeout(entry.timer);
return entry;
}
function finishHeld() {
const entry = endHeld();
if (!entry) return;
viewer.close();
// Through the session's hook, so an inline video still starting on the same URL keeps it until it settles.
if (entry.revoke) revokeObjectURL(entry.url);
const request = entry.pending;
if (!request || scope.signal.aborted || session.entry !== request.entry || session.state.phase !== 'ready') return;
if (request.auto) autoOpen(request.entry);
else openViewer(request.trigger);
}
function retire(preview, url) {
const waiting = new AbortController();
const entry = { preview, waiting, revoke: false, timer: setTimeout(() => finishRetiring(url), STARTING_VIDEO_MS) };
@@ -159,7 +129,6 @@ export function mountInspect(main, { parentScope, api, sender }) {
}
const finishAllRetiring = () => { for (const url of [...retiring.keys()]) finishRetiring(url); };
const revokeObjectURL = url => {
if (held?.url === url) { held.revoke = true; return; }
const entry = retiring.get(url);
if (entry) entry.revoke = true;
else URL.revokeObjectURL(url);
@@ -167,8 +136,7 @@ export function mountInspect(main, { parentScope, api, sender }) {
const session = createLocalSession({ parentScope: sessionScope, pageTarget: null, revokeObjectURL });
const viewer = createMediaViewer({ api, container: main, parentScope: scope, inspectionController: session.controller, sender,
adapter: localFileAdapter, onRestoreFocus: () => { if (open.isConnected) open.focus({ preventScroll: true }); },
// A held viewer closed by hand (Escape, Close) ends its hold at once: its video is gone.
onClose: () => { viewerKey = null; if (held) finishHeld(); } });
onClose: () => { viewerKey = null; } });
let viewerKey = null, media = null, mediaUrl = null, statusLine = null, rejectionLine = null, rejectionTimer = 0, announced = null, chosen = null;
function statusElement(message) {
@@ -241,7 +209,7 @@ export function mountInspect(main, { parentScope, api, sender }) {
scope.cleanup(session.subscribe(state => {
// D155: a new selection can start while the viewer shows the previous one; it closes first, so no element holds
// the old URL when the session revokes it (the new file's viewer opens once it is ready).
if (viewerKey !== null && (state.phase !== 'ready' || session.entry?.key !== viewerKey)) { closeViewer(); viewerKey = null; }
if (viewerKey !== null && (state.phase !== 'ready' || session.entry?.key !== viewerKey)) { viewer.close(); viewerKey = null; }
showMedia(state);
showStatus(state.message);
const entry = session.entry;
@@ -252,14 +220,17 @@ export function mountInspect(main, { parentScope, api, sender }) {
// while this selection is still the current one and no other dialog is open (the viewer is replaced). D62:
// a video opens it once the inline preview has settled, so a replacement while it starts never tears down a
// second starting video in the viewer (that deadlocked WebKit).
if (AUTO_OPEN.has(state.message)) autoOpen(entry);
if (AUTO_OPEN.has(state.message)) whenSettled(media, () => {
if (scope.signal.aborted || session.entry !== entry || session.state.phase !== 'ready' || document.querySelector('dialog[open]:not(#viewer-dialog)')) return;
openViewer(input);
});
// An unreadable or failed preview says so in its alert; the toast never claims metadata is available.
else if (TOASTABLE.has(state.message)) showWorkspaceToast(INSPECT_COPY.toast);
}
}));
// A replacement or page leave revokes the URL after this subscriber dropped the media; a dispose closes the
// viewer (on abort), drops the media, then ends the session.
scope.cleanup(() => { const entry = endHeld(); viewer.dispose(); if (entry?.revoke) revokeObjectURL(entry.url); });
scope.cleanup(() => viewer.dispose());
scope.cleanup(() => { releaseMedia(media); media = null; mediaUrl = null; });
scope.cleanup(finishAllRetiring);
scope.cleanup(() => sessionScope.dispose());
@@ -268,11 +239,9 @@ export function mountInspect(main, { parentScope, api, sender }) {
// cleared, so a page restored from the back-forward cache never names a file beside the empty selection.
function leave() {
clearRejection();
const entry = endHeld();
viewer.close();
releaseMedia(media); media = null; mediaUrl = null;
finishAllRetiring();
if (entry?.revoke) revokeObjectURL(entry.url);
session.clear();
chosen = null; input.value = '';
}
@@ -318,20 +287,10 @@ export function mountInspect(main, { parentScope, api, sender }) {
for (const type of ['loadedmetadata', 'error']) preview.node.addEventListener(type, done, { once: true, signal: settled.signal });
scope.cleanup(done);
}
/** D155: the viewer opens on `entry` once its inline preview has settled, while it is still the current selection. */
function autoOpen(entry) {
whenSettled(media, () => {
if (scope.signal.aborted || session.entry !== entry || session.state.phase !== 'ready' || document.querySelector('dialog[open]:not(#viewer-dialog)')) return;
// #483: the previous viewer is still closing around its starting video; this one opens once it has.
if (held) { held.pending = { entry, auto: true }; return; }
openViewer(input);
});
}
/** The viewer on the current selection, details showing (D155 amends D97); focus returns to `trigger` on close. */
function openViewer(trigger) {
const entry = session.entry;
if (!entry || session.state.phase !== 'ready') return;
if (held) { held.pending = { entry, trigger, auto: false }; return; }
viewer.open({ items: [entry], selectedKey: entry.key, trigger, details: 'shown' });
viewerKey = entry.key;
}
+2 -46
View File
@@ -14,9 +14,6 @@
* it to the host draft of `gallery_root_paths`; Remove takes one out. Both wait for the one Save row.
* - "Watch for new outputs" (`monitoring_enabled`) and "Worker threads" (`worker_threads`, a range up to the server's
* `cpu_count`, D113) are host drafts too. "Enrich existing metadata" reviews a scan of every source.
* - D158 (#247): "Save prompts from other workflows" (`save_foreign_runs`), a host draft below "Watch for new outputs",
* applied by the server at once (the monitor reads it for every file); disabled, saying why, while monitoring is off,
* unless the running monitor still watches until a restart (D128 `stopping`): then it can still be set.
* - "Thumbnail generation → Review options" opens "Thumbnail maintenance" (thumbnail-review.mjs, S5 Task 14), which
* starts the thumbnail Activity job.
*
@@ -66,21 +63,6 @@ export const LOAD_FAILED_TEXT = 'Sources could not be loaded. Your saved data ha
export const DUPLICATE_TEXT = 'This folder is already a source.';
export const VALIDATE_FAILED_TEXT = 'The folder could not be checked. Try again.';
export const REFRESH_FAILED_TEXT = 'Files could not be refreshed. Try again.';
/** D158 (#247): the row the concept does not draw, below "Watch for new outputs". */
export const FOREIGN_RUNS = Object.freeze({ id: 'foreign-runs', key: 'save_foreign_runs', label: 'Save prompts from other workflows',
description: 'New outputs of workflows without a PromptManager node add their prompt to the library when it is certain.',
off: 'Applies while Watch for new outputs is on.' });
/**
* D158: the row for the drafted `save_foreign_runs` (`value`) and `monitoring_enabled` (`monitoring`): disabled until
* read, and while monitoring is off, when `note` says why (the value is kept and saved as it is). `stopping`: a source
* is still watched until the restart (D128), so the value can still be set; the note stays.
*/
export function foreignRunsState({ value, monitoring, stopping = false }) {
if (value === undefined) return { disabled: true, checked: null, note: null };
const off = monitoring === false;
return { disabled: off && !stopping, checked: value === true, note: off ? FOREIGN_RUNS.off : null };
}
// Labels of the sources added in this page session (validate-source answers them), for the rows not saved yet.
const addedLabels = new Map();
@@ -285,16 +267,13 @@ export function mountSources(container, { api, activity, scope, drafts, notify =
const monitoring = settingRow({ id: 'monitoring', label: 'Watch for new outputs',
description: 'New files enter the same library. Existing scans do not count as new runs.', control: { type: 'toggle' },
scope, onChange: value => drafts.set('host', 'monitoring_enabled', value) });
const foreign = settingRow({ id: FOREIGN_RUNS.id, label: FOREIGN_RUNS.label, description: FOREIGN_RUNS.description,
control: { type: 'toggle' }, scope, onChange: value => drafts.set('host', FOREIGN_RUNS.key, value) });
const workers = settingRow({ id: 'workers', label: 'Worker threads',
description: 'Balance background scans and thumbnail work against generation.', control: { type: 'range', value: 4 },
scope, onChange: value => drafts.set('host', 'worker_threads', value) });
const hostRows = [[monitoring, 'monitoring', 'monitoring_enabled'], [foreign, FOREIGN_RUNS.id, FOREIGN_RUNS.key],
[workers, 'workers', 'worker_threads']]
const hostRows = [[monitoring, 'monitoring', 'monitoring_enabled'], [workers, 'workers', 'worker_threads']]
.map(([row, id, key]) => ({ input: controlOf(row, id), key, text: row.children[0], note: null }));
for (const row of hostRows) row.input.dataset.key = row.key;
surface.append(monitoring, foreign, workers,
surface.append(monitoring, workers,
operationRow({ scope, title: 'Enrich existing metadata', description: 'Resolve both prompts, deduplicate recipes, preserve uncertain links.',
label: 'Review scan', onClick: scanAll }),
operationRow({ scope, title: 'Thumbnail generation', description: 'Generate missing previews; rebuilding all is a separate explicit option.',
@@ -369,10 +348,6 @@ export function mountSources(container, { api, activity, scope, drafts, notify =
for (const row of hostRows) {
const value = drafts.get('host', row.key);
row.input.disabled = value === undefined;
if (row.key === FOREIGN_RUNS.key) {
paintForeign(row, value, unavailable);
continue;
}
if (row.key === 'worker_threads') {
// D113: the range ends at the server's core count (the concept's fixed 16).
const cpus = drafts.get('host', 'cpu_count');
@@ -385,24 +360,6 @@ export function mountSources(container, { api, activity, scope, drafts, notify =
}
}
// D158: disabled while monitoring is drafted off, with the reason under the description.
function paintForeign(row, value, unavailable) {
const stopping = sources?.effective?.some(source => source.watch === 'stopping') === true;
const state = foreignRunsState({ value, monitoring: drafts.get('host', 'monitoring_enabled'), stopping });
row.input.disabled = state.disabled;
if (state.checked !== null) row.input.checked = state.checked;
const note = unavailable && value === undefined ? HOST_UNAVAILABLE : state.note;
if (note && row.note?.textContent !== note) {
row.note?.remove();
row.note = el('p', note, 'small');
row.note.id = `${FOREIGN_RUNS.id}-note`;
row.text.append(row.note);
} else if (!note && row.note) { row.note.remove(); row.note = null; }
// The note is the switch's description, so a screen reader says why it is disabled.
if (row.note) row.input.setAttribute('aria-describedby', row.note.id);
else row.input.removeAttribute('aria-describedby');
}
// The sources changed since this page read them (a refused per-source scan): read them and the saved values again.
function reread() {
load();
@@ -421,7 +378,6 @@ export function mountSources(container, { api, activity, scope, drafts, notify =
if (answer === STALE || scope.signal.aborted) return;
sources = answer?.sources ?? null;
paintList();
paintControls();
}
// A save (or a reread after a conflict) changed the saved sources or monitoring: read their state again. The first