Compare commits

..
Author SHA1 Message Date
Vito Sansevero 157ad4df7b Merge remote-tracking branch 'origin/main' into feat/249-system-shell 2026-10-23 04:25:39 -07:00
Vito Sansevero a2b52c9cff fix(system): make two health tests portable to Windows
- Windows forbids "?" in file names, so the URI-character open test
  uses "#", "%" and a space there.
- ntpath.realpath on Python 3.11+ returns a NUL-byte path instead of
  raising, so the bad-path test raises as POSIX does to reach the
  missing-files failure path on every platform.
2026-10-23 03:54:20 -07:00
Vito Sansevero 2323aa0be5 fix(system): thumbnail row read, tracked health job and guarded host save
- The Overview's thumbnail row is its own read of the thumbnail status:
  "Checking thumbnails…" until it answers, unavailable when it fails.
- The health job is tracked in the Activity store (new store.track) and
  followed through its event stream instead of polling every 150 ms.
- A host save before a read returned a revision fails without a request;
  System reads the host values on mount and after a conflict.
- "100+ problems found" when the integrity count is capped.
- D-NEW-S2-5 is D123 everywhere; the unused Overview hand-off is gone.
- The concept spec builds the health report from the concept's own rows
  and asserts the full unmapped path list; surface tops pin the 18px
  offset.
- TESTING.md: D96's reason, the read-only (WAL/SHM) standard, the UNC URI
  and the new behavior. Coverage floors raised for the touched files.
2026-10-23 03:35:34 -07:00
Vito Sansevero 2ba0e6c34a fix(system): keep health reports per database and off the event loop
- A restore that replaces the database forgets the stored health report,
  and a check that started before the swap is not kept (database
  generation stamped at start).
- The Overview no longer walks every source for the missing-thumbnail
  count; the page reads /images/thumbnails/status on its own.
- Image sources for the health check and the Overview resolve in the
  executor; a failure to start the check answers the coded envelope.
- Any exception fails only its own check; missing files reads its rows in
  the snapshot and stats the files after it ended.
- The read-only URI keeps Windows UNC paths (empty authority).
- An integrity count at SQLite's limit is flagged as capped.
2026-10-23 03:35:33 -07:00
Vito Sansevero 3adc79512e Merge remote-tracking branch 'origin/main' into feat/249-system-shell 2026-10-23 02:42:54 -07:00
Vito Sansevero e68ab6ee98 test(system): coverage floors for the native Overview
Adds the system-overview.mjs baseline and raises the system.mjs floors to their measured coverage; no other floor changes.
2026-10-23 02:07:44 -07:00
Vito Sansevero 0fef050d94 Merge remote-tracking branch 'origin/main' into feat/249-system-shell 2026-10-23 01:52:15 -07:00
Vito Sansevero f43a222ffc feat(system): native Overview with health report and statistics
System > Overview renders the concept's metrics and its Library health and
Workspace preferences surfaces from GET /system/overview: real counts,
"Not checked" until a health check ran, no invented review count, backup
or thumbnail status (D93). Explore opens "Usage statistics" with the real
runs and ratings per category and the Library totals (D96); the
statistics route now also returns rated_images.

Maintenance shows the concept's notice with Run health checks above its
hand-off: the read-only check runs as a registry read job and opens the
"Health report" with a chip per real result, missing files per source
(D95). A load failure shows Retry and no zeros; late answers never
repaint a newer section.

Adds the System concept comparison (shell, Overview, both dialogs, the
notice, mutation checks), the System workflow spec and its fixture
hooks, and the TESTING.md "Native System" section.
2026-10-23 01:52:07 -07:00
Vito Sansevero a997fc0bcd feat(system): native System route, drafts and shared review dialog
- #/system/<section> renders the concept's System shell: section nav,
  settings layout and Save row with truthful status (D92); each section
  not delivered yet hands off to its legacy tools (D94). Overview's System
  card opens #/system/overview (Q12).
- A module-level draft store saves browser, then host (expected_revision),
  then runtime; failed domains stay dirty, a stale revision is a conflict.
- System row builders (setting, operation, section nav) on the scope.
- openReviewDialog and backupNotice extracted from the scan review, which
  is now a thin caller with identical markup.
- Overview images gain a "rated" count (images whose linked prompt is
  rated); the missing-files health check reports per source, so partial
  availability is visible.
2026-10-23 01:20:52 -07:00
Vito Sansevero 627113b9fb feat(system): read-only health check and truthful overview counts
POST /prompt_manager/system/health runs the health check off-loop as a
read registry job of kind "health" (one at a time, beside a scan, not
cancellable, no file outcomes) and keeps the last report; GET returns it
or null. The check reads one snapshot on its own mode=ro connection,
closed in finally: integrity_check(100), foreign_key_check, stored hashes
against utils.hashing, duplicate pair rows, the legacy consistency
queries (now shared with check_consistency), and image links whose file
is gone, counted only under readable sources. An unavailable source, or
the absent Needs-review queue, makes its check unavailable and the
report incomplete instead of producing a count.

GET /prompt_manager/system/overview returns Library-eligible positive
prompts and active negative variants, indexed images, the last health
summary or null, backup null, the Needs-review count null, the effective
sources with whether they were auto-detected, and the thumbnail plan's
missing count. GET /prompt_manager/system/statistics lists runs and
average rating per category of Library parents, with the /stats totals.
2026-10-23 00:23:47 -07:00
116 changed files with 566 additions and 15681 deletions
+20 -455
View File
@@ -121,13 +121,6 @@ npm run --prefix tools/vendor test:comfyui
CI performs this setup automatically. `PM_COMFY_URL` can select another
`http://127.0.0.1:PORT/` address (default port 8189). This is an owned managed
server: a running host is never reused. Do not point it at your live checkout.
One port serves one run. The launcher refuses a port that is already in use and
reports ready only when `/system_stats` names its own base directory, so two runs
started together on 8189 fail at startup ("already in use" or "answered by a
ComfyUI host this run did not start") instead of one run testing the other's
host and losing it mid-test (`ECONNREFUSED`, `ERR_CONNECTION_REFUSED`, Network
Error). Every test checks the answering host again. Agents sharing a machine
give each run its own `PM_COMFY_URL` port.
The launcher verifies the upstream SHA and clean tracked files, snapshots only
tracked PromptManager files, and creates temporary user/input/output/database/
configuration/log directories. Personal untracked files are excluded. It runs
@@ -392,8 +385,7 @@ with source `zeroed-conditioning`. `easy ifElse` with a literal boolean follows
branch only; with a linked boolean both branches are traced, equal texts resolve and
different texts make the sampler `ambiguous` with one candidate per choice (at most
32, else `work-limit`). ConditioningCombine/Concat/Average resolve only when every
part has the same text, otherwise `unresolved` with `composite` naming the node
(but see the added-conditioning floor below for a later stage).
part has the same text, otherwise `unresolved` with `composite` naming the node.
Results that depend on a forced choice are never memoized. A `FaceDetailer`
(`detailer-stage`) is a sampler stage: the samplers reached through its `image`
ancestry are compared with it. Equal texts resolve with the detailer as the sampler;
@@ -404,65 +396,9 @@ A sampler's latent ancestry is walked through every link input of non-sampler
nodes within the same visit and depth budgets: with no earlier sampler it is a
single stage whatever the origin (VAEEncode, EmptyLatentBatch, EmptyFlux2LatentImage,
EmptyHunyuanLatentVideo, SetLatentNoiseMask, InpaintModelConditioning or
WanImageToVideo slot 2). Pass-through lineage (#304 follow-up, maintainer option B,
2026-10-23): the table's `lineage` section lists the outputs whose image or latent
continues one input's (`VAEDecode`/`VAEDecodeTiled` `samples`, `VAEEncode`/
`VAEEncodeTiled` `pixels`, `ImageUpscaleWithModel`, `ImageScale`, `ImageScaleBy`,
`ImageScaleToTotalPixels` `image`, `LatentUpscale`/`LatentUpscaleBy` `samples`,
cg-image-filter `Image Filter` slot 0 `images` and slot 1 `latents`; a selection of
the same batch), plus Reroutes. Lineage describes no prompt, so its classes are not
claimed by a prompt category. When the latent walk finds exactly one earlier sampler
and the latent descends from that sampler's output slot 0 only through lineage
nodes (a hires-fix or refiner chain, img2img from an in-graph decode, a direct
latent link), the two stages are compared like a detailer and its upstream stage:
equal role texts resolve with the later sampler named, different texts are
`ambiguous` `multistage-differs` with one candidate per stage, and an unreadable
earlier stage makes the chain `unresolved` with that stage's reason. Unanchored, the
earlier sampler is judged inside the chain when its output reaches no other file
output. A size or resolution node reading the same image beside the lineage (a
`ResolutionCalculator` feeding `ImageScale` width and height) is allowed, since the
walk then still finds only that sampler. Any other earlier sampler, or one reached
only through an unlisted image node, stays `unresolved` "Multistage sampler ancestry
is ambiguous". Added-conditioning floor: a later stage whose own positive fails only
as `composite` keeps its earlier stages' shared pair ("A later stage adds
conditioning to the base prompt", naming the earlier stage's sampler) when that
positive reaches, through Reroutes and conditioning pass-throughs, a composite the
table marks `additive` (ConditioningCombine, ConditioningConcat; never
ConditioningAverage, which can weight a part away), one of its parts is exactly the
earlier stages' positive text, and its negative text equals theirs. This is the
three-stage Flux run whose UltimateSDUpscale combines the PromptManager prompt with
a style-trigger encoder: legacy linked the PromptManager text and every stage is
conditioned on it. The positive is always labelled (maintainer ruling): its source is
`added-conditioning` (D126; with a D125 omission too,
`runtime-parts-omitted-and-added-conditioning`) and its provenance names the
composite (`added_by`), whatever the added text is; no length heuristic. A combine without the base text, a different negative, or
different earlier stages stay as they were (`composite` or `multistage-differs`).
An image-edit batch (several samplers with distinct prompts, one shared saver
prefix) stays `ambiguous`, and text generated at run time stays untraceable
(`tests/test_resolver_pass_through.py` on synthetic shapes from
`tests/passthrough_graphs.py`; the browser port is compared on the named
`select-*` chain cases and `passthrough_fuzz_cases`). Ingestion keeps the default
profile's historical latent rule (D1), so the admin scan still reaches these files
through its legacy-parser fallback. Run-time PromptManager parts (maintainer option
A, 2026-10-23): LoraManager's `TriggerWord Toggle` is a `runtime` class (its output is
the LoRA's trigger words as filtered at run time). A PromptManager (or
PromptManagerPair) `prepend_text` or `append_text` linked directly to a `runtime`
node is left out: the text is the node's own, combined with any other literal or
traced part exactly as before, the side's source is `runtime-parts-omitted`, and a
resolved pair's reason is "Text added at run time, such as trigger words, is not
included". The prompt block shows that line under the text (D125; the evidence
badge stays "Workflow inferred"). The label follows the text through encoders,
concatenations and chained PromptManagers: a side is labelled when any node on its
traced path left a part out, and its provenance lists those PromptManager nodes
(`omitted_parts`). Comparisons include what a text leaves out, so a prompt that left
out a part never equals a complete one (a branch with the toggle and one without stay
ambiguous; such stages are `multistage-differs`). Only classes in the table's
`runtime` list are left out, never another `runtime-value` source (a JoinStringMulti
with a linked list flag stays `runtime-value`). The toggle's output is never computed. A run-time node further upstream (inside a
concatenation), a run-time main `text`, or an unknown feeder keep their outcome
(`runtime-value` or `unsupported-node`). This is what legacy linked for the June
hires-fix runs (`TestRunTimePromptManagerParts`, the `select-runtime-*` parity
cases); an unlisted node with sampler-shaped inputs (positive, negative, guider,
WanImageToVideo slot 2). An earlier sampler (img2img from a decode, hires fix) stays
`unresolved` "Multistage sampler ancestry is ambiguous" until stage texts are
compared; an unlisted node with sampler-shaped inputs (positive, negative, guider,
latent_image, a pipe, video embeds, custom-sampler parts, or a `model` link together
with conditioning, neg_conditioning, samples or latent) is `unsupported-node`
naming it, here and in a detailer's image ancestry; a latent that is not a link or
@@ -1898,86 +1834,8 @@ browser fields, then the host fields (`POST /prompt_manager/settings` with
highlights their controls; `settings-changed` is a conflict and keeps the draft. System
reads the host values on mount and again after a conflict; until a read returned a
revision, a host save fails without a request, so it never sends a blind
`expected_revision: null` (which S1 merges into the file on disk). Save stays disabled
until a section has an unsaved change.
Appearance (`system-appearance.mjs`, S3) is the concept's "Browsing & display" surface
on real values. Its browser rows are this browser's display defaults in the legacy
gallery's own `gallerySettings` object (`web/js/workspace/preferences.mjs`, Q3): Default
layout and Thumbnail size (`workspaceLayout` grid/list/masonry, `workspaceDensity`
comfortable/large/compact, new keys), Load metadata on selection (`autoLoadMetadata`),
Cache metadata in this browser (`cacheMetadata`), Image loading quality (`imageQuality`:
Optimized→`medium`, Original→`high`, Fast→`low`; the thumbnail dialog's preselected quality
once S5 lands), the three video rows (`videoAutoplay`, `videoMute`, `videoLoop`) and Show
file paths (`showFilePaths`). A write re-reads the stored object and writes back only the
changed keys, every other key (legacy paging, thumbnail state, unknown keys) as stored;
the legacy gallery's Save also keeps the keys it does not edit, so the two pages stay in
sync. The workspace never reads the legacy paging keys (`defaultLimit`, `gridColumns`,
`infiniteScroll`; README continuous-browsing ruling). A browser that refuses storage
(denied, or a full quota that leaves the stored bytes as they were) keeps the change for
this tab only and the status says "Saved for this tab only; browser storage is
unavailable.", never "Saved in this browser.". Another tab's change (a `storage` event)
repaints the clean controls; a draft it overtook stays, highlighted, with the conflict
status. Open from ComfyUI (New tab→`newtab`, Popup window→`popup`) and Node search
auto-hide (seconds) (a whole number 0–300; anything else is highlighted and never sent)
are the server's `webui_display_mode` and `result_timeout`, saved through
`POST /prompt_manager/settings` with the revision. The consumers read the defaults on
their next visit, never the current view: a new Images visit starts from the layout and
size (the last visit keeps its own, a saved view opens at Grid and Comfortable as
before); with metadata off a selection reads nothing until "Load metadata"; with the
cache off selecting the shown file again re-reads it (on, the one current record is
reused); paths off leaves out only the Path row of File & provenance; the viewer reads
the video defaults.
Sources & scans (`system-sources.mjs`, S4) is the concept's "sources" body on the
server's sources. `GET /prompt_manager/settings` reports them (public paths only):
`sources.configured` (each `gallery_root_paths` entry with `valid`, `code` and the
server's `reason`), `sources.effective` (what is scanned and served, with its
`root_index` and `watch`: `watching` only when the image monitor's observer is alive,
scheduled that folder and monitoring is on, `stopping` when it still watches it but
monitoring was switched off, `manual` when monitoring is off or no monitor runs, else
`restart_pending`), `sources.auto_detected`, `monitor` and `settings.monitoring_enabled`.
`POST /prompt_manager/settings/validate-source {path}` checks one folder and writes
nothing. Every caller (validate-source, the settings save, config.json at load) refuses
a source through the same checks (`sources.source_problem`): its form first, then
containment decided as text (`normcase(normpath)` against the allowed folders' own
spellings), before anything resolves, stats or opens the path, so a UNC share, the
`\\?\` and `\\.\` namespaces, a device or any folder outside the allowed ones never
reaches the filesystem or the network (no SMB request, no existence or timing oracle).
Only then is the path resolved and contained again, which catches a symlink out. The
filesystem root and the home directory answer `outside-allowed` with the same reason as
any outside path. ComfyUI folders outside its base directory (`--output-directory`)
are path anchors too, so their sources keep a relative public path
(`renders/portraits`) that round-trips and never collides by folder name. A save whose
list holds a source the server cannot use is a 400 `invalid-source` naming its public
path (`source`, as text: a refused path is never resolved to name it), its `index` and
`reason`, and saves nothing. A save that changes the sources (compared as folders, not
spellings) holds the registry as `records` ("Source change"), so it is refused with
409 `busy` while a scan runs. `POST /prompt_manager/jobs/scan {scope, roots, paths}`
scans only the given effective `root_index`es (strict ints, at most 64), each with the
public path the page read for it; an index past the list or one that now names another
folder is 409 `sources-changed` and starts nothing (the page toasts the server's
reason and reads the sources again); the legacy `/scan` is unchanged. The section
lists one row per source ("<path> · watching | watching until restart | manual scan |
restart pending" with "Scan changes" for a watched source, else "Scan folder"), which
opens the shared "Review metadata scan" for that one source ("1 output folder") and
starts the session job over it; the job opens in Activity, and the toast only says it
started. Enrich
existing metadata and the notice's Review scan review every source. The notice shows
until this browser dismissed the 3.2.0 update notice (`pm_update_notice_dismissed`,
the legacy admin page's key); Later dismisses it without scanning, and Review scan
dismisses it as the legacy Scan does. "+ Add source" asks validate-source, then adds
the folder to the host draft of `gallery_root_paths`; Remove takes one out; "Watch for
new outputs" (`monitoring_enabled`) and Worker threads (`worker_threads`) are host
drafts too, and Save writes them with the revision. A save the server refuses because a
scan runs toasts "Another job is running: <title>." with a link to that job (D57) and
keeps the drafts; a save refused for one source says the server's message as the status
and marks that row. After a save the source list is read again; a monitoring change
waits for a restart (the running observer keeps watching until then, and the status
says "Restart ComfyUI to apply: Watch for new outputs."). Thumbnail generation's
Review options opens the legacy gallery in a new tab (D94) until its review dialog (S5).
A list that cannot be read shows "Sources could not be loaded. Your saved data has not
changed." and Retry, never an empty list.
`expected_revision: null` (which S1 merges into the file on disk). No delivered section
edits a setting yet, so Save stays disabled until Appearance (S3).
Overview (`system-overview.mjs`) reads `GET /prompt_manager/system/overview` once per
visit through `latestOnly()` on the section scope; a late answer after the section left
@@ -2006,7 +1864,7 @@ the connection's `total_changes` is 0 at close and the data reads back unchanged
`-shm` bytes are not compared: every WAL reader rewrites its read marks there, so no
reader can keep it byte-identical (`test_health_check_never_writes`).
Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121–D128 assigned):
Recorded deviations (plan D60–D64 renumbered D92–D96; D121, D122 and D123 assigned):
- **D92** (shell): no "Sample configuration" chip. The Save status is truthful: "No
unsaved changes.", "Unsaved changes.", "Saved in this browser." / "Saved to the server
@@ -2019,10 +1877,8 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121
and active saved negatives and the indexed `generated_images` rows (Q2). "Database
integrity" is "Not checked" until a check ran this session, then "N issues" (checks
that found issues, even beside a check that could not run), "Passed" or "Incomplete".
The review row is "N ambiguous image links" (the Needs review queue's pending count,
`needs_review.pending`) with the concept's line; its Review metadata action is
`aria-disabled` (CSS under D93) until the Needs review page ships (#288). A count
that is not an integer reads "Ambiguous image links" / "Not available yet." (Q6).
"Ambiguous image links" reads "Not available yet." with its Review metadata action
`aria-disabled` (CSS under D93) until the Needs review queue reports a count (Q6).
"Backup status" reads "No retained backup record." until verified backups exist (S8).
The thumbnail row is "Checking thumbnails…" until its own read answers, then "No
missing thumbnails", "N missing thumbnails" (the pending count of
@@ -2066,92 +1922,6 @@ Recorded deviations (plan D60–D66 and D81 renumbered D92–D98 and D113; D121
the rest of Maintenance, above its D94 hand-off, as the native place that runs the
health check. A press shows "Checking…" (`aria-disabled`) and opens the report; a
failed check toasts "The health check could not finish. Nothing was changed."
- **D97** (Appearance): added rows for the legacy gallery settings the concept leaves
out, after its own rows: "Show image details on cards" (`showImageInfo`; off hides a
card's title and file line, Info and Tag stay at the end of the row), "Collapse viewer
details by default" (`sidebarCollapsedByDefault`; the viewer opens with
`#viewer-details` hidden and a "Show details"/"Hide details" toggle with
`aria-expanded`, the stage taking the whole width), and the operation "Reset display
defaults" ("Restore this browser’s display defaults, then save. Other settings are
kept.", button "Reset"), which drafts the gallery's defaults for the browser rows only.
The node search timeout shows the server's real value, not the sample 30. CSS under D97.
- **D98** (Sources): each configured source's row has a plain Remove after its scan
button; nothing changes until Save. A source added this session reads "<path> · not
saved yet" with Remove only until Save; a configured source the server cannot use
reads "<path> · <server reason>" with Remove only. A line under the list says when the
list is not the whole story: "No source is configured. After saving, ComfyUI’s output
folder is used." (the last one removed), "No source is configured; ComfyUI’s output
folder is used." and "No configured source is available; ComfyUI’s output folder is
used." (auto-detected rows, listed without Remove). "restart pending" is the third
watch state. The operation "Refresh files" ("Read every source again to find files
added or removed outside ComfyUI.", button "Refresh") closes the surface: it calls the
legacy gallery's `/gallery/rescan` and toasts "Files refreshed. N files in M output
folders." or "Files could not be refreshed. Try again.". The concept's "sample path"
note is prototype copy and is dropped.
- **D113** (Sources): the Worker threads range ends at the server's `cpu_count` (the
concept's fixed `max=16`) and shows the server's real `worker_threads` (the concept's
sample 4).
- **D127** (Sources): the concept's `add-source` `simpleInput` with production
copy: the field is "Folder path" (the concept's "(sample only)" dropped), it starts
empty (not the demo "/images/new-collection"), and the action is "Add" (not "Apply in
prototype"); Enter in the field adds too. `add source dialog matches the concept`
(system-concept.spec) compares the open dialog with these three mapped, and `add
source comparison detects mutations` proves the mapping is the only difference. A
folder the server refuses keeps the dialog open with the field
`aria-invalid` and the server's reason in a `p.small[role=alert]` above the actions;
a folder the server names as one already listed says "This folder is already a
source.", and a check that cannot finish says "The folder could not be checked. Try
again."; neither adds anything.
The concept dialog does no check, so `invalid source shows the server reason`
(workspace-system-sources.spec) pins it.
- **D128** (Sources): states the concept's sample rows never show. A source the
running monitor still watches after "Watch for new outputs" was saved off reads
"<path> · watching until restart" (still "Scan changes"), and the save's status adds
"Restart ComfyUI to apply: Watch for new outputs." from `restart_required_fields`. A
configured source the effective list holds under another spelling reads "<path> ·
same folder as another source" with only Remove. A save refused for one source
(`invalid-source`) marks that row (`aria-invalid`, outlined like a D92 setting) and
the status is the server's "<path> can't be used: <reason>. Remove it or reconnect it,
then save." ("Remove it, then save." when reconnecting cannot help). Pinned by
`Watch for new outputs saves monitoring_enabled; a restart applies it` and `a save
refused for one unusable source names it and marks its row`
(workspace-system-sources.spec) and the system-state and system-sources node tests.
- **D124** (inspector): with "Load metadata on selection" off, the Images panel and
the viewer's details show "Metadata loads when you ask for it." and a "Load metadata"
button instead of reading the file. The concept has no such state, so no concept
comparison renders it: `metadata off shows an explicit load and loads once on request`
(workspace-system.spec) pins it instead.
- **D125** (prompt blocks, maintainer option A): a resolved prompt whose source is
`runtime-parts-omitted` (a PromptManager `prepend_text` or `append_text` fed directly
by a listed run-time node, such as LoraManager's TriggerWord Toggle, was left out)
shows the muted line "Text added at run time, such as trigger words, is not
included." under its text. The evidence badge stays "Workflow inferred"; the line is
never sent or copied. The concept has no such state (`a prompt that left out run-time
trigger words says so under its text (D125)` in `tests/js/prompt-blocks.test.js`).
- **D126** (prompt blocks, maintainer ruling on the added-conditioning floor): a
resolved positive whose source is `added-conditioning` (a later stage combined more
conditioning with the base prompt, as UltimateSDUpscale's ConditioningCombine of the
PromptManager and a style trigger) shows "Conditioning added at a later stage is not
included." the same way. A prompt with both shows both lines, D125 first (source
`runtime-parts-omitted-and-added-conditioning`). Always labelled, with no length or
"short trigger" heuristic (`a prompt a later stage added conditioning to says so, and
both lines show together (D126)`).
Appearance scope and states. The Default layout and Thumbnail size defaults start the
Images page only (a new visit, not the current view or a saved view); the embedded
galleries (a prompt history's Shared gallery and its All images / Repeat run dialogs)
have no toolbar, so they keep Grid/Comfortable, while the metadata, cache and card
defaults apply there too. A host row whose `/settings` read failed stays disabled with
"Server settings could not be read."; a rejected timeout says "Some changes were not
saved. Review the highlighted settings." A display default another page saved under one
of this tab's drafts flags that draft with the conflict status, whether it arrived while
System was mounted (a `storage` event) or while it was not (checked again on mount, and
before a write); the draft is never written over the other page's value without that
status first. A change kept for this tab only is stored by the next write storage accepts.
Open question (saved-views follow-up, M8): a saved view stores no layout or size and
always opens at Grid/Comfortable, even when the Default layout is Masonry. Should saved
views store a layout, or follow the default?
`tests/browser/system-concept.spec.mjs` compares the shell and Overview with the concept
(dark and light, 1600×1100 and 390×844), mapping only the deviations above; unmapped,
@@ -2164,50 +1934,21 @@ source when there are several, replaces only the chips with the fixture's expect
results, and asserts the full unmapped path list; it maps D96 and compares the
Maintenance notice unchanged. `system comparison detects mutations` removes a row,
renames a metric label, swaps the `--line` token, recolours a chip, renames a check,
removes the totals table and renames a caption. `appearance matches the concept` maps
only D97 and the real timeout (unmapped, exactly those paths differ), and `appearance
comparison detects mutations` removes a row, renames a label and an option and swaps
the `--line` token. `sources matches the concept with|without the notice` (both
themes and sizes; the fixture's monitor watches the first source, as the concept's
first sample is watched, and the concept's Later removes its notice) maps the real
sources for the two samples, D98's Remove and Refresh files, and D113; unmapped,
exactly those paths differ. `sources comparison detects mutations` removes the notice
and a source row, renames a label and the Add source button and swaps the `--line`
token. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`,
`{concept,production}-appearance-{dark,light}-{1600,390}.png`,
`{concept,production}-sources-{with,without}-notice-{dark,light}-{1600,390}.png`) are test artifacts; there
removes the totals table and renames a caption. Paired captures (`{concept,production}-system-{dark,light}-{1600,390}.png`,
`{concept,production}-{statistics,health}-{dark,light}.png`) are test artifacts; there
are no PNG baselines. `tests/browser/workspace-system.spec.mjs` covers the real counts,
the health check and metric 4, a failed check, the thumbnail row against the status
endpoint (painted after the rest when its read is held, unavailable when it fails), the review row (Q6), the section links, the Statistics dialog against the
route, a load failure with Retry and late answers after leaving, and Appearance: new
visits versus the current view and a saved view, the embedded gallery keeping Grid,
metadata off and on request, the cache (counted up to a request barrier), hidden paths,
the video defaults, the host fields with their revision, a stale one and a failed read,
reload and a second tab, a second tab's save while System was closed, storage refused,
the D97 rows and reset. `tests/browser/workspace-system-sources.spec.mjs` covers the
notice's Later and Review scan, a scoped scan from one source into Activity, Add source
with its server check and Save, a refused folder, Remove down to auto-detect, an absent
configured source, the three watch states, the monitoring and worker rows, Refresh
files, the thumbnail hand-off, a save refused by a running scan and a failed read. Its hooks are
route, a load failure with Retry and late answers after leaving. Its hooks are
`tests/browser/workspace_system_fixture.py` (`/__test__/system/*`: reset, injected
failures, an absent source, a deleted linked file, unthumbnailed files, a fake image
monitor) on the
concept-shaped fixture, with delays through the shared request holds; reset also restores
the server's host settings and its `config.json`, which must be the one inside the server's
temp root (`tests/test_workspace_system_fixture.py`).
failures, an absent source, a deleted linked file, unthumbnailed files) on the
concept-shaped fixture, with delays through the shared request holds.
```sh
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library \
tests.test_workspace_system_fixture -v
python3 -m unittest tests.test_database_health tests.test_system_api tests.test_library -v
node --test tests/js/workspace-system-state.test.js tests/js/workspace-system-controls.test.js \
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js \
tests/js/workspace-system-appearance.test.js tests/js/workspace-preferences.test.js \
tests/js/workspace-inspection-state.test.js tests/js/workspace-images-state.test.js \
tests/js/workspace-system-sources.test.js
python3 -m unittest tests.test_settings_sources tests.test_scan_scope -v
tests/js/workspace-system-overview.test.js tests/js/workspace-state.test.js
npm run --prefix tools/vendor test:workflows -- system-concept.spec.mjs workspace-system.spec.mjs \
workspace-system-sources.spec.mjs \
--project chromium --project touch --project firefox --project webkit
```
@@ -2252,24 +1993,8 @@ and skip the parse.
integers past 2**53 as their exact digits, so the card and the inspector read
the same digits. The parse runs after the file handle closes. A record over
2 MiB (UTF-8), a parser error, a recursion error, a placeholder, or a
non-integral, boolean or non-finite seed gives null. When the prompt graph's
sampler stages read different seed inputs and the inspector's resolution names
one sampler for a resolved pair (a detailer or pass-through chain, whose facts
are the named stage's, D19), the card compares that stage's seed (read by the
same parser, the other samplers hidden from it) with its own and shows no seed
only when they differ. A seed that agrees, or one the named stage does not make
known, is kept, as on main (a saved sampler beside a preview-only one keeps its
seed; an ambiguous chain, whose facts merge and fill from the parser, keeps it).
Only such graphs are resolved for the card, with `INSPECTION_PROFILE`
(`test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed`,
`test_a_seed_that_agrees_with_the_named_stage_is_kept`,
`test_a_named_ambiguous_chain_keeps_the_parser_seed`; a literal named seed needs no
second parse, and stages sharing one linked seed skip the check. Measured on a
200-card page of about 54 KB graphs, half multi-stage with per-stage seeds: 618 ms
cold against main's 387 ms, about 1.2 ms more per such card, paid once per file
through the header-facts cache below;
before this rule a KSampler then FaceDetailer chain with two seeds broke the
contract below). A card never shows a seed that differs from the inspector's
non-integral, boolean or non-finite seed gives null. A card never shows a seed
that differs from the inspector's
(`test_card_seed_equals_inspector_seed_when_both_present`, which `JSON.parse`s
the listing in Node and compares `String(seed)` with the inspector's seed,
including one past 2**53). The card shows no seed for A1111 `parameters` files
@@ -2385,162 +2110,6 @@ inspection's own stable stat (`tests/test_api_workspace_images.py`
browser fixtures clear `media_image_tags`, `media_annotation_bytes` and
`media_annotation_files` on reset and never the metadata row.
Review queue schema (#249, `database/review_schema.py`,
`tests/test_review_schema.py`): the group `media_review_metadata` (one row:
schema version, `review_index_revision`), `media_review_items` (one ambiguous
resolver result per `(byte_id, resolver_version, evidence_hash)`, citing an
annotation byte record `ON DELETE RESTRICT`, with the file's recorded
`spelled_path` (the locator is case-folded on Windows, a listing target is
not), 0..16 candidates, `limited`,
`state` pending/selected/unresolved and a revision) and `media_review_decisions`
(one row per item decision revision; `selected` exactly when a
`candidate_hash` is present; `pair_id`/`image_id` navigation is set to NULL,
never cascaded, when the pair or image is deleted, through
`idx_media_review_decisions_pair`/`_image`). Evidence is canonical JSON
(`canonical_evidence`: one object, sorted keys, no whitespace, ASCII only, no
NaN/infinity, at most 8 MiB) and `evidence_hash` is its SHA-256. The group is
valid only beside a valid annotation group (a review group without annotation
tables is refused, catalog only, before the annotation install), installs like
the annotation group (optional in old backups, required once current under
`require_library_state`, SAVEPOINT install validated before and after), and
validation re-canonicalizes every evidence document in one streamed pass. A
startup runs that pass once (`test_startup_validates_stored_evidence_once`; the
install trusts the annotation group it has just validated, and the SQL row
predicates check type and size only, because the streamed pass proves the
rest): 2.5 s to open a database of 40,000 items with 4 KiB of evidence each,
was 7.7 s with two passes. Restore and `verify_database_file` keep the full
validation, annotation group included. The browser fixtures clear
`media_review_decisions` and `media_review_items` before the annotation tables,
never the metadata row.
Needs review items (#249, `utils/review_capture.py`, `database/review.py`,
`tests/test_review.py`, `tests/test_review_outcome.py`,
`tests/test_review_choosable.py`): a file is queued only when its final outcome
is ambiguous and unlinked and there is something to choose. Only scans produce
items; the monitor links new outputs and clears pending items it resolves
(maintainer rulings, 2026-10-23). The output scan (Activity's
Scan for metadata and the legacy `/scan`) hands `record_ambiguous_file` a file
the ingestion resolver finds `ambiguous` only when the browser-parser fallback
then links nothing (status `ambiguous`) and no saved row links the file
already; an ambiguous file a saved row links is reported `unchanged` ("Already
linked to its prompt"), not Needs review. A file the fallback links, an exact
pair the ingester stores, and `conflict` and `unresolved` files are not
queued. Neither is an ambiguous file with nothing to choose: fewer than two of
its candidates are `resolved` with non-blank positive text (`reviewable`). With
none the file could not be traced; with exactly one, the untraced branches may
still have made the file, so that text is not certain either. Both report
`unresolved` ("could not be traced"), are never linked, and their pending,
undecided items whose own evidence offers no choice (a limited item counts as
a choice when it holds at least two candidates) leave the queue on the next
scan or monitor pass (`settle_untraced_review_items`). The maintainer's
second live check found 39 items whose 11 candidates were all "unresolved". The queue heals itself: after each batch, the scan removes the pending
items of every listed file (read or skipped as known) that a saved row now
links, and the monitor does the same for each ambiguous output it links
(`settle_linked_review_items`; the review index revision moves). Only an
undecided item goes (`pending`, revision 0, no decision row): a decision is
never deleted or changed, so no schema state was added. The monitor never
queues an output it linked; an unlinked file passed to `record_review_file`
follows the scan's rule (a file the fallback would link is `skipped`). The
maintainer's live check of #284/#288 found the queue at 10,232 against a review
group of 4,677: every fallback-linked ambiguous file had been queued. It captures
the file (scope `needs-review`), reads its metadata again from the capture,
resolves it with the default profile and stores only an ambiguous result,
bound to the capture's byte record (created or refreshed like an observe) and
fenced by the capture's storage epoch. Evidence is the resolver reason plus one
`{sampler_id, state, reason, positive, negative, hash}` per selectable
candidate (nested ambiguous stages flattened, duplicates dropped; each side
`{text, storage_text, provenance}`; `hash` = SHA-256 of the canonical candidate
without it). More than 16 candidates, or evidence over 8 MiB, is a `limited`
item without candidates. Recording is idempotent per `(byte_id,
resolver_version, evidence_hash)` and never touches an existing item or its
decisions; changed bytes add a new pending item and keep the old one. The
capture is released before the write; no model is loaded and no pixel is
decoded. Scan statuses, counters, outcome rows and the legacy SSE events are
unchanged apart from the already-linked and nothing-to-choose cases above; the job's `scope` gains
`review_items` (this scan's files that are in the pending queue after it, new
or already there: the review group's ambiguous rows, less files a capture could
not read or a decision already settled), `review_added` (new items),
`review_settled` (items removed because their file is linked now),
`review_untraced` (items removed because they offer nothing to choose) and
`review_errors`. The log ends with `? N images added to Needs review`,
`? N images already in Needs review`, `= N images left Needs review · now
linked`, `= N images left Needs review · nothing to choose` and `! N images could not be added to Needs review` (failures, e.g.
capture capacity), each only when N > 0. Recording captures in the background: it never takes the store's
last worker or capture slot, which stay free for an interactive Tag dialog or
Apply, and waits up to 2 s for room before counting the file as not added
(`test_background_recording_leaves_a_worker_for_the_tag_dialog`). The monitor's
pass runs after its own link, best effort: a failure is logged and the link
stays. Recording follows the registry's `records`
admission (#289): the scan's recording runs under its own `records` job, and
the monitor's (one file, not a job) is refused while a `records` or `guarded`
job or hold runs. It then records nothing, logs that the file waits for the
next scan, and answers `deferred`; the next "new" scan reads the ambiguous file
again (`test_monitor_recording_is_refused_beside_record_changing_jobs`). A new item moves `review_index_revision`. A
rescan does not copy or hash an ambiguous file whose current size and
`mtime_ns` equal the verified fingerprint of bytes an item already cites (a
stat compare); the file still stays in the default "new" scope, because it is
not a known image. The skip and a duplicate insert record the current spelling
of the locator on its items (`refresh_review_spelling`, the review index
revision moves, item revisions do not), so a case-only rename on Windows or
macOS keeps its item applicable. Residual risks of the skip, like S1's: an
in-place rewrite that keeps both size and `mtime_ns` is never re-queued (Apply
still re-hashes, so it cannot link those bytes), and a file that is deleted or
moved leaves its pending item counted in `total` (its target then formats as
`unavailable` or fails to open, and Leave unresolved clears it).
Needs review decisions and queue reads (#249, `py/api/review.py`,
`database/review.py`, `database/review_decisions.py`,
`tests/test_api_review.py`): `GET /prompt_manager/review/items?limit=1..40&cursor=`
answers `{success, library_epoch, total, items, next_cursor}` with pending items
oldest first; `total` is the exact pending count. The cursor is an HMAC keyset
on `(created_at, id)` bound to the storage epoch and `review_index_revision`: a
malformed one is 400, a well-formed one this process cannot verify (tampered,
or opened before a ComfyUI restart: the secret is per process) is 409
`review-conflict` with the current `total`, one from before a restore is 409
`storage-changed`, one from before any queue change (a decision, a new item, a
refreshed spelling) is 409 `review-conflict`.
Targets are formatted from `spelled_path` against the current roots (the
roots' real paths are the only file access); a file under no current root is
`unavailable` with a null target. Every read and decision lists the roots and
resolves real paths on a worker thread, never on the event loop. `GET .../items/{id}` adds `candidates`
(`hash`, `sampler_id`, `state` resolved/unresolved, `reason`, `label` "<positive
title> / <negative title>" from the saved prompt's Library title else the
first line, and the role texts), `dependency_fingerprint` (SHA-256 of the
file's legacy rows under every spelling of the same physical file, positive
hashes and pair links with their negative hashes, as the ingester reads them) and `sha256`; a detail over 8 MiB, or a
limited item, has no candidates. `POST .../items/{id}/decision
{expected_epoch, expected_revision, capture_token, sha256, dependency_fingerprint,
choice}` with a candidate hash (Apply) requires a capture of the item's file
whose bytes are the item's, re-hashes the current file (`file-changed`
otherwise, also for a fresh capture of new bytes), re-reads the capture and
requires that its bytes are still ambiguous and still offer the chosen
candidate (`review-conflict` otherwise; a resolver change that rewords the
reason or reorders candidates leaves the item applicable, and
`RESOLVER_VERSION` in `utils/prompt_pair_resolver.py` is the one version
ingestion provenance and review share), then in one `_pair_transaction`
checks the epoch, the item revision and state (`review-conflict` with the
current item for a repeat), the re-hashed digest, the dependency fingerprint
and the public ingester's compatible-only rules through
`_ingest_resolved_pair_in` (any refusal is 409 `association-conflict` with
zero writes, explained inline; Apply never reparents a link). Physical
identity: the roots are real paths while the monitor links under the
directory it watches, so a symlinked or junctioned root, `subst` or a mapped
drive spells the same file otherwise. Every legacy row whose spelling ends with
the file's name (the link summaries' unindexed suffix lookup) is resolved by
`normcase(realpath)` against the locator, else `samefile`; two rows of the
file refuse the choice, one is enriched under the spelling it was linked
with, and a same-name row the request did not classify is a changed
association (`TestPhysicalIdentity`, symlinked root and a portable fake
junction), and writes
the link with `kind:"user-selection"` provenance (decision id, reviewed digest,
candidate hash, resolver version, role provenance), the decision row, the
item's new state and revision and `review_index_revision`. `"unresolved"`
records the decision only and accepts `capture_token: null`. A successful
decision releases its capture. No decision changes run or usage counts or
calls an execution writer; a Tag save never changes a review item. Errors carry
fixed sentences: `review-failed` / `decision-failed` 500s never echo exception
text or paths.
Capture store (#249, `utils/media_capture.py`, `tests/test_media_capture.py`):
one explicit action captures one rooted media file into an owner-only
`prompt-manager-captures-*` temp directory, streaming it in 1 MiB chunks while
@@ -2557,12 +2126,8 @@ directories are `CaptureDenied`; the locator is
`normcase(join(realpath(root), relative))`, so a symlinked root shares the
listing's identity. Fixed limits: 256 MiB per file (stat and stream), 8
completed captures, 512 MiB reserved, 2 capture/verify workers, 900 s idle,
1800 s hard; interactive admission is nonblocking (`CaptureCapacity`), a
`background=True` capture (Needs review recording) never takes the last worker
or capture slot and waits up to `BACKGROUND_WAIT_SECONDS` (2 s) for room, and
lifetimes are tested with an injected clock. The host's on_cleanup close is
final: a recording still in flight gets a capture error instead of creating a
new temp directory after it. An acquired capture outlives expiry and release
1800 s hard; admission is nonblocking (`CaptureCapacity`) and lifetimes are
tested with an injected clock. An acquired capture outlives expiry and release
until its last use ends. Header-only dimensions (4096 px per side,
16,777,216 px) and video mark Generate unsupported; `verify` re-hashes the
current file without a copy. Every handle closes before unlink (a patched
-15
View File
@@ -122,21 +122,6 @@ try:
_global_prompt_tracker = get_prompt_tracker(_global_db)
_global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
# The Needs review pass for ambiguous new outputs runs through the API's
# capture store (the one the scan and the Tag dialog use): a linked output
# is never queued and its pending items leave the queue. It stops before
# that store is closed at shutdown.
_review_api = globals().get("api")
if _review_api is not None:
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
_review_host = getattr(config.server_instance, "app", None)
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
async def _stop_review_recording(app, monitor=_global_image_monitor):
monitor.set_review_recorder(None)
_review_host.on_cleanup.insert(0, _stop_review_recording)
# Start monitoring
_global_image_monitor.start_monitoring()
_init_logger.info("Global image monitoring system started successfully")
+2 -5
View File
@@ -637,8 +637,7 @@ class LibraryOperations:
pairs (every pair's positive is eligible). ``images`` counts indexed
``generated_images`` rows, not files on disk. An image has no rating
of its own (it shows its linked prompt's), so ``rated_images`` counts
the indexed images whose linked prompt is rated. ``review_pending`` is
the Needs review queue's pending total.
the indexed images whose linked prompt is rated.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
row = conn.execute(
@@ -646,15 +645,13 @@ class LibraryOperations:
"(SELECT COUNT(*) FROM prompt_pairs WHERE archived=0),"
"(SELECT COUNT(*) FROM generated_images),"
"(SELECT COUNT(*) FROM generated_images gi JOIN prompts r "
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL),"
"(SELECT COUNT(*) FROM media_review_items WHERE state='pending')"
"ON r.id=gi.prompt_id WHERE r.rating IS NOT NULL)"
).fetchone()
return {
"positive_prompts": row[0],
"negative_variants": row[1],
"images": row[2],
"rated_images": row[3],
"review_pending": row[4],
}
def usage_by_category(self) -> list[dict]:
+3 -20
View File
@@ -18,11 +18,6 @@ from typing import Optional, Set, Tuple
from .library_schema import library_schema_error, migrate_library_state
from .image_favorites import image_favorites_schema_error, migrate_image_favorites
from .annotation_schema import annotation_schema_error, migrate_annotation_state
from .review_schema import (
migrate_review_state,
review_schema_error,
review_without_annotations_error,
)
from .saved_views import migrate_saved_views, saved_views_schema_error
from .execution_history_schema import (
history_schema_error,
@@ -308,14 +303,7 @@ class PromptModel:
migrate_library_state(conn)
migrate_image_favorites(conn)
migrate_saved_views(conn)
# A review group without its annotation group is refused before the
# annotation install could make it look valid (catalog only; the rows
# are validated once, in migrate_review_state).
review_error = review_without_annotations_error(conn)
if review_error:
raise sqlite3.DatabaseError(review_error)
migrate_annotation_state(conn)
migrate_review_state(conn)
def _migrate_prompt_pairs(self, conn: sqlite3.Connection) -> None:
conn.execute("SAVEPOINT prompt_pair_schema")
@@ -1035,9 +1023,9 @@ class PromptModel:
Args:
path: File to inspect
require_library_state: Also require the tables every current
database has (Library state, image favorites, saved views,
image annotations and the review queue), as a migrated
restore stage must; old backups may lack them.
database has (Library state, image favorites, saved views
and image annotations), as a migrated restore stage must;
old backups may lack them.
Returns:
(ok, reason): reason is "ok" when the file is acceptable, otherwise
@@ -1107,11 +1095,6 @@ class PromptModel:
)
if annotation_error:
return False, annotation_error
review_error = review_schema_error(
conn, allow_missing=not require_library_state
)
if review_error:
return False, review_error
return True, "ok"
except sqlite3.Error as e:
return False, f"Invalid SQLite database: {e}"
+5 -12
View File
@@ -4,7 +4,6 @@ Database operations for KikoTextEncode prompt storage and retrieval.
import sqlite3
import json
import math
import datetime
import os
from typing import Optional, List, Dict, Any, Set, Tuple
@@ -14,8 +13,6 @@ from .pairs import PairDependencyError, PairOperations
from .library import LibraryOperations, titled
from .image_favorites import ImageFavoriteOperations
from .annotations import AnnotationOperations
from .review import ReviewOperations
from .review_decisions import ReviewDecisionOperations
from .saved_views import SavedViewOperations
from .library_commands import LibraryCommandOperations
from .vocabulary import VocabularyOperations
@@ -29,7 +26,6 @@ from .health import HealthOperations, iter_consistency_issues
try:
from ..utils.logging_config import get_logger
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
except ImportError:
import sys
@@ -37,7 +33,6 @@ except ImportError:
sys.path.insert(0, current_dir)
from utils.logging_config import get_logger
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
# Subquery to fetch tags from junction table, embedded in SELECT statements
@@ -347,8 +342,6 @@ class PromptDatabase(
HistoryReadOperations,
ImageFavoriteOperations,
AnnotationOperations,
ReviewOperations,
ReviewDecisionOperations,
SavedViewOperations,
HealthOperations,
):
@@ -1704,9 +1697,9 @@ class PromptDatabase(
width,
height,
image_format,
dumps_strict(metadata.get("workflow", {})),
dumps_strict(metadata.get("prompt", {})),
dumps_strict(metadata.get("parameters", {})),
json.dumps(metadata.get("workflow", {})),
json.dumps(metadata.get("prompt", {})),
json.dumps(metadata.get("parameters", {})),
),
)
conn.commit()
@@ -1911,13 +1904,13 @@ class PromptDatabase(
obj: The object to clean (dict, list, or scalar)
Returns:
Cleaned object with NaN and Infinity values replaced by None
Cleaned object with NaN values replaced by None
"""
if isinstance(obj, dict):
return {key: self._clean_nan_values(value) for key, value in obj.items()}
elif isinstance(obj, list):
return [self._clean_nan_values(item) for item in obj]
elif isinstance(obj, float) and not math.isfinite(obj):
elif isinstance(obj, float) and str(obj) == "nan":
return None
else:
return obj
+111 -139
View File
@@ -1,5 +1,6 @@
"""Atomic metadata enrichment for a single known generation context."""
import json
import os
from typing import Any, Dict
@@ -8,12 +9,8 @@ from .operations import _coerce_file_info
try:
from ..utils.hashing import generate_prompt_hash
from ..utils.json_strict import dumps_strict
from ..utils.prompt_pair_resolver import RESOLVER_VERSION
except ImportError:
from utils.hashing import generate_prompt_hash
from utils.json_strict import dumps_strict
from utils.prompt_pair_resolver import RESOLVER_VERSION
def ingest_resolved_pair(
@@ -22,143 +19,118 @@ def ingest_resolved_pair(
state = resolution.get("state", "unresolved")
if state != "resolved":
return {"status": state, "pairs_added": 0, "links_added": 0}
with db._pair_transaction() as conn:
return _ingest_resolved_pair_in(conn, db, image_path, metadata, resolution)
def _conflict() -> Dict[str, Any]:
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
def _ingest_resolved_pair_in(
conn: Any,
db: Any,
image_path: str,
metadata: Dict[str, Any],
resolution: Dict[str, Any],
*,
provenance: Dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""``ingest_resolved_pair`` inside the caller's open ``_pair_transaction``.
The compatible-only rules are the public ingester's: an image with more
than one exact row, another positive, or a prior pair with another
negative is a ``conflict`` and nothing is written. The caller owns the
transaction, so its own later failure rolls the link back too.
``provenance`` replaces the link's default resolver provenance (Needs
review records the user's selection there).
"""
positive, negative = resolution["positive"], resolution["negative"]
positive_text = positive.get("storage_text", positive["text"])
negative_text = negative.get("storage_text", negative["text"])
normalized = normalize_image_path(image_path)
if provenance is None:
provenance = {
"resolver_version": RESOLVER_VERSION,
"positive": positive,
"negative": negative,
}
encoded = dumps_strict(provenance, sort_keys=True, separators=(",", ":"))
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return _conflict()
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return _conflict()
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute("SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
# ComfyUI's NaN/Infinity are stored as null: the snapshot is strict JSON.
snapshots = {
column: dumps_strict(metadata[key], sort_keys=True)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update({key: value for key, value in info.items() if value is not None})
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
provenance = {"resolver_version": 1, "positive": positive, "negative": negative}
encoded = json.dumps(
provenance, sort_keys=True, separators=(",", ":"), allow_nan=False
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
with db._pair_transaction() as conn:
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute(
"SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
snapshots = {
column: json.dumps(metadata[key], sort_keys=True, allow_nan=False)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update(
{key: value for key, value in info.items() if value is not None}
)
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
-541
View File
@@ -1,541 +0,0 @@
"""Needs review items (#249): persist and read capture-bound ambiguous evidence.
An item is one ambiguous resolver result for one annotation byte record (the
whole-file SHA-256 of one physical file at one locator), keyed by
``(byte_id, resolver_version, evidence_hash)``. Recording is idempotent: the
same evidence for the same bytes is a duplicate and never touches an existing
item, its state, revision or decisions. Changed bytes are a new byte record
and therefore a new pending item; the old one stays.
Callers pass verified facts (the capture's digest, size and ``mtime_ns``);
nothing here reads the filesystem. Writes run in one ``_pair_transaction``
fenced by the storage epoch the capture was taken under.
"""
import hashlib
import json
import os
import re
import sqlite3
from typing import Iterator
try:
from ..utils.hashing import generate_prompt_hash
except ImportError:
from utils.hashing import generate_prompt_hash
from .annotation_schema import (
BYTES_TABLE,
FILES_TABLE,
METADATA_TABLE as ANNOTATION_METADATA_TABLE,
)
from .annotations import _new_id, _validate_locator, _validate_target
from .models import normalize_image_path, utc_now_iso
from .review_schema import (
DECISIONS_TABLE,
ITEMS_TABLE,
MAX_CANDIDATES,
MAX_SPELLED_PATH_BYTES,
METADATA_TABLE,
canonical_evidence,
evidence_digest,
evidence_offers_choice,
)
# Only this module's table constants are interpolated into the SQL below.
_INSERT_ITEM = (
f"INSERT OR IGNORE INTO {ITEMS_TABLE}(id,byte_id,spelled_path," # nosec B608
"resolver_version,evidence_hash,evidence,candidate_count,limited,state,"
"revision,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?,'pending',0,?,?)"
)
_BUMP_REVIEW_INDEX = (
f"UPDATE {METADATA_TABLE} SET " # nosec B608
"review_index_revision=review_index_revision+1 WHERE singleton_id=1"
)
# A later spelling of the same locator (a case-only rename on a case-folding
# filesystem) replaces the recorded one, so targets keep formatting.
_REFRESH_SPELLING = (
f"UPDATE {ITEMS_TABLE} SET spelled_path=? " # nosec B608
f"WHERE spelled_path<>? AND byte_id IN (SELECT b.id FROM {BYTES_TABLE} b "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE f.locator=?)"
)
_REVIEW_INDEX = (
f"SELECT review_index_revision FROM {METADATA_TABLE} " # nosec B608
"WHERE singleton_id=1"
)
_BUMP_IMAGE_INDEX = (
f"UPDATE {ANNOTATION_METADATA_TABLE} SET " # nosec B608
"image_index_revision=image_index_revision+1 WHERE singleton_id=1"
)
_ITEMS_OF_LOCATOR = (
"SELECT i.id,i.byte_id,i.resolver_version,i.evidence_hash," # nosec B608
"i.evidence,i.candidate_count,i.limited,i.state,i.revision,i.created_at,"
f"i.updated_at,b.sha256 FROM {FILES_TABLE} f "
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? ORDER BY i.created_at,i.id"
)
_QUEUED = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"LIMIT 1"
)
_PENDING_AT = (
f"SELECT 1 FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND b.size=? AND b.mtime_ns=? AND i.resolver_version=? "
"AND i.state='pending' LIMIT 1"
)
# Pending items of one file: the candidates a link settles.
_PENDING_OF_LOCATOR = (
f"SELECT i.id FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Pending items of one file with their evidence: an untraced file's settle.
_PENDING_EVIDENCE_OF_LOCATOR = (
f"SELECT i.id,i.evidence FROM {FILES_TABLE} f " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.file_id=f.id "
f"JOIN {ITEMS_TABLE} i ON i.byte_id=b.id "
"WHERE f.locator=? AND i.state='pending'"
)
# Only an undecided item goes: a decision row (RESTRICT) is never deleted.
_SETTLE_ITEM = (
f"DELETE FROM {ITEMS_TABLE} WHERE id=? AND state='pending' " # nosec B608
f"AND revision=0 AND NOT EXISTS (SELECT 1 FROM {DECISIONS_TABLE} d "
f"WHERE d.item_id={ITEMS_TABLE}.id)"
)
_EPOCH = "SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
_PENDING_COUNT = (
f"SELECT COUNT(*) FROM {ITEMS_TABLE} WHERE state='pending'" # nosec B608
)
_QUEUE_COLUMNS = (
"id",
"state",
"revision",
"created_at",
"candidate_count",
"limited",
"spelled_path",
"locator",
)
_QUEUE_SELECT = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
f"i.limited,i.spelled_path,f.locator FROM {ITEMS_TABLE} i "
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id "
)
_QUEUE_FIRST = _QUEUE_SELECT + (
"WHERE i.state='pending' ORDER BY i.created_at,i.id LIMIT ?"
)
_QUEUE_AFTER = _QUEUE_SELECT + (
"WHERE i.state='pending' AND (i.created_at,i.id)>(?,?) "
"ORDER BY i.created_at,i.id LIMIT ?"
)
_DETAIL_COLUMNS = _QUEUE_COLUMNS + (
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"updated_at",
"sha256",
)
_DETAIL = (
"SELECT i.id,i.state,i.revision,i.created_at,i.candidate_count," # nosec B608
"i.limited,i.spelled_path,f.locator,i.byte_id,i.resolver_version,"
"i.evidence_hash,i.evidence,i.updated_at,b.sha256 "
f"FROM {ITEMS_TABLE} i JOIN {BYTES_TABLE} b ON b.id=i.byte_id "
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE i.id=?"
)
# The legacy rows and pair links the ingester's compatibility rules read.
_DEPENDENT_IMAGES = (
"SELECT gi.id,p.hash,gi.image_path FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=? ORDER BY gi.id"
)
_DEPENDENT_LINKS = (
"SELECT DISTINCT ip.pair_id,n.hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE ip.image_id=? ORDER BY ip.pair_id"
)
# Rows whose spelling ends with the file's name: the candidates for another
# spelling of the same physical file (the link summaries' filename fallback).
_SAME_NAME_ROWS = (
"SELECT id,image_path FROM generated_images "
"WHERE image_path LIKE ? ESCAPE '\\' ORDER BY id"
)
_TITLES = (
"SELECT p.hash,s.title FROM prompts p "
"JOIN prompt_library_state s ON s.prompt_id=p.id "
"WHERE p.hash IN ({marks}) AND s.title IS NOT NULL ORDER BY p.id"
)
_ITEM_ID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
_ITEM_COLUMNS = (
"id",
"byte_id",
"resolver_version",
"evidence_hash",
"evidence",
"candidate_count",
"limited",
"state",
"revision",
"created_at",
"updated_at",
"sha256",
)
def _validate_spelling(spelled_path: object, locator: str) -> None:
if (
not isinstance(spelled_path, str)
or not spelled_path
or len(spelled_path.encode("utf-8", "surrogatepass")) > MAX_SPELLED_PATH_BYTES
or os.path.normcase(spelled_path) != locator
):
raise ValueError("spelled_path must be the file's spelling of its locator")
def dependent_images(conn: sqlite3.Connection, spellings) -> list:
"""``(id, image_path, positive_hash)`` of every legacy row of the file.
``spellings`` are the paths the file is linked under (the Apply path and
every alias of the same physical file); each is matched as the ingester
matches one path. Rows are distinct and ordered by id.
"""
found = {}
for spelled in dict.fromkeys(spellings):
for row in conn.execute(
_DEPENDENT_IMAGES, (normalize_image_path(spelled), spelled)
):
found[row[0]] = (row[0], row[2], row[1])
return [found[key] for key in sorted(found)]
def same_name_rows(conn: sqlite3.Connection, name: str) -> list:
"""``(id, image_path)`` of rows whose spelling may end with ``name``."""
# Imported here: operations composes this module's mixin.
from .operations import escape_like
return [
(row[0], row[1])
for row in conn.execute(_SAME_NAME_ROWS, (f"%{escape_like(name)}",))
]
def dependency_fingerprint(
conn: sqlite3.Connection, image_path: str, aliases=()
) -> str:
"""SHA-256 of what Apply's compatibility depends on for ``image_path``.
The exact legacy rows of the file under ``image_path`` and every alias
spelling of it, with their positive hashes, and each row's pair links
with the negative hash, read as the ingester reads them. Unrelated prompt
changes (tags, titles) do not move it.
"""
rows = []
for image_id, _spelled, positive_hash in dependent_images(
conn, (image_path, *aliases)
):
links = conn.execute(_DEPENDENT_LINKS, (image_id,)).fetchall()
rows.append([image_id, positive_hash, [list(link) for link in links]])
text = canonical_evidence({"v": 1, "images": rows})
return hashlib.sha256(text.encode("utf-8")).hexdigest()
class ReviewOperations:
"""PromptDatabase mixin: insert and read Needs review items."""
def record_review_item(
self,
*,
locator: str,
spelled_path: str,
sha256: str,
size: int,
mtime_ns: int,
epoch: str,
resolver_version: int,
evidence: dict,
candidate_count: int,
limited: bool,
) -> bool:
"""Store one ambiguous result for these exact bytes; True when new.
The byte record is created (or its verified fingerprint refreshed)
like an observe. ``spelled_path`` is the file's own spelling (it must
fold to ``locator``); targets are formatted from it. Raises
``LibraryCommandError("storage-changed")`` when the storage epoch is
no longer ``epoch`` (a restore since the capture) and ``ValueError``
for invalid facts or evidence.
"""
_validate_target(locator, sha256, size, mtime_ns)
_validate_spelling(spelled_path, locator)
if type(resolver_version) is not int or resolver_version < 1:
raise ValueError("resolver_version must be a positive integer")
if (
type(candidate_count) is not int
or not 0 <= candidate_count <= MAX_CANDIDATES
):
raise ValueError(f"candidate_count must be between 0 and {MAX_CANDIDATES}")
if not isinstance(limited, bool) or (limited and candidate_count):
raise ValueError("A limited item carries no candidates")
text = canonical_evidence(evidence)
evidence_hash = evidence_digest(text)
with self._pair_transaction() as conn:
self._library_epoch(conn, epoch)
now = utc_now_iso()
file_id = self._annotation_owner(conn, locator, now)
byte_id, _revision, created, moved = self._annotation_byte(
conn, file_id, sha256, size, mtime_ns, now
)
if created or moved:
conn.execute(_BUMP_IMAGE_INDEX)
cursor = conn.execute(
_INSERT_ITEM,
(
_new_id(),
byte_id,
spelled_path,
resolver_version,
evidence_hash,
text,
candidate_count,
int(limited),
now,
now,
),
)
inserted = cursor.rowcount > 0
refreshed = _refresh_spelling(conn, locator, spelled_path)
if inserted or refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
return inserted
def refresh_review_spelling(self, locator: str, spelled_path: str) -> bool:
"""Record ``spelled_path`` as the spelling of every item at ``locator``.
The rescan's skip calls it for a file it does not copy again: a
case-only rename keeps size and ``mtime_ns``, and a target formatted
from the old spelling would be refused. True when a spelling changed
(the review index revision moves; item revisions do not, so an open
decision stays valid).
"""
_validate_locator(locator)
_validate_spelling(spelled_path, locator)
with self._pair_transaction() as conn:
refreshed = _refresh_spelling(conn, locator, spelled_path)
if refreshed:
conn.execute(_BUMP_REVIEW_INDEX)
return refreshed
def review_index_revision(self) -> int:
"""The queue's revision (moved by every queue change)."""
row = self.model.get_connection().execute(_REVIEW_INDEX).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
return row[0]
def review_items_for(self, locator: str) -> list:
"""Every item of the file at ``locator``, oldest first, evidence parsed."""
_validate_locator(locator)
return list(self._iter_review_items(locator))
def _iter_review_items(self, locator: str) -> Iterator[dict]:
conn = self.model.get_connection()
for row in conn.execute(_ITEMS_OF_LOCATOR, (locator,)):
item = dict(zip(_ITEM_COLUMNS, tuple(row)))
item["evidence"] = json.loads(item["evidence"])
item["limited"] = bool(item["limited"])
yield item
# -- queue reads (#249 E3) -------------------------------------------------
def review_fingerprint_queued(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when an item cites a byte record verified at this exact stat.
A rescan uses it to skip the copy and hash of a file it already
queued: the same ``(size, mtime_ns)`` the bytes were verified with.
"""
row = (
self.model.get_connection()
.execute(_QUEUED, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_pending_at(
self, locator: str, size: int, mtime_ns: int, resolver_version: int
) -> bool:
"""True when a *pending* item cites bytes verified at this exact stat.
``review_fingerprint_queued`` also matches decided items; a file whose
only item is decided is not in the queue.
"""
row = (
self.model.get_connection()
.execute(_PENDING_AT, (locator, size, mtime_ns, resolver_version))
.fetchone()
)
return row is not None
def review_file_linked(self, spellings) -> bool:
"""True when a saved row links the file under any of ``spellings``."""
with self._annotation_read() as conn:
return bool(dependent_images(conn, spellings))
def settle_linked_review_items(self, files) -> int:
"""Remove the pending, undecided items of files that are now linked.
``files`` holds ``(locator, spellings)`` per file. An item leaves the
queue only when it is still ``pending`` at revision 0 with no decision
row and a saved row links its file under one of ``spellings``; a
decided item is never touched. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
files = [(locator, tuple(spellings)) for locator, spellings in files]
for locator, _spellings in files:
_validate_locator(locator)
conn = self.model.get_connection()
candidates = [
(locator, spellings)
for locator, spellings in files
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not candidates:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator, spellings in candidates:
if not dependent_images(conn, spellings):
continue
for (item_id,) in conn.execute(
_PENDING_OF_LOCATOR, (locator,)
).fetchall():
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
return removed
def settle_untraced_review_items(self, locators) -> int:
"""Remove the pending, undecided items that offer nothing to choose.
``locators`` are files the caller found ambiguous without a choice
(fewer than two candidates carry a prompt). Of their items, one leaves
the queue only when its own stored evidence offers no choice either
(``evidence_offers_choice``) and it is still ``pending`` at revision 0
with no decision row, the same safe delete as
``settle_linked_review_items``. Files without a pending item cost one
indexed read and open no write transaction. Returns how many items
were removed (the review index revision moves when any was).
"""
locators = list(locators)
for locator in locators:
_validate_locator(locator)
conn = self.model.get_connection()
pending = [
locator
for locator in locators
if conn.execute(_PENDING_OF_LOCATOR, (locator,)).fetchone() is not None
]
if not pending:
return 0
removed = 0
with self._pair_transaction() as conn:
for locator in pending:
rows = conn.execute(_PENDING_EVIDENCE_OF_LOCATOR, (locator,)).fetchall()
for item_id, evidence in rows:
if evidence_offers_choice(json.loads(evidence)):
continue
removed += conn.execute(_SETTLE_ITEM, (item_id,)).rowcount
if removed:
conn.execute(_BUMP_REVIEW_INDEX)
return removed
def review_queue_page(self, *, limit: int, after: tuple | None = None) -> dict:
"""One page of pending items, oldest first, from one snapshot.
``after`` is the ``(created_at, id)`` keyset of the last item already
shown. Returns ``{epoch, revision, total, items, more}``; ``total`` is
the exact pending count and ``items`` carry their ``spelled_path`` and
``locator`` (server-side only).
"""
if type(limit) is not int or limit < 1:
raise ValueError("limit must be a positive integer")
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
revision = conn.execute(_REVIEW_INDEX).fetchone()
if epoch is None or revision is None:
raise sqlite3.DatabaseError("Missing review queue metadata")
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if after is None:
rows = conn.execute(_QUEUE_FIRST, (limit + 1,)).fetchall()
else:
rows = conn.execute(_QUEUE_AFTER, (*after, limit + 1)).fetchall()
items = [_queue_row(row, _QUEUE_COLUMNS) for row in rows[:limit]]
return {
"epoch": epoch[0],
"revision": revision[0],
"total": total,
"items": items,
"more": len(rows) > limit,
}
def review_item(self, item_id: str) -> dict | None:
"""One item with its parsed evidence and the pending total, or None."""
if not isinstance(item_id, str) or _ITEM_ID.fullmatch(item_id) is None:
return None
with self._annotation_read() as conn:
epoch = conn.execute(_EPOCH).fetchone()
if epoch is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item["evidence"] = json.loads(item["evidence"])
item["epoch"] = epoch[0]
item["total"] = conn.execute(_PENDING_COUNT).fetchone()[0]
return item
def review_dependency_fingerprint(self, image_path: str, aliases=()) -> str:
with self._annotation_read() as conn:
return dependency_fingerprint(conn, image_path, aliases)
def review_same_name_rows(self, name: str) -> list:
"""``(id, image_path)`` of legacy rows that may spell a file ``name``."""
with self._annotation_read() as conn:
return same_name_rows(conn, name)
def stored_prompt_titles(self, texts) -> dict:
"""Library titles of saved prompts by text (dedup hash), when set."""
hashes = {generate_prompt_hash(text): text for text in texts if text.strip()}
if not hashes:
return {}
marks = ",".join("?" * len(hashes))
found: dict[str, tuple] = {}
conn = self.model.get_connection()
for prompt_hash, title in conn.execute(
_TITLES.format(marks=marks), list(hashes)
):
text = hashes[prompt_hash]
found[text] = found.get(text, ()) + (title,)
return found
def _refresh_spelling(conn: sqlite3.Connection, locator: str, spelled: str) -> bool:
return conn.execute(_REFRESH_SPELLING, (spelled, spelled, locator)).rowcount > 0
def _queue_row(row, columns) -> dict:
item = dict(zip(columns, tuple(row)))
item["limited"] = bool(item["limited"])
return item
-287
View File
@@ -1,287 +0,0 @@
"""Needs review decisions (#249 E3): explicit, auditable and compatible-only.
A decision is one row of ``media_review_decisions`` for one item revision.
``selected`` links the chosen candidate's pair to the file through the public
ingester's compatible-only rules (``_ingest_resolved_pair_in``) in the same
``_pair_transaction`` as the decision row, the item's new state and revision
and the review index revision; any conflict or later failure rolls all of it
back. ``unresolved`` records the decision and links nothing. No decision
counts a run, touches usage or calls an execution writer.
Callers verify the file first (the capture must be the item's bytes and the
current file must still hash to them); nothing here reads the filesystem.
"""
import os
import sqlite3
from .annotation_schema import BYTES_TABLE
from .annotations import _new_id
from .library_commands import LibraryCommandError
from .models import utc_now_iso
from .review import (
_BUMP_REVIEW_INDEX,
_DETAIL,
_DETAIL_COLUMNS,
_PENDING_COUNT,
_queue_row,
dependency_fingerprint,
dependent_images,
same_name_rows,
)
from .review_schema import DECISIONS_TABLE, ITEMS_TABLE, canonical_evidence
DECISION_KIND = "user-selection"
_INSERT_DECISION = (
f"INSERT INTO {DECISIONS_TABLE}(id,item_id,decision_revision," # nosec B608
"created_at,outcome,candidate_hash,evidence,pair_id,image_id) "
"VALUES(?,?,?,?,?,?,?,?,?)"
)
_ADVANCE_ITEM = (
f"UPDATE {ITEMS_TABLE} SET state=?,revision=revision+1," # nosec B608
"updated_at=? WHERE id=? AND revision=?"
)
_ITEM_BYTES = (
f"SELECT b.sha256 FROM {ITEMS_TABLE} i " # nosec B608
f"JOIN {BYTES_TABLE} b ON b.id=i.byte_id WHERE i.id=?"
)
class ReviewCommandError(LibraryCommandError):
"""A decision lost a race or would rewrite an association.
``code`` is ``review-conflict`` (the item was decided or changed since it
was read), ``association-conflict`` (the choice is not compatible with
the file's saved links, or those links changed), ``file-changed`` (the
item is not about these bytes) or ``review-missing``. ``item`` is the
current item read inside the transaction, ``total`` the pending count and
``fingerprint`` the current dependency fingerprint, when known;
``changed`` marks an association conflict caused by links that changed
since the reviewer read them (rather than an incompatible choice).
"""
def __init__(
self, code, epoch, *, item=None, total=None, fingerprint=None, changed=False
):
super().__init__(code, epoch)
self.item = item
self.total = total
self.fingerprint = fingerprint
self.changed = changed
# The public view of ``item``, set by the API when it formats one.
self.view = None
class _Rollback(Exception):
"""Raised inside the transaction so the context manager rolls back."""
def __init__(self, error: ReviewCommandError):
super().__init__(error.code)
self.error = error
class ReviewDecisionOperations:
"""PromptDatabase mixin: record one decision on one Needs review item."""
def decide_review_item(
self,
*,
item_id: str,
expected_epoch: str,
expected_revision: int,
sha256: str,
outcome: str,
candidate: dict | None = None,
image_path: str | None = None,
metadata: dict | None = None,
expected_fingerprint: str | None = None,
aliases: tuple = (),
known_rows: frozenset | None = None,
) -> dict:
"""Record the decision; return ``{item, total, pair}``.
``selected`` needs the evidence ``candidate`` (with its ``hash``), the
``image_path`` the scan and the monitor link under, the captured
``metadata`` and the ``expected_fingerprint`` the reviewer saw.
``aliases`` are the other spellings legacy rows of the same physical
file use (a symlinked, junctioned or mapped root) and ``known_rows``
the ids of every same-name row the caller classified; a same-name row
it did not see is a changed association.
Raises ``LibraryCommandError("storage-changed")`` after a restore and
:class:`ReviewCommandError` for every refused decision, with zero
writes either way.
"""
if outcome not in ("selected", "unresolved"):
raise ValueError("outcome must be selected or unresolved")
if outcome == "selected" and (
not isinstance(candidate, dict) or not image_path
):
raise ValueError("A selected decision needs a candidate and a file")
try:
with self._pair_transaction() as conn:
return self._decide_in(
conn,
item_id=item_id,
expected_epoch=expected_epoch,
expected_revision=expected_revision,
sha256=sha256,
outcome=outcome,
candidate=candidate,
image_path=image_path,
metadata=metadata or {},
expected_fingerprint=expected_fingerprint,
aliases=tuple(aliases),
known_rows=known_rows,
)
except _Rollback as rollback:
raise rollback.error from None
def _decide_in(self, conn: sqlite3.Connection, **command) -> dict:
epoch = self._library_epoch(conn, command["expected_epoch"])
item_id = command["item_id"]
current = _current_item(conn, item_id)
if current is None:
raise _Rollback(ReviewCommandError("review-missing", epoch))
total = conn.execute(_PENDING_COUNT).fetchone()[0]
if (
current["state"] != "pending"
or current["revision"] != command["expected_revision"]
):
raise _Rollback(
ReviewCommandError("review-conflict", epoch, item=current, total=total)
)
if conn.execute(_ITEM_BYTES, (item_id,)).fetchone()[0] != command["sha256"]:
raise _Rollback(ReviewCommandError("file-changed", epoch))
decision_id = _new_id()
pair = None
candidate = command["candidate"] if command["outcome"] == "selected" else None
if candidate is not None:
pair = self._apply_candidate(
conn, epoch, current, decision_id, total, **command
)
evidence = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"outcome": command["outcome"],
"reviewed_digest": command["sha256"],
"item_evidence_hash": current["evidence_hash"],
"resolver_version": current["resolver_version"],
"candidate": candidate,
"dependency_fingerprint": command["expected_fingerprint"],
"pair": pair,
}
now = utc_now_iso()
self._insert_review_decision(
conn,
(
decision_id,
item_id,
current["revision"] + 1,
now,
command["outcome"],
candidate["hash"] if candidate is not None else None,
canonical_evidence(evidence),
pair["id"] if pair else None,
pair["image_id"] if pair else None,
),
)
conn.execute(
_ADVANCE_ITEM, (command["outcome"], now, item_id, current["revision"])
)
conn.execute(_BUMP_REVIEW_INDEX)
return {
"item": _current_item(conn, item_id),
"total": conn.execute(_PENDING_COUNT).fetchone()[0],
"pair": pair,
"decision_id": decision_id,
}
def _apply_candidate(
self, conn, epoch, current, decision_id, total, **command
) -> dict:
"""Link the candidate's pair, compatible-only; raise on any conflict."""
# Imported here: pair_ingestion imports operations, which composes this.
from .pair_ingestion import _ingest_resolved_pair_in
image_path = command["image_path"]
aliases = command["aliases"]
fingerprint = dependency_fingerprint(conn, image_path, aliases)
known = command["known_rows"]
unseen = known is not None and any(
row_id not in known
for row_id, _path in same_name_rows(conn, os.path.basename(image_path))
)
if unseen or fingerprint != command["expected_fingerprint"]:
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
changed=True,
)
)
rows = dependent_images(conn, (image_path, *aliases))
if len(rows) > 1:
# One file linked under several spellings: no single row to enrich.
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
if rows:
# Enrich the one row of this file under the spelling it was
# linked with, so the ingester's compatibility rules see it.
image_path = rows[0][1]
candidate = command["candidate"]
positive, negative = candidate["positive"], candidate["negative"]
provenance = {
"kind": DECISION_KIND,
"decision_id": decision_id,
"reviewed_digest": command["sha256"],
"candidate_hash": candidate["hash"],
"resolver_version": current["resolver_version"],
"positive": positive,
"negative": negative,
}
result = _ingest_resolved_pair_in(
conn,
self,
image_path,
command["metadata"],
{"state": "resolved", "positive": positive, "negative": negative},
provenance=provenance,
)
if result["status"] == "conflict":
raise _Rollback(
ReviewCommandError(
"association-conflict",
epoch,
item=current,
total=total,
fingerprint=fingerprint,
)
)
return {
"id": result["pair_id"],
"image_id": result["image_id"],
"status": result["status"],
}
@staticmethod
def _insert_review_decision(conn: sqlite3.Connection, values: tuple) -> None:
conn.execute(_INSERT_DECISION, values)
def _current_item(conn: sqlite3.Connection, item_id: str) -> dict | None:
row = conn.execute(_DETAIL, (item_id,)).fetchone()
if row is None:
return None
item = _queue_row(row, _DETAIL_COLUMNS)
item.pop("evidence")
return item
-507
View File
@@ -1,507 +0,0 @@
"""Needs review schema group (#249): capture-bound ambiguous items and decisions.
The group installs like ``image_annotations``: optional in old backups,
required once a database is current, and valid only beside a valid image
annotation group, because every item cites one annotation byte record. It owns
- ``media_review_metadata``: one row, the group's schema version and the
review index revision (moved by every queue change);
- ``media_review_items``: one ambiguous resolver result for one byte record,
keyed by ``(byte_id, resolver_version, evidence_hash)``, with the file's
spelling when it was recorded (the locator is case-folded on Windows, a
target is not), its canonical evidence, candidate count, ``limited`` flag,
state and revision;
- ``media_review_decisions``: the auditable decision history of an item, one
row per decision revision, with optional navigation to the pair and image
the decision linked (nulled, never cascaded, when those are deleted).
Evidence is canonical JSON (:func:`canonical_evidence`): one object, sorted
keys, no whitespace, ASCII only (so characters, bytes and the 8 MiB bound
agree and lone surrogates survive), no NaN or infinity. ``evidence_hash`` is
the SHA-256 of the item's evidence text.
Installation never backfills anything and never touches the filesystem.
Validation reads only: catalog objects compared after whitespace compaction,
columns and foreign keys through the PRAGMAs, persisted values in single
bounded SQL passes, then one streamed pass that re-canonicalizes each evidence
document (one row in memory at a time).
"""
import hashlib
import json
import re
import sqlite3
from typing import Any, Optional
from .annotation_schema import BYTES_TABLE
from .annotation_schema import TABLES as ANNOTATION_TABLES
from .annotation_schema import _catalog as _annotation_catalog
from .annotation_schema import annotation_schema_error
try:
from ..utils.json_strict import dumps_strict
except ImportError:
from utils.json_strict import dumps_strict
SCHEMA_VERSION = 1
METADATA_TABLE = "media_review_metadata"
ITEMS_TABLE = "media_review_items"
DECISIONS_TABLE = "media_review_decisions"
TABLES = (METADATA_TABLE, ITEMS_TABLE, DECISIONS_TABLE)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
MAX_CANDIDATES = 16
# The file's own spelling when it was recorded (bytes of UTF-8 text). Locators
# are case-folded on Windows, so a target is formatted from this instead.
MAX_SPELLED_PATH_BYTES = 16384
ITEM_STATES = ("pending", "selected", "unresolved")
DECISION_OUTCOMES = ("selected", "unresolved")
_TIMESTAMP_BYTES = 64
_HEX64 = "length({c})=64 AND {c} NOT GLOB '*[^0-9a-f]*'"
_EVIDENCE_CHECK = f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES})"
_TABLE_DDL = {
METADATA_TABLE: (
f"CREATE TABLE {METADATA_TABLE} (singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
f"CHECK(schema_version={SCHEMA_VERSION}), "
"review_index_revision INTEGER NOT NULL DEFAULT 0 "
"CHECK(review_index_revision>=0))"
),
ITEMS_TABLE: (
f"CREATE TABLE {ITEMS_TABLE} (id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"byte_id TEXT NOT NULL REFERENCES {BYTES_TABLE}(id) ON DELETE RESTRICT, "
"spelled_path TEXT NOT NULL CHECK(length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}), "
"resolver_version INTEGER NOT NULL, evidence_hash TEXT NOT NULL "
f"CHECK({_HEX64.format(c='evidence_hash')}), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"candidate_count INTEGER NOT NULL "
f"CHECK(candidate_count BETWEEN 0 AND {MAX_CANDIDATES}), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), "
"state TEXT NOT NULL CHECK(state IN ('pending','selected','unresolved')), "
"revision INTEGER NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, "
"updated_at TEXT NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS_TABLE: (
f"CREATE TABLE {DECISIONS_TABLE} (id TEXT PRIMARY KEY "
"CHECK(length(id)=36), item_id TEXT NOT NULL "
f"REFERENCES {ITEMS_TABLE}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
f"({_HEX64.format(c='candidate_hash')})), "
f"evidence TEXT NOT NULL {_EVIDENCE_CHECK}, "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
_INDEX_DDL = {
# The queue reads pending items oldest first with a stable tiebreak.
"idx_media_review_items_queue": (
"CREATE INDEX idx_media_review_items_queue ON "
f"{ITEMS_TABLE}(state, created_at, id)"
),
# Deleting a pair or an image nulls decision navigation by index, not scan.
"idx_media_review_decisions_pair": (
"CREATE INDEX idx_media_review_decisions_pair ON " f"{DECISIONS_TABLE}(pair_id)"
),
"idx_media_review_decisions_image": (
"CREATE INDEX idx_media_review_decisions_image ON "
f"{DECISIONS_TABLE}(image_id)"
),
}
# ``sqlite_autoindex_<table>_<n>``: the TEXT primary key and the UNIQUE key.
_AUTOINDEXES = {ITEMS_TABLE: 2, DECISIONS_TABLE: 2}
# (name, declared type, not null, default, primary key position)
_COLUMNS = {
METADATA_TABLE: (
("singleton_id", "INTEGER", 0, None, 1),
("schema_version", "INTEGER", 1, None, 0),
("review_index_revision", "INTEGER", 1, "0", 0),
),
ITEMS_TABLE: (
("id", "TEXT", 0, None, 1),
("byte_id", "TEXT", 1, None, 0),
("spelled_path", "TEXT", 1, None, 0),
("resolver_version", "INTEGER", 1, None, 0),
("evidence_hash", "TEXT", 1, None, 0),
("evidence", "TEXT", 1, None, 0),
("candidate_count", "INTEGER", 1, None, 0),
("limited", "INTEGER", 1, None, 0),
("state", "TEXT", 1, None, 0),
("revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("updated_at", "TEXT", 1, None, 0),
),
DECISIONS_TABLE: (
("id", "TEXT", 0, None, 1),
("item_id", "TEXT", 1, None, 0),
("decision_revision", "INTEGER", 1, None, 0),
("created_at", "TEXT", 1, None, 0),
("outcome", "TEXT", 1, None, 0),
("candidate_hash", "TEXT", 0, None, 0),
("evidence", "TEXT", 1, None, 0),
("pair_id", "INTEGER", 0, None, 0),
("image_id", "INTEGER", 0, None, 0),
),
}
# (column, parent table, parent column, on delete)
_FOREIGN_KEYS = {
METADATA_TABLE: set(),
ITEMS_TABLE: {("byte_id", BYTES_TABLE, "id", "RESTRICT")},
DECISIONS_TABLE: {
("item_id", ITEMS_TABLE, "id", "RESTRICT"),
("pair_id", "prompt_pairs", "id", "SET NULL"),
("image_id", "generated_images", "id", "SET NULL"),
},
}
class _NonCanonical(ValueError):
pass
def _reject_constant(name: str) -> Any:
raise _NonCanonical(f"JSON constant {name} is not allowed")
# Built once: validation re-encodes every stored document at startup.
_ENCODER = json.JSONEncoder(
sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False
)
_DECODER = json.JSONDecoder(parse_constant=_reject_constant)
def canonical_evidence(value: Any) -> str:
"""Serialize one evidence object canonically; ``ValueError`` when invalid.
The text is a JSON object with sorted keys, no whitespace, ASCII only and
no NaN/infinity, at most ``MAX_EVIDENCE_BYTES`` bytes. A non-finite number
(ComfyUI stamps ``NaN`` into graphs) is written as ``null``, the store's
policy everywhere (``utils.json_strict.dumps_strict``).
"""
if not isinstance(value, dict):
raise ValueError("Review evidence must be a JSON object")
try:
try:
text = _ENCODER.encode(value)
except ValueError as error:
if "Out of range float" not in str(error):
raise
text = dumps_strict(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
except (TypeError, RecursionError) as error:
raise ValueError(f"Review evidence is not serializable: {error}") from None
if len(text) > MAX_EVIDENCE_BYTES:
raise ValueError("Review evidence is larger than 8 MiB")
return text
def choosable(candidate: Any) -> bool:
"""True when a candidate offers a prompt to choose: it is ``resolved`` and
its positive side carries non-blank text. The page disables any other
candidate, so it is never something to choose.
"""
if not isinstance(candidate, dict) or candidate.get("state") != "resolved":
return False
positive = candidate.get("positive")
if not isinstance(positive, dict):
return False
text = positive.get("text")
return isinstance(text, str) and bool(text.strip())
# An item is worth a decision only when at least this many candidates offer
# a prompt; with fewer there is nothing to choose between (maintainer test of
# #284/#288: 39 items whose candidates were all "unresolved").
MIN_CHOOSABLE = 2
def evidence_offers_choice(evidence: Any) -> bool:
"""False when stored evidence provably offers nothing to choose.
Unlimited evidence must list ``MIN_CHOOSABLE`` choosable candidates. A
limited item stores none, only ``candidate_total``; it may offer a choice
(more than ``MAX_CANDIDATES`` prompts) unless that total is too small.
"""
if not isinstance(evidence, dict):
return False
if evidence.get("limited"):
total = evidence.get("candidate_total")
return type(total) is int and total >= MIN_CHOOSABLE
candidates = evidence.get("candidates")
if not isinstance(candidates, list):
return False
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
def evidence_digest(text: str) -> str:
"""SHA-256 of canonical evidence text (ASCII, hence UTF-8 identical)."""
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def _is_canonical(text: Any) -> bool:
if not isinstance(text, str):
return False
try:
value = _DECODER.decode(text)
return canonical_evidence(value) == text
except (ValueError, RecursionError):
return False
def _compact(sql: str) -> str:
return re.sub(r"\s+", "", sql).lower()
def _uuid(column: str) -> str:
"""SQL: *column* is a lowercase hyphenated 36-character UUID string."""
return (
f"typeof({column}) = 'text' AND length({column}) = 36 "
f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
f"AND length(replace({column}, '-', '')) = 32 "
f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
)
def _timestamp(column: str) -> str:
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) <= {_TIMESTAMP_BYTES}"
)
def _non_negative(column: str) -> str:
return f"typeof({column}) = 'integer' AND {column} >= 0"
def _hex64(column: str) -> str:
return f"typeof({column}) = 'text' AND " + _HEX64.format(c=column)
def _evidence(column: str) -> str:
# Type and size only: the streamed pass below proves the text canonical
# (which implies printable ASCII), so a pattern scan here would read every
# document a second time.
return (
f"typeof({column}) = 'text' "
f"AND length(CAST({column} AS BLOB)) BETWEEN 2 AND {MAX_EVIDENCE_BYTES}"
)
def _nullable_integer(column: str) -> str:
return f"({column} IS NULL OR typeof({column}) = 'integer')"
def _in(column: str, values: tuple) -> str:
return f"typeof({column}) = 'text' AND {column} IN ({repr(values)[1:-1]})"
# Every row of a table must satisfy its predicate; one pass each, LIMIT 1.
_ROW_PREDICATES = {
ITEMS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("byte_id"),
"typeof(spelled_path) = 'text' AND length(CAST(spelled_path AS BLOB)) "
f"BETWEEN 1 AND {MAX_SPELLED_PATH_BYTES}",
"typeof(resolver_version) = 'integer'",
_hex64("evidence_hash"),
_evidence("evidence"),
_non_negative("candidate_count"),
f"candidate_count <= {MAX_CANDIDATES}",
"typeof(limited) = 'integer' AND limited IN (0, 1)",
_in("state", ITEM_STATES),
_non_negative("revision"),
_timestamp("created_at"),
_timestamp("updated_at"),
)
),
DECISIONS_TABLE: " AND ".join(
(
_uuid("id"),
_uuid("item_id"),
_non_negative("decision_revision"),
_timestamp("created_at"),
_in("outcome", DECISION_OUTCOMES),
"(CASE WHEN outcome = 'selected' THEN "
f"{_hex64('candidate_hash')} ELSE candidate_hash IS NULL END)",
_evidence("evidence"),
_nullable_integer("pair_id"),
_nullable_integer("image_id"),
)
),
}
def _expected_catalog() -> set:
objects = {("table", name, _compact(sql)) for name, sql in _TABLE_DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in _INDEX_DDL.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in _AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def _catalog(conn: sqlite3.Connection) -> list:
names = TABLES + tuple(_INDEX_DDL)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return conn.execute( # only "?" placeholders are interpolated
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + TABLES,
).fetchall()
def _structure_error(conn: sqlite3.Connection) -> Optional[str]:
for table, expected in _COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[4], row[5])
for row in conn.execute(f"PRAGMA table_info({table})")
)
if columns != expected:
return f"Review queue table {table} has invalid columns"
foreign_keys = {
(row[3], row[2], row[4], row[6])
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
}
if foreign_keys != _FOREIGN_KEYS[table]:
return f"Review queue table {table} has invalid foreign keys"
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
return f"Review queue table {table} contains foreign key violations"
return None
def _evidence_error(conn: sqlite3.Connection) -> Optional[str]:
"""Stream every evidence document once; one row is held at a time."""
for evidence_hash, text in conn.execute(
f"SELECT evidence_hash, evidence FROM {ITEMS_TABLE}" # nosec B608
):
if not _is_canonical(text) or evidence_digest(text) != evidence_hash:
return f"Review queue table {ITEMS_TABLE} contains invalid evidence"
for (text,) in conn.execute(
f"SELECT evidence FROM {DECISIONS_TABLE}" # nosec B608
):
if not _is_canonical(text):
return f"Review queue table {DECISIONS_TABLE} contains invalid evidence"
return None
def _value_error(conn: sqlite3.Connection) -> Optional[str]:
# Only this module's constants are interpolated into the SQL below.
rows = conn.execute(
"SELECT COUNT(*), MIN(typeof(singleton_id) = 'integer' " # nosec B608
"AND singleton_id = 1 AND typeof(schema_version) = 'integer' "
f"AND schema_version = {SCHEMA_VERSION} "
f"AND {_non_negative('review_index_revision')}) FROM {METADATA_TABLE}"
).fetchone()
if rows[0] != 1 or rows[1] != 1:
return "Review queue metadata is invalid"
for table, predicate in _ROW_PREDICATES.items():
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1" # nosec B608
).fetchone():
return f"Review queue table {table} contains invalid values"
return _evidence_error(conn)
def review_without_annotations_error(conn: sqlite3.Connection) -> Optional[str]:
"""Catalog only: a review group present without its annotation tables.
Startup asks this before the annotation group installs, which would make
a review group copied without it look valid. It reads no rows; the full
validation runs once in :func:`migrate_review_state`.
"""
if not _catalog(conn):
return None
present = {
name for kind, name, _sql in _annotation_catalog(conn) if kind == "table"
}
if present != set(ANNOTATION_TABLES):
return (
"Review queue schema requires a valid image annotation schema: "
"Missing image annotation schema"
)
return None
def review_schema_error(
conn: sqlite3.Connection,
*,
allow_missing: bool = True,
annotations_valid: bool = False,
) -> Optional[str]:
"""Reject partial or unsupported review schemas and invalid rows.
Reads only: nothing is ever repaired. A database without any of the
group's objects is acceptable for backups that predate it unless
``allow_missing`` is false. A present group requires a present, valid
image annotation group; ``annotations_valid`` says the caller has just
validated it (startup's annotation install), so only its presence is
checked again.
"""
objects = _catalog(conn)
if not objects:
return None if allow_missing else "Missing review queue schema"
if annotations_valid:
annotation_error = review_without_annotations_error(conn)
else:
annotation_error = annotation_schema_error(conn, allow_missing=False)
if annotation_error:
annotation_error = (
"Review queue schema requires a valid image annotation schema: "
+ annotation_error
)
if annotation_error:
return annotation_error
present = {name for kind, name, _sql in objects if kind == "table"}
if present != set(TABLES):
return "Incomplete review queue schema"
actual = {(kind, name, _compact(sql or "")) for kind, name, sql in objects}
if actual != _expected_catalog():
return "Review queue has an invalid schema definition"
return _structure_error(conn) or _value_error(conn)
def migrate_review_state(conn: sqlite3.Connection) -> None:
"""Install the group atomically; validate before and after creating it.
Runs after ``migrate_annotation_state`` (which has just validated the
annotation group): the group is only valid beside the annotation group it
cites. A present group is validated once, evidence included.
"""
conn.execute("SAVEPOINT media_review_schema")
try:
installed = bool(_catalog(conn))
error = review_schema_error(conn, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
if installed:
# Already complete and valid: nothing to create, and the evidence
# pass is the expensive part of validation, so it runs once.
conn.execute("RELEASE media_review_schema")
return
for sql in _TABLE_DDL.values():
conn.execute(sql.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
for sql in _INDEX_DDL.values():
conn.execute(sql.replace("INDEX", "INDEX IF NOT EXISTS", 1))
conn.execute(
f"INSERT OR IGNORE INTO {METADATA_TABLE} "
"(singleton_id, schema_version, review_index_revision) VALUES (1, ?, 0)",
(SCHEMA_VERSION,),
)
error = review_schema_error(conn, allow_missing=False, annotations_valid=True)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE media_review_schema")
except BaseException:
conn.execute("ROLLBACK TO media_review_schema")
conn.execute("RELEASE media_review_schema")
raise
+2 -12
View File
@@ -8,8 +8,6 @@ Split into domain-specific mixins for maintainability:
- AutotagRoutesMixin: auto-tagging model management and tagging
- JobRoutesMixin: the session job registry behind Activity (py/api/jobs.py)
- ImageTagJobsMixin: bulk Add tags / Auto-tag as image-tags jobs
- ReviewRoutesMixin: the Needs review queue, item detail and decisions
- ReviewRecordingMixin: Needs review recording for the scan and the monitor
- SystemRoutesMixin: System health check, Overview and usage statistics
"""
@@ -18,7 +16,6 @@ import datetime
import functools
import gzip as gzip_module
import json
import math
import ntpath
import os
import re
@@ -32,8 +29,6 @@ from .pairs import PairRoutesMixin
from .library import LibraryRoutesMixin
from .vocabulary import VocabularyRoutesMixin
from .annotations import AnnotationRoutesMixin
from .review import ReviewRoutesMixin
from .review_recording import ReviewRecordingMixin
from .saved_views import SavedViewRoutesMixin
from .history import HistoryRoutesMixin
from .images import ImageRoutesMixin
@@ -268,8 +263,6 @@ class PromptManagerAPI(
LibraryRoutesMixin,
VocabularyRoutesMixin,
AnnotationRoutesMixin,
ReviewRoutesMixin,
ReviewRecordingMixin,
SavedViewRoutesMixin,
HistoryRoutesMixin,
PromptRoutesMixin,
@@ -310,8 +303,6 @@ class PromptManagerAPI(
# created on first use with their restore subscription.
self._captures = None
self._captures_unsubscribe = None
# Set by close_annotation_captures (shutdown): no store is recreated.
self._captures_closed = False
self._cached_output_dir = None # Lazy-cached by _find_comfyui_output_dir()
self._html_cache = {} # Cached HTML file contents keyed by path
self._gallery_cache = {} # dict: path_str -> (files, timestamp)
@@ -492,7 +483,6 @@ class PromptManagerAPI(
self._register_library_routes(routes)
self._register_vocabulary_routes(routes)
self._register_annotation_routes(routes)
self._register_review_routes(routes)
self._register_saved_view_routes(routes)
self._register_history_routes(routes)
self._register_prompt_routes(routes)
@@ -619,12 +609,12 @@ class PromptManagerAPI(
return prompts
def _clean_nan_recursive(self, obj):
"""Recursively replace NaN and Infinity in nested data with None."""
"""Recursively clean NaN values from nested data structures."""
if isinstance(obj, dict):
return {key: self._clean_nan_recursive(value) for key, value in obj.items()}
elif isinstance(obj, list):
return [self._clean_nan_recursive(item) for item in obj]
elif isinstance(obj, float) and not math.isfinite(obj):
elif isinstance(obj, float) and str(obj) == "nan":
return None
else:
return obj
+32 -131
View File
@@ -4,7 +4,6 @@ import asyncio
import contextlib
from contextlib import closing
import datetime
import functools
import hashlib
import json
import math
@@ -42,10 +41,9 @@ try:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_page_json,
parse_js_json,
)
from ...utils.prompt_pair_resolver import resolve_prompt_pair
from ...utils.review_capture import reviewable
except ImportError:
from database.operations import PromptDatabase
from database.pair_ingestion import ingest_resolved_pair
@@ -53,10 +51,9 @@ except ImportError:
PLACEHOLDERS,
GenerationParamsError,
extract_generation_params,
parse_page_json,
parse_js_json,
)
from utils.prompt_pair_resolver import resolve_prompt_pair
from utils.review_capture import reviewable
try:
@@ -214,30 +211,6 @@ def _time_left(done, total, batches, elapsed):
return f" · about {minutes} min left"
def new_review_counts():
"""The scan's Needs review tally: ``added`` new items, ``queued`` files
in the queue after this scan (new or already there), ``settled`` items
removed because their file is linked now, ``untraced`` items removed
because they offer nothing to choose, and ``errors``."""
return {"added": 0, "queued": 0, "settled": 0, "untraced": 0, "errors": 0}
def _review_scope(review):
"""The scan job's ``scope`` keys for its Needs review tally.
``review_items`` is the number of this scan's files that are in the
pending queue after it: the review group's ambiguous files, less those a
capture could not read or a decision already settled.
"""
return {
"review_items": review["queued"],
"review_added": review["added"],
"review_settled": review["settled"],
"review_untraced": review["untraced"],
"review_errors": review["errors"],
}
def _scan_status_line(done, total, batches, elapsed):
""" "Processing file N of M", with an estimate once enough batches ran."""
line = f"Processing file {done:,} of {total:,}"
@@ -831,9 +804,6 @@ class AdminRoutesMixin:
result = await self._run_in_executor(
self._delete_duplicate_images_sync, image_paths, output_dir
)
if result["deleted_count"]:
# The cached listing would keep (and count) the deleted files.
self.invalidate_gallery_cache()
response_data = {
"success": True,
@@ -1371,34 +1341,28 @@ class AdminRoutesMixin:
}
)
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE, roots=None, output_dirs=None):
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE):
"""Register the scan job synchronously; its runner starts on the loop.
``files`` is the scope (``SCAN_SCOPES``). ``roots`` selects sources by
their index in ``output_dirs`` (the effective sources the caller
resolved); None scans every source, resolved when the job starts.
Raises JobBusy while another record-changing job runs.
``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another
record-changing job runs.
"""
if files not in SCAN_SCOPES:
raise ValueError(f"Unknown scan scope: {files!r}")
if roots is not None and (
output_dirs is None or not roots or max(roots) >= len(output_dirs)
):
raise ValueError("A scoped scan needs the sources its roots index")
job = self._jobs.start(
SCAN_JOB_KIND,
SCAN_JOB_TITLE,
record_changing=True,
scope={"files": files, "roots": None if roots is None else list(roots)},
runner=functools.partial(self._run_scan_job, output_dirs=output_dirs),
scope={"files": files},
runner=self._run_scan_job,
groups=SCAN_GROUP_NAMES,
)
self._scan_job = job
return job
async def _run_scan_job(self, job, output_dirs=None):
async def _run_scan_job(self, job):
final = None
async for payload in self._scan_images_events(job, output_dirs):
async for payload in self._scan_images_events(job):
job.publish(payload)
final = payload.get("type")
state = SCAN_FINAL_STATES.get(final, "finished")
@@ -1433,25 +1397,16 @@ class AdminRoutesMixin:
)
return response
async def _scan_images_events(self, job, output_dirs=None):
"""Yield progress payloads while scanning the output directories.
async def _scan_images_events(self, job):
"""Yield progress payloads while scanning every configured output directory.
Every effective source is walked, or only those ``job.scope["roots"]``
selects (indexes into ``output_dirs``). The payloads are the legacy SSE
events, unchanged; ``job`` also gets its status line, totals and one
outcome per processed file. Rows keep each source's listing index.
The payloads are the legacy SSE events, unchanged; ``job`` also gets
its status line, totals and one outcome per processed file.
"""
try:
self.logger.info("Starting image scan operation")
if output_dirs is None:
output_dirs = self._get_all_output_dirs()
selected = job.scope.get("roots")
walked = (
output_dirs
if selected is None
else [output_dirs[index] for index in selected]
)
if not walked:
output_dirs = self._get_all_output_dirs()
if not output_dirs:
self.logger.error("No output directories found")
message = (
"No output directories found. "
@@ -1461,18 +1416,18 @@ class AdminRoutesMixin:
yield {"type": "error", "message": message}
return
dir_names = [self._public_path(d) for d in walked]
folders = _plural(len(walked), "output folder")
job.scope["output_folders"] = len(walked)
dir_names = [self._public_path(d) for d in output_dirs]
folders = _plural(len(output_dirs), "output folder")
job.scope["output_folders"] = len(output_dirs)
scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")]
job.log_line(f"Scan started · {folders} · {scope}")
job.update(status_line=f"Looking for media files in {folders}")
yield self._scan_progress(
0, f"Scanning {len(walked)} directory(ies) for media files..."
0, f"Scanning {len(output_dirs)} directory(ies) for media files..."
)
media_files = await self._run_in_executor(
self._collect_output_media_sync, walked
self._collect_output_media_sync, output_dirs
)
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
job.update(
@@ -1487,13 +1442,10 @@ class AdminRoutesMixin:
known = set()
if media_files and job.scope.get("files") != "all":
known = await self._known_scan_files(job, media_files)
review = new_review_counts()
job.scope.update(_review_scope(review))
async for event in self._scan_batches(
media_files, counts, job, roots, known, review
media_files, counts, job, roots, known
):
yield event
self._log_review_items(job, review)
self.logger.info(
f"Scan completed: processed={counts['processed']}, "
@@ -1530,16 +1482,12 @@ class AdminRoutesMixin:
sizes = {str(media_file): _file_size(media_file) for media_file in media_files}
return self.db.get_known_resolved_paths(sizes)
async def _scan_batches(
self, media_files, counts, job, roots=(), known=(), review=None
):
async def _scan_batches(self, media_files, counts, job, roots=(), known=()):
"""Process ``media_files`` in executor batches, updating ``counts`` in place.
Records each file's outcome in ``job`` and yields one progress event
per batch. Files in ``known`` (path strings) are not read: they count
as processed and unchanged and are recorded as skipped. ``review``
counts the Needs review items the batches record (``job.scope``
``review_items`` / ``review_errors`` follow it after every batch).
as processed and unchanged and are recorded as skipped.
"""
total = len(media_files)
started = time.monotonic()
@@ -1562,16 +1510,9 @@ class AdminRoutesMixin:
batch_results,
roots=roots,
outcomes=outcomes,
review=review,
)
for key in counts:
counts[key] += batch_counts[key]
if review is not None:
# After the batch's links: a file linked now leaves the queue.
await self._run_in_executor(
self._settle_scanned_reviews, batch, roots, review
)
job.scope.update(_review_scope(review))
for group, row, log_line in outcomes:
job.record(group, row, log_line)
@@ -1621,20 +1562,12 @@ class AdminRoutesMixin:
return map_parallel(extract, file_batch, PromptManagerConfig.WORKER_THREADS)
def _ingest_scan_batch_sync(
self, batch_results, roots=(), outcomes=None, review=None
):
def _ingest_scan_batch_sync(self, batch_results, roots=(), outcomes=None):
"""Store the prompts found in a metadata batch (blocking DB work).
Returns the legacy counts. When ``outcomes`` is a list, every processed
file appends its ``(group, row, log_line)`` for the scan job, with the
path relative to its root in ``roots`` (see ``_scan_roots``). A file
whose final outcome is ambiguous and unlinked (the resolver found it
ambiguous and the fallback did not link it) is recorded for Needs
review (best effort, counted in ``review`` when given); one a saved
row already links is reported ``unchanged`` and never queued. An
ambiguous file with nothing to choose is reported ``unresolved`` and
its pending items without a choice leave the queue.
path relative to its root in ``roots`` (see ``_scan_roots``).
"""
counts = dict.fromkeys(SCAN_COUNT_KEYS, 0)
@@ -1646,27 +1579,13 @@ class AdminRoutesMixin:
if self._scan_job and self._scan_job.cancelled:
break
counts["processed"] += 1
ambiguous, untraced = [], []
try:
outcome = self._ingest_scanned_file(
media_file,
metadata,
detailed=True,
ambiguous=ambiguous,
untraced=untraced,
)
outcome = self._ingest_scanned_file(media_file, metadata, detailed=True)
except Exception as e:
self.logger.error(f"Error processing {media_file.name}: {e}")
counts["failed"] += 1
note(media_file, "error")
continue
if ambiguous and outcome == "ambiguous":
if self._review_file_linked(media_file, roots):
outcome = "unchanged" # already linked: not a review case
else:
self._record_scanned_review(media_file, roots, review)
elif untraced and outcome == "unresolved":
self._settle_untraced_reviews(untraced, roots, review)
if outcome is None:
note(media_file, None)
continue
@@ -1684,14 +1603,11 @@ class AdminRoutesMixin:
@staticmethod
def _graph_prompt_text(metadata):
"""The positive prompt the browser parser traces from the graphs, or None.
Records parse as the legacy pages parsed them (NaN reads as null).
"""
"""The positive prompt the browser parser traces from the graphs, or None."""
records = {}
for key in ("prompt", "workflow"):
value = metadata.get(key)
records[key] = parse_page_json(value) if isinstance(value, str) else value
records[key] = parse_js_json(value) if isinstance(value, str) else value
try:
positive = extract_generation_params(records)["positivePrompt"]
except GenerationParamsError:
@@ -1700,21 +1616,13 @@ class AdminRoutesMixin:
return None
return positive.strip() or None
def _ingest_scanned_file(
self, media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
def _ingest_scanned_file(self, media_file, metadata, detailed=False):
"""Enrich exact pairs; retain legacy positive-only import for absent pairs.
Returns the file's scan status, or None without prompt metadata. With
``detailed``, a stored or linked file returns a dict instead: its
``status``, the prompt's display ``title`` and, for a resolved pair,
the pair counts. When the resolver finds the file ambiguous (whether
or not the browser parser then links it) and ``ambiguous`` is a list,
``media_file`` is appended to it; only a file that then returns
``"ambiguous"`` (nothing linked it) is a Needs review case. An
ambiguous result with nothing to choose (``reviewable`` is False: fewer
than two candidates carry a prompt) that nothing links returns
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
the pair counts.
"""
if not metadata:
return None
@@ -1730,13 +1638,6 @@ class AdminRoutesMixin:
return None
_with_file_size(canonical, media_file)
resolution = resolve_prompt_pair(canonical)
# Ambiguous without a choice: unresolved unless something links it.
unchoosable = resolution["state"] == "ambiguous" and not reviewable(resolution)
if unchoosable:
if untraced is not None:
untraced.append(media_file)
elif resolution["state"] == "ambiguous" and ambiguous is not None:
ambiguous.append(media_file)
if resolution["state"] == "resolved":
result = ingest_resolved_pair(
self.db, str(media_file), canonical, resolution
@@ -1756,9 +1657,9 @@ class AdminRoutesMixin:
# Graphs without a recognized sampler keep the legacy extractor below.
prompt_text = self._graph_prompt_text(metadata)
if not prompt_text:
return "unresolved" if unchoosable else resolution["state"]
return resolution["state"]
elif resolution["state"] == "ambiguous":
return "unresolved" if unchoosable else "ambiguous"
return "ambiguous"
elif resolution["state"] == "unresolved" and not isinstance(
canonical.get("parameters"), str
):
+13 -35
View File
@@ -13,7 +13,6 @@ import datetime
import functools
import os
import re
import threading
from pathlib import Path
from aiohttp import web
@@ -74,9 +73,6 @@ _HEX64 = re.compile(r"[0-9a-f]{64}")
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
# secrets.token_urlsafe(32) is 43 characters; the bound only rejects junk.
_TOKEN = re.compile(r"[A-Za-z0-9_-]{1,128}")
# The store is created on first use by the event loop, the scan's executor or
# the monitor's worker (Needs review), so its creation is serialized.
_CAPTURE_STORE_LOCK = threading.Lock()
_PREVIEW_TYPES = {
".png": "image/png",
@@ -263,46 +259,31 @@ class AnnotationRoutesMixin:
Created lazily so an API object that never observes a file (most tests,
a server that never opens the Tag dialog) owns no temp directory. A
restore of the database dooms every capture (``invalidate_all``).
Thread-safe: Needs review recording calls it off the event loop.
Raises ``CaptureError`` after ``close_annotation_captures``.
"""
with _CAPTURE_STORE_LOCK:
if getattr(self, "_captures_closed", False):
# A recording still in flight at shutdown must not create a
# new temp directory after the cleanup removed the old one.
raise CaptureError("The capture store is closed")
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
if self._captures is None:
self._captures = CaptureStore()
if self._captures_unsubscribe is None:
self._captures_unsubscribe = subscribe_history_restore(
self.db.model.db_path, self._invalidate_captures
)
return self._captures
def _invalidate_captures(self) -> None:
store = self._captures
if store is not None:
store.invalidate_all()
def close_annotation_captures(self, *, final: bool = False) -> None:
"""Remove every capture file and the owned directory. Idempotent.
Taken under the store lock. ``final`` (the shutdown cleanup) also makes
``_capture_store`` refuse to create another store, so a recording
still in flight cannot leak a new temp directory after it.
"""
with _CAPTURE_STORE_LOCK:
if final:
self._captures_closed = True
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
store, self._captures = self._captures, None
def close_annotation_captures(self) -> None:
"""Remove every capture file and the owned directory. Idempotent."""
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
if unsubscribe is not None:
unsubscribe()
store, self._captures = self._captures, None
if store is not None:
store.close()
async def cleanup_annotation_captures(self, app: web.Application) -> None:
self.close_annotation_captures(final=True)
self.close_annotation_captures()
# -- shared request steps --------------------------------------------------
@@ -348,10 +329,7 @@ class AnnotationRoutesMixin:
root, error = self._scoped_root(index, scope)
if error is not None:
return error
try:
store = self._capture_store()
except CaptureError as exc: # the server is shutting down
return _capture_error(exc)
store = self._capture_store()
try:
epoch = await self._run_in_executor(self.db.get_execution_epoch)
capture = await _settled(
+21 -9
View File
@@ -3,7 +3,6 @@
import asyncio
import base64
import binascii
import functools
import hashlib
import hmac
import json
@@ -38,7 +37,6 @@ try:
from ...database.models import normalize_image_path
from ...utils.card_seed import card_seed, header_params, parsed_positive
from ...utils.parallel import map_parallel
from ...utils.json_strict import dumps_strict
from ...utils.inspection_response import (
encode_inspection_response,
library_metadata,
@@ -58,7 +56,6 @@ except ImportError:
from database.models import normalize_image_path
from utils.card_seed import card_seed, header_params, parsed_positive
from utils.parallel import map_parallel
from utils.json_strict import dumps_strict
from utils.inspection_response import (
encode_inspection_response,
library_metadata,
@@ -985,12 +982,27 @@ class ImageRoutesMixin:
# Add url and thumbnail_url so the frontend can serve them
cleaned_images = self._present_images(cleaned_images)
# Rows an older version stored may still hold NaN/Infinity: answer
# strict JSON, and leave prompt text that spells "NaN" untouched.
return web.json_response(
{"success": True, "images": cleaned_images},
dumps=functools.partial(dumps_strict, default=str),
)
# Additional fallback: convert to JSON string and clean NaN values manually
try:
response_data = {"success": True, "images": cleaned_images}
# Convert to JSON string
json_str = json.dumps(response_data, default=str)
# Clean any remaining NaN values with regex
json_str = re.sub(r":\s*NaN", ": null", json_str)
json_str = re.sub(r"\[\s*NaN\s*\]", "[null]", json_str)
json_str = re.sub(r",\s*NaN\s*,", ", null,", json_str)
json_str = re.sub(r",\s*NaN\s*\]", ", null]", json_str)
json_str = re.sub(r"\[\s*NaN\s*,", "[null,", json_str)
# Parse back to verify it's valid JSON
cleaned_data = json.loads(json_str)
return web.json_response(cleaned_data)
except Exception as json_error:
self.logger.error(f"JSON cleaning error: {json_error}")
# Fallback to original response
return web.json_response({"success": True, "images": cleaned_images})
except Exception as e:
self.logger.error(f"Get prompt images error: {e}")
+17 -93
View File
@@ -18,74 +18,29 @@ def _not_found():
return web.json_response({"success": False, "error": "Job not found"}, status=404)
MAX_SCAN_ROOTS = 64
_SCOPE_ERROR = 'The scan scope must be "new" or "all"'
_ROOTS_ERROR = (
f"roots must be a non-empty list of up to {MAX_SCAN_ROOTS} source indexes, "
"with paths naming the public path of each"
)
SOURCES_CHANGED_ERROR = "The sources changed since they were loaded. Reload them."
def _bad_request(message):
return web.json_response({"success": False, "error": message}, status=400)
def _bad_request(message, code=None):
body = {"success": False, "error": message}
if code is not None:
body["code"] = code
return web.json_response(body, status=400)
async def _scan_scope(request):
"""The requested scan scope ("new" or "all"), or None when the body is invalid.
def _scan_roots(body):
"""``(roots, error)``: ``{index: public path}`` of the selected sources, or None.
Absent ``roots`` means every source. When present it is a non-empty list
of at most MAX_SCAN_ROOTS non-negative ints (never bools or floats), and
``paths`` the same-length list of the public path the caller read for
each (GET /settings), so a stale index never scans another folder.
An empty body (also a chunked one with no bytes), or one without
``scope``, asks for the default.
"""
if "roots" not in body:
return None, None
raw = body["roots"]
paths = body.get("paths")
if not isinstance(raw, list) or not raw or len(raw) > MAX_SCAN_ROOTS:
return None, _ROOTS_ERROR
if not isinstance(paths, list) or len(paths) != len(raw):
return None, _ROOTS_ERROR
selected = {}
for value, path in zip(raw, paths):
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
return None, _ROOTS_ERROR
if not isinstance(path, str) or selected.get(value, path) != path:
return None, _ROOTS_ERROR
selected[value] = path
return dict(sorted(selected.items())), None
async def _scan_request(request):
"""``({files, roots}, None)`` for a scan start, or ``(None, message)``.
``files`` is the scope ("new" or "all"); ``roots`` the selected source
indexes or None for every source. An empty body (also a chunked one with
no bytes), or one without ``scope``, asks for the default scope.
"""
default = {"files": DEFAULT_SCAN_SCOPE, "roots": None}
if not request.can_read_body:
return default, None
return DEFAULT_SCAN_SCOPE
text = await request.text()
if not text.strip(): # e.g. a chunked POST that sent no bytes
return default, None
return DEFAULT_SCAN_SCOPE
try:
body = json.loads(text)
except ValueError:
return None, _SCOPE_ERROR
return None
if not isinstance(body, dict):
return None, _SCOPE_ERROR
files = body.get("scope", DEFAULT_SCAN_SCOPE)
if files not in SCAN_SCOPES:
return None, _SCOPE_ERROR
roots, error = _scan_roots(body)
if error:
return None, error
return {"files": files, "roots": roots}, None
return None
scope = body.get("scope", DEFAULT_SCAN_SCOPE)
return scope if scope in SCAN_SCOPES else None
def busy_response(busy: JobBusy):
@@ -109,13 +64,6 @@ def _summary_event(job):
class JobRoutesMixin:
"""Mixin providing the /prompt_manager/jobs endpoints."""
def _roots_still_listed(self, selected, output_dirs):
"""Whether each ``{index: public path}`` still names that source."""
return all(
index < len(output_dirs) and self._public_path(output_dirs[index]) == path
for index, path in selected.items()
)
def _register_job_routes(self, routes):
@routes.get("/prompt_manager/jobs")
async def list_jobs_route(request):
@@ -184,37 +132,13 @@ class JobRoutesMixin:
"""Start the metadata scan as a job (202), or 409 naming the busy job.
The body's ``scope`` is "new" (the default: new and unresolved files)
or "all". ``roots``, when given, selects sources by their index in
the effective sources (``_get_all_output_dirs``), and ``paths`` names
the public path read for each. An index past them, or one whose
source now has another public path (the list changed since it was
read), is a 409 ``sources-changed`` and starts nothing. The job walks
the sources resolved here, so a later source change cannot move it.
or "all"; anything else is a 400.
"""
scan, error = await _scan_request(request)
if error:
return _bad_request(error)
output_dirs = None
roots = None
if scan["roots"] is not None:
output_dirs = await self._run_in_executor(self._get_all_output_dirs)
listed = await self._run_in_executor(
self._roots_still_listed, scan["roots"], output_dirs
)
if not listed:
return web.json_response(
{
"success": False,
"code": "sources-changed",
"error": SOURCES_CHANGED_ERROR,
},
status=409,
)
roots = list(scan["roots"])
files = await _scan_scope(request)
if files is None:
return _bad_request('The scan scope must be "new" or "all"')
try:
job = self._start_scan_job(
scan["files"], roots=roots, output_dirs=output_dirs
)
job = self._start_scan_job(files)
except JobBusy as busy:
return busy_response(busy)
return web.json_response({"success": True, "job": job.summary()}, status=202)
+3 -4
View File
@@ -435,8 +435,7 @@ class JobRegistry:
if same_kind is not None:
return same_kind
refused = _REFUSED_BESIDE[resolve_admission(admission, record_changing)]
# A snapshot: the monitor's worker thread asks too (Needs review).
running = [job for job in tuple(self._jobs.values()) if job.running]
running = [job for job in self._jobs.values() if job.running]
running.sort(key=lambda job: job._started_seq)
for job in running:
if job.admission in refused:
@@ -467,8 +466,8 @@ class JobRegistry:
return self._jobs.get(job_id)
def active(self, kind):
"""The running job of ``kind``, or None (safe off the event loop)."""
for job in tuple(self._jobs.values()):
"""The running job of ``kind``, or None."""
for job in self._jobs.values():
if job.kind == kind and job.running:
return job
return None
-689
View File
@@ -1,689 +0,0 @@
"""Needs review routes (#249 E3): the pending queue, item detail, decisions.
``GET /prompt_manager/review/items`` lists pending items oldest first behind
an HMAC keyset cursor bound to the storage epoch and the review index
revision (any decision, new item or restore makes an old cursor stale);
``total`` is the exact pending count. ``GET .../items/{id}`` adds the
candidates and the dependency fingerprint. ``POST .../items/{id}/decision``
records one explicit decision: Apply links the chosen sampler's pair only
when the capture is the item's bytes, the current file still hashes to them,
the evidence they yield is unchanged and the ingester's compatible-only rules
allow it; Leave unresolved links nothing.
Targets are formatted from the recorded spelling against the current roots
(no file access beyond the roots' real paths); a file under no current root
is ``unavailable``. Responses carry no server path, no locator and no
exception text.
"""
import base64
import binascii
import hashlib
import hmac
import json
import os
import re
from aiohttp import web
from .annotations import _HEX64, _TOKEN, _UUID, _capture_error, _error
from .prompts import bad_request
try:
from ...database.library import display_title
from ...database.library_commands import LibraryCommandError
from ...database.review_decisions import ReviewCommandError
from ...utils.media_capture import CaptureError
from ...utils.review_capture import captured_review, review_evidence
except ImportError:
from database.library import display_title
from database.library_commands import LibraryCommandError
from database.review_decisions import ReviewCommandError
from utils.media_capture import CaptureError
from utils.review_capture import captured_review, review_evidence
ITEMS_URL = "/prompt_manager/review/items"
DEFAULT_LIMIT = 40
MAX_LIMIT = 40
MAX_CURSOR = 4096
MAX_BODY_BYTES = 64 * 1024
# The serialized detail stays under this; past it the detail is ``limited``.
MAX_DETAIL_BYTES = 8 * 1024 * 1024
UNRESOLVED = "unresolved"
_DECISION_FIELDS = frozenset(
{
"expected_epoch",
"expected_revision",
"capture_token",
"sha256",
"dependency_fingerprint",
"choice",
}
)
_CURSOR_SECRET = os.urandom(32)
# The real path of a spelling; module-level so a test can fake a junction.
_realpath = os.path.realpath
_LIMIT = re.compile(r"[0-9]{1,3}")
_NO_STORE = {"Cache-Control": "no-store"}
_STATUS = {
"review-missing": 404,
"review-conflict": 409,
"association-conflict": 409,
"file-changed": 409,
"storage-changed": 409,
"source-changed": 409,
"capture-expired": 409,
}
_MESSAGES = {
"review-missing": "This review item no longer exists. Reload the queue.",
"review-conflict": (
"This item was decided or changed since it was opened. Reload the queue."
),
"association-conflict": (
"This image is already linked to another prompt or negative, so this "
"sampler path cannot be attached safely. The existing links were "
"preserved."
),
"dependencies-changed": (
"The image's saved links changed since this item was opened. Reopen it "
"to see the current association."
),
"file-changed": (
"The file changed since it was reviewed. A metadata scan adds its new "
"bytes as a new item."
),
"storage-changed": "Prompt storage was restored; reload the review queue.",
"source-changed": (
"The source changed. Refresh the image collection before reviewing this "
"file."
),
"capture-expired": "The preview expired. Open the item again.",
"other-file": "The preview belongs to another file. Open the item again.",
"evidence-changed": (
"This file's metadata no longer gives the evidence that was reviewed. "
"Reload the queue."
),
}
_DEFAULT_MESSAGE = "The decision could not be saved. Reload the queue."
def _coded(code: str, message_key: str | None = None, **extra) -> web.Response:
message = _MESSAGES.get(message_key or code, _DEFAULT_MESSAGE)
return _error(_STATUS.get(code, 409), code, message, **extra)
def _failed() -> web.Response:
return _error(500, "review-failed", "The review queue could not be read.")
def format_review_target(spelled_path, locator, real_roots, *, pathmod=os.path):
"""``(root_index, relative)`` of the first root that lists the file, or None.
``spelled_path`` is the file's recorded spelling, ``locator`` its folded
identity and ``real_roots`` ``(index, realpath)`` pairs. The relative
path keeps the recorded spelling (a Windows locator is case-folded, a
listing target is not) and must fold back to the locator.
"""
sep = pathmod.sep
for index, real_root in real_roots:
head = spelled_path[: len(real_root)]
if pathmod.normcase(head) != pathmod.normcase(real_root):
continue
rest = spelled_path[len(real_root) :]
if not real_root.endswith(sep):
if not rest.startswith(sep):
continue
rest = rest[1:]
parts = rest.split(sep)
if any(
part in ("", ".", "..") or "/" in part or "\\" in part or ":" in part
for part in parts
):
continue
if pathmod.normcase(pathmod.join(real_root, *parts)) != locator:
continue
return index, "/".join(parts)
return None
class _StaleCursor(ValueError):
"""A well-formed cursor this process cannot verify (e.g. after a restart)."""
def _encode_cursor(epoch: str, revision: int, key: tuple) -> str:
payload = json.dumps(
{"v": 1, "e": epoch, "r": revision, "k": list(key)},
separators=(",", ":"),
ensure_ascii=True,
).encode()
signature = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=")
def _decode_cursor(token: str) -> tuple:
"""``(epoch, revision, (created_at, id))`` of a cursor.
``ValueError`` when malformed; :class:`_StaleCursor` when well-formed but
signed by another secret (the per-process secret changes on every start,
so an open page's cursor is stale after a restart, not forged).
"""
try:
if len(token) > MAX_CURSOR:
raise ValueError()
raw = base64.b64decode(
token + "=" * (-len(token) % 4), altchars=b"-_", validate=True
)
signature, payload = raw[:32], raw[32:]
if len(signature) != 32 or not payload:
raise ValueError()
expected = hmac.new(_CURSOR_SECRET, payload, hashlib.sha256).digest()
if not hmac.compare_digest(signature, expected):
raise _StaleCursor("This page of the queue is out of date")
data = json.loads(payload)
key = data["k"]
if (
data["v"] != 1
or not isinstance(data["e"], str)
or type(data["r"]) is not int
or not isinstance(key, list)
or len(key) != 2
or not all(isinstance(part, str) for part in key)
):
raise ValueError()
return data["e"], data["r"], tuple(key)
except _StaleCursor:
raise
except (ValueError, KeyError, TypeError, UnicodeError, binascii.Error):
raise ValueError("Invalid review cursor; reload the queue") from None
def _limit(value):
if value is None:
return DEFAULT_LIMIT
if _LIMIT.fullmatch(value) is None or not 1 <= int(value) <= MAX_LIMIT:
return None
return int(value)
def _side_text(side) -> str | None:
if isinstance(side, dict) and isinstance(side.get("text"), str):
return side["text"]
return None
def _workable(candidate: dict) -> bool:
return (
candidate.get("state") == "resolved"
and _side_text(candidate.get("positive")) is not None
and _side_text(candidate.get("negative")) is not None
)
def _side_label(titles: dict, side, empty: str) -> str:
"""The Library title of the side's saved prompt, else its first line."""
text = _side_text(side)
if text is None:
return empty
stored = side.get("storage_text")
stored = stored if isinstance(stored, str) else text
return display_title(titles.get(stored, ()), text) or empty
class ReviewRoutesMixin:
"""The Needs review queue over E1's schema and E2's recorded items."""
def _register_review_routes(self, routes: web.RouteTableDef) -> None:
routes.get(ITEMS_URL)(self.list_review_items)
routes.get(ITEMS_URL + "/{item_id}")(self.get_review_item)
routes.post(ITEMS_URL + "/{item_id}/decision")(self.decide_review_item)
# -- shared ----------------------------------------------------------------
def _review_roots(self):
"""``(roots, real_roots)``: the current gallery roots and their realpaths.
Blocking (it lists the roots and resolves them): call it on a worker.
"""
roots = list(self._get_all_output_dirs())
return roots, [(i, _realpath(str(root))) for i, root in enumerate(roots)]
def _review_view(self, row: dict, roots, real_roots) -> dict:
"""The public ReviewItem of one stored row (no path, no locator)."""
found = format_review_target(row["spelled_path"], row["locator"], real_roots)
target = None
if found is not None:
index, relative = found
target = {
"root_index": index,
"root_scope": self._gallery_root_scope(roots[index], index),
"path": relative,
}
return {
"id": row["id"],
"state": row["state"],
"revision": row["revision"],
"created_at": row["created_at"],
"filename": os.path.basename(row["spelled_path"]),
"target": target,
"unavailable": target is None,
"candidate_count": row["candidate_count"],
"limited": row["limited"],
}
@staticmethod
def _review_image_path(row: dict, roots, target) -> str:
"""The path the scan and the monitor link this file under."""
if target is None:
return row["spelled_path"]
root = roots[target["root_index"]]
return os.path.join(str(root), *target["path"].split("/"))
# -- queue -----------------------------------------------------------------
async def list_review_items(self, request: web.Request) -> web.Response:
"""GET ``?limit=1..40&cursor=``: one page of pending items."""
limit = _limit(request.query.get("limit"))
if limit is None:
return bad_request(f"limit must be an integer from 1 to {MAX_LIMIT}")
bound = None
cursor = request.query.get("cursor")
if cursor:
try:
bound = _decode_cursor(cursor)
except _StaleCursor:
return await self._stale_cursor()
except ValueError as exc:
return bad_request(str(exc))
try:
page, views = await self._run_in_executor(
self._review_page, limit, bound[2] if bound else None
)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
if bound is not None and bound[0] != page["epoch"]:
return _coded("storage-changed", library_epoch=page["epoch"])
if bound is not None and bound[1] != page["revision"]:
return _coded("review-conflict", total=page["total"])
items = page["items"]
next_cursor = None
if page["more"] and items:
last = items[-1]
next_cursor = _encode_cursor(
page["epoch"], page["revision"], (last["created_at"], last["id"])
)
return web.json_response(
{
"success": True,
"library_epoch": page["epoch"],
"total": page["total"],
"items": views,
"next_cursor": next_cursor,
},
headers=_NO_STORE,
)
async def _stale_cursor(self) -> web.Response:
"""409 ``review-conflict`` with the current total: reload the queue."""
try:
page = await self._run_in_executor(self.db.review_queue_page, limit=1)
except Exception:
self.logger.exception("Review queue read failed")
return _failed()
return _coded(
"review-conflict", total=page["total"], library_epoch=page["epoch"]
)
def _review_page(self, limit: int, after):
"""Worker thread: one queue page and the public views of its items."""
page = self.db.review_queue_page(limit=limit, after=after)
roots, real_roots = self._review_roots()
return page, [
self._review_view(row, roots, real_roots) for row in page["items"]
]
# -- detail ----------------------------------------------------------------
async def get_review_item(self, request: web.Request) -> web.Response:
"""GET one item with its candidates and dependency fingerprint."""
try:
read = await self._run_in_executor(
self._review_detail, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if read is None:
return _coded("review-missing")
item, view, fingerprint, candidates = read
detail = {
"success": True,
"library_epoch": item["epoch"],
**view,
"candidates": candidates,
"dependency_fingerprint": fingerprint,
"sha256": item["sha256"],
"total": item["total"],
}
if len(json.dumps(detail).encode("utf-8")) > MAX_DETAIL_BYTES:
detail = {**detail, "candidates": [], "limited": True}
return web.json_response(detail, headers=_NO_STORE)
def _review_detail(self, item_id: str):
"""Worker thread: ``(item, view, fingerprint, candidates)`` or None."""
item = self.db.review_item(item_id)
if item is None:
return None
roots, real_roots = self._review_roots()
view = self._review_view(item, roots, real_roots)
image_path = self._review_image_path(item, roots, view["target"])
aliases, _known = physical_spellings(self.db, image_path, item["locator"])
fingerprint = self.db.review_dependency_fingerprint(image_path, aliases)
return item, view, fingerprint, self._review_candidates(item)
def _review_candidates(self, item: dict) -> list:
"""The detail's candidates: role texts, a label and selectability."""
if item["limited"]:
return []
candidates = [
c for c in item["evidence"].get("candidates", []) if isinstance(c, dict)
]
texts = set()
for candidate in candidates:
for role in ("positive", "negative"):
side = candidate.get(role)
if isinstance(side, dict):
for key in ("text", "storage_text"):
if isinstance(side.get(key), str):
texts.add(side[key])
titles = self.db.stored_prompt_titles(texts)
rows = []
for candidate in candidates:
positive = _side_text(candidate.get("positive"))
negative = _side_text(candidate.get("negative"))
label = (
_side_label(titles, candidate.get("positive"), "No positive")
+ " / "
+ _side_label(titles, candidate.get("negative"), "No negative")
)
rows.append(
{
"hash": candidate.get("hash"),
"sampler_id": candidate.get("sampler_id"),
"state": "resolved" if _workable(candidate) else "unresolved",
"reason": candidate.get("reason") or "",
"label": label,
"positive": {"text": positive} if positive is not None else None,
"negative": {"text": negative} if negative is not None else None,
}
)
return rows
# -- decision --------------------------------------------------------------
async def decide_review_item(self, request: web.Request) -> web.Response:
"""POST one explicit decision on one pending item."""
data, error = await self._read_json_body(request, max_bytes=MAX_BODY_BYTES)
if error is not None:
return error
guards, error = _decision_guards(data)
if error is not None:
return error
try:
item = await self._run_in_executor(
self.db.review_item, request.match_info["item_id"]
)
except Exception:
self.logger.exception("Review item read failed")
return _failed()
if item is None:
return _coded("review-missing")
choice = guards["choice"]
candidate = None
if choice != UNRESOLVED:
candidate = next(
(
c
for c in item["evidence"].get("candidates", [])
if isinstance(c, dict) and c.get("hash") == choice
),
None,
)
if candidate is None or item["limited"]:
return bad_request("choice is not one of this item's candidates")
if not _workable(candidate):
return bad_request("This sampler path cannot be applied")
if guards["capture_token"] is None:
return bad_request("Apply needs a capture of the file")
verified = None
if guards["capture_token"] is not None:
verified, error = await self._verified_review_capture(
item, guards, need_metadata=candidate is not None
)
if error is not None:
return error
elif guards["sha256"] != item["sha256"]:
return bad_request("sha256 does not match the item")
return await self._record_decision(item, guards, candidate, verified)
async def _verified_review_capture(self, item, guards, *, need_metadata):
"""``((capture, facts, image_path, metadata), None)`` or ``(None, error)``.
The capture must be the item's file and bytes, under the expected
epoch and a current root; the current file must re-hash to the same
bytes; for Apply, the capture's metadata must still yield exactly the
reviewed evidence.
"""
store = self._captures
capture = store.get(guards["capture_token"]) if store is not None else None
if capture is None:
return None, _coded("capture-expired")
if capture.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if guards["sha256"] != capture.sha256:
return None, bad_request("sha256 does not match the preview")
if capture.sha256 != item["sha256"]:
return None, _coded("file-changed")
if guards["expected_epoch"] != capture.epoch:
current = await self._run_in_executor(self.db.get_execution_epoch)
return None, _coded("storage-changed", library_epoch=current)
try:
root, error = await self._run_in_executor(
self._scoped_root, capture.root_index, capture.scope
)
if error is not None:
return None, error
read = await self._run_in_executor(
_verify_and_read, store, capture, item, root, need_metadata
)
except CaptureError as exc:
return None, _capture_error(exc)
if read is None:
return None, _coded("source-changed")
relative, facts, captured = read
if facts.locator != item["locator"]:
return None, _coded("capture-expired", "other-file")
if facts.sha256 != item["sha256"]:
return None, _coded("file-changed")
metadata = None
if need_metadata:
if captured is None or not _candidate_still_offered(
captured[1], guards["choice"]
):
return None, _coded("review-conflict", "evidence-changed")
metadata = captured[0]
image_path = os.path.join(str(root), *relative.split("/"))
return (capture, facts, image_path, metadata), None
def _decide_sync(self, item, guards, candidate, verified):
"""Worker thread: resolve the file's spellings, decide, build the view.
Returns ``(result, view)``; a refusal raises with ``exc.view`` set.
"""
_capture, facts, image_path, metadata = verified or (None, None, None, None)
aliases, known = (), None
if candidate is not None:
aliases, known = physical_spellings(self.db, image_path, item["locator"])
try:
result = self.db.decide_review_item(
item_id=item["id"],
expected_epoch=guards["expected_epoch"],
expected_revision=guards["expected_revision"],
# The bytes the re-hash proved, or the body's for Leave unresolved.
sha256=facts.sha256 if facts is not None else guards["sha256"],
outcome="selected" if candidate is not None else UNRESOLVED,
candidate=candidate,
image_path=image_path,
metadata=metadata,
expected_fingerprint=guards["dependency_fingerprint"],
aliases=aliases,
known_rows=known,
)
except ReviewCommandError as exc:
if exc.item is not None:
roots, real_roots = self._review_roots()
exc.view = self._review_view(exc.item, roots, real_roots)
raise
roots, real_roots = self._review_roots()
return result, self._review_view(result["item"], roots, real_roots)
async def _record_decision(self, item, guards, candidate, verified):
capture = verified[0] if verified else None
try:
result, view = await self._run_in_executor(
self._decide_sync, item, guards, candidate, verified
)
except ReviewCommandError as exc:
return self._decision_refused(exc)
except LibraryCommandError as exc:
return _coded(exc.code, library_epoch=exc.epoch)
except Exception:
self.logger.exception("Review decision failed")
return _error(500, "decision-failed", "The decision could not be saved.")
if capture is not None:
# The decision used the capture; the page returns to Images.
self._captures.release(capture.token)
body = {
"success": True,
"library_epoch": guards["expected_epoch"],
"item": view,
"total": result["total"],
}
if result["pair"] is not None:
body["pair"] = result["pair"]
return web.json_response(body, headers=_NO_STORE)
def _decision_refused(self, exc: ReviewCommandError) -> web.Response:
extra = {"library_epoch": exc.epoch}
if exc.total is not None:
extra["total"] = exc.total
if exc.view is not None:
extra["item"] = exc.view
message = None
if exc.fingerprint is not None:
extra["dependency_fingerprint"] = exc.fingerprint
if exc.code == "association-conflict" and exc.changed:
message = "dependencies-changed"
return _coded(exc.code, message, **extra)
def _decision_guards(data: dict):
"""The validated decision fields, or ``(None, error)``."""
if set(data) != _DECISION_FIELDS:
return None, bad_request(
"Request must contain exactly " + ", ".join(sorted(_DECISION_FIELDS))
)
epoch, revision = data["expected_epoch"], data["expected_revision"]
if not isinstance(epoch, str) or _UUID.fullmatch(epoch) is None:
return None, bad_request("expected_epoch must be a canonical UUID")
if type(revision) is not int or revision < 0:
return None, bad_request("expected_revision must be a non-negative integer")
token = data["capture_token"]
if token is not None and (
not isinstance(token, str) or _TOKEN.fullmatch(token) is None
):
return None, bad_request("Invalid capture token")
for name in ("sha256", "dependency_fingerprint"):
value = data[name]
if not isinstance(value, str) or _HEX64.fullmatch(value) is None:
return None, bad_request(
f"{name} must be 64 lowercase hexadecimal characters"
)
choice = data["choice"]
if not isinstance(choice, str) or (
choice != UNRESOLVED and _HEX64.fullmatch(choice) is None
):
return None, bad_request("choice must be a candidate hash or unresolved")
return {**data}, None
def _verify_and_read(store, capture, item, root, need_metadata: bool):
"""Worker thread: ``(relative, facts, captured)``, or None off the root.
Formats the item's target under the capture's root (resolving the root's
real path), re-hashes the current file, then reads the capture.
"""
found = format_review_target(
item["spelled_path"],
item["locator"],
[(capture.root_index, _realpath(str(root)))],
)
if found is None:
return None
relative = found[1]
facts = store.verify(root, relative)
captured = captured_review(store, capture) if need_metadata else None
return relative, facts, captured
def _same_file(spelled: str, image_path: str, locator: str) -> bool:
"""True when ``spelled`` reaches the physical file at ``locator``."""
try:
if os.path.normcase(_realpath(spelled)) == locator:
return True
return os.path.samefile(spelled, image_path)
except (OSError, ValueError):
return False
def physical_spellings(db, image_path: str, locator: str):
"""``(aliases, known_rows)`` for the legacy rows of one physical file.
The roots are real paths, while the monitor links a file under the
directory it watches, so a symlinked or junctioned root, a ``subst`` or a
mapped drive spells the same file otherwise. Every row whose spelling
ends with the file's name (the link summaries' fallback lookup) is a
candidate; ``aliases`` are the candidates that resolve to ``locator``
(``normcase(realpath)``, as the capture forms it) or are the same file,
and ``known_rows`` every candidate id, so a decision can refuse a
same-name row it did not classify. Blocking: call it on a worker.
"""
rows = db.review_same_name_rows(os.path.basename(image_path))
aliases = tuple(
spelled
for _row_id, spelled in rows
if isinstance(spelled, str)
and spelled != image_path
and _same_file(spelled, image_path, locator)
)
return tuple(dict.fromkeys(aliases)), frozenset(row_id for row_id, _ in rows)
def _candidate_still_offered(resolution: dict, choice: str) -> bool:
"""True when the captured bytes still offer the chosen candidate.
Only what the decision depends on is re-checked: the bytes are still
``ambiguous`` and the chosen candidate (its hash covers its sampler,
sides and provenance) is still among their candidates. A resolver change
that rewords the top-level reason or reorders candidates leaves a
pending item applicable.
"""
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
return False
evidence, _count, limited = review_evidence(resolution)
if limited:
return False
return any(c.get("hash") == choice for c in evidence.get("candidates", ()))
-241
View File
@@ -1,241 +0,0 @@
"""Needs review recording for the output scan and the live monitor (#249).
The rule (maintainer test of #284/#288): a file is queued only when its
final outcome is ambiguous and unlinked. The strict resolver must find it
``ambiguous``, the scan's legacy fallback (the browser parser,
``_graph_prompt_text``) must not link it, and no saved row may link it
already. A file the fallback links, an exact pair the ingester stored and an
already linked file are never queued. Neither is an ambiguous file with
nothing to choose: fewer than two of its candidates carry a prompt
(``utils.review_capture.reviewable``); the scan reports it unresolved ("could
not be traced").
The queue heals itself: a pending item whose file is now linked (by the
fallback, a later resolution or the monitor) is removed by the next scan
that lists the file or by the monitor's next pass over it, and so is a
pending item without a choice whose file is still ambiguous without one
(``settle_untraced_review_items``). Only an undecided item goes
(``pending``, revision 0, no decision row); a decision is never deleted or
changed (``settle_linked_review_items``).
The scan records through ``_record_scanned_review`` (best effort: a failure
is logged and counted); the monitor calls ``record_review_file`` from its
worker thread after its own link. Both go through
``utils.review_capture.record_ambiguous_file`` on the API's capture store.
Nothing here links a file.
Recording changes records, so it follows the registry's ``records``
admission (#289). The scan records under its own ``records`` job. The
monitor's pass lasts one file and is not a job: it is refused while a
``records`` or ``guarded`` job runs (``deferred``), and the next scan reads
the file again.
"""
import os
from pathlib import Path
from .admin import SCAN_LOG_PREFIXES, _plural, _scan_location, _scan_roots
# The kind the monitor's recording checks admission under (never registered).
REVIEW_RECORDING_KIND = "review_recording"
# record_ambiguous_file outcomes that leave the file in the pending queue.
QUEUED_OUTCOMES = ("recorded", "limited", "duplicate")
NEW_OUTCOMES = ("recorded", "limited")
try:
from ...utils.review_capture import record_ambiguous_file
except ImportError:
from utils.review_capture import record_ambiguous_file
# The scan log's reason for items that leave the queue with nothing to choose.
NOTHING_TO_CHOOSE = "nothing to choose"
class ReviewRecordingMixin:
"""Record the scan's and the monitor's ambiguous files for Needs review."""
def _record_scanned_review(self, media_file, roots, review=None):
"""Record one ambiguous, unlinked scanned file (logged and counted)."""
try:
outcome = self._record_review_sync(media_file, roots, review)
except Exception:
self.logger.warning(
f"{media_file.name} could not be added to Needs review",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return
if review is None:
return
if outcome in NEW_OUTCOMES:
review["added"] += 1
if outcome in QUEUED_OUTCOMES:
review["queued"] += 1
def _record_review_sync(self, media_file, roots, review=None):
"""``record_ambiguous_file`` for a file under one of ``roots`` (blocking).
An ``untraced`` file (its captured bytes offer nothing to choose) also
loses its pending items that offer nothing either.
"""
located = self._review_location(media_file, roots)
if located is None:
return "skipped"
root, index, relative, _spelled = located
outcome = record_ambiguous_file(
self.db, self._capture_store(), Path(root), index, relative
)
if outcome == "untraced":
self._settle_untraced_reviews([media_file], roots, review)
return outcome
@staticmethod
def _review_location(media_file, roots, real_roots=None):
"""``(root, root_index, relative, spelled)`` of a listed file, or None.
``spelled`` is the capture's spelling of the file (the real root joined
with the relative path); ``os.path.normcase`` of it is the locator
review items are stored under. ``real_roots`` caches each root's
``realpath`` across a batch.
"""
relative, _label, index = _scan_location(media_file, roots)
root = next((r for r, _l, i in roots if i == index), None)
if index is None or root is None:
return None
if real_roots is None:
real_roots = {}
real = real_roots.get(index)
if real is None:
real = real_roots[index] = os.path.realpath(os.fspath(root))
spelled = os.path.join(real, *relative.split("/"))
return root, index, relative, spelled
def _review_file_linked(self, media_file, roots):
"""True when a saved row links the scanned file (any spelling)."""
located = self._review_location(media_file, roots)
spellings = (str(media_file),) + ((located[3],) if located else ())
return self.db.review_file_linked(spellings)
def _settle_scanned_reviews(self, media_files, roots, review=None):
"""Remove the pending items of ``media_files`` that are now linked.
Blocking; best effort like recording (a failure is logged and
counted in ``review["errors"]``). Returns how many items left.
"""
real_roots = {}
files = []
for media_file in media_files:
located = self._review_location(media_file, roots, real_roots)
if located is not None:
spelled = located[3]
files.append((os.path.normcase(spelled), (str(media_file), spelled)))
try:
settled = self.db.settle_linked_review_items(files) if files else 0
except Exception:
self.logger.warning(
"Needs review items of linked files could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["settled"] += settled
return settled
def _settle_untraced_reviews(self, media_files, roots, review=None):
"""Remove the pending items without a choice of untraced ``media_files``.
``media_files`` are files found ambiguous with fewer than two
candidates that carry a prompt. Blocking; best effort like the linked
settle (a failure is logged and counted in ``review["errors"]``).
Returns how many items left.
"""
locators = []
for media_file in media_files:
located = self._review_location(media_file, roots)
if located is not None:
locators.append(os.path.normcase(located[3]))
try:
removed = self.db.settle_untraced_review_items(locators) if locators else 0
except Exception:
self.logger.warning(
"Needs review items without a choice could not be removed",
exc_info=True,
)
if review is not None:
review["errors"] += 1
return 0
if review is not None:
review["untraced"] += removed
return removed
def _fallback_links(self, metadata):
"""True when the scan's legacy fallback would link these records."""
return bool(metadata.get("prompt")) and bool(self._graph_prompt_text(metadata))
def record_review_file(self, image_path):
"""Apply the Needs review rule to one monitor output (its worker thread).
Maps ``image_path`` to the gallery root that lists it; a file under no
current root is ``skipped``. A linked file is never queued: its
pending items are removed and the answer is ``linked``. An unlinked
file the scan's fallback would link is ``skipped`` (the next scan
links it); any other file is recorded when it is ambiguous with a
choice, and is ``untraced`` (its pending items without a choice
removed) when it is ambiguous without one. While a
job runs that a ``records`` job may not run beside, nothing is written
and the answer is ``deferred``. Errors propagate: the monitor logs
them and keeps its link.
"""
busy = self._jobs.blocker(REVIEW_RECORDING_KIND, admission="records")
if busy is not None:
self.logger.info(
f"Needs review was not updated for "
f"{os.path.basename(str(image_path))} while {busy.title} runs; "
"the next scan reads it again"
)
return "deferred"
full = os.path.abspath(str(image_path))
located = Path(os.path.realpath(os.path.dirname(full))) / os.path.basename(full)
roots = _scan_roots(self._get_all_output_dirs())
if self._review_location(located, roots) is None:
return "skipped"
if self._review_file_linked(located, roots):
self._settle_scanned_reviews([located], roots)
return "linked"
if self._fallback_links(self._extract_comfyui_metadata(str(located))):
return "skipped"
return self._record_review_sync(located, roots)
@staticmethod
def _log_review_items(job, review):
"""The scan log's Needs review summary lines (none when nothing moved)."""
prefix = SCAN_LOG_PREFIXES["review"]
if review["added"]:
job.log_line(
f"{prefix} {_plural(review['added'], 'image')} added to Needs review"
)
already = review["queued"] - review["added"]
if already:
job.log_line(
f"{prefix} {_plural(already, 'image')} already in Needs review"
)
if review["settled"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['settled'], 'image')} left Needs review · now linked"
)
if review["untraced"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['known']} "
f"{_plural(review['untraced'], 'image')} left Needs review · "
f"{NOTHING_TO_CHOOSE}"
)
if review["errors"]:
job.log_line(
f"{SCAN_LOG_PREFIXES['errors']} "
f"{_plural(review['errors'], 'image')} could not be added to "
"Needs review"
)
+31 -246
View File
@@ -4,15 +4,8 @@ A save validates the payload, commits it to config.json through the config
store and publishes it at runtime inside that commit; a save that cannot be
written changes nothing at runtime and says so. ``_config_commit_failure``
is shared with the LoRA integration's save.
GET also reports the image sources and the monitor (``sources.py``). A save
that changes the sources runs as a ``source_change`` registry hold, so it is
refused while a record-changing job (a scan) runs. ``monitoring_enabled``
takes effect at the next restart. ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing.
"""
import contextlib
import os
from aiohttp import web
@@ -22,10 +15,7 @@ from ..config_store import (
RevisionConflict,
get_config_store,
)
from . import sources
from .admin import _current_image_monitor
from .job_routes import busy_response
from .jobs import JobBusy
try:
from ...utils.validators import validate_result_timeout, validate_worker_threads
@@ -33,75 +23,13 @@ except ImportError:
from utils.validators import validate_result_timeout, validate_worker_threads
# Settings owned by POST /settings: config attribute -> its keys in config.json.
# Settings owned by POST /settings: config attribute -> (section, key).
_SETTINGS_FILE_KEYS = {
"RESULT_TIMEOUT": ("web_ui", "result_timeout"),
"WEBUI_DISPLAY_MODE": ("web_ui", "webui_display_mode"),
"INFINITE_SCROLL": ("web_ui", "infinite_scroll"),
"WORKER_THREADS": ("performance", "worker_threads"),
"MONITORING_ENABLED": ("gallery", "monitoring", "enabled"),
}
# Attributes that live on GalleryConfig; the others on PromptManagerConfig.
_GALLERY_SETTINGS = frozenset({"MONITORING_ENABLED"})
SOURCE_CHANGE_JOB_KIND = "source_change"
SOURCE_CHANGE_JOB_TITLE = "Source change"
def _source_identity(path):
"""The case-folded spelling two roots are the same directory by (Windows)."""
return os.path.normcase(path)
def _stat_identity(path):
"""``(st_dev, st_ino)`` of an existing directory, or None.
Catches case variants on a case-insensitive filesystem whose spelling
normcase keeps (macOS).
"""
try:
stat = os.stat(path)
except (OSError, ValueError):
return None
return (stat.st_dev, stat.st_ino) if stat.st_ino else None
# Refusals that a reconnected drive or folder can lift.
_RECONNECTABLE = frozenset({"not-found", "not-directory"})
def _root_key(root):
"""The directory a configured root names (resolved), else its spelling."""
from ..config import GalleryConfig
try:
return GalleryConfig.resolve_gallery_root(root)
except (OSError, TypeError, ValueError, AttributeError):
return root
def _request_error(message):
return {"code": "invalid-request", "error": message}
def _invalid_source(index, entry, code, reason):
"""The 400 body for the source at ``index`` that cannot be used.
Names it by its public spelling (computed as text: a refused path is
never resolved), so the page can mark that row and say why.
"""
source = sources.public_spelling(entry)
advice = (
"Remove it or reconnect it, then save."
if code in _RECONNECTABLE
else "Remove it, then save."
)
return {
"code": "invalid-source",
"source": source,
"index": index,
"reason": reason,
"error": f"{source} can't be used: {reason}. {advice}",
}
def _section(config, name):
@@ -119,8 +47,10 @@ def _settings_mutation(updates, new_roots):
def mutate(config):
for name, value in updates.items():
*sections, key = _SETTINGS_FILE_KEYS[name]
_set_nested(config, sections, key, value)
section, key = _SETTINGS_FILE_KEYS[name]
merged = _section(config, section)
merged[key] = value
config[section] = merged
if new_roots is not None:
gallery = _section(config, "gallery")
monitoring = _section(gallery, "monitoring")
@@ -132,16 +62,6 @@ def _settings_mutation(updates, new_roots):
return mutate
def _set_nested(config, sections, key, value):
"""``config[s1][s2]...[key] = value``, each section a fresh dict."""
if not sections:
config[key] = value
return
merged = _section(config, sections[0])
_set_nested(merged, sections[1:], key, value)
config[sections[0]] = merged
async def _read_json(request):
"""Parse a JSON object body through the package's capped reader."""
from . import _read_json_body
@@ -161,10 +81,6 @@ class SettingsRoutesMixin:
async def save_settings_route(request):
return await self.save_settings(request)
@routes.post("/prompt_manager/settings/validate-source")
async def validate_source_route(request):
return await self.validate_source(request)
async def get_settings(self, request):
"""Get current settings."""
try:
@@ -174,15 +90,11 @@ class SettingsRoutesMixin:
self._public_path(d) for d in GalleryConfig.MONITORING_DIRECTORIES
]
revision = await self._run_in_executor(self._config_revision)
source_status, monitor = await self._run_in_executor(self._source_status)
return web.json_response(
{
"success": True,
**revision,
"sources": source_status,
"monitor": monitor,
"settings": {
"monitoring_enabled": GalleryConfig.MONITORING_ENABLED,
"result_timeout": PromptManagerConfig.RESULT_TIMEOUT,
"webui_display_mode": PromptManagerConfig.WEBUI_DISPLAY_MODE,
"infinite_scroll": PromptManagerConfig.INFINITE_SCROLL,
@@ -203,61 +115,6 @@ class SettingsRoutesMixin:
status=500,
)
def _source_status(self):
"""``(sources, monitor)`` of GET /settings (blocking)."""
from ..config import GalleryConfig
output_dirs, auto_detected = self._output_dirs_with_origin()
return sources.describe_sources(
configured=list(GalleryConfig.MONITORING_DIRECTORIES),
output_dirs=output_dirs,
auto_detected=auto_detected,
monitor=_current_image_monitor(),
enabled=GalleryConfig.MONITORING_ENABLED,
)
async def validate_source(self, request):
"""Check one source path ``{path}``; writes nothing.
200 ``{success, valid, label, path, code, reason}``: a valid source
answers its label and public path, an invalid one its code and reason
only. A body without a string ``path`` is a 400 ``invalid-request``.
"""
data, error_response = await _read_json(request)
if error_response is not None:
too_large = error_response.status == 413
return web.json_response(
{
"success": False,
"code": "too-large" if too_large else "invalid-request",
"error": (
"Request body too large"
if too_large
else "Request body must be a JSON object"
),
},
status=error_response.status,
)
path = data.get("path")
if not isinstance(path, str):
return web.json_response(
{
"success": False,
"code": "invalid-request",
"error": "path must be a string",
},
status=400,
)
try:
result = await self._run_in_executor(sources.validate_source, path)
except Exception:
self.logger.exception("Source validation failed")
return web.json_response(
{"success": False, "code": None, "error": "Failed to check the source"},
status=500,
)
return web.json_response({"success": True, **result})
def _monitored_directories(self):
"""Directories the running image monitor watches, else the configured roots."""
from ..config import GalleryConfig
@@ -294,15 +151,9 @@ class SettingsRoutesMixin:
{"success": False, "error": str(ve)}, status=400
)
# Off the loop: an accepted root is resolved (filesystem access).
new_roots, error = await self._run_in_executor(
self._parse_gallery_roots, data
)
new_roots, error = self._parse_gallery_roots(data)
if error:
return web.json_response({"success": False, **error}, status=400)
roots_change = new_roots is not None and await self._run_in_executor(
self._roots_change, new_roots
)
return web.json_response({"success": False, "error": error}, status=400)
published = {}
@@ -311,30 +162,15 @@ class SettingsRoutesMixin:
# they were written, so the runtime ends on the last one.
published.update(self._publish_settings(updates, new_roots))
async with contextlib.AsyncExitStack() as stack:
hold = None
if roots_change:
try:
hold = await stack.enter_async_context(
self._jobs.hold(
SOURCE_CHANGE_JOB_KIND,
SOURCE_CHANGE_JOB_TITLE,
admission="records",
)
)
except JobBusy as busy:
return busy_response(busy)
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
if hold is not None:
hold.finish("failed")
return self._config_commit_failure(exc)
try:
_saved, revision = await self._run_in_executor(
get_config_store().commit,
_settings_mutation(updates, new_roots),
expected_revision=expected_revision,
publish=publish,
)
except (ConfigUnreadable, RevisionConflict, OSError) as exc:
return self._config_commit_failure(exc)
return web.json_response(
{
@@ -360,11 +196,10 @@ class SettingsRoutesMixin:
as ``runtime_applied: false`` with the fields that need a restart;
nothing is retried or rolled back. Never raises.
"""
from ..config import GalleryConfig, PromptManagerConfig
from ..config import PromptManagerConfig
for name, value in updates.items():
owner = GalleryConfig if name in _GALLERY_SETTINGS else PromptManagerConfig
setattr(owner, name, value)
setattr(PromptManagerConfig, name, value)
restart_fields = []
runtime_applied = True
if new_roots is not None:
@@ -377,40 +212,11 @@ class SettingsRoutesMixin:
)
runtime_applied = False
restart_fields.append("gallery_root_paths")
if "MONITORING_ENABLED" in updates:
# The monitor starts with the server: a restart applies the setting
# unless the observer already matches it.
if updates["MONITORING_ENABLED"] != self._observer_alive():
restart_fields.append("monitoring_enabled")
return {
"runtime_applied": runtime_applied,
"restart_required_fields": restart_fields,
}
@staticmethod
def _roots_change(new_roots):
"""Whether a save changes the sources (blocking: it resolves them).
Compared as directories, not spellings: the legacy page resends the
roots resolved, while config.json may hold them as hand-written
(relative, another case), and an unchanged resend is no change.
"""
from ..config import GalleryConfig
if new_roots is None:
return False
return [_root_key(root) for root in new_roots] != [
_root_key(root) for root in GalleryConfig.MONITORING_DIRECTORIES
]
@staticmethod
def _observer_alive():
"""Whether the image monitor's observer is running now. Never raises."""
try:
return sources.monitor_state(_current_image_monitor())[1]
except Exception:
return False
def _config_commit_failure(self, exc):
"""The response for a config commit that wrote nothing."""
not_saved = {"success": False, "persisted": False, "runtime_applied": False}
@@ -477,10 +283,6 @@ class SettingsRoutesMixin:
if not isinstance(data["infinite_scroll"], bool):
raise ValueError("infinite_scroll must be true or false")
updates["INFINITE_SCROLL"] = data["infinite_scroll"]
if "monitoring_enabled" in data:
if not isinstance(data["monitoring_enabled"], bool):
raise ValueError("monitoring_enabled must be true or false")
updates["MONITORING_ENABLED"] = data["monitoring_enabled"]
if "worker_threads" in data:
updates["WORKER_THREADS"] = validate_worker_threads(
data["worker_threads"], PromptManagerConfig.max_worker_threads()
@@ -493,21 +295,15 @@ class SettingsRoutesMixin:
Accepts ``gallery_root_paths`` (list, preferred) or the legacy single
``gallery_root_path`` string. Blank entries are dropped.
Each entry passes the same checks as validate-source
(``sources.source_problem``): the form, then containment as text,
before anything is resolved. Blocking (an accepted root is resolved).
Returns:
(roots, error): ``roots`` is the validated list, or ``None`` when
the payload carries no gallery root key; ``error`` is the coded
body of the 400 when validation failed (``roots`` is then None).
An unusable source is ``invalid-source`` naming its public path
and its position in the list.
the payload carries no gallery root key; ``error`` is a message
when validation failed (and ``roots`` is then ``None``).
"""
if "gallery_root_paths" in data:
raw = data["gallery_root_paths"]
if not isinstance(raw, list):
return None, _request_error("gallery_root_paths must be a list")
return None, "gallery_root_paths must be a list"
elif "gallery_root_path" in data:
raw = [data["gallery_root_path"]]
else:
@@ -516,38 +312,27 @@ class SettingsRoutesMixin:
from ..config import GalleryConfig
roots = []
seen = set()
for index, entry in enumerate(raw):
for entry in raw:
if not isinstance(entry, str):
return None, _request_error("Gallery root paths must be strings")
return None, "Gallery root paths must be strings"
entry = entry.strip()
if not entry:
continue
code, reason = sources.source_problem(entry)
if code is not None:
return None, _invalid_source(index, entry, code, reason)
root = GalleryConfig.resolve_gallery_root(entry)
keys = {("path", _source_identity(root)), ("file", _stat_identity(root))}
keys.discard(("file", None))
if keys & seen:
continue # another spelling of a root already listed
seen |= keys
roots.append(root)
ok, reason = GalleryConfig.validate_gallery_root(entry)
if not ok:
return (
None,
f"Invalid gallery root '{os.path.basename(entry)}': {reason}",
)
roots.append(GalleryConfig.resolve_gallery_root(entry))
return roots, None
def _apply_gallery_roots(self, roots):
"""Install validated gallery roots; returns True when they changed.
Another spelling of the same directories installs the resolved
spelling and is no change (nothing to restart for).
"""
"""Install validated gallery roots; returns True when they changed."""
from ..config import GalleryConfig
if roots == list(GalleryConfig.MONITORING_DIRECTORIES):
return False
if not self._roots_change(roots):
GalleryConfig.MONITORING_DIRECTORIES = roots
return False
GalleryConfig.MONITORING_DIRECTORIES = roots
self._cached_output_dir = None
self.reset_gallery_roots_cache()
-280
View File
@@ -1,280 +0,0 @@
"""Image sources as System › Sources & scans shows them (#249 S4).
- ``source_form_error`` is the lexical check a submitted source path passes
before any filesystem access: no control characters, no ``..`` part, and
only the native path forms (on Windows a drive or a UNC share, never the
``\\\\?\\`` and ``\\\\.\\`` namespaces, a drive-relative path, an alternate
data stream, a device name or a part Windows would silently trim; on POSIX
no Windows drive or backslash form).
- ``source_problem`` is the one refusal every caller answers (validate-source
and the settings save): the form check, then
``GalleryConfig.gallery_root_problem``, which decides containment as text
before it touches the filesystem.
- ``describe_sources`` (blocking: it resolves paths) reports the configured
sources with their validity, the effective sources with their listing
index and watch state, and the image monitor's state. Every path in it is
a public path (``_public_path``), never an absolute one.
A source's watch state is ``watching`` when the monitor's observer is alive,
scheduled that directory and monitoring is on; ``stopping`` when it still
watches it but monitoring was switched off (it stops at the next restart);
``manual`` when monitoring is off or this process has no monitor; else
``restart_pending`` (a source added since the monitor started, monitoring
switched on, or an observer that stopped).
"""
import logging
import ntpath
import os
import re
logger = logging.getLogger("prompt_manager.api")
MAX_SOURCE_PATH = 4096 # characters
FORM_REASONS = {
"empty": "Gallery root must be a non-empty path",
"invalid-form": "Gallery root is not a folder path this server can use",
"traversal": "Gallery root must not contain '..'",
}
_DRIVE = re.compile(r"[A-Za-z]:")
_POSIX_FOREIGN = re.compile(r"^[A-Za-z]:")
# Reserved device names: Windows opens the device for any folder of that name.
_DEVICE_NAME = re.compile(r"^(con|prn|aux|nul|com[0-9¹²³]|lpt[0-9¹²³])(\..*)?$", re.I)
# folder_paths getters and the label of the directory each one reports.
_COMFYUI_LABELS = (
("get_output_directory", "ComfyUI output"),
("get_input_directory", "ComfyUI input"),
("get_temp_directory", "ComfyUI temp"),
("get_user_directory", "ComfyUI user"),
)
WATCH_STATES = ("watching", "stopping", "manual", "restart_pending")
def source_form_error(text, *, windows=None):
"""The form code that refuses ``text`` as a source path, or None.
``windows`` selects the path rules (default: this OS). No filesystem
access happens here.
"""
if windows is None:
windows = os.name == "nt"
if not isinstance(text, str) or len(text) > MAX_SOURCE_PATH:
return "invalid-form"
if any(ord(char) < 32 or ord(char) == 127 for char in text):
return "invalid-form"
text = text.strip()
if not text:
return "empty"
return _windows_form_error(text) if windows else _posix_form_error(text)
def _posix_form_error(text):
if "\\" in text or _POSIX_FOREIGN.match(text):
return "invalid-form" # a Windows path names nothing on this server
return "traversal" if ".." in text.split("/") else None
def _windows_form_error(text):
portable = text.replace("/", "\\")
if portable.startswith(("\\\\?\\", "\\\\.\\")):
return "invalid-form" # verbatim and device namespaces
if portable.startswith("\\\\"):
drive, rest = ntpath.splitdrive(portable)
share = drive[2:].split("\\")
if len(share) != 2 or any(part in ("", ".", "..") for part in share):
return "invalid-form"
else:
drive, rest = ntpath.splitdrive(portable)
if drive and (not _DRIVE.fullmatch(drive) or not rest.startswith("\\")):
return "invalid-form" # "C:folder" is relative to that drive's cwd
if ":" in rest:
return "invalid-form" # an alternate data stream
parts = rest.split("\\")
if ".." in parts:
return "traversal"
for part in parts:
if part in ("", "."):
continue
if part[-1] in ". " or _DEVICE_NAME.match(part):
return "invalid-form"
return None
def source_problem(text):
"""``(code, reason)`` that refuses ``text`` as a source, else ``(None, "")``.
The form check runs first and touches nothing; then containment is
decided as text before any filesystem access (see
``GalleryConfig.gallery_root_problem``). Every caller answers alike.
"""
from ..config import GalleryConfig
code = source_form_error(text)
if code is not None:
return code, FORM_REASONS[code]
return GalleryConfig.gallery_root_problem(text)
def public_spelling(text):
"""How a refused source is named in a response, computed as text only.
Relative to the path anchor it lies under (its public path), else its
last part: the caller sent the path, so naming it reveals nothing, and a
refused path is never resolved to name it.
"""
from ..config import GalleryConfig, _lexical_path, _lexically_inside
text = str(text).strip()
try:
lexical = os.path.normpath(text)
key = _lexical_path(text) # the same length as ``lexical``
if not os.path.isabs(text):
return lexical.replace("\\", "/")
for anchor in GalleryConfig.path_anchors():
if _lexically_inside(key, [anchor]):
rest = lexical[len(anchor.rstrip(os.sep)) :].lstrip(os.sep)
return rest.replace(os.sep, "/") or "."
return os.path.basename(lexical) or lexical
except (TypeError, ValueError):
return ""
def identity(path):
"""The comparison key of a directory: resolved and case-folded (Windows)."""
return os.path.normcase(os.path.realpath(str(path)))
def _comfyui_labels():
"""``{identity: label}`` of the directories ComfyUI reports."""
try:
import folder_paths
except ImportError:
return {}
labels = {}
for getter_name, label in _COMFYUI_LABELS:
getter = getattr(folder_paths, getter_name, None)
if getter is None:
continue
try:
directory = getter()
except Exception:
continue
if isinstance(directory, str) and directory:
labels.setdefault(identity(directory), label)
return labels
def source_label(path, labels=None):
""" "ComfyUI output" for ComfyUI's own directories, else the folder name."""
if labels is None:
labels = _comfyui_labels()
label = labels.get(identity(path))
if label:
return label
return os.path.basename(os.path.normpath(str(path))) or str(path)
def _configured(entry, labels, public_path):
from ..config import GalleryConfig
code, reason = GalleryConfig.gallery_root_problem(entry)
try:
resolved = GalleryConfig.resolve_gallery_root(entry)
except (OSError, TypeError, ValueError):
resolved = str(entry)
return {
"label": source_label(resolved, labels),
"path": public_path(resolved),
"valid": code is None,
"code": code,
"reason": reason or None,
}
def monitor_state(monitor):
"""``(present, observer_alive, scheduled directories)`` of the image monitor."""
if monitor is None:
return False, False, []
status = monitor.get_status()
scheduled = [str(d) for d in status.get("monitored_directories") or []]
return True, bool(status.get("observer_alive")), scheduled
def watch_state(path, *, present, alive, scheduled_keys, enabled):
"""One of ``WATCH_STATES`` (see the module docstring)."""
if alive and identity(path) in scheduled_keys:
return "watching" if enabled else "stopping"
if not present or not enabled:
return "manual"
return "restart_pending"
def describe_sources(*, configured, output_dirs, auto_detected, monitor, enabled):
"""``(sources, monitor)`` for GET /settings (blocking; public paths only).
``configured`` is the configured roots, ``output_dirs`` and
``auto_detected`` what ``_output_dirs_with_origin`` returned, ``monitor``
the running ImageMonitor or None, ``enabled`` the monitoring setting.
"""
from . import _public_path
labels = _comfyui_labels()
present, alive, scheduled = monitor_state(monitor)
scheduled_keys = {identity(path) for path in scheduled}
effective = [
{
"label": source_label(path, labels),
"path": _public_path(path),
"root_index": index,
"watch": watch_state(
path,
present=present,
alive=alive,
scheduled_keys=scheduled_keys,
enabled=enabled,
),
}
for index, path in enumerate(output_dirs)
]
return (
{
"configured": [
_configured(entry, labels, _public_path) for entry in configured
],
"effective": effective,
"auto_detected": bool(auto_detected),
},
{
"present": present,
"observer_alive": alive,
"scheduled": [_public_path(path) for path in scheduled],
},
)
def validate_source(text):
"""``{valid, label, path, code, reason}`` for one submitted path (blocking).
Writes nothing. An invalid path answers its code and reason with no
label or path, so a response never echoes what was typed.
"""
from ..config import GalleryConfig
from . import _public_path
code, reason = source_problem(text)
if code is not None:
return {
"valid": False,
"label": None,
"path": None,
"code": code,
"reason": reason,
}
resolved = GalleryConfig.resolve_gallery_root(text)
return {
"valid": True,
"label": source_label(resolved),
"path": _public_path(resolved),
"code": None,
"reason": None,
}
+6 -20
View File
@@ -9,10 +9,9 @@
kept.
- ``GET /prompt_manager/system/overview`` is the System › Overview aggregate.
Every value is real or null: there is no backup status until backups report
one and no health result until a check ran; ``needs_review.pending`` is the
Needs review queue's pending total. The missing-thumbnail count is not part
of it: the page reads ``/images/thumbnails/status``, which walks every
source.
one, no health result until a check ran, and no Needs-review count until
the review queue exists. The missing-thumbnail count is not part of it:
the page reads ``/images/thumbnails/status``, which walks every source.
- ``GET /prompt_manager/system/statistics`` lists runs and ratings per
category, with the legacy ``/stats`` totals and the Overview's count of
rated images (images whose linked prompt is rated).
@@ -41,19 +40,6 @@ _HEALTH_STATUS_LINES = {
_StoredReport = namedtuple("_StoredReport", "generation report")
async def _gather_settled(*reads):
"""Await every read, then raise the first failure.
Unlike a plain gather, a failure never answers while the other executor
reads still run against the database.
"""
results = await asyncio.gather(*reads, return_exceptions=True)
for result in results:
if isinstance(result, BaseException):
raise result
return results
def _failure(message, status=500, code=None):
return web.json_response(
{"success": False, "code": code, "error": message}, status=status
@@ -168,7 +154,7 @@ class SystemRoutesMixin:
from ..config import GalleryConfig
try:
(output_dirs, auto_detected), counts = await _gather_settled(
(output_dirs, auto_detected), counts = await asyncio.gather(
self._run_in_executor(self._output_dirs_with_origin),
self._run_in_executor(self.db.library_counts),
)
@@ -188,7 +174,7 @@ class SystemRoutesMixin:
},
"health": health_summary(self._current_health_report()),
"backup": None, # no verified backup status exists yet (S8)
"needs_review": {"pending": counts["review_pending"]},
"needs_review": {"pending": None}, # no review queue yet (Q6)
"sources": {
"configured": len(GalleryConfig.MONITORING_DIRECTORIES),
"effective": len(output_dirs),
@@ -202,7 +188,7 @@ class SystemRoutesMixin:
async def get_system_statistics(self, request):
try:
by_category, totals, counts = await _gather_settled(
by_category, totals, counts = await asyncio.gather(
self._run_in_executor(self.db.usage_by_category),
self._run_in_executor(self.db.get_statistics),
self._run_in_executor(self.db.library_counts),
+25 -121
View File
@@ -31,6 +31,7 @@ routes = server_instance.routes
extension_uri = None # Will be set in __init__.py
import os # noqa: E402
from pathlib import Path # noqa: E402
from typing import Any, Dict, List, Optional, Tuple # noqa: E402
# Import logging system
@@ -50,14 +51,6 @@ config_logger = get_logger("prompt_manager.config")
# may be used as gallery roots in addition to ComfyUI's own directories.
EXTRA_GALLERY_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
# The one answer for every path outside the allowed folders. The filesystem
# root and the home directory answer it too, so no caller learns the server's
# layout (home path, user name) from a refusal (#249 S4).
OUTSIDE_ALLOWED_REASON = (
"Gallery root must be inside a ComfyUI directory (output, input, temp, user) "
f"or one listed in {EXTRA_GALLERY_ROOTS_ENV}"
)
# Environment variable overriding where config.json is read from and saved to.
CONFIG_PATH_ENV = "PROMPT_MANAGER_CONFIG_PATH"
@@ -96,29 +89,6 @@ def write_private_json(path: str, data: Any) -> None:
os.chmod(path, 0o600)
def _lexical_path(path: str) -> str:
"""``path`` normalised for comparison without touching the filesystem.
normpath collapses ``.`` and ``..`` as text and normcase folds case on
Windows; neither opens, stats nor resolves anything (no SMB request for a
UNC path). A symlink is only seen by :func:`_canonical_path` later.
"""
return os.path.normcase(os.path.normpath(path))
def _lexically_inside(path: str, parents: List[str]) -> bool:
"""Whether the normalised ``path`` is one of ``parents`` or below one.
Pure text on normalised spellings (:func:`_lexical_path` or
:func:`_canonical_path`): no filesystem access.
"""
sep = os.path.sep
return any(
path == parent or path.startswith(parent.rstrip(sep) + sep)
for parent in parents
)
def _is_filesystem_root(path: str) -> bool:
"""True for '/' on POSIX and drive roots such as 'C:\\' on Windows."""
return os.path.dirname(path) == path
@@ -298,12 +268,7 @@ class GalleryConfig:
"""
monitoring = new_config.get("monitoring", {})
if "enabled" in monitoring:
if isinstance(monitoring["enabled"], bool):
cls.MONITORING_ENABLED = monitoring["enabled"]
else: # a hand-edited "false" would otherwise switch it on
config_logger.warning(
"Ignoring gallery monitoring enabled: expected true or false"
)
cls.MONITORING_ENABLED = monitoring["enabled"]
if "directories" in monitoring:
directories = cls._validated_directories(monitoring["directories"])
if directories is not None:
@@ -372,56 +337,31 @@ class GalleryConfig:
Filesystem roots are never allowed as parents, even when listed in
the environment, because that would re-open the whole disk.
"""
return [canonical for canonical, _spelling in cls._allowed_parent_pairs()]
@classmethod
def _allowed_parent_pairs(cls) -> List[Tuple[str, str]]:
"""``(canonical, lexical spelling)`` of each allowed parent.
Only trusted directories (ComfyUI's own and the environment's) are
resolved here; the spelling is how ComfyUI or the environment wrote
the directory, so a submitted path can be compared without resolving it.
"""
pairs = []
seen = set()
parents = []
for candidate in _comfyui_directories() + _extra_gallery_roots():
try:
canonical = _canonical_path(candidate)
spelling = _lexical_path(os.path.abspath(candidate))
except (OSError, ValueError):
continue
if _is_filesystem_root(canonical) or not os.path.isdir(canonical):
continue
if canonical not in seen:
seen.add(canonical)
pairs.append((canonical, spelling))
return pairs
if canonical not in parents:
parents.append(canonical)
return parents
@classmethod
def path_anchors(cls) -> List[str]:
"""Canonical directories that relative gallery paths are resolved against.
The ComfyUI base directory comes first, followed by the parent of each
ComfyUI directory outside it (``--output-directory D:\\Renders``) and of
each directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so such a
directory is addressed by its own name (``Renders/sub``) rather than by
an absolute path, and two folders of one name stay apart. The same
anchors drive the public (relative) form of paths in API responses.
directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS, so an extra
root is addressed by its own name (``gallery/sub``) rather than by an
absolute path. The same anchors drive the public (relative) form of
paths in API responses.
"""
anchors = []
base = _comfyui_base_directory()
candidates = [base] if base else []
try:
base_key = _canonical_path(base) if base else None
except (OSError, ValueError):
base_key = None
for directory in _comfyui_directories():
try:
canonical = _canonical_path(directory)
except (OSError, ValueError):
continue
if base_key is None or not _lexically_inside(canonical, [base_key]):
candidates.append(os.path.dirname(canonical))
candidates.extend(os.path.dirname(root) for root in _extra_gallery_roots())
for candidate in candidates:
try:
@@ -459,69 +399,33 @@ class GalleryConfig:
(True, "") when the directory lies inside one of
:meth:`allowed_gallery_parents`; otherwise (False, reason).
"""
code, reason = cls.gallery_root_problem(path)
return code is None, reason
@classmethod
def gallery_root_problem(cls, path: Any) -> Tuple[Optional[str], str]:
"""``(code, reason)`` why ``path`` may not be a gallery root, else (None, "").
Codes: ``empty``, ``unresolvable``, ``outside-allowed``, ``not-found``
and ``not-directory``. No reason names the path.
Containment is decided twice. First lexically, with no filesystem
access at all: a path (UNC, ``\\\\?\\``, a device, anything) whose
normalised spelling lies outside every allowed parent answers
``outside-allowed`` before it is resolved, so the server never opens
or stats it (no SMB connection, no existence or timing oracle). Only
then is it resolved and checked again, which catches a symlink out.
The filesystem root and the home directory answer ``outside-allowed``
too: one answer for every caller (validate-source, the settings save,
config.json), and the detail goes to the log only.
"""
if not isinstance(path, str) or not path.strip():
return "empty", "Gallery root must be a non-empty path"
pairs = cls._allowed_parent_pairs()
spellings = [spelling for pair in pairs for spelling in pair]
try:
lexical = cls._lexical_candidates(path)
except (TypeError, ValueError):
return "unresolvable", "Gallery root could not be resolved"
if not any(_lexically_inside(candidate, spellings) for candidate in lexical):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
return False, "Gallery root must be a non-empty path"
try:
canonical = cls.resolve_gallery_root(path)
except (OSError, ValueError):
return "unresolvable", "Gallery root could not be resolved"
return False, "Gallery root could not be resolved"
if _is_filesystem_root(canonical):
config_logger.info("Gallery root refused: a filesystem root")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
return False, "Gallery root cannot be a filesystem root"
if canonical == _canonical_path(os.path.expanduser("~")):
config_logger.info("Gallery root refused: the home directory")
return "outside-allowed", OUTSIDE_ALLOWED_REASON
if not _lexically_inside(canonical, [canonical for canonical, _ in pairs]):
return "outside-allowed", OUTSIDE_ALLOWED_REASON
return False, "Gallery root cannot be the home directory"
if not os.path.exists(canonical):
return "not-found", "Gallery root does not exist"
return False, "Gallery root does not exist"
if not os.path.isdir(canonical):
return "not-directory", "Gallery root is not a directory"
return None, ""
return False, "Gallery root is not a directory"
@classmethod
def _lexical_candidates(cls, path: str) -> List[str]:
"""Every spelling ``path`` may resolve to, normalised as text (no IO).
candidate = Path(canonical)
for parent in cls.allowed_gallery_parents():
if candidate.is_relative_to(Path(parent)):
return True, ""
An absolute path is itself; a relative one is joined to each
:meth:`path_anchors` entry, as :meth:`resolve_gallery_root` tries them.
"""
path = path.strip()
if os.path.isabs(path):
return [_lexical_path(path)]
anchors = cls.path_anchors()
return [_lexical_path(os.path.join(anchor, path)) for anchor in anchors]
return (
False,
"Gallery root must be inside a ComfyUI directory "
f"(output, input, temp, user) or one listed in {EXTRA_GALLERY_ROOTS_ENV}",
)
class IntegrationConfig:
-4
View File
@@ -226,8 +226,6 @@ def create_app(root: Path) -> web.Application:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -593,8 +591,6 @@ def create_app(root: Path) -> web.Application:
return web.json_response("dark")
async def cleanup(application: web.Application) -> None:
# A scan of an ambiguous fixture file captures it for Needs review.
api.close_annotation_captures()
api.db.close_all()
seed()
+4 -373
View File
@@ -169,9 +169,8 @@ function mapOverviewConcept(reference, overview) {
const relabel = { 'Database integrity': 'Not checked', 'Positive prompts': number(overview.library.positive_prompts),
'Negative variants': number(overview.library.negative_variants), Images: number(overview.images.indexed) };
const rows = {
// D93: the real pending count; the action is unavailable (dimmed, as the Save row's D92) until the review page.
'3 ambiguous image links': row => ({ ...row,
title: `${number(overview.needs_review.pending)} ambiguous image link${overview.needs_review.pending === 1 ? '' : 's'}`,
// D93 (Q6): no review queue yet, so no count and the action is unavailable (dimmed, as the Save row's D92).
'3 ambiguous image links': row => ({ ...row, title: 'Ambiguous image links', description: 'Not available yet.',
action: { ...row.action, opacity: '0.55', cursor: 'not-allowed' } }),
// D93: no backup has been recorded (verified backups arrive with S8).
'Backup status': row => ({ ...row, description: 'No retained backup record.' }),
@@ -231,12 +230,12 @@ for (const theme of THEMES) for (const [width, height] of SIZES) {
});
}
// The recorded differences, by path: D93 the review row (its real count, dimmed action), the backup row, the
// The recorded differences, by path: D93 the review row (title, Not available yet, dimmed action), the backup row, the
// preferences row and the sources row, and metric 4; D96 the statistics row; D92 the chip, the status and the dimmed
// Save; D122 the nav label. (The fixture's first three metrics are the concept's own counts.)
const UNMAPPED_PATHS = [
'columns.surfaces.0.rows.0.action.cursor', 'columns.surfaces.0.rows.0.action.opacity',
'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.0.rows.0.description', 'columns.surfaces.0.rows.0.title', 'columns.surfaces.0.rows.1.description',
'columns.surfaces.1.rows.0.description', 'columns.surfaces.1.rows.1.title', 'columns.surfaces.1.rows.2.description',
'heading.actions.0', 'heading.actions.length', 'metrics.items.3.value', 'nav.label',
'save.button.cursor', 'save.button.opacity', 'save.status.text',
@@ -409,371 +408,3 @@ for (const theme of THEMES) test(`statistics and health dialogs match the concep
await page.evaluate(() => { document.querySelector('#modal-body caption').textContent = 'Runs per category'; });
expect(diffInventory(expectedStatistics, await page.evaluate(measureDialog)).map(item => item.path)).toEqual(['tables.0.caption']);
});
/** Runs in the page: Appearance's surface, every row's text and control, and the concept's row geometry and tokens. */
function measureAppearance(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const surface = layout.querySelector(':scope > section.surface');
const control = row => {
const select = row.querySelector(':scope > select');
if (select) return { kind: 'select', id: select.id, options: [...select.options].map(option => option.textContent),
selected: select.selectedOptions[0]?.textContent ?? null };
const input = row.querySelector(':scope > input');
if (input) return { kind: `input.${input.type}`, id: input.id, value: input.value };
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const chip = row.querySelector(':scope > .chip');
return chip ? { kind: name(chip), text: text(chip), ...css(chip, 'color', 'backgroundColor', 'fontSize') } : null;
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
const operations = [...surface.querySelectorAll(':scope > .operation')];
const firstRow = rows[0];
return {
layout: [...layout.children].map(name),
surface: { tree: [...surface.children].map(name), heading: text(surface.querySelector(':scope > h2')),
intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius') },
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: operations.map(row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), action: text(row.querySelector('.row-actions > button')) })),
row: css(firstRow, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopWidth',
'borderTopColor', 'marginTop'),
label: css(firstRow.querySelector('label'), 'fontSize', 'fontWeight', 'color'),
description: css(firstRow.querySelector('p'), 'fontSize', 'color', 'marginTop', 'maxWidth'),
select: css(firstRow.querySelector('select'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor', 'color', 'borderTopColor'),
heading3: css(surface.querySelector('.setting-row h3'), 'fontSize', 'fontWeight', 'color'),
number: css(surface.querySelector('.setting-row input[type="number"]'), 'fontSize', 'minWidth', 'maxWidth', 'backgroundColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D97: the rows Appearance adds for the legacy gallery settings, after the concept's own, as production renders them.
const ADDED_APPEARANCE_ROWS = [
['card-details', 'Show image details on cards', 'Show each card’s title and file details under the preview.', true],
['viewer-details', 'Collapse viewer details by default', 'Open the image viewer with its details panel hidden.', false],
];
/** The concept's Appearance with the recorded deviations applied, given the server's real settings. */
function mapAppearanceConcept(reference, settings) {
const toggleRow = reference.rows.find(row => row.control?.kind === 'switch');
const added = ADDED_APPEARANCE_ROWS.map(([id, label, description, checked]) => ({
tree: toggleRow.tree, text: [['label', label, id], ['p', description, null]],
control: { ...toggleRow.control, id, checked } }));
return {
...reference,
// D97: two added toggles and the "Reset display defaults" operation close the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.setting-row', 'div.setting-row', 'div.operation'] },
rows: [...reference.rows.map(row => row.control?.id === 'timeout'
// The node search timeout is the server's real `result_timeout`, not the concept's sample 30.
? { ...row, control: { ...row.control, value: String(settings.result_timeout) } } : row), ...added],
operations: [{ tree: ['div', 'div.row-actions'], title: 'Reset display defaults',
description: 'Restore this browser’s display defaults, then save. Other settings are kept.', action: 'Reset' }],
};
}
async function conceptAppearance(page, theme) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Appearance', exact: true }).click();
await expect(page.locator('#main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#main');
}
async function productionAppearance(page, theme) {
await openProduction(page, 'system/appearance', theme);
await expect(page.locator('#workspace-main #timeout')).toBeEnabled();
await page.mouse.move(0, 0);
return page.evaluate(measureAppearance, '#workspace-main');
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`appearance matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-appearance-${theme}-${width}.png`), fullPage: true });
const production = await productionAppearance(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-appearance-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapAppearanceConcept(concept, settings), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('appearance comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const { settings } = await serverJson(request, '/prompt_manager/settings');
const concept = await conceptAppearance(page, 'dark');
const expected = mapAppearanceConcept(concept, settings);
const production = await productionAppearance(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D97 and the real timeout are recorded.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.0', 'operations.length', 'rows.11', 'rows.12', 'rows.13', 'rows.length', 'surface.tree'].sort());
const measure = () => page.evaluate(measureAppearance, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed row.
const undoRow = await mutate(page, { remove: ['#workspace-main .setting-row:nth-of-type(3)'] });
expect(await paths()).toEqual(expect.arrayContaining(['rows.length', 'surface.tree.length']));
await undoRow();
// A changed label and a changed option.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail sizes'; });
expect(await paths()).toEqual(['rows.1.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="density"]').textContent = 'Thumbnail size'; });
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Medium'; });
expect(await paths()).toEqual(expect.arrayContaining(['rows.5.control.options.0']));
await page.evaluate(() => { document.querySelector('#workspace-main #image-quality option').textContent = 'Optimized'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['row.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: Sources & scans (app.js 679–681): the optional notice, then the "Image sources" surface with its
* panel heading, the source list, the two setting rows and the operations, with the concept's geometry and tokens.
*/
function measureSources(mainSelector) {
const layout = document.querySelector(`${mainSelector} .settings-layout`);
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const button = node => ({ text: text(node), className: node.className,
...css(node, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') });
const operation = row => ({ tree: [...row.children].map(name), title: text(row.querySelector('h3')),
description: text(row.querySelector('p')), actions: [...row.querySelectorAll('.row-actions > button')].map(button) });
const notice = layout.querySelector(':scope > .notice');
const surface = layout.querySelector(':scope > section.surface');
const heading = surface.querySelector(':scope > .panel-heading');
const list = surface.querySelector(':scope > #source-list');
const control = row => {
const toggle = row.querySelector(':scope > * input[type="checkbox"]');
if (toggle) return { kind: 'switch', id: toggle.id, wrapper: name(toggle.parentElement), checked: toggle.checked,
...css(toggle.parentElement, 'width', 'height') };
const input = row.querySelector(':scope > input');
return { kind: `input.${input.type}`, id: input.id, min: input.min, max: input.max, value: input.value,
...css(input, 'width', 'accentColor') };
};
const rows = [...surface.querySelectorAll(':scope > .setting-row')];
return {
layout: [...layout.children].map(name),
notice: notice && { tree: [...notice.children].map(name),
text: [...notice.querySelectorAll(':scope > div > *')].map(node => [node.localName, text(node)]),
actions: [...notice.querySelectorAll(':scope > .row-actions > button')].map(button),
...css(notice, 'display', 'alignItems', 'columnGap', 'borderTopColor', 'backgroundColor', 'paddingTop', 'marginBottom') },
surface: { tree: [...surface.children].map(name), intro: text(surface.querySelector(':scope > p')),
...css(surface, 'paddingTop', 'paddingLeft', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
heading: { tree: [...heading.children].map(name), title: text(heading.querySelector('h2')),
action: button(heading.querySelector('button')), ...css(heading, 'display', 'alignItems', 'justifyContent', 'marginBottom'),
h2: css(heading.querySelector('h2'), 'fontSize', 'fontWeight', 'color') },
sources: [...list.children].map(operation),
rows: rows.map(row => ({ tree: [...row.children].map(name), text: [...row.firstElementChild.children].map(node => [node.localName, text(node),
node.localName === 'label' ? node.htmlFor : null]), control: control(row) })),
operations: [...surface.querySelectorAll(':scope > .operation')].map(operation),
operation: css(list.querySelector('.operation'), 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop',
'paddingBottom', 'borderTopWidth', 'borderTopColor'),
h3: css(list.querySelector('.operation h3'), 'fontSize', 'fontWeight', 'color'),
p: css(list.querySelector('.operation p'), 'fontSize', 'color', 'marginTop'),
row: css(rows[0], 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', 'borderTopColor'),
pageScroll: document.documentElement.scrollWidth > document.documentElement.clientWidth,
};
}
// D128: "watching until restart" after monitoring was switched off while the observer still runs.
const WATCH_TEXT = { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan', restart_pending: 'restart pending' };
const WATCHED = new Set(['watching', 'stopping']);
// D98: the operation Sources adds after the concept's own, as production renders it.
const REFRESH_FILES = { title: 'Refresh files', description: 'Read every source again to find files added or removed outside ComfyUI.',
label: 'Refresh' };
/** The concept's Sources with the recorded deviations applied, given the server's real settings and sources. */
function mapSourcesConcept(reference, { settings, sources }) {
const [sample] = reference.sources;
const scan = sample.actions[0];
return {
...reference,
// The real sources replace the two samples (without the "sample path" note). D98: each configured one has Remove,
// a plain button after its scan.
sources: sources.effective.map(source => ({ ...sample, title: source.label,
description: `${source.path} · ${WATCH_TEXT[source.watch]}`,
actions: [{ ...scan, text: WATCHED.has(source.watch) ? 'Scan changes' : 'Scan folder' }, { ...scan, text: 'Remove' }] })),
rows: reference.rows.map(row => {
if (row.control.id === 'monitoring') return { ...row, control: { ...row.control, checked: settings.monitoring_enabled } };
// D113: the range ends at the server's core count (the concept's fixed 16) and shows the real worker count.
return { ...row, control: { ...row.control, max: String(settings.cpu_count), value: String(settings.worker_threads) } };
}),
// D98: "Refresh files" closes the surface.
surface: { ...reference.surface, tree: [...reference.surface.tree, 'div.operation'] },
operations: [...reference.operations, { ...reference.operations[0], title: REFRESH_FILES.title,
description: REFRESH_FILES.description, actions: [{ ...reference.operations[0].actions[0], text: REFRESH_FILES.label }] }],
};
}
async function conceptSources(page, theme, { notice }) {
await openConcept(page, 'system', theme);
await page.locator('#main .section-nav').getByRole('button', { name: 'Sources & scans', exact: true }).click();
await expect(page.locator('#main .panel-heading > h2')).toHaveText('Image sources');
if (!notice) await page.locator('#main .notice').getByRole('button', { name: 'Later', exact: true }).click();
await expect(page.locator('#main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#main');
}
async function productionSources(page, theme, { notice }) {
// about:blank has no storage; the workspace page reads the dismissal.
if (!notice) await page.addInitScript(() => { if (location.protocol.startsWith('http')) localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
await openProduction(page, 'system/sources', theme);
await expect(page.locator('#workspace-main #source-list .operation')).toHaveCount(2);
await expect(page.locator('#workspace-main #workers')).toBeEnabled();
await expect(page.locator('#workspace-main .settings-layout > .notice')).toHaveCount(notice ? 1 : 0);
await page.mouse.move(0, 0);
return page.evaluate(measureSources, '#workspace-main');
}
/** The fixture's sources as the concept shows its samples: the first one watched, the second not. */
async function watchedSources(request) {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
return serverJson(request, '/prompt_manager/settings');
}
for (const theme of THEMES) for (const [width, height] of SIZES) for (const notice of [true, false]) {
const variant = notice ? 'with' : 'without';
test(`sources matches the concept ${variant} the notice ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
const server = await watchedSources(request);
const concept = await conceptSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`concept-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
const production = await productionSources(page, theme, { notice });
await page.screenshot({ path: testInfo.outputPath(`production-sources-${variant}-notice-${theme}-${width}.png`), fullPage: true });
expect(diffInventory(mapSourcesConcept(concept, server), production)).toEqual([]);
expect(production.pageScroll).toBe(false);
});
}
test('sources comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const server = await watchedSources(request);
const concept = await conceptSources(page, 'dark', { notice: true });
const expected = mapSourcesConcept(concept, server);
const production = await productionSources(page, 'dark', { notice: true });
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only in the real sources, the D98 operation and the D113 range.
expect(diffInventory(concept, production).map(item => item.path.split('.').slice(0, 2).join('.'))
.filter((path, index, all) => all.indexOf(path) === index).sort())
.toEqual(['operations.2', 'operations.length', 'rows.1', 'sources.0', 'sources.1', 'surface.tree'].sort());
const measure = () => page.evaluate(measureSources, '#workspace-main');
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
// A removed notice and a removed source row.
const undoNotice = await mutate(page, { remove: ['#workspace-main .settings-layout > .notice'] });
expect(await paths()).toEqual(expect.arrayContaining(['layout.length', 'notice.tree.length']));
await undoNotice();
const undoRow = await mutate(page, { remove: ['#workspace-main #source-list > .operation:nth-child(2)'] });
expect(await paths()).toEqual(expect.arrayContaining(['sources.length']));
await undoRow();
// A changed label and a changed button.
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for outputs'; });
expect(await paths()).toEqual(['rows.0.text.0.1']);
await page.evaluate(() => { document.querySelector('#workspace-main label[for="monitoring"]').textContent = 'Watch for new outputs'; });
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = 'Add source'; });
expect(await paths()).toEqual(['heading.action.text']);
await page.evaluate(() => { document.querySelector('#workspace-main .panel-heading button').textContent = '+ Add source'; });
// A swapped token.
const undoToken = await mutate(page, { css: '#workspace-main .settings-layout { --line: rgb(255, 0, 0); }' });
expect(await paths()).toEqual(expect.arrayContaining(['operation.borderTopColor']));
await undoToken();
expect(diffInventory(expected, await measure())).toEqual([]);
});
/**
* Runs in the page: the open "Add image source" dialog (the concept's `simpleInput`, app.js 1138 and 1497): its title,
* the body's tree, the field's label and input, and the actions, with the concept's geometry and tokens.
*/
function measureAddSource() {
const dialog = document.querySelector('dialog#modal[open]');
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
const name = node => [node.localName, ...[...node.classList].sort()].join('.');
const text = node => node?.textContent.trim() ?? null;
const body = dialog.querySelector('#modal-body');
const field = body.querySelector(':scope > .field');
const label = field?.querySelector(':scope > label');
const input = field?.querySelector(':scope > input');
const actions = body.querySelector(':scope > .dialog-actions');
const rect = dialog.getBoundingClientRect();
return {
title: text(dialog.querySelector('#modal-title')),
tree: [...body.children].map(name),
field: field && { tree: [...field.children].map(name), ...css(field, 'display', 'flexDirection', 'rowGap', 'marginBottom') },
label: label && { text: text(label), for: label.htmlFor, ...css(label, 'fontSize', 'fontWeight', 'color') },
input: input && { id: input.id, type: input.type, value: input.value,
...css(input, 'height', 'paddingLeft', 'fontSize', 'color', 'backgroundColor', 'borderTopColor', 'borderTopLeftRadius') },
actions: { tree: [...actions.children].map(name), ...css(actions, 'display', 'justifyContent', 'columnGap', 'marginTop'),
buttons: [...actions.children].map(button => ({ text: text(button), className: button.className,
...css(button, 'backgroundColor', 'color', 'fontSize', 'paddingTop', 'paddingLeft', 'borderTopColor') })) },
width: Math.round(rect.width),
};
}
/** D127: the field is "Folder path", starts empty (no sample folder) and its action is "Add". */
function mapAddSourceConcept(reference) {
const [cancel, apply] = reference.actions.buttons;
return { ...reference, label: { ...reference.label, text: 'Folder path' }, input: { ...reference.input, value: '' },
actions: { ...reference.actions, buttons: [cancel, { ...apply, text: 'Add' }] } };
}
async function conceptAddSource(page, theme) {
await conceptSources(page, theme, { notice: false });
await page.locator('#main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
async function productionAddSource(page, theme) {
await productionSources(page, theme, { notice: false });
await page.locator('#workspace-main .panel-heading').getByRole('button', { name: '+ Add source', exact: true }).click();
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Add image source');
await page.mouse.move(0, 0);
return page.evaluate(measureAddSource);
}
for (const theme of THEMES) for (const [width, height] of SIZES) {
test(`add source dialog matches the concept ${theme} ${width}x${height}`, async ({ page, request }, testInfo) => {
await page.setViewportSize({ width, height });
await watchedSources(request);
const concept = await conceptAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`concept-add-source-${theme}-${width}.png`) });
const production = await productionAddSource(page, theme);
await page.screenshot({ path: testInfo.outputPath(`production-add-source-${theme}-${width}.png`) });
expect(diffInventory(mapAddSourceConcept(concept), production)).toEqual([]);
});
}
test('add source comparison detects mutations', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
await watchedSources(request);
const concept = await conceptAddSource(page, 'dark');
const expected = mapAddSourceConcept(concept);
const production = await productionAddSource(page, 'dark');
expect(diffInventory(expected, production)).toEqual([]);
// Unmapped, the concept differs only where D127 is recorded.
expect(diffInventory(concept, production).map(item => item.path).sort())
.toEqual(['actions.buttons.1.text', 'input.value', 'label.text']);
const paths = async () => diffInventory(expected, await page.evaluate(measureAddSource)).map(item => item.path);
// A changed label, a swapped action order and a changed style.
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder'; });
expect(await paths()).toEqual(['label.text']);
await page.evaluate(() => { document.querySelector('dialog#modal[open] label[for="action-input"]').textContent = 'Folder path'; });
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
expect(await paths()).toEqual(expect.arrayContaining(['actions.buttons.0.text', 'actions.buttons.1.text']));
await page.evaluate(() => { const actions = document.querySelector('dialog#modal[open] .dialog-actions'); actions.prepend(actions.lastElementChild); });
const undoStyle = await mutate(page, { css: 'dialog#modal #action-input { border-color: rgb(255, 0, 0); } dialog#modal .dialog-actions { gap: 31px; }' });
expect(await paths()).toEqual(expect.arrayContaining(['input.borderTopColor', 'actions.columnGap']));
await undoStyle();
expect(diffInventory(expected, await page.evaluate(measureAddSource))).toEqual([]);
});
+24 -76
View File
@@ -354,48 +354,39 @@ test('prompt deletion requires confirmation and can retry after a network failur
}
});
// The duplicate scan reads the output root only, but the fixture's input root holds a different
// shared/same.png. A bare /images/serve/<path> URL tries every gallery root in turn, so once the
// output copy is deleted it answers 200 with the input copy: check each file in its own root.
const serveIn = (root, path) => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}?root=${root}`;
const rootOf = url => new URL(url, 'http://gallery.test').searchParams.get('root');
// Which copy the scan keeps depends on file times: the fixture stamps shared/same.png with the
// wall clock, and the kernel can stamp its bulk copies a fraction of a millisecond earlier.
// listOrder rewrites the finished scan's groups so a test can pick the order it needs.
async function openDuplicateScan(page, request, listOrder = null) {
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
await expect(await request.post('/__test__/reset', { data: { extra_images: 2 } })).toBeOK();
if (listOrder) {
await page.route('**/prompt_manager/scan_duplicates/stream', async route => {
const response = await route.fetch();
const body = (await response.text()).replace(/^data: (.*)$/gm, (line, json) => {
const event = JSON.parse(json);
if (event.type !== 'complete') return line;
for (const group of event.duplicates) group.images = listOrder(group.images);
return `data: ${JSON.stringify(event)}`;
});
await route.fulfill({ response, body });
});
}
await page.goto('/prompt_manager/gallery');
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(4);
const originals = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
const output = rootOf(originals.find(url => url.includes('/bulk/')));
const input = rootOf(originals.find(url => url.includes('/shared/same.png') && rootOf(url) !== output));
expect(output).not.toBeNull();
expect(input).not.toBeNull();
await page.locator('#settingsBtn').click();
await page.locator('#scanDuplicatesBtn').click();
await expect(page.locator('#duplicatesCount')).toHaveText('1');
const candidates = page.locator('.duplicate-checkbox');
await expect(candidates).toHaveCount(2);
return { originals, output, input, candidates };
}
async function expectOnlySelectedGone(page, request, { originals, output, input }, selected, retained) {
expect((await request.get(serveIn(output, selected))).status()).toBe(404);
await expect(await request.get(serveIn(output, retained))).toBeOK();
await expect(await request.get(serveIn(input, 'shared/same.png'))).toBeOK();
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
const serve = path => `/prompt_manager/images/serve/${path.split(/[\\/]/).map(encodeURIComponent).join('/')}`;
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serve(selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serve(selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
expect((await request.get(serve(selected))).status()).toBe(404);
await expect(await request.get(serve(retained))).toBeOK();
await page.reload();
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(3);
const remaining = await page.locator('#galleryGrid img[data-original]').evaluateAll(images => images.map(image => image.dataset.original));
@@ -404,47 +395,4 @@ async function expectOnlySelectedGone(page, request, { originals, output, input
expect(originals).toContain(url);
await expect(await request.get(url)).toBeOK();
}
}
test('duplicate cleanup cancels safely and deletes only the selected file after a retry', async ({ page, request }) => {
const scan = await openDuplicateScan(page, request);
const { output, candidates } = scan;
const selected = await candidates.first().getAttribute('data-image-path');
const retained = await candidates.last().getAttribute('data-image-path');
await candidates.first().check();
let deletes = 0;
page.on('request', request => { if (request.url().endsWith('/delete_duplicate_images')) deletes++; });
confirmNextDialog(page, false, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
expect(deletes).toBe(0);
await expect(await request.get(serveIn(output, selected))).toBeOK();
await page.route('**/prompt_manager/delete_duplicate_images', route => route.abort('failed'), { times: 1 });
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
await page.locator('#removeDuplicatesBtn').click();
await expect(page.getByText('Failed to delete duplicate images', { exact: true })).toBeVisible();
await expect(page.locator('#removeDuplicatesBtn')).toBeEnabled();
await expect(await request.get(serveIn(output, selected))).toBeOK();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
test('deleting a duplicate whose path another gallery root also holds removes only the scanned copy', async ({ page, request }) => {
// The order CI saw: a bulk copy stamped first, so shared/same.png is the first Delete candidate.
const ciOrder = ['bulk/0000.png', 'shared/same.png', 'bulk/0001.png'];
const rank = image => ciOrder.indexOf(image.path.replace(/\\/g, '/'));
const scan = await openDuplicateScan(page, request, images => [...images].sort((a, b) => rank(a) - rank(b)));
const selected = await scan.candidates.first().getAttribute('data-image-path');
const retained = await scan.candidates.last().getAttribute('data-image-path');
expect(selected.replace(/\\/g, '/')).toBe('shared/same.png');
await scan.candidates.first().check();
confirmNextDialog(page, true, 'permanently delete 1 duplicate image files');
const deleted = page.waitForResponse(response => response.url().endsWith('/delete_duplicate_images'));
await page.locator('#removeDuplicatesBtn').click();
expect((await (await deleted).json())).toMatchObject({ success: true, deleted_count: 1, failed_count: 0 });
await expect(page.locator('#duplicatesModal')).toBeHidden();
await expectOnlySelectedGone(page, request, scan, selected, retained);
});
-49
View File
@@ -284,19 +284,6 @@ test('reload during a scan reattaches to the running job', async ({ page, reques
await expect(pill(page)).toBeHidden();
});
test('an empty Activity says no job has run, and a list that cannot be read says so', async ({ page }) => {
// The empty and failed states were only reached in passing by other tests, so their coverage depended on timing.
const empty = main(page).locator('section.surface > p.empty-state');
await page.goto(`${workspace}#/activity`);
await expect(empty).toHaveText('No jobs have run since ComfyUI started. Scans and other long operations appear here.');
await expect(empty).not.toHaveAttribute('role');
await expect(items(page)).toHaveCount(0);
await page.route(url => url.pathname === '/prompt_manager/jobs', route => route.abort());
await page.reload();
await expect(empty).toHaveText('Activity could not be read. It is tried again shortly.');
await expect(empty).toHaveAttribute('role', 'alert');
});
test('lost server clears the pill and marks jobs ended', async ({ page, request }) => {
// A poll interval after the restart, then failed re-reads backing off for 1 + 2 + 4 s.
test.setTimeout(90_000);
@@ -737,42 +724,6 @@ test('a held legacy job shows no Cancel in the list or its view', async ({ page,
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report']);
});
// Since #289 an item dropped its Cancel job only when it was already in the page, so the first render of jobs the
// store already knew (Activity opened from another page) kept a Cancel job on a job that had ended, and moved a
// running job's progress after its actions. The read Activity starts on opening is held, so that first render is
// what the page shows (the read's own render would otherwise repair it a moment later).
test('jobs that ended before Activity opened show no Cancel job, a running one keeps its progress first', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
const finished = await startJob(request, { title: 'Finished scan' });
await finish(request, finished.id);
const cancelled = await startJob(request, { title: 'Cancelled scan' });
await finish(request, cancelled.id, 'cancelled');
await startJob(request, { title: 'Running scan' });
// The page's first list read has all three jobs once the running one shows in the pill.
await page.goto(`${workspace}#/images`);
await expect(pill(page)).toBeVisible();
let hold;
const held = new Promise(resolve => { hold = resolve; });
// Only the first read is held; any later one goes through.
await page.route(url => url.pathname === '/prompt_manager/jobs', route => {
if (!hold) return route.continue();
hold(route);
hold = null;
});
await activityButton(page).click();
const route = await held;
await expect(items(page)).toHaveCount(3);
const item = title => items(page).filter({ has: page.getByRole('link', { name: new RegExp(`${title}$`) }) });
for (const ended of ['Finished scan', 'Cancelled scan']) {
await expect(item(ended).locator('progress, .row-actions')).toHaveCount(0);
await expect(item(ended).getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(0);
}
await expect(item('Running scan').getByRole('button', { name: 'Cancel job', exact: true })).toHaveCount(1);
expect(await item('Running scan').evaluate(node => [...node.children].map(child => child.localName + (child.className ? `.${child.className}` : ''))))
.toEqual(['div.panel-heading', 'p', 'progress', 'div.row-actions']);
await route.continue();
});
test('busy start names the running job', async ({ page, request }) => {
await page.setViewportSize({ width: 1600, height: 1100 });
// The page's timers are Playwright's, so the toast's lifetime below is run out rather than waited for.
-44
View File
@@ -463,50 +463,6 @@ test('collection viewer ignores delayed media callbacks and keeps inspection ret
await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0);
});
test('collection viewer leaving an image Viewer.js has not built yet releases it at once',async({page})=>{
await page.goto(workspace);await expect(cards(page)).toHaveCount(2);
let release,held;const waiting=new Promise(resolve=>{held=resolve;});const pending=new Promise(resolve=>{release=resolve;});
await page.route('**/images/serve/shared/same.png?*',async route=>{
const url=new URL(route.request().url());
if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();await pending;}
await route.continue().catch(()=>{});
});
await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting;
const preview=page.locator('dialog#viewer-dialog');
await page.evaluate(()=>{window.unbuiltImage=document.querySelector('#viewer-stage > img');});
await preview.getByRole('button',{name:'Next media',exact:true}).click();
await expect(preview.getByRole('button',{name:'Zoom in',exact:true})).toHaveAttribute('aria-disabled','false');
// The left image is off the stage. Viewer.js has not built it, so the viewer is destroyed and the src released at
// once: its late load must never build Viewer.js into an image without a parent (getComputedStyle(null)).
release();
await expect.poll(()=>page.evaluate(()=>[window.unbuiltImage.isConnected,window.unbuiltImage.hasAttribute('src'),!!window.unbuiltImage.viewer])).toEqual([false,false,false]);
await page.evaluate(()=>{location.hash='/overview';});await expect(page.locator('.viewer-container,#viewer-dialog')).toHaveCount(0);
});
test('collection viewer closed while its image finishes loading never builds Viewer.js afterwards',async({page})=>{
await page.goto(workspace);await expect(cards(page)).toHaveCount(2);
let held;const waiting=new Promise(resolve=>{held=resolve;});
// The image is held for the whole test: its late load is delivered below, at the moment Viewer.js cannot cancel.
await page.route('**/images/serve/shared/same.png?*',async route=>{
const url=new URL(route.request().url());
if(url.searchParams.get('root')==='0'&&url.searchParams.has('_pm_viewer')) {held();return;}
await route.continue().catch(()=>{});
});
await cards(page).first().getByRole('button',{name:'Open same.png',exact:true}).click();await waiting;
const preview=page.locator('dialog#viewer-dialog');
// Viewer.js's own cancel removes its image listeners only while `image.complete` is false. An image that has
// finished downloading but whose load event is still queued is already complete: close in that state, then deliver it.
await page.evaluate(()=>{window.finishingImage=document.querySelector('#viewer-stage > img');Object.defineProperty(window.finishingImage,'complete',{get:()=>true});});
await page.keyboard.press('Escape');await expect(preview).toHaveCount(0);
const after=await page.evaluate(async()=>{
const image=window.finishingImage;image.dispatchEvent(new Event('load'));
// Viewer.js builds an inline viewer in a zero-delay timeout after its image loads; this runs after it.
await new Promise(resolve=>setTimeout(resolve));
return {src:image.hasAttribute('src'),built:!!image.parentNode?.querySelector('.viewer-container')};
});
expect(after).toEqual({src:false,built:false});
});
test('collection viewer describes video transforms and releases native playback on navigation',async({page,request})=>{
await expect(await request.post('/__test__/reset',{data:{include_video:true}})).toBeOK();await page.goto(workspace);
await page.locator('article[data-image-key="0:clip.webm"] .image-open').click();
@@ -1,401 +0,0 @@
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
import { test, expect } from './fixtures.mjs';
import { seedConcept } from './concept-inventory.mjs';
// System › Sources & scans (#249 System S4 Task 11) on the concept-shaped real fixture, whose two sources are this
// server's `output` and `input` folders: the source list, monitoring and worker rows read GET /settings (S4 Task 9),
// Add/Remove edit the host draft behind the one Save row, and every scan starts the session job through the shared
// "Review metadata scan" (D55), scoped to one source from its row (S4 Task 10). The hooks are workspace_system_fixture.py's.
const workspace = '/prompt_manager/workspace';
const SETTINGS_URL = '/prompt_manager/settings';
const NOTICE_KEY = 'pm_update_notice_dismissed';
const OUTSIDE_REASON = 'Gallery root must be inside a ComfyUI directory (output, input, temp, user) or one listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS';
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
page.on('pageerror', error => page.systemErrors.push(error.message));
});
test.afterEach(async ({ page, request, browserName }) => {
await saveCoverage(page, browserName);
await request.post('/__test__/release');
await request.post('/__test__/jobs/restart');
await request.post('/__test__/system/reset');
expect(page.systemErrors).toEqual([]);
});
const toast = page => page.locator('#workspace-toast');
const layoutOf = page => page.locator('#workspace-main .settings-layout');
const sourceList = page => layoutOf(page).locator('#source-list');
const statusOf = page => layoutOf(page).locator('#settings-status');
const saveButton = page => layoutOf(page).getByRole('button', { name: 'Save preferences', exact: true });
const dialog = page => page.locator('dialog#modal[open]');
async function serverJson(request, url) {
const response = await request.get(url);
await expect(response).toBeOK();
return response.json();
}
/** Open Sources with the update notice dismissed (or shown, `notice: true`) once the source list has loaded. */
async function openSources(page, { notice = false } = {}) {
if (!notice) await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layoutOf(page).locator('#workers')).toBeEnabled();
return layoutOf(page);
}
/** The source list's rows: `[h3, p, [buttons]]`, then its note line when there is one. */
function rowsOf(page) {
return sourceList(page).evaluate(list => [...list.children].map(row => row.matches('.operation')
? [row.querySelector('h3').textContent, row.querySelector('p').textContent,
[...row.querySelectorAll('.row-actions > button')].map(button => button.textContent)]
: [row.localName, row.className, row.textContent]));
}
const sourceRow = (page, title) => sourceList(page).locator('.operation').filter({ has: page.getByRole('heading', { name: title, exact: true }) });
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed) without scanning', async ({ page, request }) => {
const writes = [];
page.on('request', item => { if (!['GET', 'HEAD', 'OPTIONS'].includes(item.method())) writes.push(item.url()); });
const layout = await openSources(page, { notice: true });
const notice = layout.locator(':scope > .notice');
await expect(notice.locator('h2')).toHaveText('Bring older images into your prompt history');
await expect(notice.locator('p')).toHaveText('The schema upgrades automatically. An optional metadata scan links existing positives and negatives without regenerating all thumbnails.');
await expect(notice.locator('.row-actions > button')).toHaveText(['Later', 'Review scan']);
await notice.getByRole('button', { name: 'Later', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await expect(dialog(page)).toHaveCount(0);
// The dismissal is the legacy admin page's own, so neither page shows the notice again.
await page.reload();
await expect(sourceList(page).locator('.operation').first()).toBeVisible();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
expect(writes).toEqual([]);
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Review scan opens the shared Review metadata scan with all sources', async ({ page, request }) => {
const layout = await openSources(page, { notice: true });
// The notice's Review scan dismisses it, as the legacy page's does, and reviews every source.
await layout.locator(':scope > .notice').getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
expect(await page.evaluate(key => localStorage.getItem(key), NOTICE_KEY)).toBe('3.2.0');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(layout.locator(':scope > .notice')).toHaveCount(0);
// Enrich existing metadata is the same review.
const enrich = layout.locator('.operation').filter({ hasText: 'Enrich existing metadata' });
await expect(enrich.locator('p')).toHaveText('Resolve both prompts, deduplicate recipes, preserve uncertain links.');
await enrich.getByRole('button', { name: 'Review scan', exact: true }).click();
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
expect((await serverJson(request, '/prompt_manager/jobs')).jobs).toEqual([]);
});
test('Scan changes opens it with that one source ("1 output folder") and starts a scoped job that opens in Activity', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['input'] } })).toBeOK();
const bodies = [];
page.on('request', item => { if (item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan')) bodies.push(item.postDataJSON()); });
await openSources(page);
const input = sourceRow(page, 'ComfyUI input');
await expect(input.locator('p')).toHaveText('input · watching');
await input.getByRole('button', { name: 'Scan changes', exact: true }).click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Review metadata scan');
await expect(dialog(page).locator('dl.meta-grid > div > dd')).toHaveText(['1 output folder', 'New and unresolved files']);
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
// The toast says it started (or, for a scan this quick, that it already finished); Activity shows its progress.
await expect(toast(page)).toHaveText(/^Scan (started\.|finished · )/);
expect(bodies).toEqual([{ scope: 'new', roots: [1], paths: ['input'] }]);
const id = page.url().split('/').pop();
// The job walks only that source (the count is recorded once the walk has listed it).
await expect.poll(async () => (await serverJson(request, `/prompt_manager/jobs/${id}`)).job.scope)
.toMatchObject({ files: 'new', roots: [1], output_folders: 1 });
await expect(page.locator('#workspace-main h1')).toHaveText('Scan for metadata');
});
test('Add source validates on the server, then marks the host draft dirty; Save persists', async ({ page, request }) => {
// A folder inside ComfyUI's output folder that is not a source yet.
await expect(await request.post('/__test__/system/unthumbnailed', { data: { count: 1 } })).toBeOK();
const layout = await openSources(page);
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
]);
await expect(layout.locator('.panel-heading > h2')).toHaveText('Image sources');
const add = layout.locator('.panel-heading').getByRole('button', { name: '+ Add source', exact: true });
await add.click();
await expect(dialog(page).locator('#modal-title')).toHaveText('Add image source');
await expect(dialog(page).locator('label[for="action-input"]')).toHaveText('Folder path');
await expect(dialog(page).locator('#action-input')).toHaveValue('');
await expect(dialog(page).locator('.dialog-actions > button')).toHaveText(['Cancel', 'Add']);
await dialog(page).locator('#action-input').fill('output/unthumbnailed');
const validated = page.waitForResponse(response => response.url().endsWith('/prompt_manager/settings/validate-source'));
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
expect(await (await validated).json()).toEqual({ success: true, valid: true, label: 'unthumbnailed', path: 'output/unthumbnailed', code: null, reason: null });
await expect(dialog(page)).toHaveCount(0);
await expect(add).toBeFocused();
// Validation wrote nothing: the draft is dirty and the server still has two sources.
expect(await rowsOf(page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']],
['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']],
['unthumbnailed', 'output/unthumbnailed · not saved yet', ['Remove']],
]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.effective).toHaveLength(2);
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const body = (await saved).postDataJSON();
expect(body.gallery_root_paths).toEqual(['output', 'input', 'output/unthumbnailed']);
expect(typeof body.expected_revision).toBe('string');
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// The list is the server's again: three effective sources, each scannable.
await expect(sourceRow(page, 'unthumbnailed').locator('p')).toHaveText('output/unthumbnailed · manual scan');
await expect(sourceRow(page, 'unthumbnailed').getByRole('button', { name: 'Scan folder', exact: true })).toBeVisible();
expect((await serverJson(request, SETTINGS_URL)).sources.effective.map(source => source.path)).toEqual(['output', 'input', 'output/unthumbnailed']);
});
test('invalid source shows the server reason', async ({ page, request }) => {
const layout = await openSources(page);
await layout.getByRole('button', { name: '+ Add source', exact: true }).click();
const field = dialog(page).locator('#action-input');
for (const [path, reason, submit] of [
['output/not-there', 'Gallery root does not exist', 'click'],
// Enter in the field adds too (M6).
['/etc', OUTSIDE_REASON, 'Enter'],
]) {
await field.fill(path);
if (submit === 'Enter') await field.press('Enter');
else await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText(reason);
await expect(field).toHaveAttribute('aria-invalid', 'true');
await expect(field).toHaveValue(path);
}
// A folder already listed (the server's answer names it) is not added twice.
await field.fill('input');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('This folder is already a source.');
// A check that cannot finish says so and adds nothing.
await page.route('**/prompt_manager/settings/validate-source', route => route.fulfill({ status: 500,
json: { success: false, code: null, error: 'Failed to check the source' } }));
await field.fill('output/not-checked');
await dialog(page).getByRole('button', { name: 'Add', exact: true }).click();
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveText('The folder could not be checked. Try again.');
await page.unroute('**/prompt_manager/settings/validate-source');
// Typing again clears the reason.
await field.fill('output/unthumbnailed');
await expect(dialog(page).locator('p.small[role="alert"]')).toHaveCount(0);
await expect(field).not.toHaveAttribute('aria-invalid', 'true');
await dialog(page).getByRole('button', { name: 'Cancel', exact: true }).click();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
});
test('Remove marks the draft; removing the last entry states auto-detect', async ({ page, request }) => {
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['ComfyUI output', 'output · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await sourceRow(page, 'ComfyUI output').getByRole('button', { name: 'Remove', exact: true }).click();
expect(await rowsOf(page)).toEqual([['p', 'small', 'No source is configured. After saving, ComfyUI’s output folder is used.']]);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
// Saved: ComfyUI's output folder is detected and listed without Remove.
await expect.poll(rowsOf.bind(null, page)).toEqual([
['ComfyUI output', 'output · manual scan', ['Scan folder']],
['p', 'small', 'No source is configured; ComfyUI’s output folder is used.'],
]);
const { sources } = await serverJson(request, SETTINGS_URL);
expect([sources.configured, sources.auto_detected]).toEqual([[], true]);
});
test('a configured source that does not exist says why and can be removed', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source')).toBeOK();
await openSources(page);
// The fixture's unmounted drive is not inside a ComfyUI folder, so the server says that (it never probes outside).
expect((await rowsOf(page)).at(-1)).toEqual(['unmounted-drive', `unmounted-drive · ${OUTSIDE_REASON}`, ['Remove']]);
await sourceRow(page, 'unmounted-drive').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output', 'input']);
});
test('watching only with a live observer; restart pending otherwise', async ({ page, request }) => {
// No image monitor in this process: every source is scanned by hand.
await openSources(page);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · manual scan', 'input · manual scan']);
// A live observer that scheduled output: output is watched; input was added since it started.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output'] } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · restart pending']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan folder']);
// An observer that died watches nothing.
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'], alive: false } })).toBeOK();
await page.reload();
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · restart pending', 'input · restart pending']);
});
test('Watch for new outputs saves monitoring_enabled; a restart applies it', async ({ page, request }) => {
await expect(await request.post('/__test__/system/monitor', { data: { watching: ['output', 'input'] } })).toBeOK();
const layout = await openSources(page);
const monitoring = layout.getByRole('switch', { name: 'Watch for new outputs', exact: true });
await expect(monitoring).toBeChecked();
await expect(layout.locator('.setting-row').filter({ has: page.locator('#monitoring') }).locator('p'))
.toHaveText('New files enter the same library. Existing scans do not count as new runs.');
await monitoring.uncheck();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await (await saved).json();
expect(answer.restart_required_fields).toEqual(['monitoring_enabled']);
// D128: saved for the next start, and the status says a restart applies it.
await expect(statusOf(page)).toHaveText('Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The running observer keeps watching until ComfyUI restarts, and the list says so.
const { settings, monitor } = await serverJson(request, SETTINGS_URL);
expect([settings.monitoring_enabled, monitor.observer_alive]).toEqual([false, true]);
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching until restart', 'input · watching until restart']);
await expect(sourceList(page).locator('.operation .row-actions > button:first-child')).toHaveText(['Scan changes', 'Scan changes']);
// Switched back on before the restart: the observer still runs, so nothing waits for one.
await monitoring.check();
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
await expect(sourceList(page).locator('.operation p')).toHaveText(['output · watching', 'input · watching']);
});
test('worker range max is cpu_count', async ({ page, request }) => {
const { settings } = await serverJson(request, SETTINGS_URL);
const layout = await openSources(page);
const workers = layout.locator('#workers');
await expect(layout.locator('label[for="workers"]')).toHaveText('Worker threads');
await expect(workers).toHaveAttribute('type', 'range');
await expect(workers).toHaveAttribute('min', '1');
await expect(workers).toHaveAttribute('max', String(settings.cpu_count));
await expect(workers).toHaveValue(String(settings.worker_threads));
const next = settings.worker_threads === 1 ? settings.cpu_count : 1;
await workers.fill(String(next));
await expect(statusOf(page)).toHaveText(next === settings.worker_threads ? 'No unsaved changes.' : 'Unsaved changes.');
test.skip(next === settings.worker_threads, 'One core: the range has a single value');
const saved = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
expect((await saved).postDataJSON().worker_threads).toBe(next);
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.worker_threads).toBe(next);
});
test('Refresh files calls /gallery/rescan and toasts its counts', async ({ page }) => {
const layout = await openSources(page);
const refresh = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Refresh files', exact: true }) });
await expect(refresh.locator('p')).toHaveText('Read every source again to find files added or removed outside ComfyUI.');
const answered = page.waitForResponse(response => response.url().endsWith('/prompt_manager/gallery/rescan'));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
const { total, roots } = await (await answered).json();
expect(roots).toBe(2);
await expect(toast(page)).toHaveText(`Files refreshed. ${total.toLocaleString('en-US')} files in 2 output folders.`);
// A failed refresh says so.
await page.route('**/prompt_manager/gallery/rescan', route => route.fulfill({ status: 500, json: { success: false, error: 'busy disk' } }));
await refresh.getByRole('button', { name: 'Refresh', exact: true }).click();
await expect(toast(page)).toHaveText('Files could not be refreshed. Try again.');
});
test('Thumbnail generation hands off to the existing gallery until its review dialog', async ({ page }) => {
const layout = await openSources(page);
const thumbnails = layout.locator('.operation').filter({ has: page.getByRole('heading', { name: 'Thumbnail generation', exact: true }) });
await expect(thumbnails.locator('p')).toHaveText('Generate missing previews; rebuilding all is a separate explicit option.');
const popup = page.waitForEvent('popup');
await thumbnails.getByRole('button', { name: 'Review options', exact: true }).click();
const gallery = await popup;
expect(new URL(gallery.url()).pathname).toBe('/prompt_manager/gallery');
await gallery.close();
await expect(statusOf(page)).toHaveText('No unsaved changes.');
});
test('a save refused by a running scan shows the busy link toast', async ({ page, request }) => {
const running = await (await request.post('/__test__/jobs/start', { data: { title: 'Scan for metadata' } })).json();
await openSources(page);
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
// The draft is kept for the next Save, and the server still has both sources.
await expect(sourceList(page).locator('.operation')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(2);
// An answer that does not name its job (an older server): the toast names the running one from the job list.
await page.route(`**${SETTINGS_URL}`, route => (route.request().method() === 'POST'
? route.fulfill({ status: 409, json: { success: false, code: 'busy', error: 'Scan for metadata already running' } })
: route.continue()));
await page.locator('#workspace-toast').evaluate(node => { node.textContent = ''; });
await saveButton(page).click();
await expect(toast(page)).toHaveText('Another job is running: Scan for metadata.');
await expect(toast(page).getByRole('link', { name: 'Scan for metadata', exact: true })).toHaveAttribute('href', `#/activity/${running.job.id}`);
await page.unroute(`**${SETTINGS_URL}`);
await request.post(`/__test__/jobs/${running.job.id}/finish`, { data: { state: 'finished' } });
});
test('a source list that cannot be read says so and retries', async ({ page }) => {
await page.addInitScript(key => { if (location.protocol.startsWith('http')) localStorage.setItem(key, '3.2.0'); }, NOTICE_KEY);
let failing = true;
await page.route(`**${SETTINGS_URL}`, route => (failing && route.request().method() === 'GET'
? route.fulfill({ status: 500, json: { success: false, error: 'Injected failure' } }) : route.continue()));
await page.goto(`${workspace}#/system/sources`);
await expect(sourceList(page).getByRole('alert')).toHaveText('Sources could not be loaded. Your saved data has not changed.');
await expect(sourceList(page).locator('.operation')).toHaveCount(0);
// The server settings rows say so too, and stay disabled.
await expect(layoutOf(page).locator('#workers')).toBeDisabled();
failing = false;
await sourceList(page).getByRole('button', { name: 'Retry', exact: true }).click();
await expect(sourceList(page).locator('.operation')).toHaveCount(2);
});
test('a scan from a list that changed since it was read starts nothing, says why and reads the list again', async ({ page, request }) => {
await openSources(page);
// The legacy page (or another tab) removes ComfyUI output: the page's "input" is index 1, now past the end.
await expect(await request.post(SETTINGS_URL, { data: { gallery_root_paths: ['input'] } })).toBeOK();
const refused = page.waitForResponse(response => response.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
const answer = await refused;
expect([answer.status(), (await answer.json()).code]).toEqual([409, 'sources-changed']);
await expect(toast(page)).toHaveText('The sources changed since they were loaded. Reload them.');
// No scan started (the other save ran as its own source-change hold).
expect((await serverJson(request, '/prompt_manager/jobs')).jobs.map(job => job.kind)).toEqual(['source_change']);
// Read again: the one source is index 0 now, and its scan starts.
await expect.poll(rowsOf.bind(null, page)).toEqual([['ComfyUI input', 'input · manual scan', ['Scan folder', 'Remove']]]);
await expect(statusOf(page)).toHaveText('No unsaved changes.');
const started = page.waitForRequest(item => item.method() === 'POST' && item.url().endsWith('/prompt_manager/jobs/scan'));
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Scan folder', exact: true }).click();
await dialog(page).getByRole('button', { name: 'Start scan', exact: true }).click();
expect((await started).postDataJSON()).toEqual({ scope: 'new', roots: [0], paths: ['input'] });
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
});
test('a save refused for one unusable source names it and marks its row', async ({ page, request }) => {
await expect(await request.post('/__test__/system/missing-source', { data: { inside: true } })).toBeOK();
await openSources(page);
await expect(sourceRow(page, 'unplugged').locator('p')).toHaveText('output/unplugged · manual scan');
// The drive is unplugged after the page read the list; removing another source resends the whole list.
await expect(await request.post('/__test__/system/unplug')).toBeOK();
await sourceRow(page, 'ComfyUI input').getByRole('button', { name: 'Remove', exact: true }).click();
const saved = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(SETTINGS_URL));
await saveButton(page).click();
const answer = await saved;
expect(answer.status()).toBe(400);
expect(await answer.json()).toMatchObject({ code: 'invalid-source', source: 'output/unplugged', index: 1 });
await expect(statusOf(page)).toHaveText("output/unplugged can't be used: Gallery root does not exist. Remove it or reconnect it, then save.");
await expect(sourceRow(page, 'unplugged')).toHaveAttribute('aria-invalid', 'true');
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(1);
expect((await serverJson(request, SETTINGS_URL)).sources.configured).toHaveLength(3);
// Removed, the list saves.
await sourceRow(page, 'unplugged').getByRole('button', { name: 'Remove', exact: true }).click();
await expect(sourceList(page).locator('.operation[aria-invalid="true"]')).toHaveCount(0);
await saveButton(page).click();
await expect(statusOf(page)).toHaveText('Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).sources.configured.map(source => source.path)).toEqual(['output']);
});
+7 -376
View File
@@ -11,7 +11,7 @@ const number = value => value.toLocaleString('en-US');
test.beforeEach(async ({ page, request, browserName }) => {
await startCoverage(page, browserName);
page.manifest = await seedConcept(request);
await seedConcept(request);
await expect(await request.post('/__test__/system/reset')).toBeOK();
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
page.systemErrors = [];
@@ -55,8 +55,6 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
expect([overview.library.positive_prompts, overview.images.indexed, thumbnails.pending]).toEqual([6, 24, 0]);
// I2: the Overview never walks the sources for thumbnails; the row reads the status route itself.
expect(overview).not.toHaveProperty('thumbnails');
// The fixture records no ambiguous file: the Needs review queue is empty.
expect(overview.needs_review).toEqual({ pending: 0 });
const layout = await openOverview(page);
expect(await layout.locator('.summary-metrics > .metric').evaluateAll(items => items.map(item =>
[item.querySelector('strong').textContent, item.querySelector('span').textContent]))).toEqual([
@@ -66,7 +64,7 @@ test('overview shows real counts and Not checked', async ({ page, request }) =>
['Not checked', 'Database integrity'],
]);
expect(await rowsOf(layout, 'Library health')).toEqual([
['0 ambiguous image links', 'Review metadata before assigning a positive / negative pair.', 'Review metadata'],
['Ambiguous image links', 'Not available yet.', 'Review metadata'],
['Backup status', 'No retained backup record.', 'View backups'],
['No missing thumbnails', 'Generate only missing previews when necessary.', 'Review thumbnails'],
]);
@@ -168,11 +166,10 @@ test('a failed thumbnail read is unavailable, never zero', async ({ page }) => {
await expect(layout.locator('.metric').nth(2).locator('strong')).toHaveText('24');
});
test('ambiguous row reads the pending count; its action waits for the review page', async ({ page }) => {
test('ambiguous row reads Not available yet (Q6)', async ({ page }) => {
const layout = await openOverview(page);
const row = layout.locator('.operation').filter({ hasText: 'ambiguous image links' });
await expect(row.locator('h3')).toHaveText('0 ambiguous image links');
await expect(row.locator('p')).toHaveText('Review metadata before assigning a positive / negative pair.');
const row = layout.locator('.operation').filter({ hasText: 'Ambiguous image links' });
await expect(row.locator('p')).toHaveText('Not available yet.');
const review = row.getByRole('button', { name: 'Review metadata', exact: true });
await expect(review).toHaveAttribute('aria-disabled', 'true');
await review.click({ force: true });
@@ -240,11 +237,10 @@ test('out-of-order overview responses never repaint a newer section', async ({ p
await nav.getByRole('button', { name: 'Appearance', exact: true }).click();
await nav.getByRole('button', { name: 'Overview', exact: true }).click();
await expect.poll(held).toBe(2);
// Integrations still hands off (S10), so its body is one known panel.
await nav.getByRole('button', { name: 'Integrations', exact: true }).click();
await nav.getByRole('button', { name: 'Sources & scans', exact: true }).click();
await expect(await request.post('/__test__/release')).toBeOK();
await expect.poll(held).toBe(0);
// Both late answers reached a section that had left: Integrations keeps its own body.
// Both late answers reached a section that had left: Sources keeps its own body.
await expect(layout.locator(':scope > section.handoff')).toBeVisible();
await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve))));
await expect(layout.locator('.summary-metrics')).toHaveCount(0);
@@ -254,368 +250,3 @@ test('out-of-order overview responses never repaint a newer section', async ({ p
await expect(layout.locator('.summary-metrics')).toHaveCount(1);
await expect(layout.locator('.two-col')).toHaveCount(1);
});
// System › Appearance (#249 System S3 Task 8): the browser display defaults (`gallerySettings`, preferences.mjs) and the
// two host fields behind the one Save row, and the consumers that read them on their next visit.
const SETTINGS_URL = '/prompt_manager/settings';
const INSPECT_PATH = '/prompt_manager/images/inspect/';
const statusOf = page => page.locator('#workspace-main #settings-status');
const saveOf = page => page.locator('#workspace-main .save-row > button');
const storedSettings = page => page.evaluate(() => JSON.parse(localStorage.getItem('gallerySettings') || 'null'));
const imageCards = page => page.locator('#workspace-main article.media-card');
const panel = page => page.getByRole('complementary', { name: 'Selected image details', exact: true });
async function openAppearance(page) {
await page.goto(`${workspace}#/system/appearance`);
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
// The host rows are enabled once the server settings answered.
await expect(page.locator('#timeout')).toBeEnabled();
}
async function saveAppearance(page, text) {
await expect(saveOf(page)).not.toHaveAttribute('aria-disabled', 'true');
await saveOf(page).click();
await expect(statusOf(page)).toHaveText(text);
}
/** Set toggles (by row id) through their Toolkit switches. */
async function toggle(page, values) {
for (const [id, checked] of Object.entries(values)) await page.locator(`#${id}`).setChecked(checked, { force: true });
}
/**
* A request barrier: one request issued and answered after everything the page has asked for so far, so a read the
* last action started has been seen (Playwright reports a page's requests in order).
*/
async function requestBarrier(page) {
expect(await page.evaluate(url => fetch(url, { cache: 'no-store' }).then(response => response.ok), SETTINGS_URL)).toBe(true);
}
async function openImages(page) {
await page.goto(`${workspace}#/images`);
await expect(imageCards(page).first()).toBeVisible();
}
test('defaults apply to a new visit but not to the current view or a saved view', async ({ page, request }) => {
const created = await request.post('/prompt_manager/saved-views', { data: { page: 'images', name: 'Everything',
state: { q: '', category: '', sort: 'newest', source: 'all', folder: '', media: 'all', tags: [], favorites: false } } });
expect(created.status()).toBe(201);
const view = (await created.json()).view;
try {
await openImages(page);
// The current view: the toolbar sets List.
await page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'List', exact: true }).click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await openAppearance(page);
await expect(page.locator('#default-view')).toHaveValue('grid');
await page.locator('#default-view').selectOption('masonry');
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect([stored.workspaceLayout, stored.workspaceDensity]).toEqual(['masonry', 'compact']);
// Back in Images (the page session's last visit): the current view keeps its own layout and size.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// The saved view opens as before.
await page.goto(`${workspace}#/saved/${view.id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'comfortable');
// A new visit starts from the defaults.
await openImages(page);
await page.reload();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'masonry');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
await expect(page.getByRole('group', { name: 'Image layout' }).getByRole('button', { name: 'Masonry', exact: true }))
.toHaveAttribute('aria-pressed', 'true');
} finally {
await request.delete(`/prompt_manager/saved-views/${view.id}`);
}
});
test('the defaults reach the Images page only; embedded galleries keep their own presentation', async ({ page }) => {
await openAppearance(page);
await page.locator('#default-view').selectOption('list');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
// The prompt history's shared gallery has no toolbar to change them, so it opens as it always did.
await page.goto(`${workspace}#/library/${page.manifest.prompts[0].positive_id}`);
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-layout', 'grid');
await expect(page.locator('#workspace-main .image-grid')).toHaveAttribute('data-density', 'comfortable');
// The Images page starts from them.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
await expect(page.locator('.image-grid')).toHaveAttribute('data-density', 'compact');
});
test('metadata off shows an explicit load and loads once on request', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'metadata-auto': false });
await saveAppearance(page, 'Saved in this browser.');
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
await openImages(page);
const first = imageCards(page).first();
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).getByRole('status')).toHaveText('Metadata loads when you ask for it.');
const load = panel(page).getByRole('button', { name: 'Load metadata', exact: true });
await expect(load).toBeVisible();
expect(reads).toEqual([]);
await load.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(1);
// The same file again reads nothing more (the cache is on).
await first.locator('.image-card-actions button').first().click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await requestBarrier(page);
expect(reads).toHaveLength(1);
// The viewer's details use the same inspector: another file waits there too, and loads on request.
await imageCards(page).nth(1).locator('.image-open').click();
const viewerDetails = page.locator('#viewer-details');
await expect(viewerDetails.getByRole('status')).toHaveText('Metadata loads when you ask for it.');
expect(reads).toHaveLength(1);
await viewerDetails.getByRole('button', { name: 'Load metadata', exact: true }).click();
await expect(viewerDetails.locator('.panel-heading h2')).toBeVisible();
expect(reads).toHaveLength(2);
});
test('cache off rereads the current selection; on reuses the one current record', async ({ page }) => {
const reads = [];
page.on('request', item => { if (new URL(item.url()).pathname.startsWith(INSPECT_PATH)) reads.push(item.url()); });
for (const cache of [true, false]) {
if (!cache) {
await openAppearance(page);
await toggle(page, { 'metadata-cache': false });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
}
await openImages(page);
const info = imageCards(page).first().locator('.image-card-actions button').first();
await info.click();
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
await expect.poll(() => reads.length).toBeGreaterThan(0);
await requestBarrier(page);
reads.length = 0;
// Inspect the selected file twice more.
for (const _ of [1, 2]) {
const answered = cache ? null : page.waitForResponse(response => new URL(response.url()).pathname.startsWith(INSPECT_PATH));
await info.click();
if (answered) await answered;
await expect(panel(page).locator('.panel-heading h2')).toBeVisible();
}
await requestBarrier(page);
expect(reads, `cache ${cache}`).toHaveLength(cache ? 0 : 2);
}
});
test('paths hidden only changes display', async ({ page }) => {
const rows = async () => page.locator('#viewer-details details').filter({ hasText: 'File & provenance' }).locator('dt')
.evaluateAll(terms => terms.map(term => term.textContent));
const openViewer = async () => {
await openImages(page);
await imageCards(page).first().locator('.image-open').click();
const provenance = page.locator('#viewer-details details').filter({ hasText: 'File & provenance' });
await provenance.locator('summary').click();
await expect(provenance.locator('dl')).toBeVisible();
return provenance;
};
let provenance = await openViewer();
expect(await rows()).toContain('Path');
const before = { show: await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count(),
download: await provenance.getByRole('link', { name: 'Download original', exact: true }).getAttribute('href') };
await page.keyboard.press('Escape');
await openAppearance(page);
await toggle(page, { paths: false });
await saveAppearance(page, 'Saved in this browser.');
provenance = await openViewer();
const shown = await rows();
expect(shown).not.toContain('Path');
expect(shown).toEqual(expect.arrayContaining(['Source', 'Size']));
// Everything else is as it was: the folder action and the original download stay.
expect(await provenance.getByRole('button', { name: /^Show .* in folder$/ }).count()).toBe(before.show);
await expect(provenance.getByRole('link', { name: 'Download original', exact: true })).toHaveAttribute('href', before.download);
});
test('video defaults reach the viewer', async ({ page }) => {
await openAppearance(page);
await toggle(page, { 'video-auto': true, 'video-muted': false, 'video-loop': false });
await saveAppearance(page, 'Saved in this browser.');
expect(await page.evaluate(async () => (await import('/prompt_manager/js/workspace/media-viewer.mjs')).readViewerPreferences()))
.toEqual({ videoAutoplay: true, videoMute: false, videoLoop: false });
const stored = await storedSettings(page);
expect([stored.videoAutoplay, stored.videoMute, stored.videoLoop]).toEqual([true, false, false]);
});
test('host fields round-trip through /settings with revision', async ({ page, request }) => {
const posts = [];
page.on('request', item => { if (item.method() === 'POST' && new URL(item.url()).pathname === SETTINGS_URL) posts.push(item.postDataJSON()); });
const before = await serverJson(request, SETTINGS_URL);
await openAppearance(page);
await expect(page.locator('#open-mode')).toHaveValue(before.settings.webui_display_mode);
await expect(page.locator('#timeout')).toHaveValue(String(before.settings.result_timeout));
// Only whole seconds from 0 to 300: anything else is highlighted and never sent.
for (const bad of ['301', '-1', '2.5']) {
await page.locator('#timeout').fill(bad);
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).toHaveAttribute('aria-invalid', 'true');
await expect(statusOf(page)).toHaveText('Some changes were not saved. Review the highlighted settings.');
await expect(saveOf(page)).toHaveAttribute('aria-disabled', 'true');
}
await page.locator('#timeout').fill('0');
await page.locator('#timeout').press('Tab');
await expect(page.locator('#timeout')).not.toHaveAttribute('aria-invalid');
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved to the server configuration.');
expect(posts).toEqual([{ webui_display_mode: 'popup', result_timeout: 0, expected_revision: before.revision }]);
const after = await serverJson(request, SETTINGS_URL);
expect([after.settings.webui_display_mode, after.settings.result_timeout]).toEqual(['popup', 0]);
expect(after.revision).not.toBe(before.revision);
// No browser key was written for a host field.
expect(await storedSettings(page)).toBeNull();
});
test('server settings that cannot be read say so in their rows', async ({ page }) => {
await page.route(url => url.pathname === SETTINGS_URL, route => (route.request().method() === 'GET'
? route.fulfill({ status: 500, contentType: 'application/json', body: JSON.stringify({ success: false, error: 'Injected failure' }) })
: route.continue()));
await page.goto(`${workspace}#/system/appearance`);
for (const id of ['open-mode', 'timeout']) {
const row = page.locator('#workspace-main .setting-row').filter({ has: page.locator(`#${id}`) });
await expect(row.locator('p.small')).toHaveText('Server settings could not be read.');
await expect(page.locator(`#${id}`)).toBeDisabled();
}
// The browser rows are this browser's, and work as before.
await expect(page.locator('#default-view')).toBeEnabled();
await expect(page.locator('#workspace-main .setting-row p.small')).toHaveCount(2);
});
test('stale revision shows the conflict status', async ({ page, request }) => {
await openAppearance(page);
// Another page saves after this one read the settings.
const current = await serverJson(request, SETTINGS_URL);
await expect(await request.post(SETTINGS_URL, { data: { result_timeout: 42, expected_revision: current.revision } })).toBeOK();
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#open-mode')).toHaveAttribute('aria-invalid', 'true');
// The draft is kept and the section now shows the other page's value.
await expect(page.locator('#open-mode')).toHaveValue('popup');
await expect(page.locator('#timeout')).toHaveValue('42');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('newtab');
// With the reloaded revision, Save succeeds.
await saveAppearance(page, 'Saved to the server configuration.');
expect((await serverJson(request, SETTINGS_URL)).settings.webui_display_mode).toBe('popup');
});
test('reload keeps saved values; a new tab reads them', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#image-quality').selectOption('high');
await toggle(page, { 'card-details': false });
await page.locator('#open-mode').selectOption('popup');
await saveAppearance(page, 'Saved in this browser. Saved to the server configuration.');
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#image-quality')).toHaveValue('high');
await expect(page.locator('#card-details')).not.toBeChecked();
await expect(page.locator('#open-mode')).toHaveValue('popup');
const other = await context.newPage();
await openAppearance(other);
await expect(other.locator('#image-quality')).toHaveValue('high');
await expect(other.locator('#card-details')).not.toBeChecked();
// A change saved in one tab reaches the other's clean controls; a draft it overtakes is flagged and kept.
await other.locator('#density').selectOption('large');
await page.locator('#density').selectOption('compact');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#density')).toHaveValue('large');
await expect(other.locator('#settings-status')).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(other.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await page.locator('#image-quality').selectOption('low');
await saveAppearance(page, 'Saved in this browser.');
await expect(other.locator('#image-quality')).toHaveValue('low');
await other.close();
});
test('a change another tab saved while System was closed is a conflict on return, never saved over silently', async ({ page, context }) => {
await openAppearance(page);
await page.locator('#density').selectOption('compact');
await expect(statusOf(page)).toHaveText('Unsaved changes.');
// Leave System for Images: the draft stays for the page session, but System no longer listens.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
const other = await context.newPage();
await openAppearance(other);
await other.locator('#density').selectOption('large');
await saveAppearance(other, 'Saved in this browser.');
await other.close();
// Back in System: the draft is kept and flagged, and nothing was written over the other tab's value.
await page.evaluate(() => { location.hash = '#/system/appearance'; });
await expect(page.locator('#workspace-main .settings-layout > section.surface > h2')).toHaveText('Browsing & display');
await expect(statusOf(page)).toHaveText('Settings changed elsewhere. Reload this section to see them.');
await expect(page.locator('#density')).toHaveAttribute('aria-invalid', 'true');
await expect(page.locator('#density')).toHaveValue('compact');
expect((await storedSettings(page)).workspaceDensity).toBe('large');
// Saving after the conflict was shown is an explicit choice: it wins.
await saveAppearance(page, 'Saved in this browser.');
expect((await storedSettings(page)).workspaceDensity).toBe('compact');
await expect(page.locator('#density')).not.toHaveAttribute('aria-invalid');
});
test('a browser that refuses storage is saved for this tab only, never "Saved"', async ({ page }) => {
await openAppearance(page);
await page.evaluate(() => {
Storage.prototype.setItem = function setItem() { throw new DOMException('Quota exceeded', 'QuotaExceededError'); };
});
await page.locator('#default-view').selectOption('list');
await saveAppearance(page, 'Saved for this tab only; browser storage is unavailable.');
expect(await storedSettings(page)).toBeNull();
// This tab's next Images visit still uses it.
await page.locator('#primary-nav a[data-route="images"]').click();
await expect(imageCards(page).first()).toBeVisible();
await expect(page.locator('.image-grid')).toHaveAttribute('data-layout', 'list');
});
test('added rows reach their consumers', async ({ page }) => {
await openImages(page);
await expect(imageCards(page).first().locator('.media-info h3')).toBeVisible();
await openAppearance(page);
await toggle(page, { 'card-details': false, 'viewer-details': true });
await saveAppearance(page, 'Saved in this browser.');
await page.reload();
await openImages(page);
// Show image details on cards off: no title or file line; Info and Tag stay.
const card = imageCards(page).first();
await expect(card.locator('.media-info h3')).toBeHidden();
await expect(card.locator('.image-card-actions button')).toHaveCount(2);
// Collapse viewer details by default: the viewer opens with its details hidden behind one toggle.
await card.locator('.image-open').click();
const details = page.locator('#viewer-details');
await expect(page.locator('#viewer-dialog')).toBeVisible();
await expect(details).toBeHidden();
const show = page.locator('#viewer-toolbar').getByRole('button', { name: 'Show details', exact: true });
await expect(show).toHaveAttribute('aria-expanded', 'false');
await show.click();
await expect(details).toBeVisible();
await expect(page.locator('#viewer-toolbar').getByRole('button', { name: 'Hide details', exact: true })).toHaveAttribute('aria-expanded', 'true');
});
test('reset restores defaults without touching other keys', async ({ page }) => {
await page.goto(`${workspace}#/system/appearance`);
await page.evaluate(() => localStorage.setItem('gallerySettings', JSON.stringify({ gridColumns: 5, defaultLimit: 200,
thumbnailPromptDismissed: 'kept', workspaceLayout: 'list', videoMute: false, showImageInfo: false })));
await page.reload();
await expect(page.locator('#timeout')).toBeEnabled();
await expect(page.locator('#default-view')).toHaveValue('list');
const reset = page.locator('#workspace-main .operation').filter({ hasText: 'Reset display defaults' }).getByRole('button', { name: 'Reset', exact: true });
await reset.click();
await expect(page.locator('#default-view')).toHaveValue('grid');
await expect(page.locator('#video-muted')).toBeChecked();
await expect(statusOf(page)).toHaveText('Unsaved changes.');
await saveAppearance(page, 'Saved in this browser.');
const stored = await storedSettings(page);
expect(stored).toEqual({ gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'kept', workspaceLayout: 'grid',
videoMute: true, showImageInfo: true });
});
+6 -8
View File
@@ -425,7 +425,8 @@ for (const width of [1600, 390]) test(`overview and handoff render with concept
const SYSTEM_SECTION_LABELS = ['Overview', 'Appearance', 'Sources & scans', 'Integrations', 'Auto-tagging', 'Maintenance',
'Backup & export', 'Diagnostics & logs'];
const SYSTEM_HASHES = ['overview', 'appearance', 'sources', 'integrations', 'tagging', 'maintenance', 'backups', 'diagnostics'];
const SYSTEM_HANDOFFS = { maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
const SYSTEM_HANDOFFS = { appearance: ['/prompt_manager/gallery', '/prompt_manager/admin'], sources: ['/prompt_manager/admin', '/prompt_manager/gallery'],
maintenance: ['/prompt_manager/admin', '/prompt_manager/gallery'] };
test('System is native: the concept shell, section navigation, sections, hand-offs and the Save row', async ({ page }) => {
const mutations = [];
@@ -461,17 +462,14 @@ test('System is native: the concept shell, section navigation, sections, hand-of
await expect(nav.locator('[aria-pressed="true"]')).toHaveCount(1);
// The section changes in place: focus stays on the pressed section, not the page title.
await expect(button).toBeFocused();
// S2 Task 6: Overview is native (workspace-system.spec.mjs), and so are Appearance (S3 Task 8) and Sources & scans
// (S4 Task 11, workspace-system-sources.spec.mjs, its update notice shown in a fresh browser); Maintenance shows
// its health-check notice above the hand-off (D123); every other section hands off.
// S2 Task 6: Overview is native (workspace-system.spec.mjs); Maintenance shows its health-check notice above
// the hand-off (D123); every other section hands off.
if (hash === 'overview') await expect(layout.locator(':scope > .summary-metrics')).toBeVisible();
if (hash === 'sources') await expect(layout.locator('#source-list .operation')).toHaveCount(2);
expect(await layout.evaluate(node => [...node.children].map(child => `${child.localName}.${child.className}`)))
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'], appearance: ['section.surface', 'div.save-row'],
sources: ['div.notice', 'section.surface', 'div.save-row'],
.toEqual({ overview: ['div.summary-metrics', 'div.two-col', 'div.save-row'],
maintenance: ['div.notice', 'section.surface handoff', 'div.save-row'] }[hash]
?? ['section.surface handoff', 'div.save-row']);
if (['overview', 'appearance', 'sources'].includes(hash)) continue;
if (hash === 'overview') continue;
const handoff = layout.locator('section.handoff');
await expect(handoff.locator('h2')).toHaveText('Your current tools are ready');
expect(await handoff.locator('a.workspace-link').evaluateAll(links => links.map(link => link.getAttribute('href'))))
@@ -176,8 +176,6 @@ _TABLES = (
"execution_context_texts",
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -28,8 +28,6 @@ EDITOR_METADATA = {
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+6 -96
View File
@@ -11,18 +11,11 @@ read. These per-server hooks only change what a System test needs on top:
requests for ``path`` (a ``/prompt_manager/system/`` route) with the routes'
500 error envelope;
- ``POST /__test__/system/missing-source`` configures one more source that
does not exist (an unmounted drive); with ``{"inside": true}`` it is an
existing folder inside the output folder (``output/unplugged``), which
``POST /__test__/system/unplug`` then removes (a drive unplugged while the
page shows it);
does not exist (an unmounted drive);
- ``POST /__test__/system/missing-file`` deletes the newest linked image's
file from disk, leaving its database row (a missing-file link);
- ``POST /__test__/system/unthumbnailed`` ``{count}`` writes ``count`` plain
PNGs with no thumbnail into the output folder;
- ``POST /__test__/system/monitor`` ``{watching, alive}`` stands in a fake
image monitor whose observer is ``alive`` (default true) and schedules the
named folders of this server's root (``["output"]``), so Sources reads
watching, manual scan and restart pending states (S4).
PNGs with no thumbnail into the output folder.
Delays use the shared request holds (``request_hold.py``).
"""
@@ -38,31 +31,9 @@ MAX_UNTHUMBNAILED = 50
def register_system_commands(app, routes, api, root: Path):
# System › Appearance saves the host fields to this server's config.json (S3);
# reset restores both. The file is this server's own, inside its temp root:
# reset rewrites or deletes it, so a server whose configuration points
# anywhere else (a real config.json) refuses to register these hooks.
config_path = root / "config.json"
configured_path = os.environ.get("PROMPT_MANAGER_CONFIG_PATH")
if (
configured_path is None
or Path(configured_path).resolve() != config_path.resolve()
):
raise RuntimeError(
f"PROMPT_MANAGER_CONFIG_PATH must be {config_path}, not {configured_path!r}"
)
from py.config import GalleryConfig, PromptManagerConfig
from utils import image_monitor
from py.config import GalleryConfig
configured = list(GalleryConfig.MONITORING_DIRECTORIES)
config_bytes = config_path.read_bytes() if config_path.exists() else None
host_values = (
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
)
monitor = getattr(image_monitor, "_monitor_instance", None)
failures: dict[str, int] = {}
@web.middleware
@@ -77,28 +48,10 @@ def register_system_commands(app, routes, api, root: Path):
app.middlewares.append(inject_failures)
unplugged = root / "output" / "unplugged"
def reset():
failures.clear()
if unplugged.is_dir():
unplugged.rmdir()
GalleryConfig.MONITORING_DIRECTORIES = list(configured)
api._health_report = None
(
PromptManagerConfig.RESULT_TIMEOUT,
PromptManagerConfig.WEBUI_DISPLAY_MODE,
PromptManagerConfig.WORKER_THREADS,
GalleryConfig.MONITORING_ENABLED,
) = host_values
image_monitor._monitor_instance = monitor
if config_bytes is None:
config_path.unlink(missing_ok=True)
else:
# Replace in one step, as the server writes it (Windows preflight).
staged = config_path.with_name(config_path.name + ".reset")
staged.write_bytes(config_bytes)
os.replace(staged, config_path)
@routes.post("/__test__/system/reset")
async def system_reset(request):
@@ -122,26 +75,14 @@ def register_system_commands(app, routes, api, root: Path):
@routes.post("/__test__/system/missing-source")
async def missing_source(request):
data = await request.json() if request.can_read_body else {}
if data.get("inside") is True:
unplugged.mkdir(exist_ok=True)
absent = str(unplugged)
else:
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
absent = str(root / "unmounted-drive")
if os.path.exists(absent):
raise web.HTTPConflict()
GalleryConfig.MONITORING_DIRECTORIES = [*configured, absent]
return web.json_response(
{"success": True, "configured": len(GalleryConfig.MONITORING_DIRECTORIES)}
)
@routes.post("/__test__/system/unplug")
async def unplug(request):
if not unplugged.is_dir():
raise web.HTTPConflict()
unplugged.rmdir()
return web.json_response({"success": True})
@routes.post("/__test__/system/missing-file")
async def missing_file(request):
with api.db.model.get_connection() as conn:
@@ -172,35 +113,4 @@ def register_system_commands(app, routes, api, root: Path):
api.invalidate_gallery_cache()
return web.json_response({"success": True})
@routes.post("/__test__/system/monitor")
async def fake_monitor(request):
data = await request.json()
watching, alive = data.get("watching", []), data.get("alive", True)
if (
not isinstance(watching, list)
or not all(name in ("output", "input") for name in watching)
or not isinstance(alive, bool)
):
raise web.HTTPBadRequest()
image_monitor._monitor_instance = _FakeMonitor(
[str(root / name) for name in watching], alive
)
return web.json_response({"success": True})
return reset
class _FakeMonitor:
"""The image monitor's status surface (``get_status``) with no observer thread."""
def __init__(self, directories, alive):
self.monitored_directories = directories
self._alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": list(self.monitored_directories),
"handler_active": True,
"observer_alive": self._alive,
}
@@ -44,8 +44,6 @@ MAX_EXTRA_IMAGES = 160
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -114,8 +114,6 @@ def seed_workspace_visual(api: PromptManagerAPI, root: Path) -> None:
for table in (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -81,8 +81,6 @@ MAX_EXTRA_CATEGORIES = 200
_TABLES = (
"saved_views",
"image_favorites",
"media_review_decisions",
"media_review_items",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
+1 -10
View File
@@ -1,20 +1,11 @@
import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
import runtime from '../../tools/vendor/comfyui-runtime.json' with { type: 'json' };
import { hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
export function installHostHooks(test, expectedTypes = ['PromptManager', 'PromptManagerText', 'PromptSearchList']) {
test.beforeEach(async ({ page, context, request }, testInfo) => {
const response = await request.get('/system_stats');
expect(response.ok()).toBe(true);
const stats = await response.json();
// comfyui.config.mjs's launcher reports the host it owns. The execution suite has no
// web server: it starts its own host per worker behind a port refusal.
if (testInfo.config.webServer) {
const ownedBase = process.env.PM_COMFY_HOST_BASE;
expect(ownedBase, 'the launcher reported the host it started').toBeTruthy();
expect(hostBaseDirectory(stats), 'the host answering the port is the one this run started').toBe(ownedBase);
}
const info = stats.system;
const info = (await response.json()).system;
expect(info.comfyui_version).toBe(runtime.serverVersion);
expect(info.pytorch_version).toBe(`${runtime.torchVersion}+cpu`);
expect(info.required_frontend_version).toBe(runtime.frontendVersion);
+4 -4
View File
@@ -594,12 +594,12 @@
"anchor": null,
"expect": {
"metadata_status": "available",
"state": "resolved",
"reason": "",
"state": "ambiguous",
"reason": "Multiple sampler branches",
"reason_code": null,
"selection": null,
"positive": "a red fox in snow, detailed fur",
"negative": "blurry, lowres",
"positive": null,
"negative": null,
"candidate_count": 0,
"facts": {
"seed": {
File diff suppressed because one or more lines are too long
-17
View File
@@ -355,20 +355,3 @@ test("keepalive is forwarded only when asked, so a request can outlive its page"
for (const call of fetch.calls) call.resolve(jsonResponse(200, { success: true }));
await Promise.all(writes);
});
test("ApiError keeps a validated running job on busy and a validated source on invalid-source", () => {
const body = { running: { id: "a".repeat(32), title: "Scan for metadata" }, source: "output/gone" };
const busy = new ApiError(409, "busy", "busy", null, body);
assert.deepEqual([busy.running, busy.source], [{ id: "a".repeat(32), title: "Scan for metadata" }, null]);
assert.notEqual(busy.running, body.running, "server objects must be copied");
const invalid = new ApiError(400, "refused", "invalid-source", null, body);
assert.deepEqual([invalid.running, invalid.source], [null, "output/gone"]);
for (const running of [null, "x", { id: 1, title: "t" }, { id: "a b", title: "t" }, { id: "a", title: "" },
{ id: "a", title: "x".repeat(201) }, { id: "a".repeat(65), title: "t" }]) {
assert.equal(new ApiError(409, "busy", "busy", null, { running }).running, null);
}
for (const source of [null, 7, "", "x".repeat(4097)]) {
assert.equal(new ApiError(400, "refused", "invalid-source", null, { source }).source, null);
}
assert.deepEqual([new ApiError(409, "old caller").running, new ApiError(409, "old caller").source], [null, null]);
});
-47
View File
@@ -61,50 +61,3 @@ test("infinite scrolling is off by default and must be a boolean", () => {
assert.equal(normalizeGallerySettings({ infiniteScroll: true }).infiniteScroll, true);
assert.equal(normalizeGallerySettings({ infiniteScroll: "yes" }).infiniteScroll, false);
});
test("the workspace keys have their own defaults and validation", () => {
const { WORKSPACE_LAYOUTS, WORKSPACE_DENSITIES } = require("../../web/js/gallery-settings.js");
assert.equal(DEFAULT_SETTINGS.workspaceLayout, "grid");
assert.equal(DEFAULT_SETTINGS.workspaceDensity, "comfortable");
assert.deepEqual(WORKSPACE_LAYOUTS, ["grid", "list", "masonry"]);
assert.deepEqual(WORKSPACE_DENSITIES, ["comfortable", "large", "compact"]);
assert.equal(normalizeGallerySettings({ workspaceLayout: "masonry" }).workspaceLayout, "masonry");
assert.equal(normalizeGallerySettings({ workspaceLayout: "mosaic" }).workspaceLayout, "grid");
assert.equal(normalizeGallerySettings({ workspaceDensity: "large" }).workspaceDensity, "large");
assert.equal(normalizeGallerySettings({ workspaceDensity: 3 }).workspaceDensity, "comfortable");
// The legacy gallery's own view mode is separate: it has no masonry.
assert.equal(normalizeGallerySettings({ defaultViewMode: "masonry" }).defaultViewMode, "grid");
});
test("a legacy gallery Save keeps the workspace keys and every other stored key", () => {
const fs = require("node:fs");
const path = require("node:path");
const vm = require("node:vm");
const settings = require("../../web/js/gallery-settings.js");
const storage = new Map([["gallerySettings", JSON.stringify({ workspaceLayout: "masonry", workspaceDensity: "compact", someOtherTool: 1 })]]);
const controls = {
lazyLoadingToggle: { checked: true }, imageQualitySelect: { value: "high" }, defaultViewModeSelect: { value: "list" },
defaultLimitSelect: { value: "200" }, gridColumnsSelect: { value: "6" }, showImageInfoToggle: { checked: false },
autoLoadMetadataToggle: { checked: true }, cacheMetadataToggle: { checked: true }, showFilePathsToggle: { checked: false },
sidebarCollapsedByDefaultToggle: { checked: true }, infiniteScrollToggle: { checked: false }, debugModeToggle: { checked: false },
apiTimeoutInput: { value: "30" }, checkThumbnailsAtStartup: { checked: true }, videoAutoplayToggle: { checked: true },
videoMuteToggle: { checked: false }, videoLoopToggle: { checked: true },
};
const context = vm.createContext({
document: { addEventListener() {}, getElementById: id => controls[id] },
window: {}, console: { debug() {}, warn() {}, error() {} }, GallerySettings: settings,
localStorage: { getItem: key => storage.get(key) ?? null, setItem: (key, value) => storage.set(key, value) },
});
const file = path.join(__dirname, "../../web/js/gallery.js");
const Manager = vm.runInContext(`${fs.readFileSync(file, "utf8")}\nGalleryManager;`, context, { filename: file });
const manager = Object.create(Manager.prototype);
Object.assign(manager, { showNotification() {}, hideSettings() {}, applySettings() {} });
manager.saveSettings();
const saved = JSON.parse(storage.get("gallerySettings"));
assert.equal(saved.workspaceLayout, "masonry");
assert.equal(saved.workspaceDensity, "compact");
assert.equal(saved.someOtherTool, 1);
assert.equal(saved.defaultViewMode, "list");
assert.equal(saved.showImageInfo, false);
assert.equal(saved.videoMute, false);
});
-26
View File
@@ -82,32 +82,6 @@ test('a zeroed conditioning negative shows as an explicit empty negative without
}finally{scope.dispose();f.restore();}
});
test('a prompt that left out run-time trigger words says so under its text (D125)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div'),sent=[];
const note='Text added at run time, such as trigger words, is not included.';
try{
module.mountPromptBlocks(container,{parentScope:scope,sender:{open:roles=>sent.push(roles)},prompts:{positive:{status:'resolved',text:'own text',source:'runtime-parts-omitted'},negative:{status:'resolved',text:'blur',source:'graph-inferred'}}});
const notes=f.walk(container,node=>node.textContent===note);
assert.equal(notes.length,1,'one note, under the positive only');
assert.equal(notes[0].parent.attributes['aria-label'],'Positive prompt');
container.click(f.walk(container,node=>node.dataset.sendRole==='pair')[0]);
assert.deepEqual(sent,[{positive:'own text',negative:'blur'}],'the note is never sent');
}finally{scope.dispose();f.restore();}
});
test('a prompt a later stage added conditioning to says so, and both lines show together (D126)',()=>{
const f=dom(),scope=createScope(),container=new f.Element('div');
const added='Conditioning added at a later stage is not included.', runtime='Text added at run time, such as trigger words, is not included.';
try{
const view=module.mountPromptBlocks(container,{parentScope:scope,prompts:{positive:{status:'resolved',text:'base',source:'added-conditioning'}}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[added]);
view.update({positive:{status:'resolved',text:'base',source:'runtime-parts-omitted-and-added-conditioning'}});
assert.deepEqual(f.walk(container,node=>node.className==='small muted').map(node=>node.textContent),[runtime,added]);
view.update({positive:{status:'resolved',text:'base',source:'graph-inferred'}});
assert.equal(f.walk(container,node=>node.className==='small muted').length,0);
}finally{scope.dispose();f.restore();}
});
const unresolved=message=>({status:'unresolved',text:null,source:null,...(message===undefined?{}:{message})});
const statusLines=(f,container)=>f.walk(container,node=>node.tagName==='p'&&node.className==='prompt-text muted').map(node=>node.textContent);
test('unsupported node status names the class',()=>{
-16
View File
@@ -478,19 +478,3 @@ test('a refresh deferred behind a page load is dropped when the stream is dispos
await new Promise(resolve=>setImmediate(resolve));
assert.equal(requests.length,2,'no refresh runs after dispose');
});
test('defaults apply to a new visit but not to the current view or a saved view', async () => {
const { initialImagesView } = await source;
assert.equal(typeof initialImagesView, 'function', 'initialImagesView is exported');
const preferences = { workspaceLayout: 'masonry', workspaceDensity: 'compact' };
// A new visit starts from System › Appearance.
assert.deepEqual(initialImagesView({ preferences }), { layout: 'masonry', density: 'compact' });
// The current view (the last visit) keeps its own toolbar choices.
assert.deepEqual(initialImagesView({ prior: { layout: 'list', density: 'large' }, preferences }), { layout: 'list', density: 'large' });
assert.deepEqual(initialImagesView({ prior: {}, preferences }), { layout: 'grid', density: 'comfortable' });
// A saved view opens as it always did.
assert.deepEqual(initialImagesView({ saved: { q: '' }, preferences }), { layout: 'grid', density: 'comfortable' });
// Without readable preferences nothing changes.
assert.deepEqual(initialImagesView({}), { layout: 'grid', density: 'comfortable' });
assert.deepEqual(initialImagesView({ preferences: {} }), { layout: 'grid', density: 'comfortable' });
});
@@ -825,73 +825,3 @@ test('refresh re-reads the selected file and keeps the shown record until the ne
assert.equal(controller.state.record.identity.path, 'folder/b.png');
controller.dispose(); assert.equal(await controller.refresh(), undefined);
});
test('metadata off shows an explicit load and loads once on request', async () => {
const { api, requests } = pendingApi(), updates = [];
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
controller.subscribe(state => updates.push(state));
// Selecting reads nothing: the state says the metadata waits for a request.
assert.equal(controller.select(image()), undefined);
assert.equal(requests.length, 0);
assert.equal(controller.state.deferred, true);
assert.equal(controller.state.loading, false);
assert.equal(controller.state.record, null);
// Selecting the same file again still reads nothing.
controller.select({ ...image() });
assert.equal(requests.length, 0);
const load = controller.load();
assert.equal(requests.length, 1);
assert.equal(controller.state.loading, true);
assert.equal(controller.state.deferred, false);
// A second press while it reads, and the same file selected again once it loaded, read nothing more.
controller.load();
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await load;
assert.equal(controller.state.record.generation.seed.value, '9007199254740993');
controller.load(); controller.select(image());
assert.equal(requests.length, 1);
// Another file waits for its own request again.
controller.select({ ...image(), path: 'folder/b.png', filename: 'b.png', url: image().url.replace('a%20%23%25', 'b') });
assert.equal(controller.state.deferred, true);
assert.equal(requests.length, 1);
// The default controller loads on selection, as before.
const eager = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts });
eager.select(image());
assert.equal(requests.length, 2);
assert.equal(eager.state.deferred, false);
eager.dispose(); controller.dispose();
assert.equal(controller.load(), undefined);
});
test('cache off rereads the current selection; on reuses the one current record', async () => {
for (const cache of [true, false]) {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache });
const first = controller.select(image());
// A read in flight is shared either way.
controller.select(image());
assert.equal(requests.length, 1);
requests[0].resolve(envelope()); await first;
const again = controller.select(image());
assert.equal(requests.length, cache ? 1 : 2, `cache ${cache}`);
if (!cache) { requests[1].resolve(envelope()); await again; assert.equal(controller.state.record.identity.path, 'folder/a #%.png'); }
controller.dispose();
}
});
test('cache off never rereads a selection whose source changed, and refresh never loads a deferred one', async () => {
const { api, requests } = pendingApi();
const controller = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, cache: false });
const first = controller.select(image());
requests[0].reject(new ApiError(409, refreshMessage, 'source-changed')); await first;
assert.equal(controller.state.refreshRequired, true);
controller.select(image());
assert.equal(requests.length, 1);
controller.dispose();
const deferred = contract('createInspectionController')({ api, extractFacts: extractGenerationFacts, autoLoad: false });
deferred.select(image());
assert.equal(deferred.refresh(), undefined);
assert.equal(requests.length, 1);
assert.equal(deferred.state.deferred, true);
deferred.dispose();
});
-11
View File
@@ -136,17 +136,6 @@ test('an unusable semantics table is refused', () => {
for (const doc of [null, {schema: 2}, {schema: 1, samplers: []}]) assert.throws(() => resolver.compileSemantics(doc), resolver.SemanticsError);
});
test('the lineage section is compiled, and a bad one refuses the table', () => {
const doc = JSON.parse(fs.readFileSync(new URL('../../web/js/inspection-semantics.json', import.meta.url), 'utf8'));
assert.deepEqual(semantics.lineage.get('Image Filter\u00001'), { strategy: 'continue', input: 'latents' });
// Lineage describes no prompt: its classes are not table classes for the stage walk.
assert.equal(semantics.allClasses.has('VAEEncode'), false);
const row = { class: 'VAEEncode', slot: 0, strategy: 'continue', input: 'pixels' };
for (const lineage of [undefined, {}, [null], [{ ...row, slot: -1 }], [{ ...row, strategy: 'passthrough' }], [{ ...row, input: '' }], [row, row]]) {
assert.throws(() => resolver.compileSemantics({ ...doc, lineage }), resolver.SemanticsError, JSON.stringify(lineage));
}
});
// The committed server answers (tests/fixtures/local-inspect/resolver-vectors.json, written by
// `python3 -m tests.local_resolver_cases --write` and checked current by the Python suite).
test('committed server answers equal the local evidence', () => {
+5 -38
View File
@@ -124,51 +124,18 @@ test('real inspection navigation preserves slideshow across two advances and def
} finally {view.dispose();controller.dispose();parentScope.dispose();globalThis.setTimeout=oldSetTimeout;globalThis.clearTimeout=oldClearTimeout;f.restore();}
});
test('Viewer.js is built only on a loaded image and an unbuilt viewer is released at once', async () => {
const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs');
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c'),occurrenceMedia('2',0,'d')];
const instances = [];
// On a loaded image Viewer.js schedules its inline build; until it runs `ready` is false and destroy() cancels it.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = false; this.viewed = false; instances.push(this); }
destroy() { this.destroyed++; } resize() {} };
const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame};
Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}});
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}});
try {
const stageImage = () => f.walk(f.container,n => n.tagName === 'img')[0];
view.open({items,selectedKey:'d'.repeat(64)});
// Viewer.js's own cancel misses an image whose load event is still queued, so none exists before the load.
const unloaded = stageImage(), dropped = [];unloaded.removeAttribute = key => dropped.push(key);
assert.equal(instances.length, 0, 'no Viewer.js before the image loads');
f.walk(f.container,n => n.attributes['aria-label'] === 'Previous media')[0].fire('click');
assert.deepEqual(dropped, ['src'], 'an image left before it loads is released at once');
unloaded.fire('load');
assert.equal(instances.length, 0, 'its late load builds no Viewer.js');
stageImage().fire('load');
const [loading] = instances, image = loading.node, removed = [];
image.removeAttribute = key => removed.push(key);
// Next drops the image whose build is pending; a deferred destroy would let that build run there.
f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click');
stageImage().fire('load');assert.equal(instances.length, 2, 'the next image builds its own Viewer.js once loaded');
assert.equal(loading.destroyed, 1, 'an unbuilt viewer is destroyed at once');
assert.deepEqual(removed, ['src'], 'and its image is released at once');
loading.options.viewed.call(loading);
assert.equal(loading.destroyed, 1, 'never destroyed twice');
} finally { view.dispose(); parentScope.dispose(); f.restore(); Object.assign(globalThis, globals); }
});
test('an inline Viewer still loading its image is destroyed only after it has viewed it', async () => {
const {reportedLocationAdapter:adapter} = await import('../../web/js/workspace/history-media.mjs');
const f = mountedDom(), parentScope = createScope(), controller = borrowedController(), items = [occurrenceMedia('1',1,'c')];
const instances = [];
// A built (`ready`) inline Viewer.js's destroy() leaves an in-flight view running; its callback then needs a live instance.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.ready = true; this.viewed = false; instances.push(this); }
// Viewer.js's inline destroy() leaves an in-flight view running; its callback then needs a live instance.
f.win.Viewer = class { constructor(node,options) { this.node = node; this.options = options; this.destroyed = 0; this.viewed = false; instances.push(this); }
destroy() { this.destroyed++; } resize() {} };
const globals = {ResizeObserver:globalThis.ResizeObserver, requestAnimationFrame:globalThis.requestAnimationFrame, cancelAnimationFrame:globalThis.cancelAnimationFrame};
Object.assign(globalThis, {ResizeObserver:class { observe() {} disconnect() {} }, requestAnimationFrame:() => 1, cancelAnimationFrame() {}});
const view = module.createMediaViewer({container:f.container,parentScope,inspectionController:controller,adapter,sender:{open(){}}});
try {
view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load');
view.open({items,selectedKey:'c'.repeat(64)});
const [loading] = instances, image = loading.node, removed = [];
image.removeAttribute = key => removed.push(key);
f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click');
@@ -181,7 +148,7 @@ test('an inline Viewer still loading its image is destroyed only after it has vi
loading.options.viewed.call(loading);
assert.equal(loading.destroyed, 1, 'never destroyed twice');
view.open({items,selectedKey:'c'.repeat(64)});f.walk(f.container,n => n.tagName === 'img')[0].fire('load');
view.open({items,selectedKey:'c'.repeat(64)});
const shown = instances[1]; shown.viewed = true; shown.options.viewed.call(shown);
f.walk(f.container,n => n.attributes['aria-label'] === 'Close image viewer')[0].fire('click');
assert.equal(shown.destroyed, 1, 'a viewed image is destroyed at once');
@@ -269,7 +236,7 @@ test('viewer uses adapter.mediaUrl for image and video', async () => {
const img = f.walk(f.container,n => n.tagName === 'img')[0];
assert.equal(img.src, image.url, 'the blob URL is used exactly');
assert.doesNotMatch(img.src, /\?/);
img.fire('load');assert.equal(instances[0].node, img, 'Viewer.js shows the blob image');
assert.equal(instances[0].node, img, 'Viewer.js shows the blob image');
const download = f.walk(f.container,n => n.attributes['aria-label'] === 'Download original')[0];
assert.equal(download.href, image.url);assert.equal(download.download, 'a.png');
f.walk(f.container,n => n.attributes['aria-label'] === 'Next media')[0].fire('click');
-193
View File
@@ -1,193 +0,0 @@
// Browser display defaults (#249 System S3 Task 7): the workspace reads and writes the legacy gallery's own
// `gallerySettings` object (Q3, "One display configuration"), writes only the keys it changed, never consumes the
// legacy paging keys, and says when this browser refused to keep a change.
import test from 'node:test';
import assert from 'node:assert/strict';
import gallerySettings from '../../web/js/gallery-settings.js';
const module = await import('../../web/js/workspace/preferences.mjs').catch(error => {
if (error.code !== 'ERR_MODULE_NOT_FOUND') throw error;
return {};
});
globalThis.GallerySettings = gallerySettings;
const KEY = 'gallerySettings';
/** A localStorage stand-in holding `text` under the settings key; `setItem` can be made to throw. */
function storage(text = null, { quota = false } = {}) {
const writes = [];
const store = {
text,
writes,
getItem(key) { assert.equal(key, KEY); return store.text; },
setItem(key, value) {
assert.equal(key, KEY);
writes.push(value);
if (quota) { const error = new Error('Quota exceeded'); error.name = 'QuotaExceededError'; throw error; }
store.text = value;
},
removeItem() { assert.fail('Preferences never remove the shared settings'); },
};
return store;
}
const denied = () => { const error = new Error('Denied'); error.name = 'SecurityError'; throw error; };
test('normalizes new workspace keys and keeps legacy keys', () => {
assert.equal(typeof module.readPreferences, 'function', 'preferences.mjs exports readPreferences');
module.resetTabPreferences();
const legacy = { defaultViewMode: 'list', gridColumns: 5, defaultLimit: 200, thumbnailPromptDismissed: 'sig', extra: 1 };
const store = storage(JSON.stringify({ ...legacy, workspaceLayout: 'masonry', workspaceDensity: 'compact', showFilePaths: false }));
const { values, persisted } = module.readPreferences(() => store);
assert.equal(persisted, true);
assert.equal(values.workspaceLayout, 'masonry');
assert.equal(values.workspaceDensity, 'compact');
assert.equal(values.showFilePaths, false);
// Invalid workspace values fall back to the defaults, as every legacy key does.
store.text = JSON.stringify({ workspaceLayout: 'mosaic', workspaceDensity: 7, videoMute: 'no' });
const fallback = module.readPreferences(() => store).values;
assert.equal(fallback.workspaceLayout, 'grid');
assert.equal(fallback.workspaceDensity, 'comfortable');
assert.equal(fallback.videoMute, true);
// The legacy normalizer knows the new keys and still keeps unknown ones.
const normalized = gallerySettings.normalizeGallerySettings({ ...legacy, workspaceLayout: 'list' });
assert.equal(normalized.workspaceLayout, 'list');
assert.equal(normalized.workspaceDensity, 'comfortable');
assert.equal(normalized.extra, 1);
assert.equal(normalized.thumbnailPromptDismissed, 'sig');
assert.deepEqual(gallerySettings.WORKSPACE_LAYOUTS, ['grid', 'list', 'masonry']);
assert.deepEqual(gallerySettings.WORKSPACE_DENSITIES, ['comfortable', 'large', 'compact']);
});
test('write touches only dirty keys', () => {
module.resetTabPreferences();
const stored = { defaultViewMode: 'list', gridColumns: 5, thumbnailsGenerated: true, extra: { kept: true }, videoMute: false };
const store = storage(JSON.stringify(stored));
const result = module.writePreferences({ workspaceLayout: 'masonry', showImageInfo: false }, () => store);
assert.deepEqual(result, { persisted: true, conflicts: [] });
assert.equal(store.writes.length, 1);
// Every other key, legacy or unknown, is written back exactly as it was stored: no defaults are filled in.
assert.deepEqual(JSON.parse(store.text), { ...stored, workspaceLayout: 'masonry', showImageInfo: false });
// It re-reads the stored object, so a key another page wrote after this one read is kept.
store.text = JSON.stringify({ ...JSON.parse(store.text), gridColumns: 9 });
module.writePreferences({ videoLoop: false }, () => store);
assert.equal(JSON.parse(store.text).gridColumns, 9);
assert.equal(JSON.parse(store.text).workspaceLayout, 'masonry');
// An empty or malformed stored value starts from an empty object, never the defaults.
for (const text of [null, '{broken', '[]', '42']) {
const fresh = storage(text);
assert.equal(module.writePreferences({ cacheMetadata: false }, () => fresh).persisted, true);
assert.deepEqual(JSON.parse(fresh.text), { cacheMetadata: false });
}
// Only workspace keys with valid values are accepted; nothing is written for a bad change.
for (const bad of [{ gridColumns: 4 }, { defaultLimit: 50 }, { workspaceLayout: 'mosaic' }, { videoMute: 'yes' }, { imageQuality: 'ultra' }]) {
const untouched = storage(JSON.stringify(stored));
assert.throws(() => module.writePreferences(bad, () => untouched), TypeError);
assert.equal(untouched.writes.length, 0);
}
// A key changed elsewhere since this tab read it (its baseline) is reported, and the explicit save still wins.
const elsewhere = storage(JSON.stringify({ videoLoop: false }));
const outcome = module.writePreferences({ videoLoop: true, videoMute: false }, () => elsewhere, { baseline: { videoLoop: true, videoMute: true } });
assert.deepEqual(outcome, { persisted: true, conflicts: ['videoLoop'] });
assert.equal(JSON.parse(elsewhere.text).videoLoop, true);
});
test('denied storage reports tab-only', () => {
module.resetTabPreferences();
const read = module.readPreferences(denied);
assert.equal(read.persisted, false);
assert.equal(read.values.workspaceLayout, 'grid');
const result = module.writePreferences({ workspaceLayout: 'list', videoAutoplay: true }, denied);
assert.deepEqual(result, { persisted: false, conflicts: [] });
// The change is kept for this tab: every reader in it (a new Images visit, the viewer) sees it.
const tab = module.readPreferences(denied);
assert.equal(tab.persisted, false);
assert.equal(tab.values.workspaceLayout, 'list');
assert.equal(tab.values.videoAutoplay, true);
// A getItem that throws is denied storage too.
const throwing = { getItem: denied, setItem: denied };
assert.equal(module.readPreferences(() => throwing).values.workspaceLayout, 'list');
assert.equal(module.writePreferences({ workspaceDensity: 'large' }, () => throwing).persisted, false);
// Once storage accepts a write, the tab copy of that key gives way to the stored value.
const working = storage(JSON.stringify({ workspaceLayout: 'masonry' }));
assert.equal(module.writePreferences({ workspaceLayout: 'masonry' }, () => working).persisted, true);
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
assert.equal(module.readPreferences(denied).values.videoAutoplay, false);
});
test('quota error keeps previous bytes', () => {
module.resetTabPreferences();
const before = JSON.stringify({ gridColumns: 6, workspaceDensity: 'large' });
const store = storage(before, { quota: true });
const result = module.writePreferences({ workspaceDensity: 'compact' }, () => store);
assert.deepEqual(result, { persisted: false, conflicts: [] });
assert.equal(store.writes.length, 1);
assert.equal(store.text, before);
// This tab keeps the change it could not store.
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'compact');
module.resetTabPreferences();
assert.equal(module.readPreferences(() => store).values.workspaceDensity, 'large');
});
/** A window stand-in that records its storage listeners. */
function fakeWindow() {
const listeners = new Set();
return {
listeners,
addEventListener(type, listener) { assert.equal(type, 'storage'); listeners.add(listener); },
removeEventListener(type, listener) { assert.equal(type, 'storage'); listeners.delete(listener); },
fire(event) { for (const listener of [...listeners]) listener(event); },
};
}
test('storage event updates clean values and flags dirty conflicts', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ workspaceLayout: 'list', videoMute: false }));
const win = fakeWindow();
const seen = [];
const unsubscribe = module.subscribePreferences(win, next => seen.push(next), () => store);
assert.equal(win.listeners.size, 1);
// Another key, or another storage area's event, is not ours.
win.fire({ key: 'pm_update_notice_dismissed', storageArea: store });
win.fire({ key: KEY, storageArea: { other: true } });
assert.equal(seen.length, 0);
win.fire({ key: KEY, storageArea: store });
win.fire({ key: null, storageArea: store }); // localStorage.clear() elsewhere
assert.equal(seen.length, 2);
assert.equal(seen[0].values.workspaceLayout, 'list');
unsubscribe();
assert.equal(win.listeners.size, 0);
// The reconciliation of the System drafts: clean values follow the other page; a key this tab has a draft for and
// the other page changed is a conflict, and the draft stays.
const baseline = { workspaceLayout: 'grid', videoMute: true, videoLoop: true };
const next = { workspaceLayout: 'list', videoMute: true, videoLoop: false };
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { videoMute: false, videoLoop: false }, next }), { conflicts: [] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'masonry' }, next }), { conflicts: ['workspaceLayout'] });
assert.deepEqual(module.reconcilePreferences({ baseline, drafts: { workspaceLayout: 'list' }, next }), { conflicts: [] });
});
test('legacy paging keys are never consumed by the workspace', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ defaultLimit: 500, gridColumns: 12, infiniteScroll: true, lazyLoading: false,
defaultViewMode: 'list', debugMode: true, apiTimeout: 9, thumbnailsGenerated: true, checkThumbnailsAtStartup: false }));
const { values } = module.readPreferences(() => store);
assert.deepEqual(Object.keys(values).sort(), [...module.WORKSPACE_PREFERENCE_KEYS].sort());
for (const key of ['defaultLimit', 'gridColumns', 'infiniteScroll', 'lazyLoading', 'defaultViewMode', 'debugMode', 'apiTimeout',
'thumbnailsGenerated', 'checkThumbnailsAtStartup', 'thumbnailPromptDismissed']) assert.equal(Object.hasOwn(values, key), false, key);
// The display defaults are the gallery's own defaults for these keys.
const defaults = module.preferenceDefaults();
for (const key of module.WORKSPACE_PREFERENCE_KEYS) assert.equal(defaults[key], gallerySettings.DEFAULT_SETTINGS[key], key);
});
test('a later save flushes the changes kept for this tab only', () => {
module.resetTabPreferences();
const store = storage(JSON.stringify({ gridColumns: 4 }), { quota: true });
assert.equal(module.writePreferences({ workspaceLayout: 'list' }, () => store).persisted, false);
// Storage accepts writes again: saving another key stores the tab-only change with it, so "Saved" is true for both.
const working = storage(store.text);
assert.deepEqual(module.writePreferences({ videoLoop: false }, () => working), { persisted: true, conflicts: [] });
assert.deepEqual(JSON.parse(working.text), { gridColumns: 4, workspaceLayout: 'list', videoLoop: false });
// Nothing is overlaid any more: another page's later change reads through.
working.text = JSON.stringify({ ...JSON.parse(working.text), workspaceLayout: 'masonry' });
assert.equal(module.readPreferences(() => working).values.workspaceLayout, 'masonry');
module.resetTabPreferences();
});
@@ -1,188 +0,0 @@
// System › Appearance (#249 System S3 Task 8): the concept's "Browsing & display" rows (app.js 676–678) on the drafts
// of the one Save row, their stored values (Q3 mappings), the host timeout's 0–300 whole seconds and the D97 rows.
const test = require('node:test');
const assert = require('node:assert/strict');
const gallerySettings = require('../../web/js/gallery-settings.js');
const appearanceModule = import('../../web/js/workspace/system-appearance.mjs');
const stateModule = import('../../web/js/workspace/system-state.mjs');
const coreModule = import('../../web/js/shared/toolkit/core.js');
globalThis.GallerySettings = gallerySettings;
/** A minimal DOM whose elements count their live listeners (a listener's signal aborts it, as in a browser). */
function fakeDOM() {
const previous = global.document;
const elements = [];
class Element {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.attributes = {};
this.listeners = [];
this.dataset = {};
this.className = '';
this.id = '';
this.value = '';
this._text = '';
elements.push(this);
}
get textContent() { return this._text + this.children.map(child => child.textContent).join(''); }
set textContent(text) { this.children = []; this._text = String(text); }
append(...nodes) {
for (const node of nodes) {
const child = typeof node === 'string' ? Object.assign(new Element('#text'), { _text: node }) : node;
child.parentNode = this;
this.children.push(child);
}
}
remove() { if (this.parentNode) this.parentNode.children = this.parentNode.children.filter(child => child !== this); this.parentNode = null; }
replaceChildren(...nodes) { this.children = []; this._text = ''; this.append(...nodes); }
setAttribute(name, value) { this.attributes[name] = String(value); }
getAttribute(name) { return Object.hasOwn(this.attributes, name) ? this.attributes[name] : null; }
removeAttribute(name) { delete this.attributes[name]; }
addEventListener(type, callback, options = {}) {
if (options.signal?.aborted) return;
const entry = { type, callback };
this.listeners.push(entry);
options.signal?.addEventListener('abort', () => { this.listeners = this.listeners.filter(item => item !== entry); });
}
removeEventListener(type, callback) { this.listeners = this.listeners.filter(item => item.type !== type || item.callback !== callback); }
dispatch(type) { for (const { callback } of this.listeners.filter(item => item.type === type)) callback({ type, target: this }); }
click() { this.dispatch('click'); }
}
global.document = {
createElement: tag => new Element(tag),
createTextNode: text => Object.assign(new Element('#text'), { _text: String(text) }),
};
const walk = (node, found = []) => { found.push(node); node.children.forEach(child => walk(child, found)); return found; };
return {
listenerCount: () => elements.reduce((sum, element) => sum + element.listeners.length, 0),
find: (root, predicate) => walk(root).find(predicate),
tags: node => node.children.map(child => child.tagName.toLowerCase()),
restore() { global.document = previous; },
};
}
const BROWSER = { workspaceLayout: 'grid', workspaceDensity: 'comfortable', autoLoadMetadata: true, cacheMetadata: true,
imageQuality: 'medium', videoAutoplay: false, videoMute: true, videoLoop: true, showFilePaths: true, showImageInfo: true,
sidebarCollapsedByDefault: false };
async function mounted({ host = { webui_display_mode: 'newtab', result_timeout: 5 }, browser = BROWSER, hostFails = false } = {}) {
const { mountAppearance } = await appearanceModule;
const { createSystemDrafts } = await stateModule;
const { createScope } = await coreModule;
const dom = fakeDOM();
let hostValues = null;
const drafts = createSystemDrafts({
readBrowser: () => ({ ...browser }), readRuntime: () => ({}),
readHost: async () => { if (hostFails) throw new Error('offline'); return { values: hostValues, revision: 'sha256:one' }; },
writeBrowser: async () => ({ persisted: true }), writeHost: async () => ({ success: true, persisted: true }), writeRuntime: async () => ({}),
});
await drafts.load('browser');
const scope = createScope();
const [surface] = mountAppearance(null, { scope, drafts });
const byId = id => dom.find(surface, node => node.dataset?.setting === id);
return { dom, drafts, scope, surface, byId, loadHost: async () => { hostValues = host; await drafts.load('host'); },
fixHost: () => { hostFails = false; } };
}
test('the surface is the concept section with the concept rows in order, then the D97 rows', async () => {
const { APPEARANCE_ROWS } = await appearanceModule;
const { dom, surface, scope } = await mounted();
try {
assert.equal(surface.className, 'surface');
assert.equal(surface.children[0].textContent, 'Browsing & display');
assert.equal(surface.children[1].textContent, 'One set of defaults; toolbar changes apply to the current view.');
const rows = surface.children.slice(2);
const titles = rows.map(row => row.children[0].children[0].textContent);
assert.deepEqual(titles, ['Default layout', 'Thumbnail size', 'Continuous browsing', 'Load metadata on selection',
'Cache metadata in this browser', 'Image loading quality', 'Autoplay videos', 'Mute videos by default', 'Loop videos',
'Show file paths', 'Open from ComfyUI', 'Node search auto-hide (seconds)', 'Show image details on cards',
'Collapse viewer details by default', 'Reset display defaults']);
const continuous = rows[2];
assert.deepEqual(dom.tags(continuous.children[0]), ['h3', 'p']);
assert.equal(continuous.children[1].className, 'chip blue');
assert.equal(continuous.children[1].textContent, 'Infinite scroll');
assert.equal(rows.at(-1).className, 'operation');
assert.equal(APPEARANCE_ROWS.filter(Array.isArray).length, 13);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
test('added rows reach their consumers: each control edits its stored key with the Q3 values', async () => {
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
// Host rows stay disabled until the server settings answered.
assert.equal(byId('open-mode').disabled, true);
assert.equal(byId('timeout').disabled, true);
await loadHost();
assert.equal(byId('open-mode').disabled, false);
assert.equal(byId('timeout').value, '5');
assert.deepEqual(byId('image-quality').children.map(option => [option.value, option.textContent]),
[['medium', 'Optimized'], ['high', 'Original'], ['low', 'Fast']]);
assert.deepEqual(byId('open-mode').children.map(option => [option.value, option.textContent]),
[['newtab', 'New tab'], ['popup', 'Popup window']]);
const edits = [['default-view', 'masonry'], ['density', 'large'], ['image-quality', 'low'], ['open-mode', 'popup']];
for (const [id, value] of edits) { byId(id).value = value; byId(id).dispatch('change'); }
for (const id of ['metadata-auto', 'metadata-cache', 'video-muted', 'video-loop', 'paths', 'card-details']) {
byId(id).checked = false; byId(id).dispatch('change');
}
for (const id of ['video-auto', 'viewer-details']) { byId(id).checked = true; byId(id).dispatch('change'); }
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'masonry', workspaceDensity: 'large', imageQuality: 'low',
autoLoadMetadata: false, cacheMetadata: false, videoMute: false, videoLoop: false, showFilePaths: false, showImageInfo: false,
videoAutoplay: true, sidebarCollapsedByDefault: true });
assert.deepEqual(drafts.changes('host'), { webui_display_mode: 'popup' });
} finally { scope.dispose(); dom.restore(); }
});
test('the auto-hide timeout takes whole seconds from 0 to 300 only', async () => {
const { resultTimeout } = await appearanceModule;
assert.deepEqual(['0', '30', ' 300 ', '301', '-1', '2.5', '', 'abc', '1e2'].map(resultTimeout), [0, 30, 300, null, null, null, null, null, null]);
const { dom, drafts, scope, byId, loadHost } = await mounted();
try {
await loadHost();
const timeout = byId('timeout');
timeout.value = '45'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 45 });
timeout.value = '301'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), {});
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
timeout.value = '0'; timeout.dispatch('change');
assert.deepEqual(drafts.changes('host'), { result_timeout: 0 });
assert.deepEqual([...drafts.failed('host')], []);
} finally { scope.dispose(); dom.restore(); }
});
test('reset restores defaults without touching other keys', async () => {
const { DISPLAY_KEYS } = await appearanceModule;
const changed = { ...BROWSER, workspaceLayout: 'list', videoMute: false, showImageInfo: false, sidebarCollapsedByDefault: true };
const { dom, drafts, scope, surface, loadHost } = await mounted({ browser: changed });
try {
await loadHost();
drafts.set('host', 'result_timeout', 9);
const reset = dom.find(surface.children.at(-1), node => node.tagName === 'BUTTON');
assert.equal(reset.textContent, 'Reset');
reset.click();
assert.deepEqual(drafts.changes('browser'), { workspaceLayout: 'grid', videoMute: true, showImageInfo: true, sidebarCollapsedByDefault: false });
// The host fields and every key outside the display defaults are left as they are.
assert.deepEqual(drafts.changes('host'), { result_timeout: 9 });
assert.deepEqual([...DISPLAY_KEYS].sort(), Object.keys(BROWSER).sort());
} finally { scope.dispose(); dom.restore(); }
});
test('host rows that could not be read stay disabled and say so; a later read clears it', async () => {
const { HOST_UNAVAILABLE } = await appearanceModule;
const { dom, drafts, scope, byId, surface, fixHost } = await mounted({ hostFails: true });
const notes = () => surface.children.slice(2).flatMap(row => row.children[0].children.slice(2).map(note => [row.children[0].children[0].textContent, note.className, note.textContent]));
try {
assert.deepEqual(notes(), []);
assert.equal(await drafts.reloadHost(), false);
assert.deepEqual(notes(), [['Open from ComfyUI', 'small', HOST_UNAVAILABLE], ['Node search auto-hide (seconds)', 'small', HOST_UNAVAILABLE]]);
assert.equal(byId('timeout').disabled, true);
// Repaints keep one line per row.
drafts.set('browser', 'videoMute', false);
assert.equal(notes().length, 2);
fixHost();
assert.equal(await drafts.reloadHost(), true);
assert.deepEqual(notes(), []);
} finally { scope.dispose(); assert.equal(dom.listenerCount(), 0); dom.restore(); }
});
-145
View File
@@ -1,145 +0,0 @@
// System › Sources & scans (#249 System S4 Task 11): the concept's "Image sources" list (app.js 679–681) built from the
// server's configured, effective and watched sources (GET /settings), the host draft of `gallery_root_paths`, the
// 3.2.0 update notice's dismissal and the Refresh files toast (D98).
const test = require('node:test');
const assert = require('node:assert/strict');
const sourcesModule = import('../../web/js/workspace/system-sources.mjs');
const STATUS = Object.freeze({
configured: [
{ label: 'ComfyUI output', path: 'output', valid: true, code: null, reason: null },
{ label: 'ComfyUI input', path: 'input', valid: true, code: null, reason: null },
],
effective: [
{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'watching' },
{ label: 'ComfyUI input', path: 'input', root_index: 1, watch: 'manual' },
],
auto_detected: false,
});
test('each effective source is the concept row: label, "<path> · watching|manual scan", Scan changes or Scan folder', async () => {
const { sourceRows } = await sourcesModule;
const saved = ['output', 'input'];
const { rows, note } = sourceRows({ sources: STATUS, saved, current: saved });
assert.equal(note, null);
assert.deepEqual(rows, [
{ key: 'output', title: 'ComfyUI output', description: 'output · watching', scan: { label: 'Scan changes', root: 0, path: 'output' },
remove: 'output' },
{ key: 'input', title: 'ComfyUI input', description: 'input · manual scan', scan: { label: 'Scan folder', root: 1, path: 'input' },
remove: 'input' },
]);
});
test('watching only with a live observer; restart pending otherwise (D98)', async () => {
const { sourceRows, WATCH_COPY } = await sourcesModule;
assert.deepEqual(WATCH_COPY, { watching: 'watching', stopping: 'watching until restart', manual: 'manual scan',
restart_pending: 'restart pending' });
const sources = { ...STATUS, effective: STATUS.effective.map(source => ({ ...source, watch: 'restart_pending' })) };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · restart pending', 'Scan folder'], ['input · restart pending', 'Scan folder']]);
});
test('a configured source that cannot be used shows the server reason and only Remove', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'unmounted-drive', path: 'unmounted-drive', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const saved = ['output', 'input', 'unmounted-drive'];
const { rows } = sourceRows({ sources, saved, current: saved });
assert.deepEqual(rows.at(-1), { key: 'unmounted-drive', title: 'unmounted-drive',
description: 'unmounted-drive · Gallery root does not exist', scan: null, remove: 'unmounted-drive' });
});
test('an added source is listed as not saved yet with its validated label; a removed one leaves the list', async () => {
const { sourceRows } = await sourcesModule;
const labels = new Map([['output/archive', 'archive']]);
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['input', 'output/archive'], labels });
assert.equal(note, null);
assert.deepEqual(rows.map(row => [row.title, row.description, row.scan?.label ?? null, row.remove]), [
['ComfyUI input', 'input · manual scan', 'Scan folder', 'input'],
['archive', 'output/archive · not saved yet', null, 'output/archive'],
]);
// Without a recorded label (a reload of System), the folder name stands in.
assert.equal(sourceRows({ sources: STATUS, saved: [], current: ['output/new-collection'] }).rows[0].title, 'new-collection');
});
test('removing the last entry states auto-detect (D98)', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const { rows, note } = sourceRows({ sources: STATUS, saved: ['output', 'input'], current: [] });
assert.deepEqual(rows, []);
assert.equal(note, NOTES.removedAll);
assert.equal(NOTES.removedAll, 'No source is configured. After saving, ComfyUI’s output folder is used.');
});
test('auto-detected sources are listed without Remove, with the line that says how they were chosen', async () => {
const { sourceRows, NOTES } = await sourcesModule;
const auto = { configured: [], effective: [{ label: 'ComfyUI output', path: 'output', root_index: 0, watch: 'manual' }], auto_detected: true };
assert.deepEqual(sourceRows({ sources: auto, saved: [], current: [] }), { rows: [
{ key: 'auto:0', title: 'ComfyUI output', description: 'output · manual scan', scan: { label: 'Scan folder', root: 0, path: 'output' },
remove: null },
], note: NOTES.autoDetected });
// Every configured source unusable: ComfyUI's output folder is used instead.
const fallback = { ...auto, configured: [{ label: 'gone', path: 'gone', valid: false, code: 'not-found', reason: 'Gallery root does not exist' }] };
const result = sourceRows({ sources: fallback, saved: ['gone'], current: ['gone'] });
assert.deepEqual(result.rows.map(row => [row.title, row.remove]), [['gone', 'gone'], ['ComfyUI output', null]]);
assert.equal(result.note, NOTES.fallback);
assert.equal(sourceRows({ sources: { configured: [], effective: [], auto_detected: false }, saved: [], current: [] }).note, NOTES.none);
});
test('a path listed twice in the draft is one row', async () => {
const { sourceRows } = await sourcesModule;
assert.equal(sourceRows({ sources: STATUS, saved: ['output', 'input'], current: ['output', 'output'] }).rows.length, 1);
});
test('roots equal to the saved list are the saved list itself, so the draft is clean again', async () => {
const { draftRoots } = await sourcesModule;
const saved = ['output', 'input'];
assert.equal(draftRoots(['output', 'input'], saved), saved);
assert.deepEqual(draftRoots(['input'], saved), ['input']);
assert.notEqual(draftRoots(['input', 'output'], saved), saved);
assert.deepEqual(draftRoots([], undefined), []);
});
test('notice Later persists the 3.2.0 dismissal (pm_update_notice_dismissed); storage that throws keeps it for this view', async () => {
const { noticeDismissed, dismissNotice, NOTICE_KEY, NOTICE_VERSION } = await sourcesModule;
assert.deepEqual([NOTICE_KEY, NOTICE_VERSION], ['pm_update_notice_dismissed', '3.2.0']);
const values = new Map();
const storage = { getItem: key => values.get(key) ?? null, setItem: (key, value) => values.set(key, String(value)) };
assert.equal(noticeDismissed(() => storage), false);
values.set(NOTICE_KEY, '3.1.0');
assert.equal(noticeDismissed(() => storage), false);
assert.equal(dismissNotice(() => storage), true);
assert.equal(values.get(NOTICE_KEY), '3.2.0');
assert.equal(noticeDismissed(() => storage), true);
const denied = () => { throw new Error('SecurityError'); };
assert.equal(noticeDismissed(denied), false);
assert.equal(dismissNotice(denied), false);
assert.equal(dismissNotice(() => ({ setItem() { throw new Error('QuotaExceededError'); } })), false);
});
test('Refresh files toasts the rescan counts, or says it finished when they are missing', async () => {
const { refreshedText } = await sourcesModule;
assert.equal(refreshedText({ total: 1234, roots: 2 }), 'Files refreshed. 1,234 files in 2 output folders.');
assert.equal(refreshedText({ total: 1, roots: 1 }), 'Files refreshed. 1 file in 1 output folder.');
assert.equal(refreshedText({ total: -1, roots: 1 }), 'Files refreshed.');
assert.equal(refreshedText(null), 'Files refreshed.');
});
test('monitoring switched off: a source still watched says so until the restart (D128)', async () => {
const { sourceRows } = await sourcesModule;
const sources = { ...STATUS, effective: [{ ...STATUS.effective[0], watch: 'stopping' }, STATUS.effective[1]] };
const { rows } = sourceRows({ sources, saved: ['output', 'input'], current: ['output', 'input'] });
assert.deepEqual(rows.map(row => [row.description, row.scan.label]),
[['output · watching until restart', 'Scan changes'], ['input · manual scan', 'Scan folder']]);
});
test('a valid configured source missing from the effective list is another spelling of a listed folder (D128)', async () => {
const { sourceRows, DUPLICATE_SOURCE } = await sourcesModule;
assert.equal(DUPLICATE_SOURCE, 'same folder as another source');
const sources = { ...STATUS, configured: [...STATUS.configured,
{ label: 'output', path: 'output/.', valid: true, code: null, reason: null }] };
const saved = ['output', 'input', 'output/.'];
assert.deepEqual(sourceRows({ sources, saved, current: saved }).rows.at(-1), { key: 'output/.', title: 'output',
description: 'output/. · same folder as another source', scan: null, remove: 'output/.' });
});
-105
View File
@@ -192,7 +192,6 @@ test('status text per outcome', async () => {
partial: 'Some changes were not saved. Review the highlighted settings.',
tabOnly: 'Saved for this tab only; browser storage is unavailable.',
conflict: 'Settings changed elsewhere. Reload this section to see them.',
restart: 'Restart ComfyUI to apply:',
});
const clean = { browser: 'clean', host: 'clean', runtime: 'clean' };
assert.equal(statusText(clean), 'No unsaved changes.');
@@ -260,107 +259,3 @@ test('a failed read keeps the store usable and drafts survive a reload (Q10)', a
assert.equal(failing.dirty(), false);
assert.throws(() => createSystemDrafts({ ...domains, writeHost: null }), TypeError);
});
test('a rejected value drops its draft and is highlighted; a conflict elsewhere keeps the drafts', async () => {
const domains = adapters({ browser: { videoMute: true, videoLoop: true }, host: { result_timeout: 5 } });
const drafts = await loaded(domains);
assert.deepEqual(drafts.saved('browser'), { videoMute: true, videoLoop: true });
drafts.set('host', 'result_timeout', 12);
assert.equal(drafts.dirty('host'), true);
drafts.reject('host', 'result_timeout');
assert.equal(drafts.dirty('host'), false);
assert.deepEqual([...drafts.failed('host')], ['result_timeout']);
assert.equal(drafts.get('host', 'result_timeout'), 5);
// A valid value clears the highlight.
drafts.set('host', 'result_timeout', 7);
assert.deepEqual([...drafts.failed('host')], []);
drafts.discard();
drafts.set('browser', 'videoMute', false);
drafts.conflict('browser', []);
assert.equal(drafts.statusText(), 'Unsaved changes.');
drafts.conflict('browser', ['videoMute']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
assert.deepEqual([...drafts.failed('browser')], ['videoMute']);
assert.equal(drafts.get('browser', 'videoMute'), false);
assert.throws(() => drafts.reject('nowhere', 'x'), TypeError);
});
test('a rejected value says why in the status until the next edit', async () => {
const drafts = await loaded(adapters({ host: { result_timeout: 5 } }));
drafts.reject('host', 'result_timeout');
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
drafts.set('host', 'result_timeout', 9);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a browser write refused as changed elsewhere keeps the draft and reads the browser values again', async () => {
const domains = adapters({ browser: { workspaceDensity: 'comfortable' } });
const drafts = await loaded(domains);
drafts.set('browser', 'workspaceDensity', 'compact');
domains.outcome.browser = () => { throw Object.assign(new Error('changed'), { code: 'settings-changed' }); };
assert.deepEqual(await drafts.save(), { browser: 'conflict', host: 'clean', runtime: 'clean' });
assert.deepEqual(domains.calls.map(([name]) => name), ['writeBrowser', 'readBrowser']);
assert.equal(drafts.get('browser', 'workspaceDensity'), 'compact');
assert.deepEqual([...drafts.failed('browser')], ['workspaceDensity']);
assert.equal(drafts.statusText(), 'Settings changed elsewhere. Reload this section to see them.');
});
test('host read state says whether the server settings could be read', async () => {
const { createSystemDrafts } = await source;
const domains = adapters({ host: { result_timeout: 5 } });
let fail = true;
const drafts = createSystemDrafts({ ...domains, readHost: async () => { if (fail) throw new Error('offline'); return { values: {}, revision: 'r' }; } });
assert.equal(drafts.hostState(), 'pending');
assert.equal(await drafts.reloadHost(), false);
assert.equal(drafts.hostState(), 'failed');
fail = false;
assert.equal(await drafts.reloadHost(), true);
assert.equal(drafts.hostState(), 'loaded');
});
test('a host save names the settings that apply after a restart (D128)', async () => {
const { statusText, RESTART_LABELS } = await source;
assert.deepEqual(RESTART_LABELS, { monitoring_enabled: 'Watch for new outputs' });
const saved = { browser: 'clean', host: 'saved', runtime: 'clean' };
assert.equal(statusText({ ...saved, restart: ['monitoring_enabled'] }),
'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// A field without a row label (the sources say "restart pending" per row) adds nothing.
assert.equal(statusText({ ...saved, restart: ['gallery_root_paths'] }), 'Saved to the server configuration.');
assert.equal(statusText({ ...saved, restart: [] }), 'Saved to the server configuration.');
const domains = adapters({ host: { monitoring_enabled: true } });
const drafts = await loaded(domains);
domains.outcome.host = () => ({ success: true, persisted: true, revision: 'sha256:two',
restart_required_fields: ['monitoring_enabled', 7] });
drafts.set('host', 'monitoring_enabled', false);
assert.deepEqual(await drafts.save(), { browser: 'clean', host: 'saved', runtime: 'clean', restart: ['monitoring_enabled'] });
assert.equal(drafts.statusText(), 'Saved to the server configuration. Restart ComfyUI to apply: Watch for new outputs.');
// The next edit is unsaved again.
drafts.set('host', 'monitoring_enabled', true);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
test('a host save refused for one source says the server reason and names it until the next edit', async () => {
const domains = adapters({ host: { gallery_root_paths: ['output', 'output/gone'] } });
const drafts = await loaded(domains);
const message = "output/gone can't be used: Gallery root does not exist. Remove it or reconnect it, then save.";
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source', source: 'output/gone' }); };
drafts.set('host', 'gallery_root_paths', ['output/gone']);
const outcome = await drafts.save();
assert.deepEqual(outcome, { browser: 'clean', host: 'failed', runtime: 'clean',
invalid: { source: 'output/gone', message } });
assert.equal(drafts.statusText(), message);
assert.equal(drafts.invalidSource(), 'output/gone');
assert.deepEqual([...drafts.failed('host')], ['gallery_root_paths']);
// Any other failure keeps the generic copy and names no source.
domains.outcome.host = () => { throw Object.assign(new Error('busy'), { code: 'busy' }); };
await drafts.save();
assert.equal(drafts.statusText(), 'Some changes were not saved. Review the highlighted settings.');
assert.equal(drafts.invalidSource(), null);
domains.outcome.host = () => { throw Object.assign(new Error(message), { code: 'invalid-source' }); };
await drafts.save();
assert.equal(drafts.invalidSource(), null);
drafts.set('host', 'gallery_root_paths', ['output']);
assert.equal(drafts.invalidSource(), null);
assert.equal(drafts.statusText(), 'Unsaved changes.');
});
-2
View File
@@ -29,7 +29,6 @@ VECTORS_PATH = (
# Committed record sets stay small; the large bound cases are built at test time.
MAX_VECTOR_CHARS = 20_000
COMMITTED_FUZZ = (0, 60)
COMMITTED_PASSTHROUGH = (0, 40)
SPACES = ["", " ", "\t", "\n", "\x1c", "\x85", " ", "", "
 "]
WORDS = ["fox", "snow", "café", "猫", "\U0001f600", "a,", "detailed fur", ""]
@@ -1381,7 +1380,6 @@ def committed_vectors():
}
cases.update(fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.selection_fuzz_cases(*COMMITTED_FUZZ))
cases.update(selection.passthrough_fuzz_cases(*COMMITTED_PASSTHROUGH))
vectors = [
{
"name": name,
+1 -304
View File
@@ -12,10 +12,8 @@ import json
import random
try:
from tests import passthrough_graphs as shapes
from tests.local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import passthrough_graphs as shapes
from local_resolver_cases import EXECUTION_KEY, SPACES, WORDS, _Builder
@@ -439,213 +437,10 @@ def _parameters():
return cases
def _passthrough():
"""Chains joined through lineage nodes, and the floor for added conditioning."""
cases = {
"hires-fix": shapes.hires_fix(),
"hires-fix-differs": shapes.hires_fix(second_positive=shapes.OTHER),
"hires-fix-unlisted": shapes.hires_fix(filter_class="MysteryImageEffect"),
"hires-fix-generated": shapes.generated_text_chain(),
"flux-three-stage": shapes.flux_three_stage(),
"flux-concat": shapes.flux_three_stage(composite="ConditioningConcat"),
"flux-average": shapes.flux_three_stage(composite="ConditioningAverage"),
"edit-batch": shapes.edit_batch(3),
}
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
cases["flux-without-base"] = graph
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-other-negative"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
graph["10"]["inputs"]["latent_image"] = ["20", 0]
cases["hires-fix-cycle"] = graph
graph = shapes.hires_fix()
graph["17"]["inputs"]["pixels"] = ["18", 0]
graph["18"] = {"class_type": "Reroute", "inputs": {"value": ["16", 0]}}
cases["hires-fix-reroute"] = graph
graph = shapes.hires_fix()
graph["24"] = {"class_type": "SaveImage", "inputs": {"images": ["11", 0]}}
cases["hires-fix-both-saved"] = graph
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["99", 0]
cases["hires-fix-missing-latent"] = graph
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
cases["flux-negative-unsupported"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-reroute-input"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
cases["flux-encoder-parts"] = graph
cases.update(_passthrough_edges())
cases.update(_runtime_parts())
return cases
def _runtime_parts():
"""Option A: a PromptManager part fed by a run-time node is left out."""
cases = {"trigger-prepend": shapes.hires_fix_with_trigger_words()}
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="masterpiece", append_text=["8", 0])
graph["20"]["inputs"]["latent_image"] = ["4", 0]
cases["trigger-append-single"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
cases["traced-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
cases["unknown-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["indirect-prepend"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"].update(prepend_text="", text=["8", 0])
cases["runtime-main-text"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["class_type"] = "PromptManagerPair"
graph["2"]["inputs"]["negative_text"] = shapes.NEGATIVE
graph["10"]["inputs"]["negative"] = ["2", 1]
graph["20"]["inputs"]["negative"] = ["2", 1]
cases["trigger-pair"] = graph
graph = shapes.hires_fix_with_trigger_words()
graph["30"] = {"class_type": "PromptSearchList", "inputs": {"search": "x"}}
graph["2"]["inputs"]["append_text"] = ["30", 0]
cases["search-list-append"] = graph
for shape in ("text-encode", "chained", "concat"):
cases[f"through-{shape}"] = shapes.trigger_words_through(shape)
cases["branches-with-and-without"] = (
shapes.branches_with_and_without_trigger_words()
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
cases["stage-with-and-without"] = graph
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
cases["join-multi-flag"] = graph
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
cases["flux-other-full-prompt"] = graph
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
cases["flux-trigger"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-chained"] = graph
return {f"runtime-{name}": graph for name, graph in cases.items()}
def _passthrough_edges():
"""Lineage and floor edges: branching refiners, other slots, cycles, guiders."""
cases = {}
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.4)
graph["31"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-two-refiners"] = graph
graph = shapes.hires_fix()
graph["10"]["class_type"] = "SamplerCustom"
graph["11"]["inputs"]["samples"] = ["10", 1]
cases["hires-fix-denoised-slot"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["30"]["inputs"]["latent_image"] = ["32", 0]
cases["hires-fix-self-latent"] = graph
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["32", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["32", 0]}}
graph["32"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["31", 0]}}
graph["33"] = {"class_type": "SaveImage", "inputs": {"images": ["30", 0]}}
cases["hires-fix-lineage-cycle"] = graph
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
cases["flux-base-stages-differ"] = graph
graph = shapes.flux_three_stage()
del graph["60"]["inputs"]["negative"]
cases["flux-no-last-negative"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
graph["40"]["inputs"]["positive"] = ["39", 0]
graph["50"]["inputs"]["positive"] = ["39", 0]
cases["flux-unreadable-base"] = graph
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "MysteryUpscaler",
"inputs": {"image": ["51", 0], "model": ["1", 0], "positive": ["34", 0]},
}
graph["52"]["inputs"]["images"] = ["39", 0]
cases["flux-unsupported-ancestry"] = graph
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["62", 0],
"guider": ["63", 0],
"sampler": ["64", 0],
"sigmas": ["65", 0],
"latent_image": ["66", 0],
},
}
graph["62"] = {"class_type": "RandomNoise", "inputs": {"noise_seed": 1}}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"model": ["1", 0], "positive": ["35", 0], "negative": ["33", 0]},
}
graph["64"] = {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}}
graph["65"] = {"class_type": "BasicScheduler", "inputs": {"steps": 4}}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["52", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
cases["flux-custom-advanced-last"] = graph
return {f"edge-{name}": graph for name, graph in cases.items()}
def selection_cases():
"""Named record sets for every Task 9 path, each a ``read_inspection_file`` dict."""
cases = {}
for group in (_samplers(), _previews(), _detailers(), _passthrough()):
for group in (_samplers(), _previews(), _detailers()):
for name, graph in group.items():
cases[f"select-{name}"] = {"prompt": json.dumps(graph, ensure_ascii=False)}
cases.update(_parameters())
@@ -777,101 +572,3 @@ def selection_fuzz_cases(seed, count):
records.update(builder.records)
cases[f"select-fuzz-{seed}-{index}"] = records
return cases
_LINEAGE_HOPS = (
("VAEDecode", "samples"),
("VAEDecodeTiled", "samples"),
("Image Filter", "images"),
("ImageUpscaleWithModel", "image"),
("ImageScale", "image"),
("ImageScaleBy", "image"),
("VAEEncode", "pixels"),
("LatentUpscaleBy", "samples"),
("Reroute", "value"),
("MysteryImageEffect", "image"),
)
def passthrough_fuzz_cases(seed, count):
"""Seeded sampler chains joined by random lineage (and unlisted) hops.
Each later stage reads the earlier one's output through zero to four hops,
with a shared, a different, an added-to (combine, concat, average) or a
run-time positive, and a shared or different negative, under a saver or a
preview. The server is the oracle; nothing states the expected answer.
"""
rng = random.Random(seed)
cases = {}
for index in range(count):
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": shapes.prompt_manager(rng.choice([shapes.POSITIVE, "fox"])),
"3": shapes.encode(shapes.NEGATIVE),
"4": {"class_type": "EmptyLatentBatch", "inputs": {"batch_size": 2}},
"5": shapes.encode(shapes.STYLE),
"6": shapes.encode(shapes.OTHER),
}
previous, next_id = ["4", 0], 10
for stage in range(rng.randint(2, 3)):
positive = ["2", 0]
choice = rng.randrange(8)
if stage and choice == 1:
positive = ["6", 0]
elif stage and choice in (2, 3, 4):
name = (
"ConditioningCombine",
"ConditioningConcat",
"ConditioningAverage",
)[choice - 2]
names = shapes._COMPOSITE_INPUTS[name]
parts = [["2", 0], rng.choice([["5", 0], ["6", 0]])]
rng.shuffle(parts)
graph[str(next_id)] = {
"class_type": name,
"inputs": dict(zip(names, parts)),
}
positive, next_id = [str(next_id), 0], next_id + 1
elif stage and choice == 5:
graph[str(next_id)] = {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["2", 0]},
}
positive, next_id = [str(next_id), 0], next_id + 1
negative = ["6", 0] if stage and rng.random() < 0.15 else ["3", 0]
latent = previous
if stage:
image = previous
for _ in range(rng.randrange(5)):
name, key = rng.choice(_LINEAGE_HOPS)
graph[str(next_id)] = {"class_type": name, "inputs": {key: image}}
image, next_id = [str(next_id), 0], next_id + 1
latent = image
sampler = shapes.ksampler(positive, negative, latent)
if stage and rng.random() < 0.2:
sampler = {
"class_type": "UltimateSDUpscale",
"inputs": {
"image": latent,
"model": ["1", 0],
"positive": positive,
"negative": negative,
},
}
graph[str(next_id)] = sampler
previous, next_id = [str(next_id), 0], next_id + 1
if rng.random() < 0.3:
saver = rng.choice(["SaveImage", "PreviewImage"])
graph[str(next_id)] = {
"class_type": saver,
"inputs": {"images": previous},
}
next_id += 1
graph[str(next_id)] = {
"class_type": "SaveImage",
"inputs": {"images": previous},
}
cases[f"select-passthrough-fuzz-{seed}-{index}"] = {
"prompt": json.dumps(graph, ensure_ascii=False)
}
return cases
-332
View File
@@ -1,332 +0,0 @@
"""Synthetic multi-stage graphs for the pass-through lineage rule.
Each builder reproduces one graph shape with neutral texts and generic names:
a hires-fix or refiner chain (sampler, decode, image filter, model upscale,
scale, encode, sampler), a three-stage Flux run whose last stage combines the
base prompt with a style trigger, an image-edit batch of samplers with distinct
prompts, and a chain whose prompt is generated at run time. They are shared by
the resolver tests and the browser-port parity cases; no user file is involved.
"""
import copy
POSITIVE = "a lighthouse on a cliff, golden hour"
NEGATIVE = "blurry, lowres"
OTHER = "a red bicycle in the rain"
STYLE = "film grain, analog photo"
def encode(text):
return {"class_type": "CLIPTextEncode", "inputs": {"text": text, "clip": ["1", 1]}}
def prompt_manager(text, prepend=""):
return {
"class_type": "PromptManager",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
"clip": ["1", 1],
},
}
def ksampler(positive, negative, latent, denoise=1.0):
return {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"steps": 20,
"cfg": 7.0,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": denoise,
"model": ["1", 0],
"positive": positive,
"negative": negative,
"latent_image": latent,
},
}
def hires_fix(second_positive=None, filter_class="Image Filter"):
"""Two KSamplers joined in image space, like a hires-fix or refiner run.
Sampler 10 runs on a custom batch latent; its output goes through a tiled
decode, an interactive image filter, a model upscale and a scale (sized by a
resolution node reading the filtered image), then is encoded again for
sampler 20 (denoise 0.25), which is decoded and saved.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": prompt_manager(POSITIVE),
"3": encode(NEGATIVE),
"4": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1024, "height": 512, "batch_size": 4},
},
"10": ksampler(["2", 0], ["3", 0], ["4", 0]),
"11": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["10", 0], "vae": ["1", 2]},
},
"12": {
"class_type": filter_class,
"inputs": {"timeout": 600, "ontimeout": "send none", "images": ["11", 0]},
},
"13": {"class_type": "UpscaleModelLoader", "inputs": {"model_name": "x4"}},
"14": {
"class_type": "ImageUpscaleWithModel",
"inputs": {"upscale_model": ["13", 0], "image": ["12", 0]},
},
"15": {
"class_type": "ResolutionCalculator",
"inputs": {"scale_factor": 1.5, "image": ["12", 0]},
},
"16": {
"class_type": "ImageScale",
"inputs": {
"upscale_method": "nearest-exact",
"width": ["15", 0],
"height": ["15", 1],
"crop": "disabled",
"image": ["14", 0],
},
},
"17": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["16", 0], "vae": ["1", 2]},
},
"20": ksampler(["2", 0], ["3", 0], ["17", 0], denoise=0.25),
"21": {
"class_type": "VAEDecodeTiled",
"inputs": {"tile_size": 512, "samples": ["20", 0], "vae": ["1", 2]},
},
"22": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "hires", "images": ["21", 0]},
},
"23": {"class_type": "PlaySound|pysssss", "inputs": {"any": ["21", 0]}},
}
if second_positive is not None:
graph["5"] = encode(second_positive)
graph["20"]["inputs"]["positive"] = ["5", 0]
return graph
_COMPOSITE_INPUTS = {
"ConditioningCombine": ("conditioning_1", "conditioning_2"),
"ConditioningConcat": ("conditioning_to", "conditioning_from"),
"ConditioningAverage": ("conditioning_to", "conditioning_from"),
}
def flux_three_stage(combined=(("30", 0), ("31", 0)), composite="ConditioningCombine"):
"""FluxGuidance base, a refiner through an image filter, then a tiled upscale.
Samplers 40 and 50 share FluxGuidance(PromptManager 30) and a zeroed empty
negative. Sampler 50 refines 40's filtered image (denoise 0.25). The final
UltimateSDUpscale 60 reads 50's image, scaled down, and a positive that is
``composite`` of ``combined`` (by default the base PromptManager and a
style-trigger encoder 31).
"""
names = _COMPOSITE_INPUTS[composite]
graph = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux"}},
"2": {"class_type": "DualCLIPLoader", "inputs": {"type": "flux"}},
"3": {"class_type": "VAELoader", "inputs": {"vae_name": "ae"}},
"30": prompt_manager(POSITIVE),
"31": encode(STYLE),
"32": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "", "clip": ["2", 0]},
},
"33": {
"class_type": "ConditioningZeroOut",
"inputs": {"conditioning": ["32", 0]},
},
"34": {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["30", 0]},
},
"35": {
"class_type": composite,
"inputs": dict(zip(names, map(list, combined))),
},
"36": {
"class_type": "EmptyLatentBatch",
"inputs": {"width": 1080, "height": 1920, "batch_size": 4},
},
"40": ksampler(["34", 0], ["33", 0], ["36", 0]),
"41": {
"class_type": "VAEDecode",
"inputs": {"samples": ["40", 0], "vae": ["3", 0]},
},
"42": {
"class_type": "Image Filter",
"inputs": {"timeout": 600, "images": ["41", 0]},
},
"43": {"class_type": "PreviewImage", "inputs": {"images": ["42", 0]}},
"44": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["42", 0], "vae": ["3", 0]},
},
"50": ksampler(["34", 0], ["33", 0], ["44", 0], denoise=0.25),
"51": {
"class_type": "VAEDecode",
"inputs": {"samples": ["50", 0], "vae": ["3", 0]},
},
"52": {
"class_type": "ImageScaleDownBy",
"inputs": {"scale_by": 0.75, "images": ["51", 0]},
},
"53": {"class_type": "PreviewImage", "inputs": {"images": ["51", 0]}},
"54": {
"class_type": "Image Comparer (rgthree)",
"inputs": {"image_a": ["51", 0], "image_b": ["42", 0]},
},
"60": {
"class_type": "UltimateSDUpscale",
"inputs": {
"upscale_by": 2.0,
"seed": 7,
"steps": 4,
"denoise": 0.35,
"image": ["52", 0],
"model": ["1", 0],
"positive": ["35", 0],
"negative": ["33", 0],
"vae": ["3", 0],
},
},
"61": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "flux", "images": ["60", 0]},
},
}
return graph
def edit_batch(count=3):
"""Image-edit samplers with distinct prompts, all saved with one prefix."""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "LoadImage", "inputs": {"image": "ref.png"}},
"3": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["2", 0], "vae": ["1", 2]},
},
"4": {"class_type": "PrimitiveString", "inputs": {"value": "edit"}},
}
for index in range(count):
base = 100 + index * 10
graph[str(base)] = {
"class_type": "TextEncodeQwenImageEdit",
"inputs": {
"prompt": f"edit number {index}: change the pose",
"clip": ["1", 1],
},
}
graph[str(base + 1)] = ksampler([str(base), 0], [str(base), 0], ["3", 0])
graph[str(base + 2)] = {
"class_type": "VAEDecode",
"inputs": {"samples": [str(base + 1), 0], "vae": ["1", 2]},
}
graph[str(base + 3)] = {
"class_type": "SaveImage",
"inputs": {"filename_prefix": ["4", 0], "images": [str(base + 2), 0]},
}
return graph
def trigger_toggle(words="trig9word"):
"""LoraManager's TriggerWord Toggle: its output is decided at run time."""
return {
"class_type": "TriggerWord Toggle (LoraManager)",
"inputs": {
"group_mode": True,
"default_active": True,
"toggle_trigger_words": {"__value__": [{"text": words, "active": True}]},
"orinalMessage": words,
"trigger_words": ["7", 2],
},
}
def hires_fix_with_trigger_words():
"""``hires_fix`` whose PromptManager prepends LoRA trigger words (node 8)."""
graph = hires_fix()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["1", 1], "text": "<lora:x:1>"},
}
graph["8"] = trigger_toggle()
graph["2"]["inputs"]["prepend_text"] = ["8", 0]
return graph
def prompt_manager_text(text, prepend=""):
"""PromptManagerText: a STRING PromptManager (prepend/main/append)."""
return {
"class_type": "PromptManagerText",
"inputs": {
"text": text,
"category": "",
"tags": "",
"search_text": "",
"prepend_text": prepend,
"append_text": "",
},
}
def trigger_words_through(shape):
"""``hires_fix_with_trigger_words`` with nodes between the toggle and sampler.
``text-encode``: PromptManagerText (prepend <- toggle) -> CLIPTextEncode.
``chained``: a PromptManager whose ``text`` reads that PromptManagerText.
``concat``: PromptManagerText -> StringConcatenate(..., "extra") -> encoder.
"""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager_text(POSITIVE, ["8", 0])
if shape == "chained":
graph["2"]["inputs"].update(prepend_text="", text=["9", 0])
elif shape == "text-encode":
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["9", 0]}}
else:
graph["19"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": ["9", 0], "string_b": "extra", "delimiter": ", "},
}
graph["2"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["19", 0]}}
return graph
def branches_with_and_without_trigger_words():
"""Two saved branches: one PromptManager prepends trigger words, one does not."""
graph = hires_fix_with_trigger_words()
graph["9"] = prompt_manager(POSITIVE)
graph["20"]["inputs"].update(positive=["9", 0], latent_image=["4", 0])
return graph
def generated_text_chain():
"""``hires_fix`` whose positive text comes from a run-time text generator."""
graph = hires_fix()
graph["2"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["6", 0], "clip": ["1", 1]},
}
graph["6"] = {
"class_type": "TextGenerate",
"inputs": {"prompt": "write a prompt", "model": ["1", 0]},
}
return graph
def copied(graph):
return copy.deepcopy(graph)
+2 -11
View File
@@ -30,11 +30,6 @@ FILES = "media_annotation_files"
BYTES = "media_annotation_bytes"
IMAGE_TAGS = "media_image_tags"
TABLES = (METADATA, FILES, BYTES, IMAGE_TAGS)
REVIEW_TABLES = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
@@ -207,14 +202,10 @@ class AnnotationSchemaTests(unittest.TestCase):
)
def drop_group(self, path):
"""Remove the group from a closed database file (an old backup).
The review queue group cites annotation bytes and predates nothing, so
an old backup lacks it as well.
"""
"""Remove the group from a closed database file (an old backup)."""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in REVIEW_TABLES + tuple(reversed(TABLES)):
for table in reversed(TABLES):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
+3 -45
View File
@@ -52,7 +52,6 @@ class AdminAPITestCase(AioHTTPTestCase):
self.outside_dir = Path(self.tmpdir) / "outside"
self.outside_dir.mkdir()
self.addAsyncCleanup(self._drain_and_remove)
self.config_path = os.path.join(self.tmpdir, "config.json")
self._orig_env = {
key: os.environ.pop(key, None) for key in (CONFIG_PATH_ENV, EXTRA_ROOTS_ENV)
@@ -90,26 +89,6 @@ class AdminAPITestCase(AioHTTPTestCase):
sys.modules.pop("folder_paths", None)
else:
sys.modules["folder_paths"] = self._orig_folder_paths
async def _drain_and_remove(self):
"""The last cleanup: no job outlives the temporary database.
Registered first, so it runs after every cleanup a test added (which
may release its own gated jobs). A job still running then, such as a
scan a failed assertion never awaited, is cancelled and awaited
before the database and the tree are removed.
"""
api = getattr(self, "api", None)
jobs = api._jobs.list() if api is not None else []
tasks = [
job.task for job in jobs if job.task is not None and not job.task.done()
]
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
if api is not None:
# A scan of an ambiguous file captures it for Needs review.
api.close_annotation_captures()
PromptModel.close_all_instances() # restore may have swapped api.db
assert_closed(self, self.tmpdir)
shutil.rmtree(self.tmpdir, ignore_errors=True)
@@ -889,23 +868,6 @@ class TestDeleteDuplicatesEndpoint(AdminAPITestCase):
self.assertFalse(target.exists())
self.assertTrue(thumb_dir.exists())
async def test_deleted_files_leave_the_cached_gallery_listing(self):
# The listing is cached for 30 s; a delete must not leave the gallery
# listing (and counting) a file that is gone.
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
for name in ("keep.png", "gone.png"):
(self.output_dir / name).write_bytes(b"same bytes")
async def listed():
resp = await self.client.get("/prompt_manager/images/output")
data = await resp.json()
return data["total"], sorted(i["filename"] for i in data["images"])
self.assertEqual(await listed(), (2, ["gone.png", "keep.png"]))
resp = await self._post({"image_paths": ["gone.png"]})
self.assertEqual((await resp.json())["deleted_count"], 1)
self.assertEqual(await listed(), (1, ["keep.png"]))
def test_unexpected_error_per_file_is_reported_without_path(self):
self.api._delete_one_duplicate = lambda *a: (_ for _ in ()).throw(
PermissionError(13, "Permission denied", os.path.join(self.tmpdir, "x.png"))
@@ -2242,9 +2204,7 @@ class TestActivityJobs(AdminAPITestCase):
for name in names:
(self.output_dir / name).write_bytes(b"not an image")
def fake_ingest(
media_file, metadata, detailed=False, ambiguous=None, untraced=None
):
def fake_ingest(media_file, metadata, detailed=False):
key = Path(media_file).relative_to(self.output_dir).as_posix()
outcome = names[key]
if isinstance(outcome, Exception):
@@ -2373,10 +2333,8 @@ class TestActivityJobs(AdminAPITestCase):
first_done, release = threading.Event(), threading.Event()
original = self.api._ingest_scan_batch_sync
def gated(batch_results, roots=(), outcomes=None, review=None):
counts = original(
batch_results, roots=roots, outcomes=outcomes, review=review
)
def gated(batch_results, roots=(), outcomes=None):
counts = original(batch_results, roots=roots, outcomes=outcomes)
first_done.set()
release.wait(5)
return counts
+1 -9
View File
@@ -1045,15 +1045,7 @@ class TestFailures(AnnotationAPITestCase):
await self.api.cleanup_annotation_captures(None)
self.assertFalse(os.path.exists(directory))
self.assertNotIn(identity, observers._restore_observers)
# The shutdown cleanup is final: a later observation (or a monitor
# recording still in flight) creates no new store or directory.
status, body = await self.observe({"target": self.target("sub/a.png")})
self.assertEqual((status, body["code"]), (500, "capture-error"), body)
self.assertIsNone(self.api._captures)
self.assertNotIn(identity, observers._restore_observers)
# An explicit, non-final close still lets a later observation start
# a fresh store (the tests' own store swaps rely on it).
self.api._captures_closed = False
# A later observation starts a fresh store.
await self.observe_ok()
self.assertNotEqual(self.api._captures.directory, directory)
-13
View File
@@ -471,19 +471,6 @@ class TestUiRoutes(CoreRoutesTestCase):
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
async def test_review_routes_are_registered(self):
table = {
(route.method, route.resource.canonical)
for route in self.app.router.routes()
if route.resource is not None
}
for entry in (
("GET", "/prompt_manager/review/items"),
("GET", "/prompt_manager/review/items/{item_id}"),
("POST", "/prompt_manager/review/items/{item_id}/decision"),
):
self.assertIn(entry, table)
class TestStaticRoutes(CoreRoutesTestCase):
File diff suppressed because it is too large Load Diff
-57
View File
@@ -68,55 +68,6 @@ class CardSeed(unittest.TestCase):
self.assertIn(expected, (111, 222))
self.assertEqual(info_seed(info), str(expected))
def test_a_chain_the_inspector_names_one_stage_of_shows_no_differing_seed(self):
# The inspector shows the named final stage's seed (D19, pass-through
# chains); the parser shows another stage's. Never two seeds: none.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {
"class_type": "FaceDetailer",
"inputs": {
"seed": 333,
"image": ["20", 0],
"model": ["4", 0],
"positive": ["6", 0],
"negative": ["7", 0],
},
}
info = graph(seeded(111), extra=extra)
self.assertEqual(extract_from_records(info["prompt"], None)["seed"], 111)
self.assertIsNone(info_seed(info))
# The same chain on one seed keeps it; so does a refiner over a decode.
extra["21"]["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), extra=extra)), "111")
refiner = seeded(222, latent_image=["22", 0])
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
del extra["21"]
self.assertIsNone(info_seed(graph(seeded(111), refiner, extra=extra)))
refiner["inputs"]["seed"] = 111
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_a_seed_that_agrees_with_the_named_stage_is_kept(self):
# Review I3: a saved sampler (seed 7) beside a preview-only sampler
# (seed 5): the parser and the inspector both show 7, as on main.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["21"] = {"class_type": "SaveImage", "inputs": {"images": ["20", 0]}}
extra["30"] = {"class_type": "VAEDecode", "inputs": {"samples": ["11", 0]}}
extra["31"] = {"class_type": "PreviewImage", "inputs": {"images": ["30", 0]}}
info = graph(seeded(7), seeded(5), extra=extra)
self.assertEqual(info_seed(info), "7")
def test_a_named_ambiguous_chain_keeps_the_parser_seed(self):
# Review M2: multistage-differs names a sampler, but its facts merge,
# and the inspector fills the parser's seed: the card keeps it.
extra = {"2": {"class_type": "EmptyLatentImage", "inputs": {}}}
extra["8"] = encoder("another scene")
extra["20"] = {"class_type": "VAEDecode", "inputs": {"samples": ["10", 0]}}
extra["22"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["20", 0]}}
refiner = ksampler(("8", 0), ("7", 0), seed=222, latent_image=["22", 0])
self.assertEqual(info_seed(graph(seeded(111), refiner, extra=extra)), "111")
def test_card_seed_linked_input_follows_shared_parser(self):
primitive = {"class_type": "PrimitiveNode", "inputs": {"value": 555}}
info = graph(seeded(["20", 0]), extra={"20": primitive})
@@ -256,7 +207,6 @@ class CardSeed(unittest.TestCase):
self.assertIsNone(parsed_positive(unparsable))
def test_every_shared_case_reads_the_projected_graphs(self):
withheld = []
for name, case in CASES.items():
with self.subTest(case=name):
info = {k: v for k, v in case.items() if v is not None}
@@ -275,18 +225,11 @@ class CardSeed(unittest.TestCase):
except GenerationParamsError:
self.assertIsNone(params)
continue
if name == "shared-prompt-pipeline":
# The inspector names the detailer (seed 333); the parser
# reads the base sampler (111): the card shows none.
self.assertEqual(expected["seed"], 111)
expected = {**expected, "seed": None}
withheld.append(name)
self.assertEqual(params, expected)
# Prompt text is the same as from a plain parse of the records.
if len(values) == len(info):
raw = extract_from_records(case["prompt"], case["workflow"])
self.assertEqual(params["positivePrompt"], raw["positivePrompt"])
self.assertEqual(withheld, ["shared-prompt-pipeline"])
if __name__ == "__main__":
+5 -84
View File
@@ -24,9 +24,6 @@ from database.operations import PromptDatabase
from tests.open_handles import assert_closed
from utils.hashing import generate_prompt_hash
# A bound on thread hand-offs that only a hung test reaches (never a pacing timer).
THREAD_WAIT_SECONDS = 120
def _count_prompts(path):
"""Row count read through a brand-new connection (closed before return)."""
@@ -353,13 +350,7 @@ class TestRestoreFromFile(BackupTestCase):
def worker():
self.db.search_prompts(text="old") # opens this thread's connection
ready.set()
# Wait for the restore itself, not for a timer: a stalled Windows
# runner once took longer than 5 s to restore, and the worker then
# read the old database before the restore had run.
if not proceed.wait(THREAD_WAIT_SECONDS):
results["error"] = "the restore did not finish"
self.db.close()
return
proceed.wait(5)
try:
results["texts"] = [p["text"] for p in self.db.search_prompts()]
except Exception as exc: # noqa: BLE001 - reported through results
@@ -369,15 +360,12 @@ class TestRestoreFromFile(BackupTestCase):
thread = threading.Thread(target=worker)
thread.start()
self.assertTrue(ready.wait(THREAD_WAIT_SECONDS))
ready.wait(5)
try:
self.db.model.restore_from_file(src)
finally:
proceed.set()
thread.join(THREAD_WAIT_SECONDS)
self.db.model.restore_from_file(src)
proceed.set()
thread.join(5)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
self.assertEqual(results["texts"], ["restored"])
self.assertEqual([p["text"] for p in self.db.search_prompts()], ["restored"])
@@ -466,9 +454,6 @@ class TestRestoreFromFile(BackupTestCase):
def test_restore_old_backup_without_annotations(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
@@ -505,70 +490,6 @@ class TestRestoreFromFile(BackupTestCase):
0,
)
def test_restore_annotations_era_backup_without_review_queue(self):
tables = (
"media_review_decisions",
"media_review_items",
"media_review_metadata",
)
src = self._make_source_db(["predates the review queue"])
with closing(sqlite3.connect(src)) as conn:
for table in tables:
conn.execute(f"DROP TABLE {table}")
conn.commit()
self.assertTrue(PromptModel.verify_database_file(src)[0])
ok, reason = PromptModel.verify_database_file(src, require_library_state=True)
self.assertFalse(ok)
self.assertEqual(reason, "Missing review queue schema")
self.db.model.restore_from_file(src)
self.assertEqual(_texts(self.path), ["predates the review queue"])
with closing(sqlite3.connect(self.path)) as conn:
self.assertEqual(
conn.execute("SELECT * FROM media_review_metadata").fetchall(),
[(1, 1, 0)],
)
self.assertEqual(
conn.execute("SELECT COUNT(*) FROM media_review_items").fetchone()[0],
0,
)
ok, reason = PromptModel.verify_database_file(
self.path, require_library_state=True
)
self.assertTrue(ok, reason)
with closing(sqlite3.connect(src)) as conn: # the source is untouched
self.assertEqual(
conn.execute(
"SELECT COUNT(*) FROM sqlite_master WHERE name LIKE 'media_review%'"
).fetchone()[0],
0,
)
def test_restore_rejects_review_queue_without_annotations(self):
src = self._make_source_db(["review without annotations"])
with closing(sqlite3.connect(src)) as conn:
conn.execute("PRAGMA foreign_keys=OFF")
for table in (
"media_image_tags",
"media_annotation_bytes",
"media_annotation_files",
"media_annotation_metadata",
):
conn.execute(f"DROP TABLE {table}")
conn.commit()
before = Path(src).read_bytes()
live_before = _texts(self.path)
ok, reason = PromptModel.verify_database_file(src)
self.assertFalse(ok)
self.assertIn("image annotation", reason)
with self.assertRaises(ValueError):
self.db.model.restore_from_file(src)
self.assertEqual(Path(src).read_bytes(), before)
self.assertEqual(_texts(self.path), live_before)
def test_restore_preserves_favorites_and_closes_handles(self):
self.db.set_image_favorite("output", "live.png")
with tempfile.TemporaryDirectory() as directory:
+3 -18
View File
@@ -23,11 +23,7 @@ sys.modules.setdefault("server", _mock_server)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from py.config import ( # noqa: E402
OUTSIDE_ALLOWED_REASON,
GalleryConfig,
PromptManagerConfig,
)
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
@@ -162,10 +158,9 @@ class GalleryRootTestCase(unittest.TestCase):
class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_filesystem_root_rejected(self):
"""Refused as outside the allowed folders: no reason names the layout."""
ok, reason = GalleryConfig.validate_gallery_root(os.path.abspath(os.sep))
self.assertFalse(ok)
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
self.assertIn("root", reason.lower())
@unittest.skipUnless(os.name == "nt", "drive roots only exist on Windows")
def test_windows_drive_root_rejected(self):
@@ -175,17 +170,7 @@ class TestValidateGalleryRootRejects(GalleryRootTestCase):
def test_home_directory_rejected(self):
ok, reason = GalleryConfig.validate_gallery_root(os.path.expanduser("~"))
self.assertFalse(ok)
self.assertEqual(reason, OUTSIDE_ALLOWED_REASON)
def test_home_directory_inside_an_allowed_parent_is_still_refused(self):
"""Even when the environment opens the home's parent, home answers
like any outside path."""
home = os.path.expanduser("~")
os.environ[EXTRA_ROOTS_ENV] = os.path.dirname(home)
self.assertEqual(
GalleryConfig.gallery_root_problem(home),
("outside-allowed", OUTSIDE_ALLOWED_REASON),
)
self.assertIn("home", reason.lower())
def test_directory_outside_comfyui_rejected(self):
outside = Path(self.tmpdir) / "elsewhere"
-45
View File
@@ -5,10 +5,7 @@ returns for the same input (tests/test_generation_params_js_parity.py checks
that directly when Node is available).
"""
import json
import os
import shutil
import subprocess # nosec B404 - runs a fixed Node one-liner
import sys
import unittest
@@ -20,7 +17,6 @@ from utils.generation_params import ( # noqa: E402
extract_generation_params,
extract_from_records,
parse_js_json,
parse_page_json,
)
try:
@@ -223,47 +219,6 @@ class ParsesLikeJavaScript(unittest.TestCase):
run("seed-beyond-double-precision")["seed"], 4974052037430000001.0
)
def test_page_parse_reads_nan_numbers_as_null(self):
# gallery.js and admin.js replace NaN with null before JSON.parse.
self.assertEqual(
parse_page_json('{"a": NaN, "b": [NaN], "t": "NaN"}'),
{"a": None, "b": [None], "t": "NaN"},
)
self.assertEqual(
parse_page_json('{"seed": 9007199254740993}'), {"seed": 9007199254740992.0}
)
for text in ('{"a": Infinity}', "[-Infinity]", "not json", "", None):
with self.subTest(text=text):
self.assertIsNone(parse_page_json(text))
@unittest.skipUnless(shutil.which("node"), "Node is not installed")
def test_page_parse_equals_the_legacy_pages_for_nan_numbers(self):
# gallery.js/admin.js: two NaN regexes, then JSON.parse (Infinity throws).
texts = [
'{"1": {"is_changed": [NaN]}, "2": {"is_changed": NaN, "v": 1.5}}',
'{"a":NaN,"b":[NaN, 2]}',
'{"a": Infinity}',
'{"seed": 9007199254740993}',
]
script = (
"const texts = JSON.parse(require('fs').readFileSync(0, 'utf8'));"
"console.log(JSON.stringify(texts.map((t) => { try { return {v: JSON.parse("
"t.replace(/:\\s*NaN\\b/g, ': null').replace(/\\bNaN\\b/g, 'null'))}; }"
" catch (e) { return null; } })));"
)
completed = subprocess.run( # nosec B603 B607 - fixed argv, no shell
["node", "-e", script],
input=json.dumps(texts),
capture_output=True,
text=True,
check=True,
timeout=60,
)
expected = json.loads(completed.stdout)
for text, page in zip(texts, expected):
with self.subTest(text=text):
self.assertEqual(parse_page_json(text), page and page["v"])
def test_duplicate_keys_keep_the_last_value(self):
self.assertEqual(parse_js_json('{"a": 1, "a": 2}'), {"a": 2})
+3 -8
View File
@@ -800,15 +800,10 @@ class TestUpstreamVerifiedNodeSemantics(unittest.TestCase):
prompt["31"] = dict(prompt["3"], inputs=dict(prompt["3"]["inputs"]))
prompt["31"]["inputs"]["latent_image"] = ["3", 0]
prompt["8"]["inputs"]["samples"] = ["31", 0]
# 31 continues 3's latent: the stages are compared, as for KSampler.
for anchor in (None, "9"):
result = _inspect({"prompt": prompt}, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "31")
)
prompt["31"]["inputs"]["positive"] = ["7", 0]
# Unanchored, two listed samplers are two branches, as for KSampler.
self.assertEqual(_inspect({"prompt": prompt})["state"], "ambiguous")
result = _inspect({"prompt": prompt}, output_node_id="9")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_seed_variance_enhancer_passes_its_conditioning(self):
prompt = _f02(
+3 -105
View File
@@ -55,9 +55,6 @@ def _minimal():
"runtime": [],
"savers": {},
"non_file_outputs": [],
"lineage": [
{"class": "VAEEncode", "slot": 0, "strategy": "continue", "input": "pixels"}
],
"stage_inputs": {"shaped": ["positive"], "with_model": ["samples"]},
"facts": {
"configuration_outputs": {},
@@ -202,26 +199,16 @@ class SpecEntriesTest(unittest.TestCase):
},
)
# Combine and concat keep every part in effect (additive); an average
# blends its parts, so a part may be weighted away.
self.assertEqual(
table.composites["ConditioningCombine"],
{
"strategy": "combine",
"inputs": ("conditioning_1", "conditioning_2"),
"additive": True,
},
{"strategy": "combine", "inputs": ("conditioning_1", "conditioning_2")},
)
for name, additive in (
("ConditioningConcat", {"additive": True}),
("ConditioningAverage", {}),
):
for name in ("ConditioningConcat", "ConditioningAverage"):
self.assertEqual(
table.composites[name],
{
"strategy": "combine",
"inputs": ("conditioning_to", "conditioning_from"),
**additive,
},
)
@@ -270,12 +257,7 @@ class SpecEntriesTest(unittest.TestCase):
)
# D18: decided at run time.
# LoraManager's TriggerWord Toggle outputs the LoRA's trigger words as
# filtered at run time (option A leaves them out of a PromptManager).
self.assertEqual(
table.runtime,
frozenset({"PromptSearchList", "TriggerWord Toggle (LoraManager)"}),
)
self.assertEqual(table.runtime, frozenset({"PromptSearchList"}))
comfy = "{name}_{counter:05}_.{ext}"
for name, image in (
@@ -482,89 +464,6 @@ class SpecEntriesTest(unittest.TestCase):
self.assertIsInstance(table.composites["ConditioningCombine"]["inputs"], tuple)
class LineageTableTest(_TableCase):
"""``lineage``: outputs whose image or latent continues one input's."""
def test_repository_lineage_lists_the_pass_through_nodes(self):
table = load_semantics()
self.assertEqual(
{key: dict(entry) for key, entry in table.lineage.items()},
{
("VAEDecode", 0): {"strategy": "continue", "input": "samples"},
("VAEDecodeTiled", 0): {"strategy": "continue", "input": "samples"},
("VAEEncode", 0): {"strategy": "continue", "input": "pixels"},
("VAEEncodeTiled", 0): {"strategy": "continue", "input": "pixels"},
("ImageUpscaleWithModel", 0): {
"strategy": "continue",
"input": "image",
},
("ImageScale", 0): {"strategy": "continue", "input": "image"},
("ImageScaleBy", 0): {"strategy": "continue", "input": "image"},
("ImageScaleToTotalPixels", 0): {
"strategy": "continue",
"input": "image",
},
("LatentUpscale", 0): {"strategy": "continue", "input": "samples"},
("LatentUpscaleBy", 0): {"strategy": "continue", "input": "samples"},
("Image Filter", 0): {"strategy": "continue", "input": "images"},
("Image Filter", 1): {"strategy": "continue", "input": "latents"},
},
)
# Lineage describes no prompt: its classes are not prompt categories.
self.assertFalse({name for name, _ in table.lineage} & table.all_classes)
def test_lineage_rows_are_validated(self):
def row(**changes):
entry = {
"class": "VAEEncode",
"slot": 0,
"strategy": "continue",
"input": "pixels",
}
entry.update(changes)
return entry
cases = (
({"lineage": {}}, "must be a list"),
({"lineage": ["VAEEncode"]}, "must be an object"),
({"lineage": [row(strategy="passthrough")]}, "unknown strategy"),
({"lineage": [row(slot=-1)]}, "slot"),
({"lineage": [row(slot=True)]}, "slot"),
({"lineage": [row(input="")]}, "input name"),
({"lineage": [row(note="x")]}, "field"),
({"lineage": [row(), row()]}, "duplicate"),
({"lineage": [row(**{"class": ""})]}, "class"),
)
for index, (change, fragment) in enumerate(cases):
with self.subTest(case=index):
doc = _minimal()
doc.update(change)
self.assertRejected(doc, fragment)
doc = _minimal()
doc["lineage"].append(row(slot=1, input="latents"))
self.assertEqual(len(self.load_doc(doc).lineage), 2)
def test_a_lineage_class_may_also_carry_conditioning(self):
# InpaintModelConditioning-like: conditioning slots, a latent slot.
doc = _minimal()
doc["lineage"].append(
{"class": "FluxGuidance", "slot": 1, "strategy": "continue", "input": "x"}
)
self.assertIn(("FluxGuidance", 1), self.load_doc(doc).lineage)
def test_composite_additive_must_be_a_boolean(self):
doc = _minimal()
doc["composites"]["ConditioningCombine"] = {
"strategy": "combine",
"inputs": ["conditioning_1", "conditioning_2"],
"additive": "yes",
}
self.assertRejected(doc, "additive must be a boolean")
doc["composites"]["ConditioningCombine"]["additive"] = True
table = self.load_doc(doc)
self.assertIs(table.composites["ConditioningCombine"]["additive"], True)
class ValidationTest(_TableCase):
def test_minimal_table_loads(self):
table = self.load_doc(_minimal())
@@ -602,7 +501,6 @@ class ValidationTest(_TableCase):
{"literal", "strip-newlines", "join-multi", "concat"}
),
"savers": frozenset({"comfy-counter", "vhs-counter"}),
"lineage": frozenset({"continue"}),
},
)
with self.assertRaises(TypeError):
-1
View File
@@ -1744,7 +1744,6 @@ class LibraryCounts(unittest.TestCase):
"negative_variants": 0,
"images": 0,
"rated_images": 0,
"review_pending": 0,
},
)
+1 -28
View File
@@ -52,7 +52,7 @@ from utils.prompt_inspector import (
)
from utils.prompt_inspector import inspect_prompt_evidence, unreadable_evidence
from utils.prompt_pair_resolver import ADDED_CONDITIONING_REASON, INSPECTION_PROFILE
from utils.prompt_pair_resolver import INSPECTION_PROFILE
try:
from tests import inspection_parity as parity
@@ -1293,33 +1293,6 @@ class LocalResolverParityTests(unittest.TestCase):
self.assertEqual(
answers["select-preview-dropped"]["resolution"]["sampler_id"], "3"
)
# Pass-through lineage: the chain shapes reach every new outcome.
self.assertEqual(
{
name: answers[f"select-{name}"]["resolution"]["reason"]
for name in ("hires-fix", "flux-three-stage", "hires-fix-differs")
},
{
"hires-fix": "",
"flux-three-stage": ADDED_CONDITIONING_REASON,
"hires-fix-differs": "Sampler stages have different prompts",
},
)
def test_local_roles_match_fuzzed_pass_through_chains(self):
cases = {}
for seed in range(4):
cases.update(selection_cases.passthrough_fuzz_cases(seed, 120))
answers = self._compare(cases)
reasons = [answer["resolution"]["reason"] for answer in answers.values()]
for reason in (
"",
"Sampler stages have different prompts",
"A later stage adds conditioning to the base prompt",
"Combined conditioning has different texts",
"Multiple sampler branches",
):
self.assertGreater(reasons.count(reason), 5, reason)
def test_local_roles_match_fuzzed_selection_sets(self):
cases = {}
-88
View File
@@ -471,94 +471,6 @@ class CaptureStoreTestCase(unittest.TestCase):
self.assertEqual(results["capture"].size, 60)
self.capture(workers, "b.png")
def _hold_worker(self, store, relative, **options):
"""Start a capture that holds a worker until ``proceed`` is set."""
entered, proceed = threading.Event(), threading.Event()
original = CaptureStore._read
results = {}
def blocking(store_, stream, size):
if threading.current_thread() is thread and not entered.is_set():
entered.set()
self.assertTrue(proceed.wait(WAIT))
return original(store_, stream, size)
def worker():
try:
results["capture"] = self.capture(store, relative, **options)
except Exception as error: # pragma: no cover - reported below
results["error"] = error
patcher = patch.object(CaptureStore, "_read", blocking)
patcher.start()
self.addCleanup(patcher.stop)
thread = threading.Thread(target=worker)
thread.start()
self.assertTrue(entered.wait(WAIT))
def finish():
proceed.set()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertNotIn("error", results)
return results["capture"]
self.addCleanup(proceed.set)
return finish
def test_background_capture_never_takes_the_last_worker(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
# The second background capture waits, then fails: one worker is free
# but it is the interactive request's.
started = time.monotonic()
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.05):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertGreaterEqual(time.monotonic() - started, 0.04)
# An interactive capture (Tag dialog, Apply) still gets the last one.
interactive = self.capture(store, "c.png")
self.assertEqual(interactive.size, 60)
finish()
def test_background_capture_waits_for_a_worker_to_free(self):
for name in ("a", "b"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_workers=2)
finish = self._hold_worker(store, "a.png", background=True)
waited = {}
def background():
waited["capture"] = self.capture(store, "b.png", background=True)
thread = threading.Thread(target=background)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", WAIT):
thread.start()
finish()
thread.join(WAIT)
self.assertFalse(thread.is_alive())
self.assertEqual(waited["capture"].size, 60)
def test_single_worker_store_admits_no_background_capture(self):
self.write("a.png", bytes(60))
store = self.make_store(max_workers=1)
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "a.png", background=True)
self.assertEqual(self.capture(store, "a.png").size, 60)
def test_background_capture_leaves_the_last_capture_slot(self):
for name in ("a", "b", "c"):
self.write(f"{name}.png", bytes(60))
store = self.make_store(max_captures=2)
self.capture(store, "a.png")
with patch("utils.media_capture.BACKGROUND_WAIT_SECONDS", 0.01):
with self.assertRaises(CaptureCapacity):
self.capture(store, "b.png", background=True)
self.assertEqual(self.capture(store, "c.png").size, 60)
# -- lifetime ------------------------------------------------------------
def test_idle_and_hard_expiry_and_in_use_retention(self):
-360
View File
@@ -1,360 +0,0 @@
"""ComfyUI's ``NaN``/``Infinity`` metadata ingests, and is stored as strict JSON.
ComfyUI serializes the prompt with Python's ``json.dumps``, so a node whose
``IS_CHANGED`` returns ``float("nan")`` (the "always re-run" idiom) writes
``"is_changed": [NaN]`` into every saved PNG; widgets can hold ``Infinity``.
The legacy pages replaced ``NaN`` with ``null`` before ``JSON.parse``, so such
files must ingest. Whatever is stored or answered as JSON stays strict (the
non-finite numbers become ``null``) and the prompt text and hash never change.
"""
import json
import logging
import os
import sqlite3
import sys
import tempfile
import time
import types
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
from PIL import Image, PngImagePlugin
from database.operations import PromptDatabase
from tests.history_waits import drain_writer
from tests.test_api_admin import AdminAPITestCase
from tests.test_api_images import ImageAPITestCase
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
# The words "NaN" and "Infinity" inside the prompt are text, never numbers.
POSITIVE = " A cat named NaN beside an Infinity pool, ratio: NaN, [NaN] "
NEGATIVE = "blurry"
NAN, INF = float("nan"), float("inf")
def graph(positive=POSITIVE, negative=NEGATIVE):
"""An API prompt graph with non-finite numbers where ComfyUI writes them."""
g = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": positive}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": negative}},
"3": {
"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["2", 0], "model": ["4", 0]},
},
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"5": {
"class_type": "Image Filter",
"inputs": {"strength": INF, "floor": -INF},
"_meta": {"title": "Image Filter"},
"is_changed": [NAN],
},
}
return g
WORKFLOW = {
"nodes": [
{"id": 5, "type": "Image Filter", "widgets_values": [NAN, INF, -INF]},
],
"links": [],
}
def comfy_text(value):
"""The record text ComfyUI writes: Python's default, non-strict JSON."""
text = json.dumps(value)
assert "NaN" in text and "Infinity" in text
return text
def write_png(path, prompt_graph=None, workflow=WORKFLOW):
info = PngImagePlugin.PngInfo()
info.add_text("prompt", comfy_text(prompt_graph or graph()))
if workflow is not None:
info.add_text("workflow", comfy_text(workflow))
path.parent.mkdir(parents=True, exist_ok=True)
with Image.new("RGB", (4, 4)) as image:
image.save(path, pnginfo=info)
return path
def _reject(name):
raise ValueError(f"non-strict JSON constant {name}")
def strict_loads(text):
return json.loads(text, parse_constant=_reject)
class StrictRows:
"""Assertions over the rows a write left behind."""
def image_rows(self, db):
with db.model.get_connection() as conn:
return [
dict(row)
for row in conn.execute(
"SELECT prompt_id, workflow_data, prompt_metadata, parameters "
"FROM generated_images ORDER BY id"
)
]
def assert_strict_snapshot(self, row, *, workflow=True):
prompt = strict_loads(row["prompt_metadata"])
self.assertEqual(prompt["5"]["is_changed"], [None])
self.assertEqual(prompt["5"]["inputs"], {"strength": None, "floor": None})
self.assertEqual(prompt["1"]["inputs"]["text"], POSITIVE)
stored_workflow = strict_loads(row["workflow_data"])
if workflow:
self.assertEqual(
stored_workflow["nodes"][0]["widgets_values"], [None, None, None]
)
if row["parameters"] is not None:
strict_loads(row["parameters"])
def assert_prompt_text(self, db, prompt_id):
with db.model.get_connection() as conn:
row = conn.execute(
"SELECT text, hash FROM prompts WHERE id=?", (prompt_id,)
).fetchone()
self.assertEqual(row["text"], POSITIVE.strip())
self.assertEqual(row["hash"], generate_prompt_hash(POSITIVE))
def dumps_strict(value, **kwargs):
from utils.json_strict import dumps_strict as dumps
return dumps(value, **kwargs)
class TestDumpsStrict(unittest.TestCase):
def test_finite_values_serialize_byte_identically(self):
value = {"b": [1, 2.5, "NaN", None, True], "a": {"x": "Infinity"}}
for kwargs in ({}, {"sort_keys": True, "separators": (",", ":")}):
with self.subTest(kwargs=kwargs):
self.assertEqual(
dumps_strict(value, **kwargs),
json.dumps(value, allow_nan=False, **kwargs),
)
def test_non_finite_numbers_become_null_and_strings_stay(self):
value = {"n": NAN, "i": [INF, -INF], "s": "NaN Infinity", "t": (NAN, 1)}
encoded = dumps_strict(value, sort_keys=True)
self.assertEqual(
strict_loads(encoded),
{"n": None, "i": [None, None], "s": "NaN Infinity", "t": [None, 1]},
)
def test_other_errors_still_raise(self):
cycle = []
cycle.append(cycle)
for value in (cycle, [NAN, cycle]):
with self.assertRaisesRegex(ValueError, "Circular"):
dumps_strict(value)
with self.assertRaises(TypeError):
dumps_strict({"x": object()})
self.assertEqual(
dumps_strict({"x": object()}, default=lambda _: NAN), '{"x": null}'
)
class TestScanIngestsNonFiniteMetadata(StrictRows, AdminAPITestCase):
def ingest(self, name, prompt_graph=None):
path = write_png(self.output_dir / name, prompt_graph)
extracted = self.api._extract_comfyui_metadata(str(path))
return self.api._ingest_scan_batch_sync([(path, extracted)])
def test_resolved_pair_with_nan_is_added_with_strict_snapshots(self):
counts = self.ingest("nan.png")
self.assertEqual(counts["failed"], 0)
self.assertEqual(counts["added"], 1)
rows = self.image_rows(self.api.db)
self.assertEqual(len(rows), 1)
self.assert_strict_snapshot(rows[0])
self.assert_prompt_text(self.api.db, rows[0]["prompt_id"])
pairs = self.api.db.list_prompt_pairs()["pairs"]
self.assertEqual(
[(p["positive_text"], p["negative_text"]) for p in pairs],
[(POSITIVE.strip(), NEGATIVE)],
)
# A repeated scan compares the same strict snapshot: unchanged.
self.assertEqual(self.ingest("nan.png")["unchanged"], 1)
async def test_scan_route_reports_no_failures(self):
from py.config import GalleryConfig
write_png(self.output_dir / "nan.png")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
resp = await self.client.request("POST", "/prompt_manager/scan")
body = await resp.text()
self.assertEqual(resp.status, 200)
self.assertIn('"type": "complete"', body)
self.assertIn('"added": 1', body)
self.assertIn('"failed": 0', body)
class TestMonitorIngestsNonFiniteMetadata(StrictRows, unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.db = PromptDatabase(str(Path(self.temp.name) / "db.sqlite"))
self.pid = self.db.save_prompt(
POSITIVE.strip(), prompt_hash=generate_prompt_hash(POSITIVE)
)
self.tracker = SimpleNamespace(
pop_next_prompt=lambda: None,
get_current_prompt=lambda: None,
extend_prompt_timeout=lambda *args: None,
)
self.handler = ImageGenerationHandler(self.db, self.tracker)
self.handler.logger = logging.getLogger("test.nonfinite.monitor")
self.handler.logger.disabled = True
def tearDown(self):
self.db.close_all()
assert_closed(self, self.temp.name)
self.temp.cleanup()
def test_new_image_with_nan_links_its_pair(self):
# The monitor links only images a PromptManager node produced.
g = graph()
g["1"]["class_type"] = "PromptManager"
path = write_png(Path(self.temp.name) / "out" / "nan.png", g)
self.handler.process_new_image(str(path))
rows = self.image_rows(self.db)
self.assertEqual([row["prompt_id"] for row in rows], [self.pid])
self.assert_strict_snapshot(rows[0])
self.assert_prompt_text(self.db, self.pid)
pairs = self.db.list_prompt_pairs()["pairs"]
self.assertEqual(len(pairs), 1)
self.assertEqual(pairs[0]["image_count"], 1)
def test_legacy_link_stores_strict_json(self):
# The plain prompt link (no exact pair) must not write NaN either.
path = str(Path(self.temp.name) / "plain.png")
metadata = {"prompt": graph(), "workflow": WORKFLOW, "parameters": {"x": NAN}}
self.assertTrue(self.db.link_image_to_prompt(self.pid, path, metadata))
row = self.image_rows(self.db)[0]
self.assert_strict_snapshot(row)
self.assertEqual(strict_loads(row["parameters"]), {"x": None})
def test_stored_images_read_back_without_non_finite_numbers(self):
path = str(Path(self.temp.name) / "legacy.png")
self.db.link_image_to_prompt(self.pid, path, {"prompt": graph()})
with self.db.model.get_connection() as conn:
# A row an older version wrote with Python's default json.dumps.
conn.execute(
"UPDATE generated_images SET prompt_metadata=?", (comfy_text(graph()),)
)
conn.commit()
images = self.db.get_prompt_images(str(self.pid))
strict_loads(json.dumps(images, default=str))
class TestInspectorAnswersStrictJSON(ImageAPITestCase):
async def test_inspect_route_resolves_and_answers_strict_json(self):
write_png(self.output_dir / "nan.png")
response = await self.client.get(
"/prompt_manager/images/inspect/nan.png", params={"root": "0"}
)
self.assertEqual(response.status, 200)
result = strict_loads(await response.text())
self.assertEqual(result["resolution"]["state"], "resolved")
self.assertEqual(
result["prompts"]["positive"],
{"status": "resolved", "text": POSITIVE, "source": "graph-inferred"},
)
async def test_prompt_images_route_answers_strict_json(self):
pid = self.api.db.save_prompt(
POSITIVE.strip(), prompt_hash=generate_prompt_hash(POSITIVE)
)
path = str(write_png(self.output_dir / "linked.png"))
self.api.db.link_image_to_prompt(pid, path, {"prompt": graph()})
with self.api.db.model.get_connection() as conn:
conn.execute(
"UPDATE generated_images SET prompt_metadata=?", (comfy_text(graph()),)
)
conn.commit()
response = await self.client.get(f"/prompt_manager/prompts/{pid}/images")
self.assertEqual(response.status, 200)
result = strict_loads(await response.text())
self.assertEqual(len(result["images"]), 1)
stored = result["images"][0]["prompt_metadata"]
self.assertEqual(stored["1"]["inputs"]["text"], POSITIVE)
self.assertEqual(stored["5"]["is_changed"], [None])
class TestHistoryCapturesNonFiniteGraphs(unittest.TestCase):
"""The queued graph ComfyUI hands the hook may carry float('nan')."""
JOB = "a0c26da3-eb26-405c-b143-c662a84c3112"
def setUp(self):
self.directory = tempfile.TemporaryDirectory()
self.addCleanup(self.directory.cleanup)
self.db = PromptDatabase(os.path.join(self.directory.name, "history.db"))
self.addCleanup(self.db.model.reset_schema_cache)
self.addCleanup(self.db.model.close_all)
def test_queued_graph_with_nan_records_its_run_and_contexts(self):
from tests.test_execution_history_hooks import HostQueue, status
from utils.execution_context_capture import record_runtime_context
from utils.execution_history import register_history_hook
queue = HostQueue()
registration = register_history_hook(
SimpleNamespace(prompt_queue=queue), lambda: self.db
)
self.addCleanup(lambda: registration.dispose(3))
self.addCleanup(registration.writer.invalidate)
deadline = time.monotonic() + 3
while not registration.writer.status().accepting:
self.assertLess(time.monotonic(), deadline, "writer never accepted")
time.sleep(0.005)
g = {
"p": {
"class_type": "PromptManagerPair",
"inputs": {"text": POSITIVE, "negative_text": NEGATIVE},
},
"s": {
"class_type": "KSampler",
"inputs": {"positive": ["p", 0], "negative": ["p", 1], "cfg": INF},
"is_changed": [NAN],
},
"5": graph()["5"],
}
extra = {"extra_pnginfo": {"workflow": WORKFLOW}}
queue.results.append(((0, self.JOB, g, extra, ["s"]), 1))
queue.get()
host = types.ModuleType("comfy_execution.utils")
host.get_executing_context = lambda: SimpleNamespace(
prompt_id=self.JOB, node_id="p", list_index=0
)
with patch.dict(sys.modules, {"comfy_execution.utils": host}):
for role, text in (("positive", POSITIVE), ("negative", NEGATIVE)):
record_runtime_context(
"PromptManagerPair", text, "", "", text, role=role
)
queue.task_done(1, {}, status("execution_success", job=self.JOB))
drain_writer(registration.writer, clock=time.monotonic)
with self.db.model.get_connection() as conn:
conn.row_factory = sqlite3.Row
runs = [dict(r) for r in conn.execute("SELECT * FROM execution_runs")]
self.assertEqual([run["outcome"] for run in runs], ["succeeded"])
contexts = self.db.get_execution_contexts(runs[0]["run_id"])
self.assertEqual(contexts["state"], "complete")
self.assertEqual(contexts["contexts"][0]["positive"]["raw"], POSITIVE)
strict_loads(json.dumps(contexts))
if __name__ == "__main__":
unittest.main()
-66
View File
@@ -160,69 +160,3 @@ class TestPairIngestion(unittest.TestCase):
info = self.db.get_image_prompt_info(self.path)
self.assertEqual(info["pair_context"]["positive_text"], "cat, lora_trigger")
self.assertEqual(info["pair_context"]["text_source"], "execution")
def test_public_ingest_resolved_pair_contract_unchanged(self):
"""The public ingester keeps its result shape and its own transaction;
the extracted ``_ingest_resolved_pair_in`` runs in the caller's."""
from database import pair_ingestion
self.assertTrue(
hasattr(pair_ingestion, "_ingest_resolved_pair_in"),
"_ingest_resolved_pair_in is not extracted",
)
for state in ("ambiguous", "unresolved", "conflict"):
self.assertEqual(
ingest_resolved_pair(self.db, self.path, {}, {"state": state}),
{"status": state, "pairs_added": 0, "links_added": 0},
)
first = ingest_resolved_pair(self.db, self.path, {}, resolved())
self.assertEqual(
set(first),
{"status", "pairs_added", "links_added", "pair_id", "image_id"},
)
self.assertEqual(first["status"], "added")
(link,) = self.db.get_image_prompt_pairs(first["image_id"])
self.assertEqual(
link["provenance"],
{
"resolver_version": 1,
"positive": resolved()["positive"],
"negative": resolved()["negative"],
},
)
conn = self.db.model.get_connection()
self.assertFalse(conn.in_transaction)
# Inside the caller's transaction, with the caller's provenance; an
# exception afterwards rolls the link back with the caller's writes.
provenance = {"kind": "user-selection", "resolver_version": 1}
with self.assertRaises(RuntimeError):
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
self.assertEqual(inner["status"], "added")
raise RuntimeError("caller failed")
self.assertIsNone(self.db.get_image_prompt_info(self.path + "2"))
with self.db._pair_transaction() as held:
inner = pair_ingestion._ingest_resolved_pair_in(
held,
self.db,
self.path + "2",
{},
resolved("dog"),
provenance=provenance,
)
(link,) = self.db.get_image_prompt_pairs(inner["image_id"])
self.assertEqual(link["provenance"], provenance)
with self.db._pair_transaction() as held:
conflict = pair_ingestion._ingest_resolved_pair_in(
held, self.db, self.path, {}, resolved("bird")
)
self.assertEqual(
conflict, {"status": "conflict", "pairs_added": 0, "links_added": 0}
)
+2 -120
View File
@@ -82,8 +82,7 @@ class TestPairScan(AdminAPITestCase):
"class_type": "BasicGuider",
"inputs": {"width": 2048, "height": 2048},
}
# Ambiguous with nothing to choose: unresolved, never a review item.
self.assertEqual(self._scanned("untraceable.png", g), "unresolved")
self.assertEqual(self._scanned("untraceable.png", g), "ambiguous")
self.assertEqual(self.api.db.list_prompt_pairs()["total"], 0)
self.assertEqual(self._texts(), [])
@@ -160,129 +159,12 @@ class TestPairScan(AdminAPITestCase):
"\nSteps: 20, Sampler: Euler"
)
}
# No candidate to choose: reported unresolved, still not guessed.
self.assertEqual(
self.api._ingest_scanned_file(self.output_dir / "ambiguous.png", meta),
"unresolved",
"ambiguous",
)
self.assertEqual(self.api.db.get_recent_prompts()["prompts"], [])
def _nan_scanned(self, name, prompt_graph):
"""Scan ``prompt_graph`` as ComfyUI writes it: ``is_changed`` is NaN."""
stamped = {
node_id: {**node, "is_changed": [float("nan")]}
for node_id, node in prompt_graph.items()
}
text = json.dumps(stamped)
self.assertIn("NaN", text)
return self._scanned_text(name, text)
def _scanned_text(self, name, prompt_text):
meta = {"prompt": prompt_text}
return self.api._ingest_scanned_file(self.output_dir / name, meta)
@staticmethod
def _hires_fix_graph(positive):
"""A PromptManager prompt on a base sampler and a hires-fix sampler.
The base latent comes from a custom batch node and the hires pass from
a custom image filter: no sampler branch traces, so the scan links the
prompt the legacy pages show.
"""
return {
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "m.safetensors"},
},
"2": {
"class_type": "MyTriggerWords",
"inputs": {"trigger_words": ["1", 2]},
},
"3": {
"class_type": "PromptManager",
"inputs": {
"text": positive,
"prepend_text": ["2", 0],
"append_text": "",
"clip": ["1", 1],
},
},
"4": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "blur", "clip": ["1", 1]},
},
"5": {"class_type": "MyLatentBatch", "inputs": {"batch_size": 6}},
"6": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["5", 0],
},
},
"7": {
"class_type": "VAEDecode",
"inputs": {"samples": ["6", 0], "vae": ["1", 2]},
},
"8": {"class_type": "MyImageFilter", "inputs": {"images": ["7", 0]}},
"9": {
"class_type": "VAEEncode",
"inputs": {"pixels": ["8", 0], "vae": ["1", 2]},
},
"14": {
"class_type": "KSampler",
"inputs": {
"seed": 7,
"denoise": 0.25,
"model": ["1", 0],
"positive": ["3", 0],
"negative": ["4", 0],
"latent_image": ["9", 0],
},
},
"15": {
"class_type": "VAEDecode",
"inputs": {"samples": ["14", 0], "vae": ["1", 2]},
},
"16": {"class_type": "SaveImage", "inputs": {"images": ["15", 0]}},
}
def test_hires_fix_prompt_links_with_or_without_nan_is_changed(self):
# Legacy parity: the legacy pages read NaN as null before parsing, so they
# showed this prompt; the scan left NaN-stamped files in review.
for name, scan in (("plain.png", None), ("stamped.png", "nan")):
with self.subTest(name=name):
g = self._hires_fix_graph("cat on a sofa")
if scan:
status = self._nan_scanned(name, g)
else:
status = self._scanned(name, g)
self.assertIn(status, ("added", "linked"))
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_untraced_single_sampler_links_the_legacy_prompt_with_nan(self):
g = self._hires_fix_graph("cat on a sofa")
for node_id in ("7", "8", "9", "14", "15"):
del g[node_id]
g["16"]["inputs"]["images"] = ["6", 0]
self.assertEqual(self._nan_scanned("single.png", g), "added")
self.assertEqual(self._texts(), ["cat on a sofa"])
def test_nan_inside_prompt_text_is_kept(self):
# Only the NaN number reads as null; legacy's regex also rewrote text.
g = self._hires_fix_graph("NaN cat, value: NaN")
self.assertEqual(self._nan_scanned("text.png", g), "added")
self.assertEqual(self._texts(), ["NaN cat, value: NaN"])
def test_infinity_is_still_unreadable_like_the_legacy_pages(self):
text = json.dumps(self._hires_fix_graph("cat")).replace(
'"seed": 7', '"seed": Infinity'
)
self.assertIn(self._scanned_text("inf.png", text), ("ambiguous", "unresolved"))
self.assertEqual(self._texts(), [])
def test_raw_png_workflow_is_stored_as_object_and_repeat_is_unchanged(self):
path = self.output_dir / "workflow.png"
workflow = {"nodes": [{"id": 1, "type": "PromptManagerPair"}]}
-4
View File
@@ -258,10 +258,6 @@ class PairSchemaTests(unittest.TestCase):
],
[(1, 1, 0)],
)
self.assertEqual(
[tuple(r) for r in conn.execute("SELECT * FROM media_review_metadata")],
[(1, 1, 0)],
)
self.assertTrue(PromptModel.verify_database_file(live.db_path)[0])
self.assertTrue(
PromptModel.verify_database_file(live.db_path, require_library_state=True)[
+20 -31
View File
@@ -1245,7 +1245,7 @@ class TestGuiderSamplers(unittest.TestCase):
source["3"]["inputs"]["guider"] = [previous, 0]
self.assertEqual(_inspect(source)["reason_code"], "depth-limit")
def test_custom_advanced_upstream_of_a_sampler_is_a_compared_stage(self):
def test_custom_advanced_upstream_of_a_sampler_is_multistage(self):
source = _guided()
source["41"] = {
"class_type": "KSampler",
@@ -1256,18 +1256,15 @@ class TestGuiderSamplers(unittest.TestCase):
},
}
source["8"]["inputs"]["samples"] = ["41", 0]
# 41 continues 3's latent and reads the same texts as 3's guider.
for anchor in ("9", None):
result = _inspect(source, output_node_id=anchor)
self.assertEqual(
(result["state"], result["sampler_id"]), ("resolved", "41")
)
source["41"]["inputs"]["negative"] = ["6", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(result["sampler_id"], "41")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], "Multistage sampler ancestry is ambiguous")
unanchored = _inspect(source)
self.assertEqual(unanchored["state"], "ambiguous")
self.assertEqual(
sorted(item["sampler_id"] for item in result["candidates"]), ["3", "41"]
sorted(item["sampler_id"] for item in unanchored["candidates"]),
["3", "41"],
)
@@ -2635,19 +2632,17 @@ class TestLatentAncestry(unittest.TestCase):
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
self.assertEqual(_inspect(source, output_node_id="9"), result)
def test_upstream_sampler_through_vaedecode_vaeencode_is_compared(self):
# The latent continues the first stage's through lineage nodes: the
# stages are compared (spec 9.3), equal prompts resolve, others differ.
result = _inspect(_image_to_image(parity.POS), output_node_id="63")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "61"))
self.assertEqual(_sides(result), (parity.POS, parity.NEG))
result = _inspect(_image_to_image(OTHER), output_node_id="63")
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
[item["sampler_id"] for item in result["candidates"]], ["61", "3"]
)
def test_upstream_sampler_through_vaedecode_vaeencode_is_multistage(self):
for positive in (OTHER, parity.POS):
with self.subTest(positive=positive):
result = _inspect(_image_to_image(positive), output_node_id="63")
# Until PR3 compares stage texts, never one stage's prompt.
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["sampler_id"], "61")
self.assertEqual(result["reason"], MULTISTAGE)
self.assertIsNone(result["reason_code"])
self.assertIsNone(result["positive"])
self.assertIsNone(result["negative"])
# An upstream sampler reached through a non-latent input is a stage too.
source = _latent_from(
{
@@ -2964,15 +2959,9 @@ class TestLatentAncestry(unittest.TestCase):
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"], {"id": "60", "class_type": "MysteryUpscaler"})
# The listed stage is now an earlier sampler stage, continued through
# VAEEncode: its prompt equals this stage's, so the chain resolves.
# The listed stage is now an earlier sampler stage in the ancestry.
source["60"]["class_type"] = "UltimateSDUpscale"
result = _inspect(source, output_node_id="9")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "3"))
# Reached only through an unlisted image node, it stays unattributed.
source["61"] = {"class_type": "MysteryFilter", "inputs": {"image": ["60", 0]}}
source["5"]["inputs"]["pixels"] = ["61", 0]
result = _inspect(source, output_node_id="9")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_latent_walk_budget_exhaustion_is_work_limit(self):
-419
View File
@@ -1,419 +0,0 @@
"""Multi-stage chains joined through image-space pass-through nodes.
A hires-fix or refiner run decodes sampler A's latent, filters, upscales and
scales the image, then encodes it again for sampler B. The semantics table's
``lineage`` section lists the nodes through which the latent or image carries
on unchanged in kind. When B's latent descends from A only through those nodes
and the stages share one pair, the chain resolves to it. A last stage that adds
conditioning to the base prompt (ConditioningCombine of the base and a style
trigger) keeps the base pair: the floor legacy linked. Everything else stays
ambiguous or unresolved. Ingestion (the default profile) is unchanged (D1).
"""
import json
import unittest
from utils import prompt_pair_resolver as resolver
from utils.prompt_inspector import inspect_prompt_evidence
from utils.prompt_pair_resolver import resolve_prompt_pair
try:
from tests import passthrough_graphs as shapes
except ImportError: # discovered with ``-s tests``
import passthrough_graphs as shapes
def _inspect(graph, **options):
return resolve_prompt_pair(
{"prompt": graph}, profile=resolver.INSPECTION_PROFILE, **options
)
def _texts(result):
return tuple(
result[role]["text"] if isinstance(result[role], dict) else None
for role in ("positive", "negative")
)
class TestHiresFixChain(unittest.TestCase):
def test_shared_pair_resolves_through_image_space_pass_through(self):
graph = shapes.hires_fix()
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(result["sampler_id"], "20")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertIsNone(result["reason_code"])
def test_the_earlier_stage_is_judged_in_its_chain(self):
# Sampler 10 only feeds sampler 20, so the file is 20's: one candidate.
result = _inspect(shapes.hires_fix())
self.assertEqual(result["candidates"], [])
def test_inspector_shows_the_shared_prompt(self):
evidence = inspect_prompt_evidence({"prompt": json.dumps(shapes.hires_fix())})
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
[evidence["prompts"][role]["text"] for role in ("positive", "negative")],
[shapes.POSITIVE, shapes.NEGATIVE],
)
def test_different_stage_prompts_are_multistage_ambiguous(self):
result = _inspect(shapes.hires_fix(second_positive=shapes.OTHER))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason_code"], "multistage-differs")
self.assertEqual(
sorted(_texts(item)[0] for item in result["candidates"]),
sorted([shapes.POSITIVE, shapes.OTHER]),
)
def test_an_unlisted_image_node_breaks_the_lineage(self):
# The walk still finds the earlier sampler, but not through listed
# pass-through nodes: the stages are never compared.
result = _inspect(shapes.hires_fix(filter_class="MysteryImageEffect"))
self.assertEqual(result["state"], "ambiguous")
later = next(
item for item in result["candidates"] if item["sampler_id"] == "20"
)
self.assertEqual(later["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_second_earlier_sampler_in_the_ancestry_stays_unattributed(self):
graph = shapes.hires_fix()
graph["30"] = shapes.ksampler(["2", 0], ["3", 0], ["4", 0])
graph["31"] = {"class_type": "VAEDecode", "inputs": {"samples": ["30", 0]}}
graph["32"] = {
"class_type": "ImageBlend",
"inputs": {"image1": ["14", 0], "image2": ["31", 0], "blend_factor": 0.5},
}
graph["16"]["inputs"]["image"] = ["32", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], resolver.MULTISTAGE_UNATTRIBUTED_REASON)
def test_a_direct_latent_link_is_a_chain_too(self):
graph = shapes.hires_fix()
graph["20"]["inputs"]["latent_image"] = ["10", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual((result["state"], result["sampler_id"]), ("resolved", "20"))
def test_an_unreadable_earlier_stage_blocks_the_chain(self):
graph = shapes.hires_fix(second_positive=shapes.POSITIVE)
graph["10"]["inputs"]["positive"] = ["6", 0]
graph["6"] = {"class_type": "MysteryPrompt", "inputs": {"seed": 1}}
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryPrompt")
def test_generated_text_stays_untraceable(self):
result = _inspect(shapes.generated_text_chain())
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "TextGenerate")
def test_ingestion_is_unchanged(self):
# D1: the default profile keeps its historical latent rule.
result = resolve_prompt_pair({"prompt": shapes.hires_fix()})
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(
{item["reason"] for item in result["candidates"]},
{"Unsupported latent ancestry"},
)
class TestFluxThreeStage(unittest.TestCase):
def test_added_style_conditioning_keeps_the_base_pair(self):
graph = shapes.flux_three_stage()
for anchor in (None, "61"):
with self.subTest(anchor=anchor):
result = _inspect(graph, output_node_id=anchor)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
self.assertEqual(result["sampler_id"], "50")
# D126: the positive says a later stage added conditioning.
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
self.assertEqual(
result["negative"]["provenance"]["text_source"],
"zeroed-conditioning",
)
def test_any_added_conditioning_is_labelled_never_measured(self):
# Ruling I2: a whole other prompt combined in is labelled the same way;
# no length or "short trigger" heuristic.
graph = shapes.flux_three_stage()
graph["31"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], shapes.POSITIVE)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.ADDED_CONDITIONING_SOURCE,
)
def test_trigger_words_and_added_conditioning_carry_both_labels(self):
graph = shapes.flux_three_stage()
graph["7"] = {
"class_type": "Lora Loader (LoraManager)",
"inputs": {"model": ["1", 0], "clip": ["2", 0], "text": "x"},
}
graph["8"] = shapes.trigger_toggle()
graph["30"]["inputs"]["prepend_text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["state"], "resolved")
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_AND_ADDED_SOURCE,
)
def test_a_custom_advanced_last_stage_keeps_its_failure(self):
# Review M2: a guider's combined positive is not floored.
graph = shapes.flux_three_stage()
graph["60"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {"guider": ["63", 0], "latent_image": ["66", 0]},
}
graph["63"] = {
"class_type": "CFGGuider",
"inputs": {"positive": ["35", 0], "negative": ["33", 0]},
}
graph["66"] = {"class_type": "VAEEncode", "inputs": {"pixels": ["51", 0]}}
graph["67"] = {"class_type": "VAEDecode", "inputs": {"samples": ["60", 0]}}
graph["61"]["inputs"]["images"] = ["67", 0]
result = _inspect(graph, output_node_id="61")
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_concat_also_adds_to_the_base_prompt(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningConcat"))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, ""))
def test_an_average_blends_the_base_away(self):
result = _inspect(shapes.flux_three_stage(composite="ConditioningAverage"))
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_combine_without_the_base_prompt_is_not_the_base_pair(self):
graph = shapes.flux_three_stage(combined=(("37", 0), ("31", 0)))
graph["37"] = shapes.encode(shapes.OTHER)
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_a_different_last_negative_is_not_the_base_pair(self):
graph = shapes.flux_three_stage()
graph["38"] = shapes.encode("watermark")
graph["60"]["inputs"]["negative"] = ["38", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
def test_different_base_stages_are_never_floored(self):
graph = shapes.flux_three_stage()
graph["39"] = shapes.encode(shapes.OTHER)
graph["40"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertNotEqual(result["state"], "resolved")
def test_a_rerouted_combine_is_followed(self):
graph = shapes.flux_three_stage()
graph["39"] = {"class_type": "Reroute", "inputs": {"input": ["35", 0]}}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(result["reason"], resolver.ADDED_CONDITIONING_REASON)
def test_an_unreadable_last_negative_or_encoder_parts_keep_the_failure(self):
graph = shapes.flux_three_stage()
graph["38"] = {"class_type": "MysteryNegative", "inputs": {"seed": 1}}
graph["60"]["inputs"]["negative"] = ["38", 0]
self.assertEqual(_inspect(graph)["reason_code"], "composite")
# Differing encoder parts are a composite too, but not an added stage.
graph = shapes.flux_three_stage()
graph["39"] = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {"text_g": shapes.POSITIVE, "text_l": shapes.STYLE},
}
graph["60"]["inputs"]["positive"] = ["39", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "composite")
)
self.assertEqual(result["node"]["class_type"], "CLIPTextEncodeSDXL")
def test_inspector_shows_the_base_prompt(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.flux_three_stage())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(evidence["prompts"]["positive"]["text"], shapes.POSITIVE)
class TestRunTimePromptManagerParts(unittest.TestCase):
"""Maintainer option A: a run-time prepend or append is left out, labelled."""
def test_trigger_words_prepend_resolves_to_the_prompt_manager_text(self):
for anchor in (None, "22"):
with self.subTest(anchor=anchor):
result = _inspect(
shapes.hires_fix_with_trigger_words(), output_node_id=anchor
)
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result), (shapes.POSITIVE, shapes.NEGATIVE))
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
provenance = result["positive"]["provenance"]
self.assertEqual(
provenance["text_source"], resolver.RUNTIME_PARTS_SOURCE
)
self.assertEqual(result["positive"]["storage_text"], shapes.POSITIVE)
# The negative is untouched by the positive's run-time part.
self.assertEqual(
result["negative"]["provenance"]["text_source"], "graph-inferred"
)
def test_append_and_a_single_sampler_are_labelled_too(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = "masterpiece"
graph["2"]["inputs"]["append_text"] = ["8", 0]
graph["20"]["inputs"]["latent_image"] = ["4", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["state"], "resolved")
# The literal prepend is still combined exactly as before.
self.assertEqual(_texts(result)[0], "masterpiece " + shapes.POSITIVE)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
def test_the_inspector_shows_the_prompt_and_its_source(self):
evidence = inspect_prompt_evidence(
{"prompt": json.dumps(shapes.hires_fix_with_trigger_words())}
)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(
evidence["prompts"]["positive"],
{
"status": "resolved",
"text": shapes.POSITIVE,
"source": resolver.RUNTIME_PARTS_SOURCE,
},
)
def test_literal_and_traceable_parts_are_combined_as_before(self):
graph = shapes.hires_fix_with_trigger_words()
graph["8"] = {"class_type": "PrimitiveString", "inputs": {"value": "trig9word"}}
result = _inspect(graph)
self.assertEqual(_texts(result)[0], "trig9word " + shapes.POSITIVE)
self.assertEqual(result["reason"], "")
self.assertEqual(
result["positive"]["provenance"]["text_source"], "graph-inferred"
)
def test_only_a_run_time_feeder_is_left_out(self):
# An unknown feeder stays unsupported, naming it.
graph = shapes.hires_fix_with_trigger_words()
graph["8"]["class_type"] = "MysteryWords"
result = _inspect(graph)
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"]["class_type"], "MysteryWords")
# A run-time node deeper inside the part makes the whole part run time,
# but it is not a direct feeder: never leave out a traced literal.
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = {
"class_type": "StringConcatenate",
"inputs": {"string_a": "kept", "string_b": ["8", 0], "delimiter": ""},
}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_main_text_from_a_run_time_node_is_never_left_out(self):
graph = shapes.hires_fix_with_trigger_words()
graph["2"]["inputs"]["prepend_text"] = ""
graph["2"]["inputs"]["text"] = ["8", 0]
result = _inspect(graph)
self.assertEqual(result["reason_code"], "runtime-value")
def test_the_label_survives_nodes_between_the_prompt_manager_and_sampler(self):
# Review C1: encoders, chained PromptManagers and concatenations.
expected = {
"text-encode": shapes.POSITIVE,
"chained": shapes.POSITIVE,
"concat": shapes.POSITIVE + ", extra",
}
for shape, text in expected.items():
with self.subTest(shape=shape):
result = _inspect(shapes.trigger_words_through(shape))
self.assertEqual(result["state"], "resolved")
self.assertEqual(_texts(result)[0], text)
self.assertEqual(result["reason"], resolver.RUNTIME_PARTS_REASON)
self.assertEqual(
result["positive"]["provenance"]["text_source"],
resolver.RUNTIME_PARTS_SOURCE,
)
def test_a_partial_prompt_never_equals_a_complete_one(self):
# Review C2: never collapse a branch that left out trigger words with
# one that did not, nor compare such stages as equal.
result = _inspect(shapes.branches_with_and_without_trigger_words())
self.assertEqual(
(result["state"], result["reason"]),
("ambiguous", "Multiple sampler branches"),
)
graph = shapes.hires_fix_with_trigger_words()
graph["9"] = shapes.prompt_manager(shapes.POSITIVE)
graph["20"]["inputs"]["positive"] = ["9", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(
(result["state"], result["reason_code"]),
("ambiguous", "multistage-differs"),
)
# Two PromptManagers leaving out two different toggles differ too.
graph["8b"] = shapes.trigger_toggle("otherword")
graph["9"]["inputs"]["prepend_text"] = ["8b", 0]
result = _inspect(graph, output_node_id="22")
self.assertEqual(result["reason_code"], "multistage-differs")
def test_only_listed_run_time_classes_are_left_out(self):
# Review M1: a JoinStringMulti with a linked list flag is run time
# too, but it is not a listed run-time node: never left out.
graph = shapes.hires_fix()
graph["9"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": " ",
"return_list": ["98", 0],
"string_1": "static words",
"string_2": "more",
},
}
graph["98"] = {"class_type": "PrimitiveBoolean", "inputs": {"value": False}}
graph["2"]["inputs"]["prepend_text"] = ["9", 0]
result = _inspect(graph)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "runtime-value")
)
self.assertEqual(result["node"]["class_type"], "JoinStringMulti")
def test_ingestion_is_unchanged(self):
result = resolve_prompt_pair({"prompt": shapes.hires_fix_with_trigger_words()})
self.assertEqual(result["state"], "ambiguous")
class TestStillAmbiguous(unittest.TestCase):
def test_edit_batch_with_distinct_prompts_stays_ambiguous(self):
result = _inspect(shapes.edit_batch(4))
self.assertEqual(result["state"], "ambiguous")
self.assertEqual(result["reason"], "Multiple sampler branches")
self.assertEqual(len({_texts(item)[0] for item in result["candidates"]}), 4)
if __name__ == "__main__":
unittest.main()
-1053
View File
File diff suppressed because it is too large Load Diff
-335
View File
@@ -1,335 +0,0 @@
"""Needs review queues a file only when there is something to choose.
The maintainer's live queue (#284 / #288) held 39 items whose 11 candidates
all read "unresolved", "No positive / No negative": the ingestion resolver
found several sampler branches but could trace no prompt on any of them.
An ambiguous result is reviewable only when at least two candidates carry a
resolved prompt text. With none, the file could not be traced; with exactly
one, the other branches may still have made the file, so the one text is
not certain either (the resolver never flattens unknown semantics). Both
stay unresolved and are never queued, and a rescan removes the pending,
undecided items such files already have.
"""
import os
import unittest
from unittest.mock import patch
try:
from tests.test_review import DELIMITERS, write_png
from tests.test_review_outcome import OutcomeTestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_review import DELIMITERS, write_png
from test_review_outcome import OutcomeTestCase
from database.review_schema import evidence_offers_choice
from utils import review_capture
def opaque_graph(*opaque, traced=()):
"""Several KSamplers: ``opaque`` positives come from an encoder the
ingestion resolver does not know, ``traced`` ones from CLIPTextEncode.
The first sampler is opaque, so the scan's browser-parser fallback links
nothing either.
"""
graph = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "m"}},
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
}
sides = [("OpaqueTextEncode", "prompt", t) for t in opaque]
sides += [("CLIPTextEncode", "text", t) for t in traced]
for index, (encoder_class, key, text) in enumerate(sides):
encoder, sampler = str(10 + 2 * index), str(11 + 2 * index)
graph[encoder] = {"class_type": encoder_class, "inputs": {key: text}}
graph[sampler] = {
"class_type": "KSampler",
"inputs": {
"model": ["1", 0],
"positive": [encoder, 0],
"negative": ["2", 0],
},
}
return graph
def _always_reviewable(_resolution):
return True
class ChoosableTestCase(OutcomeTestCase):
def _record_pre_fix(self, name):
"""An item as the pre-fix recorder stored it (no choosable test)."""
with patch.object(review_capture, "reviewable", _always_reviewable):
self._record(name)
def _write(self, name, graph):
write_png(self.output_dir / name, prompt=graph)
return self.output_dir / name
class TestChoosableCandidates(unittest.TestCase):
def _resolution(self, *states):
candidates = []
for index, state in enumerate(states):
side = {"text": f"t{index}"} if state == "resolved" else None
candidates.append(
{"sampler_id": str(index), "state": state, "positive": side}
)
return {"state": "ambiguous", "candidates": candidates}
def test_two_resolved_candidates_are_reviewable(self):
resolution = self._resolution("resolved", "resolved", "unresolved")
self.assertTrue(review_capture.reviewable(resolution))
def test_none_or_one_resolved_candidate_is_not_reviewable(self):
for states in ((), ("unresolved",) * 11, ("resolved", "unresolved")):
with self.subTest(states=states):
resolution = self._resolution(*states)
self.assertFalse(review_capture.reviewable(resolution))
def test_blank_or_missing_text_does_not_count(self):
resolution = self._resolution("resolved", "resolved")
resolution["candidates"][0]["positive"] = {"text": " "}
self.assertFalse(review_capture.reviewable(resolution))
resolution["candidates"][0]["positive"] = None
self.assertFalse(review_capture.reviewable(resolution))
def test_nested_ambiguous_stage_counts_its_own_candidates(self):
inner = self._resolution("resolved", "resolved")["candidates"]
resolution = {
"state": "ambiguous",
"candidates": [{"state": "ambiguous", "candidates": inner}],
}
self.assertTrue(review_capture.reviewable(resolution))
def test_a_result_that_is_not_ambiguous_is_not_reviewable(self):
resolution = self._resolution("resolved", "resolved")
resolution["state"] = "resolved"
self.assertFalse(review_capture.reviewable(resolution))
class TestStoredEvidenceOffersChoice(unittest.TestCase):
def test_unlimited_evidence_needs_two_choosable_candidates(self):
side = {"text": "cat"}
two = [{"state": "resolved", "positive": side}] * 2
self.assertTrue(evidence_offers_choice({"candidates": two}))
one = two[:1] + [{"state": "unresolved", "positive": None}]
self.assertFalse(evidence_offers_choice({"candidates": one}))
def test_limited_evidence_offers_a_choice_only_with_two_candidates(self):
self.assertTrue(
evidence_offers_choice({"limited": True, "candidate_total": 18})
)
for total in (0, 1, None, "18", True):
with self.subTest(total=total):
evidence = {"limited": True, "candidate_total": total}
self.assertFalse(evidence_offers_choice(evidence))
def test_malformed_evidence_offers_nothing(self):
for evidence in (None, [], {"candidates": "x"}, {}):
with self.subTest(evidence=evidence):
self.assertFalse(evidence_offers_choice(evidence))
class TestNothingToChooseIsNotQueued(ChoosableTestCase):
async def test_no_candidate_with_text_could_not_be_traced(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_items"], 0)
(row,) = job["outcomes"]["review"]
self.assertEqual(row["path"], "none.png")
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"]["review"], 1, job["counters"])
self.assertEqual(self._capture_files(), [])
async def test_exactly_one_candidate_with_text_stays_unresolved(self):
"""One traced branch beside untraced ones is not certain: no link."""
self._roots()
self._write("one.png", opaque_graph("cat", traced=("dog",)))
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
self.assertEqual(job["counters"].get("added", 0), 0, job["counters"])
self.assertEqual(job["counters"].get("known", 0), 0, job["counters"])
async def test_two_candidates_with_text_are_queued(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
(row,) = job["outcomes"]["review"]
self.assertIn("more than one prompt could match", row["detail"])
async def test_parameters_delimiter_ambiguity_is_not_queued(self):
"""Two A1111 negative delimiters offer no candidate to choose."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
job = await self._scan()
self.assertEqual(self._items(), [])
(row,) = job["outcomes"]["review"]
self.assertIn("could not be traced", row["detail"])
def test_recorder_reports_untraced_and_stores_nothing(self):
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
for name in ("none.png", "one.png"):
with self.subTest(name=name):
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertEqual(outcome, "untraced")
self.assertEqual(self._items(), [])
self.assertEqual(self._capture_files(), [])
def test_monitor_does_not_queue_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
class TestUntracedItemsLeaveTheQueue(ChoosableTestCase):
async def test_rescan_removes_pending_items_without_a_choice(self):
"""The maintainer's 39 items: queued before the rule, gone on rescan."""
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._write("one.png", opaque_graph("cat", traced=("dog",)))
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
for name in ("none.png", "one.png", "two.png"):
self._record_pre_fix(name)
revision = self._review_revision()
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(self._pending(), 1)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_untraced"], 2)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertIn(
"= 2 images left Needs review · nothing to choose", "\n".join(job["log"])
)
async def test_decided_items_of_untraced_files_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
self._write(name, opaque_graph("cat", traced=(name, name + "2")))
self._record_pre_fix(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# The files now carry graphs without a choice, queued pre-fix.
for name in ("sel.png", "unres.png", "pending.png"):
self._write(name, opaque_graph("a cow", "dog"))
self._record_pre_fix(name)
self.assertEqual(self._pending(), 3)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_untraced"], 3)
async def test_an_item_with_a_choice_survives_beside_an_untraced_one(self):
"""Removal judges each item by its own evidence: an item that offers
a choice is not removed although the file's new bytes offer none."""
self._roots()
self._write("f.png", opaque_graph("x", traced=("cat", "dog")))
self._record_pre_fix("f.png")
(choosable,) = self._items()
self._write("f.png", opaque_graph("a cow", "dog"))
self._record_pre_fix("f.png")
self.assertEqual(self._pending(), 2)
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [choosable["id"]])
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_limited_items_go_only_without_enough_candidates(self):
"""A pre-fix delimiter item (no candidate) goes; a limited item of 18
candidates may hold a choice and stays, though the bytes now offer
none."""
self._roots()
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
self._record_pre_fix("delims.png")
many = [f"prompt {n}" for n in range(17)]
self._write("many.png", opaque_graph("x", traced=many))
self._record_pre_fix("many.png")
kept = [i["id"] for i in self._items() if "many" in i["locator"]]
self._write("many.png", opaque_graph("a cow", "dog"))
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], kept)
self.assertEqual(job["scope"]["review_untraced"], 1)
async def test_choosable_pending_item_stays(self):
self._roots()
self._write("two.png", opaque_graph("x", traced=("cat", "dog")))
await self._scan()
(item,) = self._items()
job = await self._scan("all")
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"]["review_untraced"], 0)
async def test_a_failed_removal_is_logged_and_counted_not_fatal(self):
import logging
import sqlite3
self._roots()
self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
with (
patch.object(
self.api.db,
"settle_untraced_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan("all")
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
await self._scan("all")
self.assertEqual(self._pending(), 0)
def test_monitor_removes_the_pending_item_of_an_untraced_output(self):
self._roots()
path = self._write("none.png", opaque_graph("cat", "dog"))
self._record_pre_fix("none.png")
# A pre-fix item of other bytes keeps the stat shortcut out of the way.
self._write("none.png", opaque_graph("a cow", "dog"))
self.assertEqual(self.api.record_review_file(str(path)), "untraced")
self.assertEqual(self._items(), [])
if __name__ == "__main__":
unittest.main()
-374
View File
@@ -1,374 +0,0 @@
"""Needs review takes only files whose final outcome is ambiguous and unlinked.
The maintainer's live scan (#284 / #288) queued every file the strict
resolver found ambiguous, also those the legacy browser-parser fallback then
linked and those already linked, so the queue (10,232) was far larger than
the scan's Needs review group. The rule now: a file is queued only when the
resolver finds it ambiguous, the fallback does not link it and no saved row
links it. A pending, undecided item whose file is linked (by the fallback, a
later resolution or the monitor) is removed on the next scan or monitor
pass; an item that carries a decision is never touched.
"""
import json
import logging
import os
import sqlite3
import unittest
from unittest.mock import patch
try:
from tests.test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
except ImportError: # discovered with tests/ as the top-level directory
from test_review import (
DELIMITERS,
NOW,
ReviewTestCase,
ambiguous_graph,
resolved_graph,
unlinkable_graph,
write_png,
)
from utils.hashing import generate_prompt_hash
from utils.image_monitor import ImageGenerationHandler
from utils.prompt_tracker import PromptTracker
from utils import review_capture
def nan_stamped(graph):
"""ComfyUI's ``is_changed: [NaN]`` stamp (written as a bare NaN literal)."""
for node in graph.values():
node["is_changed"] = [float("nan")]
return graph
class OutcomeTestCase(ReviewTestCase):
def _pending(self):
return self.api.db.library_counts()["review_pending"]
def _record(self, name):
"""An item as the pre-fix producers recorded it (any ambiguous file)."""
outcome = review_capture.record_ambiguous_file(
self.api.db, self.api._captures, self.output_dir, 0, name
)
self.assertIn(outcome, ("recorded", "limited"))
def _decide(self, item, outcome):
conn = self.api.db.model.get_connection()
candidate = None
if outcome == "selected":
candidate = json.loads(item["evidence"])["candidates"][0]["hash"]
with conn:
conn.execute(
"UPDATE media_review_items SET state=?, revision=1 WHERE id=?",
(outcome, item["id"]),
)
conn.execute(
"INSERT INTO media_review_decisions(id,item_id,decision_revision,"
"created_at,outcome,candidate_hash,evidence) VALUES(?,?,?,?,?,?,?)",
(
f"00000000-0000-4000-8000-{len(self._decisions()) + 1:012d}",
item["id"],
1,
NOW,
outcome,
candidate,
'{"kind":"user-selection"}',
),
)
def _decisions(self):
conn = self.api.db.model.get_connection()
return [
tuple(row)
for row in conn.execute(
"SELECT id,item_id,outcome FROM media_review_decisions ORDER BY id"
)
]
def _link(self, path, text="already saved"):
prompt_id = self.api.db.save_prompt(
text=text, prompt_hash=generate_prompt_hash(text)
)
self.api.db.link_image_to_prompt(prompt_id, str(path))
return prompt_id
@staticmethod
def _rows(job, group):
return sorted(row["path"] for row in job["outcomes"].get(group, []))
class TestOnlyUnlinkedAmbiguousFilesAreQueued(OutcomeTestCase):
async def test_nan_stamped_graph_the_fallback_links_is_not_queued(self):
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self.assertIn(b"NaN", path.read_bytes())
job = await self._scan()
self.assertEqual(job["counters"]["added"], 1, job["counters"])
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._pending(), 0)
async def test_fallback_link_to_a_known_prompt_is_not_queued(self):
self._roots()
write_png(self.output_dir / "a.png", prompt=ambiguous_graph("cat", "dog"))
await self._scan()
write_png(self.output_dir / "b.png", prompt=ambiguous_graph("cat", "dog"))
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
async def test_already_linked_file_is_not_queued(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._link(path)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(job["scope"]["review_items"], 0)
self.assertEqual(self._rows(job, "review"), [])
self.assertEqual(self._rows(job, "known"), ["linked.png"])
async def test_genuinely_ambiguous_file_is_queued(self):
self._roots()
path = self.output_dir / "two.png"
write_png(path, prompt=nan_stamped(unlinkable_graph("cat", "dog")))
# Beside it, a file the fallback links: only the first is queued.
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
job = await self._scan()
(item,) = self._items()
self.assertEqual(os.path.basename(item["locator"]), "two.png")
self.assertEqual(item["state"], "pending")
self.assertEqual(item["candidate_count"], 3)
self.assertEqual(job["scope"]["review_items"], 1)
self.assertEqual(self._rows(job, "review"), ["two.png"])
self.assertEqual(self._pending(), 1)
self.assertIn("? 1 image added to Needs review", "\n".join(job["log"]))
async def test_counter_and_queue_agree_with_the_review_group(self):
"""``review_items`` and the queue total equal the review group less
its unresolved files, on the first scan and on a rescan."""
self._roots()
for name in ("u1.png", "u2.png"):
write_png(self.output_dir / name, prompt=unlinkable_graph("cat", name))
write_png(self.output_dir / "delims.png", parameters=DELIMITERS)
write_png(self.output_dir / "linked.png", prompt=ambiguous_graph("a", "b"))
unresolved = {"1": {"class_type": "KSampler", "inputs": {}}}
write_png(self.output_dir / "none.png", prompt=unresolved)
write_png(self.output_dir / "one.png", prompt=resolved_graph())
for _ in range(2):
job = await self._scan()
review = self._rows(job, "review")
unresolved_rows = sorted(
row["path"]
for row in job["outcomes"]["review"]
if "could not be traced" in row["detail"]
)
queued = [name for name in review if name not in unresolved_rows]
# delims.png has nothing to choose: "could not be traced".
self.assertEqual(queued, ["u1.png", "u2.png"], job)
self.assertIn("delims.png", unresolved_rows)
self.assertEqual(job["scope"]["review_items"], 2)
self.assertEqual(self._pending(), 2)
self.assertIn("? 2 images already in Needs review", "\n".join(job["log"]))
class TestQueueSelfHeals(OutcomeTestCase):
async def test_pending_item_of_a_fallback_linked_file_is_removed(self):
"""The maintainer's case: queued by the old rule, linked on rescan."""
self._roots()
path = self.output_dir / "nan.png"
write_png(path, prompt=nan_stamped(ambiguous_graph("cat", "dog")))
self._record("nan.png")
revision = self._review_revision()
self.assertEqual(self._pending(), 1)
job = await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
self.assertGreater(self._review_revision(), revision)
self.assertEqual(job["scope"]["review_settled"], 1)
self.assertIn("= 1 image left Needs review · now linked", "\n".join(job["log"]))
async def test_pending_item_of_an_already_linked_file_is_removed(self):
self._roots()
path = self.output_dir / "linked.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("linked.png")
self._link(path)
await self._scan()
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_pending_item_of_a_skipped_known_file_is_removed(self):
"""A file the "new" scope skips as known still settles its old item."""
self._roots()
path = self.output_dir / "one.png"
write_png(path, prompt=unlinkable_graph("cat", "dog"))
self._record("one.png")
write_png(path, prompt=resolved_graph())
first = await self._scan()
self.assertEqual(self._rows(first, "added"), ["one.png"])
# An item a pre-fix scan left for the file's old bytes.
self.assertEqual(self._items(), [])
write_png(path, prompt=unlinkable_graph("cat", "dog"), color=(0, 9, 0))
self._record("one.png")
write_png(path, prompt=resolved_graph())
second = await self._scan()
self.assertEqual(self._rows(second, "known"), ["one.png"])
self.assertEqual(self._items(), [])
async def test_a_failed_settle_is_logged_and_counted_not_fatal(self):
self._roots()
write_png(self.output_dir / "nan.png", prompt=ambiguous_graph("cat", "dog"))
self._record("nan.png")
with (
patch.object(
self.api.db,
"settle_linked_review_items",
side_effect=sqlite3.OperationalError("database is locked"),
),
self.assertLogs("prompt_manager.api", logging.WARNING) as logs,
):
job = await self._scan()
self.assertEqual(job["state"], "finished")
self.assertEqual(job["scope"]["review_errors"], 1)
self.assertEqual(job["scope"]["review_settled"], 0)
self.assertTrue(any("could not be removed" in line for line in logs.output))
self.assertEqual(self._pending(), 1)
# The next scan settles it.
await self._scan()
self.assertEqual(self._pending(), 0)
async def test_still_unlinked_pending_item_stays(self):
self._roots()
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
await self._scan()
(item,) = self._items()
job = await self._scan()
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
self.assertEqual(job["scope"].get("review_settled", 0), 0)
async def test_decided_items_are_never_touched(self):
self._roots()
for name in ("sel.png", "unres.png"):
write_png(self.output_dir / name, prompt=ambiguous_graph("cat", name))
self._record(name)
items = {os.path.basename(i["locator"]): i for i in self._items()}
self._decide(items["sel.png"], "selected")
self._decide(items["unres.png"], "unresolved")
before = (self._items(), self._decisions())
# A pending neighbour the fallback links: it alone leaves the queue.
write_png(self.output_dir / "pending.png", prompt=ambiguous_graph("x", "y"))
self._record("pending.png")
self.assertEqual(self._pending(), 1)
job = await self._scan("all")
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
self.assertEqual(job["scope"]["review_settled"], 1)
class TestMonitorRule(OutcomeTestCase):
def _handler(self):
handler = ImageGenerationHandler(self.api.db, PromptTracker(self.api.db))
handler.review_recorder = self.api.record_review_file
return handler
def _context(self):
prompt_id = self.api.db.save_prompt(
text="latest", prompt_hash=generate_prompt_hash("latest")
)
return {"id": prompt_id, "text": "latest", "fallback": True}
async def test_monitor_linked_ambiguous_output_is_not_queued(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
async def test_monitor_link_removes_the_files_pending_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
self.assertEqual(self._pending(), 1)
self._handler().link_image_to_prompt(
str(path), self._context(), {"prompt": graph}
)
self.assertEqual(self._items(), [])
self.assertEqual(self._pending(), 0)
async def test_monitor_never_touches_a_decided_item(self):
self._roots()
path = self.output_dir / "two.png"
graph = unlinkable_graph("cat", "dog")
write_png(path, prompt=graph)
self._record("two.png")
(item,) = self._items()
self._decide(item, "unresolved")
before = (self._items(), self._decisions())
other = self.output_dir / "other.png"
write_png(other, prompt=graph, color=(0, 0, 9))
self._record("other.png")
handler, context = self._handler(), self._context()
handler.link_image_to_prompt(str(path), context, {"prompt": graph})
handler.link_image_to_prompt(str(other), context, {"prompt": graph})
self.assertEqual((self._items(), self._decisions()), before)
self.assertEqual(self._pending(), 0)
async def test_record_review_file_applies_the_scan_rule(self):
self._roots()
write_png(self.output_dir / "fallback.png", prompt=ambiguous_graph("a", "b"))
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
linked = self.output_dir / "linked.png"
write_png(linked, prompt=unlinkable_graph("cat", "dog"))
self._link(linked)
record = self.api.record_review_file
self.assertEqual(record(str(self.output_dir / "fallback.png")), "skipped")
self.assertEqual(record(str(linked)), "linked")
self.assertEqual(record(str(self.output_dir / "two.png")), "recorded")
self.assertEqual(len(self._items()), 1)
if __name__ == "__main__":
unittest.main()
-918
View File
@@ -1,918 +0,0 @@
"""Needs review schema group (#249): install, validation and restore safety."""
import hashlib
import json
import re
import sqlite3
import tempfile
import unittest
from contextlib import closing
from pathlib import Path
from unittest.mock import patch
try:
from tests.open_handles import assert_closed
except ImportError: # discovered with tests/ as the top-level directory
from open_handles import assert_closed
from database.models import PromptModel
from database.operations import PromptDatabase
try: # guarded so a missing module fails assertions, not test loading
from database.review_schema import (
canonical_evidence,
evidence_digest,
migrate_review_state,
review_schema_error,
)
except ImportError:
canonical_evidence = None
evidence_digest = None
migrate_review_state = None
review_schema_error = None
METADATA = "media_review_metadata"
ITEMS = "media_review_items"
DECISIONS = "media_review_decisions"
TABLES = (METADATA, ITEMS, DECISIONS)
ANNOTATION_TABLES = (
"media_annotation_metadata",
"media_annotation_files",
"media_annotation_bytes",
"media_image_tags",
)
MAX_EVIDENCE_BYTES = 8 * 1024 * 1024
# The exact DDL the plan pins; the module must produce precisely these objects.
DDL = {
METADATA: (
f"CREATE TABLE {METADATA}(singleton_id INTEGER PRIMARY KEY "
"CHECK(singleton_id=1), schema_version INTEGER NOT NULL "
"CHECK(schema_version=1), review_index_revision INTEGER NOT NULL "
"DEFAULT 0 CHECK(review_index_revision>=0))"
),
ITEMS: (
f"CREATE TABLE {ITEMS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
"byte_id TEXT NOT NULL REFERENCES media_annotation_bytes(id) "
"ON DELETE RESTRICT, spelled_path TEXT NOT NULL CHECK(length(CAST("
"spelled_path AS BLOB)) BETWEEN 1 AND 16384), "
"resolver_version INTEGER NOT NULL, "
"evidence_hash TEXT NOT NULL CHECK(length(evidence_hash)=64 AND "
"evidence_hash NOT GLOB '*[^0-9a-f]*'), evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"candidate_count INTEGER NOT NULL CHECK(candidate_count BETWEEN 0 AND 16), "
"limited INTEGER NOT NULL CHECK(limited IN (0,1)), state TEXT NOT NULL "
"CHECK(state IN ('pending','selected','unresolved')), revision INTEGER "
"NOT NULL CHECK(revision>=0), created_at TEXT NOT NULL, updated_at TEXT "
"NOT NULL, UNIQUE(byte_id, resolver_version, evidence_hash))"
),
DECISIONS: (
f"CREATE TABLE {DECISIONS}(id TEXT PRIMARY KEY CHECK(length(id)=36), "
f"item_id TEXT NOT NULL REFERENCES {ITEMS}(id) ON DELETE RESTRICT, "
"decision_revision INTEGER NOT NULL, created_at TEXT NOT NULL, "
"outcome TEXT NOT NULL CHECK(outcome IN ('selected','unresolved')), "
"candidate_hash TEXT CHECK(candidate_hash IS NULL OR "
"(length(candidate_hash)=64 AND candidate_hash NOT GLOB '*[^0-9a-f]*')), "
"evidence TEXT NOT NULL "
f"CHECK(length(CAST(evidence AS BLOB)) <= {MAX_EVIDENCE_BYTES}), "
"pair_id INTEGER REFERENCES prompt_pairs(id) ON DELETE SET NULL, "
"image_id INTEGER REFERENCES generated_images(id) ON DELETE SET NULL, "
"UNIQUE(item_id, decision_revision), "
"CHECK((outcome='selected') = (candidate_hash IS NOT NULL)))"
),
}
INDEXES = {
"idx_media_review_items_queue": (
f"CREATE INDEX idx_media_review_items_queue ON {ITEMS}"
"(state, created_at, id)"
),
"idx_media_review_decisions_pair": (
f"CREATE INDEX idx_media_review_decisions_pair ON {DECISIONS}(pair_id)"
),
"idx_media_review_decisions_image": (
f"CREATE INDEX idx_media_review_decisions_image ON {DECISIONS}(image_id)"
),
}
AUTOINDEXES = {ITEMS: 2, DECISIONS: 2}
LEGACY_TABLES = (
"prompts",
"generated_images",
"tags",
"prompt_tags",
"prompt_pairs",
"prompt_pair_tags",
"image_favorites",
) + ANNOTATION_TABLES
FILE_ID = "0f8e4b2a-1c3d-4e5f-8a9b-0c1d2e3f4a5b"
BYTE_ID = "6a1b2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d"
ITEM_ID = "1d2e3f4a-5b6c-4d7e-8f9a-0b1c2d3e4f5a"
DECISION_ID = "2e3f4a5b-6c7d-4e8f-9a0b-1c2d3e4f5a6b"
OTHER_ID = "7b2c3d4e-5f6a-4b7c-9d0e-1f2a3b4c5d6e"
DIGEST = hashlib.sha256(b"img").hexdigest()
CANDIDATE = hashlib.sha256(b"candidate").hexdigest()
NOW = "2026-10-22T00:00:00.000000+00:00"
EVIDENCE = '{"candidates":[],"sha256":"' + DIGEST + '"}'
EVIDENCE_HASH = hashlib.sha256(EVIDENCE.encode("ascii")).hexdigest()
def _compact(sql):
return re.sub(r"\s+", "", sql or "").lower()
def expected_catalog():
objects = {("table", name, _compact(sql)) for name, sql in DDL.items()}
objects |= {("index", name, _compact(sql)) for name, sql in INDEXES.items()}
objects |= {
("index", f"sqlite_autoindex_{table}_{n}", "")
for table, count in AUTOINDEXES.items()
for n in range(1, count + 1)
}
return objects
def catalog(conn):
names = tuple(TABLES) + tuple(INDEXES)
marks = ",".join("?" * len(names))
table_marks = ",".join("?" * len(TABLES))
return {
(kind, name, _compact(sql))
for kind, name, sql in conn.execute(
"SELECT type,name,sql FROM sqlite_master "
f"WHERE name IN ({marks}) OR tbl_name IN ({table_marks})", # nosec B608
names + tuple(TABLES),
)
}
def rows(conn, sql, *params):
"""Plain tuples whatever row factory *conn* uses."""
return [tuple(row) for row in conn.execute(sql, params)]
def metadata(conn):
return rows(conn, f"SELECT schema_version,review_index_revision FROM {METADATA}")
def table_rows(conn, tables):
found = {}
for table in tables:
present = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)
).fetchone()
if present:
found[table] = rows(conn, f"SELECT * FROM {table}") # nosec B608
return found
def insert_byte(conn, file_id=FILE_ID, byte_id=BYTE_ID, locator="/out/img.png"):
conn.execute(
"INSERT INTO media_annotation_files VALUES(?,?,?)", (file_id, locator, NOW)
)
conn.execute(
"INSERT INTO media_annotation_bytes VALUES(?,?,?,?,?,?,?,0)",
(byte_id, file_id, DIGEST, 3, 1_700_000_000_000_000_000, NOW, NOW),
)
def insert_item(conn, *, item_id=ITEM_ID, evidence=EVIDENCE, candidates=2, **extra):
values = {
"id": item_id,
"byte_id": BYTE_ID,
"spelled_path": "/out/img.png",
"resolver_version": 1,
"evidence_hash": hashlib.sha256(evidence.encode("utf-8")).hexdigest(),
"evidence": evidence,
"candidate_count": candidates,
"limited": 0,
"state": "pending",
"revision": 0,
"created_at": NOW,
"updated_at": NOW,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {ITEMS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_decision(conn, *, decision_id=DECISION_ID, **extra):
values = {
"id": decision_id,
"item_id": ITEM_ID,
"decision_revision": 1,
"created_at": NOW,
"outcome": "selected",
"candidate_hash": CANDIDATE,
"evidence": EVIDENCE,
"pair_id": None,
"image_id": None,
}
values.update(extra)
columns = ",".join(values)
marks = ",".join("?" * len(values))
conn.execute(
f"INSERT INTO {DECISIONS}({columns}) VALUES({marks})", # nosec B608
tuple(values.values()),
)
def insert_group_rows(conn, **item):
insert_byte(conn)
insert_item(conn, **item)
insert_decision(conn)
def populate_legacy(conn):
conn.execute(
"INSERT INTO prompts(id,text,hash,category,notes,run_count) VALUES"
"(7,'positive','p','keep','notes',12)"
)
conn.execute("INSERT INTO prompts(id,text,hash) VALUES(9,'negative','n')")
conn.execute(
"INSERT INTO generated_images(id,prompt_id,image_path,filename,file_path) "
"VALUES(4,7,'/tmp/image.png','image.png','/tmp/image.png')"
)
conn.execute("INSERT INTO tags(id,name) VALUES(3,'keep-tag')")
conn.execute("INSERT INTO prompt_tags VALUES(7,3)")
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(6,7,9)"
)
conn.execute("INSERT INTO prompt_pair_tags VALUES(6,3)")
conn.execute("INSERT INTO image_favorites VALUES('root','a.png',?)", (NOW,))
def full_capacity_evidence():
"""Sixteen candidates of control, lone-surrogate, astral and combining
text, exactly 8 MiB."""
text = '\x00\x07\x1f
\ud800 é é "q" \\ \U0001f9ea'
candidates = [
{
"hash": hashlib.sha256(str(n).encode()).hexdigest(),
"negative": {"text": text * 3},
"positive": {"text": text * 5},
"sampler_id": str(n),
"state": "resolved",
}
for n in range(16)
]
value = {"candidates": candidates, "pad": ""}
base = len(canonical_evidence(value).encode("utf-8"))
value["pad"] = "x" * (MAX_EVIDENCE_BYTES - base)
return value
class ReviewSchemaTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.addCleanup(assert_closed, self, self.temp.name)
self.addCleanup(PromptModel.reset_schema_cache)
self.models = []
self.addCleanup(self.close_models)
self.path = str(Path(self.temp.name) / "live.db")
def close_models(self):
for model in self.models:
model.close_all()
def model(self, path=None):
model = PromptModel(path or self.path)
self.models.append(model)
return model
def require_module(self):
self.assertIsNotNone(
review_schema_error, "database.review_schema is not implemented"
)
def drop_tables(self, path, tables=TABLES):
"""Remove tables from a closed database file (an old backup)."""
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
for table in reversed(tables):
raw.execute(f"DROP TABLE {table}")
def test_fresh_install_and_reopen_is_idempotent(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
insert_group_rows(conn)
snapshot = table_rows(conn, TABLES)
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(table_rows(conn, TABLES), snapshot)
# Running the migration again on a current database changes nothing.
migrate_review_state(conn)
self.assertEqual(table_rows(conn, TABLES), snapshot)
model.close_all()
# Items and decisions keep the byte record and the item they cite.
PromptModel.reset_schema_cache()
model = self.model()
with model.get_connection() as conn:
with self.assertRaises(sqlite3.IntegrityError):
conn.execute("DELETE FROM media_annotation_files")
with self.assertRaises(sqlite3.IntegrityError):
conn.execute(f"DELETE FROM {ITEMS}")
self.assertEqual(table_rows(conn, TABLES), snapshot)
def test_startup_validates_stored_evidence_once(self):
"""A reopen streams the evidence once and the annotation group twice
(its own install), never again for the review group (I3)."""
self.require_module()
import database.annotation_schema as annotation_module
import database.review_schema as review_module
model = self.model()
with model.get_connection() as conn:
insert_group_rows(conn)
model.close_all()
for _ in range(2):
PromptModel.reset_schema_cache()
with (
patch.object(
review_module,
"_evidence_error",
wraps=review_module._evidence_error,
) as evidence,
patch.object(
review_module,
"annotation_schema_error",
wraps=review_module.annotation_schema_error,
) as nested,
patch.object(
annotation_module,
"_value_error",
wraps=annotation_module._value_error,
) as annotation_rows,
):
model = self.model()
model.get_connection()
self.assertEqual(evidence.call_count, 1)
self.assertEqual(nested.call_count, 0)
self.assertEqual(annotation_rows.call_count, 2)
model.close_all()
def test_old_database_without_group_migrates_without_touching_legacy_rows(self):
self.require_module()
for label, dropped in (
("annotations only", TABLES),
("predates annotations", ANNOTATION_TABLES + TABLES),
):
with self.subTest(label):
path = str(Path(self.temp.name) / f"{label.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if dropped == TABLES:
insert_byte(conn)
model.close_all()
self.drop_tables(path, dropped)
with closing(sqlite3.connect(path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(
review_schema_error(raw, allow_missing=False),
"Missing review queue schema",
)
legacy_rows_before = table_rows(raw, LEGACY_TABLES)
self.assertTrue(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[
0
]
)
PromptModel.reset_schema_cache()
model = self.model(path)
with model.get_connection() as conn:
self.assertEqual(catalog(conn), expected_catalog())
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(metadata(conn), [(1, 0)])
counts = [
conn.execute(f"SELECT COUNT(*) FROM {t}").fetchone()[0]
for t in (ITEMS, DECISIONS)
]
legacy_rows_after = {
table: value
for table, value in table_rows(conn, LEGACY_TABLES).items()
if table in legacy_rows_before
}
self.assertEqual(counts, [0, 0])
self.assertEqual(legacy_rows_after, legacy_rows_before)
self.assertEqual(
PromptModel.verify_database_file(path, require_library_state=True),
(True, "ok"),
)
model.close_all()
def test_annotations_only_installation_is_valid(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
model.close_all()
self.drop_tables(self.path)
with closing(sqlite3.connect(self.path)) as raw:
self.assertIsNone(review_schema_error(raw))
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
def test_review_group_without_annotation_group_is_invalid(self):
self.require_module()
model = self.model()
model.close_all()
self.drop_tables(self.path, ANNOTATION_TABLES)
with closing(sqlite3.connect(self.path)) as raw:
for allow_missing in (True, False):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("image annotation", error.lower())
self.assertFalse(PromptModel.verify_database_file(self.path)[0])
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model()
# An invalid annotation group invalidates the review group beside it.
path = self.tampered(
"bad annotations",
("UPDATE media_annotation_bytes SET sha256=upper(sha256)",),
checks_off=True,
)
with closing(sqlite3.connect(path)) as raw:
self.assertIn("image annotation", review_schema_error(raw).lower())
def tampered(self, name, statements, *, checks_off=False, rows_in=True):
"""A closed current database whose group was altered after the fact."""
path = str(Path(self.temp.name) / f"{name.replace(' ', '-')}.db")
model = self.model(path)
with model.get_connection() as conn:
populate_legacy(conn)
if rows_in:
insert_group_rows(conn)
model.close_all()
with closing(sqlite3.connect(path)) as raw, raw:
raw.execute("PRAGMA foreign_keys=OFF")
if checks_off:
raw.execute("PRAGMA ignore_check_constraints=ON")
for statement in statements:
if isinstance(statement, tuple):
raw.execute(*statement)
else:
raw.execute(statement)
return path
def assert_rejected_unrepaired(self, path):
with closing(sqlite3.connect(path)) as raw:
schema_before = raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall()
rows_before = table_rows(raw, TABLES)
for allow_missing in (False, True):
error = review_schema_error(raw, allow_missing=allow_missing)
self.assertIsNotNone(error)
self.assertIn("review", error.lower())
self.assertFalse(PromptModel.verify_database_file(path)[0])
self.assertFalse(
PromptModel.verify_database_file(path, require_library_state=True)[0]
)
PromptModel.reset_schema_cache()
with self.assertRaises(sqlite3.DatabaseError):
self.model(path)
PromptModel.reset_schema_cache()
with closing(sqlite3.connect(path)) as raw:
self.assertEqual(
raw.execute(
"SELECT type,name,tbl_name,sql FROM sqlite_master ORDER BY name"
).fetchall(),
schema_before,
)
self.assertEqual(table_rows(raw, TABLES), rows_before)
def test_partial_or_malformed_group_is_rejected_without_repair(self):
self.require_module()
cases = {
"missing metadata": (f"DROP TABLE {METADATA}",),
"missing items": (f"DROP TABLE {DECISIONS}", f"DROP TABLE {ITEMS}"),
"missing decisions": (f"DROP TABLE {DECISIONS}",),
"reserved view": (
f"DROP TABLE {DECISIONS}",
f"CREATE VIEW {DECISIONS} AS SELECT 'x' AS id",
),
"extra trigger": (
f"CREATE TRIGGER media_review_trigger AFTER INSERT ON {ITEMS} "
"BEGIN SELECT 1; END",
),
"extra index": (
f"CREATE INDEX idx_media_review_extra ON {ITEMS}(updated_at)",
),
"extra column": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("evidence TEXT", "note TEXT, evidence TEXT"),
),
"wrong primary key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace("id TEXT PRIMARY KEY", "id TEXT", 1),
),
"wrong foreign key": (
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(
"generated_images(id) ON DELETE SET NULL",
"generated_images(id) ON DELETE CASCADE",
),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"wrong check": (
f"DROP TABLE {METADATA}",
DDL[METADATA].replace("schema_version=1", "schema_version=2"),
f"INSERT INTO {METADATA} VALUES(1,2,0)",
),
"missing unique": (
"DROP INDEX idx_media_review_decisions_pair",
"DROP INDEX idx_media_review_decisions_image",
f"DROP TABLE {DECISIONS}",
DDL[DECISIONS].replace(" UNIQUE(item_id, decision_revision),", ""),
INDEXES["idx_media_review_decisions_pair"],
INDEXES["idx_media_review_decisions_image"],
),
"missing queue index": ("DROP INDEX idx_media_review_items_queue",),
"wrong index columns": (
"DROP INDEX idx_media_review_items_queue",
INDEXES["idx_media_review_items_queue"].replace(", id", ""),
),
"orphan item": ((f"UPDATE {ITEMS} SET byte_id=?", (OTHER_ID,)),),
"orphan decision": ((f"UPDATE {DECISIONS} SET item_id=?", (OTHER_ID,)),),
"dangling pair": (f"UPDATE {DECISIONS} SET pair_id=404",),
"bad item uuid": (
(f"UPDATE {ITEMS} SET id=?", ("x" * 36,)),
(f"UPDATE {DECISIONS} SET item_id=?", ("x" * 36,)),
),
"uppercase decision uuid": (f"UPDATE {DECISIONS} SET id=upper(id)",),
"null decision id": (f"UPDATE {DECISIONS} SET id=NULL",),
"blob ids": (f"UPDATE {DECISIONS} SET id=CAST(id AS BLOB)",),
"no metadata row": (f"DELETE FROM {METADATA}",),
"long created_at": ((f"UPDATE {ITEMS} SET created_at=?", ("x" * 65,)),),
"blob updated_at": (f"UPDATE {ITEMS} SET updated_at=X'00'",),
"text resolver version": (f"UPDATE {ITEMS} SET resolver_version='one'",),
"blob spelled path": (
f"UPDATE {ITEMS} SET spelled_path=CAST(spelled_path AS BLOB)",
),
"evidence hash mismatch": (
(f"UPDATE {ITEMS} SET evidence_hash=?", (CANDIDATE,)),
),
}
for label, statements in cases.items():
with self.subTest(label):
self.assert_rejected_unrepaired(self.tampered(label, statements))
unchecked = {
"two metadata rows": (f"INSERT INTO {METADATA} VALUES(2,1,0)",),
"wrong schema version": (f"UPDATE {METADATA} SET schema_version=2",),
"negative index revision": (
f"UPDATE {METADATA} SET review_index_revision=-1",
),
"uppercase evidence hash": (
f"UPDATE {ITEMS} SET evidence_hash=upper(evidence_hash)",
),
"bad state": (f"UPDATE {ITEMS} SET state='done'",),
"limited two": (f"UPDATE {ITEMS} SET limited=2",),
"empty spelled path": (f"UPDATE {ITEMS} SET spelled_path=''",),
"negative revision": (f"UPDATE {ITEMS} SET revision=-1",),
"negative candidates": (f"UPDATE {ITEMS} SET candidate_count=-1",),
"bad outcome": (f"UPDATE {DECISIONS} SET outcome='maybe'",),
"short candidate hash": (
f"UPDATE {DECISIONS} SET candidate_hash=substr(candidate_hash,1,63)",
),
"negative decision revision": (
f"UPDATE {DECISIONS} SET decision_revision=-1",
),
}
for label, statements in unchecked.items():
with self.subTest(label):
path = self.tampered(label, statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_selected_decision_requires_candidate_hash(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
insert_item(conn)
for outcome, candidate in (("selected", None), ("unresolved", CANDIDATE)):
with self.subTest(outcome):
with self.assertRaises(sqlite3.IntegrityError):
insert_decision(conn, outcome=outcome, candidate_hash=candidate)
insert_decision(conn, outcome="unresolved", candidate_hash=None)
insert_decision(conn, decision_id=OTHER_ID, decision_revision=2)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
self.assertEqual(
rows(conn, f"SELECT outcome FROM {DECISIONS} ORDER BY 1"),
[("selected",), ("unresolved",)],
)
with self.assertRaises(sqlite3.IntegrityError): # one per revision
insert_decision(conn, decision_id=ITEM_ID, decision_revision=2)
model.close_all()
path = self.tampered(
"selected without hash",
(f"UPDATE {DECISIONS} SET candidate_hash=NULL",),
checks_off=True,
)
self.assert_rejected_unrepaired(path)
def test_oversize_or_noncanonical_evidence_rejected(self):
self.require_module()
self.assertEqual(
canonical_evidence({"b": [1, "é"], "a": None}),
'{"a":null,"b":[1,"\\u00e9"]}',
)
self.assertEqual(
evidence_digest(EVIDENCE), hashlib.sha256(EVIDENCE.encode()).hexdigest()
)
# The store's non-finite policy (utils.json_strict): written as null.
self.assertEqual(
canonical_evidence({"x": float("nan"), "y": [float("-inf")]}),
'{"x":null,"y":[null]}',
)
for bad in (
[],
"text",
{"x": object()},
{"x": object(), "y": float("nan")},
{1: 1, "a": 2},
):
with self.subTest(value=repr(bad)), self.assertRaises(ValueError):
canonical_evidence(bad)
with self.assertRaises(ValueError):
canonical_evidence({"pad": "x" * MAX_EVIDENCE_BYTES})
oversize = '{"pad":"' + "x" * (MAX_EVIDENCE_BYTES - 9) + '"}'
self.assertEqual(len(oversize), MAX_EVIDENCE_BYTES + 1)
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, evidence=oversize)
model.close_all()
noncanonical = {
"spaces": '{"candidates": [], "sha256": "x"}',
"unsorted": '{"sha256":"x","candidates":[]}',
"raw unicode": '{"text":"é"}',
"duplicate key": '{"a":1,"a":2}',
"not an object": "[]",
"not json": "{",
"nan": '{"a":NaN}',
"empty": "",
}
for label, evidence in noncanonical.items():
for table in (ITEMS, DECISIONS):
with self.subTest(label, table=table):
digest = hashlib.sha256(evidence.encode("utf-8")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (evidence,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
if table == DECISIONS:
statements.pop()
self.assert_rejected_unrepaired(
self.tampered(f"{label} {table}", statements)
)
for table in (ITEMS, DECISIONS):
with self.subTest("oversize", table=table):
digest = hashlib.sha256(oversize.encode("ascii")).hexdigest()
statements = [
(f"UPDATE {table} SET evidence=?", (oversize,)),
(f"UPDATE {ITEMS} SET evidence_hash=?", (digest,)),
]
path = self.tampered(f"oversize {table}", statements, checks_off=True)
self.assert_rejected_unrepaired(path)
def test_seventeen_candidates_rejected(self):
self.require_module()
model = self.model()
with model.get_connection() as conn:
insert_byte(conn)
with self.assertRaises(sqlite3.IntegrityError):
insert_item(conn, candidates=17)
insert_item(conn, candidates=16)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
model.close_all()
path = self.tampered(
"seventeen", (f"UPDATE {ITEMS} SET candidate_count=17",), checks_off=True
)
self.assert_rejected_unrepaired(path)
def test_prompt_or_image_deletion_nulls_navigation_and_keeps_decision(self):
self.require_module()
db = PromptDatabase(self.path)
self.models.append(db.model)
with db.model.get_connection() as conn:
populate_legacy(conn)
conn.execute(
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
"VALUES(8,9,NULL)"
)
insert_byte(conn)
insert_item(conn)
insert_decision(conn, pair_id=8, image_id=4)
conn.commit()
self.assertTrue(db.delete_prompt_pair(8))
with db.model.get_connection() as conn:
self.assertEqual(
rows(conn, f"SELECT pair_id, image_id FROM {DECISIONS}"), [(None, 4)]
)
with db.model.get_connection() as conn:
conn.execute("DELETE FROM prompt_pairs WHERE id=6")
conn.commit()
self.assertTrue(db.delete_prompt(7))
with db.model.get_connection() as conn:
self.assertEqual(
rows(
conn,
f"SELECT id, item_id, outcome, candidate_hash, pair_id, image_id "
f"FROM {DECISIONS}",
),
[(DECISION_ID, ITEM_ID, "selected", CANDIDATE, None, None)],
)
self.assertEqual(rows(conn, f"SELECT id FROM {ITEMS}"), [(ITEM_ID,)])
self.assertIsNone(review_schema_error(conn, allow_missing=False))
# The SET NULL actions look decisions up by index, not by a scan.
for column in ("pair_id", "image_id"):
plan = " ".join(
str(row[3])
for row in conn.execute(
f"EXPLAIN QUERY PLAN UPDATE {DECISIONS} " # nosec B608
f"SET {column}=NULL WHERE {column}=?",
(1,),
)
)
self.assertIn(f"idx_media_review_decisions_{column[:-3]}", plan)
def test_full_capacity_unicode_control_pair_round_trips_within_8_mib(self):
self.require_module()
value = full_capacity_evidence()
evidence = canonical_evidence(value)
self.assertEqual(len(evidence.encode("utf-8")), MAX_EVIDENCE_BYTES)
self.assertTrue(evidence.isascii())
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_byte(conn)
insert_item(conn, evidence=evidence, candidates=16)
insert_decision(conn, evidence=evidence)
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
for restore_from in (source_path, backup_file):
with self.subTest(Path(restore_from).name):
live.restore_from_file(restore_from)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertIsNone(review_schema_error(conn, allow_missing=False))
stored = conn.execute(
f"SELECT i.evidence, d.evidence FROM {ITEMS} i "
f"JOIN {DECISIONS} d ON d.item_id = i.id"
).fetchone()
self.assertEqual(json.loads(stored[0]), value)
self.assertEqual(json.loads(stored[1]), value)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_install_failure_rolls_back_and_retries(self):
self.require_module()
with patch(
"database.review_schema.review_schema_error",
side_effect=[None, "injected final validation failure"],
):
with self.assertRaisesRegex(sqlite3.DatabaseError, "injected"):
self.model()
with closing(sqlite3.connect(self.path)) as raw:
self.assertEqual(catalog(raw), set())
self.assertIsNotNone(
raw.execute(
"SELECT 1 FROM sqlite_master WHERE name='media_annotation_bytes'"
).fetchone()
)
PromptModel.reset_schema_cache()
model = self.model()
self.assertEqual(
PromptModel.verify_database_file(model.db_path, require_library_state=True),
(True, "ok"),
)
with model.get_connection() as conn:
self.assertEqual(metadata(conn), [(1, 0)])
def test_wal_backup_restore_round_trips_group(self):
self.require_module()
source_path = str(Path(self.temp.name) / "source.db")
source = self.model(source_path)
with source.get_connection() as conn:
conn.execute("PRAGMA wal_autocheckpoint=0")
insert_group_rows(conn)
conn.execute(f"UPDATE {METADATA} SET review_index_revision=3")
conn.commit()
source_rows = table_rows(conn, TABLES)
self.assertGreater(Path(source_path + "-wal").stat().st_size, 0)
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
source_before = Path(source_path).read_bytes()
backup_file = str(Path(self.temp.name) / "backup.db")
self.assertTrue(source.backup_database(backup_file))
with closing(sqlite3.connect(backup_file)) as raw:
self.assertEqual(table_rows(raw, TABLES), source_rows)
self.assertIsNone(review_schema_error(raw, allow_missing=False))
safety = live.restore_from_file(source_path)
self.assertEqual(Path(source_path).read_bytes(), source_before)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
self.assertEqual(metadata(conn), [(1, 3)])
with closing(sqlite3.connect(safety)) as raw:
self.assertEqual(rows(raw, f"SELECT id FROM {ITEMS}"), [(OTHER_ID,)])
PromptModel.reset_schema_cache()
live.restore_from_file(backup_file)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), source_rows)
PromptModel.close_all_instances()
assert_closed(self, self.temp.name)
def test_staged_failure_preserves_source_live_safety_and_handles(self):
self.require_module()
live = self.model()
with live.get_connection() as conn:
insert_byte(conn)
insert_item(conn, item_id=OTHER_ID)
live_rows = table_rows(conn, TABLES)
bad = self.tampered(
"restore bad", (f"UPDATE {ITEMS} SET state='done'",), checks_off=True
)
old = str(Path(self.temp.name) / "old.db")
self.model(old).close_all()
self.drop_tables(old)
for source, patched in ((bad, False), (old, True)):
with self.subTest(source=Path(source).name):
before = Path(source).read_bytes()
safety = sorted(Path(self.temp.name).glob("live.db.backup_*"))
if patched:
context = patch(
"database.models.migrate_review_state",
side_effect=sqlite3.OperationalError("staging failed"),
)
expected = sqlite3.OperationalError
else:
context = patch.object(PromptModel, "backup_database")
expected = ValueError
with context as hook, self.assertRaises(expected):
live.restore_from_file(source)
if not patched:
hook.assert_not_called()
self.assertEqual(Path(source).read_bytes(), before)
self.assertEqual(
sorted(Path(self.temp.name).glob("live.db.backup_*")), safety
)
assert_closed(self, source)
with live.get_connection() as conn:
self.assertEqual(table_rows(conn, TABLES), live_rows)
self.assertFalse(
PromptModel.verify_database_file(old, require_library_state=True)[0]
)
self.assertTrue(PromptModel.verify_database_file(old)[0])
def test_verify_requires_group_only_with_require_library_state(self):
self.require_module()
self.model().close_all()
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(True, "ok"),
)
self.drop_tables(self.path)
self.assertEqual(PromptModel.verify_database_file(self.path), (True, "ok"))
self.assertEqual(
PromptModel.verify_database_file(self.path, require_library_state=True),
(False, "Missing review queue schema"),
)
if __name__ == "__main__":
unittest.main()
+5 -237
View File
@@ -38,11 +38,6 @@ def _graph(text):
}
def _scope_part(scope):
"""The per-source scope keys this module pins (other features add counters)."""
return {key: scope[key] for key in ("files", "roots", "output_folders")}
class ScanScopeTestCase(AdminAPITestCase):
"""Helpers: link files the way the scan and the monitor do, spy on reads."""
@@ -265,7 +260,7 @@ class TestScanScope(ScanScopeTestCase):
async def test_empty_read_body_asks_for_the_default_scope(self):
"""A chunked POST with no bytes can read a body: still the default."""
from py.api.job_routes import _scan_request
from py.api.job_routes import _scan_scope
class Request:
can_read_body = True
@@ -276,21 +271,10 @@ class TestScanScope(ScanScopeTestCase):
async def text(self):
return self._text
default = ({"files": "new", "roots": None}, None)
self.assertEqual(await _scan_request(Request("")), default)
self.assertEqual(await _scan_request(Request(" \n")), default)
self.assertEqual(
await _scan_request(
Request(
'{"scope": "all", "roots": [2, 0, 2],'
' "paths": ["output/c", "output/a", "output/c"]}'
)
),
({"files": "all", "roots": {0: "output/a", 2: "output/c"}}, None),
)
request, error = await _scan_request(Request("{"))
self.assertIsNone(request)
self.assertIsInstance(error, str)
self.assertEqual(await _scan_scope(Request("")), "new")
self.assertEqual(await _scan_scope(Request(" \n")), "new")
self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all")
self.assertIsNone(await _scan_scope(Request("{")))
async def test_invalid_scope_is_refused(self):
for body in ({"scope": "everything"}, {"scope": 1}, ["new"]):
@@ -308,222 +292,6 @@ class TestScanScope(ScanScopeTestCase):
self.assertIsNone(self.api._scan_job)
class TestPerSourceScope(ScanScopeTestCase):
"""``roots``: scan only the selected effective sources (#249 S4 Task 10)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
_write_png(self.a / "in_a.png", prompt_text="a prompt in a")
_write_png(self.b / "in_b.png", prompt_text="a prompt in b")
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
async def _refused(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 400, data)
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_scoped_scan_only_walks_selected_root(self):
walked = []
original = self.api._collect_output_media_sync
def spy(output_dirs):
walked.append([os.path.basename(str(d)) for d in output_dirs])
return original(output_dirs)
self.api._collect_output_media_sync = spy
reads = self._spy_reads()
job = await self._scan({"scope": "all", "roots": [1], "paths": ["output/b"]})
self.assertEqual(walked, [["b"]])
self.assertEqual(reads, ["in_b.png"])
self.assertEqual(
_scope_part(job["scope"]),
{"files": "all", "roots": [1], "output_folders": 1},
)
self.assertEqual(
job["outcomes"]["added"],
[
{
"path": "in_b.png",
"root": "output/b",
"root_index": 1,
"detail": job["outcomes"]["added"][0]["detail"],
}
],
)
self.assertEqual(job["counters"]["added"], 1)
self.assertEqual(self.api._scan_job.last_event["directories"], ["output/b"])
async def test_duplicate_roots_are_one_selection(self):
job = await self._scan(
{"roots": [1, 0, 1], "paths": ["output/b", "output/a", "output/b"]}
)
self.assertEqual(job["scope"]["roots"], [0, 1])
self.assertEqual(job["scope"]["output_folders"], 2)
self.assertEqual(job["counters"]["added"], 2)
async def test_scope_line_counts_selected_folders(self):
job = await self._scan({"roots": [0], "paths": ["output/a"]})
self.assertTrue(
job["log"][0].endswith(
"Scan started · 1 output folder · new and unresolved files"
),
job["log"][0],
)
self.assertEqual(job["scope"]["output_folders"], 1)
unscoped = await self._scan({"scope": "all"})
self.assertEqual(
_scope_part(unscoped["scope"]),
{"files": "all", "roots": None, "output_folders": 2},
)
self.assertIn("Scan started · 2 output folders · all files", unscoped["log"][0])
async def _stale(self, body):
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", json=body)
data = await resp.json()
self.assertEqual(resp.status, 409, data)
self.assertEqual(data["code"], "sources-changed")
self.assertFalse(data["success"])
self.assertNotIn(self.tmpdir, json.dumps(data))
self.assertIsNone(self.api._scan_job)
self.assertEqual(self.api._jobs.list(), [])
return data
async def test_index_past_the_sources_is_409_without_job(self):
await self._stale({"roots": [2], "paths": ["output/c"]})
await self._stale({"roots": [0, 64], "paths": ["output/a", "output/x"]})
async def test_stale_index_never_scans_another_source(self):
"""The list changed after the page read it: source 0 was removed, so
old index 1 (b) is past the end and old index 0 now names b."""
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.b)]
await self._stale({"roots": [1], "paths": ["output/b"]})
await self._stale({"roots": [0], "paths": ["output/a"]})
job = await self._scan({"roots": [0], "paths": ["output/b"]})
self.assertEqual(job["outcomes"]["added"][0]["root"], "output/b")
async def test_roots_need_their_paths(self):
for body in (
{"roots": [0]},
{"roots": [0], "paths": None},
{"roots": [0], "paths": "output/a"},
{"roots": [0], "paths": []},
{"roots": [0, 1], "paths": ["output/a"]},
{"roots": [0], "paths": [0]},
{"roots": [0, 0], "paths": ["output/a", "output/b"]},
):
with self.subTest(body=body):
await self._refused(body)
async def test_bool_and_float_roots_rejected(self):
for roots in (
[True],
[False],
[0.0],
[1.5],
["0"],
[None],
[-1],
[0, None],
[],
None,
0,
"0",
{"0": 0},
list(range(65)),
):
with self.subTest(roots=roots):
paths = ["output/a"] * len(roots) if isinstance(roots, list) else []
await self._refused({"roots": roots, "paths": paths})
async def test_legacy_scan_unscoped_unchanged(self):
from py.api.admin import SCAN_COUNT_KEYS
resp = await self.client.request("POST", "/prompt_manager/scan", data="{}")
payloads = _sse_payloads(await resp.text())
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a", "output/b"])
self.assertEqual(final["added"], 2)
self.assertEqual(
_scope_part(self.api._scan_job.scope),
{"files": "all", "roots": None, "output_folders": 2},
)
async def test_legacy_scan_attaches_to_a_scoped_job_as_is(self):
import threading
from py.api.admin import SCAN_COUNT_KEYS
gate = threading.Event()
original = self.api._collect_output_media_sync
def gated(output_dirs):
gate.wait(5)
return original(output_dirs)
self.api._collect_output_media_sync = gated
# The legacy route has decided to attach once it streams the job.
attached = asyncio.Event()
stream = self.api._stream_scan_job
async def attaching(request, job):
attached.set()
return await stream(request, job)
self.api._stream_scan_job = attaching
resp = await self.client.request(
"POST",
"/prompt_manager/jobs/scan",
json={"roots": [0], "paths": ["output/a"]},
)
self.assertEqual(resp.status, 202)
started = (await resp.json())["job"]
legacy = asyncio.ensure_future(
self.client.request("POST", "/prompt_manager/scan", data="{}")
)
try:
await asyncio.wait_for(attached.wait(), 10)
finally:
gate.set()
payloads = _sse_payloads(await (await legacy).text())
await asyncio.wait_for(self.api._scan_job.task, 10)
self.assertEqual(self.api._scan_job.id, started["id"])
self.assertEqual(len(self.api._jobs.list()), 1)
final = payloads[-1]
self.assertEqual(set(final), set(SCAN_COUNT_KEYS) | {"type", "directories"})
self.assertEqual(final["directories"], ["output/a"])
self.assertEqual(final["added"], 1)
async def test_scoped_scan_uses_the_sources_it_was_started_with(self):
"""Sources saved after the start never change what the job walks."""
resolved = []
original = self.api._get_all_output_dirs
def counting():
resolved.append(1)
return original()
self.api._get_all_output_dirs = counting
job = await self._scan({"roots": [1], "paths": ["output/b"]})
self.assertEqual(len(resolved), 1) # the request's own resolution only
self.assertEqual(job["outcomes"]["added"][0]["root_index"], 1)
class TestKnownResolvedPaths(ScanScopeTestCase):
"""``PromptDatabase.get_known_resolved_paths``: one streamed query."""
-959
View File
@@ -1,959 +0,0 @@
"""System › Sources & scans backend (#249 S4 Task 9).
``GET /prompt_manager/settings`` reports the configured, effective and
watched sources and the monitoring setting; ``POST`` saves
``monitoring_enabled`` (a restart applies it) and refuses a source change
while a record-changing job runs; ``POST /prompt_manager/settings/validate-source``
checks one path and writes nothing. No response carries an absolute path.
"""
import asyncio
import functools
import json
import ntpath
import os
import sys
import types
import unittest
from contextlib import contextmanager
from unittest.mock import patch
try:
from tests.test_api_admin import AdminAPITestCase
except ImportError: # discovered with tests/ as the top-level directory
from test_api_admin import AdminAPITestCase
from py import config as config_module # noqa: E402
from py.api import settings_routes, sources # noqa: E402
from py.config import GalleryConfig # noqa: E402
VALIDATE = "/prompt_manager/settings/validate-source"
SETTINGS = "/prompt_manager/settings"
@contextmanager
def no_filesystem_access_for(marker, test):
"""Fail ``test`` if anything resolves, stats or opens a path holding ``marker``.
Trusted directories (ComfyUI's, the anchors) may still be resolved.
"""
real = {
"realpath": os.path.realpath,
"stat": os.stat,
"lstat": os.lstat,
}
def guard(name):
def call(path, *args, **kwargs):
if isinstance(path, (str, bytes)) and marker in os.fsdecode(path):
test.fail(f"{name}({path!r}) touched an outside path")
return real[name](path, *args, **kwargs)
return call
with (
patch("os.path.realpath", guard("realpath")),
patch("os.stat", guard("stat")),
patch("os.lstat", guard("lstat")),
):
yield
class FakeMonitor:
"""The image monitor as get_settings sees it: status and scheduled roots."""
def __init__(self, scheduled, alive=True):
self.monitored_directories = [str(path) for path in scheduled]
self.alive = alive
def get_status(self):
return {
"running": True,
"monitored_directories": self.monitored_directories,
"handler_active": True,
"observer_alive": self.alive,
}
class SourcesTestCase(AdminAPITestCase):
async def asyncSetUp(self):
await super().asyncSetUp()
self.a = self.output_dir / "a"
self.b = self.output_dir / "b"
self.a.mkdir()
self.b.mkdir()
enabled = GalleryConfig.MONITORING_ENABLED
self.addCleanup(setattr, GalleryConfig, "MONITORING_ENABLED", enabled)
GalleryConfig.MONITORING_ENABLED = True
self._install_monitor(None)
def _install_monitor(self, monitor):
import utils.image_monitor as im_mod
orig = im_mod._monitor_instance
im_mod._monitor_instance = monitor
self.addCleanup(setattr, im_mod, "_monitor_instance", orig)
async def _get(self):
resp = await self.client.request("GET", "/prompt_manager/settings")
body = await resp.text()
self.assertEqual(resp.status, 200, body)
self.assertNotIn(self.tmpdir, body)
return json.loads(body)
async def _post(self, url, payload):
resp = await self.client.request("POST", url, json=payload)
body = await resp.text()
self.assertNotIn(self.tmpdir, body)
return resp.status, json.loads(body)
def _config_bytes(self):
if not os.path.exists(self.config_path):
return None
with open(self.config_path, "rb") as f:
return f.read()
def _states(self, data):
return {row["path"]: row["watch"] for row in data["sources"]["effective"]}
class TestSourceStatus(SourcesTestCase):
async def test_sources_list_configured_effective_and_auto_detected(self):
missing = self.output_dir / "gone"
missing.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [
self._root(self.a),
self._root(missing),
]
missing.rmdir() # an unmounted drive: configured, no longer a directory
data = await self._get()
self.assertEqual(
data["sources"]["configured"],
[
{
"label": "a",
"path": "output/a",
"valid": True,
"code": None,
"reason": None,
},
{
"label": "gone",
"path": "output/gone",
"valid": False,
"code": "not-found",
"reason": "Gallery root does not exist",
},
],
)
self.assertEqual(
data["sources"]["effective"],
[{"label": "a", "path": "output/a", "root_index": 0, "watch": "manual"}],
)
self.assertFalse(data["sources"]["auto_detected"])
# The legacy keys are unchanged.
self.assertEqual(
data["settings"]["gallery_root_paths"], ["output/a", "output/gone"]
)
async def test_empty_list_means_auto_detect(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["x"]}}})
# Detection (the upward search for a ComfyUI tree) finds the output.
self.api._find_comfyui_output_dir = lambda: str(self.output_dir)
status, saved = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": []}
)
self.assertEqual(status, 200, saved)
self.assertEqual(
self._read_config()["gallery"]["monitoring"]["directories"], []
)
data = await self._get()
self.assertEqual(data["sources"]["configured"], [])
self.assertTrue(data["sources"]["auto_detected"])
self.assertEqual(
data["sources"]["effective"],
[
{
"label": "ComfyUI output",
"path": "output",
"root_index": 0,
"watch": "manual",
}
],
)
async def test_watching_requires_live_observer_with_root(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(self.b)]
# No monitor in this process: nothing is watched, a restart cannot help.
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": False, "observer_alive": False, "scheduled": []},
)
self.assertEqual(
self._states(data), {"output/a": "manual", "output/b": "manual"}
)
# A dead observer watches nothing, even the roots it scheduled.
self._install_monitor(FakeMonitor([self.a], alive=False))
data = await self._get()
self.assertEqual(
data["monitor"],
{"present": True, "observer_alive": False, "scheduled": ["output/a"]},
)
self.assertEqual(
self._states(data),
{"output/a": "restart_pending", "output/b": "restart_pending"},
)
# A live observer watches its scheduled root; a root configured since
# it started waits for a restart.
self._install_monitor(FakeMonitor([self.a]))
data = await self._get()
self.assertTrue(data["monitor"]["observer_alive"])
self.assertEqual(
self._states(data), {"output/a": "watching", "output/b": "restart_pending"}
)
# Monitoring switched off: the live observer still watches until the
# restart ("stopping"), and nothing else is pending.
GalleryConfig.MONITORING_ENABLED = False
data = await self._get()
self.assertFalse(data["settings"]["monitoring_enabled"])
self.assertEqual(
self._states(data), {"output/a": "stopping", "output/b": "manual"}
)
async def test_scheduled_root_matches_through_its_spelling(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
spelled = os.path.join(str(self.output_dir), "b", "..", "a")
self._install_monitor(FakeMonitor([spelled]))
data = await self._get()
self.assertEqual(self._states(data), {"output/a": "watching"})
self.assertEqual(data["monitor"]["scheduled"], ["output/a"])
async def test_status_failure_is_500_without_paths(self):
def boom(*args, **kwargs):
raise OSError(2, "No such file", f"{self.tmpdir}/secret")
with patch.object(sources, "describe_sources", boom):
resp = await self.client.request("GET", "/prompt_manager/settings")
self.assertEqual(resp.status, 500)
self.assertNotIn(self.tmpdir, await resp.text())
class TestMonitoringSetting(SourcesTestCase):
async def test_monitoring_enabled_strict_bool_and_restart_flag(self):
self._write_config({"custom": {"keep": True}})
before = self._config_bytes()
for value in ("false", 0, 1, None, [], {"on": True}):
with self.subTest(value=value):
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": value}
)
self.assertEqual(status, 400, data)
self.assertFalse(data["success"])
self.assertEqual(self._config_bytes(), before)
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
self._install_monitor(FakeMonitor([self.output_dir]))
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": False}
)
self.assertEqual(status, 200, data)
self.assertTrue(data["persisted"])
self.assertTrue(data["restart_required"])
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
saved = self._read_config()
self.assertIs(saved["gallery"]["monitoring"]["enabled"], False)
self.assertEqual(saved["custom"], {"keep": True})
self.assertFalse(GalleryConfig.MONITORING_ENABLED)
self.assertFalse((await self._get())["settings"]["monitoring_enabled"])
# Back on while the observer still runs: nothing waits for a restart.
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertIs(self._read_config()["gallery"]["monitoring"]["enabled"], True)
async def test_enabling_without_a_live_observer_needs_a_restart(self):
GalleryConfig.MONITORING_ENABLED = False
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
self.assertTrue(GalleryConfig.MONITORING_ENABLED)
def test_config_file_enabled_must_be_a_bool(self):
GalleryConfig.MONITORING_ENABLED = True
with self.assertLogs("prompt_manager.config", level="WARNING"):
GalleryConfig.update_config({"monitoring": {"enabled": "false"}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, True)
GalleryConfig.update_config({"monitoring": {"enabled": False}})
self.assertIs(GalleryConfig.MONITORING_ENABLED, False)
class TestSourceChangeHold(SourcesTestCase):
async def test_root_save_refused_while_scan_job_runs(self):
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a)]
self._write_config({"gallery": {"monitoring": {"directories": ["kept"]}}})
before = self._config_bytes()
gate = asyncio.Event()
async def runner(job):
await gate.wait()
scan = self.api._jobs.start(
"scan", "Scan for metadata", record_changing=True, runner=runner
)
try:
status, data = await self._post(
"/prompt_manager/settings",
{"gallery_root_paths": [str(self.b)]},
)
self.assertEqual(status, 409, data)
self.assertEqual(data["code"], "busy")
self.assertEqual(
data["running"], {"id": scan.id, "title": "Scan for metadata"}
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [self._root(self.a)])
# A save without sources, or one resending the current sources
# (the legacy page always does), is not a source change.
status, data = await self._post(
"/prompt_manager/settings", {"result_timeout": 9}
)
self.assertEqual(status, 200, data)
status, data = await self._post(
"/prompt_manager/settings",
{"result_timeout": 8, "gallery_root_paths": [str(self.a)]},
)
self.assertEqual(status, 200, data)
finally:
gate.set()
await asyncio.wait_for(scan.task, 5)
async def test_root_save_runs_as_a_source_change_hold(self):
seen = []
original = settings_routes.get_config_store
def store():
seen.extend(
(job.kind, job.title, job.admission, job.cancellable)
for job in self.api._jobs.list()
if job.running
)
return original()
with patch.object(settings_routes, "get_config_store", store):
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual(status, 200, data)
self.assertIn(("source_change", "Source change", "records", False), seen)
self.assertFalse(any(job.running for job in self.api._jobs.list()))
async def test_failed_root_save_ends_the_hold_failed(self):
os.makedirs(self.config_path) # config.json is a directory: unreadable
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.b)]}
)
self.assertGreaterEqual(status, 400, data)
held = [j for j in self.api._jobs.list() if j.kind == "source_change"]
self.assertEqual([job.state for job in held], ["failed"])
async def test_mixed_case_windows_paths_dedupe(self):
"""Two spellings of one Windows folder are saved once (normcase)."""
with (
patch.object(sources, "source_problem", return_value=(None, "")),
patch.object(
GalleryConfig, "resolve_gallery_root", side_effect=lambda path: path
),
patch.object(settings_routes, "_source_identity", ntpath.normcase),
patch.object(self.api, "_apply_gallery_roots", return_value=True),
):
roots, error = self.api._parse_gallery_roots(
{
"gallery_root_paths": [
"C:\\ComfyUI\\Output\\Renders",
"c:/comfyui/output/renders",
"C:\\ComfyUI\\output\\other",
]
}
)
self.assertIsNone(error)
self.assertEqual(
roots, ["C:\\ComfyUI\\Output\\Renders", "C:\\ComfyUI\\output\\other"]
)
async def test_case_variant_of_one_folder_is_saved_once_natively(self):
variant = str(self.a)[:-1] + "A"
if not os.path.isdir(variant):
self.skipTest("case-sensitive filesystem")
status, data = await self._post(
"/prompt_manager/settings", {"gallery_root_paths": [str(self.a), variant]}
)
self.assertEqual(status, 200, data)
self.assertEqual(len(GalleryConfig.MONITORING_DIRECTORIES), 1)
class TestValidateSource(SourcesTestCase):
async def test_validate_source_writes_nothing(self):
self._write_config({"gallery": {"monitoring": {"directories": []}}})
before = self._config_bytes()
roots = list(GalleryConfig.MONITORING_DIRECTORIES)
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 200, data)
self.assertEqual(
data,
{
"success": True,
"valid": True,
"label": "a",
"path": "output/a",
"code": None,
"reason": None,
},
)
self.assertEqual(self._config_bytes(), before)
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, roots)
self.assertEqual(self.api._jobs.list(), [])
async def test_relative_path_resolves_against_comfyui(self):
status, data = await self._post(VALIDATE, {"path": "output/b"})
self.assertEqual((status, data["valid"], data["path"]), (200, True, "output/b"))
status, data = await self._post(VALIDATE, {"path": "output"})
self.assertEqual(data["label"], "ComfyUI output")
async def test_invalid_sources_answer_a_code(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
cases = {
"": "empty",
" ": "empty",
str(self.output_dir / "missing"): "not-found",
str(a_file): "not-directory",
str(self.outside_dir): "outside-allowed",
os.path.abspath(os.sep): "outside-allowed",
os.path.expanduser("~"): "outside-allowed",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
"output/../output/a": "traversal",
str(self.a) + "\x00": "invalid-form",
"\\\\?\\C:\\ComfyUI\\output": "invalid-form",
"x" * 5000: "invalid-form",
}
if os.name != "nt":
cases["C:\\ComfyUI\\output"] = "invalid-form"
for path, code in cases.items():
with self.subTest(code=code, path=path[:40]):
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(status, 200, data)
self.assertFalse(data["valid"])
self.assertEqual(data["code"], code)
self.assertIsInstance(data["reason"], str)
self.assertIsNone(data["label"])
self.assertIsNone(data["path"])
self.assertNotIn(os.path.expanduser("~"), json.dumps(data))
async def test_outside_paths_answer_alike_whether_they_exist(self):
"""No existence oracle: outside the allowed folders every path looks
the same, existing or not, file or folder."""
a_file = self.outside_dir / "file.txt"
a_file.write_bytes(b"x")
answers = []
for path in (
self.outside_dir,
self.outside_dir / "missing",
a_file,
os.path.join(self.tmpdir, "no", "such", "tree"),
os.path.abspath(os.sep),
os.path.expanduser("~"),
):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append((status, data))
self.assertEqual(answers[0][1]["code"], "outside-allowed")
self.assertEqual([answer for answer in answers], [answers[0]] * len(answers))
async def test_settings_save_refuses_outside_paths_alike(self):
"""The legacy save answers every outside path alike too: the
filesystem root, the home directory and a link to it included."""
(self.outside_dir / "x").mkdir()
link = os.path.join(self.tmpdir, "link-home")
try:
os.symlink(os.path.expanduser("~"), link, target_is_directory=True)
except (OSError, NotImplementedError):
link = None
paths = [
str(self.outside_dir / "x"),
self.tmpdir + "/nothing/x",
os.path.abspath(os.sep),
os.path.expanduser("~"),
]
if link:
paths.append(link)
answers = []
for path in paths:
resp = await self.client.request(
"POST", SETTINGS, json={"gallery_root_paths": [path]}
)
body = await resp.text()
self.assertNotIn(os.path.expanduser("~"), body)
data = json.loads(body)
# Only the name the caller sent may differ.
answers.append(
(
resp.status,
{k: v for k, v in data.items() if k not in ("source", "error")},
)
)
self.assertEqual(
data["error"],
f"{data['source']} can't be used: "
f"{config_module.OUTSIDE_ALLOWED_REASON}. Remove it, then save.",
)
self.assertEqual(
answers[0],
(
400,
{
"success": False,
"code": "invalid-source",
"index": 0,
"reason": config_module.OUTSIDE_ALLOWED_REASON,
},
),
)
self.assertEqual(answers, [answers[0]] * len(answers))
self.assertEqual(GalleryConfig.MONITORING_DIRECTORIES, [])
def test_every_caller_folds_root_and_home_alike(self):
for path in (os.path.abspath(os.sep), os.path.expanduser("~")):
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
GalleryConfig.validate_gallery_root(path),
(False, config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
async def test_outside_path_is_never_resolved_or_stat(self):
"""Containment is decided as text first: no realpath, stat or lstat of
an outside path, existing or not, by any caller (I2)."""
probe = self.outside_dir / "probe-x"
probe.mkdir()
(probe / "deep").mkdir()
cases = [str(probe), str(probe / "deep"), str(probe / "missing")]
with no_filesystem_access_for("probe-x", self):
for path in cases:
with self.subTest(path=path):
self.assertEqual(
GalleryConfig.gallery_root_problem(path)[0], "outside-allowed"
)
self.assertEqual(
sources.validate_source(path)["code"], "outside-allowed"
)
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": [path]}
)
self.assertEqual(error["code"], "invalid-source")
status, data = await self._post(VALIDATE, {"path": path})
self.assertEqual(data["code"], "outside-allowed")
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
# A traversal that normalises outside is refused as text too.
sneaky = os.path.join(
str(self.output_dir), "..", "..", "outside", "probe-x"
)
self.assertEqual(
GalleryConfig.gallery_root_problem(sneaky)[0], "outside-allowed"
)
async def test_settings_save_applies_the_form_checks(self):
"""POST /settings refuses what validate-source refuses, by its form."""
for path, code in {
str(self.a) + "\x00": "invalid-form",
os.path.join(str(self.output_dir), "b", "..", "a"): "traversal",
}.items():
with self.subTest(code=code):
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [path]}
)
self.assertEqual((status, data["code"]), (400, "invalid-source"))
self.assertEqual(data["reason"], sources.FORM_REASONS[code])
windows = functools.partial(sources.source_form_error, windows=True)
with (
patch.object(sources, "source_form_error", windows),
# The probes hold a token no trusted directory does (on Windows the
# temporary ComfyUI tree itself sits on C:).
no_filesystem_access_for("PMProbe", self),
):
for path in (
"\\\\?\\C:\\PMProbe\\output",
"\\\\.\\pipe\\PMProbe",
"C:\\PMProbe\\output\\CON",
"C:\\PMProbe\\output\\a:stream",
"C:PMProbe",
):
with self.subTest(path=path):
_roots, error = self.api._parse_gallery_roots(
{"gallery_root_paths": ["output/a", path]}
)
self.assertEqual(error["code"], "invalid-source")
self.assertEqual(error["index"], 1)
self.assertEqual(
error["reason"], sources.FORM_REASONS["invalid-form"]
)
async def test_inside_paths_say_why_they_are_not_usable(self):
a_file = self.output_dir / "file.png"
a_file.write_bytes(b"x")
status, missing = await self._post(
VALIDATE, {"path": str(self.output_dir / "missing")}
)
status, not_dir = await self._post(VALIDATE, {"path": str(a_file)})
self.assertEqual(
(missing["code"], not_dir["code"]), ("not-found", "not-directory")
)
async def test_malformed_request_is_400(self):
for body in ({}, {"path": 42}, {"path": None}, ["x"]):
with self.subTest(body=body):
status, data = await self._post(VALIDATE, body)
self.assertEqual(status, 400, data)
self.assertEqual(data["code"], "invalid-request")
self.assertFalse(data["success"])
class TestSourceEdges(SourcesTestCase):
async def test_validation_failure_is_500_without_paths(self):
def boom(path):
raise OSError(13, "Permission denied", f"{self.tmpdir}/secret")
with patch.object(sources, "validate_source", boom):
status, data = await self._post(VALIDATE, {"path": str(self.a)})
self.assertEqual(status, 500)
self.assertEqual(data["error"], "Failed to check the source")
async def test_oversized_body_is_413_too_large(self):
resp = await self.client.request(
"POST", VALIDATE, data=json.dumps({"path": "x" * (2 << 20)})
)
self.assertEqual(resp.status, 413)
self.assertEqual((await resp.json())["code"], "too-large")
async def test_unreadable_monitor_counts_as_no_live_observer(self):
class Broken:
def get_status(self):
raise RuntimeError("observer gone")
self._install_monitor(Broken())
status, data = await self._post(
"/prompt_manager/settings", {"monitoring_enabled": True}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], ["monitoring_enabled"])
def test_labels_skip_getters_that_fail_or_are_missing(self):
import sys
import types
def fails():
raise RuntimeError("no input dir")
fake = types.SimpleNamespace(
get_output_directory=lambda: str(self.output_dir),
get_input_directory=fails,
get_temp_directory=lambda: None,
)
with patch.dict(sys.modules, {"folder_paths": fake}):
self.assertEqual(sources.source_label(self.output_dir), "ComfyUI output")
self.assertEqual(sources.source_label(self.a), "a")
with patch.dict(sys.modules, {"folder_paths": None}):
self.assertEqual(sources.source_label(self.output_dir), "output")
class TestOutsideBaseDirectories(SourcesTestCase):
"""ComfyUI folders outside the base directory (``--output-directory``):
their sources round-trip and two folders of one name stay apart (I3)."""
async def asyncSetUp(self):
await super().asyncSetUp()
self.renders = self.outside_dir / "renders"
self.input_dir = self.comfy_dir / "input"
(self.renders / "portraits").mkdir(parents=True)
(self.input_dir / "portraits").mkdir(parents=True)
sys.modules["folder_paths"] = types.SimpleNamespace(
base_path=str(self.comfy_dir),
get_output_directory=lambda: str(self.renders),
get_input_directory=lambda: str(self.input_dir),
)
async def test_source_outside_base_round_trips(self):
status, rendered = await self._post(
VALIDATE, {"path": str(self.renders / "portraits")}
)
self.assertEqual(
(rendered["valid"], rendered["path"]), (True, "renders/portraits")
)
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [rendered["path"]]}
)
self.assertEqual(status, 200, data)
self.assertEqual(
GalleryConfig.MONITORING_DIRECTORIES,
[self._root(self.renders / "portraits")],
)
configured = (await self._get())["sources"]["configured"]
self.assertEqual(
[(row["path"], row["valid"]) for row in configured],
[("renders/portraits", True)],
)
# The page resends what it read: still the same folder, no change.
status, data = await self._post(
SETTINGS, {"gallery_root_paths": ["renders/portraits"]}
)
self.assertEqual((status, data["restart_required_fields"]), (200, []))
async def test_same_name_folders_stay_apart(self):
answers = []
for path in (self.renders / "portraits", self.input_dir / "portraits"):
status, data = await self._post(VALIDATE, {"path": str(path)})
answers.append(data["path"])
self.assertEqual(answers, ["renders/portraits", "input/portraits"])
status, data = await self._post(SETTINGS, {"gallery_root_paths": answers})
self.assertEqual(status, 200, data)
sources_ = (await self._get())["sources"]
self.assertEqual([row["path"] for row in sources_["configured"]], answers)
self.assertEqual([row["path"] for row in sources_["effective"]], answers)
class TestUnusableSourceSave(SourcesTestCase):
async def test_unusable_source_is_named_and_saves_nothing(self):
"""A configured source that went missing blocks the save and is named."""
gone = self.output_dir / "gone"
gone.mkdir()
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.a), self._root(gone)]
self._write_config({"custom": 1})
before = self._config_bytes()
gone.rmdir() # an unplugged drive
status, data = await self._post(
SETTINGS,
{
"gallery_root_paths": ["output/a", "output/gone", "output/b"],
"worker_threads": 2,
},
)
self.assertEqual(status, 400, data)
self.assertEqual(
data,
{
"success": False,
"code": "invalid-source",
"source": "output/gone",
"index": 1,
"reason": "Gallery root does not exist",
"error": "output/gone can't be used: Gallery root does not exist. "
"Remove it or reconnect it, then save.",
},
)
self.assertEqual(self._config_bytes(), before)
async def test_malformed_roots_are_coded_invalid_request(self):
for roots in ("output/a", [1]):
with self.subTest(roots=roots):
status, data = await self._post(SETTINGS, {"gallery_root_paths": roots})
self.assertEqual((status, data["code"]), (400, "invalid-request"))
class TestUnchangedRootsTakeNoHold(SourcesTestCase):
async def test_resent_spelling_of_configured_roots_is_no_change(self):
"""config.json holds a hand-written spelling; the legacy page resends
it resolved: no source change, so no hold and no 409 during a scan (M1)."""
GalleryConfig.MONITORING_DIRECTORIES = ["output/a"]
gate = asyncio.Event()
async def runner(job):
await gate.wait()
job.finish("finished")
self.api._jobs.start(
"scan", "Metadata scan", record_changing=True, runner=runner
)
try:
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.a)], "result_timeout": 30}
)
self.assertEqual(status, 200, data)
self.assertEqual(data["restart_required_fields"], [])
self.assertEqual([j.kind for j in self.api._jobs.list()], ["scan"])
status, data = await self._post(
SETTINGS, {"gallery_root_paths": [str(self.b)]}
)
self.assertEqual((status, data["code"]), (409, "busy"))
finally:
gate.set()
class TestWindowsContainment(unittest.TestCase):
"""Windows path forms, with ntpath fakes: anything outside the allowed
folders (a UNC share, the verbatim and device namespaces, another drive)
is refused as text, before any path is resolved or opened (I2)."""
ALLOWED = "c:\\comfyui\\output"
NAS = "\\\\nas\\share\\renders"
def setUp(self):
self.touched = []
def realpath(path, *args, **kwargs):
self.touched.append(path)
return ntpath.normcase(ntpath.normpath(path))
def probe(path):
self.touched.append(path)
return True
names = ("normcase", "normpath", "isabs", "join", "dirname", "basename", "sep")
fake_path = types.SimpleNamespace(
**{name: getattr(ntpath, name) for name in names},
realpath=realpath,
exists=probe,
isdir=probe,
expanduser=lambda path: "c:\\users\\someone",
)
fake_os = types.SimpleNamespace(
path=fake_path, environ={}, pathsep=";", name="nt"
)
for target, value in (
(patch.object(config_module, "os", fake_os), None),
(
patch.object(
GalleryConfig,
"_allowed_parent_pairs",
classmethod(
lambda cls: [(self.ALLOWED, self.ALLOWED), (self.NAS, self.NAS)]
),
),
None,
),
(
patch.object(
GalleryConfig,
"path_anchors",
classmethod(lambda cls: ["c:\\comfyui", "\\\\nas\\share"]),
),
None,
),
):
target.start()
self.addCleanup(target.stop)
def test_outside_forms_are_refused_untouched(self):
for path in (
"\\\\attacker\\share\\x",
"//attacker/share/x",
"\\\\?\\C:\\ComfyUI\\output\\a",
"\\\\?\\UNC\\attacker\\share",
"\\\\.\\pipe\\x",
"\\\\.\\C:\\ComfyUI\\output",
"D:\\Renders",
"C:\\Windows",
"C:\\ComfyUI\\output\\..\\..\\Windows",
"C:\\ComfyUI\\output2",
"\\\\nas\\share\\other",
"..\\..\\Windows",
):
with self.subTest(path=path):
self.touched.clear()
self.assertEqual(
GalleryConfig.gallery_root_problem(path),
("outside-allowed", config_module.OUTSIDE_ALLOWED_REASON),
)
self.assertEqual(self.touched, [])
def test_allowed_spellings_are_accepted(self):
for path in (
"C:\\ComfyUI\\output\\a",
"c:/comfyui/OUTPUT/a",
"output\\a",
"\\\\NAS\\share\\renders\\a", # a UNC spelling of an allowed extra root
):
with self.subTest(path=path):
self.assertEqual(GalleryConfig.gallery_root_problem(path), (None, ""))
class TestSourceFormError(unittest.TestCase):
"""The lexical check that runs before any filesystem access."""
def test_windows_forms(self):
err = sources.source_form_error
for path in (
"C:\\ComfyUI\\output",
"C:/ComfyUI/output",
"\\\\server\\share\\output",
"D:\\images",
):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=True))
for path, code in {
"C:output": "invalid-form", # drive-relative
"C:\\out\\a:stream": "invalid-form", # alternate data stream
"C:\\out\\CON": "invalid-form",
"C:\\out\\nul.txt": "invalid-form",
"C:\\out\\folder.": "invalid-form",
"C:\\out\\folder \\sub": "invalid-form",
"\\\\?\\C:\\out": "invalid-form",
"//?/C:/out": "invalid-form",
"\\\\.\\pipe\\x": "invalid-form",
"\\\\server": "invalid-form",
"C:\\out\\..\\in": "traversal",
"output\\..\\input": "traversal",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=True), code)
def test_posix_forms(self):
err = sources.source_form_error
for path in ("/srv/comfy/output", "output/renders", "out/CON", "a b/c."):
with self.subTest(path=path):
self.assertIsNone(err(path, windows=False))
for path, code in {
"C:\\ComfyUI\\output": "invalid-form",
"C:/ComfyUI/output": "invalid-form",
"\\\\server\\share": "invalid-form",
"out\\renders": "invalid-form",
"/srv/../etc": "traversal",
"../output": "traversal",
"/srv/out\x00": "invalid-form",
"/srv/out\n": "invalid-form",
"": "empty",
}.items():
with self.subTest(path=path):
self.assertEqual(err(path, windows=False), code)
if __name__ == "__main__":
unittest.main()
+1 -58
View File
@@ -292,7 +292,7 @@ class TestOverviewRoute(SystemAPITestCase):
body = await self.json("GET", "/prompt_manager/system/overview")
self.assertIsNone(body["health"])
self.assertIsNone(body["backup"])
self.assertEqual(body["needs_review"], {"pending": 0})
self.assertEqual(body["needs_review"], {"pending": None})
self.save("Any")
await self.run_health()
report = (await self.json("GET", "/prompt_manager/system/health"))["report"]
@@ -416,63 +416,6 @@ class TestStatisticsRoute(SystemAPITestCase):
)
class TestFailedReadsSettleFirst(SystemAPITestCase):
"""A failed read answers only after the route's other reads finished.
Answering on the first failure (``asyncio.gather`` without
``return_exceptions``) left the other executor reads running against a
database the caller may already be closing; on Python 3.10 CI that
crashed the test process inside SQLite.
"""
async def assert_answers_after_reads(self, url, failing, slow, error):
raised, release, finished = (threading.Event() for _ in range(3))
real = getattr(*slow)
def fail(*args, **kwargs):
raised.set()
raise RuntimeError("secret")
def held(*args, **kwargs):
release.wait()
try:
return real(*args, **kwargs)
finally:
finished.set()
with (
patch.object(*failing, side_effect=fail),
patch.object(*slow, side_effect=held),
self.assertLogs("prompt_manager.api", "ERROR"),
):
answer = asyncio.ensure_future(self.json("GET", url, 500))
try:
await asyncio.get_running_loop().run_in_executor(None, raised.wait)
done, _ = await asyncio.wait({answer}, timeout=0.5)
self.assertEqual(done, set(), "answered while a read still ran")
finally:
release.set()
body = await answer
self.assertTrue(finished.is_set())
self.assertEqual(body, {"success": False, "code": None, "error": error})
async def test_statistics(self):
await self.assert_answers_after_reads(
"/prompt_manager/system/statistics",
(self.api.db, "usage_by_category"),
(self.api.db, "get_statistics"),
"Failed to read statistics",
)
async def test_overview(self):
await self.assert_answers_after_reads(
"/prompt_manager/system/overview",
(self.api.db, "library_counts"),
(self.api, "_output_dirs_with_origin"),
"Failed to read the overview",
)
if __name__ == "__main__":
import unittest
-38
View File
@@ -1,38 +0,0 @@
"""The System browser hooks only ever reset their own server's config.json (#249 S3)."""
import os
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from tests.browser.workspace_system_fixture import register_system_commands
class TestWorkspaceSystemFixture(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)
self.root = Path(self.tmp.name) / "server"
self.root.mkdir()
def test_refuses_a_config_path_outside_its_root(self):
elsewhere = Path(self.tmp.name) / "config.json"
elsewhere.write_bytes(b'{"real": true}')
for value in (str(elsewhere), None):
env = {
k: v for k, v in os.environ.items() if k != "PROMPT_MANAGER_CONFIG_PATH"
}
if value is not None:
env["PROMPT_MANAGER_CONFIG_PATH"] = value
with (
self.subTest(value=value),
mock.patch.dict(os.environ, env, clear=True),
):
with self.assertRaisesRegex(RuntimeError, "PROMPT_MANAGER_CONFIG_PATH"):
register_system_commands(None, None, None, self.root)
self.assertEqual(elsewhere.read_bytes(), b'{"real": true}')
if __name__ == "__main__":
unittest.main()
-14
View File
@@ -51,17 +51,3 @@ test('CI retains stock, execution and pack suites in order with uploaded executi
assert.deepEqual(commands, ['test:comfyui', 'test:executions', 'test:compat']);
assert.match(workflow, /path: tools\/vendor\/test-results\//);
});
test('real-host config waits for its own launcher, never for whichever server holds the port', async t => {
const previous = process.env.PM_COMFY_ROOT;
process.env.PM_COMFY_ROOT = '/unused-pinned-checkout';
t.after(() => { if (previous === undefined) delete process.env.PM_COMFY_ROOT; else process.env.PM_COMFY_ROOT = previous; });
const { default: config } = await import('../../tools/vendor/comfyui.config.mjs');
const [server] = config.webServer;
assert.equal(server.url, undefined);
assert.equal(server.port, undefined);
assert.equal(server.reuseExistingServer, false);
const ready = server.wait.stdout.exec('Starting server\nPM_COMFY_HOST_READY base-directory=/tmp/comfyassets-host-a b\r\nlater');
assert.equal(ready.groups.pm_comfy_host_base, '/tmp/comfyassets-host-a b');
assert.equal(server.wait.stdout.exec('To see the GUI go to: http://127.0.0.1:8189'), null);
});
+1 -50
View File
@@ -6,8 +6,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { once } from 'node:events';
import { createServer } from 'node:net';
import { createServer as createHttpServer } from 'node:http';
import { launchComfyHost, assertPortFree, waitForOwnHost, hostBaseDirectory } from '../../tools/vendor/comfyui-host.mjs';
import { launchComfyHost } from '../../tools/vendor/comfyui-host.mjs';
async function fixture(t, program) {
const directory = await mkdtemp(join(tmpdir(), 'pm-host-test-'));
@@ -233,51 +232,3 @@ test('execution launcher refuses an occupied port without touching an existing s
await new Promise(resolve => server.close(resolve));
}
});
// A stand-in ComfyUI that answers /system_stats with its own argv, as the real host does.
const answersOwnArgv = `const args = process.argv.slice(1);
require('node:http').createServer((request, response) => {
response.setHeader('content-type', 'application/json');
response.end(JSON.stringify({ system: { argv: args } }));
}).listen(Number(args[args.indexOf('--port') + 1]), '127.0.0.1');`;
async function listen(t, port, handler = (request, response) => response.end()) {
const server = createHttpServer(handler);
await new Promise((resolve, reject) => { server.once('error', reject); server.listen(port, '127.0.0.1', resolve); });
t.after(() => new Promise(resolve => server.close(resolve)));
return server;
}
test('real-host launcher refuses a port another run already holds and leaves that server running', async t => {
const server = await listen(t, 8199);
await assert.rejects(assertPortFree(8199), /8199 is already in use.*never shared/s);
assert.equal(server.listening, true);
});
test('real-host launcher is ready only when its own ComfyUI answers the port', async t => {
const options = await fixture(t, answersOwnArgv);
await assertPortFree(options.port);
const host = await launchComfyHost(options);
t.after(() => host.stop());
await waitForOwnHost(host, options.port);
const stats = await (await fetch(`http://127.0.0.1:${options.port}/system_stats`)).json();
assert.equal(hostBaseDirectory(stats), host.stateDirectory);
});
test('real-host launcher refuses a host it did not start that answers before its own binds', async t => {
await listen(t, 8199, (request, response) => {
response.setHeader('content-type', 'application/json');
response.end(JSON.stringify({ system: { argv: ['main.py', '--base-directory', '/other-run-state'] } }));
});
const options = await fixture(t, idle);
const host = await launchComfyHost(options);
t.after(() => host.stop());
await assert.rejects(waitForOwnHost(host, options.port), /did not start \(base directory \/other-run-state\)/);
});
test('real-host launcher reports a host that exits before it answers', async t => {
const options = await fixture(t, 'process.exit(3);');
const host = await launchComfyHost(options);
t.after(() => host.stop());
await assert.rejects(waitForOwnHost(host, options.port), /exited before it answered on port 8199 \(exit=3/);
});
+3 -58
View File
@@ -1,5 +1,4 @@
import { writeFile } from 'node:fs/promises';
import { createServer } from 'node:net';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { launchComfyHost as launchSharedHost } from '@comfyassets/toolkit/testing/host';
@@ -8,9 +7,6 @@ import runtime from './comfyui-runtime.json' with { type: 'json' };
export { launchSharedHost };
const repositoryRoot = fileURLToPath(new URL('../../', import.meta.url));
// comfyui.config.mjs waits for this line; its named group becomes PM_COMFY_HOST_BASE in the test workers.
export const HOST_READY_MARKER = 'PM_COMFY_HOST_READY';
const READY_POLL_MS = 250;
export async function preparePromptManagerPlugin({ installed, stateDirectory }) {
const configPath = join(installed, 'config.json');
@@ -31,68 +27,17 @@ export function launchComfyHost({ checkout, repository = repositoryRoot,
});
}
export async function assertPortFree(port) {
await new Promise((resolve, reject) => {
const probe = createServer();
probe.once('error', error => reject(new Error(`ComfyUI host port ${port} is already in use, often by another `
+ 'real-host run. A managed host is never shared: stop that run or set PM_COMFY_URL to a free '
+ 'http://127.0.0.1:PORT/ address.', { cause: error })));
probe.listen({ host: '127.0.0.1', port, exclusive: true }, () => probe.close(error => error ? reject(error) : resolve()));
});
}
export function hostBaseDirectory(stats) {
const argv = stats?.system?.argv;
if (!Array.isArray(argv)) return null;
const index = argv.indexOf('--base-directory');
return index >= 0 ? argv[index + 1] ?? null : null;
}
// Ready means this launcher's own ComfyUI answers: /system_stats reports its base directory.
// Any other answer on the port comes from a host this run did not start, so the run must not use it.
export async function waitForOwnHost(host, port, { fetchImpl = fetch } = {}) {
let exited = null;
host.finished.then(result => { exited = result; });
const url = `http://127.0.0.1:${port}/system_stats`;
for (;;) {
let stats = null;
try { stats = await (await fetchImpl(url)).json(); }
catch { stats = null; }
if (stats) {
const base = hostBaseDirectory(stats);
if (base === host.stateDirectory) return;
throw new Error(`Port ${port} is answered by a ComfyUI host this run did not start `
+ `(base directory ${base ?? 'unknown'}); another run is using the port.`);
}
if (exited) {
throw new Error(`ComfyUI host exited before it answered on port ${port} `
+ `(exit=${exited.code}, signal=${exited.signal || 'none'}).`);
}
await new Promise(resolve => setTimeout(resolve, READY_POLL_MS));
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) {
try {
const port = Number(process.env.PM_COMFY_PORT || 8189);
await assertPortFree(port);
const host = await launchComfyHost({ checkout: process.env.PM_COMFY_ROOT, port });
const host = await launchComfyHost({ checkout: process.env.PM_COMFY_ROOT,
port: Number(process.env.PM_COMFY_PORT || 8189) });
const stop = () => { host.stop().catch(error => { console.error(error.message); process.exitCode = 1; }); };
process.once('SIGINT', stop);
process.once('SIGTERM', stop);
let refused = false;
try {
await waitForOwnHost(host, port);
console.log(`${HOST_READY_MARKER} base-directory=${host.stateDirectory}`);
} catch (error) {
refused = true;
console.error(error.message);
await host.stop();
}
const result = await host.finished;
process.removeListener('SIGINT', stop);
process.removeListener('SIGTERM', stop);
if (result.code) console.error(`ComfyUI host exited with status ${result.code}; see tools/vendor/test-results/comfyui-host.log`);
process.exitCode = refused ? 1 : result.code;
process.exitCode = result.code;
} catch (error) { console.error(error.message); process.exitCode = 1; }
}
+1 -6
View File
@@ -1,6 +1,5 @@
import { defineConfig } from './node_modules/@playwright/test/index.mjs';
import { fileURLToPath } from 'node:url';
import { HOST_READY_MARKER } from './comfyui-host.mjs';
if (!process.env.PM_COMFY_ROOT) throw new Error('Set PM_COMFY_ROOT to a pinned disposable ComfyUI checkout');
const host = new URL(process.env.PM_COMFY_URL || 'http://127.0.0.1:8189/');
@@ -8,11 +7,7 @@ if (host.protocol !== 'http:' || host.hostname !== '127.0.0.1' || !host.port) th
const servers = [{
command: 'node tools/vendor/comfyui-host.mjs',
cwd: fileURLToPath(new URL('../../', import.meta.url)),
// No URL wait: any server on the port would satisfy it, including another run's host that
// then stops mid-test. The launcher refuses a used port and prints this line only once its
// own ComfyUI answers; the group reaches the tests as PM_COMFY_HOST_BASE.
wait: { stdout: new RegExp(`^${HOST_READY_MARKER} base-directory=(?<pm_comfy_host_base>.+?)\r?$`, 'm') },
timeout: 120000, reuseExistingServer: false,
url: new URL('system_stats', host).href, timeout: 120000, reuseExistingServer: false,
env: { PM_COMFY_PORT: host.port },
gracefulShutdown: { signal: 'SIGTERM', timeout: 10000 },
}];
+11 -29
View File
@@ -7,7 +7,7 @@
},
"api-client.js": {
"lines": 98,
"branches": 96,
"branches": 95,
"functions": 98,
"statements": 98
},
@@ -37,15 +37,15 @@
},
"gallery-settings.js": {
"lines": 98,
"branches": 94,
"branches": 90,
"functions": 98,
"statements": 98
},
"gallery.js": {
"lines": 41,
"lines": 40,
"branches": 63,
"functions": 44,
"statements": 41
"statements": 40
},
"html-escape.js": {
"lines": 98,
@@ -163,7 +163,7 @@
},
"workspace/activity-scan.mjs": {
"lines": 96,
"branches": 78,
"branches": 73,
"functions": 98,
"statements": 96
},
@@ -401,12 +401,6 @@
"functions": 98,
"statements": 98
},
"workspace/preferences.mjs": {
"lines": 98,
"branches": 96,
"functions": 98,
"statements": 98
},
"workspace/prompt-blocks.mjs": {
"lines": 90,
"branches": 92,
@@ -449,12 +443,6 @@
"functions": 98,
"statements": 98
},
"workspace/system-appearance.mjs": {
"lines": 98,
"branches": 93,
"functions": 98,
"statements": 98
},
"workspace/system-controls.mjs": {
"lines": 98,
"branches": 87,
@@ -467,23 +455,17 @@
"functions": 98,
"statements": 96
},
"workspace/system-sources.mjs": {
"lines": 98,
"branches": 88,
"functions": 95,
"statements": 98
},
"workspace/system-state.mjs": {
"lines": 98,
"branches": 94,
"functions": 98,
"branches": 92,
"functions": 92,
"statements": 98
},
"workspace/system.mjs": {
"lines": 98,
"branches": 92,
"functions": 86,
"statements": 98
"lines": 93,
"branches": 85,
"functions": 69,
"statements": 93
},
"workspace/theme.mjs": {
"lines": 98,
+3 -95
View File
@@ -6,28 +6,19 @@ the inspection projection (``project_graph``: strict JSON, the depth and node
bounds, big integers and decimals as text), and only graphs it parsed, through
the shared parser (``extract_generation_params``, the Python twin of the
browser parser the inspector fills from since #251). No separate seed
heuristic and no extra file reads. One parse feeds both the seed and the
positive that confirms a first-line card title. One exception keeps the card
from ever showing a seed the inspector does not: when sampler stages read
different seed inputs and the inspector's resolution names one of them for a
resolved pair (whose facts are that stage's), the card shows no seed when the
named stage's seed differs from the parser's. Only such graphs are resolved,
with the inspector's own profile.
heuristic, no graph resolution and no extra file reads. One parse feeds both
the seed and the positive that confirms a first-line card title.
"""
import json
import re
from collections.abc import Mapping
from .generation_params import (
PLACEHOLDERS,
GenerationParamsError,
_is_sampler,
extract_generation_params,
)
from .inspection_response import project_graph
from .inspection_semantics import SemanticsError, get_semantics
from .prompt_pair_resolver import INSPECTION_PROFILE, resolve_prompt_pair
__all__ = [
"MAX_RECORD_BYTES",
@@ -82,93 +73,10 @@ def header_params(info) -> dict | None:
if not graphs:
return None
try:
params = extract_generation_params(graphs)
return extract_generation_params(graphs)
except (GenerationParamsError, RecursionError):
# Defence in depth: a parser failure gives no card facts, never a 500.
return None
if isinstance(params, dict) and _inspector_names_another_seed(
graphs, card_seed(params)
):
return {**params, "seed": None}
return params
_SEED_INPUTS = ("seed", "noise_seed", "noise")
def _stage_seed(graph: Mapping, node_id: str) -> str | None:
"""The seed the parser reads for one sampler node, or None when unknown.
A literal seed input is that seed. A linked one is read by the parser
itself when it treats the node as a sampler, with every other such node
hidden from it (no ``positive`` input), so linked seeds resolve as they do
on the card; any other linked seed is unknown.
"""
node = graph.get(node_id)
if not isinstance(node, Mapping) or not isinstance(node.get("inputs"), Mapping):
return None
inputs = node["inputs"]
literal = next((inputs[k] for k in _SEED_INPUTS if k in inputs), None)
if not isinstance(literal, (list, tuple)):
# A literal needs no second parse (the cost is measured in TESTING.md).
return card_seed({"seed": literal})
if _is_sampler(node):
only = {
key: (
{**other, "inputs": {**other["inputs"], "positive": None}}
if key != node_id and _is_sampler(other)
else other
)
for key, other in graph.items()
}
try:
return card_seed(extract_generation_params({"prompt": only}))
except (GenerationParamsError, RecursionError):
return None
return None
def _inspector_names_another_seed(graphs: dict, parsed: str | None) -> bool:
"""Whether the inspector shows another stage's seed than the parser's.
True when the prompt graph's sampler stages read different seed inputs, the
inspection resolution names one sampler for a resolved pair (the
inspector's facts are then that stage's, D19), and that stage's seed is
known and differs from ``parsed``. A seed that agrees is kept.
"""
if parsed is None:
return False
graph = graphs.get("prompt")
if not isinstance(graph, Mapping):
return False
try:
samplers = get_semantics().samplers
except SemanticsError:
return False
seeds = set()
for node in graph.values():
if not isinstance(node, Mapping) or not isinstance(node.get("inputs"), Mapping):
continue
if not isinstance(node.get("class_type"), str):
continue
if node["class_type"] not in samplers:
continue
inputs = node["inputs"]
value = next((inputs[key] for key in _SEED_INPUTS if key in inputs), None)
seeds.add(json.dumps(value, sort_keys=True, default=str))
if len(seeds) < 2:
return False
try:
resolution = resolve_prompt_pair(
{"prompt": dict(graph)}, profile=INSPECTION_PROFILE
)
except RecursionError:
return True
named = resolution["sampler_id"]
if resolution["state"] != "resolved" or named is None:
return False
seed = _stage_seed(graph, named)
return seed is not None and seed != parsed
def card_seed(params: dict | None) -> str | None:
-25
View File
@@ -28,7 +28,6 @@ __all__ = [
"extract_generation_params",
"extract_from_records",
"parse_js_json",
"parse_page_json",
]
PLACEHOLDERS = MappingProxyType(
@@ -954,30 +953,6 @@ def parse_js_json(text):
return None
def _nan_as_null(name):
"""The legacy pages' NaN cleaning; Infinity still throws in ``JSON.parse``."""
if name == "NaN":
return None
raise ValueError(f"{name} is not JSON")
def parse_page_json(text):
"""A metadata record as the legacy gallery and admin pages parsed it.
They replace ``NaN`` with ``null`` and then ``JSON.parse`` (ComfyUI writes
``"is_changed": [NaN]`` for nodes that always re-run, PromptManager among
them). Only the NaN number reads as null here: the pages' regex also
rewrote the word inside strings, which changed prompt text. None where the
parse throws.
"""
if not isinstance(text, str):
return None
try:
return json.loads(text, parse_constant=_nan_as_null, parse_int=_double_integer)
except (ValueError, RecursionError):
return None
def extract_from_records(prompt_text, workflow_text):
"""``extract_generation_params`` for raw ``prompt``/``workflow`` record text.
+1 -39
View File
@@ -27,7 +27,7 @@ import os
import queue
import time
import threading
from typing import Callable, Optional, Dict, Any, Tuple
from typing import Optional, Dict, Any, Tuple
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
@@ -75,10 +75,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
self.prompt_tracker = prompt_tracker
self.metadata_extractor = ComfyUIMetadataExtractor()
self.logger = get_logger("prompt_manager.image_monitor")
# The Needs review pass for ambiguous outputs (the API's
# ``record_review_file``, wired at start-up): a linked output is
# never queued and its pending items leave the queue. None: no pass.
self.review_recorder: Optional[Callable[[str], str]] = None
# Read from GalleryConfig if available, otherwise use defaults
try:
@@ -583,29 +579,6 @@ class ImageGenerationHandler(FileSystemEventHandler):
)
except Exception as e:
self.logger.error(f"Failed to link image to prompt: {e}")
return
if resolution["state"] == "ambiguous":
self._record_review(image_path)
def _record_review(self, image_path: str) -> None:
"""The Needs review pass for an ambiguous output, best effort.
The output is linked by now, so it is never queued; a pending item
an earlier scan left for its path leaves the queue. The link above
stays whatever happens here: a failure is logged, never raised.
"""
recorder = getattr(self, "review_recorder", None)
if recorder is None:
return
try:
outcome = recorder(image_path)
except Exception:
self.logger.warning(
f"{os.path.basename(image_path)} could not be added to Needs review",
exc_info=True,
)
return
self.logger.debug(f"Needs review {outcome}: {os.path.basename(image_path)}")
class ImageMonitor:
@@ -635,16 +608,6 @@ class ImageMonitor:
self.handler = None
self.monitored_directories = []
self.logger = get_logger("prompt_manager.image_monitor")
self.review_recorder: Optional[Callable[[str], str]] = None
def set_review_recorder(self, recorder: Optional[Callable[[str], str]]) -> None:
"""Record ambiguous outputs through ``recorder`` (None stops recording).
Applies to the running handler and to every handler started later.
"""
self.review_recorder = recorder
if self.handler is not None:
self.handler.review_recorder = recorder
def start_monitoring(self, output_directories: Optional[list] = None):
"""
@@ -702,7 +665,6 @@ class ImageMonitor:
# Only a started observer becomes self.observer, so a failed start never
# leaves the monitor stuck "already running" with nothing to stop.
handler = ImageGenerationHandler(self.db_manager, self.prompt_tracker)
handler.review_recorder = self.review_recorder
observer = Observer()
scheduled = []
for output_dir in valid_directories:
+4 -42
View File
@@ -12,10 +12,8 @@ valid result. Importing this module reads nothing, and the default (ingestion)
profile never imports it, so a missing or bad table can never break ingestion.
An invalid table raises ``SemanticsError`` so a bad edit fails loudly instead of
silently changing what the inspector resolves. The file is bounded in size and
nesting before it is validated. Every class appears in exactly one prompt
category; classes that are not listed stay ``unsupported-node``. ``lineage``
(image and latent pass-through outputs for multi-stage chains) describes no
prompt, so its classes are not claimed by a category.
nesting before it is validated. Every class appears in exactly one category;
classes that are not listed stay ``unsupported-node``.
"""
import json
@@ -46,7 +44,6 @@ STRATEGIES: Mapping[str, FrozenSet[str]] = MappingProxyType(
"encoders": frozenset({"single", "parts-equal", "pm-compose", "flag-parts"}),
"strings": frozenset({"literal", "strip-newlines", "join-multi", "concat"}),
"savers": frozenset({"comfy-counter", "vhs-counter"}),
"lineage": frozenset({"continue"}),
}
)
# Shapes of the facts each sampler class provides (consumed by comfy-metadata.js).
@@ -75,7 +72,7 @@ _CATEGORIES = frozenset(
+ _CLASS_CATEGORIES
+ _SLOT_CATEGORIES
+ _SET_CATEGORIES
+ ("lineage", "stage_inputs", "facts")
+ ("stage_inputs", "facts")
)
# Input names that make an unlisted node a possible sampler stage in an ancestry
# walk (``shaped``: pipes, video embeds and custom-sampler parts carry a prompt
@@ -114,7 +111,6 @@ class Semantics:
runtime: FrozenSet[str]
savers: Mapping[str, Mapping[str, Any]]
non_file_outputs: FrozenSet[str]
lineage: Mapping[Tuple[str, int], Mapping[str, Any]]
stage_inputs: Mapping[str, Tuple[str, ...]]
facts: Mapping[str, Any]
all_classes: FrozenSet[str]
@@ -229,12 +225,8 @@ def _check_selector(entry: dict, strategy: str, where: str) -> None:
def _check_composite(entry: dict, strategy: str, where: str) -> None:
# ``additive``: every part stays in effect (combine, concat), so a later
# stage that adds parts to an earlier stage's prompt keeps that prompt.
_fields(entry, where, frozenset({"strategy", "inputs"}), frozenset({"additive"}))
_fields(entry, where, frozenset({"strategy", "inputs"}), frozenset())
_names(entry["inputs"], where, minimum=2)
if "additive" in entry and not isinstance(entry["additive"], bool):
raise SemanticsError(f"{where}: additive must be a boolean")
def _check_saver(entry: dict, strategy: str, where: str) -> None:
@@ -364,35 +356,6 @@ def _slot_category(doc: dict, category: str, owners: _Owners) -> Mapping:
return MappingProxyType(result)
def _lineage(doc: dict) -> Mapping:
"""``(class, slot)`` outputs whose image or latent continues one input's.
These describe an output's image or latent, never its prompt, so they are
not prompt categories: a class listed here is not claimed, and stays
unsupported (or listed elsewhere) for text. Each output is listed once.
"""
rows = doc["lineage"]
if not isinstance(rows, list):
raise SemanticsError("category lineage must be a list")
result = {}
for index, row in enumerate(rows):
where = f"lineage[{index}]"
if not isinstance(row, dict):
raise SemanticsError(f"{where}: entry must be an object")
name = _name(row.get("class"), where, "class")
slot = _slot(row, where)
entry = {
key: value for key, value in row.items() if key not in ("class", "slot")
}
_strategy(entry, "lineage", where)
_fields(entry, where, frozenset({"strategy", "input"}), frozenset())
_name(entry["input"], where)
if (name, slot) in result:
raise SemanticsError(f"lineage: duplicate output key {(name, slot)!r}")
result[(name, slot)] = _freeze(entry)
return MappingProxyType(result)
def _set_category(doc: dict, category: str, owners: _Owners) -> FrozenSet[str]:
rows = doc[category]
if not isinstance(rows, list):
@@ -489,7 +452,6 @@ def _validate(doc: Any) -> Semantics:
for category in _SET_CATEGORIES:
tables[category] = _set_category(doc, category, owners)
return Semantics(
lineage=_lineage(doc),
stage_inputs=_stage_inputs(doc),
facts=_facts(doc),
all_classes=owners.classes(),
-42
View File
@@ -1,42 +0,0 @@
"""Strict JSON output for values that may carry ComfyUI's non-finite numbers.
ComfyUI writes its metadata with Python's default ``json.dumps``, so a graph can
hold ``NaN``, ``Infinity`` and ``-Infinity`` (an ``IS_CHANGED`` that returns
``float("nan")`` puts ``"is_changed": [NaN]`` into every saved prompt). Those
literals are not JSON: browsers' ``JSON.parse`` and ``allow_nan=False`` reject
them. Reading follows the legacy pages (``parse_page_json`` where the browser
parser is mirrored; Python's ``json.loads`` elsewhere). Whatever PromptManager
stores or answers as JSON goes through ``dumps_strict``, which writes each
non-finite number as ``null`` (the legacy pages' own replacement for NaN) and
leaves strings, including the words ``NaN`` or ``Infinity`` inside prompt
text, untouched.
"""
import json
from typing import Any
__all__ = ["dumps_strict"]
def _null(_constant: str) -> None:
return None
def dumps_strict(value: Any, **kwargs: Any) -> str:
"""``json.dumps(value, allow_nan=False, **kwargs)`` with non-finite numbers null.
A value without non-finite numbers serializes byte for byte as the plain
call does. Any other error (a cycle, an unserializable object) still raises.
"""
kwargs.pop("allow_nan", None)
try:
return json.dumps(value, allow_nan=False, **kwargs)
except ValueError as error:
if "Out of range float" not in str(error):
raise
# The non-strict text spells each non-finite number as a bare literal, never
# inside a string, so parsing it back maps exactly those numbers to null.
lenient = json.dumps(value, allow_nan=True, default=kwargs.get("default"))
return json.dumps(
json.loads(lenient, parse_constant=_null), allow_nan=False, **kwargs
)
+16 -46
View File
@@ -46,10 +46,6 @@ DEFAULT_MAX_BYTES = 512 * 2**20
DEFAULT_MAX_WORKERS = 2
DEFAULT_IDLE_SECONDS = 900
DEFAULT_HARD_SECONDS = 1800
# Background recording (Needs review from the scan and the monitor) may use
# every worker and capture slot but the last one, which stays free for an
# interactive Tag dialog or Apply; it waits this long for room before failing.
BACKGROUND_WAIT_SECONDS = 2.0
MAX_INFERENCE_SIDE = 4096
MAX_INFERENCE_PIXELS = 16_777_216
MAX_RELATIVE_LENGTH = 1024
@@ -84,7 +80,7 @@ class CaptureError(Exception):
class CaptureCapacity(CaptureError):
"""Count, byte or worker admission failed (interactive: nonblocking)."""
"""Count, byte or worker admission failed (nonblocking)."""
code = "review-capacity"
@@ -224,8 +220,6 @@ class CaptureStore:
# mkdtemp creates the directory with owner-only permissions.
self.directory = tempfile.mkdtemp(prefix=_DIRECTORY_PREFIX, dir=parent)
self._lock = threading.Lock()
# Signalled whenever a worker or a capture slot frees up.
self._room = threading.Condition(self._lock)
self._entries: dict[str, _Entry] = {}
self._completed_bytes = 0
self._pending = 0
@@ -244,14 +238,8 @@ class CaptureStore:
epoch: str,
scope: str,
root_index: int = 0,
background: bool = False,
) -> Capture:
"""Copy and hash one rooted media file into a private capture.
A ``background`` capture never takes the last worker or capture slot
and waits up to ``BACKGROUND_WAIT_SECONDS`` for room; an interactive
one is admitted or refused at once.
"""
"""Copy and hash one rooted media file into a private capture."""
if isinstance(root_index, bool) or not isinstance(root_index, int):
raise CaptureDenied("Invalid gallery root")
if root_index < 0:
@@ -262,7 +250,7 @@ class CaptureStore:
parts, extension = _relative_parts(relative)
candidate, locator = _resolve(root, parts)
before = self._pre_stat(candidate)
self._admit(before.st_size, reserve=True, background=background)
self._admit(before.st_size, reserve=True)
try:
capture = self._copy(
candidate, locator, extension, before, root_index, epoch, scope
@@ -398,52 +386,35 @@ class CaptureStore:
# -- admission -------------------------------------------------------------
def _admit(self, size: int, *, reserve: bool, background: bool = False) -> None:
def _admit(self, size: int, *, reserve: bool) -> None:
try:
self._admit_locked(size, reserve=reserve, background=background)
self._admit_locked(size, reserve=reserve)
finally:
self._flush()
def _admit_locked(self, size: int, *, reserve: bool, background: bool) -> None:
deadline = time.monotonic() + BACKGROUND_WAIT_SECONDS
def _admit_locked(self, size: int, *, reserve: bool) -> None:
with self._lock:
while True:
if self._closed:
raise RuntimeError("The capture store is closed")
self._sweep_locked(self._clock())
try:
self._check_room_locked(size, reserve, background)
break
except CaptureCapacity:
remaining = deadline - time.monotonic()
if not background or remaining <= 0:
raise
self._room.wait(remaining)
if self._closed:
raise RuntimeError("The capture store is closed")
self._sweep_locked(self._clock())
if self._workers >= self.max_workers:
raise CaptureCapacity("Too many captures are in progress")
if reserve:
if len(self._entries) + self._pending >= self.max_captures:
raise CaptureCapacity("Too many captures are held")
reserved = self._completed_bytes + self._pending_bytes
if reserved + size > self.max_bytes:
raise CaptureCapacity("Capture byte budget exhausted")
self._pending += 1
self._pending_bytes += size
self._workers += 1
def _check_room_locked(self, size: int, reserve: bool, background: bool) -> None:
"""Raise ``CaptureCapacity`` unless one more capture or verify fits."""
spare = 1 if background else 0
if self._workers + spare >= self.max_workers:
raise CaptureCapacity("Too many captures are in progress")
if reserve:
held = len(self._entries) + self._pending
if held + spare >= self.max_captures:
raise CaptureCapacity("Too many captures are held")
reserved = self._completed_bytes + self._pending_bytes
if reserved + size > self.max_bytes:
raise CaptureCapacity("Capture byte budget exhausted")
def _release_admission(self, size: int, *, reserve: bool) -> None:
with self._lock:
self._workers -= 1
if reserve:
self._pending -= 1
self._pending_bytes -= size
self._room.notify_all()
last_out = self._closed and self._pending == 0 and self._workers == 0
if last_out:
self._remove_directory()
@@ -486,7 +457,6 @@ class CaptureStore:
if self._entries.pop(entry.capture.token, None) is not None:
self._completed_bytes -= entry.capture.size
self._stale.append(entry.capture.path)
self._room.notify_all()
def _flush(self) -> None:
"""Unlink files released under the lock, without holding it (slow I/O)."""
+23 -288
View File
@@ -39,12 +39,6 @@ from .pair_execution import ExecutionEvidenceError, read_pair_execution
_LOGGER = logging.getLogger("prompt_manager.prompt_pair_resolver")
# The ingestion resolver's version, recorded in pair-link provenance and on
# Needs review items. Bump it when a change would resolve the same metadata
# otherwise; a pending item stays applicable while its chosen candidate is
# still offered by the captured bytes.
RESOLVER_VERSION = 1
@dataclass(frozen=True)
class OutputAdapter:
@@ -197,18 +191,6 @@ EMPTY_POSITIVE_REASON = "Positive prompt is empty"
# Several sampler branches, every one resolved to the same pair (#249).
IDENTICAL_BRANCHES_REASON = "Every sampler branch uses the same prompt pair"
MULTISTAGE_UNATTRIBUTED_REASON = "Multistage sampler ancestry is ambiguous"
# A later stage's positive combines the earlier stages' shared prompt with more
# conditioning (a style trigger): that shared pair is the run's prompt (floor).
ADDED_CONDITIONING_REASON = "A later stage adds conditioning to the base prompt"
# Maintainer option A (2026-10-23): a PromptManager prepend or append fed
# directly by a run-time node (LoRA trigger words) is left out, never guessed;
# the text is the node's own, labelled by this source and reason (D125). A side
# is labelled when any node on its traced path left a part out.
RUNTIME_PARTS_SOURCE = "runtime-parts-omitted"
RUNTIME_PARTS_REASON = "Text added at run time, such as trigger words, is not included"
# The added-conditioning floor's positive (D126), and both labels at once.
ADDED_CONDITIONING_SOURCE = "added-conditioning"
RUNTIME_AND_ADDED_SOURCE = "runtime-parts-omitted-and-added-conditioning"
_INCOMPATIBLE_SLOT = "Output slot has an incompatible type"
# Before 3.2.3 a save patch overwrote every PromptManager node's saved ``text``
# with the last-run node's prompt (comfy-metadata.js ``createContext``).
@@ -328,51 +310,15 @@ def _role_text(side: Any) -> Optional[str]:
return side["text"] if isinstance(side, dict) else None
def _left_out(side: Any) -> tuple:
"""What a side's text leaves out: the PromptManager nodes that omitted a
run-time part, and the composite whose added conditioning is not included.
Empty for a complete text, so a partial prompt never equals a complete one.
"""
provenance = side.get("provenance") if isinstance(side, dict) else None
if not isinstance(provenance, dict):
return ()
return (
tuple(provenance.get("omitted_parts", ())),
provenance.get("added_by"),
)
def _outcome_key(result: Dict[str, Any]) -> tuple:
"""What a reader sees of a result: state, the two role texts, and what
each text leaves out."""
"""What a reader sees of a result: state and the two role texts."""
return (
result["state"],
_role_text(result["positive"]),
_role_text(result["negative"]),
_left_out(result["positive"]),
_left_out(result["negative"]),
)
def _reroute_value(inputs: dict) -> Any:
"""A Reroute's one input (``value``, ``input`` or ``in``), else ``None``."""
keys = [key for key in ("value", "input", "in") if key in inputs]
return inputs[keys[0]] if len(keys) == 1 else None
def _parts_reason(*sides: dict) -> str:
"""The run-time parts reason when a side left out a PromptManager part."""
if any(side["provenance"]["text_source"] == RUNTIME_PARTS_SOURCE for side in sides):
return RUNTIME_PARTS_REASON
return ""
def _combined_positive(result: Dict[str, Any]) -> bool:
"""Whether a stage failed only because its positive combines texts."""
return result["reason_code"] == "composite" and result["positive"] is None
def _legacy_result(result: Dict[str, Any]) -> Dict[str, Any]:
"""The default profile's wire: today's keys plus ``reason_code: None``."""
legacy = {key: value for key, value in result.items() if key != "node"}
@@ -428,8 +374,6 @@ class _GraphResolver:
self.active: set = set()
self.metadata = metadata or {}
self.text_sources: dict = {}
# PromptManager outputs that left out a run-time part (option A).
self.omitting: set = set()
# Forced runtime selector choices (inspection); a result traced through
# one is "tainted" and never memoized, since it depends on the choice.
self.choices: dict = {}
@@ -579,12 +523,11 @@ class _GraphResolver:
if adapter.compose or adapter.storage_input:
storage_text = text
if adapter.compose:
omitted: list = []
before, before_path = self.part(
inputs.get("prepend_text", ""), depth, omitted
before, before_path, _ = self.text(
inputs.get("prepend_text", ""), "string", depth + 1
)
after, after_path = self.part(
inputs.get("append_text", ""), depth, omitted
after, after_path, _ = self.text(
inputs.get("append_text", ""), "string", depth + 1
)
self.scan(
len(before)
@@ -603,8 +546,6 @@ class _GraphResolver:
)
)
path = before_path + path + after_path
if omitted:
self.omitting.add((node_id, slot))
elif (
name in ("StringConcatenate", "Text Concatenate")
and slot == 0
@@ -674,30 +615,6 @@ class _GraphResolver:
self.tainted = outer_tainted or self.tainted
self.active.discard(key)
def part(self, value: Any, depth: int, omitted: list) -> Tuple[str, list]:
"""A PromptManager prepend or append: traced, or left out when run time.
Only with the semantics table (inspection), and only when the input is
a link to a node the table lists as ``runtime`` (such as LoraManager's
TriggerWord Toggle), which raises ``runtime-value`` at itself: that part
is not in the file, so it is left out and recorded in ``omitted``.
Anything else, including a run-time source further upstream or another
``runtime-value`` node (a JoinStringMulti list), keeps its failure.
"""
try:
text, path, _ = self.text(value, "string", depth + 1)
except _Unresolved as error:
if (
self.semantics is None
or error.code != "runtime-value"
or not _link(value)
or self.node(str(value[0]))[0] not in self.semantics.runtime
):
raise
omitted.append(str(value[0]))
return "", []
return text, path
def historical(
self, adapter: OutputAdapter, inputs: dict, depth: int
) -> Tuple[str, list, str]:
@@ -1068,15 +985,6 @@ class _GraphResolver:
if guider_id is not None:
# The conditioning path starts below the guider the sampler reads.
provenance["guider_id"] = guider_id
omitted: list = []
for step in path:
if (step["node_id"], step["output_slot"]) in self.omitting:
if step["node_id"] not in omitted:
omitted.append(step["node_id"])
if omitted:
# Through encoders, concatenations and chained PromptManagers (D125).
provenance["text_source"] = RUNTIME_PARTS_SOURCE
provenance["omitted_parts"] = omitted
return {"text": text, "storage_text": storage_text, "provenance": provenance}
def guider(self, value: Any) -> Tuple[str, dict, Mapping, str, int]:
@@ -1153,18 +1061,19 @@ class _GraphResolver:
reason_code="depth-limit",
)
own = self.sides(sampler_id)
# An unresolved stage is final, unless its positive combines texts: it
# may then add conditioning to its earlier stages' prompt (the floor).
combined = own["state"] == "unresolved" and _combined_positive(own)
if own["state"] == "unresolved" and not combined:
if own["state"] == "unresolved":
return own
name, inputs = self.node(sampler_id)
entry = self.semantics.samplers.get(name)
if entry is None or entry["strategy"] != _DETAILER:
return own
try:
upstream = self.earlier_stages(sampler_id)
upstream = self.upstream_stages(
sampler_id, inputs.get(entry["image_stage"])
)
if any(node_id in later for node_id in upstream):
raise _Unresolved("Cyclic sampler stage ancestry", "malformed")
except _Unresolved as error:
if combined:
return own
return _result(
"unresolved",
sampler_id=sampler_id,
@@ -1177,10 +1086,6 @@ class _GraphResolver:
later = later | {sampler_id}
stages = [self.stage(node_id, later) for node_id in upstream]
blocked = next((item for item in stages if item["state"] == "unresolved"), None)
if combined:
if blocked is not None:
return own
return self.added_conditioning(sampler_id, stages) or own
if blocked is not None:
# A stage that cannot be read cannot be compared: never pick one.
return _result(
@@ -1203,128 +1108,6 @@ class _GraphResolver:
reason_code="multistage-differs",
)
def earlier_stages(self, sampler_id: str) -> list:
"""The stages a sampler continues: a detailer's image ancestry, or the
one sampler its latent descends from through listed lineage nodes.
The latent walk already ran in ``sides`` (its outcome is cached); a
sampler without a latent input, or whose walk found no earlier sampler,
has none.
"""
name, inputs = self.node(sampler_id)
entry = self.semantics.samplers[name]
if entry["strategy"] == _DETAILER:
return self.upstream_stages(sampler_id, inputs.get(entry["image_stage"]))
walked = self.latent_walks.get(sampler_id)
if walked is None or walked[2] is None:
return []
return [walked[2]]
def lineage_predecessor(self, value: Any) -> Optional[str]:
"""The sampler whose output a latent input continues, or ``None``.
Follows only the table's ``lineage`` outputs (decode, image filter,
upscale, scale, encode) and Reroutes, from the input back to a listed
sampler's output slot 0. Any other node ends the lineage: its output is
not known to continue one image or latent. At most ``max_depth`` hops;
each node is read through ``node`` (one scan charge per node).
"""
seen: set = set()
while _link(value) and len(seen) < self.max_depth:
node_id, slot = str(value[0]), value[1]
if node_id in seen:
return None
seen.add(node_id)
name, inputs = self.node(node_id)
if name in self.stage_classes:
return node_id if slot == 0 else None
if name == "Reroute" and slot == 0:
value = _reroute_value(inputs)
continue
entry = self.semantics.lineage.get((name, slot))
if entry is None:
return None
value = inputs.get(entry["input"])
return None
def added_conditioning(self, sampler_id: str, stages: list) -> Optional[dict]:
"""The earlier stages' shared pair when this stage only adds to it.
This stage's positive reaches (through Reroutes and conditioning
pass-throughs) an ``additive`` composite, one of whose parts is exactly
the earlier stages' positive text, and its negative is exactly theirs:
every stage is conditioned on that prompt, the last one with more. Any
other shape, or a part that cannot be traced, is ``None``.
"""
flat: list = []
for item in stages:
flat.extend(item["candidates"] if item["state"] == "ambiguous" else [item])
base = flat[0]
if base["state"] != "resolved" or len({_outcome_key(i) for i in flat}) != 1:
return None
name, inputs = self.node(sampler_id)
entry = self.semantics.samplers[name]
if entry["strategy"] == "custom-advanced":
return None
conditioning = entry["conditioning"]
try:
found = self.composite_of(inputs.get(conditioning["positive"]))
if found is None:
return None
parts, depth, composite = found
texts = [self.text(part, "conditioning", depth)[0] for part in parts]
if base["positive"]["text"] not in texts:
return None
if conditioning["negative"] not in inputs:
return None
negative = self.side(inputs[conditioning["negative"]], sampler_id)
except _Unresolved:
return None
if negative["text"] != base["negative"]["text"]:
return None
# D126: the positive does not include what the later stage added.
provenance = dict(base["positive"]["provenance"])
provenance["text_source"] = (
RUNTIME_AND_ADDED_SOURCE
if provenance["text_source"] == RUNTIME_PARTS_SOURCE
else ADDED_CONDITIONING_SOURCE
)
provenance["added_by"] = composite
positive = {**base["positive"], "provenance": provenance}
return _result(
"resolved",
positive,
base["negative"],
base["sampler_id"],
ADDED_CONDITIONING_REASON,
)
def composite_of(self, value: Any) -> Optional[Tuple[list, int]]:
"""An ``additive`` composite a conditioning input reaches, or ``None``.
Reroutes and conditioning pass-throughs (FluxGuidance and the like) are
followed; returns the composite's part values, the depth reached and
its node ID. ``text`` follows a composite only at output slot 0, so a
stage whose positive failed as ``composite`` reached it there.
"""
depth = 0
while _link(value) and depth < self.max_depth:
node_id, slot = str(value[0]), value[1]
name, inputs = self.node(node_id)
depth += 1
if name == "Reroute" and slot == 0:
value = _reroute_value(inputs)
continue
forward = self.semantics.conditioning.get((name, slot))
if forward is not None and forward["strategy"] == "passthrough":
value = inputs.get(forward["input"])
continue
composite = self.semantics.composites.get(name)
if composite is None or composite.get("additive") is not True:
return None
return [inputs.get(part) for part in composite["inputs"]], depth, node_id
return None
@staticmethod
def walk_from(value: list) -> Tuple[deque, set]:
"""A breadth-first queue holding one start link, and its queued IDs."""
@@ -1400,10 +1183,7 @@ class _GraphResolver:
if self.semantics.samplers[self.graph[node_id]["class_type"]]["strategy"]
== _DETAILER
]
# A sampler whose latent continues another's (hires-fix, refiner) is
# judged with it in ``stage``, like a detailer's upstream samplers.
continued = self.continued_stages(sampler_ids) if len(sampler_ids) > 1 else []
if not detailers and not continued:
if not detailers:
return sampler_ids
try:
consumers = self.consumers()
@@ -1420,36 +1200,9 @@ class _GraphResolver:
absorbed.add(node_id)
except _Unresolved:
continue
for node_id in continued:
self.budget.reset(self.per_candidate_visits)
try:
if not self.reaches_output(node_id, consumers):
absorbed.add(node_id)
except _Unresolved:
continue
kept = [node_id for node_id in sampler_ids if node_id not in absorbed]
return kept or sampler_ids
def continued_stages(self, sampler_ids: list) -> list:
"""Lineage predecessors of the non-detailer candidates, in order, once.
A predecessor equal to its own sampler (a cycle) is left out; a walk
that cannot read a node names none.
"""
found: list = []
for node_id in sampler_ids:
entry = self.semantics.samplers[self.graph[node_id]["class_type"]]
if entry["strategy"] == _DETAILER or entry.get("latent") is None:
continue
try:
_, inputs = self.node(node_id)
earlier = self.lineage_predecessor(inputs.get(entry["latent"]))
except _Unresolved:
continue
if earlier is not None and earlier != node_id and earlier not in found:
found.append(earlier)
return found
def file_stages(self, sampler_ids: list) -> list:
"""Stages not proven preview-only; all of them when every one is.
@@ -1628,13 +1381,7 @@ class _GraphResolver:
reason=absent_reason,
)
negative = self.side(inputs[negative_name], sampler_id, guider_id, depth)
return _result(
"resolved",
positive,
negative,
sampler_id,
_parts_reason(positive, negative),
)
return _result("resolved", positive, negative, sampler_id)
except _Unresolved as error:
return _result(
"unresolved",
@@ -1685,7 +1432,7 @@ class _GraphResolver:
return
raise self.unsupported("Unsupported latent ancestry", node_id)
def latent_checked(self, sampler_id: str, value: Any) -> Optional[str]:
def latent_checked(self, sampler_id: str, value: Any) -> None:
"""``latent_stages`` once per sampler per resolution.
A candidate reusing another candidate's walk is charged its visit cost
@@ -1694,29 +1441,26 @@ class _GraphResolver:
out of work is never reused, since it depends on the remaining budget.
"""
if sampler_id in self.latent_walks:
error, cost, earlier = self.latent_walks[sampler_id]
error, cost = self.latent_walks[sampler_id]
if sampler_id not in self.latent_charged:
self.budget.charge(cost)
self.latent_charged.add(sampler_id)
else:
before = self.budget.allowance
earlier = None
try:
earlier = self.latent_stages(sampler_id, value)
self.latent_stages(sampler_id, value)
error = None
except _Unresolved as raised:
if raised.code == "work-limit":
raise
error = raised
cost = before - self.budget.allowance
self.latent_walks[sampler_id] = (error, cost, earlier)
self.latent_walks[sampler_id] = (error, before - self.budget.allowance)
self.latent_charged.add(sampler_id)
if error is not None:
node = dict(error.node) if error.node is not None else None
raise _Unresolved(str(error), error.code, node)
return earlier
def latent_stages(self, origin: str, value: Any) -> Optional[str]:
def latent_stages(self, origin: str, value: Any) -> None:
"""Semantics table: is there an earlier sampler in the latent ancestry?
Every link input of non-sampler nodes is walked breadth-first (encoders,
@@ -1725,14 +1469,10 @@ class _GraphResolver:
(VAEEncode, EmptyLatentBatch, WanImageToVideo slot 2, ...). An unlisted
node with sampler-shaped inputs could itself be a stage, so it is
``unsupported-node``. Budget or depth exhaustion is never single stage.
One earlier sampler that the latent continues through ``lineage``
nodes (``lineage_predecessor``) is returned, to be compared in
``stage``; any other sampler found leaves the stages unattributed.
"""
if not _link(value):
raise _Unresolved("Unknown latent ancestry", "malformed")
queue, queued = self.walk_from(value)
earlier, lineage, traced = None, None, False
while queue:
node_id, depth = queue.popleft()
if node_id == origin:
@@ -1742,17 +1482,12 @@ class _GraphResolver:
self.budget.spend()
name, inputs = self.node(node_id)
if name in self.stage_classes:
if not traced:
lineage, traced = self.lineage_predecessor(value), True
if node_id != lineage:
# Not the one continued stage: no stage's prompt is chosen.
raise _Unresolved(MULTISTAGE_UNATTRIBUTED_REASON)
earlier = node_id
continue
# ``multistage-differs`` is reserved for when stage texts are
# compared (PR3); until then no stage's prompt is chosen.
raise _Unresolved(MULTISTAGE_UNATTRIBUTED_REASON)
if self.stage_shaped(name, inputs):
raise self.unsupported("Unsupported latent ancestry", node_id)
self.enqueue(queue, queued, inputs, depth + 1)
return earlier
def anchor(self, node_id: str) -> str:
seen = set()
-299
View File
@@ -1,299 +0,0 @@
"""Capture-bound Needs review items for ambiguous files (#249 E2).
The output scan and the live monitor hand every file the ingestion resolver
finds ``ambiguous`` to :func:`record_ambiguous_file`. It captures the file
(the bounded private copy of ``utils.media_capture``, hashed while copied),
reads the metadata again from the capture, resolves it with the default
(ingestion) profile and stores only an ``ambiguous`` result, bound to the
capture's byte record. The evidence a reviewer chooses from is therefore the
evidence of exactly those bytes. An ambiguous result is stored only when it
is ``reviewable``: at least two candidates carry a resolved prompt text. With
none the file could not be traced; with exactly one, the untraced branches
may still have made the file, so that text is not certain and the file stays
unresolved (the resolver never flattens unknown semantics into a guess).
Nothing here links a file, loads a model or decodes pixels; the capture is
released before the database write.
Evidence is small on purpose (startup validates every row): the resolver's
reason and, per candidate sampler, ``{sampler_id, state, reason, positive,
negative}`` where each side is ``{text, storage_text, provenance}``. A
candidate's ``hash`` is the SHA-256 of its canonical JSON without the hash.
More than ``MAX_CANDIDATES`` candidates or evidence over the 8 MiB bound make
a ``limited`` item without candidates (a result without any candidate is not
``reviewable`` and is never stored).
"""
from __future__ import annotations
import hashlib
import json
import logging
import os
from pathlib import Path
from typing import Any
try:
from ..database.library_commands import LibraryCommandError
from ..database.review_schema import (
MAX_CANDIDATES,
MIN_CHOOSABLE,
canonical_evidence,
choosable,
)
from .media_capture import CaptureCapacity, CaptureError
from .prompt_inspector import MetadataReadError, read_inspection_file
from .prompt_pair_resolver import RESOLVER_VERSION, resolve_prompt_pair
except ImportError: # imported with the repo root on sys.path
from database.library_commands import LibraryCommandError
from database.review_schema import (
MAX_CANDIDATES,
MIN_CHOOSABLE,
canonical_evidence,
choosable,
)
from utils.media_capture import CaptureCapacity, CaptureError
from utils.prompt_inspector import MetadataReadError, read_inspection_file
from utils.prompt_pair_resolver import RESOLVER_VERSION, resolve_prompt_pair
logger = logging.getLogger("prompt_manager.review_capture")
# Capture scope label: review captures are never handed to a client.
REVIEW_SCOPE = "needs-review"
# Nested ambiguous stages are flattened at most this deep.
MAX_CANDIDATE_DEPTH = 8
def record_ambiguous_file(db, store, root, root_index: int, relative: str) -> str:
"""Persist the ambiguous evidence of one rooted file.
Returns ``recorded`` (new item with candidates), ``limited`` (new item
without candidates), ``duplicate`` (these bytes already carry this
evidence, or the file's current stat is the verified fingerprint of
bytes an item cites, so it is not even copied, and that item is still
pending), ``decided`` (as ``duplicate``, but the item carries a decision,
so the file is not in the queue), ``untraced`` (ambiguous in its captured
bytes, but not ``reviewable``: nothing to choose, nothing stored) or
``skipped`` (the file
changed, vanished, is not ambiguous in its captured bytes, or storage was
restored meanwhile).
``CaptureCapacity`` and database errors propagate to the caller, which
logs and counts them.
"""
spelled_path, queued = _queued_spelling(db, root, relative)
if queued is not None:
return _already_queued(db, *queued)
epoch = db.get_execution_epoch()
try:
# Background: the last worker stays free for an interactive request.
capture = store.capture(
root,
relative,
epoch=epoch,
scope=REVIEW_SCOPE,
root_index=root_index,
background=True,
)
except CaptureCapacity:
raise
except CaptureError as error:
logger.debug("Review capture skipped (%s): %s", error.code, error)
return "skipped"
try:
captured = captured_review(store, capture)
finally:
store.release(capture.token)
resolution = captured[1] if captured is not None else None
if resolution is None or resolution.get("state") != "ambiguous":
return "skipped"
if not reviewable(resolution):
return "untraced"
evidence, count, limited = review_evidence(resolution)
try:
inserted = db.record_review_item(
locator=capture.locator,
spelled_path=spelled_path,
sha256=capture.sha256,
size=capture.size,
mtime_ns=capture.mtime_ns,
epoch=capture.epoch,
resolver_version=RESOLVER_VERSION,
evidence=evidence,
candidate_count=count,
limited=limited,
)
except LibraryCommandError:
return "skipped"
if not inserted:
return _already_queued(db, capture.locator, capture.size, capture.mtime_ns)
return "limited" if limited else "recorded"
def _already_queued(db, locator: str, size: int, mtime_ns: int) -> str:
"""``duplicate`` while the existing item is pending, else ``decided``."""
if db.review_pending_at(locator, size, mtime_ns, RESOLVER_VERSION):
return "duplicate"
return "decided"
def _queued_spelling(db, root, relative: str) -> tuple[str, tuple | None]:
"""The file's spelling, and ``(locator, size, mtime_ns)`` when queued.
The second value is None unless the file's current stat is already
queued.
The spelling is the capture's own candidate path (the real root joined
with the listed relative path), so it folds to the capture's locator. A
file whose size and ``mtime_ns`` equal the verified fingerprint of bytes
an item already cites is not copied or hashed again (maintainer ruling:
the rescan stays cheap and the file stays in the default scope); its
items take the current spelling. A stat failure leaves the decision to
the capture.
"""
spelled = os.path.join(os.path.realpath(os.fspath(root)), *relative.split("/"))
try:
facts = os.lstat(spelled)
except OSError:
return spelled, None
locator = os.path.normcase(spelled)
if db.review_fingerprint_queued(
locator, facts.st_size, facts.st_mtime_ns, RESOLVER_VERSION
):
# Not copied again, but a case-only rename keeps the stat: record
# the current spelling so the item's target still opens.
db.refresh_review_spelling(locator, spelled)
return spelled, (locator, facts.st_size, facts.st_mtime_ns)
return spelled, None
def captured_review(store, capture) -> tuple[dict, dict] | None:
"""``(metadata, resolution)`` of the capture's own bytes, else None.
``metadata`` is what the scan hands the ingester: prompt and workflow
JSON-decoded and ``file_info`` (size, dimensions, format) from the
capture. None when the capture is gone or carries no prompt metadata.
"""
if store.acquire(capture.token) is None:
return None
try:
try:
read = read_inspection_file(Path(capture.path))
except MetadataReadError:
return None
finally:
store.release_use(capture.token)
canonical = dict(read.metadata)
for key in ("prompt", "workflow"):
if isinstance(canonical.get(key), str):
try:
canonical[key] = json.loads(canonical[key])
except (ValueError, TypeError, RecursionError):
# Malformed metadata stays as text for the resolver to reject.
pass
if not canonical.get("prompt") and not canonical.get("parameters"):
return None
info = {"size": capture.size}
if read.width is not None and read.height is not None:
info["dimensions"] = [read.width, read.height]
if read.format is not None:
info["format"] = read.format
canonical["file_info"] = info
return canonical, resolve_prompt_pair(canonical)
def reviewable(resolution: Any) -> bool:
"""True when an ambiguous result offers a choice: ``MIN_CHOOSABLE`` of its
selectable candidates (nested ambiguous stages flattened) are choosable.
Counted before the evidence is bounded, so a ``limited`` item (more than
``MAX_CANDIDATES`` candidates) with prompts to choose is still queued.
"""
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
return False
count = 0
for candidate in _flattened(resolution.get("candidates"), 0):
if choosable(candidate):
count += 1
if count >= MIN_CHOOSABLE:
return True
return False
def review_evidence(resolution: dict) -> tuple[dict, int, bool]:
"""``(evidence, candidate_count, limited)`` for one ambiguous result."""
base = {
"resolver_version": RESOLVER_VERSION,
"reason": _text(resolution.get("reason")),
}
candidates: list[dict] = []
seen: set[str] = set()
total = 0
for candidate in _flattened(resolution.get("candidates"), 0):
total += 1
try:
projected = _projection(candidate)
digest = hashlib.sha256(
canonical_evidence(projected).encode("utf-8")
).hexdigest()
except (ValueError, TypeError, KeyError):
return _limited(base, total), 0, True
if digest in seen:
total -= 1
continue
seen.add(digest)
candidates.append({**projected, "hash": digest})
if not candidates or len(candidates) > MAX_CANDIDATES:
return _limited(base, total), 0, True
evidence = {**base, "candidates": candidates}
try:
canonical_evidence(evidence)
except ValueError:
return _limited(base, total), 0, True
return evidence, len(candidates), False
def _limited(base: dict, total: int) -> dict:
return {**base, "limited": True, "candidate_total": total, "candidates": []}
def _flattened(candidates: Any, depth: int):
"""Selectable candidates; a nested ambiguous stage yields its own."""
if not isinstance(candidates, list):
return
for candidate in candidates:
if not isinstance(candidate, dict):
continue
nested = candidate.get("candidates")
if (
candidate.get("state") == "ambiguous"
and isinstance(nested, list)
and nested
and depth < MAX_CANDIDATE_DEPTH
):
yield from _flattened(nested, depth + 1)
else:
yield candidate
def _projection(candidate: dict) -> dict:
return {
"sampler_id": candidate.get("sampler_id"),
"state": _text(candidate.get("state")),
"reason": _text(candidate.get("reason")),
"positive": _side(candidate.get("positive")),
"negative": _side(candidate.get("negative")),
}
def _side(side: Any) -> dict | None:
if not isinstance(side, dict):
return None
text = side["text"]
return {
"text": text,
"storage_text": side.get("storage_text", text),
"provenance": side.get("provenance"),
}
def _text(value: Any) -> str:
return value if isinstance(value, str) else ""
-15
View File
@@ -40,21 +40,6 @@
Number.isSafeInteger(current.tag_revision) && current.tag_revision >= 0) {
this.current = { tags: current.tags.slice(), tag_revision: current.tag_revision };
}
// A busy answer names the running job that holds the slot; an
// invalid-source answer names the source it refused. Only
// well-formed values are kept.
this.running = null;
const running = current && current.running;
if (this.code === "busy" && running && typeof running.id === "string" &&
/^[A-Za-z0-9_-]{1,64}$/.test(running.id) && typeof running.title === "string" &&
running.title.length > 0 && running.title.length <= 200) {
this.running = { id: running.id, title: running.title };
}
this.source = null;
if (this.code === "invalid-source" && current && typeof current.source === "string" &&
current.source.length > 0 && current.source.length <= 4096) {
this.source = current.source;
}
}
}
+1 -1
View File
@@ -462,7 +462,7 @@
"ControlNetApplyAdvanced", "ControlNetApplySD3", "InpaintModelConditioning", "WanImageToVideo", "ConditioningZeroOut",
"CLIPTextEncode", "CLIPTextEncodeSDXL", "CLIPTextEncodeFlux", "CLIPTextEncodeSD3", "TextEncodeQwenImageEdit",
"TextEncodeQwenImageEditPlus", "TextEncodeQwenImage21", "KikoTextEncode", "StringConstant", "StringConstantMultiline",
"JoinStringMulti", "PromptSearchList", "TriggerWord Toggle (LoraManager)", "PreviewImage", "PreviewAny", "SaveImageWebsocket",
"JoinStringMulti", "PromptSearchList", "PreviewImage", "PreviewAny", "SaveImageWebsocket",
"Image Comparer (rgthree)", "DetailerForEach", "UltimateSDUpscale", "UltimateSDUpscaleNoUpscale",
"ControlNetInpaintingAliMamaApply", "LTXVConditioning", "FluxKontextMultiReferenceLatentMethod", "Text to Conditioning",
"TextInput", "DisplayText", "ClownsharKSampler_Beta", "KSampler (Efficient)", "ttN KSampler_v2",
+1 -11
View File
@@ -4,9 +4,6 @@
*
* Settings are per browser (localStorage key "gallerySettings"). Anything
* stored can be stale or hand-edited, so every value is checked before use.
* The workspace shares this object ("One display configuration", #249 System
* S3): it adds workspaceLayout and workspaceDensity, and every key either page
* does not know is kept.
*
* Loadable in the browser (window.GallerySettings) and in Node for tests.
*/
@@ -14,9 +11,6 @@
"use strict";
const VIEW_MODES = ["grid", "list"];
// The workspace Images defaults; the legacy gallery's defaultViewMode has no masonry.
const WORKSPACE_LAYOUTS = ["grid", "list", "masonry"];
const WORKSPACE_DENSITIES = ["comfortable", "large", "compact"];
const LIMIT_OPTIONS = [50, 100, 200, 500];
const MIN_COLUMNS = 2;
const MAX_COLUMNS = 12;
@@ -40,8 +34,6 @@
videoAutoplay: false,
videoMute: true,
videoLoop: true,
workspaceLayout: "grid",
workspaceDensity: "comfortable",
});
function toInt(value) {
@@ -80,8 +72,6 @@
videoAutoplay: bool(merged.videoAutoplay, DEFAULT_SETTINGS.videoAutoplay),
videoMute: bool(merged.videoMute, DEFAULT_SETTINGS.videoMute),
videoLoop: bool(merged.videoLoop, DEFAULT_SETTINGS.videoLoop),
workspaceLayout: WORKSPACE_LAYOUTS.includes(merged.workspaceLayout) ? merged.workspaceLayout : DEFAULT_SETTINGS.workspaceLayout,
workspaceDensity: WORKSPACE_DENSITIES.includes(merged.workspaceDensity) ? merged.workspaceDensity : DEFAULT_SETTINGS.workspaceDensity,
};
}
@@ -91,7 +81,7 @@
return `repeat(${n}, minmax(0, 1fr))`;
}
const api = { DEFAULT_SETTINGS, LIMIT_OPTIONS, VIEW_MODES, WORKSPACE_LAYOUTS, WORKSPACE_DENSITIES, normalizeGallerySettings, gridColumnsStyle };
const api = { DEFAULT_SETTINGS, LIMIT_OPTIONS, VIEW_MODES, normalizeGallerySettings, gridColumnsStyle };
if (typeof module !== "undefined" && module.exports) module.exports = api;
else root.GallerySettings = api;
})(typeof window !== "undefined" ? window : globalThis);
+2 -3
View File
@@ -1768,10 +1768,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
}
saveSettings() {
// The workspace shares this object (its layout and density, #249 System S3), so a Save starts from what is
// stored and keeps every key this dialog does not edit.
const settings = {
...this.getSettings(),
lazyLoading: document.getElementById('lazyLoadingToggle').checked,
imageQuality: document.getElementById('imageQualitySelect').value,
defaultViewMode: document.getElementById('defaultViewModeSelect').value,
@@ -1785,6 +1782,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
infiniteScroll: document.getElementById('infiniteScrollToggle').checked,
debugMode: document.getElementById('debugModeToggle').checked,
apiTimeout: parseInt(document.getElementById('apiTimeoutInput').value),
thumbnailsGenerated: this.getSettings().thumbnailsGenerated, // Preserve this
thumbnailPromptDismissed: this.getSettings().thumbnailPromptDismissed,
checkThumbnailsAtStartup: document.getElementById('checkThumbnailsAtStartup').checked,
// Video settings
videoAutoplay: document.getElementById('videoAutoplayToggle').checked,
+3 -17
View File
@@ -124,8 +124,8 @@
}
},
"composites": {
"ConditioningCombine": { "strategy": "combine", "inputs": ["conditioning_1", "conditioning_2"], "additive": true },
"ConditioningConcat": { "strategy": "combine", "inputs": ["conditioning_to", "conditioning_from"], "additive": true },
"ConditioningCombine": { "strategy": "combine", "inputs": ["conditioning_1", "conditioning_2"] },
"ConditioningConcat": { "strategy": "combine", "inputs": ["conditioning_to", "conditioning_from"] },
"ConditioningAverage": { "strategy": "combine", "inputs": ["conditioning_to", "conditioning_from"] }
},
"encoders": [
@@ -175,7 +175,7 @@
}
}
],
"runtime": ["PromptSearchList", "TriggerWord Toggle (LoraManager)"],
"runtime": ["PromptSearchList"],
"savers": {
"SaveImage": {
"strategy": "comfy-counter",
@@ -215,20 +215,6 @@
}
},
"non_file_outputs": ["PreviewImage", "PreviewAny", "SaveImageWebsocket", "Image Comparer (rgthree)"],
"lineage": [
{ "class": "VAEDecode", "slot": 0, "strategy": "continue", "input": "samples" },
{ "class": "VAEDecodeTiled", "slot": 0, "strategy": "continue", "input": "samples" },
{ "class": "VAEEncode", "slot": 0, "strategy": "continue", "input": "pixels" },
{ "class": "VAEEncodeTiled", "slot": 0, "strategy": "continue", "input": "pixels" },
{ "class": "ImageUpscaleWithModel", "slot": 0, "strategy": "continue", "input": "image" },
{ "class": "ImageScale", "slot": 0, "strategy": "continue", "input": "image" },
{ "class": "ImageScaleBy", "slot": 0, "strategy": "continue", "input": "image" },
{ "class": "ImageScaleToTotalPixels", "slot": 0, "strategy": "continue", "input": "image" },
{ "class": "LatentUpscale", "slot": 0, "strategy": "continue", "input": "samples" },
{ "class": "LatentUpscaleBy", "slot": 0, "strategy": "continue", "input": "samples" },
{ "class": "Image Filter", "slot": 0, "strategy": "continue", "input": "images" },
{ "class": "Image Filter", "slot": 1, "strategy": "continue", "input": "latents" }
],
"stage_inputs": {
"shaped": [
"latent_image",
+8 -22
View File
@@ -17,8 +17,6 @@ export const REPORT_DONE = 'Job report downloaded.';
export const REPORT_FAILED = 'The report could not be downloaded. Try again.';
const JOBS_REPORT_BASE = '/prompt_manager/jobs';
export const START_FAILED = 'The scan could not start. Try again.';
// A per-source scan whose sources changed since they were read (S4): the server's 409 says so.
const SOURCES_CHANGED_CODE = 'sources-changed';
const FALLBACK_NAME = 'prompts_backup.db';
const REPORT_FALLBACK_NAME = 'prompt-manager-report.json';
@@ -83,37 +81,25 @@ export function busyToast(running) {
/**
* Start the scan job over `files` ('new': new and unresolved files, or 'all') and open its view; a refused or failed
* start says why and stays on the page. `roots` (System › Sources, S4) limits it to those effective source indexes, and
* `paths` names the public path read for each: when the sources changed since (409 `sources-changed`) nothing starts,
* the toast says the server's reason and `onStale` (which every caller with `roots` passes) reads them again.
* start says why and stays on the page.
*/
export async function startMetadataScan({ store, scope, files = 'new', roots = null, paths = null, onStale,
notify = showWorkspaceToast, navigate = navigateTo }) {
export async function startMetadataScan({ store, scope, files = 'new', notify = showWorkspaceToast, navigate = navigateTo }) {
try {
const result = await store.start('scan', roots ? { scope: files, roots, paths } : { scope: files });
const result = await store.start('scan', { scope: files });
if (scope.signal.aborted) return;
if (result.busy !== undefined) {
notify(busyToast(result.busy));
return;
}
navigate(`#/activity/${result.job.id}`);
} catch (error) {
if (scope.signal.aborted) return;
if (error?.code === SOURCES_CHANGED_CODE) {
notify(error.message);
onStale();
return;
}
notify(START_FAILED);
} catch {
if (!scope.signal.aborted) notify(START_FAILED);
}
}
/**
* Open "Review metadata scan" for `sources` output folders; with `roots` and `paths` (one source's index and public
* path, System › Sources) the scan reads only those, and `onStale` runs when they changed since they were read.
*/
export function openMetadataScan({ scope, store, sources, roots = null, paths = null, onStale, trigger = null }) {
/** Open "Review metadata scan" for `sources` output folders. */
export function openMetadataScan({ scope, store, sources, trigger = null }) {
openScanReviewDialog({ scope, sources, trigger,
onBackup: () => downloadDatabaseBackup(),
onStart: ({ scope: files }) => startMetadataScan({ store, scope, files, roots, paths, onStale }) });
onStart: ({ scope: files }) => startMetadataScan({ store, scope, files }) });
}
+3 -5
View File
@@ -148,20 +148,18 @@ function itemFor(job, scope) {
chip.textContent = state;
detail.textContent = jobDetail(next);
const running = next.state === 'running';
// parentNode, not isConnected: on the first render the item is not in the page yet, so a job that had already
// ended when Activity opened kept its Cancel job (and a running one's progress moved after its actions).
if (running) {
progress.value = next.progress;
progress.textContent = `${next.progress}%`;
progress.setAttribute('aria-label', `${next.title} progress`);
if (!progress.parentNode) item.insertBefore(progress, actions.parentNode ? actions : null);
if (!progress.isConnected) item.insertBefore(progress, actions.isConnected ? actions : null);
} else {
progress.remove();
}
// Truth rule: a job that cannot be cancelled (a held legacy-page job) shows no Cancel job.
if (running && next.cancellable) {
if (!actions.parentNode) item.append(actions);
} else if (actions.parentNode) {
if (!actions.isConnected) item.append(actions);
} else if (actions.isConnected) {
// A focused Cancel job that goes away hands its focus to the item's title link, never to the page body.
if (actions.contains(document.activeElement)) link.focus();
actions.remove();

Some files were not shown because too many files have changed in this diff Show More