Compare commits

...
Author SHA1 Message Date
Vito Sansevero dfbf5dfd05 feat(library): Filters shows its active count; final review follow-ups
- D44: the shared toolbar's Filters button reads "Filters · N" while N
  "Refine this view" filters are active (Library: Tags; Images: Source,
  Folder, Media, Tags), so a Browse-set Tags filter is visible on the
  shelf. The text is the accessible name. Unit and browser tests cover
  Browse -> "Filters · 1" -> Clear -> "Filters" and the Images counts.
- Renumber the alphabetical tag chips deviation D42 -> D43 (D42 is #264's).
- The caption count is a role=status region written only when it changes
  and kept shown while the shelf reloads.
- usedLabel names the year for a use in another calendar year (D23).
- prompt-blocks: same `title` option API as #264 (`subject` suffix); the
  region-name helper's parameter no longer shadows it.
- Scope library.css .row-actions rules to the history panels; fix stale
  CSS and spec comments.
- Retry re-reading a failed category list and the no-op Apply/search
  paths are covered by a browser test.
- TESTING.md: D23 amendments (year, button vs link trade-off), D44, the
  legacy Library link wording.
2026-10-20 22:57:54 -07:00
Vito Sansevero b2ec01734e test(workspace): reach the legacy Library from Overview, not the shelf
The shelf no longer links the legacy Library (#249 E3). The shell
destinations test now expects no legacy link on Prompt library and
asserts Overview's "Open existing prompt library" card still links
/prompt_manager/admin and that the page loads.
2026-10-20 22:36:58 -07:00
Vito Sansevero e45f67dbf5 test(library): concept comparison for the shelf
library-concept.spec.mjs compares the Prompt library shelf with the
immutable concept on the concept-shaped fixture: heading, toolbar,
caption, grid and the six cards (art, title row, prompt block, meta,
tags, footer) at 1600x1100 and 390x844 in both themes and motion
settings, the 3-column grid at 1800x1100, and the behaviors in both
documents (title and Explore history open history, the footer send
carries the positive only, copy toasts "Prompt copied.", the "neon"
search and the Landscapes category). A mutation test proves the
comparison notices a footer padding change and a removed variants chip.

mapLibraryConcept applies only recorded deviations: D14, D16, D23 (title
min-width, singular variant chip, spoken rating, the two-line clamp the
concept's samples exceed at 390px), D35, D36, D40 and the new D42 (tag
chips in the store's alphabetical order). Each re-laid-out part is
checked against the concept page with that deviation applied at runtime.

Corrections the comparison demonstrated:
- the fixture creates the concept's prompts last to first, so Recently
  added lists them in the concept's order;
- the shelf no longer ends with the legacy "Current prompt library"
  link; Overview's legacy cards keep it, as with "Current gallery".

The shared inventory gains the Library section, and reflow/withoutItems
move into concept-inventory.mjs for both comparison specs.
2026-10-20 21:29:20 -07:00
Vito Sansevero ae88f9cd6e feat(library): concept prompt cards with real runs and last use
The Prompt library shelf renders the concept's article.prompt-card:
.card-art (up to two previews), .card-title with a title button and the
category chip, the positive prompt block, .card-meta ("− N negative
variants", "N runs", "N images"), .tags, and .card-footer with the rating
and last-use line and the positive send plus Explore history buttons.

D23: "N runs" is the parent's run_count; the footer reads Used today /
yesterday / N days ago / <Mon D> in the viewer's local calendar days, or
"Not used yet"; stars carry a visually hidden "Rated N out of 5" and an
unrated parent shows no stars. The positive is clamped to two lines while
copy and send use the full text. Long titles wrap inside the title row
and the category chip never breaks mid-word. Card actions are named after
the prompt ("Copy positive prompt: <title>") through a new title option on
the shared prompt blocks. The production .library-card CSS is removed and
the row estimates follow the clamped card height.

The shelf error now reads "Prompt library could not be loaded. Retry."
(Refresh is gone). TESTING.md records D23, the production-only shelf copy
(D40) and the concept search placeholder kept on the Library (D41).
2026-10-20 21:04:22 -07:00
Vito Sansevero c0ba8ac5b9 feat(library): concept toolbar and caption
The Prompt library shelf uses the shared concept toolbar (search, category,
sort, Filters): sorts are Recently added / Name A-Z / Highest rated, then
Oldest first; the Tags filter moves to "Refine this view" (D14) and Refresh
library is removed (D15), with Retry kept and re-reading failed categories.
The caption is the concept's .section-caption, "N positive prompts · M
negative variants" over the filtered set from variant_total.

Library-embedded media cards keep the favorite star working: the Library
gallery read already reports favorite (one batched lookup per page, now
covered by a test), and star answers are delivered to every mounted embedded
gallery, so a star toggles, repaints and survives a reload. The checkbox and
selection bar stay Images-only.
2026-10-20 20:32:10 -07:00
Vito 00bda8f442 feat(inspector): concept prompt blocks, side panel and viewer details (#260)
* feat(workspace): concept prompt blocks

Shared prompt blocks (Images panel, viewer details, Library cards and
saved texts, recorded runs) now render the approved mock's markup
(app.js promptBlock a298-302, concept.css s550-588), #249 D22:

- section.prompt-section.<role>[aria-label="<Role> prompt"] >
  .prompt-heading (span.role-label "+ Positive prompt" / "- Negative
  prompt"; .prompt-actions: copy button.icon-btn "Copy <role> prompt",
  then send button.send.icon-btn "Send <role> prompt to ComfyUI") +
  p.prompt-text.
- No per-block source line; a file's evidence moves to the panel badge
  (D2) and "File & provenance" (D3). Recorded runs keep every field's
  source in "Frozen field provenance".
- Empty text reads "No negative prompt (explicitly empty)" and still
  sends an explicit clear. Absent, ambiguous and unresolved roles render
  p.prompt-text.muted with the existing status strings and no actions;
  unknown role names render nothing and cannot be sent or copied.
- Copy shows the shared toast "Prompt copied."; without a clipboard it
  opens the concept's "Copy prompt" select-to-copy dialog.
- The data-send-role / data-copy-role contract is unchanged.

The D22 interim overrides (26px actions, prompt-text margins, copy
status, source line) are retired so the concept's rules apply; the
action buttons measure the mock's 31x31. pre.prompt-text styling stays
only for the Library editor's Archive dialog and frozen field
provenance, which keep their own markup until their page rebuilds.

Test selector mapping:
  .prompt-block / .prompt-positive  -> .prompt-section(.positive)
  .prompt-block-heading h3          -> .prompt-heading span.role-label
  region pre / pre.prompt-text      -> p.prompt-text
  "Empty negative prompt"           -> "No negative prompt (explicitly empty)"
  "Send <role> prompt"              -> "Send <role> prompt to ComfyUI"
  .prompt-source / copy status line -> removed (toast or Copy prompt dialog)
  copy button 30x26                 -> 31x31 (measured in the mock)

* feat(inspector): concept side panel

The Images side panel is the concept's aside#image-inspector.detail-panel in the
order of app.js imageDetails: plain cover, heading with the format chip, filename,
the D21 evidence badge and category chip, prompt blocks, tag chips, the six facts
and the footer (Send to ComfyUI pair, Prompt history). Provenance, raw records and
downloads stay in the viewer's full details. Positive-scope tag chips set the
Images Tags filter to that tag.

* fix(inspector): mark the side panel's overlay rules with their D2 decision

The overlay lint (tests/vendor/concept-css.test.mjs) requires a /* D<n> */
marker on every workspace.css rule that overrides a concept selector; the
three .detail-panel rules from D2 lacked one.

* feat(inspector): full details in the viewer

The viewer's #viewer-details renders the concept's imageDetails(i, true):
the side panel content without the cover in a class-less wrapper, then
"File & provenance" and "Workflow & raw metadata" (pre[tabindex=0] > code
and the two downloads with their unavailable reasons), then the pair send
("Send to ComfyUI", labelled "Send prompt pair to ComfyUI: <title>") and
Prompt history buttons. One renderer now serves both variants.

Provenance wording (maintainer, 2026-10-20): the six per-fact
"<Fact>: Configuration evidence · <source>" lines and FACT_SOURCES are
gone; the concept's "Prompt evidence: <badge>" line carries the D21 badge
in lower case and is left out without a badge. The honest facts stay
(current-file note, Source, Path, Size, Dimensions, Format, Media type,
Modified, occurrence, association, Download original). An unverified
format reads "PNG extension", no stray leading dot.

inspector.css and images.css are deleted; only rules with a live
consumer moved to the overlay under #249 D3 (inspector wrapping, the
inert aria-disabled download, the narrow Images container, the Library
and recorded-run prompt blocks, the Archive negative's prompt-block, the
prompt sender) plus a D20 wrap rule for the viewer's buttons at 200% zoom.

D1 review follow-ups: library.css no longer restyles recorded-run
p.prompt-text.muted (m2); the copy fallback has a seam and unit tests for
a missing clipboard and a refused write (m3); a stale test title is
renamed (m4); the fallback trigger falls back to the same-role button
when the clicked one was detached by a re-render (m7); TESTING.md records
the D20 and D22 deviations (m1).

Selector mapping in tests: .file-inspector-inner -> .panel-heading,
.generation-facts -> .meta-grid (first), .file-facts -> details .meta-grid,
.file-name -> #viewer-details > div > p.small, .file-inspector-footer ->
#viewer-details, viewer "Send prompt pair" -> /^Send prompt pair to ComfyUI: /.

* test(comfyui): real-host specs read the concept prompt blocks

library-sender: an explicitly empty saved negative now reads the concept's
"No negative prompt (explicitly empty)" (D22). The prompt-sender spec's
panel and viewer selectors (complementary "Selected image details", the
card image opening the viewer, .meta-grid facts, the labelled pair send)
moved with the D3 commit.

* test(comfyui): the frozen run's prompt regions read the concept prompt text

* fix(inspector): coarse viewer-detail targets and Prompt history on the shown route

- Prompt history closes the viewer and, when its target is the route
  already shown, re-renders it as the concept's go() does (navigateTo
  plus a router refresh event), instead of doing nothing.
- The viewer details' disclosure summaries and "Download original" keep
  40px targets on coarse pointers (#249 D34), as TESTING.md promises.
- Drop the dead `.prompt-positive` rules and merge the two
  `pre.prompt-text` rules.
- TESTING.md records the side panel's D20/D21 deviations.
2026-10-20 20:15:21 -07:00
Vito 728db36ac3 ci: fix the frontend coverage timeout and the WebKit Load more race (#262)
* ci: give the frontend coverage job a 40-minute limit with measured headroom

The job runs every page workflow twice (chromium, then touch) on one
worker with V8 coverage, so it takes about twice a single-browser job.
Recent runs took 18m26s-24m57s and PR head 6359999 was cancelled at the
25-minute limit while still passing. Coverage, floors and commands are
unchanged.

* test(images): dispatch the production Load more press so the observer cannot pre-empt it

A pointer click scrolls the continuation row into view, which lets the
page's own infinite-scroll observer append the final batch and hide the
button before the click lands (WebKit, run 39512528039). Dispatch the
press as the concept side already does, after asserting the button is
visible and enabled; the count, card, toast and loading-row assertions
are unchanged.
2026-10-20 19:47:37 -07:00
Vito 5ded3cfd39 test(history): prove bounded close with events, not wall-clock bounds (#261)
The real-clock subtest of the bounded-close test asserted close(0.02)
returned in under 0.2 s; a slow macOS runner took 0.26 s. The failed
subtest then skipped its release and join, so its worker stayed blocked
holding a charge on the process-wide SHARED_CONTEXT_BUDGET, and the next
subtest saw 621 != 0. Not a production race: close(2) only returns True
once the worker has exited and released its leases.

Prove boundedness with events instead: close() must return False while
the in-flight write is still blocked (it has not left its fake write).
Keep the exact upper bound only under the manual clock, where it is
deterministic. Release and join each subtest's worker in a finally so a
failure cannot leak a charge into the next subtest. The commit-gate test
gets the same treatment: submit() and close(0) must return before the
blocked invalidation and commit finish, instead of a 0.1 s bound.
2026-10-20 18:01:34 -07:00
Vito 2b8126b7be feat(images): card selection checkbox and selection bar (Set category, Export metadata) (#259)
* feat(images): card selection checkbox and the concept selection bar

Every Images card's overlay starts with the concept's "Select <filename>"
checkbox before the star. Checked image keys form the page's selection,
kept across Load more, infinite scroll and bounded re-mounting and cleared
whenever the collection is replaced. The #bulkbar between the filter row
and the gallery shows "N selected" with Set category, Export metadata and
Clear (the first two are wired by the next tasks); Clear empties it.

The interim D8 star rule is removed, and the concept comparison now
measures the overlay with its checkbox and the bar with one card checked.

* feat(images): export the selected images' metadata as one JSON bundle

The selection bar's Export metadata reads each checked image's inspection,
at most four at a time, and downloads prompt-manager-images-metadata.json:
format, version 1, exported_at and, in listing order, the same envelope the
inspector's "Download metadata JSON" saves (one shared builder,
metadataEnvelope). The button is aria-disabled and ignores a second press
while it runs, the selection is kept, and the toast reports the exported
count, the images that could not be read, or that none could (no download).

Also: the bar and its count are built only on the Images page, the
Library-embedded gallery is asserted to have no checkbox or bar, a sort
change is asserted to clear the selection, the checkbox label test asserts
the toggle and an unchanged selection and inspector, and TESTING.md records
that a return to Images keeps the selection while a reload starts empty.

* feat(images): set the category of the selected images' linked prompts

The selection bar's Set category opens the concept's simpleInput dialog
("Categorize selected images", one Category field, Cancel / Apply) and
posts the selection's distinct linked positive ids to
/prompt_manager/bulk/category. Images have no category of their own; the
listing's category filter matches the linked positive's, so that is what
changes. With no linked image checked nothing is posted and a toast says
so; on success the toast counts the prompts and any skipped images, then
the categories and the listing reload (clearing the selection); a failure
keeps the selection. The button ignores a second press while it runs.

The bulk category route now rejects anything but a non-empty list of at
most 1000 positive integer ids (deduplicated in order) and validates the
category with validate_category; bad JSON or a non-object body is a 400.
The response shape and the legacy admin request are unchanged.

* test(images): zoomed card keyboard order follows the concept's checkbox before the star

The card's Tab order is image, Select checkbox, star, Inspect (concept app.js:398), both ways; List hides the overlays as the concept does, so there the image is followed by Inspect.

* fix(api): answer 400 for bulk category ids beyond SQLite's INTEGER range

An id of 2**63 or more passed the parser and made sqlite3 raise OverflowError, which the route turned into a 500.

* refactor(images): move the selection bar into images-selection.mjs and harden its actions

- images.mjs is back under 800 lines; the bar, its checked set, Export metadata and Set category live in images-selection.mjs.
- Busy flags release in finally; the dead guards are gone and the category targets are taken when the modal dialog opens.
- The category write no longer carries the page signal; leaving Images mid-write drops the saved visit so the return reloads.
- The categories and the listing reload together after Set category.
- The Category field stops at the server's 100 characters; the server's updated_count is a tested pure helper.
- New browser tests: leaving mid-export and mid-write, Blob URL revoke, updated_count 0 and a 400 rejection.
- Additive coverage baseline entries for images-category, images-export and images-selection.
- TESTING.md D8 records the dialog deviations and these rulings.
2026-10-20 16:04:15 -07:00
Vito 86d82186ee feat(images): concept Images page — toolbar, cards, star, viewer, compared against the mock (#258)
* feat(workspace): shared concept toolbar and dialogs

Add the concept's shared page toolbar (search.toolbar with the search
field, category filter, sort and Filters) and its dialogs in one modal
shell: "Refine this view" (Source / Folder / Media / Tags, Apply filters),
"Import & inspect" with only the real operations (Open inspector, Rescan),
and the "Copy prompt" select-to-copy fallback. The sort lists the
concept's three options first (D14); there is no Save view (D16). Every
dialog returns focus to its trigger on close or Escape.

Refs #249

* fix(workspace): bind shared dialogs to the page and its route

- Drop the invented Refine note; every field is real (D14).
- Ignore a late close event once the shell was reopened or replaced, so
  an action that opens the next dialog keeps it open.
- Dialog openers require the page scope; disposing it closes and removes
  the shell unless a newer page's dialog is open. Every route change
  closes the shared dialog before the next page mounts.
- Clear filters also reports category '' so the page clears the toolbar
  category, as the concept does.
- The toolbar requires its scope and falls back to the first sort for an
  unknown value; dialog callbacks are validated up front.

Refs #249

* feat(images): concept toolbar, filter row and caption

The Images page now renders the approved mock's heading, shared toolbar,
filter row, section caption and infinite continuation (#249, Task C2).

- Heading: "Your generations, their prompts, and the next idea." with
  Import (upload icon) before New prompt (primary).
- Toolbar (shared mountToolbar): search over filenames, linked prompt
  text, titles and tags; category (the Library's categories); sort
  Recently added / Name A–Z / Highest rated, then Oldest first and
  Name Z–A (D14).
- Filter row: Grid/List/Masonry segmented control, Thumbnail size,
  the scope chip ("All output folders" or "<source> · <folder>") and
  "Group by positive prompt →".
- Caption: "N of M images · K positive prompts" from positive_count; a
  plain cursor page keeps the first page's count and an unknown count
  hides the "· K positive prompts" part. The date span shows
  "<Today|Yesterday|weekday> · <Month D>" of the newest loaded result
  under a date sort and is hidden otherwise (D17).
- Continuation: "N of M images · more load as you scroll" with Load
  more, the concept's end copy, Retry for a failed page, and the
  concept's "K more images loaded. N of M." toast.
- The listing sends category, tags and sort=rating (PR B params).

Old control -> new control:
- "Search filenames" field -> toolbar search "Search workspace"
  (placeholder "Search prompts, tags, filenames…"), also matching
  prompt text and tags
- Source select -> Filters -> "Refine this view" Source
- Folder select -> Refine Folder (all sources' folders and ancestors)
- Media select -> Refine Media
- (new) Tags -> Refine Tags; Category -> toolbar category select
- Sort select (Newest/Oldest/Filename A–Z/Z–A) -> toolbar sort
  (Recently added/Oldest first/Name A–Z/Name Z–A, plus Highest rated)
- Layout buttons -> .segmented Grid/List/Masonry in the filter row
- Thumbnail size field -> label.small + select#image-size
- "Group by positive prompt" link -> button.chip "Group by positive
  prompt →" (#/library)
- Refresh -> Import -> "Scan an output folder" -> Rescan (D15)
- "Current gallery" link -> removed; Overview's tool cards link the
  legacy gallery (D29)
- "All images" h2 + count -> .section-caption #result-count
- .image-continuation -> .infinite-continuation (Retry kept)

Legacy behavior kept: video items, keyboard grid navigation, scroll and
filter restoration, bounded DOM, continuation retry, source-change
recovery (now disabling the toolbar, Filters and Import). The Filters
and Import dialogs close on route change and browser Back.

The shared toolbar (C1) called .some() on select.options, which is an
HTMLOptionsCollection in a browser (only the unit tests' fake DOM gave
an array); setValue and setCategories now spread it first. The layout
segmented control may wrap at extreme zoom (D31) instead of scrolling
the 390px page sideways.

The retired .image-filters, .image-view-controls, .image-toolbar,
.image-layouts, .image-collection, .image-continuation and duplicated
.image-workspace grid rules are gone from images.css.

* test(workspace): model select.options as a browser collection

The toolbar units' fake DOM returned select.options as an Array, so the
toolbar's .some() call passed there while it threw in every browser
(fixed in 12228e2). options is now an indexed, iterable, frozen
HTMLOptionsCollection-like object without Array methods; against the C1
toolbar three units fail with "sort.options.some is not a function".
The units also assert that an unknown sort falls back to the first
option and that a still-listed category survives a category refresh.

* fix(images): concept Load more focus, empty state and review rulings

Review of #249 C2:
- Load more and Retry focus the first appended image, as the concept
  does (app.js loadMoreImages). Focus moves without scrolling so the
  visible cards keep their place (bounded-DOM anchor), and only while
  focus is still on the pressed control: a viewer or a field the user
  moved to keeps it. When a last page adds nothing and the button hides,
  focus goes to the last image, never to the page; a failed page focuses
  Retry.
- The concept's empty state beside the grid: .empty "No matching
  images. Try another search or category."
- D17: the caption date shows only under Newest first (", YYYY" outside
  the current year) and is hidden under Oldest first and the name and
  rating sorts.
- en-US number grouping in the embedded count too.
- Refine "Folder" defaults to the concept's "All output folders". It is
  not narrowed by Source: the concept has no Source field, and the list
  holds every source's folders, so any pair can be chosen in one dialog.
- The shared toast moves to toast.mjs; library-editor re-exports it.
- The D32 overlay (extreme zoom) wraps the layout segmented control and
  the continuation row instead of pushing them out of the 390px column.
- The legacy gallery checks moved to Overview's "Open existing image
  gallery" card (D29).

Mapping additions: grid with no results -> .empty concept copy beside
the grid; Load more press -> focus the first new image (concept).

* feat(images): concept image cards; image opens the viewer

Cards are the concept media card: .media-art > button.image-open with the
image or video, and .media-info with the D10 title (linked title only when
the link is confirmed, else the filename; title attribute = path), the D11
subtitle "W x H · EXT · Seed N" (seed shown as the listing's string) and an
Inspect info button. The byte size and the Preview button are gone.

Activating the image (click, Enter, Space) opens the viewer and focus
returns to it on Escape; Info selects the image into the panel (D19).
Arrow, Home and End navigation, bounded rendering and scroll restoration
are kept.

Columns and gap now come from the concept grid's computed style
(gridMetrics) instead of a production breakpoint table, so the concept
breakpoints apply; a 120px minimum card bounds extreme zoom (D3, D31).
images.css card rules are retired into marked overlay rules. The scope
chip reads "All output folders" when no folder is chosen (D14).

* fix(images): pair-linked cards take the positive's title; concept Info, alt and size

- Pair links title the card, inspector and viewer from the linked positive's
  Library title or confirmed first line, never the pair title (the negative
  variant's name). The card now equals the concept card, title included.
- Info drops aria-pressed; the card's .selected carries selection, as in the
  concept.
- Card images take the card title as alt and the listing's width/height.
- One shared minimum card width for layout and the narrow-card class.
- A debounced search that lands after a sort change no longer reloads page 1.
- 200% zoom test also runs at an 800px viewport and device scale 2.

Refs #249

* feat(images): concept hover zoom

* feat(images): favorite images from the card

* feat(images): concept viewer shell

* fix(images): one favorite request per file across leaving and returning

The pending set and the receiver of each answer now live at module level
beside the saved visit. A press on a returned Images page waits for the
request the earlier page sent, and its success or rollback reaches the
card on screen, or the saved visit when no Images page is mounted.

* fix(images): mark a star busy while its favorite request is pending

The star carries aria-disabled="true" until the answer arrives, also on a
card rendered after returning to Images, so the ignored second press is
announced as unavailable instead of doing nothing silently.

* perf(images): repaint one card on a favorite toggle instead of the collection

stream.updateItem tells onChange which item it patched, and the Images
page then skips the full layout pass (computed style, grid metrics and
layout over every loaded item) because no card changed size or place.

* docs(images): say the lone-star overlay rule is interim until the checkbox PR

* test(images): concept comparison for the Images page

images-concept.spec.mjs compares the production Images page with the
immutable concept on the concept-shaped fixture: heading, toolbar, filter
row, caption, grid, the first six cards, continuation and panel box in
Grid, List and Masonry, at 1600x1100 and 390x844, dark and light, normal
and reduced motion; plus hover zoom, the viewer title, focus restore,
Info selection, search and Load more with its toast. mapImagesConcept
applies only recorded deviations (D3, D8 and D16 follow-up PRs, D9, D10,
D14, D15, D17, D29, D31, D32); removed row items flow again from the
concept's own boxes. A mutation check proves the comparison fails on a
.media-info padding change and on removed Inspect buttons.

Corrections the comparison demonstrated:
- the grid and continuation carry the concept's ids (image-grid,
  gallery-continuation);
- bounded rendering reserves the card footer at its fractional height:
  offsetHeight rounded a wrapped 67.5px row up, so at 390px every card
  was a pixel taller and the grid 6px taller than the concept.

Two mismatches wait for a ruling and are pinned on both sides: the
category order (concept prompt order, production alphabetical) and the
singular count line ("1 positive prompt").

* fix(images): ignore sub-pixel noise when the measured card footer grows

Under page zoom the fractional footer height (rect / scale) can carry float
noise above the stored value, which relaid the grid and scrolled for nothing.

* test(images): map D35-D37 and widen the Images concept comparison

- D35 (singular counts) and D36 (alphabetical categories) are mapped
  deviations instead of pinned values; a search for exactly one image
  checks both singular nouns, and a one-image Load more checks the toast
  and continuation.
- Grid, cards and panel at 1200x900 (2 columns, 275px panel) and
  1800x1100 (4 columns); Large and Compact join the layout loop.
- Masonry tops and heights compare unrounded within one layout unit per
  card above in the column (D3), not as rounded integers.
- The Save view and sample-tag re-layout is checked against the concept
  page with those items removed at runtime.
- The Load more row is compared before the press in both documents (D37).
- Two seedless fixture images (seedless_images on the concept reset)
  prove every row reserves the tallest footer (D3).
- D10 paths come from the fixture manifest; the D8 star margin is mapped;
  list-row derivations cite D3 and D9.

* fix(images): search never matches a pair's own variant title

Search covers the filename, the linked positive's text and Library title,
and its tags (#249 D12). The pair title names a negative variant, so a
variant name alone no longer admits the positive's images.

* fix(viewer): keep the Media controls group and drop a stale position description

The concept viewer toolbar is again a named group (#249 D34 #8), and
Previous/Next are described by the position line only while a single
item is loaded.

* docs(history): drop the remaining-polish notes the concept viewer and cards closed

The viewer now takes the concept stage and toolbar, and Images cards open
the viewer, select into the panel and favorite. Only the select checkbox
(#249 D8 follow-up) and the tag button (#249 D9) stay outside #209.

* test(images): the singular copy when scrolling loads one image; countCopy in the shared inventory helper

* refactor(images): retire the replaced images.css rules and lint the kept page sheets

The unused .image-facts rules go. The three prompt-block overrides on
concept selectors are marked #249 D22 interim until PR D deletes the
file, and every kept page stylesheet now passes the overlay lint.

* test(images): the extreme-zoom anchor probe reads the fractional footer measurement

Since the concept comparison (#249 C7) the grid measures footers with
getBoundingClientRect, so the probe that hooked offsetHeight never fired.

* test(images): cover the masonry fallback when a finite size underflows its ratio

* docs(images): concept Images page and recorded deviations

TESTING.md documents the Images page controls, the recorded #249
deviations and how the earlier workspace filters moved, and notes that
every kept page stylesheet is linted. The frontend coverage baseline
adds the toast, toolbar and workspace dialog modules.

* fix(images): show that Import → Rescan runs, and how it ended

The old Refresh button disabled itself while /gallery/rescan ran; Import →
Rescan closed its dialog and showed nothing. Now a toast says the rescan
started ("Rescanning output folders…") and how it ended ("Rescan finished.
N files in M output folders.", from the scan's own total and roots, or
"Rescan could not finish. Try again."). A reopened Import dialog shows
Rescan as "Rescanning…" and aria-disabled, and ignores a press on it.

Failing before: the held-rescan browser test saw no toast and no busy
state; the dialog unit test found no "Rescanning…" button.

* fix(images): rewrite the live status only when its text changes

A star press republishes the stream, and the textContent setter replaced
the continuation status, the count and the caption date with the same
text, so a polite role="status" region could announce it again. Text and
the role attribute are now written only when they change.

Failing before: a MutationObserver saw 10 records on those nodes after a
favorite toggle; now none.

* fix(images): Refine's Folder list follows the chosen Source

The workspace subfolder discovery returned only the union of every
source's folders, so "Source: temp" could be paired with a folder that
exists only under output and silently match nothing. In workspace mode
each root now lists its own folders, and "Refine this view" offers only
the chosen Source's folders: changing Source keeps a folder it also has
and otherwise returns Folder to "All output folders", as the old field
row did when it reloaded a Source's folders.

Failing before: the API test found no per-root folders (KeyError), the
dialog unit test and the browser test still offered output-only folders
under the input source.

* fix(images): an unchanged Refine apply keeps the loaded page

"Apply filters" (and "Clear filters" with nothing set) reset the stream,
refetched the first page and dropped the reader's scroll position even
when no filter changed. Like the toolbar's equality guard, an unchanged
refinement now does nothing.

Failing before: the browser test counted one listing fetch after an
unchanged Apply; now none, and the loaded cards stay in place.

* fix(images): one discovery alert when sources and categories both fail

With the server unreachable on load, the sources and the categories
status paragraphs each fired role="alert" back to back. The page now
keeps one alert, written once both first loads settled, naming whatever
failed ("Sources, folders and categories could not be loaded. Rescan
from Import to retry." when both did).

Failing before: getByRole('alert') resolved to two elements.

* fix(images): sort toolbar categories alphabetically on the client

D36 promises alphabetical real categories, which held only while the
server sorted them and a restored category was among its list. The page
now sorts them case-insensitively itself, a restored category included.

Failing before: the toolbar listed a server's [zeta, Alpha, beta] in
that order.

* refactor(db): drop the unused pair title from the link summary query

_link_summary has not read the pair's own title since the card took the
linked positive's Library title (D10), so the query no longer selects
it; the row indexes and the stale comment are updated. All values stay
bound.

Failing before: the summary SQL still selected pp.title.

* test(images): end negative checks on real signals, not fixed sleeps

The "no second request" checks waited a fixed 150 ms or 1000 ms and the
hover probe 450 ms, which can pass vacuously on a slow Firefox or WebKit
run. They now read the fetches the page's code started (the API client
calls fetch in the same task as the press), wait for a timer of the
toolbar's own debounce delay set after the debounced report, and wait
for the hover transition to finish.

Checked by mutation: without the favorite pending guard or the toolbar
equality guard, the rewritten checks fail.

* docs(testing): record the Rescan feedback, Source-bound folders and category sort

The Images contract now names the Rescan toasts and busy state (D15),
the Folder list that follows Source and the no-op unchanged refinement
(D14), and the page-side category sort (D36).

* test(viewer): escape every regex metacharacter in the filename match

* fix(viewer): title a linked card at once instead of waiting for its inspection

On a slow runner the viewer showed only the filename until the inspection
answered, while the concept titles it immediately. The card's linked
title is already loaded, so the header uses it on open.

The favorite test now waits for the first request to settle before the
Space press, which the one-request-per-file guard otherwise ignores.

* fix(images): cap the search at the listing API's 500 characters

A longer paste was sent as is, the API rejected it and the collection
showed "Images could not be loaded". The legacy search capped it.
2026-10-20 11:31:53 -07:00
Vito 6c82098207 feat(images): data for the concept cards — linked titles, seed, search and filters, counts, favorites (#257)
* feat(images): link summary on workspace image items

* fix(images): carry title source and cut first-line titles like the Library

Summaries gain title_source (stored or text). The inspector always shows a
stored title and confirms only a first-line title against the file, so a
stored title equal to the first line is no longer withheld. A link-title read
failure no longer marks a readable link unavailable. First-line titles over
80 characters end with an ellipsis, as the Library's title() does. The card
gate for unconfirmed first-line titles is pinned as an expected failure for B2.

* feat(images): unambiguous seed on workspace items

Workspace items gain `seed`, read with the shared generation-parameter
parser from the prompt/workflow records the header open already loaded
(each at most 2 MiB; no extra file reads, videos give None). A
placeholder or non-integer seed gives None. JSON integers beyond 2**53
are recovered exactly, so the card prints the digits the inspector shows.

The same parse confirms first-line card titles: a `text` title shows only
when the file's parsed positive gives the same title. One helper applies
that rule on the card and in the inspector, which now reads the positive
from its own inspection records with the same parser.

* fix(images): a deeply nested record no longer fails the workspace listing

The Python parser recurses over nested arrays where JavaScript String()
does not, so a crafted 2 KB PNG turned every workspace page holding it
(and its inspect route) into a 500. Card facts now treat RecursionError
like any record the parser cannot use: no seed and no confirmed title.

Also: Library previews skip the card parse they never show; the parse
runs after the file handle closes; the exact-seed read uses the one JSON
reader (parse_js_json(..., exact_integers=True)) and the public
number_string; record size is decided from the text length unless only
the UTF-8 byte count can tell.

* feat(images): search and filter workspace images by linked prompts

The workspace listing now takes category (exact), tags (comma list, all
required) and sort=rating, and q also matches the linked positive's text,
Library title, pair title and tag names. One bounded, parameterized SELECT
(get_image_link_matches / get_image_paths_matching) returns normalized
paths of unambiguously linked images; the listing is intersected with it.
Unlinked and ambiguous files are excluded by a category or tag filter and
sort last by rating, with a newest-first tie-break. The cursor scope covers
the new filters and sort; unknown values are still rejected.

The derived listing values and sort orders are cached per gallery scan so a
50,000-file listing with 20,000 links pages well under 150 ms at p95.

* test(images): count the filter budget everywhere, time it in one job

The 150 ms wall-clock budget failed under the CI coverage tracer, so it
is split. test_filter_work_is_bounded runs on every job and under
coverage: one filter SELECT per request, the ambiguity probe is an index
search (EXPLAIN QUERY PLAN), the per-scan values are built on the cold
request only, and a cursor page calls the sort key at most
ceil(log2 n)+3 times. test_filter_budget runs only with
PROMPTMANAGER_PERF_BUDGET=1 and no tracer, asserts the best of three
rounds' p95, and has its own non-required "perf budget" job. TESTING.md
records the convention.

* fix(images): rating sort works however many link rows the database holds

The filter SELECT counted every generated_images row against a 500,000
row cap and materialized all of them first, so a large history made
sort=rating answer 400 with nothing to narrow. The rows are now streamed
and only paths in the listing's normalized set are kept, so memory is
bounded by the listing; the cap and LinkFilterTooBroad are gone.

* fix(images): rebuild workspace listing values only where the scan changed

A TTL rescan that finds the same files reuses the previous values and
sort orders; one that finds a few changes rebuilds only those files.
Gallery invalidation and a gallery-roots change drop the derived values,
so old roots no longer hold memory. category is trimmed like tags.

* fix(images): a monitor invalidation rebuilds only the new files' values

The image monitor invalidates the gallery whenever it sees a new file,
and that dropped every derived workspace value, so the next request paid
a full cold build. Invalidation now keeps the per-file values and drops
only the sort orders; the rescan reuses every unchanged file and builds
only new or changed ones. A gallery-roots change still drops everything.

* feat(library): expose run count, last use and variant totals

* fix(library): an out-of-range last use reads as unknown, not a failed list

* perf(images): read positive counts only on a plain listing's first page

* feat(images): persist image favorites

Favorites are stored in image_favorites keyed by the gallery root label
and the relative path, set and cleared through PUT/DELETE
/prompt_manager/images/favorite with a {root_index, path} body validated
against the rooted listing. Workspace image cards carry favorite: bool.
The table is created by migration, travels with backups, is optional in
old backups and required after a restore's migration.

* fix(images): key favorites by the real root and case-normalized path

Folder names are not unique across gallery roots, so two roots shown as
"output" shared each other's stars. Favorites are now keyed by the
root's real, case-normalized path (server-side only) and a
case-normalized relative path, so a Windows case-only rename keeps its
star while distinct Linux files stay distinct. Toggles look the file up
in a per-scan dictionary instead of walking the listing.

* fix(images): favorites lookup follows monitor invalidation and the stored key bound

A monitor invalidation now also drops the favorites lookup, which held the
previous scan list until the next toggle. PUT refuses a favorite whose root
or relative key would exceed the schema's 4096-character bound, so the live
database keeps passing startup and restore validation.

* docs(testing): workspace data fields for the concept pages

* fix(images): card facts read the inspector's projected graphs

The inspector hands its browser parser only graphs the inspection
projection parsed (strict JSON, depth 64, node ceiling); a deeper record
is "limited" there, so the legacy fill never runs and no seed shows. The
card parsed the raw records instead and showed a seed the inspector did
not, e.g. for a 900-deep class_type beside a normal sampler.

project_graph() is now the one projection step, used by the inspection
response and by the card. The card runs the shared parser on the
projected graphs, the same values the browser receives: big integers
arrive as their exact digits, so the separate exact-integer re-parse and
the parse_js_json flag it needed are gone.

* fix(images): send the card seed as a decimal string

JSON.parse rounds a number past 2**53, so a listing seed such as
9223372036854788153 reached the page as 9223372036854788000 while the
inspector showed the file's digits. Every non-null card seed is now its
decimal string.

The Node parity harness now JSON.parses the real listing response and
compares String(seed) with the seed the inspector shows, including a
seed past 2**53. TESTING.md describes the string contract and drops the
removed exact re-read.

* fix(images): bound digit-string seeds and isolate each card's parse

A seed recorded as a 5000-digit string reached int(), which raises
ValueError past Python's integer string limit, and the listing answered
500. A digit-string seed now has at most 20 digits (2**64 fits), and the
conversion falls back to no seed on ValueError.

Each card's parse is also guarded in the page builder: an unexpected
error in one file's records logs a warning and gives that card no seed
and no first-line title, and the rest of the page still lists.

* fix(images): filename search survives a failed prompt-link read

On main a workspace q matched only file paths, so search worked with a
broken database. Since search also matches linked prompts, any q turned
a failed link read into a 500. A q without category, tags or the rating
sort now falls back to filename matches with a null positive_count and
logs the failure with its traceback; those linked filters still fail.

* fix(images): card reads PNG records with the inspector's reader

The card read PNG text through Pillow's header info, where a later
duplicate prompt chunk silently wins and text after IDAT is missed. The
inspector's bounded reader refuses conflicting duplicates and reads text
after IDAT, so a file with two different prompt chunks showed a card
seed the inspector refused.

PNG cards now read their records with read_inspection_file, the reader
the inspector uses. When it refuses a file the card keeps its
dimensions from Pillow and shows no seed. Other formats still use
Pillow's header info, as the inspector's header read does.

* perf(images): read each card's header facts once per gallery scan

Every page parsed each card's prompt and workflow records again, about
3.4 ms per 50 KB record set and 25 ms per 400 KB workflow, inside
ComfyUI's server process, on every scroll-back and refresh.

Dimensions, the seed and the parsed positive's display title are now
cached per root by (path, mtime), beside the scan's per-file values and
invalidated the same way: a rescan keeps unchanged files and prunes the
rest, a roots change drops them all, and only files of the current scan
are stored. An unreadable header is not stored. Links and favorites
are still read on every page.

* ci: run the perf budget on main pushes and on demand only

The wall-clock budget has under 2x margin on shared ubuntu runners, and
a red exact-head check blocks a merge that must never be rerun to green.
The job now runs on each push to main and on workflow_dispatch, not on
pull requests. TESTING.md says a red run on main means fix the code.

* fix(library): cut display titles at 80 UTF-16 units, never inside a pair

The server counted code points while the Library's JavaScript counts UTF-16
units, so a first line of emoji kept 80 emoji on cards but showed 40 in the
Library. Both now cut at 80 UTF-16 units and drop a character that would
straddle the limit instead of leaving half a surrogate pair.
2026-10-20 00:18:32 -07:00
Vito 9d398f9862 feat(workspace): concept shell — sidebar, topbar, heading and footer on the approved mock's CSS (#256)
* build(workspace): vendor the approved concept stylesheet with a drift check

* refactor(workspace): adopt the concept stylesheet as the workspace base

* feat(workspace): concept sidebar with real navigation counts

* feat(workspace): concept topbar with Theme and ComfyUI link

* refactor(workspace): one concept heading and footer for every route

* fix(workspace): keep the canvas link on the proxy mount

A resolver failure no longer stops the shell from mounting: the ComfyUI link falls back to the root read from the page's own mount path (never another origin). Tests cover a non-standard /ComfyBackendDirect mount, including that no request escapes it.

* test(workspace): concept fixture and shell comparison

* test(workspace): concept fixture carries the concept inspector facts

* fix(workspace): match the concept brand name and compare shell hover states

* test(workspace): shell comparison mutation check and docs

Add mutate() to the shared concept inventory and a shell mutation check: a
nav-row min-height rule and a removed .breadcrumbs each make the comparison
fail with the changed path. Harden the resolver-failure guard test with an
injection count, make the overlay lint fail when workspace.css is missing,
mark the library.css .row-actions leftovers for PR E, and document the
concept CSS base, the overlay lint, the comparison method and the recorded
shell deviations in TESTING.md.

* fix(workspace): focus the page title on a route change

A sidebar or history navigation now focuses #page-title, as the concept
does on every hashchange (app.js 1168), so a screen reader announces the
page heading instead of the main region. The first render leaves focus
alone and the skip link still focuses main. The heading module's comment
now describes what the shell does.

* fix(workspace): give the Overview nav entry its own home glyph

Overview reused the concept's Activity pulse byte for byte, so once the
Activity row lands two entries would share one icon. Overview now has an
outline home glyph. The shell comparison takes it from the D28 ruling
instead of the concept's activity path, and the icon test checks that
every approved glyph is distinct.

* test(workspace): accept only #249 decision markers and state the lint scope

The overlay lint accepted any D-number, so another issue's ruling such
as "D5 (#245)" could mark a concept override. A marker now has to be a
D-number with no other issue cited; the vocabulary rule that cited #245
reads "#245 decision 5". TESTING.md now states that the lint covers
workspace.css only, lists the unmarked overrides still in images.css,
and says PRs C and D run the lint over each page sheet they keep.

* perf(workspace): re-read only Library and Tags counts after a Library write

A confirmed Library or vocabulary write cannot change the Images count,
but it re-read all three, including the Images output listing. It now
re-reads only the Library and Tags counts. Each count has its own read,
so a write never cancels an Images read that is still in flight.
2026-10-19 23:04:40 -07:00
Vito a44cd27c2e fix(parse): build array String() without recursion (#255)
utils.generation_params converted nested arrays to strings recursively,
so a crafted file with a value nested about 900 arrays deep raised
RecursionError, where the browser parser returns a result. The join now
runs on an explicit stack. A JS-parity case with 900-deep values pins it.
2026-10-19 22:04:32 -07:00
Vito 673433b2ae fix(inspect): keep the server result when the legacy parser throws (#254)
#251 fills undecided roles from the browser parser, which throws on some
malformed graphs (for example a null workflow node). The error escaped
the record builder, so the inspector could not load such files at all.
The fill-in is now skipped when the parser throws.
2026-10-19 21:43:01 -07:00
Vito 36564b4767 feat(parse): one generation-parameter parser in Python; scan links what the resolver cannot decide (#253)
* feat(parse): port the browser generation-parameter parser to Python

Adds utils/generation_params.py, a line-for-line port of
extractGenerationParams in web/js/comfy-metadata.js, the parser behind
the gallery, admin and metadata pages. The scan and live monitor can use
it so every surface reads prompts the same way. The JavaScript parser is
unchanged.

The port reproduces the JavaScript value semantics the result depends on
(property order, truthiness, String(), trim(), strict equality, Map keys,
JSON.parse number rounding, and the inputs on which it throws), and runs
its recursive tracing on an explicit stack so crafted graphs cannot
exhaust the Python stack.

Tests:
- behaviour tests for each tracing rule, with JavaScript results as
  expected values, including the visit-limit boundary;
- golden vectors of the JavaScript parser's results for every named case
  and every PNG fixture, so the comparison runs without Node;
- a live comparison against the JavaScript parser when Node is present;
- a randomized differential test over 9,000 generated graphs and
  workflows, coherent and corrupted, including inputs where the
  JavaScript throws.

Five deliberate porting mistakes (trim, truthiness, prepend handling,
visit limit, pair slot) each fail the tests. On a 38,602-file output
folder, all 18,491 files with a workflow give identical results.

* fix(scan): link the traced prompt when the pair resolver cannot decide

The output scan skipped every file the pair resolver called ambiguous or
unresolved, so workflows where several samplers share one prompt (a base
pass, an upscale pass and a detailer) never reached the Library from a
scan. It now links the positive prompt the shared parser traces for
those files, the prompt the inspector and the legacy pages show. Files
the resolver decides are stored as before, and graphs without a
recognized sampler keep the legacy extractor (#252).

Read-only scans of a 38,602-file output folder into temporary databases:
- links: 1,500 before, 7,954 after; none of the 1,500 dropped or changed;
- of the added links that differ from the pre-redesign scan, 1,275
  replace a number, a LoRA tag, a fragment or a truncated prompt that
  scan stored.

* refactor(parse): return a uniform pair from the SameValueZero key
2026-10-19 20:56:51 -07:00
Vito d6eba05d21 fix(inspect): show the prompts and settings the legacy parser finds (#251)
* fix(inspect): show the prompts and settings the legacy parser finds

The workspace inspector took its prompts from the new server resolver,
which refused workflows where several samplers share one prompt (base
pass, upscale pass, FaceDetailer), and showed "Multiple possible
prompts" for files the legacy gallery and admin pages read correctly.

The inspector now runs the same extractGenerationParams those pages
use and shows its positive, negative, seed, steps, CFG, sampler and
model whenever it finds them. Prompts recorded at execution or saved
in the Library are never replaced; when the legacy parser finds
nothing, the server result stays.

Read-only check over a 38,602-file output folder: prompts shown went
from 1,798 to 7,542 files, and every one of the 7,236 positives and
9,421 negatives the legacy parser finds is shown identically.

* fix(inspect): keep definite server prompts over the legacy parse

The legacy parser follows ConditioningZeroOut back to the positive text
node and shows the positive as the negative. Apply its result only to
roles and settings the server could not decide, so a definite answer
such as an explicitly empty negative is never replaced.

Node callers (the corpus facts tool and the parity test support) now
pass the parser explicitly instead of relying on a browser global.

On a 38,602-file output folder, prompts shown stay at 7,542 files. Where
the server and the legacy parser both answer, they differ only in
surrounding whitespace (793 roles), except 227 zeroed negatives the
legacy parser misread and one prompt that is a JSON block.

* fix(inspect): never fill a setting the file does not record

The legacy parser can take a FaceDetailer as the generation sampler and
report its CFG for a workflow whose guider records none. Fill a setting
from the legacy parse only when the server left it unresolved or
ambiguous; a known value or a definite "not recorded" stays.

The corpus and parity Node tools again measure extractGenerationFacts
on its own, the contract their fixture vectors describe.

* fix(viewer): destroy an inline Viewer only after its pending view settles

Viewer.js's inline destroy() only unbinds and leaves an in-flight image
view running; when it finishes it reads the destroyed item list and
throws "c is undefined". Closing the preview while an image was still
loading hit this in Firefox about once in 15 runs of the recorded-run
history test. A closed viewer is now destroyed once that view settles,
its image fails, or a 10 second bound passes.

The history test passed 60 of 60 runs in Firefox with the fix.

* fix(viewer): keep the image src until a closed Viewer is released

Viewer.js rebuilds its item list from images that still have a src. On
close, the src was removed before the viewer cleanup ran, so a pending
view found no items and threw "c is undefined" (WebKit and Firefox).
With Viewer.js, the src is now removed after the viewer is destroyed,
once its pending view settles.

The recorded-run history test passed 40 of 40 runs in WebKit and in
Firefox.
2026-10-19 20:05:55 -07:00
Vito 6d2213ead5 feat(inspect): declarative node support matrix (#250)
* feat(inspect): add a declarative inspection node semantics table

* fix(inspect): bound and isolate the semantics table

Bound the semantics loader: at most 1 MiB is read, containers may nest
at most eight levels, and a RecursionError from the JSON parser becomes
a SemanticsError instead of escaping.

The table is no longer loaded at import. get_semantics() reads it on
first use and caches only a valid result, and the resolver imports it
lazily, only for a profile with semantics on. The default ingestion
profile never imports the module, so a missing or corrupt table cannot
break scan, monitor or admin import. Inspection of a prompt graph with
an unusable table is unresolved with the reason "Inspection node
semantics are unavailable" and an error log, rather than a crash.

INSPECTION_PROFILE now has semantics on; no strategy consumes the table
yet, so resolution is unchanged.

* feat(inspect): resolve guider-based samplers

Under INSPECTION_PROFILE the resolver discovers samplers from the
semantics table and follows SamplerCustomAdvanced through its guider
link (Reroutes allowed) to a listed guider:

- CFGGuider: positive and negative, so the pair resolves.
- BasicGuider: positive only. The state is "absent" with the reason
  "Guider has no negative input", the existing missing-negative path.
- Any other guider, including DualCFGGuider (D17), is unsupported-node
  and names that guider.

Guider hops draw visits, scan and depth from the existing budgets. A
SamplerCustomAdvanced upstream in latent ancestry is an earlier stage,
so the result stays multistage-ambiguous. Positive and negative
provenance record guider_id. FaceDetailer (detailer-stage) is not a
candidate yet: deciding it needs output ancestry. DEFAULT_PROFILE never
loads the table and is unchanged.

Vector changes (inspection profile only, default vectors unchanged):
- f07b: no-sampler -> resolved (CFGGuider).
- f17: no-sampler -> unsupported-node (FluxGuidance, Task 2.3).
- f01b: no-sampler -> unsupported-node (EmptyFlux2LatentImage latent,
  Task 2.5).
- f01: no-sampler -> ambiguous (24 SamplerCustomAdvanced branches).

The no-sampler tests (resolver and inspect route) now use a sampler
class the inspector does not know, because f17 has a supported one.

* feat(inspect): trace conditioning through supported pass-through and selector nodes

* feat(inspect): treat detailers as sampler stages

* fix(inspect): choose selectors that share one condition together

* fix(inspect): bound detailer stage chains by the depth limit

* feat(inspect): support common text encoders and string sources

* fix(inspect): keep pair execution evidence behind unreadable string sources

* feat(inspect): accept any non-sampler latent origin

* fix(inspect): queue each node once in breadth-first ancestry walks

The latent and detailer image walks queued every link before checking
whether its node was already seen, and popped from the list head. A dense
shared fan-in grew the queue far past the visit budget before any visit
was charged. Both walks now use a deque and enqueue each node at most
once, charging every examined edge as scanned work; outcomes are
unchanged.

* fix(inspect): treat unlisted pipe and embed samplers as possible stages

The latent and detailer ancestry walks only flagged unlisted nodes with a
linked latent_image, guider, positive or negative input, so easy-use pipe
samplers, Impact pipe detailers and embed-driven video samplers were walked
through and the final sampler's prompt resolved silently. A separate
stage-shaped input set (adding pipes, embeds and custom-sampler parts)
now makes such nodes unsupported-node, named.

* fix(inspect): walk each sampler's latent ancestry once per resolution

Every selector branch run and candidate re-walked the same latent
ancestry, so conditioning-carrying origins such as WanImageToVideo turned
a three-selector graph from ambiguous into work-limit. The walk outcome is
now cached per sampler; branch runs of one candidate pay for it once and
other candidates are charged its visit cost, as the text memo does. Walks
that ran out of work are not cached. Every link input is still walked.

* fix(inspect): treat unlisted model-driven samplers as possible stages

Unlisted samplers that link a model with conditioning or a latent under
other input names (XlabsSampler, generic video samplers) were still walked
through, so the final sampler's prompt resolved silently. An unlisted node
with a linked model plus conditioning, neg_conditioning, samples or latent
is now unsupported-node, named; model patches such as LoraLoader and
ModelSampling nodes are not.

* feat(inspect): compute facts from the resolved sampler

* fix(inspect): merge facts across sampler branches when ambiguous

* fix(inspect): merge facts over unresolved stages and share stage inputs via the table

* feat(inspect): explain why a prompt could not be resolved

* feat(inspect): port legacy parser rules the new resolver missed (PromptManager)

R1: before 3.2.3 a save patch overwrote every PromptManager node's saved
text with the last-run node's prompt. The legacy viewer recognizes the
signature (comfy-metadata.js createContext: two or more PromptManager
workflow nodes with one API text and different widget values). Inspection
now refuses those nodes as malformed instead of resolving to another
node's prompt; the inspector hands the parsed workflow to the resolver for
this check only. Lands before the new pass-throughs (R6) that would
otherwise make these graphs resolve to the wrong text.

R7: this repository's node classes up to 07a5af3 (PromptManagerV2,
Positive, Negative, Detailer, V2Text, PositiveText, NegativeText) resolve
with their own outputs and their exact strip and ", " join.

The default (ingestion) profile ignores both.

* feat(inspect): port legacy parser rules the new resolver missed (A1111)

R3: an accumulating batch saver repeats the A1111 parameters record once
per batch item, identical except "Batch index: N". The bounded PNG reader
now keeps the last copy (Pillow's rule, which #165 relied on) instead of
calling the file unreadable. Any other difference, including a missing or
second batch index, is still a conflict, and decoding the copies draws from
the duplicate role budget.

R4: parameters text that starts with "Negative prompt:" has an empty
positive (#165), instead of showing the delimiter line as the positive.

R5: the settings footer may be followed by sd-dynamic-prompts' one-line
"Template:" and "Negative Template:" records, and may be "Steps: N" alone
when there is no full footer.

R10: commas inside A1111's quoted settings values no longer split them, so
an "ADetailer prompt" with a comma keeps the other facts known.

The facts' A1111 reader mirrors R4, R5 and R10; the default (ingestion)
profile keeps the historical split. Also from the Task 2.7 review: a guard
test fails on any uncoded unresolved reason without a user line (M1), and
the encoder bounds role lines to the client's 512 code units (M4).

* feat(inspect): port legacy parser rules the new resolver missed (installed nodes)

R6 and R9 for nodes whose sources are installed beside ComfyUI, each
entry's input names and output slots read from the node's code (the task
report lists file and line):

- detailer stages UltimateSDUpscale (image), UltimateSDUpscaleNoUpscale
  (upscaled_image) and DetailerForEach (image), judged with their upstream
  samplers like FaceDetailer;
- pass-throughs ControlNetInpaintingAliMamaApply and LTXVConditioning
  (positive 0, negative 1) and FluxKontextMultiReferenceLatentMethod;
- encoder Text to Conditioning, strings TextInput and DisplayText;
- facts model pass-throughs Lora Loader (LoraManager), Power Lora Loader
  (rgthree), ModelPatchTorchSettings, CFGNorm, PerturbedAttentionGuidance.

Impact Pack encodes a detailer's non-empty wildcard instead of its positive
input, so FaceDetailer and DetailerForEach now list it as a prompt
override: a literal wildcard is decided at run time and a linked one names
its source, instead of resolving to the positive input. The overwritten
PromptManager refusal (R1) still holds behind the new pass-throughs. The
default profile does not read the table.

* feat(inspect): port legacy parser rules the new resolver missed (upstream nodes)

R6 and R9 for nodes not installed beside ComfyUI here. Their input names
and output slots come from each pack's source at the commit the saved
workflows record in the node's cnr_id and ver properties (the task report
lists repository, commit, file and line):

- samplers ClownsharKSampler_Beta (RES4LYF), KSampler (Efficient) and
  ttN KSampler_v2 (latent input "latent");
- pass-throughs SeedVarianceEnhancer and ApplyInstantIDAdvanced (positive
  1, negative 2; output 0 is the model);
- selector Switch conditioning [Crystools];
- encoder smZ CLIPTextEncode: its text when with_SDXL is false; when true
  the connected CLIP decides at run time which inputs are read, so the
  prompt is known only when text, text_g and text_l agree;
- strings ttN text, CR Text, CR Prompt Text and TextBoxMira;
- facts model pass-through CR Apply LoRA Stack.

Inputs that replace what a sampler reads are not followed: RES4LYF options
(may carry a guider, and with sigmas replace the settings), an Efficiency
script and a ttN XY plot leave the prompt unsupported and the facts
unresolved. The overwritten PromptManager refusal (R1) still holds behind
SeedVarianceEnhancer. The default profile does not read the table.

* test(inspect): cover batch copies past the role budget and orphan widgets

* fix(inspect): match the batch index only in the settings footer

Review I-1: duplicate A1111 parameters copies were compared after blanking
"Batch index: N" anywhere in the record, so copies whose prompts differ
only by that phrase were accepted. The field is now blanked only inside
the record's one "Steps: N, ..." footer line, which must hold it exactly
once; any other difference stays a conflict.

Review M-4: the inspector parses the workflow record only when the API
graph can show the overwritten-PromptManager signature (a PromptManager
class, or a node with PromptManager's search_text input).

* fix(inspect): refuse overwritten PromptManager text inside subgraphs

Controller ruling on review M-1: PromptManager nodes inside subgraphs run
as "<instance>:<inner>" API IDs, nested instances chaining their IDs. The
overwritten-text check now walks subgraph instances (bounded in depth and
nodes) and reads a text widget stored by name, so these nodes are refused
like top-level ones, anchored or not. A definition without an instance
runs nothing and is ignored.

Review M-5: the historical adapters skip an explicit null prepend or append,
as the 07a5af3 code's "if prepend_text:" did.

* test(inspect): import unittest one way in the inspection suites

Code scanning flagged the mixed 'import unittest' and 'from unittest import
mock' forms. Use 'import unittest' plus 'import unittest.mock' everywhere.

* fix(history): name the recorded run's current-file prompts apart from its frozen fields

Now that the inspector resolves more saved files, the Recorded run dialog can
hold two regions named "Positive prompt" (and two of each Copy/Send action):
the frozen run field and the inline current file's prompt. Assistive
technology could not tell them apart, and the real-host chronology test read
two <pre> elements.

Prompt blocks take an optional context label. The inline current file inside
a recorded run names its regions "Current file positive prompt" and its
actions "Copy/Send current file positive prompt"; the visible headings stay
as in the concept. The real-host test now also scopes its frozen-field reads
to the "Frozen run prompt texts" section, and a disposable-server workflow
pins the distinct names with a resolved current file.

* fix(inspect): drop preview-only stages from unanchored candidates

A stage whose outputs reach only non-file outputs (PreviewImage and the other
non_file_outputs) cannot have written a saved file, yet it stayed an
unanchored candidate, so a preview-only FaceDetailer made the saved branch
ambiguous (f01b).

The inspection profile now drops such stages before detailer absorption,
using two linear backward passes over the graph (charged like the consumer
map; a failed pass keeps every stage). A stage without consumers, or one that
also reaches any other terminal, is kept, and nothing is dropped when every
stage is preview-only. Raw plus detailed double saves both write files and
stay ambiguous. A file in the host temp folder may itself be a preview, so it
is inspected with PREVIEW_INSPECTION_PROFILE, which keeps those stages.

Workspace facts follow the same choice: an absent or unresolved result that
names a sampler merges facts across that sampler's stages only, so the
dropped detailer's settings no longer blur the saved file's facts.

f01b now resolves (absent negative, guider has none); its vector is updated
and the expected failure removed. The default profile is unchanged:
differential against 39dcc40 identical over 160,256 cases. Over 60,054
inspection cases no previously resolved or absent result changed.

* fix(history): name the current file's prompt pair button apart from the frozen run's

The recorded run dialog nests the current-file inspector, whose footer repeated the "Send prompt pair" name. Name it from the prompt context, scope the real-host test to the frozen region, and add a dialog-wide duplicate-name check.
2026-10-19 17:26:37 -07:00
Vito cbf46a7573 feat(vocabulary): native Tags & categories screen (#245)
* feat(vocabulary): add workspace vocabulary state

* feat(vocabulary): render the native Tags & categories screen

* feat(vocabulary): add guarded vocabulary dialogs

* fix(vocabulary): recover unconfirmed deletes and list every category in Manage

* refactor(vocabulary): use the shared plural helper

* test(vocabulary): compare the native screen with the approved concept

* fix(vocabulary): wrap long tag names so row actions stay visible on narrow screens

* docs(vocabulary): document the native vocabulary screen and gates

Also aligns the shell link focus-ring offset with the concept (3px) and makes
the chrome comparison poll the document title.

* fix(vocabulary): refresh lists when a write lands after its dialog closes

A rename, delete, create or category apply that commits after its dialog
was disposed (Chromium force-closing the modal on a repeated Escape, or
Back while the write is in flight) returned before onInvalidate, so the
lists were never re-read and a cached Library shelf kept showing the old
names. Like the recipe editor, a disposed command now still reports any
outcome that may have changed storage (receipt, conflict, restore,
unconfirmed) without rendering; only a failed write stays silent.

* fix(vocabulary): give the Manage categories datalist its own id

The proposed-name datalist reused the recipe editor's `categories` id. It was safe only because routes dispose each other's dialogs; it is now `vocabulary-category-options`.

* docs(vocabulary): correct vocabulary testing notes

Name the Tag column and the Create tag heading action, say that a failed Load more moves focus to Retry, place bulk actions on the admin page, describe late writes after a closed dialog, Browse's exact filter and the full deviation list, and use a test port that is not the live concept server's.

* test(vocabulary): raise the vocabulary dialog branch floor

coverage:update after the late-write and datalist fixes: workspace/vocabulary-dialog.mjs branches 87 -> 88. No floor is lowered.

* fix(vocabulary): keep 390px row actions visible regardless of font metrics

The one-column Tags table reserved a fixed 6em for the tag name. With
CI's stock DejaVu Sans fallback (about 9% wider than the pinned image's)
Tag + Prompts + Images + stacked actions needed 351px of a 336px wrap,
so Browse and Rename / merge were pushed out of view on every row.

At 700px and below the Tag column now has no fixed minimum and the
actions ask for no more than their stacked width, so all remaining space
goes to the tag name: short names stay on one line wherever the
concept's own table fits, and a wider font wraps the name instead of
clipping an action. A new test widens the metrics with letter spacing
and fails without this change in every engine.

* ci: run browser jobs in the pinned Playwright image

Concept comparisons measure geometry, which follows the fonts a browser
falls back to. The concept names Inter, which neither environment ships:
the pinned image falls back to WenQuanYi Zen Hei, a stock ubuntu-latest
runner to DejaVu Sans. Under DejaVu the concept's own Tags table needs
more than a 390px viewport (its page scrolls sideways and, under touch
emulation, the widened layout viewport resizes its dialogs), so the
390px comparisons failed in CI while passing locally.

Both browser jobs now run in mcr.microsoft.com/playwright:v1.64.1-noble,
the image local runs use, instead of installing browsers on the runner.
A tooling test keeps the tag equal to @playwright/test in the lockfile.

* fix(vocabulary): ignore a category preview edited while it is pending

Editing the Category or proposed name while a slow Preview was in flight
was a no-op in the saving phase, so the late response still became an
applicable preview: the form showed the new values while Apply would
submit the old source and target.

An edit during a pending preview now marks it stale. Its late result
returns the dialog to idle (Preview again) instead of showing an impact
or offering Apply; a plain failure of a stale preview is not shown
either, while a restore or conflict still is, since it describes
storage. The inputs stay editable during the request, as in the recipe
editor, which disables only its actions while saving.

* ci: use the image's Python and Node in containerized browser jobs

Inside mcr.microsoft.com/playwright:v1.64.1-noble, actions/setup-python
failed before any test ran: `spawn /__t/Python/3.12.15/x64/bin/pip
ENOENT`. The toolcache builds come from the runner, and their entry
points name host interpreter paths that do not exist in the container.

The jobs now use the image's Python 3.12 and Node 24. The image has no
ensurepip and the job runs as uid 1001 without root, so a checksum-pinned
pip wheel bootstraps a user-space venv for requirements-dev.txt; a step
checks that the image Node is 24. The tooling test forbids setup-python
and setup-node in these jobs and requires the pinned bootstrap.
2026-10-19 12:05:27 -07:00
Vito 1ba9e22824 fix(database): stop SQLite write-lock starvation under concurrent saves (#246)
* test(database): require identical WAL/NORMAL settings on managed connections (#242)

Concurrent writers failed with 'database is locked' on Windows runners
despite the 30 s busy timeout: every WAL commit flushed to disk while it
held the write lock, so polling waiters starved. These tests pin the
shared per-connection settings, including synchronous=NORMAL in WAL mode.

* fix(database): stop WAL commits from holding the write lock across a flush (#242)

Every managed connection now goes through configure_connection: WAL,
foreign keys, the 30 s busy timeout and, when WAL is active,
synchronous=NORMAL. Under FULL each commit flushed to disk while it held
SQLite's only write lock; on slow Windows disks a burst of writers (a
scan beside ComfyUI saves, or the concurrency tests) starved a polling
waiter past the busy timeout and it failed with 'database is locked'.

With a 60 ms emulated flush on Linux, 8 threads x 50 save+use calls went
from 49.6 s with a 32.1 s worst call (and 'database is locked' in the
unit test) to 0.7 s with a 0.4 s worst call. NORMAL in WAL mode keeps the
database consistent after an application crash; only an OS crash or
power loss can drop the most recent commits. In-memory or non-WAL
databases keep SQLite's FULL default.

* test(history): hold the first concurrent writer open past the bounded wait (#242)

The concurrent replay tests expected every peer to succeed, but each peer
only waits 250 ms for BEGIN IMMEDIATE by design. On a slow Windows runner
three queued commits exceeded that and a peer raised 'database is
locked'. Holding the first writer's transaction open for 500 ms makes the
same failure deterministic on any host (20/20 locally).

* test(history): let concurrent replay peers queue like ordinary writers (#242)

History writes deliberately give up after a bounded 250 ms wait so they
never stall ComfyUI; the existing bounded-timeout test still proves that.
The concurrent replay tests assert that every peer succeeds, which is a
serialization and first-capture property, so they now raise the budget
to the shared 30 s busy timeout while the first writer is held open.
The budget is one named constant, EXECUTION_BUSY_TIMEOUT_MS, shared by
the transaction helper and the background writer.

Before: 20/20 'database is locked' with the held writer. After: 0/20.

* test(history): wait on restore handoff events, not a 5 s clock (#242)

The retired-owner projection tests park a history read at a barrier
while the test performs a full multipart restore, then release it. The
barrier gave up after 5 s and raised inside the route handler, so on a
slow Windows runner (a 694 s suite run) aiohttp answered with its
text/plain 500 and the test reported 'history route missing'.

Every wait already ends on an event; the ceiling now only stops a broken
test from hanging (60 s). With restore slowed by 5.5 s locally, all 8
subtests failed with text/plain before and pass after.

* docs(testing): add the concurrent SQLite test rule to the Windows preflight (#242)
2026-10-19 11:10:47 -07:00
Vito 048d599386 fix(inspect): bound inspection work, not graph size (#244)
* test(inspect): add synthetic inspection parity fixtures and legacy characterization

* test(inspect): pin test-time parity vectors and keep fixtures fully synthetic

* fix(inspect): bound resolver work instead of graph size

* fix(inspect): bound shared-node and stripped-text work in the inspection profile

Keep one memo per resolution whose entries carry their unfolded visit cost;
a memo hit charges that cost to the current candidate's allowance only.
Cache node validation and charge input keys, stripped characters and copied
provenance entries to a whole-resolution work budget before doing the work.
The default profile is unchanged.

* fix(inspect): keep memoized depth honest across inspection candidates

A memo entry now records its height below the reusing depth; a hit raises
depth-limit exactly where a fresh walk would, so each candidate's outcome
matches its anchored result and does not depend on trace or node order.
The default profile is unchanged.

* test(inspect): pin memo height propagation

Three samplers share one chain, each later one hanging below a memo hit.
The third candidate must report depth-limit in both trace orders and when
anchored; removing the memo-hit height propagation fails this test. The
provenance test now asserts a plain subset.

* fix(inspect): tolerate ComfyUI node metadata quirks in projections

Projections now take a kind. Each prompt node keeps exactly class_type and
inputs; _meta, is_changed and other node keys are validated and dropped, so
"is_changed": [NaN] no longer marks the whole prompt graph malformed. A
nonfinite value inside inputs, anywhere in a workflow, or in a retained
carrier member projects to null and is never reported as a number. The
carrier applies the prompt/workflow kinds to its members. Exact numbers are
unchanged (Decimal spelling, unsafe integers as strings) and duplicate keys
are still malformed. The 500-node gate is replaced by a 20,000-node backstop
(workflow count includes subgraph definitions). Raw records are untouched.

The f14 inspection vector's facts become known; default vectors are unchanged.

* fix(inspect): budget metadata per record so large workflows never hide prompts

PNG inspection no longer counts every text chunk against one shared 2 MiB
budget. Role records (prompt, parameters, the pair execution record) keep a
2 MiB budget together, encoded and decoded, and exceeding it still fails the
whole read. A workflow has its own 4 MiB budget: a chunk over it is skipped
by seek, and a compressed one stops inflating at budget + 1. Only the
workflow is then reported as limited (record format "limited", no raw text,
workflow_graph limited); prompts, facts and the prompt graph are unaffected.

Each text chunk's keyword is read first (at most 80 bytes). Chunks that are
not retained are skipped by length without being read or checksummed, so
memory stays bounded. Byte-identical duplicates of a retained record are
accepted and do not consume budget; differing duplicates still raise
"Duplicate prompt metadata", and a duplicate of a limited workflow stays
limited. The chunk-count limit is unchanged.

InspectionRead gains limited_records (empty by default). The reader is only
used by the inspection routes, so ingestion, scan and import are unchanged.

Vectors: f04-inspection (2.5 MB workflow, within the new workflow budget)
and f15-inspection (identical duplicate chunks) now read and resolve, with
the same outcome and facts as f05-inspection. Default vectors are unchanged.

* fix(inspect): keep prompt evidence longest under the response budget

* fix(inspect): bound identical duplicate chunk reads

* fix(inspect): fail closed when role duplicates exceed their budget

* fix(inspect): use the bounded inspection profile end to end

Inspection routes resolve with INSPECTION_PROFILE and expose reason_code
and node on the resolution; scan, live monitor and admin import keep the
default profile (pinned by spies over the real resolver on f03).

JS facts drop the entries and workflow node-count gates; per-lookup visit
sets and the server's node ceiling bound the work. The validator accepts
the textless `limited` workflow format, follows the workflow-first
response omission order, and checks the closed reason-code vocabulary.
The file inspector labels limited records and disables their workflow
download.

vectors.json: 12 inspection vectors gain their reason codes; f03 facts
become known and f01 facts ambiguous. Default vectors are unchanged. A
new parity test asserts vectors.json facts through the workspace JS.

* chore(tools): add read-only inspection corpus tool

tools/inspection_corpus.py runs the inspection route pipeline and the
workspace JS facts of a chosen code tree over a media folder, read-only,
and reports the #240 invariants H1 to H8 with aggregate counts. Rows hold
statuses and SHA-256 digests of resolved role texts, never file names or
prompt text unless --with-paths is given. Outputs inside the root or a code
tree are refused.

The facts-over-vectors test now fails instead of skipping when CI is set
and Node.js is missing.

* docs(testing): document bounded inspection work

* fix(tools): fail the corpus run when facts or inspection did not run

Rows whose facts were never computed (no or exiting Node.js) now fail H3,
and a new harness line fails on any tool exception or missing facts, so a
run where nothing was measured can no longer exit 0. Unreadable files stay
inspector outcomes. A missing --node binary is refused, an exited harness
is reaped, H8 has a 50 ms floor for tiny baselines, and JSONL rows are
sorted by id.

* docs(inspect): correct ceiling wording and bandit annotations

Describe the two 20,000-node ceilings (prompt entries, workflow UI plus
subgraph nodes) in TESTING.md, and use bare nosec ids so Bandit stops
warning about unparsed test names.

* test(inspect): make parity observation tuples explicit
2026-10-19 09:54:29 -07:00
Vito 44b6ed2420 fix(library): editor follow-ups after storage restore and history selection parity (#243)
* fix(library): close recipe editor follow-ups from the native editor slice

- After restored storage, Create text revision and Archive are disabled so
  they cannot re-read an old numeric ID; only Reload Library remains.
- The storage-changed exit in Edit, revision and Add negative states that
  Reload Library closes the dialog and discards the entries (the concept
  has no discard confirmation, so the copy stays inline).
- When Save becomes disabled after an unconfirmed write or a conflict,
  focus moves to the recovery action instead of falling to the body; the
  archive confirmation does the same.
- A reuse toast only says the entered parent metadata was not applied when
  the controller found it differs from the stored metadata.
- Prompt history opens on its first negative variant, as the concept does,
  so the heading Send starts enabled.
- "1 linked image" is singular on cards, variant lanes and the embedded
  gallery count.

* fix(library): treat a blank category as equal to a stored uncategorized one

A reuse that left the category blank sent "Uncategorized" and compared it
with the stored NULL, so the toast claimed the entered metadata was not
applied. Both sides now normalize blank and "Uncategorized" the same way
before comparing; a genuinely different category still discloses it.

* test(coverage): raise frontend coverage floors after editor follow-ups

* fix(library): singular counts for one item and honest discard copy

- One shared countOf/noun helper (web/js/workspace/plural.mjs) now words
  linked images, saved negative variants (card chip and history count),
  distinct recorded runs and the Images toolbar count in the singular for
  exactly one; every other count keeps its existing wording.
- The restored-storage status says "Leaving this dialog discards your
  entries." because Cancel, the close button and Escape discard the
  draft too, not only Reload Library.
- The new helper gets its coverage floor; no other floor changes.

* fix(library): open every prompt history visit on its first negative

The concept resets to the first negative variant on every navigation to a
prompt (navigate: state.variant = 0) and selects another pair only right
after a save names it. The history no longer remembers a clicked variant
or an explicit All images choice across visits: the per-visit memory is
replaced by a one-shot saved selection that the next history mount (or the
same visit's reconciliation) consumes, and a verified pair outside the
loaded page is kept only for that saved selection. Within a visit, clicks
and All images behave as before.
2026-10-19 06:50:59 -07:00
Vito ab333d3838 feat(vocabulary): backend for native tags and categories (#241)
* feat(tags): keep unused tags in the shared vocabulary

* feat(vocabulary): add bounded tag and category reads

* feat(vocabulary): guard tag create, rename and merge

* feat(vocabulary): preview and apply category consolidation

* refactor(vocabulary): share canonical token hashing

* test(vocabulary): prove retained tags survive restore

* test(vocabulary): cover category paging tie-break and clarify contract docs

* test(vocabulary): assert rejected tag creations write nothing
2026-10-19 05:47:36 -07:00
Vito 39dcc40114 fix(maintenance): keep titled and history-referenced prompts when pruning (#239)
* fix(maintenance): keep titled and history-referenced prompts when pruning

prune_orphaned_prompts now also keeps prompts with a prompt_library_state
row (titled or explicit Library parents, whose state would cascade away)
and prompts that are the currently valid navigation hint of a recorded
run (execution_run_contexts positive/negative_prompt_id in the current
storage epoch whose exact raw snapshot text still matches). Stale hints
that history reads already ignore do not block pruning.

Fixes #237

* fix(maintenance): mirror unique-text hint rule and compute live hints once

The prune exempted a hinted prompt even when another prompt shared its
exact raw text, although history reads treat such a hint as dead. The
live-hint set now applies the reader's rule (current storage epoch, exact
raw text matching exactly one prompt, which is the hinted one) and lives
beside _prompt_hint in execution_contexts as LIVE_HINT_CTES.

The set is materialized once per statement and excluded with NOT IN, so
the prune no longer scans execution_run_contexts once per prompt. No
schema or index change.

Refs #237
2026-10-19 05:10:12 -07:00
Vito 8bcde4fbe4 feat(library): native recipe editor UI, pair archive browsing and export (#235)
* feat(library): manage native editor drafts and recovery

* fix(library): fence pair archive retries against restored storage

* feat(library): add native recipe editor dialogs

* fix(library): read archive counts through the editor controller

* feat(library): reconcile editor saves and archived pairs

Wire the concept's heading actions (Library Export and + New prompt, Images New prompt, history ← Prompt library, Edit metadata and pair Send to ComfyUI) and the Negative variants + Add negative and Show archived controls. Saves navigate or reconcile in place through createLibraryDetail.accept, invalidate cached shelf state by write generation, browse and restore archived pairs, and offer an explicit Library reload when storage was restored.

* fix(library): refresh archived pairs and align editor outcomes

* test(library): enforce approved native editor presentation

* test(library): verify editor host and coverage compatibility

Adds the rejected connected-target host assertion, pins the stock host listing at 80 cases, documents the native editor, and keeps Save/Archive focus after a failed write.

* docs(testing): correct host case totals and cover archive focus

* fix(library): fence editor reloads after restore and settle outcome copy
2026-10-19 01:46:33 -07:00
Vito e5da4e1d9d feat(library): backend for native recipe editing, pair archive browsing and portable export (#233)
* feat(library): persist recoverable parent state

* feat(library): expose versioned parent metadata reads

* feat(library): protect parent metadata and archive writes

* feat(library): save native recipes with atomic outcomes

* feat(library): download complete portable library json

* fix(library): name export owners list positives per spec

* fix(library): normalise native category and harden restore and id errors

* fix(library): measure stored titles past embedded NULs
2026-10-18 20:03:15 -07:00
Vito b2b817781a ci: cancel superseded PR runs and make the bandit job a real gate (#231)
* ci: cancel superseded PR runs and make the bandit job a real gate

No PR workflow had a concurrency group, so a new push left the previous
run occupying 16 Python matrix runners, the real ComfyUI job and the
three viewerjs jobs for up to 25 minutes. Add a group per workflow and
PR ref that cancels the older run. Pushes to main use the run id as
their group, so no main commit is ever cancelled or dropped as pending.

Remove the lint job from code-quality.yml. It ran black and flake8 a
second time, partly with --exit-zero, next to the protected "black and
flake8" job in test.yml.

The security job ran bandit with "|| true" and could never fail. At
medium severity it reports 74 findings today, so it now gates on high
severity only and prints the medium report without failing. The only
two high findings were SHA1 and MD5 used as a gallery entry id and a
LoRA preview cache filename; mark both usedforsecurity=False, which
leaves every hash value unchanged.

Closes #230.

* ci: verify superseded PR runs are cancelled

* ci: require the bandit high-severity check

Review on #231 pointed out that a failing bandit job did not block a
merge because its context was not required. Name the job
"bandit (high severity)" so the required context is unambiguous, and
add it to .github/branch-protection.json and the TESTING.md list of
required checks.
2026-10-18 18:00:21 -07:00
Vito 2aa39ca1a6 ci: run Firefox and WebKit page workflows in parallel jobs (#229)
* ci: run Firefox and WebKit page workflows in parallel jobs

The viewerjs job ran the Chromium/touch coverage pass and then the
Firefox and WebKit page workflows serially on one runner. The suite
grew from 396 to 496 tests in #227 and the job passed its 25 minute
cap with every test green, which #227 papered over by raising the
budget to 40 minutes.

Split the workflow into a frontend-coverage job, a fail-fast: false
matrix with one runner per browser, and an aggregate viewerjs job that
needs both and fails unless each result is success. The aggregate keeps
the exact "viewerjs" context that branch protection requires, so
.github/branch-protection.json is unchanged. Browser projects, the
single Playwright worker and per-test timeouts are untouched. Coverage
keeps the frontend-test-reports artifact; Firefox and WebKit traces
move to browser-traces-<browser>.

Closes #228.

* ci: keep the frontend coverage job on a 25 minute budget

Merging main carried the 40 minute budget from #227 onto the new
frontend-coverage job. That job no longer runs the Firefox and WebKit
workflows and took 9m31s in run 39178111649 at 396 tests, so the
original 25 minute budget is the right one for all three jobs.
2026-10-18 17:34:57 -07:00
Vito ecd21165a7 feat(history): add chronological runs and exact output inspection (#227)
* feat(history): add bounded chronological read projections (#209)

* fix(history): validate internal continuation positions (#209)

* feat(history): expose fenced history windows and run details (#209)

* fix(history): release retired projection connections (#209)

* feat(history): resolve reported outputs through current authorization (#209)

* fix(history): reject invalid inspection targets (#209)

* feat(history): add scoped history state and current-file adapter (#209)

* fix(history): preserve slideshow and clear stale reentry actions (#209)

* feat(history): render recorded executions and frozen run details (#209)

* fix(history): recover receipts and refresh capture evidence (#209)

* test(history): verify chronological history across restore and host modes (#209)

* fix(history): enforce chronological marker visual parity (#209)

* fix(history): resolve final chronological review findings

Address #209 recovery, bounded visit restoration, exact receipt availability, and portable regression coverage.

* test(history): repair Windows fixtures and review feedback

Address #209 and PR #227 native path, cleanup, media and review-comment expectations. Document Windows portability preflight without changing production behavior.

* ci: raise the viewerjs job budget for the larger browser suite

The cross-browser workflows suite grew from 396 to 496 tests on this
branch, which pushed the whole viewerjs job to about 25.5 minutes
against the 25 minute cap set in #181. Two exact-head runs on this
branch timed out without any test failing: run 39173019283 was killed
on test 474 of 496, and run 39175125474 printed "496 passed (12.8m)"
and was then cancelled during Playwright shutdown.

Raise only the job budget. Browser projects, workers and per-test
timeouts are unchanged. Splitting the browsers into parallel jobs is
tracked separately.
2026-10-18 17:17:05 -07:00
Vito 10197026be fix(database): refuse unsafe restore file fallback (#226)
* fix(database): refuse unsafe restore file fallback (#223)

* fix(database): retain safety backup limit after restore refusal (#223)
2026-10-18 10:26:29 -07:00
Vito ac23f22a37 fix(logging): close retired owned handlers (#225)
* fix(logging): close replaced owned handlers (#218)

* fix(logging): keep retired file handlers closed (#218)
2026-10-18 10:01:19 -07:00
Vito d9537a726c feat(workspace): complete shared inspector and collection viewer (#224)
* feat(inspector): expose bounded physical file evidence

* fix(inspector): distinguish ambiguous saved associations

* feat(inspector): derive conservative generation facts

* fix(inspector): guard fact ancestry and inspection state

* feat(inspector): share complete file details and exports

* fix(inspector): preserve fact states and touch targets

* feat(viewer): navigate the loaded media collection

Refs #216

* fix(viewer): refit after current stage geometry changes

Ref #216. Observe the current media stage after readiness and wrapped title changes without rebuilding focused controls or resetting transforms at unchanged dimensions.

* feat(viewer): add lifecycle-safe slideshow and media preferences

* fix(viewer): resume gallery continuation after close (#216)

* fix(images): guard source recovery and wrap viewer actions

Clear stale root and folder intent, cancel old pages, and wait for current source discovery before loading a refreshed collection. Preserve safe retry through failed, superseded, and disposed discovery.

Wrap narrow viewer footer actions and wait for measured gallery viewport settling in responsive checks. Refs #216; addresses PR #224 review.

* test(viewer): await gallery layout after viewport changes

Require actual inner, root client, and document widths to agree after each responsive-case viewport and CSS zoom transition. Keep the existing exact frame, overflow, focus, and coarse target assertions unchanged.

Observed PR #224 CI initial mobile resize at537px before pending card relayout settled to356px. Refs #216.
2026-10-18 09:41:53 -07:00
Vito e7cfcc2bfb style(workspace): align browsing with the approved concept (#219)
* style(workspace): align approved shell and visual fixtures

* test(workflows): preserve evidence across browser projects

* style(workspace): match approved image browsing geometry

* style(workspace): match natural library card layout

* fix(workspace): retain wheel motion during row correction

* fix(workspace): contain zoomed controls and image metadata (#215)
2026-10-18 02:53:26 -07:00
Vito 4ef236aaf7 feat(history): persist exact run output references (#214)
* feat(history): define bounded output reference contracts

* feat(database): migrate execution output evidence to schema v3

* feat(history): persist immutable run output references

* test(history): isolate mocked node gallery monitor

* feat(history): capture bounded output descriptors with shared leases

* fix(history): authenticate payload owners and release failed captures

* feat(history): attach output evidence to exact finalized attempts

* test(history): verify run output evidence on real ComfyUI

* fix(history): release idle worker payload references

* docs(history): clarify defensive lease cleanup

* test(history): align output fixtures with native path styles

* test(history): separate restore completion from writer drain

* test(history): bound batch drains by progress and total time
2026-10-17 23:26:37 -07:00
Vito bae542c9ac feat(history): persist exact execution prompt contexts (#213)
* feat(history): define exact bounded execution contexts

* fix(history): retain observable sources and validate evidence

* feat(history): migrate execution contexts to schema v2

* feat(history): persist immutable execution context snapshots

* fix(history): preserve package identity for execution contexts

* feat(history): bound context payloads beside lifecycle finalization

* fix(history): fence context commits and resolve stored run identity

* fix(history): bound persisted identifier bytes before restore reads

* feat(history): bind exact runtime context to execution attempts

* test(history): discover owner invalidation callback regression

* feat(history): capture cached prompt strings without mutating host caches

* test(history): verify durable contexts on real ComfyUI hosts

* fix(history): scope transforms to captured source paths

* test(ci): update execution context tooling fixtures (#209)

* fix(history): harden context validation and worker fixtures (#209)

* fix(history): honor close deadlines after early joins (#209)
2026-10-17 19:51:22 -07:00
Vito 763e6b37b1 feat(history): record durable ComfyUI execution attempts (#212)
* feat(history): normalize bounded execution evidence (#209)

* fix(history): bound lifecycle kind checks before hashing (#209)

* feat(history): persist execution attempts with restore fencing (#209)

* test(api): bootstrap standalone backup restore fixtures (#209)

* feat(history): isolate bounded asynchronous ledger writes (#209)

* feat(history): observe actual queue attempts without altering execution (#209)

* fix(database): honor package-scoped configuration (#211)

* test(history): exercise real cached and headless executions (#209)

* fix(history): reject unsupported uniqueness and isolate package fixtures (#209)

* test(history): support Python 3.10 package fixture cleanup (#209)
2026-10-17 16:42:07 -07:00
Vito fdba6fd012 feat(workspace): group saved negatives in the native Prompt Library (#210)
* feat(workspace): group saved negative variants in native library (#208)

* fix(library): preserve full name order and batch previews (#208)
2026-10-17 14:38:03 -07:00
Vito dc345671ae feat(workspace): inspect and send positive and negative prompts (#206)
* feat(workspace): inspect and send positive and negative prompts

* docs(workspace): capture complete inspector and viewer previews

* docs(inspector): explain the subprocess cleanup race

* fix(workspace): defer observed gallery layout writes (#207)

* fix(inspector): keep path-only prompt associations unverified
2026-10-17 13:43:19 -07:00
Vito 76e9b51079 feat(bridge): add acknowledged ComfyUI prompt destinations (#203)
* feat(bridge): add guarded PromptManager prompt destinations

* fix(ci): install shared contracts before frontend unit tests

* test(comfyui): wait for paired editor resize before measuring layout
2026-10-17 13:14:02 -07:00
Vito 590aa2cd0e feat(workspace): add native Images browsing and infinite scroll (#201)
* feat(workspace): add native image browsing and continuous loading

* test(workspace): assert empty collections through their status

* fix(workspace): rescan refreshed images and expose ancestor folders
2026-10-17 11:22:10 -07:00
Vito d4bb5c5ba9 feat(ui): add the tested opt-in Prompt Manager workspace shell (#198)
* docs(ui): prototype unified prompt management application

* docs(ui): add masonry and per-image tagging to application concept

* docs(ui): make gallery concept use continuous infinite scrolling

* feat(ui): introduce the tested application workspace shell

* fix(docs): preserve concept controls across cached navigation

* test(ci): emit Python tracebacks for native test crashes

* test(ui): cover workspace restoration across cached navigation
2026-10-17 10:36:30 -07:00
Vito caa9649f32 feat: reuse positive and negative prompt combinations (#195)
* feat(db): add pair schema and stage restore migrations for #193

* feat(database): persist reusable ordered prompt pairs (#193)

* feat(api): expose reusable pairs and atomic metadata enrichment (#193)

* feat(metadata): resolve canonical prompt pairs with bounded provenance (#193)

* feat(scan): enrich image pairs without guessing or duplicate links (#193)

* fix(database): preserve pair tags during global maintenance (#193)

* feat(tracking): enrich live image pairs and cancel scans safely (#193)

* feat(nodes): encode paired prompts and count cached recipe runs (#193)

* fix(database): validate restored pair ownership and provenance (#193)

* feat(admin): report pair scan outcomes and cancellation (#193)

* feat(ui): add paired prompt editor and explicit recipe library for #193

* fix(autocomplete): include reusable pair metadata vocabulary (#193)

* fix(pairs): honor recent sorting and conservative scan outcomes (#193)

* fix(ui): protect paired draft saves and clarify negative variants (#193)

* fix(metadata): preserve bounded paired execution evidence (#193)

* fix(tracking): follow actual pair output roles and prefer execution evidence (#193)

* docs(pairs): document live validation and retain renderer screenshots (#193)

* test(api): inject disposable databases before API startup (#194)

* test(api): canonicalize portable temporary gallery roots (#194)

* docs(ui): prototype branching negative variant history (#193)

* docs(ui): show selected negative beneath positive in timeline concept (#193)

* docs: retain approved timeline for later gallery redesign (#193)

* test(db): close restore fixture connections before cleanup (#194)

* test(db): reuse existing handle guards in all pair fixtures (#194)

* chore: clarify rollback test and intentional exception handling (#193)
2026-10-17 08:00:42 -07:00
Vito 3870590704 feat(ui): add Split Prompt Manager workspace (#192)
* feat(ui): add Split Prompt Manager workspace (#186)

* test(ui): use consistent unittest imports (#186)
2026-10-17 06:47:46 -07:00
Vito a52ff4e052 feat(ui): add Focused Prompt Manager with shared autocomplete (#191)
* feat(ui): add Focused Prompt Manager node (#186)

* feat(ui): autocomplete saved Focused metadata with shared controls (#186)

* feat(ui): autocomplete Focused library filters (#186)
2026-10-17 05:31:24 -07:00
Vito 035e816db6 feat(ui): adopt shared toolkit for Prompt Manager Classic (#190)
* feat(ui): adopt shared toolkit for Prompt Manager Classic (#186)

* test(ui): retain Text node Web UI coverage alongside Classic

* fix(tooling): reject lookalike toolkit download URLs (#186)
2026-10-16 22:18:18 -07:00
Vito f98f6fa08d test(ci): validate shared toolkit in real PromptManager hosts (#189)
* test(ci): prove shared toolkit with real PromptManager host coverage

* test(ci): match KikoTools Python 3.13 compatibility coverage
2026-10-16 20:56:50 -07:00
Vito ce82dfb912 fix: quiet routine console logging and prevent duplicate output (#183) 2026-10-16 09:58:05 -07:00
Vito 5ded3189c1 test: cover backup restore and destructive browser workflows (#185) 2026-10-16 07:56:15 -07:00
Vito f1b1d4731b test: exercise real page workflows and enforce frontend coverage (#181) 2026-10-16 06:45:20 -07:00
Vito e9fe586442 fix: repair prompt edit hashes, auto-tag review, and stale UI responses (#180)
* fix: keep edited prompt hashes consistent

Fixes #176

* fix: align gallery auto-tag review with scoped image APIs

Fixes #177

* fix: keep Tags prompt results on the current filter

Fixes #178

* fix: ignore stale metadata in admin image viewers

Fixes #179

* test: compare canonical gallery paths on macOS
2026-10-16 06:14:06 -07:00
Vito 6d9f3fc8fb fix: preserve gallery image identity and ignore stale metadata (#175)
* fix: preserve gallery root identity across media and metadata

Fixes #173

* fix: ignore stale gallery metadata responses

Fixes #174

* test: use platform-valid gallery filenames on Windows
2026-10-16 05:35:03 -07:00
dependabot[bot] 911b801883 deps(pip)(deps-dev): update aiohttp requirement (#172)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.5
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-16 05:23:48 -07:00
dependabot[bot] a576cff69f deps(pip)(deps-dev): update black requirement (#171)
Updates the requirements on [black](https://github.com/psf/black) to permit the latest version.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/26.10.0...26.10.1)

---
updated-dependencies:
- dependency-name: black
  dependency-version: 26.10.1
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-16 05:23:28 -07:00
Vito cf0cef3f25 feat: vendor Viewer.js reproducibly and automate release updates (#170)
Refs #169. Pin and verify pristine distribution assets, upgrade to 1.15.2, and add tested maintainer tooling plus desktop and touch browser checks.
2026-10-16 04:56:25 -07:00
Vito 7b6430e20a fix: stop misleading thumbnail prompts on gallery startup (#168)
* fix: count pending thumbnails accurately across gallery roots

* fix: clear thumbnails across all configured gallery roots

* fix: log skipped thumbnail directory cleanup
2026-10-16 04:56:07 -07:00
Vito 611c19d2f4 fix: keep infinite-scroll gallery rendering and viewer work bounded (#166) 2026-10-16 04:17:20 -07:00
ghosttigerllc-bitandkai 2763e9f9fa Read A1111/Forge generation parameters: fix the SELECT and parse multi-line prompts (#165)
* get_image_prompt_info: include gi.parameters in SELECT and return value

The function selects workflow_data, prompt_metadata and generation_time from
generated_images but omits the parameters column, so the A1111/Forge
generation-parameters string (steps, sampler, CFG, seed, LoRA list) never
reaches the API even when present in the database. Add it to the SELECT and
include it in the returned dict.

* parse A1111/Forge parameters: multi-line prompts, structured settings

_parse_comfyui_prompt took lines[0] of the parameters chunk as the whole
positive prompt, truncating every multi-line prompt to its first line, and
matched 'negative prompt:' only as a full line start. Split on the trailing
'Steps: ...' settings line instead, keep the full multi-line positive prompt,
and extract Steps / Sampler / Schedule type / CFG scale / Seed as structured
fields alongside the raw string.

Together with the get_image_prompt_info SELECT fix (previous commit), the
full A1111/Forge generation recipe now flows from the database to the API.

* style: apply black to the A1111 parameters parser

CI's black --check flagged py/api/__init__.py (slice spacing and two long
lines in the settings extraction). Formatting only; the AST is unchanged.

---------

Co-authored-by: kai <kai@hermes.local>
2026-10-16 03:50:26 -07:00
Vito Sansevero 3bb21c621c chore: release 3.3.1
- Fix Prompt Manager UI routing behind a ComfyUI proxy mount such as
  SwarmUI's /ComfyBackendDirect/ (#164, thanks @ghosttigerllc-bit)
- Restore runtime minimums Pillow>=8.0.0 and watchdog>=2.1.0: nothing
  requires the newer floors, and raising them forces upgrades in users'
  shared ComfyUI environment
2026-10-14 03:26:05 -07:00
ghosttigerllc-bitandZayne 374ce6e7a8 Fix Prompt Manager UI routing behind a ComfyUI proxy (#164)
* Fix gallery folder filters behind SwarmUI proxy

* Route Prompt Manager UI requests through proxy mount

* Use Comfy frontend base for Prompt Manager node requests

---------

Thanks @ghosttigerllc-bit

Co-authored-by: Zayne <noreply@gtb.ai>
2026-10-14 03:18:29 -07:00
dependabot[bot] e2ec6df3d4 deps(pip)(deps-dev): update numpy requirement from >=1.24 to >=2.2.6 (#162)
Updates the requirements on [numpy](https://github.com/numpy/numpy) to permit the latest version.
- [Release notes](https://github.com/numpy/numpy/releases)
- [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst)
- [Commits](https://github.com/numpy/numpy/compare/v1.24.0...v2.2.6)

---
updated-dependencies:
- dependency-name: numpy
  dependency-version: 2.2.6
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:46:02 -07:00
dependabot[bot] 38daedf131 deps(pip)(deps-dev): update flake8 requirement from >=7.0.0 to >=7.4.1 (#160)
Updates the requirements on [flake8](https://github.com/pycqa/flake8) to permit the latest version.
- [Commits](https://github.com/pycqa/flake8/compare/7.0.0...7.4.1)

---
updated-dependencies:
- dependency-name: flake8
  dependency-version: 7.4.1
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:45:59 -07:00
dependabot[bot] 1e1b92a1ba deps(pip)(deps-dev): update aiohttp requirement from >=3.8.0 to >=3.14.4 (#159)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.4
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:45:55 -07:00
dependabot[bot] ef66d7bd3e deps(pip)(deps): update watchdog requirement from >=2.1.0 to >=6.0.0 (#156)
Updates the requirements on [watchdog](https://github.com/gorakhargosh/watchdog) to permit the latest version.
- [Release notes](https://github.com/gorakhargosh/watchdog/releases)
- [Changelog](https://github.com/gorakhargosh/watchdog/blob/master/changelog.rst)
- [Commits](https://github.com/gorakhargosh/watchdog/compare/v2.1.0...v6.0.0)

---
updated-dependencies:
- dependency-name: watchdog
  dependency-version: 6.0.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:45:52 -07:00
dependabot[bot] f00d6408d8 deps(pip)(deps-dev): update pre-commit requirement (#163)
Updates the requirements on [pre-commit](https://github.com/pre-commit/pre-commit) to permit the latest version.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pre-commit/pre-commit/compare/v4.0.0...v4.6.2)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-version: 4.6.2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:32:43 -07:00
dependabot[bot] 6962fecf86 deps(pip)(deps-dev): update coverage requirement from >=7.0 to >=7.16.2 (#161)
Updates the requirements on [coverage](https://github.com/coveragepy/coveragepy) to permit the latest version.
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](https://github.com/coveragepy/coveragepy/compare/7.0.0...7.16.2)

---
updated-dependencies:
- dependency-name: coverage
  dependency-version: 7.16.2
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:32:38 -07:00
dependabot[bot] 90d706c943 deps(pip)(deps-dev): update bandit requirement from >=1.7.0 to >=1.9.4 (#158)
Updates the requirements on [bandit](https://github.com/PyCQA/bandit) to permit the latest version.
- [Release notes](https://github.com/PyCQA/bandit/releases)
- [Commits](https://github.com/PyCQA/bandit/compare/1.7.0...1.9.4)

---
updated-dependencies:
- dependency-name: bandit
  dependency-version: 1.9.4
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:32:31 -07:00
dependabot[bot] d17be56e1b deps(pip)(deps-dev): update black requirement from >=24.0.0 to >=26.10.0 (#157)
Updates the requirements on [black](https://github.com/psf/black) to permit the latest version.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/24.1a1...26.10.0)

---
updated-dependencies:
- dependency-name: black
  dependency-version: 26.10.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:32:28 -07:00
dependabot[bot] 94366938d1 deps(pip)(deps-dev): update setuptools requirement from >=61 to >=84.0.0 (#155)
Updates the requirements on [setuptools](https://github.com/pypa/setuptools) to permit the latest version.
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v61.0.0...v84.0.0)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-version: 84.0.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:32:23 -07:00
dependabot[bot] e13ea3c92f deps(pip)(deps): update pillow requirement from >=8.0.0 to >=12.3.0 (#154)
Updates the requirements on [pillow](https://github.com/python-pillow/Pillow) to permit the latest version.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/8.0.0...12.3.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 12.3.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 07:32:20 -07:00
Vito da322f8c94 Hardening: security fixes, queue-time run counting, background jobs, 85%+ test coverage (#153)
* chore: valid packaging metadata, flake8 config file, ignore local config with secrets

pyproject.toml carried requirements.txt comment lines as dependencies, which
fails PEP 508 parsing. flake8 never read [tool.flake8] from pyproject, so the
settings move to .flake8 and pre-commit runs the full rule set. config.json
(holds the CivitAI API key and machine paths) and other local artifacts are
now ignored. coverage joins the dev requirements.

* ci: test on Linux, Windows and macOS with a coverage gate and lint job

* refactor: drop validate_workflow_name, the column it validated no longer exists

* chore: keep agent worktrees out of coverage measurement

* fix: image routes serve only media files inside allowed directories

serve_image and serve_output_image now reject anything whose extension
is not in IMAGE_EXTENSIONS, fail closed when no output directory is
configured, and answer 403 (not 404) for dot-dot, absolute and
symlink-escaping paths. Containment is checked after realpath+normcase
so it behaves the same on Windows, macOS and Linux. 500 responses use
safe_error_message so OSError text no longer leaks absolute paths.

* fix: bound page sizes on list endpoints

/search, /recent and /images/recent share parse_page_params: limit is
clamped to [1, MAX_PAGE_LIMIT=500], offset to >= 0, and non-integer
values answer 400 instead of 500. /search now honours offset and all
three return a pagination block with the effective limit and offset.

* perf: thumbnail generation no longer blocks the event loop

generate_thumbnails_with_progress is split into _thumbnail_targets
(scan, executor), _generate_one (one PIL/ffmpeg job, executor) and an
async driver that awaits each file via _run_in_executor, emits one
progress frame per file and yields between files. Per-file failures
are reported as file_error events (error stays fatal for the UI) and
the loop continues. The wildcard Access-Control-Allow-Origin header is
gone. The blocking POST variant reuses the same helpers. The scan never
follows symlinks and is capped by MAX_SCAN_DEPTH / MAX_THUMBNAIL_FILES.

* fix: cap every request-driven resource in the image routes

- /images/recent, /images/all and /images/output parse limit/offset
  through parse_page_params on every path (absent, 0, negative, float
  strings), and the clamped values are what reach the DB. /images/all
  gets its own MAX_BULK_LIMIT ceiling instead of an unbounded query;
  /images/recent caps its fetch window at MAX_RECENT_IMAGES_WINDOW.
- parse_page_params also clamps offset to MAX_PAGE_OFFSET so values past
  int64 no longer surface as a 500 from SQLite.
- Gallery and thumbnail scans share _iter_media_files: no symlink
  following, MAX_SCAN_DEPTH, and MAX_GALLERY_FILES / MAX_THUMBNAIL_FILES
  caps. Thumbnail size comes only from named presets.
- JSON bodies on /images/link and /images/generate-thumbnails are read
  in bounded chunks; over MAX_JSON_BODY_BYTES answers 413, malformed or
  non-object JSON answers 400.
- /images/link only accepts media paths inside the allowed directories
  (403 otherwise), closing a path-existence oracle.

* test: cover image and prompt routes

Every route in py/api/images.py and py/api/prompts.py now has a success
path and an error path test: save/update/delete/rate/tag/category/bulk
endpoints, image list/link/unlink/delete/metadata/scan endpoints, the
thumbnail SSE stream, clear-thumbnails, and the video thumbnail chain
(fake cv2 success/failure, ffmpeg absent or failing, PIL placeholder).
Combined statement+branch coverage: images.py 97.7%, prompts.py 97.4%.

* fix: autotag endpoints no longer read arbitrary paths or run on GET

* fix: CivitAI key is never returned or sent off-host

* fix: autotag/single checks real paths and DB rows against output dirs

* fix: CivitAI downloads never follow redirects

* refactor: autotag model loading is injectable

* test: cover autotag and lora integration

* fix: saving LoRA settings without a key keeps the stored CivitAI key

* refactor: share the internal error message in autotag routes

* fix: never treat negative-only PromptManager nodes as the run prompt

run_prompt_nodes fell back to every PromptManager node whenever no positive
node was traced, including graphs that do have a sampler and whose only
PromptManager node feeds its negative input. Fall back only when the graph
has no recognisable sampler (new has_sampler helper). Samplers named like
one are recognised even without a model link. Adversarial graphs (cycles,
10k-node fan-out, non-dict inputs) are covered.

* fix: do not follow negative text inputs when tracing the positive prompt

_TEXT_KEY matched input names such as text_negative, so a styler node feeding
a sampler's positive input pulled its negative PromptManager node into the
positive trace. Input names matching (^|_)neg(ative)?($|_) are now skipped
(new follows_input helper).

* fix: count prompt runs when ComfyUI dequeues the prompt, not when it is submitted

The on-prompt hook runs before ComfyUI validates the request, so counting
there bumped run_count for prompts that were rejected or deleted from the
queue. The hook now only resolves the positive PromptManager texts, mints a
UUID4 prompt_id when the client sent none, and stashes the hashes under that
id (QueuedRuns, bounded to 1000 entries). PromptQueue.get is wrapped
idempotently; when a prompt is dequeued for execution its stashed runs are
counted (or remembered as pending first use). Servers without a prompt_queue
fall back to counting at submit time with one warning.

__init__ defines _global_db before the monitoring block and passes a factory
that raises RuntimeError while the database is unavailable; the package is
now imported under test with server and folder_paths stubbed.

* fix: keep batch image queue aligned when unrelated workflows save images

The metadata strategy returned None for images whose embedded workflow has
no positive PromptManager node, so the handler popped the batch FIFO for
them and every later batch image linked to the wrong prompt. It now returns
a tri-state: linked (prompt found), foreign (no PromptManager prompt in the
workflow, so the image is skipped and no fallback applies) or unknown (only
then the queue, snapshot, tracker and recent-prompt fallbacks run).

* fix: image monitor validates directories before creating the observer

start_monitoring assigned self.observer before checking that any directory
existed, so with no valid directory the monitor was stuck "already running"
and stop_monitoring raised on join of a never-started observer. Directories
are validated first and only a started observer is kept; stop_monitoring is
safe to call at any time and always joins. New is_monitoring property.

The per-image threading.Timer is replaced by one daemon worker: paths are
deduplicated and bounded (MAX_PENDING_IMAGES), processed in creation order
(batch images pop the prompt queue in order) after the processing delay and
once the file size is unchanged across two reads.

* refactor: one prompt hash implementation

utils.hashing.generate_prompt_hash is now the only prompt hash:
PromptManagerBase._generate_hash delegates to it and the inline
hashlib.sha256 copies in the prompt tracker and image monitor are gone.

Remove PromptManagerBase._open_web_interface (it targeted a web_interface
folder that never existed) and the unused restart_gallery.py script.

IS_CHANGED on PromptManager and PromptManagerText now includes category and
tags, so editing either re-runs the node and updates the saved metadata.

* fix: stop ComfyUI from loading the admin bundle into the canvas page

WEB_DIRECTORY = "web" made ComfyUI inject every .js under web/ (admin.js,
gallery.js, vendored Tailwind and ViewerJS) into the canvas page. The node
extension moves to web/comfy/prompt_manager.js (content unchanged; its
../../scripts/app.js import resolves against the served URL, which is the
same) and WEB_DIRECTORY points at that folder only. Admin and gallery assets
are still served from web/ by the extension's own routes.

tests/test_sort_contract.py still reads web/prompt_manager.js and must be
updated to the new path by its owner.

* fix: image monitor recent-prompt fallback reads the paginated result

get_recent_prompts returns {"prompts": [...]} but _get_fallback_prompt
indexed it like a list, so the last-resort link to the most recent prompt
always failed silently. Accept both shapes.

* test: cover prompt_manager_base, prompt_manager and prompt_search_list

Node base: init wiring, metadata update on re-save, run counting, database
error paths, tag parsing, search and recent API wrappers, gallery start
failure, LoRA trigger-word injection (disabled, missing modules, enabled).
PromptManager: missing CLIP error, encoding of the combined text with a fake
CLIP, tracking failure tolerance, empty text. PromptManagerText: tracking
failure tolerance. PromptSearchList: filters, multipart and LoRA-only
skipping, newline collapsing, preview, empty and error results.

* fix: diagnostics dependency check survives a watchdog without __version__

check_dependencies read watchdog.__version__ for its log line; recent
watchdog releases have no such attribute, so the AttributeError escaped and
took the whole diagnostics run down. Read it with a default.

Also covers utils/diagnostics.py: database and images-table checks against
a real temporary database, file system check, ComfyUI output detection
(folder_paths, configured directories, relative fallback, nothing found),
dependencies, the full run and the test image link.

* test: cover utils/logging_config

Isolated logger manager on a temporary log directory: memory buffer order,
level filter and 1000-entry cap, rotation and backup listing, tail and whole
file reads, path traversal rejection, truncation (including removal
errors), level changes across prompt_manager loggers, handler rebuilds.

* test: cover utils/comfyui_integration

Singleton re-initialisation keeps registrations, latest prompt per thread,
cross-thread fallback with the five-minute recency limit, global lookup for
the current thread, and cleanup of old registrations.

* test: cover __init__

Version parsing from pyproject (and the unknown fallback), and a server
whose add_on_prompt_handler raises: nodes and monitoring still come up.

* docs: point at the moved canvas extension under web/comfy

* fix: give every thread its own sqlite connection

PromptModel.get_connection() handed every thread the same
check_same_thread=False connection, so a commit or rollback from the
watchdog, executor or queue-hook thread could land on another thread's
half-done write (sqlite3 raised 'bad parameter or other API misuse'
under concurrent saves). Connections now live in threading.local, each
with WAL, Row factory, foreign_keys and busy_timeout, and are tracked so
they can be closed. PromptModel.close() / PromptDatabase.close() close
the calling thread's connection so tests and restores can delete the
file on Windows.

* fix: saving an existing prompt concurrently returns its id

Node execution, the queue hook and the image monitor all do
get_prompt_by_hash() then save_prompt(). When two of them race on the
same text the second INSERT failed on UNIQUE(hash) and surfaced an
IntegrityError. save_prompt now catches it, re-fetches by hash and
returns the existing id, so eight threads saving one prompt end with a
single row and every record_prompt_use counted.

* fix: backups include WAL data and restores are integrity-checked

The database runs in WAL mode, so a shutil.copy2 of prompts.db silently
dropped every committed row still in prompts.db-wal; the backup
download and the pre-restore safety copy both did that. Restore also
overwrote the live file with connections open, read the whole upload
into memory before checking the 100 MB cap and never ran
PRAGMA integrity_check.

PromptModel.backup_database now uses the SQLite online backup API,
verify_database_file opens a candidate read-only and checks header,
integrity and the prompts table, and restore_from_file backs up the
live database, closes every thread's connection, copies the file,
removes stale -wal/-shm and re-runs migrations. The admin handlers
stream the upload to a temp file with an early size cutoff, reject
anything verify_database_file refuses with a 400 and keep the
{success: ...} JSON shape. The schema-init connection is now closed
instead of leaking.

* fix: migrations are idempotent and schema init runs once per process

Every PromptDatabase() (nodes, image monitor, API, diagnostics) re-ran
table creation and all migrations. Schema init is now cached per
normalised path (class-level set under a lock) and re-run only when the
file disappeared, after a restore, or after reset_schema_cache().

The copy-and-swap migrations drop a stale *_new table left by an
interrupted run, execute inside an explicit transaction with
PRAGMA foreign_keys = OFF (DROP TABLE prompts used to cascade-delete
every generated_images row), roll back cleanly and log and re-raise
sqlite3.Error instead of swallowing it.

* fix: image uniqueness by full path, escaped LIKE search, no usage stamp on save

generated_images was UNIQUE(prompt_id, filename) on the basename, so
ComfyUI's same-named files in different date folders were silently
ignored by INSERT OR IGNORE, and the lastrowid == 0 branch after it
never ran. The table now carries file_path (normcase/normpath/abspath
of the image path) with UNIQUE(prompt_id, file_path); the migration
collapses rows that normalise to one file and supersedes the old
basename migration, which would otherwise undo it on every start.

search_prompts and the tag search escaped nothing, so 100% matched
'100 percent' and a_b matched aXb; user text now goes through
escape_like with ESCAPE. save_prompt no longer stamps last_used_at, the
usage backfill runs only on the upgrade that adds the columns, and the
Recently Used / Most Used orders put never-run prompts (NULL) after run
ones instead of first.

* fix: escape LIKE wildcards in image prompt search

search_images_by_prompt interpolated the user's text into LIKE
unescaped, so '50%' also matched '50 percent'. It now goes through the
same escape_like helper as search_prompts. Also drops an unused import
and an unused cursor assignment flagged by pyflakes.

* test: cover database operations

Behaviour tests for export (json/csv/errors), duplicate detection and
merging, image metadata storage and lookup, missing-file cleanup,
statistics, bulk operations, orphan pruning, consistency checks,
subfolder listing, tag queries and management edge cases, legacy tag
column parsing, config-driven path resolution, model housekeeping and
the TEXT -> INTEGER prompt_id migration. database/operations.py and
database/models.py are both above 85% statement+branch coverage.

* test: isolate the clock and log buffer from background threads of other tests

* fix: gallery roots are restricted to ComfyUI directories

A gallery root could be set to '/' (or 'C:\') through POST
/prompt_manager/settings because the deny list only matched a few
system prefixes. The root was persisted to config.json and fed into the
image-serving containment check, which then allowed any file on disk to
be read through /prompt_manager/images/serve/.

- GalleryConfig.validate_gallery_root() accepts a directory only when its
  realpath lies inside a ComfyUI directory (folder_paths output/input/
  temp/user) or a directory listed in PROMPT_MANAGER_EXTRA_GALLERY_ROOTS;
  filesystem roots, the home directory, files and missing paths are
  rejected.
- save_settings returns 400 and persists nothing when any root fails.
- _get_all_output_dirs() filters configured roots through the validator
  and never falls back to a rejected root.
- PROMPT_MANAGER_CONFIG_PATH relocates config.json; load/save default
  to it.

* fix: log config accepts only known keys

POST /prompt_manager/logs/config forwarded the whole request body into
PromptManagerLogger.update_config(), so any key could be injected into
the logging configuration. The handler now accepts only LOG_CONFIG_KEYS
(level, max_file_size, backup_count, console_logging, file_logging,
buffer_size), type-checks each value (level name, non-negative int,
bool) and answers 400 listing unknown keys. Malformed or non-object
bodies are 400 instead of 500.

* fix: no absolute server paths in API responses

Settings, diagnostics, duplicate/output scans and the logging endpoints
returned absolute filesystem paths, and several error envelopes echoed
str(e) which for OSError carries the full path.

- _public_path() renders a path relative to the ComfyUI base directory
  (folder_paths.base_path, else the output dir's parent) or to the
  parent of an opted-in extra gallery root, falling back to the
  basename; _public_error() reduces OSError to strerror + basename.
- GET /settings returns gallery roots and monitored directories in that
  form; POST /settings resolves relative roots against the same anchors
  (GalleryConfig.resolve_gallery_root) so the GET output round-trips.
- Diagnostics output_dirs, monitored_directories and the missing-DB
  message no longer carry absolute paths; the image monitor is reached
  through the package-or-top-level import fallback instead of a
  three-dot relative import that failed outside the package.
- Duplicate scan 'path' is now relative to the output root and the
  delete endpoint accepts relative paths.
- /logs/files 'path' and /logs/stats 'log_directory' use the public
  form; unexpected errors answer with a generic message.

* perf: run duplicate and output scans off the event loop

The duplicate scan walked the output tree and hashed every file on the
aiohttp event loop, and the duplicate delete removed files inline; the
output scan ran its per-file database work one executor hop at a time.

- _find_duplicate_images_sync / _describe_media_file do the walk and
  hashing; find_duplicate_images() runs them via _run_in_executor.
- _delete_duplicate_images_sync / _delete_one_duplicate do the removal
  (absolute or output-relative paths) off the loop.
- The output scan is split into _collect_output_media_sync,
  _extract_batch_metadata_sync and _ingest_scan_batch_sync (one
  executor call per batch for all DB work) with a small SSE generator.
- Media collection is shared and de-duplicates by normcase'd path.

* refactor: settings reads the image monitor through its module

get_settings located the ImageMonitor singleton by scanning sys.modules
for any module with 'image_monitor' in its name, a workaround for the
package being importable under two identities. It now uses the same
try-package / except-top-level import as the rest of the API
(_current_image_monitor) and falls back to the configured roots when no
monitor is running.

* fix: clamp the logs limit and reject non-integers

GET /prompt_manager/logs accepted any limit up to 1000 and answered 500
when the value was not an integer. The limit is now clamped to [1, 500]
and a non-integer value is a 400.

* test: cover core API module and gzip middleware

* refactor: split diagnostics, maintenance and HTML serving into helpers

- run_diagnostics composes four module-level probes (_diagnose_database,
  _diagnose_dependencies, _diagnose_output_dirs, _diagnose_image_monitor).
- run_maintenance is table-driven: _maintenance_runners maps each
  operation name to a callable and _run_maintenance_operations reports
  per-operation success or a public error message.
- _read_json answers 400 for malformed or non-object JSON bodies in
  settings, duplicate delete, test-link and maintenance instead of 500.
- test_image_link delegates to _link_test_image.
- The four HTML page routes share _serve_html_page.

* test: cover admin and core API routes

Route tests for stats, cleanup, duplicate scan/delete, settings,
diagnostics, test-link, maintenance (every operation, success and
failure), the output scan internals, and contract-level tests for
backup/restore (status codes and envelope only).

* test: make admin, logging and diagnostics tests independent of test order

The diagnostics message no longer carries the absolute path, the restore test
closes its raw sqlite connections before the live file is replaced, the logs
tests declare their own folder_paths base directory, and the isolated logger
filters every handler to the test thread.

* fix: truncating logs no longer rolls the main log over or pads it with zeros

truncate_logs emptied prompt_manager.log through a second open(), leaving the
rotating handler's stream at its old offset: the next record was written past
the new end (zero padding on POSIX) and tell() still reported the old size, so
the file rolled over and resurrected the backup that had just been deleted.
It also logged once per file from inside the deletion loop, which could do the
same with a small size limit. The main log is now truncated through the
handler's own stream and a single summary line is logged afterwards.

* fix: HTML escaping covers attribute context

escapeHtml now escapes & < > " ' (shared web/js/html-escape.js, loaded by
admin, gallery and metadata pages) instead of the textContent/innerHTML
round-trip that left quotes intact, so a tag, category, checkpoint or file
name containing a quote can no longer break out of a data-*, value or title
attribute.

Inline onclick handlers that interpolated server data now carry data-action
and data-* attributes and are dispatched through one delegated listener per
page (web/js/data-actions.js); the img onerror fallback uses
data-fallback-src with a capturing error listener. Every remaining template
value written into an attribute or text node (prompts, checkpoints, file
paths, log lines, directory names, categories, suggestions) is escaped.

metadata.html no longer builds an "explorer /select" shell command from the
file name (Electron-only branch); the path is copied to the clipboard.

* refactor: remove shadowed and dead frontend code

admin.js defined parsePNGMetadata, extractComfyUIData, showFullPrompt,
showWorkflowData, downloadWorkflowJSON and copyAllMetadata twice; the first
definitions were unreachable because the later ones shadow them.
updateMetadataPanel, showMetadataError, copyPrompt and tryFallbackMetadata
only ever wrote to #metadataContent, an element no page creates, so they are
gone too. gallery.js defined escapeHtml and showNotification twice. The admin
keydown/resize handlers referenced closeImageViewer, previousImage, nextImage
and applyImageSizing, none of which exist, and the imageViewMode state they
used; only the Escape-closes-gallery branch remains.

Debug console.log calls are removed; the ones that were the only statement of
a catch block are console.warn so parse failures stay visible.

* fix: stale responses, double submits and leaked listeners

web/js/api-client.js wraps fetch with one failure path (ApiError for non-2xx,
success:false and non-JSON bodies) and latestOnly(), which aborts the previous
in-flight request and resolves superseded calls to STALE. The admin search and
recent-page loads share one guard, the gallery page load has its own, so fast
typing or clicking can no longer paint an older result over a newer one.

In-place prompt editing used a blur listener with {once:true} plus a keydown
listener added on every edit, which sent the PUT twice and stacked handlers.
The list now has one keydown/focusout pair bound at construction and
finishEdit() clears the edit state before touching the element, so a save is
sent exactly once. Mutations re-fetch the page currently shown (refreshList)
instead of calling search(), and delete, bulk delete, edit, restore and scan no
longer reload the whole page.

Pagination state is committed only after a successful fetch via the pure
ListState.nextPageState(current, response) in both admin and gallery, so a
failed or aborted page request leaves the current page intact. The gallery
lightbox removed its keydown handlers on a non-existent 'remove' event; a
MutationObserver now drops them when the modal leaves the document.

web/js/sse-stream.js reads server-sent events over fetch so POST-only progress
endpoints can stream.

* fix: follow the POST-only autotag streams and the new LoRA and list contracts

autotag/download and autotag/start are POST-only now; EventSource can only
GET, so both pages stream their progress through SseStream.connect(url,
{method: 'POST'}), which keeps the onmessage/onerror/close surface the
handlers already use. autotag/single is called with image_id and the re-tag
dialog shows the server's filename instead of splitting an absolute path.

lora/status reports has_civitai_api_key instead of the key: the field shows a
'key is stored' placeholder, an empty field keeps the key on save and a
'Clear the stored key on save' checkbox sends clear_civitai_api_key.

Thumbnail generation reports per-file failures as file_error events; both
progress views count them (and show error_count from progress) without
closing the stream. The auto-tag review follows pagination.has_more on the
capped /images/all endpoint, and /search is paged like /recent so paging in
search mode no longer switches the list back to recent prompts.

* refactor: one PNG metadata parser

admin.js, gallery.js and metadata.html each carried a copy of the PNG chunk
walker that reads ComfyUI's tEXt/iTXt entries (admin.js had two until the
shadowed one was removed). web/js/png-metadata.js is now the only parser:
it checks the signature, bounds-checks every chunk so a truncated file or a
bogus length cannot throw, decodes tEXt as Latin-1 and iTXt as UTF-8, skips
compressed zTXt/iTXt payloads instead of decoding them as garbage, and stops
at IEND. The three page-level parsePNGMetadata functions delegate to it.

* fix: tags deep link renders and admin modals close

admin.html loaded tags-page.js after admin.js, so opening the page at #/tags
found no TagsPageManager and rendered nothing; the script order is now
theme-bridge, html-escape, api-client, png-metadata, (list-state, sse-stream,
data-actions), viewer, comfy-metadata, prompt-list-sort, view-router,
tags-page, admin. Route resolution lives in the pure ViewRouter.resolveView().

The auto-tag modal's close buttons called a bare hideModal(), which only
gallery.js defines; they call window.admin.hideModal(). PromptAdmin is
constructed from startAdmin() on DOMContentLoaded (or immediately when the
document is already parsed) instead of at script evaluation time.

* test: cover frontend modules

browser-globals.test.js loads every page module through vm with a fake
window, the way the pages do, and asserts the global each one exposes
(HtmlEscape/escapeHtml, DataActions, ApiClient, ListState, SseStream,
PngMetadata, ViewRouter, PromptListSort, ComfyMetadata). sse-stream gains
cases for a POST body, missing handlers and close() before or during
delivery. Every pure module is above 85% line coverage under
node --test --experimental-test-coverage.

* fix: page image listings in SQL and drop three unused helpers

get_recent_images and search_images_by_prompt take limit/offset so the recent
and search image routes no longer over-fetch a 5000-row window and slice it.
generate_content_hash, PromptModel.migrate_database and find_duplicates had no
callers outside their own tests. The isolated logger tests re-apply their
thread filter after update_config rebuilds the handlers. Long lines in the
touched database and image modules are wrapped so the full flake8 hook passes.

* fix: restore through SQLite's backup API, close_all for shared files, safer output scans

restore_from_file copied the uploaded file over the live database and deleted
its WAL while other PromptModel instances (nodes, image monitor, queue hook)
still held connections, which corrupts on POSIX and fails on Windows. It now
copies pages through the online backup API into the live connection so every
open handle sees the new content. PromptDatabase.close_all() closes the
connections of every thread, which tests need before deleting a database.
Output scans skip symlinked files, filesystem entries get a stable digest id
instead of a per-process hash(), and thumbnail streaming stops when the
client disconnects.

* style: wrap long lines in node modules

Split tooltip, description and docstring strings that exceed 88 columns
using implicit concatenation; string values are unchanged. Drop the unused
typing imports from prompt_search_list and mark the deliberately late node
imports in __init__ with noqa: E402.

* style: wrap long lines in the API layer, autotag and config

Split long f-string log messages, response messages, URL builders and
docstrings; DEFAULT_PROMPT and SYSTEM_PROMPT are rewritten as implicit
concatenations with identical values. Drop the unused shutil import in
admin and the unused HFCacheInfo import in autotag; keep the
hf_hub_download/snapshot_download availability probe with noqa: F401 and
mark the deliberately late imports in config with noqa: E402.

* style: wrap long lines and drop unused imports in utils

Re-wrap module docstrings, comments and f-string log messages; remove the
unused typing.List and PngInfo imports; mark the PIL availability probe in
diagnostics with noqa: F401; turn two placeholder-less f-strings into plain
strings; strip trailing whitespace inside the diagnostics SQL.

* style: wrap long lines in tests

Split long docstrings and SQL literals (values unchanged) and drop the
unused os import from test_path_security.

* chore: flake8 ignores the gitignored standalone tagger script

* fix: escape image dimensions and workflow JSON before rendering

openGalleryModal interpolated width, height and file_size (stored from a
POST body, so possibly strings) straight into data-caption and the card
text. showWorkflowData wrote JSON.stringify(workflow) raw into a blank
same-origin window, so a workflow containing </pre><script> ran. Both now
go through ImageHelpers (new pure module, node:test covered); a string
file_size no longer reaches formatFileSize's toFixed either, and a blocked
popup shows a notice instead of throwing.

* fix: escape remaining server values rendered through innerHTML

Maintenance and diagnostics results, review-tag errors, video info and
metadata error messages, consistency issues, recent image filenames,
output directory names, the prompt rating attribute, gallery media types
and duplicate group headers were interpolated unescaped. The copy
fallback modal also inlined the prompt text inside <textarea>, which a
closing tag could break out of; it now sets textarea.value.

* fix: address gallery auto-tag review images by path, not digest id

The gallery review flow builds its list from /prompt_manager/images/output,
whose id is a 16-hex path digest, but posted it as image_id to
autotag/single, so the server looked up a database row and every request
404ed. autotagSingleBody (pure, tested) now sends path for output-folder
entries and keeps image_id for database rows, which the admin review flow
(/images/all) still uses.

* fix: read thumbnail generation progress over POST with SseStream

images/generate-thumbnails/progress is becoming POST-only, which the
browser EventSource cannot request. Both thumbnail progress consumers in
the gallery now use SseStream.connect(url, {method: 'POST'}) exactly as
the autotag streams do. SseStream gains addEventListener/removeEventListener
for named event: frames (tested) so the status, start, progress,
file_error, complete and error listeners stay as they were, and an onclose
guard rejects when the stream ends without a terminal frame.

* fix: drop the unused copy-text data-action handler

Nothing in the admin markup emits data-action="copy-text"; the prompt
copy buttons call copyToClipboard directly.

* fix(lora): confine LoraManager paths to custom_nodes and hide server paths

- is_civitai_url also refuses URLs with userinfo or a non-443 port, so
  https://evil.com\@civitai.com/x and https://civitai.com:8443/x are rejected.
- resolve_lora_manager_path / custom_nodes_directories: lora/enable and
  lora/scan only accept a LoraManager directory that is a direct child of
  ComfyUI's custom_nodes (folder_paths base_path or registered custom_nodes
  folders, else this package's parent) and whose name contains "lora",
  compared via realpath+normcase; relative public paths resolve against the
  API path anchors. Anything else is 400, so find_lora_directories can no
  longer widen the image allow-list with arbitrary trees.
- lora/enable persists through PromptManagerConfig.save_to_file(), which
  honours PROMPT_MANAGER_CONFIG_PATH, instead of a hard-coded config.json.
- detect/status/enable return path and detected_path through _public_path;
  every lora handler reports exceptions via _public_error, and lora/scan
  errors raised after the SSE stream opened become an error event.
- Tests close the database before unlinking it.

* fix(autotag): contain batch rows, require media extensions, bound the scan

- The batch tagger runs every generated_images row through
  _resolve_allowed_image_path; rows that exist but lie outside the served
  output directories are skipped and counted in skipped_outside_roots on
  progress and complete events, and the engine receives the resolved path.
- _resolve_allowed_image_path requires a media extension (.png .jpg .jpeg
  .webp .gif .bmp .tiff .tif), so autotag/single returns 403 for other files.
- autotag/start holds a module-level lock; a concurrent start gets 409
  {"success": false, "error": "Auto-tagging is already running"}.
- scan_output_dir walks with a 50,000-file and 12-level cap, skips symlinked
  files and thumbnails, returns path through _public_path, relative_path in
  POSIX form, and a truncated flag.
- Autotag handlers report exceptions via the shared _public_error; the
  local copy is removed. Tests close the database before unlinking it.

* test: close every database before unlinking and anchor public paths in image tests

Each owned test that opens a PromptDatabase now calls close_all() before
removing the file, and temp directories holding a database use
ignore_cleanup_errors, so the suite tears down cleanly on Windows. The
image API fixture lays out a fake ComfyUI tree (<tmp>/output) and pins
GalleryConfig.path_anchors to it so responses can be checked for
root-relative paths.

* fix: validate image file_info before it reaches the generated_images columns

link_image_to_prompt stored size, dimensions and format straight from the
request body, so a string such as '"><img src=x>' ended up in INTEGER
columns and was echoed back to the gallery. Size and dimensions are now
coerced with int() and bounded (size 0..10^12, width/height 0..65535),
format is matched against a whitelist of media formats, and anything
else is stored as NULL while the image still links. Non-dict metadata or
file_info no longer aborts the link.

* fix: clamp paging on tag listings and the page-derived recent offset

tags/stats, tags/{tag}/prompts and tags/filter parsed limit/offset with a
bare int(), so limit=999999 was passed straight to SQLite and a huge
offset raised OverflowError (500). They now share parse_page_params
(limit 1..500, offset 0..MAX_PAGE_OFFSET, 400 on non-integers), and
/recent clamps the offset it derives from ?page= the same way.

* fix: make the thumbnail progress stream POST-only

GET /prompt_manager/images/generate-thumbnails/progress wrote thumbnails
to disk, so any prefetch, link preview or <img src> could start a full
output-directory scan. The route now accepts POST only (GET answers
405); quality still comes from the query string so the SSE client can
keep the same URL shape.

* fix: describe images relative to the ComfyUI tree in API responses

Gallery and prompt responses carried absolute server paths: /images/output
entries (path, root_dir), thumbnails_path, the images/prompt lookup, every
image list (image_path plus its normalised file_path duplicate) and the
preview images attached to prompt listings. They now go through
_public_path: /images/output 'path' is relative to its root (what the
delete and autotag endpoints resolve against), 'root_dir' and every
other image_path are relative to the ComfyUI base, and file_path is
dropped. recent/all/search image lists also gain url/thumbnail_url via
the same enrichment the prompt-images route already had. Remaining
str(e) error bodies in both modules use the basename-only error text.

* fix: check thumbnail directory containment with a real path test

clear_thumbnails compared resolved paths with str.startswith, so a
thumbnails directory that resolved to a sibling with a shared prefix
(out vs out2) passed the safety check and was emptied. It now uses the
module's _is_within (realpath + normcase + Path.is_relative_to).

* fix: decode tag names from the URL only once

The tag lookup, rename and delete routes ran urllib.parse.unquote on
match_info values that aiohttp had already percent-decoded, so a tag
literally named '%41' (requested as %2541) was handled as 'A'. The
second decode is dropped.

* fix: validate tags on every tag mutation route

add_prompt_tag, add_tags_to_prompt, bulk_add_tags, merge_tags and
rename_tag accepted whatever the body carried; a non-string tag raised
AttributeError (500) and over-long or control-character tags were
stored. They now run utils.validators.validate_tags through a shared
tags_error helper and answer 400 with the validator's message. Blank
entries in add_tags_to_prompt are still skipped rather than rejected.

* fix: export every prompt by paging instead of truncating at 10000

/prompt_manager/export fetched a single page of 10000 prompts and
silently dropped the rest. It now walks the table in EXPORT_PAGE_SIZE
pages (1000) inside one executor job until a short page is returned.

* fix: gate the LoraManager serving roots and accept relative link paths

A LoraManager path hand-edited into config.json went straight into
find_lora_directories and so into the image-serving allow-list. When a
path is configured it must now pass lora_utils.resolve_lora_manager_path
(a custom_nodes child whose name contains 'lora'); otherwise, or when the
resolver is unavailable, no LoRA directories are served. An empty path
keeps the custom_nodes-only auto-detection.

POST /images/link now also accepts the relative forms the API hands out
(root-relative 'sub/a.png' and ComfyUI-relative 'output/sub/a.png'),
resolving them against the allowed roots and anchors; '..' segments in
a relative path are refused and unknown files are 404.

* fix: review flow uses the served image url and the lora-dirs test pins its config

Image paths in API responses are now relative to the ComfyUI tree, so the
auto-tag review page can no longer derive a serve URL from image_path; the
server always provides url. The lora-dirs serving test assumed an empty
LoraManager path, which only held in a checkout without a local config.json.

* test: stop leaking a mocked utils.comfyui_integration and mock the IS_CHANGED clock

test_image_monitor_singleton replaced utils.comfyui_integration in
sys.modules with a MagicMock and never restored it, so running it before
test_comfyui_integration failed eight tests. The mock was unused; drop it
and install the server stub with setdefault in every owned test module.

The PromptSearchList IS_CHANGED test slept 10 ms between two time.time()
reads; the Windows clock ticks every 15.6 ms, so it was flaky there. Mock
the clock with an incrementing counter instead.

* fix: resolve_text follows a node shared by two branches

The cycle guard used one visited set for the whole walk, so a string node
read twice (a diamond, or StringConcatenate with the same source on both
inputs) made the second branch None and the prompt text unknowable. Cycle
detection is now scoped to the current path; a separate visit counter
keeps MAX_VISITS as the bound on total work.

* fix: bound resolve_text on diamond chains with memoisation, depth and length caps

A per-path visited set alone makes a chain of diamonds (node i reads node
i+1 on both inputs) exponential in depth. Each node's text is now
memoised so it is expanded once, every expansion is charged to the shared
MAX_VISITS budget, chains deeper than MAX_DEPTH (64) are unknowable so
recursion stays bounded, and text longer than MAX_TEXT_LENGTH (100k) is
unknowable so StringConcatenate doubling cannot exhaust memory.

* fix: static routes resolve under web/ with an extension allow-list

/prompt_manager/lib/{filepath} and /js/{filepath} only rejected '..' and a
leading '/'. On Windows os.path.join(root, 'C:/x') or a leading backslash
re-anchors the path, so any file on the host could be read. Requests are
now refused when the path is absolute on any platform (slash, backslash,
drive letter, UNC prefix), carries an empty, '.' or '..' segment, has an
extension outside the allow-list, or resolves (symlinks included) outside
the static directory; the resolved real path is served with its MIME type.

* feat: capped JSON body reader and single-flight lock for long admin jobs

Add _read_json_body(request, max_bytes=1_000_000) to py/api: reads the
body in chunks, answers 413 past the cap (declared or streamed) and 400
for invalid or non-object JSON. Every JSON POST in the admin and logging
handlers goes through it; it is exposed as PromptManagerAPI._read_json_body
for the other route mixins.

Add _acquire_long_job(name) / _long_job_busy_response(label) on the API
object: one asyncio.Lock per job family, handed out as an async context
manager or None when busy, so a handler answers 409 instead of starting a
second scan. The duplicate scan and delete share the 'duplicates' family.

* fix: harden database restore and keep server paths out of admin responses

Restore: verify_database_file and restore_from_file run through
_run_in_executor; the upload is also opened read-only with
PRAGMA trusted_schema=OFF and refused when sqlite_master lists a trigger
or a view (a prompts database has neither, a crafted one could use them
to run SQL against the live data); at most MAX_SAFETY_BACKUPS (10)
'<db>.backup_*' files are kept next to the live database, oldest first.

Responses: backup_created and the maintenance statistics database_path
go through _public_path, and backup/restore failure messages through
_public_error, so no absolute server path appears in an admin body.

* fix: validate hand-edited gallery roots everywhere and bound output scans

A root written straight into config.json skipped the containment check
the settings endpoint applies. GalleryConfig.update_config (and so
load_from_file) now drops entries that fail validate_gallery_root with a
warning, _find_comfyui_output_dir validates the configured directory
before using it, and ImageMonitor.start_monitoring validates every
directory before scheduling the observer.

The duplicate cleaner only deletes files with a media extension and
never anything under a thumbnails directory. _collect_media_files walks
with os.walk(followlinks=False), skips symlinked files and directories,
prunes thumbnails, stops MAX_SCAN_DEPTH (12) levels down and after
MAX_SCAN_FILES (50000) files.

* fix: bound log file size and backup count in the logging config endpoint

max_file_size must be between 1 MB and 1024 MB (bytes) and backup_count
between 0 and 50; anything else is a 400. Also close every database
connection in the logging test teardown.

* fix: private config.json, merged settings save, synthetic test-link path

PromptManagerConfig.save_to_file and the settings endpoint write
config.json through write_private_json: created with mode 0600 (chmod on
POSIX, best effort on Windows) because the file can hold an API key.
_persist_settings merges web_ui and gallery.monitoring.directories into
the existing file instead of overwriting it, so the integrations section
and any other key survive a settings save.

The test-link diagnostic no longer stores a client-supplied path; it
records a marker file name under the output directory.

* fix: image worker thread exits on shutdown and restarts cleanly

ImageGenerationHandler.shutdown() clears the pending set, sets a stop
event the worker's delay and settle waits listen to, and puts a None
sentinel on the queue; the worker exits within the current wait. Each
worker owns its queue and stop event so a draining worker never swallows
an image scheduled after shutdown. ImageMonitor.stop_monitoring calls it.

* test: close all database connections before deleting temp files

Windows refuses to unlink an open SQLite file and executor threads keep
their own connections, so every owned test that creates a PromptDatabase
calls close_all() before cleanup; the diagnostics temp dir ignores
cleanup errors.

* fix: output scan runs as a background job that survives the browser going away

Closing the dashboard tab or clicking outside the scan modal raised
ClientConnectionResetError out of scan_images and abandoned the scan
mid-batch, leaving a half-scanned database. Two overlapping scans could
also run at once.

- py/api/scan_job.py: ScanJob fans progress out to any number of SSE
  subscribers (latest snapshot first, then live events)
- POST /prompt_manager/scan starts the scan as an asyncio task, or attaches
  to the one already running; a write to a dead socket ends that stream
  quietly while the scan continues
- GET /prompt_manager/scan/status reports the running flag and last event
- dashboard: dismissing the modal while a scan runs minimizes it to a
  progress pill, the page re-attaches to a running scan after a reload,
  a lost stream reconnects, and leaving the page mid-scan asks first

* feat: worker-thread setting speeds up the output scan and thumbnail generation

Both jobs processed one file at a time on a single core. Pillow decodes the
whole image to reach trailing PNG text chunks and releases the GIL while it
does, so a bounded thread pool scales almost linearly: on a 1 GB sample of
400 PNGs the metadata pass dropped from 18 s to 2.5 s at 8 threads, and 120
thumbnails from 4.6 s to 0.6 s.

- utils/parallel.py: map_parallel(func, items, workers) keeps order, runs on
  the caller's thread when workers is 1, and caps the pool at the item count
- PromptManagerConfig.WORKER_THREADS defaults to half the detected cores,
  is clamped to max_worker_threads() when loaded from config.json, and is
  persisted under "performance"
- settings API returns worker_threads and cpu_count and validates saves;
  the settings modal gets a "Worker threads" slider capped at the core count
- the scan reads each metadata batch on the configured threads; thumbnail
  generation processes batches of files per executor job, still emitting
  one progress event per file

Threads rather than processes: this runs inside ComfyUI's server, where
forking a CUDA-initialised process is unsafe and the spawn start method
would re-import ComfyUI's main module in every child.

* fix: scan modal ignores backdrop clicks so a focus click cannot dismiss it

Only its own buttons dismiss it: Cancel before a scan starts, Hide (minimize
to the progress pill) while one runs.

* feat: scan reads prompts embedded in videos; unreadable files no longer log errors

The output scan handed every .mp4 to Pillow and logged one ERROR per file.
ComfyUI's video savers store the queued graph as JSON in the container's
comment tag, so those files carry prompts worth linking.

- utils/video_metadata.py reads the comment/description tag through
  ffprobe with a timeout, normalised to the PNG text-chunk shape so the
  existing parser handles both; without ffprobe videos are skipped and a
  single INFO notice is logged
- _extract_comfyui_metadata routes videos there and logs unreadable images
  as warnings, since the scan carries on past them

Test hygiene: every aiohttp test base class overrode tearDownAsync without
calling super(), so no test client was ever closed. That produced thousands
of ResourceWarnings, "Unclosed client session" errors from asyncio that
broke log assertions, and doubled the suite's run time.

* fix: thumbnail runs skip fresh thumbnails before the cap and report what remains

The per-run cap of 10,000 applied to the directory walk itself, so a
library with more files than that never had thumbnails generated for the
rest, and a run that found 9,987 existing thumbnails spent its whole
budget on skipping them.

- _plan_thumbnails walks the whole tree (bounded like the gallery listing),
  counts files whose thumbnail is already newer as skipped without queuing
  them, and caps only the pending files; the leftover count is returned
- start and complete payloads carry skipped and remaining; total_images
  counts every file considered and the processing rate uses the files
  actually worked on
- the gallery page shows a warning naming the pending count instead of a
  success toast when a run stopped at the cap

* feat(gallery): warn with the pending count when a thumbnail run stops at the cap

* fix: thumbnails are written atomically and served under mtime-versioned urls

Cards showed "Failed to load" for thumbnails that were valid on disk. The
browser had fetched them while the thumbnail run was still writing the
file and, because image responses are cached for an hour, kept the broken
bytes. The card also had no fallback to the original.

- image and video thumbnails are written to a hidden temp sibling and
  renamed into place, so a partial file is never visible at the final path
- every thumbnail_url (gallery listing, prompt images, duplicates, autotag
  listing) carries ?v=<mtime>, so a regenerated thumbnail gets a new URL
  instead of the cached old one
- a 404 from the image serve route is marked no-store; ComfyUI's cache
  middleware would otherwise let browsers cache the miss for an hour
- image-helpers gains fallbackImageSource(): the original is tried once
  before a card gives up

Test hygiene: the autotag tests now stub ComfyUI's server module like the
other API test modules, so the module passes when run on its own.

* fix(gallery): display settings apply at load, columns work, media info toggles

Default View Mode, Items per Page and Grid Columns were only applied when
Save was pressed, never when the page loaded; the column setting swapped a
Tailwind class that was never compiled (only 3, 6 and 8 exist) and a wider
breakpoint class overrode it anyway; Show Media Info was saved and never
read.

- gallery-settings.js validates what localStorage holds (view mode, limit
  option, column range, booleans) and falls back to defaults per value
- settings are applied in the constructor before the first load; changing
  the page size reloads, toggling media info re-renders
- the column count sets an explicit grid template through a CSS variable
  that wins on desktop widths
- media info lines honour the toggle in grid and list views
- image load failures are handled by one delegated listener that tries the
  original before showing the failed-to-load card

* feat(gallery): duplicate scan runs as a background job with progress; infinite scrolling

The duplicate scan was one blocking request with a spinner: no progress,
a modal that could not be put away, and a result lost if the tab closed.

- POST /prompt_manager/scan_duplicates/stream starts the scan as a
  background job (same ScanJob fan-out as the output scan) or attaches to
  the running one; GET /prompt_manager/scan_duplicates/status reports it.
  The job holds the duplicates lock, so deletes are refused while it runs.
- hashing runs on the configured worker threads and reports progress
  about every percent
- the gallery modal shows a progress bar, closing it minimizes to a pill
  that reopens it, results wait in the pill if the modal was closed, and a
  reload re-attaches to a running scan

Infinite scrolling is a new Display Setting (off by default): the next page
is appended when the viewport nears the bottom, pagination controls hide,
and the load-more rule lives in list-state.js where it is unit tested.
Thumb-size buttons and the column setting no longer override each other.

* fix(gallery): Rescan Output Folder drops the server listing cache and rewalks

The button only re-requested the listing, which the server answers from a
30-second in-memory cache, so a rescan showed exactly what was already on
screen.

- POST /prompt_manager/gallery/rescan invalidates the gallery cache,
  rewalks every output root and returns the file and root counts
- the button calls it, closes the settings modal, reloads the folder tree
  and the first page, and reports what it found

* feat: tag search box autocompletes, narrowed by the tags already typed

- PromptDatabase.suggest_tags(prefix, with_tags, limit): tags starting
  with the prefix, counted over the prompts that carry every context tag,
  context tags excluded, ordered by count then name; LIKE wildcards in the
  prefix are escaped
- GET /prompt_manager/tags/suggest?q=&with=&limit= exposes it (limit
  clamped to 1..50, context capped at 10 tags, registered before the
  {tag_name} routes)
- tag-autocomplete.js splits the comma-separated input into committed tags
  and the one being typed, builds the query, applies a pick and handles
  wrap-around keyboard selection
- the dashboard Tags box shows suggestions with counts on focus and while
  typing, arrow keys and Enter/Tab pick, Escape closes, and each pick
  immediately offers the next tag narrowed by the previous ones

* feat(dashboard): infinite scrolling for the prompt list, saved default plus session toggle

- PromptManagerConfig.INFINITE_SCROLL (web_ui.infinite_scroll) is reported
  and saved through the settings API; only booleans are accepted
- Settings modal gains an "Infinite Scrolling" default; saving applies it
  at once
- the results header gains an "Infinite scroll" checkbox that overrides the
  default for the current session without saving
- when on, the next page is appended as the list nears its bottom (same
  load-more rule as the gallery), page controls hide, a short page keeps
  loading until the list fills, and a refresh after an edit restarts from
  the top

* fix: first CI run on Windows and macOS; tests now enforce closed database files

The first push of the branch ran the Tests matrix on all three operating
systems. Every Python job failed; the causes were:

- numpy missing in CI: the WD14 engine tests import it (requirements-dev)
- macOS: temp directories resolve through /private; the admin API tests
  now use the real path like the gallery roots they compare against
- Windows: the startup banner printed an emoji, which raises
  UnicodeEncodeError on a legacy-code-page console and aborted loading
  the whole node; the banner is ASCII now and a test loads the package
  under a cp1252 stdout
- Windows: gallery subfolders came back with backslashes, which neither
  the folder tree nor the search filter understand; they are "/" joined
  on every platform. A relative DEFAULT_DB_PATH kept its forward slashes
  after being joined to the extension root; it is normalised now
- Windows: sqlite connections opened with `with sqlite3.connect()` are
  not closed by the context manager and linger until garbage collection,
  which holds a file lock there. Production sites (vacuum, diagnostics,
  restore inspection) close explicitly; a restore that swaps the API's
  database object closes the previous one; PromptModel keeps a registry
  so a shutdown or a test can close every instance
- Windows: served files were still open when their temp directory was
  removed (tests now read the body), a log download test compared text
  mode newlines, two tests compared paths with a hard-coded separator

tests/open_handles.py makes the Linux run fail the way Windows would:
every database test base checks, right before deleting its files, that
nothing in the process still holds them open.

* fix: Windows case-folded gallery roots in tests, "/" subfolders everywhere, longer busy timeout

Second CI run on Windows:
- gallery roots are stored resolved and case-folded there; the admin API
  tests now compare against that form instead of the mixed-case temp path
- the subfolder fallback for images outside every root also uses "/", and
  the test expects it
- a relative diagnostics db path is normalised after joining
- "database is locked" under many writers on a slow disk: connections now
  wait up to 30 seconds (connect timeout and busy_timeout), long enough for
  a scan or thumbnail run while ComfyUI saves images

* fix: serve every gallery video format, validate settings before applying, dedupe output dirs

Review findings on #153 (Codex):
- the image serve route allowed only three video types while the gallery
  lists seven, so .avi/.mkv/.m4v/.wmv cards opened to a 403; the serve
  allow-list now covers every gallery video format
- a settings payload with one invalid field still applied the valid ones
  before returning 400; the whole payload is validated first and nothing
  changes on failure. webui_display_mode is validated too

Third Windows CI run:
- the same output directory spelled twice (different case or a link) was
  listed twice in diagnostics and served roots; both dedupe on the
  resolved, case-folded path
- the root-less subfolder test expects the "/" form

Code-quality bot: index definitions are (name, target) pairs instead of
implicitly concatenated strings; redundant local asyncio imports and
two trivial lambdas in test stubs removed.

* docs: version 3.3.0, README changelog and refreshed sections

Bump to 3.3.0: new features (worker threads, background scans, tag
autocomplete, infinite scrolling, video prompt scanning) and API contract
changes (POST-only side-effect routes, relative image paths, paginated
search) go beyond a patch release.

README: changelog entries for 3.2.4 and 3.3.0, refreshed bulk scanning,
gallery settings, search, configuration and testing sections, and the
roadmap's recently-completed list.
2026-10-08 18:53:11 -07:00
Vito 6c7e4d5b35 feat: release 3.2.4 — recently used sorting, run counts, positive-prompt image linking (#152)
* chore: start 3.2.4

* feat: track prompt usage (run_count, last_used_at) with server-side sorting

- Migration adds last_used_at and run_count; existing prompts are estimated
  from linked images (count, newest image time, else created_at). The
  backfill only touches NULL rows, so it self-heals rows written after a
  downgrade or an interrupted upgrade.
- Every node run records a use, including re-runs of an existing prompt;
  a tracking failure is logged and never blocks generation.
- search_prompts/get_recent_prompts accept a whitelisted sort key
  (last_used, created, run_count, rating, text); default stays newest first.

Refs #90

* feat: sort param on /recent and /search; microsecond last_used_at

- /prompt_manager/recent and /search accept ?sort= (whitelisted in the DB
  layer; unknown values fall back to newest first).
- last_used_at is written with microseconds from Python so a re-run in the
  same millisecond as another save still sorts first; ordering stays correct
  against millisecond values from the backfill.

Refs #90

* feat: server-side sort in admin UI with Recently Used / Most Used and run badges

- New web/js/prompt-list-sort.js owns the sort options, validation and
  request URLs; admin.js fills the dropdown from it and re-fetches from the
  server on change, so every sort spans all pages (the old client-side sort
  only reordered the current page).
- Default sort is Recently Used; prompt cards show a run-count badge with
  the last-used time on hover.
- The node's Recent button now lists recently used prompts.
- Contract test keeps the UI sort keys and the node's key in sync with the
  server whitelist.

Refs #90

* fix: link images and count runs by the positive prompt, not node execution

With a negative-prompt PromptManager node in the workflow, every executed
PromptManager node pushed onto the image-linking queue, so images could be
linked to the negative prompt or to a previous run. ComfyUI also skips
unchanged nodes, so re-runs of the same prompt were never counted.

- utils/prompt_graph.py finds the PromptManager nodes feeding a sampler's
  positive input (KSampler, CFG/Basic guiders, ControlNet and combine
  pass-throughs, PromptManagerText into encoders), falling back to all
  PromptManager nodes when none can be traced.
- Runs are counted by a ComfyUI on-prompt hook when a workflow is queued;
  prompts not yet saved are counted when the node first saves them.
- Nodes receive the queued graph (hidden PROMPT/UNIQUE_ID). Only the run's
  positive prompt becomes the current prompt; only batch items with linked
  text are pushed onto the FIFO queue.
- The image monitor links from the image's own metadata first, using the
  positive node, and falls back to the queue for batch items.

Refs #90

* fix: address review — count cached runs of linked text, keep repeated pending runs

- Text from known pure string nodes (core PrimitiveString/StringConcatenate,
  WAS Text Concatenate/Multiline, pysssss ShowText) is resolved at queue
  time, so cached re-runs of such prompts are counted and their images link
  from metadata. Unknown nodes are never guessed; the node compares its real
  text with the queue-time resolution and falls back to counting and
  queueing itself on any mismatch.
- A new prompt queued several times before its first run keeps every
  pending run; the first save records all of them.
- Registration flag is a threading.Event; consistent unittest imports;
  logging instead of print in the init fallback.

* refactor: unquote PendingFirstUse annotation so Tuple usage is visible
2026-10-07 13:52:09 -07:00
525 changed files with 126601 additions and 5281 deletions
+4
View File
@@ -14,6 +14,10 @@ omit =
*/node_modules/*
*/static/*
*/media/*
*/htmlcov/*
*/.claude/*
restart_gallery.py
standalone_tagger.py
branch = true
parallel = true
+6
View File
@@ -0,0 +1,6 @@
[flake8]
max-line-length = 88
extend-ignore = E203, W503
exclude = reference/, .git, __pycache__, web/, venv/, htmlcov/, build/, dist/, .claude/, docs/, standalone_tagger.py
per-file-ignores =
tests/*:E402
+30
View File
@@ -0,0 +1,30 @@
{
"required_status_checks": {
"strict": true,
"contexts": [
"py3.10 on ubuntu-latest",
"py3.11 on ubuntu-latest",
"py3.12 on ubuntu-latest",
"py3.13 on ubuntu-latest",
"py3.10 on windows-latest",
"py3.11 on windows-latest",
"py3.12 on windows-latest",
"py3.13 on windows-latest",
"py3.10 on macos-latest",
"py3.11 on macos-latest",
"py3.12 on macos-latest",
"py3.13 on macos-latest",
"coverage gate (85%)",
"black and flake8",
"frontend modules",
"viewerjs",
"real ComfyUI compatibility",
"bandit (high severity)"
]
},
"enforce_admins": true,
"required_pull_request_reviews": null,
"restrictions": null,
"allow_force_pushes": false,
"allow_deletions": false
}
+12
View File
@@ -46,3 +46,15 @@ updates:
update-types:
- "minor"
- "patch"
# Maintainer browser test dependencies; Viewer.js itself has an asset-aware updater.
- package-ecosystem: "npm"
directory: "/tools/vendor"
schedule:
interval: "weekly"
allow:
- dependency-name: "@playwright/test"
labels:
- "dependencies"
commit-message:
prefix: "deps(vendor-tools)"
+13 -36
View File
@@ -9,42 +9,15 @@ on:
pull_request:
branches: [main]
# Superseded PR pushes cancel their older run. Every main push gets its own
# group (run_id), so no main commit's run is ever cancelled or dropped as pending.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Set up Python 3.10
uses: actions/setup-python@v7
with:
python-version: "3.10"
- name: Cache pip dependencies
uses: actions/cache@v6
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-quality-${{ hashFiles('requirements-dev.txt') }}
restore-keys: |
${{ runner.os }}-pip-quality-
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Check formatting with Black
run: black --check --diff .
- name: Lint with flake8
run: |
# Fail on syntax errors and undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# Warnings as non-blocking
flake8 . --count --exit-zero --max-line-length=88 --statistics --exclude=reference/,.git,__pycache__,web/
security:
name: bandit (high severity)
runs-on: ubuntu-latest
steps:
@@ -58,5 +31,9 @@ jobs:
- name: Install bandit
run: pip install bandit[toml]
- name: Security scan
run: bandit -r . -ll --exclude=./tests,./reference,./venv -q || true
# Medium findings are reported without failing until they are triaged.
# High severity is a hard gate.
- name: Security report (medium and above, informational)
run: bandit -r . -ll -q --exclude ./tests,./reference,./venv,./tools/vendor/node_modules --exit-zero
- name: Security scan (high severity gate)
run: bandit -r . -lll -q --exclude ./tests,./reference,./venv,./tools/vendor/node_modules
+172 -13
View File
@@ -8,13 +8,26 @@ on:
branches: [main]
push:
branches: [main]
workflow_dispatch:
# Superseded PR pushes cancel their older run. Every main push gets its own
# group (run_id), so no main commit's run is ever cancelled or dropped as pending.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
PYTHONFAULTHANDLER: "1"
jobs:
test:
runs-on: ubuntu-latest
name: py${{ matrix.python-version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v7
@@ -23,14 +36,8 @@ jobs:
uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip dependencies
uses: actions/cache@v6
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('requirements-dev.txt') }}
restore-keys: |
${{ runner.os }}-pip-${{ matrix.python-version }}-
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: |
@@ -38,13 +45,165 @@ jobs:
pip install -r requirements-dev.txt
- name: Run tests
run: python -m unittest discover tests/ -v
run: python -m unittest discover -s tests -v
js-test:
comfyui:
name: real ComfyUI compatibility
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: "24"
cache: npm
cache-dependency-path: tools/vendor/package-lock.json
- name: Read the reviewed runtime pins
id: runtime
run: |
node --input-type=module <<'JS' >> "$GITHUB_OUTPUT"
import runtime from './tools/vendor/comfyui-runtime.json' with { type: 'json' };
for (const [key, value] of Object.entries(runtime)) console.log(`${key}=${value}`);
import crossPack from './tools/vendor/cross-pack-runtime.json' with { type: 'json' };
console.log(`kikoRepository=${crossPack.repository}`);
console.log(`kikoRevision=${crossPack.revision}`);
JS
- name: Check out pinned ComfyUI
uses: actions/checkout@v7
with:
repository: ${{ steps.runtime.outputs.repository }}
ref: ${{ steps.runtime.outputs.revision }}
path: .ci-comfyui
persist-credentials: false
- name: Check out pinned KikoTools for compatibility
uses: actions/checkout@v7
with:
repository: ${{ steps.runtime.outputs.kikoRepository }}
ref: ${{ steps.runtime.outputs.kikoRevision }}
path: .ci-kikotools
persist-credentials: false
- uses: actions/setup-python@v7
with:
python-version: ${{ steps.runtime.outputs.pythonVersion }}
cache: pip
cache-dependency-path: |
requirements-dev.txt
.ci-comfyui/requirements.txt
tools/vendor/comfyui-runtime.json
- name: Install CPU runtime and test dependencies
run: |
python -m pip install --upgrade pip
pip install "torch==${{ steps.runtime.outputs.torchVersion }}" "torchvision==${{ steps.runtime.outputs.torchvisionVersion }}" --index-url https://download.pytorch.org/whl/cpu
pip install -r .ci-comfyui/requirements.txt -r requirements-dev.txt "torch==${{ steps.runtime.outputs.torchVersion }}" "torchvision==${{ steps.runtime.outputs.torchvisionVersion }}"
- run: npm ci --prefix tools/vendor --ignore-scripts
- run: npx --prefix tools/vendor playwright install --with-deps chromium
- name: Classic and Nodes 2.0 startup and compatibility
env:
PM_COMFY_ROOT: ${{ github.workspace }}/.ci-comfyui
PM_TEST_PYTHON: python
run: npm run --prefix tools/vendor test:comfyui
- name: Real cached and headless execution history
env:
PM_COMFY_ROOT: ${{ github.workspace }}/.ci-comfyui
PM_TEST_PYTHON: python
run: npm run --prefix tools/vendor test:executions
- name: Independent and combined pack compatibility
env:
PM_COMFY_ROOT: ${{ github.workspace }}/.ci-comfyui
PM_KIKO_ROOT: ${{ github.workspace }}/.ci-kikotools
PM_TEST_PYTHON: python
run: npm run --prefix tools/vendor test:compat
- name: Upload real-host reports and startup logs
if: always()
uses: actions/upload-artifact@v7
with:
name: real-comfyui-reports
path: tools/vendor/test-results/
if-no-files-found: ignore
retention-days: 14
coverage:
name: coverage gate (85%)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
# Pure-JS frontend modules, tested with the runner's preinstalled Node
- uses: actions/setup-python@v7
with:
python-version: "3.12"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Measure coverage
run: |
python -m coverage run -m unittest discover -s tests
python -m coverage combine
python -m coverage report --fail-under=85
perf-budget:
# The reference wall-clock measurement (TESTING.md "Performance budgets").
# Shared-runner timing varies, so it never gates a pull request: it runs on
# each push to main and on demand. A red run on main means fix the code.
name: perf budget
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.13"
cache: pip
cache-dependency-path: requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
- name: Workspace filter budget
env:
PROMPTMANAGER_PERF_BUDGET: "1"
run: python -m unittest -v tests.test_api_workspace_images.TestWorkspaceImages.test_filter_budget
lint:
name: black and flake8
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install black flake8
- run: black --check .
- run: flake8 .
js-test:
name: frontend modules
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: "24"
- name: Check shipped extension and page modules parse
run: |
shopt -s globstar nullglob
for f in web/comfy/**/*.js web/comfy/**/*.mjs web/js/**/*.js web/js/**/*.mjs; do
node --input-type=module --check < "$f" || exit 1
done
- name: Install pinned shared test contracts
run: npm ci --prefix tools/vendor --ignore-scripts
- name: Run JS tests
run: node --test tests/js/*.test.js
+128
View File
@@ -0,0 +1,128 @@
name: Viewer.js vendor and compatibility
on:
pull_request:
branches: [main]
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
# Superseded PR pushes cancel their older run. Every main push gets its own
# group (run_id), so no main commit's run is ever cancelled or dropped as pending.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
frontend-coverage:
name: frontend coverage (chromium and touch)
runs-on: ubuntu-latest
# Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
# same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
# tests/vendor/browser-image.test.mjs checks it). Browsers and fonts come from the image.
container:
image: mcr.microsoft.com/playwright:v1.64.1-noble
options: --user 1001
# This job runs every page workflow twice (chromium, then touch) on one worker with V8 coverage on, so it
# takes about twice a single-browser job. Measured 2026-10-20: whole job 18m26s-24m57s (main 5ded3cf
# 22m15s; PR head 6359999 was cancelled at the old 25-minute limit while still passing), coverage step
# alone 20m55s-22m17s+, against ~12-15 minutes for the firefox/webkit jobs. 40 minutes leaves headroom
# for suite growth while still stopping a hung run; re-measure if a run passes 30 minutes.
timeout-minutes: 40
steps:
- uses: actions/checkout@v7
# The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
# builds, whose entry points name host paths that do not exist in the container. The image has no
# ensurepip and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv.
- name: Python environment
env:
PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
run: |
venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
python3 -m venv --without-pip "$venv"
curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
echo "$PIP_SHA256 $wheel" | sha256sum -c -
"$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
"$venv/bin/python" -m pip install --no-cache-dir -r requirements-dev.txt
echo "$venv/bin" >> "$GITHUB_PATH"
- run: python --version && node --version && node --version | grep -q '^v24\.'
- run: npm ci --prefix tools/vendor --ignore-scripts
- run: npm run --prefix tools/vendor check
- run: npm run --prefix tools/vendor test
- name: Frontend unit, viewer, workflow, and coverage checks
run: npm run --prefix tools/vendor test:coverage
- name: Upload frontend coverage and Chromium traces
if: always()
uses: actions/upload-artifact@v7
with:
name: frontend-test-reports
path: |
tools/vendor/test-results/
coverage/js/
!coverage/js/raw/**
retention-days: 14
browser-workflows:
name: page workflows (${{ matrix.browser }})
runs-on: ubuntu-latest
# Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
# same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
# tests/vendor/browser-image.test.mjs checks it). Browsers and fonts come from the image.
container:
image: mcr.microsoft.com/playwright:v1.64.1-noble
options: --user 1001
timeout-minutes: 25
strategy:
fail-fast: false
matrix:
browser: [firefox, webkit]
steps:
- uses: actions/checkout@v7
# The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
# builds, whose entry points name host paths that do not exist in the container. The image has no
# ensurepip and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv.
- name: Python environment
env:
PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
run: |
venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
python3 -m venv --without-pip "$venv"
curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
echo "$PIP_SHA256 $wheel" | sha256sum -c -
"$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
"$venv/bin/python" -m pip install --no-cache-dir -r requirements-dev.txt
echo "$venv/bin" >> "$GITHUB_PATH"
- run: python --version && node --version && node --version | grep -q '^v24\.'
- run: npm ci --prefix tools/vendor --ignore-scripts
- name: Page workflows in ${{ matrix.browser }}
run: npm run --prefix tools/vendor test:workflows -- --project ${{ matrix.browser }}
- name: Upload browser traces
if: always()
uses: actions/upload-artifact@v7
with:
name: browser-traces-${{ matrix.browser }}
path: tools/vendor/test-results/
retention-days: 14
# Branch protection requires the exact context "viewerjs". This job keeps that
# name and fails unless every frontend job above succeeded, including when one
# was cancelled or skipped.
viewerjs:
needs: [frontend-coverage, browser-workflows]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Require every frontend job to succeed
env:
COVERAGE_RESULT: ${{ needs.frontend-coverage.result }}
WORKFLOWS_RESULT: ${{ needs.browser-workflows.result }}
run: |
echo "frontend-coverage: $COVERAGE_RESULT"
echo "browser-workflows: $WORKFLOWS_RESULT"
test "$COVERAGE_RESULT" = success && test "$WORKFLOWS_RESULT" = success
+83
View File
@@ -0,0 +1,83 @@
name: Update vendored Viewer.js
on:
schedule:
- cron: '30 7 * * *'
workflow_dispatch:
inputs:
version:
description: 'Stable version to vendor; leave blank for the latest npm release'
required: false
type: string
permissions:
contents: read
concurrency:
group: viewerjs-update
cancel-in-progress: false
jobs:
update:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
actions: write
env:
GH_TOKEN: ${{ github.token }}
steps:
- uses: actions/checkout@v7
with:
ref: ${{ github.event.repository.default_branch }}
- uses: actions/setup-node@v7
with:
node-version: '24'
cache: npm
cache-dependency-path: tools/vendor/package-lock.json
- run: npm ci --prefix tools/vendor --ignore-scripts
- name: Update the dependency and regenerate assets
env:
VIEWER_VERSION: ${{ inputs.version }}
run: |
if [ -n "$VIEWER_VERSION" ]; then
npm run --prefix tools/vendor update -- --version "$VIEWER_VERSION"
else
npm run --prefix tools/vendor update
fi
- name: Detect changes
id: changes
run: |
if git diff --quiet -- tools/vendor/package.json tools/vendor/package-lock.json web/lib/viewerjs; then
echo 'changed=false' >> "$GITHUB_OUTPUT"
else
echo 'changed=true' >> "$GITHUB_OUTPUT"
fi
- if: steps.changes.outputs.changed == 'true'
run: |
npm run --prefix tools/vendor check
npm run --prefix tools/vendor test
node --test tests/js/*.test.js
npx --prefix tools/vendor playwright install --with-deps chromium
npm run --prefix tools/vendor test:browser
- name: Open the update PR
if: steps.changes.outputs.changed == 'true'
env:
BASE_BRANCH: ${{ github.event.repository.default_branch }}
run: |
version=$(node -p "require('./web/lib/viewerjs/provenance.json').version")
branch="deps/viewerjs-$version"
existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --state all --head "$branch" --json number --jq 'length')
if [ "$existing" != '0' ]; then
echo "A PR already exists for Viewer.js $version; leaving it for review."
exit 0
fi
remote=$(git ls-remote origin "refs/heads/$branch")
if [ -z "$remote" ]; then
git switch -c "$branch"
git add tools/vendor/package.json tools/vendor/package-lock.json web/lib/viewerjs
git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' commit -m "deps: update vendored Viewer.js to $version"
git push origin "HEAD:refs/heads/$branch"
fi
gh pr create --repo "$GITHUB_REPOSITORY" --base "$BASE_BRANCH" --head "$branch" --title "deps: update vendored Viewer.js to $version" --body-file tools/vendor/update-pr.md
gh workflow run viewerjs-check.yml --repo "$GITHUB_REPOSITORY" --ref "$branch"
+17
View File
@@ -9,6 +9,12 @@ AGENTS.md
CLAUDE.md
# Local files
# Local PromptManager state (config.json holds API keys and machine paths)
config.json
resume.sh
*.backup_*
promptmanager_logs/
custom_data/
data/
docs/
logs/
@@ -194,3 +200,14 @@ cython_debug/
# PyPI configuration file
.pypirc
# Maintainer-only JavaScript tooling and browser test artifacts
node_modules/
test-results/
playwright-report/
# Combined frontend coverage reports
coverage/
# Disposable upstream checkout for real-host CI
.ci-comfyui/
+1 -1
View File
@@ -9,7 +9,7 @@ repos:
- id: flake8
name: flake8
entry: flake8 --select=E9,F63,F7,F82 --show-source --exclude=reference/,web/
entry: flake8 --show-source
language: system
types: [python]
+112 -10
View File
@@ -139,6 +139,35 @@ The comprehensive gallery system provides complete management of all ComfyUI out
_Powered by [ViewerJS](https://github.com/fengyuanchen/viewerjs) for professional image viewing capabilities._
Viewer.js is vendored locally. See [Viewer.js maintenance](tools/vendor/README.md)
for reproducible updates, verification, and automated update PRs.
Prompt Manager appears as **Prompt Manager (Classic)** in the Add Node menu.
It keeps the existing layout and behavior while using shared
[ComfyAssets Toolkit](https://github.com/ComfyAssets/ComfyAssets-Toolkit) controls
that follow ComfyUI's light/dark theme. Existing workflows keep the same internal
node identifier and sockets. The reusable architecture is inspired by
[Pixaroma](https://github.com/pixaroma/ComfyUI-Pixaroma); no implementation or assets
were copied. See the [toolkit maintenance notes](tools/vendor/README.md#shared-comfyassets-browser-controls).
**Prompt Manager (Focused)** adds an editor-first layout with collapsible
composition and metadata fields. Open **Browse library** to search saved prompts,
filter by saved category and comma-separated tags, preview a selection, then choose
**Use prompt**; **Load metadata** controls whether
category and tags come with it. Metadata fields suggest saved categories and tags;
choose with the mouse or arrow keys and Enter. Tag selections append `, ` for the
next tag; category remains a single value. New values can still be typed freely.
Prepend and append STRING sockets remain visible
when the fields are collapsed, so LoRA trigger outputs can connect directly.
Encoding, storage, and CONDITIONING/text outputs match Classic. Both layouts
support ComfyUI Classic and Nodes 2.0.
**Prompt Manager (Split)** keeps the same editor and library visible together.
Wide nodes place the library beside the editor; narrow nodes stack the panels.
It shares Focused's category/tag autocomplete, search filters, preview-before-use,
workflow persistence, and connectable prepend/append inputs. Encoding and storage
match Classic and Focused.
### 🔬 ComfyUI Workflow Analysis
- **PNG Metadata Parsing**: Advanced PNG chunk analysis to extract embedded ComfyUI data
@@ -164,7 +193,7 @@ _Powered by [ViewerJS](https://github.com/fengyuanchen/viewerjs) for professiona
- **Full-text search** across all stored prompts
- **Category filtering** for organized prompt collections
- **Tag-based search** with support for multiple tags
- **Tag-based search** with support for multiple tags, with autocomplete that narrows to tags seen together with the ones already typed
- **Rating filters** to find your best prompts
- **Date range filtering** for temporal searches
- **Recent prompts** quick access
@@ -306,9 +335,12 @@ The comprehensive gallery system provides complete access to all your ComfyUI ou
Configure your viewing experience in the settings panel:
- **Performance Settings**: Grid columns, image quality, lazy loading
- **Display Settings**: Default view, items per page, grid columns, media info, infinite scrolling
- **Performance Settings**: Image quality, lazy loading
- **Video Settings**: Autoplay, mute by default, loop videos
- **Thumbnail Management**: Generate optimized thumbnails for faster loading
- **Thumbnail Management**: Generate optimized thumbnails for faster loading (runs on the configured worker threads; a run that stops at the per-run cap tells you how many remain)
- **Find Duplicate Images**: Hashes every file on the worker threads with a progress bar; close the window and it keeps running, results wait in a pill
- **Rescan Output Folder**: Drops the server-side listing cache and rewalks the folders
#### **🖼️ Image Viewing Experience**
@@ -358,14 +390,16 @@ Use the standalone metadata viewer to analyze any ComfyUI-generated PNG:
Import existing ComfyUI images into your database:
1. **Open admin dashboard** and click **"📸 Scan Images"**
2. **Start scan** to analyze your entire output directory
3. **Monitor progress** with real-time status updates
2. **Start scan** to analyze every configured output directory
3. **Monitor progress** with real-time status updates. The scan runs on the server: closing the window only minimizes it to a progress pill, and reloading the page re-attaches to a running scan
4. **Review results** showing:
- Images found and processed
- Prompts extracted and linked
- Any errors or issues encountered
5. **Access imported data** through the normal gallery interface
The scan reads PNG text chunks and A1111-style parameters, and, when `ffprobe` is installed, the prompt ComfyUI's video savers embed in MP4/MOV/WebM files. Files are read on the number of **worker threads** set in Settings → Performance (default: half your cores), which makes a large library several times faster.
### 🏷️ AI AutoTag
Automatically tag your entire image collection using AI vision models:
@@ -497,7 +531,7 @@ Keep your database optimized with built-in maintenance tools:
### Database Location
By default, the database is saved as `example_prompts.db` in the node directory. This file contains all your prompts and linked images and can be backed up or shared.
By default, the database is saved as `prompts.db` in the extension directory. It contains your prompt library and image links. See the [database path compatibility note](#database-path-compatibility-211) before upgrading a deployment with a custom path.
### Searching Prompts
@@ -664,7 +698,8 @@ CREATE TABLE generated_images (
- **`utils/diagnostics.py`** - System diagnostics and health checks
- **`web/admin.html`** - Advanced admin dashboard with metadata panel
- **`web/index.html`** - Simple web interface
- **`web/js/prompt_manager.js`** - Dashboard JavaScript
- **`web/comfy/prompt_manager.js`** - ComfyUI canvas extension (the only file ComfyUI loads)
- **`web/js/admin.js`**, **`web/js/gallery.js`** - Dashboard and gallery JavaScript
- **`web/js/tags-page.js`** - Tag management JavaScript
- **`web/metadata.html`** - Standalone PNG metadata viewer
@@ -699,8 +734,11 @@ ComfyUI_PromptManager/
│ ├── gallery.html # Image gallery
│ ├── index.html # Simple web interface
│ ├── metadata.html # Standalone metadata viewer
│ ├── comfy/
│ │ └── prompt_manager.js # ComfyUI canvas extension (WEB_DIRECTORY)
│ └── js/
│ ├── prompt_manager.js # Dashboard JavaScript
│ ├── admin.js # Dashboard JavaScript
│ ├── gallery.js # Gallery JavaScript
│ └── tags-page.js # Tag management JavaScript
├── tests/
│ ├── __init__.py
@@ -735,6 +773,32 @@ Relative paths are resolved against the extension directory. Absolute paths are
}
```
### Database Path Compatibility (#211)
PromptManager now honors the `database.default_path` setting in `config.json`
when loaded as a ComfyUI custom node. Relative paths resolve against the extension
directory, not ComfyUI's working directory; absolute paths are used as-is. Earlier
versions could ignore this setting and use `<extension>/prompts.db` instead.
Honoring a different configured path can therefore show an empty or different
library. PromptManager does not automatically move or merge database files.
Before upgrading or changing this setting:
1. Identify the database your current installation actually uses and confirm its
library contents. Check both the configured path and the prior fallback
`<extension>/prompts.db`; keep track of which existing file you intend to use.
2. Preserve a consistent backup of the active database with the dashboard's
Backup/Restore backup action or SQLite's online backup API. Do not copy only
the main file of a live SQLite database: committed data may still be in its
`-wal` file. For a filesystem copy, stop ComfyUI cleanly first and preserve any
remaining `-wal` file together with the main database.
3. Set `database.default_path` to the existing intended database file. An absolute
path makes the selection explicit; do not move or overwrite either database
while choosing the path.
4. Restart ComfyUI, then verify that the expected library appears and the active
database path resolves to the file you selected. Keep the backup until both
checks succeed.
### Gallery & Monitoring Settings
Configure the automatic image detection and gallery system:
@@ -763,11 +827,15 @@ Configure the web interface behavior:
# Web UI settings (PromptManagerConfig class)
RESULT_TIMEOUT = 5 # Seconds to auto-hide results in ComfyUI node
WEBUI_DISPLAY_MODE = 'newtab' # 'popup' or 'newtab' for Web UI button
INFINITE_SCROLL = False # Dashboard list loads the next page on scroll by default
SHOW_TEST_BUTTON = False # Show API test button in node UI
```
All of these are editable in the dashboard Settings modal and saved to `config.json` (which is private to the installation and holds the CivitAI key, so it is never committed).
### Performance Tuning
- **Worker threads** (Settings → Performance): threads used by the image rescan, thumbnail generation and duplicate detection. Default is half the detected cores, maximum is the core count. Pillow releases the GIL while decoding, so the scan and thumbnails scale almost linearly; keep a few cores free while ComfyUI is generating.
- The database automatically creates indexes for optimal search performance
- Regular `VACUUM` operations keep the database optimized
- Consider backing up the database periodically
@@ -802,7 +870,7 @@ print(f"Average rating: {info['average_rating']}")
# Create backup
db.model.backup_database("backup_prompts.db")
# The database file can be copied directly for backup
# The online backup includes committed data still in the WAL.
```
## Development
@@ -811,9 +879,14 @@ db.model.backup_database("backup_prompts.db")
```bash
cd ComfyUI_PromptManager
python -m pytest tests/ -v
pip install -r requirements-dev.txt
python -m unittest discover -s tests # Python (what CI runs)
node --test tests/js/*.test.js # frontend pure modules
python -m coverage run -m unittest discover -s tests && python -m coverage combine && python -m coverage report
```
CI runs the suite on Linux, Windows and macOS for Python 3.10 to 3.12 with an 85% coverage gate. Tests that delete a database file first check that nothing in the process still holds it open (`tests/open_handles.py`), so a connection leak fails on Linux the way it would on Windows.
### Code Style
The project follows PEP 8 guidelines with:
@@ -874,6 +947,10 @@ MIT License - see LICENSE file for details.
### Recently Completed
- **✅ Security & Reliability Hardening (3.3.0)**: validated gallery roots, contained file serving, safe backups and restores, background scans, 85%+ test coverage on all three operating systems
- **✅ Worker Threads**: scan, thumbnails and duplicate detection run on a configurable thread pool
- **✅ Tag Autocomplete & Infinite Scrolling**: faster browsing in the dashboard and gallery
- **✅ Queue-Time Run Counting (3.2.4)**: run counts and image linking follow the ComfyUI queue
- **✅ LoRA Manager Integration**: Import LoRA metadata, trigger words, and preview images
- **✅ Folder Filter**: Browse and filter prompts by output subdirectory
- **✅ Multi-Directory Gallery**: Scan multiple output directories simultaneously
@@ -893,6 +970,31 @@ MIT License - see LICENSE file for details.
## Changelog
### v3.3.1 (Proxy Routing Fix)
- **🌐 Proxy support**: the dashboard, gallery, tag page and metadata viewer now work when ComfyUI is reached through a proxy mount such as SwarmUI's `/ComfyBackendDirect/`. Scripts, images, API calls, progress streams and gallery folder filters stay under the mount, and the canvas node's buttons use ComfyUI's `fileURL()`. Direct ComfyUI URLs are unchanged. Thanks to @ghosttigerllc-bit ([#164](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/164))
- **📦 Dependencies**: the runtime minimums stay at `Pillow>=8.0.0` and `watchdog>=2.1.0`, so updating Prompt Manager never forces upgrades in the shared ComfyUI environment
### v3.3.0 (Hardening, Background Jobs & Worker Threads)
- **🔒 Security**: gallery roots restricted to ComfyUI directories and validated wherever they are read; media-only, fail-closed image serving; Windows path escape in static routes closed; quote-safe HTML escaping with inline handlers moved to data attributes; the CivitAI key is never returned and only sent to HTTPS CivitAI hosts; auto-tag paths contained; side-effect routes are POST-only; backups include the WAL and restores are size-capped, integrity-checked and applied through SQLite's online backup API
- **🧵 Worker Threads**: a Settings slider (default half the cores) used by the image rescan, thumbnail generation and duplicate detection. Measured: a 1 GB sample scanned in 2.5 s instead of 18 s at 8 threads
- **🔁 Background Jobs**: the image rescan and the duplicate scan run on the server, stream progress, survive a closed tab, minimize to a progress pill and re-attach after a reload
- **🎬 Video Prompts**: the scan reads the prompt ComfyUI's video savers embed in MP4/MOV/WebM containers (needs `ffprobe`)
- **🖼️ Thumbnails**: written atomically and served under versioned URLs so a regenerated thumbnail never shows a stale cached copy; the per-run cap applies only to thumbnails that still need generating and the gallery says how many remain
- **🏷️ Tag Autocomplete**: the dashboard tag box suggests tags, narrowed to those that co-occur with the tags already typed
- **♾️ Infinite Scrolling**: optional in the gallery and the dashboard list (saved default plus a session toggle)
- **⚙️ Gallery Settings** now apply at load; grid columns, media info and the Rescan button work as labelled
- **🐛 Correctness**: run counts increment when ComfyUI dequeues a prompt; negative prompts are never traced; per-thread SQLite connections and idempotent migrations (an old rebuild could delete every image row); `truncate_logs` no longer rolls the log over; the ComfyUI canvas loads only the canvas extension; Windows console banners are ASCII-safe; subfolders use `/` on every platform
- **🧪 Tests & CI**: 1,460 Python and 142 JavaScript tests (from 510 and 26), coverage 96% with an 85% gate, CI on Linux, Windows and macOS for Python 3.10–3.12, flake8 clean
- **⚠️ API changes**: image paths in responses are relative to the ComfyUI tree (use `url`); `autotag/download`, `autotag/start` and the thumbnail progress stream are POST-only; `lora/status` returns `has_civitai_api_key` instead of the key; `/search` is paginated
### v3.2.4 (Recently Used Sorting & Queue-Time Run Counts)
- **🕒 Recently Used Sort**: sort the dashboard by last use, most used or rating, server-side across pages
- **🔢 Run Counts**: `run_count` and `last_used_at` are recorded through the ComfyUI queue hook, so cached nodes still count
- **🔗 Positive-Prompt Image Linking**: generated images link to the positive prompt traced through the queued graph
### v3.2.1 (LoRA Manager Integration)
- **🔗 LoRA Manager Integration**: Import LoRA metadata, trigger words, and CivitAI example images from [ComfyUI-Lora-Manager](https://github.com/willmiao/ComfyUI-Lora-Manager) into your prompt database
+1283
View File
File diff suppressed because it is too large Load Diff
+90 -10
View File
@@ -3,11 +3,14 @@ ComfyUI_PromptManager: A comprehensive ComfyUI custom node that extends text enc
with persistent prompt storage, advanced search capabilities, and an automatic image
gallery system.
This module provides two main node types:
This module provides prompt node types:
- PromptManager: CLIP encoding node that outputs CONDITIONING
- PromptManagerText: Text-only node that outputs STRING with prepend/append functionality
- PromptManagerFocused: Editor-first layout using the same encoding contract
- PromptManagerSplit: Side-by-side editor and library using the same contract
- PromptManagerText: Text-only node that outputs STRING with prepend/append
functionality
Both nodes share the same database backend for persistent prompt storage and include
The nodes share the same database backend for persistent prompt storage and include
an automatic image gallery system that monitors ComfyUI output directories and links
generated images to their source prompts.
@@ -37,24 +40,37 @@ def get_version():
return "unknown"
from .prompt_manager import PromptManager
from .prompt_manager_text import PromptManagerText
from .prompt_search_list import PromptSearchList
from .prompt_manager import ( # noqa: E402
PromptManager,
PromptManagerFocused,
PromptManagerSplit,
)
from .prompt_manager_pair import PromptManagerPair # noqa: E402
from .prompt_manager_text import PromptManagerText # noqa: E402
from .prompt_search_list import PromptSearchList # noqa: E402
NODE_CLASS_MAPPINGS = {
"PromptManager": PromptManager,
"PromptManagerFocused": PromptManagerFocused,
"PromptManagerSplit": PromptManagerSplit,
"PromptManagerPair": PromptManagerPair,
"PromptManagerText": PromptManagerText,
"PromptSearchList": PromptSearchList,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"PromptManager": "Prompt Manager",
"PromptManager": "Prompt Manager (Classic)",
"PromptManagerFocused": "Prompt Manager (Focused)",
"PromptManagerSplit": "Prompt Manager (Split)",
"PromptManagerPair": "Prompt Manager (+/-)",
"PromptManagerText": "Prompt Manager Text",
"PromptSearchList": "Prompt Search List",
}
# Define path to web directory for UI components
WEB_DIRECTORY = "web"
# Canvas extension only. ComfyUI loads every .js under this directory into the
# canvas page, so the admin/gallery bundles and vendored libraries under web/
# are served by our own routes instead (see py/api).
WEB_DIRECTORY = "web/comfy"
# Add API routes (same pattern as ComfyUI_Assets)
try:
@@ -71,6 +87,9 @@ try:
routes = config.routes
api = PromptManagerAPI()
api.add_routes(routes)
host_app = getattr(config.server_instance, "app", None)
if host_app is not None and hasattr(host_app, "on_cleanup"):
host_app.on_cleanup.append(api.cleanup_history_reads)
except Exception as e:
# Log to internal logging system without console spam
@@ -82,6 +101,10 @@ except Exception as e:
except Exception:
pass
# Shared database, set up with the monitoring system below. Stays None when that
# fails so the queue hook can report the database as unavailable instead of crashing.
_global_db = None
# Start image monitoring globally at module import time
# This ensures images are linked regardless of which node is used in the workflow
try:
@@ -111,12 +134,69 @@ except Exception as e:
except Exception:
print(f"[ComfyUI-PromptManager] Warning: Failed to start image monitoring: {e}")
def _db_factory():
"""The shared database for the queue hook; called lazily per request."""
if _global_db is None:
raise RuntimeError("PromptManager database unavailable")
return _global_db
# Count prompt runs when ComfyUI dequeues workflows (it skips unchanged nodes, so
# node execution alone misses re-runs). Separate from monitoring so either can fail.
try:
from .py.config import server_instance
from .utils.usage_tracking import register_queue_hook
register_queue_hook(server_instance, _db_factory)
except Exception as e:
try:
from .utils.logging_config import get_logger
get_logger("prompt_manager.init").error(f"Failed to register usage hook: {e}")
except Exception:
import logging
logging.getLogger("prompt_manager.init").error(
f"Failed to register usage hook: {e}"
)
def _history_db_factory() -> "PromptDatabase":
"""Initialize history on its worker even if monitoring initialization failed."""
if _global_db is not None:
return _global_db
from .database.operations import PromptDatabase
return PromptDatabase()
_history_registration = None
try:
from .py.config import server_instance
from .utils.execution_history import register_history_hook
_history_registration = register_history_hook(server_instance, _history_db_factory)
except Exception:
try:
from .utils.logging_config import get_logger
get_logger("prompt_manager.init").error("Failed to register execution history")
except Exception:
import logging
logging.getLogger("prompt_manager.init").error(
"Failed to register execution history"
)
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
# Print startup message with loaded tools
print()
print(f"\033[94m[ComfyUI-PromptManager] Version:\033[0m {get_version()}")
for node_key, display_name in NODE_DISPLAY_NAME_MAPPINGS.items():
print(f"🫶 \033[94mLoaded:\033[0m {display_name}")
# ASCII only: a Windows console using a legacy code page cannot print emoji,
# and a UnicodeEncodeError here would abort loading the whole node.
print(f" \033[94mLoaded:\033[0m {display_name}")
print(f"\033[94mTotal: {len(NODE_CLASS_MAPPINGS)} tools loaded\033[0m")
print()
+406
View File
@@ -0,0 +1,406 @@
"""Owned execution-context v1 tables within execution-history schema v2.
Content IDs hash exact scalar values; evidence JSON contains metadata only.
All validation scans use cursors and bounded per-row payloads.
"""
import json
import sqlite3
from .execution_history_schema import _compact, _uuid_check
try:
from ..utils.execution_contexts import (
BUNDLE_STATES,
CAPTURE_REASONS,
EVIDENCE_SOURCES,
UNKNOWN_REASONS,
MAX_CONTEXT_BYTES,
ContextBundle,
PromptSnapshot,
RecipeContext,
TextEvidence,
validate_bundle,
content_digest,
text_size,
)
except ImportError:
from utils.execution_contexts import (
BUNDLE_STATES,
CAPTURE_REASONS,
EVIDENCE_SOURCES,
UNKNOWN_REASONS,
MAX_CONTEXT_BYTES,
ContextBundle,
PromptSnapshot,
RecipeContext,
TextEvidence,
validate_bundle,
content_digest,
text_size,
)
FIELDS = ("raw", "effective", "prepend", "append")
TABLE_COLUMNS = {
"execution_context_texts": (
("text_id", "TEXT", 1, 1),
("text_value", "TEXT", 1, 0),
),
"execution_prompt_snapshots": (
("snapshot_id", "TEXT", 1, 1),
*((f"{field}_text_id", "TEXT", 0, 0) for field in FIELDS),
),
"execution_contexts": (
("context_id", "TEXT", 1, 1),
("positive_snapshot_id", "TEXT", 0, 0),
("negative_snapshot_id", "TEXT", 0, 0),
),
"execution_run_contexts": (
("run_id", "TEXT", 1, 1),
("anchor_id", "TEXT", 1, 2),
("context_id", "TEXT", 1, 0),
("evidence_json", "TEXT", 1, 0),
("positive_prompt_id", "INTEGER", 0, 0),
("negative_prompt_id", "INTEGER", 0, 0),
("pair_id", "INTEGER", 0, 0),
("reference_epoch", "TEXT", 0, 0),
("capture_version", "INTEGER", 1, 0),
),
"execution_context_capture": (
("run_id", "TEXT", 1, 1),
("state", "TEXT", 1, 0),
("reason", "TEXT", 1, 0),
("format_version", "INTEGER", 1, 0),
),
}
def _digest_check(column: str, nullable: bool = False) -> str:
check = (
f"typeof({column}) = 'text' AND length({column}) = 64 "
f"AND length(CAST({column} AS BLOB)) = 64 "
f"AND {column} NOT GLOB '*[^0-9a-f]*'"
)
return f"{column} IS NULL OR ({check})" if nullable else check
def _bounded_uuid_check(column: str) -> str:
return f"{_uuid_check(column)} AND length(CAST({column} AS BLOB)) = 36"
def _enum_check(column: str, values: frozenset[str]) -> str:
return (
f"typeof({column}) = 'text' AND {column} IN ("
+ ",".join("'" + value + "'" for value in sorted(values))
+ ")"
)
TABLE_CHECKS = {
"execution_context_texts": (
_digest_check("text_id"),
"typeof(text_value) = 'text' AND length(CAST(text_value AS BLOB)) <= 65536",
),
"execution_prompt_snapshots": (
_digest_check("snapshot_id"),
*(_digest_check(f"{field}_text_id", True) for field in FIELDS),
),
"execution_contexts": (
_digest_check("context_id"),
_digest_check("positive_snapshot_id", True),
_digest_check("negative_snapshot_id", True),
"positive_snapshot_id IS NOT NULL OR negative_snapshot_id IS NOT NULL",
),
"execution_run_contexts": (
_bounded_uuid_check("run_id"),
"typeof(anchor_id) = 'text' AND length(anchor_id) BETWEEN 1 AND 128 "
"AND length(CAST(anchor_id AS BLOB)) <= 512",
_digest_check("context_id"),
"typeof(evidence_json) = 'text' AND length(CAST(evidence_json AS "
"BLOB)) <= 8192",
*(
f"{column} IS NULL OR (typeof({column}) = 'integer' AND {column} > 0)"
for column in ("positive_prompt_id", "negative_prompt_id", "pair_id")
),
f"reference_epoch IS NULL OR ({_bounded_uuid_check('reference_epoch')})",
"typeof(capture_version) = 'integer' AND capture_version = 1",
),
"execution_context_capture": (
_bounded_uuid_check("run_id"),
_enum_check("state", BUNDLE_STATES),
_enum_check("reason", CAPTURE_REASONS),
"length(CAST(reason AS BLOB)) <= 64",
"typeof(format_version) = 'integer' AND format_version = 1",
),
}
# (source column, destination table, destination column); actions always NO ACTION.
TABLE_FOREIGN_KEYS = {
"execution_context_texts": (),
"execution_prompt_snapshots": tuple(
(f"{field}_text_id", "execution_context_texts", "text_id") for field in FIELDS
),
"execution_contexts": tuple(
(f"{role}_snapshot_id", "execution_prompt_snapshots", "snapshot_id")
for role in ("positive", "negative")
),
"execution_run_contexts": (
("run_id", "execution_runs", "run_id"),
("context_id", "execution_contexts", "context_id"),
),
"execution_context_capture": (("run_id", "execution_runs", "run_id"),),
}
# Only SQLite-generated primary-key indexes are unique. No secondary indexes required.
def install_context_schema(conn: sqlite3.Connection) -> None:
"""Install tables inside the caller's transaction, without committing."""
for table, columns in TABLE_COLUMNS.items():
definitions = [
f"{name} {kind}{' NOT NULL' if required else ''}"
for name, kind, required, _ in columns
]
primary = [name for name, _, _, order in columns if order]
definitions.append(f"PRIMARY KEY ({', '.join(primary)})")
definitions.extend(f"CHECK({check})" for check in TABLE_CHECKS[table])
definitions.extend(
f"FOREIGN KEY ({column}) REFERENCES {target}({key}) "
"ON UPDATE NO ACTION ON DELETE NO ACTION"
for column, target, key in TABLE_FOREIGN_KEYS[table]
)
conn.execute(f"CREATE TABLE {table} ({', '.join(definitions)})")
def _unique_object(pairs: list[tuple[str, object]]) -> dict:
result = {}
for key, value in pairs:
if key in result:
raise ValueError("Duplicate evidence key")
result[key] = value
return result
def _snapshot_values(conn: sqlite3.Connection, snapshot_id: str) -> tuple:
row = conn.execute(
"SELECT raw_text_id,effective_text_id,prepend_text_id,append_text_id "
"FROM execution_prompt_snapshots WHERE snapshot_id=?",
(snapshot_id,),
).fetchone()
if row is None:
raise ValueError("Missing snapshot")
values = []
for text_id in row:
if text_id is None:
values.append(None)
else:
text = conn.execute(
"SELECT text_value FROM execution_context_texts WHERE text_id=?",
(text_id,),
).fetchone()
if text is None:
raise ValueError("Missing context text")
text_size(text[0])
values.append(text[0])
return tuple(values)
def _validate_evidence(
conn: sqlite3.Connection, anchor_id: str, context_id: str, encoded: str
) -> tuple[RecipeContext, bool]:
text_size(encoded, 8192)
evidence = json.loads(encoded, object_pairs_hook=_unique_object)
if (
type(evidence) is not dict
or set(evidence)
!= {"format_version", "role_provenance", "positive", "negative"}
or type(evidence["format_version"]) is not int
or evidence["format_version"] != 1
or evidence["role_provenance"] != "inferred"
):
raise ValueError("Invalid context evidence")
row = conn.execute(
"SELECT positive_snapshot_id,negative_snapshot_id FROM "
"execution_contexts WHERE context_id=?",
(context_id,),
).fetchone()
if row is None:
raise ValueError("Missing context")
complete = True
snapshots = []
for role, snapshot_id in zip(("positive", "negative"), row):
metadata = evidence[role]
if snapshot_id is None:
if metadata is not None:
raise ValueError("Unexpected role evidence")
complete = False
snapshots.append(None)
continue
values = _snapshot_values(conn, snapshot_id)
if type(metadata) is not dict or set(metadata) != set(FIELDS):
raise ValueError("Invalid snapshot evidence")
fields = []
for field, value in zip(FIELDS, values):
item = metadata[field]
if type(item) is not dict or set(item) != {
"provenance",
"source",
"reason",
}:
raise ValueError("Invalid field evidence")
provenance, source, reason = (
item["provenance"],
item["source"],
item["reason"],
)
if (
type(provenance) is not str
or type(source) is not str
or source not in EVIDENCE_SOURCES.get(provenance, ())
):
raise ValueError("Invalid evidence source")
if value is None:
complete = False
if (
provenance != "unknown"
or type(reason) is not str
or reason not in UNKNOWN_REASONS
):
raise ValueError("Unknown evidence requires reason")
elif provenance == "unknown" or reason is not None:
raise ValueError("Known evidence metadata mismatch")
fields.append(TextEvidence(value, provenance, source, reason))
snapshots.append(PromptSnapshot(*fields))
return RecipeContext(anchor_id, *snapshots, evidence["role_provenance"]), complete
def context_schema_error(conn: sqlite3.Connection) -> str | None:
"""Validate exact owned schema, bounded evidence, fingerprints and links."""
if conn.execute(
"SELECT 1 FROM sqlite_master WHERE type IN ('trigger','view') LIMIT 1"
).fetchone():
return "Unsupported execution context trigger/view"
tables = {
row[0]: row[1]
for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
if row[0] in TABLE_COLUMNS
}
if tables.keys() != TABLE_COLUMNS.keys():
return "Incomplete execution context schema"
for table, expected in TABLE_COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[5])
for row in conn.execute(f"PRAGMA table_xinfo({table})")
)
if columns != expected:
return f"{table} has invalid execution context columns"
sql = _compact(tables[table])
checks = TABLE_CHECKS[table]
if sql.count("check(") != len(checks) or any(
f"check({_compact(check)})" not in sql for check in checks
):
return f"{table} has invalid execution context checks"
foreign = tuple(
sorted(
(row[3], row[2], row[4], row[5], row[6], row[7])
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
)
)
expected_foreign = tuple(
sorted(
(*key, "NO ACTION", "NO ACTION", "NONE")
for key in TABLE_FOREIGN_KEYS[table]
)
)
if foreign != expected_foreign:
return f"{table} has invalid execution context foreign keys"
for index in conn.execute(f"PRAGMA index_list({table})"):
if not index[2]:
continue
name = index[1].replace('"', '""')
keys = tuple(
row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
)
primary = tuple(column[0] for column in expected if column[3])
if (
index[3] != "pk"
or index[4]
or tuple(key[2] for key in keys) != primary
or any(key[3] or key[4] != "BINARY" for key in keys)
):
return f"{table} has unexpected execution context uniqueness"
predicate = " AND ".join(f"({check})" for check in checks)
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
).fetchone():
return f"{table} contains invalid execution context records"
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
return f"{table} contains orphan execution context records"
if conn.execute(
"SELECT 1 FROM execution_run_contexts GROUP BY run_id HAVING "
"COUNT(*) > 16 LIMIT 1"
).fetchone():
return "Too many execution contexts per run"
try:
for text_id, text in conn.execute(
"SELECT text_id,text_value FROM execution_context_texts"
):
text_size(text)
if content_digest("text", (text,)) != text_id:
raise ValueError("Invalid text fingerprint")
for (snapshot_id,) in conn.execute(
"SELECT snapshot_id FROM execution_prompt_snapshots"
):
if (
content_digest("snapshot", _snapshot_values(conn, snapshot_id))
!= snapshot_id
):
raise ValueError("Invalid snapshot fingerprint")
for context_id, positive, negative in conn.execute(
"SELECT context_id,positive_snapshot_id,negative_snapshot_id FROM "
"execution_contexts"
):
if content_digest("context", (positive, negative)) != context_id:
raise ValueError("Invalid context fingerprint")
payload_size = 0
for snapshot_id in (positive, negative):
if snapshot_id is not None:
payload_size += sum(
text_size(value)
for value in _snapshot_values(conn, snapshot_id)
if value is not None
)
if payload_size > MAX_CONTEXT_BYTES:
raise ValueError("Oversized detached context")
previous_run = None
contexts = ()
for run_id, anchor_id, context_id, evidence in conn.execute(
"SELECT run_id,anchor_id,context_id,evidence_json FROM "
"execution_run_contexts ORDER BY run_id,anchor_id"
):
if run_id != previous_run:
contexts = ()
previous_run = run_id
context, complete = _validate_evidence(
conn, anchor_id, context_id, evidence
)
status = conn.execute(
"SELECT state,reason FROM execution_context_capture WHERE run_id=?",
(run_id,),
).fetchone()
if status is None:
raise ValueError("Context link has no capture header")
state, reason = status
if state == "complete" and not complete:
raise ValueError("Incomplete evidence marked complete")
candidate = (*contexts, context)
validate_bundle(ContextBundle(candidate, state, reason))
contexts = candidate
if conn.execute(
"SELECT 1 FROM execution_context_capture c WHERE state='complete' "
"AND NOT EXISTS (SELECT 1 FROM execution_run_contexts r WHERE "
"r.run_id=c.run_id) LIMIT 1"
).fetchone():
raise ValueError("Complete capture has no contexts")
except (ValueError, TypeError, RecursionError, OverflowError) as exc:
return f"Invalid execution context content: {type(exc).__name__}"
return None
+411
View File
@@ -0,0 +1,411 @@
"""Immutable execution capture and exact, epoch-fenced library navigation hints."""
from contextlib import ExitStack
import json
import sqlite3
from typing import Callable, ContextManager, Iterator
try:
from ..utils.execution_history_events import RunFinish, encode_command
from ..utils.execution_contexts import (
CAPTURE_REASONS,
ContextBundle,
content_digest,
validate_bundle,
)
except ImportError:
from utils.execution_history_events import RunFinish, encode_command
from utils.execution_contexts import (
CAPTURE_REASONS,
ContextBundle,
content_digest,
validate_bundle,
)
FIELDS = ("raw", "effective", "prepend", "append")
ROLES = ("positive", "negative")
def _prepare(bundle: ContextBundle) -> list[dict]:
validate_bundle(bundle)
if bundle.reason not in CAPTURE_REASONS:
raise ValueError("Invalid capture reason")
prepared = []
complete = bool(bundle.contexts)
for context in bundle.contexts:
evidence = {"format_version": 1, "role_provenance": context.role_provenance}
snapshots = []
for role in ROLES:
snapshot = getattr(context, role)
if snapshot is None:
snapshots.append(None)
evidence[role] = None
complete = False
continue
values = tuple(getattr(snapshot, field).value for field in FIELDS)
complete = complete and all(value is not None for value in values)
snapshots.append((content_digest("snapshot", values), values))
evidence[role] = {
field: {
"provenance": getattr(snapshot, field).provenance,
"source": getattr(snapshot, field).source,
"reason": getattr(snapshot, field).reason,
}
for field in FIELDS
}
if all(snapshot is None for snapshot in snapshots):
raise ValueError("Context must contain a snapshot")
context_id = content_digest(
"context", tuple(s[0] if s else None for s in snapshots)
)
encoded = json.dumps(
evidence,
ensure_ascii=False,
allow_nan=False,
separators=(",", ":"),
sort_keys=True,
)
if len(encoded.encode("utf-8")) > 8192:
raise ValueError("Oversized context evidence")
prepared.append(
{
"anchor_id": context.anchor_id,
"context_id": context_id,
"snapshots": snapshots,
"evidence_json": encoded,
}
)
if bundle.state == "complete" and not complete:
raise ValueError("Incomplete scalar capture")
return sorted(prepared, key=lambda item: item["anchor_id"])
def _existing_row(
conn: sqlite3.Connection, table: str, columns: tuple[str, ...], values: tuple
) -> bool:
row = conn.execute(
f"SELECT {','.join(columns[1:])} FROM {table} WHERE {columns[0]}=?",
(values[0],),
).fetchone()
if row is None:
return False
if tuple(row) != values[1:]:
raise ValueError("Execution context digest collision")
return True
def _content_rows(item: dict) -> Iterator[tuple[str, tuple[str, ...], tuple]]:
snapshot_ids = []
for snapshot in item["snapshots"]:
if snapshot is None:
snapshot_ids.append(None)
continue
snapshot_id, values = snapshot
text_ids = []
for value in values:
text_id = None if value is None else content_digest("text", (value,))
if value is not None:
yield "execution_context_texts", ("text_id", "text_value"), (
text_id,
value,
)
text_ids.append(text_id)
yield "execution_prompt_snapshots", (
"snapshot_id",
*(f"{field}_text_id" for field in FIELDS),
), (snapshot_id, *text_ids)
snapshot_ids.append(snapshot_id)
yield "execution_contexts", (
"context_id",
"positive_snapshot_id",
"negative_snapshot_id",
), (item["context_id"], *snapshot_ids)
def _insert_exact(
conn: sqlite3.Connection, table: str, columns: tuple[str, ...], values: tuple
) -> None:
if not _existing_row(conn, table, columns, values):
names = ",".join(columns)
placeholders = ",".join("?" for _ in columns)
conn.execute(f"INSERT INTO {table} ({names}) VALUES ({placeholders})", values)
def _role_hints(role: str) -> str:
return (
f"SELECT r.{role}_prompt_id, tx.text_value "
"FROM execution_run_contexts r "
"JOIN execution_contexts c ON c.context_id = r.context_id "
f"JOIN execution_prompt_snapshots s ON s.snapshot_id = c.{role}_snapshot_id "
"JOIN execution_context_texts tx ON tx.text_id = s.raw_text_id "
f"WHERE r.{role}_prompt_id IS NOT NULL AND r.reference_epoch = ("
"SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id = 1)"
)
# Set form of the read-time prompt-hint rule in get_execution_contexts():
# a stored role hint is live only in the current storage epoch, and only when
# its exact raw snapshot text matches exactly one prompt, which is the hinted
# one (_prompt_hint). Evaluated once per statement, never per prompt row.
# Defines the CTE ``live_hint_prompts(prompt_id)``; prefix it with WITH.
LIVE_HINT_CTES = (
"stored_hints(prompt_id, raw) AS ("
+ _role_hints("positive")
+ " UNION "
+ _role_hints("negative")
+ "), unique_hint_texts(prompt_id, raw) AS ("
"SELECT MIN(id), text COLLATE BINARY FROM prompts "
"WHERE text COLLATE BINARY IN (SELECT raw FROM stored_hints) "
"GROUP BY text COLLATE BINARY HAVING COUNT(*) = 1"
"), live_hint_prompts(prompt_id) AS ("
"SELECT h.prompt_id FROM stored_hints h JOIN unique_hint_texts u "
"ON u.prompt_id = h.prompt_id AND u.raw = h.raw COLLATE BINARY)"
)
LIVE_HINT_PROMPT_IDS_SQL = (
"WITH " + LIVE_HINT_CTES + " SELECT DISTINCT prompt_id FROM live_hint_prompts"
)
def _prompt_hint(conn: sqlite3.Connection, raw: str | None) -> int | None:
if raw is None:
return None
rows = conn.execute(
"SELECT id FROM prompts WHERE text=? COLLATE BINARY LIMIT 2", (raw,)
).fetchall()
return rows[0][0] if len(rows) == 1 else None
def _hints(
conn: sqlite3.Connection, positive: str | None, negative: str | None
) -> tuple:
pos_id, neg_id = _prompt_hint(conn, positive), _prompt_hint(conn, negative)
pair_id = None
negative_ambiguous = (
negative is not None
and neg_id is None
and conn.execute(
"SELECT 1 FROM prompts WHERE text=? COLLATE BINARY LIMIT 1", (negative,)
).fetchone()
is not None
)
if pos_id is not None and negative is not None and not negative_ambiguous:
rows = conn.execute(
"SELECT pp.id FROM prompt_pairs pp LEFT JOIN prompts n ON "
"n.id=pp.negative_prompt_id "
"WHERE pp.positive_prompt_id=? AND ((pp.negative_prompt_id IS NULL AND "
"?='') "
"OR (pp.negative_prompt_id=? AND n.text=? COLLATE BINARY)) LIMIT 2",
(pos_id, negative, neg_id, negative),
).fetchall()
if len(rows) == 1:
pair_id = rows[0][0]
return pos_id, neg_id, pair_id
def _values(conn: sqlite3.Connection, snapshot_id: str | None) -> dict | None:
if snapshot_id is None:
return None
row = conn.execute(
"SELECT r.text_value,e.text_value,p.text_value,a.text_value "
"FROM execution_prompt_snapshots s "
"LEFT JOIN execution_context_texts r ON r.text_id=s.raw_text_id "
"LEFT JOIN execution_context_texts e ON e.text_id=s.effective_text_id "
"LEFT JOIN execution_context_texts p ON p.text_id=s.prepend_text_id "
"LEFT JOIN execution_context_texts a ON a.text_id=s.append_text_id "
"WHERE s.snapshot_id=?",
(snapshot_id,),
).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing execution context snapshot")
return dict(zip(FIELDS, row))
def _write_capture(
conn: sqlite3.Connection,
run_id: str,
expected_epoch: str,
bundle: ContextBundle,
prepared: list[dict],
) -> bool:
capture = conn.execute(
"SELECT state,reason,format_version FROM execution_context_capture "
"WHERE run_id=?",
(run_id,),
).fetchone()
if capture is not None:
for item in prepared:
for table, columns, values in _content_rows(item):
_existing_row(conn, table, columns, values)
links = conn.execute(
"SELECT anchor_id,context_id,evidence_json FROM "
"execution_run_contexts WHERE run_id=? ORDER BY anchor_id",
(run_id,),
).fetchall()
expected = [
(item["anchor_id"], item["context_id"], item["evidence_json"])
for item in prepared
]
# Compare JSON objects so serialization order never creates a conflict.
original = [(row[0], row[1], json.loads(row[2])) for row in links]
proposed = [
(anchor, context, json.loads(evidence))
for anchor, context, evidence in expected
]
if capture[0] != "conflicting" and (
tuple(capture) != (bundle.state, bundle.reason, bundle.format_version)
or original != proposed
):
conn.execute(
"UPDATE execution_context_capture SET "
"state='conflicting',reason='conflicting' WHERE run_id=?",
(run_id,),
)
return True
for item in prepared:
for table, columns, values in _content_rows(item):
_insert_exact(conn, table, columns, values)
raw = [snapshot[1][0] if snapshot else None for snapshot in item["snapshots"]]
positive_id, negative_id, pair_id = _hints(conn, *raw)
conn.execute(
"INSERT INTO "
"execution_run_contexts(run_id,anchor_id,context_id,evidence_json,"
"positive_prompt_id,negative_prompt_id,pair_id,reference_epoch,"
"capture_version) VALUES(?,?,?,?,?,?,?,?,1)",
(
run_id,
item["anchor_id"],
item["context_id"],
item["evidence_json"],
positive_id,
negative_id,
pair_id,
expected_epoch,
),
)
conn.execute(
"INSERT INTO "
"execution_context_capture(run_id,state,reason,format_version) "
"VALUES(?,?,?,?)",
(run_id, bundle.state, bundle.reason, bundle.format_version),
)
return True
class ExecutionContextOperations:
"""Compose with ExecutionHistoryOperations on PromptDatabase."""
def write_execution_contexts(
self, run_id: str, expected_epoch: str, bundle: ContextBundle
) -> bool:
prepared = _prepare(bundle)
with self._execution_transaction(expected_epoch) as conn:
if (
conn.execute(
"SELECT 1 FROM execution_runs WHERE run_id=? AND storage_epoch=?",
(run_id, expected_epoch),
).fetchone()
is None
):
return False
return _write_capture(conn, run_id, expected_epoch, bundle, prepared)
def write_execution_contexts_for_finish(
self,
command: RunFinish,
bundle: ContextBundle,
*,
generation_guard: Callable[[], ContextManager[bool]] | None = None,
) -> bool:
encode_command(command)
if type(command) is not RunFinish:
raise ValueError("Expected execution finish command")
prepared = _prepare(bundle)
# Enter the guard after BEGIN, and keep it until COMMIT/ROLLBACK exits.
with ExitStack() as guards:
with self._execution_transaction(command.storage_epoch) as conn:
if generation_guard is not None and not guards.enter_context(
generation_guard()
):
return False
row = conn.execute(
"SELECT run_id FROM execution_runs WHERE session_id=? "
"AND queue_item_id=? AND storage_epoch=?",
(command.session_id, command.queue_item_id, command.storage_epoch),
).fetchone()
if row is None:
return False
return _write_capture(
conn, row[0], command.storage_epoch, bundle, prepared
)
def get_execution_contexts(self, run_id: str) -> dict:
"""Return first-capture scalars; hints are None unless currently valid.
reference_epoch remains historical evidence even when hints are invalidated.
Reads never rebind a first-capture reference or alter stored capture data.
"""
conn = self.model.get_connection()
own_transaction = not conn.in_transaction
if own_transaction:
conn.execute("BEGIN")
try:
capture = conn.execute(
"SELECT state,reason,format_version FROM execution_context_capture "
"WHERE run_id=?",
(run_id,),
).fetchone()
if capture is None:
return {
"state": "not_captured",
"reason": "missing",
"format_version": None,
"contexts": [],
}
epoch = self.get_execution_epoch()
contexts = []
rows = conn.execute(
"SELECT r.*,c.positive_snapshot_id,c.negative_snapshot_id FROM "
"execution_run_contexts r JOIN execution_contexts c ON "
"c.context_id=r.context_id WHERE r.run_id=? ORDER BY r.anchor_id",
(run_id,),
)
for row in rows:
positive = _values(conn, row["positive_snapshot_id"])
negative = _values(conn, row["negative_snapshot_id"])
hints = (None, None, None)
if row["reference_epoch"] == epoch:
current = _hints(
conn,
positive["raw"] if positive else None,
negative["raw"] if negative else None,
)
hints = tuple(
row[key] if row[key] == value else None
for key, value in zip(
("positive_prompt_id", "negative_prompt_id", "pair_id"),
current,
)
)
contexts.append(
{
"anchor_id": row["anchor_id"],
"context_id": row["context_id"],
"positive": positive,
"negative": negative,
"evidence": json.loads(row["evidence_json"]),
"positive_prompt_id": hints[0],
"negative_prompt_id": hints[1],
"pair_id": hints[2],
"reference_epoch": row["reference_epoch"],
}
)
return {
"state": capture[0],
"reason": capture[1],
"format_version": capture[2],
"contexts": contexts,
}
finally:
if own_transaction:
conn.rollback()
+198
View File
@@ -0,0 +1,198 @@
"""Transactional attempt writes and process-local restore invalidation."""
import os
import sqlite3
import threading
from contextlib import contextmanager
from typing import Callable, Iterator
try:
from ..utils.execution_history_events import (
RunFinish,
RunStart,
classify_evidence,
encode_command,
)
from ..utils.logging_config import get_logger
except ImportError:
from utils.execution_history_events import (
RunFinish,
RunStart,
classify_evidence,
encode_command,
)
from utils.logging_config import get_logger
_logger = get_logger("prompt_manager.database.execution_history")
# Bounded wait for a history write lock. History is best effort and must
# never stall ComfyUI's queue: a write that cannot start in time raises
# "database is locked" and the writer records a gap instead of waiting.
EXECUTION_BUSY_TIMEOUT_MS = 250
_restore_lock = threading.Lock()
_restore_observers: dict[str, dict[object, Callable[[], None]]] = {}
def _database_identity(db_path: str) -> str:
return os.path.normcase(os.path.realpath(db_path))
def subscribe_history_restore(
db_path: str, callback: Callable[[], None]
) -> Callable[[], None]:
"""Observe successful restores of the canonical database path."""
identity = _database_identity(db_path)
token = object()
with _restore_lock:
_restore_observers.setdefault(identity, {})[token] = callback
def unsubscribe() -> None:
with _restore_lock:
observers = _restore_observers.get(identity)
if observers is not None:
observers.pop(token, None)
if not observers:
_restore_observers.pop(identity, None)
return unsubscribe
def notify_history_restore(db_path: str) -> None:
"""Notify outside the registry lock; observers cannot fail a restore."""
with _restore_lock:
callbacks = tuple(
_restore_observers.get(_database_identity(db_path), {}).values()
)
for callback in callbacks:
try:
callback()
except Exception:
_logger.warning("Execution history restore observer failed", exc_info=True)
class HistoryEpochChanged(Exception):
"""A command belongs to storage replaced by a completed restore."""
def __init__(self, storage_epoch: str):
self.storage_epoch = storage_epoch
super().__init__("Execution history storage epoch changed")
class ExecutionHistoryOperations:
"""Mixin using the calling thread's existing managed model connection."""
def get_execution_epoch(self) -> str:
row = (
self.model.get_connection()
.execute(
"SELECT storage_epoch FROM execution_history_metadata "
"WHERE singleton_id=1"
)
.fetchone()
)
if row is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
return row[0]
@contextmanager
def _execution_transaction(
self, expected_epoch: str
) -> Iterator[sqlite3.Connection]:
conn = self.model.get_connection()
if conn.in_transaction:
raise sqlite3.ProgrammingError(
"Execution history requires its own transaction"
)
timeout = conn.execute("PRAGMA busy_timeout").fetchone()[0]
conn.execute(f"PRAGMA busy_timeout={int(EXECUTION_BUSY_TIMEOUT_MS)}")
try:
conn.execute("BEGIN IMMEDIATE")
try:
epoch = self.get_execution_epoch()
if epoch != expected_epoch:
raise HistoryEpochChanged(epoch)
yield conn
conn.commit()
except BaseException:
conn.rollback()
raise
finally:
conn.execute(f"PRAGMA busy_timeout={timeout}")
def write_execution_start(self, command: RunStart) -> bool:
encode_command(command)
if type(command) is not RunStart:
raise ValueError("Expected execution start command")
with self._execution_transaction(command.storage_epoch) as conn:
conn.execute(
"INSERT INTO execution_runs "
"(run_id,session_id,queue_item_id,job_id,storage_epoch,record_version,"
"observed_dequeue_at,outcome,diagnostics_omitted) "
"VALUES(?,?,?,?,?,1,?,'observed',0) "
"ON CONFLICT(session_id,queue_item_id) DO NOTHING",
(
command.run_id,
command.session_id,
command.queue_item_id,
command.job_id,
command.storage_epoch,
command.observed_dequeue_at,
),
)
return True
def write_execution_finish(self, command: RunFinish) -> bool:
encode_command(command)
if type(command) is not RunFinish:
raise ValueError("Expected execution finish command")
with self._execution_transaction(command.storage_epoch) as conn:
row = conn.execute(
"SELECT * FROM execution_runs WHERE session_id=? AND queue_item_id=? "
"AND storage_epoch=?",
(command.session_id, command.queue_item_id, command.storage_epoch),
).fetchone()
if row is None:
return False
completion = classify_evidence(row["job_id"], command.evidence)
if row["outcome"] == "unknown":
return True
if row["outcome"] != "observed":
contradictory = completion.reason == "conflicting_terminal" or (
completion.outcome in ("succeeded", "failed", "interrupted")
and completion.outcome != row["outcome"]
)
if contradictory:
conn.execute(
"UPDATE execution_runs SET outcome='unknown',"
"reason='conflicting_terminal' WHERE run_id=?",
(row["run_id"],),
)
return True
conn.execute(
"UPDATE execution_runs SET observed_callback_at=?,"
"executor_started_at=?,"
"terminal_event_at=?,outcome=?,reason=?,diagnostics_omitted=?,"
"node_id=?,"
"node_type=?,exception_type=? WHERE run_id=?",
(
command.observed_callback_at,
completion.executor_started_at,
completion.terminal_event_at,
completion.outcome,
completion.reason,
int(completion.diagnostics_omitted),
completion.node_id,
completion.node_type,
completion.exception_type,
row["run_id"],
),
)
return True
def get_execution_run(self, run_id: str) -> dict | None:
row = (
self.model.get_connection()
.execute("SELECT * FROM execution_runs WHERE run_id=?", (run_id,))
.fetchone()
)
return dict(row) if row is not None else None
+349
View File
@@ -0,0 +1,349 @@
"""Versioned additive schema for observed execution attempts."""
import re
import sqlite3
from uuid import uuid4
SCHEMA_VERSION = 3
def _uuid_check(column: str) -> str:
return (
f"typeof({column}) = 'text' AND length({column}) = 36 "
f"AND substr({column}, 9, 1) = '-' AND substr({column}, 14, 1) = '-' "
f"AND substr({column}, 19, 1) = '-' AND substr({column}, 24, 1) = '-' "
f"AND length(replace({column}, '-', '')) = 32 "
f"AND replace({column}, '-', '') NOT GLOB '*[^0-9a-f]*'"
)
def _integer_check(column: str, *, nullable: bool = False) -> str:
expression = f"typeof({column}) = 'integer' AND {column} >= 0"
return f"{column} IS NULL OR ({expression})" if nullable else expression
_METADATA_CHECKS = (
"singleton_id = 1",
"schema_version = 3",
_uuid_check("storage_epoch"),
)
_RUN_CHECKS = (
*(
_uuid_check(column)
for column in ("run_id", "session_id", "job_id", "storage_epoch")
),
_integer_check("queue_item_id"),
"record_version = 1",
_integer_check("observed_dequeue_at"),
*(
_integer_check(column, nullable=True)
for column in (
"observed_callback_at",
"executor_started_at",
"terminal_event_at",
)
),
"outcome IN ('observed', 'succeeded', 'failed', 'interrupted', 'unknown')",
"(outcome = 'observed' AND reason IS NULL AND "
"observed_callback_at IS NULL) OR (outcome != 'observed' AND "
"reason IS NOT NULL AND observed_callback_at IS NOT NULL AND "
"((outcome = 'succeeded' AND reason = 'terminal_success') OR "
"(outcome = 'failed' AND reason = 'terminal_error') OR (outcome = "
"'interrupted' AND reason = 'terminal_interrupted') OR (outcome = "
"'unknown' AND reason IN ('missing_terminal', "
"'conflicting_terminal', 'invalid_evidence', 'evidence_limit'))))",
"diagnostics_omitted IN (0, 1)",
*(
f"{column} IS NULL OR (typeof({column}) = 'text' "
f"AND length({column}) <= {limit})"
for column, limit in (
("node_id", 128),
("node_type", 128),
("exception_type", 256),
)
),
)
_TABLE_COLUMNS = {
"execution_history_metadata": (
("singleton_id", "INTEGER", 1, 1),
("schema_version", "INTEGER", 1, 0),
("storage_epoch", "TEXT", 1, 0),
),
"execution_runs": (
("run_id", "TEXT", 1, 1),
("session_id", "TEXT", 1, 0),
("queue_item_id", "INTEGER", 1, 0),
("job_id", "TEXT", 1, 0),
("storage_epoch", "TEXT", 1, 0),
("record_version", "INTEGER", 1, 0),
("observed_dequeue_at", "INTEGER", 1, 0),
("observed_callback_at", "INTEGER", 0, 0),
("executor_started_at", "INTEGER", 0, 0),
("terminal_event_at", "INTEGER", 0, 0),
("outcome", "TEXT", 1, 0),
("reason", "TEXT", 0, 0),
("diagnostics_omitted", "INTEGER", 1, 0),
("node_id", "TEXT", 0, 0),
("node_type", "TEXT", 0, 0),
("exception_type", "TEXT", 0, 0),
),
}
_TABLE_CHECKS = {
"execution_history_metadata": _METADATA_CHECKS,
"execution_runs": _RUN_CHECKS,
}
# Row-only guards preserve the supported v1 DDL fingerprint while bounding reads.
_ROW_BYTE_CHECKS = {
"execution_history_metadata": ("length(CAST(storage_epoch AS BLOB)) = 36",),
"execution_runs": (
*(
f"length(CAST({column} AS BLOB)) = 36"
for column in (
"run_id",
"session_id",
"job_id",
"storage_epoch",
)
),
*(
f"{column} IS NULL OR length(CAST({column} AS BLOB)) <= {limit}"
for column, limit in (
("node_id", 512),
("node_type", 512),
("exception_type", 1024),
)
),
),
}
_INDEXES = {
"idx_execution_runs_identity": (True, ("session_id", "queue_item_id")),
"idx_execution_runs_observed": (False, ("observed_dequeue_at", "run_id")),
}
def _compact(sql: str) -> str:
return "".join(
token if token.startswith("'") else re.sub(r"\s+", "", token).lower()
for token in re.findall(r"'(?:''|[^'])*'|[^']+", sql)
)
def history_schema_error(
conn: sqlite3.Connection, *, allow_missing: bool = True
) -> str | None:
"""Reject partial/unsupported history schemas and invalid persisted records."""
tables = {
row[0]: row[1]
for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
if row[0] in _TABLE_COLUMNS
}
from .execution_context_schema import TABLE_COLUMNS, context_schema_error
from .execution_output_schema import (
TABLE_COLUMNS as OUTPUT_COLUMNS,
output_schema_error,
)
output_present = any(
row[0] in OUTPUT_COLUMNS
for row in conn.execute("SELECT name FROM sqlite_master")
)
if not tables and output_present:
return "Execution outputs require execution history"
context_present = any(
row[0] in TABLE_COLUMNS
for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'")
)
if not tables and context_present:
return "Execution contexts require execution history"
if not tables:
return None if allow_missing else "Missing execution history schema"
if tables.keys() != _TABLE_COLUMNS.keys():
return "Incomplete execution history schema"
metadata = conn.execute(
"SELECT CASE WHEN typeof(schema_version) = 'integer' "
"THEN schema_version ELSE NULL END FROM execution_history_metadata LIMIT 2"
).fetchall()
if len(metadata) != 1 or len(metadata[0]) != 1 or metadata[0][0] not in (1, 2, 3):
return "Unsupported execution history metadata/version"
version = metadata[0][0]
if version < 3 and output_present:
return "Unexpected execution outputs for history v1/v2"
if version == 1 and context_present:
return "Unexpected execution contexts for history v1"
if not allow_missing and version != SCHEMA_VERSION:
return "Execution history migration required"
for table, expected in _TABLE_COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[5])
for row in conn.execute(f"PRAGMA table_info({table})")
)
if columns != expected:
return f"{table} has invalid execution history columns"
sql = _compact(tables[table])
checks = _TABLE_CHECKS[table]
if table == "execution_history_metadata":
checks = tuple(
check.replace("schema_version = 3", f"schema_version = {version}")
for check in checks
)
if sql.count("check(") != len(checks) or any(
f"check({_compact(check)})" not in sql for check in checks
):
return f"{table} has invalid execution history checks"
if conn.execute(f"PRAGMA foreign_key_list({table})").fetchone():
return f"{table} has unexpected execution history foreign keys"
for index in conn.execute(f"PRAGMA index_list({table})"):
if not index[2]:
continue
if table == "execution_runs" and index[1] == "idx_execution_runs_identity":
continue
name = index[1].replace('"', '""')
keys = tuple(
row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
)
primary = tuple(column[0] for column in expected if column[3])
if (
index[3] != "pk"
or index[4]
or tuple(key[2] for key in keys) != primary
or any(key[3] or key[4] != "BINARY" for key in keys)
):
return f"{table} has unexpected execution history uniqueness"
predicate = " AND ".join(
f"({check})" for check in (*checks, *_ROW_BYTE_CHECKS[table])
)
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
).fetchone():
return f"{table} contains invalid execution history records"
metadata = conn.execute(
"SELECT singleton_id,schema_version,storage_epoch FROM "
"execution_history_metadata"
).fetchall()
if len(metadata) != 1 or metadata[0][0] != 1 or metadata[0][1] != version:
return "Unsupported execution history metadata/version"
indexes = {row[1]: row for row in conn.execute("PRAGMA index_list(execution_runs)")}
for name, (unique, columns) in _INDEXES.items():
row = indexes.get(name)
if row is None or bool(row[2]) != unique or row[4]:
return f"Missing or invalid execution history index {name}"
actual = tuple(conn.execute(f"PRAGMA index_xinfo({name})"))
keys = tuple(item for item in actual if item[5])
if tuple(item[2] for item in keys) != columns or any(
item[3] or item[4] != "BINARY" for item in keys
):
return f"Invalid execution history index {name}"
if version >= 2:
error = context_schema_error(conn)
if error:
return error
return output_schema_error(conn) if version == 3 else None
def install_context_schema(conn: sqlite3.Connection) -> None:
from .execution_context_schema import install_context_schema as install
install(conn)
def install_output_schema(conn: sqlite3.Connection) -> None:
from .execution_output_schema import install_output_schema as install
install(conn)
def migrate_execution_history(conn: sqlite3.Connection) -> None:
"""Install history, contexts and outputs atomically, without backfill."""
conn.execute("SAVEPOINT execution_history_schema")
try:
error = history_schema_error(conn)
if error:
raise sqlite3.DatabaseError(error)
present = conn.execute(
"SELECT 1 FROM sqlite_master WHERE name IN "
"('execution_runs','execution_history_metadata') LIMIT 1"
).fetchone()
if not present:
for table, columns in _TABLE_COLUMNS.items():
definitions = [
f"{name} {kind}{' NOT NULL' if required else ''}"
f"{' PRIMARY KEY' if primary else ''}"
for name, kind, required, primary in columns
]
definitions.extend(f"CHECK({check})" for check in _TABLE_CHECKS[table])
conn.execute(f"CREATE TABLE {table} ({', '.join(definitions)})")
for name, (unique, columns) in _INDEXES.items():
conn.execute(
f"CREATE {'UNIQUE ' if unique else ''}INDEX {name} "
f"ON execution_runs({', '.join(columns)})"
)
conn.execute(
"INSERT INTO execution_history_metadata VALUES(1,?,?)",
(SCHEMA_VERSION, str(uuid4())),
)
install_context_schema(conn)
install_output_schema(conn)
else:
version, epoch = conn.execute(
"SELECT schema_version,storage_epoch FROM execution_history_metadata"
).fetchone()
if version < SCHEMA_VERSION:
conn.execute(
"ALTER TABLE execution_history_metadata RENAME TO "
"execution_history_metadata_old"
)
definitions = [
f"{name} {kind}{' NOT NULL' if required else ''}"
f"{' PRIMARY KEY' if primary else ''}"
for name, kind, required, primary in _TABLE_COLUMNS[
"execution_history_metadata"
]
]
definitions.extend(f"CHECK({check})" for check in _METADATA_CHECKS)
conn.execute(
"CREATE TABLE execution_history_metadata "
f"({', '.join(definitions)})"
)
conn.execute(
"INSERT INTO execution_history_metadata VALUES(1,?,?)",
(SCHEMA_VERSION, epoch),
)
conn.execute("DROP TABLE execution_history_metadata_old")
if version == 1:
install_context_schema(conn)
install_output_schema(conn)
# Read projections need reverse content lookup and the complete run order.
for table, columns in (
("execution_prompt_snapshots", "raw_text_id,snapshot_id"),
("execution_contexts", "positive_snapshot_id,context_id"),
("execution_run_contexts", "context_id,run_id,anchor_id"),
("execution_runs", "observed_dequeue_at,session_id,queue_item_id,run_id"),
):
conn.execute(
f"CREATE INDEX IF NOT EXISTS idx_history_read_{table} "
f"ON {table}({columns})"
)
error = history_schema_error(conn, allow_missing=False)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE execution_history_schema")
except BaseException:
conn.execute("ROLLBACK TO execution_history_schema")
conn.execute("RELEASE execution_history_schema")
raise
def rotate_history_epoch(conn: sqlite3.Connection) -> str:
"""Publish a fresh staging epoch without rewriting restored run epochs."""
epoch = str(uuid4())
with conn:
cursor = conn.execute(
"UPDATE execution_history_metadata SET storage_epoch=? WHERE "
"singleton_id=1",
(epoch,),
)
if cursor.rowcount != 1:
raise sqlite3.DatabaseError("Missing execution history metadata")
return epoch
+218
View File
@@ -0,0 +1,218 @@
"""Owned schema and bounded restore validation for frozen output locators."""
import sqlite3
from .execution_history_schema import _compact, _uuid_check
try:
from ..utils.execution_outputs import (
BUNDLE_OVERHEAD,
MAX_BUNDLE_BYTES,
MAX_IMAGE_POSITIONS,
MAX_REFERENCE_BYTES,
REFERENCE_OVERHEAD,
STATE_REASONS,
OutputBundle,
OutputReference,
validate_output_bundle,
)
except ImportError:
from utils.execution_outputs import (
BUNDLE_OVERHEAD,
MAX_BUNDLE_BYTES,
MAX_IMAGE_POSITIONS,
MAX_REFERENCE_BYTES,
REFERENCE_OVERHEAD,
STATE_REASONS,
OutputBundle,
OutputReference,
validate_output_bundle,
)
TABLE_COLUMNS = {
"execution_output_capture": (
("run_id", "TEXT", 1, 1),
("state", "TEXT", 1, 0),
("reason", "TEXT", 1, 0),
("format_version", "INTEGER", 1, 0),
("source", "TEXT", 1, 0),
),
"execution_run_outputs": (
("run_id", "TEXT", 1, 1),
("output_node_id", "TEXT", 1, 2),
("image_ordinal", "INTEGER", 1, 3),
("folder_type", "TEXT", 1, 0),
("subfolder", "TEXT", 1, 0),
("filename", "TEXT", 1, 0),
("path_style", "TEXT", 1, 0),
("root_key", "TEXT", 0, 0),
),
}
TABLE_FOREIGN_KEYS = {
"execution_output_capture": (("run_id", "execution_runs", "run_id"),),
"execution_run_outputs": (("run_id", "execution_output_capture", "run_id"),),
}
def _text_check(column: str, maximum: int, *, characters: int | None = None) -> str:
return (
f"typeof({column}) = 'text' AND length({column}) <= "
f"{maximum if characters is None else characters} "
f"AND length(CAST({column} AS BLOB)) <= {maximum} "
f"AND instr(CAST({column} AS BLOB), x'00') = 0"
)
def _enum_check(column: str, values: tuple | frozenset) -> str:
return (
f"{_text_check(column, 32)} AND {column} IN ("
+ ",".join("'" + value + "'" for value in sorted(values))
+ ")"
)
_RUN_ID_CHECK = f"{_uuid_check('run_id')} AND {_text_check('run_id', 36)}"
TABLE_CHECKS = {
"execution_output_capture": (
_RUN_ID_CHECK,
_enum_check("state", tuple(STATE_REASONS)),
_enum_check("reason", frozenset.union(*STATE_REASONS.values())),
"typeof(format_version) = 'integer' AND format_version = 1",
f"{_text_check('source', 18)} AND source = 'host_final_history'",
),
"execution_run_outputs": (
_RUN_ID_CHECK,
f"{_text_check('output_node_id', 512, characters=128)} "
"AND length(output_node_id) >= 1",
"typeof(image_ordinal) = 'integer' AND image_ordinal BETWEEN 0 AND 255",
_enum_check("folder_type", ("output", "temp")),
_text_check("subfolder", 2048),
f"{_text_check('filename', 1024)} AND length(filename) >= 1",
_enum_check("path_style", ("posix", "windows")),
f"root_key IS NULL OR ({_text_check('root_key', 64)} "
"AND length(root_key) = 64 AND root_key NOT GLOB '*[^0-9a-f]*')",
),
}
def _table_sql(table: str) -> str:
columns = TABLE_COLUMNS[table]
definitions = [
f"{name} {kind}{' NOT NULL' if required else ''}"
for name, kind, required, _ in columns
]
primary = [name for name, _, _, order in columns if order]
definitions.append(f"PRIMARY KEY ({', '.join(primary)})")
definitions.extend(f"CHECK({check})" for check in TABLE_CHECKS[table])
definitions.extend(
f"FOREIGN KEY ({column}) REFERENCES {target}({key}) "
"ON UPDATE NO ACTION ON DELETE NO ACTION"
for column, target, key in TABLE_FOREIGN_KEYS[table]
)
return f"CREATE TABLE {table} ({', '.join(definitions)})"
def install_output_schema(conn: sqlite3.Connection) -> None:
"""Install the two owned tables within the caller's migration transaction."""
for table in TABLE_COLUMNS:
conn.execute(_table_sql(table))
def output_schema_error(conn: sqlite3.Connection) -> str | None:
"""Reject altered schemas and invalid rows before returning bounded DTOs."""
if conn.execute(
"SELECT 1 FROM sqlite_master WHERE type IN ('trigger','view') LIMIT 1"
).fetchone():
return "Unsupported execution output trigger/view"
tables = {
row[0]: row[1]
for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
if row[0] in TABLE_COLUMNS
}
if tables.keys() != TABLE_COLUMNS.keys():
return "Incomplete execution output schema"
for table, expected in TABLE_COLUMNS.items():
columns = tuple(
(row[1], row[2].upper(), row[3], row[5])
for row in conn.execute(f"PRAGMA table_xinfo({table})")
)
if columns != expected or _compact(tables[table]) != _compact(
_table_sql(table)
):
return f"{table} has invalid execution output schema"
foreign = tuple(
sorted(
(row[3], row[2], row[4], row[5], row[6], row[7])
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
)
)
if foreign != tuple(
(*key, "NO ACTION", "NO ACTION", "NONE")
for key in TABLE_FOREIGN_KEYS[table]
):
return f"{table} has invalid execution output foreign keys"
for index in conn.execute(f"PRAGMA index_list({table})"):
name = index[1].replace('"', '""')
keys = tuple(
row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
)
if (
not index[2]
or index[3] != "pk"
or index[4]
or tuple(key[2] for key in keys)
!= tuple(column[0] for column in expected if column[3])
or any(key[3] or key[4] != "BINARY" for key in keys)
):
return f"{table} has unexpected execution output index/uniqueness"
# Run every scalar guard in SQLite before any evidence text is fetched.
predicate = " AND ".join(f"({check})" for check in TABLE_CHECKS[table])
if conn.execute(
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
).fetchone():
return f"{table} contains invalid execution output records"
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
return f"{table} contains orphan execution output records"
reference_bytes = f"{REFERENCE_OVERHEAD} + " + " + ".join(
f"coalesce(length(CAST({column} AS BLOB)), 0)"
for column in (
"output_node_id",
"folder_type",
"subfolder",
"filename",
"path_style",
"root_key",
)
)
if conn.execute(
"SELECT 1 FROM execution_run_outputs "
f"WHERE ({reference_bytes}) > {MAX_REFERENCE_BYTES} LIMIT 1"
).fetchone():
return "Oversized execution output reference"
if conn.execute(
"SELECT 1 FROM execution_run_outputs GROUP BY run_id "
f"HAVING COUNT(*) > {MAX_IMAGE_POSITIONS} OR "
f"{BUNDLE_OVERHEAD} + SUM({reference_bytes}) > {MAX_BUNDLE_BYTES} LIMIT 1"
).fetchone():
return "Oversized execution output bundle"
try:
for run_id, state, reason, version, source in conn.execute(
"SELECT run_id,state,reason,format_version,source "
"FROM execution_output_capture"
):
references = tuple(
OutputReference(*row)
for row in conn.execute(
"SELECT output_node_id,image_ordinal,folder_type,subfolder,"
"filename,path_style,root_key FROM execution_run_outputs "
"WHERE run_id=? ORDER BY output_node_id,image_ordinal",
(run_id,),
)
)
validate_output_bundle(
OutputBundle(references, state, reason, version, source)
)
except (ValueError, TypeError, OverflowError, sqlite3.Error) as exc:
return f"Invalid execution output content: {type(exc).__name__}"
return None
+153
View File
@@ -0,0 +1,153 @@
"""Immutable run-owned output references, independent of mutable galleries."""
import sqlite3
from contextlib import ExitStack
from typing import Callable, ContextManager
try:
from ..utils.execution_history_events import RunFinish, encode_command
from ..utils.execution_outputs import OutputBundle, validate_output_bundle
except ImportError:
from utils.execution_history_events import RunFinish, encode_command
from utils.execution_outputs import OutputBundle, validate_output_bundle
_REFERENCE_FIELDS = (
"output_node_id",
"image_ordinal",
"folder_type",
"subfolder",
"filename",
"path_style",
"root_key",
)
_REFERENCE_COLUMNS = ",".join(_REFERENCE_FIELDS)
def _capture(conn: sqlite3.Connection, run_id: str) -> sqlite3.Row | None:
capture = conn.execute(
"SELECT state,reason,format_version,source FROM execution_output_capture "
"WHERE run_id=?",
(run_id,),
).fetchone()
# Legacy callers may disable FK enforcement; never adopt orphan evidence.
if capture is None:
if conn.execute(
"SELECT 1 FROM execution_run_outputs WHERE run_id=? LIMIT 1", (run_id,)
).fetchone():
raise sqlite3.DatabaseError("Output references have no capture header")
elif (
conn.execute(
"SELECT 1 FROM execution_runs WHERE run_id=?", (run_id,)
).fetchone()
is None
):
raise sqlite3.DatabaseError("Output capture has no execution parent")
return capture
def _references(conn: sqlite3.Connection, run_id: str) -> list[tuple]:
return [
tuple(row)
for row in conn.execute(
f"SELECT {_REFERENCE_COLUMNS} FROM execution_run_outputs "
"WHERE run_id=? ORDER BY output_node_id,image_ordinal",
(run_id,),
)
]
class ExecutionOutputOperations:
"""Compose with ExecutionHistoryOperations on PromptDatabase."""
def write_execution_outputs_for_finish(
self,
command: RunFinish,
bundle: OutputBundle,
*,
generation_guard: Callable[[], ContextManager[bool]] | None = None,
) -> bool:
encode_command(command)
if type(command) is not RunFinish:
raise ValueError("Expected execution finish command")
validate_output_bundle(bundle)
header = (bundle.state, bundle.reason, bundle.format_version, bundle.source)
prepared = [
tuple(getattr(reference, field) for field in _REFERENCE_FIELDS)
for reference in bundle.references
]
# Enter after BEGIN and retain the guard until COMMIT/ROLLBACK finishes.
with ExitStack() as guards:
with self._execution_transaction(command.storage_epoch) as conn:
if generation_guard is not None and not guards.enter_context(
generation_guard()
):
return False
parent = conn.execute(
"SELECT run_id FROM execution_runs WHERE session_id=? "
"AND queue_item_id=? AND storage_epoch=?",
(command.session_id, command.queue_item_id, command.storage_epoch),
).fetchone()
if parent is None:
return False
run_id = parent[0]
capture = _capture(conn, run_id)
if capture is not None:
if capture[0] != "conflicting" and (
tuple(capture) != header
or _references(conn, run_id) != prepared
):
conn.execute(
"UPDATE execution_output_capture SET "
"state='conflicting',reason='conflicting' WHERE run_id=?",
(run_id,),
)
return True
conn.execute(
"INSERT INTO execution_output_capture "
"(run_id,state,reason,format_version,source) VALUES(?,?,?,?,?)",
(run_id, *header),
)
conn.executemany(
f"INSERT INTO execution_run_outputs (run_id,{_REFERENCE_COLUMNS}) "
"VALUES(?,?,?,?,?,?,?,?)",
[(run_id, *reference) for reference in prepared],
)
return True
def get_execution_outputs(self, run_id: str) -> dict:
"""Read frozen evidence in one snapshot without resolving its locators."""
conn = self.model.get_connection()
own_transaction = not conn.in_transaction
if own_transaction:
conn.execute("BEGIN")
try:
capture = _capture(conn, run_id)
if capture is None:
return {
"state": "not_captured",
"reason": None,
"format_version": None,
"source": None,
"references": [],
}
references = []
for row in _references(conn, run_id):
reference = dict(zip(_REFERENCE_FIELDS, row))
reference.update(
file_identity="unverified",
newly_generated="unknown",
context_membership="unknown",
transient=reference["folder_type"] == "temp",
)
references.append(reference)
return {
"state": capture[0],
"reason": capture[1],
"format_version": capture[2],
"source": capture[3],
"references": references,
}
finally:
if own_transaction:
conn.rollback()
+127
View File
@@ -0,0 +1,127 @@
"""Bounded exact-path saved display within the caller's history snapshot."""
import os
import sqlite3
try:
from ..utils.inspection_response import MAX_LIBRARY_BYTES, _library_projection
except ImportError:
from utils.inspection_response import MAX_LIBRARY_BYTES, _library_projection
def _display_bytes(value: bytes | None) -> str | None:
if value is None:
return None
return value.decode("utf-8", "replace")
def _tags(
conn: sqlite3.Connection, owner: int, *, pair: bool
) -> tuple[list[str], bool]:
table, column = (
("prompt_pair_tags", "pair_id") if pair else ("prompt_tags", "prompt_id")
)
tags = []
used = 0
cursor = conn.execute(
f"SELECT substr(CAST(t.name AS BLOB),1,?) FROM {table} pt "
f"JOIN tags t ON t.id=pt.tag_id WHERE pt.{column}=? LIMIT 129",
(MAX_LIBRARY_BYTES + 1, owner),
)
for row in cursor:
used += len(row[0])
if used > MAX_LIBRARY_BYTES or len(tags) == 128:
return [], True
tags.append(_display_bytes(row[0]))
return tags, False
def _saved_projection(conn: sqlite3.Connection, row: sqlite3.Row) -> tuple[dict, dict]:
saved = {"prompt_id": row["prompt_id"]}
pairs = list(
conn.execute(
"SELECT pp.id,pp.positive_prompt_id,"
"substr(CAST(pp.title AS BLOB),1,?) AS title,"
"substr(CAST(pp.category AS BLOB),1,?) AS category "
"FROM image_prompt_pairs ip JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"WHERE ip.image_id=? LIMIT 2",
(MAX_LIBRARY_BYTES + 1, MAX_LIBRARY_BYTES + 1, row["id"]),
)
)
if len(pairs) > 1:
return saved, _library_projection({"status": "limited"})
pair = pairs[0] if pairs else None
title = pair["title"] if pair is not None else None
category = pair["category"] if pair is not None else row["category"]
tags, limited = _tags(
conn,
pair["id"] if pair is not None else row["prompt_id"],
pair=pair is not None,
)
limited = limited or any(
value is not None and len(value) > MAX_LIBRARY_BYTES
for value in (title, category)
)
if pair is not None:
saved["pair"] = {
"id": pair["id"],
"positive_prompt_id": pair["positive_prompt_id"],
}
library = _library_projection(
{
"status": "linked",
"positive_id": row["prompt_id"],
"pair_id": pair["id"] if pair is not None else None,
"title": _display_bytes(title),
"category": _display_bytes(category),
"tags": tags,
"metadata_scope": "pair" if pair is not None else "positive",
}
)
if limited:
library = _library_projection({"status": "limited", "tags": []})
return saved, library
def read_history_gallery(
conn: sqlite3.Connection, paths: tuple[str, ...]
) -> dict[str, dict]:
"""Read at most two rows/path and two pair associations/unique row; no writes."""
if not conn.in_transaction:
raise sqlite3.ProgrammingError("Saved history display requires caller snapshot")
if (
type(paths) is not tuple
or len(paths) > 100
or any(
type(path) is not str
or not os.path.isabs(path)
or path != os.path.normcase(os.path.normpath(path))
or "\0" in path
for path in paths
)
):
raise ValueError("Invalid history gallery paths")
result = {}
for path in dict.fromkeys(paths):
rows = list(
conn.execute(
"SELECT gi.id,gi.prompt_id,"
"substr(CAST(p.category AS BLOB),1,?) AS category "
"FROM generated_images gi JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=? LIMIT 2",
(MAX_LIBRARY_BYTES + 1, path, path),
)
)
match = "none" if not rows else "unique" if len(rows) == 1 else "ambiguous"
saved, library = None, _library_projection(
{"status": "none" if not rows else "ambiguous"}
)
if match == "unique":
saved, library = _saved_projection(conn, rows[0])
result[path] = {
"gallery_match": match,
"gallery_image_id": rows[0]["id"] if match == "unique" else None,
"saved": saved,
"library": library,
}
return result
+527
View File
@@ -0,0 +1,527 @@
"""Snapshot-owned, bounded projections of immutable execution evidence."""
import codecs
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
import hashlib
import json
import re
import sqlite3
from typing import Iterator
from .execution_contexts import content_digest
from .execution_history import HistoryEpochChanged
from .execution_outputs import _capture, _REFERENCE_COLUMNS
from .library import _ELIGIBLE
RunKey = tuple[int, str, int, str]
class HistoryCaptureChanged(ValueError):
"""A continuation no longer describes the stored output capture header."""
@dataclass(frozen=True)
class WindowPosition:
raw_text_id: str
upper: RunKey
before: RunKey
@dataclass(frozen=True)
class OutputPosition:
capture_fingerprint: str
after: tuple[str, int]
_DIGEST = re.compile(r"[0-9a-f]{64}")
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
def _valid_digest(value: object) -> bool:
return type(value) is str and _DIGEST.fullmatch(value) is not None
def _valid_run_key(value: object) -> bool:
return (
type(value) is tuple
and len(value) == 4
and all(type(value[i]) is int and 0 <= value[i] <= 2**63 - 1 for i in (0, 2))
and all(
type(value[i]) is str and _UUID.fullmatch(value[i]) is not None
for i in (1, 3)
)
)
def _valid_coordinate(value: object) -> bool:
if type(value) is not tuple or len(value) != 2:
return False
node, ordinal = value
if not (
type(node) is str
and 1 <= len(node) <= 128
and "\0" not in node
and type(ordinal) is int
and 0 <= ordinal <= 255
):
return False
try:
return len(node.encode("utf-8")) <= 512
except UnicodeError:
return False
def _validate_window_position(position: WindowPosition | None) -> None:
if position is not None and not (
type(position) is WindowPosition
and _valid_digest(position.raw_text_id)
and _valid_run_key(position.upper)
and _valid_run_key(position.before)
):
raise ValueError("Invalid history window position")
def _validate_output_position(position: OutputPosition | None) -> None:
if position is not None and not (
type(position) is OutputPosition
and _valid_digest(position.capture_fingerprint)
and _valid_coordinate(position.after)
):
raise ValueError("Invalid history output position")
_snapshots: ContextVar[tuple] = ContextVar("history_read_snapshots", default=())
_KEY = "(r.observed_dequeue_at,r.session_id,r.queue_item_id,r.run_id)"
_ORDER = (
"r.observed_dequeue_at DESC,r.session_id COLLATE BINARY DESC,"
"r.queue_item_id DESC,r.run_id COLLATE BINARY DESC"
)
_MATCHING = (
"SELECT DISTINCT rc.run_id FROM execution_prompt_snapshots s "
"JOIN execution_contexts c ON c.positive_snapshot_id=s.snapshot_id "
"JOIN execution_run_contexts rc ON rc.context_id=c.context_id "
"WHERE s.raw_text_id=?"
)
def _limit(limit: int) -> None:
if type(limit) is not int or not 1 <= limit <= 100:
raise ValueError("History limit must be an integer from 1 to 100")
def _key(row: dict) -> RunKey:
return tuple(
row[name]
for name in ("observed_dequeue_at", "session_id", "queue_item_id", "run_id")
)
def _context_capture(conn: sqlite3.Connection, run_id: str) -> dict:
row = conn.execute(
"SELECT state,reason,format_version FROM execution_context_capture "
"WHERE run_id=?",
(run_id,),
).fetchone()
return (
dict(row)
if row is not None
else {"state": "not_captured", "reason": "missing", "format_version": None}
)
def _output_header(conn: sqlite3.Connection, run_id: str) -> tuple[dict, int | None]:
row = _capture(conn, run_id)
if row is None:
return (
dict(state="not_captured", reason=None, format_version=None, source=None),
None,
)
count = conn.execute(
"SELECT COUNT(*) FROM execution_run_outputs WHERE run_id=?", (run_id,)
).fetchone()[0]
return dict(row), count
def _list_status(capture: dict) -> str:
if capture["state"] == "not_captured":
return "not_captured"
return "complete" if capture["state"] == "complete" else "incomplete"
def _output_summary(conn: sqlite3.Connection, run_id: str) -> dict:
capture, count = _output_header(conn, run_id)
return dict(
capture, retained_occurrence_count=count, list_status=_list_status(capture)
)
def _fingerprint(capture: dict, count: int | None) -> str:
return hashlib.sha256(
json.dumps(
[
1,
*(capture[k] for k in ("state", "reason", "format_version", "source")),
count,
],
ensure_ascii=False,
allow_nan=False,
separators=(",", ":"),
).encode("utf-8")
).hexdigest()
def _receipt(run_id: str, row: sqlite3.Row) -> dict:
return {
"occurrence": {
"run_id": run_id,
"output_node_id": row["output_node_id"],
"image_ordinal": row["image_ordinal"],
},
"reference": {
key: row[key]
for key in (
"folder_type",
"subfolder",
"filename",
"path_style",
"root_key",
)
},
"file_identity": "unverified",
"newly_generated": "unknown",
"context_membership": "unknown",
"transient": row["folder_type"] == "temp",
}
class HistoryReadOperations:
"""Read evidence only inside an explicitly owned, epoch-first snapshot."""
@contextmanager
def recorded_history_snapshot(
self, expected_epoch: str | None = None
) -> Iterator[str]:
conn = self.model.get_connection()
if conn.in_transaction:
raise sqlite3.ProgrammingError(
"History reads require their own transaction"
)
conn.execute("BEGIN")
token = _snapshots.set((*_snapshots.get(), conn))
try:
epoch = self.get_execution_epoch()
if expected_epoch is not None and epoch != expected_epoch:
raise HistoryEpochChanged(epoch)
yield epoch
finally:
try:
conn.rollback()
finally:
_snapshots.reset(token)
def _history_connection(self) -> sqlite3.Connection:
conn = self.model.get_connection()
if not conn.in_transaction or conn not in _snapshots.get():
raise sqlite3.ProgrammingError(
"History read requires recorded_history_snapshot"
)
return conn
def list_history_window(
self,
positive_id: int,
*,
limit: int = 40,
after: WindowPosition | None = None,
restart: bool = False,
) -> dict | None:
conn = self._history_connection()
_limit(limit)
_validate_window_position(after)
if type(restart) is not bool or (
restart and (after is None or after.before != after.upper)
):
raise ValueError("Invalid history restart position")
saved = conn.execute(
"SELECT CASE WHEN typeof(p.text)='text' AND "
"length(CAST(p.text AS BLOB))<=65536 THEN p.text ELSE NULL END AS text "
f"FROM prompts p WHERE p.id=? AND ({_ELIGIBLE})",
(positive_id,),
).fetchone()
if saved is None:
return None
text = saved["text"]
raw_id = None if text is None else content_digest("text", (text,))
if after is not None and after.raw_text_id != raw_id:
raise ValueError("History position text scope changed")
group = {
"positive_id": positive_id,
"raw_text_id": raw_id,
"relation": "exact_frozen_raw_text",
"scope_status": "uncapturable_text" if text is None else "no_matches",
}
result = dict(
group=group,
recorded_run_count=0,
items=[],
context_summaries=[],
has_more=False,
next_position=None,
resume_position=None,
)
if (
raw_id is None
or conn.execute(
"SELECT 1 FROM execution_context_texts WHERE text_id=? "
"AND text_value=? COLLATE BINARY",
(raw_id, text),
).fetchone()
is None
):
return result
where = f"r.run_id IN ({_MATCHING})"
params = [raw_id]
if after is not None:
where += f" AND {_KEY}<=(?,?,?,?)"
params.extend(after.upper)
count = conn.execute(
f"SELECT COUNT(*) FROM execution_runs r WHERE {where}", params
).fetchone()[0]
result["recorded_run_count"] = count
group["scope_status"] = "matched" if count else "no_matches"
if after is not None and not restart:
where += f" AND {_KEY}<(?,?,?,?)"
params.extend(after.before)
rows = [
dict(row)
for row in conn.execute(
f"SELECT r.* FROM execution_runs r WHERE {where} "
f"ORDER BY {_ORDER} LIMIT ?",
(*params, limit + 1),
)
]
more = len(rows) > limit
rows = rows[:limit]
if not rows:
return result
upper = after.upper if after is not None else _key(rows[0])
result["resume_position"] = WindowPosition(raw_id, upper, upper)
if more:
result["next_position"] = WindowPosition(raw_id, upper, _key(rows[-1]))
result["has_more"] = more
rows.reverse()
by_run = {row["run_id"]: row for row in rows}
for row in rows:
row["diagnostics_available"] = any(
row[field] is not None
for field in ("node_id", "node_type", "exception_type")
)
for field in ("node_id", "node_type", "exception_type"):
del row[field]
row["diagnostics_omitted"] = bool(row["diagnostics_omitted"])
row["context_capture"] = _context_capture(conn, row["run_id"])
row["output_summary"] = _output_summary(conn, row["run_id"])
row["matching_anchors"] = []
placeholders = ",".join("?" for _ in rows)
anchors = conn.execute(
"SELECT rc.run_id,rc.anchor_id,rc.context_id,rc.evidence_json,"
"c.positive_snapshot_id,c.negative_snapshot_id,"
"substr(CAST(t.text_value AS BLOB),1,512) AS prefix,"
"length(CAST(t.text_value AS BLOB)) AS byte_length "
"FROM execution_run_contexts rc JOIN execution_contexts c "
"ON c.context_id=rc.context_id JOIN execution_prompt_snapshots p "
"ON p.snapshot_id=c.positive_snapshot_id "
"LEFT JOIN execution_prompt_snapshots n "
"ON n.snapshot_id=c.negative_snapshot_id "
"LEFT JOIN execution_context_texts t ON t.text_id=n.raw_text_id "
f"WHERE rc.run_id IN ({placeholders}) AND p.raw_text_id=? "
"ORDER BY rc.anchor_id COLLATE BINARY,rc.run_id COLLATE BINARY",
(*by_run, raw_id),
)
summaries = {}
evidence_by_context = {}
for anchor in anchors:
reference = {
key: anchor[key]
for key in (
"anchor_id",
"context_id",
"positive_snapshot_id",
"negative_snapshot_id",
)
}
by_run[anchor["run_id"]]["matching_anchors"].append(reference)
context_id = anchor["context_id"]
if context_id not in summaries:
prefix = b"" if anchor["byte_length"] == 0 else anchor["prefix"]
preview = (
None
if prefix is None
else codecs.getincrementaldecoder("utf-8")().decode(
prefix, final=False
)[:100]
)
evidence_by_context[context_id] = []
summaries[context_id] = {
key: anchor[key]
for key in (
"context_id",
"positive_snapshot_id",
"negative_snapshot_id",
)
}
summaries[context_id]["negative_raw"] = {
"state": (
"unknown"
if preview is None
else "explicit_empty" if not preview else "known"
),
"preview": preview,
"truncated": (
False
if preview is None
else len(preview.encode("utf-8")) < anchor["byte_length"]
),
"evidence": evidence_by_context[context_id],
}
negative = json.loads(anchor["evidence_json"])["negative"]
evidence = (
negative["raw"]
if negative is not None
else dict(provenance="unknown", source="unknown", reason="missing")
)
evidence_by_context[context_id].append(
{
"run_id": anchor["run_id"],
"anchor_id": anchor["anchor_id"],
**evidence,
}
)
# Context order follows its first chronological matching run, then anchor.
ordered = {}
for row in rows:
for anchor in row["matching_anchors"]:
ordered.setdefault(
anchor["context_id"], summaries[anchor["context_id"]]
)
result["items"] = rows
result["context_summaries"] = list(ordered.values())
return result
def get_history_run(self, run_id: str) -> dict | None:
conn = self._history_connection()
run = self.get_execution_run(run_id)
if run is None:
return None
capture = self.get_execution_contexts(run_id)
snapshots = {
row["anchor_id"]: row
for row in conn.execute(
"SELECT rc.anchor_id,c.positive_snapshot_id,c.negative_snapshot_id,"
"p.raw_text_id AS positive_raw_text_id "
"FROM execution_run_contexts rc JOIN execution_contexts c "
"ON c.context_id=rc.context_id LEFT JOIN execution_prompt_snapshots p "
"ON p.snapshot_id=c.positive_snapshot_id WHERE rc.run_id=?",
(run_id,),
)
}
contexts = capture.pop("contexts")
for context in contexts:
row = snapshots[context["anchor_id"]]
context.update(
{
key: row[key]
for key in (
"positive_snapshot_id",
"negative_snapshot_id",
"positive_raw_text_id",
)
}
)
pair = context["pair_id"]
context["navigation"] = {
"positive_available": context["positive_prompt_id"] is not None
and conn.execute(
f"SELECT 1 FROM prompts p WHERE p.id=? AND ({_ELIGIBLE})",
(context["positive_prompt_id"],),
).fetchone()
is not None,
"negative_available": context["negative_prompt_id"] is not None,
"pair_available": pair is not None
and conn.execute(
"SELECT 1 FROM prompt_pairs WHERE id=? AND archived=0", (pair,)
).fetchone()
is not None,
}
return dict(
run=run,
context_capture=capture,
contexts=contexts,
output_summary=_output_summary(conn, run_id),
)
def list_history_outputs(
self,
run_id: str,
*,
limit: int = 40,
after: OutputPosition | None = None,
) -> dict | None:
conn = self._history_connection()
_limit(limit)
_validate_output_position(after)
# Validate header/orphans even when the requested parent does not exist.
capture, count = _output_header(conn, run_id)
if self.get_execution_run(run_id) is None:
return None
fingerprint = _fingerprint(capture, count)
if after is not None and after.capture_fingerprint != fingerprint:
raise HistoryCaptureChanged("History output capture changed")
where = "run_id=?"
params = [run_id]
if after is not None:
where += " AND (output_node_id COLLATE BINARY,image_ordinal)>(?,?)"
params.extend(after.after)
rows = list(
conn.execute(
f"SELECT {_REFERENCE_COLUMNS} FROM execution_run_outputs WHERE {where} "
"ORDER BY output_node_id COLLATE BINARY,image_ordinal LIMIT ?",
(*params, limit + 1),
)
)
more = len(rows) > limit
rows = rows[:limit]
position = (
OutputPosition(
fingerprint, (rows[-1]["output_node_id"], rows[-1]["image_ordinal"])
)
if more
else None
)
return dict(
capture=capture,
retained_occurrence_count=count,
list_status=_list_status(capture),
capture_fingerprint=fingerprint,
items=[_receipt(run_id, row) for row in rows],
has_more=more,
next_position=position,
)
def get_history_occurrence(
self, run_id: str, node: str, ordinal: int
) -> dict | None:
conn = self._history_connection()
capture, _ = _output_header(conn, run_id)
row = conn.execute(
f"SELECT {_REFERENCE_COLUMNS} FROM execution_run_outputs "
"WHERE run_id=? AND output_node_id=? COLLATE BINARY AND image_ordinal=?",
(run_id, node, ordinal),
).fetchone()
return (
dict(capture=capture, receipt=_receipt(run_id, row))
if row is not None
else None
)
+139
View File
@@ -0,0 +1,139 @@
"""Persisted image favorites, keyed by gallery root + relative path.
The root key is the root's real, case-normalized path and the relative key is
case-normalized the same way (see ``py/api/images.py``); both stay
server-side. Folder names alone are not unique across roots, so a favorite
follows the directory itself when the configured roots are reordered.
Optional in old backups, required once a database is current (see
``library_schema.py``).
"""
import re
import sqlite3
from typing import Iterable
TABLE = "image_favorites"
MAX_KEY_LENGTH = 4096
_LOOKUP_CHUNK = 500
_DDL = (
f"CREATE TABLE {TABLE} (root_key TEXT NOT NULL, relative_path TEXT NOT NULL, "
"created_at TEXT NOT NULL, PRIMARY KEY(root_key, relative_path))"
)
_AUTOINDEX = f"sqlite_autoindex_{TABLE}_1"
_COLUMNS = (
("root_key", "TEXT", 1, 1),
("relative_path", "TEXT", 1, 2),
("created_at", "TEXT", 1, 0),
)
def _compact(sql: str) -> str:
return re.sub(r"\s+", "", sql).lower()
def image_favorites_schema_error(
conn: sqlite3.Connection, *, allow_missing: bool = True
) -> str | None:
"""Reject unsupported favorites schemas and invalid persisted values.
Reads only. A missing table is acceptable for backups that predate it
unless ``allow_missing`` is false.
"""
objects = conn.execute(
"SELECT type,name,sql FROM sqlite_master WHERE name=? "
"OR (type IN ('index','trigger') AND tbl_name=?)",
(TABLE, TABLE),
).fetchall()
if not objects:
return None if allow_missing else "Missing image favorites schema"
tables = [sql for kind, _name, sql in objects if kind == "table"]
others = {(kind, name, sql) for kind, name, sql in objects if kind != "table"}
if len(tables) != 1 or others != {("index", _AUTOINDEX, None)}:
return "Image favorites name is reserved by an unsupported schema object"
if _compact(tables[0] or "") != _compact(_DDL):
return "Image favorites has an invalid table definition"
columns = tuple(
(row[1], row[2].upper(), row[3], row[5])
for row in conn.execute(f"PRAGMA table_info({TABLE})")
)
if columns != _COLUMNS:
return "Image favorites has invalid columns"
# Bytes bound characters from above (UTF-8 is at most 4 bytes each).
limit = MAX_KEY_LENGTH * 4
if conn.execute(
f"SELECT 1 FROM {TABLE} WHERE typeof(root_key) != 'text' "
"OR typeof(relative_path) != 'text' OR typeof(created_at) != 'text' "
f"OR length(CAST(root_key AS BLOB)) > {limit} "
f"OR length(CAST(relative_path AS BLOB)) > {limit} "
"OR length(CAST(created_at AS BLOB)) > 64 LIMIT 1"
).fetchone():
return "Image favorites contains invalid values"
for root, path in conn.execute(
f"SELECT CAST(root_key AS BLOB), CAST(relative_path AS BLOB) FROM {TABLE} "
f"WHERE length(CAST(root_key AS BLOB)) > {MAX_KEY_LENGTH} "
f"OR length(CAST(relative_path AS BLOB)) > {MAX_KEY_LENGTH}"
):
for raw in (root, path):
if len(bytes(raw).decode("utf-8", "replace")) > MAX_KEY_LENGTH:
return "Image favorites contains invalid values"
return None
def migrate_image_favorites(conn: sqlite3.Connection) -> None:
"""Install the table atomically; validate before and after creating it."""
conn.execute("SAVEPOINT image_favorites_schema")
try:
error = image_favorites_schema_error(conn)
if error:
raise sqlite3.DatabaseError(error)
conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
error = image_favorites_schema_error(conn, allow_missing=False)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE image_favorites_schema")
except BaseException:
conn.execute("ROLLBACK TO image_favorites_schema")
conn.execute("RELEASE image_favorites_schema")
raise
class ImageFavoriteOperations:
"""PromptDatabase mixin: set, clear and look up image favorites."""
def set_image_favorite(self, root_key: str, relative_path: str) -> bool:
"""Mark one file as a favorite; repeating it is a no-op. Returns True."""
from .models import utc_now_iso # models imports this module's schema
with self.model.get_connection() as conn:
conn.execute(
"INSERT OR IGNORE INTO image_favorites VALUES (?, ?, ?)",
(root_key, relative_path, utc_now_iso()),
)
return True
def clear_image_favorite(self, root_key: str, relative_path: str) -> bool:
"""Remove one favorite; True when a row was removed."""
with self.model.get_connection() as conn:
cursor = conn.execute(
"DELETE FROM image_favorites WHERE root_key=? AND relative_path=?",
(root_key, relative_path),
)
return cursor.rowcount > 0
def get_image_favorites(
self, root_key: str, relative_paths: Iterable[str]
) -> set[str]:
"""The subset of *relative_paths* favorited under *root_key*."""
wanted = list(dict.fromkeys(relative_paths))
found: set[str] = set()
conn = self.model.get_connection()
for start in range(0, len(wanted), _LOOKUP_CHUNK):
chunk = wanted[start : start + _LOOKUP_CHUNK]
marks = ",".join("?" * len(chunk))
rows = conn.execute( # only "?" placeholders are interpolated
"SELECT relative_path FROM image_favorites WHERE root_key=? "
f"AND relative_path IN ({marks})", # nosec B608
(root_key, *chunk),
)
found.update(path for (path,) in rows)
return found
+628
View File
@@ -0,0 +1,628 @@
"""Read-only positive-parent views over existing prompt and pair tables."""
import datetime
import sqlite3
from collections.abc import Iterator
from contextlib import contextmanager
from .library_tokens import library_edit_token
_ELIGIBLE = """(EXISTS(SELECT 1 FROM prompt_pairs ep WHERE ep.positive_prompt_id=p.id)
OR EXISTS(SELECT 1 FROM generated_images eg WHERE eg.prompt_id=p.id)
OR NOT EXISTS(SELECT 1 FROM prompt_pairs en WHERE en.negative_prompt_id=p.id)
OR EXISTS(SELECT 1 FROM prompt_library_state es WHERE es.prompt_id=p.id))"""
_FROM = "prompts p LEFT JOIN prompt_library_state ls ON ls.prompt_id=p.id"
# The characters JavaScript's String.prototype.trim() removes, so a title the
# display treats as blank also falls back to the prompt text for name sorting.
_TRIM_CODES = (9, 10, 11, 12, 13, 32, 160, 5760, *range(8192, 8203))
_TRIM_CODES += (8232, 8233, 8239, 8287, 12288, 65279)
_BLANK = f"char({','.join(map(str, _TRIM_CODES))})"
_JS_TRIM = "".join(map(chr, _TRIM_CODES))
# The Library's display title: a stored title, else the prompt's first line.
DISPLAY_TITLE_LIMIT = 80
def _utf16_units(text: str) -> int:
"""JavaScript ``length``: characters beyond the BMP count as two units."""
return len(text) + sum(1 for ch in text if ord(ch) > 0xFFFF)
def _utf16_prefix(text: str, units: int) -> str:
"""The longest prefix within ``units`` UTF-16 units, never splitting a pair."""
used = 0
for index, ch in enumerate(text):
used += 2 if ord(ch) > 0xFFFF else 1
if used > units:
return text[:index]
return text
def titled(stored_titles: tuple, text: object) -> tuple[str | None, str | None]:
"""Return ``(title, source)``: a stored title, else the text's first line.
``source`` is ``"stored"`` for the first non-blank stored title (blank as
in JavaScript's trim, like the Library view), ``"text"`` for the first line,
and None when both are empty. The first line is trimmed and, past
DISPLAY_TITLE_LIMIT UTF-16 units, cut with an ellipsis like the Library's
``libraryTitle()`` (never inside a surrogate pair).
"""
for title in stored_titles:
if isinstance(title, str) and title.strip(_JS_TRIM):
return title, "stored"
if not isinstance(text, str):
return None, None
first = text.split("\n")[0].strip(_JS_TRIM)
if _utf16_units(first) > DISPLAY_TITLE_LIMIT:
first = _utf16_prefix(first, DISPLAY_TITLE_LIMIT) + "\u2026"
return (first, "text") if first else (None, None)
def display_title(stored_titles: tuple, text: object) -> str | None:
"""The Library display title of ``titled``, without its source."""
return titled(stored_titles, text)[0]
def utc_iso(value: object) -> str | None:
"""A stored timestamp as UTC ISO text, or None when absent or unreadable.
Usage writes are already UTC ISO; a naive value (SQLite's CURRENT_TIMESTAMP
form) is read as UTC, and an offset is converted to UTC. One bad row must
not fail a whole Library read, so a value past the datetime range after
conversion (OverflowError) is unreadable too. Python 3.11+ fromisoformat
accepts more ISO forms than 3.10 (e.g. a 2-digit fraction); every form the
app writes (utc_now_iso, the 3.2.4 backfill, CURRENT_TIMESTAMP, Z) parses
the same on both, so only hand-edited values can differ by version.
"""
if not isinstance(value, str):
return None
text = value.strip()
if text.endswith(("Z", "z")):
text = text[:-1] + "+00:00"
try:
stamp = datetime.datetime.fromisoformat(text)
if stamp.tzinfo is None:
stamp = stamp.replace(tzinfo=datetime.timezone.utc)
return stamp.astimezone(datetime.timezone.utc).isoformat()
except (ValueError, OverflowError):
return None
_NAME = (
f"LOWER(CASE WHEN TRIM(COALESCE(ls.title,''),{_BLANK})!='' "
"THEN ls.title ELSE p.text END)"
)
_ORDERS = {
"newest": ("COALESCE(julianday(p.created_at),0)", "DESC"),
"oldest": ("COALESCE(julianday(p.created_at),0)", "ASC"),
"name": (_NAME, "ASC"),
"rating": ("COALESCE(p.rating,0)", "DESC"),
}
# Pair archive scopes for variant and image reads, as conditions on alias pp.
ARCHIVE_SCOPES = {
"active": " AND pp.archived=0",
"archived": " AND pp.archived=1",
"all": "",
}
def _escape(value: str) -> str:
return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
def _limit(value: int) -> None:
if type(value) is not int or not 1 <= value <= 200:
raise ValueError("limit must be between 1 and 200")
def _archive(value: str) -> str:
"""Return the pair-archive condition for one scope (alias ``pp``)."""
if not isinstance(value, str) or value not in ARCHIVE_SCOPES:
raise ValueError("archive must be one of active, archived or all")
return ARCHIVE_SCOPES[value]
def _filter(text: str, category: str | None, tags: list[str]) -> tuple[str, list]:
if not isinstance(text, str) or len(text) > 500:
raise ValueError("Search text must be a string of at most 500 characters")
if category is not None and (not isinstance(category, str) or len(category) > 200):
raise ValueError("Invalid library category")
if (
not isinstance(tags, list)
or len(tags) > 30
or any(not isinstance(tag, str) or not tag or len(tag) > 200 for tag in tags)
):
raise ValueError("Invalid library tags")
parent, pair, parent_params, pair_params = [], ["pp.archived=0"], [], []
if text:
parent.append("(p.text LIKE ? ESCAPE '\\' OR ls.title LIKE ? ESCAPE '\\')")
parent_params.extend([f"%{_escape(text)}%"] * 2)
pair.append(
"(p.text LIKE ? ESCAPE '\\' OR ls.title LIKE ? ESCAPE '\\' "
"OR n.text LIKE ? ESCAPE '\\' OR pp.title LIKE ? ESCAPE '\\')"
)
pair_params.extend([f"%{_escape(text)}%"] * 4)
if category is not None:
parent.append("p.category=?")
pair.append("pp.category=?")
parent_params.append(category)
pair_params.append(category)
for tag in tags:
parent.append(
"EXISTS(SELECT 1 FROM prompt_tags pt JOIN tags t ON t.id=pt.tag_id "
"WHERE pt.prompt_id=p.id AND t.name=?)"
)
pair.append(
"EXISTS(SELECT 1 FROM prompt_pair_tags pt JOIN tags t ON t.id=pt.tag_id "
"WHERE pt.pair_id=pp.id AND t.name=?)"
)
parent_params.append(tag)
pair_params.append(tag)
condition = (
"("
+ (" AND ".join(parent) or "1=1")
+ (
") OR EXISTS(SELECT 1 FROM prompt_pairs pp "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE pp.positive_prompt_id=p.id AND "
)
+ " AND ".join(pair)
+ ")"
)
return f"({_ELIGIBLE}) AND ({condition})", [*parent_params, *pair_params]
class LibraryOperations:
"""Group reusable negative variants without changing legacy prompt queries.
Each public read runs inside ``library_snapshot``; the ``_``-prefixed
helpers take that snapshot's connection and epoch so routes can share it.
"""
@contextmanager
def library_snapshot(
self, *, include_ceiling: bool = False
) -> Iterator[tuple[sqlite3.Connection, str, int | None]]:
"""Hold one read transaction, reading the epoch (and ceiling) first."""
conn = self.model.get_connection()
if conn.in_transaction:
raise sqlite3.ProgrammingError("Library reads require their own snapshot")
conn.execute("BEGIN")
try:
row = conn.execute(
"SELECT storage_epoch,"
+ (
"(SELECT COALESCE(MAX(id),0) FROM prompts)"
if include_ceiling
else "NULL"
)
+ " FROM execution_history_metadata WHERE singleton_id=1"
).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
yield conn, row[0], row[1]
finally:
if conn.in_transaction:
conn.rollback()
def _library_parents(
self,
conn: sqlite3.Connection,
epoch: str,
where: str,
params: list,
suffix: str = "",
sort_expression: str | None = None,
) -> list[dict]:
rows = conn.execute(
"SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes,p.created_at,"
"p.run_count,p.last_used_at,"
+ (f"{sort_expression} AS _sort," if sort_expression else "")
+ "(SELECT COUNT(*) FROM prompt_pairs v WHERE v.positive_prompt_id=p.id "
"AND v.archived=0) AS variant_count,"
"(SELECT COUNT(*) FROM generated_images g "
"WHERE g.prompt_id=p.id) AS image_count "
"FROM " + _FROM + " WHERE " + where + suffix,
params,
).fetchall()
result = [{**dict(row), "tags": []} for row in rows]
if result:
by_id = {row["id"]: row for row in result}
marks = ",".join("?" for _ in result)
for row in conn.execute(
"SELECT pt.prompt_id,t.name FROM prompt_tags pt "
"JOIN tags t ON t.id=pt.tag_id "
f"WHERE pt.prompt_id IN ({marks}) ORDER BY t.name",
list(by_id),
):
by_id[row["prompt_id"]]["tags"].append(row["name"])
for row in result:
# Usage is shown, never compared: the token hashes owner fields only.
row["run_count"] = row["run_count"] if type(row["run_count"]) is int else 0
row["last_used_at"] = utc_iso(row["last_used_at"])
row["edit_token"] = library_edit_token(epoch, row)
return result
def _list_library(
self,
conn: sqlite3.Connection,
epoch: str,
ceiling: int,
*,
text: str = "",
category: str | None = None,
tags: list[str] | None = None,
sort: str = "newest",
limit: int = 40,
after: dict | None = None,
) -> dict:
_limit(limit)
if sort not in _ORDERS:
raise ValueError("Unknown library sort order")
where, params = _filter(text, category, [] if tags is None else tags)
if after is not None:
if (
not isinstance(after, dict)
or set(after) != {"ceiling", "value", "id"}
or type(after["ceiling"]) is not int
or type(after["id"]) is not int
):
raise ValueError("Invalid library cursor")
ceiling = after["ceiling"]
where += " AND p.id<=?"
params.append(ceiling)
# One SELECT counts the filtered parents and their active variants.
total, variant_total = conn.execute(
"SELECT COUNT(*),COALESCE(SUM((SELECT COUNT(*) FROM prompt_pairs v "
"WHERE v.positive_prompt_id=p.id AND v.archived=0)),0) FROM "
+ _FROM
+ " WHERE "
+ where,
params,
).fetchone()
expression, direction = _ORDERS[sort]
if after is not None:
operator = ">" if direction == "ASC" else "<"
where += (
f" AND ({expression}{operator}? OR "
f"({expression}=? AND p.id{operator}?))"
)
params.extend([after["value"], after["value"], after["id"]])
rows = self._library_parents(
conn,
epoch,
where,
[*params, limit + 1],
f" ORDER BY {expression} {direction},p.id {direction} LIMIT ?",
sort_expression=expression,
)
has_more = len(rows) > limit
rows = rows[:limit]
next_key = None
if has_more:
next_key = {
"ceiling": ceiling,
"value": rows[-1]["_sort"],
"id": rows[-1]["id"],
}
for row in rows:
row.pop("_sort")
return {
"library_epoch": epoch,
"items": rows,
"total": total,
"variant_total": variant_total,
"has_more": has_more,
"next_key": next_key,
}
def list_library(
self,
*,
text: str = "",
category: str | None = None,
tags: list[str] | None = None,
sort: str = "newest",
limit: int = 40,
after: dict | None = None,
) -> dict:
with self.library_snapshot(include_ceiling=True) as (conn, epoch, ceiling):
return self._list_library(
conn,
epoch,
ceiling,
text=text,
category=category,
tags=tags,
sort=sort,
limit=limit,
after=after,
)
def _is_library_positive(self, conn: sqlite3.Connection, positive_id: int) -> bool:
return (
conn.execute(
"SELECT 1 FROM prompts p WHERE p.id=? AND " + _ELIGIBLE, (positive_id,)
).fetchone()
is not None
)
def _get_library_positive(
self,
conn: sqlite3.Connection,
epoch: str,
positive_id: int,
*,
variant_limit: int = 40,
variant_after: int | None = None,
archive: str = "active",
) -> dict | None:
_limit(variant_limit)
scope = _archive(archive)
if variant_after is not None and (
type(variant_after) is not int or variant_after <= 0
):
raise ValueError("Invalid variant cursor")
rows = self._library_parents(
conn, epoch, _ELIGIBLE + " AND p.id=?", [positive_id]
)
if not rows:
return None
positive = rows[0]
positive["archived_variant_count"] = conn.execute(
"SELECT COUNT(*) FROM prompt_pairs "
"WHERE positive_prompt_id=? AND archived=1",
(positive_id,),
).fetchone()[0]
variant_total = {
"active": positive["variant_count"],
"archived": positive["archived_variant_count"],
"all": positive["variant_count"] + positive["archived_variant_count"],
}[archive]
where = "pp.positive_prompt_id=?" + scope
params = [positive_id]
if variant_after is not None:
where += " AND pp.id<?"
params.append(variant_after)
variants = [
dict(row)
for row in conn.execute(
"SELECT pp.id,COALESCE(n.text,'') AS negative_text,"
"pp.category,pp.title,pp.archived,"
"pp.negative_prompt_id IS NULL AS explicit_empty,"
"(SELECT COUNT(DISTINCT image_id) FROM image_prompt_pairs ip "
"WHERE ip.pair_id=pp.id) AS image_count "
"FROM prompt_pairs pp "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE " + where + " ORDER BY pp.id DESC LIMIT ?",
[*params, variant_limit + 1],
)
]
variants_has_more = len(variants) > variant_limit
variants = variants[:variant_limit]
by_id = {row["id"]: row for row in variants}
for row in variants:
row["tags"] = []
row["archived"] = bool(row["archived"])
row["explicit_empty"] = bool(row["explicit_empty"])
if by_id:
marks = ",".join("?" for _ in by_id)
for row in conn.execute(
"SELECT pt.pair_id,t.name FROM prompt_pair_tags pt "
"JOIN tags t ON t.id=pt.tag_id "
f"WHERE pt.pair_id IN ({marks}) ORDER BY t.name",
list(by_id),
):
by_id[row["pair_id"]]["tags"].append(row["name"])
unassigned = conn.execute(
"SELECT COUNT(*) FROM generated_images g WHERE g.prompt_id=? "
"AND NOT EXISTS(SELECT 1 FROM image_prompt_pairs ip "
"WHERE ip.image_id=g.id)",
(positive_id,),
).fetchone()[0]
return {
"library_epoch": epoch,
"positive": positive,
"variants": variants,
"unassigned_image_count": unassigned,
"variant_total": variant_total,
"variants_has_more": variants_has_more,
"variants_next_key": variants[-1]["id"] if variants_has_more else None,
}
def get_library_positive(
self,
positive_id: int,
*,
variant_limit: int = 40,
variant_after: int | None = None,
archive: str = "active",
) -> dict | None:
with self.library_snapshot() as (conn, epoch, _ceiling):
return self._get_library_positive(
conn,
epoch,
positive_id,
variant_limit=variant_limit,
variant_after=variant_after,
archive=archive,
)
def _get_library_options(self, conn: sqlite3.Connection, epoch: str) -> dict:
rows = conn.execute(
"SELECT category FROM (SELECT p.category AS category "
"FROM prompts p WHERE "
+ _ELIGIBLE
+ (" UNION SELECT category FROM prompt_pairs " "WHERE archived=0) ")
+ "WHERE category IS NOT NULL AND TRIM(category)!='' "
"ORDER BY category COLLATE NOCASE,category"
).fetchall()
return {
"library_epoch": epoch,
"categories": [row["category"] for row in rows],
}
def get_library_options(self) -> dict:
with self.library_snapshot() as (conn, epoch, _ceiling):
return self._get_library_options(conn, epoch)
def get_library_previews(self, positive_ids: list[int]) -> dict[int, list[dict]]:
"""Read at most two links per parent in one bounded page query."""
if (
not isinstance(positive_ids, list)
or len(positive_ids) > 200
or any(type(value) is not int or value <= 0 for value in positive_ids)
):
raise ValueError("Invalid preview parent IDs")
if not positive_ids:
return {}
result = {value: [] for value in positive_ids}
marks = ",".join("?" for _ in positive_ids)
rows = self.model.get_connection().execute(
"WITH ranked AS (SELECT id,prompt_id,image_path,ROW_NUMBER() OVER "
"(PARTITION BY prompt_id ORDER BY id DESC) AS position "
f"FROM generated_images WHERE prompt_id IN ({marks})) "
"SELECT id,prompt_id,image_path FROM ranked WHERE position<=2 "
"ORDER BY prompt_id,id DESC",
positive_ids,
)
for row in rows:
result[row["prompt_id"]].append(dict(row))
return result
def _list_library_images(
self,
conn: sqlite3.Connection,
epoch: str,
positive_id: int,
*,
pair_id: int | None = None,
limit: int = 40,
after: int | None = None,
archive: str = "active",
) -> dict:
_limit(limit)
scope = _archive(archive)
if after is not None and (type(after) is not int or after <= 0):
raise ValueError("Invalid image cursor")
if (
pair_id is not None
and not conn.execute(
"SELECT 1 FROM prompt_pairs pp WHERE pp.id=? "
"AND pp.positive_prompt_id=?" + scope,
(pair_id, positive_id),
).fetchone()
):
if conn.execute(
"SELECT 1 FROM prompt_pairs WHERE id=? AND positive_prompt_id=?",
(pair_id, positive_id),
).fetchone():
raise ValueError("Variant is not in the requested archive scope")
raise ValueError("Variant does not belong to this positive prompt")
where, params = "g.prompt_id=?", [positive_id]
if pair_id is not None:
where += (
" AND EXISTS(SELECT 1 FROM image_prompt_pairs ip "
"WHERE ip.image_id=g.id AND ip.pair_id=?)"
)
params.append(pair_id)
total = conn.execute(
"SELECT COUNT(*) FROM generated_images g WHERE " + where, params
).fetchone()[0]
if after is not None:
where += " AND g.id<?"
params.append(after)
rows = conn.execute(
"SELECT g.id,g.image_path FROM generated_images g WHERE "
+ where
+ " ORDER BY g.id DESC LIMIT ?",
[*params, limit + 1],
).fetchall()
return {
"library_epoch": epoch,
"images": [dict(row) for row in rows[:limit]],
"total": total,
"has_more": len(rows) > limit,
"next_key": rows[limit - 1]["id"] if len(rows) > limit else None,
}
def list_library_images(
self,
positive_id: int,
*,
pair_id: int | None = None,
limit: int = 40,
after: int | None = None,
archive: str = "active",
) -> dict:
with self.library_snapshot() as (conn, epoch, _ceiling):
return self._list_library_images(
conn,
epoch,
positive_id,
pair_id=pair_id,
limit=limit,
after=after,
archive=archive,
)
def export_library(self) -> dict:
"""Read every portable owner and pair from one snapshot, writing nothing.
Local IDs order the rows internally and never appear in the document.
"""
with self.library_snapshot() as (conn, _epoch, _ceiling):
owners = {}
order = []
for row in conn.execute(
"SELECT p.id,p.text,ls.title,p.category,p.rating,p.notes FROM "
+ _FROM
+ " WHERE "
+ _ELIGIBLE
+ " ORDER BY p.id"
):
owners[row["id"]] = {
"positive_text": row["text"],
"title": row["title"],
"category": row["category"],
"tags": [],
"rating": row["rating"],
"notes": row["notes"],
"variants": [],
}
order.append(row["id"])
for row in conn.execute(
"SELECT pt.prompt_id,t.name FROM prompt_tags pt "
"JOIN tags t ON t.id=pt.tag_id ORDER BY pt.prompt_id,t.name"
):
if row["prompt_id"] in owners:
owners[row["prompt_id"]]["tags"].append(row["name"])
variants = {}
for row in conn.execute(
"SELECT pp.id,pp.positive_prompt_id,"
"COALESCE(n.text,'') AS negative_text,"
"pp.title,pp.category,pp.rating,pp.notes,pp.archived "
"FROM prompt_pairs pp "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"ORDER BY pp.id"
):
variant = {
"negative_text": row["negative_text"],
"title": row["title"],
"category": row["category"],
"tags": [],
"rating": row["rating"],
"notes": row["notes"],
"archived": bool(row["archived"]),
}
variants[row["id"]] = variant
owners[row["positive_prompt_id"]]["variants"].append(variant)
for row in conn.execute(
"SELECT pt.pair_id,t.name FROM prompt_pair_tags pt "
"JOIN tags t ON t.id=pt.tag_id ORDER BY pt.pair_id,t.name"
):
variants[row["pair_id"]]["tags"].append(row["name"])
return {
"schema_version": 1,
"kind": "prompt-manager-library",
"positives": [owners[key] for key in order],
}
+310
View File
@@ -0,0 +1,310 @@
"""Transactional native Library commands guarded by epoch and edit token."""
import re
import sqlite3
from .library import _ELIGIBLE
from .models import utc_now_iso
try:
from ..utils.validators import (
validate_category,
validate_prompt_text,
validate_tags,
)
except ImportError:
from utils.validators import validate_category, validate_prompt_text, validate_tags
METADATA_FIELDS = frozenset({"title", "category", "tags", "rating", "notes"})
MAX_TEXT_FIELD = 10000
_UUID = re.compile(r"[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}")
_TOKEN = re.compile(r"[0-9a-f]{64}")
# Parent metadata stored on the prompt row; title lives in prompt_library_state.
_PROMPT_COLUMNS = ("category", "rating", "notes")
class LibraryCommandError(Exception):
"""A native command lost a race against storage or another writer.
``code`` is a wire code (``metadata-conflict``, ``storage-changed`` or
``vocabulary-conflict`` for a tag or category that changed since it was read),
``epoch`` the storage epoch read inside the transaction and ``positive``
the current owner record when it may be disclosed.
"""
def __init__(self, code: str, epoch: str, positive: dict | None = None):
super().__init__(code)
self.code = code
self.epoch = epoch
self.positive = positive
def validate_library_metadata(metadata: dict, *, require_title: bool = False) -> None:
"""Reject anything but a nonempty allowlisted parent metadata object."""
if not isinstance(metadata, dict) or not metadata:
raise ValueError("metadata must be a nonempty object")
unknown = set(metadata) - METADATA_FIELDS
if unknown:
raise ValueError("Unknown metadata fields: " + ", ".join(sorted(unknown)))
if require_title and (
not isinstance(metadata.get("title"), str) or not metadata["title"].strip()
):
raise ValueError("title is required")
for name in ("title", "notes"):
value = metadata.get(name)
if value is not None and not isinstance(value, str):
raise ValueError(f"{name} must be a string or null")
if value is not None and len(value) > MAX_TEXT_FIELD:
raise ValueError(f"{name} is too long")
if metadata.get("category") is not None:
validate_category(metadata["category"])
if "tags" in metadata:
if not isinstance(metadata["tags"], list):
raise ValueError("tags must be a list of strings")
validate_tags(metadata["tags"])
rating = metadata.get("rating")
if rating is not None and (type(rating) is not int or not 1 <= rating <= 5):
raise ValueError("rating must be an integer between 1 and 5 or null")
def normalize_library_metadata(metadata: dict) -> dict:
"""Return validated metadata with category trimmed and blank stored as NULL."""
if "category" not in metadata:
return metadata
category = metadata["category"]
return {**metadata, "category": (category or "").strip() or None}
def normalize_library_tags(tags: list[str]) -> list[str]:
"""Trim and deduplicate validated tags, keeping first-seen order."""
return list(dict.fromkeys(tag.strip() for tag in tags))
def _validate_epoch(expected_epoch: str) -> None:
if not isinstance(expected_epoch, str) or not _UUID.fullmatch(expected_epoch):
raise ValueError("expected_epoch must be a canonical UUID")
def _validate_guards(positive_id: int, expected_epoch: str, if_match: str) -> None:
if type(positive_id) is not int or not 1 <= positive_id <= 2**63 - 1:
raise ValueError("Invalid positive prompt ID")
_validate_epoch(expected_epoch)
if not isinstance(if_match, str) or not _TOKEN.fullmatch(if_match):
raise ValueError("if_match must be 64 lowercase hexadecimal characters")
def _validate_negative_text(negative_text: str) -> None:
"""Require an explicit string; blank deliberately means the empty negative."""
if not isinstance(negative_text, str):
raise ValueError("negative_text must be explicit; use an empty string for none")
if negative_text.strip():
validate_prompt_text(negative_text)
class LibraryCommandOperations:
"""Connection-level parent commands mixed into PromptDatabase."""
def _library_epoch(self, conn: sqlite3.Connection, expected_epoch: str) -> str:
row = conn.execute(
"SELECT storage_epoch FROM execution_history_metadata WHERE singleton_id=1"
).fetchone()
if row is None:
raise sqlite3.DatabaseError("Missing execution history metadata")
if row[0] != expected_epoch:
raise LibraryCommandError("storage-changed", row[0])
return row[0]
def _library_parent(
self, conn: sqlite3.Connection, epoch: str, positive_id: int
) -> dict | None:
rows = self._library_parents(
conn, epoch, _ELIGIBLE + " AND p.id=?", [positive_id]
)
return rows[0] if rows else None
def _ensure_library_parent(self, conn: sqlite3.Connection, prompt_id: int) -> None:
"""Register explicit parent intent once; an existing row is never touched."""
if conn.execute(
"SELECT 1 FROM prompt_library_state WHERE prompt_id=?", (prompt_id,)
).fetchone():
return
conn.execute(
"INSERT INTO prompt_library_state(prompt_id,title) VALUES(?,NULL)",
(prompt_id,),
)
def _apply_library_metadata(
self, conn: sqlite3.Connection, current: dict, metadata: dict
) -> None:
"""Write only fields that differ; stamp ``prompts.updated_at`` once."""
prompt_id = current["id"]
changed = False
if "title" in metadata and metadata["title"] != current["title"]:
conn.execute(
"UPDATE prompt_library_state SET title=? WHERE prompt_id=?",
(metadata["title"], prompt_id),
)
changed = True
columns = {
name: metadata[name]
for name in _PROMPT_COLUMNS
if name in metadata and metadata[name] != current[name]
}
if "tags" in metadata:
tags = normalize_library_tags(metadata["tags"])
if set(tags) != set(current["tags"]):
self._sync_prompt_tags(conn, prompt_id, tags)
changed = True
if columns or changed:
columns["updated_at"] = utc_now_iso()
conn.execute(
"UPDATE prompts SET "
+ ",".join(f"{name}=?" for name in columns)
+ " WHERE id=?",
[*columns.values(), prompt_id],
)
def update_library_parent(
self,
positive_id: int,
*,
expected_epoch: str,
if_match: str,
metadata: dict,
) -> dict | None:
"""Apply explicit parent metadata if the epoch and edit token still match.
Returns ``{library_epoch, positive}`` read before commit, or None when
the positive is missing or not a Library parent. Raises
LibraryCommandError for a rotated epoch or a stale token.
"""
_validate_guards(positive_id, expected_epoch, if_match)
validate_library_metadata(metadata)
metadata = normalize_library_metadata(metadata)
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
current = self._library_parent(conn, epoch, positive_id)
if current is None:
return None
if current["edit_token"] != if_match:
raise LibraryCommandError("metadata-conflict", epoch, current)
self._ensure_library_parent(conn, positive_id)
self._apply_library_metadata(conn, current, metadata)
positive = self._library_parent(conn, epoch, positive_id)
if positive is None:
raise sqlite3.DatabaseError("Library parent vanished during update")
return {"library_epoch": epoch, "positive": positive}
def _recipe_negative(
self, conn: sqlite3.Connection, negative_text: str
) -> int | None:
if not negative_text.strip():
return None
return self._pair_component_outcome(conn, negative_text)[0]
def _recipe_receipt(
self,
conn: sqlite3.Connection,
epoch: str,
positive_id: int,
pair_id: int,
*,
positive_created: bool,
pair_created: bool,
) -> dict:
"""Read both owners inside the write transaction for the receipt."""
positive = self._library_parent(conn, epoch, positive_id)
pairs = self._read_pair_rows(conn, "pp.id = ?", [pair_id])
if positive is None or not pairs:
raise sqlite3.DatabaseError("Library recipe vanished during save")
return {
"library_epoch": epoch,
"positive": positive,
"pair": pairs[0],
"positive_created": positive_created,
"pair_created": pair_created,
# Parent metadata is applied only to a newly inserted positive.
"parent_metadata_applied": positive_created,
}
def create_library_recipe(
self,
positive_text: str,
negative_text: str,
*,
expected_epoch: str,
parent_metadata: dict,
) -> dict:
"""Create or reuse a positive and its ordered pair in one transaction.
Submitted parent metadata applies only when the positive component is
newly inserted; a reused component keeps its own metadata. New pairs get
empty pair metadata. Reused pairs keep theirs, including archive state.
Every success registers the positive as a Library parent.
"""
_validate_epoch(expected_epoch)
validate_prompt_text(positive_text)
_validate_negative_text(negative_text)
validate_library_metadata(parent_metadata, require_title=True)
parent_metadata = normalize_library_metadata(parent_metadata)
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
positive_id, positive_created = self._pair_component_outcome(
conn, positive_text
)
negative_id = self._recipe_negative(conn, negative_text)
pair_id, pair_created = self._save_pair_components(
conn, positive_id, negative_id
)
self._ensure_library_parent(conn, positive_id)
if positive_created:
current = self._library_parent(conn, epoch, positive_id)
if current is None:
raise sqlite3.DatabaseError("New Library parent is not eligible")
self._apply_library_metadata(conn, current, parent_metadata)
return self._recipe_receipt(
conn,
epoch,
positive_id,
pair_id,
positive_created=positive_created,
pair_created=pair_created,
)
def add_library_negative(
self,
positive_id: int,
negative_text: str,
*,
expected_epoch: str,
if_match: str,
) -> dict | None:
"""Create or reuse a pair under the exact stored positive.
The stored positive ID and text are used as they are; only the negative
is resolved. Returns None when the positive is missing or not a Library
parent, and raises LibraryCommandError for a rotated epoch or a stale
token, in both cases before any insertion.
"""
_validate_guards(positive_id, expected_epoch, if_match)
_validate_negative_text(negative_text)
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
current = self._library_parent(conn, epoch, positive_id)
if current is None:
return None
if current["edit_token"] != if_match:
raise LibraryCommandError("metadata-conflict", epoch, current)
negative_id = self._recipe_negative(conn, negative_text)
pair_id, pair_created = self._save_pair_components(
conn, positive_id, negative_id
)
self._ensure_library_parent(conn, positive_id)
return self._recipe_receipt(
conn,
epoch,
positive_id,
pair_id,
positive_created=False,
pair_created=pair_created,
)
+86
View File
@@ -0,0 +1,86 @@
"""Optional-old, required-current schema for Library parent state."""
import re
import sqlite3
TABLE = "prompt_library_state"
MAX_TITLE_LENGTH = 10_000
_DDL = (
f"CREATE TABLE {TABLE} (prompt_id INTEGER PRIMARY KEY, title TEXT, "
"FOREIGN KEY(prompt_id) REFERENCES prompts(id) ON DELETE CASCADE)"
)
_COLUMNS = (("prompt_id", "INTEGER", 0, 1), ("title", "TEXT", 0, 0))
_FOREIGN_KEYS = {("prompt_id", "prompts", "id", "CASCADE")}
def _compact(sql: str) -> str:
return re.sub(r"\s+", "", sql).lower()
def library_schema_error(
conn: sqlite3.Connection, *, allow_missing: bool = True
) -> str | None:
"""Reject unsupported Library state schemas and invalid persisted values.
Reads only: nothing is ever repaired. A missing table is acceptable for
backups that predate it unless ``allow_missing`` is false.
"""
objects = conn.execute(
"SELECT type,sql FROM sqlite_master WHERE name=? "
"OR (type IN ('index','trigger') AND tbl_name=?)",
(TABLE, TABLE),
).fetchall()
if not objects:
return None if allow_missing else "Missing Library state schema"
tables = [sql for kind, sql in objects if kind == "table"]
if len(tables) != 1 or len(objects) != 1:
return "Library state name is reserved by an unsupported schema object"
if _compact(tables[0] or "") != _compact(_DDL):
return "Library state has an invalid table definition"
columns = tuple(
(row[1], row[2].upper(), row[3], row[5])
for row in conn.execute(f"PRAGMA table_info({TABLE})")
)
if columns != _COLUMNS:
return "Library state has invalid columns"
foreign_keys = {
(row[3], row[2], row[4], row[6])
for row in conn.execute(f"PRAGMA foreign_key_list({TABLE})")
}
if foreign_keys != _FOREIGN_KEYS:
return "Library state has invalid foreign key definitions"
if conn.execute(f"PRAGMA foreign_key_check({TABLE})").fetchone():
return "Library state contains foreign key violations"
# length() stops at an embedded NUL, so bound by bytes in SQL (characters
# never outnumber bytes, and UTF-8 uses at most 4 bytes per character) and
# count the few remaining candidates as decoded characters.
if conn.execute(
f"SELECT 1 FROM {TABLE} WHERE typeof(title) NOT IN ('null','text') "
f"OR length(CAST(title AS BLOB)) > {MAX_TITLE_LENGTH * 4} LIMIT 1"
).fetchone():
return "Library state contains invalid titles"
for (raw,) in conn.execute(
f"SELECT CAST(title AS BLOB) FROM {TABLE} "
f"WHERE length(CAST(title AS BLOB)) > {MAX_TITLE_LENGTH}"
):
if len(bytes(raw).decode("utf-8", "replace")) > MAX_TITLE_LENGTH:
return "Library state contains invalid titles"
return None
def migrate_library_state(conn: sqlite3.Connection) -> None:
"""Install the table atomically; validate before and after creating it."""
conn.execute("SAVEPOINT prompt_library_state_schema")
try:
error = library_schema_error(conn)
if error:
raise sqlite3.DatabaseError(error)
conn.execute(_DDL.replace("CREATE TABLE", "CREATE TABLE IF NOT EXISTS", 1))
error = library_schema_error(conn, allow_missing=False)
if error:
raise sqlite3.DatabaseError(error)
conn.execute("RELEASE prompt_library_state_schema")
except BaseException:
conn.execute("ROLLBACK TO prompt_library_state_schema")
conn.execute("RELEASE prompt_library_state_schema")
raise
+37
View File
@@ -0,0 +1,37 @@
"""Narrow comparison tokens for a Library parent's own metadata projection."""
import hashlib
import json
_DOMAIN = "prompt-manager-library-parent:v1\n"
def canonical_token(domain: str, preimage: dict) -> str:
"""SHA-256 of ``domain`` plus the canonical JSON of ``preimage``."""
payload = json.dumps(
preimage,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
allow_nan=False,
)
return hashlib.sha256((domain + payload).encode("utf-8")).hexdigest()
def library_edit_token(epoch: str, positive: dict) -> str:
"""Hash the parent's owner fields under the current storage epoch.
Timestamps, counts, pairs and previews are excluded, and an absent state
row projects the same null title as a default one.
"""
preimage = {
"epoch": epoch,
"id": positive["id"],
"text": positive["text"],
"title": positive["title"],
"category": positive["category"],
"notes": positive["notes"],
"rating": positive["rating"],
"tags": sorted(set(positive["tags"])),
}
return canonical_token(_DOMAIN, preimage)
+854 -196
View File
File diff suppressed because it is too large Load Diff
+700 -204
View File
File diff suppressed because it is too large Load Diff
+136
View File
@@ -0,0 +1,136 @@
"""Atomic metadata enrichment for a single known generation context."""
import json
import os
from typing import Any, Dict
from .models import normalize_image_path
from .operations import _coerce_file_info
try:
from ..utils.hashing import generate_prompt_hash
except ImportError:
from utils.hashing import generate_prompt_hash
def ingest_resolved_pair(
db: Any, image_path: str, metadata: Dict[str, Any], resolution: Dict[str, Any]
) -> Dict[str, Any]:
state = resolution.get("state", "unresolved")
if state != "resolved":
return {"status": state, "pairs_added": 0, "links_added": 0}
positive, negative = resolution["positive"], resolution["negative"]
positive_text = positive.get("storage_text", positive["text"])
negative_text = negative.get("storage_text", negative["text"])
normalized = normalize_image_path(image_path)
provenance = {"resolver_version": 1, "positive": positive, "negative": negative}
encoded = json.dumps(
provenance, sort_keys=True, separators=(",", ":"), allow_nan=False
)
with db._pair_transaction() as conn:
images = conn.execute(
"SELECT gi.*,p.hash AS positive_hash FROM generated_images gi "
"JOIN prompts p ON p.id=gi.prompt_id "
"WHERE gi.file_path=? OR gi.image_path=?",
(normalized, image_path),
).fetchall()
if len(images) > 1 or (
images and images[0]["positive_hash"] != generate_prompt_hash(positive_text)
):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
existing = images[0] if images else None
if existing:
prior = conn.execute(
"SELECT n.hash AS negative_hash FROM image_prompt_pairs ip "
"JOIN prompt_pairs pp ON pp.id=ip.pair_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE ip.image_id=?",
(existing["id"],),
).fetchall()
expected = (
generate_prompt_hash(negative_text) if negative_text.strip() else None
)
if any(row["negative_hash"] != expected for row in prior):
return {"status": "conflict", "pairs_added": 0, "links_added": 0}
prior_pair = conn.execute(
"SELECT pp.id FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id "
"WHERE p.hash=? AND n.hash IS ?",
(
generate_prompt_hash(positive_text),
generate_prompt_hash(negative_text) if negative_text.strip() else None,
),
).fetchone()
pair_id = db._save_pair(conn, positive_text, negative_text)
pair = conn.execute(
"SELECT * FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone()
pairs_added = int(prior_pair is None)
file_size, width, height, image_format = _coerce_file_info(metadata)
snapshots = {
column: json.dumps(metadata[key], sort_keys=True, allow_nan=False)
for column, key in (
("workflow_data", "workflow"),
("prompt_metadata", "prompt"),
("parameters", "parameters"),
)
if key in metadata
}
info = {
"file_size": file_size,
"width": width,
"height": height,
"format": image_format,
}
snapshots.update(
{key: value for key, value in info.items() if value is not None}
)
changed = False
if existing:
image_id = existing["id"]
updates = {
key: value for key, value in snapshots.items() if existing[key] != value
}
if updates:
conn.execute(
"UPDATE generated_images SET "
+ ",".join(f"{key}=?" for key in updates)
+ " WHERE id=?",
[*updates.values(), image_id],
)
changed = True
else:
columns = ["prompt_id", "image_path", "filename", "file_path", *snapshots]
values = [
pair["positive_prompt_id"],
image_path,
os.path.basename(image_path),
normalized,
*snapshots.values(),
]
image_id = conn.execute(
"INSERT INTO generated_images("
+ ",".join(columns)
+ ") VALUES("
+ ",".join("?" for _ in columns)
+ ")",
values,
).lastrowid
linked = conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? "
"AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()
db._link_image_to_pair(conn, pair_id, image_id, provenance)
status = (
"added"
if existing is None
else "enriched" if changed or not linked else "unchanged"
)
return {
"status": status,
"pairs_added": pairs_added,
"links_added": int(not linked),
"pair_id": pair_id,
"image_id": image_id,
}
+433
View File
@@ -0,0 +1,433 @@
"""Ordered reusable prompt pairs and their generation provenance."""
import json
import sqlite3
from contextlib import contextmanager
from typing import Any, Dict, Iterator, List, Optional
from .models import pair_provenance_error, utc_now_iso
try:
from ..utils.hashing import generate_prompt_hash
except ImportError:
from utils.hashing import generate_prompt_hash
class PairDependencyError(ValueError):
"""An operation would change or remove content used by a saved pair."""
def _validate_metadata(
title: Optional[str],
category: Optional[str],
tags: Optional[List[str]],
rating: Optional[int],
notes: Optional[str],
) -> Optional[List[str]]:
for name, value in (("title", title), ("category", category), ("notes", notes)):
if value is not None and not isinstance(value, str):
raise ValueError(f"{name} must be a string")
if rating is not None and (type(rating) is not int or not 1 <= rating <= 5):
raise ValueError("Rating must be between 1 and 5")
if tags is None:
return None
if not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags):
raise ValueError("tags must be a list of strings")
return list(dict.fromkeys(tag.strip() for tag in tags if tag.strip()))
class PairOperations:
"""Connection-level pair operations mixed into PromptDatabase."""
@contextmanager
def _pair_transaction(self) -> Iterator[sqlite3.Connection]:
conn = self.model.get_connection()
if conn.in_transaction:
raise RuntimeError("Pair operations require their own transaction")
with conn:
conn.execute("BEGIN IMMEDIATE")
yield conn
def _ensure_prompts_unpaired(
self, conn: sqlite3.Connection, prompt_ids: List[int]
) -> None:
if not prompt_ids:
return
placeholders = ",".join("?" for _ in prompt_ids)
row = conn.execute(
f"SELECT id FROM prompt_pairs WHERE positive_prompt_id IN ({placeholders}) "
f"OR negative_prompt_id IN ({placeholders}) LIMIT 1",
[*prompt_ids, *prompt_ids],
).fetchone()
if row:
raise PairDependencyError(
f"Prompt belongs to saved pair {row['id']}; delete the pair first "
"or save changed text as a new prompt"
)
def _pair_component(self, conn: sqlite3.Connection, text: str) -> int:
return self._pair_component_outcome(conn, text)[0]
def _pair_component_outcome(
self, conn: sqlite3.Connection, text: str
) -> tuple[int, bool]:
"""Resolve a component by its dedup hash; report whether it was inserted.
The first stored spelling wins: the hash ignores outer whitespace and
case, internal whitespace stays significant. ``created`` comes from the
INSERT result under the caller's held write transaction.
"""
prompt_hash = generate_prompt_hash(text)
now = utc_now_iso()
created = (
conn.execute(
"INSERT INTO prompts(text, hash, created_at, updated_at) "
"VALUES(?,?,?,?) ON CONFLICT(hash) DO NOTHING",
(text.strip(), prompt_hash, now, now),
).rowcount
== 1
)
row = conn.execute(
"SELECT id FROM prompts WHERE hash = ?", (prompt_hash,)
).fetchone()
return row["id"], created
def _sync_pair_tags(
self, conn: sqlite3.Connection, pair_id: int, tags: List[str]
) -> None:
conn.execute("DELETE FROM prompt_pair_tags WHERE pair_id = ?", (pair_id,))
for tag_id in self._ensure_tags(conn, tags).values():
conn.execute(
"INSERT INTO prompt_pair_tags(pair_id,tag_id) VALUES(?,?)",
(pair_id, tag_id),
)
def save_prompt_pair(
self,
positive_text: str,
negative_text: str = "",
*,
title: Optional[str] = None,
category: Optional[str] = None,
tags: Optional[List[str]] = None,
rating: Optional[int] = None,
notes: Optional[str] = None,
) -> int:
"""Save an ordered pair atomically; repeat saves preserve its metadata."""
with self._pair_transaction() as conn:
return self._save_pair(
conn,
positive_text,
negative_text,
title=title,
category=category,
tags=tags,
rating=rating,
notes=notes,
)
def _save_pair(
self,
conn: sqlite3.Connection,
positive_text: str,
negative_text: str = "",
*,
title: Optional[str] = None,
category: Optional[str] = None,
tags: Optional[List[str]] = None,
rating: Optional[int] = None,
notes: Optional[str] = None,
) -> int:
"""Save an ordered pair atomically; repeat saves preserve its metadata."""
if not isinstance(positive_text, str) or not positive_text.strip():
raise ValueError("Positive prompt text cannot be empty")
if not isinstance(negative_text, str):
raise ValueError("Negative prompt text must be a string; use '' for empty")
# Validate metadata before any component insertion.
_validate_metadata(title, category, tags, rating, notes)
positive = self._pair_component(conn, positive_text)
negative = (
self._pair_component(conn, negative_text) if negative_text.strip() else None
)
return self._save_pair_components(
conn,
positive,
negative,
title=title,
category=category,
tags=tags,
rating=rating,
notes=notes,
)[0]
def _save_pair_components(
self,
conn: sqlite3.Connection,
positive_id: int,
negative_id: Optional[int],
*,
title: Optional[str] = None,
category: Optional[str] = None,
tags: Optional[List[str]] = None,
rating: Optional[int] = None,
notes: Optional[str] = None,
) -> tuple[int, bool]:
"""Create or reuse the ordered pair of existing component IDs.
Returns ``(pair_id, created)``. A reused pair keeps all of its metadata
and archive state; metadata applies only to a newly inserted pair.
"""
tags = _validate_metadata(title, category, tags, rating, notes)
existing = conn.execute(
"SELECT id FROM prompt_pairs WHERE positive_prompt_id = ? "
"AND negative_prompt_id IS ?",
(positive_id, negative_id),
).fetchone()
if existing:
return existing["id"], False
now = utc_now_iso()
pair_id = conn.execute(
"INSERT INTO prompt_pairs(positive_prompt_id,negative_prompt_id,title,"
"category,rating,notes,created_at,updated_at) VALUES(?,?,?,?,?,?,?,?)",
(positive_id, negative_id, title, category, rating, notes, now, now),
).lastrowid
if tags:
self._sync_pair_tags(conn, pair_id, tags)
return pair_id, True
def _read_pair_rows(
self, conn: sqlite3.Connection, where: str, params: List[Any], suffix: str = ""
) -> List[Dict[str, Any]]:
rows = conn.execute(
"SELECT pp.*, p.text AS positive_text, "
"COALESCE(n.text,'') AS negative_text,"
"(SELECT COUNT(DISTINCT image_id) FROM image_prompt_pairs ip "
"WHERE ip.pair_id=pp.id) AS image_count FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE " + where + suffix,
params,
).fetchall()
pairs = [dict(row) for row in rows]
if not pairs:
return pairs
by_id = {pair["id"]: pair for pair in pairs}
for pair in pairs:
pair["archived"] = bool(pair["archived"])
pair["tags"] = []
placeholders = ",".join("?" for _ in by_id)
for row in conn.execute(
"SELECT pt.pair_id,t.name FROM prompt_pair_tags pt JOIN tags t ON "
f"t.id=pt.tag_id WHERE pt.pair_id IN ({placeholders}) ORDER BY t.name",
list(by_id),
):
by_id[row["pair_id"]]["tags"].append(row["name"])
return pairs
def get_prompt_pair(self, pair_id: int) -> Optional[Dict[str, Any]]:
"""Return components, pair metadata and image count, including archives."""
rows = self._read_pair_rows(self.model.get_connection(), "pp.id = ?", [pair_id])
return rows[0] if rows else None
def list_prompt_pairs(
self,
*,
text: Optional[str] = None,
category: Optional[str] = None,
tags: Optional[List[str]] = None,
include_archived: bool = False,
limit: int = 50,
offset: int = 0,
sort: str = "newest",
) -> Dict[str, Any]:
"""List pairs with all-tag filtering and stable paginated ordering."""
orders = {
"newest": "pp.id DESC",
"last_used_desc": "julianday(pp.last_used_at) DESC, pp.id DESC",
}
if sort not in orders:
raise ValueError("Unknown pair sort order")
if type(limit) is not int or not 1 <= limit <= 500:
raise ValueError("limit must be between 1 and 500")
if type(offset) is not int or offset < 0:
raise ValueError("offset must be a nonnegative integer")
tags = _validate_metadata(None, category, tags, None, None)
if text is not None and not isinstance(text, str):
raise ValueError("text must be a string")
clauses, params = ["1=1"], []
if not include_archived:
clauses.append("pp.archived=0")
if category is not None:
clauses.append("pp.category = ?")
params.append(category)
if text:
clauses.append("(p.text LIKE ? OR n.text LIKE ? OR pp.title LIKE ?)")
params.extend([f"%{text}%"] * 3)
for tag in tags or []:
clauses.append(
"EXISTS(SELECT 1 FROM prompt_pair_tags pt JOIN tags t "
"ON t.id=pt.tag_id WHERE pt.pair_id=pp.id AND t.name=?)"
)
params.append(tag)
where = " AND ".join(clauses)
conn = self.model.get_connection()
total = conn.execute(
"SELECT COUNT(*) FROM prompt_pairs pp "
"JOIN prompts p ON p.id=pp.positive_prompt_id "
"LEFT JOIN prompts n ON n.id=pp.negative_prompt_id WHERE " + where,
params,
).fetchone()[0]
pairs = self._read_pair_rows(
conn,
where,
[*params, limit, offset],
f" ORDER BY {orders[sort]} LIMIT ? OFFSET ?",
)
return {
"pairs": pairs,
"total": total,
"limit": limit,
"offset": offset,
"has_more": offset + len(pairs) < total,
}
def update_prompt_pair_metadata(
self,
pair_id: int,
*,
title: Optional[str] = None,
category: Optional[str] = None,
tags: Optional[List[str]] = None,
rating: Optional[int] = None,
notes: Optional[str] = None,
) -> bool:
"""Update pair metadata explicitly; None preserves and empty values clear."""
tags = _validate_metadata(title, category, tags, rating, notes)
changes = {
key: value
for key, value in (
("title", title),
("category", category),
("rating", rating),
("notes", notes),
)
if value is not None
}
if not changes and tags is None:
return False
with self._pair_transaction() as conn:
if not conn.execute(
"SELECT 1 FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone():
return False
changes["updated_at"] = utc_now_iso()
conn.execute(
"UPDATE prompt_pairs SET "
+ ",".join(f"{key}=?" for key in changes)
+ " WHERE id=?",
[*changes.values(), pair_id],
)
if tags is not None:
self._sync_pair_tags(conn, pair_id, tags)
return True
def archive_prompt_pair(self, pair_id: int, archived: bool = True) -> bool:
"""Hide a reusable pair without changing components or generation history."""
if type(archived) is not bool:
raise ValueError("archived must be a boolean")
with self._pair_transaction() as conn:
return (
conn.execute(
"UPDATE prompt_pairs SET archived=?,updated_at=? WHERE id=?",
(int(archived), utc_now_iso(), pair_id),
).rowcount
> 0
)
def delete_prompt_pair(self, pair_id: int) -> bool:
"""Delete unused pair metadata; generated pairs must be archived instead."""
with self._pair_transaction() as conn:
if conn.execute(
"SELECT 1 FROM image_prompt_pairs WHERE pair_id=? LIMIT 1", (pair_id,)
).fetchone():
raise PairDependencyError(
"Pair has generated images; archive it instead"
)
return (
conn.execute("DELETE FROM prompt_pairs WHERE id=?", (pair_id,)).rowcount
> 0
)
def record_prompt_pair_use(self, pair_id: int, times: int = 1) -> bool:
"""Count pair uses independently of standalone prompt usage."""
if type(times) is not int or times < 1:
raise ValueError("times must be a positive integer")
with self._pair_transaction() as conn:
return (
conn.execute(
"UPDATE prompt_pairs SET run_count=run_count+?,last_used_at=? "
"WHERE id=?",
(times, utc_now_iso(), pair_id),
).rowcount
> 0
)
def link_image_to_pair(
self, pair_id: int, image_id: int, provenance: Dict[str, Any]
) -> int:
"""Associate an image with immutable run provenance, idempotently."""
with self._pair_transaction() as conn:
return self._link_image_to_pair(conn, pair_id, image_id, provenance)
def _link_image_to_pair(
self,
conn: sqlite3.Connection,
pair_id: int,
image_id: int,
provenance: Dict[str, Any],
) -> int:
"""Associate an image with immutable run provenance, idempotently."""
if not isinstance(provenance, dict):
raise ValueError("provenance must be an object")
try:
encoded = json.dumps(
provenance, sort_keys=True, separators=(",", ":"), allow_nan=False
)
except (TypeError, ValueError, RecursionError) as error:
raise ValueError("provenance must contain JSON values") from error
error = pair_provenance_error(encoded)
if error:
raise ValueError(error)
pair = conn.execute(
"SELECT positive_prompt_id FROM prompt_pairs WHERE id=?", (pair_id,)
).fetchone()
image = conn.execute(
"SELECT prompt_id FROM generated_images WHERE id=?", (image_id,)
).fetchone()
if not pair or not image:
raise ValueError("Pair and image must exist")
if pair["positive_prompt_id"] != image["prompt_id"]:
raise ValueError("Image must belong to the pair's positive prompt")
conn.execute(
"INSERT INTO image_prompt_pairs(pair_id,image_id,provenance,created_at) "
"VALUES(?,?,?,?) ON CONFLICT(pair_id,image_id,provenance) DO NOTHING",
(pair_id, image_id, encoded, utc_now_iso()),
)
return conn.execute(
"SELECT id FROM image_prompt_pairs "
"WHERE pair_id=? AND image_id=? AND provenance=?",
(pair_id, image_id, encoded),
).fetchone()["id"]
def get_image_prompt_pairs(self, image_id: int) -> List[Dict[str, Any]]:
"""Return every recorded pair association with decoded provenance."""
rows = (
self.model.get_connection()
.execute(
"SELECT * FROM image_prompt_pairs WHERE image_id=? ORDER BY id",
(image_id,),
)
.fetchall()
)
return [
{**dict(row), "provenance": json.loads(row["provenance"])} for row in rows
]
+435
View File
@@ -0,0 +1,435 @@
"""Bounded tag and category reads, and guarded tag commands.
A count is the number of eligible Library parents that carry the value
directly or through one active saved pair, which is exactly what
``list_library(tags=[name])`` / ``list_library(category=value)`` return.
Tag commands run in one ``_pair_transaction``: the storage epoch is checked
first, then the exact ``{id, name}`` identity, and the receipt is read on the
same connection before commit. Owner rows and their timestamps never change.
Category consolidation is previewed in one ``library_snapshot`` and applied in
one ``_pair_transaction`` that recomputes the owner projection and requires the
preview's ``impact_token``; only rows carrying the exact source change.
"""
import sqlite3
from .library import _ELIGIBLE, _limit
from .library_commands import _TOKEN, LibraryCommandError, _validate_epoch
from .library_tokens import canonical_token
from .models import utc_now_iso
try:
from ..utils.validators import validate_category, validate_tags
except ImportError:
from utils.validators import validate_category, validate_tags
_CATEGORY_DOMAIN = "prompt-manager-vocabulary-category:v1\n"
def _tag_rows(page: str) -> str:
"""Count rows for the registry rows selected by ``page`` (id, name)."""
# Eligible parent components carrying a tag in the page, plus the
# positive parent of every active pair carrying it (always eligible).
return (
"WITH page AS (" + page + "),"
"owners(tag_id,pid) AS ("
"SELECT pt.tag_id,p.id FROM prompts p "
"JOIN prompt_tags pt ON pt.prompt_id=p.id "
"WHERE pt.tag_id IN (SELECT id FROM page) AND " + _ELIGIBLE + " "
"UNION SELECT ppt.tag_id,pp.positive_prompt_id FROM prompt_pairs pp "
"JOIN prompt_pair_tags ppt ON ppt.pair_id=pp.id "
"WHERE pp.archived=0 AND ppt.tag_id IN (SELECT id FROM page)) "
"SELECT page.id,page.name,COUNT(DISTINCT owners.pid) AS prompt_count "
"FROM page LEFT JOIN owners ON owners.tag_id=page.id "
"GROUP BY page.id ORDER BY page.id"
)
_TAG_PAGE = _tag_rows("SELECT id,name FROM tags WHERE id>? ORDER BY id LIMIT ?")
_TAG_ONE = _tag_rows("SELECT id,name FROM tags WHERE id=?")
# Every owner kind a tag can sit on, for the delete confirmation.
_TAG_IMPACT = (
"SELECT "
"(SELECT COUNT(*) FROM prompts p JOIN prompt_tags pt ON pt.prompt_id=p.id "
"WHERE pt.tag_id=:tag AND " + _ELIGIBLE + ") AS positive_prompts,"
"(SELECT COUNT(*) FROM prompts p JOIN prompt_tags pt ON pt.prompt_id=p.id "
"WHERE pt.tag_id=:tag AND NOT " + _ELIGIBLE + ") AS other_prompts,"
"(SELECT COUNT(*) FROM prompt_pairs pp JOIN prompt_pair_tags ppt "
"ON ppt.pair_id=pp.id WHERE ppt.tag_id=:tag AND pp.archived=0) "
"AS negative_variants,"
"(SELECT COUNT(*) FROM prompt_pairs pp JOIN prompt_pair_tags ppt "
"ON ppt.pair_id=pp.id WHERE ppt.tag_id=:tag AND pp.archived!=0) "
"AS archived_negative_variants"
)
_MAX_ID = 2**63 - 1
def validate_vocabulary_tag_name(value: object) -> str:
"""Return the trimmed name, applying tag rules plus the native comma ban."""
if not isinstance(value, str):
raise ValueError("Tag name must be a string")
validate_tags([value])
name = value.strip()
if "," in name:
raise ValueError("Tag names cannot contain commas")
return name
def _validate_tag_id(tag_id: object) -> None:
if type(tag_id) is not int or not 1 <= tag_id <= _MAX_ID:
raise ValueError("Invalid tag ID")
# The /library/options universe: nonblank values on eligible parents or on
# active pairs, grouped exactly (whitespace and case variants stay distinct).
_CATEGORY_GROUPS = (
"WITH owners(value,pid) AS ("
"SELECT p.category,p.id FROM prompts p WHERE p.category IS NOT NULL "
"AND TRIM(p.category)!='' AND " + _ELIGIBLE + " "
"UNION SELECT pp.category,pp.positive_prompt_id FROM prompt_pairs pp "
"WHERE pp.archived=0 AND pp.category IS NOT NULL AND TRIM(pp.category)!=''),"
"grouped AS (SELECT value,MIN(pid) AS first_pid,"
"COUNT(DISTINCT pid) AS prompt_count FROM owners GROUP BY value) "
)
# Every owner of one exact category value, in id order, with the flag the
# impact counts split on.
_CATEGORY_COMPONENTS = (
"SELECT p.id," + _ELIGIBLE + " AS eligible FROM prompts p "
"WHERE p.category=? ORDER BY p.id"
)
_CATEGORY_PAIRS = "SELECT id,archived FROM prompt_pairs WHERE category=? ORDER BY id"
_CATEGORY_EXISTS = (
"SELECT EXISTS(SELECT 1 FROM prompts WHERE category=:v)"
" OR EXISTS(SELECT 1 FROM prompt_pairs WHERE category=:v)"
)
# Pairs after components, so a failing pair update rolls both back.
_CATEGORY_UPDATES = (
"UPDATE prompts SET category=?,updated_at=? WHERE category=?",
"UPDATE prompt_pairs SET category=?,updated_at=? WHERE category=?",
)
def category_impact_token(
epoch: str,
source: str,
target: str,
target_exists: bool,
components: list,
pairs: list,
) -> str:
"""Hash a category projection like ``library_edit_token`` (no secret)."""
preimage = {
"epoch": epoch,
"source": source,
"target": target,
"target_exists": bool(target_exists),
"components": sorted([int(i), bool(flag)] for i, flag in components),
"pairs": sorted([int(i), bool(flag)] for i, flag in pairs),
}
return canonical_token(_CATEGORY_DOMAIN, preimage)
def _validate_category_change(source: object, target: object) -> str:
"""Check the exact source and return the trimmed, different target."""
if not isinstance(source, str) or not source.strip():
raise ValueError("source must be a nonblank category value")
if not isinstance(target, str):
raise ValueError("Category must be a string")
validate_category(target)
target = target.strip()
if not target:
raise ValueError("New category name cannot be blank")
if target == source:
raise ValueError("New category name must differ from the current one")
return target
def _validate_impact_token(impact_token: object) -> None:
if not isinstance(impact_token, str) or not _TOKEN.fullmatch(impact_token):
raise ValueError("impact_token must be 64 lowercase hexadecimal characters")
def _tag_after(after: object) -> int:
if after is None:
return 0
if type(after) is not int or after <= 0:
raise ValueError("Invalid vocabulary cursor")
return after
def _category_after(after: object) -> tuple[int, str] | None:
if after is None:
return None
if (
not isinstance(after, list)
or len(after) != 2
or type(after[0]) is not int
or not isinstance(after[1], str)
):
raise ValueError("Invalid vocabulary cursor")
return after[0], after[1]
class VocabularyOperations:
"""Vocabulary reads and tag commands.
``_`` helpers take an open ``library_snapshot`` or ``_pair_transaction``
connection.
"""
def _vocabulary_tag(self, conn: sqlite3.Connection, tag_id: int) -> dict | None:
row = conn.execute(_TAG_ONE, (tag_id,)).fetchone()
return dict(row) if row else None
def _vocabulary_identity(
self, conn: sqlite3.Connection, epoch: str, tag_id: int, name: str
) -> dict:
"""The stored tag, or vocabulary-conflict when it is gone or renamed."""
tag = self._vocabulary_tag(conn, tag_id)
if tag is None or tag["name"] != name:
raise LibraryCommandError("vocabulary-conflict", epoch)
return tag
@staticmethod
def _tag_owner_counts(conn: sqlite3.Connection, tag_id: int) -> dict:
components = conn.execute(
"SELECT COUNT(DISTINCT prompt_id) FROM prompt_tags WHERE tag_id=?",
(tag_id,),
).fetchone()[0]
pairs = conn.execute(
"SELECT COUNT(DISTINCT pair_id) FROM prompt_pair_tags WHERE tag_id=?",
(tag_id,),
).fetchone()[0]
return {"components_changed": components, "pairs_changed": pairs}
def create_vocabulary_tag(self, name: str, *, expected_epoch: str) -> dict:
"""Insert an unused registry row, or reuse the exact existing name."""
_validate_epoch(expected_epoch)
name = validate_vocabulary_tag_name(name)
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
row = conn.execute("SELECT id FROM tags WHERE name=?", (name,)).fetchone()
created = row is None
if created:
tag_id = conn.execute(
"INSERT INTO tags(name) VALUES(?)", (name,)
).lastrowid
else:
tag_id = row[0]
return {
"library_epoch": epoch,
"created": created,
"tag": self._vocabulary_tag(conn, tag_id),
}
def rename_vocabulary_tag(
self, tag_id: int, *, expected_epoch: str, name: str, new_name: str
) -> dict:
"""Rename ``{tag_id, name}``, merging into an existing exact ``new_name``.
Links move across components and all pairs, archived included; an
owner already carrying the target keeps one link.
"""
_validate_tag_id(tag_id)
_validate_epoch(expected_epoch)
if not isinstance(name, str):
raise ValueError("name must be the tag's current name")
new_name = validate_vocabulary_tag_name(new_name)
if new_name == name:
raise ValueError("New name must differ from the current name")
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
self._vocabulary_identity(conn, epoch, tag_id, name)
counts = self._tag_owner_counts(conn, tag_id)
target = conn.execute(
"SELECT id FROM tags WHERE name=?", (new_name,)
).fetchone()
if target is None:
conn.execute("UPDATE tags SET name=? WHERE id=?", (new_name, tag_id))
result_id = tag_id
else:
result_id = target[0]
self._move_tag_links(conn, tag_id, result_id)
conn.execute("DELETE FROM tags WHERE id=?", (tag_id,))
return {
"library_epoch": epoch,
"merged": target is not None,
"tag": self._vocabulary_tag(conn, result_id),
**counts,
}
def delete_vocabulary_tag(
self, tag_id: int, *, expected_epoch: str, name: str
) -> dict:
"""Remove the registry row ``{tag_id, name}``; its links cascade."""
_validate_tag_id(tag_id)
_validate_epoch(expected_epoch)
if not isinstance(name, str):
raise ValueError("name must be the tag's current name")
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
self._vocabulary_identity(conn, epoch, tag_id, name)
counts = self._tag_owner_counts(conn, tag_id)
conn.execute("DELETE FROM tags WHERE id=?", (tag_id,))
return {"library_epoch": epoch, "deleted": True, **counts}
def vocabulary_tag_impact(self, tag_id: int) -> dict | None:
"""Owner counts by kind for one tag, or None when it does not exist."""
_validate_tag_id(tag_id)
with self.library_snapshot() as (conn, epoch, _ceiling):
if not conn.execute("SELECT 1 FROM tags WHERE id=?", (tag_id,)).fetchone():
return None
row = conn.execute(_TAG_IMPACT, {"tag": tag_id}).fetchone()
return {"library_epoch": epoch, "impact": dict(row)}
@staticmethod
def _category_projection(
conn: sqlite3.Connection, epoch: str, source: str, target: str
) -> dict:
"""Owners of ``source``, impact counts and the token over them."""
components = [
[row[0], bool(row[1])]
for row in conn.execute(_CATEGORY_COMPONENTS, (source,))
]
pairs = [
[row[0], bool(row[1])] for row in conn.execute(_CATEGORY_PAIRS, (source,))
]
if not components and not pairs:
raise LibraryCommandError("vocabulary-conflict", epoch)
target_exists = bool(
conn.execute(_CATEGORY_EXISTS, {"v": target}).fetchone()[0]
)
eligible = sum(flag for _id, flag in components)
archived = sum(flag for _id, flag in pairs)
return {
"target_exists": target_exists,
"impact": {
"positive_prompts": eligible,
"other_prompts": len(components) - eligible,
"negative_variants": len(pairs) - archived,
"archived_negative_variants": archived,
},
"impact_token": category_impact_token(
epoch, source, target, target_exists, components, pairs
),
}
def preview_vocabulary_category(
self, *, expected_epoch: str, source: str, target: str
) -> dict:
"""Read-only impact of renaming the exact ``source`` to ``target``."""
_validate_epoch(expected_epoch)
target = _validate_category_change(source, target)
with self.library_snapshot() as (conn, epoch, _ceiling):
if epoch != expected_epoch:
raise LibraryCommandError("storage-changed", epoch)
projection = self._category_projection(conn, epoch, source, target)
return {
"library_epoch": epoch,
"source": source,
"target": target,
**projection,
}
def apply_vocabulary_category(
self, *, expected_epoch: str, source: str, target: str, impact_token: str
) -> dict:
"""Set the exact ``source`` to ``target`` on every component and pair.
The recomputed projection must still match the preview's token; a
mismatch is ``vocabulary-conflict`` with nothing written.
"""
_validate_epoch(expected_epoch)
target = _validate_category_change(source, target)
_validate_impact_token(impact_token)
with self._pair_transaction() as conn:
epoch = self._library_epoch(conn, expected_epoch)
projection = self._category_projection(conn, epoch, source, target)
if projection["impact_token"] != impact_token:
raise LibraryCommandError("vocabulary-conflict", epoch)
stamp = utc_now_iso()
for statement in _CATEGORY_UPDATES:
conn.execute(statement, (target, stamp, source))
return {
"library_epoch": epoch,
"source": source,
"target": target,
"consolidated": projection["target_exists"],
"impact": projection["impact"],
}
def _vocabulary_tags(
self,
conn: sqlite3.Connection,
epoch: str,
*,
limit: int = 100,
after: int | None = None,
) -> dict:
_limit(limit)
start = _tag_after(after)
total = conn.execute("SELECT COUNT(*) FROM tags").fetchone()[0]
rows = [dict(row) for row in conn.execute(_TAG_PAGE, (start, limit + 1))]
has_more = len(rows) > limit
rows = rows[:limit]
return {
"library_epoch": epoch,
"tags": rows,
"total": total,
"has_more": has_more,
"next_key": rows[-1]["id"] if has_more else None,
}
def _vocabulary_categories(
self,
conn: sqlite3.Connection,
epoch: str,
*,
limit: int = 100,
after: list | None = None,
) -> dict:
_limit(limit)
key = _category_after(after)
total = conn.execute(
_CATEGORY_GROUPS + "SELECT COUNT(*) FROM grouped"
).fetchone()[0]
where, params = "", []
if key is not None:
where = " WHERE first_pid>? OR (first_pid=? AND value>?)"
params = [key[0], key[0], key[1]]
rows = conn.execute(
_CATEGORY_GROUPS
+ "SELECT value,first_pid,prompt_count FROM grouped"
+ where
+ " ORDER BY first_pid,value LIMIT ?",
[*params, limit + 1],
).fetchall()
has_more = len(rows) > limit
rows = rows[:limit]
return {
"library_epoch": epoch,
"categories": [
{"value": row["value"], "prompt_count": row["prompt_count"]}
for row in rows
],
"total": total,
"has_more": has_more,
"next_key": (
[rows[-1]["first_pid"], rows[-1]["value"]] if has_more else None
),
}
def list_vocabulary_tags(self, *, limit: int = 100, after: int | None = None):
with self.library_snapshot() as (conn, epoch, _ceiling):
return self._vocabulary_tags(conn, epoch, limit=limit, after=after)
def list_vocabulary_categories(
self, *, limit: int = 100, after: list | None = None
) -> dict:
with self.library_snapshot() as (conn, epoch, _ceiling):
return self._vocabulary_categories(conn, epoch, limit=limit, after=after)
+84
View File
@@ -0,0 +1,84 @@
# Application Redesign Development Plan
The maintainer approved the [interactive concept](concepts/workspace/README.md)
on 2026-10-17. It is a design reference with simulated operations, not production
code. This plan proposes implementation order and acceptance gates.
## Delivery status
- Foundation merged in [#198](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/198).
- [#200](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/200) delivers native Images browsing, physical-file details, infinite scroll, Grid/List/Masonry, Viewer.js and video playback. The workspace opens on Images; original application entry points remain available.
- Shared role-aware sending and the physical-file prompt inspector merged in [#203](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/203) and [#206](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/206).
- The native Prompt Library merged in [#210](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/210) for [#208](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/208): one positive parent, saved negative variants, full selected text and a shared filtered image gallery. It does not invent chronological runs from counters or scan dates.
- [Durable history storage](prompt-history-storage.md) records job attempts and outcomes from [#212](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/212), frozen prompt contexts from [#213](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/213), and observed run-owned output locators from [#214](https://github.com/ComfyAssets/ComfyUI_PromptManager/pull/214). Those locators do not verify file bytes or establish gallery-image associations. That association contract and the chronological execution timeline remain pending under [#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209).
- [#215](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/215) brings the shell, Images and natural-height Prompt Library presentation into the approved geometry and themes while retaining their existing operations. [#216](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/216) adds the shared current-file inspector with exact generation facts, scoped provenance, raw/download evidence and loaded-collection viewer controls, including a slideshow that pauses at native video. This delivery does not establish whole-application parity, change the default entry point or constitute a release. Recorded chronological run/context browsing continues under [#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209); image tagging/review, System consolidation and remaining organization capabilities continue under #196.
## Scope and compatibility
Rebuild the standalone application around one shared shell. Preserve existing
node schemas, saved workflows, backend capabilities, pinned Viewer.js and the
ComfyAssets toolkit. Keep existing application entry points usable during the
transition. Do not change the running maintainer instance during automated tests.
Use the concept's feature inventory as a parity checklist. Each row needs an
implementation issue, observable acceptance criteria and automated test coverage.
Record any intentional behavior change explicitly; no silent feature removal.
## Proposed delivery sequence
Use a tracking issue and small issue-linked branches and PRs. Dependencies may
require splitting these milestones further.
| Milestone | Deliverable and acceptance focus |
| --- | --- |
| 1. Foundation | Production shell, routing, shared theme and disposable page-test harness. Verify startup, navigation, dark/light changes, cleanup and existing entry points. Establish discovered-test and coverage baselines before changes. |
| 2. Image workspace | Real API-backed filters, Grid/List/Masonry, readable thumbnails, selection and shared Viewer.js inspector. Continuous infinite scroll only, with accessible Load more fallback, stable ordering, retry and restored browsing context. Bound retained DOM and test large datasets. |
| 3. Prompt library and history | Real positive/negative associations, selected full negative, run history and shared image filtering. Design any missing execution-history persistence and migration before wiring the timeline; never invent historical runs from incomplete metadata. |
| 4. Send to ComfyUI | Shared paper-plane actions wherever prompt text appears; role-specific and pair sends, destination preview, acknowledgement and failure recovery. Cover explicit empty versus unknown negatives and connected inputs. Never queue or replace a workflow. |
| 5. Editing and review | Prompt editing, vocabulary management, per-image manual/auto-tag approval and Needs review. Preserve cancellation semantics and distinguish tag approval from prompt-association decisions. |
| 6. System and maintenance | Settings, sources/scans, integrations, jobs, diagnostics, backup/restore and maintenance impact previews. Cover existing capabilities before retiring their old controls. |
| 7. Additional organization | Favorites independent of ratings, collections, saved views, Activity, scheduled backups and portable import previews. Specify persistence, validation and recovery separately for each capability. |
| 8. Cutover | Complete the parity checklist, migration and upgrade validation, accessibility/performance checks, documentation and targeted maintainer live testing. Switch the default only after these pass; preserve compatible URLs. |
## TDD and test layers
For each behavior, first write and run a meaningful failing test. Implement the
smallest working change, verify it passes, then refactor with the suite green.
Include failing-before/passing-after evidence in the PR.
- **Unit:** filtering and selection state, cursor handling, prompt roles,
formatting, validation, navigation and lifecycle cleanup.
- **Integration:** real HTTP handlers and disposable SQLite/filesystem fixtures;
persistence, migration from supported old schemas, idempotent scans, restore,
rollback and reference-safe deletion. Mock inference and external services.
- **Smoke:** shipped production pages start and navigate with real fixture APIs;
test populated, empty and error states, assets and unexpected console failures.
- **Browser workflows:** production HTML/JS across Chromium, Firefox, WebKit and
touch; editing/reload, infinite append, viewer cleanup, tagging approval,
keyboard access, delayed responses, cancellation and retries.
- **Host integration:** pinned disposable CPU ComfyUI, Classic and Nodes 2.0,
switching both ways and workflow reload for host-facing changes. Exercise
independent and combined pack installations for shared bridge/toolkit changes.
Retain existing test discovery and coverage gates. Carry forward database
injection and open-handle checks into new fixtures. Add new frontend modules to
coverage measurement and its baseline; do not lower floors or replace existing
regression coverage with mockup-only tests.
## GitHub and release gates
Extend existing required workflows: the Python 3.10–3.13 matrix on Linux,
Windows and macOS; 85% Python coverage; formatting; frontend units and per-file
coverage floors; vendor checks; browser workflows; real ComfyUI compatibility.
Publish traces, screenshots, logs and coverage artifacts on failures. See
[TESTING.md](../TESTING.md) for commands and fixture isolation requirements.
Every PR must map acceptance criteria to tests, report relevant checks and include
screenshots for visual changes. All required checks must pass before merging.
Use maintainer live review at meaningful visual and host-integration milestones;
capture discovered regressions in automated tests. Notify the maintainer when
Python changes require a ComfyUI restart.
Keep version/publication changes separate from intermediate UI work: the existing
registry workflow publishes on main/master changes to `pyproject.toml` or manual
dispatch. Do not trigger a release merely to preview a redesign milestone.
File diff suppressed because one or more lines are too long
+138
View File
@@ -0,0 +1,138 @@
# Prompt Manager application redesign concept
Status: visual concept approved by the maintainer, 2026-10-17. Production
implementation is progressing in tested slices; see the [development plan](../../application-redesign.md).
The production workspace now has native Images browsing; this concept remains the
approved target for the complete application.
## Review the mockup
Run `python tools/preview-prompt-timeline.py --bind 0.0.0.0 --port 8767` and
open `/workspace/`. The original approved timeline remains available at `/`.
The server exposes an explicit allowlist of concept files and pinned assets.
All examples are synthetic; there are no application API calls, database access,
real model downloads, or ComfyUI handoffs. Local file inspection previews an
image in the browser without uploading it. Exports contain sample data only.
The sidebar opens Images, Prompt library, Tags & categories, Inspect a file,
Activity, System and Design review. System includes Overview, Appearance,
Sources & scans, Integrations, Auto-tagging, Maintenance, Backup & export, and
Diagnostics & logs. The in-app Design review contains the feature map below.
## Proposed direction
One application shell replaces separate Admin, Gallery and Metadata shells.
Use the approved shared ComfyAssets theme: ComfyUI dark/light, blue primary
actions, green positive roles, red negative roles and errors. Preserve existing
capabilities and services while consolidating controls and clarifying their scope.
Browsing is image-led with readable thumbnails and one metadata inspector.
Prompt library groups by positive; negative variants and runs share its gallery.
The full selected negative appears directly beneath the positive in history.
Favorites express preference independently of the existing quality rating.
Collections organize projects without duplicating recipe text; saved views keep
search/filter context. New capabilities are proposals, not implemented features.
Maintenance moves out of the browsing toolbar. Checks, impact preview, backups,
and apply steps have separate responsibilities. Distinguish prompt text reuse
from image-file duplicates, and metadata enrichment from thumbnail generation.
No repeated global thumbnail-generation prompt on opening the gallery.
## Send to ComfyUI
Use the same original paper-plane outline, blue accent and accessible role label
on every positive/negative prompt block, including image details and the viewer.
A pair action sends both. Preview destination and exact fields before applying;
positive-only and negative-only actions leave the other role untouched. An
explicitly empty negative means clear the target field; unknown metadata must
never be interpreted as empty. Sending text neither queues execution nor
replaces a workflow.
Production needs a shared host bridge with versioned payloads, origin validation,
destination discovery, capability checks, disconnected-host handling, linked-input
checks and acknowledgements. Preview actual target values and report success only
after acknowledgement. Default to asking for a destination; never guess between
multiple nodes or positive and negative encoder inputs. Verify Classic and Nodes
2.0 with both packs installed independently and together.
## Existing feature inventory
| Current feature | Proposed home / behavior | Disposition |
| --- | --- | --- |
| Admin prompt cards / per-prompt gallery | Prompt library + one prompt history workspace | Retained, rebuilt |
| Standalone image gallery / Admin image modal | One Images workspace + shared Viewer.js viewer | Consolidated |
| Video playback, autoplay / mute / loop, viewer slideshow and transforms | Shared media viewer; playback controls in Appearance; same image navigation | Retained |
| Metadata page / viewer sidebar / full-text popups | One reusable metadata inspector; inline complete prompt text | Consolidated |
| Prompt add, edit, rating, notes, copy, delete | Recipe editor, shared details, contextual bulk actions; reference-safe archive | Retained, clarified |
| Categories, tag rename / merge, autocomplete | Tags & categories; same vocabulary in every editor | Retained |
| Grid/list, thumbnail size, columns, pages/infinite scroll | Infinite scroll only; one Appearance configuration and contextual layout / density overrides | Simplified |
| Scan, rescan, monitoring directories, worker threads | Sources & scans; explicit metadata enrichment vs thumbnail work | Consolidated |
| Ollama, HuggingFace, WD14 variants, thresholds, review | One Auto-tagging configuration and review flow | Retained |
| Thumbnail quality, size limit, metadata cache, paths | Appearance + cache controls in Maintenance | Retained |
| Prompt dedup, hash checks, consistency, unused prompts | Maintenance plan with dependency checks and impact preview | Retained, safer flow |
| Image dedup, orphan image links, browser cache | Separate reviewed operations in Maintenance | Retained |
| Database backup / restore, JSON export | Backup & export; recovery and portable recipes clearly distinguished | Retained |
| Diagnostics, image-link checks, logs, rotation, log levels | System → Diagnostics & logs | Retained |
| LoRA detection/import, trigger injection, CivitAI key | System → Integrations; positive-only triggers | Retained |
| Statistics / counts | System overview + usage breakdown; context counts in the workspace | Retained |
| Send prompts to ComfyUI from every prompt surface | Role-aware paper-plane action + destination / overwrite preview | New proposal |
| Reusable negatives / approved branch timeline | Prompt history with selected negative directly beneath positive | Approved concept |
| Favorites, saved searches, project collections | Sidebar views; collections do not duplicate recipe text | New proposal |
| Ambiguous metadata review, source provenance, run comparisons | Needs review queue; recorded vs inferred evidence; pair/run selection | New / expanded |
| Unified jobs, cancellation, resumable progress visibility | Activity center; cancellation distinguished from hiding progress | New proposal |
| Scheduled backups, pre-change safety snapshots | Explicit recovery policy after storage configuration | New proposal |
## Implementation boundaries and test requirements
Keep Viewer.js 1.15.2 and the shared toolkit as reproducible pinned artifacts;
this mock uses their actual checked-in files without modifying them. Preserve
legacy routes during the transition, current node layouts/schemas, existing
controllers where reusable, and all existing test/coverage gates.
The timeline needs persisted execution records to represent real runs. Saved
views, collections, metadata review resolution, activity history, scheduled
backups, portable JSON import, and the ComfyUI bridge need separate implementation
and persistence work. Preserve source evidence and distinguish recorded from
inferred metadata. Existing schema migration is automatic; old image enrichment
remains an optional explicit scan. The one-time upgrade notice belongs in the
new shell, with Scan and Later choices.
Implement in reviewed slices: shared application shell and inspector; reusable
prompt blocks/send bridge; image workspace; grouped library/history; settings and
maintenance; optional organization and workflow improvements. Write regressions
first for production behavior. Retain Python, frontend unit, Chromium/Firefox/
WebKit/touch and managed Classic/Nodes 2.0 tests. Add behavior tests for shared
filters, navigation state, viewer cleanup, interrupted jobs, dependency-safe
maintenance, backup/restore, paired metadata and correct send destinations.
Screenshots in `screenshots/` record the reviewed starting point. The approved
standalone timeline in `../prompt-variant-timeline.html` remains unchanged.
## Review polish
History actions use a shared branch icon. Images can switch among Grid, List and
Masonry; Masonry preserves each image's proportions and keeps DOM order while
packing cards with a responsive layout. It does not require experimental native
masonry support. Resize observers are disconnected when leaving the view.
Every image card includes a tagging action beside Info, including cards in Needs
review. Choose WD14 SwinV2, WD14 ViT, Ollama or HuggingFace; adjust thresholds or
instructions, generate sample suggestions, edit tags manually, and approve only
the desired suggestions. Changes are staged until approval, apply to that image
alone and do not resolve uncertain prompt associations. Changing models discards
previous suggestions. Cancel preserves existing tags. All inference is simulated.
## Continuous browsing decision
The redesign uses infinite scroll only. Remove page numbers, previous/next page
buttons, paged/infinite mode selection and results-per-page controls. Keep request
batching internal. The mock appends 12 images at a time in Grid, List and Masonry;
it preserves existing card nodes, focus and selection while appending. It also
retains the loaded range when switching layouts or revisiting a search. A
keyboard-accessible Load more action provides an alternate trigger for the same
continuous stream, not pagination. Exhausted and empty results are explicit.
Production will need bounded rendering, stable cursors, scroll restoration and
loading/error/retry behavior for large collections. Existing backend pagination
can continue providing internal request batches; no production tests or routes
are removed as part of this prototype.
File diff suppressed because it is too large Load Diff
+105
View File
@@ -0,0 +1,105 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta name="referrer" content="no-referrer" />
<title>Images · Prompt Manager concept</title>
<link rel="stylesheet" href="/toolkit/controls.css" />
<link rel="stylesheet" href="/vendor/viewer.css" />
<link rel="stylesheet" href="/workspace/style.css" />
<script src="/vendor/viewer.js" defer></script>
<script src="/workspace/app.js" type="module"></script>
</head>
<body class="ca-tools">
<a class="skip" href="#main">Skip to workspace</a>
<div class="app-shell">
<aside class="sidebar">
<a class="brand" href="#images"
><span class="brand-mark">P<span>±</span></span
><span>Prompt Manager<small>COMFYASSETS</small></span></a
>
<div class="sidebar-label">WORKSPACE</div>
<nav id="primary-nav" aria-label="Workspace"></nav>
<div class="sidebar-label">
SAVED VIEWS <span class="new-label">PROPOSED</span>
</div>
<nav id="saved-nav" aria-label="Saved views"></nav>
<div class="sidebar-label">
COLLECTIONS <span class="new-label">PROPOSED</span>
</div>
<nav id="collections-nav" aria-label="Collections"></nav>
<div class="sidebar-bottom">
<nav id="system-nav" aria-label="Tools and system"></nav>
<div class="connection">
<i></i>
<div>
ComfyUI workspace<small
>Demo connection · Classic / Nodes 2.0</small
>
</div>
</div>
</div>
</aside>
<div class="workspace">
<header class="topbar">
<div class="breadcrumbs">
ComfyAssets <span>/</span> <span id="breadcrumb">Images</span>
</div>
<div class="top-actions">
<span class="concept-badge">Interactive concept</span
><button id="activity-open" type="button">
Activity <span class="count" id="activity-count">2</span></button
><label class="theme-label" for="theme"
>Theme
<select id="theme" aria-label="Theme">
<option value="dark">Dark</option>
<option value="light">Light</option>
<option value="system">System</option>
</select></label
><a class="canvas-link" href="#send"
>ComfyUI <span aria-hidden="true">↗</span></a
>
</div>
</header>
<main id="main" tabindex="-1"></main>
<footer class="app-footer">
<span
>Design prototype · sample images and data · actions stay in this
tab</span
><button type="button" id="review-link">What changes & why</button>
</footer>
</div>
</div>
<div id="toast" class="toast" role="status" aria-atomic="true"></div>
<dialog id="modal" aria-labelledby="modal-title">
<div class="dialog-header">
<h2 id="modal-title"></h2>
<button type="button" id="modal-close" aria-label="Close dialog">
×
</button>
</div>
<div id="modal-body"></div>
</dialog>
<dialog id="viewer-dialog" aria-labelledby="viewer-title">
<div class="dialog-header">
<div>
<h2 id="viewer-title">Image details</h2>
<p class="muted small">
One viewer · image, prompts, generation data, workflow
</p>
</div>
<button type="button" id="viewer-close" aria-label="Close image viewer">
×
</button>
</div>
<div class="viewer-layout">
<div class="viewer-left">
<div id="viewer-stage"></div>
<div class="viewer-toolbar" id="viewer-toolbar"></div>
</div>
<aside id="viewer-details" aria-label="Image metadata"></aside>
</div>
</dialog>
</body>
</html>
Binary file not shown.

After

Width:  |  Height:  |  Size: 243 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 365 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 441 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 440 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 443 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 478 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 176 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 156 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 319 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 150 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 128 KiB

File diff suppressed because it is too large Load Diff
+84
View File
@@ -0,0 +1,84 @@
# Durable Prompt History
Status: the durable job ledger, immutable prompt contexts and run-owned output
references are implemented for
[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Gallery
image associations and the chronological timeline remain pending; #209 stays open.
## Recorded jobs
New jobs observed at the ComfyUI queue's dequeue boundary receive an immutable
run identity and a host-session/queue-item identity. The public ComfyUI job UUID
is a correlation field: submitting it twice records two attempts. Rejected graphs
and jobs removed from the queue before dequeue create no execution rows. Cached
jobs are recorded even when no PromptManager node runs, and headless jobs do not
require a browser or WebSocket client.
The additive `execution_runs` table stores dequeue time, observed callback time,
optional executor event times and an explicit outcome: `observed`, `succeeded`,
`failed`, `interrupted` or `unknown`. Matching stored terminal events determine
outcomes. Missing or conflicting terminal evidence remains unknown; a process
crash can leave an observed row without an invented completion time. Bounded
safe diagnostics retain node ID/type and exception type, never graph text,
arbitrary extra data, credentials, exception messages, traceback, inputs or
arbitrary output payloads. A bounded background writer keeps storage failures from
changing host execution and reports capture gaps rather than inventing rows.
Existing databases begin with zero execution rows. Migration and backup restore
preserve prompts, pairs, image links and aggregate counters; storage epochs keep
stale queued writes from being replayed into restored data. There is no
historical backfill from counters, scan dates, image timestamps or saved recipes.
[Testing](../TESTING.md#durable-execution-ledger) describes the real CPU host
matrix, historical failure proof, privacy checks and cleanup. Python deployment
requires restarting ComfyUI.
## What the existing UI proves
`prompts.run_count` and `prompt_pairs.run_count` remain aggregate usage counters,
not completed-run counts. `last_used_at` is the latest aggregate update; legacy
migration may have estimated counts from linked images. `generated_images`
identifies linked files; image counts and insertion times are not run counts or
execution times. Image/pair association creation time can reflect an old-image
scan. A saved pair's creation time is its recipe date.
The native Prompt Library shows saved negative variants, linked images and
aggregate uses. It does not expose the ledger as chronological dots or repeat-run
branches. `PromptTracker` identities and PNG recipe evidence remain separate
from durable host job identity.
## Immutable contexts and output references
Prompt contexts retain bounded raw/effective positive, negative and composition
snapshots with nullable recipe navigation hints and explicit evidence provenance.
Unknown text remains SQL NULL; a known empty negative remains empty text. Cached
evidence belongs to its exact cache provider and attempt, rather than inheriting
unrelated recent prompts. Deletion and restore can remove navigation hints without
rewriting historical snapshots or rebinding them to current library rows.
Schema v3 adds `execution_output_capture` and `execution_run_outputs` without
backfilling older runs or modifying gallery image links. At final queue completion,
the writer attaches bounded output evidence from that attempt's final host history
to its immutable run ID. The internal `get_execution_outputs(run_id)` reader
returns state, reason, format version, source and ordered references. A complete
capture with `none_reported` means the final host history reported no images;
`not_captured`, partial, unavailable and conflicting evidence remain explicit.
Failed and interrupted jobs can still have complete evidence for images reported
before termination. Running jobs do not gain a final capture merely because an
image node finished.
Each reference records output node ID, image ordinal, folder type, subfolder,
filename, path style and an opaque configured-root key. A cached repeat can retain
the same locators under a new run. Locators do not verify file bytes, continued
existence, a newly generated file or membership in any prompt context: the reader
reports file identity as `unverified`, generation and context membership as
`unknown`, and temp outputs as transient. Capture never reads, resolves or serves
the files and adds no public route. PNG metadata is not required. Same-path file
replacement and existing gallery provenance retain their separate semantics.
## Remaining history work
Define a deduplicated run/gallery-image relationship that preserves existing image
provenance and same-path replacement protection. Add a chronological timeline only
after its navigation and evidence semantics can honestly represent prompt-level
execution chronology, batched outputs and repeat-run branches.
+34
View File
@@ -0,0 +1,34 @@
# Prompt inspection and sending
The opt-in workspace at `/prompt_manager/workspace#/images` shows full positive
and negative prompts in both Image details and the full-image preview. Existing
gallery and metadata tools remain available during the application rebuild.
Each prompt shows its evidence source: recorded execution, workflow inference,
or embedded generation parameters. Path-only saved-library associations are
marked unverified and cannot be sent: a filename alone cannot prove that an image
has not been replaced. Missing, ambiguous, and unreadable prompts remain visible
as distinct states. An explicitly
empty negative is a resolved value and can clear a destination's negative field.
Use the blue paper-plane action beside a prompt, or **Send prompt pair**. Keep a
ComfyUI tab open under the same origin and deployment prefix. Choose a window and
compatible Prompt Manager node, select **Preview changes**, review the complete
before/after text and preserved prepend/append composition, then **Apply to
ComfyUI**. The dialog reports success only after a matching acknowledgement.
It does not load workflows or queue generation. If confirmation is lost, inspect
ComfyUI and preview again before another attempt. Closing the dialog cannot undo
an update already sent.
## Read-only API
`GET /prompt_manager/images/inspect/{relative_path}?root={root_index}` requires
an explicit configured root, including when two roots contain the same filename.
The response contains `image` identity and `prompts.positive` / `prompts.negative`.
Each role has `status` (`resolved`, `absent`, `unresolved`, or `ambiguous`), `text`
(string only when resolved, otherwise null), and `source`. Inspection does not
index files, alter saved prompts, run scans, or decode image pixels. PNG text and
video probe output are bounded; unsupported or malformed evidence remains
unresolved rather than becoming sendable placeholder text.
Python route changes require restarting ComfyUI. Browser assets require a refresh.
+49
View File
@@ -0,0 +1,49 @@
# Prompt sending bridge
Issue #202 establishes the transport and host adapter before the redesigned
inspector displays prompt text. Gallery paper-plane controls and normalized
metadata follow in a separate slice; existing node layouts are unchanged.
## Shared transport
PromptManager consumes the exact Toolkit 0.2.0 artifact recorded in
`web/js/shared/toolkit/provenance.json`. The shared module has no ComfyUI imports.
Its client discovers explicit host sessions and compatible destinations, requests
a preview, and applies that preview using a one-use, client-bound, expiring ticket.
Success requires an acknowledgement with verified destination values.
BroadcastChannel connects independently opened tabs on the same origin and storage
partition. Deployment prefixes have separate channel namespaces. Exact origin,
protocol and peer checks reject unrelated messages. Same-origin scripts are
trusted; this transport is not authentication against other scripts on that origin.
Unsupported messaging is reported without breaking existing ComfyUI nodes.
An apply timeout or abort after posting means the outcome is unknown. It cannot
prove the host cancelled the write. The sender must inspect a fresh destination
preview before deciding to send again; automatic retry must not overwrite edits.
## PromptManager adapter
Classic, Text, Focused and Split accept positive text; +/- accepts either role or
both. Plain CLIP encoders are not assigned roles by title or position. An omitted
role remains untouched; an explicitly empty string clears only the selected
field. Unknown/ambiguous metadata must not become an empty string.
Previews show original and requested text plus positive prepend/append composition.
Connected composition is marked unknown rather than guessing upstream values.
Connected destination fields reject writes. Applying rechecks graph, node, widgets,
values and connections. Ordinary serialization cache synchronization is accepted
only when it preserves live values. Both requested roles are checked before writing.
Widget values, visible inputs, serialization and composed previews are synchronized.
Callback failures restore local fields; arbitrary external callback side effects
cannot be undone. Sending never queues execution or loads/replaces a workflow.
## Lifecycle and validation
Page hiding closes the host and invalidates tickets; restored pages mount one fresh
host. Client disposal clears pending timers/listeners and closes its channel.
Shared protocol contracts run against the actual bundled file, alongside adapter
units, cross-browser tab workflows and real Classic/Nodes 2.0 scenarios. Pinned
PromptManager-only, KikoTools-only and combined disposable hosts verify coexistence
and workflow reload without modifying either live installation.
+111
View File
@@ -0,0 +1,111 @@
# Reusable positive and negative prompts
Tracking: [#193](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/193).
This work builds on the merged Split node in PR #192. Keep the feature unmerged
until maintainer live testing is complete.
## Storage contract
Keep existing prompt IDs, text hashes, metadata, image links and node schemas.
A component can serve either role. Ordered combinations refer to shared prompt
rows; an explicitly empty negative uses a null reference. Missing or ambiguous
metadata does not mean an empty negative. Pair metadata belongs to the
combination; saving another combination does not overwrite component metadata.
Existing trim/lowercase hash normalization remains unchanged.
Saved components are immutable while referenced. To change text, create or reuse
a component and save a new combination. Archive combinations that have generation
history. Pruning and legacy editing/deletion must respect these dependencies.
Historical workflow snapshots remain intact, including their original text.
## Migration and recovery
Create pair tables after existing legacy rebuild migrations, in an atomic,
repeatable migration. Never scan media during schema initialization. Test fresh,
legacy and current databases, repeated startup, injected failures and retries.
Preserve old backup compatibility. Migrate and validate a disposable backup copy
before replacing live database pages. Reject broken pair schemas, foreign keys,
triggers and views. Use SQLite backup APIs so committed WAL data is included.
## Metadata and scanning
Resolve positive and negative from the same sampler/output context. Track both
node ID and output slot; bound graph traversal and text construction. Unknown
conditioning operations and conflicting sampler candidates remain unresolved.
Never guess that the first encoder or most recent negative belongs to an image.
Import high-confidence pairs atomically with their image association and source
provenance. Repeated scans must not duplicate components, pairs or image links,
regenerate thumbnails, or increment generation run counts. Preserve existing
conflicting/manual associations for review. Report new, enriched, unchanged,
ambiguous, unresolved and failed results separately.
## Live node
Use the shared toolkit and Split foundation. Positive and negative editors have
equal size. Only positive receives prepend, append and automatic LoRA triggers.
Keep upstream connections authoritative. Library selection previews both fields
before an explicit load. Saving text changes creates another reusable combination.
Verify Classic and Nodes 2.0 startup, switching both directions, reload, caching,
linked inputs, popup behavior and lifecycle cleanup before maintainer testing.
## Delivery gate
Keep all existing coverage floors and test suites. Record regression failures
before implementing each behavior. Tests use disposable state and the managed
CPU host; never mutate the maintainer's running ComfyUI or personal database.
Python changes require a ComfyUI restart. Live approval is required before merge.
## Live testing
1. Download a database backup from Admin before the first restart. The additive
schema migration runs on startup; existing workflows and prompt IDs stay valid.
2. Restart ComfyUI, reload the browser, and add **Prompt Manager (+/-)** from
**ComfyAssets / Prompts**. Connect CLIP and the two conditioning outputs to the
sampler's positive and negative inputs.
3. Enter both texts, save a pair, then keep positive unchanged and change negative.
Recent should show both combinations with distinct negative excerpts. Selecting
one previews both; **Use pair** loads both together.
4. Test category/tag autocomplete and metadata opt-in, resizing, light/dark,
switching Classic / Nodes 2.0 both ways, and workflow reload. Connected fields
stay authoritative; run the node to save connected metadata values.
5. Queue the same recipe twice and change only negative once. Usage should count
dequeued jobs even when ComfyUI caches the node. Prepend, append and LoRA triggers
affect positive only. Gallery text is labelled as recorded during execution or
recovered from workflow metadata.
6. Run an output metadata scan to enrich existing images. Repeat it: pairs and
links must remain deduplicated, with unchanged results rather than new links.
Conflicting or ambiguous files need review and are preserved.
Fresh executions embed bounded text evidence in PNG metadata. A cached node may
not emit fresh evidence, so its graph-derived text is explicitly marked inferred.
Multiple distinct batch records without an image anchor remain unresolved.
Rescanning after interruption is safe through idempotent per-file writes; this
version does not persist a filesystem scan cursor across server restarts.
The API can request cancellation with `POST /prompt_manager/scan/cancel`; the
existing dashboard Hide button continues to hide progress without cancelling.
Pair APIs are under `/prompt_manager/pairs`: POST saves a combination, GET lists
and filters, GET/PATCH/DELETE `/{id}` reads, edits metadata or removes an unused
pair. Changing text creates another combination. Null metadata updates preserve
existing values; empty strings/lists clear supported fields. Archive a pair with
`PATCH {"archived":true}`. GET `/export` provides versioned JSON containing both
texts and pair metadata; this is an export format, not a JSON import endpoint.
## Gallery redesign concept retained
The maintainer approved the interactive
[negative variant timeline concept](concepts/prompt-variant-timeline.html) on
2026-10-17. Preserve it for the later gallery redesign; it is not production UI
and does not expand the current node work.
Keep one positive parent and one shared gallery, with red branches for reusable
negative variants and smaller dots for repeat runs. Show the selected negative
in full directly beneath the unchanged positive; selecting another variant or
run updates that text and filters the shared gallery. Comparison shows both
negatives. Distinguish explicitly empty negatives from an all-variants view and
saved variants from executed runs. Production history needs persisted execution
records; the prototype's timeline uses illustrative sample data.
The Admin upgrade/scan notice remains a separate follow-up: explain automatic
schema migration and offer an optional metadata enrichment scan or Later.
Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

+13
View File
@@ -0,0 +1,13 @@
# Native Library visual evidence
Captured from the production workspace using disposable synthetic browser-test
fixtures. No personal images, library, or running ComfyUI state was accessed.
Solid-color media are intentional test images.
- `library-dark.png`: grouped positive shelf, dark appearance.
- `library-light.png`: the same shelf, light appearance.
- `history-selected-negative.png`: selected negative directly below its positive,
saved-variant branches and the filtered shared gallery.
- `history-mobile.png`: the same history at a 390-pixel viewport.
These screenshots document implemented behavior, rather than a design mockup.
Binary file not shown.

After

Width:  |  Height:  |  Size: 139 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 162 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 117 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 94 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 90 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 106 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 105 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 99 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 224 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 231 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 231 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Some files were not shown because too many files have changed in this diff Show More