* feat(images): store images' own categories in the library database (#332) One new table, image_categories, and its category index, created empty by a forward-only migration that reads and writes only its own objects. Startup checks the structure; restore also checks the values. The store sets, clears and reads own categories by file key (the deepest output source holding the file and the path below it), and copies a removed duplicate's category to a kept copy that has none. * feat(images): duplicate cleanup keeps images' own categories (#332) The kept copy's own category wins; a kept copy without one receives the removed copy's, keyed by the deepest output source holding each file. The removed copy's row is never deleted, so a give-back or a failed delete loses nothing. The job log reports how many were kept. * feat(images): set, remove and list images' own categories (#332) PUT /prompt_manager/images/category sets the own category of up to 500 listed files, DELETE removes it (listed or not), and GET /prompt_manager/images/categories lists the values whose file is listed. Targets are the Images listing identity of an output source, checked by media_target (shared with Collections). The routes never touch prompts, recipes or links; POST /prompt_manager/bulk/category is unchanged. * feat(vocabulary): count and rename image categories (#332) Tags & categories lists every category value with image_count, the stored image_categories rows holding it. Values only images hold follow every prompt-backed value (first_pid at SQLite's INTEGER maximum), so keyset cursors cross the boundary. /library/options keeps its own query and never lists image-only values. Manage category counts image rows (impact.images), treats an image-held target as existing, renames image rows in the same transaction, allows renaming an image-only value, and hashes the sorted image rowids into the impact token so a change between preview and apply is refused with vocabulary-conflict. Every image-category statement reads idx_image_categories_category. * feat(images): the category filter matches each image's effective category (#332) An image's own category wins; without one it inherits its link's: the recipe's category when set, else the positive prompt's (MC4, one rule with the inspector chip). Ambiguous and unlinked images inherit nothing, so an unlinked image with its own category is now listed. The link read no longer filters by category in SQL; the page filter, select all N matching and saved views compare the effective value. A failed own-category read answers the existing listing-failed 500. * fix(images): key symlinked output folders and keep key work off the event loop (#332) Review fixes: a source matches by its configured spelling too, so a detected output folder that is a symlink keys its files (routes and duplicate cleanup). File keys are computed in the executor. A path no source holds is a plain 403, a lone surrogate a 400. The install repairs a table that lacks its category index instead of refusing to open. * feat(inspector): show an image's own category in the inspector and History (#332) The inspector's library facts gain category_scope "image": an image's own category overrides the inherited one for any link status (linked, ambiguous, none). Images reads it for output sources only; History reads it once per inspection inside its snapshot, keyed by the same file key from the symlink-resolved output folder. A failed read hides the chip rather than show a possibly wrong value. The chip's hidden prefix names the origin ("Image category: "); the visible origin text comes in PR 3. * perf(images): keep the category filter's link read narrow; record D141, D142, D145 (#332) Under a category the link read keeps only links whose raw recipe or positive category is the value (Python still decides the inherited one), the links of images whose own category is the value are read by path, and plain reads skip the inherited-category work. The own-category map is keyed by normalized path. TESTING.md records D141, D142 and D145 and amends D13, D36 and D39; the filter budget seeds 2,000 own categories. * feat(images): set or remove a category on all N matching images (#332) The category routes also accept the Images listing filter of "Select all N matching" (#392) instead of explicit items, resolved now and capped at 500 images like an explicit selection. * fix(images): Windows path keys for own categories; plan check for the category read (#332) Review fixes: on Windows the own-category map keys each file with normalize_image_path (abspath drops trailing dots and spaces), and the plan test also covers the category's narrowing clause. * test(vendor): count the #332 host case in the execution matrix image-attribution.spec.mjs gains the image category route case, so the execution config lists 92 tests (3 attribution cases per project). * test(legacy): record the image category routes in the legacy contract snapshot (#332) #332 adds PUT/DELETE /prompt_manager/images/category and GET /prompt_manager/images/categories; the legacy pages' routes are unchanged.
This commit is contained in:
+51
-5
@@ -1344,7 +1344,7 @@ How the earlier workspace Images filters moved (old → new). The legacy `/galle
|
||||
| Folder | "Refine this view" → Folder (D14); shown in the scope chip |
|
||||
| Media (All media / Images / Videos) | "Refine this view" → Media (D14) |
|
||||
| Sort: Newest first / Oldest first / Filename A–Z / Filename Z–A | toolbar sort: Recently added / Oldest first / Name A–Z / Name Z–A, plus Highest rated (D13, D14) |
|
||||
| (none) | toolbar category and "Refine this view" → Tags, through the linked positive (D13, D14) |
|
||||
| (none) | toolbar category (the effective category, D141) and "Refine this view" → Tags, through the linked positive (D13, D14) |
|
||||
| Refresh | Import → "Scan an output folder" → Rescan (D15) |
|
||||
| Grid / List / Masonry, Comfortable / Large / Compact | the same choices in the concept filter row |
|
||||
| Card image → side panel; preview icon → viewer | image → viewer; Info → side panel (D19) |
|
||||
@@ -1427,7 +1427,8 @@ Recorded deviations on this page (spec §8):
|
||||
`overscroll-behavior: contain`.
|
||||
- **D35**: counts pluralize ("1 positive prompt", "1 image", "1 more image loaded").
|
||||
- **D36**: the category list is the Library's, sorted alphabetically (case-insensitive)
|
||||
by the page, whatever order the server sends.
|
||||
by the page, whatever order the server sends (amended by D141: Images adds its
|
||||
image categories).
|
||||
- **D37**: production loads 40 images per page (the concept 12), so the continuation
|
||||
and toast use 40; after Escape production returns focus to the opening image.
|
||||
- **D44** (#249 PR E): Filters reads "Filters · N" while N of Source, Folder, Media and
|
||||
@@ -1513,15 +1514,46 @@ Recorded deviations on this page (spec §8):
|
||||
- **D39** (ruled 2026-10-21: fallback): a recipe-linked image shows its recipe's own
|
||||
category when set, otherwise the linked positive prompt's, and its recipe's own tags
|
||||
when it has any, otherwise the positive's; each field falls back on its own and the
|
||||
two tag sets are never merged. Set category (#259) writes the positive's category and
|
||||
the Images category and Tags filters match the positive's, so the panel now agrees
|
||||
with them whenever the recipe adds nothing. The inspection response names each
|
||||
two tag sets are never merged. The Images Tags filter matches the positive's tags.
|
||||
The category filter matches the same inherited category as the panel (D142, #332), and
|
||||
an image's own category overrides both (D141, D146). The inspection response names each
|
||||
field's source (`library.category_scope`, `library.tags_scope`: `pair`, `positive` or
|
||||
null); the category chip's visually hidden prefix (D21) says "Recipe category: " or
|
||||
"Positive prompt category: " from it, and tag chips are Tags filter buttons only when
|
||||
they are the positive's (#260 D2). History's media inspector uses the same rule. The
|
||||
concept shows the prompt's category and tags for every image and the concept
|
||||
fixture's recipes have neither, so the inspector comparison needs no mapping for it.
|
||||
- **D141** (#332, amends D13 and D36): the Images category filter matches each image's
|
||||
effective category: its own category (Images Set category, PR 3) when it has one,
|
||||
else its inherited one (D142), else none. An image whose own category differs never
|
||||
matches its prompt's category, and an unlinked or ambiguous image with an own
|
||||
category is listed under it (the members-only gate now applies to the Tags filter
|
||||
only). "Select all N matching" and saved views use the same value. The caption's
|
||||
positive-prompt count still counts the linked positives of the images shown. With a
|
||||
category set the listing reads the own categories once per source (a primary-key
|
||||
range); a failed read is the existing 500 `listing-failed`, never an unfiltered page.
|
||||
The link read then keeps only links whose raw recipe or positive category is the
|
||||
value (a superset of those that inherit it; Python decides), and reads the links of
|
||||
files whose own category is the value by path, so their rating and positive count.
|
||||
`test_filter_budget` seeds 2,000 own categories on its 50,000-file fixture.
|
||||
The option list adds the image categories of listed files
|
||||
(`GET /prompt_manager/images/categories`, PR 3).
|
||||
- **D142** (#332 MC4, amends D39's filter note): the inherited category is one rule,
|
||||
`inherited_category` in `utils/inspection_response.py`: a linked recipe's own
|
||||
category when it is not blank, else the linked positive's. Before #332 the filter
|
||||
matched the positive's even when the chip showed the recipe's. The link reads select
|
||||
the raw recipe and positive categories and never decide inheritance in SQL
|
||||
(`tests/test_api_workspace_image_categories.py` checks the filter and the chip agree
|
||||
on blank values stored as spaces, tabs, U+00A0 and BLOB). The concept has no recipe
|
||||
categories, so this is behavior only.
|
||||
- **D145** (#332, DC4): History's media inspector shows an image's own category with
|
||||
the chip rule of the side panel and viewer. Only the inspection request reads it, once,
|
||||
inside the History snapshot: the file is the symlink-resolved output folder plus the
|
||||
receipt's subfolder and filename, keyed with `image_category_key` over the configured
|
||||
output sources, as Images keys it. A temp output or a file no source holds has none;
|
||||
a failed read hides the chip (DC5). The outputs list and media requests read nothing
|
||||
new. The chip's hidden prefix reads "Image category: " for an own category; the
|
||||
visible origin text and its comparison are D146 (PR 3).
|
||||
- **D42**: the concept never shows a toast and a page alert together. Production keeps
|
||||
the concept's fixed toast (centred, 22px from the bottom, 12px at 700px and below)
|
||||
and, when an Import → Rescan toast leaves the Images discovery alert
|
||||
@@ -5749,6 +5781,20 @@ rotates the epoch, so a pre-restore `expected_epoch` is refused with `storage-ch
|
||||
Owner rows and their timestamps never change on a tag command; a category apply only
|
||||
touches rows carrying the exact source.
|
||||
|
||||
Image categories (#332 §7, MC9, MC10, DC7): each `GET categories` row also carries
|
||||
`image_count`, the stored `image_categories` rows holding that exact value (files
|
||||
missing on disk included). A value prompts and images share appears once, ordered by
|
||||
its oldest positive; a value only images hold appears once after every prompt-backed
|
||||
value (`first_pid` 9223372036854775807 in the keyset cursor), ordered by value, with
|
||||
`prompt_count` 0. `/library/options` keeps its own query and never lists image-only
|
||||
values. Category preview and apply report `impact.images` (rows holding the source),
|
||||
the target exists when an image row holds it, an image-only value can be renamed, the
|
||||
apply renames the image rows in the same transaction (file keys and rowids kept, so a
|
||||
merge cannot collide), and the impact token hashes the sorted image rowids, so an image
|
||||
row gaining or losing the value between preview and apply is 409
|
||||
`vocabulary-conflict` with nothing written. Every image-category statement reads
|
||||
`idx_image_categories_category` (`test_category_reads_use_the_image_category_index`).
|
||||
|
||||
Two legacy routes change additively: `GET /tags` now lists unused registry tags, and
|
||||
`/tags/suggest` without context appends unused names at count 0. Unused tags become
|
||||
deletable only through the native Delete tag (vocabulary UI PR), not on the legacy
|
||||
|
||||
+122
-60
@@ -7,7 +7,12 @@ import json
|
||||
import math
|
||||
import datetime
|
||||
import os
|
||||
from typing import Optional, List, Dict, Any, Set, Tuple
|
||||
from typing import Any, Dict, Iterable, List, Optional, Sequence, Set, Tuple
|
||||
|
||||
try:
|
||||
from ..utils.inspection_response import inherited_category
|
||||
except ImportError:
|
||||
from utils.inspection_response import inherited_category
|
||||
|
||||
from .models import PromptModel, normalize_image_path, utc_now_iso
|
||||
from .pairs import PairDependencyError, PairOperations # noqa: F401 (re-exported)
|
||||
@@ -238,13 +243,17 @@ LEFT JOIN prompt_library_state pls ON pls.prompt_id = pp.positive_prompt_id
|
||||
WHERE gi.image_path IN ({marks}) OR gi.file_path IN ({marks}){fallback}"""
|
||||
# One image row describes a card only when no other row claims the same file
|
||||
# (as in get_image_link_summaries, several rows are ambiguous). The pair scope
|
||||
# and the linked positive follow _LINK_SUMMARY_SQL; category, rating, tags and
|
||||
# the searchable text and titles come from that positive (spec D12/D13). Every
|
||||
# and the linked positive follow _LINK_SUMMARY_SQL; rating, tags and the
|
||||
# searchable text and titles come from that positive (spec D12/D13). The raw
|
||||
# recipe and positive categories are read for the inherited category, which
|
||||
# Python decides (``inherited_category``, #332 MC4); SQL never filters by
|
||||
# category. Every
|
||||
# writer since the 3.2.4 migration stores file_path, so it is the path key.
|
||||
# The prompt-level conditions are uncorrelated IN sets SQLite builds once, and
|
||||
# the CASE evaluates the per-row ambiguity probe only for admitted rows.
|
||||
_LINK_FILTER_SQL = """SELECT gi.file_path AS file_path, p.rating AS rating,
|
||||
{hit} AS matched, p.id AS positive
|
||||
{hit} AS matched, p.id AS positive, pp.id IS NOT NULL AS has_pair,
|
||||
pp.category AS pair_category, p.category AS category
|
||||
FROM generated_images gi
|
||||
LEFT JOIN prompt_pairs pp ON pp.id = (
|
||||
SELECT CASE WHEN COUNT(DISTINCT ip.pair_id) = 1 THEN MIN(ip.pair_id) END
|
||||
@@ -256,20 +265,28 @@ WHERE gi.file_path IS NOT NULL AND CASE WHEN {admit} THEN NOT EXISTS (
|
||||
# _LINK_FILTER_SQL's rows as one JSON array, built by SQLite in one step (#354).
|
||||
# Python's sqlite3 releases and retakes the GIL around every step, so a read
|
||||
# that steps once per linked row waits on a busy ComfyUI thread thousands of
|
||||
# times. A BLOB file_path or rating (no writer stores one) makes the value NULL
|
||||
# (or json_array refuses it), and the read streams instead.
|
||||
# times. A BLOB value (no writer stores one) makes the value NULL (or
|
||||
# json_array refuses it), and the read streams instead. The raw categories are
|
||||
# put in only for a category read (#332), so plain reads stay as small as #354
|
||||
# measured them.
|
||||
_LINK_MATCHES_JSON_SQL = """SELECT CASE
|
||||
WHEN max(typeof(file_path) = 'blob' OR typeof(rating) = 'blob') THEN NULL
|
||||
ELSE json_group_array(json_array(file_path, rating, matched, positive)) END
|
||||
WHEN max({blob}) THEN NULL
|
||||
ELSE json_group_array(json_array({columns})) END
|
||||
FROM ({select})"""
|
||||
_LINK_PLAIN_COLUMNS = ("file_path", "rating", "matched", "positive")
|
||||
_LINK_CATEGORY_COLUMNS = _LINK_PLAIN_COLUMNS + ("has_pair", "pair_category", "category")
|
||||
# The columns a BLOB can reach: matched, positive and has_pair are computed.
|
||||
_LINK_STORED_COLUMNS = ("file_path", "rating", "pair_category", "category")
|
||||
# Set when this SQLite has no JSON functions: every later read streams.
|
||||
_one_step_unsupported = False
|
||||
# One warning per process when a read falls back to streaming.
|
||||
_one_step_warned = False
|
||||
# The linked positive of each given path (one unambiguous link, pair scope as
|
||||
# in _LINK_FILTER_SQL) with its category; {hit} as in link_filter_query. Used
|
||||
# for the few files whose current image tags meet a Tags filter (#249 B3).
|
||||
_LINK_FACTS_SQL = """SELECT gi.file_path, p.rating, {hit}, p.id, p.category
|
||||
# in _LINK_FILTER_SQL) with the raw categories of its inherited category; {hit}
|
||||
# as in link_filter_query. Used for the few files whose current image tags meet
|
||||
# a Tags filter (#249 B3).
|
||||
_LINK_FACTS_SQL = """SELECT gi.file_path, p.rating, {hit}, p.id,
|
||||
pp.id IS NOT NULL, pp.category, p.category
|
||||
FROM generated_images gi
|
||||
LEFT JOIN prompt_pairs pp ON pp.id = (
|
||||
SELECT CASE WHEN COUNT(DISTINCT ip.pair_id) = 1 THEN MIN(ip.pair_id) END
|
||||
@@ -298,8 +315,24 @@ _LINK_FILTER_TAG = (
|
||||
)
|
||||
|
||||
|
||||
# One unambiguously linked image's facts for the Images filters, a plain tuple
|
||||
# (hit, rating, positive id, inherited category): built per linked row, so no
|
||||
# NamedTuple. The category is the inherited one (#332 MC4: the recipe's when
|
||||
# set, else the positive's), never the image's own category.
|
||||
LinkMatch = Tuple[bool, Optional[int], int, Any]
|
||||
|
||||
|
||||
def _link_match(rating, matched, positive, has_pair, pair_category, category):
|
||||
inherited, _scope = inherited_category(category, pair_category, bool(has_pair))
|
||||
return (bool(matched), rating, positive, inherited)
|
||||
|
||||
|
||||
def link_filter_query(
|
||||
q: str, category: Optional[str], tags: List[str], *, every_link: bool = False
|
||||
q: str,
|
||||
tags: List[str],
|
||||
*,
|
||||
every_link: bool = False,
|
||||
category: Optional[str] = None,
|
||||
) -> Tuple[str, Dict[str, Any]]:
|
||||
"""The workspace filter SELECT and its named parameters (all bound).
|
||||
|
||||
@@ -313,7 +346,9 @@ def link_filter_query(
|
||||
params["pattern"] = f"%{escape_like(q)}%"
|
||||
admitted = []
|
||||
if category is not None:
|
||||
admitted.append("p.category = :category")
|
||||
# Only narrows: an inherited category is one of the two raw values;
|
||||
# which one is decided in Python (inherited_category, #332 MC4).
|
||||
admitted.append("(p.category = :category OR pp.category = :category)")
|
||||
params["category"] = category
|
||||
for index, tag in enumerate(tags):
|
||||
admitted.append(_LINK_FILTER_TAG.format(index=index))
|
||||
@@ -324,9 +359,37 @@ def link_filter_query(
|
||||
return sql, params
|
||||
|
||||
|
||||
def link_matches_json_query(sql: str) -> str:
|
||||
"""*sql* from link_filter_query, wrapped to return its rows as one JSON value."""
|
||||
return _LINK_MATCHES_JSON_SQL.format(select=sql)
|
||||
def link_matches_json_query(sql: str, categories: bool = False) -> str:
|
||||
"""*sql* from link_filter_query, wrapped to return its rows as one JSON value.
|
||||
|
||||
With *categories* each row also carries the raw values the inherited
|
||||
category is decided from; otherwise only path, rating, hit and positive.
|
||||
"""
|
||||
columns = _LINK_CATEGORY_COLUMNS if categories else _LINK_PLAIN_COLUMNS
|
||||
blob = " OR ".join(
|
||||
f"typeof({name}) = 'blob'" for name in _LINK_STORED_COLUMNS if name in columns
|
||||
)
|
||||
return _LINK_MATCHES_JSON_SQL.format(
|
||||
blob=blob, columns=", ".join(columns), select=sql
|
||||
)
|
||||
|
||||
|
||||
def _link_matches_from_rows(
|
||||
rows: Iterable[Sequence[Any]], categories: bool, wanted: Optional[Set[str]]
|
||||
) -> Dict[str, LinkMatch]:
|
||||
"""LinkMatch per wanted path from link rows (one-step or streamed).
|
||||
|
||||
Without *categories* the inherited category is None, as the caller then
|
||||
never compares it.
|
||||
"""
|
||||
matches: Dict[str, LinkMatch] = {}
|
||||
for path, rating, matched, positive, *raw in rows:
|
||||
if wanted is None or path in wanted:
|
||||
if categories:
|
||||
matches[path] = _link_match(rating, matched, positive, *raw[:3])
|
||||
else:
|
||||
matches[path] = (bool(matched), rating, positive, None)
|
||||
return matches
|
||||
|
||||
|
||||
def _streamed_link_matches(
|
||||
@@ -334,13 +397,10 @@ def _streamed_link_matches(
|
||||
sql: str,
|
||||
params: Dict[str, Any],
|
||||
wanted: Optional[Set[str]],
|
||||
) -> Dict[str, Tuple[bool, Optional[int], int]]:
|
||||
categories: bool = False,
|
||||
) -> Dict[str, LinkMatch]:
|
||||
"""The link read stepped once per row (the fallback of the one-step read)."""
|
||||
matches: Dict[str, Tuple[bool, Optional[int], int]] = {}
|
||||
for path, rating, matched, positive in conn.execute(sql, params):
|
||||
if wanted is None or path in wanted:
|
||||
matches[path] = (bool(matched), rating, positive)
|
||||
return matches
|
||||
return _link_matches_from_rows(conn.execute(sql, params), categories, wanted)
|
||||
|
||||
|
||||
_ASCII_FOLD = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
|
||||
@@ -2576,47 +2636,55 @@ class PromptDatabase(
|
||||
def get_image_link_matches(
|
||||
self,
|
||||
q: str,
|
||||
category: Optional[str],
|
||||
tags: List[str],
|
||||
*,
|
||||
every_link: bool = False,
|
||||
wanted: Optional[Set[str]] = None,
|
||||
) -> Dict[str, Tuple[bool, Optional[int], int]]:
|
||||
"""Map normalized paths of unambiguously linked images to
|
||||
(hit, rating, positive id).
|
||||
category: Optional[str] = None,
|
||||
) -> Dict[str, LinkMatch]:
|
||||
"""Map normalized paths of unambiguously linked images to a LinkMatch
|
||||
(hit, rating, positive id, inherited category).
|
||||
|
||||
Rows are limited to images whose linked positive has *category* and
|
||||
every tag in *tags* (exact names). ``hit`` says whether *q* occurs in
|
||||
Rows are limited to images whose linked positive has every tag in
|
||||
*tags* (exact names). The category filter is applied by the caller to
|
||||
the effective category (#332). ``hit`` says whether *q* occurs in
|
||||
the positive's text, Library title or a tag name; a pair's own title
|
||||
names its negative variant and is not searched.
|
||||
Without category or tags only hits are returned, unless *every_link*
|
||||
Without tags only hits are returned, unless *every_link*
|
||||
asks for every linked image (the rating sort and the positive count).
|
||||
The positive id is the card's linked positive.
|
||||
The positive id is the card's linked positive. With *category* only
|
||||
links whose raw recipe or positive category is that value are read (a
|
||||
superset of the links that inherit it; the caller compares the
|
||||
inherited value) and the inherited category is decided per row;
|
||||
otherwise it is None, which keeps the plain reads cheap.
|
||||
|
||||
One SELECT, returned as one JSON value (#354): SQLite steps once, so
|
||||
the read takes the GIL back a constant two or three times instead of
|
||||
once per linked row while ComfyUI generates. Peak memory therefore
|
||||
grows with the database's matching linked rows (about 530 bytes each,
|
||||
freed on return); the result is bounded by *wanted* (the listing's
|
||||
normalized paths) when given. Where the JSON step fails (no JSON
|
||||
functions, a BLOB value, a JSON text over SQLite's length limit) the
|
||||
rows are streamed as before.
|
||||
more with the raw categories of a category read, freed on return); the
|
||||
result is bounded by *wanted* (the listing's normalized paths) when
|
||||
given. Where the JSON step fails (no JSON functions, a BLOB value, a
|
||||
JSON text over SQLite's length limit) the rows are streamed as before.
|
||||
"""
|
||||
sql, params = link_filter_query(q, category, tags, every_link=every_link)
|
||||
sql, params = link_filter_query(
|
||||
q, tags, every_link=every_link, category=category
|
||||
)
|
||||
categories = category is not None
|
||||
with self.model.get_connection() as conn:
|
||||
rows = None
|
||||
if not _one_step_unsupported:
|
||||
rows = self._one_step_link_rows(conn, sql, params)
|
||||
rows = self._one_step_link_rows(conn, sql, params, categories)
|
||||
if rows is None:
|
||||
return _streamed_link_matches(conn, sql, params, wanted)
|
||||
return {
|
||||
path: (bool(matched), rating, positive)
|
||||
for path, rating, matched, positive in rows
|
||||
if wanted is None or path in wanted
|
||||
}
|
||||
return _streamed_link_matches(conn, sql, params, wanted, categories)
|
||||
return _link_matches_from_rows(rows, categories, wanted)
|
||||
|
||||
def _one_step_link_rows(
|
||||
self, conn: sqlite3.Connection, sql: str, params: Dict[str, Any]
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
sql: str,
|
||||
params: Dict[str, Any],
|
||||
categories: bool = False,
|
||||
) -> Optional[List[List[Any]]]:
|
||||
"""*sql*'s rows from one JSON value, or None when this read must stream.
|
||||
|
||||
@@ -2626,7 +2694,7 @@ class PromptDatabase(
|
||||
global _one_step_unsupported, _one_step_warned
|
||||
error: Optional[str] = None
|
||||
try:
|
||||
cursor = conn.execute(link_matches_json_query(sql), params)
|
||||
cursor = conn.execute(link_matches_json_query(sql, categories), params)
|
||||
try:
|
||||
((document,),) = cursor.fetchall()
|
||||
finally:
|
||||
@@ -2653,9 +2721,9 @@ class PromptDatabase(
|
||||
|
||||
def get_image_link_facts(
|
||||
self, paths: List[str], q: str = ""
|
||||
) -> Dict[str, Tuple[bool, Optional[int], int, Optional[str], Tuple[str, ...]]]:
|
||||
) -> Dict[str, Tuple[bool, Optional[int], int, Any, Tuple[str, ...]]]:
|
||||
"""Map each normalized path with one unambiguous link to its positive's
|
||||
(hit, rating, positive id, category, tag names).
|
||||
(hit, rating, positive id, inherited category, tag names).
|
||||
|
||||
The link is resolved as in get_image_link_matches, and ``hit`` says
|
||||
whether *q* occurs in the positive's text, Library title or a tag
|
||||
@@ -2666,7 +2734,7 @@ class PromptDatabase(
|
||||
if q:
|
||||
hit = _LINK_FILTER_HIT
|
||||
params["pattern"] = f"%{escape_like(q)}%"
|
||||
rows: Dict[str, Tuple[bool, Optional[int], int, Optional[str]]] = {}
|
||||
rows: Dict[str, LinkMatch] = {}
|
||||
names: Dict[int, List[str]] = {}
|
||||
unique = list(dict.fromkeys(paths))
|
||||
with self.model.get_connection() as conn:
|
||||
@@ -2675,10 +2743,8 @@ class PromptDatabase(
|
||||
bound = {f"path{i}": path for i, path in enumerate(chunk)}
|
||||
marks = ",".join(f":{key}" for key in bound)
|
||||
sql = _LINK_FACTS_SQL.format(hit=hit, marks=marks)
|
||||
for path, rating, matched, positive, category in conn.execute(
|
||||
sql, {**params, **bound}
|
||||
):
|
||||
rows[path] = (bool(matched), rating, positive, category)
|
||||
for path, *row in conn.execute(sql, {**params, **bound}):
|
||||
rows[path] = _link_match(*row)
|
||||
positives = sorted({row[2] for row in rows.values()})
|
||||
for start in range(0, len(positives), _FACTS_CHUNK):
|
||||
chunk = positives[start : start + _FACTS_CHUNK]
|
||||
@@ -2691,20 +2757,16 @@ class PromptDatabase(
|
||||
path: (*row, tuple(names.get(row[2], ()))) for path, row in rows.items()
|
||||
}
|
||||
|
||||
def get_image_paths_matching(
|
||||
self, q: str, category: Optional[str], tags: List[str]
|
||||
) -> Set[str]:
|
||||
"""Normalized paths whose linked positive matches *q*, *category* and *tags*.
|
||||
def get_image_paths_matching(self, q: str, tags: List[str]) -> Set[str]:
|
||||
"""Normalized paths whose linked positive matches *q* and *tags*.
|
||||
|
||||
See get_image_link_matches; an empty *q* matches every linked image
|
||||
the category and tags admit.
|
||||
the tags admit.
|
||||
"""
|
||||
return {
|
||||
path
|
||||
for path, (hit, _rating, _positive) in self.get_image_link_matches(
|
||||
q, category, tags
|
||||
).items()
|
||||
if hit or not q
|
||||
for path, match in self.get_image_link_matches(q, tags).items()
|
||||
if match[0] or not q
|
||||
}
|
||||
|
||||
def get_prompt_id_for_image(self, image_path: str) -> Optional[int]:
|
||||
|
||||
+55
-15
@@ -13,7 +13,9 @@ same connection before commit. Owner rows and their timestamps never change.
|
||||
|
||||
Category consolidation is previewed in one ``library_snapshot`` and applied in
|
||||
one ``_pair_transaction`` that recomputes the owner projection and requires the
|
||||
preview's ``impact_token``; only rows carrying the exact source change.
|
||||
preview's ``impact_token``; only rows carrying the exact source change. Image
|
||||
rows holding the source (``image_categories``, #332) are owners too: they are
|
||||
counted, hashed into the token and renamed with the prompts and pairs.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
@@ -97,16 +99,30 @@ def _validate_tag_id(tag_id: object) -> None:
|
||||
raise ValueError("Invalid tag ID")
|
||||
|
||||
|
||||
# The /library/options universe: nonblank values on eligible parents or on
|
||||
# active pairs, grouped exactly (whitespace and case variants stay distinct).
|
||||
_CATEGORY_GROUPS = (
|
||||
# Category values in Tags & categories (#332 §7): the prompt-backed values
|
||||
# (nonblank values on eligible parents or on active pairs, grouped exactly, so
|
||||
# whitespace and case variants stay distinct; /library/options lists the same
|
||||
# prompt-backed set with its own query) plus the values only image rows hold.
|
||||
# A prompt-backed value is ordered by its oldest owning positive; an image-only
|
||||
# value sorts after all of them (first_pid is SQLite's INTEGER maximum, above
|
||||
# any prompt id), then by value. ``image_count`` counts stored rows, including
|
||||
# files missing on disk. No FULL OUTER JOIN, so it runs on SQLite 3.31+.
|
||||
_IMAGE_ONLY_FIRST_PID = 2**63 - 1
|
||||
_CATEGORY_ROWS = (
|
||||
"WITH owners(value,pid) AS ("
|
||||
"SELECT p.category,p.id FROM prompts p WHERE p.category IS NOT NULL "
|
||||
"AND TRIM(p.category)!='' AND " + _ELIGIBLE + " "
|
||||
"UNION SELECT pp.category,pp.positive_prompt_id FROM prompt_pairs pp "
|
||||
"WHERE pp.archived=0 AND pp.category IS NOT NULL AND TRIM(pp.category)!=''),"
|
||||
"grouped AS (SELECT value,MIN(pid) AS first_pid,"
|
||||
"COUNT(DISTINCT pid) AS prompt_count FROM owners GROUP BY value) "
|
||||
"image_values(value,image_count) AS ("
|
||||
"SELECT category,COUNT(*) FROM image_categories GROUP BY category),"
|
||||
"grouped(value,first_pid,prompt_count) AS ("
|
||||
"SELECT value,MIN(pid),COUNT(DISTINCT pid) FROM owners GROUP BY value "
|
||||
"UNION ALL SELECT value," + str(_IMAGE_ONLY_FIRST_PID) + ",0 FROM image_values "
|
||||
"WHERE value NOT IN (SELECT value FROM owners)),"
|
||||
"counted AS (SELECT g.value,g.first_pid,g.prompt_count,"
|
||||
"COALESCE(i.image_count,0) AS image_count FROM grouped g "
|
||||
"LEFT JOIN image_values i ON i.value=g.value) "
|
||||
)
|
||||
|
||||
|
||||
@@ -117,15 +133,23 @@ _CATEGORY_COMPONENTS = (
|
||||
"WHERE p.category=? ORDER BY p.id"
|
||||
)
|
||||
_CATEGORY_PAIRS = "SELECT id,archived FROM prompt_pairs WHERE category=? ORDER BY id"
|
||||
# Image rows holding one exact value (#332 DC7), an index search.
|
||||
_IMAGE_CATEGORY_ROWIDS = (
|
||||
"SELECT rowid FROM image_categories WHERE category=? ORDER BY rowid"
|
||||
)
|
||||
_CATEGORY_EXISTS = (
|
||||
"SELECT EXISTS(SELECT 1 FROM prompts WHERE category=:v)"
|
||||
" OR EXISTS(SELECT 1 FROM prompt_pairs WHERE category=:v)"
|
||||
" OR EXISTS(SELECT 1 FROM image_categories WHERE category=:v)"
|
||||
)
|
||||
|
||||
# Pairs after components, so a failing pair update rolls both back.
|
||||
# Pairs after components and image rows last, so a failing update rolls every
|
||||
# earlier one back. An image row is keyed by its file, never by its value, so
|
||||
# merging into a value other files already hold cannot conflict.
|
||||
_CATEGORY_UPDATES = (
|
||||
"UPDATE prompts SET category=?,updated_at=? WHERE category=?",
|
||||
"UPDATE prompt_pairs SET category=?,updated_at=? WHERE category=?",
|
||||
"UPDATE image_categories SET category=?,updated_at=? WHERE category=?",
|
||||
)
|
||||
|
||||
|
||||
@@ -136,8 +160,15 @@ def category_impact_token(
|
||||
target_exists: bool,
|
||||
components: list,
|
||||
pairs: list,
|
||||
*,
|
||||
images: list = (),
|
||||
) -> str:
|
||||
"""Hash a category projection like ``library_edit_token`` (no secret)."""
|
||||
"""Hash a category projection like ``library_edit_token`` (no secret).
|
||||
|
||||
``images`` are the rowids of the image rows holding ``source`` (#332 DC7).
|
||||
With none the preimage has no ``images`` key, so a prompt-only token keeps
|
||||
the value it had before image categories existed.
|
||||
"""
|
||||
preimage = {
|
||||
"epoch": epoch,
|
||||
"source": source,
|
||||
@@ -146,6 +177,8 @@ def category_impact_token(
|
||||
"components": sorted([int(i), bool(flag)] for i, flag in components),
|
||||
"pairs": sorted([int(i), bool(flag)] for i, flag in pairs),
|
||||
}
|
||||
if images:
|
||||
preimage["images"] = sorted(int(i) for i in images)
|
||||
return canonical_token(_CATEGORY_DOMAIN, preimage)
|
||||
|
||||
|
||||
@@ -325,7 +358,8 @@ class VocabularyOperations:
|
||||
pairs = [
|
||||
[row[0], bool(row[1])] for row in conn.execute(_CATEGORY_PAIRS, (source,))
|
||||
]
|
||||
if not components and not pairs:
|
||||
images = [row[0] for row in conn.execute(_IMAGE_CATEGORY_ROWIDS, (source,))]
|
||||
if not components and not pairs and not images:
|
||||
raise LibraryCommandError("vocabulary-conflict", epoch)
|
||||
target_exists = bool(
|
||||
conn.execute(_CATEGORY_EXISTS, {"v": target}).fetchone()[0]
|
||||
@@ -339,9 +373,10 @@ class VocabularyOperations:
|
||||
"other_prompts": len(components) - eligible,
|
||||
"negative_variants": len(pairs) - archived,
|
||||
"archived_negative_variants": archived,
|
||||
"images": len(images),
|
||||
},
|
||||
"impact_token": category_impact_token(
|
||||
epoch, source, target, target_exists, components, pairs
|
||||
epoch, source, target, target_exists, components, pairs, images=images
|
||||
),
|
||||
}
|
||||
|
||||
@@ -365,7 +400,8 @@ class VocabularyOperations:
|
||||
def apply_vocabulary_category(
|
||||
self, *, expected_epoch: str, source: str, target: str, impact_token: str
|
||||
) -> dict:
|
||||
"""Set the exact ``source`` to ``target`` on every component and pair.
|
||||
"""Set the exact ``source`` to ``target`` on every component, pair
|
||||
and image row.
|
||||
|
||||
The recomputed projection must still match the preview's token; a
|
||||
mismatch is ``vocabulary-conflict`` with nothing written.
|
||||
@@ -422,15 +458,15 @@ class VocabularyOperations:
|
||||
_limit(limit)
|
||||
key = _category_after(after)
|
||||
total = conn.execute(
|
||||
_CATEGORY_GROUPS + "SELECT COUNT(*) FROM grouped"
|
||||
_CATEGORY_ROWS + "SELECT COUNT(*) FROM counted"
|
||||
).fetchone()[0]
|
||||
where, params = "", []
|
||||
if key is not None:
|
||||
where = " WHERE first_pid>? OR (first_pid=? AND value>?)"
|
||||
params = [key[0], key[0], key[1]]
|
||||
rows = conn.execute(
|
||||
_CATEGORY_GROUPS
|
||||
+ "SELECT value,first_pid,prompt_count FROM grouped"
|
||||
_CATEGORY_ROWS
|
||||
+ "SELECT value,first_pid,prompt_count,image_count FROM counted"
|
||||
+ where
|
||||
+ " ORDER BY first_pid,value LIMIT ?",
|
||||
[*params, limit + 1],
|
||||
@@ -440,7 +476,11 @@ class VocabularyOperations:
|
||||
return {
|
||||
"library_epoch": epoch,
|
||||
"categories": [
|
||||
{"value": row["value"], "prompt_count": row["prompt_count"]}
|
||||
{
|
||||
"value": row["value"],
|
||||
"prompt_count": row["prompt_count"],
|
||||
"image_count": row["image_count"],
|
||||
}
|
||||
for row in rows
|
||||
],
|
||||
"total": total,
|
||||
|
||||
+53
-5
@@ -5,6 +5,7 @@ import base64
|
||||
import binascii
|
||||
from functools import wraps
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
from typing import Callable
|
||||
@@ -27,6 +28,7 @@ from .history_current import (
|
||||
snapshot_current_output_scope,
|
||||
)
|
||||
from .inspection_service import CurrentFileChanged, inspect_current_file
|
||||
from .images import image_category_key
|
||||
|
||||
try:
|
||||
from ...database.execution_history import HistoryEpochChanged
|
||||
@@ -39,7 +41,11 @@ try:
|
||||
WindowPosition,
|
||||
)
|
||||
from ...utils.history_read_fence import HistoryReadFence
|
||||
from ...utils.inspection_response import encode_history_inspection_response
|
||||
from ...utils.inspection_response import (
|
||||
OWN_CATEGORY_UNREAD,
|
||||
apply_own_category,
|
||||
encode_history_inspection_response,
|
||||
)
|
||||
except ImportError:
|
||||
from database.execution_history import HistoryEpochChanged
|
||||
from database.history_gallery import read_history_gallery
|
||||
@@ -51,7 +57,11 @@ except ImportError:
|
||||
WindowPosition,
|
||||
)
|
||||
from utils.history_read_fence import HistoryReadFence
|
||||
from utils.inspection_response import encode_history_inspection_response
|
||||
from utils.inspection_response import (
|
||||
OWN_CATEGORY_UNREAD,
|
||||
apply_own_category,
|
||||
encode_history_inspection_response,
|
||||
)
|
||||
|
||||
|
||||
_RUN_MAX_BYTES = 8 * 1024 * 1024
|
||||
@@ -677,6 +687,32 @@ class HistoryRoutesMixin:
|
||||
) -> web.Response:
|
||||
return await self._history_current(request, "inspection")
|
||||
|
||||
def _history_own_category(self, reference: dict, scope):
|
||||
"""The recorded output's own category (#332), inside the snapshot.
|
||||
|
||||
The path is the symlink-resolved output folder plus the receipt's
|
||||
subfolder and filename, keyed by ``image_category_key`` over the
|
||||
configured output sources as Images keys it. A temp output or a path
|
||||
no source holds has none. A failed read is OWN_CATEGORY_UNREAD.
|
||||
"""
|
||||
if (
|
||||
reference.get("folder_type") != "output"
|
||||
or scope.canonical_output is None
|
||||
or candidate_locator(reference, scope) is None
|
||||
):
|
||||
return None
|
||||
path = os.path.join(
|
||||
str(scope.canonical_output), reference["subfolder"], reference["filename"]
|
||||
)
|
||||
try:
|
||||
key = image_category_key(self._get_all_output_dirs(), path)
|
||||
return None if key is None else self.db.image_category(*key)
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
"History image category could not be read", exc_info=True
|
||||
)
|
||||
return OWN_CATEGORY_UNREAD
|
||||
|
||||
async def _history_current(
|
||||
self, request: web.Request, kind: str
|
||||
) -> web.StreamResponse:
|
||||
@@ -701,6 +737,7 @@ class HistoryRoutesMixin:
|
||||
if scope.current_scope != current_scope:
|
||||
raise _current_stale()
|
||||
result = db.get_history_occurrence(run_id, node, ordinal)
|
||||
own = None
|
||||
if result is not None:
|
||||
receipt = result["receipt"]
|
||||
path = candidate_locator(receipt["reference"], scope)
|
||||
@@ -708,6 +745,10 @@ class HistoryRoutesMixin:
|
||||
db.model.get_connection(),
|
||||
(path,) if path is not None else (),
|
||||
)
|
||||
if kind == "inspection":
|
||||
own = self._history_own_category(
|
||||
receipt["reference"], scope
|
||||
)
|
||||
finally:
|
||||
db.model.close()
|
||||
if result is None:
|
||||
@@ -726,9 +767,16 @@ class HistoryRoutesMixin:
|
||||
resolved.canonical_target,
|
||||
source_label=resolved.public["source_label"],
|
||||
saved=saved.get("saved"),
|
||||
library=saved.get(
|
||||
"library",
|
||||
{"status": "unavailable" if saved_unavailable else "none"},
|
||||
library=apply_own_category(
|
||||
saved.get(
|
||||
"library",
|
||||
{
|
||||
"status": (
|
||||
"unavailable" if saved_unavailable else "none"
|
||||
)
|
||||
},
|
||||
),
|
||||
own,
|
||||
),
|
||||
recorded=recorded_output_lookup(
|
||||
db, close=True, reference=receipt["reference"]
|
||||
|
||||
@@ -29,9 +29,11 @@ from .prompts import bad_request
|
||||
|
||||
try:
|
||||
from ...database.image_categories import MAX_CATEGORY_LENGTH, MAX_KEY_LENGTH
|
||||
from ...database.models import normalize_image_path
|
||||
from ...utils.validators import validate_category
|
||||
except ImportError:
|
||||
from database.image_categories import MAX_CATEGORY_LENGTH, MAX_KEY_LENGTH
|
||||
from database.models import normalize_image_path
|
||||
from utils.validators import validate_category
|
||||
|
||||
CATEGORY_MEDIA_MESSAGE = "Only media files can have a category"
|
||||
@@ -143,7 +145,7 @@ class ImageCategoryRoutesMixin:
|
||||
# -- reading ---------------------------------------------------------------
|
||||
|
||||
def _own_category_map(self, sources, listings) -> dict:
|
||||
"""``{root / listed path: own category}`` for the listed files (blocking).
|
||||
"""``{normalized listed path: own category}`` for the listed files (blocking).
|
||||
|
||||
One primary-key range read per source; each stored row is matched to
|
||||
the listing of every selected root that holds its source, so the work
|
||||
@@ -162,19 +164,30 @@ class ImageCategoryRoutesMixin:
|
||||
for _index, root, files in listings:
|
||||
listing = self._favorite_listing(root, files)
|
||||
root_key = _favorite_root_key(root)
|
||||
# normalize_image_path(root / listed) without a call per file: the
|
||||
# listed spellings are clean relative POSIX paths. Windows abspath
|
||||
# also drops a component's trailing dots and spaces, so it keeps
|
||||
# the call there.
|
||||
base = os.path.join(normalize_image_path(str(root)), "")
|
||||
|
||||
def normalized(listed, base=base, root=root):
|
||||
if os.name == "nt":
|
||||
return normalize_image_path(str(root / listed))
|
||||
return base + os.path.normcase(listed.replace("/", os.sep))
|
||||
|
||||
nested = [
|
||||
item for item in _nested_sources(root, sources) if item[1] != root_key
|
||||
]
|
||||
for relative, category in by_source.get(root_key, {}).items():
|
||||
listed = listing.get(relative)
|
||||
if listed is not None and _source_prefix(relative, nested) is None:
|
||||
found[root / listed] = category
|
||||
found[normalized(listed)] = category
|
||||
for prefix, source_key in nested:
|
||||
for relative, category in by_source.get(source_key, {}).items():
|
||||
full = f"{prefix}/{relative}"
|
||||
listed = listing.get(full)
|
||||
if listed is not None and _source_prefix(full, nested) == prefix:
|
||||
found[root / listed] = category
|
||||
found[normalized(listed)] = category
|
||||
return found
|
||||
|
||||
async def _source_listings(self, roots) -> list[tuple]:
|
||||
|
||||
+97
-25
@@ -66,6 +66,8 @@ try:
|
||||
from ...utils.parallel import map_parallel
|
||||
from ...utils.json_strict import dumps_strict
|
||||
from ...utils.inspection_response import (
|
||||
OWN_CATEGORY_UNREAD,
|
||||
apply_own_category,
|
||||
encode_inspection_response,
|
||||
library_metadata,
|
||||
)
|
||||
@@ -87,6 +89,8 @@ except ImportError:
|
||||
from utils.parallel import map_parallel
|
||||
from utils.json_strict import dumps_strict
|
||||
from utils.inspection_response import (
|
||||
OWN_CATEGORY_UNREAD,
|
||||
apply_own_category,
|
||||
encode_inspection_response,
|
||||
library_metadata,
|
||||
)
|
||||
@@ -565,7 +569,8 @@ class _ImageTagMatch(NamedTuple):
|
||||
|
||||
tags: bool # every filter tag is in its image tags or linked positive tags
|
||||
query: bool # ``q`` is a case-folded substring of one image tag name
|
||||
link: tuple | None # (hit, rating, positive id) of its unambiguous link
|
||||
# (hit, rating, positive id, inherited category) of its unambiguous link
|
||||
link: tuple | None
|
||||
|
||||
|
||||
def _current_fingerprint(path: Path) -> tuple[int, int] | None:
|
||||
@@ -1220,20 +1225,27 @@ def _admitted_entries(
|
||||
links: dict | None,
|
||||
favorites: frozenset | None = None,
|
||||
image_tags: dict | None = None,
|
||||
own_categories: dict | None = None,
|
||||
):
|
||||
"""Yield ``(position, entry, link match)`` for each entry the filters admit.
|
||||
|
||||
*links* maps normalized paths to (hit, rating, positive id) for every
|
||||
unambiguously linked image (see PromptDatabase.get_image_link_matches); a
|
||||
category or tag filter admits only those, and ``q`` admits a filename match
|
||||
or a linked-positive hit. *favorites*, when given, admits only those listed
|
||||
*links* maps normalized paths to (hit, rating, positive id, inherited
|
||||
category) for every unambiguously linked image (see
|
||||
PromptDatabase.get_image_link_matches); a tag filter admits only those,
|
||||
and ``q`` admits a filename match or a linked-positive hit. A category
|
||||
admits a file whose effective category equals it (#332 MC5): its own
|
||||
category (*own_categories*, ``{normalized path: category}``) when it has one,
|
||||
else the inherited one, so an unlinked file with its own category is
|
||||
admitted. *favorites*, when given, admits only those listed
|
||||
paths (``favorites=1``). *image_tags* maps normalized paths to their
|
||||
_ImageTagMatch: a tag filter also admits a file whose current image tags
|
||||
(with its linked positive's tags) carry every tag, and ``q`` also admits an
|
||||
image tag name hit.
|
||||
"""
|
||||
query, media, folder = filters["query"], filters["media"], filters["folder"]
|
||||
members_only = filters["category"] is not None or bool(filters["tags"])
|
||||
category = filters["category"]
|
||||
members_only = bool(filters["tags"])
|
||||
own = own_categories or {}
|
||||
native_folder = folder.replace("/", os.sep)
|
||||
nested = native_folder + os.sep
|
||||
exact = filters.get("path", "")
|
||||
@@ -1252,6 +1264,12 @@ def _admitted_entries(
|
||||
image = None if image_tags is None else image_tags.get(entry.normalized)
|
||||
if members_only and match is None and not (image and image.tags):
|
||||
continue
|
||||
if category is not None:
|
||||
effective = own.get(entry.normalized) if own else None
|
||||
if effective is None and match is not None:
|
||||
effective = match[3]
|
||||
if effective != category:
|
||||
continue
|
||||
if (
|
||||
query
|
||||
and query not in entry.folded
|
||||
@@ -1270,6 +1288,7 @@ def _filtered_page(
|
||||
links: dict | None,
|
||||
favorites: frozenset | None = None,
|
||||
image_tags: dict | None = None,
|
||||
own_categories: dict | None = None,
|
||||
) -> tuple[int, list, int | None]:
|
||||
"""Count the *entries* the filters admit and take *size* from *start* on.
|
||||
|
||||
@@ -1279,7 +1298,7 @@ def _filtered_page(
|
||||
"""
|
||||
total, page, positives = 0, [], set()
|
||||
for position, entry, match in _admitted_entries(
|
||||
entries, filters, links, favorites, image_tags
|
||||
entries, filters, links, favorites, image_tags, own_categories
|
||||
):
|
||||
total += 1
|
||||
if match is not None:
|
||||
@@ -2219,9 +2238,10 @@ class ImageRoutesMixin:
|
||||
stat'ed (never every listed file), and a candidate's tags count only
|
||||
while a byte record is verified at its present ``(size, mtime_ns)``
|
||||
(spec amendment S1). Under a Tags filter the candidates' linked
|
||||
positives are read too, so their tags and category combine.
|
||||
positives are read too, so their tags combine; the category is checked
|
||||
once, on the effective category, by _admitted_entries (#332).
|
||||
"""
|
||||
tags, query, category = filters["tags"], filters["query"], filters["category"]
|
||||
tags, query = filters["tags"], filters["query"]
|
||||
wanted = set(tags)
|
||||
located = self._workspace_locators(listings)
|
||||
# The tag index answers which files carry a relevant name (#379); no
|
||||
@@ -2251,11 +2271,9 @@ class ImageRoutesMixin:
|
||||
link = facts.get(entry.normalized)
|
||||
carried = set(names).union(link[4] if link else ())
|
||||
matches[entry.normalized] = _ImageTagMatch(
|
||||
bool(wanted)
|
||||
and wanted <= carried
|
||||
and (category is None or (link is not None and link[3] == category)),
|
||||
bool(wanted) and wanted <= carried,
|
||||
bool(query) and any(query in name.casefold() for name in names),
|
||||
link[:3] if link else None,
|
||||
tuple(link[:4]) if link else None,
|
||||
)
|
||||
return matches
|
||||
|
||||
@@ -2264,10 +2282,14 @@ class ImageRoutesMixin:
|
||||
) -> tuple:
|
||||
"""The ordered entries and what admits them (blocking; run in executor).
|
||||
|
||||
Returns ``(entries, key, links, image_tags, favorites)``: the listing in
|
||||
the filters' order, its keyset key, and the link, image-tag and favorite
|
||||
maps _admitted_entries reads. One SELECT (get_image_link_matches) reads
|
||||
every linked positive when *read_links*. A failed read raises when the
|
||||
Returns ``(entries, key, links, image_tags, favorites, own)``: the
|
||||
listing in the filters' order, its keyset key, and the link, image-tag,
|
||||
favorite and own-category maps _admitted_entries reads. One SELECT
|
||||
(get_image_link_matches) reads every linked positive when *read_links*;
|
||||
under a category it keeps only links whose raw recipe or positive
|
||||
category is that value (the inherited one is decided in Python), and
|
||||
the links of files whose own category is that value are read by path,
|
||||
so their rating and positive still count (#332). A failed read raises when the
|
||||
filters need it (category, tags, the rating sort) or *strict*; else the
|
||||
listing stays usable without links (a search then matches filenames
|
||||
only). A failed image-tag read raises under a Tags filter; a search then
|
||||
@@ -2279,10 +2301,10 @@ class ImageRoutesMixin:
|
||||
try:
|
||||
links = self.db.get_image_link_matches(
|
||||
filters["text"],
|
||||
category,
|
||||
tags,
|
||||
every_link=True,
|
||||
wanted=self._workspace_wanted(listings),
|
||||
category=category,
|
||||
)
|
||||
except Exception:
|
||||
if strict or category is not None or tags or order == "rating":
|
||||
@@ -2296,6 +2318,13 @@ class ImageRoutesMixin:
|
||||
),
|
||||
exc_info=True,
|
||||
)
|
||||
own = None
|
||||
if category is not None:
|
||||
# A failed read raises: an unfiltered page must never pass for the
|
||||
# category (DC6).
|
||||
own = self._own_category_map(self._get_all_output_dirs(), listings)
|
||||
if links is not None:
|
||||
links = self._own_category_links(links, own, filters)
|
||||
image_tags = None
|
||||
if filters["query"] or tags:
|
||||
try:
|
||||
@@ -2342,7 +2371,30 @@ class ImageRoutesMixin:
|
||||
favorites = (
|
||||
self._workspace_favorite_paths(listings) if filters["favorites"] else None
|
||||
)
|
||||
return entries, key, links, image_tags, favorites
|
||||
return entries, key, links, image_tags, favorites, own
|
||||
|
||||
def _own_category_links(self, links: dict, own: dict, filters: dict) -> dict:
|
||||
"""*links* plus the links of files whose own category is the filter's.
|
||||
|
||||
Their inherited category differs, so the category's link read left
|
||||
them out; read by path (get_image_link_facts), kept only when their
|
||||
positive has every filter tag, as the link read would.
|
||||
"""
|
||||
category, wanted = filters["category"], set(filters["tags"])
|
||||
missing = [
|
||||
path
|
||||
for path, value in own.items()
|
||||
if value == category and path not in links
|
||||
]
|
||||
if not missing:
|
||||
return links
|
||||
facts = self.db.get_image_link_facts(missing, filters["text"])
|
||||
extra = {
|
||||
path: tuple(fact[:4])
|
||||
for path, fact in facts.items()
|
||||
if wanted <= set(fact[4])
|
||||
}
|
||||
return {**links, **extra} if extra else links
|
||||
|
||||
def _select_workspace_page(
|
||||
self,
|
||||
@@ -2371,7 +2423,7 @@ class ImageRoutesMixin:
|
||||
or bool(filters["tags"])
|
||||
or order == "rating"
|
||||
)
|
||||
entries, key, links, image_tags, favorites = self._workspace_admission(
|
||||
entries, key, links, image_tags, favorites, own = self._workspace_admission(
|
||||
listings, filters, read_links=needed or after is None
|
||||
)
|
||||
start = 0
|
||||
@@ -2382,7 +2434,7 @@ class ImageRoutesMixin:
|
||||
else:
|
||||
start = bisect_right(entries, after, key=key)
|
||||
total, page, positive_count = _filtered_page(
|
||||
entries, start, limit + 1, filters, links, favorites, image_tags
|
||||
entries, start, limit + 1, filters, links, favorites, image_tags, own
|
||||
)
|
||||
if not needed and after is not None:
|
||||
positive_count = _COUNT_OMITTED
|
||||
@@ -2404,12 +2456,12 @@ class ImageRoutesMixin:
|
||||
|
||||
The links are read strictly: the positives must be the listing's own.
|
||||
"""
|
||||
entries, _key, links, image_tags, favorites = self._workspace_admission(
|
||||
entries, _key, links, image_tags, favorites, own = self._workspace_admission(
|
||||
listings, filters, read_links=True, strict=True
|
||||
)
|
||||
targets, positives, unlinked, scopes = [], {}, 0, {}
|
||||
for _position, entry, match in _admitted_entries(
|
||||
entries, filters, links, favorites, image_tags
|
||||
entries, filters, links, favorites, image_tags, own
|
||||
):
|
||||
if len(targets) == cap:
|
||||
return ListingSelection(targets, list(positives), unlinked, True)
|
||||
@@ -3728,13 +3780,20 @@ class ImageRoutesMixin:
|
||||
except Exception:
|
||||
# The link stays known; only its title is withheld.
|
||||
self.logger.warning("Inspection link title could not be read")
|
||||
# An output file's own category (#332); LoRA roots have none.
|
||||
own = (
|
||||
self._read_own_category(outputs, path)
|
||||
if index < len(outputs)
|
||||
else None
|
||||
)
|
||||
try:
|
||||
observed = inspect_current_file(
|
||||
path,
|
||||
source_label=_public_path(roots[index]),
|
||||
saved=saved,
|
||||
library=self._inspection_library(
|
||||
saved, unavailable=unavailable
|
||||
library=apply_own_category(
|
||||
self._inspection_library(saved, unavailable=unavailable),
|
||||
own,
|
||||
),
|
||||
image_tags=self._inspection_image_tags(
|
||||
roots[index], Path(relative).parts
|
||||
@@ -3802,6 +3861,19 @@ class ImageRoutesMixin:
|
||||
|
||||
return read
|
||||
|
||||
def _read_own_category(self, sources, path):
|
||||
"""One file's own category, None, or OWN_CATEGORY_UNREAD (blocking).
|
||||
|
||||
Keyed by ``image_category_key`` over the output sources, as Images
|
||||
writes it. A failed read is logged and hides the category (DC5).
|
||||
"""
|
||||
try:
|
||||
key = image_category_key(sources, path)
|
||||
return None if key is None else self.db.image_category(*key)
|
||||
except Exception:
|
||||
self.logger.warning("Image category could not be read", exc_info=True)
|
||||
return OWN_CATEGORY_UNREAD
|
||||
|
||||
def _inspection_library(self, saved: dict | None, *, unavailable: bool) -> dict:
|
||||
"""The inspector's Library facts; the route sets the title afterwards."""
|
||||
result = {
|
||||
|
||||
@@ -383,3 +383,32 @@ test('the D117 wrap rule targets only the settings summary button', async () =>
|
||||
assert.deepEqual(rules.map(line => line.trim()), ['.ca-tools #viewer-details .settings-summary { white-space: normal; /* D117 */ }']);
|
||||
assert.doesNotMatch(css, /#viewer-details > div > button/);
|
||||
});
|
||||
|
||||
test('category chip names its origin and shows an own category for any link status (#332)', async () => {
|
||||
const base = await localRecord(localPng('a.png'));
|
||||
const chipTexts = record => {
|
||||
const {f, done} = renderFull(record);
|
||||
try {
|
||||
return f.walk(f.container, n => /(^| )chip( |$)/.test(n.className ?? '')).map(n => n.textContent);
|
||||
} finally { done(); }
|
||||
};
|
||||
const cases = [
|
||||
[{status:'none', category:'Scans', category_scope:'image'}, 'Image category: Scans'],
|
||||
[{status:'ambiguous', category:'Scans', category_scope:'image'}, 'Image category: Scans'],
|
||||
[{status:'linked', category:'Portraits', category_scope:'image'}, 'Image category: Portraits'],
|
||||
[{status:'linked', category:'Landscapes', category_scope:'positive'}, 'Positive prompt category: Landscapes'],
|
||||
[{status:'linked', category:'Recipes', category_scope:'pair'}, 'Recipe category: Recipes'],
|
||||
];
|
||||
for (const [library, text] of cases) {
|
||||
const chips = chipTexts({...base, library:{...base.library, tags:[], tags_scope:null, ...library}});
|
||||
assert.ok(chips.includes(text), `${JSON.stringify(library)} -> ${chips}`);
|
||||
}
|
||||
for (const library of [
|
||||
{status:'none', category:'Inherited', category_scope:'positive'},
|
||||
{status:'linked', category:null, category_scope:null},
|
||||
{status:'none', category:null, category_scope:null},
|
||||
]) {
|
||||
const chips = chipTexts({...base, library:{...base.library, tags:[], tags_scope:null, ...library}});
|
||||
assert.equal(chips.some(chip => /category: /.test(chip)), false, JSON.stringify(library));
|
||||
}
|
||||
});
|
||||
|
||||
@@ -278,6 +278,7 @@ test('record rejects root path media and invalid versioned fields with retryable
|
||||
data => { data.resolution.sampler_id = {}; },
|
||||
data => { data.library.tags_scope = 'recipe'; },
|
||||
data => { data.library.category_scope = 'recipe'; },
|
||||
data => { data.library.tags_scope = 'image'; },
|
||||
data => { delete data.library.category_scope; },
|
||||
data => { delete data.library.tags_scope; },
|
||||
]) {
|
||||
@@ -942,3 +943,14 @@ test("names the server quotes (parentheses, '!*') still match their original and
|
||||
assert.equal(record.download.url, entry.url);
|
||||
assert.deepEqual(record.preview, { url: entry.thumbnail_url });
|
||||
});
|
||||
|
||||
test('record accepts an image\'s own category on any link status (#332)', () => {
|
||||
const toRecord = contract('toInspectionRecord');
|
||||
for (const status of ['linked', 'none', 'ambiguous', 'unavailable']) {
|
||||
const data = envelope();
|
||||
Object.assign(data.library, { status, category: 'Scans', category_scope: 'image' });
|
||||
const record = toRecord(data, image(), extractGenerationFacts);
|
||||
assert.equal(record.library.category, 'Scans');
|
||||
assert.equal(record.library.category_scope, 'image');
|
||||
}
|
||||
});
|
||||
|
||||
@@ -62,12 +62,16 @@ class TestVocabularyAPI(ImageAPITestCase):
|
||||
},
|
||||
)
|
||||
self.assertEqual(
|
||||
categories["categories"], [{"value": "Animal", "prompt_count": 1}]
|
||||
categories["categories"],
|
||||
[{"value": "Animal", "prompt_count": 1, "image_count": 0}],
|
||||
)
|
||||
_, rest = await self.get(
|
||||
"categories", limit=1, cursor=categories["next_cursor"]
|
||||
)
|
||||
self.assertEqual(rest["categories"], [{"value": "Pet", "prompt_count": 1}])
|
||||
self.assertEqual(
|
||||
rest["categories"],
|
||||
[{"value": "Pet", "prompt_count": 1, "image_count": 0}],
|
||||
)
|
||||
self.assertFalse(rest["has_more"])
|
||||
|
||||
async def test_vocabulary_cursor_is_signed_epoch_scoped(self):
|
||||
@@ -488,6 +492,7 @@ class TestVocabularyAPI(ImageAPITestCase):
|
||||
"other_prompts": 0,
|
||||
"negative_variants": 1,
|
||||
"archived_negative_variants": 0,
|
||||
"images": 0,
|
||||
}
|
||||
self.assertEqual(
|
||||
preview,
|
||||
@@ -528,6 +533,73 @@ class TestVocabularyAPI(ImageAPITestCase):
|
||||
self.assertEqual(response.status, 409)
|
||||
self.assertEqual(replay["code"], "vocabulary-conflict")
|
||||
|
||||
async def test_category_rows_report_images_and_image_only_values(self):
|
||||
self._save_prompt("Owner", category="Prompt value")
|
||||
_, options_before = await self.get_options()
|
||||
self.api.db.set_image_categories(
|
||||
[("root", "a.png"), ("root", "b.png")], "Prompt value"
|
||||
)
|
||||
self.api.db.set_image_categories([("root", "c.png")], "Image value")
|
||||
response, first = await self.get("categories", limit=1)
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(
|
||||
first["categories"],
|
||||
[{"value": "Prompt value", "prompt_count": 1, "image_count": 2}],
|
||||
)
|
||||
self.assertEqual(first["total"], 2)
|
||||
_, rest = await self.get("categories", limit=1, cursor=first["next_cursor"])
|
||||
self.assertEqual(
|
||||
rest["categories"],
|
||||
[{"value": "Image value", "prompt_count": 0, "image_count": 1}],
|
||||
)
|
||||
self.assertFalse(rest["has_more"])
|
||||
# The Prompt library's category list does not gain image values.
|
||||
_, options_after = await self.get_options()
|
||||
self.assertEqual(options_after, options_before)
|
||||
self.assertEqual(options_after["categories"], ["Prompt value"])
|
||||
|
||||
async def get_options(self):
|
||||
response = await self.client.get("/prompt_manager/library/options")
|
||||
return response, await response.json()
|
||||
|
||||
async def test_category_routes_rename_image_only_value(self):
|
||||
epoch = self.api.db.get_execution_epoch()
|
||||
self._save_prompt("Bystander", category="Kept")
|
||||
self.api.db.set_image_categories(
|
||||
[("root", "a.png"), ("second", "b.png")], "Image value"
|
||||
)
|
||||
prompts_before = self.category_rows()
|
||||
body = {"expected_epoch": epoch, "source": "Image value", "target": "Moved"}
|
||||
response, preview = await self.send("POST", "categories/preview", body)
|
||||
self.assertEqual(response.status, 200, preview)
|
||||
impact = {
|
||||
"positive_prompts": 0,
|
||||
"other_prompts": 0,
|
||||
"negative_variants": 0,
|
||||
"archived_negative_variants": 0,
|
||||
"images": 2,
|
||||
}
|
||||
self.assertEqual(preview["impact"], impact)
|
||||
stale = preview["impact_token"]
|
||||
self.api.db.set_image_categories([("root", "late.png")], "Image value")
|
||||
response, refused = await self.send(
|
||||
"POST", "categories/apply", {**body, "impact_token": stale}
|
||||
)
|
||||
self.assertEqual(response.status, 409)
|
||||
self.assertEqual(refused["code"], "vocabulary-conflict")
|
||||
self.assertEqual(self.api.db.image_category("root", "a.png"), "Image value")
|
||||
_, preview = await self.send("POST", "categories/preview", body)
|
||||
response, applied = await self.send(
|
||||
"POST",
|
||||
"categories/apply",
|
||||
{**body, "impact_token": preview["impact_token"]},
|
||||
)
|
||||
self.assertEqual(response.status, 200, applied)
|
||||
self.assertEqual(applied["impact"], {**impact, "images": 3})
|
||||
for key in (("root", "a.png"), ("second", "b.png"), ("root", "late.png")):
|
||||
self.assertEqual(self.api.db.image_category(*key), "Moved")
|
||||
self.assertEqual(self.category_rows(), prompts_before)
|
||||
|
||||
async def test_preview_validation(self):
|
||||
epoch = self.api.db.get_execution_epoch()
|
||||
self._save_prompt("Owner", category="Scenery")
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
"""Images filters by the effective category (#332 MC4, MC5, DC6).
|
||||
|
||||
An image's effective category is its own category when it has one, else the
|
||||
category its link supplies (the recipe's when set, else the positive's), else
|
||||
none. The category filter, "select all N matching" and saved views match it.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
import test_api_workspace_images as workspace
|
||||
from test_image_categories import category_checksum
|
||||
except ImportError:
|
||||
from tests import test_api_workspace_images as workspace
|
||||
from tests.test_image_categories import category_checksum
|
||||
|
||||
from py.api.images import image_category_key, parse_listing_filter
|
||||
|
||||
|
||||
class WorkspaceCategoryCase(workspace.ImageAPITestCase):
|
||||
"""Listing helpers plus own-category writes keyed as the routes key them."""
|
||||
|
||||
media = workspace.TestWorkspaceImages.media
|
||||
get = workspace.TestWorkspaceImages.get
|
||||
tag = workspace.TestWorkspaceImageTags.tag
|
||||
names = workspace.TestWorkspaceImageTags.names
|
||||
|
||||
async def tearDownAsync(self):
|
||||
self.api.db.close_all()
|
||||
workspace.assert_closed(self, self._temp_db.name, self._output_tmp.name)
|
||||
await super().tearDownAsync()
|
||||
|
||||
def own(self, path, value, sources=None):
|
||||
sources = sources or self.api._get_all_output_dirs()
|
||||
self.api.db.set_image_categories([image_category_key(sources, path)], value)
|
||||
|
||||
async def listed(self, **params):
|
||||
response, body = await self.get(sort="name", **params)
|
||||
self.assertEqual(response.status, 200, body)
|
||||
return self.names(body), body
|
||||
|
||||
|
||||
class TestEffectiveCategoryFilter(WorkspaceCategoryCase):
|
||||
def batch(self):
|
||||
"""Four images of "castle at dusk" (Landscapes); one is overridden."""
|
||||
castle = self._save_prompt("castle at dusk", category="Landscapes", rating=4)
|
||||
files = [self.media(f"castle_{n}.png", mtime=100 + n) for n in range(4)]
|
||||
for path in files:
|
||||
self.api.db.link_image_to_prompt(castle, str(path))
|
||||
self.own(files[0], "Portraits")
|
||||
return castle, files
|
||||
|
||||
async def test_category_filter_prefers_own_category(self):
|
||||
castle, _files = self.batch()
|
||||
conn = self.api.db.model.get_connection()
|
||||
before = category_checksum(conn)
|
||||
names, body = await self.listed(category="Portraits")
|
||||
self.assertEqual(names, ["castle_0.png"])
|
||||
# The caption still counts the prompt behind the image shown.
|
||||
self.assertEqual(body["positive_count"], 1)
|
||||
names, body = await self.listed(category="Landscapes")
|
||||
self.assertEqual(names, ["castle_1.png", "castle_2.png", "castle_3.png"])
|
||||
self.assertEqual(body["total"], 3)
|
||||
names, _ = await self.listed()
|
||||
self.assertEqual(len(names), 4)
|
||||
self.assertEqual(category_checksum(conn), before)
|
||||
self.assertEqual(self.api.db.get_prompt_by_id(castle)["category"], "Landscapes")
|
||||
|
||||
async def test_category_filter_lists_unlinked_with_own(self):
|
||||
scans = [self.media(f"scan_{n}.png", mtime=100 + n) for n in range(2)]
|
||||
self.media("plain.png", mtime=50)
|
||||
for path in scans:
|
||||
self.own(path, "Scans")
|
||||
names, body = await self.listed(category="Scans")
|
||||
self.assertEqual(names, ["scan_0.png", "scan_1.png"])
|
||||
self.assertEqual(body["positive_count"], 0)
|
||||
|
||||
async def test_recipe_category_is_inherited(self):
|
||||
"""MC4: a recipe's own category, not the positive's, is inherited."""
|
||||
db = self.api.db
|
||||
self._save_prompt("Pair positive", category="Night")
|
||||
pair = db.save_prompt_pair("Pair positive", "Noise", category="Recipes")
|
||||
blank = db.save_prompt_pair("Pair positive", "Other noise", category=" ")
|
||||
recipe = self.media("recipe.png", mtime=103)
|
||||
blank_recipe = self.media("blank_recipe.png", mtime=102)
|
||||
positive = db.get_prompt_pair(pair)["positive_prompt_id"]
|
||||
for path, pair_id in ((recipe, pair), (blank_recipe, blank)):
|
||||
image = db.link_image_to_prompt(positive, str(path))
|
||||
db.link_image_to_pair(pair_id, image, {"source": "fixture"})
|
||||
self.assertEqual((await self.listed(category="Recipes"))[0], ["recipe.png"])
|
||||
# A blank recipe category falls back to the positive's.
|
||||
self.assertEqual((await self.listed(category="Night"))[0], ["blank_recipe.png"])
|
||||
# An own category still wins over the recipe's.
|
||||
self.own(recipe, "Mine")
|
||||
self.assertEqual((await self.listed(category="Recipes"))[0], [])
|
||||
self.assertEqual((await self.listed(category="Mine"))[0], ["recipe.png"])
|
||||
|
||||
async def test_ambiguous_files_inherit_nothing_but_keep_their_own(self):
|
||||
first = self._save_prompt("Owner one", category="Night")
|
||||
second = self._save_prompt("Owner two", category="Night")
|
||||
ambiguous = self.media("ambiguous.png")
|
||||
for owner in (first, second):
|
||||
self.api.db.link_image_to_prompt(owner, str(ambiguous))
|
||||
self.assertEqual((await self.listed(category="Night"))[0], [])
|
||||
self.own(ambiguous, "Night")
|
||||
self.assertEqual((await self.listed(category="Night"))[0], ["ambiguous.png"])
|
||||
|
||||
async def test_combines_with_tags_search_and_rating(self):
|
||||
stone = self._save_prompt(
|
||||
"castle at dusk", category="Landscapes", tags=["stone"], rating=4
|
||||
)
|
||||
files = [self.media(f"castle_{n}.png", mtime=100 + n) for n in range(2)]
|
||||
for path in files:
|
||||
self.api.db.link_image_to_prompt(stone, str(path))
|
||||
self.own(files[0], "Portraits")
|
||||
other = self._save_prompt("harbor at noon", category="Portraits", rating=2)
|
||||
harbor = self.media("harbor.png", mtime=90)
|
||||
self.api.db.link_image_to_prompt(other, str(harbor))
|
||||
names, _ = await self.listed(category="Portraits", tags="stone")
|
||||
self.assertEqual(names, ["castle_0.png"])
|
||||
names, _ = await self.listed(category="Portraits", q="harbor")
|
||||
self.assertEqual(names, ["harbor.png"])
|
||||
names, _ = await self.listed(category="Portraits", q="castle_0")
|
||||
self.assertEqual(names, ["castle_0.png"])
|
||||
response, body = await self.get(category="Portraits", sort="rating")
|
||||
self.assertEqual(response.status, 200, body)
|
||||
self.assertEqual(self.names(body), ["castle_0.png", "harbor.png"])
|
||||
self.assertEqual(body["positive_count"], 2)
|
||||
|
||||
async def test_image_tag_admitted_file_keeps_its_inherited_category(self):
|
||||
night = self._save_prompt("Night owl", category="Night", tags=["a"])
|
||||
tagged = self.media("tagged.png", mtime=103)
|
||||
unlinked = self.media("unlinked.png", mtime=102)
|
||||
self.api.db.link_image_to_prompt(night, str(tagged))
|
||||
self.tag(tagged, ["b"])
|
||||
self.tag(unlinked, ["a", "b"])
|
||||
names, _ = await self.listed(tags="a,b", category="Night")
|
||||
self.assertEqual(names, ["tagged.png"])
|
||||
self.own(unlinked, "Night")
|
||||
names, _ = await self.listed(tags="a,b", category="Night")
|
||||
self.assertEqual(names, ["tagged.png", "unlinked.png"])
|
||||
|
||||
async def test_cursor_pages_walk_only_the_category(self):
|
||||
files = [self.media(f"img_{n:02}.png", mtime=100 + n) for n in range(7)]
|
||||
for path in files[::2]:
|
||||
self.own(path, "Even")
|
||||
seen, cursor = [], None
|
||||
while True:
|
||||
params = {"category": "Even", "limit": 2, "sort": "name"}
|
||||
if cursor:
|
||||
params["cursor"] = cursor
|
||||
response, body = await self.get(**params)
|
||||
self.assertEqual(response.status, 200, body)
|
||||
seen += self.names(body)
|
||||
self.assertEqual(body["total"], 4)
|
||||
cursor = body["next_cursor"]
|
||||
if not cursor:
|
||||
break
|
||||
self.assertEqual(seen, ["img_00.png", "img_02.png", "img_04.png", "img_06.png"])
|
||||
|
||||
async def test_an_outer_source_lists_a_nested_sources_own_category(self):
|
||||
inner = self.output_dir / "nested"
|
||||
nested = self.media("nested/a.png")
|
||||
self.media("b.png")
|
||||
self.api._get_all_output_dirs = lambda: [self.output_dir, inner]
|
||||
self.own(nested, "Scans")
|
||||
names, _ = await self.listed(category="Scans", root="0")
|
||||
self.assertEqual(names, ["nested/a.png"])
|
||||
names, _ = await self.listed(category="Scans", root="1")
|
||||
self.assertEqual(names, ["a.png"])
|
||||
# The outer source's own key for that path is not the file's key.
|
||||
from py.api.images import _favorite_root_key
|
||||
|
||||
self.api.db.clear_image_categories([image_category_key([inner], nested)])
|
||||
self.api.db.set_image_categories(
|
||||
[(_favorite_root_key(self.output_dir), "nested/a.png")], "Stale"
|
||||
)
|
||||
self.assertEqual((await self.listed(category="Stale"))[0], [])
|
||||
|
||||
async def test_a_gone_files_category_is_not_listed(self):
|
||||
gone = self.media("gone.png")
|
||||
self.own(gone, "Scans")
|
||||
os.unlink(gone)
|
||||
self.api.invalidate_gallery_cache()
|
||||
self.assertEqual((await self.listed(category="Scans"))[0], [])
|
||||
|
||||
async def test_a_failed_own_category_read_is_listing_failed(self):
|
||||
self.media("a.png")
|
||||
|
||||
def broken(*_args, **_kwargs):
|
||||
raise sqlite3.OperationalError("secret detail")
|
||||
|
||||
with patch.object(self.api.db, "own_categories_of", side_effect=broken):
|
||||
with self.assertLogs("prompt_manager", level="ERROR"):
|
||||
response, body = await self.get(category="Scans")
|
||||
self.assertEqual(response.status, 500)
|
||||
self.assertNotIn("secret", str(body))
|
||||
# Without a category nothing new is read.
|
||||
response, body = await self.get()
|
||||
self.assertEqual(response.status, 200, body)
|
||||
|
||||
async def test_select_all_matching_uses_the_effective_category(self):
|
||||
_castle, files = self.batch()
|
||||
resolved = await self.api.resolve_listing_filter(
|
||||
parse_listing_filter({"category": "Portraits"}), 100
|
||||
)
|
||||
self.assertEqual([t[2] for t in resolved.targets], ["castle_0.png"])
|
||||
resolved = await self.api.resolve_listing_filter(
|
||||
parse_listing_filter({"category": "Landscapes"}), 100
|
||||
)
|
||||
self.assertEqual(len(resolved.targets), 3)
|
||||
self.assertEqual(len(files), 4)
|
||||
|
||||
async def test_saved_view_category_means_the_effective_category(self):
|
||||
"""CUJ 8: a saved view's category is the listing's query value."""
|
||||
_castle, files = self.batch()
|
||||
self.own(files[1], "Portraits")
|
||||
names, _ = await self.listed(category="Portraits")
|
||||
self.assertEqual(names, ["castle_0.png", "castle_1.png"])
|
||||
|
||||
|
||||
class TestFilterAndChipAgreeOnBlanks(WorkspaceCategoryCase):
|
||||
"""The filter and the inspector use one inherited-category rule."""
|
||||
|
||||
async def chip(self, path):
|
||||
response = await self.client.get(
|
||||
f"/prompt_manager/images/inspect/{path}", params={"root": "0"}
|
||||
)
|
||||
body = await response.json()
|
||||
self.assertEqual(response.status, 200, body)
|
||||
return body["library"]["category"]
|
||||
|
||||
async def test_blank_recipe_values_agree(self):
|
||||
db = self.api.db
|
||||
positive = self._save_prompt("Pair positive", category="Night")
|
||||
blanks = {"spaces": " ", "tab": "\t", "nbsp": " ", "set": "Recipes"}
|
||||
for index, (label, value) in enumerate(blanks.items()):
|
||||
pair = db.save_prompt_pair("Pair positive", f"noise {label}")
|
||||
path = self.media(f"{label}.png", mtime=100 + index)
|
||||
image = db.link_image_to_prompt(positive, str(path))
|
||||
db.link_image_to_pair(pair, image, {"source": "fixture"})
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE prompt_pairs SET category=? WHERE id=?", (value, pair)
|
||||
)
|
||||
blob = db.save_prompt_pair("Pair positive", "noise blob")
|
||||
path = self.media("blob.png", mtime=90)
|
||||
image = db.link_image_to_prompt(positive, str(path))
|
||||
db.link_image_to_pair(blob, image, {"source": "fixture"})
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE prompt_pairs SET category=CAST(' ' AS BLOB) WHERE id=?",
|
||||
(blob,),
|
||||
)
|
||||
for name in ("spaces", "tab", "nbsp", "set", "blob"):
|
||||
with self.subTest(name=name):
|
||||
shown = await self.chip(f"{name}.png")
|
||||
for value in ("Night", "Recipes"):
|
||||
listed = f"{name}.png" in (await self.listed(category=value))[0]
|
||||
self.assertEqual(listed, shown == value, (name, value, shown))
|
||||
@@ -1347,7 +1347,7 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
_, data = await self.get(q="harbor")
|
||||
self.assertEqual(self._names(data), ["night_a.png"])
|
||||
self.assertEqual(
|
||||
self.api.db.get_image_paths_matching("zebra", None, []),
|
||||
self.api.db.get_image_paths_matching("zebra", []),
|
||||
{
|
||||
normalize_image_path(str(files[name]))
|
||||
for name in ("night_ab", "day_ab", "titled")
|
||||
@@ -1407,8 +1407,10 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
for query in statements
|
||||
if query.lstrip().upper().startswith(("SELECT", "WITH"))
|
||||
]
|
||||
# Plus the one stream of tagged image records q and tags read (B3).
|
||||
self.assertEqual(len(selects), 4, selects)
|
||||
# Plus the one stream of tagged image records q and tags read (B3),
|
||||
# and the category's own-category read of the one source (#332).
|
||||
self.assertEqual(len(selects), 5, selects)
|
||||
self.assertEqual(sum("image_categories" in q for q in selects), 1, selects)
|
||||
self.assertEqual(sum("image_favorites" in q for q in selects), 1, selects)
|
||||
self.assertEqual(sum("media_image_tags" in q for q in selects), 1, selects)
|
||||
|
||||
@@ -1528,15 +1530,16 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
elsewhere = self.api.db.get_prompt_id_for_image("/elsewhere/7.png")
|
||||
self.assertEqual(
|
||||
self.api.db.get_image_link_matches(
|
||||
"", "Night", [], every_link=True, wanted=wanted
|
||||
"", [], every_link=True, wanted=wanted, category="Night"
|
||||
),
|
||||
{
|
||||
normalize_image_path(str(files["night_ab"])): (
|
||||
True,
|
||||
5,
|
||||
self.api.db.get_prompt_id_for_image(str(files["night_ab"])),
|
||||
"Night",
|
||||
),
|
||||
"/elsewhere/7.png": (True, 5, elsewhere),
|
||||
"/elsewhere/7.png": (True, 5, elsewhere, "Night"),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1588,6 +1591,24 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
for i in range(links)
|
||||
],
|
||||
)
|
||||
# 2,000 images (one in 25 at 50,000 files) carry their own category
|
||||
# (#332), so a category filter also reads and matches those.
|
||||
from py.api.images import _favorite_root_key
|
||||
|
||||
root_key = _favorite_root_key(self.output_dir)
|
||||
with self.api.db.model.get_connection() as conn:
|
||||
conn.executemany(
|
||||
"INSERT INTO image_categories VALUES(?,?,?,?)",
|
||||
[
|
||||
(
|
||||
root_key,
|
||||
listed[i][0].relative_to(self.output_dir).as_posix(),
|
||||
f"cat-{i % 10}",
|
||||
"2026-10-27T00:00:00+00:00",
|
||||
)
|
||||
for i in range(1, files, 25)[:2_000]
|
||||
],
|
||||
)
|
||||
return listed
|
||||
|
||||
_BUDGET_REQUESTS = (
|
||||
@@ -1678,7 +1699,13 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
# q and tags also stream the tagged image records once
|
||||
# (#249 B3); none are tagged here, so nothing is stat'ed.
|
||||
streamed = "q" in params or "tags" in params
|
||||
self.assertEqual(len(found), 2 if streamed else 1, found)
|
||||
# A category also reads the own categories, one primary
|
||||
# key range per source (#332).
|
||||
own = "category" in params
|
||||
self.assertEqual(len(found), 1 + streamed + own, found)
|
||||
self.assertEqual(
|
||||
sum("image_categories" in query for query in found), int(own)
|
||||
)
|
||||
self.assertEqual(
|
||||
sum("media_image_tags" in query for query in found),
|
||||
int(streamed),
|
||||
@@ -1739,7 +1766,7 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
self.assertEqual(self.api._workspace_order_cache, {})
|
||||
# The per-row ambiguity probe is an index search, never a scan.
|
||||
# The read runs it wrapped in one JSON aggregate (#354): same plan.
|
||||
sql, params = link_filter_query("lantern", "cat-3", ["tag-1"], every_link=True)
|
||||
sql, params = link_filter_query("lantern", ["tag-1"], every_link=True)
|
||||
for query in (sql, link_matches_json_query(sql)):
|
||||
with self.subTest(wrapped=query is not sql):
|
||||
with self.api.db.model.get_connection() as conn:
|
||||
@@ -1755,6 +1782,25 @@ class TestWorkspaceImages(ImageAPITestCase):
|
||||
self.assertFalse(
|
||||
[step for step in plan if step.startswith("SCAN other")]
|
||||
)
|
||||
# The category's narrowing clause (#332) keeps every join an index
|
||||
# search, plain and wrapped with the raw categories.
|
||||
sql, params = link_filter_query(
|
||||
"", ["tag-1"], every_link=True, category="cat-3"
|
||||
)
|
||||
for query in (sql, link_matches_json_query(sql, categories=True)):
|
||||
with self.subTest(category_wrapped=query is not sql):
|
||||
with self.api.db.model.get_connection() as conn:
|
||||
plan = [
|
||||
row[3]
|
||||
for row in conn.execute("EXPLAIN QUERY PLAN " + query, params)
|
||||
]
|
||||
self.assertIn(
|
||||
"SEARCH other USING COVERING INDEX idx_image_file_path "
|
||||
"(file_path=?)",
|
||||
plan,
|
||||
)
|
||||
scans = [step for step in plan if step.startswith("SCAN")]
|
||||
self.assertTrue(all(step.startswith("SCAN gi") for step in scans), plan)
|
||||
|
||||
@unittest.skipUnless(
|
||||
PERF_BUDGET and not TRACED,
|
||||
|
||||
@@ -143,9 +143,13 @@ class LinkMatchesTestCase(unittest.TestCase):
|
||||
|
||||
def streamed(self, q, category, tags, every_link=False, wanted=None):
|
||||
"""Today's read, called directly (the fallback)."""
|
||||
sql, params = link_filter_query(q, category, tags, every_link=every_link)
|
||||
sql, params = link_filter_query(
|
||||
q, tags, every_link=every_link, category=category
|
||||
)
|
||||
with self.db.model.get_connection() as conn:
|
||||
return operations._streamed_link_matches(conn, sql, params, wanted)
|
||||
return operations._streamed_link_matches(
|
||||
conn, sql, params, wanted, category is not None
|
||||
)
|
||||
|
||||
def selects(self):
|
||||
found = [s for s in self.statements if s.lstrip().upper().startswith("SELECT")]
|
||||
@@ -156,17 +160,17 @@ class LinkMatchesTestCase(unittest.TestCase):
|
||||
class TestOneStepLinkRead(LinkMatchesTestCase):
|
||||
def test_plain_read_returns_every_unambiguous_link(self):
|
||||
self.seed()
|
||||
matches = self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
matches = self.db.get_image_link_matches("", [], every_link=True)
|
||||
self.assertEqual(
|
||||
matches,
|
||||
{
|
||||
"/out/a.png": (True, 5, 1),
|
||||
QUOTED: (True, None, 2),
|
||||
BACKSLASHED: (True, None, 2),
|
||||
UNICODE: (True, 3, 3),
|
||||
NEWLINE: (True, 5, 1),
|
||||
"/out/pair.png": (True, 4, 4),
|
||||
"/out/two-pairs.png": (True, 3, 3),
|
||||
"/out/a.png": (True, 5, 1, None),
|
||||
QUOTED: (True, None, 2, None),
|
||||
BACKSLASHED: (True, None, 2, None),
|
||||
UNICODE: (True, 3, 3, None),
|
||||
NEWLINE: (True, 5, 1, None),
|
||||
"/out/pair.png": (True, 4, 4, None),
|
||||
"/out/two-pairs.png": (True, 3, 3, None),
|
||||
},
|
||||
)
|
||||
self.assertIsNone(matches[QUOTED][1])
|
||||
@@ -176,21 +180,21 @@ class TestOneStepLinkRead(LinkMatchesTestCase):
|
||||
def test_search_hits_follow_text_title_and_tags(self):
|
||||
self.seed()
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches("lantern", None, []),
|
||||
self.db.get_image_link_matches("lantern", []),
|
||||
{
|
||||
"/out/a.png": (True, 5, 1),
|
||||
NEWLINE: (True, 5, 1),
|
||||
UNICODE: (True, 3, 3),
|
||||
"/out/two-pairs.png": (True, 3, 3),
|
||||
"/out/a.png": (True, 5, 1, None),
|
||||
NEWLINE: (True, 5, 1, None),
|
||||
UNICODE: (True, 3, 3, None),
|
||||
"/out/two-pairs.png": (True, 3, 3, None),
|
||||
},
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches("lantern", "cat-a", []),
|
||||
self.db.get_image_link_matches("lantern", [], category="cat-a"),
|
||||
{
|
||||
"/out/a.png": (True, 5, 1),
|
||||
NEWLINE: (True, 5, 1),
|
||||
QUOTED: (False, None, 2),
|
||||
BACKSLASHED: (False, None, 2),
|
||||
"/out/a.png": (True, 5, 1, "cat-a"),
|
||||
NEWLINE: (True, 5, 1, "cat-a"),
|
||||
QUOTED: (False, None, 2, "cat-a"),
|
||||
BACKSLASHED: (False, None, 2, "cat-a"),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -206,7 +210,11 @@ class TestOneStepLinkRead(LinkMatchesTestCase):
|
||||
expected = self.streamed(q, category, tags, every_link, wanted)
|
||||
self.statements.clear()
|
||||
actual = self.db.get_image_link_matches(
|
||||
q, category, list(tags), every_link=every_link, wanted=wanted
|
||||
q,
|
||||
list(tags),
|
||||
every_link=every_link,
|
||||
wanted=wanted,
|
||||
category=category,
|
||||
)
|
||||
self.assertEqual(actual, expected)
|
||||
self.assertEqual(
|
||||
@@ -218,10 +226,8 @@ class TestOneStepLinkRead(LinkMatchesTestCase):
|
||||
self.assertIn("json_group_array", found[0])
|
||||
|
||||
def test_empty_database_returns_nothing_in_one_statement(self):
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches("", None, [], every_link=True), {}
|
||||
)
|
||||
self.assertEqual(self.db.get_image_link_matches("x", "c", ["t"]), {})
|
||||
self.assertEqual(self.db.get_image_link_matches("", [], every_link=True), {})
|
||||
self.assertEqual(self.db.get_image_link_matches("x", ["t"], category="c"), {})
|
||||
found = self.selects()
|
||||
self.assertEqual(len(found), 2, found)
|
||||
self.assertTrue(all("json_group_array" in query for query in found))
|
||||
@@ -230,20 +236,51 @@ class TestOneStepLinkRead(LinkMatchesTestCase):
|
||||
self.seed()
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches(
|
||||
"", None, [], every_link=True, wanted={"/out/pair.png", "/x.png"}
|
||||
"", [], every_link=True, wanted={"/out/pair.png", "/x.png"}
|
||||
),
|
||||
{"/out/pair.png": (True, 4, 4)},
|
||||
{"/out/pair.png": (True, 4, 4, None)},
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches("", None, [], every_link=True, wanted=set()),
|
||||
self.db.get_image_link_matches("", [], every_link=True, wanted=set()),
|
||||
{},
|
||||
)
|
||||
|
||||
def test_category_read_carries_the_inherited_category_in_one_step(self):
|
||||
# A recipe's own category wins over its positive's (#332 MC4); the
|
||||
# read only narrows to links holding the value, the caller compares.
|
||||
self.seed()
|
||||
with self.db.model.get_connection() as conn:
|
||||
conn.execute("UPDATE prompt_pairs SET category='cat-a' WHERE id=1")
|
||||
self.statements.clear()
|
||||
for category, expected in (
|
||||
("cat-a", {"/out/pair.png": (True, 4, 4, "cat-a")}),
|
||||
("cat-b", {"/out/pair.png": (True, 4, 4, "cat-a")}),
|
||||
):
|
||||
with self.subTest(category=category):
|
||||
actual = self.db.get_image_link_matches(
|
||||
"", [], every_link=True, category=category, wanted={"/out/pair.png"}
|
||||
)
|
||||
self.assertEqual(actual, expected)
|
||||
self.assertEqual(
|
||||
actual, self.streamed("", category, [], True, {"/out/pair.png"})
|
||||
)
|
||||
found = [query for query in self.selects() if "json_group_array" in query]
|
||||
self.assertEqual(len(found), 2, found)
|
||||
self.assertTrue(all("pair_category" in query for query in found))
|
||||
|
||||
def test_plain_read_leaves_the_categories_out_of_the_json(self):
|
||||
self.seed()
|
||||
self.statements.clear()
|
||||
self.db.get_image_link_matches("", [], every_link=True)
|
||||
(query,) = self.selects()
|
||||
aggregate = query[query.index("json_array(") :].split(")", 1)[0]
|
||||
self.assertEqual(aggregate, "json_array(file_path, rating, matched, positive")
|
||||
|
||||
def test_read_releases_its_snapshot(self):
|
||||
# A statement left open after the read would pin its WAL snapshot, and
|
||||
# a TRUNCATE checkpoint from another connection would report busy.
|
||||
self.seed()
|
||||
self.assertTrue(self.db.get_image_link_matches("", None, [], every_link=True))
|
||||
self.assertTrue(self.db.get_image_link_matches("", [], every_link=True))
|
||||
other = sqlite3.connect(str(self.root / "prompts.db"))
|
||||
try:
|
||||
other.execute("UPDATE prompts SET rating=2 WHERE id=1")
|
||||
@@ -258,12 +295,15 @@ class TestOneStepLinkRead(LinkMatchesTestCase):
|
||||
|
||||
def test_wrapped_query_keeps_the_ambiguity_probe_an_index_search(self):
|
||||
self.seed()
|
||||
sql, params = link_filter_query("lantern", "cat-a", ["t1"], every_link=True)
|
||||
sql, params = link_filter_query(
|
||||
"lantern", ["t1"], every_link=True, category="cat-a"
|
||||
)
|
||||
with self.db.model.get_connection() as conn:
|
||||
plan = [
|
||||
row[3]
|
||||
for row in conn.execute(
|
||||
"EXPLAIN QUERY PLAN " + operations.link_matches_json_query(sql),
|
||||
"EXPLAIN QUERY PLAN "
|
||||
+ operations.link_matches_json_query(sql, categories=True),
|
||||
params,
|
||||
)
|
||||
]
|
||||
@@ -286,10 +326,10 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
) as attempts,
|
||||
self.assertLogs(LOGGER, "WARNING") as logged,
|
||||
):
|
||||
first = self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
first = self.db.get_image_link_matches("", [], every_link=True)
|
||||
found = self.selects()
|
||||
self.assertEqual(attempts.call_count, 1)
|
||||
second = self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
second = self.db.get_image_link_matches("", [], every_link=True)
|
||||
again = self.selects()
|
||||
self.assertEqual(first, expected)
|
||||
self.assertEqual(second, expected)
|
||||
@@ -308,11 +348,11 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
self.seed()
|
||||
self.add_image(2, b"/out/blob.png")
|
||||
expected = self.streamed("", None, [], True)
|
||||
self.assertEqual(expected[b"/out/blob.png"], (True, None, 2))
|
||||
self.assertEqual(expected[b"/out/blob.png"], (True, None, 2, None))
|
||||
self.statements.clear()
|
||||
with self.assertLogs(LOGGER, "WARNING") as logged:
|
||||
first = self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
second = self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
first = self.db.get_image_link_matches("", [], every_link=True)
|
||||
second = self.db.get_image_link_matches("", [], every_link=True)
|
||||
self.assertEqual(first, expected)
|
||||
self.assertEqual(second, expected)
|
||||
self.assertEqual(len(logged.records), 1, logged.output)
|
||||
@@ -330,7 +370,7 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
self.assertIn(b"\x00", expected)
|
||||
with self.assertLogs(LOGGER, "WARNING"):
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches("", None, [], every_link=True), expected
|
||||
self.db.get_image_link_matches("", [], every_link=True), expected
|
||||
)
|
||||
self.assertFalse(operations._one_step_unsupported)
|
||||
|
||||
@@ -342,12 +382,28 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
conn.execute("UPDATE prompts SET rating=? WHERE id=3", (b"\x00",))
|
||||
conn.execute("PRAGMA ignore_check_constraints=OFF")
|
||||
expected = self.streamed("", None, [], True)
|
||||
self.assertEqual(expected[UNICODE], (True, b"\x00", 3))
|
||||
self.assertEqual(expected[UNICODE], (True, b"\x00", 3, None))
|
||||
with self.assertLogs(LOGGER, "WARNING"):
|
||||
self.assertEqual(
|
||||
self.db.get_image_link_matches("", None, [], every_link=True), expected
|
||||
self.db.get_image_link_matches("", [], every_link=True), expected
|
||||
)
|
||||
|
||||
def test_blob_category_streams_a_category_read(self):
|
||||
self.seed()
|
||||
with self.db.model.get_connection() as conn:
|
||||
# Admitted by its recipe's text category; the positive's is a BLOB.
|
||||
conn.execute("UPDATE prompts SET category=? WHERE id=4", (b"cat-b",))
|
||||
conn.execute("UPDATE prompt_pairs SET category='cat-b' WHERE id=1")
|
||||
expected = self.streamed("", "cat-b", [], True)
|
||||
self.assertEqual(expected["/out/pair.png"], (True, 4, 4, "cat-b"))
|
||||
with self.assertLogs(LOGGER, "WARNING") as logged:
|
||||
actual = self.db.get_image_link_matches(
|
||||
"", [], every_link=True, category="cat-b"
|
||||
)
|
||||
self.assertEqual(actual, expected)
|
||||
self.assertIn("BLOB", logged.output[0])
|
||||
self.assertFalse(operations._one_step_unsupported)
|
||||
|
||||
def test_json_text_over_the_length_limit_streams(self):
|
||||
self.seed()
|
||||
expected = self.streamed("", None, [], True)
|
||||
@@ -355,7 +411,7 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
limit = conn.setlimit(sqlite3.SQLITE_LIMIT_LENGTH, 200)
|
||||
try:
|
||||
with self.assertLogs(LOGGER, "WARNING") as logged:
|
||||
actual = self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
actual = self.db.get_image_link_matches("", [], every_link=True)
|
||||
finally:
|
||||
conn.setlimit(sqlite3.SQLITE_LIMIT_LENGTH, limit)
|
||||
self.assertEqual(actual, expected)
|
||||
@@ -374,7 +430,7 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
self.assertNoLogs(LOGGER, "WARNING"),
|
||||
self.assertRaisesRegex(sqlite3.OperationalError, "^interrupted$"),
|
||||
):
|
||||
self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
self.db.get_image_link_matches("", [], every_link=True)
|
||||
finally:
|
||||
conn.set_progress_handler(None, 1)
|
||||
self.assertFalse(operations._one_step_warned)
|
||||
@@ -393,7 +449,7 @@ class TestStreamedFallback(LinkMatchesTestCase):
|
||||
self.assertLogs(LOGGER, "WARNING"),
|
||||
):
|
||||
with self.assertRaises(sqlite3.OperationalError):
|
||||
self.db.get_image_link_matches("", None, [], every_link=True)
|
||||
self.db.get_image_link_matches("", [], every_link=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
"""The inspector and History show an image's own category (#332 §5, DC4, DC5)."""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from tests import test_api_history_outputs as history
|
||||
from tests.test_api_images import make_png
|
||||
from py.api.images import _favorite_root_key, image_category_key
|
||||
from utils.inspection_response import (
|
||||
OWN_CATEGORY_UNREAD,
|
||||
_library_projection,
|
||||
apply_own_category,
|
||||
)
|
||||
|
||||
|
||||
class InspectCase(history.ImageAPITestCase):
|
||||
async def tearDownAsync(self):
|
||||
self.api.db.close_all()
|
||||
await super().tearDownAsync()
|
||||
|
||||
def own(self, path, value, sources=None):
|
||||
sources = sources or self.api._get_all_output_dirs()
|
||||
self.api.db.set_image_categories([image_category_key(sources, path)], value)
|
||||
|
||||
async def library(self, path, root="0"):
|
||||
response = await self.client.get(
|
||||
f"/prompt_manager/images/inspect/{path}", params={"root": root}
|
||||
)
|
||||
body = await response.json()
|
||||
self.assertEqual(response.status, 200, body)
|
||||
return body["library"]
|
||||
|
||||
|
||||
class TestInspectorOwnCategory(InspectCase):
|
||||
async def test_inspect_reports_image_scope(self):
|
||||
path = make_png(self.output_dir / "image.png")
|
||||
self.own(path, "Portraits")
|
||||
library = await self.library("image.png")
|
||||
self.assertEqual(library["status"], "none")
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]), ("Portraits", "image")
|
||||
)
|
||||
|
||||
async def test_own_overrides_linked_recipe_and_ambiguous(self):
|
||||
db = self.api.db
|
||||
linked = make_png(self.output_dir / "linked.png")
|
||||
self._link_image(linked, "castle at dusk")
|
||||
prompt = db.get_prompt_id_for_image(str(linked))
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE prompts SET category='Landscapes' WHERE id=?", (prompt,)
|
||||
)
|
||||
library = await self.library("linked.png")
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]),
|
||||
("Landscapes", "positive"),
|
||||
)
|
||||
self.own(linked, "Portraits")
|
||||
library = await self.library("linked.png")
|
||||
self.assertEqual(library["status"], "linked")
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]), ("Portraits", "image")
|
||||
)
|
||||
recipe = make_png(self.output_dir / "recipe.png")
|
||||
self._save_prompt("Pair positive", category="Night")
|
||||
pair = db.save_prompt_pair("Pair positive", "Noise", category="Recipes")
|
||||
positive = db.get_prompt_pair(pair)["positive_prompt_id"]
|
||||
image = db.link_image_to_prompt(positive, str(recipe))
|
||||
db.link_image_to_pair(pair, image, {"source": "fixture"})
|
||||
library = await self.library("recipe.png")
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]), ("Recipes", "pair")
|
||||
)
|
||||
self.own(recipe, "Mine")
|
||||
library = await self.library("recipe.png")
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]), ("Mine", "image")
|
||||
)
|
||||
ambiguous = make_png(self.output_dir / "ambiguous.png")
|
||||
self._link_image(ambiguous, "owner one")
|
||||
self._link_image(ambiguous, "owner two")
|
||||
self.own(ambiguous, "Scans")
|
||||
library = await self.library("ambiguous.png")
|
||||
self.assertEqual(library["status"], "ambiguous")
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]), ("Scans", "image")
|
||||
)
|
||||
|
||||
async def test_a_failed_read_hides_the_category(self):
|
||||
linked = make_png(self.output_dir / "linked.png")
|
||||
self._link_image(linked, "castle at dusk")
|
||||
|
||||
def broken(*_args, **_kwargs):
|
||||
raise sqlite3.OperationalError("secret")
|
||||
|
||||
with patch.object(self.api.db, "image_category", side_effect=broken):
|
||||
with self.assertLogs("prompt_manager", level="WARNING"):
|
||||
library = await self.library("linked.png")
|
||||
self.assertEqual(library["status"], "linked")
|
||||
self.assertIsNone(library["category"])
|
||||
self.assertIsNone(library["category_scope"])
|
||||
|
||||
async def test_a_lora_root_reads_no_own_category(self):
|
||||
lora = self.comfy_dir / "loras"
|
||||
make_png(lora / "preview.png")
|
||||
roots = [self.output_dir, lora]
|
||||
reads = []
|
||||
real = self.api.db.image_category
|
||||
|
||||
def recorded(*args):
|
||||
reads.append(args)
|
||||
return real(*args)
|
||||
|
||||
with (
|
||||
patch.object(self.api, "_media_roots", lambda outputs: roots),
|
||||
patch.object(self.api, "_linked_only_error", self._no_error),
|
||||
patch.object(self.api.db, "image_category", side_effect=recorded),
|
||||
):
|
||||
response = await self.client.get(
|
||||
"/prompt_manager/images/inspect/preview.png", params={"root": "1"}
|
||||
)
|
||||
body = await response.json()
|
||||
self.assertEqual(response.status, 200, body)
|
||||
self.assertEqual(reads, [])
|
||||
self.assertIsNone(body["library"]["category"])
|
||||
|
||||
@staticmethod
|
||||
async def _no_error(*_args):
|
||||
return None
|
||||
|
||||
|
||||
class TestProjection(unittest.TestCase):
|
||||
def test_scope_image_is_kept_for_category_only(self):
|
||||
base = {"status": "none", "scope": "saved-library", "tags": []}
|
||||
projected = _library_projection(
|
||||
{**base, "category": "Scans", "category_scope": "image"}
|
||||
)
|
||||
self.assertEqual(projected["category_scope"], "image")
|
||||
projected = _library_projection({**base, "tags_scope": "image"})
|
||||
self.assertNotIn("tags_scope", projected)
|
||||
|
||||
def test_apply_own_category_three_cases(self):
|
||||
library = {"status": "linked", "category": "L", "category_scope": "positive"}
|
||||
self.assertEqual(
|
||||
apply_own_category(library, "Mine"),
|
||||
{"status": "linked", "category": "Mine", "category_scope": "image"},
|
||||
)
|
||||
self.assertEqual(apply_own_category(library, None), library)
|
||||
self.assertEqual(
|
||||
apply_own_category(library, OWN_CATEGORY_UNREAD),
|
||||
{"status": "linked", "category": None, "category_scope": None},
|
||||
)
|
||||
self.assertEqual(library["category"], "L") # never mutated
|
||||
|
||||
|
||||
class TestHistoryOwnCategory(history.ImageAPITestCase):
|
||||
"""Only the inspection request reads the own category, keyed as Images."""
|
||||
|
||||
# The History fixture's helpers, without inheriting its tests.
|
||||
_base = history.HistoryOutputAPITests
|
||||
ref = _base.ref
|
||||
capture = _base.capture
|
||||
outputs = _base.outputs
|
||||
current = _base.current
|
||||
media = _base.media
|
||||
|
||||
async def get_application(self):
|
||||
app = await super().get_application()
|
||||
self.temp_dir = self.comfy_dir / "temp"
|
||||
self.temp_dir.mkdir()
|
||||
self.host_output = str(self.output_dir)
|
||||
self.host_temp = str(self.temp_dir)
|
||||
self.patches = history.ExitStack()
|
||||
self.addCleanup(self.patches.close)
|
||||
for name, value in (
|
||||
("folder_paths.get_output_directory", lambda: self.host_output),
|
||||
("folder_paths.get_temp_directory", lambda: self.host_temp),
|
||||
):
|
||||
self.patches.enter_context(patch(name, value, create=True))
|
||||
self.patches.enter_context(
|
||||
patch.object(
|
||||
history.GalleryConfig,
|
||||
"allowed_gallery_parents",
|
||||
lambda: [str(self.comfy_dir)],
|
||||
)
|
||||
)
|
||||
return app
|
||||
|
||||
async def tearDownAsync(self):
|
||||
self.api.close_history_reads()
|
||||
self.patches.close()
|
||||
await super().tearDownAsync()
|
||||
|
||||
async def inspect(self, start):
|
||||
descriptor = await self.media(start)
|
||||
response = await self.current(descriptor["inspection_url"])
|
||||
body = await response.json()
|
||||
self.assertEqual(response.status, 200, body)
|
||||
return body["library"]
|
||||
|
||||
async def test_history_inspection_shows_own_category(self):
|
||||
path = make_png(self.output_dir / "image.png")
|
||||
self.api.db.set_image_categories(
|
||||
[image_category_key([self.output_dir], path)], "Portraits"
|
||||
)
|
||||
start = self.capture([self.ref()])
|
||||
library = await self.inspect(start)
|
||||
self.assertEqual(
|
||||
(library["category"], library["category_scope"]), ("Portraits", "image")
|
||||
)
|
||||
|
||||
async def test_nested_sources_give_the_images_key(self):
|
||||
inner = self.output_dir / "nested"
|
||||
path = make_png(inner / "image.png")
|
||||
self.api._get_all_output_dirs = lambda: [self.output_dir, inner]
|
||||
self.api.db.set_image_categories(
|
||||
[(_favorite_root_key(inner), "image.png")], "S"
|
||||
)
|
||||
start = self.capture([self.ref(subfolder="nested")])
|
||||
library = await self.inspect(start)
|
||||
self.assertEqual(library["category"], "S")
|
||||
self.assertEqual(
|
||||
image_category_key([self.output_dir, inner], path),
|
||||
(_favorite_root_key(inner), "image.png"),
|
||||
)
|
||||
|
||||
async def test_a_symlinked_output_folder_still_shows_it(self):
|
||||
real = self.comfy_dir / "real-output"
|
||||
real.mkdir()
|
||||
link = self.comfy_dir / "linked-output"
|
||||
self._make_symlink(real, link, target_is_directory=True)
|
||||
make_png(real / "image.png")
|
||||
self.host_output = str(link)
|
||||
self.api._get_all_output_dirs = lambda: [real]
|
||||
self.api.db.set_image_categories([(_favorite_root_key(real), "image.png")], "S")
|
||||
start = self.capture([self.ref()])
|
||||
library = await self.inspect(start)
|
||||
self.assertEqual(library["category"], "S")
|
||||
|
||||
async def test_outputs_and_media_read_no_own_category(self):
|
||||
make_png(self.output_dir / "image.png")
|
||||
start = self.capture([self.ref()])
|
||||
reads = []
|
||||
real = self.api.db.image_category
|
||||
|
||||
def recorded(*args):
|
||||
reads.append(args)
|
||||
return real(*args)
|
||||
|
||||
with patch.object(self.api.db, "image_category", side_effect=recorded):
|
||||
descriptor = await self.media(start)
|
||||
for field in ("url", "thumbnail_url"):
|
||||
response = await self.current(descriptor[field])
|
||||
await response.read()
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(reads, [])
|
||||
await self.current(descriptor["inspection_url"])
|
||||
self.assertEqual(len(reads), 1)
|
||||
|
||||
async def test_a_temp_output_has_none(self):
|
||||
reference = {"folder_type": "temp", "subfolder": "", "filename": "a.png"}
|
||||
self.assertIsNone(self.api._history_own_category(reference, None))
|
||||
|
||||
async def test_a_failed_read_hides_it(self):
|
||||
make_png(self.output_dir / "image.png")
|
||||
start = self.capture([self.ref()])
|
||||
|
||||
def broken(*_args):
|
||||
raise sqlite3.OperationalError("secret")
|
||||
|
||||
with patch.object(self.api.db, "image_category", side_effect=broken):
|
||||
with self.assertLogs("prompt_manager", level="WARNING"):
|
||||
library = await self.inspect(start)
|
||||
self.assertIsNone(library["category"])
|
||||
self.assertTrue(os.path.exists(self.output_dir / "image.png"))
|
||||
@@ -1426,3 +1426,32 @@ class TestPerRecordBudgets(unittest.TestCase):
|
||||
self.assertEqual(metadata["record_formats"], {"workflow": "limited"})
|
||||
self.assertEqual(metadata["workflow_graph"]["status"], "limited")
|
||||
self.assertEqual(metadata["prompt_graph"]["status"], "absent")
|
||||
|
||||
|
||||
class TestInheritedCategory(unittest.TestCase):
|
||||
"""#332 MC4: one inherited-category rule for the chip, filter and History."""
|
||||
|
||||
def test_recipe_set_blank_and_absent(self):
|
||||
from utils.inspection_response import inherited_category, library_metadata
|
||||
|
||||
cases = [
|
||||
(("Night", "Recipes", True), ("Recipes", "pair")),
|
||||
(("Night", " ", True), ("Night", "positive")),
|
||||
(("Night", " \t", True), ("Night", "positive")),
|
||||
(("Night", b" ", True), ("Night", "positive")),
|
||||
(("Night", None, True), ("Night", "positive")),
|
||||
(("Night", "Recipes", False), ("Night", "positive")),
|
||||
((" ", None, False), (" ", "positive")),
|
||||
((None, None, False), (None, "positive")),
|
||||
]
|
||||
for args, expected in cases:
|
||||
with self.subTest(args=args):
|
||||
self.assertEqual(inherited_category(*args), expected)
|
||||
positive, recipe, has_recipe = args
|
||||
metadata = library_metadata(
|
||||
positive, ["t"], (recipe, []) if has_recipe else None
|
||||
)
|
||||
self.assertEqual(
|
||||
(metadata["category"], metadata["category_scope"]), expected
|
||||
)
|
||||
self.assertEqual(metadata["tags_scope"], "positive")
|
||||
|
||||
+291
-1
@@ -13,6 +13,11 @@ from database.library_commands import LibraryCommandError
|
||||
from database.operations import PromptDatabase
|
||||
from utils.hashing import generate_prompt_hash
|
||||
|
||||
try:
|
||||
from test_image_categories import category_checksum
|
||||
except ImportError:
|
||||
from tests.test_image_categories import category_checksum
|
||||
|
||||
|
||||
class TestVocabularyReads(unittest.TestCase):
|
||||
def setUp(self):
|
||||
@@ -168,7 +173,8 @@ class TestVocabularyReads(unittest.TestCase):
|
||||
self.assertEqual(page["total"], 4)
|
||||
self.assertFalse(page["has_more"])
|
||||
for row in page["categories"]:
|
||||
self.assertEqual(set(row), {"value", "prompt_count"})
|
||||
self.assertEqual(set(row), {"value", "prompt_count", "image_count"})
|
||||
self.assertEqual(row["image_count"], 0)
|
||||
self.assertEqual(
|
||||
row["prompt_count"],
|
||||
self.db.list_library(category=row["value"])["total"],
|
||||
@@ -238,6 +244,111 @@ class TestVocabularyReads(unittest.TestCase):
|
||||
with self.assertRaises(ValueError):
|
||||
self.read("list_vocabulary_tags", **options)
|
||||
|
||||
def own(self, value, *paths, root="root"):
|
||||
"""Give each relative path under ``root`` the own category ``value``."""
|
||||
self.db.set_image_categories([(root, path) for path in paths], value)
|
||||
|
||||
def all_categories(self, limit=200):
|
||||
rows, pages, after = [], [], None
|
||||
while True:
|
||||
page = self.read("list_vocabulary_categories", limit=limit, after=after)
|
||||
pages.append(page)
|
||||
rows.extend(page["categories"])
|
||||
if not page["has_more"]:
|
||||
return rows, pages
|
||||
after = page["next_key"]
|
||||
|
||||
def test_category_list_includes_image_only_values_last(self):
|
||||
self.save("Oldest", category="Zeta")
|
||||
self.save("Newer", category="Alpha")
|
||||
self.own("Zeta", "a.png", "b.png")
|
||||
self.own("Only images B", "c.png")
|
||||
self.own("Only images A", "d.png", "e.png", "gone.png")
|
||||
rows, _pages = self.all_categories()
|
||||
self.assertEqual(
|
||||
rows,
|
||||
[
|
||||
{"value": "Zeta", "prompt_count": 1, "image_count": 2},
|
||||
{"value": "Alpha", "prompt_count": 1, "image_count": 0},
|
||||
{"value": "Only images A", "prompt_count": 0, "image_count": 3},
|
||||
{"value": "Only images B", "prompt_count": 0, "image_count": 1},
|
||||
],
|
||||
)
|
||||
page = self.read("list_vocabulary_categories")
|
||||
self.assertEqual(page["total"], 4)
|
||||
|
||||
def test_category_cursor_crosses_into_image_only_values(self):
|
||||
for index in range(3):
|
||||
self.save(f"Owner {index}", category=f"P{index}")
|
||||
self.own("P1", "shared.png")
|
||||
self.own("I-b", "x.png")
|
||||
self.own("I-a", "y.png")
|
||||
self.own("I-c", "z.png", root="other")
|
||||
for limit in (1, 2, 3, 4):
|
||||
with self.subTest(limit=limit):
|
||||
rows, pages = self.all_categories(limit=limit)
|
||||
self.assertEqual(
|
||||
[row["value"] for row in rows],
|
||||
["P0", "P1", "P2", "I-a", "I-b", "I-c"],
|
||||
)
|
||||
self.assertEqual({page["total"] for page in pages}, {6})
|
||||
self.assertEqual(
|
||||
[row["image_count"] for row in rows], [0, 1, 0, 1, 1, 1]
|
||||
)
|
||||
first = self.read("list_vocabulary_categories", limit=3)
|
||||
self.assertEqual(first["next_key"], [3, "P2"])
|
||||
second = self.read("list_vocabulary_categories", limit=1, after=[3, "P2"])
|
||||
self.assertEqual(second["next_key"], [2**63 - 1, "I-a"])
|
||||
|
||||
def test_image_rows_on_a_prompt_value_count_once(self):
|
||||
self.save("Owner", category="Shared")
|
||||
self.db.save_prompt_pair("Owner", "Noise", category="Shared")
|
||||
self.own("Shared", *[f"{n}.png" for n in range(5)])
|
||||
self.own("Shared", "elsewhere.png", root="second")
|
||||
rows, _pages = self.all_categories()
|
||||
self.assertEqual(
|
||||
rows, [{"value": "Shared", "prompt_count": 1, "image_count": 6}]
|
||||
)
|
||||
|
||||
def test_library_options_ignore_image_categories(self):
|
||||
self.save("Owner", category="Prompt value")
|
||||
before = self.db.get_library_options()
|
||||
self.own("Image value", "a.png")
|
||||
self.own("Prompt value", "b.png")
|
||||
self.assertEqual(self.db.get_library_options(), before)
|
||||
self.assertEqual(before["categories"], ["Prompt value"])
|
||||
|
||||
def test_category_reads_use_the_image_category_index(self):
|
||||
from database import vocabulary
|
||||
|
||||
self.own("Value", "a.png")
|
||||
statements = [
|
||||
vocabulary._CATEGORY_ROWS + "SELECT COUNT(*) FROM counted",
|
||||
vocabulary._CATEGORY_ROWS
|
||||
+ "SELECT value,first_pid,prompt_count,image_count FROM counted "
|
||||
"WHERE first_pid>? OR (first_pid=? AND value>?) "
|
||||
"ORDER BY first_pid,value LIMIT ?",
|
||||
]
|
||||
plans = []
|
||||
with self.db.model.get_connection() as conn:
|
||||
for sql in statements:
|
||||
params = [1] * sql.count("?")
|
||||
plans.extend(
|
||||
row[-1] for row in conn.execute("EXPLAIN QUERY PLAN " + sql, params)
|
||||
)
|
||||
for sql in (
|
||||
vocabulary._IMAGE_CATEGORY_ROWIDS,
|
||||
*vocabulary._CATEGORY_UPDATES,
|
||||
):
|
||||
params = [1] * sql.count("?")
|
||||
plans.extend(
|
||||
row[-1] for row in conn.execute("EXPLAIN QUERY PLAN " + sql, params)
|
||||
)
|
||||
image_steps = [step for step in plans if "image_categories" in step]
|
||||
self.assertTrue(image_steps)
|
||||
for step in image_steps:
|
||||
self.assertIn("idx_image_categories_category", step, step)
|
||||
|
||||
def test_reads_share_one_snapshot_and_report_epoch(self):
|
||||
self.save("Cat", category="Animal", tags=["cat"])
|
||||
epoch = self.db.get_execution_epoch()
|
||||
@@ -741,6 +852,7 @@ class TestVocabularyCategoryCommands(unittest.TestCase):
|
||||
"other_prompts": 1,
|
||||
"negative_variants": 2,
|
||||
"archived_negative_variants": 1,
|
||||
"images": 0,
|
||||
},
|
||||
)
|
||||
self.assertFalse(result["target_exists"])
|
||||
@@ -810,6 +922,7 @@ class TestVocabularyCategoryCommands(unittest.TestCase):
|
||||
"other_prompts": 0,
|
||||
"negative_variants": 0,
|
||||
"archived_negative_variants": 0,
|
||||
"images": 0,
|
||||
},
|
||||
)
|
||||
self.assertEqual(self.rows("prompts")[variant]["category"], "Other")
|
||||
@@ -902,6 +1015,171 @@ class TestVocabularyCategoryCommands(unittest.TestCase):
|
||||
self.apply(bad)
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
|
||||
def own(self, value, *paths, root="root"):
|
||||
self.db.set_image_categories([(root, path) for path in paths], value)
|
||||
|
||||
def image_rows(self):
|
||||
with self.db.model.get_connection() as conn:
|
||||
return {
|
||||
(row[0], row[1]): (row[2], row[3], row[4])
|
||||
for row in conn.execute(
|
||||
"SELECT root_key,relative_path,category,updated_at,rowid "
|
||||
"FROM image_categories ORDER BY rowid"
|
||||
)
|
||||
}
|
||||
|
||||
def checksum(self):
|
||||
with self.db.model.get_connection() as conn:
|
||||
return category_checksum(conn)
|
||||
|
||||
def test_preview_counts_image_rows(self):
|
||||
self.owners()
|
||||
self.own("Scenery", "a.png", "b.png", "missing.png")
|
||||
self.own("Scenery", "c.png", root="second")
|
||||
self.own("Other", "d.png")
|
||||
before = self.image_rows()
|
||||
result = self.preview()
|
||||
self.assertEqual(result["impact"]["images"], 4)
|
||||
self.assertEqual(self.image_rows(), before)
|
||||
self.assertTrue(self.preview(target="Other")["target_exists"])
|
||||
|
||||
def test_apply_renames_image_categories(self):
|
||||
self.owners()
|
||||
bystander = self.save("Bystander", category="Other")
|
||||
self.own("Scenery", "a.png", "b.png")
|
||||
self.own("Scenery", "c.png", root="second")
|
||||
self.own(" Scenery ", "padded.png")
|
||||
self.own("Other", "d.png")
|
||||
self.age_rows()
|
||||
with self.db.model.get_connection() as conn:
|
||||
conn.execute("UPDATE image_categories SET updated_at=?", (_OLD_STAMP,))
|
||||
images_before = self.image_rows()
|
||||
prompts_before, pairs_before = self.snapshot()
|
||||
preview = self.preview()
|
||||
receipt = self.apply(preview["impact_token"])
|
||||
self.assertEqual(receipt["impact"]["images"], 3)
|
||||
images_after = self.image_rows()
|
||||
self.assertEqual(set(images_after), set(images_before))
|
||||
renamed = {("root", "a.png"), ("root", "b.png"), ("second", "c.png")}
|
||||
for key, (value, stamp, rowid) in images_after.items():
|
||||
old_value, old_stamp, old_rowid = images_before[key]
|
||||
self.assertEqual(rowid, old_rowid, key)
|
||||
if key in renamed:
|
||||
self.assertEqual(value, "Landscape")
|
||||
self.assertNotEqual(stamp, _OLD_STAMP)
|
||||
else:
|
||||
self.assertEqual((value, stamp), (old_value, old_stamp))
|
||||
prompts_after, pairs_after = self.snapshot()
|
||||
self.assertEqual(prompts_after[bystander], prompts_before[bystander])
|
||||
for before, after in (
|
||||
(prompts_before, prompts_after),
|
||||
(pairs_before, pairs_after),
|
||||
):
|
||||
for row_id, row in after.items():
|
||||
if before[row_id]["category"] != "Scenery":
|
||||
self.assertEqual(row, before[row_id])
|
||||
# Merging into a value images already hold keeps every file key.
|
||||
merge = self.preview(source="Landscape", target="Other")
|
||||
self.assertTrue(merge["target_exists"])
|
||||
self.assertEqual(merge["impact"]["images"], 3)
|
||||
receipt = self.apply(merge["impact_token"], source="Landscape", target="Other")
|
||||
self.assertTrue(receipt["consolidated"])
|
||||
merged = self.image_rows()
|
||||
self.assertEqual(set(merged), set(images_before))
|
||||
self.assertEqual(
|
||||
sorted(key for key, row in merged.items() if row[0] == "Other"),
|
||||
sorted([*renamed, ("root", "d.png")]),
|
||||
)
|
||||
|
||||
def test_image_only_value_can_be_renamed(self):
|
||||
self.save("Bystander", category="Kept")
|
||||
self.db.save_prompt_pair("Pair", "Noise", category="Kept")
|
||||
self.own("Image only", "a.png", "b.png")
|
||||
self.age_rows()
|
||||
before = self.checksum()
|
||||
preview = self.preview(source="Image only", target="Renamed")
|
||||
self.assertEqual(
|
||||
preview["impact"],
|
||||
{
|
||||
"positive_prompts": 0,
|
||||
"other_prompts": 0,
|
||||
"negative_variants": 0,
|
||||
"archived_negative_variants": 0,
|
||||
"images": 2,
|
||||
},
|
||||
)
|
||||
self.assertFalse(preview["target_exists"])
|
||||
receipt = self.apply(
|
||||
preview["impact_token"], source="Image only", target="Renamed"
|
||||
)
|
||||
self.assertFalse(receipt["consolidated"])
|
||||
self.assertEqual({row[0] for row in self.image_rows().values()}, {"Renamed"})
|
||||
self.assertEqual(self.checksum(), before)
|
||||
# An image-only target counts as existing.
|
||||
self.own("Image target", "c.png")
|
||||
self.assertTrue(
|
||||
self.preview(source="Renamed", target="Image target")["target_exists"]
|
||||
)
|
||||
# Nothing holds the value any more: refused.
|
||||
with self.assertRaises(LibraryCommandError) as caught:
|
||||
self.preview(source="Image only", target="Again")
|
||||
self.assertEqual(caught.exception.code, "vocabulary-conflict")
|
||||
|
||||
def test_token_refused_after_an_image_row_gains_or_loses_the_value(self):
|
||||
self.owners()
|
||||
self.own("Scenery", "a.png", "b.png")
|
||||
|
||||
def gain():
|
||||
self.own("Scenery", "late.png")
|
||||
|
||||
def lose():
|
||||
self.db.clear_image_categories([("root", "a.png")])
|
||||
|
||||
def move_away():
|
||||
self.own("Elsewhere", "b.png")
|
||||
|
||||
def target_on_image():
|
||||
self.own("Landscape", "target.png")
|
||||
|
||||
for mutate in (gain, lose, move_away, target_on_image):
|
||||
with self.subTest(mutate.__name__):
|
||||
token = self.preview()["impact_token"]
|
||||
mutate()
|
||||
before = self.snapshot(), self.image_rows()
|
||||
with self.assertRaises(LibraryCommandError) as caught:
|
||||
self.apply(token)
|
||||
self.assertEqual(caught.exception.code, "vocabulary-conflict")
|
||||
self.assertEqual((self.snapshot(), self.image_rows()), before)
|
||||
|
||||
def test_image_category_failure_rolls_back_prompts(self):
|
||||
self.owners()
|
||||
self.own("Scenery", "a.png")
|
||||
token = self.preview()["impact_token"]
|
||||
with self.db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"CREATE TRIGGER abort_images BEFORE UPDATE ON image_categories "
|
||||
"BEGIN SELECT RAISE(ABORT, 'boom'); END"
|
||||
)
|
||||
before = self.snapshot(), self.image_rows()
|
||||
with self.assertRaises(sqlite3.Error):
|
||||
self.apply(token)
|
||||
self.assertEqual((self.snapshot(), self.image_rows()), before)
|
||||
|
||||
def test_category_exists_reads_the_image_index(self):
|
||||
from database import vocabulary
|
||||
|
||||
with self.db.model.get_connection() as conn:
|
||||
plan = [
|
||||
row[-1]
|
||||
for row in conn.execute(
|
||||
"EXPLAIN QUERY PLAN " + vocabulary._CATEGORY_EXISTS, {"v": "x"}
|
||||
)
|
||||
]
|
||||
steps = [step for step in plan if "image_categories" in step]
|
||||
self.assertTrue(steps, plan)
|
||||
for step in steps:
|
||||
self.assertIn("idx_image_categories_category", step)
|
||||
|
||||
def test_category_apply_failure_rolls_back(self):
|
||||
self.owners()
|
||||
token = self.preview()["impact_token"]
|
||||
@@ -942,6 +1220,18 @@ class TestSharedTokenHashing(unittest.TestCase):
|
||||
),
|
||||
"cefe7f6a5544b7f6a203f5527810899c6af568aa7abc1b75adffbfb5ae6251ae",
|
||||
)
|
||||
with_images = category_impact_token(
|
||||
"e", "S", "T", True, [(1, True)], [], images=[9, 4]
|
||||
)
|
||||
self.assertRegex(with_images, r"\A[0-9a-f]{64}\Z")
|
||||
self.assertEqual(
|
||||
with_images,
|
||||
category_impact_token("e", "S", "T", True, [(1, True)], [], images=[4, 9]),
|
||||
)
|
||||
self.assertNotEqual(
|
||||
with_images,
|
||||
category_impact_token("e", "S", "T", True, [(1, True)], [], images=[4]),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -300,6 +300,49 @@ def _blank(value: str | bytes | None) -> bool:
|
||||
return value is None or not value.strip()
|
||||
|
||||
|
||||
def inherited_category(
|
||||
positive_category: str | bytes | None,
|
||||
recipe_category: str | bytes | None,
|
||||
has_recipe: bool,
|
||||
) -> tuple:
|
||||
"""``(value, scope)`` of the category an image's link supplies (#332 MC4).
|
||||
|
||||
The one rule for the inspector chip, the Images category filter and
|
||||
History (D39): a linked recipe's own category when it is not blank,
|
||||
``(recipe category, "pair")``; otherwise the positive's value unchanged,
|
||||
blank or not, ``(positive category, "positive")``.
|
||||
"""
|
||||
if has_recipe and not _blank(recipe_category):
|
||||
return recipe_category, "pair"
|
||||
return positive_category, "positive"
|
||||
|
||||
|
||||
class _OwnCategoryUnread:
|
||||
"""The own-category read failed: hide the category (#332 DC5)."""
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return "OWN_CATEGORY_UNREAD"
|
||||
|
||||
|
||||
OWN_CATEGORY_UNREAD = _OwnCategoryUnread()
|
||||
|
||||
|
||||
def apply_own_category(library: dict, own) -> dict:
|
||||
"""*library* with an image's own category applied (#332 §5).
|
||||
|
||||
``own`` is the own category (a string, shown with scope ``"image"``
|
||||
whatever the link status), None (the inherited values stay) or
|
||||
OWN_CATEGORY_UNREAD (category and scope hidden rather than possibly
|
||||
wrong). Returns a new dict.
|
||||
"""
|
||||
result = dict(library)
|
||||
if own is OWN_CATEGORY_UNREAD:
|
||||
result.update(category=None, category_scope=None)
|
||||
elif isinstance(own, str):
|
||||
result.update(category=own, category_scope="image")
|
||||
return result
|
||||
|
||||
|
||||
def library_metadata(
|
||||
positive_category: str | bytes | None,
|
||||
positive_tags: list,
|
||||
@@ -312,13 +355,14 @@ def library_metadata(
|
||||
the linked positive prompt's, the values Set category writes and the Images
|
||||
filters match. Tag sets are never merged. ``recipe`` is (category, tags).
|
||||
"""
|
||||
category, category_scope = positive_category, "positive"
|
||||
category, category_scope = inherited_category(
|
||||
positive_category,
|
||||
recipe[0] if recipe is not None else None,
|
||||
recipe is not None,
|
||||
)
|
||||
tags, tags_scope = positive_tags, "positive"
|
||||
if recipe is not None:
|
||||
if not _blank(recipe[0]):
|
||||
category, category_scope = recipe[0], "pair"
|
||||
if recipe[1]:
|
||||
tags, tags_scope = recipe[1], "pair"
|
||||
if recipe is not None and recipe[1]:
|
||||
tags, tags_scope = recipe[1], "pair"
|
||||
return {
|
||||
"category": category,
|
||||
"category_scope": category_scope,
|
||||
@@ -345,9 +389,13 @@ def _library_projection(library: dict) -> dict:
|
||||
value = library.get(key)
|
||||
if value is None or isinstance(value, str):
|
||||
result[key] = value
|
||||
for key in ("category_scope", "tags_scope"):
|
||||
# An image's own category (#332) has the scope "image"; tags never do.
|
||||
for key, scopes in (
|
||||
("category_scope", ("pair", "positive", "image", None)),
|
||||
("tags_scope", ("pair", "positive", None)),
|
||||
):
|
||||
scope = library.get(key)
|
||||
if scope in ("pair", "positive", None):
|
||||
if scope in scopes:
|
||||
result[key] = scope
|
||||
tags = library.get("tags")
|
||||
result["tags"] = (
|
||||
|
||||
@@ -20,6 +20,8 @@ const REPRESENTATIONS = {
|
||||
limited: 'Record exceeds inspection limits; text is unavailable',
|
||||
};
|
||||
const TEXTLESS = new Set(['invalid', 'limited']);
|
||||
// #332: the hidden prefix that names where a category chip's value came from.
|
||||
const CATEGORY_PREFIX = {image: 'Image category: ', positive: 'Positive prompt category: ', pair: 'Recipe category: '};
|
||||
const REVEAL_URL = '/prompt_manager/images/reveal';
|
||||
// D38 Show in folder: the route's answers, in the maintainer's words.
|
||||
const REVEAL_OPENED = 'Opened the folder on this computer.';
|
||||
@@ -265,7 +267,10 @@ export function mountFileInspector(container, {api, sender, parentScope, variant
|
||||
inner.append(heading, el('p', filename, 'small'));
|
||||
const badges = el('div', undefined, 'tags'), badge = evidenceBadge(record.prompts);
|
||||
if (badge) badges.append(el('span', badge, 'chip blue'));
|
||||
if (linked && library.category) badges.append(prefixedChip(library.category_scope === 'pair' ? 'Recipe category: ' : 'Positive prompt category: ', library.category));
|
||||
// #332: an image's own category shows for any link status; an inherited one only when linked.
|
||||
if (library?.category && (library.category_scope === 'image' || linked)) {
|
||||
badges.append(prefixedChip(CATEGORY_PREFIX[library.category_scope] ?? CATEGORY_PREFIX.positive, library.category));
|
||||
}
|
||||
if (badges.childNodes.length) inner.append(badges);
|
||||
const prompts = el('div'); inner.append(prompts);
|
||||
const blocks = mountPromptBlocks(prompts, {sender, parentScope:contentScope, includePairAction:false, contextLabel:promptContext, title});
|
||||
|
||||
@@ -201,7 +201,9 @@ function resolutionFor(value) {
|
||||
function libraryFor(value) {
|
||||
if (!object(value) || !['linked', 'none', 'ambiguous', 'unavailable', 'limited'].includes(value.status) || value.scope !== 'saved-library' ||
|
||||
!Array.isArray(value.tags) || value.tags.some(tag => typeof tag !== 'string') ||
|
||||
![value.category_scope, value.tags_scope].every(scope => [null, 'pair', 'positive'].includes(scope))) throw new TypeError('Invalid inspection library');
|
||||
![null, 'pair', 'positive', 'image'].includes(value.category_scope) ||
|
||||
![null, 'pair', 'positive'].includes(value.tags_scope)) throw new TypeError('Invalid inspection library');
|
||||
// #332: an image's own category (scope "image") may sit on any link status; tags never have that scope.
|
||||
// D39: each field names its own source, so a recipe's category can sit beside its positive's tags.
|
||||
const library = { status: value.status, scope: value.scope, tags: [...value.tags], category_scope: value.category_scope, tags_scope: value.tags_scope };
|
||||
for (const key of ['positive_id', 'pair_id', 'title', 'category']) {
|
||||
|
||||
Reference in New Issue
Block a user