feat(maintenance): repair old Pair prompts' empty category and tags (#550)
Adds System > Maintenance > "Pair category and tags" (pair_metadata), a reviewed plan that fills the empty category and empty tag set of prompts Pair node runs created before #546, from their non-archived saved pairs. - Only Pair-created prompts (earliest pair has them as the positive, saved within 2 s), only empty fields, only when every pair with a value agrees. - Archived, edited (updated_at not within 2 s of created_at), conflicting and over-20-tag fields are kept and listed first in the plan. - Apply recomputes the fingerprint inside BEGIN IMMEDIATE and writes guarded statements; a guard that hits nothing rolls everything back. Never writes prompt_pairs, prompt_pair_tags, updated_at, tags or Library state. - Follows the safety backup policy like the other operations. - The job's downloadable report lists every filled prompt (no 200-row cap): Job.record_counted(complete=True). - D192: the row beyond the concept, mapped in the concept comparison.
This commit is contained in:
Vito Sansevero
+43
-1
@@ -2943,7 +2943,8 @@ change: ComfyUI needs a restart.
|
||||
A Pair save, by a run or the node's Save pair button, writes the category and tags to the positive prompt only when that
|
||||
save creates the prompt row (maintainer ruling). An existing prompt never changes on a run: empty, cleared or set by a person, including one first seen as
|
||||
another pair's negative. The negative stays empty, the saved pair keeps its own metadata, and `save_prompt_pair` writes
|
||||
no prompt metadata unless called with `fill_new_positive`. No backfill; optional repair is #550.
|
||||
no prompt metadata unless called with `fill_new_positive`. No backfill on a run; the opt-in repair for prompts older
|
||||
Pair runs saved is System › Maintenance › "Pair category and tags" (#550, "Pair category and tags repair" below).
|
||||
Tests: `tests/test_prompt_pair_node.py`, `tests/test_api_pairs.py`, `tests/comfyui/pair-node.spec.mjs`. A Python change: ComfyUI needs a restart.
|
||||
|
||||
### Delete… on the archived shelf (#492)
|
||||
@@ -4272,6 +4273,12 @@ the per-section hand-off panel, was retired by S12 (#329): every section is nati
|
||||
primary stays the concept's plain button; the plan it opens has the primary.
|
||||
`duplicates dialog matches the concept` and `duplicates comparison detects mutations`
|
||||
(system-maintenance-concept.spec) pin the mapping.
|
||||
- **D192** (Maintenance, #550, beyond the concept): Database adds "Pair category and
|
||||
tags" after "Unused prompt components", a plain Review plan row (no danger mark: it
|
||||
deletes nothing) with the line "Fill an empty category or tags on prompts older Pair
|
||||
node runs saved, from their saved pairs. Only empty fields of never-edited prompts."
|
||||
`system-maintenance-concept.spec.mjs` maps it (`mapPairRow`) as a copy of Optimize
|
||||
database's row.
|
||||
- **D102** (Maintenance, S7, Q5, #222): Metadata cache clears only the browser entries
|
||||
Prompt Manager owns as a cache (`web/js/cache-ownership.js`). The inventory of every
|
||||
storage writer has none (see "Browser cache ownership" below), so the row's line adds
|
||||
@@ -4809,6 +4816,41 @@ replaced the database since the review; the plan is dropped). The job:
|
||||
prompt ids only), logs the backup by its file name, and prunes safety backups to the
|
||||
newest 10. With no backup the log and the status line say "No safety backup".
|
||||
|
||||
#### Pair category and tags repair (#550)
|
||||
|
||||
`pair_metadata` ("Pair category and tags", D192; `database/pair_metadata_repair.py`) fills
|
||||
the empty category and the empty tag set of prompts that Pair node runs before #546
|
||||
saved, from their saved pairs. It is reviewed and applied like the operations above and
|
||||
follows the safety backup policy like them (no forced backup). Rules
|
||||
(`docs/egm/post-3.4.1/550-design.md`):
|
||||
|
||||
- Considered: prompts the Pair node created as a positive: their earliest pair (lowest
|
||||
id, either role, archived included) has them as the positive and its `created_at` is
|
||||
within 2 s of theirs. A prompt first saved by the PromptManager node, or first as a
|
||||
negative, is out of scope and not counted.
|
||||
- Sources: the prompt's non-archived pairs where it is the positive; pairs with a blank
|
||||
category or no tags are ignored for that field. Category values are trimmed and
|
||||
case-sensitive, tag sets compared by tag id; a field fills when exactly one value is
|
||||
left. A whitespace-only category counts as empty.
|
||||
- Kept, per field, in this order: archived prompt, edited prompt (`updated_at` not within
|
||||
2 s of `created_at`; NULL or unreadable counts as edited), pairs that disagree, a tag
|
||||
set over 20 tags. A prompt can fill one field and keep the other.
|
||||
- The plan lists protected rows first, then fills, then a "Record" row, within the 50
|
||||
detail rows; at most 20,000 prompts per plan ("review again for the rest").
|
||||
- The fingerprint covers each considered prompt's computed result (values, tag ids and
|
||||
names, source pair ids), never a pair's title, notes, rating, run count or use
|
||||
stamps, so a Pair run between review and apply still applies.
|
||||
- The apply's writes are guarded (`category` still blank, no `prompt_tags` row) and a
|
||||
guard that hits nothing rolls the whole transaction back as `plan-changed`. It never
|
||||
writes `prompt_pairs`, `prompt_pair_tags`, `updated_at`, a tag or Library state.
|
||||
- The job lists 200 changed rows ("Prompt 12: category “night”; tags a, b"); its
|
||||
Download report lists every one (`truncated: false`, `Job.record_counted(complete=True)`).
|
||||
|
||||
Tests: `tests/test_maintenance_pair_metadata.py`, `tests/test_api_maintenance_pair_metadata.py`,
|
||||
`tests/test_jobs.py`, `tests/js/workspace-system-maintenance.test.js`,
|
||||
`tests/browser/workspace-system-maintenance.spec.mjs` (hook `/__test__/system/pair-metadata`).
|
||||
A Python change: ComfyUI needs a restart.
|
||||
|
||||
The legacy restore (`POST /prompt_manager/restore`) takes a backup of any size the disks
|
||||
have room for, so a real library's safety backup restores: the upload, its staged copy
|
||||
and the swap (the upload twice in the temporary folder, once more plus the current
|
||||
|
||||
+15
-2
@@ -2,7 +2,8 @@
|
||||
|
||||
Four operations, each the native owner of a legacy ``/maintenance`` one and
|
||||
built on the same queries, so a preview counts what the legacy cleanup would
|
||||
change:
|
||||
change (``workflow_storage`` and #550's ``pair_metadata``, which fills old
|
||||
Pair prompts' empty category and tags, live in their own modules):
|
||||
|
||||
- ``duplicate_prompts``: prompts whose text is equal ignoring case and outer
|
||||
spaces (``DUPLICATE_PROMPTS_SQL``, legacy ``cleanup_duplicates``); each
|
||||
@@ -60,6 +61,9 @@ MAINTENANCE_OPERATIONS = (
|
||||
"missing_links",
|
||||
# Applied as its own batched job (database/workflow_storage.py).
|
||||
"workflow_storage",
|
||||
# #550: fill old Pair prompts' empty category and tags
|
||||
# (database/pair_metadata_repair.py).
|
||||
"pair_metadata",
|
||||
)
|
||||
MAX_DETAILS = 50 # detail rows a plan lists
|
||||
MAX_RESULT_ROWS = 200 # changed records an apply describes (the job lists 200)
|
||||
@@ -533,6 +537,10 @@ class MaintenanceOperations:
|
||||
lambda conn: (self._link_rows(conn), _mark_rows(conn)),
|
||||
)
|
||||
return self._plan_missing_links(rows, roots, _stale_marks(marks, roots))
|
||||
if operation == "pair_metadata":
|
||||
from .pair_metadata_repair import plan_pair_metadata
|
||||
|
||||
return _read_snapshot(self.model.db_path, plan_pair_metadata)
|
||||
reader = (
|
||||
self._plan_duplicate_prompts
|
||||
if operation == "duplicate_prompts"
|
||||
@@ -800,7 +808,8 @@ class MaintenanceOperations:
|
||||
records changed and up to ``MAX_RESULT_ROWS`` descriptions of them
|
||||
(``{prompt_id[, kept_id]}`` or ``{image_path}``). ``marks`` are a
|
||||
missing-links plan's stale removal marks; that apply also returns
|
||||
``marks``, how many it forgot.
|
||||
``marks``, how many it forgot. ``pair_metadata`` returns every filled
|
||||
prompt, uncapped (its job keeps the complete record, #550).
|
||||
"""
|
||||
if operation not in MAINTENANCE_OPERATIONS or operation == "workflow_storage":
|
||||
raise ValueError(f"Unknown maintenance operation {operation!r}")
|
||||
@@ -815,6 +824,10 @@ class MaintenanceOperations:
|
||||
"unused_components": self._apply_unused_components,
|
||||
}
|
||||
with self._pair_transaction() as conn:
|
||||
if operation == "pair_metadata":
|
||||
from .pair_metadata_repair import apply_pair_metadata
|
||||
|
||||
return apply_pair_metadata(conn, fingerprint, targets)
|
||||
if operation == "missing_links":
|
||||
return self._apply_missing_links(
|
||||
conn, fingerprint, targets, roots, list(marks)
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
"""Pair category and tags repair (#550): the ``pair_metadata`` maintenance plan.
|
||||
|
||||
Before #546 a Pair node run saved its category and tags on the saved pair
|
||||
only; the positive prompt it created stayed Uncategorized and untagged. This
|
||||
plan fills the **empty** category and the **empty** tag set of those prompts
|
||||
from their saved pairs (``docs/egm/post-3.4.1/550-design.md``):
|
||||
|
||||
- Considered: prompts the Pair node created as a positive (their earliest
|
||||
pair, in either role, has them as the positive and its ``created_at`` is
|
||||
within ``PAIR_CREATED_TOLERANCE`` of theirs) whose category is blank
|
||||
and/or that have no tags.
|
||||
- Sources: the prompt's non-archived pairs where it is the positive. Pairs
|
||||
with a blank category or no tags are ignored for that field. A field fills
|
||||
when exactly one value remains (category: trimmed, case-sensitive; tags:
|
||||
the tag-id set).
|
||||
- Kept (protected), in this order: archived prompt, edited prompt
|
||||
(``updated_at`` not within the tolerance of ``created_at``), pairs that
|
||||
disagree, a tag set over ``MAX_LIBRARY_TAGS``.
|
||||
|
||||
The apply runs in the caller's ``BEGIN IMMEDIATE`` transaction, recomputes
|
||||
the fingerprint, then writes guarded statements (the field must still be
|
||||
empty) that never touch a pair, a negative or ``updated_at``. A guard that
|
||||
hits nothing raises ``PlanChanged`` so the whole transaction rolls back.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
|
||||
from . import maintenance as _maintenance
|
||||
from .library_commands import MAX_LIBRARY_TAGS
|
||||
from .maintenance import (
|
||||
MAX_DETAILS,
|
||||
MaintenanceDraft,
|
||||
PlanChanged,
|
||||
_excerpt,
|
||||
_Fingerprint,
|
||||
_marks,
|
||||
_plural,
|
||||
)
|
||||
|
||||
OPERATION = "pair_metadata"
|
||||
# Seconds: the Pair save stamps its new prompt and its pair with separate
|
||||
# utc_now_iso() calls in one transaction (microseconds apart).
|
||||
PAIR_CREATED_TOLERANCE = 2.0
|
||||
# Fill prompts one plan applies; the rest are applied by the next review.
|
||||
# Bounds the complete record a job keeps in memory (about 100 bytes a row).
|
||||
MAX_PLAN_FILLS = 20000
|
||||
DETAIL_VALUES = 3
|
||||
DETAIL_TAGS = 5
|
||||
MEDIA_LABEL = "No image file or saved pair is changed."
|
||||
EMPTY_LABEL = "No Pair prompts are missing a category or tags"
|
||||
RECORD_NOTE = (
|
||||
"The job's report lists every filled field. Download it to keep it: "
|
||||
"Activity forgets jobs after a restart."
|
||||
)
|
||||
# Protected classes in precedence order: (key, detail, reason one, many).
|
||||
EDITED_DETAIL = (
|
||||
"Kept: edited after it was saved; its empty category or tags may be intentional"
|
||||
)
|
||||
PROTECTED = (
|
||||
(
|
||||
"archived",
|
||||
"Kept: archived in the Library",
|
||||
"archived in the Library",
|
||||
"archived in the Library",
|
||||
),
|
||||
(
|
||||
"edited",
|
||||
EDITED_DETAIL,
|
||||
"edited after it was saved",
|
||||
"edited after they were saved",
|
||||
),
|
||||
("conflict", None, "where pairs disagree", "where pairs disagree"),
|
||||
(
|
||||
"too-many-tags",
|
||||
f"Kept: its pair has more than {MAX_LIBRARY_TAGS} tags",
|
||||
f"whose pair has more than {MAX_LIBRARY_TAGS} tags",
|
||||
f"whose pair has more than {MAX_LIBRARY_TAGS} tags",
|
||||
),
|
||||
)
|
||||
_PROTECTED_ORDER = [key for key, *_rest in PROTECTED]
|
||||
|
||||
_CANDIDATES_SQL = """
|
||||
SELECT p.id, p.hash, p.text, p.created_at, p.updated_at,
|
||||
COALESCE(TRIM(p.category), '') = '' AS blank_category,
|
||||
NOT EXISTS (SELECT 1 FROM prompt_tags pt WHERE pt.prompt_id = p.id)
|
||||
AS untagged,
|
||||
EXISTS (SELECT 1 FROM prompt_archive pa WHERE pa.prompt_id = p.id)
|
||||
AS archived,
|
||||
(SELECT CASE WHEN pp.positive_prompt_id = p.id
|
||||
THEN pp.created_at END
|
||||
FROM prompt_pairs pp
|
||||
WHERE pp.positive_prompt_id = p.id OR pp.negative_prompt_id = p.id
|
||||
ORDER BY pp.id LIMIT 1) AS first_pair_at
|
||||
FROM prompts p
|
||||
WHERE EXISTS (SELECT 1 FROM prompt_pairs pp WHERE pp.positive_prompt_id = p.id)
|
||||
AND (COALESCE(TRIM(p.category), '') = ''
|
||||
OR NOT EXISTS (SELECT 1 FROM prompt_tags pt WHERE pt.prompt_id = p.id))
|
||||
ORDER BY p.id
|
||||
"""
|
||||
_SOURCES_SQL = (
|
||||
"SELECT id, positive_prompt_id, TRIM(COALESCE(category, '')) FROM prompt_pairs "
|
||||
"WHERE archived = 0 AND positive_prompt_id IN ({marks}) ORDER BY id"
|
||||
)
|
||||
_PAIR_TAGS_SQL = (
|
||||
"SELECT ppt.pair_id, t.id, t.name FROM prompt_pair_tags ppt "
|
||||
"JOIN tags t ON t.id = ppt.tag_id WHERE ppt.pair_id IN ({marks}) "
|
||||
"ORDER BY ppt.pair_id, t.id"
|
||||
)
|
||||
|
||||
|
||||
def parse_stamp(value):
|
||||
"""A stored timestamp as an aware UTC datetime, or None when unreadable."""
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
return None
|
||||
try:
|
||||
stamp = datetime.datetime.fromisoformat(value.strip())
|
||||
except ValueError:
|
||||
return None
|
||||
if stamp.tzinfo is None:
|
||||
stamp = stamp.replace(tzinfo=datetime.timezone.utc)
|
||||
return stamp
|
||||
|
||||
|
||||
def _close(first, second):
|
||||
"""Whether two stored stamps parse and lie within the tolerance."""
|
||||
a, b = parse_stamp(first), parse_stamp(second)
|
||||
if a is None or b is None:
|
||||
return False
|
||||
return abs((a - b).total_seconds()) <= PAIR_CREATED_TOLERANCE
|
||||
|
||||
|
||||
def _batches(ids):
|
||||
size = _maintenance.ID_BATCH
|
||||
for start in range(0, len(ids), size):
|
||||
yield ids[start : start + size]
|
||||
|
||||
|
||||
def _sources(conn, ids):
|
||||
"""``{prompt_id: [(pair_id, category, ((tag_id, name), ...)), ...]}``."""
|
||||
pairs = {}
|
||||
for batch in _batches(ids):
|
||||
for pair_id, prompt_id, category in conn.execute(
|
||||
_SOURCES_SQL.format(marks=_marks(batch)), batch
|
||||
):
|
||||
pairs.setdefault(prompt_id, []).append([pair_id, category, []])
|
||||
by_pair = {pair[0]: pair for group in pairs.values() for pair in group}
|
||||
pair_ids = sorted(by_pair)
|
||||
for batch in _batches(pair_ids):
|
||||
for pair_id, tag_id, name in conn.execute(
|
||||
_PAIR_TAGS_SQL.format(marks=_marks(batch)), batch
|
||||
):
|
||||
by_pair[pair_id][2].append((tag_id, name))
|
||||
return {
|
||||
prompt_id: [(pid, category, tuple(tags)) for pid, category, tags in group]
|
||||
for prompt_id, group in pairs.items()
|
||||
}
|
||||
|
||||
|
||||
def _field(values, guard):
|
||||
"""``(class, value)`` of one field from its distinct values, or None."""
|
||||
if not values:
|
||||
return None
|
||||
if guard:
|
||||
return (guard, None)
|
||||
if len(values) > 1:
|
||||
return ("conflict", sorted(values))
|
||||
return ("fill", next(iter(values)))
|
||||
|
||||
|
||||
def _classify(row, sources):
|
||||
"""The result of one considered prompt, or None when nothing applies."""
|
||||
guard = None
|
||||
if row["archived"]:
|
||||
guard = "archived"
|
||||
elif not _close(row["updated_at"], row["created_at"]):
|
||||
guard = "edited"
|
||||
category = tags = None
|
||||
tag_pair = None
|
||||
if row["blank_category"]:
|
||||
category = _field({c for _p, c, _t in sources if c}, guard)
|
||||
if row["untagged"]:
|
||||
sets = {}
|
||||
for pair_id, _category, pair_tags in sources:
|
||||
if pair_tags:
|
||||
sets.setdefault(frozenset(pair_tags), pair_id)
|
||||
tags = _field(set(sets), guard)
|
||||
if tags and tags[0] == "fill":
|
||||
tag_pair = sets[tags[1]]
|
||||
if len(tags[1]) > MAX_LIBRARY_TAGS:
|
||||
tags = ("too-many-tags", None)
|
||||
else:
|
||||
tags = ("fill", tuple(sorted(tags[1])))
|
||||
elif tags and tags[0] == "conflict":
|
||||
tags = ("conflict", [sorted(s) for s in sorted(sets, key=sorted)])
|
||||
if category is None and tags is None:
|
||||
return None
|
||||
return {
|
||||
"id": row["id"],
|
||||
"hash": row["hash"],
|
||||
"text": row["text"],
|
||||
"category": category,
|
||||
"tags": tags,
|
||||
"tag_pair": tag_pair if tags and tags[0] == "fill" else None,
|
||||
"pairs": [pair_id for pair_id, _c, _t in sources],
|
||||
}
|
||||
|
||||
|
||||
def _state(conn):
|
||||
"""``(results, fingerprint)``: every considered prompt's result, by id."""
|
||||
considered = []
|
||||
for row in conn.execute(_CANDIDATES_SQL):
|
||||
row = dict(
|
||||
zip(
|
||||
(
|
||||
"id",
|
||||
"hash",
|
||||
"text",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
"blank_category",
|
||||
"untagged",
|
||||
"archived",
|
||||
"first_pair_at",
|
||||
),
|
||||
tuple(row),
|
||||
)
|
||||
)
|
||||
if _close(row["created_at"], row["first_pair_at"]):
|
||||
considered.append(row)
|
||||
sources = _sources(conn, [row["id"] for row in considered])
|
||||
fingerprint = _Fingerprint(OPERATION)
|
||||
results = []
|
||||
for row in considered:
|
||||
result = _classify(row, sources.get(row["id"], []))
|
||||
if result is None:
|
||||
continue
|
||||
results.append(result)
|
||||
fingerprint.add(
|
||||
[
|
||||
result["id"],
|
||||
result["hash"],
|
||||
result["category"],
|
||||
result["tags"],
|
||||
result["tag_pair"],
|
||||
result["pairs"],
|
||||
]
|
||||
)
|
||||
return results, fingerprint.hexdigest()
|
||||
|
||||
|
||||
def _fills(result):
|
||||
return (
|
||||
bool(result["category"] and result["category"][0] == "fill"),
|
||||
bool(result["tags"] and result["tags"][0] == "fill"),
|
||||
)
|
||||
|
||||
|
||||
def _targets(results):
|
||||
targets = []
|
||||
for result in results:
|
||||
category, tags = _fills(result)
|
||||
if category or tags:
|
||||
targets.append([result["id"], category, tags])
|
||||
return targets[:MAX_PLAN_FILLS]
|
||||
|
||||
|
||||
def _protected_classes(result):
|
||||
"""The protected classes of a result's fields, in precedence order."""
|
||||
found = {
|
||||
field[0]
|
||||
for field in (result["category"], result["tags"])
|
||||
if field and field[0] != "fill"
|
||||
}
|
||||
return [key for key in _PROTECTED_ORDER if key in found]
|
||||
|
||||
|
||||
def _more(values, limit):
|
||||
shown = ", ".join(values[:limit])
|
||||
return shown + (", …" if len(values) > limit else "")
|
||||
|
||||
|
||||
def _fill_detail(result):
|
||||
parts = []
|
||||
category, tags = _fills(result)
|
||||
if category:
|
||||
parts.append(f"category “{result['category'][1]}”")
|
||||
if tags:
|
||||
names = sorted(name for _id, name in result["tags"][1])
|
||||
noun = "tag" if len(names) == 1 else "tags"
|
||||
parts.append(f"{len(names)} {noun}: {_more(names, DETAIL_TAGS)}")
|
||||
pairs = [str(pair_id) for pair_id in result["pairs"]]
|
||||
label = "pair" if len(pairs) == 1 else "pairs"
|
||||
return f"Fills {' and '.join(parts)} ({label} {_more(pairs, DETAIL_VALUES)})"
|
||||
|
||||
|
||||
def _protected_detail(result, key):
|
||||
if key != "conflict":
|
||||
return next(detail for k, detail, *_r in PROTECTED if k == key)
|
||||
if result["category"] and result["category"][0] == "conflict":
|
||||
values = result["category"][1]
|
||||
return f"Kept: pairs disagree on category: {_more(values, DETAIL_VALUES)}"
|
||||
return "Kept: pairs disagree on tags"
|
||||
|
||||
|
||||
def _details(results, targets):
|
||||
rows = []
|
||||
for key in _PROTECTED_ORDER:
|
||||
for result in results:
|
||||
if key in _protected_classes(result):
|
||||
rows.append(
|
||||
{
|
||||
"label": _excerpt(result["text"]),
|
||||
"detail": _protected_detail(result, key),
|
||||
}
|
||||
)
|
||||
planned = {target[0] for target in targets}
|
||||
for result in results:
|
||||
if result["id"] in planned:
|
||||
rows.append(
|
||||
{"label": _excerpt(result["text"]), "detail": _fill_detail(result)}
|
||||
)
|
||||
if not targets:
|
||||
return rows[:MAX_DETAILS]
|
||||
return rows[: MAX_DETAILS - 1] + [{"label": "Record", "detail": RECORD_NOTE}]
|
||||
|
||||
|
||||
def _scope_label(targets, total):
|
||||
if not targets:
|
||||
return EMPTY_LABEL
|
||||
categories = sum(1 for _id, category, _tags in targets if category)
|
||||
tag_sets = sum(1 for _id, _category, tags in targets if tags)
|
||||
label = (
|
||||
f"{_plural(len(targets), 'prompt')} to fill: "
|
||||
f"{_plural(categories, 'category', 'categories')}, "
|
||||
f"{_plural(tag_sets, 'tag set')}"
|
||||
)
|
||||
if total > len(targets):
|
||||
label += (
|
||||
f" (the first {len(targets):,} of {total:,}; review again for the rest)"
|
||||
)
|
||||
return label
|
||||
|
||||
|
||||
def plan_pair_metadata(conn):
|
||||
"""The reviewed ``pair_metadata`` plan on ``conn``'s snapshot."""
|
||||
results, fingerprint = _state(conn)
|
||||
targets = _targets(results)
|
||||
total = sum(1 for result in results if any(_fills(result)))
|
||||
counts = {key: 0 for key in _PROTECTED_ORDER}
|
||||
held = 0
|
||||
for result in results:
|
||||
classes = _protected_classes(result)
|
||||
held += bool(classes)
|
||||
for key in classes:
|
||||
counts[key] += 1
|
||||
reasons = [
|
||||
f"{counts[key]:,} {one if counts[key] == 1 else many}"
|
||||
for key, _detail, one, many in PROTECTED
|
||||
if counts[key]
|
||||
]
|
||||
return MaintenanceDraft(
|
||||
operation=OPERATION,
|
||||
fingerprint=fingerprint,
|
||||
targets=targets,
|
||||
scope={"count": len(targets), "label": _scope_label(targets, total)},
|
||||
protected={"count": held, "reasons": reasons},
|
||||
media={"deletes_files": False, "label": MEDIA_LABEL},
|
||||
details=_details(results, targets),
|
||||
)
|
||||
|
||||
|
||||
def _fill_categories(conn, fills):
|
||||
"""Set each blank category; raises PlanChanged when one is no longer blank."""
|
||||
cursor = conn.executemany(
|
||||
"UPDATE prompts SET category = ? WHERE id = ? "
|
||||
"AND COALESCE(TRIM(category), '') = ''",
|
||||
fills,
|
||||
)
|
||||
if cursor.rowcount != len(fills):
|
||||
raise PlanChanged(OPERATION)
|
||||
|
||||
|
||||
def _fill_tags(conn, fills, expected):
|
||||
"""Copy each source pair's tags onto its untagged prompt."""
|
||||
cursor = conn.executemany(
|
||||
"INSERT INTO prompt_tags(prompt_id, tag_id) "
|
||||
"SELECT ?, tag_id FROM prompt_pair_tags WHERE pair_id = ? "
|
||||
"AND NOT EXISTS (SELECT 1 FROM prompt_tags WHERE prompt_id = ?)",
|
||||
fills,
|
||||
)
|
||||
if cursor.rowcount != expected:
|
||||
raise PlanChanged(OPERATION)
|
||||
|
||||
|
||||
def apply_pair_metadata(conn, fingerprint, targets):
|
||||
"""Apply the reviewed fills on ``conn`` (inside BEGIN IMMEDIATE).
|
||||
|
||||
Returns ``{changed, categories, tag_sets, rows}``; ``rows`` lists every
|
||||
filled prompt (``{prompt_id, category?, tags?}``), never capped.
|
||||
"""
|
||||
results, current = _state(conn)
|
||||
if current != fingerprint or _targets(results) != targets:
|
||||
raise PlanChanged(OPERATION)
|
||||
by_id = {result["id"]: result for result in results}
|
||||
categories, tags, rows = [], [], []
|
||||
tag_count = 0
|
||||
for prompt_id, fill_category, fill_tags in targets:
|
||||
result = by_id[prompt_id]
|
||||
row = {"prompt_id": prompt_id}
|
||||
if fill_category:
|
||||
row["category"] = result["category"][1]
|
||||
categories.append((row["category"], prompt_id))
|
||||
if fill_tags:
|
||||
row["tags"] = sorted(name for _id, name in result["tags"][1])
|
||||
tags.append((prompt_id, result["tag_pair"], prompt_id))
|
||||
tag_count += len(result["tags"][1])
|
||||
rows.append(row)
|
||||
if categories:
|
||||
_fill_categories(conn, categories)
|
||||
if tags:
|
||||
_fill_tags(conn, tags, tag_count)
|
||||
return {
|
||||
"changed": len(rows),
|
||||
"categories": len(categories),
|
||||
"tag_sets": len(tags),
|
||||
"rows": rows,
|
||||
}
|
||||
+30
-17
@@ -204,6 +204,8 @@ class Job:
|
||||
self.total = 0
|
||||
self.counters = dict.fromkeys(groups, 0)
|
||||
self.outcomes = {group: [] for group in groups}
|
||||
# Groups whose report lists every row, beyond MAX_OUTCOME_ROWS (#550).
|
||||
self.complete = {}
|
||||
self.log = []
|
||||
self._cancel_requested = False
|
||||
self.last_event = None
|
||||
@@ -297,31 +299,36 @@ class Job:
|
||||
)
|
||||
self.log_line(log_line)
|
||||
|
||||
def record_counted(self, group, count, rows=()):
|
||||
def record_counted(self, group, count, rows=(), *, complete=False):
|
||||
"""Count ``count`` outcomes in ``group`` at once, listing ``rows``.
|
||||
|
||||
For outcomes that need no log line each (a thumbnail job's already
|
||||
fresh previews): the counter is exact, ``rows`` (``record``'s row
|
||||
shape) are listed up to MAX_OUTCOME_ROWS, and nothing is logged.
|
||||
``complete`` also keeps every row for the report (#550: a record of
|
||||
each change that must not be cut short; the caller bounds it).
|
||||
"""
|
||||
if group not in self.counters:
|
||||
raise ValueError(f"Unknown outcome group {group!r} for {self.kind}")
|
||||
self.counters[group] += max(0, int(count))
|
||||
listed = self.outcomes[group]
|
||||
kept = self.complete.setdefault(group, []) if complete else None
|
||||
for row in rows:
|
||||
if len(listed) >= MAX_OUTCOME_ROWS:
|
||||
if kept is None and len(listed) >= MAX_OUTCOME_ROWS:
|
||||
break
|
||||
index = row.get("root_index")
|
||||
if isinstance(index, bool) or not isinstance(index, int) or index < 0:
|
||||
index = None
|
||||
listed.append(
|
||||
{
|
||||
"path": _relative_path(row.get("path")),
|
||||
"root": str(row.get("root") or ""),
|
||||
"root_index": index,
|
||||
"detail": str(row.get("detail") or ""),
|
||||
}
|
||||
)
|
||||
shaped = {
|
||||
"path": _relative_path(row.get("path")),
|
||||
"root": str(row.get("root") or ""),
|
||||
"root_index": index,
|
||||
"detail": str(row.get("detail") or ""),
|
||||
}
|
||||
if kept is not None:
|
||||
kept.append(shaped)
|
||||
if len(listed) < MAX_OUTCOME_ROWS:
|
||||
listed.append(dict(shaped))
|
||||
|
||||
def finish(self, state="finished"):
|
||||
"""End the job in ``state`` and release every subscriber.
|
||||
@@ -406,8 +413,9 @@ class Job:
|
||||
"""The downloadable report: the summary, every outcome group and the log.
|
||||
|
||||
Each group carries its exact count, its listed rows (at most
|
||||
MAX_OUTCOME_ROWS) and whether the list is cut short. Rows and log lines
|
||||
already hold root-relative paths only.
|
||||
MAX_OUTCOME_ROWS, or every row of a ``complete`` group) and whether
|
||||
the list is cut short. Rows and log lines already hold root-relative
|
||||
paths only.
|
||||
"""
|
||||
summary = self.summary()
|
||||
summary.pop("listed", None)
|
||||
@@ -417,16 +425,21 @@ class Job:
|
||||
"generated_at": self._clock(),
|
||||
"job": summary,
|
||||
"outcomes": {
|
||||
group: {
|
||||
"count": self.counters.get(group, len(rows)),
|
||||
"rows": [dict(row) for row in rows],
|
||||
"truncated": self.counters.get(group, len(rows)) > len(rows),
|
||||
}
|
||||
group: self._report_group(group, rows)
|
||||
for group, rows in self.outcomes.items()
|
||||
},
|
||||
"log": list(self.log),
|
||||
}
|
||||
|
||||
def _report_group(self, group, listed):
|
||||
rows = self.complete.get(group, listed)
|
||||
count = self.counters.get(group, len(rows))
|
||||
return {
|
||||
"count": count,
|
||||
"rows": [dict(row) for row in rows],
|
||||
"truncated": count > len(rows),
|
||||
}
|
||||
|
||||
def report_filename(self):
|
||||
"""The download name: prompt-manager-KIND-YYYYMMDD-HHMM.json (local time)."""
|
||||
stamp = time.strftime("%Y%m%d-%H%M", time.localtime(self._clock()))
|
||||
|
||||
+47
-5
@@ -1,8 +1,9 @@
|
||||
"""Maintenance routes: reviewed, read-only plans (#249 System S6).
|
||||
|
||||
``POST /prompt_manager/maintenance/preview {operation, options:{}}`` reviews
|
||||
one operation (``duplicate_prompts``, ``optimize``, ``unused_components`` or
|
||||
``missing_links``; ``database/maintenance.py``; ``duplicate_files``,
|
||||
one operation (``duplicate_prompts``, ``optimize``, ``unused_components``,
|
||||
``missing_links`` or ``pair_metadata``, #550; ``database/maintenance.py``;
|
||||
``duplicate_files``,
|
||||
``py/api/duplicate_files.py``; ``rotated_logs``, the rotated log copies,
|
||||
``py/api/log_cleanup.py``, which has its own apply job) off the event loop and
|
||||
answers a ``Plan``: ``{success, token, operation, expires_at, scope:{count,
|
||||
@@ -97,7 +98,11 @@ MAINTENANCE_TITLES = {
|
||||
"unused_components": "Unused prompt components",
|
||||
"missing_links": "Missing-file links",
|
||||
"workflow_storage": "Store each workflow once",
|
||||
# #550, beyond the concept: fills old Pair prompts' empty category and tags.
|
||||
"pair_metadata": "Pair category and tags",
|
||||
}
|
||||
# Operations whose job keeps every changed row for its report (#550).
|
||||
COMPLETE_RECORD = frozenset({"pair_metadata"})
|
||||
_OPERATION_ERROR = "operation must be one of: " + ", ".join(PREVIEW_OPERATIONS)
|
||||
MAINTENANCE_GROUPS = ("changed", "protected", "errors")
|
||||
MAX_TOKEN_LENGTH = 128
|
||||
@@ -281,9 +286,27 @@ def short_space_message(short):
|
||||
)
|
||||
|
||||
|
||||
def _changed_line(operation, count, marks=0):
|
||||
def _pair_metadata_line(result):
|
||||
count = result["changed"]
|
||||
noun = (
|
||||
"prompt from its saved pairs"
|
||||
if count == 1
|
||||
else "prompts from their saved pairs"
|
||||
)
|
||||
categories = result.get("categories", 0)
|
||||
tag_sets = result.get("tag_sets", 0)
|
||||
return (
|
||||
f"Filled {count:,} {noun} "
|
||||
f"({categories:,} {'category' if categories == 1 else 'categories'}, "
|
||||
f"{tag_sets:,} {'tag set' if tag_sets == 1 else 'tag sets'})"
|
||||
)
|
||||
|
||||
|
||||
def _changed_line(operation, count, marks=0, result=None):
|
||||
if operation == "optimize":
|
||||
return "Optimized the database"
|
||||
if operation == "pair_metadata":
|
||||
return _pair_metadata_line(result or {"changed": count})
|
||||
if operation == "duplicate_prompts":
|
||||
noun = "duplicate prompt" if count == 1 else "duplicate prompts"
|
||||
return f"Merged {count:,} {noun}"
|
||||
@@ -668,9 +691,16 @@ class MaintenanceRoutesMixin:
|
||||
rows = await self._run_in_executor(
|
||||
self._maintenance_rows, plan.operation, result["rows"]
|
||||
)
|
||||
job.record_counted("changed", result["changed"], rows)
|
||||
job.record_counted(
|
||||
"changed",
|
||||
result["changed"],
|
||||
rows,
|
||||
complete=plan.operation in COMPLETE_RECORD,
|
||||
)
|
||||
job.record_counted("protected", payload["protected"])
|
||||
line = _changed_line(plan.operation, result["changed"], result.get("marks", 0))
|
||||
line = _changed_line(
|
||||
plan.operation, result["changed"], result.get("marks", 0), result
|
||||
)
|
||||
job.log_line(line)
|
||||
job.update(progress=100, status_line=f"{line} · {backup_note(backup)}")
|
||||
return None
|
||||
@@ -691,6 +721,8 @@ class MaintenanceRoutesMixin:
|
||||
}
|
||||
)
|
||||
return rows
|
||||
if operation == "pair_metadata":
|
||||
return [{"path": "", "detail": _filled_detail(item)} for item in described]
|
||||
if operation == "duplicate_prompts":
|
||||
return [
|
||||
{
|
||||
@@ -708,6 +740,16 @@ class MaintenanceRoutesMixin:
|
||||
]
|
||||
|
||||
|
||||
def _filled_detail(item):
|
||||
"""``Prompt 12: category “portrait”; tags a, b``: one filled prompt (#550)."""
|
||||
parts = []
|
||||
if "category" in item:
|
||||
parts.append(f"category “{item['category']}”")
|
||||
if item.get("tags"):
|
||||
parts.append("tags " + ", ".join(item["tags"]))
|
||||
return f"Prompt {item['prompt_id']}: " + "; ".join(parts)
|
||||
|
||||
|
||||
def backup_note(backup):
|
||||
"""The status line's backup part: the file name, or that there is none."""
|
||||
return f"Safety backup {backup}" if backup else NO_BACKUP
|
||||
|
||||
@@ -64,11 +64,31 @@ function measureMaintenance(mainSelector) {
|
||||
};
|
||||
}
|
||||
|
||||
/** D102: the Metadata cache row says why its review is unavailable and its button is aria-disabled (D93's dimmed style). */
|
||||
function mapMaintenanceConcept(reference) {
|
||||
/**
|
||||
* D192 (#550, beyond the concept): Database adds "Pair category and tags" after "Unused prompt components", shaped like
|
||||
* the concept's plain Review plan row (Optimize database's).
|
||||
*/
|
||||
function mapPairRow(reference) {
|
||||
return {
|
||||
...reference,
|
||||
surfaces: reference.surfaces.map(surface => ({ ...surface, rows: surface.rows.map(row => row.title !== 'Metadata cache' ? row : {
|
||||
surfaces: reference.surfaces.map(surface => {
|
||||
const plain = surface.rows.find(row => row.title === 'Optimize database');
|
||||
const at = surface.rows.findIndex(row => row.title === 'Unused prompt components');
|
||||
if (!plain || at < 0) return surface;
|
||||
return { ...surface, tree: [...surface.tree, 'div.operation'], rows: [...surface.rows.slice(0, at + 1),
|
||||
{ ...plain, title: 'Pair category and tags', description: 'Fill an empty category or tags on prompts older ' +
|
||||
'Pair node runs saved, from their saved pairs. Only empty fields of never-edited prompts.' },
|
||||
...surface.rows.slice(at + 1)] };
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
/** D102: the Metadata cache row says why its review is unavailable and its button is aria-disabled (D93's dimmed style). */
|
||||
function mapMaintenanceConcept(reference) {
|
||||
const mapped = mapPairRow(reference);
|
||||
return {
|
||||
...mapped,
|
||||
surfaces: mapped.surfaces.map(surface => ({ ...surface, rows: surface.rows.map(row => row.title !== 'Metadata cache' ? row : {
|
||||
...row, description: `${row.description} Nothing is cached in this browser.`,
|
||||
actions: row.actions.map(action => ({ ...action, opacity: '0.55', cursor: 'not-allowed' })) }) })),
|
||||
};
|
||||
@@ -209,8 +229,8 @@ test('maintenance comparison detects mutations', async ({ page, request }) => {
|
||||
const expected = mapMaintenanceConcept(concept);
|
||||
const production = await productionMaintenance(page, 'dark');
|
||||
expect(diffInventory(expected, production)).toEqual([]);
|
||||
// Unmapped, the section differs only on the D102 row.
|
||||
expect(diffInventory(concept, production).map(item => item.path).sort()).toEqual([
|
||||
// Mapped only for the D192 row, the section differs only on the D102 row.
|
||||
expect(diffInventory(mapPairRow(concept), production).map(item => item.path).sort()).toEqual([
|
||||
'surfaces.1.rows.3.actions.0.cursor', 'surfaces.1.rows.3.actions.0.opacity', 'surfaces.1.rows.3.description']);
|
||||
const measure = () => page.evaluate(measureMaintenance, '#workspace-main');
|
||||
const paths = async () => diffInventory(expected, await measure()).map(item => item.path);
|
||||
|
||||
@@ -64,7 +64,9 @@ test('the section shows the concept rows; Metadata cache follows Q5 (D102)', asy
|
||||
item.querySelector('.row-actions > button').textContent, item.querySelector('.row-actions > button').className]),
|
||||
]))).toEqual([
|
||||
['Database', [['Integrity & relationship checks', 'Check', ''], ['Duplicate prompt text', 'Review plan', ''],
|
||||
['Optimize database', 'Review plan', ''], ['Unused prompt components', 'Review plan', 'danger']]],
|
||||
['Optimize database', 'Review plan', ''], ['Unused prompt components', 'Review plan', 'danger'],
|
||||
// D192 (#550): beyond the concept.
|
||||
['Pair category and tags', 'Review plan', '']]],
|
||||
['Images & cache', [['Missing-file links', 'Review plan', 'danger'], ['Duplicate image files', 'Find duplicates', ''],
|
||||
['Thumbnail cache', 'Review options', ''], ['Metadata cache', 'Review plan', '']]],
|
||||
]);
|
||||
@@ -177,6 +179,53 @@ test("plan dialog shows the server's scope, protected, backup policy and media l
|
||||
expect(await serverJson(request, '/__test__/system/digest')).toEqual(before);
|
||||
});
|
||||
|
||||
test('D192: Pair category and tags reviews old Pair prompts, keeps an edited one and fills the other (#550)',
|
||||
async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/system/pair-metadata')).toBeOK();
|
||||
const answers = [];
|
||||
page.on('response', response => { if (response.url().endsWith(PREVIEW_URL)) answers.push(response.json()); });
|
||||
await openMaintenance(page);
|
||||
const pairRow = row(page, 'Pair category and tags');
|
||||
await expect(pairRow.locator('p')).toHaveText('Fill an empty category or tags on prompts older Pair node runs saved, ' +
|
||||
'from their saved pairs. Only empty fields of never-edited prompts.');
|
||||
await pairRow.getByRole('button', { name: 'Review plan', exact: true }).click();
|
||||
const plan = dialog(page);
|
||||
await expect(plan.locator('#modal-body > h3')).toHaveText('Pair category and tags');
|
||||
const answer = await answers[0];
|
||||
expect(answer.operation).toBe('pair_metadata');
|
||||
expect(await gridOf(page)).toEqual([
|
||||
['Scope', '1 prompt to fill: 1 category, 1 tag set'],
|
||||
['Protected', '1 edited after it was saved'],
|
||||
['Backup', `Create and verify before changes (${answer.backup.destination_label})`],
|
||||
['Media on disk', 'No image file or saved pair is changed.'],
|
||||
]);
|
||||
await expect(plan.locator('details > summary')).toHaveText('Listed records (3)');
|
||||
await plan.locator('details > summary').click();
|
||||
// Protected rows come first, the Record row last.
|
||||
const lines = (await plan.locator('details > pre').textContent()).split('\n');
|
||||
expect(lines).toHaveLength(3);
|
||||
expect(lines[0]).toBe('S550 harbor · Kept: edited after it was saved; its empty category or tags may be intentional');
|
||||
expect(lines[1]).toMatch(/^S550 lantern · Fills category “night” and 2 tags: s550-glow, s550-warm \(pair \d+\)$/);
|
||||
expect(lines[2]).toMatch(/^Record · The job's report lists every filled field\. Download it to keep it/);
|
||||
await plan.getByRole('button', { name: 'Apply reviewed operation', exact: true }).click();
|
||||
await expect(dialog(page)).toHaveCount(0);
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
const job = await ended(request, page.url().split('/').pop());
|
||||
expect([job.title, job.state, job.counters.changed, job.counters.protected])
|
||||
.toEqual(['Pair category and tags', 'finished', 1, 1]);
|
||||
const main = page.locator('#workspace-main');
|
||||
await expect(main.locator('h1')).toHaveText('Pair category and tags');
|
||||
await expect(main.getByRole('button', { name: 'Download report', exact: true })).toBeVisible();
|
||||
const library = await serverJson(request, '/prompt_manager/library');
|
||||
const lantern = library.items.find(item => item.text === 'S550 lantern');
|
||||
const harbor = library.items.find(item => item.text === 'S550 harbor');
|
||||
const filled = (await serverJson(request, `/prompt_manager/library/${lantern.id}?archive=all`)).positive;
|
||||
expect([filled.category, [...filled.tags].sort()]).toEqual(['night', ['s550-glow', 's550-warm']]);
|
||||
expect((await serverJson(request, `/prompt_manager/library/${harbor.id}?archive=all`)).positive.category).toBeFalsy();
|
||||
await page.goto(`${workspace}#/library`);
|
||||
await expect(page.locator(`article[data-prompt-id="${lantern.id}"] .card-title > .chip`)).toHaveText('night');
|
||||
});
|
||||
|
||||
test('a plan with nothing to do shows only Close, never Apply', async ({ page, request }) => {
|
||||
const applies = [];
|
||||
page.on('request', item => { if (item.url().endsWith(APPLY_URL)) applies.push(item.url()); });
|
||||
|
||||
@@ -41,6 +41,11 @@ read. These per-server hooks only change what a System test needs on top:
|
||||
- ``POST /__test__/system/duplicate-prompts`` (S7) adds two prompts whose
|
||||
text differs only in case and spacing (a duplicate prompt text group);
|
||||
``{"again": true}`` adds a third, so a reviewed plan no longer matches;
|
||||
- ``POST /__test__/system/pair-metadata`` (#550) saves two saved pairs the
|
||||
way the Pair node did before #546 (the pair holds the category and tags,
|
||||
its new positive prompt none): ``S550 lantern`` (category ``night``, tags
|
||||
``s550-glow``, ``s550-warm``) fills; ``S550 harbor`` was edited after it was
|
||||
saved, so the plan keeps it;
|
||||
- ``POST /__test__/system/duplicate-files`` (S7) writes one PNG's bytes three
|
||||
times: ``output/s7-dupes/same.png``, ``input/s7-dupes/same.png`` (the same
|
||||
relative path in the other source) and ``output/s7-dupes/copy.png``;
|
||||
@@ -344,6 +349,25 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
conn.commit()
|
||||
return web.json_response({"success": True})
|
||||
|
||||
@routes.post("/__test__/system/pair-metadata")
|
||||
async def pair_metadata(request):
|
||||
def seed():
|
||||
api.db.save_prompt_pair(
|
||||
"S550 lantern",
|
||||
"S550 blurry",
|
||||
category="night",
|
||||
tags=["s550-glow", "s550-warm"],
|
||||
)
|
||||
api.db.save_prompt_pair("S550 harbor", category="night")
|
||||
with api.db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE prompts SET updated_at='2099-01-01T00:00:00+00:00' "
|
||||
"WHERE text='S550 harbor'"
|
||||
)
|
||||
|
||||
await api._run_in_executor(seed)
|
||||
return web.json_response({"success": True})
|
||||
|
||||
@routes.post("/__test__/system/duplicate-files")
|
||||
async def duplicate_files(request):
|
||||
body = await request.json() if request.can_read_body else {}
|
||||
|
||||
@@ -21,8 +21,23 @@ test('the section lists the concept rows in its order, labels and danger marks',
|
||||
assert.deepEqual(MAINTENANCE_SURFACES.map(([heading]) => heading), surfaces.map(text => text.replace('&', '&')));
|
||||
const production = MAINTENANCE_SURFACES.flatMap(([, rows]) => rows.map(([title, description, label, , danger = false]) =>
|
||||
[title, description, label, danger]));
|
||||
assert.deepEqual(production, concept.map(([title, ...rest]) => [title.replace('&', '&'), ...rest]));
|
||||
assert.equal(production.length, 8);
|
||||
// D192 (#550, beyond the concept): "Pair category and tags" follows "Unused prompt components" in Database.
|
||||
const expected = concept.map(([title, ...rest]) => [title.replace('&', '&'), ...rest]);
|
||||
expected.splice(expected.findIndex(([title]) => title === 'Unused prompt components') + 1, 0, PAIR_ROW);
|
||||
assert.deepEqual(production, expected);
|
||||
assert.equal(production.length, 9);
|
||||
});
|
||||
|
||||
const PAIR_ROW = Object.freeze(['Pair category and tags', 'Fill an empty category or tags on prompts older Pair node runs ' +
|
||||
'saved, from their saved pairs. Only empty fields of never-edited prompts.', 'Review plan', false]);
|
||||
|
||||
test('D192: the Pair category and tags row reviews the pair_metadata plan under its title (#550)', async () => {
|
||||
const { MAINTENANCE_SURFACES } = await maintenance;
|
||||
const { OPERATION_TITLES } = await review;
|
||||
const [, database] = MAINTENANCE_SURFACES[0];
|
||||
const row = database.find(([title]) => title === PAIR_ROW[0]);
|
||||
assert.deepEqual(row, [PAIR_ROW[0], PAIR_ROW[1], PAIR_ROW[2], 'pair_metadata']);
|
||||
assert.equal(OPERATION_TITLES.pair_metadata, 'Pair category and tags');
|
||||
});
|
||||
|
||||
const PLAN = Object.freeze({
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Pair category and tags repair through the maintenance routes (#550).
|
||||
|
||||
``pair_metadata`` is reviewed and applied like the other maintenance plans:
|
||||
``options`` must be ``{}``, the job is titled "Pair category and tags", the
|
||||
safety backup follows the ``maintenance.safety_backup`` policy, and the job's
|
||||
downloadable report lists every filled prompt, never cut at 200 rows.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
_mock_server = MagicMock()
|
||||
_mock_server.PromptServer.instance.routes = MagicMock()
|
||||
sys.modules.setdefault("server", _mock_server)
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from py.api.jobs import MAX_OUTCOME_ROWS # noqa: E402
|
||||
from py.api.maintenance import MAINTENANCE_TITLES # noqa: E402
|
||||
from py.config import PromptManagerConfig # noqa: E402
|
||||
|
||||
try:
|
||||
from tests.test_api_maintenance import WAIT, ApplyTestCase # noqa: E402
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_maintenance import WAIT, ApplyTestCase # noqa: E402
|
||||
|
||||
OPERATION = "pair_metadata"
|
||||
TITLE = "Pair category and tags"
|
||||
|
||||
|
||||
class PairMetadataAPITestCase(ApplyTestCase):
|
||||
def pair(self, positive, negative="", **metadata):
|
||||
pair_id = self.api.db.save_prompt_pair(positive, negative, **metadata)
|
||||
with self.api.db.model.get_connection() as conn:
|
||||
prompt = conn.execute(
|
||||
"SELECT positive_prompt_id FROM prompt_pairs WHERE id=?", (pair_id,)
|
||||
).fetchone()[0]
|
||||
return pair_id, prompt
|
||||
|
||||
def category(self, prompt_id):
|
||||
with self.api.db.model.get_connection() as conn:
|
||||
return conn.execute(
|
||||
"SELECT category FROM prompts WHERE id=?", (prompt_id,)
|
||||
).fetchone()[0]
|
||||
|
||||
|
||||
class TestPairMetadataRoutes(PairMetadataAPITestCase):
|
||||
async def test_preview_accepts_the_operation_and_no_options(self):
|
||||
self.assertEqual(MAINTENANCE_TITLES[OPERATION], TITLE)
|
||||
self.pair("a cat", category="portrait", tags=["a"])
|
||||
plan = await self.preview(OPERATION)
|
||||
self.assertEqual(plan["operation"], OPERATION)
|
||||
self.assertEqual(plan["scope"]["count"], 1)
|
||||
self.assertEqual(plan["backup"]["policy"], "on")
|
||||
refused = await self.preview(OPERATION, 400, options={"all": True})
|
||||
self.assertEqual(refused["code"], "invalid-request")
|
||||
|
||||
async def test_apply_fills_with_a_backup_and_names_it(self):
|
||||
_pair, prompt = self.pair("a cat", "neg", category="portrait", tags=["a", "b"])
|
||||
plan, summary, job = await self.run_apply(OPERATION)
|
||||
self.assertEqual(summary["title"], TITLE)
|
||||
self.assertEqual(summary["admission"], "records")
|
||||
self.assertEqual(job.state, "finished", job.log)
|
||||
self.assertEqual(self.category(prompt), "portrait")
|
||||
[backup] = self.backups()
|
||||
self.assertEqual(
|
||||
job.status_line,
|
||||
"Filled 1 prompt from its saved pairs (1 category, 1 tag set) · "
|
||||
f"Safety backup {backup}",
|
||||
)
|
||||
self.assertEqual(
|
||||
job.outcomes["changed"][0]["detail"],
|
||||
f"Prompt {prompt}: category “portrait”; tags a, b",
|
||||
)
|
||||
self.assertEqual(job.counters["changed"], 1)
|
||||
|
||||
async def test_protected_prompts_are_counted(self):
|
||||
_a, prompt = self.pair("conflict", "n1", category="x")
|
||||
self.pair("conflict", "n2", category="y")
|
||||
self.pair("fill me", category="portrait")
|
||||
plan, _summary, job = await self.run_apply(OPERATION)
|
||||
self.assertEqual(plan["protected"]["reasons"], ["1 where pairs disagree"])
|
||||
self.assertEqual(job.counters["protected"], 1)
|
||||
self.assertIsNone(self.category(prompt))
|
||||
|
||||
async def test_policy_off_writes_without_a_backup(self):
|
||||
PromptManagerConfig.SAFETY_BACKUP = False
|
||||
_pair, prompt = self.pair("no backup", category="portrait")
|
||||
plan, _summary, job = await self.run_apply(OPERATION)
|
||||
self.assertEqual(plan["backup"]["policy"], "off")
|
||||
self.assertEqual(job.state, "finished", job.log)
|
||||
self.assertIn("No safety backup (the policy is off)", "\n".join(job.log))
|
||||
self.assertTrue(job.status_line.endswith("· No safety backup"))
|
||||
self.assertEqual(self.backups(), [])
|
||||
self.assertEqual(self.category(prompt), "portrait")
|
||||
|
||||
async def test_reviewed_with_the_policy_on_keeps_the_backup(self):
|
||||
self.pair("promise", category="portrait")
|
||||
plan = await self.preview(OPERATION)
|
||||
PromptManagerConfig.SAFETY_BACKUP = False
|
||||
body = await self.apply(plan["token"])
|
||||
job = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
self.assertEqual(len(self.backups()), 1)
|
||||
|
||||
async def test_plan_changed_writes_nothing(self):
|
||||
pair_id, prompt = self.pair("drift", category="portrait")
|
||||
plan = await self.preview(OPERATION)
|
||||
self.api.db.update_prompt_pair_metadata(pair_id, category="landscape")
|
||||
body = await self.apply(plan["token"])
|
||||
job = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
self.assertEqual(job.state, "failed")
|
||||
self.assertEqual(job.last_event["code"], "plan-changed")
|
||||
self.assertIsNone(self.category(prompt))
|
||||
|
||||
async def test_report_lists_every_filled_prompt(self):
|
||||
PromptManagerConfig.SAFETY_BACKUP = False
|
||||
count = MAX_OUTCOME_ROWS + 5
|
||||
for n in range(count):
|
||||
self.pair(f"bulk {n:03d}", category="portrait")
|
||||
_plan, _summary, job = await self.run_apply(OPERATION)
|
||||
self.assertEqual(job.state, "finished", job.log)
|
||||
self.assertEqual(len(job.outcomes["changed"]), MAX_OUTCOME_ROWS)
|
||||
response = await self.client.request(
|
||||
"GET", f"/prompt_manager/jobs/{job.id}/report"
|
||||
)
|
||||
self.assertEqual(response.status, 200)
|
||||
report = await response.json()
|
||||
changed = report["outcomes"]["changed"]
|
||||
self.assertEqual(changed["count"], count)
|
||||
self.assertEqual(len(changed["rows"]), count)
|
||||
self.assertFalse(changed["truncated"])
|
||||
self.assertTrue(
|
||||
all(
|
||||
row["detail"].endswith("category “portrait”") for row in changed["rows"]
|
||||
)
|
||||
)
|
||||
@@ -490,6 +490,22 @@ class TestJobOutcomesAndLog(unittest.TestCase):
|
||||
with self.assertRaises(ValueError):
|
||||
job.record_counted("bogus", 1)
|
||||
|
||||
def test_record_counted_complete_keeps_every_row_for_the_report(self):
|
||||
# #550: a repair's record of each change is never cut at the cap.
|
||||
job = Job("maintenance", "Pair category and tags", groups=("changed", "errors"))
|
||||
rows = [
|
||||
{"path": "", "detail": f"Prompt {i}"} for i in range(MAX_OUTCOME_ROWS + 5)
|
||||
]
|
||||
job.record_counted("changed", len(rows), rows, complete=True)
|
||||
self.assertEqual(len(job.outcomes["changed"]), MAX_OUTCOME_ROWS)
|
||||
report = job.report()["outcomes"]
|
||||
self.assertEqual(len(report["changed"]["rows"]), MAX_OUTCOME_ROWS + 5)
|
||||
self.assertEqual(
|
||||
report["changed"]["rows"][-1]["detail"], f"Prompt {MAX_OUTCOME_ROWS + 4}"
|
||||
)
|
||||
self.assertFalse(report["changed"]["truncated"])
|
||||
self.assertEqual(report["errors"], {"count": 0, "rows": [], "truncated": False})
|
||||
|
||||
def test_thumbnail_job_groups(self):
|
||||
self.assertEqual(
|
||||
jobs_module.THUMBNAIL_GROUP_NAMES, ("generated", "skipped", "errors")
|
||||
|
||||
@@ -0,0 +1,467 @@
|
||||
"""Pair category and tags repair (#550): the ``pair_metadata`` maintenance plan.
|
||||
|
||||
Before #546 a Pair node run saved its category and tags on the saved pair
|
||||
only, so the positive prompt it created stayed Uncategorized and untagged.
|
||||
``pair_metadata`` reviews those prompts and fills only their empty fields
|
||||
from the prompt's non-archived pairs when every pair with a value agrees.
|
||||
It never touches a negative, a saved pair or ``updated_at``, and a prompt
|
||||
the Pair node did not create, that was edited, archived, or whose pairs
|
||||
disagree, is kept (``550-design.md``).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from database import maintenance # noqa: E402
|
||||
from database import pair_metadata_repair as repair # noqa: E402
|
||||
|
||||
try:
|
||||
from tests.test_maintenance_plans import MaintenanceDatabaseCase # noqa: E402
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_maintenance_plans import MaintenanceDatabaseCase # noqa: E402
|
||||
|
||||
OPERATION = "pair_metadata"
|
||||
|
||||
|
||||
class PairMetadataCase(MaintenanceDatabaseCase):
|
||||
def pair(self, positive, negative="", **metadata):
|
||||
"""A pre-#546 Pair save: the pair gets the metadata, the prompt none."""
|
||||
pair_id = self.db.save_prompt_pair(positive, negative, **metadata)
|
||||
return pair_id, self.positive_of(pair_id)
|
||||
|
||||
def positive_of(self, pair_id):
|
||||
return self.sql(
|
||||
"SELECT positive_prompt_id FROM prompt_pairs WHERE id=?", pair_id
|
||||
)[0][0]
|
||||
|
||||
def sql(self, statement, *args):
|
||||
with self.db.model.get_connection() as conn:
|
||||
return [tuple(row) for row in conn.execute(statement, args)]
|
||||
|
||||
def run_sql(self, statement, *args):
|
||||
with self.db.model.get_connection() as conn:
|
||||
conn.execute(statement, args)
|
||||
|
||||
def category(self, prompt_id):
|
||||
return self.sql("SELECT category FROM prompts WHERE id=?", prompt_id)[0][0]
|
||||
|
||||
def tags(self, prompt_id):
|
||||
return sorted(
|
||||
row[0]
|
||||
for row in self.sql(
|
||||
"SELECT t.name FROM prompt_tags pt JOIN tags t ON t.id=pt.tag_id "
|
||||
"WHERE pt.prompt_id=?",
|
||||
prompt_id,
|
||||
)
|
||||
)
|
||||
|
||||
def preview(self, operation=OPERATION, roots=None):
|
||||
return self.db.preview_maintenance(operation, [])
|
||||
|
||||
def apply(self, draft):
|
||||
return self.db.apply_maintenance(
|
||||
draft.operation, draft.fingerprint, draft.targets, []
|
||||
)
|
||||
|
||||
def edit_stamp(self, prompt_id, updated_at="2099-01-01T00:00:00+00:00"):
|
||||
self.run_sql(
|
||||
"UPDATE prompts SET updated_at=? WHERE id=?", updated_at, prompt_id
|
||||
)
|
||||
|
||||
def details(self, draft):
|
||||
return [(row["label"], row["detail"]) for row in draft.details]
|
||||
|
||||
|
||||
class TestClassification(PairMetadataCase):
|
||||
def test_fills_empty_category_and_tags_from_the_pair(self):
|
||||
pair_id, prompt = self.pair(
|
||||
"a cat on a sofa", "blurry", category="portrait", tags=["b", "a"]
|
||||
)
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.operation, OPERATION)
|
||||
self.assertEqual(draft.targets, [[prompt, True, True]])
|
||||
self.assertEqual(draft.scope["count"], 1)
|
||||
self.assertEqual(
|
||||
draft.scope["label"], "1 prompt to fill: 1 category, 1 tag set"
|
||||
)
|
||||
self.assertEqual(draft.protected, {"count": 0, "reasons": []})
|
||||
self.assertEqual(
|
||||
draft.media,
|
||||
{
|
||||
"deletes_files": False,
|
||||
"label": "No image file or saved pair is changed.",
|
||||
},
|
||||
)
|
||||
self.assertIn(
|
||||
(
|
||||
"a cat on a sofa",
|
||||
f"Fills category “portrait” and 2 tags: a, b (pair {pair_id})",
|
||||
),
|
||||
self.details(draft),
|
||||
)
|
||||
self.assertEqual(draft.details[-1]["label"], "Record")
|
||||
self.assertEqual(draft.details[-1]["detail"], repair.RECORD_NOTE)
|
||||
|
||||
def test_empty_plan(self):
|
||||
self.save("plain prompt")
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [])
|
||||
self.assertEqual(draft.scope["count"], 0)
|
||||
self.assertEqual(
|
||||
draft.scope["label"], "No Pair prompts are missing a category or tags"
|
||||
)
|
||||
self.assertEqual(draft.details, [])
|
||||
|
||||
def test_prompt_already_complete_is_not_counted(self):
|
||||
_pair, prompt = self.pair("done", category="portrait", tags=["a"])
|
||||
self.run_sql("UPDATE prompts SET category='portrait' WHERE id=?", prompt)
|
||||
self.db.set_prompt_tags(prompt, ["a"])
|
||||
self.edit_stamp(
|
||||
prompt, self.sql("SELECT created_at FROM prompts WHERE id=?", prompt)[0][0]
|
||||
)
|
||||
self.assertEqual(self.preview().scope["count"], 0)
|
||||
|
||||
def test_whitespace_category_is_blank_on_prompt_and_pair(self):
|
||||
_p1, blank_prompt = self.pair("ws prompt", category=" portrait ")
|
||||
self.run_sql("UPDATE prompts SET category=' ' WHERE id=?", blank_prompt)
|
||||
_p2, blank_pair = self.pair("ws pair", category=" ")
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [[blank_prompt, True, False]])
|
||||
self.assertIn("“portrait”", self.details(draft)[0][1])
|
||||
self.assertNotIn(blank_pair, [t[0] for t in draft.targets])
|
||||
|
||||
def test_mixed_fields_category_fills_while_tags_conflict(self):
|
||||
_a, prompt = self.pair("mixed", "n1", category="portrait", tags=["a"])
|
||||
self.pair("mixed", "n2", category="portrait", tags=["b"])
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [[prompt, True, False]])
|
||||
self.assertEqual(draft.scope["count"], 1)
|
||||
self.assertEqual(draft.protected["count"], 1)
|
||||
self.assertEqual(draft.protected["reasons"], ["1 where pairs disagree"])
|
||||
self.assertIn(("mixed", "Kept: pairs disagree on tags"), self.details(draft))
|
||||
|
||||
def test_mixed_fields_tags_fill_while_category_conflicts(self):
|
||||
_a, prompt = self.pair("mixed2", "n1", category="portrait", tags=["a"])
|
||||
self.pair("mixed2", "n2", category="landscape", tags=["a"])
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [[prompt, False, True]])
|
||||
self.assertIn(
|
||||
("mixed2", "Kept: pairs disagree on category: landscape, portrait"),
|
||||
self.details(draft),
|
||||
)
|
||||
|
||||
def test_category_set_and_tags_empty_fills_tags_only(self):
|
||||
_a, prompt = self.pair("has cat", category="portrait", tags=["a"])
|
||||
self.run_sql("UPDATE prompts SET category='mine' WHERE id=?", prompt)
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [[prompt, False, True]])
|
||||
|
||||
def test_pairs_without_a_value_are_ignored(self):
|
||||
_a, prompt = self.pair("partial", "n1", category="portrait", tags=["a"])
|
||||
self.pair("partial", "n2") # no category, no tags
|
||||
self.assertEqual(self.preview().targets, [[prompt, True, True]])
|
||||
|
||||
def test_equal_values_across_pairs_fill(self):
|
||||
_a, prompt = self.pair("same", "n1", category="portrait", tags=["a", "b"])
|
||||
self.pair("same", "n2", category="portrait", tags=["b", "a"])
|
||||
self.assertEqual(self.preview().targets, [[prompt, True, True]])
|
||||
|
||||
def test_category_is_case_sensitive(self):
|
||||
_a, prompt = self.pair("case", "n1", category="Portrait")
|
||||
self.pair("case", "n2", category="portrait")
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [])
|
||||
self.assertEqual(draft.protected["count"], 1)
|
||||
|
||||
def test_edited_prompt_is_protected(self):
|
||||
cases = {
|
||||
"later": "2099-01-01T00:00:00+00:00",
|
||||
"null": None,
|
||||
"legacy": "2020-01-01 10:00:00",
|
||||
"garbage": "yesterday",
|
||||
}
|
||||
for name, stamp in cases.items():
|
||||
with self.subTest(name=name):
|
||||
_a, prompt = self.pair(f"edited {name}", category="portrait")
|
||||
self.edit_stamp(prompt, stamp)
|
||||
draft = self.preview()
|
||||
self.assertNotIn(prompt, [t[0] for t in draft.targets])
|
||||
self.assertIn(
|
||||
(
|
||||
f"edited {name}",
|
||||
"Kept: edited after it was saved; its empty category or "
|
||||
"tags may be intentional",
|
||||
),
|
||||
self.details(draft),
|
||||
)
|
||||
self.run_sql(
|
||||
"DELETE FROM prompt_pairs WHERE positive_prompt_id=?", prompt
|
||||
)
|
||||
|
||||
def test_first_release_stamps_microseconds_apart_are_never_edited(self):
|
||||
_a, prompt = self.pair("caa9649f", category="portrait")
|
||||
self.run_sql(
|
||||
"UPDATE prompts SET created_at=?, updated_at=? WHERE id=?",
|
||||
"2026-10-18T10:00:00.000100+00:00",
|
||||
"2026-10-18T10:00:00.000180+00:00",
|
||||
prompt,
|
||||
)
|
||||
self.run_sql(
|
||||
"UPDATE prompt_pairs SET created_at=? WHERE positive_prompt_id=?",
|
||||
"2026-10-18T10:00:00.000400+00:00",
|
||||
prompt,
|
||||
)
|
||||
self.assertEqual(self.preview().targets, [[prompt, True, False]])
|
||||
|
||||
def test_edited_beats_conflict(self):
|
||||
_a, prompt = self.pair("both", "n1", category="portrait")
|
||||
self.pair("both", "n2", category="landscape")
|
||||
self.edit_stamp(prompt)
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.protected["reasons"], ["1 edited after it was saved"])
|
||||
|
||||
def test_archived_prompt_is_protected(self):
|
||||
_a, prompt = self.pair("archived", category="portrait")
|
||||
self.run_sql(
|
||||
"INSERT INTO prompt_archive(prompt_id, archived_at) VALUES(?, ?)",
|
||||
prompt,
|
||||
"2026-10-31T00:00:00+00:00",
|
||||
)
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [])
|
||||
self.assertEqual(draft.protected["reasons"], ["1 archived in the Library"])
|
||||
self.assertIn(
|
||||
("archived", "Kept: archived in the Library"), self.details(draft)
|
||||
)
|
||||
|
||||
def test_archived_pairs_are_no_source(self):
|
||||
pair_id, prompt = self.pair("old pair", "n1", category="portrait")
|
||||
self.db.archive_prompt_pair(pair_id)
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [])
|
||||
self.assertEqual(draft.protected["count"], 0)
|
||||
# An archived pair also never disagrees with a live one.
|
||||
self.pair("old pair", "n2", category="landscape")
|
||||
self.assertEqual(self.preview().targets, [[prompt, True, False]])
|
||||
self.assertIn("landscape", self.details(self.preview())[0][1])
|
||||
|
||||
def test_prompt_not_created_by_its_pair_is_out_of_scope(self):
|
||||
prompt = self.save("saved by the node first")
|
||||
self.run_sql(
|
||||
"UPDATE prompts SET created_at=?, updated_at=? WHERE id=?",
|
||||
"2026-01-01T00:00:00+00:00",
|
||||
"2026-01-01T00:00:00+00:00",
|
||||
prompt,
|
||||
)
|
||||
self.db.save_prompt_pair("saved by the node first", category="portrait")
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [])
|
||||
self.assertEqual(draft.protected["count"], 0)
|
||||
|
||||
def test_negative_is_never_a_target_and_positive_role_sources_only(self):
|
||||
_a, both = self.pair("dual", "shared negative", category="portrait")
|
||||
neg = self.sql("SELECT id FROM prompts WHERE text='shared negative'")[0][0]
|
||||
# The negative of pair A is the positive of pair B, made later: not
|
||||
# Pair-created as a positive, so out of scope.
|
||||
self.db.save_prompt_pair("shared negative", category="landscape")
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [[both, True, False]])
|
||||
self.assertNotIn(neg, [t[0] for t in draft.targets])
|
||||
|
||||
def test_positive_first_then_negative_is_filled_from_positive_pairs_only(self):
|
||||
_a, prompt = self.pair("pos first", category="portrait")
|
||||
self.db.save_prompt_pair("another", "pos first", category="landscape")
|
||||
draft = self.preview()
|
||||
self.assertIn([prompt, True, False], draft.targets)
|
||||
self.assertIn("“portrait”", dict(self.details(draft))["pos first"])
|
||||
|
||||
def test_more_than_twenty_tags_is_protected(self):
|
||||
tags = [f"t{n:02d}" for n in range(21)]
|
||||
_a, prompt = self.pair("many tags", category="portrait", tags=tags)
|
||||
draft = self.preview()
|
||||
self.assertEqual(draft.targets, [[prompt, True, False]])
|
||||
self.assertEqual(
|
||||
draft.protected["reasons"], ["1 whose pair has more than 20 tags"]
|
||||
)
|
||||
self.assertIn(
|
||||
("many tags", "Kept: its pair has more than 20 tags"), self.details(draft)
|
||||
)
|
||||
|
||||
def test_protected_rows_come_first_within_fifty(self):
|
||||
_c, conflict = self.pair("zz conflict", "n1", category="x")
|
||||
self.pair("zz conflict", "n2", category="y")
|
||||
for n in range(60):
|
||||
self.pair(f"fill {n:02d}", category="portrait")
|
||||
draft = self.preview()
|
||||
self.assertEqual(len(draft.details), maintenance.MAX_DETAILS)
|
||||
self.assertEqual(draft.details[0]["label"], "zz conflict")
|
||||
self.assertEqual(draft.details[-1]["label"], "Record")
|
||||
self.assertEqual(draft.scope["count"], 60)
|
||||
|
||||
def test_same_library_same_fingerprint_and_pair_use_does_not_change_it(self):
|
||||
pair_id, _prompt = self.pair("fp", category="portrait", tags=["a"])
|
||||
first = self.preview().fingerprint
|
||||
self.db.record_prompt_pair_use(pair_id)
|
||||
self.db.update_prompt_pair_metadata(pair_id, notes="a note", title="t")
|
||||
self.assertEqual(self.preview().fingerprint, first)
|
||||
|
||||
|
||||
class TestPreviewWritesNothing(PairMetadataCase):
|
||||
def test_preview_reads_one_read_only_snapshot(self):
|
||||
self.pair("cat", category="portrait", tags=["a"])
|
||||
before = self.sql("SELECT * FROM prompts ORDER BY id")
|
||||
tags = self.sql("SELECT * FROM prompt_tags ORDER BY prompt_id, tag_id")
|
||||
connect = maintenance.sqlite3.connect
|
||||
opened = []
|
||||
|
||||
def spy(*args, **kwargs):
|
||||
opened.append(args[0])
|
||||
return connect(*args, **kwargs)
|
||||
|
||||
with patch.object(maintenance.sqlite3, "connect", side_effect=spy):
|
||||
self.preview()
|
||||
self.assertEqual(len(opened), 1)
|
||||
self.assertTrue(str(opened[0]).endswith("?mode=ro"))
|
||||
self.assertEqual(self.sql("SELECT * FROM prompts ORDER BY id"), before)
|
||||
self.assertEqual(
|
||||
self.sql("SELECT * FROM prompt_tags ORDER BY prompt_id, tag_id"), tags
|
||||
)
|
||||
|
||||
|
||||
class TestApply(PairMetadataCase):
|
||||
def snapshot_pairs(self):
|
||||
return (
|
||||
self.sql("SELECT * FROM prompt_pairs ORDER BY id"),
|
||||
self.sql("SELECT * FROM prompt_pair_tags ORDER BY pair_id, tag_id"),
|
||||
)
|
||||
|
||||
def test_apply_fills_only_empty_fields_and_leaves_pairs_alone(self):
|
||||
_a, prompt = self.pair("apply me", "neg", category="portrait", tags=["a", "b"])
|
||||
_b, partial = self.pair("partial", "n1", category="portrait", tags=["a"])
|
||||
self.pair("partial", "n2", category="portrait", tags=["c"])
|
||||
neg = self.sql("SELECT id FROM prompts WHERE text='neg'")[0][0]
|
||||
pairs = self.snapshot_pairs()
|
||||
stamps = self.sql("SELECT id, created_at, updated_at FROM prompts ORDER BY id")
|
||||
state = self.sql("SELECT * FROM prompt_library_state ORDER BY prompt_id")
|
||||
tag_rows = self.sql("SELECT id, name FROM tags ORDER BY id")
|
||||
draft = self.preview()
|
||||
result = self.apply(draft)
|
||||
self.assertEqual(result["changed"], 2)
|
||||
self.assertEqual(
|
||||
result["rows"],
|
||||
[
|
||||
{"prompt_id": prompt, "category": "portrait", "tags": ["a", "b"]},
|
||||
{"prompt_id": partial, "category": "portrait"},
|
||||
],
|
||||
)
|
||||
self.assertEqual(result["categories"], 2)
|
||||
self.assertEqual(result["tag_sets"], 1)
|
||||
self.assertEqual(
|
||||
(self.category(prompt), self.tags(prompt)), ("portrait", ["a", "b"])
|
||||
)
|
||||
self.assertEqual((self.category(partial), self.tags(partial)), ("portrait", []))
|
||||
self.assertEqual((self.category(neg), self.tags(neg)), (None, []))
|
||||
self.assertEqual(self.snapshot_pairs(), pairs)
|
||||
self.assertEqual(
|
||||
self.sql("SELECT id, created_at, updated_at FROM prompts ORDER BY id"),
|
||||
stamps,
|
||||
)
|
||||
self.assertEqual(
|
||||
self.sql("SELECT * FROM prompt_library_state ORDER BY prompt_id"), state
|
||||
)
|
||||
self.assertEqual(self.sql("SELECT id, name FROM tags ORDER BY id"), tag_rows)
|
||||
|
||||
def test_second_run_finds_nothing_and_a_partial_fill_completes_later(self):
|
||||
_a, prompt = self.pair("partial", "n1", category="portrait", tags=["a"])
|
||||
p2, _same = self.pair("partial", "n2", category="portrait", tags=["c"])
|
||||
self.apply(self.preview())
|
||||
again = self.preview()
|
||||
self.assertEqual(again.scope["count"], 0)
|
||||
self.assertEqual(again.protected["count"], 1) # tags still disagree
|
||||
self.db.update_prompt_pair_metadata(p2, tags=["a"])
|
||||
self.apply(self.preview())
|
||||
self.assertEqual(self.tags(prompt), ["a"])
|
||||
self.assertEqual(self.preview().protected["count"], 0)
|
||||
|
||||
def test_drift_raises_plan_changed_and_writes_nothing(self):
|
||||
def pair_category(ids):
|
||||
self.db.update_prompt_pair_metadata(ids[0], category="landscape")
|
||||
|
||||
def prompt_category(ids):
|
||||
self.run_sql("UPDATE prompts SET category='mine' WHERE id=?", ids[1])
|
||||
|
||||
def new_pair(ids):
|
||||
self.db.save_prompt_pair("drift", "other neg", category="landscape")
|
||||
|
||||
def tag_rename(ids):
|
||||
tag = self.sql("SELECT id FROM tags WHERE name='a'")[0][0]
|
||||
self.run_sql("UPDATE tags SET name='renamed' WHERE id=?", tag)
|
||||
|
||||
def edited(ids):
|
||||
self.edit_stamp(ids[1])
|
||||
|
||||
for change in (pair_category, prompt_category, new_pair, tag_rename, edited):
|
||||
with self.subTest(change=change.__name__):
|
||||
ids = self.pair("drift", "n", category="portrait", tags=["a"])
|
||||
draft = self.preview()
|
||||
change(ids)
|
||||
before = self.sql("SELECT id, category FROM prompts ORDER BY id")
|
||||
tags = self.sql("SELECT * FROM prompt_tags ORDER BY prompt_id")
|
||||
with self.assertRaises(maintenance.PlanChanged):
|
||||
self.apply(draft)
|
||||
self.assertEqual(
|
||||
self.sql("SELECT id, category FROM prompts ORDER BY id"), before
|
||||
)
|
||||
self.assertEqual(
|
||||
self.sql("SELECT * FROM prompt_tags ORDER BY prompt_id"), tags
|
||||
)
|
||||
self.run_sql("DELETE FROM prompt_pairs")
|
||||
self.run_sql("DELETE FROM prompts")
|
||||
self.run_sql("DELETE FROM tags")
|
||||
|
||||
def test_pair_run_between_preview_and_apply_still_applies(self):
|
||||
pair_id, prompt = self.pair("busy", category="portrait")
|
||||
draft = self.preview()
|
||||
self.db.record_prompt_pair_use(pair_id)
|
||||
self.apply(draft)
|
||||
self.assertEqual(self.category(prompt), "portrait")
|
||||
|
||||
def test_guard_failure_rolls_back_everything(self):
|
||||
_a, first = self.pair("first", category="portrait")
|
||||
_b, second = self.pair("second", category="portrait")
|
||||
draft = self.preview()
|
||||
real = repair._fill_categories
|
||||
|
||||
def racing(conn, fills):
|
||||
# Another writer's value appears inside the transaction.
|
||||
conn.execute("UPDATE prompts SET category='x' WHERE id=?", (second,))
|
||||
return real(conn, fills)
|
||||
|
||||
with patch.object(repair, "_fill_categories", side_effect=racing):
|
||||
with self.assertRaises(maintenance.PlanChanged):
|
||||
self.apply(draft)
|
||||
self.assertIsNone(self.category(first))
|
||||
self.assertIsNone(self.category(second))
|
||||
|
||||
def test_more_than_two_hundred_fills_are_all_returned(self):
|
||||
for n in range(205):
|
||||
self.pair(f"bulk {n:03d}", category="portrait")
|
||||
result = self.apply(self.preview())
|
||||
self.assertEqual(result["changed"], 205)
|
||||
self.assertEqual(len(result["rows"]), 205)
|
||||
|
||||
def test_batches_bind_few_ids(self):
|
||||
for n in range(5):
|
||||
self.pair(f"batch {n}", category="portrait", tags=["a"])
|
||||
with patch.object(maintenance, "ID_BATCH", 2):
|
||||
draft = self.preview()
|
||||
result = self.apply(draft)
|
||||
self.assertEqual(result["changed"], 5)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -49,6 +49,8 @@ export const OPERATION_TITLES = Object.freeze({
|
||||
duplicate_prompts: 'Duplicate prompt text',
|
||||
optimize: 'Optimize database',
|
||||
unused_components: 'Unused prompt components',
|
||||
// D192 (#550): beyond the concept, System › Maintenance › Database.
|
||||
pair_metadata: 'Pair category and tags',
|
||||
missing_links: 'Missing-file links',
|
||||
duplicate_files: 'Duplicate image files',
|
||||
// Reviewed and applied through the API and seen in Activity; Maintenance has no row for it (the concept has none).
|
||||
|
||||
@@ -2,10 +2,12 @@
|
||||
* System › Maintenance (#249 System S7 Task 19): the concept's `renderSystem` "maintenance" body (app.js 688–690).
|
||||
*
|
||||
* - The notice "Check first. Review changes. Then apply." with "Run health checks" (D123), then the surfaces
|
||||
* "Database" and "Images & cache" with the concept's eight operations, in its order, labels and danger marks.
|
||||
* "Database" and "Images & cache" with the concept's eight operations, in its order, labels and danger marks, plus
|
||||
* D192 (#550, beyond the concept): "Pair category and tags" after "Unused prompt components".
|
||||
* - Integrity & relationship checks → Check runs the same read-only health check and opens the Health report (D95);
|
||||
* its "Review missing links" reviews a fresh Missing-file links plan, never a cleanup.
|
||||
* - Duplicate prompt text, Optimize database, Unused prompt components and Missing-file links → Review plan open the
|
||||
* - Duplicate prompt text, Optimize database, Unused prompt components, Pair category and tags (`pair_metadata`: fill
|
||||
* old Pair prompts' empty category and tags from their saved pairs) and Missing-file links → Review plan open the
|
||||
* D100 plan review (maintenance-review.mjs), which applies only what it showed, as an Activity job.
|
||||
* - Duplicate image files → Find duplicates starts the duplicate search as an Activity job and opens it; its view
|
||||
* offers Review groups (D101) once it found some.
|
||||
@@ -39,6 +41,9 @@ export const MAINTENANCE_SURFACES = Object.freeze([
|
||||
['Optimize database', 'Reclaim unused SQLite space. This does not delete prompts or media.', 'Review plan', 'optimize'],
|
||||
['Unused prompt components', 'Preview unreferenced components. Pair references and history remain protected.', 'Review plan',
|
||||
'unused_components', true],
|
||||
// D192 (#550): beyond the concept. Fills only empty fields, deletes nothing (no danger mark).
|
||||
['Pair category and tags', 'Fill an empty category or tags on prompts older Pair node runs saved, from their saved ' +
|
||||
'pairs. Only empty fields of never-edited prompts.', 'Review plan', 'pair_metadata'],
|
||||
]],
|
||||
['Images & cache', [
|
||||
['Missing-file links', 'Remove stale database links only. Files on disk are never deleted by this operation.', 'Review plan',
|
||||
|
||||
Reference in New Issue
Block a user