Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
99ae2e8c7f | ||
|
|
08103adde5 |
@@ -221,10 +221,7 @@ jobs:
|
||||
- uses: actions/setup-python@v7
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: pip
|
||||
cache-dependency-path: requirements-dev.txt
|
||||
# The exact black and flake8 pins from requirements-dev.txt, so a new release cannot turn CI red.
|
||||
- run: pip install $(grep -E '^(black|flake8)==' requirements-dev.txt)
|
||||
- run: pip install black flake8
|
||||
- run: black --check .
|
||||
- run: flake8 .
|
||||
|
||||
@@ -238,8 +235,6 @@ jobs:
|
||||
- uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: "24"
|
||||
cache: npm
|
||||
cache-dependency-path: tools/vendor/package-lock.json
|
||||
|
||||
- name: Check shipped extension and page modules parse
|
||||
run: |
|
||||
|
||||
@@ -13,11 +13,9 @@ repos:
|
||||
language: system
|
||||
types: [python]
|
||||
|
||||
# Fails at the severity CI enforces (code-quality.yml: -lll, high only), never stricter.
|
||||
# pre-commit appends the staged files, so no shell wrapper can drop the first one.
|
||||
- id: bandit
|
||||
name: bandit
|
||||
entry: bandit -lll -q
|
||||
entry: bash -c 'bandit "$@" -ll -q 2>&1; true'
|
||||
language: system
|
||||
types: [python]
|
||||
exclude: ^(tests/|reference/)
|
||||
|
||||
+11
-143
@@ -3200,16 +3200,12 @@ the per-section hand-off panel, was retired by S12 (#329): every section is nati
|
||||
again"; an Unload that found nothing loaded only reads the status again. At phone width
|
||||
the concept's 120px select cap cuts every engine label, so `#tag-model` alone keeps its
|
||||
natural width there and its row wraps the select under the text (`workspace.css`
|
||||
`/* D132 */`). The concept's cap cuts Existing tags too ("Replace after review"); by the
|
||||
maintainer's decision (#398) that parity is not kept: `#tag-existing` never shrinks for
|
||||
its row text and at phone width keeps its natural width beside the text (its row does
|
||||
not wrap; the text column narrows). 1600 is as the concept draws it. The comparison
|
||||
expects `selectsFit` true for every select and measures each row's `flexWrap` and
|
||||
whether its control sits under the text. Pinned by `auto-tagging matches the concept`
|
||||
(both themes and sizes), `auto-tagging at 390 wraps only the engine row`, `auto-tagging
|
||||
selects fit every option at 390/320 wide` (every option of both selects, both themes, no
|
||||
page scroll) and the mutation check (a wrapped row, a squeezed engine and Existing tags
|
||||
select) (system-tagging-concept.spec), and the model, Download, Unload and Preview
|
||||
`/* D132 */`); Existing tags stays as the concept draws it, cut at 390 as there. The
|
||||
comparison maps `selectsFit` only for the engine select and measures each row's
|
||||
`flexWrap` and whether its control sits under the text. Pinned by `auto-tagging matches
|
||||
the concept` (both themes and sizes), `auto-tagging at 390 wraps only the engine row`,
|
||||
`auto-tagging engine select fits at 320 wide` and the mutation check (it includes a
|
||||
wrapped row) (system-tagging-concept.spec), and the model, Download, Unload and Preview
|
||||
tests of workspace-system-tagging.spec.
|
||||
- **D133** (Auto-tagging, S11): a "Suggestions per image" row (`#tag-suggestions`,
|
||||
number 1–100, host key `autotag_suggestion_limit`) follows the two confidence rows:
|
||||
@@ -3830,9 +3826,7 @@ threshold off that grid is refused ("General-tag confidence must be a multiple o
|
||||
0.05.", `validate_saved_threshold`; `draftProblem` holds back Preview tagging review and
|
||||
the batch Review with the same sentence) and falls back on load; an accepted one is
|
||||
stored exactly on the grid (#369). A per-request threshold keeps its own check: `/image-annotations/suggest` and
|
||||
`/jobs/autotag` take 0–1; the legacy `/autotag/start` passes its options through
|
||||
`validate_options` and starts the `autotag` job titled "Tag output images (legacy page)";
|
||||
the legacy `/autotag/single` only requires a number. `GET /autotag/models` reports the
|
||||
`/jobs/autotag` take 0–1; the legacy `/autotag/single` and `/autotag/start` only require a number. `GET /autotag/models` reports the
|
||||
saved values (`wd14_general_threshold`, `wd14_character_threshold`, `suggestion_limit`,
|
||||
`default_prompt`, `model_type`, `existing`), so the Tag image and Auto-tag selected
|
||||
dialogs start from them; a preset (the bulk review, Preview tagging review) wins.
|
||||
@@ -3898,57 +3892,6 @@ images", "Tag output images (legacy page)", "Tagging one image (legacy page)",
|
||||
is pinned by `tests/test_autotag_schema.py` (forward-only: one empty `WITHOUT ROWID`
|
||||
table, optional in old backups, refused without the annotation group).
|
||||
|
||||
- **The legacy pages (#379, PR 2)** write image tags, never prompt tags.
|
||||
`POST /autotag/start` (the admin and gallery batch) reads `model_type`, `prompt`,
|
||||
`skip_tagged` (Skip or Merge), `keep_in_memory` and the thresholds (else the saved
|
||||
ones), checks them with `validate_options` (WD14: both thresholds; GGUF/HF:
|
||||
`instructions` = the non-blank `prompt`, else the saved template; the saved
|
||||
"Suggestions per image") and starts the `autotag` job "Tag output images (legacy
|
||||
page)" (gate owner `legacy-batch`) through the native start, so the safety backup and
|
||||
the disk check apply. A second legacy batch answers JSON 409 `{success: false, error:
|
||||
"Auto-tagging is already running"}` (also when another legacy start wins the race past
|
||||
that check); `SseStream` hands a non-2xx JSON `error` to `onerror` as `serverMessage`,
|
||||
which both pages show instead of "Connection lost during auto tagging". Any other refusal (a job in the way: "<title>
|
||||
already running", `model-unavailable`, `insufficient-space`, `model-busy`, a bad
|
||||
option) is one `{type: "error", status: "error", message}` frame. The stream relays
|
||||
the job: the phases as `{type: "progress", progress: 0 | 5 | 8, status: "Taking a
|
||||
safety backup…" | "Loading model…" | "Listing output images…"}`, each image as
|
||||
`{type: "progress", progress: 10 + progress × 85 / 100, status, processed, tagged,
|
||||
skipped}` (at most every 0.5 s; `processed` = Tagged + Unchanged + Errors), and the end
|
||||
as `complete` (`processed`, `tagged`, `skipped`, `errors`, `skipped_outside_roots: 0`,
|
||||
`status` "Complete" or "No still images found", `model_status`, `model_loaded`),
|
||||
`{type: "cancelled", message: "Cancelled. Tags already added were kept."}` or an
|
||||
`error` frame with the job's status line. A job quiet for 5 s repeats its last frame,
|
||||
and a closed stream cancels the job (the legacy pages have no other stop).
|
||||
`POST /autotag/single` takes `existing` (`"skip"` or `"merge"`, default merge). Under
|
||||
Skip, an image whose current bytes are auto-tagged answers `{success, already_tagged:
|
||||
true, image_tags, prompt_id, filename}` after one stat, without the model gate.
|
||||
Otherwise the image is captured and the model runs on the capture under the gate with
|
||||
the saved cap; the answer adds `tags`, `sha256` (the captured bytes) and `image_tags`,
|
||||
and `service.custom_prompt` is never set. Refusals: `409 model-unavailable`, `413
|
||||
too-large`, `404 file-not-found`, `409 file-changed`, `503 busy`, `415 not-a-still`
|
||||
(the image is marked), `409 model-busy`. `POST /autotag/apply` takes exactly
|
||||
`{image_id, sha256, tags}` or `{path, root_index?, sha256, tags}` (64 KiB), re-hashes
|
||||
the file and unions the tags into that image (`add_image_tags`, marked): `{success,
|
||||
added_tags, total_tags}`; an empty list only marks the image. A body with `prompt_id`
|
||||
is `400` "This page is out of date. Reload it."; a changed file `409 file-changed`
|
||||
("The image changed since its tags were generated."); a full set `409 tag-limit`; a
|
||||
restore `409 storage-changed`. The admin review asks each image with Skip first: an
|
||||
auto-tagged one shows "Image Already Auto-tagged" / "A tagging model already tagged
|
||||
this image. Generate new tags?" with its image tags (Skip, Skip All Tagged, Re-tag asks
|
||||
again with Merge); the gallery review always merges. Apply & Next sends the image and
|
||||
its `sha256` (linked or not). The batch copy: "Skip images already auto-tagged",
|
||||
"Auto tagging complete! Tagged N images." ("1 image"), and under both progress bars
|
||||
"Closing this window stops tagging. Tags already added are kept. Prompt Manager's
|
||||
Activity page shows the time left.". Pinned by `tests/test_autotag_legacy.py`,
|
||||
`tests/js/image-helpers.test.js` and `legacy-autotag.spec`.
|
||||
- **Tag image dialog marks (DEC4).** A save after Generate suggestions ran on that
|
||||
capture marks the image auto-tagged (also a replayed save, and a saved retry after
|
||||
`tag-conflict`); a save without Generate does not. The API keeps the suggested capture
|
||||
tokens; each new one drops the tokens whose capture is gone (`expires_in`, which never
|
||||
resets a capture's idle clock), a release or a saved receipt drops its own, and
|
||||
closing the captures clears them (`TestDialogMarks`).
|
||||
|
||||
No browser test downloads or runs a model: the injected `TestModel` answers, its
|
||||
`download_model` raises unless `/__test__/tagging/download {mode: hold | succeed |
|
||||
fail}` installs a fake that waits on a fixture event, and every tagging reset releases
|
||||
@@ -3983,30 +3926,7 @@ modals use the same routes and keep working.
|
||||
other: each file's identity is checked just before its unlink.
|
||||
- **Clear all logs** (`POST /logs/truncate`, also the admin page's Clear) empties the
|
||||
active log through its handler, deletes the rotated copies (Rule R) and clears the
|
||||
buffer; other files and links are kept. Response keys are unchanged. It deletes or
|
||||
empties no more than its confirmation showed (#389):
|
||||
- `/logs/files` rows carry `identity`, the string `"<size>:<mtime_ns>:<ino>"` (a
|
||||
string: `mtime_ns` and many inodes are beyond a browser's exact integers). The
|
||||
workspace sends `{"expected": [{name, identity}]}` with the active and rotated rows
|
||||
its dialog was built from. Under the files lock and every handler lock each listed
|
||||
rotated copy must still match (`_matches_review`) and the listed active log must be
|
||||
the same inode, or nothing changes: 409 `logs-changed`. Only listed copies are
|
||||
deleted; one that appeared after the confirmation is kept. A malformed list is 400.
|
||||
- Without a body (the legacy admin page, whose confirm lists nothing; also the
|
||||
workspace when the file list could not be read and the dialog said "deletes its
|
||||
rotated copies") every rotated copy present at the locked rescan is deleted.
|
||||
- Either way, before anything changes (best effort, not a cross-process lock: a
|
||||
write before our own last write, or any while file logging is off here, is not
|
||||
seen): a second writer of the active log (our
|
||||
handler's flushed stream is behind the file's size, or its file is now a rotated
|
||||
copy: another ComfyUI sharing the folder) is 409 `logs-foreign-writer`; an active
|
||||
log with more than one name (`st_nlink > 1` by `lstat` or the handler's `fstat`,
|
||||
NTFS included) is 409 `logs-hard-link`, never truncated (replacing it instead would
|
||||
rename a file our handler holds open, which Windows refuses). Each copy is
|
||||
`lstat`-checked again just before its unlink; one that changed stops the clear there
|
||||
and keeps it, the copies after it and the active log, which is emptied last.
|
||||
- The browser fixture grows the active log through the handler's own stream, since a
|
||||
second handle appending to it is exactly what Clear refuses.
|
||||
buffer; other files and links are kept. Response keys are unchanged.
|
||||
- **Runtime settings** (`GET/POST /logs/config`): the level (DEBUG…CRITICAL) and console
|
||||
logging apply until the server restarts; the retention values are shown read-only in
|
||||
the workspace and the API keeps accepting them.
|
||||
@@ -4126,33 +4046,6 @@ list stay unscoped: any row's control proves them.
|
||||
| API timeout `apiTimeout` | `gallery-settings.js:37` | Legacy gallery (S12-10; read by no request) | kept in legacy | — | defaults are a complete, sane settings object (gallery-settings.test.js) |
|
||||
| config.json-only keys | `py/config.py:212–234,722–739`; `py/api/diagnostics_report.py:185` | S12 report (read-only) | covered | `diagnostics` · button · "Download diagnostic report" | test_report_lists_config_only_keys (test_api_diagnostics_report.py) |
|
||||
|
||||
### Legacy System UI removed by S13
|
||||
|
||||
System S13 (#330) points the legacy pages' System buttons at native System (a link that
|
||||
opens `workspace#/system/<id>` in a new tab) and removes the legacy copy of a capability
|
||||
only when its row above is `covered`. One row per removed capability: Capability is the
|
||||
exact Capability cell of its coverage row, Page is `admin` or `gallery`, Removed ids and
|
||||
Removed methods list what went (`—` for none), and Native test copies that row's Test
|
||||
cell. An id that stays on a System link is not listed, only its handler.
|
||||
|
||||
`tests/vendor/legacy-coverage.test.mjs` fails when a row cites a capability that is not
|
||||
`covered` or copies another test, when an id or method is listed twice for one page, or
|
||||
when a listed id is still on its page (`id="…"` in the markup or the script, a
|
||||
`getElementById`/`querySelector('#…')` lookup, an entry of a modal-id list) or a listed
|
||||
method is still defined or called (`.name(`). A deletion listed nowhere is caught by the
|
||||
kept-flow tests (`kept admin flows still work` in `legacy-system-links.spec.mjs` and the
|
||||
legacy page workflows), not by this check.
|
||||
|
||||
| Capability | Page | Removed ids | Removed methods | Native test |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Diagnostics run | admin | `#diagnosticsModal`, `#runDiagnosticsBtn`, `#diagnosticsContent` | `showDiagnosticsModal()`, `closeDiagnostics()`, `runDiagnostics()`, `renderDiagnostics()` | Run checks opens the shared Health report (workspace-system-diagnostics.spec.mjs) |
|
||||
| Synthetic image-link test | admin | `#testImageLinkBtn` | `testImageLink()` | Create test link confirms first and names the prompt (workspace-system-diagnostics.spec.mjs) |
|
||||
| View logs with level and limit | admin | `#logsModal`, `#logLevel`, `#logLimit`, `#refreshLogsBtn`, `#logsContainer` | `showLogsModal()`, `closeLogs()`, `refreshLogs()`, `displayLogs()` | level filter is minimum-level over all six choices (workspace-system-diagnostics.spec.mjs) |
|
||||
| Log statistics | admin | `#logStatsTotal`, `#logStatsErrors`, `#logStatsWarnings`, `#logStatsLevel`, `#logStatsSize` | `loadLogStats()`, `formatBytes()` | log statistics show entries, errors, warnings, files and size (workspace-system-diagnostics.spec.mjs) |
|
||||
| Log files list and download | admin | `#downloadLogsBtn`, `#logFilesList` | `loadLogFiles()`, `downloadLogs()`, `downloadLogFile()` | each listed file downloads by its returned name and a link has no Download (workspace-system-diagnostics.spec.mjs) |
|
||||
| Clear/truncate logs | admin | `#clearLogsBtn` | `clearLogs()` | Clear all logs states both counts, confirms first and reads everything again (workspace-system-diagnostics.spec.mjs) |
|
||||
| Set log level (incl. CRITICAL), console logging | admin | `#updateLogConfigBtn`, `#setLogLevel`, `#consoleLogging` | `loadLogConfig()`, `updateLogConfig()` | runtime level (including CRITICAL) and console logging save and say they apply until the server restarts (workspace-system-diagnostics.spec.mjs) |
|
||||
|
||||
### Browser cache ownership (#222)
|
||||
|
||||
The legacy gallery's Clear cache used to delete every localStorage key starting `pm_` or
|
||||
@@ -4547,17 +4440,8 @@ impact and command receipts report `image_count` / `images` /
|
||||
`(size, mtime_ns)` equals its verified fingerprint; an in-place rewrite that
|
||||
preserves both keeps the old set matching until the file is next observed. The
|
||||
Images Tags filter (a file matches when every tag is on its linked positive or
|
||||
its current image tag set) and `q` search find their candidates through the tag
|
||||
index (#379), never by streaming every image tag row: `image_tag_ids` reads the
|
||||
tag names that tag at least one image (an `EXISTS` on `idx_media_image_tags_tag`)
|
||||
and keeps a name when it equals a Tags filter tag exactly or contains `q`
|
||||
case-folded; `image_tag_locators` reads the files carrying those ids (`tag_id IN`,
|
||||
500 ids a chunk). Only the listed ones among them are stat'ed, and only their
|
||||
current records count; a linked positive's prompt tags still combine with the
|
||||
image tags under a Tags filter. `test_indexed_candidates_equal_the_full_stream`
|
||||
compares the lookup with the old full stream (kept in the test as the reference)
|
||||
and `test_tag_search_reads_image_tags_only_by_tag_id` traces every statement; the
|
||||
|
||||
its current image tag set) and `q` search stream the tagged records once and
|
||||
stat only the listed files whose records carry a filter tag or a `q` hit; the
|
||||
inspector's `image_tags` facet (`current` / `none` / `unavailable`) uses the
|
||||
inspection's own stable stat (`tests/test_api_workspace_images.py`
|
||||
`TestWorkspaceImageTags`, `tests/test_api_prompt_inspector.py`). The
|
||||
@@ -4646,23 +4530,7 @@ files, oldest forgotten first, keyed by 16-byte digests: 8.6 MiB measured at
|
||||
the cap). A file whose inspection raises is logged, counted in
|
||||
`review_errors` and reported `unresolved`; the scan goes on. An older version's
|
||||
pending, undecided item of bytes now untraced leaves the queue
|
||||
(`settle_stale_review_items`), so an item never stays stuck at 409.
|
||||
Saver anchor (#390, `tests/test_review_saver_anchor.py`): the scan and Apply
|
||||
resolve with the file's name as the workspace inspector does, so the one saver
|
||||
whose `filename_prefix` names the file anchors its branch. A file is never both
|
||||
unlinked and unqueued (maintainer ruling): when only that anchor resolves it
|
||||
(the named inspection run resolves, the unnamed one does not) and neither the
|
||||
default profile nor the parser floor links it, the scan queues it with the
|
||||
anchor's pair as its one candidate (evidence `anchor: "saver"`; the page
|
||||
preselects it), never linking it. Such an item counts as a choice for the
|
||||
untraced settle. Files already linked or resolved by the default path are
|
||||
unchanged. No match, several matches or an unfinished walk anchor nothing, and
|
||||
the file is queued as before. `REVIEW_RULE_VERSION` is folded into inspection
|
||||
item versions, so older inspection items are recorded again once. Apply accepts
|
||||
the anchored pair and refuses a choice the anchor contradicts. Measured on the
|
||||
maintainer's 40,890 outputs: the queue is unchanged (249 files); the 242
|
||||
`gpu_unlock_bench` files the inspector resolves are linked by the parser
|
||||
floor. The 16-candidate limited
|
||||
(`settle_stale_review_items`), so an item never stays stuck at 409. The 16-candidate limited
|
||||
rule and the 8 MiB bound apply unchanged. The queue heals itself: after each batch, the scan removes the pending
|
||||
items of every listed file (read or skipped as known) that a saved row now
|
||||
links, and the monitor does the same for each ambiguous output it links
|
||||
|
||||
+21
-37
@@ -110,16 +110,12 @@ _MARKED_LOCATORS = (
|
||||
f"SELECT DISTINCT f.locator FROM {AUTOTAG_TABLE} m " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.id=m.byte_id JOIN {FILES_TABLE} f ON f.id=b.file_id"
|
||||
)
|
||||
# Tag search (#379): the names that tag at least one image (the tag index
|
||||
# answers the EXISTS), then the locators carrying chosen names (by tag_id).
|
||||
_IMAGE_TAG_NAMES = (
|
||||
"SELECT t.id,t.name FROM tags t WHERE EXISTS " # nosec B608
|
||||
f"(SELECT 1 FROM {IMAGE_TAGS_TABLE} mit WHERE mit.tag_id=t.id)"
|
||||
)
|
||||
_TAG_LOCATORS = (
|
||||
f"SELECT DISTINCT f.locator FROM {IMAGE_TAGS_TABLE} mit " # nosec B608
|
||||
f"JOIN {BYTES_TABLE} b ON b.id=mit.byte_id "
|
||||
f"JOIN {FILES_TABLE} f ON f.id=b.file_id WHERE mit.tag_id IN ({{marks}})"
|
||||
_RECORDS = (
|
||||
"SELECT f.locator,b.id,b.size,b.mtime_ns,t.name " # nosec B608
|
||||
f"FROM {FILES_TABLE} f JOIN {BYTES_TABLE} b ON b.file_id=f.id "
|
||||
f"LEFT JOIN {IMAGE_TAGS_TABLE} mit ON mit.byte_id=b.id "
|
||||
"LEFT JOIN tags t ON t.id=mit.tag_id "
|
||||
"ORDER BY f.locator,b.first_seen_at,b.id,mit.position"
|
||||
)
|
||||
|
||||
|
||||
@@ -520,32 +516,20 @@ class AnnotationOperations:
|
||||
with self._annotation_read() as conn:
|
||||
return {row[0] for row in conn.execute(_MARKED_LOCATORS)}
|
||||
|
||||
def image_tag_ids(self, wanted, query: str) -> list[int]:
|
||||
"""Ids of the tag names on at least one image that a search wants.
|
||||
def iter_image_tag_records(self) -> Iterator[tuple[str, int, int, list[str]]]:
|
||||
"""Stream every byte record as ``(locator, size, mtime_ns, tags)``.
|
||||
|
||||
A name counts when it is in ``wanted`` exactly (the Tags filter) or
|
||||
contains ``query`` case-folded (the text search; ``query`` is
|
||||
already folded). One read of the names that tag an image.
|
||||
One SELECT ordered by locator, then by the bytes' first sighting, with
|
||||
each set's names in position order.
|
||||
"""
|
||||
wanted = set(wanted)
|
||||
with self._annotation_read() as conn:
|
||||
return [
|
||||
tag_id
|
||||
for tag_id, name in conn.execute(_IMAGE_TAG_NAMES)
|
||||
if name in wanted or (query and query in name.casefold())
|
||||
]
|
||||
|
||||
def image_tag_locators(self, tag_ids) -> set[str]:
|
||||
"""Locators with any byte record (current or not) carrying one of
|
||||
``tag_ids``, through the tag index, in chunks of 500 ids."""
|
||||
ids = list(dict.fromkeys(tag_ids))
|
||||
found: set[str] = set()
|
||||
with self._annotation_read() as conn:
|
||||
for start in range(0, len(ids), _LOOKUP_CHUNK):
|
||||
chunk = ids[start : start + _LOOKUP_CHUNK]
|
||||
marks = ",".join("?" * len(chunk))
|
||||
found.update(
|
||||
row[0]
|
||||
for row in conn.execute(_TAG_LOCATORS.format(marks=marks), chunk)
|
||||
)
|
||||
return found
|
||||
conn = self.model.get_connection()
|
||||
current: tuple | None = None
|
||||
for locator, byte_id, size, mtime_ns, name in conn.execute(_RECORDS):
|
||||
if current is None or current[0] != byte_id:
|
||||
if current is not None:
|
||||
yield current[1]
|
||||
current = (byte_id, (locator, size, mtime_ns, []))
|
||||
if name is not None:
|
||||
current[1][3].append(name)
|
||||
if current is not None:
|
||||
yield current[1]
|
||||
|
||||
@@ -1808,6 +1808,37 @@ class PromptDatabase(
|
||||
reader = DocumentReader(conn)
|
||||
return [self._image_row_to_dict(row, reader) for row in cursor]
|
||||
|
||||
def autotag_batch_candidates(self) -> List[Dict[str, Any]]:
|
||||
"""The images a Tag output images batch looks at (#328).
|
||||
|
||||
``{id, image_path, prompt_id, prompt_tags}`` for every image with a
|
||||
stored path that is linked to a prompt, newest first (the
|
||||
``get_all_images`` order, ties newest row first). Never reads
|
||||
``workflow_data`` or the other documents: a batch needs none of them.
|
||||
"""
|
||||
sql = (
|
||||
"SELECT gi.id, gi.image_path, gi.prompt_id, "
|
||||
"(SELECT GROUP_CONCAT(t.name, '|||') FROM prompt_tags pt "
|
||||
"JOIN tags t ON pt.tag_id = t.id WHERE pt.prompt_id = gi.prompt_id) "
|
||||
"AS prompt_tags "
|
||||
"FROM generated_images gi "
|
||||
"INNER JOIN prompts p ON gi.prompt_id = p.id "
|
||||
"WHERE gi.image_path IS NOT NULL AND gi.image_path != '' "
|
||||
"ORDER BY gi.generation_time DESC, gi.id DESC"
|
||||
)
|
||||
with self.model.get_connection() as conn:
|
||||
return [
|
||||
{
|
||||
"id": row["id"],
|
||||
"image_path": row["image_path"],
|
||||
"prompt_id": row["prompt_id"],
|
||||
"prompt_tags": (
|
||||
row["prompt_tags"].split("|||") if row["prompt_tags"] else []
|
||||
),
|
||||
}
|
||||
for row in conn.execute(sql)
|
||||
]
|
||||
|
||||
def get_all_images(
|
||||
self,
|
||||
limit: int = 0,
|
||||
|
||||
@@ -229,15 +229,13 @@ def evidence_offers_choice(evidence: Any) -> bool:
|
||||
"""
|
||||
if not isinstance(evidence, dict):
|
||||
return False
|
||||
# The saver anchor's one pair (#390) is a suggestion to confirm.
|
||||
needed = 1 if evidence.get("anchor") == "saver" else MIN_CHOOSABLE
|
||||
if evidence.get("limited"):
|
||||
total = evidence.get("candidate_total")
|
||||
return type(total) is int and total >= needed
|
||||
return type(total) is int and total >= MIN_CHOOSABLE
|
||||
candidates = evidence.get("candidates")
|
||||
if not isinstance(candidates, list):
|
||||
return False
|
||||
return sum(1 for c in candidates if choosable(c)) >= needed
|
||||
return sum(1 for c in candidates if choosable(c)) >= MIN_CHOOSABLE
|
||||
|
||||
|
||||
def evidence_digest(text: str) -> str:
|
||||
|
||||
+193
-3
@@ -363,9 +363,6 @@ class PromptManagerAPI(
|
||||
# Private media captures of the annotation routes (py/api/annotations.py),
|
||||
# created on first use with their restore subscription.
|
||||
self._captures = None
|
||||
# Capture tokens Generate suggestions ran on: a save of one marks the
|
||||
# image auto-tagged (#379).
|
||||
self._suggested_tokens = set()
|
||||
self._captures_unsubscribe = None
|
||||
# Set by close_annotation_captures (shutdown): no store is recreated.
|
||||
self._captures_closed = False
|
||||
@@ -1107,3 +1104,196 @@ class PromptManagerAPI(
|
||||
result["parameters"][field] = metadata[field]
|
||||
|
||||
return result
|
||||
|
||||
def _extract_readable_prompt(self, parsed_data):
|
||||
"""Extract human-readable prompt text from ComfyUI/A1111 data."""
|
||||
|
||||
def safe_to_string(value):
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
elif isinstance(value, list):
|
||||
return " ".join(str(item) for item in value if item)
|
||||
elif value is not None:
|
||||
return str(value)
|
||||
return None
|
||||
|
||||
# First check if we already extracted a positive prompt (A1111 format)
|
||||
if parsed_data.get("positive_prompt"):
|
||||
return parsed_data["positive_prompt"]
|
||||
|
||||
# Check if prompt is already a string
|
||||
if isinstance(parsed_data.get("prompt"), str):
|
||||
return parsed_data["prompt"]
|
||||
|
||||
# Check if prompt is a simple value that can be converted
|
||||
if parsed_data.get("prompt") and not isinstance(
|
||||
parsed_data.get("prompt"), dict
|
||||
):
|
||||
return safe_to_string(parsed_data["prompt"])
|
||||
|
||||
prompt_data = parsed_data.get("prompt")
|
||||
if isinstance(prompt_data, dict):
|
||||
# Use the enhanced logic from parse-metadata.py
|
||||
positive_prompt = self._extract_positive_prompt_from_comfyui_data(
|
||||
prompt_data
|
||||
)
|
||||
if positive_prompt:
|
||||
return positive_prompt
|
||||
|
||||
# Check workflow data if available
|
||||
workflow_data = parsed_data.get("workflow")
|
||||
if isinstance(workflow_data, dict):
|
||||
positive_prompt = self._extract_positive_prompt_from_comfyui_data(
|
||||
workflow_data
|
||||
)
|
||||
if positive_prompt:
|
||||
return positive_prompt
|
||||
|
||||
# Check parameters for positive prompt
|
||||
if parsed_data.get("parameters", {}).get("positive"):
|
||||
return safe_to_string(parsed_data["parameters"]["positive"])
|
||||
|
||||
return None
|
||||
|
||||
def _get_node_inputs(self, node):
|
||||
"""Safely get inputs from a node, handling both dict and list formats.
|
||||
|
||||
Old format: inputs is a dict with direct key-value pairs
|
||||
inputs = {"text": "my prompt", "seed": 123}
|
||||
|
||||
New format: inputs is a list of connection objects
|
||||
inputs = [
|
||||
{"name": "text", "type": "STRING", "link": null,
|
||||
"widget": {"name": "text"}},
|
||||
{"name": "clip", "type": "CLIP", "link": 11}
|
||||
]
|
||||
"""
|
||||
if not isinstance(node, dict):
|
||||
return {}
|
||||
|
||||
inputs = node.get("inputs", {})
|
||||
|
||||
# If inputs is already a dict, return it
|
||||
if isinstance(inputs, dict):
|
||||
return inputs
|
||||
|
||||
# If inputs is a list, convert to dict format
|
||||
if isinstance(inputs, list):
|
||||
inputs_dict = {}
|
||||
for input_item in inputs:
|
||||
if isinstance(input_item, dict) and "name" in input_item:
|
||||
name = input_item["name"]
|
||||
inputs_dict[name] = input_item
|
||||
return inputs_dict
|
||||
|
||||
return {}
|
||||
|
||||
def _find_text_in_node(self, node):
|
||||
"""Try to find text content in a node using various strategies.
|
||||
|
||||
Handles both old and new workflow formats.
|
||||
"""
|
||||
if not isinstance(node, dict):
|
||||
return None
|
||||
|
||||
# Strategy 1: Check normalized inputs for 'text' field
|
||||
inputs = self._get_node_inputs(node)
|
||||
if "text" in inputs and isinstance(inputs["text"], str):
|
||||
return inputs["text"]
|
||||
|
||||
# Strategy 2: For text encoder nodes, check widgets_values
|
||||
class_type = node.get("class_type", node.get("type", ""))
|
||||
text_encoder_types = [
|
||||
"CLIPTextEncode",
|
||||
"CLIPTextEncodeSDXL",
|
||||
"CLIPTextEncodeSDXLRefiner",
|
||||
"CLIPTextEncodeFlux",
|
||||
"PromptManager",
|
||||
"PromptManagerText",
|
||||
"BNK_CLIPTextEncoder",
|
||||
"Text Encoder",
|
||||
"CLIP Text Encode",
|
||||
]
|
||||
|
||||
if any(
|
||||
encoder_type.lower() in class_type.lower()
|
||||
for encoder_type in text_encoder_types
|
||||
):
|
||||
widgets_values = node.get("widgets_values", [])
|
||||
if widgets_values and len(widgets_values) > 0:
|
||||
if isinstance(widgets_values[0], str) and widgets_values[0].strip():
|
||||
return widgets_values[0]
|
||||
|
||||
return None
|
||||
|
||||
def _extract_positive_prompt_from_comfyui_data(self, data):
|
||||
"""Extract positive prompt from ComfyUI data.
|
||||
|
||||
Handles both old and new formats.
|
||||
"""
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
|
||||
# Build nodes dictionary
|
||||
nodes_by_id = {}
|
||||
if "nodes" in data:
|
||||
# Handle nodes array format
|
||||
for node in data["nodes"]:
|
||||
if isinstance(node, dict):
|
||||
nid = node.get("id")
|
||||
if nid is not None:
|
||||
nodes_by_id[nid] = node
|
||||
else:
|
||||
# Handle flat dictionary format (node_id -> node_data)
|
||||
for nid_str, node in data.items():
|
||||
try:
|
||||
nid = int(nid_str)
|
||||
except (ValueError, TypeError):
|
||||
nid = nid_str
|
||||
if isinstance(node, dict):
|
||||
if "id" in node:
|
||||
nid = node["id"]
|
||||
nodes_by_id[nid] = node
|
||||
|
||||
if not nodes_by_id:
|
||||
return None
|
||||
|
||||
# First, try to find positive/negative connection pattern
|
||||
pos_id = None
|
||||
for node in nodes_by_id.values():
|
||||
if isinstance(node, dict):
|
||||
inputs = self._get_node_inputs(node)
|
||||
if "positive" in inputs and "negative" in inputs:
|
||||
try:
|
||||
pos_input = inputs["positive"]
|
||||
if isinstance(pos_input, list) and len(pos_input) > 0:
|
||||
pos_id = int(pos_input[0])
|
||||
break
|
||||
except (ValueError, TypeError, IndexError):
|
||||
continue
|
||||
|
||||
# Get text from the positive node
|
||||
if pos_id is not None and pos_id in nodes_by_id:
|
||||
text_val = self._find_text_in_node(nodes_by_id[pos_id])
|
||||
if text_val:
|
||||
return text_val
|
||||
|
||||
# Fallback: find any text encoder node with text content
|
||||
text_nodes = []
|
||||
for node in nodes_by_id.values():
|
||||
if isinstance(node, dict):
|
||||
text_val = self._find_text_in_node(node)
|
||||
if text_val:
|
||||
# Try to determine if this is positive or negative
|
||||
node_title = node.get("title", "").lower()
|
||||
if "neg" not in node_title and "negative" not in node_title:
|
||||
# Prioritize non-negative prompts
|
||||
text_nodes.insert(0, text_val)
|
||||
else:
|
||||
text_nodes.append(text_val)
|
||||
|
||||
# Return the first positive-looking prompt
|
||||
if text_nodes:
|
||||
return text_nodes[0]
|
||||
|
||||
return None
|
||||
|
||||
+12
-32
@@ -47,7 +47,7 @@ try:
|
||||
extract_generation_params,
|
||||
parse_page_json,
|
||||
)
|
||||
from ...utils.prompt_pair_resolver import resolve_prompt_pair, sampler_less_prompt
|
||||
from ...utils.prompt_pair_resolver import resolve_prompt_pair
|
||||
from ...utils.review_capture import reviewable
|
||||
except ImportError:
|
||||
from database.pair_ingestion import blank_positive, ingest_resolved_pair
|
||||
@@ -62,7 +62,7 @@ except ImportError:
|
||||
extract_generation_params,
|
||||
parse_page_json,
|
||||
)
|
||||
from utils.prompt_pair_resolver import resolve_prompt_pair, sampler_less_prompt
|
||||
from utils.prompt_pair_resolver import resolve_prompt_pair
|
||||
from utils.review_capture import reviewable
|
||||
|
||||
|
||||
@@ -1630,14 +1630,7 @@ class AdminRoutesMixin:
|
||||
if self._review_file_linked(media_file, roots):
|
||||
outcome = "unchanged" # already linked: not a review case
|
||||
else:
|
||||
# The records name the item versions that count; the
|
||||
# saver anchor (#390) is checked on the captured bytes.
|
||||
self._record_scanned_review(
|
||||
media_file,
|
||||
roots,
|
||||
review,
|
||||
scanned=(records.get("canonical"), None),
|
||||
)
|
||||
self._record_scanned_review(media_file, roots, review)
|
||||
elif untraced and outcome == "unresolved":
|
||||
# Untraced by the default profile: the inspection profile may
|
||||
# still offer a choice (queued for review, never linked).
|
||||
@@ -1698,9 +1691,9 @@ class AdminRoutesMixin:
|
||||
``"ambiguous"`` (nothing linked it) is a Needs review case. An
|
||||
ambiguous result with nothing to choose (``reviewable`` is False: fewer
|
||||
than two candidates carry a prompt) that nothing links returns
|
||||
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``.
|
||||
The parsed records of either ambiguous file go to
|
||||
``records["canonical"]`` when ``records`` is a dict.
|
||||
``"unresolved"`` instead, and ``media_file`` goes to ``untraced``;
|
||||
its parsed records go to ``records["canonical"]`` when ``records`` is
|
||||
a dict.
|
||||
"""
|
||||
if not metadata:
|
||||
# No prompt metadata (--disable-metadata, bare savers): only the
|
||||
@@ -1730,8 +1723,6 @@ class AdminRoutesMixin:
|
||||
records["canonical"] = canonical
|
||||
elif resolution["state"] == "ambiguous" and ambiguous is not None:
|
||||
ambiguous.append(media_file)
|
||||
if records is not None:
|
||||
records["canonical"] = canonical
|
||||
if resolution["state"] == "resolved":
|
||||
result = ingest_resolved_pair(
|
||||
self.db, str(media_file), canonical, resolution
|
||||
@@ -1763,24 +1754,13 @@ class AdminRoutesMixin:
|
||||
canonical.get("parameters"), str
|
||||
):
|
||||
return resolution["state"]
|
||||
elif canonical.get("prompt"):
|
||||
# No node takes a positive or negative input (a guider or wrapper
|
||||
# graph): the one wired text, never a guess (#252).
|
||||
found = sampler_less_prompt(canonical["prompt"])
|
||||
prompt_text = found["text"]
|
||||
if prompt_text is None and isinstance(canonical.get("parameters"), str):
|
||||
# A metadata saver also wrote A1111 text: its positive prompt.
|
||||
parsed = self._parse_comfyui_prompt(metadata)["positive_prompt"]
|
||||
prompt_text = parsed.strip() if parsed and parsed.strip() else None
|
||||
if prompt_text is None:
|
||||
linked = self._link_recorded_run(
|
||||
media_file, canonical, detailed, history
|
||||
)
|
||||
if linked is not None:
|
||||
return linked
|
||||
return "unresolved" if found["state"] == "ambiguous" else "found"
|
||||
else:
|
||||
prompt_text = self._parse_comfyui_prompt(metadata)["positive_prompt"]
|
||||
parsed = self._parse_comfyui_prompt(metadata)
|
||||
if not (parsed.get("prompt") or parsed.get("parameters")):
|
||||
return None
|
||||
prompt_text = self._extract_readable_prompt(parsed)
|
||||
if prompt_text is not None and not isinstance(prompt_text, str):
|
||||
prompt_text = str(prompt_text)
|
||||
if not (prompt_text and prompt_text.strip()):
|
||||
return "found"
|
||||
prompt_text = prompt_text.strip()
|
||||
|
||||
@@ -296,7 +296,6 @@ class AnnotationRoutesMixin:
|
||||
self._captures_closed = True
|
||||
unsubscribe, self._captures_unsubscribe = self._captures_unsubscribe, None
|
||||
store, self._captures = self._captures, None
|
||||
self._suggested_tokens = set()
|
||||
if unsubscribe is not None:
|
||||
unsubscribe()
|
||||
if store is not None:
|
||||
@@ -516,7 +515,6 @@ class AnnotationRoutesMixin:
|
||||
store = self._captures
|
||||
if store is not None and _TOKEN.fullmatch(token) is not None:
|
||||
store.release(token)
|
||||
self._suggested_tokens.discard(token)
|
||||
return web.json_response(
|
||||
{"success": True}, headers={"Cache-Control": "no-store"}
|
||||
)
|
||||
@@ -566,9 +564,6 @@ class AnnotationRoutesMixin:
|
||||
return _coded("capture-expired", _OTHER_FILE)
|
||||
if facts.sha256 != capture.sha256:
|
||||
return _coded("file-changed")
|
||||
# A save after Generate suggestions ran on this capture marks the
|
||||
# image auto-tagged (#379), so a Skip batch leaves the review alone.
|
||||
suggested = token in self._suggested_tokens
|
||||
try:
|
||||
receipt = await self._run_in_executor(
|
||||
self.db.replace_image_tags,
|
||||
@@ -579,7 +574,6 @@ class AnnotationRoutesMixin:
|
||||
expected_epoch=expected_epoch,
|
||||
expected_tag_revision=revision,
|
||||
tags=tags,
|
||||
mark_autotagged=suggested,
|
||||
)
|
||||
except AnnotationCommandError as exc:
|
||||
current = exc.current or {}
|
||||
@@ -596,23 +590,10 @@ class AnnotationRoutesMixin:
|
||||
except Exception:
|
||||
self.logger.exception("Image tag save failed")
|
||||
return _error(500, "save-failed", "The tags could not be saved.")
|
||||
# Dropped only after a saved or replayed receipt: a retry after a
|
||||
# tag-conflict still marks.
|
||||
self._suggested_tokens.discard(token)
|
||||
return web.json_response(
|
||||
{"success": True, **receipt}, headers={"Cache-Control": "no-store"}
|
||||
)
|
||||
|
||||
def _remember_suggested(self, store, token: str) -> None:
|
||||
"""Note that Generate ran on *token*; forget tokens whose capture is
|
||||
gone. ``expires_in`` does not reset a capture's idle clock."""
|
||||
self._suggested_tokens = {
|
||||
kept
|
||||
for kept in self._suggested_tokens
|
||||
if store.expires_in(kept) is not None
|
||||
}
|
||||
self._suggested_tokens.add(token)
|
||||
|
||||
# -- suggest ---------------------------------------------------------------
|
||||
|
||||
async def suggest_image_tags(self, request: web.Request) -> web.Response:
|
||||
@@ -694,7 +675,6 @@ class AnnotationRoutesMixin:
|
||||
finally:
|
||||
if not started:
|
||||
store.release_use(token)
|
||||
self._remember_suggested(store, token)
|
||||
return web.json_response(
|
||||
{
|
||||
"success": True,
|
||||
|
||||
+415
-526
File diff suppressed because it is too large
Load Diff
@@ -483,59 +483,13 @@ class BackupRestoreRoutesMixin:
|
||||
self.logger.info(f"Removed {len(removed)} old safety backups")
|
||||
|
||||
async def _publish(self, model, staged):
|
||||
"""Publish ``staged``; on a page-copy failure prune and re-raise.
|
||||
|
||||
A cancel (server stop) waits for the publishing thread (#360): the
|
||||
caller's cleanup and its registry admission must outlive it. When the
|
||||
copy then succeeded the library was replaced, so it is adopted
|
||||
(``_after_publish``, itself run to its end) before the cancel goes
|
||||
on; a copy that failed is handled as usual, then the cancel goes on.
|
||||
"""
|
||||
future = self._in_executor_task(model.publish_staged, staged)
|
||||
cancelled = await _outlive_cancel(future)
|
||||
"""Publish ``staged``; on a page-copy failure prune and re-raise."""
|
||||
try:
|
||||
backup_path = future.result()
|
||||
return await self._run_in_executor(model.publish_staged, staged)
|
||||
except sqlite3.OperationalError as exc:
|
||||
if getattr(exc, "_prompt_manager_restore_backup_created", False) is True:
|
||||
await self._prune_after(model.db_path, failed=True)
|
||||
if cancelled:
|
||||
raise asyncio.CancelledError from exc
|
||||
raise
|
||||
except Exception as exc:
|
||||
if cancelled:
|
||||
raise asyncio.CancelledError from exc
|
||||
raise
|
||||
if cancelled:
|
||||
self.logger.info("The restore finished while the server was stopping")
|
||||
# A second cancel (the loop's own shutdown) must not cut it short.
|
||||
adopted = asyncio.ensure_future(self._after_publish(model.db_path))
|
||||
await _outlive_cancel(adopted)
|
||||
if not adopted.cancelled() and adopted.exception() is not None:
|
||||
self.logger.error(
|
||||
"Could not adopt the restored library",
|
||||
exc_info=adopted.exception(),
|
||||
)
|
||||
raise asyncio.CancelledError
|
||||
return backup_path
|
||||
|
||||
async def _stage_to_end(self, model, temp_path):
|
||||
"""Stage ``temp_path``; a cancel waits for the worker (#360).
|
||||
|
||||
The upload stays until the staging thread is done with it, and a
|
||||
stage it finished after the cancel is removed before the cancel goes
|
||||
on.
|
||||
"""
|
||||
future = self._in_executor_task(model.stage_restore, temp_path)
|
||||
if await _outlive_cancel(future):
|
||||
if not future.cancelled() and future.exception() is None:
|
||||
await self._discard_held([_Orphan(future.result())])
|
||||
raise asyncio.CancelledError
|
||||
return future.result()
|
||||
|
||||
def _in_executor_task(self, func, *args):
|
||||
"""``_run_in_executor(func, *args)`` as a task ``_outlive_cancel`` can
|
||||
wait for (shielded, the executor call is never cancelled)."""
|
||||
return asyncio.ensure_future(self._run_in_executor(func, *args))
|
||||
|
||||
async def _after_publish(self, db_path):
|
||||
"""Everything after the copy replaced the library: best effort, logged.
|
||||
@@ -655,7 +609,7 @@ class BackupRestoreRoutesMixin:
|
||||
{"success": False, "error": REQUESTS_BUSY}, status=409
|
||||
)
|
||||
model = self.db.model
|
||||
staged = await self._stage_to_end(model, temp_path)
|
||||
staged = await self._run_in_executor(model.stage_restore, temp_path)
|
||||
try:
|
||||
backup_path = await self._publish(model, staged)
|
||||
finally:
|
||||
@@ -973,30 +927,6 @@ class BackupRestoreRoutesMixin:
|
||||
job.update(progress=100, status_line=line)
|
||||
|
||||
|
||||
async def _outlive_cancel(future):
|
||||
"""Await ``future`` (an executor call's task) to its end, even through
|
||||
cancels (#360).
|
||||
|
||||
A cancelled task cannot stop the worker thread, so the files it uses and
|
||||
the registry admission it runs under must outlive it: the cancel is held
|
||||
until the worker returned. Returns whether the task was cancelled
|
||||
meanwhile; the caller reads the outcome from ``future`` and re-raises
|
||||
``CancelledError``. A future cancelled itself (an executor shut down) is
|
||||
not a cancel of the task: ``future.result()`` raises it. Never catches
|
||||
``GeneratorExit``: a closed loop still closes the coroutine.
|
||||
"""
|
||||
cancelled = False
|
||||
while not future.done():
|
||||
try:
|
||||
await asyncio.shield(future)
|
||||
except asyncio.CancelledError:
|
||||
if not future.cancelled():
|
||||
cancelled = True
|
||||
except Exception:
|
||||
pass # the caller reads it from the future
|
||||
return cancelled
|
||||
|
||||
|
||||
class _Orphan:
|
||||
"""A stage with no store entry, in the shape ``_discard_held`` takes."""
|
||||
|
||||
|
||||
+16
-10
@@ -575,6 +575,12 @@ def _current_fingerprint(path: Path) -> tuple[int, int] | None:
|
||||
return observed.st_size, observed.st_mtime_ns
|
||||
|
||||
|
||||
def _image_tags_relevant(names: list, wanted: set, query: str) -> bool:
|
||||
if wanted and not wanted.isdisjoint(names):
|
||||
return True
|
||||
return bool(query) and any(query in name.casefold() for name in names)
|
||||
|
||||
|
||||
def _workspace_key(entry: tuple, order: str) -> tuple:
|
||||
"""The keyset sort key of *entry* (a _Listed or a 4-tuple entry)."""
|
||||
if not isinstance(entry, _Listed):
|
||||
@@ -2164,9 +2170,8 @@ class ImageRoutesMixin:
|
||||
def _image_tag_matches(self, listings: list[tuple], filters: dict) -> dict:
|
||||
"""Normalized path -> _ImageTagMatch for files whose image tags count.
|
||||
|
||||
Looks the relevant tag names up (exact under Tags, a case-folded
|
||||
substring for ``q``) and their files through the tag index, keeping
|
||||
only listed locators; only those candidates are
|
||||
Streams the tagged byte records once and keeps only listed locators
|
||||
carrying a filter tag or a ``q`` hit; only those candidates are
|
||||
stat'ed (never every listed file), and a candidate's tags count only
|
||||
while a byte record is verified at its present ``(size, mtime_ns)``
|
||||
(spec amendment S1). Under a Tags filter the candidates' linked
|
||||
@@ -2175,15 +2180,16 @@ class ImageRoutesMixin:
|
||||
tags, query, category = filters["tags"], filters["query"], filters["category"]
|
||||
wanted = set(tags)
|
||||
located = self._workspace_locators(listings)
|
||||
# The tag index answers which files carry a relevant name (#379); no
|
||||
# read streams every image tag row.
|
||||
tag_ids = self.db.image_tag_ids(wanted, query)
|
||||
candidates = {}
|
||||
if tag_ids:
|
||||
for locator in self.db.image_tag_locators(tag_ids):
|
||||
records = self.db.iter_image_tag_records()
|
||||
try:
|
||||
for locator, _size, _mtime_ns, names in records:
|
||||
entry = located.get(locator)
|
||||
if entry is not None:
|
||||
candidates[locator] = entry
|
||||
if entry is not None and locator not in candidates:
|
||||
if _image_tags_relevant(names, wanted, query):
|
||||
candidates[locator] = entry
|
||||
finally:
|
||||
records.close()
|
||||
stats = {}
|
||||
for locator, entry in candidates.items():
|
||||
fingerprint = _current_fingerprint(entry.path)
|
||||
|
||||
@@ -11,11 +11,6 @@ import stat
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
try:
|
||||
from ...utils.logging_config import ClearRefused, parse_confirmed
|
||||
except ImportError: # loaded as the top-level ``py`` package (tests)
|
||||
from utils.logging_config import ClearRefused, parse_confirmed
|
||||
|
||||
# Keys PromptManagerLogger.update_config() understands. Anything else is
|
||||
# rejected so a client cannot inject arbitrary entries into the logging config.
|
||||
LOG_CONFIG_KEYS = frozenset(
|
||||
@@ -268,43 +263,10 @@ class LoggingRoutesMixin:
|
||||
)
|
||||
|
||||
async def truncate_logs(self, request):
|
||||
"""Clear all logs: what ``{"expected": [...]}`` lists, or all (#389).
|
||||
|
||||
``expected`` holds the ``{name, identity}`` rows of ``/logs/files`` the
|
||||
confirmation showed. Without a body (the legacy admin page, whose
|
||||
confirmation lists nothing) every rotated copy present now is deleted.
|
||||
A malformed list is 400; a clear that would empty or delete what was
|
||||
not confirmed changes nothing and is 409 with the reason's ``code``.
|
||||
"""
|
||||
"""Truncate all log files."""
|
||||
try:
|
||||
args = ()
|
||||
if request.can_read_body:
|
||||
data, error_response = await self._read_json_body(request)
|
||||
if error_response is not None:
|
||||
return error_response
|
||||
if "expected" in data:
|
||||
try:
|
||||
parse_confirmed(data["expected"])
|
||||
except ValueError:
|
||||
return web.json_response(
|
||||
{
|
||||
"success": False,
|
||||
"error": "expected must list name and identity"
|
||||
" rows from /logs/files",
|
||||
},
|
||||
status=400,
|
||||
)
|
||||
args = (data["expected"],)
|
||||
logger_manager = self._get_logger_manager()
|
||||
try:
|
||||
results = await self._run_in_executor(
|
||||
logger_manager.truncate_logs, *args
|
||||
)
|
||||
except ClearRefused as refused:
|
||||
return web.json_response(
|
||||
{"success": False, "error": str(refused), "code": refused.code},
|
||||
status=409,
|
||||
)
|
||||
results = await self._run_in_executor(logger_manager.truncate_logs)
|
||||
|
||||
return web.json_response(
|
||||
{
|
||||
|
||||
+1
-11
@@ -677,14 +677,7 @@ def _verify_and_read(store, capture, item, root, need_metadata: bool):
|
||||
facts = store.verify(root, relative)
|
||||
captured = None
|
||||
if need_metadata:
|
||||
# The file's own name, so an inspection item resolves anchored on its
|
||||
# saver as it was recorded (#390).
|
||||
captured = captured_review(
|
||||
store,
|
||||
capture,
|
||||
item_profile(item["evidence"]),
|
||||
relative.rsplit("/", 1)[-1],
|
||||
)
|
||||
captured = captured_review(store, capture, item_profile(item["evidence"]))
|
||||
return relative, facts, captured
|
||||
|
||||
|
||||
@@ -730,9 +723,6 @@ def _candidate_still_offered(resolution: dict, choice: str) -> bool:
|
||||
that rewords the top-level reason or reorders candidates leaves a
|
||||
pending item applicable.
|
||||
"""
|
||||
if isinstance(resolution, dict) and resolution.get("anchor") == "saver":
|
||||
# Only the saver anchor resolves the bytes (#390): its pair alone.
|
||||
resolution = {"state": "ambiguous", "candidates": [resolution]}
|
||||
if not isinstance(resolution, dict) or resolution.get("state") != "ambiguous":
|
||||
return False
|
||||
evidence, _count, limited = review_evidence(resolution)
|
||||
|
||||
@@ -88,8 +88,7 @@ class ReviewRecordingMixin:
|
||||
def _record_scanned_review(self, media_file, roots, review=None, scanned=None):
|
||||
"""Record one ambiguous, unlinked scanned file (logged and counted).
|
||||
|
||||
``scanned`` is the scan's ``(metadata, inspection_resolution)``; the
|
||||
resolution is None when the scan did not run the inspection profile.
|
||||
``scanned`` is the scan's ``(metadata, inspection_resolution)``.
|
||||
"""
|
||||
try:
|
||||
outcome = self._record_review_sync(
|
||||
@@ -200,7 +199,7 @@ class ReviewRecordingMixin:
|
||||
key = _memory_key(locator)
|
||||
if state is not None and memory.get(key) == state:
|
||||
return None
|
||||
resolution = review_capture.inspection_choice(canonical, media_file.name)
|
||||
resolution = review_capture.inspection_choice(canonical)
|
||||
if resolution is not None:
|
||||
memory.pop(key, None)
|
||||
return canonical, resolution
|
||||
|
||||
+11
-23
@@ -88,9 +88,9 @@ from .thumbnail_encode import shutdown_encoders, writer_settings
|
||||
from .thumbnail_jobs import ThumbnailOptions
|
||||
|
||||
try:
|
||||
from ...utils.parallel import map_parallel, note_refusal, refused_at_shutdown
|
||||
from ...utils.parallel import map_parallel
|
||||
except ImportError: # tests import the package top-level
|
||||
from utils.parallel import map_parallel, note_refusal, refused_at_shutdown
|
||||
from utils.parallel import map_parallel
|
||||
|
||||
# The placeholder frame a video without a preview is answered with (JPEG
|
||||
# bytes, never written to disk); written previews follow the settings.
|
||||
@@ -410,8 +410,7 @@ class AutoThumbnailer:
|
||||
|
||||
def enqueue(self, src, dst, is_video):
|
||||
"""Queue *dst*'s preview for the worker; False when already queued,
|
||||
full, closed or its worker is refused at interpreter exit. Never
|
||||
blocks on a write."""
|
||||
full or closed. Never blocks on a write."""
|
||||
key = _dst_key(dst)
|
||||
with self._changed:
|
||||
if self._closed or key in self._queued:
|
||||
@@ -421,41 +420,30 @@ class AutoThumbnailer:
|
||||
self._queued.add(key)
|
||||
self._queue.append((key, Path(src), Path(dst), bool(is_video)))
|
||||
self._busy = True
|
||||
started = True
|
||||
if self._start_worker and not self._worker_running:
|
||||
started = self._start_worker_locked()
|
||||
if not started:
|
||||
# Refused at interpreter exit: no worker will take it (#375).
|
||||
self._queue.pop()
|
||||
self._queued.discard(key)
|
||||
self._start_worker_locked()
|
||||
self._changed.notify_all()
|
||||
return started
|
||||
return True
|
||||
|
||||
def _start_worker_locked(self):
|
||||
"""Start the worker (lock held); False only when the interpreter is
|
||||
exiting (#375: logged once at DEBUG, the caller drops the file).
|
||||
Another start failure leaves the file queued and the next enqueue
|
||||
tries again."""
|
||||
"""Start the worker (lock held). A start that fails leaves the file
|
||||
queued and the next enqueue tries again."""
|
||||
worker = threading.Thread(target=self._run, name=WORKER_NAME, daemon=True)
|
||||
worker.pm_preview_token = _MODULE_TOKEN
|
||||
worker.pm_preview_close = self.close
|
||||
self._worker_running = True
|
||||
try:
|
||||
worker.start()
|
||||
except RuntimeError as error:
|
||||
except RuntimeError:
|
||||
self._worker_running = False
|
||||
self._busy = False
|
||||
if refused_at_shutdown(error):
|
||||
note_refusal(self._logger, "The automatic preview worker")
|
||||
return False
|
||||
self._logger.warning(
|
||||
"The automatic preview worker could not start; the next output "
|
||||
"tries again",
|
||||
exc_info=True,
|
||||
)
|
||||
return True
|
||||
return
|
||||
self._worker = worker
|
||||
return True
|
||||
|
||||
def _run(self):
|
||||
try:
|
||||
@@ -802,8 +790,8 @@ class AutoThumbnailMixin:
|
||||
|
||||
Returns ``queued``, ``fresh`` (its preview is current) or ``skipped``
|
||||
(not a media file under a current root, a link, a preview itself,
|
||||
already queued, after the app's cleanup, or when its worker is refused
|
||||
at interpreter exit). Never writes on the caller's thread.
|
||||
already queued, or after the app's cleanup). Never writes on the
|
||||
caller's thread.
|
||||
"""
|
||||
if self._auto_thumbnails_shut:
|
||||
return "skipped"
|
||||
|
||||
@@ -8,8 +8,8 @@ coverage>=7.16.2
|
||||
numpy>=2.2.6 # WD14 engine tests
|
||||
|
||||
# Code quality
|
||||
black==26.10.1
|
||||
flake8==7.4.1
|
||||
black>=26.10.1
|
||||
flake8>=7.4.1
|
||||
|
||||
# Security scanning
|
||||
bandit[toml]>=1.9.4
|
||||
|
||||
@@ -286,8 +286,7 @@ const continuationText=page=>page.locator('.infinite-continuation > p').first().
|
||||
async function searchOf(page,term) {
|
||||
const before=await countText(page);
|
||||
await page.locator('#search').fill(term);
|
||||
// Production's caption reads "Loading images…" until the search's listing answers (#391); the concept's is instant.
|
||||
await expect.poll(async()=>{const text=await countText(page);return text!==before&&text!=='Loading images…';}).toBe(true);
|
||||
await expect.poll(()=>countText(page)).not.toBe(before);
|
||||
await expect.poll(()=>page.locator(CARDS).count()).toBeLessThan(FIXTURE_IMAGES);
|
||||
await settle(page);
|
||||
const result={count:await countText(page),continuation:await continuationText(page),cards:await page.locator(`${CARDS} .image-open`).evaluateAll(nodes=>nodes.map(node=>node.getAttribute('aria-label')))};
|
||||
|
||||
@@ -228,8 +228,7 @@ async function productionSendOf(page,context) {
|
||||
async function tagOf(page) {
|
||||
const before=await page.locator('#result-count').textContent();
|
||||
await panelOf(page).locator('.detail-inner > .tags').last().getByRole('button',{name:'cinematic',exact:true}).click();
|
||||
// Production's caption reads "Loading images…" until the tag's listing answers (#391); the concept's search is instant.
|
||||
await expect.poll(async()=>{const text=await page.locator('#result-count').textContent();return text!==before&&text!=='Loading images…';}).toBe(true);
|
||||
await expect.poll(()=>page.locator('#result-count').textContent()).not.toBe(before);
|
||||
await settle(page);
|
||||
return {count:await page.locator('#result-count').textContent(),
|
||||
cards:await page.locator(`${CARDS} .image-open`).evaluateAll(nodes=>nodes.map(node=>node.getAttribute('aria-label')))};
|
||||
@@ -329,25 +328,6 @@ test('a recipe-linked image without recipe metadata shows its positive prompt me
|
||||
await expect(panelOf(page).locator('.detail-footer button')).toHaveText(['Send to ComfyUI','Prompt history']);
|
||||
});
|
||||
|
||||
// #391: a tag chip replaces the collection; until its listing answers the caption says it is loading, never that the
|
||||
// filter matched nothing ("0 of 0 images" was painted before the response and read by a fast reader).
|
||||
test('a tag chip collection reads as loading, not empty, until its listing answers (#391)',async({page})=>{
|
||||
await page.setViewportSize({width:1600,height:1100});
|
||||
await openProductionPanel(page,'dark');
|
||||
let release,arrived;
|
||||
const held=new Promise(resolve=>{release=resolve;}),reached=new Promise(resolve=>{arrived=resolve;});
|
||||
await page.route(url=>url.pathname==='/prompt_manager/images/output'&&url.searchParams.get('tags')==='cinematic',
|
||||
async route=>{arrived();await held;await route.continue();},{times:1});
|
||||
const count=page.locator('#result-count');
|
||||
await panelOf(page).locator('.detail-inner > .tags').last().getByRole('button',{name:'cinematic',exact:true}).click();
|
||||
await reached;
|
||||
await expect(count).toHaveText('Loading images…');
|
||||
await expect(page.locator(CARDS)).toHaveCount(0);
|
||||
release();
|
||||
await expect(count).toHaveText(countCopy('8 of 8 images · 2 positive prompts'));
|
||||
await expect(page.locator(CARDS)).toHaveCount(8);
|
||||
});
|
||||
|
||||
// Task C3 (D88, #249 PR C): the concept's chip row is the image's own tags (`i.tags`, app.js a402). With those names
|
||||
// saved as the file's current image tags, production draws the row from them (Tags filter buttons in the panel, plain
|
||||
// in the viewer, the linked positive's equal tags not repeated) and still equals the concept, in both sizes.
|
||||
|
||||
@@ -95,14 +95,8 @@ async function watchProgress(page) {
|
||||
await page.evaluate(() => {
|
||||
window.progressSeen = [];
|
||||
const node = document.getElementById('autoTagProgressPercent');
|
||||
// Each change from its own record: frames that land in one task share one
|
||||
// callback, and the node's text by then is only the last of them.
|
||||
new MutationObserver(records => {
|
||||
for (const record of records) {
|
||||
if (record.type === 'characterData') window.progressSeen.push(record.target.data);
|
||||
for (const added of record.addedNodes) window.progressSeen.push(added.textContent);
|
||||
}
|
||||
}).observe(node, { childList: true, characterData: true, subtree: true });
|
||||
new MutationObserver(() => window.progressSeen.push(node.textContent))
|
||||
.observe(node, { childList: true, characterData: true, subtree: true });
|
||||
});
|
||||
}
|
||||
|
||||
@@ -114,58 +108,25 @@ async function runGalleryBatch(page) {
|
||||
await page.locator('#startAutoTagBtn').click();
|
||||
await done;
|
||||
await expect(page.locator('#autoTagProgressModal')).toBeHidden();
|
||||
// #379: the batch counts images, never prompts.
|
||||
await expect(page.getByText(/^Auto tagging complete! Tagged \d+ images?\.$/)).toBeVisible();
|
||||
}
|
||||
|
||||
const CLOSE_NOTE = 'Closing this window stops tagging. Tags already added are kept. Prompt Manager\'s Activity page shows the '
|
||||
+ 'time left.';
|
||||
|
||||
async function serverJobs(request) {
|
||||
const response = await request.get('/prompt_manager/jobs');
|
||||
await expect(response).toBeOK();
|
||||
return (await response.json()).jobs;
|
||||
}
|
||||
|
||||
async function runNativeBatch(request) {
|
||||
const started = await request.post('/prompt_manager/jobs/autotag', { data: { model_type: 'wd14-swinv2', existing: 'skip',
|
||||
options: { general_threshold: 0.35, character_threshold: 0.85 } } });
|
||||
expect(started.status()).toBe(202);
|
||||
const { job } = await started.json();
|
||||
await expect.poll(async () => (await serverJobs(request)).find(item => item.id === job.id)?.state).toBe('finished');
|
||||
await expect(page.getByText(/^Auto tagging complete! Applied tags to \d+ prompts\.$/)).toBeVisible();
|
||||
}
|
||||
|
||||
test('gallery batch modal counts progress and closes on complete', async ({ page, request }) => {
|
||||
await page.goto('/prompt_manager/gallery');
|
||||
await runGalleryBatch(page);
|
||||
const seen = await page.evaluate(() => window.progressSeen);
|
||||
// The phase frames (model 5%, listing 8%) then the per-image frames (from 10%) move the bar.
|
||||
expect(seen).toContain('8%');
|
||||
expect(seen.some(value => parseInt(value, 10) >= 10)).toBe(true);
|
||||
await expect(page.locator('#autoTagCloseNote')).toHaveText(CLOSE_NOTE);
|
||||
// The batch ran as the autotag job of the legacy page.
|
||||
expect((await serverJobs(request)).filter(job => job.kind === 'autotag').map(job => [job.title, job.state]))
|
||||
.toEqual([['Tag output images (legacy page)', 'finished']]);
|
||||
// The fixture's prompts are already tagged, so the batch skips them; its progress frames still move the bar.
|
||||
expect(seen).toContain('95%');
|
||||
expect((await downloads(request)).length).toBe(0);
|
||||
});
|
||||
|
||||
test('admin batch copy names images, not prompts (#379)', async ({ page }) => {
|
||||
await page.goto('/prompt_manager/admin');
|
||||
await page.locator('#autoTagBtn').click();
|
||||
await expect(page.locator('input[name="autoTagMode"][value="skip"] + span')).toHaveText('Skip images already auto-tagged');
|
||||
await expect(page.getByText('"auto-scanned" is not counted as a real tag')).toHaveCount(0);
|
||||
await expect(page.locator('#autoTagCloseNote')).toHaveText(CLOSE_NOTE);
|
||||
await expect(page.locator('#retagConfirmModal h3')).toHaveText('Image Already Auto-tagged');
|
||||
await expect(page.locator('#retagConfirmModal')).toContainText('A tagging model already tagged this image. Generate new tags?');
|
||||
});
|
||||
|
||||
test('admin review opens the first image', async ({ page }) => {
|
||||
await page.goto('/prompt_manager/admin');
|
||||
await page.locator('#autoTagBtn').click();
|
||||
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
|
||||
await page.locator('#startReviewBtn').click();
|
||||
// #379: no image is auto-tagged yet (prompt tags no longer count), so the review opens on the first image.
|
||||
await expect(page.locator('#retagConfirmModal')).toBeHidden();
|
||||
// The first image's prompt is already tagged: confirm a new pass, then the review opens on that image.
|
||||
await page.locator('#retagConfirmBtn').click();
|
||||
await expect(page.locator('#autoTagReviewModal')).toBeVisible();
|
||||
await expect(page.locator('#reviewImageName')).not.toHaveText(/^(image_name\.png|Unknown|)$/);
|
||||
await expect(page.getByText('Failed to start review mode')).toHaveCount(0);
|
||||
@@ -189,70 +150,3 @@ test('gallery unload works and is refused while the gate is held', async ({ page
|
||||
await unload.click();
|
||||
await expect(page.getByText('WD14-SWINV2 model unloaded successfully')).toBeVisible();
|
||||
});
|
||||
|
||||
/** The admin review lists every row, videos too: Skip any image that is not one until the confirmation shows. */
|
||||
async function untilAutoTagged(page, confirm) {
|
||||
// Each image's skip check answers before its confirmation or its error shows.
|
||||
const failed = page.locator('#reviewTagsVisible .text-pm-error');
|
||||
for (let tries = 0; tries < 5; tries++) {
|
||||
await expect.poll(async () => (await confirm.isVisible()) || (await failed.count()) > 0).toBe(true);
|
||||
if (await confirm.isVisible()) return;
|
||||
const answered = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/single'));
|
||||
await page.locator('#skipReviewBtn').click();
|
||||
await answered;
|
||||
}
|
||||
await expect(confirm).toBeVisible();
|
||||
}
|
||||
|
||||
// #379: the admin review asks before re-tagging an image a model already tagged, and Apply & Next writes that image's
|
||||
// tags (its bytes, never its prompt).
|
||||
test('admin review: an auto-tagged image asks first; Re-tag then Apply & Next tags the image', async ({ page, request }) => {
|
||||
await runNativeBatch(request);
|
||||
await page.goto('/prompt_manager/admin');
|
||||
await page.locator('#autoTagBtn').click();
|
||||
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
|
||||
await page.locator('#startReviewBtn').click();
|
||||
const confirm = page.locator('#retagConfirmModal');
|
||||
await untilAutoTagged(page, confirm);
|
||||
await expect(confirm.locator('#retagExistingTags span').first()).toHaveText('sandstone');
|
||||
await page.locator('#retagConfirmBtn').click();
|
||||
await expect(page.locator('#autoTagReviewModal')).toBeVisible();
|
||||
await expect(page.locator('#reviewTagsVisible .tag-chip').first()).toContainText('sandstone');
|
||||
const applied = page.waitForRequest(item => item.url().endsWith('/prompt_manager/autotag/apply'));
|
||||
const answer = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/apply'));
|
||||
await page.locator('#applyReviewBtn').click();
|
||||
const body = (await applied).postDataJSON();
|
||||
expect(body.prompt_id).toBeUndefined();
|
||||
expect(body.sha256).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect((await answer).status()).toBe(200);
|
||||
});
|
||||
|
||||
test('admin review: Skip All Tagged passes every auto-tagged image', async ({ page, request }) => {
|
||||
await runNativeBatch(request);
|
||||
await page.goto('/prompt_manager/admin');
|
||||
await page.locator('#autoTagBtn').click();
|
||||
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
|
||||
await page.locator('#startReviewBtn').click();
|
||||
const confirm = page.locator('#retagConfirmModal');
|
||||
await untilAutoTagged(page, confirm);
|
||||
await page.locator('#retagSkipAllBtn').click();
|
||||
await expect(page.locator('#retagConfirmModal')).toBeHidden();
|
||||
await expect(page.getByText('Review complete!')).toBeVisible();
|
||||
});
|
||||
|
||||
test('gallery review: Apply & Next tags the image by its path and bytes', async ({ page }) => {
|
||||
await page.goto('/prompt_manager/gallery');
|
||||
await page.locator('#autoTagBtn').click();
|
||||
await page.locator('input[name="autoTagModel"][value="wd14-swinv2"]').check();
|
||||
const suggested = page.waitForRequest(item => item.url().endsWith('/prompt_manager/autotag/single'));
|
||||
await page.locator('#startReviewBtn').click();
|
||||
expect((await suggested).postDataJSON().existing).toBe('merge');
|
||||
await expect(page.locator('#reviewTagsContainer .tag-chip').first()).toContainText('sandstone');
|
||||
const applied = page.waitForRequest(item => item.url().endsWith('/prompt_manager/autotag/apply'));
|
||||
const answer = page.waitForResponse(item => item.url().endsWith('/prompt_manager/autotag/apply'));
|
||||
await page.locator('#applyReviewBtn').click();
|
||||
const body = (await applied).postDataJSON();
|
||||
expect(Object.keys(body).sort()).toEqual(['path', 'root_index', 'sha256', 'tags'].filter(key => key in body).sort());
|
||||
expect(body.prompt_id).toBeUndefined();
|
||||
expect((await answer).status()).toBe(200);
|
||||
});
|
||||
|
||||
@@ -1,163 +0,0 @@
|
||||
// System S13 (#330): the legacy admin and gallery pages open native System for their System tools. Each replaced
|
||||
// control is a plain link (`workspace#/system/<id>`, a new tab), and the legacy flows S13 keeps still work.
|
||||
import { startCoverage, saveCoverage } from '../../tools/vendor/browser-coverage.mjs';
|
||||
import { test, expect } from './fixtures.mjs';
|
||||
|
||||
const PAGES = Object.freeze({ admin: '/prompt_manager/admin', gallery: '/prompt_manager/gallery' });
|
||||
const LABELS = Object.freeze({
|
||||
overview: 'Overview',
|
||||
appearance: 'Appearance',
|
||||
sources: 'Sources & scans',
|
||||
integrations: 'Integrations',
|
||||
tagging: 'Auto-tagging',
|
||||
maintenance: 'Maintenance',
|
||||
backups: 'Backup & export',
|
||||
diagnostics: 'Diagnostics & logs',
|
||||
});
|
||||
|
||||
// One row per System link of the design's §2 map: the page, the control's label, how to find it, the section it opens
|
||||
// and what a test must arrange before the control shows.
|
||||
const LINKS = Object.freeze([
|
||||
{ page: 'admin', label: 'Diagnostics', locate: page => page.locator('#diagnosticsBtn'), section: 'diagnostics' },
|
||||
{ page: 'admin', label: 'View Logs', locate: page => page.locator('#logsBtn'), section: 'diagnostics' },
|
||||
]);
|
||||
|
||||
async function post(request, url, data) {
|
||||
await expect(await request.post(url, data === undefined ? {} : { data })).toBeOK();
|
||||
}
|
||||
|
||||
test.beforeEach(async ({ page, request, browserName }) => {
|
||||
await startCoverage(page, browserName);
|
||||
await post(request, '/__test__/reset');
|
||||
await post(request, '/__test__/system/reset');
|
||||
await post(request, '/__test__/tagging/reset');
|
||||
// The 3.2.0 update notice would cover the admin page's toolbar.
|
||||
await page.addInitScript(() => { localStorage.setItem('pm_update_notice_dismissed', '3.2.0'); });
|
||||
page.legacyErrors = [];
|
||||
page.on('pageerror', error => page.legacyErrors.push(error.message));
|
||||
});
|
||||
|
||||
test.afterEach(async ({ page, request, browserName }) => {
|
||||
await saveCoverage(page, browserName);
|
||||
await post(request, '/__test__/tagging/reset');
|
||||
await post(request, '/__test__/system/reset');
|
||||
expect(page.legacyErrors).toEqual([]);
|
||||
});
|
||||
|
||||
/** Opens the legacy page with the link showing; returns the link. */
|
||||
async function openLink(page, request, link) {
|
||||
if (link.arrange) await link.arrange(page, request);
|
||||
await page.goto(PAGES[link.page]);
|
||||
if (link.reveal) await link.reveal(page);
|
||||
const control = link.locate(page);
|
||||
await expect(control).toBeVisible();
|
||||
return control;
|
||||
}
|
||||
|
||||
for (const link of LINKS) {
|
||||
test(`legacy ${link.page} ${link.label} opens System ${link.section}`, async ({ page, context, request }) => {
|
||||
const control = await openLink(page, request, link);
|
||||
const legacyUrl = page.url();
|
||||
await expect(control).toHaveAttribute('href', `workspace#/system/${link.section}`);
|
||||
await expect(control).toHaveAttribute('target', '_blank');
|
||||
await expect(control).toHaveAttribute('rel', 'noopener');
|
||||
const opened = context.waitForEvent('page');
|
||||
await control.click();
|
||||
const system = await opened;
|
||||
await expect(system).toHaveURL(new RegExp(`/prompt_manager/workspace#/system/${link.section}$`));
|
||||
// System's page heading is "System"; the open section is the pressed button of its section switcher.
|
||||
const main = system.locator('#workspace-main');
|
||||
await expect(main.getByRole('heading', { level: 1, name: 'System', exact: true })).toBeVisible();
|
||||
await expect(main.getByRole('group', { name: 'System sections', exact: true })
|
||||
.getByRole('button', { name: LABELS[link.section], exact: true })).toHaveAttribute('aria-pressed', 'true');
|
||||
expect(page.url()).toBe(legacyUrl);
|
||||
await system.close();
|
||||
});
|
||||
}
|
||||
|
||||
test('a System link sends no request from the legacy page', async ({ page, context, request }) => {
|
||||
for (const link of LINKS) {
|
||||
const control = await openLink(page, request, link);
|
||||
// The page's own start-up reads (list, film strips, settings) finish first; then the click must add none.
|
||||
await page.waitForLoadState('networkidle');
|
||||
const sent = [];
|
||||
const record = item => sent.push(`${item.method()} ${item.url()}`);
|
||||
page.on('request', record);
|
||||
const opened = context.waitForEvent('page');
|
||||
await control.click();
|
||||
const system = await opened;
|
||||
await system.waitForLoadState('domcontentloaded');
|
||||
page.off('request', record);
|
||||
expect(sent, `${link.page} ${link.label}`).toEqual([]);
|
||||
await system.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('kept admin flows still work', async ({ page, context }) => {
|
||||
await page.goto(PAGES.admin);
|
||||
const texts = page.locator('#resultsList .prompt-text');
|
||||
await expect(texts.first()).toBeVisible();
|
||||
|
||||
// K1: sorting reorders the list.
|
||||
for (const [value, order] of [['text_asc', 1], ['text_desc', -1]]) {
|
||||
const listed = page.waitForResponse(item => /\/prompt_manager\/(recent|search)\?/.test(item.url()) && item.url().includes(`sort=${value}`));
|
||||
await page.locator('#sortBy').selectOption(value);
|
||||
await listed;
|
||||
await expect.poll(async () => {
|
||||
const shown = await texts.allTextContents();
|
||||
// The server sorts case-insensitively (SQLite NOCASE), not by locale.
|
||||
const key = text => text.toLowerCase();
|
||||
const sorted = [...shown].sort((a, b) => order * (key(a) < key(b) ? -1 : key(a) > key(b) ? 1 : 0));
|
||||
return shown.length > 1 && shown.join('\n') === sorted.join('\n');
|
||||
}).toBe(true);
|
||||
}
|
||||
// K3: Add Prompt saves a prompt the list then shows.
|
||||
await page.locator('#addPromptBtn').click();
|
||||
await page.locator('#addPromptText').fill('Kept flow lighthouse');
|
||||
const saved = page.waitForResponse(item => item.request().method() === 'POST' && item.url().endsWith('/prompt_manager/save'));
|
||||
await page.locator('#saveNewPromptBtn').click();
|
||||
expect((await saved).ok()).toBe(true);
|
||||
await expect(page.locator('#addPromptModal')).toBeHidden();
|
||||
await expect(texts.filter({ hasText: 'Kept flow lighthouse' })).toBeVisible();
|
||||
|
||||
// K1: a search narrows the list, and a selection enables the bulk buttons.
|
||||
const all = await texts.allTextContents();
|
||||
const word = all[0].trim().split(/\s+/)[0];
|
||||
const searched = page.waitForResponse(item => item.url().includes('/prompt_manager/search?') && item.url().includes('text='));
|
||||
await page.locator('#searchText').fill(word);
|
||||
await page.locator('#searchBtn').click();
|
||||
await searched;
|
||||
await expect.poll(async () => {
|
||||
const shown = await texts.allTextContents();
|
||||
return shown.length > 0 && shown.length < all.length && shown.every(text => text.toLowerCase().includes(word.toLowerCase()));
|
||||
}).toBe(true);
|
||||
await expect(page.locator('#bulkDeleteBtn')).toBeDisabled();
|
||||
await page.locator('#resultsList .prompt-checkbox').first().check();
|
||||
for (const id of ['#bulkDeleteBtn', '#bulkTagBtn', '#bulkCategoryBtn']) await expect(page.locator(id)).toBeEnabled();
|
||||
|
||||
// K4: Stats reads the statistics again; Export downloads the prompts.
|
||||
const stats = page.waitForResponse(item => item.url().endsWith('/prompt_manager/stats') && item.ok());
|
||||
await page.locator('#statsBtn').click();
|
||||
await stats;
|
||||
await expect(page.getByText('Stats refreshed successfully!')).toBeVisible();
|
||||
const download = page.waitForEvent('download');
|
||||
await page.locator('#exportBtn').click();
|
||||
expect((await download).suggestedFilename()).toMatch(/^prompts-\d{4}-\d{2}-\d{2}\.json$/);
|
||||
|
||||
// K5: Scan Images opens its dialog (with the quick backup) without starting a scan.
|
||||
await page.locator('#scanBtn').click();
|
||||
await expect(page.locator('#scanModal')).toBeVisible();
|
||||
await expect(page.locator('#quickBackupBtn')).toBeVisible();
|
||||
await page.locator('#cancelScan').click();
|
||||
await expect(page.locator('#scanModal')).toBeHidden();
|
||||
|
||||
// K7: Metadata and Gallery open their pages in a new tab.
|
||||
for (const [id, path] of [['#metadataBtn', '/prompt_manager/gallery.html'], ['#galleryBtn', '/prompt_manager/gallery']]) {
|
||||
const opened = context.waitForEvent('page');
|
||||
await page.locator(id).click();
|
||||
const tab = await opened;
|
||||
await tab.waitForLoadState('domcontentloaded');
|
||||
expect(new URL(tab.url()).pathname).toBe(path);
|
||||
await tab.close();
|
||||
}
|
||||
});
|
||||
@@ -127,9 +127,9 @@ function mapTaggingConcept(reference, { template, phone }) {
|
||||
actions: { ...reference.actions, buttons: [preview, ['Download again', ''], ['Unload', '']] },
|
||||
// D134: the prompt-tag batch, reviewed first.
|
||||
operations: [{ tree: ['div', 'div.row-actions'], title: 'Tag output images', description: BATCH, actions: [['Review', '']] }],
|
||||
// D132 (#398): every select shows its whole chosen option at every width (at 390 the concept's 120px cap cuts
|
||||
// them; production lets each grow past it).
|
||||
selectsFit: reference.selectsFit.map(() => true),
|
||||
// D132: the engine select shows its whole chosen option at every width (at 390 the concept's 120px cap cuts it;
|
||||
// `#tag-model` alone may grow there). Existing tags keeps the concept's own fit, cut or not.
|
||||
selectsFit: reference.selectsFit.map((fit, index) => (index === 0 ? true : fit)),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -168,28 +168,16 @@ for (const theme of THEMES) for (const [width, height] of SIZES) {
|
||||
});
|
||||
}
|
||||
|
||||
// D132 (#398) at phone width and below the concept's smallest size: every option of every select, once chosen, shows
|
||||
// whole, with no page scroll.
|
||||
for (const theme of THEMES) for (const width of [390, 320]) {
|
||||
test(`auto-tagging selects fit every option at ${width} wide ${theme}`, async ({ page }, testInfo) => {
|
||||
await page.setViewportSize({ width, height: 844 });
|
||||
await productionTagging(page, theme);
|
||||
await page.locator('#workspace-main #tag-existing').selectOption('replace_after_review');
|
||||
await page.locator('#workspace-main section.surface').screenshot({ path: testInfo.outputPath(`production-tagging-selects-${theme}-${width}.png`) });
|
||||
const selects = page.locator('#workspace-main section.surface > .setting-row > select');
|
||||
await expect(selects).toHaveCount(2);
|
||||
for (let index = 0; index < 2; index += 1) {
|
||||
const select = selects.nth(index);
|
||||
const values = await select.evaluate(node => [...node.options].map(option => option.value));
|
||||
for (const value of values) {
|
||||
await select.selectOption(value);
|
||||
const measured = await page.evaluate(measureTagging, '#workspace-main');
|
||||
expect(measured.selectsFit, `${await select.getAttribute('id')}=${value}`).toEqual([true, true]);
|
||||
expect(measured.pageScroll, value).toBe(false);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
// D132 below the concept's smallest size: on a 320px phone the engine select still shows its whole option.
|
||||
for (const theme of THEMES) test(`auto-tagging engine select fits at 320 wide ${theme}`, async ({ page }) => {
|
||||
await page.setViewportSize({ width: 320, height: 700 });
|
||||
const production = await productionTagging(page, theme);
|
||||
for (const model of ['hf', 'gguf', 'wd14-swinv2']) {
|
||||
await page.locator('#workspace-main #tag-model').selectOption(model);
|
||||
expect((await page.evaluate(measureTagging, '#workspace-main')).selectsFit[0], model).toBe(true);
|
||||
}
|
||||
expect(production.pageScroll).toBe(false);
|
||||
});
|
||||
|
||||
test('auto-tagging comparison detects mutations', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
@@ -234,9 +222,6 @@ test('auto-tagging comparison detects mutations', async ({ page, request }) => {
|
||||
const undoSqueeze = await mutate(page, { css: '#workspace-main #tag-model { max-width: 60px !important; min-width: 0 !important; }' });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['selectsFit.0']));
|
||||
await undoSqueeze();
|
||||
const undoSqueezeExisting = await mutate(page, { css: '#workspace-main #tag-existing { max-width: 60px !important; min-width: 0 !important; }' });
|
||||
expect(await paths()).toEqual(expect.arrayContaining(['selectsFit.1']));
|
||||
await undoSqueezeExisting();
|
||||
// A row that wraps where the concept's does not.
|
||||
const undoWrap = await mutate(page, { css: '#workspace-main .setting-row:has(> #tag-existing) { flex-wrap: wrap; }' });
|
||||
expect(await paths()).toEqual(['rows.1.flexWrap']);
|
||||
@@ -248,14 +233,14 @@ test('auto-tagging comparison detects mutations', async ({ page, request }) => {
|
||||
expect(diffInventory(expected, await measure())).toEqual([]);
|
||||
});
|
||||
|
||||
// D132 at phone width: only the engine row wraps; Existing tags grows beside its text instead (#398).
|
||||
// D132 at phone width, unmapped: only the engine row wraps; Existing tags keeps the concept's cut select.
|
||||
test('auto-tagging at 390 wraps only the engine row', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 390, height: 844 });
|
||||
const concept = await conceptTagging(page, 'dark');
|
||||
const production = await productionTagging(page, 'dark');
|
||||
expect(diffInventory(mapTaggingConcept(concept, await savedTemplate(request, page)), production)).toEqual([]);
|
||||
expect(production.rows.map(row => row.flexWrap)).toEqual(['wrap', 'nowrap', 'nowrap', 'nowrap', 'nowrap']);
|
||||
expect(production.selectsFit).toEqual([true, true]);
|
||||
expect(production.selectsFit[1]).toBe(concept.selectsFit[1]);
|
||||
});
|
||||
|
||||
/** Runs in the page: the open review dialog: its title, body and the concept's tokens. */
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
// A restore the open workspace page has not noticed yet (#413).
|
||||
//
|
||||
// Specs that restore through the API while a page is open check what the page does with a restore it did not see:
|
||||
// its next write is refused as "Prompt storage was restored since this dialog opened". But the page's Activity store
|
||||
// reads the job list every 15 s (activity-store.mjs POLL_MS), and a read the server answers while the restore job runs
|
||||
// makes the page follow it: when it finishes, the page is shown again from the restored data (S9, index.mjs
|
||||
// onRestoreFinished), which closes any open dialog. On a slow runner a restore can take longer than the time left to
|
||||
// the next read (CI run 40646112668: the restore took 16 s), so whether the page saw it depended on timing.
|
||||
//
|
||||
// `armUnseenRestores(page)`, before the page loads, passes every job-list read of the page through one route, so it
|
||||
// knows which reads the server has not answered yet. `unseenRestore(page, restoring)` then closes the gate, waits
|
||||
// for those reads' answers (sent before the restore began, they cannot list it), runs `restoring()`, and only after
|
||||
// it has answered lets the held reads go on: they read the restore as already ended, as a tab that read its job list
|
||||
// just before and just after the restore does. A restore that is never seen running never reloads the page.
|
||||
const JOBS = '**/prompt_manager/jobs';
|
||||
const gates = new WeakMap();
|
||||
|
||||
/** Route `page`'s job-list reads through the restore gate. Call before the page's first navigation. */
|
||||
export async function armUnseenRestores(page) {
|
||||
const gate = { closed: null, inFlight: 0, idle: [] };
|
||||
gates.set(page, gate);
|
||||
await page.route(JOBS, async route => {
|
||||
if (gate.closed) {
|
||||
await gate.closed;
|
||||
return route.fallback();
|
||||
}
|
||||
gate.inFlight += 1;
|
||||
try {
|
||||
const response = await route.fetch();
|
||||
await route.fulfill({ response }).catch(() => {});
|
||||
} catch {
|
||||
await route.abort().catch(() => {});
|
||||
} finally {
|
||||
gate.inFlight -= 1;
|
||||
if (gate.inFlight === 0) for (const resolve of gate.idle.splice(0)) resolve();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/** Run `restoring()` (an API restore) unseen by `page`'s Activity store; resolves with what it resolves with. */
|
||||
export async function unseenRestore(page, restoring) {
|
||||
const gate = gates.get(page);
|
||||
if (!gate) throw new Error('armUnseenRestores(page) must run before the page loads');
|
||||
let open;
|
||||
gate.closed = new Promise(resolve => { open = resolve; });
|
||||
try {
|
||||
// Reads already on their way were sent before the restore: their answers cannot list it.
|
||||
if (gate.inFlight > 0) await new Promise(resolve => gate.idle.push(resolve));
|
||||
return await restoring();
|
||||
} finally {
|
||||
gate.closed = null;
|
||||
open();
|
||||
}
|
||||
}
|
||||
@@ -132,8 +132,7 @@ test('the gallery cache clear keeps preferences and other tools caches', async (
|
||||
}))).toEqual({ theme: 'light', other: 'keep', prefixed: 'keep', notice: '3.2.0', caches: ['other-tool-gallery-thumbnails'] });
|
||||
});
|
||||
|
||||
// #379: the reviewed tags are that image's own tags; no prompt changes.
|
||||
test('actual gallery review generates and persists tags for the selected root\'s image', async ({ page, request }) => {
|
||||
test('actual gallery review generates and persists tags for the selected root', async ({ page, request }) => {
|
||||
await page.goto('/prompt_manager/gallery');
|
||||
await expect(page.locator('#galleryGrid .image-item')).toHaveCount(2);
|
||||
await page.locator('#autoTagBtn').click();
|
||||
@@ -141,15 +140,13 @@ test('actual gallery review generates and persists tags for the selected root\'s
|
||||
await page.locator('#startReviewBtn').click();
|
||||
const generation = await generated;
|
||||
expect(generation.request().postDataJSON().root_index).toBe(0);
|
||||
await expect(page.locator('#reviewTagsContainer')).toContainText('sandstone');
|
||||
await expect(page.locator('#reviewTagsContainer')).toContainText('generated');
|
||||
const result = page.waitForResponse(response => response.url().endsWith('/prompt_manager/autotag/apply'));
|
||||
await page.locator('#applyReviewBtn').click();
|
||||
expect((await (await result).json()).success).toBe(true);
|
||||
const data = await (await request.get('/prompt_manager/recent?limit=20')).json();
|
||||
expect(data.results.find(prompt => prompt.text === 'Cat portrait').tags).toEqual(['cats']);
|
||||
expect(data.results.find(prompt => prompt.text === 'Dog landscape').tags).toEqual(['dogs']);
|
||||
const tagged = await (await request.get('/prompt_manager/images/output?mode=workspace&tags=sandstone')).json();
|
||||
expect(tagged.images.map(image => `${image.root_index}:${image.path}`)).toEqual(['0:shared/same.png']);
|
||||
expect(data.results.find(prompt => prompt.text === 'Cat portrait').tags).toContain('generated');
|
||||
expect(data.results.find(prompt => prompt.text === 'Dog landscape').tags).not.toContain('generated');
|
||||
});
|
||||
|
||||
test('actual Tags page reloads the selected filter and opens its image viewer', async ({ page }) => {
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
|
||||
import {test,expect} from './fixtures.mjs';
|
||||
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
|
||||
import {readFile,writeFile} from 'node:fs/promises';
|
||||
import {createHash} from 'node:crypto';
|
||||
|
||||
@@ -17,13 +16,13 @@ const marker=(page,id,anchor='10')=>recorded(page).locator(`button[data-run-id="
|
||||
const exact=async locator=>locator.evaluate(node=>node.textContent);
|
||||
|
||||
test.beforeEach(async({page,request,browserName})=>{
|
||||
await startCoverage(page,browserName);await armUnseenRestores(page);page.historyErrors=[];page.on('pageerror',error=>page.historyErrors.push(error.message));
|
||||
await startCoverage(page,browserName);page.historyErrors=[];page.on('pageerror',error=>page.historyErrors.push(error.message));
|
||||
const reset=await request.post('/__test__/reset',{data:{chronological_history:true}});await expect(reset).toBeOK();page.fixture=await reset.json();
|
||||
});
|
||||
test.afterEach(async({page,browserName})=>{await saveCoverage(page,browserName);expect(page.historyErrors).toEqual([]);});
|
||||
|
||||
const command=async(request,data)=>{const response=await request.post('/__test__/history/commands',{data});await expect(response).toBeOK();return response.json();};
|
||||
const restoreSnapshot=async(page,request,bytes)=>{const response=await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'history.db',mimeType:'application/octet-stream',buffer:bytes}}}));await expect(response).toBeOK();expect((await response.json()).success).toBe(true);};
|
||||
const restoreSnapshot=async(request,bytes)=>{const response=await request.post('/prompt_manager/restore',{multipart:{database_file:{name:'history.db',mimeType:'application/octet-stream',buffer:bytes}}});await expect(response).toBeOK();expect((await response.json()).success).toBe(true);};
|
||||
const historyRead=async(request,id)=>{const response=await request.get(`/prompt_manager/library/${id}/history`);await expect(response).toBeOK();return response.json();};
|
||||
const frozenContexts=contexts=>contexts.map(({navigation,positive_prompt_id,negative_prompt_id,pair_id,...frozen})=>frozen);
|
||||
const immutableReceipts=items=>items.map(({current_candidate,...receipt})=>receipt);
|
||||
@@ -63,7 +62,7 @@ for(const boundary of ['window','detail','candidate','inspection','media']) test
|
||||
await command(request,{action:'arm',boundary});const pending=request.get(target);
|
||||
try {
|
||||
await expect.poll(async()=> (await(await request.get('/__test__/history/checkpoint')).json()).entered).toBe(true);
|
||||
await restoreSnapshot(page,request,bytes);
|
||||
await restoreSnapshot(request,bytes);
|
||||
} finally {await command(request,{action:'release'});}
|
||||
const retired=await pending;expect(retired.status()).toBe(409);expect((await retired.json()).code).toBe('history-changed');
|
||||
for(const url of [`/prompt_manager/library/${f.detail_id}/history?limit=1&cursor=${encodeURIComponent(firstWindow.next_cursor)}`,`${base}?epoch=${epoch}`,`${base}/outputs?epoch=${epoch}`,`${base}/outputs?epoch=${epoch}&cursor=${encodeURIComponent(outputs.next_cursor)}`,media.url,media.thumbnail_url,media.inspection_url]) {
|
||||
@@ -131,7 +130,7 @@ test('archive delete merge and prune preserve frozen evidence without repairing
|
||||
|
||||
test('actual restore with reused numeric owner rejects stale history and offers explicit saved recovery',async({page,request})=>{
|
||||
const f=page.fixture;await open(page,f.detail_id);const original=await exact(saved(page).getByRole('region',{name:'Positive prompt',exact:true}).locator('p.prompt-text'));
|
||||
const source=await request.get('/__test__/history/reused-snapshot');await expect(source).toBeOK();await restoreSnapshot(page,request,await source.body());
|
||||
const source=await request.get('/__test__/history/reused-snapshot');await expect(source).toBeOK();await restoreSnapshot(request,await source.body());
|
||||
await marker(page,f.runs.shared).click();await expect(dialog(page)).toContainText('Recorded evidence changed');await page.keyboard.press('Escape');
|
||||
await expect(page.locator('.library-parent-status')).toContainText('changed');expect(await exact(saved(page).getByRole('region',{name:'Positive prompt',exact:true}).locator('p.prompt-text'))).toBe(original);
|
||||
await expect(recorded(page).getByRole('button',{name:'Reload recorded history',exact:true})).toBeVisible();
|
||||
@@ -143,12 +142,12 @@ for(const outcome of ['success','error'])test(`delayed actual old browser ${outc
|
||||
const f=page.fixture;await open(page,f.detail_id);const snapshot=await request.get('/__test__/history/snapshot');const bytes=await snapshot.body();
|
||||
let entered,release;const started=new Promise(resolve=>{entered=resolve;}),held=new Promise(resolve=>{release=resolve;});
|
||||
await page.route(new RegExp(`/history/runs/${f.runs.shared}\\?`),async route=>{
|
||||
if(outcome==='error')await restoreSnapshot(page,request,bytes);
|
||||
if(outcome==='error')await restoreSnapshot(request,bytes);
|
||||
const response=await route.fetch();expect(response.status()).toBe(outcome==='success'?200:409);entered();await held;
|
||||
await route.fulfill({response}).catch(()=>{});
|
||||
});
|
||||
try {
|
||||
await marker(page,f.runs.shared).click();await started;if(outcome==='success')await restoreSnapshot(page,request,bytes);
|
||||
await marker(page,f.runs.shared).click();await started;if(outcome==='success')await restoreSnapshot(request,bytes);
|
||||
await page.evaluate(()=>{location.hash='#/overview';});await expect(page.locator('#workspace-main h1')).toHaveText('Overview');release();
|
||||
await page.unroute(new RegExp(`/history/runs/${f.runs.shared}\\?`));await open(page,f.detail_id);
|
||||
await expect(recorded(page).locator('button[data-run-id]')).toHaveCount(0);
|
||||
@@ -259,7 +258,7 @@ test('inspection first discovers a real restore and immediately retires every su
|
||||
if(changed)window.recordedRecoveryStates.push({receipts:run.querySelectorAll('[data-receipt]').length,active:[...run.querySelectorAll('[data-send-role],[data-copy-role],[data-view-receipt]')].filter(node=>!node.disabled&&node.checkVisibility()).length});
|
||||
});observer.observe(run,{subtree:true,childList:true,attributes:true,characterData:true});
|
||||
});
|
||||
await restoreSnapshot(page,request,await snapshot.body());
|
||||
await restoreSnapshot(request,await snapshot.body());
|
||||
} finally {await command(request,{action:'release'});}
|
||||
const response=await inspectionRead;expect(response.status()).toBe(409);expect((await response.json()).code).toBe('history-changed');
|
||||
await expect(dialog(page).locator('.recorded-run-status')).toContainText('Recorded evidence changed');
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import {writeFile} from 'node:fs/promises';
|
||||
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
|
||||
import {test,expect} from './fixtures.mjs';
|
||||
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
|
||||
import {captureCanonicalState} from './history-canonical-comparison.mjs';
|
||||
import {inventoryOf,mapConcept,diffInventory} from './library-editor-canonical-comparison.mjs';
|
||||
|
||||
@@ -14,7 +13,7 @@ const RATINGS=[['5','5 / 5'],['4','4 / 5'],['3','3 / 5'],['2','2 / 5'],['1','1 /
|
||||
const editorDialog=page=>page.locator('dialog.library-editor');
|
||||
|
||||
test.beforeEach(async({page,request,browserName})=>{
|
||||
await startCoverage(page,browserName);await armUnseenRestores(page);
|
||||
await startCoverage(page,browserName);
|
||||
const reset=await request.post('/__test__/library-editor/reset',{data:{}});await expect(reset).toBeOK();page.fixture=await reset.json();
|
||||
page.editorErrors=[];page.on('pageerror',error=>page.editorErrors.push(error.message));
|
||||
page.writes=[];page.on('request',item=>{if(item.method()!=='GET')page.writes.push({method:item.method(),url:new URL(item.url()).pathname,body:item.postDataJSON()});});
|
||||
@@ -261,8 +260,7 @@ test('recovery states render inline and never replay a write',async({page,reques
|
||||
page.writes.length=0;
|
||||
dialog=await openEditor(page,{mode:'new'});await ready(dialog);
|
||||
const snapshot=await request.get('/__test__/library-editor/snapshot');await expect(snapshot).toBeOK();
|
||||
const buffer=await snapshot.body();
|
||||
const restore=await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer}}}));
|
||||
const restore=await request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:await snapshot.body()}}});
|
||||
await expect(restore).toBeOK();page.writes.length=0;
|
||||
await field(dialog,'Title').fill('After restore');await field(dialog,'Positive prompt').fill('Restored storage prompt');await button(dialog,'Save').click();
|
||||
await expect(note(dialog)).toHaveText('Prompt storage was restored since this dialog opened. Nothing was saved.');
|
||||
@@ -590,7 +588,7 @@ test('saved refresh failure retries only reads',async({page})=>{
|
||||
|
||||
test('restored storage after a confirmed save or before an archive offers an explicit Library reload',async({page,request})=>{
|
||||
const f=page.fixture;const snapshot=await(await request.get('/__test__/library-editor/snapshot')).body();
|
||||
const restore=()=>unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}}));
|
||||
const restore=()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}});
|
||||
await openHistory(page,f.positive_id);
|
||||
await page.route(`**/prompt_manager/library/${f.positive_id}`,async route=>{
|
||||
if(route.request().method()!=='PATCH')return route.continue();
|
||||
@@ -617,7 +615,7 @@ test('restored storage after a confirmed save or before an archive offers an exp
|
||||
const DISCARDS='Leaving this dialog discards your entries.';
|
||||
test('restored storage before an Edit or Add negative save offers only the Library reload exit',async({page,request})=>{
|
||||
const f=page.fixture;const snapshot=await(await request.get('/__test__/library-editor/snapshot')).body();
|
||||
const restore=()=>unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}}));
|
||||
const restore=()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'editor.db',mimeType:'application/octet-stream',buffer:snapshot}}});
|
||||
await openHistory(page,f.positive_id);
|
||||
await page.getByRole('button',{name:'Edit metadata',exact:true}).click();let dialog=editorDialog(page);await ready(dialog);
|
||||
await expect(await restore()).toBeOK();
|
||||
|
||||
@@ -47,17 +47,6 @@ async function serverJson(request, url) {
|
||||
return response.json();
|
||||
}
|
||||
|
||||
/**
|
||||
* The restore job's own end (#361): its progress stream closes when the job takes its final state, however long the
|
||||
* disk-bound publish takes (bounded by the test, not by a 5 s expect), and the job finished. The toast the page shows
|
||||
* for it is then the page's reaction alone, checked with the default timeout.
|
||||
*/
|
||||
async function restoreFinished(request, id) {
|
||||
await expect(await request.get(`/prompt_manager/jobs/${id}/events`)).toBeOK();
|
||||
const { job } = await serverJson(request, `/prompt_manager/jobs/${id}`);
|
||||
expect([job.kind, job.state]).toEqual(['restore', 'finished']);
|
||||
}
|
||||
|
||||
async function openBackups(page) {
|
||||
// A fresh page each time: the drafts live for the page session (Q10), and a hash-only goto would not reload.
|
||||
if (page.url().endsWith('#/system/backups')) await page.reload();
|
||||
@@ -227,7 +216,6 @@ test('Restore starts the job, opens Activity; views reload after it finishes', a
|
||||
const { job } = await (await applied).json();
|
||||
expect([job.kind, job.title, job.cancellable]).toEqual(['restore', 'Database restore', false]);
|
||||
await expect(page).toHaveURL(`${workspace}#/activity/${job.id}`);
|
||||
await restoreFinished(request, job.id);
|
||||
await expect(toast(page)).toHaveText(/^Database restore finished · Restored \d+ prompts?, /);
|
||||
// The library is the backup's again, and the page says so without a reload: the sidebar counts were read again.
|
||||
expect(await libraryTotal(request)).toBe(restoredTotal);
|
||||
@@ -256,8 +244,7 @@ test('a finished restore shows the open page again from the restored data', asyn
|
||||
const card = page.locator('#workspace-main article.prompt-card').filter({ hasText: 'S9 prompt the restore removes' });
|
||||
await expect(card).toHaveCount(1);
|
||||
await expect(await request.post('/__test__/release')).toBeOK();
|
||||
const { job } = await (await applied).json();
|
||||
await restoreFinished(request, job.id);
|
||||
await applied;
|
||||
await expect(toast(page)).toHaveText(/^Database restore finished · /);
|
||||
// The shelf was read again from the restored library, on the same route.
|
||||
await expect(card).toHaveCount(0);
|
||||
@@ -435,7 +422,6 @@ test('a refused restore reopens its review: Restore retries it, Discard releases
|
||||
await dialog(page).getByRole('button', { name: 'Restore', exact: true }).click();
|
||||
const { job } = await (await applied).json();
|
||||
await expect(page).toHaveURL(`${workspace}#/activity/${job.id}`);
|
||||
await restoreFinished(request, job.id);
|
||||
await expect(toast(page)).toHaveText(/^Database restore finished · /);
|
||||
expect(await stagesOf(request)).toBe(0);
|
||||
});
|
||||
|
||||
@@ -516,7 +516,7 @@ test('a rollover after the review deletes nothing and Activity offers Review aga
|
||||
test('Clear all logs states both counts, confirms first and reads everything again', async ({ page, request }) => {
|
||||
await seedLogs(request, LOG_FOLDER);
|
||||
const truncates = [];
|
||||
page.on('request', item => { if (item.url().endsWith('/prompt_manager/logs/truncate')) truncates.push(item.postDataJSON()); });
|
||||
page.on('request', item => { if (item.url().endsWith('/prompt_manager/logs/truncate')) truncates.push(item.url()); });
|
||||
await openDiagnostics(page);
|
||||
await expect(lines(page)).toContainText('Database backup verified');
|
||||
const details = await openFiles(page);
|
||||
@@ -535,10 +535,6 @@ test('Clear all logs states both counts, confirms first and reads everything aga
|
||||
await dialog(page).getByRole('button', { name: 'Clear', exact: true }).click();
|
||||
await expect(toast(page)).toHaveText('Logs cleared · 3 rotated copies deleted.');
|
||||
expect(truncates).toHaveLength(1);
|
||||
// #389: Clear sends back exactly the files the dialog was built from, each with its identity string.
|
||||
expect(truncates[0].expected.map(row => row.name).sort()).toEqual(['prompt_manager.log', 'prompt_manager.log.1',
|
||||
'prompt_manager.log.2', 'prompt_manager.log.3']);
|
||||
for (const row of truncates[0].expected) expect(row.identity).toMatch(/^\d+:\d+:\d+$/);
|
||||
// Read again: the viewer, the file line and the list.
|
||||
await expect(lines(page)).not.toContainText('Database backup verified');
|
||||
await expect(details.locator(':scope > p.small').first()).toHaveText(/rotated copies: 0$/);
|
||||
@@ -568,23 +564,6 @@ test('Clear all logs reads the files again before it states what it deletes', as
|
||||
expect(await names(request)).toEqual(['prompt_manager.log', 'prompt_manager.log.1', 'prompt_manager.log.2', 'prompt_manager.log.3']);
|
||||
});
|
||||
|
||||
test('a rollover while Clear all logs is open deletes nothing and says why (#389)', async ({ page, request }) => {
|
||||
await seedLogs(request, { rotated: [1024, 1024], active: 100 });
|
||||
await openDiagnostics(page);
|
||||
await openFiles(page);
|
||||
await operation(page, 'Clear all logs').getByRole('button', { name: 'Clear all logs…', exact: true }).click();
|
||||
await expect(dialog(page).locator('#modal-body > p')).toHaveText(/^Empties prompt_manager\.log, deletes 2 rotated copies /);
|
||||
// The log rolls over after the confirmation was built: the copies it listed are not the ones there now.
|
||||
await expect(await request.post('/__test__/logs/rotate')).toBeOK();
|
||||
const rolled = await names(request);
|
||||
const answer = page.waitForResponse(response => response.url().endsWith('/prompt_manager/logs/truncate'));
|
||||
await dialog(page).getByRole('button', { name: 'Clear', exact: true }).click();
|
||||
expect((await answer).status()).toBe(409);
|
||||
await expect(toast(page)).toHaveText('The log files changed since you confirmed; nothing was cleared. '
|
||||
+ 'Read the list again and clear again.');
|
||||
expect(await names(request)).toEqual(rolled);
|
||||
});
|
||||
|
||||
test('each listed file downloads by its returned name and a link has no Download', async ({ page, request }) => {
|
||||
await seedLogs(request, LOG_FOLDER);
|
||||
await openDiagnostics(page);
|
||||
|
||||
@@ -513,41 +513,6 @@ test('search finds an image by its image tag', async ({ page, request }) => {
|
||||
await expect(card(page, UNLINKED)).toBeVisible();
|
||||
});
|
||||
|
||||
// #379 CUJ 8: the Tags filter finds images by their own image tags through the tag index; no prompt carries them, and
|
||||
// selecting a match shows its prompt. A linked prompt's tag still combines with the image tags.
|
||||
test('the Tags filter finds images by image tags only, and selecting one shows its prompt (#379)', async ({ page, request }) => {
|
||||
await saveImageTags(request, FIRST, ['beach', '1girl', 'waves']);
|
||||
await saveImageTags(request, SECOND, ['beach', 'waves']);
|
||||
await saveImageTags(request, UNLINKED, ['beach', '1girl', 'waves', 'gulls']);
|
||||
await page.goto(images);
|
||||
await expect(page.locator('.gallery-body article[data-image-key]')).toHaveCount(manifest.files.length);
|
||||
const filters = page.locator('#workspace-main search.toolbar').getByRole('button', { name: /^Filters(?: · \d+)?$/ });
|
||||
const refine = page.getByRole('dialog', { name: 'Refine this view', exact: true });
|
||||
const applyTags = async value => {
|
||||
await filters.click();
|
||||
await refine.getByLabel('Tags', { exact: true }).fill(value);
|
||||
const listing = page.waitForRequest(item => item.url().includes('/images/output?')
|
||||
&& new URL(item.url()).searchParams.get('tags') !== null);
|
||||
await refine.getByRole('button', { name: 'Apply filters', exact: true }).click();
|
||||
await listing;
|
||||
};
|
||||
await applyTags('beach, 1girl, waves');
|
||||
// Every listed image whose image tags hold all three; SECOND lacks 1girl.
|
||||
await expect(page.locator('.gallery-body article[data-image-key]')).toHaveCount(2);
|
||||
await expect(card(page, FIRST)).toBeVisible();
|
||||
await expect(card(page, UNLINKED)).toBeVisible();
|
||||
await expect(card(page, SECOND)).toHaveCount(0);
|
||||
// The newest match is selected; the inspector shows its linked prompt.
|
||||
await infoButton(page, FIRST).click();
|
||||
await expect(card(page, FIRST)).toHaveClass(/\bselected\b/);
|
||||
await expect(panel(page).locator('.panel-heading h2')).toHaveText(manifest.title);
|
||||
// The linked positive's own tag combines: SECOND has beach and waves itself and minimal through its prompt.
|
||||
await applyTags(`beach, waves, ${manifest.prompt_tags[0]}`);
|
||||
await expect(page.locator('.gallery-body article[data-image-key]')).toHaveCount(2);
|
||||
await expect(card(page, FIRST)).toBeVisible();
|
||||
await expect(card(page, SECOND)).toBeVisible();
|
||||
});
|
||||
|
||||
// Task C4 rulings: a save re-reads the selected file's details without a loading flash, and under a Tags filter or a
|
||||
// search the loaded listing is re-read in place (scroll and selection kept while the file still matches).
|
||||
const cards = page => page.locator('.gallery-body article[data-image-key]');
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
|
||||
import {writeFile} from 'node:fs/promises';
|
||||
import {test,expect} from './fixtures.mjs';
|
||||
import {armUnseenRestores,unseenRestore} from './unseen-restore.mjs';
|
||||
import {captureCanonicalState} from './history-canonical-comparison.mjs';
|
||||
import {pageInventoryOf,inventoryOf,diffInventory,mapVocabularyConcept,mapVocabularyDialog} from './vocabulary-canonical-comparison.mjs';
|
||||
|
||||
@@ -41,7 +40,7 @@ async function expectLibrary(page,{tags='',category='',count}) {
|
||||
|
||||
let manifest;
|
||||
test.beforeEach(async({page,request,browserName})=>{
|
||||
await startCoverage(page,browserName);await armUnseenRestores(page);manifest=await seed(request);
|
||||
await startCoverage(page,browserName);manifest=await seed(request);
|
||||
page.vocabularyErrors=[];page.on('pageerror',error=>page.vocabularyErrors.push(error.message));
|
||||
});
|
||||
test.afterEach(async({page,browserName})=>{await saveCoverage(page,browserName);expect(page.vocabularyErrors).toEqual([]);});
|
||||
@@ -265,8 +264,7 @@ const toast=page=>page.locator('#workspace-toast');
|
||||
const actionNames=scope=>scope.locator('.dialog-actions button').evaluateAll(nodes=>nodes.map(node=>node.textContent.trim()));
|
||||
const snapshot=async request=>{const response=await request.get('/__test__/vocabulary/snapshot');await expect(response).toBeOK();return response.json();};
|
||||
const backup=async request=>{const response=await request.get('/__test__/vocabulary/backup');await expect(response).toBeOK();return response.body();};
|
||||
// A restore the open page has not seen, so its dialog stays to report it (#413).
|
||||
const restore=async(page,request,buffer)=>expect(await unseenRestore(page,()=>request.post('/prompt_manager/restore',{multipart:{database_file:{name:'vocabulary.db',mimeType:'application/octet-stream',buffer}}}))).toBeOK();
|
||||
const restore=async(request,buffer)=>expect(await request.post('/prompt_manager/restore',{multipart:{database_file:{name:'vocabulary.db',mimeType:'application/octet-stream',buffer}}})).toBeOK();
|
||||
const tagNames=rows=>rows.tags.map(([,name])=>name);
|
||||
function trackWrites(page) {
|
||||
const writes=[];page.on('request',item=>{if(item.method()!=='GET')writes.push({method:item.method(),url:new URL(item.url()).pathname,body:item.postDataJSON()});});return writes;
|
||||
@@ -484,7 +482,7 @@ test('conflict and restore states keep the dialog and write nothing',async({page
|
||||
|
||||
const before=await snapshot(request);
|
||||
box=await openDialog(page,main(page).getByRole('button',{name:'+ Create tag',exact:true}),'Create tag');
|
||||
await restore(page,request,saved);
|
||||
await restore(request,saved);
|
||||
await field(box,'Tag name').fill('sunrise');await button(box,'Apply').click();
|
||||
await expect(note(box)).toHaveText('Prompt storage was restored since this dialog opened. Nothing was changed.');
|
||||
await expect(button(box,'Apply')).toBeDisabled();
|
||||
@@ -624,7 +622,7 @@ test('delete confirmation read failure, missing tag and restore keep Delete disa
|
||||
await expect(button(box,'Delete')).toBeDisabled();await expect(tagRow(page,'calm').locator('td:nth-child(2)')).toHaveText('2');
|
||||
await button(box,'Cancel').click();
|
||||
|
||||
box=await openRename(page,'mist');await restore(page,request,saved);
|
||||
box=await openRename(page,'mist');await restore(request,saved);
|
||||
await button(box,'Delete tag…').click();
|
||||
await expect(note(box)).toHaveText('Prompt storage was restored since this dialog opened. Nothing was changed.');
|
||||
await expect(button(box,'Delete')).toBeDisabled();expect(await actionNames(box)).toEqual(['Reload','Cancel','Delete']);
|
||||
|
||||
@@ -123,34 +123,6 @@ def _touch(path: Path, size: int, moment: datetime.datetime) -> None:
|
||||
os.utime(path, (stamp, stamp))
|
||||
|
||||
|
||||
def _append_active(manager, log_dir: Path, size: int) -> None:
|
||||
"""Grow the active log as this process's handler does (#389).
|
||||
|
||||
A second handle appending to the file the handler writes is what Clear
|
||||
all logs refuses as another program writing to it, so the bytes go
|
||||
through the handler's own stream; with file logging off, to the file.
|
||||
"""
|
||||
|
||||
def same(path) -> bool:
|
||||
return os.path.normcase(os.path.realpath(path)) == os.path.normcase(
|
||||
os.path.realpath(log_dir / ACTIVE)
|
||||
)
|
||||
|
||||
for handler in manager._file_handlers():
|
||||
stream = getattr(handler, "stream", None)
|
||||
if stream is None or not same(handler.baseFilename):
|
||||
continue
|
||||
handler.acquire()
|
||||
try:
|
||||
stream.write("x" * size)
|
||||
stream.flush()
|
||||
finally:
|
||||
handler.release()
|
||||
return
|
||||
with open(log_dir / ACTIVE, "ab") as handle:
|
||||
handle.write(b"x" * size)
|
||||
|
||||
|
||||
def _bounded(value, name, top=MAX_SEED_BYTES):
|
||||
if isinstance(value, bool) or not isinstance(value, int) or not 0 <= value <= top:
|
||||
raise web.HTTPBadRequest(text=f"{name} must be an integer 0..{top}")
|
||||
@@ -197,7 +169,8 @@ def register_diagnostics_commands(app, routes, api, root: Path):
|
||||
for index, (time_text, level, message) in enumerate(CONCEPT_RECORDS):
|
||||
record = _record(time_text, level, message, index)
|
||||
manager.add_to_buffer(record, record.getMessage())
|
||||
_append_active(manager, log_dir, CONCEPT_ACTIVE)
|
||||
with open(log_dir / ACTIVE, "ab") as handle:
|
||||
handle.write(b"x" * CONCEPT_ACTIVE)
|
||||
for number, size in enumerate(CONCEPT_ROTATED, start=1):
|
||||
moment = ROTATED_TIME - datetime.timedelta(hours=number)
|
||||
_touch(log_dir / f"{ACTIVE}.{number}", size, moment)
|
||||
@@ -221,7 +194,8 @@ def register_diagnostics_commands(app, routes, api, root: Path):
|
||||
manager.add_to_buffer(record, record.getMessage())
|
||||
active = _bounded(data.get("active", 0), "active")
|
||||
if active:
|
||||
_append_active(manager, log_dir, active)
|
||||
with open(log_dir / ACTIVE, "ab") as handle:
|
||||
handle.write(b"x" * active)
|
||||
rotated = data.get("rotated", [])
|
||||
if not isinstance(rotated, list) or len(rotated) > 20:
|
||||
raise web.HTTPBadRequest(text="rotated must be a list of sizes")
|
||||
|
||||
@@ -35,8 +35,7 @@ read. These per-server hooks only change what a System test needs on top:
|
||||
previews for Convert outdated;
|
||||
- ``POST /__test__/system/jxl-encoder`` ``{available}`` (#317) makes the
|
||||
server read as having (``cjxl``) or lacking a JPEG XL encoder without
|
||||
probing, whatever the host has installed (#371: ``false`` stands in for the
|
||||
detector itself); reset restores the thumbnail settings and the real detection.
|
||||
probing; reset restores the thumbnail settings and the real detection.
|
||||
|
||||
- ``POST /__test__/system/duplicate-prompts`` (S7) adds two prompts whose
|
||||
text differs only in case and spacing (a duplicate prompt text group);
|
||||
@@ -119,11 +118,6 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
}
|
||||
from py.api import thumbnail_encode
|
||||
|
||||
real_detect = thumbnail_encode._detect
|
||||
|
||||
def _no_encoder():
|
||||
return None
|
||||
|
||||
monitor = getattr(image_monitor, "_monitor_instance", None)
|
||||
failures: dict[str, int] = {}
|
||||
|
||||
@@ -153,7 +147,6 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
api._health_report = None
|
||||
for (owner, name), value in host_values.items():
|
||||
setattr(owner, name, value)
|
||||
thumbnail_encode._detect = real_detect
|
||||
thumbnail_encode.reset_encoders()
|
||||
image_monitor._monitor_instance = monitor
|
||||
api.__dict__.pop("_library_resolver", None)
|
||||
@@ -305,12 +298,7 @@ def register_system_commands(app, routes, api, root: Path):
|
||||
available = data.get("available")
|
||||
if not isinstance(available, bool):
|
||||
raise web.HTTPBadRequest()
|
||||
# Test-only: the fixture, not the host, decides (#371). Without an encoder the
|
||||
# detector itself stands in, so a re-probe (a "none found" is probed again on
|
||||
# every settings read) cannot find the host's real encoder; with one the cached
|
||||
# answer stands, and a failed write still probes the real detector as before.
|
||||
# Reset restores the real detector.
|
||||
thumbnail_encode._detect = real_detect if available else _no_encoder
|
||||
# Test-only: stand in the cached detection (no probe).
|
||||
with thumbnail_encode._detect_lock:
|
||||
thumbnail_encode._detected = "cjxl" if available else None
|
||||
return web.json_response({"success": True})
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
"""Hang guards for waits on disk-bound work (tests only).
|
||||
|
||||
A test that waits for a job or handoff whose work is file and SQLite I/O
|
||||
(a safety backup with fsync, a restore, hashing and unlinking files) waits
|
||||
on that state; the ceiling below only ends a real hang. Hosted Windows
|
||||
runners stall disk I/O for tens of seconds at a time: a restore took over
|
||||
10 s (merge queue, #382), and on PR #405 every test setup in a 50 s window
|
||||
took up to 9 s against 0.1-0.6 s around it while the test bodies stayed
|
||||
fast, so a one-file duplicate cleanup missed a 10 s bound (#409).
|
||||
|
||||
The ceiling is per wait and stays under the suite's 120 s per-test hang
|
||||
watchdog. A test with several such waits in a row can reach the watchdog
|
||||
first; either way a real hang fails, the watchdog with every thread's stack.
|
||||
"""
|
||||
|
||||
DISK_BOUND_SECONDS = 90
|
||||
Vendored
-16
@@ -113,19 +113,3 @@ by their own suites. The expectations start as the observed baseline. A change
|
||||
that alters an outcome edits the vector in the same commit.
|
||||
`tests/js/legacy-metadata-characterization.test.js` pins what
|
||||
the legacy viewers show on the same files.
|
||||
|
||||
# Legacy contract snapshot
|
||||
|
||||
`legacy_contract_snapshot.json` records what the legacy admin and gallery pages
|
||||
talk to while System S13 (#330) moves their tools to native System: every
|
||||
registered route as `"<METHOD> <path>"` (`routes`) and the key paths of
|
||||
`GET /prompt_manager/settings` to depth 2 (`settings_keys`; list items give the
|
||||
union of their keys as `name[].key`). `tests/test_legacy_contract_snapshot.py`
|
||||
compares a temp-database API against it. Record it again only for an intended
|
||||
change, and say why in the PR body:
|
||||
|
||||
```sh
|
||||
PM_RECORD_LEGACY_SNAPSHOT=1 python3 -m unittest tests.test_legacy_contract_snapshot
|
||||
```
|
||||
|
||||
Cutover (#334) removes the test and this fixture.
|
||||
|
||||
-200
@@ -1,200 +0,0 @@
|
||||
{
|
||||
"routes": [
|
||||
"DELETE /prompt_manager/delete/{prompt_id}",
|
||||
"DELETE /prompt_manager/image-annotations/capture/{token}",
|
||||
"DELETE /prompt_manager/images/favorite",
|
||||
"DELETE /prompt_manager/images/{image_id}",
|
||||
"DELETE /prompt_manager/pairs/{pair_id}",
|
||||
"DELETE /prompt_manager/prompts/{prompt_id}/tags",
|
||||
"DELETE /prompt_manager/restore/preview/{token}",
|
||||
"DELETE /prompt_manager/saved-views/{view_id}",
|
||||
"DELETE /prompt_manager/tags/{tag_name}",
|
||||
"GET /prompt_manager/admin",
|
||||
"GET /prompt_manager/autotag/models",
|
||||
"GET /prompt_manager/backup",
|
||||
"GET /prompt_manager/backup/status",
|
||||
"GET /prompt_manager/categories",
|
||||
"GET /prompt_manager/diagnostics",
|
||||
"GET /prompt_manager/export",
|
||||
"GET /prompt_manager/gallery",
|
||||
"GET /prompt_manager/gallery.html",
|
||||
"GET /prompt_manager/gallery/subfolders",
|
||||
"GET /prompt_manager/history/runs/{run_id}",
|
||||
"GET /prompt_manager/history/runs/{run_id}/output/current/inspection",
|
||||
"GET /prompt_manager/history/runs/{run_id}/output/current/media",
|
||||
"GET /prompt_manager/history/runs/{run_id}/output/current/thumbnail",
|
||||
"GET /prompt_manager/history/runs/{run_id}/outputs",
|
||||
"GET /prompt_manager/image-annotations/capture/{token}",
|
||||
"GET /prompt_manager/images/all",
|
||||
"GET /prompt_manager/images/inspect/{image_path}",
|
||||
"GET /prompt_manager/images/output",
|
||||
"GET /prompt_manager/images/prompt/{image_path}",
|
||||
"GET /prompt_manager/images/recent",
|
||||
"GET /prompt_manager/images/search",
|
||||
"GET /prompt_manager/images/serve/{filepath}",
|
||||
"GET /prompt_manager/images/thumbnails/plan",
|
||||
"GET /prompt_manager/images/thumbnails/status",
|
||||
"GET /prompt_manager/images/{image_id}/file",
|
||||
"GET /prompt_manager/jobs",
|
||||
"GET /prompt_manager/jobs/{job_id}",
|
||||
"GET /prompt_manager/jobs/{job_id}/events",
|
||||
"GET /prompt_manager/jobs/{job_id}/report",
|
||||
"GET /prompt_manager/js/{filepath}",
|
||||
"GET /prompt_manager/lib/{filepath}",
|
||||
"GET /prompt_manager/library",
|
||||
"GET /prompt_manager/library/export",
|
||||
"GET /prompt_manager/library/options",
|
||||
"GET /prompt_manager/library/{positive_id}",
|
||||
"GET /prompt_manager/library/{positive_id}/history",
|
||||
"GET /prompt_manager/library/{positive_id}/history/pairs/{pair_id}/newest",
|
||||
"GET /prompt_manager/library/{positive_id}/images",
|
||||
"GET /prompt_manager/logs",
|
||||
"GET /prompt_manager/logs/config",
|
||||
"GET /prompt_manager/logs/download/{filename}",
|
||||
"GET /prompt_manager/logs/files",
|
||||
"GET /prompt_manager/logs/stats",
|
||||
"GET /prompt_manager/lora/detect",
|
||||
"GET /prompt_manager/lora/status",
|
||||
"GET /prompt_manager/lora/trigger-words",
|
||||
"GET /prompt_manager/maintenance/duplicates/{job_id}",
|
||||
"GET /prompt_manager/pairs",
|
||||
"GET /prompt_manager/pairs/export",
|
||||
"GET /prompt_manager/pairs/{pair_id}",
|
||||
"GET /prompt_manager/prompts/{prompt_id}/images",
|
||||
"GET /prompt_manager/recent",
|
||||
"GET /prompt_manager/review/events",
|
||||
"GET /prompt_manager/review/items",
|
||||
"GET /prompt_manager/review/items/{item_id}",
|
||||
"GET /prompt_manager/saved-views",
|
||||
"GET /prompt_manager/scan/status",
|
||||
"GET /prompt_manager/scan_duplicates",
|
||||
"GET /prompt_manager/scan_duplicates/status",
|
||||
"GET /prompt_manager/scan_output_dir",
|
||||
"GET /prompt_manager/search",
|
||||
"GET /prompt_manager/settings",
|
||||
"GET /prompt_manager/stats",
|
||||
"GET /prompt_manager/subfolders",
|
||||
"GET /prompt_manager/system/diagnostics",
|
||||
"GET /prompt_manager/system/health",
|
||||
"GET /prompt_manager/system/library",
|
||||
"GET /prompt_manager/system/overview",
|
||||
"GET /prompt_manager/system/statistics",
|
||||
"GET /prompt_manager/tags",
|
||||
"GET /prompt_manager/tags/filter",
|
||||
"GET /prompt_manager/tags/stats",
|
||||
"GET /prompt_manager/tags/suggest",
|
||||
"GET /prompt_manager/tags/{tag_name}/prompts",
|
||||
"GET /prompt_manager/test",
|
||||
"GET /prompt_manager/vocabulary/categories",
|
||||
"GET /prompt_manager/vocabulary/tags",
|
||||
"GET /prompt_manager/vocabulary/tags/{tag_id}/impact",
|
||||
"GET /prompt_manager/web",
|
||||
"GET /prompt_manager/workspace",
|
||||
"PATCH /prompt_manager/library/{positive_id}",
|
||||
"PATCH /prompt_manager/pairs/{pair_id}",
|
||||
"PATCH /prompt_manager/saved-views/{view_id}",
|
||||
"PATCH /prompt_manager/vocabulary/tags/{tag_id}",
|
||||
"POST /prompt_manager/autotag/apply",
|
||||
"POST /prompt_manager/autotag/batch/preview",
|
||||
"POST /prompt_manager/autotag/download/{model_type}",
|
||||
"POST /prompt_manager/autotag/single",
|
||||
"POST /prompt_manager/autotag/start",
|
||||
"POST /prompt_manager/autotag/unload",
|
||||
"POST /prompt_manager/bulk/category",
|
||||
"POST /prompt_manager/bulk/delete",
|
||||
"POST /prompt_manager/bulk/tags",
|
||||
"POST /prompt_manager/cleanup",
|
||||
"POST /prompt_manager/delete_duplicate_images",
|
||||
"POST /prompt_manager/diagnostics/test-link",
|
||||
"POST /prompt_manager/gallery/rescan",
|
||||
"POST /prompt_manager/image-annotations/observe",
|
||||
"POST /prompt_manager/image-annotations/suggest",
|
||||
"POST /prompt_manager/images/clear-thumbnails",
|
||||
"POST /prompt_manager/images/generate-thumbnails",
|
||||
"POST /prompt_manager/images/generate-thumbnails/progress",
|
||||
"POST /prompt_manager/images/link",
|
||||
"POST /prompt_manager/images/reveal",
|
||||
"POST /prompt_manager/jobs/autotag",
|
||||
"POST /prompt_manager/jobs/duplicates",
|
||||
"POST /prompt_manager/jobs/image-tags",
|
||||
"POST /prompt_manager/jobs/model-download",
|
||||
"POST /prompt_manager/jobs/scan",
|
||||
"POST /prompt_manager/jobs/thumbnails",
|
||||
"POST /prompt_manager/jobs/{job_id}/cancel",
|
||||
"POST /prompt_manager/library",
|
||||
"POST /prompt_manager/library/{positive_id}/variants",
|
||||
"POST /prompt_manager/logs/config",
|
||||
"POST /prompt_manager/logs/truncate",
|
||||
"POST /prompt_manager/lora/enable",
|
||||
"POST /prompt_manager/lora/import/apply",
|
||||
"POST /prompt_manager/lora/import/preview",
|
||||
"POST /prompt_manager/lora/refresh-cache",
|
||||
"POST /prompt_manager/lora/scan",
|
||||
"POST /prompt_manager/maintenance",
|
||||
"POST /prompt_manager/maintenance/apply",
|
||||
"POST /prompt_manager/maintenance/preview",
|
||||
"POST /prompt_manager/pairs",
|
||||
"POST /prompt_manager/prompts/tags",
|
||||
"POST /prompt_manager/prompts/{prompt_id}/tags",
|
||||
"POST /prompt_manager/restore",
|
||||
"POST /prompt_manager/restore/apply",
|
||||
"POST /prompt_manager/restore/preview",
|
||||
"POST /prompt_manager/review/items/{item_id}/decision",
|
||||
"POST /prompt_manager/save",
|
||||
"POST /prompt_manager/saved-views",
|
||||
"POST /prompt_manager/scan",
|
||||
"POST /prompt_manager/scan/cancel",
|
||||
"POST /prompt_manager/scan_duplicates/stream",
|
||||
"POST /prompt_manager/settings",
|
||||
"POST /prompt_manager/settings/validate-source",
|
||||
"POST /prompt_manager/system/health",
|
||||
"POST /prompt_manager/system/library/keep",
|
||||
"POST /prompt_manager/system/library/move",
|
||||
"POST /prompt_manager/tags/merge",
|
||||
"POST /prompt_manager/vocabulary/categories/apply",
|
||||
"POST /prompt_manager/vocabulary/categories/preview",
|
||||
"POST /prompt_manager/vocabulary/tags",
|
||||
"POST /prompt_manager/vocabulary/tags/{tag_id}/delete",
|
||||
"PUT /prompt_manager/image-annotations/tags",
|
||||
"PUT /prompt_manager/images/favorite",
|
||||
"PUT /prompt_manager/prompts/{prompt_id}",
|
||||
"PUT /prompt_manager/prompts/{prompt_id}/rating",
|
||||
"PUT /prompt_manager/tags/{tag_name}"
|
||||
],
|
||||
"settings_keys": [
|
||||
"monitor",
|
||||
"monitor.observer_alive",
|
||||
"monitor.present",
|
||||
"monitor.scheduled",
|
||||
"revision",
|
||||
"settings",
|
||||
"settings.autotag_character_threshold",
|
||||
"settings.autotag_existing",
|
||||
"settings.autotag_general_threshold",
|
||||
"settings.autotag_model_type",
|
||||
"settings.autotag_suggestion_limit",
|
||||
"settings.autotag_template",
|
||||
"settings.cpu_count",
|
||||
"settings.gallery_root_path",
|
||||
"settings.gallery_root_paths",
|
||||
"settings.infinite_scroll",
|
||||
"settings.monitored_directories",
|
||||
"settings.monitoring_enabled",
|
||||
"settings.result_timeout",
|
||||
"settings.safety_backup",
|
||||
"settings.thumbnail_edge",
|
||||
"settings.thumbnail_format",
|
||||
"settings.thumbnail_quality",
|
||||
"settings.webui_display_mode",
|
||||
"settings.worker_threads",
|
||||
"sources",
|
||||
"sources.auto_detected",
|
||||
"sources.configured",
|
||||
"sources.effective",
|
||||
"success",
|
||||
"thumbnail_encoders",
|
||||
"thumbnail_encoders.jpeg",
|
||||
"thumbnail_encoders.jxl",
|
||||
"thumbnail_encoders.webp"
|
||||
]
|
||||
}
|
||||
@@ -36,48 +36,33 @@ function fixture() {
|
||||
const reply = (index, data) => requests[index].resolve({ json: async () => data });
|
||||
return { manager, requests, reply, notifications };
|
||||
}
|
||||
const SHA = 'a'.repeat(64);
|
||||
test('gallery review retains root identity and applies the tags to the image by its sha256 (#379)', async () => {
|
||||
test('gallery review retains root identity and applies tags with the returned prompt ID', async () => {
|
||||
const { manager, requests, reply } = fixture();
|
||||
const load = manager.loadNextReviewImage();
|
||||
assert.equal(requests[0].body.root_index, 1);
|
||||
assert.equal(requests[0].body.existing, 'merge');
|
||||
reply(0, { success: true, prompt_id: 42, tags: ['smile'], sha256: SHA, image_tags: [] });
|
||||
await load;
|
||||
manager.loadNextReviewImage = () => {};
|
||||
const apply = manager.applyReviewTags();
|
||||
assert.equal(requests[1].url, '/prompt_manager/autotag/apply');
|
||||
assert.deepEqual(requests[1].body, ImageHelpers.autotagApplyBody(
|
||||
{ id: 'digest', path: 'same.png', root_index: 1 }, SHA, ['smile']));
|
||||
assert.equal('prompt_id' in requests[1].body, false);
|
||||
reply(1, { success: true });
|
||||
await apply;
|
||||
});
|
||||
test('an unlinked image applies to the image like a linked one', async () => {
|
||||
const { manager, requests, reply } = fixture();
|
||||
const load = manager.loadNextReviewImage();
|
||||
reply(0, { success: true, prompt_id: null, tags: ['smile'], sha256: SHA, image_tags: [] });
|
||||
await load;
|
||||
manager.loadNextReviewImage = () => {};
|
||||
const apply = manager.applyReviewTags();
|
||||
assert.equal(requests[1].body.sha256, SHA);
|
||||
assert.deepEqual(requests[1].body.tags, ['smile']);
|
||||
reply(1, { success: true });
|
||||
await apply;
|
||||
});
|
||||
test('an answer without a captured sha256 never sends an apply', async () => {
|
||||
const { manager, requests, reply } = fixture();
|
||||
const load = manager.loadNextReviewImage();
|
||||
reply(0, { success: true, prompt_id: 42, tags: ['smile'] });
|
||||
await load;
|
||||
manager.loadNextReviewImage = () => {};
|
||||
await manager.applyReviewTags();
|
||||
assert.equal(requests.filter(r => r.url === '/prompt_manager/autotag/apply').length, 0);
|
||||
const apply = manager.applyReviewTags();
|
||||
assert.deepEqual(requests[1].body, { prompt_id: 42, tags: ['smile'] });
|
||||
reply(1, { success: true });
|
||||
await apply;
|
||||
});
|
||||
test('unlinked images do not submit an invalid apply request or advance', async () => {
|
||||
const { manager, requests, reply } = fixture();
|
||||
const load = manager.loadNextReviewImage();
|
||||
reply(0, { success: true, prompt_id: null, tags: ['smile'] });
|
||||
await load;
|
||||
const apply = manager.applyReviewTags();
|
||||
if (requests[1]) reply(1, { success: false, error: "Unexpected apply" });
|
||||
await apply;
|
||||
assert.equal(requests.length, 1);
|
||||
assert.equal(manager.autoTagState.reviewIndex, 0);
|
||||
});
|
||||
test('a failed apply keeps the current image available for retry', async () => {
|
||||
const { manager, requests, reply } = fixture();
|
||||
const load = manager.loadNextReviewImage();
|
||||
reply(0, { success: true, prompt_id: 42, tags: ['smile'], sha256: SHA, image_tags: [] });
|
||||
reply(0, { success: true, prompt_id: 42, tags: ['smile'] });
|
||||
await load;
|
||||
const apply = manager.applyReviewTags();
|
||||
reply(1, { success: false, error: 'Try again' });
|
||||
|
||||
@@ -5,8 +5,6 @@ const {
|
||||
formatImageCaption,
|
||||
workflowDocumentHtml,
|
||||
autotagSingleBody,
|
||||
autotagApplyBody,
|
||||
autotagCompleteMessage,
|
||||
fallbackImageSource,
|
||||
} = require("../../web/js/image-helpers.js");
|
||||
|
||||
@@ -123,27 +121,3 @@ test('metadata lookup preserves encoded filename characters and works behind a p
|
||||
assert.equal(metadataLookupUrl('/prompt_manager/images/serve/a.png?v=123'), '/prompt_manager/images/prompt/a.png');
|
||||
assert.equal(metadataLookupUrl('/elsewhere/a.png'), null);
|
||||
});
|
||||
|
||||
test("autotagSingleBody asks to skip auto-tagged images only when told (#379)", () => {
|
||||
assert.equal(autotagSingleBody({ id: 1 }, WD14).existing, "merge");
|
||||
assert.equal(autotagSingleBody({ id: 1 }, { ...WD14, existing: "skip" }).existing, "skip");
|
||||
assert.equal(autotagSingleBody({ id: 1 }, { ...WD14, existing: "odd" }).existing, "merge");
|
||||
});
|
||||
|
||||
test("autotagApplyBody names the image, its bytes and the tags, never a prompt (#379)", () => {
|
||||
const sha = "a".repeat(64);
|
||||
assert.deepEqual(autotagApplyBody({ id: 4, prompt_id: 7 }, sha, ["bird"]), { image_id: 4, sha256: sha, tags: ["bird"] });
|
||||
assert.deepEqual(autotagApplyBody({ id: "digest", path: "sub/a.png", root_index: 1 }, sha, []),
|
||||
{ path: "sub/a.png", root_index: 1, sha256: sha, tags: [] });
|
||||
assert.deepEqual(autotagApplyBody({ id: 2 }, undefined, null), { image_id: 2, sha256: "", tags: [] });
|
||||
const tags = ["x"];
|
||||
autotagApplyBody({ id: 1 }, sha, tags).tags.push("y");
|
||||
assert.deepEqual(tags, ["x"]);
|
||||
});
|
||||
|
||||
test("autotagCompleteMessage counts images (#379)", () => {
|
||||
assert.equal(autotagCompleteMessage(2), "Auto tagging complete! Tagged 2 images.");
|
||||
assert.equal(autotagCompleteMessage(1), "Auto tagging complete! Tagged 1 image.");
|
||||
assert.equal(autotagCompleteMessage(0), "Auto tagging complete! Tagged 0 images.");
|
||||
assert.equal(autotagCompleteMessage(undefined), "Auto tagging complete! Tagged 0 images.");
|
||||
});
|
||||
|
||||
@@ -97,30 +97,6 @@ test("connect reports a non-2xx response through onerror and never onmessage", a
|
||||
assert.equal(source.readyState, source.CLOSED);
|
||||
});
|
||||
|
||||
test("a non-2xx JSON answer's error sentence reaches onerror as serverMessage", async () => {
|
||||
let error = null;
|
||||
const fetchImpl = async () => new Response(
|
||||
JSON.stringify({ success: false, error: "Auto-tagging is already running" }),
|
||||
{ status: 409, headers: { "Content-Type": "application/json" } },
|
||||
);
|
||||
const source = connect("/x", { fetchImpl });
|
||||
source.onerror = (err) => { error = err; };
|
||||
await until(() => error !== null);
|
||||
assert.match(error.message, /409/);
|
||||
assert.equal(error.status, 409);
|
||||
assert.equal(error.serverMessage, "Auto-tagging is already running");
|
||||
});
|
||||
|
||||
test("a non-2xx answer without a JSON error has no serverMessage", async () => {
|
||||
let error = null;
|
||||
const fetchImpl = async () => new Response("nope", { status: 500 });
|
||||
const source = connect("/x", { fetchImpl });
|
||||
source.onerror = (err) => { error = err; };
|
||||
await until(() => error !== null);
|
||||
assert.equal(error.status, 500);
|
||||
assert.equal(error.serverMessage, undefined);
|
||||
});
|
||||
|
||||
test("connect reports a network failure through onerror", async () => {
|
||||
let error = null;
|
||||
const source = connect("/x", { fetchImpl: async () => { throw new TypeError("Failed to fetch"); } });
|
||||
|
||||
@@ -234,23 +234,6 @@ test('the count line reads N of M images and K positive prompts only when known'
|
||||
assert.equal(countLine({loaded:1200,total:3400,positiveCount:0}),'1,200 of 3,400 images · 0 positive prompts');
|
||||
});
|
||||
|
||||
test('the count line reads as loading, never as an empty result, until a first page answers (#391)', async () => {
|
||||
const {countLine,createImageStream}=await source;
|
||||
assert.equal(countLine({loaded:0,total:0,positiveCount:null,loading:true,error:null}),'Loading images…');
|
||||
// A failed first page and a settled empty one say what the server said; a later page keeps the loaded count.
|
||||
assert.equal(countLine({loaded:0,total:0,positiveCount:null,loading:false,error:new Error('x')}),'0 of 0 images');
|
||||
assert.equal(countLine({loaded:0,total:0,positiveCount:0,loading:false,error:null}),'0 of 0 images · 0 positive prompts');
|
||||
assert.equal(countLine({loaded:40,total:102,positiveCount:2,loading:true,error:null}),'40 of 102 images · 2 positive prompts');
|
||||
// While a reset's first page is in flight the line reads as loading (the cleared state before it is replaced in
|
||||
// the same task, so it is never painted).
|
||||
let answer;const lines=[];
|
||||
const stream=createImageStream({api:{get:()=>new Promise(resolve=>{answer=resolve;})},onChange:state=>lines.push(countLine(state))});
|
||||
const loading=stream.reset({tags:['cinematic']});
|
||||
assert.equal(lines.at(-1),'Loading images…');
|
||||
answer({images:[{path:'a.png',root_index:0}],total:1,has_more:false,next_cursor:null,positive_count:1});await loading;
|
||||
assert.equal(lines.at(-1),'1 of 1 image · 1 positive prompt');
|
||||
});
|
||||
|
||||
test('the caption date names the newest result day under Newest first only', async () => {
|
||||
const {captionDate}=await source;
|
||||
const now=new Date(2026,9,17,15,0);
|
||||
|
||||
@@ -175,28 +175,6 @@ test('Clear all logs states both counts and its toast says what was done', async
|
||||
assert.equal(clearToast(undefined), 'Logs cleared · 0 rotated copies deleted.');
|
||||
});
|
||||
|
||||
test('Clear all logs sends back exactly the files its confirmation showed (#389)', async () => {
|
||||
const { clearBody, clearFailureText, COPY } = await diagnostics;
|
||||
const files = [
|
||||
{ filename: 'prompt_manager.log', kind: 'active', identity: '10:1761000000123456789:9007199254740993' },
|
||||
{ filename: 'prompt_manager.log.1', kind: 'rotated', identity: '20:1761000000123456789:42' },
|
||||
{ filename: 'prompt_manager.log.4', kind: 'link', identity: '5:1:7' },
|
||||
{ filename: 'prompt_manager.log.bak', kind: 'other', identity: '1:1:8' },
|
||||
{ filename: 'prompt_manager.log.2', kind: 'rotated' },
|
||||
];
|
||||
// Identities stay strings: mtime_ns and large inodes are not exact JS numbers.
|
||||
assert.deepEqual(clearBody(files), { expected: [
|
||||
{ name: 'prompt_manager.log', identity: '10:1761000000123456789:9007199254740993' },
|
||||
{ name: 'prompt_manager.log.1', identity: '20:1761000000123456789:42' },
|
||||
] });
|
||||
assert.deepEqual(clearBody([]), { expected: [] });
|
||||
assert.deepEqual(clearBody(null), {});
|
||||
const refused = Object.assign(new Error('The log files changed since you confirmed; nothing was cleared.'), { status: 409 });
|
||||
assert.equal(clearFailureText(refused), refused.message);
|
||||
assert.equal(clearFailureText(Object.assign(new Error('boom'), { status: 500 })), COPY.clearFailed);
|
||||
assert.equal(clearFailureText(undefined), COPY.clearFailed);
|
||||
});
|
||||
|
||||
test('the test-link dialog names the prompt and how Missing-file links removes the record', async () => {
|
||||
const { testLinkSentence, promptName } = await diagnostics;
|
||||
assert.equal(testLinkSentence({ id: 812, text: 'Misty harbor' }), 'Writes a placeholder image record '
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
"""A temp SQLite database in a folder no other test process shares (#346).
|
||||
|
||||
``PromptManagerAPI`` sweeps its database's folder at startup for the stages,
|
||||
uploads and safety-backup copies an interrupted restore or backup left. A
|
||||
fixture database directly in the system temp folder made every other test
|
||||
process that started an API (another worktree's suite, a pre-push hook) delete
|
||||
this process's in-flight restore files, and left safety backups behind there.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
|
||||
|
||||
def private_temp_db(case):
|
||||
"""Return a closed ``NamedTemporaryFile`` handle for an empty ``.db`` file.
|
||||
|
||||
The file lives in a new folder that ``case`` removes at cleanup, after its
|
||||
own teardown has closed and unlinked the database (cleanups run after
|
||||
``tearDown``), together with anything a restore or backup left next to it.
|
||||
"""
|
||||
folder = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
|
||||
case.addCleanup(folder.cleanup)
|
||||
handle = tempfile.NamedTemporaryFile(delete=False, suffix=".db", dir=folder.name)
|
||||
handle.close()
|
||||
return handle
|
||||
@@ -430,9 +430,7 @@ class ReplaceImageTagsTests(AnnotationTestCase):
|
||||
self.assertTrue(
|
||||
self.db.refresh_image_fingerprint(loc, DIGEST_A, SIZE, MTIME + 1)
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.image_tag_locators(self.db.image_tag_ids(["img"], "")), {loc}
|
||||
)
|
||||
list(self.db.iter_image_tag_records())
|
||||
self.assertEqual(self.db.get_prompt_by_id(prompt)["tags"], ["prompt-tag"])
|
||||
|
||||
|
||||
@@ -553,26 +551,14 @@ class CurrentImageTagsTests(AnnotationTestCase):
|
||||
self.assertEqual(current, {locators[0]: ["z", "y"], locators[502]: ["x"]})
|
||||
lookups = [s for s in statements if "media_annotation_files" in s]
|
||||
self.assertEqual(len(lookups), 2)
|
||||
# The tag search finds every record's file by tag id, and the empty
|
||||
# set's file never.
|
||||
self.assertEqual(
|
||||
self.db.image_tag_locators(self.db.image_tag_ids(["z", "x"], "")),
|
||||
{locators[0], locators[502]},
|
||||
list(self.db.iter_image_tag_records()),
|
||||
[
|
||||
(locators[0], SIZE, MTIME, ["z", "y"]),
|
||||
(locators[250], SIZE, MTIME + 1, []),
|
||||
(locators[502], SIZE, MTIME + 2, ["x"]),
|
||||
],
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.current_image_tags({locators[250]: (SIZE, MTIME + 1)}),
|
||||
{locators[250]: []},
|
||||
)
|
||||
|
||||
def test_tag_locators_cover_more_ids_than_one_chunk(self):
|
||||
# A broad text search can match more than 500 names: every id's file
|
||||
# comes back across the chunks.
|
||||
locators = [self.locator(f"t{i:04}.png") for i in range(503)]
|
||||
for i, loc in enumerate(locators):
|
||||
self.replace(loc, tags=[f"name_{i:04}"])
|
||||
ids = self.db.image_tag_ids([], "name_")
|
||||
self.assertEqual(len(ids), 503)
|
||||
self.assertEqual(self.db.image_tag_locators(ids), set(locators))
|
||||
|
||||
|
||||
class AutotagMarkTests(AnnotationTestCase):
|
||||
|
||||
+3
-2
@@ -8,6 +8,7 @@ without requiring a full ComfyUI server.
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
@@ -17,7 +18,6 @@ from aiohttp.test_utils import AioHTTPTestCase
|
||||
|
||||
from database.operations import PromptDatabase
|
||||
from py.api import PromptManagerAPI
|
||||
from tests.private_temp import private_temp_db
|
||||
from utils.hashing import generate_prompt_hash
|
||||
|
||||
|
||||
@@ -25,7 +25,8 @@ class APITestCase(AioHTTPTestCase):
|
||||
"""Base class that stands up a test aiohttp app with PromptManager routes."""
|
||||
|
||||
async def get_application(self):
|
||||
self._temp_db = private_temp_db(self)
|
||||
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
|
||||
self._temp_db.close()
|
||||
|
||||
app = web.Application()
|
||||
routes = web.RouteTableDef()
|
||||
|
||||
+21
-13
@@ -846,14 +846,6 @@ class TestSettingsReadMonitorThroughModule(AdminAPITestCase):
|
||||
self.assertEqual(data["settings"]["gallery_root_paths"], ["output"])
|
||||
|
||||
|
||||
def _wired_prompt(text):
|
||||
"""A sampler-less graph whose one encoder feeds a guider (#252)."""
|
||||
return {
|
||||
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": text}},
|
||||
"2": {"class_type": "BasicGuider", "inputs": {"conditioning": ["1", 0]}},
|
||||
}
|
||||
|
||||
|
||||
def _raise(*args, **kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
@@ -2194,7 +2186,11 @@ class TestOutputScanInternals(AdminAPITestCase):
|
||||
|
||||
def test_ingest_outcomes(self):
|
||||
f = Path("gen.png")
|
||||
meta = {"prompt": json.dumps(_wired_prompt("a cat"))}
|
||||
meta = {
|
||||
"prompt": json.dumps(
|
||||
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a cat"}}}
|
||||
)
|
||||
}
|
||||
self.assertIsNone(self.api._ingest_scanned_file(f, {}))
|
||||
self.assertIsNone(self.api._ingest_scanned_file(f, {"prompt": "{}"}))
|
||||
self.assertEqual(
|
||||
@@ -2206,19 +2202,31 @@ class TestOutputScanInternals(AdminAPITestCase):
|
||||
|
||||
def test_ingest_link_failures_are_tolerated(self):
|
||||
f = Path("gen.png")
|
||||
meta = {"prompt": json.dumps(_wired_prompt("a dog"))}
|
||||
meta = {
|
||||
"prompt": json.dumps(
|
||||
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a dog"}}}
|
||||
)
|
||||
}
|
||||
self.api.db.link_image_to_prompt = _raise
|
||||
self.assertEqual(self.api._ingest_scanned_file(f, meta), "added")
|
||||
self.assertEqual(self.api._ingest_scanned_file(f, meta), "found")
|
||||
|
||||
def test_ingest_save_failure_counts_as_found(self):
|
||||
f = Path("gen.png")
|
||||
meta = {"prompt": json.dumps(_wired_prompt("a bird"))}
|
||||
meta = {
|
||||
"prompt": json.dumps(
|
||||
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a bird"}}}
|
||||
)
|
||||
}
|
||||
self.api.db.save_prompt = lambda *a, **k: None
|
||||
self.assertEqual(self.api._ingest_scanned_file(f, meta), "found")
|
||||
|
||||
def test_batch_counts_and_per_file_errors(self):
|
||||
meta = {"prompt": json.dumps(_wired_prompt("a fish"))}
|
||||
meta = {
|
||||
"prompt": json.dumps(
|
||||
{"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a fish"}}}
|
||||
)
|
||||
}
|
||||
self.api.db.get_prompt_by_hash = _raise
|
||||
counts = self.api._ingest_scan_batch_sync(
|
||||
[(Path("a.png"), meta), (Path("b.png"), {})]
|
||||
@@ -2907,7 +2915,7 @@ class TestScanReadsVideos(AdminAPITestCase):
|
||||
shutil.which("ffmpeg") and shutil.which("ffprobe"), "ffmpeg not installed"
|
||||
)
|
||||
async def test_scan_adds_prompts_found_in_videos(self):
|
||||
graph = _wired_prompt("a cat")
|
||||
graph = {"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "a cat"}}}
|
||||
_write_mp4(
|
||||
self.output_dir / "clip.mp4", json.dumps({"prompt": json.dumps(graph)})
|
||||
)
|
||||
|
||||
@@ -363,20 +363,23 @@ class TestModelGate(SuggestTestCase):
|
||||
prompt_id = self._save_prompt("batch prompt")
|
||||
self.api.db.link_image_to_prompt(prompt_id, str(self.output_dir / "sub/a.png"))
|
||||
started, proceed = threading.Event(), threading.Event()
|
||||
# The legacy batch is the autotag job (#379): it suggests per image.
|
||||
self.tagger.block = (started, proceed)
|
||||
|
||||
def slow_generate(image_path, **kwargs):
|
||||
started.set()
|
||||
proceed.wait(WAIT)
|
||||
return ["batch"]
|
||||
|
||||
self.tagger.generate_tags = slow_generate
|
||||
batch = asyncio.ensure_future(
|
||||
self.client.post("/prompt_manager/autotag/start?model_type=wd14-vit")
|
||||
)
|
||||
try:
|
||||
await asyncio.get_running_loop().run_in_executor(None, started.wait, WAIT)
|
||||
calls = len(self.tagger.calls)
|
||||
status, body = await self.suggest(self.suggest_body(observed, "gguf"))
|
||||
self.assertEqual((status, body["code"]), (409, "model-busy"))
|
||||
self.assertEqual(len(self.tagger.calls), calls)
|
||||
self.assertEqual(self.tagger.calls, [])
|
||||
self.assertEqual(self.tagger.loaded, "wd14-vit") # not swapped
|
||||
finally:
|
||||
self.tagger.block = None
|
||||
proceed.set()
|
||||
response = await batch
|
||||
await response.read()
|
||||
|
||||
+413
-71
@@ -5,7 +5,6 @@ no network request is made.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
@@ -35,8 +34,6 @@ from aiohttp.test_utils import AioHTTPTestCase
|
||||
|
||||
from database.operations import PromptDatabase
|
||||
from py.api import PromptManagerAPI
|
||||
from tests.private_temp import private_temp_db
|
||||
from py.api.autotag_routes import MODEL_BUSY_ERROR
|
||||
from py.api.jobs import JobBusy
|
||||
from utils.hashing import generate_prompt_hash
|
||||
|
||||
@@ -73,7 +70,6 @@ class FakeAutotagService:
|
||||
self.status_calls = 0
|
||||
self.status_threads = []
|
||||
self.status_paths = {}
|
||||
self.suggest_calls = 0
|
||||
|
||||
def get_models_status(self):
|
||||
self.status_calls += 1
|
||||
@@ -119,32 +115,14 @@ class FakeAutotagService:
|
||||
raise self.generate_error
|
||||
return list(self.tags)
|
||||
|
||||
def generate_tag_suggestions(
|
||||
self,
|
||||
image_path,
|
||||
*,
|
||||
limit,
|
||||
prompt=None,
|
||||
general_threshold=None,
|
||||
character_threshold=None,
|
||||
):
|
||||
"""The suggestion entry point (#379): the fixed tags, at most ``limit``."""
|
||||
self.suggest_calls += 1
|
||||
tags = self.generate_tags(
|
||||
image_path,
|
||||
prompt=prompt,
|
||||
general_threshold=general_threshold,
|
||||
character_threshold=character_threshold,
|
||||
)
|
||||
return [(name, None) for name in tags[:limit]]
|
||||
|
||||
|
||||
class AutotagAPITestCase(AioHTTPTestCase):
|
||||
"""Stands up the PromptManager routes with a temp DB, temp output dir and a
|
||||
fake engine."""
|
||||
|
||||
async def get_application(self):
|
||||
self._temp_db = private_temp_db(self)
|
||||
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
|
||||
self._temp_db.close()
|
||||
self.tmp_root = Path(tempfile.mkdtemp())
|
||||
self.output_dir = self.tmp_root / "output"
|
||||
self.output_dir.mkdir()
|
||||
@@ -197,17 +175,8 @@ class AutotagAPITestCase(AioHTTPTestCase):
|
||||
)
|
||||
|
||||
def _make_image(self, directory, name="img.png"):
|
||||
"""A small, valid image (the legacy review captures and probes it)."""
|
||||
from PIL import Image
|
||||
|
||||
path = directory / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
shade = sum(name.encode()) % 256
|
||||
fmt = {".jpg": "JPEG", ".jpeg": "JPEG", ".webp": "WEBP"}.get(
|
||||
path.suffix.lower(), "PNG"
|
||||
)
|
||||
with Image.new("RGB", (4, 4), (shade, 64, 128)) as image:
|
||||
image.save(path, fmt)
|
||||
path.write_bytes(b"\x89PNG\r\n\x1a\n")
|
||||
return path
|
||||
|
||||
def _link_image(self, prompt_id, path):
|
||||
@@ -247,9 +216,10 @@ class TestAutotagSingleContainment(AutotagAPITestCase):
|
||||
self.assertEqual(data["tags"], ["1girl", "smile"])
|
||||
self.assertEqual(data["prompt_id"], prompt_id)
|
||||
self.assertEqual(len(self.service.generate_calls), 1)
|
||||
# #379: the model reads a private capture of the image, never the file.
|
||||
self.assertNotIn(str(self.output_dir), self.service.generate_calls[0][0])
|
||||
self.assertEqual(data["sha256"], hashlib.sha256(image.read_bytes()).hexdigest())
|
||||
self.assertEqual(
|
||||
os.path.realpath(self.service.generate_calls[0][0]),
|
||||
os.path.realpath(str(image)),
|
||||
)
|
||||
|
||||
async def test_path_inside_output_dir_is_tagged(self):
|
||||
image = self._make_image(self.output_dir)
|
||||
@@ -479,6 +449,238 @@ class TestAutotagSideEffectRoutesArePostOnly(AutotagAPITestCase):
|
||||
self.assertTrue((await resp.json())["success"])
|
||||
|
||||
|
||||
class TestAutotagBatch(AutotagAPITestCase):
|
||||
"""Batch tagging over the DB image list with SSE progress reporting."""
|
||||
|
||||
def _seed(self):
|
||||
"""Three prompts: A (one image), B (two images), C (missing file)."""
|
||||
a = self._save_prompt("prompt a")
|
||||
b = self._save_prompt("prompt b")
|
||||
c = self._save_prompt("prompt c")
|
||||
self._link_image(a, self._make_image(self.output_dir, "a1.png"))
|
||||
self._link_image(b, self._make_image(self.output_dir, "b1.png"))
|
||||
self._link_image(b, self._make_image(self.output_dir, "b2.png"))
|
||||
self._link_image(c, self.output_dir / "missing.png")
|
||||
return a, b, c
|
||||
|
||||
async def _start(self, query="model_type=gguf", body=None):
|
||||
if body is None:
|
||||
resp = await self.client.request(
|
||||
"POST", f"/prompt_manager/autotag/start?{query}"
|
||||
)
|
||||
else:
|
||||
resp = await self._post_json(f"/prompt_manager/autotag/start?{query}", body)
|
||||
return resp, self._sse_events(await resp.text())
|
||||
|
||||
async def test_batch_tags_each_prompt_once_and_reports_counts(self):
|
||||
a, b, c = self._seed()
|
||||
|
||||
resp, events = await self._start()
|
||||
|
||||
self.assertEqual(resp.status, 200)
|
||||
done = events[-1]
|
||||
self.assertEqual(done["type"], "complete")
|
||||
self.assertEqual(done["processed"], 2)
|
||||
self.assertEqual(done["tagged"], 2)
|
||||
self.assertEqual(done["skipped"], 2)
|
||||
self.assertEqual(done["errors"], 0)
|
||||
self.assertEqual(len(self.service.generate_calls), 2)
|
||||
self.assertEqual(self.api.db.get_prompt_by_id(a)["tags"], ["1girl", "smile"])
|
||||
self.assertEqual(self.api.db.get_prompt_by_id(b)["tags"], ["1girl", "smile"])
|
||||
self.assertEqual(self.api.db.get_prompt_by_id(c)["tags"], [])
|
||||
progress = [e for e in events if e["type"] == "progress"]
|
||||
self.assertTrue(all(0 <= e["progress"] <= 100 for e in progress))
|
||||
self.assertTrue(any("processed" in e for e in progress))
|
||||
|
||||
async def test_rows_outside_output_dirs_are_skipped_and_counted(self):
|
||||
inside = self._save_prompt("inside")
|
||||
outside = self._save_prompt("outside")
|
||||
self._link_image(inside, self._make_image(self.output_dir, "in.png"))
|
||||
secret = self._make_image(self.secret_dir, "password.png")
|
||||
self._link_image(outside, secret)
|
||||
|
||||
_, events = await self._start()
|
||||
|
||||
done = events[-1]
|
||||
self.assertEqual(done["type"], "complete")
|
||||
self.assertEqual(done["processed"], 1)
|
||||
self.assertEqual(done["tagged"], 1)
|
||||
self.assertEqual(done["skipped"], 1)
|
||||
self.assertEqual(done["skipped_outside_roots"], 1)
|
||||
self.assertEqual(len(self.service.generate_calls), 1)
|
||||
self.assertEqual(
|
||||
os.path.realpath(self.service.generate_calls[0][0]),
|
||||
os.path.realpath(str(self.output_dir / "in.png")),
|
||||
)
|
||||
self.assertEqual(self.api.db.get_prompt_by_id(outside)["tags"], [])
|
||||
progress = [e for e in events if e["type"] == "progress" and "skipped" in e]
|
||||
self.assertTrue(progress)
|
||||
self.assertTrue(all("skipped_outside_roots" in e for e in progress))
|
||||
body = json.dumps(events)
|
||||
self.assertNotIn(str(self.secret_dir), body)
|
||||
|
||||
async def test_second_start_while_running_is_409(self):
|
||||
self._seed()
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def slow_generate(image_path, **kwargs):
|
||||
started.set()
|
||||
release.wait(5)
|
||||
return ["slow"]
|
||||
|
||||
self.service.generate_tags = slow_generate
|
||||
first = asyncio.ensure_future(self._start())
|
||||
try:
|
||||
await asyncio.get_event_loop().run_in_executor(None, started.wait, 5)
|
||||
self.assertTrue(started.is_set())
|
||||
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/autotag/start?model_type=gguf"
|
||||
)
|
||||
|
||||
self.assertEqual(resp.status, 409)
|
||||
self.assertEqual(
|
||||
await resp.json(),
|
||||
{"success": False, "error": "Auto-tagging is already running"},
|
||||
)
|
||||
finally:
|
||||
release.set()
|
||||
resp1, events = await first
|
||||
self.assertEqual(resp1.status, 200)
|
||||
self.assertEqual(events[-1]["type"], "complete")
|
||||
|
||||
async def test_lock_is_released_after_a_run(self):
|
||||
self._seed()
|
||||
|
||||
resp1, _ = await self._start()
|
||||
resp2, events = await self._start()
|
||||
|
||||
self.assertEqual(resp1.status, 200)
|
||||
self.assertEqual(resp2.status, 200)
|
||||
self.assertEqual(events[-1]["type"], "complete")
|
||||
|
||||
async def test_lock_is_released_when_the_run_fails(self):
|
||||
self._seed()
|
||||
self.service.load_error = RuntimeError("no model")
|
||||
|
||||
_, events = await self._start()
|
||||
self.assertEqual(events[-1]["type"], "error")
|
||||
|
||||
self.service.load_error = None
|
||||
resp, events = await self._start()
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(events[-1]["type"], "complete")
|
||||
|
||||
async def test_already_tagged_prompts_are_skipped_unless_asked(self):
|
||||
p = self._save_prompt("tagged", tags=["cat"])
|
||||
self._link_image(p, self._make_image(self.output_dir, "t.png"))
|
||||
|
||||
_, events = await self._start("model_type=gguf&skip_tagged=true")
|
||||
self.assertEqual(events[-1]["skipped"], 1)
|
||||
self.assertEqual(self.service.generate_calls, [])
|
||||
|
||||
_, events = await self._start("model_type=gguf&skip_tagged=false")
|
||||
self.assertEqual(events[-1]["tagged"], 1)
|
||||
self.assertEqual(
|
||||
self.api.db.get_prompt_by_id(p)["tags"], ["cat", "1girl", "smile"]
|
||||
)
|
||||
|
||||
async def test_prompt_already_holding_the_tags_counts_as_skipped(self):
|
||||
p = self._save_prompt("same", tags=["1girl", "smile"])
|
||||
self._link_image(p, self._make_image(self.output_dir, "s.png"))
|
||||
|
||||
_, events = await self._start("model_type=gguf&skip_tagged=false")
|
||||
|
||||
self.assertEqual(events[-1]["skipped"], 1)
|
||||
self.assertEqual(events[-1]["tagged"], 0)
|
||||
|
||||
async def test_generation_errors_are_counted_and_do_not_abort(self):
|
||||
self._seed()
|
||||
self.service.generate_error = RuntimeError("boom")
|
||||
|
||||
_, events = await self._start()
|
||||
|
||||
self.assertEqual(events[-1]["type"], "complete")
|
||||
# a1, b1 and b2 each fail: a failed prompt is retried on its next image
|
||||
self.assertEqual(events[-1]["errors"], 3)
|
||||
self.assertEqual(events[-1]["tagged"], 0)
|
||||
|
||||
async def test_json_body_parameters_are_accepted(self):
|
||||
self._seed()
|
||||
|
||||
_, events = await self._start(
|
||||
"", {"model_type": "wd14-vit", "general_threshold": 0.5, "prompt": "p"}
|
||||
)
|
||||
|
||||
self.assertEqual(events[-1]["type"], "complete")
|
||||
self.assertEqual(self.service.load_calls, [("wd14-vit", True)])
|
||||
self.assertEqual(self.service.generate_calls[0][2], 0.5)
|
||||
self.assertEqual(self.service.custom_prompt, "p")
|
||||
|
||||
async def test_keep_in_memory_false_unloads_after_batch(self):
|
||||
self._seed()
|
||||
|
||||
_, events = await self._start("model_type=gguf&keep_in_memory=false")
|
||||
|
||||
self.assertEqual(events[-1]["model_loaded"], False)
|
||||
self.assertEqual(self.service.unload_calls, 1)
|
||||
|
||||
async def test_model_not_downloaded_is_an_error_event(self):
|
||||
self.service.downloaded = False
|
||||
|
||||
_, events = await self._start()
|
||||
|
||||
self.assertEqual(events[-1]["type"], "error")
|
||||
self.assertEqual(self.service.load_calls, [])
|
||||
|
||||
async def test_load_failure_is_an_error_event_without_details(self):
|
||||
self.service.load_error = RuntimeError("/srv/models/secret.gguf is corrupt")
|
||||
|
||||
_, events = await self._start()
|
||||
|
||||
self.assertEqual(events[-1]["type"], "error")
|
||||
self.assertNotIn("/srv/models", events[-1]["message"])
|
||||
|
||||
async def test_non_numeric_threshold_is_400(self):
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/autotag/start?general_threshold=abc"
|
||||
)
|
||||
self.assertEqual(resp.status, 400)
|
||||
|
||||
async def test_invalid_json_body_is_400(self):
|
||||
resp = await self.client.request(
|
||||
"POST",
|
||||
"/prompt_manager/autotag/start",
|
||||
data="{not json",
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
self.assertEqual(resp.status, 400)
|
||||
|
||||
async def test_client_disconnect_stops_the_batch_and_unloads(self):
|
||||
self._seed()
|
||||
real_write = web.StreamResponse.write
|
||||
state = {"writes": 0}
|
||||
|
||||
async def flaky_write(response, data):
|
||||
state["writes"] += 1
|
||||
if b"Processing " in data:
|
||||
raise ConnectionResetError("Cannot write to closing transport")
|
||||
return await real_write(response, data)
|
||||
|
||||
with patch.object(web.StreamResponse, "write", flaky_write):
|
||||
try:
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/autotag/start?keep_in_memory=false"
|
||||
)
|
||||
await resp.read()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
self.assertEqual(len(self.service.generate_calls), 1)
|
||||
self.assertEqual(self.service.unload_calls, 1)
|
||||
|
||||
|
||||
class TestAutotagModelsAndDownloadErrors(AutotagAPITestCase):
|
||||
|
||||
async def test_models_error_is_500(self):
|
||||
@@ -632,15 +834,7 @@ class TestAutotagSingleEdgeCases(AutotagAPITestCase):
|
||||
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(self.service.load_calls, [("wd14-vit", True)])
|
||||
# #379: the request's prompt is for this call only.
|
||||
self.assertEqual(self.service.custom_prompt, "")
|
||||
resp = await self._post_json(
|
||||
"/prompt_manager/autotag/single",
|
||||
{"path": str(image), "model_type": "gguf", "prompt": "terse"},
|
||||
)
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertEqual(self.service.generate_calls[-1][1], "terse")
|
||||
self.assertEqual(self.service.custom_prompt, "")
|
||||
self.assertEqual(self.service.custom_prompt, "terse")
|
||||
|
||||
async def test_image_id_without_stored_path_is_404(self):
|
||||
prompt_id = self._save_prompt()
|
||||
@@ -656,6 +850,64 @@ class TestAutotagSingleEdgeCases(AutotagAPITestCase):
|
||||
self.assertEqual(resp.status, 404)
|
||||
|
||||
|
||||
class TestApplyAutotag(AutotagAPITestCase):
|
||||
|
||||
async def test_missing_prompt_id_is_400(self):
|
||||
resp = await self._post_json("/prompt_manager/autotag/apply", {"tags": ["a"]})
|
||||
self.assertEqual(resp.status, 400)
|
||||
|
||||
async def test_no_tags_is_a_noop(self):
|
||||
resp = await self._post_json("/prompt_manager/autotag/apply", {"prompt_id": 1})
|
||||
data = await resp.json()
|
||||
self.assertTrue(data["success"])
|
||||
self.assertEqual(data["message"], "No tags to apply")
|
||||
|
||||
async def test_unknown_prompt_is_404(self):
|
||||
resp = await self._post_json(
|
||||
"/prompt_manager/autotag/apply", {"prompt_id": 999, "tags": ["a"]}
|
||||
)
|
||||
self.assertEqual(resp.status, 404)
|
||||
|
||||
async def test_tags_are_merged_without_duplicates(self):
|
||||
prompt_id = self._save_prompt(tags=["cat"])
|
||||
|
||||
resp = await self._post_json(
|
||||
"/prompt_manager/autotag/apply",
|
||||
{"prompt_id": prompt_id, "tags": ["cat", "dog"]},
|
||||
)
|
||||
|
||||
data = await resp.json()
|
||||
self.assertEqual(data["added_tags"], ["dog"])
|
||||
self.assertEqual(data["total_tags"], 2)
|
||||
self.assertEqual(
|
||||
self.api.db.get_prompt_by_id(prompt_id)["tags"], ["cat", "dog"]
|
||||
)
|
||||
|
||||
async def test_string_tags_from_db_are_split(self):
|
||||
prompt_id = self._save_prompt()
|
||||
with patch.object(
|
||||
self.api.db,
|
||||
"get_prompt_by_id",
|
||||
return_value={"id": prompt_id, "tags": "cat, dog"},
|
||||
):
|
||||
resp = await self._post_json(
|
||||
"/prompt_manager/autotag/apply",
|
||||
{"prompt_id": prompt_id, "tags": ["dog", "owl"]},
|
||||
)
|
||||
data = await resp.json()
|
||||
self.assertEqual(data["added_tags"], ["owl"])
|
||||
self.assertEqual(data["total_tags"], 3)
|
||||
|
||||
async def test_db_error_is_500(self):
|
||||
with patch.object(
|
||||
self.api.db, "get_prompt_by_id", side_effect=RuntimeError("db")
|
||||
):
|
||||
resp = await self._post_json(
|
||||
"/prompt_manager/autotag/apply", {"prompt_id": 1, "tags": ["a"]}
|
||||
)
|
||||
self.assertEqual(resp.status, 500)
|
||||
|
||||
|
||||
class TestAutotagErrorBodiesHideServerPaths(AutotagAPITestCase):
|
||||
"""Every autotag handler reports exceptions through the shared public helper."""
|
||||
|
||||
@@ -680,12 +932,9 @@ class TestAutotagErrorBodiesHideServerPaths(AutotagAPITestCase):
|
||||
await self._assert_hidden(resp)
|
||||
|
||||
async def test_apply_error(self):
|
||||
image = self._make_image(self.output_dir)
|
||||
digest = hashlib.sha256(image.read_bytes()).hexdigest()
|
||||
with patch.object(self.api.db, "add_image_tags", side_effect=self._locked()):
|
||||
with patch.object(self.api.db, "get_prompt_by_id", side_effect=self._locked()):
|
||||
resp = await self._post_json(
|
||||
"/prompt_manager/autotag/apply",
|
||||
{"path": str(image), "sha256": digest, "tags": ["a"]},
|
||||
"/prompt_manager/autotag/apply", {"prompt_id": 1, "tags": ["a"]}
|
||||
)
|
||||
await self._assert_hidden(resp)
|
||||
|
||||
@@ -759,13 +1008,10 @@ class TestLegacyRoutesShareTheModelGate(AutotagAPITestCase):
|
||||
"POST", "/prompt_manager/autotag/download/gguf"
|
||||
)
|
||||
)
|
||||
# The legacy batch is read as a stream: one error frame (#379).
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/autotag/start?model_type=gguf"
|
||||
)
|
||||
self.assertEqual(
|
||||
self._sse_events(await resp.text()),
|
||||
[{"type": "error", "status": "error", "message": MODEL_BUSY_ERROR}],
|
||||
await self._busy(
|
||||
await self.client.request(
|
||||
"POST", "/prompt_manager/autotag/start?model_type=gguf"
|
||||
)
|
||||
)
|
||||
await self._busy(
|
||||
await self.client.request("POST", "/prompt_manager/autotag/unload")
|
||||
@@ -780,7 +1026,7 @@ class TestLegacyRoutesShareTheModelGate(AutotagAPITestCase):
|
||||
self.assertEqual(self.service.unload_calls, 0)
|
||||
self.assertEqual(self.service.get_loaded_model_type(), "wd14-vit")
|
||||
|
||||
# A refused start left no job behind: the next start runs.
|
||||
# A refused start released the batch lock: the next start runs.
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/autotag/start?model_type=gguf"
|
||||
)
|
||||
@@ -992,7 +1238,7 @@ if __name__ == "__main__":
|
||||
|
||||
|
||||
class TestGalleryReviewContract(AutotagAPITestCase):
|
||||
async def test_root_relative_gallery_targets_generate_and_apply_to_their_image(
|
||||
async def test_root_relative_gallery_targets_generate_and_apply_to_their_prompt(
|
||||
self,
|
||||
):
|
||||
second = self.tmp_root / "second-output"
|
||||
@@ -1009,22 +1255,16 @@ class TestGalleryReviewContract(AutotagAPITestCase):
|
||||
self.assertEqual(response.status, 200)
|
||||
data = await response.json()
|
||||
self.assertEqual(data["prompt_id"], pid)
|
||||
self.assertEqual(
|
||||
self.service.generate_calls[-1][0], str((root / "same.png").resolve())
|
||||
)
|
||||
response = await self._post_json(
|
||||
"/prompt_manager/autotag/apply",
|
||||
{
|
||||
"path": "same.png",
|
||||
"root_index": index,
|
||||
"sha256": data["sha256"],
|
||||
"tags": data["tags"],
|
||||
},
|
||||
{"prompt_id": data["prompt_id"], "tags": data["tags"]},
|
||||
)
|
||||
self.assertEqual(response.status, 200)
|
||||
# The image's own tags; the prompt is never written (#379).
|
||||
self.assertEqual(self.api.db.get_prompt_by_id(pid)["tags"], [])
|
||||
locator = os.path.normcase(os.path.realpath(str(root / "same.png")))
|
||||
self.assertEqual(
|
||||
self.api.db.read_image_tags(locator, data["sha256"])["tags"],
|
||||
data["tags"],
|
||||
set(self.api.db.get_prompt_by_id(pid)["tags"]), set(data["tags"])
|
||||
)
|
||||
|
||||
async def test_gallery_unlinked_image_returns_no_prompt_id(self):
|
||||
@@ -1818,6 +2058,108 @@ class TestTagOutputImagesJob(BatchJobTestCase):
|
||||
self.assertTrue(inside)
|
||||
|
||||
|
||||
# The legacy batch's per-image frames over BatchJobTestCase's library, newest
|
||||
# first, with every throttle check passing (pinned from main's generator).
|
||||
GOLDEN_LEGACY_ROWS = [
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 24,
|
||||
"status": "Processing 1/6...",
|
||||
"processed": 1,
|
||||
"tagged": 1,
|
||||
"skipped": 0,
|
||||
"skipped_outside_roots": 0,
|
||||
},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 38,
|
||||
"status": "Skipping 2/6 (outside allowed directories)...",
|
||||
"processed": 1,
|
||||
"tagged": 1,
|
||||
"skipped": 1,
|
||||
"skipped_outside_roots": 1,
|
||||
},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 52,
|
||||
"status": "Skipping 3/6 (file missing)...",
|
||||
"processed": 1,
|
||||
"tagged": 1,
|
||||
"skipped": 2,
|
||||
"skipped_outside_roots": 1,
|
||||
},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 66,
|
||||
"status": "Processing 4/6...",
|
||||
"processed": 2,
|
||||
"tagged": 2,
|
||||
"skipped": 2,
|
||||
"skipped_outside_roots": 1,
|
||||
},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 80,
|
||||
"status": "Skipping 5/6 (prompt already processed)...",
|
||||
"processed": 2,
|
||||
"tagged": 2,
|
||||
"skipped": 3,
|
||||
"skipped_outside_roots": 1,
|
||||
},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 95,
|
||||
"status": "Skipping 6/6 (already tagged)...",
|
||||
"processed": 2,
|
||||
"tagged": 2,
|
||||
"skipped": 4,
|
||||
"skipped_outside_roots": 1,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
class TestLegacyBatchFrames(BatchJobTestCase):
|
||||
|
||||
async def test_legacy_batch_events_unchanged(self):
|
||||
"""Golden frames of the legacy batch over the seeded library."""
|
||||
self._seed()
|
||||
ticks = iter(range(1000)) # every throttle check sees a second pass
|
||||
with patch("py.api.autotag_routes._monotonic", side_effect=lambda: next(ticks)):
|
||||
resp = await self.client.post(
|
||||
"/prompt_manager/autotag/start?model_type=wd14-vit&skip_tagged=true"
|
||||
)
|
||||
frames = self._sse_events(await resp.text()) # the stream runs here
|
||||
self.assertEqual(
|
||||
frames,
|
||||
[
|
||||
{"type": "progress", "progress": 0, "status": "Loading model..."},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 5,
|
||||
"status": "Model loaded. Fetching all images from database...",
|
||||
},
|
||||
{
|
||||
"type": "progress",
|
||||
"progress": 10,
|
||||
"status": "Found 6 images. Processing...",
|
||||
},
|
||||
*GOLDEN_LEGACY_ROWS,
|
||||
{
|
||||
"type": "complete",
|
||||
"progress": 100,
|
||||
"processed": 2,
|
||||
"tagged": 2,
|
||||
"skipped": 4,
|
||||
"errors": 0,
|
||||
"skipped_outside_roots": 1,
|
||||
"status": "Complete",
|
||||
"model_status": "Model kept in memory",
|
||||
"model_loaded": True,
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class TestTagOutputImagesEdges(BatchJobTestCase):
|
||||
|
||||
async def test_cancel_during_backup_loads_no_model(self):
|
||||
|
||||
@@ -835,6 +835,7 @@ class TestMetadataParsing(unittest.TestCase):
|
||||
self.assertEqual(parsed["positive_prompt"], "a cat")
|
||||
self.assertEqual(parsed["negative_prompt"], "dog")
|
||||
self.assertEqual(parsed["parameters"]["parameters"], meta["parameters"])
|
||||
self.assertEqual(self.api._extract_readable_prompt(parsed), "a cat")
|
||||
|
||||
def test_parse_comfyui_fields(self):
|
||||
meta = {
|
||||
@@ -855,6 +856,97 @@ class TestMetadataParsing(unittest.TestCase):
|
||||
self.assertEqual(parsed["prompt"], "plain text")
|
||||
self.assertEqual(parsed["workflow"], {"nodes": []})
|
||||
|
||||
def test_extract_readable_prompt_variants(self):
|
||||
extract = self.api._extract_readable_prompt
|
||||
self.assertEqual(extract({"prompt": "plain"}), "plain")
|
||||
self.assertEqual(extract({"prompt": ["a", "", "b"]}), "a b")
|
||||
self.assertEqual(extract({"prompt": 42}), "42")
|
||||
self.assertEqual(
|
||||
extract({"prompt": None, "parameters": {"positive": ["x", "y"]}}), "x y"
|
||||
)
|
||||
self.assertIsNone(extract({"prompt": {}, "parameters": {}}))
|
||||
|
||||
graph = {
|
||||
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "pos"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "neg"}},
|
||||
"3": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["1", 0], "negative": ["2", 0]},
|
||||
},
|
||||
}
|
||||
self.assertEqual(extract({"prompt": graph}), "pos")
|
||||
self.assertEqual(extract({"prompt": {"x": 1}, "workflow": graph}), "pos")
|
||||
|
||||
def test_node_inputs_and_text_lookup(self):
|
||||
self.assertEqual(self.api._get_node_inputs("nope"), {})
|
||||
self.assertEqual(self.api._get_node_inputs({"inputs": 5}), {})
|
||||
self.assertEqual(self.api._get_node_inputs({"inputs": {"a": 1}}), {"a": 1})
|
||||
listed = self.api._get_node_inputs(
|
||||
{"inputs": [{"name": "text", "link": None}, {"nope": 1}, "junk"]}
|
||||
)
|
||||
self.assertEqual(list(listed), ["text"])
|
||||
|
||||
self.assertIsNone(self.api._find_text_in_node(None))
|
||||
self.assertEqual(self.api._find_text_in_node({"inputs": {"text": "hi"}}), "hi")
|
||||
self.assertEqual(
|
||||
self.api._find_text_in_node(
|
||||
{"type": "CLIPTextEncode", "widgets_values": ["w"]}
|
||||
),
|
||||
"w",
|
||||
)
|
||||
self.assertIsNone(
|
||||
self.api._find_text_in_node(
|
||||
{"type": "CLIPTextEncode", "widgets_values": [" "]}
|
||||
)
|
||||
)
|
||||
self.assertIsNone(self.api._find_text_in_node({"type": "KSampler"}))
|
||||
|
||||
def test_positive_prompt_from_nodes_array_and_fallbacks(self):
|
||||
extract = self.api._extract_positive_prompt_from_comfyui_data
|
||||
self.assertIsNone(extract("x"))
|
||||
self.assertIsNone(extract({}))
|
||||
self.assertIsNone(extract({"nodes": []}))
|
||||
|
||||
workflow = {
|
||||
"nodes": [
|
||||
{"id": 1, "type": "CLIPTextEncode", "widgets_values": ["pos"]},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "CLIPTextEncode",
|
||||
"widgets_values": ["neg"],
|
||||
"title": "Negative",
|
||||
},
|
||||
{"id": 3, "type": "KSampler", "inputs": [{"name": "seed", "link": 4}]},
|
||||
"junk",
|
||||
]
|
||||
}
|
||||
self.assertEqual(extract(workflow), "pos")
|
||||
|
||||
only_negative = {
|
||||
"nodes": [
|
||||
{
|
||||
"id": 2,
|
||||
"type": "CLIPTextEncode",
|
||||
"widgets_values": ["neg"],
|
||||
"title": "neg",
|
||||
},
|
||||
]
|
||||
}
|
||||
self.assertEqual(extract(only_negative), "neg")
|
||||
|
||||
bad_link = {
|
||||
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "pos"}},
|
||||
"3": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": "oops", "negative": []},
|
||||
},
|
||||
"4": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["zz", 0], "negative": []},
|
||||
},
|
||||
}
|
||||
self.assertEqual(extract(bad_link), "pos")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -18,19 +18,15 @@ from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.disk_waits import DISK_BOUND_SECONDS
|
||||
from tests.open_handles import open_handles_under
|
||||
from tests.test_api_maintenance import ApplyTestCase, _digest
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from disk_waits import DISK_BOUND_SECONDS
|
||||
from open_handles import open_handles_under
|
||||
from test_api_maintenance import ApplyTestCase, _digest
|
||||
|
||||
from py.api import duplicate_files # noqa: E402
|
||||
from py.config import GalleryConfig, PromptManagerConfig # noqa: E402
|
||||
|
||||
# Gates and server-state polls. Waits on a find or cleanup job's task use
|
||||
# DISK_BOUND_SECONDS: those jobs hash, back up and unlink files (#409).
|
||||
WAIT = 10
|
||||
FIND = "/prompt_manager/jobs/duplicates"
|
||||
GROUPS = "/prompt_manager/maintenance/duplicates/"
|
||||
@@ -87,7 +83,7 @@ class DuplicateFilesTestCase(ApplyTestCase):
|
||||
self.assertEqual(response.status, 202, text)
|
||||
self.assertNotIn(self.tmpdir, text)
|
||||
job = self.api._jobs.get(json.loads(text)["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
return job
|
||||
|
||||
async def groups(self, job_id, status=200, **query):
|
||||
@@ -131,7 +127,7 @@ class DuplicateFilesTestCase(ApplyTestCase):
|
||||
plan = await self.review(job.id, [self.keep_of(group, path)])
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
return plan, cleanup
|
||||
|
||||
async def until(self, predicate):
|
||||
@@ -182,7 +178,7 @@ class TestFindJob(DuplicateFilesTestCase):
|
||||
gate.set()
|
||||
body = await (await legacy).text()
|
||||
job = self.api._jobs.get(summary["id"])
|
||||
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
events = _sse_payloads(body)
|
||||
done = events[-1]
|
||||
self.assertEqual(done["type"], "complete")
|
||||
@@ -245,7 +241,7 @@ class TestFindJob(DuplicateFilesTestCase):
|
||||
self.assertTrue((await cancel.json())["cancellation_requested"])
|
||||
gate.set()
|
||||
job = self.api._jobs.get(job_id)
|
||||
await asyncio.wait_for(asyncio.shield(job.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
last = _sse_payloads(await (await stream).text())[-1]
|
||||
self.assertEqual(last, {"type": "error", "message": "Duplicate scan cancelled"})
|
||||
@@ -345,8 +341,6 @@ class TestReview(DuplicateFilesTestCase):
|
||||
self.assertEqual(plan["protected"], {"count": 0, "reasons": []})
|
||||
|
||||
async def test_cannot_delete_keeper_or_outside_group(self):
|
||||
# About which files may go, not the backup: skip its fsync-heavy copy.
|
||||
PromptManagerConfig.SAFETY_BACKUP = False
|
||||
self.trio()
|
||||
self.write("e.png", b"other pair")
|
||||
self.write("f.png", b"other pair")
|
||||
@@ -378,7 +372,7 @@ class TestReview(DuplicateFilesTestCase):
|
||||
self.assertEqual([row["label"] for row in plan["details"]], ["b.png"])
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.counters["removed"], 1)
|
||||
self.assertTrue((self.output_dir / "a.png").is_file())
|
||||
self.assertTrue((self.output_dir / "sub" / "c.png").is_file())
|
||||
@@ -502,7 +496,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
plan = await self.review(job.id, keep)
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertTrue(kept.is_file())
|
||||
self.assertFalse(gone.exists())
|
||||
self.assertFalse(video_thumb.exists())
|
||||
@@ -514,7 +508,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
self.write("b.png", b"new bytes") # changed since the review
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.counters, {"removed": 1, "kept": 2, "errors": 0})
|
||||
self.assertEqual((self.output_dir / "b.png").read_bytes(), b"new bytes")
|
||||
self.assertFalse((self.output_dir / "sub" / "c.png").exists())
|
||||
@@ -528,7 +522,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
self.write("a.png", b"edited keeper")
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.counters["removed"], 0)
|
||||
self.assertTrue((self.output_dir / "b.png").is_file())
|
||||
self.assertTrue((self.output_dir / "sub" / "c.png").is_file())
|
||||
@@ -555,7 +549,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.state, "failed")
|
||||
self.assertEqual(cleanup.counters, {"removed": 1, "kept": 1, "errors": 1})
|
||||
self.assertEqual(
|
||||
@@ -590,7 +584,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
):
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.counters, {"removed": 0, "kept": 3, "errors": 1})
|
||||
self.assertEqual(
|
||||
cleanup.outcomes["errors"][0]["detail"], duplicate_files.DELETE_FAILED
|
||||
@@ -625,7 +619,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
with context:
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
finally:
|
||||
if handle is not None: # pragma: no cover - Windows
|
||||
handle.close()
|
||||
@@ -648,7 +642,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.counters["removed"], 2)
|
||||
self.assertEqual(open_at_unlink, [])
|
||||
|
||||
@@ -709,7 +703,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
with patch.object(duplicate_files, "_unlink", side_effect=unlink):
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
# Its records moved to the identical kept copy before the unlink failed.
|
||||
[image] = self.api.db.get_prompt_images(prompt)
|
||||
self.assertTrue(image["image_path"].endswith("a.png"))
|
||||
@@ -725,7 +719,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
self.assertEqual((busy["code"], busy["running"]["id"]), ("busy", find["id"]))
|
||||
gate.set()
|
||||
await asyncio.wait_for(
|
||||
asyncio.shield(self.api._jobs.get(find["id"]).task), DISK_BOUND_SECONDS
|
||||
asyncio.shield(self.api._jobs.get(find["id"]).task), WAIT
|
||||
)
|
||||
# The token survived the refusal; the cleanup holds off a new find.
|
||||
release = asyncio.Event()
|
||||
@@ -743,7 +737,7 @@ class TestCleanupJob(DuplicateFilesTestCase):
|
||||
self.assertEqual(legacy.status, 409)
|
||||
release.set()
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
|
||||
async def test_cleanup_refused_up_front_when_the_backup_does_not_fit(self):
|
||||
from py.api import maintenance
|
||||
@@ -971,7 +965,7 @@ class TestReviewHelpers(DuplicateFilesTestCase):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.second_source())]
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.state, "failed")
|
||||
self.assertEqual(cleanup.last_event["code"], "sources-changed")
|
||||
self.assertEqual(self.backups(), [])
|
||||
@@ -991,7 +985,7 @@ class TestReviewHelpers(DuplicateFilesTestCase):
|
||||
):
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
self.assertEqual(cleanup.state, "failed")
|
||||
self.assertEqual(
|
||||
cleanup.outcomes["errors"][-1]["detail"], duplicate_files.RECORDS_FAILED
|
||||
|
||||
@@ -19,17 +19,17 @@ from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.disk_waits import DISK_BOUND_SECONDS
|
||||
from tests.open_handles import open_handles_under
|
||||
from tests import test_api_duplicate_files as dup_tests
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
import test_api_duplicate_files as dup_tests
|
||||
from disk_waits import DISK_BOUND_SECONDS
|
||||
from open_handles import open_handles_under
|
||||
|
||||
from py.api import duplicate_files # noqa: E402
|
||||
from py.config import GalleryConfig # noqa: E402
|
||||
|
||||
WAIT = 10
|
||||
|
||||
|
||||
def _locator(path):
|
||||
return os.path.normcase(os.path.realpath(str(path)))
|
||||
@@ -49,7 +49,7 @@ class SafetyTestCase(dup_tests.DuplicateFilesTestCase):
|
||||
async def apply_plan(self, plan):
|
||||
body = await self.apply(plan["token"])
|
||||
cleanup = self.api._jobs.get(body["job"]["id"])
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), DISK_BOUND_SECONDS)
|
||||
await asyncio.wait_for(asyncio.shield(cleanup.task), WAIT)
|
||||
return cleanup
|
||||
|
||||
|
||||
|
||||
@@ -1169,47 +1169,3 @@ class HistoryHostCleanupTests(unittest.IsolatedAsyncioTestCase):
|
||||
fence.validate(epoch, generation)
|
||||
with self.assertRaises(sqlite3.Error):
|
||||
api._history_fence()
|
||||
|
||||
|
||||
class SharedTempFolderRestoreTests(ImageAPITestCase):
|
||||
"""#346: a restore survives another suite starting in the shared temp folder.
|
||||
|
||||
Every ``PromptManagerAPI`` sweeps its database's folder for stages and
|
||||
uploads an interrupted restore left. A fixture database directly in the
|
||||
shared temp folder let any other test process (another worktree's suite,
|
||||
a pre-push hook) delete this restore's files mid-request. The shared
|
||||
folder here is a scratch stand-in for the system temp folder, so the test
|
||||
never sweeps the real one.
|
||||
"""
|
||||
|
||||
async def get_application(self):
|
||||
shared = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
|
||||
# Cleanups run last-in first-out: this one after the fixture's own.
|
||||
self.addCleanup(shared.cleanup)
|
||||
self.shared_tmp = shared.name
|
||||
with patch.object(tempfile, "tempdir", shared.name):
|
||||
return await super().get_application()
|
||||
|
||||
async def test_restore_survives_another_suite_sweeping_the_shared_folder(self):
|
||||
from database import restore_stage
|
||||
from database.models import PromptModel
|
||||
|
||||
self.api.db.save_prompt("kept by the upload")
|
||||
source = self.comfy_dir / "upload.db"
|
||||
self.assertTrue(self.api.db.model.backup_database(str(source)))
|
||||
stage = PromptModel.stage_restore
|
||||
swept = []
|
||||
|
||||
def stage_then_other_suite_starts(model, src_path):
|
||||
staged = stage(model, src_path)
|
||||
# What another suite's PromptManagerAPI does at startup.
|
||||
other = os.path.join(self.shared_tmp, "other-suite.db")
|
||||
swept.extend(restore_stage.sweep_restore_temps(other))
|
||||
return staged
|
||||
|
||||
form = FormData()
|
||||
form.add_field("database_file", source.read_bytes(), filename="upload.db")
|
||||
with patch.object(PromptModel, "stage_restore", stage_then_other_suite_starts):
|
||||
response = await self.client.post("/prompt_manager/restore", data=form)
|
||||
self.assertEqual(response.status, 200, await response.text())
|
||||
self.assertEqual(swept, [], "another suite's sweep reached this restore")
|
||||
|
||||
@@ -42,7 +42,6 @@ from py import lora_utils # noqa: E402
|
||||
from py import lora_manager_layout as layout_module # noqa: E402
|
||||
from py.api.thumbnail_settings import ThumbnailSettings # noqa: E402
|
||||
from py.api import PromptManagerAPI # noqa: E402
|
||||
from tests.private_temp import private_temp_db # noqa: E402
|
||||
from py.api.prompts import MAX_PAGE_LIMIT # noqa: E402
|
||||
from py.config import GalleryConfig # noqa: E402
|
||||
from utils.hashing import generate_prompt_hash # noqa: E402
|
||||
@@ -61,7 +60,8 @@ class ImageAPITestCase(AioHTTPTestCase):
|
||||
"""App with PromptManager routes, a temp DB and a temp output directory."""
|
||||
|
||||
async def get_application(self):
|
||||
self._temp_db = private_temp_db(self)
|
||||
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
|
||||
self._temp_db.close()
|
||||
|
||||
self._output_tmp = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
|
||||
# A fake ComfyUI tree: <tmp>/output is the gallery root and <tmp> is the
|
||||
|
||||
@@ -415,74 +415,5 @@ class TestApply(LogCleanupTestCase):
|
||||
self.assertEqual(job.status_line, "The log cleanup failed.")
|
||||
|
||||
|
||||
TRUNCATE = "/prompt_manager/logs/truncate"
|
||||
FILES = "/prompt_manager/logs/files"
|
||||
|
||||
|
||||
class TestClearAllLogsConfirmed(LogCleanupTestCase):
|
||||
"""#389: POST /logs/truncate deletes no more than the confirmation listed."""
|
||||
|
||||
async def confirmed(self):
|
||||
response = await self.client.request("GET", FILES)
|
||||
data = await response.json()
|
||||
return [
|
||||
{"name": row["filename"], "identity": row["identity"]}
|
||||
for row in data["files"]
|
||||
if row["kind"] in ("active", "rotated")
|
||||
]
|
||||
|
||||
def handler(self):
|
||||
return next(
|
||||
h
|
||||
for h in self.manager.logger.handlers
|
||||
if isinstance(h, logging.FileHandler)
|
||||
)
|
||||
|
||||
async def test_a_rollover_after_the_confirmation_is_refused(self):
|
||||
self.write("prompt_manager.log.1", b"a" * 100)
|
||||
self.write("prompt_manager.log.2", b"b" * 200)
|
||||
expected = await self.confirmed()
|
||||
self.app_logger.info("written after the confirmation")
|
||||
self.handler().doRollover()
|
||||
before = self.snapshot()
|
||||
body = await self.post(TRUNCATE, {"expected": expected}, 409)
|
||||
self.assertEqual(body["success"], False)
|
||||
self.assertEqual(body["code"], "logs-changed")
|
||||
self.assertIn("nothing was cleared", body["error"])
|
||||
self.assertEqual(self.snapshot(), before)
|
||||
|
||||
async def test_the_confirmed_files_are_cleared(self):
|
||||
self.write("prompt_manager.log.1", b"a" * 100)
|
||||
expected = await self.confirmed()
|
||||
body = await self.post(TRUNCATE, {"expected": expected}, 200)
|
||||
self.assertEqual(
|
||||
sorted(body["results"]["truncated"]),
|
||||
["prompt_manager.log", "prompt_manager.log.1"],
|
||||
)
|
||||
self.assertEqual(self.names(), ["prompt_manager.log"])
|
||||
|
||||
async def test_the_legacy_call_without_a_body_still_clears(self):
|
||||
self.write("prompt_manager.log.1", b"a" * 100)
|
||||
response = await self.client.request("POST", TRUNCATE)
|
||||
self.assertEqual(response.status, 200, await response.text())
|
||||
self.assertEqual(self.names(), ["prompt_manager.log"])
|
||||
|
||||
async def test_a_foreign_writer_is_refused_with_a_clear_message(self):
|
||||
self.write("prompt_manager.log.1", b"a" * 100)
|
||||
with open(self.path("prompt_manager.log"), "ab") as other:
|
||||
other.write(b"line from another ComfyUI\n")
|
||||
body = await self.post(TRUNCATE, {"expected": await self.confirmed()}, 409)
|
||||
self.assertEqual(body["code"], "logs-foreign-writer")
|
||||
self.assertIn("Another program is writing", body["error"])
|
||||
self.assertIn("prompt_manager.log.1", self.names())
|
||||
|
||||
async def test_a_malformed_confirmation_is_400(self):
|
||||
self.write("prompt_manager.log.1", b"a" * 100)
|
||||
for expected in ("all", [{"name": "prompt_manager.log.1", "identity": 5}]):
|
||||
with self.subTest(expected=expected):
|
||||
await self.post(TRUNCATE, {"expected": expected}, 400)
|
||||
self.assertIn("prompt_manager.log.1", self.names())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -36,7 +36,6 @@ from aiohttp.test_utils import AioHTTPTestCase
|
||||
|
||||
from database.operations import PromptDatabase
|
||||
from py.api import PromptManagerAPI
|
||||
from tests.private_temp import private_temp_db
|
||||
from py import config_store, lora_import, lora_utils
|
||||
from py.config import IntegrationConfig, PromptManagerConfig
|
||||
from py.api import settings_routes as settings_module
|
||||
@@ -58,7 +57,8 @@ class LoraAPITestCase(AioHTTPTestCase):
|
||||
"""Stands up the PromptManager routes with a temp DB and neutral LoRA config."""
|
||||
|
||||
async def get_application(self):
|
||||
self._temp_db = private_temp_db(self)
|
||||
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
|
||||
self._temp_db.close()
|
||||
self.tmp_root = Path(tempfile.mkdtemp())
|
||||
self.custom_nodes = self.tmp_root / "custom_nodes"
|
||||
self.custom_nodes.mkdir()
|
||||
|
||||
@@ -5,6 +5,7 @@ Route tests for the prompt API (py/api/prompts.py).
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@@ -28,7 +29,6 @@ except ImportError: # discovered with tests/ as the top-level directory
|
||||
|
||||
from database.operations import PromptDatabase # noqa: E402
|
||||
from py.api import PromptManagerAPI # noqa: E402
|
||||
from tests.private_temp import private_temp_db # noqa: E402
|
||||
import py.api.prompts as prompts_module # noqa: E402
|
||||
from py.api.prompts import ( # noqa: E402
|
||||
BULK_CATEGORY_MAX_PROMPTS,
|
||||
@@ -43,7 +43,8 @@ class PromptAPITestCase(AioHTTPTestCase):
|
||||
"""App with PromptManager routes and a temporary SQLite database."""
|
||||
|
||||
async def get_application(self):
|
||||
self._temp_db = private_temp_db(self)
|
||||
self._temp_db = tempfile.NamedTemporaryFile(delete=False, suffix=".db")
|
||||
self._temp_db.close()
|
||||
|
||||
app = web.Application()
|
||||
routes = web.RouteTableDef()
|
||||
|
||||
@@ -1040,8 +1040,8 @@ class TestApplyRefusals(ReviewRoutesTestCase):
|
||||
_path, item, detail, observed, cat = await self._opened()
|
||||
real = review_capture_module.resolve_prompt_pair
|
||||
|
||||
def reworded(metadata, **kwargs):
|
||||
resolution = real(metadata, **kwargs)
|
||||
def reworded(metadata):
|
||||
resolution = real(metadata)
|
||||
return {
|
||||
**resolution,
|
||||
"reason": "a reworded reason from a newer resolver",
|
||||
@@ -1065,8 +1065,8 @@ class TestApplyRefusals(ReviewRoutesTestCase):
|
||||
_path, item, detail, observed, cat = await self._opened()
|
||||
real = review_capture_module.resolve_prompt_pair
|
||||
|
||||
def without_cat(metadata, **kwargs):
|
||||
resolution = real(metadata, **kwargs)
|
||||
def without_cat(metadata):
|
||||
resolution = real(metadata)
|
||||
kept = [
|
||||
c
|
||||
for c in resolution["candidates"]
|
||||
|
||||
@@ -1831,39 +1831,6 @@ class TestWorkspaceCursorEncoding(unittest.TestCase):
|
||||
self.assertEqual(_workspace_cursor_key(token, "scope", order), key)
|
||||
|
||||
|
||||
# The pre-#379 candidate read, kept as the differential reference for the
|
||||
# indexed lookup (`image_tag_ids` + `image_tag_locators`): every byte record
|
||||
# streamed with its tag names, a locator kept when any record's names are
|
||||
# relevant (`_image_tags_relevant`).
|
||||
_REFERENCE_RECORDS = (
|
||||
"SELECT f.locator,b.id,t.name FROM media_annotation_files f "
|
||||
"JOIN media_annotation_bytes b ON b.file_id=f.id "
|
||||
"LEFT JOIN media_image_tags mit ON mit.byte_id=b.id "
|
||||
"LEFT JOIN tags t ON t.id=mit.tag_id "
|
||||
"ORDER BY f.locator,b.first_seen_at,b.id,mit.position"
|
||||
)
|
||||
|
||||
|
||||
def _image_tags_relevant(names, wanted, query):
|
||||
if wanted and not wanted.isdisjoint(names):
|
||||
return True
|
||||
return bool(query) and any(query in name.casefold() for name in names)
|
||||
|
||||
|
||||
def reference_candidates(db, wanted, query):
|
||||
"""Locators the old full stream admitted for (``wanted``, ``query``)."""
|
||||
records = {}
|
||||
for locator, byte_id, name in db.model.get_connection().execute(_REFERENCE_RECORDS):
|
||||
names = records.setdefault((locator, byte_id), [])
|
||||
if name is not None:
|
||||
names.append(name)
|
||||
return {
|
||||
locator
|
||||
for (locator, _byte), names in records.items()
|
||||
if _image_tags_relevant(names, set(wanted), query)
|
||||
}
|
||||
|
||||
|
||||
class TestWorkspaceImageTags(ImageAPITestCase):
|
||||
"""B3: image tags in the Tags filter and search (spec amendment S1)."""
|
||||
|
||||
@@ -2071,7 +2038,7 @@ class TestWorkspaceImageTags(ImageAPITestCase):
|
||||
leak = "no such table: media_annotation_files"
|
||||
with patch.object(
|
||||
self.api.db,
|
||||
"image_tag_locators",
|
||||
"iter_image_tag_records",
|
||||
side_effect=sqlite3.OperationalError(leak),
|
||||
):
|
||||
# As a failed link read does under a Tags filter: a fixed coded
|
||||
@@ -2089,140 +2056,6 @@ class TestWorkspaceImageTags(ImageAPITestCase):
|
||||
self.assertEqual(self.names(body), ["dunes_by_name.png"])
|
||||
self.assertIn("image tags", "\n".join(logged.output))
|
||||
|
||||
def seed_differential(self):
|
||||
"""Mixed-case and underscore names, a stale record, an unlisted file."""
|
||||
a = self.media("a.png", mtime=101)
|
||||
b = self.media("b.png", mtime=102)
|
||||
c = self.media("c.png", mtime=103)
|
||||
self.tag(a, ["Beach", "long_hair", "1girl"])
|
||||
self.tag(b, ["beach", "waves"])
|
||||
self.tag(c, ["Sunset"])
|
||||
# c's bytes change: its old record (Sunset) is stale, the new one current.
|
||||
make_png(c, size=(5, 5))
|
||||
os.utime(c, (104, 104))
|
||||
self.tag(c, ["night_sky", "BEACH"])
|
||||
outside = Path(self._output_tmp.name) / "elsewhere" / "x.png"
|
||||
outside.parent.mkdir(parents=True, exist_ok=True)
|
||||
make_png(outside)
|
||||
self.tag(outside, ["beach", "outside_only"])
|
||||
|
||||
async def test_indexed_candidates_equal_the_full_stream(self):
|
||||
self.seed_differential()
|
||||
cases = [
|
||||
(["beach"], ""),
|
||||
(["Beach"], ""),
|
||||
(["long_hair", "waves"], ""),
|
||||
([], "beach"),
|
||||
([], "hair"),
|
||||
([], "_"),
|
||||
([], "sun"),
|
||||
(["nope"], "zzz"),
|
||||
(["outside_only"], ""),
|
||||
(["beach"], "night"),
|
||||
]
|
||||
db = self.api.db
|
||||
for wanted, query in cases:
|
||||
with self.subTest(wanted=wanted, query=query):
|
||||
indexed = set(db.image_tag_locators(db.image_tag_ids(wanted, query)))
|
||||
self.assertEqual(indexed, reference_candidates(db, wanted, query))
|
||||
# The API answers are the indexed lookup's: exact names under Tags, a
|
||||
# case-folded substring under q, current records only.
|
||||
_, body = await self.get(tags="beach")
|
||||
self.assertEqual(sorted(self.names(body)), ["b.png"])
|
||||
_, body = await self.get(q="beach")
|
||||
self.assertEqual(sorted(self.names(body)), ["a.png", "b.png", "c.png"])
|
||||
_, body = await self.get(q="sunset")
|
||||
self.assertEqual(self.names(body), [])
|
||||
|
||||
async def test_tag_search_reads_image_tags_only_by_tag_id(self):
|
||||
self.seed_differential()
|
||||
statements = []
|
||||
original = self.api.db.model.get_connection
|
||||
|
||||
def trace_connection():
|
||||
connection = original()
|
||||
connection.set_trace_callback(statements.append)
|
||||
return connection
|
||||
|
||||
with patch.object(
|
||||
self.api.db.model, "get_connection", side_effect=trace_connection
|
||||
):
|
||||
for params in ({"tags": "beach"}, {"q": "hair"}):
|
||||
response, _ = await self.get(**params)
|
||||
self.assertEqual(response.status, 200)
|
||||
reads = [
|
||||
sql
|
||||
for sql in statements
|
||||
if "media_image_tags" in sql and sql.lstrip().upper().startswith("SELECT")
|
||||
]
|
||||
self.assertTrue(reads)
|
||||
for sql in reads:
|
||||
# Only a tag-id lookup, an EXISTS probe on the tag index, or the
|
||||
# current records' own tags (byte_id IN) read the junction.
|
||||
self.assertTrue(
|
||||
"tag_id IN" in sql or "EXISTS" in sql or "byte_id IN" in sql, sql
|
||||
)
|
||||
|
||||
async def test_the_image_tag_lookup_never_reads_prompt_tags(self):
|
||||
self.seed_differential()
|
||||
linked = self.media("linked.png", mtime=105)
|
||||
prompt = self._save_prompt(
|
||||
"Prompt with its own tag", tags=["prompt_only_beach"]
|
||||
)
|
||||
self.api.db.link_image_to_prompt(prompt, str(linked))
|
||||
db = self.api.db
|
||||
# The tag is only on a prompt: the image-tag lookup never yields it,
|
||||
# under the Tags filter or the text search.
|
||||
self.assertEqual(db.image_tag_ids(["prompt_only_beach"], ""), [])
|
||||
self.assertEqual(db.image_tag_ids([], "prompt_only"), [])
|
||||
self.assertEqual(
|
||||
db.image_tag_locators(db.image_tag_ids([], "beach")),
|
||||
reference_candidates(db, [], "beach"),
|
||||
)
|
||||
# The search still matches through the linked positive's prompt tags
|
||||
# (the design keeps that, #379 "Behavior that does not change").
|
||||
_, body = await self.get(q="prompt_only")
|
||||
self.assertEqual(self.names(body), ["linked.png"])
|
||||
|
||||
@unittest.skipUnless(os.environ.get("PM_PERF") == "1", "timing run (PM_PERF=1)")
|
||||
async def test_tag_search_timing(self):
|
||||
"""Linux timing for the PR body: 23,600 records, 10 tags each."""
|
||||
import hashlib
|
||||
import statistics
|
||||
import time
|
||||
|
||||
db = self.api.db
|
||||
epoch = db.get_execution_epoch()
|
||||
names = [f"name_{n:04d}" for n in range(2000)]
|
||||
root = os.path.realpath(self.output_dir)
|
||||
for n in range(23_600):
|
||||
path = self.output_dir / "perf" / f"{n:05d}.png"
|
||||
path.parent.mkdir(exist_ok=True)
|
||||
path.write_bytes(b"x")
|
||||
os.utime(path, (n + 1, n + 1))
|
||||
stat = path.stat()
|
||||
db.replace_image_tags(
|
||||
locator=os.path.normcase(os.path.join(root, "perf", path.name)),
|
||||
sha256=hashlib.sha256(str(n).encode()).hexdigest(),
|
||||
size=stat.st_size,
|
||||
mtime_ns=stat.st_mtime_ns,
|
||||
expected_epoch=epoch,
|
||||
expected_tag_revision=0,
|
||||
tags=[names[(n * 7 + k) % 2000] for k in range(10)],
|
||||
)
|
||||
await self.get(limit=1) # the listing, warm
|
||||
for label, params in (
|
||||
("one tag", {"tags": "name_0042"}),
|
||||
("text", {"q": "e_1"}),
|
||||
):
|
||||
times = []
|
||||
for _ in range(5):
|
||||
started = time.perf_counter()
|
||||
response, _ = await self.get(limit=100, **params)
|
||||
times.append(time.perf_counter() - started)
|
||||
self.assertEqual(response.status, 200)
|
||||
print(f"PM_PERF {label}: median {statistics.median(times) * 1000:.0f} ms")
|
||||
|
||||
async def test_legacy_output_failure_logs_its_traceback(self):
|
||||
self.media("legacy.png", mtime=100)
|
||||
with (
|
||||
|
||||
@@ -1,617 +0,0 @@
|
||||
"""The legacy pages and the Tag image dialog tag images, never prompts (#379, PR 2).
|
||||
|
||||
- ``POST /autotag/start`` (legacy admin and gallery batch) starts the
|
||||
``autotag`` job titled "Tag output images (legacy page)" and relays it as
|
||||
the legacy frames; closing the stream cancels it.
|
||||
- ``POST /autotag/single`` suggests on a capture with the saved cap and
|
||||
answers the captured ``sha256``; ``existing: "skip"`` answers
|
||||
``already_tagged`` for an auto-tagged image without the model gate.
|
||||
- ``POST /autotag/apply`` unions into that image's tags and marks it; a
|
||||
``prompt_id`` body is refused.
|
||||
- A Tag image dialog save after Generate suggestions marks the image.
|
||||
|
||||
No real model runs: the per-image fake from test_autotag_batch answers.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
import threading
|
||||
from unittest.mock import patch
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
try:
|
||||
from test_autotag_batch import BatchTestCase, _absolute, _strings
|
||||
except ImportError:
|
||||
from tests.test_autotag_batch import BatchTestCase, _absolute, _strings
|
||||
|
||||
from database.execution_history_schema import rotate_history_epoch
|
||||
from py import model_gate
|
||||
from py.api import autotag_routes, image_tag_jobs
|
||||
from py.config import AutotagConfig
|
||||
|
||||
START = "/prompt_manager/autotag/start"
|
||||
SINGLE = "/prompt_manager/autotag/single"
|
||||
APPLY = "/prompt_manager/autotag/apply"
|
||||
SUGGEST = "/prompt_manager/image-annotations/suggest"
|
||||
OBSERVE = "/prompt_manager/image-annotations/observe"
|
||||
TAGS = "/prompt_manager/image-annotations/tags"
|
||||
WAIT = 30.0
|
||||
LEGACY_TITLE = "Tag output images (legacy page)"
|
||||
|
||||
|
||||
def frames_of(text):
|
||||
return [
|
||||
json.loads(line[len("data: ") :])
|
||||
for line in text.splitlines()
|
||||
if line.startswith("data: ")
|
||||
]
|
||||
|
||||
|
||||
class LegacyTestCase(BatchTestCase):
|
||||
async def legacy(self, query="model_type=wd14-vit&skip_tagged=true"):
|
||||
response = await self.client.post(f"{START}?{query}")
|
||||
return response, frames_of(await response.text())
|
||||
|
||||
def digest(self, path):
|
||||
return hashlib.sha256((self.output_dir / path).read_bytes()).hexdigest()
|
||||
|
||||
async def single(self, body):
|
||||
response = await self.client.post(SINGLE, json=body)
|
||||
return response.status, await response.json()
|
||||
|
||||
async def apply(self, body):
|
||||
response = await self.client.post(APPLY, json=body)
|
||||
return response.status, await response.json()
|
||||
|
||||
def snapshot_prompts(self):
|
||||
conn = sqlite3.connect(self._temp_db.name)
|
||||
try:
|
||||
return [
|
||||
conn.execute(f"SELECT * FROM {t} ORDER BY 1,2").fetchall()
|
||||
for t in ("prompts", "prompt_tags")
|
||||
]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
class TestLegacyBatch(LegacyTestCase):
|
||||
async def test_the_legacy_batch_tags_each_image_and_relays_frames(self):
|
||||
prompt = self.prompt("one prompt", tags=["beach"])
|
||||
a = self.image("a.png", (1, 2, 3), sees=["bird"])
|
||||
b = self.image("b.png", (4, 5, 6), sees=["waves"])
|
||||
self.link(prompt, a)
|
||||
self.link(prompt, b)
|
||||
before = self.snapshot_prompts()
|
||||
ticks = iter(range(1000))
|
||||
with patch.object(
|
||||
autotag_routes, "_monotonic", side_effect=lambda: next(ticks)
|
||||
):
|
||||
response, frames = await self.legacy()
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(self.stored("a.png")["tags"], ["bird"])
|
||||
self.assertEqual(self.stored("b.png")["tags"], ["waves"])
|
||||
self.assertEqual(self.snapshot_prompts(), before)
|
||||
phases = [f["status"] for f in frames if f.get("progress") in (0, 5, 8)]
|
||||
self.assertEqual(
|
||||
phases,
|
||||
sorted(
|
||||
phases,
|
||||
key=[
|
||||
"Taking a safety backup…",
|
||||
"Loading model…",
|
||||
"Listing output images…",
|
||||
].index,
|
||||
),
|
||||
)
|
||||
self.assertIn("Listing output images…", phases)
|
||||
rows = [f for f in frames if f["type"] == "progress" and "processed" in f]
|
||||
self.assertTrue(rows)
|
||||
for row in rows:
|
||||
self.assertEqual(
|
||||
set(row),
|
||||
{"type", "progress", "status", "processed", "tagged", "skipped"},
|
||||
)
|
||||
self.assertTrue(10 <= row["progress"] <= 95, row)
|
||||
self.assertEqual(
|
||||
frames[-1],
|
||||
{
|
||||
"type": "complete",
|
||||
"progress": 100,
|
||||
"processed": 2,
|
||||
"tagged": 2,
|
||||
"skipped": 0,
|
||||
"errors": 0,
|
||||
"skipped_outside_roots": 0,
|
||||
"status": "Complete",
|
||||
"model_status": "Model kept in memory",
|
||||
"model_loaded": True,
|
||||
},
|
||||
)
|
||||
job = self.api._jobs.list()[0]
|
||||
self.assertEqual(
|
||||
(job.title, job.kind, job.state), (LEGACY_TITLE, "autotag", "finished")
|
||||
)
|
||||
self.assertEqual([s for s in _strings(frames) if _absolute(s)], [])
|
||||
|
||||
# Skip: the marked images are skipped before the model runs.
|
||||
self.tagger.calls.clear()
|
||||
_, again = await self.legacy()
|
||||
self.assertEqual(again[-1]["skipped"], 2)
|
||||
self.assertEqual(again[-1]["processed"], 0)
|
||||
self.assertEqual(self.tagger.calls, [])
|
||||
|
||||
async def test_no_still_images_and_keep_in_memory_false(self):
|
||||
_, frames = await self.legacy("model_type=wd14-vit&keep_in_memory=false")
|
||||
self.assertEqual(frames[-1]["status"], "No still images found")
|
||||
self.assertEqual(frames[-1]["model_loaded"], False)
|
||||
self.assertEqual(frames[-1]["model_status"], "Model unloaded")
|
||||
self.assertEqual(self.tagger.unload_calls, 1)
|
||||
|
||||
async def test_language_model_options_come_from_the_page_or_the_template(self):
|
||||
self.image("a.png", (1, 1, 1))
|
||||
await self.legacy("model_type=gguf&prompt=Tags%20only")
|
||||
self.assertEqual(self.tagger.calls[-1]["prompt"], "Tags only")
|
||||
self.assertEqual(self.tagger.calls[-1]["limit"], 10)
|
||||
self.image("b.png", (2, 2, 2))
|
||||
await self.relist()
|
||||
await self.legacy("model_type=gguf&prompt=%20")
|
||||
self.assertEqual(self.tagger.calls[-1]["prompt"], AutotagConfig.TEMPLATE)
|
||||
self.assertEqual(self.tagger.custom_prompt, "legacy prompt")
|
||||
|
||||
async def test_each_refusal_is_one_error_frame(self):
|
||||
self.tagger.downloaded["wd14-vit"] = False
|
||||
_, frames = await self.legacy()
|
||||
self.assertEqual(len(frames), 1)
|
||||
self.assertEqual(frames[0]["type"], "error")
|
||||
self.assertEqual(frames[0]["status"], "error")
|
||||
self.assertIn("WD14 ViT is not downloaded", frames[0]["message"])
|
||||
self.tagger.downloaded["wd14-vit"] = True
|
||||
|
||||
_, frames = await self.legacy("model_type=wd14-vit&general_threshold=2")
|
||||
self.assertEqual(
|
||||
frames,
|
||||
[
|
||||
{
|
||||
"type": "error",
|
||||
"status": "error",
|
||||
"message": "general_threshold must be between 0 and 1",
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
owner = model_gate.new_owner("image-annotations")
|
||||
self.assertTrue(model_gate.try_acquire(owner))
|
||||
try:
|
||||
_, frames = await self.legacy()
|
||||
finally:
|
||||
model_gate.release(owner)
|
||||
self.assertEqual(frames[0]["message"], autotag_routes.MODEL_BUSY_ERROR)
|
||||
self.assertEqual(self.api._jobs.list(), [])
|
||||
self.assertIsNone(model_gate.holder())
|
||||
|
||||
response = await self.client.post(f"{START}?general_threshold=abc")
|
||||
self.assertEqual(response.status, 400)
|
||||
|
||||
async def test_busy_jobs_refuse_the_legacy_batch(self):
|
||||
self.image("a.png", (1, 1, 1))
|
||||
entered, proceed = threading.Event(), threading.Event()
|
||||
self.tagger.block = (entered, proceed)
|
||||
status, data = await self.start_batch()
|
||||
self.assertEqual(status, 202, data)
|
||||
job = self.api._jobs.get(data["job"]["id"])
|
||||
try:
|
||||
await self.wait_for(entered)
|
||||
_, frames = await self.legacy()
|
||||
self.assertEqual(
|
||||
frames,
|
||||
[
|
||||
{
|
||||
"type": "error",
|
||||
"status": "error",
|
||||
"message": "Tag output images already running",
|
||||
}
|
||||
],
|
||||
)
|
||||
finally:
|
||||
self.tagger.block = None
|
||||
proceed.set()
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
|
||||
# A second legacy batch keeps its JSON 409.
|
||||
self.image("b.png", (2, 2, 2))
|
||||
await self.relist()
|
||||
entered, proceed = threading.Event(), threading.Event()
|
||||
self.tagger.block = (entered, proceed)
|
||||
first = asyncio.ensure_future(
|
||||
self.legacy("model_type=wd14-vit&skip_tagged=false")
|
||||
)
|
||||
try:
|
||||
await self.wait_for(entered)
|
||||
second = await self.client.post(f"{START}?model_type=wd14-vit")
|
||||
self.assertEqual(second.status, 409)
|
||||
self.assertEqual(
|
||||
await second.json(),
|
||||
{"success": False, "error": "Auto-tagging is already running"},
|
||||
)
|
||||
finally:
|
||||
self.tagger.block = None
|
||||
proceed.set()
|
||||
_, frames = await first
|
||||
self.assertEqual(frames[-1]["type"], "complete")
|
||||
|
||||
async def test_a_legacy_start_that_loses_the_race_gets_the_json_409(self):
|
||||
# Two starts can both pass the title check before either registers its
|
||||
# job; the loser's busy refusal names the legacy batch and answers the
|
||||
# same JSON 409 as a second legacy batch.
|
||||
refusal = image_tag_jobs.Refusal(
|
||||
409,
|
||||
"busy",
|
||||
f"{autotag_routes.LEGACY_BATCH_TITLE} already running",
|
||||
web.json_response({}, status=409),
|
||||
)
|
||||
|
||||
async def refused(**_kwargs):
|
||||
return None, refusal
|
||||
|
||||
with patch.object(self.api, "_start_autotag", side_effect=refused):
|
||||
response = await self.client.post(f"{START}?model_type=wd14-vit")
|
||||
self.assertEqual(response.status, 409)
|
||||
self.assertEqual(
|
||||
await response.json(),
|
||||
{"success": False, "error": "Auto-tagging is already running"},
|
||||
)
|
||||
|
||||
async def test_closing_the_stream_cancels_the_job(self):
|
||||
for n in range(3):
|
||||
self.image(f"{n}.png", (n, n, n))
|
||||
real_write = web.StreamResponse.write
|
||||
|
||||
async def flaky_write(response, data):
|
||||
if b'"processed"' in data:
|
||||
raise ConnectionResetError("Cannot write to closing transport")
|
||||
return await real_write(response, data)
|
||||
|
||||
with patch.object(web.StreamResponse, "write", flaky_write):
|
||||
try:
|
||||
response = await self.client.post(f"{START}?model_type=wd14-vit")
|
||||
await response.read()
|
||||
except Exception: # nosec B110 - the client sees the dropped stream
|
||||
pass
|
||||
job = self.api._jobs.list()[0]
|
||||
await asyncio.wait_for(asyncio.shield(job.task), WAIT)
|
||||
self.assertEqual(job.state, "cancelled")
|
||||
self.assertLess(job.counters["tagged"], 3)
|
||||
|
||||
async def test_a_quiet_job_repeats_its_last_frame(self):
|
||||
self.image("a.png", (1, 1, 1))
|
||||
entered, proceed = threading.Event(), threading.Event()
|
||||
self.tagger.block = (entered, proceed)
|
||||
with patch.object(autotag_routes, "LEGACY_REPEAT_AFTER", 0.05):
|
||||
run = asyncio.ensure_future(self.legacy())
|
||||
await self.wait_for(entered)
|
||||
await asyncio.sleep(0.3)
|
||||
self.tagger.block = None
|
||||
proceed.set()
|
||||
_, frames = await run
|
||||
statuses = [f.get("status") for f in frames]
|
||||
repeated = [s for s in statuses if s and statuses.count(s) > 1]
|
||||
self.assertTrue(repeated, statuses)
|
||||
self.assertEqual(frames[-1]["type"], "complete")
|
||||
|
||||
async def test_a_cancel_from_activity_ends_with_a_cancelled_frame(self):
|
||||
for n in range(2):
|
||||
self.image(f"{n}.png", (n, n, n))
|
||||
entered, proceed = threading.Event(), threading.Event()
|
||||
self.tagger.block = (entered, proceed)
|
||||
run = asyncio.ensure_future(self.legacy())
|
||||
try:
|
||||
await self.wait_for(entered)
|
||||
job = self.api._jobs.active("autotag")
|
||||
await self.client.post(f"/prompt_manager/jobs/{job.id}/cancel")
|
||||
finally:
|
||||
self.tagger.block = None
|
||||
proceed.set()
|
||||
_, frames = await run
|
||||
self.assertEqual(
|
||||
frames[-1],
|
||||
{
|
||||
"type": "cancelled",
|
||||
"message": "Cancelled. Tags already added were kept.",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class TestLegacySingleAndApply(LegacyTestCase):
|
||||
async def test_single_suggests_on_a_capture_and_apply_tags_only_that_image(self):
|
||||
prompt = self.prompt("shared prompt", tags=["keep"])
|
||||
a = self.image("a.png", (1, 2, 3), sees=["bird", "sky"])
|
||||
b = self.image("b.png", (4, 5, 6), sees=["cat"])
|
||||
image_a = self.link(prompt, a)
|
||||
self.link(prompt, b)
|
||||
self.saved_set("b.png", b.read_bytes(), ["hand"])
|
||||
before = self.snapshot_prompts()
|
||||
status, data = await self.single(
|
||||
{"image_id": image_a, "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertEqual(status, 200, data)
|
||||
self.assertEqual(data["tags"], ["bird", "sky"])
|
||||
self.assertEqual(data["sha256"], self.digest("a.png"))
|
||||
self.assertEqual(data["image_tags"], [])
|
||||
self.assertEqual(data["prompt_id"], prompt)
|
||||
self.assertEqual(self.tagger.custom_prompt, "legacy prompt")
|
||||
# Inference read the private capture, not the file.
|
||||
self.assertNotIn(str(self.output_dir), self.tagger.read_paths[-1])
|
||||
status, applied = await self.apply(
|
||||
{"image_id": image_a, "sha256": data["sha256"], "tags": ["bird"]}
|
||||
)
|
||||
self.assertEqual(
|
||||
(status, applied),
|
||||
(200, {"success": True, "added_tags": ["bird"], "total_tags": 1}),
|
||||
)
|
||||
self.assertEqual(self.stored("a.png")["tags"], ["bird"])
|
||||
self.assertEqual(self.stored("b.png")["tags"], ["hand"])
|
||||
self.assertEqual(self.snapshot_prompts(), before)
|
||||
self.assertEqual(self.marked(), {self.locator("a.png")})
|
||||
|
||||
async def test_skip_answers_already_tagged_without_the_gate(self):
|
||||
a = self.image("a.png", (1, 2, 3), sees=["bird"])
|
||||
await self.run_batch()
|
||||
self.tagger.calls.clear()
|
||||
owner = model_gate.new_owner("image-tag-job")
|
||||
self.assertTrue(model_gate.try_acquire(owner))
|
||||
try:
|
||||
status, data = await self.single(
|
||||
{
|
||||
"path": "a.png",
|
||||
"root_index": 0,
|
||||
"existing": "skip",
|
||||
"model_type": "wd14-vit",
|
||||
}
|
||||
)
|
||||
finally:
|
||||
model_gate.release(owner)
|
||||
self.assertEqual(status, 200, data)
|
||||
self.assertEqual(data["already_tagged"], True)
|
||||
self.assertEqual(data["image_tags"], ["bird"])
|
||||
self.assertEqual(self.tagger.calls, [])
|
||||
# Hand tags alone do not count; merge always suggests.
|
||||
b = self.image("b.png", (9, 9, 9))
|
||||
self.saved_set("b.png", b.read_bytes(), ["hand"])
|
||||
status, data = await self.single(
|
||||
{"path": str(b), "existing": "skip", "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertNotIn("already_tagged", data)
|
||||
self.assertEqual(data["image_tags"], ["hand"])
|
||||
status, data = await self.single(
|
||||
{"path": str(a), "existing": "merge", "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertNotIn("already_tagged", data)
|
||||
|
||||
async def test_single_uses_the_saved_cap_and_the_refusal_table(self):
|
||||
self.image("a.png", (1, 2, 3))
|
||||
AutotagConfig.SUGGESTION_LIMIT = 3
|
||||
status, data = await self.single(
|
||||
{"path": "a.png", "root_index": 0, "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertEqual((status, len(data["tags"])), (200, 3))
|
||||
(self.output_dir / "broken.png").write_bytes(b"not an image")
|
||||
await self.relist()
|
||||
status, data = await self.single(
|
||||
{"path": "broken.png", "root_index": 0, "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertEqual((status, data["code"]), (415, "not-a-still"))
|
||||
self.assertIn(self.locator("broken.png"), self.marked())
|
||||
self.tagger.downloaded["gguf"] = False
|
||||
status, data = await self.single(
|
||||
{"path": "a.png", "root_index": 0, "model_type": "gguf"}
|
||||
)
|
||||
self.assertEqual((status, data["code"]), (409, "model-unavailable"))
|
||||
self.api._capture_store().max_file = 16
|
||||
status, data = await self.single(
|
||||
{"path": "a.png", "root_index": 0, "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertEqual((status, data["code"]), (413, "too-large"))
|
||||
self.api._capture_store().max_file = 2**20
|
||||
owner = model_gate.new_owner("image-annotations")
|
||||
self.assertTrue(model_gate.try_acquire(owner))
|
||||
try:
|
||||
status, data = await self.single(
|
||||
{"path": "a.png", "root_index": 0, "model_type": "wd14-vit"}
|
||||
)
|
||||
finally:
|
||||
model_gate.release(owner)
|
||||
self.assertEqual((status, data["code"]), (409, "model-busy"))
|
||||
|
||||
async def test_apply_body_and_answer_table(self):
|
||||
a = self.image("a.png", (1, 2, 3))
|
||||
digest = self.digest("a.png")
|
||||
status, data = await self.apply({"prompt_id": 1, "tags": ["a"]})
|
||||
self.assertEqual(
|
||||
(status, data["error"]), (400, "This page is out of date. Reload it.")
|
||||
)
|
||||
status, _ = await self.apply(
|
||||
{"path": "a.png", "root_index": 0, "sha256": "x", "tags": ["a"]}
|
||||
)
|
||||
self.assertEqual(status, 400)
|
||||
status, _ = await self.apply(
|
||||
{"path": "a.png", "root_index": 0, "sha256": digest, "tags": ["bad,name"]}
|
||||
)
|
||||
self.assertEqual(status, 400)
|
||||
status, _ = await self.apply({"path": "a.png", "sha256": digest, "tags": []})
|
||||
self.assertEqual(status, 403) # a bare relative path is not inside a root
|
||||
# An empty reviewed list only marks the image.
|
||||
status, data = await self.apply({"path": str(a), "sha256": digest, "tags": []})
|
||||
self.assertEqual((status, data["added_tags"]), (200, []))
|
||||
self.assertEqual(self.marked(), {self.locator("a.png")})
|
||||
# The file changed since its tags were generated.
|
||||
status, data = await self.apply(
|
||||
{"path": str(a), "sha256": "0" * 64, "tags": ["x"]}
|
||||
)
|
||||
self.assertEqual((status, data["code"]), (409, "file-changed"))
|
||||
# Too many tags.
|
||||
full = self.image("full.png", (7, 7, 7))
|
||||
self.saved_set("full.png", full.read_bytes(), [f"t{i}" for i in range(100)])
|
||||
status, data = await self.apply(
|
||||
{"path": str(full), "sha256": self.digest("full.png"), "tags": ["one_more"]}
|
||||
)
|
||||
self.assertEqual((status, data["code"]), (409, "tag-limit"))
|
||||
# Gone.
|
||||
status, data = await self.apply(
|
||||
{"path": "gone.png", "root_index": 0, "sha256": digest, "tags": ["x"]}
|
||||
)
|
||||
self.assertIn(status, (403, 404))
|
||||
|
||||
async def test_apply_after_a_restore_writes_nothing(self):
|
||||
a = self.image("a.png", (1, 2, 3))
|
||||
digest = self.digest("a.png")
|
||||
real = self.api.db.get_execution_epoch
|
||||
|
||||
def stale_epoch():
|
||||
epoch = real()
|
||||
conn = sqlite3.connect(self._temp_db.name)
|
||||
try:
|
||||
rotate_history_epoch(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
return epoch
|
||||
|
||||
with patch.object(self.api.db, "get_execution_epoch", side_effect=stale_epoch):
|
||||
status, data = await self.apply(
|
||||
{"path": str(a), "sha256": digest, "tags": ["x"]}
|
||||
)
|
||||
self.assertEqual((status, data["code"]), (409, "storage-changed"))
|
||||
self.assertEqual(self.stored("a.png")["tags"], [])
|
||||
|
||||
async def test_image_id_and_path_bodies_reach_the_same_bytes(self):
|
||||
prompt = self.prompt("p")
|
||||
a = self.image("sub/a.png", (1, 2, 3), sees=["x"])
|
||||
image_id = self.link(prompt, a)
|
||||
_, by_id = await self.single({"image_id": image_id, "model_type": "wd14-vit"})
|
||||
_, by_path = await self.single(
|
||||
{"path": "sub/a.png", "root_index": 0, "model_type": "wd14-vit"}
|
||||
)
|
||||
self.assertEqual(by_id["sha256"], by_path["sha256"])
|
||||
status, _ = await self.apply(
|
||||
{"image_id": image_id, "sha256": by_id["sha256"], "tags": ["x"]}
|
||||
)
|
||||
self.assertEqual(status, 200)
|
||||
status, data = await self.apply(
|
||||
{
|
||||
"path": "sub/a.png",
|
||||
"root_index": 0,
|
||||
"sha256": by_path["sha256"],
|
||||
"tags": ["y"],
|
||||
}
|
||||
)
|
||||
self.assertEqual((status, data["total_tags"]), (200, 2))
|
||||
|
||||
|
||||
class TestDialogMarks(LegacyTestCase):
|
||||
"""DEC4: a Tag image dialog save after Generate suggestions marks the image."""
|
||||
|
||||
def target(self, path):
|
||||
return {
|
||||
"root_index": 0,
|
||||
"root_scope": self.api._gallery_root_scope(self.output_dir, 0),
|
||||
"path": path,
|
||||
}
|
||||
|
||||
async def observe(self, path):
|
||||
response = await self.client.post(OBSERVE, json={"target": self.target(path)})
|
||||
data = await response.json()
|
||||
self.assertEqual(response.status, 200, data)
|
||||
return data
|
||||
|
||||
async def suggest(self, observed, path):
|
||||
response = await self.client.post(
|
||||
SUGGEST,
|
||||
json={
|
||||
"target": self.target(path),
|
||||
"capture_token": observed["capture"]["token"],
|
||||
"expected_epoch": observed["library_epoch"],
|
||||
"model_type": "wd14-vit",
|
||||
"options": {"general_threshold": 0.35, "character_threshold": 0.85},
|
||||
},
|
||||
)
|
||||
self.assertEqual(response.status, 200, await response.text())
|
||||
|
||||
async def save(self, observed, path, tags, revision=None):
|
||||
response = await self.client.put(
|
||||
TAGS,
|
||||
json={
|
||||
"target": self.target(path),
|
||||
"capture_token": observed["capture"]["token"],
|
||||
"sha256": observed["capture"]["sha256"],
|
||||
"expected_epoch": observed["library_epoch"],
|
||||
"expected_tag_revision": (
|
||||
observed["tag_revision"] if revision is None else revision
|
||||
),
|
||||
"tags": tags,
|
||||
},
|
||||
)
|
||||
return response.status, await response.json()
|
||||
|
||||
async def test_save_after_generate_marks_and_a_skip_batch_leaves_it(self):
|
||||
self.image("d.png", (1, 2, 3), sees=["bird", "sky"])
|
||||
observed = await self.observe("d.png")
|
||||
await self.suggest(observed, "d.png")
|
||||
status, _ = await self.save(observed, "d.png", ["sky"]) # bird removed
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(self.marked(), {self.locator("d.png")})
|
||||
job = await self.run_batch(existing="skip")
|
||||
self.assertEqual(self.rows(job, "skipped"), [("d.png", "Already auto-tagged")])
|
||||
self.assertEqual(self.stored("d.png")["tags"], ["sky"])
|
||||
|
||||
async def test_save_without_generate_does_not_mark(self):
|
||||
self.image("e.png", (1, 2, 3))
|
||||
observed = await self.observe("e.png")
|
||||
status, _ = await self.save(observed, "e.png", ["hand"])
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(self.marked(), set())
|
||||
|
||||
async def test_a_conflict_then_a_saved_retry_marks(self):
|
||||
f = self.image("f.png", (1, 2, 3))
|
||||
observed = await self.observe("f.png")
|
||||
await self.suggest(observed, "f.png")
|
||||
self.saved_set("f.png", f.read_bytes(), ["other"]) # another writer
|
||||
status, data = await self.save(observed, "f.png", ["mine"])
|
||||
self.assertEqual((status, data["code"]), (409, "tag-conflict"))
|
||||
self.assertEqual(self.marked(), set())
|
||||
status, _ = await self.save(
|
||||
observed, "f.png", ["mine"], revision=data["tag_revision"]
|
||||
)
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(self.marked(), {self.locator("f.png")})
|
||||
|
||||
async def test_another_capture_does_not_inherit_the_mark(self):
|
||||
self.image("g.png", (1, 2, 3))
|
||||
self.image("h.png", (4, 5, 6))
|
||||
first = await self.observe("g.png")
|
||||
await self.suggest(first, "g.png")
|
||||
second = await self.observe("h.png")
|
||||
status, _ = await self.save(second, "h.png", ["x"])
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(self.marked(), set())
|
||||
self.assertIn(first["capture"]["token"], self.api._suggested_tokens)
|
||||
|
||||
async def test_gone_tokens_are_dropped_on_the_next_insert_without_touching_clocks(
|
||||
self,
|
||||
):
|
||||
self.image("i.png", (1, 2, 3))
|
||||
self.image("j.png", (4, 5, 6))
|
||||
first = await self.observe("i.png")
|
||||
await self.suggest(first, "i.png")
|
||||
store = self.api._capture_store()
|
||||
store.release(first["capture"]["token"])
|
||||
second = await self.observe("j.png")
|
||||
seen = []
|
||||
real_get = store.get
|
||||
with patch.object(
|
||||
store, "get", side_effect=lambda t: seen.append(t) or real_get(t)
|
||||
):
|
||||
await self.suggest(second, "j.png")
|
||||
self.assertEqual(seen, [])
|
||||
self.assertEqual(self.api._suggested_tokens, {second["capture"]["token"]})
|
||||
@@ -1398,6 +1398,50 @@ class TestModelHousekeeping(DatabaseTestCase):
|
||||
self.assertIsNot(self.db.model.get_connection(), failing)
|
||||
|
||||
|
||||
class TestAutotagBatchCandidates(DatabaseTestCase):
|
||||
"""#328: the batch's candidate read never loads workflow documents."""
|
||||
|
||||
def test_autotag_candidates_skip_workflow_data(self):
|
||||
from utils.hashing import generate_prompt_hash
|
||||
|
||||
first = self.db.save_prompt(
|
||||
text="a", tags=["one", "two"], prompt_hash=generate_prompt_hash("a")
|
||||
)
|
||||
second = self.db.save_prompt(text="b", prompt_hash=generate_prompt_hash("b"))
|
||||
old = self.db.link_image_to_prompt(
|
||||
first, "/out/a.png", {"workflow": {"big": "x" * 1000}}
|
||||
)
|
||||
new = self.db.link_image_to_prompt(second, "/out/b.png")
|
||||
statements = []
|
||||
conn = self.db.model.get_connection() # this thread's own connection
|
||||
conn.set_trace_callback(statements.append)
|
||||
try:
|
||||
rows = self.db.autotag_batch_candidates()
|
||||
finally:
|
||||
conn.set_trace_callback(None)
|
||||
self.assertEqual(
|
||||
rows[0],
|
||||
{
|
||||
"id": new,
|
||||
"image_path": "/out/b.png",
|
||||
"prompt_id": second,
|
||||
"prompt_tags": [],
|
||||
},
|
||||
)
|
||||
self.assertEqual(
|
||||
{**rows[1], "prompt_tags": sorted(rows[1]["prompt_tags"])},
|
||||
{
|
||||
"id": old,
|
||||
"image_path": "/out/a.png",
|
||||
"prompt_id": first,
|
||||
"prompt_tags": ["one", "two"],
|
||||
},
|
||||
)
|
||||
self.assertEqual(len(rows), 2)
|
||||
self.assertTrue(statements)
|
||||
self.assertFalse(any("workflow_data" in sql for sql in statements))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
@@ -31,12 +31,6 @@ from utils.execution_history_writer import HistoryWriter
|
||||
# 0.19 s on macOS py3.13 (PR #351).
|
||||
NON_BLOCKING_CEILING = 1.0
|
||||
|
||||
# Saturation tests block the worker while they queue 1023 more items. The
|
||||
# block must hold until the test releases it (#346/#352): a 3 s timer let a
|
||||
# loaded runner unblock the worker mid-fill and drain it. This is only a hang
|
||||
# guard; the test, or cleanup, always sets the release.
|
||||
SATURATION_HOLD_SECONDS = 60
|
||||
|
||||
|
||||
class ManualClock:
|
||||
def __init__(self):
|
||||
@@ -90,26 +84,6 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
threading.Event().wait(0.005)
|
||||
self.fail("finite deadline expired waiting for worker behavior")
|
||||
|
||||
def wait_for_worker_pass(self, writer):
|
||||
"""Wait until the idle worker has run one whole loop pass from now.
|
||||
|
||||
A worker with no accepted work ends every pass by parking on its wake
|
||||
event. The first park counted may end a pass already under way; the
|
||||
second ends a pass that began after this call, so whatever the worker
|
||||
decides at the current clock value has happened (state, not a
|
||||
wall-clock pause that a loaded runner can outlast).
|
||||
"""
|
||||
wake = writer._wake
|
||||
park = wake.wait
|
||||
parks = []
|
||||
|
||||
def counted(timeout=None):
|
||||
parks.append(timeout)
|
||||
return park(timeout)
|
||||
|
||||
with patch.object(wake, "wait", counted):
|
||||
self.wait_for(lambda: len(parks) >= 2)
|
||||
|
||||
def writer(self, factory=None, ready=True):
|
||||
writer = HistoryWriter(factory or (lambda: self.db), clock=self.clock)
|
||||
self.writers.append(writer)
|
||||
@@ -327,7 +301,7 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
|
||||
def write(command):
|
||||
entered.set()
|
||||
release.wait(SATURATION_HOLD_SECONDS)
|
||||
release.wait(3)
|
||||
return original(command)
|
||||
|
||||
with patch.object(self.db, "write_execution_start", write):
|
||||
@@ -374,12 +348,9 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
first = self.sealed(writer)
|
||||
charge = first.lease.byte_count
|
||||
|
||||
left = threading.Event()
|
||||
|
||||
def write(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
left.set()
|
||||
return original(*args, **kwargs)
|
||||
|
||||
with patch.object(self.db, "write_execution_contexts_for_finish", write):
|
||||
@@ -425,21 +396,8 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
before = monotonic()
|
||||
self.assertFalse(writer.close(0.02))
|
||||
elapsed = monotonic() - before
|
||||
# close() waits until its own monotonic deadline, so the lower
|
||||
# bound holds on any runner.
|
||||
self.assertGreaterEqual(elapsed, 0.02)
|
||||
if early_join:
|
||||
# Manual clock: only the patched join/sleep advance it, so this
|
||||
# exact bound proves early joins do not spin forever.
|
||||
self.assertLess(elapsed, 0.2)
|
||||
else:
|
||||
# Real clock: a loaded runner may run late, but close() must
|
||||
# not wait anywhere near the 3 s the blocked write holds.
|
||||
self.assertLess(elapsed, 0.02 + NON_BLOCKING_CEILING)
|
||||
# Bounded on the real clock too: close() gave up while the
|
||||
# in-flight write was still blocked, instead of waiting it out.
|
||||
self.assertFalse(left.is_set())
|
||||
self.assertTrue(writer.status().alive)
|
||||
self.assertLess(elapsed, 0.2)
|
||||
self.assertEqual(pool.charged_bytes, charge)
|
||||
replacement = HistoryWriter(lambda: self.db, context_budget=pool)
|
||||
self.writers.append(replacement)
|
||||
@@ -813,7 +771,7 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
self.wait_for(lambda: not writer.status().accepting)
|
||||
self.assertFalse(writer.submit(fresh))
|
||||
self.clock.advance(4.99)
|
||||
self.wait_for_worker_pass(writer)
|
||||
threading.Event().wait(0.08)
|
||||
self.assertEqual(probes, [0])
|
||||
self.clock.advance(0.01)
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
@@ -836,12 +794,12 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
writer = self.writer(factory, ready=False)
|
||||
self.wait_for(lambda: len(calls) == 1)
|
||||
self.clock.advance(4.99)
|
||||
self.wait_for_worker_pass(writer)
|
||||
threading.Event().wait(0.08)
|
||||
self.assertEqual(calls, [0])
|
||||
self.clock.advance(0.01)
|
||||
self.wait_for(lambda: len(calls) == 2)
|
||||
self.clock.advance(4.99)
|
||||
self.wait_for_worker_pass(writer)
|
||||
threading.Event().wait(0.08)
|
||||
self.assertEqual(calls, [0, 5])
|
||||
self.clock.advance(0.01)
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
@@ -895,7 +853,7 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
)
|
||||
writer.invalidate()
|
||||
release.set()
|
||||
self.wait_for_worker_pass(writer)
|
||||
threading.Event().wait(0.08)
|
||||
self.assertFalse(writer.status().accepting)
|
||||
self.assertIsNone(writer.status().epoch)
|
||||
self.assertFalse(writer.submit(self.start_command(epoch=stale_epoch)))
|
||||
@@ -913,7 +871,7 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
lock.execute("BEGIN IMMEDIATE")
|
||||
before = time.monotonic()
|
||||
self.assertTrue(writer.submit(command))
|
||||
self.assertLess(time.monotonic() - before, NON_BLOCKING_CEILING)
|
||||
self.assertLess(time.monotonic() - before, 0.1)
|
||||
self.wait_for(lambda: not writer.status().accepting)
|
||||
lock.rollback()
|
||||
self.clock.advance(5)
|
||||
|
||||
@@ -18,7 +18,6 @@ from utils.execution_output_capture import seal_outputs
|
||||
from utils.execution_outputs import snapshot_output_roots
|
||||
from utils.execution_history_writer import HistoryWriter
|
||||
from tests.history_waits import drain_writer
|
||||
from tests.disk_waits import DISK_BOUND_SECONDS
|
||||
|
||||
try:
|
||||
from tests import test_execution_history_writer as fixtures
|
||||
@@ -29,17 +28,11 @@ except ImportError:
|
||||
# Upper bound a blocked fake write waits for its release. Tests prove ordering
|
||||
# with events, so this only caps a failing run and never decides a pass.
|
||||
BLOCK_SECONDS = 10
|
||||
|
||||
# Saturation tests block the worker while they queue 1023 more items. The
|
||||
# block must hold until the test releases it (#346/#352): a 3 s timer let a
|
||||
# loaded runner unblock the worker mid-fill and drain it. This is only a hang
|
||||
# guard; the test, or cleanup, always sets the release.
|
||||
SATURATION_HOLD_SECONDS = 60
|
||||
|
||||
# A restore is a full verify/migrate/backup/replace cycle and is bound by the
|
||||
# disk: a busy Windows runner took over 10 s (merge queue, #382). Waits that
|
||||
# span a restore use the shared disk-bound hang guard (tests/disk_waits.py).
|
||||
RESTORE_SECONDS = DISK_BOUND_SECONDS
|
||||
# span a restore use this ceiling, which stays under the runner's 120 s hang
|
||||
# watchdog, so a real hang still fails with the thread dump.
|
||||
RESTORE_SECONDS = 90
|
||||
|
||||
|
||||
class OutputWriterTests(fixtures.unittest.TestCase):
|
||||
@@ -598,7 +591,7 @@ class OutputWriterTests(fixtures.unittest.TestCase):
|
||||
|
||||
def write(command):
|
||||
entered.set()
|
||||
release.wait(SATURATION_HOLD_SECONDS)
|
||||
release.wait(3)
|
||||
return original(command)
|
||||
|
||||
with patch.object(self.db, "write_execution_start", write):
|
||||
|
||||
@@ -1,266 +0,0 @@
|
||||
"""A thread or pool refused at interpreter shutdown is quiet (#375).
|
||||
|
||||
When the interpreter exits, ``threading._shutdown`` first shuts every
|
||||
``ThreadPoolExecutor`` (``submit`` then raises "cannot schedule new futures
|
||||
after interpreter shutdown") and joins the non-daemon threads; only then do
|
||||
the ``atexit`` callbacks (the package teardown) run. Our daemon threads keep
|
||||
working meanwhile. ``Thread.start`` is refused ("can't create new thread at
|
||||
interpreter shutdown") only later, once the interpreter is finalizing; the
|
||||
unit tests cover that case, the child process the pool case. Such a refusal
|
||||
is logged once at DEBUG, with no traceback, and leaves nothing pending.
|
||||
|
||||
``threading.Thread`` is replaced only in the module under test (a copy of
|
||||
the ``threading`` namespace), never process-wide. The child-process test runs
|
||||
the real exit: a daemon thread queues previews while the main thread returns.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import threading
|
||||
import types
|
||||
import unittest
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from py.api import thumbnail_auto
|
||||
from utils import image_monitor, parallel
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
CHILD_CEILING_SECONDS = 60 # the hang watchdog allows 120 s per test
|
||||
REPORT_PREFIX = "PM_EXIT_REPORT "
|
||||
THREAD_REFUSAL = "can't create new thread at interpreter shutdown"
|
||||
POOL_REFUSAL = "cannot schedule new futures after interpreter shutdown"
|
||||
|
||||
|
||||
def _refusing_threading():
|
||||
"""A ``threading`` namespace whose ``Thread.start`` is refused."""
|
||||
|
||||
class RefusedThread(threading.Thread):
|
||||
def start(self):
|
||||
raise RuntimeError(THREAD_REFUSAL)
|
||||
|
||||
namespace = types.SimpleNamespace(**vars(threading))
|
||||
namespace.Thread = RefusedThread
|
||||
return namespace
|
||||
|
||||
|
||||
class _ShutPool(ThreadPoolExecutor):
|
||||
def submit(self, *args, **kwargs):
|
||||
raise RuntimeError(POOL_REFUSAL)
|
||||
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def __init__(self):
|
||||
super().__init__(logging.DEBUG)
|
||||
self.records = []
|
||||
|
||||
def emit(self, record):
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
def _capture(testcase, name):
|
||||
parallel._noted.clear() # "once" is per process: start each test afresh
|
||||
logger = logging.getLogger(name)
|
||||
handler = _Capture()
|
||||
previous = logger.level
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
def restore():
|
||||
logger.removeHandler(handler)
|
||||
logger.setLevel(previous)
|
||||
|
||||
testcase.addCleanup(restore)
|
||||
return handler.records
|
||||
|
||||
|
||||
def _assert_one_debug_line(testcase, records):
|
||||
"""Exactly one refusal line, at DEBUG and without a traceback."""
|
||||
refusals = [r for r in records if "interpreter is exiting" in r.getMessage()]
|
||||
testcase.assertEqual(len(refusals), 1, [r.getMessage() for r in records])
|
||||
testcase.assertEqual(refusals[0].levelno, logging.DEBUG)
|
||||
testcase.assertEqual([r for r in records if r.levelno > logging.DEBUG], [])
|
||||
testcase.assertTrue(all(r.exc_info is None for r in records))
|
||||
|
||||
|
||||
class TestScheduleRefusedAtShutdown(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.handler = image_monitor.ImageGenerationHandler(
|
||||
mock.MagicMock(), mock.MagicMock()
|
||||
)
|
||||
self.addCleanup(self.handler.close)
|
||||
|
||||
def test_schedule_returns_false_and_leaves_nothing_pending(self):
|
||||
records = _capture(self, "prompt_manager.image_monitor")
|
||||
with mock.patch.object(image_monitor, "threading", _refusing_threading()):
|
||||
self.assertFalse(self.handler.schedule("/out/a.png"))
|
||||
self.assertFalse(self.handler.schedule("/out/b.png"))
|
||||
self.assertEqual(self.handler.pending_count(), 0)
|
||||
_assert_one_debug_line(self, records)
|
||||
|
||||
def test_another_start_failure_still_raises_and_leaves_nothing_pending(self):
|
||||
class Failing(threading.Thread):
|
||||
def start(self):
|
||||
raise RuntimeError("can't start new thread")
|
||||
|
||||
namespace = types.SimpleNamespace(**vars(threading))
|
||||
namespace.Thread = Failing
|
||||
with mock.patch.object(image_monitor, "threading", namespace):
|
||||
with self.assertRaises(RuntimeError):
|
||||
self.handler.schedule("/out/a.png")
|
||||
self.assertEqual(self.handler.pending_count(), 0)
|
||||
|
||||
def test_schedule_before_shutdown_is_unchanged(self):
|
||||
self.handler.processing_delay = 3600
|
||||
self.assertTrue(self.handler.schedule("/out/a.png"))
|
||||
self.assertEqual(self.handler.pending_count(), 1)
|
||||
|
||||
|
||||
class TestPreviewQueueRefusedAtShutdown(unittest.TestCase):
|
||||
def thumbnailer(self, logger):
|
||||
made = thumbnail_auto.AutoThumbnailer(
|
||||
lambda *a, **k: {"action": "generated"}, lambda: True, logger
|
||||
)
|
||||
self.addCleanup(made.close)
|
||||
return made
|
||||
|
||||
def test_enqueue_is_refused_quietly_and_drops_the_file(self):
|
||||
logger = logging.getLogger("prompt_manager.test_exit_window")
|
||||
records = _capture(self, logger.name)
|
||||
thumbnailer = self.thumbnailer(logger)
|
||||
src = Path("/out/a.png")
|
||||
with mock.patch.object(thumbnail_auto, "threading", _refusing_threading()):
|
||||
self.assertFalse(thumbnailer.enqueue(src, Path("/out/t/a.webp"), False))
|
||||
self.assertFalse(thumbnailer.enqueue(src, Path("/out/t/b.webp"), False))
|
||||
self.assertFalse(thumbnailer._worker_running)
|
||||
self.assertEqual(len(thumbnailer._queue), 0)
|
||||
self.assertEqual(thumbnailer._queued, set())
|
||||
self.assertTrue(thumbnailer.wait_idle(0))
|
||||
_assert_one_debug_line(self, records)
|
||||
|
||||
|
||||
class TestMapParallelRefusedAtShutdown(unittest.TestCase):
|
||||
def test_a_shut_pool_runs_the_items_on_the_calling_thread(self):
|
||||
records = _capture(self, "prompt_manager.parallel")
|
||||
with mock.patch.object(parallel, "ThreadPoolExecutor", _ShutPool):
|
||||
self.assertEqual(
|
||||
parallel.map_parallel(lambda n: n * n, [1, 2, 3], 4), [1, 4, 9]
|
||||
)
|
||||
_assert_one_debug_line(self, records)
|
||||
|
||||
def test_another_pool_error_still_raises(self):
|
||||
class Broken(ThreadPoolExecutor):
|
||||
def submit(self, *args, **kwargs):
|
||||
raise RuntimeError("something else")
|
||||
|
||||
with mock.patch.object(parallel, "ThreadPoolExecutor", Broken):
|
||||
with self.assertRaises(RuntimeError):
|
||||
parallel.map_parallel(lambda n: n, [1, 2], 2)
|
||||
|
||||
|
||||
CHILD = textwrap.dedent("""
|
||||
import threading
|
||||
|
||||
# Registered before concurrent.futures is imported, so it runs after the
|
||||
# pool's own exit hook: from here on, a pool refuses new work.
|
||||
exiting = threading.Event()
|
||||
threading._register_atexit(exiting.set)
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
root, tmp = sys.argv[1], sys.argv[2]
|
||||
sys.path.insert(0, root)
|
||||
|
||||
from py.api.thumbnail_auto import AutoThumbnailer
|
||||
|
||||
logging.basicConfig(level=logging.WARNING, stream=sys.stderr)
|
||||
logger = logging.getLogger("prompt_manager.exit_child")
|
||||
|
||||
def generate(src, dst, is_video, *, stop, guard, track):
|
||||
Path(dst).write_bytes(b"preview")
|
||||
return {"action": "generated"}
|
||||
|
||||
thumbnailer = AutoThumbnailer(
|
||||
generate, lambda: True, logger, workers=lambda: 4
|
||||
)
|
||||
queued = threading.Event()
|
||||
|
||||
def late_monitor():
|
||||
exiting.wait()
|
||||
# One batch of two: the worker maps it on a pool.
|
||||
with thumbnailer._changed:
|
||||
for name in ("a", "b"):
|
||||
src = Path(tmp, name + ".png")
|
||||
src.write_bytes(b"x")
|
||||
thumbnailer.enqueue(src, Path(tmp, name + "_thumb.webp"), False)
|
||||
queued.set()
|
||||
|
||||
def holder():
|
||||
# Non-daemon: the interpreter waits for it, inside the exit window.
|
||||
exiting.wait()
|
||||
queued.wait(30)
|
||||
idle = thumbnailer.wait_idle(30)
|
||||
written = sorted(
|
||||
n for n in os.listdir(tmp) if n.endswith("_thumb.webp")
|
||||
)
|
||||
print("PM_EXIT_REPORT " + json.dumps({"idle": idle, "written": written}))
|
||||
sys.stdout.flush()
|
||||
|
||||
threading.Thread(target=late_monitor, daemon=True).start()
|
||||
threading.Thread(target=holder).start()
|
||||
# Return normally, as ComfyUI does after Ctrl+C.
|
||||
""")
|
||||
|
||||
|
||||
class TestExitWindowChild(unittest.TestCase):
|
||||
def test_previews_queued_in_the_exit_window_log_no_traceback(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
script = os.path.join(tmp, "child.py")
|
||||
with open(script, "w", encoding="utf-8") as handle:
|
||||
handle.write(CHILD)
|
||||
env = dict(os.environ)
|
||||
env.pop("PROMPT_MANAGER_CONFIG_PATH", None)
|
||||
env["PROMPT_MANAGER_DATA_DIR"] = tmp
|
||||
env["PYTHONIOENCODING"] = "utf-8"
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, script, ROOT, tmp],
|
||||
cwd=ROOT,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
timeout=CHILD_CEILING_SECONDS,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
except subprocess.TimeoutExpired as hung:
|
||||
self.fail(
|
||||
f"the child did not exit within {CHILD_CEILING_SECONDS} s\n"
|
||||
f"stdout:\n{hung.stdout}\nstderr:\n{hung.stderr}"
|
||||
)
|
||||
detail = f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
|
||||
self.assertEqual(completed.returncode, 0, detail)
|
||||
reports = [
|
||||
line[len(REPORT_PREFIX) :]
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.startswith(REPORT_PREFIX)
|
||||
]
|
||||
self.assertEqual(len(reports), 1, detail)
|
||||
report = json.loads(reports[0])
|
||||
self.assertTrue(report["idle"], detail)
|
||||
# The batch still ran: on the worker's own thread.
|
||||
self.assertEqual(report["written"], ["a_thumb.webp", "b_thumb.webp"], detail)
|
||||
self.assertNotIn("Traceback", completed.stderr, detail)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -535,9 +535,7 @@ class TestHeldJobStates(_LegacyHoldCase):
|
||||
self.assertIn("event: error", text)
|
||||
self.assertEqual(await self._listed(), [("thumbnails", "failed")])
|
||||
|
||||
async def test_legacy_autotag_refused_at_start_leaves_no_job(self):
|
||||
# #379: the legacy batch is the autotag job; a model that is not
|
||||
# downloaded refuses the start (one error frame) and registers nothing.
|
||||
async def test_legacy_autotag_that_ends_in_error_is_failed(self):
|
||||
service = MagicMock()
|
||||
service.get_models_status.return_value = {} # model not downloaded
|
||||
with patch("py.autotag.get_autotag_service", return_value=service):
|
||||
@@ -547,7 +545,7 @@ class TestHeldJobStates(_LegacyHoldCase):
|
||||
text = await resp.text()
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertIn('"type": "error"', text)
|
||||
self.assertEqual(await self._listed(), [])
|
||||
self.assertEqual(await self._listed(), [("autotag", "failed")])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,136 +0,0 @@
|
||||
"""The legacy pages' backend contract does not move under System S13 (#330).
|
||||
|
||||
S13 points the legacy admin and gallery buttons at native System and removes
|
||||
their duplicate dialogs, changing no backend file. This snapshot pins what the
|
||||
legacy pages talk to: every registered route (method and path) and the key
|
||||
paths of ``GET /prompt_manager/settings``, the one route whose legacy request
|
||||
body S13 shrinks. Record again only with a reason in the PR body:
|
||||
|
||||
PM_RECORD_LEGACY_SNAPSHOT=1 python3 -m unittest tests.test_legacy_contract_snapshot
|
||||
|
||||
Cutover (#334) removes this test and its fixture.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from tests.test_api_admin import AdminAPITestCase
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_admin import AdminAPITestCase
|
||||
|
||||
from py.config import GalleryConfig # noqa: E402
|
||||
|
||||
SNAPSHOT = (
|
||||
Path(__file__).resolve().parent / "fixtures" / "legacy_contract_snapshot.json"
|
||||
)
|
||||
RECORD_ENV = "PM_RECORD_LEGACY_SNAPSHOT"
|
||||
SETTINGS = "/prompt_manager/settings"
|
||||
DEPTH = 2
|
||||
CHANGED = (
|
||||
"the legacy contract changed; if the change is intended (a new route, a new"
|
||||
" settings key), record it with PM_RECORD_LEGACY_SNAPSHOT=1 and say why in"
|
||||
" the PR body"
|
||||
)
|
||||
|
||||
|
||||
def key_paths(value, prefix="", depth=1):
|
||||
"""The key paths of a JSON object to ``DEPTH`` levels.
|
||||
|
||||
``{"a": {"b": 1}}`` gives ``a`` and ``a.b``; the items of a list of
|
||||
objects give the union of their keys as ``a[].b``.
|
||||
"""
|
||||
paths = set()
|
||||
for key, child in value.items():
|
||||
path = f"{prefix}{key}"
|
||||
paths.add(path)
|
||||
if depth >= DEPTH:
|
||||
continue
|
||||
if isinstance(child, dict):
|
||||
paths |= key_paths(child, f"{path}.", depth + 1)
|
||||
elif isinstance(child, list):
|
||||
for item in child:
|
||||
if isinstance(item, dict):
|
||||
paths |= key_paths(item, f"{path}[].", depth + 1)
|
||||
return paths
|
||||
|
||||
|
||||
class TestLegacyContractSnapshot(AdminAPITestCase):
|
||||
"""Routes and settings keys against tests/fixtures/legacy_contract_snapshot.json.
|
||||
|
||||
The base case isolates config.json (``PROMPT_MANAGER_CONFIG_PATH`` in the
|
||||
temp tree), stubs ComfyUI's ``folder_paths`` and resets the gallery roots;
|
||||
nothing here reads the maintainer's install.
|
||||
"""
|
||||
|
||||
async def asyncSetUp(self):
|
||||
await super().asyncSetUp()
|
||||
self.api._get_all_output_dirs = lambda: [str(self.output_dir)]
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [str(self.output_dir)]
|
||||
|
||||
def _routes(self):
|
||||
return sorted(
|
||||
{
|
||||
f"{route.method} {route.resource.canonical}"
|
||||
for route in self.app.router.routes()
|
||||
if route.method != "HEAD"
|
||||
}
|
||||
)
|
||||
|
||||
async def _settings_keys(self):
|
||||
resp = await self.client.request("GET", SETTINGS)
|
||||
body = await resp.json()
|
||||
self.assertEqual(resp.status, 200, body)
|
||||
return sorted(key_paths(body))
|
||||
|
||||
def _snapshot(self):
|
||||
with open(SNAPSHOT, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
def _record(self, key, value):
|
||||
recorded = self._snapshot() if SNAPSHOT.exists() else {}
|
||||
recorded[key] = value
|
||||
with open(SNAPSHOT, "w", encoding="utf-8", newline="\n") as f:
|
||||
json.dump(recorded, f, indent=2)
|
||||
f.write("\n")
|
||||
|
||||
async def test_routes_match_the_snapshot(self):
|
||||
routes = self._routes()
|
||||
if os.environ.get(RECORD_ENV) == "1":
|
||||
self._record("routes", routes)
|
||||
self.assertGreater(len(routes), 50)
|
||||
self.assertEqual(routes, self._snapshot()["routes"], CHANGED)
|
||||
|
||||
async def test_settings_response_keeps_its_keys(self):
|
||||
keys = await self._settings_keys()
|
||||
if os.environ.get(RECORD_ENV) == "1":
|
||||
self._record("settings_keys", keys)
|
||||
self.assertIn("settings.infinite_scroll", keys)
|
||||
self.assertEqual(keys, self._snapshot()["settings_keys"], CHANGED)
|
||||
|
||||
|
||||
class TestKeyPaths(unittest.TestCase):
|
||||
def test_depth_and_list_items(self):
|
||||
value = {
|
||||
"success": True,
|
||||
"settings": {"a": 1, "deep": {"hidden": 1}},
|
||||
"rows": [{"path": "x"}, {"label": "y"}],
|
||||
}
|
||||
self.assertEqual(
|
||||
sorted(key_paths(value)),
|
||||
[
|
||||
"rows",
|
||||
"rows[].label",
|
||||
"rows[].path",
|
||||
"settings",
|
||||
"settings.a",
|
||||
"settings.deep",
|
||||
"success",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+1
-158
@@ -115,12 +115,7 @@ class TestPairScan(AdminAPITestCase):
|
||||
self.assertEqual(second["pairs_added"], 0)
|
||||
|
||||
def test_legacy_single_prompt_second_pass_not_reported_as_new_link(self):
|
||||
# A sampler-less graph: one encoder wired into a guider (#252).
|
||||
single = {
|
||||
"1": graph()["1"],
|
||||
"2": {"class_type": "BasicGuider", "inputs": {"conditioning": ["1", 0]}},
|
||||
}
|
||||
meta = {"prompt": json.dumps(single)}
|
||||
meta = {"prompt": json.dumps({"1": graph()["1"]})}
|
||||
path = self.output_dir / "old.png"
|
||||
self.assertEqual(self.api._ingest_scanned_file(path, meta), "added")
|
||||
self.assertEqual(self.api._ingest_scanned_file(path, meta), "unchanged")
|
||||
@@ -297,155 +292,3 @@ class TestPairScan(AdminAPITestCase):
|
||||
self.api.db.get_image_prompt_info(str(path))["workflow_data"], workflow
|
||||
)
|
||||
self.assertEqual(self.api._ingest_scanned_file(path, metadata), "unchanged")
|
||||
|
||||
|
||||
def flux_graph(text="a red fox", encoder="CLIPTextEncode"):
|
||||
"""A sampler-less Flux graph: the prompt reaches a guider's conditioning (#252)."""
|
||||
return {
|
||||
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux.safetensors"}},
|
||||
"2": {"class_type": "DualCLIPLoader", "inputs": {"clip_name1": "t5"}},
|
||||
"6": {"class_type": encoder, "inputs": {"text": text, "clip": ["2", 0]}},
|
||||
"22": {
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {"model": ["1", 0], "conditioning": ["6", 0]},
|
||||
},
|
||||
"13": {
|
||||
"class_type": "SamplerCustomAdvanced",
|
||||
"inputs": {"guider": ["22", 0], "sigmas": ["17", 0]},
|
||||
},
|
||||
"17": {"class_type": "BasicScheduler", "inputs": {"steps": 20}},
|
||||
"9": {"class_type": "SaveImage", "inputs": {"images": ["13", 0]}},
|
||||
}
|
||||
|
||||
|
||||
class TestSamplerLessScan(AdminAPITestCase):
|
||||
"""Graphs where no node takes a positive or negative input (#252)."""
|
||||
|
||||
def _scanned(self, name, prompt_graph, workflow=None):
|
||||
meta = {"prompt": json.dumps(prompt_graph)}
|
||||
if workflow is not None:
|
||||
meta["workflow"] = json.dumps(workflow)
|
||||
return self.api._ingest_scanned_file(self.output_dir / name, meta)
|
||||
|
||||
def _texts(self):
|
||||
return [p["text"] for p in self.api.db.get_recent_prompts()["prompts"]]
|
||||
|
||||
def test_the_one_wired_prompt_is_linked(self):
|
||||
self.assertEqual(self._scanned("flux.png", flux_graph()), "added")
|
||||
self.assertEqual(self._texts(), ["a red fox"])
|
||||
|
||||
def test_a_text_node_feeding_the_encoder_is_one_prompt(self):
|
||||
g = flux_graph(text=["30", 0])
|
||||
g["30"] = {"class_type": "TextInput", "inputs": {"text": "a red fox"}}
|
||||
self.assertEqual(self._scanned("textinput.png", g), "added")
|
||||
self.assertEqual(self._texts(), ["a red fox"])
|
||||
|
||||
def test_a_display_node_that_feeds_nothing_is_never_the_prompt(self):
|
||||
# A real scan stored a "=== GIT CLONE REPORT ===" display text this way.
|
||||
g = flux_graph()
|
||||
g["40"] = {
|
||||
"class_type": "easy showAnything",
|
||||
"inputs": {"text": "=== GIT CLONE REPORT ===\nTotal repositories: 1"},
|
||||
}
|
||||
self.assertEqual(self._scanned("report.png", g), "added")
|
||||
self.assertEqual(self._texts(), ["a red fox"])
|
||||
|
||||
def test_only_unconsumed_texts_store_nothing(self):
|
||||
g = {
|
||||
"40": {"class_type": "easy showAnything", "inputs": {"text": "report"}},
|
||||
"41": {"class_type": "Textbox", "inputs": {"text": "install note"}},
|
||||
}
|
||||
self.assertEqual(self._scanned("notes.png", g), "found")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_two_wired_texts_are_ambiguous_and_stay_unlinked(self):
|
||||
# A LoRA loader's text input is wired too: the scan stored "<lora:...>".
|
||||
g = flux_graph()
|
||||
g["50"] = {
|
||||
"class_type": "Lora Loader (LoraManager)",
|
||||
"inputs": {
|
||||
"model": ["1", 0],
|
||||
"clip": ["2", 0],
|
||||
"text": "<lora:style:1.00>",
|
||||
},
|
||||
}
|
||||
g["22"]["inputs"]["model"] = ["50", 0]
|
||||
g["6"]["inputs"]["clip"] = ["50", 1]
|
||||
self.assertEqual(self._scanned("lora.png", g), "unresolved")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_an_untraceable_wired_text_makes_the_graph_ambiguous(self):
|
||||
# The encoder's text comes from a node no rule reads: never guess the other.
|
||||
g = flux_graph(text=["60", 0])
|
||||
g["60"] = {"class_type": "SomeLLMNode", "inputs": {"seed": 1}}
|
||||
g["50"] = {
|
||||
"class_type": "Lora Loader (LoraManager)",
|
||||
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
|
||||
}
|
||||
g["22"]["inputs"]["model"] = ["50", 0]
|
||||
self.assertEqual(self._scanned("llm.png", g), "unresolved")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_this_repositorys_prompt_node_is_authoritative(self):
|
||||
g = flux_graph(text="a red fox", encoder="PromptManager")
|
||||
g["50"] = {
|
||||
"class_type": "Lora Loader (LoraManager)",
|
||||
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
|
||||
}
|
||||
g["22"]["inputs"]["model"] = ["50", 0]
|
||||
self.assertEqual(self._scanned("own.png", g), "added")
|
||||
self.assertEqual(self._texts(), ["a red fox"])
|
||||
|
||||
def test_a_numeric_text_is_never_stored_as_a_prompt(self):
|
||||
self.assertEqual(self._scanned("number.png", flux_graph(text="2048")), "found")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_workflow_widget_values_are_never_read(self):
|
||||
# The API text is linked from a node no rule reads; the UI workflow
|
||||
# still holds a stale widget value, which the old scan stored.
|
||||
g = flux_graph(text=["70", 0])
|
||||
g["70"] = {"class_type": "PromptBuilder", "inputs": {"prompt": "x", "n": 2}}
|
||||
workflow = {
|
||||
"nodes": [
|
||||
{"id": 6, "type": "CLIPTextEncode", "widgets_values": ["stale text"]}
|
||||
]
|
||||
}
|
||||
self.assertEqual(self._scanned("stale.png", g, workflow), "unresolved")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_a_lora_loaders_trigger_word_output_is_not_its_text(self):
|
||||
# The encoder reads the loader's trigger-word output (slot 2), not its
|
||||
# text input: the prompt is unknown, never the "<lora:...>" syntax.
|
||||
g = flux_graph(text=["50", 2])
|
||||
g["50"] = {
|
||||
"class_type": "Lora Loader (LoraManager)",
|
||||
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
|
||||
}
|
||||
g["22"]["inputs"]["model"] = ["50", 0]
|
||||
self.assertEqual(self._scanned("trigger.png", g), "unresolved")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_a_seeded_text_node_holds_a_template_not_the_prompt(self):
|
||||
g = flux_graph(text=["80", 0])
|
||||
g["80"] = {
|
||||
"class_type": "RandomPrompt",
|
||||
"inputs": {"text": "a {red|blue} fox", "seed": 5},
|
||||
}
|
||||
self.assertEqual(self._scanned("random.png", g), "unresolved")
|
||||
self.assertEqual(self._texts(), [])
|
||||
|
||||
def test_a1111_parameters_beside_an_undecided_graph_are_read(self):
|
||||
# Metadata savers write both; the old scan read the A1111 positive.
|
||||
g = flux_graph()
|
||||
g["50"] = {
|
||||
"class_type": "Lora Loader (LoraManager)",
|
||||
"inputs": {"model": ["1", 0], "text": "<lora:style:1.00>"},
|
||||
}
|
||||
g["22"]["inputs"]["model"] = ["50", 0]
|
||||
meta = {
|
||||
"prompt": json.dumps(g),
|
||||
"parameters": "a red fox\nNegative prompt: blur\nSteps: 20",
|
||||
}
|
||||
path = self.output_dir / "saver.png"
|
||||
self.assertEqual(self.api._ingest_scanned_file(path, meta), "added")
|
||||
self.assertEqual(self._texts(), ["a red fox"])
|
||||
|
||||
@@ -1,233 +0,0 @@
|
||||
"""A restore cut off by a server stop waits for its worker thread (#360).
|
||||
|
||||
A cancelled handler (or job) task cannot stop the executor thread that
|
||||
stages or publishes. Its cleanup (the stage, the upload) and the registry's
|
||||
``restore`` admission must outlive that thread, and the live library ends as
|
||||
the old or the restored one, never partial, with the safety backup kept.
|
||||
|
||||
Every positive wait is on an event. The one bounded wait (``SETTLE``) is a
|
||||
negative check: the cancelled task must still be pending while its worker is
|
||||
held.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import threading
|
||||
from unittest.mock import patch
|
||||
|
||||
from database import restore_stage
|
||||
from database.models import PromptModel
|
||||
from py.api import admin as admin_mod
|
||||
from py.api.jobs import RESTORE_JOB_KIND
|
||||
|
||||
try:
|
||||
from tests.test_restore_plans import RestoreTestCase, _texts
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_restore_plans import RestoreTestCase, _texts
|
||||
|
||||
WAIT = 60 # a ceiling only a hung test reaches
|
||||
SETTLE = 0.3 # the negative check's bound: the old code's cleanup takes ms
|
||||
|
||||
|
||||
class _Gate:
|
||||
"""Hold one blocking call on its worker thread until released."""
|
||||
|
||||
def __init__(self, real):
|
||||
self.real = real
|
||||
self.entered = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.done = threading.Event()
|
||||
|
||||
def method(self):
|
||||
"""A function to patch onto the class (bound like the real one)."""
|
||||
|
||||
def held(*args):
|
||||
return self(*args)
|
||||
|
||||
return held
|
||||
|
||||
def __call__(self, *args):
|
||||
self.entered.set()
|
||||
self.release.wait(WAIT)
|
||||
try:
|
||||
return self.real(*args)
|
||||
finally:
|
||||
self.done.set()
|
||||
|
||||
|
||||
class ShutdownRestoreCase(RestoreTestCase):
|
||||
def gate(self, real):
|
||||
gate = _Gate(real)
|
||||
# A failed check never leaves the worker held until WAIT.
|
||||
self._gates = getattr(self, "_gates", []) + [gate]
|
||||
self.addCleanup(gate.release.set)
|
||||
return gate
|
||||
|
||||
def stages(self):
|
||||
"""The stage folders (the upload's prefix starts the same way)."""
|
||||
return [
|
||||
name
|
||||
for name in super().stages()
|
||||
if os.path.isdir(os.path.join(self.tmpdir, name))
|
||||
]
|
||||
|
||||
def uploads(self):
|
||||
"""The upload files (not the -wal/-shm its readers may add)."""
|
||||
return sorted(
|
||||
name
|
||||
for name in os.listdir(self.tmpdir)
|
||||
if name.startswith(restore_stage.UPLOAD_PREFIX) and name.endswith(".tmp")
|
||||
)
|
||||
|
||||
def fake_upload(self, texts):
|
||||
"""Make ``_receive_restore_upload`` hand over a prepared upload."""
|
||||
self.upload(texts, "source.db")
|
||||
source = os.path.join(self.tmpdir, "uploads", "source.db")
|
||||
target = os.path.join(self.tmpdir, restore_stage.UPLOAD_PREFIX + "x.tmp")
|
||||
shutil.copyfile(source, target)
|
||||
|
||||
async def receive(request, uploads):
|
||||
uploads.append(target)
|
||||
return target, None
|
||||
|
||||
return patch.object(self.api, "_receive_restore_upload", receive)
|
||||
|
||||
async def wait_thread(self, event):
|
||||
loop = asyncio.get_running_loop()
|
||||
self.assertTrue(await loop.run_in_executor(None, event.wait, WAIT))
|
||||
|
||||
async def assert_still_pending(self, task):
|
||||
done, _ = await asyncio.wait({task}, timeout=SETTLE)
|
||||
if done:
|
||||
for gate in getattr(self, "_gates", ()):
|
||||
gate.release.set() # let the worker end before teardown
|
||||
self.fail("the cancelled task ended while its worker still ran")
|
||||
|
||||
|
||||
class TestLegacyRestoreAtShutdown(ShutdownRestoreCase):
|
||||
async def test_cancel_during_publish_waits_for_the_worker(self):
|
||||
self.save("old")
|
||||
old_db = self.api.db
|
||||
gate = self.gate(PromptModel.publish_staged)
|
||||
with (
|
||||
self.fake_upload(["restored"]),
|
||||
patch.object(PromptModel, "publish_staged", gate.method()),
|
||||
):
|
||||
task = asyncio.ensure_future(self.api.restore_database(None))
|
||||
await self.wait_thread(gate.entered)
|
||||
self.assertEqual(len(self.stages()), 1)
|
||||
task.cancel()
|
||||
await self.assert_still_pending(task)
|
||||
# The worker still has its stage and upload, and runs alone.
|
||||
self.assertEqual(len(self.stages()), 1)
|
||||
self.assertEqual(len(self.uploads()), 1)
|
||||
self.assertIsNotNone(self.api._jobs.active(RESTORE_JOB_KIND))
|
||||
gate.release.set()
|
||||
with self.assertRaises(asyncio.CancelledError):
|
||||
await asyncio.wait_for(task, WAIT)
|
||||
self.assertTrue(gate.done.is_set())
|
||||
self.assertEqual(_texts(self.db_path), ["restored"])
|
||||
self.assertIsNot(self.api.db, old_db) # the restored library adopted
|
||||
self.assertEqual(len(self.backups()), 1) # the safety backup is kept
|
||||
self.assertEqual(self.stages(), [])
|
||||
self.assertEqual(self.uploads(), [])
|
||||
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
|
||||
|
||||
async def test_cancel_during_staging_waits_and_changes_nothing(self):
|
||||
self.save("old")
|
||||
gate = self.gate(PromptModel.stage_restore)
|
||||
with (
|
||||
self.fake_upload(["restored"]),
|
||||
patch.object(PromptModel, "stage_restore", gate.method()),
|
||||
):
|
||||
task = asyncio.ensure_future(self.api.restore_database(None))
|
||||
await self.wait_thread(gate.entered)
|
||||
task.cancel()
|
||||
await self.assert_still_pending(task)
|
||||
self.assertEqual(len(self.uploads()), 1) # still being read
|
||||
self.assertIsNotNone(self.api._jobs.active(RESTORE_JOB_KIND))
|
||||
gate.release.set()
|
||||
with self.assertRaises(asyncio.CancelledError):
|
||||
await asyncio.wait_for(task, WAIT)
|
||||
self.assertEqual(_texts(self.db_path), ["old"])
|
||||
self.assertEqual(self.backups(), [])
|
||||
self.assertEqual(self.stages(), []) # the finished stage is removed
|
||||
self.assertEqual(self.uploads(), [])
|
||||
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
|
||||
|
||||
async def test_a_publish_that_fails_after_the_cancel_still_cancels(self):
|
||||
self.save("old")
|
||||
|
||||
def refuse(model, staged):
|
||||
raise sqlite3.OperationalError("refused")
|
||||
|
||||
gate = self.gate(refuse)
|
||||
with (
|
||||
self.fake_upload(["restored"]),
|
||||
patch.object(PromptModel, "publish_staged", gate.method()),
|
||||
):
|
||||
task = asyncio.ensure_future(self.api.restore_database(None))
|
||||
await self.wait_thread(gate.entered)
|
||||
task.cancel()
|
||||
await self.assert_still_pending(task)
|
||||
gate.release.set()
|
||||
# Not answered as a 500: the cancel goes on.
|
||||
with self.assertRaises(asyncio.CancelledError):
|
||||
await asyncio.wait_for(task, WAIT)
|
||||
self.assertEqual(_texts(self.db_path), ["old"])
|
||||
self.assertEqual(self.stages(), [])
|
||||
self.assertEqual(self.uploads(), [])
|
||||
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
|
||||
|
||||
async def test_a_second_cancel_still_adopts_the_restored_library(self):
|
||||
self.save("old")
|
||||
old_db = self.api.db
|
||||
publish = self.gate(PromptModel.publish_staged)
|
||||
prune = self.gate(admin_mod._prune_safety_backups)
|
||||
with (
|
||||
self.fake_upload(["restored"]),
|
||||
patch.object(PromptModel, "publish_staged", publish.method()),
|
||||
patch.object(admin_mod, "_prune_safety_backups", prune),
|
||||
):
|
||||
task = asyncio.ensure_future(self.api.restore_database(None))
|
||||
await self.wait_thread(publish.entered)
|
||||
task.cancel()
|
||||
await self.assert_still_pending(task)
|
||||
publish.release.set()
|
||||
await self.wait_thread(prune.entered) # adopting the library
|
||||
task.cancel() # the loop's own shutdown cancels again
|
||||
await self.assert_still_pending(task)
|
||||
prune.release.set()
|
||||
with self.assertRaises(asyncio.CancelledError):
|
||||
await asyncio.wait_for(task, WAIT)
|
||||
self.assertEqual(_texts(self.db_path), ["restored"])
|
||||
self.assertIsNot(self.api.db, old_db)
|
||||
self.assertEqual(len(self.backups()), 1)
|
||||
self.assertEqual(self.stages(), [])
|
||||
self.assertIsNone(self.api._jobs.active(RESTORE_JOB_KIND))
|
||||
|
||||
|
||||
class TestReviewedRestoreAtShutdown(ShutdownRestoreCase):
|
||||
async def test_cancelled_job_waits_for_the_publish(self):
|
||||
self.save("old")
|
||||
old_db = self.api.db
|
||||
body = await self.preview(self.upload(["restored"]))
|
||||
gate = self.gate(PromptModel.publish_staged)
|
||||
with patch.object(PromptModel, "publish_staged", gate.method()):
|
||||
started = await self.apply(body["token"])
|
||||
job = self.api._jobs.get(started["job"]["id"])
|
||||
await self.wait_thread(gate.entered)
|
||||
job.task.cancel()
|
||||
await self.assert_still_pending(job.task)
|
||||
self.assertEqual(len(self.stages()), 1)
|
||||
self.assertTrue(job.running) # its exclusive admission holds
|
||||
gate.release.set()
|
||||
with self.assertRaises(asyncio.CancelledError):
|
||||
await asyncio.wait_for(job.task, WAIT)
|
||||
self.assertEqual(_texts(self.db_path), ["restored"])
|
||||
self.assertIsNot(self.api.db, old_db)
|
||||
self.assertEqual(len(self.backups()), 1)
|
||||
self.assertEqual(self.stages(), [])
|
||||
self.assertFalse(job.running)
|
||||
@@ -298,21 +298,19 @@ class TestQwenEditBatchIsQueued(InspectionTestCase):
|
||||
|
||||
class TestDefaultProfileStillDecides(InspectionTestCase):
|
||||
async def test_ambiguous_file_with_texts_keeps_the_default_profile(self):
|
||||
"""A default-ambiguous file with a choice is recorded from the
|
||||
default profile, even when the inspection profile would offer more
|
||||
candidates. That profile only checks the saver anchor (#390): no
|
||||
saver names this file, so it anchors nothing."""
|
||||
"""A default-ambiguous file with a choice never runs the inspection
|
||||
profile, even when that profile would offer more candidates."""
|
||||
self._roots()
|
||||
graph = {
|
||||
**_prefixed(qwen_edit_graph("make it night"), "q"),
|
||||
**unlinkable_graph("cat", "dog"),
|
||||
}
|
||||
self._write("unsaved_00001_.png", graph=graph)
|
||||
self._write(graph=graph)
|
||||
|
||||
with patch.object(
|
||||
review_capture,
|
||||
"inspection_choice",
|
||||
side_effect=AssertionError("inspection choices were asked"),
|
||||
"resolve_inspection",
|
||||
side_effect=AssertionError("inspection profile ran"),
|
||||
):
|
||||
await self._scan()
|
||||
|
||||
@@ -323,24 +321,17 @@ class TestDefaultProfileStillDecides(InspectionTestCase):
|
||||
self.assertEqual(item["resolver_version"], review_capture.RESOLVER_VERSION)
|
||||
self.assertEqual(_texts(evidence), ["cat", "dog"])
|
||||
|
||||
async def test_apply_of_a_default_item_inspects_only_for_the_anchor(
|
||||
self,
|
||||
):
|
||||
async def test_apply_of_a_default_item_never_runs_the_inspection_profile(self):
|
||||
self._roots()
|
||||
write_png(self.output_dir / "two.png", prompt=unlinkable_graph("cat", "dog"))
|
||||
await self._scan()
|
||||
with (
|
||||
_counted(review_capture, "resolve_inspection") as calls,
|
||||
patch.object(
|
||||
review_capture,
|
||||
"inspection_choice",
|
||||
side_effect=AssertionError("inspection choices were asked"),
|
||||
),
|
||||
with patch.object(
|
||||
review_capture,
|
||||
"resolve_inspection",
|
||||
side_effect=AssertionError("inspection profile ran"),
|
||||
):
|
||||
_item, (status, data) = await self._apply("cat")
|
||||
self.assertEqual(status, 200, data)
|
||||
# The saver anchor check (#390): with the file's name, nothing matches.
|
||||
self.assertEqual([c[1] for c in calls], ["two.png"])
|
||||
self.assertEqual(self._linked_text(self.output_dir / "two.png"), "cat")
|
||||
|
||||
|
||||
@@ -378,16 +369,15 @@ class TestApplyInspectionCandidate(InspectionTestCase):
|
||||
calls = []
|
||||
real = review_capture.resolve_inspection
|
||||
|
||||
def spy(metadata, file_name=None):
|
||||
calls.append((metadata, file_name))
|
||||
return real(metadata, file_name)
|
||||
def spy(metadata):
|
||||
calls.append(metadata)
|
||||
return real(metadata)
|
||||
|
||||
with patch.object(review_capture, "resolve_inspection", side_effect=spy):
|
||||
_item, (status, data) = await self._apply("make it night")
|
||||
|
||||
self.assertEqual(status, 200, data)
|
||||
self.assertEqual(len(calls), 1)
|
||||
self.assertEqual(calls[0][1], NAME) # anchored on its saver (#390)
|
||||
|
||||
async def test_table_change_keeps_the_item_applicable(self):
|
||||
"""A table change for a class the graph uses moves the item's
|
||||
@@ -863,8 +853,8 @@ class TestInspectionResolverVersion(InspectionTestCase):
|
||||
version = _version()
|
||||
real = review_capture.resolve_inspection
|
||||
|
||||
def newer(metadata, file_name=None):
|
||||
resolution = real(metadata, file_name)
|
||||
def newer(metadata):
|
||||
resolution = real(metadata)
|
||||
return {
|
||||
**resolution,
|
||||
"candidates": [
|
||||
@@ -909,10 +899,10 @@ class TestScanContainment(InspectionTestCase):
|
||||
self._write("b.png", graph=qwen_edit_graph("snow", "rain"))
|
||||
real = review_capture.resolve_inspection
|
||||
|
||||
def fails_for_a(metadata, file_name=None):
|
||||
def fails_for_a(metadata):
|
||||
if "make it night" in json.dumps(metadata):
|
||||
raise RuntimeError("inspection blew up")
|
||||
return real(metadata, file_name)
|
||||
return real(metadata)
|
||||
|
||||
with (
|
||||
patch.object(review_capture, "resolve_inspection", side_effect=fails_for_a),
|
||||
|
||||
@@ -1,333 +0,0 @@
|
||||
"""The scan and Needs review anchor a file on the saver that wrote it (#390).
|
||||
|
||||
The workspace inspector (since #383) passes the file's name to the resolver,
|
||||
which anchors the file on the one saver whose ``filename_prefix`` names it
|
||||
(``_GraphResolver.saver_stages``). The scan and Needs review use the same
|
||||
rule. Maintainer ruling: a file never falls through both the link and the
|
||||
queue, so when only that anchor resolves a file the default profile and the
|
||||
parser floor do not link, it is queued with the anchor's pair as its one
|
||||
choice (never linked automatically). No match, several matches or an
|
||||
unfinished walk anchor nothing, so such a file is queued as before. Apply
|
||||
re-resolves with the file's name, so a choice the anchor contradicts is
|
||||
refused as stale evidence.
|
||||
"""
|
||||
|
||||
import json
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.test_review import unlinkable_graph, write_png
|
||||
from tests.test_review_inspection import (
|
||||
PROMPTS,
|
||||
_counted,
|
||||
InspectionTestCase,
|
||||
qwen_edit_graph,
|
||||
)
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_review import unlinkable_graph, write_png
|
||||
from test_review_inspection import (
|
||||
PROMPTS,
|
||||
_counted,
|
||||
InspectionTestCase,
|
||||
qwen_edit_graph,
|
||||
)
|
||||
|
||||
from utils import review_capture
|
||||
|
||||
|
||||
def branch_graph(*prompts, prefixes=None):
|
||||
"""A Qwen-edit batch whose every SaveImage has its own literal prefix."""
|
||||
graph = qwen_edit_graph(*prompts)
|
||||
savers = [n for n in graph.values() if n["class_type"] == "SaveImage"]
|
||||
names = prefixes or [f"branch_{index}" for index in range(len(savers))]
|
||||
for saver, prefix in zip(savers, names):
|
||||
saver["inputs"]["filename_prefix"] = prefix
|
||||
return graph
|
||||
|
||||
|
||||
def saved_unlinkable_graph(*positives):
|
||||
"""``unlinkable_graph`` whose branches save under their own prefix.
|
||||
|
||||
The default profile finds it ambiguous with a choice and the scan's
|
||||
browser-parser floor links nothing, so it is queued from the default
|
||||
profile unless the saver that wrote the file anchors its branch.
|
||||
"""
|
||||
graph = unlinkable_graph(*positives)
|
||||
samplers = [
|
||||
node_id
|
||||
for node_id, node in graph.items()
|
||||
if node["class_type"] == "KSampler" and node_id != "3"
|
||||
]
|
||||
for index, sampler in enumerate(samplers):
|
||||
graph[f"{sampler}1"] = {
|
||||
"class_type": "VAEDecode",
|
||||
"inputs": {"samples": [sampler, 0], "vae": ["1", 2]},
|
||||
}
|
||||
graph[f"{sampler}2"] = {
|
||||
"class_type": "SaveImage",
|
||||
"inputs": {
|
||||
"images": [f"{sampler}1", 0],
|
||||
"filename_prefix": f"side_{index}",
|
||||
},
|
||||
}
|
||||
return graph
|
||||
|
||||
|
||||
def _without_name(real):
|
||||
"""``resolve_inspection`` as it was before #390: the file name ignored."""
|
||||
|
||||
def unnamed(metadata, file_name=None):
|
||||
return real(metadata)
|
||||
|
||||
return unnamed
|
||||
|
||||
|
||||
def _only_candidate(test, item):
|
||||
"""The positive text of an anchored item's one candidate."""
|
||||
evidence = json.loads(item["evidence"])
|
||||
test.assertEqual(evidence.get("anchor"), "saver")
|
||||
test.assertEqual(evidence.get("profile"), "inspection")
|
||||
(candidate,) = evidence["candidates"]
|
||||
return candidate["positive"]["text"]
|
||||
|
||||
|
||||
class TestInspectionItemsAnchor(InspectionTestCase):
|
||||
async def test_file_its_saver_names_is_queued_with_the_anchor_choice(self):
|
||||
self._roots()
|
||||
path = self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(_only_candidate(self, item), "add a red hat")
|
||||
self.assertEqual(item["candidate_count"], 1)
|
||||
self.assertEqual(
|
||||
item["resolver_version"],
|
||||
review_capture.inspection_review_version(
|
||||
{"prompt": branch_graph(*PROMPTS)}
|
||||
),
|
||||
)
|
||||
(row,) = job["outcomes"]["review"]
|
||||
self.assertEqual(row["path"], "branch_1_00001_.png")
|
||||
# Queued, never linked: ingestion stays on the default profile.
|
||||
self.assertIsNone(self._linked_text(path))
|
||||
|
||||
async def test_anchored_item_stays_on_rescan(self):
|
||||
self._roots()
|
||||
self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
|
||||
await self._scan()
|
||||
(item,) = self._items()
|
||||
|
||||
job = await self._scan("all")
|
||||
|
||||
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
|
||||
self.assertEqual(job["scope"]["review_untraced"], 0)
|
||||
|
||||
async def test_file_no_saver_names_stays_queued(self):
|
||||
self._roots()
|
||||
self._write("elsewhere_00001_.png", graph=branch_graph(*PROMPTS))
|
||||
|
||||
await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(item["candidate_count"], len(PROMPTS))
|
||||
self.assertNotIn("anchor", json.loads(item["evidence"]))
|
||||
|
||||
async def test_file_several_savers_name_stays_queued(self):
|
||||
self._roots()
|
||||
graph = branch_graph(*PROMPTS, prefixes=["same", "same", "other"])
|
||||
self._write("same_00001_.png", graph=graph)
|
||||
|
||||
await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(item["candidate_count"], len(PROMPTS))
|
||||
|
||||
async def test_old_rule_item_is_replaced_by_the_anchor_choice(self):
|
||||
self._roots()
|
||||
self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
|
||||
real = review_capture.resolve_inspection
|
||||
with (
|
||||
patch.object(review_capture, "REVIEW_RULE_VERSION", 0),
|
||||
patch.object(
|
||||
review_capture, "resolve_inspection", side_effect=_without_name(real)
|
||||
),
|
||||
):
|
||||
await self._scan()
|
||||
(old,) = self._items() # queued under the old rule with every branch
|
||||
self.assertEqual(old["candidate_count"], len(PROMPTS))
|
||||
|
||||
await self._scan("all")
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertNotEqual(item["id"], old["id"])
|
||||
self.assertEqual(_only_candidate(self, item), "add a red hat")
|
||||
|
||||
async def test_apply_links_the_anchor_choice(self):
|
||||
self._require_routes()
|
||||
self._roots()
|
||||
path = self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
|
||||
await self._scan()
|
||||
|
||||
_item, (status, data) = await self._apply("add a red hat")
|
||||
|
||||
self.assertEqual(status, 200, data)
|
||||
self.assertEqual(self._linked_text(path), "add a red hat")
|
||||
|
||||
async def test_apply_refuses_an_old_choice_the_anchor_contradicts(self):
|
||||
self._require_routes()
|
||||
self._roots()
|
||||
path = self._write("branch_1_00001_.png", graph=branch_graph(*PROMPTS))
|
||||
real = review_capture.resolve_inspection
|
||||
with (
|
||||
patch.object(review_capture, "REVIEW_RULE_VERSION", 0),
|
||||
patch.object(
|
||||
review_capture, "resolve_inspection", side_effect=_without_name(real)
|
||||
),
|
||||
):
|
||||
await self._scan()
|
||||
item, detail, observed = await self._open()
|
||||
# branch_1 is the saver of "add a red hat"; choose another branch.
|
||||
wrong = self._candidate(detail, "make it night")
|
||||
|
||||
status, data = await self._decide(
|
||||
item["id"], self._body(detail, observed, wrong["hash"])
|
||||
)
|
||||
|
||||
self.assertEqual(status, 409, data)
|
||||
self.assertEqual(data["code"], "review-conflict", data)
|
||||
self.assertIsNone(self._linked_text(path))
|
||||
|
||||
|
||||
class TestDefaultItemsAnchor(InspectionTestCase):
|
||||
async def test_anchor_resolves_and_nothing_links_so_it_is_queued(self):
|
||||
"""Maintainer ruling: never unlinked and unqueued. The anchor resolves
|
||||
the file, the default profile and the parser floor link nothing, so
|
||||
it is queued with the anchor's pair as its choice."""
|
||||
self._roots()
|
||||
path = self.output_dir / "side_0_00001_.png"
|
||||
graph = saved_unlinkable_graph("cat", "dog")
|
||||
write_png(path, prompt=graph)
|
||||
self.assertEqual(
|
||||
review_capture.resolve_prompt_pair({"prompt": graph})["state"], "ambiguous"
|
||||
)
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertEqual(_only_candidate(self, item), "cat")
|
||||
(row,) = job["outcomes"]["review"]
|
||||
self.assertIn("more than one prompt could match", row["detail"])
|
||||
self.assertIsNone(self._linked_text(path))
|
||||
|
||||
async def test_default_ambiguous_file_no_saver_names_stays_queued(self):
|
||||
self._roots()
|
||||
write_png(
|
||||
self.output_dir / "unsaved_00001_.png",
|
||||
prompt=saved_unlinkable_graph("cat", "dog"),
|
||||
)
|
||||
|
||||
await self._scan()
|
||||
|
||||
(item,) = self._items()
|
||||
evidence = json.loads(item["evidence"])
|
||||
self.assertNotIn("profile", evidence)
|
||||
self.assertNotIn("anchor", evidence)
|
||||
self.assertEqual(item["resolver_version"], review_capture.RESOLVER_VERSION)
|
||||
|
||||
async def test_file_the_parser_floor_links_is_not_queued(self):
|
||||
"""Already linked by the default path: unchanged by the anchor."""
|
||||
self._roots()
|
||||
graph = saved_unlinkable_graph("cat", "dog")
|
||||
del graph["3"] # the parser floor now reads a real sampler
|
||||
path = self.output_dir / "side_0_00001_.png"
|
||||
write_png(path, prompt=graph)
|
||||
|
||||
await self._scan()
|
||||
|
||||
self.assertEqual(self._items(), [])
|
||||
self.assertIsNotNone(self._linked_text(path))
|
||||
|
||||
async def test_apply_refuses_an_old_default_choice_the_anchor_contradicts(self):
|
||||
self._require_routes()
|
||||
self._roots()
|
||||
path = self.output_dir / "side_0_00001_.png"
|
||||
write_png(path, prompt=saved_unlinkable_graph("cat", "dog"))
|
||||
with patch.object(review_capture, "_captured_inspection", return_value=None):
|
||||
await self._scan() # queued from the default profile, as before #390
|
||||
item, detail, observed = await self._open()
|
||||
# side_0 is the saver of "cat"; choose the other branch.
|
||||
wrong = self._candidate(detail, "dog")
|
||||
|
||||
status, data = await self._decide(
|
||||
item["id"], self._body(detail, observed, wrong["hash"])
|
||||
)
|
||||
|
||||
self.assertEqual(status, 409, data)
|
||||
self.assertEqual(data["code"], "review-conflict", data)
|
||||
self.assertIsNone(self._linked_text(path))
|
||||
|
||||
async def test_old_default_item_is_replaced_by_the_anchor_choice(self):
|
||||
self._roots()
|
||||
write_png(
|
||||
self.output_dir / "side_0_00001_.png",
|
||||
prompt=saved_unlinkable_graph("cat", "dog"),
|
||||
)
|
||||
with patch.object(review_capture, "_captured_inspection", return_value=None):
|
||||
await self._scan() # queued from the default profile, as before #390
|
||||
(old,) = self._items()
|
||||
self.assertEqual(old["resolver_version"], review_capture.RESOLVER_VERSION)
|
||||
|
||||
await self._scan("all")
|
||||
|
||||
(item,) = self._items()
|
||||
self.assertNotEqual(item["id"], old["id"])
|
||||
self.assertEqual(_only_candidate(self, item), "cat")
|
||||
|
||||
with _counted(review_capture, "resolve_inspection") as calls:
|
||||
await self._scan("all") # now queued under its version: no re-read
|
||||
self.assertEqual(calls, [])
|
||||
self.assertEqual([i["id"] for i in self._items()], [item["id"]])
|
||||
|
||||
async def test_decided_anchored_item_is_kept(self):
|
||||
self._require_routes()
|
||||
self._roots()
|
||||
write_png(
|
||||
self.output_dir / "side_0_00001_.png",
|
||||
prompt=saved_unlinkable_graph("cat", "dog"),
|
||||
)
|
||||
await self._scan()
|
||||
item, detail, observed = await self._open()
|
||||
status, data = await self._decide(
|
||||
item["id"], self._body(detail, observed, "unresolved")
|
||||
)
|
||||
self.assertEqual(status, 200, data)
|
||||
before = (self._items(), self._decisions())
|
||||
|
||||
await self._scan("all")
|
||||
|
||||
self.assertEqual((self._items(), self._decisions()), before)
|
||||
|
||||
async def test_apply_accepts_an_old_default_choice_the_anchor_names(self):
|
||||
self._require_routes()
|
||||
self._roots()
|
||||
path = self.output_dir / "side_0_00001_.png"
|
||||
write_png(path, prompt=saved_unlinkable_graph("cat", "dog"))
|
||||
with patch.object(review_capture, "_captured_inspection", return_value=None):
|
||||
await self._scan()
|
||||
|
||||
_item, (status, data) = await self._apply("cat")
|
||||
|
||||
self.assertEqual(status, 200, data)
|
||||
self.assertEqual(self._linked_text(path), "cat")
|
||||
|
||||
|
||||
class TestSaverAnchored(InspectionTestCase):
|
||||
def test_only_the_anchor_counts(self):
|
||||
anchored = {"prompt": saved_unlinkable_graph("cat", "dog")}
|
||||
self.assertTrue(review_capture.saver_anchored(anchored, "side_1_00001_.png"))
|
||||
self.assertFalse(review_capture.saver_anchored(anchored, "nope_00001_.png"))
|
||||
self.assertFalse(review_capture.saver_anchored(anchored, None))
|
||||
# One sampler resolves without any name: the anchor decides nothing.
|
||||
single = {"prompt": branch_graph("make it night")}
|
||||
self.assertFalse(review_capture.saver_anchored(single, "branch_0_00001_.png"))
|
||||
@@ -1,112 +0,0 @@
|
||||
"""``sampler_less_prompt``: the one wired text of a graph with no sampler (#252)."""
|
||||
|
||||
import time
|
||||
import unittest
|
||||
|
||||
from utils.prompt_pair_resolver import sampler_less_prompt
|
||||
|
||||
|
||||
def _guided(**nodes):
|
||||
"""``nodes`` plus a guider consuming node ``"1"``."""
|
||||
graph = {"9": {"class_type": "BasicGuider", "inputs": {"conditioning": ["1", 0]}}}
|
||||
graph.update(nodes)
|
||||
return graph
|
||||
|
||||
|
||||
class TestSamplerLessPrompt(unittest.TestCase):
|
||||
def test_not_a_graph_or_too_large_is_absent(self):
|
||||
for graph in (None, [], {}, "text"):
|
||||
self.assertEqual(sampler_less_prompt(graph)["state"], "absent")
|
||||
big = {str(i): {"class_type": "X", "inputs": {}} for i in range(3)}
|
||||
self.assertEqual(sampler_less_prompt(big, max_visits=2)["state"], "absent")
|
||||
|
||||
def test_one_literal_text(self):
|
||||
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": " a fox "}}})
|
||||
self.assertEqual(
|
||||
sampler_less_prompt(graph),
|
||||
{"state": "resolved", "text": "a fox", "texts": ["a fox"]},
|
||||
)
|
||||
|
||||
def test_a_number_literal_is_never_a_prompt(self):
|
||||
for value in (2048, "2048", "1.5", " 7 "):
|
||||
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": value}}})
|
||||
self.assertEqual(sampler_less_prompt(graph)["state"], "absent", value)
|
||||
|
||||
def test_a_long_digit_run_with_a_letter_is_text_and_fast(self):
|
||||
text = "9" * 50_000 + "x"
|
||||
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": text}}})
|
||||
started = time.monotonic()
|
||||
self.assertEqual(sampler_less_prompt(graph)["text"], text)
|
||||
self.assertLess(time.monotonic() - started, 1.0)
|
||||
|
||||
def test_number_like_words_are_prompts(self):
|
||||
for value in ("nan", "inf", "Infinity", "1_000"):
|
||||
graph = _guided(**{"1": {"class_type": "Enc", "inputs": {"text": value}}})
|
||||
self.assertEqual(sampler_less_prompt(graph)["text"], value)
|
||||
|
||||
def test_an_own_node_with_unreadable_text_beside_another_text_is_ambiguous(self):
|
||||
graph = _guided(
|
||||
**{
|
||||
"1": {"class_type": "PromptManager", "inputs": {"text": ["5", 0]}},
|
||||
"5": {"class_type": "LLM", "inputs": {"seed": 3}},
|
||||
"2": {"class_type": "Enc", "inputs": {"text": "<lora:x:1>"}},
|
||||
"3": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
|
||||
}
|
||||
)
|
||||
self.assertEqual(sampler_less_prompt(graph)["state"], "ambiguous")
|
||||
|
||||
def test_unhashable_class_type_is_an_unknown_node_not_a_crash(self):
|
||||
graph = _guided(**{"1": {"class_type": ["x"], "inputs": {"text": "a fox"}}})
|
||||
self.assertEqual(sampler_less_prompt(graph)["text"], "a fox")
|
||||
|
||||
def test_own_negative_node_is_not_authoritative(self):
|
||||
graph = _guided(
|
||||
**{
|
||||
"1": {"class_type": "Enc", "inputs": {"text": "a fox"}},
|
||||
"2": {
|
||||
"class_type": "PromptManagerNegative",
|
||||
"inputs": {"text": "blur"},
|
||||
},
|
||||
"3": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
|
||||
}
|
||||
)
|
||||
found = sampler_less_prompt(graph)
|
||||
self.assertEqual(found["state"], "ambiguous")
|
||||
self.assertEqual(found["texts"], ["a fox", "blur"])
|
||||
|
||||
def test_own_node_text_wins_over_an_untraceable_text(self):
|
||||
graph = _guided(
|
||||
**{
|
||||
"1": {"class_type": "PromptManagerText", "inputs": {"text": "a fox"}},
|
||||
"2": {"class_type": "Enc", "inputs": {"text": ["5", 0]}},
|
||||
"5": {"class_type": "LLM", "inputs": {"seed": 3}},
|
||||
"6": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
|
||||
}
|
||||
)
|
||||
self.assertEqual(sampler_less_prompt(graph)["text"], "a fox")
|
||||
|
||||
def test_two_own_texts_are_ambiguous(self):
|
||||
graph = _guided(
|
||||
**{
|
||||
"1": {"class_type": "PromptManager", "inputs": {"text": "a fox"}},
|
||||
"2": {"class_type": "PromptManager", "inputs": {"text": "a cat"}},
|
||||
"3": {"class_type": "Sink", "inputs": {"c": ["2", 0]}},
|
||||
}
|
||||
)
|
||||
self.assertEqual(sampler_less_prompt(graph)["state"], "ambiguous")
|
||||
|
||||
def test_only_an_untraceable_text_is_ambiguous_with_nothing_to_show(self):
|
||||
graph = _guided(
|
||||
**{
|
||||
"1": {"class_type": "Enc", "inputs": {"text": ["5", 0]}},
|
||||
"5": {"class_type": "LLM", "inputs": {"seed": 3}},
|
||||
}
|
||||
)
|
||||
self.assertEqual(
|
||||
sampler_less_prompt(graph),
|
||||
{"state": "ambiguous", "text": None, "texts": []},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -978,27 +978,6 @@ class TestThumbnailOnCapture(AutoThumbnailCase):
|
||||
(self.output_dir / "thumbnails" / f"{name}_thumb.webp").is_file()
|
||||
)
|
||||
|
||||
async def test_a_worker_refused_at_interpreter_exit_skips_quietly(self):
|
||||
"""#375: refused once the interpreter exits; nothing stays queued."""
|
||||
|
||||
class Refused(threading.Thread):
|
||||
def start(self):
|
||||
raise RuntimeError("can't create new thread at interpreter shutdown")
|
||||
|
||||
namespace = type(sys)("threading_copy")
|
||||
namespace.__dict__.update(vars(threading))
|
||||
namespace.Thread = Refused
|
||||
thumbnailer = self.api._auto_thumbnails()
|
||||
path = make_png(self.output_dir / "late.png", size=(32, 32))
|
||||
with (
|
||||
patch.object(auto, "threading", namespace),
|
||||
self.assertNoLogs("prompt_manager.api", "INFO"),
|
||||
):
|
||||
self.assertEqual(self.api.queue_capture_thumbnail(str(path)), "skipped")
|
||||
self.assertFalse(thumbnailer._worker_running)
|
||||
self.assertEqual(len(thumbnailer._queue), 0)
|
||||
self.assertTrue(thumbnailer.wait_idle(0))
|
||||
|
||||
def test_queue_is_bounded(self):
|
||||
thumbnailer = auto.AutoThumbnailer(
|
||||
lambda *a: {"action": "generated"},
|
||||
|
||||
@@ -709,8 +709,7 @@ class TestRotatedCopyRule(IsolatedLoggerTestCase):
|
||||
self.assertNotEqual(link["size"], os.path.getsize(target))
|
||||
for row in rows.values():
|
||||
self.assertEqual(
|
||||
set(row),
|
||||
{"filename", "path", "size", "modified", "is_main", "kind", "identity"},
|
||||
set(row), {"filename", "path", "size", "modified", "is_main", "kind"}
|
||||
)
|
||||
|
||||
def test_an_unreadable_timestamp_does_not_break_the_scan(self):
|
||||
@@ -764,217 +763,6 @@ class TestRotatedCopyRule(IsolatedLoggerTestCase):
|
||||
self.manager.log_file_entry(name)
|
||||
|
||||
|
||||
class TestClearOnlyWhatWasConfirmed(IsolatedLoggerTestCase):
|
||||
"""#389: Clear all logs empties or deletes no more than its confirmation showed."""
|
||||
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.logger.info("active log line") # the active log exists
|
||||
self.outside = os.path.join(self.tmp.name, "outside")
|
||||
os.makedirs(self.outside)
|
||||
|
||||
def _path(self, name):
|
||||
return os.path.join(self.log_dir, name)
|
||||
|
||||
def _write(self, name, data=b"rotated\n"):
|
||||
with open(self._path(name), "wb") as handle:
|
||||
handle.write(data)
|
||||
|
||||
def _read(self, path):
|
||||
with open(path, "rb") as handle:
|
||||
return handle.read()
|
||||
|
||||
def _snapshot(self):
|
||||
return {name: self._read(self._path(name)) for name in os.listdir(self.log_dir)}
|
||||
|
||||
def _assert_unchanged(self, before, growing="prompt_manager.log"):
|
||||
"""Nothing deleted or emptied (the refusal itself logs to ``growing``)."""
|
||||
after = self._snapshot()
|
||||
self.assertEqual(sorted(after), sorted(before))
|
||||
for name, data in before.items():
|
||||
if name == growing:
|
||||
self.assertTrue(after[name].startswith(data), name)
|
||||
else:
|
||||
self.assertEqual(after[name], data, name)
|
||||
|
||||
def _confirmed(self):
|
||||
"""What the workspace dialog showed: the listed active and rotated rows."""
|
||||
return [
|
||||
{"name": row["filename"], "identity": row["identity"]}
|
||||
for row in self.manager.get_log_files()
|
||||
if row["kind"] in ("active", "rotated")
|
||||
]
|
||||
|
||||
def _file_handler(self):
|
||||
return next(
|
||||
h
|
||||
for h in self.manager.logger.handlers
|
||||
if isinstance(h, logging.FileHandler)
|
||||
)
|
||||
|
||||
def test_listed_identity_is_a_string_that_survives_json_numbers(self):
|
||||
# mtime_ns (~1.7e18) and many inodes exceed 2**53: as JSON numbers a
|
||||
# browser would round them and every confirmed clear would be refused.
|
||||
self._write("prompt_manager.log.1")
|
||||
for row in self.manager.get_log_files():
|
||||
with self.subTest(name=row["filename"]):
|
||||
self.assertIsInstance(row["identity"], str)
|
||||
self.assertRegex(row["identity"], r"^\d+:-?\d+:\d+$")
|
||||
|
||||
def test_a_copy_dated_before_1970_can_be_confirmed(self):
|
||||
self._write("prompt_manager.log.1")
|
||||
try:
|
||||
os.utime(self._path("prompt_manager.log.1"), ns=(0, -(10**9)))
|
||||
except (OSError, OverflowError, ValueError) as error:
|
||||
self.skipTest(f"this filesystem cannot store a time before 1970: {error}")
|
||||
confirmed = self._confirmed()
|
||||
if not any(":-" in row["identity"] for row in confirmed):
|
||||
self.skipTest("the filesystem did not keep the negative time")
|
||||
result = self.manager.truncate_logs(confirmed)
|
||||
self.assertIn("prompt_manager.log.1", result["truncated"])
|
||||
|
||||
def test_a_rollover_after_the_confirmation_deletes_nothing(self):
|
||||
self._write("prompt_manager.log.1", b"one" * 10)
|
||||
self._write("prompt_manager.log.2", b"two" * 10)
|
||||
confirmed = self._confirmed()
|
||||
self.logger.info("written after the confirmation")
|
||||
self._file_handler().doRollover() # .2 -> .3, .1 -> .2, log -> .1
|
||||
before = self._snapshot()
|
||||
with self.assertRaises(logging_config.ClearRefused) as caught:
|
||||
self.manager.truncate_logs(confirmed)
|
||||
self.assertEqual(caught.exception.code, "logs-changed")
|
||||
self._assert_unchanged(before)
|
||||
|
||||
def test_a_copy_that_appeared_after_the_confirmation_is_kept(self):
|
||||
self._write("prompt_manager.log.1", b"one")
|
||||
confirmed = self._confirmed()
|
||||
self._write("prompt_manager.log.5", b"not shown")
|
||||
result = self.manager.truncate_logs(confirmed)
|
||||
self.assertEqual(result["errors"], [])
|
||||
self.assertEqual(
|
||||
sorted(result["truncated"]), ["prompt_manager.log", "prompt_manager.log.1"]
|
||||
)
|
||||
self.assertEqual(
|
||||
sorted(os.listdir(self.log_dir)),
|
||||
["prompt_manager.log", "prompt_manager.log.5"],
|
||||
)
|
||||
|
||||
def _clear(self, confirmed=None):
|
||||
"""Run a clear and return what it raised (None), so the files are
|
||||
asserted first: on the old code the test fails on lost bytes."""
|
||||
try:
|
||||
if confirmed is None:
|
||||
self.manager.truncate_logs() # the legacy page's call
|
||||
else:
|
||||
self.manager.truncate_logs(confirmed)
|
||||
except Exception as error: # noqa: BLE001 - asserted below
|
||||
return error
|
||||
return None
|
||||
|
||||
def test_a_foreign_append_to_the_active_log_is_refused(self):
|
||||
# A second ComfyUI sharing the folder writes the same active log.
|
||||
self._write("prompt_manager.log.1")
|
||||
listings = (None, "listed")
|
||||
for listing in listings:
|
||||
with self.subTest(listed=listing is not None):
|
||||
confirmed = self._confirmed() if listing else None
|
||||
with open(self._path("prompt_manager.log"), "ab") as other:
|
||||
other.write(b"line from another process\n")
|
||||
before = self._snapshot()
|
||||
error = self._clear(confirmed)
|
||||
self._assert_unchanged(before)
|
||||
self.assertEqual(getattr(error, "code", None), "logs-foreign-writer")
|
||||
self.logger.info("our own line") # our stream catches up again
|
||||
|
||||
@unittest.skipIf(os.name == "nt", "Windows cannot rename a log that is open")
|
||||
def test_a_foreign_rotation_of_our_log_is_refused(self):
|
||||
# Another process rolled the shared log over: our handler still writes
|
||||
# the file now named .1, and a new active log is the other's.
|
||||
os.rename(self._path("prompt_manager.log"), self._path("prompt_manager.log.1"))
|
||||
self._write("prompt_manager.log", b"the other process's log\n")
|
||||
before = self._snapshot()
|
||||
with self.assertRaises(logging_config.ClearRefused) as caught:
|
||||
self.manager.truncate_logs()
|
||||
self.assertEqual(caught.exception.code, "logs-foreign-writer")
|
||||
# Our handler still writes the renamed file, so the refusal lands there.
|
||||
self._assert_unchanged(before, growing="prompt_manager.log.1")
|
||||
|
||||
def test_a_copy_that_changes_mid_clear_stops_the_clear(self):
|
||||
# The identity is checked again just before each unlink: another
|
||||
# process rotating while we delete stops the clear there.
|
||||
for n in (1, 2):
|
||||
self._write(f"prompt_manager.log.{n}", b"r" * n)
|
||||
confirmed = self._confirmed()
|
||||
real_unlink = os.unlink
|
||||
calls = []
|
||||
|
||||
def unlink_then_rotate(path, *args, **kwargs):
|
||||
real_unlink(path, *args, **kwargs)
|
||||
if not calls:
|
||||
with open(self._path("prompt_manager.log.2"), "ab") as handle:
|
||||
handle.write(b"rotated in by another process")
|
||||
calls.append(os.path.basename(path))
|
||||
|
||||
active_before = self._read(self._path("prompt_manager.log"))
|
||||
with mock.patch.object(logging_config.os, "unlink", unlink_then_rotate):
|
||||
result = self.manager.truncate_logs(confirmed)
|
||||
self.assertEqual(calls, ["prompt_manager.log.1"])
|
||||
self.assertEqual(result["truncated"], ["prompt_manager.log.1"])
|
||||
self.assertEqual(len(result["errors"]), 1)
|
||||
self.assertIn("prompt_manager.log.2", result["errors"][0])
|
||||
self.assertTrue(os.path.exists(self._path("prompt_manager.log.2")))
|
||||
# The active log is emptied last, so it is kept too.
|
||||
self.assertTrue(
|
||||
self._read(self._path("prompt_manager.log")).startswith(active_before)
|
||||
)
|
||||
|
||||
def _require_hard_link(self, source, link):
|
||||
try:
|
||||
os.link(source, link)
|
||||
except (OSError, NotImplementedError, AttributeError) as error:
|
||||
self.skipTest(f"this filesystem cannot make hard links: {error}")
|
||||
|
||||
def test_a_hard_linked_active_log_is_never_emptied(self):
|
||||
other_name = os.path.join(self.outside, "kept-by-the-user.log")
|
||||
self._require_hard_link(self._path("prompt_manager.log"), other_name)
|
||||
kept = self._read(other_name)
|
||||
self.assertTrue(kept)
|
||||
for listed in (False, True):
|
||||
with self.subTest(listed=listed):
|
||||
error = self._clear(self._confirmed() if listed else None)
|
||||
self.assertTrue(self._read(other_name).startswith(kept))
|
||||
self.assertEqual(getattr(error, "code", None), "logs-hard-link")
|
||||
|
||||
def test_a_hard_linked_active_log_with_file_logging_off_is_kept(self):
|
||||
self.manager.update_config({"file_logging": False})
|
||||
other_name = os.path.join(self.outside, "kept-by-the-user.log")
|
||||
self._require_hard_link(self._path("prompt_manager.log"), other_name)
|
||||
kept = self._read(other_name)
|
||||
error = self._clear()
|
||||
self.assertEqual(self._read(other_name), kept)
|
||||
self.assertEqual(getattr(error, "code", None), "logs-hard-link")
|
||||
|
||||
def test_a_malformed_confirmation_is_refused_before_anything_changes(self):
|
||||
self._write("prompt_manager.log.1")
|
||||
before = self._snapshot()
|
||||
for expected in (
|
||||
"prompt_manager.log.1",
|
||||
[{"name": "prompt_manager.log.1"}],
|
||||
[{"name": "prompt_manager.log.1", "identity": 7}],
|
||||
[{"name": "prompt_manager.log.1", "identity": "1:2"}],
|
||||
[{"name": 1, "identity": "1:2:3"}],
|
||||
[
|
||||
{"name": "prompt_manager.log.1", "identity": "1:2:3"},
|
||||
{"name": "prompt_manager.log.1", "identity": "1:2:3"},
|
||||
],
|
||||
["prompt_manager.log.1"],
|
||||
):
|
||||
with self.subTest(expected=expected):
|
||||
with self.assertRaises(ValueError):
|
||||
self.manager.truncate_logs(expected)
|
||||
self._assert_unchanged(before)
|
||||
|
||||
|
||||
class TestConfiguration(IsolatedLoggerTestCase):
|
||||
def test_in_flight_record_cannot_reopen_retired_file_handler(self) -> None:
|
||||
handler = next(
|
||||
|
||||
@@ -1,26 +1,12 @@
|
||||
"""The System browser hooks only ever reset their own server's config.json (#249 S3)
|
||||
and decide JPEG XL availability whatever the host has installed (#371)."""
|
||||
"""The System browser hooks only ever reset their own server's config.json (#249 S3)."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from aiohttp import web
|
||||
from aiohttp.test_utils import TestClient, TestServer
|
||||
|
||||
_mock_server = mock.MagicMock()
|
||||
_mock_server.PromptServer.instance.routes = mock.MagicMock()
|
||||
sys.modules.setdefault("server", _mock_server)
|
||||
|
||||
from py.api import thumbnail_encode # noqa: E402
|
||||
from tests.browser.workspace_system_fixture import ( # noqa: E402
|
||||
register_system_commands,
|
||||
)
|
||||
from tests.browser.workspace_system_fixture import register_system_commands
|
||||
|
||||
|
||||
class TestWorkspaceSystemFixture(unittest.TestCase):
|
||||
@@ -47,48 +33,6 @@ class TestWorkspaceSystemFixture(unittest.TestCase):
|
||||
register_system_commands(None, None, None, self.root)
|
||||
self.assertEqual(elsewhere.read_bytes(), b'{"real": true}')
|
||||
|
||||
def test_no_encoder_holds_even_when_the_host_has_one(self):
|
||||
"""``{available: false}`` survives the settings read's re-probe (#371)."""
|
||||
config = self.root / "config.json"
|
||||
env = {**os.environ, "PROMPT_MANAGER_CONFIG_PATH": str(config)}
|
||||
api = types.SimpleNamespace()
|
||||
app, routes = web.Application(), web.RouteTableDef()
|
||||
real_detect = thumbnail_encode._detect
|
||||
with mock.patch.dict(os.environ, env, clear=True):
|
||||
reset = register_system_commands(app, routes, api, self.root)
|
||||
app.add_routes(routes)
|
||||
self.addCleanup(thumbnail_encode.reset_encoders)
|
||||
self.addCleanup(setattr, thumbnail_encode, "_detect", real_detect)
|
||||
|
||||
async def scenario():
|
||||
async with TestClient(TestServer(app)) as client:
|
||||
|
||||
async def post(available):
|
||||
response = await client.post(
|
||||
"/__test__/system/jxl-encoder", json={"available": available}
|
||||
)
|
||||
self.assertEqual(response.status, 200)
|
||||
|
||||
await post(False)
|
||||
status = thumbnail_encode.encoder_status()["jxl"]
|
||||
self.assertFalse(status["available"])
|
||||
self.assertIsNone(
|
||||
thumbnail_encode.detect_jxl_encoder(recheck_missing=True)
|
||||
)
|
||||
await post(True)
|
||||
status = thumbnail_encode.encoder_status()["jxl"]
|
||||
self.assertEqual(status["encoder"], "cjxl")
|
||||
await post(False)
|
||||
|
||||
# The host's real detector finds an encoder (Pillow can save JPEG XL).
|
||||
with mock.patch.object(thumbnail_encode, "_pillow_can_save_jxl", lambda: True):
|
||||
asyncio.run(scenario())
|
||||
# Reset gives detection back to the host.
|
||||
reset()
|
||||
self.assertIs(thumbnail_encode._detect, real_detect)
|
||||
status = thumbnail_encode.encoder_status()["jxl"]
|
||||
self.assertEqual(status["encoder"], "pillow")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Vendored
-29
@@ -247,32 +247,3 @@ test('the Python matrix and coverage run the suite through the watchdog runner',
|
||||
assert.doesNotMatch(found.get(id), /unittest discover/, `${id} no longer runs the bare unittest discover`);
|
||||
}
|
||||
});
|
||||
|
||||
// CI hygiene (#352): lint tools are pinned, the bandit hook can fail, the JS job caches npm.
|
||||
const readRepoFile = path => readFileSync(new URL(`../../${path}`, import.meta.url), 'utf8');
|
||||
|
||||
test('black and flake8 are pinned exactly and the lint job installs those pins', () => {
|
||||
const requirements = readRepoFile('requirements-dev.txt');
|
||||
assert.match(requirements, /^black==\d+\.\d+\.\d+$/m, 'black is pinned exactly');
|
||||
assert.match(requirements, /^flake8==\d+\.\d+\.\d+$/m, 'flake8 is pinned exactly');
|
||||
const lint = jobs(readWorkflow('test.yml')).get('lint');
|
||||
assert.match(lint, /pip install \$\(grep -E '\^\(black\|flake8\)==' requirements-dev\.txt\)/);
|
||||
assert.doesNotMatch(lint, /pip install black flake8/, 'lint no longer installs the latest releases');
|
||||
});
|
||||
|
||||
test('the frontend modules job caches npm keyed on the vendor lockfile', () => {
|
||||
const setupNode = steps(jobs(readWorkflow('test.yml')).get('js-test'))
|
||||
.find(step => /uses: actions\/setup-node@v\d+/.test(step));
|
||||
assert.match(setupNode, /^ {10}cache: npm$/m);
|
||||
assert.match(setupNode, /^ {10}cache-dependency-path: tools\/vendor\/package-lock\.json$/m);
|
||||
});
|
||||
|
||||
test('the bandit pre-commit hook fails at the severity CI enforces and passes every file', () => {
|
||||
const quality = readWorkflow('code-quality.yml');
|
||||
assert.match(quality, /^ {8}run: bandit -r \. -lll -q --exclude /m, 'CI fails on high severity');
|
||||
const hook = /- id: bandit\n([\s\S]*?)(?=\n {6}(?:- id:|#)|\n*$)/.exec(readRepoFile('.pre-commit-config.yaml'))?.[1];
|
||||
assert.ok(hook, 'the bandit hook exists');
|
||||
assert.match(hook, /^ {8}entry: bandit -lll -q$/m, 'the hook enforces the same -lll level as CI');
|
||||
assert.doesNotMatch(hook, /true|exit-zero|bash -c/, 'nothing masks the exit code or drops a file');
|
||||
assert.doesNotMatch(hook, /pass_filenames: false/, 'the hook scans the staged files');
|
||||
});
|
||||
|
||||
+1
-98
@@ -1,7 +1,6 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { COLUMNS, CONTROL_EXEMPTIONS, PAGES, REMOVAL_COLUMNS, STATUSES, declaredTitles, duplicateRemovals, idUses, methodUses,
|
||||
parseControl, parseCoverageTable, parseRemovalTable, readCoverageTable, readPage, readRemovalTable, removalProblems,
|
||||
import { COLUMNS, CONTROL_EXEMPTIONS, STATUSES, declaredTitles, parseControl, parseCoverageTable, readCoverageTable,
|
||||
rowProblems } from '../../tools/vendor/legacy-coverage.mjs';
|
||||
|
||||
// TESTING.md "Legacy System capability coverage" (#249 System S12, #329): every legacy System capability names an
|
||||
@@ -115,99 +114,3 @@ test('a covered row names its native control unless it is exempt', () => {
|
||||
assert.deepEqual([row.Status, row['Native control']], ['covered', '—']);
|
||||
}
|
||||
});
|
||||
|
||||
// TESTING.md "Legacy System UI removed by S13" (#330): a legacy element is removed only when its capability is
|
||||
// `covered`, the removal copies that row's native test, and the element is really gone from its page.
|
||||
const COVERAGE_HEADER = '| Capability | Legacy anchor | Owner | Status | Native control | Test |\n| --- | --- | --- | --- | --- | --- |';
|
||||
const REMOVAL_HEADER = '| Capability | Page | Removed ids | Removed methods | Native test |\n| --- | --- | --- | --- | --- |';
|
||||
const FIXTURE_COVERAGE = parseCoverageTable(['### Legacy System capability coverage', '', COVERAGE_HEADER,
|
||||
'| View logs | `admin.html:1` | S12 Logs | covered | `diagnostics` · combobox · "Level" | level filter works (logs.spec.mjs) |',
|
||||
'| Infinite scrolling default | `admin.html:2` | Legacy prompt list | kept in legacy | — | scrolls (list-state.test.js) |',
|
||||
''].join('\n'));
|
||||
const removalRow = cells => parseRemovalTable(['### Legacy System UI removed by S13', '', REMOVAL_HEADER, `| ${cells} |`, '']
|
||||
.join('\n'))[0];
|
||||
const EMPTY_PAGE = () => ({ html: '<button id="kept">Kept</button>', script: 'class Admin {\n kept() {}\n}\n' });
|
||||
|
||||
test('every removed legacy element cites a covered capability and its native test', () => {
|
||||
const coverage = readCoverageTable();
|
||||
const rows = readRemovalTable();
|
||||
for (const row of rows) {
|
||||
const cited = coverage.find(item => item.Capability === row.Capability);
|
||||
assert.ok(cited, `TESTING.md:${row.line} cites "${row.Capability}"`);
|
||||
assert.equal(cited.Status, 'covered', `TESTING.md:${row.line} ${row.Capability}`);
|
||||
assert.equal(row['Native test'], cited.Test, `TESTING.md:${row.line} ${row.Capability}`);
|
||||
assert.ok(Object.keys(PAGES).includes(row.Page), `TESTING.md:${row.line} page ${row.Page}`);
|
||||
}
|
||||
assert.deepEqual(duplicateRemovals(rows), []);
|
||||
assert.deepEqual(REMOVAL_COLUMNS, ['Capability', 'Page', 'Removed ids', 'Removed methods', 'Native test']);
|
||||
});
|
||||
|
||||
test('a removed id or method is gone from its page', () => {
|
||||
const coverage = readCoverageTable();
|
||||
for (const row of readRemovalTable()) {
|
||||
assert.deepEqual(removalProblems(row, coverage, page => readPage(page)), [], `TESTING.md:${row.line} ${row.Capability}`);
|
||||
}
|
||||
// Each way a page can still hold an id or a method is found; a call through another object, or a longer name, is not
|
||||
// a definition.
|
||||
const page = {
|
||||
html: '<div id="logsModal"></div><button onclick="window.admin.closeLogs()">x</button>',
|
||||
script: [
|
||||
"document.getElementById('logLevel').value;",
|
||||
"this.showModal('diagnosticsModal');",
|
||||
'<span data-id="logsContainer"></span>',
|
||||
'document.querySelector("#logLimit option");',
|
||||
"const backdrops = ['settingsModal', 'duplicatesModal'];",
|
||||
'class Admin {',
|
||||
' async refreshLogs(level) {',
|
||||
' }',
|
||||
'} displayLogs(logs) {',
|
||||
' }',
|
||||
'this.loadLogStats();',
|
||||
'// downloadLogs() {',
|
||||
'other.formatBytesLater(1);',
|
||||
].join('\n'),
|
||||
};
|
||||
assert.deepEqual(idUses('logsModal', page), ['id="logsModal" in the markup']);
|
||||
assert.deepEqual(idUses('logLevel', page), ["getElementById('logLevel')"]);
|
||||
assert.deepEqual(idUses('logLimit', page), ["querySelector('#logLimit')"]);
|
||||
assert.deepEqual(idUses('duplicatesModal', page), ["'duplicatesModal' in a modal list"]);
|
||||
assert.deepEqual(idUses('logLevelX', page), []);
|
||||
assert.deepEqual(idUses('diagnosticsModal', page), ["a …Modal('diagnosticsModal') call"]);
|
||||
assert.deepEqual(idUses('logsContainer', page), []);
|
||||
assert.deepEqual(methodUses('closeLogs', page), ['.closeLogs( called in the markup']);
|
||||
assert.deepEqual(methodUses('refreshLogs', page), ['refreshLogs() defined']);
|
||||
assert.deepEqual(methodUses('displayLogs', page), ['displayLogs() defined']);
|
||||
assert.deepEqual(methodUses('loadLogStats', page), ['.loadLogStats( called']);
|
||||
assert.deepEqual(methodUses('downloadLogs', page), []);
|
||||
assert.deepEqual(methodUses('formatBytes', page), []);
|
||||
// One id or method is listed on one row per page.
|
||||
const twice = [removalRow('View logs | admin | `#logsModal` | — | level filter works (logs.spec.mjs)'),
|
||||
{ ...removalRow('View logs | admin | `#logsModal` | — | level filter works (logs.spec.mjs)'), line: 9 }];
|
||||
assert.match(duplicateRemovals(twice).join(), /admin #logsModal is listed on rows/);
|
||||
assert.deepEqual(removalProblems(removalRow('View logs | admin | `logsModal`, `#ok` | `close` | level filter works (logs.spec.mjs)'),
|
||||
FIXTURE_COVERAGE, EMPTY_PAGE), ['malformed entry "logsModal"', 'malformed entry "close"']);
|
||||
});
|
||||
|
||||
test('a removal citing a kept-in-legacy capability fails', () => {
|
||||
const row = removalRow('Infinite scrolling default | admin | `#infiniteScrollDefault` | — | scrolls (list-state.test.js)');
|
||||
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, EMPTY_PAGE), ['cites a kept in legacy capability']);
|
||||
const unknown = removalRow('Something else | admin | — | — | scrolls (list-state.test.js)');
|
||||
assert.deepEqual(removalProblems(unknown, FIXTURE_COVERAGE, EMPTY_PAGE), ['cites no single coverage row "Something else"']);
|
||||
});
|
||||
|
||||
test('a removal whose native test differs fails', () => {
|
||||
const row = removalRow('View logs | admin | `#logsModal` | `showLogsModal()` | another title (logs.spec.mjs)');
|
||||
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, EMPTY_PAGE),
|
||||
['its native test differs from the coverage row\'s "level filter works (logs.spec.mjs)"']);
|
||||
assert.deepEqual(removalProblems({ ...row, 'Native test': 'level filter works (logs.spec.mjs)' }, FIXTURE_COVERAGE, EMPTY_PAGE), []);
|
||||
});
|
||||
|
||||
test('a removed id still on its page fails', () => {
|
||||
const row = removalRow('View logs | admin | `#logsModal` | `showLogsModal()` | level filter works (logs.spec.mjs)');
|
||||
const page = () => ({ html: '<div id="logsModal" class="hidden"></div>', script: 'class Admin {\n showLogsModal() {\n }\n}\n' });
|
||||
assert.deepEqual(removalProblems(row, FIXTURE_COVERAGE, page), [
|
||||
'#logsModal is still on the admin page: id="logsModal" in the markup',
|
||||
'showLogsModal() is still on the admin page: showLogsModal() defined',
|
||||
]);
|
||||
assert.deepEqual(removalProblems({ ...row, Page: 'metadata' }, FIXTURE_COVERAGE, page), ['unknown page "metadata"']);
|
||||
});
|
||||
|
||||
Vendored
+8
-140
@@ -42,30 +42,25 @@ function cells(line) {
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The rows, keyed by `columns`, of the first Markdown table under `heading` (`label` names it in errors). */
|
||||
function parseTable(markdown, heading, columns, label) {
|
||||
/** The table's rows as objects keyed by COLUMNS, from the section that starts at HEADING. */
|
||||
export function parseCoverageTable(markdown) {
|
||||
const lines = markdown.split('\n');
|
||||
const start = lines.findIndex(line => line.trim() === heading);
|
||||
if (start === -1) throw new Error(`TESTING.md has no "${heading}" section`);
|
||||
const start = lines.findIndex(line => line.trim() === HEADING);
|
||||
if (start === -1) throw new Error(`TESTING.md has no "${HEADING}" section`);
|
||||
const header = lines.findIndex((line, index) => index > start && line.trim().startsWith('|'));
|
||||
if (header === -1 || lines.slice(start + 1, header).some(line => /^#{1,3} /.test(line))) {
|
||||
throw new Error(`The ${label} section has no table`);
|
||||
throw new Error('The coverage section has no table');
|
||||
}
|
||||
const found = cells(lines[header]);
|
||||
if (found.join('|') !== columns.join('|')) throw new Error(`The ${label} table's columns are ${found.join(', ')}`);
|
||||
const columns = cells(lines[header]);
|
||||
if (columns.join('|') !== COLUMNS.join('|')) throw new Error(`The coverage table's columns are ${columns.join(', ')}`);
|
||||
const rows = [];
|
||||
for (let index = header + 2; index < lines.length && lines[index].trim().startsWith('|'); index += 1) {
|
||||
const values = cells(lines[index]);
|
||||
rows.push({ line: index + 1, ...Object.fromEntries(columns.map((column, at) => [column, values[at] ?? ''])) });
|
||||
rows.push({ line: index + 1, ...Object.fromEntries(COLUMNS.map((column, at) => [column, values[at] ?? ''])) });
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
/** The table's rows as objects keyed by COLUMNS, from the section that starts at HEADING. */
|
||||
export function parseCoverageTable(markdown) {
|
||||
return parseTable(markdown, HEADING, COLUMNS, 'coverage');
|
||||
}
|
||||
|
||||
/**
|
||||
* A Native control cell: `` `<section>` · <role> · "<name>" ``, optionally ` · in "<row title>"` when the name is shared
|
||||
* by several rows of the section → {section, role, name, row}; `—` → null; else it throws.
|
||||
@@ -218,130 +213,3 @@ export function rowProblems(row, { root = ROOT, locate = file => testFile(file,
|
||||
export function readCoverageTable(root = ROOT) {
|
||||
return parseCoverageTable(readFileSync(join(root, 'TESTING.md'), 'utf8'));
|
||||
}
|
||||
|
||||
// ── Legacy System UI removed by S13 (#330) ──────────────────────────────────────────────────────────────────────────
|
||||
// TESTING.md "Legacy System UI removed by S13": one row per covered capability whose legacy UI S13 removed, with the
|
||||
// element ids and methods that went. A row must cite a `covered` coverage row and copy its Test cell, and none of its
|
||||
// ids or methods may still be on its page.
|
||||
export const REMOVAL_HEADING = '### Legacy System UI removed by S13';
|
||||
export const REMOVAL_COLUMNS = Object.freeze(['Capability', 'Page', 'Removed ids', 'Removed methods', 'Native test']);
|
||||
/** Each legacy page's markup and script. */
|
||||
export const PAGES = Object.freeze({
|
||||
admin: Object.freeze({ html: 'web/admin.html', script: 'web/js/admin.js' }),
|
||||
gallery: Object.freeze({ html: 'web/gallery.html', script: 'web/js/gallery.js' }),
|
||||
});
|
||||
|
||||
/** The removal table's rows, from the section that starts at REMOVAL_HEADING. */
|
||||
export function parseRemovalTable(markdown) {
|
||||
return parseTable(markdown, REMOVAL_HEADING, REMOVAL_COLUMNS, 'removal');
|
||||
}
|
||||
|
||||
/** The removal table read from TESTING.md. */
|
||||
export function readRemovalTable(root = ROOT) {
|
||||
return parseRemovalTable(readFileSync(join(root, 'TESTING.md'), 'utf8'));
|
||||
}
|
||||
|
||||
/** A Removed ids or Removed methods cell → its entries without backticks (`—` → none). */
|
||||
export function listCell(cell) {
|
||||
if (cell.trim() === NONE) return [];
|
||||
return cell.split(',').map(entry => entry.trim().replace(/^`|`$/g, '')).filter(Boolean);
|
||||
}
|
||||
|
||||
/** The removed ids of a row (without `#`) and its malformed entries. */
|
||||
function removedIds(row) {
|
||||
const entries = listCell(row['Removed ids']);
|
||||
return { ids: entries.filter(entry => /^#[\w-]+$/.test(entry)).map(entry => entry.slice(1)),
|
||||
bad: entries.filter(entry => !/^#[\w-]+$/.test(entry)) };
|
||||
}
|
||||
|
||||
/** The removed methods of a row (without `()`) and its malformed entries. */
|
||||
function removedMethods(row) {
|
||||
const entries = listCell(row['Removed methods']);
|
||||
return { names: entries.filter(entry => /^[A-Za-z_$][\w$]*\(\)$/.test(entry)).map(entry => entry.slice(0, -2)),
|
||||
bad: entries.filter(entry => !/^[A-Za-z_$][\w$]*\(\)$/.test(entry)) };
|
||||
}
|
||||
|
||||
const escapeRegExp = text => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
|
||||
/**
|
||||
* Where `id` is still on a page: `id="<id>"` in its markup or script; in its script a lookup, a `…Modal("<id>")` call
|
||||
* (`showModal`, `hideModal`) or a modal-list entry.
|
||||
*/
|
||||
export function idUses(id, { html, script }) {
|
||||
const name = escapeRegExp(id);
|
||||
const code = stripJsComments(script);
|
||||
const uses = [];
|
||||
// `id=`, not `data-id=`: the attribute starts after a space or a tag's `<name`.
|
||||
const attribute = new RegExp(`(?:^|\\s)id=["']${name}["']`, 'm');
|
||||
if (attribute.test(html)) uses.push(`id="${id}" in the markup`);
|
||||
if (attribute.test(code)) uses.push(`id="${id}" in the script`);
|
||||
if (new RegExp(`getElementById\\(\\s*["'\`]${name}["'\`]\\s*\\)`).test(code)) uses.push(`getElementById('${id}')`);
|
||||
if (new RegExp(`querySelector(?:All)?\\(\\s*["'\`]#${name}\\b`).test(code)) uses.push(`querySelector('#${id}')`);
|
||||
if (new RegExp(`\\w*Modal\\(\\s*["'\`]${name}["'\`]`).test(code)) uses.push(`a …Modal('${id}') call`);
|
||||
// A string list of modal ids (the gallery's backdrop list): the entry next to another `…Modal` entry.
|
||||
for (const list of code.matchAll(/\[[^\[\]]*\]/g)) {
|
||||
const entry = new RegExp(`["']${name}["']`);
|
||||
if (entry.test(list[0]) && /["'][\w-]*Modal["']/.test(list[0].replace(entry, ''))) {
|
||||
uses.push(`'${id}' in a modal list`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
return uses;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where method `name` is still on a page: a definition in its script, or a `.<name>(` call (`this.`, `window.admin.`)
|
||||
* in its script or markup.
|
||||
*/
|
||||
export function methodUses(name, { html, script }) {
|
||||
const method = escapeRegExp(name);
|
||||
const code = stripJsComments(script);
|
||||
const uses = [];
|
||||
if (new RegExp(`(?:^|[\\s;}])(?:async\\s+)?${method}\\s*\\([^)]*\\)\\s*\\{`, 'm').test(code)) uses.push(`${name}() defined`);
|
||||
if (new RegExp(`\\.${method}\\s*\\(`).test(code)) uses.push(`.${name}( called`);
|
||||
if (new RegExp(`\\.${method}\\s*\\(`).test(html)) uses.push(`.${name}( called in the markup`);
|
||||
return uses;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every problem of one removal row (empty: sound). `coverage` is the coverage table's rows; `readPage(page)` returns
|
||||
* that page's `{html, script}`.
|
||||
*/
|
||||
export function removalProblems(row, coverage, readPage) {
|
||||
const problems = [];
|
||||
const cited = coverage.filter(item => item.Capability === row.Capability);
|
||||
if (cited.length !== 1) problems.push(`cites no single coverage row "${row.Capability}"`);
|
||||
else if (cited[0].Status !== 'covered') problems.push(`cites a ${cited[0].Status} capability`);
|
||||
else if (cited[0].Test !== row['Native test']) problems.push(`its native test differs from the coverage row's "${cited[0].Test}"`);
|
||||
if (!Object.hasOwn(PAGES, row.Page)) {
|
||||
problems.push(`unknown page "${row.Page}"`);
|
||||
return problems;
|
||||
}
|
||||
const { ids, bad: badIds } = removedIds(row);
|
||||
const { names, bad: badNames } = removedMethods(row);
|
||||
for (const entry of [...badIds, ...badNames]) problems.push(`malformed entry "${entry}"`);
|
||||
const page = readPage(row.Page);
|
||||
for (const id of ids) for (const use of idUses(id, page)) problems.push(`#${id} is still on the ${row.Page} page: ${use}`);
|
||||
for (const name of names) for (const use of methodUses(name, page)) problems.push(`${name}() is still on the ${row.Page} page: ${use}`);
|
||||
return problems;
|
||||
}
|
||||
|
||||
/** Ids and methods listed on more than one row of the same page. */
|
||||
export function duplicateRemovals(rows) {
|
||||
const seen = new Map();
|
||||
const duplicates = [];
|
||||
for (const row of rows) {
|
||||
for (const entry of [...listCell(row['Removed ids']), ...listCell(row['Removed methods'])]) {
|
||||
const key = `${row.Page} ${entry}`;
|
||||
if (seen.has(key)) duplicates.push(`${key} is listed on rows ${seen.get(key)} and ${row.line}`);
|
||||
else seen.set(key, row.line);
|
||||
}
|
||||
}
|
||||
return duplicates;
|
||||
}
|
||||
|
||||
/** A page's markup and script from the repository. */
|
||||
export function readPage(page, root = ROOT) {
|
||||
const files = PAGES[page];
|
||||
return { html: readFileSync(join(root, files.html), 'utf8'), script: readFileSync(join(root, files.script), 'utf8') };
|
||||
}
|
||||
|
||||
+3
-13
@@ -35,7 +35,6 @@ from watchdog.events import FileSystemEventHandler
|
||||
|
||||
from .hashing import generate_prompt_hash
|
||||
from .metadata_extractor import ComfyUIMetadataExtractor
|
||||
from .parallel import note_refusal, refused_at_shutdown
|
||||
from .logging_config import get_logger
|
||||
from .prompt_graph import resolve_text, run_prompt_nodes
|
||||
from .prompt_pair_resolver import resolve_prompt_pair
|
||||
@@ -195,9 +194,8 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
and once its size has stopped changing.
|
||||
|
||||
Returns:
|
||||
False when the path is already pending, the pending set is full,
|
||||
the handler is closed or the interpreter is exiting (its worker
|
||||
cannot start)
|
||||
False when the path is already pending, the pending set is full or
|
||||
the handler is closed
|
||||
"""
|
||||
with self._pending_lock:
|
||||
if self._closed:
|
||||
@@ -213,15 +211,7 @@ class ImageGenerationHandler(FileSystemEventHandler):
|
||||
return False
|
||||
due = time.monotonic() + self.processing_delay
|
||||
self._pending[image_path] = (prompt_snapshot, due)
|
||||
try:
|
||||
self._ensure_worker()
|
||||
except RuntimeError as error:
|
||||
# No worker will ever take it: never leave it pending (#375).
|
||||
self._pending.pop(image_path, None)
|
||||
if not refused_at_shutdown(error):
|
||||
raise
|
||||
note_refusal(self.logger, "The image worker")
|
||||
return False
|
||||
self._ensure_worker()
|
||||
self._work.put(image_path)
|
||||
return True
|
||||
|
||||
|
||||
+53
-239
@@ -39,35 +39,6 @@ class RotatedChanged(Exception):
|
||||
self.name = name
|
||||
|
||||
|
||||
# Clear all logs (#389): why it changed nothing, as the API and the pages say it.
|
||||
CLEAR_CHANGED = (
|
||||
"The log files changed since you confirmed; nothing was cleared."
|
||||
" Read the list again and clear again."
|
||||
)
|
||||
CLEAR_FOREIGN_WRITER = (
|
||||
"Another program is writing to prompt_manager.log, such as a second ComfyUI"
|
||||
" sharing this log folder; nothing was cleared. Stop it and clear again."
|
||||
)
|
||||
CLEAR_HARD_LINK = (
|
||||
"prompt_manager.log has another name (a hard link); emptying it would empty"
|
||||
" that file too, so nothing was cleared. Remove the other name, then clear"
|
||||
" again."
|
||||
)
|
||||
# A confirmed row's identity: "<size>:<mtime_ns>:<ino>", a string because
|
||||
# mtime_ns and many inodes are larger than a browser's exact integers.
|
||||
# mtime_ns is negative for a time before 1970 (a restored archive, say).
|
||||
_IDENTITY = re.compile(r"([0-9]{1,20}):(-?[0-9]{1,20}):([0-9]{1,40})")
|
||||
MAX_CONFIRMED_ROWS = 10000
|
||||
|
||||
|
||||
class ClearRefused(Exception):
|
||||
"""Clear all logs changed nothing; ``code`` names why, the message says it."""
|
||||
|
||||
def __init__(self, code: str, message: str):
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
def _name_key(name: str) -> str:
|
||||
"""How a name compares: case-folded where the filesystem folds it (Windows)."""
|
||||
return os.path.normcase(name)
|
||||
@@ -116,37 +87,35 @@ def scan_log_folder(log_dir) -> List[Dict[str, Any]]:
|
||||
be read is left out. Never logs: the cleanup calls it while it holds the
|
||||
file handler's lock.
|
||||
"""
|
||||
rows = []
|
||||
try:
|
||||
names = [entry.name for entry in os.scandir(log_dir)]
|
||||
except OSError:
|
||||
return []
|
||||
rows = (scan_log_entry(log_dir, name) for name in names)
|
||||
return [row for row in rows if row is not None]
|
||||
|
||||
|
||||
def scan_log_entry(log_dir, name) -> Optional[Dict[str, Any]]:
|
||||
"""One entry's scan row (``lstat``), or None when it cannot be read."""
|
||||
path = os.path.join(log_dir, name)
|
||||
try:
|
||||
st = os.lstat(path)
|
||||
kind = _kind(name, path, st)
|
||||
except OSError:
|
||||
return None
|
||||
try:
|
||||
modified = datetime.fromtimestamp(st.st_mtime).isoformat()
|
||||
except (OSError, OverflowError, ValueError): # a time before 1970 on Windows
|
||||
modified = ""
|
||||
return {
|
||||
"name": name,
|
||||
"kind": kind,
|
||||
"size": st.st_size,
|
||||
"mtime_ns": st.st_mtime_ns,
|
||||
"ino": st.st_ino,
|
||||
"modified": modified,
|
||||
"_dir": stat.S_ISDIR(st.st_mode) and kind != "link",
|
||||
"_regular": stat.S_ISREG(st.st_mode) and kind != "link",
|
||||
"_nlink": st.st_nlink,
|
||||
}
|
||||
return rows
|
||||
for name in names:
|
||||
path = os.path.join(log_dir, name)
|
||||
try:
|
||||
st = os.lstat(path)
|
||||
kind = _kind(name, path, st)
|
||||
except OSError:
|
||||
continue
|
||||
try:
|
||||
modified = datetime.fromtimestamp(st.st_mtime).isoformat()
|
||||
except (OSError, OverflowError, ValueError): # a time before 1970 on Windows
|
||||
modified = ""
|
||||
rows.append(
|
||||
{
|
||||
"name": name,
|
||||
"kind": kind,
|
||||
"size": st.st_size,
|
||||
"mtime_ns": st.st_mtime_ns,
|
||||
"ino": st.st_ino,
|
||||
"modified": modified,
|
||||
"_dir": stat.S_ISDIR(st.st_mode) and kind != "link",
|
||||
"_regular": stat.S_ISREG(st.st_mode) and kind != "link",
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def public_row(row: Dict[str, Any]) -> Dict[str, Any]:
|
||||
@@ -158,37 +127,6 @@ def rotated_sort_key(row: Dict[str, Any]) -> int:
|
||||
return _rotated_number(row["name"]) or 0
|
||||
|
||||
|
||||
def file_identity(row: Dict[str, Any]) -> str:
|
||||
"""A listed file's identity as the client sends it back: size, mtime_ns, ino."""
|
||||
return f"{row['size']}:{row['mtime_ns']}:{row['ino']}"
|
||||
|
||||
|
||||
def parse_confirmed(expected) -> List[Dict[str, Any]]:
|
||||
"""The rows a Clear all logs confirmation showed, as review rows.
|
||||
|
||||
``expected`` is a list of ``{name, identity}`` objects from
|
||||
``/logs/files``; anything else raises ValueError.
|
||||
"""
|
||||
if not isinstance(expected, list) or len(expected) > MAX_CONFIRMED_ROWS:
|
||||
raise ValueError("expected must be a list of listed log files")
|
||||
rows, seen = [], set()
|
||||
for item in expected:
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError("each expected file must be an object")
|
||||
name, identity = item.get("name"), item.get("identity")
|
||||
if not isinstance(name, str) or not isinstance(identity, str):
|
||||
raise ValueError("each expected file needs a name and an identity")
|
||||
match = _IDENTITY.fullmatch(identity)
|
||||
if match is None:
|
||||
raise ValueError("an expected file's identity is not size:mtime_ns:ino")
|
||||
size, mtime_ns, ino = (int(part) for part in match.groups())
|
||||
if _name_key(name) in seen:
|
||||
raise ValueError("an expected file is listed twice")
|
||||
seen.add(_name_key(name))
|
||||
rows.append({"name": name, "size": size, "mtime_ns": mtime_ns, "ino": ino})
|
||||
return rows
|
||||
|
||||
|
||||
def log_file_entry(log_dir, name) -> Path:
|
||||
"""The path of a downloadable log file named ``name`` (blocking).
|
||||
|
||||
@@ -219,11 +157,6 @@ def _stream_is(stream, ino: int) -> bool:
|
||||
return not (ino and open_ino and open_ino != ino)
|
||||
|
||||
|
||||
def _stream_ino_is(opened: os.stat_result, ino: int) -> bool:
|
||||
"""Whether a stream's ``fstat`` is the file with inode ``ino`` (0: unknown)."""
|
||||
return not (ino and opened.st_ino and opened.st_ino != ino)
|
||||
|
||||
|
||||
def _error_text(error: BaseException) -> str:
|
||||
"""An OSError's reason and file name, without the folder it is in."""
|
||||
if isinstance(error, OSError):
|
||||
@@ -499,7 +432,6 @@ class PromptManagerLogger:
|
||||
"modified": row["modified"],
|
||||
"is_main": row["kind"] == "active",
|
||||
"kind": row["kind"],
|
||||
"identity": file_identity(row),
|
||||
}
|
||||
)
|
||||
|
||||
@@ -598,14 +530,8 @@ class PromptManagerLogger:
|
||||
finally:
|
||||
handler.release()
|
||||
if not handled:
|
||||
flags = os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0)
|
||||
fd = os.open(str(log_file), flags)
|
||||
try:
|
||||
if os.fstat(fd).st_nlink > 1: # emptying it would empty the other
|
||||
raise ClearRefused("logs-hard-link", CLEAR_HARD_LINK)
|
||||
os.ftruncate(fd, 0)
|
||||
finally:
|
||||
os.close(fd)
|
||||
flags = os.O_WRONLY | os.O_TRUNC | getattr(os, "O_NOFOLLOW", 0)
|
||||
os.close(os.open(str(log_file), flags))
|
||||
|
||||
def delete_rotated(self, expected: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Delete exactly the reviewed rotated copies, or none of them (blocking).
|
||||
@@ -656,126 +582,47 @@ class PromptManagerLogger:
|
||||
)
|
||||
return {"removed": removed, "failed": failed}
|
||||
|
||||
def _active_streams(self, active: Optional[Dict[str, Any]]):
|
||||
"""``fstat`` and offset of every handler stream writing the active name."""
|
||||
target = os.path.normcase(
|
||||
os.path.realpath(os.path.join(self.log_dir, ACTIVE_LOG_NAME))
|
||||
)
|
||||
for handler in self._file_handlers():
|
||||
stream = getattr(handler, "stream", None)
|
||||
base = getattr(handler, "baseFilename", None)
|
||||
if stream is None or base is None:
|
||||
continue
|
||||
if os.path.normcase(os.path.realpath(base)) != target:
|
||||
continue
|
||||
try:
|
||||
stream.flush()
|
||||
fd = stream.fileno()
|
||||
yield os.fstat(fd), os.lseek(fd, 0, os.SEEK_CUR)
|
||||
except (OSError, ValueError, AttributeError):
|
||||
continue
|
||||
def truncate_logs(self) -> Dict[str, Any]:
|
||||
"""Empty the active log and delete its rotated copies (Rule R).
|
||||
|
||||
def _check_clear_is_safe(self, rows, active) -> None:
|
||||
"""Refuse a clear that would empty or delete what nobody confirmed.
|
||||
|
||||
Called under the files lock and every handler lock, so nothing of this
|
||||
process writes or rolls over meanwhile; what changes is another
|
||||
program's doing. Our handler's stream is flushed: a file larger than
|
||||
the stream's own offset was appended to by someone else, and a stream
|
||||
whose file is now a rotated copy was rolled over by someone else.
|
||||
Either is a second writer (another ComfyUI sharing the folder). This
|
||||
is best effort, not a lock: writes before our own last write, or any
|
||||
write while this process has file logging off, go unseen. An
|
||||
active log with more than one name (a hard link) is never emptied: the
|
||||
other name would be emptied too.
|
||||
"""
|
||||
rotated = {row["ino"] for row in rows if row["kind"] == "rotated"}
|
||||
rotated.discard(0)
|
||||
nlink = active["_nlink"] if active is not None else 1
|
||||
for opened, offset in self._active_streams(active):
|
||||
if (opened.st_ino and opened.st_ino in rotated) or opened.st_size > offset:
|
||||
raise ClearRefused("logs-foreign-writer", CLEAR_FOREIGN_WRITER)
|
||||
if active is not None and _stream_ino_is(opened, active["ino"]):
|
||||
nlink = max(nlink, opened.st_nlink)
|
||||
if nlink > 1:
|
||||
raise ClearRefused("logs-hard-link", CLEAR_HARD_LINK)
|
||||
|
||||
@staticmethod
|
||||
def _confirmed_targets(confirmed, current, active) -> List[Dict[str, Any]]:
|
||||
"""The rotated copies a confirmation listed, each as it was listed.
|
||||
|
||||
Refuses (``logs-changed``) when any listed copy is missing or differs
|
||||
in kind, size, ``mtime_ns`` or inode, or when the listed active log is
|
||||
now another file (it was rolled over). A copy nobody listed is kept.
|
||||
"""
|
||||
targets = []
|
||||
for reviewed in confirmed:
|
||||
key = _name_key(reviewed["name"])
|
||||
row = current.get(key)
|
||||
if key == _name_key(ACTIVE_LOG_NAME):
|
||||
ino = reviewed["ino"]
|
||||
if active is None or (ino and active["ino"] and active["ino"] != ino):
|
||||
raise ClearRefused("logs-changed", CLEAR_CHANGED)
|
||||
continue
|
||||
if _rotated_number(reviewed["name"]) is None:
|
||||
continue # never deleted, listed or not
|
||||
if not _matches_review(row, reviewed):
|
||||
raise ClearRefused("logs-changed", CLEAR_CHANGED)
|
||||
targets.append(row)
|
||||
return targets
|
||||
|
||||
def truncate_logs(self, expected=None) -> Dict[str, Any]:
|
||||
"""Empty the active log and delete its rotated copies (Rule R, #389).
|
||||
|
||||
``expected`` is what the confirmation showed: ``{name, identity}``
|
||||
rows of ``/logs/files`` (``parse_confirmed``; ValueError when
|
||||
malformed). Then only the listed rotated copies are deleted, and only
|
||||
when each still has the listed size, ``mtime_ns`` and inode and the
|
||||
active log is still the listed file; otherwise ClearRefused
|
||||
(``logs-changed``) with nothing changed. Without it (the legacy admin
|
||||
page, whose confirmation lists nothing) every rotated copy present now
|
||||
is deleted.
|
||||
|
||||
Either way, a second writer of the active log (``logs-foreign-writer``)
|
||||
or an active log with a hard link (``logs-hard-link``) refuses the
|
||||
whole clear first. Each copy's identity is checked again just before
|
||||
its unlink; one that changed (another process rolling over meanwhile)
|
||||
stops the clear there, keeping it, the rest and the active log, which
|
||||
is emptied last. Every other file in the log folder, and every link,
|
||||
is kept. Also clears the memory buffer. Runs under ``_files_lock`` and
|
||||
the file handlers' locks and logs only once they are released.
|
||||
Every other file in the log folder, and every link, is kept. Also
|
||||
clears the memory buffer. Runs under ``_files_lock`` and the file
|
||||
handlers' locks, so no rollover recreates a just-deleted copy, and
|
||||
logs only once they are released.
|
||||
|
||||
Returns:
|
||||
Dictionary with:
|
||||
- truncated: List of successfully truncated files
|
||||
- errors: List of error messages for failed operations
|
||||
"""
|
||||
confirmed = None if expected is None else parse_confirmed(expected)
|
||||
results = {"truncated": [], "errors": []}
|
||||
refused = None
|
||||
|
||||
with self._files_lock:
|
||||
held = self._acquire_file_handlers()
|
||||
try:
|
||||
rows = scan_log_folder(self.log_dir)
|
||||
current = {_name_key(row["name"]): row for row in rows}
|
||||
active = next((row for row in rows if row["kind"] == "active"), None)
|
||||
try:
|
||||
if confirmed is None:
|
||||
targets = [row for row in rows if row["kind"] == "rotated"]
|
||||
else:
|
||||
targets = self._confirmed_targets(confirmed, current, active)
|
||||
self._check_clear_is_safe(rows, active)
|
||||
except ClearRefused as error:
|
||||
refused = error
|
||||
else:
|
||||
self._clear(sorted(targets, key=rotated_sort_key), active, results)
|
||||
active = [row for row in rows if row["kind"] == "active"]
|
||||
rotated = sorted(
|
||||
(row for row in rows if row["kind"] == "rotated"),
|
||||
key=rotated_sort_key,
|
||||
)
|
||||
for row in active + rotated:
|
||||
path = os.path.join(self.log_dir, row["name"])
|
||||
try:
|
||||
if row["kind"] == "active":
|
||||
self._truncate_main_log(Path(path), row["ino"])
|
||||
else:
|
||||
os.unlink(path)
|
||||
results["truncated"].append(row["name"])
|
||||
except OSError as error:
|
||||
results["errors"].append(
|
||||
f"Failed to truncate {row['name']}: {_error_text(error)}"
|
||||
)
|
||||
with self._buffer_lock:
|
||||
self._log_buffer.clear()
|
||||
finally:
|
||||
self._release_handlers(held)
|
||||
|
||||
if refused is not None:
|
||||
self.logger.warning("Clear all logs refused (%s)", refused.code)
|
||||
raise refused
|
||||
if results["truncated"]:
|
||||
self.logger.info("Truncated log files: %s", ", ".join(results["truncated"]))
|
||||
for error_msg in results["errors"]:
|
||||
@@ -783,39 +630,6 @@ class PromptManagerLogger:
|
||||
|
||||
return results
|
||||
|
||||
def _clear(self, targets, active, results) -> None:
|
||||
"""Delete ``targets`` (each re-checked) then empty ``active`` (locks held)."""
|
||||
for reviewed in targets:
|
||||
name = reviewed["name"]
|
||||
if not _matches_review(scan_log_entry(self.log_dir, name), reviewed):
|
||||
results["errors"].append(
|
||||
f"{name} changed while clearing (another program may be writing"
|
||||
" to this log folder); it, the copies after it and the active"
|
||||
" log were kept"
|
||||
)
|
||||
return
|
||||
try:
|
||||
os.unlink(os.path.join(self.log_dir, name))
|
||||
results["truncated"].append(name)
|
||||
except OSError as error:
|
||||
results["errors"].append(
|
||||
f"Failed to truncate {name}: {_error_text(error)}"
|
||||
)
|
||||
if active is not None:
|
||||
try:
|
||||
self._truncate_main_log(
|
||||
Path(self.log_dir, active["name"]), active["ino"]
|
||||
)
|
||||
results["truncated"].append(active["name"])
|
||||
except ClearRefused as error:
|
||||
results["errors"].append(str(error))
|
||||
except OSError as error:
|
||||
results["errors"].append(
|
||||
f"Failed to truncate {active['name']}: {_error_text(error)}"
|
||||
)
|
||||
with self._buffer_lock:
|
||||
self._log_buffer.clear()
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]):
|
||||
"""Update logging configuration.
|
||||
|
||||
|
||||
+2
-67
@@ -5,51 +5,17 @@ a CUDA-initialised, multi-threaded process is unsafe and the spawn start
|
||||
method would re-import ComfyUI's main module in every child. Pillow releases
|
||||
the GIL while decoding, resizing and encoding, so threads still scale close
|
||||
to linearly for both jobs.
|
||||
|
||||
Interpreter exit (#375): ``threading._shutdown`` shuts every thread pool
|
||||
(``submit`` then raises "cannot schedule new futures after interpreter
|
||||
shutdown") before it joins the non-daemon threads and long before the package
|
||||
teardown runs, while our daemon threads keep working. Measured on 3.12 and
|
||||
3.14, ``Thread.start`` still succeeds in that window; it is refused ("can't
|
||||
create new thread at interpreter shutdown") only once the interpreter is
|
||||
marked finalizing, after the ``atexit`` callbacks. ``refused_at_shutdown``
|
||||
recognises both; ``note_refusal`` logs one DEBUG line per place, with no
|
||||
traceback.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
SHUTDOWN_REFUSAL = "interpreter shutdown"
|
||||
_logger = logging.getLogger("prompt_manager.parallel")
|
||||
_noted = set()
|
||||
_noted_lock = threading.Lock()
|
||||
|
||||
|
||||
def refused_at_shutdown(error):
|
||||
"""Whether ``error`` is a thread or pool start refused at interpreter exit."""
|
||||
return isinstance(error, RuntimeError) and SHUTDOWN_REFUSAL in str(error)
|
||||
|
||||
|
||||
def note_refusal(logger, what):
|
||||
"""Log once per ``what``, at DEBUG and without a traceback, that ``what``
|
||||
was refused because the interpreter is exiting."""
|
||||
key = (logger.name, what)
|
||||
with _noted_lock:
|
||||
if key in _noted:
|
||||
return
|
||||
_noted.add(key)
|
||||
logger.debug(f"{what}: refused, the interpreter is exiting")
|
||||
|
||||
|
||||
def map_parallel(func, items, workers):
|
||||
"""Apply ``func`` to every item on up to ``workers`` threads, keeping order.
|
||||
|
||||
``workers`` of 1 (or fewer) runs on the calling thread with no pool. An
|
||||
exception in ``func`` propagates to the caller, so callers that must
|
||||
survive bad input wrap ``func`` themselves. A pool refused at interpreter
|
||||
exit runs the items it did not start on the calling thread (#375).
|
||||
survive bad input wrap ``func`` themselves.
|
||||
"""
|
||||
items = list(items)
|
||||
workers = max(1, int(workers or 1))
|
||||
@@ -57,35 +23,4 @@ def map_parallel(func, items, workers):
|
||||
return [func(item) for item in items]
|
||||
max_workers = min(workers, len(items))
|
||||
with ThreadPoolExecutor(max_workers, thread_name_prefix="pm-worker") as pool:
|
||||
futures = []
|
||||
try:
|
||||
for item in items:
|
||||
futures.append(pool.submit(func, item))
|
||||
except RuntimeError as error:
|
||||
if not refused_at_shutdown(error):
|
||||
raise
|
||||
note_refusal(_logger, "A worker pool")
|
||||
# Drop what was queued but not started; it runs here instead.
|
||||
# (A refused Thread.start leaves its item queued without a future
|
||||
# here: it is cancelled too, unless a busy pool thread took it in
|
||||
# between, the one case where an item can run twice.)
|
||||
pool.shutdown(wait=True, cancel_futures=True)
|
||||
else:
|
||||
return [future.result() for future in futures]
|
||||
return _run_rest(func, items, futures)
|
||||
|
||||
|
||||
def _run_rest(func, items, futures):
|
||||
"""The pool's finished results, and every item it did not run, run here.
|
||||
|
||||
After ``shutdown(cancel_futures=True)`` each submitted future either ran
|
||||
(its result or exception is kept) or was cancelled before it started.
|
||||
"""
|
||||
results = []
|
||||
for index, item in enumerate(items):
|
||||
future = futures[index] if index < len(futures) else None
|
||||
if future is not None and not future.cancelled():
|
||||
results.append(future.result())
|
||||
else:
|
||||
results.append(func(item))
|
||||
return results
|
||||
return list(pool.map(func, items))
|
||||
|
||||
@@ -2578,115 +2578,3 @@ def resolve_text_input(
|
||||
return resolver.text(inputs.get(input_name), "string")[0]
|
||||
except _Unresolved:
|
||||
return None
|
||||
|
||||
|
||||
_NUMBER = re.compile(r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?")
|
||||
|
||||
|
||||
def _numeric(text: str) -> bool:
|
||||
"""Whether ``text`` is a plain number (a widget value such as ``2048``)."""
|
||||
return _NUMBER.fullmatch(text) is not None
|
||||
|
||||
|
||||
def sampler_less_prompt(graph: Any, *, max_visits: int = 500) -> Dict[str, Any]:
|
||||
"""The positive prompt of an API graph with no sampler pair, if one (#252).
|
||||
|
||||
For graphs ``_no_sampler`` finds ``absent`` (no node takes a ``positive``
|
||||
or ``negative`` input, such as a Flux guider or a video wrapper). Only the
|
||||
executed API graph is read, never the UI workflow. A candidate is a node
|
||||
with a ``text`` input whose output another node consumes: a text that
|
||||
feeds nothing did not make the image. Its text is traced as a STRING
|
||||
(``resolve_text_input``); a linked text no rule reads is unknown unless
|
||||
it is built from a candidate's text (``_covered``). Blank and
|
||||
numeric texts are never prompts, and a node with a seed input holds a
|
||||
template, not the prompt (unknown). This repository's own positive-text
|
||||
nodes are authoritative: when they give a text, only they count.
|
||||
|
||||
Returns ``{"state", "text", "texts"}``: ``resolved`` with the one distinct
|
||||
text; ``ambiguous`` when several texts or an unknown one could be the
|
||||
prompt (``texts`` lists the readable ones); ``absent`` when none.
|
||||
"""
|
||||
absent = {"state": "absent", "text": None, "texts": []}
|
||||
if not isinstance(graph, dict) or not graph or len(graph) > max_visits:
|
||||
return absent
|
||||
nodes = {str(key): value for key, value in graph.items()}
|
||||
consumed = {
|
||||
str(value[0])
|
||||
for node in nodes.values()
|
||||
if isinstance(node, dict) and isinstance(node.get("inputs"), dict)
|
||||
for value in node["inputs"].values()
|
||||
if _link(value)
|
||||
}
|
||||
own: list = []
|
||||
other: list = []
|
||||
unknown = False
|
||||
for node_id, node in nodes.items():
|
||||
inputs = node.get("inputs") if isinstance(node, dict) else None
|
||||
if node_id not in consumed or not _has_text_input(node):
|
||||
continue
|
||||
if _seeded(inputs):
|
||||
# Generated at run time (a wildcard or a language model): its
|
||||
# text input is a template, not the prompt that ran.
|
||||
unknown = True
|
||||
continue
|
||||
value = inputs["text"]
|
||||
text = resolve_text_input(nodes, node_id, "text", max_visits=max_visits)
|
||||
if text is None:
|
||||
# A literal that is not a string (a number) is never a prompt.
|
||||
if _link(value) and not _covered(nodes, value):
|
||||
unknown = True
|
||||
continue
|
||||
text = text.strip()
|
||||
if not text or _numeric(text):
|
||||
continue
|
||||
kind = node.get("class_type")
|
||||
if isinstance(kind, str) and kind in _OWN_CLASSES and "Negative" not in kind:
|
||||
own.append(text)
|
||||
else:
|
||||
other.append(text)
|
||||
texts = list(dict.fromkeys(own or other))
|
||||
if len(texts) == 1 and (own or not unknown):
|
||||
return {"state": "resolved", "text": texts[0], "texts": texts}
|
||||
if texts or unknown:
|
||||
return {"state": "ambiguous", "text": None, "texts": texts}
|
||||
return absent
|
||||
|
||||
|
||||
def _has_text_input(node: Any) -> bool:
|
||||
inputs = node.get("inputs") if isinstance(node, dict) else None
|
||||
return isinstance(inputs, dict) and "text" in inputs
|
||||
|
||||
|
||||
def _seeded(inputs: dict) -> bool:
|
||||
"""A seed input: the node's string output is a run-time value (the
|
||||
resolver's ``runtime-value`` rule)."""
|
||||
return any(isinstance(name, str) and name.endswith("seed") for name in inputs)
|
||||
|
||||
|
||||
def _covered(nodes: dict, link: list) -> bool:
|
||||
"""Whether an unreadable text link is built from a candidate's text.
|
||||
|
||||
True when a node upstream of ``link`` is read through its first output
|
||||
(slot 0, the legacy encoder rule's text) and has an unseeded ``text``
|
||||
input: that candidate is read on its own, and whatever else is joined to
|
||||
it at run time (a caption, a language model) is no stored prompt to
|
||||
choose. Another output of such a node (a loader's trigger words) is not
|
||||
its text. False when the text comes only from inputs no candidate holds
|
||||
(a builder's ``prompt`` field): the prompt is unknown then.
|
||||
"""
|
||||
seen, stack = set(), [(str(link[0]), link[1])]
|
||||
while stack:
|
||||
node_id, slot = stack.pop()
|
||||
if (node_id, slot) in seen:
|
||||
continue
|
||||
seen.add((node_id, slot))
|
||||
node = nodes.get(node_id)
|
||||
inputs = node.get("inputs") if isinstance(node, dict) else None
|
||||
if not isinstance(inputs, dict):
|
||||
continue
|
||||
if "text" in inputs and slot == 0 and not _seeded(inputs):
|
||||
return True
|
||||
stack.extend(
|
||||
(str(value[0]), value[1]) for value in inputs.values() if _link(value)
|
||||
)
|
||||
return False
|
||||
|
||||
+31
-176
@@ -40,19 +40,6 @@ while it is pending and undecided (never beside a decision, see
|
||||
profile the item recorded (``captured_review(..., item_profile(evidence))``).
|
||||
The inspection profile only ever produces review choices; ingestion's own
|
||||
links stay on the default profile, which never loads the table (D1).
|
||||
|
||||
The saver anchor (#390): the inspection profile resolves a file with its
|
||||
name, as the workspace inspector does since #383, so the one saver whose
|
||||
``filename_prefix`` names the file anchors its branch. When that anchor alone
|
||||
resolves a file the default profile left ambiguous and nothing links
|
||||
(``saver_anchored``), the scan queues it with the anchor's pair as its one
|
||||
candidate (evidence ``anchor: "saver"``), never linking it: a file never
|
||||
falls through both the link and the queue (maintainer ruling). No match,
|
||||
several matches or an unfinished walk anchor nothing and the file is queued
|
||||
as before. Apply re-resolves with the file's name, so only the anchored
|
||||
branch can be applied. ``REVIEW_RULE_VERSION`` is folded into inspection
|
||||
item versions, so inspection items recorded before the anchor are recorded
|
||||
again by the next scan.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -117,16 +104,10 @@ MAX_CANDIDATE_DEPTH = 8
|
||||
# The resolver profile an item's evidence was built with (``profile``).
|
||||
DEFAULT = "default"
|
||||
INSPECTION = "inspection"
|
||||
# The ``anchor`` an item recorded from the saver anchor names (#390).
|
||||
SAVER_ANCHOR = "saver"
|
||||
ANCHOR_REASON = "The saver that wrote this file names its branch"
|
||||
# Inspection item versions: above every plausible ``RESOLVER_VERSION`` and,
|
||||
# with 40 bits of the folded digest, below 2**53 (exact as a JSON number).
|
||||
INSPECTION_VERSION_BASE = 1 << 40
|
||||
_INSPECTION_VERSION_HEX = 10
|
||||
# How the scan and Apply use the inspection profile, folded into inspection
|
||||
# item versions. 1: they resolve with the file's name (the saver anchor, #390).
|
||||
REVIEW_RULE_VERSION = 1
|
||||
|
||||
|
||||
# Above this many distinct node classes the whole table is the fingerprint.
|
||||
@@ -173,92 +154,38 @@ def inspection_review_version(metadata: Any = None) -> int | None:
|
||||
"""The ``resolver_version`` of an item recorded from the inspection profile.
|
||||
|
||||
Folds the profile with its budgets, ``RESOLVER_VERSION``,
|
||||
``INSPECTION_RESOLVER_VERSION``, ``REVIEW_RULE_VERSION`` and
|
||||
``semantics_version(metadata)``, so a change of either resolver, of how
|
||||
review uses the profile, of the profile's bounds or of the table about
|
||||
the graph's classes moves it. None when the table cannot be loaded.
|
||||
``INSPECTION_RESOLVER_VERSION`` and ``semantics_version(metadata)``, so
|
||||
a change of either resolver, of the profile's bounds or of the table
|
||||
about the graph's classes moves it. None when the table cannot be loaded.
|
||||
"""
|
||||
table = semantics_version(metadata)
|
||||
if table is None:
|
||||
return None
|
||||
profile = repr(astuple(INSPECTION_PROFILE))
|
||||
seed = f"{profile}:{RESOLVER_VERSION}:{INSPECTION_RESOLVER_VERSION}:{table}"
|
||||
if REVIEW_RULE_VERSION: # 0 is the seed items had before the rule existed
|
||||
seed += f":review-rule-{REVIEW_RULE_VERSION}"
|
||||
folded = hashlib.sha256(seed.encode("utf-8")).hexdigest()
|
||||
return INSPECTION_VERSION_BASE + int(folded[:_INSPECTION_VERSION_HEX], 16)
|
||||
|
||||
|
||||
def resolve_inspection(metadata: dict, file_name: str | None = None) -> dict:
|
||||
"""``metadata`` resolved with the inspection profile (bounded work).
|
||||
|
||||
``file_name`` (the file's last path part) anchors the file on the one
|
||||
saver that wrote it, as in the workspace inspector (#390).
|
||||
"""
|
||||
return resolve_prompt_pair(
|
||||
metadata, profile=INSPECTION_PROFILE, file_name=file_name
|
||||
)
|
||||
def resolve_inspection(metadata: dict) -> dict:
|
||||
"""``metadata`` resolved with the inspection profile (bounded work)."""
|
||||
return resolve_prompt_pair(metadata, profile=INSPECTION_PROFILE)
|
||||
|
||||
|
||||
def inspection_choice(metadata: dict, file_name: str | None = None) -> dict | None:
|
||||
"""The inspection profile's result when the file is a review case.
|
||||
def inspection_choice(metadata: dict) -> dict | None:
|
||||
"""The inspection profile's ambiguous result when it offers a choice.
|
||||
|
||||
The scan asks only for a file the default profile found ambiguous
|
||||
without a choice that nothing links. The result is ambiguous with a
|
||||
choice, or ``resolved`` when ``file_name`` anchors the file on its saver
|
||||
and only that anchor resolves it (``saver_anchored``): the item then
|
||||
offers the anchor's pair. None otherwise.
|
||||
without a choice that nothing links.
|
||||
"""
|
||||
if inspection_review_version(metadata) is None:
|
||||
return None
|
||||
resolution = resolve_inspection(metadata, file_name)
|
||||
if resolution.get("state") == "resolved":
|
||||
return _anchor_of(metadata, file_name, resolution)
|
||||
resolution = resolve_inspection(metadata)
|
||||
if resolution.get("state") != "ambiguous" or not reviewable(resolution):
|
||||
return None
|
||||
return resolution
|
||||
|
||||
|
||||
def _anchor_of(metadata: Any, file_name: str | None, named: dict) -> dict | None:
|
||||
"""``named`` tagged ``anchor: "saver"`` when only the saver anchor
|
||||
resolves the file, to a pair with a positive text to choose; else None.
|
||||
|
||||
``named`` is ``metadata`` resolved with ``file_name``; the profile runs
|
||||
once more without the name only when it is resolved and choosable.
|
||||
"""
|
||||
if not file_name or named.get("state") != "resolved":
|
||||
return None
|
||||
try:
|
||||
offered = choosable(_projection(named))
|
||||
except (KeyError, TypeError):
|
||||
return None
|
||||
if not offered or resolve_inspection(metadata).get("state") == "resolved":
|
||||
return None
|
||||
return {**named, "anchor": SAVER_ANCHOR}
|
||||
|
||||
|
||||
def saver_anchored(metadata: Any, file_name: str | None) -> bool:
|
||||
"""True when the saver that wrote ``file_name`` resolves its branch (#390).
|
||||
|
||||
The inspection profile resolves ``metadata`` with the file's name, and
|
||||
without it the file does not resolve: the anchor alone decides it. A file
|
||||
the default profile left ambiguous is then queued with the anchor's pair
|
||||
as its one candidate. No name, an unusable table, no matching saver,
|
||||
several, or an unfinished walk anchor nothing (False). The profile
|
||||
without the name runs only when the named run resolved.
|
||||
"""
|
||||
return _anchored(metadata, file_name) is not None
|
||||
|
||||
|
||||
def _anchored(metadata: Any, file_name: str | None) -> dict | None:
|
||||
"""The anchored result ``saver_anchored`` describes, else None."""
|
||||
if not file_name or not isinstance(metadata, dict):
|
||||
return None
|
||||
if inspection_review_version(metadata) is None:
|
||||
return None
|
||||
return _anchor_of(metadata, file_name, resolve_inspection(metadata, file_name))
|
||||
|
||||
|
||||
def item_profile(evidence: Any) -> str:
|
||||
"""The profile an item's stored evidence was built with."""
|
||||
if isinstance(evidence, dict) and evidence.get("profile") == INSPECTION:
|
||||
@@ -319,9 +246,7 @@ def record_ambiguous_file(
|
||||
"""
|
||||
versions = item_versions(scanned[0] if scanned is not None else None)
|
||||
spelled_path, queued = _queued_spelling(db, root, relative, versions)
|
||||
if queued is not None and not (
|
||||
inspection and _default_item_to_anchor(db, queued, scanned, relative)
|
||||
):
|
||||
if queued is not None:
|
||||
return _already_queued(db, *queued, versions)
|
||||
epoch = db.get_execution_epoch()
|
||||
try:
|
||||
@@ -347,24 +272,24 @@ def record_ambiguous_file(
|
||||
if resolution is None or resolution.get("state") != "ambiguous":
|
||||
return "skipped"
|
||||
profile, version = DEFAULT, RESOLVER_VERSION
|
||||
named = _captured_inspection(captured[0], relative, scanned) if inspection else None
|
||||
anchored = named is not None and named[0].get("anchor") == SAVER_ANCHOR
|
||||
if anchored:
|
||||
# Only the saver anchor resolves it (#390): offer that one pair.
|
||||
profile, version = INSPECTION, named[1]
|
||||
resolution = {
|
||||
"state": "ambiguous",
|
||||
"reason": ANCHOR_REASON,
|
||||
"candidates": [named[0]],
|
||||
}
|
||||
elif not reviewable(resolution):
|
||||
if named is None:
|
||||
if not reviewable(resolution):
|
||||
version = inspection_review_version(captured[0]) if inspection else None
|
||||
if version is None:
|
||||
return "untraced"
|
||||
profile, (resolution, version) = INSPECTION, named
|
||||
profile = INSPECTION
|
||||
if (
|
||||
scanned is not None
|
||||
and scanned[1] is not None
|
||||
and _same_records(captured[0], scanned[0])
|
||||
):
|
||||
resolution = scanned[1] # the same records: no second run
|
||||
else:
|
||||
# Heavier, so only for a file ambiguous with nothing to choose.
|
||||
resolution = resolve_inspection(captured[0])
|
||||
if resolution.get("state") != "ambiguous" or not reviewable(resolution):
|
||||
return "untraced"
|
||||
evidence, count, limited = review_evidence(
|
||||
resolution, profile, version, captured[0], anchored=anchored
|
||||
resolution, profile, version, captured[0]
|
||||
)
|
||||
try:
|
||||
inserted = db.record_review_item(
|
||||
@@ -388,60 +313,6 @@ def record_ambiguous_file(
|
||||
return "limited" if limited else "recorded"
|
||||
|
||||
|
||||
def _default_item_to_anchor(db, queued: tuple, scanned, relative: str) -> bool:
|
||||
"""True when the bytes' only pending item is a default item recorded
|
||||
before #390 and the saver anchor alone resolves the scanned records.
|
||||
|
||||
The file is then captured again so the anchored item supersedes that
|
||||
pending, undecided item. Decided bytes and bytes already queued under
|
||||
the current inspection version are never re-read.
|
||||
"""
|
||||
if scanned is None or scanned[0] is None:
|
||||
return False
|
||||
version = inspection_review_version(scanned[0])
|
||||
if version is None or db.review_pending_at(*queued, version):
|
||||
return False
|
||||
if not db.review_pending_at(*queued, RESOLVER_VERSION):
|
||||
return False
|
||||
return _anchored(scanned[0], _file_name(relative)) is not None
|
||||
|
||||
|
||||
def _captured_inspection(metadata: dict, relative: str, scanned) -> tuple | None:
|
||||
"""``(resolution, item_version)`` of the captured records under the
|
||||
inspection profile, anchored on the file's name; None when the table is
|
||||
unusable.
|
||||
|
||||
A result tagged ``anchor: "saver"`` means the saver anchor alone
|
||||
resolves the file (#390); any other resolved result is returned as
|
||||
``unresolved`` (no review case, as before). The scan's result is reused
|
||||
when the capture holds the same records (``inspection_choice`` applies
|
||||
the same rule).
|
||||
"""
|
||||
version = inspection_review_version(metadata)
|
||||
if version is None:
|
||||
return None
|
||||
if (
|
||||
scanned is not None
|
||||
and scanned[1] is not None
|
||||
and _same_records(metadata, scanned[0])
|
||||
):
|
||||
return scanned[1], version # the same records: no second run
|
||||
name = _file_name(relative)
|
||||
resolution = resolve_inspection(metadata, name)
|
||||
if resolution.get("state") == "resolved":
|
||||
# Resolved without the anchor too: no review case, as before.
|
||||
resolution = _anchor_of(metadata, name, resolution) or {
|
||||
**resolution,
|
||||
"state": "unresolved",
|
||||
}
|
||||
return resolution, version
|
||||
|
||||
|
||||
def _file_name(relative: str) -> str:
|
||||
"""The last part of a ``/``-separated relative path (the saved name)."""
|
||||
return relative.rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def _already_queued(db, locator: str, size: int, mtime_ns: int, versions) -> str:
|
||||
"""``duplicate`` while an existing item is pending, else ``decided``."""
|
||||
for version in versions:
|
||||
@@ -483,19 +354,14 @@ def _queued_spelling(db, root, relative: str, versions) -> tuple[str, tuple | No
|
||||
return spelled, None
|
||||
|
||||
|
||||
def captured_review(
|
||||
store, capture, profile: str = DEFAULT, file_name: str | None = None
|
||||
) -> tuple[dict, dict] | None:
|
||||
def captured_review(store, capture, profile: str = DEFAULT) -> tuple[dict, dict] | None:
|
||||
"""``(metadata, resolution)`` of the capture's own bytes, else None.
|
||||
|
||||
``metadata`` is what the scan hands the ingester: prompt and workflow
|
||||
JSON-decoded and ``file_info`` (size, dimensions, format) from the
|
||||
capture. ``resolution`` comes from ``profile`` (``DEFAULT`` or
|
||||
``INSPECTION``; the latter anchored on ``file_name``, the file's own
|
||||
name, as the item was recorded). With ``file_name`` (Apply), a default
|
||||
result its saver anchor alone resolves (``saver_anchored``) is that
|
||||
anchored result, so only the anchored branch is still offered. None
|
||||
when the capture is gone or carries no prompt metadata.
|
||||
``INSPECTION``). None when the capture is gone or carries no prompt
|
||||
metadata.
|
||||
"""
|
||||
if store.acquire(capture.token) is None:
|
||||
return None
|
||||
@@ -523,15 +389,8 @@ def captured_review(
|
||||
info["format"] = read.format
|
||||
canonical["file_info"] = info
|
||||
if profile == INSPECTION:
|
||||
named = resolve_inspection(canonical, file_name)
|
||||
return canonical, _anchor_of(canonical, file_name, named) or named
|
||||
resolution = resolve_prompt_pair(canonical)
|
||||
if file_name is not None and resolution.get("state") == "ambiguous":
|
||||
anchored = _anchored(canonical, file_name)
|
||||
if anchored is not None:
|
||||
# Its saver names its branch (#390): only that one is offered.
|
||||
return canonical, anchored
|
||||
return canonical, resolution
|
||||
return canonical, resolve_inspection(canonical)
|
||||
return canonical, resolve_prompt_pair(canonical)
|
||||
|
||||
|
||||
def reviewable(resolution: Any) -> bool:
|
||||
@@ -557,14 +416,12 @@ def review_evidence(
|
||||
profile: str = DEFAULT,
|
||||
version: int | None = None,
|
||||
metadata: Any = None,
|
||||
anchored: bool = False,
|
||||
) -> tuple[dict, int, bool]:
|
||||
"""``(evidence, candidate_count, limited)`` for one ambiguous result.
|
||||
|
||||
Inspection evidence also names its ``profile`` and ``semantics``, the
|
||||
table fingerprint of ``metadata``'s graph it was resolved with;
|
||||
``version`` is the item's version. ``anchored`` evidence (the saver
|
||||
anchor's one pair, #390) also names ``anchor: "saver"``.
|
||||
``version`` is the item's version.
|
||||
"""
|
||||
base = {
|
||||
"resolver_version": RESOLVER_VERSION if version is None else version,
|
||||
@@ -573,8 +430,6 @@ def review_evidence(
|
||||
if profile == INSPECTION:
|
||||
base["profile"] = INSPECTION
|
||||
base["semantics"] = semantics_version(metadata)
|
||||
if anchored:
|
||||
base["anchor"] = SAVER_ANCHOR
|
||||
candidates: list[dict] = []
|
||||
seen: set[str] = set()
|
||||
total = 0
|
||||
|
||||
+165
-12
@@ -168,11 +168,10 @@
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Settings
|
||||
</button>
|
||||
<a id="diagnosticsBtn" href="workspace#/system/diagnostics" target="_blank" rel="noopener"
|
||||
title="Opens System › Diagnostics & logs in a new tab"
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
<button id="diagnosticsBtn"
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Diagnostics
|
||||
</a>
|
||||
</button>
|
||||
<button id="maintenanceBtn"
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Maintenance
|
||||
@@ -193,11 +192,10 @@
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Auto Tag
|
||||
</button>
|
||||
<a id="logsBtn" href="workspace#/system/diagnostics" target="_blank" rel="noopener"
|
||||
title="Opens System › Diagnostics & logs in a new tab"
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
<button id="logsBtn"
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
View Logs
|
||||
</a>
|
||||
</button>
|
||||
<button id="metadataBtn"
|
||||
class="px-2.5 py-1 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Metadata
|
||||
@@ -643,6 +641,37 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Diagnostics Modal -->
|
||||
<div id="diagnosticsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
|
||||
<div class="bg-pm-surface rounded-pm-lg max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
|
||||
<div class="p-4 border-b border-pm">
|
||||
<div class="flex items-center justify-between">
|
||||
<h3 class="text-sm font-semibold text-pm">Gallery System Diagnostics</h3>
|
||||
<button onclick="window.admin.closeDiagnostics()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
|
||||
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="p-4 overflow-y-auto max-h-[calc(90vh-120px)]">
|
||||
<div class="flex items-center justify-between mb-3">
|
||||
<button id="runDiagnosticsBtn" class="px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Run Diagnostics
|
||||
</button>
|
||||
<button id="testImageLinkBtn" class="px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Test Image Link
|
||||
</button>
|
||||
</div>
|
||||
<div id="diagnosticsContent" class="space-y-3">
|
||||
<div class="text-center py-6 text-pm-secondary text-xs">
|
||||
Click "Run Diagnostics" to check the gallery system status
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Maintenance Modal -->
|
||||
<div id="maintenanceModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
|
||||
<div class="bg-pm-surface rounded-pm-lg max-w-4xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
|
||||
@@ -835,6 +864,130 @@
|
||||
<span class="inline-block w-1.5 h-1.5 rounded-full bg-pm-accent mr-1.5 align-middle animate-pulse"></span><span id="scanPillText" class="align-middle">Scanning...</span>
|
||||
</button>
|
||||
|
||||
<!-- Logs Modal -->
|
||||
<div id="logsModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
|
||||
<div class="bg-pm-surface rounded-pm-lg max-w-6xl w-full max-h-[90vh] overflow-hidden mx-4 border border-pm shadow-pm">
|
||||
<div class="p-4 border-b border-pm">
|
||||
<div class="flex items-center justify-between">
|
||||
<h3 class="text-sm font-semibold text-pm">System Logs</h3>
|
||||
<button onclick="window.admin.closeLogs()" class="p-1 rounded-pm-sm hover:bg-pm-hover transition-colors">
|
||||
<svg class="w-4 h-4 text-pm-secondary" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Log Controls -->
|
||||
<div class="p-4 border-b border-pm">
|
||||
<div class="grid grid-cols-1 md:grid-cols-5 gap-3 mb-3">
|
||||
<div>
|
||||
<label class="block text-xs font-medium text-pm-secondary mb-1">Log Level</label>
|
||||
<select id="logLevel" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
|
||||
<option value="">All Levels</option>
|
||||
<option value="DEBUG">Debug</option>
|
||||
<option value="INFO">Info</option>
|
||||
<option value="WARNING">Warning</option>
|
||||
<option value="ERROR">Error</option>
|
||||
<option value="CRITICAL">Critical</option>
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label class="block text-xs font-medium text-pm-secondary mb-1">Limit</label>
|
||||
<select id="logLimit" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
|
||||
<option value="50">Last 50</option>
|
||||
<option value="100" selected>Last 100</option>
|
||||
<option value="200">Last 200</option>
|
||||
<option value="500">Last 500</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="flex items-end">
|
||||
<button id="refreshLogsBtn" class="w-full px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Refresh
|
||||
</button>
|
||||
</div>
|
||||
<div class="flex items-end">
|
||||
<button id="downloadLogsBtn" class="w-full px-2.5 py-1.5 bg-pm-surface border border-pm hover:bg-pm-hover text-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Download
|
||||
</button>
|
||||
</div>
|
||||
<div class="flex items-end">
|
||||
<button id="clearLogsBtn" class="w-full px-2.5 py-1.5 text-pm-error bg-transparent hover:bg-pm-error-tint border border-pm text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Clear Logs
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Log Stats -->
|
||||
<div class="grid grid-cols-2 md:grid-cols-5 gap-3 mb-3">
|
||||
<div class="bg-pm-input rounded-pm-md p-2">
|
||||
<div class="text-[11px] text-pm-secondary">Total Logs</div>
|
||||
<div class="text-sm font-semibold text-pm-accent" id="logStatsTotal">-</div>
|
||||
</div>
|
||||
<div class="bg-pm-input rounded-pm-md p-2">
|
||||
<div class="text-[11px] text-pm-secondary">Errors</div>
|
||||
<div class="text-sm font-semibold text-pm-error" id="logStatsErrors">-</div>
|
||||
</div>
|
||||
<div class="bg-pm-input rounded-pm-md p-2">
|
||||
<div class="text-[11px] text-pm-secondary">Warnings</div>
|
||||
<div class="text-sm font-semibold text-pm-warning" id="logStatsWarnings">-</div>
|
||||
</div>
|
||||
<div class="bg-pm-input rounded-pm-md p-2">
|
||||
<div class="text-[11px] text-pm-secondary">Current Level</div>
|
||||
<div class="text-sm font-semibold text-pm-success" id="logStatsLevel">-</div>
|
||||
</div>
|
||||
<div class="bg-pm-input rounded-pm-md p-2">
|
||||
<div class="text-[11px] text-pm-secondary">Log Size</div>
|
||||
<div class="text-sm font-semibold text-pm" id="logStatsSize">-</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Log Settings -->
|
||||
<div class="grid grid-cols-1 md:grid-cols-3 gap-3">
|
||||
<div>
|
||||
<label class="block text-xs font-medium text-pm-secondary mb-1">Set Log Level</label>
|
||||
<select id="setLogLevel" class="w-full px-2.5 py-1.5 bg-pm-input border border-pm rounded-pm-sm text-pm text-[13px] focus:outline-none">
|
||||
<option value="DEBUG">Debug</option>
|
||||
<option value="INFO" selected>Info</option>
|
||||
<option value="WARNING">Warning</option>
|
||||
<option value="ERROR">Error</option>
|
||||
<option value="CRITICAL">Critical</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="flex items-center">
|
||||
<label class="flex items-center gap-1.5 text-pm">
|
||||
<input type="checkbox" id="consoleLogging" checked class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
|
||||
<span class="text-xs">Console Logging</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="flex items-end">
|
||||
<button id="updateLogConfigBtn" class="w-full px-2.5 py-1.5 bg-pm-accent hover:bg-pm-accent-hover text-pm-accent-fg text-xs font-medium rounded-pm-sm transition-colors">
|
||||
Update Config
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Log Viewer -->
|
||||
<div class="flex-1 overflow-hidden">
|
||||
<div class="h-96 overflow-y-auto bg-pm-primary font-mono text-xs">
|
||||
<div id="logsContainer" class="p-3 space-y-0.5">
|
||||
<div class="text-center py-6 text-pm-muted text-xs">
|
||||
Click "Refresh" to load logs
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Log Files -->
|
||||
<div class="p-4 border-t border-pm">
|
||||
<h4 class="text-xs font-semibold text-pm-secondary uppercase tracking-wide mb-3">Log Files</h4>
|
||||
<div id="logFilesList" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-2">
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Add Prompt Modal -->
|
||||
<div id="addPromptModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
|
||||
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-2xl w-full mx-4 border border-pm max-h-[90vh] overflow-y-auto shadow-pm">
|
||||
@@ -1034,13 +1187,14 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
|
||||
<div class="space-y-1.5">
|
||||
<label class="flex items-center p-2.5 bg-pm-input rounded-pm-md cursor-pointer hover:bg-pm-hover border border-pm">
|
||||
<input type="radio" name="autoTagMode" value="skip" checked class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
|
||||
<span class="ml-2.5 text-pm text-xs">Skip images already auto-tagged</span>
|
||||
<span class="ml-2.5 text-pm text-xs">Skip images with existing tags</span>
|
||||
</label>
|
||||
<label class="flex items-center p-2.5 bg-pm-input rounded-pm-md cursor-pointer hover:bg-pm-hover border border-pm">
|
||||
<input type="radio" name="autoTagMode" value="retag" class="w-3.5 h-3.5 accent-[var(--pm-accent)]">
|
||||
<span class="ml-2.5 text-pm text-xs">Re-tag all images</span>
|
||||
</label>
|
||||
</div>
|
||||
<p class="text-pm-muted text-[11px] mt-1">Note: "auto-scanned" is not counted as a real tag</p>
|
||||
</div>
|
||||
|
||||
<!-- Action Buttons -->
|
||||
@@ -1139,7 +1293,6 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
|
||||
<div class="w-full bg-pm-input rounded-full h-1.5">
|
||||
<div id="autoTagProgressBar" class="bg-pm-accent h-1.5 rounded-full transition-all duration-300" style="width: 0%"></div>
|
||||
</div>
|
||||
<p id="autoTagCloseNote" class="text-pm-muted text-[11px] mt-1.5">Closing this window stops tagging. Tags already added are kept. Prompt Manager's Activity page shows the time left.</p>
|
||||
</div>
|
||||
|
||||
<div class="grid grid-cols-3 gap-3 mb-4">
|
||||
@@ -1179,7 +1332,7 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
|
||||
<!-- Re-tag Confirmation Modal -->
|
||||
<div id="retagConfirmModal" class="fixed inset-0 bg-black/60 backdrop-blur-[2px] hidden items-center justify-center z-50">
|
||||
<div class="bg-pm-surface rounded-pm-lg p-4 max-w-lg w-full mx-4 border border-pm shadow-pm">
|
||||
<h3 class="text-sm font-semibold text-pm mb-3">Image Already Auto-tagged</h3>
|
||||
<h3 class="text-sm font-semibold text-pm mb-3">Image Already Tagged</h3>
|
||||
|
||||
<div class="mb-3">
|
||||
<img id="retagPreviewImage" src="" alt="Preview" class="w-full h-40 object-contain bg-pm-primary rounded-pm-md">
|
||||
@@ -1192,7 +1345,7 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p class="text-pm-secondary text-xs mb-4">A tagging model already tagged this image. Generate new tags?</p>
|
||||
<p class="text-pm-secondary text-xs mb-4">This image already has tags. Would you like to generate new tags?</p>
|
||||
|
||||
<div class="flex gap-2 justify-end">
|
||||
<button id="retagSkipBtn"
|
||||
|
||||
@@ -682,7 +682,6 @@ No sentences, no commentary, no captions. Only tags. Keep length short but descr
|
||||
<div class="w-full bg-pm-input rounded-full h-2">
|
||||
<div id="autoTagProgressBar" class="bg-pm-accent h-2 rounded-full transition-all duration-300" style="width: 0%"></div>
|
||||
</div>
|
||||
<p id="autoTagCloseNote" class="text-pm-muted text-xs mt-1.5">Closing this window stops tagging. Tags already added are kept. Prompt Manager's Activity page shows the time left.</p>
|
||||
</div>
|
||||
|
||||
<!-- Stats -->
|
||||
|
||||
+508
-52
@@ -47,6 +47,7 @@
|
||||
edit: ({ promptId }) => this.editPrompt(promptId),
|
||||
delete: ({ promptId }) => this.deletePrompt(promptId),
|
||||
"open-film": ({ promptId, index }) => this.openFilmStripViewer(promptId, index),
|
||||
"download-log": ({ filename }) => this.downloadLogFile(filename),
|
||||
"remove-review-tag": ({ index }) => this.removeReviewTag(index),
|
||||
"pick-tag-suggestion": ({ tag }) => this.pickTagSuggestion(tag),
|
||||
};
|
||||
@@ -127,12 +128,14 @@
|
||||
document.getElementById("exportBtn").addEventListener("click", () => this.exportPrompts());
|
||||
document.getElementById("settingsBtn").addEventListener("click", () => this.showSettingsModal());
|
||||
document.getElementById("statsBtn").addEventListener("click", () => this.showStats());
|
||||
document.getElementById("diagnosticsBtn").addEventListener("click", () => this.showDiagnosticsModal());
|
||||
document.getElementById("maintenanceBtn").addEventListener("click", () => this.showMaintenanceModal());
|
||||
document.getElementById("backupBtn").addEventListener("click", () => this.backupDatabase());
|
||||
document.getElementById("restoreBtn").addEventListener("click", () => this.showRestoreModal());
|
||||
document.getElementById("scanBtn").addEventListener("click", () => this.showScanModal());
|
||||
document.getElementById("autoTagBtn").addEventListener("click", () => this.showAutoTagModal());
|
||||
document.getElementById("addPromptBtn").addEventListener("click", () => this.showAddPromptModal());
|
||||
document.getElementById("logsBtn").addEventListener("click", () => this.showLogsModal());
|
||||
document.getElementById("metadataBtn").addEventListener("click", () => this.openMetadataViewer());
|
||||
document.getElementById("galleryBtn").addEventListener("click", () => this.openGallery());
|
||||
|
||||
@@ -212,6 +215,10 @@
|
||||
document.getElementById("confirmRestore").addEventListener("click", () => this.confirmRestore());
|
||||
document.getElementById("cancelRestore").addEventListener("click", () => this.hideModal("restoreModal"));
|
||||
|
||||
// Diagnostics modal
|
||||
document.getElementById("runDiagnosticsBtn").addEventListener("click", () => this.runDiagnostics());
|
||||
document.getElementById("testImageLinkBtn").addEventListener("click", () => this.testImageLink());
|
||||
|
||||
// Maintenance modal
|
||||
document.getElementById("runMaintenanceBtn").addEventListener("click", () => this.runMaintenance());
|
||||
document.getElementById("selectAllMaintenanceBtn").addEventListener("click", () => this.selectAllMaintenance());
|
||||
@@ -223,6 +230,12 @@
|
||||
document.getElementById("scanPill").addEventListener("click", () => this.showScanModal());
|
||||
document.getElementById("quickBackupBtn").addEventListener("click", () => this.quickBackup());
|
||||
|
||||
// Logs modal
|
||||
document.getElementById("refreshLogsBtn").addEventListener("click", () => this.refreshLogs());
|
||||
document.getElementById("downloadLogsBtn").addEventListener("click", () => this.downloadLogs());
|
||||
document.getElementById("clearLogsBtn").addEventListener("click", () => this.clearLogs());
|
||||
document.getElementById("updateLogConfigBtn").addEventListener("click", () => this.updateLogConfig());
|
||||
|
||||
// Auto Tag modals
|
||||
document.getElementById("startAutoTagBtn").addEventListener("click", () => this.startAutoTag());
|
||||
document.getElementById("startReviewBtn").addEventListener("click", () => this.startReview());
|
||||
@@ -1788,6 +1801,15 @@
|
||||
return `${size.toFixed(1)} ${units[unitIndex]}`;
|
||||
}
|
||||
|
||||
// Diagnostics functionality
|
||||
showDiagnosticsModal() {
|
||||
this.showModal("diagnosticsModal");
|
||||
}
|
||||
|
||||
closeDiagnostics() {
|
||||
this.hideModal("diagnosticsModal");
|
||||
}
|
||||
|
||||
// Maintenance functionality
|
||||
showMaintenanceModal() {
|
||||
this.showModal("maintenanceModal");
|
||||
@@ -1959,6 +1981,160 @@
|
||||
resultsContainer.innerHTML = html;
|
||||
}
|
||||
|
||||
async runDiagnostics() {
|
||||
const content = document.getElementById("diagnosticsContent");
|
||||
content.innerHTML = `
|
||||
<div class="text-center py-4">
|
||||
<div class="w-12 h-12 border-4 border-orange-500/30 border-t-orange-500 rounded-full animate-spin mx-auto mb-4"></div>
|
||||
<p class="text-pm-secondary">Running diagnostics...</p>
|
||||
</div>
|
||||
`;
|
||||
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/diagnostics');
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
if (data.success) {
|
||||
this.renderDiagnostics(data.diagnostics);
|
||||
} else {
|
||||
throw new Error(data.error || 'Diagnostics failed');
|
||||
}
|
||||
} else {
|
||||
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Diagnostics error:', error);
|
||||
content.innerHTML = `
|
||||
<div class="bg-pm-error-tint border border-pm-error rounded-pm-sm p-4">
|
||||
<h4 class="text-pm-error font-medium mb-2">❌ Diagnostics Failed</h4>
|
||||
<p class="text-pm-secondary">${escapeHtml(error.message)}</p>
|
||||
</div>
|
||||
`;
|
||||
}
|
||||
}
|
||||
|
||||
renderDiagnostics(diagnostics) {
|
||||
const content = document.getElementById("diagnosticsContent");
|
||||
|
||||
let html = '<div class="space-y-4">';
|
||||
|
||||
// Summary
|
||||
html += '<div class="bg-pm-surface rounded-pm-sm p-4">';
|
||||
html += '<h4 class="text-pm font-medium mb-3">📋 Diagnostic Summary</h4>';
|
||||
html += '<div class="grid grid-cols-1 md:grid-cols-2 gap-3">';
|
||||
|
||||
for (const [category, result] of Object.entries(diagnostics)) {
|
||||
const status = result.status === 'ok' ? '✅ PASS' : (result.status === 'warning' ? '⚠️ WARNING' : '❌ FAIL');
|
||||
const statusColor = result.status === 'ok' ? 'text-pm-success' : (result.status === 'warning' ? 'text-pm-warning' : 'text-pm-error');
|
||||
|
||||
html += `
|
||||
<div class="flex justify-between items-center p-2 bg-pm-input rounded">
|
||||
<span class="text-pm-secondary capitalize">${escapeHtml(category)}</span>
|
||||
<span class="${statusColor} font-mono text-sm">${status}</span>
|
||||
</div>
|
||||
`;
|
||||
}
|
||||
|
||||
html += '</div></div>';
|
||||
|
||||
// Detailed results
|
||||
for (const [category, result] of Object.entries(diagnostics)) {
|
||||
const bgColor = result.status === 'ok' ? 'bg-pm-success-tint border-pm-success' :
|
||||
(result.status === 'warning' ? 'bg-pm-warning/20 border-pm-warning' : 'bg-pm-error-tint border-pm-error');
|
||||
|
||||
html += `<div class="${bgColor} border rounded-pm-sm p-4">`;
|
||||
html += `<h4 class="text-pm font-medium mb-2 capitalize">${escapeHtml(category)}</h4>`;
|
||||
|
||||
if (result.message) {
|
||||
html += `<p class="text-pm-secondary mb-2">${escapeHtml(result.message)}</p>`;
|
||||
}
|
||||
|
||||
// Show specific details based on category
|
||||
if (category === 'database' && result.status === 'ok') {
|
||||
html += `<div class="text-sm text-pm-secondary">`;
|
||||
html += `<p>Prompts: ${escapeHtml(result.prompt_count || 0)}</p>`;
|
||||
html += `<p>Images table: ${result.has_images_table ? 'Yes' : 'No'}</p>`;
|
||||
html += `</div>`;
|
||||
}
|
||||
|
||||
if (category === 'images_table' && result.status === 'ok') {
|
||||
html += `<div class="text-sm text-pm-secondary">`;
|
||||
html += `<p>Images: ${escapeHtml(result.image_count || 0)}</p>`;
|
||||
if (result.recent_images && result.recent_images.length > 0) {
|
||||
html += `<p>Recent images:</p>`;
|
||||
html += `<ul class="ml-4 list-disc">`;
|
||||
result.recent_images.slice(0, 3).forEach(img => {
|
||||
html += `<li>${escapeHtml(img.filename)} → Prompt ${escapeHtml(img.prompt_id)}</li>`;
|
||||
});
|
||||
html += `</ul>`;
|
||||
}
|
||||
html += `</div>`;
|
||||
}
|
||||
|
||||
if (category === 'comfyui_output' && result.output_dirs) {
|
||||
html += `<div class="text-sm text-pm-secondary">`;
|
||||
html += `<p>Output directories found:</p>`;
|
||||
html += `<ul class="ml-4 list-disc">`;
|
||||
result.output_dirs.forEach(dir => {
|
||||
html += `<li>${escapeHtml(dir)}</li>`;
|
||||
});
|
||||
html += `</ul>`;
|
||||
html += `</div>`;
|
||||
}
|
||||
|
||||
if (category === 'dependencies' && result.dependencies) {
|
||||
html += `<div class="text-sm text-pm-secondary">`;
|
||||
for (const [dep, available] of Object.entries(result.dependencies)) {
|
||||
const status = available ? '✅' : '❌';
|
||||
html += `<p>${status} ${escapeHtml(dep)}</p>`;
|
||||
}
|
||||
html += `</div>`;
|
||||
}
|
||||
|
||||
html += '</div>';
|
||||
}
|
||||
|
||||
html += '</div>';
|
||||
content.innerHTML = html;
|
||||
}
|
||||
|
||||
async testImageLink() {
|
||||
if (this.prompts.length === 0) {
|
||||
this.showNotification('No prompts available for testing', 'warning');
|
||||
return;
|
||||
}
|
||||
|
||||
// Use the first prompt for testing
|
||||
const testPrompt = this.prompts[0];
|
||||
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/diagnostics/test-link', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
prompt_id: testPrompt.id,
|
||||
image_path: '/test/fake/image.png'
|
||||
})
|
||||
});
|
||||
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
if (data.success) {
|
||||
this.showNotification('✅ Test image link created successfully!', 'success');
|
||||
// Refresh the gallery to show the test image
|
||||
setTimeout(() => this.refreshList(), 1000);
|
||||
} else {
|
||||
throw new Error(data.result?.message || 'Test failed');
|
||||
}
|
||||
} else {
|
||||
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Test link error:', error);
|
||||
this.showNotification(`❌ Test failed: ${error.message}`, 'error');
|
||||
}
|
||||
}
|
||||
|
||||
// Database backup and restore functionality
|
||||
async backupDatabase() {
|
||||
try {
|
||||
@@ -2321,6 +2497,273 @@
|
||||
this.showNotification(`Scan failed: ${state.message}`, "error");
|
||||
}
|
||||
|
||||
// Logs functionality
|
||||
showLogsModal() {
|
||||
this.showModal("logsModal");
|
||||
this.loadLogStats();
|
||||
this.loadLogFiles();
|
||||
this.loadLogConfig();
|
||||
}
|
||||
|
||||
closeLogs() {
|
||||
this.hideModal("logsModal");
|
||||
}
|
||||
|
||||
async loadLogStats() {
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/logs/stats');
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
const stats = data.stats;
|
||||
document.getElementById("logStatsTotal").textContent = stats.buffer_count || 0;
|
||||
document.getElementById("logStatsErrors").textContent = stats.level_counts?.ERROR || 0;
|
||||
document.getElementById("logStatsWarnings").textContent = stats.level_counts?.WARNING || 0;
|
||||
document.getElementById("logStatsLevel").textContent = stats.current_level || "INFO";
|
||||
document.getElementById("logStatsSize").textContent = this.formatBytes(stats.total_log_size || 0);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to load log stats:", error);
|
||||
}
|
||||
}
|
||||
|
||||
async loadLogConfig() {
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/logs/config');
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
const config = data.config;
|
||||
document.getElementById("setLogLevel").value = config.level || "INFO";
|
||||
document.getElementById("consoleLogging").checked = config.console_logging !== false;
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to load log config:", error);
|
||||
}
|
||||
}
|
||||
|
||||
async loadLogFiles() {
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/logs/files');
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
const container = document.getElementById("logFilesList");
|
||||
|
||||
if (data.files.length === 0) {
|
||||
container.innerHTML = '<div class="col-span-full text-center text-pm-muted">No log files found</div>';
|
||||
return;
|
||||
}
|
||||
|
||||
container.innerHTML = data.files.map(file => `
|
||||
<div class="bg-pm-surface rounded-pm-sm p-3">
|
||||
<div class="flex items-center justify-between mb-2">
|
||||
<span class="text-sm font-medium text-pm">${escapeHtml(file.filename)}</span>
|
||||
${file.is_main ? '<span class="bg-pm-accent text-xs px-2 py-1 rounded">Active</span>' : ''}
|
||||
</div>
|
||||
<div class="text-xs text-pm-secondary mb-2">
|
||||
Size: ${this.formatBytes(file.size)} | Modified: ${new Date(file.modified).toLocaleString()}
|
||||
</div>
|
||||
<button data-action="download-log" data-filename="${escapeHtml(file.filename)}"
|
||||
class="w-full px-3 py-1 bg-pm-success hover:bg-pm-success text-pm text-xs rounded transition-colors">
|
||||
📥 Download
|
||||
</button>
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to load log files:", error);
|
||||
}
|
||||
}
|
||||
|
||||
async refreshLogs() {
|
||||
const level = document.getElementById("logLevel").value;
|
||||
const limit = parseInt(document.getElementById("logLimit").value);
|
||||
|
||||
try {
|
||||
const url = new URL(ApiClient.resolveUrl('/prompt_manager/logs'), window.location.origin);
|
||||
if (level) url.searchParams.set('level', level);
|
||||
url.searchParams.set('limit', limit.toString());
|
||||
|
||||
const response = await ApiClient.fetchUrl(url);
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
this.displayLogs(data.logs);
|
||||
this.showNotification(`Loaded ${data.logs.length} log entries`, "success");
|
||||
} else {
|
||||
this.showNotification(`Failed to load logs: ${data.error}`, "error");
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to refresh logs:", error);
|
||||
this.showNotification("Failed to refresh logs", "error");
|
||||
}
|
||||
}
|
||||
|
||||
displayLogs(logs) {
|
||||
const container = document.getElementById("logsContainer");
|
||||
|
||||
if (logs.length === 0) {
|
||||
container.innerHTML = '<div class="text-center py-4 text-pm-muted">No logs found</div>';
|
||||
return;
|
||||
}
|
||||
|
||||
container.innerHTML = logs.map(log => {
|
||||
const levelColors = {
|
||||
DEBUG: 'text-pm-secondary',
|
||||
INFO: 'text-pm-accent',
|
||||
WARNING: 'text-pm-warning',
|
||||
ERROR: 'text-pm-error',
|
||||
CRITICAL: 'text-pm-error'
|
||||
};
|
||||
|
||||
const levelColor = levelColors[log.level] || 'text-pm-secondary';
|
||||
const timestamp = new Date(log.timestamp).toLocaleString();
|
||||
|
||||
return `
|
||||
<div class="border-l-2 border-pm pl-3 py-1 hover:bg-pm-surface transition-colors">
|
||||
<div class="flex items-start space-x-2 text-sm">
|
||||
<span class="text-pm-muted text-xs font-mono w-24 flex-shrink-0">${escapeHtml(timestamp.split(' ')[1])}</span>
|
||||
<span class="${levelColor} font-semibold w-16 flex-shrink-0">${escapeHtml(log.level)}</span>
|
||||
<span class="text-pm-accent text-xs w-32 flex-shrink-0">${escapeHtml(log.logger)}</span>
|
||||
<span class="text-pm-secondary flex-1">${escapeHtml(log.message)}</span>
|
||||
</div>
|
||||
<div class="text-xs text-pm-muted ml-44">
|
||||
${escapeHtml(log.filename)}:${escapeHtml(log.lineno)}
|
||||
</div>
|
||||
</div>
|
||||
`;
|
||||
}).join('');
|
||||
|
||||
// Auto-scroll to bottom
|
||||
container.scrollTop = container.scrollHeight;
|
||||
}
|
||||
|
||||
async downloadLogs() {
|
||||
const level = document.getElementById("logLevel").value;
|
||||
const limit = parseInt(document.getElementById("logLimit").value);
|
||||
|
||||
try {
|
||||
const url = new URL(ApiClient.resolveUrl('/prompt_manager/logs'), window.location.origin);
|
||||
if (level) url.searchParams.set('level', level);
|
||||
url.searchParams.set('limit', limit.toString());
|
||||
|
||||
const response = await ApiClient.fetchUrl(url);
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
const logText = data.logs.map(log =>
|
||||
`${log.timestamp} [${log.level}] ${log.logger}: ${log.message} (${log.filename}:${log.lineno})`
|
||||
).join('\n');
|
||||
|
||||
const blob = new Blob([logText], { type: 'text/plain' });
|
||||
const url2 = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url2;
|
||||
a.download = `prompt_manager_logs_${new Date().toISOString().slice(0, 19).replace(/:/g, '-')}.txt`;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
URL.revokeObjectURL(url2);
|
||||
|
||||
this.showNotification("Logs downloaded successfully", "success");
|
||||
} else {
|
||||
this.showNotification(`Failed to download logs: ${data.error}`, "error");
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to download logs:", error);
|
||||
this.showNotification("Failed to download logs", "error");
|
||||
}
|
||||
}
|
||||
|
||||
async downloadLogFile(filename) {
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl(`/prompt_manager/logs/download/${encodeURIComponent(filename)}`);
|
||||
|
||||
if (response.ok) {
|
||||
const blob = await response.blob();
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = filename;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
URL.revokeObjectURL(url);
|
||||
|
||||
this.showNotification(`Downloaded ${filename}`, "success");
|
||||
} else {
|
||||
this.showNotification(`Failed to download ${filename}`, "error");
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to download log file:", error);
|
||||
this.showNotification("Failed to download log file", "error");
|
||||
}
|
||||
}
|
||||
|
||||
async clearLogs() {
|
||||
if (!confirm("Are you sure you want to clear all log files? This cannot be undone.")) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/logs/truncate', {
|
||||
method: 'POST'
|
||||
});
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
this.showNotification(data.message, "success");
|
||||
this.refreshLogs();
|
||||
this.loadLogStats();
|
||||
this.loadLogFiles();
|
||||
} else {
|
||||
this.showNotification(`Failed to clear logs: ${data.error}`, "error");
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to clear logs:", error);
|
||||
this.showNotification("Failed to clear logs", "error");
|
||||
}
|
||||
}
|
||||
|
||||
async updateLogConfig() {
|
||||
const level = document.getElementById("setLogLevel").value;
|
||||
const consoleLogging = document.getElementById("consoleLogging").checked;
|
||||
|
||||
try {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/logs/config', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
level: level,
|
||||
console_logging: consoleLogging
|
||||
})
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
this.showNotification("Log configuration updated", "success");
|
||||
this.loadLogStats();
|
||||
} else {
|
||||
this.showNotification(`Failed to update config: ${data.error}`, "error");
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to update log config:", error);
|
||||
this.showNotification("Failed to update log configuration", "error");
|
||||
}
|
||||
}
|
||||
|
||||
formatBytes(bytes) {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
|
||||
}
|
||||
|
||||
// Pagination methods
|
||||
updatePaginationControls() {
|
||||
const { currentPage, totalPages, total, limit } = this.pagination;
|
||||
@@ -3358,7 +3801,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
} else if (data.type === 'complete') {
|
||||
this.autoTagState.eventSource.close();
|
||||
this.hideModal("autoTagProgressModal");
|
||||
this.showNotification(ImageHelpers.autotagCompleteMessage(data.tagged), 'success');
|
||||
this.showNotification(`Auto tagging complete! Applied tags to ${data.tagged || 0} prompts.`, 'success');
|
||||
this.refreshList(); // Refresh the prompt list
|
||||
} else if (data.type === 'error') {
|
||||
this.autoTagState.eventSource.close();
|
||||
@@ -3376,7 +3819,7 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
console.error('EventSource readyState:', this.autoTagState.eventSource.readyState);
|
||||
this.autoTagState.eventSource.close();
|
||||
this.hideModal("autoTagProgressModal");
|
||||
this.showNotification((error && error.serverMessage) || 'Connection lost during auto tagging', 'error');
|
||||
this.showNotification('Connection lost during auto tagging', 'error');
|
||||
};
|
||||
} catch (error) {
|
||||
console.error('Auto tag error:', error);
|
||||
@@ -3451,14 +3894,10 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** POST /autotag/single for `image` (#379: `existing` "skip" answers already_tagged for an auto-tagged image). */
|
||||
async requestReviewTags(image, existing) {
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/single', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(ImageHelpers.autotagSingleBody(image, { ...this.autoTagState, existing }))
|
||||
});
|
||||
return response.json();
|
||||
// Check if tags array has "real" tags (excluding auto-scanned and prepend:* metadata)
|
||||
getRealTags(tags) {
|
||||
if (!tags || !Array.isArray(tags)) return [];
|
||||
return tags.filter(tag => tag !== 'auto-scanned' && !tag.startsWith('prepend:') && !tag.startsWith('append:'));
|
||||
}
|
||||
|
||||
// Show the re-tag confirmation modal
|
||||
@@ -3510,48 +3949,26 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
// always provides a servable url; never rebuild it from the path.
|
||||
const imageUrl = this.getImageUrl(image);
|
||||
|
||||
this.autoTagState.currentImage = image;
|
||||
this.autoTagState.currentSha256 = null;
|
||||
this.autoTagState.tagsExpanded = false; // Reset accordion for new image
|
||||
// Check if image already has real tags (excluding auto-scanned)
|
||||
const realTags = this.getRealTags(image.prompt_tags);
|
||||
|
||||
let data;
|
||||
try {
|
||||
// Each image is first asked with "skip": an image a tagging model already tagged answers
|
||||
// already_tagged (its image tags, never its prompt's) without running the model (#379).
|
||||
data = await this.requestReviewTags(image, 'skip');
|
||||
if (data.success && data.already_tagged) {
|
||||
if (this.autoTagState.skipAllTagged) {
|
||||
this.autoTagState.reviewIndex++;
|
||||
return this.loadNextReviewImage();
|
||||
}
|
||||
const choice = await this.showRetagConfirmation(image, imageUrl, data.image_tags || []);
|
||||
if (choice === 'skip' || choice === 'skipAll') {
|
||||
this.autoTagState.reviewIndex++;
|
||||
return this.loadNextReviewImage();
|
||||
}
|
||||
// choice === 'retag': generate new tags for it
|
||||
this.showReviewImage(image, imageUrl);
|
||||
data = await this.requestReviewTags(image, 'merge');
|
||||
} else {
|
||||
this.showReviewImage(image, imageUrl);
|
||||
if (realTags.length > 0) {
|
||||
// Skip if "Skip All Tagged" was selected
|
||||
if (this.autoTagState.skipAllTagged) {
|
||||
this.autoTagState.reviewIndex++;
|
||||
return this.loadNextReviewImage();
|
||||
}
|
||||
if (data.success) {
|
||||
this.autoTagState.currentTags = data.tags;
|
||||
this.autoTagState.currentSha256 = data.sha256 || null;
|
||||
this.renderReviewTags();
|
||||
} else {
|
||||
document.getElementById('reviewTagsVisible').innerHTML =
|
||||
`<div class="text-pm-error">Error: ${escapeHtml(data.error)}</div>`;
|
||||
|
||||
// Show confirmation modal
|
||||
const choice = await this.showRetagConfirmation(image, imageUrl, realTags);
|
||||
|
||||
if (choice === 'skip' || choice === 'skipAll') {
|
||||
this.autoTagState.reviewIndex++;
|
||||
return this.loadNextReviewImage();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error generating tags:', error);
|
||||
this.showReviewImage(image, imageUrl);
|
||||
document.getElementById('reviewTagsVisible').innerHTML =
|
||||
'<div class="text-pm-error">Failed to generate tags</div>';
|
||||
// choice === 'retag' - continue with tagging
|
||||
}
|
||||
}
|
||||
|
||||
showReviewImage(image, imageUrl) {
|
||||
document.getElementById('reviewImage').src = imageUrl;
|
||||
// The record's file name, else the last segment of its path (#328: `filename` was never declared).
|
||||
const filename = image.filename || String(image.image_path || image.path || '').split(/[\\/]/).pop();
|
||||
@@ -3560,6 +3977,37 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
document.getElementById('reviewTagsHidden').innerHTML = '';
|
||||
document.getElementById('reviewTagsAccordion').classList.add('hidden');
|
||||
document.getElementById('reviewTagsToggle').classList.add('hidden');
|
||||
|
||||
// Store prompt_id from database for applying tags later
|
||||
this.autoTagState.currentPromptId = image.prompt_id;
|
||||
this.autoTagState.tagsExpanded = false; // Reset accordion for new image
|
||||
|
||||
try {
|
||||
// Review rows come from /images/all (database ids), so this resolves to image_id
|
||||
const requestBody = ImageHelpers.autotagSingleBody(image, this.autoTagState);
|
||||
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/single', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(requestBody)
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
this.autoTagState.currentTags = data.tags;
|
||||
// Use prompt_id from database, fallback to API response
|
||||
this.autoTagState.currentPromptId = image.prompt_id || data.prompt_id;
|
||||
this.renderReviewTags();
|
||||
} else {
|
||||
document.getElementById('reviewTagsVisible').innerHTML =
|
||||
`<div class="text-pm-error">Error: ${escapeHtml(data.error)}</div>`;
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Error generating tags:', error);
|
||||
document.getElementById('reviewTagsVisible').innerHTML =
|
||||
'<div class="text-pm-error">Failed to generate tags</div>';
|
||||
}
|
||||
}
|
||||
|
||||
/** /prompt_manager/images/all is capped per call; collect every page. */
|
||||
@@ -3678,9 +4126,15 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
|
||||
async applyReviewTags() {
|
||||
// #379: the reviewed tags go to this image (linked or not), never its prompt; an empty
|
||||
// list still marks the image as reviewed by a tagging model.
|
||||
if (!this.autoTagState.currentSha256) {
|
||||
if (this.autoTagState.currentTags.length === 0) {
|
||||
this.skipReviewImage();
|
||||
return;
|
||||
}
|
||||
|
||||
const promptId = this.autoTagState.currentPromptId;
|
||||
|
||||
if (!promptId) {
|
||||
this.showNotification('No linked prompt found for this image', 'warning');
|
||||
this.skipReviewImage();
|
||||
return;
|
||||
}
|
||||
@@ -3689,8 +4143,10 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/apply', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(ImageHelpers.autotagApplyBody(
|
||||
this.autoTagState.currentImage, this.autoTagState.currentSha256, this.autoTagState.currentTags))
|
||||
body: JSON.stringify({
|
||||
prompt_id: promptId,
|
||||
tags: this.autoTagState.currentTags
|
||||
})
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
+16
-20
@@ -3560,24 +3560,19 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
} else if (data.type === 'complete') {
|
||||
this.autoTagState.eventSource.close();
|
||||
this.hideModal("autoTagProgressModal");
|
||||
this.showNotification(ImageHelpers.autotagCompleteMessage(data.tagged), 'success');
|
||||
this.showNotification(`Auto tagging complete! Applied tags to ${data.tagged || 0} prompts.`, 'success');
|
||||
this.loadImages(); // Refresh the gallery
|
||||
} else if (data.type === 'error') {
|
||||
this.autoTagState.eventSource.close();
|
||||
this.hideModal("autoTagProgressModal");
|
||||
this.showNotification(`Auto tag error: ${data.message}`, 'error');
|
||||
} else if (data.type === 'cancelled') {
|
||||
// Cancelled in Prompt Manager's Activity (#379): as its own cancel.
|
||||
this.autoTagState.eventSource.close();
|
||||
this.hideModal("autoTagProgressModal");
|
||||
this.showNotification('Auto tagging cancelled', 'info');
|
||||
}
|
||||
};
|
||||
|
||||
this.autoTagState.eventSource.onerror = (error) => {
|
||||
this.autoTagState.eventSource.onerror = () => {
|
||||
this.autoTagState.eventSource.close();
|
||||
this.hideModal("autoTagProgressModal");
|
||||
this.showNotification((error && error.serverMessage) || 'Connection lost during auto tagging', 'error');
|
||||
this.showNotification('Connection lost during auto tagging', 'error');
|
||||
};
|
||||
} catch (error) {
|
||||
console.error('Auto tag error:', error);
|
||||
@@ -3634,8 +3629,6 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
this.reviewRequest = request;
|
||||
this.autoTagState.currentTags = [];
|
||||
this.autoTagState.reviewPromptId = null;
|
||||
this.autoTagState.reviewImage = null;
|
||||
this.autoTagState.reviewSha256 = null;
|
||||
if (this.autoTagState.reviewIndex >= this.autoTagState.reviewImages.length) {
|
||||
this.hideModal("autoTagReviewModal");
|
||||
this.showNotification('Review complete!', 'success');
|
||||
@@ -3650,9 +3643,8 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
document.getElementById('reviewTagsContainer').innerHTML = '<div class="text-pm-secondary">Generating tags...</div>';
|
||||
|
||||
try {
|
||||
// Gallery entries come from /images/output: id is a path digest, so address by path.
|
||||
// The gallery review always suggests (it never skipped tagged images, #379).
|
||||
const requestBody = ImageHelpers.autotagSingleBody(image, { ...this.autoTagState, existing: 'merge' });
|
||||
// Gallery entries come from /images/output: id is a path digest, so address by path
|
||||
const requestBody = ImageHelpers.autotagSingleBody(image, this.autoTagState);
|
||||
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/single', {
|
||||
method: 'POST',
|
||||
@@ -3665,8 +3657,6 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
if (this.reviewRequest !== request) return;
|
||||
if (data.success) {
|
||||
this.autoTagState.reviewPromptId = data.prompt_id ?? null;
|
||||
this.autoTagState.reviewImage = image;
|
||||
this.autoTagState.reviewSha256 = data.sha256 || null;
|
||||
this.autoTagState.currentTags = data.tags;
|
||||
this.renderReviewTags();
|
||||
} else {
|
||||
@@ -3707,12 +3697,16 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
}
|
||||
|
||||
async applyReviewTags() {
|
||||
// #379: the reviewed tags go to this image (linked or not), never its prompt; an empty
|
||||
// list still marks the image as reviewed by a tagging model.
|
||||
if (!this.autoTagState.reviewSha256) {
|
||||
if (this.autoTagState.currentTags.length === 0) {
|
||||
this.skipReviewImage();
|
||||
return;
|
||||
}
|
||||
|
||||
const promptId = this.autoTagState.reviewPromptId;
|
||||
if (!promptId) {
|
||||
this.showNotification('This image has no linked prompt. Scan the output folder first.', 'warning');
|
||||
return;
|
||||
}
|
||||
const request = this.reviewRequest;
|
||||
if (this.reviewApplying === request) return;
|
||||
this.reviewApplying = request;
|
||||
@@ -3721,8 +3715,10 @@ Seed: ${this.currentMetadata.seed || 'Unknown'}`;
|
||||
const response = await ApiClient.fetchUrl('/prompt_manager/autotag/apply', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(ImageHelpers.autotagApplyBody(
|
||||
this.autoTagState.reviewImage, this.autoTagState.reviewSha256, this.autoTagState.currentTags))
|
||||
body: JSON.stringify({
|
||||
prompt_id: promptId,
|
||||
tags: this.autoTagState.currentTags
|
||||
})
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
+10
-44
@@ -70,28 +70,10 @@
|
||||
* path digest as id, which would 404, so those are addressed by path.
|
||||
*
|
||||
* @param {{id?: unknown, path?: unknown}} image
|
||||
* `existing` (#379): "skip" answers `already_tagged` for an image a model
|
||||
* already tagged; anything else is "merge" (always suggest).
|
||||
*
|
||||
* @param {{id?: unknown, path?: unknown}} image
|
||||
* @param {{modelType: string, generalThreshold?: number, characterThreshold?: number, prompt?: string, existing?: string}} settings
|
||||
* @param {{modelType: string, generalThreshold?: number, characterThreshold?: number, prompt?: string}} settings
|
||||
* @returns {object} JSON-serialisable request body
|
||||
*/
|
||||
function autotagSingleBody(image, settings) {
|
||||
const target = autotagTarget(image);
|
||||
const existing = settings.existing === "skip" ? "skip" : "merge";
|
||||
const modelType = String(settings.modelType || "");
|
||||
const params = modelType.startsWith("wd14")
|
||||
? {
|
||||
general_threshold: settings.generalThreshold,
|
||||
character_threshold: settings.characterThreshold,
|
||||
}
|
||||
: { prompt: settings.prompt };
|
||||
return { ...target, model_type: modelType, existing, ...params };
|
||||
}
|
||||
|
||||
/** The image a legacy single/apply body names: its row id, or its path (and root). */
|
||||
function autotagTarget(image) {
|
||||
const entry = image || {};
|
||||
const target =
|
||||
typeof entry.path === "string" && entry.path && !Number.isInteger(entry.id)
|
||||
@@ -100,22 +82,14 @@
|
||||
if ("path" in target && Number.isInteger(entry.root_index)) {
|
||||
target.root_index = entry.root_index;
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
/**
|
||||
* Body for POST /prompt_manager/autotag/apply (#379): the image, the
|
||||
* `sha256` /autotag/single answered and the reviewed tags. The tags go to
|
||||
* that image only, never to its prompt; an empty list marks it reviewed.
|
||||
*/
|
||||
function autotagApplyBody(image, sha256, tags) {
|
||||
return { ...autotagTarget(image), sha256: String(sha256 || ""), tags: Array.isArray(tags) ? [...tags] : [] };
|
||||
}
|
||||
|
||||
/** The legacy batch's completion toast (#379): images, never prompts. */
|
||||
function autotagCompleteMessage(tagged) {
|
||||
const count = Number.isSafeInteger(tagged) && tagged > 0 ? tagged : 0;
|
||||
return `Auto tagging complete! Tagged ${count} ${count === 1 ? "image" : "images"}.`;
|
||||
const modelType = String(settings.modelType || "");
|
||||
const params = modelType.startsWith("wd14")
|
||||
? {
|
||||
general_threshold: settings.generalThreshold,
|
||||
character_threshold: settings.characterThreshold,
|
||||
}
|
||||
: { prompt: settings.prompt };
|
||||
return { ...target, model_type: modelType, ...params };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -148,15 +122,7 @@
|
||||
}
|
||||
}
|
||||
|
||||
const api = {
|
||||
formatImageCaption,
|
||||
workflowDocumentHtml,
|
||||
autotagSingleBody,
|
||||
autotagApplyBody,
|
||||
autotagCompleteMessage,
|
||||
fallbackImageSource,
|
||||
metadataLookupUrl,
|
||||
};
|
||||
const api = { formatImageCaption, workflowDocumentHtml, autotagSingleBody, fallbackImageSource, metadataLookupUrl };
|
||||
if (typeof module !== "undefined" && module.exports) module.exports = api;
|
||||
else root.ImageHelpers = api;
|
||||
})(typeof window !== "undefined" ? window : globalThis);
|
||||
|
||||
+1
-15
@@ -59,20 +59,6 @@
|
||||
* @returns {{onmessage: Function|null, onerror: Function|null, onclose: Function|null,
|
||||
* readyState: number, close: Function, CONNECTING: number, OPEN: number, CLOSED: number}}
|
||||
*/
|
||||
// A non-2xx answer as an Error carrying its status and, when the body is
|
||||
// the API's JSON {error}, that sentence as serverMessage for the page.
|
||||
async function httpError(response) {
|
||||
const error = new Error(`HTTP ${response.status}`);
|
||||
error.status = response.status;
|
||||
try {
|
||||
const data = await response.json();
|
||||
if (data && typeof data.error === "string" && data.error) error.serverMessage = data.error;
|
||||
} catch (_) {
|
||||
// Not JSON: the status alone.
|
||||
}
|
||||
return error;
|
||||
}
|
||||
|
||||
function connect(url, { method = "GET", body, fetchImpl } = {}) {
|
||||
const doFetch = fetchImpl || ((...args) => globalThis.fetch(...args));
|
||||
const controller = new AbortController();
|
||||
@@ -117,7 +103,7 @@
|
||||
if (body !== undefined) init.body = body;
|
||||
const requestUrl = root.ApiClient ? root.ApiClient.resolveUrl(url) : url;
|
||||
const response = await doFetch(requestUrl, init);
|
||||
if (!response.ok) throw await httpError(response);
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
if (source.readyState === CLOSED) return;
|
||||
source.readyState = OPEN;
|
||||
|
||||
|
||||
@@ -275,13 +275,8 @@ export function createImageStream({ api, onChange = () => {}, batchSize = 40, re
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The concept's count line (app.js a446): "N of M images · K positive prompts", without the count while unknown, and
|
||||
* the page's first "Loading images…" while a first page loads.
|
||||
*/
|
||||
export function countLine({ loaded, total, positiveCount, loading = false, error = null }) {
|
||||
// A first page in flight has no count yet: "0 of 0 images" would claim the filter matched nothing (#391).
|
||||
if (loading && !loaded && !error) return 'Loading images…';
|
||||
/** The concept's count line (app.js a446): "N of M images · K positive prompts", without the count while unknown. */
|
||||
export function countLine({ loaded, total, positiveCount }) {
|
||||
const images = `${loaded.toLocaleString('en-US')} of ${total.toLocaleString('en-US')} ${noun(total, 'image')}`;
|
||||
return isCount(positiveCount) ? `${images} · ${positiveCount.toLocaleString('en-US')} ${noun(positiveCount, 'positive prompt')}` : images;
|
||||
}
|
||||
|
||||
@@ -225,23 +225,6 @@ export function clearSentence(files) {
|
||||
return `Empties ${ACTIVE_LOG}, deletes ${number(rotated.length)} rotated ${noun(rotated.length, 'copy', 'copies')} (${sizeText(size)}) ${tail}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* #389: the `/logs/truncate` body for the files a confirmation showed. Their active and rotated rows go back with the
|
||||
* server's `identity`, so the server deletes only those and refuses (409) when any changed meanwhile. A list that could
|
||||
* not be read (`null`, the general sentence) sends no list: the server then clears what is there, as the sentence says.
|
||||
*/
|
||||
export function clearBody(files) {
|
||||
if (!Array.isArray(files)) return {};
|
||||
const expected = files.filter(row => (row?.kind === 'active' || row?.kind === 'rotated') && typeof row.filename === 'string'
|
||||
&& typeof row.identity === 'string').map(row => ({ name: row.filename, identity: row.identity }));
|
||||
return { expected };
|
||||
}
|
||||
|
||||
/** A refused clear (409) says why in the server's words; any other failure the general line. */
|
||||
export function clearFailureText(error) {
|
||||
return error?.status === 409 && typeof error.message === 'string' && error.message ? error.message : COPY.clearFailed;
|
||||
}
|
||||
|
||||
/** CUJ 7: the toast for `/logs/truncate`'s `results` ({truncated: [names], errors: [messages]}). */
|
||||
export function clearToast(results) {
|
||||
const truncated = Array.isArray(results?.truncated) ? results.truncated : [];
|
||||
@@ -486,14 +469,13 @@ function logsSurface({ api, activity, scope, drafts, navigate, notify, latestOnl
|
||||
const list = el('div', undefined, 'operation-list');
|
||||
let listed = null;
|
||||
// The confirmation names what Clear deletes now: the files are read again before it opens (a read that fails
|
||||
// gives the general sentence, never an old count). Clear sends exactly the files the dialog was built from (#389).
|
||||
// gives the general sentence, never an old count).
|
||||
const clear = operationRow({ scope, title: 'Clear all logs', description: COPY.clearDescription, label: 'Clear all logs…',
|
||||
danger: true, onClick: button => pressing(button, async () => {
|
||||
await readFiles();
|
||||
if (scope.signal.aborted) return;
|
||||
const shown = listed;
|
||||
confirmDialog({ scope, trigger: button, title: 'Clear all logs', sentence: clearSentence(shown), label: 'Clear',
|
||||
danger: true, onConfirm: () => clearLogs(shown) });
|
||||
confirmDialog({ scope, trigger: button, title: 'Clear all logs', sentence: clearSentence(listed), label: 'Clear',
|
||||
danger: true, onConfirm: clearLogs });
|
||||
}) });
|
||||
files.append(el('summary', 'Log files & retention'), line, review, retention, list, clear);
|
||||
const runtime = runtimeRows({ scope, drafts });
|
||||
@@ -580,12 +562,12 @@ function logsSurface({ api, activity, scope, drafts, navigate, notify, latestOnl
|
||||
readConfig();
|
||||
}
|
||||
|
||||
async function clearLogs(shown) {
|
||||
async function clearLogs() {
|
||||
let answer;
|
||||
try {
|
||||
answer = await api.post(TRUNCATE_URL, clearBody(shown));
|
||||
} catch (error) {
|
||||
if (!scope.signal.aborted) notify(clearFailureText(error));
|
||||
answer = await api.post(TRUNCATE_URL, {});
|
||||
} catch {
|
||||
if (!scope.signal.aborted) notify(COPY.clearFailed);
|
||||
}
|
||||
if (scope.signal.aborted) return;
|
||||
if (answer) notify(clearToast(answer.results));
|
||||
|
||||
@@ -433,13 +433,10 @@ pre.prompt-text { font: inherit; font-size: 12px; line-height: 1.75; margin-bloc
|
||||
}
|
||||
/* D132: at phone width the concept's 120px cap cuts every engine label ("WD14 SwinV2 · ~400 MB"); the Tagging model
|
||||
select alone keeps its natural width there (its widest engine label) and, with no room beside the row text, wraps
|
||||
under it. Existing tags (#398) never shrinks for its row text and, past the cap, keeps its natural width beside the
|
||||
text ("Replace after review"), bounded by its row like the engine select. */
|
||||
.setting-row #tag-existing { flex-shrink: 0; /* D132 */ }
|
||||
under it. */
|
||||
@media (max-width: 700px) {
|
||||
#workspace-main .setting-row:has(> #tag-model) { flex-wrap: wrap; /* D132 */ }
|
||||
#workspace-main .setting-row #tag-model { flex-shrink: 0; max-width: 100% !important; /* D132 */ }
|
||||
#workspace-main .setting-row #tag-existing { max-width: 100% !important; /* D132 */ }
|
||||
}
|
||||
|
||||
/* D109: System › Diagnostics & logs' added Lines field sits in the concept's toolbar like its Level select, and a value
|
||||
|
||||
Reference in New Issue
Block a user