Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e2c1cc8fc9 | ||
|
|
66b3d078db | ||
|
|
f95c7adfc9 | ||
|
|
a32e66729b |
+60
-2
@@ -550,8 +550,9 @@ Recorded shell deviations:
|
||||
configured sources.", button "Review scan"). It opens the concept's `scan-review`,
|
||||
"Review metadata scan": no sample-plan `.demo-note` (prototype copy); Sources and
|
||||
Scope in a `dl.meta-grid` (the concept's maintenance plan uses `dl`; `scan-review`
|
||||
uses a `div`) reading "N output folders" (the discovered sources) and "All files"
|
||||
(fixed in Activity-1; Activity-2 adds the scope option); the concept's sentence; then
|
||||
uses a `div`) reading "N output folders" (the discovered sources) and the concept's
|
||||
"New and unresolved files" (the D58 checkbox switches it to "All files"); the
|
||||
concept's sentence; then
|
||||
a backup `.notice` (the concept's System notice box): `h3` "Back up the database first
|
||||
(recommended)", "Downloads a copy of the prompts database before the scan changes any
|
||||
records." and "Back up now", which downloads the existing `/prompt_manager/backup`
|
||||
@@ -624,6 +625,63 @@ Recorded shell deviations:
|
||||
is running: <title>." with the title linking that job's view (the shared toast now
|
||||
accepts link parts). A toast with a link stays 20 s instead of 6.5 s, and any toast
|
||||
stays while the pointer or focus is on it.
|
||||
- **D58** (scan scope, Activity-2 Task 4): the concept's `scan-review` states a fixed
|
||||
scope, "New and unresolved files". Production keeps that value as the default and adds
|
||||
the concept's own System toggle after the `dl.meta-grid`: a `.setting-row` with the
|
||||
label "All files", the hint "Also read the files already linked to their prompt
|
||||
again." (the switch's `aria-describedby`) and the Toolkit switch (`label.ca-switch`, a
|
||||
checkbox with `role="switch"`, as the concept's `hydrateSwitches` renders it),
|
||||
unchecked. Checking it switches the Scope value to "All files"; Start scan sends
|
||||
`POST /prompt_manager/jobs/scan {scope: "new" | "all"}` (no body, an empty body or no
|
||||
`scope` is "new"; another value is a 400).
|
||||
- In "new" the scan skips, before reading any metadata, the files the Library already
|
||||
links with a resolved link and that are unchanged on disk: exactly one
|
||||
`generated_images` row for the file, exactly one distinct pair link on it
|
||||
(`image_prompt_pairs`, written only by resolved ingestion), and a stored
|
||||
`file_size` equal to the file's size in this walk. Both paths go through
|
||||
`normalize_image_path` (absolute, normalized, `os.path.normcase`); a row without
|
||||
`file_path` matches by its `image_path`.
|
||||
- Read again: files linked without a pair (the monitor's fallback and FIFO links, the
|
||||
legacy positive-only import), linked to more than one prompt or pair, not linked, a
|
||||
file whose size changed (one replaced at the same path, e.g. a reused ComfyUI
|
||||
counter name, is read and its conflict lands under Needs review), and a row with no
|
||||
stored size (unknown is not known). The scan now stores each file's size when it
|
||||
ingests it, so a file read once is skipped by the next "new" pass; the first pass
|
||||
after the upgrade fills the missing sizes, which reports those files as enriched
|
||||
(Added) instead of unchanged.
|
||||
- The known set is one streamed query that keeps only the scanned paths. If it cannot
|
||||
be read, the scan logs "Known files could not be read · reading every file" and
|
||||
reads every file.
|
||||
- A skipped file counts as processed and unchanged in the legacy counts (not as
|
||||
found) and under Already known, with the detail "Skipped · already linked" and the
|
||||
log line "= <root>/<path> · Skipped · already linked"; the log's first line names
|
||||
the scope ("Scan started · N output folders · new and unresolved files" or "· all
|
||||
files"). A legacy-page scan that attaches to a running "new" job therefore shows
|
||||
the skipped files in its `unchanged` count.
|
||||
- "All files" reads every file, as the scan did in Activity-1, and the legacy
|
||||
`/prompt_manager/scan` route always does when it starts the scan (its SSE payload
|
||||
keys are unchanged).
|
||||
- `activity-concept.spec.mjs` compares the switch with the concept's System → Sources
|
||||
toggle ("Watch for new outputs"), without its place-dependent top margin;
|
||||
`workspace-activity.spec.mjs` covers the Scope value, the description and the
|
||||
request both ways; `tests/test_scan_scope.py` covers the skip, the size rule, the
|
||||
replaced file, the re-read of unresolved files, "all", the legacy route, the
|
||||
fallback, the empty body, the 400 and Windows case folding.
|
||||
- **D59** (job report and log, Activity-2 Task 5): the job view's "Download report" is
|
||||
enabled. It fetches `GET /prompt_manager/jobs/<id>/report` through the api-client URL
|
||||
(so a proxy mount works) and saves a JSON attachment named
|
||||
`prompt-manager-<kind>-<YYYYMMDD-HHMM>.json` (server local time), then toasts "Job
|
||||
report downloaded." (or "The report could not be downloaded. Try again."). The report
|
||||
holds the summary, every counter, each outcome group's exact `count`, its listed
|
||||
`rows` (at most 200) and a `truncated` flag, the scope and the log; rows and log lines
|
||||
carry root-relative paths only, never an absolute path (an unknown id is a 404).
|
||||
A job lost with a restart has no report on the server, so its Download report is
|
||||
disabled; a view left while the report downloads still saves it but shows no toast.
|
||||
The scan log opens at its newest line every time it is opened and follows new lines
|
||||
only while the reader is at the bottom; a reader who scrolled up keeps the position
|
||||
while it stays open (checked on Chromium, Firefox and WebKit).
|
||||
`test_report_is_attachment_without_absolute_paths` and the browser tests `download
|
||||
report saves the job JSON` and `log keeps the reader's scroll position` cover it.
|
||||
- **D29**: no prototype copy. There is no "Interactive concept" badge and no
|
||||
"What changes & why" button. The footer reads "ComfyAssets · Local tools. Your
|
||||
library.", and the title has no " concept" suffix. Save view's dialog action reads
|
||||
|
||||
@@ -181,6 +181,21 @@ AND p.id NOT IN (SELECT prompt_id FROM live_hint_prompts)
|
||||
"""
|
||||
|
||||
|
||||
# The scan's known set (#249 Activity-2): every image row, with its stored file
|
||||
# size and whether its pair link is resolved: exactly one distinct pair, the rule
|
||||
# _LINK_SUMMARY_SQL and get_image_prompt_info use. A file is known and resolved
|
||||
# when it has exactly one image row, that row is resolved, and the row's stored
|
||||
# size equals the file's size now; several rows (one file linked to more than
|
||||
# one prompt) are ambiguous, and an unknown (NULL) stored size is not known. The
|
||||
# rows are streamed in fetches of KNOWN_PATHS_FETCH_SIZE and only the scanned
|
||||
# paths are kept, so memory follows the scan (at most MAX_SCAN_FILES paths), not
|
||||
# the table.
|
||||
_KNOWN_PATHS_SQL = """SELECT gi.file_path, gi.image_path, gi.file_size,
|
||||
(SELECT COUNT(DISTINCT ip.pair_id) FROM image_prompt_pairs ip
|
||||
WHERE ip.image_id = gi.id) = 1
|
||||
FROM generated_images gi"""
|
||||
KNOWN_PATHS_FETCH_SIZE = 1000
|
||||
|
||||
# One SELECT reads the link summaries of at most this many paths: three bound
|
||||
# parameters per path stays under SQLite's historical 999-variable limit.
|
||||
LINK_SUMMARY_BATCH = 200
|
||||
@@ -2280,6 +2295,52 @@ class PromptDatabase(
|
||||
"parameters": row[8] if len(row) > 8 else None,
|
||||
}
|
||||
|
||||
def get_known_resolved_paths(self, files: Dict[str, Optional[int]]) -> Set[str]:
|
||||
"""The paths in ``files`` the Library already links with a resolved link.
|
||||
|
||||
``files`` maps each scanned path to its size on disk (None when it could
|
||||
not be read). A path qualifies when exactly one ``generated_images`` row
|
||||
claims the file, that row has exactly one pair link
|
||||
(``image_prompt_pairs``, written only by resolved ingestion), and its
|
||||
stored ``file_size`` is not NULL and equals the given size, so a file
|
||||
replaced at the same path is read again. Files linked without a pair
|
||||
(the monitor's fallback and FIFO links, the legacy positive-only
|
||||
import), linked to several prompts or pairs, or not linked at all do
|
||||
not qualify. Both sides go through ``normalize_image_path`` (absolute,
|
||||
normalized, ``os.path.normcase``), so a Windows row stored in another
|
||||
case or spelling still matches. One query, streamed; returns the
|
||||
matching keys of ``files``.
|
||||
"""
|
||||
wanted: Dict[str, List[Tuple[str, Optional[int]]]] = {}
|
||||
for path, size in files.items():
|
||||
key = normalize_image_path(str(path))
|
||||
wanted.setdefault(key, []).append((str(path), size))
|
||||
if not wanted:
|
||||
return set()
|
||||
rows_seen: Dict[str, int] = {}
|
||||
stored_sizes: Dict[str, int] = {}
|
||||
with self.model.get_connection() as conn:
|
||||
cursor = conn.execute(_KNOWN_PATHS_SQL)
|
||||
while True:
|
||||
rows = cursor.fetchmany(KNOWN_PATHS_FETCH_SIZE)
|
||||
if not rows:
|
||||
break
|
||||
for file_path, image_path, file_size, is_resolved in rows:
|
||||
# image_path is NOT NULL; file_path is NULL only on old rows.
|
||||
key = normalize_image_path(file_path or image_path)
|
||||
if key not in wanted:
|
||||
continue
|
||||
rows_seen[key] = rows_seen.get(key, 0) + 1
|
||||
if is_resolved and file_size is not None:
|
||||
stored_sizes[key] = file_size
|
||||
return {
|
||||
path
|
||||
for key, stored_size in stored_sizes.items()
|
||||
if rows_seen[key] == 1
|
||||
for path, size in wanted[key]
|
||||
if size is not None and size == stored_size
|
||||
}
|
||||
|
||||
def get_image_link_summaries(
|
||||
self, paths: List[str], allow_filename_fallback: bool = True
|
||||
) -> Dict[str, Dict[str, Any]]:
|
||||
|
||||
+88
-21
@@ -17,7 +17,14 @@ from pathlib import Path
|
||||
from aiohttp import web
|
||||
|
||||
from .prompts import thumbnail_url_for
|
||||
from .jobs import SCAN_GROUP_NAMES, SCAN_GROUPS, Job, JobBusy
|
||||
from .jobs import (
|
||||
DEFAULT_SCAN_SCOPE,
|
||||
SCAN_GROUP_NAMES,
|
||||
SCAN_GROUPS,
|
||||
SCAN_SCOPES,
|
||||
Job,
|
||||
JobBusy,
|
||||
)
|
||||
|
||||
try:
|
||||
from ...utils.parallel import map_parallel
|
||||
@@ -150,6 +157,8 @@ SCAN_FINAL_STATES = {
|
||||
"cancelled": "cancelled",
|
||||
"error": "failed",
|
||||
}
|
||||
# The scope as the scan log's first line names it (the review dialog's words).
|
||||
SCAN_SCOPE_LOG = {"new": "new and unresolved files", "all": "all files"}
|
||||
MIN_BATCHES_FOR_ESTIMATE = 2 # "about X min left" waits for this many batches
|
||||
# Log line prefix per outcome group (spec: + added, = known, ? review, ! error,
|
||||
# - no usable metadata).
|
||||
@@ -174,6 +183,7 @@ SCAN_DETAILS = {
|
||||
"linked": "Linked to an existing prompt",
|
||||
"enriched": "Metadata added to its prompt",
|
||||
"unchanged": "Already linked to its prompt",
|
||||
"skipped": "Skipped · already linked",
|
||||
"ambiguous": "Needs review: more than one prompt could match",
|
||||
"unresolved": "Needs review: the prompt could not be traced in the workflow",
|
||||
"conflict": "Needs review: already linked to a different prompt",
|
||||
@@ -197,6 +207,30 @@ def _scan_status_line(done, total, batches, elapsed):
|
||||
return f"{line} · about {minutes} min left"
|
||||
|
||||
|
||||
def _file_size(path):
|
||||
"""``path``'s size on disk, or None when it cannot be read."""
|
||||
try:
|
||||
return os.path.getsize(path)
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def _with_file_size(canonical, media_file):
|
||||
"""Record the file's size in ``file_info`` unless the metadata carries one.
|
||||
|
||||
Stored as ``generated_images.file_size``; the "new" scan scope skips a known
|
||||
file only while its size is unchanged.
|
||||
"""
|
||||
info = canonical.get("file_info")
|
||||
info = dict(info) if isinstance(info, dict) else {}
|
||||
if info.get("size") is None:
|
||||
size = _file_size(media_file)
|
||||
if size is not None:
|
||||
info["size"] = size
|
||||
if info:
|
||||
canonical["file_info"] = info
|
||||
|
||||
|
||||
def _scan_roots(output_dirs):
|
||||
"""(absolute root, public label, listing index), the most specific first.
|
||||
|
||||
@@ -1426,7 +1460,7 @@ class AdminRoutesMixin:
|
||||
job = self._jobs.active(SCAN_JOB_KIND)
|
||||
if job is None:
|
||||
try:
|
||||
job = self._start_scan_job()
|
||||
job = self._start_scan_job("all")
|
||||
except JobBusy as busy:
|
||||
return self._long_job_busy_response(busy.running.title)
|
||||
return await self._stream_scan_job(request, job)
|
||||
@@ -1450,16 +1484,19 @@ class AdminRoutesMixin:
|
||||
}
|
||||
)
|
||||
|
||||
def _start_scan_job(self):
|
||||
def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE):
|
||||
"""Register the scan job synchronously; its runner starts on the loop.
|
||||
|
||||
Raises JobBusy while another record-changing job runs.
|
||||
``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another
|
||||
record-changing job runs.
|
||||
"""
|
||||
if files not in SCAN_SCOPES:
|
||||
raise ValueError(f"Unknown scan scope: {files!r}")
|
||||
job = self._jobs.start(
|
||||
SCAN_JOB_KIND,
|
||||
SCAN_JOB_TITLE,
|
||||
record_changing=True,
|
||||
scope={"mode": "all"},
|
||||
scope={"files": files},
|
||||
runner=self._run_scan_job,
|
||||
groups=SCAN_GROUP_NAMES,
|
||||
)
|
||||
@@ -1525,7 +1562,8 @@ class AdminRoutesMixin:
|
||||
dir_names = [self._public_path(d) for d in output_dirs]
|
||||
folders = _plural(len(output_dirs), "output folder")
|
||||
job.scope["output_folders"] = len(output_dirs)
|
||||
job.log_line(f"Scan started · {folders}")
|
||||
scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")]
|
||||
job.log_line(f"Scan started · {folders} · {scope}")
|
||||
job.update(status_line=f"Looking for media files in {folders}")
|
||||
yield self._scan_progress(
|
||||
0, f"Scanning {len(output_dirs)} directory(ies) for media files..."
|
||||
@@ -1544,7 +1582,12 @@ class AdminRoutesMixin:
|
||||
5, f"Found {len(media_files)} media files to process...", counts
|
||||
)
|
||||
roots = _scan_roots(output_dirs)
|
||||
async for event in self._scan_batches(media_files, counts, job, roots):
|
||||
known = set()
|
||||
if media_files and job.scope.get("files") != "all":
|
||||
known = await self._known_scan_files(job, media_files)
|
||||
async for event in self._scan_batches(
|
||||
media_files, counts, job, roots, known
|
||||
):
|
||||
yield event
|
||||
|
||||
self.logger.info(
|
||||
@@ -1565,11 +1608,29 @@ class AdminRoutesMixin:
|
||||
"message": "An internal error occurred. Check server logs for details.",
|
||||
}
|
||||
|
||||
async def _scan_batches(self, media_files, counts, job, roots=()):
|
||||
async def _known_scan_files(self, job, media_files):
|
||||
"""The "new" scope's skip set; a failed read falls back to reading all."""
|
||||
try:
|
||||
return await self._run_in_executor(self._known_scan_files_sync, media_files)
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
"Known files could not be read; the scan reads every file",
|
||||
exc_info=True,
|
||||
)
|
||||
job.log_line("Known files could not be read · reading every file")
|
||||
return set()
|
||||
|
||||
def _known_scan_files_sync(self, media_files):
|
||||
"""Stat each scanned file and ask the database which are known (blocking)."""
|
||||
sizes = {str(media_file): _file_size(media_file) for media_file in media_files}
|
||||
return self.db.get_known_resolved_paths(sizes)
|
||||
|
||||
async def _scan_batches(self, media_files, counts, job, roots=(), known=()):
|
||||
"""Process ``media_files`` in executor batches, updating ``counts`` in place.
|
||||
|
||||
Records each file's outcome in ``job`` and yields one progress event
|
||||
per batch.
|
||||
per batch. Files in ``known`` (path strings) are not read: they count
|
||||
as processed and unchanged and are recorded as skipped.
|
||||
"""
|
||||
total = len(media_files)
|
||||
started = time.monotonic()
|
||||
@@ -1578,18 +1639,23 @@ class AdminRoutesMixin:
|
||||
if job.cancelled:
|
||||
return
|
||||
batch = media_files[batch_start : batch_start + SCAN_BATCH_SIZE]
|
||||
batch_results = await self._run_in_executor(
|
||||
self._extract_batch_metadata_sync, batch
|
||||
)
|
||||
outcomes = []
|
||||
batch_counts = await self._run_in_executor(
|
||||
self._ingest_scan_batch_sync,
|
||||
batch_results,
|
||||
roots=roots,
|
||||
outcomes=outcomes,
|
||||
)
|
||||
for key in counts:
|
||||
counts[key] += batch_counts[key]
|
||||
skipped = [f for f in batch if str(f) in known]
|
||||
to_read = [f for f in batch if str(f) not in known]
|
||||
outcomes = [_scan_outcome(f, roots, "skipped") for f in skipped]
|
||||
counts["processed"] += len(skipped)
|
||||
counts["unchanged"] += len(skipped)
|
||||
if to_read:
|
||||
batch_results = await self._run_in_executor(
|
||||
self._extract_batch_metadata_sync, to_read
|
||||
)
|
||||
batch_counts = await self._run_in_executor(
|
||||
self._ingest_scan_batch_sync,
|
||||
batch_results,
|
||||
roots=roots,
|
||||
outcomes=outcomes,
|
||||
)
|
||||
for key in counts:
|
||||
counts[key] += batch_counts[key]
|
||||
for group, row, log_line in outcomes:
|
||||
job.record(group, row, log_line)
|
||||
|
||||
@@ -1713,6 +1779,7 @@ class AdminRoutesMixin:
|
||||
pass
|
||||
if not canonical.get("prompt") and not canonical.get("parameters"):
|
||||
return None
|
||||
_with_file_size(canonical, media_file)
|
||||
resolution = resolve_prompt_pair(canonical)
|
||||
if resolution["state"] == "resolved":
|
||||
result = ingest_resolved_pair(
|
||||
|
||||
+55
-3
@@ -10,13 +10,38 @@ import json
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from .jobs import JobBusy
|
||||
from .jobs import DEFAULT_SCAN_SCOPE, SCAN_SCOPES, JobBusy
|
||||
|
||||
|
||||
def _not_found():
|
||||
return web.json_response({"success": False, "error": "Job not found"}, status=404)
|
||||
|
||||
|
||||
def _bad_request(message):
|
||||
return web.json_response({"success": False, "error": message}, status=400)
|
||||
|
||||
|
||||
async def _scan_scope(request):
|
||||
"""The requested scan scope ("new" or "all"), or None when the body is invalid.
|
||||
|
||||
An empty body (also a chunked one with no bytes), or one without
|
||||
``scope``, asks for the default.
|
||||
"""
|
||||
if not request.can_read_body:
|
||||
return DEFAULT_SCAN_SCOPE
|
||||
text = await request.text()
|
||||
if not text.strip(): # e.g. a chunked POST that sent no bytes
|
||||
return DEFAULT_SCAN_SCOPE
|
||||
try:
|
||||
body = json.loads(text)
|
||||
except ValueError:
|
||||
return None
|
||||
if not isinstance(body, dict):
|
||||
return None
|
||||
scope = body.get("scope", DEFAULT_SCAN_SCOPE)
|
||||
return scope if scope in SCAN_SCOPES else None
|
||||
|
||||
|
||||
def _summary_event(job):
|
||||
return f"data: {json.dumps(job.summary())}\n\n".encode("utf-8")
|
||||
|
||||
@@ -37,6 +62,10 @@ class JobRoutesMixin:
|
||||
async def job_detail_route(request):
|
||||
return await self.job_detail(request)
|
||||
|
||||
@routes.get("/prompt_manager/jobs/{job_id}/report")
|
||||
async def job_report_route(request):
|
||||
return await self.job_report(request)
|
||||
|
||||
@routes.get("/prompt_manager/jobs/{job_id}/events")
|
||||
async def job_events_route(request):
|
||||
return await self.job_events(request)
|
||||
@@ -56,6 +85,22 @@ class JobRoutesMixin:
|
||||
return _not_found()
|
||||
return web.json_response({"success": True, "job": job.detail()})
|
||||
|
||||
async def job_report(self, request):
|
||||
"""The job's whole record as a JSON attachment (no absolute paths)."""
|
||||
job = self._jobs.get(request.match_info["job_id"])
|
||||
if job is None:
|
||||
return _not_found()
|
||||
body = json.dumps(job.report(), indent=2)
|
||||
disposition = f'attachment; filename="{job.report_filename()}"'
|
||||
return web.Response(
|
||||
text=body,
|
||||
content_type="application/json",
|
||||
headers={
|
||||
"Content-Disposition": disposition,
|
||||
"Cache-Control": "no-store",
|
||||
},
|
||||
)
|
||||
|
||||
async def cancel_job(self, request):
|
||||
"""Stop a job at its next file boundary; a finished job is a no-op."""
|
||||
job_id = request.match_info["job_id"]
|
||||
@@ -65,9 +110,16 @@ class JobRoutesMixin:
|
||||
return web.json_response({"success": True, "cancellation_requested": requested})
|
||||
|
||||
async def start_scan_job(self, request):
|
||||
"""Start the metadata scan as a job (202), or 409 naming the busy job."""
|
||||
"""Start the metadata scan as a job (202), or 409 naming the busy job.
|
||||
|
||||
The body's ``scope`` is "new" (the default: new and unresolved files)
|
||||
or "all"; anything else is a 400.
|
||||
"""
|
||||
files = await _scan_scope(request)
|
||||
if files is None:
|
||||
return _bad_request('The scan scope must be "new" or "all"')
|
||||
try:
|
||||
job = self._start_scan_job()
|
||||
job = self._start_scan_job(files)
|
||||
except JobBusy as busy:
|
||||
running = busy.running
|
||||
return web.json_response(
|
||||
|
||||
@@ -35,12 +35,19 @@ MAX_FINISHED_JOBS = 20 # finished jobs this session keeps
|
||||
# The metadata scan's outcome groups, in counter-tile order of the spec's
|
||||
# mapping, and the per-file scan status each group collects. Files without
|
||||
# usable metadata have no status and go to "no_metadata".
|
||||
# Which files a scan reads (#249 Activity-2): "new" skips the files the Library
|
||||
# already links with a resolved link (get_known_resolved_paths) before reading
|
||||
# them, so new, unlinked, unresolved and ambiguous files are read; "all" reads
|
||||
# every file again. The workspace job defaults to "new"; legacy /scan is "all".
|
||||
SCAN_SCOPES = ("new", "all")
|
||||
DEFAULT_SCAN_SCOPE = "new"
|
||||
SCAN_GROUP_NAMES = ("added", "known", "review", "no_metadata", "errors")
|
||||
SCAN_GROUPS = {
|
||||
"added": "added",
|
||||
"linked": "added",
|
||||
"enriched": "added",
|
||||
"unchanged": "known",
|
||||
"skipped": "known", # "new" scope: known and resolved, not read again
|
||||
"ambiguous": "review",
|
||||
"conflict": "review",
|
||||
"unresolved": "review",
|
||||
@@ -240,6 +247,36 @@ class Job:
|
||||
"log": list(self.log),
|
||||
}
|
||||
|
||||
def report(self):
|
||||
"""The downloadable report: the summary, every outcome group and the log.
|
||||
|
||||
Each group carries its exact count, its listed rows (at most
|
||||
MAX_OUTCOME_ROWS) and whether the list is cut short. Rows and log lines
|
||||
already hold root-relative paths only.
|
||||
"""
|
||||
summary = self.summary()
|
||||
summary.pop("listed", None)
|
||||
summary.pop("log_length", None)
|
||||
return {
|
||||
"report_version": 1,
|
||||
"generated_at": self._clock(),
|
||||
"job": summary,
|
||||
"outcomes": {
|
||||
group: {
|
||||
"count": self.counters.get(group, len(rows)),
|
||||
"rows": [dict(row) for row in rows],
|
||||
"truncated": self.counters.get(group, len(rows)) > len(rows),
|
||||
}
|
||||
for group, rows in self.outcomes.items()
|
||||
},
|
||||
"log": list(self.log),
|
||||
}
|
||||
|
||||
def report_filename(self):
|
||||
"""The download name: prompt-manager-KIND-YYYYMMDD-HHMM.json (local time)."""
|
||||
stamp = time.strftime("%Y%m%d-%H%M", time.localtime(self._clock()))
|
||||
return f"prompt-manager-{self.kind}-{stamp}.json"
|
||||
|
||||
|
||||
class JobRegistry:
|
||||
"""Every job of this server session, running and recently finished."""
|
||||
|
||||
@@ -63,16 +63,35 @@ function measureNotice(selector) {
|
||||
};
|
||||
}
|
||||
|
||||
/** D55: the concept's dialog with the production differences applied. */
|
||||
/** A concept `.setting-row` switch: its frame, its label and hint, and its checkbox (not its place-dependent margin). */
|
||||
function measureToggle(selector) {
|
||||
const row = document.querySelector(selector);
|
||||
const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null;
|
||||
return {
|
||||
children: [...row.children].map(node => [node.localName, ...node.classList].join('.')),
|
||||
frame: css(row, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom',
|
||||
'borderTopWidth', 'borderTopStyle', 'borderTopColor'),
|
||||
label: css(row.querySelector('label'), 'fontSize', 'fontWeight', 'color'),
|
||||
hint: css(row.querySelector('p'), 'fontSize', 'color', 'marginTop'),
|
||||
box: css(row.querySelector('input[type="checkbox"]'), 'accentColor', 'width', 'height', 'appearance',
|
||||
'backgroundColor', 'borderTopLeftRadius'),
|
||||
role: row.querySelector('input').getAttribute('role'),
|
||||
labelled: row.querySelector('label').htmlFor === row.querySelector('input').id,
|
||||
};
|
||||
}
|
||||
|
||||
/** D55 and D58: the concept's dialog with the production differences applied. */
|
||||
function mapReviewConcept(reference) {
|
||||
return {
|
||||
...reference,
|
||||
// No sample-plan note (prototype copy); the backup box sits before the actions.
|
||||
// No sample-plan note (prototype copy); the D58 "All files" toggle follows the grid; the backup box sits
|
||||
// before the actions.
|
||||
body: [...reference.body.filter(item => item !== 'p.demo-note').slice(0, -1), 'div.notice', reference.body.at(-1)]
|
||||
.map(item => item === 'div.meta-grid' ? 'dl.meta-grid' : item),
|
||||
// A `dl`, as in the concept's maintenance plan; real sources and the fixed Activity-1 scope.
|
||||
.flatMap(item => item === 'div.meta-grid' ? ['dl.meta-grid', 'div.setting-row'] : [item]),
|
||||
// A `dl`, as in the concept's maintenance plan; real sources. The scope keeps the concept's
|
||||
// "New and unresolved files" until "All files" is checked.
|
||||
grid: { ...reference.grid, tag: 'dl', entries: reference.grid.entries.map(entry => ({ ...entry,
|
||||
dd: { Sources: '2 output folders', Scope: 'All files' }[entry.dt] })) },
|
||||
dd: { Sources: '2 output folders', Scope: 'New and unresolved files' }[entry.dt] })) },
|
||||
actions: { ...reference.actions, buttons: reference.actions.buttons.map(button => button.text === 'Simulate scan'
|
||||
? { ...button, text: 'Start scan' } : button) },
|
||||
};
|
||||
@@ -92,7 +111,8 @@ async function conceptReview(page, theme) {
|
||||
await page.getByRole('combobox', { name: 'Theme', exact: true }).selectOption(theme);
|
||||
await page.locator('#main').getByRole('button', { name: 'Manage sources', exact: true }).click();
|
||||
await expect(page.locator('#main .notice')).toBeVisible();
|
||||
return { review, notice: await page.evaluate(measureNotice, '#main .notice') };
|
||||
return { review, notice: await page.evaluate(measureNotice, '#main .notice'),
|
||||
toggle: await page.evaluate(measureToggle, '#main .setting-row:has(input[type="checkbox"])') };
|
||||
}
|
||||
|
||||
async function productionReview(page, theme) {
|
||||
@@ -103,7 +123,16 @@ async function productionReview(page, theme) {
|
||||
await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Review metadata scan');
|
||||
// The swapped dialog can open under the pointer that pressed Review scan; measure it at rest.
|
||||
await page.mouse.move(0, 0);
|
||||
return { review: await page.evaluate(measureReview), notice: await page.evaluate(measureNotice, 'dialog#modal[open] .notice') };
|
||||
const review = await page.evaluate(measureReview);
|
||||
const notice = await page.evaluate(measureNotice, 'dialog#modal[open] .notice');
|
||||
// The concept's switch is on; compare ours on too, then turn it off again (the default).
|
||||
const allFiles = page.getByRole('switch', { name: 'All files', exact: true });
|
||||
await allFiles.check();
|
||||
await page.mouse.move(0, 0);
|
||||
const toggle = await page.evaluate(measureToggle, 'dialog#modal[open] .setting-row');
|
||||
await allFiles.uncheck();
|
||||
await page.mouse.move(0, 0);
|
||||
return { review, notice, toggle };
|
||||
}
|
||||
|
||||
for (const theme of THEMES) test(`review dialog matches the concept scan-review ${theme}`, async ({ page, request }) => {
|
||||
@@ -114,19 +143,29 @@ for (const theme of THEMES) test(`review dialog matches the concept scan-review
|
||||
const production = await productionReview(page, theme);
|
||||
expect(diffInventory(mapReviewConcept(concept.review), production.review)).toEqual([]);
|
||||
expect(diffInventory(concept.notice, production.notice)).toEqual([]);
|
||||
expect(diffInventory(concept.toggle, production.toggle)).toEqual([]);
|
||||
await expect(page.locator('dialog#modal[open] .setting-row > div > label')).toHaveText('All files');
|
||||
await expect(page.locator('dialog#modal[open] .setting-row input')).not.toBeChecked();
|
||||
await expect(page.getByRole('switch', { name: 'All files', exact: true })).toBeVisible();
|
||||
await expect(page.locator('dialog#modal[open] .notice h3')).toHaveText('Back up the database first (recommended)');
|
||||
await expect(page.locator('dialog#modal[open] .notice .row-actions button')).toHaveText(['Back up now']);
|
||||
|
||||
// Mutation check: one CSS change and one removed element are both noticed, then undone to an exact match again.
|
||||
const undoCss = await mutate(page, { css: '#modal .meta-grid { gap: 3px; } #modal .notice { padding: 2px; }' });
|
||||
const undoCss = await mutate(page, { css: '#modal .meta-grid { gap: 3px; } #modal .notice { padding: 2px; } #modal .setting-row { gap: 3px; }' });
|
||||
const cssMutated = await page.evaluate(measureReview);
|
||||
const noticeMutated = await page.evaluate(measureNotice, 'dialog#modal[open] .notice');
|
||||
await page.getByRole('switch', { name: 'All files', exact: true }).check();
|
||||
await page.mouse.move(0, 0);
|
||||
const toggleMutated = await page.evaluate(measureToggle, 'dialog#modal[open] .setting-row');
|
||||
await page.getByRole('switch', { name: 'All files', exact: true }).uncheck();
|
||||
await page.mouse.move(0, 0);
|
||||
expect(diffInventory(concept.toggle, toggleMutated).map(item => item.path)).toEqual(['frame.columnGap']);
|
||||
expect(diffInventory(mapReviewConcept(concept.review), cssMutated).map(item => item.path)).toEqual(expect.arrayContaining(['grid.columnGap', 'grid.rowGap']));
|
||||
expect(diffInventory(concept.notice, noticeMutated).map(item => item.path)).toEqual(['frame.paddingTop', 'frame.paddingLeft']);
|
||||
await undoCss();
|
||||
const undoDom = await mutate(page, { remove: ['#modal-body > p.small'] });
|
||||
const domMutated = await page.evaluate(() => [...document.querySelectorAll('#modal-body > *')].map(node => node.localName));
|
||||
expect(domMutated).toEqual(['dl', 'div', 'div']);
|
||||
expect(domMutated).toEqual(['dl', 'div', 'div', 'div']);
|
||||
await undoDom();
|
||||
expect(diffInventory(mapReviewConcept(concept.review), await page.evaluate(measureReview))).toEqual([]);
|
||||
});
|
||||
|
||||
@@ -392,16 +392,69 @@ test('start scan creates a job, toasts, and opens its view', async ({ page, requ
|
||||
await expect(scanLog(page).locator('pre')).toContainText('Scan finished');
|
||||
});
|
||||
|
||||
// ── Task 4 (Activity-2): scan scope, new and unresolved files by default ─────────────────────
|
||||
|
||||
test('all files checkbox changes the scope line and the request', async ({ page }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const bodies = [];
|
||||
page.on('request', item => {
|
||||
if (item.method() === 'POST' && new URL(item.url()).pathname === '/prompt_manager/jobs/scan') bodies.push(item.postDataJSON());
|
||||
});
|
||||
const review = page.getByRole('dialog', { name: 'Review metadata scan', exact: true });
|
||||
const scopeValue = review.locator('.meta-grid > div').filter({ has: page.locator('dt', { hasText: 'Scope' }) }).locator('dd');
|
||||
// The concept's switch: a checkbox with role=switch.
|
||||
const allFiles = review.getByRole('switch', { name: 'All files', exact: true });
|
||||
const openReview = async () => {
|
||||
await page.goto(`${workspace}#/images`);
|
||||
await page.getByRole('button', { name: 'Import', exact: true }).click();
|
||||
await page.getByRole('dialog', { name: 'Import & inspect', exact: true }).locator('.operation')
|
||||
.filter({ hasText: 'Scan for metadata' }).getByRole('button', { name: 'Review scan', exact: true }).click();
|
||||
await expect(review).toBeVisible();
|
||||
};
|
||||
|
||||
// The default: the concept's "New and unresolved files", the box unchecked; the request asks for "new".
|
||||
await openReview();
|
||||
await expect(scopeValue).toHaveText('New and unresolved files');
|
||||
await expect(allFiles).not.toBeChecked();
|
||||
await expect(review.locator('.setting-row p')).toHaveText('Also read the files already linked to their prompt again.');
|
||||
await expect(allFiles).toHaveAccessibleDescription('Also read the files already linked to their prompt again.');
|
||||
// Checking and unchecking switches the Scope value both ways.
|
||||
await allFiles.check();
|
||||
await expect(scopeValue).toHaveText('All files');
|
||||
await allFiles.uncheck();
|
||||
await expect(scopeValue).toHaveText('New and unresolved files');
|
||||
await review.getByRole('button', { name: 'Start scan', exact: true }).click();
|
||||
await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/);
|
||||
await expect.poll(() => bodies.length).toBe(1);
|
||||
expect(bodies[0]).toEqual({ scope: 'new' });
|
||||
await expect(heading(page).locator('h1')).toHaveText('Scan for metadata');
|
||||
// Wait for the first scan to end so the second is not refused as busy.
|
||||
await scanLog(page).locator('summary').click();
|
||||
await expect(scanLog(page).locator('pre')).toContainText('Scan finished', { timeout: 15000 });
|
||||
await expect(scanLog(page).locator('pre')).toContainText('new and unresolved files');
|
||||
|
||||
// Checked: "All files", and the request asks for "all".
|
||||
await openReview();
|
||||
await allFiles.check();
|
||||
await expect(scopeValue).toHaveText('All files');
|
||||
await review.getByRole('button', { name: 'Start scan', exact: true }).click();
|
||||
await expect.poll(() => bodies.length).toBe(2);
|
||||
expect(bodies[1]).toEqual({ scope: 'all' });
|
||||
await scanLog(page).locator('summary').click();
|
||||
await expect(scanLog(page).locator('pre')).toContainText('Scan started · ', { timeout: 15000 });
|
||||
await expect(scanLog(page).locator('pre')).toContainText('all files');
|
||||
});
|
||||
|
||||
test('job view counters follow the mapping and update live', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request);
|
||||
await step(request, job.id, { processed: 8240, counters: { added: 1420, known: 6511, review: 37, no_metadata: 254, errors: 18 } });
|
||||
await openJob(page, job.id);
|
||||
// The header: the title, the detail line, then Download report (Task 5) and Cancel scan (danger).
|
||||
// The header: the title, the detail line, then Download report (enabled in Task 5) and Cancel scan (danger).
|
||||
await expect(heading(page).locator('h1')).toHaveText('Scan for metadata');
|
||||
await expect(heading(page).locator('p.muted')).toHaveText(/^Started \d\d:\d\d · 2 output folders · closing this view does not cancel it$/);
|
||||
await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel scan']);
|
||||
await expect(heading(page).getByRole('button', { name: 'Download report', exact: true })).toBeDisabled();
|
||||
await expect(heading(page).getByRole('button', { name: 'Download report', exact: true })).toBeEnabled();
|
||||
await expect(heading(page).getByRole('button', { name: 'Cancel scan', exact: true })).toHaveClass(/danger/);
|
||||
// Progress: the bar, the server's line and the five tiles in order and colour.
|
||||
await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Processing file 8,240 of 19,622');
|
||||
@@ -558,6 +611,19 @@ test('job view renders 200-row cap note', async ({ page, request }) => {
|
||||
await step(request, job.id, { processed: 260, bulk: { group: 'added', count: 250 }, records: [
|
||||
...Array.from({ length: 10 }, (_, index) => ({ group: 'no_metadata', path: `misc/plain_${index}.png`, detail: 'No prompt metadata in the file' })),
|
||||
] });
|
||||
// Every detail read of this job: in flight, and whether one has returned the full listing.
|
||||
const detailUrl = `/prompt_manager/jobs/${job.id}`;
|
||||
const isDetail = item => new URL(item.url()).pathname === detailUrl;
|
||||
const detailReads = { inFlight: 0, full: false };
|
||||
page.on('request', item => { if (isDetail(item)) detailReads.inFlight += 1; });
|
||||
page.on('requestfailed', item => { if (isDetail(item)) detailReads.inFlight -= 1; });
|
||||
page.on('response', async response => {
|
||||
if (!isDetail(response)) return;
|
||||
const body = await response.json().catch(() => null);
|
||||
const outcomes = body?.job?.outcomes ?? {};
|
||||
if (outcomes.added?.length === 200 && outcomes.no_metadata?.length === 10) detailReads.full = true;
|
||||
detailReads.inFlight -= 1;
|
||||
});
|
||||
await openJob(page, job.id);
|
||||
await expect(tiles(page).first().locator('dd')).toHaveText('250');
|
||||
await expect(fileRows(page)).toHaveCount(200);
|
||||
@@ -573,6 +639,9 @@ test('job view renders 200-row cap note', async ({ page, request }) => {
|
||||
|
||||
// Review M12: with the list capped and the log closed, more files only update the counters; the detail is read
|
||||
// again only when the log opens with new lines.
|
||||
// The reads from opening the view have settled: one returned the full listing and none is in flight. The view
|
||||
// re-reads only while what the summary lists differs from the last read, so no further read is due.
|
||||
await expect.poll(() => detailReads.full && detailReads.inFlight === 0).toBe(true);
|
||||
const reads = [];
|
||||
page.on('request', item => { if (new URL(item.url()).pathname === `/prompt_manager/jobs/${job.id}`) reads.push(Date.now()); });
|
||||
await step(request, job.id, { processed: 300, bulk: { group: 'added', count: 40 } });
|
||||
@@ -639,6 +708,8 @@ test('an unknown or restarted job view says so', async ({ page, request }) => {
|
||||
await expect(await request.post('/__test__/jobs/restart')).toBeOK();
|
||||
await expect(heading(page).locator('p.muted')).toHaveText(LOST);
|
||||
await expect(heading(page).getByRole('button', { name: 'Cancel scan', exact: true })).toHaveCount(0);
|
||||
// A lost job has no report on the server (review M9): Download report is disabled rather than failing.
|
||||
await expect(heading(page).getByRole('button', { name: 'Download report', exact: true })).toBeDisabled();
|
||||
// What was read before the restart stays on the page.
|
||||
await expect(fileRows(page)).toHaveCount(1);
|
||||
await expect(pill(page)).toBeHidden();
|
||||
@@ -662,3 +733,69 @@ test('a finished job the server pruned leaves its open view honest', async ({ pa
|
||||
await expect(heading(page).locator('h1')).toBeFocused();
|
||||
await expect(page).toHaveTitle('Activity · Prompt Manager');
|
||||
});
|
||||
|
||||
// Activity-2 Task 5 (D59): Download report and the log's scroll position.
|
||||
test('download report saves the job JSON', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request);
|
||||
await step(request, job.id, { processed: 2, records: [
|
||||
{ group: 'added', path: 'sub/a.png', root: 'output', detail: 'Added' },
|
||||
{ group: 'errors', path: 'broken/b.png', root: 'output', detail: 'Error' },
|
||||
] });
|
||||
await finish(request, job.id);
|
||||
await openJob(page, job.id);
|
||||
const button = heading(page).getByRole('button', { name: 'Download report', exact: true });
|
||||
await expect(button).toBeEnabled();
|
||||
const [download] = await Promise.all([page.waitForEvent('download'), button.click()]);
|
||||
expect(download.suggestedFilename()).toMatch(/^prompt-manager-scan-\d{8}-\d{4}\.json$/);
|
||||
const path = await download.path();
|
||||
const report = JSON.parse(await (await import('node:fs/promises')).readFile(path, 'utf8'));
|
||||
expect(report.job.id).toBe(job.id);
|
||||
expect(report.outcomes.added.rows.map(row => row.path)).toEqual(['sub/a.png']);
|
||||
expect(report.outcomes.errors.truncated).toBe(false);
|
||||
expect(report.log.length).toBeGreaterThan(0);
|
||||
await expect(toast(page)).toHaveText('Job report downloaded.');
|
||||
});
|
||||
|
||||
test('log keeps the reader\'s scroll position', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request);
|
||||
const more = (from, count) => Array.from({ length: count }, (_, index) => ({ group: 'known', path: `log/f_${from + index}.png`,
|
||||
root: 'output', detail: 'Known' }));
|
||||
await step(request, job.id, { processed: 60, records: more(0, 60) });
|
||||
await openJob(page, job.id);
|
||||
await scanLog(page).locator('summary').click();
|
||||
const pre = scanLog(page).locator('pre');
|
||||
await expect(pre).toContainText('log/f_59.png');
|
||||
const metrics = () => pre.evaluate(node => ({ top: node.scrollTop, max: node.scrollHeight - node.clientHeight }));
|
||||
// Opening starts at the newest line.
|
||||
await expect.poll(async () => { const m = await metrics(); return m.max > 0 && m.top >= m.max - 2; }).toBe(true);
|
||||
// Pinned to the bottom: new lines are followed.
|
||||
await step(request, job.id, { processed: 90, records: more(60, 30) });
|
||||
await expect(pre).toContainText('log/f_89.png');
|
||||
await expect.poll(async () => { const m = await metrics(); return m.top >= m.max - 2; }).toBe(true);
|
||||
// Scrolled up: the position stays while more lines arrive.
|
||||
await pre.evaluate(node => { node.scrollTop = 40; });
|
||||
await step(request, job.id, { processed: 120, records: more(90, 30) });
|
||||
await expect(pre).toContainText('log/f_119.png');
|
||||
expect((await metrics()).top).toBe(40);
|
||||
// Closed and opened again (review M6): every open starts at the newest line, also after lines arrived meanwhile.
|
||||
await scanLog(page).locator('summary').click();
|
||||
await expect(scanLog(page)).not.toHaveAttribute('open', '');
|
||||
await step(request, job.id, { processed: 150, records: more(120, 30) });
|
||||
await scanLog(page).locator('summary').click();
|
||||
await expect(pre).toContainText('log/f_149.png');
|
||||
await expect.poll(async () => { const m = await metrics(); return m.max > 0 && m.top >= m.max - 2; }).toBe(true);
|
||||
});
|
||||
|
||||
test('a failed report download says so and leaves the button usable', async ({ page, request }) => {
|
||||
await page.setViewportSize({ width: 1600, height: 1100 });
|
||||
const job = await startJob(request);
|
||||
await finish(request, job.id);
|
||||
await openJob(page, job.id);
|
||||
await page.route(`**/prompt_manager/jobs/${job.id}/report`, route => route.fulfill({ status: 500, body: 'no' }));
|
||||
const button = heading(page).getByRole('button', { name: 'Download report', exact: true });
|
||||
await button.click();
|
||||
await expect(toast(page)).toHaveText('The report could not be downloaded. Try again.');
|
||||
await expect(button).not.toHaveAttribute('aria-disabled', 'true');
|
||||
});
|
||||
|
||||
@@ -2060,7 +2060,7 @@ test('scan for metadata opens the review dialog from Import', async ({ page, req
|
||||
await openScanReview(page);
|
||||
const dialog = scanReview(page);
|
||||
await expect(dialog.locator('dl.meta-grid > div > dt')).toHaveText(['Sources', 'Scope']);
|
||||
await expect(dialog.locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'All files']);
|
||||
await expect(dialog.locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']);
|
||||
await expect(dialog.locator('#modal-body > p.small')).toHaveText('Enrich positive / negative associations, reuse canonical text and report unchanged files. Do not inflate run counts or recreate existing thumbnails.');
|
||||
await expect(dialog.locator('.notice h3')).toHaveText('Back up the database first (recommended)');
|
||||
await expect(dialog.locator('.dialog-actions button')).toHaveText(['Cancel', 'Start scan']);
|
||||
|
||||
@@ -129,7 +129,7 @@ def register_activity_commands(app, routes, api):
|
||||
str(data.get("title", "Scan for metadata")),
|
||||
record_changing=True,
|
||||
scope={
|
||||
"mode": "all",
|
||||
"files": str(data.get("files", "new")),
|
||||
"output_folders": count(data.get("output_folders"), 2),
|
||||
},
|
||||
runner=fake_runner(ending),
|
||||
|
||||
@@ -11,7 +11,7 @@ const ID_C = 'c'.repeat(32);
|
||||
function summary(overrides = {}) {
|
||||
return { id: ID_A, kind: 'scan', title: 'Scan for metadata', state: 'running', started_at: 1000, finished_at: null,
|
||||
progress: 0, status_line: '', processed: 0, total: 0,
|
||||
counters: { added: 0, known: 0, review: 0, no_metadata: 0, errors: 0 }, scope: { mode: 'all', output_folders: 2 },
|
||||
counters: { added: 0, known: 0, review: 0, no_metadata: 0, errors: 0 }, scope: { files: 'new', output_folders: 2 },
|
||||
...overrides };
|
||||
}
|
||||
|
||||
|
||||
@@ -2094,6 +2094,72 @@ class TestActivityJobs(AdminAPITestCase):
|
||||
(legacy["processed"], legacy["found"], legacy["failed"]), (6, 4, 1)
|
||||
)
|
||||
|
||||
async def test_report_is_attachment_without_absolute_paths(self):
|
||||
"""Task 5: the report downloads as JSON, whole and without server paths."""
|
||||
import re
|
||||
|
||||
from py.api.jobs import MAX_OUTCOME_ROWS
|
||||
|
||||
async def runner(job):
|
||||
absolute = f"{self.tmpdir}/output/deep/abs.png"
|
||||
job.record(
|
||||
"added",
|
||||
{"path": absolute, "root": "output", "root_index": 0, "detail": "ok"},
|
||||
f"+ {absolute} · ok",
|
||||
)
|
||||
for index in range(MAX_OUTCOME_ROWS + 5):
|
||||
job.record(
|
||||
"known",
|
||||
{"path": f"k/{index}.png", "root": "output", "detail": "x"},
|
||||
f"= k/{index}.png",
|
||||
)
|
||||
job.update(processed=MAX_OUTCOME_ROWS + 6, total=MAX_OUTCOME_ROWS + 6)
|
||||
|
||||
job = self.api._jobs.start(
|
||||
"scan",
|
||||
"Scan for metadata",
|
||||
record_changing=True,
|
||||
scope={"files": "new", "output_folders": 1},
|
||||
runner=runner,
|
||||
groups=("added", "known", "review", "no_metadata", "errors"),
|
||||
)
|
||||
await asyncio.wait_for(job.task, 10)
|
||||
|
||||
resp = await self.client.request("GET", f"/prompt_manager/jobs/{job.id}/report")
|
||||
text = await resp.text()
|
||||
self.assertEqual(resp.status, 200)
|
||||
self.assertIn("application/json", resp.headers["Content-Type"])
|
||||
disposition = resp.headers["Content-Disposition"]
|
||||
self.assertRegex(
|
||||
disposition,
|
||||
r'^attachment; filename="prompt-manager-scan-\d{8}-\d{4}\.json"$',
|
||||
)
|
||||
self.assertNotIn(self.tmpdir, text)
|
||||
self.assertIsNone(
|
||||
re.search(r"[A-Za-z]:[\\/]|(?<![\w.])/(?:tmp|home|var)/", text)
|
||||
)
|
||||
report = json.loads(text)
|
||||
self.assertEqual(report["job"]["id"], job.id)
|
||||
self.assertEqual(report["job"]["scope"]["files"], "new")
|
||||
self.assertEqual(report["job"]["counters"]["known"], MAX_OUTCOME_ROWS + 5)
|
||||
added = report["outcomes"]["added"]
|
||||
self.assertEqual(
|
||||
(added["count"], len(added["rows"]), added["truncated"]), (1, 1, False)
|
||||
)
|
||||
self.assertEqual(added["rows"][0]["path"], "abs.png")
|
||||
known = report["outcomes"]["known"]
|
||||
self.assertEqual(
|
||||
(known["count"], len(known["rows"]), known["truncated"]),
|
||||
(MAX_OUTCOME_ROWS + 5, MAX_OUTCOME_ROWS, True),
|
||||
)
|
||||
self.assertEqual(report["outcomes"]["errors"]["rows"], [])
|
||||
self.assertFalse(report["outcomes"]["errors"]["truncated"])
|
||||
self.assertEqual(report["log"], job.detail()["log"])
|
||||
self.assertIn("generated_at", report)
|
||||
|
||||
resp = await self.client.request("GET", "/prompt_manager/jobs/nope/report")
|
||||
self.assertEqual(resp.status, 404)
|
||||
|
||||
async def test_cancel_after_first_batch_keeps_partial_counts(self):
|
||||
"""Review M7: the real scan stops at the batch boundary, keeping batch 1."""
|
||||
import threading
|
||||
|
||||
+13
-3
@@ -13,12 +13,14 @@ import unittest
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from py.api.jobs import ( # noqa: E402
|
||||
DEFAULT_SCAN_SCOPE,
|
||||
LOG_CAP_LINE,
|
||||
MAX_FINISHED_JOBS,
|
||||
MAX_LOG_LINES,
|
||||
MAX_OUTCOME_ROWS,
|
||||
SCAN_GROUP_NAMES,
|
||||
SCAN_GROUPS,
|
||||
SCAN_SCOPES,
|
||||
Job,
|
||||
JobBusy,
|
||||
JobRegistry,
|
||||
@@ -155,7 +157,7 @@ class TestJobRegistry(unittest.IsolatedAsyncioTestCase):
|
||||
"scan",
|
||||
"Scan for metadata",
|
||||
record_changing=True,
|
||||
scope={"mode": "all"},
|
||||
scope={"files": "new", "output_folders": 2},
|
||||
runner=_gated_runner(gate),
|
||||
)
|
||||
self.assertEqual(job.state, "running")
|
||||
@@ -181,7 +183,7 @@ class TestJobRegistry(unittest.IsolatedAsyncioTestCase):
|
||||
"log_length",
|
||||
},
|
||||
)
|
||||
self.assertEqual(summary["scope"], {"mode": "all"})
|
||||
self.assertEqual(summary["scope"], {"files": "new", "output_folders": 2})
|
||||
# What the detail would list, so a view reads it only when that changed.
|
||||
self.assertEqual((summary["listed"], summary["log_length"]), ({}, 0))
|
||||
self.assertIsNone(summary["finished_at"])
|
||||
@@ -371,7 +373,12 @@ class TestJobOutcomesAndLog(unittest.TestCase):
|
||||
job.record("bogus", {"path": "a.png", "root": "output"}, "x")
|
||||
|
||||
def test_detail_and_progress(self):
|
||||
job = Job("scan", "Scan", scope={"mode": "all"}, groups=SCAN_GROUP_NAMES)
|
||||
job = Job(
|
||||
"scan",
|
||||
"Scan",
|
||||
scope={"files": "new", "output_folders": 2},
|
||||
groups=SCAN_GROUP_NAMES,
|
||||
)
|
||||
job.update(progress=150, status_line="Processing file 3 of 4", processed=3)
|
||||
job.update(total=4)
|
||||
job.record("known", {"path": "a.png", "root": "output"}, "= a.png")
|
||||
@@ -395,12 +402,15 @@ class TestJobOutcomesAndLog(unittest.TestCase):
|
||||
"linked": "added",
|
||||
"enriched": "added",
|
||||
"unchanged": "known",
|
||||
# Activity-2: a file the "new" scope skipped is Already known.
|
||||
"skipped": "known",
|
||||
"ambiguous": "review",
|
||||
"conflict": "review",
|
||||
"unresolved": "review",
|
||||
"failed": "errors",
|
||||
},
|
||||
)
|
||||
self.assertEqual((SCAN_SCOPES, DEFAULT_SCAN_SCOPE), (("new", "all"), "new"))
|
||||
self.assertEqual(
|
||||
SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors")
|
||||
)
|
||||
|
||||
@@ -0,0 +1,390 @@
|
||||
"""Scan scope (#249 Activity-2 Task 4): new and unresolved files by default.
|
||||
|
||||
A "Scan for metadata" job started without ``scope`` (or with ``"new"``) skips
|
||||
the files the Library already links with a resolved link, before their
|
||||
metadata is read, and counts them as Already known. ``"all"`` reads every file
|
||||
again, as the legacy ``/prompt_manager/scan`` route always does.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
try:
|
||||
from tests.test_api_admin import AdminAPITestCase, _sse_payloads, _write_png
|
||||
except ImportError: # discovered with tests/ as the top-level directory
|
||||
from test_api_admin import AdminAPITestCase, _sse_payloads, _write_png
|
||||
|
||||
from database import operations # noqa: E402
|
||||
from database.pair_ingestion import ingest_resolved_pair # noqa: E402
|
||||
from py.config import GalleryConfig # noqa: E402
|
||||
from utils.prompt_pair_resolver import resolve_prompt_pair # noqa: E402
|
||||
|
||||
SKIPPED_DETAIL = "Skipped · already linked"
|
||||
ON_DISK = object() # _link_resolved: store the file's size on disk
|
||||
|
||||
|
||||
def _graph(text):
|
||||
return {
|
||||
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": text}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
|
||||
"3": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["1", 0], "negative": ["2", 0]},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class ScanScopeTestCase(AdminAPITestCase):
|
||||
"""Helpers: link files the way the scan and the monitor do, spy on reads."""
|
||||
|
||||
def _link_resolved(self, path, text, size=ON_DISK):
|
||||
"""``path`` linked with a resolved pair link (what a resolved scan writes).
|
||||
|
||||
The row stores ``size`` as its file size: by default the file's size on
|
||||
disk (None when the file does not exist).
|
||||
"""
|
||||
if size is ON_DISK:
|
||||
size = os.path.getsize(path) if os.path.exists(path) else None
|
||||
metadata = {"prompt": _graph(text)}
|
||||
if size is not None:
|
||||
metadata["file_info"] = {"size": size}
|
||||
result = ingest_resolved_pair(
|
||||
self.api.db, str(path), metadata, resolve_prompt_pair(metadata)
|
||||
)
|
||||
self.assertEqual(result["status"], "added")
|
||||
|
||||
def _link_without_pair(self, path, text):
|
||||
"""``path`` linked to a prompt with no pair link (fallback / FIFO link)."""
|
||||
prompt_id = self.api.db.save_prompt(text, None, [], None, None)
|
||||
self.assertTrue(self.api.db.link_image_to_prompt(prompt_id, str(path)))
|
||||
return prompt_id
|
||||
|
||||
def _spy_reads(self):
|
||||
reads = []
|
||||
original = self.api._extract_comfyui_metadata
|
||||
|
||||
def spy(path):
|
||||
reads.append(os.path.basename(path))
|
||||
return original(path)
|
||||
|
||||
self.api._extract_comfyui_metadata = spy
|
||||
return reads
|
||||
|
||||
async def _scan(self, body=None):
|
||||
kwargs = {} if body is None else {"json": body}
|
||||
resp = await self.client.request("POST", "/prompt_manager/jobs/scan", **kwargs)
|
||||
data = await resp.json()
|
||||
self.assertEqual(resp.status, 202, data)
|
||||
await asyncio.wait_for(self.api._scan_job.task, 10)
|
||||
detail = await self.client.request(
|
||||
"GET", f"/prompt_manager/jobs/{data['job']['id']}"
|
||||
)
|
||||
return (await detail.json())["job"]
|
||||
|
||||
|
||||
class TestScanScope(ScanScopeTestCase):
|
||||
|
||||
async def test_new_scope_skips_known_resolved_files_without_reading_them(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
known = self.output_dir / "known.png"
|
||||
_write_png(known, prompt_text="a red square")
|
||||
_write_png(self.output_dir / "new.png", prompt_text="a blue circle")
|
||||
self._link_resolved(known, "a red square")
|
||||
reads = self._spy_reads()
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(reads, ["new.png"])
|
||||
self.assertEqual(job["scope"]["files"], "new")
|
||||
self.assertEqual(job["counters"]["known"], 1)
|
||||
self.assertEqual(job["counters"]["added"], 1)
|
||||
self.assertEqual(job["processed"], 2)
|
||||
self.assertEqual(
|
||||
job["outcomes"]["known"],
|
||||
[
|
||||
{
|
||||
"path": "known.png",
|
||||
"root": "output",
|
||||
"root_index": 0,
|
||||
"detail": SKIPPED_DETAIL,
|
||||
}
|
||||
],
|
||||
)
|
||||
self.assertIn(f"= output/known.png · {SKIPPED_DETAIL}", "\n".join(job["log"]))
|
||||
self.assertTrue(
|
||||
job["log"][0].endswith(
|
||||
"Scan started · 1 output folder · new and unresolved files"
|
||||
)
|
||||
)
|
||||
legacy = self.api._scan_job.last_event
|
||||
self.assertEqual((legacy["processed"], legacy["unchanged"]), (2, 1))
|
||||
self.assertTrue(job["status_line"].startswith("Processing file 2 of 2"))
|
||||
|
||||
async def test_whole_known_batch_skips_the_executor_reads(self):
|
||||
from py.api import admin
|
||||
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
for name in ("a.png", "b.png"):
|
||||
_write_png(self.output_dir / name, prompt_text=f"prompt {name}")
|
||||
self._link_resolved(self.output_dir / name, f"prompt {name}")
|
||||
_write_png(self.output_dir / "c.png", prompt_text="prompt c")
|
||||
reads = self._spy_reads()
|
||||
|
||||
with patch.object(admin, "SCAN_BATCH_SIZE", 1):
|
||||
job = await self._scan({"scope": "new"})
|
||||
|
||||
self.assertEqual(reads, ["c.png"])
|
||||
self.assertEqual(job["counters"]["known"], 2)
|
||||
self.assertEqual(job["counters"]["added"], 1)
|
||||
|
||||
async def test_unresolved_files_are_rescanned_in_new_scope(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
names = ("fallback.png", "conflict.png", "unlinked.png")
|
||||
for name in names:
|
||||
_write_png(self.output_dir / name, prompt_text=f"text of {name}")
|
||||
# Linked without a resolved pair link (the monitor's fallback / FIFO link).
|
||||
self._link_without_pair(self.output_dir / "fallback.png", "text of fallback")
|
||||
# Resolved once, then also linked to a second prompt: ambiguous.
|
||||
conflict = self.output_dir / "conflict.png"
|
||||
self._link_resolved(conflict, "text of conflict.png")
|
||||
self._link_without_pair(conflict, "another prompt")
|
||||
reads = self._spy_reads()
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(sorted(reads), sorted(names))
|
||||
self.assertNotIn(SKIPPED_DETAIL, json.dumps(job["outcomes"]))
|
||||
|
||||
async def test_all_scope_reads_everything(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
known = self.output_dir / "known.png"
|
||||
_write_png(known, prompt_text="a red square")
|
||||
self._link_resolved(known, "a red square")
|
||||
reads = self._spy_reads()
|
||||
|
||||
job = await self._scan({"scope": "all"})
|
||||
|
||||
self.assertEqual(reads, ["known.png"])
|
||||
self.assertEqual(job["scope"]["files"], "all")
|
||||
self.assertEqual(job["counters"]["known"], 1)
|
||||
self.assertNotEqual(job["outcomes"]["known"][0]["detail"], SKIPPED_DETAIL)
|
||||
self.assertTrue(
|
||||
job["log"][0].endswith("Scan started · 1 output folder · all files")
|
||||
)
|
||||
|
||||
async def test_legacy_scan_route_reads_everything(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
known = self.output_dir / "known.png"
|
||||
_write_png(known, prompt_text="a red square")
|
||||
self._link_resolved(known, "a red square")
|
||||
reads = self._spy_reads()
|
||||
|
||||
resp = await self.client.request("POST", "/prompt_manager/scan", data="{}")
|
||||
payloads = _sse_payloads(await resp.text())
|
||||
|
||||
self.assertEqual(reads, ["known.png"])
|
||||
self.assertEqual(self.api._scan_job.scope["files"], "all")
|
||||
self.assertEqual(payloads[-1]["type"], "complete")
|
||||
self.assertEqual(payloads[-1]["unchanged"], 1)
|
||||
|
||||
async def test_file_replaced_at_a_known_path_is_reread_and_needs_review(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
path = self.output_dir / "out_00012.png"
|
||||
_write_png(path, prompt_text="the first prompt")
|
||||
self._link_resolved(path, "the first prompt")
|
||||
before = os.path.getsize(path)
|
||||
# ComfyUI reused the name for a new image with another prompt.
|
||||
_write_png(path, prompt_text="a much longer second prompt for the same name")
|
||||
self.assertNotEqual(os.path.getsize(path), before)
|
||||
reads = self._spy_reads()
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(reads, ["out_00012.png"])
|
||||
self.assertEqual(job["counters"]["review"], 1)
|
||||
self.assertEqual(job["counters"]["known"], 0)
|
||||
self.assertIn("different prompt", job["outcomes"]["review"][0]["detail"])
|
||||
|
||||
async def test_scanned_files_store_their_size_so_the_next_new_scan_skips(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
_write_png(self.output_dir / "fresh.png", prompt_text="a fresh prompt")
|
||||
first = await self._scan()
|
||||
self.assertEqual(first["counters"]["added"], 1)
|
||||
reads = self._spy_reads()
|
||||
|
||||
second = await self._scan()
|
||||
|
||||
self.assertEqual(reads, [])
|
||||
self.assertEqual(second["outcomes"]["known"][0]["detail"], SKIPPED_DETAIL)
|
||||
|
||||
async def test_stored_null_size_is_reread(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
path = self.output_dir / "old.png"
|
||||
_write_png(path, prompt_text="an old prompt")
|
||||
self._link_resolved(path, "an old prompt", size=None)
|
||||
reads = self._spy_reads()
|
||||
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(reads, ["old.png"])
|
||||
# The read stores the size it was missing, so the file is known next time.
|
||||
self.assertEqual(job["counters"]["added"], 1)
|
||||
self.assertEqual(
|
||||
self.api.db.get_known_resolved_paths({str(path): os.path.getsize(path)}),
|
||||
{str(path)},
|
||||
)
|
||||
|
||||
async def test_known_set_failure_falls_back_to_reading_every_file(self):
|
||||
GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)]
|
||||
known = self.output_dir / "known.png"
|
||||
_write_png(known, prompt_text="a red square")
|
||||
self._link_resolved(known, "a red square")
|
||||
reads = self._spy_reads()
|
||||
|
||||
def broken(files):
|
||||
raise sqlite3.OperationalError("database is locked")
|
||||
|
||||
self.api.db.get_known_resolved_paths = broken
|
||||
with self.assertLogs("prompt_manager.api", level="WARNING"):
|
||||
job = await self._scan()
|
||||
|
||||
self.assertEqual(job["state"], "finished")
|
||||
self.assertEqual(reads, ["known.png"])
|
||||
self.assertIn(
|
||||
"Known files could not be read · reading every file", "\n".join(job["log"])
|
||||
)
|
||||
|
||||
async def test_empty_read_body_asks_for_the_default_scope(self):
|
||||
"""A chunked POST with no bytes can read a body: still the default."""
|
||||
from py.api.job_routes import _scan_scope
|
||||
|
||||
class Request:
|
||||
can_read_body = True
|
||||
|
||||
def __init__(self, text):
|
||||
self._text = text
|
||||
|
||||
async def text(self):
|
||||
return self._text
|
||||
|
||||
self.assertEqual(await _scan_scope(Request("")), "new")
|
||||
self.assertEqual(await _scan_scope(Request(" \n")), "new")
|
||||
self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all")
|
||||
self.assertIsNone(await _scan_scope(Request("{")))
|
||||
|
||||
async def test_invalid_scope_is_refused(self):
|
||||
for body in ({"scope": "everything"}, {"scope": 1}, ["new"]):
|
||||
with self.subTest(body=body):
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/jobs/scan", json=body
|
||||
)
|
||||
self.assertEqual(resp.status, 400)
|
||||
data = await resp.json()
|
||||
self.assertFalse(data["success"])
|
||||
resp = await self.client.request(
|
||||
"POST", "/prompt_manager/jobs/scan", data="not json"
|
||||
)
|
||||
self.assertEqual(resp.status, 400)
|
||||
self.assertIsNone(self.api._scan_job)
|
||||
|
||||
|
||||
class TestKnownResolvedPaths(ScanScopeTestCase):
|
||||
"""``PromptDatabase.get_known_resolved_paths``: one streamed query."""
|
||||
|
||||
async def test_returns_the_given_paths_that_are_known_and_resolved(self):
|
||||
known, fallback = self.output_dir / "known.png", self.output_dir / "fb.png"
|
||||
self._link_resolved(known, "a red square", size=10)
|
||||
self._link_without_pair(fallback, "a fallback")
|
||||
asked = {str(known): 10, str(fallback): 10, str(self.output_dir / "n.png"): 10}
|
||||
|
||||
self.assertEqual(self.api.db.get_known_resolved_paths(asked), {str(known)})
|
||||
self.assertEqual(self.api.db.get_known_resolved_paths({}), set())
|
||||
|
||||
async def test_size_must_match_and_a_stored_null_size_is_not_known(self):
|
||||
same, other = self.output_dir / "same.png", self.output_dir / "other.png"
|
||||
unknown = self.output_dir / "unknown.png"
|
||||
self._link_resolved(same, "same size", size=10)
|
||||
self._link_resolved(other, "other size", size=10)
|
||||
self._link_resolved(unknown, "no stored size", size=None)
|
||||
asked = {str(same): 10, str(other): 11, str(unknown): 10}
|
||||
|
||||
self.assertEqual(self.api.db.get_known_resolved_paths(asked), {str(same)})
|
||||
# A file the walk could not stat is not known either.
|
||||
self.assertEqual(self.api.db.get_known_resolved_paths({str(same): None}), set())
|
||||
|
||||
async def test_row_without_file_path_matches_through_image_path(self):
|
||||
known = self.output_dir / "legacy.png"
|
||||
self._link_resolved(known, "a legacy row", size=10)
|
||||
with sqlite3.connect(self.api.db.model.db_path) as conn:
|
||||
conn.execute("UPDATE generated_images SET file_path=NULL")
|
||||
conn.close()
|
||||
|
||||
found = self.api.db.get_known_resolved_paths({str(known): 10})
|
||||
self.assertEqual(found, {str(known)})
|
||||
|
||||
async def test_stored_spelling_is_normalized_like_the_scanned_path(self):
|
||||
known = self.output_dir / "sub" / "known.png"
|
||||
self._link_resolved(known, "a red square", size=10)
|
||||
spelled = os.path.join(str(self.output_dir), "sub", "..", "sub", "known.png")
|
||||
with sqlite3.connect(self.api.db.model.db_path) as conn:
|
||||
conn.execute("UPDATE generated_images SET file_path=?", (spelled,))
|
||||
conn.close()
|
||||
|
||||
found = self.api.db.get_known_resolved_paths({str(known): 10})
|
||||
self.assertEqual(found, {str(known)})
|
||||
|
||||
async def test_one_query_streamed_in_small_fetches(self):
|
||||
paths = [self.output_dir / f"{index}.png" for index in range(5)]
|
||||
for path in paths:
|
||||
self._link_resolved(path, f"prompt {path.name}", size=10)
|
||||
original = self.api.db.model.get_connection
|
||||
opened = []
|
||||
|
||||
def counting():
|
||||
opened.append(1)
|
||||
return original()
|
||||
|
||||
self.api.db.model.get_connection = counting
|
||||
with patch.object(operations, "KNOWN_PATHS_FETCH_SIZE", 2):
|
||||
found = self.api.db.get_known_resolved_paths({str(p): 10 for p in paths})
|
||||
self.assertEqual(found, {str(p) for p in paths})
|
||||
self.assertEqual(len(opened), 1)
|
||||
|
||||
async def test_windows_path_case_matches_known_set(self):
|
||||
stored = self.output_dir / "Sub" / "Known.PNG"
|
||||
self._link_resolved(stored, "a red square", size=10)
|
||||
# A row written before paths were case-folded keeps its own spelling.
|
||||
with sqlite3.connect(self.api.db.model.db_path) as conn:
|
||||
conn.execute(
|
||||
"UPDATE generated_images SET file_path=?", (str(stored).upper(),)
|
||||
)
|
||||
conn.close()
|
||||
asked = str(self.output_dir / "sub" / "known.png")
|
||||
|
||||
with patch("os.path.normcase", lambda path: path.lower()):
|
||||
self.assertEqual(self.api.db.get_known_resolved_paths({asked: 10}), {asked})
|
||||
if os.path.normcase("A") == "A": # a case-sensitive platform
|
||||
self.assertEqual(self.api.db.get_known_resolved_paths({asked: 10}), set())
|
||||
|
||||
|
||||
class TestSkippedOutcome(unittest.TestCase):
|
||||
|
||||
def test_skipped_file_is_already_known_with_its_own_detail(self):
|
||||
from py.api.admin import _scan_outcome
|
||||
|
||||
root = os.path.abspath("out")
|
||||
group, row, line = _scan_outcome(
|
||||
os.path.join(root, "a", "b.png"), [(root, "output", 0)], "skipped"
|
||||
)
|
||||
self.assertEqual(group, "known")
|
||||
self.assertEqual(row["detail"], SKIPPED_DETAIL)
|
||||
self.assertEqual(row["path"], "a/b.png")
|
||||
self.assertEqual(line, f"= output/a/b.png · {SKIPPED_DETAIL}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -3,7 +3,7 @@
|
||||
* concept's classes.
|
||||
*
|
||||
* - Header: the page heading names the job; its line is the Activity item's (D54), and its actions are Download report
|
||||
* (disabled until Activity-2) and, while the job runs, "Cancel scan" (danger).
|
||||
* (the job's JSON report, D59) and, while the job runs, "Cancel scan" (danger).
|
||||
* - Progress (`section.surface`): `<progress>`, the server's "Processing file N of M · about X min left", and the
|
||||
* outcome counters as `dl.stat-row` tiles: Added to Library (green), Already known, Needs review (warning), No usable
|
||||
* metadata, Errors (red).
|
||||
@@ -21,6 +21,7 @@ import { showWorkspaceToast } from './toast.mjs';
|
||||
import { noun } from './plural.mjs';
|
||||
import { parseJob, JOBS_URL, LOST_DETAIL } from './activity-store.mjs';
|
||||
import { jobDetail, CANCEL_FAILED_TOAST } from './activity.mjs';
|
||||
import { downloadJobReport } from './activity-scan.mjs';
|
||||
import { createMediaViewer } from './media-viewer.mjs';
|
||||
import { createInspectionController } from './inspection-state.mjs';
|
||||
import { createPromptSender } from './prompt-sender.mjs';
|
||||
@@ -122,7 +123,6 @@ function section(title, id) {
|
||||
export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
const parts = pageHeading(main);
|
||||
const download = createButton('Download report');
|
||||
download.disabled = true;
|
||||
const cancel = createButton('Cancel scan', 'danger');
|
||||
|
||||
const progressPart = section('Progress', 'job-progress-title');
|
||||
@@ -271,6 +271,10 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
document.title = `${job.title} · Prompt Manager`;
|
||||
parts.description.textContent = jobDetail(job);
|
||||
cancel.textContent = scan ? 'Cancel scan' : 'Cancel job';
|
||||
// A job lost with the server has no report there (review M9).
|
||||
const lost = job.state === 'lost';
|
||||
if (lost && document.activeElement === download) parts.h1.focus();
|
||||
download.disabled = lost;
|
||||
if (job.state === 'running') {
|
||||
if (!cancel.isConnected) parts.actions.append(cancel);
|
||||
} else if (cancel.isConnected) {
|
||||
@@ -290,9 +294,11 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
logLabel.textContent = `● ${scan ? 'Scan' : 'Job'} log`;
|
||||
const text = detail?.log.length ? detail.log.join('\n') : 'No log lines yet.';
|
||||
if (pre.textContent !== text) {
|
||||
// Replacing the text clamps the scroll position, so keep it: pinned readers follow the end, others stay.
|
||||
const { scrollTop } = pre;
|
||||
const atEnd = pre.scrollTop + pre.clientHeight >= pre.scrollHeight - 2;
|
||||
pre.textContent = text;
|
||||
if (atEnd) pre.scrollTop = pre.scrollHeight;
|
||||
pre.scrollTop = atEnd ? pre.scrollHeight : scrollTop;
|
||||
}
|
||||
renderRows(job);
|
||||
}
|
||||
@@ -323,7 +329,12 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
reading = false;
|
||||
}
|
||||
render();
|
||||
if (readAgain) { readAgain = false; scheduleRead(); }
|
||||
// Read once more only if the summary moved past what this read returned (review: no redundant re-read).
|
||||
if (readAgain) {
|
||||
readAgain = false;
|
||||
const job = store.find(jobId);
|
||||
if (job && job.state !== 'lost' && detailChanged(job)) scheduleRead();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -347,10 +358,23 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) {
|
||||
}
|
||||
|
||||
scope.listen(log, 'toggle', () => {
|
||||
// Every open starts at the newest line: while closed the log has no layout box (Firefox, WebKit), so a kept
|
||||
// position would be lost anyway (review M6).
|
||||
if (log.open) pre.scrollTop = pre.scrollHeight;
|
||||
const job = store.find(jobId);
|
||||
if (log.open && job && job.state !== 'lost' && detailChanged(job)) scheduleRead();
|
||||
});
|
||||
|
||||
scope.listen(download, 'click', async () => {
|
||||
if (download.disabled || download.getAttribute('aria-disabled') === 'true') return;
|
||||
download.setAttribute('aria-disabled', 'true');
|
||||
try {
|
||||
// A view left while the report downloads shows no toast (review M10).
|
||||
await downloadJobReport(jobId, { signal: scope.signal });
|
||||
} finally {
|
||||
download.removeAttribute('aria-disabled');
|
||||
}
|
||||
});
|
||||
scope.listen(cancel, 'click', async () => {
|
||||
if (cancel.getAttribute('aria-disabled') === 'true') return;
|
||||
cancel.setAttribute('aria-disabled', 'true');
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* Scan for metadata (#249 Activity-1, D55): Images → Import → "Scan for metadata" → Review scan opens the concept's
|
||||
* "Review metadata scan". Its backup box downloads the existing `/prompt_manager/backup` copy and toasts the outcome;
|
||||
* Start scan starts the session job through the Activity store, which toasts "Scan started.", and opens the job's view.
|
||||
* Start scan starts the session job through the Activity store over the chosen scope (new and unresolved files, or
|
||||
* all files with the D58 checkbox), which toasts "Scan started.", and opens the job's view.
|
||||
* A start refused because another record-changing job runs toasts "Another job is running: <title>." with a link to
|
||||
* that job (D57).
|
||||
*/
|
||||
@@ -12,13 +13,29 @@ import { navigateTo } from './state.mjs';
|
||||
export const BACKUP_URL = '/prompt_manager/backup';
|
||||
export const BACKUP_DONE = 'Database backup downloaded.';
|
||||
export const BACKUP_FAILED = 'The database could not be backed up. Try again.';
|
||||
export const REPORT_DONE = 'Job report downloaded.';
|
||||
export const REPORT_FAILED = 'The report could not be downloaded. Try again.';
|
||||
const JOBS_REPORT_BASE = '/prompt_manager/jobs';
|
||||
export const START_FAILED = 'The scan could not start. Try again.';
|
||||
const FALLBACK_NAME = 'prompts_backup.db';
|
||||
const REPORT_FALLBACK_NAME = 'prompt-manager-report.json';
|
||||
|
||||
/** The download name from a Content-Disposition header, or the fallback. */
|
||||
export function backupFilename(disposition) {
|
||||
/** The download name from a Content-Disposition header, or `fallback`. */
|
||||
export function backupFilename(disposition, fallback = FALLBACK_NAME) {
|
||||
const match = /filename="?([^";]+)"?/i.exec(disposition || '');
|
||||
return match ? match[1] : FALLBACK_NAME;
|
||||
return match ? match[1] : fallback;
|
||||
}
|
||||
|
||||
/** Save a fetched response through a temporary link, named by `filename`. */
|
||||
function saveResponse(blob, filename) {
|
||||
const url = URL.createObjectURL(blob);
|
||||
const link = document.createElement('a');
|
||||
link.href = url;
|
||||
link.download = filename;
|
||||
document.body.append(link);
|
||||
link.click();
|
||||
link.remove();
|
||||
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||
}
|
||||
|
||||
/** Download the prompts database copy the legacy admin page also uses; toasts the outcome. */
|
||||
@@ -27,14 +44,7 @@ export async function downloadDatabaseBackup({ fetchUrl = window.ApiClient.fetch
|
||||
const response = await fetchUrl(BACKUP_URL);
|
||||
if (!response.ok) throw new Error(`Backup failed with ${response.status}`);
|
||||
const blob = await response.blob();
|
||||
const url = URL.createObjectURL(blob);
|
||||
const link = document.createElement('a');
|
||||
link.href = url;
|
||||
link.download = backupFilename(response.headers.get('Content-Disposition'));
|
||||
document.body.append(link);
|
||||
link.click();
|
||||
link.remove();
|
||||
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||
saveResponse(blob, backupFilename(response.headers.get('Content-Disposition')));
|
||||
notify(BACKUP_DONE);
|
||||
return true;
|
||||
} catch {
|
||||
@@ -43,16 +53,39 @@ export async function downloadDatabaseBackup({ fetchUrl = window.ApiClient.fetch
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Download a job's JSON report (Activity-2, D59) through the api-client URL, so a proxy mount works; toasts the outcome
|
||||
* unless `signal` (the view's scope) was aborted meanwhile.
|
||||
*/
|
||||
export async function downloadJobReport(jobId, { fetchUrl = window.ApiClient.fetchUrl, notify = showWorkspaceToast,
|
||||
signal = null } = {}) {
|
||||
try {
|
||||
const response = await fetchUrl(`${JOBS_REPORT_BASE}/${encodeURIComponent(jobId)}/report`);
|
||||
if (!response.ok) throw new Error(`Report failed with ${response.status}`);
|
||||
const blob = await response.blob();
|
||||
saveResponse(blob, backupFilename(response.headers.get('Content-Disposition'), REPORT_FALLBACK_NAME));
|
||||
// The file is saved either way; a view left meanwhile (`signal` aborted) shows no toast.
|
||||
if (!signal?.aborted) notify(REPORT_DONE);
|
||||
return true;
|
||||
} catch {
|
||||
if (!signal?.aborted) notify(REPORT_FAILED);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** The toast for a refused start, linking the job that holds the slot. */
|
||||
export function busyToast(running) {
|
||||
if (!running) return 'Another job is running. Try again when it ends.';
|
||||
return ['Another job is running: ', { text: running.title, href: `#/activity/${running.id}` }, '.'];
|
||||
}
|
||||
|
||||
/** Start the scan job and open its view; a refused or failed start says why and stays on the page. */
|
||||
export async function startMetadataScan({ store, scope, notify = showWorkspaceToast, navigate = navigateTo }) {
|
||||
/**
|
||||
* Start the scan job over `files` ('new': new and unresolved files, or 'all') and open its view; a refused or failed
|
||||
* start says why and stays on the page.
|
||||
*/
|
||||
export async function startMetadataScan({ store, scope, files = 'new', notify = showWorkspaceToast, navigate = navigateTo }) {
|
||||
try {
|
||||
const result = await store.start('scan');
|
||||
const result = await store.start('scan', { scope: files });
|
||||
if (scope.signal.aborted) return;
|
||||
if (result.busy !== undefined) {
|
||||
notify(busyToast(result.busy));
|
||||
@@ -68,5 +101,5 @@ export async function startMetadataScan({ store, scope, notify = showWorkspaceTo
|
||||
export function openMetadataScan({ scope, store, sources, trigger = null }) {
|
||||
openScanReviewDialog({ scope, sources, trigger,
|
||||
onBackup: () => downloadDatabaseBackup(),
|
||||
onStart: () => startMetadataScan({ store, scope }) });
|
||||
onStart: ({ scope: files }) => startMetadataScan({ store, scope, files }) });
|
||||
}
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
*
|
||||
* Every opener takes the page `scope`: disposing it closes and removes the shell, so a dialog never outlives its page.
|
||||
*/
|
||||
import { createButton } from '../shared/toolkit/core.js';
|
||||
import { createButton, createSwitch } from '../shared/toolkit/core.js';
|
||||
|
||||
const FIELD_LABELS = Object.freeze({ source: 'Source', folder: 'Folder', media: 'Media', tags: 'Tags' });
|
||||
const FIELD_ORDER = Object.keys(FIELD_LABELS);
|
||||
@@ -342,22 +342,44 @@ export const SCAN_REVIEW_SENTENCE = 'Enrich positive / negative associations, re
|
||||
const BACKUP_LABEL = 'Back up now';
|
||||
const BACKUP_BUSY_LABEL = 'Backing up…';
|
||||
|
||||
export const SCAN_SCOPE_LABELS = { new: 'New and unresolved files', all: 'All files' };
|
||||
export const SCAN_ALL_FILES_HINT = 'Also read the files already linked to their prompt again.';
|
||||
|
||||
/**
|
||||
* The concept's "Review metadata scan" (`scan-review`, D55). `sources` is the number of output folders the scan reads;
|
||||
* the scope is fixed to "All files" in Activity-1. `onBackup` runs the database backup (the button is aria-disabled
|
||||
* and reads "Backing up…" until it settles, and the dialog stays open); Start scan closes the dialog, then `onStart`.
|
||||
* The concept's "Review metadata scan" (`scan-review`, D55). `sources` is the number of output folders the scan reads.
|
||||
* The scope is the concept's "New and unresolved files"; the "All files" checkbox (D58, the concept's `.setting-row`
|
||||
* switch, unchecked) switches the Scope value to "All files". `onBackup` runs the database backup (the button is
|
||||
* aria-disabled and reads "Backing up…" until it settles, and the dialog stays open); Start scan closes the dialog, then
|
||||
* calls `onStart({scope: 'new' | 'all'})`.
|
||||
*/
|
||||
export function openScanReviewDialog({ scope, sources, onBackup, onStart, trigger = null }) {
|
||||
requireScope(scope);
|
||||
requireCallbacks({ onBackup, onStart });
|
||||
if (!Number.isSafeInteger(sources) || sources < 0) throw new TypeError('The scan review needs its source count');
|
||||
const grid = el('dl', undefined, 'meta-grid');
|
||||
const values = {};
|
||||
for (const [term, value] of [['Sources', `${sources.toLocaleString('en-US')} output ${sources === 1 ? 'folder' : 'folders'}`],
|
||||
['Scope', 'All files']]) {
|
||||
['Scope', SCAN_SCOPE_LABELS.new]]) {
|
||||
const entry = el('div');
|
||||
entry.append(el('dt', term), el('dd', value));
|
||||
values[term] = el('dd', value);
|
||||
entry.append(el('dt', term), values[term]);
|
||||
grid.append(entry);
|
||||
}
|
||||
// The concept's System toggle: a `.setting-row` whose checkbox is the Toolkit switch (concept `hydrateSwitches`).
|
||||
const toggle = createSwitch('', '', 'scan-all-files');
|
||||
const allFiles = toggle.querySelector('input');
|
||||
allFiles.checked = false;
|
||||
const option = el('div', undefined, 'setting-row');
|
||||
const optionText = el('div');
|
||||
const optionLabel = el('label', SCAN_SCOPE_LABELS.all);
|
||||
optionLabel.htmlFor = allFiles.id;
|
||||
const hint = el('p', SCAN_ALL_FILES_HINT);
|
||||
hint.id = 'scan-all-files-hint';
|
||||
allFiles.setAttribute('aria-describedby', hint.id);
|
||||
optionText.append(optionLabel, hint);
|
||||
option.append(optionText, toggle);
|
||||
const chosenScope = () => (allFiles.checked ? 'all' : 'new');
|
||||
allFiles.addEventListener('change', () => { values.Scope.textContent = SCAN_SCOPE_LABELS[chosenScope()]; });
|
||||
const notice = el('div', undefined, 'notice');
|
||||
const copy = el('div');
|
||||
copy.append(el('h3', 'Back up the database first (recommended)'),
|
||||
@@ -381,10 +403,11 @@ export function openScanReviewDialog({ scope, sources, onBackup, onStart, trigge
|
||||
const start = createButton('Start scan', 'ca-primary');
|
||||
cancel.addEventListener('click', () => finish());
|
||||
start.addEventListener('click', () => {
|
||||
const files = chosenScope();
|
||||
finish();
|
||||
onStart();
|
||||
onStart({ scope: files });
|
||||
});
|
||||
const actions = el('div', undefined, 'dialog-actions');
|
||||
actions.append(cancel, start);
|
||||
show('Review metadata scan', [grid, el('p', SCAN_REVIEW_SENTENCE, 'small'), notice, actions], { scope, trigger });
|
||||
show('Review metadata scan', [grid, option, el('p', SCAN_REVIEW_SENTENCE, 'small'), notice, actions], { scope, trigger });
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user