diff --git a/TESTING.md b/TESTING.md index 8069def..21aaa78 100644 --- a/TESTING.md +++ b/TESTING.md @@ -550,8 +550,9 @@ Recorded shell deviations: configured sources.", button "Review scan"). It opens the concept's `scan-review`, "Review metadata scan": no sample-plan `.demo-note` (prototype copy); Sources and Scope in a `dl.meta-grid` (the concept's maintenance plan uses `dl`; `scan-review` - uses a `div`) reading "N output folders" (the discovered sources) and "All files" - (fixed in Activity-1; Activity-2 adds the scope option); the concept's sentence; then + uses a `div`) reading "N output folders" (the discovered sources) and the concept's + "New and unresolved files" (the D58 checkbox switches it to "All files"); the + concept's sentence; then a backup `.notice` (the concept's System notice box): `h3` "Back up the database first (recommended)", "Downloads a copy of the prompts database before the scan changes any records." and "Back up now", which downloads the existing `/prompt_manager/backup` @@ -624,6 +625,63 @@ Recorded shell deviations: is running: ." with the title linking that job's view (the shared toast now accepts link parts). A toast with a link stays 20 s instead of 6.5 s, and any toast stays while the pointer or focus is on it. +- **D58** (scan scope, Activity-2 Task 4): the concept's `scan-review` states a fixed + scope, "New and unresolved files". Production keeps that value as the default and adds + the concept's own System toggle after the `dl.meta-grid`: a `.setting-row` with the + label "All files", the hint "Also read the files already linked to their prompt + again." (the switch's `aria-describedby`) and the Toolkit switch (`label.ca-switch`, a + checkbox with `role="switch"`, as the concept's `hydrateSwitches` renders it), + unchecked. Checking it switches the Scope value to "All files"; Start scan sends + `POST /prompt_manager/jobs/scan {scope: "new" | "all"}` (no body, an empty body or no + `scope` is "new"; another value is a 400). + - In "new" the scan skips, before reading any metadata, the files the Library already + links with a resolved link and that are unchanged on disk: exactly one + `generated_images` row for the file, exactly one distinct pair link on it + (`image_prompt_pairs`, written only by resolved ingestion), and a stored + `file_size` equal to the file's size in this walk. Both paths go through + `normalize_image_path` (absolute, normalized, `os.path.normcase`); a row without + `file_path` matches by its `image_path`. + - Read again: files linked without a pair (the monitor's fallback and FIFO links, the + legacy positive-only import), linked to more than one prompt or pair, not linked, a + file whose size changed (one replaced at the same path, e.g. a reused ComfyUI + counter name, is read and its conflict lands under Needs review), and a row with no + stored size (unknown is not known). The scan now stores each file's size when it + ingests it, so a file read once is skipped by the next "new" pass; the first pass + after the upgrade fills the missing sizes, which reports those files as enriched + (Added) instead of unchanged. + - The known set is one streamed query that keeps only the scanned paths. If it cannot + be read, the scan logs "Known files could not be read · reading every file" and + reads every file. + - A skipped file counts as processed and unchanged in the legacy counts (not as + found) and under Already known, with the detail "Skipped · already linked" and the + log line "= <root>/<path> · Skipped · already linked"; the log's first line names + the scope ("Scan started · N output folders · new and unresolved files" or "· all + files"). A legacy-page scan that attaches to a running "new" job therefore shows + the skipped files in its `unchanged` count. + - "All files" reads every file, as the scan did in Activity-1, and the legacy + `/prompt_manager/scan` route always does when it starts the scan (its SSE payload + keys are unchanged). + - `activity-concept.spec.mjs` compares the switch with the concept's System → Sources + toggle ("Watch for new outputs"), without its place-dependent top margin; + `workspace-activity.spec.mjs` covers the Scope value, the description and the + request both ways; `tests/test_scan_scope.py` covers the skip, the size rule, the + replaced file, the re-read of unresolved files, "all", the legacy route, the + fallback, the empty body, the 400 and Windows case folding. +- **D59** (job report and log, Activity-2 Task 5): the job view's "Download report" is + enabled. It fetches `GET /prompt_manager/jobs/<id>/report` through the api-client URL + (so a proxy mount works) and saves a JSON attachment named + `prompt-manager-<kind>-<YYYYMMDD-HHMM>.json` (server local time), then toasts "Job + report downloaded." (or "The report could not be downloaded. Try again."). The report + holds the summary, every counter, each outcome group's exact `count`, its listed + `rows` (at most 200) and a `truncated` flag, the scope and the log; rows and log lines + carry root-relative paths only, never an absolute path (an unknown id is a 404). + A job lost with a restart has no report on the server, so its Download report is + disabled; a view left while the report downloads still saves it but shows no toast. + The scan log opens at its newest line every time it is opened and follows new lines + only while the reader is at the bottom; a reader who scrolled up keeps the position + while it stays open (checked on Chromium, Firefox and WebKit). + `test_report_is_attachment_without_absolute_paths` and the browser tests `download + report saves the job JSON` and `log keeps the reader's scroll position` cover it. - **D29**: no prototype copy. There is no "Interactive concept" badge and no "What changes & why" button. The footer reads "ComfyAssets · Local tools. Your library.", and the title has no " concept" suffix. Save view's dialog action reads diff --git a/database/operations.py b/database/operations.py index 2584498..26b28d1 100644 --- a/database/operations.py +++ b/database/operations.py @@ -181,6 +181,21 @@ AND p.id NOT IN (SELECT prompt_id FROM live_hint_prompts) """ +# The scan's known set (#249 Activity-2): every image row, with its stored file +# size and whether its pair link is resolved: exactly one distinct pair, the rule +# _LINK_SUMMARY_SQL and get_image_prompt_info use. A file is known and resolved +# when it has exactly one image row, that row is resolved, and the row's stored +# size equals the file's size now; several rows (one file linked to more than +# one prompt) are ambiguous, and an unknown (NULL) stored size is not known. The +# rows are streamed in fetches of KNOWN_PATHS_FETCH_SIZE and only the scanned +# paths are kept, so memory follows the scan (at most MAX_SCAN_FILES paths), not +# the table. +_KNOWN_PATHS_SQL = """SELECT gi.file_path, gi.image_path, gi.file_size, + (SELECT COUNT(DISTINCT ip.pair_id) FROM image_prompt_pairs ip + WHERE ip.image_id = gi.id) = 1 +FROM generated_images gi""" +KNOWN_PATHS_FETCH_SIZE = 1000 + # One SELECT reads the link summaries of at most this many paths: three bound # parameters per path stays under SQLite's historical 999-variable limit. LINK_SUMMARY_BATCH = 200 @@ -2280,6 +2295,52 @@ class PromptDatabase( "parameters": row[8] if len(row) > 8 else None, } + def get_known_resolved_paths(self, files: Dict[str, Optional[int]]) -> Set[str]: + """The paths in ``files`` the Library already links with a resolved link. + + ``files`` maps each scanned path to its size on disk (None when it could + not be read). A path qualifies when exactly one ``generated_images`` row + claims the file, that row has exactly one pair link + (``image_prompt_pairs``, written only by resolved ingestion), and its + stored ``file_size`` is not NULL and equals the given size, so a file + replaced at the same path is read again. Files linked without a pair + (the monitor's fallback and FIFO links, the legacy positive-only + import), linked to several prompts or pairs, or not linked at all do + not qualify. Both sides go through ``normalize_image_path`` (absolute, + normalized, ``os.path.normcase``), so a Windows row stored in another + case or spelling still matches. One query, streamed; returns the + matching keys of ``files``. + """ + wanted: Dict[str, List[Tuple[str, Optional[int]]]] = {} + for path, size in files.items(): + key = normalize_image_path(str(path)) + wanted.setdefault(key, []).append((str(path), size)) + if not wanted: + return set() + rows_seen: Dict[str, int] = {} + stored_sizes: Dict[str, int] = {} + with self.model.get_connection() as conn: + cursor = conn.execute(_KNOWN_PATHS_SQL) + while True: + rows = cursor.fetchmany(KNOWN_PATHS_FETCH_SIZE) + if not rows: + break + for file_path, image_path, file_size, is_resolved in rows: + # image_path is NOT NULL; file_path is NULL only on old rows. + key = normalize_image_path(file_path or image_path) + if key not in wanted: + continue + rows_seen[key] = rows_seen.get(key, 0) + 1 + if is_resolved and file_size is not None: + stored_sizes[key] = file_size + return { + path + for key, stored_size in stored_sizes.items() + if rows_seen[key] == 1 + for path, size in wanted[key] + if size is not None and size == stored_size + } + def get_image_link_summaries( self, paths: List[str], allow_filename_fallback: bool = True ) -> Dict[str, Dict[str, Any]]: diff --git a/py/api/admin.py b/py/api/admin.py index 8519cd1..bc900ee 100644 --- a/py/api/admin.py +++ b/py/api/admin.py @@ -17,7 +17,14 @@ from pathlib import Path from aiohttp import web from .prompts import thumbnail_url_for -from .jobs import SCAN_GROUP_NAMES, SCAN_GROUPS, Job, JobBusy +from .jobs import ( + DEFAULT_SCAN_SCOPE, + SCAN_GROUP_NAMES, + SCAN_GROUPS, + SCAN_SCOPES, + Job, + JobBusy, +) try: from ...utils.parallel import map_parallel @@ -150,6 +157,8 @@ SCAN_FINAL_STATES = { "cancelled": "cancelled", "error": "failed", } +# The scope as the scan log's first line names it (the review dialog's words). +SCAN_SCOPE_LOG = {"new": "new and unresolved files", "all": "all files"} MIN_BATCHES_FOR_ESTIMATE = 2 # "about X min left" waits for this many batches # Log line prefix per outcome group (spec: + added, = known, ? review, ! error, # - no usable metadata). @@ -174,6 +183,7 @@ SCAN_DETAILS = { "linked": "Linked to an existing prompt", "enriched": "Metadata added to its prompt", "unchanged": "Already linked to its prompt", + "skipped": "Skipped · already linked", "ambiguous": "Needs review: more than one prompt could match", "unresolved": "Needs review: the prompt could not be traced in the workflow", "conflict": "Needs review: already linked to a different prompt", @@ -197,6 +207,30 @@ def _scan_status_line(done, total, batches, elapsed): return f"{line} · about {minutes} min left" +def _file_size(path): + """``path``'s size on disk, or None when it cannot be read.""" + try: + return os.path.getsize(path) + except OSError: + return None + + +def _with_file_size(canonical, media_file): + """Record the file's size in ``file_info`` unless the metadata carries one. + + Stored as ``generated_images.file_size``; the "new" scan scope skips a known + file only while its size is unchanged. + """ + info = canonical.get("file_info") + info = dict(info) if isinstance(info, dict) else {} + if info.get("size") is None: + size = _file_size(media_file) + if size is not None: + info["size"] = size + if info: + canonical["file_info"] = info + + def _scan_roots(output_dirs): """(absolute root, public label, listing index), the most specific first. @@ -1426,7 +1460,7 @@ class AdminRoutesMixin: job = self._jobs.active(SCAN_JOB_KIND) if job is None: try: - job = self._start_scan_job() + job = self._start_scan_job("all") except JobBusy as busy: return self._long_job_busy_response(busy.running.title) return await self._stream_scan_job(request, job) @@ -1450,16 +1484,19 @@ class AdminRoutesMixin: } ) - def _start_scan_job(self): + def _start_scan_job(self, files=DEFAULT_SCAN_SCOPE): """Register the scan job synchronously; its runner starts on the loop. - Raises JobBusy while another record-changing job runs. + ``files`` is the scope (``SCAN_SCOPES``). Raises JobBusy while another + record-changing job runs. """ + if files not in SCAN_SCOPES: + raise ValueError(f"Unknown scan scope: {files!r}") job = self._jobs.start( SCAN_JOB_KIND, SCAN_JOB_TITLE, record_changing=True, - scope={"mode": "all"}, + scope={"files": files}, runner=self._run_scan_job, groups=SCAN_GROUP_NAMES, ) @@ -1525,7 +1562,8 @@ class AdminRoutesMixin: dir_names = [self._public_path(d) for d in output_dirs] folders = _plural(len(output_dirs), "output folder") job.scope["output_folders"] = len(output_dirs) - job.log_line(f"Scan started · {folders}") + scope = SCAN_SCOPE_LOG[job.scope.get("files", "all")] + job.log_line(f"Scan started · {folders} · {scope}") job.update(status_line=f"Looking for media files in {folders}") yield self._scan_progress( 0, f"Scanning {len(output_dirs)} directory(ies) for media files..." @@ -1544,7 +1582,12 @@ class AdminRoutesMixin: 5, f"Found {len(media_files)} media files to process...", counts ) roots = _scan_roots(output_dirs) - async for event in self._scan_batches(media_files, counts, job, roots): + known = set() + if media_files and job.scope.get("files") != "all": + known = await self._known_scan_files(job, media_files) + async for event in self._scan_batches( + media_files, counts, job, roots, known + ): yield event self.logger.info( @@ -1565,11 +1608,29 @@ class AdminRoutesMixin: "message": "An internal error occurred. Check server logs for details.", } - async def _scan_batches(self, media_files, counts, job, roots=()): + async def _known_scan_files(self, job, media_files): + """The "new" scope's skip set; a failed read falls back to reading all.""" + try: + return await self._run_in_executor(self._known_scan_files_sync, media_files) + except Exception: + self.logger.warning( + "Known files could not be read; the scan reads every file", + exc_info=True, + ) + job.log_line("Known files could not be read · reading every file") + return set() + + def _known_scan_files_sync(self, media_files): + """Stat each scanned file and ask the database which are known (blocking).""" + sizes = {str(media_file): _file_size(media_file) for media_file in media_files} + return self.db.get_known_resolved_paths(sizes) + + async def _scan_batches(self, media_files, counts, job, roots=(), known=()): """Process ``media_files`` in executor batches, updating ``counts`` in place. Records each file's outcome in ``job`` and yields one progress event - per batch. + per batch. Files in ``known`` (path strings) are not read: they count + as processed and unchanged and are recorded as skipped. """ total = len(media_files) started = time.monotonic() @@ -1578,18 +1639,23 @@ class AdminRoutesMixin: if job.cancelled: return batch = media_files[batch_start : batch_start + SCAN_BATCH_SIZE] - batch_results = await self._run_in_executor( - self._extract_batch_metadata_sync, batch - ) - outcomes = [] - batch_counts = await self._run_in_executor( - self._ingest_scan_batch_sync, - batch_results, - roots=roots, - outcomes=outcomes, - ) - for key in counts: - counts[key] += batch_counts[key] + skipped = [f for f in batch if str(f) in known] + to_read = [f for f in batch if str(f) not in known] + outcomes = [_scan_outcome(f, roots, "skipped") for f in skipped] + counts["processed"] += len(skipped) + counts["unchanged"] += len(skipped) + if to_read: + batch_results = await self._run_in_executor( + self._extract_batch_metadata_sync, to_read + ) + batch_counts = await self._run_in_executor( + self._ingest_scan_batch_sync, + batch_results, + roots=roots, + outcomes=outcomes, + ) + for key in counts: + counts[key] += batch_counts[key] for group, row, log_line in outcomes: job.record(group, row, log_line) @@ -1713,6 +1779,7 @@ class AdminRoutesMixin: pass if not canonical.get("prompt") and not canonical.get("parameters"): return None + _with_file_size(canonical, media_file) resolution = resolve_prompt_pair(canonical) if resolution["state"] == "resolved": result = ingest_resolved_pair( diff --git a/py/api/job_routes.py b/py/api/job_routes.py index 1d63eb4..71b35cb 100644 --- a/py/api/job_routes.py +++ b/py/api/job_routes.py @@ -10,13 +10,38 @@ import json from aiohttp import web -from .jobs import JobBusy +from .jobs import DEFAULT_SCAN_SCOPE, SCAN_SCOPES, JobBusy def _not_found(): return web.json_response({"success": False, "error": "Job not found"}, status=404) +def _bad_request(message): + return web.json_response({"success": False, "error": message}, status=400) + + +async def _scan_scope(request): + """The requested scan scope ("new" or "all"), or None when the body is invalid. + + An empty body (also a chunked one with no bytes), or one without + ``scope``, asks for the default. + """ + if not request.can_read_body: + return DEFAULT_SCAN_SCOPE + text = await request.text() + if not text.strip(): # e.g. a chunked POST that sent no bytes + return DEFAULT_SCAN_SCOPE + try: + body = json.loads(text) + except ValueError: + return None + if not isinstance(body, dict): + return None + scope = body.get("scope", DEFAULT_SCAN_SCOPE) + return scope if scope in SCAN_SCOPES else None + + def _summary_event(job): return f"data: {json.dumps(job.summary())}\n\n".encode("utf-8") @@ -37,6 +62,10 @@ class JobRoutesMixin: async def job_detail_route(request): return await self.job_detail(request) + @routes.get("/prompt_manager/jobs/{job_id}/report") + async def job_report_route(request): + return await self.job_report(request) + @routes.get("/prompt_manager/jobs/{job_id}/events") async def job_events_route(request): return await self.job_events(request) @@ -56,6 +85,22 @@ class JobRoutesMixin: return _not_found() return web.json_response({"success": True, "job": job.detail()}) + async def job_report(self, request): + """The job's whole record as a JSON attachment (no absolute paths).""" + job = self._jobs.get(request.match_info["job_id"]) + if job is None: + return _not_found() + body = json.dumps(job.report(), indent=2) + disposition = f'attachment; filename="{job.report_filename()}"' + return web.Response( + text=body, + content_type="application/json", + headers={ + "Content-Disposition": disposition, + "Cache-Control": "no-store", + }, + ) + async def cancel_job(self, request): """Stop a job at its next file boundary; a finished job is a no-op.""" job_id = request.match_info["job_id"] @@ -65,9 +110,16 @@ class JobRoutesMixin: return web.json_response({"success": True, "cancellation_requested": requested}) async def start_scan_job(self, request): - """Start the metadata scan as a job (202), or 409 naming the busy job.""" + """Start the metadata scan as a job (202), or 409 naming the busy job. + + The body's ``scope`` is "new" (the default: new and unresolved files) + or "all"; anything else is a 400. + """ + files = await _scan_scope(request) + if files is None: + return _bad_request('The scan scope must be "new" or "all"') try: - job = self._start_scan_job() + job = self._start_scan_job(files) except JobBusy as busy: running = busy.running return web.json_response( diff --git a/py/api/jobs.py b/py/api/jobs.py index 4918e4c..a5a4b0e 100644 --- a/py/api/jobs.py +++ b/py/api/jobs.py @@ -35,12 +35,19 @@ MAX_FINISHED_JOBS = 20 # finished jobs this session keeps # The metadata scan's outcome groups, in counter-tile order of the spec's # mapping, and the per-file scan status each group collects. Files without # usable metadata have no status and go to "no_metadata". +# Which files a scan reads (#249 Activity-2): "new" skips the files the Library +# already links with a resolved link (get_known_resolved_paths) before reading +# them, so new, unlinked, unresolved and ambiguous files are read; "all" reads +# every file again. The workspace job defaults to "new"; legacy /scan is "all". +SCAN_SCOPES = ("new", "all") +DEFAULT_SCAN_SCOPE = "new" SCAN_GROUP_NAMES = ("added", "known", "review", "no_metadata", "errors") SCAN_GROUPS = { "added": "added", "linked": "added", "enriched": "added", "unchanged": "known", + "skipped": "known", # "new" scope: known and resolved, not read again "ambiguous": "review", "conflict": "review", "unresolved": "review", @@ -240,6 +247,36 @@ class Job: "log": list(self.log), } + def report(self): + """The downloadable report: the summary, every outcome group and the log. + + Each group carries its exact count, its listed rows (at most + MAX_OUTCOME_ROWS) and whether the list is cut short. Rows and log lines + already hold root-relative paths only. + """ + summary = self.summary() + summary.pop("listed", None) + summary.pop("log_length", None) + return { + "report_version": 1, + "generated_at": self._clock(), + "job": summary, + "outcomes": { + group: { + "count": self.counters.get(group, len(rows)), + "rows": [dict(row) for row in rows], + "truncated": self.counters.get(group, len(rows)) > len(rows), + } + for group, rows in self.outcomes.items() + }, + "log": list(self.log), + } + + def report_filename(self): + """The download name: prompt-manager-KIND-YYYYMMDD-HHMM.json (local time).""" + stamp = time.strftime("%Y%m%d-%H%M", time.localtime(self._clock())) + return f"prompt-manager-{self.kind}-{stamp}.json" + class JobRegistry: """Every job of this server session, running and recently finished.""" diff --git a/tests/browser/activity-concept.spec.mjs b/tests/browser/activity-concept.spec.mjs index 4d6d37b..e3c8f9f 100644 --- a/tests/browser/activity-concept.spec.mjs +++ b/tests/browser/activity-concept.spec.mjs @@ -63,16 +63,35 @@ function measureNotice(selector) { }; } -/** D55: the concept's dialog with the production differences applied. */ +/** A concept `.setting-row` switch: its frame, its label and hint, and its checkbox (not its place-dependent margin). */ +function measureToggle(selector) { + const row = document.querySelector(selector); + const css = (node, ...keys) => node ? Object.fromEntries(keys.map(key => [key, getComputedStyle(node)[key]])) : null; + return { + children: [...row.children].map(node => [node.localName, ...node.classList].join('.')), + frame: css(row, 'display', 'alignItems', 'justifyContent', 'columnGap', 'paddingTop', 'paddingBottom', + 'borderTopWidth', 'borderTopStyle', 'borderTopColor'), + label: css(row.querySelector('label'), 'fontSize', 'fontWeight', 'color'), + hint: css(row.querySelector('p'), 'fontSize', 'color', 'marginTop'), + box: css(row.querySelector('input[type="checkbox"]'), 'accentColor', 'width', 'height', 'appearance', + 'backgroundColor', 'borderTopLeftRadius'), + role: row.querySelector('input').getAttribute('role'), + labelled: row.querySelector('label').htmlFor === row.querySelector('input').id, + }; +} + +/** D55 and D58: the concept's dialog with the production differences applied. */ function mapReviewConcept(reference) { return { ...reference, - // No sample-plan note (prototype copy); the backup box sits before the actions. + // No sample-plan note (prototype copy); the D58 "All files" toggle follows the grid; the backup box sits + // before the actions. body: [...reference.body.filter(item => item !== 'p.demo-note').slice(0, -1), 'div.notice', reference.body.at(-1)] - .map(item => item === 'div.meta-grid' ? 'dl.meta-grid' : item), - // A `dl`, as in the concept's maintenance plan; real sources and the fixed Activity-1 scope. + .flatMap(item => item === 'div.meta-grid' ? ['dl.meta-grid', 'div.setting-row'] : [item]), + // A `dl`, as in the concept's maintenance plan; real sources. The scope keeps the concept's + // "New and unresolved files" until "All files" is checked. grid: { ...reference.grid, tag: 'dl', entries: reference.grid.entries.map(entry => ({ ...entry, - dd: { Sources: '2 output folders', Scope: 'All files' }[entry.dt] })) }, + dd: { Sources: '2 output folders', Scope: 'New and unresolved files' }[entry.dt] })) }, actions: { ...reference.actions, buttons: reference.actions.buttons.map(button => button.text === 'Simulate scan' ? { ...button, text: 'Start scan' } : button) }, }; @@ -92,7 +111,8 @@ async function conceptReview(page, theme) { await page.getByRole('combobox', { name: 'Theme', exact: true }).selectOption(theme); await page.locator('#main').getByRole('button', { name: 'Manage sources', exact: true }).click(); await expect(page.locator('#main .notice')).toBeVisible(); - return { review, notice: await page.evaluate(measureNotice, '#main .notice') }; + return { review, notice: await page.evaluate(measureNotice, '#main .notice'), + toggle: await page.evaluate(measureToggle, '#main .setting-row:has(input[type="checkbox"])') }; } async function productionReview(page, theme) { @@ -103,7 +123,16 @@ async function productionReview(page, theme) { await expect(page.locator('dialog#modal[open] #modal-title')).toHaveText('Review metadata scan'); // The swapped dialog can open under the pointer that pressed Review scan; measure it at rest. await page.mouse.move(0, 0); - return { review: await page.evaluate(measureReview), notice: await page.evaluate(measureNotice, 'dialog#modal[open] .notice') }; + const review = await page.evaluate(measureReview); + const notice = await page.evaluate(measureNotice, 'dialog#modal[open] .notice'); + // The concept's switch is on; compare ours on too, then turn it off again (the default). + const allFiles = page.getByRole('switch', { name: 'All files', exact: true }); + await allFiles.check(); + await page.mouse.move(0, 0); + const toggle = await page.evaluate(measureToggle, 'dialog#modal[open] .setting-row'); + await allFiles.uncheck(); + await page.mouse.move(0, 0); + return { review, notice, toggle }; } for (const theme of THEMES) test(`review dialog matches the concept scan-review ${theme}`, async ({ page, request }) => { @@ -114,19 +143,29 @@ for (const theme of THEMES) test(`review dialog matches the concept scan-review const production = await productionReview(page, theme); expect(diffInventory(mapReviewConcept(concept.review), production.review)).toEqual([]); expect(diffInventory(concept.notice, production.notice)).toEqual([]); + expect(diffInventory(concept.toggle, production.toggle)).toEqual([]); + await expect(page.locator('dialog#modal[open] .setting-row > div > label')).toHaveText('All files'); + await expect(page.locator('dialog#modal[open] .setting-row input')).not.toBeChecked(); + await expect(page.getByRole('switch', { name: 'All files', exact: true })).toBeVisible(); await expect(page.locator('dialog#modal[open] .notice h3')).toHaveText('Back up the database first (recommended)'); await expect(page.locator('dialog#modal[open] .notice .row-actions button')).toHaveText(['Back up now']); // Mutation check: one CSS change and one removed element are both noticed, then undone to an exact match again. - const undoCss = await mutate(page, { css: '#modal .meta-grid { gap: 3px; } #modal .notice { padding: 2px; }' }); + const undoCss = await mutate(page, { css: '#modal .meta-grid { gap: 3px; } #modal .notice { padding: 2px; } #modal .setting-row { gap: 3px; }' }); const cssMutated = await page.evaluate(measureReview); const noticeMutated = await page.evaluate(measureNotice, 'dialog#modal[open] .notice'); + await page.getByRole('switch', { name: 'All files', exact: true }).check(); + await page.mouse.move(0, 0); + const toggleMutated = await page.evaluate(measureToggle, 'dialog#modal[open] .setting-row'); + await page.getByRole('switch', { name: 'All files', exact: true }).uncheck(); + await page.mouse.move(0, 0); + expect(diffInventory(concept.toggle, toggleMutated).map(item => item.path)).toEqual(['frame.columnGap']); expect(diffInventory(mapReviewConcept(concept.review), cssMutated).map(item => item.path)).toEqual(expect.arrayContaining(['grid.columnGap', 'grid.rowGap'])); expect(diffInventory(concept.notice, noticeMutated).map(item => item.path)).toEqual(['frame.paddingTop', 'frame.paddingLeft']); await undoCss(); const undoDom = await mutate(page, { remove: ['#modal-body > p.small'] }); const domMutated = await page.evaluate(() => [...document.querySelectorAll('#modal-body > *')].map(node => node.localName)); - expect(domMutated).toEqual(['dl', 'div', 'div']); + expect(domMutated).toEqual(['dl', 'div', 'div', 'div']); await undoDom(); expect(diffInventory(mapReviewConcept(concept.review), await page.evaluate(measureReview))).toEqual([]); }); diff --git a/tests/browser/workspace-activity.spec.mjs b/tests/browser/workspace-activity.spec.mjs index f61eac5..3970de0 100644 --- a/tests/browser/workspace-activity.spec.mjs +++ b/tests/browser/workspace-activity.spec.mjs @@ -392,16 +392,69 @@ test('start scan creates a job, toasts, and opens its view', async ({ page, requ await expect(scanLog(page).locator('pre')).toContainText('Scan finished'); }); +// ── Task 4 (Activity-2): scan scope, new and unresolved files by default ───────────────────── + +test('all files checkbox changes the scope line and the request', async ({ page }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const bodies = []; + page.on('request', item => { + if (item.method() === 'POST' && new URL(item.url()).pathname === '/prompt_manager/jobs/scan') bodies.push(item.postDataJSON()); + }); + const review = page.getByRole('dialog', { name: 'Review metadata scan', exact: true }); + const scopeValue = review.locator('.meta-grid > div').filter({ has: page.locator('dt', { hasText: 'Scope' }) }).locator('dd'); + // The concept's switch: a checkbox with role=switch. + const allFiles = review.getByRole('switch', { name: 'All files', exact: true }); + const openReview = async () => { + await page.goto(`${workspace}#/images`); + await page.getByRole('button', { name: 'Import', exact: true }).click(); + await page.getByRole('dialog', { name: 'Import & inspect', exact: true }).locator('.operation') + .filter({ hasText: 'Scan for metadata' }).getByRole('button', { name: 'Review scan', exact: true }).click(); + await expect(review).toBeVisible(); + }; + + // The default: the concept's "New and unresolved files", the box unchecked; the request asks for "new". + await openReview(); + await expect(scopeValue).toHaveText('New and unresolved files'); + await expect(allFiles).not.toBeChecked(); + await expect(review.locator('.setting-row p')).toHaveText('Also read the files already linked to their prompt again.'); + await expect(allFiles).toHaveAccessibleDescription('Also read the files already linked to their prompt again.'); + // Checking and unchecking switches the Scope value both ways. + await allFiles.check(); + await expect(scopeValue).toHaveText('All files'); + await allFiles.uncheck(); + await expect(scopeValue).toHaveText('New and unresolved files'); + await review.getByRole('button', { name: 'Start scan', exact: true }).click(); + await expect(page).toHaveURL(/#\/activity\/[0-9a-f]{32}$/); + await expect.poll(() => bodies.length).toBe(1); + expect(bodies[0]).toEqual({ scope: 'new' }); + await expect(heading(page).locator('h1')).toHaveText('Scan for metadata'); + // Wait for the first scan to end so the second is not refused as busy. + await scanLog(page).locator('summary').click(); + await expect(scanLog(page).locator('pre')).toContainText('Scan finished', { timeout: 15000 }); + await expect(scanLog(page).locator('pre')).toContainText('new and unresolved files'); + + // Checked: "All files", and the request asks for "all". + await openReview(); + await allFiles.check(); + await expect(scopeValue).toHaveText('All files'); + await review.getByRole('button', { name: 'Start scan', exact: true }).click(); + await expect.poll(() => bodies.length).toBe(2); + expect(bodies[1]).toEqual({ scope: 'all' }); + await scanLog(page).locator('summary').click(); + await expect(scanLog(page).locator('pre')).toContainText('Scan started · ', { timeout: 15000 }); + await expect(scanLog(page).locator('pre')).toContainText('all files'); +}); + test('job view counters follow the mapping and update live', async ({ page, request }) => { await page.setViewportSize({ width: 1600, height: 1100 }); const job = await startJob(request); await step(request, job.id, { processed: 8240, counters: { added: 1420, known: 6511, review: 37, no_metadata: 254, errors: 18 } }); await openJob(page, job.id); - // The header: the title, the detail line, then Download report (Task 5) and Cancel scan (danger). + // The header: the title, the detail line, then Download report (enabled in Task 5) and Cancel scan (danger). await expect(heading(page).locator('h1')).toHaveText('Scan for metadata'); await expect(heading(page).locator('p.muted')).toHaveText(/^Started \d\d:\d\d · 2 output folders · closing this view does not cancel it$/); await expect(heading(page).locator('.heading-actions > button')).toHaveText(['Download report', 'Cancel scan']); - await expect(heading(page).getByRole('button', { name: 'Download report', exact: true })).toBeDisabled(); + await expect(heading(page).getByRole('button', { name: 'Download report', exact: true })).toBeEnabled(); await expect(heading(page).getByRole('button', { name: 'Cancel scan', exact: true })).toHaveClass(/danger/); // Progress: the bar, the server's line and the five tiles in order and colour. await expect(progressSection(page).locator('.panel-heading > span.small')).toHaveText('Processing file 8,240 of 19,622'); @@ -558,6 +611,19 @@ test('job view renders 200-row cap note', async ({ page, request }) => { await step(request, job.id, { processed: 260, bulk: { group: 'added', count: 250 }, records: [ ...Array.from({ length: 10 }, (_, index) => ({ group: 'no_metadata', path: `misc/plain_${index}.png`, detail: 'No prompt metadata in the file' })), ] }); + // Every detail read of this job: in flight, and whether one has returned the full listing. + const detailUrl = `/prompt_manager/jobs/${job.id}`; + const isDetail = item => new URL(item.url()).pathname === detailUrl; + const detailReads = { inFlight: 0, full: false }; + page.on('request', item => { if (isDetail(item)) detailReads.inFlight += 1; }); + page.on('requestfailed', item => { if (isDetail(item)) detailReads.inFlight -= 1; }); + page.on('response', async response => { + if (!isDetail(response)) return; + const body = await response.json().catch(() => null); + const outcomes = body?.job?.outcomes ?? {}; + if (outcomes.added?.length === 200 && outcomes.no_metadata?.length === 10) detailReads.full = true; + detailReads.inFlight -= 1; + }); await openJob(page, job.id); await expect(tiles(page).first().locator('dd')).toHaveText('250'); await expect(fileRows(page)).toHaveCount(200); @@ -573,6 +639,9 @@ test('job view renders 200-row cap note', async ({ page, request }) => { // Review M12: with the list capped and the log closed, more files only update the counters; the detail is read // again only when the log opens with new lines. + // The reads from opening the view have settled: one returned the full listing and none is in flight. The view + // re-reads only while what the summary lists differs from the last read, so no further read is due. + await expect.poll(() => detailReads.full && detailReads.inFlight === 0).toBe(true); const reads = []; page.on('request', item => { if (new URL(item.url()).pathname === `/prompt_manager/jobs/${job.id}`) reads.push(Date.now()); }); await step(request, job.id, { processed: 300, bulk: { group: 'added', count: 40 } }); @@ -639,6 +708,8 @@ test('an unknown or restarted job view says so', async ({ page, request }) => { await expect(await request.post('/__test__/jobs/restart')).toBeOK(); await expect(heading(page).locator('p.muted')).toHaveText(LOST); await expect(heading(page).getByRole('button', { name: 'Cancel scan', exact: true })).toHaveCount(0); + // A lost job has no report on the server (review M9): Download report is disabled rather than failing. + await expect(heading(page).getByRole('button', { name: 'Download report', exact: true })).toBeDisabled(); // What was read before the restart stays on the page. await expect(fileRows(page)).toHaveCount(1); await expect(pill(page)).toBeHidden(); @@ -662,3 +733,69 @@ test('a finished job the server pruned leaves its open view honest', async ({ pa await expect(heading(page).locator('h1')).toBeFocused(); await expect(page).toHaveTitle('Activity · Prompt Manager'); }); + +// Activity-2 Task 5 (D59): Download report and the log's scroll position. +test('download report saves the job JSON', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const job = await startJob(request); + await step(request, job.id, { processed: 2, records: [ + { group: 'added', path: 'sub/a.png', root: 'output', detail: 'Added' }, + { group: 'errors', path: 'broken/b.png', root: 'output', detail: 'Error' }, + ] }); + await finish(request, job.id); + await openJob(page, job.id); + const button = heading(page).getByRole('button', { name: 'Download report', exact: true }); + await expect(button).toBeEnabled(); + const [download] = await Promise.all([page.waitForEvent('download'), button.click()]); + expect(download.suggestedFilename()).toMatch(/^prompt-manager-scan-\d{8}-\d{4}\.json$/); + const path = await download.path(); + const report = JSON.parse(await (await import('node:fs/promises')).readFile(path, 'utf8')); + expect(report.job.id).toBe(job.id); + expect(report.outcomes.added.rows.map(row => row.path)).toEqual(['sub/a.png']); + expect(report.outcomes.errors.truncated).toBe(false); + expect(report.log.length).toBeGreaterThan(0); + await expect(toast(page)).toHaveText('Job report downloaded.'); +}); + +test('log keeps the reader\'s scroll position', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const job = await startJob(request); + const more = (from, count) => Array.from({ length: count }, (_, index) => ({ group: 'known', path: `log/f_${from + index}.png`, + root: 'output', detail: 'Known' })); + await step(request, job.id, { processed: 60, records: more(0, 60) }); + await openJob(page, job.id); + await scanLog(page).locator('summary').click(); + const pre = scanLog(page).locator('pre'); + await expect(pre).toContainText('log/f_59.png'); + const metrics = () => pre.evaluate(node => ({ top: node.scrollTop, max: node.scrollHeight - node.clientHeight })); + // Opening starts at the newest line. + await expect.poll(async () => { const m = await metrics(); return m.max > 0 && m.top >= m.max - 2; }).toBe(true); + // Pinned to the bottom: new lines are followed. + await step(request, job.id, { processed: 90, records: more(60, 30) }); + await expect(pre).toContainText('log/f_89.png'); + await expect.poll(async () => { const m = await metrics(); return m.top >= m.max - 2; }).toBe(true); + // Scrolled up: the position stays while more lines arrive. + await pre.evaluate(node => { node.scrollTop = 40; }); + await step(request, job.id, { processed: 120, records: more(90, 30) }); + await expect(pre).toContainText('log/f_119.png'); + expect((await metrics()).top).toBe(40); + // Closed and opened again (review M6): every open starts at the newest line, also after lines arrived meanwhile. + await scanLog(page).locator('summary').click(); + await expect(scanLog(page)).not.toHaveAttribute('open', ''); + await step(request, job.id, { processed: 150, records: more(120, 30) }); + await scanLog(page).locator('summary').click(); + await expect(pre).toContainText('log/f_149.png'); + await expect.poll(async () => { const m = await metrics(); return m.max > 0 && m.top >= m.max - 2; }).toBe(true); +}); + +test('a failed report download says so and leaves the button usable', async ({ page, request }) => { + await page.setViewportSize({ width: 1600, height: 1100 }); + const job = await startJob(request); + await finish(request, job.id); + await openJob(page, job.id); + await page.route(`**/prompt_manager/jobs/${job.id}/report`, route => route.fulfill({ status: 500, body: 'no' })); + const button = heading(page).getByRole('button', { name: 'Download report', exact: true }); + await button.click(); + await expect(toast(page)).toHaveText('The report could not be downloaded. Try again.'); + await expect(button).not.toHaveAttribute('aria-disabled', 'true'); +}); diff --git a/tests/browser/workspace-images.spec.mjs b/tests/browser/workspace-images.spec.mjs index 0e799ed..99dbbba 100644 --- a/tests/browser/workspace-images.spec.mjs +++ b/tests/browser/workspace-images.spec.mjs @@ -2060,7 +2060,7 @@ test('scan for metadata opens the review dialog from Import', async ({ page, req await openScanReview(page); const dialog = scanReview(page); await expect(dialog.locator('dl.meta-grid > div > dt')).toHaveText(['Sources', 'Scope']); - await expect(dialog.locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'All files']); + await expect(dialog.locator('dl.meta-grid > div > dd')).toHaveText(['2 output folders', 'New and unresolved files']); await expect(dialog.locator('#modal-body > p.small')).toHaveText('Enrich positive / negative associations, reuse canonical text and report unchanged files. Do not inflate run counts or recreate existing thumbnails.'); await expect(dialog.locator('.notice h3')).toHaveText('Back up the database first (recommended)'); await expect(dialog.locator('.dialog-actions button')).toHaveText(['Cancel', 'Start scan']); diff --git a/tests/browser/workspace_activity_fixture.py b/tests/browser/workspace_activity_fixture.py index 40a9054..77f102a 100644 --- a/tests/browser/workspace_activity_fixture.py +++ b/tests/browser/workspace_activity_fixture.py @@ -129,7 +129,7 @@ def register_activity_commands(app, routes, api): str(data.get("title", "Scan for metadata")), record_changing=True, scope={ - "mode": "all", + "files": str(data.get("files", "new")), "output_folders": count(data.get("output_folders"), 2), }, runner=fake_runner(ending), diff --git a/tests/js/workspace-activity-store.test.js b/tests/js/workspace-activity-store.test.js index 211825c..89eb72d 100644 --- a/tests/js/workspace-activity-store.test.js +++ b/tests/js/workspace-activity-store.test.js @@ -11,7 +11,7 @@ const ID_C = 'c'.repeat(32); function summary(overrides = {}) { return { id: ID_A, kind: 'scan', title: 'Scan for metadata', state: 'running', started_at: 1000, finished_at: null, progress: 0, status_line: '', processed: 0, total: 0, - counters: { added: 0, known: 0, review: 0, no_metadata: 0, errors: 0 }, scope: { mode: 'all', output_folders: 2 }, + counters: { added: 0, known: 0, review: 0, no_metadata: 0, errors: 0 }, scope: { files: 'new', output_folders: 2 }, ...overrides }; } diff --git a/tests/test_api_admin.py b/tests/test_api_admin.py index e7cc02b..363a9b1 100644 --- a/tests/test_api_admin.py +++ b/tests/test_api_admin.py @@ -2094,6 +2094,72 @@ class TestActivityJobs(AdminAPITestCase): (legacy["processed"], legacy["found"], legacy["failed"]), (6, 4, 1) ) + async def test_report_is_attachment_without_absolute_paths(self): + """Task 5: the report downloads as JSON, whole and without server paths.""" + import re + + from py.api.jobs import MAX_OUTCOME_ROWS + + async def runner(job): + absolute = f"{self.tmpdir}/output/deep/abs.png" + job.record( + "added", + {"path": absolute, "root": "output", "root_index": 0, "detail": "ok"}, + f"+ {absolute} · ok", + ) + for index in range(MAX_OUTCOME_ROWS + 5): + job.record( + "known", + {"path": f"k/{index}.png", "root": "output", "detail": "x"}, + f"= k/{index}.png", + ) + job.update(processed=MAX_OUTCOME_ROWS + 6, total=MAX_OUTCOME_ROWS + 6) + + job = self.api._jobs.start( + "scan", + "Scan for metadata", + record_changing=True, + scope={"files": "new", "output_folders": 1}, + runner=runner, + groups=("added", "known", "review", "no_metadata", "errors"), + ) + await asyncio.wait_for(job.task, 10) + + resp = await self.client.request("GET", f"/prompt_manager/jobs/{job.id}/report") + text = await resp.text() + self.assertEqual(resp.status, 200) + self.assertIn("application/json", resp.headers["Content-Type"]) + disposition = resp.headers["Content-Disposition"] + self.assertRegex( + disposition, + r'^attachment; filename="prompt-manager-scan-\d{8}-\d{4}\.json"$', + ) + self.assertNotIn(self.tmpdir, text) + self.assertIsNone( + re.search(r"[A-Za-z]:[\\/]|(?<![\w.])/(?:tmp|home|var)/", text) + ) + report = json.loads(text) + self.assertEqual(report["job"]["id"], job.id) + self.assertEqual(report["job"]["scope"]["files"], "new") + self.assertEqual(report["job"]["counters"]["known"], MAX_OUTCOME_ROWS + 5) + added = report["outcomes"]["added"] + self.assertEqual( + (added["count"], len(added["rows"]), added["truncated"]), (1, 1, False) + ) + self.assertEqual(added["rows"][0]["path"], "abs.png") + known = report["outcomes"]["known"] + self.assertEqual( + (known["count"], len(known["rows"]), known["truncated"]), + (MAX_OUTCOME_ROWS + 5, MAX_OUTCOME_ROWS, True), + ) + self.assertEqual(report["outcomes"]["errors"]["rows"], []) + self.assertFalse(report["outcomes"]["errors"]["truncated"]) + self.assertEqual(report["log"], job.detail()["log"]) + self.assertIn("generated_at", report) + + resp = await self.client.request("GET", "/prompt_manager/jobs/nope/report") + self.assertEqual(resp.status, 404) + async def test_cancel_after_first_batch_keeps_partial_counts(self): """Review M7: the real scan stops at the batch boundary, keeping batch 1.""" import threading diff --git a/tests/test_jobs.py b/tests/test_jobs.py index 9dffb42..7f87f85 100644 --- a/tests/test_jobs.py +++ b/tests/test_jobs.py @@ -13,12 +13,14 @@ import unittest sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) from py.api.jobs import ( # noqa: E402 + DEFAULT_SCAN_SCOPE, LOG_CAP_LINE, MAX_FINISHED_JOBS, MAX_LOG_LINES, MAX_OUTCOME_ROWS, SCAN_GROUP_NAMES, SCAN_GROUPS, + SCAN_SCOPES, Job, JobBusy, JobRegistry, @@ -155,7 +157,7 @@ class TestJobRegistry(unittest.IsolatedAsyncioTestCase): "scan", "Scan for metadata", record_changing=True, - scope={"mode": "all"}, + scope={"files": "new", "output_folders": 2}, runner=_gated_runner(gate), ) self.assertEqual(job.state, "running") @@ -181,7 +183,7 @@ class TestJobRegistry(unittest.IsolatedAsyncioTestCase): "log_length", }, ) - self.assertEqual(summary["scope"], {"mode": "all"}) + self.assertEqual(summary["scope"], {"files": "new", "output_folders": 2}) # What the detail would list, so a view reads it only when that changed. self.assertEqual((summary["listed"], summary["log_length"]), ({}, 0)) self.assertIsNone(summary["finished_at"]) @@ -371,7 +373,12 @@ class TestJobOutcomesAndLog(unittest.TestCase): job.record("bogus", {"path": "a.png", "root": "output"}, "x") def test_detail_and_progress(self): - job = Job("scan", "Scan", scope={"mode": "all"}, groups=SCAN_GROUP_NAMES) + job = Job( + "scan", + "Scan", + scope={"files": "new", "output_folders": 2}, + groups=SCAN_GROUP_NAMES, + ) job.update(progress=150, status_line="Processing file 3 of 4", processed=3) job.update(total=4) job.record("known", {"path": "a.png", "root": "output"}, "= a.png") @@ -395,12 +402,15 @@ class TestJobOutcomesAndLog(unittest.TestCase): "linked": "added", "enriched": "added", "unchanged": "known", + # Activity-2: a file the "new" scope skipped is Already known. + "skipped": "known", "ambiguous": "review", "conflict": "review", "unresolved": "review", "failed": "errors", }, ) + self.assertEqual((SCAN_SCOPES, DEFAULT_SCAN_SCOPE), (("new", "all"), "new")) self.assertEqual( SCAN_GROUP_NAMES, ("added", "known", "review", "no_metadata", "errors") ) diff --git a/tests/test_scan_scope.py b/tests/test_scan_scope.py new file mode 100644 index 0000000..6044714 --- /dev/null +++ b/tests/test_scan_scope.py @@ -0,0 +1,390 @@ +"""Scan scope (#249 Activity-2 Task 4): new and unresolved files by default. + +A "Scan for metadata" job started without ``scope`` (or with ``"new"``) skips +the files the Library already links with a resolved link, before their +metadata is read, and counts them as Already known. ``"all"`` reads every file +again, as the legacy ``/prompt_manager/scan`` route always does. +""" + +import asyncio +import json +import os +import sqlite3 +import unittest +from unittest.mock import patch + +try: + from tests.test_api_admin import AdminAPITestCase, _sse_payloads, _write_png +except ImportError: # discovered with tests/ as the top-level directory + from test_api_admin import AdminAPITestCase, _sse_payloads, _write_png + +from database import operations # noqa: E402 +from database.pair_ingestion import ingest_resolved_pair # noqa: E402 +from py.config import GalleryConfig # noqa: E402 +from utils.prompt_pair_resolver import resolve_prompt_pair # noqa: E402 + +SKIPPED_DETAIL = "Skipped · already linked" +ON_DISK = object() # _link_resolved: store the file's size on disk + + +def _graph(text): + return { + "1": {"class_type": "CLIPTextEncode", "inputs": {"text": text}}, + "2": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}}, + "3": { + "class_type": "KSampler", + "inputs": {"positive": ["1", 0], "negative": ["2", 0]}, + }, + } + + +class ScanScopeTestCase(AdminAPITestCase): + """Helpers: link files the way the scan and the monitor do, spy on reads.""" + + def _link_resolved(self, path, text, size=ON_DISK): + """``path`` linked with a resolved pair link (what a resolved scan writes). + + The row stores ``size`` as its file size: by default the file's size on + disk (None when the file does not exist). + """ + if size is ON_DISK: + size = os.path.getsize(path) if os.path.exists(path) else None + metadata = {"prompt": _graph(text)} + if size is not None: + metadata["file_info"] = {"size": size} + result = ingest_resolved_pair( + self.api.db, str(path), metadata, resolve_prompt_pair(metadata) + ) + self.assertEqual(result["status"], "added") + + def _link_without_pair(self, path, text): + """``path`` linked to a prompt with no pair link (fallback / FIFO link).""" + prompt_id = self.api.db.save_prompt(text, None, [], None, None) + self.assertTrue(self.api.db.link_image_to_prompt(prompt_id, str(path))) + return prompt_id + + def _spy_reads(self): + reads = [] + original = self.api._extract_comfyui_metadata + + def spy(path): + reads.append(os.path.basename(path)) + return original(path) + + self.api._extract_comfyui_metadata = spy + return reads + + async def _scan(self, body=None): + kwargs = {} if body is None else {"json": body} + resp = await self.client.request("POST", "/prompt_manager/jobs/scan", **kwargs) + data = await resp.json() + self.assertEqual(resp.status, 202, data) + await asyncio.wait_for(self.api._scan_job.task, 10) + detail = await self.client.request( + "GET", f"/prompt_manager/jobs/{data['job']['id']}" + ) + return (await detail.json())["job"] + + +class TestScanScope(ScanScopeTestCase): + + async def test_new_scope_skips_known_resolved_files_without_reading_them(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + known = self.output_dir / "known.png" + _write_png(known, prompt_text="a red square") + _write_png(self.output_dir / "new.png", prompt_text="a blue circle") + self._link_resolved(known, "a red square") + reads = self._spy_reads() + + job = await self._scan() + + self.assertEqual(reads, ["new.png"]) + self.assertEqual(job["scope"]["files"], "new") + self.assertEqual(job["counters"]["known"], 1) + self.assertEqual(job["counters"]["added"], 1) + self.assertEqual(job["processed"], 2) + self.assertEqual( + job["outcomes"]["known"], + [ + { + "path": "known.png", + "root": "output", + "root_index": 0, + "detail": SKIPPED_DETAIL, + } + ], + ) + self.assertIn(f"= output/known.png · {SKIPPED_DETAIL}", "\n".join(job["log"])) + self.assertTrue( + job["log"][0].endswith( + "Scan started · 1 output folder · new and unresolved files" + ) + ) + legacy = self.api._scan_job.last_event + self.assertEqual((legacy["processed"], legacy["unchanged"]), (2, 1)) + self.assertTrue(job["status_line"].startswith("Processing file 2 of 2")) + + async def test_whole_known_batch_skips_the_executor_reads(self): + from py.api import admin + + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + for name in ("a.png", "b.png"): + _write_png(self.output_dir / name, prompt_text=f"prompt {name}") + self._link_resolved(self.output_dir / name, f"prompt {name}") + _write_png(self.output_dir / "c.png", prompt_text="prompt c") + reads = self._spy_reads() + + with patch.object(admin, "SCAN_BATCH_SIZE", 1): + job = await self._scan({"scope": "new"}) + + self.assertEqual(reads, ["c.png"]) + self.assertEqual(job["counters"]["known"], 2) + self.assertEqual(job["counters"]["added"], 1) + + async def test_unresolved_files_are_rescanned_in_new_scope(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + names = ("fallback.png", "conflict.png", "unlinked.png") + for name in names: + _write_png(self.output_dir / name, prompt_text=f"text of {name}") + # Linked without a resolved pair link (the monitor's fallback / FIFO link). + self._link_without_pair(self.output_dir / "fallback.png", "text of fallback") + # Resolved once, then also linked to a second prompt: ambiguous. + conflict = self.output_dir / "conflict.png" + self._link_resolved(conflict, "text of conflict.png") + self._link_without_pair(conflict, "another prompt") + reads = self._spy_reads() + + job = await self._scan() + + self.assertEqual(sorted(reads), sorted(names)) + self.assertNotIn(SKIPPED_DETAIL, json.dumps(job["outcomes"])) + + async def test_all_scope_reads_everything(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + known = self.output_dir / "known.png" + _write_png(known, prompt_text="a red square") + self._link_resolved(known, "a red square") + reads = self._spy_reads() + + job = await self._scan({"scope": "all"}) + + self.assertEqual(reads, ["known.png"]) + self.assertEqual(job["scope"]["files"], "all") + self.assertEqual(job["counters"]["known"], 1) + self.assertNotEqual(job["outcomes"]["known"][0]["detail"], SKIPPED_DETAIL) + self.assertTrue( + job["log"][0].endswith("Scan started · 1 output folder · all files") + ) + + async def test_legacy_scan_route_reads_everything(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + known = self.output_dir / "known.png" + _write_png(known, prompt_text="a red square") + self._link_resolved(known, "a red square") + reads = self._spy_reads() + + resp = await self.client.request("POST", "/prompt_manager/scan", data="{}") + payloads = _sse_payloads(await resp.text()) + + self.assertEqual(reads, ["known.png"]) + self.assertEqual(self.api._scan_job.scope["files"], "all") + self.assertEqual(payloads[-1]["type"], "complete") + self.assertEqual(payloads[-1]["unchanged"], 1) + + async def test_file_replaced_at_a_known_path_is_reread_and_needs_review(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + path = self.output_dir / "out_00012.png" + _write_png(path, prompt_text="the first prompt") + self._link_resolved(path, "the first prompt") + before = os.path.getsize(path) + # ComfyUI reused the name for a new image with another prompt. + _write_png(path, prompt_text="a much longer second prompt for the same name") + self.assertNotEqual(os.path.getsize(path), before) + reads = self._spy_reads() + + job = await self._scan() + + self.assertEqual(reads, ["out_00012.png"]) + self.assertEqual(job["counters"]["review"], 1) + self.assertEqual(job["counters"]["known"], 0) + self.assertIn("different prompt", job["outcomes"]["review"][0]["detail"]) + + async def test_scanned_files_store_their_size_so_the_next_new_scan_skips(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + _write_png(self.output_dir / "fresh.png", prompt_text="a fresh prompt") + first = await self._scan() + self.assertEqual(first["counters"]["added"], 1) + reads = self._spy_reads() + + second = await self._scan() + + self.assertEqual(reads, []) + self.assertEqual(second["outcomes"]["known"][0]["detail"], SKIPPED_DETAIL) + + async def test_stored_null_size_is_reread(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + path = self.output_dir / "old.png" + _write_png(path, prompt_text="an old prompt") + self._link_resolved(path, "an old prompt", size=None) + reads = self._spy_reads() + + job = await self._scan() + + self.assertEqual(reads, ["old.png"]) + # The read stores the size it was missing, so the file is known next time. + self.assertEqual(job["counters"]["added"], 1) + self.assertEqual( + self.api.db.get_known_resolved_paths({str(path): os.path.getsize(path)}), + {str(path)}, + ) + + async def test_known_set_failure_falls_back_to_reading_every_file(self): + GalleryConfig.MONITORING_DIRECTORIES = [self._root(self.output_dir)] + known = self.output_dir / "known.png" + _write_png(known, prompt_text="a red square") + self._link_resolved(known, "a red square") + reads = self._spy_reads() + + def broken(files): + raise sqlite3.OperationalError("database is locked") + + self.api.db.get_known_resolved_paths = broken + with self.assertLogs("prompt_manager.api", level="WARNING"): + job = await self._scan() + + self.assertEqual(job["state"], "finished") + self.assertEqual(reads, ["known.png"]) + self.assertIn( + "Known files could not be read · reading every file", "\n".join(job["log"]) + ) + + async def test_empty_read_body_asks_for_the_default_scope(self): + """A chunked POST with no bytes can read a body: still the default.""" + from py.api.job_routes import _scan_scope + + class Request: + can_read_body = True + + def __init__(self, text): + self._text = text + + async def text(self): + return self._text + + self.assertEqual(await _scan_scope(Request("")), "new") + self.assertEqual(await _scan_scope(Request(" \n")), "new") + self.assertEqual(await _scan_scope(Request('{"scope": "all"}')), "all") + self.assertIsNone(await _scan_scope(Request("{"))) + + async def test_invalid_scope_is_refused(self): + for body in ({"scope": "everything"}, {"scope": 1}, ["new"]): + with self.subTest(body=body): + resp = await self.client.request( + "POST", "/prompt_manager/jobs/scan", json=body + ) + self.assertEqual(resp.status, 400) + data = await resp.json() + self.assertFalse(data["success"]) + resp = await self.client.request( + "POST", "/prompt_manager/jobs/scan", data="not json" + ) + self.assertEqual(resp.status, 400) + self.assertIsNone(self.api._scan_job) + + +class TestKnownResolvedPaths(ScanScopeTestCase): + """``PromptDatabase.get_known_resolved_paths``: one streamed query.""" + + async def test_returns_the_given_paths_that_are_known_and_resolved(self): + known, fallback = self.output_dir / "known.png", self.output_dir / "fb.png" + self._link_resolved(known, "a red square", size=10) + self._link_without_pair(fallback, "a fallback") + asked = {str(known): 10, str(fallback): 10, str(self.output_dir / "n.png"): 10} + + self.assertEqual(self.api.db.get_known_resolved_paths(asked), {str(known)}) + self.assertEqual(self.api.db.get_known_resolved_paths({}), set()) + + async def test_size_must_match_and_a_stored_null_size_is_not_known(self): + same, other = self.output_dir / "same.png", self.output_dir / "other.png" + unknown = self.output_dir / "unknown.png" + self._link_resolved(same, "same size", size=10) + self._link_resolved(other, "other size", size=10) + self._link_resolved(unknown, "no stored size", size=None) + asked = {str(same): 10, str(other): 11, str(unknown): 10} + + self.assertEqual(self.api.db.get_known_resolved_paths(asked), {str(same)}) + # A file the walk could not stat is not known either. + self.assertEqual(self.api.db.get_known_resolved_paths({str(same): None}), set()) + + async def test_row_without_file_path_matches_through_image_path(self): + known = self.output_dir / "legacy.png" + self._link_resolved(known, "a legacy row", size=10) + with sqlite3.connect(self.api.db.model.db_path) as conn: + conn.execute("UPDATE generated_images SET file_path=NULL") + conn.close() + + found = self.api.db.get_known_resolved_paths({str(known): 10}) + self.assertEqual(found, {str(known)}) + + async def test_stored_spelling_is_normalized_like_the_scanned_path(self): + known = self.output_dir / "sub" / "known.png" + self._link_resolved(known, "a red square", size=10) + spelled = os.path.join(str(self.output_dir), "sub", "..", "sub", "known.png") + with sqlite3.connect(self.api.db.model.db_path) as conn: + conn.execute("UPDATE generated_images SET file_path=?", (spelled,)) + conn.close() + + found = self.api.db.get_known_resolved_paths({str(known): 10}) + self.assertEqual(found, {str(known)}) + + async def test_one_query_streamed_in_small_fetches(self): + paths = [self.output_dir / f"{index}.png" for index in range(5)] + for path in paths: + self._link_resolved(path, f"prompt {path.name}", size=10) + original = self.api.db.model.get_connection + opened = [] + + def counting(): + opened.append(1) + return original() + + self.api.db.model.get_connection = counting + with patch.object(operations, "KNOWN_PATHS_FETCH_SIZE", 2): + found = self.api.db.get_known_resolved_paths({str(p): 10 for p in paths}) + self.assertEqual(found, {str(p) for p in paths}) + self.assertEqual(len(opened), 1) + + async def test_windows_path_case_matches_known_set(self): + stored = self.output_dir / "Sub" / "Known.PNG" + self._link_resolved(stored, "a red square", size=10) + # A row written before paths were case-folded keeps its own spelling. + with sqlite3.connect(self.api.db.model.db_path) as conn: + conn.execute( + "UPDATE generated_images SET file_path=?", (str(stored).upper(),) + ) + conn.close() + asked = str(self.output_dir / "sub" / "known.png") + + with patch("os.path.normcase", lambda path: path.lower()): + self.assertEqual(self.api.db.get_known_resolved_paths({asked: 10}), {asked}) + if os.path.normcase("A") == "A": # a case-sensitive platform + self.assertEqual(self.api.db.get_known_resolved_paths({asked: 10}), set()) + + +class TestSkippedOutcome(unittest.TestCase): + + def test_skipped_file_is_already_known_with_its_own_detail(self): + from py.api.admin import _scan_outcome + + root = os.path.abspath("out") + group, row, line = _scan_outcome( + os.path.join(root, "a", "b.png"), [(root, "output", 0)], "skipped" + ) + self.assertEqual(group, "known") + self.assertEqual(row["detail"], SKIPPED_DETAIL) + self.assertEqual(row["path"], "a/b.png") + self.assertEqual(line, f"= output/a/b.png · {SKIPPED_DETAIL}") + + +if __name__ == "__main__": + unittest.main() diff --git a/web/js/workspace/activity-job-view.mjs b/web/js/workspace/activity-job-view.mjs index b3a6346..0732989 100644 --- a/web/js/workspace/activity-job-view.mjs +++ b/web/js/workspace/activity-job-view.mjs @@ -3,7 +3,7 @@ * concept's classes. * * - Header: the page heading names the job; its line is the Activity item's (D54), and its actions are Download report - * (disabled until Activity-2) and, while the job runs, "Cancel scan" (danger). + * (the job's JSON report, D59) and, while the job runs, "Cancel scan" (danger). * - Progress (`section.surface`): `<progress>`, the server's "Processing file N of M · about X min left", and the * outcome counters as `dl.stat-row` tiles: Added to Library (green), Already known, Needs review (warning), No usable * metadata, Errors (red). @@ -21,6 +21,7 @@ import { showWorkspaceToast } from './toast.mjs'; import { noun } from './plural.mjs'; import { parseJob, JOBS_URL, LOST_DETAIL } from './activity-store.mjs'; import { jobDetail, CANCEL_FAILED_TOAST } from './activity.mjs'; +import { downloadJobReport } from './activity-scan.mjs'; import { createMediaViewer } from './media-viewer.mjs'; import { createInspectionController } from './inspection-state.mjs'; import { createPromptSender } from './prompt-sender.mjs'; @@ -122,7 +123,6 @@ function section(title, id) { export function mountActivityJobView(main, { store, scope, api, jobId }) { const parts = pageHeading(main); const download = createButton('Download report'); - download.disabled = true; const cancel = createButton('Cancel scan', 'danger'); const progressPart = section('Progress', 'job-progress-title'); @@ -271,6 +271,10 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { document.title = `${job.title} · Prompt Manager`; parts.description.textContent = jobDetail(job); cancel.textContent = scan ? 'Cancel scan' : 'Cancel job'; + // A job lost with the server has no report there (review M9). + const lost = job.state === 'lost'; + if (lost && document.activeElement === download) parts.h1.focus(); + download.disabled = lost; if (job.state === 'running') { if (!cancel.isConnected) parts.actions.append(cancel); } else if (cancel.isConnected) { @@ -290,9 +294,11 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { logLabel.textContent = `● ${scan ? 'Scan' : 'Job'} log`; const text = detail?.log.length ? detail.log.join('\n') : 'No log lines yet.'; if (pre.textContent !== text) { + // Replacing the text clamps the scroll position, so keep it: pinned readers follow the end, others stay. + const { scrollTop } = pre; const atEnd = pre.scrollTop + pre.clientHeight >= pre.scrollHeight - 2; pre.textContent = text; - if (atEnd) pre.scrollTop = pre.scrollHeight; + pre.scrollTop = atEnd ? pre.scrollHeight : scrollTop; } renderRows(job); } @@ -323,7 +329,12 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { reading = false; } render(); - if (readAgain) { readAgain = false; scheduleRead(); } + // Read once more only if the summary moved past what this read returned (review: no redundant re-read). + if (readAgain) { + readAgain = false; + const job = store.find(jobId); + if (job && job.state !== 'lost' && detailChanged(job)) scheduleRead(); + } } /** @@ -347,10 +358,23 @@ export function mountActivityJobView(main, { store, scope, api, jobId }) { } scope.listen(log, 'toggle', () => { + // Every open starts at the newest line: while closed the log has no layout box (Firefox, WebKit), so a kept + // position would be lost anyway (review M6). + if (log.open) pre.scrollTop = pre.scrollHeight; const job = store.find(jobId); if (log.open && job && job.state !== 'lost' && detailChanged(job)) scheduleRead(); }); + scope.listen(download, 'click', async () => { + if (download.disabled || download.getAttribute('aria-disabled') === 'true') return; + download.setAttribute('aria-disabled', 'true'); + try { + // A view left while the report downloads shows no toast (review M10). + await downloadJobReport(jobId, { signal: scope.signal }); + } finally { + download.removeAttribute('aria-disabled'); + } + }); scope.listen(cancel, 'click', async () => { if (cancel.getAttribute('aria-disabled') === 'true') return; cancel.setAttribute('aria-disabled', 'true'); diff --git a/web/js/workspace/activity-scan.mjs b/web/js/workspace/activity-scan.mjs index 9fdfd09..8ec2c87 100644 --- a/web/js/workspace/activity-scan.mjs +++ b/web/js/workspace/activity-scan.mjs @@ -1,7 +1,8 @@ /** * Scan for metadata (#249 Activity-1, D55): Images → Import → "Scan for metadata" → Review scan opens the concept's * "Review metadata scan". Its backup box downloads the existing `/prompt_manager/backup` copy and toasts the outcome; - * Start scan starts the session job through the Activity store, which toasts "Scan started.", and opens the job's view. + * Start scan starts the session job through the Activity store over the chosen scope (new and unresolved files, or + * all files with the D58 checkbox), which toasts "Scan started.", and opens the job's view. * A start refused because another record-changing job runs toasts "Another job is running: <title>." with a link to * that job (D57). */ @@ -12,13 +13,29 @@ import { navigateTo } from './state.mjs'; export const BACKUP_URL = '/prompt_manager/backup'; export const BACKUP_DONE = 'Database backup downloaded.'; export const BACKUP_FAILED = 'The database could not be backed up. Try again.'; +export const REPORT_DONE = 'Job report downloaded.'; +export const REPORT_FAILED = 'The report could not be downloaded. Try again.'; +const JOBS_REPORT_BASE = '/prompt_manager/jobs'; export const START_FAILED = 'The scan could not start. Try again.'; const FALLBACK_NAME = 'prompts_backup.db'; +const REPORT_FALLBACK_NAME = 'prompt-manager-report.json'; -/** The download name from a Content-Disposition header, or the fallback. */ -export function backupFilename(disposition) { +/** The download name from a Content-Disposition header, or `fallback`. */ +export function backupFilename(disposition, fallback = FALLBACK_NAME) { const match = /filename="?([^";]+)"?/i.exec(disposition || ''); - return match ? match[1] : FALLBACK_NAME; + return match ? match[1] : fallback; +} + +/** Save a fetched response through a temporary link, named by `filename`. */ +function saveResponse(blob, filename) { + const url = URL.createObjectURL(blob); + const link = document.createElement('a'); + link.href = url; + link.download = filename; + document.body.append(link); + link.click(); + link.remove(); + setTimeout(() => URL.revokeObjectURL(url), 1000); } /** Download the prompts database copy the legacy admin page also uses; toasts the outcome. */ @@ -27,14 +44,7 @@ export async function downloadDatabaseBackup({ fetchUrl = window.ApiClient.fetch const response = await fetchUrl(BACKUP_URL); if (!response.ok) throw new Error(`Backup failed with ${response.status}`); const blob = await response.blob(); - const url = URL.createObjectURL(blob); - const link = document.createElement('a'); - link.href = url; - link.download = backupFilename(response.headers.get('Content-Disposition')); - document.body.append(link); - link.click(); - link.remove(); - setTimeout(() => URL.revokeObjectURL(url), 1000); + saveResponse(blob, backupFilename(response.headers.get('Content-Disposition'))); notify(BACKUP_DONE); return true; } catch { @@ -43,16 +53,39 @@ export async function downloadDatabaseBackup({ fetchUrl = window.ApiClient.fetch } } +/** + * Download a job's JSON report (Activity-2, D59) through the api-client URL, so a proxy mount works; toasts the outcome + * unless `signal` (the view's scope) was aborted meanwhile. + */ +export async function downloadJobReport(jobId, { fetchUrl = window.ApiClient.fetchUrl, notify = showWorkspaceToast, + signal = null } = {}) { + try { + const response = await fetchUrl(`${JOBS_REPORT_BASE}/${encodeURIComponent(jobId)}/report`); + if (!response.ok) throw new Error(`Report failed with ${response.status}`); + const blob = await response.blob(); + saveResponse(blob, backupFilename(response.headers.get('Content-Disposition'), REPORT_FALLBACK_NAME)); + // The file is saved either way; a view left meanwhile (`signal` aborted) shows no toast. + if (!signal?.aborted) notify(REPORT_DONE); + return true; + } catch { + if (!signal?.aborted) notify(REPORT_FAILED); + return false; + } +} + /** The toast for a refused start, linking the job that holds the slot. */ export function busyToast(running) { if (!running) return 'Another job is running. Try again when it ends.'; return ['Another job is running: ', { text: running.title, href: `#/activity/${running.id}` }, '.']; } -/** Start the scan job and open its view; a refused or failed start says why and stays on the page. */ -export async function startMetadataScan({ store, scope, notify = showWorkspaceToast, navigate = navigateTo }) { +/** + * Start the scan job over `files` ('new': new and unresolved files, or 'all') and open its view; a refused or failed + * start says why and stays on the page. + */ +export async function startMetadataScan({ store, scope, files = 'new', notify = showWorkspaceToast, navigate = navigateTo }) { try { - const result = await store.start('scan'); + const result = await store.start('scan', { scope: files }); if (scope.signal.aborted) return; if (result.busy !== undefined) { notify(busyToast(result.busy)); @@ -68,5 +101,5 @@ export async function startMetadataScan({ store, scope, notify = showWorkspaceTo export function openMetadataScan({ scope, store, sources, trigger = null }) { openScanReviewDialog({ scope, sources, trigger, onBackup: () => downloadDatabaseBackup(), - onStart: () => startMetadataScan({ store, scope }) }); + onStart: ({ scope: files }) => startMetadataScan({ store, scope, files }) }); } diff --git a/web/js/workspace/workspace-dialogs.mjs b/web/js/workspace/workspace-dialogs.mjs index 6872a68..72b3e86 100644 --- a/web/js/workspace/workspace-dialogs.mjs +++ b/web/js/workspace/workspace-dialogs.mjs @@ -20,7 +20,7 @@ * * Every opener takes the page `scope`: disposing it closes and removes the shell, so a dialog never outlives its page. */ -import { createButton } from '../shared/toolkit/core.js'; +import { createButton, createSwitch } from '../shared/toolkit/core.js'; const FIELD_LABELS = Object.freeze({ source: 'Source', folder: 'Folder', media: 'Media', tags: 'Tags' }); const FIELD_ORDER = Object.keys(FIELD_LABELS); @@ -342,22 +342,44 @@ export const SCAN_REVIEW_SENTENCE = 'Enrich positive / negative associations, re const BACKUP_LABEL = 'Back up now'; const BACKUP_BUSY_LABEL = 'Backing up…'; +export const SCAN_SCOPE_LABELS = { new: 'New and unresolved files', all: 'All files' }; +export const SCAN_ALL_FILES_HINT = 'Also read the files already linked to their prompt again.'; + /** - * The concept's "Review metadata scan" (`scan-review`, D55). `sources` is the number of output folders the scan reads; - * the scope is fixed to "All files" in Activity-1. `onBackup` runs the database backup (the button is aria-disabled - * and reads "Backing up…" until it settles, and the dialog stays open); Start scan closes the dialog, then `onStart`. + * The concept's "Review metadata scan" (`scan-review`, D55). `sources` is the number of output folders the scan reads. + * The scope is the concept's "New and unresolved files"; the "All files" checkbox (D58, the concept's `.setting-row` + * switch, unchecked) switches the Scope value to "All files". `onBackup` runs the database backup (the button is + * aria-disabled and reads "Backing up…" until it settles, and the dialog stays open); Start scan closes the dialog, then + * calls `onStart({scope: 'new' | 'all'})`. */ export function openScanReviewDialog({ scope, sources, onBackup, onStart, trigger = null }) { requireScope(scope); requireCallbacks({ onBackup, onStart }); if (!Number.isSafeInteger(sources) || sources < 0) throw new TypeError('The scan review needs its source count'); const grid = el('dl', undefined, 'meta-grid'); + const values = {}; for (const [term, value] of [['Sources', `${sources.toLocaleString('en-US')} output ${sources === 1 ? 'folder' : 'folders'}`], - ['Scope', 'All files']]) { + ['Scope', SCAN_SCOPE_LABELS.new]]) { const entry = el('div'); - entry.append(el('dt', term), el('dd', value)); + values[term] = el('dd', value); + entry.append(el('dt', term), values[term]); grid.append(entry); } + // The concept's System toggle: a `.setting-row` whose checkbox is the Toolkit switch (concept `hydrateSwitches`). + const toggle = createSwitch('', '', 'scan-all-files'); + const allFiles = toggle.querySelector('input'); + allFiles.checked = false; + const option = el('div', undefined, 'setting-row'); + const optionText = el('div'); + const optionLabel = el('label', SCAN_SCOPE_LABELS.all); + optionLabel.htmlFor = allFiles.id; + const hint = el('p', SCAN_ALL_FILES_HINT); + hint.id = 'scan-all-files-hint'; + allFiles.setAttribute('aria-describedby', hint.id); + optionText.append(optionLabel, hint); + option.append(optionText, toggle); + const chosenScope = () => (allFiles.checked ? 'all' : 'new'); + allFiles.addEventListener('change', () => { values.Scope.textContent = SCAN_SCOPE_LABELS[chosenScope()]; }); const notice = el('div', undefined, 'notice'); const copy = el('div'); copy.append(el('h3', 'Back up the database first (recommended)'), @@ -381,10 +403,11 @@ export function openScanReviewDialog({ scope, sources, onBackup, onStart, trigge const start = createButton('Start scan', 'ca-primary'); cancel.addEventListener('click', () => finish()); start.addEventListener('click', () => { + const files = chosenScope(); finish(); - onStart(); + onStart({ scope: files }); }); const actions = el('div', undefined, 'dialog-actions'); actions.append(cancel, start); - show('Review metadata scan', [grid, el('p', SCAN_REVIEW_SENTENCE, 'small'), notice, actions], { scope, trigger }); + show('Review metadata scan', [grid, option, el('p', SCAN_REVIEW_SENTENCE, 'small'), notice, actions], { scope, trigger }); }