diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3e1914e..a530023 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -87,9 +87,14 @@ covered by the `styles.css` manifest. whole models folder with its subfolders for the models page's type chips. - `api/scanner.py` and `scraper.py` own scan orchestration and enrichment; both scan routes map request fields to scraper switches in one place (`_scraper_flags`). + `civitai_client.py` is the scraper's network side (version by hash, model page, media), + telling "Civitai does not know the file" from "Civitai could not be asked". + `api/scan_report.py` turns the scraper's per-model report lines into one result per + scan: kept as the last scan (`/anomalous/last_scan`), recorded in the activity log, each + model located so the browser can open it (`/anomalous/scan_model`). `api/scan_summary.py` counts models matched on Civitai, unmatched and not yet scanned - for the scan page, by the same sidecar rules the scraper uses (`sidecar_info`, - `is_unmatched` in `model_identity.py`). + for the scan page and lists the last two, by the same sidecar rules the scraper uses + (`sidecar_info`, `is_unmatched`, `unmatched_reason` in `model_identity.py`). - `api/workflow_schema.py`, `api/recipe_schema.py`, `api/recipe_images.py`, and `api/recipe_store.py` own recipe validation/shaping, images, CRUD, history, and integrity receipts; `api/recipes.py` is the HTTP facade. `api/recipe_constants.py` @@ -282,7 +287,8 @@ covered by the `styles.css` manifest. The activity log (docs/decisions AD-018): `api/activity_log.py` keeps the entries (newest first, bounded, in the ComfyUI user folder), records this plugin's and Anomalous_TTS's successful write requests in a middleware, and serves - `/anomalous/activity`; `activity_canvas.js` records canvas changes by comparing a + `/anomalous/activity` (a handler may attach what it changed, `request[ACTIVITY_DETAIL]`; + scans record themselves when they end); `activity_canvas.js` records canvas changes by comparing a snapshot taken when the user presses inside Anomalous with the canvas when they next press or type outside it (`activity_diff.js` holds the pure snapshot and difference); `activity_log.js` is the client and the words for each entry; @@ -293,9 +299,12 @@ covered by the `styles.css` manifest. including Esc on the workspace panel (`nbPanel`, below the header and right of the rail; the list column steps aside while it is open). `ui_scan_page.js` is the scan page (a tool page like the doctor): counts, scan and - look-up-again buttons, advanced options. `scan_runner.js` starts and follows scans - (every folder, picked models, or one model from its card) and refreshes node - drop-downs, hashes and the grid afterwards; folder visibility/order lives in + look-up-again buttons, advanced options; `ui_scan_lists.js` renders its last-scan result + and the unmatched / not-scanned lists, whose rows open a model or scan it; + `scan_results.js` holds the words for scan results, shared with the activity page. + `scan_runner.js` starts and follows scans (every folder, picked or listed models, or one + model from its card), shows the result and refreshes node drop-downs, hashes and the + grid afterwards; folder visibility/order lives in `ui_folder_manager.js`; and help content lives in `ui_help.js`. `scan_progress.js` owns the scan progress panel (`updateScanProgress` / `finishScanProgress` / `failScanProgress`): inside the scan diff --git a/CHANGELOG.md b/CHANGELOG.md index b82c26f..ac0c117 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,8 @@ - **Deleting goes to the Recycle Bin (删除进回收站)**: deleting a model (with its info and covers), an output image, generated audio, a recipe (with its versions and images), a material, a prompt note or a parameter note, replacing or resetting a cover, and the duplicate copy a file rename finds, all move the files to the system Recycle Bin instead of erasing them. On a USB stick or network drive, which has no Recycle Bin, nothing is deleted and the message says to delete it by hand. The confirmations say where things go. An unused route that could erase every `.civitai.info` in the library (with the names you set) is gone. - **Scan page (扫描页)**: the rail's scan button and Home's "Scan model folders" open a page instead of the old seven-switch dialog. It shows how many models are matched on Civitai, not matched, and not scanned yet; one button scans the new ones, with progress in the page (it runs in the background, so the window can be closed). Offline scanning, renaming files on disk, fetching everything again, picking models and the Civitai API key are under **Advanced**; the two risky switches turn themselves off after each scan. The "record model fingerprints in workflows" switch moved to Settings → Model settings. - **Unmatched models can be looked up again (没匹配上的模型可以重新查找)**: a model that Civitai did not know, or that was scanned while Civitai could not be reached, used to stay "local only" for good unless every model was fetched again (which also replaced your own covers). **Look up the N unmatched models again** retries just those, keeps your covers, and reuses the saved hash when the file is unchanged, so it takes seconds instead of re-hashing gigabytes. +- **Which models, and what a scan did (是哪些模型、扫描做了什么)**: on the scan page, **Not matched** and **Not scanned yet** open a list of those models, with their type and folder; unmatched ones say why (Civitai does not have the file, Civitai could not be reached then, or it was an offline scan) and the base model inferred. Each row opens the model (Back returns to the scan page), scans it or looks it up again, or searches Civitai for its name. After a scan the page shows its result: when, how long, how many matched, inferred, failed, covers downloaded and files renamed, and each model it changed with what happened; the progress box says the same and, when it floats over the canvas, has **Show the result**. A model's card scan looks an unmatched model up again and its message says what it found. +- **Online and offline in one scan (联网和离线在同一次扫描里)**: a scan asks Civitai first and infers the base model from the file for what Civitai does not know, as before; now it also remembers why. When Civitai stops answering (two models in a row), the rest of the scan infers from the files right away instead of waiting out every timeout, and the next online scan looks those models up again by itself, as it does for models from an offline scan. Models Civitai answered "not found" for wait for **Look up again**. A model that fails (a locked file, say) no longer ends the scan of the others; it is listed as failed with the reason. - **Your edits come first (你改的永远优先)**: names, notes and download links you set in the model editor now have a file of their own next to the model (`.anomalous.json`) that scans never write, and they win over Civitai's information, which wins over what is inferred from the file. Edits saved by earlier versions keep working. Editing a model that was never scanned used to make it look scanned, so scans skipped it for good; it is now scanned like any new model. - **Covers you chose are never replaced (你换的封面不会被替换)**: a cover counts as Civitai's only while it is the very image the scan downloaded; any other cover is yours. **Fetch scanned models again** now keeps your covers (and names and notes), updates only Civitai's own, and asks before it starts. - **Where a model's information comes from (信息来源标注)**: cards look as before for models matched on Civitai; a base model inferred from the file shows as "≈ SDXL" with a dashed outline, and a model not scanned yet says so, each with a tooltip on what to do. The detail header adds the source (Civitai, inferred from the file, not scanned) and which fields you edited. @@ -22,6 +24,7 @@ ### 🕘 Activity log (操作记录) - **See what Anomalous changed (看 Anomalous 改了什么)**: the new **Activity** page on the rail lists, by day, every change the plugin made: on the canvas, which node settings changed (from → to), which nodes were added or removed, and which workflow was opened; in files, models, covers, recipes, materials, notes, presets, images, audio and imported characters. Filter by canvas or files; open a canvas entry for its details and **Find on canvas** to jump to the node. Your own edits on the canvas are not recorded as the plugin's. The home page shows the latest few. +- **More detail (记录更详细)**: a scan is recorded when it ends, with what it found and each model it changed (open one from the entry), instead of a bare "scanned models" when it started. Editing a model records each field from → to, and deleting one lists the files moved to the Recycle Bin. - It only records; nothing is undone for you. **Clear** removes the log itself, never your changes. The newest 500 entries (at most 2 MB) are kept in ComfyUI's user folder. ### 🎙️ Audio & Voice Studio (音频与声音工作台) diff --git a/api/__init__.py b/api/__init__.py index bcad303..aa640e2 100644 --- a/api/__init__.py +++ b/api/__init__.py @@ -1,6 +1,6 @@ from . import ( activity_log, audio_catalog, folder_types, gallery_routes, materials, media_routes, model_catalog, - model_media, model_metadata, model_resolution, model_type_listing, recipe_packages, recipes, scan_summary, + model_media, model_metadata, model_resolution, model_type_listing, recipe_packages, recipes, scan_report, scan_summary, translation_routes, version_manager, ) from .scanner import * @@ -106,6 +106,8 @@ def setup_routes(app): app.router.add_post('/anomalous/resolve_paths_to_previews', model_catalog.api_resolve_paths_to_previews) app.router.add_post('/anomalous/scan_missing_models', api_scan_missing_models) app.router.add_get('/anomalous/scan_summary', scan_summary.api_scan_summary) + app.router.add_get('/anomalous/last_scan', scan_report.api_last_scan) + app.router.add_get('/anomalous/scan_model', scan_report.api_scan_model) # Audio & Voice Studio Routes app.router.add_get('/anomalous/audio_stream', audio_catalog.api_serve_audio) diff --git a/api/activity_log.py b/api/activity_log.py index 65af2d5..1315f85 100644 --- a/api/activity_log.py +++ b/api/activity_log.py @@ -26,14 +26,13 @@ MAX_LOG_BYTES = 2 * 1024 * 1024 # long canvas entries could otherwise outgrow t MAX_VALUE_CHARS = 400 MAX_CHANGES = 60 -# Write routes worth a line in the log → the action id the browser has words for. +# Write routes worth a line in the log → the action id the browser has words for. Scans are +# recorded when they end, with what they found (api/scan_report.py), not here. ROUTE_ACTIONS = { '/anomalous/delete_model': 'model_delete', '/anomalous/update_metadata': 'model_edit', '/anomalous/set_custom_cover': 'model_cover', '/anomalous/upload_custom_cover': 'model_cover', - '/anomalous/scan': 'scan', - '/anomalous/scan_all': 'scan', '/anomalous/save_recipe': 'recipe_save', '/anomalous/update_recipe': 'recipe_edit', '/anomalous/delete_recipe': 'recipe_delete', @@ -63,6 +62,10 @@ ROUTE_ACTIONS = { TARGET_FIELDS = ('name', 'character', 'filename', 'target_filename', 'recipe_filename', 'notebook_filename', 'new_name', 'path', 'file', 'target') +# What a handler attaches to its request for the log: request[ACTIVITY_DETAIL] = {fields, files}. +# web.RequestKey is newer than the aiohttp some ComfyUI installs have; a plain key works there. +ACTIVITY_DETAIL = web.RequestKey("activity_detail", dict) if hasattr(web, "RequestKey") else "activity_detail" + _lock = threading.RLock() _entries = None @@ -173,6 +176,20 @@ async def _request_body(request): return {} +def _clean_detail(detail): + """What a handler attached for the log (request[ACTIVITY_DETAIL]): the fields it changed + ({field, before, after}) and the files it moved, values shortened.""" + if not isinstance(detail, dict): + return None + cleaned = {} + if isinstance(detail.get('fields'), list): + cleaned['fields'] = [{key: _short(item[key]) for key in ('field', 'before', 'after') if key in item} + for item in detail['fields'][:MAX_CHANGES] if isinstance(item, dict)] + if isinstance(detail.get('files'), list): + cleaned['files'] = [_short(name) for name in detail['files'][:MAX_CHANGES]] + return {key: value for key, value in cleaned.items() if value} or None + + @web.middleware async def activity_middleware(request, handler): response = await handler(request) @@ -180,7 +197,8 @@ async def activity_middleware(request, handler): if action and not _failed(response): try: target = _target_of(await _request_body(request)) - await asyncio.get_running_loop().run_in_executor(None, add_entry, 'file', action, target) + detail = _clean_detail(request.get(ACTIVITY_DETAIL)) + await asyncio.get_running_loop().run_in_executor(None, add_entry, 'file', action, target, detail) except Exception: # noqa: BLE001 - the log must never fail the action it records logging.getLogger(__name__).warning('Anomalous activity log: could not record %s', request.path, exc_info=True) return response diff --git a/api/metadata.py b/api/metadata.py index 2466d09..b60a918 100644 --- a/api/metadata.py +++ b/api/metadata.py @@ -11,9 +11,9 @@ from aiohttp import web import folder_paths import struct try: - from ..model_identity import USER_INFO_SUFFIX, is_unmatched, read_json, sidecar_file_hash, sidecar_info + from ..model_identity import USER_INFO_SUFFIX, is_unmatched, read_json, sidecar_file_hash, sidecar_info, unmatched_reason except ImportError: - from model_identity import USER_INFO_SUFFIX, is_unmatched, read_json, sidecar_file_hash, sidecar_info + from model_identity import USER_INFO_SUFFIX, is_unmatched, read_json, sidecar_file_hash, sidecar_info, unmatched_reason # Fields the model editor sets: .anomalous.json key -> metadata key. USER_FIELDS = {"custom_name": "custom_name", "custom_notes": "custom_notes", "source_url": "source_url"} @@ -102,7 +102,8 @@ def _read_metadata(file_path): "custom_name": "", "custom_notes": "", # "civitai" (matched), "local" (only what the file tells: no match, or offline) or "" (not scanned) - "info_source": "", + "info_source": "", + "unmatched_reason": "", # for "local": not_found | network | offline | "" (scanned before this was kept) "user_fields": [], } @@ -163,7 +164,8 @@ def _read_metadata(file_path): scan = sidecar_info(base_path) if scan is not None: - metadata["info_source"] = "local" if is_unmatched(scan) else "civitai" + metadata["info_source"] = "local" if is_unmatched(scan) else "civitai" + metadata["unmatched_reason"] = unmatched_reason(scan) # The user's own layer comes last and wins, also when a field was cleared on purpose (""). user = read_json(base_path + USER_INFO_SUFFIX) or {} for key, field in USER_FIELDS.items(): diff --git a/api/model_metadata.py b/api/model_metadata.py index 0313fff..5c1ae6c 100644 --- a/api/model_metadata.py +++ b/api/model_metadata.py @@ -14,6 +14,8 @@ from .model_constants import ( CIVITAI_BACKUP_SUFFIXES, MEDIA_EXTENSIONS, MODEL_EXTENSIONS, PREVIEW_SUFFIXES, SIDECAR_SUFFIXES, ) +from .activity_log import ACTIVITY_DETAIL +from .metadata import get_metadata from .model_catalog import _resolve_paths_to_model_info_sync from .trash import move_to_trash, trash_failure from .utils import atomic_write_json @@ -117,6 +119,7 @@ async def api_delete_model(request): except Exception as e: return web.json_response({"status": "error", "message": trash_failure(e)}) deleted_files = [os.path.basename(path) for path in moved] + request[ACTIVITY_DETAIL] = {"files": deleted_files} # 前端期待的成功状态是 "success" 而不是 "ok" return web.json_response({ @@ -194,6 +197,7 @@ async def api_update_metadata(request): if os.path.exists(user_file) and read_json(user_file) is None: return web.json_response({"status": "error", "message": f"{os.path.basename(user_file)} is unreadable; nothing was changed."}) user_data = read_json(user_file) or {"format": 1} + before = get_metadata(file_path) # what was shown, for the activity log if 'custom_name' in data: user_data["custom_name"] = str(custom_name) if 'custom_notes' in data: @@ -231,6 +235,13 @@ async def api_update_metadata(request): elif os.path.exists(new_file_path) and new_file_path != file_path: return web.json_response({"status": "error", "message": "A file with the target physical name already exists."}) + sent = [key for key in ("custom_name", "custom_notes") if key in data] + sent += ["source_url"] if custom_source_url is not None else [] + fields = [{"field": key, "before": before.get(key) or "", "after": user_data[key]} + for key in sent if user_data[key] != (before.get(key) or "")] + if new_filename != filename: + fields.append({"field": "filename", "before": filename, "after": new_filename}) + request[ACTIVITY_DETAIL] = {"fields": fields} response_data = { "status": "success", "new_filename": new_filename, diff --git a/api/scan_report.py b/api/scan_report.py new file mode 100644 index 0000000..05b686f --- /dev/null +++ b/api/scan_report.py @@ -0,0 +1,166 @@ +"""What a scan did, model by model. + +The scraper appends one JSON line per model it changed to a report file (--report-file), +plus events: "civitai_down" when Civitai stopped answering, "done" with the count of models +left as they were. When a scan ends, ScanJob turns the lines of all its folders into one +result: kept as the last scan (GET /anomalous/last_scan, shown on the scan page) and recorded +in the activity log, with each model located by type, path index and relative path so the +browser can open it (GET /anomalous/scan_model). +""" + +import asyncio +import json +import os +import time + +from aiohttp import web +import folder_paths + +from .activity_log import MAX_CHANGES, add_entry, log_path +from .folder_types import get_active_folder_types +from .model_catalog import _model_info_for_path +from .path_utils import atomic_write_json, resolve_folder_subdir + +MAX_FILES = 2000 # models kept in the last scan's result +FILE_FIELDS = ("status", "reason", "name", "base", "cover", "error") + + +def last_scan_path(): + return os.path.join(os.path.dirname(log_path()), "last_scan.json") + + +def locate_root(directory): + """(type, path_idx) of the models folder that is `directory`, shown types first; None if none.""" + target = os.path.normcase(os.path.realpath(directory)) + active = get_active_folder_types() + types = active + [name for name in folder_paths.folder_names_and_paths if name not in active] + for folder_type in types: + try: + paths = folder_paths.get_folder_paths(folder_type) or [] + except Exception: + continue + for path_idx, path in enumerate(paths): + if os.path.normcase(os.path.realpath(path)) == target: + return folder_type, path_idx + return None + + +def read_report(path): + """(model entries, events) from a report file; both empty when there is none.""" + entries, events = [], [] + try: + with open(path, encoding="utf-8") as source: + for line in source: + try: + item = json.loads(line) + except ValueError: + continue + if isinstance(item, dict): + (events if "event" in item else entries).append(item) + except OSError: + pass + return entries, events + + +class ScanJob: + """One scan from start to end: what to call it, its options, and its folders' reports.""" + + def __init__(self, kind, data): + self.kind = kind # "all" | "picked" | "one" + self.options = {key: bool(data.get(key)) for key in ("offline_only", "retry_unmatched", "force_overwrite")} + self.started = time.time() + self.files = [] + self.unchanged = 0 + self.civitai_down = False + self.errors = [] + + def add_folder(self, report_file, base_dir, located=None): + """Reads (and removes) one scraper run's report. `located`: (type, path_idx) of base_dir.""" + located = located or locate_root(base_dir) or ("", 0) + entries, events = read_report(report_file) + try: + os.remove(report_file) + except OSError: + pass + for event in events: + if event.get("event") == "done": + self.unchanged += int(event.get("unchanged") or 0) + elif event.get("event") == "civitai_down": + self.civitai_down = True + for entry in entries: + path = entry.get("renamed_to") if entry.get("renamed_to") and entry.get("status") != "failed" else entry.get("file") + if not isinstance(path, str): + continue + item = {key: entry[key] for key in FILE_FIELDS if entry.get(key)} + item.update(type=located[0], path_idx=located[1], filename=os.path.basename(path), + rel=os.path.relpath(path, base_dir).replace(os.sep, "/")) + if entry.get("renamed_to") and item["status"] != "failed": + item["renamed_from"] = os.path.basename(entry.get("file") or "") + if entry.get("duplicate_of"): + item["duplicate"] = True + self.files.append(item) + + def add_error(self, message): + self.errors.append(str(message)[:300]) + + def result(self): + files = self.files + counts = { + "matched": sum(item["status"] == "matched" for item in files), + "inferred": sum(item["status"] == "inferred" for item in files), + "failed": sum(item["status"] == "failed" for item in files), + "covers": sum(item.get("cover") == "new" for item in files), + "renamed": sum(bool(item.get("renamed_from")) for item in files), + "unchanged": self.unchanged, + } + return { + "kind": self.kind, "options": self.options, "started": self.started, + "finished": time.time(), "counts": counts, "civitai_down": self.civitai_down, + "errors": self.errors[:10], "files": files[:MAX_FILES], + } + + def finish(self): + """Keeps the result as the last scan and records it in the activity log. Never raises: + the scan itself is done, the record of it is a convenience.""" + result = self.result() + try: + atomic_write_json(last_scan_path(), result) + except Exception as error: # noqa: BLE001 + print(f"[Anomalous Browser] Could not keep the scan result: {error}") + try: + target = result["files"][0]["filename"] if self.kind == "one" and result["files"] else "" + detail = {"scan": {key: result[key] for key in ("kind", "options", "counts", "civitai_down", "errors")}, + "files": result["files"][:MAX_CHANGES], "total": len(self.files), + "seconds": round(result["finished"] - result["started"])} + add_entry("file", "scan_done", target, detail) + except Exception as error: # noqa: BLE001 + print(f"[Anomalous Browser] Could not record the scan in the activity log: {error}") + return result + + +async def api_last_scan(request): + """GET /anomalous/last_scan - the last scan's result, or {} before the first one.""" + def read(): + try: + with open(last_scan_path(), encoding="utf-8") as source: + data = json.load(source) + return data if isinstance(data, dict) else {} + except (OSError, ValueError): + return {} + return web.json_response(await asyncio.to_thread(read)) + + +async def api_scan_model(request): + """GET /anomalous/scan_model?type&path_idx&rel - one model as the grid lists it, to open it.""" + rel = request.query.get("rel", "").replace("\\", "/").strip("/") + subfolder, filename = os.path.split(rel) + try: + path_idx = int(request.query.get("path_idx", 0)) + base_dir, target_dir = resolve_folder_subdir(request.query.get("type", ""), path_idx, "/" + subfolder) + except (ValueError, KeyError, TypeError): + return web.json_response({"error": "Invalid model location"}, status=400) + file_path = os.path.join(target_dir, filename) + if not filename or filename in (".", "..") or not os.path.isfile(file_path): + return web.json_response({"error": "Model not found"}, status=404) + model = await asyncio.to_thread(_model_info_for_path, request.query.get("type"), path_idx, base_dir, file_path) + return web.json_response({"model": model}) diff --git a/api/scan_summary.py b/api/scan_summary.py index a055bc1..5986623 100644 --- a/api/scan_summary.py +++ b/api/scan_summary.py @@ -1,4 +1,6 @@ -"""What the scan page shows before a scan: how many models are matched on Civitai, unmatched, or new.""" +"""What the scan page shows before a scan: how many models are matched on Civitai, unmatched or +new, and which ones the last two are (located like the scan report's models, so the page can +open them).""" import asyncio import os @@ -6,18 +8,25 @@ import os from aiohttp import web from .folder_types import get_active_scan_paths +from .scan_report import locate_root try: - from ..model_identity import is_unmatched, sidecar_info + from ..model_identity import is_unmatched, sidecar_info, unmatched_reason except ImportError: # loaded outside the package (tests) - from model_identity import is_unmatched, sidecar_info + from model_identity import is_unmatched, sidecar_info, unmatched_reason + +MAX_LISTED = 2000 # models listed per kind; the counts stay exact def _summarize(): - """Counts the files the scanner handles (.safetensors in the active scan folders), each once.""" + """Counts the files the scanner handles (.safetensors in the active scan folders), each once. + `pending`: unmatched models Civitai was never asked about (offline, or unreachable then), + which the next online scan looks up.""" seen = set() - total = matched = unmatched = 0 + total = matched = 0 + unmatched, new = [], [] for base_dir in get_active_scan_paths(): + located = locate_root(base_dir) or ("", 0) for root, _dirs, files in os.walk(base_dir): for filename in files: if not filename.endswith(".safetensors"): @@ -29,15 +38,23 @@ def _summarize(): seen.add(real) total += 1 info = sidecar_info(os.path.splitext(path)[0]) - if info is None: - continue - if is_unmatched(info): - unmatched += 1 - else: + if info is not None and not is_unmatched(info): matched += 1 - return {"total": total, "matched": matched, "unmatched": unmatched, "new": total - matched - unmatched} + continue + item = {"type": located[0], "path_idx": located[1], "filename": filename, + "rel": os.path.relpath(path, base_dir).replace(os.sep, "/")} + if info is None: + new.append(item) + else: + item.update(reason=unmatched_reason(info), base=info.get("baseModel") or "") + unmatched.append(item) + return { + "total": total, "matched": matched, "unmatched": len(unmatched), "new": len(new), + "pending": sum(item["reason"] in ("offline", "network") for item in unmatched), + "unmatched_models": unmatched[:MAX_LISTED], "new_models": new[:MAX_LISTED], + } async def api_scan_summary(request): - """GET /anomalous/scan_summary - {total, matched, unmatched, new}.""" + """GET /anomalous/scan_summary - counts, and the unmatched and new models.""" return web.json_response(await asyncio.to_thread(_summarize)) diff --git a/api/scanner.py b/api/scanner.py index 9a19356..cdd0dfc 100644 --- a/api/scanner.py +++ b/api/scanner.py @@ -13,14 +13,17 @@ import uuid from aiohttp import web import folder_paths import struct -from .folder_types import get_active_folder_types, get_active_scan_paths +from .folder_types import get_active_scan_paths from .path_utils import resolve_folder_subdir +from .scan_report import ScanJob, locate_root try: from ..model_policies import is_physical_rename_protected except ImportError: from model_policies import is_physical_rename_protected +PLUGIN_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +SCRAPER_PATH = os.path.join(PLUGIN_DIR, "scraper.py") SCAN_SESSION_ID = uuid.uuid4().hex SCAN_MARKER_LOCK = threading.Lock() ACTIVE_SCAN_MARKERS = set() @@ -278,82 +281,75 @@ async def api_scan_status(request): return web.json_response(data) async def api_scan_folder(request): - """Launches the scraper in the background for a specific directory.""" + """Launches the scraper in the background for one folder (a model card's scan).""" folder_type = request.query.get('type', 'checkpoints') subfolder = request.query.get('subfolder', '/') try: path_idx = int(request.query.get('path_idx', 0)) - except: + except (TypeError, ValueError): path_idx = 0 - + try: base_dir, target_dir = resolve_folder_subdir(folder_type, path_idx, subfolder) except (ValueError, KeyError): return web.json_response({"status": "error", "message": "Invalid folder type"}) - if not os.path.exists(target_dir): return web.json_response({"status": "error", "message": "Directory does not exist"}) - - # Get the scraper path relative to this script - plugin_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - scraper_path = os.path.join(plugin_dir, "scraper.py") - - if not os.path.exists(scraper_path): + if not os.path.exists(SCRAPER_PATH): return web.json_response({"status": "error", "message": "scraper.py not found in extension directory"}) - + print(f"[Anomalous Browser] Starting background scan for: {target_dir}") - try: data = await request.json() except ValueError: data = None if not isinstance(data, dict): return web.json_response({"status": "error", "message": "Invalid request body"}, status=400) - + flags = _scraper_flags(data, False, not is_physical_rename_protected( folder_type=folder_type, folder_path=target_dir, )) - target_files_list = data.get("target_files", []) if not target_files_list: target_files_str = request.query.get('target_files', '') target_files_list = [f.strip() for f in target_files_str.split(',')] if target_files_str else [] - + claimed = False try: marker_file = os.path.join(target_dir, '.scan_in_progress') progress_file = os.path.join(target_dir, '.scan_progress.json') targets_file = os.path.join(target_dir, '.scan_targets.json') - claimed, recovered = _claim_scan_marker( - marker_file, - "folder", - (progress_file, targets_file), - ) + report_file = os.path.join(target_dir, '.scan_report.jsonl') + artifacts = (progress_file, targets_file, report_file) + claimed, recovered = _claim_scan_marker(marker_file, "folder", artifacts) if not claimed: return web.json_response({"status": "error", "message": "Scan already in progress"}, status=409) if target_files_list: with open(targets_file, 'w', encoding='utf-8') as f: - __import__('json').dump(target_files_list, f) + json.dump(target_files_list, f) + job = ScanJob("one" if len(target_files_list) == 1 else "picked", data) def run_bg(): try: - cmd = [sys.executable, scraper_path, target_dir, "--folder-type", folder_type, *flags] - cmd.extend(["--progress-file", progress_file]) - process = subprocess.Popen(cmd, cwd=plugin_dir) + cmd = [sys.executable, SCRAPER_PATH, target_dir, "--folder-type", folder_type, *flags] + cmd.extend(["--progress-file", progress_file, "--report-file", report_file]) + process = subprocess.Popen(cmd, cwd=PLUGIN_DIR) _update_scan_marker(marker_file, worker_pid=process.pid) return_code = process.wait() + job.add_folder(report_file, base_dir, (folder_type, path_idx)) if return_code != 0: + job.add_error(f"Scanner exited with code {return_code}") result_file = os.path.join(target_dir, '.scan_result.json') with open(result_file, 'w', encoding='utf-8') as f: json.dump({"success": 0, "fail": 1, "error": f"Scanner exited with code {return_code}"}, f) finally: _clear_folder_caches() - _release_scan_marker(marker_file, (progress_file, targets_file)) - + job.finish() # before the marker goes: whoever sees the scan end finds its result + _release_scan_marker(marker_file, artifacts) + threading.Thread(target=run_bg, daemon=True).start() - return web.json_response({ "status": "ok", "message": "Scan started in background. Check console for details.", @@ -364,85 +360,122 @@ async def api_scan_folder(request): _release_scan_marker(marker_file, (progress_file, targets_file)) return web.json_response({"status": "error", "message": str(e)}) + +def _global_scan_plan(data): + """The folders a scan-page scan runs through: [(scan dir, (type, path_idx) or None, base dir, + file names or None, folder type for the rename policy)]. `data["targets"]`, when given, is + the picked models: [{type, path_idx, subfolder, files}]; otherwise every active folder.""" + targets = data.get("targets") + if not targets: + plan = [] + for base_dir in get_active_scan_paths(): + if os.path.exists(base_dir): + plan.append((base_dir, locate_root(base_dir), base_dir, None, _protected_type_for_path(base_dir))) + return plan + if not isinstance(targets, list): + raise ValueError("targets must be a list") + plan = [] + for target in targets: + files = target.get("files") if isinstance(target, dict) else None + if not isinstance(files, list) or not all(isinstance(name, str) for name in files): + raise ValueError("each target needs a list of file names") + path_idx = int(target.get("path_idx", 0)) + base_dir, target_dir = resolve_folder_subdir(target.get("type"), path_idx, target.get("subfolder") or "/") + if files and os.path.isdir(target_dir): + plan.append((target_dir, (target["type"], path_idx), base_dir, files, target["type"])) + return plan + + async def api_scan_all(request): - plugin_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - scraper_path = os.path.join(plugin_dir, "scraper.py") - marker_file = os.path.join(plugin_dir, '.global_scan_in_progress') - progress_file = os.path.join(plugin_dir, '.global_scan_progress.json') - + """Scans every active models folder, or the picked models (`targets`), one folder after another.""" + marker_file = os.path.join(PLUGIN_DIR, '.global_scan_in_progress') + progress_file = os.path.join(PLUGIN_DIR, '.global_scan_progress.json') + targets_file = os.path.join(PLUGIN_DIR, '.global_scan_targets.json') + report_file = os.path.join(PLUGIN_DIR, '.global_scan_report.jsonl') + artifacts = (progress_file, targets_file, report_file) + try: data = await request.json() except ValueError: data = None if not isinstance(data, dict): return web.json_response({"status": "error", "message": "Invalid request body"}, status=400) - + try: + plan = _global_scan_plan(data) if data.get("targets") else None + except (ValueError, KeyError, TypeError) as e: + return web.json_response({"status": "error", "message": f"Invalid targets: {e}"}, status=400) + picked = sum(len(files) for _dir, _loc, _base, files, _type in plan or ()) + job = ScanJob("all" if plan is None else "one" if picked == 1 else "picked", data) + claimed = False try: - claimed, recovered = _claim_scan_marker(marker_file, "global", (progress_file,)) + claimed, recovered = _claim_scan_marker(marker_file, "global", artifacts) if not claimed: return web.json_response({"status": "error", "message": "Global scan already in progress"}, status=409) def run_global_bg(): try: - paths_to_scan = get_active_scan_paths() - paths_to_scan = [path for path in paths_to_scan if os.path.exists(path)] - _update_scan_state( - marker_file, - phase="preparing", - folder_total=len(paths_to_scan), - folder_current=0, - folder="", - ) - for folder_index, base_dir in enumerate(paths_to_scan, 1): - if not os.path.exists(base_dir): continue + folders = plan if plan is not None else _global_scan_plan(data) + _update_scan_state(marker_file, phase="preparing", folder_total=len(folders), folder_current=0, folder="") + for folder_index, (scan_dir, located, base_dir, files, policy_type) in enumerate(folders, 1): try: - print(f"[Anomalous Browser] Global scan processing: {base_dir}") + print(f"[Anomalous Browser] Global scan processing: {scan_dir}") _remove_file(progress_file) _update_scan_state( marker_file, phase="enumerating", - folder_total=len(paths_to_scan), + folder_total=len(folders), folder_current=folder_index, - folder=os.path.basename(os.path.normpath(base_dir)) or base_dir, + folder=os.path.basename(os.path.normpath(scan_dir)) or scan_dir, error="", ) - cmd = [sys.executable, scraper_path, base_dir] - protected_type = _protected_type_for_path(base_dir) - if protected_type: - cmd.extend(["--folder-type", protected_type]) + cmd = [sys.executable, SCRAPER_PATH, scan_dir] + if policy_type: + cmd.extend(["--folder-type", policy_type]) cmd.extend(_scraper_flags(data, True, not is_physical_rename_protected( - folder_type=protected_type, - folder_path=base_dir, + folder_type=policy_type, + folder_path=scan_dir, ))) - cmd.extend(["--progress-file", progress_file]) - process = subprocess.Popen(cmd, cwd=plugin_dir) + if files is not None: + with open(targets_file, 'w', encoding='utf-8') as f: + json.dump(files, f) + cmd.extend(["--targets-file", targets_file]) + if job.civitai_down: # an earlier folder found Civitai unreachable + cmd.append("--civitai-down") + cmd.extend(["--progress-file", progress_file, "--report-file", report_file]) + process = subprocess.Popen(cmd, cwd=PLUGIN_DIR) _update_scan_marker(marker_file, worker_pid=process.pid) return_code = process.wait() _update_scan_marker(marker_file, worker_pid=0) + job.add_folder(report_file, base_dir, located) if return_code != 0: - message = f"Scanner exited with code {return_code}: {base_dir}" + message = f"Scanner exited with code {return_code}: {scan_dir}" _update_scan_state(marker_file, error=message) + job.add_error(message) print(f"[Anomalous Browser] {message}") except Exception as e: _update_scan_state(marker_file, error=str(e)) - print(f"[Anomalous Browser] Global scan error on {base_dir}: {e}") + job.add_error(f"{scan_dir}: {e}") + print(f"[Anomalous Browser] Global scan error on {scan_dir}: {e}") finally: _clear_folder_caches() - _release_scan_marker(marker_file, (progress_file,)) - + job.finish() # before the marker goes: whoever sees the scan end finds its result + _release_scan_marker(marker_file, artifacts) + threading.Thread(target=run_global_bg, daemon=True).start() return web.json_response({"status": "ok", "message": "Global scan started", "recovered": recovered}) except Exception as e: if claimed: - _release_scan_marker(marker_file, (progress_file,)) + _release_scan_marker(marker_file, artifacts) return web.json_response({"status": "error", "message": str(e)}) + async def api_global_scan_status(request): - plugin_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - marker_file = os.path.join(plugin_dir, '.global_scan_in_progress') - progress_file = os.path.join(plugin_dir, '.global_scan_progress.json') + marker_file = os.path.join(PLUGIN_DIR, '.global_scan_in_progress') + progress_file = os.path.join(PLUGIN_DIR, '.global_scan_progress.json') return web.json_response(_scan_status_payload(marker_file, progress_file, (progress_file,))) + + GLOBAL_SCAN_STATE = { "scanning": False, "total": 0, @@ -465,7 +498,7 @@ async def api_scan_missing_models(request): sys.path.insert(0, plugin_dir) try: - from scraper import calculate_sha256, fetch_civitai_info, infer_base_model_from_header, computed_file_identity + from scraper import CivitaiUnreachable, calculate_sha256, computed_file_identity, fetch_civitai_info, local_record except ImportError: return web.json_response({"status": "error", "message": "Failed to load scraper module"}) @@ -510,27 +543,13 @@ async def api_scan_missing_models(request): try: file_hash = calculate_sha256(file_path) - civitai_data = fetch_civitai_info(file_hash) + try: + civitai_data, reason = fetch_civitai_info(file_hash), "not_found" + except CivitaiUnreachable: + civitai_data, reason = None, "network" + if not civitai_data: # what the file itself tells + civitai_data = local_record(file_path, os.path.dirname(file_path), file_hash, reason) - # Fallback 3: Local Offline Inference - if not civitai_data: - inferred_base = infer_base_model_from_header(file_path) - if inferred_base == 'Unknown': - inferred_base = "" - - civitai_data = { - "id": -1, - "modelId": -1, - "name": os.path.splitext(filename)[0], - "baseModel": inferred_base, - "description": "

Automatically inferred by Anomalous Local Engine.

", - "model": { - "name": os.path.splitext(filename)[0], - "type": "LORA" if "lora" in file_path.lower() else "Checkpoint" - }, - "files": [{"hashes": {"SHA256": file_hash}}] - } - civitai_data["anomalous_file_identity"] = computed_file_identity(file_path, file_hash) # Save info file diff --git a/civitai_client.py b/civitai_client.py new file mode 100644 index 0000000..76c2517 --- /dev/null +++ b/civitai_client.py @@ -0,0 +1,107 @@ +"""Civitai over the network, for the scanner: a model version by file hash, a model page's +description, and preview media. The API key comes from the plugin's config.json. + +fetch_civitai_info tells the two failures apart: None means Civitai answered that it does not +know the file; CivitaiUnreachable means it could not be asked (no connection, timeouts, errors). +""" + +import json +import os +import time +import urllib.error +import urllib.request +from typing import Dict, Optional + +USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + + +class CivitaiUnreachable(Exception): + """Civitai did not answer (network, timeout or server error) after the retries.""" + + +def _load_api_key(): + plugin_dir = os.path.dirname(os.path.abspath(__file__)) + for config_path in (os.path.join(plugin_dir, "api", "config.json"), os.path.join(plugin_dir, "config.json")): + if not os.path.exists(config_path): + continue + try: + with open(config_path, 'r', encoding='utf-8') as f: + key = json.load(f).get("CIVITAI_API_KEY", "") + if isinstance(key, str) and key.strip(): + return key.strip() + except Exception as e: + print(f"[-] 读取 config.json 失败: {e}") + return None + + +CIVITAI_API_KEY = _load_api_key() + + +def _headers(): + headers = {"User-Agent": USER_AGENT} + if CIVITAI_API_KEY: + headers["Authorization"] = f"Bearer {CIVITAI_API_KEY}" + return headers + + +def fetch_civitai_info(file_hash: str, max_retries: int = 3) -> Optional[Dict]: + """Civitai's record of the model version with this file hash; None when Civitai does not + know the file (404). Raises CivitaiUnreachable when it could not be asked.""" + url = f"https://civitai.com/api/v1/model-versions/by-hash/{file_hash}" + for attempt in range(max_retries): + try: + with urllib.request.urlopen(urllib.request.Request(url, headers=_headers()), timeout=15) as response: + return json.loads(response.read().decode('utf-8')) + except urllib.error.HTTPError as e: + if e.code == 404: + print(f"\033[93m[Skip] 模型 Hash {file_hash} 未在 Civitai 找到 (404)。\033[0m") + return None + print(f"[-] 请求异常，状态码: {e.code} (尝试 {attempt+1}/{max_retries})") + except urllib.error.URLError as e: + print(f"[-] 网络请求超时或异常: {e.reason} (尝试 {attempt+1}/{max_retries})") + except Exception as e: + print(f"[-] 未知异常: {e} (尝试 {attempt+1}/{max_retries})") + if attempt < max_retries - 1: + time.sleep(2) + print(f"\033[93m[Skip] 模型 Hash {file_hash} 网络重试失败。\033[0m") + raise CivitaiUnreachable(file_hash) + + +def fetch_model_description(model_id) -> str: + """The model page's description (the version record only has the version's); '' when unavailable.""" + try: + request = urllib.request.Request(f"https://civitai.com/api/v1/models/{model_id}", headers=_headers()) + with urllib.request.urlopen(request, timeout=10) as response: + return json.loads(response.read().decode('utf-8')).get("description") or "" + except Exception as e: + print(f"[-] 获取模型主页详细说明失败: {e}") + return "" + + +def download_media(url: str, base_path: str, max_retries: int = 3): + """Downloads an image or video to base_path + the extension its type calls for; the path, or None.""" + for attempt in range(max_retries): + try: + with urllib.request.urlopen(urllib.request.Request(url, headers=_headers()), timeout=15) as response: + content_type = response.headers.get("Content-Type", "").lower() + ext = ".png" + if "video/mp4" in content_type: ext = ".mp4" + elif "video/webm" in content_type: ext = ".webm" + elif "image/jpeg" in content_type: ext = ".jpg" + elif "image/webp" in content_type: ext = ".webp" + elif url.endswith(".mp4"): ext = ".mp4" + final_path = base_path + ext + with open(final_path, 'wb') as f: + while True: + chunk = response.read(8192) + if not chunk: + break + f.write(chunk) + return final_path + except urllib.error.HTTPError as e: + print(f"[-] 媒体下载失败，状态码: {e.code} (尝试 {attempt+1}/{max_retries})") + except urllib.error.URLError as e: + print(f"[-] 媒体下载网络异常: {e.reason} (尝试 {attempt+1}/{max_retries})") + except Exception: + pass + return None diff --git a/docs/architecture/backend.md b/docs/architecture/backend.md index 429bfaa..237998a 100644 --- a/docs/architecture/backend.md +++ b/docs/architecture/backend.md @@ -128,6 +128,22 @@ file count, current index, and filename. Status responses merge worker progress with parent-owned folder progress. The frontend can reconstruct this state by polling after its UI has been reopened. +`scraper.py --report-file` appends one JSON line per model it changed (`matched`, +`inferred` with its reason, `failed` with the error, renames and covers) and events +(`civitai_down`, `done` with the unchanged count). One model's error is reported and +the scan goes on. `ScanJob` in `api/scan_report.py` reads each folder's report when its +worker exits and, before the scan's marker is released, writes the scan's result to +`user/anomalous/last_scan.json` and the activity log, so whoever sees the scan end can +read its result. The scan page's picked or listed models go to `/anomalous/scan_all` as +`targets` (`--targets-file` per folder), so one scan has one result. + +An unmatched model's `.info` (`id` -1) keeps why: `anomalous_unmatched_reason` is +`not_found` (Civitai answered 404), `network` (no answer after the retries) or +`offline`. Online scans look up `network` and `offline` models again; `not_found` ones +wait for "look up again". After `CIVITAI_DOWN_AFTER` models in a row without an answer +a scan stops asking (`--civitai-down` carries this to the next folders) and infers the +rest from the files, instead of waiting for every timeout. + ## Metadata and cache behavior The output gallery keeps one ordered directory snapshot for at most ten seconds diff --git a/model_identity.py b/model_identity.py index 290d4c9..9ce7fa7 100644 --- a/model_identity.py +++ b/model_identity.py @@ -48,6 +48,17 @@ def is_unmatched(info): return isinstance(info, dict) and info.get("id") == -1 +# Why an unmatched model has no Civitai record: Civitai does not know the file; Civitai could +# not be reached; the scan was offline. Scans before this was recorded leave it empty. +UNMATCHED_REASONS = ("not_found", "network", "offline") + + +def unmatched_reason(info): + """One of UNMATCHED_REASONS for an unmatched model's info, else "".""" + reason = info.get("anomalous_unmatched_reason") if is_unmatched(info) else "" + return reason if reason in UNMATCHED_REASONS else "" + + def normalise_sha256(value): if not isinstance(value, str) or not re.fullmatch(r"[0-9a-fA-F]{64}", value.strip()): return "" diff --git a/scraper.py b/scraper.py index f53be97..2eeca4d 100644 --- a/scraper.py +++ b/scraper.py @@ -16,14 +16,12 @@ import json import hashlib import re import urllib.request -import urllib.error -import urllib.parse import argparse import shutil -from typing import Dict, Optional from model_policies import is_physical_rename_protected from recycle_bin import move_to_trash -from model_identity import USER_INFO_SUFFIX, computed_file_identity, is_unmatched, scan_info_path, sidecar_file_hash, sidecar_info as read_local_info +from model_identity import USER_INFO_SUFFIX, computed_file_identity, is_unmatched, scan_info_path, sidecar_file_hash, sidecar_info as read_local_info, unmatched_reason +from civitai_client import CIVITAI_API_KEY, CivitaiUnreachable, download_media, fetch_civitai_info, fetch_model_description # Fixed tuples avoid rebuilding long extension lists for every scanned model. @@ -89,30 +87,6 @@ def count_scan_files(target_folder, target_files_basenames, unmatched_only=False ) return total -# ============================================================================== -# CIVITAI API 配置读取 -# 请在插件目录 (Anomalous_Model_Browser) 下新建 config.json 文件： -# { "CIVITAI_API_KEY": "你的KEY" } -# ============================================================================== -CIVITAI_API_KEY = None -plugin_dir = os.path.dirname(os.path.abspath(__file__)) -config_paths = [os.path.join(plugin_dir, "api", "config.json"), os.path.join(plugin_dir, "config.json")] -for config_path in config_paths: - if not os.path.exists(config_path): - continue - try: - with open(config_path, 'r', encoding='utf-8') as f: - cfg = json.load(f) - configured_key = cfg.get("CIVITAI_API_KEY", "") - if isinstance(configured_key, str) and configured_key.strip(): - CIVITAI_API_KEY = configured_key.strip() - break - except Exception as e: - print(f"[-] 读取 config.json 失败: {e}") - -if not CIVITAI_API_KEY: - print("[!] 未配置 Civitai API Key。部分限制级模型或将无法获取图片。") - def calculate_sha256(file_path: str) -> str: """计算文件的 SHA256 哈希值 (用于 Civitai 匹配)""" sha256_hash = hashlib.sha256() @@ -167,71 +141,205 @@ def sanitize_filename(name: str) -> str: name = re.sub(r'[\\/*?:"<>|#]', "", name) return name.strip(' .') -def fetch_civitai_info(file_hash: str, max_retries: int = 3) -> Optional[Dict]: - """向 Civitai API 获取模型信息，支持重试机制""" - url = f"https://civitai.com/api/v1/model-versions/by-hash/{file_hash}" - headers = { - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - } - if CIVITAI_API_KEY: - headers["Authorization"] = f"Bearer {CIVITAI_API_KEY}" - - for attempt in range(max_retries): - try: - req = urllib.request.Request(url, headers=headers) - with urllib.request.urlopen(req, timeout=15) as response: - return json.loads(response.read().decode('utf-8')) - except urllib.error.HTTPError as e: - if e.code == 404: - print(f"\033[93m[Skip] 模型 Hash {file_hash} 未在 Civitai 找到 (404)，已跳过。\033[0m") - return None - print(f"[-] 请求异常，状态码: {e.code} (尝试 {attempt+1}/{max_retries})") - except urllib.error.URLError as e: - print(f"[-] 网络请求超时或异常: {e.reason} (尝试 {attempt+1}/{max_retries})") - except Exception as e: - print(f"[-] 未知异常: {e} (尝试 {attempt+1}/{max_retries})") - - if attempt < max_retries - 1: - time.sleep(2) +CIVITAI_DOWN_AFTER = 2 # models in a row Civitai did not answer for before the rest of the run stays offline - print(f"\033[93m[Skip] 模型 Hash {file_hash} 网络重试失败，已跳过该文件。\033[0m") - return None -def download_media(url: str, base_path: str, max_retries: int = 3): - """下载图片或视频并自动识别扩展名""" - headers = { - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" +def write_report(args, **entry): + """One line of the scan report (--report-file): a model's outcome, or an event of the run.""" + if not args.report_file: + return + try: + with open(args.report_file, 'a', encoding='utf-8') as f: + f.write(json.dumps(entry, ensure_ascii=False) + "\n") + except OSError as e: + print(f"[-] 扫描记录写入失败: {e}") + + +def local_record(file_path, root, file_hash, reason): + """What the file alone tells (id -1), with why Civitai's record is missing.""" + inferred_base = infer_base_model_from_header(file_path) + stem = os.path.splitext(os.path.basename(file_path))[0] + print(f"[*] 使用本地哈希重建基础元数据 ({os.path.basename(file_path)})") + return { + "id": -1, + "modelId": -1, + "name": stem, + "baseModel": "" if inferred_base == 'Unknown' else inferred_base, + "description": "

Automatically inferred by Anomalous Local Engine.

", + "model": {"name": stem, "type": "LORA" if "lora" in root.lower() else "Checkpoint"}, + "files": [{"hashes": {"SHA256": file_hash}}], + "anomalous_unmatched_reason": reason, } - if CIVITAI_API_KEY: - headers["Authorization"] = f"Bearer {CIVITAI_API_KEY}" - - for attempt in range(max_retries): - try: - req = urllib.request.Request(url, headers=headers) - with urllib.request.urlopen(req, timeout=15) as response: - content_type = response.headers.get("Content-Type", "").lower() - ext = ".png" # default - if "video/mp4" in content_type: ext = ".mp4" - elif "video/webm" in content_type: ext = ".webm" - elif "image/jpeg" in content_type: ext = ".jpg" - elif "image/webp" in content_type: ext = ".webp" - elif url.endswith(".mp4"): ext = ".mp4" - - final_path = base_path + ext - with open(final_path, 'wb') as f: - while True: - chunk = response.read(8192) - if not chunk: - break - f.write(chunk) - return final_path - except urllib.error.HTTPError as e: - print(f"[-] 媒体下载失败，状态码: {e.code} (尝试 {attempt+1}/{max_retries})") - except urllib.error.URLError as e: - print(f"[-] 媒体下载网络异常: {e.reason} (尝试 {attempt+1}/{max_retries})") - except Exception as e: - pass - return None + + +def ask_civitai(args, state, file_hash): + """(Civitai's record or None, why it is missing: "not_found" | "network" | "offline" | "").""" + if args.offline_only: + return None, "offline" + if state["civitai_down"]: + return None, "network" + try: + data = fetch_civitai_info(file_hash) + except CivitaiUnreachable: + state["network_failures"] += 1 + if state["network_failures"] >= CIVITAI_DOWN_AFTER: + state["civitai_down"] = True + print("\033[93m[!] 连续几个模型都连不上 Civitai，本次其余模型改为从文件推断，之后可以“重新联网查找”。\033[0m") + write_report(args, event="civitai_down") + return None, "network" + state["network_failures"] = 0 + return data, ("" if data else "not_found") + + +def scan_one(args, state, rename_log, root, filename, previous): + """Scans one model; returns its report entry, or None when nothing about it changed.""" + file_path = os.path.join(root, filename) + old_base = os.path.splitext(file_path)[0] + online = not args.offline_only and not state["civitai_down"] + + # A .civitai.info holding only edits from older versions is not a scan result. + info_exists = scan_info_path(old_base) is not None + # Models Civitai was never asked about (an offline scan, no connection then) are looked up + # by every online scan; models Civitai did not know wait for "look up again". + unasked = is_unmatched(previous) and unmatched_reason(previous) in ("offline", "network") + if args.force_overwrite or args.retry_unmatched or (online and unasked): + info_exists = False + + preview_exists = args.skip_media or any(os.path.exists(old_base + ext) for ext in COVER_SUFFIXES) + needs_rename = bool(args.virtual_rename or (not args.skip_rename and args.physical_rename)) + if info_exists and preview_exists and not needs_rename: + print(f"[*] 已跳过 (信息满足要求): {filename}") + return None + + print(f"\n---> 处理文件: {filename} (位于 {root})") + civitai_data = None + file_hash = None + reason = "" + if info_exists: # a rename, or a missing cover fetched from the saved record: no new lookup + civitai_data = read_local_info(old_base) + if civitai_data: + print(f"[*] 使用本地已有的信息") + + if not civitai_data: + if is_unmatched(previous): # the hash from last time, if the file is unchanged + file_hash = sidecar_file_hash(previous, file_path, (previous.get("files") or [{}])[0])[0] + file_hash = file_hash or calculate_sha256(file_path) + civitai_data, reason = ask_civitai(args, state, file_hash) + + # What was read from disk is not fetched again: only fresh Civitai data needs the model + # page's description and the cover. + fresh = civitai_data is not None and file_hash is not None + if not civitai_data: + if args.skip_local_metadata: + print(f"[*] Civitai 获取失败，且禁用了本地元数据解析。跳过 {filename}") + return {"status": "failed", "error": "no_local_metadata", "reason": reason} + civitai_data = local_record(file_path, root, file_hash, reason) + entry = {"status": "matched" if fresh else "inferred" if reason else "unchanged"} + if reason: + entry["reason"] = reason + + model_id = civitai_data.get("modelId") + if fresh and model_id and model_id != -1 and not state["civitai_down"]: + description = fetch_model_description(model_id) + if description: + civitai_data["description"] = description + if not isinstance(civitai_data.get("model"), dict): + civitai_data["model"] = {} + civitai_data["model"]["description"] = description + + model_name = sanitize_filename(civitai_data.get("model", {}).get("name", "UnknownModel")) + version_name = sanitize_filename(civitai_data.get("name", "UnknownVersion")) + if fresh: + entry["name"] = f"{civitai_data.get('model', {}).get('name', '')} · {civitai_data.get('name', '')}".strip(" ·") + entry["base"] = civitai_data.get("baseModel") or "" + new_filename = f"{model_name}_{version_name}.safetensors" + new_file_path = os.path.join(root, new_filename) + new_base = os.path.splitext(new_file_path)[0] + + info_data = civitai_data + if file_hash: + info_data["anomalous_file_identity"] = computed_file_identity(file_path, file_hash) + # Civitai's name for the card; a name set in the model editor lives in + # .anomalous.json and is shown instead. + if args.virtual_rename: + info_data["anomalous_custom_name"] = f"{model_name}_{version_name}" + if not args.dry_run: + with open(old_base + ".info", 'w', encoding='utf-8') as f: + json.dump(info_data, f, ensure_ascii=True, indent=4) + print(f"[+] 写入纯净版 Civitai 描述 -> .info") + else: + print(f"[Dry-Run] 拟生成标准描述信息 -> .info") + + media_url = None + if not args.skip_media: + media_url = next((img.get("url") for img in civitai_data.get("images") or [] if img.get("url")), None) + civitai_covers, user_covers = cover_owners(old_base) + # Download when the data is fresh, or when the model has no cover at all. + if media_url and not state["civitai_down"] and (fresh or not (civitai_covers or user_covers)): + if not args.dry_run: + print(f"[*] 正在下载预览媒体...") + saved_path = download_media(media_url, old_base + ".civitai_bak") + if saved_path: + print(f"[+] 媒体下载成功 -> {os.path.basename(saved_path)}") + entry["cover"] = "kept" if user_covers else "new" + if user_covers: + # The user's cover always stays; Civitai's image is kept as the backup. + print(f"[*] 保留你自己的封面，Civitai 封面存为备份") + else: + ext = os.path.splitext(saved_path)[1] + preview_path = old_base + (ext if ext.startswith('.preview.') else f".preview{ext}") + # Earlier copies of Civitai's image under another name give way to the new one. + older = [c for c in civitai_covers if c != preview_path] + try: + if older: + move_to_trash(*older) + except Exception as e: + print(f"[-] 旧的 Civitai 封面没有移走: {e}") + shutil.copy2(saved_path, preview_path) + else: + print(f"[Dry-Run] 拟下载预览媒体...") + + if args.skip_rename or not args.physical_rename: + print(f"[*] 物理重命名已跳过。仅保存 .info 及其可能包含的虚拟重命名。") + elif file_path != new_file_path and new_filename != filename: + if os.path.exists(new_file_path): + print(f"[*] 目标文件名已存在，正在验证内容是否完全相同: {filename}") + try: + files_identical = os.path.getsize(file_path) == os.path.getsize(new_file_path) + if files_identical: + files_identical = calculate_sha256(file_path) == calculate_sha256(new_file_path) + except OSError: + files_identical = False + + if not files_identical: + print(f"\033[91m[-] 目标名称冲突但文件内容不同，已保留两个模型并跳过重命名: {filename}\033[0m") + return {**entry, "status": "failed", "error": "name_conflict", "renamed_to": new_file_path} + if not args.dry_run: + try: + print(f"[*] Hash 一致，把已确认的重复副本移到回收站: {filename}") + move_to_trash(file_path, *[old_base + ext for ext in SIDECAR_SUFFIXES]) + entry["duplicate_of"] = new_file_path + except Exception as e: + print(f"[-] 重复副本没有移走，两份都保留 (可能文件被占用或这个盘没有回收站): {e}") + return {**entry, "status": "failed", "error": "duplicate_kept"} + else: + print(f"[Dry-Run] Hash 一致，拟删除重复副本及其附属文件: {filename}") + elif not args.dry_run: + os.rename(file_path, new_file_path) + rename_log[file_path] = new_file_path + for ext in SIDECAR_SUFFIXES: + if os.path.exists(old_base + ext): + os.replace(old_base + ext, new_base + ext) + print(f"[+] 物理重命名完成: {filename} ==> {new_filename}") + entry["renamed_to"] = new_file_path + else: + print(f"[Dry-Run] 拟物理重命名文件: {filename} ==> {new_filename}") + print(f"[Dry-Run] 拟连带重命名附属文件 (.info / .png 等)") + else: + print("[*] 文件名已符合规范，无需重命名。") + changed = entry["status"] != "unchanged" or set(entry) & {"cover", "renamed_to", "duplicate_of"} + return entry if changed else None + def main(): parser = argparse.ArgumentParser(description="ComfyUI 模型 Civitai 嗅探与重命名工具") @@ -249,6 +357,9 @@ def main(): parser.add_argument("--target-files", type=str, default="", help="仅扫描逗号分隔的具体文件(相对路径)") parser.add_argument("--folder-type", default="", help="由 ComfyUI 传入的模型目录类型，用于执行安全策略") parser.add_argument("--progress-file", default="", help=argparse.SUPPRESS) + parser.add_argument("--targets-file", default="", help=argparse.SUPPRESS) # JSON list of file names + parser.add_argument("--report-file", default="", help=argparse.SUPPRESS) # one JSON line per changed model + parser.add_argument("--civitai-down", action="store_true", help=argparse.SUPPRESS) # an earlier folder could not reach it args = parser.parse_args() if args.offline_only: # Offline means no network at all: previews and model pages come from Civitai. @@ -303,22 +414,21 @@ def main(): # 模式二：正常嗅探与重命名 # ========================== rename_log = {} - success_count = 0 - fail_count = 0 if os.path.exists(backup_log_path): with open(backup_log_path, 'r', encoding='utf-8') as f: rename_log = json.load(f) target_files_basenames = [] - target_file_path = os.path.join(target_folder, '.scan_targets.json') - if os.path.exists(target_file_path): + for targets_path in (os.path.join(target_folder, '.scan_targets.json'), args.targets_file): + if not targets_path or not os.path.exists(targets_path): + continue try: - with open(target_file_path, 'r', encoding='utf-8') as f: - target_files_basenames = [os.path.basename(t.strip()) for t in __import__('json').load(f)] - os.remove(target_file_path) - except: + with open(targets_path, 'r', encoding='utf-8') as f: + target_files_basenames += [os.path.basename(t.strip()) for t in json.load(f)] + if targets_path != args.targets_file: # the scanner removes its own file + os.remove(targets_path) + except (OSError, ValueError, AttributeError): pass - if args.target_files: target_files_basenames.extend([os.path.basename(t.strip()) for t in args.target_files.split(',')]) @@ -327,22 +437,25 @@ def main(): print("==================================================") print("[警告]: 当前处于 Dry-Run (空跑) 模式，不会修改系统中的任何文件！") print("==================================================") + if not CIVITAI_API_KEY and not args.offline_only: + print("[!] 未配置 Civitai API Key。部分限制级模型或将无法获取图片。") write_scan_progress(args.progress_file, "enumerating") total_files = count_scan_files(target_folder, target_files_basenames, args.retry_unmatched) write_scan_progress(args.progress_file, "scanning", total_files) current_file = 0 last_progress_write = 0.0 + state = {"network_failures": 0, "civitai_down": args.civitai_down} + counts = {"success": 0, "fail": 0, "unchanged": 0} for root, _, files in os.walk(target_folder): for filename in files: if not filename.endswith(".safetensors"): continue - if target_files_basenames and filename not in target_files_basenames: continue - previous_info = read_local_info(os.path.join(root, os.path.splitext(filename)[0])) if args.retry_unmatched else None - if args.retry_unmatched and not is_unmatched(previous_info): + previous = read_local_info(os.path.join(root, os.path.splitext(filename)[0])) + if args.retry_unmatched and not is_unmatched(previous): continue current_file += 1 @@ -351,218 +464,28 @@ def main(): write_scan_progress(args.progress_file, "scanning", total_files, current_file, filename) last_progress_write = now - file_path = os.path.join(root, filename) - old_base = os.path.splitext(file_path)[0] - - # A .civitai.info holding only edits from older versions is not a scan result. - info_exists = scan_info_path(old_base) is not None - if args.force_overwrite or args.retry_unmatched: - info_exists = False - - preview_exists = args.skip_media - if not preview_exists: - for ext in COVER_SUFFIXES: - if os.path.exists(old_base + ext): - preview_exists = True - break - - needs_rename = False - if not args.skip_rename and args.physical_rename: - needs_rename = True - elif args.virtual_rename: - needs_rename = True - - if info_exists and preview_exists and not needs_rename: - print(f"[*] 已跳过 (信息满足要求): {filename}") + try: + entry = scan_one(args, state, rename_log, root, filename, previous) + except Exception as e: # one model's trouble does not end the scan of the others + print(f"\033[91m[-] 处理失败，已跳过: {filename}: {e}\033[0m") + entry = {"status": "failed", "error": str(e)[:300]} + if entry is None: + counts["unchanged"] += 1 continue - - print(f"\n---> 处理文件: {filename} (位于 {root})") - - civitai_data = None - file_hash = None - if info_exists and needs_rename: - info_path = scan_info_path(old_base) - try: - with open(info_path, 'r', encoding='utf-8') as f: - civitai_data = json.load(f) - print(f"[*] 本地信息存在，直接进入重命名流程") - except: - pass - - if not civitai_data: - if previous_info: # --retry-unmatched: the hash from last time, if the file is unchanged - file_hash = sidecar_file_hash(previous_info, file_path, (previous_info.get("files") or [{}])[0])[0] - file_hash = file_hash or calculate_sha256(file_path) - if not args.offline_only: - civitai_data = fetch_civitai_info(file_hash) + counts["fail" if entry["status"] == "failed" else "success"] += 1 + write_report(args, file=os.path.join(root, filename), **entry) - # Fallback 3: Local Offline Inference (if Civitai still fails or offline_only) - if not civitai_data: - if args.skip_local_metadata: - print(f"[*] Civitai 获取失败，且禁用了本地元数据解析。跳过 {filename}") - fail_count += 1 - continue - # 尝试离线推断底模 - inferred_base = infer_base_model_from_header(file_path) - if inferred_base == 'Unknown': - inferred_base = "" - - print(f"[*] 使用本地哈希重建基础元数据 ({filename})") - civitai_data = { - "id": -1, - "modelId": -1, - "name": os.path.splitext(filename)[0], - "baseModel": inferred_base, - "description": "

Automatically inferred by Anomalous Local Engine.

", - "model": { - "name": os.path.splitext(filename)[0], - "type": "LORA" if "lora" in root.lower() else "Checkpoint" - }, - "files": [{"hashes": {"SHA256": file_hash}}] - } - - # --- 额外获取模型主页的说明文字 --- - # What was read from disk is not fetched again: only fresh Civitai data needs the - # model page's description and the cover. - fresh = file_hash is not None and not args.offline_only - model_id = civitai_data.get("modelId") - if fresh and model_id and model_id != -1: - try: - headers = { - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - } - if CIVITAI_API_KEY: - headers["Authorization"] = f"Bearer {CIVITAI_API_KEY}" - req = urllib.request.Request(f"https://civitai.com/api/v1/models/{model_id}", headers=headers) - with urllib.request.urlopen(req, timeout=10) as m_resp: - m_data = json.loads(m_resp.read().decode('utf-8')) - if "description" in m_data and m_data["description"]: - civitai_data["description"] = m_data["description"] - if "model" not in civitai_data or not isinstance(civitai_data["model"], dict): - civitai_data["model"] = {} - civitai_data["model"]["description"] = m_data["description"] - except Exception as e: - print(f"[-] 获取模型主页详细说明失败: {e}") - # ---------------------------------- - - model_name = sanitize_filename(civitai_data.get("model", {}).get("name", "UnknownModel")) - version_name = sanitize_filename(civitai_data.get("name", "UnknownVersion")) - - new_filename = f"{model_name}_{version_name}.safetensors" - new_file_path = os.path.join(root, new_filename) - new_base = os.path.splitext(new_file_path)[0] - - # ========================================== - # 兼容性大刀阔斧改革：直接保存全宇宙最原汁原味的格式 - # ========================================== - info_data = civitai_data - if file_hash: - info_data["anomalous_file_identity"] = computed_file_identity(file_path, file_hash) - - # Civitai's name for the card; a name set in the model editor lives in - # .anomalous.json and is shown instead. - if args.virtual_rename: - info_data["anomalous_custom_name"] = f"{model_name}_{version_name}" - - if not args.dry_run: - info_path = old_base + ".info" - with open(info_path, 'w', encoding='utf-8') as f: - json.dump(info_data, f, ensure_ascii=True, indent=4) - print(f"[+] 写入纯净版 Civitai 描述 -> .info") - else: - print(f"[Dry-Run] 拟生成标准描述信息 -> .info") - - media_url = None - images = civitai_data.get("images", []) - if images and len(images) > 0: - for img_obj in images: - if not args.skip_media: - if img_obj.get("url"): - media_url = img_obj.get("url") - break - - civitai_covers, user_covers = cover_owners(old_base) - # Download when the data is fresh, or when the model has no cover at all. - if media_url and not args.skip_media and (fresh or not (civitai_covers or user_covers)): - if not args.dry_run: - print(f"[*] 正在下载预览媒体...") - saved_path = download_media(media_url, old_base + ".civitai_bak") - if saved_path: - print(f"[+] 媒体下载成功 -> {os.path.basename(saved_path)}") - if user_covers: - # The user's cover always stays; Civitai's image is kept as the backup. - print(f"[*] 保留你自己的封面，Civitai 封面存为备份") - else: - ext = os.path.splitext(saved_path)[1] - preview_path = old_base + (ext if ext.startswith('.preview.') else f".preview{ext}") - # Earlier copies of Civitai's image under another name give way to the new one. - older = [c for c in civitai_covers if c != preview_path] - try: - if older: - move_to_trash(*older) - except Exception as e: - print(f"[-] 旧的 Civitai 封面没有移走: {e}") - shutil.copy2(saved_path, preview_path) - else: - print(f"[Dry-Run] 拟下载预览媒体...") - - if args.skip_rename or not args.physical_rename: - print(f"[*] 物理重命名已跳过。仅保存 .info 及其可能包含的虚拟重命名。") - success_count += 1 - elif file_path != new_file_path and new_filename != filename: - if os.path.exists(new_file_path): - print(f"[*] 目标文件名已存在，正在验证内容是否完全相同: {filename}") - try: - files_identical = os.path.getsize(file_path) == os.path.getsize(new_file_path) - if files_identical: - files_identical = calculate_sha256(file_path) == calculate_sha256(new_file_path) - except OSError: - files_identical = False - - if not files_identical: - print(f"\033[91m[-] 目标名称冲突但文件内容不同，已保留两个模型并跳过重命名: {filename}\033[0m") - fail_count += 1 - elif not args.dry_run: - try: - print(f"[*] Hash 一致，把已确认的重复副本移到回收站: {filename}") - move_to_trash(file_path, *[old_base + ext for ext in SIDECAR_SUFFIXES]) - success_count += 1 - except Exception as e: - print(f"[-] 重复副本没有移走，两份都保留 (可能文件被占用或这个盘没有回收站): {e}") - fail_count += 1 - else: - print(f"[Dry-Run] Hash 一致，拟删除重复副本及其附属文件: {filename}") - else: - if not args.dry_run: - os.rename(file_path, new_file_path) - rename_log[file_path] = new_file_path - - for ext in SIDECAR_SUFFIXES: - old_ext = old_base + ext - new_ext = new_base + ext - if os.path.exists(old_ext): - os.replace(old_ext, new_ext) - - print(f"[+] 物理重命名完成: {filename} ==> {new_filename}") - success_count += 1 - else: - print(f"[Dry-Run] 拟物理重命名文件: {filename} ==> {new_filename}") - print(f"[Dry-Run] 拟连带重命名附属文件 (.info / .png 等)") - else: - print("[*] 文件名已符合规范，无需重命名。") - success_count += 1 - - # Save scan results write_scan_progress(args.progress_file, "complete", total_files, total_files) + write_report(args, event="done", unchanged=counts["unchanged"], civitai_down=state["civitai_down"]) if not args.dry_run: result_path = os.path.join(target_folder, ".scan_result.json") try: with open(result_path, 'w', encoding='utf-8') as f: - json.dump({"success": success_count, "fail": fail_count}, f) + json.dump({"success": counts["success"], "fail": counts["fail"]}, f) except Exception as e: print(f"[-] 保存统计结果失败: {e}") - + if not args.dry_run and rename_log: with open(backup_log_path, 'w', encoding='utf-8') as f: json.dump(rename_log, f, ensure_ascii=True, indent=4) diff --git a/web/modules/activity_log.js b/web/modules/activity_log.js index 88a4f71..905c792 100644 --- a/web/modules/activity_log.js +++ b/web/modules/activity_log.js @@ -5,13 +5,14 @@ */ import { translate as t } from './locales.js'; +import { countsLine, fileOutcome, scanScope, STATUS_MARKS } from './scan_results.js'; // Server action id → locale key. const ACTION_KEYS = { model_delete: 'activityModelDelete', model_edit: 'activityModelEdit', model_cover: 'activityModelCover', - scan: 'activityScan', + scan: 'activityScan', // entries from before scans were recorded with their result recipe_save: 'activityRecipeSave', recipe_edit: 'activityRecipeEdit', recipe_delete: 'activityRecipeDelete', @@ -70,10 +71,41 @@ export function entrySummary(entry) { if (opened) return t('activityCanvasOpened', { name: opened.node || '—', count: opened.after || '?' }); return t('activityCanvasChanged', { count: entry.detail?.total || changes.length }); } + if (entry.action === 'scan_done') { + const scan = entry.detail?.scan || {}; + return t('activityScanDone', { scope: scanScope(scan, entry.target), counts: countsLine(scan.counts) }); + } const words = t(ACTION_KEYS[entry.action] || 'activityUnknown'); return entry.target ? t('activityWithTarget', { action: words, target: entry.target }) : words; } +const FIELD_KEYS = { + custom_name: 'modelSourceFieldName', custom_notes: 'modelSourceFieldNotes', + source_url: 'modelSourceFieldLink', filename: 'activityFieldFilename', +}; + +/** + * The lines a file entry opens to: [{text, model?}], `model` being a scanned model the line + * can open ({type, path_idx, rel, filename}). Empty for entries without details. + */ +export function fileDetailLines(entry) { + const detail = entry.detail || {}; + const empty = t('activityEmptyValue'); + const lines = []; + if (detail.scan?.civitai_down) lines.push({ text: t('scanResultCivitaiDown') }); + for (const error of detail.scan?.errors || []) lines.push({ text: error }); + for (const item of detail.files || []) { + if (typeof item === 'string') lines.push({ text: t('activityFileMoved', { name: item }) }); + else lines.push({ text: `${STATUS_MARKS[item.status] || '·'} ${item.filename}: ${fileOutcome(item)}`, model: item.rel ? item : null }); + } + for (const field of detail.fields || []) { + lines.push({ text: t('activityFieldChange', { + field: t(FIELD_KEYS[field.field] || field.field), before: field.before || empty, after: field.after || empty, + }) }); + } + return lines; +} + /** One line per canvas change (not for `opened`). */ export function changeLine(change) { const empty = t('activityEmptyValue'); diff --git a/web/modules/locales.js b/web/modules/locales.js index dc74f06..bb32dde 100644 --- a/web/modules/locales.js +++ b/web/modules/locales.js @@ -289,19 +289,18 @@ export const i18n = { collapseAll: '收起全部', expandAll: '展开全部', scanPageTitle: "扫描模型", - scanPageLead: "读取模型文件夹里的 .safetensors 模型，按文件哈希从 Civitai 获取封面、触发词和底模。已经扫过的模型不会重复下载。", + scanPageLead: "读取模型文件夹里的 .safetensors 模型，按文件哈希从 Civitai 获取封面、触发词和底模；Civitai 上找不到的，从模型文件推断底模。已经扫过的模型不会重复下载。", scanPageTotal: "模型总数", scanPageMatched: "已匹配 Civitai", scanPageUnmatched: "没匹配上", scanPageNew: "还没扫", scanPageSummaryFailed: "暂时读不到模型数量，仍然可以扫描。", - scanPageScanNew: "扫描 {count} 个新模型", + scanPageScanNew: "扫描 {count} 个待查的模型", scanPageScanAgain: "再检查一遍", scanPageScanPicked: "扫描挑选的 {count} 个模型", scanPageScanning: "正在扫描…", - scanPageRetry: "重新联网查找 {count} 个没匹配上的模型", - scanPageHint: "扫描在后台进行，可以关掉窗口继续用 ComfyUI，进度会显示在右下角。", - scanPageHintRetry: "“没匹配上”是上次扫描时 Civitai 没找到，或者当时连不上网。重新查找不会改动你的封面，文件没变时也不用重新计算哈希。扫描在后台进行，可以关掉窗口。", + scanPageRetry: "全部重新联网查找（{count} 个没匹配上的）", + scanPageHint: "联网扫描先查 Civitai，查不到的从文件推断；连不上 Civitai 时会自动改成离线推断，下次再补查。扫描在后台进行，可以关掉窗口继续用 ComfyUI。", scanPageHintPicked: "只扫描在高级选项里挑选的模型。", scanPageAdvanced: "高级选项", scanPageScope: "扫描哪些模型", @@ -312,7 +311,7 @@ export const i18n = { scanPageNothingPicked: "还没有挑选模型。", scanPageHow: "怎么扫描", scanPageOffline: "离线扫描", - scanPageOfflineHelp: "不联网，只从模型文件推断底模，不会有封面和触发词。网络不通时用。", + scanPageOfflineHelp: "不联网，只从模型文件推断底模，没有封面和触发词。之后联网扫描会自动补查这些模型。", scanPageDisplayName: "用 Civitai 上的名称作为显示名", scanPageDisplayNameHelp: "只改卡片上显示的名字，不改文件。你在编辑窗口里改过的名字会保留。", scanPageFileRename: "同时修改磁盘上的文件名", @@ -332,11 +331,60 @@ export const i18n = { scanPageApiKeyCleared: "API Key 已清除。", scanPageApiKeyFailed: "保存失败：{error}", scanPageStartFailed: "扫描没能开始：{error}", + scanReasonNotFound: "Civitai 上没有这个文件", + scanReasonNetwork: "当时连不上 Civitai", + scanReasonOffline: "离线扫描，还没查 Civitai", + scanReasonUnknown: "Civitai 上没找到", + scanErrorNameConflict: "目标文件名已被另一个不同的模型占用，没有改名", + scanErrorDuplicateKept: "发现完全相同的副本，但没能移到回收站，两份都保留了", + scanCountMatched: "新匹配 {count}", + scanCountInferred: "从文件推断 {count}", + scanCountFailed: "失败 {count}", + scanCountCovers: "下载封面 {count}", + scanCountRenamed: "改名 {count}", + scanCountNothing: "没有需要更新的模型", + scanFileMatched: "已匹配 Civitai：{name}", + scanFileInferredBase: "{reason}，底模从文件推断为 {base}", + scanFileInferred: "{reason}，文件里也没读出底模", + scanFileFailed: "没有完成：{error}", + scanFileUpdated: "已更新", + scanFileRenamed: "原文件名 {from}", + scanFileDuplicate: "重复副本已移到回收站", + scanFileCoverKept: "保留了你的封面，Civitai 封面存为备份", + scanScopeOne: "扫描 {name}", + scanScopePicked: "扫描挑选的模型", + scanScopeRetry: "重新联网查找", + scanScopeAll: "扫描全部模型", + scanCivitaiDownShort: "中途连不上 Civitai", + scanOneUnchanged: "这个模型的信息已经是最新的", + scanOneTitle: "扫描：{name}", + scanModelGone: "找不到 {name}，可能已被移动或删除。", + scanRowOpen: "查看", + scanRowRetry: "重新查找", + scanRowSearch: "去 Civitai 搜", + scanRowScan: "扫描", + scanShowAll: "显示全部 {count} 个", + scanResultTitle: "上次扫描 · {when} · {scope} · 用时 {duration}", + scanSeconds: "{count} 秒", + scanMinutes: "{count} 分钟", + scanResultOffline: "这是离线扫描：没有查 Civitai，下次联网扫描会补查这些模型。", + scanResultCivitaiDown: "扫描中途连不上 Civitai，之后的模型先从文件推断了底模。网络恢复后，下次扫描会自动补查它们。", + scanResultNothing: "没有模型需要更新（{count} 个已是最新）。", + scanResultUnchanged: "另有 {count} 个模型已是最新，没有改动。", + scanListNew: "还没扫 · {count}", + scanListUnmatched: "没匹配上 · {count}", + scanListNewHint: "这些模型还没扫描过，卡片上标着“未扫描”。可以逐个扫描，也可以用上面的按钮一起扫。", + scanListUnmatchedHint: "Civitai 上没找到这些模型，底模是从文件推断的（卡片上的“≈”）。“当时连不上”和“离线扫描”的，下次联网扫描会自动补查；Civitai 上确实没有的（比如自己训练、合并或从别处下载的），可以去 Civitai 搜一下，或在编辑窗口里填上下载地址。", + scanListUnmatchedHintOffline: "Civitai 上没找到这些模型，底模是从文件推断的。离线扫描时不能重新查找。", + scanPageShowList: "查看是哪些模型", + scanPageScanNewOffline: "离线扫描 {count} 个新模型", + scanPageHintOffline: "离线扫描只读模型文件推断底模，不联网，没有封面和触发词。之后联网扫描会自动补查这些模型。", + scanProgressShowResult: "查看结果", modelSourceLabel: "来源", modelSourceCivitai: "Civitai", modelSourceLocal: "从文件推断", modelSourceLocalShort: "本地", - modelSourceLocalHint: "底模是从模型文件推断的：Civitai 上没找到这个模型，或者扫描时连不上 Civitai。可以在扫描页“重新联网查找”。", + modelSourceLocalHint: "{reason}，底模是从模型文件推断的。可以在扫描页重新联网查找。", modelSourceNotScanned: "未扫描", modelSourceNotScannedHint: "还没扫描：到扫描页补全封面、触发词和底模。", modelSourceEdited: "你改过：{fields}", @@ -860,6 +908,10 @@ export const i18n = { activityModelEdit: "修改模型信息", activityModelCover: "更换模型封面", activityScan: "扫描模型", + activityScanDone: "{scope}：{counts}", + activityFieldFilename: "文件名", + activityFileMoved: "移到回收站：{name}", + activityFieldChange: "{field}：{before} → {after}", activityRecipeSave: "保存工作流配方", activityRecipeEdit: "修改配方", activityRecipeDelete: "删除配方", @@ -2121,19 +2173,18 @@ export const i18n = { collapseAll: 'Collapse All', expandAll: 'Expand All', scanPageTitle: "Scan models", - scanPageLead: "Reads the .safetensors models in your model folders and fetches covers, trigger words and base models from Civitai by file hash. Models already scanned are not downloaded again.", + scanPageLead: "Reads the .safetensors models in your model folders and fetches covers, trigger words and base models from Civitai by file hash; for models Civitai does not know, the base model is inferred from the file. Models already scanned are not downloaded again.", scanPageTotal: "Models", scanPageMatched: "Matched on Civitai", scanPageUnmatched: "Not matched", scanPageNew: "Not scanned yet", scanPageSummaryFailed: "Couldn't count the models right now; you can still scan.", - scanPageScanNew: "Scan {count} new models", + scanPageScanNew: "Scan {count} models not looked up yet", scanPageScanAgain: "Check again", scanPageScanPicked: "Scan the {count} picked models", scanPageScanning: "Scanning…", - scanPageRetry: "Look up the {count} unmatched models again", - scanPageHint: "The scan runs in the background: you can close this window and keep using ComfyUI; progress shows at the bottom right.", - scanPageHintRetry: "“Not matched” models were not found on Civitai last time, or Civitai could not be reached then. Looking them up again keeps your covers and, for unchanged files, reuses the saved hash. Scans run in the background; you can close this window.", + scanPageRetry: "Look up all {count} unmatched models again", + scanPageHint: "An online scan asks Civitai first and infers from the file what Civitai does not know; when Civitai cannot be reached it falls back to the file and looks again next time. Scans run in the background; you can close this window and keep using ComfyUI.", scanPageHintPicked: "Only the models picked under Advanced are scanned.", scanPageAdvanced: "Advanced", scanPageScope: "Which models", @@ -2144,7 +2195,7 @@ export const i18n = { scanPageNothingPicked: "No models picked yet.", scanPageHow: "How to scan", scanPageOffline: "Offline scan", - scanPageOfflineHelp: "No internet: only the base model is inferred from the file; no covers or trigger words. For when you are offline.", + scanPageOfflineHelp: "No internet: only the base model is inferred from the file; no covers or trigger words. The next online scan looks these models up.", scanPageDisplayName: "Use Civitai's name as the display name", scanPageDisplayNameHelp: "Only the name shown on cards; files keep their names. Names you set in the model editor are kept.", scanPageFileRename: "Also rename the files on disk", @@ -2164,11 +2215,60 @@ export const i18n = { scanPageApiKeyCleared: "API key cleared.", scanPageApiKeyFailed: "Could not save: {error}", scanPageStartFailed: "The scan could not start: {error}", + scanReasonNotFound: "Civitai does not know this file", + scanReasonNetwork: "Civitai could not be reached then", + scanReasonOffline: "Offline scan, Civitai not asked yet", + scanReasonUnknown: "Not found on Civitai", + scanErrorNameConflict: "a different model already has the target file name; not renamed", + scanErrorDuplicateKept: "an identical copy could not be moved to the Recycle Bin; both kept", + scanCountMatched: "{count} matched", + scanCountInferred: "{count} inferred from the file", + scanCountFailed: "{count} failed", + scanCountCovers: "{count} covers downloaded", + scanCountRenamed: "{count} renamed", + scanCountNothing: "No model needed an update", + scanFileMatched: "Matched on Civitai: {name}", + scanFileInferredBase: "{reason}; base model inferred from the file: {base}", + scanFileInferred: "{reason}; the file does not tell its base model either", + scanFileFailed: "Not done: {error}", + scanFileUpdated: "Updated", + scanFileRenamed: "was {from}", + scanFileDuplicate: "duplicate copy moved to the Recycle Bin", + scanFileCoverKept: "your cover kept, Civitai’s saved as the backup", + scanScopeOne: "Scanned {name}", + scanScopePicked: "Scanned the picked models", + scanScopeRetry: "Looked up again", + scanScopeAll: "Scanned all models", + scanCivitaiDownShort: "Civitai stopped answering", + scanOneUnchanged: "This model’s information is already up to date", + scanOneTitle: "Scanning: {name}", + scanModelGone: "{name} is not there any more; it may have been moved or deleted.", + scanRowOpen: "Open", + scanRowRetry: "Look up again", + scanRowSearch: "Search Civitai", + scanRowScan: "Scan", + scanShowAll: "Show all {count}", + scanResultTitle: "Last scan · {when} · {scope} · took {duration}", + scanSeconds: "{count} s", + scanMinutes: "{count} min", + scanResultOffline: "This was an offline scan: Civitai was not asked; the next online scan looks these models up.", + scanResultCivitaiDown: "Civitai stopped answering during the scan, so the base model of the models after that was inferred from their files. The next scan looks them up once the connection is back.", + scanResultNothing: "No model needed an update ({count} already up to date).", + scanResultUnchanged: "{count} more models were already up to date.", + scanListNew: "Not scanned · {count}", + scanListUnmatched: "Not matched · {count}", + scanListNewHint: "These models have not been scanned; their cards say “Not scanned”. Scan them one by one, or all with the button above.", + scanListUnmatchedHint: "Civitai did not find these models; their base model was inferred from the file (“≈” on the card). Those Civitai could not be asked about (no connection, offline scan) are looked up by the next online scan. For models Civitai really does not have (your own, merges, from elsewhere), search Civitai or set the download link in the model editor.", + scanListUnmatchedHintOffline: "Civitai did not find these models; their base model was inferred from the file. Looking them up again needs an online scan.", + scanPageShowList: "Show which models", + scanPageScanNewOffline: "Scan {count} new models offline", + scanPageHintOffline: "An offline scan only reads the files to infer the base model: no internet, no covers or trigger words. The next online scan looks these models up.", + scanProgressShowResult: "Show the result", modelSourceLabel: "Source", modelSourceCivitai: "Civitai", modelSourceLocal: "Inferred from the file", modelSourceLocalShort: "Local", - modelSourceLocalHint: "The base model was inferred from the file: Civitai does not know this model, or could not be reached during the scan. Try \"Look up again\" on the scan page.", + modelSourceLocalHint: "{reason}; the base model was inferred from the file. You can look it up again on the scan page.", modelSourceNotScanned: "Not scanned", modelSourceNotScannedHint: "Not scanned yet: the scan page fills in cover, trigger words and base model.", modelSourceEdited: "Edited by you: {fields}", @@ -2692,6 +2792,10 @@ export const i18n = { activityModelEdit: "Edited model info", activityModelCover: "Changed model cover", activityScan: "Scanned models", + activityScanDone: "{scope}: {counts}", + activityFieldFilename: "file name", + activityFileMoved: "Moved to the Recycle Bin: {name}", + activityFieldChange: "{field}: {before} → {after}", activityRecipeSave: "Saved workflow recipe", activityRecipeEdit: "Edited recipe", activityRecipeDelete: "Deleted recipe", diff --git a/web/modules/model_source.js b/web/modules/model_source.js index 761a294..395cad2 100644 --- a/web/modules/model_source.js +++ b/web/modules/model_source.js @@ -7,6 +7,7 @@ import { translate as t } from './locales.js'; import { escapeHtml } from './safe_dom.js'; +import { reasonText } from './scan_results.js'; const FIELD_KEYS = { custom_name: 'modelSourceFieldName', custom_notes: 'modelSourceFieldNotes', source_url: 'modelSourceFieldLink' }; @@ -32,7 +33,7 @@ export function renderSourceBadge(model) { if (meta.info_source === 'local') { badge.classList.add('is-inferred', ...(baseModel ? [colourClass(baseModel)] : [])); badge.textContent = `≈ ${baseModel || t('modelSourceLocalShort')}`; - badge.title = t('modelSourceLocalHint'); + badge.title = t('modelSourceLocalHint', { reason: reasonText(meta.unmatched_reason) }); } else if (!meta.info_source && scannable(model)) { badge.classList.add('is-unscanned'); badge.textContent = t('modelSourceNotScanned'); @@ -57,6 +58,6 @@ export function renderSourceLine(span, model, meta) { const edited = (meta.user_fields || []).map(field => t(FIELD_KEYS[field] || field)); if (edited.length) parts.push(escapeHtml(t('modelSourceEdited', { fields: edited.join(t('modelSourceListSep')) }))); span.innerHTML = parts.join(' · '); - span.title = meta.info_source === 'local' ? t('modelSourceLocalHint') + span.title = meta.info_source === 'local' ? t('modelSourceLocalHint', { reason: reasonText(meta.unmatched_reason) }) : !meta.info_source && scannable(model) ? t('modelSourceNotScannedHint') : ''; } diff --git a/web/modules/scan_progress.js b/web/modules/scan_progress.js index ddccc1d..289059e 100644 --- a/web/modules/scan_progress.js +++ b/web/modules/scan_progress.js @@ -138,17 +138,18 @@ export function updateScanProgress(status, titleText = '') { } -function closeLater(panel) { +function closeLater(panel, ms = 4000) { if (closeTimer) clearTimeout(closeTimer); if (host) return; // on the scan page the result stays until the page is left closeTimer = setTimeout(() => { if (panel === current) removePanel(); closeTimer = null; - }, 4000); + }, ms); } -export function finishScanProgress(detailText = '') { +/** The scan has ended. `action` ({label, onClick}): a button while the box floats, e.g. to the result. */ +export function finishScanProgress(detailText = '', action = null) { const panel = ensurePanel(); panel.classList.remove('is-error'); panel.classList.add('is-complete'); @@ -166,8 +167,20 @@ export function finishScanProgress(detailText = '') { panel.querySelector('.anomalous-scan-progress-detail').textContent = translate('scanProgressFinished'); } } - panel.querySelector('.anomalous-scan-progress-item').textContent = ''; - closeLater(panel); + const item = panel.querySelector('.anomalous-scan-progress-item'); + item.textContent = ''; + if (action && !host) { + const open = document.createElement('button'); + open.type = 'button'; + open.className = 'anomalous-scan-progress-action'; + open.textContent = action.label; + open.onclick = () => { + removePanel(); + action.onClick(); + }; + item.appendChild(open); + } + closeLater(panel, action && !host ? 12000 : 4000); } diff --git a/web/modules/scan_results.js b/web/modules/scan_results.js new file mode 100644 index 0000000..cbb0973 --- /dev/null +++ b/web/modules/scan_results.js @@ -0,0 +1,68 @@ +/** + * Words for what a scan found (the result GET /anomalous/last_scan returns, also kept in the + * activity log): a model's outcome, why an unmatched model has no Civitai record, a scan's + * counts. Shared by the scan page, the activity page and the progress box. No DOM. + */ + +import { translate as t } from './locales.js'; + +const REASON_KEYS = { not_found: 'scanReasonNotFound', network: 'scanReasonNetwork', offline: 'scanReasonOffline' }; +const ERROR_KEYS = { name_conflict: 'scanErrorNameConflict', duplicate_kept: 'scanErrorDuplicateKept' }; +const COUNT_KEYS = [ + ['matched', 'scanCountMatched'], ['inferred', 'scanCountInferred'], ['failed', 'scanCountFailed'], + ['covers', 'scanCountCovers'], ['renamed', 'scanCountRenamed'], +]; + +/** The mark before a model's line. */ +export const STATUS_MARKS = { matched: '✓', inferred: '≈', failed: '✕', unchanged: '·' }; + +/** Why an unmatched model has no Civitai record. */ +export function reasonText(reason) { + return t(REASON_KEYS[reason] || 'scanReasonUnknown'); +} + +/** Models whose lookup did not happen (offline, or no connection): the next online scan asks. */ +export function isPending(reason) { + return reason === 'offline' || reason === 'network'; +} + +/** "3 matched · 1 inferred …", the parts that are not zero. */ +export function countsLine(counts = {}) { + const parts = COUNT_KEYS.filter(([key]) => counts[key]).map(([key, label]) => t(label, { count: counts[key] })); + return parts.length ? parts.join(' · ') : t('scanCountNothing'); +} + +/** What happened to one model in a scan. */ +export function fileOutcome(item) { + let text; + if (item.status === 'matched') text = t('scanFileMatched', { name: item.name || item.filename }); + else if (item.status === 'inferred') { + text = item.base ? t('scanFileInferredBase', { reason: reasonText(item.reason), base: item.base }) + : t('scanFileInferred', { reason: reasonText(item.reason) }); + } else if (item.status === 'failed') text = t('scanFileFailed', { error: ERROR_KEYS[item.error] ? t(ERROR_KEYS[item.error]) : item.error || '?' }); + else text = t('scanFileUpdated'); + if (item.renamed_from) text += ` · ${t('scanFileRenamed', { from: item.renamed_from })}`; + if (item.duplicate) text += ` · ${t('scanFileDuplicate')}`; + if (item.cover === 'kept') text += ` · ${t('scanFileCoverKept')}`; + return text; +} + +/** Which models a scan covered: all, picked, or one (`target`: its file name). */ +export function scanScope(scan, target = '') { + if (scan?.kind === 'one') return t('scanScopeOne', { name: target || '?' }); + if (scan?.kind === 'picked') return t('scanScopePicked'); + return t(scan?.options?.retry_unmatched ? 'scanScopeRetry' : 'scanScopeAll'); +} + +/** The progress box's line when a scan has ended. */ +export function resultLine(result) { + if (!result?.counts) return ''; + if (result.kind === 'one') return oneModelLine(result); + return result.civitai_down ? `${countsLine(result.counts)} · ${t('scanCivitaiDownShort')}` : countsLine(result.counts); +} + +/** After one model's scan (its card's radar button). */ +export function oneModelLine(result) { + const item = result?.files?.[0]; + return item ? fileOutcome(item) : t('scanOneUnchanged'); +} diff --git a/web/modules/scan_runner.js b/web/modules/scan_runner.js index 9dae40b..10f3b68 100644 --- a/web/modules/scan_runner.js +++ b/web/modules/scan_runner.js @@ -1,13 +1,15 @@ /** - * Starting scans and following them: a scan of every model folder (or of picked models) - * from the scan page, and the precision scan of one model from its card. Progress goes - * to scan_progress.js; when a scan ends, node drop-downs, hashes and the model grid are - * refreshed. No page DOM here: ui_scan_page.js renders the page. + * Starting scans and following them: every model folder, picked models or listed models from + * the scan page (all through /anomalous/scan_all), and one model from its card. Progress goes + * to scan_progress.js; when a scan ends its result (GET /anomalous/last_scan) is shown, and + * node drop-downs, hashes and the model grid are refreshed. No page DOM here: ui_scan_page.js + * renders the page. */ import { app } from '../../../scripts/app.js'; import { translate as t } from './locales.js'; import { updateScanProgress, finishScanProgress, failScanProgress } from './scan_progress.js'; +import { oneModelLine, resultLine } from './scan_results.js'; import { showWorkbenchToast } from './ui_prompt_toast.js'; const POLL_MS = 2000; @@ -50,6 +52,31 @@ export function scanRequestBody(options) { }; } +/** Listed models ({type, path_idx, rel}) as the scan's `targets`, one per folder. */ +export function targetsForItems(items) { + const folders = new Map(); + for (const item of items) { + const cut = item.rel.lastIndexOf('/'); + const subfolder = cut < 0 ? '/' : `/${item.rel.slice(0, cut)}`; + const key = `${item.type}|${item.path_idx}|${subfolder}`; + if (!folders.has(key)) folders.set(key, { type: item.type, path_idx: item.path_idx, subfolder, files: [] }); + folders.get(key).files.push(item.filename); + } + return [...folders.values()]; +} + +/** The model picker's selection (Map "type|path_idx|subfolder" -> Set of file names) as `targets`. */ +function targetsForSelection(selection) { + const targets = []; + for (const [folderKey, files] of selection.entries()) { + const [type, pathIdx, ...rest] = folderKey.split('|'); + if (type && pathIdx !== undefined && rest.length && files.size) { + targets.push({ type, path_idx: Number(pathIdx), subfolder: rest.join('|'), files: [...files] }); + } + } + return targets; +} + async function postJson(url, body) { const res = await fetch(url, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); const data = await res.json().catch(() => ({})); @@ -57,13 +84,22 @@ async function postJson(url, body) { return data; } +/** The last scan's result ({} before the first, or when it cannot be read). */ +export async function fetchLastScan() { + try { + return await (await fetch('/anomalous/last_scan')).json(); + } catch { + return {}; + } +} + /** Polls a status URL until the scan there stops; resolves with the last status. */ -function followScan(statusUrl, extra = {}) { +function followScan(statusUrl, extra = {}, everyMs = POLL_MS, title = '') { return new Promise((resolve, reject) => { const poll = setInterval(async () => { try { const status = await (await fetch(statusUrl)).json(); - updateScanProgress({ ...status, ...extra }); + updateScanProgress({ ...status, ...extra }, title); if (!status.scanning) { clearInterval(poll); resolve(status); @@ -72,7 +108,7 @@ function followScan(statusUrl, extra = {}) { clearInterval(poll); reject(error); } - }, POLL_MS); + }, everyMs); }); } @@ -87,141 +123,54 @@ async function afterScan(owner, autoFix) { owner.loadModels?.(); } -/** Picked models: one folder after the other (`selection`: Map "type|path_idx|subfolder" -> Set of file names). */ -async function scanSelection(selection, body) { - const folders = [...selection.entries()].filter(([, files]) => files.size > 0); - let current = 0; - updateScanProgress({ scanning: true, phase: 'preparing', folder_total: folders.length, folder_current: 0 }); - let last = {}; - for (const [folderKey, files] of folders) { - current += 1; - const [type, pathIdx, ...rest] = folderKey.split('|'); - const subfolder = rest.join('|'); - if (!type || pathIdx === undefined || !rest.length) continue; - const params = new URLSearchParams({ type, path_idx: pathIdx, subfolder }); - try { - await postJson(`/anomalous/scan?${params}`, { ...body, target_files: [...files] }); - last = await followScan(`/anomalous/scan_status?${params}`, { folder_total: folders.length, folder_current: current, folder: subfolder }); - } catch (error) { - console.error('[AMB] Scan failed for folder:', folderKey, error); - last = { error: String(error.message || error) }; - } - } - return last; +/** The ended scan's line, with a way to its result when the progress box floats. */ +export function showScanResult(owner, result) { + finishScanProgress(resultLine(result), { + label: t('scanProgressShowResult'), + onClick: () => { + owner.show?.(); + owner.openScanPage?.(); + }, + }); } /** - * Runs one scan from the scan page: every active model folder, or `options.selection`. - * Resolves when it has ended (true) or could not start (false, with the reason shown). + * Runs one scan from the scan page: every active model folder, `options.selection` (the model + * picker) or `options.targets` (listed models). Resolves with the scan's result when it has + * ended, or null when it could not start (the reason shown). */ export async function startScan(owner, options) { - if (running) return false; + if (running) return null; running = true; setActiveScanButtonState(true); const body = scanRequestBody(options); + const targets = options.targets || (options.selection ? targetsForSelection(options.selection) : null); + if (targets) body.targets = targets; try { - let status; - if (options.selection) { - status = await scanSelection(options.selection, body); - } else { - const data = await postJson('/anomalous/scan_all', body); - updateScanProgress({ scanning: true, phase: 'preparing', recovered: data.recovered }); - status = await followScan('/anomalous/global_scan_status'); - } + const data = await postJson('/anomalous/scan_all', body); + updateScanProgress({ scanning: true, phase: 'preparing', recovered: data.recovered }); + const status = await followScan('/anomalous/global_scan_status'); + const result = await fetchLastScan(); if (status.interrupted) failScanProgress(t('scanProgressInterrupted')); - else finishScanProgress(); + else showScanResult(owner, result); await afterScan(owner, options.autoFix); - return true; + return result; } catch (error) { failScanProgress(t('scanPageStartFailed', { error: String(error.message || error) })); - return false; + return null; } finally { running = false; setActiveScanButtonState(false); } } -export async function formatScanCompletionToast(model) { - const isZh = window.anomalous_browser_lang === 'zh'; - const fallbackName = model.name || model.filename; - try { - const findRes = await fetch('/anomalous/find_model?search=' + encodeURIComponent(model.filename)); - if (findRes.ok) { - const updated = await findRes.json(); - const target = updated?.model || updated || model; - if (target && target.metadata) { - const meta = target.metadata; - const isCivitai = Boolean( - (meta.id && meta.id !== -1) || - (meta.modelId && meta.modelId !== -1) || - (meta.model_id && meta.model_id !== -1) || - (meta.version_id && meta.version_id !== -1) || - meta.civitai_url - ); - const hasPreview = Boolean(target.preview_url); - const baseModel = meta.baseModel; - - if (isCivitai && hasPreview) { - return isZh ? `✓ 已从 Civitai 获取封面与模型信息！` : `✓ Civitai cover & metadata fetched!`; - } - if (isCivitai && !hasPreview) { - return isZh ? `✓ 已匹配到 Civitai 信息（线上未提供封面）` : `✓ Civitai metadata matched (no cover online)`; - } - if (!isCivitai) { - if (baseModel) { - return isZh - ? `ℹ️ 非 Civitai 模型：已识别底模为 [${baseModel}]` - : `ℹ️ Non-Civitai model: inferred base model [${baseModel}]`; - } - return isZh - ? `ℹ️ 未在 Civitai 匹配到此模型` - : `ℹ️ No Civitai match found for this model`; - } - } - } - } catch { - // fallback to standard text - } - return isZh ? `✓ 模型 [${fallbackName}] 扫描完成！` : `✓ Model [${fallbackName}] scanned!`; -} - -function pollDirectScanStatus(params, titleText, model, onComplete) { - const statusUrl = '/anomalous/scan_status?' + params.toString(); - const poll = setInterval(async () => { - try { - const statusRes = await fetch(statusUrl); - const statusData = await statusRes.json(); - updateScanProgress(statusData, titleText); - - if (!statusData.scanning) { - clearInterval(poll); - if (statusData.interrupted) { - failScanProgress(t('scanProgressInterrupted')); - showWorkbenchToast(window.anomalous_browser_lang === 'zh' ? '扫描被中断' : 'Scan interrupted'); - } else { - const toastMsg = await formatScanCompletionToast(model); - finishScanProgress(toastMsg); - showWorkbenchToast(toastMsg); - } - onComplete(true); - } - } catch (err) { - clearInterval(poll); - failScanProgress(String(err)); - onComplete(false); - } - }, 1200); - return poll; -} - /** The radar button on a model card: scan just that model, in its own folder. */ export async function triggerDirectModelScan(model, triggerBtn = null, browserInstance = null) { const browser = browserInstance || this || {}; if (!model || !model.filename) return; const modelLabel = model.name || model.filename; - const isZh = window.anomalous_browser_lang === 'zh'; - const titleText = isZh ? `精准扫描: ${modelLabel}` : `Scanning: ${modelLabel}`; + const titleText = t('scanOneTitle', { name: modelLabel }); if (triggerBtn) { triggerBtn.classList.add('anomalous-radar-spinning'); @@ -254,45 +203,38 @@ export async function triggerDirectModelScan(model, triggerBtn = null, browserIn path_idx: browser.currentPathIdx || 0, subfolder: browser.currentSubfolder || '/', }); - const reqBody = { + // A model that was looked up without an answer is looked up again. + await postJson(`/anomalous/scan?${params}`, { target_files: [model.filename], offline_only: false, skip_rename: true, virtual_rename: false, physical_rename: false, - force_overwrite: false - }; - - const res = await fetch('/anomalous/scan?' + params.toString(), { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(reqBody) + force_overwrite: false, + retry_unmatched: model.metadata?.info_source === 'local', }); - - const data = await res.json(); - if (data.status === 'ok') { - showWorkbenchToast(isZh ? `开始精准扫描: ${modelLabel}` : `Scanning model: ${modelLabel}`); - pollDirectScanStatus(params, titleText, model, async () => { - resetBtn(); - if (typeof browser.loadModels === 'function') { - browser.loadModels(); - } - try { - if (app?.refreshComboInNodes) await app.refreshComboInNodes(); - if (window.anomalous_reload_hashes) await window.anomalous_reload_hashes(); - } catch (e) { - console.warn('[AMB] Error reloading hashes or combo nodes:', e); - } - }); + showWorkbenchToast(titleText); + const status = await followScan(`/anomalous/scan_status?${params}`, {}, 1200, titleText); + resetBtn(); + if (status.interrupted) { + failScanProgress(t('scanProgressInterrupted')); + showWorkbenchToast(t('scanProgressInterrupted')); } else { - resetBtn(); - const errMsg = data.message || (isZh ? '扫描启动失败' : 'Failed to start scan'); - failScanProgress(errMsg); - showWorkbenchToast(errMsg); + const line = oneModelLine(await fetchLastScan()); + finishScanProgress(line); + showWorkbenchToast(line); + } + browser.loadModels?.(); + try { + if (app?.refreshComboInNodes) await app.refreshComboInNodes(); + if (window.anomalous_reload_hashes) await window.anomalous_reload_hashes(); + } catch (e) { + console.warn('[AMB] Error reloading hashes or combo nodes:', e); } } catch (e) { resetBtn(); - failScanProgress(String(e)); - showWorkbenchToast(String(e)); + const message = String(e.message || e) || t('scanPageStartFailed', { error: '' }); + failScanProgress(message); + showWorkbenchToast(message); } } diff --git a/web/modules/ui_activity.js b/web/modules/ui_activity.js index 7dd672e..f19b84f 100644 --- a/web/modules/ui_activity.js +++ b/web/modules/ui_activity.js @@ -1,12 +1,13 @@ /** - * The activity page (everything Anomalous changed, by day, with filters, details and - * "find on canvas") and the home page's "recent" list. Reads through activity_log.js; - * each render cancels the previous one's request. + * The activity page (everything Anomalous changed, by day, with filters, details, + * "find on canvas" and "open" for scanned models) and the home page's "recent" list. + * Reads through activity_log.js; each render cancels the previous one's request. */ import { app } from "../../../scripts/app.js"; import { translate as t } from './locales.js'; -import { changeLine, clearActivity, dayLabel, entrySummary, fetchActivity, timeLabel } from './activity_log.js'; +import { changeLine, clearActivity, dayLabel, entrySummary, fetchActivity, fileDetailLines, timeLabel } from './activity_log.js'; +import { openListedModel } from './ui_scan_lists.js'; import { changeNodeId } from './activity_diff.js'; import { anomalousConfirm } from './ui_dialog.js'; @@ -45,34 +46,50 @@ function locateNode(change, note) { app.canvas.setDirty?.(true, true); } -function renderEntry(entry, { compact = false } = {}) { +/** What an entry opens to: [{text, label?, onClick?}]. */ +function entryLines(entry, owner, note) { + if (entry.source !== 'canvas') { + return fileDetailLines(entry).map(line => (line.model && owner ? { + text: line.text, + label: t('scanRowOpen'), + onClick: () => openListedModel(owner, line.model, () => owner.goTo('activity')), + } : line)); + } + return (entry.detail?.changes || []).filter(change => change.kind !== 'opened').map(change => ({ + text: changeLine(change), + ...(change.kind !== 'removed' ? { label: t('activityLocate'), onClick: () => locateNode(change, note) } : {}), + })); +} + +function renderEntry(entry, { compact = false, owner = null } = {}) { const row = el('div', `anomalous-activity-entry is-${entry.source === 'canvas' ? 'canvas' : 'file'}`); const head = el('div', 'anomalous-activity-head'); + const summary = entrySummary(entry); head.append(el('span', 'anomalous-activity-time', timeLabel(entry.time)), el('span', 'anomalous-activity-source', t(entry.source === 'canvas' ? 'activityCanvas' : 'activityFiles')), - el('span', 'anomalous-activity-summary', entrySummary(entry))); + el('span', 'anomalous-activity-summary', summary)); + head.title = summary; row.appendChild(head); - const changes = (entry.detail?.changes || []).filter(change => change.kind !== 'opened'); - if (compact || !changes.length) return row; + if (compact) return row; + const note = el('div', 'anomalous-activity-note'); + const lines = entryLines(entry, owner, note); + if (!lines.length) return row; - // Canvas entries open to one line per change. + // An entry with details opens to one line each. head.classList.add('is-expandable'); head.tabIndex = 0; head.setAttribute('role', 'button'); head.setAttribute('aria-expanded', 'false'); const details = el('div', 'anomalous-activity-details'); details.hidden = true; - const note = el('div', 'anomalous-activity-note'); - for (const change of changes) { + for (const { text, label, onClick } of lines) { const line = el('div', 'anomalous-activity-change'); - line.appendChild(el('span', 'anomalous-activity-change-text', changeLine(change))); - line.title = changeLine(change); - if (change.kind !== 'removed') { - line.appendChild(button('anomalous-activity-locate', t('activityLocate'), () => locateNode(change, note))); - } + line.appendChild(el('span', 'anomalous-activity-change-text', text)); + line.title = text; + if (onClick) line.appendChild(button('anomalous-activity-locate', label, onClick)); details.appendChild(line); } - const shown = changes.length; + const shown = entry.source === 'canvas' ? lines.length : (entry.detail?.files || []).length; const total = entry.detail?.total || shown; if (total > shown) details.appendChild(el('div', 'anomalous-activity-more', t('activityMore', { count: total - shown }))); details.appendChild(note); @@ -91,7 +108,7 @@ function renderEntry(entry, { compact = false } = {}) { return row; } -function renderList(list, entries) { +function renderList(list, entries, owner) { list.replaceChildren(); let day = null; for (const entry of entries) { @@ -100,7 +117,7 @@ function renderList(list, entries) { day = label; list.appendChild(el('div', 'anomalous-activity-day', label)); } - list.appendChild(renderEntry(entry)); + list.appendChild(renderEntry(entry, { owner })); } } @@ -120,7 +137,7 @@ export function renderActivityPage(owner, panel) { fetchActivity({ limit: PAGE_SIZE, source: filter, signal }) .then(entries => { if (signal.aborted) return; - if (entries.length) renderList(list, entries); + if (entries.length) renderList(list, entries, owner); else list.replaceChildren(el('div', 'anomalous-activity-empty', t('activityEmpty'))); }) .catch((error) => { diff --git a/web/modules/ui_scan_lists.js b/web/modules/ui_scan_lists.js new file mode 100644 index 0000000..c5516c0 --- /dev/null +++ b/web/modules/ui_scan_lists.js @@ -0,0 +1,178 @@ +/** + * The scan page's model lists: the last scan's result (what each model got, why Civitai had + * nothing) and the models still unmatched or not scanned, from GET /anomalous/scan_summary. + * Every row opens its model; rows offer "scan" / "look up again" and a Civitai search. + * The page passes what the buttons do (`actions`); this module only renders. + */ + +import { translate as t } from './locales.js'; +import { dayLabel, timeLabel } from './activity_log.js'; +import { countsLine, fileOutcome, isPending, reasonText, scanScope, STATUS_MARKS } from './scan_results.js'; +import { typeLabel } from './ui_model_types.js'; + +const SHOWN_ROWS = 100; // more behind "show all" +const RESULT_ROWS = 8; + +function el(tag, className, text) { + const node = document.createElement(tag); + if (className) node.className = className; + if (text !== undefined) node.textContent = text; + return node; +} + +function button(className, label, onClick) { + const node = el('button', className, label); + node.type = 'button'; + node.onclick = onClick; + return node; +} + +/** Opens a listed model's detail; its back button returns to `onBack` (the scan page). */ +export async function openListedModel(owner, item, onBack) { + const params = new URLSearchParams({ type: item.type, path_idx: item.path_idx, rel: item.rel }); + const data = await fetch(`/anomalous/scan_model?${params}`).then(res => res.json()).catch(() => ({})); + if (!data.model) { + alert(t('scanModelGone', { name: item.filename })); + return; + } + owner.hideAllPanels(); + owner.historyStack = []; + owner.currentType = data.model.type; + owner.currentPathIdx = data.model.path_idx; + owner.currentSubfolder = data.model.subfolder || '/'; + owner.currentDetailModel = data.model; + owner.recipeModelReturn = onBack; // set after hideAllPanels, which clears it + owner.showDetail(data.model); +} + +/** Civitai's model search for a file name, to find a model by hand. */ +function searchCivitai(item) { + const query = item.filename.replace(/\.safetensors$/i, '').replace(/[_-]+/g, ' '); + window.open(`https://civitai.com/search/models?query=${encodeURIComponent(query)}`, '_blank', 'noopener'); +} + +function where(item) { + const folder = item.rel.includes('/') ? item.rel.slice(0, item.rel.lastIndexOf('/')) : ''; + return [item.type ? typeLabel({ type: item.type }) : '', folder].filter(Boolean).join(' · '); +} + +/** One model: mark, file name, where it is, what about it, and its buttons. */ +function row(item, detail, buttons, mark = '') { + const line = el('div', `anomalous-scan-row${item.status ? ` is-${item.status}` : ''}`); + if (mark) line.append(el('span', 'anomalous-scan-row-mark', mark)); + const copy = el('div', 'anomalous-scan-row-copy'); + const name = el('strong', 'anomalous-scan-row-name', item.filename); + name.title = item.rel; + copy.append(name, el('small', 'anomalous-scan-row-where', where(item))); + if (detail) copy.append(el('small', 'anomalous-scan-row-detail', detail)); + const actions = el('div', 'anomalous-scan-row-actions'); + for (const [label, onClick, disabled] of buttons) { + const node = button('anomalous-scan-row-btn', label, onClick); + node.disabled = Boolean(disabled); + actions.append(node); + } + line.append(copy, actions); + return line; +} + +/** Buttons for an unmatched model. */ +function unmatchedButtons(item, actions) { + return [ + [t('scanRowOpen'), () => actions.open(item)], + ...(actions.offline ? [] : [[t('scanRowRetry'), () => actions.scan([item], { retry: true }), actions.busy]]), + [t('scanRowSearch'), () => searchCivitai(item)], + ]; +} + +/** Rows into `list`, the first `limit`, then a button for the rest. */ +function fillRows(list, items, render, limit) { + const shown = items.slice(0, limit); + list.replaceChildren(...shown.map(render)); + if (items.length > shown.length) { + list.append(button('anomalous-scan-more', t('scanShowAll', { count: items.length }), () => fillRows(list, items, render, Infinity))); + } +} + +/** The last scan: when, what, the counts, and each model it changed. Null before the first scan. */ +export function renderLastScan(result, actions) { + if (!result?.counts) return null; + const box = el('section', 'anomalous-scan-card anomalous-scan-result'); + const seconds = Math.max(0, Math.round((result.finished || 0) - (result.started || 0))); + const target = result.kind === 'one' ? result.files?.[0]?.filename : ''; + box.append( + el('div', 'anomalous-scan-card-title', t('scanResultTitle', { + when: `${dayLabel(result.finished)} ${timeLabel(result.finished)}`, + scope: scanScope(result, target), + duration: seconds < 60 ? t('scanSeconds', { count: seconds }) : t('scanMinutes', { count: Math.round(seconds / 60) }), + })), + el('div', 'anomalous-scan-result-counts', countsLine(result.counts)), + ); + if (result.options?.offline_only) box.append(el('p', 'anomalous-scan-muted', t('scanResultOffline'))); + if (result.civitai_down) box.append(el('p', 'anomalous-scan-note is-warn', t('scanResultCivitaiDown'))); + for (const error of result.errors || []) box.append(el('p', 'anomalous-scan-note is-error', error)); + + const files = result.files || []; + if (!files.length) { + box.append(el('p', 'anomalous-scan-muted', t('scanResultNothing', { count: result.counts.unchanged || 0 }))); + return box; + } + const list = el('div', 'anomalous-scan-rows'); + const render = item => row(item, fileOutcome(item), + item.status === 'inferred' ? unmatchedButtons(item, actions) : [[t('scanRowOpen'), () => actions.open(item)]], + STATUS_MARKS[item.status] || ''); + fillRows(list, files, render, RESULT_ROWS); + box.append(list); + if (result.counts.unchanged) box.append(el('p', 'anomalous-scan-muted', t('scanResultUnchanged', { count: result.counts.unchanged }))); + return box; +} + +/** The models not scanned yet or unmatched, one tab each; `state.tab` remembers the open one. */ +export function renderModelLists(summary, actions, state) { + const unmatched = [...(summary?.unmatched_models || [])] + .sort((a, b) => Number(isPending(b.reason)) - Number(isPending(a.reason))); + const tabs = [['new', t('scanListNew', { count: summary?.new || 0 }), summary?.new_models || []], + ['unmatched', t('scanListUnmatched', { count: summary?.unmatched || 0 }), unmatched]]; + if (!tabs.some(([, , items]) => items.length)) return null; + if (!tabs.some(([key, , items]) => key === state.tab && items.length)) state.tab = tabs.find(([, , items]) => items.length)[0]; + + const box = el('section', 'anomalous-scan-card anomalous-scan-lists'); + box.id = 'anomalous-scan-lists'; + const bar = el('div', 'anomalous-scan-tabs'); + bar.setAttribute('role', 'tablist'); + const hint = el('p', 'anomalous-scan-muted'); + const list = el('div', 'anomalous-scan-rows'); + const show = (key) => { + state.tab = key; + bar.querySelectorAll('button').forEach(tab => tab.setAttribute('aria-selected', String(tab.dataset.tab === key))); + const items = tabs.find(([tabKey]) => tabKey === key)[2]; + if (key === 'new') { + hint.textContent = t('scanListNewHint'); + fillRows(list, items, item => row(item, '', [ + [t('scanRowOpen'), () => actions.open(item)], + [t('scanRowScan'), () => actions.scan([item]), actions.busy], + ]), SHOWN_ROWS); + } else { + hint.textContent = t(actions.offline ? 'scanListUnmatchedHintOffline' : 'scanListUnmatchedHint'); + fillRows(list, items, item => row(item, [reasonText(item.reason), item.base ? `≈ ${item.base}` : ''].filter(Boolean).join(' · '), + unmatchedButtons(item, actions), '≈'), SHOWN_ROWS); + } + }; + for (const [key, label, items] of tabs) { + const tab = button('anomalous-scan-tab', label, () => show(key)); + tab.dataset.tab = key; + tab.setAttribute('role', 'tab'); + tab.disabled = !items.length; + bar.append(tab); + } + box.append(bar, hint, list); + show(state.tab); + return box; +} + +/** A count on the page that opens its list (unmatched, not scanned). */ +export function showList(page, state, key) { + state.tab = key; + const lists = page.querySelector('#anomalous-scan-lists'); + lists?.querySelector(`[data-tab="${key}"]`)?.click(); + lists?.scrollIntoView({ behavior: 'smooth', block: 'start' }); +} diff --git a/web/modules/ui_scan_page.js b/web/modules/ui_scan_page.js index dcedc62..bfe38fb 100644 --- a/web/modules/ui_scan_page.js +++ b/web/modules/ui_scan_page.js @@ -1,14 +1,16 @@ /** - * The scan page: how many models are matched on Civitai, unmatched or new + * The scan page: how many models are matched on Civitai, unmatched or not scanned * (GET /anomalous/scan_summary), one button to scan, one to look unmatched models up - * again, progress in the page, and the rarer choices folded under "Advanced". + * again, progress in the page, the last scan's result and the unmatched / not scanned + * models (ui_scan_lists.js), and the rarer choices folded under "Advanced". * Scans run through scan_runner.js; the progress box is scan_progress.js's panel, * hosted here while the page is shown. */ import { translate as t } from './locales.js'; -import { isScanRunning, startScan } from './scan_runner.js'; +import { isScanRunning, startScan, targetsForItems } from './scan_runner.js'; import { setScanProgressHost } from './scan_progress.js'; +import { openListedModel, renderLastScan, renderModelLists, showList } from './ui_scan_lists.js'; // Choices kept while the browser is open. The two risky ones go back off after each scan. const options = { @@ -20,6 +22,7 @@ const options = { forceOverwrite: false, autoFix: true, }; +const listState = { tab: '' }; // the open list: 'new' | 'unmatched' function el(tag, className, text) { const node = document.createElement(tag); @@ -50,8 +53,14 @@ async function fetchJson(url) { } } -function stat(value, label, tone) { - const box = el('div', `anomalous-scan-stat${tone ? ` is-${tone}` : ''}`); +/** A count; with `onClick` it opens the list of those models. */ +function stat(value, label, tone, onClick) { + const box = el(onClick ? 'button' : 'div', `anomalous-scan-stat${tone ? ` is-${tone}` : ''}`); + if (onClick) { + box.type = 'button'; + box.onclick = onClick; + box.title = t('scanPageShowList'); + } box.append(el('strong', 'anomalous-scan-stat-value', value ?? '–'), el('span', 'anomalous-scan-stat-label', label)); return box; } @@ -144,26 +153,45 @@ async function saveApiKey() { } } +/** A scan from this page. `extra`: { retryUnmatched } for every unmatched model, or + * { targets, retryUnmatched } for the models of a list row. */ async function run(owner, panel, extra = {}) { - if (options.scope === 'picked' && !extra.retryUnmatched && pickedCount() === 0) { + const whole = !extra.retryUnmatched && !extra.targets; + if (whole && options.scope === 'picked' && pickedCount() === 0) { alert(t('scanPageNothingPicked')); return; } - if (options.forceOverwrite && !extra.retryUnmatched && !confirm(t('scanPageForceConfirm'))) return; - const selection = options.scope === 'picked' && !extra.retryUnmatched ? options.selection : null; + if (whole && options.forceOverwrite && !confirm(t('scanPageForceConfirm'))) return; + const selection = whole && options.scope === 'picked' ? options.selection : null; const started = startScan(owner, { ...options, ...extra, selection }); renderScanPage(owner, panel); // buttons show "scanning" await started; - options.physicalRename = false; - options.forceOverwrite = false; + if (whole) { + options.physicalRename = false; + options.forceOverwrite = false; + } if (panel.isConnected && panel.style.display !== 'none') renderScanPage(owner, panel); } +/** What the list rows' buttons do. */ +function listActions(owner, panel, busy) { + return { + busy, + offline: options.offline, + open: item => openListedModel(owner, item, () => openScanPage(owner)), + scan: (items, { retry = false } = {}) => run(owner, panel, { targets: targetsForItems(items), retryUnmatched: retry }), + }; +} + /** Renders the page into `panel` (again after each scan, so the counts are fresh). */ export async function renderScanPage(owner, panel) { const token = (panel._scanRender = (panel._scanRender || 0) + 1); - const [summary, config] = await Promise.all([fetchJson('/anomalous/scan_summary'), fetchJson('/anomalous/config')]); + const [summary, config, last] = await Promise.all([ + fetchJson('/anomalous/scan_summary'), fetchJson('/anomalous/config'), fetchJson('/anomalous/last_scan'), + ]); if (token !== panel._scanRender) return; + // Closed or left meanwhile: the progress box keeps floating instead of hiding in the page. + if (panel.style.display === 'none' || !owner.modal?.classList.contains('visible')) return; const page = el('div', 'anomalous-scan-page'); page.append(el('h1', 'anomalous-scan-title', t('scanPageTitle')), el('p', 'anomalous-scan-lead', t('scanPageLead'))); @@ -172,8 +200,10 @@ export async function renderScanPage(owner, panel) { stats.append( stat(summary?.total, t('scanPageTotal')), stat(summary?.matched, t('scanPageMatched'), 'ok'), - stat(summary?.unmatched, t('scanPageUnmatched'), summary?.unmatched ? 'warn' : ''), - stat(summary?.new, t('scanPageNew'), summary?.new ? 'new' : ''), + stat(summary?.unmatched, t('scanPageUnmatched'), summary?.unmatched ? 'warn' : '', + summary?.unmatched ? () => showList(page, listState, 'unmatched') : null), + stat(summary?.new, t('scanPageNew'), summary?.new ? 'new' : '', + summary?.new ? () => showList(page, listState, 'new') : null), ); page.append(stats); if (!summary) page.append(el('p', 'anomalous-scan-muted', t('scanPageSummaryFailed'))); @@ -181,9 +211,11 @@ export async function renderScanPage(owner, panel) { const busy = isScanRunning(); const actions = el('div', 'anomalous-scan-actions'); const picked = options.scope === 'picked'; + // An online scan also looks up the models earlier scans could not ask Civitai about. + const waiting = (summary?.new || 0) + (options.offline ? 0 : summary?.pending || 0); const label = busy ? t('scanPageScanning') : picked ? t('scanPageScanPicked', { count: pickedCount() }) - : summary?.new ? t('scanPageScanNew', { count: summary.new }) + : waiting ? t(options.offline ? 'scanPageScanNewOffline' : 'scanPageScanNew', { count: waiting }) : t('scanPageScanAgain'); const primary = button('anomalous-scan-primary', label, () => run(owner, panel)); primary.disabled = busy; @@ -194,12 +226,15 @@ export async function renderScanPage(owner, panel) { actions.append(retry); } page.append(actions); - const hint = picked ? t('scanPageHintPicked') - : summary?.unmatched && !options.offline ? t('scanPageHintRetry') : t('scanPageHint'); + const hint = picked ? t('scanPageHintPicked') : t(options.offline ? 'scanPageHintOffline' : 'scanPageHint'); page.append(el('p', 'anomalous-scan-muted', hint)); const progressHost = el('div', 'anomalous-scan-progress-host'); - page.append(progressHost, renderAdvanced(owner, panel, Boolean(config?.has_api_key))); + page.append(progressHost); + const rowActions = listActions(owner, panel, busy); + const result = renderLastScan(last, rowActions); + const lists = renderModelLists(summary, rowActions, listState); + page.append(...[result, lists].filter(Boolean), renderAdvanced(owner, panel, Boolean(config?.has_api_key))); panel.replaceChildren(page); setScanProgressHost(progressHost); // A scan that ends elsewhere (a card's radar, another tab) refreshes the counts too. diff --git a/web/modules/ui_scan_watch.js b/web/modules/ui_scan_watch.js index fa32641..2998cbf 100644 --- a/web/modules/ui_scan_watch.js +++ b/web/modules/ui_scan_watch.js @@ -1,11 +1,11 @@ /** - * Follows scans started anywhere (the wizard, a single model, another tab): shows their - * progress, turns the rail's scan button into a spinner, and reloads the models when done. + * Follows scans started anywhere (the scan page, a single model, another tab): shows their + * progress and result, turns the rail's scan button into a spinner, and reloads the models when done. */ import { translate as t } from './locales.js'; -import { updateScanProgress, finishScanProgress, failScanProgress } from './scan_progress.js'; -import { setScanButtonState } from './scan_runner.js'; +import { updateScanProgress, failScanProgress } from './scan_progress.js'; +import { fetchLastScan, setScanButtonState, showScanResult } from './scan_runner.js'; const POLL_MS = 3000; @@ -33,7 +33,7 @@ export function watchScans(owner) { } else if (!active && scanning) { scanning = false; if (button) setScanButtonState(button, false); - finishScanProgress(); + showScanResult(owner, await fetchLastScan()); owner.loadModels(); owner.onScanFinished?.(); if (window.anomalous_reload_hashes) await window.anomalous_reload_hashes(); diff --git a/web/styles/00-foundation-models.css b/web/styles/00-foundation-models.css index fbdac41..71d056a 100644 --- a/web/styles/00-foundation-models.css +++ b/web/styles/00-foundation-models.css @@ -903,6 +903,22 @@ html:not(.anomalous-floating-entry-enabled) [data-setting-id="Anomalous.ModelBro font-size: 11px; } +.anomalous-scan-progress-action { + margin-top: 8px; + padding: 4px 12px; + border: 1px solid rgba(138, 180, 248, 0.45); + border-radius: 999px; + background: transparent; + color: #d7dde8; + font-size: 12px; + cursor: pointer; +} + +.anomalous-scan-progress-action:hover { + background: rgba(138, 180, 248, 0.14); + color: #ffffff; +} + .anomalous-scan-progress.is-complete .anomalous-scan-progress-fill { background: #81c995; } diff --git a/web/styles/18-scan-page.css b/web/styles/18-scan-page.css index beb716b..472a81a 100644 --- a/web/styles/18-scan-page.css +++ b/web/styles/18-scan-page.css @@ -44,6 +44,18 @@ border: 1px solid rgba(255, 255, 255, 0.08); border-radius: 12px; background: rgba(255, 255, 255, 0.03); + font: inherit; + text-align: left; +} + +/* Unmatched and not scanned open their list. */ +button.anomalous-scan-stat { + cursor: pointer; +} + +button.anomalous-scan-stat:hover { + border-color: rgba(165, 180, 252, 0.4); + background: rgba(255, 255, 255, 0.06); } .anomalous-scan-stat-value { @@ -228,7 +240,165 @@ border-top: 1px solid rgba(255, 255, 255, 0.06); } +/* The scan page's cards: the last scan's result and the model lists. */ +.anomalous-scan-card { + display: flex; + flex-direction: column; + gap: 8px; + padding: 14px 16px; + border: 1px solid rgba(255, 255, 255, 0.08); + border-radius: 12px; + background: rgba(255, 255, 255, 0.02); +} + +.anomalous-scan-card-title { + color: #9ca3af; + font-size: 12.5px; +} + +.anomalous-scan-result-counts { + color: #f3f4f6; + font-size: 15px; + font-weight: 600; +} + +.anomalous-scan-note { + margin: 0; + padding: 8px 10px; + border-radius: 8px; + font-size: 12.5px; + line-height: 1.55; +} + +.anomalous-scan-note.is-warn { + background: rgba(252, 211, 77, 0.08); + color: #fcd34d; +} + +.anomalous-scan-note.is-error { + background: rgba(248, 113, 113, 0.08); + color: #fca5a5; +} + +.anomalous-scan-tabs { + display: flex; + flex-wrap: wrap; + gap: 6px; +} + +.anomalous-scan-tab { + padding: 5px 14px; + border: 1px solid rgba(255, 255, 255, 0.1); + border-radius: 999px; + background: transparent; + color: #9ca3af; + font-size: 13px; + cursor: pointer; +} + +.anomalous-scan-tab[aria-selected="true"] { + border-color: rgba(165, 180, 252, 0.45); + background: rgba(165, 180, 252, 0.12); + color: #e5e7eb; +} + +.anomalous-scan-tab:disabled { + opacity: 0.45; + cursor: default; +} + +.anomalous-scan-rows { + display: flex; + flex-direction: column; +} + +.anomalous-scan-row { + display: flex; + align-items: center; + gap: 10px; + padding: 8px 0; + border-top: 1px solid rgba(255, 255, 255, 0.05); +} + +.anomalous-scan-row-mark { + flex-shrink: 0; + width: 16px; + color: #fcd34d; + text-align: center; +} + +.anomalous-scan-row.is-matched .anomalous-scan-row-mark { color: #86efac; } +.anomalous-scan-row.is-failed .anomalous-scan-row-mark { color: #fca5a5; } +.anomalous-scan-row.is-unchanged .anomalous-scan-row-mark { color: #9ca3af; } + +.anomalous-scan-row-copy { + display: flex; + flex: 1; + flex-direction: column; + gap: 2px; + min-width: 0; +} + +.anomalous-scan-row-name, +.anomalous-scan-row-where, +.anomalous-scan-row-detail { + overflow: hidden; + white-space: nowrap; + text-overflow: ellipsis; +} + +.anomalous-scan-row-name { + color: #e5e7eb; + font-size: 13px; + font-weight: 600; +} + +.anomalous-scan-row-where { + color: #6b7280; + font-size: 11.5px; +} + +.anomalous-scan-row-detail { + color: #9ca3af; + font-size: 12px; +} + +.anomalous-scan-row-actions { + display: flex; + flex-shrink: 0; + gap: 6px; +} + +.anomalous-scan-row-btn, +.anomalous-scan-more { + padding: 3px 10px; + border: 1px solid rgba(255, 255, 255, 0.12); + border-radius: 999px; + background: transparent; + color: #d1d5db; + font-size: 12px; + cursor: pointer; +} + +.anomalous-scan-row-btn:hover:not(:disabled), +.anomalous-scan-more:hover { + border-color: rgba(165, 180, 252, 0.45); + color: #ffffff; +} + +.anomalous-scan-row-btn:disabled { + opacity: 0.45; + cursor: default; +} + +.anomalous-scan-more { + align-self: flex-start; + margin-top: 8px; +} + @media (max-width: 560px) { .anomalous-scan-stats { grid-template-columns: repeat(2, minmax(0, 1fr)); } .anomalous-scan-page { padding: 24px 16px 32px; } + .anomalous-scan-row { flex-wrap: wrap; } + .anomalous-scan-row-actions { width: 100%; padding-left: 26px; } }