DemonGatanjieuandClaude Opus 5.5
e466e9dfcf
The scan page now says which models are unmatched or not scanned yet (type, folder, why Civitai had nothing) and what each scan did, model by model; rows open the model, scan it, look it up again or search Civitai. The progress box shows the counts and, when floating, a way to the result. The activity log records a scan when it ends with its result, and model edits (from -> to) and deletions (files moved) in detail. - scraper.py reports one JSON line per changed model (--report-file); a failing model no longer ends the scan. Unmatched info keeps why (not_found / network / offline); online scans look network/offline ones up again. After two models in a row without an answer the rest of the scan infers from the files (--civitai-down for the next folders) instead of waiting out every timeout. - civitai_client.py: the network side moved out of scraper.py; tells a 404 from Civitai being unreachable (CivitaiUnreachable). - api/scan_report.py: ScanJob turns a scan's reports into one result, kept as the last scan (/anomalous/last_scan) and logged; /anomalous/scan_model opens a listed model. - /anomalous/scan_all takes picked or listed models as `targets`, so one scan has one result; scan_summary lists unmatched and new models with their reason. - Activity middleware takes details a handler attaches (request[ACTIVITY_DETAIL]). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
61 lines
2.4 KiB
Python
61 lines
2.4 KiB
Python
"""What the scan page shows before a scan: how many models are matched on Civitai, unmatched or
|
|
new, and which ones the last two are (located like the scan report's models, so the page can
|
|
open them)."""
|
|
|
|
import asyncio
|
|
import os
|
|
|
|
from aiohttp import web
|
|
|
|
from .folder_types import get_active_scan_paths
|
|
from .scan_report import locate_root
|
|
|
|
try:
|
|
from ..model_identity import is_unmatched, sidecar_info, unmatched_reason
|
|
except ImportError: # loaded outside the package (tests)
|
|
from model_identity import is_unmatched, sidecar_info, unmatched_reason
|
|
|
|
MAX_LISTED = 2000 # models listed per kind; the counts stay exact
|
|
|
|
|
|
def _summarize():
|
|
"""Counts the files the scanner handles (.safetensors in the active scan folders), each once.
|
|
`pending`: unmatched models Civitai was never asked about (offline, or unreachable then),
|
|
which the next online scan looks up."""
|
|
seen = set()
|
|
total = matched = 0
|
|
unmatched, new = [], []
|
|
for base_dir in get_active_scan_paths():
|
|
located = locate_root(base_dir) or ("", 0)
|
|
for root, _dirs, files in os.walk(base_dir):
|
|
for filename in files:
|
|
if not filename.endswith(".safetensors"):
|
|
continue
|
|
path = os.path.join(root, filename)
|
|
real = os.path.realpath(path)
|
|
if real in seen:
|
|
continue
|
|
seen.add(real)
|
|
total += 1
|
|
info = sidecar_info(os.path.splitext(path)[0])
|
|
if info is not None and not is_unmatched(info):
|
|
matched += 1
|
|
continue
|
|
item = {"type": located[0], "path_idx": located[1], "filename": filename,
|
|
"rel": os.path.relpath(path, base_dir).replace(os.sep, "/")}
|
|
if info is None:
|
|
new.append(item)
|
|
else:
|
|
item.update(reason=unmatched_reason(info), base=info.get("baseModel") or "")
|
|
unmatched.append(item)
|
|
return {
|
|
"total": total, "matched": matched, "unmatched": len(unmatched), "new": len(new),
|
|
"pending": sum(item["reason"] in ("offline", "network") for item in unmatched),
|
|
"unmatched_models": unmatched[:MAX_LISTED], "new_models": new[:MAX_LISTED],
|
|
}
|
|
|
|
|
|
async def api_scan_summary(request):
|
|
"""GET /anomalous/scan_summary - counts, and the unmatched and new models."""
|
|
return web.json_response(await asyncio.to_thread(_summarize))
|