diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index fda0e02..d0a5170 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -113,6 +113,12 @@ covered by the `styles.css` manifest. User guide: `docs/guides/mcp.md`. - `api/lora_info.py` reads a LoRA's safetensors header to tell whether it trains the text encoder (`/anomalous/lora_info`), so inserting it wires the CLIP line only then. +- `api/model_download.py` downloads the models a workflow misses (Model Check's + "Download", `/anomalous/download/*`): one at a time into `.part`, continued after + an interruption, checked against the SHA-256 before it gets its name, never over an + existing file; also where downloads go by default (`download_settings.json`). + `api/download_sources.py` finds the one exact file: Civitai by hash, a Civitai version + link, or a Hugging Face / GitHub file link the workflow carries. - `api/workflow_schema.py`, `api/recipe_schema.py`, `api/recipe_images.py`, and `api/recipe_store.py` own recipe validation/shaping, images, CRUD, history, and integrity receipts; `api/recipes.py` is the HTTP facade. `api/recipe_constants.py` @@ -470,7 +476,12 @@ covered by the `styles.css` manifest. say "doctor") and puts a found file into its node; `ui_doctor.js` is the Model Check page and `ui_doctor_banner.js` the bar over the canvas when an opened workflow misses models (it also owns the check-on-open preference that Settings → Workflows and the - bar's "Don't show again" set). `ui_node_assistant.js` owns the current-node panel (model actions, LoRA + bar's "Don't show again" set). `model_download.js` finds a missing model's source, + decides where its file goes (the download settings, `{base}` = its base model's folder), + follows the downloads and puts each finished file into its nodes (the placement rules + are `download_places.js`, without imports so they are tested alone); `ui_model_download.js` + is the page's Download buttons, progress and the dialog that shows and changes where + each file goes. `ui_node_assistant.js` owns the current-node panel (model actions, LoRA insertion, model cards and history), `ui_node_model_picker.js` owns native combo replacement, `ui_node_parameters.js` renders the panel's parameters section, and `node_parameter_sets.js` merges the node type's saved values (material files and recipe diff --git a/CHANGELOG.md b/CHANGELOG.md index b56e917..800efc6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,14 +52,22 @@ - **Model folders, readable (模型文件夹设置更好懂)**: folder management is a view of the settings page instead of a pop-up. Grouping is two plain choices, **By model type** or **By folders on disk** (it was "Category mode / Physical mode"); the types shown on the Models page come first, each with its usual name (Checkpoint, Diffusion…) and folder name, a switch and ↑↓ to order them; the ones that are off follow, and the many folders ComfyUI registers that few people use (configs, datasets…) fold away. Each change applies at once; there is no Save and reload button any more. - The output gallery shows 512 px thumbnails instead of full images (cached like model cards), so scrolling through hundreds of outputs no longer fills the browser's memory and graphics memory with full-size pictures. Dragging an image to the canvas and the large viewer still use the original. When the browser has been closed for 90 seconds, its gallery images are released too, like the model grid. +### ⬇️ Download missing models from Model Check (模型检查一键下载缺失模型) +- **Download** next to a missing model, and **Download all** for every one, when its source is certain: Civitai found the very file by the workflow's fingerprint, or the workflow carries a link to the file (a shared workflow's model links, or the download links in ComfyUI's own templates on Hugging Face). A file name alone is never enough, and a link to a different file than the workflow's is not offered. +- **You see where it goes before it starts**: the dialog shows, per file, its size, source and base model, the full path it will be saved to, and the free disk space. The models folder (when you have several, e.g. through extra_model_paths) and the subfolder can be changed right there; the folder you pick is remembered per type. +- **Your folder habits (按你的文件夹习惯放)**: Settings → Model downloads sets the default. *Where the workflow says* (default) puts the file in the subfolder the workflow names, so nothing in the workflow changes; a model without one goes into the folder set below (`Downloads`). *A set folder* always uses that folder, and the node is moved to the new place (Ctrl+Z undoes it). Write `{base}` in the folder to sort by base model (SDXL, Flux, Illustrious…); a folder of that name you already have is used, in whatever spelling. +- **Safe by default**: it downloads into a temporary file and checks the fingerprint before giving it its name, so a broken or wrong download never shows up in your models; an existing file is never replaced; an interrupted download continues where it stopped. `.ckpt` / `.pt` files get a warning (they can run code), and a source without a fingerprint says it cannot be checked. Your Civitai API key (needed for some models; Settings → Model downloads, or the scan page) is only sent to Civitai. +- Progress shows in the row with Cancel; when a file is done, the nodes using it get it, a short message says so, the new files are scanned for their information and cover, and the activity log records the download (source, size, whether the fingerprint matched). +- **Ctrl+Z after a scan (扫描后的 Ctrl+Z)**: after any scan, the first Ctrl+Z used to seem to do nothing: the workflow's saved fingerprints had changed, which ComfyUI counted as a step. A change of fingerprints alone no longer takes a step. + ### 🧩 Inserting a LoRA wires it properly (插入 LoRA 连线更聪明) - **Current node → Insert LoRA before / after** now works on the usual workflows it refused before: a checkpoint whose CLIP feeds both prompts (both now hang behind the LoRA), and UNet / Flux / Anima workflows with their own CLIP loader. Whether the CLIP line goes through the LoRA is decided by the LoRA file itself: one that also trains the text encoder (most SD 1.5 / SDXL / Illustrious LoRAs) takes it, from the checkpoint or the CLIP loader; one that does not (most Flux and Anima LoRAs) is inserted as a model-only LoRA loader. Inserting before a sampler moves only that sampler's model, and the CLIP of the prompts wired into it. - + ### 🤖 AI apps can use your library (MCP，让 AI 用你的库) - While ComfyUI runs, AI apps that speak MCP (Claude Code, Claude Desktop, Cursor, Cherry Studio, VS Code…) can connect to `http://127.0.0.1:8188/anomalous/mcp` and ask about your library: models with their Civitai information, trigger words and your notes, which models are unmatched and why, output images and the settings that made them, combos, workflow recipes, saved prompts, generated speech and the activity log. A model's cover or an image's thumbnail can be attached for apps that look at pictures. - It can also act on the workflow open in ComfyUI: rewrite or extend the prompts, swap a node's model, add a LoRA (it goes after the model loader and the LoRAs already there; one that also trains the text encoder takes the CLIP line too, from the checkpoint or, in a UNet workflow, from its CLIP loader, and everything those fed, such as both prompts, moves behind it; one that does not, like most Flux and Anima LoRAs, patches the model only), place one of your combos, check and put back missing models, and run the workflow; and start scans, or have an Anomalous TTS voice read a line. Every canvas change is one Ctrl+Z step, shows a note on the page and is in the activity log marked "AI". Nothing deletes or renames files; AI apps ask before calling a tool that changes something. - Only this computer can connect, and requests from web pages, other hosts or through a proxy are refused. It costs no memory while nobody is connected. Setup for each app: `docs/guides/mcp.md`. - + ### 🕘 Activity log (操作记录) - **See what Anomalous changed (看 Anomalous 改了什么)**: the new **Activity** page on the rail lists, by day, every change the plugin made: on the canvas, which node settings changed (from → to), which nodes were added or removed, and which workflow was opened; in files, models, covers, recipes, materials, notes, presets, images, audio and imported characters. Filter by canvas or files; open a canvas entry for its details and **Find on canvas** to jump to the node. Your own edits on the canvas are not recorded as the plugin's. The home page shows the latest few. - **More detail (记录更详细)**: a scan is recorded when it ends, with what it found and each model it changed (open one from the entry), instead of a bare "scanned models" when it started. Editing a model records each field from → to, and deleting one lists the files moved to the Recycle Bin. diff --git a/api/__init__.py b/api/__init__.py index b20f197..dd5898c 100644 --- a/api/__init__.py +++ b/api/__init__.py @@ -1,7 +1,7 @@ from . import ( activity_log, audio_catalog, folder_types, gallery_routes, materials, media_routes, model_catalog, model_media, model_metadata, model_resolution, model_type_listing, kept_images, node_material, recipe_packages, recipes, scan_report, scan_summary, - lora_info, mcp_server, translation_routes, version_manager, + lora_info, mcp_server, model_download, translation_routes, version_manager, ) from .scanner import * from .config import * @@ -108,7 +108,7 @@ def setup_routes(app): app.router.add_post('/anomalous/resolve_paths_to_previews', model_catalog.api_resolve_paths_to_previews) app.router.add_get('/anomalous/scan_summary', scan_summary.api_scan_summary) app.router.add_get('/anomalous/last_scan', scan_report.api_last_scan) - app.router.add_get('/anomalous/scan_model', scan_report.api_scan_model) + app.router.add_get('/anomalous/scan_model', scan_report.api_scan_model) app.router.add_get('/anomalous/lora_info', lora_info.api_lora_info) # Audio & Voice Studio Routes @@ -118,5 +118,6 @@ def setup_routes(app): app.router.add_get('/anomalous/audio_previews', audio_catalog.api_get_audio_previews) app.router.add_post('/anomalous/save_audio_preview', audio_catalog.api_save_audio_preview) version_manager.register_routes(app) - activity_log.register_routes(app) + activity_log.register_routes(app) mcp_server.register_routes(app) + model_download.register_routes(app) diff --git a/api/download_sources.py b/api/download_sources.py new file mode 100644 index 0000000..20d4e6d --- /dev/null +++ b/api/download_sources.py @@ -0,0 +1,237 @@ +"""Where a model the open workflow is missing can be downloaded from, for Model Check's +"Download" (api/model_download.py fetches it). + +Only sources that name one exact file count: Civitai by the file's hash (the workflow's +fingerprint), a Civitai model version or file link, or a direct file link on Hugging Face +(or its mirror) or GitHub, as a shared workflow carries them (AMB's model links, ComfyUI's own +`properties.models`). A file name alone is never a source. Each answer says the file's size +and SHA-256 when the site gives them, so the download can be checked. +""" + +import json +import re +import urllib.error +import urllib.parse +import urllib.request + +try: + from ..civitai_client import USER_AGENT, _load_api_key +except ImportError: # loaded outside the package (tests) + from civitai_client import USER_AGENT, _load_api_key + +CIVITAI_HOSTS = ("civitai.com", "civitai.red") +HF_HOSTS = ("huggingface.co", "hf-mirror.com") +OTHER_HOSTS = ("github.com",) +TIMEOUT = 15 +SHA256 = re.compile(r"^[0-9a-fA-F]{64}$") + + +class SourceUnreachable(Exception): + """The site did not answer (network, timeout, server error).""" + + +def host_of(url): + try: + parsed = urllib.parse.urlsplit(str(url).strip()) + except ValueError: + return "" + return (parsed.hostname or "").lower() if parsed.scheme == "https" else "" + + +def _site(host, sites): + return any(host == site or host.endswith("." + site) for site in sites) + + +def allowed_download_url(url): + """Whether `url` is an https link on a site downloads may start from.""" + host = host_of(url) + return bool(host) and _site(host, CIVITAI_HOSTS + HF_HOSTS + OTHER_HOSTS) + + +def is_civitai(url): + return _site(host_of(url), CIVITAI_HOSTS) + + +def civitai_headers(): + headers = {"User-Agent": USER_AGENT} + key = _load_api_key() + if key: + headers["Authorization"] = f"Bearer {key}" + return headers + + +def _get_json(url, headers): + """The JSON at `url`; None on 404; SourceUnreachable when it could not be asked.""" + try: + with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=TIMEOUT) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + if error.code in (400, 404): + return None + raise SourceUnreachable(f"HTTP {error.code}") from error + except (urllib.error.URLError, OSError, ValueError) as error: + raise SourceUnreachable(str(error)) from error + + +def _civitai_page(version): + domain = "civitai.red" if (version.get("model") or {}).get("nsfw") or (version.get("nsfwLevel") or 1) > 1 else "civitai.com" + model_id = version.get("modelId") + return f"https://{domain}/models/{model_id}?modelVersionId={version.get('id')}" if model_id else "" + + +def _pick_civitai_file(files, file_hash="", name=""): + """The version's file with this hash (SHA-256 or a prefix such as AutoV2), else this name; None if neither.""" + files = [item for item in files or [] if isinstance(item, dict) and item.get("downloadUrl")] + wanted = str(file_hash or "").upper() + if wanted: + for item in files: + hashes = {str(value).upper() for value in (item.get("hashes") or {}).values()} + sha = str((item.get("hashes") or {}).get("SHA256") or "").upper() + if wanted in hashes or (len(wanted) >= 10 and sha.startswith(wanted)): + return item + return None + if name: + for item in files: + if str(item.get("name", "")).lower() == name.lower(): + return item + return None + + +def _from_civitai_version(version, file_hash="", name=""): + picked = _pick_civitai_file(version.get("files"), file_hash, name) + if not picked: + return None + sha = str((picked.get("hashes") or {}).get("SHA256") or "") + return { + "found": True, + "source": "civitai", + "download_url": picked["downloadUrl"], + "file_name": picked.get("name") or "", + "size": int(float(picked.get("sizeKB") or 0) * 1024), + "sha256": sha.lower() if SHA256.match(sha) else "", + "format": ((picked.get("metadata") or {}).get("format") or ""), + "base_model": version.get("baseModel") or "", + "model_name": (version.get("model") or {}).get("name") or "", + "version_name": version.get("name") or "", + "page": _civitai_page(version), + } + + +def civitai_by_hash(file_hash): + version = _get_json(f"https://civitai.com/api/v1/model-versions/by-hash/{urllib.parse.quote(file_hash)}", civitai_headers()) + return _from_civitai_version(version, file_hash) if version else None + + +def _civitai_version_id(url): + parsed = urllib.parse.urlsplit(url) + query = urllib.parse.parse_qs(parsed.query) + if query.get("modelVersionId"): + return query["modelVersionId"][0] + match = re.search(r"/api/download/models/(\d+)", parsed.path) or re.search(r"/model-versions/(\d+)", parsed.path) + return match.group(1) if match else "" + + +def civitai_by_link(url, name, file_hash=""): + """A Civitai link that names a model version: its file of this hash or name.""" + version_id = _civitai_version_id(url) + if not version_id.isdigit(): + return None + version = _get_json(f"https://civitai.com/api/v1/model-versions/{version_id}", civitai_headers()) + return _from_civitai_version(version, file_hash, name) if version else None + + +class _NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): + return None + + +def _hf_file_url(url): + """https:////resolve// for a blob or resolve link to a file, else ''.""" + parsed = urllib.parse.urlsplit(url) + parts = [part for part in parsed.path.split("/") if part] + for marker in ("resolve", "blob"): + if marker in parts: + at = parts.index(marker) + if at >= 2 and len(parts) > at + 2: + parts[at] = "resolve" + return urllib.parse.urlunsplit(("https", parsed.netloc, "/" + "/".join(parts), "", "")) + return "" + + +def hugging_face_file(url): + """A Hugging Face file link: its size and SHA-256 from the site's headers.""" + file_url = _hf_file_url(url) + if not file_url: + return None + opener = urllib.request.build_opener(_NoRedirect) + request = urllib.request.Request(file_url, method="HEAD", headers={"User-Agent": USER_AGENT}) + try: + response = opener.open(request, timeout=TIMEOUT) + headers, redirected = response.headers, False + except urllib.error.HTTPError as error: + if error.code in (401, 403): + return {"found": False, "reason": "gated", "page": url} + if error.code == 404: + return None + if error.code not in (301, 302, 303, 307, 308): + raise SourceUnreachable(f"HTTP {error.code}") from error + headers, redirected = error.headers, True + except (urllib.error.URLError, OSError) as error: + raise SourceUnreachable(str(error)) from error + etag = str(headers.get("x-linked-etag") or headers.get("etag") or "").strip('"').replace("W/", "").strip('"') + # A redirect's own length is not the file's: without x-linked-size (small files kept + # outside large-file storage) the size is unknown. + size = headers.get("x-linked-size") or (0 if redirected else headers.get("content-length")) or 0 + return { + "found": True, + "source": "huggingface", + "download_url": file_url, + "file_name": urllib.parse.unquote(file_url.rsplit("/", 1)[-1]), + "size": int(size) if str(size).isdigit() else 0, + "sha256": etag.lower() if SHA256.match(etag) else "", + "format": "", + "base_model": "", + "model_name": "", + "version_name": "", + "page": url, + } + + +def plain_link(url): + """A direct file link elsewhere (a GitHub release asset): no size or hash to check against.""" + path = urllib.parse.urlsplit(url).path + if "/releases/download/" not in path: + return None + return { + "found": True, "source": "link", "download_url": url, + "file_name": urllib.parse.unquote(path.rsplit("/", 1)[-1]), "size": 0, "sha256": "", + "format": "", "base_model": "", "model_name": "", "version_name": "", "page": url, + } + + +def find_source(file_hash="", url="", name=""): + """Where one missing model can come from: {found, source, download_url, file_name, size, + sha256, base_model, model_name, version_name, page}, or {found: False, reason}.""" + looked = False + try: + if file_hash: + looked = True + found = civitai_by_hash(file_hash) + if found: + return found + if url and host_of(url): + looked = True + host = host_of(url) + if _site(host, CIVITAI_HOSTS): + found = civitai_by_link(url, name, file_hash) + elif _site(host, HF_HOSTS): + found = hugging_face_file(url) + elif _site(host, OTHER_HOSTS): + found = plain_link(url) + else: + found = None + if found: + return found + except SourceUnreachable as error: + return {"found": False, "reason": "network", "error": str(error)} + return {"found": False, "reason": "not_found" if looked else "no_source"} diff --git a/api/model_download.py b/api/model_download.py new file mode 100644 index 0000000..6e61e02 --- /dev/null +++ b/api/model_download.py @@ -0,0 +1,424 @@ +"""Downloading the models the open workflow is missing (Model Check's "Download"). + +POST /anomalous/download/lookup {items: [{key, hash, url, value, types}]} -> where each can come + from (download_sources.find_source) and the model folders of its type: [{index, path, free, + subfolders}]. +POST /anomalous/download/start {download_url, type, root, rel, sha256, size, ...} -> {id} +GET /anomalous/download/status -> {jobs} +POST /anomalous/download/cancel {id} +GET/POST /anomalous/download/settings -> where downloads go ({place, folder}), kept in + user/anomalous/download_settings.json. + +One download runs at a time, into ".part" next to where the file goes; an interrupted one +continues from there. The finished file is checked against the SHA-256 the workflow or the site +gives, and only then gets its name; a file that is already there is never replaced. The Civitai +key goes only to Civitai (not along a redirect to its file storage). +""" + +import asyncio +import hashlib +import json +import os +import shutil +import threading +import time +import urllib.error +import urllib.request +import uuid + +from aiohttp import web +import folder_paths + +from .activity_log import add_entry, log_path +from .download_sources import SHA256, USER_AGENT, allowed_download_url, find_source, is_civitai, _load_api_key +from .path_utils import atomic_write_json, resolve_within + +MODEL_EXTENSIONS = (".safetensors", ".ckpt", ".pt", ".pth", ".bin", ".sft", ".gguf") +CHUNK = 1024 * 1024 +SPACE_MARGIN = 200 * 1024 * 1024 +MAX_SUBFOLDERS = 300 +KEEP_FINISHED = 50 +DEFAULT_SETTINGS = {"place": "workflow", "folder": "Downloads"} + +_lock = threading.Lock() +_jobs = {} # id -> job dict, in the order started +_queue = [] +_worker = None + + +class DownloadError(Exception): + def __init__(self, code, message=""): + super().__init__(message or code) + self.code = code + + +async def _body(request): + """The request's JSON object, or {} when it is not one (each route then refuses it).""" + try: + data = await request.json() + except ValueError: + return {} + return data if isinstance(data, dict) else {} + + +# --- settings --------------------------------------------------------------------------- + +def settings_path(): + return os.path.join(os.path.dirname(log_path()), "download_settings.json") + + +def read_settings(): + settings = dict(DEFAULT_SETTINGS) + try: + with open(settings_path(), encoding="utf-8") as source: + saved = json.load(source) + if saved.get("place") in ("workflow", "folder"): + settings["place"] = saved["place"] + if isinstance(saved.get("folder"), str): + settings["folder"] = saved["folder"] + except (OSError, ValueError, AttributeError): + pass + return settings + + +def clean_folder(folder): + """A relative folder ('' = the type's own folder), '/' separated; ValueError if it climbs out.""" + parts = [part.strip() for part in str(folder or "").replace("\\", "/").split("/") if part.strip()] + for part in parts: + if part in (".", "..") or ":" in part or any(char in part for char in '<>"|?*'): + raise ValueError("Invalid folder") + return "/".join(parts) + + +async def api_settings(request): + if request.method == "POST": + data = await _body(request) + settings = read_settings() + if data.get("place") in ("workflow", "folder"): + settings["place"] = data["place"] + if "folder" in data: + try: + # {base} is filled with the model's base model when a download starts. + settings["folder"] = clean_folder(data.get("folder")) + except ValueError: + return web.json_response({"error": "Invalid folder"}, status=400) + atomic_write_json(settings_path(), settings) + return web.json_response(settings) + return web.json_response(read_settings()) + + +# --- lookup ----------------------------------------------------------------------------- + +def folder_type_for(types): + known = getattr(folder_paths, "folder_names_and_paths", {}) + for name in types or []: + if isinstance(name, str) and name in known: + return name + return "" + + +def _subfolders(root): + found = [] + for current, dirs, _files in os.walk(root): + dirs[:] = sorted(name for name in dirs if not name.startswith(".")) + depth = os.path.relpath(current, root).count(os.sep) + for name in dirs: + found.append(os.path.relpath(os.path.join(current, name), root).replace(os.sep, "/")) + if len(found) >= MAX_SUBFOLDERS: + return found + if depth >= 1: + dirs[:] = [] + return found + + +def roots_of(folder_type): + """The model folders of a type: [{index, path, exists, free, subfolders}].""" + roots = [] + for index, path in enumerate(folder_paths.get_folder_paths(folder_type) or []): + exists = os.path.isdir(path) + probe = path if exists else os.path.dirname(path) + try: + free = shutil.disk_usage(probe).free if os.path.isdir(probe) else 0 + except OSError: + free = 0 + roots.append({"index": index, "path": path, "exists": exists, "free": free, + "subfolders": _subfolders(path) if exists else []}) + return roots + + +def same_file(expected, sha256): + """Whether a fingerprint (full SHA-256 or a prefix of 10+, AutoV2) names this SHA-256.""" + expected, sha256 = str(expected or "").lower(), str(sha256 or "").lower() + return len(expected) >= 10 and sha256.startswith(expected) + + +def lookup_item(item): + file_hash = str(item.get("hash") or "").strip() + name = os.path.basename(str(item.get("value") or "").replace("\\", "/")) + found = find_source(file_hash, str(item.get("url") or "").strip(), name) + if found.get("found") and file_hash and found.get("sha256") and len(file_hash) >= 10 \ + and all(c in "0123456789abcdefABCDEF" for c in file_hash) and not same_file(file_hash, found["sha256"]): + # The link names another file than the one the workflow was saved with. + return {"found": False, "reason": "different_file", "page": found.get("page", "")} + return found + + +async def api_lookup(request): + """Items already looked up (`known`) or with nothing to look up by only get their type and + folders. Once a site does not answer, the rest are not asked (offline: no wait per model).""" + items = (await _body(request)).get("items") + if not isinstance(items, list): + return web.json_response({"error": "items must be a list"}, status=400) + results, types, offline = [], {}, False + for item in items[:100]: + if not isinstance(item, dict): + continue + folder_type = folder_type_for(item.get("types")) + if item.get("known"): + result = {"found": False, "reason": "known"} + elif not (item.get("hash") or item.get("url")): + result = {"found": False, "reason": "no_source"} + elif offline: + result = {"found": False, "reason": "network"} + else: + result = await asyncio.to_thread(lookup_item, item) + offline = result.get("reason") == "network" + result.update(key=item.get("key"), type=folder_type) + if folder_type and folder_type not in types: + types[folder_type] = await asyncio.to_thread(roots_of, folder_type) + results.append(result) + return web.json_response({"results": results, "roots": types, "settings": read_settings()}) + + +# --- downloads -------------------------------------------------------------------------- + +def _public(job): + return {key: job[key] for key in ("id", "key", "state", "error", "received", "total", "rel", "type", + "root", "value", "file", "source", "verified")} + + +def plan_job(data): + """A queued job from a start request, or ValueError / DownloadError saying why not.""" + url = str(data.get("download_url") or "") + if not allowed_download_url(url): + raise DownloadError("bad_source") + folder_type = folder_type_for([data.get("type")]) + if not folder_type: + raise DownloadError("bad_type") + paths = folder_paths.get_folder_paths(folder_type) or [] + root_index = data.get("root", 0) + if not isinstance(root_index, int) or not 0 <= root_index < len(paths): + raise DownloadError("bad_root") + rel = clean_folder(data.get("rel")) + if not rel.lower().endswith(MODEL_EXTENSIONS): + raise DownloadError("bad_name") + root = os.path.realpath(paths[root_index]) + dest = resolve_within(root, *rel.split("/")) + if os.path.exists(dest): + raise DownloadError("exists") + size = int(data.get("size") or 0) + probe = root + while probe and not os.path.isdir(probe): + probe = os.path.dirname(probe) + if size and probe and shutil.disk_usage(probe).free < size + SPACE_MARGIN: + raise DownloadError("no_space") + sha256 = str(data.get("sha256") or "").lower() + with _lock: + if any(job["dest"] == dest and job["state"] in ("queued", "running", "verifying") for job in _jobs.values()): + raise DownloadError("already") + return { + "id": uuid.uuid4().hex[:12], "key": str(data.get("key") or ""), "state": "queued", "error": "", + "received": 0, "total": size, "rel": rel, "type": folder_type, "root": root_index, + "value": str(data.get("value") or ""), "file": os.path.basename(dest), "dest": dest, "url": url, + "sha256": sha256 if SHA256.match(sha256) else "", "expected": str(data.get("hash") or "").lower(), + "source": str(data.get("source") or ""), "page": str(data.get("page") or ""), + "verified": False, "cancel": False, "finished": 0, + } + + +def _open(job, start): + headers = {"User-Agent": USER_AGENT} + if start: + headers["Range"] = f"bytes={start}-" + request = urllib.request.Request(job["url"], headers=headers) + key = _load_api_key() if is_civitai(job["url"]) else None + if key: # not carried along the redirect to the file storage + request.add_unredirected_header("Authorization", f"Bearer {key}") + try: + return urllib.request.urlopen(request, timeout=60) + except urllib.error.HTTPError as error: + if error.code in (401, 403): + raise DownloadError("needs_key" if is_civitai(job["url"]) and not key else "forbidden") from error + if error.code == 404: + raise DownloadError("gone") from error + if error.code == 416 and start: + raise DownloadError("restart") from error + raise DownloadError("network", f"HTTP {error.code}") from error + except (urllib.error.URLError, OSError) as error: + raise DownloadError("network", str(error)) from error + + +def _hash_existing(path, digest): + with open(path, "rb") as source: + for chunk in iter(lambda: source.read(CHUNK), b""): + digest.update(chunk) + + +def _fetch(job): + part = job["dest"] + ".part" + os.makedirs(os.path.dirname(job["dest"]), exist_ok=True) + digest = hashlib.sha256() + start = os.path.getsize(part) if os.path.isfile(part) else 0 + try: + response = _open(job, start) + except DownloadError as error: + if error.code != "restart": + raise + start, response = 0, _open(job, 0) + with response: + if start and response.status != 206: + start = 0 # the site sends the whole file again + if "text/html" in str(response.headers.get("Content-Type", "")).lower(): + raise DownloadError("not_a_file") # a login or error page, not the model + if start: + _hash_existing(part, digest) + length = int(response.headers.get("Content-Length") or 0) + job["total"] = start + length if length else job["total"] + job["received"] = start + with open(part, "ab" if start else "wb") as target: + while True: + if job["cancel"]: + raise DownloadError("cancelled") + chunk = response.read(CHUNK) + if not chunk: + break + target.write(chunk) + digest.update(chunk) + job["received"] += len(chunk) + return part, digest.hexdigest() + + +def _finish(job, part, sha256): + job["state"] = "verifying" + if job["total"] and job["received"] != job["total"]: + raise DownloadError("network", "incomplete") + expected = job["sha256"] or (job["expected"] if len(job["expected"]) >= 10 else "") + if expected and not same_file(expected, sha256): + os.remove(part) # our own unfinished download, not a file of the user's + raise DownloadError("hash_mismatch") + job["verified"] = bool(expected) + if os.path.exists(job["dest"]): + raise DownloadError("exists") + os.replace(part, job["dest"]) + # ComfyUI's file lists and the browser's model metadata see the new file. + for cache in (getattr(folder_paths, "filename_list_cache", None), getattr(folder_paths, "cache_helper", None)): + try: + cache.clear() + except Exception: + pass + try: + from .metadata import clear_metadata_cache + clear_metadata_cache() + except Exception as error: + print(f"[Anomalous Browser] Download: could not clear the model cache: {error}") + + +def _record(job, sha256): + add_entry("file", "model_download", job["rel"], {"download": { + "type": job["type"], "path_idx": job["root"], "rel": job["rel"], "file": job["file"], + "size": job["received"], "sha256": sha256, "verified": job["verified"], + "source": job["source"], "page": job["page"], "value": job["value"], + }}) + + +def run_job(job): + job["state"] = "running" + try: + part, sha256 = _fetch(job) + _finish(job, part, sha256) + job["state"] = "done" + try: + _record(job, sha256) + except Exception as error: + print(f"[Anomalous Browser] Download: could not record it in the activity log: {error}") + except DownloadError as error: + job["state"] = "cancelled" if error.code == "cancelled" else "failed" + job["error"] = error.code + if error.code == "cancelled" and os.path.isfile(job["dest"] + ".part"): + os.remove(job["dest"] + ".part") # our own unfinished download + if error.code not in ("cancelled",): + print(f"[Anomalous Browser] Download of {job['rel']} failed: {error}") + except Exception as error: + job["state"], job["error"] = "failed", "unknown" + print(f"[Anomalous Browser] Download of {job['rel']} failed: {error}") + job["finished"] = time.time() + + +def _work(): + global _worker + while True: + with _lock: + job = next((item for item in _queue if item["state"] == "queued"), None) + if job is None: + _queue.clear() + _worker = None + return + _queue.remove(job) + run_job(job) + + +def enqueue(job): + global _worker + with _lock: + _jobs[job["id"]] = job + _queue.append(job) + finished = [item for item in _jobs.values() if item["finished"]] + for old in finished[:-KEEP_FINISHED]: + _jobs.pop(old["id"], None) + if _worker is None: + _worker = threading.Thread(target=_work, daemon=True, name="anomalous-download") + _worker.start() + + +async def api_start(request): + data = await _body(request) + if not data: + return web.json_response({"error": "bad_request"}, status=400) + try: + job = await asyncio.to_thread(plan_job, data) + except DownloadError as error: + return web.json_response({"error": error.code}, status=409 if error.code in ("exists", "already") else 400) + except (ValueError, TypeError) as error: + return web.json_response({"error": "bad_path", "message": str(error)}, status=400) + enqueue(job) + return web.json_response(_public(job)) + + +async def api_status(request): + with _lock: + jobs = [_public(job) for job in _jobs.values()] + return web.json_response({"jobs": jobs}) + + +async def api_cancel(request): + data = await _body(request) + with _lock: + job = _jobs.get(str(data.get("id") or "")) + if not job: + return web.json_response({"error": "unknown"}, status=404) + if job["state"] == "queued": + job["state"], job["error"], job["finished"] = "cancelled", "cancelled", time.time() + if job in _queue: + _queue.remove(job) + else: + job["cancel"] = True + return web.json_response(_public(job)) + + +def register_routes(app): + app.router.add_post("/anomalous/download/lookup", api_lookup) + app.router.add_post("/anomalous/download/start", api_start) + app.router.add_get("/anomalous/download/status", api_status) + app.router.add_post("/anomalous/download/cancel", api_cancel) + app.router.add_get("/anomalous/download/settings", api_settings) + app.router.add_post("/anomalous/download/settings", api_settings) diff --git a/docs/architecture/backend.md b/docs/architecture/backend.md index 59134a8..4600346 100644 --- a/docs/architecture/backend.md +++ b/docs/architecture/backend.md @@ -177,6 +177,32 @@ server's own `/anomalous/scan_all`, `/anomalous_tts/characters`, `/prompt` and `/history` over loopback, so they run exactly as from the UI (speech waits up to two minutes for the file). Canvas changes are posted to the activity log with `via: "mcp"`. +## Model downloads + +`api/model_download.py` serves Model Check's downloads. `POST /anomalous/download/lookup` +asks `api/download_sources.py` where each missing model comes from: Civitai by the +workflow's hash (the version's file with that SHA-256 or AutoV2), else a link the +workflow carries — a Civitai link naming a model version, a Hugging Face or hf-mirror +file link (size and SHA-256 from the site's headers), a GitHub release asset. A file +name alone is never a source, and a link whose file has another SHA-256 than the +workflow's hash is refused (`different_file`). Once a site does not answer, the rest of +the lookup is skipped (`network`). It also returns each type's model folders with free +space and subfolders (two levels). + +`POST /anomalous/download/start` takes the source, type, root index and a relative path +and refuses: a non-https link or one off Civitai, Hugging Face, hf-mirror or GitHub; a +path leaving the root (`resolve_within`) or not ending in a model extension; a file that +exists; not enough space. Jobs run one at a time in a worker thread into +`.part`; a new start for the same file continues it with a Range request. The +Civitai key is sent to Civitai only, as an unredirected header, so it never reaches the +file storage the download redirects to. An HTML answer (a login page) fails as +`not_a_file`, 401/403 as `needs_key` or `forbidden`. The finished file must match the +source's SHA-256 (else the workflow's hash) or it is removed as `hash_mismatch`; then it +is renamed into place, ComfyUI's file-list caches are cleared and the activity log gets +a `model_download` entry. `GET /anomalous/download/status` lists the jobs of this run, +`POST /anomalous/download/cancel` stops one (its `.part` goes). The default place is +`user/anomalous/download_settings.json` (`{place: workflow | folder, folder}`). + ## Metadata and cache behavior The output gallery keeps one ordered directory snapshot for at most ten seconds diff --git a/docs/architecture/frontend.md b/docs/architecture/frontend.md index 8064c51..220b47d 100644 --- a/docs/architecture/frontend.md +++ b/docs/architecture/frontend.md @@ -86,7 +86,15 @@ The main surfaces are: missing-model recovery. `model_check.js` decides each workflow model's state and applies fixes; `ui_doctor.js` (the page) and `ui_doctor_banner.js` (the bar shown after a workflow opens, unless its "Don't show again" or Settings → Workflows turned the - check on opening off) only render it and call it on a press. + check on opening off) only render it and call it on a press. A missing model with a + certain source gets Download (`ui_model_download.js`; `model_download.js` holds the + logic): the dialog names, per file, the type's folder (a choice when there are several, + remembered per type in `localStorage`) and the subfolder from Settings → Model downloads + — where the workflow says (so the node needs no change) or a set folder, `{base}` there + being the model's base model, matched to an existing folder in any spelling — and the + file keeps the workflow's name. When a download ends, the model lists reload, nodes + still naming that model get the file (one Ctrl+Z step each), and once nothing is + downloading the new files are scanned in one targeted scan. `ui_node_assistant.js` owns the current-node panel and model history, `ui_node_model_picker.js` owns the native-widget model replacer, and `ui_node_parameters.js` with `node_parameter_sets.js` own the panel's one list of saved values for the node type; diff --git a/docs/architecture/model-resolution.md b/docs/architecture/model-resolution.md index 386511c..eaa9d11 100644 --- a/docs/architecture/model-resolution.md +++ b/docs/architecture/model-resolution.md @@ -152,6 +152,12 @@ press puts them in: the bar's or the doctor page's for hash matches, the page's per-row button for a size-only candidate. A scan with "fix the open workflow" on puts hash matches in when it ends. +A model that is not here can be downloaded from the page when its source names one +exact file: Civitai by the workflow's hash, or a file link the workflow carries. The +download is checked against the SHA-256 before it gets its name, so a wrong file never +lands in a models folder; the file keeps the workflow's name, and nodes are pointed at +it only when it went to another folder than the workflow says. + Provenance-rich workflows skip the redundant full filename-to-hash cache refresh. Legacy workflows without injected provenance may refresh it for compatibility. This is an I/O optimization only and does not change identity evidence. diff --git a/web/hash_resolver.js b/web/hash_resolver.js index 26f7443..37c6cca 100644 --- a/web/hash_resolver.js +++ b/web/hash_resolver.js @@ -3,6 +3,7 @@ import { findWorkflowHashRecord } from './modules/recipe_provenance.js'; import { requiresHashForModelRecovery } from './modules/model_policies.js'; import { detectPlatform, isModelFilename } from './modules/model_source_links.js'; import { usableSourceUrl } from './modules/model_source_data.js'; +import { absorbFingerprintChange } from './modules/canvas_history.js'; // Global cache for hashes: filename -> hash window.anomalous_hash_cache = window.anomalous_hash_cache || {}; @@ -47,6 +48,7 @@ app.registerExtension({ window.anomalous_is_empty_state = true; console.warn("[Anomalous] Failed to fetch hashes", e); } + absorbFingerprintChange(app); }; // Pre-fetch all hashes on startup so that dragging generated images (without opening UI) still intercepts diff --git a/web/modules/activity_log.js b/web/modules/activity_log.js index 90ebc4a..4257d7c 100644 --- a/web/modules/activity_log.js +++ b/web/modules/activity_log.js @@ -34,6 +34,7 @@ const ACTION_KEYS = { voice_import: 'activityVoiceImport', voice_settings: 'activityVoiceSettings', voice_storage: 'activityVoiceStorage', + model_download: 'activityModelDownload', }; async function request(url, options) { @@ -80,6 +81,11 @@ export function entrySummary(entry) { return entry.target ? t('activityWithTarget', { action: words, target: entry.target }) : words; } +function formatSize(bytes) { + const value = Number(bytes) || 0; + return value >= 1024 ** 3 ? `${(value / 1024 ** 3).toFixed(2)} GB` : `${Math.max(0.1, value / 1024 ** 2).toFixed(1)} MB`; +} + const FIELD_KEYS = { custom_name: 'modelSourceFieldName', custom_notes: 'modelSourceFieldNotes', source_url: 'modelSourceFieldLink', filename: 'activityFieldFilename', @@ -95,6 +101,15 @@ export function fileDetailLines(entry) { const lines = []; if (detail.scan?.civitai_down) lines.push({ text: t('scanResultCivitaiDown') }); for (const error of detail.scan?.errors || []) lines.push({ text: error }); + const download = detail.download; + if (download) { + lines.push({ + text: t('activityDownloadLine', { file: download.rel || download.file, size: formatSize(download.size), source: download.source || '?' }), + model: { type: download.type, path_idx: download.path_idx, rel: download.rel, filename: download.file }, + }); + lines.push({ text: t(download.verified ? 'activityDownloadChecked' : 'activityDownloadUnchecked') }); + if (download.value && download.value !== download.rel) lines.push({ text: t('activityDownloadFor', { value: download.value }) }); + } for (const item of detail.files || []) { if (typeof item === 'string') lines.push({ text: t('activityFileMoved', { name: item }) }); else lines.push({ text: `${STATUS_MARKS[item.status] || '·'} ${item.filename}: ${fileOutcome(item)}`, model: item.rel ? item : null }); diff --git a/web/modules/canvas_history.js b/web/modules/canvas_history.js index b5bffb6..722f21a 100644 --- a/web/modules/canvas_history.js +++ b/web/modules/canvas_history.js @@ -2,7 +2,8 @@ * ComfyUI's Ctrl+Z history. It takes a step on its own events (a mouse button or key let * go); a canvas write made by a drop (no mouse-up follows a drag), in a click handler or * after a request finishes would otherwise land in no step, and Ctrl+Z would skip it. - * Call after every canvas write. Takes `app` so pure modules can use it. + * Call after every canvas write. Takes `app` so pure modules can use it. Also keeps a change of + * fingerprints alone (absorbFingerprintChange) out of the history. */ export function recordCanvasStep(app) { @@ -14,3 +15,35 @@ export function recordCanvasStep(app) { console.warn('[AMB] Ctrl+Z history: step not recorded.', error); } } + +// What a saved workflow carries from the fingerprint cache (hash_resolver.js), not the canvas. +const DERIVED = ['anomalous_hashes', 'anomalous_model_sources']; + +function withoutDerived(state) { + const extra = { ...(state?.extra || {}) }; + for (const key of DERIVED) delete extra[key]; + return { ...state, extra }; +} + +/** + * After the fingerprint cache changes (a scan, a download), the workflow serializes with + * other fingerprints although nothing on the canvas changed; ComfyUI would take that as a + * step, and the next Ctrl+Z would seem to do nothing. When that is the only difference, the + * history's current state takes it without a step. + */ +export function absorbFingerprintChange(app) { + try { + const tracker = app?.extensionManager?.workflow?.activeWorkflow?.changeTracker; + const graph = app?.rootGraph || app?.graph; + if (!tracker?.activeState || typeof graph?.serialize !== 'function') return; + const current = JSON.parse(JSON.stringify(graph.serialize())); + const a = withoutDerived(tracker.activeState); + const b = withoutDerived(current); + const same = typeof tracker.constructor?.graphEqual === 'function' + ? tracker.constructor.graphEqual(a, b) + : JSON.stringify(a) === JSON.stringify(b); + if (same) tracker.activeState = current; + } catch (error) { + console.warn('[AMB] Ctrl+Z history: could not take the new fingerprints in.', error); + } +} diff --git a/web/modules/download_places.js b/web/modules/download_places.js new file mode 100644 index 0000000..2a66579 --- /dev/null +++ b/web/modules/download_places.js @@ -0,0 +1,59 @@ +/** + * Where a downloaded model goes inside its type's models folder (Model Check's downloads; + * model_download.js, ui_model_download.js). The download settings say: where the workflow + * says (its own subfolder, so the node needs no change; a bare name goes into the settings' + * folder) or always the settings' folder, `{base}` there standing for the model's base model + * (an existing folder of that name, in any spelling, is used). The file keeps the workflow's + * name. No imports, so it is tested on its own. + */ + +const UNSAFE_FORMATS = /\.(ckpt|pt|pth|bin)$/i; +// Civitai's base models -> the folder name most people use for them. +const BASE_FAMILIES = [ + [/^sd ?1\.[45]/i, 'SD1.5'], [/^sdxl/i, 'SDXL'], [/^sd ?3/i, 'SD3'], [/^pony/i, 'Pony'], + [/^illustrious/i, 'Illustrious'], [/^noob/i, 'NoobAI'], [/^flux/i, 'Flux'], [/^hunyuan/i, 'Hunyuan'], + [/^wan/i, 'Wan'], [/^qwen/i, 'Qwen'], [/^hidream/i, 'HiDream'], [/^chroma/i, 'Chroma'], +]; + +const slashes = (value) => String(value || '').replace(/\\/g, '/'); +const fileName = (value) => slashes(value).split('/').pop(); +const dirName = (value) => slashes(value).split('/').slice(0, -1).join('/'); +const squash = (value) => String(value || '').toLowerCase().replace(/[^a-z0-9]/g, ''); + +/** The folder name `{base}` stands for among `siblings` (the folders beside it): one of them + * that names the model's base model, in any spelling, else the base model's usual name. */ +export function baseFolder(baseModel, siblings = []) { + const base = String(baseModel || '').trim(); + if (!base) return ''; + const family = BASE_FAMILIES.find(([pattern]) => pattern.test(base))?.[1] || base.replace(/[<>:"/\\|?*]+/g, ' ').replace(/ +/g, ' ').trim(); + const names = new Set([squash(family), squash(base)]); + return siblings.find(name => names.has(squash(name))) || family; +} + +/** The settings' folder with `{base}` filled in (dropped when the base model is unknown); + * `subfolders` are the type folder's existing ones ("a", "a/b"). */ +export function expandFolder(folder, baseModel, subfolders = []) { + const parts = []; + for (const raw of slashes(folder).split('/')) { + let part = raw.trim(); + if (/\{base\}/i.test(part)) { + const prefix = parts.join('/').toLowerCase(); + const siblings = subfolders.filter(path => dirName(path).toLowerCase() === prefix).map(fileName); + part = part.replace(/\{base\}/gi, baseFolder(baseModel, siblings)).trim(); + } + if (part) parts.push(part); + } + return parts.join('/'); +} + +/** Where in its type's folder an entry's file goes: "sub/folder/name.ext", '/' separated. */ +export function destinationFor(entry, info, settings, subfolders = []) { + const name = fileName(entry.value); + const own = dirName(entry.value); + if (settings?.place !== 'folder' && own) return `${own}/${name}`; + const folder = expandFolder(settings?.folder ?? 'Downloads', info?.base_model, subfolders); + return folder ? `${folder}/${name}` : name; +} + +/** Whether a file of this kind can carry code that runs when it is loaded. */ +export const unsafeFormat = (value) => UNSAFE_FORMATS.test(String(value || '')); diff --git a/web/modules/locales.js b/web/modules/locales.js index 855d961..30eaddf 100644 --- a/web/modules/locales.js +++ b/web/modules/locales.js @@ -680,7 +680,7 @@ export const i18n = { doctorSources: '模型去哪下载', doctorNoModels: '当前工作流没有用到模型文件。', doctorAllHere: '{count} 个模型都在本机，可以直接运行。', - doctorHowItWorks: '“换上”只会用同一个文件（指纹一致）。其余的请自己挑一个本地模型，或者去 Civitai 下载。', + doctorHowItWorks: '“换上”只会用同一个文件（指纹一致）。缺的模型能确定来源的（C 站按指纹找到、或工作流带着文件链接）可以直接下载；其余的请自己挑一个本地模型，或者去 Civitai 找。', doctorToHandle: '需要处理', doctorReplaced: '刚换上的', doctorWas: '原来是 {file}', @@ -701,6 +701,44 @@ export const i18n = { doctorWhyChanged: '文件还在，但内容和保存工作流时不一样（可能换成了别的版本）。ComfyUI 会照常使用它。', doctorWhyNotHere: '按指纹在本机找过，没有这个文件。', doctorWhyNoRecord: '工作流里没有记下它的指纹，没法自动找，只能自己挑或下载。', + downloadLooking: '查找下载来源…', + downloadOne: '下载 · {size}', + downloadOneNoSize: '下载', + downloadAll: '下载全部 {count} 个（{size}）', + downloadNoNetwork: '连不上下载网站，暂时查不到来源。', + downloadDifferentFile: '工作流里的链接指向的不是同一个文件，没有提供下载。', + downloadGated: '这个文件要先在 Hugging Face 上登录并同意使用条款，请到网页上下载。', + downloadTitle: '下载缺失的模型', + downloadTitleMany: '下载 {count} 个缺失的模型', + downloadSaveTo: '保存到', + downloadRootNew: '（还没有这个文件夹，会新建）', + downloadUnsafeFormat: '⚠ 这是 .ckpt / .pt 一类的格式，加载时可能运行文件里夹带的代码。只下载你信任的来源。', + downloadNoCheck: 'ⓘ 这个来源没有给出文件指纹，下完没法核对是不是同一个文件。', + downloadSpace: '这次要下载 {need}，磁盘剩余 {free}。', + downloadRules: '默认保存到哪里，在 设置 → 模型下载 里改', + downloadStart: '开始下载', + downloadQueued: '排队中…', + downloadVerifying: '核对文件…', + downloadProgress: '{percent}% · {done} / {total}', + downloadRetry: '重试', + downloadSetKey: '填 API Key', + downloadDone: '已下载：{name}', + downloadDonePutIn: '已下载并换上：{name}', + downloadError_network: '下载中断了（网络问题）。点重试会从断开的地方接着下。', + downloadError_needs_key: '这个模型要登录 Civitai 才能下载：填上你的 Civitai API Key 再点下载。', + downloadError_forbidden: '网站拒绝了下载（可能要登录或没有权限）。', + downloadError_gone: '网站上已经没有这个文件了。', + downloadError_hash_mismatch: '下载到的文件和工作流记录的指纹对不上，已经丢弃，没有放进模型文件夹。', + downloadError_exists: '那里已经有同名文件了，不会覆盖。换个位置或文件名。', + downloadError_already: '这个文件已经在下载了。', + downloadError_no_space: '磁盘空间不够。', + downloadError_not_a_file: '网站返回的是网页而不是模型文件（多半要登录）。', + downloadError_bad_name: '文件名要以模型格式结尾，比如 .safetensors。', + downloadError_bad_path: '这个位置不行：只能放在模型文件夹里面。', + downloadError_bad_source: '这个下载来源不在允许的网站里。', + downloadError_bad_type: '认不出这个模型该放进哪类文件夹。', + downloadError_bad_root: '这个模型文件夹不存在了。', + downloadError_unknown: '下载失败了，详细原因在 ComfyUI 的控制台里。', doctorProof: '指纹', doctorProofSaved: '工作流里记的', doctorProofLocal: '本机同名文件', @@ -932,6 +970,11 @@ export const i18n = { activityVoiceImport: "导入角色", activityVoiceSettings: "修改角色设置", activityVoiceStorage: "更改角色存放位置", + activityModelDownload: "下载模型", + activityDownloadLine: "{file}（{size}，来自 {source}）", + activityDownloadChecked: "✓ 指纹核对一致", + activityDownloadUnchecked: "来源没有指纹，没有核对", + activityDownloadFor: "工作流里写的是 {value}", sidebarToggle: '收起/展开列表', sidebarDoctor: '模型检查', sidebarAssistant: '当前节点', @@ -975,6 +1018,17 @@ export const i18n = { settingsWorkflows: "工作流", settingsCheckOnOpen: '打开工作流时检查缺失模型', settingsCheckOnOpenHelp: '打开或拖进工作流时，缺模型就在画布上方提示，本机有同一个文件的可以一键换上。关掉后，要检查就点左侧的模型检查。', + settingsDownloads: '模型下载', + settingsDownloadPlace: '缺失的模型下载到', + settingsDownloadPlaceHelp: '“工作流写的位置”：按工作流里记的子文件夹放，下完节点不用改；工作流没写子文件夹的，放进下面这个文件夹。“固定文件夹”：总是放进下面这个文件夹，下完自动把节点改到新位置（可以 Ctrl+Z）。每次下载前，确认框里还能改。', + settingsDownloadPlaceWorkflow: '工作流写的位置', + settingsDownloadPlaceFolder: '固定文件夹', + settingsDownloadFolder: '文件夹', + settingsDownloadFolderHelp: '在各类模型自己的文件夹里面，比如 loras 下的 Downloads。写 {base} 会换成模型的底模，比如 SDXL、Flux、Illustrious；已经有这个名字的文件夹（不分大小写和符号）就用它。留空就放在类型文件夹本身。', + settingsDownloadFolderRoot: '（类型文件夹本身）', + settingsDownloadKey: 'Civitai API Key', + settingsDownloadKeyHelp: '有些模型要登录 Civitai 才能下载，填上你的 Key 就能下。只保存在本机。', + settingsDownloadKeySet: '设置…', settingsOpening: "打开方式与窗口", settingsEntryHelp: "在哪里打开这个浏览器。快捷键在 ComfyUI 设置 → Anomalous Model Browser 里设置。", settingsLayout: "窗口布局", @@ -2616,7 +2670,7 @@ export const i18n = { doctorSources: 'Where to download', doctorNoModels: 'The open workflow uses no model files.', doctorAllHere: 'All {count} models are on this computer; the workflow can run.', - doctorHowItWorks: '"Put in" only uses the same file (same fingerprint). For the others, pick a local model yourself or download it from Civitai.', + doctorHowItWorks: '"Put in" only uses the same file (same fingerprint). A missing model whose source is certain (found on Civitai by its fingerprint, or a file link the workflow carries) can be downloaded right here; for the others, pick a local model yourself or look on Civitai.', doctorToHandle: 'To handle', doctorReplaced: 'Just put in', doctorWas: 'Was {file}', @@ -2637,6 +2691,44 @@ export const i18n = { doctorWhyChanged: 'The file is there, but its content differs from when the workflow was saved (maybe another version). ComfyUI uses it as usual.', doctorWhyNotHere: 'Looked for its fingerprint on this computer: not here.', doctorWhyNoRecord: 'The workflow has no fingerprint for it, so it cannot be found automatically. Pick one or download it.', + downloadLooking: 'Looking for a download…', + downloadOne: 'Download · {size}', + downloadOneNoSize: 'Download', + downloadAll: 'Download all {count} ({size})', + downloadNoNetwork: 'Cannot reach the download sites right now.', + downloadDifferentFile: 'The workflow\'s link points to a different file, so it is not offered.', + downloadGated: 'This file needs a Hugging Face login and accepting its terms; download it on the page.', + downloadTitle: 'Download the missing model', + downloadTitleMany: 'Download {count} missing models', + downloadSaveTo: 'Save to', + downloadRootNew: '(not there yet, will be created)', + downloadUnsafeFormat: '⚠ A .ckpt / .pt kind of file can run code it carries when it loads. Only download it from a source you trust.', + downloadNoCheck: 'ⓘ The source gives no fingerprint, so the file cannot be checked after downloading.', + downloadSpace: 'Downloading {need}; {free} free on the disk.', + downloadRules: 'Change where downloads go by default in Settings → Model downloads', + downloadStart: 'Download', + downloadQueued: 'Waiting…', + downloadVerifying: 'Checking the file…', + downloadProgress: '{percent}% · {done} / {total}', + downloadRetry: 'Retry', + downloadSetKey: 'Set API key', + downloadDone: 'Downloaded: {name}', + downloadDonePutIn: 'Downloaded and put in: {name}', + downloadError_network: 'The download broke off (network). Retry continues where it stopped.', + downloadError_needs_key: 'This model needs a Civitai login: set your Civitai API key, then download again.', + downloadError_forbidden: 'The site refused the download (a login or permission may be needed).', + downloadError_gone: 'The site no longer has this file.', + downloadError_hash_mismatch: 'The downloaded file does not match the workflow\'s fingerprint; it was thrown away, not put into the models folder.', + downloadError_exists: 'A file of that name is already there and is never replaced. Choose another place or name.', + downloadError_already: 'This file is already downloading.', + downloadError_no_space: 'Not enough disk space.', + downloadError_not_a_file: 'The site sent a web page instead of the model (probably a login).', + downloadError_bad_name: 'The name must end in a model format such as .safetensors.', + downloadError_bad_path: 'Not a place it can go: only inside the models folder.', + downloadError_bad_source: 'This download source is not one of the allowed sites.', + downloadError_bad_type: 'Cannot tell which kind of models folder this goes into.', + downloadError_bad_root: 'That models folder no longer exists.', + downloadError_unknown: 'The download failed; the ComfyUI console says why.', doctorProof: 'Fingerprint', doctorProofSaved: 'In the workflow', doctorProofLocal: 'Local file of that name', @@ -2868,6 +2960,11 @@ export const i18n = { activityVoiceImport: "Imported character", activityVoiceSettings: "Changed character settings", activityVoiceStorage: "Changed where characters are kept", + activityModelDownload: "Downloaded a model", + activityDownloadLine: "{file} ({size}, from {source})", + activityDownloadChecked: "✓ Fingerprint matches", + activityDownloadUnchecked: "The source gave no fingerprint; not checked", + activityDownloadFor: "The workflow named it {value}", sidebarToggle: 'Show or hide the list', sidebarDoctor: 'Model Check', sidebarAssistant: 'Current node', @@ -2911,6 +3008,17 @@ export const i18n = { settingsWorkflows: "Workflows", settingsCheckOnOpen: 'Check for missing models when a workflow opens', settingsCheckOnOpenHelp: 'When you open or drop in a workflow, a bar over the canvas says which models are missing and puts in the ones this computer has under another name in one press. Turned off, run Model Check from the rail when you want it.', + settingsDownloads: 'Model downloads', + settingsDownloadPlace: 'Download missing models to', + settingsDownloadPlaceHelp: '"Where the workflow says": into the subfolder the workflow names, so the node needs no change; a model without one goes into the folder below. "A set folder": always into the folder below, and the node is changed to the new place afterwards (Ctrl+Z undoes it). The dialog before each download can still change it.', + settingsDownloadPlaceWorkflow: 'Where the workflow says', + settingsDownloadPlaceFolder: 'A set folder', + settingsDownloadFolder: 'Folder', + settingsDownloadFolderHelp: 'Inside each kind\'s own models folder, such as Downloads under loras. {base} becomes the model\'s base model, such as SDXL, Flux or Illustrious; a folder of that name you already have (any case or punctuation) is used. Empty means the kind\'s folder itself.', + settingsDownloadFolderRoot: '(the kind\'s folder itself)', + settingsDownloadKey: 'Civitai API key', + settingsDownloadKeyHelp: 'Some models need a Civitai login to download; your key lets them. Kept on this computer only.', + settingsDownloadKeySet: 'Set…', settingsOpening: "Opening and window", settingsEntryHelp: "Where you open this browser from. Shortcuts are set in ComfyUI's settings under Anomalous Model Browser.", settingsLayout: "Window layout", diff --git a/web/modules/model_check.js b/web/modules/model_check.js index 32f8db2..cce7427 100644 --- a/web/modules/model_check.js +++ b/web/modules/model_check.js @@ -195,6 +195,7 @@ export function applyModelFix(entry, target = entry.target, { step = true } = {} // ComfyUI clears its own missing-model mark through this hook. node.onWidgetChanged?.(widget.name, target, value, widget); if (app.lastNodeErrors?.[node.id]) delete app.lastNodeErrors[node.id]; + if (target === value) return true; // the same name, now found (a download): nothing to undo markReplaced(widget, value, target); // Without a step ComfyUI keeps the old name: a reload or Ctrl+Z brings the missing model back. if (step) recordCanvasStep(app); diff --git a/web/modules/model_download.js b/web/modules/model_download.js new file mode 100644 index 0000000..305ce2f --- /dev/null +++ b/web/modules/model_download.js @@ -0,0 +1,194 @@ +/** + * Downloading the models the open workflow is missing, behind Model Check's "Download" + * (api/model_download.py does the downloading; ui_model_download.js shows it, and + * download_places.js says where a file goes). No DOM. + * + * When a download ends, the model lists reload, every node still + * naming that model gets the file (one Ctrl+Z step each), and the files of the batch are + * scanned for their information and covers once nothing is downloading. + */ + +import { app } from "../../../scripts/app.js"; +import { translate as t } from './locales.js'; +import { applyModelFix } from './model_check.js'; +import { inferModelFolderTypes } from './model_policies.js'; +import { startScan, targetsForItems } from './scan_runner.js'; +import { showWorkbenchToast } from './ui_prompt_toast.js'; + +const POLL_MS = 1000; +const ACTIVE = new Set(['queued', 'running', 'verifying']); + +const slashes = (value) => String(value || '').replace(/\\/g, '/'); +const fileName = (value) => slashes(value).split('/').pop(); + +const jobs = new Map(); // id -> latest status +const listeners = new Set(); +const lookups = new Map(); // lookup key -> result +let pollTimer = 0; +let batch = []; +let scanOwner = null; + +async function json(url, options) { + const response = await fetch(url, options); + const data = await response.json().catch(() => ({})); + if (!response.ok) { + const error = new Error(data.error || `HTTP ${response.status}`); + error.code = data.error || 'http'; + throw error; + } + return data; +} + +const post = (url, body) => json(url, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); + +/** Where downloads go: { place: 'workflow' | 'folder', folder }. */ +export const fetchDownloadSettings = () => json('/anomalous/download/settings'); +export const saveDownloadSettings = (changes) => post('/anomalous/download/settings', changes); + +/** The link a shared workflow keeps for a model: AMB's, else ComfyUI's own (`properties.models`). */ +export function workflowLink(entry) { + const value = entry.value; + const kept = app.graph?.extra?.anomalous_model_sources || {}; + const saved = kept[value] || kept[slashes(value)] || kept[fileName(value)] || kept[`${entry.node.id}_${value}`]; + const url = typeof saved === 'string' ? saved : saved?.url; + if (url) return url; + const name = fileName(value).toLowerCase(); + const native = [ + ...(entry.node.properties?.models || []), + ...(app.extensionManager?.workflow?.activeWorkflow?.activeState?.models || []), + ]; + return native.find(model => String(model?.name || '').split(/[\\/]/).pop().toLowerCase() === name)?.url || ''; +} + +const lookupKey = (item) => `${item.hash}|${item.url}|${item.value}`; + +/** + * Where each entry can be downloaded from: { found: Map entry -> { found, source, size, sha256, + * base_model, page, type, roots, reason… }, settings }. Sources are asked once per page load + * (a network failure again next time); the model folders (free space, subfolders) every time. + */ +export async function lookupDownloads(entries) { + const items = entries.map((entry, index) => { + const item = { key: String(index), hash: entry.record?.hash || '', url: workflowLink(entry), value: entry.value }; + return { ...item, types: inferModelFolderTypes(entry.node, entry.widget), known: lookups.has(lookupKey(item)) }; + }); + const data = await post('/anomalous/download/lookup', { items }); + const found = new Map(); + for (const result of data.results || []) { + const item = items.find(candidate => candidate.key === result.key); + if (!item) continue; + let info = item.known ? lookups.get(lookupKey(item)) : result; + if (!item.known && result.reason !== 'network' && (item.hash || item.url)) lookups.set(lookupKey(item), result); + if (!info || (!item.hash && !item.url)) continue; + info = { ...info, type: result.type, roots: data.roots?.[result.type] || [], link: item.url }; + found.set(entries[Number(item.key)], info); + } + return { found, settings: data.settings || { place: 'workflow', folder: 'Downloads' } }; +} + +function notify() { + for (const listener of listeners) { + try { listener(jobs); } catch (error) { console.warn('[AMB] Download: a listener failed.', error); } + } +} + +/** Calls `listener(jobs)` whenever a download moves on; returns the way to stop. */ +export function watchDownloads(listener) { + listeners.add(listener); + return () => listeners.delete(listener); +} + +/** The newest download of a workflow value (its model), if any. */ +export function downloadFor(value) { + let latest = null; + for (const job of jobs.values()) if (job.value === value) latest = job; + return latest; +} + +export const downloadsActive = () => [...jobs.values()].some(job => ACTIVE.has(job.state)); +export const doneCount = () => [...jobs.values()].filter(job => job.state === 'done').length; + +/** Takes in the downloads already running (started before the page reloaded). */ +export async function syncDownloads(owner) { + const latest = (await json('/anomalous/download/status')).jobs || []; + for (const job of latest) if (!jobs.has(job.id)) jobs.set(job.id, job); + if (downloadsActive()) { + scanOwner = scanOwner || owner; + poll(); + } +} + +/** Starts (or continues) one download; `choice` = { root, rel }. Rejects with `.code` when refused. */ +export async function startDownload(owner, entry, info, choice) { + scanOwner = owner; + const job = await post('/anomalous/download/start', { + key: entry.value, value: entry.value, download_url: info.download_url, type: info.type, + root: choice.root, rel: choice.rel, size: info.size || 0, sha256: info.sha256 || '', + hash: entry.record?.hash || '', source: info.source, page: info.page || '', + }); + jobs.set(job.id, job); + notify(); + poll(); + return job; +} + +export async function cancelDownload(id) { + const job = await post('/anomalous/download/cancel', { id }); + jobs.set(job.id, job); + notify(); +} + +function poll() { + if (pollTimer) return; + pollTimer = setTimeout(async () => { + pollTimer = 0; + let latest = []; + try { + latest = (await json('/anomalous/download/status')).jobs || []; + } catch (error) { + console.warn('[AMB] Download: status failed.', error); + } + for (const job of latest) { + const before = jobs.get(job.id); + jobs.set(job.id, job); + if (job.state === 'done' && before && before.state !== 'done') await afterDownload(job); + } + notify(); + if (downloadsActive()) poll(); + else scanBatch(); + }, POLL_MS); +} + +/** ComfyUI's drop-downs (the server already cleared its file-list caches). */ +async function reloadModelLists() { + try { + await app.refreshComboInNodes?.(); + } catch (error) { + console.warn('[AMB] Download: could not reload the model lists.', error); + } +} + +/** Puts the new file into every node still naming the model it was downloaded for. */ +async function afterDownload(job) { + await reloadModelLists(); + const rel = slashes(job.rel); + let put = 0; + for (const node of app.graph?._nodes || []) { + for (const widget of node.widgets || []) { + if (widget.type !== 'combo' || widget.value !== job.value) continue; + const target = (widget.options?.values || []).find(value => typeof value === 'string' && slashes(value) === rel); + // Same place as the workflow says: nothing changes, ComfyUI's missing mark goes. + if (target && applyModelFix({ node, widget, value: job.value }, target) && target !== job.value) put += 1; + } + } + showWorkbenchToast(t(put ? 'downloadDonePutIn' : 'downloadDone', { name: job.file })); + batch.push({ type: job.type, path_idx: job.root, rel, filename: job.file }); +} + +/** The downloaded files' information and covers: one scan of the whole batch. */ +async function scanBatch() { + if (!batch.length || !scanOwner) return; + const items = batch; + batch = []; + await startScan(scanOwner, { targets: targetsForItems(items) }); +} diff --git a/web/modules/ui_doctor.js b/web/modules/ui_doctor.js index aad1fe2..cede2f7 100644 --- a/web/modules/ui_doctor.js +++ b/web/modules/ui_doctor.js @@ -2,13 +2,14 @@ * Model Check, the tool page: the models the open workflow uses and what is wrong with each * (model_check.js), with what can be done about it: put in the same file found under * another name or folder, take or leave a file of the same name or size, pick one by hand (the - * node model picker) or look it up on Civitai. Nothing in the workflow changes without a - * press here. Uses the scan page's layout classes (18-scan-page.css). + * node model picker), download it when its source is known (ui_model_download.js) or look it + * up on Civitai. Nothing in the workflow changes without a press here. Uses the scan page's layout classes (18-scan-page.css). */ import { translate as t } from './locales.js'; import { applyModelFix, checkWorkflowModels, fixWorkflowModels, isProblem, markReplaced, replacedFrom } from './model_check.js'; import { updateDoctorBanner } from './ui_doctor_banner.js'; +import { fillDownloads } from './ui_model_download.js'; const MODEL_EXT = /\.(safetensors|ckpt|pt|bin|pth|sft|gguf)$/i; const CANDIDATE_WHY = { size: 'doctorWhyCandidate', name: 'doctorWhySameName', 'name-size': 'doctorWhyLikely' }; @@ -113,7 +114,7 @@ function pickByHand(owner, panel, entry) { }); } -function problemRow(owner, panel, entry) { +function problemRow(owner, panel, entry, downloads) { const row = el('div', `anomalous-scan-row anomalous-doctor-row is-${entry.state}`); const copy = el('div', 'anomalous-scan-row-copy'); copy.append( @@ -127,6 +128,12 @@ function problemRow(owner, panel, entry) { if (!applyModelFix(entry)) alert(t('doctorChangedMeanwhile')); renderDoctorPage(owner, panel); }); + if (entry.state === 'missing') { + // Filled once its source is known (ui_model_download.js). + const slot = el('span', 'anomalous-download-slot'); + downloads.set(entry, slot); + actions.append(slot); + } if (entry.state === 'fixable') actions.append(put('doctorPutIn')); if (entry.state === 'candidate') actions.append(put('doctorUseCandidate')); if (entry.state !== 'fixable') { @@ -221,7 +228,8 @@ export async function renderDoctorPage(owner, panel, { refresh = false } = {}) { : t('doctorHowItWorks'); page.append(el('p', 'anomalous-scan-muted', verdict)); - if (problems.length) page.append(card(t('doctorToHandle'), problems.map(entry => problemRow(owner, panel, entry)))); + const downloads = new Map(); + if (problems.length) page.append(card(t('doctorToHandle'), problems.map(entry => problemRow(owner, panel, entry, downloads)))); if (hint) page.append(hint); if (replaced.length) { page.append(card(t('doctorReplaced'), replaced.map(entry => simpleRow(entry, t('doctorWas', { file: replacedFrom(entry.widget) }))))); @@ -235,6 +243,8 @@ export async function renderDoctorPage(owner, panel, { refresh = false } = {}) { page.append(fold); } panel.replaceChildren(page); + fillDownloads(owner, panel, downloads, actions, () => renderDoctorPage(owner, panel)) + .catch(error => console.warn('[AMB] Doctor: downloads unavailable.', error)); } /** The rail's doctor button, Home's card and the canvas banner's "Show". */ diff --git a/web/modules/ui_model_download.js b/web/modules/ui_model_download.js new file mode 100644 index 0000000..ba69b67 --- /dev/null +++ b/web/modules/ui_model_download.js @@ -0,0 +1,285 @@ +/** + * Model Check's downloads, on its page (ui_doctor.js): each missing model with a source gets + * "Download" (and the page "Download all"); a dialog shows, per file, where it will be saved — + * the folder of its type (a choice when there are several) and the subfolder the download + * settings give, both changeable — its size, source and anything to know (a format that can + * carry code, nothing to check it against), and the free space. While a file downloads its + * row shows how far, with Cancel; a failed one says why, with Retry (it continues where it + * stopped). model_download.js does the work. + */ + +import { translate as t } from './locales.js'; +import { destinationFor, unsafeFormat } from './download_places.js'; +import { + cancelDownload, doneCount, downloadFor, lookupDownloads, startDownload, syncDownloads, watchDownloads, +} from './model_download.js'; +import { saveApiKey } from './ui_scan_page.js'; +import { openSettingsPage } from './ui_settings_page.js'; + +const ROOT_KEY = 'anomalous_download_root_'; +const ACTIVE = ['queued', 'running', 'verifying']; +const SOURCES = { civitai: 'Civitai', huggingface: 'Hugging Face', link: 'GitHub' }; +const NOT_FOUND = { network: 'downloadNoNetwork', different_file: 'downloadDifferentFile', gated: 'downloadGated' }; + +function el(tag, className, text) { + const node = document.createElement(tag); + if (className) node.className = className; + if (text !== undefined) node.textContent = text; + return node; +} + +function button(className, label, onClick) { + const node = el('button', className, label); + node.type = 'button'; + node.onclick = onClick; + return node; +} + +export function formatSize(bytes) { + const value = Number(bytes) || 0; + if (value >= 1024 ** 3) return `${(value / 1024 ** 3).toFixed(2)} GB`; + if (value >= 1024 ** 2) return `${(value / 1024 ** 2).toFixed(1)} MB`; + return `${Math.max(1, Math.round(value / 1024))} KB`; +} + +const fileName = (value) => String(value).split(/[\\/]/).pop(); + +function rememberedRoot(type, roots) { + let index = 0; + try { index = Number(localStorage.getItem(ROOT_KEY + type) || 0); } catch { /* the first one */ } + return roots.some(root => root.index === index) ? index : (roots.find(root => root.exists)?.index ?? 0); +} + +function rememberRoot(type, index) { + try { localStorage.setItem(ROOT_KEY + type, String(index)); } catch { /* not kept */ } +} + +const joinPath = (root, rel) => `${String(root).replace(/[\\/]+$/, '')}${String(root).includes('\\') ? '\\' : '/'}${ + String(root).includes('\\') ? rel.replace(/\//g, '\\') : rel}`; + +/** One file's part of the dialog; `read()` gives { root, rel } as chosen. */ +function dialogItem(entry, info, settings) { + // ComfyUI also lists folders that do not exist (output/loras…): offered only when none does. + const existing = (info.roots || []).filter(root => root.exists); + const roots = existing.length ? existing : (info.roots || []).slice(0, 1); + const box = el('div', 'anomalous-download-item'); + const head = el('div', 'anomalous-download-head'); + head.append(el('strong', 'anomalous-download-name', fileName(entry.value))); + const facts = [SOURCES[info.source] || info.source, info.size ? formatSize(info.size) : '', info.base_model].filter(Boolean); + head.append(el('span', 'anomalous-download-facts', facts.join(' · '))); + box.append(head); + if (info.model_name) box.append(el('div', 'anomalous-download-model', [info.model_name, info.version_name].filter(Boolean).join(' — '))); + + let root = roots.find(item => item.index === rememberedRoot(info.type, roots)) || roots[0] || { index: 0, path: '', subfolders: [] }; + const line = el('label', 'anomalous-download-where'); + line.append(el('span', 'anomalous-download-label', t('downloadSaveTo'))); + if (roots.length > 1) { + const select = el('select', 'anomalous-download-root'); + for (const item of roots) { + const option = el('option', '', item.path + (item.exists ? '' : ` ${t('downloadRootNew')}`)); + option.value = String(item.index); + option.selected = item.index === root.index; + select.append(option); + } + select.onchange = () => { + root = roots.find(item => String(item.index) === select.value) || root; + // Its own folders decide {base}, unless the path was typed by hand. + if (!input.dataset.edited) input.value = destinationFor(entry, info, settings, root.subfolders); + fillList(); + showPath(); + }; + line.append(select); + } + const input = el('input', 'anomalous-download-rel'); + input.type = 'text'; + input.spellcheck = false; + input.value = destinationFor(entry, info, settings, root.subfolders); + const list = el('datalist'); + list.id = `anomalous-download-list-${Math.random().toString(36).slice(2)}`; + input.setAttribute('list', list.id); + const fillList = () => list.replaceChildren(...(root.subfolders || []).map(folder => { + const option = el('option'); + option.value = `${folder}/${fileName(entry.value)}`; + return option; + })); + fillList(); + line.append(input, list); + const full = el('div', 'anomalous-download-path'); + const showPath = () => { full.textContent = root.path ? joinPath(root.path, input.value.trim()) : input.value.trim(); }; + input.oninput = () => { + input.dataset.edited = '1'; + showPath(); + }; + showPath(); + box.append(line, full); + + if (unsafeFormat(entry.value) || unsafeFormat(info.file_name)) box.append(el('div', 'anomalous-download-note is-warn', t('downloadUnsafeFormat'))); + if (!info.sha256 && !(entry.record?.hash?.length >= 10)) box.append(el('div', 'anomalous-download-note', t('downloadNoCheck'))); + const error = el('div', 'anomalous-download-note is-bad'); + error.hidden = true; + box.append(error); + return { + box, + read: () => ({ root: root.index, rel: input.value.trim().replace(/\\/g, '/'), rootInfo: root }), + fail: (text) => { error.textContent = text; error.hidden = !text; }, + }; +} + +function errorText(code) { + const key = `downloadError_${code}`; + const text = t(key); + return text === key ? t('downloadError_unknown') : text; +} + +/** The dialog for `items` ([{ entry, info }]); resolves once downloads started or it was closed. */ +export function openDownloadDialog(owner, items, settings) { + return new Promise((resolve) => { + const overlay = el('div', 'anomalous-dialog-overlay anomalous-download-overlay'); + const dialog = el('div', 'anomalous-download-dialog'); + dialog.setAttribute('role', 'dialog'); + dialog.append(el('h3', 'anomalous-download-title', t(items.length > 1 ? 'downloadTitleMany' : 'downloadTitle', { count: items.length }))); + const parts = items.map(({ entry, info }) => ({ entry, info, ...dialogItem(entry, info, settings) })); + const listBox = el('div', 'anomalous-download-items'); + listBox.append(...parts.map(part => part.box)); + dialog.append(listBox); + + const total = items.reduce((sum, { info }) => sum + (Number(info.size) || 0), 0); + const space = el('div', 'anomalous-download-space'); + const showSpace = () => { + const roots = new Map(parts.map(part => [part.read().rootInfo.path, part.read().rootInfo])); + const free = Math.min(...[...roots.values()].map(root => Number(root.free) || 0)); + space.textContent = t('downloadSpace', { need: total ? formatSize(total) : '?', free: free ? formatSize(free) : '?' }); + space.classList.toggle('is-bad', Boolean(total && free && free < total)); + }; + showSpace(); + dialog.addEventListener('change', showSpace); + const rules = button('anomalous-download-rules', t('downloadRules'), () => { + close(); + openSettingsPage(owner); + }); + dialog.append(space, rules); + + const footer = el('div', 'anomalous-download-footer'); + const close = () => { overlay.remove(); document.removeEventListener('keydown', onKey, true); resolve(false); }; + const onKey = (event) => { if (event.key === 'Escape') { event.stopPropagation(); close(); } }; + document.addEventListener('keydown', onKey, true); + const go = button('anomalous-scan-primary', t('downloadStart'), async () => { + go.disabled = true; + let left = 0; + for (const part of parts) { + const choice = part.read(); + part.fail(''); + try { + await startDownload(owner, part.entry, part.info, choice); + rememberRoot(part.info.type, choice.root); + part.box.remove(); + } catch (error) { + left += 1; + part.fail(errorText(error.code)); + } + } + go.disabled = false; + if (!left) { + overlay.remove(); + document.removeEventListener('keydown', onKey, true); + resolve(true); + } + }); + footer.append(button('anomalous-scan-secondary', t('dialogCancel'), close), go); + dialog.append(footer); + overlay.append(dialog); + overlay.onclick = (event) => { if (event.target === overlay) close(); }; + document.body.append(overlay); + go.focus(); + }); +} + +/** What a row shows for its model's download: progress, a failure, or nothing yet. */ +function jobView(owner, job, retry) { + const box = el('span', 'anomalous-download-status'); + if (ACTIVE.includes(job.state)) { + const percent = job.total ? Math.floor((job.received / job.total) * 100) : 0; + const text = job.state === 'queued' ? t('downloadQueued') + : job.state === 'verifying' ? t('downloadVerifying') + : t('downloadProgress', { percent, done: formatSize(job.received), total: job.total ? formatSize(job.total) : '?' }); + const bar = el('span', 'anomalous-download-bar'); + bar.style.setProperty('--amb-download-done', `${percent}%`); + box.append(el('span', 'anomalous-download-text', text), bar, + button('anomalous-scan-row-btn', t('dialogCancel'), () => cancelDownload(job.id))); + } else if (job.state === 'failed') { + box.append(el('span', 'anomalous-download-text is-bad', errorText(job.error))); + if (job.error === 'needs_key') { + box.append(button('anomalous-scan-row-btn is-main', t('downloadSetKey'), async () => { + if (await saveApiKey()) retry(); + })); + } else if (!['hash_mismatch', 'gone', 'exists'].includes(job.error)) { + box.append(button('anomalous-scan-row-btn', t('downloadRetry'), retry)); + } + } + return box; +} + +/** + * Fills the download parts of a rendered Model Check page: `slots` (Map entry -> the span in + * its row) and `pageActions` (the page's buttons) for "Download all". `rerender` redraws the + * page (after a download ends, the model is there). + */ +export async function fillDownloads(owner, panel, slots, pageActions, rerender) { + panel._downloadUnwatch?.(); + if (!slots.size) return; + for (const slot of slots.values()) slot.replaceChildren(el('span', 'anomalous-download-text', t('downloadLooking'))); + let looked; + try { + await syncDownloads(owner); + looked = await lookupDownloads([...slots.keys()]); + } catch (error) { + console.warn('[AMB] Download: looking for sources failed.', error); + for (const slot of slots.values()) slot.replaceChildren(); + return; + } + if (!panel.isConnected || [...slots.values()].some(slot => !slot.isConnected)) return; // redrawn meanwhile + const { found, settings } = looked; + const ready = []; + const draw = () => { + for (const [entry, slot] of slots) { + const info = found.get(entry); + const job = downloadFor(entry.value); + const start = () => openDownloadDialog(owner, [{ entry, info }], settings); + if (job && job.state !== 'cancelled' && job.state !== 'done') { + slot.replaceChildren(jobView(owner, job, start)); + } else if (info?.found) { + slot.replaceChildren(button('anomalous-scan-row-btn is-main', info.size + ? t('downloadOne', { size: formatSize(info.size) }) : t('downloadOneNoSize'), start)); + } else if (info && NOT_FOUND[info.reason]) { + slot.replaceChildren(el('span', 'anomalous-download-text', t(NOT_FOUND[info.reason]))); + } else { + slot.replaceChildren(); + } + } + }; + // Not downloading now (an earlier, ended download of the same name does not count). + const idle = (entry) => !ACTIVE.includes(downloadFor(entry.value)?.state); + for (const [entry] of slots) { + if (found.get(entry)?.found && idle(entry)) ready.push({ entry, info: found.get(entry) }); + } + if (ready.length > 1) { + const size = ready.reduce((sum, item) => sum + (Number(item.info.size) || 0), 0); + pageActions.prepend(button('anomalous-scan-primary', t('downloadAll', { count: ready.length, size: formatSize(size) }), () => { + const waiting = ready.filter(({ entry }) => idle(entry)); + if (waiting.length) openDownloadDialog(owner, waiting, settings); + })); + } + draw(); + let finished = doneCount(); + const unwatch = watchDownloads((jobs) => { + if (!panel.isConnected || panel.style.display === 'none') return; + const done = [...jobs.values()].filter(job => job.state === 'done').length; + if (done !== finished) { + finished = done; + rerender(); + } else { + draw(); + } + }); + panel._downloadUnwatch = unwatch; +} diff --git a/web/modules/ui_scan_page.js b/web/modules/ui_scan_page.js index 657893d..e0b5dcd 100644 --- a/web/modules/ui_scan_page.js +++ b/web/modules/ui_scan_page.js @@ -176,8 +176,8 @@ function settingsPage(owner, panel, hasKey, progressHost) { return page; } -/** Asks for the key and saves it; true when saved. */ -async function saveApiKey() { +/** Asks for the Civitai key and saves it; true when saved (also Model Check's downloads). */ +export async function saveApiKey() { const value = prompt(t('scanPageApiKeyPrompt'), ''); if (value === null) return false; try { diff --git a/web/modules/ui_settings_page.js b/web/modules/ui_settings_page.js index 9223067..fa79678 100644 --- a/web/modules/ui_settings_page.js +++ b/web/modules/ui_settings_page.js @@ -2,7 +2,7 @@ * The settings page (the rail's gear): one card per subject — how the browser looks, * model cards and what they cost in memory (with the card image cache), model folders * (a view of its own, ui_folder_manager.js, `owner.settingsView === 'folders'`), - * workflows, how the browser opens, and help. A tool page like the scan page, with a + * workflows, where Model Check's downloads go, how the browser opens, and help. A tool page like the scan page, with a * way back to the page you came from. Display preferences go through `owner.displayPrefs` * (ui_settings_hub.js); language, theme and opening mode are ComfyUI settings. */ @@ -17,6 +17,8 @@ import { copyDiagnostics } from './feedback.js'; import { renderFolderPage } from './ui_folder_manager.js'; import { openFeedbackDialog } from './ui_feedback_dialog.js'; import { checkOnOpen, setCheckOnOpen } from './ui_doctor_banner.js'; +import { fetchDownloadSettings, saveDownloadSettings } from './model_download.js'; +import { saveApiKey } from './ui_scan_page.js'; const t = (key, params) => translate(key, params); // The full written guide: the README, at its Chinese half for Chinese. @@ -196,6 +198,30 @@ function workflowGroup(owner) { }))); } +/** Where Model Check's downloads go (model_download.js); filled once the settings are read. */ +function downloadGroup(owner, redraw) { + const box = group('settingsDownloads'); + fetchDownloadSettings().then(settings => { + const folder = el('input', 'anomalous-download-folder'); + folder.type = 'text'; + folder.spellcheck = false; + folder.value = settings.folder; + folder.placeholder = t('settingsDownloadFolderRoot'); + folder.onchange = () => saveDownloadSettings({ folder: folder.value }) + .then(saved => { folder.value = saved.folder; folder.classList.remove('is-bad'); }) + .catch(() => folder.classList.add('is-bad')); + box.append( + row('settingsDownloadPlace', 'settingsDownloadPlaceHelp', segment( + [['workflow', 'settingsDownloadPlaceWorkflow'], ['folder', 'settingsDownloadPlaceFolder']], settings.place, + value => saveDownloadSettings({ place: value }).then(redraw).catch(() => {}))), + row('settingsDownloadFolder', 'settingsDownloadFolderHelp', folder), + row('settingsDownloadKey', 'settingsDownloadKeyHelp', button('anomalous-scan-secondary anomalous-scan-small-btn', + t('settingsDownloadKeySet'), () => saveApiKey())), + ); + }).catch(error => console.warn('[AMB] Settings: download settings unavailable.', error)); + return box; +} + function openingGroup(owner, redraw) { const mode = comfySetting(ENTRY_MODE_SETTING_ID) || 'floating'; return group('settingsOpening', @@ -255,6 +281,7 @@ function render(owner) { cardsGroup(owner, redraw), foldersGroup(owner), workflowGroup(owner), + downloadGroup(owner, redraw), openingGroup(owner, redraw), helpGroup(owner), ); diff --git a/web/styles.css b/web/styles.css index 8610d5d..eed7154 100644 --- a/web/styles.css +++ b/web/styles.css @@ -20,3 +20,4 @@ @import url("./styles/19-model-doctor.css?v=20261005-mute-1"); @import url("./styles/20-current-node.css?v=20261004-studio-1"); @import url("./styles/21-prompt-studio.css?v=20261004-draftx-1"); +@import url("./styles/22-model-download.css?v=20261006-download-1"); diff --git a/web/styles/22-model-download.css b/web/styles/22-model-download.css new file mode 100644 index 0000000..bb3e472 --- /dev/null +++ b/web/styles/22-model-download.css @@ -0,0 +1,182 @@ +/* Model Check's downloads: the row's button and progress, and the dialog (ui_model_download.js). */ + +.anomalous-download-slot:empty { display: none; } + +.anomalous-download-slot, +.anomalous-download-status { + display: inline-flex; + align-items: center; + flex-wrap: wrap; + gap: 6px; +} + +.anomalous-download-text { + color: var(--amb-text-muted); + font-size: 12px; + line-height: 1.4; +} + +.anomalous-download-text.is-bad { color: #fca5a5; } + +.anomalous-download-bar { + position: relative; + width: 56px; + height: 5px; + border-radius: 999px; + background: var(--amb-border); + overflow: hidden; +} + +.anomalous-download-bar::after { + content: ""; + position: absolute; + inset: 0 auto 0 0; + width: var(--amb-download-done, 0%); + background: var(--amb-link); + transition: width 0.4s ease; +} + +.anomalous-download-overlay { + position: fixed; + inset: 0; + z-index: 1000001; + display: flex; + align-items: center; + justify-content: center; + padding: 20px; + box-sizing: border-box; + background: rgba(0, 0, 0, 0.6); + backdrop-filter: blur(4px); +} + +.anomalous-download-dialog { + display: flex; + flex-direction: column; + gap: 14px; + width: min(620px, 100%); + max-height: 90vh; + padding: 22px 24px; + box-sizing: border-box; + overflow-y: auto; + border: 1px solid var(--amb-border-strong); + border-radius: 16px; + background: var(--amb-bg-panel, #24252b); + color: var(--amb-text-main); + box-shadow: 0 24px 70px rgba(0, 0, 0, 0.55); +} + +.anomalous-download-title { + margin: 0; + font-size: 1.15rem; +} + +.anomalous-download-items { + display: flex; + flex-direction: column; + gap: 10px; +} + +.anomalous-download-item { + display: flex; + flex-direction: column; + gap: 6px; + padding: 12px; + border: 1px solid var(--amb-border); + border-radius: 10px; + background: var(--amb-bg-card); +} + +.anomalous-download-head { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: 10px; + min-width: 0; +} + +.anomalous-download-name { + min-width: 0; + overflow-wrap: anywhere; +} + +.anomalous-download-facts, +.anomalous-download-model { + flex-shrink: 0; + color: var(--amb-text-muted); + font-size: 12px; +} + +.anomalous-download-model { flex-shrink: 1; } + +.anomalous-download-where { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 6px; + font-size: 12.5px; +} + +.anomalous-download-label { color: var(--amb-text-soft, var(--amb-text-muted)); } + +.anomalous-download-root, +.anomalous-download-rel, +.anomalous-download-folder { + min-width: 0; + padding: 5px 8px; + border: 1px solid var(--amb-border-strong); + border-radius: 8px; + background: var(--amb-bg-input, rgba(0, 0, 0, 0.25)); + color: var(--amb-text-main); + font: inherit; + font-size: 12.5px; +} + +.anomalous-download-root { max-width: 100%; } +.anomalous-download-rel { flex: 1 1 240px; } +.anomalous-download-folder { width: 220px; } +.anomalous-download-folder.is-bad { border-color: #fca5a5; } + +.anomalous-download-path { + color: var(--amb-text-dim, var(--amb-text-muted)); + font-size: 11.5px; + overflow-wrap: anywhere; +} + +.anomalous-download-note { + color: var(--amb-text-muted); + font-size: 12px; + line-height: 1.5; +} + +.anomalous-download-note.is-warn { color: #fcd34d; } +.anomalous-download-note.is-bad { color: #fca5a5; } + +.anomalous-download-space { + color: var(--amb-text-muted); + font-size: 12.5px; +} + +.anomalous-download-space.is-bad { color: #fca5a5; } + +.anomalous-download-rules { + align-self: flex-start; + padding: 0; + border: none; + background: none; + color: var(--amb-link); + font-size: 12px; + cursor: pointer; +} + +.anomalous-download-rules:hover { text-decoration: underline; } + +.anomalous-download-footer { + display: flex; + justify-content: flex-end; + gap: 10px; +} + +@media (max-width: 520px) { + .anomalous-download-head { flex-direction: column; gap: 2px; } + .anomalous-download-dialog { padding: 16px; } +}