From c41af60cd9cdf18079acd7dbbb74f523bfef59ee Mon Sep 17 00:00:00 2001 From: DemonGatanjieu <207458273+DemonGatanjieu@users.noreply.github.com> Date: Fri, 23 Oct 2026 19:50:57 +0800 Subject: [PATCH] Tell the AI what loads a model, file model patches right, add gallery pictures to messages - model_kind: Anima LLLite, Qwen-Image block-wise and Z-Image Fun ControlNets (and other ModelPatchLoader files) go to model_patches, not controlnet; import and Tidy follow. - api/mcp_model_use.py: search_models / get_model say which node types load each model, or where a misfiled one belongs; move_model moves it there behind a new "files" gate (asks by default, never overwrites). - An empty drop-down in add_node says ComfyUI found no file; the task tab switch is no longer logged as the user's canvas change. - The folded task list stays while the pointer is in the box around its button and the list. - The message box's picture button picks gallery pictures; "@" offers the gallery's newest and searches it as you type. Co-Authored-By: Claude Opus 5.5 --- ARCHITECTURE.md | 16 ++- CHANGELOG.md | 10 ++ api/action_catalog.py | 5 +- api/ai_gates.py | 8 +- api/mcp_canvas_edit.py | 12 ++- api/mcp_model_use.py | 166 ++++++++++++++++++++++++++++++ api/mcp_server.py | 3 +- api/mcp_task_tools.py | 4 +- api/mcp_tools.py | 17 +-- api/model_import.py | 2 +- api/model_placement.py | 19 +++- docs/guides/mcp.md | 9 +- model_kind.py | 9 +- tests/test_ai_toolbox.py | 2 +- tests/test_mcp.py | 29 ++++-- tests/test_mcp_canvas_edit.py | 2 +- tests/test_mcp_model_use.py | 90 ++++++++++++++++ tests/test_model_import.py | 5 + tests/test_model_placement.py | 11 +- web/modules/canvas_build.js | 1 + web/modules/locales.js | 42 ++++++++ web/modules/mcp_bridge.js | 1 + web/modules/model_policies.js | 1 + web/modules/ui_ai_gallery_pick.js | 164 +++++++++++++++++++++++++++++ web/modules/ui_ai_gate.js | 9 +- web/modules/ui_ai_mention.js | 60 +++++++++-- web/modules/ui_ai_message.js | 18 +++- web/modules/ui_ai_task_list.js | 51 ++++++--- web/modules/ui_model_import.js | 4 +- web/styles.css | 2 +- web/styles/46-ai-mention.css | 81 +++++++++++++++ 31 files changed, 769 insertions(+), 84 deletions(-) create mode 100644 api/mcp_model_use.py create mode 100644 tests/test_mcp_model_use.py create mode 100644 web/modules/ui_ai_gallery_pick.js diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index dee2849..ee308a7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -161,6 +161,10 @@ covered by the `styles.css` manifest. ComfyUI's websocket and waits for its answer; `web/modules/mcp_bridge.js` is the page side, doing it with Anomalous's canvas code (one Ctrl+Z step each, logged as the AI's; the outcome carries the log entry's id as `_activity`, for the AI workbench's undo). + `api/mcp_model_use.py` offers search_models and get_model (mcp_tools.py finds the files) + with what loads each model: the node types whose drop-down lists it (from `/object_info`), + or, for a file whose header says it belongs in another folder (model_placement.py), where, + and `move_model` (behind the "files" gate) moves it there as Tidy does. `api/mcp_canvas_edit.py` holds the tools that build a workflow where the user watches: `find_nodes` reads this ComfyUI's node types from its own `/object_info` (kept two minutes); `add_node`, `connect_nodes`, `set_values`, `remove_nodes` and `new_workflow` (a @@ -267,9 +271,10 @@ covered by the `styles.css` manifest. and the rail's dot (`owner.railNews`). A task's parts stay in the page between redraws: a step is drawn again only when it changed (`syncRows`), so a scrolled diagram or strip and half-typed text survive the AI's steps coming in. `ui_ai_message.js` is the message box under - a task (made once per task); `ui_ai_mention.js` gives it "@" (the task's pictures, combos, - models, workflows, prompts, node packs and node types from the materials picker's data, chips - of what is named) and "/" (`ui_ai_commands.js`: stop, undo, side views, new task, canvas mode, + a task (made once per task); `ui_ai_mention.js` gives it "@" (the task's pictures, the + gallery's newest and those its search finds, combos, models, workflows, prompts, node packs + and node types from the materials picker's data, chips of what is named), and + `ui_ai_gallery_pick.js` its picture button (gallery pictures picked into the message) and "/" (`ui_ai_commands.js`: stop, undo, side views, new task, canvas mode, words to fill in); `api/ai_message_refs.py` checks what the "@" name and gives each to the AI with the message (in full, pictures attached; get_toolbox shares its `lookup`), `tests/test_ai_message_refs.py`. `ui_ai_step_body.js` is what a timeline step says under its @@ -380,7 +385,7 @@ covered by the `styles.css` manifest. and the scanned models, never replaces a file (" (2)"). - `api/model_placement.py` is the models page's Tidy check (`/anomalous/placement/*`): models whose header names another type than their folder, identical files (by SHA-256), and moving - a model with its sidecars to where it belongs. + a model with its sidecars to where it belongs; `belongs()` checks one file for the AI. - `api/model_usage.py` is Tidy's "not used lately" (`GET /anomalous/usage/check`, this computer only, on a press): when each model and node pack was last used, from the run record, the PNG outputs' prompts (image_search's cached records, which keep the node types too), FLAC / @@ -424,7 +429,8 @@ covered by the `styles.css` manifest. tells, shared with the standalone scanner (which runs as its own process, so the API never imports `scraper.py`). `model_kind.py` reads a .safetensors / .gguf header (never formats that can carry code) and says which models folder the file belongs in and its base - model; model import and the scanner's offline base model use it. + model (model patches such as Anima LLLite go to `model_patches`, not `controlnet`); model + import and the scanner's offline base model use it. - `recycle_bin.py` is the only way the plugin deletes the user's files (models, covers, outputs, recipes, materials, notes, audio): to the system Recycle Bin, together, or not at all (`TrashUnavailable` on drives without one). `api/trash.py` is the API's import of it diff --git a/CHANGELOG.md b/CHANGELOG.md index 59718d9..0387775 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,16 @@ - 右边这一栏改成一张单独的卡片，不再用一条线硬隔开；默认窄一些，给中间的时间线留出地方。鼠标移到它左边缘会出现小竖条，拖动调宽窄（记得住，双击恢复）。 - 左边的任务列表用窗口左上角的按钮收起 / 展开；收起后鼠标移到按钮上，列表会临时浮出来，点任务或移开就收回。 - 素材在右边这一栏里时，鼠标移过卡片不再自动跳进详细说明（之前很难操作），点卡片上的 ⓘ 才看。 +- 左边任务列表收起后，鼠标从按钮往下移去点任务时列表不会再中途消失（之前移得慢一点就会收回）：鼠标在按钮和列表围成的范围里都算还在用，离开后才收回。 +- The folded task list no longer vanishes on the way down from its button: it stays while the pointer is anywhere in the box around the button and the list, however slowly it moves. +- **模型补丁不再被当成 ControlNet**：Anima LLLite、Qwen-Image 分块 ControlNet、Z-Image Fun ControlNet 这类文件名字里常带 ControlNet，但只有「加载模型补丁」节点能读，它只看 `model_patches` 文件夹。现在拖进模型页会自动放进 `model_patches`；已经放在 `controlnet` 里的，模型页的“整理”会提示挪过去。 +- Model patches (Anima LLLite, the Qwen-Image block-wise and Z-Image Fun ControlNets) are told from ControlNets by their header: imported into `model_patches`, and Tidy offers to move those already in `controlnet`. +- **AI 找模型时直接知道该用哪个节点**：`search_models` / `get_model` 给每个模型写上能加载它的节点；放错文件夹的会说该放哪、放在这里加载不了，同名文件已经在对的地方时会直接指给 AI。新工具 `move_model` 把放错的模型挪过去（新关卡“挪动模型文件”，默认先问你；从不覆盖文件）。AI 不必再去命令行翻 ComfyUI 的源码。AI 加节点时如果下拉列表是空的，会被告知“ComfyUI 没找到文件，可能放错了文件夹”。 +- search_models and get_model say which node types load each model, and where a misfiled one belongs; `move_model` moves it there behind a new "files" gate (asks by default, never overwrites). An empty drop-down in add_node now says ComfyUI found no file for it. +- AI 第一次在任务专用标签页上动手时，切换标签页不再被记成“你在画布上的改动”告诉 AI。 +- The AI switching to its task tab is no longer reported to it as a canvas change of the user's. +- **工作台里直接用图库的图**：留言框左边新增图片按钮，打开图库（最新的在前，可按提示词、模型、seed 搜索），点选几张后 **加进消息**；打 **@** 时也能看到图库最新的图，打字会顺带在图库里搜。AI 收到的是图片本身和它的生成参数。 +- Gallery pictures in the AI workbench: the message box's picture button picks them from the gallery (newest first, searchable), and "@" offers the gallery's newest pictures and searches it as you type; the AI gets each picture with what made it. - 右边那一栏的宽度把手更好抓了：卡片左边整条空隙（从上到下）都能拖，把手平时就淡淡地显示，鼠标靠近时整条边亮起；点一下后也能用 ← → 调宽窄。 - The side column's grip catches the pointer across the whole gap left of the card, top to bottom, shows faintly at rest and lights the edge when pointed at; focused, ← → resize it. - **节点结构的简单看法**：除了完整的方块图，现在默认是一张一眼能看懂的说明：每一段采样用哪个模型加哪些 LoRA（带权重）、正负提示词、从空白画布还是从上一段的结果开始（用哪个放大模型、缩放多少），最后保存在哪；和上一段一样的就写“和第 1 段一样”。点任意一项定位到画布上的节点；“简单 / 完整”随时切换，AMB 记得你的选择。 diff --git a/api/action_catalog.py b/api/action_catalog.py index 2524954..bd0b4d5 100644 --- a/api/action_catalog.py +++ b/api/action_catalog.py @@ -4,7 +4,7 @@ Every tool (mcp_tools.py reads, mcp_actions.py acts, mcp_task_tools.py the AI's entry here: its kind (the AI workbench's colour, words and counts; None = not a step of its own), the stage of the work it belongs to (the flow over a task: models, combo, prompt, canvas, run, pictures, voice, library) and its gate (what it touches or costs, for the user's -rules on what needs their OK: canvas, gpu, download, network; None = nothing to guard). +rules on what needs their OK: canvas, gpu, download, network, files; None = nothing to guard). tests/test_action_catalog.py keeps this catalog and the tools in step. The user's own changes in Anomalous (activity log actions) get stages too, so the ones made @@ -16,7 +16,7 @@ from collections import namedtuple Entry = namedtuple("Entry", "kind stage gate") STAGES = ("model", "combo", "prompt", "canvas", "run", "picture", "voice", "library") -GATES = ("canvas", "gpu", "download", "network") +GATES = ("canvas", "gpu", "download", "network", "files") TOOLS = { # reading @@ -51,6 +51,7 @@ TOOLS = { "run_workflow": Entry("run", "run", "gpu"), "download_missing_models": Entry("download", "model", "download"), "download_model": Entry("download", "model", "download"), + "move_model": Entry("act", "model", "files"), "scan_models": Entry("scan", "library", "network"), "speak": Entry("speech", "voice", "gpu"), "open_in_anomalous": Entry("show", "model", None), diff --git a/api/ai_gates.py b/api/ai_gates.py index 449c8ee..65f1a83 100644 --- a/api/ai_gates.py +++ b/api/ai_gates.py @@ -1,6 +1,6 @@ """Gates: what the user lets AI apps do through Anomalous's MCP endpoint without asking first. -Each acting tool has a gate in action_catalog.py (canvas, gpu, download, network) and each gate +Each acting tool has a gate in action_catalog.py (canvas, gpu, download, network, files) and each gate a policy the user sets on the AI workbench: "auto" (go ahead), "ask" (wait for the user's OK) or "never". mcp_server.py asks check() before running a tool. With "ask" the tool's step waits in state "asking", showing what it would do, for up to WAIT_SECONDS: the user's Allow lets the @@ -20,7 +20,7 @@ from . import action_catalog, ai_tasks from .path_utils import atomic_write_json POLICIES = ("auto", "ask", "never") -DEFAULTS = {"canvas": "auto", "gpu": "auto", "download": "ask", "network": "auto"} +DEFAULTS = {"canvas": "auto", "gpu": "auto", "download": "ask", "network": "auto", "files": "ask"} WAIT_SECONDS = 50 # below the minute many AI apps give a tool call ALLOWED_SECONDS = 30 * 60 # an Allow the AI did not wait for holds this long for the same call MAX_ARGS = 8 @@ -58,8 +58,8 @@ def describe(tool, arguments): for key, value in list((arguments or {}).items())[:MAX_ARGS]: if key == "task": continue - if key == "model_id": - value = ai_tasks._model_result(value).get("name") or value + if key == "model_id": # its name, and the folder it is in + value = " · ".join(filter(None, (ai_tasks._model_result(value).get("name") or value, str(value).split(":")[0]))) out.append({"label": key, "value": ai_tasks._short(value, 400)}) return out diff --git a/api/mcp_canvas_edit.py b/api/mcp_canvas_edit.py index 0c98996..25fbf84 100644 --- a/api/mcp_canvas_edit.py +++ b/api/mcp_canvas_edit.py @@ -28,7 +28,7 @@ _info = {"at": 0.0, "data": None} _info_lock = threading.Lock() -def _object_info(ctx): +def object_info(ctx): """ComfyUI's node definitions, as its own page reads them (kept a little while).""" with _info_lock: if _info["data"] is None or time.monotonic() - _info["at"] > INFO_SECONDS: @@ -39,6 +39,12 @@ def _object_info(ctx): return _info["data"] +def forget_object_info(): + """After a model moved: the next look reads ComfyUI's lists anew.""" + with _info_lock: + _info["data"] = None + + def _input(name, conf, required, choices): kind, options = (conf + [None, None])[:2] if isinstance(conf, list) else (conf, None) options = options if isinstance(options, dict) else {} @@ -68,7 +74,7 @@ def _allowed(ctx, types): """Refuses node types the user's "only" picks leave out, before the page is asked.""" if not ctx.get("task"): return - info = _object_info(ctx) + info = object_info(ctx) refused = ai_toolbox.refused(ctx["task"], types, info) if refused: box = ai_toolbox.effective(ai_tasks.get_task(ctx["task"])) or {} @@ -93,7 +99,7 @@ def _node(type_name, info, choices): def find_nodes(ctx, query="", types=None): - info = _object_info(ctx) + info = object_info(ctx) if types: missing = [name for name in types if name not in info] found = [_node(name, info[name], MAX_CHOICES_NAMED) for name in types if name in info][:MAX_FOUND] diff --git a/api/mcp_model_use.py b/api/mcp_model_use.py new file mode 100644 index 0000000..733fd81 --- /dev/null +++ b/api/mcp_model_use.py @@ -0,0 +1,166 @@ +"""Where a model loads, for the AI: search_models and get_model (mcp_tools.py finds the files) +say which node types list each file among their choices (ComfyUI's /object_info, kept a little +while by mcp_canvas_edit.py), and, when a file's header says it is another kind than the folder +it is in (model_placement.py, the models page's Tidy view), where it belongs and that nothing +loads it where it is. So the AI picks the right node at once, or has the model moved, instead of +searching ComfyUI itself. move_model moves one model with its covers and notes as Tidy does, +behind the "files" gate (the user says yes first unless they let it go ahead).""" + +import os +import threading + +import folder_paths + +from . import mcp_canvas_edit, mcp_tools, model_placement +from .mcp_actions import _action +from .mcp_tools import _LIMIT, _QUERY, ToolError +from .model_constants import MODEL_EXTENSIONS +from .model_import import Refused, import_types + +MAX_LOADERS = 6 + +_lock = threading.Lock() +_index = {"of": None, "inputs": [], "by_choice": {}} + + +def _plain(name): + return str(name).replace("\\", "/") + + +def _choices(conf): + if not isinstance(conf, list) or not conf: + return None + if isinstance(conf[0], list): + return conf[0] + options = conf[1] if len(conf) > 1 and isinstance(conf[1], dict) else {} + return options.get("options") if conf[0] == "COMBO" and isinstance(options.get("options"), list) else None + + +def _built(info): + """The drop-downs of model files in node definitions `info`: [(node type, choices)] and choice -> their places.""" + with _lock: + if _index["of"] is info: + return _index + inputs, by_choice = [], {} + for node_type, item in info.items(): + for section in ("required", "optional"): + for conf in ((item.get("input") or {}).get(section) or {}).values(): + listed = [_plain(choice) for choice in _choices(conf) or [] if isinstance(choice, str)] + if not any(choice.lower().endswith(MODEL_EXTENSIONS) for choice in listed): + continue + for choice in listed: + by_choice.setdefault(choice, []).append(len(inputs)) + inputs.append((node_type, frozenset(listed))) + _index.update(of=info, inputs=inputs, by_choice=by_choice) + return _index + + +def _folder(folder_type): + return frozenset(_plain(name) for name in folder_paths.get_filename_list(folder_type)) + + +def _readers(ctx, folder_type, rel=None): + """Node types with a drop-down of the whole folder `folder_type` (one that lists `rel`, when given).""" + index = _built(mcp_canvas_edit.object_info(ctx)) + folder = _folder(folder_type) + if not folder: + return [] + places = index["by_choice"].get(_plain(rel), []) if rel else range(len(index["inputs"])) + found = [] + for at in places: + node_type, choices = index["inputs"][at] + if folder <= choices and node_type not in found: + found.append(node_type) + return found[:MAX_LOADERS] + + +def _about(ctx, model): + """`model` (from mcp_tools) with what loads it: loaded_by, or belongs_in and placement when it is misfiled.""" + folder_type, root, base_dir, path = mcp_tools._resolve_model(model["model_id"]) + rel = os.path.relpath(path, base_dir).replace(os.sep, "/") + home = model_placement.belongs(model_placement.one(folder_type, root, rel, path)) + if not home or home["works"]: + model["loaded_by"] = _readers(ctx, folder_type, rel) + return model + kind, to = home["kind"], home["to"] + readers = _readers(ctx, kind) + words = (f"Its header says it is a {kind} file, but it is in {folder_type}: the nodes listing it there cannot load it" + + (f"; {', '.join(readers)} load {kind} files." if readers else ".")) + there = os.path.join(folder_paths.get_folder_paths(kind)[to["root"]], *to["rel"].split("/")) + if os.path.isfile(there): + words += (f" A file of the same name is already in {kind} (model_id {kind}:{to['root']}:{to['rel']}): use that one. " + "If it is the same file, this copy is extra; the user can recycle it in Anomalous's models page (Tidy).") + else: + words += f" move_model with to_type {kind} moves it there (the user confirms first)." + model.update(belongs_in=kind, placement=words) + return model + + +def search_models(ctx, query="", type="", base_model="", source="", limit=20): + found = mcp_tools.search_models(query, type, base_model, source, limit) + for model in found["models"]: + _about(ctx, model) + return found + + +def get_model(ctx, model_id, include_cover=False): + detail, extra = mcp_tools.get_model(model_id, include_cover) + return _about(ctx, detail), extra + + +def move_model(ctx, model_id, to_type): + folder_type, root, base_dir, path = mcp_tools._resolve_model(model_id) + rel = os.path.relpath(path, base_dir).replace(os.sep, "/") + item = model_placement.one(folder_type, root, rel, path) + home = model_placement.belongs(item) + to_type = str(to_type or "") + if to_type == folder_type: + raise ToolError(f"It is in {to_type} already.") + if to_type not in import_types(): + raise ToolError(f"to_type is one of {', '.join(import_types())}.") + to = home["to"] if home and home["kind"] == to_type else model_placement.home_for(item, to_type) + if not to: + raise ToolError(f"This ComfyUI has no {to_type} folder.") + if os.path.exists(os.path.join(folder_paths.get_folder_paths(to_type)[to["root"]], *to["rel"].split("/"))): + raise ToolError(f"A file of the same name is already in {to_type} (model_id {to_type}:{to['root']}:{to['rel']}); " + "nothing was moved. Use that one; the user can recycle an extra copy in Anomalous's models page (Tidy).") + try: + moved = model_placement.move({"type": folder_type, "root": root, "rel": rel, + "to_type": to_type, "to_root": to["root"], "to_rel": to["rel"]}) + except Refused as error: + raise ToolError(f"Not moved ({error.code}).") + mcp_canvas_edit.forget_object_info() + answer = {"moved": {"from": f"{folder_type}/{rel}", "to": f"{moved['type']}/{moved['rel']}"}, + "model_id": f"{moved['type']}:{moved['root']}:{moved['rel']}", + "loaded_by": _readers(ctx, moved["type"], moved["rel"])} + if moved["left"]: + answer["left_behind"] = moved["left"] # covers or notes that could not go along + return answer + + +MODEL_TOOLS = dict([ + _action("search_models", "Search models", + "Find model files (checkpoints, LoRAs, VAEs…) by name, trigger word or base model. Returns model_id for " + "get_model, and loaded_by: the node types whose drop-down lists the file (use one of them). A file in the " + "wrong folder for what it is comes with belongs_in and placement instead: nothing loads it where it is.", + search_models, { + "query": _QUERY, + "type": {"type": "string", "description": "ComfyUI models folder, e.g. checkpoints, loras, vae, diffusion_models, model_patches."}, + "base_model": {"type": "string", "description": "Part of the base model name, e.g. SDXL, Pony, Flux, SD 1.5."}, + "source": {"type": "string", "enum": ["civitai", "inferred_from_file", "not_scanned"], + "description": "Where the model's information came from."}, + "limit": _LIMIT}, read_only=True), + _action("get_model", "Model details", + "Everything known about one model: Civitai name and link, trigger words, base model, description, " + "the user's own name and notes, and loaded_by (or belongs_in and placement) as in search_models. " + "include_cover attaches its cover picture.", + get_model, {"model_id": {"type": "string", "description": "From search_models."}, + "include_cover": {"type": "boolean", "default": False}}, ["model_id"], read_only=True), + _action("move_model", "Move a model to its folder", + "Moves a model the user has in the wrong models folder (search_models says belongs_in) to where it loads, " + "with its covers and notes, as Anomalous's Tidy view does; never over a file. The answer gives its new " + "model_id and loaded_by; the canvas lists it from then on.", + move_model, {"model_id": {"type": "string", "description": "From search_models."}, + "to_type": {"type": "string", "description": "The models folder to move it to: belongs_in from search_models."}}, + ["model_id", "to_type"], destructive=True), +]) diff --git a/api/mcp_server.py b/api/mcp_server.py index b429e40..d5f5e05 100644 --- a/api/mcp_server.py +++ b/api/mcp_server.py @@ -27,10 +27,11 @@ from . import ai_gates, ai_panels, ai_run_tracking, ai_task_canvas, ai_task_runs from .local_guard import is_local as _is_local, loopback_host as _loopback_host from .mcp_actions import ACTIONS from .mcp_canvas_edit import CANVAS_TOOLS +from .mcp_model_use import MODEL_TOOLS from .mcp_task_tools import TASK_TOOLS from .mcp_tools import TOOLS as READ_TOOLS, call_tool -TOOLS = {**READ_TOOLS, **ACTIONS, **CANVAS_TOOLS, **TASK_TOOLS, **mcp_apps.APP_TOOLS} +TOOLS = {**READ_TOOLS, **MODEL_TOOLS, **ACTIONS, **CANVAS_TOOLS, **TASK_TOOLS, **mcp_apps.APP_TOOLS} CAPABILITIES = { "tools": {"listChanged": False}, "prompts": {"listChanged": False}, "resources": {"listChanged": False}, "extensions": {mcp_apps.EXTENSION: {"mimeTypes": [mcp_apps.MIME]}}, # the card in the conversation (mcp_apps.py) diff --git a/api/mcp_task_tools.py b/api/mcp_task_tools.py index 7a7536f..e79979c 100644 --- a/api/mcp_task_tools.py +++ b/api/mcp_task_tools.py @@ -177,9 +177,9 @@ def get_toolbox(ctx, task=None): if found.get("not_found"): answer["page_only_nodes"] = found["not_found"] # helpers that live only in the page: add_node takes them if box.get("packs"): - from .mcp_canvas_edit import _object_info + from .mcp_canvas_edit import object_info - info = _object_info(ctx) + info = object_info(ctx) answer["packs"] = [] for pack in box["packs"]: types = sorted(name for name, item in info.items() if ai_toolbox.source_of(item) == pack) diff --git a/api/mcp_tools.py b/api/mcp_tools.py index 8e14261..5f25055 100644 --- a/api/mcp_tools.py +++ b/api/mcp_tools.py @@ -1,6 +1,7 @@ """The MCP endpoint's (mcp_server.py) reading tools: views of the library (mcp_actions.py acts). -`call_tool` runs any tool of a registry, these and mcp_actions.py's. +`call_tool` runs any tool of a registry, these and mcp_actions.py's. search_models and get_model +are offered through mcp_model_use.py, which adds what loads each model. Each tool is a plain function over the same stores the browser's pages use (model catalog and sidecars, output gallery, combos, workflow recipes, saved prompts, generated audio, scan @@ -444,20 +445,6 @@ TOOLS = dict([ _tool("library_overview", "Library overview", "How many models of each type and base model the user has, how many are scanned, and how many " "combos, workflow recipes and saved prompts they kept. A good first call.", library_overview), - _tool("search_models", "Search models", - "Find model files (checkpoints, LoRAs, VAEs…) by name, trigger word or base model. Returns model_id for get_model.", - search_models, { - "query": _QUERY, - "type": {"type": "string", "description": "ComfyUI models folder, e.g. checkpoints, loras, vae, diffusion_models."}, - "base_model": {"type": "string", "description": "Part of the base model name, e.g. SDXL, Pony, Flux, SD 1.5."}, - "source": {"type": "string", "enum": ["civitai", "inferred_from_file", "not_scanned"], - "description": "Where the model's information came from."}, - "limit": _LIMIT}), - _tool("get_model", "Model details", - "Everything known about one model: Civitai name and link, trigger words, base model, description, " - "the user's own name and notes. include_cover attaches its cover picture.", - get_model, {"model_id": {"type": "string", "description": "From search_models."}, - "include_cover": {"type": "boolean", "default": False}}, ["model_id"]), _tool("scan_report", "Scan status", "Which models are matched on Civitai, which are not and why, which were never scanned, and what the last scan did.", scan_report), diff --git a/api/model_import.py b/api/model_import.py index a46fd4e..2989709 100644 --- a/api/model_import.py +++ b/api/model_import.py @@ -44,7 +44,7 @@ from .model_download import ( from .path_utils import resolve_within KINDS = ("checkpoints", "loras", "vae", "text_encoders", "diffusion_models", "controlnet", "embeddings", - "clip_vision", "upscale_models", "ipadapter") + "clip_vision", "upscale_models", "ipadapter", "model_patches") SEARCH_DEPTH = 2 SEARCH_LIMIT = 20000 TIME_SLACK_MS = 2000 diff --git a/api/model_placement.py b/api/model_placement.py index f6eb91d..eea1877 100644 --- a/api/model_placement.py +++ b/api/model_placement.py @@ -12,7 +12,8 @@ POST /anomalous/placement/move {type, root, rel, to_type, to_root, to_rel} -> mo and its sidecars (covers, info, the user's notes); never over a file (" (2)"). Only for the computer ComfyUI runs on, unless remote changes are allowed (local_guard.may_change). -Removing a copy goes through /anomalous/delete_model (Recycle Bin). Headers and computed +belongs() is the same check for one file (the AI's search_models and move_model, +api/mcp_model_use.py). Removing a copy goes through /anomalous/delete_model (Recycle Bin). Headers and computed SHA-256s are kept in memory per file (path, size, time). """ @@ -116,7 +117,7 @@ def _card(item): return {"name": name or stem, "version": version, "preview_url": url} -def _home_for(item, kind): +def home_for(item, kind): """Where a file of `kind` goes: its own subfolder and name, in a folder of that type, preferably on its drive (a move is then a rename).""" roots = [] @@ -136,7 +137,7 @@ def misplaced(files): kind = result.get("kind") if not kind or not result.get("sure") or kind == item["type"] or kind not in import_types(): continue - home = _home_for(item, kind) + home = home_for(item, kind) if home: found.append({"type": item["type"], "root": item["root"], "rel": item["rel"], "size": item["size"], "kind": kind, "base": result.get("base", ""), "works": (item["type"], kind) in WORKS_ANYWAY, @@ -144,6 +145,18 @@ def misplaced(files): return found +def one(folder_type, root, rel, path): + """One model file as library() lists it.""" + stat = os.stat(path) + return {"type": folder_type, "root": root, "rel": rel, "path": path, "size": stat.st_size, "dev": stat.st_dev, "stat": stat} + + +def belongs(item): + """misplaced() of one file (from one()): where it belongs, or None when it is where it goes or its kind is unsure.""" + found = misplaced([item]) + return found[0] if found else None + + def _known_hash(item): try: value = str(get_metadata(item["path"]).get("hash") or "").lower() diff --git a/docs/guides/mcp.md b/docs/guides/mcp.md index 0f84742..565e9b3 100644 --- a/docs/guides/mcp.md +++ b/docs/guides/mcp.md @@ -85,7 +85,7 @@ claude mcp add --transport http anomalous http://127.0.0.1:8188/anomalous/mcp | 工具 | 查什么 | |---|---| | library_overview | 各类模型的数量、底模分布、扫描情况，存了多少搭配、工作流和提示词 | -| search_models / get_model | 按名字、触发词、底模、来源找模型；看某个模型的 C 站信息、描述、你写的备注，可以附上封面 | +| search_models / get_model | 按名字、触发词、底模、来源找模型；看某个模型的 C 站信息、描述、你写的备注，可以附上封面。每个模型都写明哪些节点能加载它；放错文件夹的（比如名字叫 ControlNet、其实是模型补丁的 Anima LLLite）会说明该放哪、放在这里加载不了 | | scan_report | 哪些模型匹配上了 C 站，哪些没有、原因是什么，哪些还没扫，上次扫描做了什么 | | search_images / get_image_info | 按提示词、模型、参数找出图；看某张图的全部生成参数，可以附上缩略图 | | list_combos | 你的搭配（主模型 + LoRA + 提示词） | @@ -110,6 +110,7 @@ claude mcp add --transport http anomalous http://127.0.0.1:8188/anomalous/mcp | download_missing_models | 下载当前工作流缺的模型：只下能确定来源的（C 站按指纹找到、或工作流带着文件链接），放在设置里“模型下载”说的位置，核对指纹后自动换上 | | download_model | 按链接（C 站模型版本、Hugging Face / GitHub 文件）或 C 站指纹下载一个模型到指定类型的文件夹，不覆盖已有文件 | | download_status | 看下载进度和结果 | +| move_model | 把放错文件夹的模型挪到它该在的文件夹（封面、备注跟着走），和模型页“整理”里的“挪过去”一样；不会盖掉同名文件。默认先问你 | | list_voices / speak | 列出 Anomalous TTS 的角色；让某个角色读一段台词并保存（最多等两分钟） | | begin_task / report | 在 AI 工作台里起一个任务的标题；给你留言、交上它用别的方式做出来的图、结束任务（见第 5 节） | | ask_user | 在 AI 工作台里问你一个问题：几个选项（AI 推荐的那个标着“推荐”），加一行“都不是？写下你的想法”；AI 在同一步里等你回答 | @@ -151,7 +152,7 @@ claude mcp add --transport http anomalous http://127.0.0.1:8188/anomalous/mcp **运行进度**：AI 运行工作流后，时间线上那一步下面有进度条（排队时前面还有几个；运行时哪个节点、第几步、节点做完几个）。进度条右边是百分比，旁边是红色的 **■ 停止**：一点就停这一次运行（还在排队的移出队列，正在跑的立刻中断，只动这次运行），AI 也会收到“你停止了运行”，不会自己再跑一遍。任务工具栏的运行状态按钮旁边也有一个小 ■。进度条下面写着当前节点 **约剩多久**、每步几秒、**显存剩多少**，再下面一行是 ComfyUI **控制台**最新的那一行（比如 `Model Initializing ...`、`Requested to load …`），点开能看最近几行；这些 AI 也同样看得到。出图刚排上队，AMB 会读这次的工作流算出每个采样器要在多大的图上采样：如果是放大模型放大后又整张重新采样、尺寸超过这张显卡的舒适范围、或者要解码特别大的图，会出现黄色提醒，告诉你可能很慢或爆显存，以及怎么改（只想放大就用纯模型超分，或者先缩小、减步数、分块解码）；AI 收到同样的提醒，会被要求先问你要不要停（它也能用 `stop_run` 停自己的运行）。AI 用 `run_status` 问同样的进度，或者等到跑完再继续，不会凭时间猜；运行期间 AI 调用别的工具时，回复里的 `your_run` 也带着一句当前进度，跑完后告诉它一次结果，所以它不用自己去翻输出文件夹或者看屏幕。 -**流程图和关卡**：任务标题下面的一排流程按 AI 做事的先后列出它经过的阶段（模型、搭配、提示词、运行、挑图……），点一下跳到那一步。任务列表上方的 **关卡** 里，改画布、用显卡出图配音、下载模型、联网扫描四类事各自可以选 **直接做 / 先问我 / 不许做**（默认只有下载先问）。选了“先问我”，AI 的那一步会停在时间线上列出它要做什么，你点 **允许** 才做；点 **不要** 时可以写原因或想怎么改，AI 会收到。AI 最多等 50 秒，等不及的话会先去问你的回答（`get_answers`），你允许后它再做一次，不会问第二遍。 +**流程图和关卡**：任务标题下面的一排流程按 AI 做事的先后列出它经过的阶段（模型、搭配、提示词、运行、挑图……），点一下跳到那一步。任务列表上方的 **关卡** 里，改画布、用显卡出图配音、下载模型、联网扫描、挪动模型文件五类事各自可以选 **直接做 / 先问我 / 不许做**（默认下载和挪文件先问）。选了“先问我”，AI 的那一步会停在时间线上列出它要做什么，你点 **允许** 才做；点 **不要** 时可以写原因或想怎么改，AI 会收到。AI 最多等 50 秒，等不及的话会先去问你的回答（`get_answers`），你允许后它再做一次，不会问第二遍。 **AI 干活时你自己改的东西**：AI 在做一个任务时（10 分钟内调用过），你在 AMB 里做的改动，比如画布上的改动、改模型信息、存搭配或提示词，会以“你在 AMB 里改了”出现在这个任务的时间线上，画布改动带修改前后。AI 下一次调用任何工具时会收到它（并被告知保留你的改动，不要改回去），时间线上写着它是否已经知道。操作记录页里，AI 做的和 AI 干活时你做的记录旁边有 **AI 任务 →**。 @@ -175,7 +176,7 @@ claude mcp add --transport http anomalous http://127.0.0.1:8188/anomalous/mcp **决策记录**：讨论在哪里都行，在 AI 自己的对话框里聊也可以，但商量出的结论会留在 AMB。AI 用 `log_decision` 把“定了什么、为什么、没选哪些、谁定的、跟哪几步有关”记成时间线上一张带 📌 的卡片；你在 AMB 里自己做的选择（面板上的回答、允许或不允许、交出去的素材、AI 改画布的地方、停止运行）会自动算进去。窗口右上角的 **决策** 按时间列出这个任务的所有决策，每条都能跳回时间线上的那一步；AI 跑了图或改了画布却很久没记录决策时，这里会轻轻提一句。AI 下次接手这个任务时，也会先收到这些决策。 -**留言框里的 @ 和 /**：在任务下面的留言框里打 **@**，会弹出一张能搜索的列表：这个任务出过的图、搭配、模型、工作流、提示词、节点包、节点。选中的会写成“@名字”，留言框下面显示成小标签；发出去时 AI 收到的是你指的那个东西本身（模型的准确 ID、搭配的参数、图片和它的生成参数），不用它去猜。在开头打 **/** 是快捷指令：/停止、/撤销、/展台、/结构、/素材、/专用标签页、/当前画布、/新任务 马上生效，不用等 AI；/重跑、/解释、/继续 帮你写好一句话，改一改再发。↑↓ 选，回车或 Tab 确认，Esc 关掉。 +**留言框里的 @ 和 /**：在任务下面的留言框里打 **@**，会弹出一张能搜索的列表：这个任务出过的图、图库里最新的图（打字时也按提示词、模型、seed 在图库里搜）、搭配、模型、工作流、提示词、节点包、节点。留言框左边的 **图片按钮** 打开图库：点选几张（一次最多 8 张），点 **加进消息**，它们就像 @ 一样写进留言。选中的会写成“@名字”，留言框下面显示成小标签；发出去时 AI 收到的是你指的那个东西本身（模型的准确 ID、搭配的参数、图片和它的生成参数），不用它去猜。在开头打 **/** 是快捷指令：/停止、/撤销、/展台、/结构、/素材、/专用标签页、/当前画布、/新任务 马上生效，不用等 AI；/重跑、/解释、/继续 帮你写好一句话，改一改再发。↑↓ 选，回车或 Tab 确认，Esc 关掉。 **AI 画的面板**（`show_panel`）显示在工作台右边的**展台**上，后面再多步骤也不会被顶走。右边这一栏一次只显示一样东西，用窗口右上角的三个按钮切换：**展台**（有问题等你回答时带个小黄点）、**节点结构**（AI 跑过的节点结构图，大图；上面可以选看哪一次运行，时间线里运行那一步的“在右侧看大图”也会打开它）、**素材**。再点一下正亮着的那个就收起这一栏；切走再切回来，展台的位置和挑了一半的素材都还在。任务标题和那一排按钮在时间线往下翻时一直停在顶上。时间线上只留一行“在展台打开 ▸”。展台默认跟着最新的面板，你翻回旧面板就停在你看的那张；窗口窄的时候，时间线和展台变成两个标签页切换。右边这一栏是一张单独的卡片，它左边的整条空隙都能拖着调宽窄（鼠标靠近时整条边会亮起；点一下后也能按 ← → 调；AMB 记得你调的宽度，双击恢复默认），时间线总会留够地方。窗口左上角的按钮收起 / 展开左边的任务列表；收起以后，鼠标移到这个按钮上，任务列表会临时浮出来，点一个任务或者鼠标移开就自动收回。素材在右边这一栏里时，鼠标移过卡片不会再跳进详细说明，点右边的 ⓘ 才看。小窗里素材是一行一个的列表（图标、名字、来源、勾选），搜索框旁边三个小图标是“全部 / 已选 / 常用”；展开成大窗口就是带预览的卡片。这些地方的滚动条平时是隐藏的，鼠标移到能滚动的那一块上才显出来。AI 可以用插件提供的“积木”拼面板：标题和文字（也可以是提示、警告）、图片（一排、网格，或者两张并排对比）、你的模型（带封面的卡片，点了打开）、表格、列表、参数、进度条。它还能**问你**：几个按钮（单选或多选）、一行输入框。你的回答会交出去，等 AI 用 `get_answers` 来取（它可以等你最多一分钟）；每个问题下面有一个状态标签：琥珀色的 **已发送 · 等 AI 来取**，AI 拿到后变成绿色的 **AI 已收到 · 时间**。你的留言、批注、发给它的图片也是一样。不想答的题点 **这题跳过**，AI 会知道你跳过了，自己拿主意。AMB 没法主动叫醒 AI，所以回答之后要等它下一次来问。 @@ -212,4 +213,4 @@ While ComfyUI runs, Anomalous serves an MCP endpoint at `http://127.0.0.1:8188/a Since 1.62 the **AI workbench** page keeps every step an AI app takes through this endpoint, grouped into tasks (named with `begin_task`, else one app's calls without a 20-minute pause): what it is currently running (`running` in real time) and what it looked up, changed, ran, downloaded, scanned or spoke, its notes (`report`, which also hands in pictures made another way), and what each step made, with links to the picture, the model, the combo, the canvas change (find or undo) and the audio. It updates live; the rail gets a dot while you are elsewhere. With `show_panel` the AI draws a panel there from fixed blocks (text, pictures to compare, your models as cards, tables, settings, progress, buttons and a text line); the user's answers and task messages are returned on `get_answers` (which can wait up to 55 s) and are also automatically attached once to the AI's next successful tool call. Blocks are checked on the server and drawn as text; nothing the AI sends runs. -Reading tools: library overview, model search and details (optionally with the cover), scan status, output-image search and generation settings (optionally with a thumbnail), combos, workflow recipes, saved prompts, generated audio and the activity log. Acting tools: read the open workflow, write prompts, swap a node's model, add a LoRA, place a combo, check and fix missing models, run the workflow, open a model in Anomalous, start scans, list Anomalous TTS voices and have one speak. Canvas actions need ComfyUI open in a browser on this computer; each is one Ctrl+Z step and is logged as the AI's. Nothing deletes or renames files. Only this computer may connect; requests through a proxy, from other hosts or from web pages are refused. When your AI app uses a cloud model, what the tools return (and any attached picture) is sent to that provider. +Reading tools: library overview, model search and details (optionally with the cover; each model says which node types load it, and a model in the wrong folder for what it is says where it belongs), scan status, output-image search and generation settings (optionally with a thumbnail), combos, workflow recipes, saved prompts, generated audio and the activity log. Acting tools: read the open workflow, write prompts, swap a node's model, add a LoRA, place a combo, check and fix missing models, run the workflow, open a model in Anomalous, start scans, list Anomalous TTS voices and have one speak. Canvas actions need ComfyUI open in a browser on this computer; each is one Ctrl+Z step and is logged as the AI's. Nothing deletes or renames files; `move_model` moves a misfiled model (with its covers and notes) to its folder, after the user says yes unless the "files" gate lets it go ahead. Only this computer may connect; requests through a proxy, from other hosts or from web pages are refused. When your AI app uses a cloud model, what the tools return (and any attached picture) is sent to that provider. diff --git a/model_kind.py b/model_kind.py index 18c4cae..cec4a33 100644 --- a/model_kind.py +++ b/model_kind.py @@ -30,6 +30,11 @@ LORA_MARKS = ("lora_up.", "lora_down.", "lora_a.", "lora_b.", "lora.up.", "lora. "lokr_w1", ".lora_linear_layer.", "dora_scale") CONTROL_MARKS = ("control_model_", "input_hint_block", "controlnet_cond_embedding", "controlnet_blocks", "controlnet_x_embedder", "zero_convs", "controlnet_mid_block") +# Tensors only a model patch has (ComfyUI's "Load Model Patch" node, models/model_patches): Anima +# LLLite, the Qwen-Image block-wise and Z-Image Fun ControlNets, SigLIP features, Wan's Uni3C. Some +# are called ControlNets, but the ControlNet loader cannot read them. +PATCH_KEYS = {"lllite_conditioning1.conv1.weight", "controlnet_blocks.0.y_rms.weight", "feature_embedder.mid_layer_norm.bias", + "control_all_x_embedder.2-1.weight", "controlnet_patch_embedding.weight"} DIFFUSION_STARTS = ("double_blocks.", "single_blocks.", "joint_blocks.", "transformer_blocks.", "single_transformer_blocks.", "input_blocks.", "blocks.0.self_attn", "net.blocks.", "net.x_embedder.") @@ -210,6 +215,8 @@ def _kind(keys, raw, text): names = set(keys) if names and names <= EMBEDDING_KEYS and names & {"emb_params", "clip_l", "clip_g", "string_to_param"}: return "embeddings", True + if names & PATCH_KEYS: + return "model_patches", True if any(mark in text for mark in CONTROL_MARKS): return "controlnet", True diffusion = any(key.startswith(("model.diffusion_model.", "diffusion_model.") + DIFFUSION_STARTS) for key in keys) @@ -252,7 +259,7 @@ def classify(fmt, header): raw = "\n".join(keys).lower() text = raw.replace(".", "_") kind, sure = _kind(keys, raw, text) - if kind in ("vae", "text_encoders", "clip_vision", "upscale_models", "ipadapter"): + if kind in ("vae", "text_encoders", "clip_vision", "upscale_models", "ipadapter", "model_patches"): base, base_sure = "", True # shared by several base models: the type's own folder elif kind == "embeddings": base, base_sure = _embedding_base(header), True diff --git a/tests/test_ai_toolbox.py b/tests/test_ai_toolbox.py index 1a097cc..c056d6a 100644 --- a/tests/test_ai_toolbox.py +++ b/tests/test_ai_toolbox.py @@ -31,7 +31,7 @@ class ToolboxTests(unittest.TestCase): isolate() if os.path.exists(FAVOURITES): os.remove(FAVOURITES) - for target, value in ((ai_toolbox, "_path"), (mcp_canvas_edit, "_object_info")): + for target, value in ((ai_toolbox, "_path"), (mcp_canvas_edit, "object_info")): patcher = mock.patch.object(target, value, (lambda: FAVOURITES) if value == "_path" else (lambda ctx: INFO)) patcher.start() self.addCleanup(patcher.stop) diff --git a/tests/test_mcp.py b/tests/test_mcp.py index 37bd441..5029ed7 100644 --- a/tests/test_mcp.py +++ b/tests/test_mcp.py @@ -13,9 +13,12 @@ from aiohttp.test_utils import make_mocked_request PLUGIN = Path(__file__).resolve().parents[1] sys.path[:0] = [str(PLUGIN), str(PLUGIN.parents[1])] -from api import ai_gates, ai_panels, ai_tasks, lora_info, mcp_actions, mcp_apps, mcp_bridge, mcp_server, mcp_tools # noqa: E402 +from api import ( # noqa: E402 + ai_gates, ai_panels, ai_tasks, lora_info, mcp_actions, mcp_apps, mcp_bridge, mcp_canvas_edit, mcp_model_use, mcp_server, mcp_tools, +) MODERN = "2026-07-28" +CTX = {"base_url": "http://127.0.0.1:1"} # Every call is kept as an AI workbench step: keep them in a temporary folder, never the user's. _AI_TASKS = tempfile.mkdtemp(prefix="amb_mcp_ai_tasks_") ai_tasks.tasks_dir = lambda: _AI_TASKS @@ -135,18 +138,24 @@ class ToolTests(unittest.TestCase): metas = {str(lora): {"name": "Neon City", "baseModel": "SDXL 1.0", "trainedWords": ["neon"], "info_source": "civitai", "description": "

Bright lights

", "custom_notes": "mine"}, str(self.root / "plain.ckpt"): {"info_source": ""}} + # ComfyUI's LoRA loader lists the whole loras folder (with its own separators). + listed = ["plain.ckpt", "sub\\neon_city.safetensors"] + info = {"LoraLoader": {"input": {"required": {"lora_name": [listed]}}}} + tools = mcp_model_use.MODEL_TOOLS with mock.patch.object(mcp_tools, "_iter_search_models", return_value=files), \ mock.patch.object(mcp_tools, "get_metadata", side_effect=lambda path: metas[path]), \ - mock.patch.object(mcp_tools, "resolve_folder_subdir", side_effect=lambda t, i, s: (str(self.root), str(self.root / s.strip("/")))): - found = mcp_tools.call_tool(mcp_tools.TOOLS, "search_models", {"query": "neon sdxl"})["structuredContent"] + mock.patch.object(mcp_tools, "resolve_folder_subdir", side_effect=lambda t, i, s: (str(self.root), str(self.root / s.strip("/")))), \ + mock.patch.object(mcp_canvas_edit, "object_info", return_value=info), \ + mock.patch.object(mcp_model_use.folder_paths, "get_filename_list", return_value=listed): + found = mcp_tools.call_tool(tools, "search_models", {"query": "neon sdxl"}, CTX)["structuredContent"] self.assertEqual(found["total_matches"], 1) model = found["models"][0] - self.assertEqual((model["model_id"], model["source"]), ("loras:0:sub/neon_city.safetensors", "civitai")) - self.assertEqual(mcp_tools.call_tool(mcp_tools.TOOLS, "search_models", {"source": "not_scanned"})["structuredContent"]["total_matches"], 1) - detail = mcp_tools.call_tool(mcp_tools.TOOLS, "get_model", {"model_id": model["model_id"]})["structuredContent"] - self.assertEqual((detail["description"], detail["user_notes"]), ("Bright lights", "mine")) + self.assertEqual((model["model_id"], model["source"], model["loaded_by"]), ("loras:0:sub/neon_city.safetensors", "civitai", ["LoraLoader"])) + self.assertEqual(mcp_tools.call_tool(tools, "search_models", {"source": "not_scanned"}, CTX)["structuredContent"]["total_matches"], 1) + detail = mcp_tools.call_tool(tools, "get_model", {"model_id": model["model_id"]}, CTX)["structuredContent"] + self.assertEqual((detail["description"], detail["user_notes"], detail["loaded_by"]), ("Bright lights", "mine", ["LoraLoader"])) for bad in ("loras:0:../../secret.safetensors", "loras:0:sub/missing.safetensors", "nonsense"): - result = mcp_tools.call_tool(mcp_tools.TOOLS, "get_model", {"model_id": bad}) + result = mcp_tools.call_tool(tools, "get_model", {"model_id": bad}, CTX) self.assertTrue(result["isError"], bad) def test_image_settings_come_from_the_png_and_stay_inside_output(self): @@ -167,8 +176,8 @@ class ToolTests(unittest.TestCase): self.assertTrue(mcp_tools.call_tool(mcp_tools.TOOLS, "get_image_info", {"image": "../out.png"})["isError"]) def test_bad_arguments_are_tool_errors(self): - self.assertTrue(mcp_tools.call_tool(mcp_tools.TOOLS, "get_model", {})["isError"]) - self.assertTrue(mcp_tools.call_tool(mcp_tools.TOOLS, "search_models", {"delete": True})["isError"]) + self.assertTrue(mcp_tools.call_tool(mcp_model_use.MODEL_TOOLS, "get_model", {}, CTX)["isError"]) + self.assertTrue(mcp_tools.call_tool(mcp_model_use.MODEL_TOOLS, "search_models", {"delete": True}, CTX)["isError"]) class BridgeTests(unittest.TestCase): diff --git a/tests/test_mcp_canvas_edit.py b/tests/test_mcp_canvas_edit.py index b18f01d..58fa456 100644 --- a/tests/test_mcp_canvas_edit.py +++ b/tests/test_mcp_canvas_edit.py @@ -27,7 +27,7 @@ INFO = { class FindNodesTests(unittest.TestCase): def setUp(self): - patcher = mock.patch.object(mcp_canvas_edit, "_object_info", return_value=INFO) + patcher = mock.patch.object(mcp_canvas_edit, "object_info", return_value=INFO) patcher.start() self.addCleanup(patcher.stop) diff --git a/tests/test_mcp_model_use.py b/tests/test_mcp_model_use.py new file mode 100644 index 0000000..2f2391a --- /dev/null +++ b/tests/test_mcp_model_use.py @@ -0,0 +1,90 @@ +"""What loads a model, for the AI (api/mcp_model_use.py): a model patch filed under ControlNet is +said to belong in model_patches, move_model puts it there, and a copy already there is never +overwritten. Files go to temporary folders.""" +import json +import os +import struct +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +PLUGIN = Path(__file__).resolve().parents[1] +sys.path[:0] = [str(PLUGIN), str(PLUGIN.parents[1])] + +from api import mcp_canvas_edit, mcp_model_use, mcp_tools, model_placement # noqa: E402 + +CTX = {"base_url": "http://127.0.0.1:1"} + + +def safetensors(*names): + raw = json.dumps({name: {"dtype": "F16", "shape": [4, 4], "data_offsets": [0, 0]} for name in names}).encode() + return struct.pack(" plain(text).split('/').pop().replace(/\.[^.]+$/, ''); const stem = name(value); const close = stem ? choices.filter(item => name(item).includes(stem) || (name(item).length > 2 && stem.includes(name(item)))) : []; + if (!choices.length) throw new Error(`${where} has no choices at all: ComfyUI found no file for it. If the file is in another models folder, search_models says where it belongs (move_model moves it).`); throw new Error(`${where} has no choice "${value}"; ${close.length ? `close: ${shown(close)}` : `choices: ${shown(choices)}`}`); } if (typeof widget.value === 'number') { diff --git a/web/modules/locales.js b/web/modules/locales.js index 7f60c5c..a91bf56 100644 --- a/web/modules/locales.js +++ b/web/modules/locales.js @@ -1475,14 +1475,21 @@ export const i18n = { aiGate_gpu: "用显卡出图、配音", aiGate_download: "下载模型", aiGate_network: "联网扫描模型", + aiGate_files: "挪动模型文件", aiGateWhy_canvas: "写提示词、换模型、加 LoRA、放搭配、补模型（都能撤销）", aiGateWhy_gpu: "运行工作流、配音：会占用显卡", aiGateWhy_download: "下载模型文件：占硬盘，可能很大", aiGateWhy_network: "到 Civitai 查模型信息", + aiGateWhy_files: "把放错文件夹的模型挪到对的地方（封面和备注跟着走）", aiGateQ_canvas: "能不能改画布？", aiGateQ_gpu: "能不能用显卡跑？", aiGateQ_download: "能不能下载？", aiGateQ_network: "能不能联网查？", + aiGateQ_files: "能不能挪这个模型？", + aiGateArg_model_id: "模型", + aiGateArg_to_type: "挪到", + aiGateArg_url: "链接", + aiGateArg_type: "类型", aiPolicy_auto: "直接做", aiPolicy_ask: "先问我", aiPolicy_never: "不许做", @@ -1587,6 +1594,15 @@ export const i18n = { aiMentionHead: "提到：这个任务的图、搭配、模型、工作流、提示词、节点包、节点", aiMentionNone: "没找到，换个词试试", aiMentionPicture: "这个任务的图", + aiMentionGallery: "图库里的图", + aiPicsButton: "从图库选图给 AI", + aiPicsSearch: "搜图库：提示词、模型、seed:123…", + aiPicsAdd: "加进消息", + aiPicsPicked: "选了 {count} 张（最多 {max} 张）", + aiPicsHint: "点图片来选，最多 {max} 张。AI 会收到图片本身，和它是用什么做出来的。", + aiPicsFull: "这条消息放不下更多图了，先发出去吧。", + aiPicsNone: "没找到这样的图", + aiPicsEmpty: "图库里还没有图片", aiMentionRemove: "不提这个了", aiCommandHead: "快捷指令：马上生效，或者帮你写好一句话", aiCommandNone: "没有这个指令", @@ -1759,6 +1775,7 @@ export const i18n = { aiTool_run_workflow: "运行工作流", aiTool_download_missing_models: "下载缺的模型", aiTool_download_model: "下载模型", + aiTool_move_model: "挪动模型", aiTool_scan_models: "扫描模型", aiTool_speak: "配音", aiTool_open_in_anomalous: "打开模型页给你看", @@ -2032,6 +2049,7 @@ export const i18n = { importKind_clip_vision: 'CLIP Vision', importKind_upscale_models: '放大模型', importKind_ipadapter: 'IPAdapter', + importKind_model_patches: '模型补丁', importKindHelp_checkpoints: '大模型：出图的主体，文件里自带 VAE 和文本编码器，用「Checkpoint 加载器」加载。', importKindHelp_loras: 'LoRA：小模型，给大模型加角色、画风或概念，接在大模型后面用；要和大模型的底模对得上。', importKindHelp_vae: 'VAE：负责图片和潜空间互相转换，影响颜色和细节；大模型多半自带，单独下载的放这里。', @@ -2042,6 +2060,7 @@ export const i18n = { importKindHelp_clip_vision: 'CLIP Vision：让模型看懂参考图，IPAdapter、图生视频等会用到。', importKindHelp_upscale_models: '放大模型：把图片放大并补上细节。', importKindHelp_ipadapter: 'IPAdapter：用参考图控制画风或人物。', + importKindHelp_model_patches: '模型补丁：给某个底模加控制或功能的小文件，比如 Anima LLLite、Qwen-Image 和 Z-Image 的 ControlNet。名字里常带 ControlNet，但要用「加载模型补丁」节点加载。', importKindUnknown: '认不出这是哪类模型，请在上面选一下。', importKindUnreadable: '这种格式不能安全地打开来看，请在上面选它的类型。', importBase: '底模：{base}', @@ -2095,6 +2114,7 @@ export const i18n = { tidyLoader_clip_vision: 'CLIP Vision 加载器', tidyLoader_upscale_models: '放大模型加载器', tidyLoader_ipadapter: 'IPAdapter 加载器', + tidyLoader_model_patches: '模型补丁加载器', modelTabAll: '全部模型', modelBasesLabel: '按底模筛选', modelBasesAll: '全部', @@ -2109,6 +2129,7 @@ export const i18n = { tidyWhy_loras: '这是 LoRA，要接在大模型后面、用「{right}」加载，放在这里找不到它。', tidyWhy_text_encoders: '这是文本编码器（把提示词变成模型看得懂的东西），要用「{right}」加载。', tidyWhy_vae: '这是 VAE，要用「{right}」加载。', + tidyWhy_model_patches: '这是模型补丁（比如 Anima LLLite），名字里常带 ControlNet，但要用「{right}」加载，它只看 model_patches 文件夹。', tidyChangePlace: '换个位置', tidyDupWhy: '这几个文件内容一模一样，留一份就够了。点封面选要留的那份。', tidyKeep: '留这份', @@ -4714,14 +4735,21 @@ export const i18n = { aiGate_gpu: "Use the GPU (runs, speech)", aiGate_download: "Download models", aiGate_network: "Scan models online", + aiGate_files: "Move model files", aiGateWhy_canvas: "Write prompts, swap models, add LoRAs, place combos, put back models (all undoable)", aiGateWhy_gpu: "Run the workflow or speak a line: takes the GPU", aiGateWhy_download: "Download model files: disk space, possibly large", aiGateWhy_network: "Look models up on Civitai", + aiGateWhy_files: "Move a model in the wrong folder to the right one (covers and notes go along)", aiGateQ_canvas: "May it change the canvas?", aiGateQ_gpu: "May it use the GPU?", aiGateQ_download: "May it download?", aiGateQ_network: "May it go online?", + aiGateQ_files: "May it move this model?", + aiGateArg_model_id: "Model", + aiGateArg_to_type: "Move to", + aiGateArg_url: "Link", + aiGateArg_type: "Type", aiPolicy_auto: "Go ahead", aiPolicy_ask: "Ask me", aiPolicy_never: "Never", @@ -4826,6 +4854,15 @@ export const i18n = { aiMentionHead: "Name: this task's pictures, combos, models, workflows, prompts, node packs, nodes", aiMentionNone: "Nothing found; try other words", aiMentionPicture: "This task's picture", + aiMentionGallery: "Gallery picture", + aiPicsButton: "Pick pictures from the gallery for the AI", + aiPicsSearch: "Search the gallery: prompt words, a model, seed:123…", + aiPicsAdd: "Add to the message", + aiPicsPicked: "{count} picked (up to {max})", + aiPicsHint: "Click pictures to pick them, up to {max}. The AI gets the pictures and what made them.", + aiPicsFull: "This message takes no more pictures; send it first.", + aiPicsNone: "No picture like that", + aiPicsEmpty: "No pictures in the gallery yet", aiMentionRemove: "Do not name this", aiCommandHead: "Commands: act at once, or write a sentence for you", aiCommandNone: "No such command", @@ -4998,6 +5035,7 @@ export const i18n = { aiTool_run_workflow: "Ran the workflow", aiTool_download_missing_models: "Downloaded missing models", aiTool_download_model: "Downloaded a model", + aiTool_move_model: "Moved a model", aiTool_scan_models: "Scanned models", aiTool_speak: "Spoke a line", aiTool_open_in_anomalous: "Showed you a model", @@ -5271,6 +5309,7 @@ export const i18n = { importKind_clip_vision: 'CLIP Vision', importKind_upscale_models: 'Upscale model', importKind_ipadapter: 'IPAdapter', + importKind_model_patches: 'Model patch', importKindHelp_checkpoints: 'Checkpoint: the main image model, with its VAE and text encoder inside; loaded with "Load Checkpoint".', importKindHelp_loras: 'LoRA: a small add-on for a character, style or idea, chained after the main model; its base model must match.', importKindHelp_vae: 'VAE: turns images into latents and back, affecting colour and detail; most checkpoints carry one, separate ones go here.', @@ -5281,6 +5320,7 @@ export const i18n = { importKindHelp_clip_vision: 'CLIP Vision: lets a model see a reference image; IPAdapter and image-to-video use it.', importKindHelp_upscale_models: 'Upscale model: enlarges an image and adds detail.', importKindHelp_ipadapter: 'IPAdapter: steers the style or a person from a reference image.', + importKindHelp_model_patches: 'Model patch: a small file that adds control or a feature to one base model, such as Anima LLLite or the Qwen-Image and Z-Image ControlNets. Often called a ControlNet, it loads with the Load Model Patch node.', importKindUnknown: 'Cannot tell what kind of model this is; choose its type above.', importKindUnreadable: 'This format cannot be looked into safely; choose its type above.', importBase: 'Base model: {base}', @@ -5334,6 +5374,7 @@ export const i18n = { tidyLoader_clip_vision: 'Load CLIP Vision', tidyLoader_upscale_models: 'Load Upscale Model', tidyLoader_ipadapter: 'IPAdapter loader', + tidyLoader_model_patches: 'Model patch loader', modelTabAll: 'All models', modelBasesLabel: 'Filter by base model', modelBasesAll: 'All', @@ -5348,6 +5389,7 @@ export const i18n = { tidyWhy_loras: 'This is a LoRA: it goes after a main model with "{right}" and cannot be found here.', tidyWhy_text_encoders: 'This is a text encoder (it turns the prompt into what the model understands); it loads with "{right}".', tidyWhy_vae: 'This is a VAE; it loads with "{right}".', + tidyWhy_model_patches: 'This is a model patch (Anima LLLite, for one): often called a ControlNet, it loads with "{right}", which only looks in model_patches.', tidyChangePlace: 'Change place', tidyDupWhy: 'These files are exactly the same; one is enough. Click a cover to choose the one to keep.', tidyKeep: 'Keep', diff --git a/web/modules/mcp_bridge.js b/web/modules/mcp_bridge.js index 3370814..43de42d 100644 --- a/web/modules/mcp_bridge.js +++ b/web/modules/mcp_bridge.js @@ -309,6 +309,7 @@ async function handle(detail, getOwner) { const { _task_canvas: taskCanvas, ...given } = args || {}; let outcome; let tab = null; // a tab made or opened for the task is kept even when the action then fails + getOwner()?.flushCanvasActivity?.(); // the user's own changes so far, before the task's tab may come to the front try { outcome = { ok: true, value: (await onTaskTab(action, taskCanvas, () => run(given, getOwner), (made) => { tab = made; })) ?? null }; } catch (error) { diff --git a/web/modules/model_policies.js b/web/modules/model_policies.js index 84450d5..c315555 100644 --- a/web/modules/model_policies.js +++ b/web/modules/model_policies.js @@ -25,6 +25,7 @@ export function inferModelFolderTypes(node, widget) { const nodeType = String(node?.type || '').toLowerCase(); const key = `${widgetName} ${nodeType}`; + if (nodeType === 'modelpatchloader') return ['model_patches']; if (key.includes('clip_vision') || key.includes('clip vision') || nodeType.includes('clipvision')) return ['clip_vision']; if (key.includes('vae_approx') || key.includes('vae approx')) return ['vae_approx']; if (widgetName.includes('vae') || nodeType === 'vaeloader' || key.includes('vae loader')) return ['vae']; diff --git a/web/modules/ui_ai_gallery_pick.js b/web/modules/ui_ai_gallery_pick.js new file mode 100644 index 0000000..6ea716b --- /dev/null +++ b/web/modules/ui_ai_gallery_pick.js @@ -0,0 +1,164 @@ +/** + * Pictures from the gallery for the AI workbench's message box (ui_ai_message.js): a panel over + * the box with the newest output pictures (GET /anomalous/gallery_images, as 512 px thumbnails), + * searched the way the gallery searches (words of the prompt, a model, "seed:123"…), more as it + * scrolls. Clicking pictures picks them, up to the number the message still takes; "Add" gives + * them to `onPick([image])`, and the box names them as "@" does, so the AI gets the pictures + * themselves with what made them. Esc, a click outside, or the button again closes it. + */ + +import { translate as t } from './locales.js'; +import { icon } from './ui_ai_icons.js'; + +const PAGE = 40; +const SEARCH_MS = 250; +const MORE_PX = 120; + +function el(tag, className, text) { + const node = document.createElement(tag); + if (className) node.className = className; + if (text !== undefined) node.textContent = String(text); + return node; +} + +/** An output picture's 512 px thumbnail ({filename, subfolder}). */ +export const thumbUrl = image => `/anomalous/output_thumbnail?${new URLSearchParams({ filename: image.filename, subfolder: image.subfolder || '' })}`; + +/** An output picture as "@" names it. */ +export function pictureItem(image, gallery = false) { + const id = [image.subfolder, image.filename].filter(Boolean).join('/'); + return { kind: 'pictures', id, title: image.filename, image, gallery, haystack: id.toLowerCase() }; +} + +/** Newest output pictures (matching `query`, when given): {images, pages}. */ +export async function galleryPictures(page, limit, query = '', signal = undefined) { + const params = new URLSearchParams({ page: String(page), limit: String(limit) }); + if (query) params.set('q', query); + const response = await fetch(`/anomalous/gallery_images?${params}`, { signal }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + const data = await response.json(); + return { images: data.images || [], pages: data.pages || 0 }; +} + +/** + * The panel for message box `box`; `room()` says how many more pictures the message takes. + * Returns {toggle(), close()}. + */ +export function galleryPicker(box, room, onPick) { + let panel = null; + let stop = null; // ends the open panel's listeners and requests + + function close() { + stop?.(); + stop = null; + panel?.remove(); + panel = null; + } + + function open() { + const picked = new Map(); + const abort = new AbortController(); + let page = 0; + let pages = 1; + let query = ''; + let loading = false; + let timer = null; + panel = el('div', 'anomalous-ai-pics'); + panel.dataset.anomalousEscape = '1'; + const search = el('input', 'anomalous-ai-pics-search'); + search.type = 'search'; + search.placeholder = t('aiPicsSearch'); + const grid = el('div', 'anomalous-ai-pics-grid'); + const count = el('span', 'anomalous-ai-pics-count'); + const add = el('button', 'anomalous-ai-panel-send', t('aiPicsAdd')); + add.type = 'button'; + const foot = el('div', 'anomalous-ai-pics-foot'); + foot.append(count, add); + panel.append(search, grid, foot); + box.append(panel); + + const max = () => Math.max(0, room()); + const drawCount = () => { + count.textContent = picked.size ? t('aiPicsPicked', { count: picked.size, max: max() }) : t(max() ? 'aiPicsHint' : 'aiPicsFull', { max: max() }); + add.disabled = !picked.size; + }; + function tile(image) { + const item = pictureItem(image, true); + const node = el('button', 'anomalous-ai-pics-tile'); + node.type = 'button'; + node.title = item.id; + const img = el('img'); + img.src = thumbUrl(image); + img.alt = ''; + img.loading = 'lazy'; + const mark = el('span', 'anomalous-ai-pics-mark'); + mark.append(icon('check', 12)); + node.append(img, mark); + node.onclick = () => { + if (picked.has(item.id)) picked.delete(item.id); + else if (picked.size < max()) picked.set(item.id, item); + node.classList.toggle('is-picked', picked.has(item.id)); + drawCount(); + }; + node.classList.toggle('is-picked', picked.has(item.id)); + return node; + } + async function more() { + if (loading || page >= pages) return; + loading = true; + const asked = query; + try { + const found = await galleryPictures(page + 1, PAGE, asked, abort.signal); + if (asked !== query || !panel) return; + page += 1; + pages = found.pages; + grid.querySelector('.anomalous-ai-pics-none')?.remove(); + grid.append(...found.images.filter(image => image.media_type !== 'video').map(tile)); + if (!grid.children.length) grid.append(el('div', 'anomalous-ai-pics-none', t(query ? 'aiPicsNone' : 'aiPicsEmpty'))); + } catch (error) { + if (error.name !== 'AbortError') grid.replaceChildren(el('div', 'anomalous-ai-pics-none', error.message)); + } finally { + loading = false; + } + } + search.addEventListener('keydown', (event) => { if (event.key !== 'Escape') event.stopPropagation(); }); + search.addEventListener('input', () => { + clearTimeout(timer); + timer = setTimeout(() => { + query = search.value.trim(); + page = 0; + pages = 1; + grid.replaceChildren(); + void more(); + }, SEARCH_MS); + }); + grid.addEventListener('scroll', () => { + if (grid.scrollTop + grid.clientHeight > grid.scrollHeight - MORE_PX) void more(); + }); + add.onclick = () => { + const items = [...picked.values()]; + close(); + onPick(items); + }; + const outside = (event) => { + if (!panel?.contains(event.target) && !event.target.closest?.('.anomalous-ai-message-pics')) close(); + }; + const escape = (event) => { if (event.key === 'Escape') setTimeout(close); }; // after the window's own Esc saw the panel + document.addEventListener('pointerdown', outside, true); + document.addEventListener('keydown', escape); + stop = () => { + clearTimeout(timer); + abort.abort(); + document.removeEventListener('pointerdown', outside, true); + document.removeEventListener('keydown', escape); + }; + drawCount(); + void more(); + search.focus(); + } + + return { + toggle: () => (panel ? close() : open()), + close, + }; +} diff --git a/web/modules/ui_ai_gate.js b/web/modules/ui_ai_gate.js index 687f8e1..1bf4d1c 100644 --- a/web/modules/ui_ai_gate.js +++ b/web/modules/ui_ai_gate.js @@ -2,7 +2,7 @@ * Gates on the AI workbench (api/ai_gates.py): the card under a step that waits for the user's OK * (what the AI wants to do; Allow, No with a reason for the AI, and "from now on, go ahead"), what * the user answered, and the Gates window where each kind of action is set to go ahead, ask first - * or never (changing the canvas, using the GPU, downloading, going online), and whether a new + * or never (changing the canvas, using the GPU, downloading, going online, moving model files), and whether a new * task works in a canvas tab of its own (api/ai_task_canvas.py). */ @@ -10,7 +10,7 @@ import { translate as t } from './locales.js'; import { anomalousAlert } from './ui_dialog.js'; import { aiWindow } from './ui_ai_connect.js'; -export const GATES = ['canvas', 'gpu', 'download', 'network']; +export const GATES = ['canvas', 'gpu', 'download', 'network', 'files']; const POLICIES = ['auto', 'ask', 'never']; function el(tag, className, text) { @@ -51,7 +51,10 @@ export function gateCard(step, taskId, changed) { el('div', 'anomalous-ai-gate-why', t(`aiGateWhy_${asked.gate}`))); if (asked.lines?.length) { const lines = el('dl', 'anomalous-ai-gate-lines'); - for (const line of asked.lines) lines.append(el('dt', '', line.label), el('dd', '', line.value)); + for (const line of asked.lines) { + const label = t(`aiGateArg_${line.label}`); + lines.append(el('dt', '', label === `aiGateArg_${line.label}` ? line.label : label), el('dd', '', line.value)); + } card.append(lines); } const reason = el('input', 'anomalous-ai-gate-reason'); diff --git a/web/modules/ui_ai_mention.js b/web/modules/ui_ai_mention.js index 5833628..cf73c37 100644 --- a/web/modules/ui_ai_mention.js +++ b/web/modules/ui_ai_mention.js @@ -1,7 +1,9 @@ /** * "@" and "/" in the AI workbench's message box (ui_ai_message.js), as chat apps have them. - * "@" lists what the user can point the AI at: this task's pictures, combos, models, workflows, - * prompts, custom node packs and node types (the materials picker's data, ai_toolbox_data.js). + * "@" lists what the user can point the AI at: this task's pictures, the gallery's newest ones + * (and, for what is typed, those the gallery's search finds: prompt words, models, seed), combos, + * models, workflows, prompts, custom node packs and node types (the materials picker's data, + * ai_toolbox_data.js). The picture button's panel (ui_ai_gallery_pick.js) adds pictures as well. * Taking one writes "@name" into the text and keeps what it is; the message then carries exactly * what each "@" points at (api/ai_message_refs.py), shown as chips under the box until it is * sent. "/" at the start lists the commands (ui_ai_commands.js): some act at once, some fill in @@ -13,10 +15,12 @@ import { keptCatalog, prefetchToolbox, searchItems } from './ai_toolbox_data.js' import { coverUrl } from './ui_ai_toolbox_cards.js'; import { sourceBadge } from './ui_ai_toolbox_icons.js'; import { packItems, pageNodes } from './ui_ai_toolbox.js'; -import { outputUrl } from './ai_task_view.js'; import { icon } from './ui_ai_icons.js'; +import { galleryPictures, pictureItem, thumbUrl } from './ui_ai_gallery_pick.js'; const SHOWN = 8; +const RECENT = 30; // the gallery's newest pictures "@" offers +const SEARCH_MS = 250; const ORDER = ['pictures', 'combos', 'models', 'workflows', 'prompts', 'packs', 'nodes']; function el(tag, className, text) { @@ -36,7 +40,7 @@ function pictures(task) { const id = [item.subfolder, item.filename].filter(Boolean).join('/'); if (seen.has(id)) continue; seen.add(id); - out.push({ kind: 'pictures', id, title: item.filename, image: item, haystack: id.toLowerCase() }); + out.push(pictureItem(item)); } } return out; @@ -55,7 +59,7 @@ function everything(task) { function lead(item) { const box = el('span', 'anomalous-ai-mention-lead'); - const url = item.kind === 'pictures' ? outputUrl(item.image) : ['combos', 'workflows', 'models'].includes(item.kind) ? coverUrl(item) : ''; + const url = item.kind === 'pictures' ? thumbUrl(item.image) : ['combos', 'workflows', 'models'].includes(item.kind) ? coverUrl(item) : ''; if (url) { const img = el('img'); img.src = url; @@ -71,13 +75,14 @@ function lead(item) { return box; } -const kindWords = item => [item.kind === 'pictures' ? t('aiMentionPicture') : t(`aiToolboxKind_${item.kind}`), +const kindWords = item => [item.kind === 'pictures' ? t(item.gallery ? 'aiMentionGallery' : 'aiMentionPicture') : t(`aiToolboxKind_${item.kind}`), item.kind === 'models' ? item.type : item.kind === 'prompts' && item.source ? t(`aiTbPromptFrom_${item.source}`) : ''].filter(Boolean).join(' · '); /** * "@" and "/" for `field` in the message box `box`. `source`: {task(), commands()}. Returns * {keydown(event) (true when the list took the key), refs(text) (what the "@" still in `text` - * point at), clear()}. + * point at), add(item) (names it at the end of the text), pictures() and count() (how many + * pictures, and things, the text names), clear()}. */ export function mentions(field, box, source) { const list = el('div', 'anomalous-ai-mention'); @@ -88,6 +93,31 @@ export function mentions(field, box, source) { box.append(list, chips); let refs = []; let open = null; // {mode: 'at' | 'slash', start, items, at} + let recent = null; // the gallery's newest pictures, read when "@" first opens + let found = { query: null, items: [] }; // what the gallery's search found for a query + let timer = null; + + /** The gallery's pictures for `query`: the newest at once, what its search finds once it answers. */ + function gallery(query) { + const keep = images => images.filter(image => image.media_type !== 'video').map(image => pictureItem(image, true)); + if (!recent) { + recent = []; + galleryPictures(1, RECENT).then(({ images }) => { + recent = keep(images); + if (open?.mode === 'at') update(); + }).catch(() => {}); + } + clearTimeout(timer); + if (query.length >= 2 && found.query !== query) { + timer = setTimeout(() => { + galleryPictures(1, SHOWN, query).then(({ images }) => { + found = { query, items: keep(images) }; + if (open?.mode === 'at') update(); + }).catch(() => {}); + }, SEARCH_MS); + } + return { newest: recent, searched: found.query === query ? found.items : [] }; + } const close = () => { open = null; @@ -174,7 +204,12 @@ export function mentions(field, box, source) { const words = query.toLowerCase(); items = source.commands().filter(command => !words || command.name.toLowerCase().includes(words) || command.id.includes(words)); } else { - items = searchItems(everything(source.task()), query).slice(0, SHOWN); + const all = everything(source.task()); + const { newest, searched } = gallery(query); + const known = new Set(all.filter(item => item.kind === 'pictures').map(item => item.id)); + const matched = searchItems([...all, ...newest.filter(item => !known.has(item.id))], query); + const shown = new Set(matched.filter(item => item.kind === 'pictures').map(item => item.id)); + items = [...matched, ...searched.filter(item => !shown.has(item.id))].slice(0, SHOWN); } open = { mode: slash ? 'slash' : 'at', start: caret - query.length - 1, items, at: 0 }; draw(); @@ -205,6 +240,15 @@ export function mentions(field, box, source) { return false; }, refs: text => refs.filter(ref => text.includes(`@${ref.label}`)).map(({ kind, id, label }) => ({ kind, id, label })), + add(item) { + if (refs.some(ref => ref.kind === item.kind && ref.id === item.id)) return; + const text = field.value.replace(/\s+$/, ''); + field.value = `${text}${text ? ' ' : ''}@${item.title} `; + refs.push({ kind: item.kind, id: item.id, label: item.title, item }); + drawChips(); + }, + pictures: () => refs.filter(ref => ref.kind === 'pictures' && field.value.includes(`@${ref.label}`)).length, + count: () => refs.filter(ref => field.value.includes(`@${ref.label}`)).length, clear() { refs = []; close(); diff --git a/web/modules/ui_ai_message.js b/web/modules/ui_ai_message.js index 744a3a3..bb2e9b0 100644 --- a/web/modules/ui_ai_message.js +++ b/web/modules/ui_ai_message.js @@ -1,13 +1,19 @@ /** * The AI workbench's message box (ui_ai_workbench.js): under a task's timeline, where the user * writes to the AI on that task, with "@" to point at things and "/" for commands - * (ui_ai_mention.js). The text and what its "@" point at go to `onSend(text, refs)`; the AI gets + * (ui_ai_mention.js), and a picture button for pictures from the gallery (ui_ai_gallery_pick.js). + * The text and what its "@" point at go to `onSend(text, refs)`; the AI gets * them the next time it asks (get_answers), which the hint says, and says to ask in the AI app * once the task ended. Made once per task, so half-typed text survives the task's redraws. */ import { translate as t } from './locales.js'; import { mentions } from './ui_ai_mention.js'; +import { galleryPicker } from './ui_ai_gallery_pick.js'; +import { iconButton } from './ui_ai_icons.js'; + +const MAX_REFS = 12; // api/ai_message_refs.py +const MAX_PICTURES = 8; // a message's pictures, as ai_pictures.py hands them function el(tag, className, text) { const node = document.createElement(tag); @@ -49,7 +55,15 @@ export function messageBox(onSend, source) { }); line.append(field, send); box.append(line, note); - if (source) at = mentions(field, box, source); + if (source) { + at = mentions(field, box, source); + const room = () => Math.min(MAX_PICTURES - at.pictures(), MAX_REFS - at.count()); + const picker = galleryPicker(box, room, (items) => { + for (const item of items) at.add(item); + field.focus(); + }); + line.prepend(iconButton('anomalous-ai-message-pics', 'image', t('aiPicsButton'), () => picker.toggle(), { iconOnly: true, size: 17 })); + } box.note = note; return box; } diff --git a/web/modules/ui_ai_task_list.js b/web/modules/ui_ai_task_list.js index afebb91..ef9e778 100644 --- a/web/modules/ui_ai_task_list.js +++ b/web/modules/ui_ai_task_list.js @@ -101,31 +101,54 @@ export function renderTaskList(box, view) { box.replaceChildren(head, actions, ...(filters.children.length > 1 ? [filters] : []), list); } -const PEEK_LEAVE_MS = 280; +const PEEK_LEAVE_MS = 350; +const PEEK_SLACK = 12; /** * The button that shows or folds the column (`onToggle()`), for the window's title bar. While - * the column is folded, the pointer on it, or on the column it brings, shows the column over - * the page; leaving both, or picking a task, puts it away again. + * the column is folded, the pointer on it brings the column over the page; it stays while the + * pointer is anywhere in the box around the button and the column (the way down between them + * too, however slowly it is crossed), and goes a moment after the pointer leaves that box, or + * at once when a task is picked. */ export function listToggle(page, onToggle) { - const node = iconButton('anomalous-ai-list-toggle', 'sidebar', t('aiTasksToggle'), () => { + let timer = null; + const stop = () => { + clearTimeout(timer); + timer = null; page.classList.remove('is-peeking'); + document.removeEventListener('pointermove', follow); + }; + const node = iconButton('anomalous-ai-list-toggle', 'sidebar', t('aiTasksToggle'), () => { + stop(); onToggle(); }, { iconOnly: true }); - let timer = null; - const peek = (on) => { + // The box around the button and the column it brings: the pointer inside keeps it shown. + function inside(x, y) { + const list = page.querySelector('.anomalous-ai-tasks')?.getBoundingClientRect(); + const button = node.getBoundingClientRect(); + if (!list?.width) return false; + return x >= Math.min(list.left, button.left) - PEEK_SLACK && x <= Math.max(list.right, button.right) + PEEK_SLACK + && y >= Math.min(list.top, button.top) - PEEK_SLACK && y <= Math.max(list.bottom, button.bottom) + PEEK_SLACK; + } + function follow(event) { + if (!page.isConnected || !node.isConnected || !page.classList.contains('is-peeking')) return stop(); + if (inside(event.clientX, event.clientY)) { + clearTimeout(timer); + timer = null; + } else if (!timer) { + timer = setTimeout(stop, PEEK_LEAVE_MS); + } + } + node.addEventListener('pointerenter', () => { + if (!page.classList.contains('tasks-collapsed')) return; clearTimeout(timer); - if (on) page.classList.toggle('is-peeking', page.classList.contains('tasks-collapsed')); - else timer = setTimeout(() => page.classList.remove('is-peeking'), PEEK_LEAVE_MS); - }; - node.addEventListener('pointerenter', () => peek(true)); - node.addEventListener('pointerleave', () => peek(false)); + timer = null; + page.classList.add('is-peeking'); + document.addEventListener('pointermove', follow); + }); if (!page.dataset.peeks) { page.dataset.peeks = '1'; - const inList = target => Boolean(target?.closest?.('.anomalous-ai-tasks')); - page.addEventListener('pointerover', (event) => { if (page.classList.contains('is-peeking') && inList(event.target)) peek(true); }); - page.addEventListener('pointerout', (event) => { if (inList(event.target) && !inList(event.relatedTarget)) peek(false); }); page.addEventListener('click', (event) => { if (event.target.closest('.anomalous-ai-task-row')) page.classList.remove('is-peeking'); }); } return node; diff --git a/web/modules/ui_model_import.js b/web/modules/ui_model_import.js index f6c0ead..944f451 100644 --- a/web/modules/ui_model_import.js +++ b/web/modules/ui_model_import.js @@ -18,11 +18,11 @@ import { showWorkbenchToast } from './ui_prompt_toast.js'; import { openSettingsPage } from './ui_settings_page.js'; const KINDS = ['checkpoints', 'loras', 'vae', 'text_encoders', 'diffusion_models', 'controlnet', 'embeddings', - 'clip_vision', 'upscale_models', 'ipadapter']; + 'clip_vision', 'upscale_models', 'ipadapter', 'model_patches']; const LEVELS = { sure: ['●●●', 'doctorLevelSure'], likely: ['●●○', 'doctorLevelLikely'] }; const ACCEPT = '.safetensors,.sft,.gguf,.ckpt,.pt,.pth,.bin'; // Types whose files serve several base models: no base model folder. -const SHARED = ['vae', 'text_encoders', 'clip_vision', 'upscale_models', 'ipadapter']; +const SHARED = ['vae', 'text_encoders', 'clip_vision', 'upscale_models', 'ipadapter', 'model_patches']; function el(tag, className, text) { const node = document.createElement(tag); diff --git a/web/styles.css b/web/styles.css index 3e1c000..8237d75 100644 --- a/web/styles.css +++ b/web/styles.css @@ -44,5 +44,5 @@ @import url("./styles/43-ai-toolbox.css?v=20261024-facetmenu"); @import url("./styles/44-ai-status.css?v=20261024-stop"); @import url("./styles/45-ai-side-views.css?v=20261024-grip"); -@import url("./styles/46-ai-mention.css?v=20261024-mention2"); +@import url("./styles/46-ai-mention.css?v=20261024-pics"); @import url("./styles/47-ai-decisions.css?v=20261024-links"); diff --git a/web/styles/46-ai-mention.css b/web/styles/46-ai-mention.css index 394800b..af99532 100644 --- a/web/styles/46-ai-mention.css +++ b/web/styles/46-ai-mention.css @@ -68,3 +68,84 @@ .anomalous-ai-mention-chip-name { min-width: 0; overflow: hidden; color: var(--amb-text-main); text-overflow: ellipsis; white-space: nowrap; } .anomalous-ai-mention-chip-x { display: inline-flex; padding: 3px; border: 0; border-radius: 5px; background: none; color: var(--amb-text-soft); cursor: pointer; } .anomalous-ai-mention-chip-x:hover { background: rgba(var(--amb-ink-rgb), 0.1); color: var(--amb-text-main); } + +/* The picture button and its panel of gallery pictures (ui_ai_gallery_pick.js), over the box like the list. */ +.anomalous-ai-message-pics { + display: inline-flex; + flex: none; + align-items: center; + justify-content: center; + width: 36px; + border: 1px solid var(--amb-border); + border-radius: 8px; + background: none; + color: var(--amb-text-soft); + cursor: pointer; +} +.anomalous-ai-message-pics:hover, .anomalous-ai-message-pics:focus-visible { border-color: var(--amb-link); color: var(--amb-text-main); outline: none; } +.anomalous-ai-pics { + position: absolute; + z-index: 6; + right: 12px; + bottom: calc(100% - 6px); + left: 12px; + display: flex; + flex-direction: column; + gap: 8px; + height: min(420px, 60vh); + padding: 10px; + border: 1px solid var(--amb-border-strong, var(--amb-border)); + border-radius: 12px; + background: var(--amb-bg-panel); + box-shadow: 0 16px 44px rgba(0, 0, 0, 0.42); +} +.anomalous-ai-pics-search { + flex: none; + padding: 7px 11px; + border: 1px solid var(--amb-border); + border-radius: 8px; + background: var(--amb-lt-sunken, rgba(0, 0, 0, 0.25)); + color: var(--amb-text-main); + font: inherit; + font-size: 13px; +} +.anomalous-ai-pics-search:focus { border-color: color-mix(in srgb, var(--amb-link) 60%, transparent); outline: none; } +.anomalous-ai-pics-grid { + display: grid; + flex: 1; + grid-template-columns: repeat(auto-fill, minmax(92px, 1fr)); + grid-auto-rows: max-content; + gap: 6px; + min-height: 0; + overflow-y: auto; +} +.anomalous-ai-pics-tile { + position: relative; + aspect-ratio: 1; + padding: 0; + overflow: hidden; + border: 2px solid transparent; + border-radius: 8px; + background: rgba(var(--amb-ink-rgb), 0.05); + cursor: pointer; +} +.anomalous-ai-pics-tile img { display: block; width: 100%; height: 100%; object-fit: cover; } +.anomalous-ai-pics-tile:hover { border-color: color-mix(in srgb, var(--amb-link) 55%, transparent); } +.anomalous-ai-pics-tile.is-picked { border-color: var(--amb-link); } +.anomalous-ai-pics-mark { + position: absolute; + top: 5px; + right: 5px; + display: none; + align-items: center; + justify-content: center; + width: 20px; + height: 20px; + border-radius: 50%; + background: var(--amb-link); + color: #fff; +} +.anomalous-ai-pics-tile.is-picked .anomalous-ai-pics-mark { display: flex; } +.anomalous-ai-pics-none { grid-column: 1 / -1; padding: 18px 6px; color: var(--amb-text-soft); font-size: 12.5px; text-align: center; } +.anomalous-ai-pics-foot { display: flex; flex: none; align-items: center; gap: 10px; } +.anomalous-ai-pics-count { flex: 1; min-width: 0; color: var(--amb-text-dim); font-size: 12px; }